Skip to content

Fix interactive responses while using .WithFiles() method #104

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Kr1sek wants to merge 1 commit into from
Closed

Fix interactive responses while using .WithFiles() method #104

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/MaIN.Services/Services/Abstract/ILLMService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ public interface ILLMService
/// <returns></returns>
Task<ChatResult?> AskMemory(Chat chat,
ChatMemoryOptions memoryOptions,
ChatRequestOptions requestOptions,
CancellationToken cancellationToken = default);

/// <summary>
Expand Down
4 changes: 2 additions & 2 deletions src/MaIN.Services/Services/LLMService/AnthropicService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@
if (HasFiles(lastMessage))
{
var result = ChatHelper.ExtractMemoryOptions(lastMessage);
var memoryResult = await AskMemory(chat, result, cancellationToken);
var memoryResult = await AskMemory(chat, result, options, cancellationToken);
resultBuilder.Append(memoryResult!.Message.Content);
lastMessage.MarkProcessed();
UpdateSessionCache(chat.Id, resultBuilder.ToString(), options.CreateSession);
Expand Down Expand Up @@ -488,7 +488,7 @@
{
requestBody["tools"] = chat.ToolsConfiguration.Tools.Select(t => new
{
name = t.Function.Name,

Check warning on line 491 in src/MaIN.Services/Services/LLMService/AnthropicService.cs

View workflow job for this annotation

GitHub Actions / build 

Dereference of a possibly null reference.
description = t.Function.Description,
input_schema = t.Function.Parameters
}).ToList();
Expand Down Expand Up @@ -531,7 +531,7 @@
return messages;
}

public async Task<ChatResult?> AskMemory(Chat chat, ChatMemoryOptions memoryOptions, CancellationToken cancellationToken = default)
public async Task<ChatResult?> AskMemory(Chat chat, ChatMemoryOptions memoryOptions, ChatRequestOptions requestOptions, CancellationToken cancellationToken = default)
{
throw new NotSupportedException("Embeddings are not supported by the Anthropic. Document reading requires embedding support.");
}
Expand Down Expand Up @@ -824,9 +824,9 @@

file class AnthropicModelListResponse
{
public List<AnthropicModelInfo> Data { get; set; }

Check warning on line 827 in src/MaIN.Services/Services/LLMService/AnthropicService.cs

View workflow job for this annotation

GitHub Actions / build 

Non-nullable property 'Data' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the property as nullable.
}

file class AnthropicModelInfo
{
public string Id { get; set; }

Check warning on line 832 in src/MaIN.Services/Services/LLMService/AnthropicService.cs

View workflow job for this annotation

GitHub Actions / build 

Non-nullable property 'Id' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the property as nullable.
Expand Down
3 changes: 2 additions & 1 deletion src/MaIN.Services/Services/LLMService/DeepSeekService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
public override async Task<ChatResult?> AskMemory(
Chat chat,
ChatMemoryOptions memoryOptions,
ChatRequestOptions requestOptions,
CancellationToken cancellationToken = default)
{
var lastMsg = chat.Messages.Last();
Expand All @@ -63,7 +64,7 @@
chat.Messages.Last().Files = [];
var result = await Send(chat, new ChatRequestOptions(), cancellationToken);
chat.Messages.Last().Content = lastMsg.Content;
return result;
return await CreateChatResult(chat, result.Message.Content, [], requestOptions);

Check warning on line 67 in src/MaIN.Services/Services/LLMService/DeepSeekService.cs

View workflow job for this annotation

GitHub Actions / build 

Dereference of a possibly null reference.
}

private string ComposeMessage(Message lastMsg, string[] filePaths)
Expand Down
3 changes: 2 additions & 1 deletion src/MaIN.Services/Services/LLMService/GeminiService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ protected override void ValidateApiKey()
public override async Task<ChatResult?> AskMemory(
Chat chat,
ChatMemoryOptions memoryOptions,
ChatRequestOptions requestOptions,
CancellationToken cancellationToken = default)
{
if (!chat.Messages.Any())
Expand All @@ -91,7 +92,7 @@ protected override void ValidateApiKey()
var retrievedContext = await kernel.AskAsync(userQuery, cancellationToken: cancellationToken);
chat.Messages.Last().MarkProcessed();
await kernel.DeleteIndexAsync(cancellationToken: cancellationToken);
return CreateChatResult(chat, retrievedContext.Result, []);
return await CreateChatResult(chat, retrievedContext.Result, [], requestOptions);
}
}

Expand Down
1 change: 1 addition & 0 deletions src/MaIN.Services/Services/LLMService/GroqCloudService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ protected override void ValidateApiKey()
public override async Task<ChatResult?> AskMemory(
Chat chat,
ChatMemoryOptions memoryOptions,
ChatRequestOptions requestOptions,
CancellationToken cancellationToken = default)
{
var lastMsg = chat.Messages.Last();
Expand Down
17 changes: 4 additions & 13 deletions src/MaIN.Services/Services/LLMService/LLMService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ public LLMService(
if (ChatHelper.HasFiles(lastMsg))
{
var memoryOptions = ChatHelper.ExtractMemoryOptions(lastMsg);
return await AskMemory(chat, memoryOptions, cancellationToken);
return await AskMemory(chat, memoryOptions, requestOptions, cancellationToken);
}

var model = KnownModels.GetModel(chat.Model);
Expand Down Expand Up @@ -90,6 +90,7 @@ public Task CleanSessionCache(string? id)
public async Task<ChatResult?> AskMemory(
Chat chat,
ChatMemoryOptions memoryOptions,
ChatRequestOptions requestOptions,
CancellationToken cancellationToken = default)
{
var model = KnownModels.GetModel(chat.Model);
Expand Down Expand Up @@ -127,18 +128,8 @@ public Task CleanSessionCache(string? id)
memory.generator._embedder._weights.Dispose();
memory.generator.Dispose();

return new ChatResult
{
Done = true,
CreatedAt = DateTime.Now,
Model = chat.Model,
Message = new Message
{
Content = memoryService.CleanResponseText(result.Result),
Role = nameof(AuthorRole.Assistant),
Type = MessageType.LocalLLM,
}
};
var tokens = await ProcessChatRequest(chat, model, userMessage, requestOptions, cancellationToken);
return await CreateChatResult(chat, tokens, requestOptions);
}

private async Task<List<LLMTokenValue>> ProcessChatRequest(
Expand Down
31 changes: 25 additions & 6 deletions src/MaIN.Services/Services/LLMService/OpenAiCompatibleService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ public abstract class OpenAiCompatibleService(
if (HasFiles(lastMessage))
{
var result = ChatHelper.ExtractMemoryOptions(lastMessage);
var memoryResult = await AskMemory(chat, result, cancellationToken);
var memoryResult = await AskMemory(chat, result, options, cancellationToken);
resultBuilder.Append(memoryResult!.Message.Content);
lastMessage.MarkProcessed();
UpdateSessionCache(chat.Id, resultBuilder.ToString(), options.CreateSession);
Expand All @@ -80,7 +80,7 @@ public abstract class OpenAiCompatibleService(
Type = TokenType.FullAnswer
});
}
return CreateChatResult(chat, resultBuilder.ToString(), tokens);
return await CreateChatResult(chat, resultBuilder.ToString(), tokens, options);
}

if (chat.ToolsConfiguration?.Tools != null && chat.ToolsConfiguration.Tools.Any())
Expand Down Expand Up @@ -128,7 +128,7 @@ await _notificationService.DispatchNotification(

lastMessage.MarkProcessed();
UpdateSessionCache(chat.Id, resultBuilder.ToString(), options.CreateSession);
return CreateChatResult(chat, resultBuilder.ToString(), tokens);
return await CreateChatResult(chat, resultBuilder.ToString(), tokens, options);
}

private async Task<ChatResult> ProcessWithToolsAsync(
Expand Down Expand Up @@ -279,7 +279,7 @@ await _notificationService.DispatchNotification(

chat.Messages.Last().MarkProcessed();
UpdateSessionCache(chat.Id, resultBuilder.ToString(), options.CreateSession);
return CreateChatResult(chat, resultBuilder.ToString(), tokens);
return await CreateChatResult(chat, resultBuilder.ToString(), tokens, options);
}

private async Task<List<ToolCall>?> ProcessStreamingChatWithToolsAsync(
Expand Down Expand Up @@ -438,6 +438,7 @@ await _notificationService.DispatchNotification(
public virtual async Task<ChatResult?> AskMemory(
Chat chat,
ChatMemoryOptions memoryOptions,
ChatRequestOptions requestOptions,
CancellationToken cancellationToken = default)
{
if (!chat.Messages.Any())
Expand All @@ -458,7 +459,7 @@ await _notificationService.DispatchNotification(
var retrievedContext = await kernel.AskAsync(userQuery, cancellationToken: cancellationToken);

await kernel.DeleteIndexAsync(cancellationToken: cancellationToken);
return CreateChatResult(chat, retrievedContext.Result, []);
return await CreateChatResult(chat, retrievedContext.Result, [], requestOptions);
}

public virtual async Task<string[]> GetCurrentModels()
Expand Down Expand Up @@ -714,8 +715,19 @@ internal static void MergeMessages(List<ChatMessage> conversation, List<Message>
}
}

protected static ChatResult CreateChatResult(Chat chat, string content, List<LLMTokenValue> tokens)
protected async Task<ChatResult> CreateChatResult(Chat chat, string content, List<LLMTokenValue> tokens, ChatRequestOptions requestOptions)
{
var responseText = string.Concat(tokens.Select(x => x.Text));

if (requestOptions.InteractiveUpdates)
{
await SendNotification(chat.Id, new LLMTokenValue
{
Type = TokenType.FullAnswer,
Text = responseText
}, true);
}

return new ChatResult
{
Done = true,
Expand All @@ -730,6 +742,13 @@ protected static ChatResult CreateChatResult(Chat chat, string content, List<LLM
}.MarkProcessed()
};
}

private async Task SendNotification(string chatId, LLMTokenValue token, bool isComplete)
{
await notificationService.DispatchNotification(
NotificationMessageBuilder.CreateChatCompletion(chatId, token, isComplete),
ServiceConstants.Notifications.ReceiveMessageUpdate);
}

internal static async Task<object[]> BuildMessagesArray(List<ChatMessage> conversation, Chat chat, ImageType imageType)
{
Expand Down
6 changes: 3 additions & 3 deletions src/MaIN.Services/Services/Steps/Commands/AnswerCommandHandler.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ public class AnswerCommandHandler(
switch (command.KnowledgeUsage)
{
case KnowledgeUsage.UseMemory:
result = await llmService.AskMemory(command.Chat,
new ChatMemoryOptions { Memory = command.Chat.Memory });
result = await llmService.AskMemory(command.Chat, new ChatMemoryOptions { Memory = command.Chat.Memory },
new ChatRequestOptions());
return result!.Message;
case KnowledgeUsage.UseKnowledge:
var isKnowledgeNeeded = await ShouldUseKnowledge(command.Knowledge, command.Chat);
Expand Down Expand Up @@ -138,7 +138,7 @@ await notificationService.DispatchNotification(NotificationMessageBuilder.Create
return result.Message;
}

var knowledgeResult = await llmService.AskMemory(chat, memoryOptions);
var knowledgeResult = await llmService.AskMemory(chat, memoryOptions, new ChatRequestOptions());
chat.Messages.Last().Content = originalContent;
return knowledgeResult?.Message;
}
Expand Down
6 changes: 3 additions & 3 deletions src/MaIN.Services/Services/Steps/Commands/FetchCommandHandler.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ private async Task<Message> HandleFileSource(FetchCommand command, Dictionary<st
{
FilesData = filesDictionary,
PreProcess = fileData.PreProcess
}
}, new ChatRequestOptions()
);

return result!.Message;
Expand All @@ -112,7 +112,7 @@ private async Task<Message> HandleWebSource(FetchCommand command, Dictionary<str
{
var memoryChat = command.MemoryChat;
var result = await llmServiceFactory.CreateService(command.Chat.Backend ?? settings.BackendType)
.AskMemory(memoryChat!, new ChatMemoryOptions { WebUrls = [webData!.Url] });
.AskMemory(memoryChat!, new ChatMemoryOptions { WebUrls = [webData!.Url] }, new ChatRequestOptions());
result!.Message.Role = command.ResponseType == FetchResponseType.AS_System ? "System" : "Assistant";
return result!.Message;
}
Expand All @@ -131,7 +131,7 @@ private async Task<Message> ProcessJsonResponse(Message response, FetchCommand c
var result = await llmServiceFactory.CreateService(command.Chat.Backend ?? settings.BackendType).AskMemory(command.MemoryChat!, new ChatMemoryOptions
{
TextData = chunks
});
}, new ChatRequestOptions());

result!.Message.Role = command.ResponseType == FetchResponseType.AS_System ? "System" : "Assistant";
var newMessage = result!.Message;
Expand Down
Loading