-
Notifications
You must be signed in to change notification settings - Fork 4.4k
Description
Describe the bug
When a kernel function returns ImageContent, FunctionCallsProcessor.ProcessFunctionResult() serializes it to JSON. This prevents connectors from using native image formats in tool results, even when the underlying API supports it (Gemini 3+, Anthropic Claude).
The model receives "{ \"Data\": \"base64...\", \"MimeType\": \"image/png\" }" as text instead of an actual image it can "see".
To Reproduce
-
Create a kernel function that returns
ImageContent:[KernelFunction] public ImageContent CaptureScreenshot() { var imageBytes = TakeScreenshot(); return new ImageContent(imageBytes, "image/png"); }
-
Enable auto function calling with Gemini 3 connector
-
Inspect the API request - observe that
functionResponse.responsecontains serialized JSON text instead offunctionResponse.parts[].inlineData
Expected behavior
Connectors that support images in tool results should receive the original ImageContent object and convert it to their native format:
- Gemini 3+:
functionResponse.parts[].inlineData - Anthropic:
ImageBlockParaminToolResultBlockParam.Content - OpenAI/others: Clear error message (API does not support images in tool results)
Platform
- Language: C#
- Source: main branch
- AI model: Google Gemini 3 (gemini-2.5-pro, gemini-2.5-flash)
- IDE: any
- OS: any
Additional context
| API | Image Tool Results | Native Format |
|---|---|---|
| Google Gemini 3+ | ✅ Supported | functionResponse.parts[].inlineData |
| Anthropic Claude | ✅ Supported | ImageBlockParam in ToolResultBlockParam.Content |
| Google Gemini 2.5 and earlier | ❌ Not supported | - |
| OpenAI | ❌ Not supported | - |
Related: PR #13419 (Anthropic Connector multimodal support)