Skip to content

.Net: Bug: ImageContent in tool/function results is serialized to JSON instead of native format #13430

@Cozmopolit

Description

@Cozmopolit

Describe the bug

When a kernel function returns ImageContent, FunctionCallsProcessor.ProcessFunctionResult() serializes it to JSON. This prevents connectors from using native image formats in tool results, even when the underlying API supports it (Gemini 3+, Anthropic Claude).

The model receives "{ \"Data\": \"base64...\", \"MimeType\": \"image/png\" }" as text instead of an actual image it can "see".

To Reproduce

  1. Create a kernel function that returns ImageContent:

    [KernelFunction]
    public ImageContent CaptureScreenshot()
    {
        var imageBytes = TakeScreenshot();
        return new ImageContent(imageBytes, "image/png");
    }
  2. Enable auto function calling with Gemini 3 connector

  3. Inspect the API request - observe that functionResponse.response contains serialized JSON text instead of functionResponse.parts[].inlineData

Expected behavior

Connectors that support images in tool results should receive the original ImageContent object and convert it to their native format:

  • Gemini 3+: functionResponse.parts[].inlineData
  • Anthropic: ImageBlockParam in ToolResultBlockParam.Content
  • OpenAI/others: Clear error message (API does not support images in tool results)

Platform

  • Language: C#
  • Source: main branch
  • AI model: Google Gemini 3 (gemini-2.5-pro, gemini-2.5-flash)
  • IDE: any
  • OS: any

Additional context

API Image Tool Results Native Format
Google Gemini 3+ ✅ Supported functionResponse.parts[].inlineData
Anthropic Claude ✅ Supported ImageBlockParam in ToolResultBlockParam.Content
Google Gemini 2.5 and earlier ❌ Not supported -
OpenAI ❌ Not supported -

Related: PR #13419 (Anthropic Connector multimodal support)

Metadata

Metadata

Assignees

No one assigned

    Labels

    .NETIssue or Pull requests regarding .NET codebugSomething isn't workingtriage

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions