MEAI representation for Mistral built-in tools

[verdie-g]

Mistral's /v1/conversations supports 4 built-in tools. I would like to confirm how to represent them with MEAI objects.

Input

_client.GetResponseAsync("Hi", new ChatOptions
        {
            Tools =
            [
                new HostedWebSearchTool(),
                new HostedCodeInterpreterTool(),
                new HostedImageGenerationTool(),
                new HostedFileSearchTool
                {
                    Inputs = [new HostedVectorStoreContent("019b6700-a895-75a5-81c7-318f97645bd7")],
                },
            ],
        }

would generate this request

{
  "model": "mistral-small-latest",
  "inputs": [
    {
      "type": "message.input",
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Hi"
        }
      ]
    }
  ],
  "tools": [
    {
      "type": "web_search"
    },
    {
      "type": "code_interpreter"
    },
    {
      "type": "image_generation"
    },
    {
      "type": "document_library",
      "library_ids": [
        "019b6700-a895-75a5-81c7-318f97645bd7"
      ]
    }
  ]
}

Q1: Is HostedVectorStoreContent the right content to specify a library id?

Output

Web Search

            {
              "object": "conversation.response",
              "conversation_id": "conv_019b655c73d575a4b026a57753826cf8",
              "outputs": [
                {
                  "object": "entry",
                  "type": "tool.execution",
                  "created_at": "2025-12-28T14:28:36.565958Z",
                  "completed_at": "2025-12-28T14:28:37.153563Z",
                  "id": "tool_exec_019b655c74557142bf221e736c80708a",
                  "name": "web_search",
                  "arguments": "{\"query\": \"Mistral AI\"}",
                  "function": "web_search",
                  "info": {}
                },
                {
                  "object": "entry",
                  "type": "message.output",
                  "created_at": "2025-12-28T14:28:37.304953Z",
                  "completed_at": "2025-12-28T14:28:38.278256Z",
                  "id": "msg_019b655c77387030959b66a714ecc98d",
                  "model": "mistral-small-latest",
                  "role": "assistant",
                  "content": [
                    {
                      "type": "text",
                      "text": "The first result from the web search for \"Mistral AI\" is their official website..."
                    },
                    {
                      "type": "tool_reference",
                      "tool": "web_search",
                      "title": "Frontier AI LLMs, assistants, agents, services | Mistral AI",
                      "url": "https://mistral.ai/",
                      "favicon": null,
                      "description": "The most powerful AI platform for enterprises..."
                    },
                    {
                      "type": "text",
                      "text": "."
                    }
                  ]
                }
              ],
              "usage": {
                "prompt_tokens": 177,
                "completion_tokens": 134,
                "total_tokens": 4240,
                "connector_tokens": 3929,
                "connectors": {
                  "web_search": 1
                }
              }
            }

• There is no WebSearchToolCallContent/WebSearchToolResultContent so I'm just using a plain AIContent with additional props { toolName: "web_search", arguments: "{\"query\": \"Mistral AI\"}" }
• In the second message, text is mapped to a TextContent, and tool_reference adds CitationAnnotation to the previous content

Q2: there is a concept of premium web search. How to represent it? Maybe an additional prop on HostedWebSearchTool?

Q3: $.usage.connector_tokens is the number of tokens used by the built-in tools. Should it be mapped to UsageDetails.ReasoningTokenCount?

Q4: with streaming, the tool_reference will be in its own chunk. Is it fine to create a ChatResponseUpdate with a single AIContent with the AIAnnotation?

Code Interpreter

            {
              "object": "conversation.response",
              "conversation_id": "conv_019b65926d3d7767a449919a4371273a",
              "outputs": [
                {
                  "object": "entry",
                  "type": "tool.execution",
                  "created_at": "2025-12-28T15:27:33.840174Z",
                  "completed_at": "2025-12-28T15:27:35.171693Z",
                  "id": "tool_exec_019b65926dd0713d95cce2a6ce2f8386",
                  "name": "code_interpreter",
                  "arguments": "{\"code\": \"# Calculate the sum of 2 and 2\\nresult = 2 + 2\\nresult\"}",
                  "function": "code_interpreter",
                  "info": {
                    "code": "# Calculate the sum of 2 and 2\nresult = 2 + 2\nresult",
                    "code_output": "4\n"
                  }
                },
                {
                  "object": "entry",
                  "type": "message.output",
                  "created_at": "2025-12-28T15:27:35.235560Z",
                  "completed_at": "2025-12-28T15:27:35.307702Z",
                  "id": "msg_019b6592734371a29d7338184d7c692d",
                  "model": "mistral-small-latest",
                  "role": "assistant",
                  "content": "The result of \\(2 + 2\\) is \\(4\\)."
                }
              ],
              "usage": {
                "prompt_tokens": 79,
                "completion_tokens": 44,
                "total_tokens": 136,
                "connector_tokens": 13,
                "connectors": {
                  "code_interpreter": 1
                }
              }
            }

• For the first message, I create two contents. One CodeInterpreterToolCallContent with "{\"code\": \"# Calculate the sum of 2 and 2\\nresult = 2 + 2\\nresult\"}" as input and CodeInterpreterToolResultContent with "4\n" as output.
• The second message has one TextContent

Q5: any risk to represent the first message as 2 AIContents?

Image Generation

            {
              "object": "conversation.response",
              "conversation_id": "conv_019b659f0f00701d9f3899641b644604",
              "outputs": [
                {
                  "object": "entry",
                  "type": "tool.execution",
                  "created_at": "2025-12-28T15:41:21.719204Z",
                  "completed_at": "2025-12-28T15:41:26.056385Z",
                  "id": "tool_exec_019b659f0fb77108a8030f3b7f4678ef",
                  "name": "image_generation",
                  "arguments": "{\"prompt\": \"a red square\"}",
                  "function": "generate_image",
                  "info": {}
                },
                {
                  "object": "entry",
                  "type": "message.output",
                  "created_at": "2025-12-28T15:41:27.072608Z",
                  "completed_at": "2025-12-28T15:41:27.072608Z",
                  "id": "msg_019b659f240b70b289530b582990efbb",
                  "model": "mistral-small-latest",
                  "role": "assistant",
                  "content": [
                    {
                      "type": "tool_file",
                      "tool": "image_generation",
                      "file_id": "cf11559a-39cb-487d-acc1-19a86750601c",
                      "file_name": "image_generated_0",
                      "file_type": "png"
                    }
                  ]
                }
              ],
              "usage": {
                "prompt_tokens": 121,
                "completion_tokens": 162,
                "total_tokens": 440,
                "connector_tokens": 157,
                "connectors": {
                  "image_generation": 1
                }
              }
            }

• First message has a ImageGenerationToolCallContent. Its ImageId is the same as the MessageId. The arguments are added as additional props
• The second message has a ImageGenerationToolResultContent. The file_name and file_type are added as additional props.

Q6: ImageGenerationToolCallContent.ImageId is an awkward name in that case since we don't have the image yet. Shouldn't it be GenerationId or just Id?

File Search Tool

            {
              "object": "conversation.response",
              "conversation_id": "conv_019b6701d92076cba7e28a80a6097456",
              "outputs": [
                {
                  "object": "entry",
                  "type": "tool.execution",
                  "created_at": "2025-12-28T22:08:53.228375Z",
                  "completed_at": "2025-12-28T22:08:53.804397Z",
                  "id": "tool_exec_019b6701d9ec7379b0f9d7031edd6f13",
                  "name": "document_library",
                  "arguments": "{\"libraries_ids\": [\"019b6700-a895-75a5-81c7-318f97645bd7\"], \"query\": \"cats\"}",
                  "function": "library_search",
                  "info": {}
                },
                {
                  "object": "entry",
                  "type": "message.output",
                  "created_at": "2025-12-28T22:08:53.886176Z",
                  "completed_at": "2025-12-28T22:09:00.001755Z",
                  "id": "msg_019b6701dc7e715f84c649b39aa6d144",
                  "model": "mistral-small-latest",
                  "role": "assistant",
                  "content": "### Cat Library Search Results\n\n#### cat_document_1.txt\n- **Description**: ..."
                }
              ],
              "usage": {
                "prompt_tokens": 203,
                "completion_tokens": 459,
                "total_tokens": 1353,
                "connector_tokens": 691,
                "connectors": {
                  "document_library": 1
                }
              }
            }

• In the first message, an AIContent is used with additional props { toolName: "document_library", arguments: "{\"libraries_ids\": [\"019b6700-a895-75a5-81c7-318f97645bd7\"], \"query\": \"cats\"}" }
• Second message is a TextContent

[stephentoub]

I'm missing context. Are you implementing IChatClient to talk to the mistral endpoint?

[verdie-g]

Yes I'm implementing an IChatClient for the Mistral endpoint /v1/conversations

[stephentoub]

Is HostedVectorStoreContent the right content to specify a library id?

Looks right, assuming "library" is mistral's server-side representation for a vector / file store that can be searched for RAG-like purposes.

There is no WebSearchToolCallContent/WebSearchToolResultContent

Yes, we need to add that soon but haven't yet.
#6057 (comment)

there is a concept of premium web search. How to represent it? Maybe an additional prop on HostedWebSearchTool?

Using an AdditionalProperty is reasonable. Once you start getting into provider-specific tooling (and tooling options), you can also consider either creating your own custom AITool-derived type, or if you have an underlying library that provides its own tool representation, have an AsAITool method that just wraps it with an AITool shim and then have your IChatClient implementation understand how to unwrap it. You can do that in addition to looking for AdditionalProperties.

$.usage.connector_tokens is the number of tokens used by the built-in tools. Should it be mapped to UsageDetails.ReasoningTokenCount?

Probably better to have it in AdditionalCounts, since if I'm understanding correctly it's not actually about reasoning. Once counts for tool use become more common place, we can also look at exposing something strongly-typed.

with streaming, the tool_reference will be in its own chunk. Is it fine to create a ChatResponseUpdate with a single AIContent with the AIAnnotation?

Yup

any risk to represent the first message as 2 AIContents?

You can represent it however you like. The content should be as portable as possible, but ideally providers just ignore things they don't understand.

ImageGenerationToolCallContent.ImageId is an awkward name in that case since we don't have the image yet. Shouldn't it be GenerationId or just Id?

@ericstj?