Include AIFunctionDeclaration.ReturnJsonSchema in gen_ai.tool.definitions?

[verdie-g]

While I was adding a span tag to include the tool definitions, I noticed that it is actually already supported by MEAI (again!). We are not using that version yet but I was wondering if AIFunctionDeclaration.ReturnJsonSchema should be included too in gen_ai.tool.definitions? We would like anything that can impact the response of the LLM to be logged.

extensions/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/OpenTelemetryChatClient.cs

Lines 401 to 408 in 01354ad

	_ = activity.AddTag(
	OpenTelemetryConsts.GenAI.Tool.Definitions,
	JsonSerializer.Serialize(options.Tools.OfType<AIFunctionDeclaration>().Select(t => new OtelFunction
	{
	Name = t.Name,
	Description = t.Description,
	Parameters = t.JsonSchema,
	}), OtelContext.Default.IEnumerableOtelFunction));

[stephentoub]

The otel spec doesn't have support for return schemas (neither do most llm providers). I'm not against including it, but I'd prefer to wait until otel standardizes it.

[verdie-g]

neither do most llm providers

Hmm isn't that a miss here (:

https://github.com/googleapis/google-cloud-dotnet/pull/15134/files#r2416606899

The otel spec doesn't have support for return schemas

It doesn't seem like a strict schema is defined for gen_ai.tool.definitions so it feels like we can just put whatever?

https://github.com/open-telemetry/semantic-conventions/blob/4f638232672366897e483ebf13d22a21dc4c1364/model/gen-ai/registry.yaml#L299

Additionally, I noticed that the example has a

"type": "function",

It would be nice to also include web search, code interpreter, etc. in gen_ai.tool.definitions

[stephentoub]

Hmm isn't that a miss here (:

It was deliberate. I'd initially added it, but it failed for really common schemas, like string or arrays. I could not find any documentation about what else might be invalid schema according to gemini's handling, and thus opted to leave it out rather than risk it blowing up randomly based on some corner case I'm not aware of. The response schema doesn't affect what the model produces, it could only serve to inform it with a bit more information about how to interpret function results it gets back, so it did not seem like the right trade-off right now.

It doesn't seem like a strict schema is defined for gen_ai.tool.definitions

It's being worked on, e.g.
open-telemetry/semantic-conventions#2793
which does add a response schema, and I'd like whatever we do to match, hence my preference to wait until those discussions wrap up.

[verdie-g]

Makes perfect sense