Recommended strategy to handle invalid tool input

[verdie-g]

If the LLM returns a tool call with an object that doesn't respect the tool input schema, a JsonException will be raised and the message Error: Function failed is returned to the LLM.

I'm trying to see what the possibilities are to make the LLM retry the tool call. I'm experimenting with FunctionInvokingChatClient.IncludeDetailedErrors and I could try with a custom FunctionInvoker to send back a more explicit message to ask for a retry.

Would you have any other suggestion?

[stephentoub]

Both of those are valid. As are:

• deriving from FunctionInvokingChatClient and overriding methods like CreateResponseMessages
• wrapping the AIFunctions with your own DelegatingAIFunctions that catch the inner exceptions and return whatever new result you want to use instead
• inserting your own middleware IChatClient after the FICC that inspects FunctionResultContent that comes through and rewriting the Result based on a non-null Exception it contains

[ouvreboite]

@stephentoub would it make sense to push some of that logic into MEAI? LLM calling with improper schema is quite common (incorrect enum value, string instead of int, ...), so that seems like a shared concern.

As offering good standard behavior may be difficult, one possible solution would be to still call the tool when the marshalling of the argument fails, to let the developper handle it as they see fit.

Similarly to how IServiceProvider, CancellationToken and AIFunctionArguments are exposed to the tool (as possible argument), we could catch the potential deserialization exception in InvokeCoreAsync.InvokeCoreAsync and expose it to the tool as a custom AiArgumentDeserializationException (or other)

So a tool implementation could look like:

// in MEAI
public class AiArgumentDeserializationException(string Parameter, Exception Inner) : Exception;

// in Custom agent code
public enum Foo{Bar}
public string MyTool(Foo foo, AIFunctionArguments args, AiArgumentDeserializationException ex){
  if(ex!= null){
    //developper has access to the inner exception and the raw argument and can shape its message how it wants
    return $"Oups, looks like your {ex.Parameter} argument's value {args[ex.Parameter]?.ToString()} is invalid: {ex.Inner.Message}"; 
  }
  //happy path
  return "success";
}

[verdie-g]

Talked offline with @ouvreboite, after thinking exactly what our problem is we came up with this "While the happy path works great, we want MEAI to be more reliable to LLM behaving weirdly".

Our two problems of the day are tool calls not respecting the tool input schema, and gemini returning malformed_function_call.

That latter is very annoying because it is returned even through the OpenAPI endpoint of Vertex AI, so the OpenAI client just throws (https://issuetracker.google.com/u/2/issues/455479512). We can't wait any longer for googleapis/google-cloud-dotnet#13815 so we will copy your PR to support MEAI and I think we'll get lucky enough to reach that code path that returns a decent error message to the LLM.

The second problem about tool call with invalid arguments, IncludeDetailedErrors should help but we would want more flexibility about what is returned to the LLM. For example, we would like to customize the error message to make it clearer to the LLM that it should retry the tool. Or maybe if the tool call contains an enum member that just does not exist, we would like to try returning a better message than the generated JsonException. For this problem, would it make sense to have a FunctionInvokingChatClient option that generates the error message from the exception? A Func<Exception, string> or Func<FunctionInvocationResult, string> or Func<FunctionInvocationResult, object>.

[stephentoub]

We can't wait any longer for googleapis/google-cloud-dotnet#13815 so we will copy your PR to support MEAI

@amanda-tarafa, can it be published soon?

For this problem, would it make sense to have a FunctionInvokingChatClient option that generates the error message from the exception? A Func<Exception, string> or Func<FunctionInvocationResult, string> or Func<FunctionInvocationResult, object>.

FunctionInvoker isn't sufficient?

ficc.FunctionInvoker = async (ctx, ct) =>
{
    try
    {
        return await ctx.Function.InvokeAsync(ctx.Arguments, ct);
    }
    catch (Exception ex)
    {
        return $"Something went wrong. This is a customized message about it. {ex}";
    }
};

[verdie-g]

FunctionInvoker isn't sufficient?

I didn't realize it was that easy to implement. That should work, thanks!

[amanda-tarafa]

I'm on it, expectation is that we can release some time this week.

[verdie-g]

@stephentoub one last question maybe, with a pattern like to try to force a retry in case of incoherent input

ficc.FunctionInvoker = async (ctx, ct) =>
{
    try
    {
        return await ctx.Function.InvokeAsync(ctx.Arguments, ct);
    }
    catch (MyRetryException e)
    {
        return $"""
            Something went wrong: {e.Message}.

            Fix the errors and try again.
       """;
    }
};

how can I implement a retry limit?

MaximumConsecutiveErrorsPerRequest will not work because no exceptions are thrown back to the FICC.