Fallback to full trajectory condensation on malformed message errors

[neubig]

Summary

When we encounter a malformed message error from the language model, we should fall back to condensing the entire trajectory into a single message, similarly to how Claude Code does it.

Background

This approach was discussed as a way to handle edge cases in condensation. Currently, when we hit malformed message errors, we don't have a robust fallback mechanism. Claude Code's approach of condensing everything into a single message could help avoid some of the edge cases we've been trying to fix (e.g., mixture of thinking blocks, wrong order of action-observation event pairs).

Proposed Solution

Implement a fallback mechanism that:

1. Detects malformed message errors from the language model
2. Falls back to condensing the entire trajectory into a single comprehensive message (similar to Claude Code's approach)
3. Keeps the existing condensation approach as the default, using this only as a fallback

Additional Context

• This should be a fallback strategy, not the default behavior
• We want to keep the option to not condense everything from scratch, as starting from scratch has a fairly high likelihood of confusing the agent (especially with open-source models)
• The existing condensation approach should remain the primary method, with this as a safety net for error cases

Related Discussion

See: #1518 (comment)

[csmith49]

See my explanation of error 2 in #1575 -- this may be the only real way to handle condensations with Opus 4.5 due to Anthropic's requirements.

One challenge we'll struggle with in the short term is recovering from things like context window exceptions: if the agent is experiencing a context window exception and the condenser has to restart entirely, we can't just use the same LLM and generate a summary without running into the exact same context window exception.

We might be able to work around that with a kind of hierarchical summarization: split the history into chunks, summarize each chunk separately, then summarize the summaries into one. We'll lose some context across each chunk boundary but maybe we can get it tuned alright.

[neubig]

We could possibly convert the trajectory into text?

[csmith49]

We actually already do that as a holdover from v0. We don't send the trajectory up as messages, we convert them all into one big string which gets sent as a single message. Not sure how that impacts the token consumption.

Some context: this was the behavior in v0 because it was surprisingly difficult to convert events into messages. With v1 we refactored the logic and can do so pretty trivially, but we haven't made that change because of the potential impact on condensation behavior.

[neubig]

Got it, makes sense!

[enyst]

One challenge we'll struggle with in the short term is recovering from things like context window exceptions: if the agent is experiencing a context window exception and the condenser has to restart entirely, we can't just use the same LLM and generate a summary without running into the exact same context window exception.

Just to note, it depends a bit, at the lower end. If the context window is exceeded with < 8k tokens or so, it may work? As I think we have a large system prompt and tools descriptions, and we can reasonably exclude them from the condensed events, even when we say “all” events.

A few other alternatives might be:
(though not all are mutually exclusive)

• catch the exception in the condenser, truncate large observations (e.g. first x… end x chars), then summarize.
• catch the exception in the condenser, drop earliest agent events until it works. This should work in all expected use cases.
• on a best effort basis, we could set the max tokens to 80% of the context window, and not apply it for the summary call. This will have exceptions of course, but it may help in common use cases.
• we could reasonably treat user messages separately, and then exclude them/most from the bulk.

We might be able to work around that with a kind of hierarchical summarization: split the history into chunks, summarize each chunk separately, then summarize the summaries into one. We'll lose some context across each chunk boundary but maybe we can get it tuned alright.

This is probably the best in some form. We’ve been trying in the past something like

user message
(summary of agent actions)
user message
(summary of agent actions)
…

Interestingly, these days it’s a… good question how often will it happen to have user messages in a long running task in the same context window. I mean, I think it’s becoming more usual that the entire context window could be occupied by tool calls. 🤔

[csmith49]

Good ideas.

I've started with the simplest approach possible (hoping it's not a problem) to reduce the error rate and then will explore some combination of approaches. Something like truncating large events + dropping early agent events will be pretty quick to pull together.

Interestingly, these days it’s a… good question how often will it happen to have user messages in a long running task in the same context window. I mean, I think it’s becoming more usual that the entire context window could be occupied by tool calls. 🤔

Absolutely. And probably makes the strategy of just summarizing agent actions between user messages worth trying again.

[csmith49]

Been working on #1596, which adds a control-flow path that will trigger a hard context reset using the summarizer if a condensation is needed but one can't be found. That PR won't implicitly handle malformed messages, but it's the first step. Next steps are capturing those malformed messages like we do context window exceptions and somehow getting them to trigger the hard context reset.