fix: create isolated runtime per eval execution

[Chibionos]

Summary

• Create a new runtime with unique runtime_id for each eval execution
• This ensures each eval has its own LangGraph thread_id with clean state
• Prevents message accumulation across sequential eval runs

Problem

Previously, all evals shared a single runtime with the same thread_id, causing the LangGraph checkpointer to persist and accumulate messages across eval runs. This led to 400 bad request errors on sequential eval executions.

Root Cause Analysis

Regression Source

This bug was introduced in PR #1055 (akshaya/single_eval_runtime) by @akshaylive, merged on Dec 30, 2025.

Field	Value
PR	#1055
Commit	e51d942
Title	"refactor(EvalRuntimeInstance): evaluate specific runtime instance"
Rationale	"Doing this will avoid creation of temporary runtimes everywhere"

What Changed

Before PR #1055 - Each evaluation created its own runtime with a unique runtime_id:

# In execute_runtime() - called per eval
runtime = await self.factory.new_runtime(
    entrypoint=self.context.entrypoint or "",
    runtime_id=execution_id,  # Unique per eval
)

After PR #1055 - A single runtime was created and shared across all evaluations:

# In execute() - called once for all evals  
runtime = await self.factory.new_runtime(
    entrypoint=self.context.entrypoint or "",
    runtime_id=self.execution_id,  # Same for ALL evals
)

Why This Breaks Sequential Evals

When using LangGraph agents with SQLite checkpointer:

1. runtime_id maps to thread_id for LangGraph conversation state
2. All evals shared the same thread_id
3. First eval runs fine, stores conversation state in checkpointer
4. Second eval tries to continue from corrupted/unexpected state → 400 error

Why Testing Missed It

The original PR was tested with the calculator agent which doesn't use LangGraph with persistent state. The issue only manifests with stateful LangGraph agents where conversation history accumulates across the shared thread.

Solution

Each eval execution now gets its own runtime with a unique runtime_id (using execution_id). The runtime is properly disposed after the eval completes.

Test plan

• Run evaluations with multiple sequential evals (e.g., calculator_same_as_agent with 5 evals)
• Verify all evals run with isolated state
• Verify linting passes

🤖 Generated with Claude Code

[saksharthakkar]

left a comment, other than that... lgtm!

[akshaylive]

Thanks for the fix!

[akshaylive]

One minor comment: can you please rename to eval_execution_id and runtime_execution_id for clarity? I think the lack of distinction is what lead to the bug to begin with.