Eval runtimes are force-stopped after ~20–24 minutes of runtime-api inactivity, causing mid-run 404s

[simonrosenberg]

Description

During evaluations, runtimes are being force-stopped by the runtime-api’s idle GC after ~20–24 minutes with no runtime-api traffic. Once the controller stops the pod/service, any subsequent conversation/exec/health request returns 404 Remote conversation not found, aborting the run. This happens even though the evaluation is still in progress; the only trigger is that the runtime-api has not received requests for ~20+ minutes.

Impact

• Evaluations fail mid-run with 404s.
• No conversation artifacts are saved for the affected runs.

Evidence (from runtime-api logs)

• Repeated pattern: Stopping idle runtime (idle for N seconds) → Force-stopping runtime … → Runtime stopped successfully → pod/service removal messages (Calico/felix, log tailers closing).
• The 404s occur immediately after the forced stop.

[simonrosenberg]

Context: SWTBench eval run eval-21011832642-gpt-5-mini (model litellm_proxy/gpt-5-mini-2025-08-07, eval_limit=100, run_id eval_output.swtbench). Artifacts: gs://openhands-evaluation-results/eval-21011832642-gpt-5-
mini_litellm_proxy-gpt-5-mini-2025-08-07_26-01-15-00-54.tar.gz

What happened:

• 93 submitted, 89 completed, 4 error_instances (all matplotlib repro tests), 0 solver/env build issues.
• Seven instances never finished because the runtimes were force-stopped mid-run by the runtime-api idle GC (~20–24m of no runtime-api traffic), leading to 404s on the next poll/command.

Idle-GC failures (runtime force-stopped → 404):

• astropy__astropy-14995 (runtime skxskqdlcxeqfdxh): idle GC at 22:55:10Z “Stopping idle runtime (idle for 1231s)”; pod removed; instance 404s.
• astropy__astropy-14309 (runtime zflgtgabyxubpuxk): idle GC at 22:50:12Z “idle for 1482s”; pod removed; instance 404s.
• django__django-16082 (runtime yfkgrgribjfdjvdn): idle GC at 22:45:12Z “idle for 1212s”; pod removed; instance 404s.
• django__django-16493 (runtime vblciydybrfzqloa): idle GC at 23:10:10Z “idle for 1429s”; pod removed; instance 404s.
• sphinx-doc__sphinx-11445 (runtime njzrltcgyivswquw): idle GC at 22:55:09Z “idle for 1340s”; pod removed; instance 404s.
• sphinx-doc__sphinx-9461 (runtime eejrmndqmcvcszdh): idle GC at 22:55:08Z “idle for 1429s”; pod removed; instance 404s.
• sympy__sympy-12096 (runtime khlvvmqytkkmymgo): never created; runtime-api spammed “Runtime not found in k8s” then force-stopped after ~1m.

Symptom pattern:

• Runtime-api logs: “Stopping idle runtime (idle for N seconds)” → “Force-stopping runtime … / Runtime stopped successfully” → pod/service removed (Calico/felix, log tailer closed).
• Next conversation/exec poll hits 404 (Remote conversation not found), aborting the instance; no conversation archives saved.

[simonrosenberg]

GH_TOKEN=... gh api
repos/OpenHands/software-agent-sdk/actions/workflows/run-eval.yml/dispatches
-f inputs[ref]=main
-f inputs[benchmark]=swebench
-f inputs[eval_limit]=50
-f inputs[model_ids]=gpt-5-mini-2025-08-07
-f inputs[sdk_ref]=main
-f inputs[benchmarks_branch]=feature/runtime-pod-mapping
-f inputs[eval_branch]=main
-f inputs[max_retries]=0
-f inputs[reason]='debug runtime instability with 50 image run'

Tracking

• SDK workflow dispatch: success (Run Eval) — https://github.com/OpenHands/software-agent-sdk/actions/runs/20958766672
• Eval workflow: success — https://github.com/OpenHands/evaluation/actions/runs/20958773406 (swebench, eval_limit=50, model=gpt-5-mini-2025-08-07, benchmarks_branch=feature/runtime-pod-mapping, reason="debug runtime instability with 50 image run")
• Eval deployment: unique_eval_name=eval-20958773406-gpt-5-mini, namespace=evaluation-jobs, pod=eval-eval-20958773406-gpt-5-mini-ngx9p (Running). Datadog logs: https://us5.datadoghq.com/logs?query=kube_namespace%3Aevaluation-jobs%20kube_job%3Aeval-eval-20958773406-gpt-5-mini&cols=host%2Cservice&index=%2A&messageDisplay=inline&stream_sort=time%2Cdesc&viz=stream&live=true
• Benchmarks image build (feature/runtime-pod-mapping, temp branch dispatch-c4e01ecf-f5fc-491e-99ca-f1615eb431ef): success — https://github.com/OpenHands/benchmarks/actions/runs/20958818532
• Runtime/pod mapping (PR Track runtime/pod mappings per attempt #305): runtime-api logs show runtime_id+session (e.g., Starting runtime in k8s: mkbfibokdasdervb for session agent-server-5c02bc82-8b1b-4a8d-b5de-f094f673f043); eval logs carry instance/attempt with hostname runtime-mkbfibokdasdervb-... (e.g., instance django__django-12155, attempt 1, hostname runtime-mkbfibokdasdervb-86d5659c4f-pnbd4). That links instance ↔ runtime_id ↔ pod.

Failures

• django__django-10097

  • Runtime vxttxvmwywsveyrk started 2026-01-13T14:02:14Z (runtime-api logs).
  • Force-stopped at 2026-01-13T14:25:08Z with “Stopping idle runtime (idle for 1229.0 seconds)” (Datadog query: service:runtime-api vxttxvmwywsveyrk).
  • Instance log shows 404 at 14:25:08Z on /api/conversations/d4dd696c-... against that runtime; then a replacement runtime uifqvqqcvurwzwni was created and the instance later completed.

• django__django-15278

  • Runtime zjuvyyqbdegceavs started 2026-01-13T14:01:00Z (instance log).
  • Force-stopped ~2026-01-13T14:25Z; instance log shows 404 on /api/conversations/b3bd466e-... at that time and then a new runtime avcoufnsnpybvmci was created and the instance later completed.
  • Datadog: runtime-api logs for zjuvyyqbdegceavs show start; idle stop not explicitly logged in our pull, but timestamps line up with ~24m inactivity before the 404.

• sphinx-doc__sphinx-8595

  • Runtime rdnmjasdxvsdfeha started 2026-01-13T13:58:15Z (runtime_runs entry).
  • 404s later at 14:25Z on conversations 25cb0f1d-... and e35763d1-...; likely the runtime was force-stopped for idleness after ~27m with no traffic. Runtime-api logs for the runtime-id not captured in the artifact, but
    the behavior matches the idle-stop window.

• Where: benchmarks/utils/evaluation.py and benchmarks/utils/models.py.

• RuntimeRun/RemoteRuntimeAllocation model: A per-allocation record with runtime_id, session_id, runtime_url, workspace_type, resource_factor, critic_attempt (outer), retry (inner, 1‑indexed), and started_at (UTC).
  Validation requires at least runtime_id or session_id.

• Collection: On each allocation in _process_one_mp, if the workspace is APIRemoteWorkspace, it captures those fields and appends to runtime_runs. On success or failure, runtime_runs is attached to the EvalOutput (including
  error outputs), so it ends up in the serialized results.

• Logging (Datadog-friendly):

  • Info log when a runtime is allocated:
    [child] runtime allocated instance= attempt=<critic_attempt> retry= workspace= runtime_id= session_id= resource_factor=
  • Warning on runtime init failures (including “Runtime not yet ready”):
    [child] runtime init failure instance= attempt=<critic_attempt> retry= runtime_id= session_id= error=<...>

• Scope: Only remote runtimes are logged/tracked; non-remote workspaces are ignored for mapping.

• Datadog use: These structured logs let you correlate instance/attempt ↔ runtime_id/pod ↔ session_id in Datadog (search by instance= or runtime_id=), and the runtime_runs array in outputs gives a durable mapping
  even after pods are gone.