SWTBench profiling

[simonrosenberg]

GH_TOKEN=... gh api
repos/OpenHands/software-agent-sdk/actions/workflows/run-eval.yml/dispatches
-f inputs[ref]=main
-f inputs[benchmark]=swtbench
-f inputs[eval_limit]=100
-f inputs[model_ids]=gpt-5-mini-2025-08-07
-f inputs[sdk_ref]=main
-f inputs[benchmarks_branch]=feature/swtbench-profiling
-f inputs[eval_branch]=feature/swtbench-profiling
-f inputs[max_retries]=0
-f inputs[reason]='swtbench profiling'

[simonrosenberg]

GH_TOKEN=... gh api
repos/OpenHands/software-agent-sdk/actions/workflows/run-eval.yml/dispatches
-f inputs[ref]=main
-f inputs[benchmark]=swtbench
-f inputs[eval_limit]=100
-f inputs[model_ids]=gpt-5-mini-2025-08-07
-f inputs[sdk_ref]=main
-f inputs[benchmarks_branch]=feature/swtbench-profiling
-f inputs[eval_branch]=feature/swtbench-profiling
-f inputs[max_retries]=0
-f inputs[reason]='swtbench profiling'

Triggered SWTBench profiling run.

• Dispatch: OpenHands/software-agent-sdk run-eval.yml workflow_dispatch run 21002217458 (main) — https://github.com/OpenHands/software-agent-sdk/actions/runs/21002217458
  Inputs: benchmark=swtbench, eval_limit=100, model_ids=gpt-5-mini-2025-08-07, sdk_ref=main, benchmarks_branch=feature/swtbench-profiling, eval_branch=feature/swtbench-profiling, max_retries=0, reason="swtbench profiling"
• Downstream evaluation: OpenHands/evaluation eval-job.yml run 21002222706 (branch feature/swtbench-profiling) — https://github.com/OpenHands/evaluation/actions/runs/21002222706

Both workflows succeeded. Downstream comment steps skipped (workflow_dispatch without PR context).

Update on SWTBench profiling run: eval job appears stuck.

Job: evaluation-jobs/eval-eval-21002222706-gpt-5-mini (from workflow_dispatch 21002222706) has been Running ~3h48m with 0/1 completions; pod eval-eval-21002222706-gpt-5-mini-qbtvp is 2/2 Ready, no restarts.
/tmp/eval.log shows swtbench-eval reached 96% (93/97) and stopped emitting output ~2h ago; main eval processes are idle.
Inside the pod’s dind, container sharp_nightingale has been running ~1h50, executing conda env create --file environment.yml (heavy env with pyqt/wxpython/gtk3). It is consuming CPU (~100%) and preventing completion; log driver is none so no stdout.
Other exec.eval containers are idle (tail -f /dev/null).
Likely stalled on environment solve/build for the last SWTBench instance. Options: wait longer, or kill the container/job and re-trigger (ideally with a prebuilt env/image to avoid per-instance conda env creation). Let me know and I can take the next action.

[simonrosenberg]

GH_TOKEN=... gh api
repos/OpenHands/software-agent-sdk/actions/workflows/run-eval.yml/dispatches
-f inputs[benchmark]=swtbench
-f inputs[eval_limit]=100
-f inputs[model_ids]=gpt-5-mini-2025-08-07
-f inputs[sdk_ref]=main
-f inputs[benchmarks_branch]=feature/mamba-swtbench-profiling
-f inputs[eval_branch]=feature/swtbench-profiling
-f inputs[max_retries]=0
-f inputs[reason]='swtbench profiling'

Triggered the workflow with the provided command.\n\n- Workflow: Run Eval (swtbench) swtbench profiling\n- Run: https://github.com/OpenHands/software-agent-sdk/actions/runs/21003153892\n- Status: completed (success)\n- Run number: 906\n- Branch/SHA: main @ ef1c2dce89adba7e0b814c36fd75c2625a211b5e\n- Timing: started 2026-01-14T17:13:37Z, finished 2026-01-14T17:13:51Z\n\nLet me know if you want any follow-up or logs.

Investigation summary for the swtbench run (job eval-eval-21003158960-gpt-5-mini):\n\n- The eval pod is still running because two dind workers are stuck at :\n - baf573bd9071 (epic_colden), started ~19:16Z, PID 244840\n - 02a8d1c91614 (brave_davinci), started ~19:21Z, PID 272683\n- Output counts show only 96 unique instances processed:\n - output.jsonl: 96 lines\n - output.critic_attempt_1/2 combined: 96 unique instance_ids\n - output_errors.jsonl: 0 lines\n- This means 4 instances never ran; the harness logged "Evaluation complete" but we are missing 4/100 due to the stuck env-build containers.\n\nImpact: the swtbench evaluation is incomplete (96/100) and the job will not terminate until those containers are cleaned up. Suggest killing the two stuck containers (or adding timeouts/cleanup in the workflow) and rerunning the missing instances.

• The job is stuck on two dind workers pegged at ~100% CPU running mamba env create --file environment.yml (confirmed mamba, not conda).
• Containers baf573bd9071 and 02a8d1c91614 are each running mamba env create at ~100% CPU.
• This is preventing completion; 4 instances remain unevaluated (only 96/100 processed).

[simonrosenberg]

Triggered micromamba/timeout repro run for SWTBench and it completed successfully.

Changes in branch:

• Switched SWTBench patching to install/use micromamba (lighter libmamba solver) and wrapped env create/update/install calls with a 5-minute timeout to prevent solver hangs.
• Branches: benchmarks feature/mamba-swtbench-profiling (commit 76d3848) + eval feature/swtbench-profiling.

Run details:

• SDK dispatch: run-eval.yml workflow_dispatch run 21011825818 (main) — https://github.com/OpenHands/software-agent-sdk/actions/runs/21011825818
  Inputs: benchmark=swtbench, eval_limit=100, model_ids=gpt-5-mini-2025-08-07, sdk_ref=main, benchmarks_branch=feature/mamba-swtbench-profiling, eval_branch=feature/swtbench-profiling, max_retries=0, reason="swtbench micromamba profiling"
• Benchmarks build: build-swtbench-images.yml run 21011904320 (branch dispatch-40e77313-c785-4f40-8c02-d075189551ff from base feature/mamba-swtbench-profiling) — https://github.com/OpenHands/benchmarks/actions/runs/21011904320 (success)
• Evaluation: eval-job.yml run 21011832642 (feature/swtbench-profiling) — https://github.com/OpenHands/evaluation/actions/runs/21011832642 (success)
  K8s job: evaluation-jobs/eval-eval-21011832642-gpt-5-mini (swtbench, sdk commit ef1c2dc). Completed without stalls; env builds proceeded with micromamba+5m timeout.

Next: artifacts (output.jsonl/report.json) are in the evaluation run; PUSH_TO_INDEX=false for this trigger. Let me know if you want them pulled or pushed to index.

I pulled the GCS build artifact for this run (gs://openhands-evaluation-results/eval-21011832642-gpt-5-mini_litellm_proxy-gpt-5-mini-2025-08-07_26-01-15-00-54.tar.gz) and unpacked the build logs tar (/tmp_eval_artifacts/
build_logs/build-logs-21011904320). The manifest lists every per-instance image build, but the corresponding log files in builds/eth-sri/SWT-bench_Verified_bm25_27k_zsp/test/logs/.../build-*.log are zero bytes—so there’s no
timing detail captured there.

On the evaluation side, output.jsonl has zero error entries and report.json shows 93 submitted / 89 completed / 4 error instances (all matplotlib failures), with no indication of env build timeouts. Searching the eval logs
for “timeout” only hits command wrappers ([timeout: 120000.0s]), not micromamba env creation errors, and there are no micromamba strings in the logs (upstream doesn’t echo the solver command).

So: all micromamba env builds appear to have succeeded, and no installs were cancelled. However, the GCS build logs are empty, so I can’t provide precise build durations. If we need timing, we’ll have to enable verbose
logging in the build workflow or in the micromamba patch to emit start/end timestamps around micromamba env create/update/install.