swtbench.run_infer getting stuck?

[enochii]

Hello,

When I run the command below with a small max_iterations (say 4):

python -m benchmarks.swtbench.run_infer .llm_config/$conf.json   \
        --max-iterations 4 --select instances.txt \
        --output-dir ./evaluation_results --max-attempts 1 \
        --critic finish_with_patch >> onecase-$conf.out 2>&1

It got stuck after reaching thie limit while keeping ouputing some heatbeat messages (see below). Is there something wrong on my side?

....
<messages in previous iterations omitted>

$ python /workspace/django/reproduction.py

Tokens: ↑ input 66.86K • cache hit 86.72% • ↓ output 741 • $ 0.1021

[DOCKER] {"asctime": "2026-01-14 14:08:15,906", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
Observation ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Tool: terminal
Result:
<messages omitted>

📁 Working directory: /workspace
🐍 Python interpreter: /opt/miniconda3/bin/python
❌ Exit code: 1

[DOCKER] {"asctime": "2026-01-14 14:08:16,924", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:17,934", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:18,943", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:19,952", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:20,960", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:21,968", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:22,975", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:23,984", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:24,993", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:26,005", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:27,014", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:28,023", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:29,033", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:30,042", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:31,051", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:32,060", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:33,071", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:34,079", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:35,089", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:36,096", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:37,104", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:38,133", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:39,145", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:40,154", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:41,164", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:42,173", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:43,181", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:44,190", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:45,202", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:46,212", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}
[DOCKER] {"asctime": "2026-01-14 14:08:47,222", "levelname": "INFO", "name": "uvicorn.access", "filename": "h11_impl.py", "lineno": 473, "message": "127.0.0.1:37452 - \"GET /api/conversations/3f808e14-3149-4001-9ff0-59a8d4469571 HTTP/1.1\" 200"}

[juanmichelini]

Hey @enochii ! --max-iterations 4 is very low. For reference we use --max-iterations 100 or --max-iterations 500 when aiming for a result.

Having said that, it shouldn't get stuck at 4 iterations though, does that happen with any particular instance?

[enochii]

Hi, @juanmichelini

Thanks for your quick reply.

It seems that this issue might have been caused by my previous modifications to part of the Docker interaction code. After switching to the latest version, the problem no longer occurs.

Originally, I used --max-iterations 4 only for a quick test. I was also wondering about the configuration parameters you used when evaluating SWT-Bench, such as max-iterations, max-attempts, max-retries, and related settings. Could you please share the exact configuration used in your experiments?

In addition, would it be possible for you to share the TRAJ files (instance_<INSTANCE_ID>.output.log) of OpenHands (with GPT-5 and GPT-5-mini) that were used for the SWT-Bench leaderboard?

By the way, I noticed that the commands in the README appear to be outdated.

python -m benchmarks.swtbench.run_infer \
    --llm-config llm_config.toml \
    --max-iterations 30 \
    --select instances.txt \
    --eval-output-dir ./evaluation_results \
    --max-attempts 3 \
    --critic finish_with_patch

Should it be (?):

python -m benchmarks.swtbench.run_infer \
    --max-iterations 30 \
    --select instances.txt \
    --output-dir ./evaluation_results \
    --max-attempts 3 \
    --critic finish_with_patch
    llm_config.toml