Add prebaked SWT-bench eval image support

[simonrosenberg]

Prebake SWT-bench eval env/base images and preload them in eval jobs.

• add swtbench image utilities to list required env/base tags and build/push prebaked eval env images
• workflow inputs to build prebaked eval env images and publish to GHCR
• optional micromamba patch flag so env hashes stay stable by default

Acceptance: swtbench run-eval with eval_limit=1 completed using prebaked env/base images from ghcr.io/openhands/swtbench-eval.

[neubig]

Do you have the instance-size one eval ready yet, or is this still wip?

[simonrosenberg]

Do you have the instance-size one eval ready yet, or is this still wip?

do you mean: using highcore nodes and parallelizing multiple evals on it? That's already merged
https://github.com/OpenHands/evaluation/pull/131/changes

[neubig]

Sorry for the vague message. The "acceptance criterion" for this PR was that you were able to run an eval with a single instance. I wanted to ask if you had successfully done that yet.

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • Build SWT-Bench Images

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #318 at branch `swtbench-prebaked-eval`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

[simonrosenberg]

Sorry for the vague message. The "acceptance criterion" for this PR was that you were able to run an eval with a single instance. I wanted to ask if you had successfully done that yet.

Ah OK!
Yes it works for one image. Also what's good is building the images for eval is one-shot since it doesn't depend on our agent version.
However, some images are extremely slow to build (on 40 images, a single image took +3hours while all others take less than 3 minutes) so it might take a while to build the 433 images. But the mamba fallback seems to work too honestly.

[simonrosenberg]

Sorry for the vague message. The "acceptance criterion" for this PR was that you were able to run an eval with a single instance. I wanted to ask if you had successfully done that yet.

testing now on 100 images