Single-image benches keep fail-fast; >8 distinct images (swe's 500
per-instance) now pull concurrently with resume state inside the run
itself -- exactly like the old auto-pull behavior, just resilient:
scoring proceeds with whatever images landed, missing ones score 0
and only a total wipeout fails the bench. Ctrl+C-safe (state file),
network recovery resumes automatically on the next run.
Co-Authored-By: Claude <noreply@anthropic.com>
- naming corrected to the Docker Hub truth es verified 500/500:
swebench/sweb.eval.x86_64.{instance_id.lower(), __->_1776_}:latest
(my repo-base rewrite was wrong; per-instance images ARE published)
- 'evalharness sandbox pull swe_bench_verified': concurrent pulls,
resume state in <cache-dir>/swe_pull_state.json, --dry-run N,
--retry-failed; self-contained (own dataset registry, no evalscope
import)
- save/load intentionally omitted per user call (pull-only for now)
Co-Authored-By: Claude <noreply@anthropic.com>
