EvalHarness

History

sora 587274d8b6 sandbox pull: SWE-bench per-instance pre-pull with resume (es port)

- naming corrected to the Docker Hub truth es verified 500/500:
  swebench/sweb.eval.x86_64.{instance_id.lower(), __->_1776_}:latest
  (my repo-base rewrite was wrong; per-instance images ARE published)
- 'evalharness sandbox pull swe_bench_verified': concurrent pulls,
  resume state in <cache-dir>/swe_pull_state.json, --dry-run N,
  --retry-failed; self-contained (own dataset registry, no evalscope
  import)
- save/load intentionally omitted per user call (pull-only for now)

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-17 09:36:20 +00:00

__init__.py

Fix images_for_samples export (swe prefetch import error)

2026-08-27 15:27:30 +00:00

base.py

LocalSandbox.exec: accept entrypoint kwarg (signature parity)

2026-09-17 07:24:48 +00:00

bg_prefetch.py

Comparison-driven fixes: MCQ choices in prompt + letter contract, Answer: suffix for QA, markdown answer cleaning, trivia_qa answer_phrase priority, DROP gold-as-alternatives (OR) official semantics, retry on 5xx, numpy/scipy compat

2026-08-24 15:48:42 +00:00

docker.py

Mirror chain trimmed to 6 mainstream sources; manual-load guidance