Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
210 Commits 1 Branch 0 Tags
Commit Graph

5 Commits

Author SHA1 Message Date
sora
e9e3230e46 swe_agentic: auto-release pulled images at process exit 
Per-instance images are 1-4GB each; a --limit 10 run pulls up to 10 of
them and previously stranded the whole footprint on Ctrl+C/exit. Every
image THIS process pulls is now registered and released by an atexit
hook (containers first, then rmi) -- images that already existed
locally are never touched. EVALHARNESS_KEEP_SWE_IMAGES=1 opts out for
prefetch-style runs that want to keep them.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 09:03:17 +00:00
sora
f09fe184fe swe_agentic: ensure per-instance image before docker run 
docker run's implicit pull prints nothing and the 120s start timeout
kills runs on slow mirrors; ensure_image brings the progress display,
mirror chain and watchdog to per-instance pulls (each sample = its own
image, 10 samples = 10 pulls).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 08:54:30 +00:00
sora
a93996094d swe_agentic: sentinel patch must end with a newline 
The captured diff was stripped, and git apply rejects diffs whose last
line lacks a trailing newline ('corrupt patch at line N') -- the first
agentic score was resolved=0 with patch_apply_failed despite a correct
fix. Verified: same payload + newline applies clean in the official
container.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 08:22:59 +00:00
sora
2a2335f842 swe_agentic: accept adapter kwarg in __init__ (get_env uniform signature) 
Without it get_env('swe_agentic') raised TypeError at env creation --
and the swallowed exception silently degraded the run to SINGLE-TURN
prose generation (421s of model monologue, no container, empty
trajectory). This was the agentic 'run' that looked like it worked.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 04:01:48 +00:00
sora
f662006517 swe_bench_verified_agentic: multi-turn SWE agent (mini-swe-agent protocol) 
Ports es's swe_bench_agentic_adapter into our plugin architecture:
- env swe_agentic: per-sample LONG-RUNNING container (official sweb
  image, /testbed, bash -lc like the testbed startup files expect),
  single bash tool via function calling, sentinel-submission protocol
  (COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT + patch), git-diff fallback;
  observations capped at 30k chars
- dataset swe_bench_verified_agentic: same princeton source/converter,
  separate bench name so both variants coexist
- recipe: recovered patch + OFFICIAL test_patch applied in-container,
  FAIL_TO_PASS + capped PASS_TO_PASS via conda testbed pytest, 1800s
- config: max_turns 250, env swe_agentic

Single-turn swe_bench_verified is untouched.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 02:29:35 +00:00