Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
209 Commits 1 Branch 0 Tags
Commit Graph

4 Commits

Author SHA1 Message Date
sora
f09fe184fe swe_agentic: ensure per-instance image before docker run 
docker run's implicit pull prints nothing and the 120s start timeout
kills runs on slow mirrors; ensure_image brings the progress display,
mirror chain and watchdog to per-instance pulls (each sample = its own
image, 10 samples = 10 pulls).

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 08:54:30 +00:00
sora
a93996094d swe_agentic: sentinel patch must end with a newline 
The captured diff was stripped, and git apply rejects diffs whose last
line lacks a trailing newline ('corrupt patch at line N') -- the first
agentic score was resolved=0 with patch_apply_failed despite a correct
fix. Verified: same payload + newline applies clean in the official
container.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 08:22:59 +00:00
sora
2a2335f842 swe_agentic: accept adapter kwarg in __init__ (get_env uniform signature) 
Without it get_env('swe_agentic') raised TypeError at env creation --
and the swallowed exception silently degraded the run to SINGLE-TURN
prose generation (421s of model monologue, no container, empty
trajectory). This was the agentic 'run' that looked like it worked.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 04:01:48 +00:00
sora
f662006517 swe_bench_verified_agentic: multi-turn SWE agent (mini-swe-agent protocol) 
Ports es's swe_bench_agentic_adapter into our plugin architecture:
- env swe_agentic: per-sample LONG-RUNNING container (official sweb
  image, /testbed, bash -lc like the testbed startup files expect),
  single bash tool via function calling, sentinel-submission protocol
  (COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT + patch), git-diff fallback;
  observations capped at 30k chars
- dataset swe_bench_verified_agentic: same princeton source/converter,
  separate bench name so both variants coexist
- recipe: recovered patch + OFFICIAL test_patch applied in-container,
  FAIL_TO_PASS + capped PASS_TO_PASS via conda testbed pytest, 1800s
- config: max_turns 250, env swe_agentic

Single-turn swe_bench_verified is untouched.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-18 02:29:35 +00:00