Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

swe scorer: metadata from env_state (positional pair misalign fix)

Browse Source 
sample.metadata vs env_state mismatch confirmed: sample[0] said
astropy-12907 while the paired prediction held 13453's test_patch --
make_test_spec built the eval_script for the WRONG bug, scoring real
patches against wrong tests (0/10 despite manual verification that
the patches are correct). The env_state IS the actual instance the
agent ran; it takes priority, sample.metadata fills gaps.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-18 15:28:05 +00:00
parent 4618c990b1
commit 014e8bcd9b
1 changed files with 8 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
evalharness/eval/recipes/agent.py
View File

@ -340,7 +340,14 @@ def _swe_official_reward(pred, target, sample, ctx):
patch = es.get('patch') or pred or ''
if not patch or '(no patch produced)' in patch:
return ({'acc': 0.0}, {'acc': {'error': 'no patch'}})
md = dict(sample.metadata or {})
# metadata from ENV_STATE, not sample.metadata: the runner shuffles
# samples and restores predictions by checkpoint key, so positional
# pairing can misalign -- env_state carries the ACTUAL instance the
# agent ran (verified: sample.metadata said 12907 while env_state
# held 13453's test_patch, building an eval_script for the wrong bug)
md = dict(es)
md.update({k: v for k, v in (sample.metadata or {}).items()
if k not in md or not md.get(k)})
# official make_test_spec needs the raw instance fields
for k in ('FAIL_TO_PASS', 'PASS_TO_PASS'):
v = md.get(k)