Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

swe scorer: return 'resolved' key (recipe metric name, not 'acc')

Browse Source 
The scorer returned {'acc': ...} while the recipe registers the metric
as 'resolved' -- the aggregator looked for 'resolved' in scores, found
nothing, and reported 0.0 despite per-sample resolved=1.0 in the
checkpoint. Key names now match the recipe registration.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-18 17:15:37 +00:00
parent 9889fb6c0e
commit 8c3d32bf2e
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
evalharness/eval/recipes/agent.py
View File

@ -375,8 +375,8 @@ def _swe_official_reward(pred, target, sample, ctx):
if have: if have:
try: try:
res = _official_swebench_eval(md, patch) res = _official_swebench_eval(md, patch)
return ({'acc': res.get('resolved', 0.0)}, return ({'resolved': res.get('resolved', 0.0)},
{'acc': {'mode': 'official_swebench', {'resolved': {'mode': 'official_swebench',
'resolved': res.get('resolved'), 'resolved': res.get('resolved'),
'error': res.get('error', ''), 'error': res.get('error', ''),
'turns_used': es.get('turns_used')}}) 'turns_used': es.get('turns_used')}})