Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

swe scorer: unwrap grader's per-instance return shape

Browse Source 
get_eval_report returns {instance_id: {resolved: bool, ...}}, not a flat
dict. report.get('resolved') was always None -> every sample scored 0
despite the grader judging resolved=true (confirmed by direct grader
call on the real test output: astropy-13453 resolved=true).

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-18 16:14:00 +00:00
parent 7085624da0
commit 9889fb6c0e
1 changed files with 10 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
evalharness/eval/recipes/agent.py
View File

@ -323,8 +323,16 @@ def _official_swebench_eval(instance_meta: dict, patch: str, timeout: int = 1800
test_log_path=log_dir / 'test_output.txt', test_log_path=log_dir / 'test_output.txt',
include_tests_status=True, include_tests_status=True,
) )
return {'resolved': float(report.get('resolved', 0.0)), # get_eval_report returns {instance_id: {resolved: bool, ...}},
'report': {k: v for k, v in report.items() # NOT a flat dict -- report.get('resolved') was always None, so
# every instance scored 0 even when the grader said resolved=true
inner = report.get(test_spec.instance_id) or report
if isinstance(inner, dict):
resolved = float(inner.get('resolved', 0) or 0)
else:
resolved = 0.0
return {'resolved': resolved,
'report': {k: v for k, v in (inner if isinstance(inner, dict) else {}).items()
if isinstance(v, (int, float, str, bool))}} if isinstance(v, (int, float, str, bool))}}
finally: finally:
try: try: