Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

swe_bench_verified end-to-end verified (oracle patch -> official sweb image -> resolved); image naming mapped to local swebench/ set; ladder2 complete 28/28

Browse Source
This commit is contained in:
sora 2026-08-26 06:40:55 +00:00
parent 3d16e9647a
commit 0e1e31ba10
1 changed files with 8 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
evalharness/data/datasets/swe_bench_verified.py
View File

@ -19,10 +19,17 @@ from ..spec import DatasetSpec
def swe_bench_verified(): def swe_bench_verified():
def to_sample(record: dict) -> Sample: def to_sample(record: dict) -> Sample:
instance_id = record['instance_id'] instance_id = record['instance_id']
# image naming: local swebench/ set uses {repo}_1776_{repo}-{num}; the
# newer official layout is sweb.eval.x86_64.{repo}__{repo}-{num}
img = f'sweb.eval.x86_64.{instance_id}'
if '__' in instance_id:
repo, num = instance_id.rsplit('-', 1)
r = repo.split('__')[0]
img = f'swebench/sweb.eval.x86_64.{r}_1776_{r.split("__")[-1]}-{num}'
return Sample( return Sample(
input=record['problem_statement'], input=record['problem_statement'],
target=record['patch'], # gold patch (for oracle/oracle-check only) target=record['patch'], # gold patch (for oracle/oracle-check only)
sandbox=SandboxSpec(image=f'sweb.eval.x86_64.{instance_id}'), # official image naming sandbox=SandboxSpec(image=img),
metadata={ metadata={
'instance_id': instance_id, 'instance_id': instance_id,
'repo': record['repo'], 'repo': record['repo'],