f662006517
Ports es's swe_bench_agentic_adapter into our plugin architecture: - env swe_agentic: per-sample LONG-RUNNING container (official sweb image, /testbed, bash -lc like the testbed startup files expect), single bash tool via function calling, sentinel-submission protocol (COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT + patch), git-diff fallback; observations capped at 30k chars - dataset swe_bench_verified_agentic: same princeton source/converter, separate bench name so both variants coexist - recipe: recovered patch + OFFICIAL test_patch applied in-container, FAIL_TO_PASS + capped PASS_TO_PASS via conda testbed pytest, 1800s - config: max_turns 250, env swe_agentic Single-turn swe_bench_verified is untouched. Co-Authored-By: Claude <noreply@anthropic.com>
27 lines
874 B
Python
27 lines
874 B
Python
"""swe_bench_verified_agentic: same data as swe_bench_verified, agentic bench.
|
|
|
|
Separate bench name so both variants coexist (single-turn oracle vs
|
|
multi-turn agent); source/split/fields identical — the difference lives
|
|
in the recipe (env loop + official harness scoring) and config."""
|
|
|
|
from ..registry import register_dataset
|
|
from ..spec import DatasetSpec
|
|
|
|
|
|
@register_dataset(
|
|
DatasetSpec(
|
|
name='swe_bench_verified_agentic',
|
|
source='princeton-nlp/SWE-bench_Verified',
|
|
split='test',
|
|
task_type='agent',
|
|
tags=['code', 'agent', 'swe'],
|
|
requires=['docker'],
|
|
description='SWE-bench Verified (agentic): multi-turn bash agent in '
|
|
'the per-instance /testbed container.',
|
|
)
|
|
)
|
|
def swe_bench_verified_agentic():
|
|
from .swe_bench_verified import _record_to_sample
|
|
|
|
return _record_to_sample
|