Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

official swe scorer: PIP_INDEX_URL tuna in eval containers

Browse Source 
The official eval_script runs pip install -e '.[test]' -- same pypi.org
unreachability as agent containers would burn the 1800s timeout on
network retries. Same tuna mirror injected.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-18 10:45:51 +00:00
parent b3c837d3f3
commit 4618c990b1
1 changed files with 6 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
evalharness/eval/recipes/agent.py
View File

@ -269,7 +269,12 @@ def _official_swebench_eval(instance_meta: dict, patch: str, timeout: int = 1800
image_name = test_spec.instance_image_key
container = client.containers.run(
image_name, command='tail -f /dev/null', detach=True,
name=f'eh-swe-official-{test_spec.instance_id[:40]}')
name=f'eh-swe-official-{test_spec.instance_id[:40]}',
# the official eval_script runs `pip install -e .[test]`; pypi.org
# is unreachable from this host's containers -> same tuna mirror
# as the agent containers
environment={'PIP_INDEX_URL':
'https://pypi.tuna.tsinghua.edu.cn/simple'})
try:
patch_file = log_dir / 'patch.diff'
patch_file.write_text(patch or '')