env per-bench from YAML config; swe image naming to official repo-base

- config env entries: bfcl_v3->bfcl_mock, tau2_bench->tau2_official;
  the eval loop previously POPPED the env key and discarded it -- now
  it feeds args.env (CLI --env still works as default/fallback)
- swe_bench_verified images: per-instance names are never published;
  the official docker.io layout is repo-level BASE images
  (swebench/sweb.eval.x86_64.{repo}) with per-instance images built
  on top -- naming corrected (note: the swebench/* namespace is
  currently blocked on every reachable CN mirror all the same)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-17 08:33:08 +00:00
parent b3c10d23f2
commit 5c14d11852
2 changed files with 7 additions and 1 deletions

View File

@ -788,8 +788,12 @@ def _cmd_eval_run(args) -> int:
# YAML config: per-bench generation params, AUTO-LOADED # YAML config: per-bench generation params, AUTO-LOADED
# (single .yaml in config/ = the default; --config overrides) # (single .yaml in config/ = the default; --config overrides)
bench_cfg = _load_bench_cfg(args, name) bench_cfg = _load_bench_cfg(args, name)
# env: per-bench from YAML, CLI --env as fallback/default
_env_cfg = bench_cfg.pop('env', '') or ''
if _env_cfg:
args.env = _env_cfg
# strip non-generation keys (they go to run_eval kwargs) # strip non-generation keys (they go to run_eval kwargs)
for k in ('judge', 'judge_url', 'env', 'max_turns', for k in ('judge', 'judge_url', 'max_turns',
'limit', 'limit_per_task', 'concurrency'): 'limit', 'limit_per_task', 'concurrency'):
bench_cfg.pop(k, None) bench_cfg.pop(k, None)

View File

@ -54,7 +54,9 @@ openai_mrcr:
max_input_tokens: 128000 max_input_tokens: 128000
bfcl_v3: bfcl_v3:
max_tokens: 4096 max_tokens: 4096
env: bfcl_mock # agent 模式:工具调用轨迹 + 官方 AST 判分
general_fc: general_fc:
max_tokens: 4096 max_tokens: 4096
tau2_bench: tau2_bench:
max_tokens: 16384 max_tokens: 16384
env: tau2_official # agent 模式:官方引擎