env per-bench from YAML config; swe image naming to official repo-base
- config env entries: bfcl_v3->bfcl_mock, tau2_bench->tau2_official;
the eval loop previously POPPED the env key and discarded it -- now
it feeds args.env (CLI --env still works as default/fallback)
- swe_bench_verified images: per-instance names are never published;
the official docker.io layout is repo-level BASE images
(swebench/sweb.eval.x86_64.{repo}) with per-instance images built
on top -- naming corrected (note: the swebench/* namespace is
currently blocked on every reachable CN mirror all the same)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
b3c10d23f2
commit
5c14d11852
@ -788,8 +788,12 @@ def _cmd_eval_run(args) -> int:
|
|||||||
# YAML config: per-bench generation params, AUTO-LOADED
|
# YAML config: per-bench generation params, AUTO-LOADED
|
||||||
# (single .yaml in config/ = the default; --config overrides)
|
# (single .yaml in config/ = the default; --config overrides)
|
||||||
bench_cfg = _load_bench_cfg(args, name)
|
bench_cfg = _load_bench_cfg(args, name)
|
||||||
|
# env: per-bench from YAML, CLI --env as fallback/default
|
||||||
|
_env_cfg = bench_cfg.pop('env', '') or ''
|
||||||
|
if _env_cfg:
|
||||||
|
args.env = _env_cfg
|
||||||
# strip non-generation keys (they go to run_eval kwargs)
|
# strip non-generation keys (they go to run_eval kwargs)
|
||||||
for k in ('judge', 'judge_url', 'env', 'max_turns',
|
for k in ('judge', 'judge_url', 'max_turns',
|
||||||
'limit', 'limit_per_task', 'concurrency'):
|
'limit', 'limit_per_task', 'concurrency'):
|
||||||
bench_cfg.pop(k, None)
|
bench_cfg.pop(k, None)
|
||||||
|
|
||||||
|
|||||||
@ -54,7 +54,9 @@ openai_mrcr:
|
|||||||
max_input_tokens: 128000
|
max_input_tokens: 128000
|
||||||
bfcl_v3:
|
bfcl_v3:
|
||||||
max_tokens: 4096
|
max_tokens: 4096
|
||||||
|
env: bfcl_mock # agent 模式:工具调用轨迹 + 官方 AST 判分
|
||||||
general_fc:
|
general_fc:
|
||||||
max_tokens: 4096
|
max_tokens: 4096
|
||||||
tau2_bench:
|
tau2_bench:
|
||||||
max_tokens: 16384
|
max_tokens: 16384
|
||||||
|
env: tau2_official # agent 模式:官方引擎
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user