sora 5c14d11852 env per-bench from YAML config; swe image naming to official repo-base ...

- config env entries: bfcl_v3->bfcl_mock, tau2_bench->tau2_official;
  the eval loop previously POPPED the env key and discarded it -- now
  it feeds args.env (CLI --env still works as default/fallback)
- swe_bench_verified images: per-instance names are never published;
  the official docker.io layout is repo-level BASE images
  (swebench/sweb.eval.x86_64.{repo}) with per-instance images built
  on top -- naming corrected (note: the swebench/* namespace is
  currently blocked on every reachable CN mirror all the same)

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-17 08:33:08 +00:00

..

agent

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

config

env per-bench from YAML config; swe image naming to official repo-base

2026-09-17 08:33:08 +00:00

data

aria2c multi-connection dataset downloads (CDN edge roulette fix)

2026-09-17 08:03:34 +00:00

eval

Strip <think> blocks from raw predictions before extraction

2026-09-17 07:41:33 +00:00

fingerprint

fingerprint: integrate fp_fusion model fingerprint benchmark

2026-09-11 03:52:21 +00:00

model

run_eval: construct the judge only when the recipe uses llm_judge

2026-09-17 08:07:49 +00:00

progress

Idle watchdog back to 300s default; begin_bench clears the stale bar

2026-09-17 06:08:33 +00:00

sandbox

LocalSandbox.exec: accept entrypoint kwarg (signature parity)

2026-09-17 07:24:48 +00:00

themes

Fixes: theme markup closing tags ([bold green]...[/bold green]); evaluate() runs off the event loop (sympy equivalence on hard math froze the progress clock for the whole bench)

2026-09-11 04:36:51 +00:00

third_party/bfcl

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

viz

Unified, nicer result tables + conda env setup in README

2026-09-10 06:30:22 +00:00

__init__.py

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

__main__.py

EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show

2026-08-24 03:35:11 +00:00

cli.py

env per-bench from YAML config; swe image naming to official repo-base

2026-09-17 08:33:08 +00:00

hooks.py

Pluginize the run shell: @register_progress (rich/plain), @register_theme, @register_hook (on_benchmark_failed/done), @register_prober; CLI gains --progress-plugin/--theme

2026-09-11 03:26:19 +00:00