sora e9d7e7f4eb README overhaul: features overview, config section, perf stats, FAQ ...

- accurate output structure (summary.xlsx/csv + per-bench xlsx; drop
  stale detail.md/summary.md mentions)
- dedicated YAML-config section documenting the keys that actually work
  (generation params + repeats + max_input_tokens) and precedence
- perf-stats table: what is always collected vs --perf streaming-only
- FAQ: endpoint probe, context-overflow shrink, thinking-mode notes
  (non-stream chat_template_kwargs vs cloud-API param), auto-stream
- config/README.md rewritten to match the flat-key reality (old file
  documented a directory scheme + judge/env keys that are not consumed)

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-14 02:30:38 +00:00

..

agent

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

config

README overhaul: features overview, config section, perf stats, FAQ

2026-09-14 02:30:38 +00:00

data

Fix NameError _fmt (module-level helper restored); aime/hmmt max_tokens 32768->8192 (non-stream path, gateway honors nothink)

2026-09-11 07:20:02 +00:00

eval

Drop summary.md from outputs; reports switch to report.jsonl (header line + one sample per line, round-trip verified)

2026-09-10 10:40:00 +00:00

fingerprint

fingerprint: integrate fp_fusion model fingerprint benchmark

2026-09-11 03:52:21 +00:00

model

Fix perf stats (wrong import path), per-repeat checkpoints, README

2026-09-11 13:38:04 +00:00

progress

Progress bar: colored fields (cur=yellow, elapsed=green, eta=cyan, retries=red, rate=dim)

2026-09-11 09:45:01 +00:00

sandbox

dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes

2026-09-08 05:55:04 +00:00

themes

Fixes: theme markup closing tags ([bold green]...[/bold green]); evaluate() runs off the event loop (sympy equivalence on hard math froze the progress clock for the whole bench)

2026-09-11 04:36:51 +00:00

third_party/bfcl

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

viz

Unified, nicer result tables + conda env setup in README

2026-09-10 06:30:22 +00:00

__init__.py

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

__main__.py

EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show

2026-08-24 03:35:11 +00:00

cli.py

Fix perf stats (wrong import path), per-repeat checkpoints, README

2026-09-11 13:38:04 +00:00

hooks.py

Pluginize the run shell: @register_progress (rich/plain), @register_theme, @register_hook (on_benchmark_failed/done), @register_prober; CLI gains --progress-plugin/--theme

2026-09-11 03:26:19 +00:00