Adding config/sample_counts.yaml made TWO yamls in config/, and the
auto-load rule was 'exactly one yaml' -- so every bench silently lost
its repeats/temperature/max_tokens (humaneval ran once instead of 3).
Manifest renamed to .json, and the rule hardened: a lone yaml still
wins, otherwise default.yaml wins explicitly. Verified: aime 12x +
humaneval 3x repeats active again.
Co-Authored-By: Claude <noreply@anthropic.com>
- sample-counts manifest (config/sample_counts.yaml, harvested from
real runs): uncached benches still show exact numbers in the plan
instead of 'counts when datasets load' -- 'cache+est.' marks the mix
- '--concurrency auto' is now an alias for --auto-concurrency
- Concurrency row shows 'auto (start 8, gate decides)' when the gate
drives, instead of a bare misleading 8
Also verified end-to-end: thinking-mode humaneval rep1/rep2 both
pass 98.8%, matching the es reference runs (98.17/98.78/98.78) on the
same model -- framework alignment holds on the thinking path too.
Co-Authored-By: Claude <noreply@anthropic.com>
syy's es runs: humaneval x3, gpqa x2, aime25/26 x12 (already aligned),
mmlu_pro/longbench_v2 x1 (temp=0 deterministic -- repeats are noise,
and 3x 12k samples is pure cost). temp=1 benches get 3.
Co-Authored-By: Claude <noreply@anthropic.com>
- accurate output structure (summary.xlsx/csv + per-bench xlsx; drop
stale detail.md/summary.md mentions)
- dedicated YAML-config section documenting the keys that actually work
(generation params + repeats + max_input_tokens) and precedence
- perf-stats table: what is always collected vs --perf streaming-only
- FAQ: endpoint probe, context-overflow shrink, thinking-mode notes
(non-stream chat_template_kwargs vs cloud-API param), auto-stream
- config/README.md rewritten to match the flat-key reality (old file
documented a directory scheme + judge/env keys that are not consumed)
Co-Authored-By: Claude <noreply@anthropic.com>
