7921688149
- sample-counts manifest (config/sample_counts.yaml, harvested from real runs): uncached benches still show exact numbers in the plan instead of 'counts when datasets load' -- 'cache+est.' marks the mix - '--concurrency auto' is now an alias for --auto-concurrency - Concurrency row shows 'auto (start 8, gate decides)' when the gate drives, instead of a bare misleading 8 Also verified end-to-end: thinking-mode humaneval rep1/rep2 both pass 98.8%, matching the es reference runs (98.17/98.78/98.78) on the same model -- framework alignment holds on the thinking path too. Co-Authored-By: Claude <noreply@anthropic.com>
31 lines
605 B
YAML
31 lines
605 B
YAML
# Full-dataset sample counts per benchmark (fallback for the Run Plan
|
|
# when the dataset is not cached locally yet -- harvested from real runs;
|
|
# update when a benchmark's source changes).
|
|
humaneval: 164
|
|
aime24: 30
|
|
aime25: 30
|
|
aime26: 30
|
|
hmmt26: 30
|
|
gpqa_diamond: 198
|
|
mmlu: 285
|
|
mmlu_pro: 12032
|
|
cmmlu: 11528
|
|
gsm8k: 1319
|
|
arc: 2376
|
|
hellaswag: 10042
|
|
winogrande: 1267
|
|
drop: 9535
|
|
longbench_v2: 503
|
|
live_code_bench: 1055
|
|
bigcodebench: 1140
|
|
trivia_qa: 17944
|
|
simple_qa: 4326
|
|
hle: 2500
|
|
imo_answerbench: 89
|
|
openai_mrcr: 1300
|
|
bfcl_v3: 2000
|
|
general_fc: 400
|
|
tau2_bench: 184
|
|
competition_math: 5000
|
|
swe_bench_verified: 500
|