370953729b
- perf_stats aggregator lives in eval/, not model/: the import failed silently and EVERY perf column was empty (not just ttft). Now warns on stderr instead of swallowing. - repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2 previously restored repeat 1's predictions and finished instantly with identical scores. rep1 keeps the legacy key (existing checkpoints still resume). - repeats summary: report the MEAN score and aggregate time/tokens over ALL runs (was: last run only). - README: six-benchmark command as the primary example. Co-Authored-By: Claude <noreply@anthropic.com>
76 lines
850 B
YAML
76 lines
850 B
YAML
default:
|
|
temperature: 0.0
|
|
top_p: 1.0
|
|
stream: true
|
|
max_tokens: 32768
|
|
|
|
aime24:
|
|
temperature: 1.0
|
|
repeats: 12
|
|
|
|
aime25:
|
|
temperature: 1.0
|
|
repeats: 12
|
|
|
|
aime26:
|
|
temperature: 1.0
|
|
repeats: 12
|
|
|
|
hmmt26:
|
|
temperature: 1.0
|
|
repeats: 12
|
|
|
|
imo_answerbench:
|
|
temperature: 1.0
|
|
|
|
gpqa_diamond:
|
|
temperature: 1.0
|
|
max_tokens: 8192
|
|
|
|
mmlu:
|
|
max_tokens: 8192
|
|
|
|
mmlu_pro:
|
|
max_tokens: 8192
|
|
|
|
cmmlu:
|
|
max_tokens: 8192
|
|
|
|
arc:
|
|
max_tokens: 8192
|
|
|
|
hellaswag:
|
|
max_tokens: 8192
|
|
|
|
winogrande:
|
|
max_tokens: 8192
|
|
|
|
simple_qa:
|
|
max_tokens: 8192
|
|
|
|
trivia_qa:
|
|
max_tokens: 8192
|
|
|
|
humaneval:
|
|
temperature: 1.0
|
|
|
|
live_code_bench:
|
|
temperature: 1.0
|
|
|
|
longbench_v2:
|
|
max_tokens: 8192
|
|
max_input_tokens: 128000
|
|
|
|
openai_mrcr:
|
|
max_tokens: 8192
|
|
max_input_tokens: 128000
|
|
|
|
bfcl_v3:
|
|
max_tokens: 4096
|
|
|
|
general_fc:
|
|
max_tokens: 4096
|
|
|
|
tau2_bench:
|
|
max_tokens: 16384
|