Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

thinking.yaml: 64k budgets for the full-thinking ladder (es parity)

Browse Source 
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-17 02:10:50 +00:00
parent b1c88bef25
commit e9b79a2a41
1 changed files with 60 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

60
evalharness/config/thinking.yaml Normal file
View File

@ -0,0 +1,60 @@
default:
temperature: 0.0
top_p: 1.0
stream: true
max_tokens: 200336
aime24:
temperature: 1.0
repeats: 12
max_tokens: 200336
aime25:
temperature: 1.0
repeats: 12
max_tokens: 200336
aime26:
temperature: 1.0
repeats: 12
max_tokens: 200336
hmmt26:
temperature: 1.0
repeats: 12
max_tokens: 200336
imo_answerbench:
temperature: 1.0
gpqa_diamond:
temperature: 1.0
repeats: 3
max_tokens: 200336
mmlu:
max_tokens: 200336
mmlu_pro:
max_tokens: 200336
cmmlu:
max_tokens: 200336
arc:
max_tokens: 200336
hellaswag:
max_tokens: 200336
winogrande:
max_tokens: 200336
simple_qa:
max_tokens: 200336
trivia_qa:
max_tokens: 200336
humaneval:
temperature: 1.0
repeats: 3
live_code_bench:
temperature: 1.0
longbench_v2:
max_tokens: 200336
max_input_tokens: 128000
openai_mrcr:
max_tokens: 200336
max_input_tokens: 128000
bfcl_v3:
max_tokens: 4096
general_fc:
max_tokens: 4096
tau2_bench:
max_tokens: 16384