Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

gen_profiles.yaml moved to evalharness/config/ (search path added); /data2/hf_home deleted (8.6G freed)

Browse Source 
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-11 07:48:41 +00:00
parent 471e0ca1a9
commit 77b67f8bbc
2 changed files with 32 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
evalharness/config/gen_profiles.yaml Normal file
View File

@ -0,0 +1,31 @@
# Generation-parameter profiles (YAML).
# Usage: evalharness eval run <bench> --model ... --profile <name>
# Resolution: DatasetSpec.gen_config < profile.default < profile.<bench> < gen_kwargs
dp4-nothink:
default: {temperature: 0.0, max_tokens: 32768, top_p: 1.0}
qwen3-es-parity:
default: {temperature: 0.0, max_tokens: 32768}
simple_qa: {max_tokens: 1024}
hle: {max_tokens: 8192}
gpqa_diamond: {temperature: 1.0, max_tokens: 8192}
aime24: {temperature: 1.0, max_tokens: 8192}
aime25: {temperature: 1.0, max_tokens: 8192}
aime26: {temperature: 1.0, max_tokens: 8192}
hmmt26: {temperature: 1.0, max_tokens: 8192}
imo_answerbench: {temperature: 1.0, max_tokens: 8192}
glm53-nothink:
default: {temperature: 0.0, max_tokens: 8192, top_p: 1.0}
aime24: {temperature: 1.0, max_tokens: 8192}
aime25: {temperature: 1.0, max_tokens: 8192}
aime26: {temperature: 1.0, max_tokens: 8192}
hmmt26: {temperature: 1.0, max_tokens: 8192}
imo_answerbench: {temperature: 1.0, max_tokens: 8192}
gpqa_diamond: {temperature: 1.0, max_tokens: 8192}
humaneval: {temperature: 1.0, max_tokens: 32768}
live_code_bench: {temperature: 1.0, max_tokens: 32768}
t1-short:
default: {temperature: 1.0, max_tokens: 8192, top_p: 1.0}

1
evalharness/model/gen_profiles.py
View File

@ -71,6 +71,7 @@ def _candidate_yaml_paths():
if env:
paths.append(Path(env))
paths.append(Path('gen_profiles.yaml'))
paths.append(Path(__file__).parent.parent / 'config' / 'gen_profiles.yaml')
paths.append(Path.home() / '.config' / 'evalharness' / 'gen_profiles.yaml')
return paths