Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cmmlu few_shot_num 5->0 (es code default is 0-shot despite docstring claiming 5); 28-bench parity check all green

Browse Source
This commit is contained in:
sora 2026-08-27 06:43:09 +00:00
parent e11cae038b
commit 7ce04b8ac6
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
evalharness/data/datasets/cmmlu.py
View File

@ -20,7 +20,7 @@ from ..spec import DatasetSpec
subset='anatomy', # 67 subjects; override with --subset <subject> or 'all' subset='anatomy', # 67 subjects; override with --subset <subject> or 'all'
split='test', split='test',
few_shot_split='dev', few_shot_split='dev',
few_shot_num=5, few_shot_num=0, # es default is 0-shot (docstring says 5 but code says 0)
task_type='mcq', task_type='mcq',
tags=['zh', 'knowledge'], tags=['zh', 'knowledge'],
description='Chinese multiple-choice QA (official content, ModelScope mirror).', description='Chinese multiple-choice QA (official content, ModelScope mirror).',