7.0 KiB
7.0 KiB
MMLU-Redux
概述
MMLU-Redux 是 MMLU 基准测试的一个改进版本，修正了原始数据集中的错误答案。它通过修复不正确的标准答案标签、缺失的正确选项以及存在歧义的问题，解决了原始 MMLU 数据集中已知的错误。
任务描述
- 任务类型：多项选择知识评估
- 输入：包含四个选项的问题
- 输出：正确答案的字母（A/B/C/D）
- 领域：涵盖 STEM、人文学科、社会科学及其他领域的 57 个学科
主要特性
- 修正了原始 MMLU 基准测试中的错误
- 修复的错误类型包括：
no_correct_answer：缺少正确选项的问题wrong_groundtruth：标准答案错误的问题multiple_correct_answers：答案存在歧义的问题
- 与原始 MMLU 保持相同的 57 个学科覆盖范围
- 与 MMLU 评估框架兼容
评估说明
- 默认使用 test 数据划分进行评估
- 主要指标：准确率（Accuracy）（对多答案问题使用包含标志）
- 使用思维链（Chain-of-Thought, CoT）提示
- 仅支持零样本（zero-shot）评估（不支持少样本）
- 结果按学科和类别（STEM、人文学科、社会科学、其他）汇总
属性
| 属性 | 值 |
|---|---|
| 基准测试名称 | mmlu_redux |
| 数据集 ID | AI-ModelScope/mmlu-redux-2.0 |
| 论文 | N/A |
| 标签 | Knowledge, MCQ |
| 指标 | acc |
| 默认样本数 | 0-shot |
| 评估划分 | test |
数据统计
| 指标 | 值 |
|---|---|
| 总样本数 | 5,700 |
| 提示词长度（平均） | 600.81 字符 |
| 提示词长度（最小/最大） | 255 / 5082 字符 |
各子集统计数据：
| 子集 | 样本数 | 提示平均长度 | 提示最小长度 | 提示最大长度 |
|---|---|---|---|---|
abstract_algebra |
100 | 399.13 | 285 | 525 |
anatomy |
100 | 454.09 | 323 | 788 |
astronomy |
100 | 516.53 | 297 | 1087 |
business_ethics |
100 | 538.35 | 274 | 927 |
clinical_knowledge |
100 | 453.01 | 303 | 757 |
college_biology |
100 | 550.59 | 343 | 1081 |
college_chemistry |
100 | 451.35 | 292 | 890 |
college_computer_science |
100 | 631.74 | 336 | 1344 |
college_mathematics |
100 | 451.77 | 255 | 709 |
college_medicine |
100 | 669.42 | 308 | 5082 |
college_physics |
100 | 500.71 | 317 | 797 |
computer_security |
100 | 476.01 | 291 | 1115 |
conceptual_physics |
100 | 372.72 | 284 | 539 |
econometrics |
100 | 610.01 | 304 | 1036 |
electrical_engineering |
100 | 373.01 | 286 | 585 |
elementary_mathematics |
100 | 403.98 | 264 | 797 |
formal_logic |
100 | 598.04 | 318 | 1275 |
global_facts |
100 | 389.82 | 303 | 746 |
high_school_biology |
100 | 574.54 | 328 | 1078 |
high_school_chemistry |
100 | 496.56 | 271 | 1093 |
high_school_computer_science |
100 | 649.34 | 278 | 1786 |
high_school_european_history |
100 | 1840.88 | 855 | 3045 |
high_school_geography |
100 | 417.76 | 306 | 616 |
high_school_government_and_politics |
100 | 539.42 | 384 | 864 |
high_school_macroeconomics |
100 | 509.35 | 319 | 756 |
high_school_mathematics |
100 | 409.45 | 282 | 846 |
high_school_microeconomics |
100 | 508.99 | 325 | 1070 |
high_school_physics |
100 | 600.82 | 324 | 1414 |
high_school_psychology |
100 | 476.49 | 309 | 1055 |
high_school_statistics |
100 | 736.04 | 376 | 1772 |
high_school_us_history |
100 | 1643.04 | 782 | 2595 |
high_school_world_history |
100 | 1749.59 | 749 | 3834 |
human_aging |
100 | 416.87 | 323 | 689 |
human_sexuality |
100 | 457.5 | 307 | 1145 |
international_law |
100 | 636.61 | 332 | 986 |
jurisprudence |
100 | 518.24 | 307 | 1039 |
logical_fallacies |
100 | 516.62 | 333 | 902 |
machine_learning |
100 | 513.09 | 315 | 806 |
management |
100 | 399.38 | 294 | 645 |
marketing |
100 | 474.85 | 335 | 757 |
medical_genetics |
100 | 414.32 | 292 | 658 |
miscellaneous |
100 | 388.56 | 277 | 1268 |
moral_disputes |
100 | 534.28 | 325 | 872 |
moral_scenarios |
100 | 620.62 | 563 | 767 |
nutrition |
100 | 505.6 | 318 | 926 |
philosophy |
100 | 462.44 | 319 | 1155 |
prehistory |
100 | 468.62 | 320 | 943 |
professional_accounting |
100 | 650.46 | 341 | 1226 |
professional_law |
100 | 1370.21 | 359 | 2928 |
professional_medicine |
100 | 1003.93 | 610 | 1735 |
professional_psychology |
100 | 577.98 | 317 | 1502 |
public_relations |
100 | 472.39 | 300 | 1188 |
security_studies |
100 | 1029.35 | 317 | 2066 |
sociology |
100 | 530.23 | 335 | 834 |
us_foreign_policy |
100 | 490.28 | 305 | 754 |
virology |
100 | 447.81 | 302 | 1383 |
world_religions |
100 | 353.31 | 287 | 557 |
样例示例
子集: abstract_algebra
{
"input": [
{
"id": "f937b8b4",
"content": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\n\nStatement 1 | If T: V -> W is a linear transformation and dim(V ) < dim(W) < 1, then T must be injective. Statement 2 | Let dim(V) = n and suppose that T: V -> V is linear. If T is injective, then it is a bijection.\n\nA) True, True\nB) False, False\nC) True, False\nD) False, True"
}
],
"choices": [
"True, True",
"False, False",
"True, False",
"False, True"
],
"target": [
"A"
],
"id": 0,
"group_id": 0,
"metadata": {
"error_type": "bad_question_clarity",
"correct_answer": "0",
"potential_reason": "Statement 2 is true and well defined. \r\nHowever, statement 1 is not well defined: The dimension of a vector space is a nonnegative number, and since dim(V) < dim(W) < 1, this means dim(V) has to be negative. Taking this statement literally, the implication is vacuously true as the premise cannot be satisfed, but I doubt that was what the question is trying to test. "
}
}
提示模板
提示模板：
Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}
使用方法
使用 CLI
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets mmlu_redux \
--limit 10 # 正式评估时请删除此行
使用 Python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['mmlu_redux'],
dataset_args={
'mmlu_redux': {
# subset_list: ['abstract_algebra', 'anatomy', 'astronomy'] # 可选，用于评估特定子集
}
},
limit=10, # 正式评估时请删除此行
)
run_task(task_cfg=task_cfg)