evalstone/evalscope/docs/zh/benchmarks/medxpertqa.md

MedXpertQA

概述

MedXpertQA 是一个专家级医学多项选择基准测试，旨在评估高级医学知识与推理能力。该基准包含独立的纯文本（Text-only）和多模态（Multimodal, MM）两个赛道，题目源自具有挑战性的医学考试试题，并经由持证医师审核。

任务描述

• 任务类型：单答案医学多项选择题
• 输入：一道临床或生物医学问题及其选项，可选附带最多六张图像
• 输出：一个答案字母（Text 赛道为 A-J，MM 赛道为 A-E）
• 领域：涵盖17个医学专科和11个人体系统

主要特点

• 测试集包含4,450道题目：其中2,450道为Text题目（含十个选项），2,000道为MM题目（含五个选项）
• MM赛道包含放射影像、病理切片、光学图像、照片、示意图、图表、表格、文档及生命体征图像
• 所有题目均标注了医学任务类型、人体系统和问题类型；其中3,307道测试题侧重推理能力，1,143道侧重理解能力
• 题目经过难度筛选、选项增强、数据泄露缓解以及多轮专家评审

评估说明

• 主要指标：准确率（Accuracy），通过预测答案字母与标准答案的精确匹配计算
• 默认提示词采用 EvalScope 的零样本思维链（zero-shot chain-of-thought）模板，保留官方指定的逐步推理指令及严格的答案字母评分格式
• 应将 max_tokens 设置得足够高，以确保模型能完整输出所需的最终行 ANSWER: [LETTER]；否则，若推理过程被截断，解析器可能回退到提取最后一个有效的大写字母作为答案
• 结果分别报告 Text 和 MM 子集的表现，并通过样本加权聚合计算整体得分
• MM图像存储在 images.zip（约517 MB）中，直接从压缩包读取，无需额外解压副本
• 公开数据集共包含4,460条记录（含10个开发样例）；本集成仅评估其中4,450道预留的测试题
• 论文 | GitHub

属性

属性	值
基准测试名称	medxpertqa
数据集ID	evalscope/MedXpertQA
论文	Paper
标签	MCQ, Medical, MultiModal, Reasoning
指标	accuracy
默认示例数	0-shot
评估划分	test

数据统计

指标	值
总样本数	4,450
提示词长度（平均）	1135.22 字符
提示词长度（最小/最大）	346 / 4771 字符

各子集统计：

子集	样本数	提示词平均长度	提示词最小长度	提示词最大长度
Text	2,450	1337.92	435	4771
MM	2,000	886.91	346	2335

图像统计：

指标	值
图像总数	2,852
每样本图像数	最小: 1, 最大: 6, 平均: 1.43
分辨率范围	323x34 - 4248x2144
格式	jpeg, png

样例示例

子集: Text

{
  "input": [
    {
      "id": "3f1d2f2a",
      "content": "You are a helpful medical assistant."
    },
    {
      "id": "1a9f9143",
      "content": [
        {
          "text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D,E,F,G,H,I,J. Think step by step before answering.\n\nWhich pat ... [TRUNCATED 885 chars] ... ere posterior wear undergoing shoulder arthroplasty\nI) 58-year-old male with glenoid retroversion of 12-degrees undergoing shoulder arthroplasty\nJ) 55-year-old male with glenoid retroversion of 8-degrees undergoing total shoulder arthroplasty"
        }
      ]
    }
  ],
  "choices": [
    "70-year-old male with glenoid retroversion of 18-degrees undergoing shoulder arthroplasty",
    "70-year-old female with humeral anteversion of 13-degrees undergoing shoulder arthroplasty",
    "63-year-old female with glenoid retroversion of 22-degrees and mild posterior wear undergoing shoulder arthroplasty",
    "65-year-old female with glenoid retroversion of 25-degrees undergoing shoulder arthroplasty",
    "65-year-old female with a glenoid retroversion of 13-degrees undergoing shoulder arthroplasty",
    "68-year-old female with glenoid retroversion of 20-degrees undergoing reverse shoulder arthroplasty",
    "72-year-old male with glenoid retroversion of 15-degrees undergoing shoulder arthroplasty",
    "65-year-old female with glenoid retroversion of 30-degrees and severe posterior wear undergoing shoulder arthroplasty",
    "58-year-old male with glenoid retroversion of 12-degrees undergoing shoulder arthroplasty",
    "55-year-old male with glenoid retroversion of 8-degrees undergoing total shoulder arthroplasty"
  ],
  "target": "E",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "id": "Text-0",
    "medical_task": "Basic Science",
    "body_system": "Skeletal",
    "question_type": "Reasoning",
    "images": []
  }
}

注：部分内容为显示目的已截断。

提示模板

系统提示：

You are a helpful medical assistant.

提示模板：

Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.

{question}

{choices}

使用方法

使用 CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets medxpertqa \
    --limit 10  # 正式评估时请删除此行

使用 Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['medxpertqa'],
    dataset_args={
        'medxpertqa': {
            # subset_list: ['Text', 'MM']  # 可选，用于评估特定子集
        }
    },
    limit=10,  # 正式评估时请删除此行
)

run_task(task_cfg=task_cfg)