13274243a0
Keep K3 suite selection and report-schema scoring in bash, merge K3/vision dataset_args into dpv4 yamls, and pin EvalScope at 735d920ee911 with local patches. Co-authored-by: Cursor <cursoragent@cursor.com>
168 lines
6.2 KiB
Markdown
168 lines
6.2 KiB
Markdown
# MedXpertQA
|
||
|
||
|
||
## 概述
|
||
|
||
MedXpertQA 是一个专家级医学多项选择基准测试，旨在评估高级医学知识与推理能力。该基准包含独立的纯文本（Text-only）和多模态（Multimodal, MM）两个赛道，题目源自具有挑战性的医学考试试题，并经由持证医师审核。
|
||
|
||
## 任务描述
|
||
|
||
- **任务类型**：单答案医学多项选择题
|
||
- **输入**：一道临床或生物医学问题及其选项，可选附带最多六张图像
|
||
- **输出**：一个答案字母（Text 赛道为 A-J，MM 赛道为 A-E）
|
||
- **领域**：涵盖17个医学专科和11个人体系统
|
||
|
||
## 主要特点
|
||
|
||
- 测试集包含4,450道题目：其中2,450道为Text题目（含十个选项），2,000道为MM题目（含五个选项）
|
||
- MM赛道包含放射影像、病理切片、光学图像、照片、示意图、图表、表格、文档及生命体征图像
|
||
- 所有题目均标注了医学任务类型、人体系统和问题类型；其中3,307道测试题侧重推理能力，1,143道侧重理解能力
|
||
- 题目经过难度筛选、选项增强、数据泄露缓解以及多轮专家评审
|
||
|
||
## 评估说明
|
||
|
||
- 主要指标：**准确率（Accuracy）**，通过预测答案字母与标准答案的精确匹配计算
|
||
- 默认提示词采用 EvalScope 的零样本思维链（zero-shot chain-of-thought）模板，保留官方指定的逐步推理指令及严格的答案字母评分格式
|
||
- 应将 `max_tokens` 设置得足够高，以确保模型能完整输出所需的最终行 `ANSWER: [LETTER]`；否则，若推理过程被截断，解析器可能回退到提取最后一个有效的大写字母作为答案
|
||
- 结果分别报告 Text 和 MM 子集的表现，并通过样本加权聚合计算整体得分
|
||
- MM图像存储在 `images.zip`（约517 MB）中，直接从压缩包读取，无需额外解压副本
|
||
- 公开数据集共包含4,460条记录（含10个开发样例）；本集成仅评估其中4,450道预留的测试题
|
||
- [论文](https://arxiv.org/abs/2501.18362) | [GitHub](https://github.com/TsinghuaC3I/MedXpertQA)
|
||
|
||
|
||
## 属性
|
||
|
||
| 属性 | 值 |
|
||
|----------|-------|
|
||
| **基准测试名称** | `medxpertqa` |
|
||
| **数据集ID** | [evalscope/MedXpertQA](https://modelscope.cn/datasets/evalscope/MedXpertQA/summary) |
|
||
| **论文** | [Paper](https://arxiv.org/abs/2501.18362) |
|
||
| **标签** | `MCQ`, `Medical`, `MultiModal`, `Reasoning` |
|
||
| **指标** | `accuracy` |
|
||
| **默认示例数** | 0-shot |
|
||
| **评估划分** | `test` |
|
||
|
||
|
||
## 数据统计
|
||
|
||
| 指标 | 值 |
|
||
|--------|-------|
|
||
| 总样本数 | 4,450 |
|
||
| 提示词长度（平均） | 1135.22 字符 |
|
||
| 提示词长度（最小/最大） | 346 / 4771 字符 |
|
||
|
||
**各子集统计：**
|
||
|
||
| 子集 | 样本数 | 提示词平均长度 | 提示词最小长度 | 提示词最大长度 |
|
||
|--------|---------|-------------|------------|------------|
|
||
| `Text` | 2,450 | 1337.92 | 435 | 4771 |
|
||
| `MM` | 2,000 | 886.91 | 346 | 2335 |
|
||
|
||
**图像统计：**
|
||
|
||
| 指标 | 值 |
|
||
|--------|-------|
|
||
| 图像总数 | 2,852 |
|
||
| 每样本图像数 | 最小: 1, 最大: 6, 平均: 1.43 |
|
||
| 分辨率范围 | 323x34 - 4248x2144 |
|
||
| 格式 | jpeg, png |
|
||
|
||
|
||
## 样例示例
|
||
|
||
**子集**: `Text`
|
||
|
||
```json
|
||
{
|
||
"input": [
|
||
{
|
||
"id": "3f1d2f2a",
|
||
"content": "You are a helpful medical assistant."
|
||
},
|
||
{
|
||
"id": "1a9f9143",
|
||
"content": [
|
||
{
|
||
"text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D,E,F,G,H,I,J. Think step by step before answering.\n\nWhich pat ... [TRUNCATED 885 chars] ... ere posterior wear undergoing shoulder arthroplasty\nI) 58-year-old male with glenoid retroversion of 12-degrees undergoing shoulder arthroplasty\nJ) 55-year-old male with glenoid retroversion of 8-degrees undergoing total shoulder arthroplasty"
|
||
}
|
||
]
|
||
}
|
||
],
|
||
"choices": [
|
||
"70-year-old male with glenoid retroversion of 18-degrees undergoing shoulder arthroplasty",
|
||
"70-year-old female with humeral anteversion of 13-degrees undergoing shoulder arthroplasty",
|
||
"63-year-old female with glenoid retroversion of 22-degrees and mild posterior wear undergoing shoulder arthroplasty",
|
||
"65-year-old female with glenoid retroversion of 25-degrees undergoing shoulder arthroplasty",
|
||
"65-year-old female with a glenoid retroversion of 13-degrees undergoing shoulder arthroplasty",
|
||
"68-year-old female with glenoid retroversion of 20-degrees undergoing reverse shoulder arthroplasty",
|
||
"72-year-old male with glenoid retroversion of 15-degrees undergoing shoulder arthroplasty",
|
||
"65-year-old female with glenoid retroversion of 30-degrees and severe posterior wear undergoing shoulder arthroplasty",
|
||
"58-year-old male with glenoid retroversion of 12-degrees undergoing shoulder arthroplasty",
|
||
"55-year-old male with glenoid retroversion of 8-degrees undergoing total shoulder arthroplasty"
|
||
],
|
||
"target": "E",
|
||
"id": 0,
|
||
"group_id": 0,
|
||
"metadata": {
|
||
"id": "Text-0",
|
||
"medical_task": "Basic Science",
|
||
"body_system": "Skeletal",
|
||
"question_type": "Reasoning",
|
||
"images": []
|
||
}
|
||
}
|
||
```
|
||
|
||
*注：部分内容为显示目的已截断。*
|
||
|
||
## 提示模板
|
||
|
||
**系统提示：**
|
||
```text
|
||
You are a helpful medical assistant.
|
||
```
|
||
|
||
**提示模板：**
|
||
```text
|
||
Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.
|
||
|
||
{question}
|
||
|
||
{choices}
|
||
```
|
||
|
||
## 使用方法
|
||
|
||
### 使用 CLI
|
||
|
||
```bash
|
||
evalscope eval \
|
||
--model YOUR_MODEL \
|
||
--api-url OPENAI_API_COMPAT_URL \
|
||
--api-key EMPTY_TOKEN \
|
||
--datasets medxpertqa \
|
||
--limit 10 # 正式评估时请删除此行
|
||
```
|
||
|
||
### 使用 Python
|
||
|
||
```python
|
||
from evalscope import run_task
|
||
from evalscope.config import TaskConfig
|
||
|
||
task_cfg = TaskConfig(
|
||
model='YOUR_MODEL',
|
||
api_url='OPENAI_API_COMPAT_URL',
|
||
api_key='EMPTY_TOKEN',
|
||
datasets=['medxpertqa'],
|
||
dataset_args={
|
||
'medxpertqa': {
|
||
# subset_list: ['Text', 'MM'] # 可选，用于评估特定子集
|
||
}
|
||
},
|
||
limit=10, # 正式评估时请删除此行
|
||
)
|
||
|
||
run_task(task_cfg=task_cfg)
|
||
```
|