183 lines
5.7 KiB
Markdown
183 lines
5.7 KiB
Markdown
# CMMMU
|
||
|
||
|
||
## Overview
|
||
|
||
CMMU (Chinese Massive Multi-discipline Multimodal Understanding) includes manually collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines in Chinese. It is the Chinese counterpart to MMMU.
|
||
|
||
## Task Description
|
||
|
||
- **Task Type**: Chinese Multimodal Question Answering
|
||
- **Input**: Image(s) + question in Chinese with answer choices
|
||
- **Output**: Correct answer choice
|
||
- **Language**: Chinese
|
||
|
||
## Key Features
|
||
|
||
- 30 subjects across 6 core disciplines
|
||
- Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering
|
||
- 39 heterogeneous image types (charts, diagrams, maps, tables, etc.)
|
||
- College-level difficulty
|
||
- Multiple question types (multiple-choice, true/false, short answer)
|
||
|
||
## Evaluation Notes
|
||
|
||
- Default configuration uses **0-shot** evaluation
|
||
- Evaluates on validation split
|
||
- Simple accuracy metric
|
||
- Chinese language prompts used
|
||
|
||
|
||
## Properties
|
||
|
||
| Property | Value |
|
||
|----------|-------|
|
||
| **Benchmark Name** | `cmmmu` |
|
||
| **Dataset ID** | [lmms-lab/CMMMU](https://modelscope.cn/datasets/lmms-lab/CMMMU/summary) |
|
||
| **Paper** | N/A |
|
||
| **Tags** | `Chinese`, `Knowledge`, `MultiModal`, `QA` |
|
||
| **Metrics** | `acc` |
|
||
| **Default Shots** | 0-shot |
|
||
| **Evaluation Split** | `val` |
|
||
|
||
|
||
## Data Statistics
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Total Samples | 900 |
|
||
| Prompt Length (Mean) | 185.59 chars |
|
||
| Prompt Length (Min/Max) | 91 / 1045 chars |
|
||
|
||
**Per-Subset Statistics:**
|
||
|
||
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|
||
|--------|---------|-------------|------------|------------|
|
||
| `设计` | 18 | 216.39 | 136 | 347 |
|
||
| `音乐` | 21 | 149.57 | 110 | 185 |
|
||
| `艺术` | 16 | 166.44 | 143 | 240 |
|
||
| `艺术理论` | 33 | 171.21 | 118 | 268 |
|
||
| `经济` | 20 | 227.3 | 105 | 412 |
|
||
| `会计` | 39 | 197.05 | 103 | 528 |
|
||
| `金融` | 29 | 166 | 106 | 333 |
|
||
| `管理` | 22 | 236.73 | 135 | 423 |
|
||
| `营销` | 16 | 191.62 | 131 | 370 |
|
||
| `物理` | 35 | 266.11 | 146 | 497 |
|
||
| `地理` | 49 | 160.02 | 93 | 326 |
|
||
| `化学` | 42 | 187.4 | 103 | 322 |
|
||
| `生物` | 35 | 162.49 | 91 | 293 |
|
||
| `数学` | 43 | 188.23 | 101 | 370 |
|
||
| `临床医学` | 28 | 183.39 | 94 | 310 |
|
||
| `公共卫生` | 46 | 183.04 | 110 | 271 |
|
||
| `基础医学` | 32 | 146.56 | 92 | 219 |
|
||
| `诊断学与实验室医学` | 12 | 150.08 | 116 | 189 |
|
||
| `制药` | 35 | 181.77 | 97 | 413 |
|
||
| `历史` | 25 | 185.84 | 111 | 264 |
|
||
| `心理学` | 29 | 144.83 | 96 | 207 |
|
||
| `文献学` | 7 | 163.86 | 109 | 227 |
|
||
| `社会学` | 24 | 165 | 100 | 249 |
|
||
| `计算机科学` | 35 | 218.89 | 101 | 532 |
|
||
| `电子学` | 29 | 179.48 | 110 | 318 |
|
||
| `机械工程` | 40 | 196.1 | 98 | 817 |
|
||
| `能源和电力` | 32 | 186.44 | 100 | 330 |
|
||
| `材料` | 38 | 194.5 | 102 | 476 |
|
||
| `建筑学` | 49 | 175.39 | 97 | 415 |
|
||
| `农业` | 21 | 215.86 | 96 | 1045 |
|
||
|
||
**Image Statistics:**
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| Total Images | 1,023 |
|
||
| Images per Sample | min: 1, max: 5, mean: 1.14 |
|
||
| Resolution Range | 112x38 - 1500x3000 |
|
||
| Formats | jpeg, png |
|
||
|
||
|
||
## Sample Example
|
||
|
||
**Subset**: `设计`
|
||
|
||
```json
|
||
{
|
||
"input": [
|
||
{
|
||
"id": "47c0e169",
|
||
"content": [
|
||
{
|
||
"text": "请回答以下多项选择题，并选出正确选项。这些题目可能包括单选和多选题型。如果所提供的信息不足以确定一个明确的答案，那么请根据可用的数据和你的判断来选择最可能正确的选项。\n\n问题："
|
||
},
|
||
{
|
||
"image": "[BASE64_IMAGE: png, ~17.4KB]"
|
||
},
|
||
{
|
||
"text": "为一幅灰度图，要为它局部添加颜色以得到右图所示的效果，正确的操作步骤是（ ）。\n选项：\n(A) 先将色彩模式转为RGB，然后用工具箱中的 【画笔工具】上色\n(B) 先将色彩模式转为RGB，制作局部选区，然后打开【色相/饱和度】对话框，在其中点中【着色】项,调节色彩属性参数\n(C) 先将色彩模式转为RGB，制作局部选区，然后打开【可选颜色】对话框,调节参数\n(D) 打开【色相/饱和度】对话框，直接调节色彩属性参数\n\n正确答案：\n"
|
||
}
|
||
]
|
||
}
|
||
],
|
||
"target": "B",
|
||
"id": 0,
|
||
"group_id": 0,
|
||
"subset_key": "设计",
|
||
"metadata": {
|
||
"id": "1900",
|
||
"type": "选择",
|
||
"source_type": "website",
|
||
"analysis": null,
|
||
"distribution": "本科",
|
||
"difficulty_level": "easy",
|
||
"subcategory": "设计",
|
||
"category": "艺术与设计",
|
||
"subfield": "['图像编辑', '色彩调整']",
|
||
"img_type": "['屏幕截图']",
|
||
"answer": "B",
|
||
"option1": "先将色彩模式转为RGB，然后用工具箱中的 【画笔工具】上色",
|
||
"option2": "先将色彩模式转为RGB，制作局部选区，然后打开【色相/饱和度】对话框，在其中点中【着色】项,调节色彩属性参数",
|
||
"option3": "先将色彩模式转为RGB，制作局部选区，然后打开【可选颜色】对话框,调节参数",
|
||
"option4": "打开【色相/饱和度】对话框，直接调节色彩属性参数"
|
||
}
|
||
}
|
||
```
|
||
|
||
## Prompt Template
|
||
|
||
*No prompt template defined.*
|
||
|
||
## Usage
|
||
|
||
### Using CLI
|
||
|
||
```bash
|
||
evalscope eval \
|
||
--model YOUR_MODEL \
|
||
--api-url OPENAI_API_COMPAT_URL \
|
||
--api-key EMPTY_TOKEN \
|
||
--datasets cmmmu \
|
||
--limit 10 # Remove this line for formal evaluation
|
||
```
|
||
|
||
### Using Python
|
||
|
||
```python
|
||
from evalscope import run_task
|
||
from evalscope.config import TaskConfig
|
||
|
||
task_cfg = TaskConfig(
|
||
model='YOUR_MODEL',
|
||
api_url='OPENAI_API_COMPAT_URL',
|
||
api_key='EMPTY_TOKEN',
|
||
datasets=['cmmmu'],
|
||
dataset_args={
|
||
'cmmmu': {
|
||
# subset_list: ['设计', '音乐', '艺术'] # optional, evaluate specific subsets
|
||
}
|
||
},
|
||
limit=10, # Remove this line for formal evaluation
|
||
)
|
||
|
||
run_task(task_cfg=task_cfg)
|
||
```
|
||
|
||
|