{
  "meta": {
    "pretty_name": "Video-MME-v2",
    "dataset_id": "MME-Benchmarks/Video-MME-v2",
    "paper_url": "https://arxiv.org/abs/2604.05015",
    "tags": [
      "MultiModal",
      "MCQ"
    ],
    "metrics": [
      "acc"
    ],
    "few_shot_num": 0,
    "eval_split": "test",
    "train_split": "",
    "subset_list": [
      "all"
    ],
    "description": "\n## Overview\n\nVideo-MME-v2 is a public comprehensive video understanding benchmark. It contains 800 videos,\n3,200 multiple-choice QA instances, and word-level subtitles with timestamps. The native adapter\nuses the shared `DatasetHub` abstraction for both annotation loading and optional media archive\ndownloads, so it exercises the same reusable video benchmark path as MVBench.\n\n## Task Description\n\n- **Task Type**: Video multiple-choice question answering\n- **Input**: Video URL or archived MP4 + question + answer choices\n- **Output**: Single correct answer letter\n- **Subsets**: `all`, `level_1`, `level_2`, `level_3`, `logic`, `relevance`\n\n## Evaluation Notes\n\n- Default configuration uses **0-shot** evaluation\n- Primary metric: **Accuracy**\n- The default video source is the public `url` field for lightweight smoke tests\n- Set `extra_params.video_source` to `archive` to download and use the official MP4 archives\n- Set `extra_params.use_subtitles` to `true` to include word-level subtitles in the prompt\n",
    "prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {
      "dataset_id": {
        "type": "str",
        "description": "Dataset repository ID or local dataset root for Video-MME-v2.",
        "value": "MME-Benchmarks/Video-MME-v2"
      },
      "dataset_hub": {
        "type": "str",
        "description": "Dataset hub used to load annotations, subtitles, and optional video archives.",
        "value": "modelscope",
        "choices": [
          "huggingface",
          "modelscope",
          "local"
        ]
      },
      "dataset_revision": {
        "type": "str",
        "description": "Optional dataset revision; leave empty to use the hub default.",
        "value": ""
      },
      "video_source": {
        "type": "str",
        "description": "Use public URL fields for lightweight tests or official archived MP4 files.",
        "value": "url",
        "choices": [
          "url",
          "archive"
        ]
      },
      "use_subtitles": {
        "type": "bool",
        "description": "Include Video-MME-v2 subtitle text in the prompt.",
        "value": false
      },
      "subtitle_word_limit": {
        "type": "int",
        "description": "Maximum number of subtitle words included per sample when subtitles are enabled.",
        "value": 512
      }
    },
    "sandbox_config": {},
    "category": "vlm"
  },
  "statistics": {
    "total_samples": 3200,
    "subset_stats": [
      {
        "name": "all",
        "sample_count": 3200
      },
      {
        "name": "level_1",
        "sample_count": 686
      },
      {
        "name": "level_2",
        "sample_count": 834
      },
      {
        "name": "level_3",
        "sample_count": 837
      },
      {
        "name": "logic",
        "sample_count": 1124
      },
      {
        "name": "relevance",
        "sample_count": 2076
      }
    ],
    "computed_at": "2026-05-12T11:49:48Z"
  },
  "sample_example": {
    "subset": "all",
    "record": {
      "video_id": "001",
      "url": "https://www.youtube.com/watch?v=AYSYelOQtQI",
      "group_type": "logic",
      "group_structure": "[1,2,3,4]",
      "question_id": "001-1",
      "question": "What is the ethnicity of the protagonist's mother?",
      "options": "A. Malaysian.\nB. British.\nC. Singaporean.\nD. German.\nE. Canadian.\nF. Chinese.\nG. American.\nH. Cannot be determined.",
      "answer": "F",
      "level": null,
      "second_head": null,
      "third_head": null
    }
  },
  "readme": {
    "en": "# Video-MME-v2\n\n\n## Overview\n\nVideo-MME-v2 is a public comprehensive video understanding benchmark. It contains 800 videos,\n3,200 multiple-choice QA instances, and word-level subtitles with timestamps. The native adapter\nuses the shared `DatasetHub` abstraction for both annotation loading and optional media archive\ndownloads, so it exercises the same reusable video benchmark path as MVBench.\n\n## Task Description\n\n- **Task Type**: Video multiple-choice question answering\n- **Input**: Video URL or archived MP4 + question + answer choices\n- **Output**: Single correct answer letter\n- **Subsets**: `all`, `level_1`, `level_2`, `level_3`, `logic`, `relevance`\n\n## Evaluation Notes\n\n- Default configuration uses **0-shot** evaluation\n- Primary metric: **Accuracy**\n- The default video source is the public `url` field for lightweight smoke tests\n- Set `extra_params.video_source` to `archive` to download and use the official MP4 archives\n- Set `extra_params.use_subtitles` to `true` to include word-level subtitles in the prompt\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `videomme_v2` |\n| **Dataset ID** | [MME-Benchmarks/Video-MME-v2](https://modelscope.cn/datasets/MME-Benchmarks/Video-MME-v2/summary) |\n| **Paper** | [Paper](https://arxiv.org/abs/2604.05015) |\n| **Tags** | `MCQ`, `MultiModal` |\n| **Metrics** | `acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 3,200 |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `all` | 3,200 | N/A | N/A | N/A |\n| `level_1` | 686 | N/A | N/A | N/A |\n| `level_2` | 834 | N/A | N/A | N/A |\n| `level_3` | 837 | N/A | N/A | N/A |\n| `logic` | 1,124 | N/A | N/A | N/A |\n| `relevance` | 2,076 | N/A | N/A | N/A |\n\n## Sample Example\n\n*Sample example not available.*\n\n## Prompt Template\n\n**Prompt Template:**\n```text\nAnswer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}\n```\n\n## Extra Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `dataset_id` | `str` | `MME-Benchmarks/Video-MME-v2` | Dataset repository ID or local dataset root for Video-MME-v2. |\n| `dataset_hub` | `str` | `modelscope` | Dataset hub used to load annotations, subtitles, and optional video archives. Choices: ['huggingface', 'modelscope', 'local'] |\n| `dataset_revision` | `str` | `` | Optional dataset revision; leave empty to use the hub default. |\n| `video_source` | `str` | `url` | Use public URL fields for lightweight tests or official archived MP4 files. Choices: ['url', 'archive'] |\n| `use_subtitles` | `bool` | `False` | Include Video-MME-v2 subtitle text in the prompt. |\n| `subtitle_word_limit` | `int` | `512` | Maximum number of subtitle words included per sample when subtitles are enabled. |\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets videomme_v2 \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['videomme_v2'],\n    dataset_args={\n        'videomme_v2': {\n            # subset_list: ['all', 'level_1', 'level_2']  # optional, evaluate specific subsets\n            # extra_params: {}  # uses default extra parameters\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# Video-MME-v2\n\n\n## 概述\n\nVideo-MME-v2 是一个公开的综合性视频理解基准测试。它包含 800 个视频、3,200 个多选问答样本，以及带有时间戳的词级字幕。该基准测试的原生适配器使用共享的 `DatasetHub` 抽象来加载标注数据并可选地下载媒体归档文件，因此其复用了与 MVBench 相同的可重用视频基准测试路径。\n\n## 任务描述\n\n- **任务类型**：视频多选问答（MCQ）\n- **输入**：视频 URL 或归档的 MP4 文件 + 问题 + 答案选项\n- **输出**：单个正确答案字母\n- **子集**：`all`、`level_1`、`level_2`、`level_3`、`logic`、`relevance`\n\n## 评估说明\n\n- 默认配置使用 **0-shot** 评估\n- 主要指标：**准确率（Accuracy）**\n- 默认视频源为公开的 `url` 字段，用于轻量级冒烟测试\n- 将 `extra_params.video_source` 设置为 `archive` 以下载并使用官方 MP4 归档文件\n- 将 `extra_params.use_subtitles` 设置为 `true` 以在提示中包含词级字幕\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `videomme_v2` |\n| **数据集ID** | [MME-Benchmarks/Video-MME-v2](https://modelscope.cn/datasets/MME-Benchmarks/Video-MME-v2/summary) |\n| **论文** | [Paper](https://arxiv.org/abs/2604.05015) |\n| **标签** | `MCQ`, `MultiModal` |\n| **指标** | `acc` |\n| **默认示例数** | 0-shot |\n| **评估划分** | `test` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 3,200 |\n\n**各子集统计信息：**\n\n| 子集 | 样本数 | 提示词平均长度 | 提示词最小长度 | 提示词最大长度 |\n|--------|---------|-------------|------------|------------|\n| `all` | 3,200 | N/A | N/A | N/A |\n| `level_1` | 686 | N/A | N/A | N/A |\n| `level_2` | 834 | N/A | N/A | N/A |\n| `level_3` | 837 | N/A | N/A | N/A |\n| `logic` | 1,124 | N/A | N/A | N/A |\n| `relevance` | 2,076 | N/A | N/A | N/A |\n\n## 样例示例\n\n*样例示例不可用。*\n\n## 提示模板\n\n**提示模板：**\n```text\nAnswer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}\n```\n\n## 额外参数\n\n| 参数 | 类型 | 默认值 | 描述 |\n|-----------|------|---------|-------------|\n| `dataset_id` | `str` | `MME-Benchmarks/Video-MME-v2` | Video-MME-v2 的数据集仓库 ID 或本地数据集根目录。 |\n| `dataset_hub` | `str` | `modelscope` | 用于加载标注、字幕和可选视频归档的 dataset hub。选项：['huggingface', 'modelscope', 'local'] |\n| `dataset_revision` | `str` | `` | 可选的数据集版本；留空则使用 hub 默认版本。 |\n| `video_source` | `str` | `url` | 使用公开 URL 字段进行轻量级测试，或使用官方归档的 MP4 文件。选项：['url', 'archive'] |\n| `use_subtitles` | `bool` | `False` | 在提示中包含 Video-MME-v2 的字幕文本。 |\n| `subtitle_word_limit` | `int` | `512` | 启用字幕时，每个样本最多包含的字幕词数。 |\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets videomme_v2 \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['videomme_v2'],\n    dataset_args={\n        'videomme_v2': {\n            # subset_list: ['all', 'level_1', 'level_2']  # 可选，用于评估特定子集\n            # extra_params: {}  # 使用默认额外参数\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "50b6825d91f0b1de50b73d90ba48a172",
    "needs_translation": false
  },
  "updated_at": "2026-05-18T10:23:17.437027",
  "translation_updated_at": "2026-05-18T10:23:22"
}