4f33521567
- Upgrade evalscope/evalscope from dev snapshot to upstream v1.9.1 - New benchmarks available: deep_swe, skillsbench, toolathlon, terminal_bench_v2_1, swe_bench_pro, browsecomp, gdpval, mcp_atlas, etc. - Reapply local patches: - api/model/generate_config.py: add max_completion_tokens - api/model/model.py: treat EMPTY api_key as unset - models/utils/openai.py: pass max_completion_tokens; handle choice.index=None - benchmarks/swe_bench/utils.py: guard None instance_id/client - api/evaluator/cache.py: remove model_name from cache/report paths
145 lines
17 KiB
JSON
145 lines
17 KiB
JSON
{
|
||
"meta": {
|
||
"pretty_name": "TVBench",
|
||
"dataset_id": "evalscope/TVBench",
|
||
"paper_url": null,
|
||
"tags": [
|
||
"MultiModal",
|
||
"Video",
|
||
"MCQ"
|
||
],
|
||
"metrics": [
|
||
"acc"
|
||
],
|
||
"few_shot_num": 0,
|
||
"eval_split": "train",
|
||
"train_split": "",
|
||
"subset_list": [
|
||
"action_count"
|
||
],
|
||
"description": "\n## Overview\n\nTVBench is a temporal video understanding benchmark for evaluating whether multimodal models can reason over dynamic visual events rather than isolated frames. It covers a broad set of video reasoning skills, including action recognition, action counting, temporal localization, action order understanding, egocentric action sequencing, object counting, object shuffling, moving direction recognition, scene transition reasoning, and unexpected action detection.\n\nThe EvalScope native adapter reads the official per-task JSON annotations from the dataset repository and resolves the corresponding video archives on demand. This keeps the default smoke-test path lightweight while still supporting the full benchmark through `subset_list`.\n\n## Task Description\n\n- **Task Type**: Video multiple-choice question answering (MCQ)\n- **Input**: A video clip or a time-bounded video segment, a natural-language question, and 2-4 answer candidates\n- **Output**: A single answer option letter selected from the provided candidates\n- **Default Subset**: `action_count`, selected because it is available as a standard MP4 archive in the public dataset repository\n- **Supported Subsets**: `action_antonym`, `action_count`, `action_localization`, `action_sequence`, `egocentric_sequence`, `moving_direction`, `object_count`, `object_shuffle`, `scene_transition`, and `unexpected_action`\n\n## Evaluation Notes\n\n- Default evaluation uses 0-shot Chain-of-Thought multiple-choice prompting via `MultipleChoiceTemplate.SINGLE_ANSWER_COT`.\n- Primary metric: Accuracy (`acc`). The dataset answer is stored as candidate text and is converted to the corresponding option letter before scoring.\n- Some subsets provide `start`/`end` fields. The adapter passes these values to `ContentVideo` and also adds a concise segment instruction to the prompt.\n- Video files are downloaded lazily from subset-specific archives. `egocentric_sequence` uses segmented archives under `video/egocentric_sequence/<prefix>.zip`.\n- The `action_antonym` annotations reference AVI files. If the repository media archive is unavailable, configure `extra_params.video_dir` to a local directory containing the AVI files.\n- The adapter supports local video layouts that are either flat (`video_dir/<video>`) or grouped by subset (`video_dir/<subset>/<video>`).\n",
|
||
"prompt_template": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}",
|
||
"system_prompt": "",
|
||
"few_shot_prompt_template": "",
|
||
"aggregation": "mean",
|
||
"extra_params": {
|
||
"video_dir": {
|
||
"type": "str",
|
||
"description": "Optional local directory containing TVBench video files. It may be organized flat or by subset.",
|
||
"value": ""
|
||
}
|
||
},
|
||
"sandbox_config": {},
|
||
"category": "vlm"
|
||
},
|
||
"statistics": {
|
||
"total_samples": 536,
|
||
"subset_stats": [
|
||
{
|
||
"name": "action_count",
|
||
"sample_count": 536,
|
||
"prompt_length_mean": 326.44,
|
||
"prompt_length_min": 290,
|
||
"prompt_length_max": 358,
|
||
"prompt_length_std": 24.64,
|
||
"target_length_mean": 1,
|
||
"multimodal": {
|
||
"has_images": false,
|
||
"has_audio": false,
|
||
"has_video": true,
|
||
"video": {
|
||
"count_total": 536,
|
||
"count_per_sample": {
|
||
"min": 1,
|
||
"max": 1,
|
||
"mean": 1
|
||
},
|
||
"duration": null,
|
||
"resolutions": [],
|
||
"formats": [
|
||
"mp4"
|
||
]
|
||
}
|
||
}
|
||
}
|
||
],
|
||
"prompt_length": {
|
||
"mean": 326.44,
|
||
"min": 290,
|
||
"max": 358,
|
||
"std": 24.64
|
||
},
|
||
"target_length_mean": 1,
|
||
"computed_at": "2026-07-08T17:05:25.499892",
|
||
"multimodal": {
|
||
"has_images": false,
|
||
"has_audio": false,
|
||
"has_video": true,
|
||
"video": {
|
||
"count_total": 536,
|
||
"count_per_sample": {
|
||
"min": 1,
|
||
"max": 1,
|
||
"mean": 1
|
||
},
|
||
"duration": null,
|
||
"resolutions": [],
|
||
"formats": [
|
||
"mp4"
|
||
]
|
||
}
|
||
}
|
||
},
|
||
"sample_example": {
|
||
"data": {
|
||
"input": [
|
||
{
|
||
"id": "4b521c81",
|
||
"content": [
|
||
{
|
||
"text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\n\nThe person makes sets of repeated actions. How many times did the person repeat the action in the last set?\n\nA) 2\nB) 3\nC) 5\nD) 4"
|
||
},
|
||
{
|
||
"video": "~/.cache/modelscope/hub/datasets/tvbench/videos/action_count/video_8686.mp4",
|
||
"format": "mp4"
|
||
}
|
||
]
|
||
}
|
||
],
|
||
"choices": [
|
||
"2",
|
||
"3",
|
||
"5",
|
||
"4"
|
||
],
|
||
"target": "D",
|
||
"id": 0,
|
||
"group_id": 0,
|
||
"metadata": {
|
||
"video": "video_8686.mp4",
|
||
"answer": "4",
|
||
"subset": "action_count",
|
||
"start": null,
|
||
"end": null,
|
||
"fps": null,
|
||
"accurate_start": null,
|
||
"accurate_end": null,
|
||
"video_length": null,
|
||
"question_id": null,
|
||
"dataset_id": "evalscope/TVBench",
|
||
"dataset_hub": "modelscope"
|
||
}
|
||
},
|
||
"subset": "action_count",
|
||
"truncated": false
|
||
},
|
||
"readme": {
|
||
"en": "# TVBench\n\n\n## Overview\n\nTVBench is a temporal video understanding benchmark for evaluating whether multimodal models can reason over dynamic visual events rather than isolated frames. It covers a broad set of video reasoning skills, including action recognition, action counting, temporal localization, action order understanding, egocentric action sequencing, object counting, object shuffling, moving direction recognition, scene transition reasoning, and unexpected action detection.\n\nThe EvalScope native adapter reads the official per-task JSON annotations from the dataset repository and resolves the corresponding video archives on demand. This keeps the default smoke-test path lightweight while still supporting the full benchmark through `subset_list`.\n\n## Task Description\n\n- **Task Type**: Video multiple-choice question answering (MCQ)\n- **Input**: A video clip or a time-bounded video segment, a natural-language question, and 2-4 answer candidates\n- **Output**: A single answer option letter selected from the provided candidates\n- **Default Subset**: `action_count`, selected because it is available as a standard MP4 archive in the public dataset repository\n- **Supported Subsets**: `action_antonym`, `action_count`, `action_localization`, `action_sequence`, `egocentric_sequence`, `moving_direction`, `object_count`, `object_shuffle`, `scene_transition`, and `unexpected_action`\n\n## Evaluation Notes\n\n- Default evaluation uses 0-shot Chain-of-Thought multiple-choice prompting via `MultipleChoiceTemplate.SINGLE_ANSWER_COT`.\n- Primary metric: Accuracy (`acc`). The dataset answer is stored as candidate text and is converted to the corresponding option letter before scoring.\n- Some subsets provide `start`/`end` fields. The adapter passes these values to `ContentVideo` and also adds a concise segment instruction to the prompt.\n- Video files are downloaded lazily from subset-specific archives. `egocentric_sequence` uses segmented archives under `video/egocentric_sequence/<prefix>.zip`.\n- The `action_antonym` annotations reference AVI files. If the repository media archive is unavailable, configure `extra_params.video_dir` to a local directory containing the AVI files.\n- The adapter supports local video layouts that are either flat (`video_dir/<video>`) or grouped by subset (`video_dir/<subset>/<video>`).\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `tvbench` |\n| **Dataset ID** | [evalscope/TVBench](https://modelscope.cn/datasets/evalscope/TVBench/summary) |\n| **Paper** | N/A |\n| **Tags** | `MCQ`, `MultiModal`, `Video` |\n| **Metrics** | `acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `train` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 536 |\n| Prompt Length (Mean) | 326.44 chars |\n| Prompt Length (Min/Max) | 290 / 358 chars |\n\n**Video Statistics:**\n\n| Metric | Value |\n|--------|-------|\n| Total Videos | 536 |\n| Videos per Sample | min: 1, max: 1, mean: 1 |\n| Formats | mp4 |\n\n\n## Sample Example\n\n**Subset**: `action_count`\n\n```json\n{\n \"input\": [\n {\n \"id\": \"4b521c81\",\n \"content\": [\n {\n \"text\": \"Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\\n\\nThe person makes sets of repeated actions. How many times did the person repeat the action in the last set?\\n\\nA) 2\\nB) 3\\nC) 5\\nD) 4\"\n },\n {\n \"video\": \"~/.cache/modelscope/hub/datasets/tvbench/videos/action_count/video_8686.mp4\",\n \"format\": \"mp4\"\n }\n ]\n }\n ],\n \"choices\": [\n \"2\",\n \"3\",\n \"5\",\n \"4\"\n ],\n \"target\": \"D\",\n \"id\": 0,\n \"group_id\": 0,\n \"metadata\": {\n \"video\": \"video_8686.mp4\",\n \"answer\": \"4\",\n \"subset\": \"action_count\",\n \"start\": null,\n \"end\": null,\n \"fps\": null,\n \"accurate_start\": null,\n \"accurate_end\": null,\n \"video_length\": null,\n \"question_id\": null,\n \"dataset_id\": \"evalscope/TVBench\",\n \"dataset_hub\": \"modelscope\"\n }\n}\n```\n\n## Prompt Template\n\n**Prompt Template:**\n```text\nAnswer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}\n```\n\n## Extra Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `video_dir` | `str` | `` | Optional local directory containing TVBench video files. It may be organized flat or by subset. |\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n --model YOUR_MODEL \\\n --api-url OPENAI_API_COMPAT_URL \\\n --api-key EMPTY_TOKEN \\\n --datasets tvbench \\\n --limit 10 # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n model='YOUR_MODEL',\n api_url='OPENAI_API_COMPAT_URL',\n api_key='EMPTY_TOKEN',\n datasets=['tvbench'],\n dataset_args={\n 'tvbench': {\n # extra_params: {} # uses default extra parameters\n }\n },\n limit=10, # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
|
||
"zh": "# TVBench\n\n\n## 概述\n\nTVBench 是一个用于评估多模态模型是否能够对动态视觉事件（而非孤立帧）进行推理的时序视频理解基准测试。它涵盖了广泛的视频推理能力，包括动作识别、动作计数、时序定位、动作顺序理解、第一人称视角动作排序、物体计数、物体重排、运动方向识别、场景转换推理以及异常动作检测。\n\nEvalScope 原生适配器从数据集仓库中读取官方提供的按任务划分的 JSON 注释，并按需解析对应的视频压缩包。这种方式使得默认的冒烟测试路径保持轻量，同时仍可通过 `subset_list` 支持完整的基准测试。\n\n## 任务描述\n\n- **任务类型**：视频多项选择题问答（MCQ）\n- **输入**：一段视频片段或带时间范围的视频片段、一个自然语言问题以及 2-4 个候选答案\n- **输出**：从提供的候选答案中选择一个选项字母\n- **默认子集**：`action_count`，因其在公开数据集仓库中以标准 MP4 压缩包形式提供\n- **支持的子集**：`action_antonym`、`action_count`、`action_localization`、`action_sequence`、`egocentric_sequence`、`moving_direction`、`object_count`、`object_shuffle`、`scene_transition` 和 `unexpected_action`\n\n## 评估说明\n\n- 默认评估使用 0-shot Chain-of-Thought 多项选择提示模板 `MultipleChoiceTemplate.SINGLE_ANSWER_COT`。\n- 主要指标：准确率（`acc`）。数据集中的答案以候选文本形式存储，在评分前会转换为对应的选项字母。\n- 部分子集提供 `start`/`end` 字段。适配器会将这些值传递给 `ContentVideo`，并在提示中添加简明的片段说明。\n- 视频文件按需从子集特定的压缩包中懒加载下载。`egocentric_sequence` 使用位于 `video/egocentric_sequence/<prefix>.zip` 下的分段压缩包。\n- `action_antonym` 的注释引用 AVI 文件。如果仓库媒体压缩包不可用，请通过配置 `extra_params.video_dir` 指向包含 AVI 文件的本地目录。\n- 适配器支持两种本地视频布局：扁平结构（`video_dir/<video>`）或按子集分组结构（`video_dir/<subset>/<video>`）。\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `tvbench` |\n| **数据集ID** | [evalscope/TVBench](https://modelscope.cn/datasets/evalscope/TVBench/summary) |\n| **论文** | N/A |\n| **标签** | `MCQ`, `MultiModal`, `Video` |\n| **指标** | `acc` |\n| **默认示例数** | 0-shot |\n| **评估分割** | `train` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 536 |\n| 提示词长度（平均） | 326.44 字符 |\n| 提示词长度（最小/最大） | 290 / 358 字符 |\n\n**视频统计信息：**\n\n| 指标 | 值 |\n|--------|-------|\n| 总视频数 | 536 |\n| 每样本视频数 | 最小: 1, 最大: 1, 平均: 1 |\n| 格式 | mp4 |\n\n\n## 样例示例\n\n**子集**: `action_count`\n\n```json\n{\n \"input\": [\n {\n \"id\": \"4b521c81\",\n \"content\": [\n {\n \"text\": \"Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\\n\\nThe person makes sets of repeated actions. How many times did the person repeat the action in the last set?\\n\\nA) 2\\nB) 3\\nC) 5\\nD) 4\"\n },\n {\n \"video\": \"~/.cache/modelscope/hub/datasets/tvbench/videos/action_count/video_8686.mp4\",\n \"format\": \"mp4\"\n }\n ]\n }\n ],\n \"choices\": [\n \"2\",\n \"3\",\n \"5\",\n \"4\"\n ],\n \"target\": \"D\",\n \"id\": 0,\n \"group_id\": 0,\n \"metadata\": {\n \"video\": \"video_8686.mp4\",\n \"answer\": \"4\",\n \"subset\": \"action_count\",\n \"start\": null,\n \"end\": null,\n \"fps\": null,\n \"accurate_start\": null,\n \"accurate_end\": null,\n \"video_length\": null,\n \"question_id\": null,\n \"dataset_id\": \"evalscope/TVBench\",\n \"dataset_hub\": \"modelscope\"\n }\n}\n```\n\n## 提示模板\n\n**提示模板：**\n```text\nAnswer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.\n\n{question}\n\n{choices}\n```\n\n## 额外参数\n\n| 参数 | 类型 | 默认值 | 描述 |\n|-----------|------|---------|-------------|\n| `video_dir` | `str` | `` | 可选的本地目录，包含 TVBench 视频文件。目录结构可以是扁平的，也可以按子集组织。 |\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n --model YOUR_MODEL \\\n --api-url OPENAI_API_COMPAT_URL \\\n --api-key EMPTY_TOKEN \\\n --datasets tvbench \\\n --limit 10 # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n model='YOUR_MODEL',\n api_url='OPENAI_API_COMPAT_URL',\n api_key='EMPTY_TOKEN',\n datasets=['tvbench'],\n dataset_args={\n 'tvbench': {\n # extra_params: {} # 使用默认额外参数\n }\n },\n limit=10, # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
|
||
"content_hash": "e8674a114f8b1d91a11b6e90859a96fc",
|
||
"needs_translation": false
|
||
},
|
||
"updated_at": "2026-07-13T17:06:08.945393",
|
||
"translation_updated_at": "2026-07-13T17:06:26"
|
||
}
|