{
  "meta": {
    "pretty_name": "MSVD",
    "dataset_id": "evalscope/MSVD",
    "paper_url": "https://aclanthology.org/P11-1020/",
    "tags": [
      "MultiModal",
      "ImageCaptioning"
    ],
    "metrics": [
      "Bleu_1",
      "Bleu_2",
      "Bleu_3",
      "Bleu_4",
      "METEOR",
      "ROUGE_L",
      "CIDEr"
    ],
    "few_shot_num": 0,
    "eval_split": "test",
    "train_split": "",
    "subset_list": [
      "default"
    ],
    "description": "\n## Overview\n\nMSVD is a classic video captioning benchmark with short web videos annotated by many human captions.\nThe native adapter treats each video as one evaluation sample and uses all available captions as references.\n\n## Task Description\n\n- **Task Type**: Video captioning\n- **Input**: Video clip\n- **Output**: One concise natural-language caption\n- **Domains**: Open-domain video understanding and description\n\n## Evaluation Notes\n\n- Default data source: `evalscope/MSVD` on ModelScope, `test` split\n- Hugging Face `VLM2Vec/MSVD` remains available by setting `extra_params.dataset_hub=\"huggingface\"`\n- Primary metric: **CIDEr**\n- Additional metrics: BLEU-1/2/3/4, METEOR, ROUGE-L\n- Set `extra_params.video_dir` when the dataset only provides video file names and local media files are required\n",
    "prompt_template": "Describe the video in one concise sentence.",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {
      "dataset_hub": {
        "type": "str",
        "description": "Dataset hub used to load MSVD annotations.",
        "value": "modelscope",
        "choices": [
          "huggingface",
          "modelscope",
          "local"
        ]
      },
      "eval_split": {
        "type": "str",
        "description": "Source split to load; defaults to test.",
        "value": ""
      },
      "dataset_revision": {
        "type": "str",
        "description": "Optional dataset revision; leave empty to use the hub default.",
        "value": ""
      },
      "video_dir": {
        "type": "str",
        "description": "Optional local directory containing MSVD video files.",
        "value": ""
      },
      "video_extension": {
        "type": "str",
        "description": "Optional extension override for local videos, for example \"mp4\".",
        "value": ""
      }
    },
    "sandbox_config": {},
    "category": "vlm"
  },
  "statistics": {
    "total_samples": 670,
    "subset_stats": [
      {
        "name": "default",
        "sample_count": 670,
        "prompt_length_mean": 43,
        "prompt_length_min": 43,
        "prompt_length_max": 43,
        "prompt_length_std": null,
        "target_length_mean": 1399.59,
        "multimodal": {
          "has_images": false,
          "has_audio": false,
          "has_video": true,
          "video": {
            "count_total": 670,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "duration": null,
            "resolutions": [],
            "formats": [
              "mp4"
            ]
          }
        }
      }
    ],
    "prompt_length": {
      "mean": 43,
      "min": 43,
      "max": 43,
      "std": null
    },
    "target_length_mean": 1399.59,
    "computed_at": "2026-06-09T15:45:49.069606",
    "multimodal": {
      "has_images": false,
      "has_audio": false,
      "has_video": true,
      "video": {
        "count_total": 670,
        "count_per_sample": {
          "min": 1,
          "max": 1,
          "mean": 1
        },
        "duration": null,
        "resolutions": [],
        "formats": [
          "mp4"
        ]
      }
    }
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "a4b83275",
          "content": [
            {
              "text": "Describe the video in one concise sentence."
            },
            {
              "video": "fr9H1WLcF1A_256_261.avi",
              "format": "mp4"
            }
          ]
        }
      ],
      "target": "[\"two young men are playing table tennis\", \"men are playing table tennis\", \"two men are playing table tennis\", \"two men are playing a tabletennis\", \"a couple of people are playing a game of ping pong\", \"peoples are playing table tennis\", \"two ... [TRUNCATED 306 chars] ... e boys are playing\", \"people are playing ping pong\", \"18 kids and counting show the newest baby josie\", \"there is some kids and playing to each other\", \"the men played pingpong together\", \"the boys are playing ping pong\", \"2 boys is playing\"]",
      "id": 0,
      "group_id": 0,
      "metadata": {
        "references": [
          "two young men are playing table tennis",
          "men are playing table tennis",
          "two men are playing table tennis",
          "two men are playing a tabletennis",
          "a couple of people are playing a game of ping pong",
          "peoples are playing table tennis",
          "two men are playing pingpong",
          "two guys play table tennis",
          "two people are playing ping pong",
          "two boys are playing ping pong",
          "... [TRUNCATED 12 more items] ..."
        ],
        "subset": "default",
        "dataset_id": "evalscope/MSVD",
        "dataset_hub": "modelscope",
        "video": "fr9H1WLcF1A_256_261.avi",
        "video_id": "fr9H1WLcF1A_256_261",
        "source": "MSVD"
      }
    },
    "subset": "default",
    "truncated": false
  },
  "readme": {
    "en": "# MSVD\n\n\n## Overview\n\nMSVD is a classic video captioning benchmark with short web videos annotated by many human captions.\nThe native adapter treats each video as one evaluation sample and uses all available captions as references.\n\n## Task Description\n\n- **Task Type**: Video captioning\n- **Input**: Video clip\n- **Output**: One concise natural-language caption\n- **Domains**: Open-domain video understanding and description\n\n## Evaluation Notes\n\n- Default data source: `evalscope/MSVD` on ModelScope, `test` split\n- Hugging Face `VLM2Vec/MSVD` remains available by setting `extra_params.dataset_hub=\"huggingface\"`\n- Primary metric: **CIDEr**\n- Additional metrics: BLEU-1/2/3/4, METEOR, ROUGE-L\n- Set `extra_params.video_dir` when the dataset only provides video file names and local media files are required\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `msvd` |\n| **Dataset ID** | [evalscope/MSVD](https://modelscope.cn/datasets/evalscope/MSVD/summary) |\n| **Paper** | [Paper](https://aclanthology.org/P11-1020/) |\n| **Tags** | `ImageCaptioning`, `MultiModal` |\n| **Metrics** | `Bleu_1`, `Bleu_2`, `Bleu_3`, `Bleu_4`, `METEOR`, `ROUGE_L`, `CIDEr` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 670 |\n| Prompt Length (Mean) | 43 chars |\n| Prompt Length (Min/Max) | 43 / 43 chars |\n\n**Video Statistics:**\n\n| Metric | Value |\n|--------|-------|\n| Total Videos | 670 |\n| Videos per Sample | min: 1, max: 1, mean: 1 |\n| Formats | mp4 |\n\n\n## Sample Example\n\n**Subset**: `default`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"a4b83275\",\n      \"content\": [\n        {\n          \"text\": \"Describe the video in one concise sentence.\"\n        },\n        {\n          \"video\": \"fr9H1WLcF1A_256_261.avi\",\n          \"format\": \"mp4\"\n        }\n      ]\n    }\n  ],\n  \"target\": \"[\\\"two young men are playing table tennis\\\", \\\"men are playing table tennis\\\", \\\"two men are playing table tennis\\\", \\\"two men are playing a tabletennis\\\", \\\"a couple of people are playing a game of ping pong\\\", \\\"peoples are playing table tennis\\\", \\\"two ... [TRUNCATED 306 chars] ... e boys are playing\\\", \\\"people are playing ping pong\\\", \\\"18 kids and counting show the newest baby josie\\\", \\\"there is some kids and playing to each other\\\", \\\"the men played pingpong together\\\", \\\"the boys are playing ping pong\\\", \\\"2 boys is playing\\\"]\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"references\": [\n      \"two young men are playing table tennis\",\n      \"men are playing table tennis\",\n      \"two men are playing table tennis\",\n      \"two men are playing a tabletennis\",\n      \"a couple of people are playing a game of ping pong\",\n      \"peoples are playing table tennis\",\n      \"two men are playing pingpong\",\n      \"two guys play table tennis\",\n      \"two people are playing ping pong\",\n      \"two boys are playing ping pong\",\n      \"... [TRUNCATED 12 more items] ...\"\n    ],\n    \"subset\": \"default\",\n    \"dataset_id\": \"evalscope/MSVD\",\n    \"dataset_hub\": \"modelscope\",\n    \"video\": \"fr9H1WLcF1A_256_261.avi\",\n    \"video_id\": \"fr9H1WLcF1A_256_261\",\n    \"source\": \"MSVD\"\n  }\n}\n```\n\n## Prompt Template\n\n**Prompt Template:**\n```text\nDescribe the video in one concise sentence.\n```\n\n## Extra Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `dataset_hub` | `str` | `modelscope` | Dataset hub used to load MSVD annotations. Choices: ['huggingface', 'modelscope', 'local'] |\n| `eval_split` | `str` | `` | Source split to load; defaults to test. |\n| `dataset_revision` | `str` | `` | Optional dataset revision; leave empty to use the hub default. |\n| `video_dir` | `str` | `` | Optional local directory containing MSVD video files. |\n| `video_extension` | `str` | `` | Optional extension override for local videos, for example \"mp4\". |\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets msvd \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['msvd'],\n    dataset_args={\n        'msvd': {\n            # extra_params: {}  # uses default extra parameters\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# MSVD\n\n\n## 概述\n\nMSVD 是一个经典的视频描述（video captioning）基准测试，包含大量带有多个人工标注字幕的短视频。\n原生适配器将每个视频视为一个评估样本，并使用所有可用字幕作为参考。\n\n## 任务描述\n\n- **任务类型**：视频描述（Video captioning）\n- **输入**：视频片段\n- **输出**：一条简洁的自然语言字幕\n- **领域**：开放域视频理解与描述\n\n## 评估说明\n\n- 默认数据源：ModelScope 上的 `evalscope/MSVD`，使用 `test` 划分\n- 通过设置 `extra_params.dataset_hub=\"huggingface\"`，仍可使用 Hugging Face 上的 `VLM2Vec/MSVD`\n- 主要指标：**CIDEr**\n- 其他指标：BLEU-1/2/3/4、METEOR、ROUGE-L\n- 当数据集仅提供视频文件名且需要本地媒体文件时，请设置 `extra_params.video_dir`\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `msvd` |\n| **数据集ID** | [evalscope/MSVD](https://modelscope.cn/datasets/evalscope/MSVD/summary) |\n| **论文** | [Paper](https://aclanthology.org/P11-1020/) |\n| **标签** | `ImageCaptioning`, `MultiModal` |\n| **指标** | `Bleu_1`, `Bleu_2`, `Bleu_3`, `Bleu_4`, `METEOR`, `ROUGE_L`, `CIDEr` |\n| **默认示例数** | 0-shot |\n| **评估划分** | `test` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 670 |\n| 提示词长度（平均） | 43 字符 |\n| 提示词长度（最小/最大） | 43 / 43 字符 |\n\n**视频统计信息：**\n\n| 指标 | 值 |\n|--------|-------|\n| 视频总数 | 670 |\n| 每样本视频数 | 最小: 1, 最大: 1, 平均: 1 |\n| 格式 | mp4 |\n\n\n## 样例示例\n\n**子集**：`default`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"a4b83275\",\n      \"content\": [\n        {\n          \"text\": \"Describe the video in one concise sentence.\"\n        },\n        {\n          \"video\": \"fr9H1WLcF1A_256_261.avi\",\n          \"format\": \"mp4\"\n        }\n      ]\n    }\n  ],\n  \"target\": \"[\\\"two young men are playing table tennis\\\", \\\"men are playing table tennis\\\", \\\"two men are playing table tennis\\\", \\\"two men are playing a tabletennis\\\", \\\"a couple of people are playing a game of ping pong\\\", \\\"peoples are playing table tennis\\\", \\\"two ... [TRUNCATED 306 chars] ... e boys are playing\\\", \\\"people are playing ping pong\\\", \\\"18 kids and counting show the newest baby josie\\\", \\\"there is some kids and playing to each other\\\", \\\"the men played pingpong together\\\", \\\"the boys are playing ping pong\\\", \\\"2 boys is playing\\\"]\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"references\": [\n      \"two young men are playing table tennis\",\n      \"men are playing table tennis\",\n      \"two men are playing table tennis\",\n      \"two men are playing a tabletennis\",\n      \"a couple of people are playing a game of ping pong\",\n      \"peoples are playing table tennis\",\n      \"two men are playing pingpong\",\n      \"two guys play table tennis\",\n      \"two people are playing ping pong\",\n      \"two boys are playing ping pong\",\n      \"... [TRUNCATED 12 more items] ...\"\n    ],\n    \"subset\": \"default\",\n    \"dataset_id\": \"evalscope/MSVD\",\n    \"dataset_hub\": \"modelscope\",\n    \"video\": \"fr9H1WLcF1A_256_261.avi\",\n    \"video_id\": \"fr9H1WLcF1A_256_261\",\n    \"source\": \"MSVD\"\n  }\n}\n```\n\n## 提示模板\n\n**提示模板：**\n```text\nDescribe the video in one concise sentence.\n```\n\n## 额外参数\n\n| 参数 | 类型 | 默认值 | 描述 |\n|-----------|------|---------|-------------|\n| `dataset_hub` | `str` | `modelscope` | 用于加载 MSVD 标注的数据集平台。选项：['huggingface', 'modelscope', 'local'] |\n| `eval_split` | `str` | `` | 要加载的数据划分；默认为 test。 |\n| `dataset_revision` | `str` | `` | 可选的数据集版本；留空则使用平台默认版本。 |\n| `video_dir` | `str` | `` | 包含 MSVD 视频文件的本地目录（可选）。 |\n| `video_extension` | `str` | `` | 本地视频文件的扩展名覆盖项（可选），例如 \"mp4\"。 |\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets msvd \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['msvd'],\n    dataset_args={\n        'msvd': {\n            # extra_params: {}  # 使用默认额外参数\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "8b9a7bf6e44733c00672786b2b6afb04",
    "needs_translation": false
  },
  "updated_at": "2026-06-09T15:45:54.351306",
  "translation_updated_at": "2026-06-09T15:46:01"
}