{
  "meta": {
    "pretty_name": "BabyVision",
    "dataset_id": "evalscope/BabyVision",
    "paper_url": null,
    "tags": [
      "MultiModal",
      "Reasoning",
      "QA"
    ],
    "metrics": [
      "acc"
    ],
    "few_shot_num": 0,
    "eval_split": "train",
    "train_split": "",
    "subset_list": [
      "Fine-grained Discrimination",
      "Spatial Perception",
      "Visual Pattern Recognition",
      "Visual Tracking"
    ],
    "description": "\n## Overview\n\nBabyVision is a visual perception benchmark that evaluates the fundamental visual abilities of\nmultimodal large language models through tasks inspired by infant and early childhood visual\ndevelopment. It focuses on fine-grained discrimination, spatial perception, visual pattern\nrecognition, and visual tracking.\n\n## Task Description\n\n- **Task Type**: Visual Perception (Choice + Fill-in-the-blank)\n- **Input**: Image + question\n- **Output**: Choice letter or free-form short answer\n- **Domains**: Fine-grained discrimination, spatial perception, visual pattern recognition, visual tracking\n\n## Key Features\n\n- 388 test samples across 4 major visual ability categories and 22 subtypes\n- Two answer types: choice (135 samples) and blank (253 samples)\n- Subtypes include: Find the different, Find the same, Count clusters, Maze,\n  3D cube unfold, Pattern completion, Paper folding, Rotation patterns, etc.\n- Tests low-level visual perception rather than high-level reasoning or knowledge\n- Includes Chain-of-Thought (CoT) reference for analysis\n\n## Evaluation Notes\n\n- Default evaluation uses the **train** split (388 samples, single split dataset)\n- Primary metric: **Accuracy** via LLM-as-judge\n- Subsets organized by `type` field (4 categories)\n- LLM judge evaluates both choice and blank answer types uniformly\n- Requires `judge_model_args` configuration for LLM judge\n",
    "prompt_template": "",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {},
    "sandbox_config": {},
    "category": "vlm"
  },
  "statistics": {
    "total_samples": 388,
    "subset_stats": [
      {
        "name": "Fine-grained Discrimination",
        "sample_count": 163,
        "prompt_length_mean": 152.09,
        "prompt_length_min": 33,
        "prompt_length_max": 450,
        "prompt_length_std": 78.17,
        "target_length_mean": 6.91,
        "multimodal": {
          "has_images": true,
          "has_audio": false,
          "has_video": false,
          "image": {
            "count_total": 163,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "resolutions": [
              "1094x390",
              "1104x1027",
              "1212x882",
              "1256x1017",
              "213x270",
              "216x275",
              "228x228",
              "234x228",
              "239x225",
              "243x225"
            ],
            "resolution_range": {
              "min": "228x228",
              "max": "758x1810"
            },
            "formats": [
              "jpeg",
              "png"
            ]
          }
        }
      },
      {
        "name": "Spatial Perception",
        "sample_count": 91,
        "prompt_length_mean": 157.18,
        "prompt_length_min": 73,
        "prompt_length_max": 370,
        "prompt_length_std": 61.52,
        "target_length_mean": 4.68,
        "multimodal": {
          "has_images": true,
          "has_audio": false,
          "has_video": false,
          "image": {
            "count_total": 91,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "resolutions": [
              "1034x1152",
              "1050x854",
              "1071x816",
              "1086x1184",
              "1088x872",
              "1092x916",
              "1096x954",
              "1100x1112",
              "1179x1619",
              "1254x537"
            ],
            "resolution_range": {
              "min": "209x168",
              "max": "1179x1619"
            },
            "formats": [
              "jpeg",
              "png"
            ]
          }
        }
      },
      {
        "name": "Visual Pattern Recognition",
        "sample_count": 51,
        "prompt_length_mean": 178.92,
        "prompt_length_min": 94,
        "prompt_length_max": 319,
        "prompt_length_std": 53.47,
        "target_length_mean": 3.73,
        "multimodal": {
          "has_images": true,
          "has_audio": false,
          "has_video": false,
          "image": {
            "count_total": 51,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "resolutions": [
              "1023x605",
              "1224x1110",
              "164x226",
              "246x571",
              "264x600",
              "271x600",
              "303x717",
              "332x600",
              "360x476",
              "373x673"
            ],
            "resolution_range": {
              "min": "164x226",
              "max": "1224x1110"
            },
            "formats": [
              "jpeg",
              "png"
            ]
          }
        }
      },
      {
        "name": "Visual Tracking",
        "sample_count": 83,
        "prompt_length_mean": 201.45,
        "prompt_length_min": 97,
        "prompt_length_max": 389,
        "prompt_length_std": 64.88,
        "target_length_mean": 9.33,
        "multimodal": {
          "has_images": true,
          "has_audio": false,
          "has_video": false,
          "image": {
            "count_total": 83,
            "count_per_sample": {
              "min": 1,
              "max": 1,
              "mean": 1
            },
            "resolutions": [
              "1012x984",
              "1400x1406",
              "1502x1211",
              "1690x1466",
              "1744x1080",
              "174x144",
              "174x147",
              "180x144",
              "189x144",
              "189x147"
            ],
            "resolution_range": {
              "min": "174x144",
              "max": "2378x1448"
            },
            "formats": [
              "jpeg",
              "png",
              "webp"
            ]
          }
        }
      }
    ],
    "prompt_length": {
      "mean": 167.37,
      "min": 33,
      "max": 450,
      "std": 71.34
    },
    "target_length_mean": 6.49,
    "computed_at": "2026-07-03T11:09:46.867162",
    "multimodal": {
      "has_images": true,
      "has_audio": false,
      "has_video": false,
      "image": {
        "count_total": 388,
        "count_per_sample": {
          "min": 1,
          "max": 1,
          "mean": 1
        },
        "resolutions": [
          "1012x984",
          "1023x605",
          "1034x1152",
          "1050x854",
          "1071x816",
          "1086x1184",
          "1088x872",
          "1092x916",
          "1094x390",
          "1096x954"
        ],
        "resolution_range": {
          "min": "174x144",
          "max": "2378x1448"
        },
        "formats": [
          "jpeg",
          "png",
          "webp"
        ]
      }
    }
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "8b83904b",
          "content": [
            {
              "image": "[BASE64_IMAGE: jpeg, ~77.0KB]"
            },
            {
              "text": "The image shows a total of 49 tiger patterns arranged in 7 rows and 7 columns. One of them is different from the others. Which row and column is it in? The answer format is (x,y). (For example, the answer for the 2nd row and 3rd column is (2,3))."
            }
          ]
        }
      ],
      "target": "(4,7)",
      "id": 0,
      "group_id": 0,
      "subset_key": "Fine-grained Discrimination",
      "metadata": {
        "taskId": 445,
        "type": "Fine-grained Discrimination",
        "subtype": "Find the different",
        "ansType": "blank",
        "coT": "The image shows 49 tiger patterns arranged in 7 rows and 7 columns.\nNow, we need to find the coordinates of the one tiger pattern that is different from the other 48.\nIt can be observed that the tiger in the fourth row and seventh column has no ears (the ears are located in the upper right corner of each tiger pattern), while the other 48 tigers have ears.\nTherefore, the correct answer is (4,7)."
      }
    },
    "subset": "Fine-grained Discrimination",
    "truncated": false
  },
  "readme": {
    "en": "# BabyVision\n\n\n## Overview\n\nBabyVision is a visual perception benchmark that evaluates the fundamental visual abilities of\nmultimodal large language models through tasks inspired by infant and early childhood visual\ndevelopment. It focuses on fine-grained discrimination, spatial perception, visual pattern\nrecognition, and visual tracking.\n\n## Task Description\n\n- **Task Type**: Visual Perception (Choice + Fill-in-the-blank)\n- **Input**: Image + question\n- **Output**: Choice letter or free-form short answer\n- **Domains**: Fine-grained discrimination, spatial perception, visual pattern recognition, visual tracking\n\n## Key Features\n\n- 388 test samples across 4 major visual ability categories and 22 subtypes\n- Two answer types: choice (135 samples) and blank (253 samples)\n- Subtypes include: Find the different, Find the same, Count clusters, Maze,\n  3D cube unfold, Pattern completion, Paper folding, Rotation patterns, etc.\n- Tests low-level visual perception rather than high-level reasoning or knowledge\n- Includes Chain-of-Thought (CoT) reference for analysis\n\n## Evaluation Notes\n\n- Default evaluation uses the **train** split (388 samples, single split dataset)\n- Primary metric: **Accuracy** via LLM-as-judge\n- Subsets organized by `type` field (4 categories)\n- LLM judge evaluates both choice and blank answer types uniformly\n- Requires `judge_model_args` configuration for LLM judge\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `baby_vision` |\n| **Dataset ID** | [evalscope/BabyVision](https://modelscope.cn/datasets/evalscope/BabyVision/summary) |\n| **Paper** | N/A |\n| **Tags** | `MultiModal`, `QA`, `Reasoning` |\n| **Metrics** | `acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `train` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 388 |\n| Prompt Length (Mean) | 167.37 chars |\n| Prompt Length (Min/Max) | 33 / 450 chars |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `Fine-grained Discrimination` | 163 | 152.09 | 33 | 450 |\n| `Spatial Perception` | 91 | 157.18 | 73 | 370 |\n| `Visual Pattern Recognition` | 51 | 178.92 | 94 | 319 |\n| `Visual Tracking` | 83 | 201.45 | 97 | 389 |\n\n**Image Statistics:**\n\n| Metric | Value |\n|--------|-------|\n| Total Images | 388 |\n| Images per Sample | min: 1, max: 1, mean: 1 |\n| Resolution Range | 174x144 - 2378x1448 |\n| Formats | jpeg, png, webp |\n\n\n## Sample Example\n\n**Subset**: `Fine-grained Discrimination`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"8b83904b\",\n      \"content\": [\n        {\n          \"image\": \"[BASE64_IMAGE: jpeg, ~77.0KB]\"\n        },\n        {\n          \"text\": \"The image shows a total of 49 tiger patterns arranged in 7 rows and 7 columns. One of them is different from the others. Which row and column is it in? The answer format is (x,y). (For example, the answer for the 2nd row and 3rd column is (2,3)).\"\n        }\n      ]\n    }\n  ],\n  \"target\": \"(4,7)\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"subset_key\": \"Fine-grained Discrimination\",\n  \"metadata\": {\n    \"taskId\": 445,\n    \"type\": \"Fine-grained Discrimination\",\n    \"subtype\": \"Find the different\",\n    \"ansType\": \"blank\",\n    \"coT\": \"The image shows 49 tiger patterns arranged in 7 rows and 7 columns.\\nNow, we need to find the coordinates of the one tiger pattern that is different from the other 48.\\nIt can be observed that the tiger in the fourth row and seventh column has no ears (the ears are located in the upper right corner of each tiger pattern), while the other 48 tigers have ears.\\nTherefore, the correct answer is (4,7).\"\n  }\n}\n```\n\n## Prompt Template\n\n*No prompt template defined.*\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets baby_vision \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['baby_vision'],\n    dataset_args={\n        'baby_vision': {\n            # subset_list: ['Fine-grained Discrimination', 'Spatial Perception', 'Visual Pattern Recognition']  # optional, evaluate specific subsets\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# BabyVision\n\n\n## 概述\n\nBabyVision 是一个视觉感知基准测试，通过受婴儿及幼儿视觉发展启发的任务，评估多模态大语言模型的基础视觉能力。该基准聚焦于细粒度辨别、空间感知、视觉模式识别和视觉追踪能力。\n\n## 任务描述\n\n- **任务类型**：视觉感知（选择题 + 填空题）\n- **输入**：图像 + 问题\n- **输出**：选项字母或自由格式的简短答案\n- **领域**：细粒度辨别、空间感知、视觉模式识别、视觉追踪\n\n## 主要特点\n\n- 包含 388 个测试样本，覆盖 4 大视觉能力类别和 22 个子类型\n- 两种答案类型：选择题（135 个样本）和填空题（253 个样本）\n- 子类型包括：找不同、找相同、数簇、迷宫、立方体展开图、图案补全、折纸、旋转图案等\n- 侧重于低层次视觉感知，而非高层次推理或知识\n- 提供思维链（Chain-of-Thought, CoT）参考用于分析\n\n## 评估说明\n\n- 默认评估使用 **train** 切分（388 个样本，单切分数据集）\n- 主要指标：通过 LLM-as-judge 计算的 **准确率（Accuracy）**\n- 子集按 `type` 字段组织（4 个类别）\n- LLM 评判器统一评估选择题和填空题两种答案类型\n- 需要配置 `judge_model_args` 以启用 LLM 评判器\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `baby_vision` |\n| **数据集ID** | [evalscope/BabyVision](https://modelscope.cn/datasets/evalscope/BabyVision/summary) |\n| **论文** | N/A |\n| **标签** | `MultiModal`, `QA`, `Reasoning` |\n| **指标** | `acc` |\n| **默认示例数** | 0-shot |\n| **评估切分** | `train` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 388 |\n| 提示词长度（平均） | 167.37 字符 |\n| 提示词长度（最小/最大） | 33 / 450 字符 |\n\n**各子集统计信息：**\n\n| 子集 | 样本数 | 提示平均长度 | 提示最小长度 | 提示最大长度 |\n|--------|---------|-------------|------------|------------|\n| `Fine-grained Discrimination` | 163 | 152.09 | 33 | 450 |\n| `Spatial Perception` | 91 | 157.18 | 73 | 370 |\n| `Visual Pattern Recognition` | 51 | 178.92 | 94 | 319 |\n| `Visual Tracking` | 83 | 201.45 | 97 | 389 |\n\n**图像统计信息：**\n\n| 指标 | 值 |\n|--------|-------|\n| 图像总数 | 388 |\n| 每样本图像数 | 最小: 1, 最大: 1, 平均: 1 |\n| 分辨率范围 | 174x144 - 2378x1448 |\n| 格式 | jpeg, png, webp |\n\n\n## 样例示例\n\n**子集**: `Fine-grained Discrimination`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"8b83904b\",\n      \"content\": [\n        {\n          \"image\": \"[BASE64_IMAGE: jpeg, ~77.0KB]\"\n        },\n        {\n          \"text\": \"The image shows a total of 49 tiger patterns arranged in 7 rows and 7 columns. One of them is different from the others. Which row and column is it in? The answer format is (x,y). (For example, the answer for the 2nd row and 3rd column is (2,3)).\"\n        }\n      ]\n    }\n  ],\n  \"target\": \"(4,7)\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"subset_key\": \"Fine-grained Discrimination\",\n  \"metadata\": {\n    \"taskId\": 445,\n    \"type\": \"Fine-grained Discrimination\",\n    \"subtype\": \"Find the different\",\n    \"ansType\": \"blank\",\n    \"coT\": \"The image shows 49 tiger patterns arranged in 7 rows and 7 columns.\\nNow, we need to find the coordinates of the one tiger pattern that is different from the other 48.\\nIt can be observed that the tiger in the fourth row and seventh column has no ears (the ears are located in the upper right corner of each tiger pattern), while the other 48 tigers have ears.\\nTherefore, the correct answer is (4,7).\"\n  }\n}\n```\n\n## 提示模板\n\n*未定义提示模板。*\n\n## 使用方法\n\n### 通过命令行（CLI）\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets baby_vision \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 通过 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['baby_vision'],\n    dataset_args={\n        'baby_vision': {\n            # subset_list: ['Fine-grained Discrimination', 'Spatial Perception', 'Visual Pattern Recognition']  # 可选，用于评估特定子集\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "e1a80e171a3094d96b695738ff04de7e",
    "needs_translation": false
  },
  "updated_at": "2026-07-03T11:09:47.265495",
  "translation_updated_at": "2026-07-03T16:19:29"
}