sora 13274243a0 Bump vendored EvalScope and add K3-ready DPV4 configs.

Keep K3 suite selection and report-schema scoring in bash, merge K3/vision dataset_args into dpv4 yamls, and pin EvalScope at 735d920ee911 with local patches.

Co-authored-by: Cursor <cursoragent@cursor.com>

2026-09-02 07:30:48 +00:00

3.6 KiB

Raw Blame History

BabyVision

Overview

BabyVision is a visual perception benchmark that evaluates the fundamental visual abilities of multimodal large language models through tasks inspired by infant and early childhood visual development. It focuses on fine-grained discrimination, spatial perception, visual pattern recognition, and visual tracking.

Task Description

Task Type: Visual Perception (Choice + Fill-in-the-blank)
Input: Image + question
Output: Choice letter or free-form short answer
Domains: Fine-grained discrimination, spatial perception, visual pattern recognition, visual tracking

Key Features

388 test samples across 4 major visual ability categories and 22 subtypes
Two answer types: choice (135 samples) and blank (253 samples)
Subtypes include: Find the different, Find the same, Count clusters, Maze, 3D cube unfold, Pattern completion, Paper folding, Rotation patterns, etc.
Tests low-level visual perception rather than high-level reasoning or knowledge
Includes Chain-of-Thought (CoT) reference for analysis

Evaluation Notes

Default evaluation uses the train split (388 samples, single split dataset)
Primary metric: Accuracy via LLM-as-judge
Subsets organized by type field (4 categories)
LLM judge evaluates both choice and blank answer types uniformly
Requires an LLM judge configured through judge.models

Properties

Property	Value
Benchmark Name	`baby_vision`
Dataset ID	evalscope/BabyVision
Paper	N/A
Tags	`MultiModal`, `QA`, `Reasoning`
Metrics	`accuracy`
Default Shots	0-shot
Evaluation Split	`train`

Data Statistics

Statistics not available.

Sample Example

Subset: Fine-grained Discrimination

{
  "input": [
    {
      "id": "1256c961",
      "content": [
        {
          "image": "[BASE64_IMAGE: jpeg, ~77.0KB]"
        },
        {
          "text": "The image shows a total of 49 tiger patterns arranged in 7 rows and 7 columns. One of them is different from the others. Which row and column is it in? The answer format is (x,y). (For example, the answer for the 2nd row and 3rd column is (2,3)).\nThink about the question and give your final answer in \\boxed{Answer} format."
        }
      ]
    }
  ],
  "target": "(4,7)",
  "id": 0,
  "group_id": 0,
  "subset_key": "Fine-grained Discrimination",
  "metadata": {
    "taskId": 445,
    "type": "Fine-grained Discrimination",
    "subtype": "Find the different",
    "ansType": "blank",
    "coT": "The image shows 49 tiger patterns arranged in 7 rows and 7 columns.\nNow, we need to find the coordinates of the one tiger pattern that is different from the other 48.\nIt can be observed that the tiger in the fourth row and seventh column has no ears (the ears are located in the upper right corner of each tiger pattern), while the other 48 tigers have ears.\nTherefore, the correct answer is (4,7)."
  }
}

Prompt Template

No prompt template defined.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets baby_vision \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['baby_vision'],
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

3.6 KiB Raw Blame History