GSM8K-V

Overview

GSM8K-V is a purely visual multi-image mathematical reasoning benchmark that systematically transforms each GSM8K math word problem into its visual counterpart. It enables clean within-item comparison across modalities for multimodal math evaluation.

Task Description

Task Type: Visual Mathematical Word Problem Solving
Input: Multiple images representing a math problem + question
Output: Numerical answer in \boxed{} format
Domains: Visual math reasoning, scene understanding, arithmetic

Key Features

Visual transformation of GSM8K text problems
Multi-image input format for complete problem representation
Enables text-vs-visual modality comparisons
Categorized by problem type and subcategory
Preserves original problem difficulty levels

Evaluation Notes

Default evaluation uses the train split
Primary metric: Accuracy with numeric comparison
Answers should be in \boxed{} format with number only
Step-by-step reasoning encouraged before final answer
Metadata includes original text question for comparison

Properties

Property	Value
Benchmark Name	`gsm8k_v`
Dataset ID	evalscope/GSM8K-V
Paper	N/A
Tags	`Math`, `MultiModal`, `Reasoning`
Metrics	`acc`
Default Shots	0-shot
Evaluation Split	`train`

Data Statistics

Metric	Value
Total Samples	1,319
Prompt Length (Mean)	492.4 chars
Prompt Length (Min/Max)	410 / 789 chars

Image Statistics:

Metric	Value
Total Images	5,344
Images per Sample	min: 2, max: 11, mean: 4.05
Resolution Range	1024x1024 - 1024x1024
Formats	jpeg

Sample Example

Subset: default

{
  "input": [
    {
      "id": "9e0072fd",
      "content": [
        {
          "text": "You are an expert at solving mathematical word problems. Please solve the following problem step by step, showing your reasoning.\n\nWhen providing your final answer:\n- If the answer can be expressed as a whole number (integer), provide it as a ... [TRUNCATED] ... s Janet (white woman, shoulder-length brown hair, wearing a light blue blouse and khaki pants) make each day at the farmers' market?\n\nPlease think step by step. After your reasoning, put your final answer within \\boxed{} with the number only."
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~153.1KB]"
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~175.1KB]"
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~124.7KB]"
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~172.8KB]"
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~117.4KB]"
        }
      ]
    }
  ],
  "target": "18.0",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "index": 1,
    "category": "other",
    "subcategory": "count",
    "original_question": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?"
  }
}

Note: Some content was truncated for display.

Prompt Template

Prompt Template:

You are an expert at solving mathematical word problems. Please solve the following problem step by step, showing your reasoning.

When providing your final answer:
- If the answer can be expressed as a whole number (integer), provide it as an integer
Problem: {question}

Please think step by step. After your reasoning, put your final answer within \boxed{{}} with the number only.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets gsm8k_v \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['gsm8k_v'],
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

4.3 KiB Raw Permalink Blame History Unescape Escape