ERQA

Overview

ERQA (Embodied Reasoning QA) is a benchmark for evaluating spatial reasoning and embodied understanding capabilities of multimodal large language models. It tests models' ability to reason about trajectories, actions, spatial relationships, and task planning in egocentric robotic scenarios.

Task Description

Task Type: Embodied Spatial Reasoning (Multiple Choice)
Input: Egocentric image(s) + multiple-choice question (A/B/C/D)
Output: Single answer letter (A/B/C/D)
Domain: Robotics, spatial reasoning, embodied AI

Key Features

400 questions across 8 reasoning categories
Multi-image support (some questions require reasoning across multiple views)
Categories: Trajectory Reasoning, Action Reasoning, Pointing, State Estimation, Spatial Reasoning, Multi-view Reasoning, Task Reasoning, Other
Egocentric perspective from robotic manipulation scenarios

Evaluation Notes

Default configuration uses 0-shot evaluation
Evaluates on test split
Primary metric: Accuracy on multiple-choice questions
Answers are single letters (A/B/C/D)

Properties

Property	Value
Benchmark Name	`erqa`
Dataset ID	evalscope/ERQA
Paper	N/A
Tags	`MCQ`, `MultiModal`, `Reasoning`
Metrics	`acc`
Default Shots	0-shot
Evaluation Split	`test`

Data Statistics

Metric	Value
Total Samples	400
Prompt Length (Mean)	289.05 chars
Prompt Length (Min/Max)	147 / 834 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
`Trajectory Reasoning`	66	307.3	161	834
`Action Reasoning`	72	271.97	175	503
`Pointing`	34	300.82	178	676
`State Estimation`	55	314.84	147	735
`Spatial Reasoning`	84	286.21	152	819
`Multi-view Reasoning`	37	277.78	196	475
`Task Reasoning`	38	280.18	152	737
`Other`	14	231.79	188	409

Image Statistics:

Metric	Value
Total Images	630
Images per Sample	min: 1, max: 16, mean: 1.57
Resolution Range	200x150 - 3072x4080
Formats	jpeg

Sample Example

Subset: Trajectory Reasoning

{
  "input": [
    {
      "id": "eec4c5cf",
      "content": [
        {
          "text": "If the yellow robot gripper follows the yellow trajectory, what will happen? Choices: A. Robot puts the soda on the wooden steps. B. Robot moves the soda in front of the wooden steps. C. Robot moves the soda to the very top of the wooden steps. D. Robot picks up the soda can and moves it up. Please answer directly with only the letter of the correct option and nothing else."
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~16.0KB]"
        }
      ]
    }
  ],
  "choices": [
    "A",
    "B",
    "C",
    "D"
  ],
  "target": "A",
  "id": 0,
  "group_id": 0,
  "subset_key": "Trajectory Reasoning",
  "metadata": {
    "question_id": "ERQA_1",
    "question_type": "Trajectory Reasoning"
  }
}

Prompt Template

Prompt Template:

{question}

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets erqa \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['erqa'],
    dataset_args={
        'erqa': {
            # subset_list: ['Trajectory Reasoning', 'Action Reasoning', 'Pointing']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

3.9 KiB Raw Permalink Blame History