evalstone/evalscope/docs/en/benchmarks/emb_spatial_bench.md

EmbSpatial-Bench

Overview

EmbSpatial-Bench is a benchmark for evaluating embodied spatial understanding of large vision-language models (LVLMs). The benchmark is automatically derived from embodied scenes and covers 6 spatial relationships from an egocentric perspective: close, far, above, under, left, and right.

Task Description

• Task Type: Multiple-Choice Visual Question Answering (VQA)
• Input: An egocentric RGB image + a spatial reasoning question with 4 candidate answers
• Output: A single letter (A / B / C / D) identifying the correct object or spatial relationship
• Domains: Embodied AI, spatial reasoning (MP3D and AI2Thor environments)

Key Features

• 3,640 human-verified evaluation questions derived from two embodied environments (MP3D and AI2Thor)
• 6 spatial relation categories: close, far, above, under, left, right
• Each question requires selecting the most spatially accurate answer from 4 options
• Designed to expose the gap between current LVLMs and qualified embodied intelligence

Evaluation Notes

• Default evaluation uses the embspatial_bench.json file (3,640 samples)
• Primary metric: Accuracy (accuracy)
• Answer indices are 0-based in the dataset (0 → A, 1 → B, 2 → C, 3 → D)
• Images are stored as JPEG base64 strings in the JSON file
• Subsets are organized by the relation field (6 spatial categories)
• Paper | GitHub

Properties

Property	Value
Benchmark Name	emb_spatial_bench
Dataset ID	evalscope/EmbSpatial-Bench
Paper	Paper
Tags	MCQ, MultiModal, Reasoning
Metrics	accuracy
Default Shots	0-shot
Evaluation Split	test

Data Statistics

Metric	Value
Total Samples	3,640
Prompt Length (Mean)	369.34 chars
Prompt Length (Min/Max)	265 / 490 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
close	612	293.54	274	326
far	594	292.68	265	326
above	596	405.61	360	486
under	602	404.35	350	490
left	616	408.86	359	486
right	620	409.47	352	481

Image Statistics:

Metric	Value
Total Images	3,640
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	300x300 - 1296x968
Formats	jpeg

Sample Example

Subset: close

{
  "input": [
    {
      "id": "3f406c29",
      "content": [
        {
          "image": "[BASE64_IMAGE: jpeg, ~35.2KB]"
        },
        {
          "text": "Among the listed objects, which one is closest to your current location in the image?\n(A) table\n(B) towel\n(C) door\n(D) basket\nAnswer with only the letter of the correct option. The last line of your response should be of the format: ANSWER: [LETTER] where LETTER is one of A, B, C, D."
        }
      ]
    }
  ],
  "target": "D",
  "id": 0,
  "group_id": 0,
  "subset_key": "close",
  "metadata": {
    "question_id": "mp3d_0",
    "relation": "close",
    "data_source": "mp3d"
  }
}

Prompt Template

Prompt Template:

Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.

{question}

{choices}

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets emb_spatial_bench \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['emb_spatial_bench'],
    dataset_args={
        'emb_spatial_bench': {
            # subset_list: ['close', 'far', 'above']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)