evalstone/evalscope/docs/en/benchmarks/screenspot_pro.md

ScreenSpot-Pro

Overview

ScreenSpot-Pro is a GUI grounding benchmark built from authentic high-resolution screenshots of professional desktop software. Given a natural-language instruction, a model must locate the target UI element on the screen, which stresses fine-grained localization on large, densely populated displays.

Task Description

• Task Type: GUI grounding (single click-point prediction)
• Input: A full-resolution desktop screenshot + an English instruction describing the target UI element
• Output: One click point [x, y] normalized to the range 0 to 1, given after an Answer: marker
• Domain: Professional desktop applications across CAD, Creative, Dev, Office, OS and Scientific software

Key Features

• 1,581 expert-annotated instructions over 26 applications and 3 platforms (Windows, macOS, Linux)
• Screenshots are genuinely high-resolution (up to 6016x3384), so target elements often occupy well under 0.1% of the image
• Samples are grouped into six professional domains (CAD, Creative, Dev, OS, Office, Scientific), each exposed as a subset
• Every element is labelled as text or icon, enabling separate reporting for textual versus iconographic targets
• Ground-truth boxes are pixel coordinates paired with the original image size, and are normalized before scoring

Evaluation Notes

• Primary metric: accuracy — a prediction is correct when the predicted point falls inside the ground-truth bounding box
• Secondary metrics: text_acc and icon_acc, each averaged over the samples of the corresponding ui_type
• Predictions are read from the answer line that the prompt requires (Answer: [x, y]), so reasoning traces cannot be mistaken for the answer. Replies ignoring the format fall back to scanning for unambiguous point notation only ([x, y] pairs or <bbox> tags); loose notation such as x=.., y=.. and bare numbers is accepted only on the answer line, because in free prose it harvests layout bounds and ordinals instead of a click point
• A reply truncated before its answer line yields no prediction and scores 0 rather than a coordinate invented from its reasoning, so allow enough max_tokens for the model to finish answering
• Ground truth is normalized to [0, 1], and predictions are mapped into the same space by magnitude: values in [0, 1] are taken as normalized, values up to 1000 as the thousandths grid many VLMs emit, and larger values as pixels of the image the model received (every screenshot is at least 1920 px wide, so genuine pixel answers are classified correctly)
• The dataset ships a single train split, which is used as the evaluation split
• Images are large; max_image_bytes in dataset_args can cap the request size, and pixel-space predictions are normalized with the size of the image actually sent
• Paper | GitHub

Properties

Property	Value
Benchmark Name	screenspot_pro
Dataset ID	lmms-lab/ScreenSpot-Pro
Paper	Paper
Tags	Agent, Grounding, MultiModal
Metrics	accuracy, text_acc, icon_acc
Default Shots	0-shot
Evaluation Split	train

Data Statistics

Metric	Value
Total Samples	1,581
Prompt Length (Mean)	319.22 chars
Prompt Length (Min/Max)	295 / 395 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
CAD	261	313.18	296	344
Creative	341	318.02	296	395
Dev	299	329.79	296	392
OS	196	317.57	297	382
Office	230	320.76	296	372
Scientific	254	314.44	295	353

Image Statistics:

Metric	Value
Total Images	1,581
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	1920x1080 - 6016x3384
Formats	png

Sample Example

Subset: CAD

{
  "input": [
    {
      "id": "2d87e94f",
      "content": [
        {
          "image": "[BASE64_IMAGE: png, ~933.3KB]"
        },
        {
          "text": "Identify the UI element for the instruction and give a single click point. Coordinates must be normalized to the range 0 to 1 relative to the image size. Do not output a bounding box.\nInstruction: Mark dimensions\nEnd your reply with the final answer on its own last line, formatted exactly as: Answer: [x, y]"
        }
      ]
    }
  ],
  "target": "[0.1672, 0.0435, 0.1802, 0.1019]",
  "id": 0,
  "group_id": 0,
  "subset_key": "CAD",
  "metadata": {
    "id": "inventor_windows_0",
    "sent_size": [
      3840,
      1080
    ],
    "bbox_norm": [
      0.1671875,
      0.04351851851851852,
      0.18020833333333333,
      0.10185185185185185
    ],
    "ui_type": "text",
    "application": "inventor",
    "platform": "windows"
  }
}

Prompt Template

No prompt template defined.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets screenspot_pro \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['screenspot_pro'],
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)