RefCOCO

Overview

RefCOCO is a dataset for training and evaluating models on Referring Expression Comprehension (REC). It contains images, object bounding boxes, and free-form natural-language expressions that uniquely describe target objects within MSCOCO images.

Task Description

Task Type: Referring Expression Comprehension / Image Captioning
Input: Image (with visualization) + referring expression
Output: Bounding box coordinates or caption
Domains: Visual grounding, object localization, image understanding

Key Features

Created via Amazon Mechanical Turk annotations
Three evaluation modes:
- bbox: Image captioning with bounding box visualization
- seg: Image captioning with segmentation visualization
- bbox_rec: Grounding task - output normalized bounding box coordinates
Expressions uniquely identify target objects in complex scenes
Multiple subsets: test, val, testA, testB

Evaluation Notes

Evaluation mode configurable via eval_mode parameter
Multiple metrics for comprehensive evaluation:
- Grounding: IoU, ACC@0.1/0.3/0.5/0.7/0.9, Center_ACC
- Captioning: BLEU (1-4), METEOR, ROUGE_L, CIDEr
Bounding boxes output as normalized coordinates [x1/W, y1/H, x2/W, y2/H]
Requires pycocoevalcap for caption metrics

Properties

Property	Value
Benchmark Name	`refcoco`
Dataset ID	lmms-lab/RefCOCO
Paper	N/A
Tags	`Grounding`, `ImageCaptioning`, `Knowledge`, `MultiModal`
Metrics	`IoU`, `ACC@0.1`, `ACC@0.3`, `ACC@0.5`, `ACC@0.7`, `ACC@0.9`, `Center_ACC`, `Bleu_1`, `Bleu_2`, `Bleu_3`, `Bleu_4`, `METEOR`, `ROUGE_L`, `CIDEr`
Default Shots	0-shot
Evaluation Split	`N/A`

Data Statistics

Metric	Value
Total Samples	17,596
Prompt Length (Mean)	146 chars
Prompt Length (Min/Max)	146 / 146 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
`test`	5,000	146	146	146
`val`	8,811	146	146	146
`testA`	1,975	146	146	146
`testB`	1,810	146	146	146

Image Statistics:

Metric	Value
Total Images	13,785
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	300x176 - 640x640
Formats	jpeg

Sample Example

Subset: test

{
  "input": [
    {
      "id": "53a494fc",
      "content": [
        {
          "text": "Please carefully observe the area circled in the image and come up with a caption for the area.\nAnswer the question using a single word or phrase."
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~57.6KB]"
        }
      ]
    }
  ],
  "target": "['guy petting elephant', 'foremost person', 'green shirt']",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "question_id": "469306",
    "iscrowd": 0,
    "file_name": "COCO_train2014_000000296747_0.jpg",
    "answer": [
      "guy petting elephant",
      "foremost person",
      "green shirt"
    ],
    "original_bbox": [
      59.04999923706055,
      93.23999786376953,
      375.0199890136719,
      362.5799865722656
    ],
    "bbox": [],
    "eval_mode": "bbox"
  }
}

Prompt Template

No prompt template defined.

Extra Parameters

Parameter	Type	Default	Description
`eval_mode`	`str`	`bbox`	Control the evaluation mode used by RefCOCO. bbox: image caption task, visualize the original image with bounding box; seg: image caption task, visualize the original image with segmentation; bbox_rec: grounding task, recognize bounding box coordinates. Choices: ['bbox', 'seg', 'bbox_rec']

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets refcoco \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['refcoco'],
    dataset_args={
        'refcoco': {
            # subset_list: ['test', 'val', 'testA']  # optional, evaluate specific subsets
            # extra_params: {}  # uses default extra parameters
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

4.5 KiB Raw Permalink Blame History