165 lines
4.5 KiB
Markdown
165 lines
4.5 KiB
Markdown
# RefCOCO
|
|
|
|
|
|
## Overview
|
|
|
|
RefCOCO is a dataset for training and evaluating models on Referring Expression Comprehension (REC). It contains images, object bounding boxes, and free-form natural-language expressions that uniquely describe target objects within MSCOCO images.
|
|
|
|
## Task Description
|
|
|
|
- **Task Type**: Referring Expression Comprehension / Image Captioning
|
|
- **Input**: Image (with visualization) + referring expression
|
|
- **Output**: Bounding box coordinates or caption
|
|
- **Domains**: Visual grounding, object localization, image understanding
|
|
|
|
## Key Features
|
|
|
|
- Created via Amazon Mechanical Turk annotations
|
|
- Three evaluation modes:
|
|
- `bbox`: Image captioning with bounding box visualization
|
|
- `seg`: Image captioning with segmentation visualization
|
|
- `bbox_rec`: Grounding task - output normalized bounding box coordinates
|
|
- Expressions uniquely identify target objects in complex scenes
|
|
- Multiple subsets: test, val, testA, testB
|
|
|
|
## Evaluation Notes
|
|
|
|
- Evaluation mode configurable via `eval_mode` parameter
|
|
- Multiple metrics for comprehensive evaluation:
|
|
- Grounding: IoU, ACC@0.1/0.3/0.5/0.7/0.9, Center_ACC
|
|
- Captioning: BLEU (1-4), METEOR, ROUGE_L, CIDEr
|
|
- Bounding boxes output as normalized coordinates [x1/W, y1/H, x2/W, y2/H]
|
|
- Requires pycocoevalcap for caption metrics
|
|
|
|
|
|
## Properties
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **Benchmark Name** | `refcoco` |
|
|
| **Dataset ID** | [lmms-lab/RefCOCO](https://modelscope.cn/datasets/lmms-lab/RefCOCO/summary) |
|
|
| **Paper** | N/A |
|
|
| **Tags** | `Grounding`, `ImageCaptioning`, `Knowledge`, `MultiModal` |
|
|
| **Metrics** | `IoU`, `ACC@0.1`, `ACC@0.3`, `ACC@0.5`, `ACC@0.7`, `ACC@0.9`, `Center_ACC`, `Bleu_1`, `Bleu_2`, `Bleu_3`, `Bleu_4`, `METEOR`, `ROUGE_L`, `CIDEr` |
|
|
| **Default Shots** | 0-shot |
|
|
| **Evaluation Split** | `N/A` |
|
|
|
|
|
|
## Data Statistics
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Total Samples | 17,596 |
|
|
| Prompt Length (Mean) | 146 chars |
|
|
| Prompt Length (Min/Max) | 146 / 146 chars |
|
|
|
|
**Per-Subset Statistics:**
|
|
|
|
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|
|
|--------|---------|-------------|------------|------------|
|
|
| `test` | 5,000 | 146 | 146 | 146 |
|
|
| `val` | 8,811 | 146 | 146 | 146 |
|
|
| `testA` | 1,975 | 146 | 146 | 146 |
|
|
| `testB` | 1,810 | 146 | 146 | 146 |
|
|
|
|
**Image Statistics:**
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Total Images | 13,785 |
|
|
| Images per Sample | min: 1, max: 1, mean: 1 |
|
|
| Resolution Range | 300x176 - 640x640 |
|
|
| Formats | jpeg |
|
|
|
|
|
|
## Sample Example
|
|
|
|
**Subset**: `test`
|
|
|
|
```json
|
|
{
|
|
"input": [
|
|
{
|
|
"id": "53a494fc",
|
|
"content": [
|
|
{
|
|
"text": "Please carefully observe the area circled in the image and come up with a caption for the area.\nAnswer the question using a single word or phrase."
|
|
},
|
|
{
|
|
"image": "[BASE64_IMAGE: jpeg, ~57.6KB]"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"target": "['guy petting elephant', 'foremost person', 'green shirt']",
|
|
"id": 0,
|
|
"group_id": 0,
|
|
"metadata": {
|
|
"question_id": "469306",
|
|
"iscrowd": 0,
|
|
"file_name": "COCO_train2014_000000296747_0.jpg",
|
|
"answer": [
|
|
"guy petting elephant",
|
|
"foremost person",
|
|
"green shirt"
|
|
],
|
|
"original_bbox": [
|
|
59.04999923706055,
|
|
93.23999786376953,
|
|
375.0199890136719,
|
|
362.5799865722656
|
|
],
|
|
"bbox": [],
|
|
"eval_mode": "bbox"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Prompt Template
|
|
|
|
*No prompt template defined.*
|
|
|
|
## Extra Parameters
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `eval_mode` | `str` | `bbox` | Control the evaluation mode used by RefCOCO. bbox: image caption task, visualize the original image with bounding box; seg: image caption task, visualize the original image with segmentation; bbox_rec: grounding task, recognize bounding box coordinates. Choices: ['bbox', 'seg', 'bbox_rec'] |
|
|
|
|
## Usage
|
|
|
|
### Using CLI
|
|
|
|
```bash
|
|
evalscope eval \
|
|
--model YOUR_MODEL \
|
|
--api-url OPENAI_API_COMPAT_URL \
|
|
--api-key EMPTY_TOKEN \
|
|
--datasets refcoco \
|
|
--limit 10 # Remove this line for formal evaluation
|
|
```
|
|
|
|
### Using Python
|
|
|
|
```python
|
|
from evalscope import run_task
|
|
from evalscope.config import TaskConfig
|
|
|
|
task_cfg = TaskConfig(
|
|
model='YOUR_MODEL',
|
|
api_url='OPENAI_API_COMPAT_URL',
|
|
api_key='EMPTY_TOKEN',
|
|
datasets=['refcoco'],
|
|
dataset_args={
|
|
'refcoco': {
|
|
# subset_list: ['test', 'val', 'testA'] # optional, evaluate specific subsets
|
|
# extra_params: {} # uses default extra parameters
|
|
}
|
|
},
|
|
limit=10, # Remove this line for formal evaluation
|
|
)
|
|
|
|
run_task(task_cfg=task_cfg)
|
|
```
|
|
|
|
|