POPE

Overview

POPE (Polling-based Object Probing Evaluation) is a benchmark specifically designed to evaluate object hallucination in Large Vision-Language Models (LVLMs). It tests models' ability to accurately identify objects present in images through yes/no questions.

Task Description

Task Type: Object Hallucination Detection (Yes/No Q&A)
Input: Image with question "Is there a [object] in the image?"
Output: YES or NO answer
Focus: Measuring accuracy vs. hallucination rate

Key Features

Three sampling strategies: random, popular, adversarial
Tests for false positive object claims (hallucination)
Based on MSCOCO images
Simple yes/no question format for objective evaluation
Measures alignment between model responses and visual content

Evaluation Notes

Default configuration uses 0-shot evaluation
Five metrics: accuracy, precision, recall, F1 score, yes_ratio
F1 score is the primary aggregation metric
Three subsets: popular, adversarial, random
"Popular" and "adversarial" subsets are more challenging
yes_ratio indicates model's tendency to answer "yes"

Properties

Property	Value
Benchmark Name	`pope`
Dataset ID	lmms-lab/POPE
Paper	N/A
Tags	`Hallucination`, `MultiModal`, `Yes/No`
Metrics	`accuracy`, `precision`, `recall`, `f1_score`, `yes_ratio`
Default Shots	0-shot
Evaluation Split	`N/A`
Aggregation	`f1`

Data Statistics

Metric	Value
Total Samples	9,000
Prompt Length (Mean)	79.4 chars
Prompt Length (Min/Max)	75 / 87 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
`popular`	3,000	79.27	75	87
`adversarial`	3,000	79.36	75	87
`random`	3,000	79.59	75	87

Image Statistics:

Metric	Value
Total Images	9,000
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	500x243 - 640x640
Formats	jpeg

Sample Example

Subset: popular

{
  "input": [
    {
      "id": "8847a5a3",
      "content": [
        {
          "text": "Is there a snowboard in the image?\nPlease answer YES or NO without an explanation."
        },
        {
          "image": "[BASE64_IMAGE: png, ~87.2KB]"
        }
      ]
    }
  ],
  "target": "YES",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "id": "3000",
    "answer": "YES",
    "category": "popular",
    "question_id": "1"
  }
}

Prompt Template

Prompt Template:

{question}
Please answer YES or NO without an explanation.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets pope \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['pope'],
    dataset_args={
        'pope': {
            # subset_list: ['popular', 'adversarial', 'random']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

3.4 KiB Raw Blame History