HallusionBench

Overview

HallusionBench is an advanced diagnostic benchmark designed to evaluate image-context reasoning and detect hallucination tendencies in Large Vision-Language Models (LVLMs). It specifically tests models' susceptibility to language hallucination and visual illusion.

Task Description

Task Type: Hallucination Detection and Visual Reasoning
Input: Image + yes/no question about image content
Output: YES or NO answer
Domains: Hallucination detection, visual reasoning, factual accuracy

Key Features

Specifically designed to probe hallucination behaviors
Tests both language hallucination and visual illusion
Organized by categories and subcategories for detailed analysis
Uses grouped accuracy metrics for robust evaluation
Questions require precise image-context reasoning

Evaluation Notes

Default evaluation uses the image split
Multiple accuracy metrics:
- aAcc: Answer-level accuracy (per-question)
- fAcc: Figure-level accuracy (all questions per figure correct)
- qAcc: Question-level accuracy (grouped by question type)
Requires simple YES/NO answers without explanation
Aggregation at subcategory, category, and overall levels

Properties

Property	Value
Benchmark Name	`hallusion_bench`
Dataset ID	lmms-lab/HallusionBench
Paper	N/A
Tags	`Hallucination`, `MultiModal`, `Yes/No`
Metrics	`aAcc`, `qAcc`, `fAcc`
Default Shots	0-shot
Evaluation Split	`image`
Aggregation	`f1`

Data Statistics

Metric	Value
Total Samples	951
Prompt Length (Mean)	136.78 chars
Prompt Length (Min/Max)	76 / 292 chars

Image Statistics:

Metric	Value
Total Images	951
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	388x56 - 5291x4536
Formats	png

Sample Example

Subset: default

{
  "input": [
    {
      "id": "ba75d669",
      "content": [
        {
          "text": "Is China, Hongkong SAR, the leading importing country of gold, silverware, and jewelry with the highest import value in 2018?\nPlease answer YES or NO without an explanation."
        },
        {
          "image": "[BASE64_IMAGE: png, ~143.0KB]"
        }
      ]
    }
  ],
  "target": "NO",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "category": "VS",
    "subcategory": "chart",
    "visual_input": "1",
    "set_id": "0",
    "figure_id": "1",
    "question_id": "0",
    "gt_answer": "0",
    "gt_answer_details": "Switzerland is the leading importing country of gold, silverware, and jewelry with the highest import value in 2018?"
  }
}

Prompt Template

Prompt Template:

{question}
Please answer YES or NO without an explanation.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets hallusion_bench \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['hallusion_bench'],
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

3.4 KiB Raw Blame History