HPD-v2

Overview

HPD-v2 (Human Preference Dataset v2) is a text-to-image benchmark that evaluates generated images based on human preferences. It uses the HPSv2.1 score metric trained on large-scale human preference data.

Task Description

Task Type: Text-to-Image Generation Evaluation
Input: Text prompt for image generation
Output: Generated image evaluated against human preferences
Metric: HPSv2.1 Score

Key Features

Human preference-aligned evaluation metric
Trained on large-scale human preference data
Tests aesthetic quality and prompt alignment
Supports diverse prompt categories
Objective, reproducible scoring

Evaluation Notes

Default configuration uses 0-shot evaluation
HPSv2.1 Score metric measures human preference alignment
Supports local prompt files
Category tags available in metadata
Can evaluate existing images or generate new ones

Properties

Property	Value
Benchmark Name	`hpdv2`
Dataset ID	AI-ModelScope/T2V-Eval-Prompts
Paper	N/A
Tags	`TextToImage`
Metrics	`HPSv2.1Score`
Default Shots	0-shot
Evaluation Split	`test`

Data Statistics

Metric	Value
Total Samples	3,200
Prompt Length (Mean)	81.71 chars
Prompt Length (Min/Max)	9 / 404 chars

Sample Example

Subset: HPDv2

{
  "input": [
    {
      "id": "c971b3c7",
      "content": "Spongebob depicted in the style of Dragon Ball Z."
    }
  ],
  "id": 0,
  "group_id": 0,
  "metadata": {
    "id": "HPDv2_0",
    "prompt": "Spongebob depicted in the style of Dragon Ball Z.",
    "category": "Animation",
    "tags": {
      "category": "Animation"
    },
    "image_path": ""
  }
}

Prompt Template

No prompt template defined.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets hpdv2 \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['hpdv2'],
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

2.4 KiB Raw Permalink Blame History