General-T2I

Overview

General Text-to-Image is a customizable benchmark adapter for evaluating text-to-image generation models with user-provided prompts and images.

Task Description

Task Type: Custom Text-to-Image Evaluation
Input: User-provided text prompts
Output: Generated images evaluated using configurable metrics
Flexibility: Supports local prompt files

Key Features

Flexible custom evaluation framework
Supports local prompt file loading
Configurable metrics (default: PickScore)
User-defined subset naming from file path
Compatible with pre-generated images

Evaluation Notes

Default configuration uses 0-shot evaluation
Primary metric: PickScore (configurable)
Supports custom dataset paths and prompt files
Automatically extracts subset name from file path
Can evaluate existing images via image_path field

Properties

Property	Value
Benchmark Name	`general_t2i`
Dataset ID	`general_t2i`
Paper	N/A
Tags	`Custom`, `TextToImage`
Metrics	`PickScore`
Default Shots	0-shot
Evaluation Split	`test`

Data Statistics

Statistics not available.

Sample Example

Sample example not available.

Prompt Template

No prompt template defined.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets general_t2i \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['general_t2i'],
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

1.9 KiB Raw Blame History