{
  "meta": {
    "pretty_name": "EvalMuse",
    "dataset_id": "AI-ModelScope/T2V-Eval-Prompts",
    "paper_url": null,
    "tags": [
      "TextToImage"
    ],
    "metrics": [
      "FGA_BLIP2Score"
    ],
    "few_shot_num": 0,
    "eval_split": "test",
    "train_split": "",
    "subset_list": [
      "EvalMuse"
    ],
    "description": "\n## Overview\n\nEvalMuse is a text-to-image benchmark that evaluates the quality and semantic alignment of generated images using fine-grained analysis with the FGA-BLIP2Score metric.\n\n## Task Description\n\n- **Task Type**: Text-to-Image Generation Evaluation\n- **Input**: Text prompt for image generation\n- **Output**: Generated image evaluated for quality and semantic fidelity\n- **Metric**: FGA-BLIP2Score (Fine-Grained Analysis with BLIP-2)\n\n## Key Features\n\n- Fine-grained semantic alignment evaluation\n- Uses BLIP-2 vision-language model for scoring\n- Evaluates both image quality and prompt adherence\n- Supports diverse prompt categories\n- Objective, reproducible metrics\n\n## Evaluation Notes\n\n- Default configuration uses **0-shot** evaluation\n- Only **FGA_BLIP2Score** metric is supported\n- Evaluates images from the **test** split\n- Can evaluate pre-generated images or generate new ones\n",
    "prompt_template": "",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {},
    "sandbox_config": {},
    "category": "aigc"
  },
  "statistics": {
    "total_samples": 199,
    "subset_stats": [
      {
        "name": "EvalMuse",
        "sample_count": 199,
        "prompt_length_mean": 61.05,
        "prompt_length_min": 8,
        "prompt_length_max": 347,
        "prompt_length_std": 46.36,
        "target_length_mean": null
      }
    ],
    "prompt_length": {
      "mean": 61.05,
      "min": 8,
      "max": 347,
      "std": 46.36
    },
    "target_length_mean": null,
    "computed_at": "2026-01-28T11:17:03.739115"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "f8b508f4",
          "content": "wide angle photograph at night, award winning interior design apartment, night outside, dark, moody dim faint lighting, wood panel walls, cozy and calm, fabrics, textiles, pillows, lamps, colorful copper brass accents, secluded, many light sources, lamps, hardwood floors, book shelf, couch, desk, plants"
        }
      ],
      "id": 0,
      "group_id": 0,
      "metadata": {
        "prompt": "wide angle photograph at night, award winning interior design apartment, night outside, dark, moody dim faint lighting, wood panel walls, cozy and calm, fabrics, textiles, pillows, lamps, colorful copper brass accents, secluded, many light sources, lamps, hardwood floors, book shelf, couch, desk, plants",
        "category": "",
        "tags": [
          "photograph (activity)",
          "night (attribute)",
          "interior design apartment (location)",
          "outside (location)",
          "dark (attribute)",
          "moody dim faint lighting (attribute)",
          "wood panel walls (object)",
          "cozy and calm (attribute)",
          "fabrics (material)",
          "textiles (material)",
          "pillows (object)",
          "lamps (object)",
          "colorful (attribute)",
          "copper brass accents (material)",
          "secluded (attribute)",
          "many light sources (attribute)",
          "hardwood floors (material)",
          "book shelf (object)",
          "couch (object)",
          "desk (object)",
          "plants (object)",
          "wide angle (attribute)"
        ],
        "id": "EvalMuse_0",
        "image_path": ""
      }
    },
    "subset": "EvalMuse",
    "truncated": false
  },
  "readme": {
    "en": "# EvalMuse\n\n\n## Overview\n\nEvalMuse is a text-to-image benchmark that evaluates the quality and semantic alignment of generated images using fine-grained analysis with the FGA-BLIP2Score metric.\n\n## Task Description\n\n- **Task Type**: Text-to-Image Generation Evaluation\n- **Input**: Text prompt for image generation\n- **Output**: Generated image evaluated for quality and semantic fidelity\n- **Metric**: FGA-BLIP2Score (Fine-Grained Analysis with BLIP-2)\n\n## Key Features\n\n- Fine-grained semantic alignment evaluation\n- Uses BLIP-2 vision-language model for scoring\n- Evaluates both image quality and prompt adherence\n- Supports diverse prompt categories\n- Objective, reproducible metrics\n\n## Evaluation Notes\n\n- Default configuration uses **0-shot** evaluation\n- Only **FGA_BLIP2Score** metric is supported\n- Evaluates images from the **test** split\n- Can evaluate pre-generated images or generate new ones\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `evalmuse` |\n| **Dataset ID** | [AI-ModelScope/T2V-Eval-Prompts](https://modelscope.cn/datasets/AI-ModelScope/T2V-Eval-Prompts/summary) |\n| **Paper** | N/A |\n| **Tags** | `TextToImage` |\n| **Metrics** | `FGA_BLIP2Score` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 199 |\n| Prompt Length (Mean) | 61.05 chars |\n| Prompt Length (Min/Max) | 8 / 347 chars |\n\n## Sample Example\n\n**Subset**: `EvalMuse`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"f8b508f4\",\n      \"content\": \"wide angle photograph at night, award winning interior design apartment, night outside, dark, moody dim faint lighting, wood panel walls, cozy and calm, fabrics, textiles, pillows, lamps, colorful copper brass accents, secluded, many light sources, lamps, hardwood floors, book shelf, couch, desk, plants\"\n    }\n  ],\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"prompt\": \"wide angle photograph at night, award winning interior design apartment, night outside, dark, moody dim faint lighting, wood panel walls, cozy and calm, fabrics, textiles, pillows, lamps, colorful copper brass accents, secluded, many light sources, lamps, hardwood floors, book shelf, couch, desk, plants\",\n    \"category\": \"\",\n    \"tags\": [\n      \"photograph (activity)\",\n      \"night (attribute)\",\n      \"interior design apartment (location)\",\n      \"outside (location)\",\n      \"dark (attribute)\",\n      \"moody dim faint lighting (attribute)\",\n      \"wood panel walls (object)\",\n      \"cozy and calm (attribute)\",\n      \"fabrics (material)\",\n      \"textiles (material)\",\n      \"pillows (object)\",\n      \"lamps (object)\",\n      \"colorful (attribute)\",\n      \"copper brass accents (material)\",\n      \"secluded (attribute)\",\n      \"many light sources (attribute)\",\n      \"hardwood floors (material)\",\n      \"book shelf (object)\",\n      \"couch (object)\",\n      \"desk (object)\",\n      \"plants (object)\",\n      \"wide angle (attribute)\"\n    ],\n    \"id\": \"EvalMuse_0\",\n    \"image_path\": \"\"\n  }\n}\n```\n\n## Prompt Template\n\n*No prompt template defined.*\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets evalmuse \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['evalmuse'],\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# EvalMuse\n\n\n## 概述\n\nEvalMuse 是一个文本到图像的基准测试，使用 FGA-BLIP2Score 指标进行细粒度分析，以评估生成图像的质量和语义对齐程度。\n\n## 任务描述\n\n- **任务类型**：文本到图像生成评估\n- **输入**：用于图像生成的文本提示\n- **输出**：评估生成图像的质量和语义保真度\n- **指标**：FGA-BLIP2Score（基于 BLIP-2 的细粒度分析）\n\n## 主要特性\n\n- 细粒度语义对齐评估\n- 使用 BLIP-2 视觉-语言模型进行评分\n- 同时评估图像质量和对提示的遵循程度\n- 支持多样化的提示类别\n- 提供客观、可复现的指标\n\n## 评估说明\n\n- 默认配置使用 **0-shot** 评估\n- 仅支持 **FGA_BLIP2Score** 指标\n- 评估 **test** 分割中的图像\n- 可评估预生成的图像或生成新图像\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `evalmuse` |\n| **数据集ID** | [AI-ModelScope/T2V-Eval-Prompts](https://modelscope.cn/datasets/AI-ModelScope/T2V-Eval-Prompts/summary) |\n| **论文** | N/A |\n| **标签** | `TextToImage` |\n| **指标** | `FGA_BLIP2Score` |\n| **默认Shots** | 0-shot |\n| **评估分割** | `test` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 199 |\n| 提示词长度（平均） | 61.05 字符 |\n| 提示词长度（最小/最大） | 8 / 347 字符 |\n\n## 样例示例\n\n**子集**：`EvalMuse`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"f8b508f4\",\n      \"content\": \"wide angle photograph at night, award winning interior design apartment, night outside, dark, moody dim faint lighting, wood panel walls, cozy and calm, fabrics, textiles, pillows, lamps, colorful copper brass accents, secluded, many light sources, lamps, hardwood floors, book shelf, couch, desk, plants\"\n    }\n  ],\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"prompt\": \"wide angle photograph at night, award winning interior design apartment, night outside, dark, moody dim faint lighting, wood panel walls, cozy and calm, fabrics, textiles, pillows, lamps, colorful copper brass accents, secluded, many light sources, lamps, hardwood floors, book shelf, couch, desk, plants\",\n    \"category\": \"\",\n    \"tags\": [\n      \"photograph (activity)\",\n      \"night (attribute)\",\n      \"interior design apartment (location)\",\n      \"outside (location)\",\n      \"dark (attribute)\",\n      \"moody dim faint lighting (attribute)\",\n      \"wood panel walls (object)\",\n      \"cozy and calm (attribute)\",\n      \"fabrics (material)\",\n      \"textiles (material)\",\n      \"pillows (object)\",\n      \"lamps (object)\",\n      \"colorful (attribute)\",\n      \"copper brass accents (material)\",\n      \"secluded (attribute)\",\n      \"many light sources (attribute)\",\n      \"hardwood floors (material)\",\n      \"book shelf (object)\",\n      \"couch (object)\",\n      \"desk (object)\",\n      \"plants (object)\",\n      \"wide angle (attribute)\"\n    ],\n    \"id\": \"EvalMuse_0\",\n    \"image_path\": \"\"\n  }\n}\n```\n\n## 提示模板\n\n*未定义提示模板。*\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets evalmuse \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['evalmuse'],\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "6e7107f190a2d151250757254442a2a4",
    "needs_translation": false
  },
  "updated_at": "2026-01-28T17:31:32.506303",
  "translation_updated_at": "2026-01-28T16:09:53Z"
}