{
  "meta": {
    "pretty_name": "GAIA",
    "dataset_id": "gaia-benchmark/GAIA",
    "paper_url": null,
    "tags": [
      "Agent",
      "Reasoning",
      "MultiTurn"
    ],
    "metrics": [
      "acc"
    ],
    "few_shot_num": 0,
    "eval_split": "validation",
    "train_split": "",
    "subset_list": [
      "2023_level1",
      "2023_level2",
      "2023_level3"
    ],
    "description": "\n## Overview\n\nGAIA (General AI Assistants) is a benchmark of 450+ questions targeting next-generation LLMs with tool use, web browsing and multi-step reasoning. Each question has an unambiguous short answer and is bucketed into one of three difficulty levels.\n\n## Task Description\n\n- **Task Type**: Tool-use Agent (multi-turn)\n- **Input**: Natural-language question, optionally with one referenced attachment file (PDF / xlsx / image / audio / ...)\n- **Output**: A short final answer (number / short phrase / comma-separated list)\n- **Splits**: ``validation`` (answers public) and ``test`` (answers private — evaluation skipped)\n\n## Key Features\n\n- ReAct agent loop with a single ``bash`` tool inside a Docker sandbox (default image ``python:3.11``, includes ``curl`` / ``wget`` / ``git``).\n- Attachment files are mounted read-only at ``/shared_files`` inside the sandbox.\n- Rule-based scorer ported verbatim from the official GAIA leaderboard (no LLM judge).\n- Dataset downloaded from ModelScope (``gaia-benchmark/GAIA``) by default; set ``dataset_hub='huggingface'`` to load from Hugging Face instead.\n\n## Evaluation Notes\n\n- Requires Docker daemon running locally (or a remote sandbox engine via the ms_enclave configuration).\n- ``extra_params.max_steps`` caps the agent loop length (default 50).\n- ``extra_params.command_timeout`` sets per-``bash`` command timeout (default 180s, mirrors inspect_ai).\n- Network is enabled by default — many questions require browsing/searching.\n- Use ``subset_list`` to restrict to specific difficulty levels, e.g. ``['2023_level1']``, ``['2023_level1', '2023_level2']`` or ``['2023_all']`` (default).\n- [Usage Documentation](https://evalscope.readthedocs.io/en/latest/third_party/gaia.html)\n",
    "prompt_template": "{question}",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {
      "max_steps": {
        "type": "int",
        "description": "Maximum number of agent steps per sample.",
        "value": 50
      },
      "command_timeout": {
        "type": "float",
        "description": "Default per-bash-command timeout in seconds.",
        "value": 180.0
      },
      "docker_image": {
        "type": "str",
        "description": "Docker image used as the per-sample sandbox.",
        "value": "python:3.11"
      },
      "network_enabled": {
        "type": "bool",
        "description": "Allow the sandbox to access the network (GAIA browsing questions need this).",
        "value": true
      }
    },
    "sandbox_config": {},
    "category": "agent"
  },
  "statistics": {
    "total_samples": 165,
    "subset_stats": [
      {
        "name": "2023_level1",
        "sample_count": 53,
        "prompt_length_mean": 906.53,
        "prompt_length_min": 604,
        "prompt_length_max": 2582,
        "prompt_length_std": 383.72,
        "target_length_mean": 14.68
      },
      {
        "name": "2023_level2",
        "sample_count": 86,
        "prompt_length_mean": 816.66,
        "prompt_length_min": 596,
        "prompt_length_max": 1275,
        "prompt_length_std": 158.69,
        "target_length_mean": 8.81
      },
      {
        "name": "2023_level3",
        "sample_count": 26,
        "prompt_length_mean": 917.08,
        "prompt_length_min": 621,
        "prompt_length_max": 1497,
        "prompt_length_std": 188.76,
        "target_length_mean": 11.46
      }
    ],
    "prompt_length": {
      "mean": 861.35,
      "min": 596,
      "max": 2582,
      "std": 259.55
    },
    "target_length_mean": 11.12,
    "computed_at": "2026-05-26T13:02:15.194551"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "93e8257c",
          "content": "Please answer the question below. You should:\n\n- Return only your answer, which should be a number, or a short phrase with as few words as possible, or a comma separated list of numbers and/or strings.\n- If the answer is a number, return only ... [TRUNCATED 431 chars] ... Earth and the Moon its closest approach? Please use the minimum perigee value on the Wikipedia page for the Moon when carrying out your calculation. Round your result to the nearest 1000 hours and do not use any comma separators if necessary."
        }
      ],
      "target": "17",
      "id": 0,
      "group_id": 0,
      "tools": [
        {
          "name": "bash",
          "description": "Execute a bash command inside the sandbox environment. Returns the combined stdout / stderr output of the command.",
          "parameters": {
            "properties": {
              "command": {
                "type": "string",
                "description": "The bash command to execute."
              },
              "timeout": {
                "type": "number",
                "description": "Maximum execution time in seconds (default: 60).",
                "default": 60
              }
            },
            "required": [
              "command"
            ]
          }
        }
      ],
      "metadata": {
        "task_id": "e1fc63a2-da7a-432f-be78-7c4a95598703",
        "level": "1",
        "file_name": "",
        "file_path": "",
        "Annotator Metadata": {
          "Steps": "1. Googled Eliud Kipchoge marathon pace to find 4min 37sec/mile\n2. Converted into fractions of hours.\n3. Found moon periapsis in miles (225,623 miles).\n4. Multiplied the two to find the number of hours and rounded to the nearest 100 hours.",
          "Number of steps": "4",
          "How long did this take?": "20 Minutes",
          "Tools": "1. A web browser.\n2. A search engine.\n3. A calculator.",
          "Number of tools": "3"
        }
      }
    },
    "subset": "2023_level1",
    "truncated": false
  },
  "readme": {
    "en": "# GAIA\n\n\n## Overview\n\nGAIA (General AI Assistants) is a benchmark of 450+ questions targeting next-generation LLMs with tool use, web browsing and multi-step reasoning. Each question has an unambiguous short answer and is bucketed into one of three difficulty levels.\n\n## Task Description\n\n- **Task Type**: Tool-use Agent (multi-turn)\n- **Input**: Natural-language question, optionally with one referenced attachment file (PDF / xlsx / image / audio / ...)\n- **Output**: A short final answer (number / short phrase / comma-separated list)\n- **Splits**: ``validation`` (answers public) and ``test`` (answers private — evaluation skipped)\n\n## Key Features\n\n- ReAct agent loop with a single ``bash`` tool inside a Docker sandbox (default image ``python:3.11``, includes ``curl`` / ``wget`` / ``git``).\n- Attachment files are mounted read-only at ``/shared_files`` inside the sandbox.\n- Rule-based scorer ported verbatim from the official GAIA leaderboard (no LLM judge).\n- Dataset downloaded from ModelScope (``gaia-benchmark/GAIA``) by default; set ``dataset_hub='huggingface'`` to load from Hugging Face instead.\n\n## Evaluation Notes\n\n- Requires Docker daemon running locally (or a remote sandbox engine via the ms_enclave configuration).\n- ``extra_params.max_steps`` caps the agent loop length (default 50).\n- ``extra_params.command_timeout`` sets per-``bash`` command timeout (default 180s, mirrors inspect_ai).\n- Network is enabled by default — many questions require browsing/searching.\n- Use ``subset_list`` to restrict to specific difficulty levels, e.g. ``['2023_level1']``, ``['2023_level1', '2023_level2']`` or ``['2023_all']`` (default).\n- [Usage Documentation](https://evalscope.readthedocs.io/en/latest/third_party/gaia.html)\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `gaia` |\n| **Dataset ID** | [gaia-benchmark/GAIA](https://modelscope.cn/datasets/gaia-benchmark/GAIA/summary) |\n| **Paper** | N/A |\n| **Tags** | `Agent`, `MultiTurn`, `Reasoning` |\n| **Metrics** | `acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `validation` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 165 |\n| Prompt Length (Mean) | 861.35 chars |\n| Prompt Length (Min/Max) | 596 / 2582 chars |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `2023_level1` | 53 | 906.53 | 604 | 2582 |\n| `2023_level2` | 86 | 816.66 | 596 | 1275 |\n| `2023_level3` | 26 | 917.08 | 621 | 1497 |\n\n## Sample Example\n\n**Subset**: `2023_level1`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"93e8257c\",\n      \"content\": \"Please answer the question below. You should:\\n\\n- Return only your answer, which should be a number, or a short phrase with as few words as possible, or a comma separated list of numbers and/or strings.\\n- If the answer is a number, return only ... [TRUNCATED 431 chars] ... Earth and the Moon its closest approach? Please use the minimum perigee value on the Wikipedia page for the Moon when carrying out your calculation. Round your result to the nearest 1000 hours and do not use any comma separators if necessary.\"\n    }\n  ],\n  \"target\": \"17\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"tools\": [\n    {\n      \"name\": \"bash\",\n      \"description\": \"Execute a bash command inside the sandbox environment. Returns the combined stdout / stderr output of the command.\",\n      \"parameters\": {\n        \"properties\": {\n          \"command\": {\n            \"type\": \"string\",\n            \"description\": \"The bash command to execute.\"\n          },\n          \"timeout\": {\n            \"type\": \"number\",\n            \"description\": \"Maximum execution time in seconds (default: 60).\",\n            \"default\": 60\n          }\n        },\n        \"required\": [\n          \"command\"\n        ]\n      }\n    }\n  ],\n  \"metadata\": {\n    \"task_id\": \"e1fc63a2-da7a-432f-be78-7c4a95598703\",\n    \"level\": \"1\",\n    \"file_name\": \"\",\n    \"file_path\": \"\",\n    \"Annotator Metadata\": {\n      \"Steps\": \"1. Googled Eliud Kipchoge marathon pace to find 4min 37sec/mile\\n2. Converted into fractions of hours.\\n3. Found moon periapsis in miles (225,623 miles).\\n4. Multiplied the two to find the number of hours and rounded to the nearest 100 hours.\",\n      \"Number of steps\": \"4\",\n      \"How long did this take?\": \"20 Minutes\",\n      \"Tools\": \"1. A web browser.\\n2. A search engine.\\n3. A calculator.\",\n      \"Number of tools\": \"3\"\n    }\n  }\n}\n```\n\n## Prompt Template\n\n**Prompt Template:**\n```text\n{question}\n```\n\n## Extra Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `max_steps` | `int` | `50` | Maximum number of agent steps per sample. |\n| `command_timeout` | `float` | `180.0` | Default per-bash-command timeout in seconds. |\n| `docker_image` | `str` | `python:3.11` | Docker image used as the per-sample sandbox. |\n| `network_enabled` | `bool` | `True` | Allow the sandbox to access the network (GAIA browsing questions need this). |\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets gaia \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['gaia'],\n    dataset_args={\n        'gaia': {\n            # subset_list: ['2023_level1', '2023_level2', '2023_level3']  # optional, evaluate specific subsets\n            # extra_params: {}  # uses default extra parameters\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# GAIA\n\n\n## 概述\n\nGAIA（General AI Assistants）是一个包含 450 多个问题的基准测试，旨在评估具备工具使用、网页浏览和多步推理能力的新一代大语言模型（LLM）。每个问题都有一个明确的简短答案，并被划分为三个难度等级之一。\n\n## 任务描述\n\n- **任务类型**：工具使用型智能体（多轮交互）\n- **输入**：自然语言问题，可选附带一个引用附件文件（PDF / xlsx / 图像 / 音频 / ...）\n- **输出**：一个简短的最终答案（数字 / 短语 / 逗号分隔的列表）\n- **数据划分**：``validation``（答案公开）和 ``test``（答案私有 — 跳过评估）\n\n## 核心特性\n\n- 基于 ReAct 智能体循环，内置单个 ``bash`` 工具，运行于 Docker 沙箱中（默认镜像为 ``python:3.11``，已预装 ``curl`` / ``wget`` / ``git``）。\n- 附件文件以只读方式挂载至沙箱内的 ``/shared_files`` 目录。\n- 评分器直接移植自官方 GAIA 排行榜的规则逻辑（不使用 LLM 作为评判）。\n- 默认从 ModelScope 下载数据集（``gaia-benchmark/GAIA``）；设置 ``dataset_hub='huggingface'`` 可改为从 Hugging Face 加载。\n\n## 评估说明\n\n- 需要在本地运行 Docker 守护进程（或通过 ms_enclave 配置使用远程沙箱引擎）。\n- ``extra_params.max_steps`` 限制智能体循环的最大步数（默认为 50）。\n- ``extra_params.command_timeout`` 设置每条 ``bash`` 命令的超时时间（默认 180 秒，与 inspect_ai 一致）。\n- 默认启用网络访问 — 许多问题需要进行网页浏览或搜索。\n- 使用 ``subset_list`` 可限定特定难度级别，例如 ``['2023_level1']``、``['2023_level1', '2023_level2']`` 或默认的 ``['2023_all']``。\n- [使用文档](https://evalscope.readthedocs.io/zh-cn/latest/third_party/gaia.html)\n\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `gaia` |\n| **数据集ID** | [gaia-benchmark/GAIA](https://modelscope.cn/datasets/gaia-benchmark/GAIA/summary) |\n| **论文** | N/A |\n| **标签** | `Agent`, `MultiTurn`, `Reasoning` |\n| **指标** | `acc` |\n| **默认示例数** | 0-shot |\n| **评估划分** | `validation` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 165 |\n| 提示词长度（平均） | 861.35 字符 |\n| 提示词长度（最小/最大） | 596 / 2582 字符 |\n\n**各子集统计信息：**\n\n| 子集 | 样本数 | 提示词平均长度 | 提示词最小长度 | 提示词最大长度 |\n|--------|---------|-------------|------------|------------|\n| `2023_level1` | 53 | 906.53 | 604 | 2582 |\n| `2023_level2` | 86 | 816.66 | 596 | 1275 |\n| `2023_level3` | 26 | 917.08 | 621 | 1497 |\n\n## 样例示例\n\n**子集**: `2023_level1`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"93e8257c\",\n      \"content\": \"Please answer the question below. You should:\\n\\n- Return only your answer, which should be a number, or a short phrase with as few words as possible, or a comma separated list of numbers and/or strings.\\n- If the answer is a number, return only ... [TRUNCATED 431 chars] ... Earth and the Moon its closest approach? Please use the minimum perigee value on the Wikipedia page for the Moon when carrying out your calculation. Round your result to the nearest 1000 hours and do not use any comma separators if necessary.\"\n    }\n  ],\n  \"target\": \"17\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"tools\": [\n    {\n      \"name\": \"bash\",\n      \"description\": \"Execute a bash command inside the sandbox environment. Returns the combined stdout / stderr output of the command.\",\n      \"parameters\": {\n        \"properties\": {\n          \"command\": {\n            \"type\": \"string\",\n            \"description\": \"The bash command to execute.\"\n          },\n          \"timeout\": {\n            \"type\": \"number\",\n            \"description\": \"Maximum execution time in seconds (default: 60).\",\n            \"default\": 60\n          }\n        },\n        \"required\": [\n          \"command\"\n        ]\n      }\n    }\n  ],\n  \"metadata\": {\n    \"task_id\": \"e1fc63a2-da7a-432f-be78-7c4a95598703\",\n    \"level\": \"1\",\n    \"file_name\": \"\",\n    \"file_path\": \"\",\n    \"Annotator Metadata\": {\n      \"Steps\": \"1. Googled Eliud Kipchoge marathon pace to find 4min 37sec/mile\\n2. Converted into fractions of hours.\\n3. Found moon periapsis in miles (225,623 miles).\\n4. Multiplied the two to find the number of hours and rounded to the nearest 100 hours.\",\n      \"Number of steps\": \"4\",\n      \"How long did this take?\": \"20 Minutes\",\n      \"Tools\": \"1. A web browser.\\n2. A search engine.\\n3. A calculator.\",\n      \"Number of tools\": \"3\"\n    }\n  }\n}\n```\n\n## 提示模板\n\n**提示模板：**\n```text\n{question}\n```\n\n## 额外参数\n\n| 参数 | 类型 | 默认值 | 描述 |\n|-----------|------|---------|-------------|\n| `max_steps` | `int` | `50` | 每个样本允许的最大智能体步数。 |\n| `command_timeout` | `float` | `180.0` | 每条 bash 命令的默认超时时间（秒）。 |\n| `docker_image` | `str` | `python:3.11` | 用作每个样本沙箱环境的 Docker 镜像。 |\n| `network_enabled` | `bool` | `True` | 是否允许沙箱访问网络（GAIA 中涉及浏览的问题需要此选项）。 |\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets gaia \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['gaia'],\n    dataset_args={\n        'gaia': {\n            # subset_list: ['2023_level1', '2023_level2', '2023_level3']  # 可选，用于评估特定子集\n            # extra_params: {}  # 使用默认额外参数\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "628ec8e67277a0ae28b0f6a8a6a4d09e",
    "needs_translation": false
  },
  "updated_at": "2026-06-01T14:08:06.294860",
  "translation_updated_at": "2026-06-01T14:08:20"
}