{
  "meta": {
    "pretty_name": "GSM8K",
    "dataset_id": "AI-ModelScope/gsm8k",
    "paper_url": "https://arxiv.org/abs/2110.14168",
    "tags": [
      "Math",
      "Reasoning"
    ],
    "metrics": [
      {
        "acc": {
          "numeric": true
        }
      }
    ],
    "few_shot_num": 4,
    "eval_split": "test",
    "train_split": "train",
    "subset_list": [
      "main"
    ],
    "description": "\n## Overview\n\nGSM8K (Grade School Math 8K) is a high-quality dataset of 8.5K linguistically diverse grade school math word problems created by human problem writers. The dataset is specifically designed to evaluate and improve the multi-step mathematical reasoning capabilities of language models.\n\n## Task Description\n\n- **Task Type**: Mathematical Word Problem Solving\n- **Input**: Natural language math word problem\n- **Output**: Numerical answer derived through step-by-step reasoning\n- **Difficulty**: Grade school level (2-8 reasoning steps required)\n\n## Key Features\n\n- Problems require basic arithmetic operations (addition, subtraction, multiplication, division)\n- Solutions involve 2 to 8 sequential reasoning steps\n- High linguistic diversity in problem formulations\n- Human-written problems ensuring natural language quality\n- Clear numerical answers for objective evaluation\n\n## Evaluation Notes\n\n- Default configuration uses **4-shot** examples with Chain-of-Thought (CoT) prompting\n- Answers should be formatted within `\\boxed{}` for proper extraction\n- The metric extracts numerical values for accuracy comparison\n- Supports both zero-shot and few-shot evaluation modes\n",
    "prompt_template": "{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
    "system_prompt": "",
    "few_shot_prompt_template": "Here are some examples of how to solve similar problems:\n\n{fewshot}\n\n{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.",
    "aggregation": "mean",
    "extra_params": {},
    "sandbox_config": {},
    "category": "llm"
  },
  "statistics": {
    "total_samples": 1319,
    "subset_stats": [
      {
        "name": "main",
        "sample_count": 1319,
        "prompt_length_mean": 1966.87,
        "prompt_length_min": 1800,
        "prompt_length_max": 2575,
        "prompt_length_std": 97.61,
        "target_length_mean": 2.29
      }
    ],
    "prompt_length": {
      "mean": 1966.87,
      "min": 1800,
      "max": 2575,
      "std": 97.61
    },
    "target_length_mean": 2.29,
    "computed_at": "2026-01-28T11:13:21.430820"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "0bc7f97b",
          "content": "Here are some examples of how to solve similar problems:\n\nNatalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\n\nReasoning:\nNatalia sold 48/ ... [TRUNCATED] ... ds every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?\nPlease reason step by step, and put your final answer within \\boxed{}."
        }
      ],
      "target": "18",
      "id": 0,
      "group_id": 0,
      "metadata": {
        "reasoning": "Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.\nShe makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s market."
      }
    },
    "subset": "main",
    "truncated": true
  },
  "readme": {
    "en": "# GSM8K\n\n\n## Overview\n\nGSM8K (Grade School Math 8K) is a high-quality dataset of 8.5K linguistically diverse grade school math word problems created by human problem writers. The dataset is specifically designed to evaluate and improve the multi-step mathematical reasoning capabilities of language models.\n\n## Task Description\n\n- **Task Type**: Mathematical Word Problem Solving\n- **Input**: Natural language math word problem\n- **Output**: Numerical answer derived through step-by-step reasoning\n- **Difficulty**: Grade school level (2-8 reasoning steps required)\n\n## Key Features\n\n- Problems require basic arithmetic operations (addition, subtraction, multiplication, division)\n- Solutions involve 2 to 8 sequential reasoning steps\n- High linguistic diversity in problem formulations\n- Human-written problems ensuring natural language quality\n- Clear numerical answers for objective evaluation\n\n## Evaluation Notes\n\n- Default configuration uses **4-shot** examples with Chain-of-Thought (CoT) prompting\n- Answers should be formatted within `\\boxed{}` for proper extraction\n- The metric extracts numerical values for accuracy comparison\n- Supports both zero-shot and few-shot evaluation modes\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `gsm8k` |\n| **Dataset ID** | [AI-ModelScope/gsm8k](https://modelscope.cn/datasets/AI-ModelScope/gsm8k/summary) |\n| **Paper** | [Paper](https://arxiv.org/abs/2110.14168) |\n| **Tags** | `Math`, `Reasoning` |\n| **Metrics** | `acc` |\n| **Default Shots** | 4-shot |\n| **Evaluation Split** | `test` |\n| **Train Split** | `train` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 1,319 |\n| Prompt Length (Mean) | 1966.87 chars |\n| Prompt Length (Min/Max) | 1800 / 2575 chars |\n\n## Sample Example\n\n**Subset**: `main`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"0bc7f97b\",\n      \"content\": \"Here are some examples of how to solve similar problems:\\n\\nNatalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\\n\\nReasoning:\\nNatalia sold 48/ ... [TRUNCATED] ... ds every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?\\nPlease reason step by step, and put your final answer within \\\\boxed{}.\"\n    }\n  ],\n  \"target\": \"18\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"reasoning\": \"Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.\\nShe makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s market.\"\n  }\n}\n```\n\n*Note: Some content was truncated for display.*\n\n## Prompt Template\n\n**Prompt Template:**\n```text\n{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.\n```\n\n<details>\n<summary>Few-shot Template</summary>\n\n```text\nHere are some examples of how to solve similar problems:\n\n{fewshot}\n\n{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.\n```\n\n</details>\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets gsm8k \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['gsm8k'],\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# GSM8K\n\n## 概述\n\nGSM8K（Grade School Math 8K）是一个高质量的数据集，包含由人工编写者创作的 8.5K 道语言多样化的中小学数学应用题。该数据集专门用于评估和提升语言模型的多步数学推理能力。\n\n## 任务描述\n\n- **任务类型**：数学应用题求解  \n- **输入**：自然语言描述的数学应用题  \n- **输出**：通过逐步推理得出的数值答案  \n- **难度**：小学水平（需 2–8 步推理）\n\n## 主要特点\n\n- 问题涉及基本算术运算（加、减、乘、除）  \n- 解答过程包含 2 到 8 个连续推理步骤  \n- 问题表述具有高度的语言多样性  \n- 人工编写确保自然语言质量  \n- 答案为明确的数值，便于客观评估  \n\n## 评估说明\n\n- 默认配置使用 **4-shot** 示例并结合思维链（Chain-of-Thought, CoT）提示  \n- 答案应使用 `\\boxed{}` 格式包裹以便正确提取  \n- 评估指标会提取数值进行准确率比较  \n- 支持零样本（zero-shot）和少样本（few-shot）两种评估模式  \n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `gsm8k` |\n| **数据集ID** | [AI-ModelScope/gsm8k](https://modelscope.cn/datasets/AI-ModelScope/gsm8k/summary) |\n| **论文** | [Paper](https://arxiv.org/abs/2110.14168) |\n| **标签** | `Math`, `Reasoning` |\n| **指标** | `acc` |\n| **默认样本数** | 4-shot |\n| **评估划分** | `test` |\n| **训练划分** | `train` |\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 1,319 |\n| 提示词长度（平均） | 1966.87 字符 |\n| 提示词长度（最小/最大） | 1800 / 2575 字符 |\n\n## 样例示例\n\n**子集**: `main`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"0bc7f97b\",\n      \"content\": \"Here are some examples of how to solve similar problems:\\n\\nNatalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?\\n\\nReasoning:\\nNatalia sold 48/ ... [TRUNCATED] ... ds every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?\\nPlease reason step by step, and put your final answer within \\\\boxed{}.\"\n    }\n  ],\n  \"target\": \"18\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"reasoning\": \"Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.\\nShe makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s market.\"\n  }\n}\n```\n\n*注：部分内容因展示需要已被截断。*\n\n## 提示模板\n\n**提示模板：**\n```text\n{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.\n```\n\n<details>\n<summary>少样本模板</summary>\n\n```text\nHere are some examples of how to solve similar problems:\n\n{fewshot}\n\n{question}\nPlease reason step by step, and put your final answer within \\boxed{{}}.\n```\n\n</details>\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets gsm8k \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['gsm8k'],\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "0bec17b778c58b45a31517042993da7e",
    "needs_translation": false
  },
  "updated_at": "2026-01-28T17:31:32.243854",
  "translation_updated_at": "2026-01-28T16:09:53Z"
}