{
  "meta": {
    "pretty_name": "ProcessBench",
    "dataset_id": "Qwen/ProcessBench",
    "paper_url": null,
    "tags": [
      "Math",
      "Reasoning"
    ],
    "metrics": [
      "error_acc",
      "correct_acc",
      "simple_f1_score"
    ],
    "few_shot_num": 0,
    "eval_split": "test",
    "train_split": "",
    "subset_list": [
      "gsm8k",
      "math",
      "olympiadbench",
      "omnimath"
    ],
    "description": "\n## Overview\n\nProcessBench is a benchmark for evaluating AI models on mathematical reasoning process verification. It tests the ability to identify errors in step-by-step mathematical solutions across various difficulty levels from GSM8K to OmniMath.\n\n## Task Description\n\n- **Task Type**: Mathematical Reasoning Error Detection\n- **Input**: Math problem + step-by-step solution (tagged paragraphs)\n- **Output**: Index of first error paragraph (or -1 if correct)\n- **Domains**: Math reasoning verification, error detection\n\n## Key Features\n\n- Four difficulty subsets:\n  - `gsm8k`: Grade school math problems\n  - `math`: Competition math problems\n  - `olympiadbench`: Olympiad-level problems\n  - `omnimath`: Advanced mathematical reasoning\n- Tests process supervision and verification abilities\n- Requires analyzing step-by-step reasoning for errors\n\n## Evaluation Notes\n\n- Default evaluation uses the **test** split\n- Multiple metrics tracked:\n  - `error_acc`: Accuracy on detecting error locations\n  - `correct_acc`: Accuracy on identifying correct solutions\n  - `simple_f1_score`: F1 score balancing both\n- Answers should be in \\boxed{} format (paragraph index or -1)\n- Aggregation method: **F1** score\n",
    "prompt_template": "The following is a math problem and a solution (split into paragraphs, enclosed with tags and indexed from 0):\n\n[Math Problem]\n\n{problem}\n\n[Solution]\n\n{tagged_response}\n\nYour task is to review and critique the solution paragraph by paragraph. Once you identify an error in a paragraph, return the index of the paragraph where the earliest error occurs. Otherwise, return the index of -1 (which typically denotes \"not found\").\n\nPlease put your final answer (i.e., the index) in \boxed{{}}.\n",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "f1",
    "extra_params": {},
    "sandbox_config": {},
    "category": "llm"
  },
  "statistics": {
    "total_samples": 3400,
    "subset_stats": [
      {
        "name": "gsm8k",
        "sample_count": 400,
        "prompt_length_mean": 1824.26,
        "prompt_length_min": 876,
        "prompt_length_max": 4520,
        "prompt_length_std": 436.07,
        "target_length_mean": 1.48
      },
      {
        "name": "math",
        "sample_count": 1000,
        "prompt_length_mean": 2297.11,
        "prompt_length_min": 690,
        "prompt_length_max": 7565,
        "prompt_length_std": 978.71,
        "target_length_mean": 1.41
      },
      {
        "name": "olympiadbench",
        "sample_count": 1000,
        "prompt_length_mean": 3166.77,
        "prompt_length_min": 1129,
        "prompt_length_max": 9005,
        "prompt_length_std": 1029.56,
        "target_length_mean": 1.37
      },
      {
        "name": "omnimath",
        "sample_count": 1000,
        "prompt_length_mean": 3206.82,
        "prompt_length_min": 832,
        "prompt_length_max": 8550,
        "prompt_length_std": 1087.59,
        "target_length_mean": 1.27
      }
    ],
    "prompt_length": {
      "mean": 2764.83,
      "min": 690,
      "max": 9005,
      "std": 1111.99
    },
    "target_length_mean": 1.37,
    "computed_at": "2026-01-28T11:16:12.678164"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "aca63163",
          "content": "The following is a math problem and a solution (split into paragraphs, enclosed with tags and indexed from 0):\n\n[Math Problem]\n\nSue lives in a fun neighborhood.  One weekend, the neighbors decided to play a prank on Sue.  On Friday morning, t ... [TRUNCATED] ... nce you identify an error in a paragraph, return the index of the paragraph where the earliest error occurs. Otherwise, return the index of -1 (which typically denotes \"not found\").\n\nPlease put your final answer (i.e., the index) in \boxed{}.\n"
        }
      ],
      "target": "1",
      "id": 0,
      "group_id": 0,
      "metadata": {
        "steps": [
          "To find out how many more pink plastic flamingos were out than white plastic flamingos at noon on Sunday, we can break down the problem into steps. First, on Friday, the neighbors start with 18 pink plastic flamingos.",
          "On Saturday, they take back one third of the flamingos. Since there were 18 flamingos, \\(1/3 \\times 18 = 6\\) flamingos are taken back. So, they have \\(18 - 6 = 12\\) flamingos left in their possession. Then, they paint these 6 flamingos white and put them back out on Sue's front yard. Now, Sue has the original 12 pink flamingos plus the 6 new white ones. Thus, by the end of Saturday, Sue has \\(12 + 6 = 18\\) pink flamingos and 6 white flamingos.",
          "On Sunday, the neighbors add another 18 pink plastic flamingos to Sue's front yard. By the end of Sunday morning, Sue has \\(18 + 18 = 36\\) pink flamingos and still 6 white flamingos.",
          "To find the difference, subtract the number of white flamingos from the number of pink flamingos: \\(36 - 6 = 30\\). Therefore, at noon on Sunday, there were 30 more pink plastic flamingos out than white plastic flamingos. The answer is \\(\\boxed{30}\\)."
        ],
        "tagged_response": "<paragraph_0>\nTo find out how many more pink plastic flamingos were out than white plastic flamingos at noon on Sunday, we can break down the problem into steps. First, on Friday, the neighbors start with 18 pink plastic flamingos.\n</paragrap ... [TRUNCATED] ...  subtract the number of white flamingos from the number of pink flamingos: \\(36 - 6 = 30\\). Therefore, at noon on Sunday, there were 30 more pink plastic flamingos out than white plastic flamingos. The answer is \\(\\boxed{30}\\).\n</paragraph_3>",
        "final_answer_correct": false
      }
    },
    "subset": "gsm8k",
    "truncated": true
  },
  "readme": {
    "en": "# ProcessBench\n\n\n## Overview\n\nProcessBench is a benchmark for evaluating AI models on mathematical reasoning process verification. It tests the ability to identify errors in step-by-step mathematical solutions across various difficulty levels from GSM8K to OmniMath.\n\n## Task Description\n\n- **Task Type**: Mathematical Reasoning Error Detection\n- **Input**: Math problem + step-by-step solution (tagged paragraphs)\n- **Output**: Index of first error paragraph (or -1 if correct)\n- **Domains**: Math reasoning verification, error detection\n\n## Key Features\n\n- Four difficulty subsets:\n  - `gsm8k`: Grade school math problems\n  - `math`: Competition math problems\n  - `olympiadbench`: Olympiad-level problems\n  - `omnimath`: Advanced mathematical reasoning\n- Tests process supervision and verification abilities\n- Requires analyzing step-by-step reasoning for errors\n\n## Evaluation Notes\n\n- Default evaluation uses the **test** split\n- Multiple metrics tracked:\n  - `error_acc`: Accuracy on detecting error locations\n  - `correct_acc`: Accuracy on identifying correct solutions\n  - `simple_f1_score`: F1 score balancing both\n- Answers should be in \\boxed{} format (paragraph index or -1)\n- Aggregation method: **F1** score\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `process_bench` |\n| **Dataset ID** | [Qwen/ProcessBench](https://modelscope.cn/datasets/Qwen/ProcessBench/summary) |\n| **Paper** | N/A |\n| **Tags** | `Math`, `Reasoning` |\n| **Metrics** | `error_acc`, `correct_acc`, `simple_f1_score` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n| **Aggregation** | `f1` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 3,400 |\n| Prompt Length (Mean) | 2764.83 chars |\n| Prompt Length (Min/Max) | 690 / 9005 chars |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `gsm8k` | 400 | 1824.26 | 876 | 4520 |\n| `math` | 1,000 | 2297.11 | 690 | 7565 |\n| `olympiadbench` | 1,000 | 3166.77 | 1129 | 9005 |\n| `omnimath` | 1,000 | 3206.82 | 832 | 8550 |\n\n## Sample Example\n\n**Subset**: `gsm8k`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"aca63163\",\n      \"content\": \"The following is a math problem and a solution (split into paragraphs, enclosed with tags and indexed from 0):\\n\\n[Math Problem]\\n\\nSue lives in a fun neighborhood.  One weekend, the neighbors decided to play a prank on Sue.  On Friday morning, t ... [TRUNCATED] ... nce you identify an error in a paragraph, return the index of the paragraph where the earliest error occurs. Otherwise, return the index of -1 (which typically denotes \\\"not found\\\").\\n\\nPlease put your final answer (i.e., the index) in \\boxed{}.\\n\"\n    }\n  ],\n  \"target\": \"1\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"steps\": [\n      \"To find out how many more pink plastic flamingos were out than white plastic flamingos at noon on Sunday, we can break down the problem into steps. First, on Friday, the neighbors start with 18 pink plastic flamingos.\",\n      \"On Saturday, they take back one third of the flamingos. Since there were 18 flamingos, \\\\(1/3 \\\\times 18 = 6\\\\) flamingos are taken back. So, they have \\\\(18 - 6 = 12\\\\) flamingos left in their possession. Then, they paint these 6 flamingos white and put them back out on Sue's front yard. Now, Sue has the original 12 pink flamingos plus the 6 new white ones. Thus, by the end of Saturday, Sue has \\\\(12 + 6 = 18\\\\) pink flamingos and 6 white flamingos.\",\n      \"On Sunday, the neighbors add another 18 pink plastic flamingos to Sue's front yard. By the end of Sunday morning, Sue has \\\\(18 + 18 = 36\\\\) pink flamingos and still 6 white flamingos.\",\n      \"To find the difference, subtract the number of white flamingos from the number of pink flamingos: \\\\(36 - 6 = 30\\\\). Therefore, at noon on Sunday, there were 30 more pink plastic flamingos out than white plastic flamingos. The answer is \\\\(\\\\boxed{30}\\\\).\"\n    ],\n    \"tagged_response\": \"<paragraph_0>\\nTo find out how many more pink plastic flamingos were out than white plastic flamingos at noon on Sunday, we can break down the problem into steps. First, on Friday, the neighbors start with 18 pink plastic flamingos.\\n</paragrap ... [TRUNCATED] ...  subtract the number of white flamingos from the number of pink flamingos: \\\\(36 - 6 = 30\\\\). Therefore, at noon on Sunday, there were 30 more pink plastic flamingos out than white plastic flamingos. The answer is \\\\(\\\\boxed{30}\\\\).\\n</paragraph_3>\",\n    \"final_answer_correct\": false\n  }\n}\n```\n\n*Note: Some content was truncated for display.*\n\n## Prompt Template\n\n**Prompt Template:**\n```text\nThe following is a math problem and a solution (split into paragraphs, enclosed with tags and indexed from 0):\n\n[Math Problem]\n\n{problem}\n\n[Solution]\n\n{tagged_response}\n\nYour task is to review and critique the solution paragraph by paragraph. Once you identify an error in a paragraph, return the index of the paragraph where the earliest error occurs. Otherwise, return the index of -1 (which typically denotes \"not found\").\n\nPlease put your final answer (i.e., the index) in \boxed{{}}.\n\n```\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets process_bench \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['process_bench'],\n    dataset_args={\n        'process_bench': {\n            # subset_list: ['gsm8k', 'math', 'olympiadbench']  # optional, evaluate specific subsets\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# ProcessBench\n\n## 概述\n\nProcessBench 是一个用于评估 AI 模型在数学推理过程验证能力的基准测试。它测试模型在从 GSM8K 到 OmniMath 等不同难度级别的数学问题中，识别分步解答错误的能力。\n\n## 任务描述\n\n- **任务类型**：数学推理错误检测\n- **输入**：数学问题 + 分步解答（带标签的段落）\n- **输出**：第一个错误段落的索引（若解答正确则为 -1）\n- **领域**：数学推理验证、错误检测\n\n## 主要特性\n\n- 包含四个难度子集：\n  - `gsm8k`：小学数学问题\n  - `math`：竞赛数学问题\n  - `olympiadbench`：奥林匹克级别问题\n  - `omnimath`：高级数学推理\n- 测试过程监督与验证能力\n- 要求逐段分析推理过程以发现错误\n\n## 评估说明\n\n- 默认使用 **test** 数据划分进行评估\n- 跟踪多项指标：\n  - `error_acc`：检测错误位置的准确率\n  - `correct_acc`：识别正确解答的准确率\n  - `simple_f1_score`：综合两项的 F1 分数\n- 答案应使用 \\boxed{} 格式（段落索引或 -1）\n- 聚合方法：**F1** 分数\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `process_bench` |\n| **数据集 ID** | [Qwen/ProcessBench](https://modelscope.cn/datasets/Qwen/ProcessBench/summary) |\n| **论文** | N/A |\n| **标签** | `Math`, `Reasoning` |\n| **指标** | `error_acc`, `correct_acc`, `simple_f1_score` |\n| **默认示例数量** | 0-shot |\n| **评估划分** | `test` |\n| **聚合方式** | `f1` |\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 3,400 |\n| 提示词长度（平均） | 2764.83 字符 |\n| 提示词长度（最小/最大） | 690 / 9005 字符 |\n\n**各子集统计数据：**\n\n| 子集 | 样本数 | 提示平均长度 | 提示最小长度 | 提示最大长度 |\n|--------|---------|-------------|------------|------------|\n| `gsm8k` | 400 | 1824.26 | 876 | 4520 |\n| `math` | 1,000 | 2297.11 | 690 | 7565 |\n| `olympiadbench` | 1,000 | 3166.77 | 1129 | 9005 |\n| `omnimath` | 1,000 | 3206.82 | 832 | 8550 |\n\n## 样例示例\n\n**子集**: `gsm8k`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"aca63163\",\n      \"content\": \"The following is a math problem and a solution (split into paragraphs, enclosed with tags and indexed from 0):\\n\\n[Math Problem]\\n\\nSue lives in a fun neighborhood.  One weekend, the neighbors decided to play a prank on Sue.  On Friday morning, t ... [TRUNCATED] ... nce you identify an error in a paragraph, return the index of the paragraph where the earliest error occurs. Otherwise, return the index of -1 (which typically denotes \\\"not found\\\").\\n\\nPlease put your final answer (i.e., the index) in \\boxed{}.\\n\"\n    }\n  ],\n  \"target\": \"1\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"steps\": [\n      \"To find out how many more pink plastic flamingos were out than white plastic flamingos at noon on Sunday, we can break down the problem into steps. First, on Friday, the neighbors start with 18 pink plastic flamingos.\",\n      \"On Saturday, they take back one third of the flamingos. Since there were 18 flamingos, \\\\(1/3 \\\\times 18 = 6\\\\) flamingos are taken back. So, they have \\\\(18 - 6 = 12\\\\) flamingos left in their possession. Then, they paint these 6 flamingos white and put them back out on Sue's front yard. Now, Sue has the original 12 pink flamingos plus the 6 new white ones. Thus, by the end of Saturday, Sue has \\\\(12 + 6 = 18\\\\) pink flamingos and 6 white flamingos.\",\n      \"On Sunday, the neighbors add another 18 pink plastic flamingos to Sue's front yard. By the end of Sunday morning, Sue has \\\\(18 + 18 = 36\\\\) pink flamingos and still 6 white flamingos.\",\n      \"To find the difference, subtract the number of white flamingos from the number of pink flamingos: \\\\(36 - 6 = 30\\\\). Therefore, at noon on Sunday, there were 30 more pink plastic flamingos out than white plastic flamingos. The answer is \\\\(\\\\boxed{30}\\\\).\"\n    ],\n    \"tagged_response\": \"<paragraph_0>\\nTo find out how many more pink plastic flamingos were out than white plastic flamingos at noon on Sunday, we can break down the problem into steps. First, on Friday, the neighbors start with 18 pink plastic flamingos.\\n</paragrap ... [TRUNCATED] ...  subtract the number of white flamingos from the number of pink flamingos: \\\\(36 - 6 = 30\\\\). Therefore, at noon on Sunday, there were 30 more pink plastic flamingos out than white plastic flamingos. The answer is \\\\(\\\\boxed{30}\\\\).\\n</paragraph_3>\",\n    \"final_answer_correct\": false\n  }\n}\n```\n\n*注：部分内容因展示需要已被截断。*\n\n## 提示模板\n\n**提示模板：**\n```text\nThe following is a math problem and a solution (split into paragraphs, enclosed with tags and indexed from 0):\n\n[Math Problem]\n\n{problem}\n\n[Solution]\n\n{tagged_response}\n\nYour task is to review and critique the solution paragraph by paragraph. Once you identify an error in a paragraph, return the index of the paragraph where the earliest error occurs. Otherwise, return the index of -1 (which typically denotes \"not found\").\n\nPlease put your final answer (i.e., the index) in \\boxed{}.\n\n```\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets process_bench \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['process_bench'],\n    dataset_args={\n        'process_bench': {\n            # subset_list: ['gsm8k', 'math', 'olympiadbench']  # 可选，用于评估特定子集\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "ca5bc409abb72d62abe6ccf44b6d48bc",
    "needs_translation": false
  },
  "updated_at": "2026-01-28T17:31:32.431671",
  "translation_updated_at": "2026-01-28T16:09:53Z"
}