{
  "meta": {
    "pretty_name": "FRAMES",
    "dataset_id": "iic/frames",
    "paper_url": null,
    "tags": [
      "Reasoning",
      "LongContext"
    ],
    "metrics": [
      "acc"
    ],
    "few_shot_num": 0,
    "eval_split": "test",
    "train_split": "",
    "subset_list": [
      "default"
    ],
    "description": "\n## Overview\n\nFRAMES is a comprehensive evaluation dataset designed to test the capabilities of Retrieval-Augmented Generation (RAG) systems. It evaluates factuality, retrieval accuracy, and reasoning abilities in long-context scenarios.\n\n## Task Description\n\n- **Task Type**: RAG Evaluation / Long-Context QA\n- **Input**: Wikipedia context documents + question\n- **Output**: Factual answer in specified format\n- **Domains**: Factuality, retrieval, multi-hop reasoning\n\n## Key Features\n\n- Tests core RAG capabilities: factuality, retrieval, reasoning\n- Provides Wikipedia-sourced context documents\n- Questions require synthesizing information from multiple sources\n- Evaluates both retrieval quality and answer generation\n- Supports both exact match and LLM judge evaluation\n\n## Evaluation Notes\n\n- Default evaluation uses the **test** split\n- Primary metric: **Accuracy** with both exact match and LLM judge\n- Response format: \"Therefore, the answer is (answer here)\"\n- Uses normalized answer comparison for exact matching\n- LLM judge provides flexible semantic matching\n",
    "prompt_template": "Please read the following text and answer the question below.\n\n<text>\n{context}\n</text>\n\n{question}\n\nFormat your response as follows: \"Therefore, the answer is (insert answer here)\".",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {},
    "sandbox_config": {},
    "category": "llm"
  },
  "statistics": {
    "total_samples": 824,
    "subset_stats": [
      {
        "name": "test",
        "sample_count": 824,
        "prompt_length_mean": 67976.16,
        "prompt_length_min": 235,
        "prompt_length_max": 557427,
        "prompt_length_std": 67482.95,
        "target_length_mean": 30.7
      }
    ],
    "prompt_length": {
      "mean": 67976.16,
      "min": 235,
      "max": 557427,
      "std": 67482.95
    },
    "target_length_mean": 30.7,
    "computed_at": "2026-01-28T11:13:27.268572"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "6845108b",
          "content": "Please read the following text and answer the question below.\n\n<text>\nPresident_of_the_United_States\nThe president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs ... [TRUNCATED] ... irst lady of the United States' mother and her surname is the same as the second assassinated president's mother's maiden name, what is my future wife's name? \n\nFormat your response as follows: \"Therefore, the answer is (insert answer here)\"."
        }
      ],
      "target": "Jane Ballou",
      "id": 0,
      "group_id": 0,
      "metadata": {
        "context": "President_of_the_United_States\nThe president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs the executive branch of the federal government and is the commander-i ... [TRUNCATED] ... Garfield from the U.S. National Library of Medicine. Contains medical bulletins issued by attending physicians D. Hayes Agnes, J.K. Barnes, D. W. Bliss, Frank H. Hamilton, Robert Reyburn, and J.J. Woodward between July 6 – September 19, 1881.",
        "wiki_items": [
          {
            "title": "President_of_the_United_States",
            "link": "https://en.wikipedia.org/wiki/President_of_the_United_States",
            "frames_prompt_id": "[0, 261]",
            "original_link": "['https://en.wikipedia.org/wiki/President_of_the_United_States', 'https://en.wikipedia.org/wiki/President_of_the_United_States#History_and_development']",
            "text": "The president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs the executive branch of the federal government and is the commander-in-chief of the United States Ar ... [TRUNCATED] ... assuming office.\n\nSee also\nOutline of American politics\n\nNotes\nReferences\nFurther reading\nExternal links\n\nWhite House homepage\nUnited States Presidents Collection. General Collection, Beinecke Rare Book and Manuscript Library, Yale University"
          },
          {
            "title": "James_Buchanan",
            "link": "https://en.wikipedia.org/wiki/James_Buchanan",
            "frames_prompt_id": "[0]",
            "original_link": "['https://en.wikipedia.org/wiki/James_Buchanan']",
            "text": "James Buchanan Jr. ( bew-KAN-ən; April 23, 1791 – June 1, 1868) was the 15th president of the United States, serving from 1857 to 1861. Buchanan also served as the secretary of state from 1845 to 1849 and represented Pennsylvania in both hous ... [TRUNCATED] ... Letters Shapell Manuscript Foundation\nMr. Buchanans Administration on the Eve of the Rebellion. President Buchanans memoirs.\nInaugural Address Archived August 9, 2020, at the Wayback Machine\nFourth Annual Message to Congress, December 3, 1860"
          },
          {
            "title": "Harriet_Lane",
            "link": "https://en.wikipedia.org/wiki/Harriet_Lane",
            "frames_prompt_id": "[0]",
            "original_link": "['https://en.wikipedia.org/wiki/Harriet_Lane']",
            "text": "Harriet Rebecca Lane Johnston (May 9, 1830 – July 3, 1903) acted as first lady of the United States during the administration of her uncle, lifelong bachelor president James Buchanan, from 1857 to 1861. She has been described as the first of  ... [TRUNCATED] ... nan Dying (play). (Ms. Johnston is a character in Updike's fictional play about President Buchanan.)\n\nExternal links\nWorks by or about Harriet Lane at the Internet Archive\n\"Harriet Lane\". First Ladies: Influence & Image. firstladies.org. CNN."
          },
          {
            "title": "List_of_presidents_of_the_United_States_who_died_in_office",
            "link": "https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States_who_died_in_office",
            "frames_prompt_id": "[0]",
            "original_link": "['https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States_who_died_in_office']",
            "text": "Since the office was established in 1789, 45 persons have served as president of the United States. Of these, eight have died in office: four were assassinated, and four died of natural causes. In each of these instances, the vice president h ... [TRUNCATED] ... te University Press. ISBN 0-87338-210-2.\nVowell, Sarah (2005). Assassination Vacation. Simon and Schuster. ISBN 0-7432-6003-1.\n\nExternal links\nThe Mortal Presidency Archived June 3, 2015, at the Wayback Machine (Shapell Manuscript Foundation)"
          },
          {
            "title": "James_A._Garfield",
            "link": "https://en.wikipedia.org/wiki/James_A._Garfield",
            "frames_prompt_id": "[0]",
            "original_link": "['https://en.wikipedia.org/wiki/James_A._Garfield']",
            "text": "James Abram Garfield (November 19, 1831 – September 19, 1881) was the 20th president of the United States, serving from March 1881 until his assassination in September that year. A preacher, lawyer, and Civil War general, Garfield served nine ... [TRUNCATED] ... Garfield from the U.S. National Library of Medicine. Contains medical bulletins issued by attending physicians D. Hayes Agnes, J.K. Barnes, D. W. Bliss, Frank H. Hamilton, Robert Reyburn, and J.J. Woodward between July 6 – September 19, 1881."
          }
        ]
      }
    },
    "subset": "test",
    "truncated": true
  },
  "readme": {
    "en": "# FRAMES\n\n\n## Overview\n\nFRAMES is a comprehensive evaluation dataset designed to test the capabilities of Retrieval-Augmented Generation (RAG) systems. It evaluates factuality, retrieval accuracy, and reasoning abilities in long-context scenarios.\n\n## Task Description\n\n- **Task Type**: RAG Evaluation / Long-Context QA\n- **Input**: Wikipedia context documents + question\n- **Output**: Factual answer in specified format\n- **Domains**: Factuality, retrieval, multi-hop reasoning\n\n## Key Features\n\n- Tests core RAG capabilities: factuality, retrieval, reasoning\n- Provides Wikipedia-sourced context documents\n- Questions require synthesizing information from multiple sources\n- Evaluates both retrieval quality and answer generation\n- Supports both exact match and LLM judge evaluation\n\n## Evaluation Notes\n\n- Default evaluation uses the **test** split\n- Primary metric: **Accuracy** with both exact match and LLM judge\n- Response format: \"Therefore, the answer is (answer here)\"\n- Uses normalized answer comparison for exact matching\n- LLM judge provides flexible semantic matching\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `frames` |\n| **Dataset ID** | [iic/frames](https://modelscope.cn/datasets/iic/frames/summary) |\n| **Paper** | N/A |\n| **Tags** | `LongContext`, `Reasoning` |\n| **Metrics** | `acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 824 |\n| Prompt Length (Mean) | 67976.16 chars |\n| Prompt Length (Min/Max) | 235 / 557427 chars |\n\n## Sample Example\n\n**Subset**: `test`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"6845108b\",\n      \"content\": \"Please read the following text and answer the question below.\\n\\n<text>\\nPresident_of_the_United_States\\nThe president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs ... [TRUNCATED] ... irst lady of the United States' mother and her surname is the same as the second assassinated president's mother's maiden name, what is my future wife's name? \\n\\nFormat your response as follows: \\\"Therefore, the answer is (insert answer here)\\\".\"\n    }\n  ],\n  \"target\": \"Jane Ballou\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"context\": \"President_of_the_United_States\\nThe president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs the executive branch of the federal government and is the commander-i ... [TRUNCATED] ... Garfield from the U.S. National Library of Medicine. Contains medical bulletins issued by attending physicians D. Hayes Agnes, J.K. Barnes, D. W. Bliss, Frank H. Hamilton, Robert Reyburn, and J.J. Woodward between July 6 – September 19, 1881.\",\n    \"wiki_items\": [\n      {\n        \"title\": \"President_of_the_United_States\",\n        \"link\": \"https://en.wikipedia.org/wiki/President_of_the_United_States\",\n        \"frames_prompt_id\": \"[0, 261]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/President_of_the_United_States', 'https://en.wikipedia.org/wiki/President_of_the_United_States#History_and_development']\",\n        \"text\": \"The president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs the executive branch of the federal government and is the commander-in-chief of the United States Ar ... [TRUNCATED] ... assuming office.\\n\\nSee also\\nOutline of American politics\\n\\nNotes\\nReferences\\nFurther reading\\nExternal links\\n\\nWhite House homepage\\nUnited States Presidents Collection. General Collection, Beinecke Rare Book and Manuscript Library, Yale University\"\n      },\n      {\n        \"title\": \"James_Buchanan\",\n        \"link\": \"https://en.wikipedia.org/wiki/James_Buchanan\",\n        \"frames_prompt_id\": \"[0]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/James_Buchanan']\",\n        \"text\": \"James Buchanan Jr. ( bew-KAN-ən; April 23, 1791 – June 1, 1868) was the 15th president of the United States, serving from 1857 to 1861. Buchanan also served as the secretary of state from 1845 to 1849 and represented Pennsylvania in both hous ... [TRUNCATED] ... Letters Shapell Manuscript Foundation\\nMr. Buchanans Administration on the Eve of the Rebellion. President Buchanans memoirs.\\nInaugural Address Archived August 9, 2020, at the Wayback Machine\\nFourth Annual Message to Congress, December 3, 1860\"\n      },\n      {\n        \"title\": \"Harriet_Lane\",\n        \"link\": \"https://en.wikipedia.org/wiki/Harriet_Lane\",\n        \"frames_prompt_id\": \"[0]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/Harriet_Lane']\",\n        \"text\": \"Harriet Rebecca Lane Johnston (May 9, 1830 – July 3, 1903) acted as first lady of the United States during the administration of her uncle, lifelong bachelor president James Buchanan, from 1857 to 1861. She has been described as the first of  ... [TRUNCATED] ... nan Dying (play). (Ms. Johnston is a character in Updike's fictional play about President Buchanan.)\\n\\nExternal links\\nWorks by or about Harriet Lane at the Internet Archive\\n\\\"Harriet Lane\\\". First Ladies: Influence & Image. firstladies.org. CNN.\"\n      },\n      {\n        \"title\": \"List_of_presidents_of_the_United_States_who_died_in_office\",\n        \"link\": \"https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States_who_died_in_office\",\n        \"frames_prompt_id\": \"[0]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States_who_died_in_office']\",\n        \"text\": \"Since the office was established in 1789, 45 persons have served as president of the United States. Of these, eight have died in office: four were assassinated, and four died of natural causes. In each of these instances, the vice president h ... [TRUNCATED] ... te University Press. ISBN 0-87338-210-2.\\nVowell, Sarah (2005). Assassination Vacation. Simon and Schuster. ISBN 0-7432-6003-1.\\n\\nExternal links\\nThe Mortal Presidency Archived June 3, 2015, at the Wayback Machine (Shapell Manuscript Foundation)\"\n      },\n      {\n        \"title\": \"James_A._Garfield\",\n        \"link\": \"https://en.wikipedia.org/wiki/James_A._Garfield\",\n        \"frames_prompt_id\": \"[0]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/James_A._Garfield']\",\n        \"text\": \"James Abram Garfield (November 19, 1831 – September 19, 1881) was the 20th president of the United States, serving from March 1881 until his assassination in September that year. A preacher, lawyer, and Civil War general, Garfield served nine ... [TRUNCATED] ... Garfield from the U.S. National Library of Medicine. Contains medical bulletins issued by attending physicians D. Hayes Agnes, J.K. Barnes, D. W. Bliss, Frank H. Hamilton, Robert Reyburn, and J.J. Woodward between July 6 – September 19, 1881.\"\n      }\n    ]\n  }\n}\n```\n\n*Note: Some content was truncated for display.*\n\n## Prompt Template\n\n**Prompt Template:**\n```text\nPlease read the following text and answer the question below.\n\n<text>\n{context}\n</text>\n\n{question}\n\nFormat your response as follows: \"Therefore, the answer is (insert answer here)\".\n```\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets frames \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['frames'],\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# FRAMES\n\n\n## 概述\n\nFRAMES 是一个全面的评估数据集，旨在测试检索增强生成（Retrieval-Augmented Generation, RAG）系统的能力。它在长上下文场景中评估事实性、检索准确性和推理能力。\n\n## 任务描述\n\n- **任务类型**：RAG 评估 / 长上下文问答（Long-Context QA）\n- **输入**：维基百科上下文文档 + 问题\n- **输出**：指定格式的事实性答案\n- **领域**：事实性、检索、多跳推理\n\n## 主要特点\n\n- 测试 RAG 的核心能力：事实性、检索、推理\n- 提供源自维基百科的上下文文档\n- 问题需要综合多个来源的信息\n- 同时评估检索质量和答案生成效果\n- 支持精确匹配（exact match）和大语言模型（LLM）裁判两种评估方式\n\n## 评估说明\n\n- 默认使用 **test** 数据划分进行评估\n- 主要指标：**准确率（Accuracy）**，结合精确匹配和 LLM 裁判\n- 回答格式：`\"Therefore, the answer is (answer here)\"`\n- 精确匹配采用标准化后的答案比较\n- LLM 裁判提供灵活的语义匹配能力\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `frames` |\n| **数据集ID** | [iic/frames](https://modelscope.cn/datasets/iic/frames/summary) |\n| **论文** | N/A |\n| **标签** | `LongContext`, `Reasoning` |\n| **指标** | `acc` |\n| **默认示例数** | 0-shot |\n| **评估划分** | `test` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 824 |\n| 提示词长度（平均） | 67976.16 字符 |\n| 提示词长度（最小/最大） | 235 / 557427 字符 |\n\n## 样例示例\n\n**子集**: `test`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"6845108b\",\n      \"content\": \"Please read the following text and answer the question below.\\n\\n<text>\\nPresident_of_the_United_States\\nThe president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs ... [TRUNCATED] ... irst lady of the United States' mother and her surname is the same as the second assassinated president's mother's maiden name, what is my future wife's name? \\n\\nFormat your response as follows: \\\"Therefore, the answer is (insert answer here)\\\".\"\n    }\n  ],\n  \"target\": \"Jane Ballou\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"context\": \"President_of_the_United_States\\nThe president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs the executive branch of the federal government and is the commander-i ... [TRUNCATED] ... Garfield from the U.S. National Library of Medicine. Contains medical bulletins issued by attending physicians D. Hayes Agnes, J.K. Barnes, D. W. Bliss, Frank H. Hamilton, Robert Reyburn, and J.J. Woodward between July 6 – September 19, 1881.\",\n    \"wiki_items\": [\n      {\n        \"title\": \"President_of_the_United_States\",\n        \"link\": \"https://en.wikipedia.org/wiki/President_of_the_United_States\",\n        \"frames_prompt_id\": \"[0, 261]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/President_of_the_United_States', 'https://en.wikipedia.org/wiki/President_of_the_United_States#History_and_development']\",\n        \"text\": \"The president of the United States (POTUS) is the head of state and head of government of the United States of America. The president directs the executive branch of the federal government and is the commander-in-chief of the United States Ar ... [TRUNCATED] ... assuming office.\\n\\nSee also\\nOutline of American politics\\n\\nNotes\\nReferences\\nFurther reading\\nExternal links\\n\\nWhite House homepage\\nUnited States Presidents Collection. General Collection, Beinecke Rare Book and Manuscript Library, Yale University\"\n      },\n      {\n        \"title\": \"James_Buchanan\",\n        \"link\": \"https://en.wikipedia.org/wiki/James_Buchanan\",\n        \"frames_prompt_id\": \"[0]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/James_Buchanan']\",\n        \"text\": \"James Buchanan Jr. ( bew-KAN-ən; April 23, 1791 – June 1, 1868) was the 15th president of the United States, serving from 1857 to 1861. Buchanan also served as the secretary of state from 1845 to 1849 and represented Pennsylvania in both hous ... [TRUNCATED] ... Letters Shapell Manuscript Foundation\\nMr. Buchanans Administration on the Eve of the Rebellion. President Buchanans memoirs.\\nInaugural Address Archived August 9, 2020, at the Wayback Machine\\nFourth Annual Message to Congress, December 3, 1860\"\n      },\n      {\n        \"title\": \"Harriet_Lane\",\n        \"link\": \"https://en.wikipedia.org/wiki/Harriet_Lane\",\n        \"frames_prompt_id\": \"[0]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/Harriet_Lane']\",\n        \"text\": \"Harriet Rebecca Lane Johnston (May 9, 1830 – July 3, 1903) acted as first lady of the United States during the administration of her uncle, lifelong bachelor president James Buchanan, from 1857 to 1861. She has been described as the first of  ... [TRUNCATED] ... nan Dying (play). (Ms. Johnston is a character in Updike's fictional play about President Buchanan.)\\n\\nExternal links\\nWorks by or about Harriet Lane at the Internet Archive\\n\\\"Harriet Lane\\\". First Ladies: Influence & Image. firstladies.org. CNN.\"\n      },\n      {\n        \"title\": \"List_of_presidents_of_the_United_States_who_died_in_office\",\n        \"link\": \"https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States_who_died_in_office\",\n        \"frames_prompt_id\": \"[0]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States_who_died_in_office']\",\n        \"text\": \"Since the office was established in 1789, 45 persons have served as president of the United States. Of these, eight have died in office: four were assassinated, and four died of natural causes. In each of these instances, the vice president h ... [TRUNCATED] ... te University Press. ISBN 0-87338-210-2.\\nVowell, Sarah (2005). Assassination Vacation. Simon and Schuster. ISBN 0-7432-6003-1.\\n\\nExternal links\\nThe Mortal Presidency Archived June 3, 2015, at the Wayback Machine (Shapell Manuscript Foundation)\"\n      },\n      {\n        \"title\": \"James_A._Garfield\",\n        \"link\": \"https://en.wikipedia.org/wiki/James_A._Garfield\",\n        \"frames_prompt_id\": \"[0]\",\n        \"original_link\": \"['https://en.wikipedia.org/wiki/James_A._Garfield']\",\n        \"text\": \"James Abram Garfield (November 19, 1831 – September 19, 1881) was the 20th president of the United States, serving from March 1881 until his assassination in September that year. A preacher, lawyer, and Civil War general, Garfield served nine ... [TRUNCATED] ... Garfield from the U.S. National Library of Medicine. Contains medical bulletins issued by attending physicians D. Hayes Agnes, J.K. Barnes, D. W. Bliss, Frank H. Hamilton, Robert Reyburn, and J.J. Woodward between July 6 – September 19, 1881.\"\n      }\n    ]\n  }\n}\n```\n\n*注：部分内容为显示目的已截断。*\n\n## 提示模板\n\n**提示模板：**\n```text\nPlease read the following text and answer the question below.\n\n<text>\n{context}\n</text>\n\n{question}\n\nFormat your response as follows: \"Therefore, the answer is (insert answer here)\".\n```\n\n## 使用方法\n\n### 使用命令行（CLI）\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets frames \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['frames'],\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "ed8f7dc8de5ea7339e610134d9800bac",
    "needs_translation": false
  },
  "updated_at": "2026-01-28T17:31:32.231481",
  "translation_updated_at": "2026-01-28T16:09:53Z"
}