{
  "meta": {
    "pretty_name": "CL-bench",
    "dataset_id": "tencent-community/CL-bench",
    "paper_url": null,
    "tags": [
      "InstructionFollowing",
      "Reasoning"
    ],
    "metrics": [
      "acc"
    ],
    "few_shot_num": 0,
    "eval_split": "train",
    "train_split": "",
    "subset_list": [
      "default"
    ],
    "description": "\n## Overview\n\nCL-bench represents a step towards building LMs with this fundamental capability (Context Learning), making them more intelligent and advancing their deployment in real-world scenarios. This benchmark is specifically designed to evaluate a model's ability to learn specific, often novel or long-tail knowledge directly from the context and apply it to solve problems, simulating real-world learning processes.\n\n**Resources:**\n[Homepage](https://github.com/Tencent-Hunyuan/CL-bench) | [Dataset](https://huggingface.co/datasets/tencent/CL-bench)\n\n## Task Description\n\n- **Task Type**: In-Context Learning & Reasoning (Context-dependent QA)\n- **Input**: A context containing new rules, fictional information, or specific logic, followed by a query\n- **Output**: A solution derived strictly from the provided context (not pre-trained knowledge)\n- **Difficulty**: Varied, requiring understanding of novel concepts defined in the prompt\n\n## Key Features\n\n- **Contamination-Free**: Uses synthetic or highly specific data (e.g., fictional laws, new programming syntax) to ensure models cannot rely on memorized training data\n- **Real-world Simulation**: Mimics scenarios where humans learn new tasks by reading instructions or documentation\n- **Diverse Domains**: Covers logic reasoning, rule following, language understanding, and puzzle solving within a given context\n- **Evaluation of Adaptability**: Measures the \"learning\" capability rather than just \"retrieval\" or \"memory\"\n\n## Evaluation Notes\n\n- Focuses on the model's ability to follow strict instructions provided in the context\n- Evaluation typically checks if the reasoning process utilizes the unique information given in the prompt\n- Answers are often evaluated against specific ground-truth rules defined in the context\n- Crucial for assessing how well models can adapt to private data or dynamic environments without fine-tuning\n",
    "prompt_template": "",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {},
    "sandbox_config": {},
    "category": "llm"
  },
  "statistics": {
    "total_samples": 1899,
    "subset_stats": [
      {
        "name": "default",
        "sample_count": 1899,
        "prompt_length_mean": 42491.73,
        "prompt_length_min": 6595,
        "prompt_length_max": 320655,
        "prompt_length_std": 47544.85,
        "target_length_mean": 3147.75
      }
    ],
    "prompt_length": {
      "mean": 42491.73,
      "min": 6595,
      "max": 320655,
      "std": 47544.85
    },
    "target_length_mean": 3147.75,
    "computed_at": "2026-02-07T10:36:33.002868"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "b0a2c972",
          "content": "You are an AI designed to play the Twisted Cryptids board Game developed by Unstable Unicorns. Your purpose is to teach rules, host/referee a session, simulate AI opponents if requested, track game state, and provide strategy advice for the g ... [TRUNCATED 861 chars] ...  brief rationale, then a single short quip).\n\nYou take the role of an eccentric conspiracy theorist when talking to the user, but constrain your tone to brief, optional quips that never obscure or delay rules, turn summaries, or move prompts.",
          "role": "system"
        },
        {
          "id": "109b41de",
          "content": "RULE BOOK\nTM\nTM\nIt’s so hard to feel seen as a Cryptid these days. In this easy-to-learn strategy game,\nyou’ll navigate your love-hate relationship with humans and outwit your fellow Cryptids\nto earn your status as a true legend. As hikers, h ... [TRUNCATED 158305 chars] ...  game\n02:23:53.880 designers what the gloi cryp it is to be fair we still have a dark horse because\n02:24:00.560 uh the worm is still underway yeah and who knows how gloopy they will\n\nThis is my first time playing, what do Sightings Cards do?",
          "role": "user"
        }
      ],
      "target": [
        "The response should define what a Sighting card is and its role when playing Twisted Cryptids. For example, Sighting cards are revealed to award/penalize Myth and trigger effects that may move Humans.",
        "The response should name the four different sighting card types, namely: Decoys, Hoaxes, Silhouettes, and Real Deals.",
        "The response should state the requirements for when sighting cards can be revealed: for example, during the dusk phase of each round, during an encounter at a site in the wilderness, when a player has at least 1 hiding spot at the encounter site.",
        "The response should explain how sighting cards are scored once revealed. For example, by gaining or losing myth that depends on the distribution of human types present at the site of the encounter, as specified on the sighting card.",
        "The response should explain the following gameplay sequence in order, which happens when a sighting is going to be revealed: 1. reveal the top face down sighting card in the corresponding stack; 2. apply myth gain/loss shown on the card; 3. r ... [TRUNCATED 161 chars] ... lace it on its corresponding stack (or if that sighting reveal emptied the stack, return the hiding spot token to the box for the rest of the game); 6. the next player who has a hiding spot at the site of the encounter reveals their sighting.",
        "The response should state that players can look at their own sighting cards at any time, but other player's sightings remain hidden until revealed.",
        "The response should state that a player can gain 7 Myth from any of their real deal sighting cards that have not been revealed by the end of the game (after 5 rounds).",
        "The response should explain how the four different sighting card types work. For example,\r\n1. Decoys: Are objects arranged to look just like you! Use them to trick Hunters and earn Myth.\r\n2. Hoaxes: Enlist the help of woodland creatures to de ... [TRUNCATED 239 chars] ... lly you! Expose yourself attacking, sleeping, or engaged in your favorite activity to the right Humans and get rewarded with Myth for your bravery. But beware, you could lose a lot of Myth if the wrong Humans are at the Site of the Encounter.",
        "The response should state that each player has nine sighting cards in Twisted Cryptids.",
        "The response should state that at the start of the game each player forms 3 stacks of sighting cards that correspond to each player's 3 hiding spot tokens they placed on the board.",
        "... [TRUNCATED 4 more items] ..."
      ],
      "id": 0,
      "group_id": 0,
      "metadata": {
        "task_id": "71a2cd92-6978-4ea8-a37f-d99728129d89",
        "context_category": "Rule System Application",
        "sub_category": "Game Mechanics"
      }
    },
    "subset": "default",
    "truncated": false
  },
  "readme": {
    "en": "# CL-bench\n\n\n## Overview\n\nCL-bench represents a step towards building LMs with this fundamental capability (Context Learning), making them more intelligent and advancing their deployment in real-world scenarios. This benchmark is specifically designed to evaluate a model's ability to learn specific, often novel or long-tail knowledge directly from the context and apply it to solve problems, simulating real-world learning processes.\n\n**Resources:**\n[Homepage](https://github.com/Tencent-Hunyuan/CL-bench) | [Dataset](https://huggingface.co/datasets/tencent/CL-bench)\n\n## Task Description\n\n- **Task Type**: In-Context Learning & Reasoning (Context-dependent QA)\n- **Input**: A context containing new rules, fictional information, or specific logic, followed by a query\n- **Output**: A solution derived strictly from the provided context (not pre-trained knowledge)\n- **Difficulty**: Varied, requiring understanding of novel concepts defined in the prompt\n\n## Key Features\n\n- **Contamination-Free**: Uses synthetic or highly specific data (e.g., fictional laws, new programming syntax) to ensure models cannot rely on memorized training data\n- **Real-world Simulation**: Mimics scenarios where humans learn new tasks by reading instructions or documentation\n- **Diverse Domains**: Covers logic reasoning, rule following, language understanding, and puzzle solving within a given context\n- **Evaluation of Adaptability**: Measures the \"learning\" capability rather than just \"retrieval\" or \"memory\"\n\n## Evaluation Notes\n\n- Focuses on the model's ability to follow strict instructions provided in the context\n- Evaluation typically checks if the reasoning process utilizes the unique information given in the prompt\n- Answers are often evaluated against specific ground-truth rules defined in the context\n- Crucial for assessing how well models can adapt to private data or dynamic environments without fine-tuning\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `cl_bench` |\n| **Dataset ID** | [tencent-community/CL-bench](https://modelscope.cn/datasets/tencent-community/CL-bench/summary) |\n| **Paper** | N/A |\n| **Tags** | `InstructionFollowing`, `Reasoning` |\n| **Metrics** | `acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `train` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 1,899 |\n| Prompt Length (Mean) | 42491.73 chars |\n| Prompt Length (Min/Max) | 6595 / 320655 chars |\n\n## Sample Example\n\n**Subset**: `default`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"b0a2c972\",\n      \"content\": \"You are an AI designed to play the Twisted Cryptids board Game developed by Unstable Unicorns. Your purpose is to teach rules, host/referee a session, simulate AI opponents if requested, track game state, and provide strategy advice for the g ... [TRUNCATED 861 chars] ...  brief rationale, then a single short quip).\\n\\nYou take the role of an eccentric conspiracy theorist when talking to the user, but constrain your tone to brief, optional quips that never obscure or delay rules, turn summaries, or move prompts.\",\n      \"role\": \"system\"\n    },\n    {\n      \"id\": \"109b41de\",\n      \"content\": \"RULE BOOK\\nTM\\nTM\\nIt’s so hard to feel seen as a Cryptid these days. In this easy-to-learn strategy game,\\nyou’ll navigate your love-hate relationship with humans and outwit your fellow Cryptids\\nto earn your status as a true legend. As hikers, h ... [TRUNCATED 158305 chars] ...  game\\n02:23:53.880 designers what the gloi cryp it is to be fair we still have a dark horse because\\n02:24:00.560 uh the worm is still underway yeah and who knows how gloopy they will\\n\\nThis is my first time playing, what do Sightings Cards do?\",\n      \"role\": \"user\"\n    }\n  ],\n  \"target\": [\n    \"The response should define what a Sighting card is and its role when playing Twisted Cryptids. For example, Sighting cards are revealed to award/penalize Myth and trigger effects that may move Humans.\",\n    \"The response should name the four different sighting card types, namely: Decoys, Hoaxes, Silhouettes, and Real Deals.\",\n    \"The response should state the requirements for when sighting cards can be revealed: for example, during the dusk phase of each round, during an encounter at a site in the wilderness, when a player has at least 1 hiding spot at the encounter site.\",\n    \"The response should explain how sighting cards are scored once revealed. For example, by gaining or losing myth that depends on the distribution of human types present at the site of the encounter, as specified on the sighting card.\",\n    \"The response should explain the following gameplay sequence in order, which happens when a sighting is going to be revealed: 1. reveal the top face down sighting card in the corresponding stack; 2. apply myth gain/loss shown on the card; 3. r ... [TRUNCATED 161 chars] ... lace it on its corresponding stack (or if that sighting reveal emptied the stack, return the hiding spot token to the box for the rest of the game); 6. the next player who has a hiding spot at the site of the encounter reveals their sighting.\",\n    \"The response should state that players can look at their own sighting cards at any time, but other player's sightings remain hidden until revealed.\",\n    \"The response should state that a player can gain 7 Myth from any of their real deal sighting cards that have not been revealed by the end of the game (after 5 rounds).\",\n    \"The response should explain how the four different sighting card types work. For example,\\r\\n1. Decoys: Are objects arranged to look just like you! Use them to trick Hunters and earn Myth.\\r\\n2. Hoaxes: Enlist the help of woodland creatures to de ... [TRUNCATED 239 chars] ... lly you! Expose yourself attacking, sleeping, or engaged in your favorite activity to the right Humans and get rewarded with Myth for your bravery. But beware, you could lose a lot of Myth if the wrong Humans are at the Site of the Encounter.\",\n    \"The response should state that each player has nine sighting cards in Twisted Cryptids.\",\n    \"The response should state that at the start of the game each player forms 3 stacks of sighting cards that correspond to each player's 3 hiding spot tokens they placed on the board.\",\n    \"... [TRUNCATED 4 more items] ...\"\n  ],\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"task_id\": \"71a2cd92-6978-4ea8-a37f-d99728129d89\",\n    \"context_category\": \"Rule System Application\",\n    \"sub_category\": \"Game Mechanics\"\n  }\n}\n```\n\n## Prompt Template\n\n*No prompt template defined.*\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets cl_bench \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['cl_bench'],\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# CL-bench\n\n## 概述\n\nCL-bench 旨在推动构建具备上下文学习（Context Learning）这一基础能力的大语言模型，使其更加智能，并促进其在现实场景中的部署。该基准测试专门用于评估模型能否直接从上下文中学习特定的、通常是新颖或长尾的知识，并将其应用于解决问题，从而模拟真实世界中的学习过程。\n\n**资源：**  \n[主页](https://github.com/Tencent-Hunyuan/CL-bench) | [数据集](https://huggingface.co/datasets/tencent/CL-bench)\n\n## 任务描述\n\n- **任务类型**：上下文学习与推理（上下文相关的问答）\n- **输入**：包含新规则、虚构信息或特定逻辑的上下文，后接一个查询\n- **输出**：严格基于所提供上下文得出的答案（而非依赖预训练知识）\n- **难度**：多样，要求理解提示中定义的新颖概念\n\n## 核心特性\n\n- **无污染性**：使用合成数据或高度特定的数据（例如虚构法律、新编程语法），确保模型无法依赖记忆中的训练数据\n- **真实场景模拟**：模拟人类通过阅读说明或文档来学习新任务的情境\n- **多领域覆盖**：涵盖在给定上下文内的逻辑推理、规则遵循、语言理解和谜题求解\n- **适应性评估**：衡量模型的“学习”能力，而不仅仅是“检索”或“记忆”能力\n\n## 评估说明\n\n- 重点关注模型是否能严格遵循上下文中提供的指令\n- 评估通常检查推理过程是否利用了提示中给出的独特信息\n- 答案通常根据上下文中定义的具体真实规则进行评判\n- 对评估模型在无需微调的情况下适应私有数据或动态环境的能力至关重要\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `cl_bench` |\n| **数据集ID** | [tencent-community/CL-bench](https://modelscope.cn/datasets/tencent-community/CL-bench/summary) |\n| **论文** | 无 |\n| **标签** | `InstructionFollowing`, `Reasoning` |\n| **指标** | `acc` |\n| **默认示例数** | 0-shot |\n| **评估分割** | `train` |\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 1,899 |\n| 提示词长度（平均） | 42491.73 字符 |\n| 提示词长度（最小/最大） | 6595 / 320655 字符 |\n\n## 样例示例\n\n**子集**：`default`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"b0a2c972\",\n      \"content\": \"You are an AI designed to play the Twisted Cryptids board Game developed by Unstable Unicorns. Your purpose is to teach rules, host/referee a session, simulate AI opponents if requested, track game state, and provide strategy advice for the g ... [TRUNCATED 861 chars] ...  brief rationale, then a single short quip).\\n\\nYou take the role of an eccentric conspiracy theorist when talking to the user, but constrain your tone to brief, optional quips that never obscure or delay rules, turn summaries, or move prompts.\",\n      \"role\": \"system\"\n    },\n    {\n      \"id\": \"109b41de\",\n      \"content\": \"RULE BOOK\\nTM\\nTM\\nIt’s so hard to feel seen as a Cryptid these days. In this easy-to-learn strategy game,\\nyou’ll navigate your love-hate relationship with humans and outwit your fellow Cryptids\\nto earn your status as a true legend. As hikers, h ... [TRUNCATED 158305 chars] ...  game\\n02:23:53.880 designers what the gloi cryp it is to be fair we still have a dark horse because\\n02:24:00.560 uh the worm is still underway yeah and who knows how gloopy they will\\n\\nThis is my first time playing, what do Sightings Cards do?\",\n      \"role\": \"user\"\n    }\n  ],\n  \"target\": [\n    \"The response should define what a Sighting card is and its role when playing Twisted Cryptids. For example, Sighting cards are revealed to award/penalize Myth and trigger effects that may move Humans.\",\n    \"The response should name the four different sighting card types, namely: Decoys, Hoaxes, Silhouettes, and Real Deals.\",\n    \"The response should state the requirements for when sighting cards can be revealed: for example, during the dusk phase of each round, during an encounter at a site in the wilderness, when a player has at least 1 hiding spot at the encounter site.\",\n    \"The response should explain how sighting cards are scored once revealed. For example, by gaining or losing myth that depends on the distribution of human types present at the site of the encounter, as specified on the sighting card.\",\n    \"The response should explain the following gameplay sequence in order, which happens when a sighting is going to be revealed: 1. reveal the top face down sighting card in the corresponding stack; 2. apply myth gain/loss shown on the card; 3. r ... [TRUNCATED 161 chars] ... lace it on its corresponding stack (or if that sighting reveal emptied the stack, return the hiding spot token to the box for the rest of the game); 6. the next player who has a hiding spot at the site of the encounter reveals their sighting.\",\n    \"The response should state that players can look at their own sighting cards at any time, but other player's sightings remain hidden until revealed.\",\n    \"The response should state that a player can gain 7 Myth from any of their real deal sighting cards that have not been revealed by the end of the game (after 5 rounds).\",\n    \"The response should explain how the four different sighting card types work. For example,\\r\\n1. Decoys: Are objects arranged to look just like you! Use them to trick Hunters and earn Myth.\\r\\n2. Hoaxes: Enlist the help of woodland creatures to de ... [TRUNCATED 239 chars] ... lly you! Expose yourself attacking, sleeping, or engaged in your favorite activity to the right Humans and get rewarded with Myth for your bravery. But beware, you could lose a lot of Myth if the wrong Humans are at the Site of the Encounter.\",\n    \"The response should state that each player has nine sighting cards in Twisted Cryptids.\",\n    \"The response should state that at the start of the game each player forms 3 stacks of sighting cards that correspond to each player's 3 hiding spot tokens they placed on the board.\",\n    \"... [TRUNCATED 4 more items] ...\"\n  ],\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"task_id\": \"71a2cd92-6978-4ea8-a37f-d99728129d89\",\n    \"context_category\": \"Rule System Application\",\n    \"sub_category\": \"Game Mechanics\"\n  }\n}\n```\n\n## 提示模板\n\n*未定义提示模板。*\n\n## 使用方法\n\n### 使用命令行接口（CLI）\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets cl_bench \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['cl_bench'],\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "7e04a3a0df9a8ad8090f23b533f9f423",
    "needs_translation": false
  },
  "updated_at": "2026-02-07T10:36:33.238019",
  "translation_updated_at": "2026-02-07T10:36:51Z"
}