{
  "meta": {
    "pretty_name": "C-Eval",
    "dataset_id": "evalscope/ceval",
    "paper_url": null,
    "tags": [
      "Knowledge",
      "MCQ",
      "Chinese"
    ],
    "metrics": [
      "acc"
    ],
    "few_shot_num": 5,
    "eval_split": "val",
    "train_split": "dev",
    "subset_list": [
      "computer_network",
      "operating_system",
      "computer_architecture",
      "college_programming",
      "college_physics",
      "college_chemistry",
      "advanced_mathematics",
      "probability_and_statistics",
      "discrete_mathematics",
      "electrical_engineer",
      "metrology_engineer",
      "high_school_mathematics",
      "high_school_physics",
      "high_school_chemistry",
      "high_school_biology",
      "middle_school_mathematics",
      "middle_school_biology",
      "middle_school_physics",
      "middle_school_chemistry",
      "veterinary_medicine",
      "college_economics",
      "business_administration",
      "marxism",
      "mao_zedong_thought",
      "education_science",
      "teacher_qualification",
      "high_school_politics",
      "high_school_geography",
      "middle_school_politics",
      "middle_school_geography",
      "modern_chinese_history",
      "ideological_and_moral_cultivation",
      "logic",
      "law",
      "chinese_language_and_literature",
      "art_studies",
      "professional_tour_guide",
      "legal_professional",
      "high_school_chinese",
      "high_school_history",
      "middle_school_history",
      "civil_servant",
      "sports_science",
      "plant_protection",
      "basic_medicine",
      "clinical_medicine",
      "urban_and_rural_planner",
      "accountant",
      "fire_engineer",
      "environmental_impact_assessment_engineer",
      "tax_accountant",
      "physician"
    ],
    "description": "\n## Overview\n\nC-Eval is a comprehensive Chinese evaluation benchmark designed to assess the knowledge and reasoning abilities of language models in Chinese. It covers 52 subjects ranging from STEM to humanities and social sciences, with questions from middle school to professional examination levels.\n\n## Task Description\n\n- **Task Type**: Multiple-Choice Question Answering (Chinese)\n- **Input**: Chinese question with four answer choices (A, B, C, D)\n- **Output**: Single correct answer letter\n- **Subjects**: 52 subjects organized into 4 categories (STEM, Social Science, Humanities, Other)\n\n## Key Features\n\n- 13,948 multiple-choice questions across 52 subjects\n- Questions sourced from Chinese middle school, high school, college, and professional exams\n- Covers diverse domains including mathematics, physics, law, medicine, and more\n- Includes explanations for validation split questions\n- Standard benchmark for Chinese language model evaluation\n\n## Evaluation Notes\n\n- Default configuration uses **5-shot** examples from the dev split\n- Questions and prompts are in Chinese\n- Answers should follow the format: \"答案：[LETTER]\"\n- Results can be aggregated by subject or category\n- Use `subset_list` parameter to evaluate specific subjects\n",
    "prompt_template": "以下是中国关于{subject}的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\"答案：[LETTER]\"（不带引号），其中 [LETTER] 是 A、B、C、D 中的一个。\n\n问题：{question}\n选项：\n{choices}\n",
    "system_prompt": "",
    "few_shot_prompt_template": "以下是一些示例问题：\n\n{fewshot}\n\n",
    "aggregation": "mean",
    "extra_params": {},
    "sandbox_config": {},
    "category": "llm"
  },
  "statistics": {
    "total_samples": 1346,
    "subset_stats": [
      {
        "name": "computer_network",
        "sample_count": 19,
        "prompt_length_mean": 1245.42,
        "prompt_length_min": 1201,
        "prompt_length_max": 1313,
        "prompt_length_std": 36.51,
        "target_length_mean": 1
      },
      {
        "name": "operating_system",
        "sample_count": 19,
        "prompt_length_mean": 1216.16,
        "prompt_length_min": 1187,
        "prompt_length_max": 1282,
        "prompt_length_std": 24.52,
        "target_length_mean": 1
      },
      {
        "name": "computer_architecture",
        "sample_count": 21,
        "prompt_length_mean": 1654.67,
        "prompt_length_min": 1622,
        "prompt_length_max": 1732,
        "prompt_length_std": 32.79,
        "target_length_mean": 1
      },
      {
        "name": "college_programming",
        "sample_count": 37,
        "prompt_length_mean": 1745.03,
        "prompt_length_min": 1660,
        "prompt_length_max": 2189,
        "prompt_length_std": 119.01,
        "target_length_mean": 1
      },
      {
        "name": "college_physics",
        "sample_count": 19,
        "prompt_length_mean": 2071.16,
        "prompt_length_min": 1986,
        "prompt_length_max": 2184,
        "prompt_length_std": 50.61,
        "target_length_mean": 1
      },
      {
        "name": "college_chemistry",
        "sample_count": 24,
        "prompt_length_mean": 2152.96,
        "prompt_length_min": 2107,
        "prompt_length_max": 2291,
        "prompt_length_std": 51.23,
        "target_length_mean": 1
      },
      {
        "name": "advanced_mathematics",
        "sample_count": 19,
        "prompt_length_mean": 6271.68,
        "prompt_length_min": 6130,
        "prompt_length_max": 6605,
        "prompt_length_std": 120.05,
        "target_length_mean": 1
      },
      {
        "name": "probability_and_statistics",
        "sample_count": 18,
        "prompt_length_mean": 4700.39,
        "prompt_length_min": 4527,
        "prompt_length_max": 4987,
        "prompt_length_std": 155.05,
        "target_length_mean": 1
      },
      {
        "name": "discrete_mathematics",
        "sample_count": 16,
        "prompt_length_mean": 1176.75,
        "prompt_length_min": 1104,
        "prompt_length_max": 1365,
        "prompt_length_std": 73.72,
        "target_length_mean": 1
      },
      {
        "name": "electrical_engineer",
        "sample_count": 37,
        "prompt_length_mean": 1137.81,
        "prompt_length_min": 1093,
        "prompt_length_max": 1264,
        "prompt_length_std": 37.14,
        "target_length_mean": 1
      },
      {
        "name": "metrology_engineer",
        "sample_count": 24,
        "prompt_length_mean": 1187.08,
        "prompt_length_min": 1140,
        "prompt_length_max": 1312,
        "prompt_length_std": 42.17,
        "target_length_mean": 1
      },
      {
        "name": "high_school_mathematics",
        "sample_count": 18,
        "prompt_length_mean": 2749.11,
        "prompt_length_min": 2670,
        "prompt_length_max": 2894,
        "prompt_length_std": 66.35,
        "target_length_mean": 1
      },
      {
        "name": "high_school_physics",
        "sample_count": 19,
        "prompt_length_mean": 1365,
        "prompt_length_min": 1263,
        "prompt_length_max": 1538,
        "prompt_length_std": 64.22,
        "target_length_mean": 1
      },
      {
        "name": "high_school_chemistry",
        "sample_count": 19,
        "prompt_length_mean": 1625.79,
        "prompt_length_min": 1536,
        "prompt_length_max": 1739,
        "prompt_length_std": 60.37,
        "target_length_mean": 1
      },
      {
        "name": "high_school_biology",
        "sample_count": 19,
        "prompt_length_mean": 1159.95,
        "prompt_length_min": 1103,
        "prompt_length_max": 1243,
        "prompt_length_std": 38.43,
        "target_length_mean": 1
      },
      {
        "name": "middle_school_mathematics",
        "sample_count": 19,
        "prompt_length_mean": 2141.68,
        "prompt_length_min": 2054,
        "prompt_length_max": 2413,
        "prompt_length_std": 90.3,
        "target_length_mean": 1
      },
      {
        "name": "middle_school_biology",
        "sample_count": 21,
        "prompt_length_mean": 1822.52,
        "prompt_length_min": 1769,
        "prompt_length_max": 1910,
        "prompt_length_std": 38.48,
        "target_length_mean": 1
      },
      {
        "name": "middle_school_physics",
        "sample_count": 19,
        "prompt_length_mean": 1596.53,
        "prompt_length_min": 1544,
        "prompt_length_max": 1721,
        "prompt_length_std": 42.61,
        "target_length_mean": 1
      },
      {
        "name": "middle_school_chemistry",
        "sample_count": 20,
        "prompt_length_mean": 1916.65,
        "prompt_length_min": 1857,
        "prompt_length_max": 2045,
        "prompt_length_std": 43.66,
        "target_length_mean": 1
      },
      {
        "name": "veterinary_medicine",
        "sample_count": 23,
        "prompt_length_mean": 1254.04,
        "prompt_length_min": 1210,
        "prompt_length_max": 1350,
        "prompt_length_std": 42.61,
        "target_length_mean": 1
      },
      {
        "name": "college_economics",
        "sample_count": 55,
        "prompt_length_mean": 1919.85,
        "prompt_length_min": 1863,
        "prompt_length_max": 2141,
        "prompt_length_std": 51.05,
        "target_length_mean": 1
      },
      {
        "name": "business_administration",
        "sample_count": 33,
        "prompt_length_mean": 1608.52,
        "prompt_length_min": 1547,
        "prompt_length_max": 1754,
        "prompt_length_std": 55.37,
        "target_length_mean": 1
      },
      {
        "name": "marxism",
        "sample_count": 19,
        "prompt_length_mean": 1061.32,
        "prompt_length_min": 1033,
        "prompt_length_max": 1104,
        "prompt_length_std": 21.55,
        "target_length_mean": 1
      },
      {
        "name": "mao_zedong_thought",
        "sample_count": 24,
        "prompt_length_mean": 1485.62,
        "prompt_length_min": 1449,
        "prompt_length_max": 1545,
        "prompt_length_std": 27.53,
        "target_length_mean": 1
      },
      {
        "name": "education_science",
        "sample_count": 29,
        "prompt_length_mean": 1369.24,
        "prompt_length_min": 1338,
        "prompt_length_max": 1444,
        "prompt_length_std": 27.07,
        "target_length_mean": 1
      },
      {
        "name": "teacher_qualification",
        "sample_count": 44,
        "prompt_length_mean": 1516.55,
        "prompt_length_min": 1457,
        "prompt_length_max": 1638,
        "prompt_length_std": 37.68,
        "target_length_mean": 1
      },
      {
        "name": "high_school_politics",
        "sample_count": 19,
        "prompt_length_mean": 2062,
        "prompt_length_min": 1955,
        "prompt_length_max": 2189,
        "prompt_length_std": 64.75,
        "target_length_mean": 1
      },
      {
        "name": "high_school_geography",
        "sample_count": 19,
        "prompt_length_mean": 1059.53,
        "prompt_length_min": 1026,
        "prompt_length_max": 1216,
        "prompt_length_std": 44.28,
        "target_length_mean": 1
      },
      {
        "name": "middle_school_politics",
        "sample_count": 21,
        "prompt_length_mean": 1653.24,
        "prompt_length_min": 1595,
        "prompt_length_max": 1714,
        "prompt_length_std": 36.11,
        "target_length_mean": 1
      },
      {
        "name": "middle_school_geography",
        "sample_count": 12,
        "prompt_length_mean": 1063.58,
        "prompt_length_min": 1020,
        "prompt_length_max": 1143,
        "prompt_length_std": 38.78,
        "target_length_mean": 1
      },
      {
        "name": "modern_chinese_history",
        "sample_count": 23,
        "prompt_length_mean": 1355.7,
        "prompt_length_min": 1313,
        "prompt_length_max": 1447,
        "prompt_length_std": 32.16,
        "target_length_mean": 1
      },
      {
        "name": "ideological_and_moral_cultivation",
        "sample_count": 19,
        "prompt_length_mean": 760.21,
        "prompt_length_min": 727,
        "prompt_length_max": 830,
        "prompt_length_std": 24.3,
        "target_length_mean": 1
      },
      {
        "name": "logic",
        "sample_count": 22,
        "prompt_length_mean": 2436.41,
        "prompt_length_min": 2358,
        "prompt_length_max": 2572,
        "prompt_length_std": 63.83,
        "target_length_mean": 1
      },
      {
        "name": "law",
        "sample_count": 24,
        "prompt_length_mean": 1799.29,
        "prompt_length_min": 1729,
        "prompt_length_max": 1931,
        "prompt_length_std": 50.17,
        "target_length_mean": 1
      },
      {
        "name": "chinese_language_and_literature",
        "sample_count": 23,
        "prompt_length_mean": 954.83,
        "prompt_length_min": 937,
        "prompt_length_max": 983,
        "prompt_length_std": 14.83,
        "target_length_mean": 1
      },
      {
        "name": "art_studies",
        "sample_count": 33,
        "prompt_length_mean": 793.3,
        "prompt_length_min": 774,
        "prompt_length_max": 844,
        "prompt_length_std": 12.98,
        "target_length_mean": 1
      },
      {
        "name": "professional_tour_guide",
        "sample_count": 29,
        "prompt_length_mean": 924.41,
        "prompt_length_min": 902,
        "prompt_length_max": 1004,
        "prompt_length_std": 21.91,
        "target_length_mean": 1
      },
      {
        "name": "legal_professional",
        "sample_count": 23,
        "prompt_length_mean": 2856.17,
        "prompt_length_min": 2718,
        "prompt_length_max": 2978,
        "prompt_length_std": 75.58,
        "target_length_mean": 1
      },
      {
        "name": "high_school_chinese",
        "sample_count": 19,
        "prompt_length_mean": 2295.79,
        "prompt_length_min": 2205,
        "prompt_length_max": 2418,
        "prompt_length_std": 55.48,
        "target_length_mean": 1
      },
      {
        "name": "high_school_history",
        "sample_count": 20,
        "prompt_length_mean": 1221.9,
        "prompt_length_min": 1164,
        "prompt_length_max": 1300,
        "prompt_length_std": 40.21,
        "target_length_mean": 1
      },
      {
        "name": "middle_school_history",
        "sample_count": 22,
        "prompt_length_mean": 1069.73,
        "prompt_length_min": 1034,
        "prompt_length_max": 1149,
        "prompt_length_std": 27.08,
        "target_length_mean": 1
      },
      {
        "name": "civil_servant",
        "sample_count": 47,
        "prompt_length_mean": 1973,
        "prompt_length_min": 1849,
        "prompt_length_max": 2186,
        "prompt_length_std": 87.68,
        "target_length_mean": 1
      },
      {
        "name": "sports_science",
        "sample_count": 19,
        "prompt_length_mean": 1810.26,
        "prompt_length_min": 1789,
        "prompt_length_max": 1874,
        "prompt_length_std": 22.55,
        "target_length_mean": 1
      },
      {
        "name": "plant_protection",
        "sample_count": 22,
        "prompt_length_mean": 1678.09,
        "prompt_length_min": 1653,
        "prompt_length_max": 1745,
        "prompt_length_std": 19.36,
        "target_length_mean": 1
      },
      {
        "name": "basic_medicine",
        "sample_count": 19,
        "prompt_length_mean": 938.05,
        "prompt_length_min": 920,
        "prompt_length_max": 976,
        "prompt_length_std": 14.91,
        "target_length_mean": 1
      },
      {
        "name": "clinical_medicine",
        "sample_count": 22,
        "prompt_length_mean": 1119.41,
        "prompt_length_min": 1086,
        "prompt_length_max": 1209,
        "prompt_length_std": 33.2,
        "target_length_mean": 1
      },
      {
        "name": "urban_and_rural_planner",
        "sample_count": 46,
        "prompt_length_mean": 1428.8,
        "prompt_length_min": 1373,
        "prompt_length_max": 1591,
        "prompt_length_std": 55.04,
        "target_length_mean": 1
      },
      {
        "name": "accountant",
        "sample_count": 49,
        "prompt_length_mean": 1605.92,
        "prompt_length_min": 1511,
        "prompt_length_max": 1808,
        "prompt_length_std": 70.35,
        "target_length_mean": 1
      },
      {
        "name": "fire_engineer",
        "sample_count": 31,
        "prompt_length_mean": 1240.81,
        "prompt_length_min": 1168,
        "prompt_length_max": 1402,
        "prompt_length_std": 60.16,
        "target_length_mean": 1
      },
      {
        "name": "environmental_impact_assessment_engineer",
        "sample_count": 31,
        "prompt_length_mean": 1269.97,
        "prompt_length_min": 1209,
        "prompt_length_max": 1388,
        "prompt_length_std": 43.24,
        "target_length_mean": 1
      },
      {
        "name": "tax_accountant",
        "sample_count": 49,
        "prompt_length_mean": 1970.65,
        "prompt_length_min": 1879,
        "prompt_length_max": 2099,
        "prompt_length_std": 58.35,
        "target_length_mean": 1
      },
      {
        "name": "physician",
        "sample_count": 49,
        "prompt_length_mean": 1010.59,
        "prompt_length_min": 983,
        "prompt_length_max": 1065,
        "prompt_length_std": 19.74,
        "target_length_mean": 1
      }
    ],
    "prompt_length": {
      "mean": 1643.61,
      "min": 727,
      "max": 6605,
      "std": 804.87
    },
    "target_length_mean": 1,
    "computed_at": "2026-01-28T11:13:30.036665"
  },
  "sample_example": {
    "data": {
      "input": [
        {
          "id": "73073a35",
          "content": "以下是一些示例问题：\n\n问题：下列设备属于资源子网的是____。\n选项：\nA. 计算机软件\nB. 网桥\nC. 交换机\nD. 路由器\n解析：1. 首先，资源子网是指提供共享资源的网络，如打印机、文件服务器等。\r\n2. 其次，我们需要了解选项中设备的功能。网桥、交换机和路由器的主要功能是实现不同网络之间的通信和数据传输，是通信子网设备。而计算机软件可以提供共享资源的功能。\n答案：A\n\n问题：滑动窗口的作用是____。\n选项：\nA. 流量控制\nB. 拥塞控制\nC. 路由控制\nD. 差错 ... [TRUNCATED] ... Mbps，所以答案为min{80Mbps, 100Mbps}=80Mbps，选C。\n答案：C\n\n\n以下是中国关于计算机网络的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\"答案：[LETTER]\"（不带引号），其中 [LETTER] 是 A、B、C、D 中的一个。\n\n问题：使用位填充方法，以01111110为位首flag，数据为011011111111111111110010，求问传送时要添加几个0____\n选项：\nA. 1\nB. 2\nC. 3\nD. 4\n"
        }
      ],
      "choices": [
        "1",
        "2",
        "3",
        "4"
      ],
      "target": "C",
      "id": 0,
      "group_id": 0,
      "metadata": {
        "id": 0,
        "explanation": "",
        "subject": "computer_network"
      }
    },
    "subset": "computer_network",
    "truncated": true
  },
  "readme": {
    "en": "# C-Eval\n\n\n## Overview\n\nC-Eval is a comprehensive Chinese evaluation benchmark designed to assess the knowledge and reasoning abilities of language models in Chinese. It covers 52 subjects ranging from STEM to humanities and social sciences, with questions from middle school to professional examination levels.\n\n## Task Description\n\n- **Task Type**: Multiple-Choice Question Answering (Chinese)\n- **Input**: Chinese question with four answer choices (A, B, C, D)\n- **Output**: Single correct answer letter\n- **Subjects**: 52 subjects organized into 4 categories (STEM, Social Science, Humanities, Other)\n\n## Key Features\n\n- 13,948 multiple-choice questions across 52 subjects\n- Questions sourced from Chinese middle school, high school, college, and professional exams\n- Covers diverse domains including mathematics, physics, law, medicine, and more\n- Includes explanations for validation split questions\n- Standard benchmark for Chinese language model evaluation\n\n## Evaluation Notes\n\n- Default configuration uses **5-shot** examples from the dev split\n- Questions and prompts are in Chinese\n- Answers should follow the format: \"答案：[LETTER]\"\n- Results can be aggregated by subject or category\n- Use `subset_list` parameter to evaluate specific subjects\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `ceval` |\n| **Dataset ID** | [evalscope/ceval](https://modelscope.cn/datasets/evalscope/ceval/summary) |\n| **Paper** | N/A |\n| **Tags** | `Chinese`, `Knowledge`, `MCQ` |\n| **Metrics** | `acc` |\n| **Default Shots** | 5-shot |\n| **Evaluation Split** | `val` |\n| **Train Split** | `dev` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 1,346 |\n| Prompt Length (Mean) | 1643.61 chars |\n| Prompt Length (Min/Max) | 727 / 6605 chars |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `computer_network` | 19 | 1245.42 | 1201 | 1313 |\n| `operating_system` | 19 | 1216.16 | 1187 | 1282 |\n| `computer_architecture` | 21 | 1654.67 | 1622 | 1732 |\n| `college_programming` | 37 | 1745.03 | 1660 | 2189 |\n| `college_physics` | 19 | 2071.16 | 1986 | 2184 |\n| `college_chemistry` | 24 | 2152.96 | 2107 | 2291 |\n| `advanced_mathematics` | 19 | 6271.68 | 6130 | 6605 |\n| `probability_and_statistics` | 18 | 4700.39 | 4527 | 4987 |\n| `discrete_mathematics` | 16 | 1176.75 | 1104 | 1365 |\n| `electrical_engineer` | 37 | 1137.81 | 1093 | 1264 |\n| `metrology_engineer` | 24 | 1187.08 | 1140 | 1312 |\n| `high_school_mathematics` | 18 | 2749.11 | 2670 | 2894 |\n| `high_school_physics` | 19 | 1365 | 1263 | 1538 |\n| `high_school_chemistry` | 19 | 1625.79 | 1536 | 1739 |\n| `high_school_biology` | 19 | 1159.95 | 1103 | 1243 |\n| `middle_school_mathematics` | 19 | 2141.68 | 2054 | 2413 |\n| `middle_school_biology` | 21 | 1822.52 | 1769 | 1910 |\n| `middle_school_physics` | 19 | 1596.53 | 1544 | 1721 |\n| `middle_school_chemistry` | 20 | 1916.65 | 1857 | 2045 |\n| `veterinary_medicine` | 23 | 1254.04 | 1210 | 1350 |\n| `college_economics` | 55 | 1919.85 | 1863 | 2141 |\n| `business_administration` | 33 | 1608.52 | 1547 | 1754 |\n| `marxism` | 19 | 1061.32 | 1033 | 1104 |\n| `mao_zedong_thought` | 24 | 1485.62 | 1449 | 1545 |\n| `education_science` | 29 | 1369.24 | 1338 | 1444 |\n| `teacher_qualification` | 44 | 1516.55 | 1457 | 1638 |\n| `high_school_politics` | 19 | 2062 | 1955 | 2189 |\n| `high_school_geography` | 19 | 1059.53 | 1026 | 1216 |\n| `middle_school_politics` | 21 | 1653.24 | 1595 | 1714 |\n| `middle_school_geography` | 12 | 1063.58 | 1020 | 1143 |\n| `modern_chinese_history` | 23 | 1355.7 | 1313 | 1447 |\n| `ideological_and_moral_cultivation` | 19 | 760.21 | 727 | 830 |\n| `logic` | 22 | 2436.41 | 2358 | 2572 |\n| `law` | 24 | 1799.29 | 1729 | 1931 |\n| `chinese_language_and_literature` | 23 | 954.83 | 937 | 983 |\n| `art_studies` | 33 | 793.3 | 774 | 844 |\n| `professional_tour_guide` | 29 | 924.41 | 902 | 1004 |\n| `legal_professional` | 23 | 2856.17 | 2718 | 2978 |\n| `high_school_chinese` | 19 | 2295.79 | 2205 | 2418 |\n| `high_school_history` | 20 | 1221.9 | 1164 | 1300 |\n| `middle_school_history` | 22 | 1069.73 | 1034 | 1149 |\n| `civil_servant` | 47 | 1973 | 1849 | 2186 |\n| `sports_science` | 19 | 1810.26 | 1789 | 1874 |\n| `plant_protection` | 22 | 1678.09 | 1653 | 1745 |\n| `basic_medicine` | 19 | 938.05 | 920 | 976 |\n| `clinical_medicine` | 22 | 1119.41 | 1086 | 1209 |\n| `urban_and_rural_planner` | 46 | 1428.8 | 1373 | 1591 |\n| `accountant` | 49 | 1605.92 | 1511 | 1808 |\n| `fire_engineer` | 31 | 1240.81 | 1168 | 1402 |\n| `environmental_impact_assessment_engineer` | 31 | 1269.97 | 1209 | 1388 |\n| `tax_accountant` | 49 | 1970.65 | 1879 | 2099 |\n| `physician` | 49 | 1010.59 | 983 | 1065 |\n\n## Sample Example\n\n**Subset**: `computer_network`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"73073a35\",\n      \"content\": \"以下是一些示例问题：\\n\\n问题：下列设备属于资源子网的是____。\\n选项：\\nA. 计算机软件\\nB. 网桥\\nC. 交换机\\nD. 路由器\\n解析：1. 首先，资源子网是指提供共享资源的网络，如打印机、文件服务器等。\\r\\n2. 其次，我们需要了解选项中设备的功能。网桥、交换机和路由器的主要功能是实现不同网络之间的通信和数据传输，是通信子网设备。而计算机软件可以提供共享资源的功能。\\n答案：A\\n\\n问题：滑动窗口的作用是____。\\n选项：\\nA. 流量控制\\nB. 拥塞控制\\nC. 路由控制\\nD. 差错 ... [TRUNCATED] ... Mbps，所以答案为min{80Mbps, 100Mbps}=80Mbps，选C。\\n答案：C\\n\\n\\n以下是中国关于计算机网络的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\\\"答案：[LETTER]\\\"（不带引号），其中 [LETTER] 是 A、B、C、D 中的一个。\\n\\n问题：使用位填充方法，以01111110为位首flag，数据为011011111111111111110010，求问传送时要添加几个0____\\n选项：\\nA. 1\\nB. 2\\nC. 3\\nD. 4\\n\"\n    }\n  ],\n  \"choices\": [\n    \"1\",\n    \"2\",\n    \"3\",\n    \"4\"\n  ],\n  \"target\": \"C\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"id\": 0,\n    \"explanation\": \"\",\n    \"subject\": \"computer_network\"\n  }\n}\n```\n\n*Note: Some content was truncated for display.*\n\n## Prompt Template\n\n**Prompt Template:**\n```text\n以下是中国关于{subject}的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\"答案：[LETTER]\"（不带引号），其中 [LETTER] 是 A、B、C、D 中的一个。\n\n问题：{question}\n选项：\n{choices}\n\n```\n\n<details>\n<summary>Few-shot Template</summary>\n\n```text\n以下是一些示例问题：\n\n{fewshot}\n\n\n```\n\n</details>\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets ceval \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['ceval'],\n    dataset_args={\n        'ceval': {\n            # subset_list: ['computer_network', 'operating_system', 'computer_architecture']  # optional, evaluate specific subsets\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# C-Eval\n\n\n## 概述\n\nC-Eval 是一个全面的中文评估基准，旨在评估语言模型在中文语境下的知识与推理能力。该基准涵盖 52 个学科，范围从 STEM（科学、技术、工程和数学）到人文与社会科学，题目难度覆盖初中至专业资格考试水平。\n\n## 任务描述\n\n- **任务类型**：多项选择题问答（中文）\n- **输入**：一道包含四个选项（A、B、C、D）的中文问题\n- **输出**：单个正确答案字母\n- **学科分类**：52 个学科，分为 4 大类（STEM、社会科学、人文学科、其他）\n\n## 主要特点\n\n- 共计 13,948 道多选题，覆盖 52 个学科\n- 题目来源包括中国初中、高中、大学及专业资格考试\n- 涵盖数学、物理、法律、医学等多个领域\n- 验证集（validation split）题目附带解析\n- 是中文语言模型评估的标准基准之一\n\n## 评估说明\n\n- 默认配置使用来自开发集（dev split）的 **5-shot** 示例\n- 所有问题和提示均为中文\n- 答案格式应为：\"答案：[LETTER]\"\n- 结果可按学科或大类进行汇总\n- 可通过 `subset_list` 参数指定评估特定学科\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `ceval` |\n| **数据集ID** | [evalscope/ceval](https://modelscope.cn/datasets/evalscope/ceval/summary) |\n| **论文** | N/A |\n| **标签** | `Chinese`, `Knowledge`, `MCQ` |\n| **指标** | `acc` |\n| **默认示例数** | 5-shot |\n| **评估集** | `val` |\n| **训练集** | `dev` |\n\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 1,346 |\n| 提示词长度（平均） | 1643.61 字符 |\n| 提示词长度（最小/最大） | 727 / 6605 字符 |\n\n**各子集统计数据：**\n\n| 子集 | 样本数 | 提示平均长度 | 提示最小长度 | 提示最大长度 |\n|--------|---------|-------------|------------|------------|\n| `computer_network` | 19 | 1245.42 | 1201 | 1313 |\n| `operating_system` | 19 | 1216.16 | 1187 | 1282 |\n| `computer_architecture` | 21 | 1654.67 | 1622 | 1732 |\n| `college_programming` | 37 | 1745.03 | 1660 | 2189 |\n| `college_physics` | 19 | 2071.16 | 1986 | 2184 |\n| `college_chemistry` | 24 | 2152.96 | 2107 | 2291 |\n| `advanced_mathematics` | 19 | 6271.68 | 6130 | 6605 |\n| `probability_and_statistics` | 18 | 4700.39 | 4527 | 4987 |\n| `discrete_mathematics` | 16 | 1176.75 | 1104 | 1365 |\n| `electrical_engineer` | 37 | 1137.81 | 1093 | 1264 |\n| `metrology_engineer` | 24 | 1187.08 | 1140 | 1312 |\n| `high_school_mathematics` | 18 | 2749.11 | 2670 | 2894 |\n| `high_school_physics` | 19 | 1365 | 1263 | 1538 |\n| `high_school_chemistry` | 19 | 1625.79 | 1536 | 1739 |\n| `high_school_biology` | 19 | 1159.95 | 1103 | 1243 |\n| `middle_school_mathematics` | 19 | 2141.68 | 2054 | 2413 |\n| `middle_school_biology` | 21 | 1822.52 | 1769 | 1910 |\n| `middle_school_physics` | 19 | 1596.53 | 1544 | 1721 |\n| `middle_school_chemistry` | 20 | 1916.65 | 1857 | 2045 |\n| `veterinary_medicine` | 23 | 1254.04 | 1210 | 1350 |\n| `college_economics` | 55 | 1919.85 | 1863 | 2141 |\n| `business_administration` | 33 | 1608.52 | 1547 | 1754 |\n| `marxism` | 19 | 1061.32 | 1033 | 1104 |\n| `mao_zedong_thought` | 24 | 1485.62 | 1449 | 1545 |\n| `education_science` | 29 | 1369.24 | 1338 | 1444 |\n| `teacher_qualification` | 44 | 1516.55 | 1457 | 1638 |\n| `high_school_politics` | 19 | 2062 | 1955 | 2189 |\n| `high_school_geography` | 19 | 1059.53 | 1026 | 1216 |\n| `middle_school_politics` | 21 | 1653.24 | 1595 | 1714 |\n| `middle_school_geography` | 12 | 1063.58 | 1020 | 1143 |\n| `modern_chinese_history` | 23 | 1355.7 | 1313 | 1447 |\n| `ideological_and_moral_cultivation` | 19 | 760.21 | 727 | 830 |\n| `logic` | 22 | 2436.41 | 2358 | 2572 |\n| `law` | 24 | 1799.29 | 1729 | 1931 |\n| `chinese_language_and_literature` | 23 | 954.83 | 937 | 983 |\n| `art_studies` | 33 | 793.3 | 774 | 844 |\n| `professional_tour_guide` | 29 | 924.41 | 902 | 1004 |\n| `legal_professional` | 23 | 2856.17 | 2718 | 2978 |\n| `high_school_chinese` | 19 | 2295.79 | 2205 | 2418 |\n| `high_school_history` | 20 | 1221.9 | 1164 | 1300 |\n| `middle_school_history` | 22 | 1069.73 | 1034 | 1149 |\n| `civil_servant` | 47 | 1973 | 1849 | 2186 |\n| `sports_science` | 19 | 1810.26 | 1789 | 1874 |\n| `plant_protection` | 22 | 1678.09 | 1653 | 1745 |\n| `basic_medicine` | 19 | 938.05 | 920 | 976 |\n| `clinical_medicine` | 22 | 1119.41 | 1086 | 1209 |\n| `urban_and_rural_planner` | 46 | 1428.8 | 1373 | 1591 |\n| `accountant` | 49 | 1605.92 | 1511 | 1808 |\n| `fire_engineer` | 31 | 1240.81 | 1168 | 1402 |\n| `environmental_impact_assessment_engineer` | 31 | 1269.97 | 1209 | 1388 |\n| `tax_accountant` | 49 | 1970.65 | 1879 | 2099 |\n| `physician` | 49 | 1010.59 | 983 | 1065 |\n\n## 样例示例\n\n**子集**: `computer_network`\n\n```json\n{\n  \"input\": [\n    {\n      \"id\": \"73073a35\",\n      \"content\": \"以下是一些示例问题：\\n\\n问题：下列设备属于资源子网的是____。\\n选项：\\nA. 计算机软件\\nB. 网桥\\nC. 交换机\\nD. 路由器\\n解析：1. 首先，资源子网是指提供共享资源的网络，如打印机、文件服务器等。\\r\\n2. 其次，我们需要了解选项中设备的功能。网桥、交换机和路由器的主要功能是实现不同网络之间的通信和数据传输，是通信子网设备。而计算机软件可以提供共享资源的功能。\\n答案：A\\n\\n问题：滑动窗口的作用是____。\\n选项：\\nA. 流量控制\\nB. 拥塞控制\\nC. 路由控制\\nD. 差错 ... [TRUNCATED] ... Mbps，所以答案为min{80Mbps, 100Mbps}=80Mbps，选C。\\n答案：C\\n\\n\\n以下是中国关于计算机网络的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\\\"答案：[LETTER]\\\"（不带引号），其中 [LETTER] 是 A、B、C、D 中的一个。\\n\\n问题：使用位填充方法，以01111110为位首flag，数据为011011111111111111110010，求问传送时要添加几个0____\\n选项：\\nA. 1\\nB. 2\\nC. 3\\nD. 4\\n\"\n    }\n  ],\n  \"choices\": [\n    \"1\",\n    \"2\",\n    \"3\",\n    \"4\"\n  ],\n  \"target\": \"C\",\n  \"id\": 0,\n  \"group_id\": 0,\n  \"metadata\": {\n    \"id\": 0,\n    \"explanation\": \"\",\n    \"subject\": \"computer_network\"\n  }\n}\n```\n\n*注：部分内容因展示需要已被截断。*\n\n## 提示模板\n\n**提示模板：**\n```text\n以下是中国关于{subject}的单项选择题，请选出对的选项。你的回答的最后一行应该是这样的格式：\"答案：[LETTER]\"（不带引号），其中 [LETTER] 是 A、B、C、D 中的一个。\n\n问题：{question}\n选项：\n{choices}\n\n```\n\n<details>\n<summary>少样本（Few-shot）模板</summary>\n\n```text\n以下是一些示例问题：\n\n{fewshot}\n\n\n```\n\n</details>\n\n## 使用方法\n\n### 使用命令行（CLI）\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets ceval \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['ceval'],\n    dataset_args={\n        'ceval': {\n            # subset_list: ['computer_network', 'operating_system', 'computer_architecture']  # 可选，用于评估特定子集\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "ba3a022d5cf53c633532b501aec54fe3",
    "needs_translation": false
  },
  "updated_at": "2026-01-28T17:31:32.194461",
  "translation_updated_at": "2026-01-28T16:09:53Z"
}