# 👍 Contribute Benchmark

EvalScope, as the official evaluation tool of [ModelScope](https://modelscope.cn), is continuously optimizing its benchmark evaluation features! We invite you to refer to this tutorial to easily add your own benchmark and share your contributions with the community. Let's work together to enhance EvalScope and make our tool even better!

Below, we will introduce how to add two types of benchmark evaluations: **General Text Reasoning** and **Multiple Choice**, which mainly include three steps: uploading the dataset, registering the dataset, and writing the evaluation task.

## Basic Concepts

```{tip}
You can skip this section and start directly from [Preparing Benchmark Evaluation Dataset](#1-preparing-benchmark-evaluation-dataset). Refer back to the specific implementation when encountering code you don't understand.
```

The evaluation process of EvalScope mainly includes the following steps:
1. **Data Preparation**: Load and preprocess the dataset through `DataAdapter`.
2. **Task Definition**: Define the configuration of the evaluation task through `TaskConfig`, including models, datasets, evaluation metrics, etc.
3. **Evaluation Execution**: Execute the evaluation task through the `run_task` function and output the evaluation results.

Among them, `DataAdapter` is the core component of benchmark evaluation that we need to focus on.

### DataAdapter Architecture and Call Flow

DataAdapter adopts a Pipeline architecture, supporting custom behavior through hook methods. Taking `DefaultDataAdapter` as an example, the complete evaluation process is as follows:

```
1. Data Loading Phase
   load_dataset() 
   ├── load() 
   │   ├── load_from_remote() / load_from_disk()
   │   │   ├── load_subsets()
   │   │   │   └── load_subset() / load_fewshot_subset()
   │   │   │       └── record_to_sample() [User Implementation]
   │   │   └── _post_process_samples()
   │   │       └── process_sample_input()
   │   │           ├── sample_to_fewshot() [User Implementation]
   │   │           ├── format_fewshot_template() [Optional User Implementation]
   │   │           └── format_prompt_template() [Optional User Implementation]
   │   └── Returns DatasetDict

2. Model Inference Phase (Per Sample)
   run_inference()
   ├── _on_inference_start() [Hook Method]
   ├── _on_inference() [Hook Method]
   └── _on_inference_end() [Hook Method]
       └── Returns TaskState

3. Metric Calculation Phase (Per Sample)
   calculate_metrics()
   ├── filter_prediction()
   │   └── extract_answer() [Optional User Implementation]
   ├── match_score() / score_with_judge_contracts()
   └── Returns SampleScore

4. Result Aggregation Phase
   aggregate_scores()
   └── Returns List[AggScore]

5. Report Generation Phase
   generate_report()
   ├── _on_generate_report() [Hook Method]
   └── _on_generate_report_end() [Hook Method]
       └── Returns Report
```

### Adding an LLM-Judged Benchmark

Users enable judging with the typed `judge` configuration described in [Judge Parameters](../get_started/parameters.md#judge-parameters):

```python
TaskConfig(
    model='MODEL_UNDER_TEST',
    datasets=['your_benchmark'],
    judge={
        'strategy': 'llm',
        'models': {'model_id': 'JUDGE_MODEL', 'api_url': 'OPENAI_COMPATIBLE_URL', 'api_key': 'API_KEY'},
    },
)
```

For an adapter author, judge I/O belongs to `evalscope.api.judge`; do not call `self.llm_judge.judge()` or parse a model reply in the adapter.

1. Declare `scoring_policy`: use `JUDGE_ONLY` when rule scoring is not meaningful, `JUDGE_DEFAULT` when rules remain available but `auto` should judge, and `RULE_DEFAULT` when `auto` should keep rule scoring.
2. Implement the single adapter entry point, `judge_definition(context)`. For an ordinary one-verdict task, return `JudgeDefinition.labels(...)` or `JudgeDefinition.numeric(...)` with a Pydantic verdict schema. These helpers append the JSON output instruction and keep the prompt, schema, and metric mapping together.
3. For a rubric, multiple claims, or staged task, define a Pydantic verdict schema and `OutputContract`, then return `JudgeDefinition.workflow(cases=..., request=..., reduce=...)`. Pass `expand=...`, `fallback=...`, or `finalize=...` only when the workflow needs them. The callbacks can be nested in `judge_definition()` or private adapter helpers, but they must be owned by the returned definition rather than exposed as adapter hooks. Append `case.output_contract.instruction()` in a custom request unless the official fixed JSON instruction exactly matches that schema. Use `CaseVerdict.metadata` rather than encoding state into `case_id`.
4. When a deterministic rule settles a sample without a model call, return `JudgeDefinition.skip(score, reason='...')`. `reason` is required and is persisted as `Score.metadata['judge_skip_reason']` with `Score.metadata['judge_skipped'] = True`; the web review panel labels this as rule-based scoring rather than an LLM verdict.
5. Add a scripted-judge test covering a valid JSON verdict, malformed/prose output, and a transport error. Invalid judge replies are excluded from the metric; they are not converted to a zero score and are not automatically retried by the executor.

The executor owns request dispatch, position swaps, repeats, multi-judge quorum, aggregation, and review diagnostics. The model transport owns its own retry policy through `generation_config`.

### Core Data Structures

#### 1. Sample Object
Represents a single evaluation sample, including input, target answer, and metadata:

```python
@dataclass
class Sample:
    input: Any                    # Input content (question text or list of chat messages)
    target: str                   # Target answer (correct answer)
    choices: Optional[List[str]] = None    # Choices (used for multiple choice questions)
    subset_key: Optional[str] = None       # Subset division key (used for grouping by category)
    metadata: Optional[Dict] = None        # Metadata (reasoning process, ID, etc.)
    tools: Optional[List] = None           # Tool call information
```

#### 2. TaskState Object
Represents the complete state of a single inference task:

```python
@dataclass
class TaskState:
    model: str                    # Model name
    sample: Sample               # Input sample
    messages: List[ChatMessage]  # Chat message history
    output: ModelOutput          # Model raw output
    completed: bool              # Whether the task is completed
    sample_id: Optional[str] = None      # Sample ID
    group_id: Optional[str] = None       # Group ID
    metadata: Optional[Dict] = None      # Task metadata
```

#### 3. ModelOutput Object
Represents the raw output of the model:

```python
@dataclass
class ModelOutput:
    completion: str              # Text generated by the model
    message: ChatMessage         # Formatted chat message
    # Other model-specific fields...
```

#### 4. Score Object
Represents the scoring result of a single sample:

```python
class Score(BaseModel):
    value: Dict[str, int | float | bool] = Field(default_factory=dict)  # E.g. {"accuracy": 1.0}
    extracted_prediction: Optional[str] = None
    prediction: Optional[str] = None
    explanation: Optional[str] = None
    metadata: Optional[Dict] = Field(default_factory=dict)
    main_score_name: Optional[str] = None  # Selects one value within this sample only
```

#### 5. SampleScore Object
Encapsulates the complete scoring information of a single sample:

```python
class SampleScore(BaseModel):
    score: Score
    sample_id: Optional[str | int] = None
    group_id: Optional[str | int] = None
    sample_metadata: Optional[Dict] = None
```

#### 6. AggScore Object
Represents aggregated scoring statistics:

```python
class AggScore(BaseModel):
    score: float = 0.0
    metric_name: str = ''        # Canonical measured concept, e.g. "accuracy"
    aggregation: str = 'identity'
    dimensions: Dict[str, str | int | float | bool] = Field(default_factory=dict)
    num: int = 0
    ids: Optional[List[str | int]] = None
    metadata: Optional[Dict] = None
```

#### 7. DatasetDict Object
Manages multiple dataset subsets:

```python
class DatasetDict(dict):
    """Dataset dictionary, keys are subset names, values are Dataset objects"""
    
    @classmethod
    def from_dataset(cls, dataset, subset_list=None, limit=None, repeats=1):
        """Create a multi-subset dataset dictionary from a single dataset"""
        pass
```

### Core Methods of DataAdapter

Based on the above call flow, here are the key methods that need to be implemented by the user or can be optionally overridden:

#### Methods That Must Be Implemented

1. **`record_to_sample(record: Dict[str, Any]) -> Sample`**
   - **Purpose**: Convert raw data records into standard Sample objects
   - **Input**: Raw record dictionary from the dataset
   - **Output**: Standardized Sample object
   - **Example**:
   ```python
   def record_to_sample(self, record: Dict[str, Any]) -> Sample:
       return Sample(
           input=record['question'],
           target=record['answer'],
           metadata={'reasoning': record.get('explanation', '')}
       )
   ```

#### Methods That Can Be Optionally Implemented

2. **`sample_to_fewshot(sample: Sample) -> str`**
   - **Purpose**: Convert sample into a few-shot example string
   - **Input**: Sample object
   - **Output**: Formatted few-shot example text
   - **Call Timing**: When constructing few-shot prompts

3. **`extract_answer(prediction: str, task_state: TaskState) -> str`**
   - **Purpose**: Extract the final answer from the model's raw output
   - **Input**: Model prediction text and task state
   - **Output**: Extracted answer string
   - **Call Timing**: Before calculating metrics for answer cleaning

4. **`format_prompt_template(sample: Sample) -> str`**
   - **Purpose**: Format the basic prompt template
   - **Input**: Sample object
   - **Output**: Formatted prompt text
   - **Default Implementation**: Uses `prompt_template.format(question=sample.input)`

5. **`format_fewshot_template(fewshot: str, sample: Sample) -> str`**
   - **Purpose**: Format the prompt template containing few-shot examples
   - **Input**: Few-shot example string and Sample object
   - **Output**: Complete few-shot prompt
   - **Default Implementation**: Uses `few_shot_prompt_template.format()`

6. **`sample_filter(sample: Sample) -> bool`**
   - **Purpose**: Filter dataset samples
   - **Input**: Sample object
   - **Output**: Whether to retain the sample
   - **Default Implementation**: Returns True (retains all samples)

### Hook Method System

DataAdapter provides a hook method system, supporting custom logic insertion at key points:

#### Inference Phase Hooks
- **`_on_inference_start(model, sample)`**: Before inference starts
- **`_on_inference(model, sample)`**: Execute inference
- **`_on_inference_end(model, sample, model_output, output_dir)`**: After inference ends

#### Report Generation Hooks
- **`_on_generate_report(scores, model_name)`**: Generate report
- **`_on_generate_report_end(report, output_dir)`**: After report generation

### Adapter Types

EvalScope provides two main adapter base classes:

1. **`DefaultDataAdapter`**: Basic adapter for general text reasoning tasks
   - Suitable for open-ended question answering, mathematical reasoning, code generation, etc.
   - Requires custom answer extraction logic

2. **`MultiChoiceAdapter`**: Specialized adapter for multiple choice tasks
   - Inherits from `DefaultDataAdapter`
   - Built-in choice formatting and answer extraction logic
   - Supports single-choice and multiple-choice modes

Principles for choosing adapter types:
- If the task involves selecting answers from fixed options → Use `MultiChoiceAdapter`
- If the task requires generating open-ended answers → Use `DefaultDataAdapter`

## 1. Preparing Benchmark Evaluation Dataset

You have two ways to prepare the benchmark evaluation dataset:

1. **Upload to ModelScope (Recommended)**: Upload the dataset to the ModelScope platform, so other users can easily load your dataset, making it more convenient to use and benefiting more users from your contribution. If you need to upload to ModelScope, refer to the [Dataset Upload Tutorial](https://www.modelscope.cn/docs/datasets/create).

2. **Local Use**: You can also directly use the local dataset for evaluation, suitable for datasets that are still in development or contain sensitive information.

Regardless of the method chosen, ensure that the data format is correct and can be loaded. If using a local dataset, you can test with the following code:

```python
from modelscope import MsDataset

dataset = MsDataset.load("/path/to/your/dataset")  # Replace with your dataset
```

## 2. Creating File Structure

First, [Fork EvalScope](https://github.com/modelscope/evalscope/fork) repository, i.e., create your own EvalScope repository copy, and clone it locally.

```bash
git clone https://github.com/your_username/evalscope.git
cd evalscope
```

Then, add benchmark evaluation in the `evalscope/benchmarks/` directory, with the structure as follows:

```text
evalscope/benchmarks/
├── benchmark_name
│   ├── __init__.py
│   ├── benchmark_name_adapter.py
│   └── ...
```
Specifically for `GSM8K` and `MMLU-Pro`, the structure is as follows:

```text
evalscope/benchmarks/
├── gsm8k
│   ├── __init__.py
│   ├── gsm8k_adapter.py
├── mmlu_pro
│   ├── __init__.py
│   ├── mmlu_pro_adapter.py
│   └── ...
```

## 3. Writing Evaluation Logic

Below, we will take **GSM8K** and **MMLU-Pro** as examples to introduce two types of evaluation tasks: **General Text Reasoning** and **Multiple Choice**.

### General Text Reasoning

General text reasoning tasks usually require the model to analyze and reason about the given problem and then generate an answer. Taking GSM8K (mathematical reasoning) as an example:

We need to register `Benchmark` and implement the `GSM8KAdapter` class in `gsm8k_adapter.py`:

```python
from typing import Any, Dict
from evalscope.api.benchmark import BenchmarkMeta, DefaultDataAdapter
from evalscope.api.dataset import Sample
from evalscope.api.evaluator import TaskState
from evalscope.api.registry import register_benchmark
from evalscope.constants import Tags

# Define prompt template
PROMPT_TEMPLATE = """
Solve the following math problem step by step. The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem.

{question}

Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \\boxed command.

Reasoning:
""".lstrip()

# Register benchmark evaluation
@register_benchmark(
    BenchmarkMeta(
        name='gsm8k',                          # Benchmark test name
        pretty_name='GSM8K',                   # Readable name
        dataset_id='AI-ModelScope/gsm8k',      # Dataset ID or local path
        tags=[Tags.MATH, Tags.REASONING],      # Tags
        description='GSM8K (Grade School Math 8K) is a dataset of grade school math problems, designed to evaluate the mathematical reasoning abilities of AI models.',
        subset_list=['main'],                  # Subset list
        few_shot_num=4,                       # Few-shot example number
        train_split='train',                  # Training set split name
        eval_split='test',                    # Evaluation set split name
        metric_list=['accuracy'],             # Canonical evaluation metrics
        prompt_template=PROMPT_TEMPLATE,      # Prompt template
    )
)
class GSM8KAdapter(DefaultDataAdapter):
    
    def record_to_sample(self, record: Dict[str, Any]) -> Sample:
        """Convert raw data records into Sample objects"""
        DELIM = '####'
        question = record['question']
        answer = record['answer'].split(DELIM)
        target = answer.pop().strip()  # Extract final answer
        reasoning = DELIM.join(answer)  # Extract reasoning process
        
        return Sample(
            input=question,
            target=target,
            metadata={'reasoning': reasoning.strip()}
        )
    
    def sample_to_fewshot(self, sample: Sample) -> str:
        """Convert sample into few-shot example"""
        if sample.metadata:
            return (
                f'{sample.input}\n\nReasoning:\n' + 
                f"{sample.metadata['reasoning']}\n\n" + 
                f'ANSWER: {sample.target}'
            )
        else:
            return ''
    
    def extract_answer(self, prediction: str, task_state: TaskState):
        """Extract answer from model prediction"""
        from evalscope.filters.extraction import RegexFilter
        
        # Use regular expression to extract numeric answer
        regex = RegexFilter(regex_pattern=r'(-?[0-9.,]{2,})|(-?[0-9]+)', group_select=-1)
        res = regex(prediction)
        return res.replace(',', '').replace('+', '').strip().strip('.')
```

### Multiple Choice

Multiple choice tasks require the model to select the correct answer from given options. Taking MMLU-Pro as an example, we need to inherit `MultiChoiceAdapter`:

```python
from typing import Any, Dict
from evalscope.api.benchmark import BenchmarkMeta, MultiChoiceAdapter
from evalscope.api.dataset import Sample
from evalscope.api.registry import register_benchmark
from evalscope.constants import Tags

# Define prompt template
USER_PROMPT_TEMPLATE = """Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.

Question:
{question}
Options:
{choices}
""".lstrip()

SUBSET_LIST = [
    'computer science', 'math', 'chemistry', 'engineering', 'law', 'biology', 
    'health', 'physics', 'business', 'philosophy', 'economics', 'other', 
    'psychology', 'history'
]

@register_benchmark(
    BenchmarkMeta(
        name='mmlu_pro',
        pretty_name='MMLU-Pro',
        tags=[Tags.MULTIPLE_CHOICE, Tags.KNOWLEDGE],
        description='MMLU-Pro is a benchmark for evaluating language models on multiple-choice questions across various subjects.',
        dataset_id='modelscope/MMLU-Pro',
        subset_list=SUBSET_LIST,
        metric_list=['accuracy'],
        few_shot_num=5,
        train_split='validation',
        eval_split='test',
        prompt_template=USER_PROMPT_TEMPLATE,
    )
)
class MMLUProAdapter(MultiChoiceAdapter):
    
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.reformat_subset = True  # Enable subset division
    
    def record_to_sample(self, record: Dict[str, Any]) -> Sample:
        """Convert raw data records into Sample objects"""
        return Sample(
            input=record['question'],
            choices=record['options'],      # Choice list
            target=record['answer'],        # Correct answer (e.g., 'A')
            subset_key=record['category'].lower(),  # Key for subset division
            metadata={
                'cot_content': record['cot_content'],
                'subject': record['category'].lower(),
                'question_id': record['question_id'],
            },
        )
    
    def sample_to_fewshot(self, sample: Sample) -> str:
        """Convert sample into few-shot example"""
        q_str = f"""Question:\n{str(sample.input)}"""
        options = sample.choices if sample.choices is not None else []
        
        # Format choices
        opt_str_list = []
        for i, opt in enumerate(options):
            opt_str_list.append(f"""{chr(65 + i)} {opt}""")
        opt_str = f"""Options:\n{'\n'.join(opt_str_list)}"""
        
        # Handle answer and reasoning process
        ans_str = sample.metadata['cot_content'] if sample.metadata is not None else ''
        ans_str = ans_str.replace('The answer is', 'ANSWER:')
        ans_opt = ans_str.split('ANSWER:')[-1].split('.')[0].strip().strip('(').strip(')')
        ans_str = ans_str.replace(f'ANSWER: ({ans_opt})', f'ANSWER: {ans_opt}')
        
        final_str = '\n'.join([q_str, opt_str, ans_str])
        return final_str
```

### Key Differences Explanation

**General Text Reasoning** vs **Multiple Choice**:

1. **Inherited Base Class**:
   - General Text Reasoning: Inherits `DefaultDataAdapter`
   - Multiple Choice: Inherits `MultiChoiceAdapter`

2. **Sample Object Structure**:
   - General Text Reasoning: Mainly includes `input` and `target`
   - Multiple Choice: Additionally includes `choices` (choice list)

3. **Answer Extraction Method**:
   - General Text Reasoning: Requires custom `extract_answer()` method
   - Multiple Choice: `MultiChoiceAdapter` provides standard answer extraction logic

4. **Prompt Template**:
   - General Text Reasoning: Focuses more on guiding the reasoning process
   - Multiple Choice: Focuses on displaying choices and answer format

### Metric Semantics and the Primary Metric

Reports do not guess what a metric means. How a metric is displayed — its name, its optimization
direction, its unit, its scale and its precision — comes from a central catalog at
`evalscope/metrics/semantics/catalog.py`, and each benchmark states which of its metrics carries
the conclusion.

**Most new benchmarks need no catalog change at all.** Reusing an existing canonical metric name
(`accuracy`, `f1`, `exact_match`, `pass_rate`, ...) means the semantics are already declared:

```python
metric_list=['accuracy'],
```

Two cases are worth a line from you:

1. **Your benchmark reports several metrics or several variants of one metric.** Declare exactly
   which emitted identity is primary. A selector may constrain the aggregation and any structured
   dimensions such as `k`, `scope`, or `threshold`:

   ```python
   from evalscope.api.metric.semantics import MetricSelector

   metric_list=['precision', 'recall', 'f1', 'accuracy'],
   primary_metric=MetricSelector(name='f1', aggregation='mean'),
   ```

   A single non-diagnostic identity is implicitly primary. If several non-diagnostic identities
   are emitted, omitting the selector makes report generation fail instead of guessing from list
   order. A selector must match exactly one emitted identity, and its name must be declared in
   `metric_list`.

2. **Your benchmark introduces a new canonical metric name.** Add one line to `METRIC_DEFINITIONS`,
   referencing the baseline that describes it:

   ```python
   # evalscope/metrics/semantics/catalog.py
   METRIC_DEFINITIONS['my_new_score'] = MetricEntry(baseline='quality.accuracy.ratio')
   ```

Keep the naming layers separate:

- `metric_list`, `Score.value`, and custom `AggScore.metric_name` use canonical names such as
  `accuracy`. A small set of legacy aliases is normalized for compatibility, but new adapters
  should not introduce more aliases.
- `AggScore` stores `metric_name`, `aggregation`, and `dimensions` separately. Do not encode
  `mean`, `pass@k`, thresholds, or scopes into the metric name.
- The catalog is keyed by the canonical metric name. Aggregation-specific meaning belongs in
  `AGGREGATION_SEMANTICS`; benchmark-specific name collisions belong in
  `BENCHMARK_METRIC_OVERRIDES`.
- `Score.main_score_name` selects one value in a sample, `BenchmarkMeta.primary_metric` declares the
  report-level primary identity, and `Report.primary_metric_identity` persists that identity.

After changing `primary_metric`, refresh its generated metadata cache with
`make docs-update BENCHMARK="<name>" FORCE=1`; do not edit `_meta/*.json` by hand.

An undeclared metric degrades to a diagnostic, which displays the stored value without claiming a
direction or unit and logs the catalog entry to add. Dynamic variants do not require catalog
enumeration: values such as `k`, question type, threshold, and token range belong in structured
dimensions and share the canonical metric's semantics.

## 4. Running Evaluation

Debug the code to see if it can run normally.

**GSM8K Example**:
```python
from evalscope import run_task, TaskConfig

task_cfg = TaskConfig(
    model='Qwen/Qwen2.5-0.5B-Instruct',
    datasets=['gsm8k'],
    limit=10,
    debug=True
)
run_task(task_cfg=task_cfg)
```

**MMLU-Pro Example**:
```python
from evalscope import run_task, TaskConfig

task_cfg = TaskConfig(
    model='Qwen/Qwen2.5-0.5B-Instruct',
    datasets=['mmlu_pro'],
    limit=10,
    dataset_args={'mmlu_pro': {'subset_list': ['computer science', 'math']}},
    debug=True
)
run_task(task_cfg=task_cfg)
```

Output Example:

```text
+-----------------------+-----------+-----------------+------------------+-------+---------+---------+
| Model                 | Dataset   | Metric          | Subset           |   Num |   Score | Cat.0   |
+=======================+===========+=================+==================+=======+=========+=========+
| Qwen2.5-0.5B-Instruct | gsm8k     | Accuracy ↑      | main             |    10 |     30% | default |
+-----------------------+-----------+-----------------+------------------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | mmlu_pro  | Accuracy ↑      | computer science |    10 |     10% | default |
+-----------------------+-----------+-----------------+------------------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | mmlu_pro  | Accuracy ↑      | math             |    10 |     10% | default |
+-----------------------+-----------+-----------------+------------------+-------+---------+---------+
```

## 5. Benchmark Evaluation Document Generation

After completing the benchmark evaluation implementation, you can use the tools provided by EvalScope to generate standard documents. This will ensure that your benchmark evaluation has a consistent document format and can be easily understood and used by other users.

To generate Chinese and English documents, run the following command, which will generate documents based on registration information:

```bash
pip install -e '.[docs]'
make docs
```

## 6. Submitting PR
After completing the implementation and documentation generation, run all repository checks before submitting a [PR](https://github.com/modelscope/evalscope/pulls). This command applies safe Ruff fixes and formatting before validating the remaining hooks:
```bash
make lint
```
Once the checks pass, your contribution is ready for review. For the complete development workflow, see the [Contributing Guide](https://github.com/modelscope/evalscope/blob/main/CONTRIBUTING.md). Give it a try 🚀