117 lines
4.2 KiB
Markdown
117 lines
4.2 KiB
Markdown
# MiniMax-Vendor-Verifier
|
|
|
|
|
|
## Overview
|
|
|
|
MiniMax-Vendor-Verifier is a multi-validator deployment-correctness check for MiniMax M2 / M2.5 / M2.7 vendors. Each prompt row carries an optional ``check_type`` tag that routes it through specific validators, plus an always-on ``error_only_reasoning`` detector for the most common deployment regression. Adapted from [MiniMax-Provider-Verifier](https://github.com/MiniMax-AI/MiniMax-Provider-Verifier).
|
|
|
|
## Task Description
|
|
|
|
- **Task Type**: Vendor-deployment correctness check (multi-dimensional)
|
|
- **Input**: Multi-turn chat messages with optional tool definitions, plus per-row routing tags (``check_type``, ``expected_tool_call``)
|
|
- **Output**: Vendor's chat-completion response, scored against the validator(s) selected for that row
|
|
- **Dispatch**: Rows without ``check_type`` default to the ``tool_calls`` validator; rows with ``check_type`` run only the listed validators
|
|
|
|
## Key Features
|
|
|
|
- Five upstream validators ported as pure functions:
|
|
- ``tool_calls`` — JSON-schema validation of arguments + array-command soundness check, plus a confusion matrix over ``expected_tool_call``
|
|
- ``error_only_reasoning`` (always-on) — flags responses with reasoning but no content and no tool calls (a deployment regression)
|
|
- ``contains_russian_characters_unicode`` — language-following check; fails when Cyrillic codepoints leak into the response
|
|
- ``repeat_n_gram`` — degenerate-repetition detector (any 3-gram appearing 4 or more times)
|
|
- ``scenario_check`` — verifies the model preserves the declared JSON property order, catching providers that re-sort ``parameters.properties``
|
|
- Per-validator denominator in the report: ``num=0`` indicates no row in the subset triggered that validator (not a failure)
|
|
- Hosted dataset preserves the upstream sample.jsonl plus per-loop baseline traces for M2.5 / M2.7
|
|
|
|
## Evaluation Notes
|
|
|
|
- Default configuration uses **0-shot** evaluation; the ``default`` subset has 102 rows
|
|
- Metrics: **tool_calls_match_rate**, **schema_accuracy**, **error_only_reasoning_rate**, **language_following_success_rate**, **repeat_ngram_pass_rate**, **scenario_check_pass_rate**
|
|
- Per upstream guidance, a correctly-deployed vendor should hit ``tool_calls_match_rate ≈ 0.98``, ``schema_accuracy ≥ 0.98``, ``error_only_reasoning_rate = 0``, and ``scenario_check_pass_rate = 1.0``
|
|
- When using ``--limit``, the rarer ``check_type`` rows (scenario / repeat / language) may not all be sampled; check the per-validator ``num`` column
|
|
|
|
|
|
## Properties
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **Benchmark Name** | `minimax_verifier` |
|
|
| **Dataset ID** | [evalscope/MiniMaxVendorVerifier](https://modelscope.cn/datasets/evalscope/MiniMaxVendorVerifier/summary) |
|
|
| **Paper** | N/A |
|
|
| **Tags** | `Agent`, `FunctionCalling` |
|
|
| **Metrics** | `tool_calls_match_rate`, `schema_accuracy`, `error_only_reasoning_rate`, `language_following_success_rate`, `repeat_ngram_pass_rate`, `scenario_check_pass_rate` |
|
|
| **Default Shots** | 0-shot |
|
|
| **Evaluation Split** | `test` |
|
|
|
|
|
|
## Data Statistics
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Total Samples | 102 |
|
|
| Prompt Length (Mean) | 72251.38 chars |
|
|
| Prompt Length (Min/Max) | 16 / 341252 chars |
|
|
|
|
## Sample Example
|
|
|
|
**Subset**: `default`
|
|
|
|
```json
|
|
{
|
|
"input": [
|
|
{
|
|
"id": "6cc50a79",
|
|
"content": "日本ではどのような時にお年玉を渡しますか？",
|
|
"role": "user"
|
|
}
|
|
],
|
|
"target": "",
|
|
"id": 0,
|
|
"group_id": 0,
|
|
"tools": [],
|
|
"metadata": {
|
|
"check_type": [
|
|
"contains_russian_characters_unicode"
|
|
],
|
|
"expected_tool_call": null,
|
|
"tools_raw": []
|
|
}
|
|
}
|
|
```
|
|
|
|
## Prompt Template
|
|
|
|
*No prompt template defined.*
|
|
|
|
## Usage
|
|
|
|
### Using CLI
|
|
|
|
```bash
|
|
evalscope eval \
|
|
--model YOUR_MODEL \
|
|
--api-url OPENAI_API_COMPAT_URL \
|
|
--api-key EMPTY_TOKEN \
|
|
--datasets minimax_verifier \
|
|
--limit 10 # Remove this line for formal evaluation
|
|
```
|
|
|
|
### Using Python
|
|
|
|
```python
|
|
from evalscope import run_task
|
|
from evalscope.config import TaskConfig
|
|
|
|
task_cfg = TaskConfig(
|
|
model='YOUR_MODEL',
|
|
api_url='OPENAI_API_COMPAT_URL',
|
|
api_key='EMPTY_TOKEN',
|
|
datasets=['minimax_verifier'],
|
|
limit=10, # Remove this line for formal evaluation
|
|
)
|
|
|
|
run_task(task_cfg=task_cfg)
|
|
```
|
|
|
|
|