MiniMax-Vendor-Verifier

Overview

MiniMax-Vendor-Verifier is a multi-validator deployment-correctness check for MiniMax M2 / M2.5 / M2.7 vendors. Each prompt row carries an optional check_type tag that routes it through specific validators, plus an always-on error_only_reasoning detector for the most common deployment regression. Adapted from MiniMax-Provider-Verifier.

Task Description

Task Type: Vendor-deployment correctness check (multi-dimensional)
Input: Multi-turn chat messages with optional tool definitions, plus per-row routing tags (check_type, expected_tool_call)
Output: Vendor's chat-completion response, scored against the validator(s) selected for that row
Dispatch: Rows without check_type default to the tool_calls validator; rows with check_type run only the listed validators

Key Features

Five upstream validators ported as pure functions:
- tool_calls — JSON-schema validation of arguments + array-command soundness check, plus a confusion matrix over expected_tool_call
- error_only_reasoning (always-on) — flags responses with reasoning but no content and no tool calls (a deployment regression)
- contains_russian_characters_unicode — language-following check; fails when Cyrillic codepoints leak into the response
- repeat_n_gram — degenerate-repetition detector (any 3-gram appearing 4 or more times)
- scenario_check — verifies the model preserves the declared JSON property order, catching providers that re-sort parameters.properties
Per-validator denominator in the report: num=0 indicates no row in the subset triggered that validator (not a failure)
Hosted dataset preserves the upstream sample.jsonl plus per-loop baseline traces for M2.5 / M2.7

Evaluation Notes

Default configuration uses 0-shot evaluation; the default subset has 102 rows
Metrics: tool_calls_match_rate, schema_accuracy, error_only_reasoning_rate, language_following_success_rate, repeat_ngram_pass_rate, scenario_check_pass_rate
Per upstream guidance, a correctly-deployed vendor should hit tool_calls_match_rate ≈ 0.98, schema_accuracy ≥ 0.98, error_only_reasoning_rate = 0, and scenario_check_pass_rate = 1.0
When using --limit, the rarer check_type rows (scenario / repeat / language) may not all be sampled; check the per-validator num column

Properties

Property	Value
Benchmark Name	`minimax_verifier`
Dataset ID	evalscope/MiniMaxVendorVerifier
Paper	N/A
Tags	`Agent`, `FunctionCalling`
Metrics	`tool_calls_match_rate`, `schema_accuracy`, `error_only_reasoning_rate`, `language_following_success_rate`, `repeat_ngram_pass_rate`, `scenario_check_pass_rate`
Default Shots	0-shot
Evaluation Split	`test`

Data Statistics

Metric	Value
Total Samples	102
Prompt Length (Mean)	72251.38 chars
Prompt Length (Min/Max)	16 / 341252 chars

Sample Example

Subset: default

{
  "input": [
    {
      "id": "6cc50a79",
      "content": "日本ではどのような時にお年玉を渡しますか？",
      "role": "user"
    }
  ],
  "target": "",
  "id": 0,
  "group_id": 0,
  "tools": [],
  "metadata": {
    "check_type": [
      "contains_russian_characters_unicode"
    ],
    "expected_tool_call": null,
    "tools_raw": []
  }
}

Prompt Template

No prompt template defined.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets minimax_verifier \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['minimax_verifier'],
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)

4.2 KiB Raw Blame History