evalstone/evalscope/docs/en/benchmarks/toolathlon.md

Toolathlon Official Service Wrapper

Overview

Toolathlon is an agent benchmark for realistic, long-horizon tool use across many MCP-backed software environments. This EvalScope benchmark is a wrapper around the official Toolathlon remote evaluation service, not a local reimplementation of the MCP environments or official evaluator.

Task Description

• Task Type: Long-horizon agent tool use with multi-turn function calling
• Input: Realistic software tasks selected from the Toolathlon-Verified task set
• Output: Model responses and tool calls executed by the official Toolathlon agent loop
• Domain: MCP-backed productivity, development, data, and web software environments

Key Features

• Uses the official Toolathlon service for MCP environments, task containers, agent execution, and scoring
• Supports official-service private mode with a local or intranet OpenAI-compatible model endpoint
• Represents one remote Toolathlon job as one EvalScope sample, while task_list and limit select tasks in the job
• Retries transient polling and result-download failures without cancelling the remote evaluation
• Keeps failed archive downloads pending for later retries and validates downloaded output paths
• Keeps the model API key on the EvalScope side and passes it to the local WebSocket relay through the environment
• Includes the Toolathlon-Verified task list inspected from official repository commit b7bbac3f9a1f381b095c878debe1a47dd164ad85

Evaluation Notes

• The primary metric is accuracy reported by the official Toolathlon scorer
• Aggregate scores are read from official statistics, including average_success_rate, with count and per-task fallbacks for compatible service outputs
• Missing, non-finite, or out-of-range scores fail validation instead of being silently reported as zero
• EvalScope evaluation semantics are versioned as v1.1; the wrapper follows Toolathlon service protocol 1.3
• Runtime requires httpx, websockets, and an OpenAI-compatible endpoint reachable from the EvalScope process
• The shared public service has queue and IP-based usage limits; use a dedicated or self-hosted service for sustained evaluation

Usage Guide

See the Toolathlon usage guide for public-service limits, private-mode data flow, self-hosted service setup, and EvalScope configuration examples:

• https://evalscope.readthedocs.io/en/latest/third_party/toolathlon.html

Official sources:

• https://github.com/hkust-nlp/Toolathlon
• https://github.com/hkust-nlp/Toolathlon/blob/main/EVAL_SERVICE_README.md

Properties

Property	Value
Benchmark Name	toolathlon
Dataset ID	Toolathlon
Paper	N/A
Tags	Agent, FunctionCalling, MultiTurn
Metrics	accuracy
Default Shots	0-shot
Evaluation Split	test

Data Statistics

Metric	Value
Total Samples	1
Prompt Length (Mean)	50 chars
Prompt Length (Min/Max)	50 / 50 chars

Sample Example

Subset: default

{
  "input": [
    {
      "id": "ba8702fc",
      "content": "Run Toolathlon official remote evaluation service."
    }
  ],
  "target": "",
  "id": 0,
  "metadata": {
    "task_list": [
      "ab-testing",
      "academic-pdf-report",
      "academic-warning",
      "add-bibtex",
      "apply-phd-email",
      "arrange-workspace",
      "canvas-arrange-exam",
      "canvas-art-manager",
      "canvas-art-quiz",
      "canvas-do-quiz",
      "... [TRUNCATED 98 more items] ..."
    ],
    "mode": "private"
  }
}

Note: Some content was truncated for display.

Prompt Template

Prompt Template:

{question}

Extra Parameters

Parameter	Type	Default	Description
mode	str	private	Toolathlon service mode. EvalScope supports private mode for this wrapper. Choices: ['private']
server_host	str	47.253.6.47	Official Toolathlon evaluation service host.
server_port	int	8080	Official Toolathlon HTTP service port.
ws_proxy_port	int	8081	Official Toolathlon WebSocket proxy port for private mode.
workers	int	10	Number of parallel Toolathlon workers requested from the official service.
provider	str	unified	Toolathlon model provider type. Choices: ['unified', 'openai_stateful_responses']
task_list	list	[]	Optional Toolathlon task names to evaluate. Empty uses the bundled Toolathlon-Verified list.
task_list_file	str	``	Optional file containing one Toolathlon task name per line.
model_params	dict	{}	Extra model parameters forwarded to Toolathlon, merged after TaskConfig.generation_config.
job_id	str	``	Optional Toolathlon job id. Reuse to resume an incomplete official-service job.
force_redownload	bool	False	Force redownload of Toolathlon result archives.
override_output_dir	bool	False	Clear the Toolathlon output directory when it already contains files.
skip_container_restart	bool	False	Skip Toolathlon container restart. Use only for small debug task subsets.
trust_env_in_httpx	bool	False	Allow httpx to use proxy environment variables.
timeout_seconds	int	14400	Maximum time to wait for a Toolathlon official-service job.
poll_interval	int	5	Polling interval in seconds for Toolathlon job status.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets toolathlon \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['toolathlon'],
    dataset_args={
        'toolathlon': {
            # extra_params: {}  # uses default extra parameters
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)