evalstone/evalscope/docs/en/user_guides/agent/native.md

Native AgentLoop Mode

Turn a regular benchmark (GSM8K, AIME, IFEval, SWE-bench, …) into a multi-turn tool-use task to evaluate the model's own tool-use and multi-step reasoning. EvalScope runs generate → parse → call tool → observe → generate on each iteration until the model submits a final answer or hits the step cap. The full trajectory is recorded as an AgentTrace for replay in the UI.

To evaluate an off-the-shelf agent CLI such as Claude Code / Codex, see External Agent Bridge Mode.

When to use

• Evaluate a model's multi-step reasoning and tool-use ability Example: does qwen-plus proactively call python_exec to verify arithmetic on GSM8K? Can it reason step-by-step and self-check on AIME?

• Let a model autonomously explore code-fix tasks Example: on SWE-bench, give the model a bash tool to read code, run tests, and submit patches — no human in the loop.

• Plug in MCP ecosystem tools to expand model capabilities Example: attach fetch, Brave Search, or other MCP servers so the model can search the web when running benchmarks like GAIA.

Quick start

Smallest runnable example — have qwen-plus verify GSM8K answers with python_exec.

from evalscope import TaskConfig, run_task
from evalscope.api.agent import NativeAgentConfig

task_config = TaskConfig(
    model='qwen-plus',
    api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
    api_key='<your-key>',
    eval_type='openai_api',
    datasets=['gsm8k'],
    limit=5,
    generation_config={'parallel_tool_calls': True},
    agent_config=NativeAgentConfig(
        strategy='function_calling',
        tools=['python_exec'],
        environment='docker',
        environment_extra={
            'sandbox_config': {
                'image': 'python:3.11-slim',
            },
        },
        max_steps=5,
        kwargs={'system_prompt': 'Use python_exec to verify your calculations.'},
    ),
)
run_task(task_config)

EvalScope auto-injects the python_exec tool definition into the model's context, executes the generated Python in a Docker container, feeds stdout/stderr back as an observation, and loops until the model uses submit to deliver its answer.

Common configuration

Most-used NativeAgentConfig fields:

Field	Description	Recommended
strategy	Interaction protocol	function_calling (default); use react or swe_bench_backticks if the model lacks function-calling
tools	Tool whitelist	On demand, e.g. ['python_exec'] / ['bash']
environment	Where tools or an external agent run	local (dev) / docker (production)
environment_extra	Agent environment constructor options	Docker image, timeout, mounts, and other environment-specific settings
max_steps	Max iterations per sample	Math/QA 5-10, code fixes 100+
skills_dir	Optional host Agent Skills directory	EvalScope makes the skills available before the agent loop starts
kwargs	Strategy kwargs	Most commonly {'system_prompt': '...'} to steer the model

Available strategy values:

Name	Use when
function_calling (default)	The model supports native function calling
react	Reasoning-first; let the model think before acting
swe_bench_toolcall	SWE-bench tool-call protocol
swe_bench_backticks	SWE-bench backticks text protocol (model lacks function calling)

Available tools (tools):

Name	Description
bash	Run shell commands
python_exec	Run Python code
submit	Submit the final answer (auto-injected; do not configure manually)

- `local` has no filesystem isolation; use `docker` in production.
- `agent_config` also accepts a plain dict; `TaskConfig` converts it to `NativeAgentConfig` automatically.

Agent Skills

Set NativeAgentConfig.skills_dir to a host directory whose immediate children are skill folders containing SKILL.md. EvalScope makes those skills available before the agent loop starts, and adds a short prompt hint when skill_prompt_nudge=True.

MCP server tools

Beyond the built-in tools, NativeAgentConfig accepts arbitrary MCP (Model Context Protocol) servers via mcp_servers — plug in fetch, web search, GitHub, and the rest of the MCP ecosystem without writing a custom EvalScope tool.

Install

pip install evalscope[mcp]

evalscope[mcp] bundles the official mcp Python SDK plus mcp-server-fetch (universal HTTP fetching, no API key). Other MCP servers can be installed on demand or run via uvx / npx.

For search-agent benchmark examples that combine search MCP servers with fetch/browser-style tools, see the DeepSearchQA and WideSearch guides.

Quick start

Attach HTTP fetching and a remote MCP tool to a GAIA evaluation:

import sys
from evalscope import TaskConfig, run_task
from evalscope.api.agent import NativeAgentConfig
from evalscope.api.agent.mcp import MCPServerConfigStdio, MCPServerConfigHTTP

task_config = TaskConfig(
    model='qwen3-max',
    api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
    eval_type='openai_api',
    datasets=['gaia'],
    agent_config=NativeAgentConfig(
        mcp_servers=[
            # Spawn a local MCP server process
            MCPServerConfigStdio(
                command=sys.executable,
                args=['-m', 'mcp_server_fetch', '--ignore-robots-txt'],
                name='fetch',
            ),
            # Connect to a remote MCP endpoint
            MCPServerConfigHTTP(
                url='https://mcp.api-inference.modelscope.net/<server-id>/mcp',
                headers={'Authorization': 'Bearer <TOKEN>'},
                name='modelscope_tool',
            ),
        ],
    ),
    dataset_args={'gaia': {'subset_list': ['2023_level1']}},
    limit=5,
)
run_task(task_config)

Once configured, the model sees both built-in tools and every tool advertised by the MCP servers, picking them by name. If an MCP tool call fails, the error is fed back to the model so it can retry — the evaluation loop keeps running.

Connection methods

EvalScope supports three MCP transports:

Config class	When to use	Key parameters
MCPServerConfigStdio	Spawn a local MCP server process	command, args, env
MCPServerConfigHTTP	Connect to a remote Streamable HTTP endpoint (recommended)	url, headers
MCPServerConfigSSE	Connect to a legacy SSE endpoint	url, headers

All config classes share name (display name in logs) and tools (tool whitelist, defaults to 'all'). For the full parameter list, see the source.

- For CI/production, prefer `python -m mcp_server_<x>` after `pip install` — no per-run package fetch.
- `uvx mcp-server-<x>` is convenient for ad-hoc runs; first call downloads the package.
- When a remote service offers both `/mcp` and `/sse`, prefer `/mcp` (Streamable HTTP).

SWE-bench agentic benchmarks

The swe_bench_*_agentic family (swe_bench_verified_agentic, swe_bench_verified_mini_agentic, swe_bench_lite_agentic) owns the per-sample Docker environment and required bash tool. Explicit NativeAgentConfig values can still override loop settings such as strategy and step limit. Keep dataset_args.extra_params for benchmark-specific image preparation options.

from evalscope.api.agent import NativeAgentConfig

task_config = TaskConfig(
    model='qwen-plus',
    api_url='...',
    api_key='...',
    eval_type='openai_api',
    datasets=['swe_bench_verified_mini_agentic'],
    agent_config=NativeAgentConfig(
        strategy='swe_bench_toolcall',
        max_steps=250,
    ),
    dataset_args={
        'swe_bench_verified_mini_agentic': {
            'extra_params': {
                'build_docker_images': True,
                'pull_remote_images_if_available': True,
            },
        },
    },
    limit=3,
    generation_config={'max_tokens': 4096, 'temperature': 0.0},
)
run_task(task_config)

Use NativeAgentConfig(strategy='swe_bench_backticks') for models that do not support function calling. Common extra_params keys are build_docker_images, pull_remote_images_if_available, and force_arch. Full reference in the benchmark docs.

Prerequisites: `pip install evalscope[swe_bench]`, Docker installed locally.

FAQ

The model never calls any tool

1. Check tool-name spelling in tools.
2. Confirm the model supports function calling; if not, switch to a text protocol like react or swe_bench_backticks.
3. Use kwargs={'system_prompt': '...'} to explicitly steer the model toward tool use.

How big should max_steps be?

• Math / simple QA: 3–10 is usually enough.
• Code fixes / SWE-bench: keep the default 250; lower values produce a lot of max_steps_exceeded.