Native AgentLoop Mode#

Turn a regular benchmark (GSM8K, AIME, IFEval, SWE-bench, …) into a multi-turn tool-use task to evaluate the model’s own tool-use and multi-step reasoning. EvalScope runs generate → parse → call tool → observe → generate on each iteration until the model submits a final answer or hits the step cap. The full trajectory is recorded as an AgentTrace for replay in the UI.

To evaluate an off-the-shelf agent CLI such as Claude Code / Codex, see External Agent Bridge Mode.

When to use#

Evaluate a model’s multi-step reasoning and tool-use ability Example: does qwen-plus proactively call python_exec to verify arithmetic on GSM8K? Can it reason step-by-step and self-check on AIME?
Let a model autonomously explore code-fix tasks Example: on SWE-bench, give the model a bash tool to read code, run tests, and submit patches — no human in the loop.
Plug in MCP ecosystem tools to expand model capabilities Example: attach fetch, Brave Search, or other MCP servers so the model can search the web when running benchmarks like GAIA.

Quick start#

Smallest runnable example — have qwen-plus verify GSM8K answers with python_exec.

from evalscope import TaskConfig, run_task
from evalscope.api.agent import NativeAgentConfig

task_config = TaskConfig(
    model='qwen-plus',
    api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
    api_key='<your-key>',
    eval_type='openai_api',
    datasets=['gsm8k'],
    limit=5,
    generation_config={'parallel_tool_calls': True},
    agent_config=NativeAgentConfig(
        strategy='function_calling',
        tools=['python_exec'],
        environment='docker',
        environment_extra={'image': 'python:3.11-slim'},
        max_steps=5,
        kwargs={'system_prompt': 'Use python_exec to verify your calculations.'},
    ),
)
run_task(task_config)

EvalScope auto-injects the python_exec tool definition into the model’s context, executes the generated Python in a Docker container, feeds stdout/stderr back as an observation, and loops until the model uses submit to deliver its answer.

Common configuration#

Most-used NativeAgentConfig fields:

Field	Description	Recommended
`strategy`	Interaction protocol	`function_calling` (default); use `react` or `swe_bench_backticks` if the model lacks function-calling
`tools`	Tool whitelist	On demand, e.g. `['python_exec']` / `['bash']`
`environment`	Where tools run	`local` (dev) / `docker` (production)
`environment_extra`	Sandbox constructor kwargs	For `docker`: `{'image': '...', 'timeout': 60}`; see Sandbox Environment
`max_steps`	Max iterations per sample	Math/QA `5-10`, code fixes `100+`
`skills_dir`	Optional host Agent Skills directory	EvalScope makes the skills available before the agent loop starts
`kwargs`	Strategy kwargs	Most commonly `{'system_prompt': '...'}` to steer the model

Available strategy values:

Name	Use when
`function_calling` (default)	The model supports native function calling
`react`	Reasoning-first; let the model think before acting
`swe_bench_toolcall`	SWE-bench tool-call protocol
`swe_bench_backticks`	SWE-bench backticks text protocol (model lacks function calling)

Available tools (tools):

Name	Description
`bash`	Run shell commands
`python_exec`	Run Python code
`submit`	Submit the final answer (auto-injected; do not configure manually)

Tip

local has no filesystem isolation; use docker in production.
agent_config also accepts a plain dict; TaskConfig converts it to NativeAgentConfig automatically.

Agent Skills#

Set NativeAgentConfig.skills_dir to a host directory whose immediate children are skill folders containing SKILL.md. EvalScope makes those skills available before the agent loop starts, and adds a short prompt hint when skill_prompt_nudge=True.

MCP server tools#

Beyond the built-in tools, NativeAgentConfig accepts arbitrary MCP (Model Context Protocol) servers via mcp_servers — plug in fetch, web search, GitHub, and the rest of the MCP ecosystem without writing a custom EvalScope tool.

Install#

pip install evalscope[mcp]

evalscope[mcp] bundles the official mcp Python SDK plus mcp-server-fetch (universal HTTP fetching, no API key). Other MCP servers can be installed on demand or run via uvx / npx.

Quick start#

Attach HTTP fetching and a remote MCP tool to a GAIA evaluation:

import sys
from evalscope import TaskConfig, run_task
from evalscope.api.agent import NativeAgentConfig
from evalscope.api.agent.mcp import MCPServerConfigStdio, MCPServerConfigHTTP

task_config = TaskConfig(
    model='qwen3-max',
    api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
    eval_type='openai_api',
    datasets=['gaia'],
    agent_config=NativeAgentConfig(
        mcp_servers=[
            # Spawn a local MCP server process
            MCPServerConfigStdio(
                command=sys.executable,
                args=['-m', 'mcp_server_fetch', '--ignore-robots-txt'],
                name='fetch',
            ),
            # Connect to a remote MCP endpoint
            MCPServerConfigHTTP(
                url='https://mcp.api-inference.modelscope.net/<server-id>/mcp',
                headers={'Authorization': 'Bearer <TOKEN>'},
                name='modelscope_tool',
            ),
        ],
    ),
    dataset_args={'gaia': {'subset_list': ['2023_level1']}},
    limit=5,
)
run_task(task_config)

Once configured, the model sees both built-in tools and every tool advertised by the MCP servers, picking them by name. If an MCP tool call fails, the error is fed back to the model so it can retry — the evaluation loop keeps running.

Connection methods#

EvalScope supports three MCP transports:

Config class	When to use	Key parameters
`MCPServerConfigStdio`	Spawn a local MCP server process	`command`, `args`, `env`
`MCPServerConfigHTTP`	Connect to a remote Streamable HTTP endpoint (recommended)	`url`, `headers`
`MCPServerConfigSSE`	Connect to a legacy SSE endpoint	`url`, `headers`

All config classes share name (display name in logs) and tools (tool whitelist, defaults to 'all'). For the full parameter list, see the source.

Tip

For CI/production, prefer python -m mcp_server_<x> after pip install — no per-run package fetch.
uvx mcp-server-<x> is convenient for ad-hoc runs; first call downloads the package.
When a remote service offers both /mcp and /sse, prefer /mcp (Streamable HTTP).

SWE-bench agentic benchmarks#

The swe_bench_*_agentic family (swe_bench_verified_agentic, swe_bench_verified_mini_agentic, swe_bench_lite_agentic) owns the per-sample Docker environment and required bash tool. Explicit NativeAgentConfig values can still override loop settings such as strategy and step limit. Keep dataset_args.extra_params for benchmark-specific image preparation options.

from evalscope.api.agent import NativeAgentConfig

task_config = TaskConfig(
    model='qwen-plus',
    api_url='...',
    api_key='...',
    eval_type='openai_api',
    datasets=['swe_bench_verified_mini_agentic'],
    agent_config=NativeAgentConfig(
        strategy='swe_bench_toolcall',
        max_steps=250,
    ),
    dataset_args={
        'swe_bench_verified_mini_agentic': {
            'extra_params': {
                'build_docker_images': True,
                'pull_remote_images_if_available': True,
            },
        },
    },
    limit=3,
    generation_config={'max_tokens': 4096, 'temperature': 0.0},
)
run_task(task_config)

Use NativeAgentConfig(strategy='swe_bench_backticks') for models that do not support function calling. Common extra_params keys are build_docker_images, pull_remote_images_if_available, and force_arch. Full reference in the benchmark docs.

Important

Prerequisites: pip install evalscope[swe_bench], Docker installed locally.

FAQ#

The model never calls any tool

Check tool-name spelling in tools.
Confirm the model supports function calling; if not, switch to a text protocol like react or swe_bench_backticks.
Use kwargs={'system_prompt': '...'} to explicitly steer the model toward tool use.

How big should max_steps be?

Math / simple QA: 3–10 is usually enough.
Code fixes / SWE-bench: keep the default 250; lower values produce a lot of max_steps_exceeded.