Toolathlon

Toolathlon evaluates long-horizon agent tool use in realistic MCP-backed software environments. EvalScope integrates it as a wrapper around the official Toolathlon evaluation service. EvalScope does not reimplement the MCP environments, task containers, agent loop, or official scorer.

Install

pip install evalscope[toolathlon]

The EvalScope wrapper only needs the client-side dependencies httpx and websockets. You do not need to install the full Toolathlon repository when you use the official public service.

Service Modes

EvalScope supports Toolathlon official service private mode.

In private mode, the Toolathlon service runs the task containers, MCP environments, agent loop, and scoring. EvalScope starts a local WebSocket relay; when the service needs a model response, the request is relayed back to your local or intranet OpenAI-compatible endpoint. This allows api_url=http://localhost:8000/v1 to work even when the Toolathlon service is remote.

The official Toolathlon client also supports public mode, where the service directly calls a public model API with the submitted API key. The EvalScope wrapper intentionally uses private mode so the model endpoint and API key stay on the EvalScope side.

Official Public Service

The default service is the official public service:

• HTTP service: 47.253.6.47:8080

• WebSocket proxy: 47.253.6.47:8081

The public service is intended for quick trials and debugging. According to the official documentation, it applies IP-based limits over a 24-hour window:

• 180 minutes cumulative execution time per IP per 24 hours

• 3 evaluation requests per IP per 24 hours

• While cumulative execution time is under 180 minutes, requests are not capped by the 3-request limit

• After cumulative execution time exceeds 180 minutes, the 3-request-per-24-hour cap applies

If submission returns HTTP 503 Service Unavailable, the service is usually busy because the official server runs one evaluation job at a time. If submission returns HTTP 429, the IP-based rate limit was reached.

Quick Debug Run

Use a small task_list first:

from evalscope import TaskConfig, run_task

task_cfg = TaskConfig(
    model='your-model-name',
    api_url='http://localhost:8000/v1',
    api_key='your-local-api-key',
    datasets=['toolathlon'],
    dataset_args={
        'toolathlon': {
            'extra_params': {
                'task_list': ['find-alita-paper'],
                'skip_container_restart': True,
            }
        }
    },
    limit=1,
)

run_task(task_cfg)

Use skip_container_restart=True only for small debugging subsets. For formal runs, remove it so the official service can restart task containers between jobs.

Self-hosted Official Service

Self-hosting removes the public-service queue and public IP limits, but it is not a single docker compose up deployment. The official Docker image docker.io/lockon0927/toolathlon-task-image:1016beta is mainly the task/runtime image; it does not package every service, account, credential, and evaluator process required by the Toolathlon evaluation service.

Before starting a self-hosted service, follow the official Toolathlon materials:

• README.md: installation dependencies, Docker/Podman setup, configs/global_configs.py, app credential setup, local app deployment, and local smoke checks

• global_preparation/how2register_accounts.md: accounts, tokens, and sessions required by MCP-backed tasks

• configs/global_configs_example.py: copy to configs/global_configs.py and set podman_or_docker

• global_preparation/pull_toolathlon_image.sh: pull the official task image

• global_preparation/deploy_containers.sh: deploy local services such as Canvas, email, WooCommerce, and k8s/kind

• EVAL_SERVICE_README.md: start the official eval_server.py HTTP service and private-mode WebSocket proxy

After the full Toolathlon environment is ready, start the official service from the Toolathlon repository:

python eval_server.py 8080 8081 3 10 180

The arguments are:

• server_port: HTTP service port

• ws_proxy_port: WebSocket proxy port for private mode

• max_submissions_per_ip: request limit per IP per 24 hours, or -1 for unlimited

• max_workers: maximum Toolathlon workers per job

• max_duration_minutes: cumulative execution-time limit per IP per 24 hours, or -1 for unlimited

For an internal service without rate limiting:

python eval_server.py 8080 8081 -1 10 -1

Use a Self-hosted Service from EvalScope

Point EvalScope to the self-hosted service through extra_params:

from evalscope import TaskConfig, run_task

task_cfg = TaskConfig(
    model='your-model-name',
    api_url='http://localhost:8000/v1',
    api_key='your-local-api-key',
    datasets=['toolathlon'],
    dataset_args={
        'toolathlon': {
            'extra_params': {
                'server_host': 'your.toolathlon.server',
                'server_port': 8080,
                'ws_proxy_port': 8081,
                'workers': 10,
                'task_list': ['find-alita-paper'],
            }
        }
    },
    limit=1,
)

run_task(task_cfg)

For CLI usage, put the same values under toolathlon.extra_params in --dataset-args.

Operational Notes

• The official server runs one evaluation job at a time. Increasing workers controls parallelism inside a job, not the number of concurrent submitted jobs.

• The official service receives prompts, model responses, and tool context in order to drive the agent loop and scoring.

• Keep client_version and WebSocket client protocol aligned with the official service version. This EvalScope wrapper follows Toolathlon service protocol 1.3.

• For sustained evaluation, prefer a dedicated official service or a self-hosted service over the shared public endpoint.

Official sources:

• Toolathlon repository

• Toolathlon evaluation service README