GAIA#

Introduction#

GAIA (General AI Assistants) is a benchmark of 450+ questions designed to evaluate next-generation LLMs equipped with tool use, web browsing and multi-step reasoning. Each question has a single unambiguous short answer (a number, a short phrase, or a comma-separated list).

Questions are split into three difficulty levels:

Level	Description
`2023_level1`	Should be solvable by capable LLMs with basic tool use
`2023_level2`	Requires more autonomous planning and richer tool use
`2023_level3`	Indicates a strong jump in agent capability — multi-modal sources, long chains of reasoning

Each level provides a public validation split (with answers) and a private test split (answers withheld for the official leaderboard). EvalScope currently supports validation only.

GAIA’s evalscope integration drives a multi-turn ReAct agent loop inside a per-sample Docker container, with a single bash tool. The official rule-based scorer (number / list / string normalization) is ported verbatim from the GAIA leaderboard.

Install Dependencies#

GAIA evaluation runs the agent inside Docker:

Install Docker: see the Docker Installation Guide. Make sure the daemon is running.
Install evalscope:
```
pip install evalscope
```

The first run will pull the python:3.11 image (~1.1GB) and download the GAIA dataset snapshot from ModelScope (~110MB) — these are cached for subsequent runs.

Note

The benchmark uses the gaia-benchmark/GAIA ModelScope mirror by default — no HuggingFace gating, no token required. Set dataset_hub='huggingface' if you prefer the original repo (you’ll need to accept the dataset terms first).

Datasets#

A single benchmark gaia covers all three difficulty levels via subset_list:

Configuration	Loads
`subset_list=['2023_level1']`	Level 1 only
`subset_list=['2023_level1', '2023_level2']`	Levels 1 + 2
`subset_list=['2023_level1', '2023_level2', '2023_level3']` (default)	All three levels

About 1/3 of GAIA questions reference an attachment file (PDF / xlsx / image / audio / …). EvalScope mounts the dataset’s 2023/validation/ directory read-only into the sandbox at /shared_files, so the agent can access referenced files via the path hint embedded in the prompt.

Run Example#

The example below mirrors test_gaia in tests/benchmark/test_agent.py:

import os
from evalscope import TaskConfig, run_task

task_cfg = TaskConfig(
    model='qwen3-max',
    api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
    api_key=os.getenv('DASHSCOPE_API_KEY'),
    eval_type='openai_api',
    datasets=['gaia'],
    dataset_args={
        'gaia': {
            'subset_list': ['2023_level1'],            # Pick which level(s) to run
            'extra_params': {
                'max_steps': 50,                       # Maximum agent loop steps per sample
                'command_timeout': 180.0,              # Per-bash-command timeout in seconds
                'docker_image': 'python:3.11',         # Sandbox image; full image bundles curl/wget/git
                'network_enabled': True,               # Required: most questions need network access
            }
        }
    },
    eval_batch_size=5,  # Number of parallel sandboxes
    limit=5,            # Limit samples for quick testing; remove for the full validation set
    generation_config={
        'temperature': 0.7,
        'parallel_tool_calls': True,
        'stream': True,
    }
)
run_task(task_cfg=task_cfg)

Example final result:

+-----------+---------+----------+-------------+-------+---------+---------+
| Model     | Dataset | Metric   | Subset      |   Num |   Score | Cat.0   |
+===========+=========+==========+=============+=======+=========+=========+
| qwen3-max | gaia    | mean_acc | 2023_level1 |     5 |     0.2 | default |
+-----------+---------+----------+-------------+-------+---------+---------+

Parameters#

Parameter	Type	Default	Description
`max_steps`	int	`50`	Maximum agent loop steps per sample. Mirrors inspect_ai’s `message_limit=100` (~50 ReAct turns).
`command_timeout`	float	`180.0`	Per-bash-command timeout in seconds.
`docker_image`	str	`python:3.11`	Sandbox image. The full `python:3.11` ships `curl` / `wget` / `git`; use `python:3.11-slim` (~130MB) only if you bake your own tools.
`network_enabled`	bool	`True`	Allow the sandbox to access the network. Most GAIA questions need it.

Note

GAIA spawns one Docker container per sample. Each container is destroyed once the sample completes. Tune eval_batch_size according to your machine resources — each in-flight sample holds a container open.

Scoring#

The scorer is a verbatim port of the official GAIA leaderboard scorer (Apache 2.0):

Numeric ground truth: strip $ / % / ,, parse as float, compare for exact equality.
List ground truth (contains , or ;): split on those delimiters, compare element-wise (number-aware).
String ground truth: strip whitespace, lowercase, remove punctuation, compare for equality.

No LLM judge is involved. The agent must produce an answer that exactly matches the canonical normalization above — typically by calling the auto-injected submit(answer=...) tool.

Adding web access via MCP#

GAIA’s bash-only sandbox can curl / wget raw HTML, but cannot run a JS-rendered browser or hit gated search APIs. The simplest way to give the agent real browsing power is to plug an MCP server (e.g. mcp-server-fetch for HTTP, mcp-server-brave-search for keyword search) — the host-side MCP servers run outside the per-sample Docker, no sandbox image change needed.

pip install evalscope[mcp]

evalscope[mcp] ships with mcp-server-fetch already bundled, so no further install is needed for the example below.

import sys
from evalscope import TaskConfig, run_task
from evalscope.api.agent import NativeAgentConfig
from evalscope.api.agent.mcp import MCPServerConfigStdio

task_cfg = TaskConfig(
    model='qwen3-max',
    api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
    eval_type='openai_api',
    datasets=['gaia'],
    dataset_args={'gaia': {'subset_list': ['2023_level1']}},
    agent_config=NativeAgentConfig(
        mcp_servers=[
            MCPServerConfigStdio(
                command=sys.executable,
                # ``--ignore-robots-txt`` avoids stalling when the upstream
                # robots.txt request is intermittently blocked.
                args=['-m', 'mcp_server_fetch', '--ignore-robots-txt'],
                name='fetch',
            ),
        ],
    ),
    limit=5,
)
run_task(task_cfg)

The agent now sees bash, submit and fetch as tools and can call fetch(url=...) to read web pages directly — no docker-side library install, no curl plumbing.

See Native Agent Loop → MCP server tools for the full configuration reference (stdio / HTTP transports, tool whitelist, env vars, etc.).

Known Limitations#

No web_search / web_browser tool yet: GAIA Phase-1 ships only the bash tool. The agent can curl / wget and parse HTML with python3, but JavaScript-rendered pages and gated search APIs cannot be reached. Browsing-heavy questions will score significantly lower than implementations with a real browser tool.
No support for test split: only validation (which has public answers) is supported. To submit to the official leaderboard you’ll need to capture model predictions and format them yourself.
Long bash outputs may exceed model input length: a curl of a large HTML page accumulates fast in the agent’s message history and can hit the model’s max input length on harder samples. Set ignore_errors=True on TaskConfig (or in the test) so the rest of the run continues when that happens.