SkillsBench

SkillsBench evaluates whether agents can use task-bundled Agent Skills. Each task contains task.md, environment/Dockerfile, environment/skills/, oracle/, and verifier/. EvalScope builds a local Docker image from the task environment, runs the agent or oracle in a container, then executes the task’s verifier/test.sh and reads /logs/verifier/reward.txt as the score.

Prerequisites

• Docker is installed and running.

• The SkillsBench repository is cloned locally, and you pass the tasks/ directory path.

• Real agent runs require agent_config, for example the Codex external runner.

• External runners execute inside the task container. Codex/OpenCode/Gemini CLI runners probe for the CLI first and can install it during runner setup when auto_install=True.

• Verifier scripts may install dependencies through apt/pip/uv, so network access is usually required.

Skill Mode

The default skill_mode is no-skill.

• no-skill: EvalScope removes environment/skills from the temporary build context and strips skill copy/path residues from the Dockerfile.

• with-skill: EvalScope injects task-bundled skills into /skills; the runner installs them into its own skill discovery path before launch.

self-gen is not supported in this version. EvalScope does not automatically compute lift. Run no-skill and with-skill separately, then compare the two runs with EvalScope’s run comparison tools.

Oracle Smoke

Use oracle first to validate the image, paths, and verifier contract:

from evalscope import TaskConfig, run_task

run_task(TaskConfig(
    model='dummy',
    datasets=['skillsbench'],
    limit=1,
    dataset_args={
        'skillsbench': {
            'extra_params': {
                'tasks_dir': '/path/to/skillsbench/tasks',
                'task_ids': ['offer-letter-generator'],
                'runner': 'oracle',
                'skill_mode': 'no-skill',
            }
        }
    },
))

Real Agent

no-skill:

from evalscope import TaskConfig, run_task

run_task(TaskConfig(
    model='your-model',
    datasets=['skillsbench'],
    limit=1,
    agent_config={
        'mode': 'external',
        'framework': 'codex',
        'environment': 'docker',
        'timeout': 900,
        'kwargs': {
            'auto_install': True,
        },
    },
    dataset_args={
        'skillsbench': {
            'extra_params': {
                'tasks_dir': '/path/to/skillsbench/tasks',
                'task_ids': ['offer-letter-generator'],
                'skill_mode': 'no-skill',
            }
        }
    },
))

For with-skill, only change skill_mode:

dataset_args={
    'skillsbench': {
        'extra_params': {
            'tasks_dir': '/path/to/skillsbench/tasks',
            'task_ids': ['offer-letter-generator'],
            'skill_mode': 'with-skill',
        }
    }
}

Image Cache

EvalScope creates a temporary build context for each task and skill mode, then tags the local Docker image from the context hash. no-skill and with-skill use different cache keys. The first run builds the image; later runs with the same context reuse it. Set force_rebuild=true to rebuild.

The task image remains the source of truth for task dependencies and verifier inputs. If that image already contains the external CLI, set the runner’s auto_install to False; otherwise leave auto_install=True so the runner installs the CLI inside the task container during setup.

Current Limits

• The adapter targets the official SkillsBench tasks/ set by default and does not include tasks-extra/ automatically.

• The supported verifier contract is verifier/test.sh writing /logs/verifier/reward.txt.

• Full BenchFlow verifier features such as reward-kit, llm-judge, agent-judge, ors-episode, and multi-service verifiers are not supported.

• EvalScope saves key agent/verifier logs and metadata by default, not the full container filesystem.