Welcome to the EvalScope Tutorial!

Getting Started

To help users quickly get started, we recommend the following flow:

• For those who want to use EvalScope, we recommend first reading the QuickStart section to set up the environment and initiate a mini-experiment to familiarize yourself with the process.

• For some basic usages, we suggest users read the Tutorials, which include how to perform offline evaluations with EvalScope, how to use Arena models for evaluations, how to utilize other evaluation backends, and how to use the model service stress testing tool.

• If you wish to customize more modules, such as adding datasets and models, we provide the AdvancedTutorials.

• Additionally, we offer Third-PartyTools to help users quickly evaluate models, such as using ToolBench for evaluations.

• Finally, we provide BestPractices to assist users with evaluations, such as how to use Swift for evaluations.

We always welcome users’ PRs and Issues to improve EvalScope.

Contents

Quick Start

• Introduction
• Installation
• Basic Usage
• Supported Datasets

Tutorials

• Offline Evaluation
• Arena Mode
• OpenCompass Backend
• VLMEvalKit Backend
• Stress Testing

Advanced Tutorials

• Custom Dataset Evaluation
• Custom Model Evaluation

Third-Party Tools

• ToolBench
• LongBench-Write

Best Practices

• ms-swift Integration
• Full-Chain LLM Training
• Experiments