Model Inference Stress Testing

A stress testing tool for large language models that can be customized to support various dataset formats and different API protocol formats, with default support for the OpenAI API format.

• Quick Start
  • Environment Preparation
  • Basic Usage
• Parameter Description
  • Basic Settings
  • Network Configuration
  • Request Control
  • Prompt Settings
  • Dataset Configuration
  • Model Settings
  • Data Storage
• Examples
  • Using Local Model Inference
  • Using prompt
  • Complex Requests
  • Using query-template
  • Using wandb to Record Test Results
  • Debugging Requests
• Speed Benchmark Testing
  • Online API Inference
  • Local Transformer Inference
  • Local vLLM Inference
• Custom Usage
  • Custom Result Analysis
  • Custom Request API
  • Custom Dataset