Model Inference Stress Testing

A stress testing tool for large language models that can be customized to support various dataset formats and different API protocol formats, with default support for the OpenAI API format.

• Quick Start
  • Environment Preparation
  • Basic Usage
  • Visualizing Test Results
• Parameter
  • Basic Settings
  • Network Configuration
  • Request Control
  • SLA Settings
  • Prompt Settings
  • Dataset Configuration
  • Model Settings
  • Data Storage
  • Other Parameters
• Examples
  • Using Local Model Inference
  • Using prompt
  • Complex Requests
  • Using query-template
  • Using the Random Dataset
  • Using the Random Multimodal Dataset
  • Embedding Model Stress Testing
  • Rerank Model Stress Testing
  • Visualizing Test Results
  • Debugging Requests
• SLA Auto-Tuning
  • Features
  • Parameter Description
  • Supported Metrics and Operators
  • --sla-params Logic
  • Workflow
  • Usage Examples
• Speed Benchmark Testing
  • Online API Inference
  • Local Transformer Inference
  • Local vLLM Inference
• vLLM Bench vs Evalscope Perf Load Testing Comparison
  • TL;DR: Quick Comparison Recipe
  • Environment and Prerequisites
  • Unified Server Configuration
  • Parameter Alignment Guide (Key Mappings)
  • Consistency Validation: Minimum Example (1 Concurrent / 1 Request)
  • Full Load Test: 50 Concurrency / 1000 Requests
  • Metric Definitions and Naming Correspondence
  • Common Sources of Discrepancies and Troubleshooting Suggestions
• Custom Usage
  • Custom Result Analysis
  • Custom API Requests
  • Custom Dataset
  • Notes