Model Inference Stress Testing

A stress testing tool for large language models that can be customized to support various dataset formats and different API protocol formats, with default support for the OpenAI API format.

• Quick Start
  • Environment Preparation
  • Basic Usage
  • Visualizing Test Results
• Parameter
  • Basic Settings
  • Network Configuration
  • Request Control
  • SLA Settings
  • Dataset Configuration
  • Input Construction
  • Multi-turn Conversation
  • Production Traffic Replay
  • Model and Generation
  • Output
  • Other Parameters
• Examples
  • Local Model Inference
  • Input Construction
  • Request Configuration
  • Load Patterns
  • Embedding and Rerank
  • Debugging Requests
  • Visualizing Results
• Multi-turn Conversation Benchmark
  • How It Works
  • Parameters
  • Datasets
  • Output Metrics
• SLA Auto-Tuning
  • Features
  • Parameter Description
  • Supported Metrics and Operators
  • --sla-params Logic
  • Workflow
  • Usage Examples
• Speed Benchmark Testing
  • Online API Inference
  • Local Transformer Inference
  • Local vLLM Inference
• vLLM Bench vs Evalscope Perf Load Testing Comparison
  • TL;DR: Quick Comparison Recipe
  • Environment and Prerequisites
  • Unified Server Configuration
  • Parameter Alignment Guide (Key Mappings)
  • Consistency Validation: Minimum Example (1 Concurrent / 1 Request)
  • Full Load Test: 50 Concurrency / 1000 Requests
  • Metric Definitions and Naming Correspondence
  • Common Sources of Discrepancies and Troubleshooting Suggestions
• Custom Usage
  • Custom Result Analysis
  • Custom API Requests
  • Custom Dataset
  • Notes