evalstone/evalscope/skills/evalscope/perf-reference.md

Perf Parameter Reference

Complete parameter reference for evalscope perf. All parameters can also be discovered via evalscope perf --help.

Model and API

Parameter	Type	Default	Description
--model	str	(required)	Model name or path
--api	str	openai	API protocol: openai, local, local_vllm, dashscope, embedding, rerank, custom
--url	str	http://127.0.0.1:8877/v1/chat/completions	API endpoint URL
--port	int	8877	Port for local inference server
--api-key	str	None	API authentication key
--tokenizer-path	str	None	Tokenizer path for accurate token counting
--attn-implementation	str	None	Attention implementation (local inference only)

API Types

Value	Use Case
openai	OpenAI-compatible API endpoints (default)
local	Auto-start local inference server using the model
local_vllm	Auto-start local vLLM inference server
dashscope	Alibaba Cloud DashScope API
embedding	OpenAI-compatible embedding API
rerank	OpenAI-compatible reranking API
custom	Custom API implementation

Connection Settings

Parameter	Type	Default	Description
--headers	key=value pairs	None	Extra HTTP headers (e.g. --headers key1=val1 key2=val2)
--connect-timeout	int	None	Network connection timeout (seconds)
--read-timeout	int	None	Network read timeout (seconds)
--total-timeout	int	21600	Total request timeout in seconds (default: 6 hours)
--no-test-connection	flag	false	Skip connection test before starting benchmark

Concurrency and Load

Parameter	Type	Default	Description
--parallel	int (multiple)	1	Number of concurrent requests. Supports multiple values for gradient tests
-n, --number	int (multiple)	1000	Total number of requests. Must match --parallel count
--rate	float	-1	Requests per second limit (-1 = unlimited)
--sleep-interval	int	5	Sleep seconds between consecutive perf runs

Important: When providing multiple values for --parallel and --number, they are paired positionally:

# Run 3 tests: (parallel=1, n=100), (parallel=5, n=500), (parallel=10, n=1000)
--parallel 1 5 10 --number 100 500 1000

SLA Auto-Tuning

Automatically find the maximum concurrency or rate that satisfies SLA constraints.

Parameter	Type	Default	Description
--sla-auto-tune	flag	false	Enable SLA auto-tuning
--sla-variable	str	parallel	Variable to tune: parallel or rate
--sla-params	JSON str	None	SLA constraints as JSON array
--sla-num-runs	int	3	Runs to average per configuration
--sla-upper-bound	int	65536	Upper bound for binary search
--sla-lower-bound	int	1	Lower bound for binary search
--sla-number-multiplier	float	None	Multiplier: number = round(variable * N). Default: 2

SLA Params Format

--sla-params accepts a JSON array of constraint objects:

[
  {"name": "latency", "operator": "<=", "value": 5.0},
  {"name": "ttft", "operator": "<=", "value": 0.5},
  {"name": "output_token_throughput", "operator": ">=", "value": 100}
]

Available constraint names: latency, ttft, tpot, request_throughput, output_token_throughput, total_token_throughput.

Operators: <=, >=, <, >, ==.

Dataset / Prompt Settings

Parameter	Type	Default	Description
--dataset	str	openqa	Dataset type (see below)
--dataset-path	str	None	Path to custom dataset file
--prompt	str	None	Use a fixed prompt text for all requests
--query-template	str	None	Jinja2 template for request formatting
--apply-chat-template	flag	auto	Apply chat template to prompts (auto-detected from URL)

Dataset Types

Type	Description
openqa	Open-ended QA prompts (default, good for general testing)
random	Random token sequences (for precise input length control)
speed_benchmark	Speed benchmark with fixed prompts
random_vl	Random vision-language data (for multimodal models)
line_by_line	Read prompts from a file, one per line
flickr8k	Flickr8K vision-language dataset
longalpaca	Long-context prompts
embedding_dataset	For embedding model benchmarks
rerank_dataset	For reranking model benchmarks
custom	Custom dataset plugin

Prompt Length Control (for random dataset)

Parameter	Type	Default	Description
--min-prompt-length	int	0	Minimum input prompt length. Unit is tokens when --tokenizer-path is provided, otherwise characters (len(prompt))
--max-prompt-length	int	131072 (no cap)	Maximum input prompt length. Unit is tokens when --tokenizer-path is provided, otherwise characters (len(prompt))
--prefix-length	int	0	Fixed prefix length for random prompts

Vision-Language Settings (for random_vl dataset)

Parameter	Type	Default	Description
--image-width	int	224	Image width in pixels
--image-height	int	224	Image height in pixels
--image-format	str	RGB	Image format
--image-num	int	1	Number of images per request
--image-patch-size	int	28	Patch size for image tokenizer (local token calculation)

Response Settings

Parameter	Type	Default	Description
--max-tokens	int	2048	Maximum output tokens
--min-tokens	int	None	Minimum output tokens (forces model to generate at least N tokens)
--temperature	float	0.0	Sampling temperature
--top-p	float	None	Top-p (nucleus) sampling
--top-k	int	None	Top-k sampling
--stream / --no-stream	flag	stream	Enable/disable streaming (default: enabled)
--frequency-penalty	float	None	Frequency penalty
--repetition-penalty	float	None	Repetition penalty
--n-choices	int	None	Number of response choices
--seed	int	None	Random seed
--stop	str (multiple)	None	Stop sequences
--stop-token-ids	str (multiple)	None	Stop token IDs
--logprobs	flag	None	Return log probabilities
--extra-args	JSON str	{}	Additional API-specific arguments

Output and Logging

Parameter	Type	Default	Description
--outputs-dir	str	outputs	Output directory
--no-timestamp	flag	false	Don't add timestamp to output directory
--name	str	None	Custom name for the run (used in DB and visualizer)
--debug	flag	false	Enable debug logging
--log-every-n-query	int	10	Log progress every N queries
--enable-progress-tracker	flag	false	Write progress.json for tracking

Visualization Integration

Parameter	Type	Default	Description
--visualizer	str	None	Visualizer: wandb, swanlab, clearml
--wandb-api-key	str	None	W&B API key
--swanlab-api-key	str	None	SwanLab API key

Performance Tuning Knobs

Parameter	Type	Default	Description
--db-commit-interval	int	1000	Rows buffered before SQLite commit
--queue-size-multiplier	int	5	Request queue maxsize = parallel * multiplier
--in-flight-task-multiplier	int	2	Max scheduled tasks = parallel * multiplier