Benchmark SGLang inference performance across different Tensor Parallelism (TP) × Data Parallelism (DP) configurations for large MoE models (e.g., DeepSeek-V4-Flash).

What This Experiment Does

This experiment systematically tests how different parallel strategies affect serving performance:

Config	TP	DP	GPUs per replica	Replicas
tp2_dp4	2	4	2	4
tp4_dp2	4	2	4	2
tp8_dp1	8	1	8	1

For each configuration, it runs a matrix of input/output sequence lengths and concurrency levels, measuring:

TTFT (Time To First Token)
TPOT (Time Per Output Token)
Throughput (requests/s, output tokens/s)
GPU memory usage
SLO compliance (TTFT P95 < 3000ms, TPOT mean < 50ms)

Quick Start

1. Prerequisites

NVIDIA GPU server with Docker and NVIDIA Container Toolkit
Model weights mounted locally (e.g., /data/models/DeepSeek-V4-Flash)
ShareGPT-format dataset JSON for the random sampler (e.g., /data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json)
SGLang Docker image pulled locally

2. Configure the Experiment

Edit config.env to match your environment:

# Model settings
MODEL_NAME="DeepSeek-V4-Flash"
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
SERVED_MODEL_NAME="deepseek-v4-flash"

# Docker image (change this if you use a custom or private registry image)
DOCKER_IMAGE="lmsysorg/sglang:latest"

# Dataset path (change this to your local dataset)
DATASET_PATH="/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"

# GPU devices to use
CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

# TP×DP configurations to test
PARALLEL_CONFIGS=(
  "2 4"
  "4 2"
  "8 1"
)

3. Run the Experiment

Option A: Persistent Container Mode (Recommended)

Starts a long-running container and reuses it across configurations. This avoids reloading the model and preserves DeepGEMM JIT cache, saving significant time.

# Step 1: Start the persistent container (run once)
bash start_sglang_container.sh

# Step 2: Run the benchmark
CONTAINER_NAME="dsv4_h200_sglang_tp_dp_matrix_sglang_container" \
  bash run_bench.sh

Option B: One-Container-Per-Config Mode

Each TP×DP configuration starts a fresh Docker container. Slower but simpler.

# Just run the main script
bash run_bench.sh

4. View Results

Results are saved under results/<run_id>/:

results/
└── full_20260710-063109/
    ├── tp2_dp4/
    │   ├── results.json          # Parsed metrics per scenario
    │   ├── raw_outputs/          # Raw benchmark output files
    │   ├── gpu_logs/             # GPU memory CSV logs
    │   └── logs/                 # Detailed logs per scenario
    ├── tp4_dp2/
    │   └── ...
    ├── tp8_dp1/
    │   └── ...
    ├── comparison.md             # Cross-config comparison report
    └── logs/
        └── orchestrator.log      # Main experiment log

Generate the comparison report manually:

python3 compare.py --run-root results/<run_id> --output results/<run_id>/comparison.md

Configuration Reference

`config.env` — Main Configuration File

Variable	Default	Description
`MODEL_NAME`	`DeepSeek-V4-Flash`	Human-readable model name
`MODEL_PATH`	`/data/models/DeepSeek-V4-Flash`	Local path to model weights
`SERVED_MODEL_NAME`	`deepseek-v4-flash`	Model name exposed by the server
`DOCKER_IMAGE`	`lmsysorg/sglang:latest`	SGLang Docker image
`DATASET_PATH`	`/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json`	Local dataset for random sampler
`CUDA_VISIBLE_DEVICES`	`0,1,2,3,4,5,6,7`	GPU devices to use
`PARALLEL_CONFIGS`	`("2 4" "4 2" "8 1")`	TP×DP configurations to test
`CONTEXT_LENGTH`	`1048576`	Max context length (tokens)
`MAX_RUNNING_REQUESTS`	`128`	Max concurrent requests
`MEM_FRACTION_STATIC`	`0.88`	Static memory fraction
`MOE_RUNNER_BACKEND`	`marlin`	MoE backend (marlin, flashinfer, etc.)
`USE_DOCKER`	`1`	Run server in Docker (1) or native venv (0)
`USE_DOCKER_CLIENT`	`1`	Run benchmark client in Docker (1) or native (0)
`SGLANG_PORT`	`30031`	Server port
`MATRIX_FILE`	`matrix.json`	Scenario matrix definition
`MATRIX_MODE`	`Y`	Test only Y (mandatory), or Y+P (include optional)
`CONCURRENCY_SAMPLES`	`2`	Concurrency sampling density
`SCENARIO_TIMEOUT_S`	`1800`	Per-scenario timeout (seconds)
`DRY_RUN`	`0`	If 1, only print plan without running
`GRID_LIMIT`	`0`	Limit scenarios per config (0 = all)

`matrix.json` — Test Scenario Matrix

Defines which (Input Sequence Length, Output Sequence Length) combinations to test:

{
  "mode": "Y",
  "matrix": {
    "1024": { "128": "Y", "256": "Y", ... },
    "4096": { "128": "Y", "256": "Y", ... }
  },
  "concurrency": {
    "1024": { "low": 1, "high": 128 },
    "4096": { "low": 1, "high": 64 }
  }
}

Mark meanings:

Y = Must test; failure stops the config
P = Optional; failure is recorded but continues
N = Skip

`smoke_matrix.json` — Quick Verification

Small matrix for fast testing. Use it to verify the setup works:

MATRIX_FILE="${PWD}/smoke_matrix.json" \
  GRID_LIMIT=4 \
  bash run_bench.sh

Script Reference

Script	Purpose	When to Run
`run_bench.sh`	Main experiment orchestrator	Always run this
`start_sglang_container.sh`	Start a persistent Docker container	Before run_bench.sh if using persistent mode
`run_sglang_in_container.sh`	Start/restart server inside persistent container	Called by run_bench.sh
`start_sglang_dp.sh`	Entry point for starting server	Called by run_bench.sh
`start_sglang_docker.sh`	Start a fresh Docker container per config	Called by start_sglang_dp.sh
`generate_scenarios.py`	Generate scenario TSV from matrix.json	Called by run_bench.sh
`compare.py`	Generate cross-config comparison report	Called by run_bench.sh, or run manually

Migrating to a New Server

When moving this experiment to a different server, update these fields in config.env:

Model path: MODEL_PATH → your local model weights directory
Docker image: DOCKER_IMAGE → your SGLang image (e.g., myregistry/sglang:v0.4.0)
Dataset path: DATASET_PATH → your local ShareGPT-format JSON
GPU devices: CUDA_VISIBLE_DEVICES → match your hardware
TP×DP configs: PARALLEL_CONFIGS → adjust to your GPU count
- Total GPUs must equal TP × DP for each config
- Example: 8 GPUs → (2 4), (4 2), (8 1) are all valid

Optional adjustments:

CONTEXT_LENGTH → model's max context length
MEM_FRACTION_STATIC → adjust if you hit OOM
MOE_RUNNER_BACKEND → marlin, flashinfer, etc.

Troubleshooting

Symptom	Cause	Fix
`docker: invalid reference format`	`DOCKER_IMAGE` not exported to subshell	Already fixed; ensure `run_bench.sh` exports it
Server health check timeout	Model loading takes >20 min	Increase wait loops in `start_sglang_docker.sh` or use persistent container mode
`CUDA out of memory`	Config exceeds GPU memory	Reduce `MEM_FRACTION_STATIC` or test smaller ISL/DSL
Benchmark client can't reach server	Network isolation	Ensure both server and client use `--network host`
Stale container from previous run	Container name collision	`docker rm -f <container_name>` before restarting

Monitoring a Running Experiment

# Watch the main log
tail -f /tmp/full_experiment.log

# Check if the orchestrator is still running
ps aux | grep run_bench | grep -v grep

# Check GPU utilization
watch -n 1 nvidia-smi

# Check server health
curl http://127.0.0.1:30031/health

Output Files

results/<run_id>/<config_label>/results.json — Parsed metrics per scenario
results/<run_id>/<config_label>/raw_outputs/ — Raw sglang.bench_serving JSONL output
results/<run_id>/<config_label>/gpu_logs/ — GPU memory usage CSV
results/<run_id>/comparison.md — Side-by-side comparison across all configs
results/<run_id>/skipped_after_oom.csv — Record of OOM/failed scenarios

License

Follows the project's existing license. Model weights and Docker images are subject to their respective licenses.

README.md Unescape Escape

SGLang TP×DP Matrix Benchmark