sskj/experiments/dsv4_h200_sglang_tp_dp_matrix/README.md

SGLang TP×DP Matrix Benchmark

Benchmark SGLang inference performance across different Tensor Parallelism (TP) × Data Parallelism (DP) configurations for large MoE models (e.g., DeepSeek-V4-Flash).

What This Experiment Does

This experiment systematically tests how different parallel strategies affect serving performance:

Config	TP	DP	GPUs per replica	Replicas
tp2_dp4	2	4	2	4
tp4_dp2	4	2	4	2
tp8_dp1	8	1	8	1

For each configuration, it runs a matrix of input/output sequence lengths and concurrency levels, measuring:

• TTFT (Time To First Token)
• TPOT (Time Per Output Token)
• Throughput (requests/s, output tokens/s)
• GPU memory usage
• SLO compliance (TTFT P95 < 3000ms, TPOT mean < 50ms)

Quick Start

1. Prerequisites

• NVIDIA GPU server with Docker and NVIDIA Container Toolkit
• Model weights mounted locally (e.g., /data/models/DeepSeek-V4-Flash)
• ShareGPT-format dataset JSON for the random sampler (e.g., /data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json)
• SGLang Docker image pulled locally

2. Configure the Experiment

Edit config.env to match your environment:

# Model settings
MODEL_NAME="DeepSeek-V4-Flash"
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
SERVED_MODEL_NAME="deepseek-v4-flash"

# Docker image (change this if you use a custom or private registry image)
DOCKER_IMAGE="lmsysorg/sglang:latest"

# Dataset path (change this to your local dataset)
DATASET_PATH="/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"

# GPU devices to use
CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

# TP×DP configurations to test
PARALLEL_CONFIGS=(
  "2 4"
  "4 2"
  "8 1"
)

3. Run the Experiment

Option A: Persistent Container Mode (Recommended)

Starts a long-running container and reuses it across configurations. This avoids reloading the model and preserves DeepGEMM JIT cache, saving significant time.

# Step 1: Start the persistent container (run once)
bash start_sglang_container.sh

# Step 2: Run the benchmark
CONTAINER_NAME="dsv4_h200_sglang_tp_dp_matrix_sglang_container" \
  bash run_bench.sh

Option B: One-Container-Per-Config Mode

Each TP×DP configuration starts a fresh Docker container. Slower but simpler.

# Just run the main script
bash run_bench.sh

4. View Results

Results are saved under results/<run_id>/:

results/
└── full_20260710-063109/
    ├── tp2_dp4/
    │   ├── results.json          # Parsed metrics per scenario
    │   ├── raw_outputs/          # Raw benchmark output files
    │   ├── gpu_logs/             # GPU memory CSV logs
    │   └── logs/                 # Detailed logs per scenario
    ├── tp4_dp2/
    │   └── ...
    ├── tp8_dp1/
    │   └── ...
    ├── comparison.md             # Cross-config comparison report
    └── logs/
        └── orchestrator.log      # Main experiment log

Generate the comparison report manually:

python3 compare.py --run-root results/<run_id> --output results/<run_id>/comparison.md

Configuration Reference

config.env — Main Configuration File

Variable	Default	Description
MODEL_NAME	DeepSeek-V4-Flash	Human-readable model name
MODEL_PATH	/data/models/DeepSeek-V4-Flash	Local path to model weights
SERVED_MODEL_NAME	deepseek-v4-flash	Model name exposed by the server
DOCKER_IMAGE	lmsysorg/sglang:latest	SGLang Docker image
DATASET_PATH	/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json	Local dataset for random sampler
CUDA_VISIBLE_DEVICES	0,1,2,3,4,5,6,7	GPU devices to use
PARALLEL_CONFIGS	("2 4" "4 2" "8 1")	TP×DP configurations to test
CONTEXT_LENGTH	1048576	Max context length (tokens)
MAX_RUNNING_REQUESTS	128	Max concurrent requests
MEM_FRACTION_STATIC	0.88	Static memory fraction
MOE_RUNNER_BACKEND	marlin	MoE backend (marlin, flashinfer, etc.)
USE_DOCKER	1	Run server in Docker (1) or native venv (0)
USE_DOCKER_CLIENT	1	Run benchmark client in Docker (1) or native (0)
SGLANG_PORT	30031	Server port
MATRIX_FILE	matrix.json	Scenario matrix definition
MATRIX_MODE	Y	Test only Y (mandatory), or Y+P (include optional)
CONCURRENCY_SAMPLES	2	Concurrency sampling density
SCENARIO_TIMEOUT_S	1800	Per-scenario timeout (seconds)
DRY_RUN	0	If 1, only print plan without running
GRID_LIMIT	0	Limit scenarios per config (0 = all)

matrix.json — Test Scenario Matrix

Defines which (Input Sequence Length, Output Sequence Length) combinations to test:

{
  "mode": "Y",
  "matrix": {
    "1024": { "128": "Y", "256": "Y", ... },
    "4096": { "128": "Y", "256": "Y", ... }
  },
  "concurrency": {
    "1024": { "low": 1, "high": 128 },
    "4096": { "low": 1, "high": 64 }
  }
}

Mark meanings:

• Y = Must test; failure stops the config
• P = Optional; failure is recorded but continues
• N = Skip

smoke_matrix.json — Quick Verification

Small matrix for fast testing. Use it to verify the setup works:

MATRIX_FILE="${PWD}/smoke_matrix.json" \
  GRID_LIMIT=4 \
  bash run_bench.sh

Script Reference

Script	Purpose	When to Run
run_bench.sh	Main experiment orchestrator	Always run this
start_sglang_container.sh	Start a persistent Docker container	Before run_bench.sh if using persistent mode
run_sglang_in_container.sh	Start/restart server inside persistent container	Called by run_bench.sh
start_sglang_dp.sh	Entry point for starting server	Called by run_bench.sh
start_sglang_docker.sh	Start a fresh Docker container per config	Called by start_sglang_dp.sh
generate_scenarios.py	Generate scenario TSV from matrix.json	Called by run_bench.sh
compare.py	Generate cross-config comparison report	Called by run_bench.sh, or run manually

Migrating to a New Server

When moving this experiment to a different server, update these fields in config.env:

1. Model path: MODEL_PATH → your local model weights directory
2. Docker image: DOCKER_IMAGE → your SGLang image (e.g., myregistry/sglang:v0.4.0)
3. Dataset path: DATASET_PATH → your local ShareGPT-format JSON
4. GPU devices: CUDA_VISIBLE_DEVICES → match your hardware
5. TP×DP configs: PARALLEL_CONFIGS → adjust to your GPU count
   • Total GPUs must equal TP × DP for each config
   • Example: 8 GPUs → (2 4), (4 2), (8 1) are all valid

Optional adjustments:

• CONTEXT_LENGTH → model's max context length
• MEM_FRACTION_STATIC → adjust if you hit OOM
• MOE_RUNNER_BACKEND → marlin, flashinfer, etc.

Troubleshooting

Symptom	Cause	Fix
docker: invalid reference format	DOCKER_IMAGE not exported to subshell	Already fixed; ensure run_bench.sh exports it
Server health check timeout	Model loading takes >20 min	Increase wait loops in start_sglang_docker.sh or use persistent container mode
CUDA out of memory	Config exceeds GPU memory	Reduce MEM_FRACTION_STATIC or test smaller ISL/DSL
Benchmark client can't reach server	Network isolation	Ensure both server and client use --network host
Stale container from previous run	Container name collision	docker rm -f <container_name> before restarting

Monitoring a Running Experiment

# Watch the main log
tail -f /tmp/full_experiment.log

# Check if the orchestrator is still running
ps aux | grep run_bench | grep -v grep

# Check GPU utilization
watch -n 1 nvidia-smi

# Check server health
curl http://127.0.0.1:30031/health

Output Files

• results/<run_id>/<config_label>/results.json — Parsed metrics per scenario
• results/<run_id>/<config_label>/raw_outputs/ — Raw sglang.bench_serving JSONL output
• results/<run_id>/<config_label>/gpu_logs/ — GPU memory usage CSV
• results/<run_id>/comparison.md — Side-by-side comparison across all configs
• results/<run_id>/skipped_after_oom.csv — Record of OOM/failed scenarios

License

Follows the project's existing license. Model weights and Docker images are subject to their respective licenses.