# SGLang TP×DP Matrix Benchmark

Benchmark SGLang inference performance across different Tensor Parallelism (TP) × Data Parallelism (DP) configurations for large MoE models (e.g., DeepSeek-V4-Flash).

## What This Experiment Does

This experiment systematically tests how different parallel strategies affect serving performance:

| Config | TP | DP | GPUs per replica | Replicas |
|--------|----|----|------------------|----------|
| tp2_dp4 | 2 | 4 | 2 | 4 |
| tp4_dp2 | 4 | 2 | 4 | 2 |
| tp8_dp1 | 8 | 1 | 8 | 1 |

For each configuration, it runs a matrix of input/output sequence lengths and concurrency levels, measuring:
- **TTFT** (Time To First Token)
- **TPOT** (Time Per Output Token)
- **Throughput** (requests/s, output tokens/s)
- **GPU memory usage**
- **SLO compliance** (TTFT P95 < 3000ms, TPOT mean < 50ms)

## Quick Start

### 1. Prerequisites

- NVIDIA GPU server with Docker and NVIDIA Container Toolkit
- Model weights mounted locally (e.g., `/data/models/DeepSeek-V4-Flash`)
- ShareGPT-format dataset JSON for the random sampler (e.g., `/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json`)
- SGLang Docker image pulled locally

### 2. Configure the Experiment

Edit `config.env` to match your environment:

```bash
# Model settings
MODEL_NAME="DeepSeek-V4-Flash"
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
SERVED_MODEL_NAME="deepseek-v4-flash"

# Docker image (change this if you use a custom or private registry image)
DOCKER_IMAGE="lmsysorg/sglang:latest"

# Dataset path (change this to your local dataset)
DATASET_PATH="/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"

# GPU devices to use
CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

# TP×DP configurations to test
PARALLEL_CONFIGS=(
  "2 4"
  "4 2"
  "8 1"
)
```

### 3. Run the Experiment

#### Option A: Persistent Container Mode (Recommended)

Starts a long-running container and reuses it across configurations. This avoids reloading the model and preserves DeepGEMM JIT cache, saving significant time.

```bash
# Step 1: Start the persistent container (run once)
bash start_sglang_container.sh

# Step 2: Run the benchmark
CONTAINER_NAME="dsv4_h200_sglang_tp_dp_matrix_sglang_container" \
  bash run_bench.sh
```

#### Option B: One-Container-Per-Config Mode

Each TP×DP configuration starts a fresh Docker container. Slower but simpler.

```bash
# Just run the main script
bash run_bench.sh
```

### 4. View Results

Results are saved under `results/<run_id>/`:

```
results/
└── full_20260710-063109/
    ├── tp2_dp4/
    │   ├── results.json          # Parsed metrics per scenario
    │   ├── raw_outputs/          # Raw benchmark output files
    │   ├── gpu_logs/             # GPU memory CSV logs
    │   └── logs/                 # Detailed logs per scenario
    ├── tp4_dp2/
    │   └── ...
    ├── tp8_dp1/
    │   └── ...
    ├── comparison.md             # Cross-config comparison report
    └── logs/
        └── orchestrator.log      # Main experiment log
```

Generate the comparison report manually:

```bash
python3 compare.py --run-root results/<run_id> --output results/<run_id>/comparison.md
```

## Configuration Reference

### `config.env` — Main Configuration File

| Variable | Default | Description |
|----------|---------|-------------|
| `MODEL_NAME` | `DeepSeek-V4-Flash` | Human-readable model name |
| `MODEL_PATH` | `/data/models/DeepSeek-V4-Flash` | **Local path to model weights** |
| `SERVED_MODEL_NAME` | `deepseek-v4-flash` | Model name exposed by the server |
| `DOCKER_IMAGE` | `lmsysorg/sglang:latest` | **SGLang Docker image** |
| `DATASET_PATH` | `/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json` | **Local dataset for random sampler** |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3,4,5,6,7` | GPU devices to use |
| `PARALLEL_CONFIGS` | `("2 4" "4 2" "8 1")` | TP×DP configurations to test |
| `CONTEXT_LENGTH` | `1048576` | Max context length (tokens) |
| `MAX_RUNNING_REQUESTS` | `128` | Max concurrent requests |
| `MEM_FRACTION_STATIC` | `0.88` | Static memory fraction |
| `MOE_RUNNER_BACKEND` | `marlin` | MoE backend (marlin, flashinfer, etc.) |
| `USE_DOCKER` | `1` | Run server in Docker (1) or native venv (0) |
| `USE_DOCKER_CLIENT` | `1` | Run benchmark client in Docker (1) or native (0) |
| `SGLANG_PORT` | `30031` | Server port |
| `MATRIX_FILE` | `matrix.json` | Scenario matrix definition |
| `MATRIX_MODE` | `Y` | Test only Y (mandatory), or Y+P (include optional) |
| `CONCURRENCY_SAMPLES` | `2` | Concurrency sampling density |
| `SCENARIO_TIMEOUT_S` | `1800` | Per-scenario timeout (seconds) |
| `DRY_RUN` | `0` | If 1, only print plan without running |
| `GRID_LIMIT` | `0` | Limit scenarios per config (0 = all) |

### `matrix.json` — Test Scenario Matrix

Defines which (Input Sequence Length, Output Sequence Length) combinations to test:

```json
{
  "mode": "Y",
  "matrix": {
    "1024": { "128": "Y", "256": "Y", ... },
    "4096": { "128": "Y", "256": "Y", ... }
  },
  "concurrency": {
    "1024": { "low": 1, "high": 128 },
    "4096": { "low": 1, "high": 64 }
  }
}
```

Mark meanings:
- **Y** = Must test; failure stops the config
- **P** = Optional; failure is recorded but continues
- **N** = Skip

### `smoke_matrix.json` — Quick Verification

Small matrix for fast testing. Use it to verify the setup works:

```bash
MATRIX_FILE="${PWD}/smoke_matrix.json" \
  GRID_LIMIT=4 \
  bash run_bench.sh
```

## Script Reference

| Script | Purpose | When to Run |
|--------|---------|-------------|
| `run_bench.sh` | **Main experiment orchestrator** | Always run this |
| `start_sglang_container.sh` | Start a persistent Docker container | Before run_bench.sh if using persistent mode |
| `run_sglang_in_container.sh` | Start/restart server inside persistent container | Called by run_bench.sh |
| `start_sglang_dp.sh` | Entry point for starting server | Called by run_bench.sh |
| `start_sglang_docker.sh` | Start a fresh Docker container per config | Called by start_sglang_dp.sh |
| `generate_scenarios.py` | Generate scenario TSV from matrix.json | Called by run_bench.sh |
| `compare.py` | Generate cross-config comparison report | Called by run_bench.sh, or run manually |

## Migrating to a New Server

When moving this experiment to a different server, update these fields in `config.env`:

1. **Model path**: `MODEL_PATH` → your local model weights directory
2. **Docker image**: `DOCKER_IMAGE` → your SGLang image (e.g., `myregistry/sglang:v0.4.0`)
3. **Dataset path**: `DATASET_PATH` → your local ShareGPT-format JSON
4. **GPU devices**: `CUDA_VISIBLE_DEVICES` → match your hardware
5. **TP×DP configs**: `PARALLEL_CONFIGS` → adjust to your GPU count
   - Total GPUs must equal `TP × DP` for each config
   - Example: 8 GPUs → `(2 4)`, `(4 2)`, `(8 1)` are all valid

Optional adjustments:
- `CONTEXT_LENGTH` → model's max context length
- `MEM_FRACTION_STATIC` → adjust if you hit OOM
- `MOE_RUNNER_BACKEND` → `marlin`, `flashinfer`, etc.

## Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| `docker: invalid reference format` | `DOCKER_IMAGE` not exported to subshell | Already fixed; ensure `run_bench.sh` exports it |
| Server health check timeout | Model loading takes >20 min | Increase wait loops in `start_sglang_docker.sh` or use persistent container mode |
| `CUDA out of memory` | Config exceeds GPU memory | Reduce `MEM_FRACTION_STATIC` or test smaller ISL/DSL |
| Benchmark client can't reach server | Network isolation | Ensure both server and client use `--network host` |
| Stale container from previous run | Container name collision | `docker rm -f <container_name>` before restarting |

## Monitoring a Running Experiment

```bash
# Watch the main log
tail -f /tmp/full_experiment.log

# Check if the orchestrator is still running
ps aux | grep run_bench | grep -v grep

# Check GPU utilization
watch -n 1 nvidia-smi

# Check server health
curl http://127.0.0.1:30031/health
```

## Output Files

- `results/<run_id>/<config_label>/results.json` — Parsed metrics per scenario
- `results/<run_id>/<config_label>/raw_outputs/` — Raw `sglang.bench_serving` JSONL output
- `results/<run_id>/<config_label>/gpu_logs/` — GPU memory usage CSV
- `results/<run_id>/comparison.md` — Side-by-side comparison across all configs
- `results/<run_id>/skipped_after_oom.csv` — Record of OOM/failed scenarios

## License

Follows the project's existing license. Model weights and Docker images are subject to their respective licenses.