fix(sglang-tp-dp-matrix): export DOCKER_IMAGE/USE_DOCKER_CLIENT and add persistent container support

- Fix docker invalid reference format by exporting DOCKER_IMAGE and
  USE_DOCKER_CLIENT to bash subshells (run_bench_serving)
- Add persistent container mode: start_sglang_container.sh +
  run_sglang_in_container.sh for faster config switching
- Add README.md with setup and migration guide

experiments/dsv4_h200_sglang_tp_dp_matrix/README.md

@@ -0,0 +1,234 @@
+# SGLang TP×DP Matrix Benchmark
+
+Benchmark SGLang inference performance across different Tensor Parallelism (TP) × Data Parallelism (DP) configurations for large MoE models (e.g., DeepSeek-V4-Flash).
+
+## What This Experiment Does
+
+This experiment systematically tests how different parallel strategies affect serving performance:
+
+| Config | TP | DP | GPUs per replica | Replicas |
+|--------|----|----|------------------|----------|
+| tp2_dp4 | 2 | 4 | 2 | 4 |
+| tp4_dp2 | 4 | 2 | 4 | 2 |
+| tp8_dp1 | 8 | 1 | 8 | 1 |
+
+For each configuration, it runs a matrix of input/output sequence lengths and concurrency levels, measuring:
+- **TTFT** (Time To First Token)
+- **TPOT** (Time Per Output Token)
+- **Throughput** (requests/s, output tokens/s)
+- **GPU memory usage**
+- **SLO compliance** (TTFT P95 < 3000ms, TPOT mean < 50ms)
+
+## Quick Start
+
+### 1. Prerequisites
+
+- NVIDIA GPU server with Docker and NVIDIA Container Toolkit
+- Model weights mounted locally (e.g., `/data/models/DeepSeek-V4-Flash`)
+- ShareGPT-format dataset JSON for the random sampler (e.g., `/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json`)
+- SGLang Docker image pulled locally
+
+### 2. Configure the Experiment
+
+Edit `config.env` to match your environment:
+
+```bash
+# Model settings
+MODEL_NAME="DeepSeek-V4-Flash"
+MODEL_PATH="/data/models/DeepSeek-V4-Flash"
+SERVED_MODEL_NAME="deepseek-v4-flash"
+
+# Docker image (change this if you use a custom or private registry image)
+DOCKER_IMAGE="lmsysorg/sglang:latest"
+
+# Dataset path (change this to your local dataset)
+DATASET_PATH="/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"
+
+# GPU devices to use
+CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
+
+# TP×DP configurations to test
+PARALLEL_CONFIGS=(
+  "2 4"
+  "4 2"
+  "8 1"
+)
+```
+
+### 3. Run the Experiment
+
+#### Option A: Persistent Container Mode (Recommended)
+
+Starts a long-running container and reuses it across configurations. This avoids reloading the model and preserves DeepGEMM JIT cache, saving significant time.
+
+```bash
+# Step 1: Start the persistent container (run once)
+bash start_sglang_container.sh
+
+# Step 2: Run the benchmark
+CONTAINER_NAME="dsv4_h200_sglang_tp_dp_matrix_sglang_container" \
+  bash run_bench.sh
+```
+
+#### Option B: One-Container-Per-Config Mode
+
+Each TP×DP configuration starts a fresh Docker container. Slower but simpler.
+
+```bash
+# Just run the main script
+bash run_bench.sh
+```
+
+### 4. View Results
+
+Results are saved under `results/<run_id>/`:
+
+```
+results/
+└── full_20260710-063109/
+    ├── tp2_dp4/
+    │   ├── results.json          # Parsed metrics per scenario
+    │   ├── raw_outputs/          # Raw benchmark output files
+    │   ├── gpu_logs/             # GPU memory CSV logs
+    │   └── logs/                 # Detailed logs per scenario
+    ├── tp4_dp2/
+    │   └── ...
+    ├── tp8_dp1/
+    │   └── ...
+    ├── comparison.md             # Cross-config comparison report
+    └── logs/
+        └── orchestrator.log      # Main experiment log
+```
+
+Generate the comparison report manually:
+
+```bash
+python3 compare.py --run-root results/<run_id> --output results/<run_id>/comparison.md
+```
+
+## Configuration Reference
+
+### `config.env` — Main Configuration File
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MODEL_NAME` | `DeepSeek-V4-Flash` | Human-readable model name |
+| `MODEL_PATH` | `/data/models/DeepSeek-V4-Flash` | **Local path to model weights** |
+| `SERVED_MODEL_NAME` | `deepseek-v4-flash` | Model name exposed by the server |
+| `DOCKER_IMAGE` | `lmsysorg/sglang:latest` | **SGLang Docker image** |
+| `DATASET_PATH` | `/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json` | **Local dataset for random sampler** |
+| `CUDA_VISIBLE_DEVICES` | `0,1,2,3,4,5,6,7` | GPU devices to use |
+| `PARALLEL_CONFIGS` | `("2 4" "4 2" "8 1")` | TP×DP configurations to test |
+| `CONTEXT_LENGTH` | `1048576` | Max context length (tokens) |
+| `MAX_RUNNING_REQUESTS` | `128` | Max concurrent requests |
+| `MEM_FRACTION_STATIC` | `0.88` | Static memory fraction |
+| `MOE_RUNNER_BACKEND` | `marlin` | MoE backend (marlin, flashinfer, etc.) |
+| `USE_DOCKER` | `1` | Run server in Docker (1) or native venv (0) |
+| `USE_DOCKER_CLIENT` | `1` | Run benchmark client in Docker (1) or native (0) |
+| `SGLANG_PORT` | `30031` | Server port |
+| `MATRIX_FILE` | `matrix.json` | Scenario matrix definition |
+| `MATRIX_MODE` | `Y` | Test only Y (mandatory), or Y+P (include optional) |
+| `CONCURRENCY_SAMPLES` | `2` | Concurrency sampling density |
+| `SCENARIO_TIMEOUT_S` | `1800` | Per-scenario timeout (seconds) |
+| `DRY_RUN` | `0` | If 1, only print plan without running |
+| `GRID_LIMIT` | `0` | Limit scenarios per config (0 = all) |
+
+### `matrix.json` — Test Scenario Matrix
+
+Defines which (Input Sequence Length, Output Sequence Length) combinations to test:
+
+```json
+{
+  "mode": "Y",
+  "matrix": {
+    "1024": { "128": "Y", "256": "Y", ... },
+    "4096": { "128": "Y", "256": "Y", ... }
+  },
+  "concurrency": {
+    "1024": { "low": 1, "high": 128 },
+    "4096": { "low": 1, "high": 64 }
+  }
+}
+```
+
+Mark meanings:
+- **Y** = Must test; failure stops the config
+- **P** = Optional; failure is recorded but continues
+- **N** = Skip
+
+### `smoke_matrix.json` — Quick Verification
+
+Small matrix for fast testing. Use it to verify the setup works:
+
+```bash
+MATRIX_FILE="${PWD}/smoke_matrix.json" \
+  GRID_LIMIT=4 \
+  bash run_bench.sh
+```
+
+## Script Reference
+
+| Script | Purpose | When to Run |
+|--------|---------|-------------|
+| `run_bench.sh` | **Main experiment orchestrator** | Always run this |
+| `start_sglang_container.sh` | Start a persistent Docker container | Before run_bench.sh if using persistent mode |
+| `run_sglang_in_container.sh` | Start/restart server inside persistent container | Called by run_bench.sh |
+| `start_sglang_dp.sh` | Entry point for starting server | Called by run_bench.sh |
+| `start_sglang_docker.sh` | Start a fresh Docker container per config | Called by start_sglang_dp.sh |
+| `generate_scenarios.py` | Generate scenario TSV from matrix.json | Called by run_bench.sh |
+| `compare.py` | Generate cross-config comparison report | Called by run_bench.sh, or run manually |
+
+## Migrating to a New Server
+
+When moving this experiment to a different server, update these fields in `config.env`:
+
+1. **Model path**: `MODEL_PATH` → your local model weights directory
+2. **Docker image**: `DOCKER_IMAGE` → your SGLang image (e.g., `myregistry/sglang:v0.4.0`)
+3. **Dataset path**: `DATASET_PATH` → your local ShareGPT-format JSON
+4. **GPU devices**: `CUDA_VISIBLE_DEVICES` → match your hardware
+5. **TP×DP configs**: `PARALLEL_CONFIGS` → adjust to your GPU count
+   - Total GPUs must equal `TP × DP` for each config
+   - Example: 8 GPUs → `(2 4)`, `(4 2)`, `(8 1)` are all valid
+
+Optional adjustments:
+- `CONTEXT_LENGTH` → model's max context length
+- `MEM_FRACTION_STATIC` → adjust if you hit OOM
+- `MOE_RUNNER_BACKEND` → `marlin`, `flashinfer`, etc.
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| `docker: invalid reference format` | `DOCKER_IMAGE` not exported to subshell | Already fixed; ensure `run_bench.sh` exports it |
+| Server health check timeout | Model loading takes >20 min | Increase wait loops in `start_sglang_docker.sh` or use persistent container mode |
+| `CUDA out of memory` | Config exceeds GPU memory | Reduce `MEM_FRACTION_STATIC` or test smaller ISL/DSL |
+| Benchmark client can't reach server | Network isolation | Ensure both server and client use `--network host` |
+| Stale container from previous run | Container name collision | `docker rm -f <container_name>` before restarting |
+
+## Monitoring a Running Experiment
+
+```bash
+# Watch the main log
+tail -f /tmp/full_experiment.log
+
+# Check if the orchestrator is still running
+ps aux | grep run_bench | grep -v grep
+
+# Check GPU utilization
+watch -n 1 nvidia-smi
+
+# Check server health
+curl http://127.0.0.1:30031/health
+```
+
+## Output Files
+
+- `results/<run_id>/<config_label>/results.json` — Parsed metrics per scenario
+- `results/<run_id>/<config_label>/raw_outputs/` — Raw `sglang.bench_serving` JSONL output
+- `results/<run_id>/<config_label>/gpu_logs/` — GPU memory usage CSV
+- `results/<run_id>/comparison.md` — Side-by-side comparison across all configs
+- `results/<run_id>/skipped_after_oom.csv` — Record of OOM/failed scenarios
+
+## License
+
+Follows the project's existing license. Model weights and Docker images are subject to their respective licenses.

experiments/dsv4_h200_sglang_tp_dp_matrix/run_bench.sh

@@ -12,9 +12,6 @@ source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
 # shellcheck source=/dev/null
 source "${SCRIPT_DIR}/config.env"
 
-# Export variables used inside functions that are called via bash -c subshells.
-export DATASET_PATH MODEL_PATH RESULT_BASE
-
 RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
 RESULT_BASE="${SCRIPT_DIR}/results"
 MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR}/matrix.json}"
@@ -24,6 +21,9 @@ GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
 DRY_RUN="${DRY_RUN:-0}"
 GRID_LIMIT="${GRID_LIMIT:-0}"
 
+# Export variables used inside functions that are called via bash -c subshells.
+export DATASET_PATH MODEL_PATH RESULT_BASE DOCKER_IMAGE USE_DOCKER_CLIENT
+
 if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
   PYTHON="${VENV_CLIENT}/bin/python"
 else
@@ -55,17 +55,29 @@ stop_server() {
   if [[ -f "$pid_file" ]]; then
     local pid
     pid="$(cat "$pid_file")"
-    if kill -0 "$pid" 2>/dev/null; then
+    # In container-reuse mode, pid is the server PID inside the container.
-      log "stopping sglang server pid=${pid} (tp=${tp}, dp=${dp})"
+    if [[ -n "${CONTAINER_NAME:-}" ]]; then
-      kill "$pid" 2>/dev/null || true
+      if docker exec "$CONTAINER_NAME" kill -0 "$pid" 2>/dev/null; then
-      sleep 5
+        log "stopping sglang server inside container (tp=${tp}, dp=${dp}, pid=${pid})"
-      kill -9 "$pid" 2>/dev/null || true
+        docker exec "$CONTAINER_NAME" kill "$pid" 2>/dev/null || true
+        sleep 5
+        docker exec "$CONTAINER_NAME" kill -9 "$pid" 2>/dev/null || true
+      fi
+    else
+      if kill -0 "$pid" 2>/dev/null; then
+        log "stopping sglang server pid=${pid} (tp=${tp}, dp=${dp})"
+        kill "$pid" 2>/dev/null || true
+        sleep 5
+        kill -9 "$pid" 2>/dev/null || true
+      fi
     fi
     rm -f "$pid_file"
   fi
 
-  # Fallback: remove any Docker container started by this experiment.
+  # Fallback: remove any Docker container started by this experiment (legacy mode).
-  docker rm -f "${EXPERIMENT}_sglang_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
+  if [[ -z "${CONTAINER_NAME:-}" ]]; then
+    docker rm -f "${EXPERIMENT}_sglang_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
+  fi
 
   # Fallback: kill any sglang serve processes for this model.
   pkill -9 -f "sglang serve.*${MODEL_NAME}" 2>/dev/null || true
@@ -101,8 +113,13 @@ start_server() {
   local dp="$2"
 
   log "starting sglang server tp=${tp} dp=${dp}"
-  bash "${SCRIPT_DIR}/start_sglang_dp.sh" "$tp" "$dp" \
+  if [[ -n "${CONTAINER_NAME:-}" ]]; then
-    >> "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1
+    bash "${SCRIPT_DIR}/run_sglang_in_container.sh" "$tp" "$dp" \
+      >> "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1
+  else
+    bash "${SCRIPT_DIR}/start_sglang_dp.sh" "$tp" "$dp" \
+      >> "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1
+  fi
 
   if ! is_server_healthy; then
     log "error: sglang server tp=${tp} dp=${dp} failed health check on port ${SGLANG_PORT}"

experiments/dsv4_h200_sglang_tp_dp_matrix/run_sglang_in_container.sh

@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+# Run SGLang server inside the already-running benchmark container.
+# Usage: run_sglang_in_container.sh <TP> <DP>
+set -e
+
+TP="${1}"
+DP="${2}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
+mkdir -p "${RUNTIME_BASE}/logs"
+
+PORT="${SGLANG_PORT:-30031}"
+NAME="${EXPERIMENT}_sglang_container"
+PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_sglang_tp${TP}_dp${DP}.pid"
+
+LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_sglang_in_container_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
+rm -f "$PID_FILE"
+
+# Build server args.
+SERVER_ARGS=(
+  serve
+  --model-path "$MODEL_PATH"
+  --trust-remote-code
+  --tp-size "$TP"
+  --moe-runner-backend "$MOE_RUNNER_BACKEND"
+  --context-length "$CONTEXT_LENGTH"
+  --max-running-requests "$MAX_RUNNING_REQUESTS"
+  --mem-fraction-static "$MEM_FRACTION_STATIC"
+  --host 0.0.0.0
+  --port "$PORT"
+)
+
+if [[ "$DP" -gt 1 ]]; then
+  SERVER_ARGS+=(
+    --dp-size "$DP"
+  )
+fi
+
+SERVER_ARGS_STR="${SERVER_ARGS[*]}"
+
+echo "=== Starting SGLang server inside container (TP=${TP}, DP=${DP}) ==="
+echo "Container: $NAME"
+echo "Command: sglang ${SERVER_ARGS_STR}"
+echo "Log: $LOG"
+
+# Kill any existing sglang process inside container first.
+docker exec "$NAME" pkill -9 -f "sglang serve" 2>/dev/null || true
+sleep 3
+
+# Start sglang serve inside the container in background.
+# We use nohup so it survives after docker exec returns.
+docker exec -d \
+  -e CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}" \
+  -e PYTHONUNBUFFERED=1 \
+  "$NAME" \
+  bash -c "nohup sglang ${SERVER_ARGS_STR} > /tmp/sglang_server.log 2>&1 &"
+
+# Wait for the server process to appear inside container.
+sleep 2
+SERVER_PID=$(docker exec "$NAME" pgrep -f "sglang serve" | head -n 1 || true)
+if [[ -z "$SERVER_PID" ]]; then
+  echo "ERROR: sglang server process not found inside container"
+  docker exec "$NAME" cat /tmp/sglang_server.log 2>/dev/null | tail -50 || true
+  exit 1
+fi
+
+echo "Server PID inside container: $SERVER_PID"
+echo "$SERVER_PID" > "$PID_FILE"
+
+echo "Waiting for health on port ${PORT}..."
+for i in $(seq 1 360); do
+  if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
+    echo "SGLang server is ready at http://127.0.0.1:${PORT}"
+    echo "Log: $LOG"
+    exit 0
+  fi
+  # Check if server process is still alive.
+  if ! docker exec "$NAME" kill -0 "$SERVER_PID" 2>/dev/null; then
+    echo "ERROR: SGLang server exited early"
+    docker exec "$NAME" cat /tmp/sglang_server.log 2>/dev/null | tail -200 || true
+    exit 1
+  fi
+  echo "Waiting... ($i/360)"
+  sleep 5
+done
+
+echo "ERROR: SGLang server not healthy after 360 retries (30 mins)"
+docker exec "$NAME" cat /tmp/sglang_server.log 2>/dev/null | tail -200 || true
+exit 1

experiments/dsv4_h200_sglang_tp_dp_matrix/start_sglang_container.sh

@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# Start a long-running Docker container for SGLang benchmarking.
+# The container stays alive (sleep infinity) so we can docker exec into it
+# to run different TP×DP configurations without losing DeepGEMM JIT cache.
+#
+# Usage: start_sglang_container.sh
+set -e
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
+mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
+
+IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:latest}"
+PORT="${SGLANG_PORT:-30031}"
+NAME="${EXPERIMENT}_sglang_container"
+PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_container.pid"
+
+# Persistent cache directory on host for DeepGEMM JIT kernels.
+CACHE_DIR="${RUNTIME_BASE}/cache"
+mkdir -p "${CACHE_DIR}/deep_gemm" "${CACHE_DIR}/tvm-ffi"
+
+# Clean up any stale container.
+docker rm -f "$NAME" >/dev/null 2>&1 || true
+
+LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_container_$(date +%Y%m%d_%H%M%S).log"
+rm -f "$PID_FILE"
+
+echo "=== Starting SGLang benchmark container ==="
+echo "Image: $IMAGE"
+echo "Container name: $NAME"
+echo "Host port: $PORT"
+echo "Cache dir: $CACHE_DIR"
+echo "Log: $LOG"
+
+# Start a long-running container.
+# We override the entrypoint to sleep infinity so the container stays alive.
+docker run -d \
+  --name "$NAME" \
+  --gpus all \
+  --ipc host \
+  --shm-size 16g \
+  --entrypoint /bin/bash \
+  -p "${PORT}:${PORT}" \
+  -v "${MODEL_PATH}:${MODEL_PATH}:ro" \
+  -v "${CACHE_DIR}/deep_gemm:/root/.cache/deep_gemm" \
+  -v "${CACHE_DIR}/tvm-ffi:/root/.cache/tvm-ffi" \
+  -v "${RUNTIME_BASE}/tmp:/tmp" \
+  -e CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}" \
+  -e PYTHONUNBUFFERED=1 \
+  "$IMAGE" \
+  -c "sleep infinity" \
+  > "$LOG" 2>&1
+
+# Wait a moment for container to be ready.
+sleep 2
+
+if ! docker ps --filter "name=$NAME" --format '{{.Names}}' | grep -q "$NAME"; then
+  echo "ERROR: Container failed to start"
+  tail -50 "$LOG"
+  exit 1
+fi
+
+# Record the container ID for later use.
+docker inspect -f '{{.Id}}' "$NAME" > "$PID_FILE"
+
+echo "Container $NAME is running"
+echo "Log: $LOG"