qqtang/sskj
1
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
sskj/experiments/p800/dsv4_p800_sglang_tp_dp_matrix
History

README.md

DSV4 P800 SGLang TP×DP Matrix

Kunlun P800 XPU + SGLang-XPU + DeepSeek-V4-Flash-INT8 TP×DP matrix benchmark, modeled after experiments/h20/dsv4_h20_sglang_tp_dp_matrix.

What it does

Runs each parallel configuration with its own server instance and sweeps the ISL/OSL matrix in matrix.json:

  • TP=2, DP=4 → 2 XPUs per replica, 4 replicas
  • TP=4, DP=2 → 4 XPUs per replica, 2 replicas
  • TP=8, DP=1 → 8 XPUs, no data parallelism (production configuration)

Differences from the H20 version:

  • Server runs in the P800 vendor image with the proven INT8 launch args (NSA attention backend, w8a8_int8 quantization); the benchmark client runs inside the same container via docker exec.
  • ISL is capped at 131072 because P800 INT8 sustains at most ~131k input tokens; concurrency ranges match the H20 matrix (up to 128 at ISL=1024). Scenarios beyond server capacity are recorded as failed/skipped and the run moves on.
  • XPU memory sampling uses xpu-smi instead of nvidia-smi.

--dp-size support on this image was probed empirically (2026-07-16, results/smoke_tpdp): TP=4/DP=2 starts and serves correctly (replicas land on disjoint XPU groups), while TP=2/DP=4 fails during model loading with a weight-loading OOM. Root cause: expert weights are sharded only across the TP group (this matrix passes no --ep-size, so experts fall back to TP sharding; the image also asserts ep_size <= tp_size, so EP can never exceed TP anyway) and DP replicas do not share expert shards. The INT8 checkpoint holds ~264 GiB of routed-expert weights out of ~274 GiB total, so each rank carries roughly 274/TP GiB: TP=8/DP=1 ≈ 34 GiB, TP=4/DP=2 ≈ 68 GiB, TP=2/DP=4 ≈ 137 GiB — more than the 96 GiB an XPU has, hence the OOM. TP=4 is therefore the minimum viable TP for this model on P800. Configs that cannot start are recorded as SKIPPED_SERVICE_START_FAILED and the run continues with the next config.

Quick Start

# Full matrix (server per config -> warmup -> scenarios -> parse -> compare)
bash experiments/p800/dsv4_p800_sglang_tp_dp_matrix/run_bench.sh

# Dry run: print server args and scenario plan only
DRY_RUN=1 bash experiments/p800/dsv4_p800_sglang_tp_dp_matrix/run_bench.sh

# Adaptive concurrency search (find saturation point per shape)
bash experiments/p800/dsv4_p800_sglang_tp_dp_matrix/run_adaptive_concurrency.sh

Results land in results/<RUN_ID>/ (fixed matrix) or adaptive_results/<RUN_ID>/ (adaptive search).

Files

File Purpose
config.env Experiment-level configuration (model, port, TP×DP configs, server settings)
matrix.json ISL/OSL matrix with Y/P/N marks and per-ISL concurrency ranges
generate_scenarios.py Expands matrix.json into a scenario TSV
start_sglang_docker.sh Start the P800 SGLang Docker container for a TP×DP config
start_sglang_dp.sh Alias for start_sglang_docker.sh (H20 naming parity)
run_bench.sh Orchestrator: server per config → warmup → scenarios → parse → compare
compare.py Cross-config comparison report (comparison.md)
adaptive_config.env Adaptive concurrency search settings
run_adaptive_concurrency.sh Adaptive saturation search (shared adaptive_bench_lib.sh)
run_adaptive_concurrency_add16.sh Same search starting at concurrency 16, step +16

Platform

This experiment targets platforms/kunlun_p800.env. The container uses the dedicated name sglang-dsv4-flash-tpdp so it never interferes with the sglang-dsv4-flash container used by the other P800 experiments.