Phase 2 Hardware Attribution Analysis

Run

Run ID: dsv4pro-phase2-20260731-130125
Status: COMPLETED
Time: 2026-07-31 13:01:25 to 13:27:51 CST
Duration: 26 minutes 26 seconds
Topology: two Pro6000D nodes, SGLang TP16 / EP2
Bench result: 8 completed, 0 failed, no OOM

The command was run only on 174.1.51.5:

cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution

RUN_ID=dsv4pro-phase2-20260731-130125
tmux new-session -d -s dsv4pro-phase2 \
  "RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"

The entry script started the Worker and its collectors on 174.1.51.7 over SSH. The full Phase 2 command must not be run a second time on the Worker.

Endpoint Results

Case	Input TPS	Output TPS	TTFT P95	TPOT P95	E2E P95
128K -> 1, C=1	2,618.53	0.02	50.036 s	-	50.036 s
32K -> 1, C=16	3,116.20	0.10	161.899 s	-	161.899 s
1K -> 1K, C=32	447.41	447.41	10.144 s	65.63 ms	73.189 s
1K -> 4K, C=16	79.35	317.41	1.727 s	50.01 ms	206.427 s
128K -> 1K, C=1	1,610.89	12.59	48.489 s	32.12 ms	81.343 s

Mixed Prefill/Decode

Decode metric	Control	With one 128K Prefill	Change
Output TPS	454.39	345.20	-24.03%
TTFT P95	9.437 s	9.869 s	+4.58%
TPOT P95	66.17 ms	110.36 ms	+66.79%
E2E P95	72.225 s	117.941 s	+63.30%

This reproduces the Phase 1 contention result. Prefill mainly damages ongoing Decode progress rather than its first-token latency. Global token-level ITL P95 stayed near 62 ms because a small number of synchronized long stalls can remain below the top 5% of all token gaps. Per-request TPOT and E2E expose the impact more clearly.

Hardware Evidence

GPU

During the 128K Prefill injection, average GPU utilization was 99.78% on the Head and 99.34% on the Worker; average power was about 274 W on both nodes.
During 32K -> 1, C=16 Prefill, average GPU utilization was about 82% on both nodes and power was about 265 W.
GPU clocks remained around 2.38 to 2.41 GHz. There is no evidence of a frequency collapse.
GPU memory stayed near 83.2 to 83.4 GiB of 85,651 MiB per GPU, leaving only about 2.3 GiB of headroom.
Head DCGM data is valid. During the injection window, GRACT=0.995, SMACT=0.648, SMOCC=0.215, Tensor Active=0.108, and DRAM Active=0.412.

CPU

Whole-machine CPU utilization stayed around 8% to 11% across 128 logical CPUs, with zero average I/O wait.
A few individual cores were saturated. During the injection window, 5 Head cores and 4 Worker cores averaged above 80%.
Therefore the machine is not globally CPU-bound, but scheduler, affinity, or NUMA hot threads remain plausible contributors.

Inter-node RDMA

NCCL logs on both nodes verified NET/IB + GDRDMA over mlx5_0,mlx5_3.
Both rails carried nearly identical traffic and all recorded RDMA error deltas were zero.
The largest average traffic occurred in 32K -> 1, C=16 Prefill: about 140.0 Gbit/s total transmit on the Head and 139.3 Gbit/s on the Worker, or about 70 Gbit/s per 400G rail.
This is only about 17.5% of each rail's nominal line rate. Raw RoCE bandwidth saturation is therefore not the primary ceiling, although collective latency and synchronization overhead are not ruled out.

Intra-node GPU Communication

RTX Pro 6000D has no NVLink. nvidia-smi nvlink --status reports no links.
nvidia-smi topo -m shows GPU0-3 and GPU4-7 connected through local PCIe switches (PIX), while traffic between the two four-GPU groups crosses the host bridge and CPU/NUMA interconnect (SYS).
NCCL logs show intra-node transfers through P2P/IPC; on this machine that means CUDA peer-to-peer over PCIe, not NVLink.
Phase 2 collected DCGM PCIe counters but did not run a dedicated GPU-to-GPU bandwidth/latency microbenchmark. A later short diagnostic should run p2pBandwidthLatencyTest, 8-GPU all_reduce_perf, and 16-GPU all_reduce_perf.

Preliminary Attribution

The first successful Phase 2 run supports three conclusions:

Prefill/Decode resource contention is real and reproducible.
Raw dual-rail RoCE bandwidth, whole-machine CPU capacity, and clock throttling do not explain the observed slowdown.
The next useful target is the TP16 execution timeline: GPU kernels, scheduler gaps, PCIe/RDMA collectives, and synchronization between ranks.

This is a bottleneck direction, not a final kernel-level root cause. Phase 3 should capture short Prefill, Decode, and mixed timelines rather than repeat the full benchmark matrix.

Collection Limitations

Worker DCGM exited because nvidia-dcgm hostengine was inactive on 174.1.51.7. Before another hardware run, execute on the Worker:
```
systemctl start nvidia-dcgm
systemctl is-active nvidia-dcgm
dcgmi discovery -l
```
The main Phase 2 entry still runs only on the Head.
Current Case windows include benchmark-client startup, tokenizer/data preparation, warm-up, the measured run, and client exit. Hardware averages are therefore diluted outside the true main benchmark interval. Future profiling must add explicit main-run start/end markers.
pidstat -durwt at one-second intervals generated about 850 MB on the Head and 735 MB on the Worker. The next run should use process-level sampling and/or a five-second interval.
Worker teardown logged a Gloo connection reset after the Head stopped. It occurred after all eight benchmark rows completed and is a cleanup artifact, not a benchmark failure.

5.6 KiB Raw Blame History