# Phase 2 Hardware Attribution Analysis

## Run

- Run ID: `dsv4pro-phase2-20260731-130125`
- Status: `COMPLETED`
- Time: `2026-07-31 13:01:25` to `13:27:51 CST`
- Duration: 26 minutes 26 seconds
- Topology: two Pro6000D nodes, SGLang TP16 / EP2
- Bench result: 8 completed, 0 failed, no OOM

The command was run only on `174.1.51.5`:

```bash
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution

RUN_ID=dsv4pro-phase2-20260731-130125
tmux new-session -d -s dsv4pro-phase2 \
  "RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"
```

The entry script started the Worker and its collectors on `174.1.51.7` over SSH.
The full Phase 2 command must not be run a second time on the Worker.

## Endpoint Results

| Case | Input TPS | Output TPS | TTFT P95 | TPOT P95 | E2E P95 |
|---|---:|---:|---:|---:|---:|
| 128K -> 1, C=1 | 2,618.53 | 0.02 | 50.036 s | - | 50.036 s |
| 32K -> 1, C=16 | 3,116.20 | 0.10 | 161.899 s | - | 161.899 s |
| 1K -> 1K, C=32 | 447.41 | 447.41 | 10.144 s | 65.63 ms | 73.189 s |
| 1K -> 4K, C=16 | 79.35 | 317.41 | 1.727 s | 50.01 ms | 206.427 s |
| 128K -> 1K, C=1 | 1,610.89 | 12.59 | 48.489 s | 32.12 ms | 81.343 s |

## Mixed Prefill/Decode

| Decode metric | Control | With one 128K Prefill | Change |
|---|---:|---:|---:|
| Output TPS | 454.39 | 345.20 | -24.03% |
| TTFT P95 | 9.437 s | 9.869 s | +4.58% |
| TPOT P95 | 66.17 ms | 110.36 ms | +66.79% |
| E2E P95 | 72.225 s | 117.941 s | +63.30% |

This reproduces the Phase 1 contention result. Prefill mainly damages ongoing
Decode progress rather than its first-token latency. Global token-level ITL P95
stayed near 62 ms because a small number of synchronized long stalls can remain
below the top 5% of all token gaps. Per-request TPOT and E2E expose the impact
more clearly.

## Hardware Evidence

### GPU

- During the 128K Prefill injection, average GPU utilization was 99.78% on the
  Head and 99.34% on the Worker; average power was about 274 W on both nodes.
- During 32K -> 1, C=16 Prefill, average GPU utilization was about 82% on both
  nodes and power was about 265 W.
- GPU clocks remained around 2.38 to 2.41 GHz. There is no evidence of a
  frequency collapse.
- GPU memory stayed near 83.2 to 83.4 GiB of 85,651 MiB per GPU, leaving only
  about 2.3 GiB of headroom.
- Head DCGM data is valid. During the injection window, `GRACT=0.995`,
  `SMACT=0.648`, `SMOCC=0.215`, `Tensor Active=0.108`, and
  `DRAM Active=0.412`.

### CPU

- Whole-machine CPU utilization stayed around 8% to 11% across 128 logical
  CPUs, with zero average I/O wait.
- A few individual cores were saturated. During the injection window, 5 Head
  cores and 4 Worker cores averaged above 80%.
- Therefore the machine is not globally CPU-bound, but scheduler, affinity, or
  NUMA hot threads remain plausible contributors.

### Inter-node RDMA

- NCCL logs on both nodes verified `NET/IB + GDRDMA` over
  `mlx5_0,mlx5_3`.
- Both rails carried nearly identical traffic and all recorded RDMA error
  deltas were zero.
- The largest average traffic occurred in 32K -> 1, C=16 Prefill:
  about 140.0 Gbit/s total transmit on the Head and 139.3 Gbit/s on the Worker,
  or about 70 Gbit/s per 400G rail.
- This is only about 17.5% of each rail's nominal line rate. Raw RoCE bandwidth
  saturation is therefore not the primary ceiling, although collective
  latency and synchronization overhead are not ruled out.

### Intra-node GPU Communication

- RTX Pro 6000D has no NVLink. `nvidia-smi nvlink --status` reports no links.
- `nvidia-smi topo -m` shows GPU0-3 and GPU4-7 connected through local PCIe
  switches (`PIX`), while traffic between the two four-GPU groups crosses the
  host bridge and CPU/NUMA interconnect (`SYS`).
- NCCL logs show intra-node transfers through `P2P/IPC`; on this machine that
  means CUDA peer-to-peer over PCIe, not NVLink.
- Phase 2 collected DCGM PCIe counters but did not run a dedicated GPU-to-GPU
  bandwidth/latency microbenchmark. A later short diagnostic should run
  `p2pBandwidthLatencyTest`, 8-GPU `all_reduce_perf`, and 16-GPU
  `all_reduce_perf`.

## Preliminary Attribution

The first successful Phase 2 run supports three conclusions:

1. Prefill/Decode resource contention is real and reproducible.
2. Raw dual-rail RoCE bandwidth, whole-machine CPU capacity, and clock
   throttling do not explain the observed slowdown.
3. The next useful target is the TP16 execution timeline: GPU kernels,
   scheduler gaps, PCIe/RDMA collectives, and synchronization between ranks.

This is a bottleneck direction, not a final kernel-level root cause. Phase 3
should capture short Prefill, Decode, and mixed timelines rather than repeat the
full benchmark matrix.

## Collection Limitations

- Worker DCGM exited because `nvidia-dcgm` hostengine was inactive on
  `174.1.51.7`. Before another hardware run, execute on the Worker:

  ```bash
  systemctl start nvidia-dcgm
  systemctl is-active nvidia-dcgm
  dcgmi discovery -l
  ```

  The main Phase 2 entry still runs only on the Head.

- Current Case windows include benchmark-client startup, tokenizer/data
  preparation, warm-up, the measured run, and client exit. Hardware averages
  are therefore diluted outside the true main benchmark interval. Future
  profiling must add explicit main-run start/end markers.
- `pidstat -durwt` at one-second intervals generated about 850 MB on the Head
  and 735 MB on the Worker. The next run should use process-level sampling
  and/or a five-second interval.
- Worker teardown logged a Gloo connection reset after the Head stopped. It
  occurred after all eight benchmark rows completed and is a cleanup artifact,
  not a benchmark failure.