# Phase 2 Hardware Attribution Summary

## Run

- Run ID: `dsv4pro-phase2-20260731-163620`
- Status: `COMPLETED`
- Duration: 2026-07-31 16:36:20 to 17:05:04 CST, 28m44s
- Benchmarks: 8/8 successful
- Precise benchmark windows: 8/8
- Collectors: 18 started, 18 stopped
- Run commit: `5f24b7d22f98108f6cc234edba6768d55ea0a962`
- Git dirty: `false`

## Main Result

Injecting one 128K prefill request into the `1K -> 1K, C=32` decode workload caused:

| Metric | Control | Treatment | Delta |
|---|---:|---:|---:|
| Output TPS | 453.55 | 344.89 | -23.96% |
| TTFT P95 | 9.443 s | 9.892 s | +4.76% |
| TPOT P95 | 66.24 ms | 110.45 ms | +66.75% |
| E2E P95 | 72.270 s | 118.076 s | +63.38% |

The dominant impact is sustained decode slowdown, not only first-token queueing.

## Attribution

- GPU utilization is generally 94-99%, with stable 2.39-2.42 GHz clocks. There is no evidence of frequency collapse.
- Whole-host CPU active is about 9.6-10.4%; CPU capacity is not globally saturated.
- Head and Worker show similar GPU, CPU and communication behavior. There is no obvious slow-node imbalance.
- The highest measured RDMA traffic is about 83.5 Gbit/s per 400G rail. Both rails are balanced, with zero wait/error/discard/retry deltas.
- PCIe P2P bandwidth is about 53.5 GB/s within a switch and 52.3-52.4 GB/s across NUMA, only about a 2.3% difference.
- 16-GPU 1 GiB AllReduce bus bandwidth is 39.345/39.685/39.530 GB/s for `NCCL_CROSS_NIC=0/1/2`. The spread is below 1%.

The evidence excludes raw rail saturation, whole-host CPU saturation, GPU clock throttling, severe PCIe cross-NUMA collapse, and `NCCL_CROSS_NIC` selection as the primary cause.

## Phase 3 Target

Capture short Control/Treatment timelines and identify:

1. Attention/indexer and MoE kernel serialization.
2. NCCL collective placement and compute/communication overlap.
3. Scheduler or CPU launch gaps.
4. Slow-rank synchronization around injected prefill.

Do not repeat Phase 2's long-running DCGM, CPU, RDMA or communication microbenchmarks.

## Cleanup

After the run, both nodes had no related Docker containers, no GPU compute processes, and no listeners on the Phase 2 service ports. Worker Gloo `Connection closed by peer` tracebacks occurred during orchestrated shutdown after all benchmarks completed and did not affect the results.