qqtang/sskj
1
3
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
sskj/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/analysis.md
Zhiyi Hong 1dc5612e3a [Docs] finalize Phase 2 hardware attribution
2026-07-31 17:38:36 +08:00

2.3 KiB
Raw Blame History

Phase 2 Hardware Attribution Summary

Run

  • Run ID: dsv4pro-phase2-20260731-163620
  • Status: COMPLETED
  • Duration: 2026-07-31 16:36:20 to 17:05:04 CST, 28m44s
  • Benchmarks: 8/8 successful
  • Precise benchmark windows: 8/8
  • Collectors: 18 started, 18 stopped
  • Run commit: 5f24b7d22f98108f6cc234edba6768d55ea0a962
  • Git dirty: false

Main Result

Injecting one 128K prefill request into the 1K -> 1K, C=32 decode workload caused:

Metric Control Treatment Delta
Output TPS 453.55 344.89 -23.96%
TTFT P95 9.443 s 9.892 s +4.76%
TPOT P95 66.24 ms 110.45 ms +66.75%
E2E P95 72.270 s 118.076 s +63.38%

The dominant impact is sustained decode slowdown, not only first-token queueing.

Attribution

  • GPU utilization is generally 94-99%, with stable 2.39-2.42 GHz clocks. There is no evidence of frequency collapse.
  • Whole-host CPU active is about 9.6-10.4%; CPU capacity is not globally saturated.
  • Head and Worker show similar GPU, CPU and communication behavior. There is no obvious slow-node imbalance.
  • The highest measured RDMA traffic is about 83.5 Gbit/s per 400G rail. Both rails are balanced, with zero wait/error/discard/retry deltas.
  • PCIe P2P bandwidth is about 53.5 GB/s within a switch and 52.3-52.4 GB/s across NUMA, only about a 2.3% difference.
  • 16-GPU 1 GiB AllReduce bus bandwidth is 39.345/39.685/39.530 GB/s for NCCL_CROSS_NIC=0/1/2. The spread is below 1%.

The evidence excludes raw rail saturation, whole-host CPU saturation, GPU clock throttling, severe PCIe cross-NUMA collapse, and NCCL_CROSS_NIC selection as the primary cause.

Phase 3 Target

Capture short Control/Treatment timelines and identify:

  1. Attention/indexer and MoE kernel serialization.
  2. NCCL collective placement and compute/communication overlap.
  3. Scheduler or CPU launch gaps.
  4. Slow-rank synchronization around injected prefill.

Do not repeat Phase 2's long-running DCGM, CPU, RDMA or communication microbenchmarks.

Cleanup

After the run, both nodes had no related Docker containers, no GPU compute processes, and no listeners on the Phase 2 service ports. Worker Gloo Connection closed by peer tracebacks occurred during orchestrated shutdown after all benchmarks completed and did not affect the results.