[Experiment] GLM-5.3-NVFP4 TP4PP2 torch-profiler profile on 174.1.60.5 (2026-09-06)
- Scenarios: A prefill 16k/1 cc16, B decode 16k/512 cc16, D 128k/90% prefix-hit cc8, E NCCL_PROTO=Simple protocol control
- Key findings: PP bubble disproven (96% both-stage busy, chunked prefill = natural microbatch pipeline);
prefill wall = PP0 chain compute 22.8s(65%) + TP4 AllReduce 10.4s(30%);
AR 96MB/3.7ms => busbw ~39GB/s ~ 75% of practical PCIe Gen5 (near-optimal, RING_LL kernel name is NCCL 2.28 mixed-plan naming);
NCCL_PROTO forcing refuted: docker -e never reaches scheduler ranks (sglang worker spawn drops NCCL env);
decode step 21-26ms (MoE GEMM 35%, SendRecv 18%, PP1 AllGather 273 launches outside cuda graph);
128k/90%: DSA indexer sm120_fp8_mqa_logits 28%/23% = 1.6x the sparse attention it serves;
with_stack pathological on this nightly (2 repros) - flush wedges server
- Full report: D:\sskj\reports\GLM53_TP4PP2_Profile与优化方向_2026-09-06.md
- Raw traces remain on 60.5:/data/hf_models/pp2tp4_profiles/{A_prefill,B_decode,D_128k,E_nccl_simple}/
1e8c36b7d1