== PP0 rank ==
  file: /data/hf_models/pp2tp4_profiles/B_decode/pb-1788706647.2466695-TP-0-PP-0.trace.json.gz
  baseTimeNanoseconds present: True
  span 1.579s  GPU-busy 1.167s (73.9%)  kernels 80049
  cudaMemcpy/Set GPU time: 0.3ms   cudaLaunchKernel calls: 757 (10.0ms CPU)
  NCCL total: 324.5ms (27.8% of busy, 20.5% of span)
    SendRecv         239.3ms  n=300
    AllReduce         85.0ms  n=4740
    AG_RS              0.3ms  n=30
  top kernels (of GPU-busy):
     39.3%      459.2ms  n=4320    _ZN7cutlass13device_kernelINS_4gemm6kernel13GemmUniversalINS1_17GroupProblemShap
     20.5%      239.3ms  n=300     ncclDevKernel_SendRecv(ncclDevKernelArgsStorage
     18.1%      211.7ms  n=13140   void cutlass::Kernel2
      7.3%       85.0ms  n=4740    ncclDevKernel_AllReduce_Sum_bf16_RING_LL(ncclDevKernelArgsStorage
      5.9%       68.4ms  n=2340    kernel_cutlass__dsv3_fused_a_gemm_kernel_tensorptri32gmemo2624307230721_tensorpt
      4.1%       47.8ms  n=2340    void flashinfer::sparse_mla_sm120::sparse_mla_decode_dsv3_2_kernel
      3.0%       35.1ms  n=2340    kernel_cutlass__dsv3_fused_a_gemm_kernel_tensorptri32gmemo4096102410241_tensorpt
      2.2%       25.8ms  n=2160    void tensorrt_llm::kernels::cutlass_kernels::finalizeMoeRoutingKernel
      1.4%       16.1ms  n=4620    kernel_cutlass_kernel_flashinfernormkernelsfused_add_rmsnormFusedAddRMSNormKerne
      1.3%       14.7ms  n=2160    void sglang::router_gemm_kernel
      1.0%       11.7ms  n=720     void deep_gemm::sm120_fp8_paged_mqa_logits
      0.9%       10.5ms  n=2160    void tensorrt_llm::kernels::cutlass_kernels::expandInputRowsKernel
      0.8%        9.4ms  n=4680    kernel_cutlass_kernel_flashinfernormkernelsrmsnormRMSNormKernel_object_at__tenso
      0.8%        9.0ms  n=2160    void tensorrt_llm::kernels::cutlass_kernels::doActivationKernel
      0.7%        8.1ms  n=2160    void tensorrt_llm::kernels::cutlass_kernels::fusedBuildExpertMapsSortFirstTokenK
  forwards: 12 (gap>2.0ms split)
    wall ms: mean 124.1  p50 54.0  min 10.7  max 386.7
    small forwards (<30% of p50): 1  (warmup/mixed steps)
    GPU-covered fraction inside main forwards: mean 76.5%
    inter-forward gap ms: mean 8.2  p50 2.3  p90 4.4  max 64.0

== PP1 rank ==
  file: /data/hf_models/pp2tp4_profiles/B_decode/pb-1788706647.2466695-TP-0-PP-1.trace.json.gz
  baseTimeNanoseconds present: True
  span 12.096s  GPU-busy 1.300s (10.7%)  kernels 81315
  cudaMemcpy/Set GPU time: 0.2ms   cudaLaunchKernel calls: 997 (11.2ms CPU)
  NCCL total: 255.8ms (19.7% of busy, 2.1% of span)
    AllReduce        112.4ms  n=4680
    AG_RS             85.7ms  n=273
    SendRecv          57.7ms  n=303
  top kernels (of GPU-busy):
     42.4%      551.1ms  n=4680    _ZN7cutlass13device_kernelINS_4gemm6kernel13GemmUniversalINS1_17GroupProblemShap
     16.7%      216.9ms  n=12840   void cutlass::Kernel2
      8.6%      112.4ms  n=4680    ncclDevKernel_AllReduce_Sum_bf16_RING_LL(ncclDevKernelArgsStorage
      6.6%       85.7ms  n=273     ncclDevKernel_AllGather_RING_LL(ncclDevKernelArgsStorage
      5.3%       68.5ms  n=2340    kernel_cutlass__dsv3_fused_a_gemm_kernel_tensorptri32gmemo2624307230721_tensorpt
      4.4%       57.7ms  n=303     ncclDevKernel_SendRecv(ncclDevKernelArgsStorage
      3.7%       47.7ms  n=2340    void flashinfer::sparse_mla_sm120::sparse_mla_decode_dsv3_2_kernel
      2.7%       35.3ms  n=2340    kernel_cutlass__dsv3_fused_a_gemm_kernel_tensorptri32gmemo4096102410241_tensorpt
      2.2%       28.0ms  n=2340    void tensorrt_llm::kernels::cutlass_kernels::finalizeMoeRoutingKernel
      1.3%       16.5ms  n=4740    kernel_cutlass_kernel_flashinfernormkernelsfused_add_rmsnormFusedAddRMSNormKerne
      1.2%       16.0ms  n=2340    void sglang::router_gemm_kernel
      0.9%       11.4ms  n=2340    void tensorrt_llm::kernels::cutlass_kernels::expandInputRowsKernel
      0.8%       10.0ms  n=2340    void tensorrt_llm::kernels::cutlass_kernels::doActivationKernel
      0.7%        9.3ms  n=4680    kernel_cutlass_kernel_flashinfernormkernelsrmsnormRMSNormKernel_object_at__tenso
      0.7%        8.8ms  n=2340    void tensorrt_llm::kernels::cutlass_kernels::fusedBuildExpertMapsSortFirstTokenK
  forwards: 11 (gap>2.0ms split)
    wall ms: mean 142.6  p50 56.0  min 5.6  max 346.6
    small forwards (<30% of p50): 2  (warmup/mixed steps)
    GPU-covered fraction inside main forwards: mean 84.5%
    inter-forward gap ms: mean 1052.8  p50 2.6  p90 10503.3  max 10503.3

== cross-stage bins (1.0ms, total 12.17s) ==
  both busy :     1.44s   11.8%
  only PP0  :     0.03s    0.3%   <- PP1 bubble
  only PP1  :     0.13s    1.1%   <- PP0 bubble
  neither   :    10.56s   86.8%   <- sched/CPU idle
  pipeline efficiency: both/(any busy) = 89.6%
