Phase 3 captures short distributed timelines for the completed TP16 baseline. It does not rerun Phase 2 hardware sampling or communication microbenchmarks.

Formal entry point

Run only on Head 174.1.51.5:

cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
tmux new-session -d -s dsv4pro-phase3 \
  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"

tmux attach -t dsv4pro-phase3

PyTorch Profiler and Nsight smoke tests have already passed on this image and topology. To inspect the exact commands without starting containers or sending requests, run DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun bash run_timeline_profiling.sh all first.

The script starts and stops both nodes, triggers /start_profile, retrieves Worker artifacts, exports basic nsys stats, and cleans containers and Worker staging paths. No source, Conda activation, or command on Worker is required.

Formal ranges

decode_control: 1K -> 1K, C=32; trigger after Decode becomes active, skip 2 engine steps, then capture 16 steps.
mixed_decode_with_128k_prefill: active 1K -> 1K, C=32 decode; capture 2 pure Decode steps before injecting one 128K -> 1 request, then retain a 32-step range spanning both sides of the injection.
long_prefill: isolated 128K -> 1, C=1.

The service remains under one Nsight process for all three ranges, so it uses --capture-range-end=repeat:3:defer rather than the single-range stop form. Every /start_profile request records both start_step and num_steps.

The primary run keeps CUDA Graph enabled. A later layer-wise NVTX run with disabled graphs is allowed only if these production-faithful traces cannot map the dominant kernels to model components.

Outputs

results/<RUN_ID>/
  manifest.json
  service/
  cases/
  profiles/head/
    nsys/
    torch/
  profiles/worker/
    nsys/
    torch/

Profiler runs are diagnostic and must not replace the Phase 2 no-profiler TPS baseline.