Phase 3 captures short distributed timelines for the completed TP16 baseline. It does not rerun Phase 2 hardware sampling or communication microbenchmarks.

Formal run dsv4pro-phase3-20260802-003213 completed at commit 82b7d91: three ranges produced three Head and three Worker .nsys-rep files, and all six reports passed validation and nsys stats export. Raw reports remain under the run directory on Head. Profiler-side throughput is diagnostic only.

Formal entry point

Run only on Head 174.1.51.5:

cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
tmux new-session -d -s dsv4pro-phase3 \
  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"

tmux attach -t dsv4pro-phase3

PyTorch Profiler and Nsight smoke tests have already passed on this image and topology. To inspect the exact commands without starting containers or sending requests, run DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun bash run_timeline_profiling.sh all first.

The script starts and stops both nodes, triggers /start_profile, retrieves Worker artifacts, exports basic nsys stats, and cleans containers and Worker staging paths. No source, Conda activation, or command on Worker is required.

Formal ranges

decode_control: 1K -> 1K, C=32; trigger after Decode becomes active, skip 2 engine steps, then capture 16 steps.
mixed_decode_with_128k_prefill: active 1K -> 1K, C=32 decode; wait for stable Decode activity, submit one native input_ids based 128K -> 1 request, and start /start_profile only after the first 8K chunk enters Prefill. This image blocks other API requests while /start_profile is active, so Control provides the pure Decode baseline and the 32-step mixed range captures the actual Prefill-plus-Decode treatment.
long_prefill: isolated 128K -> 1, C=1.

The service remains under one Nsight process for all three ranges, so it uses --capture-range-end=repeat:3:defer rather than the single-range stop form. Every /start_profile request records both start_step and num_steps.

The primary run keeps CUDA Graph enabled. A later layer-wise NVTX run with disabled graphs is allowed only if these production-faithful traces cannot map the dominant kernels to model components.

Outputs

results/<RUN_ID>/
  manifest.json
  service/
  cases/
  profiles/head/
    nsys/
    torch/
  profiles/worker/
    nsys/
    torch/

Profiler runs are diagnostic and must not replace the Phase 2 no-profiler TPS baseline.