sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling

DSV4-Pro / Pro6000D / 2-Node / SGLang Timeline Profiling

Phase 3 captures short distributed timelines for the completed TP16 baseline. It does not rerun Phase 2 hardware sampling or communication microbenchmarks.

One entry point

Run only on Head 174.1.51.5:

cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling

bash run_timeline_profiling.sh audit

DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun-$(date +%Y%m%d-%H%M%S) \
  bash run_timeline_profiling.sh all

RUN_ID=dsv4pro-phase3-torch-smoke-$(date +%Y%m%d-%H%M%S) \
  bash run_timeline_profiling.sh torch-smoke

RUN_ID=dsv4pro-phase3-nsys-smoke-$(date +%Y%m%d-%H%M%S) \
  bash run_timeline_profiling.sh nsys-smoke

After both smoke tests pass, run the three formal capture ranges in one model service lifetime:

RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
tmux new-session -d -s dsv4pro-phase3 \
  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"

tmux attach -t dsv4pro-phase3

The script starts and stops both nodes, triggers /start_profile, retrieves Worker artifacts, exports basic nsys stats, and cleans containers and Worker staging paths. No source, Conda activation, or command on Worker is required.

Formal ranges

1. decode_control: 1K -> 256, C=32.
2. mixed_decode_with_128k_prefill: active 1K -> 512, C=32 decode plus one 128K -> 1 injection.
3. long_prefill: isolated 128K -> 1, C=1.

The primary run keeps CUDA Graph enabled. A later layer-wise NVTX run with disabled graphs is allowed only if these production-faithful traces cannot map the dominant kernels to model components.

Outputs

results/<RUN_ID>/
  manifest.json
  service/
  cases/
  profiles/head/
    nsys/
    torch/
  profiles/worker/
    nsys/
    torch/

Profiler runs are diagnostic and must not replace the Phase 2 no-profiler TPS baseline.