# DSV4-Pro / Pro6000D / 2-Node / SGLang Timeline Profiling

Phase 3 captures short distributed timelines for the completed TP16 baseline.
It does not rerun Phase 2 hardware sampling or communication microbenchmarks.

## Formal entry point

Run only on Head `174.1.51.5`:

```bash
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
tmux new-session -d -s dsv4pro-phase3 \
  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"

tmux attach -t dsv4pro-phase3
```

PyTorch Profiler and Nsight smoke tests have already passed on this image and
topology. To inspect the exact commands without starting containers or sending
requests, run `DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun bash
run_timeline_profiling.sh all` first.

The script starts and stops both nodes, triggers `/start_profile`, retrieves
Worker artifacts, exports basic `nsys stats`, and cleans containers and Worker
staging paths. No `source`, Conda activation, or command on Worker is required.

## Formal ranges

1. `decode_control`: `1K -> 1K`, `C=32`; trigger after Decode becomes active,
   skip 2 engine steps, then capture 16 steps.
2. `mixed_decode_with_128k_prefill`: active `1K -> 1K`, `C=32` decode;
   capture 2 pure Decode steps before injecting one `128K -> 1` request, then
   retain a 32-step range spanning both sides of the injection.
3. `long_prefill`: isolated `128K -> 1`, `C=1`.

The service remains under one Nsight process for all three ranges, so it uses
`--capture-range-end=repeat:3:defer` rather than the single-range `stop` form.
Every `/start_profile` request records both `start_step` and `num_steps`.

The primary run keeps CUDA Graph enabled. A later layer-wise NVTX run with
disabled graphs is allowed only if these production-faithful traces cannot map
the dominant kernels to model components.

## Outputs

```text
results/<RUN_ID>/
  manifest.json
  service/
  cases/
  profiles/head/
    nsys/
    torch/
  profiles/worker/
    nsys/
    torch/
```

Profiler runs are diagnostic and must not replace the Phase 2 no-profiler TPS
baseline.