# DSV4-Pro / Pro6000D / 2-Node / SGLang Timeline Profiling

Phase 3 captures short distributed timelines for the completed TP16 baseline.
It does not rerun Phase 2 hardware sampling or communication microbenchmarks.

## One entry point

Run only on Head `174.1.51.5`:

```bash
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling

bash run_timeline_profiling.sh audit

DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun-$(date +%Y%m%d-%H%M%S) \
  bash run_timeline_profiling.sh all

RUN_ID=dsv4pro-phase3-torch-smoke-$(date +%Y%m%d-%H%M%S) \
  bash run_timeline_profiling.sh torch-smoke

RUN_ID=dsv4pro-phase3-nsys-smoke-$(date +%Y%m%d-%H%M%S) \
  bash run_timeline_profiling.sh nsys-smoke
```

After both smoke tests pass, run the three formal capture ranges in one model
service lifetime:

```bash
RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
tmux new-session -d -s dsv4pro-phase3 \
  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"

tmux attach -t dsv4pro-phase3
```

The script starts and stops both nodes, triggers `/start_profile`, retrieves
Worker artifacts, exports basic `nsys stats`, and cleans containers and Worker
staging paths. No `source`, Conda activation, or command on Worker is required.

## Formal ranges

1. `decode_control`: `1K -> 256`, `C=32`.
2. `mixed_decode_with_128k_prefill`: active `1K -> 512`, `C=32` decode plus one `128K -> 1` injection.
3. `long_prefill`: isolated `128K -> 1`, `C=1`.

The primary run keeps CUDA Graph enabled. A later layer-wise NVTX run with
disabled graphs is allowed only if these production-faithful traces cannot map
the dominant kernels to model components.

## Outputs

```text
results/<RUN_ID>/
  manifest.json
  service/
  cases/
  profiles/head/
    nsys/
    torch/
  profiles/worker/
    nsys/
    torch/
```

Profiler runs are diagnostic and must not replace the Phase 2 no-profiler TPS
baseline.