[Feat] finalize Phase 3 SGLang timeline capture

README.md

@@ -1,5 +1,9 @@
 # sskj — 多平台大模型推理性能基准测试项目
 
+> **更新（2026-08-01 19:34:34 CST）**
+>
+> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 正式时间线代码交付。正式 Control 与混合背景统一为 `1K -> 1K, C=32`；所有 `/start_profile` 请求显式记录 `start_step` 和 `num_steps`，Control 在活跃 Decode 后跳过 2 Step，混合场景先保留 2 个纯 Decode Step 再注入 `128K -> 1`，长 Prefill 从首个 Chunk 开始捕获。继续沿用已通过的双节点 PyTorch/Nsight smoke、16 Rank/6 份 Nsight 报告 fail-closed 门禁和单服务三段 Capture Range；正式结果尚未生成，因此不创建 `phase3_exp.html` 或 `phase3_code.html`。
+>
 > **更新（2026-08-01 15:24:52 CST）**
 >
 > 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模。正式 Run `dsv4pro-phase2_5-20260801-130007` 完成 Scout 5/5 与 Confirm 6/6；64K Prefill 在 C=16 已进入约 2,984 input tok/s、79.90 Gbit/s/rail 的平台，C=32/64 不再显著增长。拟合通信强度为 3.332 MB/input-token/rail，单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍，因此当前是模型计算/实现吞吐先饱和，不是 RDMA 先饱和。新增 `phase2_5_exp.html`、`phase2_5_code.html`、精简证据集和可复用的模型部署 RDMA 需求评估流程；实验结束后双节点容器与 16 张 GPU 已清理。

docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html

@@ -837,7 +837,7 @@ NVTX 只是在时间线上添加可读标记。启用 NVTX 不等于已经启动
 </p>
 <h3>7.2 捕获策略</h3>
 <ul>
-<li>只捕获预热后的 5 到 10 个 Engine Step。</li>
+<li>只捕获预热后的 10 到 32 个 Engine Step；混合场景窗口略长，用于覆盖 Prefill 注入前后。</li>
 <li>Prefill、Decode 和混合干扰分别生成报告。</li>
 <li>两台机器分别保存原始报告。</li>
 <li>优先保留所有 Rank；文件过大时至少保留代表 Rank 和跨机通信相关 Rank。</li>
@@ -890,7 +890,36 @@ SGLang 进程，并绑定相同 Case ID 和时间窗口。正式执行前先验
 <code>nsys</code> 版本、子进程跟踪方式及动态 Capture 机制，再固化命令；
 不直接对整轮 Benchmark 做长时间全量捕获。
 </p>
-<h3>7.4 时间线必须回答的问题</h3>
+<h3>7.4.1 当前实现与正式入口</h3>
+<p>
+代码已集中在
+<code>experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling</code>，
+仍只有 <code>run_timeline_profiling.sh</code> 一个入口。双节点 PyTorch Profiler
+和 Nsight smoke 已通过；正式 Run 尚未执行，因此暂不生成 Phase 3 的
+<code>exp/code</code> HTML 档案。
+</p>
+<table>
+<thead><tr><th>Case</th><th>负载</th><th>动态窗口</th></tr></thead>
+<tbody>
+<tr><td>Decode Control</td><td>1K → 1K，C=32</td><td>确认 Decode 活跃后跳过 2 Step，捕获 16 Step</td></tr>
+<tr><td>Mixed Treatment</td><td>1K → 1K，C=32 背景 + 128K → 1 注入</td><td>先保留 2 个纯 Decode Step，再注入 Prefill；总计捕获 32 Step</td></tr>
+<tr><td>Long Prefill</td><td>128K → 1，C=1</td><td>从首个 Chunk 开始捕获 16 Step</td></tr>
+</tbody>
+</table>
+<p>
+三段正式范围复用一次模型加载，因此 Nsight 使用
+<code>--capture-range-end=repeat:3:defer</code>。它与官方单段示例中的
+<code>stop</code> 目的相同，但允许同一个服务依次产出三段独立报告。
+</p>
+<pre><code class="language-bash"># 只在 Head 174.1.51.5 执行
+cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
+RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
+tmux new-session -d -s dsv4pro-phase3 \
+  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
+   2&gt;&amp;1 | tee /data/hzy/${RUN_ID}.log"
+tmux attach -t dsv4pro-phase3
+</code></pre>
+<h3>7.5 时间线必须回答的问题</h3>
 <ol>
 <li>Prefill 和 Decode 各自的 Top Kernel 是什么？</li>
 <li>NCCL 在关键路径上的暴露时间是多少？</li>
@@ -903,7 +932,7 @@ SGLang 进程，并绑定相同 Case ID 和时间窗口。正式执行前先验
 <li>NSA Indexer 的成本占 Sparse Attention 总成本多少？</li>
 <li>长 Prefill 到来时，Decode Kernel 为什么被延迟？</li>
 </ol>
-<h3>7.5 报告分析</h3>
+<h3>7.6 报告分析</h3>
 <pre><code class="language-bash">nsys stats &lt;REPORT&gt;.nsys-rep
 </code></pre>
 <p>重点查看：</p>

docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md

@@ -320,7 +320,7 @@ NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING
 
 ### 7.1 捕获策略
 
-- 只捕获预热后的 5 到 10 个 Engine Step。
+- 只捕获预热后的 10 到 32 个 Engine Step；混合场景窗口略长，用于覆盖 Prefill 注入前后。
 - Prefill、Decode 和混合干扰分别生成报告。
 - 两台机器分别保存原始报告。
 - 优先保留所有 Rank；文件过大时至少保留代表 Rank 和跨机通信相关 Rank。
@@ -379,6 +379,31 @@ curl -X POST http://127.0.0.1:30000/start_profile \
 
 多机 Trace 自动合并要求两台机器能访问同一个共享输出目录。没有共享目录时分别保存，再在本地汇总。
 
+当前实现位于：
+
+```text
+experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
+```
+
+正式场景为：
+
+- Decode Control：`1K -> 1K, C=32`，确认 Decode 活跃后跳过 2 Step，捕获 16 Step。
+- Mixed Treatment：`1K -> 1K, C=32` 背景先捕获 2 个纯 Decode Step，再注入 `128K -> 1`，总计捕获 32 Step。
+- Long Prefill：`128K -> 1, C=1`，从首个 Chunk 开始捕获 16 Step。
+
+双节点 PyTorch/Nsight smoke 已通过。正式 Run 尚未执行，因此暂不生成 Phase 3 的 `exp/code` HTML。正式命令只在 Head 执行：
+
+```bash
+cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
+RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
+tmux new-session -d -s dsv4pro-phase3 \
+  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
+   2>&1 | tee /data/hzy/${RUN_ID}.log"
+tmux attach -t dsv4pro-phase3
+```
+
+三段范围复用一次模型加载，因此 Nsight 使用 `--capture-range-end=repeat:3:defer`，而不是单段示例中的 `stop`。
+
 ### 7.4 时间线必须回答的问题
 
 1. Prefill 和 Decode 各自的 Top Kernel 是什么？

experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/README.md

@@ -3,29 +3,12 @@
 Phase 3 captures short distributed timelines for the completed TP16 baseline.
 It does not rerun Phase 2 hardware sampling or communication microbenchmarks.
 
-## One entry point
+## Formal entry point
 
 Run only on Head `174.1.51.5`:
 
 ```bash
 cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
-
-bash run_timeline_profiling.sh audit
-
-DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun-$(date +%Y%m%d-%H%M%S) \
-  bash run_timeline_profiling.sh all
-
-RUN_ID=dsv4pro-phase3-torch-smoke-$(date +%Y%m%d-%H%M%S) \
-  bash run_timeline_profiling.sh torch-smoke
-
-RUN_ID=dsv4pro-phase3-nsys-smoke-$(date +%Y%m%d-%H%M%S) \
-  bash run_timeline_profiling.sh nsys-smoke
-```
-
-After both smoke tests pass, run the three formal capture ranges in one model
-service lifetime:
-
-```bash
 RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
 tmux new-session -d -s dsv4pro-phase3 \
   "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
@@ -34,16 +17,28 @@ tmux new-session -d -s dsv4pro-phase3 \
 tmux attach -t dsv4pro-phase3
 ```
 
+PyTorch Profiler and Nsight smoke tests have already passed on this image and
+topology. To inspect the exact commands without starting containers or sending
+requests, run `DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun bash
+run_timeline_profiling.sh all` first.
+
 The script starts and stops both nodes, triggers `/start_profile`, retrieves
 Worker artifacts, exports basic `nsys stats`, and cleans containers and Worker
 staging paths. No `source`, Conda activation, or command on Worker is required.
 
 ## Formal ranges
 
-1. `decode_control`: `1K -> 256`, `C=32`.
+1. `decode_control`: `1K -> 1K`, `C=32`; trigger after Decode becomes active,
-2. `mixed_decode_with_128k_prefill`: active `1K -> 512`, `C=32` decode plus one `128K -> 1` injection.
+   skip 2 engine steps, then capture 16 steps.
+2. `mixed_decode_with_128k_prefill`: active `1K -> 1K`, `C=32` decode;
+   capture 2 pure Decode steps before injecting one `128K -> 1` request, then
+   retain a 32-step range spanning both sides of the injection.
 3. `long_prefill`: isolated `128K -> 1`, `C=1`.
 
+The service remains under one Nsight process for all three ranges, so it uses
+`--capture-range-end=repeat:3:defer` rather than the single-range `stop` form.
+Every `/start_profile` request records both `start_step` and `num_steps`.
+
 The primary run keeps CUDA Graph enabled. A later layer-wise NVTX run with
 disabled graphs is allowed only if these production-faithful traces cannot map
 the dominant kernels to model components.

experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/config.env

@@ -22,16 +22,22 @@ NSYS_MIXED_STEPS="${NSYS_MIXED_STEPS:-32}"
 NSYS_PREFILL_STEPS="${NSYS_PREFILL_STEPS:-16}"
 NSYS_SMOKE_STEPS="${NSYS_SMOKE_STEPS:-5}"
 TORCH_SMOKE_STEPS="${TORCH_SMOKE_STEPS:-3}"
+TORCH_SMOKE_START_STEP="${TORCH_SMOKE_START_STEP:-1}"
+NSYS_SMOKE_START_STEP="${NSYS_SMOKE_START_STEP:-1}"
+NSYS_CONTROL_START_STEP="${NSYS_CONTROL_START_STEP:-2}"
+NSYS_MIXED_START_STEP="${NSYS_MIXED_START_STEP:-0}"
+NSYS_PREFILL_START_STEP="${NSYS_PREFILL_START_STEP:-0}"
+MIXED_PRE_INJECTION_DECODE_STEPS="${MIXED_PRE_INJECTION_DECODE_STEPS:-2}"
 
 # Short profiling workloads. They reproduce the Phase 2 scheduler shapes but
 # avoid its full result matrix and long hardware sampling.
 CONTROL_ISL="${CONTROL_ISL:-1024}"
-CONTROL_OSL="${CONTROL_OSL:-256}"
+CONTROL_OSL="${CONTROL_OSL:-1024}"
 CONTROL_CONCURRENCY="${CONTROL_CONCURRENCY:-32}"
 CONTROL_PROMPTS="${CONTROL_PROMPTS:-32}"
 
 MIXED_ISL="${MIXED_ISL:-1024}"
-MIXED_OSL="${MIXED_OSL:-512}"
+MIXED_OSL="${MIXED_OSL:-1024}"
 MIXED_CONCURRENCY="${MIXED_CONCURRENCY:-32}"
 MIXED_PROMPTS="${MIXED_PROMPTS:-64}"
 MIXED_INJECTION_ISL="${MIXED_INJECTION_ISL:-131072}"

experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh

@@ -367,15 +367,16 @@ profile_done_count() {
 start_profile() {
   local case_id="$1"
   local activities_json="$2"
-  local steps="$3"
+  local start_step="$3"
+  local steps="$4"
   local output_dir="${PROFILE_ROOT_IN_CONTAINER}/torch/${case_id}"
   local response_file="${CASE_DIR}/${case_id}/start_profile_response.json"
   local payload
-  payload="{\"output_dir\":\"${output_dir}\",\"num_steps\":${steps},\"activities\":${activities_json},\"profile_id\":\"${RUN_ID}-${case_id}\",\"profile_prefix\":\"${case_id}\",\"merge_profiles\":false}"
+  payload="{\"output_dir\":\"${output_dir}\",\"start_step\":${start_step},\"num_steps\":${steps},\"activities\":${activities_json},\"profile_id\":\"${RUN_ID}-${case_id}\",\"profile_prefix\":\"${case_id}\",\"merge_profiles\":false}"
   mkdir -p "${CASE_DIR}/${case_id}"
   printf '%s\n' "${payload}" > "${CASE_DIR}/${case_id}/start_profile_request.json"
   if [[ "${DRY_RUN}" == "1" ]]; then
-    log "[DRY] POST /start_profile case=${case_id} steps=${steps} activities=${activities_json}"
+    log "[DRY] POST /start_profile case=${case_id} start_step=${start_step} steps=${steps} activities=${activities_json}"
     return 0
   fi
   curl --fail --silent --show-error -X POST \
@@ -407,9 +408,16 @@ server_decode_count() {
 wait_for_decode_activity() {
   local before="$1"
   local bench_pid="$2"
+  wait_for_decode_steps "${before}" 1 "${bench_pid}"
+}
+
+wait_for_decode_steps() {
+  local before="$1"
+  local required_steps="$2"
+  local bench_pid="$3"
   local waited=0
   while (( waited < 180 )); do
-    if (( $(server_decode_count) > before )); then
+    if (( $(server_decode_count) >= before + required_steps )); then
       return 0
     fi
     kill -0 "${bench_pid}" 2>/dev/null || return 1
@@ -482,28 +490,30 @@ run_bench() {
 run_profiled_case() {
   local case_id="$1"
   local activities_json="$2"
-  local steps="$3"
+  local start_step="$3"
-  local isl="$4"
+  local steps="$4"
-  local osl="$5"
+  local isl="$5"
-  local concurrency="$6"
+  local osl="$6"
-  local prompts="$7"
+  local concurrency="$7"
-  local seed="$8"
+  local prompts="$8"
+  local seed="$9"
   local before=0
   flush_cache
   [[ "${DRY_RUN}" == "1" ]] || before="$(profile_done_count)"
-  start_profile "${case_id}" "${activities_json}" "${steps}"
+  start_profile "${case_id}" "${activities_json}" "${start_step}" "${steps}"
   run_bench "${case_id}" "${isl}" "${osl}" "${concurrency}" "${prompts}" "${seed}"
   [[ "${DRY_RUN}" == "1" ]] || wait_for_profile_stop "${before}"
 }
 
 run_profiled_decode_case() {
   local case_id="$1"
-  local steps="$2"
+  local start_step="$2"
-  local isl="$3"
+  local steps="$3"
-  local osl="$4"
+  local isl="$4"
-  local concurrency="$5"
+  local osl="$5"
-  local prompts="$6"
+  local concurrency="$6"
-  local seed="$7"
+  local prompts="$7"
+  local seed="$8"
   local case_path="${CASE_DIR}/${case_id}"
   local decode_before=0
   local profile_before=0
@@ -517,7 +527,7 @@ run_profiled_decode_case() {
     log "[DRY] start decode workload, wait for Decode batch, then start Nsight range"
     printf '[DRY] '
     cat "${case_path}/bench_cmd.txt"
-    start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}"
+    start_profile "${case_id}" '["CUDA_PROFILER"]' "${start_step}" "${steps}"
     return 0
   fi
   decode_before="$(server_decode_count)"
@@ -532,7 +542,7 @@ run_profiled_decode_case() {
       return 1
     }
   profile_before="$(profile_done_count)"
-  start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}"
+  start_profile "${case_id}" '["CUDA_PROFILER"]' "${start_step}" "${steps}"
   wait_for_profile_stop "${profile_before}"
   wait "${bench_pid}"
 }
@@ -557,6 +567,7 @@ run_mixed_profile() {
   local background_path="${CASE_DIR}/${background_id}"
   local before=0
   local decode_before=0
+  local mixed_baseline_before=0
   flush_cache
   mkdir -p "${background_path}"
   prepare_bench_command "${background_path}/bench.jsonl" \
@@ -565,7 +576,9 @@ run_mixed_profile() {
     "${PROFILE_TIMEOUT_S}s" "${BENCH_CMD[@]}" > "${background_path}/bench_cmd.txt"
   if [[ "${DRY_RUN}" == "1" ]]; then
     log "[DRY] start decode background, wait for Decode batch, start profile, inject 128K prefill"
-    start_profile "${case_id}" '["CUDA_PROFILER"]' "${NSYS_MIXED_STEPS}"
+    start_profile "${case_id}" '["CUDA_PROFILER"]' \
+      "${NSYS_MIXED_START_STEP}" "${NSYS_MIXED_STEPS}"
+    log "[DRY] wait ${MIXED_PRE_INJECTION_DECODE_STEPS} Decode steps before injection"
     run_bench "${injection_id}" "${MIXED_INJECTION_ISL}" \
       "${MIXED_INJECTION_OSL}" 1 1 5202
     return 0
@@ -582,7 +595,16 @@ run_mixed_profile() {
     return 1
   }
   before="$(profile_done_count)"
-  start_profile "${case_id}" '["CUDA_PROFILER"]' "${NSYS_MIXED_STEPS}"
+  start_profile "${case_id}" '["CUDA_PROFILER"]' \
+    "${NSYS_MIXED_START_STEP}" "${NSYS_MIXED_STEPS}"
+  mixed_baseline_before="$(server_decode_count)"
+  wait_for_decode_steps "${mixed_baseline_before}" \
+    "${MIXED_PRE_INJECTION_DECODE_STEPS}" "${background_pid}" || {
+    kill "${background_pid}" 2>/dev/null || true
+    wait "${background_pid}" 2>/dev/null || true
+    log "ERROR: mixed profile did not capture the pre-injection Decode baseline"
+    return 1
+  }
   run_bench "${injection_id}" "${MIXED_INJECTION_ISL}" \
     "${MIXED_INJECTION_OSL}" 1 1 5202
   wait_for_profile_stop "${before}"
@@ -610,7 +632,12 @@ write_manifest() {
   "docker_image": "${DOCKER_IMAGE}",
   "cuda_graph_enabled": true,
   "tp_size": ${TP_SIZE},
-  "ep_size": ${EP_SIZE}
+  "ep_size": ${EP_SIZE},
+  "profile_windows": {
+    "decode_control": {"start_step": ${NSYS_CONTROL_START_STEP}, "num_steps": ${NSYS_CONTROL_STEPS}},
+    "mixed": {"start_step": ${NSYS_MIXED_START_STEP}, "num_steps": ${NSYS_MIXED_STEPS}, "pre_injection_decode_steps": ${MIXED_PRE_INJECTION_DECODE_STEPS}},
+    "long_prefill": {"start_step": ${NSYS_PREFILL_START_STEP}, "num_steps": ${NSYS_PREFILL_STEPS}}
+  }
 }
 EOF
 }
@@ -648,7 +675,8 @@ run_torch_smoke() {
   start_service
   write_manifest RUNNING
   run_profiled_case torch_decode_control '["CPU","GPU"]' \
-    "${TORCH_SMOKE_STEPS}" "${CONTROL_ISL}" 128 8 8 5101
+    "${TORCH_SMOKE_START_STEP}" "${TORCH_SMOKE_STEPS}" \
+    "${CONTROL_ISL}" 128 8 8 5101
   stop_service
   CLEANUP_ON_EXIT=0
   validate_torch_traces
@@ -662,7 +690,8 @@ run_nsys_smoke() {
   start_service
   write_manifest RUNNING
   run_profiled_case nsys_decode_control '["CUDA_PROFILER"]' \
-    "${NSYS_SMOKE_STEPS}" "${CONTROL_ISL}" "${CONTROL_OSL}" \
+    "${NSYS_SMOKE_START_STEP}" "${NSYS_SMOKE_STEPS}" \
+    "${CONTROL_ISL}" "${CONTROL_OSL}" \
     "${CONTROL_CONCURRENCY}" "${CONTROL_PROMPTS}" 5102
   stop_service
   CLEANUP_ON_EXIT=0
@@ -678,11 +707,13 @@ run_all() {
   start_service
   write_manifest RUNNING
   run_profiled_decode_case decode_control \
-    "${NSYS_CONTROL_STEPS}" "${CONTROL_ISL}" "${CONTROL_OSL}" \
+    "${NSYS_CONTROL_START_STEP}" "${NSYS_CONTROL_STEPS}" \
+    "${CONTROL_ISL}" "${CONTROL_OSL}" \
     "${CONTROL_CONCURRENCY}" "${CONTROL_PROMPTS}" 5301
   run_mixed_profile
   run_profiled_case long_prefill '["CUDA_PROFILER"]' \
-    "${NSYS_PREFILL_STEPS}" "${PREFILL_ISL}" "${PREFILL_OSL}" 1 1 5303
+    "${NSYS_PREFILL_START_STEP}" "${NSYS_PREFILL_STEPS}" \
+    "${PREFILL_ISL}" "${PREFILL_OSL}" 1 1 5303
   stop_service
   CLEANUP_ON_EXIT=0
   validate_nsys_reports