[Docs] finalize Phase 3 timeline profiling results

[BugFix] capture mixed trace after prefill admission
[BugFix] align mixed Phase 3 capture with prefill injection
2026-08-02 01:16:49 +08:00 · 2026-08-02 00:30:46 +08:00 · 2026-08-02 00:05:54 +08:00 · 2026-08-01 19:38:36 +08:00
6 changed files with 253 additions and 59 deletions
--- a/README.md
+++ b/README.md
@ -1,5 +1,17 @@
 # sskj — 多平台大模型推理性能基准测试项目

+> **更新（2026-08-02 01:25:30 CST）**
+>
+> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 时间线 Profiling。正式 Run `dsv4pro-phase3-20260802-003213` 在 21 分 03 秒内完成 Decode Control、Decode+128K Prefill Mixed、独立 128K Prefill 三段 Capture；Head/Worker 各生成 3 份 Nsight Systems 报告及 Stats，合计 6/6 校验通过。Control 以 GEMM/MoE（Head 44.7%）和 NCCL（31.9%）为主；Mixed 中 NSA/MLA 升至 29.7%，NCCL 为 23.1%，说明 Chunked Prefill 的 Sparse Attention/MLA 计算进入同一 TP16 GPU 关键路径。Prefill Trace 漏采 3 个 Rank 的 CUDA 活动，因此暂不判断固定慢 Rank，也不将 NCCL 单独定性为根因。实验结束后两节点容器和 16 张 GPU 均已清理。
+>
+> **更新（2026-08-02 00:35:00 CST）**
+>
+> 修复 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 混合 Capture 时序。两轮失败 Run 证明当前镜像的 `/start_profile` 会阻塞 API 事件循环：Profile 激活后发送的 128K 请求只能在捕获结束后进入 Scheduler，因此异步调用仍无法覆盖注入。最终实现先稳定 Decode 背景，再通过原生 `/generate` + `input_ids` 送入 128K 请求；确认首个 8K Chunk Prefill 已开始后才捕获 32 个 Mixed Step。Control Trace 单独提供纯 Decode 基线，Mixed Trace 捕获真实 Prefill+Decode Treatment。混合背景缩为单波 32 请求，Profiling 专用 watchdog 提升至 1800 秒，异常退出会自动将 manifest 标记为 `FAILED`。
+>
+> **更新（2026-08-01 19:34:34 CST）**
+>
+> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 正式时间线代码交付。正式 Control 与混合背景统一为 `1K -> 1K, C=32`；所有 `/start_profile` 请求显式记录 `start_step` 和 `num_steps`，Control 在活跃 Decode 后跳过 2 Step，混合场景先保留 2 个纯 Decode Step 再注入 `128K -> 1`，长 Prefill 从首个 Chunk 开始捕获。继续沿用已通过的双节点 PyTorch/Nsight smoke、16 Rank/6 份 Nsight 报告 fail-closed 门禁和单服务三段 Capture Range；正式结果尚未生成，因此不创建 `phase3_exp.html` 或 `phase3_code.html`。
+>
 > **更新（2026-08-01 15:24:52 CST）**
 >
 > 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模。正式 Run `dsv4pro-phase2_5-20260801-130007` 完成 Scout 5/5 与 Confirm 6/6；64K Prefill 在 C=16 已进入约 2,984 input tok/s、79.90 Gbit/s/rail 的平台，C=32/64 不再显著增长。拟合通信强度为 3.332 MB/input-token/rail，单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍，因此当前是模型计算/实现吞吐先饱和，不是 RDMA 先饱和。新增 `phase2_5_exp.html`、`phase2_5_code.html`、精简证据集和可复用的模型部署 RDMA 需求评估流程；实验结束后双节点容器与 16 张 GPU 已清理。
--- a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html
+++ b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html
@ -837,7 +837,7 @@ NVTX 只是在时间线上添加可读标记。启用 NVTX 不等于已经启动
 </p>
 <h3>7.2 捕获策略</h3>
 <ul>
-<li>只捕获预热后的 5 到 10 个 Engine Step。</li>
+<li>只捕获预热后的 10 到 32 个 Engine Step；混合场景窗口略长，用于覆盖 Prefill 注入前后。</li>
 <li>Prefill、Decode 和混合干扰分别生成报告。</li>
 <li>两台机器分别保存原始报告。</li>
 <li>优先保留所有 Rank；文件过大时至少保留代表 Rank 和跨机通信相关 Rank。</li>
@ -890,7 +890,36 @@ SGLang 进程，并绑定相同 Case ID 和时间窗口。正式执行前先验
 <code>nsys</code> 版本、子进程跟踪方式及动态 Capture 机制，再固化命令；
 不直接对整轮 Benchmark 做长时间全量捕获。
 </p>
-<h3>7.4 时间线必须回答的问题</h3>
+<h3>7.4.1 当前实现与正式入口</h3>
+<p>
+代码已集中在
+<code>experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling</code>，
+仍只有 <code>run_timeline_profiling.sh</code> 一个入口。双节点 PyTorch Profiler
+和 Nsight smoke 已通过；正式 Run 尚未执行，因此暂不生成 Phase 3 的
+<code>exp/code</code> HTML 档案。
+</p>
+<table>
+<thead><tr><th>Case</th><th>负载</th><th>动态窗口</th></tr></thead>
+<tbody>
+<tr><td>Decode Control</td><td>1K → 1K，C=32</td><td>确认 Decode 活跃后跳过 2 Step，捕获 16 Step</td></tr>
+<tr><td>Mixed Treatment</td><td>1K → 1K，C=32 背景 + 128K → 1 注入</td><td>先保留 2 个纯 Decode Step，再注入 Prefill；总计捕获 32 Step</td></tr>
+<tr><td>Long Prefill</td><td>128K → 1，C=1</td><td>从首个 Chunk 开始捕获 16 Step</td></tr>
+</tbody>
+</table>
+<p>
+三段正式范围复用一次模型加载，因此 Nsight 使用
+<code>--capture-range-end=repeat:3:defer</code>。它与官方单段示例中的
+<code>stop</code> 目的相同，但允许同一个服务依次产出三段独立报告。
+</p>
+<pre><code class="language-bash"># 只在 Head 174.1.51.5 执行
+cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
+RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
+tmux new-session -d -s dsv4pro-phase3 \
+  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
+   2&gt;&amp;1 | tee /data/hzy/${RUN_ID}.log"
+tmux attach -t dsv4pro-phase3
+</code></pre>
+<h3>7.5 时间线必须回答的问题</h3>
 <ol>
 <li>Prefill 和 Decode 各自的 Top Kernel 是什么？</li>
 <li>NCCL 在关键路径上的暴露时间是多少？</li>
@ -903,7 +932,7 @@ SGLang 进程，并绑定相同 Case ID 和时间窗口。正式执行前先验
 <li>NSA Indexer 的成本占 Sparse Attention 总成本多少？</li>
 <li>长 Prefill 到来时，Decode Kernel 为什么被延迟？</li>
 </ol>
-<h3>7.5 报告分析</h3>
+<h3>7.6 报告分析</h3>
 <pre><code class="language-bash">nsys stats &lt;REPORT&gt;.nsys-rep
 </code></pre>
 <p>重点查看：</p>
--- a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md
+++ b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md
@ -320,7 +320,7 @@ NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING

 ### 7.1 捕获策略

- 只捕获预热后的 5 到 10 个 Engine Step。
+- 只捕获预热后的 10 到 32 个 Engine Step；混合场景窗口略长，用于覆盖 Prefill 注入前后。
 - Prefill、Decode 和混合干扰分别生成报告。
 - 两台机器分别保存原始报告。
 - 优先保留所有 Rank；文件过大时至少保留代表 Rank 和跨机通信相关 Rank。
@ -379,6 +379,31 @@ curl -X POST http://127.0.0.1:30000/start_profile \

 多机 Trace 自动合并要求两台机器能访问同一个共享输出目录。没有共享目录时分别保存，再在本地汇总。

+当前实现位于：
+
+```text
+experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
+```
+
+正式场景为：
+
+- Decode Control：`1K -> 1K, C=32`，确认 Decode 活跃后跳过 2 Step，捕获 16 Step。
+- Mixed Treatment：`1K -> 1K, C=32` 背景先捕获 2 个纯 Decode Step，再注入 `128K -> 1`，总计捕获 32 Step。
+- Long Prefill：`128K -> 1, C=1`，从首个 Chunk 开始捕获 16 Step。
+
+双节点 PyTorch/Nsight smoke 已通过。正式 Run 尚未执行，因此暂不生成 Phase 3 的 `exp/code` HTML。正式命令只在 Head 执行：
+
+```bash
+cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
+RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
+tmux new-session -d -s dsv4pro-phase3 \
+  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
+   2>&1 | tee /data/hzy/${RUN_ID}.log"
+tmux attach -t dsv4pro-phase3
+```
+
+三段范围复用一次模型加载，因此 Nsight 使用 `--capture-range-end=repeat:3:defer`，而不是单段示例中的 `stop`。
+
 ### 7.4 时间线必须回答的问题

 1. Prefill 和 Decode 各自的 Top Kernel 是什么？
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/README.md
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/README.md
@ -3,29 +3,17 @@
 Phase 3 captures short distributed timelines for the completed TP16 baseline.
 It does not rerun Phase 2 hardware sampling or communication microbenchmarks.

-## One entry point
+Formal run `dsv4pro-phase3-20260802-003213` completed at commit `82b7d91`:
+three ranges produced three Head and three Worker `.nsys-rep` files, and all
+six reports passed validation and `nsys stats` export. Raw reports remain under
+the run directory on Head. Profiler-side throughput is diagnostic only.
+
+## Formal entry point

 Run only on Head `174.1.51.5`:

 ```bash
 cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
-
-bash run_timeline_profiling.sh audit
-
-DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun-$(date +%Y%m%d-%H%M%S) \
-  bash run_timeline_profiling.sh all
-
-RUN_ID=dsv4pro-phase3-torch-smoke-$(date +%Y%m%d-%H%M%S) \
-  bash run_timeline_profiling.sh torch-smoke
-
-RUN_ID=dsv4pro-phase3-nsys-smoke-$(date +%Y%m%d-%H%M%S) \
-  bash run_timeline_profiling.sh nsys-smoke
-```
-
-After both smoke tests pass, run the three formal capture ranges in one model
-service lifetime:
-
-```bash
 RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
 tmux new-session -d -s dsv4pro-phase3 \
  "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
@ -34,16 +22,31 @@ tmux new-session -d -s dsv4pro-phase3 \
 tmux attach -t dsv4pro-phase3
 ```

+PyTorch Profiler and Nsight smoke tests have already passed on this image and
+topology. To inspect the exact commands without starting containers or sending
+requests, run `DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun bash
+run_timeline_profiling.sh all` first.
+
 The script starts and stops both nodes, triggers `/start_profile`, retrieves
 Worker artifacts, exports basic `nsys stats`, and cleans containers and Worker
 staging paths. No `source`, Conda activation, or command on Worker is required.

 ## Formal ranges

-1. `decode_control`: `1K -> 256`, `C=32`.
-2. `mixed_decode_with_128k_prefill`: active `1K -> 512`, `C=32` decode plus one `128K -> 1` injection.
+1. `decode_control`: `1K -> 1K`, `C=32`; trigger after Decode becomes active,
+   skip 2 engine steps, then capture 16 steps.
+2. `mixed_decode_with_128k_prefill`: active `1K -> 1K`, `C=32` decode;
+   wait for stable Decode activity, submit one native `input_ids` based
+   `128K -> 1` request, and start `/start_profile` only after the first 8K
+   chunk enters Prefill. This image blocks other API requests while
+   `/start_profile` is active, so Control provides the pure Decode baseline and
+   the 32-step mixed range captures the actual Prefill-plus-Decode treatment.
 3. `long_prefill`: isolated `128K -> 1`, `C=1`.

+The service remains under one Nsight process for all three ranges, so it uses
+`--capture-range-end=repeat:3:defer` rather than the single-range `stop` form.
+Every `/start_profile` request records both `start_step` and `num_steps`.
+
 The primary run keeps CUDA Graph enabled. A later layer-wise NVTX run with
 disabled graphs is allowed only if these production-faithful traces cannot map
 the dominant kernels to model components.
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/config.env
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/config.env
@ -17,25 +17,33 @@ WORKER_STAGE_ROOT="${WORKER_STAGE_ROOT:-/data/hzy/dsv4pro_phase3_worker}"
 NSYS_TRACE="${NSYS_TRACE:-cuda,nvtx,nccl,osrt}"
 NSYS_CUDA_GRAPH_TRACE="${NSYS_CUDA_GRAPH_TRACE:-node}"
 NSYS_EXPORT_TIMEOUT_S="${NSYS_EXPORT_TIMEOUT_S:-300}"
+PROFILE_WATCHDOG_TIMEOUT_S="${PROFILE_WATCHDOG_TIMEOUT_S:-1800}"
 NSYS_CONTROL_STEPS="${NSYS_CONTROL_STEPS:-16}"
 NSYS_MIXED_STEPS="${NSYS_MIXED_STEPS:-32}"
 NSYS_PREFILL_STEPS="${NSYS_PREFILL_STEPS:-16}"
 NSYS_SMOKE_STEPS="${NSYS_SMOKE_STEPS:-5}"
 TORCH_SMOKE_STEPS="${TORCH_SMOKE_STEPS:-3}"
+TORCH_SMOKE_START_STEP="${TORCH_SMOKE_START_STEP:-1}"
+NSYS_SMOKE_START_STEP="${NSYS_SMOKE_START_STEP:-1}"
+NSYS_CONTROL_START_STEP="${NSYS_CONTROL_START_STEP:-2}"
+NSYS_MIXED_START_STEP="${NSYS_MIXED_START_STEP:-0}"
+NSYS_PREFILL_START_STEP="${NSYS_PREFILL_START_STEP:-0}"
+MIXED_PRE_INJECTION_DECODE_STEPS="${MIXED_PRE_INJECTION_DECODE_STEPS:-2}"

 # Short profiling workloads. They reproduce the Phase 2 scheduler shapes but
 # avoid its full result matrix and long hardware sampling.
 CONTROL_ISL="${CONTROL_ISL:-1024}"
-CONTROL_OSL="${CONTROL_OSL:-256}"
+CONTROL_OSL="${CONTROL_OSL:-1024}"
 CONTROL_CONCURRENCY="${CONTROL_CONCURRENCY:-32}"
 CONTROL_PROMPTS="${CONTROL_PROMPTS:-32}"

 MIXED_ISL="${MIXED_ISL:-1024}"
-MIXED_OSL="${MIXED_OSL:-512}"
+MIXED_OSL="${MIXED_OSL:-1024}"
 MIXED_CONCURRENCY="${MIXED_CONCURRENCY:-32}"
-MIXED_PROMPTS="${MIXED_PROMPTS:-64}"
+MIXED_PROMPTS="${MIXED_PROMPTS:-32}"
 MIXED_INJECTION_ISL="${MIXED_INJECTION_ISL:-131072}"
 MIXED_INJECTION_OSL="${MIXED_INJECTION_OSL:-1}"
+MIXED_INJECTION_TOKEN_ID="${MIXED_INJECTION_TOKEN_ID:-100}"

 PREFILL_ISL="${PREFILL_ISL:-131072}"
 PREFILL_OSL="${PREFILL_OSL:-1}"
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh
@ -209,6 +209,7 @@ build_server_command() {
    --mem-fraction-static "${MEM_FRACTION_STATIC}"
    --cuda-graph-max-bs-decode "${CUDA_GRAPH_MAX_BS_DECODE}"
    --max-running-requests "${MAX_RUNNING_REQUESTS}"
+    --watchdog-timeout "${PROFILE_WATCHDOG_TIMEOUT_S}"
  )
  if [[ -n "${SGLANG_EXTRA_ARGS}" ]]; then
    local -a extra_args=()
@ -367,15 +368,16 @@ profile_done_count() {
 start_profile() {
  local case_id="$1"
  local activities_json="$2"
-  local steps="$3"
+  local start_step="$3"
+  local steps="$4"
  local output_dir="${PROFILE_ROOT_IN_CONTAINER}/torch/${case_id}"
  local response_file="${CASE_DIR}/${case_id}/start_profile_response.json"
  local payload
-  payload="{\"output_dir\":\"${output_dir}\",\"num_steps\":${steps},\"activities\":${activities_json},\"profile_id\":\"${RUN_ID}-${case_id}\",\"profile_prefix\":\"${case_id}\",\"merge_profiles\":false}"
+  payload="{\"output_dir\":\"${output_dir}\",\"start_step\":${start_step},\"num_steps\":${steps},\"activities\":${activities_json},\"profile_id\":\"${RUN_ID}-${case_id}\",\"profile_prefix\":\"${case_id}\",\"merge_profiles\":false}"
  mkdir -p "${CASE_DIR}/${case_id}"
  printf '%s\n' "${payload}" > "${CASE_DIR}/${case_id}/start_profile_request.json"
  if [[ "${DRY_RUN}" == "1" ]]; then
-    log "[DRY] POST /start_profile case=${case_id} steps=${steps} activities=${activities_json}"
+    log "[DRY] POST /start_profile case=${case_id} start_step=${start_step} steps=${steps} activities=${activities_json}"
    return 0
  fi
  curl --fail --silent --show-error -X POST \
@ -407,9 +409,16 @@ server_decode_count() {
 wait_for_decode_activity() {
  local before="$1"
  local bench_pid="$2"
+  wait_for_decode_steps "${before}" 1 "${bench_pid}"
+}
+
+wait_for_decode_steps() {
+  local before="$1"
+  local required_steps="$2"
+  local bench_pid="$3"
  local waited=0
  while (( waited < 180 )); do
-    if (( $(server_decode_count) > before )); then
+    if (( $(server_decode_count) >= before + required_steps )); then
      return 0
    fi
    kill -0 "${bench_pid}" 2>/dev/null || return 1
@ -419,6 +428,26 @@ wait_for_decode_activity() {
  return 1
 }

+server_long_prefill_count() {
+  run_on_node "${HEAD_NODE}" "docker logs '${HEAD_CONTAINER}' 2>&1" \
+    | grep -c 'Prefill batch, #new-seq: 1, #new-token: 8192' || true
+}
+
+wait_for_long_prefill_activity() {
+  local before="$1"
+  local injection_pid="$2"
+  local waited=0
+  while (( waited < 180 )); do
+    if (( $(server_long_prefill_count) > before )); then
+      return 0
+    fi
+    kill -0 "${injection_pid}" 2>/dev/null || return 1
+    sleep 1
+    ((waited+=1))
+  done
+  return 1
+}
+
 prepare_bench_command() {
  local output_file="$1"
  local isl="$2"
@ -479,31 +508,78 @@ run_bench() {
    "${BENCH_CMD[@]}" > "${case_path}/bench.log" 2>&1
 }

+prepare_native_injection_payload() {
+  local case_path="$1"
+  local payload_file="${case_path}/request.json"
+  mkdir -p "${case_path}"
+  python3 -c 'import json, sys
+path, length, token_id, output_len = sys.argv[1:]
+payload = {
+    "input_ids": [int(token_id)] * int(length),
+    "sampling_params": {
+        "max_new_tokens": int(output_len),
+        "temperature": 0,
+        "ignore_eos": True,
+    },
+    "stream": False,
+}
+with open(path, "w", encoding="utf-8") as f:
+    json.dump(payload, f, separators=(",", ":"))
+' "${payload_file}" "${MIXED_INJECTION_ISL}" \
+    "${MIXED_INJECTION_TOKEN_ID}" "${MIXED_INJECTION_OSL}"
+}
+
+run_native_injection() {
+  local case_id="$1"
+  local case_path="${CASE_DIR}/${case_id}"
+  local payload_file="${case_path}/request.json"
+  local response_file="${case_path}/response.json"
+  prepare_native_injection_payload "${case_path}"
+  print_command timeout --signal=TERM --kill-after=30s \
+    "${PROFILE_TIMEOUT_S}s" curl --fail --silent --show-error \
+    -X POST "http://${HEAD_IP}:${SGLANG_PORT}/generate" \
+    -H 'Content-Type: application/json' --data-binary "@${payload_file}" \
+    -o "${response_file}" > "${case_path}/curl_cmd.txt"
+  if [[ "${DRY_RUN}" == "1" ]]; then
+    printf '[DRY] '
+    cat "${case_path}/curl_cmd.txt"
+    return 0
+  fi
+  timeout --signal=TERM --kill-after=30s "${PROFILE_TIMEOUT_S}s" \
+    curl --fail --silent --show-error \
+    -X POST "http://${HEAD_IP}:${SGLANG_PORT}/generate" \
+    -H 'Content-Type: application/json' --data-binary "@${payload_file}" \
+    -o "${response_file}"
+  python3 -m json.tool "${response_file}" >/dev/null
+}
+
 run_profiled_case() {
  local case_id="$1"
  local activities_json="$2"
-  local steps="$3"
-  local isl="$4"
-  local osl="$5"
-  local concurrency="$6"
-  local prompts="$7"
-  local seed="$8"
+  local start_step="$3"
+  local steps="$4"
+  local isl="$5"
+  local osl="$6"
+  local concurrency="$7"
+  local prompts="$8"
+  local seed="$9"
  local before=0
  flush_cache
  [[ "${DRY_RUN}" == "1" ]] || before="$(profile_done_count)"
-  start_profile "${case_id}" "${activities_json}" "${steps}"
+  start_profile "${case_id}" "${activities_json}" "${start_step}" "${steps}"
  run_bench "${case_id}" "${isl}" "${osl}" "${concurrency}" "${prompts}" "${seed}"
  [[ "${DRY_RUN}" == "1" ]] || wait_for_profile_stop "${before}"
 }

 run_profiled_decode_case() {
  local case_id="$1"
-  local steps="$2"
-  local isl="$3"
-  local osl="$4"
-  local concurrency="$5"
-  local prompts="$6"
-  local seed="$7"
+  local start_step="$2"
+  local steps="$3"
+  local isl="$4"
+  local osl="$5"
+  local concurrency="$6"
+  local prompts="$7"
+  local seed="$8"
  local case_path="${CASE_DIR}/${case_id}"
  local decode_before=0
  local profile_before=0
@ -517,7 +593,7 @@ run_profiled_decode_case() {
    log "[DRY] start decode workload, wait for Decode batch, then start Nsight range"
    printf '[DRY] '
    cat "${case_path}/bench_cmd.txt"
-    start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}"
+    start_profile "${case_id}" '["CUDA_PROFILER"]' "${start_step}" "${steps}"
    return 0
  fi
  decode_before="$(server_decode_count)"
@ -532,7 +608,7 @@ run_profiled_decode_case() {
      return 1
    }
  profile_before="$(profile_done_count)"
-  start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}"
+  start_profile "${case_id}" '["CUDA_PROFILER"]' "${start_step}" "${steps}"
  wait_for_profile_stop "${profile_before}"
  wait "${bench_pid}"
 }
@ -557,6 +633,9 @@ run_mixed_profile() {
  local background_path="${CASE_DIR}/${background_id}"
  local before=0
  local decode_before=0
+  local mixed_baseline_before=0
+  local prefill_before=0
+  local injection_pid=0
  flush_cache
  mkdir -p "${background_path}"
  prepare_bench_command "${background_path}/bench.jsonl" \
@ -564,10 +643,11 @@ run_mixed_profile() {
  print_command timeout --signal=TERM --kill-after=30s \
    "${PROFILE_TIMEOUT_S}s" "${BENCH_CMD[@]}" > "${background_path}/bench_cmd.txt"
  if [[ "${DRY_RUN}" == "1" ]]; then
-    log "[DRY] start decode background, wait for Decode batch, start profile, inject 128K prefill"
-    start_profile "${case_id}" '["CUDA_PROFILER"]' "${NSYS_MIXED_STEPS}"
-    run_bench "${injection_id}" "${MIXED_INJECTION_ISL}" \
-      "${MIXED_INJECTION_OSL}" 1 1 5202
+    log "[DRY] start decode background and wait for stable Decode activity"
+    log "[DRY] inject native 128K prefill, wait for its first 8K chunk, then start mixed profile"
+    run_native_injection "${injection_id}"
+    start_profile "${case_id}" '["CUDA_PROFILER"]' \
+      "${NSYS_MIXED_START_STEP}" "${NSYS_MIXED_STEPS}"
    return 0
  fi
  decode_before="$(server_decode_count)"
@ -581,11 +661,28 @@ run_mixed_profile() {
    log "ERROR: mixed decode background did not enter main benchmark"
    return 1
  }
+  mixed_baseline_before="$(server_decode_count)"
+  wait_for_decode_steps "${mixed_baseline_before}" \
+    "${MIXED_PRE_INJECTION_DECODE_STEPS}" "${background_pid}" || {
+    kill "${background_pid}" 2>/dev/null || true
+    wait "${background_pid}" 2>/dev/null || true
+    log "ERROR: mixed decode background did not remain stable before injection"
+    return 1
+  }
+  prefill_before="$(server_long_prefill_count)"
+  run_native_injection "${injection_id}" &
+  injection_pid=$!
+  wait_for_long_prefill_activity "${prefill_before}" "${injection_pid}" || {
+    kill "${injection_pid}" "${background_pid}" 2>/dev/null || true
+    wait "${injection_pid}" "${background_pid}" 2>/dev/null || true
+    log "ERROR: native 128K injection did not enter chunked Prefill"
+    return 1
+  }
  before="$(profile_done_count)"
-  start_profile "${case_id}" '["CUDA_PROFILER"]' "${NSYS_MIXED_STEPS}"
-  run_bench "${injection_id}" "${MIXED_INJECTION_ISL}" \
-    "${MIXED_INJECTION_OSL}" 1 1 5202
+  start_profile "${case_id}" '["CUDA_PROFILER"]' \
+    "${NSYS_MIXED_START_STEP}" "${NSYS_MIXED_STEPS}"
  wait_for_profile_stop "${before}"
+  wait "${injection_pid}"
  wait "${background_pid}"
 }

@ -610,7 +707,12 @@ write_manifest() {
  "docker_image": "${DOCKER_IMAGE}",
  "cuda_graph_enabled": true,
  "tp_size": ${TP_SIZE},
-  "ep_size": ${EP_SIZE}
+  "ep_size": ${EP_SIZE},
+  "profile_windows": {
+    "decode_control": {"start_step": ${NSYS_CONTROL_START_STEP}, "num_steps": ${NSYS_CONTROL_STEPS}},
+    "mixed": {"start_step": ${NSYS_MIXED_START_STEP}, "num_steps": ${NSYS_MIXED_STEPS}, "pre_injection_decode_steps": ${MIXED_PRE_INJECTION_DECODE_STEPS}},
+    "long_prefill": {"start_step": ${NSYS_PREFILL_START_STEP}, "num_steps": ${NSYS_PREFILL_STEPS}}
+  }
 }
 EOF
 }
@ -641,6 +743,16 @@ cleanup() {
  fi
 }

+on_exit() {
+  local exit_code=$?
+  trap - EXIT
+  if (( exit_code != 0 )) && [[ -d "${RESULT_DIR}" ]]; then
+    write_manifest FAILED || true
+  fi
+  cleanup
+  exit "${exit_code}"
+}
+
 run_torch_smoke() {
  PROFILE_MODE="torch"
  CAPTURE_RANGES=0
@ -648,7 +760,8 @@ run_torch_smoke() {
  start_service
  write_manifest RUNNING
  run_profiled_case torch_decode_control '["CPU","GPU"]' \
-    "${TORCH_SMOKE_STEPS}" "${CONTROL_ISL}" 128 8 8 5101
+    "${TORCH_SMOKE_START_STEP}" "${TORCH_SMOKE_STEPS}" \
+    "${CONTROL_ISL}" 128 8 8 5101
  stop_service
  CLEANUP_ON_EXIT=0
  validate_torch_traces
@ -662,7 +775,8 @@ run_nsys_smoke() {
  start_service
  write_manifest RUNNING
  run_profiled_case nsys_decode_control '["CUDA_PROFILER"]' \
-    "${NSYS_SMOKE_STEPS}" "${CONTROL_ISL}" "${CONTROL_OSL}" \
+    "${NSYS_SMOKE_START_STEP}" "${NSYS_SMOKE_STEPS}" \
+    "${CONTROL_ISL}" "${CONTROL_OSL}" \
    "${CONTROL_CONCURRENCY}" "${CONTROL_PROMPTS}" 5102
  stop_service
  CLEANUP_ON_EXIT=0
@ -678,11 +792,13 @@ run_all() {
  start_service
  write_manifest RUNNING
  run_profiled_decode_case decode_control \
-    "${NSYS_CONTROL_STEPS}" "${CONTROL_ISL}" "${CONTROL_OSL}" \
+    "${NSYS_CONTROL_START_STEP}" "${NSYS_CONTROL_STEPS}" \
+    "${CONTROL_ISL}" "${CONTROL_OSL}" \
    "${CONTROL_CONCURRENCY}" "${CONTROL_PROMPTS}" 5301
  run_mixed_profile
  run_profiled_case long_prefill '["CUDA_PROFILER"]' \
-    "${NSYS_PREFILL_STEPS}" "${PREFILL_ISL}" "${PREFILL_OSL}" 1 1 5303
+    "${NSYS_PREFILL_START_STEP}" "${NSYS_PREFILL_STEPS}" \
+    "${PREFILL_ISL}" "${PREFILL_OSL}" 1 1 5303
  stop_service
  CLEANUP_ON_EXIT=0
  validate_nsys_reports
@ -690,7 +806,8 @@ run_all() {
  write_manifest COMPLETED
 }

-trap cleanup EXIT INT TERM
+trap on_exit EXIT
+trap 'exit 130' INT TERM

 case "${ACTION}" in
  audit)
Author	SHA1	Message	Date
Zhiyi Hong	41e2b000a4	[Docs] finalize Phase 3 timeline profiling results	2026-08-02 01:16:49 +08:00
Zhiyi Hong	82b7d91ac0	[BugFix] capture mixed trace after prefill admission	2026-08-02 00:30:46 +08:00
Zhiyi Hong	bc491eeeed	[BugFix] align mixed Phase 3 capture with prefill injection	2026-08-02 00:05:54 +08:00
Zhiyi Hong	4628d49755	[Feat] finalize Phase 3 SGLang timeline capture	2026-08-01 19:38:36 +08:00