[Docs] finalize Phase 2.5 RDMA demand model

[Feat] add Phase 2.5 RDMA demand modeling
[BugFix] align Phase 3 captures with active decode
2026-08-01 15:29:26 +08:00 · 2026-08-01 02:43:06 +08:00 · 2026-08-01 00:50:45 +08:00
39 changed files with 5788 additions and 16 deletions
--- a/README.md
+++ b/README.md
@ -1,5 +1,17 @@
 # sskj — 多平台大模型推理性能基准测试项目
 > **更新（2026-08-01 15:24:52 CST）**
 >
 > 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模。正式 Run `dsv4pro-phase2_5-20260801-130007` 完成 Scout 5/5 与 Confirm 6/6；64K Prefill 在 C=16 已进入约 2,984 input tok/s、79.90 Gbit/s/rail 的平台，C=32/64 不再显著增长。拟合通信强度为 3.332 MB/input-token/rail，单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍，因此当前是模型计算/实现吞吐先饱和，不是 RDMA 先饱和。新增 `phase2_5_exp.html`、`phase2_5_code.html`、精简证据集和可复用的模型部署 RDMA 需求评估流程；实验结束后双节点容器与 16 张 GPU 已清理。
 >
 > **更新（2026-08-01 02:40:00 CST）**
 >
 > 新增 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模唯一入口。实验保持现有 TP16/EP2 服务参数不变，先以 `64K -> 1` 的 C=1/4/16/32/64 建立 Input TPS 与每 Rail HCA 带宽关系，再自动选择平台前、拐点和最大稳定并发，对 `64K -> 1K` 重复确认。结果将给出每 Token 跨机字节数、400G 所需 Token TPS、并发饱和曲线和“模型计算先饱和还是 RDMA 先饱和”的机器可读结论；正式结果尚未生成，因此暂不创建 Phase 2.5 HTML 档案。
 >
 > **更新（2026-07-31 19:02:00 CST）**
 >
 > Phase 3 Nsight smoke 已在 Head/Worker 各生成一份可解析报告；首份时间线确认 NCCL AllReduce、Sparse MLA、CUTLASS GEMM 与 MoE kernel 均可见。根据 smoke 修正正式 capture：Decode 对照和混合负载必须等服务实际进入 `Decode batch` 后才触发；每段等待本节点全部 8 个 rank 完成，而非任一 rank；双节点近同步停止以减少 Gloo 断链噪声。`nsys stats` 改为可写临时 SQLite、保留 Kernel/API/NVTX 高价值汇总并删除临时库，结果与报告数量均 fail-closed。
 >
 > **更新（2026-07-31 18:53:00 CST）**
 >
 > Phase 3 PyTorch Profiler 首轮 smoke 已证明双节点 16 个 rank 均可完成 trace 写盘，Head/Worker 分别生成 8 份、约 1.5/1.4 GB 压缩 trace。修复当前 OpenSSH 不接受 `scp remote:/path/.` 导致 Worker 结果未回收的问题，改为 SSH tar 流式传输；正式入口不再忽略回收失败，并新增至少 16 份 PyTorch rank trace、至少 2 份 Nsight 节点报告的结果门禁。
--- a/docs/dsv4pro_pro6000d_2node_sglang/6000D双机通信与NCCL术语入门.html
+++ b/docs/dsv4pro_pro6000d_2node_sglang/6000D双机通信与NCCL术语入门.html
@ -547,6 +547,26 @@ mlx5_3 port 1 ==> eth3 (Up)</code></pre>
            <td>查看系统或进程在各 NUMA 节点上的内存分布。</td>
            <td>Phase 2 每 5 秒保存结构化 Node0/Node1 MiB，寻找跨 NUMA 内存放置。</td>
          </tr>
          <tr>
            <td><strong>HCA Counter</strong></td>
            <td>网卡硬件维护的发送、接收、等待、丢弃和错误累计计数器。</td>
            <td>Phase 2.5 用 <code>mlx5_0/mlx5_3</code> 的 counter 差值计算正式 benchmark 窗口内的 RDMA Gbit/s。</td>
          </tr>
          <tr>
            <td><strong>bytes/input-token/rail</strong></td>
            <td>模型每处理一个输入 token，平均要在一条 Rail 上发送的字节数。</td>
            <td>当前 DSV4-Pro TP16/EP2 Scout 拟合为约 3.332 MB/token/rail；换模型或并行策略必须重新标定。</td>
          </tr>
          <tr>
            <td><strong>带宽平台 / 拐点</strong></td>
            <td>继续增加并发后，吞吐与网络带宽都几乎不再增长的位置。</td>
            <td>Phase 2.5 以相邻点的 Input TPS 和 Rail Mean 增益同时低于 5% 判断，当前拐点为 C=16。</td>
          </tr>
          <tr>
            <td><strong>渐近线 / 饱和上限</strong></td>
            <td>饱和曲线在并发继续增大时逼近、但不会明显超过的预测上限。</td>
            <td>当前 64K Prefill 的拟合上限约 80.32 Gbit/s/rail，表示模型产流量上限，不表示网卡硬件只能跑 80G。</td>
          </tr>
        </tbody>
      </table>
--- a/docs/dsv4pro_pro6000d_2node_sglang/phase2_5_code.html
+++ b/docs/dsv4pro_pro6000d_2node_sglang/phase2_5_code.html
@ -0,0 +1,167 @@
 <!doctype html>
 <html lang="zh-CN">
 <head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Phase 2.5 Code：DSV4-Pro 双机 SGLang RDMA 需求建模</title>
  <style>
    :root { --canvas:#eef3f4; --paper:#fff; --ink:#182126; --muted:#5a6970; --line:#d4dee1; --navy:#17363d; --teal:#087c72; --teal-soft:#e8f5f3; --amber:#a64c14; --amber-soft:#fff1e7; --code-bg:#17252b; --code-ink:#eaf2f3; }
    * { box-sizing:border-box; letter-spacing:0; }
    body { margin:0; color:var(--ink); background:var(--canvas); font-family:"PingFang SC","Microsoft YaHei",Arial,sans-serif; font-size:16px; line-height:1.72; }
    header { color:#f6fbfb; background:var(--navy); border-bottom:5px solid #d2692b; }
    .header-inner,main { width:min(100% - 36px,1120px); margin:0 auto; }
    .header-inner { padding:34px 0 30px; }
    h1 { margin:0; font-size:clamp(28px,4vw,42px); line-height:1.25; }
    h2 { margin:42px 0 15px; padding-bottom:8px; font-size:25px; border-bottom:2px solid #adbbc0; }
    h3 { margin:28px 0 10px; color:#21454d; font-size:19px; }
    .eyebrow { margin:0 0 6px; color:#9edbd5; font-size:13px; font-weight:700; }
    .meta { margin-top:15px; color:#d6e5e7; font-size:14px; }
    main { margin-top:30px; margin-bottom:70px; padding:38px 48px 58px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
    a { color:var(--teal); }
    code { padding:2px 5px; color:#85380d; background:var(--amber-soft); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; overflow-wrap:anywhere; }
    pre { margin:14px 0 22px; padding:16px 18px; overflow:auto; color:var(--code-ink); background:var(--code-bg); border-radius:5px; font:13px/1.62 "SFMono-Regular",Consolas,monospace; }
    pre code { padding:0; color:inherit; background:transparent; }
    table { width:100%; margin:16px 0 26px; border-collapse:collapse; font-size:14px; }
    th,td { padding:9px 11px; vertical-align:top; text-align:left; border:1px solid var(--line); overflow-wrap:anywhere; }
    th { color:#153b41; background:#eaf2f2; }
    .callout { margin:18px 0 26px; padding:14px 18px; background:var(--teal-soft); border-left:4px solid var(--teal); }
    .warning { margin:18px 0 26px; padding:14px 18px; background:var(--amber-soft); border-left:4px solid var(--amber); }
    li+li { margin-top:5px; }
    @media(max-width:760px){main{padding:28px 20px 42px}table{display:block;overflow-x:auto}}
  </style>
 </head>
 <body>
  <header>
    <div class="header-inner">
      <p class="eyebrow">STANDALONE CODE WALKTHROUGH / PHASE 2.5</p>
      <h1>DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模：代码详解</h1>
      <div class="meta">实现提交：<code>c5fa700c50c0</code>　正式 Run：<code>dsv4pro-phase2_5-20260801-130007</code>　唯一入口：<code>run_rdma_demand_modeling.sh all</code></div>
    </div>
  </header>
  <main>
    <p><a href="./推理优化计划.html">返回推理优化主计划</a> · <a href="./phase2_5_exp.html">打开 Phase 2.5 实验档案</a></p>
    <div class="callout"><strong>边界：</strong>Phase 2.5 不复制模型服务和采集器。它复用 Phase 1 的双机 TP16 服务/benchmark 与 Phase 2 的精确窗口、GPU/RDMA 采集能力，只新增“并发 Scout → 自动选点 → 业务 OSL Confirm → 需求拟合”这一层编排和分析。</div>
    <h2>1. 文件职责与调用关系</h2>
    <table>
      <thead><tr><th>文件</th><th>职责</th><th>调用关系</th></tr></thead>
      <tbody>
        <tr><td><code>config.env</code></td><td>定义 ISL/OSL、Scout 并发、重复次数、平台阈值和 400G 目标</td><td>被唯一入口 source</td></tr>
        <tr><td><code>run_rdma_demand_modeling.sh</code></td><td>生成场景、调用 Phase 2、串行运行 Scout/Confirm、清理服务</td><td>唯一人工入口</td></tr>
        <tr><td><code>rdma_demand_model.py</code></td><td>对齐 benchmark/HCA 窗口，计算 bytes/token，拟合平台并生成报告</td><td>Scout 后选点；Confirm 后最终汇总</td></tr>
        <tr><td><code>test_rdma_demand_model.py</code></td><td>覆盖 HCA counter 单位、平台选择、线性换算和拟合输出</td><td>本地/CI 回归测试</td></tr>
        <tr><td>Phase 2 <code>run_hardware_contention_attribution.sh</code></td><td>启动服务、采集 Head/Worker、切正式测量窗</td><td>由 Phase 2.5 以环境变量调用</td></tr>
        <tr><td>Phase 1 <code>run_quick_map.sh</code></td><td>双机服务启停与 SGLang benchmark</td><td>由 Phase 2 内部复用</td></tr>
      </tbody>
    </table>
    <pre><code>run_rdma_demand_modeling.sh all
  ├─ validate_config + write_manifest
  ├─ run_scout
  │    ├─ write_scenario_file(64K→1, C=1/4/16/32/64)
  │    ├─ Phase 2 all（服务 + 18 个采集器 + 5 Case）
  │    └─ rdma_demand_model.py scout → recommendation.env
  ├─ run_confirm
  │    ├─ 读取自动选择的 C=4/16/64
  │    ├─ Phase 2 all（重启服务 + 18 个采集器 + 每点 2 次）
  │    └─ rdma_demand_model.py final
  └─ Phase 2 stop → 双节点清理</code></pre>
    <h2>2. 配置层</h2>
    <p><code>config.env:5-7</code> 通过相对路径找到 Phase 1/2，不依赖执行命令所在目录。<code>config.env:10-16</code> 定义 64K Scout 与 1K Confirm；<code>18-21</code> 定义 5% 平台阈值、400G 物理目标和 360G 实用目标。</p>
    <pre><code>ISL=65536
 SCOUT_OSL=1
 CONFIRM_OSL=1024
 SCOUT_CONCURRENCIES="1 4 16 32 64"
 SCOUT_REPETITIONS=1
 CONFIRM_REPETITIONS=2
 PLATEAU_GAIN_PCT=5
 TARGET_RAIL_GBPS=400
 PRACTICAL_RAIL_GBPS=360</code></pre>
    <p><code>SAMPLE_INTERVAL_S=1</code> 只决定 HCA/GPU 时间序列分辨率；<code>SCENARIO_TIMEOUT_S=7200</code> 是单个 benchmark 的保护上限，不是期望耗时。</p>
    <h2>3. Shell 唯一入口</h2>
    <h3>3.1 参数检查与场景生成</h3>
    <p><code>run_rdma_demand_modeling.sh:42-71</code> fail-fast 检查依赖脚本、整数参数和并发列表。<code>73-100</code> 生成 Phase 2 能读取的 TSV，并为每个形状生成稳定的 case id。</p>
    <h3>3.2 复用 Phase 2，而不是复制采集代码</h3>
    <p><code>103-134</code> 构造一个数组命令，把场景、Case、重复次数和采样周期作为环境变量传给 Phase 2。它显式关闭 mixed case 与通信 microbenchmark，因为 Phase 2.5 只测模型 RDMA 需求，不重复已完成的硬件基线。</p>
    <pre><code>RUN_MIXED_CASE=0
 RUN_COMMUNICATION_BASELINE=0
 SCENARIO_FILE=.../scout.tsv
 FIXED_CASE_IDS=rdma_scout_...
 bash run_hardware_contention_attribution.sh all</code></pre>
    <h3>3.3 两阶段控制流</h3>
    <p><code>136-152</code> 跑完 Scout 后立即调用 Python，并写出 <code>recommendation.env</code>；<code>154-180</code> 读取推荐并发，生成 64K→1K Confirm。<code>217-225</code> 的 <code>run_all</code> 严格串行执行，异常信号触发 stop 清理。</p>
    <div class="warning"><strong>为什么服务会启动两次：</strong>Scout 结束后 Phase 2 会清理服务；Confirm 使用全新的 Prefix Cache、采集器和服务生命周期，避免 Scout 状态污染确认结果。</div>
    <h2>4. Python 如何从计数器变成需求模型</h2>
    <h3>4.1 精确时间窗与 HCA 单位</h3>
    <p><code>rdma_demand_model.py:87-115</code> 以 <code>(case_id,repetition)</code> 对齐 benchmark、窗口和 RDMA 汇总。<code>117-147</code> 在正式窗口内计算相邻 HCA counter 的速率；IB <code>port_*_data</code> 单位是 4-octet，因此必须乘 4，再乘 8 转为 bit/s。</p>
    <pre><code>gbps = (counter_delta × 4 bytes × 8 bits) / duration_s / 1e9</code></pre>
    <h3>4.2 单 Case 指标</h3>
    <p><code>171-281</code> 汇总四条观测边（Head/Worker × 两个 HCA）的 Rail Mean/P95/Max、双 Rail 单向合计、Rail 不均衡、错误计数和 GPU 利用率。通信强度按每条 Rail 平均发送字节计算：</p>
    <pre><code>bytes_per_input_token_per_rail
 = mean(head/worker × mlx5_0/mlx5_3 xmit_bytes)
  / total_input_tokens</code></pre>
    <p>这里不把 TX+RX 相加，因为那会把同一份跨机数据重复计数。</p>
    <h3>4.3 平台、拐点与自动选点</h3>
    <p><code>362-392</code> 比较相邻并发点。只有 Rail Mean 与 Input TPS 增益同时低于 5%，当前点才是平台候选。随后选择平台前一点、平台点和最高稳定点；本 Run 得到 <code>4 16 64</code>。</p>
    <pre><code>if bandwidth_gain &lt; 5% and input_tps_gain &lt; 5%:
    plateau_c = current_concurrency</code></pre>
    <h3>4.4 线性通信强度与饱和曲线</h3>
    <p><code>352-360</code> 用过原点线性斜率拟合 <code>rail_gbps/input_tps</code>，再还原为 bytes/token。<code>318-350</code> 用双曲线 <code>B(C)=B∞×C/(K+C)</code> 拟合并发饱和曲线；<code>394-447</code> 组合两者，判断模型计算或网络谁先到平台。</p>
    <pre><code>required_input_tps
 = target_rail_gbps / linear_gbps_per_input_tps
 if fitted_bandwidth_asymptote &lt; 360:
    verdict = COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION</code></pre>
    <p>本 Run 的 RMSE 为 0.277 Gbit/s，五个 Scout 点与饱和曲线贴合良好；拟合上限 80.32 Gbit/s，与 C=16/32/64 的 79.90/79.93/79.97 一致。</p>
    <h2>5. 输出文件如何阅读</h2>
    <table>
      <thead><tr><th>输出</th><th>用途</th></tr></thead>
      <tbody>
        <tr><td><code>rdma_case_metrics.csv</code></td><td>每次重复的 benchmark + GPU + Rail 对齐数据，是审计主表</td></tr>
        <tr><td><code>rdma_demand_model.json</code></td><td>完整拟合参数、平台点、目标 TPS 和最终 verdict</td></tr>
        <tr><td><code>rdma_demand_report.md</code></td><td>面向人的 Scout/Confirm 摘要</td></tr>
        <tr><td><code>recommendation.env</code></td><td>Shell 可直接 source 的 Confirm 并发列表</td></tr>
        <tr><td><code>{scout,confirm}/case_windows.csv</code></td><td>每个正式 benchmark 的精确起止时间</td></tr>
        <tr><td><code>{scout,confirm}/{head,worker}/rdma.csv</code></td><td>原始 HCA counter 时间序列，仅保留在服务器完整结果中</td></tr>
        <tr><td><code>commands/*.cmd.txt</code></td><td>实际传给 Phase 2 的完整可复现命令</td></tr>
      </tbody>
    </table>
    <h2>6. 测试与验收门槛</h2>
    <ul>
      <li><code>python3 -m unittest test_rdma_demand_model.py</code>：4/4 通过。</li>
      <li><code>bash -n run_rdma_demand_modeling.sh</code> 与 Python compile：通过。</li>
      <li><code>DRY_RUN=1 ... all</code>：展开 5 个 Scout 和 3×2 个 Confirm，不启动服务。</li>
      <li>正式 Run：Scout 5/5、Confirm 6/6；两个阶段各 18/18 个采集器正常启停，共保存 72 条 STARTED/STOPPED 生命周期事件。</li>
      <li>11 个测量结果全部 <code>COMPLETED</code>，<code>rdma_error_delta=0</code>。</li>
      <li>结束后 Head/Worker 均无实验容器，16 张 GPU 为 0 MiB / 0%。</li>
    </ul>
    <h2>7. 行号索引</h2>
    <table>
      <thead><tr><th>功能</th><th>文件与行</th></tr></thead>
      <tbody>
        <tr><td>配置与路径</td><td><code>config.env:3-36</code></td></tr>
        <tr><td>校验与场景生成</td><td><code>run_rdma_demand_modeling.sh:42-100</code></td></tr>
        <tr><td>Phase 2 调用</td><td><code>run_rdma_demand_modeling.sh:103-134</code></td></tr>
        <tr><td>Scout / Confirm</td><td><code>run_rdma_demand_modeling.sh:136-180</code></td></tr>
        <tr><td>唯一 all 与清理</td><td><code>run_rdma_demand_modeling.sh:206-264</code></td></tr>
        <tr><td>HCA interval rate</td><td><code>rdma_demand_model.py:117-147</code></td></tr>
        <tr><td>Case 对齐汇总</td><td><code>rdma_demand_model.py:171-281</code></td></tr>
        <tr><td>饱和拟合</td><td><code>rdma_demand_model.py:318-350</code></td></tr>
        <tr><td>平台选点</td><td><code>rdma_demand_model.py:362-392</code></td></tr>
        <tr><td>需求模型与 verdict</td><td><code>rdma_demand_model.py:394-447</code></td></tr>
        <tr><td>报告输出</td><td><code>rdma_demand_model.py:449-517</code></td></tr>
      </tbody>
    </table>
    <h2>8. 复用时必须重新标定的边界</h2>
    <p>这套代码可复用，但 <code>3.332 MB/token/rail</code> 不是通用常数。换模型、量化、TP/EP、节点切分、backend、Prefill/Decode 形状或 Prefix Cache 策略后，都必须重新跑 Scout。代码输出的是“当前部署实现的经验模型”，不是由参数量单独推导出的理论通信量。</p>
  </main>
 </body>
 </html>
--- a/docs/dsv4pro_pro6000d_2node_sglang/phase2_5_exp.html
+++ b/docs/dsv4pro_pro6000d_2node_sglang/phase2_5_exp.html
@ -0,0 +1,166 @@
 <!doctype html>
 <html lang="zh-CN">
 <head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Phase 2.5：DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模</title>
  <style>
    :root { --ink:#18202a; --muted:#5b6570; --line:#d8dde3; --paper:#fff; --page:#f3f5f7; --blue:#1769aa; --green:#16734a; --amber:#9a5a00; --code:#f0f3f6; }
    * { box-sizing:border-box; letter-spacing:0; }
    body { margin:0; background:var(--page); color:var(--ink); font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","PingFang SC",sans-serif; font-size:16px; line-height:1.68; }
    header { background:#202a34; color:#fff; border-bottom:5px solid #34a17a; }
    .header-inner, main { width:min(1120px,calc(100% - 32px)); margin:0 auto; }
    .header-inner { padding:36px 0 30px; }
    h1 { margin:4px 0 12px; font-size:clamp(27px,4vw,42px); line-height:1.2; }
    h2 { margin:38px 0 12px; padding-bottom:7px; border-bottom:2px solid var(--line); font-size:24px; }
    h3 { margin:26px 0 8px; font-size:19px; }
    .eyebrow { margin:0; color:#8fd8bd; font-size:13px; font-weight:700; }
    .meta { display:flex; flex-wrap:wrap; gap:8px 22px; color:#d7dee5; font-size:14px; }
    main { margin-top:24px; margin-bottom:48px; padding:30px 38px 42px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
    a { color:var(--blue); }
    .back { display:inline-block; margin:0 16px 10px 0; font-weight:650; }
    code { padding:1px 5px; background:var(--code); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; font-size:.92em; overflow-wrap:anywhere; }
    pre { overflow-x:auto; padding:14px 16px; background:#202a34; color:#f4f7fa; border-radius:5px; line-height:1.5; }
    pre code { padding:0; background:transparent; color:inherit; }
    table { width:100%; margin:14px 0 22px; border-collapse:collapse; font-size:14px; }
    th,td { padding:10px 11px; border:1px solid var(--line); text-align:left; vertical-align:top; }
    th { background:#edf1f4; }
    .status,.decision,.warning { padding:14px 16px; }
    .status { border-left:4px solid var(--green); background:#eef8f3; }
    .decision { border-left:4px solid var(--blue); background:#edf5fb; }
    .warning { border-left:4px solid var(--amber); background:#fff7e7; }
    .metric { font-size:20px; font-weight:750; color:var(--green); }
    li+li { margin-top:5px; }
    @media (max-width:720px) { main{padding:22px 18px 30px} table{display:block;overflow-x:auto;white-space:nowrap} }
  </style>
 </head>
 <body>
  <header>
    <div class="header-inner">
      <p class="eyebrow">FINAL EXPERIMENT RECORD / PHASE 2.5</p>
      <h1>DeepSeek-V4-Pro 双机 Pro6000D SGLang：RDMA 需求建模与并发拐点</h1>
      <div class="meta">
        <span>Run：dsv4pro-phase2_5-20260801-130007</span>
        <span>拓扑：TP16 / EP2 / 双 Rail RoCE</span>
        <span>完成：2026-08-01 15:14:11 CST</span>
      </div>
    </div>
  </header>
  <main>
    <a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
    <a class="back" href="./phase2_5_code.html">打开 Phase 2.5 代码详解</a>
    <p class="status"><strong>阶段已完成。</strong>正式 Run 用时 2 小时 14 分 04 秒；Scout 5/5、Confirm 6/6 成功，两阶段各 18/18 个采集器正常启停。所有测量窗 RDMA 错误增量为 0，结束后两节点容器和 16 张 GPU 均已清理。</p>
    <h2>1. 要回答的问题</h2>
    <p>Phase 2 只看到代表负载约 83.5 Gbit/s/rail，不能判断继续增加并发是否会逼近 400G。Phase 2.5 专门回答三个问题：</p>
    <ol>
      <li>固定模型、TP/EP 和输入形状后，Input TPS 与每 Rail RDMA 带宽是什么关系？</li>
      <li>并发增加到哪里后，模型吞吐和 RDMA 带宽不再增长？</li>
      <li>要达到 400G，需要怎样的 Input TPS；当前瓶颈先出现在模型计算还是网络？</li>
    </ol>
    <h2>2. 实验设计</h2>
    <table>
      <thead><tr><th>阶段</th><th>请求形状</th><th>并发</th><th>重复</th><th>目的</th></tr></thead>
      <tbody>
        <tr><td>Scout</td><td>64K → 1</td><td>1 / 4 / 16 / 32 / 64</td><td>1</td><td>隔离 Prefill，找吞吐与带宽平台</td></tr>
        <tr><td>Confirm</td><td>64K → 1K</td><td>自动选择 4 / 16 / 64</td><td>2</td><td>验证真实长输出不会推翻需求模型</td></tr>
      </tbody>
    </table>
    <p>服务参数沿用 Phase 1/2：SGLang nightly、TP16、EP2、双 Rail <code>mlx5_0/mlx5_3</code>、<code>NET/IB + GDRDMA</code>。每个 Case 使用冷 Prefix，并按 benchmark 正式测量窗口切片 HCA Counter。</p>
    <h2>3. 实际启动命令</h2>
    <p>只在 Head <code>174.1.51.5</code> 执行，不需要 <code>source</code> 或 <code>conda activate</code>：</p>
    <pre><code>cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
 RUN_ID=dsv4pro-phase2_5-20260801-130007
 tmux new-session -d -s dsv4pro-phase2_5 \
  "RUN_ID=${RUN_ID} bash run_rdma_demand_modeling.sh all \
   2&gt;&amp;1 | tee /data/hzy/${RUN_ID}.log"
 tmux attach -t dsv4pro-phase2_5</code></pre>
    <p>实际展开后的 Scout/Confirm 命令分别保存在结果目录的 <code>commands/scout.cmd.txt</code> 与 <code>commands/confirm.cmd.txt</code>。</p>
    <h2>4. Scout 结果：并发 16 已进入平台</h2>
    <table>
      <thead><tr><th>C</th><th>Input TPS</th><th>Rail Mean</th><th>Rail P95</th><th>Rail Max</th><th>双 Rail 单向合计</th><th>MB/input-token/rail</th><th>GPU Util</th></tr></thead>
      <tbody>
        <tr><td>1</td><td>2,709.64</td><td>70.97 Gbit/s</td><td>85.60</td><td>86.81</td><td>141.94</td><td>3.138</td><td>93.75%</td></tr>
        <tr><td>4</td><td>2,930.49</td><td>78.27 Gbit/s</td><td>86.82</td><td>88.86</td><td>156.55</td><td>3.298</td><td>97.45%</td></tr>
        <tr><td>16</td><td>2,983.77</td><td>79.90 Gbit/s</td><td>86.42</td><td>88.20</td><td>159.79</td><td>3.336</td><td>99.33%</td></tr>
        <tr><td>32</td><td>2,983.92</td><td>79.93 Gbit/s</td><td>86.23</td><td>88.22</td><td>159.86</td><td>3.344</td><td>99.49%</td></tr>
        <tr><td>64</td><td>2,991.28</td><td>79.97 Gbit/s</td><td>85.93</td><td>88.32</td><td>159.95</td><td>3.340</td><td>99.64%</td></tr>
      </tbody>
    </table>
    <p class="decision"><strong>观察结论：</strong>C=16→32 的 Input TPS 只增长 0.005%，Rail Mean 只增长 0.040%；C=32→64 也仅增长 0.247% / 0.057%。并发 16 已是平台拐点，继续加到 64 只会增加排队和 TTFT，不会增加网络压力。</p>
    <p><strong>证据：</strong><code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv</code>；原始 HCA 数据位于同一 Run 的 <code>scout/head/rdma.csv</code> 与 <code>scout/worker/rdma.csv</code>，精确窗口位于 <code>scout/case_windows.csv</code>。</p>
    <h2>5. Confirm 结果：加入 1K Decode 后仍由计算先饱和</h2>
    <table>
      <thead><tr><th>C</th><th>重复</th><th>Input TPS</th><th>Output TPS</th><th>Rail Mean</th><th>Rail P95</th><th>双 Rail单向合计</th><th>TTFT P95</th><th>TPOT P95</th></tr></thead>
      <tbody>
        <tr><td>4</td><td>2</td><td>2,072.24</td><td>32.38</td><td>57.33 Gbit/s</td><td>86.28</td><td>114.66</td><td>86.36 s</td><td>95.63 ms</td></tr>
        <tr><td>16</td><td>2</td><td>2,582.13</td><td>40.35</td><td>71.64 Gbit/s</td><td>86.30</td><td>143.29</td><td>336.85 s</td><td>356.24 ms</td></tr>
        <tr><td>64</td><td>2</td><td>2,588.96</td><td>40.45</td><td>72.19 Gbit/s</td><td>86.69</td><td>144.38</td><td>1,506.50 s</td><td>418.58 ms</td></tr>
      </tbody>
    </table>
    <p>C=16→64 的 Input TPS 仅增长 0.26%，Rail Mean 仅增长 0.76%，但 TTFT P95 从 336.85 秒升至 1,506.50 秒。对于 64K→1K，<strong>最大有意义并发仍约为 16</strong>；C=64 是容量压力点，不是推荐服务点。</p>
    <p><strong>证据：</strong>同一 Run 的 <code>confirm/bench_summary.csv</code>、<code>confirm/case_rdma_summary.csv</code>、<code>confirm/case_windows.csv</code>；两轮逐点数据在顶层 <code>rdma_case_metrics.csv</code>。</p>
    <h2>6. 400G 能否被模型负载打满</h2>
    <p>Scout 的线性比例为：</p>
    <pre><code>每 Rail 带宽（Gbit/s）
 ≈ Input TPS × 3.332 MB/input-token/rail × 8 ÷ 1e9</code></pre>
    <table>
      <thead><tr><th>目标口径</th><th>需要的 Input TPS</th><th>当前约 2,991 TPS 的差距</th></tr></thead>
      <tbody>
        <tr><td>单 Rail 400G</td><td class="metric">15,006 tok/s</td><td>约 5.02×</td></tr>
        <tr><td>单 Rail 360G（90% 实用线）</td><td>13,505 tok/s</td><td>约 4.51×</td></tr>
        <tr><td>双 Rail 单向合计 400G</td><td>7,503 tok/s</td><td>约 2.51×</td></tr>
      </tbody>
    </table>
    <p>拟合得到当前模型负载的单 Rail 渐近上限约 <strong>80.32 Gbit/s</strong>，即物理 400G 的约 20.1%。瞬时 Max 也只有 89.53 Gbit/s。结论不是“网络只能跑 80G”，而是<strong>当前 DSV4-Pro TP16/EP2 实现最多只能产生约 80G/rail 的持续 RDMA 流量</strong>；Phase 2 的 NCCL microbenchmark 已证明链路本身能达到更高通信带宽。</p>
    <p class="warning"><strong>口径提醒：</strong>400G 是每条 Rail 的线速；双 Rail 单向总量是两条 Rail 的 TX 之和。不要把 TX 与 RX 相加后声称打满，也不要把 NCCL <code>busbw GB/s</code> 与 HCA <code>Gbit/s</code> 直接比较。</p>
    <h2>7. 一套可复用的 RDMA 需求评估方法</h2>
    <ol>
      <li><strong>固定部署变量。</strong>记录模型版本、精度/量化、TP/EP/PP/DP、节点数、Attention/MoE backend、chunked prefill 和网卡拓扑。任一项变化都要重新标定。</li>
      <li><strong>先选 Prefill Scout。</strong>固定 ISL，OSL=1，取稀疏并发点如 1/4/16/32/64；每点清 Prefix Cache，并保证请求文本实际达到目标 token 数。</li>
      <li><strong>对齐正式测量窗。</strong>从 benchmark 的 main-run 起止时间切片 Head/Worker 的 <code>mlx5_*</code> HCA Counter，不能用整个进程寿命，也不能只看 <code>sar eth*</code>。</li>
      <li><strong>计算通信强度。</strong><code>bytes_per_input_token_per_rail = rail_xmit_bytes / total_input_tokens</code>。这是该模型与并行策略下“每处理一个输入 token，要在一条 Rail 发送多少字节”。</li>
      <li><strong>找并发平台。</strong>同时观察 Input TPS 和 Rail Mean；连续一点的增益都低于阈值（本实验 5%）时，记为拐点。最大 C 不等于最大有效 C。</li>
      <li><strong>推导目标吞吐。</strong><code>required_input_tps = target_rail_gbps × 1e9 / (bytes_per_token × 8)</code>。若模型的实测/拟合 TPS 上限远低于该值，网络不会先饱和。</li>
      <li><strong>用业务 OSL 复测。</strong>在平台前、拐点、最高压力点各重复至少两次，确认 Decode、KV Cache 和调度没有改变结论。</li>
      <li><strong>最后做链路对照。</strong>模型负载未打满时，用 NCCL microbenchmark 验证网络能力，把“模型产流量不足”与“网络本身跑不满”分开。</li>
    </ol>
    <h3>7.1 哪些变量会改变 bytes/token 与平台</h3>
    <table>
      <thead><tr><th>变量</th><th>可能改变的原因</th></tr></thead>
      <tbody>
        <tr><td>模型架构与层数</td><td>每 token 触发的 TP collective、MoE dispatch/combine 和激活尺寸不同</td></tr>
        <tr><td>TP / EP / PP / DP</td><td>通信参与 rank、跨机边界、collective 类型和频率改变</td></tr>
        <tr><td>Prefill / Decode、ISL / OSL</td><td>计算强度、chunk 调度、KV 访问和 collective 消息粒度不同</td></tr>
        <tr><td>并发与 batch</td><td>决定 kernel/batch 效率和 Input TPS；超过平台后只增加排队</td></tr>
        <tr><td>量化与 backend</td><td>改变计算速度；通信字节可能不同比例变化，因此会移动“计算先饱和还是网络先饱和”的边界</td></tr>
        <tr><td>Prefix Cache</td><td>命中会绕过大量 Prefill，必须单独作为另一类业务场景建模</td></tr>
      </tbody>
    </table>
    <h2>8. 最终结论</h2>
    <p class="decision"><strong>在两台 Pro6000D、DSV4-Pro、SGLang TP16/EP2 的当前实现中，RDMA 不是吞吐瓶颈。</strong>64K Prefill 在 C=16 已达到约 3K input tok/s 和 80 Gbit/s/rail 的平台；继续增加并发到 64 不会显著增加吞吐或带宽，只会令 TTFT 急剧上升。要打满单 Rail 400G，模型侧 Input TPS 需提高到约 15K，约为当前上限 5 倍。因此后续优化应先看 GPU Kernel、MoE/Attention 执行和 rank 同步，而不是扩容计算网。</p>
    <h2>9. 证据与清理</h2>
    <ul>
      <li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_report.md">自动 RDMA 需求报告</a></li>
      <li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_model.json">机器可读模型</a></li>
      <li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv">全部逐点指标</a></li>
      <li><a href="./results/dsv4pro-phase2_5-20260801-130007/run_manifest.txt">Run Manifest</a></li>
      <li><a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/head_server_cmd.txt">Scout Head 服务命令</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_server_cmd.txt">Worker 服务命令</a></li>
      <li><a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_nccl_transport.log">Confirm Head NCCL 路径</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_nccl_transport.log">Worker NCCL 路径</a></li>
    </ul>
    <p>Worker 在最后一个 Case 完成后随 Head 主动关闭进程组出现 Gloo peer-close Traceback；它发生在测量结束与结果落盘之后，不是实验失败。最终 tmux、服务容器和 GPU 进程均已退出。</p>
  </main>
 </body>
 </html>
--- a/docs/dsv4pro_pro6000d_2node_sglang/phase2_exp.html
+++ b/docs/dsv4pro_pro6000d_2node_sglang/phase2_exp.html
@ -805,6 +805,7 @@ tmux new-session -d -s dsv4pro-phase2 \
    </p>
    <p><a class="back" href="./phase1_exp.html">返回 Phase 1 实验档案</a></p>
    <p><a class="back" href="./phase2_5_exp.html">继续 Phase 2.5 RDMA 需求建模</a></p>
    <p><a class="back" href="./推理优化计划.html">返回推理优化主计划</a></p>
  </main>
 </body>
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/commands/confirm.cmd.txt
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/commands/confirm.cmd.txt
@ -0,0 +1 @@
 env DRY_RUN=0 RUN_ID=confirm RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/confirm.tsv FIXED_CASE_IDS=rdma_confirm_i65536_o1024_c4\,rdma_confirm_i65536_o1024_c16\,rdma_confirm_i65536_o1024_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=2 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all 
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/commands/scout.cmd.txt
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/commands/scout.cmd.txt
@ -0,0 +1 @@
 env DRY_RUN=0 RUN_ID=scout RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/scout.tsv FIXED_CASE_IDS=rdma_scout_i65536_o1_c1\,rdma_scout_i65536_o1_c4\,rdma_scout_i65536_o1_c16\,rdma_scout_i65536_o1_c32\,rdma_scout_i65536_o1_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=1 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all 
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/bench_summary.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/bench_summary.csv
@ -0,0 +1,7 @@
 run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
 fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:03:43+0800,2026-08-01T14:11:03+0800,440.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,406.0844679859583,bench_main_marker_plus_duration,16,0,406.0844679859583,15.997723352218085,,1048576,16384,0.03940066971621592,2582.1622905219265,40.3462857894051,2622.5085763113316,,406026.68602950143,406024.73604350234,406053.3737935184,406053.5528299224,189372.86670175308,189251.9370575028,336778.7128902273,348624.74093402264,211.7828145921294,211.89852223703926,356.41907673483706,369.22852823690624,211.7827865053078,53.14174899831414,53.806967759737745,55.23117201111745,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c16
 fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:11:03+0800,2026-08-01T14:18:24+0800,441.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,406.09330375201534,bench_main_marker_plus_duration,16,0,406.09330375201534,15.997232835777202,,1048576,16384,0.03939981243761298,2582.1061079114043,40.34540793611569,2622.45151584752,,406023.07082318654,406021.80599197163,406049.04031673505,406056.0000617319,189594.6209833055,189467.45998901315,336924.2522477143,348614.9598911492,211.56251206244485,211.68662201320043,356.06784952074946,368.88221031313344,211.56248251307755,53.15867651370354,54.19228565006051,55.67694299679715,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c16
 fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T13:58:15+0800,2026-08-01T14:00:56+0800,161.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.71871886099689,bench_main_marker_plus_duration,4,0,127.71871886099689,3.9987289259054917,,262144,4096,0.031318823393095684,2052.5104098899187,32.07047515452998,2084.5808850444487,,127678.13387226488,127678.37510351092,127682.15249372879,127682.41568037542,58779.63522000937,59286.87346848892,87520.75181677355,89912.75672257761,67.34946104814811,66.85386279083285,96.14249014501952,98.71897348003505,67.3494309836352,36.35076849604957,36.65076588222291,37.22598696593195,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c4
 fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T14:00:57+0800,2026-08-01T14:03:36+0800,159.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,125.30951252899831,bench_main_marker_plus_duration,4,0,125.30951252899831,3.9986913595171454,,262144,4096,0.03192096050229504,2091.9720674784076,32.68706355435012,2124.6591310327576,,125268.51625375275,125267.95494600083,125274.71757131862,125275.15292868251,56944.67479274317,57454.862895508995,85207.63279019155,87584.75151082734,66.7877238133036,66.2884575273625,95.11951526871552,97.68299949875195,66.78769251954503,36.23565100133419,36.565644716029055,37.02181185770315,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c4
 fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:18:31+0800,2026-08-01T14:46:05+0800,1654.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.8975476099877,bench_main_marker_plus_duration,64,0,1619.8975476099877,40.3918658202939,,4194304,65536,0.039508671455442484,2589.2402925038787,40.456879570373104,2629.6971720742517,,1022354.4435263402,950470.9995904996,1619639.9038552889,1619650.3429244931,785995.8297684161,774078.3500669932,1506265.7001026294,1560787.9135349835,231.04458822866482,223.80146996136563,418.57784627036636,439.7380822666596,231.0445591993489,56.0161369794514,56.77989884279668,58.514122637570836,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c64
 fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:46:06+0800,2026-08-01T15:13:40+0800,1654.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1620.2422464910196,bench_main_marker_plus_duration,64,0,1620.2422464910196,40.39704233770943,,4194304,65536,0.03950026617229964,2588.689443867829,40.44827256043483,2629.137716428264,,1022703.0410756742,950775.2832530241,1620010.8873196847,1620019.4079715456,786335.3449273454,774403.2157540205,1506724.735757988,1561196.831805848,231.05346642065376,223.83287831180922,418.5783493891426,439.7222371669007,231.0534379278651,56.055732507957146,56.75308680802118,58.11881933768746,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c64
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/case_rdma_summary.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/case_rdma_summary.csv
@ -0,0 +1,25 @@
 phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_0,371,3626828296476.0,3626837993952.0,71.66006119953522,71.66025280544177,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_3,371,3626914718800.0,3626921347028.0,71.66176876011448,71.66189972281393,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_0,383,3626134447036.0,3626135801124.0,71.58004520955629,71.58007193930987,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_3,383,3626207403788.0,3626212421504.0,71.58148537888337,71.58158442880648,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_0,371,3625059758564.0,3625068124448.0,71.59790115790838,71.59806639097366,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_3,371,3625136689400.0,3625142126056.0,71.59942060496823,71.59952798337986,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_0,382,3625042688652.0,3625045908572.0,71.73367967522776,71.73374339219049,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_3,382,3625122942520.0,3625131203508.0,71.73526776832192,71.73543123979464,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_0,117,894188173332.0,894192013996.0,56.331515043976715,56.33175699575861,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_3,117,908654087048.0,908656485748.0,57.24282975425881,57.24298086608198,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_0,121,908912097324.0,908914949500.0,57.10271925252004,57.102898441471176,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_3,121,894353999288.0,894353927048.0,56.188101670195095,56.188097131691926,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_0,115,894195520320.0,894197433708.0,57.32326114869037,57.32338380826259,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_3,115,908661063200.0,908663318872.0,58.25058864399148,58.250733246007684,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_0,118,908220840268.0,908222493124.0,58.56932688787363,58.5694334772174,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_3,118,893986155676.0,893986644792.0,57.65136084035526,57.651392382457814,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_0,1479,14610070965300.0,14610072489408.0,72.21898772698583,72.21899526079844,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_3,1479,14610085802320.0,14610095102220.0,72.21906106780447,72.21910703810576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_0,1527,14611153636236.0,14611146819600.0,72.16844452340027,72.1684108541891,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_3,1527,14611226802652.0,14611236350532.0,72.16880591214243,72.16885307170732,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_0,1480,14610569508844.0,14610573106108.0,72.18614224806339,72.18616002099213,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_3,1480,14610597458160.0,14610605842440.0,72.18628033667787,72.18632176071877,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_0,1527,14610266497856.0,14610260234736.0,72.18231262718308,72.18228168411397,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_3,1527,14610323772088.0,14610331114336.0,72.18259559166701,72.18263186618948,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/case_windows.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/case_windows.csv
@ -0,0 +1,7 @@
 phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,1785564254150768896,1785564660235236864,406.084467968,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/meta.json
 fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,1785564694981654016,1785565101074958080,406.093304064,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/meta.json
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,1785563926377015040,1785564054095733760,127.71871872,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/meta.json
 fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,1785564088360896000,1785564213670408960,125.30951296,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/meta.json
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1785565142582675968,1785566762480224000,1619.897548032,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/meta.json
 fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1785566797465815040,1785568417708060928,1620.242245888,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/meta.json
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/collector_status.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/collector_status.csv
@ -0,0 +1,37 @@
 wall_time_ns,node,collector,status,pid
 1785563885833567643,head,gpu_samples.csv,STARTED,2581028
 1785563885848767676,head,rdma.csv,STARTED,2581039
 1785563885862486752,head,dcgm_dmon.log,STARTED,2581053
 1785563885876598354,head,mpstat.log,STARTED,2581071
 1785563885891181421,head,pidstat.log,STARTED,2581090
 1785563885906513050,head,sar_net.log,STARTED,2581117
 1785563885921729008,head,perf_stat.log,STARTED,2581144
 1785563885938632713,head,docker_top.log,STARTED,2581179
 1785563885959612586,head,numa_samples.csv,STARTED,2581251
 1785563885980492840,worker,gpu_samples.csv,STARTED,2581294
 1785563885999012597,worker,rdma.csv,STARTED,2581356
 1785563886014677475,worker,dcgm_dmon.log,STARTED,2581408
 1785563886031239484,worker,mpstat.log,STARTED,2581455
 1785563886046314326,worker,pidstat.log,STARTED,2581464
 1785563886060934631,worker,sar_net.log,STARTED,2581478
 1785563886075086128,worker,perf_stat.log,STARTED,2581486
 1785563886090622653,worker,docker_top.log,STARTED,2581497
 1785563886106525841,worker,numa_samples.csv,STARTED,2581509
 1785568431501511663,head,gpu_samples.csv,STOPPED,2581028
 1785568431506303146,head,rdma.csv,STOPPED,2581039
 1785568431513026775,head,dcgm_dmon.log,STOPPED,2581053
 1785568431518181488,head,mpstat.log,STOPPED,2581071
 1785568431523384390,head,pidstat.log,STOPPED,2581090
 1785568431528223936,head,sar_net.log,STOPPED,2581117
 1785568431608439580,head,perf_stat.log,STOPPED,2581144
 1785568431612888149,head,docker_top.log,STOPPED,2581179
 1785568431617027570,head,numa_samples.csv,STOPPED,2581251
 1785568431620791766,worker,gpu_samples.csv,STOPPED,2581294
 1785568431624694474,worker,rdma.csv,STOPPED,2581356
 1785568431628587426,worker,dcgm_dmon.log,STOPPED,2581408
 1785568431632294659,worker,mpstat.log,STOPPED,2581455
 1785568431636153536,worker,pidstat.log,STOPPED,2581464
 1785568431639870020,worker,sar_net.log,STOPPED,2581478
 1785568431643739953,worker,perf_stat.log,STOPPED,2581486
 1785568431647558030,worker,docker_top.log,STOPPED,2581497
 1785568431651292267,worker,numa_samples.csv,STOPPED,2581509
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_nccl_transport.log
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_nccl_transport.log
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_server_cmd.txt
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_server_cmd.txt
@ -0,0 +1 @@
 docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256 
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_nccl_transport.log
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_nccl_transport.log
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_server_cmd.txt
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_server_cmd.txt
@ -0,0 +1 @@
 docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256 
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv
@ -0,0 +1,12 @@
 stage,case_id,repetition,status,isl,osl,concurrency,completed,duration_s,input_tps,output_tps,total_tps,ttft_p95_ms,tpot_p95_ms,gpu_util_mean_pct,rail_xmit_avg_gbps,rail_xmit_p95_gbps,rail_xmit_max_gbps,dual_rail_xmit_avg_gbps,rail_imbalance_pct,xmit_bytes_per_input_token_per_rail,xmit_bytes_per_total_token_per_rail,rdma_error_delta,measurement_started_at,measurement_ended_at,source_result_dir
 scout,rdma_scout_i65536_o1_c1,1,COMPLETED,65536,1,1,1,24.1862671909621,2709.6368150803128,0.04134577659729481,2709.67816085691,24167.605170048773,0.0,93.74897875816994,70.9712746848062,85.5972372945909,86.80566056858429,141.9425493696124,0.5207119247408836,3137732.762954712,3137684.8856828967,0.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
 scout,rdma_scout_i65536_o1_c4,1,COMPLETED,65536,1,4,4,89.45394851703895,2930.491100122512,0.044715745546302976,2930.535815868058,86526.69315790116,0.0,97.4494891826923,78.27464182068744,86.81919268342193,88.85894624954848,156.5492836413749,0.2703571902588796,3298012.1388320923,3297961.815928407,0.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
 scout,rdma_scout_i65536_o1_c16,1,COMPLETED,65536,1,16,16,351.42641380499117,2983.771164628114,0.045528734811830354,2983.816693362926,336717.4506017618,0.0,99.33137833282255,79.89685137869246,86.416508483831,88.20167661610235,159.79370275738495,0.1149304623247205,3335858.008945465,3335807.108568442,0.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
 scout,rdma_scout_i65536_o1_c32,1,COMPLETED,65536,1,32,32,702.8175486620166,2983.9209393567885,0.04553102019282209,2983.9664703769813,671139.2772838357,0.0,99.49393723047987,79.92852806348517,86.22587499587476,88.21644913628985,159.85705612697035,0.10091204027725086,3343929.3806581497,3343878.357123648,0.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
 scout,rdma_scout_i65536_o1_c64,1,COMPLETED,65536,1,64,64,1402.17612864502,2991.2818470623424,0.04564333873080967,2991.327490401073,1337131.1103800863,0.0,99.63928846991146,79.9743866369171,85.93278751515446,88.32212536165416,159.9487732738342,0.03496630506355109,3339597.901974201,3339546.9445318105,0.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
 confirm,rdma_confirm_i65536_o1024_c4,1,COMPLETED,65536,1024,4,4,127.71871886099689,2052.5104098899187,32.07047515452998,2084.5808850444487,87520.75181677355,96.14249014501952,97.83383458646617,56.71629143023766,86.06750356786138,88.99885476939444,113.43258286047534,1.8596562953363254,3439052.922241211,3386144.415745192,0.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
 confirm,rdma_confirm_i65536_o1024_c4,2,COMPLETED,65536,1024,4,4,125.30951252899831,2091.9720674784076,32.68706355435012,2124.6591310327576,85207.63279019155,95.11951526871552,98.63248207885304,57.94863438022769,86.4894823881113,88.5335715781298,115.89726876045536,2.1502935358359734,3438056.544746399,3385163.3671349157,0.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
 confirm,rdma_confirm_i65536_o1024_c16,1,COMPLETED,65536,1024,16,16,406.0844679859583,2582.1622905219265,40.3462857894051,2622.5085763113316,336778.7128902273,356.41907673483706,99.38883591065292,71.62084013702234,86.26345231858909,88.9399300384103,143.24168027404468,0.11410582506689998,3458520.1421022415,3405312.139916053,0.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
 confirm,rdma_confirm_i65536_o1024_c16,2,COMPLETED,65536,1024,16,16,406.09330375201534,2582.1061079114043,40.34540793611569,2622.45151584752,336924.2522477143,356.06784952074946,99.48923731257209,71.66656730160658,86.33499788736604,89.31414888004788,143.33313460321313,0.19167460586668467,3457155.7233657837,3403968.712237079,0.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
 confirm,rdma_confirm_i65536_o1024_c64,1,COMPLETED,65536,1024,64,64,1619.8975476099877,2589.2402925038787,40.456879570373104,2629.6971720742517,1506265.7001026294,418.57784627036636,99.6722631148773,72.19382480758325,86.71108451788668,89.82468221963666,144.38764961516648,0.070112013789426,3483446.6699664593,3429855.182736206,0.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
 confirm,rdma_confirm_i65536_o1024_c64,2,COMPLETED,65536,1024,64,64,1620.2422464910196,2588.689443867829,40.44827256043483,2629.137716428264,1506724.735757988,418.5783493891426,99.67225809785734,72.18433270089784,86.67873733217695,89.23672463088391,144.36866540179568,0.005496635275729702,3483400.1801579,3429809.4081554706,0.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/rdma_demand_model.json
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/rdma_demand_model.json
@ -0,0 +1,205 @@
 {
  "generated_at": "2026-08-01T15:14:11.558999+08:00",
  "formula": "rail_gbps ~= input_tps * bytes_per_input_token_per_rail * 8 / 1e9",
  "plateau_gain_pct": 5.0,
  "target_rail_gbps": 400.0,
  "practical_rail_gbps": 360.0,
  "scout": [
    {
      "stage": "scout",
      "concurrency": 1,
      "repetitions": 1,
      "input_tps": 2709.6368150803128,
      "output_tps": 0.04134577659729481,
      "total_tps": 2709.67816085691,
      "ttft_p95_ms": 24167.605170048773,
      "tpot_p95_ms": 0.0,
      "gpu_util_mean_pct": 93.74897875816994,
      "rail_xmit_avg_gbps": 70.9712746848062,
      "rail_xmit_p95_gbps": 85.5972372945909,
      "rail_xmit_max_gbps": 86.80566056858429,
      "dual_rail_xmit_avg_gbps": 141.9425493696124,
      "rail_imbalance_pct": 0.5207119247408836,
      "xmit_bytes_per_input_token_per_rail": 3137732.762954712,
      "xmit_bytes_per_total_token_per_rail": 3137684.8856828967,
      "rdma_error_delta": 0.0
    },
    {
      "stage": "scout",
      "concurrency": 4,
      "repetitions": 1,
      "input_tps": 2930.491100122512,
      "output_tps": 0.044715745546302976,
      "total_tps": 2930.535815868058,
      "ttft_p95_ms": 86526.69315790116,
      "tpot_p95_ms": 0.0,
      "gpu_util_mean_pct": 97.4494891826923,
      "rail_xmit_avg_gbps": 78.27464182068744,
      "rail_xmit_p95_gbps": 86.81919268342193,
      "rail_xmit_max_gbps": 88.85894624954848,
      "dual_rail_xmit_avg_gbps": 156.5492836413749,
      "rail_imbalance_pct": 0.2703571902588796,
      "xmit_bytes_per_input_token_per_rail": 3298012.1388320923,
      "xmit_bytes_per_total_token_per_rail": 3297961.815928407,
      "rdma_error_delta": 0.0
    },
    {
      "stage": "scout",
      "concurrency": 16,
      "repetitions": 1,
      "input_tps": 2983.771164628114,
      "output_tps": 0.045528734811830354,
      "total_tps": 2983.816693362926,
      "ttft_p95_ms": 336717.4506017618,
      "tpot_p95_ms": 0.0,
      "gpu_util_mean_pct": 99.33137833282255,
      "rail_xmit_avg_gbps": 79.89685137869246,
      "rail_xmit_p95_gbps": 86.416508483831,
      "rail_xmit_max_gbps": 88.20167661610235,
      "dual_rail_xmit_avg_gbps": 159.79370275738495,
      "rail_imbalance_pct": 0.1149304623247205,
      "xmit_bytes_per_input_token_per_rail": 3335858.008945465,
      "xmit_bytes_per_total_token_per_rail": 3335807.108568442,
      "rdma_error_delta": 0.0
    },
    {
      "stage": "scout",
      "concurrency": 32,
      "repetitions": 1,
      "input_tps": 2983.9209393567885,
      "output_tps": 0.04553102019282209,
      "total_tps": 2983.9664703769813,
      "ttft_p95_ms": 671139.2772838357,
      "tpot_p95_ms": 0.0,
      "gpu_util_mean_pct": 99.49393723047987,
      "rail_xmit_avg_gbps": 79.92852806348517,
      "rail_xmit_p95_gbps": 86.22587499587476,
      "rail_xmit_max_gbps": 88.21644913628985,
      "dual_rail_xmit_avg_gbps": 159.85705612697035,
      "rail_imbalance_pct": 0.10091204027725086,
      "xmit_bytes_per_input_token_per_rail": 3343929.3806581497,
      "xmit_bytes_per_total_token_per_rail": 3343878.357123648,
      "rdma_error_delta": 0.0
    },
    {
      "stage": "scout",
      "concurrency": 64,
      "repetitions": 1,
      "input_tps": 2991.2818470623424,
      "output_tps": 0.04564333873080967,
      "total_tps": 2991.327490401073,
      "ttft_p95_ms": 1337131.1103800863,
      "tpot_p95_ms": 0.0,
      "gpu_util_mean_pct": 99.63928846991146,
      "rail_xmit_avg_gbps": 79.9743866369171,
      "rail_xmit_p95_gbps": 85.93278751515446,
      "rail_xmit_max_gbps": 88.32212536165416,
      "dual_rail_xmit_avg_gbps": 159.9487732738342,
      "rail_imbalance_pct": 0.03496630506355109,
      "xmit_bytes_per_input_token_per_rail": 3339597.901974201,
      "xmit_bytes_per_total_token_per_rail": 3339546.9445318105,
      "rdma_error_delta": 0.0
    }
  ],
  "confirm": [
    {
      "stage": "confirm",
      "concurrency": 4,
      "repetitions": 2,
      "input_tps": 2072.2412386841634,
      "output_tps": 32.37876935444005,
      "total_tps": 2104.6200080386034,
      "ttft_p95_ms": 86364.19230348256,
      "tpot_p95_ms": 95.63100270686752,
      "gpu_util_mean_pct": 98.2331583326596,
      "rail_xmit_avg_gbps": 57.332462905232674,
      "rail_xmit_p95_gbps": 86.27849297798633,
      "rail_xmit_max_gbps": 88.76621317376211,
      "dual_rail_xmit_avg_gbps": 114.66492581046535,
      "rail_imbalance_pct": 2.0049749155861494,
      "xmit_bytes_per_input_token_per_rail": 3438554.733493805,
      "xmit_bytes_per_total_token_per_rail": 3385653.891440054,
      "rdma_error_delta": 0.0
    },
    {
      "stage": "confirm",
      "concurrency": 16,
      "repetitions": 2,
      "input_tps": 2582.1341992166654,
      "output_tps": 40.345846862760396,
      "total_tps": 2622.480046079426,
      "ttft_p95_ms": 336851.4825689708,
      "tpot_p95_ms": 356.24346312779323,
      "gpu_util_mean_pct": 99.4390366116125,
      "rail_xmit_avg_gbps": 71.64370371931446,
      "rail_xmit_p95_gbps": 86.29922510297757,
      "rail_xmit_max_gbps": 89.12703945922908,
      "dual_rail_xmit_avg_gbps": 143.28740743862892,
      "rail_imbalance_pct": 0.15289021546679232,
      "xmit_bytes_per_input_token_per_rail": 3457837.9327340126,
      "xmit_bytes_per_total_token_per_rail": 3404640.426076566,
      "rdma_error_delta": 0.0
    },
    {
      "stage": "confirm",
      "concurrency": 64,
      "repetitions": 2,
      "input_tps": 2588.964868185854,
      "output_tps": 40.45257606540397,
      "total_tps": 2629.4174442512576,
      "ttft_p95_ms": 1506495.2179303085,
      "tpot_p95_ms": 418.5780978297545,
      "gpu_util_mean_pct": 99.67226060636732,
      "rail_xmit_avg_gbps": 72.18907875424054,
      "rail_xmit_p95_gbps": 86.69491092503182,
      "rail_xmit_max_gbps": 89.53070342526028,
      "dual_rail_xmit_avg_gbps": 144.37815750848108,
      "rail_imbalance_pct": 0.03780432453257785,
      "xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
      "xmit_bytes_per_total_token_per_rail": 3429832.295445838,
      "rdma_error_delta": 0.0
    }
  ],
  "selected_confirm_concurrencies": [
    4,
    16,
    64
  ],
  "observed_plateau_concurrency": 16,
  "best_scout_bandwidth_concurrency": 64,
  "linear_gbps_per_input_tps": 0.026656368057740153,
  "modeled_bytes_per_input_token_per_rail": 3332046.007217519,
  "required_input_tps_for_400g_per_rail": 15005.795205617025,
  "required_input_tps_for_360g_per_rail": 13505.215685055322,
  "required_input_tps_for_400g_dual_rail_aggregate": 7502.897602808513,
  "bandwidth_saturation_fit": {
    "asymptote": 80.32102454669293,
    "half_saturation_c": 0.12589254117941673,
    "rmse": 0.27652726609534695
  },
  "input_tps_saturation_fit": {
    "asymptote": 3003.9418335884134,
    "half_saturation_c": 0.11220184543019636,
    "rmse": 7.680357081457988
  },
  "best_confirm_point": {
    "stage": "confirm",
    "concurrency": 64,
    "repetitions": 2,
    "input_tps": 2588.964868185854,
    "output_tps": 40.45257606540397,
    "total_tps": 2629.4174442512576,
    "ttft_p95_ms": 1506495.2179303085,
    "tpot_p95_ms": 418.5780978297545,
    "gpu_util_mean_pct": 99.67226060636732,
    "rail_xmit_avg_gbps": 72.18907875424054,
    "rail_xmit_p95_gbps": 86.69491092503182,
    "rail_xmit_max_gbps": 89.53070342526028,
    "dual_rail_xmit_avg_gbps": 144.37815750848108,
    "rail_imbalance_pct": 0.03780432453257785,
    "xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
    "xmit_bytes_per_total_token_per_rail": 3429832.295445838,
    "rdma_error_delta": 0.0
  },
  "verdict": "COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION"
 }
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/rdma_demand_report.md
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/rdma_demand_report.md
@ -0,0 +1,37 @@
 # Phase 2.5 RDMA Demand Model
 - Generated: `2026-08-01T15:14:11.558999+08:00`
 - Verdict: `COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION`
 - Model: `rail_gbps ~= input_tps x 3.332 MB/token/rail x 8`
 - Required Input TPS for 400G per rail: `15006`
 - Required Input TPS for 360G per rail: `13505`
 - Required Input TPS for 400G dual-rail aggregate: `7503`
 - Fitted rail bandwidth ceiling: `80.32 Gbit/s`
 - Observed plateau concurrency: `16`
 ## Scout: 64K -> 1
 | C | Input TPS | Rail avg | Rail P95 | Dual-rail avg | MB/input-token/rail | GPU util |
 |---:|---:|---:|---:|---:|---:|---:|
 | 1 | 2709.64 | 70.97 | 85.60 | 141.94 | 3.138 | 93.75 |
 | 4 | 2930.49 | 78.27 | 86.82 | 156.55 | 3.298 | 97.45 |
 | 16 | 2983.77 | 79.90 | 86.42 | 159.79 | 3.336 | 99.33 |
 | 32 | 2983.92 | 79.93 | 86.23 | 159.86 | 3.344 | 99.49 |
 | 64 | 2991.28 | 79.97 | 85.93 | 159.95 | 3.340 | 99.64 |
 ## Confirm: 64K -> 1K
 | C | Reps | Input TPS | Output TPS | Rail avg | Rail P95 | Dual-rail avg | TTFT P95 ms | TPOT P95 ms |
 |---:|---:|---:|---:|---:|---:|---:|---:|---:|
 | 4 | 2 | 2072.24 | 32.38 | 57.33 | 86.28 | 114.66 | 86364.19 | 95.63 |
 | 16 | 2 | 2582.13 | 40.35 | 71.64 | 86.30 | 143.29 | 336851.48 | 356.24 |
 | 64 | 2 | 2588.96 | 40.45 | 72.19 | 86.69 | 144.38 | 1506495.22 | 418.58 |
 ## Evidence
 - Per-case metrics: `rdma_case_metrics.csv`
 - Machine-readable model: `rdma_demand_model.json`
 - Scout HCA source: `scout/{head,worker}/rdma.csv`
 - Confirm HCA source: `confirm/{head,worker}/rdma.csv`
 - Exact windows: `scout/case_windows.csv`, `confirm/case_windows.csv`
 - Benchmark summaries: `scout/bench_summary.csv`, `confirm/bench_summary.csv`
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/recommendation.env
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/recommendation.env
@ -0,0 +1 @@
 CONFIRM_CONCURRENCIES='4 16 64'
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/run_manifest.txt
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/run_manifest.txt
@ -0,0 +1,13 @@
 run_id=dsv4pro-phase2_5-20260801-130007
 experiment=dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
 created_at=2026-08-01T13:00:07+0800
 git_commit=c5fa700c50c0cc1b9e20d4d2d3f03bd485ba0dbf
 git_dirty=0
 isl=65536
 scout_osl=1
 confirm_osl=1024
 scout_concurrencies=1 4 16 32 64
 scout_repetitions=1
 confirm_repetitions=2
 target_rail_gbps=400
 practical_rail_gbps=360
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scenarios/confirm.tsv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scenarios/confirm.tsv
@ -0,0 +1,4 @@
 # case_id	stage	isl	osl	concurrency	prompt_multiplier	min_prompts	warmup_requests	note
 rdma_confirm_i65536_o1024_c4	rdma_confirm	65536	1024	4	1	1	0	Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
 rdma_confirm_i65536_o1024_c16	rdma_confirm	65536	1024	16	1	1	0	Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
 rdma_confirm_i65536_o1024_c64	rdma_confirm	65536	1024	64	1	1	0	Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scenarios/scout.tsv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scenarios/scout.tsv
@ -0,0 +1,6 @@
 # case_id	stage	isl	osl	concurrency	prompt_multiplier	min_prompts	warmup_requests	note
 rdma_scout_i65536_o1_c1	rdma_scout	65536	1	1	1	1	0	Phase 2.5 scout: correlate model token rate with dual-rail RDMA
 rdma_scout_i65536_o1_c4	rdma_scout	65536	1	4	1	1	0	Phase 2.5 scout: correlate model token rate with dual-rail RDMA
 rdma_scout_i65536_o1_c16	rdma_scout	65536	1	16	1	1	0	Phase 2.5 scout: correlate model token rate with dual-rail RDMA
 rdma_scout_i65536_o1_c32	rdma_scout	65536	1	32	1	1	0	Phase 2.5 scout: correlate model token rate with dual-rail RDMA
 rdma_scout_i65536_o1_c64	rdma_scout	65536	1	64	1	1	0	Phase 2.5 scout: correlate model token rate with dual-rail RDMA
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/bench_summary.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/bench_summary.csv
@ -0,0 +1,6 @@
 run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
 fixed_rdma_scout_i65536_o1_c1,fixed,rdma_scout_i65536_o1_c1,,rdma_scout,1,65536,1,1,1,0,COMPLETED,,0,2026-08-01T13:05:54+0800,2026-08-01T13:06:53+0800,59.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,24.1862671909621,bench_main_marker_plus_duration,1,0,24.1862671909621,0.9992315809294645,,65536,1,0.04134577659729481,2709.6368150803128,0.04134577659729481,2709.67816085691,,24167.682002007496,24167.682002007496,24167.682002007496,24167.682002007496,24167.605170048773,24167.605170048773,24167.605170048773,24167.605170048773,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/bench.log,fixed_rdma_scout_i65536_o1_c1
 fixed_rdma_scout_i65536_o1_c16,fixed,rdma_scout_i65536_o1_c16,,rdma_scout,1,65536,1,16,16,0,COMPLETED,,0,2026-08-01T13:09:09+0800,2026-08-01T13:15:34+0800,385.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,351.42641380499117,bench_main_marker_plus_duration,16,0,351.42641380499117,8.628873239905923,,1048576,16,0.045528734811830354,2983.771164628114,0.045528734811830354,2983.816693362926,,189525.8736173746,189769.315014506,336717.50638925005,348428.8343418157,189525.84591050618,189769.29299000767,336717.4506017618,348428.77078671294,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/bench.log,fixed_rdma_scout_i65536_o1_c16
 fixed_rdma_scout_i65536_o1_c32,fixed,rdma_scout_i65536_o1_c32,,rdma_scout,1,65536,1,32,32,0,COMPLETED,,0,2026-08-01T13:15:40+0800,2026-08-01T13:27:58+0800,738.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.8175486620166,bench_main_marker_plus_duration,32,0,702.8175486620166,16.63411701870453,,2097152,32,0.04553102019282209,2983.9209393567885,0.04553102019282209,2983.9664703769813,,365335.9171013453,365369.92930600536,671139.3307480408,696643.7062339701,365335.8620414692,365369.8761375272,671139.2772838357,696643.6536653765,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/bench.log,fixed_rdma_scout_i65536_o1_c32
 fixed_rdma_scout_i65536_o1_c4,fixed,rdma_scout_i65536_o1_c4,,rdma_scout,1,65536,1,4,4,0,COMPLETED,,0,2026-08-01T13:06:59+0800,2026-08-01T13:09:02+0800,123.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,89.45394851703895,bench_main_marker_plus_duration,4,0,89.45394851703895,2.5881014457616573,,262144,4,0.044715745546302976,2930.491100122512,0.044715745546302976,2930.535815868058,,57878.973371509346,58521.736774011515,86526.744662947,88841.4666661853,57878.91561826109,58521.6828455159,86526.69315790116,88841.41537077782,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/bench.log,fixed_rdma_scout_i65536_o1_c4
 fixed_rdma_scout_i65536_o1_c64,fixed,rdma_scout_i65536_o1_c64,,rdma_scout,1,65536,1,64,64,0,COMPLETED,,0,2026-08-01T13:28:04+0800,2026-08-01T13:52:01+0800,1437.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1402.17612864502,bench_main_marker_plus_duration,64,0,1402.17612864502,32.732392604632125,,4194304,64,0.04564333873080967,2991.2818470623424,0.04564333873080967,2991.327490401073,,717134.0553695618,717609.0070659702,1337131.160498035,1390460.9022671543,717134.0051560618,717608.9713889814,1337131.1103800863,1390460.8547802733,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/bench.log,fixed_rdma_scout_i65536_o1_c64
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/case_rdma_summary.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/case_rdma_summary.csv
@ -0,0 +1,21 @@
 phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
 fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.006126848,bench_main_marker_plus_duration,head,mlx5_0,22,203533427700.0,203651710996.0,70.77538224308064,70.81651330239592,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.006126848,bench_main_marker_plus_duration,head,mlx5_3,22,204100301340.0,204100542584.0,70.97250317308169,70.97258706169157,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.352313088,bench_main_marker_plus_duration,worker,mlx5_0,23,207229229632.0,207304460944.0,70.99227518955745,71.01804781832155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.352313088,bench_main_marker_plus_duration,worker,mlx5_3,23,207674858740.0,207674728976.0,71.14493813350504,71.14489367915073,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,350.585191424,bench_main_marker_plus_duration,head,mlx5_0,321,3499319374812.0,3499321894556.0,79.850934047694,79.85099154570729,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,350.585191424,bench_main_marker_plus_duration,head,mlx5_3,321,3499670238252.0,3499687020088.0,79.85894039704549,79.85932334159443,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,349.898471936,bench_main_marker_plus_duration,worker,mlx5_0,331,3496131787252.0,3496130977288.0,79.93477120166395,79.93475268282916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,349.898471936,bench_main_marker_plus_duration,worker,mlx5_3,331,3496481190036.0,3496480578468.0,79.94275986836642,79.94274588561318,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,701.184411136,bench_main_marker_plus_duration,head,mlx5_0,641,7008726918884.0,7008731821472.0,79.96443511947507,79.96449105440941,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,701.184411136,bench_main_marker_plus_duration,head,mlx5_3,641,7009132360248.0,7009232426556.0,79.9690609081556,79.97020259135803,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.616064768,bench_main_marker_plus_duration,worker,mlx5_0,663,7016359452164.0,7016357965840.0,79.88840339972316,79.88838647640956,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.616064768,bench_main_marker_plus_duration,worker,mlx5_3,663,7016694022728.0,7016692492508.0,79.89221282658687,79.89219540347258,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.716937728,bench_main_marker_plus_duration,head,mlx5_0,82,866863327600.0,866884788736.0,78.16891338226691,78.17084862814438,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.716937728,bench_main_marker_plus_duration,head,mlx5_3,82,867282690684.0,867282482924.0,78.20672921268122,78.20671047804,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.006032896,bench_main_marker_plus_duration,worker,mlx5_0,84,861825370952.0,861825497804.0,78.34239018322312,78.34240171443258,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.006032896,bench_main_marker_plus_duration,worker,mlx5_3,84,862244987252.0,862244674092.0,78.38053450457852,78.38050603743919,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.218045952,bench_main_marker_plus_duration,head,mlx5_0,1280,14005295392512.0,14005304292920.0,79.96069096010922,79.96074177537255,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.218045952,bench_main_marker_plus_duration,head,mlx5_3,1280,14005533397520.0,14005553945192.0,79.96204980648541,79.96216711968765,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.136956672,bench_main_marker_plus_duration,worker,mlx5_0,1321,14008943976736.0,14008941122316.0,79.9861507329604,79.98613443522456,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
 fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.136956672,bench_main_marker_plus_duration,worker,mlx5_3,1321,14009382587800.0,14009379963032.0,79.98865504811339,79.98864006160981,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/case_windows.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/case_windows.csv
@ -0,0 +1,6 @@
 phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
 fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,1785560786217761024,1785560810404027904,24.18626688,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/meta.json
 fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,1785560979903596032,1785561331330009856,351.426413824,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/meta.json
 fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,1785561372166763008,1785562074984312064,702.817549056,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/meta.json
 fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,1785560850521112832,1785560939975062016,89.453949184,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/meta.json
 fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1785562116631968000,1785563518808096768,1402.176128768,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/meta.json
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/collector_status.csv
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/collector_status.csv
@ -0,0 +1,37 @@
 wall_time_ns,node,collector,status,pid
 1785560744455556276,head,gpu_samples.csv,STARTED,2168337
 1785560744470800548,head,rdma.csv,STARTED,2168348
 1785560744485157848,head,dcgm_dmon.log,STARTED,2168362
 1785560744499919157,head,mpstat.log,STARTED,2168380
 1785560744514378735,head,pidstat.log,STARTED,2168400
 1785560744529055413,head,sar_net.log,STARTED,2168428
 1785560744543220536,head,perf_stat.log,STARTED,2168451
 1785560744559383367,head,docker_top.log,STARTED,2168486
 1785560744579614511,head,numa_samples.csv,STARTED,2168561
 1785560744602206477,worker,gpu_samples.csv,STARTED,2168604
 1785560744619402981,worker,rdma.csv,STARTED,2168664
 1785560744634896197,worker,dcgm_dmon.log,STARTED,2168717
 1785560744650285593,worker,mpstat.log,STARTED,2168763
 1785560744666005300,worker,pidstat.log,STARTED,2168775
 1785560744681017748,worker,sar_net.log,STARTED,2168787
 1785560744694432153,worker,perf_stat.log,STARTED,2168796
 1785560744709522681,worker,docker_top.log,STARTED,2168808
 1785560744723983222,worker,numa_samples.csv,STARTED,2168818
 1785563532345901412,head,gpu_samples.csv,STOPPED,2168337
 1785563532350635898,head,rdma.csv,STOPPED,2168348
 1785563532357352405,head,dcgm_dmon.log,STOPPED,2168362
 1785563532362419425,head,mpstat.log,STOPPED,2168380
 1785563532367417490,head,pidstat.log,STOPPED,2168400
 1785563532372197961,head,sar_net.log,STOPPED,2168428
 1785563532541034088,head,perf_stat.log,STOPPED,2168451
 1785563532545727806,head,docker_top.log,STOPPED,2168486
 1785563532550309298,head,numa_samples.csv,STOPPED,2168561
 1785563532554473337,worker,gpu_samples.csv,STOPPED,2168604
 1785563532558689660,worker,rdma.csv,STOPPED,2168664
 1785563532562663636,worker,dcgm_dmon.log,STOPPED,2168717
 1785563532566892350,worker,mpstat.log,STOPPED,2168763
 1785563532571076626,worker,pidstat.log,STOPPED,2168775
 1785563532575164431,worker,sar_net.log,STOPPED,2168787
 1785563532579199145,worker,perf_stat.log,STOPPED,2168796
 1785563532583371937,worker,docker_top.log,STOPPED,2168808
 1785563532587359055,worker,numa_samples.csv,STOPPED,2168818
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/service/head_nccl_transport.log
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/service/head_nccl_transport.log
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/service/head_server_cmd.txt
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/service/head_server_cmd.txt
@ -0,0 +1 @@
 docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256 
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_nccl_transport.log
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_nccl_transport.log
--- a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_server_cmd.txt
+++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_server_cmd.txt
@ -0,0 +1 @@
 docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256 
--- a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html
+++ b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html
@ -400,7 +400,7 @@
      <div class="document-header__meta">
        <span>节点：174.1.51.5 + 174.1.51.7</span>
        <span>资源：16 × RTX PRO 6000 Blackwell</span>
-        <span>版本：2026-07-31 13:40:03 CST</span>
+        <span>版本：2026-08-01 02:45:00 CST</span>
      </div>
    </div>
  </header>
@ -415,7 +415,7 @@
      <article id="document-content">
        <h1>6000D 双机 DeepSeek-V4-Pro 推理优化计划</h1>
 <blockquote>
-<p>适用环境：<code>174.1.51.5 + 174.1.51.7</code>，每台 8 张 RTX PRO 6000 Blackwell Server Edition<br>当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例<br>当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离<br>计划版本：2026-07-31 16:46:05 CST</p>
+<p>适用环境：<code>174.1.51.5 + 174.1.51.7</code>，每台 8 张 RTX PRO 6000 Blackwell Server Edition<br>当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例<br>当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离<br>计划版本：2026-08-01 02:45:00 CST</p>
 </blockquote>
 <h2>当前执行状态与阶段档案</h2>
 <table>
@ -448,6 +448,14 @@
 <a href="./phase2_code.html">打开 Phase 2 代码详解</a>
 </td>
 </tr>
 <tr>
 <td>DeepSeek-V4-Pro / 双机 Pro6000D / SGLang RDMA 需求建模与并发拐点</td>
 <td>Phase 2.5 已完成；Scout 5/5、Confirm 6/6，C=16 进入平台，拟合上限 80.32 Gbit/s/rail</td>
 <td>
 <a href="./phase2_5_exp.html">打开 Phase 2.5 实验档案</a><br>
 <a href="./phase2_5_code.html">打开 Phase 2.5 代码详解</a>
 </td>
 </tr>
 </tbody></table>
 <p>
 <strong>阶段档案生成门禁：</strong>Phase 尚未产出正式阶段结果、完成汇总和汇报确认前，
@ -785,6 +793,24 @@ Phase 2 回答“哪个硬件/Host/通信资源在什么 Case 中升高，以及
 Phase 3 只捕获短 TP16 Timeline，回答具体 Kernel、Scheduler gap、Collective、
 Rank 同步及计算/通信重叠，不重复 Phase 2 的长时间轻量采样。
 </p>
 <h2>6.5 Phase 2.5：RDMA 需求建模与并发拐点</h2>
 <p>
 Phase 2 已证明当前代表负载最高约 83.5 Gbit/s/rail，但单个固定 Case 不能回答
 “提高并发后是否还能继续逼近 400G”。Phase 2.5 保持 TP16/EP2 和服务参数不变，
 先用 <code>64K → 1</code> 的 C=1/4/16/32/64 隔离 Prefill，拟合
 <code>rail_gbps ≈ input_tps × bytes_per_input_token_per_rail × 8</code>；
 再自动选择平台前、拐点和最大稳定并发，对 <code>64K → 1K</code> 重复两次确认。
 </p>
 <p>
 正式 Run <code>dsv4pro-phase2_5-20260801-130007</code> 已完成。64K→1 Scout 在 C=16
 达到约 2,984 input tok/s 与 79.90 Gbit/s/rail；C=32/64 均不再显著增长，拟合
 单 Rail 渐近上限为 80.32 Gbit/s。通信强度约为 3.332 MB/input-token/rail，
 要达到单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍。64K→1K
 两轮确认同样在 C=16 后进入平台，说明当前是模型计算/实现吞吐先饱和，而非 RDMA 链路先饱和。
 完整方法、逐点结果和证据路径见
 <a href="./phase2_5_exp.html">Phase 2.5 实验档案</a> 与
 <a href="./phase2_5_code.html">Phase 2.5 代码详解</a>。
 </p>
 <h2>7. Phase 3：时间线 Profiling（Nsight Systems 为主）</h2>
 <p>
 Nsight Systems、PyTorch Profiler 和 NVTX 是三件不同的东西：Nsight Systems
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/.gitignore
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/.gitignore
@ -0,0 +1,3 @@
 results/
 runtime/
 *.log
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/README.md
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/README.md
@ -0,0 +1,35 @@
 # DeepSeek-V4-Pro 双机 SGLang RDMA 需求建模
 Phase 2.5 在不改变 TP16/EP2 服务参数的前提下回答三个问题：
 1. 64K 输入下，哪个并发是 Token TPS 与 RDMA 带宽的拐点；
 2. `64K -> 1K` 真实请求最多产生多少双 Rail 流量；
 3. 当前部署会先达到 GPU/模型吞吐平台，还是先打满 400G rail。
 唯一入口：
 ```bash
 cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
 # 只展开命令，不启动服务
 DRY_RUN=1 RUN_ID=dsv4pro-phase2_5-dryrun-$(date +%Y%m%d-%H%M%S) \
  bash run_rdma_demand_modeling.sh all
 # 正式运行
 RUN_ID=dsv4pro-phase2_5-$(date +%Y%m%d-%H%M%S)
 tmux new-session -d -s dsv4pro-phase2_5 \
  "RUN_ID=${RUN_ID} bash run_rdma_demand_modeling.sh all \
   2>&1 | tee /data/hzy/${RUN_ID}.log"
 tmux attach -t dsv4pro-phase2_5
 ```
 脚本先跑 `64K -> 1` 的 C=1/4/16/32/64 探路点，再自动选择三个代表并发，
 对 `64K -> 1K` 重复两次。服务、benchmark 和 HCA 采集复用已经验证的
 Phase 1/2 实现；本目录只负责工作负载编排与带宽模型拟合。
 最终高价值结果：
 - `rdma_case_metrics.csv`：每个 Case 的 TPS、每 Rail/双 Rail 带宽和 bytes/token；
 - `rdma_demand_model.json`：拟合参数、并发拐点和理论需求；
 - `rdma_demand_report.md`：可直接用于阶段汇报的结论；
 - `recommendation.env`：Scout 自动选择的 Confirm 并发。
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/config.env
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/config.env
@ -0,0 +1,36 @@
 # DeepSeek-V4-Pro two-node SGLang RDMA demand modeling.
 EXPERIMENT="dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling"
 PHASE1_DIR="${PHASE1_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../dsv4pro_pro6000d_2node_sglang_tp16_quick_map" && pwd)}"
 PHASE2_DIR="${PHASE2_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution" && pwd)}"
 PHASE2_ENTRY="${PHASE2_ENTRY:-${PHASE2_DIR}/run_hardware_contention_attribution.sh}"
 # Workload policy. Scout isolates Prefill; confirm preserves the requested 64K -> 1K shape.
 ISL="${ISL:-65536}"
 SCOUT_OSL="${SCOUT_OSL:-1}"
 CONFIRM_OSL="${CONFIRM_OSL:-1024}"
 SCOUT_CONCURRENCIES="${SCOUT_CONCURRENCIES:-1 4 16 32 64}"
 DRY_RUN_CONFIRM_CONCURRENCIES="${DRY_RUN_CONFIRM_CONCURRENCIES:-16 32 64}"
 SCOUT_REPETITIONS="${SCOUT_REPETITIONS:-1}"
 CONFIRM_REPETITIONS="${CONFIRM_REPETITIONS:-2}"
 # A point is a plateau candidate when both bandwidth and Input TPS gains stay below this value.
 PLATEAU_GAIN_PCT="${PLATEAU_GAIN_PCT:-5}"
 TARGET_RAIL_GBPS="${TARGET_RAIL_GBPS:-400}"
 PRACTICAL_RAIL_GBPS="${PRACTICAL_RAIL_GBPS:-360}"
 # Reuse the verified Phase 2 collectors but skip its one-time communication microbenchmark.
 SAMPLE_INTERVAL_S="${SAMPLE_INTERVAL_S:-1}"
 IDLE_BASELINE_S="${IDLE_BASELINE_S:-5}"
 POST_RUN_COOLDOWN_S="${POST_RUN_COOLDOWN_S:-5}"
 CASE_COOLDOWN_S="${CASE_COOLDOWN_S:-3}"
 COLLECTOR_MAX_SECONDS="${COLLECTOR_MAX_SECONDS:-14400}"
 SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-7200}"
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
 RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/results}"
 RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
 DRY_RUN="${DRY_RUN:-0}"
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/rdma_demand_model.py
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/rdma_demand_model.py
@ -0,0 +1,563 @@
 #!/usr/bin/env python3
 """Fit model-serving RDMA demand from aligned benchmark and HCA counters."""
 from __future__ import annotations
 import argparse
 import csv
 import json
 import math
 import statistics
 from dataclasses import dataclass
 from datetime import datetime
 from pathlib import Path
 from typing import Any, Iterable
 def read_csv(path: Path) -> list[dict[str, str]]:
    if not path.is_file():
        return []
    with path.open(encoding="utf-8", newline="") as handle:
        return list(csv.DictReader(handle))
 def write_csv(path: Path, rows: list[dict[str, Any]]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    if not rows:
        path.write_text("", encoding="utf-8")
        return
    fields: list[str] = []
    for row in rows:
        for field in row:
            if field not in fields:
                fields.append(field)
    with path.open("w", encoding="utf-8", newline="") as handle:
        writer = csv.DictWriter(handle, fieldnames=fields)
        writer.writeheader()
        writer.writerows(rows)
 def number(value: Any) -> float | None:
    if value in (None, ""):
        return None
    try:
        parsed = float(value)
    except (TypeError, ValueError):
        return None
    return parsed if math.isfinite(parsed) else None
 def integer(value: Any) -> int:
    parsed = number(value)
    return int(parsed) if parsed is not None else 0
 def percentile(values: Iterable[float], fraction: float) -> float | None:
    ordered = sorted(values)
    if not ordered:
        return None
    position = (len(ordered) - 1) * fraction
    lower = math.floor(position)
    upper = math.ceil(position)
    if lower == upper:
        return ordered[lower]
    return ordered[lower] + (ordered[upper] - ordered[lower]) * (position - lower)
 def mean(values: Iterable[float | None]) -> float | None:
    present = [value for value in values if value is not None]
    return statistics.fmean(present) if present else None
 def fmt(value: Any, digits: int = 2) -> str:
    parsed = number(value)
    return "-" if parsed is None else f"{parsed:.{digits}f}"
 def parse_iso_ns(value: str) -> int:
    return int(datetime.fromisoformat(value).timestamp() * 1_000_000_000)
@dataclass(frozen=True)
 class CaseKey:
    case_id: str
    repetition: int
 def load_bench(result_dir: Path) -> dict[CaseKey, dict[str, str]]:
    rows = read_csv(result_dir / "bench_summary.csv")
    result: dict[CaseKey, dict[str, str]] = {}
    for row in rows:
        key = CaseKey(row.get("case_id", ""), integer(row.get("repetition")))
        if key.case_id:
            result[key] = row
    return result
 def load_windows(result_dir: Path) -> dict[CaseKey, dict[str, str]]:
    rows = read_csv(result_dir / "case_windows.csv")
    result: dict[CaseKey, dict[str, str]] = {}
    for row in rows:
        key = CaseKey(row.get("case_id", ""), integer(row.get("repetition")))
        if key.case_id:
            result[key] = row
    return result
 def load_case_rdma(
    result_dir: Path,
 ) -> dict[CaseKey, list[dict[str, str]]]:
    grouped: dict[CaseKey, list[dict[str, str]]] = {}
    for row in read_csv(result_dir / "case_rdma_summary.csv"):
        key = CaseKey(row.get("case_id", ""), integer(row.get("repetition")))
        grouped.setdefault(key, []).append(row)
    return grouped
 def interval_rates(
    rows: list[dict[str, str]], started_ns: int, ended_ns: int
 ) -> tuple[list[float], list[float]]:
    selected = []
    for row in rows:
        timestamp = integer(row.get("wall_time_ns"))
        if started_ns <= timestamp <= ended_ns:
            selected.append(row)
    selected.sort(key=lambda row: integer(row.get("wall_time_ns")))
    xmit_rates: list[float] = []
    rcv_rates: list[float] = []
    for previous, current in zip(selected, selected[1:]):
        delta_ns = integer(current.get("wall_time_ns")) - integer(
            previous.get("wall_time_ns")
        )
        if delta_ns <= 0:
            continue
        duration_s = delta_ns / 1e9
        for field, output in (
            ("port_xmit_data", xmit_rates),
            ("port_rcv_data", rcv_rates),
        ):
            first = number(previous.get(field))
            last = number(current.get(field))
            if first is None or last is None or last < first:
                continue
            # IB port data counters use 4-octet units.
            output.append((last - first) * 4 * 8 / duration_s / 1e9)
    return xmit_rates, rcv_rates
 def raw_rdma_by_node_hca(
    result_dir: Path,
 ) -> dict[tuple[str, str], list[dict[str, str]]]:
    grouped: dict[tuple[str, str], list[dict[str, str]]] = {}
    for node in ("head", "worker"):
        for row in read_csv(result_dir / node / "rdma.csv"):
            hca = row.get("hca", "")
            if hca:
                grouped.setdefault((node, hca), []).append(row)
    return grouped
 def load_gpu_util(result_dir: Path) -> dict[CaseKey, float]:
    grouped: dict[CaseKey, list[float]] = {}
    for row in read_csv(result_dir / "case_gpu_node_summary.csv"):
        key = CaseKey(row.get("case_id", ""), integer(row.get("repetition")))
        value = number(row.get("gpu_util_pct_mean"))
        if value is not None:
            grouped.setdefault(key, []).append(value)
    return {key: statistics.fmean(values) for key, values in grouped.items()}
 def aggregate_stage(result_dir: Path, stage_name: str) -> list[dict[str, Any]]:
    bench = load_bench(result_dir)
    windows = load_windows(result_dir)
    case_rdma = load_case_rdma(result_dir)
    raw_rdma = raw_rdma_by_node_hca(result_dir)
    gpu_util = load_gpu_util(result_dir)
    output: list[dict[str, Any]] = []
    for key, bench_row in sorted(
        bench.items(), key=lambda item: (integer(item[1].get("concurrency")), item[0].repetition)
    ):
        window = windows.get(key)
        rdma_rows = case_rdma.get(key, [])
        if not window or not rdma_rows:
            continue
        started_ns = integer(window.get("started_ns")) or parse_iso_ns(
            window.get("started_at", "")
        )
        ended_ns = integer(window.get("ended_ns")) or parse_iso_ns(
            window.get("ended_at", "")
        )
        rail_averages: list[float] = []
        rail_p95s: list[float] = []
        rail_maxima: list[float] = []
        node_averages: dict[str, float] = {}
        xmit_bytes: list[float] = []
        error_delta = 0.0
        for rdma_row in rdma_rows:
            node = rdma_row.get("node", "")
            hca = rdma_row.get("hca", "")
            average = number(rdma_row.get("xmit_gbps"))
            total_bytes = number(rdma_row.get("xmit_bytes"))
            if average is not None:
                rail_averages.append(average)
                node_averages[node] = node_averages.get(node, 0.0) + average
            if total_bytes is not None:
                xmit_bytes.append(total_bytes)
            rates, _ = interval_rates(raw_rdma.get((node, hca), []), started_ns, ended_ns)
            p95 = percentile(rates, 0.95)
            if p95 is not None:
                rail_p95s.append(p95)
            if rates:
                rail_maxima.append(max(rates))
            for field in (
                "port_xmit_wait_delta",
                "port_xmit_discards_delta",
                "port_rcv_errors_delta",
                "req_transport_retries_exceeded_delta",
                "req_rnr_retries_exceeded_delta",
                "roce_adp_retrans_delta",
                "roce_adp_retrans_to_delta",
            ):
                error_delta += number(rdma_row.get(field)) or 0.0
        total_input_tokens = number(bench_row.get("total_input_tokens"))
        total_output_tokens = number(bench_row.get("total_output_tokens"))
        total_tokens = (
            total_input_tokens + total_output_tokens
            if total_input_tokens is not None and total_output_tokens is not None
            else None
        )
        mean_xmit_bytes = mean(xmit_bytes)
        bytes_per_input = (
            mean_xmit_bytes / total_input_tokens
            if mean_xmit_bytes is not None and total_input_tokens
            else None
        )
        bytes_per_total = (
            mean_xmit_bytes / total_tokens
            if mean_xmit_bytes is not None and total_tokens
            else None
        )
        min_rail = min(rail_averages) if rail_averages else None
        max_rail = max(rail_averages) if rail_averages else None
        rail_imbalance = (
            (max_rail - min_rail) / statistics.fmean(rail_averages) * 100
            if min_rail is not None and max_rail is not None and rail_averages
            else None
        )
        output.append(
            {
                "stage": stage_name,
                "case_id": key.case_id,
                "repetition": key.repetition,
                "status": bench_row.get("status", ""),
                "isl": integer(bench_row.get("isl")),
                "osl": integer(bench_row.get("osl")),
                "concurrency": integer(bench_row.get("concurrency")),
                "completed": integer(bench_row.get("completed")),
                "duration_s": number(bench_row.get("duration_s")),
                "input_tps": number(bench_row.get("input_token_throughput")),
                "output_tps": number(bench_row.get("output_token_throughput")),
                "total_tps": number(bench_row.get("total_token_throughput")),
                "ttft_p95_ms": number(bench_row.get("ttft_p95_ms")),
                "tpot_p95_ms": number(bench_row.get("tpot_p95_ms")),
                "gpu_util_mean_pct": gpu_util.get(key),
                "rail_xmit_avg_gbps": mean(rail_averages),
                "rail_xmit_p95_gbps": mean(rail_p95s),
                "rail_xmit_max_gbps": max(rail_maxima) if rail_maxima else None,
                "dual_rail_xmit_avg_gbps": mean(node_averages.values()),
                "rail_imbalance_pct": rail_imbalance,
                "xmit_bytes_per_input_token_per_rail": bytes_per_input,
                "xmit_bytes_per_total_token_per_rail": bytes_per_total,
                "rdma_error_delta": error_delta,
                "measurement_started_at": window.get("started_at", ""),
                "measurement_ended_at": window.get("ended_at", ""),
                "source_result_dir": str(result_dir),
            }
        )
    return output
 def average_by_concurrency(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
    grouped: dict[int, list[dict[str, Any]]] = {}
    for row in rows:
        if row.get("status") == "COMPLETED":
            grouped.setdefault(integer(row.get("concurrency")), []).append(row)
    output = []
    metrics = (
        "input_tps",
        "output_tps",
        "total_tps",
        "ttft_p95_ms",
        "tpot_p95_ms",
        "gpu_util_mean_pct",
        "rail_xmit_avg_gbps",
        "rail_xmit_p95_gbps",
        "rail_xmit_max_gbps",
        "dual_rail_xmit_avg_gbps",
        "rail_imbalance_pct",
        "xmit_bytes_per_input_token_per_rail",
        "xmit_bytes_per_total_token_per_rail",
        "rdma_error_delta",
    )
    for concurrency, group in sorted(grouped.items()):
        item: dict[str, Any] = {
            "stage": group[0].get("stage"),
            "concurrency": concurrency,
            "repetitions": len(group),
        }
        for metric in metrics:
            item[metric] = mean(number(row.get(metric)) for row in group)
        output.append(item)
    return output
 def fit_hyperbola(rows: list[dict[str, Any]], field: str) -> dict[str, float | None]:
    points = [
        (float(integer(row.get("concurrency"))), number(row.get(field)))
        for row in rows
        if integer(row.get("concurrency")) > 0 and number(row.get(field)) is not None
    ]
    if len(points) < 2:
        return {"asymptote": None, "half_saturation_c": None, "rmse": None}
    best: tuple[float, float, float] | None = None
    for exponent_step in range(-20, 121):
        k = 10 ** (exponent_step / 20)
        factors = [concurrency / (k + concurrency) for concurrency, _ in points]
        denominator = sum(factor * factor for factor in factors)
        if denominator == 0:
            continue
        asymptote = sum(
            factor * value for factor, (_, value) in zip(factors, points)
        ) / denominator
        errors = [
            value - asymptote * factor
            for factor, (_, value) in zip(factors, points)
        ]
        rmse = math.sqrt(sum(error * error for error in errors) / len(errors))
        if best is None or rmse < best[0]:
            best = (rmse, asymptote, k)
    assert best is not None
    return {
        "asymptote": best[1],
        "half_saturation_c": best[2],
        "rmse": best[0],
    }
 def linear_slope(rows: list[dict[str, Any]]) -> float | None:
    points = [
        (number(row.get("input_tps")), number(row.get("rail_xmit_avg_gbps")))
        for row in rows
    ]
    present = [(x, y) for x, y in points if x is not None and y is not None]
    denominator = sum(x * x for x, _ in present)
    return sum(x * y for x, y in present) / denominator if denominator else None
 def select_concurrencies(
    scout: list[dict[str, Any]], plateau_gain_pct: float
 ) -> tuple[list[int], int | None, int | None]:
    ordered = sorted(scout, key=lambda row: integer(row.get("concurrency")))
    if not ordered:
        return [], None, None
    plateau_c = None
    for previous, current in zip(ordered, ordered[1:]):
        previous_bw = number(previous.get("rail_xmit_avg_gbps")) or 0
        current_bw = number(current.get("rail_xmit_avg_gbps")) or 0
        previous_tps = number(previous.get("input_tps")) or 0
        current_tps = number(current.get("input_tps")) or 0
        bw_gain = (current_bw / previous_bw - 1) * 100 if previous_bw else math.inf
        tps_gain = (current_tps / previous_tps - 1) * 100 if previous_tps else math.inf
        if bw_gain < plateau_gain_pct and tps_gain < plateau_gain_pct:
            plateau_c = integer(current.get("concurrency"))
            break
    best = max(
        ordered,
        key=lambda row: number(row.get("rail_xmit_avg_gbps")) or -1,
    )
    best_c = integer(best.get("concurrency"))
    cs = [integer(row.get("concurrency")) for row in ordered]
    anchor = plateau_c or best_c
    anchor_index = cs.index(anchor)
    candidates = {anchor, cs[max(0, anchor_index - 1)], cs[-1]}
    if len(candidates) < 3:
        candidates.update(cs[-3:])
    return sorted(candidates), plateau_c, best_c
 def build_model(
    scout_rows: list[dict[str, Any]],
    confirm_rows: list[dict[str, Any]],
    plateau_gain_pct: float,
    target_rail_gbps: float,
    practical_rail_gbps: float,
 ) -> dict[str, Any]:
    scout = average_by_concurrency(scout_rows)
    confirm = average_by_concurrency(confirm_rows)
    selected, plateau_c, best_scout_c = select_concurrencies(
        scout, plateau_gain_pct
    )
    slope = linear_slope(scout)
    bytes_per_token = slope * 1e9 / 8 if slope is not None else None
    required_target_tps = target_rail_gbps / slope if slope else None
    required_practical_tps = practical_rail_gbps / slope if slope else None
    required_dual_400_tps = (target_rail_gbps / 2) / slope if slope else None
    bandwidth_fit = fit_hyperbola(scout, "rail_xmit_avg_gbps")
    input_tps_fit = fit_hyperbola(scout, "input_tps")
    max_confirm = (
        max(confirm, key=lambda row: number(row.get("rail_xmit_avg_gbps")) or -1)
        if confirm
        else None
    )
    asymptote = number(bandwidth_fit.get("asymptote"))
    if asymptote is None:
        verdict = "INSUFFICIENT_DATA"
    elif asymptote < practical_rail_gbps:
        verdict = "COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION"
    else:
        verdict = "RDMA_SATURATION_POSSIBLE_WITH_MODEL_LOAD"
    return {
        "generated_at": datetime.now().astimezone().isoformat(),
        "formula": "rail_gbps ~= input_tps * bytes_per_input_token_per_rail * 8 / 1e9",
        "plateau_gain_pct": plateau_gain_pct,
        "target_rail_gbps": target_rail_gbps,
        "practical_rail_gbps": practical_rail_gbps,
        "scout": scout,
        "confirm": confirm,
        "selected_confirm_concurrencies": selected,
        "observed_plateau_concurrency": plateau_c,
        "best_scout_bandwidth_concurrency": best_scout_c,
        "linear_gbps_per_input_tps": slope,
        "modeled_bytes_per_input_token_per_rail": bytes_per_token,
        "required_input_tps_for_400g_per_rail": required_target_tps,
        "required_input_tps_for_360g_per_rail": required_practical_tps,
        "required_input_tps_for_400g_dual_rail_aggregate": required_dual_400_tps,
        "bandwidth_saturation_fit": bandwidth_fit,
        "input_tps_saturation_fit": input_tps_fit,
        "best_confirm_point": max_confirm,
        "verdict": verdict,
    }
 def write_recommendation(output_dir: Path, model: dict[str, Any]) -> None:
    values = model.get("selected_confirm_concurrencies", [])
    joined = " ".join(str(value) for value in values)
    (output_dir / "recommendation.env").write_text(
        f"CONFIRM_CONCURRENCIES='{joined}'\n", encoding="utf-8"
    )
 def write_report(output_dir: Path, model: dict[str, Any]) -> None:
    scout = model.get("scout", [])
    confirm = model.get("confirm", [])
    lines = [
        "# Phase 2.5 RDMA Demand Model",
        "",
        f"- Generated: `{model['generated_at']}`",
        f"- Verdict: `{model['verdict']}`",
        f"- Model: `rail_gbps ~= input_tps x {fmt(model.get('modeled_bytes_per_input_token_per_rail') / 1e6 if model.get('modeled_bytes_per_input_token_per_rail') else None, 3)} MB/token/rail x 8`",
        f"- Required Input TPS for 400G per rail: `{fmt(model.get('required_input_tps_for_400g_per_rail'), 0)}`",
        f"- Required Input TPS for 360G per rail: `{fmt(model.get('required_input_tps_for_360g_per_rail'), 0)}`",
        f"- Required Input TPS for 400G dual-rail aggregate: `{fmt(model.get('required_input_tps_for_400g_dual_rail_aggregate'), 0)}`",
        f"- Fitted rail bandwidth ceiling: `{fmt(model.get('bandwidth_saturation_fit', {}).get('asymptote'))} Gbit/s`",
        f"- Observed plateau concurrency: `{model.get('observed_plateau_concurrency')}`",
        "",
        "## Scout: 64K -> 1",
        "",
        "| C | Input TPS | Rail avg | Rail P95 | Dual-rail avg | MB/input-token/rail | GPU util |",
        "|---:|---:|---:|---:|---:|---:|---:|",
    ]
    for row in scout:
        bytes_per_token = number(row.get("xmit_bytes_per_input_token_per_rail"))
        lines.append(
            f"| {row['concurrency']} | {fmt(row.get('input_tps'))} | "
            f"{fmt(row.get('rail_xmit_avg_gbps'))} | {fmt(row.get('rail_xmit_p95_gbps'))} | "
            f"{fmt(row.get('dual_rail_xmit_avg_gbps'))} | "
            f"{fmt(bytes_per_token / 1e6 if bytes_per_token else None, 3)} | "
            f"{fmt(row.get('gpu_util_mean_pct'))} |"
        )
    lines.extend(
        [
            "",
            "## Confirm: 64K -> 1K",
            "",
            "| C | Reps | Input TPS | Output TPS | Rail avg | Rail P95 | Dual-rail avg | TTFT P95 ms | TPOT P95 ms |",
            "|---:|---:|---:|---:|---:|---:|---:|---:|---:|",
        ]
    )
    for row in confirm:
        lines.append(
            f"| {row['concurrency']} | {row['repetitions']} | {fmt(row.get('input_tps'))} | "
            f"{fmt(row.get('output_tps'))} | {fmt(row.get('rail_xmit_avg_gbps'))} | "
            f"{fmt(row.get('rail_xmit_p95_gbps'))} | {fmt(row.get('dual_rail_xmit_avg_gbps'))} | "
            f"{fmt(row.get('ttft_p95_ms'))} | {fmt(row.get('tpot_p95_ms'))} |"
        )
    lines.extend(
        [
            "",
            "## Evidence",
            "",
            "- Per-case metrics: `rdma_case_metrics.csv`",
            "- Machine-readable model: `rdma_demand_model.json`",
            "- Scout HCA source: `scout/{head,worker}/rdma.csv`",
            "- Confirm HCA source: `confirm/{head,worker}/rdma.csv`",
            "- Exact windows: `scout/case_windows.csv`, `confirm/case_windows.csv`",
            "- Benchmark summaries: `scout/bench_summary.csv`, `confirm/bench_summary.csv`",
        ]
    )
    (output_dir / "rdma_demand_report.md").write_text(
        "\n".join(lines) + "\n", encoding="utf-8"
    )
 def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser()
    parser.add_argument("mode", choices=("scout", "final"))
    parser.add_argument("--scout-dir", required=True, type=Path)
    parser.add_argument("--confirm-dir", type=Path)
    parser.add_argument("--output-dir", required=True, type=Path)
    parser.add_argument("--plateau-gain-pct", type=float, default=5.0)
    parser.add_argument("--target-rail-gbps", type=float, default=400.0)
    parser.add_argument("--practical-rail-gbps", type=float, default=360.0)
    return parser.parse_args()
 def main() -> None:
    args = parse_args()
    args.output_dir.mkdir(parents=True, exist_ok=True)
    scout_rows = aggregate_stage(args.scout_dir, "scout")
    if not scout_rows:
        raise SystemExit(f"no aligned scout rows found in {args.scout_dir}")
    confirm_rows: list[dict[str, Any]] = []
    if args.mode == "final":
        if args.confirm_dir is None:
            raise SystemExit("--confirm-dir is required in final mode")
        confirm_rows = aggregate_stage(args.confirm_dir, "confirm")
        if not confirm_rows:
            raise SystemExit(f"no aligned confirm rows found in {args.confirm_dir}")
    model = build_model(
        scout_rows,
        confirm_rows,
        args.plateau_gain_pct,
        args.target_rail_gbps,
        args.practical_rail_gbps,
    )
    write_csv(args.output_dir / "rdma_case_metrics.csv", scout_rows + confirm_rows)
    (args.output_dir / "rdma_demand_model.json").write_text(
        json.dumps(model, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
    )
    write_recommendation(args.output_dir, model)
    write_report(args.output_dir, model)
    print(json.dumps(model, ensure_ascii=False, indent=2))
 if __name__ == "__main__":
    main()
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/run_rdma_demand_modeling.sh
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/run_rdma_demand_modeling.sh
@ -0,0 +1,264 @@
 #!/usr/bin/env bash
 # Measure how model token throughput translates into two-node RDMA bandwidth.
 set -Eeuo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck source=/dev/null
 source "${SCRIPT_DIR}/config.env"
 ACTION="${1:-all}"
 RUN_ID="${RUN_ID:-dsv4pro-phase2_5-$(date +%Y%m%d-%H%M%S)}"
 RESULT_DIR="${RESULT_BASE}/${RUN_ID}"
 COMMAND_DIR="${RESULT_DIR}/commands"
 SCENARIO_DIR="${RESULT_DIR}/scenarios"
 ANALYZER="${SCRIPT_DIR}/rdma_demand_model.py"
 RUN_LOG_ACTIVE=0
 log() {
  printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"
 }
 print_command() {
  printf '%q ' "$@"
  printf '\n'
 }
 enable_result_logging() {
  [[ "${DRY_RUN}" == "1" || "${RUN_LOG_ACTIVE}" == "1" ]] && return 0
  mkdir -p "${RESULT_DIR}"
  exec > >(tee -a "${RESULT_DIR}/run.log") 2>&1
  RUN_LOG_ACTIVE=1
 }
 normalize_concurrencies() {
  tr ', ' '\n\n' <<< "$1" |
    sed '/^[[:space:]]*$/d' |
    awk '!seen[$0]++' |
    sort -n |
    paste -sd' ' -
 }
 validate_config() {
  [[ -x "${PHASE2_ENTRY}" ]] || {
    log "ERROR: Phase 2 entry is missing: ${PHASE2_ENTRY}"
    return 1
  }
  [[ -f "${ANALYZER}" ]] || {
    log "ERROR: analyzer is missing: ${ANALYZER}"
    return 1
  }
  local name value
  for name in ISL SCOUT_OSL CONFIRM_OSL SCOUT_REPETITIONS CONFIRM_REPETITIONS \
    SAMPLE_INTERVAL_S COLLECTOR_MAX_SECONDS SCENARIO_TIMEOUT_S; do
    value="${!name}"
    [[ "${value}" =~ ^[1-9][0-9]*$ ]] || {
      log "ERROR: ${name} must be a positive integer: ${value}"
      return 1
    }
  done
  [[ "${PLATEAU_GAIN_PCT}" =~ ^[0-9]+([.][0-9]+)?$ ]] || {
    log "ERROR: PLATEAU_GAIN_PCT must be numeric"
    return 1
  }
  local concurrency
  for concurrency in $(normalize_concurrencies "${SCOUT_CONCURRENCIES}"); do
    [[ "${concurrency}" =~ ^[1-9][0-9]*$ ]] || {
      log "ERROR: invalid scout concurrency: ${concurrency}"
      return 1
    }
  done
 }
 write_scenario_file() {
  local stage="$1"
  local osl="$2"
  local concurrencies="$3"
  local output="$4"
  mkdir -p "$(dirname "${output}")"
  printf '# case_id\tstage\tisl\tosl\tconcurrency\tprompt_multiplier\tmin_prompts\twarmup_requests\tnote\n' > "${output}"
  local concurrency case_id
  for concurrency in $(normalize_concurrencies "${concurrencies}"); do
    case_id="rdma_${stage}_i${ISL}_o${osl}_c${concurrency}"
    printf '%s\t%s\t%s\t%s\t%s\t1\t1\t0\t%s\n' \
      "${case_id}" "rdma_${stage}" "${ISL}" "${osl}" "${concurrency}" \
      "Phase 2.5 ${stage}: correlate model token rate with dual-rail RDMA" \
      >> "${output}"
  done
 }
 case_ids_for() {
  local stage="$1"
  local osl="$2"
  local concurrencies="$3"
  local result=() concurrency
  for concurrency in $(normalize_concurrencies "${concurrencies}"); do
    result+=("rdma_${stage}_i${ISL}_o${osl}_c${concurrency}")
  done
  local joined
  joined="$(IFS=,; printf '%s' "${result[*]}")"
  printf '%s\n' "${joined}"
 }
 run_phase2_stage() {
  local stage="$1"
  local scenario_file="$2"
  local case_ids="$3"
  local repetitions="$4"
  local stage_result="${RESULT_DIR}/${stage}"
  local command_file="${COMMAND_DIR}/${stage}.cmd.txt"
  local -a command=(
    env
    "DRY_RUN=${DRY_RUN}"
    "RUN_ID=${stage}"
    "RESULT_BASE=${RESULT_DIR}"
    "SCENARIO_FILE=${scenario_file}"
    "FIXED_CASE_IDS=${case_ids}"
    "RUN_MIXED_CASE=0"
    "RUN_COMMUNICATION_BASELINE=0"
    "NUM_REPETITIONS=${repetitions}"
    "SAMPLE_INTERVAL_S=${SAMPLE_INTERVAL_S}"
    "IDLE_BASELINE_S=${IDLE_BASELINE_S}"
    "POST_RUN_COOLDOWN_S=${POST_RUN_COOLDOWN_S}"
    "CASE_COOLDOWN_S=${CASE_COOLDOWN_S}"
    "COLLECTOR_MAX_SECONDS=${COLLECTOR_MAX_SECONDS}"
    "SCENARIO_TIMEOUT_S=${SCENARIO_TIMEOUT_S}"
    bash "${PHASE2_ENTRY}" all
  )
  mkdir -p "${COMMAND_DIR}"
  print_command "${command[@]}" > "${command_file}"
  log "START stage=${stage} result=${stage_result}"
  "${command[@]}"
  log "DONE  stage=${stage} result=${stage_result}"
 }
 run_scout() {
  local scenario_file="${SCENARIO_DIR}/scout.tsv"
  local concurrencies case_ids
  concurrencies="$(normalize_concurrencies "${SCOUT_CONCURRENCIES}")"
  write_scenario_file scout "${SCOUT_OSL}" "${concurrencies}" "${scenario_file}"
  case_ids="$(case_ids_for scout "${SCOUT_OSL}" "${concurrencies}")"
  run_phase2_stage scout "${scenario_file}" "${case_ids}" "${SCOUT_REPETITIONS}"
  if [[ "${DRY_RUN}" != "1" ]]; then
    python3 "${ANALYZER}" scout \
      --scout-dir "${RESULT_DIR}/scout" \
      --output-dir "${RESULT_DIR}" \
      --plateau-gain-pct "${PLATEAU_GAIN_PCT}" \
      --target-rail-gbps "${TARGET_RAIL_GBPS}" \
      --practical-rail-gbps "${PRACTICAL_RAIL_GBPS}"
  fi
 }
 confirm_concurrencies() {
  if [[ "${DRY_RUN}" == "1" ]]; then
    normalize_concurrencies "${DRY_RUN_CONFIRM_CONCURRENCIES}"
    return 0
  fi
  # shellcheck source=/dev/null
  source "${RESULT_DIR}/recommendation.env"
  normalize_concurrencies "${CONFIRM_CONCURRENCIES}"
 }
 run_confirm() {
  local scenario_file="${SCENARIO_DIR}/confirm.tsv"
  local concurrencies case_ids
  concurrencies="$(confirm_concurrencies)"
  write_scenario_file confirm "${CONFIRM_OSL}" "${concurrencies}" "${scenario_file}"
  case_ids="$(case_ids_for confirm "${CONFIRM_OSL}" "${concurrencies}")"
  run_phase2_stage confirm "${scenario_file}" "${case_ids}" "${CONFIRM_REPETITIONS}"
  if [[ "${DRY_RUN}" != "1" ]]; then
    python3 "${ANALYZER}" final \
      --scout-dir "${RESULT_DIR}/scout" \
      --confirm-dir "${RESULT_DIR}/confirm" \
      --output-dir "${RESULT_DIR}" \
      --plateau-gain-pct "${PLATEAU_GAIN_PCT}" \
      --target-rail-gbps "${TARGET_RAIL_GBPS}" \
      --practical-rail-gbps "${PRACTICAL_RAIL_GBPS}"
  fi
 }
 write_manifest() {
  local commit dirty
  commit="$(git -C "${REPO_ROOT}" rev-parse HEAD 2>/dev/null || printf unknown)"
  dirty=0
  [[ -n "$(git -C "${REPO_ROOT}" status --porcelain 2>/dev/null || true)" ]] && dirty=1
  mkdir -p "${RESULT_DIR}"
  cat > "${RESULT_DIR}/run_manifest.txt" <<EOF
 run_id=${RUN_ID}
 experiment=${EXPERIMENT}
 created_at=$(date '+%Y-%m-%dT%H:%M:%S%z')
 git_commit=${commit}
 git_dirty=${dirty}
 isl=${ISL}
 scout_osl=${SCOUT_OSL}
 confirm_osl=${CONFIRM_OSL}
 scout_concurrencies=$(normalize_concurrencies "${SCOUT_CONCURRENCIES}")
 scout_repetitions=${SCOUT_REPETITIONS}
 confirm_repetitions=${CONFIRM_REPETITIONS}
 target_rail_gbps=${TARGET_RAIL_GBPS}
 practical_rail_gbps=${PRACTICAL_RAIL_GBPS}
 EOF
 }
 stop_service() {
  env \
    "RUN_ID=${RUN_ID}-stop" \
    "RESULT_BASE=${RESULT_DIR}" \
    "RUN_COMMUNICATION_BASELINE=0" \
    bash "${PHASE2_ENTRY}" stop || true
 }
 run_all() {
  validate_config
  enable_result_logging
  write_manifest
  trap 'stop_service' INT TERM
  run_scout
  run_confirm
  trap - INT TERM
  log "Phase 2.5 complete: ${RESULT_DIR}"
 }
 main() {
  case "${ACTION}" in
    audit)
      validate_config
      log "configuration valid"
      ;;
    scout)
      validate_config
      enable_result_logging
      write_manifest
      run_scout
      ;;
    confirm)
      validate_config
      enable_result_logging
      run_confirm
      ;;
    summarize)
      python3 "${ANALYZER}" final \
        --scout-dir "${RESULT_DIR}/scout" \
        --confirm-dir "${RESULT_DIR}/confirm" \
        --output-dir "${RESULT_DIR}" \
        --plateau-gain-pct "${PLATEAU_GAIN_PCT}" \
        --target-rail-gbps "${TARGET_RAIL_GBPS}" \
        --practical-rail-gbps "${PRACTICAL_RAIL_GBPS}"
      ;;
    stop)
      stop_service
      ;;
    all)
      run_all
      ;;
    *)
      printf 'Usage: %s {audit|scout|confirm|summarize|stop|all}\n' "$0" >&2
      return 2
      ;;
  esac
 }
 main "$@"
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/test_rdma_demand_model.py
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/test_rdma_demand_model.py
@ -0,0 +1,95 @@
 #!/usr/bin/env python3
 import importlib.util
 import sys
 import tempfile
 import unittest
 from pathlib import Path
 MODULE_PATH = Path(__file__).with_name("rdma_demand_model.py")
 SPEC = importlib.util.spec_from_file_location("rdma_demand_model", MODULE_PATH)
 assert SPEC and SPEC.loader
 MODEL = importlib.util.module_from_spec(SPEC)
 sys.modules[SPEC.name] = MODEL
 SPEC.loader.exec_module(MODEL)
 class RdmaDemandModelTest(unittest.TestCase):
    def test_interval_rate_uses_four_octet_counter_units(self):
        rows = [
            {
                "wall_time_ns": "1000000000",
                "port_xmit_data": "100",
                "port_rcv_data": "200",
            },
            {
                "wall_time_ns": "2000000000",
                "port_xmit_data": "100000100",
                "port_rcv_data": "100000200",
            },
        ]
        xmit, rcv = MODEL.interval_rates(rows, 0, 3_000_000_000)
        self.assertAlmostEqual(xmit[0], 3.2)
        self.assertAlmostEqual(rcv[0], 3.2)
    def test_linear_slope_recovers_bytes_per_token(self):
        rows = [
            {"input_tps": 1000, "rail_xmit_avg_gbps": 26.8},
            {"input_tps": 2000, "rail_xmit_avg_gbps": 53.6},
            {"input_tps": 3000, "rail_xmit_avg_gbps": 80.4},
        ]
        slope = MODEL.linear_slope(rows)
        self.assertAlmostEqual(slope, 0.0268)
        self.assertAlmostEqual(slope * 1e9 / 8, 3_350_000)
    def test_plateau_selects_previous_plateau_and_max(self):
        rows = [
            {"concurrency": 1, "input_tps": 1000, "rail_xmit_avg_gbps": 27},
            {"concurrency": 4, "input_tps": 2400, "rail_xmit_avg_gbps": 64},
            {"concurrency": 16, "input_tps": 3100, "rail_xmit_avg_gbps": 83},
            {"concurrency": 32, "input_tps": 3150, "rail_xmit_avg_gbps": 84},
            {"concurrency": 64, "input_tps": 3160, "rail_xmit_avg_gbps": 84.2},
        ]
        selected, plateau, best = MODEL.select_concurrencies(rows, 5.0)
        self.assertEqual(plateau, 32)
        self.assertEqual(best, 64)
        self.assertEqual(selected, [16, 32, 64])
    def test_report_and_recommendation_are_created(self):
        rows = [
            {
                "stage": "scout",
                "concurrency": 16,
                "repetitions": 1,
                "input_tps": 3000,
                "rail_xmit_avg_gbps": 80,
                "rail_xmit_p95_gbps": 90,
                "dual_rail_xmit_avg_gbps": 160,
                "xmit_bytes_per_input_token_per_rail": 3_333_333,
                "gpu_util_mean_pct": 99,
            }
        ]
        model = {
            "generated_at": "2026-08-01T00:00:00+08:00",
            "verdict": "TEST",
            "modeled_bytes_per_input_token_per_rail": 3_333_333,
            "required_input_tps_for_400g_per_rail": 15000,
            "required_input_tps_for_360g_per_rail": 13500,
            "required_input_tps_for_400g_dual_rail_aggregate": 7500,
            "bandwidth_saturation_fit": {"asymptote": 100},
            "observed_plateau_concurrency": 32,
            "scout": rows,
            "confirm": [],
            "selected_confirm_concurrencies": [16, 32, 64],
        }
        with tempfile.TemporaryDirectory() as temporary:
            output = Path(temporary)
            MODEL.write_recommendation(output, model)
            MODEL.write_report(output, model)
            self.assertIn("16 32 64", (output / "recommendation.env").read_text())
            self.assertIn("15000", (output / "rdma_demand_report.md").read_text())
 if __name__ == "__main__":
    unittest.main()
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/.gitignore
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/.gitignore
@ -0,0 +1,3 @@
 results/
 runtime/
 *.log
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh
@ -278,16 +278,22 @@ start_service() {
  return 1
 }
-stop_service_node() {
+request_stop_node() {
  local node="$1"
  local container_name="$2"
  local role="$3"
  if [[ "${DRY_RUN}" == "1" ]]; then
    log "[DRY] ${node}: docker stop --time ${NSYS_EXPORT_TIMEOUT_S} ${container_name}"
    return 0
  fi
  run_on_node "${node}" \
    "docker stop --time '${NSYS_EXPORT_TIMEOUT_S}' '${container_name}' >/dev/null 2>&1 || true"
 }
 collect_service_node() {
  local node="$1"
  local container_name="$2"
  local role="$3"
  [[ "${DRY_RUN}" == "1" ]] && return 0
  run_on_node "${node}" "docker logs '${container_name}' 2>&1" \
    > "${SERVICE_DIR}/${role}_server.log" 2>&1 || true
  run_on_node "${node}" "docker inspect '${container_name}' 2>/dev/null" \
@ -305,12 +311,24 @@ fetch_worker_profiles() {
 }
 stop_service() {
-  stop_service_node "${HEAD_NODE}" "${HEAD_CONTAINER}" head
+  if [[ "${DRY_RUN}" == "1" ]]; then
-  stop_service_node "${WORKER_NODE}" "${WORKER_CONTAINER}" worker
+    request_stop_node "${HEAD_NODE}" "${HEAD_CONTAINER}"
    request_stop_node "${WORKER_NODE}" "${WORKER_CONTAINER}"
    return 0
  fi
  request_stop_node "${HEAD_NODE}" "${HEAD_CONTAINER}" &
  local head_stop_pid=$!
  request_stop_node "${WORKER_NODE}" "${WORKER_CONTAINER}" &
  local worker_stop_pid=$!
  wait "${head_stop_pid}"
  wait "${worker_stop_pid}"
  collect_service_node "${HEAD_NODE}" "${HEAD_CONTAINER}" head
  collect_service_node "${WORKER_NODE}" "${WORKER_CONTAINER}" worker
  fetch_worker_profiles
 }
 validate_torch_traces() {
  [[ "${DRY_RUN}" == "1" ]] && return 0
  local count
  count="$(find "${RESULT_DIR}/profiles" -type f -name '*.trace.json.gz' | wc -l)"
  if (( count < TP_SIZE )); then
@ -321,10 +339,12 @@ validate_torch_traces() {
 }
 validate_nsys_reports() {
-  local count
+  [[ "${DRY_RUN}" == "1" ]] && return 0
  local count expected
  count="$(find "${RESULT_DIR}/profiles" -type f -name '*.nsys-rep' | wc -l)"
-  if (( count < NNODES )); then
+  expected=$(( NNODES * CAPTURE_RANGES ))
-    log "ERROR: expected at least ${NNODES} Nsight reports, found ${count}"
+  if (( count < expected )); then
    log "ERROR: expected at least ${expected} Nsight reports, found ${count}"
    return 1
  fi
  log "Nsight report validation passed: ${count} reports"
@ -366,9 +386,10 @@ start_profile() {
 wait_for_profile_stop() {
  local before="$1"
  local expected=$(( TP_SIZE / NNODES ))
  local waited=0
  while (( waited < PROFILE_STOP_WAIT_S )); do
-    if (( $(profile_done_count) > before )); then
+    if (( $(profile_done_count) >= before + expected )); then
      return 0
    fi
    sleep 2
@ -378,6 +399,26 @@ wait_for_profile_stop() {
  return 1
 }
 server_decode_count() {
  run_on_node "${HEAD_NODE}" "docker logs '${HEAD_CONTAINER}' 2>&1" \
    | grep -c 'Decode batch' || true
 }
 wait_for_decode_activity() {
  local before="$1"
  local bench_pid="$2"
  local waited=0
  while (( waited < 180 )); do
    if (( $(server_decode_count) > before )); then
      return 0
    fi
    kill -0 "${bench_pid}" 2>/dev/null || return 1
    sleep 1
    ((waited+=1))
  done
  return 1
 }
 prepare_bench_command() {
  local output_file="$1"
  local isl="$2"
@ -455,6 +496,47 @@ run_profiled_case() {
  [[ "${DRY_RUN}" == "1" ]] || wait_for_profile_stop "${before}"
 }
 run_profiled_decode_case() {
  local case_id="$1"
  local steps="$2"
  local isl="$3"
  local osl="$4"
  local concurrency="$5"
  local prompts="$6"
  local seed="$7"
  local case_path="${CASE_DIR}/${case_id}"
  local decode_before=0
  local profile_before=0
  flush_cache
  mkdir -p "${case_path}"
  prepare_bench_command "${case_path}/bench.jsonl" \
    "${isl}" "${osl}" "${concurrency}" "${prompts}" "${seed}"
  print_command timeout --signal=TERM --kill-after=30s \
    "${PROFILE_TIMEOUT_S}s" "${BENCH_CMD[@]}" > "${case_path}/bench_cmd.txt"
  if [[ "${DRY_RUN}" == "1" ]]; then
    log "[DRY] start decode workload, wait for Decode batch, then start Nsight range"
    printf '[DRY] '
    cat "${case_path}/bench_cmd.txt"
    start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}"
    return 0
  fi
  decode_before="$(server_decode_count)"
  timeout --signal=TERM --kill-after=30s "${PROFILE_TIMEOUT_S}s" \
    "${BENCH_CMD[@]}" > "${case_path}/bench.log" 2>&1 &
  local bench_pid=$!
  wait_for_bench_main "${case_path}/bench.log" "${bench_pid}" \
    && wait_for_decode_activity "${decode_before}" "${bench_pid}" || {
      kill "${bench_pid}" 2>/dev/null || true
      wait "${bench_pid}" 2>/dev/null || true
      log "ERROR: ${case_id} did not reach active Decode"
      return 1
    }
  profile_before="$(profile_done_count)"
  start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}"
  wait_for_profile_stop "${profile_before}"
  wait "${bench_pid}"
 }
 wait_for_bench_main() {
  local bench_log="$1"
  local bench_pid="$2"
@ -474,6 +556,7 @@ run_mixed_profile() {
  local injection_id="${case_id}/prefill_injection"
  local background_path="${CASE_DIR}/${background_id}"
  local before=0
  local decode_before=0
  flush_cache
  mkdir -p "${background_path}"
  prepare_bench_command "${background_path}/bench.jsonl" \
@ -481,16 +564,18 @@ run_mixed_profile() {
  print_command timeout --signal=TERM --kill-after=30s \
    "${PROFILE_TIMEOUT_S}s" "${BENCH_CMD[@]}" > "${background_path}/bench_cmd.txt"
  if [[ "${DRY_RUN}" == "1" ]]; then
-    log "[DRY] start decode background, wait for main run, start profile, inject 128K prefill"
+    log "[DRY] start decode background, wait for Decode batch, start profile, inject 128K prefill"
    start_profile "${case_id}" '["CUDA_PROFILER"]' "${NSYS_MIXED_STEPS}"
    run_bench "${injection_id}" "${MIXED_INJECTION_ISL}" \
      "${MIXED_INJECTION_OSL}" 1 1 5202
    return 0
  fi
  decode_before="$(server_decode_count)"
  timeout --signal=TERM --kill-after=30s "${PROFILE_TIMEOUT_S}s" \
    "${BENCH_CMD[@]}" > "${background_path}/bench.log" 2>&1 &
  local background_pid=$!
-  wait_for_bench_main "${background_path}/bench.log" "${background_pid}" || {
+  wait_for_bench_main "${background_path}/bench.log" "${background_pid}" \
    && wait_for_decode_activity "${decode_before}" "${background_pid}" || {
    kill "${background_pid}" 2>/dev/null || true
    wait "${background_pid}" 2>/dev/null || true
    log "ERROR: mixed decode background did not enter main benchmark"
@ -537,10 +622,16 @@ generate_nsys_stats() {
    relative="${report#${RESULT_DIR}/}"
    output="${report%.nsys-rep}.stats.txt"
    docker run --rm \
-      -v "${RESULT_DIR}:/results:ro" \
+      -v "${RESULT_DIR}:/results" \
      --entrypoint nsys "${DOCKER_IMAGE}" stats \
-      --report cuda_gpu_kern_sum,cuda_api_sum,cuda_kern_exec_sum,nvtx_sum \
+      --force-export=true \
-      "/results/${relative}" > "${output}" 2>&1 || true
+      --report cuda_gpu_kern_sum,cuda_api_sum,nvtx_sum \
      "/results/${relative}" > "${output}" 2>&1
    grep -q 'CUDA GPU Kernel Summary' "${output}" || {
      log "ERROR: nsys stats missing CUDA kernel summary for ${relative}"
      return 1
    }
    rm -f "${report%.nsys-rep}.sqlite"
  done < <(find "${RESULT_DIR}/profiles" -type f -name '*.nsys-rep' | sort)
 }
@ -586,7 +677,7 @@ run_all() {
  CLEANUP_ON_EXIT=1
  start_service
  write_manifest RUNNING
-  run_profiled_case decode_control '["CUDA_PROFILER"]' \
+  run_profiled_decode_case decode_control \
    "${NSYS_CONTROL_STEPS}" "${CONTROL_ISL}" "${CONTROL_OSL}" \
    "${CONTROL_CONCURRENCY}" "${CONTROL_PROMPTS}" 5301
  run_mixed_profile
Author	SHA1	Message	Date
Zhiyi Hong	e1719bd575	[Docs] finalize Phase 2.5 RDMA demand model	2026-08-01 15:29:26 +08:00
Zhiyi Hong	c5fa700c50	[Feat] add Phase 2.5 RDMA demand modeling	2026-08-01 02:43:06 +08:00
Zhiyi Hong	119701a417	[BugFix] align Phase 3 captures with active decode	2026-08-01 00:50:45 +08:00
		`@ -0,0 +1 @@`
							env DRY_RUN=0 RUN_ID=confirm RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/confirm.tsv FIXED_CASE_IDS=rdma_confirm_i65536_o1024_c4\,rdma_confirm_i65536_o1024_c16\,rdma_confirm_i65536_o1024_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=2 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
		`@ -0,0 +1 @@`
							env DRY_RUN=0 RUN_ID=scout RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/scout.tsv FIXED_CASE_IDS=rdma_scout_i65536_o1_c1\,rdma_scout_i65536_o1_c4\,rdma_scout_i65536_o1_c16\,rdma_scout_i65536_o1_c32\,rdma_scout_i65536_o1_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=1 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
		`@ -0,0 +1 @@`
							docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256