[Docs] record completed DSV4-Pro Phase 1 quick map
This commit is contained in:
Zhiyi Hong
parent
75182c6ede
commit
25418ec174
@ -1,5 +1,9 @@
|
||||
# sskj — 多平台大模型推理性能基准测试项目
|
||||
|
||||
> **更新（2026-07-30 23:06:01 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 Phase 1 正式 quick-map。Head 与 Worker 均通过 `mlx5_0/mlx5_3` 双 Rail `NET/IB + GDRDMA` 门禁；9 个固定点和 3 个混合 A/B 结果共 12/12 成功，总用时 28 分 36 秒。32K/128K 单请求 Prefill 输入吞吐为 2,652.76/2,710.16 token/s；在 C=32 Decode 中注入一个 128K Prefill 后，Output TPS 下降 24.08%，TPOT P95 增加 66.55%。阶段 HTML 已重写为只保留成功结果，并补充正式汇总、运行清单和 Phase 2 三个诊断负载；两节点容器和 16 张 GPU 已清理。
|
||||
>
|
||||
> **更新（2026-07-30 18:40:53 CST）**
|
||||
>
|
||||
> 为 DeepSeek-V4-Pro 双机 TP16 quick-map 的唯一入口新增 `CASE_IDS` 场景过滤和未知 Case 预检，可在完整九点实验前先跑 1K/32K Prefill 与 C1/C32 Decode 四点 Sanity；运行清单会记录实际过滤条件。同步精简阶段档案：正文只保留最终成功 Run 与有效结论，历史失败压缩到末尾经验教训。
|
||||
|
||||
@ -112,6 +112,12 @@
|
||||
color: #075e58;
|
||||
}
|
||||
|
||||
.decision {
|
||||
padding: 14px 18px;
|
||||
background: var(--teal-soft);
|
||||
border-left: 4px solid var(--teal);
|
||||
}
|
||||
|
||||
h2 {
|
||||
margin: 48px 0 16px;
|
||||
padding-bottom: 9px;
|
||||
@ -228,7 +234,7 @@
|
||||
<div class="meta">
|
||||
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||
<span>拓扑：SGLang TP16 / EP2</span>
|
||||
<span>更新：2026-07-30 17:54 CST</span>
|
||||
<span>更新：2026-07-30 22:55:37 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
@ -237,9 +243,10 @@
|
||||
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||
|
||||
<p class="status">
|
||||
<strong>阶段状态：等待真实双 Rail RDMA 重跑。</strong>
|
||||
启动器与静态验证已完成；历史无效 Run 不进入最终分析。下一次仅在
|
||||
NCCL 日志通过 <code>NET/IB</code> 门禁后执行 Sanity 和完整 quick map。
|
||||
<strong>阶段状态：已完成。</strong>
|
||||
正式 Run <code>dsv4pro-phase1-full-20260730-220916</code> 在双 Rail
|
||||
<code>NET/IB + GDRDMA</code> 下完成 9 个固定点和 3 个混合 A/B 结果，
|
||||
共 12/12 成功，用时 28 分 36 秒；服务、容器与 16 张 GPU 已清理。
|
||||
</p>
|
||||
|
||||
<h2>1. 目标与边界</h2>
|
||||
@ -530,34 +537,96 @@ wait "${background_pid}"</code></pre>
|
||||
<tr><td>完整 Dry-run</td><td class="pass">通过</td><td>服务、九个固定点、混合 A/B、清理均展开成功</td></tr>
|
||||
<tr><td>真实旧 Bench JSON 解析</td><td class="pass">通过</td><td>成功解析 P50/P95/P99 与吞吐字段</td></tr>
|
||||
<tr><td>项目精简</td><td class="pass">通过</td><td>实验目录顶层仅保留一个 Shell 入口</td></tr>
|
||||
<tr><td>RDMA fail-closed</td><td class="pass">静态验证通过</td><td>非法网卡、缺失设备或未出现 <code>NET/IB</code> 时禁止 benchmark</td></tr>
|
||||
<tr><td>双 Rail 传输门禁</td><td class="pass">通过</td><td>Head 与 Worker 均识别 <code>mlx5_0/mlx5_3</code>，跨节点 Channel 使用 <code>NET/IB/*/GDRDMA</code></td></tr>
|
||||
<tr><td>四点 Sanity</td><td class="pass">4/4 通过</td><td>1K/32K Prefill 与 C1/C32 Decode 均恢复到合理量级</td></tr>
|
||||
<tr><td>冷 Prefix 口径</td><td class="pass">通过</td><td>正式测量请求的 Head 日志显示 <code>#cached-token: 0</code></td></tr>
|
||||
<tr><td>完整真机 Run</td><td class="pass">12/12 通过</td><td>固定矩阵 9/9，混合 A/B 3/3，运行期失败 0</td></tr>
|
||||
<tr><td>资源清理</td><td class="pass">通过</td><td>两节点相关容器与计算进程为 0，16 张 GPU 显存占用为 0</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>9. 最终真机结果</h2>
|
||||
<p class="pending">
|
||||
正确双 Rail RDMA 下的正式结果尚未生成。本节只接受通过传输门禁、
|
||||
冷缓存口径和结果校验的最终 Run；完成后将写入四点 Sanity、九个固定点、
|
||||
混合 A/B 与阶段结论。
|
||||
<p>
|
||||
本节只使用正式成功 Run。Profiler 与投机解码均关闭，每个 Case 只做一次快速测量，
|
||||
所以它适合决定下一步 Profile 对象，不作为需要统计置信度的最终容量认证。
|
||||
</p>
|
||||
|
||||
<h3>9.1 执行摘要</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>门禁或实验</th><th>状态</th><th>最终证据</th></tr>
|
||||
<tr><th>项目</th><th>结果</th><th>证据</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>双 Rail <code>NET/IB</code></td><td class="pending">待真机验证</td><td>头、Worker 日志均需识别 <code>mlx5_0/mlx5_3</code></td></tr>
|
||||
<tr><td>四点 Sanity</td><td class="pending">待执行</td><td>1K/32K Prefill 与 C1/C32 Decode</td></tr>
|
||||
<tr><td>九个固定点</td><td class="pending">待执行</td><td>最终 <code>summary.csv</code> 与 <code>report.md</code></td></tr>
|
||||
<tr><td>混合 A/B</td><td class="pending">待执行</td><td>Control 与 Treatment 的 Decode 指标变化</td></tr>
|
||||
<tr><td>Run ID</td><td><code>dsv4pro-phase1-full-20260730-220916</code></td><td><code>COMPLETED</code></td></tr>
|
||||
<tr><td>运行时间</td><td>28 分 36 秒</td><td>22:09:47 至 22:38:22 CST</td></tr>
|
||||
<tr><td>网络路径</td><td>双 Rail <code>NET/IB + GDRDMA</code></td><td><code>mlx5_0</code> 与 <code>mlx5_3</code></td></tr>
|
||||
<tr><td>结果完整性</td><td>12/12 成功</td><td>固定点 9/9；混合 A/B 3/3</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>9.2 Prefill</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>场景</th><th>Input TPS</th><th>TTFT P95</th><th>观察</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>1K → 1，C=1</td><td>1,969.66 tok/s</td><td>0.502 s</td><td>短请求固定开销占比更高</td></tr>
|
||||
<tr><td>32K → 1，C=1</td><td>2,652.76 tok/s</td><td>12.335 s</td><td>单请求吞吐进入稳定区间</td></tr>
|
||||
<tr><td>128K → 1，C=1</td><td>2,710.16 tok/s</td><td>48.344 s</td><td>长 Prefill 代表点</td></tr>
|
||||
<tr><td>32K → 1，C=16</td><td>3,112.77 tok/s</td><td>162.087 s</td><td>聚合吞吐仅比 C=1 高 17.3%，排队时延显著增加</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>9.3 Decode</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>1K → 1K</th><th>Output TPS</th><th>TTFT P95</th><th>TPOT P95</th><th>E2E P95</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>C=1</td><td>31.41 tok/s</td><td>0.363 s</td><td>31.47 ms</td><td>32.555 s</td></tr>
|
||||
<tr><td>C=16</td><td>295.29 tok/s</td><td>4.950 s</td><td>50.02 ms</td><td>55.444 s</td></tr>
|
||||
<tr><td>C=32</td><td>461.68 tok/s</td><td>8.022 s</td><td>63.31 ms</td><td>70.933 s</td></tr>
|
||||
<tr><td>C=64</td><td>647.42 tok/s</td><td>12.716 s</td><td>93.44 ms</td><td>101.163 s</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
Decode 吞吐到 C=64 仍在上升，但增益递减且 TPOT 明显变差。综合场景
|
||||
<code>32K → 1K，C=8</code> 的 Input/Output TPS 为
|
||||
<code>2,038.00 / 63.69</code>，TTFT P95 为 <code>82.084 s</code>，
|
||||
说明 Prefill 与 Decode 同时存在时干扰很强。
|
||||
</p>
|
||||
|
||||
<h3>9.4 混合 Prefill/Decode A/B</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>指标</th><th>A：仅 Decode</th><th>B：注入 128K Prefill</th><th>变化</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>Output TPS</td><td>455.68 tok/s</td><td>345.95 tok/s</td><td>-24.08%</td></tr>
|
||||
<tr><td>TTFT P95</td><td>9.443 s</td><td>10.194 s</td><td>+7.96%</td></tr>
|
||||
<tr><td>TPOT P95</td><td>65.88 ms</td><td>109.73 ms</td><td>+66.55%</td></tr>
|
||||
<tr><td>E2E P95</td><td>72.008 s</td><td>117.630 s</td><td>+63.36%</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="decision">
|
||||
Phase 2 优先采集三类代表负载：<code>128K → 1，C=1</code> 的纯长 Prefill、
|
||||
<code>32K → 1，C=16</code> 的并发 Prefill，以及
|
||||
<code>1K → 1K，C=32</code> 在有无 128K 注入时的混合 A/B。
|
||||
目标是区分计算、显存带宽、调度排队、跨机通信和节点不均衡。
|
||||
</p>
|
||||
<p>
|
||||
完整产物：
|
||||
<a href="./results/dsv4pro-phase1-full-20260730-220916/report.md">报告</a>、
|
||||
<a href="./results/dsv4pro-phase1-full-20260730-220916/summary.csv">逐点汇总</a>、
|
||||
<a href="./results/dsv4pro-phase1-full-20260730-220916/aggregate.csv">聚合表</a>、
|
||||
<a href="./results/dsv4pro-phase1-full-20260730-220916/run_manifest.json">运行清单</a>。
|
||||
</p>
|
||||
|
||||
<h2>10. 经验教训</h2>
|
||||
<ul>
|
||||
<li>启动参数不等于实际传输路径；开始性能测试前必须由 NCCL 日志证明 <code>NET/IB</code>。</li>
|
||||
<li>Warm-up、固定随机种子和跨 Case Prefix Cache 会改变 TTFT，冷缓存与热缓存必须分开报告。</li>
|
||||
<li>传输门禁或测量口径不成立时，性能结果不能用于归因模型、调度器或 Kernel。</li>
|
||||
<li>完整实验前先跑少量 Sanity 点，能更早发现环境和口径问题。</li>
|
||||
<li>先跑四点 Sanity 再启动完整矩阵，可以在几分钟内验证环境、口径和数量级。</li>
|
||||
</ul>
|
||||
<p>
|
||||
历史排查细节保存在
|
||||
@ -570,11 +639,11 @@ wait "${background_pid}"</code></pre>
|
||||
<h2>11. 运行命令</h2>
|
||||
<pre><code class="language-bash">cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||
|
||||
tmux new-session -d -s dsv4pro-pro6000d-2node-sglang-quick-map -c "$PWD"
|
||||
tmux send-keys -t dsv4pro-pro6000d-2node-sglang-quick-map \
|
||||
'RUN_ID=dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625 bash run_quick_map.sh all' Enter
|
||||
tmux new-session -d -s dsv4pro-phase1-full \
|
||||
"RUN_ID=dsv4pro-phase1-full-20260730-220916 bash run_quick_map.sh all \
|
||||
2>&1 | tee /data/hzy/dsv4pro_phase1_full_20260730-220916.log"
|
||||
|
||||
tmux attach -t dsv4pro-pro6000d-2node-sglang-quick-map</code></pre>
|
||||
tmux attach -t dsv4pro-phase1-full</code></pre>
|
||||
|
||||
<p>
|
||||
下一阶段：
|
||||
|
||||
@ -135,7 +135,7 @@
|
||||
<div class="meta">
|
||||
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||
<span>拓扑：SGLang TP16 / EP2</span>
|
||||
<span>更新：2026-07-30 17:54 CST</span>
|
||||
<span>更新：2026-07-30 22:55:37 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
@ -144,55 +144,35 @@
|
||||
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||
|
||||
<p class="status">
|
||||
<strong>当前状态：网络前置条件未满足，Phase 2 暂停，代码尚未开始。</strong>
|
||||
本页从第一行 Phase 2 代码开始同步维护。每次代码改动、静态验证、真机运行和
|
||||
结果判断都会在对应小节留下文件路径、命令和证据，不在阶段结束后凭记忆补写。
|
||||
<strong>当前状态：Phase 1 前置条件已通过，等待阶段汇报确认后开始实现。</strong>
|
||||
Phase 2 代码尚未创建。本页已经根据最终 Phase 1 结果选择诊断 Case；
|
||||
后续代码改动、静态验证、真机运行和结果判断会同步写入本页。
|
||||
</p>
|
||||
|
||||
<h2>1. 为什么现在不能直接进入 Phase 2</h2>
|
||||
<p>
|
||||
Phase 1 在没有 Profiler、没有 Prefix Cache 命中的条件下得到以下结果，
|
||||
但这些数值后来确认受错误 Socket 网络路径污染：
|
||||
</p>
|
||||
<h2>1. Phase 1 交接结果</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>ISL / OSL / C</th><th>输入 TPS</th><th>TTFT</th><th>结果</th></tr>
|
||||
<tr><th>代表负载</th><th>关键结果</th><th>Phase 2 用途</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>1K / 1 / 1</td><td>64.44 tok/s</td><td>15.88 s</td><td>错误网络证据，不作为基线</td></tr>
|
||||
<tr><td>32K / 1 / 1</td><td>64.96 tok/s</td><td>504.44 s</td><td>错误网络证据，不作为基线</td></tr>
|
||||
<tr><td>128K / 1 / 1</td><td>65.20 tok/s</td><td>2010.38 s</td><td>错误网络证据，不作为基线</td></tr>
|
||||
<tr><td>128K → 1，C=1</td><td>Input TPS 2,710.16；TTFT P95 48.344 s</td><td>纯长 Prefill 的计算、显存与通信归因</td></tr>
|
||||
<tr><td>32K → 1，C=16</td><td>Input TPS 3,112.77；TTFT P95 162.087 s</td><td>并发 Prefill 的排队、Chunk 调度与节点均衡</td></tr>
|
||||
<tr><td>1K → 1K，C=32 + 128K 注入</td><td>Output TPS -24.08%；TPOT P95 +66.55%</td><td>Prefill 干扰 Decode 时的硬件资源竞争</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
quick-map 容器没有 <code>/dev/infiniband</code>，NCCL 回退
|
||||
<code>NET/Socket</code>；脚本又误选低速非计算网。原网络配置冷请求控制组中，
|
||||
1K/32K TTFT 分别只有 1.458s/38.062s，比 quick-map 快约 10.9×/13.25×。
|
||||
因此约 65 token/s 不是待 profile 的模型现象，而是已定位的部署配置错误。
|
||||
</p>
|
||||
|
||||
<h3>1.1 Phase 2 前置审计</h3>
|
||||
<p>
|
||||
旧脚本较短的 TTFT 包含两个因素。其一，旧脚本固定执行 16 条同 Prompt
|
||||
Warm-up，从不清 Prefix Cache，并按固定 Seed 递增长度；17:40 的失败 Run
|
||||
还在 18:01 正式 Run 前预热了同一批 1K 请求。其二，新旧模型参数虽相同，
|
||||
NCCL Socket 接口不同，而二者容器都没有形成真实 RDMA 数据面。
|
||||
</p>
|
||||
<p>
|
||||
网络控制组明确打印 <code>NET/IB : No device found</code> 和
|
||||
<code>Using network Socket</code>。Phase 1 入口现已加入 RDMA 设备透传与
|
||||
<code>NET/IB</code> fail-closed 校验，但真机验证尚未运行。Phase 2 只有在
|
||||
双 Rail RDMA 得到运行时证据并重跑 Phase 1 后才会开始。Warm Prefix/Prefix Cache
|
||||
收益仍另立 A/B，不与冷 Prefill 混算。
|
||||
最终基线已由 Head 与 Worker 日志证明使用
|
||||
<code>mlx5_0/mlx5_3</code> 双 Rail <code>NET/IB + GDRDMA</code>，
|
||||
正式测量请求为冷 Prefix，12/12 结果成功。Phase 2 保持相同服务配置和请求口径。
|
||||
</p>
|
||||
|
||||
<h2>2. 本阶段的边界</h2>
|
||||
<ul>
|
||||
<li>只测试 SGLang，不测试 vLLM。</li>
|
||||
<li>保留模型、镜像、TP16、EP2 和显存比例；NCCL 参数必须使用修正并验证后的版本。</li>
|
||||
<li>保留模型、镜像、TP16、EP2、显存比例和已验证的双 Rail NCCL 配置。</li>
|
||||
<li>不启用 Nsight Systems、PyTorch Profiler、NCCL DEBUG 或投机解码。</li>
|
||||
<li>不调参，不尝试优化；先获得足以区分瓶颈类别的硬件证据。</li>
|
||||
<li>第一轮仍只重放 <code>32K → 1, C=1</code>，但必须与修正后的 Phase 1 结果对齐。</li>
|
||||
<li>只重放本页选出的三个代表负载，不重复 Phase 1 全矩阵。</li>
|
||||
<li>采集器从请求开始前启动，到请求结束后停止，不能中途补采后声称完整。</li>
|
||||
</ul>
|
||||
|
||||
@ -216,15 +196,16 @@
|
||||
<li>保存两节点静态快照：GPU/NIC/NUMA 拓扑、驱动、CUDA、镜像与服务命令。</li>
|
||||
<li>复用 Phase 1 已验证的 <code>run_quick_map.sh start</code> 启动同配置双机服务。</li>
|
||||
<li>在 Head 和 Worker 同时启动 GPU、CPU、网卡与 RDMA 采样，先记录 15 秒空闲基线。</li>
|
||||
<li>清空 Prefix Cache，发送一条 <code>32K → 1, C=1</code> 请求，Seed 与 Phase 1 一致。</li>
|
||||
<li>依次重放 <code>128K → 1, C=1</code> 与 <code>32K → 1, C=16</code>，请求参数和 Seed 与 Phase 1 一致。</li>
|
||||
<li>重放 <code>1K → 1K, C=32</code> Control 与 128K Prefill 注入 Treatment，保留相同注入时序。</li>
|
||||
<li>请求结束后继续采样 15 秒，再停止采集器和服务。</li>
|
||||
<li>按时间戳将请求、8K Chunk、GPU、CPU 和 Rail 指标对齐，生成摘要与判定。</li>
|
||||
<li>按时间戳将请求、GPU、CPU 和双 Rail 指标对齐，生成摘要与判定。</li>
|
||||
</ol>
|
||||
<pre><code>idle 15s │──────── 32K Prefill：4 × 8K Chunk ────────│ cooldown 15s
|
||||
↑ request_start ↑ request_end
|
||||
Head 与 Worker 的所有采集器覆盖完整时间窗</code></pre>
|
||||
<pre><code>idle 15s │ 128K C1 │ 32K C16 │ Decode Control │ Decode + Prefill │ cooldown 15s
|
||||
Head 与 Worker 的所有采集器覆盖完整诊断窗口</code></pre>
|
||||
<p>
|
||||
预计服务加载约 6 分钟、请求约 8.5 分钟，连同快照和清理应在 20 分钟左右完成。
|
||||
Phase 1 中服务加载约 5 分 30 秒，三个诊断负载合计为分钟级；
|
||||
连同静态快照、采样和清理，目标仍控制在 30 分钟内。
|
||||
</p>
|
||||
|
||||
<h2>5. 采集指标</h2>
|
||||
@ -251,7 +232,7 @@ dsv4pro_pro6000d_2node_sglang_prefill_hardware_attribution/</code></pre>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td><code>run_prefill_hardware_attribution.sh</code></td><td>唯一 Shell 入口；服务启停、双节点采集器、单 Case、Trap 清理</td><td class="pending">待实现</td></tr>
|
||||
<tr><td><code>config.env</code></td><td>Phase 1 入口路径、32K Case、采样间隔、结果路径</td><td class="pending">待实现</td></tr>
|
||||
<tr><td><code>config.env</code></td><td>Phase 1 入口路径、三个诊断 Case、采样间隔、结果路径</td><td class="pending">待实现</td></tr>
|
||||
<tr><td><code>hardware_attribution.py</code></td><td>结构化解析、时间对齐、统计摘要与报告生成</td><td class="pending">待实现</td></tr>
|
||||
<tr><td><code>tests/test_hardware_attribution.py</code></td><td>计数器差分、单位换算、统计与缺失工具回退测试</td><td class="pending">待实现</td></tr>
|
||||
<tr><td><code>README.md</code></td><td>入口命令、环境变量和结果目录说明</td><td class="pending">待实现</td></tr>
|
||||
@ -259,7 +240,7 @@ dsv4pro_pro6000d_2node_sglang_prefill_hardware_attribution/</code></pre>
|
||||
</table>
|
||||
<p class="decision">
|
||||
Phase 2 不复制双机 Docker 启停实现。唯一入口通过环境变量调用 Phase 1 的
|
||||
<code>run_quick_map.sh start/stop</code>，只新增硬件采集和 32K 请求编排。
|
||||
<code>run_quick_map.sh start/stop</code>，只新增硬件采集和三个代表负载的编排。
|
||||
顶层仍只保留一个 Shell 文件，不创建额外 tmux、launch、start 或 stop 脚本。
|
||||
</p>
|
||||
|
||||
@ -296,7 +277,7 @@ dsv4pro_pro6000d_2node_sglang_prefill_hardware_attribution/</code></pre>
|
||||
|
||||
<h2>8. 验收条件</h2>
|
||||
<ul>
|
||||
<li>Bench 的 ISL、OSL、并发、Seed、缓存状态与 Phase 1 的 32K Case 一致。</li>
|
||||
<li>Bench 的 ISL、OSL、并发、Seed、缓存状态与 Phase 1 对应 Case 一致。</li>
|
||||
<li>两节点采集器均覆盖请求开始前 15 秒到结束后 15 秒。</li>
|
||||
<li>每份时间序列有节点名、墙钟时间和单调时钟，能与服务 Chunk 日志对齐。</li>
|
||||
<li>采集器不可用时记录 <code>UNAVAILABLE</code> 和原因，不静默跳过。</li>
|
||||
@ -311,17 +292,15 @@ dsv4pro_pro6000d_2node_sglang_prefill_hardware_attribution/</code></pre>
|
||||
<tr><th>时间</th><th>代码或运行</th><th>结果</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>2026-07-30 15:46 CST</td><td>创建 Phase 2 设计与档案</td><td>代码尚未开始，等待按本页设计实现</td></tr>
|
||||
<tr><td>2026-07-30 16:35 CST</td><td>完成旧脚本与 quick-map 同口径审计</td><td>排除服务参数、OSL=1 和一次性 JIT；确认旧产物被 Warm-up、跨 Case 与前一轮 Prefix Cache 污染</td></tr>
|
||||
<tr><td>2026-07-30 17:30 CST</td><td>完成原网络配置冷请求控制组</td><td>确认 quick-map 误入低速非计算网 Socket；Phase 2 暂停，先修正并重跑 Phase 1</td></tr>
|
||||
<tr><td>2026-07-30 17:54 CST</td><td>完成 Phase 1 RDMA fail-closed 代码与 dry-run</td><td>只允许 eth0/eth3 与 mlx5_0/mlx5_3；等待真机 NET/IB 证据</td></tr>
|
||||
<tr><td>2026-07-30 15:46 CST</td><td>创建 Phase 2 设计与档案</td><td>采用单入口和独立轻量采样，不复制双机服务启动逻辑</td></tr>
|
||||
<tr><td>2026-07-30 22:55:37 CST</td><td>完成 Phase 1 阶段交接</td><td>双 Rail 门禁和 12/12 正式结果通过；选定纯 Prefill、并发 Prefill、混合干扰三个诊断负载</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>10. 真机结果</h2>
|
||||
<p class="pending">
|
||||
尚未运行，也不应立即运行。先获得修正网络后的 Phase 1 冷 32K 基线；
|
||||
Phase 2 代码实现、静态验证和 Dry-run 完成后，再向用户说明具体改动并等待阶段门。
|
||||
尚未运行。按阶段门约定，先完成 Phase 1 汇报；用户确认进入 Phase 2 后，
|
||||
再实现代码、完成静态验证和 Dry-run，并在启动真机诊断前说明具体改动。
|
||||
</p>
|
||||
|
||||
<p><a class="back" href="./phase1_dsv4pro_pro6000d_2node_sglang_quick_map.html">返回 Phase 1 实施记录</a></p>
|
||||
|
||||
@ -0,0 +1,13 @@
|
||||
suite,case_id,role,stage,isl,osl,concurrency,planned_repetitions,completed_repetitions,statuses,total_tps_mean,total_tps_cv_pct,output_tps_mean,output_tps_cv_pct,ttft_p95_mean_ms,tpot_p95_mean_ms,itl_p95_mean_ms,e2e_p95_mean_ms
|
||||
fixed,balanced_32k_to_1k_c8,,balanced,32768,1024,8,1,1,COMPLETED,2101.686722481554,,63.68747643883498,,82083.64989476977,109.32309288714016,43.84165157971437,128588.8987001963
|
||||
fixed,decode_latency_1k_to_1k_c1,,decode_latency,1024,1024,1,1,1,COMPLETED,62.82990203738489,,31.414951018692445,,363.2723209448159,31.468245639333286,31.659404194215313,32555.28760998277
|
||||
fixed,decode_throughput_1k_to_1k_c16,,decode_throughput,1024,1024,16,1,1,COMPLETED,590.5818337765435,,295.29091688827174,,4950.267374995747,50.01878498413362,49.8873026604997,55443.82167501317
|
||||
fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1024,1024,32,1,1,COMPLETED,923.3692532200665,,461.68462661003326,,8021.852347906679,63.30661669023356,62.237933481810614,70933.18627287517
|
||||
fixed,decode_throughput_1k_to_1k_c64,,decode_throughput,1024,1024,64,1,1,COMPLETED,1294.833542313111,,647.4167711565555,,12716.491229846724,93.44459645826957,87.80462378927041,101163.1970004586
|
||||
fixed,long_prefill_latency_128k_c1,,prefill_latency,131072,1,1,1,1,COMPLETED,2710.1829473156417,,0.020676897204730506,,48344.461318978574,0.0,0.0,48344.52949295519
|
||||
fixed,mid_prefill_latency_32k_c1,,prefill_latency,32768,1,1,1,1,COMPLETED,2652.8390294198,,0.08095575175988892,,12334.68782599084,0.0,0.0,12334.764264000114
|
||||
fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,32768,1,16,1,1,COMPLETED,3112.8645802960464,,0.09499418902914482,,162087.46085499297,0.0,0.0,162087.50443853205
|
||||
fixed,short_prefill_latency_1k_c1,,prefill_latency,1024,1,1,1,1,COMPLETED,1971.5883577957954,,1.9235008368739468,,501.89953204244375,0.0,0.0,501.94522901438177
|
||||
mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1024,1024,32,1,1,COMPLETED,911.3699336194111,,455.68496680970554,,9442.636363586644,65.88327712923635,61.753195300116204,72007.63789927005
|
||||
mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1024,1024,32,1,1,COMPLETED,691.8932985710949,,345.94664928554744,,10193.896457596566,109.7299457727384,61.61956858995836,117629.63820034638
|
||||
mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,131072,1,1,1,1,COMPLETED,2859.7533639992707,,0.021818020217735695,,45752.13837000774,0.0,0.0,45752.21362197772
|
||||
|
@ -0,0 +1,31 @@
|
||||
# DeepSeek-V4-Pro Pro6000D Two-Node SGLang Quick Map
|
||||
|
||||
Profiler: disabled. Speculative decoding: disabled.
|
||||
|
||||
## Aggregate results
|
||||
|
||||
| Case | Suite / role | Stage | ISL | OSL | C | Reps | Total TPS | CV | Output TPS | TTFT P95 | TPOT P95 | E2E P95 | Status |
|
||||
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| balanced_32k_to_1k_c8 | fixed / - | balanced | 32768 | 1024 | 8 | 1/1 | 2101.69 | -% | 63.69 | 82083.65 ms | 109.32 ms | 128588.90 ms | COMPLETED |
|
||||
| decode_latency_1k_to_1k_c1 | fixed / - | decode_latency | 1024 | 1024 | 1 | 1/1 | 62.83 | -% | 31.41 | 363.27 ms | 31.47 ms | 32555.29 ms | COMPLETED |
|
||||
| decode_throughput_1k_to_1k_c16 | fixed / - | decode_throughput | 1024 | 1024 | 16 | 1/1 | 590.58 | -% | 295.29 | 4950.27 ms | 50.02 ms | 55443.82 ms | COMPLETED |
|
||||
| decode_throughput_1k_to_1k_c32 | fixed / - | decode_throughput | 1024 | 1024 | 32 | 1/1 | 923.37 | -% | 461.68 | 8021.85 ms | 63.31 ms | 70933.19 ms | COMPLETED |
|
||||
| decode_throughput_1k_to_1k_c64 | fixed / - | decode_throughput | 1024 | 1024 | 64 | 1/1 | 1294.83 | -% | 647.42 | 12716.49 ms | 93.44 ms | 101163.20 ms | COMPLETED |
|
||||
| long_prefill_latency_128k_c1 | fixed / - | prefill_latency | 131072 | 1 | 1 | 1/1 | 2710.18 | -% | 0.02 | 48344.46 ms | 0.00 ms | 48344.53 ms | COMPLETED |
|
||||
| mid_prefill_latency_32k_c1 | fixed / - | prefill_latency | 32768 | 1 | 1 | 1/1 | 2652.84 | -% | 0.08 | 12334.69 ms | 0.00 ms | 12334.76 ms | COMPLETED |
|
||||
| mid_prefill_throughput_32k_c16 | fixed / - | prefill_throughput | 32768 | 1 | 16 | 1/1 | 3112.86 | -% | 0.09 | 162087.46 ms | 0.00 ms | 162087.50 ms | COMPLETED |
|
||||
| short_prefill_latency_1k_c1 | fixed / - | prefill_latency | 1024 | 1 | 1 | 1/1 | 1971.59 | -% | 1.92 | 501.90 ms | 0.00 ms | 501.95 ms | COMPLETED |
|
||||
| decode_control_1k_to_1k_c32 | mixed / control | mixed_interference | 1024 | 1024 | 32 | 1/1 | 911.37 | -% | 455.68 | 9442.64 ms | 65.88 ms | 72007.64 ms | COMPLETED |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | mixed / decode_background | mixed_interference | 1024 | 1024 | 32 | 1/1 | 691.89 | -% | 345.95 | 10193.90 ms | 109.73 ms | 117629.64 ms | COMPLETED |
|
||||
| long_prefill_injection_128k_to_1_c1 | mixed / prefill_injection | mixed_interference | 131072 | 1 | 1 | 1/1 | 2859.75 | -% | 0.02 | 45752.14 ms | 0.00 ms | 45752.21 ms | COMPLETED |
|
||||
|
||||
## Mixed-interference A/B
|
||||
|
||||
| Metric | Control | With 128K prefill | Change |
|
||||
|---|---:|---:|---:|
|
||||
| Output TPS | 455.68 | 345.95 | -24.08% |
|
||||
| TTFT P95 (ms) | 9442.64 | 10193.90 | 7.96% |
|
||||
| TPOT P95 (ms) | 65.88 | 109.73 | 66.55% |
|
||||
| E2E P95 (ms) | 72007.64 | 117629.64 | 63.36% |
|
||||
|
||||
The fixed map is descriptive and never stops on SLO. With one repetition, CV is intentionally unavailable.
|
||||
@ -0,0 +1,53 @@
|
||||
[2026-07-30 22:09:49] Starting worker node=10.101.0.13 rank=1 container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker
|
||||
[2026-07-30 22:09:55] Starting head node=10.101.0.11 rank=0 container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head
|
||||
[2026-07-30 22:09:56] Waiting for SGLang health at 10.101.0.11:30002
|
||||
[2026-07-30 22:10:55] Still starting: checks=12/600
|
||||
[2026-07-30 22:12:00] Still starting: checks=24/600
|
||||
[2026-07-30 22:13:05] Still starting: checks=36/600
|
||||
[2026-07-30 22:14:09] Still starting: checks=48/600
|
||||
[2026-07-30 22:15:14] Still starting: checks=60/600
|
||||
[2026-07-30 22:15:20] SGLang is healthy after 61 checks
|
||||
[2026-07-30 22:15:20] Verified worker NCCL transport: NET/IB with mlx5_0 and mlx5_3
|
||||
[2026-07-30 22:15:21] Verified head NCCL transport: NET/IB with mlx5_0 and mlx5_3
|
||||
[2026-07-30 22:15:21] Validating scenario file: /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv
|
||||
short_prefill_latency_1k_c1: stage=prefill_latency isl=1024 osl=1 c=1 prompts=1 warmup=1
|
||||
mid_prefill_latency_32k_c1: stage=prefill_latency isl=32768 osl=1 c=1 prompts=1 warmup=0
|
||||
long_prefill_latency_128k_c1: stage=prefill_latency isl=131072 osl=1 c=1 prompts=1 warmup=0
|
||||
mid_prefill_throughput_32k_c16: stage=prefill_throughput isl=32768 osl=1 c=16 prompts=16 warmup=0
|
||||
decode_latency_1k_to_1k_c1: stage=decode_latency isl=1024 osl=1024 c=1 prompts=1 warmup=1
|
||||
decode_throughput_1k_to_1k_c16: stage=decode_throughput isl=1024 osl=1024 c=16 prompts=16 warmup=1
|
||||
decode_throughput_1k_to_1k_c32: stage=decode_throughput isl=1024 osl=1024 c=32 prompts=32 warmup=1
|
||||
decode_throughput_1k_to_1k_c64: stage=decode_throughput isl=1024 osl=1024 c=64 prompts=64 warmup=1
|
||||
balanced_32k_to_1k_c8: stage=balanced isl=32768 osl=1024 c=8 prompts=8 warmup=0
|
||||
[2026-07-30 22:15:21] START case=short_prefill_latency_1k_c1 rep=1 isl=1024 osl=1 c=1
|
||||
[2026-07-30 22:15:55] DONE case=short_prefill_latency_1k_c1 rep=1 elapsed=34s
|
||||
[2026-07-30 22:16:00] START case=mid_prefill_latency_32k_c1 rep=1 isl=32768 osl=1 c=1
|
||||
[2026-07-30 22:16:46] DONE case=mid_prefill_latency_32k_c1 rep=1 elapsed=46s
|
||||
[2026-07-30 22:16:51] START case=long_prefill_latency_128k_c1 rep=1 isl=131072 osl=1 c=1
|
||||
[2026-07-30 22:18:13] DONE case=long_prefill_latency_128k_c1 rep=1 elapsed=81s
|
||||
[2026-07-30 22:18:18] START case=mid_prefill_throughput_32k_c16 rep=1 isl=32768 osl=1 c=16
|
||||
[2026-07-30 22:21:39] DONE case=mid_prefill_throughput_32k_c16 rep=1 elapsed=201s
|
||||
[2026-07-30 22:21:44] START case=decode_latency_1k_to_1k_c1 rep=1 isl=1024 osl=1024 c=1
|
||||
[2026-07-30 22:22:51] DONE case=decode_latency_1k_to_1k_c1 rep=1 elapsed=66s
|
||||
[2026-07-30 22:22:56] START case=decode_throughput_1k_to_1k_c16 rep=1 isl=1024 osl=1024 c=16
|
||||
[2026-07-30 22:24:25] DONE case=decode_throughput_1k_to_1k_c16 rep=1 elapsed=89s
|
||||
[2026-07-30 22:24:30] START case=decode_throughput_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||
[2026-07-30 22:26:15] DONE case=decode_throughput_1k_to_1k_c32 rep=1 elapsed=105s
|
||||
[2026-07-30 22:26:20] START case=decode_throughput_1k_to_1k_c64 rep=1 isl=1024 osl=1024 c=64
|
||||
[2026-07-30 22:28:36] DONE case=decode_throughput_1k_to_1k_c64 rep=1 elapsed=136s
|
||||
[2026-07-30 22:28:41] START case=balanced_32k_to_1k_c8 rep=1 isl=32768 osl=1024 c=8
|
||||
[2026-07-30 22:31:23] DONE case=balanced_32k_to_1k_c8 rep=1 elapsed=161s
|
||||
[2026-07-30 22:31:28] Fixed quick map complete: completed=9/9 failed=0
|
||||
[2026-07-30 22:31:28] START case=decode_control_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||
[2026-07-30 22:34:26] DONE case=decode_control_1k_to_1k_c32 rep=1 elapsed=178s
|
||||
[2026-07-30 22:34:26] START mixed background rep=1; injection delay=10s
|
||||
[2026-07-30 22:34:26] START case=decode_with_128k_prefill_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||
[2026-07-30 22:35:06] START case=long_prefill_injection_128k_to_1_c1 rep=1 isl=131072 osl=1 c=1
|
||||
[2026-07-30 22:36:24] DONE case=long_prefill_injection_128k_to_1_c1 rep=1 elapsed=78s
|
||||
[2026-07-30 22:38:09] DONE case=decode_with_128k_prefill_1k_to_1k_c32 rep=1 elapsed=223s
|
||||
[2026-07-30 22:38:14] Mixed-interference A/B complete: failed_repetitions=0
|
||||
[2026-07-30 22:38:20] Stopped head container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head on 10.101.0.11
|
||||
[2026-07-30 22:38:22] Stopped worker container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker on 10.101.0.13
|
||||
summarized 12 case runs into /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916
|
||||
[2026-07-30 22:38:22] Quick performance map complete: status=COMPLETED
|
||||
[2026-07-30 22:38:22] Results: /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916
|
||||
@ -0,0 +1,43 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"workflow_stage": "quick_performance_map",
|
||||
"run_id": "dsv4pro-phase1-full-20260730-220916",
|
||||
"status": "COMPLETED",
|
||||
"started_at": "2026-07-30T22:09:47+08:00",
|
||||
"updated_at": "2026-07-30T22:38:22+08:00",
|
||||
"suites": [
|
||||
"fixed",
|
||||
"mixed"
|
||||
],
|
||||
"engine": "sglang",
|
||||
"model_name": "DeepSeek-V4-Pro",
|
||||
"model_path": "/data/hf_models/DeepSeek-V4-Pro",
|
||||
"docker_image": "lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45",
|
||||
"head_node": "10.101.0.11",
|
||||
"worker_node": "10.101.0.13",
|
||||
"head_ip": "10.101.0.11",
|
||||
"sglang_port": 30002,
|
||||
"dist_init_port": 20002,
|
||||
"tp_size": 16,
|
||||
"ep_size": 2,
|
||||
"nnodes": 2,
|
||||
"mem_fraction_static": 0.9,
|
||||
"cuda_graph_max_bs_decode": 64,
|
||||
"max_running_requests": 256,
|
||||
"nccl_socket_ifname": "eth0",
|
||||
"nccl_ib_hca": "=mlx5_0:1,mlx5_3:1",
|
||||
"nccl_cross_nic": "1",
|
||||
"enable_rdma": true,
|
||||
"require_nccl_ib": true,
|
||||
"rdma_device_paths": "/dev/infiniband/rdma_cm,/dev/infiniband/uverbs0,/dev/infiniband/uverbs3",
|
||||
"git_commit": "75182c6ededf94511c6678552aaf489b9fc6fba6",
|
||||
"git_dirty": false,
|
||||
"scenario_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv",
|
||||
"case_ids": "",
|
||||
"notes": [
|
||||
"The fixed quick map does not stop on SLO.",
|
||||
"Profiler is disabled; these results are eligible for performance comparison.",
|
||||
"Speculative decoding is not enabled."
|
||||
],
|
||||
"ended_at": "2026-07-30T22:38:22+08:00"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -0,0 +1,13 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,balanced_32k_to_1k_c8,,balanced,1,32768,1024,8,8,0,COMPLETED,,0,2026-07-30T22:28:41+0800,2026-07-30T22:31:22+0800,161.0,8,0,128.62811431800947,7.996993535425267,,262144,8192,0.06219480120979978,2037.9992460427193,63.68747643883498,2101.686722481554,,128579.77483438299,128578.26417451724,128588.8987001963,128589.40238882786,49665.467494261975,49563.69844998699,82083.64989476977,83746.94262376754,77.14008537646238,77.23808966229743,109.32309288714016,112.17132956471158,77.14005810044779,43.40646349010058,43.84165157971437,45.188912986195646,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1,1024,1024,32,64,1,COMPLETED,,0,2026-07-30T22:31:28+0800,2026-07-30T22:34:26+0800,178.0,64,0,143.81865712802391,31.98259215252821,,65536,65536,0.44500485040010307,455.68496680970554,455.68496680970554,911.3699336194111,,71870.21023202942,71867.41446750239,72007.63789927005,72012.73389439622,7370.452653370194,6938.839260517852,9442.636363586644,9443.245556704933,63.04961640142642,63.471088219448355,65.88327712923635,67.80910156527888,63.049588087913975,60.98580302204937,61.753195300116204,63.56921844591854,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,decode_latency_1k_to_1k_c1,,decode_latency,1,1024,1024,1,1,1,COMPLETED,,0,2026-07-30T22:21:44+0800,2026-07-30T22:22:50+0800,66.0,1,0,32.59594450396253,0.9987527008467322,,1024,1024,0.03067866310419184,31.414951018692445,31.414951018692445,62.82990203738489,,32555.28760998277,32555.28760998277,32555.28760998277,32555.28760998277,363.2723209448159,363.2723209448159,363.2723209448159,363.2723209448159,31.468245639333286,31.468245639333286,31.468245639333286,31.468245639333286,31.468210095789587,31.49253799347207,31.659404194215313,32.3411487112753,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c16,,decode_throughput,1,1024,1024,16,16,1,COMPLETED,,0,2026-07-30T22:22:56+0800,2026-07-30T22:24:25+0800,89.0,16,0,55.48426674498478,15.986115557256861,,16384,16384,0.28837003602370287,295.29091688827174,295.29091688827174,590.5818337765435,,55436.11873718692,55437.53465998452,55443.82167501317,55447.12010782387,4706.02819519263,4801.126986538293,4950.267374995747,4950.924708603998,49.58953132159755,49.49641186020903,50.01878498413362,50.915034893211796,49.58950283761395,49.32670452399179,49.8873026604997,52.25673661392648,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1,1024,1024,32,32,1,COMPLETED,,0,2026-07-30T22:24:30+0800,2026-07-30T22:26:15+0800,105.0,32,0,70.97485623596003,31.975931620502816,,32768,32768,0.4508638931738606,461.68462661003326,461.68462661003326,923.3692532200665,,70921.47343050237,70921.31606096518,70933.18627287517,70935.98751158977,6650.527383468216,6178.440199000761,8021.852347906679,8023.242657405207,62.825949215087164,63.29258629570958,63.30661669023356,65.29997366317116,62.82592070965087,61.42424049903639,62.237933481810614,64.6031521842815,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c64,,decode_throughput,1,1024,1024,64,64,1,COMPLETED,,0,2026-07-30T22:26:20+0800,2026-07-30T22:28:36+0800,136.0,64,0,101.22691119497176,63.94603747793439,,65536,65536,0.6322429405825737,647.4167711565555,647.4167711565555,1294.833542313111,,101141.56026639376,101148.19569600513,101163.1970004586,101167.44335912867,7978.397482784203,6505.179281026358,12716.491229846724,12719.49685954547,91.06858532122146,92.52174521165054,93.44459645826957,94.39186767905878,91.06855870121748,86.24391999910586,87.80462378927041,90.7129056059057,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,1024,1024,32,64,0,COMPLETED,,0,2026-07-30T22:34:26+0800,2026-07-30T22:38:09+0800,223.0,64,0,189.43961485201726,31.98654958092739,,65536,65536,0.3378385246929174,345.94664928554744,345.94664928554744,691.8932985710949,,94679.9942664975,94677.34680901049,117629.63820034638,117636.8834006862,7749.917391879535,7705.110649490962,10193.896457596566,10196.455221220385,84.97563721859039,86.47172644327328,109.7299457727384,110.56891953740994,84.97560842059728,60.77706499490887,61.61956858995836,64.70431641500909,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,131072,1,1,1,0,COMPLETED,,0,2026-07-30T22:35:06+0800,2026-07-30T22:36:24+0800,78.0,1,0,45.83367280900711,0.9982227218104723,,131072,1,0.021818020217735695,2859.731545979053,0.021818020217735695,2859.7533639992707,,45752.21362197772,45752.21362197772,45752.21362197772,45752.21362197772,45752.13837000774,45752.13837000774,45752.13837000774,45752.13837000774,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,long_prefill_latency_128k_c1,,prefill_latency,1,131072,1,1,1,0,COMPLETED,,0,2026-07-30T22:16:51+0800,2026-07-30T22:18:12+0800,81.0,1,0,48.363155752944294,0.9996148667368967,,131072,1,0.020676897204730506,2710.162270418437,0.020676897204730506,2710.1829473156417,,48344.52949295519,48344.52949295519,48344.52949295519,48344.52949295519,48344.461318978574,48344.461318978574,48344.461318978574,48344.461318978574,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,mid_prefill_latency_32k_c1,,prefill_latency,1,32768,1,1,1,0,COMPLETED,,0,2026-07-30T22:16:00+0800,2026-07-30T22:16:46+0800,46.0,1,0,12.352426828991156,0.9985701137731423,,32768,1,0.08095575175988892,2652.75807366804,0.08095575175988892,2652.8390294198,,12334.764264000114,12334.764264000114,12334.764264000114,12334.764264000114,12334.68782599084,12334.68782599084,12334.68782599084,12334.68782599084,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,1,32768,1,16,16,0,COMPLETED,,0,2026-07-30T22:18:18+0800,2026-07-30T22:21:39+0800,201.0,16,0,168.4313552599633,8.771898294558559,,524288,16,0.09499418902914482,3112.7695861070174,0.09499418902914482,3112.8645802960464,,92341.41987219118,92749.04042950948,162087.50443853205,167117.798359727,92341.37187193119,92748.99350100895,162087.46085499297,167117.75140058598,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,short_prefill_latency_1k_c1,,prefill_latency,1,1024,1,1,1,1,COMPLETED,,0,2026-07-30T22:15:21+0800,2026-07-30T22:15:55+0800,34.0,1,0,0.5198853989713825,0.9654920680740482,,1024,1,1.9235008368739468,1969.6648569589215,1.9235008368739468,1971.5883577957954,,501.94522901438177,501.94522901438177,501.94522901438177,501.94522901438177,501.89953204244375,501.89953204244375,501.89953204244375,501.89953204244375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.log
|
||||
|
@ -0,0 +1,12 @@
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "balanced_32k_to_1k_c8", "role": "", "stage": "balanced", "repetition": 1, "isl": 32768, "osl": 1024, "concurrency": 8, "num_prompts": 8, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:28:41+0800", "ended_at": "2026-07-30T22:31:22+0800", "elapsed_s": 161.0, "completed": 8, "failed": 0, "duration_s": 128.62811431800947, "actual_concurrency": 7.996993535425267, "peak_concurrent_requests": null, "total_input_tokens": 262144, "total_output_tokens": 8192, "request_throughput": 0.06219480120979978, "input_token_throughput": 2037.9992460427193, "output_token_throughput": 63.68747643883498, "total_token_throughput": 2101.686722481554, "peak_output_token_throughput": null, "e2e_mean_ms": 128579.77483438299, "e2e_p50_ms": 128578.26417451724, "e2e_p95_ms": 128588.8987001963, "e2e_p99_ms": 128589.40238882786, "ttft_mean_ms": 49665.467494261975, "ttft_p50_ms": 49563.69844998699, "ttft_p95_ms": 82083.64989476977, "ttft_p99_ms": 83746.94262376754, "tpot_mean_ms": 77.14008537646238, "tpot_p50_ms": 77.23808966229743, "tpot_p95_ms": 109.32309288714016, "tpot_p99_ms": 112.17132956471158, "itl_mean_ms": 77.14005810044779, "itl_p50_ms": 43.40646349010058, "itl_p95_ms": 43.84165157971437, "itl_p99_ms": 45.188912986195646, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "decode_control_1k_to_1k_c32", "role": "control", "stage": "mixed_interference", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 64, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:31:28+0800", "ended_at": "2026-07-30T22:34:26+0800", "elapsed_s": 178.0, "completed": 64, "failed": 0, "duration_s": 143.81865712802391, "actual_concurrency": 31.98259215252821, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.44500485040010307, "input_token_throughput": 455.68496680970554, "output_token_throughput": 455.68496680970554, "total_token_throughput": 911.3699336194111, "peak_output_token_throughput": null, "e2e_mean_ms": 71870.21023202942, "e2e_p50_ms": 71867.41446750239, "e2e_p95_ms": 72007.63789927005, "e2e_p99_ms": 72012.73389439622, "ttft_mean_ms": 7370.452653370194, "ttft_p50_ms": 6938.839260517852, "ttft_p95_ms": 9442.636363586644, "ttft_p99_ms": 9443.245556704933, "tpot_mean_ms": 63.04961640142642, "tpot_p50_ms": 63.471088219448355, "tpot_p95_ms": 65.88327712923635, "tpot_p99_ms": 67.80910156527888, "itl_mean_ms": 63.049588087913975, "itl_p50_ms": 60.98580302204937, "itl_p95_ms": 61.753195300116204, "itl_p99_ms": 63.56921844591854, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_latency_1k_to_1k_c1", "role": "", "stage": "decode_latency", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 1, "num_prompts": 1, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:21:44+0800", "ended_at": "2026-07-30T22:22:50+0800", "elapsed_s": 66.0, "completed": 1, "failed": 0, "duration_s": 32.59594450396253, "actual_concurrency": 0.9987527008467322, "peak_concurrent_requests": null, "total_input_tokens": 1024, "total_output_tokens": 1024, "request_throughput": 0.03067866310419184, "input_token_throughput": 31.414951018692445, "output_token_throughput": 31.414951018692445, "total_token_throughput": 62.82990203738489, "peak_output_token_throughput": null, "e2e_mean_ms": 32555.28760998277, "e2e_p50_ms": 32555.28760998277, "e2e_p95_ms": 32555.28760998277, "e2e_p99_ms": 32555.28760998277, "ttft_mean_ms": 363.2723209448159, "ttft_p50_ms": 363.2723209448159, "ttft_p95_ms": 363.2723209448159, "ttft_p99_ms": 363.2723209448159, "tpot_mean_ms": 31.468245639333286, "tpot_p50_ms": 31.468245639333286, "tpot_p95_ms": 31.468245639333286, "tpot_p99_ms": 31.468245639333286, "itl_mean_ms": 31.468210095789587, "itl_p50_ms": 31.49253799347207, "itl_p95_ms": 31.659404194215313, "itl_p99_ms": 32.3411487112753, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c16", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 16, "num_prompts": 16, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:22:56+0800", "ended_at": "2026-07-30T22:24:25+0800", "elapsed_s": 89.0, "completed": 16, "failed": 0, "duration_s": 55.48426674498478, "actual_concurrency": 15.986115557256861, "peak_concurrent_requests": null, "total_input_tokens": 16384, "total_output_tokens": 16384, "request_throughput": 0.28837003602370287, "input_token_throughput": 295.29091688827174, "output_token_throughput": 295.29091688827174, "total_token_throughput": 590.5818337765435, "peak_output_token_throughput": null, "e2e_mean_ms": 55436.11873718692, "e2e_p50_ms": 55437.53465998452, "e2e_p95_ms": 55443.82167501317, "e2e_p99_ms": 55447.12010782387, "ttft_mean_ms": 4706.02819519263, "ttft_p50_ms": 4801.126986538293, "ttft_p95_ms": 4950.267374995747, "ttft_p99_ms": 4950.924708603998, "tpot_mean_ms": 49.58953132159755, "tpot_p50_ms": 49.49641186020903, "tpot_p95_ms": 50.01878498413362, "tpot_p99_ms": 50.915034893211796, "itl_mean_ms": 49.58950283761395, "itl_p50_ms": 49.32670452399179, "itl_p95_ms": 49.8873026604997, "itl_p99_ms": 52.25673661392648, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c32", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 32, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:24:30+0800", "ended_at": "2026-07-30T22:26:15+0800", "elapsed_s": 105.0, "completed": 32, "failed": 0, "duration_s": 70.97485623596003, "actual_concurrency": 31.975931620502816, "peak_concurrent_requests": null, "total_input_tokens": 32768, "total_output_tokens": 32768, "request_throughput": 0.4508638931738606, "input_token_throughput": 461.68462661003326, "output_token_throughput": 461.68462661003326, "total_token_throughput": 923.3692532200665, "peak_output_token_throughput": null, "e2e_mean_ms": 70921.47343050237, "e2e_p50_ms": 70921.31606096518, "e2e_p95_ms": 70933.18627287517, "e2e_p99_ms": 70935.98751158977, "ttft_mean_ms": 6650.527383468216, "ttft_p50_ms": 6178.440199000761, "ttft_p95_ms": 8021.852347906679, "ttft_p99_ms": 8023.242657405207, "tpot_mean_ms": 62.825949215087164, "tpot_p50_ms": 63.29258629570958, "tpot_p95_ms": 63.30661669023356, "tpot_p99_ms": 65.29997366317116, "itl_mean_ms": 62.82592070965087, "itl_p50_ms": 61.42424049903639, "itl_p95_ms": 62.237933481810614, "itl_p99_ms": 64.6031521842815, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c64", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 64, "num_prompts": 64, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:26:20+0800", "ended_at": "2026-07-30T22:28:36+0800", "elapsed_s": 136.0, "completed": 64, "failed": 0, "duration_s": 101.22691119497176, "actual_concurrency": 63.94603747793439, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.6322429405825737, "input_token_throughput": 647.4167711565555, "output_token_throughput": 647.4167711565555, "total_token_throughput": 1294.833542313111, "peak_output_token_throughput": null, "e2e_mean_ms": 101141.56026639376, "e2e_p50_ms": 101148.19569600513, "e2e_p95_ms": 101163.1970004586, "e2e_p99_ms": 101167.44335912867, "ttft_mean_ms": 7978.397482784203, "ttft_p50_ms": 6505.179281026358, "ttft_p95_ms": 12716.491229846724, "ttft_p99_ms": 12719.49685954547, "tpot_mean_ms": 91.06858532122146, "tpot_p50_ms": 92.52174521165054, "tpot_p95_ms": 93.44459645826957, "tpot_p99_ms": 94.39186767905878, "itl_mean_ms": 91.06855870121748, "itl_p50_ms": 86.24391999910586, "itl_p95_ms": 87.80462378927041, "itl_p99_ms": 90.7129056059057, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "decode_with_128k_prefill_1k_to_1k_c32", "role": "decode_background", "stage": "mixed_interference", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 64, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:34:26+0800", "ended_at": "2026-07-30T22:38:09+0800", "elapsed_s": 223.0, "completed": 64, "failed": 0, "duration_s": 189.43961485201726, "actual_concurrency": 31.98654958092739, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.3378385246929174, "input_token_throughput": 345.94664928554744, "output_token_throughput": 345.94664928554744, "total_token_throughput": 691.8932985710949, "peak_output_token_throughput": null, "e2e_mean_ms": 94679.9942664975, "e2e_p50_ms": 94677.34680901049, "e2e_p95_ms": 117629.63820034638, "e2e_p99_ms": 117636.8834006862, "ttft_mean_ms": 7749.917391879535, "ttft_p50_ms": 7705.110649490962, "ttft_p95_ms": 10193.896457596566, "ttft_p99_ms": 10196.455221220385, "tpot_mean_ms": 84.97563721859039, "tpot_p50_ms": 86.47172644327328, "tpot_p95_ms": 109.7299457727384, "tpot_p99_ms": 110.56891953740994, "itl_mean_ms": 84.97560842059728, "itl_p50_ms": 60.77706499490887, "itl_p95_ms": 61.61956858995836, "itl_p99_ms": 64.70431641500909, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "long_prefill_injection_128k_to_1_c1", "role": "prefill_injection", "stage": "mixed_interference", "repetition": 1, "isl": 131072, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:35:06+0800", "ended_at": "2026-07-30T22:36:24+0800", "elapsed_s": 78.0, "completed": 1, "failed": 0, "duration_s": 45.83367280900711, "actual_concurrency": 0.9982227218104723, "peak_concurrent_requests": null, "total_input_tokens": 131072, "total_output_tokens": 1, "request_throughput": 0.021818020217735695, "input_token_throughput": 2859.731545979053, "output_token_throughput": 0.021818020217735695, "total_token_throughput": 2859.7533639992707, "peak_output_token_throughput": null, "e2e_mean_ms": 45752.21362197772, "e2e_p50_ms": 45752.21362197772, "e2e_p95_ms": 45752.21362197772, "e2e_p99_ms": 45752.21362197772, "ttft_mean_ms": 45752.13837000774, "ttft_p50_ms": 45752.13837000774, "ttft_p95_ms": 45752.13837000774, "ttft_p99_ms": 45752.13837000774, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "long_prefill_latency_128k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 131072, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:16:51+0800", "ended_at": "2026-07-30T22:18:12+0800", "elapsed_s": 81.0, "completed": 1, "failed": 0, "duration_s": 48.363155752944294, "actual_concurrency": 0.9996148667368967, "peak_concurrent_requests": null, "total_input_tokens": 131072, "total_output_tokens": 1, "request_throughput": 0.020676897204730506, "input_token_throughput": 2710.162270418437, "output_token_throughput": 0.020676897204730506, "total_token_throughput": 2710.1829473156417, "peak_output_token_throughput": null, "e2e_mean_ms": 48344.52949295519, "e2e_p50_ms": 48344.52949295519, "e2e_p95_ms": 48344.52949295519, "e2e_p99_ms": 48344.52949295519, "ttft_mean_ms": 48344.461318978574, "ttft_p50_ms": 48344.461318978574, "ttft_p95_ms": 48344.461318978574, "ttft_p99_ms": 48344.461318978574, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "mid_prefill_latency_32k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 32768, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:16:00+0800", "ended_at": "2026-07-30T22:16:46+0800", "elapsed_s": 46.0, "completed": 1, "failed": 0, "duration_s": 12.352426828991156, "actual_concurrency": 0.9985701137731423, "peak_concurrent_requests": null, "total_input_tokens": 32768, "total_output_tokens": 1, "request_throughput": 0.08095575175988892, "input_token_throughput": 2652.75807366804, "output_token_throughput": 0.08095575175988892, "total_token_throughput": 2652.8390294198, "peak_output_token_throughput": null, "e2e_mean_ms": 12334.764264000114, "e2e_p50_ms": 12334.764264000114, "e2e_p95_ms": 12334.764264000114, "e2e_p99_ms": 12334.764264000114, "ttft_mean_ms": 12334.68782599084, "ttft_p50_ms": 12334.68782599084, "ttft_p95_ms": 12334.68782599084, "ttft_p99_ms": 12334.68782599084, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "mid_prefill_throughput_32k_c16", "role": "", "stage": "prefill_throughput", "repetition": 1, "isl": 32768, "osl": 1, "concurrency": 16, "num_prompts": 16, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:18:18+0800", "ended_at": "2026-07-30T22:21:39+0800", "elapsed_s": 201.0, "completed": 16, "failed": 0, "duration_s": 168.4313552599633, "actual_concurrency": 8.771898294558559, "peak_concurrent_requests": null, "total_input_tokens": 524288, "total_output_tokens": 16, "request_throughput": 0.09499418902914482, "input_token_throughput": 3112.7695861070174, "output_token_throughput": 0.09499418902914482, "total_token_throughput": 3112.8645802960464, "peak_output_token_throughput": null, "e2e_mean_ms": 92341.41987219118, "e2e_p50_ms": 92749.04042950948, "e2e_p95_ms": 162087.50443853205, "e2e_p99_ms": 167117.798359727, "ttft_mean_ms": 92341.37187193119, "ttft_p50_ms": 92748.99350100895, "ttft_p95_ms": 162087.46085499297, "ttft_p99_ms": 167117.75140058598, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "short_prefill_latency_1k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 1024, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:15:21+0800", "ended_at": "2026-07-30T22:15:55+0800", "elapsed_s": 34.0, "completed": 1, "failed": 0, "duration_s": 0.5198853989713825, "actual_concurrency": 0.9654920680740482, "peak_concurrent_requests": null, "total_input_tokens": 1024, "total_output_tokens": 1, "request_throughput": 1.9235008368739468, "input_token_throughput": 1969.6648569589215, "output_token_throughput": 1.9235008368739468, "total_token_throughput": 1971.5883577957954, "peak_output_token_throughput": null, "e2e_mean_ms": 501.94522901438177, "e2e_p50_ms": 501.94522901438177, "e2e_p95_ms": 501.94522901438177, "e2e_p99_ms": 501.94522901438177, "ttft_mean_ms": 501.89953204244375, "ttft_p50_ms": 501.89953204244375, "ttft_p95_ms": 501.89953204244375, "ttft_p99_ms": 501.89953204244375, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.log"}
|
||||
@ -400,7 +400,7 @@
|
||||
<div class="document-header__meta">
|
||||
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||
<span>资源：16 × RTX PRO 6000 Blackwell</span>
|
||||
<span>版本：2026-07-30 17:54 CST</span>
|
||||
<span>版本：2026-07-30 22:55:37 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
@ -415,7 +415,7 @@
|
||||
<article id="document-content">
|
||||
<h1>6000D 双机 DeepSeek-V4-Pro 推理优化计划</h1>
|
||||
<blockquote>
|
||||
<p>适用环境：<code>174.1.51.5 + 174.1.51.7</code>，每台 8 张 RTX PRO 6000 Blackwell Server Edition<br>当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例<br>当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离<br>计划版本：2026-07-30 18:40 CST</p>
|
||||
<p>适用环境：<code>174.1.51.5 + 174.1.51.7</code>，每台 8 张 RTX PRO 6000 Blackwell Server Edition<br>当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例<br>当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离<br>计划版本：2026-07-30 22:55:37 CST</p>
|
||||
</blockquote>
|
||||
<h2>当前执行状态与阶段档案</h2>
|
||||
<table>
|
||||
@ -434,12 +434,12 @@
|
||||
</tr>
|
||||
<tr>
|
||||
<td>DeepSeek-V4-Pro / 双机 Pro6000D / SGLang TP16 快速性能地图</td>
|
||||
<td>网络错误已定位；RDMA fail-closed 代码与 dry-run 已完成，待真机 NET/IB 验证后重跑</td>
|
||||
<td>已完成；双 Rail NET/IB + GDRDMA，固定点 9/9、混合 A/B 3/3，总用时 28 分 36 秒</td>
|
||||
<td><a href="./phase1_dsv4pro_pro6000d_2node_sglang_quick_map.html">打开实施记录</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>DeepSeek-V4-Pro / 双机 Pro6000D / SGLang Prefill 硬件指标归因</td>
|
||||
<td>设计已固化、代码尚未开始；等待修正后的 Phase 1 基线</td>
|
||||
<td>Phase 1 已交接，三个诊断负载已选定；等待阶段确认后开始代码实现</td>
|
||||
<td><a href="./phase2_dsv4pro_pro6000d_2node_sglang_prefill_hardware_attribution.html">打开 Phase 2 档案</a></td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
@ -596,18 +596,17 @@ Nsight Compute 或专项 Microbenchmark
|
||||
<a href="./phase1_dsv4pro_pro6000d_2node_sglang_quick_map.html">Phase 1：双机 SGLang TP16 快速性能地图实施记录</a>。
|
||||
代码采用单一 Shell 入口 <code>run_quick_map.sh</code>，不修改或调用旧的全天全量脚本。</p>
|
||||
<p>
|
||||
最终 Run 完成 1K、32K、128K 三个单请求 Prefill 点，输入吞吐分别为
|
||||
64.44、64.96、65.20 token/s。第 4 个 <code>32K, C=16</code> 点在稳定复现
|
||||
单序列 Chunk 推进后被主动中止；Manifest 状态为
|
||||
<code>ABORTED_EARLY_FOR_PHASE2</code>。Decode、Balanced 和混合 A/B 未执行。
|
||||
最终 Run <code>dsv4pro-phase1-full-20260730-220916</code> 已完成：
|
||||
Head 与 Worker 均通过双 Rail <code>NET/IB + GDRDMA</code> 门禁，
|
||||
9 个固定点和 3 个混合 A/B 结果共 12/12 成功，总用时 28 分 36 秒。
|
||||
单请求 32K/128K Prefill 输入吞吐为 2,652.76/2,710.16 token/s；
|
||||
1K → 1K Decode 的 Output TPS 从 C=1 的 31.41 提升到 C=64 的 647.42。
|
||||
</p>
|
||||
<p>
|
||||
旧脚本 TTFT 较短的问题已完成复核。新旧服务端关键运行参数相同；旧脚本固定
|
||||
<code>warmup_requests=16</code>、不清 Prefix Cache、按固定 Seed 递增长度，
|
||||
且正式 Run 前已有一次失败 Run 预热同一批请求。最小复现中，两次清 Cache 的
|
||||
1K 冷 TTFT 分别为 16.04 秒和 15.90 秒，OSL=128 时为 15.79 秒；旧产物的
|
||||
0.455 秒无法在冷缓存口径复现。因此当前 65 token/s 明确解释为完整冷 Prompt
|
||||
路径，Warm Prefix 性能后续单独做 A/B。
|
||||
最值得继续归因的现象有两个：32K Prefill 从 C=1 增至 C=16 时，
|
||||
聚合输入吞吐只提高 17.3%，TTFT P95 却从 12.335 秒增至 162.087 秒；
|
||||
在 C=32 Decode 中注入一个 128K Prefill 后，Output TPS 下降 24.08%，
|
||||
TPOT P95 增加 66.55%。Phase 2 将围绕这两个现象采集硬件时间序列。
|
||||
</p>
|
||||
<h3>5.1 当前固定快速矩阵</h3>
|
||||
<table>
|
||||
@ -692,7 +691,11 @@ Nsight Compute 或专项 Microbenchmark
|
||||
<li>正式测量开始 10 秒后注入一个 <code>128K → 1, C=1</code> 长 Prefill；不额外执行 128K Warm-up。</li>
|
||||
<li>比较 Output TPS、P95 TTFT、P95 TPOT 与 P95 E2E 的变化。</li>
|
||||
</ol>
|
||||
<p>该 A/B 在 Phase 1 中未执行，待 Prefill 异常完成归因后再决定是否重放。这是聚合级干扰探针，不声称具备逐请求时间线归因能力；后者在 Scheduler 与 Timeline 阶段实现。</p>
|
||||
<p>
|
||||
该 A/B 已完成：注入 128K Prefill 后，Decode Output TPS 从 455.68 降至
|
||||
345.95 token/s，TPOT P95 从 65.88 ms 增至 109.73 ms。它证明存在聚合级干扰，
|
||||
但逐请求调度时间线仍由后续阶段补齐。
|
||||
</p>
|
||||
<h3>5.3 本轮运行与停止策略</h3>
|
||||
<ul>
|
||||
<li>固定矩阵不做 Add-16 搜索，也不按 SLO 提前终止。</li>
|
||||
@ -722,10 +725,10 @@ Nsight Compute 或专项 Microbenchmark
|
||||
<p>
|
||||
本阶段的设计、代码改动与结果同步维护在
|
||||
<a href="./phase2_dsv4pro_pro6000d_2node_sglang_prefill_hardware_attribution.html">
|
||||
Phase 2：Prefill 硬件指标归因档案</a>。第一轮只重放
|
||||
<code>32K → 1, C=1</code>，目标是在约 20 分钟内区分 GPU、CPU、双 Rail、
|
||||
频率节流和节点不均衡。该请求在测量前清 Prefix Cache；旧脚本中的热缓存
|
||||
TTFT 不作为本阶段参照值。
|
||||
Phase 2：Prefill 硬件指标归因档案</a>。本阶段只重放三类代表负载：
|
||||
<code>128K → 1, C=1</code>、<code>32K → 1, C=16</code>，以及
|
||||
<code>1K → 1K, C=32</code> 的混合 A/B。目标是在约 30 分钟内区分 GPU、
|
||||
CPU、双 Rail、调度排队、频率节流和节点不均衡。
|
||||
</p>
|
||||
<h3>6.1 GPU</h3>
|
||||
<p>测试期间持续记录：</p>
|
||||
@ -762,19 +765,11 @@ numastat -p <PID>
|
||||
<li>CPU 空洞是否对应 GPU 空洞。</li>
|
||||
</ul>
|
||||
<h3>6.3 网络</h3>
|
||||
<blockquote>
|
||||
<p>
|
||||
2026-07-30 控制组已确认：宿主机具备 <code>mlx5_0/eth0</code> 与
|
||||
<code>mlx5_3/eth3</code> 两条 400G Rail，但原服务容器没有
|
||||
<code>/dev/infiniband</code>，NCCL 实际使用 <code>NET/Socket</code>。
|
||||
当前第一优先级是校正容器数据面并重做 Phase 1，不再把约 65 token/s 当作模型或算子瓶颈。
|
||||
</p>
|
||||
</blockquote>
|
||||
<p>
|
||||
唯一启动入口现已在两端预检并透传
|
||||
<code>rdma_cm/uverbs0/uverbs3</code>，且服务健康后强制从两端 NCCL INFO
|
||||
日志确认 <code>NET/IB</code> 同时识别 <code>mlx5_0/mlx5_3</code>。代码、
|
||||
单测与 dry-run 已通过；真机服务尚未启动，因此这里仍不宣称 RDMA 已验证成功。
|
||||
Phase 1 正式 Run 已从两端 NCCL 日志确认
|
||||
<code>NET/IB</code> 同时识别 <code>mlx5_0/mlx5_3</code>，跨节点 Channel
|
||||
使用 <code>GDRDMA</code>。Phase 2 继续保留相同 fail-closed 门禁，并采集两条
|
||||
Rail 的流量和错误计数。
|
||||
</p>
|
||||
<p>当前拓扑中需要分别观察两条 Compute Rail，确认：</p>
|
||||
<ul>
|
||||
@ -795,10 +790,9 @@ NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING
|
||||
<p>该日志开销较高，不应在正式性能结果中长期启用。</p>
|
||||
<h3>6.4 NCCL_CROSS_NIC 快速 A/B</h3>
|
||||
<p>
|
||||
旧 Phase 1 虽然设置了 <code>NCCL_CROSS_NIC=1</code>，但由于 NCCL 回退
|
||||
<code>NET/Socket</code>，该参数没有参与实际路径选择。先让容器真正使用
|
||||
<code>mlx5_0/mlx5_3</code> 的 <code>NET/IB</code>，再固定其余环境比较
|
||||
<code>0/1/2</code>，不能仅凭双 Rail 拓扑判断最优值。
|
||||
Phase 1 已证明 <code>NCCL_CROSS_NIC=1</code> 可以稳定完成端到端 Run，
|
||||
但这不代表它是性能最优值。固定其余环境后比较 <code>0/1/2</code>，
|
||||
不能仅凭双 Rail 拓扑判断。
|
||||
</p>
|
||||
<ol>
|
||||
<li>分别执行相同消息范围的 <code>all_reduce_perf</code>，每个值至少重复 3 次，检查错误、algbw 和 busbw。</li>
|
||||
@ -810,8 +804,8 @@ NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING
|
||||
<ul>
|
||||
<li>Phase 1 保留一份不启用 Profiler 的端到端基线，避免 TPS 和时延被诊断工具污染。</li>
|
||||
<li>GPU、CPU 和网络的轻量采样可以伴随后续基线运行，但必须从 Case 开始前启动，并使用统一时间戳与 Case ID 对齐。</li>
|
||||
<li>Phase 1 已提前结束，不把中途人工观察到的 GPU 数值伪装成完整硬件时间序列。</li>
|
||||
<li>Phase 2 首轮独立重放 32K Prefill 并完整采集轻量指标；得到瓶颈方向后，再决定是否把 Phase 2 指标与 Phase 3 的短时间线放在同一诊断 Run。</li>
|
||||
<li>Phase 1 已完整结束，其无 Profiler 结果作为后续 A/B 的请求层基线。</li>
|
||||
<li>Phase 2 重放三个代表负载并完整采集轻量指标；得到瓶颈方向后，再决定 Phase 3 捕获哪段短时间线。</li>
|
||||
<li>联合诊断 Run 的吞吐和时延只用于解释时间线；正式性能变化仍与 Phase 1 的无 Profiler 结果比较。</li>
|
||||
</ul>
|
||||
<h2>7. Phase 3：时间线 Profiling（Nsight Systems 为主）</h2>
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user