Compare commits
3 Commits
405608ad23
...
e1719bd575
| Author | SHA1 | Date | |
|---|---|---|---|
|
e1719bd575 | ||
|
|
c5fa700c50 | ||
|
|
119701a417 |
12
README.md
12
README.md
@ -1,5 +1,17 @@
|
||||
# sskj — 多平台大模型推理性能基准测试项目
|
||||
|
||||
> **更新（2026-08-01 15:24:52 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模。正式 Run `dsv4pro-phase2_5-20260801-130007` 完成 Scout 5/5 与 Confirm 6/6；64K Prefill 在 C=16 已进入约 2,984 input tok/s、79.90 Gbit/s/rail 的平台，C=32/64 不再显著增长。拟合通信强度为 3.332 MB/input-token/rail，单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍，因此当前是模型计算/实现吞吐先饱和，不是 RDMA 先饱和。新增 `phase2_5_exp.html`、`phase2_5_code.html`、精简证据集和可复用的模型部署 RDMA 需求评估流程；实验结束后双节点容器与 16 张 GPU 已清理。
|
||||
>
|
||||
> **更新（2026-08-01 02:40:00 CST）**
|
||||
>
|
||||
> 新增 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模唯一入口。实验保持现有 TP16/EP2 服务参数不变，先以 `64K -> 1` 的 C=1/4/16/32/64 建立 Input TPS 与每 Rail HCA 带宽关系，再自动选择平台前、拐点和最大稳定并发，对 `64K -> 1K` 重复确认。结果将给出每 Token 跨机字节数、400G 所需 Token TPS、并发饱和曲线和“模型计算先饱和还是 RDMA 先饱和”的机器可读结论；正式结果尚未生成，因此暂不创建 Phase 2.5 HTML 档案。
|
||||
>
|
||||
> **更新（2026-07-31 19:02:00 CST）**
|
||||
>
|
||||
> Phase 3 Nsight smoke 已在 Head/Worker 各生成一份可解析报告；首份时间线确认 NCCL AllReduce、Sparse MLA、CUTLASS GEMM 与 MoE kernel 均可见。根据 smoke 修正正式 capture：Decode 对照和混合负载必须等服务实际进入 `Decode batch` 后才触发；每段等待本节点全部 8 个 rank 完成，而非任一 rank；双节点近同步停止以减少 Gloo 断链噪声。`nsys stats` 改为可写临时 SQLite、保留 Kernel/API/NVTX 高价值汇总并删除临时库，结果与报告数量均 fail-closed。
|
||||
>
|
||||
> **更新（2026-07-31 18:53:00 CST）**
|
||||
>
|
||||
> Phase 3 PyTorch Profiler 首轮 smoke 已证明双节点 16 个 rank 均可完成 trace 写盘，Head/Worker 分别生成 8 份、约 1.5/1.4 GB 压缩 trace。修复当前 OpenSSH 不接受 `scp remote:/path/.` 导致 Worker 结果未回收的问题，改为 SSH tar 流式传输；正式入口不再忽略回收失败，并新增至少 16 份 PyTorch rank trace、至少 2 份 Nsight 节点报告的结果门禁。
|
||||
|
||||
@ -547,6 +547,26 @@ mlx5_3 port 1 ==> eth3 (Up)</code></pre>
|
||||
<td>查看系统或进程在各 NUMA 节点上的内存分布。</td>
|
||||
<td>Phase 2 每 5 秒保存结构化 Node0/Node1 MiB，寻找跨 NUMA 内存放置。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>HCA Counter</strong></td>
|
||||
<td>网卡硬件维护的发送、接收、等待、丢弃和错误累计计数器。</td>
|
||||
<td>Phase 2.5 用 <code>mlx5_0/mlx5_3</code> 的 counter 差值计算正式 benchmark 窗口内的 RDMA Gbit/s。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>bytes/input-token/rail</strong></td>
|
||||
<td>模型每处理一个输入 token，平均要在一条 Rail 上发送的字节数。</td>
|
||||
<td>当前 DSV4-Pro TP16/EP2 Scout 拟合为约 3.332 MB/token/rail；换模型或并行策略必须重新标定。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>带宽平台 / 拐点</strong></td>
|
||||
<td>继续增加并发后，吞吐与网络带宽都几乎不再增长的位置。</td>
|
||||
<td>Phase 2.5 以相邻点的 Input TPS 和 Rail Mean 增益同时低于 5% 判断，当前拐点为 C=16。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>渐近线 / 饱和上限</strong></td>
|
||||
<td>饱和曲线在并发继续增大时逼近、但不会明显超过的预测上限。</td>
|
||||
<td>当前 64K Prefill 的拟合上限约 80.32 Gbit/s/rail，表示模型产流量上限，不表示网卡硬件只能跑 80G。</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
167
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_code.html
Normal file
167
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_code.html
Normal file
@ -0,0 +1,167 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Phase 2.5 Code：DSV4-Pro 双机 SGLang RDMA 需求建模</title>
|
||||
<style>
|
||||
:root { --canvas:#eef3f4; --paper:#fff; --ink:#182126; --muted:#5a6970; --line:#d4dee1; --navy:#17363d; --teal:#087c72; --teal-soft:#e8f5f3; --amber:#a64c14; --amber-soft:#fff1e7; --code-bg:#17252b; --code-ink:#eaf2f3; }
|
||||
* { box-sizing:border-box; letter-spacing:0; }
|
||||
body { margin:0; color:var(--ink); background:var(--canvas); font-family:"PingFang SC","Microsoft YaHei",Arial,sans-serif; font-size:16px; line-height:1.72; }
|
||||
header { color:#f6fbfb; background:var(--navy); border-bottom:5px solid #d2692b; }
|
||||
.header-inner,main { width:min(100% - 36px,1120px); margin:0 auto; }
|
||||
.header-inner { padding:34px 0 30px; }
|
||||
h1 { margin:0; font-size:clamp(28px,4vw,42px); line-height:1.25; }
|
||||
h2 { margin:42px 0 15px; padding-bottom:8px; font-size:25px; border-bottom:2px solid #adbbc0; }
|
||||
h3 { margin:28px 0 10px; color:#21454d; font-size:19px; }
|
||||
.eyebrow { margin:0 0 6px; color:#9edbd5; font-size:13px; font-weight:700; }
|
||||
.meta { margin-top:15px; color:#d6e5e7; font-size:14px; }
|
||||
main { margin-top:30px; margin-bottom:70px; padding:38px 48px 58px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
|
||||
a { color:var(--teal); }
|
||||
code { padding:2px 5px; color:#85380d; background:var(--amber-soft); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; overflow-wrap:anywhere; }
|
||||
pre { margin:14px 0 22px; padding:16px 18px; overflow:auto; color:var(--code-ink); background:var(--code-bg); border-radius:5px; font:13px/1.62 "SFMono-Regular",Consolas,monospace; }
|
||||
pre code { padding:0; color:inherit; background:transparent; }
|
||||
table { width:100%; margin:16px 0 26px; border-collapse:collapse; font-size:14px; }
|
||||
th,td { padding:9px 11px; vertical-align:top; text-align:left; border:1px solid var(--line); overflow-wrap:anywhere; }
|
||||
th { color:#153b41; background:#eaf2f2; }
|
||||
.callout { margin:18px 0 26px; padding:14px 18px; background:var(--teal-soft); border-left:4px solid var(--teal); }
|
||||
.warning { margin:18px 0 26px; padding:14px 18px; background:var(--amber-soft); border-left:4px solid var(--amber); }
|
||||
li+li { margin-top:5px; }
|
||||
@media(max-width:760px){main{padding:28px 20px 42px}table{display:block;overflow-x:auto}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">STANDALONE CODE WALKTHROUGH / PHASE 2.5</p>
|
||||
<h1>DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模：代码详解</h1>
|
||||
<div class="meta">实现提交：<code>c5fa700c50c0</code> 正式 Run：<code>dsv4pro-phase2_5-20260801-130007</code> 唯一入口：<code>run_rdma_demand_modeling.sh all</code></div>
|
||||
</div>
|
||||
</header>
|
||||
<main>
|
||||
<p><a href="./推理优化计划.html">返回推理优化主计划</a> · <a href="./phase2_5_exp.html">打开 Phase 2.5 实验档案</a></p>
|
||||
<div class="callout"><strong>边界：</strong>Phase 2.5 不复制模型服务和采集器。它复用 Phase 1 的双机 TP16 服务/benchmark 与 Phase 2 的精确窗口、GPU/RDMA 采集能力，只新增“并发 Scout → 自动选点 → 业务 OSL Confirm → 需求拟合”这一层编排和分析。</div>
|
||||
|
||||
<h2>1. 文件职责与调用关系</h2>
|
||||
<table>
|
||||
<thead><tr><th>文件</th><th>职责</th><th>调用关系</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>config.env</code></td><td>定义 ISL/OSL、Scout 并发、重复次数、平台阈值和 400G 目标</td><td>被唯一入口 source</td></tr>
|
||||
<tr><td><code>run_rdma_demand_modeling.sh</code></td><td>生成场景、调用 Phase 2、串行运行 Scout/Confirm、清理服务</td><td>唯一人工入口</td></tr>
|
||||
<tr><td><code>rdma_demand_model.py</code></td><td>对齐 benchmark/HCA 窗口，计算 bytes/token，拟合平台并生成报告</td><td>Scout 后选点；Confirm 后最终汇总</td></tr>
|
||||
<tr><td><code>test_rdma_demand_model.py</code></td><td>覆盖 HCA counter 单位、平台选择、线性换算和拟合输出</td><td>本地/CI 回归测试</td></tr>
|
||||
<tr><td>Phase 2 <code>run_hardware_contention_attribution.sh</code></td><td>启动服务、采集 Head/Worker、切正式测量窗</td><td>由 Phase 2.5 以环境变量调用</td></tr>
|
||||
<tr><td>Phase 1 <code>run_quick_map.sh</code></td><td>双机服务启停与 SGLang benchmark</td><td>由 Phase 2 内部复用</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<pre><code>run_rdma_demand_modeling.sh all
|
||||
├─ validate_config + write_manifest
|
||||
├─ run_scout
|
||||
│ ├─ write_scenario_file(64K→1, C=1/4/16/32/64)
|
||||
│ ├─ Phase 2 all（服务 + 18 个采集器 + 5 Case）
|
||||
│ └─ rdma_demand_model.py scout → recommendation.env
|
||||
├─ run_confirm
|
||||
│ ├─ 读取自动选择的 C=4/16/64
|
||||
│ ├─ Phase 2 all（重启服务 + 18 个采集器 + 每点 2 次）
|
||||
│ └─ rdma_demand_model.py final
|
||||
└─ Phase 2 stop → 双节点清理</code></pre>
|
||||
|
||||
<h2>2. 配置层</h2>
|
||||
<p><code>config.env:5-7</code> 通过相对路径找到 Phase 1/2，不依赖执行命令所在目录。<code>config.env:10-16</code> 定义 64K Scout 与 1K Confirm；<code>18-21</code> 定义 5% 平台阈值、400G 物理目标和 360G 实用目标。</p>
|
||||
<pre><code>ISL=65536
|
||||
SCOUT_OSL=1
|
||||
CONFIRM_OSL=1024
|
||||
SCOUT_CONCURRENCIES="1 4 16 32 64"
|
||||
SCOUT_REPETITIONS=1
|
||||
CONFIRM_REPETITIONS=2
|
||||
PLATEAU_GAIN_PCT=5
|
||||
TARGET_RAIL_GBPS=400
|
||||
PRACTICAL_RAIL_GBPS=360</code></pre>
|
||||
<p><code>SAMPLE_INTERVAL_S=1</code> 只决定 HCA/GPU 时间序列分辨率；<code>SCENARIO_TIMEOUT_S=7200</code> 是单个 benchmark 的保护上限，不是期望耗时。</p>
|
||||
|
||||
<h2>3. Shell 唯一入口</h2>
|
||||
<h3>3.1 参数检查与场景生成</h3>
|
||||
<p><code>run_rdma_demand_modeling.sh:42-71</code> fail-fast 检查依赖脚本、整数参数和并发列表。<code>73-100</code> 生成 Phase 2 能读取的 TSV，并为每个形状生成稳定的 case id。</p>
|
||||
<h3>3.2 复用 Phase 2，而不是复制采集代码</h3>
|
||||
<p><code>103-134</code> 构造一个数组命令，把场景、Case、重复次数和采样周期作为环境变量传给 Phase 2。它显式关闭 mixed case 与通信 microbenchmark，因为 Phase 2.5 只测模型 RDMA 需求，不重复已完成的硬件基线。</p>
|
||||
<pre><code>RUN_MIXED_CASE=0
|
||||
RUN_COMMUNICATION_BASELINE=0
|
||||
SCENARIO_FILE=.../scout.tsv
|
||||
FIXED_CASE_IDS=rdma_scout_...
|
||||
bash run_hardware_contention_attribution.sh all</code></pre>
|
||||
<h3>3.3 两阶段控制流</h3>
|
||||
<p><code>136-152</code> 跑完 Scout 后立即调用 Python，并写出 <code>recommendation.env</code>；<code>154-180</code> 读取推荐并发，生成 64K→1K Confirm。<code>217-225</code> 的 <code>run_all</code> 严格串行执行，异常信号触发 stop 清理。</p>
|
||||
<div class="warning"><strong>为什么服务会启动两次：</strong>Scout 结束后 Phase 2 会清理服务；Confirm 使用全新的 Prefix Cache、采集器和服务生命周期，避免 Scout 状态污染确认结果。</div>
|
||||
|
||||
<h2>4. Python 如何从计数器变成需求模型</h2>
|
||||
<h3>4.1 精确时间窗与 HCA 单位</h3>
|
||||
<p><code>rdma_demand_model.py:87-115</code> 以 <code>(case_id,repetition)</code> 对齐 benchmark、窗口和 RDMA 汇总。<code>117-147</code> 在正式窗口内计算相邻 HCA counter 的速率；IB <code>port_*_data</code> 单位是 4-octet，因此必须乘 4，再乘 8 转为 bit/s。</p>
|
||||
<pre><code>gbps = (counter_delta × 4 bytes × 8 bits) / duration_s / 1e9</code></pre>
|
||||
<h3>4.2 单 Case 指标</h3>
|
||||
<p><code>171-281</code> 汇总四条观测边（Head/Worker × 两个 HCA）的 Rail Mean/P95/Max、双 Rail 单向合计、Rail 不均衡、错误计数和 GPU 利用率。通信强度按每条 Rail 平均发送字节计算：</p>
|
||||
<pre><code>bytes_per_input_token_per_rail
|
||||
= mean(head/worker × mlx5_0/mlx5_3 xmit_bytes)
|
||||
/ total_input_tokens</code></pre>
|
||||
<p>这里不把 TX+RX 相加，因为那会把同一份跨机数据重复计数。</p>
|
||||
|
||||
<h3>4.3 平台、拐点与自动选点</h3>
|
||||
<p><code>362-392</code> 比较相邻并发点。只有 Rail Mean 与 Input TPS 增益同时低于 5%，当前点才是平台候选。随后选择平台前一点、平台点和最高稳定点；本 Run 得到 <code>4 16 64</code>。</p>
|
||||
<pre><code>if bandwidth_gain < 5% and input_tps_gain < 5%:
|
||||
plateau_c = current_concurrency</code></pre>
|
||||
|
||||
<h3>4.4 线性通信强度与饱和曲线</h3>
|
||||
<p><code>352-360</code> 用过原点线性斜率拟合 <code>rail_gbps/input_tps</code>，再还原为 bytes/token。<code>318-350</code> 用双曲线 <code>B(C)=B∞×C/(K+C)</code> 拟合并发饱和曲线；<code>394-447</code> 组合两者，判断模型计算或网络谁先到平台。</p>
|
||||
<pre><code>required_input_tps
|
||||
= target_rail_gbps / linear_gbps_per_input_tps
|
||||
|
||||
if fitted_bandwidth_asymptote < 360:
|
||||
verdict = COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION</code></pre>
|
||||
<p>本 Run 的 RMSE 为 0.277 Gbit/s，五个 Scout 点与饱和曲线贴合良好；拟合上限 80.32 Gbit/s，与 C=16/32/64 的 79.90/79.93/79.97 一致。</p>
|
||||
|
||||
<h2>5. 输出文件如何阅读</h2>
|
||||
<table>
|
||||
<thead><tr><th>输出</th><th>用途</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>rdma_case_metrics.csv</code></td><td>每次重复的 benchmark + GPU + Rail 对齐数据，是审计主表</td></tr>
|
||||
<tr><td><code>rdma_demand_model.json</code></td><td>完整拟合参数、平台点、目标 TPS 和最终 verdict</td></tr>
|
||||
<tr><td><code>rdma_demand_report.md</code></td><td>面向人的 Scout/Confirm 摘要</td></tr>
|
||||
<tr><td><code>recommendation.env</code></td><td>Shell 可直接 source 的 Confirm 并发列表</td></tr>
|
||||
<tr><td><code>{scout,confirm}/case_windows.csv</code></td><td>每个正式 benchmark 的精确起止时间</td></tr>
|
||||
<tr><td><code>{scout,confirm}/{head,worker}/rdma.csv</code></td><td>原始 HCA counter 时间序列，仅保留在服务器完整结果中</td></tr>
|
||||
<tr><td><code>commands/*.cmd.txt</code></td><td>实际传给 Phase 2 的完整可复现命令</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>6. 测试与验收门槛</h2>
|
||||
<ul>
|
||||
<li><code>python3 -m unittest test_rdma_demand_model.py</code>：4/4 通过。</li>
|
||||
<li><code>bash -n run_rdma_demand_modeling.sh</code> 与 Python compile：通过。</li>
|
||||
<li><code>DRY_RUN=1 ... all</code>：展开 5 个 Scout 和 3×2 个 Confirm，不启动服务。</li>
|
||||
<li>正式 Run：Scout 5/5、Confirm 6/6；两个阶段各 18/18 个采集器正常启停，共保存 72 条 STARTED/STOPPED 生命周期事件。</li>
|
||||
<li>11 个测量结果全部 <code>COMPLETED</code>，<code>rdma_error_delta=0</code>。</li>
|
||||
<li>结束后 Head/Worker 均无实验容器，16 张 GPU 为 0 MiB / 0%。</li>
|
||||
</ul>
|
||||
|
||||
<h2>7. 行号索引</h2>
|
||||
<table>
|
||||
<thead><tr><th>功能</th><th>文件与行</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>配置与路径</td><td><code>config.env:3-36</code></td></tr>
|
||||
<tr><td>校验与场景生成</td><td><code>run_rdma_demand_modeling.sh:42-100</code></td></tr>
|
||||
<tr><td>Phase 2 调用</td><td><code>run_rdma_demand_modeling.sh:103-134</code></td></tr>
|
||||
<tr><td>Scout / Confirm</td><td><code>run_rdma_demand_modeling.sh:136-180</code></td></tr>
|
||||
<tr><td>唯一 all 与清理</td><td><code>run_rdma_demand_modeling.sh:206-264</code></td></tr>
|
||||
<tr><td>HCA interval rate</td><td><code>rdma_demand_model.py:117-147</code></td></tr>
|
||||
<tr><td>Case 对齐汇总</td><td><code>rdma_demand_model.py:171-281</code></td></tr>
|
||||
<tr><td>饱和拟合</td><td><code>rdma_demand_model.py:318-350</code></td></tr>
|
||||
<tr><td>平台选点</td><td><code>rdma_demand_model.py:362-392</code></td></tr>
|
||||
<tr><td>需求模型与 verdict</td><td><code>rdma_demand_model.py:394-447</code></td></tr>
|
||||
<tr><td>报告输出</td><td><code>rdma_demand_model.py:449-517</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>8. 复用时必须重新标定的边界</h2>
|
||||
<p>这套代码可复用，但 <code>3.332 MB/token/rail</code> 不是通用常数。换模型、量化、TP/EP、节点切分、backend、Prefill/Decode 形状或 Prefix Cache 策略后，都必须重新跑 Scout。代码输出的是“当前部署实现的经验模型”，不是由参数量单独推导出的理论通信量。</p>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
166
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_exp.html
Normal file
166
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_exp.html
Normal file
@ -0,0 +1,166 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Phase 2.5：DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模</title>
|
||||
<style>
|
||||
:root { --ink:#18202a; --muted:#5b6570; --line:#d8dde3; --paper:#fff; --page:#f3f5f7; --blue:#1769aa; --green:#16734a; --amber:#9a5a00; --code:#f0f3f6; }
|
||||
* { box-sizing:border-box; letter-spacing:0; }
|
||||
body { margin:0; background:var(--page); color:var(--ink); font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","PingFang SC",sans-serif; font-size:16px; line-height:1.68; }
|
||||
header { background:#202a34; color:#fff; border-bottom:5px solid #34a17a; }
|
||||
.header-inner, main { width:min(1120px,calc(100% - 32px)); margin:0 auto; }
|
||||
.header-inner { padding:36px 0 30px; }
|
||||
h1 { margin:4px 0 12px; font-size:clamp(27px,4vw,42px); line-height:1.2; }
|
||||
h2 { margin:38px 0 12px; padding-bottom:7px; border-bottom:2px solid var(--line); font-size:24px; }
|
||||
h3 { margin:26px 0 8px; font-size:19px; }
|
||||
.eyebrow { margin:0; color:#8fd8bd; font-size:13px; font-weight:700; }
|
||||
.meta { display:flex; flex-wrap:wrap; gap:8px 22px; color:#d7dee5; font-size:14px; }
|
||||
main { margin-top:24px; margin-bottom:48px; padding:30px 38px 42px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
|
||||
a { color:var(--blue); }
|
||||
.back { display:inline-block; margin:0 16px 10px 0; font-weight:650; }
|
||||
code { padding:1px 5px; background:var(--code); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; font-size:.92em; overflow-wrap:anywhere; }
|
||||
pre { overflow-x:auto; padding:14px 16px; background:#202a34; color:#f4f7fa; border-radius:5px; line-height:1.5; }
|
||||
pre code { padding:0; background:transparent; color:inherit; }
|
||||
table { width:100%; margin:14px 0 22px; border-collapse:collapse; font-size:14px; }
|
||||
th,td { padding:10px 11px; border:1px solid var(--line); text-align:left; vertical-align:top; }
|
||||
th { background:#edf1f4; }
|
||||
.status,.decision,.warning { padding:14px 16px; }
|
||||
.status { border-left:4px solid var(--green); background:#eef8f3; }
|
||||
.decision { border-left:4px solid var(--blue); background:#edf5fb; }
|
||||
.warning { border-left:4px solid var(--amber); background:#fff7e7; }
|
||||
.metric { font-size:20px; font-weight:750; color:var(--green); }
|
||||
li+li { margin-top:5px; }
|
||||
@media (max-width:720px) { main{padding:22px 18px 30px} table{display:block;overflow-x:auto;white-space:nowrap} }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">FINAL EXPERIMENT RECORD / PHASE 2.5</p>
|
||||
<h1>DeepSeek-V4-Pro 双机 Pro6000D SGLang：RDMA 需求建模与并发拐点</h1>
|
||||
<div class="meta">
|
||||
<span>Run：dsv4pro-phase2_5-20260801-130007</span>
|
||||
<span>拓扑：TP16 / EP2 / 双 Rail RoCE</span>
|
||||
<span>完成：2026-08-01 15:14:11 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
<main>
|
||||
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||
<a class="back" href="./phase2_5_code.html">打开 Phase 2.5 代码详解</a>
|
||||
|
||||
<p class="status"><strong>阶段已完成。</strong>正式 Run 用时 2 小时 14 分 04 秒；Scout 5/5、Confirm 6/6 成功，两阶段各 18/18 个采集器正常启停。所有测量窗 RDMA 错误增量为 0，结束后两节点容器和 16 张 GPU 均已清理。</p>
|
||||
|
||||
<h2>1. 要回答的问题</h2>
|
||||
<p>Phase 2 只看到代表负载约 83.5 Gbit/s/rail，不能判断继续增加并发是否会逼近 400G。Phase 2.5 专门回答三个问题：</p>
|
||||
<ol>
|
||||
<li>固定模型、TP/EP 和输入形状后，Input TPS 与每 Rail RDMA 带宽是什么关系？</li>
|
||||
<li>并发增加到哪里后，模型吞吐和 RDMA 带宽不再增长？</li>
|
||||
<li>要达到 400G，需要怎样的 Input TPS；当前瓶颈先出现在模型计算还是网络？</li>
|
||||
</ol>
|
||||
|
||||
<h2>2. 实验设计</h2>
|
||||
<table>
|
||||
<thead><tr><th>阶段</th><th>请求形状</th><th>并发</th><th>重复</th><th>目的</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Scout</td><td>64K → 1</td><td>1 / 4 / 16 / 32 / 64</td><td>1</td><td>隔离 Prefill，找吞吐与带宽平台</td></tr>
|
||||
<tr><td>Confirm</td><td>64K → 1K</td><td>自动选择 4 / 16 / 64</td><td>2</td><td>验证真实长输出不会推翻需求模型</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>服务参数沿用 Phase 1/2：SGLang nightly、TP16、EP2、双 Rail <code>mlx5_0/mlx5_3</code>、<code>NET/IB + GDRDMA</code>。每个 Case 使用冷 Prefix，并按 benchmark 正式测量窗口切片 HCA Counter。</p>
|
||||
|
||||
<h2>3. 实际启动命令</h2>
|
||||
<p>只在 Head <code>174.1.51.5</code> 执行，不需要 <code>source</code> 或 <code>conda activate</code>：</p>
|
||||
<pre><code>cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||
|
||||
RUN_ID=dsv4pro-phase2_5-20260801-130007
|
||||
tmux new-session -d -s dsv4pro-phase2_5 \
|
||||
"RUN_ID=${RUN_ID} bash run_rdma_demand_modeling.sh all \
|
||||
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||
|
||||
tmux attach -t dsv4pro-phase2_5</code></pre>
|
||||
<p>实际展开后的 Scout/Confirm 命令分别保存在结果目录的 <code>commands/scout.cmd.txt</code> 与 <code>commands/confirm.cmd.txt</code>。</p>
|
||||
|
||||
<h2>4. Scout 结果：并发 16 已进入平台</h2>
|
||||
<table>
|
||||
<thead><tr><th>C</th><th>Input TPS</th><th>Rail Mean</th><th>Rail P95</th><th>Rail Max</th><th>双 Rail 单向合计</th><th>MB/input-token/rail</th><th>GPU Util</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>1</td><td>2,709.64</td><td>70.97 Gbit/s</td><td>85.60</td><td>86.81</td><td>141.94</td><td>3.138</td><td>93.75%</td></tr>
|
||||
<tr><td>4</td><td>2,930.49</td><td>78.27 Gbit/s</td><td>86.82</td><td>88.86</td><td>156.55</td><td>3.298</td><td>97.45%</td></tr>
|
||||
<tr><td>16</td><td>2,983.77</td><td>79.90 Gbit/s</td><td>86.42</td><td>88.20</td><td>159.79</td><td>3.336</td><td>99.33%</td></tr>
|
||||
<tr><td>32</td><td>2,983.92</td><td>79.93 Gbit/s</td><td>86.23</td><td>88.22</td><td>159.86</td><td>3.344</td><td>99.49%</td></tr>
|
||||
<tr><td>64</td><td>2,991.28</td><td>79.97 Gbit/s</td><td>85.93</td><td>88.32</td><td>159.95</td><td>3.340</td><td>99.64%</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="decision"><strong>观察结论：</strong>C=16→32 的 Input TPS 只增长 0.005%，Rail Mean 只增长 0.040%；C=32→64 也仅增长 0.247% / 0.057%。并发 16 已是平台拐点，继续加到 64 只会增加排队和 TTFT，不会增加网络压力。</p>
|
||||
<p><strong>证据：</strong><code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv</code>；原始 HCA 数据位于同一 Run 的 <code>scout/head/rdma.csv</code> 与 <code>scout/worker/rdma.csv</code>，精确窗口位于 <code>scout/case_windows.csv</code>。</p>
|
||||
|
||||
<h2>5. Confirm 结果：加入 1K Decode 后仍由计算先饱和</h2>
|
||||
<table>
|
||||
<thead><tr><th>C</th><th>重复</th><th>Input TPS</th><th>Output TPS</th><th>Rail Mean</th><th>Rail P95</th><th>双 Rail单向合计</th><th>TTFT P95</th><th>TPOT P95</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>4</td><td>2</td><td>2,072.24</td><td>32.38</td><td>57.33 Gbit/s</td><td>86.28</td><td>114.66</td><td>86.36 s</td><td>95.63 ms</td></tr>
|
||||
<tr><td>16</td><td>2</td><td>2,582.13</td><td>40.35</td><td>71.64 Gbit/s</td><td>86.30</td><td>143.29</td><td>336.85 s</td><td>356.24 ms</td></tr>
|
||||
<tr><td>64</td><td>2</td><td>2,588.96</td><td>40.45</td><td>72.19 Gbit/s</td><td>86.69</td><td>144.38</td><td>1,506.50 s</td><td>418.58 ms</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>C=16→64 的 Input TPS 仅增长 0.26%，Rail Mean 仅增长 0.76%，但 TTFT P95 从 336.85 秒升至 1,506.50 秒。对于 64K→1K，<strong>最大有意义并发仍约为 16</strong>；C=64 是容量压力点，不是推荐服务点。</p>
|
||||
<p><strong>证据：</strong>同一 Run 的 <code>confirm/bench_summary.csv</code>、<code>confirm/case_rdma_summary.csv</code>、<code>confirm/case_windows.csv</code>；两轮逐点数据在顶层 <code>rdma_case_metrics.csv</code>。</p>
|
||||
|
||||
<h2>6. 400G 能否被模型负载打满</h2>
|
||||
<p>Scout 的线性比例为：</p>
|
||||
<pre><code>每 Rail 带宽（Gbit/s）
|
||||
≈ Input TPS × 3.332 MB/input-token/rail × 8 ÷ 1e9</code></pre>
|
||||
<table>
|
||||
<thead><tr><th>目标口径</th><th>需要的 Input TPS</th><th>当前约 2,991 TPS 的差距</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>单 Rail 400G</td><td class="metric">15,006 tok/s</td><td>约 5.02×</td></tr>
|
||||
<tr><td>单 Rail 360G（90% 实用线）</td><td>13,505 tok/s</td><td>约 4.51×</td></tr>
|
||||
<tr><td>双 Rail 单向合计 400G</td><td>7,503 tok/s</td><td>约 2.51×</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>拟合得到当前模型负载的单 Rail 渐近上限约 <strong>80.32 Gbit/s</strong>，即物理 400G 的约 20.1%。瞬时 Max 也只有 89.53 Gbit/s。结论不是“网络只能跑 80G”，而是<strong>当前 DSV4-Pro TP16/EP2 实现最多只能产生约 80G/rail 的持续 RDMA 流量</strong>；Phase 2 的 NCCL microbenchmark 已证明链路本身能达到更高通信带宽。</p>
|
||||
<p class="warning"><strong>口径提醒：</strong>400G 是每条 Rail 的线速；双 Rail 单向总量是两条 Rail 的 TX 之和。不要把 TX 与 RX 相加后声称打满，也不要把 NCCL <code>busbw GB/s</code> 与 HCA <code>Gbit/s</code> 直接比较。</p>
|
||||
|
||||
<h2>7. 一套可复用的 RDMA 需求评估方法</h2>
|
||||
<ol>
|
||||
<li><strong>固定部署变量。</strong>记录模型版本、精度/量化、TP/EP/PP/DP、节点数、Attention/MoE backend、chunked prefill 和网卡拓扑。任一项变化都要重新标定。</li>
|
||||
<li><strong>先选 Prefill Scout。</strong>固定 ISL，OSL=1，取稀疏并发点如 1/4/16/32/64；每点清 Prefix Cache，并保证请求文本实际达到目标 token 数。</li>
|
||||
<li><strong>对齐正式测量窗。</strong>从 benchmark 的 main-run 起止时间切片 Head/Worker 的 <code>mlx5_*</code> HCA Counter，不能用整个进程寿命，也不能只看 <code>sar eth*</code>。</li>
|
||||
<li><strong>计算通信强度。</strong><code>bytes_per_input_token_per_rail = rail_xmit_bytes / total_input_tokens</code>。这是该模型与并行策略下“每处理一个输入 token，要在一条 Rail 发送多少字节”。</li>
|
||||
<li><strong>找并发平台。</strong>同时观察 Input TPS 和 Rail Mean；连续一点的增益都低于阈值（本实验 5%）时，记为拐点。最大 C 不等于最大有效 C。</li>
|
||||
<li><strong>推导目标吞吐。</strong><code>required_input_tps = target_rail_gbps × 1e9 / (bytes_per_token × 8)</code>。若模型的实测/拟合 TPS 上限远低于该值，网络不会先饱和。</li>
|
||||
<li><strong>用业务 OSL 复测。</strong>在平台前、拐点、最高压力点各重复至少两次，确认 Decode、KV Cache 和调度没有改变结论。</li>
|
||||
<li><strong>最后做链路对照。</strong>模型负载未打满时，用 NCCL microbenchmark 验证网络能力，把“模型产流量不足”与“网络本身跑不满”分开。</li>
|
||||
</ol>
|
||||
|
||||
<h3>7.1 哪些变量会改变 bytes/token 与平台</h3>
|
||||
<table>
|
||||
<thead><tr><th>变量</th><th>可能改变的原因</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>模型架构与层数</td><td>每 token 触发的 TP collective、MoE dispatch/combine 和激活尺寸不同</td></tr>
|
||||
<tr><td>TP / EP / PP / DP</td><td>通信参与 rank、跨机边界、collective 类型和频率改变</td></tr>
|
||||
<tr><td>Prefill / Decode、ISL / OSL</td><td>计算强度、chunk 调度、KV 访问和 collective 消息粒度不同</td></tr>
|
||||
<tr><td>并发与 batch</td><td>决定 kernel/batch 效率和 Input TPS；超过平台后只增加排队</td></tr>
|
||||
<tr><td>量化与 backend</td><td>改变计算速度；通信字节可能不同比例变化，因此会移动“计算先饱和还是网络先饱和”的边界</td></tr>
|
||||
<tr><td>Prefix Cache</td><td>命中会绕过大量 Prefill，必须单独作为另一类业务场景建模</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>8. 最终结论</h2>
|
||||
<p class="decision"><strong>在两台 Pro6000D、DSV4-Pro、SGLang TP16/EP2 的当前实现中，RDMA 不是吞吐瓶颈。</strong>64K Prefill 在 C=16 已达到约 3K input tok/s 和 80 Gbit/s/rail 的平台；继续增加并发到 64 不会显著增加吞吐或带宽，只会令 TTFT 急剧上升。要打满单 Rail 400G，模型侧 Input TPS 需提高到约 15K，约为当前上限 5 倍。因此后续优化应先看 GPU Kernel、MoE/Attention 执行和 rank 同步，而不是扩容计算网。</p>
|
||||
|
||||
<h2>9. 证据与清理</h2>
|
||||
<ul>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_report.md">自动 RDMA 需求报告</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_model.json">机器可读模型</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv">全部逐点指标</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/run_manifest.txt">Run Manifest</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/head_server_cmd.txt">Scout Head 服务命令</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_server_cmd.txt">Worker 服务命令</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_nccl_transport.log">Confirm Head NCCL 路径</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_nccl_transport.log">Worker NCCL 路径</a></li>
|
||||
</ul>
|
||||
<p>Worker 在最后一个 Case 完成后随 Head 主动关闭进程组出现 Gloo peer-close Traceback；它发生在测量结束与结果落盘之后，不是实验失败。最终 tmux、服务容器和 GPU 进程均已退出。</p>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@ -805,6 +805,7 @@ tmux new-session -d -s dsv4pro-phase2 \
|
||||
</p>
|
||||
|
||||
<p><a class="back" href="./phase1_exp.html">返回 Phase 1 实验档案</a></p>
|
||||
<p><a class="back" href="./phase2_5_exp.html">继续 Phase 2.5 RDMA 需求建模</a></p>
|
||||
<p><a class="back" href="./推理优化计划.html">返回推理优化主计划</a></p>
|
||||
</main>
|
||||
</body>
|
||||
|
||||
@ -0,0 +1 @@
|
||||
env DRY_RUN=0 RUN_ID=confirm RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/confirm.tsv FIXED_CASE_IDS=rdma_confirm_i65536_o1024_c4\,rdma_confirm_i65536_o1024_c16\,rdma_confirm_i65536_o1024_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=2 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
|
||||
@ -0,0 +1 @@
|
||||
env DRY_RUN=0 RUN_ID=scout RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/scout.tsv FIXED_CASE_IDS=rdma_scout_i65536_o1_c1\,rdma_scout_i65536_o1_c4\,rdma_scout_i65536_o1_c16\,rdma_scout_i65536_o1_c32\,rdma_scout_i65536_o1_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=1 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
|
||||
@ -0,0 +1,7 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||
fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:03:43+0800,2026-08-01T14:11:03+0800,440.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,406.0844679859583,bench_main_marker_plus_duration,16,0,406.0844679859583,15.997723352218085,,1048576,16384,0.03940066971621592,2582.1622905219265,40.3462857894051,2622.5085763113316,,406026.68602950143,406024.73604350234,406053.3737935184,406053.5528299224,189372.86670175308,189251.9370575028,336778.7128902273,348624.74093402264,211.7828145921294,211.89852223703926,356.41907673483706,369.22852823690624,211.7827865053078,53.14174899831414,53.806967759737745,55.23117201111745,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c16
|
||||
fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:11:03+0800,2026-08-01T14:18:24+0800,441.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,406.09330375201534,bench_main_marker_plus_duration,16,0,406.09330375201534,15.997232835777202,,1048576,16384,0.03939981243761298,2582.1061079114043,40.34540793611569,2622.45151584752,,406023.07082318654,406021.80599197163,406049.04031673505,406056.0000617319,189594.6209833055,189467.45998901315,336924.2522477143,348614.9598911492,211.56251206244485,211.68662201320043,356.06784952074946,368.88221031313344,211.56248251307755,53.15867651370354,54.19228565006051,55.67694299679715,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c16
|
||||
fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T13:58:15+0800,2026-08-01T14:00:56+0800,161.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.71871886099689,bench_main_marker_plus_duration,4,0,127.71871886099689,3.9987289259054917,,262144,4096,0.031318823393095684,2052.5104098899187,32.07047515452998,2084.5808850444487,,127678.13387226488,127678.37510351092,127682.15249372879,127682.41568037542,58779.63522000937,59286.87346848892,87520.75181677355,89912.75672257761,67.34946104814811,66.85386279083285,96.14249014501952,98.71897348003505,67.3494309836352,36.35076849604957,36.65076588222291,37.22598696593195,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c4
|
||||
fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T14:00:57+0800,2026-08-01T14:03:36+0800,159.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,125.30951252899831,bench_main_marker_plus_duration,4,0,125.30951252899831,3.9986913595171454,,262144,4096,0.03192096050229504,2091.9720674784076,32.68706355435012,2124.6591310327576,,125268.51625375275,125267.95494600083,125274.71757131862,125275.15292868251,56944.67479274317,57454.862895508995,85207.63279019155,87584.75151082734,66.7877238133036,66.2884575273625,95.11951526871552,97.68299949875195,66.78769251954503,36.23565100133419,36.565644716029055,37.02181185770315,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c4
|
||||
fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:18:31+0800,2026-08-01T14:46:05+0800,1654.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.8975476099877,bench_main_marker_plus_duration,64,0,1619.8975476099877,40.3918658202939,,4194304,65536,0.039508671455442484,2589.2402925038787,40.456879570373104,2629.6971720742517,,1022354.4435263402,950470.9995904996,1619639.9038552889,1619650.3429244931,785995.8297684161,774078.3500669932,1506265.7001026294,1560787.9135349835,231.04458822866482,223.80146996136563,418.57784627036636,439.7380822666596,231.0445591993489,56.0161369794514,56.77989884279668,58.514122637570836,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c64
|
||||
fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:46:06+0800,2026-08-01T15:13:40+0800,1654.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1620.2422464910196,bench_main_marker_plus_duration,64,0,1620.2422464910196,40.39704233770943,,4194304,65536,0.03950026617229964,2588.689443867829,40.44827256043483,2629.137716428264,,1022703.0410756742,950775.2832530241,1620010.8873196847,1620019.4079715456,786335.3449273454,774403.2157540205,1506724.735757988,1561196.831805848,231.05346642065376,223.83287831180922,418.5783493891426,439.7222371669007,231.0534379278651,56.055732507957146,56.75308680802118,58.11881933768746,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c64
|
||||
|
@ -0,0 +1,25 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_0,371,3626828296476.0,3626837993952.0,71.66006119953522,71.66025280544177,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_3,371,3626914718800.0,3626921347028.0,71.66176876011448,71.66189972281393,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_0,383,3626134447036.0,3626135801124.0,71.58004520955629,71.58007193930987,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_3,383,3626207403788.0,3626212421504.0,71.58148537888337,71.58158442880648,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_0,371,3625059758564.0,3625068124448.0,71.59790115790838,71.59806639097366,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_3,371,3625136689400.0,3625142126056.0,71.59942060496823,71.59952798337986,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_0,382,3625042688652.0,3625045908572.0,71.73367967522776,71.73374339219049,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_3,382,3625122942520.0,3625131203508.0,71.73526776832192,71.73543123979464,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_0,117,894188173332.0,894192013996.0,56.331515043976715,56.33175699575861,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_3,117,908654087048.0,908656485748.0,57.24282975425881,57.24298086608198,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_0,121,908912097324.0,908914949500.0,57.10271925252004,57.102898441471176,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_3,121,894353999288.0,894353927048.0,56.188101670195095,56.188097131691926,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_0,115,894195520320.0,894197433708.0,57.32326114869037,57.32338380826259,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_3,115,908661063200.0,908663318872.0,58.25058864399148,58.250733246007684,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_0,118,908220840268.0,908222493124.0,58.56932688787363,58.5694334772174,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_3,118,893986155676.0,893986644792.0,57.65136084035526,57.651392382457814,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_0,1479,14610070965300.0,14610072489408.0,72.21898772698583,72.21899526079844,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_3,1479,14610085802320.0,14610095102220.0,72.21906106780447,72.21910703810576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_0,1527,14611153636236.0,14611146819600.0,72.16844452340027,72.1684108541891,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_3,1527,14611226802652.0,14611236350532.0,72.16880591214243,72.16885307170732,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_0,1480,14610569508844.0,14610573106108.0,72.18614224806339,72.18616002099213,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_3,1480,14610597458160.0,14610605842440.0,72.18628033667787,72.18632176071877,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_0,1527,14610266497856.0,14610260234736.0,72.18231262718308,72.18228168411397,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_3,1527,14610323772088.0,14610331114336.0,72.18259559166701,72.18263186618948,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
|
@ -0,0 +1,7 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,1785564254150768896,1785564660235236864,406.084467968,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,1785564694981654016,1785565101074958080,406.093304064,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,1785563926377015040,1785564054095733760,127.71871872,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,1785564088360896000,1785564213670408960,125.30951296,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1785565142582675968,1785566762480224000,1619.897548032,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1785566797465815040,1785568417708060928,1620.242245888,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/meta.json
|
||||
|
@ -0,0 +1,37 @@
|
||||
wall_time_ns,node,collector,status,pid
|
||||
1785563885833567643,head,gpu_samples.csv,STARTED,2581028
|
||||
1785563885848767676,head,rdma.csv,STARTED,2581039
|
||||
1785563885862486752,head,dcgm_dmon.log,STARTED,2581053
|
||||
1785563885876598354,head,mpstat.log,STARTED,2581071
|
||||
1785563885891181421,head,pidstat.log,STARTED,2581090
|
||||
1785563885906513050,head,sar_net.log,STARTED,2581117
|
||||
1785563885921729008,head,perf_stat.log,STARTED,2581144
|
||||
1785563885938632713,head,docker_top.log,STARTED,2581179
|
||||
1785563885959612586,head,numa_samples.csv,STARTED,2581251
|
||||
1785563885980492840,worker,gpu_samples.csv,STARTED,2581294
|
||||
1785563885999012597,worker,rdma.csv,STARTED,2581356
|
||||
1785563886014677475,worker,dcgm_dmon.log,STARTED,2581408
|
||||
1785563886031239484,worker,mpstat.log,STARTED,2581455
|
||||
1785563886046314326,worker,pidstat.log,STARTED,2581464
|
||||
1785563886060934631,worker,sar_net.log,STARTED,2581478
|
||||
1785563886075086128,worker,perf_stat.log,STARTED,2581486
|
||||
1785563886090622653,worker,docker_top.log,STARTED,2581497
|
||||
1785563886106525841,worker,numa_samples.csv,STARTED,2581509
|
||||
1785568431501511663,head,gpu_samples.csv,STOPPED,2581028
|
||||
1785568431506303146,head,rdma.csv,STOPPED,2581039
|
||||
1785568431513026775,head,dcgm_dmon.log,STOPPED,2581053
|
||||
1785568431518181488,head,mpstat.log,STOPPED,2581071
|
||||
1785568431523384390,head,pidstat.log,STOPPED,2581090
|
||||
1785568431528223936,head,sar_net.log,STOPPED,2581117
|
||||
1785568431608439580,head,perf_stat.log,STOPPED,2581144
|
||||
1785568431612888149,head,docker_top.log,STOPPED,2581179
|
||||
1785568431617027570,head,numa_samples.csv,STOPPED,2581251
|
||||
1785568431620791766,worker,gpu_samples.csv,STOPPED,2581294
|
||||
1785568431624694474,worker,rdma.csv,STOPPED,2581356
|
||||
1785568431628587426,worker,dcgm_dmon.log,STOPPED,2581408
|
||||
1785568431632294659,worker,mpstat.log,STOPPED,2581455
|
||||
1785568431636153536,worker,pidstat.log,STOPPED,2581464
|
||||
1785568431639870020,worker,sar_net.log,STOPPED,2581478
|
||||
1785568431643739953,worker,perf_stat.log,STOPPED,2581486
|
||||
1785568431647558030,worker,docker_top.log,STOPPED,2581497
|
||||
1785568431651292267,worker,numa_samples.csv,STOPPED,2581509
|
||||
|
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -0,0 +1,12 @@
|
||||
stage,case_id,repetition,status,isl,osl,concurrency,completed,duration_s,input_tps,output_tps,total_tps,ttft_p95_ms,tpot_p95_ms,gpu_util_mean_pct,rail_xmit_avg_gbps,rail_xmit_p95_gbps,rail_xmit_max_gbps,dual_rail_xmit_avg_gbps,rail_imbalance_pct,xmit_bytes_per_input_token_per_rail,xmit_bytes_per_total_token_per_rail,rdma_error_delta,measurement_started_at,measurement_ended_at,source_result_dir
|
||||
scout,rdma_scout_i65536_o1_c1,1,COMPLETED,65536,1,1,1,24.1862671909621,2709.6368150803128,0.04134577659729481,2709.67816085691,24167.605170048773,0.0,93.74897875816994,70.9712746848062,85.5972372945909,86.80566056858429,141.9425493696124,0.5207119247408836,3137732.762954712,3137684.8856828967,0.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c4,1,COMPLETED,65536,1,4,4,89.45394851703895,2930.491100122512,0.044715745546302976,2930.535815868058,86526.69315790116,0.0,97.4494891826923,78.27464182068744,86.81919268342193,88.85894624954848,156.5492836413749,0.2703571902588796,3298012.1388320923,3297961.815928407,0.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c16,1,COMPLETED,65536,1,16,16,351.42641380499117,2983.771164628114,0.045528734811830354,2983.816693362926,336717.4506017618,0.0,99.33137833282255,79.89685137869246,86.416508483831,88.20167661610235,159.79370275738495,0.1149304623247205,3335858.008945465,3335807.108568442,0.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c32,1,COMPLETED,65536,1,32,32,702.8175486620166,2983.9209393567885,0.04553102019282209,2983.9664703769813,671139.2772838357,0.0,99.49393723047987,79.92852806348517,86.22587499587476,88.21644913628985,159.85705612697035,0.10091204027725086,3343929.3806581497,3343878.357123648,0.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c64,1,COMPLETED,65536,1,64,64,1402.17612864502,2991.2818470623424,0.04564333873080967,2991.327490401073,1337131.1103800863,0.0,99.63928846991146,79.9743866369171,85.93278751515446,88.32212536165416,159.9487732738342,0.03496630506355109,3339597.901974201,3339546.9445318105,0.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
confirm,rdma_confirm_i65536_o1024_c4,1,COMPLETED,65536,1024,4,4,127.71871886099689,2052.5104098899187,32.07047515452998,2084.5808850444487,87520.75181677355,96.14249014501952,97.83383458646617,56.71629143023766,86.06750356786138,88.99885476939444,113.43258286047534,1.8596562953363254,3439052.922241211,3386144.415745192,0.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c4,2,COMPLETED,65536,1024,4,4,125.30951252899831,2091.9720674784076,32.68706355435012,2124.6591310327576,85207.63279019155,95.11951526871552,98.63248207885304,57.94863438022769,86.4894823881113,88.5335715781298,115.89726876045536,2.1502935358359734,3438056.544746399,3385163.3671349157,0.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c16,1,COMPLETED,65536,1024,16,16,406.0844679859583,2582.1622905219265,40.3462857894051,2622.5085763113316,336778.7128902273,356.41907673483706,99.38883591065292,71.62084013702234,86.26345231858909,88.9399300384103,143.24168027404468,0.11410582506689998,3458520.1421022415,3405312.139916053,0.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c16,2,COMPLETED,65536,1024,16,16,406.09330375201534,2582.1061079114043,40.34540793611569,2622.45151584752,336924.2522477143,356.06784952074946,99.48923731257209,71.66656730160658,86.33499788736604,89.31414888004788,143.33313460321313,0.19167460586668467,3457155.7233657837,3403968.712237079,0.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c64,1,COMPLETED,65536,1024,64,64,1619.8975476099877,2589.2402925038787,40.456879570373104,2629.6971720742517,1506265.7001026294,418.57784627036636,99.6722631148773,72.19382480758325,86.71108451788668,89.82468221963666,144.38764961516648,0.070112013789426,3483446.6699664593,3429855.182736206,0.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c64,2,COMPLETED,65536,1024,64,64,1620.2422464910196,2588.689443867829,40.44827256043483,2629.137716428264,1506724.735757988,418.5783493891426,99.67225809785734,72.18433270089784,86.67873733217695,89.23672463088391,144.36866540179568,0.005496635275729702,3483400.1801579,3429809.4081554706,0.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
|
@ -0,0 +1,205 @@
|
||||
{
|
||||
"generated_at": "2026-08-01T15:14:11.558999+08:00",
|
||||
"formula": "rail_gbps ~= input_tps * bytes_per_input_token_per_rail * 8 / 1e9",
|
||||
"plateau_gain_pct": 5.0,
|
||||
"target_rail_gbps": 400.0,
|
||||
"practical_rail_gbps": 360.0,
|
||||
"scout": [
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 1,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2709.6368150803128,
|
||||
"output_tps": 0.04134577659729481,
|
||||
"total_tps": 2709.67816085691,
|
||||
"ttft_p95_ms": 24167.605170048773,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 93.74897875816994,
|
||||
"rail_xmit_avg_gbps": 70.9712746848062,
|
||||
"rail_xmit_p95_gbps": 85.5972372945909,
|
||||
"rail_xmit_max_gbps": 86.80566056858429,
|
||||
"dual_rail_xmit_avg_gbps": 141.9425493696124,
|
||||
"rail_imbalance_pct": 0.5207119247408836,
|
||||
"xmit_bytes_per_input_token_per_rail": 3137732.762954712,
|
||||
"xmit_bytes_per_total_token_per_rail": 3137684.8856828967,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 4,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2930.491100122512,
|
||||
"output_tps": 0.044715745546302976,
|
||||
"total_tps": 2930.535815868058,
|
||||
"ttft_p95_ms": 86526.69315790116,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 97.4494891826923,
|
||||
"rail_xmit_avg_gbps": 78.27464182068744,
|
||||
"rail_xmit_p95_gbps": 86.81919268342193,
|
||||
"rail_xmit_max_gbps": 88.85894624954848,
|
||||
"dual_rail_xmit_avg_gbps": 156.5492836413749,
|
||||
"rail_imbalance_pct": 0.2703571902588796,
|
||||
"xmit_bytes_per_input_token_per_rail": 3298012.1388320923,
|
||||
"xmit_bytes_per_total_token_per_rail": 3297961.815928407,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 16,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2983.771164628114,
|
||||
"output_tps": 0.045528734811830354,
|
||||
"total_tps": 2983.816693362926,
|
||||
"ttft_p95_ms": 336717.4506017618,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.33137833282255,
|
||||
"rail_xmit_avg_gbps": 79.89685137869246,
|
||||
"rail_xmit_p95_gbps": 86.416508483831,
|
||||
"rail_xmit_max_gbps": 88.20167661610235,
|
||||
"dual_rail_xmit_avg_gbps": 159.79370275738495,
|
||||
"rail_imbalance_pct": 0.1149304623247205,
|
||||
"xmit_bytes_per_input_token_per_rail": 3335858.008945465,
|
||||
"xmit_bytes_per_total_token_per_rail": 3335807.108568442,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 32,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2983.9209393567885,
|
||||
"output_tps": 0.04553102019282209,
|
||||
"total_tps": 2983.9664703769813,
|
||||
"ttft_p95_ms": 671139.2772838357,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.49393723047987,
|
||||
"rail_xmit_avg_gbps": 79.92852806348517,
|
||||
"rail_xmit_p95_gbps": 86.22587499587476,
|
||||
"rail_xmit_max_gbps": 88.21644913628985,
|
||||
"dual_rail_xmit_avg_gbps": 159.85705612697035,
|
||||
"rail_imbalance_pct": 0.10091204027725086,
|
||||
"xmit_bytes_per_input_token_per_rail": 3343929.3806581497,
|
||||
"xmit_bytes_per_total_token_per_rail": 3343878.357123648,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 64,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2991.2818470623424,
|
||||
"output_tps": 0.04564333873080967,
|
||||
"total_tps": 2991.327490401073,
|
||||
"ttft_p95_ms": 1337131.1103800863,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.63928846991146,
|
||||
"rail_xmit_avg_gbps": 79.9743866369171,
|
||||
"rail_xmit_p95_gbps": 85.93278751515446,
|
||||
"rail_xmit_max_gbps": 88.32212536165416,
|
||||
"dual_rail_xmit_avg_gbps": 159.9487732738342,
|
||||
"rail_imbalance_pct": 0.03496630506355109,
|
||||
"xmit_bytes_per_input_token_per_rail": 3339597.901974201,
|
||||
"xmit_bytes_per_total_token_per_rail": 3339546.9445318105,
|
||||
"rdma_error_delta": 0.0
|
||||
}
|
||||
],
|
||||
"confirm": [
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 4,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2072.2412386841634,
|
||||
"output_tps": 32.37876935444005,
|
||||
"total_tps": 2104.6200080386034,
|
||||
"ttft_p95_ms": 86364.19230348256,
|
||||
"tpot_p95_ms": 95.63100270686752,
|
||||
"gpu_util_mean_pct": 98.2331583326596,
|
||||
"rail_xmit_avg_gbps": 57.332462905232674,
|
||||
"rail_xmit_p95_gbps": 86.27849297798633,
|
||||
"rail_xmit_max_gbps": 88.76621317376211,
|
||||
"dual_rail_xmit_avg_gbps": 114.66492581046535,
|
||||
"rail_imbalance_pct": 2.0049749155861494,
|
||||
"xmit_bytes_per_input_token_per_rail": 3438554.733493805,
|
||||
"xmit_bytes_per_total_token_per_rail": 3385653.891440054,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 16,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2582.1341992166654,
|
||||
"output_tps": 40.345846862760396,
|
||||
"total_tps": 2622.480046079426,
|
||||
"ttft_p95_ms": 336851.4825689708,
|
||||
"tpot_p95_ms": 356.24346312779323,
|
||||
"gpu_util_mean_pct": 99.4390366116125,
|
||||
"rail_xmit_avg_gbps": 71.64370371931446,
|
||||
"rail_xmit_p95_gbps": 86.29922510297757,
|
||||
"rail_xmit_max_gbps": 89.12703945922908,
|
||||
"dual_rail_xmit_avg_gbps": 143.28740743862892,
|
||||
"rail_imbalance_pct": 0.15289021546679232,
|
||||
"xmit_bytes_per_input_token_per_rail": 3457837.9327340126,
|
||||
"xmit_bytes_per_total_token_per_rail": 3404640.426076566,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 64,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2588.964868185854,
|
||||
"output_tps": 40.45257606540397,
|
||||
"total_tps": 2629.4174442512576,
|
||||
"ttft_p95_ms": 1506495.2179303085,
|
||||
"tpot_p95_ms": 418.5780978297545,
|
||||
"gpu_util_mean_pct": 99.67226060636732,
|
||||
"rail_xmit_avg_gbps": 72.18907875424054,
|
||||
"rail_xmit_p95_gbps": 86.69491092503182,
|
||||
"rail_xmit_max_gbps": 89.53070342526028,
|
||||
"dual_rail_xmit_avg_gbps": 144.37815750848108,
|
||||
"rail_imbalance_pct": 0.03780432453257785,
|
||||
"xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
|
||||
"xmit_bytes_per_total_token_per_rail": 3429832.295445838,
|
||||
"rdma_error_delta": 0.0
|
||||
}
|
||||
],
|
||||
"selected_confirm_concurrencies": [
|
||||
4,
|
||||
16,
|
||||
64
|
||||
],
|
||||
"observed_plateau_concurrency": 16,
|
||||
"best_scout_bandwidth_concurrency": 64,
|
||||
"linear_gbps_per_input_tps": 0.026656368057740153,
|
||||
"modeled_bytes_per_input_token_per_rail": 3332046.007217519,
|
||||
"required_input_tps_for_400g_per_rail": 15005.795205617025,
|
||||
"required_input_tps_for_360g_per_rail": 13505.215685055322,
|
||||
"required_input_tps_for_400g_dual_rail_aggregate": 7502.897602808513,
|
||||
"bandwidth_saturation_fit": {
|
||||
"asymptote": 80.32102454669293,
|
||||
"half_saturation_c": 0.12589254117941673,
|
||||
"rmse": 0.27652726609534695
|
||||
},
|
||||
"input_tps_saturation_fit": {
|
||||
"asymptote": 3003.9418335884134,
|
||||
"half_saturation_c": 0.11220184543019636,
|
||||
"rmse": 7.680357081457988
|
||||
},
|
||||
"best_confirm_point": {
|
||||
"stage": "confirm",
|
||||
"concurrency": 64,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2588.964868185854,
|
||||
"output_tps": 40.45257606540397,
|
||||
"total_tps": 2629.4174442512576,
|
||||
"ttft_p95_ms": 1506495.2179303085,
|
||||
"tpot_p95_ms": 418.5780978297545,
|
||||
"gpu_util_mean_pct": 99.67226060636732,
|
||||
"rail_xmit_avg_gbps": 72.18907875424054,
|
||||
"rail_xmit_p95_gbps": 86.69491092503182,
|
||||
"rail_xmit_max_gbps": 89.53070342526028,
|
||||
"dual_rail_xmit_avg_gbps": 144.37815750848108,
|
||||
"rail_imbalance_pct": 0.03780432453257785,
|
||||
"xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
|
||||
"xmit_bytes_per_total_token_per_rail": 3429832.295445838,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
"verdict": "COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION"
|
||||
}
|
||||
@ -0,0 +1,37 @@
|
||||
# Phase 2.5 RDMA Demand Model
|
||||
|
||||
- Generated: `2026-08-01T15:14:11.558999+08:00`
|
||||
- Verdict: `COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION`
|
||||
- Model: `rail_gbps ~= input_tps x 3.332 MB/token/rail x 8`
|
||||
- Required Input TPS for 400G per rail: `15006`
|
||||
- Required Input TPS for 360G per rail: `13505`
|
||||
- Required Input TPS for 400G dual-rail aggregate: `7503`
|
||||
- Fitted rail bandwidth ceiling: `80.32 Gbit/s`
|
||||
- Observed plateau concurrency: `16`
|
||||
|
||||
## Scout: 64K -> 1
|
||||
|
||||
| C | Input TPS | Rail avg | Rail P95 | Dual-rail avg | MB/input-token/rail | GPU util |
|
||||
|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 1 | 2709.64 | 70.97 | 85.60 | 141.94 | 3.138 | 93.75 |
|
||||
| 4 | 2930.49 | 78.27 | 86.82 | 156.55 | 3.298 | 97.45 |
|
||||
| 16 | 2983.77 | 79.90 | 86.42 | 159.79 | 3.336 | 99.33 |
|
||||
| 32 | 2983.92 | 79.93 | 86.23 | 159.86 | 3.344 | 99.49 |
|
||||
| 64 | 2991.28 | 79.97 | 85.93 | 159.95 | 3.340 | 99.64 |
|
||||
|
||||
## Confirm: 64K -> 1K
|
||||
|
||||
| C | Reps | Input TPS | Output TPS | Rail avg | Rail P95 | Dual-rail avg | TTFT P95 ms | TPOT P95 ms |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 4 | 2 | 2072.24 | 32.38 | 57.33 | 86.28 | 114.66 | 86364.19 | 95.63 |
|
||||
| 16 | 2 | 2582.13 | 40.35 | 71.64 | 86.30 | 143.29 | 336851.48 | 356.24 |
|
||||
| 64 | 2 | 2588.96 | 40.45 | 72.19 | 86.69 | 144.38 | 1506495.22 | 418.58 |
|
||||
|
||||
## Evidence
|
||||
|
||||
- Per-case metrics: `rdma_case_metrics.csv`
|
||||
- Machine-readable model: `rdma_demand_model.json`
|
||||
- Scout HCA source: `scout/{head,worker}/rdma.csv`
|
||||
- Confirm HCA source: `confirm/{head,worker}/rdma.csv`
|
||||
- Exact windows: `scout/case_windows.csv`, `confirm/case_windows.csv`
|
||||
- Benchmark summaries: `scout/bench_summary.csv`, `confirm/bench_summary.csv`
|
||||
@ -0,0 +1 @@
|
||||
CONFIRM_CONCURRENCIES='4 16 64'
|
||||
@ -0,0 +1,13 @@
|
||||
run_id=dsv4pro-phase2_5-20260801-130007
|
||||
experiment=dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||
created_at=2026-08-01T13:00:07+0800
|
||||
git_commit=c5fa700c50c0cc1b9e20d4d2d3f03bd485ba0dbf
|
||||
git_dirty=0
|
||||
isl=65536
|
||||
scout_osl=1
|
||||
confirm_osl=1024
|
||||
scout_concurrencies=1 4 16 32 64
|
||||
scout_repetitions=1
|
||||
confirm_repetitions=2
|
||||
target_rail_gbps=400
|
||||
practical_rail_gbps=360
|
||||
@ -0,0 +1,4 @@
|
||||
# case_id stage isl osl concurrency prompt_multiplier min_prompts warmup_requests note
|
||||
rdma_confirm_i65536_o1024_c4 rdma_confirm 65536 1024 4 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
rdma_confirm_i65536_o1024_c16 rdma_confirm 65536 1024 16 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
rdma_confirm_i65536_o1024_c64 rdma_confirm 65536 1024 64 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
|
@ -0,0 +1,6 @@
|
||||
# case_id stage isl osl concurrency prompt_multiplier min_prompts warmup_requests note
|
||||
rdma_scout_i65536_o1_c1 rdma_scout 65536 1 1 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c4 rdma_scout 65536 1 4 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c16 rdma_scout 65536 1 16 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c32 rdma_scout 65536 1 32 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c64 rdma_scout 65536 1 64 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
|
@ -0,0 +1,6 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||
fixed_rdma_scout_i65536_o1_c1,fixed,rdma_scout_i65536_o1_c1,,rdma_scout,1,65536,1,1,1,0,COMPLETED,,0,2026-08-01T13:05:54+0800,2026-08-01T13:06:53+0800,59.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,24.1862671909621,bench_main_marker_plus_duration,1,0,24.1862671909621,0.9992315809294645,,65536,1,0.04134577659729481,2709.6368150803128,0.04134577659729481,2709.67816085691,,24167.682002007496,24167.682002007496,24167.682002007496,24167.682002007496,24167.605170048773,24167.605170048773,24167.605170048773,24167.605170048773,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/bench.log,fixed_rdma_scout_i65536_o1_c1
|
||||
fixed_rdma_scout_i65536_o1_c16,fixed,rdma_scout_i65536_o1_c16,,rdma_scout,1,65536,1,16,16,0,COMPLETED,,0,2026-08-01T13:09:09+0800,2026-08-01T13:15:34+0800,385.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,351.42641380499117,bench_main_marker_plus_duration,16,0,351.42641380499117,8.628873239905923,,1048576,16,0.045528734811830354,2983.771164628114,0.045528734811830354,2983.816693362926,,189525.8736173746,189769.315014506,336717.50638925005,348428.8343418157,189525.84591050618,189769.29299000767,336717.4506017618,348428.77078671294,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/bench.log,fixed_rdma_scout_i65536_o1_c16
|
||||
fixed_rdma_scout_i65536_o1_c32,fixed,rdma_scout_i65536_o1_c32,,rdma_scout,1,65536,1,32,32,0,COMPLETED,,0,2026-08-01T13:15:40+0800,2026-08-01T13:27:58+0800,738.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.8175486620166,bench_main_marker_plus_duration,32,0,702.8175486620166,16.63411701870453,,2097152,32,0.04553102019282209,2983.9209393567885,0.04553102019282209,2983.9664703769813,,365335.9171013453,365369.92930600536,671139.3307480408,696643.7062339701,365335.8620414692,365369.8761375272,671139.2772838357,696643.6536653765,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/bench.log,fixed_rdma_scout_i65536_o1_c32
|
||||
fixed_rdma_scout_i65536_o1_c4,fixed,rdma_scout_i65536_o1_c4,,rdma_scout,1,65536,1,4,4,0,COMPLETED,,0,2026-08-01T13:06:59+0800,2026-08-01T13:09:02+0800,123.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,89.45394851703895,bench_main_marker_plus_duration,4,0,89.45394851703895,2.5881014457616573,,262144,4,0.044715745546302976,2930.491100122512,0.044715745546302976,2930.535815868058,,57878.973371509346,58521.736774011515,86526.744662947,88841.4666661853,57878.91561826109,58521.6828455159,86526.69315790116,88841.41537077782,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/bench.log,fixed_rdma_scout_i65536_o1_c4
|
||||
fixed_rdma_scout_i65536_o1_c64,fixed,rdma_scout_i65536_o1_c64,,rdma_scout,1,65536,1,64,64,0,COMPLETED,,0,2026-08-01T13:28:04+0800,2026-08-01T13:52:01+0800,1437.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1402.17612864502,bench_main_marker_plus_duration,64,0,1402.17612864502,32.732392604632125,,4194304,64,0.04564333873080967,2991.2818470623424,0.04564333873080967,2991.327490401073,,717134.0553695618,717609.0070659702,1337131.160498035,1390460.9022671543,717134.0051560618,717608.9713889814,1337131.1103800863,1390460.8547802733,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/bench.log,fixed_rdma_scout_i65536_o1_c64
|
||||
|
@ -0,0 +1,21 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.006126848,bench_main_marker_plus_duration,head,mlx5_0,22,203533427700.0,203651710996.0,70.77538224308064,70.81651330239592,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.006126848,bench_main_marker_plus_duration,head,mlx5_3,22,204100301340.0,204100542584.0,70.97250317308169,70.97258706169157,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.352313088,bench_main_marker_plus_duration,worker,mlx5_0,23,207229229632.0,207304460944.0,70.99227518955745,71.01804781832155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.352313088,bench_main_marker_plus_duration,worker,mlx5_3,23,207674858740.0,207674728976.0,71.14493813350504,71.14489367915073,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,350.585191424,bench_main_marker_plus_duration,head,mlx5_0,321,3499319374812.0,3499321894556.0,79.850934047694,79.85099154570729,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,350.585191424,bench_main_marker_plus_duration,head,mlx5_3,321,3499670238252.0,3499687020088.0,79.85894039704549,79.85932334159443,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,349.898471936,bench_main_marker_plus_duration,worker,mlx5_0,331,3496131787252.0,3496130977288.0,79.93477120166395,79.93475268282916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,349.898471936,bench_main_marker_plus_duration,worker,mlx5_3,331,3496481190036.0,3496480578468.0,79.94275986836642,79.94274588561318,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,701.184411136,bench_main_marker_plus_duration,head,mlx5_0,641,7008726918884.0,7008731821472.0,79.96443511947507,79.96449105440941,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,701.184411136,bench_main_marker_plus_duration,head,mlx5_3,641,7009132360248.0,7009232426556.0,79.9690609081556,79.97020259135803,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.616064768,bench_main_marker_plus_duration,worker,mlx5_0,663,7016359452164.0,7016357965840.0,79.88840339972316,79.88838647640956,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.616064768,bench_main_marker_plus_duration,worker,mlx5_3,663,7016694022728.0,7016692492508.0,79.89221282658687,79.89219540347258,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.716937728,bench_main_marker_plus_duration,head,mlx5_0,82,866863327600.0,866884788736.0,78.16891338226691,78.17084862814438,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.716937728,bench_main_marker_plus_duration,head,mlx5_3,82,867282690684.0,867282482924.0,78.20672921268122,78.20671047804,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.006032896,bench_main_marker_plus_duration,worker,mlx5_0,84,861825370952.0,861825497804.0,78.34239018322312,78.34240171443258,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.006032896,bench_main_marker_plus_duration,worker,mlx5_3,84,862244987252.0,862244674092.0,78.38053450457852,78.38050603743919,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.218045952,bench_main_marker_plus_duration,head,mlx5_0,1280,14005295392512.0,14005304292920.0,79.96069096010922,79.96074177537255,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.218045952,bench_main_marker_plus_duration,head,mlx5_3,1280,14005533397520.0,14005553945192.0,79.96204980648541,79.96216711968765,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.136956672,bench_main_marker_plus_duration,worker,mlx5_0,1321,14008943976736.0,14008941122316.0,79.9861507329604,79.98613443522456,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.136956672,bench_main_marker_plus_duration,worker,mlx5_3,1321,14009382587800.0,14009379963032.0,79.98865504811339,79.98864006160981,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
|
@ -0,0 +1,6 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,1785560786217761024,1785560810404027904,24.18626688,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/meta.json
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,1785560979903596032,1785561331330009856,351.426413824,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/meta.json
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,1785561372166763008,1785562074984312064,702.817549056,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/meta.json
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,1785560850521112832,1785560939975062016,89.453949184,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/meta.json
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1785562116631968000,1785563518808096768,1402.176128768,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/meta.json
|
||||
|
@ -0,0 +1,37 @@
|
||||
wall_time_ns,node,collector,status,pid
|
||||
1785560744455556276,head,gpu_samples.csv,STARTED,2168337
|
||||
1785560744470800548,head,rdma.csv,STARTED,2168348
|
||||
1785560744485157848,head,dcgm_dmon.log,STARTED,2168362
|
||||
1785560744499919157,head,mpstat.log,STARTED,2168380
|
||||
1785560744514378735,head,pidstat.log,STARTED,2168400
|
||||
1785560744529055413,head,sar_net.log,STARTED,2168428
|
||||
1785560744543220536,head,perf_stat.log,STARTED,2168451
|
||||
1785560744559383367,head,docker_top.log,STARTED,2168486
|
||||
1785560744579614511,head,numa_samples.csv,STARTED,2168561
|
||||
1785560744602206477,worker,gpu_samples.csv,STARTED,2168604
|
||||
1785560744619402981,worker,rdma.csv,STARTED,2168664
|
||||
1785560744634896197,worker,dcgm_dmon.log,STARTED,2168717
|
||||
1785560744650285593,worker,mpstat.log,STARTED,2168763
|
||||
1785560744666005300,worker,pidstat.log,STARTED,2168775
|
||||
1785560744681017748,worker,sar_net.log,STARTED,2168787
|
||||
1785560744694432153,worker,perf_stat.log,STARTED,2168796
|
||||
1785560744709522681,worker,docker_top.log,STARTED,2168808
|
||||
1785560744723983222,worker,numa_samples.csv,STARTED,2168818
|
||||
1785563532345901412,head,gpu_samples.csv,STOPPED,2168337
|
||||
1785563532350635898,head,rdma.csv,STOPPED,2168348
|
||||
1785563532357352405,head,dcgm_dmon.log,STOPPED,2168362
|
||||
1785563532362419425,head,mpstat.log,STOPPED,2168380
|
||||
1785563532367417490,head,pidstat.log,STOPPED,2168400
|
||||
1785563532372197961,head,sar_net.log,STOPPED,2168428
|
||||
1785563532541034088,head,perf_stat.log,STOPPED,2168451
|
||||
1785563532545727806,head,docker_top.log,STOPPED,2168486
|
||||
1785563532550309298,head,numa_samples.csv,STOPPED,2168561
|
||||
1785563532554473337,worker,gpu_samples.csv,STOPPED,2168604
|
||||
1785563532558689660,worker,rdma.csv,STOPPED,2168664
|
||||
1785563532562663636,worker,dcgm_dmon.log,STOPPED,2168717
|
||||
1785563532566892350,worker,mpstat.log,STOPPED,2168763
|
||||
1785563532571076626,worker,pidstat.log,STOPPED,2168775
|
||||
1785563532575164431,worker,sar_net.log,STOPPED,2168787
|
||||
1785563532579199145,worker,perf_stat.log,STOPPED,2168796
|
||||
1785563532583371937,worker,docker_top.log,STOPPED,2168808
|
||||
1785563532587359055,worker,numa_samples.csv,STOPPED,2168818
|
||||
|
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -400,7 +400,7 @@
|
||||
<div class="document-header__meta">
|
||||
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||
<span>资源：16 × RTX PRO 6000 Blackwell</span>
|
||||
<span>版本：2026-07-31 13:40:03 CST</span>
|
||||
<span>版本：2026-08-01 02:45:00 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
@ -415,7 +415,7 @@
|
||||
<article id="document-content">
|
||||
<h1>6000D 双机 DeepSeek-V4-Pro 推理优化计划</h1>
|
||||
<blockquote>
|
||||
<p>适用环境：<code>174.1.51.5 + 174.1.51.7</code>，每台 8 张 RTX PRO 6000 Blackwell Server Edition<br>当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例<br>当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离<br>计划版本：2026-07-31 16:46:05 CST</p>
|
||||
<p>适用环境：<code>174.1.51.5 + 174.1.51.7</code>，每台 8 张 RTX PRO 6000 Blackwell Server Edition<br>当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例<br>当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离<br>计划版本：2026-08-01 02:45:00 CST</p>
|
||||
</blockquote>
|
||||
<h2>当前执行状态与阶段档案</h2>
|
||||
<table>
|
||||
@ -448,6 +448,14 @@
|
||||
<a href="./phase2_code.html">打开 Phase 2 代码详解</a>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>DeepSeek-V4-Pro / 双机 Pro6000D / SGLang RDMA 需求建模与并发拐点</td>
|
||||
<td>Phase 2.5 已完成；Scout 5/5、Confirm 6/6，C=16 进入平台，拟合上限 80.32 Gbit/s/rail</td>
|
||||
<td>
|
||||
<a href="./phase2_5_exp.html">打开 Phase 2.5 实验档案</a><br>
|
||||
<a href="./phase2_5_code.html">打开 Phase 2.5 代码详解</a>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
<p>
|
||||
<strong>阶段档案生成门禁：</strong>Phase 尚未产出正式阶段结果、完成汇总和汇报确认前，
|
||||
@ -785,6 +793,24 @@ Phase 2 回答“哪个硬件/Host/通信资源在什么 Case 中升高，以及
|
||||
Phase 3 只捕获短 TP16 Timeline，回答具体 Kernel、Scheduler gap、Collective、
|
||||
Rank 同步及计算/通信重叠，不重复 Phase 2 的长时间轻量采样。
|
||||
</p>
|
||||
<h2>6.5 Phase 2.5：RDMA 需求建模与并发拐点</h2>
|
||||
<p>
|
||||
Phase 2 已证明当前代表负载最高约 83.5 Gbit/s/rail，但单个固定 Case 不能回答
|
||||
“提高并发后是否还能继续逼近 400G”。Phase 2.5 保持 TP16/EP2 和服务参数不变，
|
||||
先用 <code>64K → 1</code> 的 C=1/4/16/32/64 隔离 Prefill，拟合
|
||||
<code>rail_gbps ≈ input_tps × bytes_per_input_token_per_rail × 8</code>；
|
||||
再自动选择平台前、拐点和最大稳定并发，对 <code>64K → 1K</code> 重复两次确认。
|
||||
</p>
|
||||
<p>
|
||||
正式 Run <code>dsv4pro-phase2_5-20260801-130007</code> 已完成。64K→1 Scout 在 C=16
|
||||
达到约 2,984 input tok/s 与 79.90 Gbit/s/rail；C=32/64 均不再显著增长，拟合
|
||||
单 Rail 渐近上限为 80.32 Gbit/s。通信强度约为 3.332 MB/input-token/rail，
|
||||
要达到单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍。64K→1K
|
||||
两轮确认同样在 C=16 后进入平台，说明当前是模型计算/实现吞吐先饱和，而非 RDMA 链路先饱和。
|
||||
完整方法、逐点结果和证据路径见
|
||||
<a href="./phase2_5_exp.html">Phase 2.5 实验档案</a> 与
|
||||
<a href="./phase2_5_code.html">Phase 2.5 代码详解</a>。
|
||||
</p>
|
||||
<h2>7. Phase 3：时间线 Profiling（Nsight Systems 为主）</h2>
|
||||
<p>
|
||||
Nsight Systems、PyTorch Profiler 和 NVTX 是三件不同的东西：Nsight Systems
|
||||
|
||||
3
experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/.gitignore
vendored
Normal file
3
experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/.gitignore
vendored
Normal file
@ -0,0 +1,3 @@
|
||||
results/
|
||||
runtime/
|
||||
*.log
|
||||
@ -0,0 +1,35 @@
|
||||
# DeepSeek-V4-Pro 双机 SGLang RDMA 需求建模
|
||||
|
||||
Phase 2.5 在不改变 TP16/EP2 服务参数的前提下回答三个问题：
|
||||
|
||||
1. 64K 输入下，哪个并发是 Token TPS 与 RDMA 带宽的拐点；
|
||||
2. `64K -> 1K` 真实请求最多产生多少双 Rail 流量；
|
||||
3. 当前部署会先达到 GPU/模型吞吐平台，还是先打满 400G rail。
|
||||
|
||||
唯一入口：
|
||||
|
||||
```bash
|
||||
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||
|
||||
# 只展开命令，不启动服务
|
||||
DRY_RUN=1 RUN_ID=dsv4pro-phase2_5-dryrun-$(date +%Y%m%d-%H%M%S) \
|
||||
bash run_rdma_demand_modeling.sh all
|
||||
|
||||
# 正式运行
|
||||
RUN_ID=dsv4pro-phase2_5-$(date +%Y%m%d-%H%M%S)
|
||||
tmux new-session -d -s dsv4pro-phase2_5 \
|
||||
"RUN_ID=${RUN_ID} bash run_rdma_demand_modeling.sh all \
|
||||
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||
tmux attach -t dsv4pro-phase2_5
|
||||
```
|
||||
|
||||
脚本先跑 `64K -> 1` 的 C=1/4/16/32/64 探路点，再自动选择三个代表并发，
|
||||
对 `64K -> 1K` 重复两次。服务、benchmark 和 HCA 采集复用已经验证的
|
||||
Phase 1/2 实现；本目录只负责工作负载编排与带宽模型拟合。
|
||||
|
||||
最终高价值结果：
|
||||
|
||||
- `rdma_case_metrics.csv`：每个 Case 的 TPS、每 Rail/双 Rail 带宽和 bytes/token；
|
||||
- `rdma_demand_model.json`：拟合参数、并发拐点和理论需求；
|
||||
- `rdma_demand_report.md`：可直接用于阶段汇报的结论；
|
||||
- `recommendation.env`：Scout 自动选择的 Confirm 并发。
|
||||
@ -0,0 +1,36 @@
|
||||
# DeepSeek-V4-Pro two-node SGLang RDMA demand modeling.
|
||||
|
||||
EXPERIMENT="dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling"
|
||||
|
||||
PHASE1_DIR="${PHASE1_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../dsv4pro_pro6000d_2node_sglang_tp16_quick_map" && pwd)}"
|
||||
PHASE2_DIR="${PHASE2_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution" && pwd)}"
|
||||
PHASE2_ENTRY="${PHASE2_ENTRY:-${PHASE2_DIR}/run_hardware_contention_attribution.sh}"
|
||||
|
||||
# Workload policy. Scout isolates Prefill; confirm preserves the requested 64K -> 1K shape.
|
||||
ISL="${ISL:-65536}"
|
||||
SCOUT_OSL="${SCOUT_OSL:-1}"
|
||||
CONFIRM_OSL="${CONFIRM_OSL:-1024}"
|
||||
SCOUT_CONCURRENCIES="${SCOUT_CONCURRENCIES:-1 4 16 32 64}"
|
||||
DRY_RUN_CONFIRM_CONCURRENCIES="${DRY_RUN_CONFIRM_CONCURRENCIES:-16 32 64}"
|
||||
SCOUT_REPETITIONS="${SCOUT_REPETITIONS:-1}"
|
||||
CONFIRM_REPETITIONS="${CONFIRM_REPETITIONS:-2}"
|
||||
|
||||
# A point is a plateau candidate when both bandwidth and Input TPS gains stay below this value.
|
||||
PLATEAU_GAIN_PCT="${PLATEAU_GAIN_PCT:-5}"
|
||||
TARGET_RAIL_GBPS="${TARGET_RAIL_GBPS:-400}"
|
||||
PRACTICAL_RAIL_GBPS="${PRACTICAL_RAIL_GBPS:-360}"
|
||||
|
||||
# Reuse the verified Phase 2 collectors but skip its one-time communication microbenchmark.
|
||||
SAMPLE_INTERVAL_S="${SAMPLE_INTERVAL_S:-1}"
|
||||
IDLE_BASELINE_S="${IDLE_BASELINE_S:-5}"
|
||||
POST_RUN_COOLDOWN_S="${POST_RUN_COOLDOWN_S:-5}"
|
||||
CASE_COOLDOWN_S="${CASE_COOLDOWN_S:-3}"
|
||||
COLLECTOR_MAX_SECONDS="${COLLECTOR_MAX_SECONDS:-14400}"
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-7200}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/results}"
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
@ -0,0 +1,563 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fit model-serving RDMA demand from aligned benchmark and HCA counters."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
def read_csv(path: Path) -> list[dict[str, str]]:
|
||||
if not path.is_file():
|
||||
return []
|
||||
with path.open(encoding="utf-8", newline="") as handle:
|
||||
return list(csv.DictReader(handle))
|
||||
|
||||
|
||||
def write_csv(path: Path, rows: list[dict[str, Any]]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if not rows:
|
||||
path.write_text("", encoding="utf-8")
|
||||
return
|
||||
fields: list[str] = []
|
||||
for row in rows:
|
||||
for field in row:
|
||||
if field not in fields:
|
||||
fields.append(field)
|
||||
with path.open("w", encoding="utf-8", newline="") as handle:
|
||||
writer = csv.DictWriter(handle, fieldnames=fields)
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
|
||||
def number(value: Any) -> float | None:
|
||||
if value in (None, ""):
|
||||
return None
|
||||
try:
|
||||
parsed = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return parsed if math.isfinite(parsed) else None
|
||||
|
||||
|
||||
def integer(value: Any) -> int:
|
||||
parsed = number(value)
|
||||
return int(parsed) if parsed is not None else 0
|
||||
|
||||
|
||||
def percentile(values: Iterable[float], fraction: float) -> float | None:
|
||||
ordered = sorted(values)
|
||||
if not ordered:
|
||||
return None
|
||||
position = (len(ordered) - 1) * fraction
|
||||
lower = math.floor(position)
|
||||
upper = math.ceil(position)
|
||||
if lower == upper:
|
||||
return ordered[lower]
|
||||
return ordered[lower] + (ordered[upper] - ordered[lower]) * (position - lower)
|
||||
|
||||
|
||||
def mean(values: Iterable[float | None]) -> float | None:
|
||||
present = [value for value in values if value is not None]
|
||||
return statistics.fmean(present) if present else None
|
||||
|
||||
|
||||
def fmt(value: Any, digits: int = 2) -> str:
|
||||
parsed = number(value)
|
||||
return "-" if parsed is None else f"{parsed:.{digits}f}"
|
||||
|
||||
|
||||
def parse_iso_ns(value: str) -> int:
|
||||
return int(datetime.fromisoformat(value).timestamp() * 1_000_000_000)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CaseKey:
|
||||
case_id: str
|
||||
repetition: int
|
||||
|
||||
|
||||
def load_bench(result_dir: Path) -> dict[CaseKey, dict[str, str]]:
|
||||
rows = read_csv(result_dir / "bench_summary.csv")
|
||||
result: dict[CaseKey, dict[str, str]] = {}
|
||||
for row in rows:
|
||||
key = CaseKey(row.get("case_id", ""), integer(row.get("repetition")))
|
||||
if key.case_id:
|
||||
result[key] = row
|
||||
return result
|
||||
|
||||
|
||||
def load_windows(result_dir: Path) -> dict[CaseKey, dict[str, str]]:
|
||||
rows = read_csv(result_dir / "case_windows.csv")
|
||||
result: dict[CaseKey, dict[str, str]] = {}
|
||||
for row in rows:
|
||||
key = CaseKey(row.get("case_id", ""), integer(row.get("repetition")))
|
||||
if key.case_id:
|
||||
result[key] = row
|
||||
return result
|
||||
|
||||
|
||||
def load_case_rdma(
|
||||
result_dir: Path,
|
||||
) -> dict[CaseKey, list[dict[str, str]]]:
|
||||
grouped: dict[CaseKey, list[dict[str, str]]] = {}
|
||||
for row in read_csv(result_dir / "case_rdma_summary.csv"):
|
||||
key = CaseKey(row.get("case_id", ""), integer(row.get("repetition")))
|
||||
grouped.setdefault(key, []).append(row)
|
||||
return grouped
|
||||
|
||||
|
||||
def interval_rates(
|
||||
rows: list[dict[str, str]], started_ns: int, ended_ns: int
|
||||
) -> tuple[list[float], list[float]]:
|
||||
selected = []
|
||||
for row in rows:
|
||||
timestamp = integer(row.get("wall_time_ns"))
|
||||
if started_ns <= timestamp <= ended_ns:
|
||||
selected.append(row)
|
||||
selected.sort(key=lambda row: integer(row.get("wall_time_ns")))
|
||||
|
||||
xmit_rates: list[float] = []
|
||||
rcv_rates: list[float] = []
|
||||
for previous, current in zip(selected, selected[1:]):
|
||||
delta_ns = integer(current.get("wall_time_ns")) - integer(
|
||||
previous.get("wall_time_ns")
|
||||
)
|
||||
if delta_ns <= 0:
|
||||
continue
|
||||
duration_s = delta_ns / 1e9
|
||||
for field, output in (
|
||||
("port_xmit_data", xmit_rates),
|
||||
("port_rcv_data", rcv_rates),
|
||||
):
|
||||
first = number(previous.get(field))
|
||||
last = number(current.get(field))
|
||||
if first is None or last is None or last < first:
|
||||
continue
|
||||
# IB port data counters use 4-octet units.
|
||||
output.append((last - first) * 4 * 8 / duration_s / 1e9)
|
||||
return xmit_rates, rcv_rates
|
||||
|
||||
|
||||
def raw_rdma_by_node_hca(
|
||||
result_dir: Path,
|
||||
) -> dict[tuple[str, str], list[dict[str, str]]]:
|
||||
grouped: dict[tuple[str, str], list[dict[str, str]]] = {}
|
||||
for node in ("head", "worker"):
|
||||
for row in read_csv(result_dir / node / "rdma.csv"):
|
||||
hca = row.get("hca", "")
|
||||
if hca:
|
||||
grouped.setdefault((node, hca), []).append(row)
|
||||
return grouped
|
||||
|
||||
|
||||
def load_gpu_util(result_dir: Path) -> dict[CaseKey, float]:
|
||||
grouped: dict[CaseKey, list[float]] = {}
|
||||
for row in read_csv(result_dir / "case_gpu_node_summary.csv"):
|
||||
key = CaseKey(row.get("case_id", ""), integer(row.get("repetition")))
|
||||
value = number(row.get("gpu_util_pct_mean"))
|
||||
if value is not None:
|
||||
grouped.setdefault(key, []).append(value)
|
||||
return {key: statistics.fmean(values) for key, values in grouped.items()}
|
||||
|
||||
|
||||
def aggregate_stage(result_dir: Path, stage_name: str) -> list[dict[str, Any]]:
|
||||
bench = load_bench(result_dir)
|
||||
windows = load_windows(result_dir)
|
||||
case_rdma = load_case_rdma(result_dir)
|
||||
raw_rdma = raw_rdma_by_node_hca(result_dir)
|
||||
gpu_util = load_gpu_util(result_dir)
|
||||
output: list[dict[str, Any]] = []
|
||||
|
||||
for key, bench_row in sorted(
|
||||
bench.items(), key=lambda item: (integer(item[1].get("concurrency")), item[0].repetition)
|
||||
):
|
||||
window = windows.get(key)
|
||||
rdma_rows = case_rdma.get(key, [])
|
||||
if not window or not rdma_rows:
|
||||
continue
|
||||
started_ns = integer(window.get("started_ns")) or parse_iso_ns(
|
||||
window.get("started_at", "")
|
||||
)
|
||||
ended_ns = integer(window.get("ended_ns")) or parse_iso_ns(
|
||||
window.get("ended_at", "")
|
||||
)
|
||||
|
||||
rail_averages: list[float] = []
|
||||
rail_p95s: list[float] = []
|
||||
rail_maxima: list[float] = []
|
||||
node_averages: dict[str, float] = {}
|
||||
xmit_bytes: list[float] = []
|
||||
error_delta = 0.0
|
||||
for rdma_row in rdma_rows:
|
||||
node = rdma_row.get("node", "")
|
||||
hca = rdma_row.get("hca", "")
|
||||
average = number(rdma_row.get("xmit_gbps"))
|
||||
total_bytes = number(rdma_row.get("xmit_bytes"))
|
||||
if average is not None:
|
||||
rail_averages.append(average)
|
||||
node_averages[node] = node_averages.get(node, 0.0) + average
|
||||
if total_bytes is not None:
|
||||
xmit_bytes.append(total_bytes)
|
||||
rates, _ = interval_rates(raw_rdma.get((node, hca), []), started_ns, ended_ns)
|
||||
p95 = percentile(rates, 0.95)
|
||||
if p95 is not None:
|
||||
rail_p95s.append(p95)
|
||||
if rates:
|
||||
rail_maxima.append(max(rates))
|
||||
for field in (
|
||||
"port_xmit_wait_delta",
|
||||
"port_xmit_discards_delta",
|
||||
"port_rcv_errors_delta",
|
||||
"req_transport_retries_exceeded_delta",
|
||||
"req_rnr_retries_exceeded_delta",
|
||||
"roce_adp_retrans_delta",
|
||||
"roce_adp_retrans_to_delta",
|
||||
):
|
||||
error_delta += number(rdma_row.get(field)) or 0.0
|
||||
|
||||
total_input_tokens = number(bench_row.get("total_input_tokens"))
|
||||
total_output_tokens = number(bench_row.get("total_output_tokens"))
|
||||
total_tokens = (
|
||||
total_input_tokens + total_output_tokens
|
||||
if total_input_tokens is not None and total_output_tokens is not None
|
||||
else None
|
||||
)
|
||||
mean_xmit_bytes = mean(xmit_bytes)
|
||||
bytes_per_input = (
|
||||
mean_xmit_bytes / total_input_tokens
|
||||
if mean_xmit_bytes is not None and total_input_tokens
|
||||
else None
|
||||
)
|
||||
bytes_per_total = (
|
||||
mean_xmit_bytes / total_tokens
|
||||
if mean_xmit_bytes is not None and total_tokens
|
||||
else None
|
||||
)
|
||||
min_rail = min(rail_averages) if rail_averages else None
|
||||
max_rail = max(rail_averages) if rail_averages else None
|
||||
rail_imbalance = (
|
||||
(max_rail - min_rail) / statistics.fmean(rail_averages) * 100
|
||||
if min_rail is not None and max_rail is not None and rail_averages
|
||||
else None
|
||||
)
|
||||
output.append(
|
||||
{
|
||||
"stage": stage_name,
|
||||
"case_id": key.case_id,
|
||||
"repetition": key.repetition,
|
||||
"status": bench_row.get("status", ""),
|
||||
"isl": integer(bench_row.get("isl")),
|
||||
"osl": integer(bench_row.get("osl")),
|
||||
"concurrency": integer(bench_row.get("concurrency")),
|
||||
"completed": integer(bench_row.get("completed")),
|
||||
"duration_s": number(bench_row.get("duration_s")),
|
||||
"input_tps": number(bench_row.get("input_token_throughput")),
|
||||
"output_tps": number(bench_row.get("output_token_throughput")),
|
||||
"total_tps": number(bench_row.get("total_token_throughput")),
|
||||
"ttft_p95_ms": number(bench_row.get("ttft_p95_ms")),
|
||||
"tpot_p95_ms": number(bench_row.get("tpot_p95_ms")),
|
||||
"gpu_util_mean_pct": gpu_util.get(key),
|
||||
"rail_xmit_avg_gbps": mean(rail_averages),
|
||||
"rail_xmit_p95_gbps": mean(rail_p95s),
|
||||
"rail_xmit_max_gbps": max(rail_maxima) if rail_maxima else None,
|
||||
"dual_rail_xmit_avg_gbps": mean(node_averages.values()),
|
||||
"rail_imbalance_pct": rail_imbalance,
|
||||
"xmit_bytes_per_input_token_per_rail": bytes_per_input,
|
||||
"xmit_bytes_per_total_token_per_rail": bytes_per_total,
|
||||
"rdma_error_delta": error_delta,
|
||||
"measurement_started_at": window.get("started_at", ""),
|
||||
"measurement_ended_at": window.get("ended_at", ""),
|
||||
"source_result_dir": str(result_dir),
|
||||
}
|
||||
)
|
||||
return output
|
||||
|
||||
|
||||
def average_by_concurrency(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
grouped: dict[int, list[dict[str, Any]]] = {}
|
||||
for row in rows:
|
||||
if row.get("status") == "COMPLETED":
|
||||
grouped.setdefault(integer(row.get("concurrency")), []).append(row)
|
||||
output = []
|
||||
metrics = (
|
||||
"input_tps",
|
||||
"output_tps",
|
||||
"total_tps",
|
||||
"ttft_p95_ms",
|
||||
"tpot_p95_ms",
|
||||
"gpu_util_mean_pct",
|
||||
"rail_xmit_avg_gbps",
|
||||
"rail_xmit_p95_gbps",
|
||||
"rail_xmit_max_gbps",
|
||||
"dual_rail_xmit_avg_gbps",
|
||||
"rail_imbalance_pct",
|
||||
"xmit_bytes_per_input_token_per_rail",
|
||||
"xmit_bytes_per_total_token_per_rail",
|
||||
"rdma_error_delta",
|
||||
)
|
||||
for concurrency, group in sorted(grouped.items()):
|
||||
item: dict[str, Any] = {
|
||||
"stage": group[0].get("stage"),
|
||||
"concurrency": concurrency,
|
||||
"repetitions": len(group),
|
||||
}
|
||||
for metric in metrics:
|
||||
item[metric] = mean(number(row.get(metric)) for row in group)
|
||||
output.append(item)
|
||||
return output
|
||||
|
||||
|
||||
def fit_hyperbola(rows: list[dict[str, Any]], field: str) -> dict[str, float | None]:
|
||||
points = [
|
||||
(float(integer(row.get("concurrency"))), number(row.get(field)))
|
||||
for row in rows
|
||||
if integer(row.get("concurrency")) > 0 and number(row.get(field)) is not None
|
||||
]
|
||||
if len(points) < 2:
|
||||
return {"asymptote": None, "half_saturation_c": None, "rmse": None}
|
||||
|
||||
best: tuple[float, float, float] | None = None
|
||||
for exponent_step in range(-20, 121):
|
||||
k = 10 ** (exponent_step / 20)
|
||||
factors = [concurrency / (k + concurrency) for concurrency, _ in points]
|
||||
denominator = sum(factor * factor for factor in factors)
|
||||
if denominator == 0:
|
||||
continue
|
||||
asymptote = sum(
|
||||
factor * value for factor, (_, value) in zip(factors, points)
|
||||
) / denominator
|
||||
errors = [
|
||||
value - asymptote * factor
|
||||
for factor, (_, value) in zip(factors, points)
|
||||
]
|
||||
rmse = math.sqrt(sum(error * error for error in errors) / len(errors))
|
||||
if best is None or rmse < best[0]:
|
||||
best = (rmse, asymptote, k)
|
||||
assert best is not None
|
||||
return {
|
||||
"asymptote": best[1],
|
||||
"half_saturation_c": best[2],
|
||||
"rmse": best[0],
|
||||
}
|
||||
|
||||
|
||||
def linear_slope(rows: list[dict[str, Any]]) -> float | None:
|
||||
points = [
|
||||
(number(row.get("input_tps")), number(row.get("rail_xmit_avg_gbps")))
|
||||
for row in rows
|
||||
]
|
||||
present = [(x, y) for x, y in points if x is not None and y is not None]
|
||||
denominator = sum(x * x for x, _ in present)
|
||||
return sum(x * y for x, y in present) / denominator if denominator else None
|
||||
|
||||
|
||||
def select_concurrencies(
|
||||
scout: list[dict[str, Any]], plateau_gain_pct: float
|
||||
) -> tuple[list[int], int | None, int | None]:
|
||||
ordered = sorted(scout, key=lambda row: integer(row.get("concurrency")))
|
||||
if not ordered:
|
||||
return [], None, None
|
||||
plateau_c = None
|
||||
for previous, current in zip(ordered, ordered[1:]):
|
||||
previous_bw = number(previous.get("rail_xmit_avg_gbps")) or 0
|
||||
current_bw = number(current.get("rail_xmit_avg_gbps")) or 0
|
||||
previous_tps = number(previous.get("input_tps")) or 0
|
||||
current_tps = number(current.get("input_tps")) or 0
|
||||
bw_gain = (current_bw / previous_bw - 1) * 100 if previous_bw else math.inf
|
||||
tps_gain = (current_tps / previous_tps - 1) * 100 if previous_tps else math.inf
|
||||
if bw_gain < plateau_gain_pct and tps_gain < plateau_gain_pct:
|
||||
plateau_c = integer(current.get("concurrency"))
|
||||
break
|
||||
|
||||
best = max(
|
||||
ordered,
|
||||
key=lambda row: number(row.get("rail_xmit_avg_gbps")) or -1,
|
||||
)
|
||||
best_c = integer(best.get("concurrency"))
|
||||
cs = [integer(row.get("concurrency")) for row in ordered]
|
||||
anchor = plateau_c or best_c
|
||||
anchor_index = cs.index(anchor)
|
||||
candidates = {anchor, cs[max(0, anchor_index - 1)], cs[-1]}
|
||||
if len(candidates) < 3:
|
||||
candidates.update(cs[-3:])
|
||||
return sorted(candidates), plateau_c, best_c
|
||||
|
||||
|
||||
def build_model(
|
||||
scout_rows: list[dict[str, Any]],
|
||||
confirm_rows: list[dict[str, Any]],
|
||||
plateau_gain_pct: float,
|
||||
target_rail_gbps: float,
|
||||
practical_rail_gbps: float,
|
||||
) -> dict[str, Any]:
|
||||
scout = average_by_concurrency(scout_rows)
|
||||
confirm = average_by_concurrency(confirm_rows)
|
||||
selected, plateau_c, best_scout_c = select_concurrencies(
|
||||
scout, plateau_gain_pct
|
||||
)
|
||||
slope = linear_slope(scout)
|
||||
bytes_per_token = slope * 1e9 / 8 if slope is not None else None
|
||||
required_target_tps = target_rail_gbps / slope if slope else None
|
||||
required_practical_tps = practical_rail_gbps / slope if slope else None
|
||||
required_dual_400_tps = (target_rail_gbps / 2) / slope if slope else None
|
||||
bandwidth_fit = fit_hyperbola(scout, "rail_xmit_avg_gbps")
|
||||
input_tps_fit = fit_hyperbola(scout, "input_tps")
|
||||
max_confirm = (
|
||||
max(confirm, key=lambda row: number(row.get("rail_xmit_avg_gbps")) or -1)
|
||||
if confirm
|
||||
else None
|
||||
)
|
||||
asymptote = number(bandwidth_fit.get("asymptote"))
|
||||
if asymptote is None:
|
||||
verdict = "INSUFFICIENT_DATA"
|
||||
elif asymptote < practical_rail_gbps:
|
||||
verdict = "COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION"
|
||||
else:
|
||||
verdict = "RDMA_SATURATION_POSSIBLE_WITH_MODEL_LOAD"
|
||||
|
||||
return {
|
||||
"generated_at": datetime.now().astimezone().isoformat(),
|
||||
"formula": "rail_gbps ~= input_tps * bytes_per_input_token_per_rail * 8 / 1e9",
|
||||
"plateau_gain_pct": plateau_gain_pct,
|
||||
"target_rail_gbps": target_rail_gbps,
|
||||
"practical_rail_gbps": practical_rail_gbps,
|
||||
"scout": scout,
|
||||
"confirm": confirm,
|
||||
"selected_confirm_concurrencies": selected,
|
||||
"observed_plateau_concurrency": plateau_c,
|
||||
"best_scout_bandwidth_concurrency": best_scout_c,
|
||||
"linear_gbps_per_input_tps": slope,
|
||||
"modeled_bytes_per_input_token_per_rail": bytes_per_token,
|
||||
"required_input_tps_for_400g_per_rail": required_target_tps,
|
||||
"required_input_tps_for_360g_per_rail": required_practical_tps,
|
||||
"required_input_tps_for_400g_dual_rail_aggregate": required_dual_400_tps,
|
||||
"bandwidth_saturation_fit": bandwidth_fit,
|
||||
"input_tps_saturation_fit": input_tps_fit,
|
||||
"best_confirm_point": max_confirm,
|
||||
"verdict": verdict,
|
||||
}
|
||||
|
||||
|
||||
def write_recommendation(output_dir: Path, model: dict[str, Any]) -> None:
|
||||
values = model.get("selected_confirm_concurrencies", [])
|
||||
joined = " ".join(str(value) for value in values)
|
||||
(output_dir / "recommendation.env").write_text(
|
||||
f"CONFIRM_CONCURRENCIES='{joined}'\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def write_report(output_dir: Path, model: dict[str, Any]) -> None:
|
||||
scout = model.get("scout", [])
|
||||
confirm = model.get("confirm", [])
|
||||
lines = [
|
||||
"# Phase 2.5 RDMA Demand Model",
|
||||
"",
|
||||
f"- Generated: `{model['generated_at']}`",
|
||||
f"- Verdict: `{model['verdict']}`",
|
||||
f"- Model: `rail_gbps ~= input_tps x {fmt(model.get('modeled_bytes_per_input_token_per_rail') / 1e6 if model.get('modeled_bytes_per_input_token_per_rail') else None, 3)} MB/token/rail x 8`",
|
||||
f"- Required Input TPS for 400G per rail: `{fmt(model.get('required_input_tps_for_400g_per_rail'), 0)}`",
|
||||
f"- Required Input TPS for 360G per rail: `{fmt(model.get('required_input_tps_for_360g_per_rail'), 0)}`",
|
||||
f"- Required Input TPS for 400G dual-rail aggregate: `{fmt(model.get('required_input_tps_for_400g_dual_rail_aggregate'), 0)}`",
|
||||
f"- Fitted rail bandwidth ceiling: `{fmt(model.get('bandwidth_saturation_fit', {}).get('asymptote'))} Gbit/s`",
|
||||
f"- Observed plateau concurrency: `{model.get('observed_plateau_concurrency')}`",
|
||||
"",
|
||||
"## Scout: 64K -> 1",
|
||||
"",
|
||||
"| C | Input TPS | Rail avg | Rail P95 | Dual-rail avg | MB/input-token/rail | GPU util |",
|
||||
"|---:|---:|---:|---:|---:|---:|---:|",
|
||||
]
|
||||
for row in scout:
|
||||
bytes_per_token = number(row.get("xmit_bytes_per_input_token_per_rail"))
|
||||
lines.append(
|
||||
f"| {row['concurrency']} | {fmt(row.get('input_tps'))} | "
|
||||
f"{fmt(row.get('rail_xmit_avg_gbps'))} | {fmt(row.get('rail_xmit_p95_gbps'))} | "
|
||||
f"{fmt(row.get('dual_rail_xmit_avg_gbps'))} | "
|
||||
f"{fmt(bytes_per_token / 1e6 if bytes_per_token else None, 3)} | "
|
||||
f"{fmt(row.get('gpu_util_mean_pct'))} |"
|
||||
)
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Confirm: 64K -> 1K",
|
||||
"",
|
||||
"| C | Reps | Input TPS | Output TPS | Rail avg | Rail P95 | Dual-rail avg | TTFT P95 ms | TPOT P95 ms |",
|
||||
"|---:|---:|---:|---:|---:|---:|---:|---:|---:|",
|
||||
]
|
||||
)
|
||||
for row in confirm:
|
||||
lines.append(
|
||||
f"| {row['concurrency']} | {row['repetitions']} | {fmt(row.get('input_tps'))} | "
|
||||
f"{fmt(row.get('output_tps'))} | {fmt(row.get('rail_xmit_avg_gbps'))} | "
|
||||
f"{fmt(row.get('rail_xmit_p95_gbps'))} | {fmt(row.get('dual_rail_xmit_avg_gbps'))} | "
|
||||
f"{fmt(row.get('ttft_p95_ms'))} | {fmt(row.get('tpot_p95_ms'))} |"
|
||||
)
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Evidence",
|
||||
"",
|
||||
"- Per-case metrics: `rdma_case_metrics.csv`",
|
||||
"- Machine-readable model: `rdma_demand_model.json`",
|
||||
"- Scout HCA source: `scout/{head,worker}/rdma.csv`",
|
||||
"- Confirm HCA source: `confirm/{head,worker}/rdma.csv`",
|
||||
"- Exact windows: `scout/case_windows.csv`, `confirm/case_windows.csv`",
|
||||
"- Benchmark summaries: `scout/bench_summary.csv`, `confirm/bench_summary.csv`",
|
||||
]
|
||||
)
|
||||
(output_dir / "rdma_demand_report.md").write_text(
|
||||
"\n".join(lines) + "\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("mode", choices=("scout", "final"))
|
||||
parser.add_argument("--scout-dir", required=True, type=Path)
|
||||
parser.add_argument("--confirm-dir", type=Path)
|
||||
parser.add_argument("--output-dir", required=True, type=Path)
|
||||
parser.add_argument("--plateau-gain-pct", type=float, default=5.0)
|
||||
parser.add_argument("--target-rail-gbps", type=float, default=400.0)
|
||||
parser.add_argument("--practical-rail-gbps", type=float, default=360.0)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
scout_rows = aggregate_stage(args.scout_dir, "scout")
|
||||
if not scout_rows:
|
||||
raise SystemExit(f"no aligned scout rows found in {args.scout_dir}")
|
||||
confirm_rows: list[dict[str, Any]] = []
|
||||
if args.mode == "final":
|
||||
if args.confirm_dir is None:
|
||||
raise SystemExit("--confirm-dir is required in final mode")
|
||||
confirm_rows = aggregate_stage(args.confirm_dir, "confirm")
|
||||
if not confirm_rows:
|
||||
raise SystemExit(f"no aligned confirm rows found in {args.confirm_dir}")
|
||||
|
||||
model = build_model(
|
||||
scout_rows,
|
||||
confirm_rows,
|
||||
args.plateau_gain_pct,
|
||||
args.target_rail_gbps,
|
||||
args.practical_rail_gbps,
|
||||
)
|
||||
write_csv(args.output_dir / "rdma_case_metrics.csv", scout_rows + confirm_rows)
|
||||
(args.output_dir / "rdma_demand_model.json").write_text(
|
||||
json.dumps(model, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
|
||||
)
|
||||
write_recommendation(args.output_dir, model)
|
||||
write_report(args.output_dir, model)
|
||||
print(json.dumps(model, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -0,0 +1,264 @@
|
||||
#!/usr/bin/env bash
|
||||
# Measure how model token throughput translates into two-node RDMA bandwidth.
|
||||
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
ACTION="${1:-all}"
|
||||
RUN_ID="${RUN_ID:-dsv4pro-phase2_5-$(date +%Y%m%d-%H%M%S)}"
|
||||
RESULT_DIR="${RESULT_BASE}/${RUN_ID}"
|
||||
COMMAND_DIR="${RESULT_DIR}/commands"
|
||||
SCENARIO_DIR="${RESULT_DIR}/scenarios"
|
||||
ANALYZER="${SCRIPT_DIR}/rdma_demand_model.py"
|
||||
RUN_LOG_ACTIVE=0
|
||||
|
||||
log() {
|
||||
printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"
|
||||
}
|
||||
|
||||
print_command() {
|
||||
printf '%q ' "$@"
|
||||
printf '\n'
|
||||
}
|
||||
|
||||
enable_result_logging() {
|
||||
[[ "${DRY_RUN}" == "1" || "${RUN_LOG_ACTIVE}" == "1" ]] && return 0
|
||||
mkdir -p "${RESULT_DIR}"
|
||||
exec > >(tee -a "${RESULT_DIR}/run.log") 2>&1
|
||||
RUN_LOG_ACTIVE=1
|
||||
}
|
||||
|
||||
normalize_concurrencies() {
|
||||
tr ', ' '\n\n' <<< "$1" |
|
||||
sed '/^[[:space:]]*$/d' |
|
||||
awk '!seen[$0]++' |
|
||||
sort -n |
|
||||
paste -sd' ' -
|
||||
}
|
||||
|
||||
validate_config() {
|
||||
[[ -x "${PHASE2_ENTRY}" ]] || {
|
||||
log "ERROR: Phase 2 entry is missing: ${PHASE2_ENTRY}"
|
||||
return 1
|
||||
}
|
||||
[[ -f "${ANALYZER}" ]] || {
|
||||
log "ERROR: analyzer is missing: ${ANALYZER}"
|
||||
return 1
|
||||
}
|
||||
local name value
|
||||
for name in ISL SCOUT_OSL CONFIRM_OSL SCOUT_REPETITIONS CONFIRM_REPETITIONS \
|
||||
SAMPLE_INTERVAL_S COLLECTOR_MAX_SECONDS SCENARIO_TIMEOUT_S; do
|
||||
value="${!name}"
|
||||
[[ "${value}" =~ ^[1-9][0-9]*$ ]] || {
|
||||
log "ERROR: ${name} must be a positive integer: ${value}"
|
||||
return 1
|
||||
}
|
||||
done
|
||||
[[ "${PLATEAU_GAIN_PCT}" =~ ^[0-9]+([.][0-9]+)?$ ]] || {
|
||||
log "ERROR: PLATEAU_GAIN_PCT must be numeric"
|
||||
return 1
|
||||
}
|
||||
local concurrency
|
||||
for concurrency in $(normalize_concurrencies "${SCOUT_CONCURRENCIES}"); do
|
||||
[[ "${concurrency}" =~ ^[1-9][0-9]*$ ]] || {
|
||||
log "ERROR: invalid scout concurrency: ${concurrency}"
|
||||
return 1
|
||||
}
|
||||
done
|
||||
}
|
||||
|
||||
write_scenario_file() {
|
||||
local stage="$1"
|
||||
local osl="$2"
|
||||
local concurrencies="$3"
|
||||
local output="$4"
|
||||
mkdir -p "$(dirname "${output}")"
|
||||
printf '# case_id\tstage\tisl\tosl\tconcurrency\tprompt_multiplier\tmin_prompts\twarmup_requests\tnote\n' > "${output}"
|
||||
local concurrency case_id
|
||||
for concurrency in $(normalize_concurrencies "${concurrencies}"); do
|
||||
case_id="rdma_${stage}_i${ISL}_o${osl}_c${concurrency}"
|
||||
printf '%s\t%s\t%s\t%s\t%s\t1\t1\t0\t%s\n' \
|
||||
"${case_id}" "rdma_${stage}" "${ISL}" "${osl}" "${concurrency}" \
|
||||
"Phase 2.5 ${stage}: correlate model token rate with dual-rail RDMA" \
|
||||
>> "${output}"
|
||||
done
|
||||
}
|
||||
|
||||
case_ids_for() {
|
||||
local stage="$1"
|
||||
local osl="$2"
|
||||
local concurrencies="$3"
|
||||
local result=() concurrency
|
||||
for concurrency in $(normalize_concurrencies "${concurrencies}"); do
|
||||
result+=("rdma_${stage}_i${ISL}_o${osl}_c${concurrency}")
|
||||
done
|
||||
local joined
|
||||
joined="$(IFS=,; printf '%s' "${result[*]}")"
|
||||
printf '%s\n' "${joined}"
|
||||
}
|
||||
|
||||
run_phase2_stage() {
|
||||
local stage="$1"
|
||||
local scenario_file="$2"
|
||||
local case_ids="$3"
|
||||
local repetitions="$4"
|
||||
local stage_result="${RESULT_DIR}/${stage}"
|
||||
local command_file="${COMMAND_DIR}/${stage}.cmd.txt"
|
||||
local -a command=(
|
||||
env
|
||||
"DRY_RUN=${DRY_RUN}"
|
||||
"RUN_ID=${stage}"
|
||||
"RESULT_BASE=${RESULT_DIR}"
|
||||
"SCENARIO_FILE=${scenario_file}"
|
||||
"FIXED_CASE_IDS=${case_ids}"
|
||||
"RUN_MIXED_CASE=0"
|
||||
"RUN_COMMUNICATION_BASELINE=0"
|
||||
"NUM_REPETITIONS=${repetitions}"
|
||||
"SAMPLE_INTERVAL_S=${SAMPLE_INTERVAL_S}"
|
||||
"IDLE_BASELINE_S=${IDLE_BASELINE_S}"
|
||||
"POST_RUN_COOLDOWN_S=${POST_RUN_COOLDOWN_S}"
|
||||
"CASE_COOLDOWN_S=${CASE_COOLDOWN_S}"
|
||||
"COLLECTOR_MAX_SECONDS=${COLLECTOR_MAX_SECONDS}"
|
||||
"SCENARIO_TIMEOUT_S=${SCENARIO_TIMEOUT_S}"
|
||||
bash "${PHASE2_ENTRY}" all
|
||||
)
|
||||
|
||||
mkdir -p "${COMMAND_DIR}"
|
||||
print_command "${command[@]}" > "${command_file}"
|
||||
log "START stage=${stage} result=${stage_result}"
|
||||
"${command[@]}"
|
||||
log "DONE stage=${stage} result=${stage_result}"
|
||||
}
|
||||
|
||||
run_scout() {
|
||||
local scenario_file="${SCENARIO_DIR}/scout.tsv"
|
||||
local concurrencies case_ids
|
||||
concurrencies="$(normalize_concurrencies "${SCOUT_CONCURRENCIES}")"
|
||||
write_scenario_file scout "${SCOUT_OSL}" "${concurrencies}" "${scenario_file}"
|
||||
case_ids="$(case_ids_for scout "${SCOUT_OSL}" "${concurrencies}")"
|
||||
run_phase2_stage scout "${scenario_file}" "${case_ids}" "${SCOUT_REPETITIONS}"
|
||||
|
||||
if [[ "${DRY_RUN}" != "1" ]]; then
|
||||
python3 "${ANALYZER}" scout \
|
||||
--scout-dir "${RESULT_DIR}/scout" \
|
||||
--output-dir "${RESULT_DIR}" \
|
||||
--plateau-gain-pct "${PLATEAU_GAIN_PCT}" \
|
||||
--target-rail-gbps "${TARGET_RAIL_GBPS}" \
|
||||
--practical-rail-gbps "${PRACTICAL_RAIL_GBPS}"
|
||||
fi
|
||||
}
|
||||
|
||||
confirm_concurrencies() {
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
normalize_concurrencies "${DRY_RUN_CONFIRM_CONCURRENCIES}"
|
||||
return 0
|
||||
fi
|
||||
# shellcheck source=/dev/null
|
||||
source "${RESULT_DIR}/recommendation.env"
|
||||
normalize_concurrencies "${CONFIRM_CONCURRENCIES}"
|
||||
}
|
||||
|
||||
run_confirm() {
|
||||
local scenario_file="${SCENARIO_DIR}/confirm.tsv"
|
||||
local concurrencies case_ids
|
||||
concurrencies="$(confirm_concurrencies)"
|
||||
write_scenario_file confirm "${CONFIRM_OSL}" "${concurrencies}" "${scenario_file}"
|
||||
case_ids="$(case_ids_for confirm "${CONFIRM_OSL}" "${concurrencies}")"
|
||||
run_phase2_stage confirm "${scenario_file}" "${case_ids}" "${CONFIRM_REPETITIONS}"
|
||||
|
||||
if [[ "${DRY_RUN}" != "1" ]]; then
|
||||
python3 "${ANALYZER}" final \
|
||||
--scout-dir "${RESULT_DIR}/scout" \
|
||||
--confirm-dir "${RESULT_DIR}/confirm" \
|
||||
--output-dir "${RESULT_DIR}" \
|
||||
--plateau-gain-pct "${PLATEAU_GAIN_PCT}" \
|
||||
--target-rail-gbps "${TARGET_RAIL_GBPS}" \
|
||||
--practical-rail-gbps "${PRACTICAL_RAIL_GBPS}"
|
||||
fi
|
||||
}
|
||||
|
||||
write_manifest() {
|
||||
local commit dirty
|
||||
commit="$(git -C "${REPO_ROOT}" rev-parse HEAD 2>/dev/null || printf unknown)"
|
||||
dirty=0
|
||||
[[ -n "$(git -C "${REPO_ROOT}" status --porcelain 2>/dev/null || true)" ]] && dirty=1
|
||||
mkdir -p "${RESULT_DIR}"
|
||||
cat > "${RESULT_DIR}/run_manifest.txt" <<EOF
|
||||
run_id=${RUN_ID}
|
||||
experiment=${EXPERIMENT}
|
||||
created_at=$(date '+%Y-%m-%dT%H:%M:%S%z')
|
||||
git_commit=${commit}
|
||||
git_dirty=${dirty}
|
||||
isl=${ISL}
|
||||
scout_osl=${SCOUT_OSL}
|
||||
confirm_osl=${CONFIRM_OSL}
|
||||
scout_concurrencies=$(normalize_concurrencies "${SCOUT_CONCURRENCIES}")
|
||||
scout_repetitions=${SCOUT_REPETITIONS}
|
||||
confirm_repetitions=${CONFIRM_REPETITIONS}
|
||||
target_rail_gbps=${TARGET_RAIL_GBPS}
|
||||
practical_rail_gbps=${PRACTICAL_RAIL_GBPS}
|
||||
EOF
|
||||
}
|
||||
|
||||
stop_service() {
|
||||
env \
|
||||
"RUN_ID=${RUN_ID}-stop" \
|
||||
"RESULT_BASE=${RESULT_DIR}" \
|
||||
"RUN_COMMUNICATION_BASELINE=0" \
|
||||
bash "${PHASE2_ENTRY}" stop || true
|
||||
}
|
||||
|
||||
run_all() {
|
||||
validate_config
|
||||
enable_result_logging
|
||||
write_manifest
|
||||
trap 'stop_service' INT TERM
|
||||
run_scout
|
||||
run_confirm
|
||||
trap - INT TERM
|
||||
log "Phase 2.5 complete: ${RESULT_DIR}"
|
||||
}
|
||||
|
||||
main() {
|
||||
case "${ACTION}" in
|
||||
audit)
|
||||
validate_config
|
||||
log "configuration valid"
|
||||
;;
|
||||
scout)
|
||||
validate_config
|
||||
enable_result_logging
|
||||
write_manifest
|
||||
run_scout
|
||||
;;
|
||||
confirm)
|
||||
validate_config
|
||||
enable_result_logging
|
||||
run_confirm
|
||||
;;
|
||||
summarize)
|
||||
python3 "${ANALYZER}" final \
|
||||
--scout-dir "${RESULT_DIR}/scout" \
|
||||
--confirm-dir "${RESULT_DIR}/confirm" \
|
||||
--output-dir "${RESULT_DIR}" \
|
||||
--plateau-gain-pct "${PLATEAU_GAIN_PCT}" \
|
||||
--target-rail-gbps "${TARGET_RAIL_GBPS}" \
|
||||
--practical-rail-gbps "${PRACTICAL_RAIL_GBPS}"
|
||||
;;
|
||||
stop)
|
||||
stop_service
|
||||
;;
|
||||
all)
|
||||
run_all
|
||||
;;
|
||||
*)
|
||||
printf 'Usage: %s {audit|scout|confirm|summarize|stop|all}\n' "$0" >&2
|
||||
return 2
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
MODULE_PATH = Path(__file__).with_name("rdma_demand_model.py")
|
||||
SPEC = importlib.util.spec_from_file_location("rdma_demand_model", MODULE_PATH)
|
||||
assert SPEC and SPEC.loader
|
||||
MODEL = importlib.util.module_from_spec(SPEC)
|
||||
sys.modules[SPEC.name] = MODEL
|
||||
SPEC.loader.exec_module(MODEL)
|
||||
|
||||
|
||||
class RdmaDemandModelTest(unittest.TestCase):
|
||||
def test_interval_rate_uses_four_octet_counter_units(self):
|
||||
rows = [
|
||||
{
|
||||
"wall_time_ns": "1000000000",
|
||||
"port_xmit_data": "100",
|
||||
"port_rcv_data": "200",
|
||||
},
|
||||
{
|
||||
"wall_time_ns": "2000000000",
|
||||
"port_xmit_data": "100000100",
|
||||
"port_rcv_data": "100000200",
|
||||
},
|
||||
]
|
||||
xmit, rcv = MODEL.interval_rates(rows, 0, 3_000_000_000)
|
||||
self.assertAlmostEqual(xmit[0], 3.2)
|
||||
self.assertAlmostEqual(rcv[0], 3.2)
|
||||
|
||||
def test_linear_slope_recovers_bytes_per_token(self):
|
||||
rows = [
|
||||
{"input_tps": 1000, "rail_xmit_avg_gbps": 26.8},
|
||||
{"input_tps": 2000, "rail_xmit_avg_gbps": 53.6},
|
||||
{"input_tps": 3000, "rail_xmit_avg_gbps": 80.4},
|
||||
]
|
||||
slope = MODEL.linear_slope(rows)
|
||||
self.assertAlmostEqual(slope, 0.0268)
|
||||
self.assertAlmostEqual(slope * 1e9 / 8, 3_350_000)
|
||||
|
||||
def test_plateau_selects_previous_plateau_and_max(self):
|
||||
rows = [
|
||||
{"concurrency": 1, "input_tps": 1000, "rail_xmit_avg_gbps": 27},
|
||||
{"concurrency": 4, "input_tps": 2400, "rail_xmit_avg_gbps": 64},
|
||||
{"concurrency": 16, "input_tps": 3100, "rail_xmit_avg_gbps": 83},
|
||||
{"concurrency": 32, "input_tps": 3150, "rail_xmit_avg_gbps": 84},
|
||||
{"concurrency": 64, "input_tps": 3160, "rail_xmit_avg_gbps": 84.2},
|
||||
]
|
||||
selected, plateau, best = MODEL.select_concurrencies(rows, 5.0)
|
||||
self.assertEqual(plateau, 32)
|
||||
self.assertEqual(best, 64)
|
||||
self.assertEqual(selected, [16, 32, 64])
|
||||
|
||||
def test_report_and_recommendation_are_created(self):
|
||||
rows = [
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 16,
|
||||
"repetitions": 1,
|
||||
"input_tps": 3000,
|
||||
"rail_xmit_avg_gbps": 80,
|
||||
"rail_xmit_p95_gbps": 90,
|
||||
"dual_rail_xmit_avg_gbps": 160,
|
||||
"xmit_bytes_per_input_token_per_rail": 3_333_333,
|
||||
"gpu_util_mean_pct": 99,
|
||||
}
|
||||
]
|
||||
model = {
|
||||
"generated_at": "2026-08-01T00:00:00+08:00",
|
||||
"verdict": "TEST",
|
||||
"modeled_bytes_per_input_token_per_rail": 3_333_333,
|
||||
"required_input_tps_for_400g_per_rail": 15000,
|
||||
"required_input_tps_for_360g_per_rail": 13500,
|
||||
"required_input_tps_for_400g_dual_rail_aggregate": 7500,
|
||||
"bandwidth_saturation_fit": {"asymptote": 100},
|
||||
"observed_plateau_concurrency": 32,
|
||||
"scout": rows,
|
||||
"confirm": [],
|
||||
"selected_confirm_concurrencies": [16, 32, 64],
|
||||
}
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
output = Path(temporary)
|
||||
MODEL.write_recommendation(output, model)
|
||||
MODEL.write_report(output, model)
|
||||
self.assertIn("16 32 64", (output / "recommendation.env").read_text())
|
||||
self.assertIn("15000", (output / "rdma_demand_report.md").read_text())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
3
experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/.gitignore
vendored
Normal file
3
experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/.gitignore
vendored
Normal file
@ -0,0 +1,3 @@
|
||||
results/
|
||||
runtime/
|
||||
*.log
|
||||
@ -278,16 +278,22 @@ start_service() {
|
||||
return 1
|
||||
}
|
||||
|
||||
stop_service_node() {
|
||||
request_stop_node() {
|
||||
local node="$1"
|
||||
local container_name="$2"
|
||||
local role="$3"
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
log "[DRY] ${node}: docker stop --time ${NSYS_EXPORT_TIMEOUT_S} ${container_name}"
|
||||
return 0
|
||||
fi
|
||||
run_on_node "${node}" \
|
||||
"docker stop --time '${NSYS_EXPORT_TIMEOUT_S}' '${container_name}' >/dev/null 2>&1 || true"
|
||||
}
|
||||
|
||||
collect_service_node() {
|
||||
local node="$1"
|
||||
local container_name="$2"
|
||||
local role="$3"
|
||||
[[ "${DRY_RUN}" == "1" ]] && return 0
|
||||
run_on_node "${node}" "docker logs '${container_name}' 2>&1" \
|
||||
> "${SERVICE_DIR}/${role}_server.log" 2>&1 || true
|
||||
run_on_node "${node}" "docker inspect '${container_name}' 2>/dev/null" \
|
||||
@ -305,12 +311,24 @@ fetch_worker_profiles() {
|
||||
}
|
||||
|
||||
stop_service() {
|
||||
stop_service_node "${HEAD_NODE}" "${HEAD_CONTAINER}" head
|
||||
stop_service_node "${WORKER_NODE}" "${WORKER_CONTAINER}" worker
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
request_stop_node "${HEAD_NODE}" "${HEAD_CONTAINER}"
|
||||
request_stop_node "${WORKER_NODE}" "${WORKER_CONTAINER}"
|
||||
return 0
|
||||
fi
|
||||
request_stop_node "${HEAD_NODE}" "${HEAD_CONTAINER}" &
|
||||
local head_stop_pid=$!
|
||||
request_stop_node "${WORKER_NODE}" "${WORKER_CONTAINER}" &
|
||||
local worker_stop_pid=$!
|
||||
wait "${head_stop_pid}"
|
||||
wait "${worker_stop_pid}"
|
||||
collect_service_node "${HEAD_NODE}" "${HEAD_CONTAINER}" head
|
||||
collect_service_node "${WORKER_NODE}" "${WORKER_CONTAINER}" worker
|
||||
fetch_worker_profiles
|
||||
}
|
||||
|
||||
validate_torch_traces() {
|
||||
[[ "${DRY_RUN}" == "1" ]] && return 0
|
||||
local count
|
||||
count="$(find "${RESULT_DIR}/profiles" -type f -name '*.trace.json.gz' | wc -l)"
|
||||
if (( count < TP_SIZE )); then
|
||||
@ -321,10 +339,12 @@ validate_torch_traces() {
|
||||
}
|
||||
|
||||
validate_nsys_reports() {
|
||||
local count
|
||||
[[ "${DRY_RUN}" == "1" ]] && return 0
|
||||
local count expected
|
||||
count="$(find "${RESULT_DIR}/profiles" -type f -name '*.nsys-rep' | wc -l)"
|
||||
if (( count < NNODES )); then
|
||||
log "ERROR: expected at least ${NNODES} Nsight reports, found ${count}"
|
||||
expected=$(( NNODES * CAPTURE_RANGES ))
|
||||
if (( count < expected )); then
|
||||
log "ERROR: expected at least ${expected} Nsight reports, found ${count}"
|
||||
return 1
|
||||
fi
|
||||
log "Nsight report validation passed: ${count} reports"
|
||||
@ -366,9 +386,10 @@ start_profile() {
|
||||
|
||||
wait_for_profile_stop() {
|
||||
local before="$1"
|
||||
local expected=$(( TP_SIZE / NNODES ))
|
||||
local waited=0
|
||||
while (( waited < PROFILE_STOP_WAIT_S )); do
|
||||
if (( $(profile_done_count) > before )); then
|
||||
if (( $(profile_done_count) >= before + expected )); then
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
@ -378,6 +399,26 @@ wait_for_profile_stop() {
|
||||
return 1
|
||||
}
|
||||
|
||||
server_decode_count() {
|
||||
run_on_node "${HEAD_NODE}" "docker logs '${HEAD_CONTAINER}' 2>&1" \
|
||||
| grep -c 'Decode batch' || true
|
||||
}
|
||||
|
||||
wait_for_decode_activity() {
|
||||
local before="$1"
|
||||
local bench_pid="$2"
|
||||
local waited=0
|
||||
while (( waited < 180 )); do
|
||||
if (( $(server_decode_count) > before )); then
|
||||
return 0
|
||||
fi
|
||||
kill -0 "${bench_pid}" 2>/dev/null || return 1
|
||||
sleep 1
|
||||
((waited+=1))
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
prepare_bench_command() {
|
||||
local output_file="$1"
|
||||
local isl="$2"
|
||||
@ -455,6 +496,47 @@ run_profiled_case() {
|
||||
[[ "${DRY_RUN}" == "1" ]] || wait_for_profile_stop "${before}"
|
||||
}
|
||||
|
||||
run_profiled_decode_case() {
|
||||
local case_id="$1"
|
||||
local steps="$2"
|
||||
local isl="$3"
|
||||
local osl="$4"
|
||||
local concurrency="$5"
|
||||
local prompts="$6"
|
||||
local seed="$7"
|
||||
local case_path="${CASE_DIR}/${case_id}"
|
||||
local decode_before=0
|
||||
local profile_before=0
|
||||
flush_cache
|
||||
mkdir -p "${case_path}"
|
||||
prepare_bench_command "${case_path}/bench.jsonl" \
|
||||
"${isl}" "${osl}" "${concurrency}" "${prompts}" "${seed}"
|
||||
print_command timeout --signal=TERM --kill-after=30s \
|
||||
"${PROFILE_TIMEOUT_S}s" "${BENCH_CMD[@]}" > "${case_path}/bench_cmd.txt"
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
log "[DRY] start decode workload, wait for Decode batch, then start Nsight range"
|
||||
printf '[DRY] '
|
||||
cat "${case_path}/bench_cmd.txt"
|
||||
start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}"
|
||||
return 0
|
||||
fi
|
||||
decode_before="$(server_decode_count)"
|
||||
timeout --signal=TERM --kill-after=30s "${PROFILE_TIMEOUT_S}s" \
|
||||
"${BENCH_CMD[@]}" > "${case_path}/bench.log" 2>&1 &
|
||||
local bench_pid=$!
|
||||
wait_for_bench_main "${case_path}/bench.log" "${bench_pid}" \
|
||||
&& wait_for_decode_activity "${decode_before}" "${bench_pid}" || {
|
||||
kill "${bench_pid}" 2>/dev/null || true
|
||||
wait "${bench_pid}" 2>/dev/null || true
|
||||
log "ERROR: ${case_id} did not reach active Decode"
|
||||
return 1
|
||||
}
|
||||
profile_before="$(profile_done_count)"
|
||||
start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}"
|
||||
wait_for_profile_stop "${profile_before}"
|
||||
wait "${bench_pid}"
|
||||
}
|
||||
|
||||
wait_for_bench_main() {
|
||||
local bench_log="$1"
|
||||
local bench_pid="$2"
|
||||
@ -474,6 +556,7 @@ run_mixed_profile() {
|
||||
local injection_id="${case_id}/prefill_injection"
|
||||
local background_path="${CASE_DIR}/${background_id}"
|
||||
local before=0
|
||||
local decode_before=0
|
||||
flush_cache
|
||||
mkdir -p "${background_path}"
|
||||
prepare_bench_command "${background_path}/bench.jsonl" \
|
||||
@ -481,16 +564,18 @@ run_mixed_profile() {
|
||||
print_command timeout --signal=TERM --kill-after=30s \
|
||||
"${PROFILE_TIMEOUT_S}s" "${BENCH_CMD[@]}" > "${background_path}/bench_cmd.txt"
|
||||
if [[ "${DRY_RUN}" == "1" ]]; then
|
||||
log "[DRY] start decode background, wait for main run, start profile, inject 128K prefill"
|
||||
log "[DRY] start decode background, wait for Decode batch, start profile, inject 128K prefill"
|
||||
start_profile "${case_id}" '["CUDA_PROFILER"]' "${NSYS_MIXED_STEPS}"
|
||||
run_bench "${injection_id}" "${MIXED_INJECTION_ISL}" \
|
||||
"${MIXED_INJECTION_OSL}" 1 1 5202
|
||||
return 0
|
||||
fi
|
||||
decode_before="$(server_decode_count)"
|
||||
timeout --signal=TERM --kill-after=30s "${PROFILE_TIMEOUT_S}s" \
|
||||
"${BENCH_CMD[@]}" > "${background_path}/bench.log" 2>&1 &
|
||||
local background_pid=$!
|
||||
wait_for_bench_main "${background_path}/bench.log" "${background_pid}" || {
|
||||
wait_for_bench_main "${background_path}/bench.log" "${background_pid}" \
|
||||
&& wait_for_decode_activity "${decode_before}" "${background_pid}" || {
|
||||
kill "${background_pid}" 2>/dev/null || true
|
||||
wait "${background_pid}" 2>/dev/null || true
|
||||
log "ERROR: mixed decode background did not enter main benchmark"
|
||||
@ -537,10 +622,16 @@ generate_nsys_stats() {
|
||||
relative="${report#${RESULT_DIR}/}"
|
||||
output="${report%.nsys-rep}.stats.txt"
|
||||
docker run --rm \
|
||||
-v "${RESULT_DIR}:/results:ro" \
|
||||
-v "${RESULT_DIR}:/results" \
|
||||
--entrypoint nsys "${DOCKER_IMAGE}" stats \
|
||||
--report cuda_gpu_kern_sum,cuda_api_sum,cuda_kern_exec_sum,nvtx_sum \
|
||||
"/results/${relative}" > "${output}" 2>&1 || true
|
||||
--force-export=true \
|
||||
--report cuda_gpu_kern_sum,cuda_api_sum,nvtx_sum \
|
||||
"/results/${relative}" > "${output}" 2>&1
|
||||
grep -q 'CUDA GPU Kernel Summary' "${output}" || {
|
||||
log "ERROR: nsys stats missing CUDA kernel summary for ${relative}"
|
||||
return 1
|
||||
}
|
||||
rm -f "${report%.nsys-rep}.sqlite"
|
||||
done < <(find "${RESULT_DIR}/profiles" -type f -name '*.nsys-rep' | sort)
|
||||
}
|
||||
|
||||
@ -586,7 +677,7 @@ run_all() {
|
||||
CLEANUP_ON_EXIT=1
|
||||
start_service
|
||||
write_manifest RUNNING
|
||||
run_profiled_case decode_control '["CUDA_PROFILER"]' \
|
||||
run_profiled_decode_case decode_control \
|
||||
"${NSYS_CONTROL_STEPS}" "${CONTROL_ISL}" "${CONTROL_OSL}" \
|
||||
"${CONTROL_CONCURRENCY}" "${CONTROL_PROMPTS}" 5301
|
||||
run_mixed_profile
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user