[Docs] finalize Phase 2.5 RDMA demand model
This commit is contained in:
Zhiyi Hong
parent
c5fa700c50
commit
e1719bd575
@ -1,5 +1,9 @@
|
||||
# sskj — 多平台大模型推理性能基准测试项目
|
||||
|
||||
> **更新（2026-08-01 15:24:52 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模。正式 Run `dsv4pro-phase2_5-20260801-130007` 完成 Scout 5/5 与 Confirm 6/6；64K Prefill 在 C=16 已进入约 2,984 input tok/s、79.90 Gbit/s/rail 的平台，C=32/64 不再显著增长。拟合通信强度为 3.332 MB/input-token/rail，单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍，因此当前是模型计算/实现吞吐先饱和，不是 RDMA 先饱和。新增 `phase2_5_exp.html`、`phase2_5_code.html`、精简证据集和可复用的模型部署 RDMA 需求评估流程；实验结束后双节点容器与 16 张 GPU 已清理。
|
||||
>
|
||||
> **更新（2026-08-01 02:40:00 CST）**
|
||||
>
|
||||
> 新增 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模唯一入口。实验保持现有 TP16/EP2 服务参数不变，先以 `64K -> 1` 的 C=1/4/16/32/64 建立 Input TPS 与每 Rail HCA 带宽关系，再自动选择平台前、拐点和最大稳定并发，对 `64K -> 1K` 重复确认。结果将给出每 Token 跨机字节数、400G 所需 Token TPS、并发饱和曲线和“模型计算先饱和还是 RDMA 先饱和”的机器可读结论；正式结果尚未生成，因此暂不创建 Phase 2.5 HTML 档案。
|
||||
|
||||
@ -547,6 +547,26 @@ mlx5_3 port 1 ==> eth3 (Up)</code></pre>
|
||||
<td>查看系统或进程在各 NUMA 节点上的内存分布。</td>
|
||||
<td>Phase 2 每 5 秒保存结构化 Node0/Node1 MiB，寻找跨 NUMA 内存放置。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>HCA Counter</strong></td>
|
||||
<td>网卡硬件维护的发送、接收、等待、丢弃和错误累计计数器。</td>
|
||||
<td>Phase 2.5 用 <code>mlx5_0/mlx5_3</code> 的 counter 差值计算正式 benchmark 窗口内的 RDMA Gbit/s。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>bytes/input-token/rail</strong></td>
|
||||
<td>模型每处理一个输入 token，平均要在一条 Rail 上发送的字节数。</td>
|
||||
<td>当前 DSV4-Pro TP16/EP2 Scout 拟合为约 3.332 MB/token/rail；换模型或并行策略必须重新标定。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>带宽平台 / 拐点</strong></td>
|
||||
<td>继续增加并发后，吞吐与网络带宽都几乎不再增长的位置。</td>
|
||||
<td>Phase 2.5 以相邻点的 Input TPS 和 Rail Mean 增益同时低于 5% 判断，当前拐点为 C=16。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>渐近线 / 饱和上限</strong></td>
|
||||
<td>饱和曲线在并发继续增大时逼近、但不会明显超过的预测上限。</td>
|
||||
<td>当前 64K Prefill 的拟合上限约 80.32 Gbit/s/rail，表示模型产流量上限，不表示网卡硬件只能跑 80G。</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
167
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_code.html
Normal file
167
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_code.html
Normal file
@ -0,0 +1,167 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Phase 2.5 Code：DSV4-Pro 双机 SGLang RDMA 需求建模</title>
|
||||
<style>
|
||||
:root { --canvas:#eef3f4; --paper:#fff; --ink:#182126; --muted:#5a6970; --line:#d4dee1; --navy:#17363d; --teal:#087c72; --teal-soft:#e8f5f3; --amber:#a64c14; --amber-soft:#fff1e7; --code-bg:#17252b; --code-ink:#eaf2f3; }
|
||||
* { box-sizing:border-box; letter-spacing:0; }
|
||||
body { margin:0; color:var(--ink); background:var(--canvas); font-family:"PingFang SC","Microsoft YaHei",Arial,sans-serif; font-size:16px; line-height:1.72; }
|
||||
header { color:#f6fbfb; background:var(--navy); border-bottom:5px solid #d2692b; }
|
||||
.header-inner,main { width:min(100% - 36px,1120px); margin:0 auto; }
|
||||
.header-inner { padding:34px 0 30px; }
|
||||
h1 { margin:0; font-size:clamp(28px,4vw,42px); line-height:1.25; }
|
||||
h2 { margin:42px 0 15px; padding-bottom:8px; font-size:25px; border-bottom:2px solid #adbbc0; }
|
||||
h3 { margin:28px 0 10px; color:#21454d; font-size:19px; }
|
||||
.eyebrow { margin:0 0 6px; color:#9edbd5; font-size:13px; font-weight:700; }
|
||||
.meta { margin-top:15px; color:#d6e5e7; font-size:14px; }
|
||||
main { margin-top:30px; margin-bottom:70px; padding:38px 48px 58px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
|
||||
a { color:var(--teal); }
|
||||
code { padding:2px 5px; color:#85380d; background:var(--amber-soft); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; overflow-wrap:anywhere; }
|
||||
pre { margin:14px 0 22px; padding:16px 18px; overflow:auto; color:var(--code-ink); background:var(--code-bg); border-radius:5px; font:13px/1.62 "SFMono-Regular",Consolas,monospace; }
|
||||
pre code { padding:0; color:inherit; background:transparent; }
|
||||
table { width:100%; margin:16px 0 26px; border-collapse:collapse; font-size:14px; }
|
||||
th,td { padding:9px 11px; vertical-align:top; text-align:left; border:1px solid var(--line); overflow-wrap:anywhere; }
|
||||
th { color:#153b41; background:#eaf2f2; }
|
||||
.callout { margin:18px 0 26px; padding:14px 18px; background:var(--teal-soft); border-left:4px solid var(--teal); }
|
||||
.warning { margin:18px 0 26px; padding:14px 18px; background:var(--amber-soft); border-left:4px solid var(--amber); }
|
||||
li+li { margin-top:5px; }
|
||||
@media(max-width:760px){main{padding:28px 20px 42px}table{display:block;overflow-x:auto}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">STANDALONE CODE WALKTHROUGH / PHASE 2.5</p>
|
||||
<h1>DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模：代码详解</h1>
|
||||
<div class="meta">实现提交：<code>c5fa700c50c0</code> 正式 Run：<code>dsv4pro-phase2_5-20260801-130007</code> 唯一入口：<code>run_rdma_demand_modeling.sh all</code></div>
|
||||
</div>
|
||||
</header>
|
||||
<main>
|
||||
<p><a href="./推理优化计划.html">返回推理优化主计划</a> · <a href="./phase2_5_exp.html">打开 Phase 2.5 实验档案</a></p>
|
||||
<div class="callout"><strong>边界：</strong>Phase 2.5 不复制模型服务和采集器。它复用 Phase 1 的双机 TP16 服务/benchmark 与 Phase 2 的精确窗口、GPU/RDMA 采集能力，只新增“并发 Scout → 自动选点 → 业务 OSL Confirm → 需求拟合”这一层编排和分析。</div>
|
||||
|
||||
<h2>1. 文件职责与调用关系</h2>
|
||||
<table>
|
||||
<thead><tr><th>文件</th><th>职责</th><th>调用关系</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>config.env</code></td><td>定义 ISL/OSL、Scout 并发、重复次数、平台阈值和 400G 目标</td><td>被唯一入口 source</td></tr>
|
||||
<tr><td><code>run_rdma_demand_modeling.sh</code></td><td>生成场景、调用 Phase 2、串行运行 Scout/Confirm、清理服务</td><td>唯一人工入口</td></tr>
|
||||
<tr><td><code>rdma_demand_model.py</code></td><td>对齐 benchmark/HCA 窗口，计算 bytes/token，拟合平台并生成报告</td><td>Scout 后选点；Confirm 后最终汇总</td></tr>
|
||||
<tr><td><code>test_rdma_demand_model.py</code></td><td>覆盖 HCA counter 单位、平台选择、线性换算和拟合输出</td><td>本地/CI 回归测试</td></tr>
|
||||
<tr><td>Phase 2 <code>run_hardware_contention_attribution.sh</code></td><td>启动服务、采集 Head/Worker、切正式测量窗</td><td>由 Phase 2.5 以环境变量调用</td></tr>
|
||||
<tr><td>Phase 1 <code>run_quick_map.sh</code></td><td>双机服务启停与 SGLang benchmark</td><td>由 Phase 2 内部复用</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<pre><code>run_rdma_demand_modeling.sh all
|
||||
├─ validate_config + write_manifest
|
||||
├─ run_scout
|
||||
│ ├─ write_scenario_file(64K→1, C=1/4/16/32/64)
|
||||
│ ├─ Phase 2 all（服务 + 18 个采集器 + 5 Case）
|
||||
│ └─ rdma_demand_model.py scout → recommendation.env
|
||||
├─ run_confirm
|
||||
│ ├─ 读取自动选择的 C=4/16/64
|
||||
│ ├─ Phase 2 all（重启服务 + 18 个采集器 + 每点 2 次）
|
||||
│ └─ rdma_demand_model.py final
|
||||
└─ Phase 2 stop → 双节点清理</code></pre>
|
||||
|
||||
<h2>2. 配置层</h2>
|
||||
<p><code>config.env:5-7</code> 通过相对路径找到 Phase 1/2，不依赖执行命令所在目录。<code>config.env:10-16</code> 定义 64K Scout 与 1K Confirm；<code>18-21</code> 定义 5% 平台阈值、400G 物理目标和 360G 实用目标。</p>
|
||||
<pre><code>ISL=65536
|
||||
SCOUT_OSL=1
|
||||
CONFIRM_OSL=1024
|
||||
SCOUT_CONCURRENCIES="1 4 16 32 64"
|
||||
SCOUT_REPETITIONS=1
|
||||
CONFIRM_REPETITIONS=2
|
||||
PLATEAU_GAIN_PCT=5
|
||||
TARGET_RAIL_GBPS=400
|
||||
PRACTICAL_RAIL_GBPS=360</code></pre>
|
||||
<p><code>SAMPLE_INTERVAL_S=1</code> 只决定 HCA/GPU 时间序列分辨率；<code>SCENARIO_TIMEOUT_S=7200</code> 是单个 benchmark 的保护上限，不是期望耗时。</p>
|
||||
|
||||
<h2>3. Shell 唯一入口</h2>
|
||||
<h3>3.1 参数检查与场景生成</h3>
|
||||
<p><code>run_rdma_demand_modeling.sh:42-71</code> fail-fast 检查依赖脚本、整数参数和并发列表。<code>73-100</code> 生成 Phase 2 能读取的 TSV，并为每个形状生成稳定的 case id。</p>
|
||||
<h3>3.2 复用 Phase 2，而不是复制采集代码</h3>
|
||||
<p><code>103-134</code> 构造一个数组命令，把场景、Case、重复次数和采样周期作为环境变量传给 Phase 2。它显式关闭 mixed case 与通信 microbenchmark，因为 Phase 2.5 只测模型 RDMA 需求，不重复已完成的硬件基线。</p>
|
||||
<pre><code>RUN_MIXED_CASE=0
|
||||
RUN_COMMUNICATION_BASELINE=0
|
||||
SCENARIO_FILE=.../scout.tsv
|
||||
FIXED_CASE_IDS=rdma_scout_...
|
||||
bash run_hardware_contention_attribution.sh all</code></pre>
|
||||
<h3>3.3 两阶段控制流</h3>
|
||||
<p><code>136-152</code> 跑完 Scout 后立即调用 Python，并写出 <code>recommendation.env</code>；<code>154-180</code> 读取推荐并发，生成 64K→1K Confirm。<code>217-225</code> 的 <code>run_all</code> 严格串行执行，异常信号触发 stop 清理。</p>
|
||||
<div class="warning"><strong>为什么服务会启动两次：</strong>Scout 结束后 Phase 2 会清理服务；Confirm 使用全新的 Prefix Cache、采集器和服务生命周期，避免 Scout 状态污染确认结果。</div>
|
||||
|
||||
<h2>4. Python 如何从计数器变成需求模型</h2>
|
||||
<h3>4.1 精确时间窗与 HCA 单位</h3>
|
||||
<p><code>rdma_demand_model.py:87-115</code> 以 <code>(case_id,repetition)</code> 对齐 benchmark、窗口和 RDMA 汇总。<code>117-147</code> 在正式窗口内计算相邻 HCA counter 的速率；IB <code>port_*_data</code> 单位是 4-octet，因此必须乘 4，再乘 8 转为 bit/s。</p>
|
||||
<pre><code>gbps = (counter_delta × 4 bytes × 8 bits) / duration_s / 1e9</code></pre>
|
||||
<h3>4.2 单 Case 指标</h3>
|
||||
<p><code>171-281</code> 汇总四条观测边（Head/Worker × 两个 HCA）的 Rail Mean/P95/Max、双 Rail 单向合计、Rail 不均衡、错误计数和 GPU 利用率。通信强度按每条 Rail 平均发送字节计算：</p>
|
||||
<pre><code>bytes_per_input_token_per_rail
|
||||
= mean(head/worker × mlx5_0/mlx5_3 xmit_bytes)
|
||||
/ total_input_tokens</code></pre>
|
||||
<p>这里不把 TX+RX 相加，因为那会把同一份跨机数据重复计数。</p>
|
||||
|
||||
<h3>4.3 平台、拐点与自动选点</h3>
|
||||
<p><code>362-392</code> 比较相邻并发点。只有 Rail Mean 与 Input TPS 增益同时低于 5%，当前点才是平台候选。随后选择平台前一点、平台点和最高稳定点；本 Run 得到 <code>4 16 64</code>。</p>
|
||||
<pre><code>if bandwidth_gain < 5% and input_tps_gain < 5%:
|
||||
plateau_c = current_concurrency</code></pre>
|
||||
|
||||
<h3>4.4 线性通信强度与饱和曲线</h3>
|
||||
<p><code>352-360</code> 用过原点线性斜率拟合 <code>rail_gbps/input_tps</code>，再还原为 bytes/token。<code>318-350</code> 用双曲线 <code>B(C)=B∞×C/(K+C)</code> 拟合并发饱和曲线；<code>394-447</code> 组合两者，判断模型计算或网络谁先到平台。</p>
|
||||
<pre><code>required_input_tps
|
||||
= target_rail_gbps / linear_gbps_per_input_tps
|
||||
|
||||
if fitted_bandwidth_asymptote < 360:
|
||||
verdict = COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION</code></pre>
|
||||
<p>本 Run 的 RMSE 为 0.277 Gbit/s，五个 Scout 点与饱和曲线贴合良好；拟合上限 80.32 Gbit/s，与 C=16/32/64 的 79.90/79.93/79.97 一致。</p>
|
||||
|
||||
<h2>5. 输出文件如何阅读</h2>
|
||||
<table>
|
||||
<thead><tr><th>输出</th><th>用途</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>rdma_case_metrics.csv</code></td><td>每次重复的 benchmark + GPU + Rail 对齐数据，是审计主表</td></tr>
|
||||
<tr><td><code>rdma_demand_model.json</code></td><td>完整拟合参数、平台点、目标 TPS 和最终 verdict</td></tr>
|
||||
<tr><td><code>rdma_demand_report.md</code></td><td>面向人的 Scout/Confirm 摘要</td></tr>
|
||||
<tr><td><code>recommendation.env</code></td><td>Shell 可直接 source 的 Confirm 并发列表</td></tr>
|
||||
<tr><td><code>{scout,confirm}/case_windows.csv</code></td><td>每个正式 benchmark 的精确起止时间</td></tr>
|
||||
<tr><td><code>{scout,confirm}/{head,worker}/rdma.csv</code></td><td>原始 HCA counter 时间序列，仅保留在服务器完整结果中</td></tr>
|
||||
<tr><td><code>commands/*.cmd.txt</code></td><td>实际传给 Phase 2 的完整可复现命令</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>6. 测试与验收门槛</h2>
|
||||
<ul>
|
||||
<li><code>python3 -m unittest test_rdma_demand_model.py</code>：4/4 通过。</li>
|
||||
<li><code>bash -n run_rdma_demand_modeling.sh</code> 与 Python compile：通过。</li>
|
||||
<li><code>DRY_RUN=1 ... all</code>：展开 5 个 Scout 和 3×2 个 Confirm，不启动服务。</li>
|
||||
<li>正式 Run：Scout 5/5、Confirm 6/6；两个阶段各 18/18 个采集器正常启停，共保存 72 条 STARTED/STOPPED 生命周期事件。</li>
|
||||
<li>11 个测量结果全部 <code>COMPLETED</code>，<code>rdma_error_delta=0</code>。</li>
|
||||
<li>结束后 Head/Worker 均无实验容器，16 张 GPU 为 0 MiB / 0%。</li>
|
||||
</ul>
|
||||
|
||||
<h2>7. 行号索引</h2>
|
||||
<table>
|
||||
<thead><tr><th>功能</th><th>文件与行</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>配置与路径</td><td><code>config.env:3-36</code></td></tr>
|
||||
<tr><td>校验与场景生成</td><td><code>run_rdma_demand_modeling.sh:42-100</code></td></tr>
|
||||
<tr><td>Phase 2 调用</td><td><code>run_rdma_demand_modeling.sh:103-134</code></td></tr>
|
||||
<tr><td>Scout / Confirm</td><td><code>run_rdma_demand_modeling.sh:136-180</code></td></tr>
|
||||
<tr><td>唯一 all 与清理</td><td><code>run_rdma_demand_modeling.sh:206-264</code></td></tr>
|
||||
<tr><td>HCA interval rate</td><td><code>rdma_demand_model.py:117-147</code></td></tr>
|
||||
<tr><td>Case 对齐汇总</td><td><code>rdma_demand_model.py:171-281</code></td></tr>
|
||||
<tr><td>饱和拟合</td><td><code>rdma_demand_model.py:318-350</code></td></tr>
|
||||
<tr><td>平台选点</td><td><code>rdma_demand_model.py:362-392</code></td></tr>
|
||||
<tr><td>需求模型与 verdict</td><td><code>rdma_demand_model.py:394-447</code></td></tr>
|
||||
<tr><td>报告输出</td><td><code>rdma_demand_model.py:449-517</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>8. 复用时必须重新标定的边界</h2>
|
||||
<p>这套代码可复用，但 <code>3.332 MB/token/rail</code> 不是通用常数。换模型、量化、TP/EP、节点切分、backend、Prefill/Decode 形状或 Prefix Cache 策略后，都必须重新跑 Scout。代码输出的是“当前部署实现的经验模型”，不是由参数量单独推导出的理论通信量。</p>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
166
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_exp.html
Normal file
166
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_exp.html
Normal file
@ -0,0 +1,166 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Phase 2.5：DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模</title>
|
||||
<style>
|
||||
:root { --ink:#18202a; --muted:#5b6570; --line:#d8dde3; --paper:#fff; --page:#f3f5f7; --blue:#1769aa; --green:#16734a; --amber:#9a5a00; --code:#f0f3f6; }
|
||||
* { box-sizing:border-box; letter-spacing:0; }
|
||||
body { margin:0; background:var(--page); color:var(--ink); font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","PingFang SC",sans-serif; font-size:16px; line-height:1.68; }
|
||||
header { background:#202a34; color:#fff; border-bottom:5px solid #34a17a; }
|
||||
.header-inner, main { width:min(1120px,calc(100% - 32px)); margin:0 auto; }
|
||||
.header-inner { padding:36px 0 30px; }
|
||||
h1 { margin:4px 0 12px; font-size:clamp(27px,4vw,42px); line-height:1.2; }
|
||||
h2 { margin:38px 0 12px; padding-bottom:7px; border-bottom:2px solid var(--line); font-size:24px; }
|
||||
h3 { margin:26px 0 8px; font-size:19px; }
|
||||
.eyebrow { margin:0; color:#8fd8bd; font-size:13px; font-weight:700; }
|
||||
.meta { display:flex; flex-wrap:wrap; gap:8px 22px; color:#d7dee5; font-size:14px; }
|
||||
main { margin-top:24px; margin-bottom:48px; padding:30px 38px 42px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
|
||||
a { color:var(--blue); }
|
||||
.back { display:inline-block; margin:0 16px 10px 0; font-weight:650; }
|
||||
code { padding:1px 5px; background:var(--code); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; font-size:.92em; overflow-wrap:anywhere; }
|
||||
pre { overflow-x:auto; padding:14px 16px; background:#202a34; color:#f4f7fa; border-radius:5px; line-height:1.5; }
|
||||
pre code { padding:0; background:transparent; color:inherit; }
|
||||
table { width:100%; margin:14px 0 22px; border-collapse:collapse; font-size:14px; }
|
||||
th,td { padding:10px 11px; border:1px solid var(--line); text-align:left; vertical-align:top; }
|
||||
th { background:#edf1f4; }
|
||||
.status,.decision,.warning { padding:14px 16px; }
|
||||
.status { border-left:4px solid var(--green); background:#eef8f3; }
|
||||
.decision { border-left:4px solid var(--blue); background:#edf5fb; }
|
||||
.warning { border-left:4px solid var(--amber); background:#fff7e7; }
|
||||
.metric { font-size:20px; font-weight:750; color:var(--green); }
|
||||
li+li { margin-top:5px; }
|
||||
@media (max-width:720px) { main{padding:22px 18px 30px} table{display:block;overflow-x:auto;white-space:nowrap} }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">FINAL EXPERIMENT RECORD / PHASE 2.5</p>
|
||||
<h1>DeepSeek-V4-Pro 双机 Pro6000D SGLang：RDMA 需求建模与并发拐点</h1>
|
||||
<div class="meta">
|
||||
<span>Run：dsv4pro-phase2_5-20260801-130007</span>
|
||||
<span>拓扑：TP16 / EP2 / 双 Rail RoCE</span>
|
||||
<span>完成：2026-08-01 15:14:11 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
<main>
|
||||
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||
<a class="back" href="./phase2_5_code.html">打开 Phase 2.5 代码详解</a>
|
||||
|
||||
<p class="status"><strong>阶段已完成。</strong>正式 Run 用时 2 小时 14 分 04 秒；Scout 5/5、Confirm 6/6 成功，两阶段各 18/18 个采集器正常启停。所有测量窗 RDMA 错误增量为 0，结束后两节点容器和 16 张 GPU 均已清理。</p>
|
||||
|
||||
<h2>1. 要回答的问题</h2>
|
||||
<p>Phase 2 只看到代表负载约 83.5 Gbit/s/rail，不能判断继续增加并发是否会逼近 400G。Phase 2.5 专门回答三个问题：</p>
|
||||
<ol>
|
||||
<li>固定模型、TP/EP 和输入形状后，Input TPS 与每 Rail RDMA 带宽是什么关系？</li>
|
||||
<li>并发增加到哪里后，模型吞吐和 RDMA 带宽不再增长？</li>
|
||||
<li>要达到 400G，需要怎样的 Input TPS；当前瓶颈先出现在模型计算还是网络？</li>
|
||||
</ol>
|
||||
|
||||
<h2>2. 实验设计</h2>
|
||||
<table>
|
||||
<thead><tr><th>阶段</th><th>请求形状</th><th>并发</th><th>重复</th><th>目的</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Scout</td><td>64K → 1</td><td>1 / 4 / 16 / 32 / 64</td><td>1</td><td>隔离 Prefill，找吞吐与带宽平台</td></tr>
|
||||
<tr><td>Confirm</td><td>64K → 1K</td><td>自动选择 4 / 16 / 64</td><td>2</td><td>验证真实长输出不会推翻需求模型</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>服务参数沿用 Phase 1/2：SGLang nightly、TP16、EP2、双 Rail <code>mlx5_0/mlx5_3</code>、<code>NET/IB + GDRDMA</code>。每个 Case 使用冷 Prefix，并按 benchmark 正式测量窗口切片 HCA Counter。</p>
|
||||
|
||||
<h2>3. 实际启动命令</h2>
|
||||
<p>只在 Head <code>174.1.51.5</code> 执行，不需要 <code>source</code> 或 <code>conda activate</code>：</p>
|
||||
<pre><code>cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||
|
||||
RUN_ID=dsv4pro-phase2_5-20260801-130007
|
||||
tmux new-session -d -s dsv4pro-phase2_5 \
|
||||
"RUN_ID=${RUN_ID} bash run_rdma_demand_modeling.sh all \
|
||||
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||
|
||||
tmux attach -t dsv4pro-phase2_5</code></pre>
|
||||
<p>实际展开后的 Scout/Confirm 命令分别保存在结果目录的 <code>commands/scout.cmd.txt</code> 与 <code>commands/confirm.cmd.txt</code>。</p>
|
||||
|
||||
<h2>4. Scout 结果：并发 16 已进入平台</h2>
|
||||
<table>
|
||||
<thead><tr><th>C</th><th>Input TPS</th><th>Rail Mean</th><th>Rail P95</th><th>Rail Max</th><th>双 Rail 单向合计</th><th>MB/input-token/rail</th><th>GPU Util</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>1</td><td>2,709.64</td><td>70.97 Gbit/s</td><td>85.60</td><td>86.81</td><td>141.94</td><td>3.138</td><td>93.75%</td></tr>
|
||||
<tr><td>4</td><td>2,930.49</td><td>78.27 Gbit/s</td><td>86.82</td><td>88.86</td><td>156.55</td><td>3.298</td><td>97.45%</td></tr>
|
||||
<tr><td>16</td><td>2,983.77</td><td>79.90 Gbit/s</td><td>86.42</td><td>88.20</td><td>159.79</td><td>3.336</td><td>99.33%</td></tr>
|
||||
<tr><td>32</td><td>2,983.92</td><td>79.93 Gbit/s</td><td>86.23</td><td>88.22</td><td>159.86</td><td>3.344</td><td>99.49%</td></tr>
|
||||
<tr><td>64</td><td>2,991.28</td><td>79.97 Gbit/s</td><td>85.93</td><td>88.32</td><td>159.95</td><td>3.340</td><td>99.64%</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="decision"><strong>观察结论：</strong>C=16→32 的 Input TPS 只增长 0.005%，Rail Mean 只增长 0.040%；C=32→64 也仅增长 0.247% / 0.057%。并发 16 已是平台拐点，继续加到 64 只会增加排队和 TTFT，不会增加网络压力。</p>
|
||||
<p><strong>证据：</strong><code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv</code>；原始 HCA 数据位于同一 Run 的 <code>scout/head/rdma.csv</code> 与 <code>scout/worker/rdma.csv</code>，精确窗口位于 <code>scout/case_windows.csv</code>。</p>
|
||||
|
||||
<h2>5. Confirm 结果：加入 1K Decode 后仍由计算先饱和</h2>
|
||||
<table>
|
||||
<thead><tr><th>C</th><th>重复</th><th>Input TPS</th><th>Output TPS</th><th>Rail Mean</th><th>Rail P95</th><th>双 Rail单向合计</th><th>TTFT P95</th><th>TPOT P95</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>4</td><td>2</td><td>2,072.24</td><td>32.38</td><td>57.33 Gbit/s</td><td>86.28</td><td>114.66</td><td>86.36 s</td><td>95.63 ms</td></tr>
|
||||
<tr><td>16</td><td>2</td><td>2,582.13</td><td>40.35</td><td>71.64 Gbit/s</td><td>86.30</td><td>143.29</td><td>336.85 s</td><td>356.24 ms</td></tr>
|
||||
<tr><td>64</td><td>2</td><td>2,588.96</td><td>40.45</td><td>72.19 Gbit/s</td><td>86.69</td><td>144.38</td><td>1,506.50 s</td><td>418.58 ms</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>C=16→64 的 Input TPS 仅增长 0.26%，Rail Mean 仅增长 0.76%，但 TTFT P95 从 336.85 秒升至 1,506.50 秒。对于 64K→1K，<strong>最大有意义并发仍约为 16</strong>；C=64 是容量压力点，不是推荐服务点。</p>
|
||||
<p><strong>证据：</strong>同一 Run 的 <code>confirm/bench_summary.csv</code>、<code>confirm/case_rdma_summary.csv</code>、<code>confirm/case_windows.csv</code>；两轮逐点数据在顶层 <code>rdma_case_metrics.csv</code>。</p>
|
||||
|
||||
<h2>6. 400G 能否被模型负载打满</h2>
|
||||
<p>Scout 的线性比例为：</p>
|
||||
<pre><code>每 Rail 带宽（Gbit/s）
|
||||
≈ Input TPS × 3.332 MB/input-token/rail × 8 ÷ 1e9</code></pre>
|
||||
<table>
|
||||
<thead><tr><th>目标口径</th><th>需要的 Input TPS</th><th>当前约 2,991 TPS 的差距</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>单 Rail 400G</td><td class="metric">15,006 tok/s</td><td>约 5.02×</td></tr>
|
||||
<tr><td>单 Rail 360G（90% 实用线）</td><td>13,505 tok/s</td><td>约 4.51×</td></tr>
|
||||
<tr><td>双 Rail 单向合计 400G</td><td>7,503 tok/s</td><td>约 2.51×</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>拟合得到当前模型负载的单 Rail 渐近上限约 <strong>80.32 Gbit/s</strong>，即物理 400G 的约 20.1%。瞬时 Max 也只有 89.53 Gbit/s。结论不是“网络只能跑 80G”，而是<strong>当前 DSV4-Pro TP16/EP2 实现最多只能产生约 80G/rail 的持续 RDMA 流量</strong>；Phase 2 的 NCCL microbenchmark 已证明链路本身能达到更高通信带宽。</p>
|
||||
<p class="warning"><strong>口径提醒：</strong>400G 是每条 Rail 的线速；双 Rail 单向总量是两条 Rail 的 TX 之和。不要把 TX 与 RX 相加后声称打满，也不要把 NCCL <code>busbw GB/s</code> 与 HCA <code>Gbit/s</code> 直接比较。</p>
|
||||
|
||||
<h2>7. 一套可复用的 RDMA 需求评估方法</h2>
|
||||
<ol>
|
||||
<li><strong>固定部署变量。</strong>记录模型版本、精度/量化、TP/EP/PP/DP、节点数、Attention/MoE backend、chunked prefill 和网卡拓扑。任一项变化都要重新标定。</li>
|
||||
<li><strong>先选 Prefill Scout。</strong>固定 ISL，OSL=1，取稀疏并发点如 1/4/16/32/64；每点清 Prefix Cache，并保证请求文本实际达到目标 token 数。</li>
|
||||
<li><strong>对齐正式测量窗。</strong>从 benchmark 的 main-run 起止时间切片 Head/Worker 的 <code>mlx5_*</code> HCA Counter，不能用整个进程寿命，也不能只看 <code>sar eth*</code>。</li>
|
||||
<li><strong>计算通信强度。</strong><code>bytes_per_input_token_per_rail = rail_xmit_bytes / total_input_tokens</code>。这是该模型与并行策略下“每处理一个输入 token，要在一条 Rail 发送多少字节”。</li>
|
||||
<li><strong>找并发平台。</strong>同时观察 Input TPS 和 Rail Mean；连续一点的增益都低于阈值（本实验 5%）时，记为拐点。最大 C 不等于最大有效 C。</li>
|
||||
<li><strong>推导目标吞吐。</strong><code>required_input_tps = target_rail_gbps × 1e9 / (bytes_per_token × 8)</code>。若模型的实测/拟合 TPS 上限远低于该值，网络不会先饱和。</li>
|
||||
<li><strong>用业务 OSL 复测。</strong>在平台前、拐点、最高压力点各重复至少两次，确认 Decode、KV Cache 和调度没有改变结论。</li>
|
||||
<li><strong>最后做链路对照。</strong>模型负载未打满时，用 NCCL microbenchmark 验证网络能力，把“模型产流量不足”与“网络本身跑不满”分开。</li>
|
||||
</ol>
|
||||
|
||||
<h3>7.1 哪些变量会改变 bytes/token 与平台</h3>
|
||||
<table>
|
||||
<thead><tr><th>变量</th><th>可能改变的原因</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>模型架构与层数</td><td>每 token 触发的 TP collective、MoE dispatch/combine 和激活尺寸不同</td></tr>
|
||||
<tr><td>TP / EP / PP / DP</td><td>通信参与 rank、跨机边界、collective 类型和频率改变</td></tr>
|
||||
<tr><td>Prefill / Decode、ISL / OSL</td><td>计算强度、chunk 调度、KV 访问和 collective 消息粒度不同</td></tr>
|
||||
<tr><td>并发与 batch</td><td>决定 kernel/batch 效率和 Input TPS；超过平台后只增加排队</td></tr>
|
||||
<tr><td>量化与 backend</td><td>改变计算速度；通信字节可能不同比例变化，因此会移动“计算先饱和还是网络先饱和”的边界</td></tr>
|
||||
<tr><td>Prefix Cache</td><td>命中会绕过大量 Prefill，必须单独作为另一类业务场景建模</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>8. 最终结论</h2>
|
||||
<p class="decision"><strong>在两台 Pro6000D、DSV4-Pro、SGLang TP16/EP2 的当前实现中，RDMA 不是吞吐瓶颈。</strong>64K Prefill 在 C=16 已达到约 3K input tok/s 和 80 Gbit/s/rail 的平台；继续增加并发到 64 不会显著增加吞吐或带宽，只会令 TTFT 急剧上升。要打满单 Rail 400G，模型侧 Input TPS 需提高到约 15K，约为当前上限 5 倍。因此后续优化应先看 GPU Kernel、MoE/Attention 执行和 rank 同步，而不是扩容计算网。</p>
|
||||
|
||||
<h2>9. 证据与清理</h2>
|
||||
<ul>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_report.md">自动 RDMA 需求报告</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_model.json">机器可读模型</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv">全部逐点指标</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/run_manifest.txt">Run Manifest</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/head_server_cmd.txt">Scout Head 服务命令</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_server_cmd.txt">Worker 服务命令</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_nccl_transport.log">Confirm Head NCCL 路径</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_nccl_transport.log">Worker NCCL 路径</a></li>
|
||||
</ul>
|
||||
<p>Worker 在最后一个 Case 完成后随 Head 主动关闭进程组出现 Gloo peer-close Traceback；它发生在测量结束与结果落盘之后，不是实验失败。最终 tmux、服务容器和 GPU 进程均已退出。</p>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@ -805,6 +805,7 @@ tmux new-session -d -s dsv4pro-phase2 \
|
||||
</p>
|
||||
|
||||
<p><a class="back" href="./phase1_exp.html">返回 Phase 1 实验档案</a></p>
|
||||
<p><a class="back" href="./phase2_5_exp.html">继续 Phase 2.5 RDMA 需求建模</a></p>
|
||||
<p><a class="back" href="./推理优化计划.html">返回推理优化主计划</a></p>
|
||||
</main>
|
||||
</body>
|
||||
|
||||
@ -0,0 +1 @@
|
||||
env DRY_RUN=0 RUN_ID=confirm RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/confirm.tsv FIXED_CASE_IDS=rdma_confirm_i65536_o1024_c4\,rdma_confirm_i65536_o1024_c16\,rdma_confirm_i65536_o1024_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=2 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
|
||||
@ -0,0 +1 @@
|
||||
env DRY_RUN=0 RUN_ID=scout RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/scout.tsv FIXED_CASE_IDS=rdma_scout_i65536_o1_c1\,rdma_scout_i65536_o1_c4\,rdma_scout_i65536_o1_c16\,rdma_scout_i65536_o1_c32\,rdma_scout_i65536_o1_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=1 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
|
||||
@ -0,0 +1,7 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||
fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:03:43+0800,2026-08-01T14:11:03+0800,440.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,406.0844679859583,bench_main_marker_plus_duration,16,0,406.0844679859583,15.997723352218085,,1048576,16384,0.03940066971621592,2582.1622905219265,40.3462857894051,2622.5085763113316,,406026.68602950143,406024.73604350234,406053.3737935184,406053.5528299224,189372.86670175308,189251.9370575028,336778.7128902273,348624.74093402264,211.7828145921294,211.89852223703926,356.41907673483706,369.22852823690624,211.7827865053078,53.14174899831414,53.806967759737745,55.23117201111745,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c16
|
||||
fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:11:03+0800,2026-08-01T14:18:24+0800,441.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,406.09330375201534,bench_main_marker_plus_duration,16,0,406.09330375201534,15.997232835777202,,1048576,16384,0.03939981243761298,2582.1061079114043,40.34540793611569,2622.45151584752,,406023.07082318654,406021.80599197163,406049.04031673505,406056.0000617319,189594.6209833055,189467.45998901315,336924.2522477143,348614.9598911492,211.56251206244485,211.68662201320043,356.06784952074946,368.88221031313344,211.56248251307755,53.15867651370354,54.19228565006051,55.67694299679715,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c16
|
||||
fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T13:58:15+0800,2026-08-01T14:00:56+0800,161.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.71871886099689,bench_main_marker_plus_duration,4,0,127.71871886099689,3.9987289259054917,,262144,4096,0.031318823393095684,2052.5104098899187,32.07047515452998,2084.5808850444487,,127678.13387226488,127678.37510351092,127682.15249372879,127682.41568037542,58779.63522000937,59286.87346848892,87520.75181677355,89912.75672257761,67.34946104814811,66.85386279083285,96.14249014501952,98.71897348003505,67.3494309836352,36.35076849604957,36.65076588222291,37.22598696593195,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c4
|
||||
fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T14:00:57+0800,2026-08-01T14:03:36+0800,159.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,125.30951252899831,bench_main_marker_plus_duration,4,0,125.30951252899831,3.9986913595171454,,262144,4096,0.03192096050229504,2091.9720674784076,32.68706355435012,2124.6591310327576,,125268.51625375275,125267.95494600083,125274.71757131862,125275.15292868251,56944.67479274317,57454.862895508995,85207.63279019155,87584.75151082734,66.7877238133036,66.2884575273625,95.11951526871552,97.68299949875195,66.78769251954503,36.23565100133419,36.565644716029055,37.02181185770315,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c4
|
||||
fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:18:31+0800,2026-08-01T14:46:05+0800,1654.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.8975476099877,bench_main_marker_plus_duration,64,0,1619.8975476099877,40.3918658202939,,4194304,65536,0.039508671455442484,2589.2402925038787,40.456879570373104,2629.6971720742517,,1022354.4435263402,950470.9995904996,1619639.9038552889,1619650.3429244931,785995.8297684161,774078.3500669932,1506265.7001026294,1560787.9135349835,231.04458822866482,223.80146996136563,418.57784627036636,439.7380822666596,231.0445591993489,56.0161369794514,56.77989884279668,58.514122637570836,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c64
|
||||
fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:46:06+0800,2026-08-01T15:13:40+0800,1654.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1620.2422464910196,bench_main_marker_plus_duration,64,0,1620.2422464910196,40.39704233770943,,4194304,65536,0.03950026617229964,2588.689443867829,40.44827256043483,2629.137716428264,,1022703.0410756742,950775.2832530241,1620010.8873196847,1620019.4079715456,786335.3449273454,774403.2157540205,1506724.735757988,1561196.831805848,231.05346642065376,223.83287831180922,418.5783493891426,439.7222371669007,231.0534379278651,56.055732507957146,56.75308680802118,58.11881933768746,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c64
|
||||
|
@ -0,0 +1,25 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_0,371,3626828296476.0,3626837993952.0,71.66006119953522,71.66025280544177,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_3,371,3626914718800.0,3626921347028.0,71.66176876011448,71.66189972281393,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_0,383,3626134447036.0,3626135801124.0,71.58004520955629,71.58007193930987,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_3,383,3626207403788.0,3626212421504.0,71.58148537888337,71.58158442880648,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_0,371,3625059758564.0,3625068124448.0,71.59790115790838,71.59806639097366,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_3,371,3625136689400.0,3625142126056.0,71.59942060496823,71.59952798337986,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_0,382,3625042688652.0,3625045908572.0,71.73367967522776,71.73374339219049,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_3,382,3625122942520.0,3625131203508.0,71.73526776832192,71.73543123979464,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_0,117,894188173332.0,894192013996.0,56.331515043976715,56.33175699575861,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_3,117,908654087048.0,908656485748.0,57.24282975425881,57.24298086608198,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_0,121,908912097324.0,908914949500.0,57.10271925252004,57.102898441471176,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_3,121,894353999288.0,894353927048.0,56.188101670195095,56.188097131691926,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_0,115,894195520320.0,894197433708.0,57.32326114869037,57.32338380826259,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_3,115,908661063200.0,908663318872.0,58.25058864399148,58.250733246007684,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_0,118,908220840268.0,908222493124.0,58.56932688787363,58.5694334772174,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_3,118,893986155676.0,893986644792.0,57.65136084035526,57.651392382457814,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_0,1479,14610070965300.0,14610072489408.0,72.21898772698583,72.21899526079844,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_3,1479,14610085802320.0,14610095102220.0,72.21906106780447,72.21910703810576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_0,1527,14611153636236.0,14611146819600.0,72.16844452340027,72.1684108541891,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_3,1527,14611226802652.0,14611236350532.0,72.16880591214243,72.16885307170732,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_0,1480,14610569508844.0,14610573106108.0,72.18614224806339,72.18616002099213,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_3,1480,14610597458160.0,14610605842440.0,72.18628033667787,72.18632176071877,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_0,1527,14610266497856.0,14610260234736.0,72.18231262718308,72.18228168411397,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_3,1527,14610323772088.0,14610331114336.0,72.18259559166701,72.18263186618948,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
|
@ -0,0 +1,7 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,1785564254150768896,1785564660235236864,406.084467968,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,1785564694981654016,1785565101074958080,406.093304064,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,1785563926377015040,1785564054095733760,127.71871872,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,1785564088360896000,1785564213670408960,125.30951296,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1785565142582675968,1785566762480224000,1619.897548032,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1785566797465815040,1785568417708060928,1620.242245888,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/meta.json
|
||||
|
@ -0,0 +1,37 @@
|
||||
wall_time_ns,node,collector,status,pid
|
||||
1785563885833567643,head,gpu_samples.csv,STARTED,2581028
|
||||
1785563885848767676,head,rdma.csv,STARTED,2581039
|
||||
1785563885862486752,head,dcgm_dmon.log,STARTED,2581053
|
||||
1785563885876598354,head,mpstat.log,STARTED,2581071
|
||||
1785563885891181421,head,pidstat.log,STARTED,2581090
|
||||
1785563885906513050,head,sar_net.log,STARTED,2581117
|
||||
1785563885921729008,head,perf_stat.log,STARTED,2581144
|
||||
1785563885938632713,head,docker_top.log,STARTED,2581179
|
||||
1785563885959612586,head,numa_samples.csv,STARTED,2581251
|
||||
1785563885980492840,worker,gpu_samples.csv,STARTED,2581294
|
||||
1785563885999012597,worker,rdma.csv,STARTED,2581356
|
||||
1785563886014677475,worker,dcgm_dmon.log,STARTED,2581408
|
||||
1785563886031239484,worker,mpstat.log,STARTED,2581455
|
||||
1785563886046314326,worker,pidstat.log,STARTED,2581464
|
||||
1785563886060934631,worker,sar_net.log,STARTED,2581478
|
||||
1785563886075086128,worker,perf_stat.log,STARTED,2581486
|
||||
1785563886090622653,worker,docker_top.log,STARTED,2581497
|
||||
1785563886106525841,worker,numa_samples.csv,STARTED,2581509
|
||||
1785568431501511663,head,gpu_samples.csv,STOPPED,2581028
|
||||
1785568431506303146,head,rdma.csv,STOPPED,2581039
|
||||
1785568431513026775,head,dcgm_dmon.log,STOPPED,2581053
|
||||
1785568431518181488,head,mpstat.log,STOPPED,2581071
|
||||
1785568431523384390,head,pidstat.log,STOPPED,2581090
|
||||
1785568431528223936,head,sar_net.log,STOPPED,2581117
|
||||
1785568431608439580,head,perf_stat.log,STOPPED,2581144
|
||||
1785568431612888149,head,docker_top.log,STOPPED,2581179
|
||||
1785568431617027570,head,numa_samples.csv,STOPPED,2581251
|
||||
1785568431620791766,worker,gpu_samples.csv,STOPPED,2581294
|
||||
1785568431624694474,worker,rdma.csv,STOPPED,2581356
|
||||
1785568431628587426,worker,dcgm_dmon.log,STOPPED,2581408
|
||||
1785568431632294659,worker,mpstat.log,STOPPED,2581455
|
||||
1785568431636153536,worker,pidstat.log,STOPPED,2581464
|
||||
1785568431639870020,worker,sar_net.log,STOPPED,2581478
|
||||
1785568431643739953,worker,perf_stat.log,STOPPED,2581486
|
||||
1785568431647558030,worker,docker_top.log,STOPPED,2581497
|
||||
1785568431651292267,worker,numa_samples.csv,STOPPED,2581509
|
||||
|
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -0,0 +1,12 @@
|
||||
stage,case_id,repetition,status,isl,osl,concurrency,completed,duration_s,input_tps,output_tps,total_tps,ttft_p95_ms,tpot_p95_ms,gpu_util_mean_pct,rail_xmit_avg_gbps,rail_xmit_p95_gbps,rail_xmit_max_gbps,dual_rail_xmit_avg_gbps,rail_imbalance_pct,xmit_bytes_per_input_token_per_rail,xmit_bytes_per_total_token_per_rail,rdma_error_delta,measurement_started_at,measurement_ended_at,source_result_dir
|
||||
scout,rdma_scout_i65536_o1_c1,1,COMPLETED,65536,1,1,1,24.1862671909621,2709.6368150803128,0.04134577659729481,2709.67816085691,24167.605170048773,0.0,93.74897875816994,70.9712746848062,85.5972372945909,86.80566056858429,141.9425493696124,0.5207119247408836,3137732.762954712,3137684.8856828967,0.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c4,1,COMPLETED,65536,1,4,4,89.45394851703895,2930.491100122512,0.044715745546302976,2930.535815868058,86526.69315790116,0.0,97.4494891826923,78.27464182068744,86.81919268342193,88.85894624954848,156.5492836413749,0.2703571902588796,3298012.1388320923,3297961.815928407,0.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c16,1,COMPLETED,65536,1,16,16,351.42641380499117,2983.771164628114,0.045528734811830354,2983.816693362926,336717.4506017618,0.0,99.33137833282255,79.89685137869246,86.416508483831,88.20167661610235,159.79370275738495,0.1149304623247205,3335858.008945465,3335807.108568442,0.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c32,1,COMPLETED,65536,1,32,32,702.8175486620166,2983.9209393567885,0.04553102019282209,2983.9664703769813,671139.2772838357,0.0,99.49393723047987,79.92852806348517,86.22587499587476,88.21644913628985,159.85705612697035,0.10091204027725086,3343929.3806581497,3343878.357123648,0.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c64,1,COMPLETED,65536,1,64,64,1402.17612864502,2991.2818470623424,0.04564333873080967,2991.327490401073,1337131.1103800863,0.0,99.63928846991146,79.9743866369171,85.93278751515446,88.32212536165416,159.9487732738342,0.03496630506355109,3339597.901974201,3339546.9445318105,0.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
confirm,rdma_confirm_i65536_o1024_c4,1,COMPLETED,65536,1024,4,4,127.71871886099689,2052.5104098899187,32.07047515452998,2084.5808850444487,87520.75181677355,96.14249014501952,97.83383458646617,56.71629143023766,86.06750356786138,88.99885476939444,113.43258286047534,1.8596562953363254,3439052.922241211,3386144.415745192,0.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c4,2,COMPLETED,65536,1024,4,4,125.30951252899831,2091.9720674784076,32.68706355435012,2124.6591310327576,85207.63279019155,95.11951526871552,98.63248207885304,57.94863438022769,86.4894823881113,88.5335715781298,115.89726876045536,2.1502935358359734,3438056.544746399,3385163.3671349157,0.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c16,1,COMPLETED,65536,1024,16,16,406.0844679859583,2582.1622905219265,40.3462857894051,2622.5085763113316,336778.7128902273,356.41907673483706,99.38883591065292,71.62084013702234,86.26345231858909,88.9399300384103,143.24168027404468,0.11410582506689998,3458520.1421022415,3405312.139916053,0.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c16,2,COMPLETED,65536,1024,16,16,406.09330375201534,2582.1061079114043,40.34540793611569,2622.45151584752,336924.2522477143,356.06784952074946,99.48923731257209,71.66656730160658,86.33499788736604,89.31414888004788,143.33313460321313,0.19167460586668467,3457155.7233657837,3403968.712237079,0.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c64,1,COMPLETED,65536,1024,64,64,1619.8975476099877,2589.2402925038787,40.456879570373104,2629.6971720742517,1506265.7001026294,418.57784627036636,99.6722631148773,72.19382480758325,86.71108451788668,89.82468221963666,144.38764961516648,0.070112013789426,3483446.6699664593,3429855.182736206,0.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c64,2,COMPLETED,65536,1024,64,64,1620.2422464910196,2588.689443867829,40.44827256043483,2629.137716428264,1506724.735757988,418.5783493891426,99.67225809785734,72.18433270089784,86.67873733217695,89.23672463088391,144.36866540179568,0.005496635275729702,3483400.1801579,3429809.4081554706,0.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
|
@ -0,0 +1,205 @@
|
||||
{
|
||||
"generated_at": "2026-08-01T15:14:11.558999+08:00",
|
||||
"formula": "rail_gbps ~= input_tps * bytes_per_input_token_per_rail * 8 / 1e9",
|
||||
"plateau_gain_pct": 5.0,
|
||||
"target_rail_gbps": 400.0,
|
||||
"practical_rail_gbps": 360.0,
|
||||
"scout": [
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 1,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2709.6368150803128,
|
||||
"output_tps": 0.04134577659729481,
|
||||
"total_tps": 2709.67816085691,
|
||||
"ttft_p95_ms": 24167.605170048773,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 93.74897875816994,
|
||||
"rail_xmit_avg_gbps": 70.9712746848062,
|
||||
"rail_xmit_p95_gbps": 85.5972372945909,
|
||||
"rail_xmit_max_gbps": 86.80566056858429,
|
||||
"dual_rail_xmit_avg_gbps": 141.9425493696124,
|
||||
"rail_imbalance_pct": 0.5207119247408836,
|
||||
"xmit_bytes_per_input_token_per_rail": 3137732.762954712,
|
||||
"xmit_bytes_per_total_token_per_rail": 3137684.8856828967,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 4,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2930.491100122512,
|
||||
"output_tps": 0.044715745546302976,
|
||||
"total_tps": 2930.535815868058,
|
||||
"ttft_p95_ms": 86526.69315790116,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 97.4494891826923,
|
||||
"rail_xmit_avg_gbps": 78.27464182068744,
|
||||
"rail_xmit_p95_gbps": 86.81919268342193,
|
||||
"rail_xmit_max_gbps": 88.85894624954848,
|
||||
"dual_rail_xmit_avg_gbps": 156.5492836413749,
|
||||
"rail_imbalance_pct": 0.2703571902588796,
|
||||
"xmit_bytes_per_input_token_per_rail": 3298012.1388320923,
|
||||
"xmit_bytes_per_total_token_per_rail": 3297961.815928407,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 16,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2983.771164628114,
|
||||
"output_tps": 0.045528734811830354,
|
||||
"total_tps": 2983.816693362926,
|
||||
"ttft_p95_ms": 336717.4506017618,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.33137833282255,
|
||||
"rail_xmit_avg_gbps": 79.89685137869246,
|
||||
"rail_xmit_p95_gbps": 86.416508483831,
|
||||
"rail_xmit_max_gbps": 88.20167661610235,
|
||||
"dual_rail_xmit_avg_gbps": 159.79370275738495,
|
||||
"rail_imbalance_pct": 0.1149304623247205,
|
||||
"xmit_bytes_per_input_token_per_rail": 3335858.008945465,
|
||||
"xmit_bytes_per_total_token_per_rail": 3335807.108568442,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 32,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2983.9209393567885,
|
||||
"output_tps": 0.04553102019282209,
|
||||
"total_tps": 2983.9664703769813,
|
||||
"ttft_p95_ms": 671139.2772838357,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.49393723047987,
|
||||
"rail_xmit_avg_gbps": 79.92852806348517,
|
||||
"rail_xmit_p95_gbps": 86.22587499587476,
|
||||
"rail_xmit_max_gbps": 88.21644913628985,
|
||||
"dual_rail_xmit_avg_gbps": 159.85705612697035,
|
||||
"rail_imbalance_pct": 0.10091204027725086,
|
||||
"xmit_bytes_per_input_token_per_rail": 3343929.3806581497,
|
||||
"xmit_bytes_per_total_token_per_rail": 3343878.357123648,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 64,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2991.2818470623424,
|
||||
"output_tps": 0.04564333873080967,
|
||||
"total_tps": 2991.327490401073,
|
||||
"ttft_p95_ms": 1337131.1103800863,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.63928846991146,
|
||||
"rail_xmit_avg_gbps": 79.9743866369171,
|
||||
"rail_xmit_p95_gbps": 85.93278751515446,
|
||||
"rail_xmit_max_gbps": 88.32212536165416,
|
||||
"dual_rail_xmit_avg_gbps": 159.9487732738342,
|
||||
"rail_imbalance_pct": 0.03496630506355109,
|
||||
"xmit_bytes_per_input_token_per_rail": 3339597.901974201,
|
||||
"xmit_bytes_per_total_token_per_rail": 3339546.9445318105,
|
||||
"rdma_error_delta": 0.0
|
||||
}
|
||||
],
|
||||
"confirm": [
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 4,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2072.2412386841634,
|
||||
"output_tps": 32.37876935444005,
|
||||
"total_tps": 2104.6200080386034,
|
||||
"ttft_p95_ms": 86364.19230348256,
|
||||
"tpot_p95_ms": 95.63100270686752,
|
||||
"gpu_util_mean_pct": 98.2331583326596,
|
||||
"rail_xmit_avg_gbps": 57.332462905232674,
|
||||
"rail_xmit_p95_gbps": 86.27849297798633,
|
||||
"rail_xmit_max_gbps": 88.76621317376211,
|
||||
"dual_rail_xmit_avg_gbps": 114.66492581046535,
|
||||
"rail_imbalance_pct": 2.0049749155861494,
|
||||
"xmit_bytes_per_input_token_per_rail": 3438554.733493805,
|
||||
"xmit_bytes_per_total_token_per_rail": 3385653.891440054,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 16,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2582.1341992166654,
|
||||
"output_tps": 40.345846862760396,
|
||||
"total_tps": 2622.480046079426,
|
||||
"ttft_p95_ms": 336851.4825689708,
|
||||
"tpot_p95_ms": 356.24346312779323,
|
||||
"gpu_util_mean_pct": 99.4390366116125,
|
||||
"rail_xmit_avg_gbps": 71.64370371931446,
|
||||
"rail_xmit_p95_gbps": 86.29922510297757,
|
||||
"rail_xmit_max_gbps": 89.12703945922908,
|
||||
"dual_rail_xmit_avg_gbps": 143.28740743862892,
|
||||
"rail_imbalance_pct": 0.15289021546679232,
|
||||
"xmit_bytes_per_input_token_per_rail": 3457837.9327340126,
|
||||
"xmit_bytes_per_total_token_per_rail": 3404640.426076566,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 64,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2588.964868185854,
|
||||
"output_tps": 40.45257606540397,
|
||||
"total_tps": 2629.4174442512576,
|
||||
"ttft_p95_ms": 1506495.2179303085,
|
||||
"tpot_p95_ms": 418.5780978297545,
|
||||
"gpu_util_mean_pct": 99.67226060636732,
|
||||
"rail_xmit_avg_gbps": 72.18907875424054,
|
||||
"rail_xmit_p95_gbps": 86.69491092503182,
|
||||
"rail_xmit_max_gbps": 89.53070342526028,
|
||||
"dual_rail_xmit_avg_gbps": 144.37815750848108,
|
||||
"rail_imbalance_pct": 0.03780432453257785,
|
||||
"xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
|
||||
"xmit_bytes_per_total_token_per_rail": 3429832.295445838,
|
||||
"rdma_error_delta": 0.0
|
||||
}
|
||||
],
|
||||
"selected_confirm_concurrencies": [
|
||||
4,
|
||||
16,
|
||||
64
|
||||
],
|
||||
"observed_plateau_concurrency": 16,
|
||||
"best_scout_bandwidth_concurrency": 64,
|
||||
"linear_gbps_per_input_tps": 0.026656368057740153,
|
||||
"modeled_bytes_per_input_token_per_rail": 3332046.007217519,
|
||||
"required_input_tps_for_400g_per_rail": 15005.795205617025,
|
||||
"required_input_tps_for_360g_per_rail": 13505.215685055322,
|
||||
"required_input_tps_for_400g_dual_rail_aggregate": 7502.897602808513,
|
||||
"bandwidth_saturation_fit": {
|
||||
"asymptote": 80.32102454669293,
|
||||
"half_saturation_c": 0.12589254117941673,
|
||||
"rmse": 0.27652726609534695
|
||||
},
|
||||
"input_tps_saturation_fit": {
|
||||
"asymptote": 3003.9418335884134,
|
||||
"half_saturation_c": 0.11220184543019636,
|
||||
"rmse": 7.680357081457988
|
||||
},
|
||||
"best_confirm_point": {
|
||||
"stage": "confirm",
|
||||
"concurrency": 64,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2588.964868185854,
|
||||
"output_tps": 40.45257606540397,
|
||||
"total_tps": 2629.4174442512576,
|
||||
"ttft_p95_ms": 1506495.2179303085,
|
||||
"tpot_p95_ms": 418.5780978297545,
|
||||
"gpu_util_mean_pct": 99.67226060636732,
|
||||
"rail_xmit_avg_gbps": 72.18907875424054,
|
||||
"rail_xmit_p95_gbps": 86.69491092503182,
|
||||
"rail_xmit_max_gbps": 89.53070342526028,
|
||||
"dual_rail_xmit_avg_gbps": 144.37815750848108,
|
||||
"rail_imbalance_pct": 0.03780432453257785,
|
||||
"xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
|
||||
"xmit_bytes_per_total_token_per_rail": 3429832.295445838,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
"verdict": "COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION"
|
||||
}
|
||||
@ -0,0 +1,37 @@
|
||||
# Phase 2.5 RDMA Demand Model
|
||||
|
||||
- Generated: `2026-08-01T15:14:11.558999+08:00`
|
||||
- Verdict: `COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION`
|
||||
- Model: `rail_gbps ~= input_tps x 3.332 MB/token/rail x 8`
|
||||
- Required Input TPS for 400G per rail: `15006`
|
||||
- Required Input TPS for 360G per rail: `13505`
|
||||
- Required Input TPS for 400G dual-rail aggregate: `7503`
|
||||
- Fitted rail bandwidth ceiling: `80.32 Gbit/s`
|
||||
- Observed plateau concurrency: `16`
|
||||
|
||||
## Scout: 64K -> 1
|
||||
|
||||
| C | Input TPS | Rail avg | Rail P95 | Dual-rail avg | MB/input-token/rail | GPU util |
|
||||
|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 1 | 2709.64 | 70.97 | 85.60 | 141.94 | 3.138 | 93.75 |
|
||||
| 4 | 2930.49 | 78.27 | 86.82 | 156.55 | 3.298 | 97.45 |
|
||||
| 16 | 2983.77 | 79.90 | 86.42 | 159.79 | 3.336 | 99.33 |
|
||||
| 32 | 2983.92 | 79.93 | 86.23 | 159.86 | 3.344 | 99.49 |
|
||||
| 64 | 2991.28 | 79.97 | 85.93 | 159.95 | 3.340 | 99.64 |
|
||||
|
||||
## Confirm: 64K -> 1K
|
||||
|
||||
| C | Reps | Input TPS | Output TPS | Rail avg | Rail P95 | Dual-rail avg | TTFT P95 ms | TPOT P95 ms |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 4 | 2 | 2072.24 | 32.38 | 57.33 | 86.28 | 114.66 | 86364.19 | 95.63 |
|
||||
| 16 | 2 | 2582.13 | 40.35 | 71.64 | 86.30 | 143.29 | 336851.48 | 356.24 |
|
||||
| 64 | 2 | 2588.96 | 40.45 | 72.19 | 86.69 | 144.38 | 1506495.22 | 418.58 |
|
||||
|
||||
## Evidence
|
||||
|
||||
- Per-case metrics: `rdma_case_metrics.csv`
|
||||
- Machine-readable model: `rdma_demand_model.json`
|
||||
- Scout HCA source: `scout/{head,worker}/rdma.csv`
|
||||
- Confirm HCA source: `confirm/{head,worker}/rdma.csv`
|
||||
- Exact windows: `scout/case_windows.csv`, `confirm/case_windows.csv`
|
||||
- Benchmark summaries: `scout/bench_summary.csv`, `confirm/bench_summary.csv`
|
||||
@ -0,0 +1 @@
|
||||
CONFIRM_CONCURRENCIES='4 16 64'
|
||||
@ -0,0 +1,13 @@
|
||||
run_id=dsv4pro-phase2_5-20260801-130007
|
||||
experiment=dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||
created_at=2026-08-01T13:00:07+0800
|
||||
git_commit=c5fa700c50c0cc1b9e20d4d2d3f03bd485ba0dbf
|
||||
git_dirty=0
|
||||
isl=65536
|
||||
scout_osl=1
|
||||
confirm_osl=1024
|
||||
scout_concurrencies=1 4 16 32 64
|
||||
scout_repetitions=1
|
||||
confirm_repetitions=2
|
||||
target_rail_gbps=400
|
||||
practical_rail_gbps=360
|
||||
@ -0,0 +1,4 @@
|
||||
# case_id stage isl osl concurrency prompt_multiplier min_prompts warmup_requests note
|
||||
rdma_confirm_i65536_o1024_c4 rdma_confirm 65536 1024 4 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
rdma_confirm_i65536_o1024_c16 rdma_confirm 65536 1024 16 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
rdma_confirm_i65536_o1024_c64 rdma_confirm 65536 1024 64 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
|
@ -0,0 +1,6 @@
|
||||
# case_id stage isl osl concurrency prompt_multiplier min_prompts warmup_requests note
|
||||
rdma_scout_i65536_o1_c1 rdma_scout 65536 1 1 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c4 rdma_scout 65536 1 4 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c16 rdma_scout 65536 1 16 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c32 rdma_scout 65536 1 32 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c64 rdma_scout 65536 1 64 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
|
@ -0,0 +1,6 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||
fixed_rdma_scout_i65536_o1_c1,fixed,rdma_scout_i65536_o1_c1,,rdma_scout,1,65536,1,1,1,0,COMPLETED,,0,2026-08-01T13:05:54+0800,2026-08-01T13:06:53+0800,59.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,24.1862671909621,bench_main_marker_plus_duration,1,0,24.1862671909621,0.9992315809294645,,65536,1,0.04134577659729481,2709.6368150803128,0.04134577659729481,2709.67816085691,,24167.682002007496,24167.682002007496,24167.682002007496,24167.682002007496,24167.605170048773,24167.605170048773,24167.605170048773,24167.605170048773,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/bench.log,fixed_rdma_scout_i65536_o1_c1
|
||||
fixed_rdma_scout_i65536_o1_c16,fixed,rdma_scout_i65536_o1_c16,,rdma_scout,1,65536,1,16,16,0,COMPLETED,,0,2026-08-01T13:09:09+0800,2026-08-01T13:15:34+0800,385.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,351.42641380499117,bench_main_marker_plus_duration,16,0,351.42641380499117,8.628873239905923,,1048576,16,0.045528734811830354,2983.771164628114,0.045528734811830354,2983.816693362926,,189525.8736173746,189769.315014506,336717.50638925005,348428.8343418157,189525.84591050618,189769.29299000767,336717.4506017618,348428.77078671294,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/bench.log,fixed_rdma_scout_i65536_o1_c16
|
||||
fixed_rdma_scout_i65536_o1_c32,fixed,rdma_scout_i65536_o1_c32,,rdma_scout,1,65536,1,32,32,0,COMPLETED,,0,2026-08-01T13:15:40+0800,2026-08-01T13:27:58+0800,738.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.8175486620166,bench_main_marker_plus_duration,32,0,702.8175486620166,16.63411701870453,,2097152,32,0.04553102019282209,2983.9209393567885,0.04553102019282209,2983.9664703769813,,365335.9171013453,365369.92930600536,671139.3307480408,696643.7062339701,365335.8620414692,365369.8761375272,671139.2772838357,696643.6536653765,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/bench.log,fixed_rdma_scout_i65536_o1_c32
|
||||
fixed_rdma_scout_i65536_o1_c4,fixed,rdma_scout_i65536_o1_c4,,rdma_scout,1,65536,1,4,4,0,COMPLETED,,0,2026-08-01T13:06:59+0800,2026-08-01T13:09:02+0800,123.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,89.45394851703895,bench_main_marker_plus_duration,4,0,89.45394851703895,2.5881014457616573,,262144,4,0.044715745546302976,2930.491100122512,0.044715745546302976,2930.535815868058,,57878.973371509346,58521.736774011515,86526.744662947,88841.4666661853,57878.91561826109,58521.6828455159,86526.69315790116,88841.41537077782,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/bench.log,fixed_rdma_scout_i65536_o1_c4
|
||||
fixed_rdma_scout_i65536_o1_c64,fixed,rdma_scout_i65536_o1_c64,,rdma_scout,1,65536,1,64,64,0,COMPLETED,,0,2026-08-01T13:28:04+0800,2026-08-01T13:52:01+0800,1437.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1402.17612864502,bench_main_marker_plus_duration,64,0,1402.17612864502,32.732392604632125,,4194304,64,0.04564333873080967,2991.2818470623424,0.04564333873080967,2991.327490401073,,717134.0553695618,717609.0070659702,1337131.160498035,1390460.9022671543,717134.0051560618,717608.9713889814,1337131.1103800863,1390460.8547802733,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/bench.log,fixed_rdma_scout_i65536_o1_c64
|
||||
|
@ -0,0 +1,21 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.006126848,bench_main_marker_plus_duration,head,mlx5_0,22,203533427700.0,203651710996.0,70.77538224308064,70.81651330239592,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.006126848,bench_main_marker_plus_duration,head,mlx5_3,22,204100301340.0,204100542584.0,70.97250317308169,70.97258706169157,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.352313088,bench_main_marker_plus_duration,worker,mlx5_0,23,207229229632.0,207304460944.0,70.99227518955745,71.01804781832155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.352313088,bench_main_marker_plus_duration,worker,mlx5_3,23,207674858740.0,207674728976.0,71.14493813350504,71.14489367915073,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,350.585191424,bench_main_marker_plus_duration,head,mlx5_0,321,3499319374812.0,3499321894556.0,79.850934047694,79.85099154570729,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,350.585191424,bench_main_marker_plus_duration,head,mlx5_3,321,3499670238252.0,3499687020088.0,79.85894039704549,79.85932334159443,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,349.898471936,bench_main_marker_plus_duration,worker,mlx5_0,331,3496131787252.0,3496130977288.0,79.93477120166395,79.93475268282916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,349.898471936,bench_main_marker_plus_duration,worker,mlx5_3,331,3496481190036.0,3496480578468.0,79.94275986836642,79.94274588561318,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,701.184411136,bench_main_marker_plus_duration,head,mlx5_0,641,7008726918884.0,7008731821472.0,79.96443511947507,79.96449105440941,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,701.184411136,bench_main_marker_plus_duration,head,mlx5_3,641,7009132360248.0,7009232426556.0,79.9690609081556,79.97020259135803,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.616064768,bench_main_marker_plus_duration,worker,mlx5_0,663,7016359452164.0,7016357965840.0,79.88840339972316,79.88838647640956,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.616064768,bench_main_marker_plus_duration,worker,mlx5_3,663,7016694022728.0,7016692492508.0,79.89221282658687,79.89219540347258,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.716937728,bench_main_marker_plus_duration,head,mlx5_0,82,866863327600.0,866884788736.0,78.16891338226691,78.17084862814438,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.716937728,bench_main_marker_plus_duration,head,mlx5_3,82,867282690684.0,867282482924.0,78.20672921268122,78.20671047804,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.006032896,bench_main_marker_plus_duration,worker,mlx5_0,84,861825370952.0,861825497804.0,78.34239018322312,78.34240171443258,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.006032896,bench_main_marker_plus_duration,worker,mlx5_3,84,862244987252.0,862244674092.0,78.38053450457852,78.38050603743919,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.218045952,bench_main_marker_plus_duration,head,mlx5_0,1280,14005295392512.0,14005304292920.0,79.96069096010922,79.96074177537255,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.218045952,bench_main_marker_plus_duration,head,mlx5_3,1280,14005533397520.0,14005553945192.0,79.96204980648541,79.96216711968765,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.136956672,bench_main_marker_plus_duration,worker,mlx5_0,1321,14008943976736.0,14008941122316.0,79.9861507329604,79.98613443522456,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.136956672,bench_main_marker_plus_duration,worker,mlx5_3,1321,14009382587800.0,14009379963032.0,79.98865504811339,79.98864006160981,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
|
@ -0,0 +1,6 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,1785560786217761024,1785560810404027904,24.18626688,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/meta.json
|
||||
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,1785560979903596032,1785561331330009856,351.426413824,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/meta.json
|
||||
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,1785561372166763008,1785562074984312064,702.817549056,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/meta.json
|
||||
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,1785560850521112832,1785560939975062016,89.453949184,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/meta.json
|
||||
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1785562116631968000,1785563518808096768,1402.176128768,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/meta.json
|
||||
|
@ -0,0 +1,37 @@
|
||||
wall_time_ns,node,collector,status,pid
|
||||
1785560744455556276,head,gpu_samples.csv,STARTED,2168337
|
||||
1785560744470800548,head,rdma.csv,STARTED,2168348
|
||||
1785560744485157848,head,dcgm_dmon.log,STARTED,2168362
|
||||
1785560744499919157,head,mpstat.log,STARTED,2168380
|
||||
1785560744514378735,head,pidstat.log,STARTED,2168400
|
||||
1785560744529055413,head,sar_net.log,STARTED,2168428
|
||||
1785560744543220536,head,perf_stat.log,STARTED,2168451
|
||||
1785560744559383367,head,docker_top.log,STARTED,2168486
|
||||
1785560744579614511,head,numa_samples.csv,STARTED,2168561
|
||||
1785560744602206477,worker,gpu_samples.csv,STARTED,2168604
|
||||
1785560744619402981,worker,rdma.csv,STARTED,2168664
|
||||
1785560744634896197,worker,dcgm_dmon.log,STARTED,2168717
|
||||
1785560744650285593,worker,mpstat.log,STARTED,2168763
|
||||
1785560744666005300,worker,pidstat.log,STARTED,2168775
|
||||
1785560744681017748,worker,sar_net.log,STARTED,2168787
|
||||
1785560744694432153,worker,perf_stat.log,STARTED,2168796
|
||||
1785560744709522681,worker,docker_top.log,STARTED,2168808
|
||||
1785560744723983222,worker,numa_samples.csv,STARTED,2168818
|
||||
1785563532345901412,head,gpu_samples.csv,STOPPED,2168337
|
||||
1785563532350635898,head,rdma.csv,STOPPED,2168348
|
||||
1785563532357352405,head,dcgm_dmon.log,STOPPED,2168362
|
||||
1785563532362419425,head,mpstat.log,STOPPED,2168380
|
||||
1785563532367417490,head,pidstat.log,STOPPED,2168400
|
||||
1785563532372197961,head,sar_net.log,STOPPED,2168428
|
||||
1785563532541034088,head,perf_stat.log,STOPPED,2168451
|
||||
1785563532545727806,head,docker_top.log,STOPPED,2168486
|
||||
1785563532550309298,head,numa_samples.csv,STOPPED,2168561
|
||||
1785563532554473337,worker,gpu_samples.csv,STOPPED,2168604
|
||||
1785563532558689660,worker,rdma.csv,STOPPED,2168664
|
||||
1785563532562663636,worker,dcgm_dmon.log,STOPPED,2168717
|
||||
1785563532566892350,worker,mpstat.log,STOPPED,2168763
|
||||
1785563532571076626,worker,pidstat.log,STOPPED,2168775
|
||||
1785563532575164431,worker,sar_net.log,STOPPED,2168787
|
||||
1785563532579199145,worker,perf_stat.log,STOPPED,2168796
|
||||
1785563532583371937,worker,docker_top.log,STOPPED,2168808
|
||||
1785563532587359055,worker,numa_samples.csv,STOPPED,2168818
|
||||
|
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -450,8 +450,11 @@
|
||||
</tr>
|
||||
<tr>
|
||||
<td>DeepSeek-V4-Pro / 双机 Pro6000D / SGLang RDMA 需求建模与并发拐点</td>
|
||||
<td>Phase 2.5 代码已实现；等待正式 Run，以 64K Scout 拟合 bytes/token、带宽平台和 400G 所需 Token TPS</td>
|
||||
<td>阶段尚未完成，按门禁暂不创建实验/代码 HTML</td>
|
||||
<td>Phase 2.5 已完成；Scout 5/5、Confirm 6/6，C=16 进入平台，拟合上限 80.32 Gbit/s/rail</td>
|
||||
<td>
|
||||
<a href="./phase2_5_exp.html">打开 Phase 2.5 实验档案</a><br>
|
||||
<a href="./phase2_5_code.html">打开 Phase 2.5 代码详解</a>
|
||||
</td>
|
||||
</tr>
|
||||
</tbody></table>
|
||||
<p>
|
||||
@ -799,11 +802,14 @@ Phase 2 已证明当前代表负载最高约 83.5 Gbit/s/rail，但单个固定
|
||||
再自动选择平台前、拐点和最大稳定并发，对 <code>64K → 1K</code> 重复两次确认。
|
||||
</p>
|
||||
<p>
|
||||
本阶段必须同时给出每 Rail Mean/P95/Max、双 Rail 单向合计、Input/Output TPS、
|
||||
GPU 利用率、Rail 不均衡和 RDMA 错误增量。最终结论不是只报告“最大 C”，而是判断
|
||||
模型计算吞吐和 RDMA 链路哪个先进入平台，并形成一套换模型、TP/EP 或输入输出形状后
|
||||
可重复使用的带宽需求估算方法。正式结果完成前不创建
|
||||
<code>phase2_5_exp.html</code> 或 <code>phase2_5_code.html</code>。
|
||||
正式 Run <code>dsv4pro-phase2_5-20260801-130007</code> 已完成。64K→1 Scout 在 C=16
|
||||
达到约 2,984 input tok/s 与 79.90 Gbit/s/rail；C=32/64 均不再显著增长，拟合
|
||||
单 Rail 渐近上限为 80.32 Gbit/s。通信强度约为 3.332 MB/input-token/rail，
|
||||
要达到单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍。64K→1K
|
||||
两轮确认同样在 C=16 后进入平台，说明当前是模型计算/实现吞吐先饱和，而非 RDMA 链路先饱和。
|
||||
完整方法、逐点结果和证据路径见
|
||||
<a href="./phase2_5_exp.html">Phase 2.5 实验档案</a> 与
|
||||
<a href="./phase2_5_code.html">Phase 2.5 代码详解</a>。
|
||||
</p>
|
||||
<h2>7. Phase 3：时间线 Profiling（Nsight Systems 为主）</h2>
|
||||
<p>
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user