Compare commits
38 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
97c208cfdb | ||
|
|
b9e8c0c4a2 | ||
|
|
2370d33c73 | ||
|
|
c5d91ceafe | ||
|
|
d063bd63c9 | ||
|
|
21fcca5d16 | ||
|
|
e3476aff86 | ||
|
|
f15003d2f5 | ||
|
|
ad1853f49b | ||
|
|
0d929948a5 | ||
|
|
92517e87f0 | ||
|
|
498e333730 | ||
|
|
0c89fd4fbe | ||
|
|
4d6dac010c | ||
|
|
911a9a2fb0 | ||
|
|
a3b8f1b2ab | ||
|
|
ffda226f5a | ||
|
|
123023b6ae | ||
|
|
5c749cda03 | ||
|
|
b3165a1d3c | ||
|
|
f0ab17c561 | ||
| 9cdbc1fd22 | |||
| ba20973bea | |||
| eeec56c2cf | |||
|
|
987f1db4b0 | ||
|
|
c69831f258 | ||
|
|
fe375e1307 | ||
|
|
ddf807d458 | ||
|
|
da0e1b4372 | ||
|
|
04dfa31583 | ||
|
|
d72dbff689 | ||
|
|
3bd04698bb | ||
|
|
d9a2e3b5e5 | ||
|
|
f33c5f1d3d | ||
|
|
ea8302561e | ||
|
|
6ba04325d3 | ||
|
|
9acf9fdfdb | ||
|
|
3761d75b00 |
1
.gitattributes
vendored
1
.gitattributes
vendored
@ -1 +0,0 @@
|
||||
*.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
||||
14
.gitignore
vendored
14
.gitignore
vendored
@ -49,6 +49,9 @@ experiments/**/runtime/
|
||||
# 无关项目
|
||||
loomeval_yy/
|
||||
|
||||
# 部署层运行时产物
|
||||
deploy/logs/
|
||||
|
||||
.gitignore
|
||||
|
||||
dsv4_dspark_h20_sglang_tp_dp_matrix
|
||||
@ -73,14 +76,22 @@ build/
|
||||
sglang_sm120_cache/
|
||||
vllm_sm120_cache/
|
||||
*_sm120_cache/
|
||||
sglang_nightly_cu13_cache/
|
||||
|
||||
# 一次性 bench 输出
|
||||
bench-output/
|
||||
|
||||
# 单数 dataset/（可重新下载，与 datasets/ 同源）
|
||||
dataset/
|
||||
|
||||
# 逐请求原始日志（体积大；汇总见 results.json / report.md）
|
||||
experiments/**/raw_outputs/
|
||||
**/dummy_sharegpt.json
|
||||
|
||||
# 统一 bench CLI 的一次性测试输出（ops_ 前缀 run_id）
|
||||
experiments/pro6000/*/results/ops_*/
|
||||
experiments/910c/*/results/ops_*/
|
||||
|
||||
# Virtual environments (keep only docs under envs/)
|
||||
envs/*
|
||||
!envs/README.md
|
||||
@ -93,3 +104,6 @@ skills-lock.json
|
||||
|
||||
# 临时备份文件
|
||||
*.bak.glm52orig
|
||||
*.bak
|
||||
*.bak.*
|
||||
*.tmp
|
||||
|
||||
148
README.md
148
README.md
@ -1,137 +1,5 @@
|
||||
# sskj — 多平台大模型推理性能基准测试项目
|
||||
|
||||
> **更新（2026-08-27 13:53:26 CST）**
|
||||
>
|
||||
> 完成 Kimi-K3 八节点标准 PD 第一阶段。P 组 601-604 使用 PP8×TP4×EP4、FlashInfer MXFP4、Chunk 8K，D 组 605-608 使用 PP1×TP32×EP32、Marlin，通过 Mooncake 0.3.12.post1 和 4 Rail RDMA 传输；统一 P/D `page_size=64` 后，16K→1 与 16K→512 的 C1/C8 共 91/91 请求成功。代表结果：16K→1 C8 Input TPS 6364.31、TTFT P50/P95 20.555/21.345 秒；16K→512 C8 TPOT P50/P95 63.20/66.55 ms。详见 `experiments/pro6000/kimi3_pro6000_pd_pp8_standard/README.md`。
|
||||
>
|
||||
> **更新（2026-08-03 10:02:07 CST）**
|
||||
>
|
||||
> 将 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 正式 Run `dsv4pro-phase3-20260802-003213` 的六份原始 `.nsys-rep` 纳入 GitLab `hzy` 分支，并通过 Git LFS 保存；普通 Git 仅保存六份报告的 SHA256 清单。Bench 文本、请求响应、服务日志、环境快照不随原始 Trace 上传；约 2.44 GB 的临时 `.sqlite` 也不上传，因为它可由 `.nsys-rep` 重新导出且不构成独立证据。
|
||||
>
|
||||
> **更新（2026-08-02 01:25:30 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 时间线 Profiling。正式 Run `dsv4pro-phase3-20260802-003213` 在 21 分 03 秒内完成 Decode Control、Decode+128K Prefill Mixed、独立 128K Prefill 三段 Capture；Head/Worker 各生成 3 份 Nsight Systems 报告及 Stats，合计 6/6 校验通过。Control 以 GEMM/MoE（Head 44.7%）和 NCCL（31.9%）为主；Mixed 中 NSA/MLA 升至 29.7%，NCCL 为 23.1%，说明 Chunked Prefill 的 Sparse Attention/MLA 计算进入同一 TP16 GPU 关键路径。Prefill Trace 漏采 3 个 Rank 的 CUDA 活动，因此暂不判断固定慢 Rank，也不将 NCCL 单独定性为根因。实验结束后两节点容器和 16 张 GPU 均已清理。
|
||||
>
|
||||
> **更新（2026-08-02 00:35:00 CST）**
|
||||
>
|
||||
> 修复 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 混合 Capture 时序。两轮失败 Run 证明当前镜像的 `/start_profile` 会阻塞 API 事件循环：Profile 激活后发送的 128K 请求只能在捕获结束后进入 Scheduler，因此异步调用仍无法覆盖注入。最终实现先稳定 Decode 背景，再通过原生 `/generate` + `input_ids` 送入 128K 请求；确认首个 8K Chunk Prefill 已开始后才捕获 32 个 Mixed Step。Control Trace 单独提供纯 Decode 基线，Mixed Trace 捕获真实 Prefill+Decode Treatment。混合背景缩为单波 32 请求，Profiling 专用 watchdog 提升至 1800 秒，异常退出会自动将 manifest 标记为 `FAILED`。
|
||||
>
|
||||
> **更新（2026-08-01 19:34:34 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 正式时间线代码交付。正式 Control 与混合背景统一为 `1K -> 1K, C=32`；所有 `/start_profile` 请求显式记录 `start_step` 和 `num_steps`，Control 在活跃 Decode 后跳过 2 Step，混合场景先保留 2 个纯 Decode Step 再注入 `128K -> 1`，长 Prefill 从首个 Chunk 开始捕获。继续沿用已通过的双节点 PyTorch/Nsight smoke、16 Rank/6 份 Nsight 报告 fail-closed 门禁和单服务三段 Capture Range；正式结果尚未生成，因此不创建 `phase3_exp.html` 或 `phase3_code.html`。
|
||||
>
|
||||
> **更新（2026-08-01 15:24:52 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模。正式 Run `dsv4pro-phase2_5-20260801-130007` 完成 Scout 5/5 与 Confirm 6/6；64K Prefill 在 C=16 已进入约 2,984 input tok/s、79.90 Gbit/s/rail 的平台，C=32/64 不再显著增长。拟合通信强度为 3.332 MB/input-token/rail，单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍，因此当前是模型计算/实现吞吐先饱和，不是 RDMA 先饱和。新增 `phase2_5_exp.html`、`phase2_5_code.html`、精简证据集和可复用的模型部署 RDMA 需求评估流程；实验结束后双节点容器与 16 张 GPU 已清理。
|
||||
>
|
||||
> **更新（2026-08-01 02:40:00 CST）**
|
||||
>
|
||||
> 新增 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模唯一入口。实验保持现有 TP16/EP2 服务参数不变，先以 `64K -> 1` 的 C=1/4/16/32/64 建立 Input TPS 与每 Rail HCA 带宽关系，再自动选择平台前、拐点和最大稳定并发，对 `64K -> 1K` 重复确认。结果将给出每 Token 跨机字节数、400G 所需 Token TPS、并发饱和曲线和“模型计算先饱和还是 RDMA 先饱和”的机器可读结论；正式结果尚未生成，因此暂不创建 Phase 2.5 HTML 档案。
|
||||
>
|
||||
> **更新（2026-07-31 19:02:00 CST）**
|
||||
>
|
||||
> Phase 3 Nsight smoke 已在 Head/Worker 各生成一份可解析报告；首份时间线确认 NCCL AllReduce、Sparse MLA、CUTLASS GEMM 与 MoE kernel 均可见。根据 smoke 修正正式 capture：Decode 对照和混合负载必须等服务实际进入 `Decode batch` 后才触发；每段等待本节点全部 8 个 rank 完成，而非任一 rank；双节点近同步停止以减少 Gloo 断链噪声。`nsys stats` 改为可写临时 SQLite、保留 Kernel/API/NVTX 高价值汇总并删除临时库，结果与报告数量均 fail-closed。
|
||||
>
|
||||
> **更新（2026-07-31 18:53:00 CST）**
|
||||
>
|
||||
> Phase 3 PyTorch Profiler 首轮 smoke 已证明双节点 16 个 rank 均可完成 trace 写盘，Head/Worker 分别生成 8 份、约 1.5/1.4 GB 压缩 trace。修复当前 OpenSSH 不接受 `scp remote:/path/.` 导致 Worker 结果未回收的问题，改为 SSH tar 流式传输；正式入口不再忽略回收失败，并新增至少 16 份 PyTorch rank trace、至少 2 份 Nsight 节点报告的结果门禁。
|
||||
>
|
||||
> **更新（2026-07-31 18:43:00 CST）**
|
||||
>
|
||||
> 为 Phase 2 实验档案第 11 节的 GPU/DCGM、CPU/进程/NUMA、双 Rail RDMA、PCIe/NCCL 四组结果补充服务器证据路径。每组同时标明结构化汇总 CSV、Head/Worker 原始采样日志、通信原始输出及实际命令文件，便于从结论直接追溯最终 Run `dsv4pro-phase2-20260731-163620` 的证据。
|
||||
>
|
||||
> **更新（2026-07-31 18:38:00 CST）**
|
||||
>
|
||||
> 新增 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 时间线分析唯一入口。代码提供双节点环境审计、PyTorch Profiler smoke、Nsight Systems smoke 与三段正式 capture range，覆盖 Decode 对照、Decode 背景叠加 128K Prefill、独立 128K Prefill；沿用 Phase 1 已验证的 TP16/EP2、CUDA Graph 与双 Rail NET/IB 配置，不重复 Phase 2 的硬件采样和通信微基准。阶段尚未产出正式结果，因此按档案门禁暂不创建 `phase3_exp.html` 或 `phase3_code.html`。
|
||||
>
|
||||
> **更新（2026-07-31 17:22:20 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2 正式收口。最终 Run `dsv4pro-phase2-20260731-163620` 在 28 分 44 秒内完成 8/8 benchmark、8/8 精确测量窗口和 18/18 采集器启停；混合 Prefill 令 Decode Output TPS 下降 23.96%、TPOT P95 增加 66.75%。GPU/CPU/NUMA、双 Rail RDMA、PCIe P2P、8/16-GPU AllReduce 与 `NCCL_CROSS_NIC=0/1/2` 数据均已归档；证据排除原始 Rail 带宽饱和、整机 CPU 饱和、GPU 降频和 CROSS_NIC 选择作为首要原因。实验结束后两节点容器、端口和 16 张 GPU 均已清理。
|
||||
>
|
||||
> **更新（2026-07-31 17:08:30 CST）**
|
||||
>
|
||||
> 修正阶段档案门禁的适用范围：规则禁止在 Phase 尚无阶段结果时提前创建 HTML，但不追溯撤下已经完成正式 Run 和结果汇总的档案。Phase 2 已有正式 Run `dsv4pro-phase2-20260731-130125` 的 8/8 结果，因此恢复 `phase2_exp.html`、`phase2_code.html` 及主计划、Phase 1/2 页面之间的双向导航。
|
||||
>
|
||||
> **更新（2026-07-31 17:01:56 CST）**
|
||||
>
|
||||
> 固定阶段档案生成门禁：某个 Phase 在实验结束、结果汇总并完成汇报确认前，不创建或维护 `phaseN_exp.html` 与 `phaseN_code.html`；进行中只维护代码、原始结果和主计划状态。阶段确认完成后再一次性生成两份最终 HTML。Phase 2 尚待最终正式复跑，因此暂时撤下其两份 HTML 及导航；已完成的 Phase 1 档案继续保留。
|
||||
>
|
||||
> **更新（2026-07-31 16:46:05 CST）**
|
||||
>
|
||||
> 统一 DeepSeek-V4-Pro 推理优化档案命名与导航：Phase 1/2 实验页分别更名为 `phase1_exp.html`、`phase2_exp.html`，代码页保持 `phase1_code.html`、`phase2_code.html`。主计划中的入口统一为“打开 Phase N 实验档案 / 代码详解”，并为实验页与代码页补齐双向链接。Phase 1 状态同步为固定点 11/11、混合 A/B 3/3、阶段总结果 14/14。
|
||||
>
|
||||
> **更新（2026-07-31 16:27:58 CST）**
|
||||
>
|
||||
> 完成 Phase 2 Worker 通信代码分发的双节点真机 smoke test。Run `dsv4pro-phase2-stage-smoke-20260731-162326` 依次通过 Head/Worker P2P、两组单机 8-rank AllReduce 和一组双机 16-rank AllReduce，全部 `wrong_values=0`。两节点暂存文件 SHA256 一致；结束后 `/tmp` 暂存、通信容器和 GPU 进程均已清理。本次使用 1 MiB、单次迭代，仅验证执行链路，不作为正式性能数据。
|
||||
>
|
||||
> **更新（2026-07-31 16:18:35 CST）**
|
||||
>
|
||||
> 修复 Phase 2 双节点通信基线在 Worker 启动后立即退出的问题。唯一入口仍只在 Head `174.1.51.5` 执行；脚本现在按 `RUN_ID` 将 `communication_baseline.py` 自动暂存到 Head/Worker 的 `/tmp`，校验 SHA256 后只读挂载进通信容器，并在结果目录保存当次源码与哈希。Worker `174.1.51.7` 不再依赖同路径 Git 工作树，异常退出也会清理暂存文件。新增回归测试后 Phase 2 单元测试为 9/9。
|
||||
>
|
||||
> **更新（2026-07-31 15:59:49 CST）**
|
||||
>
|
||||
> 在 DeepSeek-V4-Pro 双机 Pro6000D 推理优化主计划和 Phase 2 实施档案中增加 `phase2_code.html` 的直接入口，便于从阶段状态、实验命令与结果页面跳转到对应代码调用关系和逐行实现说明。
|
||||
>
|
||||
> **更新（2026-07-31 15:33:02 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2 最终采集代码。提交 `30664faa41f8` 新增正式 benchmark 精确时间窗、双节点 DCGM fail-closed 门禁、进程级 5 秒 Host 采样、机内 PCIe P2P 全矩阵、单机 8-rank 与双机 16-rank AllReduce、`NCCL_CROSS_NIC=0/1/2` A/B，以及与 Phase 2 第 5 节逐项对应的结构化 CSV 和 `report.md`。通信/NCCL 术语档案同步补充 DCGM、Host Engine、SM、Warp、NUMA、NIC、PIX/SYS、algbw/busbw 与 Linux 监控工具；最终双机复跑前不进入 Phase 3。
|
||||
>
|
||||
> **更新（2026-07-31 14:15:00 CST）**
|
||||
>
|
||||
> 恢复并完善 Phase 2 档案中的采集命令与指标解释。第 5 节现按实际实现记录时间 Marker、`nvidia-smi`、DCGM Field 1001–1005/1009/1010、`mpstat`、`pidstat`、`perf`、`numastat`、`sar`、`ethtool` 和 `mlx5_0/mlx5_3` HCA Counter，并逐项说明字段含义、分析方法及对应原始/汇总文件。同步记录首轮线程级 1 秒 `pidstat` 日志过重，后续应改为进程级 5 秒采样。
|
||||
>
|
||||
> **更新（2026-07-31 13:59:12 CST）**
|
||||
>
|
||||
> 补充 Phase 2 双节点执行边界：正式实验前仅需在 Worker `174.1.51.7` 启动并验证 `nvidia-dcgm` Host Engine；完整 `run_hardware_contention_attribution.sh all` 入口仍然只在 Head `174.1.51.5` 执行，由其通过 SSH 管理 Worker 服务与采集器。文档明确列出两台节点分别需要运行的命令，避免在 Worker 重复启动整套实验。
|
||||
>
|
||||
> **更新（2026-07-31 13:40:03 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 的 Phase 2 首轮硬件归因。正式 Run `dsv4pro-phase2-20260731-130125` 在 26 分 26 秒内完成 8/8 个 benchmark，无 OOM；混合负载下 Decode Output TPS 下降 24.03%，TPOT P95 增加 66.79%。两端双 Rail `NET/IB + GDRDMA` 流量均衡、错误增量为 0，最高每 Rail 平均约 70 Gbit/s，原始 RoCE 带宽未饱和；整机 CPU 与 GPU 频率也未见全局瓶颈。档案补充了 6000D 无 NVLink、机内 PCIe P2P/IPC、跨机 GDRDMA 的真实通信路径，并记录 Worker DCGM Host Engine 未启动、Case 时间窗过宽和 `pidstat` 日志过大的采集限制。
|
||||
>
|
||||
> **更新（2026-07-31 13:11:40 CST）**
|
||||
>
|
||||
> 新增 Phase 1 与 Phase 2 的独立代码详解 HTML 档案，行号固定到提交 `ca1f2f63375c`。文档从唯一入口展开到配置来源、文件调用关系、双机服务与 RDMA 门禁、benchmark 请求生成、混合 Prefill/Decode 时序、两节点采集器、Case 时间窗切片和结构化结果，并为 `MEM_FRACTION_STATIC` 等关键变量记录“默认值定义 → Shell 传递 → 服务参数 → Run 证据”的完整追踪路径。代码档案保持独立，不加入主计划 HTML 或阶段介绍 HTML 的导航。
|
||||
>
|
||||
> **更新（2026-07-31 11:57:13 CST）**
|
||||
>
|
||||
> 实现 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 的 Phase 2 硬件与资源竞争归因。新增唯一入口 `run_hardware_contention_attribution.sh`，内部复用 Phase 1 的双机服务与 benchmark，不要求用户手工启动 Phase 1；默认重放长/并发 Prefill、普通/持续/长上下文 Decode 和混合 Prefill/Decode A/B。Head 与 Worker 在同一诊断窗口采集 GPU、DCGM、CPU、进程、NUMA、`eth0/eth3` 和 `mlx5_0/mlx5_3` RDMA 数据，并保存 Case marker、完整命令、Manifest 和结构化汇总。正式执行只需运行 Phase 2 的 `all` 入口。
|
||||
>
|
||||
> **更新（2026-07-31 10:45:24 CST）**
|
||||
>
|
||||
> 为 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 的 Phase 2 硬件归因同步宿主机监控环境：`174.1.51.5` 与 `174.1.51.7` 均安装 `sysstat 12.5.2`，`pidstat`、`mpstat`、`sar -n DEV,EDEV` 的 1 秒实时采样验证通过。两节点的 sysstat service/timer 保持 `disabled/inactive`，正式实验由编排脚本显式启停采集，避免后台周期任务干扰 Case。Phase 2 必须同时保存 Head/Worker 的 sysstat、GPU、SGLang 与 RDMA 时间序列；`docker top` 用于 PID 映射，`perf` 用于 CPU 硬件事件，`numastat` 用于 NUMA 亲和，`mlx5_0/mlx5_3` HCA 计数器用于 RoCE 数据面，均不能由 sysstat 完整替代。
|
||||
>
|
||||
> **更新（2026-07-31 00:11:25 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 的长 Decode 补测，Run `dsv4pro-phase1-long-decode-20260730-234236` 结果 2/2 成功。`1K → 4K, C=16` 生成 65,536 个目标输出 token，Output TPS 为 310.02，TPOT P95 为 50.33 ms；相比 `1K → 1K, C=16`，Output TPS 增加 4.99%，TPOT P95 仅增加 0.62%。`128K → 1K, C=1` 生成完整 1,024 token，TTFT P95 为 49.326 s，TPOT P95 为 32.24 ms；其 TTFT 与 `128K → 1` 纯 Prefill 仅差 2.03%，TPOT P95 相比 1K Context 仅增加 2.47%。Phase 1 至此共完成 11 个固定点和 3 个混合结果，14/14 成功；两节点容器和 16 张 GPU 已清理。
|
||||
>
|
||||
> **更新（2026-07-30 23:38:42 CST）**
|
||||
>
|
||||
> 为 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 Phase 1 增加两个最小长 Decode 补充场景：`1K → 4K, C=16` 用于观察持续长输出、KV 增长与稳态 Decode，`128K → 1K, C=1` 用于观察长上下文上的 Decode Attention。继续复用唯一 `run_quick_map.sh` 入口和 `CASE_IDS` 过滤，不新增启动脚本；场景验证单测由 9 点更新为 11 点，Shell、3 个 Python 单测和两点 Dry-run 已通过。
|
||||
>
|
||||
> **更新（2026-07-30 23:06:01 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 Phase 1 正式 quick-map。Head 与 Worker 均通过 `mlx5_0/mlx5_3` 双 Rail `NET/IB + GDRDMA` 门禁；9 个固定点和 3 个混合 A/B 结果共 12/12 成功，总用时 28 分 36 秒。32K/128K 单请求 Prefill 输入吞吐为 2,652.76/2,710.16 token/s；在 C=32 Decode 中注入一个 128K Prefill 后，Output TPS 下降 24.08%，TPOT P95 增加 66.55%。阶段 HTML 已重写为只保留成功结果，并补充正式汇总、运行清单和 Phase 2 三个诊断负载；两节点容器和 16 张 GPU 已清理。
|
||||
>
|
||||
> **更新（2026-07-30 18:40:53 CST）**
|
||||
>
|
||||
> 为 DeepSeek-V4-Pro 双机 TP16 quick-map 的唯一入口新增 `CASE_IDS` 场景过滤和未知 Case 预检，可在完整九点实验前先跑 1K/32K Prefill 与 C1/C32 Decode 四点 Sanity；运行清单会记录实际过滤条件。同步精简阶段档案：正文只保留最终成功 Run 与有效结论，历史失败压缩到末尾经验教训。
|
||||
>
|
||||
> **更新（2026-07-30 17:54:30 CST）**
|
||||
>
|
||||
> 为 DeepSeek-V4-Pro 双机 TP16 quick-map 加入 RDMA fail-closed 启动保护。唯一 Shell 入口现在只允许计算网 `eth0/eth3` 与其 RDMA HCA `mlx5_0/mlx5_3`，在两端预检并透传 `rdma_cm/uverbs0/uverbs3`，服务健康后必须从两端 NCCL INFO 日志证明 `NET/IB` 和两条 HCA 均已启用，否则 benchmark 不会开始。运行清单新增 RDMA 开关、强制校验和设备路径；语法、结果解析器单测、完整 dry-run 及非法网卡/HCA 负例均已通过，真机 NET/IB 验证与 Phase 1 重跑尚未执行。
|
||||
>
|
||||
> **更新（2026-07-30 17:45:18 CST）**
|
||||
>
|
||||
> 修正 DeepSeek-V4-Pro 双机 TP16 quick-map 的 NCCL Socket 网卡错误。控制组确认服务容器未暴露 `/dev/infiniband`，NCCL 实际回退 `NET/Socket`；旧 quick-map 又误选低速非计算网，导致冷 1K/32K Prefill 比 `eth0` 计算网 Socket 控制组慢约 10.9 倍/13.25 倍。默认 `NCCL_SOCKET_IFNAME` 已改为 `eth0`；旧约 65 token/s 结果降级为事故证据，Phase 2 暂停并等待修正后的 Phase 1。新增双机通信/NCCL 术语 HTML、网络审计报告，并保留原 `/data/qqt/sskj` TP16 脚本不变。
|
||||
>
|
||||
> **更新（2026-07-30 16:38:41 CST）**
|
||||
>
|
||||
> 完成 DeepSeek-V4-Pro 双机 TP16 新旧脚本 TTFT 口径审计。确认旧产物受到 16 条 Warm-up、跨 Case 固定 Seed 递增长度、未清 Prefix Cache 及前一轮残留服务状态影响；同配置冷请求稳定复现约 16 秒/1K。新增 Phase 1 结果、脚本审计和 Phase 2 设计 HTML 档案，后续 Cold/Warm Prefix 指标分开报告。
|
||||
>
|
||||
> **更新（2026-07-30 14:33:52 CST）**
|
||||
>
|
||||
> 新增独立的 `dsv4pro_pro6000d_2node_sglang_tp16_quick_map` 快速性能地图与混合干扰 A/B。实验只保留一个 Shell 入口；旧 TP16 全量脚本保持不变。首轮真机验证已确认双机 TP16 服务可用，并据实测耗时将快速矩阵缩为一波请求，同时修正 Warm-up 污染 Prefix Cache 和混合负载注入时序。
|
||||
>
|
||||
> 历史更新见 `git log`。项目目的与工作流见下方。
|
||||
> **项目目的**：当新显卡（GPU/NPU）到货时，用最短时间完成大模型在该平台上的推理性能评估与部署配置选型。
|
||||
> 当前模型：DeepSeek-V4-Flash（FP8 / INT8）；后续接入 GLM5.2，**完全复用**本项目的实验与报告流程。
|
||||
@ -176,11 +44,10 @@
|
||||
| `experiments/p800/dsv4_p800_sglang_tp_dp_matrix/` | P800 + SGLang（INT8；TP2/DP4 启动 OOM 无数据，见 config.env 注释） |
|
||||
| `experiments/pro6000/dsv4_pro6000_vllm_tp_dp_matrix/` | RTX 6000D + vLLM |
|
||||
| `experiments/pro6000/dsv4_pro6000_sglang_tp_dp_matrix/` | RTX 6000D + SGLang |
|
||||
| `experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/` | 双机 RTX 6000D + DeepSeek-V4-Pro，SGLang TP16 快速性能地图与混合干扰 A/B |
|
||||
| `experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/` | 双机 RTX 6000D + DeepSeek-V4-Pro，SGLang TP16 硬件与资源竞争归因 |
|
||||
| `experiments/pro6000/kimi3_pro6000_pd_pp8_standard/` | Kimi-K3 八节点标准 PD：P=PP8/TP4/EP4、D=PP1/TP32/EP32、Mooncake 4 Rail RDMA，含 16K→1/512 C1/C8 原始证据 |
|
||||
| `experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/` | RTX 6000D + SGLang，Kimi-K3（TP32×EP32，部署手册见 docs/KIMI_K3_DEPLOY.md） |
|
||||
| `experiments/pro6000/kimi3_pro6000_pd_rdma/` | RTX 6000D + SGLang，Kimi-K3 PD 分离（MoonCake RDMA，8 节点，见 deploy_pd.sh + docs/KIMI_K3_DEPLOY.md 附录 B） |
|
||||
|
||||
TP/DP matrix 目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_concurrency.sh` 从 C=1 指数倍增搜饱和点；`run_adaptive_concurrency_add16.sh` 从 C=16 线性 +16 步进、带 TTFT SLO 停止与回退（当前主力用法，见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`）。专项 quick-map 的入口以对应目录 README 为准。
|
||||
每个目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_concurrency.sh` 从 C=1 指数倍增搜饱和点；`run_adaptive_concurrency_add16.sh` 从 C=16 线性 +16 步进、带 TTFT SLO 停止与回退（当前主力用法，见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`）。
|
||||
|
||||
### 其他实验（H200 老形态 & 专项）
|
||||
|
||||
@ -193,6 +60,7 @@ TP/DP matrix 目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_co
|
||||
| `experiments/h200/dsv4_h200_vllm_tp{2,4,8}_custom_bench/` | 自定义压测客户端（多服务负载均衡） |
|
||||
| `experiments/h200/dsv4_h200_max_context_length/` `…/dsv4_h200_long_context_matrix/` `…/dsv4_h200_256k_4k_probe/` `…/dsv4_h200_64k_sglang_vs_vllm/` | 长上下文专项 |
|
||||
| `experiments/p800/dsv4_p800_sglang/` `…/dsv4_p800_max_context_length/` `…/dsv4_p800_long_context_matrix/` `…/dsv4_p800_256k_4k_probe/` | P800 baseline 与长上下文专项 |
|
||||
| `experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/` | RTX 6000D + SGLang，GLM-5.3-NVFP4 双场景压测标准（16k/512 高并发 + 128k/64k 90% 命中低并发；真实语料 run-id 窗口纪律 + 三套部署配置 A/B/CAR，基线与口径见目录 README；部署 profile 见 `deploy/profiles/pro6000/glm53_nvfp4_*`） |
|
||||
| `experiments/TEMPLATE/` | 老式固定场景实验模板 |
|
||||
|
||||
## 快速复现
|
||||
@ -227,6 +95,14 @@ tmux new-session -d -s h20-vllm-adaptive \
|
||||
- 各平台的镜像 / venv / 模型路径见对应实验目录的 `config.env` 与 `platforms/<chip>.env`；**模型与数据集路径是机器相关的**，换机器时按实际路径调整。
|
||||
- 环境搭建规范见 `envs/README.md` 与 `envs/UV_ENV_SETUP.md`。
|
||||
|
||||
## 运维与部署分层
|
||||
|
||||
- 模型团队用 `deploy/profiles/<platform>/<model>-<engine>.env` 与
|
||||
`python -m sskj.deploy start|stop|status` 管理服务生命周期。
|
||||
- 运维只跑测试：`python -m sskj.bench run --url http://<host>:<port> --platform <platform> --experiment <name> --profile smoke|matrix`。
|
||||
- 运维手册见 [`ops/README.md`](ops/README.md)，统一测试层复用现有
|
||||
`scripts/common/parse_backend.py` 等指标口径。
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 提交内容：实验代码 + 最终产物（results.json / report.md / adaptive 的 jsonl 与 summary）；日志、raw_outputs、gpu_logs 不入库（`.gitignore` 已排除）。
|
||||
|
||||
35
deploy/CURRENT.md
Normal file
35
deploy/CURRENT.md
Normal file
@ -0,0 +1,35 @@
|
||||
# 现役部署状态页（live 核验于 2026-09-10，60.8 当日核验；其余机器 09-09 口径）
|
||||
|
||||
> 本页记录 174.1.60.x 集群"现在跑的是什么"。改动机器前后先读这页；
|
||||
> **任何变动（停/起/改配置）完成后必须更新本页**。核验命令：`docker ps` + `nvidia-smi`。
|
||||
|
||||
## 机器状态（2026-09-09 实测）
|
||||
|
||||
| 机器 | 在役 | 口径 / 归属 | 对应 profile |
|
||||
|---|---|---|---|
|
||||
| 60.1 (6000D-1) | `glm53-pp4`（Up，8 卡满载，:30000） | **方案 D 生产**。09-09/10 两轮实验窗口停机（凌晨 D 基线战役 ~7h + 日间 A 基线战役 ~9.5h）均恢复核验（health+生成冒烟）。**D 配置升级路径已交付未执行**：latest 镜像+开 autotune+持久缓存，16k/512 cc8-64 五点 +1.2~3.2%（A/B/A 确认），profile `..._tp2pp4_latest_autotune.env`——**仅适用 D 配置**：09-10 A 基线（TP8+EAGLE）战役证明 latest 与 autotune 在 A 配置上均负收益（eager 路径对代差/ tactic 敏感，图内不敏感），A 配置维持 nightly+autotune off；**IPC@TP8 真增益 +4.6~9.1% 已验证**（vs 同镜像基线全点胜、cc8/cc64 分布不重叠，收益随 TP 度单调升：TP2 −3.5%→TP8 +9.1%）但补丁包仅兼容 latest（sglang 4 文件依赖新版 runtime_context API），net vs 生产≈0；落地=移植回 nightly 或等上游 PR #34528 合入，未执行。同轮 A 基线画像：生产 A 掉图陷阱主导场景二（cc16+ 全程 eager，TPOT 168-197ms，平台 ~70-73；已发表 95/96 是 A-s2 图覆盖 bs16 变体）；日内漂移=eager 点 8.5h 后 −6~8% 而图内点不动（CPU 侧签名）。实验全量 `experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/`（REPORT.md + REPORT_A_BASELINE.md）。`dsv4_scan` 经授权保持停止，还原说明 `/root/dsv4_scan_restore_note.txt` | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp2pp4.env`（现役）；升级候选 `..._tp2pp4_latest_autotune.env`（仅 D 配置） |
|
||||
| 60.2 (6000D-2) | `glm53-nvfp4` 实验容器（09-08 晚 TP1PP8 phase，run_phase.sh 实验链进行中） | **他人实验进行中，勿动**（方案 F PD 链已拆除；GPU7 曾有外部裸金属任务）。动卡前仍先核实归属 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_{prefill→decode 侧}.env` |
|
||||
| 60.3 | 无容器，但 8 卡被外部裸金属训练占用（`/data/mas/larm`，09-08 晚实测） | 外部任务，勿动（此前台账漏记） | — |
|
||||
| 60.4 | `glm53-nvfp4`（Up，:30000，restart=unless-stopped） | **TP8+EAGLE+custom-AR 1stage（E7b 配方）在役**（09-09 下午部署：EAGLE 4/1/5/mem0.90/MRR16/chunk8192/ctxlen270336/fp8KV+hicache3/decode 图桶 1-8/双 parser；CAR 补丁三处全注入、8 rank `SSKJ_CAR_PATCH_ACTIVE` 确认，health 200、生成冒烟、质量门 7/7 见 `/root/qg_604_car.log`）。启动 `/root/deploy_glm53_604_exp.sh`（=60.7 实验版逐字拷贝，`RESTART=yes CAR_PATCH=1`），补丁 `/root/patches/`（md5 与仓库 car_patch 归档一致）。当日早间曾短暂部署 TP2PP4 D 配方复刻（deploy_s2_test_604.sh 留盘可切回）后被本方案替换；同日经授权清退外部 vllm 评测流水线（tmux `mas` 的 run_multiseed.sh 链，--resume 可续跑） | `experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/`（deploy_glm53_604_exp.sh + car_patch/ 补丁快照） |
|
||||
| 60.5 | `glm53-nvfp4`（Up 2d，09-09 只读核验） | **NVFP4 团队生产**（deploy_glm53_605.sh，md5 fcd9109b）。生产机铁律：不实验、不重启、不覆盖脚本。**交付升级路径已更新为 v3（09-09 hit90 场景优胜 TP4PP2-hicache@0.90，池 647,040/c4 并发独立文档/hit90 cc8 out +34% vs 现役）：脚本 `experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/scripts/deploy_glm53_605_v3.sh` + 补丁束（60.8:/root/glm53_r37_patch_bundle_v3.tar.gz，md5 6922e534，需 scp 至 60.5）+ 需 docker pull cu13 镜像；未执行、未落 60.5 磁盘（60.5:/root 仅有原脚本，核验过）。此前 v2（TP2PP4-hicache，冷缓存口径优胜）被 v3 取代，仍留仓库可作"容量优先 6 条文档/512k 单条最快"备选；回滚=原脚本** | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp8eagle.env`（方案 A 口径）；待切 `glm53_nvfp4_pro6000_sglang_tp4pp2_hicache.env`（容量优先备选 `..._tp2pp4_hicache.env`） |
|
||||
| 60.6 | 无容器，但 8 卡被外部裸金属实验占用（`/data/hzy/sparse-opd-*`，09-08 晚实测） | 外部任务，勿动（此前台账漏记） | — |
|
||||
| 60.7 | 基本空（4 卡仍有 `/home/user/dirA_exp` 外部小任务，09-08 晚实测） | 09-08 已拆除清空（方案 F 前身单机实验 + 场景一深优资产留盘），不再恢复 | — |
|
||||
| 60.8 | `glm53-nvfp4`（Up，:30000，restart=unless-stopped） | **TP4PP2-nomtp@0.90 hit90 优胜口径在役**（09-09 晚接替同日早间的 TP2PP4@0.85：同 r37 栈 cu13+9 挂载+de-GLOO，仅改 tp4/pp2 + memfrac 0.90；KV 池 **647,040**；实测实例原样保留切 restart 策略，未重建容器。启动 `bash /root/deploy_ppmtp_r37.sh '--tp 4 --pp-size 2 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length 1048576' nomtp 8192 0.90 1 1 0 0`）。hit90（90% 命中 i128k/o512）out cc1/2/3/4/8/16 = **28.8/47.5/63.9/76.2/106.4/125.7**（vs TP2PP4 基线 cc4 +14%/cc8 +15%）；并发独立 128k 文档 **4 条**（cap cc4 98.5 零排队）、512k 单条 ✓ TTFT 135.8s、质量门 **7/7**；同轮判决：DP attention 容量负收益判死、DCP 对 DSA 静默算错禁用、MTP@128k accept 2.07 判负。让步项（知情）：容量 4 条（TP2PP4 为 6）、512k 单条慢 33%。回滚 TP2PP4 = 同命令改 `'--tp 2 --pp-size 4'` + memfrac 0.85。**09-10 B300 对标战役**：停役（容器 rename 保全 `glm53-nvfp4-insvc`）→ 双臂 B300 场景矩阵（TP2PP4-D 口径 24 点 + E7b 配方）→ E7b 高并发调参复测（初测 MRR16+图1-8 掉图断崖 → MRR64+图桶 1-64，`deploy_glm53_e7b_hicc.sh`，三场景 c8-c64 共 10 点全 OK；报告正文采用复测值：掉图断崖已修复、16K 仍 TP2PP4 占优（E7b 被 prefill 墙封 ~100 tok/s 平台）、decode 密集 1K→4K E7b 全档反超（c64 out 826.5 tok/s 全场最高、超 TP2PP4 72%）、边界仅 TP2PP4 可达、与 B300 绝对差 4-5×；报告飞书 wiki `A7V3wZTQeifCB4krdi6cA834nW9`，已整文更新）→ TP2PP4 高并发调参复测（初测 C=64 受 MRR48 封顶 48 活跃+16 排队、decode 图默认覆盖 bs≤256 从未掉图 → MRR64，`deploy_glm53_pp4_mrr64.sh` 逐 token 仅改 MRR；三场景 c32/c64 共 6 点两轮对齐 ≤1.8%（pass-1 + v2 有序版）+ 全新实例 MRR48 对照归因；正文采用 v2 值：4.1/4.2 c64 out +27%/+19%（433/572）、三场景 c64 TTFT P95 全塌缩（20.0/337.9/134.8s → 7.2/7.3/89.2s）、16K c64 到池顶 ~59 活跃 + 3 次回退（确定性行为）、1K 短输入 c32 付 −14% MRR64 配置代价（276→266→228 = −3% 新鲜度 −14% 配置，三实例可复现、机制未定位）、16K c32 无代价；报告已再次整文更新）→ **原容器恢复并三次核验**（rename 回 + start，fired up/health 200/16K 抽测 ok、KV 池 647,040 与启动口径逐字一致、显存水位 77.2/82.3 GiB 与停役前一致） | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp4pp2_hicache.env`；实验全量 `experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/`；B300 对标 `experiments/pro6000/glm53_nvfp4_b300_equivalent_matrix/`；前史 `experiments/pro6000/glm53_nvfp4_128k_capacity_topology/`（TP2PP4 口径）+ r37 `experiments/pro6000/glm53_ppmtp_r37_verify_graph/` |
|
||||
|
||||
## 方案 A-F 一览（GLM-5.3-NVFP4 @ pro6000，2026-09-08 双场景报告口径）
|
||||
|
||||
| 方案 | 一句话 | profile / 脚本 |
|
||||
|---|---|---|
|
||||
| A | TP8 + EAGLE，基准；60.5 生产口径 | `glm53_nvfp4_pro6000_sglang_tp8eagle.env` |
|
||||
| B | TP4 PP2 | `glm53_nvfp4_pro6000_sglang_tp4pp2.env` |
|
||||
| C | TP4 PP2 + IndexCache(freq=4) | `glm53_nvfp4_pro6000_sglang_tp4pp2_index.env` |
|
||||
| D | TP2 PP4；60.1 生产在役 | `glm53_nvfp4_pro6000_sglang_tp2pp4.env` |
|
||||
| E | TP8 + DFlash2 投机 | `glm53_nvfp4_pro6000_sglang_tp8dflash2.env` |
|
||||
| F | PD 分离双机链（mc-master→prefill→decode→router） | `glm53_nvfp4_pro6000_pd_{master,prefill,decode,router}.env` + `deploy/PD_CHAIN.md` |
|
||||
|
||||
压测数据：飞书《GLM-5.3-NVFP4 双场景压测报告》（`SZUSdEqY1oRVxGxgILBcHqPJnEc`）。
|
||||
|
||||
## 防漂移
|
||||
|
||||
每个在役容器用 `deploy/verify_profile.sh <profile.env>` 定期核验（镜像 digest + 启动参数 +
|
||||
端口），发现不一致 = 容器被人手改过，先查清归属再处理。
|
||||
96
deploy/PD_CHAIN.md
Normal file
96
deploy/PD_CHAIN.md
Normal file
@ -0,0 +1,96 @@
|
||||
# 方案F：GLM-5.3-NVFP4 PD 分离完整链（双机 6000D）编排手册
|
||||
|
||||
> 2026-09-08 实测终态。四角色、两台机、启动顺序强制。吞吐换算口径：**链合计 ÷2 = 单机等效**（与单机方案 A-E 可比）。
|
||||
> 完整压测数据见飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 `SZUSdEqY1oRVxGxgILBcHqPJnEc` / wiki `NzbMwzmKviYidRkZYRrc8GYQnrf`）。
|
||||
|
||||
## 拓扑
|
||||
|
||||
| 角色 | 机器 | 容器 | 端口 | profile |
|
||||
|---|---|---|---|---|
|
||||
| 1. mc-master（Mooncake 元数据） | 174.1.60.1 | `mc-master` | 50051 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_master.env` |
|
||||
| 2. prefill（TP4 PP2 + DFLASH 草稿） | 174.1.60.1 | `glm53-pd-smoke-prefill` | 30000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_prefill.env` |
|
||||
| 3. decode（TP8 + DFLASH v5 配方） | 174.1.60.2 | `glm53-s1-decode` | 30000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_decode.env` |
|
||||
| 4. router（MiniLB） | 174.1.60.2 | `pd-smoke-router` | 31000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_router.env` |
|
||||
|
||||
镜像统一：`lmsysorg/sglang:nightly-dev-20260828-daf63171`
|
||||
（digest `sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399`）。
|
||||
|
||||
## 启动顺序（强制）
|
||||
|
||||
```
|
||||
mc-master (60.1) → prefill (60.1) → decode (60.2) → router (60.2)
|
||||
```
|
||||
|
||||
对应脚本（`experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/`）：
|
||||
|
||||
```bash
|
||||
# 60.1（注意：60.1 日常跑生产容器 glm53-pp4，先停它，见下节）
|
||||
bash /tmp/deploy_pd_smoke_master.sh # 判据 ss :50051
|
||||
bash /tmp/deploy_pd_probe.sh p1b_prefill_tp4pp2_cps8k_radix_launch.sh # 判据 :30000/health 200，~10-15min
|
||||
# 60.2
|
||||
bash /tmp/deploy_s1_decode.sh # 判据 :30000/health 200，~10min
|
||||
bash /tmp/deploy_pd_smoke_router.sh # 判据 ss :31000
|
||||
```
|
||||
|
||||
场景一（90% 命中）必须用 **radix 变体** launch（`p1b_prefill_tp4pp2_cps8k_radix_launch.sh`，
|
||||
与基线唯一差异 = 无 `--disable-radix-cache`）。场景二 0 命中，radix 开销可忽略，两场景共用同一部署。
|
||||
|
||||
## 基础设施依赖（缺一不可）
|
||||
|
||||
| 依赖 | 位置 | 说明 |
|
||||
|---|---|---|
|
||||
| sglang 补丁树 | `/data/sglang_patch_glm53` → 容器 `/sgl-workspace/sglang` | 与镜像原版差 **11 个文件**（10 改 + 1 新增 `dflash_pp.py`），清单与 diff 见 `platforms/patches/pro6000/glm53_pd_chain/`。无补丁则 DFlash+PD 冷启动接线缺失，decode 首请求 400 |
|
||||
| mooncake wheel | `/data/flashkda_deploy/wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl` | 容器内 launch 脚本 `pip install --no-deps` 自装 |
|
||||
| DFlash2 草稿权重 | `/data/hf_models/GLM-5.3-DFlash2` | 两角色都要挂 |
|
||||
| IB 设备 | mlx5_0–mlx5_3 + `--device /dev/infiniband` + `--ulimit memlock=-1` | RDMA KV 传输通道 |
|
||||
| 环境变量 | `MOONCAKE_MASTER=174.1.60.1:50051` `MOONCAKE_PROTOCOL=rdma` | prefill/decode 容器都要 |
|
||||
|
||||
## 质量门（经 router :31000，全链路含 KV transfer + 草稿）
|
||||
|
||||
60.2 上生成 router 口径的 gate 副本：
|
||||
|
||||
```bash
|
||||
sed 's/PORT=30000/PORT=31000/' /root/quality_gate_605.sh > /tmp/quality_gate_31000.sh
|
||||
```
|
||||
|
||||
判据：GSM8K×5 + 中文推理必须过。tool call 预期不过（链配方无 parser，与方案 B/D 同口径的已知配置缺口，记录放行）。
|
||||
2026-09-08 实测 **6/7**（核心项全过，DFlash 草稿无质量损失）。
|
||||
|
||||
## 压测
|
||||
|
||||
驱动脚本：`run_pd_s1.sh` / `run_pd_s2.sh`（同 scripts 目录），跑在 60.1（语料本地，
|
||||
`--url http://174.1.60.2:31000/generate` 走 router，`--container glm53-pd-smoke-prefill` 读 prefill 日志核命中率）。
|
||||
|
||||
- **场景一**：128k/64k × cc1-4，nreq8，shared-frac 0.9，canonical 窗口 9301-9308
|
||||
- **场景二**：cc8/16/32 canonical 9311-9313（nreq 16/32/32）；cc40/64 全新窗口 pool-override
|
||||
16,384,000 / 17,203,200（nreq 40/64 单轮满波）
|
||||
- 指标读取：命中率读 prefill 容器日志（PP2 日志计数 ×2，不影响比值）；accept 从 decode 日志读
|
||||
（router 转发后 usage 字段可能缺失）
|
||||
- 每点核验：ok/failed=0、retractions、s1 hit≈0.90、s2 hit=0
|
||||
|
||||
### 2026-09-08 实测判决（13/13 干净点）
|
||||
|
||||
- **场景一**：128k cc1 TTFT 3.91s = 六方案最低（跨机重叠把 prefill 与 RDMA 传输完全藏住），
|
||||
但 cc≥2 时 decode 侧排队，TTFT 堆到 31.0s；÷2 输出吞吐 21.2-23.5 tok/s，仅为 A 的 30-57%。
|
||||
根因：decode KV 池 214,336 token 按 16k 场景定容 → 128k 仅 1 驻留、64k 仅 3。
|
||||
- **场景二**：÷2 单机等效 74-85 tok/s = A 的 75-96%，但 TTFT 较 A 减半。
|
||||
判决：**两机买 TTFT、不买吞吐**。decode MRR12 在 cc16+ 饱和；DFlash accept 在 12-batch
|
||||
verify 下掉到 1.8（单机 EAGLE 2.84）。
|
||||
- **PD 机制验证**：跨机 prefill/decode 重叠成立、mooncake RDMA 传输被吸收——机制无罪，
|
||||
容量配置（decode 池按 16k 定容）是判决主因。
|
||||
|
||||
## 拆链与生产恢复（顺序固定）
|
||||
|
||||
```bash
|
||||
# 1. 拆链
|
||||
# 60.1: docker rm -f mc-master glm53-pd-smoke-prefill
|
||||
# 60.2: docker rm -f glm53-s1-decode pd-smoke-router
|
||||
# 2. 等双机显存排空（60.1 <2000MiB；60.2 GPU7 常有外部裸金属任务 main_v2.py，核实归属勿清）
|
||||
# 3. 恢复 60.1 生产：bash /tmp/deploy_glm53_pp4.sh → health 200 + docker inspect Args 核对
|
||||
```
|
||||
|
||||
## 60.1 生产停机窗口提示
|
||||
|
||||
60.1 日常跑生产 `glm53-pp4`（方案 D，TP2PP4，8 卡满载）。拉 prefill 角色前必须停生产；
|
||||
`deploy_glm53_pp4.sh` 生成的参数与生产容器 `docker inspect` Args 已核对逐字一致，
|
||||
恢复即完全复现。全程停机 ~2-2.5h（部署+13 点压测+质量门+恢复）。
|
||||
4
deploy/manifests/GLM-5.3-DFlash2.md5
Normal file
4
deploy/manifests/GLM-5.3-DFlash2.md5
Normal file
@ -0,0 +1,4 @@
|
||||
302f15789d891aa39620209286252fed ./README.md
|
||||
ce1962554abf73c6ad953aa0ac3f20d7 ./config.json
|
||||
5e170425c97cda8f798c74041979a569 ./configuration.json
|
||||
32c81842f12e56e6ac2a1feaafd5bfa7 ./model.safetensors
|
||||
58
deploy/manifests/GLM-5.3-NVFP4.md5
Normal file
58
deploy/manifests/GLM-5.3-NVFP4.md5
Normal file
@ -0,0 +1,58 @@
|
||||
434f8692a4914f39714df2c4493c92d7 ./.gitattributes
|
||||
245254422e0ca504a5f09e37719ca718 ./LICENSE
|
||||
02e0c207304666abdd93de6a3780c52a ./README.md
|
||||
e4c9e1000a513f680dd796f76d1097e0 ./chat_template.jinja
|
||||
d42bba3f99ce9b2705bb19c41930ea0c ./config.json
|
||||
5e170425c97cda8f798c74041979a569 ./configuration.json
|
||||
ee2560496cd373851033b3c05ee461fb ./generation_config.json
|
||||
027f9b7a7bb96f30962b5be7e4e4a5f1 ./hf_quant_config.json
|
||||
57cdec3085e0805c4ad5875f61c63b68 ./model-00030-of-00047.safetensors
|
||||
be93abf8f84e890b8dd1d4187ef540c2 ./model-00001-of-00047.safetensors
|
||||
a4a627a33e69e91491c9a3d3a7c9cba9 ./model-00031-of-00047.safetensors
|
||||
500a87d1fd1dd6a93400539227361f5d ./model-00002-of-00047.safetensors
|
||||
c732e4ee2f910af666418abd547d34cb ./model-00032-of-00047.safetensors
|
||||
68024a4367c3341402766fcc40d383f6 ./model-00003-of-00047.safetensors
|
||||
94188d8acbc7fd47ee99400507793d6a ./model-00033-of-00047.safetensors
|
||||
b5870fe5e8704e879895a7b5a3be5136 ./model-00004-of-00047.safetensors
|
||||
87b17b02643b1c14ad2e90900b3dfa89 ./model-00034-of-00047.safetensors
|
||||
f2f51d0be2c49ba6569d59d8d1db5236 ./model-00005-of-00047.safetensors
|
||||
915184c001e70fea709ffbb5242e4a8c ./model-00035-of-00047.safetensors
|
||||
fe2146b7c71775e51bad2322c52c34d1 ./model-00006-of-00047.safetensors
|
||||
56d618d4ba615cc19a0898d02d82bb52 ./model-00036-of-00047.safetensors
|
||||
03c0d5ea0471553cce658b70daed7b47 ./model-00007-of-00047.safetensors
|
||||
c76ac8dd7dbddbc4a636118ad8a5f713 ./model-00038-of-00047.safetensors
|
||||
49e246d6faa068641d2858e36b6bb5cd ./model-00008-of-00047.safetensors
|
||||
8f31262bc0a079c946a9ec1cdeb4051e ./model-00037-of-00047.safetensors
|
||||
b956d449de03a2db14fee14df7aaddac ./model-00009-of-00047.safetensors
|
||||
087456e9659f4d7c8c9a4af093b7c828 ./model-00039-of-00047.safetensors
|
||||
f94083fc155df999ab9ff27f6f67b978 ./model-00010-of-00047.safetensors
|
||||
bfd9606be27f5efbb34d85dc8fad3cea ./model-00040-of-00047.safetensors
|
||||
775a242af0663e5c65bc1b41a8930314 ./model-00011-of-00047.safetensors
|
||||
a4c70502c1470d1cd520f4f2b0f968d8 ./model-00041-of-00047.safetensors
|
||||
1d9d4ca43e411b977cded63b3447b8b8 ./model-00012-of-00047.safetensors
|
||||
f00a801a8de9a31b2ff686f6f8924d41 ./model-00042-of-00047.safetensors
|
||||
5feebc534bb09cee7b7e908aab8e2108 ./model-00013-of-00047.safetensors
|
||||
b58238047ce49362d39c36b54761c4cf ./model-00043-of-00047.safetensors
|
||||
c15c8fa6325af6ca302e10425b357199 ./model-00014-of-00047.safetensors
|
||||
fb652edaa0a450b7ce333829e363ee7d ./model.safetensors.index.json
|
||||
ce7e000a8d1c076b13346374f378e1f3 ./model-00015-of-00047.safetensors
|
||||
752f6cd2e6a4a2ea824d1b513530e0b0 ./tokenizer.json
|
||||
96ab98059044ac21eff43da3d9882689 ./model-00016-of-00047.safetensors
|
||||
3312b710133454ca7150bccad6381bd8 ./tokenizer_config.json
|
||||
82429b288fe2a97cb771e78bea60a0bd ./model-00017-of-00047.safetensors
|
||||
7605255172c381931a496b316faecdb7 ./model-00018-of-00047.safetensors
|
||||
e7fdbb1bd3e82f97831f4c1c69e0403a ./model-00019-of-00047.safetensors
|
||||
f5d355f9a737d7aa270d178a1bceeeec ./model-00020-of-00047.safetensors
|
||||
50b23a8dc4932840e5ae2062ca04b60a ./model-00021-of-00047.safetensors
|
||||
285172cbc5e900681efc5783f04a346a ./model-00022-of-00047.safetensors
|
||||
accdb4eb9e0831705bcd2d39e4f3bdcb ./model-00023-of-00047.safetensors
|
||||
9182f13eb2670db2a5312aaca2f9b1da ./model-00024-of-00047.safetensors
|
||||
198809340ec4795bdcfafdf7cd3074c7 ./model-00025-of-00047.safetensors
|
||||
da5e3a1afe95fe14c5d2d1f432f4a933 ./model-00026-of-00047.safetensors
|
||||
1d15d1696fd6d2cab3ed8726c3b25112 ./model-00027-of-00047.safetensors
|
||||
6147f7b5acb9a297eab276b16a58efe7 ./model-00028-of-00047.safetensors
|
||||
529b4048cf2f18d9eeaf8d1ce37b60cd ./model-00029-of-00047.safetensors
|
||||
ffe813a8385daf74f69f632d9437085c ./model-00044-of-00047.safetensors
|
||||
f67ddcaf1b68509fd6ab8d0399632a75 ./model-00045-of-00047.safetensors
|
||||
489dc1a568e671176a98e7cc23519c17 ./model-00046-of-00047.safetensors
|
||||
3a7abe2df1bd98e749ee8733370ea78b ./model-00047-of-00047.safetensors
|
||||
15
deploy/manifests/README.md
Normal file
15
deploy/manifests/README.md
Normal file
@ -0,0 +1,15 @@
|
||||
# 权重完整性清单（md5）
|
||||
|
||||
2026-09-08 在 174.1.60.1 `/data/hf_models/` 下对两个模型目录逐文件 `md5sum` 的原样输出。
|
||||
用于：新机器部署前核对权重传输完整性、或怀疑权重被改动时做漂移检测。
|
||||
|
||||
校验方法（在权重目录下）：
|
||||
|
||||
```bash
|
||||
md5sum -c /path/to/GLM-5.3-NVFP4.md5 # 清单内路径为 ./ 相对路径
|
||||
```
|
||||
|
||||
| 清单 | 模型 | 规模 | 用途 |
|
||||
|---|---|---|---|
|
||||
| `GLM-5.3-NVFP4.md5` | GLM-5.3-NVFP4 主模型 | 47 分片 + 配置共 55 文件 | 方案 A-F 全部部署的目标模型 |
|
||||
| `GLM-5.3-DFlash2.md5` | GLM-5.3-DFlash2 草稿模型 | 单分片 model.safetensors + 配置共 4 文件 | 方案 E 与方案 F 的 DFLASH 投机草稿 |
|
||||
31
deploy/profiles/910c/dsv4_910c_vllm_tp_dp_matrix.env
Normal file
31
deploy/profiles/910c/dsv4_910c_vllm_tp_dp_matrix.env
Normal file
@ -0,0 +1,31 @@
|
||||
# DeepSeek-V4-Flash w8a8-mtp vLLM-Ascend TPxDP profile on a single 910C node
|
||||
# (8x Ascend910 = 16 dies, 64GB HBM/die).
|
||||
# Launch via BOOTSTRAP (base64) so the JSON-bearing args survive the image
|
||||
# entrypoint; the container runs vllm serve in the foreground.
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||
|
||||
PLATFORM=910c
|
||||
EXPERIMENT=dsv4_910c_vllm_tp_dp_matrix
|
||||
MODEL_NAME=DeepSeek-V4-Flash
|
||||
ENGINE=vllm
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=local/vllm-ascend:0.23-a3-dsv4-sglang
|
||||
CONTAINER_NAME=vllm-ascend-dsv4-910c_tp${TP}_dp${DP}
|
||||
MODEL_PATH=/mnt/models/DeepSeek-V4-Flash-w8a8-mtp
|
||||
SERVED_MODEL_NAME=dsv4
|
||||
PORT=30052
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=/usr/local/python3.12.13/bin/python3
|
||||
|
||||
DEVICE_VARS="ASCEND_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTORCH_NPU_ALLOC_CONF=expandable_segments:True PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 OMP_PROC_BIND=false OMP_NUM_THREADS=10 HCCL_BUFFSIZE=1024 VLLM_ASCEND_ENABLE_FLASHCOMM1=1 TASK_QUEUE_ENABLE=1 HCCL_OP_EXPANSION_MODE=AIV LD_PRELOAD=/usr/lib64/libjemalloc.so.2"
|
||||
|
||||
DOCKER_FLAGS="--privileged --ipc host --shm-size 512g --network host --ulimit memlock=-1 --device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 --device /dev/davinci4 --device /dev/davinci5 --device /dev/davinci6 --device /dev/davinci7 --device /dev/davinci8 --device /dev/davinci9 --device /dev/davinci10 --device /dev/davinci11 --device /dev/davinci12 --device /dev/davinci13 --device /dev/davinci14 --device /dev/davinci15 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc"
|
||||
|
||||
VOLUMES="/usr/local/dcmi:/usr/local/dcmi /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool /usr/local/bin/npu-smi:/usr/local/bin/npu-smi /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info /etc/ascend_install.info:/etc/ascend_install.info /etc/hccn.conf:/etc/hccn.conf ${MODEL_PATH}:${MODEL_PATH}:ro /mnt:/mnt"
|
||||
|
||||
LAUNCH_ARGS="${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --block-size 128 --tensor-parallel-size ${TP} --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION:-0.9} --max-model-len ${MAX_MODEL_LEN:-1048576} --max-num-seqs ${MAX_NUM_SEQS:-64} --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS:-10240} --api-server-count 1 --host 0.0.0.0 --port ${PORT} --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --quantization ascend --safetensors-load-strategy prefetch --enable-expert-parallel --async-scheduling --model-loader-extra-config '{\"enable_multithread_load\": \"true\", \"num_threads\": 128}' --compilation-config '{\"cudagraph_mode\": \"FULL_DECODE_ONLY\"}' --additional-config '{\"ascend_compilation_config\":{\"enable_npugraph_ex\":true,\"enable_static_kernel\":false},\"enable_cpu_binding\": true,\"multistream_overlap_shared_expert\":true}'"
|
||||
DP_FLAG="--data-parallel-size ${DP}"
|
||||
|
||||
BOOTSTRAP="for sh in /usr/local/Ascend/ascend-toolkit/set_env.sh /usr/local/Ascend/cann-9.0.1/share/info/ascendnpu-ir/bin/set_env.sh /usr/local/Ascend/cann-9.0.0/share/info/ascendnpu-ir/bin/set_env.sh /usr/local/Ascend/nnal/atb/set_env.sh; do [ -f \$sh ] && source \$sh && break; done; exec vllm serve ${LAUNCH_ARGS}"
|
||||
31
deploy/profiles/910c/glm52_910c_vllm_tp_dp_matrix.env
Normal file
31
deploy/profiles/910c/glm52_910c_vllm_tp_dp_matrix.env
Normal file
@ -0,0 +1,31 @@
|
||||
# GLM-5.2 w4a8c8 vLLM-Ascend TPxDP profile on a single 910C node
|
||||
# (8x Ascend910 = 16 dies, 64GB HBM/die). GLM5.2-tuned image.
|
||||
# Launch via BOOTSTRAP (base64) so the JSON-bearing args survive the image
|
||||
# entrypoint; the container runs vllm serve in the foreground.
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||
|
||||
PLATFORM=910c
|
||||
EXPERIMENT=glm52_910c_vllm_tp_dp_matrix
|
||||
MODEL_NAME=GLM-5.2
|
||||
ENGINE=vllm
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=local/vllm-ascend:0.23-a3-20260718-sglang
|
||||
CONTAINER_NAME=vllm-ascend-glm52-910c_tp${TP}_dp${DP}
|
||||
MODEL_PATH=/mnt/models/GLM-5.2-w4a8c8
|
||||
SERVED_MODEL_NAME=glm-5.2
|
||||
PORT=30050
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=/usr/local/python3.12.13/bin/python3
|
||||
|
||||
DEVICE_VARS="ASCEND_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTORCH_NPU_ALLOC_CONF=expandable_segments:True PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 VLLM_ASCEND_ENABLE_FLASHCOMM1=1"
|
||||
|
||||
DOCKER_FLAGS="--privileged --ipc host --shm-size 512g --network host --ulimit memlock=-1 --device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 --device /dev/davinci4 --device /dev/davinci5 --device /dev/davinci6 --device /dev/davinci7 --device /dev/davinci8 --device /dev/davinci9 --device /dev/davinci10 --device /dev/davinci11 --device /dev/davinci12 --device /dev/davinci13 --device /dev/davinci14 --device /dev/davinci15 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc"
|
||||
|
||||
VOLUMES="/usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool /usr/local/bin/npu-smi:/usr/local/bin/npu-smi /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info /etc/ascend_install.info:/etc/ascend_install.info /etc/hccn.conf:/etc/hccn.conf ${MODEL_PATH}:${MODEL_PATH}:ro /usr/local/dcmi:/usr/local/dcmi /mnt:/mnt"
|
||||
|
||||
LAUNCH_ARGS="${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --trust-remote-code --block-size 128 --tensor-parallel-size ${TP} --enable-expert-parallel --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION:-0.95} --max-model-len ${MAX_MODEL_LEN:-131072} --max-num-seqs ${MAX_NUM_SEQS:-256} --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS:-8192} --api-server-count 1 --host 0.0.0.0 --port ${PORT} --compilation-config '{\"cudagraph_mode\": \"FULL_DECODE_ONLY\"}' --additional-config '{\"enable_dsa_cp\": true,\"enable_sparse_sfa_c8\": false, \"enable_sparse_li_c8\": true,\"enable_balance_scheduling\": true,\"multistream_overlap_shared_expert\":true}'"
|
||||
DP_FLAG="--data-parallel-size ${DP}"
|
||||
|
||||
BOOTSTRAP="for sh in /usr/local/Ascend/ascend-toolkit/set_env.sh /usr/local/Ascend/cann-9.0.1/share/info/ascendnpu-ir/bin/set_env.sh /usr/local/Ascend/cann-9.0.0/share/info/ascendnpu-ir/bin/set_env.sh /usr/local/Ascend/nnal/atb/set_env.sh; do [ -f \$sh ] && source \$sh && break; done; exec vllm serve ${LAUNCH_ARGS}"
|
||||
27
deploy/profiles/p800/dsv4_w8a8_int8_sglang.env
Normal file
27
deploy/profiles/p800/dsv4_w8a8_int8_sglang.env
Normal file
@ -0,0 +1,27 @@
|
||||
# P800 + SGLang-XPU + DeepSeek-V4-Flash-INT8 deployment profile.
|
||||
# Model-team only. Ops should NOT run `python -m sskj.deploy`; they only
|
||||
# run `python -m sskj.bench` against the URL this profile serves.
|
||||
|
||||
PLATFORM=p800
|
||||
MODEL_NAME=DeepSeek-V4-Flash-INT8
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202
|
||||
CONTAINER_NAME=sglang-dsv4-flash-tpdp
|
||||
MODEL_PATH=/data1/models/DeepSeek-V4-Flash-INT8
|
||||
SERVED_MODEL_NAME=deepseek-v4-flash-int8
|
||||
PORT=30014
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
CONTAINER_PYTHON=/root/miniconda/envs/python310_torch25_cuda/bin/python
|
||||
PATCH_ROOT=${ROOT_DIR}/platforms/patches/kunlun_p800
|
||||
|
||||
DEVICE_VARS="XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 CUDA_DEVICE_ORDER=OAM_ID"
|
||||
ENGINE_ENV="SGLANG_USE_TRANSFORMERS_V5_TOKENIZER=1 XMLIR_FORCE_USE_XPU_GRAPH=1 SGLANG_DSV4_MODE=2604 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_NSA_DUAL_STREAM=true SGLANG_NSA_QUANT_WQ_B_WK=false SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1 SGLANG_OPT_DEEPGEMM_HC_PRENORM=false SGLANG_OPT_USE_TILELANG_MHC_PRE=1 SGLANG_OPT_USE_TILELANG_MHC_POST=1 SGLANG_CLEAN_REQUEST_WHEN_RETRACT=1 SGLANG_SET_CPU_AFFINITY=1 SGLANG_OPT_USE_KLX_TOPK_KERNEL=1 XSGL_INTERTYPE_BFP16=1 ENABLE_FAST_BFP16_ATTN=1 XSGL_USE_DEEP_GEMM_BMM=1 XSGL_XDNN_QUANT=1 XSGL_FUSE_RMS_NORM_QUANT=1 XSGL_TRANSPOSE_MATMUL_WEIGHT=1 XINFER_QUANT_SDNN=1 XSGL_USE_MOE_SIGMOID_GROUP_TOPK_NORM=1 XSGL_EARLY_FIRST_TOKEN=1 XSGL_ENABLE_TGEMM_FP16=1 SGLANG_ENABLE_SPEC_V2=True SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 PYTHONDONTWRITEBYTECODE=1 XTORCH_OPS_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/xtorch_ops XPU_RUNTIME_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/xcudart/lib BKCL_TREE_THRESHOLD=1048576 CUDA_ENABLE_P2P_NO_UVA=1 NCCL_IB_GID_INDEX=3 IS_DSV4=1 MC_CUSTOM_TOPO_JSON=/workspace/nic_priority_matrix_test.json SGLANG_DSV4_FP4_EXPERTS=false SGLANG_APPLY_CONFIG_BACKUP=auto BKCL_ENABLE_XDR=1 BKCL_RDMA_NICS=eth1,eth1,eth3,eth3,eth5,eth5,eth7,eth7 BKCL_RDMA_VERBS=1 XSGL_INT8_LM_HEAD=1 SGLANG_P800_ALL_GATHER_FALLBACK=0"
|
||||
|
||||
DOCKER_FLAGS="--privileged --network host --ipc host --device /dev/xpu0:/dev/xpu0 --device /dev/xpu1:/dev/xpu1 --device /dev/xpu2:/dev/xpu2 --device /dev/xpu3:/dev/xpu3 --device /dev/xpu4:/dev/xpu4 --device /dev/xpu5:/dev/xpu5 --device /dev/xpu6:/dev/xpu6 --device /dev/xpu7:/dev/xpu7 --device /dev/xpuctrl:/dev/xpuctrl"
|
||||
VOLUMES="${MODEL_PATH}:/models:ro ${MODEL_PATH}:${MODEL_PATH}:ro ${PATCH_ROOT}/nic_priority_matrix_test.json:/workspace/nic_priority_matrix_test.json:ro"
|
||||
|
||||
LAUNCH_ARGS="--host 0.0.0.0 --port ${PORT} --model-path /models --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --page-size 64 --mem-fraction-static 0.8 --tensor-parallel-size ${TP} --disable-shared-experts-fusion --quantization w8a8_int8 --kv-cache-dtype float16 --disable-piecewise-cuda-graph --cuda-graph-max-bs 32 --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --constrained-json-disable-any-whitespace --enable-metrics --enable-request-time-stats-logging --context-length 65536"
|
||||
DP_FLAG="--dp-size ${DP}"
|
||||
BOOTSTRAP="cd /workspace; find /root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true; ${CONTAINER_PYTHON} -m pip install --upgrade safetensors -q; ${CONTAINER_PYTHON} -m pip install https://files.pythonhosted.org/packages/14/8b/2a1333a6455c6fad401c2285dee6f58016c55b1cb44cae3a31f8a9cc7d83/apache_tvm_ffi-0.1.0b2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl -q; ${CONTAINER_PYTHON} -c \"import torch; torch.float8_e8m0fnu = torch.uint8; import runpy, sys; sys.argv[0] = 'sglang.launch_server'; runpy.run_module('sglang.launch_server', run_name='__main__')\" ${LAUNCH_ARGS}"
|
||||
29
deploy/profiles/p800/glm52_w8a8_int8_dynamic.env
Normal file
29
deploy/profiles/p800/glm52_w8a8_int8_dynamic.env
Normal file
@ -0,0 +1,29 @@
|
||||
# P800 + SGLang-XPU + GLM-5.2-W8A8-INT8-Dynamic deployment profile.
|
||||
# Supports the 2-node deploy previously living in deploy_glm52_2nodes.sh.
|
||||
# Model-team only. Ops should not run `python -m sskj.deploy`.
|
||||
|
||||
PLATFORM=p800
|
||||
MODEL_NAME=GLM-5.2-W8A8-INT8-Dynamic
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202
|
||||
CONTAINER_NAME=glm52_deploy
|
||||
MODEL_PATH=/data1/models/GLM-5.2-W8A8-INT8-Dynamic
|
||||
SERVED_MODEL_NAME=glm52
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
NNODES=2
|
||||
MASTER_IP=10.100.11.2
|
||||
DIST_PORT=6311
|
||||
DP_SIZE=2
|
||||
|
||||
DEVICE_VARS="XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 CUDA_DEVICE_ORDER=OAM_ID"
|
||||
ENGINE_ENV="BKCL_ENABLE_XDR=1 BKCL_ENABLE_IBGDA=1 BKCL_SOCKET_IFNAME=eth0 BKCL_RDMA_NICS=eth1,eth2,eth3,eth4,eth5,eth6,eth7,eth8 BKCL_TREE_THRESHOLD=1048576 XSHMEM_MODE=1 XSHMEM_QP_NUM_PER_RANK=32 BKCL_RDMA_VERBS=1 NCCL_IB_GID_INDEX=3 CUDA_ENABLE_P2P_NO_UVA=1 SGLANG_USE_TRANSFORMERS_V5_TOKENIZER=1 XSGL_INTERTYPE_BFP16=1 ENABLE_FAST_BFP16_ATTN=1 XSGL_USE_DEEP_GEMM_BMM=1 XSGL_XDNN_QUANT=1 XSGL_FUSE_RMS_NORM_QUANT=1 XSGL_TRANSPOSE_MATMUL_WEIGHT=1 XINFER_QUANT_SDNN=1 XMLIR_FORCE_USE_XPU_GRAPH=1 XSGL_USE_MOE_SIGMOID_GROUP_TOPK_NORM=1 XSGL_INT8_LM_HEAD=1 SGLANG_CLEAN_REQUEST_WHEN_RETRACT=1 SGLANG_ENABLE_SPEC_V2=True SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 USE_FAST_BFP16_FC=1 XPUAPI_SDNN_BF16_ROUND_MODE=3 XMLIR_ENABLE_FAST_FC=true XSGL_FAST_SWIGLU=1 SGLANG_NSA_DUAL_STREAM=true SGLANG_NSA_QUANT_WQ_B_WK=false XPU_FLASH_ATTENTION_DECODER_USE_BALANCE=true"
|
||||
|
||||
DOCKER_FLAGS="--privileged --network host --security-opt label=disable --shm-size 64m --device /dev/xpu0:/dev/xpu0 --device /dev/xpu1:/dev/xpu1 --device /dev/xpu2:/dev/xpu2 --device /dev/xpu3:/dev/xpu3 --device /dev/xpu4:/dev/xpu4 --device /dev/xpu5:/dev/xpu5 --device /dev/xpu6:/dev/xpu6 --device /dev/xpu7:/dev/xpu7 --device /dev/xpuctrl:/dev/xpuctrl"
|
||||
VOLUMES="${MODEL_PATH}:/models:ro ${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--host 0.0.0.0 --port ${PORT} --model-path /models --dist-init-addr ${MASTER_IP}:${DIST_PORT} --nnodes ${NNODES} --node-rank ${NODE_RANK} --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 65536 --chunked-prefill-size 16384 --max-running-requests 128 --tp-size 16 --ep-size 16 --dp-size ${DP_SIZE} --enable-dp-attention --moe-dense-tp-size 1 --enable-dp-lm-head --moe-a2a-backend deepep --deepep-mode auto --quantization w8a8_int8 --disable-shared-experts-fusion --kv-cache-dtype float16 --disable-radix-cache --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --enable-nsa-prefill-context-parallel --enable-single-batch-overlap"
|
||||
|
||||
BOOTSTRAP="source /root/miniconda/bin/activate python310_torch25_cuda; cd /workspace; pip install --no-index --find-links=/workspace/ /workspace/transformers-5.3.0-py3-none-any.whl 2>/dev/null || true; nohup python3 -m sglang.launch_server ${LAUNCH_ARGS} > /workspace/sglang_log_${NODE_RANK}.log 2>&1 & sleep infinity"
|
||||
34
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp16.env
Normal file
34
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp16.env
Normal file
@ -0,0 +1,34 @@
|
||||
# DeepSeek-V4-Pro SGLang multi-node TP=16 EP=2 deployment profile (2x RTX 6000D).
|
||||
# Nodes: 10.101.0.11 (rank 0) + 10.101.0.13 (rank 1), 16x RTX 6000D (85GB).
|
||||
# Key: SGLANG_SHARED_EXPERT_TP1=1 + --ep-size 2 -> TP=16 works with FP8 block=128.
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=dsv4_pro6000_sglang_tp16
|
||||
MODEL_NAME=DeepSeek-V4-Pro
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/DeepSeek-V4-Pro
|
||||
SERVED_MODEL_NAME=default
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=10.101.0.11
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||
NNODES=2
|
||||
NODE_HOSTS="10.101.0.11 10.101.0.13"
|
||||
NODE_SSH_USER=root
|
||||
MASTER_IP=10.101.0.11
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="NCCL_SOCKET_IFNAME=eth0 NCCL_DEBUG=WARN SGLANG_SHARED_EXPERT_TP1=1 PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --shm-size=20g"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --ep-size 2 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --host 0.0.0.0 --port ${PORT} --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256"
|
||||
32
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp16_eagle.env
Normal file
32
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp16_eagle.env
Normal file
@ -0,0 +1,32 @@
|
||||
# DeepSeek-V4-Pro SGLang multi-node TP=16 EP=2 + EAGLE speculative decoding
|
||||
# (2x RTX 6000D). Same topology as dsv4_pro6000_sglang_tp16, plus EAGLE flags.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=dsv4_pro6000_sglang_tp16_eagle
|
||||
MODEL_NAME=DeepSeek-V4-Pro
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/DeepSeek-V4-Pro
|
||||
SERVED_MODEL_NAME=default
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=10.101.0.11
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||
NNODES=2
|
||||
NODE_HOSTS="10.101.0.11 10.101.0.13"
|
||||
NODE_SSH_USER=root
|
||||
MASTER_IP=10.101.0.11
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="NCCL_SOCKET_IFNAME=eth0 NCCL_DEBUG=WARN SGLANG_SHARED_EXPERT_TP1=1 PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --shm-size=20g"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --ep-size 2 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --host 0.0.0.0 --port ${PORT} --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4"
|
||||
24
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp_dp_matrix.env
Normal file
24
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp_dp_matrix.env
Normal file
@ -0,0 +1,24 @@
|
||||
# DeepSeek-V4-Flash SGLang TPxDP matrix profile on a single RTX 6000D node (8 GPUs).
|
||||
# TP=2/DP=4 is excluded because Marlin weight loading OOMs on this machine.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=dsv4_pro6000_sglang_tp_dp_matrix
|
||||
MODEL_NAME=DeepSeek-V4-Flash
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
|
||||
CONTAINER_NAME=${EXPERIMENT}_sglang_tp${TP}_dp${DP}
|
||||
MODEL_PATH=/data/6000D/DeepSeek-V4-Flash
|
||||
SERVED_MODEL_NAME=deepseek-v4-flash
|
||||
PORT=30031
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --privileged --ipc=host --network host --ulimit memlock=-1 --ulimit stack=67108864"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --trust-remote-code --tp-size ${TP} --moe-runner-backend auto --mem-fraction-static 0.9 --context-length 131072 --max-running-requests 64 --host 0.0.0.0 --port ${PORT}"
|
||||
24
deploy/profiles/pro6000/dsv4_pro6000_vllm_tp_dp_matrix.env
Normal file
24
deploy/profiles/pro6000/dsv4_pro6000_vllm_tp_dp_matrix.env
Normal file
@ -0,0 +1,24 @@
|
||||
# DeepSeek-V4-Flash vLLM TPxDP matrix profile on a single RTX 6000D node (8 GPUs).
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=dsv4_pro6000_vllm_tp_dp_matrix
|
||||
MODEL_NAME=DeepSeek-V4-Flash
|
||||
ENGINE=vllm
|
||||
SERVER_CMD=vllm serve
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=vllm-sm120-dsv4:0.25.1-fi0.6.14
|
||||
CONTAINER_NAME=${EXPERIMENT}_vllm_tp${TP}_dp${DP}
|
||||
MODEL_PATH=/data/6000D/DeepSeek-V4-Flash
|
||||
SERVED_MODEL_NAME=deepseek-v4-flash
|
||||
PORT=30030
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --privileged --ipc=host --network host --ulimit memlock=-1 --ulimit stack=67108864"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--model ${MODEL_PATH} --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --tensor-parallel-size ${TP} --gpu-memory-utilization 0.9 --max-model-len 131072 --max-num-seqs 128 --host 0.0.0.0 --port ${PORT}"
|
||||
DP_FLAG="--data-parallel-size ${DP}"
|
||||
37
deploy/profiles/pro6000/glm52_pro6000_sglang_tp16.env
Normal file
37
deploy/profiles/pro6000/glm52_pro6000_sglang_tp16.env
Normal file
@ -0,0 +1,37 @@
|
||||
# GLM-5.2-FP8 SGLang multi-node TP=16 deployment profile (2x RTX 6000D).
|
||||
# node0 = 174.1.51.5 (pro6000D.1, HTTP API), node1 = 174.1.51.7 (pro6000D.3,
|
||||
# compute-only). Model ~700GB fp8 so 2-node TP=16 is mandatory.
|
||||
# Requires the tilelang/dsa patch mounts below (experiment-dir sources).
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm52_pro6000_sglang_multinode_tp16
|
||||
MODEL_NAME=GLM-5.2-FP8
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
|
||||
CONTAINER_NAME=${EXPERIMENT}_sglang_tp${TP}_dp${DP}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/GLM-5.2-FP8
|
||||
SERVED_MODEL_NAME=GLM-5.2-FP8
|
||||
PORT=30031
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.51.5
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
TP=16
|
||||
DP=1
|
||||
|
||||
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||
NNODES=2
|
||||
NODE_HOSTS="174.1.51.5 174.1.51.7"
|
||||
NODE_SSH_USER=root
|
||||
MASTER_IP=174.1.51.5
|
||||
DIST_PORT=50000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:256 NCCL_IB_HCA=mlx5_0,mlx5_3 NCCL_MIN_NCHANNELS=8 NCCL_IB_QPS_PER_CONNECTION=4 NCCL_NET_GDR_LEVEL=PHB NCCL_SOCKET_IFNAME=eth1 NCCL_IB_RETRY_CNT=14 NCCL_IB_TIMEOUT=16"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --privileged --ipc=host --network host --ulimit memlock=-1 --ulimit stack=67108864"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro ${ROOT_DIR}/sglang_nightly_cu13_cache:/root/.cache"
|
||||
PATCH_MOUNTS="${ROOT_DIR}/experiments/pro6000/glm52_pro6000_sglang_multinode_tp16/tilelang_kernel_sm120.py:/sgl-workspace/sglang/python/sglang/kernels/ops/attention/dsa/tilelang_kernel.py:ro ${ROOT_DIR}/experiments/pro6000/glm52_pro6000_sglang_multinode_tp16/dsa_backend.py:/sgl-workspace/sglang/python/sglang/srt/layers/attention/dsa_backend.py:ro ${ROOT_DIR}/experiments/pro6000/glm52_pro6000_sglang_multinode_tp16/dsa_indexer.py:/sgl-workspace/sglang/python/sglang/srt/layers/attention/dsa/dsa_indexer.py:ro"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --trust-remote-code --tp-size ${TP} --dp-size ${DP} --quantization fp8 --kv-cache-dtype bfloat16 --moe-runner-backend auto --mem-fraction-static 0.80 --context-length 131072 --max-running-requests 64 --cuda-graph-backend-decode disabled --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --host 0.0.0.0 --port ${PORT} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --nnodes ${NNODES} --node-rank ${NODE_RANK}"
|
||||
52
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_decode.env
Normal file
52
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_decode.env
Normal file
@ -0,0 +1,52 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色3: decode 节点（方案F，跑在 6000D-2 = 174.1.60.2，8 卡）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> prefill -> [本角色] -> router。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行部署脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_s1_decode.sh
|
||||
# （launch 脚本同目录 s1_decode_launch.sh）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 配方 = 方案 E v5（TP8 + DFLASH + MRR12 + fa4 + window2048）+ PD decode flags；
|
||||
# KV 池 214,336 token（比单机 E 的 243,584 少，PD 传输缓冲占显存）
|
||||
# - 容量属性（实测判决的核心）: 池按 16k 场景定容 → 128k 仅容 1 个驻留（cc4 时 TTFT
|
||||
# 堆到 31.0s）、64k 容 3、16k 容 12（MRR12 上限）。长上下文负载该池就是瓶颈
|
||||
# - 必须挂补丁树 /data/sglang_patch_glm53:/sgl-workspace/sglang（唯一改动
|
||||
# python/sglang/srt/speculative/spec_info.py 接 build_dflash_family_disagg_draft_input，
|
||||
# DFlash PD 冷启动接线）+ 装 mooncake wheel（--no-deps，launch 脚本自装）
|
||||
# - --network host + --device /dev/infiniband + --ulimit memlock=-1，IB 设备 mlx5_0-3
|
||||
# - DFlash accept 在 12-batch verify 下掉到 1.8（单机 EAGLE 2.84）——decode 侧并发
|
||||
# verify 是 DFlash 的弱势区，场景二吞吐上限由此而来
|
||||
# - 60.2 平时空闲但 GPU7 常有外部裸金属任务（main_v2.py）：动卡前核实归属，勿清
|
||||
#
|
||||
# 实测成绩: 飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 场景二判决: "两机买 TTFT、不买吞吐"（÷2 单机等效 74-85 tok/s = A 的 75-96%，TTFT 减半）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_decode
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
ROLE=pd-decode
|
||||
NODE=174.1.60.2
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-s1-decode
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1200
|
||||
|
||||
TP=8
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=12
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma PYTHONUNBUFFERED=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --shm-size 64g --ulimit memlock=-1 --device /dev/infiniband --restart unless-stopped"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/flashkda_deploy/wheels:/mc_wheels:ro /data/sglang_patch_glm53:/sgl-workspace/sglang"
|
||||
|
||||
BOOTSTRAP="pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --speculative-draft-window-size 2048 --disaggregation-mode decode --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
29
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_master.env
Normal file
29
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_master.env
Normal file
@ -0,0 +1,29 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色1: Mooncake 元数据服务 mc-master（方案F，双机 6000D-1 + 6000D-2）。
|
||||
# 四角色链之一，启动顺序强制: mc-master(60.1:50051) -> prefill(60.1:30000) -> decode(60.2:30000) -> router(60.2:31000)。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md；可执行脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_smoke_master.sh
|
||||
#
|
||||
# 关键点:
|
||||
# - 镜像自带的 mooncake_master 二进制（/opt/sglang/bin/mooncake_master），无需 GPU
|
||||
# --network host，监听 50051；prefill/decode 容器通过 MOONCAKE_MASTER=174.1.60.1:50051 注册
|
||||
# - 就绪判据: ss -tln | grep :50051（脚本 sleep 3 后检查）
|
||||
# - 拆链时必须先删本容器之外的角色再删它? 否——顺序无依赖，但生产恢复 60.1 时
|
||||
# 本容器与 glm53-pd-smoke-prefill 都要删干净、等显存排空再拉生产容器
|
||||
#
|
||||
# 实测背景（2026-09-08 方案F 双场景压测）: 13/13 干净点，质量门 6/7。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_master
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang-mooncake-master
|
||||
RUNTIME=docker
|
||||
ROLE=pd-master
|
||||
NODE=174.1.60.1
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=mc-master
|
||||
PORT=50051
|
||||
|
||||
DOCKER_FLAGS="--network host --restart unless-stopped"
|
||||
ENTRYPOINT="/opt/sglang/bin/mooncake_master"
|
||||
|
||||
READY_CHECK="ss -tln | grep :50051"
|
||||
59
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_prefill.env
Normal file
59
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_prefill.env
Normal file
@ -0,0 +1,59 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色2: prefill 节点（方案F，跑在 6000D-1 = 174.1.60.1，8 卡）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> [本角色] -> decode -> router。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行部署脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_probe.sh
|
||||
# （参数 = launch 脚本名；launch 脚本同目录 p1b_prefill_tp4pp2_cps8k_launch.sh 及其 radix 变体）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 拓扑 TP4 PP2（4卡/stage × 2 stage）+ DFLASH 草稿（草稿只跑 prefill 侧草稿 KV，
|
||||
# export SGLANG_DFLASH_PD_DRAFT_KV_TRANSFER=0 = 草稿 KV 不跨机传输）
|
||||
# - mem 0.78 低于单机方案（PD 模式下 mooncake 传输缓冲占显存）；cps 8192（非 D 方案的 16384）
|
||||
# - 基线 launch 脚本带 --disable-radix-cache；场景一（90% 命中）实测用 radix 变体
|
||||
# （p1b_prefill_tp4pp2_cps8k_radix_launch.sh，唯一差异 = 删掉该 flag）
|
||||
# - 必须挂补丁树 /data/sglang_patch_glm53:/sgl-workspace/sglang（含 1 个文件改动:
|
||||
# python/sglang/srt/speculative/spec_info.py 接 build_dflash_family_disagg_draft_input，
|
||||
# DFlash PD 冷启动接线；无此补丁 decode 首请求 400）
|
||||
# - 必须装 mooncake wheel /data/flashkda_deploy/wheels/mooncake_transfer_engine_cuda13-
|
||||
# 0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl（--no-deps，容器内 launch 脚本自装）
|
||||
# - --network host + --device /dev/infiniband + --ulimit memlock=-1，IB 设备 mlx5_0-3
|
||||
# - 前置硬检查: mc-master 50051 必须已监听（deploy_pd_probe.sh 自带）
|
||||
# - 60.1 是生产机（日常跑 glm53-pp4）：拉本角色前须停生产容器，测完等显存 <2000MiB
|
||||
# 再跑 deploy_glm53_pp4.sh 恢复
|
||||
#
|
||||
# 实测成绩: 飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 128k cc1 TTFT 3.91s = 六方案最低；跨机 prefill/decode 重叠使 RDMA 传输被完全吸收。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_prefill
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
ROLE=pd-prefill
|
||||
NODE=174.1.60.1
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-pd-smoke-prefill
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1500
|
||||
|
||||
TP=4
|
||||
PP=2
|
||||
MEM_FRACTION_STATIC=0.78
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma SGLANG_DFLASH_PD_DRAFT_KV_TRANSFER=0 PYTHONUNBUFFERED=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --shm-size 64g --ulimit memlock=-1 --device /dev/infiniband --restart unless-stopped"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/flashkda_deploy/wheels:/mc_wheels:ro /data/sglang_patch_glm53:/sgl-workspace/sglang"
|
||||
|
||||
BOOTSTRAP="pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
|
||||
# 场景一（90% 命中）变体: LAUNCH_ARGS 追加 --disable-radix-cache 删除（radix 开）。
|
||||
# 基线（本 profile 口径）= radix off。
|
||||
35
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_router.env
Normal file
35
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_router.env
Normal file
@ -0,0 +1,35 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色4: MiniLB 路由（方案F，跑在 6000D-2 = 174.1.60.2:31000）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> prefill -> decode -> [本角色]。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_smoke_router.sh
|
||||
#
|
||||
# 关键点:
|
||||
# - 镜像内自带 sglang_router（python3 -m sglang_router.launch_router），无需 GPU
|
||||
# - --prefill 参数形态: URL 后跟 prefill 侧 disaggregation-bootstrap-port（28800），
|
||||
# 这是 KV 传输握手口，漏掉则路由建立后首请求挂起；--decode 只有 URL
|
||||
# - bench 与质量门全部经 :31000/generate 打（--url http://174.1.60.2:31000/generate），
|
||||
# 即全链路含 KV transfer + DFlash 草稿
|
||||
# - usage/accept 指标 router 转发后可能缺失: 命中率读 prefill 容器日志
|
||||
# （docker logs glm53-pd-smoke-prefill，PP2 下日志计数 ×2 不影响比值），
|
||||
# accept 读 decode 容器日志
|
||||
# - 质量门脚本: 用 quality_gate_605.sh 副本 sed 's/PORT=30000/PORT=31000/' 生成
|
||||
# - 就绪判据: ss -tln | grep :31000
|
||||
#
|
||||
# 实测背景: 全链 13/13 干净点、质量门 6/7（tool call 为 parser 配置缺口，与 B/D 同口径）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_router
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang-router
|
||||
RUNTIME=docker
|
||||
ROLE=pd-router
|
||||
NODE=174.1.60.2
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=pd-smoke-router
|
||||
PORT=31000
|
||||
|
||||
DOCKER_FLAGS="--network host --restart unless-stopped --entrypoint python3"
|
||||
LAUNCH_ARGS="-m sglang_router.launch_router --pd-disaggregation --mini-lb --prefill http://174.1.60.1:30000 28800 --decode http://174.1.60.2:30000 --host 0.0.0.0 --port 31000"
|
||||
|
||||
READY_CHECK="ss -tln | grep :31000"
|
||||
@ -0,0 +1,52 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 方案 D（2026-09-08 双场景补测；6000D-1 现役生产容器 glm53-pp4 的原样配方）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_pp4.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 脚本头注释是早期 TP4PP2 版残留（写"TP4 PP2 + IndexCache"），实际配置 TP=2 PP=4，
|
||||
# 以脚本 docker run 段为准；md5 def3c64c5dc19e1d507080e3366c3762
|
||||
# - 串行 prefill 有效速率 6.5-6.7k tok/s 为各方案最高；KV 池 1,040,384 token
|
||||
# （A 的 3.76 倍，≈61.6 请求驻留，超过 MRR 48——池在高并发下不构成约束）
|
||||
# - 场景二（16k 独立输入 cc8-64）成立：cc32 起反超 TP4PP2（方案B），cc40/64 输出
|
||||
# 203.8/208.2 tok/s；TTFT p50 五档全档低于 B
|
||||
# - 场景一（90% 命中长上下文）8/8 全败：TP2 长上下文每卡 KV 读量翻倍 + PP4 低并发
|
||||
# 流水空泡，单请求 decode 仅 16-19 tok/s。生产态 radix off 前缀命中恒 0，实际表现
|
||||
# 比报告 D 行（radix-on 最好情况）更差——长上下文/共享前缀负载勿用
|
||||
# - DSA 实测：上下文长度不影响 TPOT（52.8ms 恒定），并发才是驱动
|
||||
# - 上生产须补 --tool-call-parser glm47 与 --reasoning-parser glm45
|
||||
# - bench 口径：场景二 cc40/64 用 nreq 80/128（与其他方案 40/64 单轮满波不同，
|
||||
# 已在报告表注声明）
|
||||
#
|
||||
# 实测成绩：飞书《GLM-5.3-NVFP4 双场景压测报告》方案 D 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 质量：GSM8K×5 + 中文推理通过（6/7，tool call 为 parser 配置缺口）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp2pp4
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-pp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=16384
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -0,0 +1,44 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 + hicache deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 2026-09-09 128k 低并发容量扩容实验优胜配置（60.8 现役；60.5 交付 deploy_glm53_605_v2.sh 待执行）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_128k_capacity_topology/scripts/deploy_glm53_605_v2.sh
|
||||
#
|
||||
# 与方案 D（glm53_nvfp4_pro6000_sglang_tp2pp4.env）的关键差异（勿混淆）：
|
||||
# - radix/hicache 保持开启（60.5 真实流量命中率 90%+，关 radix 不可接受）；D 为 16k 独立输入场景关了 radix
|
||||
# - cu13 镜像 + 9 补丁只读挂载（de-GLOO request_receiver / decode_cuda_graph_runner_fix 等 r37 栈遗产，
|
||||
# nomtp 下 spec 相关补丁为惰性，de-GLOO 为 PP 通用修复）；D 用旧镜像 20260828 无挂载
|
||||
# - --reasoning-parser glm45 --tool-call-parser glm47 齐备（质量门 7/7）；D 当时 6/7
|
||||
# - --context-length 1048576（模型原生 1M；D 未设）；chunk 8192（D 16384）；MRR 16（D 48）
|
||||
# - 实测（60.8，i128k/o512 冷缓存）：KV 池 909,632 token（A 的 3.29×）、并发上限 c6、
|
||||
# 单条上限 ~909k（900k 实跑通过）、c4 输入/输出 5,907/23.1 tok/s、TTFT p50 43.4s、
|
||||
# 512k 单条 TTFT 88.4s、90% 命中 c4 输入 17,625 tok/s
|
||||
# - memfrac 0.85 为验证档；PP0 stage 空闲 22GB 提示 0.88 有余量（未验证，改动须重跑质量门+容量冒烟）
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_128k_capacity_topology
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1800
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=16
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
CONTEXT_LENGTH=1048576
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 SGLANG_PP_DEGLOO=1 SGLANG_PP_SPEC_FORCE_EAGER_DRAFT=1 SGLANG_PP_FORCE_EAGER_VERIFY=0 SGLANG_PP_SPEC_DEBUG=0"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host --cap-add SYS_PTRACE -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models + 9 patch ro-mounts (full list in scripts/deploy_glm53_605_v2.sh; files live in /root + /root/sglang_patch2 on the host)"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --reasoning-parser glm45 --tool-call-parser glm47 --enable-hierarchical-cache --hicache-ratio 3 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length ${CONTEXT_LENGTH} --host 0.0.0.0 --port ${PORT}"
|
||||
@ -0,0 +1,49 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 + latest镜像 + autotune profile（方案 D 升级版）。
|
||||
# 2026-09-09 DSV4 优化点迁移实验优胜配置：五点(16k/512 cc8-64)全胜 +1.2~+3.2%，
|
||||
# A/B/A 回切确认因果成立。见 experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/。
|
||||
# 已入档未执行；60.1 生产仍跑 nightly 版（glm53-pp4，见同目录 tp2pp4.env）。
|
||||
#
|
||||
# 关键点（实测，勿随意改）:
|
||||
# - 相对 tp2pp4.env 仅两处变化：镜像 nightly-dev-20260828 → latest(d6e72886)；
|
||||
# 删 --disable-flashinfer-autotune。启动参数其余逐字相同
|
||||
# - 镜像中性实测：latest 与 nightly 同配置五点差 ≤0.7%（换镜像无风险无收益）
|
||||
# - autotune 增益钉在 tactic 缓存上：SGLANG_CACHE_DIR 与 flashinfer JIT 缓存
|
||||
# 必须挂宿主持久盘，否则重部署重抽签（增益消失/不可复现，DSV4 §12.6 同款教训）
|
||||
# - 代价：available_gpu_mem 15.64→11.79 GB（tactic 缓冲 ≈3.9GB/卡），
|
||||
# KV 池不缩（1,040,384）
|
||||
# - PCIe-IPC AllReduce 包在同一实验中五点全降（-0.4~-3.5%）判负勿叠用：
|
||||
# TP2 单对端 NCCL AR 同 switch P2P 已近最优
|
||||
# - 上生产须补 --tool-call-parser glm47 与 --reasoning-parser glm45（D 系既有缺口）
|
||||
# - 部署器：experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/scripts/deploy_glm53_exp.sh
|
||||
# （EXP_NAME=<名> EXP_IMAGE=lmsysorg/sglang:latest AUTOTUNE=1）
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp2pp4_latest_autotune
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:latest
|
||||
DOCKER_IMAGE_DIGEST=sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9
|
||||
CONTAINER_NAME=glm53-pp4-autotune
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=16384
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 SGLANG_CACHE_DIR=/root/.cache/sglang"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/glm53_exp/fi_jit_cache:/root/.cache/flashinfer /data/glm53_exp/sglang_cache:/root/.cache/sglang /data/glm53_exp/triton_cache:/root/.triton"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -0,0 +1,47 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=4 PP=2 + hicache deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 2026-09-09 hit90 场景实验优胜配置（60.8 现役；60.5 交付 deploy_glm53_605_v3.sh 待执行）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/scripts/deploy_glm53_605_v3.sh
|
||||
# 60.8 启动口径：bash /root/deploy_ppmtp_r37.sh '--tp 4 --pp-size 2 --disable-overlap-schedule
|
||||
# --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length 1048576' nomtp 8192 0.90 1 1 0 0
|
||||
#
|
||||
# 与 TP2PP4-hicache（glm53_nvfp4_pro6000_sglang_tp2pp4_hicache.env）的关键差异（勿混淆）：
|
||||
# - tp2/pp4 → tp4/pp2，memfrac 0.85 → 0.90；其余（cu13+9 挂载、chunk 8192、MRR 16、
|
||||
# hicache 3、ctx 1048576、双 parser）逐项同构
|
||||
# - 实测 KV 池 647,040（fp8 KV 18.52GB/rank，PP0 初始化后剩 10.94GB）
|
||||
# - 取胜依据（hit90=90% 命中 i128k/o512 主场景）：out cc1/2/3/4 = 28.8/47.5/63.9/76.2、
|
||||
# cc8/16 = 106.4/125.7（vs TP2PP4：cc4 +14%/cc8 +15%/cc16 +1%）；cap cc2/cc4 = 62.9/98.5
|
||||
# 零排队；质量门 7/7
|
||||
# - 让步项（知情选择）：并发独立 128k 文档 4 条（TP2PP4 为 6）；512k 单条 151.9s
|
||||
# （TP2PP4 为 114.4s，PP4 单条巨请求 prefill 流水更优）；无投机解码
|
||||
# - 判决背景：DP attention 对本模型容量负收益、DCP 对 DSA 静默算错、MTP@128k accept 2.07 判负
|
||||
# （见 experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/README.md）
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_hit90_dp_dcp_bench
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1800
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=4
|
||||
PP=2
|
||||
MEM_FRACTION_STATIC=0.90
|
||||
MAX_RUNNING_REQUESTS=16
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
CONTEXT_LENGTH=1048576
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 SGLANG_PP_DEGLOO=1 SGLANG_PP_SPEC_FORCE_EAGER_DRAFT=1 SGLANG_PP_FORCE_EAGER_VERIFY=0 SGLANG_PP_SPEC_DEBUG=0"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host --cap-add SYS_PTRACE -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models + 9 patch ro-mounts (full list in scripts/deploy_glm53_605_v3.sh; files live in /root + /root/sglang_patch2 on the host, bundle md5 6922e53439991bc13feee72f3760704f)"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --reasoning-parser glm45 --tool-call-parser glm47 --enable-hierarchical-cache --hicache-ratio 3 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length ${CONTEXT_LENGTH} --host 0.0.0.0 --port ${PORT}"
|
||||
@ -0,0 +1,43 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=4 PP=2 + IndexCache deployment profile (single RTX 6000D node).
|
||||
# 场景二（16k 独立输入、cc8-32、prefill 主导）最优配置：吞吐 +41~79%、cc32 TTFT 减半
|
||||
# （对比 TP8+EAGLE 生产配置，2026-09-07 真实语料基线）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_optimal.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - index_topk_freq=4 为模型原生默认（层轴索引复用省 75% indexer，无质量损失）
|
||||
# - 禁投机解码：PP2 与投机框架不兼容（已实测）
|
||||
# - 本文件为场景二形态（--disable-radix-cache，独立输入无前缀复用）；
|
||||
# 场景一 90% 命中对比须启用 radix（唯一差异：去掉 --disable-radix-cache，
|
||||
# 见 deploy_glm53_optimal_s1.sh）
|
||||
# - mem 0.85：0.90 下 cuda graph 捕获余量不足会运行时 OOM（KV 池 569,600 = TP8 的 2.06 倍）
|
||||
# - --disable-custom-all-reduce：TP4 over PCIe 用自定义 AR 在本栈无收益
|
||||
# - 已知缺口：未带 --tool-call-parser glm47 --reasoning-parser glm45，质量门 6/7
|
||||
# （tool call 失败纯属参数缺失，非模型问题）；上生产必须补 parser
|
||||
# - 场景一（90% 命中低并发）该配置全面劣于 TP8+EAGLE（输出吞吐 −25~−65%），勿混用选型
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp4pp2_indexcache
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=4
|
||||
DP=1
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --ipc=host --shm-size 64g --entrypoint '' -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models:ro"
|
||||
|
||||
BOOTSTRAP="exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size 2 --mem-fraction-static 0.85 --max-running-requests 48 --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size 16384 --json-model-override-args '{\"index_topk_freq\": 4}' --host 0.0.0.0 --port ${PORT}"
|
||||
@ -0,0 +1,52 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=8 + DFlash2 speculative decoding profile (single RTX 6000D node, 8 GPUs).
|
||||
# 方案 E（2026-09-08 场景一补测，6000D-2）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_tp8_dflash2.sh
|
||||
# （v5 终版底稿，含四轮 OOM 战役完整教训注释；md5 5bf2b47c9349e5855b963e571e35c096）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - v5 配方核心：MRR 48→12（verify CUDA graph 4.04→0.83GB，真正起作用的杠杆）+
|
||||
# --speculative-draft-window-size 2048 + mem0.85/cps8192。四轮 OOM 根因与推导见脚本头注释
|
||||
# - DFLASH block-diffusion 草稿 7 tokens/步，draft 权重 GLM-5.3-DFlash2，fa4 draft
|
||||
# attention（fa4 会把 draft KV 强制 bf16，fp8 需换 flashinfer/triton 后端，仅省 0.35GB 未用）
|
||||
# - KV 池 243,584 token；kv fp8_e4m3 由模型配置自动带出（无需显式 flag）
|
||||
# - 底稿（本 profile LAUNCH_ARGS）radix/AR 均为禁用；场景一实测变体共四处 delta：
|
||||
# ① 去 --disable-radix-cache（90% 命中前提）② 去 --disable-custom-all-reduce
|
||||
# （v1 CAR 与方案 A 一致开启）③ 加 --context-length 270336 ④ 加 --reasoning-parser
|
||||
# glm45 --tool-call-parser glm47（质量门 7/7 的前提）
|
||||
# - 判决：场景一 8 点全部低于方案 C、7 点低于 A——DFlash accept 低于 EAGLE（同语料
|
||||
# 2.53 vs 2.84）而每步墙钟相当，劣势全在接受率。投机栈选型维持 EAGLE3，勿用
|
||||
# DFlash2 替换（性能问题非质量问题）
|
||||
#
|
||||
# 实测成绩：飞书《GLM-5.3-NVFP4 双场景压测报告》方案 E 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 质量：变体配置下质量门 7/7（GSM8K×5、中文推理、tool call 全过），DFlash 草稿无质量损失。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp8dflash2
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-tp8-dflash2
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=900
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=8
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=12
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --speculative-draft-window-size 2048 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -0,0 +1,44 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=8 + EAGLE deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 生产标准配置（174.1.60.5 团队自用在役，2026-09-07 定稿）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_605.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - SM120 必需三项：--disable-shared-experts-fusion --moe-runner-backend
|
||||
# flashinfer_cutlass --disable-flashinfer-autotune
|
||||
# - EAGLE 4/1/5 为 cc1-2 均衡点（3/1/4、5/1/6 在 ±10% 噪声内；树式 topk>1 在 DSA 上不可用；
|
||||
# EAGLE3 本 nightly 不支持 GLM-5.3）；draft 模型自动从主权重加载
|
||||
# - KV fp8_e4m3 + hicache-ratio 3 是长上下文 90% 前缀命中场景的容量前提（KV 池 276,864 token）
|
||||
# - chunk 8192：16384 已 A/B 证伪（MoE 工作区 OOM 风险，且 AR 带宽受限无次数红利）
|
||||
# - 容器入口必须 python3 -m sglang.launch_server（镜像 entrypoint 无 shebang）
|
||||
# - 重新部署前必须等显存排空（docker rm -f 异步滞留数分钟，不等会把新 KV 池压小）
|
||||
# - 场景二高并发变体（16k 独立输入 cc8-32）：仅改 --max-running-requests 32 +
|
||||
# --cuda-graph-max-bs-decode 16 --cuda-graph-bs-decode 4 8 12 16（KV 池 276,864 ÷
|
||||
# 16,896/req = 16.4 驻留上限，decode 批自然 ≤16，图覆盖到 bs16 即可，bs24/32 纯耗显存）
|
||||
# - 上线前后过质量门 quality_gate_605.sh（PASS=7/7）
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp8eagle
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=8
|
||||
DP=1
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --ipc=host --shm-size 64g --entrypoint '' -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models:ro"
|
||||
|
||||
BOOTSTRAP="exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static 0.90 --max-running-requests 16 --chunked-prefill-size 8192 --max-prefill-tokens 16384 --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --speculative-algorithm EAGLE --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 --kv-cache-dtype fp8_e4m3 --enable-hierarchical-cache --hicache-ratio 3 --cuda-graph-max-bs-decode 8 --cuda-graph-bs-decode 1 2 3 4 6 8 --cuda-graph-max-bs-prefill 8 --context-length 270336 --reasoning-parser glm45 --tool-call-parser glm47 --host 0.0.0.0 --port ${PORT}"
|
||||
54
deploy/profiles/pro6000/kimi3_pro6000_pd_decode.env
Normal file
54
deploy/profiles/pro6000/kimi3_pro6000_pd_decode.env
Normal file
@ -0,0 +1,54 @@
|
||||
# Kimi-K3 PD 分离 - D 组 (decode) deployment profile (4x RTX 6000D).
|
||||
# Nodes: 174.1.60.5~8 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||
#
|
||||
# 这是 PD (Prefill/Decode) 分离部署的 D 组 (decode) 端 profile。
|
||||
# 配套 P 组 profile: kimi3_pro6000_pd_prefill.env
|
||||
# 配套编排脚本: experiments/pro6000/kimi3_pro6000_pd_rdma/deploy_pd.sh
|
||||
#
|
||||
# 与 P 组相同的传输/网络配置（mooncake RDMA + 计算网），区别仅在:
|
||||
# - 节点是 174.1.60.5~8
|
||||
# - --disaggregation-mode decode --port 30000
|
||||
# - dist-init-addr 174.1.60.5:20000
|
||||
# - 必须 D 组在 P 组之后启动（见 deploy_pd.sh）
|
||||
#
|
||||
# Model-team only. Ops only run benchmark against the served router URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=kimi3_pro6000_pd_decode
|
||||
MODEL_NAME=Kimi-K3
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||
SERVED_MODEL_NAME=kimi-k3
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.60.5
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- 并行度（固定，勿改）----
|
||||
TP=32
|
||||
DP=1
|
||||
EP_SIZE=32
|
||||
|
||||
# ---- Multi-node topology (D 组 = decode 4 节点) ----
|
||||
NNODES=4
|
||||
NODE_HOSTS="174.1.60.5 174.1.60.6 174.1.60.7 174.1.60.8"
|
||||
NODE_SSH_USER=root
|
||||
LOCAL_NODE_RANK=0
|
||||
MASTER_IP=174.1.60.5
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma NCCL_SOCKET_IFNAME=bond1 GLOO_SOCKET_IFNAME=bond1 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1 SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro /tmp/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:ro /data/flashkda_deploy/wheels:/mc_wheels:ro"
|
||||
|
||||
# BOOTSTRAP: 打补丁 → 装 flashkda + mooncake wheel → 按 rank 设 SGLANG_HOST_IP → 启动 decode。
|
||||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && pip install /flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl --no-deps -q && pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && export SGLANG_HOST_IP=\"174.1.60.$((5 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --disaggregation-mode decode --host 0.0.0.0 --port ${PORT}"
|
||||
60
deploy/profiles/pro6000/kimi3_pro6000_pd_prefill.env
Normal file
60
deploy/profiles/pro6000/kimi3_pro6000_pd_prefill.env
Normal file
@ -0,0 +1,60 @@
|
||||
# Kimi-K3 PD 分离 - P 组 (prefill) deployment profile (4x RTX 6000D).
|
||||
# Nodes: 174.1.60.1~4 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||
#
|
||||
# 这是 PD (Prefill/Decode) 分离部署的 P 组 (prefill) 端 profile。
|
||||
# 配套 D 组 profile: kimi3_pro6000_pd_decode.env
|
||||
# 配套编排脚本: experiments/pro6000/kimi3_pro6000_pd_rdma/deploy_pd.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 传输后端 mooncake + MOONCAKE_PROTOCOL=rdma（计算网 mlx5_0~3，4 链路）
|
||||
# - 必须关掉 PYTORCH_CUDA_ALLOC_CONF=expandable_segments
|
||||
# （实测: expandable_segments 分配的 GPU 段 mooncake RDMA 注册失败
|
||||
# Bad address [14]，见飞书文档 PD 章节 & GitHub kvcache-ai/Mooncake#2511）
|
||||
# - --disaggregation-ib-device mlx5_0~3 让 mooncake 走计算网 RDMA
|
||||
# - NCCL/GLOO 走 bond1（计算网），NCCL_IB_HCA=mlx5_0..3
|
||||
# - P 组先启动，D 组后启动（见 deploy_pd.sh）
|
||||
# - mooncake master 需先在 174.1.60.1 运行（MOONCAKE_MASTER=174.1.60.1:50051）
|
||||
# - 容器挂载 /mc_wheels 并在 BOOTSTRAP 里 pip install mooncake 0.3.12.post1
|
||||
# （镜像自带 0.3.11.post1 无 dmabuf 修复，必须升级）
|
||||
#
|
||||
# Model-team only. Ops only run benchmark against the served router URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=kimi3_pro6000_pd_prefill
|
||||
MODEL_NAME=Kimi-K3
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||
SERVED_MODEL_NAME=kimi-k3
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.60.1
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- 并行度（固定，勿改）----
|
||||
TP=32
|
||||
DP=1
|
||||
EP_SIZE=32
|
||||
|
||||
# ---- Multi-node topology (P 组 = prefill 4 节点) ----
|
||||
NNODES=4
|
||||
NODE_HOSTS="174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4"
|
||||
NODE_SSH_USER=root
|
||||
LOCAL_NODE_RANK=0
|
||||
MASTER_IP=174.1.60.1
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma NCCL_SOCKET_IFNAME=bond1 GLOO_SOCKET_IFNAME=bond1 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro /tmp/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:ro /data/flashkda_deploy/wheels:/mc_wheels:ro"
|
||||
|
||||
# BOOTSTRAP: 打补丁 → 装 flashkda + mooncake wheel → 按 rank 设 SGLANG_HOST_IP → 启动 prefill。
|
||||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && pip install /flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl --no-deps -q && pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && export SGLANG_HOST_IP=\"174.1.60.$((1 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --disaggregation-mode prefill --linear-attn-prefill-backend flashkda --host 0.0.0.0 --port ${PORT}"
|
||||
54
deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env
Normal file
54
deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env
Normal file
@ -0,0 +1,54 @@
|
||||
# Kimi-K3 SGLang multi-node TP=32 EP=32 deployment profile (4x RTX 6000D).
|
||||
# Nodes: 174.1.60.5~8 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - MoE 后端必须 marlin（K3 的 MXFP4 缩放因子为 uint8，DeepGEMM 只接受 fp32/UE8M0）
|
||||
# - RoCE: NCCL_IB_HCA=mlx5_0..3（4 张独立卡, 10.100.21-24/24, RoCEv2 GID index 3）
|
||||
# 实测 32-rank 117MB allreduce 2.4ms；勿用 mlx5_bond_0（仅 4.5GB/s）
|
||||
# - 容器必须 --ulimit memlock=-1（否则 ibv_create_cq 报 Cannot allocate memory）
|
||||
# - 不要设 NCCL_ALGO=TREE（CUDA graph 捕获报 "NCCL error: invalid usage"）
|
||||
# - 首次请求含 ~110s Triton JIT 编译，属正常现象，预热一次后回落
|
||||
# - flashkda 可选后端（与 triton prefill 性能等价）见 docs/KIMI_K3_DEPLOY.md 附录
|
||||
#
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=kimi3_pro6000_sglang_tp32ep32
|
||||
MODEL_NAME=Kimi-K3
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||
SERVED_MODEL_NAME=kimi-k3
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.60.5
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- 并行度（固定，勿改）----
|
||||
TP=32
|
||||
DP=1
|
||||
EP_SIZE=32
|
||||
|
||||
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||
NNODES=4
|
||||
NODE_HOSTS="174.1.60.5 174.1.60.6 174.1.60.7 174.1.60.8"
|
||||
NODE_SSH_USER=root
|
||||
LOCAL_NODE_RANK=0
|
||||
MASTER_IP=174.1.60.5
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="NCCL_SOCKET_IFNAME=bond0 GLOO_SOCKET_IFNAME=bond0 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro"
|
||||
|
||||
# BOOTSTRAP 在容器内执行: 打 sm_120 补丁 → 按节点 rank 计算 SGLANG_HOST_IP → 启动 sglang。
|
||||
# SGLANG_HOST_IP 必须为本节点实际 IP（174.1.60.5~8 = 5 + NODE_RANK）。
|
||||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && export SGLANG_HOST_IP=\"174.1.60.$((5 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --host 0.0.0.0 --port ${PORT}"
|
||||
@ -0,0 +1,61 @@
|
||||
# Kimi-K3 SGLang multi-node TP=32 EP=32 deployment profile — 实例 B (DP=2 的第二个实例).
|
||||
# Nodes: 174.1.60.1~4 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||
#
|
||||
# DP=2 方案 B：两个独立实例（实例 A: .5-.8, 实例 B: .1-.4），各自 TP32×EP32，
|
||||
# 前面用 sglang_router（--worker-urls）做负载均衡。实例间零通信。
|
||||
#
|
||||
# 与实例 A（kimi3_pro6000_sglang_tp32ep32）的差异仅在于节点拓扑：
|
||||
# - NODE_HOSTS = .1-.4
|
||||
# - MASTER_IP = 174.1.60.1（从 .1 执行部署，LOCAL_NODE_RANK=0 本地起 rank0）
|
||||
# - SGLANG_HOST_IP = 174.1.60.$(1 + NODE_RANK)
|
||||
# 其余并行度/网络配置与实例 A 一致。
|
||||
#
|
||||
# 关键点（与实例 A 相同，勿随意改）:
|
||||
# - MoE 后端必须 marlin（K3 的 MXFP4 缩放因子为 uint8，DeepGEMM 只接受 fp32/UE8M0）
|
||||
# - RoCE: NCCL_IB_HCA=mlx5_0..3（4 张独立卡, 10.100.21-24/24, RoCEv2 GID index 3）
|
||||
# - 容器必须 --ulimit memlock=-1（否则 ibv_create_cq 报 Cannot allocate memory）
|
||||
# - 不要设 NCCL_ALGO=TREE
|
||||
# - --disable-radix-cache 关闭前缀缓存（bench 测量纯净）
|
||||
#
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the router URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=kimi3_pro6000_sglang_tp32ep32_instB
|
||||
MODEL_NAME=Kimi-K3
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||
SERVED_MODEL_NAME=kimi-k3
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.60.1
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- 并行度（固定，勿改）----
|
||||
TP=32
|
||||
DP=1
|
||||
EP_SIZE=32
|
||||
|
||||
# ---- Multi-node topology (实例 B = .1-.4; rank 0 exposes the HTTP API) ----
|
||||
NNODES=4
|
||||
NODE_HOSTS="174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4"
|
||||
NODE_SSH_USER=root
|
||||
LOCAL_NODE_RANK=0
|
||||
MASTER_IP=174.1.60.1
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="NCCL_SOCKET_IFNAME=bond0 GLOO_SOCKET_IFNAME=bond0 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro"
|
||||
|
||||
# BOOTSTRAP 在容器内执行: 打 sm_120 补丁 → 按节点 rank 计算 SGLANG_HOST_IP → 启动 sglang。
|
||||
# SGLANG_HOST_IP 必须为本节点实际 IP（174.1.60.1~4 = 1 + NODE_RANK）。
|
||||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && export SGLANG_HOST_IP=\"174.1.60.$((1 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --host 0.0.0.0 --port ${PORT}"
|
||||
26
deploy/profiles/pro6000/qwen3_235b_pro6000_sglang_tp8.env
Normal file
26
deploy/profiles/pro6000/qwen3_235b_pro6000_sglang_tp8.env
Normal file
@ -0,0 +1,26 @@
|
||||
# Qwen3-235B-A22B SGLang TP=8 profile on a single RTX 6000D node (8 GPUs).
|
||||
# NVIDIA counterpart of p800/qwen3_235b_p800_sglang_tp8 (same workload for
|
||||
# comparability: isl=2048 osl=2048 c=16).
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=qwen3_235b_pro6000_sglang_tp8
|
||||
MODEL_NAME=Qwen3-235B-A22B
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=sglang-sm120-dsv4:0.5.15.post1-fi0.6.14-sm120fix1
|
||||
CONTAINER_NAME=qwen3_235b_pro6000_sglang_tp8
|
||||
MODEL_PATH=/data/6000D/Qwen3-235B-A22B
|
||||
SERVED_MODEL_NAME=/data/6000D/Qwen3-235B-A22B
|
||||
PORT=30010
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1800
|
||||
CONTAINER_PYTHON=python3
|
||||
TP=8
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --privileged --ipc=host --network host --ulimit memlock=-1 --ulimit stack=67108864"
|
||||
VOLUMES="/data:/data"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --trust-remote-code --tp-size ${TP} --mem-fraction-static 0.9 --context-length 8192 --max-running-requests 16 --host 0.0.0.0 --port ${PORT}"
|
||||
91
deploy/verify_profile.sh
Normal file
91
deploy/verify_profile.sh
Normal file
@ -0,0 +1,91 @@
|
||||
#!/bin/bash
|
||||
# verify_profile.sh —— 防漂移核验：运行中容器 vs 仓库 profile 声明
|
||||
# 在目标服务器上运行（需 docker 读权限，无需 GPU）。用法：
|
||||
# bash verify_profile.sh deploy/profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp2pp4.env
|
||||
# bash verify_profile.sh <profile.env> [container_name] # container_name 缺省取 profile 的 CONTAINER_NAME
|
||||
# 核验项：①容器在跑 ②镜像 registry digest ③launch 参数（token 级比对）④端口监听。
|
||||
# PD 链角色的参数在挂载的 launch 脚本内（docker Args 只有 bash /smoke_launch.sh），
|
||||
# 第③项自动跳过并提示改用库内脚本 md5 对比（scripts/ 目录各脚本头注释有 md5）。
|
||||
set -uo pipefail
|
||||
|
||||
PROFILE="$1"
|
||||
[ -r "$PROFILE" ] || { echo "FATAL: 无法读取 profile: $PROFILE"; exit 2; }
|
||||
# profile 是声明式清单：BOOTSTRAP 行引用 ${LAUNCH_ARGS}（定义在后），set -u 下直接
|
||||
# source 会炸；临时关 -u。source 完成后 LAUNCH_ARGS 已是全部变量展开后的实参串。
|
||||
set +u
|
||||
# shellcheck disable=SC1090
|
||||
source "$PROFILE"
|
||||
set -u
|
||||
CONT="${2:-${CONTAINER_NAME:-}}"
|
||||
|
||||
fail=0
|
||||
section() { printf '\n== %s ==\n' "$1"; }
|
||||
|
||||
section "容器状态"
|
||||
if [ -z "$CONT" ]; then echo "FATAL: profile 未定义 CONTAINER_NAME 且未显式传入"; exit 2; fi
|
||||
if docker ps --format '{{.Names}}' | grep -qx "$CONT"; then
|
||||
echo "OK $CONT 在跑"
|
||||
else
|
||||
echo "FAIL $CONT 未在运行"; docker ps -a --filter "name=$CONT" --format '{{.Names}} {{.Status}}' | head -3
|
||||
exit 1
|
||||
fi
|
||||
|
||||
section "镜像 digest"
|
||||
if [ -n "${DOCKER_IMAGE_DIGEST:-}" ]; then
|
||||
# RepoDigests 在 image 对象上：container 先取 image ID，再 image inspect
|
||||
img_id=$(docker inspect --format '{{.Image}}' "$CONT" 2>/dev/null)
|
||||
repo_digests=$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$img_id" 2>/dev/null)
|
||||
digest_hex="${DOCKER_IMAGE_DIGEST#sha256:}"
|
||||
if echo "$repo_digests" | grep -q "$digest_hex"; then
|
||||
echo "OK registry digest 一致: ${DOCKER_IMAGE_DIGEST}"
|
||||
elif [ "$img_id" = "sha256:${digest_hex}" ]; then
|
||||
echo "OK image ID 一致: ${DOCKER_IMAGE_DIGEST}"
|
||||
else
|
||||
echo "FAIL digest 不一致"
|
||||
echo " profile: ${DOCKER_IMAGE_DIGEST}"
|
||||
echo " 实际 image ID: ${img_id}"
|
||||
echo " 实际 RepoDigests: $(echo "$repo_digests" | head -2 | tr '\n' ' ')"
|
||||
fail=1
|
||||
fi
|
||||
else
|
||||
echo "SKIP profile 未定义 DOCKER_IMAGE_DIGEST"
|
||||
fi
|
||||
|
||||
section "启动参数"
|
||||
args=$(docker inspect --format '{{join .Args " "}}' "$CONT" 2>/dev/null)
|
||||
if echo "$args" | grep -q launch_server; then
|
||||
actual=$(echo "$args" | sed 's/.*launch_server //')
|
||||
expected="${LAUNCH_ARGS:-}"
|
||||
tr ' ' '\n' <<<"$expected" | sed '/^$/d' | sort > /tmp/vp_expected.$$
|
||||
tr ' ' '\n' <<<"$actual" | sed '/^$/d' | sort > /tmp/vp_actual.$$
|
||||
if diff -q /tmp/vp_expected.$$ /tmp/vp_actual.$$ >/dev/null; then
|
||||
echo "OK 参数一致（token 比对，共 $(wc -l < /tmp/vp_expected.$$) 项）"
|
||||
else
|
||||
echo "FAIL 参数有漂移（< profile 声明 / > 容器实际）:"
|
||||
diff /tmp/vp_expected.$$ /tmp/vp_actual.$$ | sed 's/^/ /'
|
||||
fail=1
|
||||
fi
|
||||
rm -f /tmp/vp_expected.$$ /tmp/vp_actual.$$
|
||||
else
|
||||
echo "SKIP docker Args 为 '$args' —— 参数在挂载的 launch 脚本内"
|
||||
echo " 改用库内脚本 md5 对比：experiments/.../scripts/ 各脚本头注释"
|
||||
fi
|
||||
|
||||
section "端口 ${PORT:-?}"
|
||||
if [ -n "${PORT:-}" ]; then
|
||||
if ss -tln | grep -q ":${PORT} "; then
|
||||
echo "OK :${PORT} 在监听"
|
||||
else
|
||||
echo "FAIL :${PORT} 未监听"; fail=1
|
||||
fi
|
||||
if [ -n "${HEALTH_PATH:-}" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:${PORT}${HEALTH_PATH}" 2>/dev/null)
|
||||
[ "$code" = "200" ] && echo "OK health ${HEALTH_PATH} -> 200" || { echo "FAIL health ${HEALTH_PATH} -> ${code}"; fail=1; }
|
||||
fi
|
||||
else
|
||||
echo "SKIP profile 未定义 PORT"
|
||||
fi
|
||||
|
||||
echo
|
||||
[ $fail -eq 0 ] && echo "VERDICT: PASS" || echo "VERDICT: DRIFT DETECTED"
|
||||
exit $fail
|
||||
270
docs/KIMI_K3_DEPLOY.md
Normal file
270
docs/KIMI_K3_DEPLOY.md
Normal file
@ -0,0 +1,270 @@
|
||||
# Kimi-K3 部署手册（运维版）
|
||||
|
||||
> 面向运维的一键部署手册：按顺序复制粘贴命令即可完成 Kimi-K3 在 4 节点
|
||||
> RTX 6000D 集群上的部署。**不需要理解 SGLang 引擎参数。**
|
||||
> 部署配置统一在 `deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env`，
|
||||
> 压测统一走 `python -m sskj.bench`（见 `ops/README.md`）。
|
||||
|
||||
## 0. 概述
|
||||
|
||||
| 项 | 值 |
|
||||
|---|---|
|
||||
| 模型 | Kimi-K3（Moonshot AI，2.8T 参数 MXFP4 量化，~1.5TB） |
|
||||
| 节点 | 6000D-5~8 = 174.1.60.5~8（每节点 8× RTX 6000D 85GB，共 32 卡） |
|
||||
| 镜像 | `lmsysorg/sglang:kimi-k3`（专用镜像，普通 sglang 镜像不识别该模型架构） |
|
||||
| 并行 | TP=32 × EP=32（跨 4 节点） |
|
||||
| 服务地址 | `http://174.1.60.5:30000`（OpenAI 兼容） |
|
||||
| 模型 ID | `kimi-k3` |
|
||||
| 部署方式 | `python -m sskj.deploy start --profile pro6000/kimi3_pro6000_sglang_tp32ep32` |
|
||||
|
||||
## 1. 前置检查（每步都过再继续）
|
||||
|
||||
```bash
|
||||
# 1.1 4 节点 8 卡全部空闲（应全部显示 0 MiB / 0%）
|
||||
for i in 5 6 7 8; do echo "== 174.1.60.$i"; ssh 174.1.60.$i 'nvidia-smi --query-gpu=index,memory.used --format=csv,noheader'; done
|
||||
# 注意：如显示被占用，联系模型团队确认（其他用户的作业需先让出）
|
||||
|
||||
# 1.2 磁盘空间（模型 1.5TB + 运行余量，/data 需 ≥ 2TB 可用）
|
||||
ssh 174.1.60.5 'df -h /data'
|
||||
|
||||
# 1.3 RoCE 网卡就绪（应输出 4 个 ACTIVE 且 link_layer=Ethernet）
|
||||
ssh 174.1.60.5 'cat /sys/class/infiniband/mlx5_*/ports/1/link_layer; cat /sys/class/infiniband/mlx5_*/ports/1/state'
|
||||
|
||||
# 1.4 节点互信（174.1.60.5 免密登录其余节点；若失败执行步骤 1.5）
|
||||
ssh 174.1.60.5 'for i in 6 7 8; do ssh -o StrictHostKeyChecking=no 174.1.60.$i hostname; done'
|
||||
|
||||
# 1.5 若 1.4 失败，在 174.1.60.5 上执行（把 .5 的 root 公钥发到各节点）
|
||||
ssh 174.1.60.5 'for i in 6 7 8; do ssh-copy-id -o StrictHostKeyChecking=no root@174.1.60.$i; done'
|
||||
```
|
||||
|
||||
## 2. 模型下载（仅首次，~1.5TB，需数小时）
|
||||
|
||||
模型权重放 `/data/hf_models/Kimi-K3`（4 节点同一路径，本集群已就绪则跳过本节）。
|
||||
|
||||
```bash
|
||||
# 在 174.1.60.5 上执行（国内走 ModelScope 最快；modelscope CLI 需先 pip install modelscope）
|
||||
pip3 install -q modelscope
|
||||
mkdir -p /data/hf_models
|
||||
nohup modelscope download --model moonshotai/Kimi-K3 \
|
||||
--local_dir /data/hf_models/Kimi-K3 > /data/hf_models/download_k3.log 2>&1 &
|
||||
# 查看进度
|
||||
tail -f /data/hf_models/download_k3.log
|
||||
```
|
||||
|
||||
下载完成后校验（96 个分片，共约 1.5TB）：
|
||||
|
||||
```bash
|
||||
ssh 174.1.60.5 'ls /data/hf_models/Kimi-K3/*.safetensors | wc -l; du -sh /data/hf_models/Kimi-K3'
|
||||
# 期望：96（safetensors 数量）、1.5T（总大小）
|
||||
```
|
||||
|
||||
**分发到其余 3 台**（.5 上执行；源盘 NVMe 读是瓶颈，聚合约 3.5GB/s）：
|
||||
|
||||
```bash
|
||||
ssh 174.1.60.5 'for i in 6 7 8; do rsync -aH --partial /data/hf_models/Kimi-K3/ 174.1.60.$i:/data/hf_models/Kimi-K3/ & done; wait'
|
||||
```
|
||||
|
||||
> 每节点需约 1.5T 空闲磁盘（`df -h /data` 确认）。
|
||||
|
||||
## 3. 镜像准备（已拉取则跳过）
|
||||
|
||||
```bash
|
||||
# 4 节点并行拉取（约 9.6GB；如 Docker Hub 直连失败，各机 daemon.json 需配镜像加速：
|
||||
# "registry-mirrors": ["https://docker.m.daocloud.io", "https://docker.xuanyuan.me"]）
|
||||
for i in 5 6 7 8; do ssh 174.1.60.$i 'docker pull lmsysorg/sglang:kimi-k3' & done; wait
|
||||
|
||||
# 验证镜像存在
|
||||
ssh 174.1.60.5 'docker images lmsysorg/sglang:kimi-k3 --format "{{.Repository}}:{{.Tag}} {{.Size}}"'
|
||||
```
|
||||
|
||||
## 4. 补丁与依赖分发
|
||||
|
||||
Kimi-K3 镜像在消费级 Blackwell（sm_120，RTX 6000D）上有一个内核不兼容点，
|
||||
需要打补丁（补丁源文件在本仓库 `platforms/patches/pro6000/kimi_k3/patch_k3_sm120.py`，
|
||||
修复 attn_res 融合内核误用数据中心 Blackwell 专属的 tcgen05 指令问题）。
|
||||
|
||||
```bash
|
||||
# 在 174.1.60.5 上，把补丁分发到 4 节点 /tmp（部署时容器自动挂载并执行）
|
||||
ssh 174.1.60.5 '
|
||||
for i in 5 6 7 8; do
|
||||
scp /data/yy/sskj/platforms/patches/pro6000/kimi_k3/patch_k3_sm120.py 174.1.60.$i:/tmp/patch_k3_sm120.py
|
||||
done
|
||||
for i in 5 6 7 8; do ssh 174.1.60.$i "sha256sum /tmp/patch_k3_sm120.py"; done
|
||||
'
|
||||
# 4 台输出的 sha256 必须一致（内容校验）
|
||||
```
|
||||
|
||||
## 5. 一键部署
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj
|
||||
|
||||
# 5.1 先预览要执行的命令（不实际启动）
|
||||
PYTHONPATH=src python3 -m sskj.deploy start \
|
||||
--profile pro6000/kimi3_pro6000_sglang_tp32ep32 --dry-run
|
||||
|
||||
# 5.2 正式启动（4 节点同时拉起，耗时 8~12 分钟：模型加载 + CUDA graph 捕获）
|
||||
PYTHONPATH=src python3 -m sskj.deploy start \
|
||||
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||
|
||||
# 5.3 查看状态
|
||||
PYTHONPATH=src python3 -m sskj.deploy status \
|
||||
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||
```
|
||||
|
||||
就绪判定（满足其一）：
|
||||
- 步骤 5.3 `status` 显示容器 Up 且健康检查通过
|
||||
- 或手动验证：`curl -s http://174.1.60.5:30000/health` 返回 `{"status":"ok"}`
|
||||
|
||||
## 6. 验证服务
|
||||
|
||||
```bash
|
||||
# 6.1 模型列表
|
||||
curl -s http://174.1.60.5:30000/v1/models | head -c 300
|
||||
# 应包含 "id": "kimi-k3"
|
||||
|
||||
# 6.2 推理冒烟（正确性：23×47 应算得 1081，输出可能带思考标签属正常）
|
||||
curl -s -m 300 http://174.1.60.5:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "kimi-k3", "messages": [{"role": "user", "content": "What is 23*47? Answer briefly."}], "max_tokens": 128, "temperature": 0.1}'
|
||||
|
||||
# 6.3 服务日志（确认无 ERROR）
|
||||
ssh 174.1.60.5 'docker logs kimi3_pro6000_sglang_tp32ep32_node0 2>&1 | tail -20'
|
||||
```
|
||||
|
||||
> ⚠️ **首次请求很慢（TTFT 可达 100+ 秒）是正常现象**：Triton/FlashKDA 内核首次
|
||||
> JIT 编译。发一个任意请求预热后，后续请求恢复正常（16K 输入 TTFT ~6s）。
|
||||
|
||||
## 7. 停止 / 重启
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj
|
||||
|
||||
# 停止（4 节点容器全部移除）
|
||||
PYTHONPATH=src python3 -m sskj.deploy stop \
|
||||
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||
|
||||
# 重启 = 再次执行 5.2 的 start 命令（幂等，会先清旧容器）
|
||||
|
||||
# 手动兜底（start/stop 异常时）：4 节点各执行
|
||||
for i in 5 6 7 8; do ssh 174.1.60.$i 'docker rm -f kimi3_pro6000_sglang_tp32ep32_node*'; done
|
||||
```
|
||||
|
||||
## 8. 性能自检（可选）
|
||||
|
||||
```bash
|
||||
# 稳态基准：16K 输入 TTFT / prefill 吞吐 / TPOT（先跑一次丢弃 JIT 冷启动）
|
||||
cd /data/yy/sskj
|
||||
scp -o ConnectTimeout=10 /data/flashkda_deploy/bench_16k_steady.py 174.1.60.5:/tmp/ 2>/dev/null || true
|
||||
ssh 174.1.60.5 'python3 /tmp/bench_16k_steady.py 3'
|
||||
# 参考值（稳态）：TTFT ~6s / prefill ~2700 tok/s / TPOT ~39ms / decode ~26 tok/s
|
||||
```
|
||||
|
||||
正式压测（矩阵 + 自适应并发搜索）走 bench 层：
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||
DRY_RUN=1 bash run_adaptive_concurrency_add16.sh # 先看计划
|
||||
bash run_adaptive_concurrency_add16.sh # 正式跑（放 tmux）
|
||||
```
|
||||
|
||||
## 9. 故障排查速查
|
||||
|
||||
| 现象 | 原因 | 处理 |
|
||||
|---|---|---|
|
||||
| 首次请求 TTFT 100+ 秒 | Triton/FlashKDA 内核首次 JIT 编译 | 正常现象，预热一次即可 |
|
||||
| `Not enough GPU memory for hybrid mamba state cache` | 某节点显存被其他作业占用 | 检查 4 节点 `nvidia-smi`，确认 8 卡全空闲后重启部署 |
|
||||
| `ibv_create_cq failed: Cannot allocate memory` | 容器 memlock 限制 | 部署配置已带 `--ulimit memlock=-1`，检查 profile 未被改动 |
|
||||
| `NCCL error: invalid usage`（graph 捕获时） | 误设了 `NCCL_ALGO=TREE` | 确认环境变量里没有 NCCL_ALGO |
|
||||
| 启动后 `/health` 一直不通 | 分布式初始化失败/节点未就绪 | 看 master 日志；确认 4 节点 GPU 全空闲后 `stop` + 重新 `start` |
|
||||
| 推理结果带 `<\|open\|>think` 标签 | K3 思考型模型正常输出格式 | 非故障；如需精简可调 chat template（联系模型团队） |
|
||||
| 并发压测时部分请求失败 | flashkda 后端已知问题（未修复） | 当前默认 triton 后端无此问题；如误用 flashkda 见附录 |
|
||||
|
||||
## 附录 A：flashkda 可选后端（默认不启用）
|
||||
|
||||
FlashKDA（MoonshotAI CUTLASS KDA 内核，支持 sm_120）可作为 KDA prefill 的备选后端。
|
||||
实测与默认 triton prefill **性能等价**（16K prefill 2693 vs 2695 tok/s），仅作备份用途。
|
||||
|
||||
启用方式（需模型团队协助）：
|
||||
1. 构建 wheel（脚本见 `/data/flashkda_deploy/01_build_flashkda.sh`，产物
|
||||
`flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl` 分发到 4 节点 `/tmp/`）
|
||||
2. profile 增加 wheel 挂载与安装步骤，并在 LAUNCH_ARGS 加
|
||||
`--linear-attn-prefill-backend flashkda`
|
||||
|
||||
已知问题：flashkda 后端在 4/8 并发压测下服务异常（纯 triton 无此问题），修复前不建议生产使用。
|
||||
|
||||
## 附录 B：PD 分离部署（MoonCake RDMA，8 节点）
|
||||
|
||||
非 PD 单组部署之外的另一种形态：**Prefill/Decode 分离**（P 组 4 节点做 prefill，D 组 4 节点做 decode），KV 传输走 **MoonCake RDMA**。适合长上下文、吞吐优先的场景。
|
||||
|
||||
### B.1 拓扑
|
||||
|
||||
```
|
||||
┌───────────────────┐
|
||||
client ──31000──▶ │ router (MiniLB) │ 174.1.60.5
|
||||
└───────┬───────────┘
|
||||
┌───────▼───────────┐
|
||||
P 组 (prefill) │ 174.1.60.1~4 │ TP32×EP32，4 节点 32 卡
|
||||
└── prefill 计算 │ 计算完产出 KV │
|
||||
└───────┬───────────┘
|
||||
┌───────▼───────────┐
|
||||
MoonCake RDMA│ mlx5_0~3 (RoCE) │ 4 链路并行
|
||||
└───────┬───────────┘
|
||||
┌───────▼───────────┐
|
||||
D 组 (decode) │ 174.1.60.5~8 │ TP32×EP32，4 节点 32 卡
|
||||
└── decode 生成 │ 接收 KV 逐 token │
|
||||
└───────────────────┘
|
||||
```
|
||||
|
||||
### B.2 运维准备（一次性，新环境才需执行）
|
||||
|
||||
已部署过的集群可跳过本节。步骤顺序：
|
||||
|
||||
1. **免密与文件就位**：确保 8 节点互信、docker 可用；把 `patch_k3_sm120.py`、`flash_kda-*.whl`、mooncake wheel 分发到 8 节点（路径见附录 B.2 下方说明）。
|
||||
2. **sskj 仓库**：`.5` 和 `.1` 都需 clone `/data/yy/sskj`（deploy_pd.sh 的 deploy 层命令依赖）。
|
||||
3. **拉镜像**：8 节点 `docker pull lmsysorg/sglang:kimi-k3`。
|
||||
4. **GPU 空闲**：`nvidia-smi` 全 0 MiB，否则联系模型团队确认。
|
||||
|
||||
详细命令见 `experiments/pro6000/kimi3_pro6000_pd_rdma/README.md` 的「运维一键部署」章节（含每步的可执行命令与验证）。
|
||||
|
||||
### B.3 一键部署
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj/experiments/pro6000/kimi3_pro6000_pd_rdma
|
||||
bash deploy_pd.sh start # mc-master → P 组 → D 组 → router
|
||||
bash deploy_pd.sh status
|
||||
bash deploy_pd.sh stop
|
||||
bash deploy_pd.sh restart
|
||||
```
|
||||
|
||||
### B.4 验证
|
||||
|
||||
```bash
|
||||
# 端到端（router 自动双发 P/D）
|
||||
curl -s http://174.1.60.5:31000/generate -H "Content-Type: application/json" \
|
||||
-d '{"text":"Hello","sampling_params":{"max_new_tokens":16}}'
|
||||
```
|
||||
|
||||
长输入验证（2026-08-11 实测）：1680 / 5280 tokens 输入均正常，P 组 prefill ~112 tok/s，D 组 decode ~26 tok/s。
|
||||
|
||||
### B.5 关键注意事项（务必遵守）
|
||||
|
||||
1. **禁止设置 `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`**
|
||||
expandable_segments 分配的 GPU 段，mooncake RDMA 注册报 `Bad address [14]`
|
||||
（GitHub kvcache-ai/Mooncake **#2511**）。移除后 KV 注册从 2376 条失败 → 0。
|
||||
2. **P 组必须先于 D 组启动**（prefill 需先注册 bootstrap）。
|
||||
3. **容器内必须升级 mooncake 到 0.3.12.post1**（镜像自带 0.3.11.post1 无 dmabuf 修复 #2035）。
|
||||
wheel 在 `/data/flashkda_deploy/wheels/`，profile 的 BOOTSTRAP 已含 pip install。
|
||||
4. NCCL/GLOO 走计算网（`bond1` + `NCCL_IB_HCA=mlx5_0..3`），不要走管理网 bond0。
|
||||
5. mc-master（mooncake 元数据服务）需先在 174.1.60.1 运行（deploy_pd.sh 自动处理）。
|
||||
|
||||
### B.6 与单组部署的取舍
|
||||
|
||||
| 维度 | 单组 TP32×EP32 | PD 分离（MoonCake RDMA） |
|
||||
|---|---|---|
|
||||
| 节点 | 174.1.60.5~8（4 节点） | P 174.1.60.1~4 + D 174.1.60.5~8（8 节点） |
|
||||
| KV 传输 | 无（本地） | MoonCake RDMA（计算网 4 链路） |
|
||||
| 传输后端 | - | mooncake（必须 0.3.12.post1） |
|
||||
| 适用 | 单组吞吐、简单 | 长上下文、PD 分离 |
|
||||
| profile | kimi3_pro6000_sglang_tp32ep32 | kimi3_pro6000_pd_prefill / _pd_decode |
|
||||
| 编排 | `python -m sskj.deploy` | `deploy_pd.sh` |
|
||||
@ -1,816 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>6000D 双机通信、NCCL 与 Profiling 术语入门</title>
|
||||
<style>
|
||||
:root {
|
||||
color-scheme: light;
|
||||
--ink: #17202a;
|
||||
--muted: #5f6b76;
|
||||
--line: #d9dee3;
|
||||
--panel: #f5f7f8;
|
||||
--cyan: #087b83;
|
||||
--cyan-soft: #e5f4f3;
|
||||
--orange: #a64b17;
|
||||
--orange-soft: #fff0e5;
|
||||
--green: #287a45;
|
||||
--green-soft: #eaf6ee;
|
||||
--red: #a73535;
|
||||
--red-soft: #fdecec;
|
||||
--code: #f0f2f4;
|
||||
--max: 1160px;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
|
||||
html { scroll-behavior: smooth; }
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
color: var(--ink);
|
||||
background: #fff;
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC",
|
||||
"Hiragino Sans GB", "Microsoft YaHei", sans-serif;
|
||||
font-size: 16px;
|
||||
line-height: 1.72;
|
||||
}
|
||||
|
||||
header {
|
||||
color: #fff;
|
||||
background: #172f35;
|
||||
border-bottom: 5px solid #e57932;
|
||||
}
|
||||
|
||||
.header-inner {
|
||||
max-width: var(--max);
|
||||
margin: 0 auto;
|
||||
padding: 46px 28px 40px;
|
||||
}
|
||||
|
||||
.eyebrow {
|
||||
margin: 0 0 8px;
|
||||
color: #9ed8d5;
|
||||
font-size: 13px;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
h1, h2, h3 { letter-spacing: 0; }
|
||||
|
||||
header h1 {
|
||||
max-width: 850px;
|
||||
margin: 0;
|
||||
font-size: clamp(32px, 5vw, 54px);
|
||||
line-height: 1.12;
|
||||
}
|
||||
|
||||
.header-meta {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 10px 22px;
|
||||
margin-top: 22px;
|
||||
color: #d4e4e5;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
.layout {
|
||||
display: grid;
|
||||
grid-template-columns: 240px minmax(0, 1fr);
|
||||
gap: 38px;
|
||||
max-width: var(--max);
|
||||
margin: 0 auto;
|
||||
padding: 34px 28px 70px;
|
||||
}
|
||||
|
||||
nav {
|
||||
position: sticky;
|
||||
top: 20px;
|
||||
align-self: start;
|
||||
padding-right: 20px;
|
||||
border-right: 1px solid var(--line);
|
||||
}
|
||||
|
||||
nav strong {
|
||||
display: block;
|
||||
margin-bottom: 10px;
|
||||
font-size: 13px;
|
||||
color: var(--muted);
|
||||
}
|
||||
|
||||
nav a {
|
||||
display: block;
|
||||
padding: 6px 0;
|
||||
color: #40515a;
|
||||
font-size: 14px;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
nav a:hover { color: var(--cyan); }
|
||||
|
||||
main { min-width: 0; }
|
||||
|
||||
h2 {
|
||||
margin: 48px 0 16px;
|
||||
padding-top: 8px;
|
||||
font-size: 28px;
|
||||
line-height: 1.25;
|
||||
border-top: 2px solid var(--ink);
|
||||
}
|
||||
|
||||
h2:first-child { margin-top: 0; }
|
||||
|
||||
h3 {
|
||||
margin: 30px 0 10px;
|
||||
font-size: 20px;
|
||||
line-height: 1.35;
|
||||
}
|
||||
|
||||
p { margin: 10px 0; }
|
||||
|
||||
a { color: var(--cyan); }
|
||||
|
||||
code {
|
||||
padding: 2px 5px;
|
||||
border-radius: 3px;
|
||||
background: var(--code);
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
font-size: .92em;
|
||||
}
|
||||
|
||||
pre {
|
||||
overflow-x: auto;
|
||||
margin: 14px 0;
|
||||
padding: 16px 18px;
|
||||
color: #e8f1f1;
|
||||
background: #1c292d;
|
||||
border-left: 4px solid #4cb3ae;
|
||||
border-radius: 4px;
|
||||
line-height: 1.55;
|
||||
}
|
||||
|
||||
pre code {
|
||||
padding: 0;
|
||||
color: inherit;
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
margin: 16px 0 24px;
|
||||
border-collapse: collapse;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
th, td {
|
||||
padding: 11px 12px;
|
||||
text-align: left;
|
||||
vertical-align: top;
|
||||
border: 1px solid var(--line);
|
||||
}
|
||||
|
||||
th {
|
||||
color: #26363e;
|
||||
background: #eef1f3;
|
||||
}
|
||||
|
||||
ul, ol { padding-left: 24px; }
|
||||
|
||||
.note, .warning, .finding, .good {
|
||||
margin: 18px 0;
|
||||
padding: 15px 18px;
|
||||
border-left: 4px solid;
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.note { background: var(--cyan-soft); border-color: var(--cyan); }
|
||||
.warning { background: var(--orange-soft); border-color: var(--orange); }
|
||||
.finding { background: var(--red-soft); border-color: var(--red); }
|
||||
.good { background: var(--green-soft); border-color: var(--green); }
|
||||
|
||||
.path {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(5, minmax(0, 1fr));
|
||||
gap: 8px;
|
||||
margin: 18px 0 24px;
|
||||
}
|
||||
|
||||
.path div {
|
||||
min-height: 108px;
|
||||
padding: 12px;
|
||||
border: 1px solid var(--line);
|
||||
border-top: 4px solid var(--cyan);
|
||||
background: var(--panel);
|
||||
}
|
||||
|
||||
.path b { display: block; margin-bottom: 5px; }
|
||||
|
||||
.split {
|
||||
display: grid;
|
||||
grid-template-columns: 1fr 1fr;
|
||||
gap: 18px;
|
||||
margin: 18px 0;
|
||||
}
|
||||
|
||||
.split section {
|
||||
padding: 16px 18px;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
.split h3 { margin-top: 0; }
|
||||
|
||||
.metric {
|
||||
font-variant-numeric: tabular-nums;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
footer {
|
||||
padding: 26px 28px 40px;
|
||||
color: var(--muted);
|
||||
text-align: center;
|
||||
border-top: 1px solid var(--line);
|
||||
}
|
||||
|
||||
@media (max-width: 900px) {
|
||||
.layout { grid-template-columns: 1fr; }
|
||||
nav {
|
||||
position: static;
|
||||
padding: 0 0 18px;
|
||||
border-right: 0;
|
||||
border-bottom: 1px solid var(--line);
|
||||
}
|
||||
nav a { display: inline-block; margin-right: 14px; }
|
||||
.path { grid-template-columns: 1fr; }
|
||||
.split { grid-template-columns: 1fr; }
|
||||
}
|
||||
|
||||
@media print {
|
||||
nav { display: none; }
|
||||
.layout { display: block; max-width: none; }
|
||||
header { color: #000; background: #fff; border-bottom-color: #000; }
|
||||
.header-meta, .eyebrow { color: #333; }
|
||||
pre { color: #000; background: #f4f4f4; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">Two-node communication primer</p>
|
||||
<h1>6000D 双机通信、NCCL 与 Profiling 术语入门</h1>
|
||||
<div class="header-meta">
|
||||
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||
<span>规模：16 GPU / TP16</span>
|
||||
<span>版本：2026-07-31 15:25 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="layout">
|
||||
<nav aria-label="目录">
|
||||
<strong>阅读顺序</strong>
|
||||
<a href="#mental-model">1. 一张总图</a>
|
||||
<a href="#names">2. 设备名字</a>
|
||||
<a href="#glossary">3. 核心术语</a>
|
||||
<a href="#bootstrap">4. Bootstrap</a>
|
||||
<a href="#parameters">5. NCCL 参数</a>
|
||||
<a href="#logs">6. 日志怎么读</a>
|
||||
<a href="#incident">7. 本次事故</a>
|
||||
<a href="#checklist">8. 排查清单</a>
|
||||
<a href="#sources">9. 官方资料</a>
|
||||
</nav>
|
||||
|
||||
<main>
|
||||
<h2 id="mental-model">1. 先建立一张总图</h2>
|
||||
<p>
|
||||
SGLang 不会自己搬运 16 张 GPU 之间的 Tensor。模型代码发起 TP/MoE 通信，
|
||||
NCCL 决定用什么算法、经过哪条链路把数据送到其他 rank。
|
||||
</p>
|
||||
|
||||
<div class="path" aria-label="通信路径">
|
||||
<div><b>SGLang</b>执行模型层、TP16 和 EP2</div>
|
||||
<div><b>Collective</b>AllReduce、AllGather、ReduceScatter、AllToAll</div>
|
||||
<div><b>NCCL</b>构造 rank、ring/tree 和 channel</div>
|
||||
<div><b>Transport</b>机内 P2P/IPC；跨机 NET/IB 或 NET/Socket</div>
|
||||
<div><b>硬件</b>GPU、PCIe、HCA、网卡、光模块、交换机</div>
|
||||
</div>
|
||||
|
||||
<div class="note">
|
||||
<strong>最重要的区分：</strong>
|
||||
<code>NCCL bootstrap</code> 是“启动时要完成的一件事”；
|
||||
<code>NCCL_SOCKET_IFNAME</code> 是“选择 IP 网卡的一个参数”；
|
||||
<code>NET/IB</code> 和 <code>NET/Socket</code> 才是 NCCL 实际搬运数据的传输后端。
|
||||
</div>
|
||||
|
||||
<h3>两类跨机路径</h3>
|
||||
<div class="split">
|
||||
<section>
|
||||
<h3>理想路径：RDMA</h3>
|
||||
<p><code>GPU → HCA → RoCE 网络 → HCA → GPU</code></p>
|
||||
<p>日志应出现 <code>NET/IB</code>，支持时还会出现 <code>GDRDMA</code>。</p>
|
||||
</section>
|
||||
<section>
|
||||
<h3>回退路径：TCP Socket</h3>
|
||||
<p><code>GPU/CPU → Linux Socket → ethX → TCP/IP → ethX</code></p>
|
||||
<p>日志会出现 <code>Using network Socket</code>。这不是报错，但性能通常低得多。</p>
|
||||
</section>
|
||||
</div>
|
||||
|
||||
<h2 id="names">2. eth0 和 mlx5_0 不是同一个设备</h2>
|
||||
<div class="note">
|
||||
<strong>400G 是物理 Ethernet 端口的标称链路速率。</strong>
|
||||
<code>eth0</code> 是该端口的 Linux netdev/IP 入口；
|
||||
<code>mlx5_0</code> 是映射到该端口的 RDMA Verbs/HCA 入口。
|
||||
二者相关联，但不相等，也不代表 TCP 或 RDMA 应用一定能跑到 400G。
|
||||
</div>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>名字</th><th>属于哪一层</th><th>负责什么</th><th>本机实例</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>eth0</code></td>
|
||||
<td>Linux IP 网卡接口</td>
|
||||
<td>配置 IP、TCP/UDP、路由；由 <code>NCCL_SOCKET_IFNAME</code> 选择</td>
|
||||
<td>400 Gbit/s 计算网</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>eth3</code></td>
|
||||
<td>Linux IP 网卡接口</td>
|
||||
<td>第二条计算网 Rail</td>
|
||||
<td>400 Gbit/s 计算网</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>mlx5_0</code></td>
|
||||
<td>RDMA HCA / Verbs 设备</td>
|
||||
<td>供 <code>NET/IB</code> 使用；由 <code>NCCL_IB_HCA</code> 选择</td>
|
||||
<td>对应 <code>eth0</code>，挂 switch 1</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>mlx5_3</code></td>
|
||||
<td>RDMA HCA / Verbs 设备</td>
|
||||
<td>第二条 RDMA Rail</td>
|
||||
<td>对应 <code>eth3</code>，挂 switch 2</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>/dev/infiniband/uverbs0</code></td>
|
||||
<td>Linux 字符设备</td>
|
||||
<td>容器进程访问 RDMA Verbs 的入口</td>
|
||||
<td>对应 <code>mlx5_0</code></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<pre><code>mlx5_0 port 1 ==> eth0 (Up)
|
||||
mlx5_3 port 1 ==> eth3 (Up)</code></pre>
|
||||
|
||||
<p>
|
||||
同一条物理端口可以同时暴露 Linux IP 接口和 RDMA HCA。
|
||||
<code>eth0</code> 是 IP/Socket 世界的入口，<code>mlx5_0</code> 是 RDMA
|
||||
Verbs 世界的入口。<code>ibdev2netdev</code> 输出的是映射关系，不是等号。
|
||||
本项目部署时只把 <code>eth0/eth3</code> 作为节点间计算网。
|
||||
</p>
|
||||
<pre><code>同一条 400G 物理 Ethernet 端口
|
||||
├── eth0 -> Linux netdev -> IP / TCP Socket
|
||||
└── mlx5_0 -> RDMA HCA -> RoCE / Verbs / GDRDMA</code></pre>
|
||||
<p>
|
||||
<code>400 Gbit/s = 50 GB/s</code> 只是单方向理论线速。协议开销、PCIe、
|
||||
CPU、Socket 线程、消息大小和 collective 算法都会让实际
|
||||
<code>algbw/busbw</code> 低于或采用不同统计口径。
|
||||
</p>
|
||||
|
||||
<h2 id="glossary">3. 核心术语字典</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>术语</th><th>通俗解释</th><th>在本项目中的意义</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>NCCL</strong></td>
|
||||
<td>NVIDIA 的多 GPU 通信库，负责高效实现 collective 和点对点通信。</td>
|
||||
<td>SGLang TP16 每层跨 GPU 通信最终大量落到 NCCL。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>rank</strong></td>
|
||||
<td>一个通信参与者的编号。TP16 communicator 有 rank 0–15。</td>
|
||||
<td>两台机器各 8 个 GPU rank，共 16 个。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>collective</strong></td>
|
||||
<td>一组 rank 共同参与的通信操作。</td>
|
||||
<td>TP 常见 AllReduce、AllGather、ReduceScatter；MoE 还可能有 AllToAll。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>RDMA</strong></td>
|
||||
<td>远端直接内存访问。网卡可直接读写远端内存，减少 CPU 和内核数据拷贝。</td>
|
||||
<td>双机 TP16 希望使用的高速数据路径。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>IB</strong></td>
|
||||
<td>InfiniBand。既是一套高速网络体系，也常被 NCCL 用作 Verbs/RDMA 后端的统称。</td>
|
||||
<td>NCCL 日志里的 <code>NET/IB</code> 也可承载 RoCE，不代表交换机一定是原生 IB。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>RoCE</strong></td>
|
||||
<td>RDMA over Converged Ethernet，在以太网上承载 RDMA。</td>
|
||||
<td>本项目的 400G 计算网类型。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>HCA</strong></td>
|
||||
<td>Host Channel Adapter，提供 RDMA 能力的适配器。</td>
|
||||
<td><code>mlx5_0</code>、<code>mlx5_3</code>。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>GDRDMA</strong></td>
|
||||
<td>GPUDirect RDMA，让 HCA 直接访问 GPU 显存，减少经 CPU 内存中转。</td>
|
||||
<td>跨机 GPU 通信的理想路径，日志可见 <code>via NET/IB/.../GDRDMA</code>。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Socket / TCP</strong></td>
|
||||
<td>普通 IP 网络编程路径。NCCL 找不到 RDMA 时会使用。</td>
|
||||
<td>本次脚本实际发生的回退路径。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Rail</strong></td>
|
||||
<td>一条相对独立的网络通道，通常由一张 HCA 和一套交换路径组成。</td>
|
||||
<td><code>mlx5_0/switch 1</code> 与 <code>mlx5_3/switch 2</code> 是双 Rail。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>ring / tree</strong></td>
|
||||
<td>NCCL 对 collective 的通信拓扑组织方式。</td>
|
||||
<td><code>NCCL_CROSS_NIC</code> 决定同一 ring/tree 能否跨不同 NIC。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>PFC / ECN</strong></td>
|
||||
<td>RoCE 网络控制拥塞和丢包的机制。</td>
|
||||
<td>RDMA 出现 retry、pause 或吞吐抖动时由运维检查。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>NIC</strong></td>
|
||||
<td>Network Interface Card，网卡的统称。它可以暴露普通 IP 接口，也可以提供 RDMA 能力。</td>
|
||||
<td><code>eth0/eth3</code> 是 Linux netdev 名；对应的 RDMA HCA 名是 <code>mlx5_0/mlx5_3</code>。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>NUMA</strong></td>
|
||||
<td>Non-Uniform Memory Access。双路 CPU 机器中，每个 CPU 访问本地内存更快，访问另一侧内存更慢。</td>
|
||||
<td>服务线程、GPU 和 NIC 若跨 NUMA 节点配合，可能增加 Host 侧延迟和 PCIe 路径长度。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>CUDA P2P / IPC</strong></td>
|
||||
<td>P2P 让同机 GPU 直接互访显存；IPC 让不同进程共享可访问的 GPU 内存句柄。</td>
|
||||
<td>6000D 无 NVLink，单机 8 卡的 NCCL P2P/IPC 实际经过 PCIe。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>PIX / SYS</strong></td>
|
||||
<td>NVIDIA 拓扑标签。PIX 表示 GPU 间只跨一个 PCIe Switch；SYS 表示还要跨 CPU/NUMA 互联。</td>
|
||||
<td>GPU0–3、GPU4–7 各自多为 PIX，两组之间为 SYS；P2P 微基准会分别汇总这两类路径。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>AllReduce</strong></td>
|
||||
<td>所有 rank 先归约数据，再让每个 rank 都拿到相同结果的 collective。</td>
|
||||
<td>TP16 高频使用；Phase 2 分别测单机 8 rank 和双机 16 rank。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>algbw / busbw</strong></td>
|
||||
<td><code>algbw</code> 是有效数据量除以操作时间；<code>busbw</code> 再按 collective 的理论链路流量换算，便于比较硬件通信效率。</td>
|
||||
<td>AllReduce 使用 <code>busbw = algbw × 2 × (N-1) / N</code>。两者单位通常为 GB/s，不能与 400 Gbit/s 直接混用。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>DCGM</strong></td>
|
||||
<td>NVIDIA Data Center GPU Manager，一套 GPU 健康、遥测和诊断框架。它比 <code>nvidia-smi</code> 提供更细的 GPU 活跃度计数器。</td>
|
||||
<td>Phase 2 用它采集 SM、Tensor、设备显存接口和 PCIe 活跃度；它不是 Nsight Timeline。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>DCGM Host Engine</strong></td>
|
||||
<td>DCGM 的后台服务，负责连接驱动、维护 GPU 清单并提供指标。systemd 服务通常叫 <code>nvidia-dcgm</code>，底层进程是 <code>nv-hostengine</code>。</td>
|
||||
<td>两节点都必须运行；否则 <code>dcgmi dmon</code> 客户端存在也无法采集。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>dcgmi / Field ID</strong></td>
|
||||
<td><code>dcgmi</code> 是 DCGM 命令行客户端；Field ID 是某个遥测指标的数字编号。</td>
|
||||
<td>Phase 2 使用 1001–1005、1009、1010，并把缺失样本保留为 <code>-</code>，不会当成 0。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>SM</strong></td>
|
||||
<td>Streaming Multiprocessor，GPU 执行 CUDA Warp、Tensor Core 指令和大部分计算的基本处理单元。</td>
|
||||
<td><code>sm_active</code> 高说明 SM 经常在工作，但不等于每个 SM 都满负载。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Warp</strong></td>
|
||||
<td>NVIDIA GPU 同步执行的一组线程，通常包含 32 个 CUDA 线程。</td>
|
||||
<td><code>sm_occupancy</code> 反映活跃 Warp 相对硬件可容纳 Warp 的比例。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>SM Active / Occupancy</strong></td>
|
||||
<td>前者回答“SM 有多少时间在工作”，后者回答“工作时驻留了多少 Warp”。</td>
|
||||
<td>Active 高、Occupancy 低可能来自小 Kernel、资源约束或同步，必须结合后续 Timeline 判断。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>Tensor Active</strong></td>
|
||||
<td>Tensor Core 管线处于活跃状态的时间比例。</td>
|
||||
<td>用于判断矩阵计算单元是否被充分使用；它不是模型总 FLOPS 利用率。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>DRAM Active</strong></td>
|
||||
<td>DCGM 的历史字段名，表示 GPU 设备显存接口活跃比例，不限定显存必须是主机 DRAM 或 HBM。</td>
|
||||
<td>Pro6000D 使用 GDDR7；该指标仍用于观察设备显存带宽压力。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>测量窗口 / Epoch</strong></td>
|
||||
<td>Epoch 是统一的 Unix 时间基准；测量窗口是正式 benchmark 开始到结束的精确时间段。</td>
|
||||
<td>Phase 2 用 <code>Starting main benchmark run</code> 加 benchmark duration 切片，排除数据准备和 Warm-up。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>mpstat</strong></td>
|
||||
<td>查看整机和每个逻辑 CPU 的利用率、I/O Wait 等。</td>
|
||||
<td>回答是否整机 CPU 饱和，或只有少数核心成为热点。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>pidstat</strong></td>
|
||||
<td>按进程统计 CPU、内存、I/O、缺页和上下文切换。</td>
|
||||
<td>Phase 2 使用进程级 5 秒采样，避免旧版线程级 1 秒采样产生数百 MB 日志。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>sar</strong></td>
|
||||
<td>sysstat 套件中的系统活动记录工具，可采集网卡吞吐和错误。</td>
|
||||
<td>Phase 2 只看计算网 <code>eth0/eth3</code>，与 HCA RDMA Counter 分层比较。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>perf stat</strong></td>
|
||||
<td>Linux 性能计数器工具，统计 CPU cycles、instructions、cache miss、迁移和缺页。</td>
|
||||
<td>用于判断 Host 进程是否受 CPU 执行、Cache 或调度开销限制，不提供 GPU Kernel 时间线。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>numastat</strong></td>
|
||||
<td>查看系统或进程在各 NUMA 节点上的内存分布。</td>
|
||||
<td>Phase 2 每 5 秒保存结构化 Node0/Node1 MiB，寻找跨 NUMA 内存放置。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>HCA Counter</strong></td>
|
||||
<td>网卡硬件维护的发送、接收、等待、丢弃和错误累计计数器。</td>
|
||||
<td>Phase 2.5 用 <code>mlx5_0/mlx5_3</code> 的 counter 差值计算正式 benchmark 窗口内的 RDMA Gbit/s。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>bytes/input-token/rail</strong></td>
|
||||
<td>模型每处理一个输入 token，平均要在一条 Rail 上发送的字节数。</td>
|
||||
<td>当前 DSV4-Pro TP16/EP2 Scout 拟合为约 3.332 MB/token/rail；换模型或并行策略必须重新标定。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>带宽平台 / 拐点</strong></td>
|
||||
<td>继续增加并发后，吞吐与网络带宽都几乎不再增长的位置。</td>
|
||||
<td>Phase 2.5 以相邻点的 Input TPS 和 Rail Mean 增益同时低于 5% 判断，当前拐点为 C=16。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>渐近线 / 饱和上限</strong></td>
|
||||
<td>饱和曲线在并发继续增大时逼近、但不会明显超过的预测上限。</td>
|
||||
<td>当前 64K Prefill 的拟合上限约 80.32 Gbit/s/rail，表示模型产流量上限，不表示网卡硬件只能跑 80G。</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2 id="bootstrap">4. NCCL bootstrap 到底是什么</h2>
|
||||
<p>
|
||||
NCCL 本身不是进程启动器。SGLang 先启动各个 worker，NCCL communicator 初始化时，
|
||||
rank 之间需要交换地址、唯一 ID、拓扑和连接信息，这段“先认识彼此”的过程就是 bootstrap。
|
||||
</p>
|
||||
|
||||
<ol>
|
||||
<li>每个 rank 启动并获得自己的 rank ID。</li>
|
||||
<li>通过 IP Socket 交换 NCCL unique ID 和连接信息。</li>
|
||||
<li>NCCL 探测 GPU、PCIe、HCA 和节点拓扑。</li>
|
||||
<li>构造 ring/tree/channel。</li>
|
||||
<li>选择真正的数据传输后端：P2P、SHM、NET/IB 或 NET/Socket。</li>
|
||||
</ol>
|
||||
|
||||
<div class="warning">
|
||||
<strong>容易误解的地方：</strong>
|
||||
<code>NCCL_SOCKET_IFNAME</code> 不保证“只用于 bootstrap”。
|
||||
RDMA 正常时它主要承担 bootstrap；RDMA 失败并回退 Socket 后，它也会决定大块 Tensor
|
||||
数据走哪张 IP 网卡。
|
||||
</div>
|
||||
|
||||
<h2 id="parameters">5. 常见 NCCL 参数</h2>
|
||||
|
||||
<h3>NCCL_SOCKET_IFNAME</h3>
|
||||
<p>筛选 NCCL 可使用的 Linux IP 接口。精确指定接口时可写：</p>
|
||||
<pre><code>NCCL_SOCKET_IFNAME="=eth0"</code></pre>
|
||||
<ul>
|
||||
<li>RDMA 正常：主要影响 bootstrap/OOB IP 连接。</li>
|
||||
<li>RDMA 不可用：决定 <code>NET/Socket</code> 的数据网卡。</li>
|
||||
<li>部署时只允许使用计算网 <code>eth0/eth3</code>；Socket 回退时不能落到其他接口。</li>
|
||||
</ul>
|
||||
|
||||
<h3>NCCL_IB_HCA</h3>
|
||||
<p>筛选 NCCL 的 RDMA HCA。推荐使用精确匹配：</p>
|
||||
<pre><code>NCCL_IB_HCA="=mlx5_0:1,mlx5_3:1"</code></pre>
|
||||
<p>
|
||||
这个变量只是“允许选择谁”，不会自动把宿主机 RDMA 设备送进容器。
|
||||
容器还必须看到 <code>/dev/infiniband/rdma_cm</code>、
|
||||
<code>uverbs0</code> 和 <code>uverbs3</code>。
|
||||
</p>
|
||||
|
||||
<h3>NCCL_CROSS_NIC</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>值</th><th>行为</th><th>适用直觉</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>0</code></td>
|
||||
<td>尽量让同一 ring/tree 在不同节点使用对应的同一条 Rail。</td>
|
||||
<td>每张 NIC 接不同交换机、跨 Rail 代价高的 rail-optimized 网络。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>1</code></td>
|
||||
<td>允许同一 ring/tree 在不同节点使用不同 NIC。</td>
|
||||
<td>所有 NIC 进入同一网络 Fabric，跨 NIC 不构成额外问题。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>2</code></td>
|
||||
<td>优先对应同一 NIC，但必要时允许跨 NIC。</td>
|
||||
<td>NCCL 默认的折中策略。</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="note">
|
||||
本项目的 <code>mlx5_0</code> 和 <code>mlx5_3</code> 分挂 switch 1/2，
|
||||
拓扑直觉上更偏向 <code>0</code> 或默认 <code>2</code>。最终值必须在
|
||||
<strong>真正启用 NET/IB 后</strong>用 all_reduce 和 SGLang 端到端 A/B 决定。
|
||||
当 NCCL 使用 NET/Socket 时，这个参数不参与路径选择。
|
||||
</div>
|
||||
|
||||
<h3>NCCL_DEBUG 与 NCCL_DEBUG_SUBSYS</h3>
|
||||
<pre><code>NCCL_DEBUG=INFO
|
||||
NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING</code></pre>
|
||||
<p>用于确认实际路径，诊断完成后应关闭，正式性能数据不要长期带 INFO 日志。</p>
|
||||
|
||||
<h2 id="logs">6. NCCL 日志速查</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>日志</th><th>含义</th><th>判断</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>Bootstrap: Using eth0:...</code></td>
|
||||
<td>初始化控制连接选择 eth0。</td>
|
||||
<td>只说明 bootstrap，尚不能证明数据走 RDMA。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>NET/IB : Using ... mlx5_0 ...</code></td>
|
||||
<td>NCCL 已识别 RDMA HCA。</td>
|
||||
<td>RDMA 数据后端可用。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>via NET/IB/.../GDRDMA</code></td>
|
||||
<td>跨机边通过 GPUDirect RDMA。</td>
|
||||
<td>理想证据。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>NET/IB : No device found</code></td>
|
||||
<td>容器没有可用 RDMA 设备或驱动/权限不完整。</td>
|
||||
<td>继续看是否回退 Socket。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>NET/Socket : Using 非计算网...</code></td>
|
||||
<td>跨机数据由普通 TCP Socket 传输。</td>
|
||||
<td>若误入低速非计算网，性能会严重受限。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>via P2P/IPC</code></td>
|
||||
<td>同机 GPU 通过 CUDA P2P/IPC。</td>
|
||||
<td>机内路径，不代表跨机路径。</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2 id="incident">7. 2026-07-30 Prefill 变慢事故复盘</h2>
|
||||
<h3>已观测事实</h3>
|
||||
<ul>
|
||||
<li>宿主机存在 <code>/dev/infiniband</code>，两条 400G Rail 均 Up。</li>
|
||||
<li>原脚本容器内不存在 <code>/dev/infiniband</code>。</li>
|
||||
<li>NCCL INFO 明确打印 <code>NET/IB : No device found</code> 和 <code>Using network Socket</code>。</li>
|
||||
<li>原脚本选择 400G 计算网 <code>eth0</code>；quick-map 曾误选低速非计算网。</li>
|
||||
<li>部署规定只有 <code>eth0/eth3</code> 用于节点间通信，两者均为 400G。</li>
|
||||
</ul>
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>冷缓存 Shape</th><th>原脚本网络：eth0 Socket</th><th>quick-map：错误的非计算网</th><th>差异</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>1K → 1, C=1</td>
|
||||
<td class="metric">TTFT 1.458s / 693.1 input tok/s</td>
|
||||
<td class="metric">TTFT 15.88–16.04s / 约 64 tok/s</td>
|
||||
<td>约 10.9×</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>32K → 1, C=1</td>
|
||||
<td class="metric">TTFT 38.062s / 860.5 input tok/s</td>
|
||||
<td class="metric">TTFT 504.44s / 64.96 tok/s</td>
|
||||
<td>约 13.25×</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<div class="finding">
|
||||
<strong>根因判断：</strong>
|
||||
quick-map 没有把 RDMA 设备透传进容器，却把 <code>NCCL_SOCKET_IFNAME</code>
|
||||
设成低速非计算网。NCCL 回退 NET/Socket 后，TP16 跨机数据没有进入规定的
|
||||
<code>eth0/eth3</code> 计算网。<code>NCCL_CROSS_NIC=1</code> 在没有 NET/IB
|
||||
的情况下不是致因。
|
||||
</div>
|
||||
|
||||
<h3>为什么旧日志还会比 38 秒更短</h3>
|
||||
<p>
|
||||
旧矩阵脚本还有第二个独立因素：<code>warmup_requests=16</code>、固定
|
||||
<code>seed=42</code>、ISL/OSL/C 升序运行，而且从不 flush Prefix Cache。
|
||||
因此旧日志混入缓存命中，不能直接与冷 Prefill 比较。
|
||||
</p>
|
||||
|
||||
<div class="good">
|
||||
quick-map 现在只允许 <code>eth0/eth3</code> 和
|
||||
<code>mlx5_0/mlx5_3</code>，并会透传精确 RDMA 设备、强制检查两端
|
||||
<code>NET/IB</code> 日志。代码与 dry-run 已通过；下一步是真机启动验证，
|
||||
在拿到运行时证据前不进入 Kernel 归因。
|
||||
</div>
|
||||
|
||||
<h2 id="checklist">8. 从宿主机到 NCCL 的排查清单</h2>
|
||||
<ol>
|
||||
<li>
|
||||
<strong>宿主机链路：</strong>
|
||||
<code>ethtool eth0</code>、<code>ethtool eth3</code>。
|
||||
</li>
|
||||
<li>
|
||||
<strong>HCA 映射：</strong>
|
||||
<code>ibdev2netdev</code>，确认 <code>mlx5_0→eth0</code>、
|
||||
<code>mlx5_3→eth3</code>。
|
||||
</li>
|
||||
<li>
|
||||
<strong>宿主机设备：</strong>
|
||||
<code>ls -l /dev/infiniband</code>。
|
||||
</li>
|
||||
<li>
|
||||
<strong>容器设备：</strong>
|
||||
<code>docker exec CONTAINER ls -l /dev/infiniband</code>。
|
||||
宿主机有、容器没有，NCCL 仍然用不了 RDMA。
|
||||
</li>
|
||||
<li>
|
||||
<strong>运行时证据：</strong>
|
||||
用一次 <code>NCCL_DEBUG=INFO</code> 启动，搜索
|
||||
<code>NET/IB</code>、<code>NET/Socket</code>、<code>GDRDMA</code>。
|
||||
</li>
|
||||
<li>
|
||||
<strong>硬件计数器：</strong>
|
||||
同时观察 eth0/eth3 流量和 RDMA 端口计数；不能只看环境变量。
|
||||
</li>
|
||||
<li>
|
||||
<strong>端到端 A/B：</strong>
|
||||
冷缓存、同一 prompt、同一模型参数，仅改变一个网络变量。
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<h3>最小 RDMA 设备透传验证</h3>
|
||||
<pre><code>docker run --rm \
|
||||
--device=/dev/infiniband/rdma_cm \
|
||||
--device=/dev/infiniband/uverbs0 \
|
||||
--device=/dev/infiniband/uverbs3 \
|
||||
IMAGE \
|
||||
ls -l /dev/infiniband</code></pre>
|
||||
<p>
|
||||
能看到设备只是第一关。最终仍必须从 NCCL INFO 中看到 <code>NET/IB</code>，
|
||||
并通过通信基准与 SGLang 结果确认。
|
||||
</p>
|
||||
|
||||
<h2 id="sources">9. 官方资料</h2>
|
||||
<ul>
|
||||
<li>
|
||||
<a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/setup.html">
|
||||
NVIDIA NCCL Setup：bootstrap 与通信安全边界
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html">
|
||||
NVIDIA NCCL Environment Variables：SOCKET_IFNAME、IB_HCA、CROSS_NIC
|
||||
</a>
|
||||
</li>
|
||||
<li>
|
||||
<a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html">
|
||||
NVIDIA NCCL Networking Troubleshooting
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<p>
|
||||
<a href="./推理优化计划.html">返回推理优化主计划</a>
|
||||
</p>
|
||||
</main>
|
||||
</div>
|
||||
|
||||
<footer>
|
||||
本页以两台 6000D 的真实设备映射和 2026-07-30 实测日志为例。
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
@ -1,597 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="color-scheme" content="light">
|
||||
<title>Phase 1 Code：DSV4-Pro 双机 Pro6000D SGLang 快速性能地图</title>
|
||||
<style>
|
||||
:root {
|
||||
--canvas: #eef3f4;
|
||||
--paper: #ffffff;
|
||||
--ink: #182126;
|
||||
--muted: #5a6970;
|
||||
--line: #d4dee1;
|
||||
--navy: #17363d;
|
||||
--teal: #087c72;
|
||||
--teal-soft: #e8f5f3;
|
||||
--amber: #a64c14;
|
||||
--amber-soft: #fff1e7;
|
||||
--code-bg: #17252b;
|
||||
--code-ink: #eaf2f3;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; letter-spacing: 0; }
|
||||
html { scroll-behavior: smooth; }
|
||||
body {
|
||||
margin: 0;
|
||||
color: var(--ink);
|
||||
background: var(--canvas);
|
||||
font-family: "PingFang SC", "Microsoft YaHei", "Noto Sans CJK SC", Arial, sans-serif;
|
||||
font-size: 16px;
|
||||
line-height: 1.72;
|
||||
}
|
||||
header {
|
||||
color: #f6fbfb;
|
||||
background: var(--navy);
|
||||
border-bottom: 5px solid #d2692b;
|
||||
}
|
||||
.header-inner, main { width: min(100% - 36px, 1120px); margin: 0 auto; }
|
||||
.header-inner { padding: 34px 0 30px; }
|
||||
.eyebrow { margin: 0 0 6px; color: #9edbd5; font-size: 13px; font-weight: 700; }
|
||||
h1 { margin: 0; font-size: clamp(28px, 4vw, 42px); line-height: 1.25; }
|
||||
.meta { margin-top: 15px; color: #d6e5e7; font-size: 14px; }
|
||||
main {
|
||||
margin-top: 30px;
|
||||
margin-bottom: 70px;
|
||||
padding: 38px 48px 58px;
|
||||
background: var(--paper);
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 6px;
|
||||
box-shadow: 0 12px 30px rgba(27, 45, 51, 0.07);
|
||||
}
|
||||
h2 {
|
||||
margin: 46px 0 15px;
|
||||
padding-bottom: 8px;
|
||||
font-size: 25px;
|
||||
line-height: 1.35;
|
||||
border-bottom: 2px solid #adbbc0;
|
||||
}
|
||||
h2:first-of-type { margin-top: 18px; }
|
||||
h3 { margin: 29px 0 10px; color: #21454d; font-size: 19px; }
|
||||
h4 { margin: 22px 0 8px; font-size: 16px; }
|
||||
p, ul, ol { margin-top: 0; margin-bottom: 16px; }
|
||||
li + li { margin-top: 5px; }
|
||||
a { color: var(--teal); text-underline-offset: 3px; }
|
||||
code {
|
||||
padding: 2px 5px;
|
||||
color: #85380d;
|
||||
background: var(--amber-soft);
|
||||
border-radius: 3px;
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
pre {
|
||||
margin: 14px 0 22px;
|
||||
padding: 16px 18px;
|
||||
overflow: auto;
|
||||
color: var(--code-ink);
|
||||
background: var(--code-bg);
|
||||
border-radius: 5px;
|
||||
font: 13px/1.62 "SFMono-Regular", Consolas, monospace;
|
||||
}
|
||||
pre code { padding: 0; color: inherit; background: transparent; }
|
||||
table { width: 100%; margin: 16px 0 26px; border-collapse: collapse; font-size: 14px; }
|
||||
th, td {
|
||||
padding: 9px 11px;
|
||||
vertical-align: top;
|
||||
text-align: left;
|
||||
border: 1px solid var(--line);
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
th { color: #153b41; background: #eaf2f2; }
|
||||
tbody tr:nth-child(even) { background: #fafcfc; }
|
||||
.callout {
|
||||
margin: 18px 0 26px;
|
||||
padding: 14px 18px;
|
||||
background: var(--teal-soft);
|
||||
border-left: 4px solid var(--teal);
|
||||
}
|
||||
.warning {
|
||||
margin: 18px 0 26px;
|
||||
padding: 14px 18px;
|
||||
background: var(--amber-soft);
|
||||
border-left: 4px solid var(--amber);
|
||||
}
|
||||
.toc {
|
||||
columns: 2;
|
||||
column-gap: 38px;
|
||||
margin: 16px 0 24px;
|
||||
padding-left: 22px;
|
||||
}
|
||||
.toc li { break-inside: avoid; }
|
||||
.path { font-family: "SFMono-Regular", Consolas, monospace; font-size: 13px; }
|
||||
.nowrap { white-space: nowrap; }
|
||||
footer { margin-top: 48px; padding-top: 18px; color: var(--muted); border-top: 1px solid var(--line); }
|
||||
@media (max-width: 760px) {
|
||||
main { padding: 28px 20px 42px; }
|
||||
.toc { columns: 1; }
|
||||
table { display: block; overflow-x: auto; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">Standalone Code Walkthrough / Phase 1</p>
|
||||
<h1>DSV4-Pro 双机 Pro6000D SGLang 快速性能地图：代码详解</h1>
|
||||
<div class="meta">
|
||||
行号基线：<code>ca1f2f63375c</code>
|
||||
生成时间：2026-07-31 13:04:21 CST
|
||||
入口：<code>run_quick_map.sh</code>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<p>
|
||||
<a href="./推理优化计划.html">返回推理优化主计划</a> ·
|
||||
<a href="./phase1_exp.html">打开 Phase 1 实验档案</a>
|
||||
</p>
|
||||
<div class="callout">
|
||||
<strong>文档边界：</strong>这是一份独立代码档案，只解释 Phase 1 实现，不承担阶段结论展示。
|
||||
下文的行号均绑定提交 <code>ca1f2f63375c</code>。代码变更后应先更新基线提交，再重新核对行号。
|
||||
</div>
|
||||
|
||||
<h2 id="read">1. 阅读方法</h2>
|
||||
<ul class="toc">
|
||||
<li><a href="#flow">总体控制流</a></li>
|
||||
<li><a href="#files">文件职责</a></li>
|
||||
<li><a href="#config">配置与场景</a></li>
|
||||
<li><a href="#service">双机服务启动</a></li>
|
||||
<li><a href="#bench">Benchmark 生成</a></li>
|
||||
<li><a href="#mixed">混合 Prefill/Decode</a></li>
|
||||
<li><a href="#results">指标解析与汇总</a></li>
|
||||
<li><a href="#artifacts">结果目录与数据契约</a></li>
|
||||
<li><a href="#index">函数行号索引</a></li>
|
||||
</ul>
|
||||
<p>
|
||||
行号写法例如
|
||||
<code>run_quick_map.sh:L212-L268</code>。它表示该提交中，从第 212 行到第 268 行的完整函数段，
|
||||
不是当前编辑器自动漂移后的行号。
|
||||
</p>
|
||||
|
||||
<h2 id="flow">2. 总体控制流</h2>
|
||||
<pre><code>main "$@"
|
||||
└─ ACTION=all → run_all
|
||||
├─ 校验场景与客户端
|
||||
├─ start_service
|
||||
│ ├─ Worker 节点先启动
|
||||
│ ├─ Head 节点后启动
|
||||
│ ├─ 等待 /health
|
||||
│ └─ 从两端日志验证 NET/IB + 两条 HCA
|
||||
├─ run_fixed_suite
|
||||
│ └─ TSV 每一行 → run_bench_case
|
||||
├─ run_mixed_suite
|
||||
│ └─ control → decode background + long prefill injection
|
||||
├─ stop_service
|
||||
├─ summarize_results
|
||||
└─ complete_manifest</code></pre>
|
||||
<p>
|
||||
Shell 负责生命周期、远端执行、容器和失败策略；Python 负责结果读取、指标补算、聚合与报告。
|
||||
这条分工是理解代码的第一把钥匙。
|
||||
</p>
|
||||
|
||||
<h2 id="files">3. 文件职责</h2>
|
||||
<table>
|
||||
<thead><tr><th>文件</th><th>行数</th><th>职责</th><th>主要输出</th></tr></thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td class="path">run_quick_map.sh</td><td>957</td>
|
||||
<td>唯一入口，管理双机服务、固定场景、混合场景、失败恢复与清理。</td>
|
||||
<td><code>run.log</code>、服务日志、每个 Case 的命令与原始结果。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="path">config.env</td><td>78</td>
|
||||
<td>模型、节点、SGLang、NCCL/RDMA、benchmark、超时和路径配置。</td>
|
||||
<td>被 Shell 直接 <code>source</code>，自身不产生输出。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="path">quick_map_scenarios.tsv</td><td>12</td>
|
||||
<td>固定性能地图的声明式场景表，一行对应一个 Case。</td>
|
||||
<td>输入给 <code>run_fixed_suite</code>。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="path">quick_map_results.py</td><td>716</td>
|
||||
<td>校验 bench JSON、补算百分位、生成 meta/manifest、聚合重复实验。</td>
|
||||
<td><code>summary.csv</code>、<code>summary.jsonl</code>、<code>aggregate.csv</code>、<code>report.md</code>。</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>3.1 文件之间如何调用</h3>
|
||||
<pre><code>用户
|
||||
└─ bash run_quick_map.sh all
|
||||
├─ source config.env
|
||||
│ ├─ 给 Shell 提供模型、节点、服务、NCCL 和 benchmark 变量
|
||||
│ └─ 计算 SCENARIO_FILE / RESULT_BASE / RUNTIME_BASE
|
||||
├─ 读取 quick_map_scenarios.tsv
|
||||
│ └─ 每一行变成一次 run_bench_case 调用
|
||||
├─ 调用 quick_map_results.py
|
||||
│ ├─ validate-scenarios：启动前校验 TSV
|
||||
│ ├─ write-case / mark-case-failed：维护 Case 状态
|
||||
│ ├─ write-manifest / complete-manifest：维护 Run 状态
|
||||
│ ├─ check-bench：验证 bench.json
|
||||
│ └─ summarize：生成 CSV、JSONL 和报告
|
||||
└─ tests/test_quick_map_results.py
|
||||
└─ 只测试 Python 解析和聚合，不启动模型</code></pre>
|
||||
<p>
|
||||
<code>run_quick_map.sh:L6-L16</code> 是关系的起点：先定位自身目录，再
|
||||
<code>source config.env</code>，随后把结果工具固定为同目录下的
|
||||
<code>quick_map_results.py</code>。Shell 与 Python 之间不是 import 关系，
|
||||
而是 Shell 通过 Python CLI 子命令交换 JSON/CSV 文件。
|
||||
</p>
|
||||
<table>
|
||||
<thead><tr><th>上游文件</th><th>下游文件</th><th>连接点</th><th>传递内容</th></tr></thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>config.env</code></td><td><code>run_quick_map.sh</code></td>
|
||||
<td><code>run_quick_map.sh:L8</code></td><td>Shell 变量，允许调用命令中的环境变量覆盖默认值。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>quick_map_scenarios.tsv</code></td><td><code>run_fixed_suite</code></td>
|
||||
<td><code>run_quick_map.sh:L695-L729</code></td><td>Case ID、ISL、OSL、C、请求数规则和 Warm-up。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>run_quick_map.sh</code></td><td><code>quick_map_results.py</code></td>
|
||||
<td><code>RESULT_TOOL</code>，<code>run_quick_map.sh:L13</code></td><td>命令行参数、bench JSON、meta 和 Manifest 路径。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>quick_map_results.py</code></td><td>结果目录</td>
|
||||
<td><code>quick_map_results.py:L307-L601</code></td><td>结构化 Case、Run、汇总和报告。</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>tests/test_quick_map_results.py</code></td><td><code>quick_map_results.py</code></td>
|
||||
<td>Python 单元测试</td><td>用合成数据验证字段兼容、百分位和聚合。</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2 id="config">4. 配置与场景</h2>
|
||||
<h3>4.1 配置分区</h3>
|
||||
<table>
|
||||
<thead><tr><th>代码范围</th><th>配置组</th><th>影响</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>config.env:L4-L6</code></td><td>实验与模型</td><td>实验名、模型名和两节点都能看到的模型路径。</td></tr>
|
||||
<tr><td><code>config.env:L8-L18</code></td><td>节点与并行</td><td>Head/Worker 地址、TP16、EP2、双节点 rank。</td></tr>
|
||||
<tr><td><code>config.env:L20-L22</code></td><td>镜像与缓存</td><td>SGLang 镜像、宿主机缓存目录和容器挂载。</td></tr>
|
||||
<tr><td><code>config.env:L24-L35</code></td><td>NCCL/RDMA</td><td>限定 <code>eth0/eth3</code>、<code>mlx5_0/mlx5_3</code> 以及设备透传。</td></tr>
|
||||
<tr><td><code>config.env:L37-L41</code></td><td>服务容量</td><td>显存比例、CUDA Graph Decode BS、活跃请求上限。</td></tr>
|
||||
<tr><td><code>config.env:L43-L61</code></td><td>压测</td><td>随机数据生成、请求率、重复次数、混合注入和超时。</td></tr>
|
||||
<tr><td><code>config.env:L67-L78</code></td><td>运行控制</td><td>相对路径、Case 过滤、Dry-run、断点续跑。</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>4.2 具体值在哪里看</h3>
|
||||
<p>
|
||||
配置采用 <code>VAR="${VAR:-default}"</code>。含义是：启动命令已经提供
|
||||
<code>VAR</code> 时使用外部值，否则使用 <code>config.env</code> 里的默认值。
|
||||
所以应区分“代码默认值”和“某次 Run 的实际值”。
|
||||
</p>
|
||||
<table>
|
||||
<thead><tr><th>变量</th><th>当前默认值</th><th>默认值定义</th><th>传入服务</th><th>Run 后证据</th></tr></thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>MEM_FRACTION_STATIC</code></td><td><code>0.9</code></td>
|
||||
<td><code>config.env:L38</code></td><td><code>run_quick_map.sh:L258</code> → <code>--mem-fraction-static</code></td>
|
||||
<td><code>server/head_server_cmd.txt</code>；<code>run_manifest.json</code> 的 <code>mem_fraction_static</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>CUDA_GRAPH_MAX_BS_DECODE</code></td><td><code>64</code></td>
|
||||
<td><code>config.env:L39</code></td><td><code>run_quick_map.sh:L259</code></td>
|
||||
<td>服务命令；Manifest 的 <code>cuda_graph_max_bs_decode</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>MAX_RUNNING_REQUESTS</code></td><td><code>256</code></td>
|
||||
<td><code>config.env:L40</code></td><td><code>run_quick_map.sh:L260</code></td>
|
||||
<td>服务命令；Manifest 的 <code>max_running_requests</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>TP_SIZE / EP_SIZE / NNODES</code></td><td><code>16 / 2 / 2</code></td>
|
||||
<td><code>config.env:L15-L17</code></td><td><code>run_quick_map.sh:L250-L253</code></td>
|
||||
<td>服务命令；Manifest 的 <code>tp_size/ep_size/nnodes</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>NCCL_SOCKET_IFNAME</code></td><td><code>eth0</code></td>
|
||||
<td><code>config.env:L26</code></td><td><code>run_quick_map.sh:L231</code></td>
|
||||
<td>服务命令；Manifest 的同名小写字段；NCCL 服务日志</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>NCCL_IB_HCA</code></td><td><code>=mlx5_0:1,mlx5_3:1</code></td>
|
||||
<td><code>config.env:L27</code></td><td><code>run_quick_map.sh:L232</code></td>
|
||||
<td>服务命令；Manifest；两节点 NCCL 日志</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>以 <code>MEM_FRACTION_STATIC</code> 为例，三个查看层级是：</p>
|
||||
<pre><code># 1. 看仓库默认值
|
||||
grep '^MEM_FRACTION_STATIC=' config.env
|
||||
|
||||
# 2. 看本次命令实际覆盖后的值
|
||||
source ./config.env
|
||||
printf '%s\n' "${MEM_FRACTION_STATIC}"
|
||||
|
||||
# 3. 看已经执行的 Run 最终用了什么
|
||||
grep -- '--mem-fraction-static' results/<RUN_ID>/server/head_server_cmd.txt
|
||||
python3 -c 'import json; print(json.load(open(
|
||||
"results/<RUN_ID>/run_manifest.json"))["mem_fraction_static"])'</code></pre>
|
||||
<p>
|
||||
第 3 层最可信，因为 <code>start_service_node</code> 在
|
||||
<code>run_quick_map.sh:L269-L291</code> 先展开命令，再写入
|
||||
<code><role>_server_cmd.txt</code>；<code>write_run_manifest</code>
|
||||
在 <code>L641-L676</code> 另存一份结构化配置。二者不一致时，应以实际容器命令和服务日志继续核查。
|
||||
</p>
|
||||
|
||||
<h3>4.3 TSV 如何变成请求</h3>
|
||||
<p>
|
||||
<code>quick_map_scenarios.tsv:L1</code> 定义列：
|
||||
<code>case_id, stage, isl, osl, concurrency, multiplier, minimum, warmup, note</code>。
|
||||
<code>run_fixed_suite</code> 在 <code>run_quick_map.sh:L695-L736</code> 中逐行读取。
|
||||
</p>
|
||||
<pre><code>num_prompts = concurrency × multiplier
|
||||
num_prompts = max(num_prompts, minimum)</code></pre>
|
||||
<p>
|
||||
计算位于 <code>run_quick_map.sh:L700-L718</code>。因此场景表不直接写死总请求数，
|
||||
而是让总请求数随并发扩大，同时允许 <code>minimum</code> 给低并发 Case 提供最小样本量。
|
||||
<code>CASE_IDS</code> 的过滤发生在 <code>L69-L84</code> 与 <code>L703-L705</code>。
|
||||
</p>
|
||||
|
||||
<h3>4.4 11 个固定场景</h3>
|
||||
<p>
|
||||
<code>quick_map_scenarios.tsv:L2-L12</code> 覆盖冷 Prefill、并发 Prefill、短 Decode、
|
||||
长 Decode 与长上下文 Decode。长 Prefill 和长 Decode Case 默认不做额外 Warm-up，
|
||||
避免昂贵预热和 Prefix Cache 污染；短 Decode Case保留一次 Warm-up。
|
||||
</p>
|
||||
|
||||
<h2 id="service">5. 双机服务启动</h2>
|
||||
<h3>5.1 参数校验与 RDMA 门禁</h3>
|
||||
<p>
|
||||
<code>validate_network_config</code> 位于 <code>run_quick_map.sh:L85-L140</code>。
|
||||
它不接受任意网卡，而是把计算网约束为 <code>eth0/eth3</code>，把 RDMA HCA 约束为
|
||||
<code>mlx5_0/mlx5_3</code>。开启 RDMA 时，两条 rail 和必需设备路径都必须存在。
|
||||
</p>
|
||||
<p>
|
||||
<code>preflight_rdma_devices_on_node</code> 在 <code>L141-L156</code> 逐节点检查
|
||||
<code>/dev/infiniband/rdma_cm</code>、<code>uverbs0</code>、<code>uverbs3</code>。
|
||||
这是宿主机设备存在性检查，不能证明 NCCL 最终真的用了 IB，所以后面还有日志门禁。
|
||||
</p>
|
||||
|
||||
<h3>5.2 Docker 与 SGLang 命令展开</h3>
|
||||
<p>
|
||||
<code>build_server_command</code> 位于 <code>run_quick_map.sh:L212-L268</code>。
|
||||
关键部分如下：
|
||||
</p>
|
||||
<pre><code>docker run --rm --network host --ipc host --shm-size 20g
|
||||
--device /dev/infiniband/rdma_cm
|
||||
--device /dev/infiniband/uverbs0
|
||||
--device /dev/infiniband/uverbs3
|
||||
-e NCCL_SOCKET_IFNAME=eth0,eth3
|
||||
-e NCCL_IB_HCA=mlx5_0,mlx5_3
|
||||
-e NCCL_CROSS_NIC=...
|
||||
IMAGE python3 -m sglang.launch_server
|
||||
--model-path ...
|
||||
--tp-size 16 --ep-size 2 --nnodes 2 --node-rank ...
|
||||
--dist-init-addr HEAD_IP:DIST_PORT
|
||||
--mem-fraction-static ...
|
||||
--cuda-graph-max-bs-decode ...
|
||||
--max-running-requests ...</code></pre>
|
||||
<ul>
|
||||
<li><code>--network host</code> 让容器直接使用宿主机网络栈，避免额外端口映射。</li>
|
||||
<li><code>--device</code> 把宿主机 RDMA 字符设备暴露给容器。只有环境变量而没有设备透传时，NCCL 仍可能找不到 IB。</li>
|
||||
<li><code>--node-rank</code> 区分 Head 为 0、Worker 为 1；其余模型和并行参数保持一致。</li>
|
||||
<li>完整展开命令会保存到结果目录，便于复现，而不是只留在终端历史中。</li>
|
||||
</ul>
|
||||
|
||||
<h3>5.3 为什么 Worker 先启动</h3>
|
||||
<p>
|
||||
<code>start_service</code> 位于 <code>run_quick_map.sh:L334-L388</code>。
|
||||
它先调用 Worker 的 <code>start_service_node</code>，再启动 Head，随后轮询 Head 的
|
||||
<code>/health</code>。这样 Worker 已经等待分布式 rendezvous，Head 启动后两端更容易同步进入初始化。
|
||||
</p>
|
||||
<p>
|
||||
健康检查成功还不够。<code>verify_nccl_transport_node</code>
|
||||
在 <code>L293-L325</code> 从服务日志拒绝 <code>NET/IB : No device found</code>，
|
||||
并要求看到 <code>NET/IB</code> 及两条 HCA；<code>L326-L333</code> 对两节点都执行。
|
||||
因而脚本采用 fail-closed：无法证明走 RDMA 就不开始 benchmark。
|
||||
</p>
|
||||
|
||||
<h3>5.4 停止与证据保存</h3>
|
||||
<p>
|
||||
<code>stop_service_node</code> 位于 <code>run_quick_map.sh:L389-L410</code>。
|
||||
删除容器前先保存 <code>docker inspect</code> 和最终日志，再执行强制移除。
|
||||
<code>cleanup</code> 在 <code>L849-L853</code> 配合 <code>trap</code>，保证异常退出也尝试清理两端服务。
|
||||
</p>
|
||||
|
||||
<h2 id="bench">6. Benchmark 请求生成与 Case 生命周期</h2>
|
||||
<h3>6.1 命令生成</h3>
|
||||
<p>
|
||||
<code>prepare_bench_command</code> 位于 <code>run_quick_map.sh:L416-L464</code>。
|
||||
它在 benchmark 客户端容器中运行 <code>python3 -m sglang.benchmark.serving</code>，
|
||||
使用 <code>random</code> 数据集并显式传入 ISL、OSL、并发、请求数、请求率、Warm-up 与 Seed。
|
||||
</p>
|
||||
<pre><code>--dataset-name random
|
||||
--random-input-len ISL
|
||||
--random-output-len OSL
|
||||
--num-prompts N
|
||||
--max-concurrency C
|
||||
--request-rate REQUEST_RATE
|
||||
--warmup-requests W
|
||||
--seed SEED
|
||||
--output-file bench.json
|
||||
--output-details</code></pre>
|
||||
<div class="warning">
|
||||
<strong>OSL 语义：</strong>随机 benchmark 会把目标输出长度传给服务端，并使用忽略 EOS 的生成设置，
|
||||
目标是生成足量 token。是否真正达到 OSL 仍以 <code>bench.json</code> 中的成功请求数和
|
||||
<code>total_output_tokens</code> 为准，不能只看命令参数。
|
||||
</div>
|
||||
|
||||
<h3>6.2 单个 Case 的完整流程</h3>
|
||||
<p><code>run_bench_case</code> 位于 <code>run_quick_map.sh:L545-L632</code>，顺序是：</p>
|
||||
<ol>
|
||||
<li>根据 suite、case、repetition 创建稳定结果目录。</li>
|
||||
<li>若 <code>RESUME=1</code>，由 <code>case_already_completed</code> 检查 meta 和 bench 是否完整。</li>
|
||||
<li>保存展开后的命令与 Case 元数据。</li>
|
||||
<li>记录开始时间，使用 <code>timeout</code> 执行 benchmark。</li>
|
||||
<li>调用 Python <code>check-bench</code> 校验 JSON，不把“进程退出码为 0”误当成有效结果。</li>
|
||||
<li>失败时由 <code>detect_error_type</code> 区分超时、OOM、服务失活、传输错误和普通 benchmark 失败。</li>
|
||||
<li>写入最终 <code>meta.json</code>，供后续汇总和 Phase 2 时间窗使用。</li>
|
||||
</ol>
|
||||
<p>
|
||||
<code>case_already_completed</code> 在 <code>L511-L525</code> 同时要求 meta 状态为完成、
|
||||
bench 文件存在且可解析。它避免只凭目录存在就跳过半成品。
|
||||
</p>
|
||||
|
||||
<h2 id="mixed">7. 混合 Prefill/Decode A/B</h2>
|
||||
<h3>7.1 一次 repetition 的三个角色</h3>
|
||||
<p><code>run_mixed_repetition</code> 位于 <code>run_quick_map.sh:L755-L832</code>：</p>
|
||||
<table>
|
||||
<thead><tr><th>角色</th><th>Shape</th><th>作用</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>control</td><td>1K → 1K，C=32</td><td>单独运行 Decode 背景，建立无注入基线。</td></tr>
|
||||
<tr><td>decode_background</td><td>1K → 1K，C=32</td><td>混合组中的持续 Decode 请求流。</td></tr>
|
||||
<tr><td>prefill_injection</td><td>128K → 1，C=1</td><td>在 Decode 正式测量期间注入一次长 Prefill。</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>7.2 “背景”在代码里是什么</h3>
|
||||
<p>
|
||||
背景不是 SGLang 特殊模式。它只是 Shell 把一个正常 benchmark 放到后台进程运行：
|
||||
<code>run_quick_map.sh:L783-L792</code> 的子 Shell 加 <code>&</code>。
|
||||
<code>background_pid=$!</code> 保存该进程 PID，主脚本随后还能并行发起长 Prefill。
|
||||
</p>
|
||||
<p>
|
||||
<code>wait_for_bench_main</code> 位于 <code>L738-L753</code>，轮询背景日志中的
|
||||
<code>Starting main benchmark run</code>。看到它以后再等待配置的注入延迟，避免把 Warm-up 阶段误当正式混合阶段。
|
||||
</p>
|
||||
<p>
|
||||
注入前还会在 <code>L803-L816</code> 用 <code>kill -0</code> 检查背景进程是否仍存活。
|
||||
若背景已经正常结束，Case 被重写为
|
||||
<code>BACKGROUND_FINISHED_BEFORE_INJECTION</code>，防止生成一个实际上没有重叠的“混合成功”结果。
|
||||
</p>
|
||||
|
||||
<h3>7.3 A/B 对比来自哪里</h3>
|
||||
<p>
|
||||
Shell 只负责产生 control、background 和 injection 三份原始记录。
|
||||
Python 在 <code>quick_map_results.py:L439-L580</code> 聚合同一 Case 的重复实验，
|
||||
并在报告阶段计算 percentage change。主要观察 background 相对 control 的
|
||||
Output TPS、TTFT P95、TPOT P95 与 E2E P95 变化。
|
||||
</p>
|
||||
|
||||
<h2 id="results">8. 指标解析与汇总</h2>
|
||||
<h3>8.1 为什么需要 Python 补算</h3>
|
||||
<p>
|
||||
SGLang 版本变化可能导致字段名或原始明细形态不同。
|
||||
<code>quick_map_results.py:L106-L125</code> 既支持单个 JSON 对象，也能从混合日志中寻找首个合法 JSON 行。
|
||||
<code>L152-L157</code> 用多个候选字段名读取同一指标。
|
||||
</p>
|
||||
|
||||
<h3>8.2 延迟百分位</h3>
|
||||
<p>
|
||||
<code>latency_stats</code> 位于 <code>quick_map_results.py:L208-L233</code>。
|
||||
优先读取 benchmark 已给出的 mean/P50/P95/P99；缺失时才从请求级数组补算：
|
||||
</p>
|
||||
<ul>
|
||||
<li>E2E：优先 <code>request_latencies</code>，否则用 TTFT 加该请求所有 ITL。</li>
|
||||
<li>TTFT：来自 <code>ttfts</code>。</li>
|
||||
<li>TPOT：优先 <code>tpots</code>，否则取每请求 ITL 平均值。</li>
|
||||
<li>ITL：展开所有请求的逐 token 间隔。</li>
|
||||
</ul>
|
||||
<p>
|
||||
<code>percentile_ms</code> 在 <code>L128-L140</code> 使用线性插值，并把秒转换为毫秒。
|
||||
</p>
|
||||
|
||||
<h3>8.3 吞吐与完成状态</h3>
|
||||
<p>
|
||||
<code>compute_metrics</code> 位于 <code>quick_map_results.py:L236-L272</code>。
|
||||
Total TPS 优先读取 benchmark 自带字段，缺失时才使用 Input TPS + Output TPS。
|
||||
完成数优先读取 <code>completed</code> 或 <code>successful_requests</code>；
|
||||
失败数缺失时才由尝试数减完成数。
|
||||
</p>
|
||||
|
||||
<h3>8.4 重复实验聚合</h3>
|
||||
<p>
|
||||
<code>aggregate_rows</code> 位于 <code>quick_map_results.py:L439-L481</code>。
|
||||
它按 suite/case/role 聚合 repetition，输出均值、离散程度和成功状态。
|
||||
<code>write_report</code> 在 <code>L493-L580</code> 生成面向人的 Markdown 报告，
|
||||
<code>write_csv</code> 与 <code>summarize</code> 在 <code>L581-L601</code> 生成机器可读汇总。
|
||||
</p>
|
||||
|
||||
<h2 id="artifacts">9. 结果目录与数据契约</h2>
|
||||
<pre><code>results/<RUN_ID>/
|
||||
run.log
|
||||
manifest.json
|
||||
server/
|
||||
head_command.txt
|
||||
worker_command.txt
|
||||
*.log
|
||||
*.inspect.json
|
||||
cases/
|
||||
<case_id>/rep<N>/
|
||||
bench_cmd.txt
|
||||
bench.log
|
||||
bench.json
|
||||
meta.json
|
||||
summary.csv
|
||||
summary.jsonl
|
||||
aggregate.csv
|
||||
report.md</code></pre>
|
||||
<p>
|
||||
<code>meta.json</code> 的结构由 <code>quick_map_results.py:L307-L330</code> 写入，
|
||||
包含 shape、并发、请求数、Warm-up、开始结束时间、退出码与错误分类。
|
||||
<code>manifest.json</code> 由 <code>L333-L382</code> 维护，记录模型、镜像、并行参数、
|
||||
NCCL/RDMA 参数和 Git 状态。两者共同保证结果可追溯。
|
||||
</p>
|
||||
|
||||
<h2 id="index">10. 函数行号索引</h2>
|
||||
<h3>10.1 run_quick_map.sh</h3>
|
||||
<table>
|
||||
<thead><tr><th>行号</th><th>函数</th><th>一句话职责</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>L27-L68</td><td><code>log</code> 到 <code>case_selected</code></td><td>日志、时间、命令打印、节点执行和 Case 过滤基础函数。</td></tr>
|
||||
<tr><td>L69-L140</td><td><code>validate_case_filter</code> / <code>validate_network_config</code></td><td>运行前拒绝未知 Case 和非计算网配置。</td></tr>
|
||||
<tr><td>L141-L211</td><td>RDMA、健康和客户端预检</td><td>检查设备、服务、GPU 占用和 benchmark 客户端。</td></tr>
|
||||
<tr><td>L212-L268</td><td><code>build_server_command</code></td><td>构造每个节点的完整 Docker + SGLang 命令。</td></tr>
|
||||
<tr><td>L269-L388</td><td>启动与 NCCL 验证</td><td>启动节点、等待健康、从日志证明 NET/IB 双 HCA。</td></tr>
|
||||
<tr><td>L389-L415</td><td>停止服务</td><td>保存日志与 inspect 后删除两端容器。</td></tr>
|
||||
<tr><td>L416-L510</td><td>bench 命令与 meta 参数</td><td>构造请求并准备结果元数据。</td></tr>
|
||||
<tr><td>L511-L632</td><td>断点续跑、错误分类、单 Case</td><td>执行并验证一个 benchmark Case。</td></tr>
|
||||
<tr><td>L633-L694</td><td>失败标记、Manifest、汇总、日志</td><td>Run 级元数据和结果收口。</td></tr>
|
||||
<tr><td>L695-L737</td><td><code>run_fixed_suite</code></td><td>遍历 TSV 与 repetition。</td></tr>
|
||||
<tr><td>L738-L847</td><td>混合 A/B</td><td>确保 Decode 与长 Prefill 在时间上真实重叠。</td></tr>
|
||||
<tr><td>L849-L932</td><td>清理、独立 suite、all</td><td>管理完整生命周期与最终状态。</td></tr>
|
||||
<tr><td>L933-L957</td><td><code>main</code></td><td>分发 <code>all/start/fixed/mixed/stop</code>。</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>10.2 quick_map_results.py</h3>
|
||||
<table>
|
||||
<thead><tr><th>行号</th><th>函数组</th><th>职责</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>L94-L125</td><td>JSON I/O</td><td>可靠读取原始 benchmark 输出。</td></tr>
|
||||
<tr><td>L128-L207</td><td>百分位与请求级 fallback</td><td>从明细恢复 E2E、TTFT、TPOT、ITL。</td></tr>
|
||||
<tr><td>L208-L272</td><td><code>latency_stats</code> / <code>compute_metrics</code></td><td>统一指标字段与单位。</td></tr>
|
||||
<tr><td>L275-L306</td><td><code>parse_scenarios</code></td><td>校验 TSV schema、类型和重复 Case。</td></tr>
|
||||
<tr><td>L307-L404</td><td>Case、Manifest、失败状态</td><td>维护机器可读运行状态。</td></tr>
|
||||
<tr><td>L405-L492</td><td>行构造与聚合</td><td>把每次 repetition 合并为 Case 统计。</td></tr>
|
||||
<tr><td>L493-L601</td><td>报告与汇总</td><td>输出 Markdown、CSV、JSONL。</td></tr>
|
||||
<tr><td>L602-L716</td><td>CLI</td><td>定义 Shell 调用的子命令和参数。</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<footer>
|
||||
本文只描述提交 <code>ca1f2f63375c</code> 的实现。维护时应同时更新提交基线、行号索引与关键控制流，
|
||||
不应只改文字结论。
|
||||
</footer>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@ -1,853 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="color-scheme" content="light">
|
||||
<title>Phase 1：DeepSeek-V4-Pro 双机 Pro6000D SGLang 快速性能地图</title>
|
||||
<style>
|
||||
:root {
|
||||
--canvas: #f3f6f7;
|
||||
--paper: #ffffff;
|
||||
--ink: #172126;
|
||||
--muted: #5d6b71;
|
||||
--line: #d8e0e3;
|
||||
--teal: #087e75;
|
||||
--teal-soft: #e8f5f3;
|
||||
--orange: #b65318;
|
||||
--orange-soft: #fff1e8;
|
||||
--code-bg: #18262b;
|
||||
--code-ink: #eaf2f3;
|
||||
}
|
||||
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
letter-spacing: 0;
|
||||
}
|
||||
|
||||
html {
|
||||
scroll-behavior: smooth;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
color: var(--ink);
|
||||
background: var(--canvas);
|
||||
font-family: "PingFang SC", "Microsoft YaHei", "Noto Sans CJK SC",
|
||||
Arial, sans-serif;
|
||||
font-size: 16px;
|
||||
line-height: 1.72;
|
||||
}
|
||||
|
||||
a {
|
||||
color: var(--teal);
|
||||
text-underline-offset: 3px;
|
||||
}
|
||||
|
||||
header {
|
||||
color: #f7fbfb;
|
||||
background: #17343a;
|
||||
border-bottom: 5px solid #d76a2a;
|
||||
}
|
||||
|
||||
.header-inner,
|
||||
main {
|
||||
width: min(100% - 40px, 1020px);
|
||||
margin: 0 auto;
|
||||
}
|
||||
|
||||
.header-inner {
|
||||
padding: 34px 0 30px;
|
||||
}
|
||||
|
||||
.eyebrow {
|
||||
margin: 0 0 6px;
|
||||
color: #9edbd5;
|
||||
font-size: 13px;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
h1 {
|
||||
max-width: 900px;
|
||||
margin: 0;
|
||||
font-size: clamp(28px, 4vw, 42px);
|
||||
line-height: 1.25;
|
||||
}
|
||||
|
||||
.meta {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px 20px;
|
||||
margin-top: 16px;
|
||||
color: #d4e4e6;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
main {
|
||||
margin-top: 32px;
|
||||
margin-bottom: 72px;
|
||||
padding: 40px 50px 60px;
|
||||
background: var(--paper);
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 6px;
|
||||
box-shadow: 0 12px 34px rgba(28, 43, 49, 0.07);
|
||||
}
|
||||
|
||||
.back {
|
||||
display: inline-block;
|
||||
margin-bottom: 20px;
|
||||
font-weight: 700;
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
.status {
|
||||
margin: 0 0 28px;
|
||||
padding: 14px 18px;
|
||||
background: var(--teal-soft);
|
||||
border-left: 4px solid var(--teal);
|
||||
}
|
||||
|
||||
.status strong {
|
||||
color: #075e58;
|
||||
}
|
||||
|
||||
.decision {
|
||||
padding: 14px 18px;
|
||||
background: var(--teal-soft);
|
||||
border-left: 4px solid var(--teal);
|
||||
}
|
||||
|
||||
h2 {
|
||||
margin: 48px 0 16px;
|
||||
padding-bottom: 9px;
|
||||
font-size: 25px;
|
||||
line-height: 1.35;
|
||||
border-bottom: 2px solid #afbcc1;
|
||||
}
|
||||
|
||||
h2:first-of-type {
|
||||
margin-top: 14px;
|
||||
}
|
||||
|
||||
h3 {
|
||||
margin: 30px 0 10px;
|
||||
color: #21434a;
|
||||
font-size: 19px;
|
||||
}
|
||||
|
||||
p,
|
||||
ul,
|
||||
ol {
|
||||
margin-top: 0;
|
||||
margin-bottom: 16px;
|
||||
}
|
||||
|
||||
li + li {
|
||||
margin-top: 5px;
|
||||
}
|
||||
|
||||
table {
|
||||
width: 100%;
|
||||
margin: 18px 0 28px;
|
||||
border-collapse: collapse;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
th,
|
||||
td {
|
||||
padding: 10px 12px;
|
||||
vertical-align: top;
|
||||
text-align: left;
|
||||
border: 1px solid var(--line);
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
th {
|
||||
color: #15393e;
|
||||
background: #eaf2f2;
|
||||
}
|
||||
|
||||
tbody tr:nth-child(even) {
|
||||
background: #fafcfc;
|
||||
}
|
||||
|
||||
code {
|
||||
padding: 2px 5px;
|
||||
color: #85380d;
|
||||
background: var(--orange-soft);
|
||||
border-radius: 3px;
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
font-size: 0.9em;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
|
||||
pre {
|
||||
margin: 18px 0 24px;
|
||||
padding: 16px 18px;
|
||||
overflow: auto;
|
||||
color: var(--code-ink);
|
||||
background: var(--code-bg);
|
||||
border-radius: 6px;
|
||||
line-height: 1.55;
|
||||
}
|
||||
|
||||
pre code {
|
||||
padding: 0;
|
||||
color: inherit;
|
||||
background: transparent;
|
||||
white-space: pre;
|
||||
}
|
||||
|
||||
.pending {
|
||||
color: var(--orange);
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
.pass {
|
||||
color: var(--teal);
|
||||
font-weight: 700;
|
||||
}
|
||||
|
||||
@media (max-width: 680px) {
|
||||
.header-inner,
|
||||
main {
|
||||
width: min(100% - 24px, 1020px);
|
||||
}
|
||||
|
||||
main {
|
||||
padding: 28px 18px 44px;
|
||||
}
|
||||
|
||||
table {
|
||||
display: block;
|
||||
overflow-x: auto;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">Implementation & Result Record</p>
|
||||
<h1>Phase 1：DeepSeek-V4-Pro 双机 Pro6000D SGLang 快速性能地图</h1>
|
||||
<div class="meta">
|
||||
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||
<span>拓扑：SGLang TP16 / EP2</span>
|
||||
<span>更新：2026-07-31 16:46:05 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||
<a class="back" href="./phase1_code.html">打开 Phase 1 代码详解</a>
|
||||
|
||||
<p class="status">
|
||||
<strong>阶段状态：已完成。</strong>
|
||||
正式 Run <code>dsv4pro-phase1-full-20260730-220916</code> 在双 Rail
|
||||
<code>NET/IB + GDRDMA</code> 下完成 9 个固定点和 3 个混合 A/B 结果，
|
||||
共 12/12 成功，用时 28 分 36 秒。补充 Run
|
||||
<code>dsv4pro-phase1-long-decode-20260730-234236</code> 完成长输出与
|
||||
长上下文 Decode 2/2。两个 Run 合计 11 个固定点和 3 个混合结果，
|
||||
14/14 成功；服务、容器与 16 张 GPU 已清理。
|
||||
</p>
|
||||
|
||||
<h2>1. 目标与边界</h2>
|
||||
<p>
|
||||
用数小时以内、可重复的小矩阵替代约一天以上的全量扫描，先回答
|
||||
Prefill、Decode、长上下文和混合干扰各自是否存在明显异常，再决定后续
|
||||
Timeline 和 Kernel Profiling 的捕获对象。该阶段不要求为了“跑满表格”
|
||||
而浪费算力；一旦出现稳定、可复现且足以改变调查方向的异常，就可以提前结束。
|
||||
</p>
|
||||
<ul>
|
||||
<li>只测试 SGLang，不测试 vLLM。</li>
|
||||
<li>使用双机 16 卡完整实例，不做 PD 分离。</li>
|
||||
<li>不启用 MTP、EAGLE、DSpark 或其他投机解码。</li>
|
||||
<li>本轮不启用 Profiler；正文只记录最终有效 Run，失败尝试仅在末尾总结经验。</li>
|
||||
<li>不修改或调用旧的全天全量 Benchmark 脚本。</li>
|
||||
</ul>
|
||||
|
||||
<h2>2. 精简实现</h2>
|
||||
<p>实验代码位于：</p>
|
||||
<pre><code>/data/hzy/sskj/experiments/pro6000/
|
||||
dsv4pro_pro6000d_2node_sglang_tp16_quick_map/</code></pre>
|
||||
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>文件</th>
|
||||
<th>职责</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>run_quick_map.sh</code></td>
|
||||
<td>唯一 Shell 入口：双机服务启停、固定矩阵、混合 A/B、错误处理与清理</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>config.env</code></td>
|
||||
<td>节点、模型、镜像、并行与容量参数</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>quick_map_scenarios.tsv</code></td>
|
||||
<td>十一个固定工作负载点</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>quick_map_results.py</code></td>
|
||||
<td>验证 Bench JSON，生成 CSV、JSONL 和 Markdown 汇总</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>tests/test_quick_map_results.py</code></td>
|
||||
<td>结果解析回归测试</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<p>单入口的操作面：</p>
|
||||
<pre><code class="language-bash">bash run_quick_map.sh all
|
||||
|
||||
# 仅排障时使用同一个入口
|
||||
bash run_quick_map.sh start
|
||||
bash run_quick_map.sh fixed
|
||||
bash run_quick_map.sh mixed
|
||||
bash run_quick_map.sh stop</code></pre>
|
||||
|
||||
<h2>3. 服务配置</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>配置项</th>
|
||||
<th>当前值</th>
|
||||
<th>说明</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>镜像</td>
|
||||
<td><code>lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45</code></td>
|
||||
<td>沿用已验证可加载 DSV4-Pro 的版本</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>模型</td>
|
||||
<td><code>/data/hf_models/DeepSeek-V4-Pro</code></td>
|
||||
<td>两台节点均有本地权重</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>并行</td>
|
||||
<td><code>TP=16, EP=2, nnodes=2</code></td>
|
||||
<td>每台 8 卡，共 16 Rank</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>显存比例</td>
|
||||
<td><code>0.9</code></td>
|
||||
<td>保持已知基线，不在本阶段调参</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>活跃请求上限</td>
|
||||
<td><code>256</code></td>
|
||||
<td>覆盖本轮最大并发 64</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>CUDA Graph Decode BS</td>
|
||||
<td><code>64</code></td>
|
||||
<td>覆盖固定矩阵中的 Decode C64</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>NCCL Socket 接口</td>
|
||||
<td><code>eth0</code></td>
|
||||
<td>RDMA 失败回退时也承载跨机 Tensor，不只是 bootstrap</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>RoCE HCA</td>
|
||||
<td><code>mlx5_0,mlx5_3</code></td>
|
||||
<td>启动器只透传对应的 <code>uverbs0/uverbs3</code> 与 <code>rdma_cm</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>传输后端门禁</td>
|
||||
<td><code>REQUIRE_NCCL_IB=1</code></td>
|
||||
<td>两端日志未证明 <code>NET/IB + mlx5_0 + mlx5_3</code> 时禁止开始 benchmark</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>代码分支</td>
|
||||
<td><code>hzy</code></td>
|
||||
<td>从该维护分支向中央仓库 <code>main</code> 提交合并请求</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>4. 固定快速矩阵</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Case ID</th>
|
||||
<th>ISL</th>
|
||||
<th>OSL</th>
|
||||
<th>C</th>
|
||||
<th>目的</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td><code>short_prefill_latency_1k_c1</code></td><td>1K</td><td>1</td><td>1</td><td>最小 TTFT</td></tr>
|
||||
<tr><td><code>mid_prefill_latency_32k_c1</code></td><td>32K</td><td>1</td><td>1</td><td>中长 Prefill</td></tr>
|
||||
<tr><td><code>long_prefill_latency_128k_c1</code></td><td>128K</td><td>1</td><td>1</td><td>长上下文 Prefill</td></tr>
|
||||
<tr><td><code>mid_prefill_throughput_32k_c16</code></td><td>32K</td><td>1</td><td>16</td><td>Prefill 输入吞吐</td></tr>
|
||||
<tr><td><code>decode_latency_1k_to_1k_c1</code></td><td>1K</td><td>1K</td><td>1</td><td>单请求 TPOT</td></tr>
|
||||
<tr><td><code>decode_throughput_1k_to_1k_c16</code></td><td>1K</td><td>1K</td><td>16</td><td>Decode 吞吐</td></tr>
|
||||
<tr><td><code>decode_throughput_1k_to_1k_c32</code></td><td>1K</td><td>1K</td><td>32</td><td>Decode 吞吐</td></tr>
|
||||
<tr><td><code>decode_throughput_1k_to_1k_c64</code></td><td>1K</td><td>1K</td><td>64</td><td>Decode 高并发</td></tr>
|
||||
<tr><td><code>long_output_decode_1k_to_4k_c16</code></td><td>1K</td><td>4K</td><td>16</td><td>持续 Decode 与 KV 增长</td></tr>
|
||||
<tr><td><code>long_context_decode_128k_to_1k_c1</code></td><td>128K</td><td>1K</td><td>1</td><td>长上下文上的 Decode 成本</td></tr>
|
||||
<tr><td><code>balanced_32k_to_1k_c8</code></td><td>32K</td><td>1K</td><td>8</td><td>综合压力</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
快速 Run 使用一次重复和一波测量请求，即 <code>num_prompts=C</code>。
|
||||
32K/128K Prefill 与 128K 长上下文 Decode 不做昂贵的同形状 Warm-up；
|
||||
短 Prefill、普通 Decode 与 1K → 4K 长输出 Decode 使用一个 Warm-up，
|
||||
并在正式计时前清空 Prefix Cache。固定矩阵不做 SLO 截断或自适应并发搜索。
|
||||
</p>
|
||||
|
||||
<h2>5. SGLang Benchmark 与 Prefix Cache</h2>
|
||||
<h3>5.1 <code>random</code> 如何生成 ISL</h3>
|
||||
<p>
|
||||
当前镜像的实现位于
|
||||
<code>/sgl-workspace/sglang/python/sglang/benchmark/datasets/random.py</code>。
|
||||
<code>dataset-name=random</code> 会读取 ShareGPT，打乱样本后取每条会话的首轮用户文本：
|
||||
文本过长就截断，过短就重复其 token，直到达到目标 ISL。
|
||||
<code>random-range-ratio=1.0</code> 使每条请求都使用精确的目标长度。
|
||||
本机数据集共有 94,145 行，其中 92,886 行可用、71,904 个不同首轮文本，
|
||||
因此不存在此前“两条数据只能形成两个并发请求”的问题。
|
||||
</p>
|
||||
<p>
|
||||
<code>random-ids</code> 则直接构造随机整数 token id，不读取 ShareGPT。
|
||||
当前源码同时警告这种方式可能触发 NaN，因此本阶段继续使用
|
||||
<code>random + 大规模 ShareGPT</code>，并通过清缓存隔离不同测试点。
|
||||
</p>
|
||||
|
||||
<h3>5.2 OSL 为什么能达到指定长度</h3>
|
||||
<p>
|
||||
SGLang 原生请求函数位于
|
||||
<code>/sgl-workspace/sglang/python/sglang/benchmark/serving.py</code>。
|
||||
它将目标 OSL 写入 <code>max_new_tokens</code>，并默认设置
|
||||
<code>ignore_eos=True</code>。因此模型即使提前生成 EOS，也会继续生成到指定 OSL；
|
||||
只有请求失败、超时或触及上下文限制时，实际输出才可能不足。
|
||||
</p>
|
||||
<pre><code class="language-python">sampling_params = {
|
||||
"max_new_tokens": request_func_input.output_len,
|
||||
"ignore_eos": not args.disable_ignore_eos,
|
||||
}</code></pre>
|
||||
|
||||
<h3>5.3 为什么 Warm-up 会污染 Prefix Cache</h3>
|
||||
<p>
|
||||
SGLang benchmark 的 Warm-up 直接复用 <code>input_requests[0]</code>，
|
||||
而正式测量随后仍会遍历包含该请求的完整列表。因此，只要服务启用了 Prefix Cache，
|
||||
第一条正式请求就可能命中刚刚 Warm-up 的前缀。第一次 Run 的服务日志实际出现
|
||||
<code>#cached-token: 768</code>，证明该污染在当前环境真实发生。
|
||||
</p>
|
||||
<p>
|
||||
修复方式是在每个隔离测试点传入 <code>--flush-cache</code>。benchmark 会先完成
|
||||
Warm-up，再调用服务端 <code>/flush_cache</code>，最后才启动计时。这样保留 Kernel
|
||||
和执行路径预热，同时不把 Warm-up 的 KV 前缀带入测量。混合干扰中的长 Prefill
|
||||
注入不会清缓存，避免在 Decode 背景运行时改变其服务状态；背景与注入使用不同随机种子。
|
||||
</p>
|
||||
|
||||
<h3>5.4 如何单独测试 Prefix Caching</h3>
|
||||
<ol>
|
||||
<li>调用 <code>/flush_cache</code>，发送固定长 Prompt P，记录 Cold TTFT 和 <code>#cached-token</code>。</li>
|
||||
<li>不清缓存，原样重发 P，记录 Warm TTFT；预期 cached token 明显增加、TTFT 降低。</li>
|
||||
<li>再次清缓存，发送同长度但内容不同的 Prompt Q，排除长度、JIT 和偶然波动造成的假提升。</li>
|
||||
</ol>
|
||||
<p>
|
||||
三组请求保持 OSL、采样参数和并发一致，各重复至少 3 次。Prefix Cache 是生产优化能力，
|
||||
不是“坏东西”；这里只是在无缓存性能基线中隔离它，后续会把缓存命中场景作为单独 A/B。
|
||||
</p>
|
||||
|
||||
<h2>6. 混合干扰实现</h2>
|
||||
<p>
|
||||
这里的“背景”不是 SGLang 后台线程，而是<strong>先启动并持续运行的一批
|
||||
Decode 基准流量</strong>。它既在实验期间占用 GPU，也是我们希望观察是否
|
||||
变慢的对象。混合 A/B 的问题非常具体：同样一批 Decode 请求，在没有长
|
||||
Prefill 干扰和有长 Prefill 干扰时，性能会相差多少？
|
||||
</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>组别</th><th>运行内容</th><th>作用</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>A：Control</td><td>仅运行 64 条 <code>1K → 1K, C=32</code> Decode</td><td>建立无干扰基线</td></tr>
|
||||
<tr><td>B：Treatment</td><td>运行相同 Decode，并在正式测量开始 10 秒后注入一条 <code>128K → 1</code> Prefill</td><td>测量 Prefill 对 Decode 的干扰</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<ol>
|
||||
<li>先完成 A 组，仅运行 Decode，保存对照指标。</li>
|
||||
<li>启动 B 组的 Decode 基准流量，并从日志确认它已进入正式测量，而不只是完成客户端初始化。</li>
|
||||
<li>正式测量开始 10 秒后，并行提交一个 <code>128K → 1</code> 长 Prefill。</li>
|
||||
<li>等待两类请求都结束，分别保存 Decode 流量和长 Prefill 请求的结果。</li>
|
||||
<li>用 A、B 两组 Decode 的 Output TPS、TTFT P95、TPOT P95 与 E2E P95 计算变化率；长 Prefill 自身的 TTFT 单独报告。</li>
|
||||
</ol>
|
||||
<pre><code>A：Decode ───────────────────────────────→ 结束
|
||||
|
||||
B：Decode ───────────────────────────────→ 结束
|
||||
正式测量 + 10 秒
|
||||
└─ 128K Prefill ─→ 结束
|
||||
共同占用同一服务</code></pre>
|
||||
<pre><code class="language-bash">(
|
||||
run_bench_case ... 1024 1024 32 64
|
||||
) &
|
||||
background_pid=$!
|
||||
|
||||
# 实际代码先从 bench.log 确认正式测量已经开始。
|
||||
sleep 10
|
||||
run_bench_case ... 131072 1 1 1
|
||||
wait "${background_pid}"</code></pre>
|
||||
<p>
|
||||
<code>&</code> 让 Decode benchmark 与后续 Prefill 并行；
|
||||
<code>$!</code> 取得该 Decode benchmark 的进程号；
|
||||
<code>wait</code> 等待它完成。总请求数 64、并发 32，表示最多同时有
|
||||
32 条请求在途，通常形成约两波请求。如果 Decode 流量在注入前已经结束，
|
||||
两类请求没有发生重叠，结果会被明确改写为
|
||||
<code>BACKGROUND_FINISHED_BEFORE_INJECTION</code>，避免生成虚假的“混合成功”。
|
||||
</p>
|
||||
|
||||
<h2>7. 结果与可追溯性</h2>
|
||||
<pre><code>results/<RUN_ID>/
|
||||
run_manifest.json
|
||||
run.log
|
||||
summary.csv
|
||||
summary.jsonl
|
||||
aggregate.csv
|
||||
report.md
|
||||
cases/<case_id>/rep1/
|
||||
bench_cmd.txt
|
||||
bench.jsonl
|
||||
bench.log
|
||||
meta.json
|
||||
server/
|
||||
head_server_cmd.txt
|
||||
worker_server_cmd.txt
|
||||
head_server.log
|
||||
worker_server.log</code></pre>
|
||||
<p>
|
||||
汇总保留 Request/Input/Output/Total TPS，以及 E2E、TTFT、TPOT、ITL 的
|
||||
Mean、P50、P95、P99。断点续跑前会重新解析原始 Bench JSON，不能只凭文件存在就跳过。
|
||||
</p>
|
||||
|
||||
<h2>8. 已完成验证</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>检查</th><th>结果</th><th>证据</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>Shell 语法</td><td class="pass">通过</td><td><code>bash -n run_quick_map.sh</code></td></tr>
|
||||
<tr><td>Python 单测</td><td class="pass">3/3 通过</td><td>场景唯一性、百分位回退、失败结果汇总</td></tr>
|
||||
<tr><td>完整 Dry-run</td><td class="pass">通过</td><td>服务、十一个固定点、混合 A/B、清理均展开成功</td></tr>
|
||||
<tr><td>真实旧 Bench JSON 解析</td><td class="pass">通过</td><td>成功解析 P50/P95/P99 与吞吐字段</td></tr>
|
||||
<tr><td>项目精简</td><td class="pass">通过</td><td>实验目录顶层仅保留一个 Shell 入口</td></tr>
|
||||
<tr><td>双 Rail 传输门禁</td><td class="pass">通过</td><td>Head 与 Worker 均识别 <code>mlx5_0/mlx5_3</code>，跨节点 Channel 使用 <code>NET/IB/*/GDRDMA</code></td></tr>
|
||||
<tr><td>四点 Sanity</td><td class="pass">4/4 通过</td><td>1K/32K Prefill 与 C1/C32 Decode 均恢复到合理量级</td></tr>
|
||||
<tr><td>冷 Prefix 口径</td><td class="pass">通过</td><td>正式测量请求的 Head 日志显示 <code>#cached-token: 0</code></td></tr>
|
||||
<tr><td>完整真机 Run</td><td class="pass">12/12 通过</td><td>固定矩阵 9/9，混合 A/B 3/3，运行期失败 0</td></tr>
|
||||
<tr><td>长 Decode 补测</td><td class="pass">2/2 通过</td><td>1K → 4K C16 与 128K → 1K C1 均生成完整目标 OSL</td></tr>
|
||||
<tr><td>资源清理</td><td class="pass">通过</td><td>两节点相关容器与计算进程为 0，16 张 GPU 显存占用为 0</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>9. 最终真机结果</h2>
|
||||
<p>
|
||||
本节只使用正式成功 Run。Profiler 与投机解码均关闭，每个 Case 只做一次快速测量，
|
||||
所以它适合决定下一步 Profile 对象，不作为需要统计置信度的最终容量认证。
|
||||
</p>
|
||||
|
||||
<h3>9.1 执行摘要</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>项目</th><th>结果</th><th>证据</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>Run ID</td><td><code>dsv4pro-phase1-full-20260730-220916</code></td><td><code>COMPLETED</code></td></tr>
|
||||
<tr><td>运行时间</td><td>28 分 36 秒</td><td>22:09:47 至 22:38:22 CST</td></tr>
|
||||
<tr><td>网络路径</td><td>双 Rail <code>NET/IB + GDRDMA</code></td><td><code>mlx5_0</code> 与 <code>mlx5_3</code></td></tr>
|
||||
<tr><td>正式 Run 完整性</td><td>12/12 成功</td><td>固定点 9/9；混合 A/B 3/3</td></tr>
|
||||
<tr><td>补充 Run</td><td><code>dsv4pro-phase1-long-decode-20260730-234236</code></td><td>固定点 2/2；约 12 分钟含服务启动与清理</td></tr>
|
||||
<tr><td>阶段合计</td><td>14/14 成功</td><td>固定点 11/11；混合 A/B 3/3</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>9.2 Prefill</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>场景</th><th>Input TPS</th><th>TTFT P95</th><th>观察</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>1K → 1，C=1</td><td>1,969.66 tok/s</td><td>0.502 s</td><td>短请求固定开销占比更高</td></tr>
|
||||
<tr><td>32K → 1，C=1</td><td>2,652.76 tok/s</td><td>12.335 s</td><td>单请求吞吐进入稳定区间</td></tr>
|
||||
<tr><td>128K → 1，C=1</td><td>2,710.16 tok/s</td><td>48.344 s</td><td>长 Prefill 代表点</td></tr>
|
||||
<tr><td>32K → 1，C=16</td><td>3,112.77 tok/s</td><td>162.087 s</td><td>聚合吞吐仅比 C=1 高 17.3%，排队时延显著增加</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>9.3 Decode</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>1K → 1K</th><th>Output TPS</th><th>TTFT P95</th><th>TPOT P95</th><th>E2E P95</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>C=1</td><td>31.41 tok/s</td><td>0.363 s</td><td>31.47 ms</td><td>32.555 s</td></tr>
|
||||
<tr><td>C=16</td><td>295.29 tok/s</td><td>4.950 s</td><td>50.02 ms</td><td>55.444 s</td></tr>
|
||||
<tr><td>C=32</td><td>461.68 tok/s</td><td>8.022 s</td><td>63.31 ms</td><td>70.933 s</td></tr>
|
||||
<tr><td>C=64</td><td>647.42 tok/s</td><td>12.716 s</td><td>93.44 ms</td><td>101.163 s</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
Decode 吞吐到 C=64 仍在上升，但增益递减且 TPOT 明显变差。综合场景
|
||||
<code>32K → 1K，C=8</code> 的 Input/Output TPS 为
|
||||
<code>2,038.00 / 63.69</code>，TTFT P95 为 <code>82.084 s</code>，
|
||||
说明 Prefill 与 Decode 同时存在时干扰很强。
|
||||
</p>
|
||||
|
||||
<h3>9.4 长 Decode 补测</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>场景</th><th>Output TPS</th><th>TTFT P95</th><th>TPOT P95</th><th>E2E P95</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>1K → 4K，C=16</td><td>310.02 tok/s</td><td>6.241 s</td><td>50.33 ms</td><td>211.364 s</td></tr>
|
||||
<tr><td>128K → 1K，C=1</td><td>12.43 tok/s</td><td>49.326 s</td><td>32.24 ms</td><td>82.312 s</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
<code>1K → 4K，C=16</code> 相比 <code>1K → 1K，C=16</code>，
|
||||
Output TPS 增加 4.99%，TPOT P95 只增加 0.62%。较长 Decode 没有出现
|
||||
稳态吞吐塌陷；吞吐略升是固定启动和 Prefill 成本被更多输出 token 摊薄。
|
||||
</p>
|
||||
<p>
|
||||
<code>128K → 1K，C=1</code> 的 TTFT 只比 <code>128K → 1</code>
|
||||
纯 Prefill 高 2.03%，而 TPOT P95 只比 <code>1K → 1K，C=1</code>
|
||||
高 2.47%。因此这次长上下文请求的主要新增成本在 Prefill，而不是每个 Decode
|
||||
token。表中的 12.43 Output TPS 是把 49 秒 Prefill 也计入总时长的端到端值，
|
||||
不能把它误读为纯 Decode 速率。
|
||||
</p>
|
||||
|
||||
<h3>9.5 混合 Prefill/Decode A/B</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>指标</th><th>A：仅 Decode</th><th>B：注入 128K Prefill</th><th>变化</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>Output TPS</td><td>455.68 tok/s</td><td>345.95 tok/s</td><td>-24.08%</td></tr>
|
||||
<tr><td>TTFT P95</td><td>9.443 s</td><td>10.194 s</td><td>+7.96%</td></tr>
|
||||
<tr><td>TPOT P95</td><td>65.88 ms</td><td>109.73 ms</td><td>+66.55%</td></tr>
|
||||
<tr><td>E2E P95</td><td>72.008 s</td><td>117.630 s</td><td>+63.36%</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="decision">
|
||||
Phase 2 重放五个固定代表点：
|
||||
<code>128K → 1，C=1</code>、<code>32K → 1，C=16</code>、
|
||||
<code>1K → 1K，C=32</code>、<code>1K → 4K，C=16</code>、
|
||||
<code>128K → 1K，C=1</code>，再执行有无 128K 注入的混合 A/B。
|
||||
目标是区分计算、显存带宽、调度排队、跨机通信和节点不均衡。
|
||||
</p>
|
||||
<p>
|
||||
完整产物：
|
||||
<a href="./results/dsv4pro-phase1-full-20260730-220916/report.md">报告</a>、
|
||||
<a href="./results/dsv4pro-phase1-full-20260730-220916/summary.csv">逐点汇总</a>、
|
||||
<a href="./results/dsv4pro-phase1-full-20260730-220916/aggregate.csv">聚合表</a>、
|
||||
<a href="./results/dsv4pro-phase1-full-20260730-220916/run_manifest.json">运行清单</a>；
|
||||
长 Decode 补测的
|
||||
<a href="./results/dsv4pro-phase1-long-decode-20260730-234236/report.md">报告</a>、
|
||||
<a href="./results/dsv4pro-phase1-long-decode-20260730-234236/summary.csv">逐点汇总</a>
|
||||
和
|
||||
<a href="./results/dsv4pro-phase1-long-decode-20260730-234236/run_manifest.json">运行清单</a>。
|
||||
</p>
|
||||
|
||||
<h2>10. 经验教训</h2>
|
||||
<ul>
|
||||
<li>启动参数不等于实际传输路径；开始性能测试前必须由 NCCL 日志证明 <code>NET/IB</code>。</li>
|
||||
<li>Warm-up、固定随机种子和跨 Case Prefix Cache 会改变 TTFT，冷缓存与热缓存必须分开报告。</li>
|
||||
<li>先跑四点 Sanity 再启动完整矩阵，可以在几分钟内验证环境、口径和数量级。</li>
|
||||
</ul>
|
||||
<p>
|
||||
历史排查细节保存在
|
||||
<a href="./results/script-audit-20260730/report.md">TTFT 脚本口径审计报告</a>
|
||||
与
|
||||
<a href="./results/network-path-audit-20260730/report.md">TP16 网络路径审计报告</a>，
|
||||
不作为本阶段最终结果。
|
||||
</p>
|
||||
|
||||
<h2>11. 实验复现命令</h2>
|
||||
<p class="decision">
|
||||
本节记录的是本阶段<strong>实际执行过</strong>的命令。长命令同时由程序原样保存到
|
||||
<code>results/<RUN_ID>/server/*_server_cmd.txt</code> 和每个 Case 的
|
||||
<code>bench_cmd.txt</code>；这些落盘文件是最终证据，正文中的换行仅用于阅读。
|
||||
</p>
|
||||
|
||||
<h3>11.1 实际执行：完整 Phase 1</h3>
|
||||
<p><strong>执行位置：</strong><code>174.1.51.5</code>；脚本通过 SSH 启动 <code>.7</code> Worker。</p>
|
||||
<pre><code class="language-bash">cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||
|
||||
tmux new-session -d -s dsv4pro-phase1-full \
|
||||
"RUN_ID=dsv4pro-phase1-full-20260730-220916 bash run_quick_map.sh all \
|
||||
2>&1 | tee /data/hzy/dsv4pro_phase1_full_20260730-220916.log"
|
||||
|
||||
tmux attach -t dsv4pro-phase1-full</code></pre>
|
||||
<p>
|
||||
<code>all</code> 的真实顺序是：
|
||||
<code>Worker 启动 → Head 启动 → /health → NET/IB 门禁 → fixed → mixed → stop → summarize</code>。
|
||||
</p>
|
||||
|
||||
<h3>11.2 实际执行：Worker 服务</h3>
|
||||
<details>
|
||||
<summary>展开 174.1.51.7 的完整 docker run</summary>
|
||||
<pre><code class="language-bash">docker run -d \
|
||||
--name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker \
|
||||
--gpus all \
|
||||
--network host \
|
||||
--ipc host \
|
||||
--shm-size 20g \
|
||||
--ulimit memlock=-1 \
|
||||
--ulimit stack=67108864 \
|
||||
-v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro \
|
||||
-v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache \
|
||||
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
-e NCCL_SOCKET_IFNAME=eth0 \
|
||||
-e 'NCCL_IB_HCA==mlx5_0:1,mlx5_3:1' \
|
||||
-e NCCL_CROSS_NIC=1 \
|
||||
-e NCCL_DEBUG=INFO \
|
||||
-e SGLANG_SHARED_EXPERT_TP1=1 \
|
||||
--device /dev/infiniband/rdma_cm \
|
||||
--device /dev/infiniband/uverbs0 \
|
||||
--device /dev/infiniband/uverbs3 \
|
||||
--entrypoint python3 \
|
||||
lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 \
|
||||
-m sglang.launch_server \
|
||||
--model-path /data/hf_models/DeepSeek-V4-Pro \
|
||||
--tp-size 16 \
|
||||
--ep-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--dist-init-addr 10.101.0.11:20002 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30002 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--cuda-graph-max-bs-decode 64 \
|
||||
--max-running-requests 256</code></pre>
|
||||
</details>
|
||||
|
||||
<h3>11.3 实际执行：Head 服务</h3>
|
||||
<details>
|
||||
<summary>展开 174.1.51.5 的完整 docker run</summary>
|
||||
<pre><code class="language-bash">docker run -d \
|
||||
--name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head \
|
||||
--gpus all \
|
||||
--network host \
|
||||
--ipc host \
|
||||
--shm-size 20g \
|
||||
--ulimit memlock=-1 \
|
||||
--ulimit stack=67108864 \
|
||||
-v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro \
|
||||
-v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache \
|
||||
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||
-e NCCL_SOCKET_IFNAME=eth0 \
|
||||
-e 'NCCL_IB_HCA==mlx5_0:1,mlx5_3:1' \
|
||||
-e NCCL_CROSS_NIC=1 \
|
||||
-e NCCL_DEBUG=INFO \
|
||||
-e SGLANG_SHARED_EXPERT_TP1=1 \
|
||||
--device /dev/infiniband/rdma_cm \
|
||||
--device /dev/infiniband/uverbs0 \
|
||||
--device /dev/infiniband/uverbs3 \
|
||||
--entrypoint python3 \
|
||||
lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 \
|
||||
-m sglang.launch_server \
|
||||
--model-path /data/hf_models/DeepSeek-V4-Pro \
|
||||
--tp-size 16 \
|
||||
--ep-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--dist-init-addr 10.101.0.11:20002 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30002 \
|
||||
--mem-fraction-static 0.9 \
|
||||
--cuda-graph-max-bs-decode 64 \
|
||||
--max-running-requests 256</code></pre>
|
||||
</details>
|
||||
<p>
|
||||
<code>NCCL_IB_HCA==...</code> 的两个等号不是笔误：第一个是环境变量赋值分隔符，
|
||||
第二个是 NCCL HCA 列表的“精确匹配”前缀。
|
||||
</p>
|
||||
|
||||
<h3>11.4 实际执行：代表 Benchmark</h3>
|
||||
<p>以下是正式 Run 的 <code>128K → 1, C=1</code> 冷 Prefix 命令：</p>
|
||||
<details>
|
||||
<summary>展开完整 sglang.benchmark.serving 命令</summary>
|
||||
<pre><code class="language-bash">timeout --signal=TERM --kill-after=30s 7200s \
|
||||
docker run --rm \
|
||||
--network host \
|
||||
-v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro \
|
||||
-v /data/yy/sskj/dataset/ShareGPT_V3_unfiltered_cleaned_split.json:/data/yy/sskj/dataset/ShareGPT_V3_unfiltered_cleaned_split.json:ro \
|
||||
-v /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1:/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1 \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
--entrypoint python3 \
|
||||
lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 \
|
||||
-m sglang.benchmark.serving \
|
||||
--backend sglang \
|
||||
--host 10.101.0.11 \
|
||||
--port 30002 \
|
||||
--dataset-name random \
|
||||
--dataset-path /data/yy/sskj/dataset/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||
--random-input-len 131072 \
|
||||
--random-output-len 1 \
|
||||
--random-range-ratio 1.0 \
|
||||
--num-prompts 1 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate 10000 \
|
||||
--output-file /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl \
|
||||
--output-details \
|
||||
--disable-tqdm \
|
||||
--warmup-requests 0 \
|
||||
--seed 42 \
|
||||
--flush-cache</code></pre>
|
||||
</details>
|
||||
<p>
|
||||
其余固定点使用同一命令模板，只替换 ISL、OSL、并发、请求数、Warm-up、Seed
|
||||
和输出目录。每个点的最终展开命令保存在自己的 <code>bench_cmd.txt</code>。
|
||||
混合 A/B 的并行启动顺序和两条请求命令见第 6 节及相应 Case 目录。
|
||||
</p>
|
||||
|
||||
<h3>11.5 实际执行：长 Decode 补测</h3>
|
||||
<pre><code class="language-bash">cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||
|
||||
export CASE_IDS="long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1"
|
||||
export RUN_ID="dsv4pro-phase1-long-decode-20260730-234236"
|
||||
trap 'bash run_quick_map.sh stop' EXIT INT TERM
|
||||
bash run_quick_map.sh start
|
||||
bash run_quick_map.sh fixed</code></pre>
|
||||
|
||||
<h3>11.6 停止、清理与检查</h3>
|
||||
<pre><code class="language-bash">cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||
bash run_quick_map.sh stop
|
||||
|
||||
curl -fsS http://10.101.0.11:30002/health || true
|
||||
docker ps --filter name=dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv</code></pre>
|
||||
|
||||
<p>
|
||||
下一阶段：
|
||||
<a class="back" href="./phase2_exp.html">打开 Phase 2 实验档案</a>
|
||||
</p>
|
||||
<p><a class="back" href="./推理优化计划.html">返回推理优化主计划</a></p>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@ -1,167 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Phase 2.5 Code：DSV4-Pro 双机 SGLang RDMA 需求建模</title>
|
||||
<style>
|
||||
:root { --canvas:#eef3f4; --paper:#fff; --ink:#182126; --muted:#5a6970; --line:#d4dee1; --navy:#17363d; --teal:#087c72; --teal-soft:#e8f5f3; --amber:#a64c14; --amber-soft:#fff1e7; --code-bg:#17252b; --code-ink:#eaf2f3; }
|
||||
* { box-sizing:border-box; letter-spacing:0; }
|
||||
body { margin:0; color:var(--ink); background:var(--canvas); font-family:"PingFang SC","Microsoft YaHei",Arial,sans-serif; font-size:16px; line-height:1.72; }
|
||||
header { color:#f6fbfb; background:var(--navy); border-bottom:5px solid #d2692b; }
|
||||
.header-inner,main { width:min(100% - 36px,1120px); margin:0 auto; }
|
||||
.header-inner { padding:34px 0 30px; }
|
||||
h1 { margin:0; font-size:clamp(28px,4vw,42px); line-height:1.25; }
|
||||
h2 { margin:42px 0 15px; padding-bottom:8px; font-size:25px; border-bottom:2px solid #adbbc0; }
|
||||
h3 { margin:28px 0 10px; color:#21454d; font-size:19px; }
|
||||
.eyebrow { margin:0 0 6px; color:#9edbd5; font-size:13px; font-weight:700; }
|
||||
.meta { margin-top:15px; color:#d6e5e7; font-size:14px; }
|
||||
main { margin-top:30px; margin-bottom:70px; padding:38px 48px 58px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
|
||||
a { color:var(--teal); }
|
||||
code { padding:2px 5px; color:#85380d; background:var(--amber-soft); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; overflow-wrap:anywhere; }
|
||||
pre { margin:14px 0 22px; padding:16px 18px; overflow:auto; color:var(--code-ink); background:var(--code-bg); border-radius:5px; font:13px/1.62 "SFMono-Regular",Consolas,monospace; }
|
||||
pre code { padding:0; color:inherit; background:transparent; }
|
||||
table { width:100%; margin:16px 0 26px; border-collapse:collapse; font-size:14px; }
|
||||
th,td { padding:9px 11px; vertical-align:top; text-align:left; border:1px solid var(--line); overflow-wrap:anywhere; }
|
||||
th { color:#153b41; background:#eaf2f2; }
|
||||
.callout { margin:18px 0 26px; padding:14px 18px; background:var(--teal-soft); border-left:4px solid var(--teal); }
|
||||
.warning { margin:18px 0 26px; padding:14px 18px; background:var(--amber-soft); border-left:4px solid var(--amber); }
|
||||
li+li { margin-top:5px; }
|
||||
@media(max-width:760px){main{padding:28px 20px 42px}table{display:block;overflow-x:auto}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">STANDALONE CODE WALKTHROUGH / PHASE 2.5</p>
|
||||
<h1>DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模：代码详解</h1>
|
||||
<div class="meta">实现提交：<code>c5fa700c50c0</code> 正式 Run：<code>dsv4pro-phase2_5-20260801-130007</code> 唯一入口：<code>run_rdma_demand_modeling.sh all</code></div>
|
||||
</div>
|
||||
</header>
|
||||
<main>
|
||||
<p><a href="./推理优化计划.html">返回推理优化主计划</a> · <a href="./phase2_5_exp.html">打开 Phase 2.5 实验档案</a></p>
|
||||
<div class="callout"><strong>边界：</strong>Phase 2.5 不复制模型服务和采集器。它复用 Phase 1 的双机 TP16 服务/benchmark 与 Phase 2 的精确窗口、GPU/RDMA 采集能力，只新增“并发 Scout → 自动选点 → 业务 OSL Confirm → 需求拟合”这一层编排和分析。</div>
|
||||
|
||||
<h2>1. 文件职责与调用关系</h2>
|
||||
<table>
|
||||
<thead><tr><th>文件</th><th>职责</th><th>调用关系</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>config.env</code></td><td>定义 ISL/OSL、Scout 并发、重复次数、平台阈值和 400G 目标</td><td>被唯一入口 source</td></tr>
|
||||
<tr><td><code>run_rdma_demand_modeling.sh</code></td><td>生成场景、调用 Phase 2、串行运行 Scout/Confirm、清理服务</td><td>唯一人工入口</td></tr>
|
||||
<tr><td><code>rdma_demand_model.py</code></td><td>对齐 benchmark/HCA 窗口，计算 bytes/token，拟合平台并生成报告</td><td>Scout 后选点；Confirm 后最终汇总</td></tr>
|
||||
<tr><td><code>test_rdma_demand_model.py</code></td><td>覆盖 HCA counter 单位、平台选择、线性换算和拟合输出</td><td>本地/CI 回归测试</td></tr>
|
||||
<tr><td>Phase 2 <code>run_hardware_contention_attribution.sh</code></td><td>启动服务、采集 Head/Worker、切正式测量窗</td><td>由 Phase 2.5 以环境变量调用</td></tr>
|
||||
<tr><td>Phase 1 <code>run_quick_map.sh</code></td><td>双机服务启停与 SGLang benchmark</td><td>由 Phase 2 内部复用</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<pre><code>run_rdma_demand_modeling.sh all
|
||||
├─ validate_config + write_manifest
|
||||
├─ run_scout
|
||||
│ ├─ write_scenario_file(64K→1, C=1/4/16/32/64)
|
||||
│ ├─ Phase 2 all（服务 + 18 个采集器 + 5 Case）
|
||||
│ └─ rdma_demand_model.py scout → recommendation.env
|
||||
├─ run_confirm
|
||||
│ ├─ 读取自动选择的 C=4/16/64
|
||||
│ ├─ Phase 2 all（重启服务 + 18 个采集器 + 每点 2 次）
|
||||
│ └─ rdma_demand_model.py final
|
||||
└─ Phase 2 stop → 双节点清理</code></pre>
|
||||
|
||||
<h2>2. 配置层</h2>
|
||||
<p><code>config.env:5-7</code> 通过相对路径找到 Phase 1/2，不依赖执行命令所在目录。<code>config.env:10-16</code> 定义 64K Scout 与 1K Confirm；<code>18-21</code> 定义 5% 平台阈值、400G 物理目标和 360G 实用目标。</p>
|
||||
<pre><code>ISL=65536
|
||||
SCOUT_OSL=1
|
||||
CONFIRM_OSL=1024
|
||||
SCOUT_CONCURRENCIES="1 4 16 32 64"
|
||||
SCOUT_REPETITIONS=1
|
||||
CONFIRM_REPETITIONS=2
|
||||
PLATEAU_GAIN_PCT=5
|
||||
TARGET_RAIL_GBPS=400
|
||||
PRACTICAL_RAIL_GBPS=360</code></pre>
|
||||
<p><code>SAMPLE_INTERVAL_S=1</code> 只决定 HCA/GPU 时间序列分辨率；<code>SCENARIO_TIMEOUT_S=7200</code> 是单个 benchmark 的保护上限，不是期望耗时。</p>
|
||||
|
||||
<h2>3. Shell 唯一入口</h2>
|
||||
<h3>3.1 参数检查与场景生成</h3>
|
||||
<p><code>run_rdma_demand_modeling.sh:42-71</code> fail-fast 检查依赖脚本、整数参数和并发列表。<code>73-100</code> 生成 Phase 2 能读取的 TSV，并为每个形状生成稳定的 case id。</p>
|
||||
<h3>3.2 复用 Phase 2，而不是复制采集代码</h3>
|
||||
<p><code>103-134</code> 构造一个数组命令，把场景、Case、重复次数和采样周期作为环境变量传给 Phase 2。它显式关闭 mixed case 与通信 microbenchmark，因为 Phase 2.5 只测模型 RDMA 需求，不重复已完成的硬件基线。</p>
|
||||
<pre><code>RUN_MIXED_CASE=0
|
||||
RUN_COMMUNICATION_BASELINE=0
|
||||
SCENARIO_FILE=.../scout.tsv
|
||||
FIXED_CASE_IDS=rdma_scout_...
|
||||
bash run_hardware_contention_attribution.sh all</code></pre>
|
||||
<h3>3.3 两阶段控制流</h3>
|
||||
<p><code>136-152</code> 跑完 Scout 后立即调用 Python，并写出 <code>recommendation.env</code>；<code>154-180</code> 读取推荐并发，生成 64K→1K Confirm。<code>217-225</code> 的 <code>run_all</code> 严格串行执行，异常信号触发 stop 清理。</p>
|
||||
<div class="warning"><strong>为什么服务会启动两次：</strong>Scout 结束后 Phase 2 会清理服务；Confirm 使用全新的 Prefix Cache、采集器和服务生命周期，避免 Scout 状态污染确认结果。</div>
|
||||
|
||||
<h2>4. Python 如何从计数器变成需求模型</h2>
|
||||
<h3>4.1 精确时间窗与 HCA 单位</h3>
|
||||
<p><code>rdma_demand_model.py:87-115</code> 以 <code>(case_id,repetition)</code> 对齐 benchmark、窗口和 RDMA 汇总。<code>117-147</code> 在正式窗口内计算相邻 HCA counter 的速率；IB <code>port_*_data</code> 单位是 4-octet，因此必须乘 4，再乘 8 转为 bit/s。</p>
|
||||
<pre><code>gbps = (counter_delta × 4 bytes × 8 bits) / duration_s / 1e9</code></pre>
|
||||
<h3>4.2 单 Case 指标</h3>
|
||||
<p><code>171-281</code> 汇总四条观测边（Head/Worker × 两个 HCA）的 Rail Mean/P95/Max、双 Rail 单向合计、Rail 不均衡、错误计数和 GPU 利用率。通信强度按每条 Rail 平均发送字节计算：</p>
|
||||
<pre><code>bytes_per_input_token_per_rail
|
||||
= mean(head/worker × mlx5_0/mlx5_3 xmit_bytes)
|
||||
/ total_input_tokens</code></pre>
|
||||
<p>这里不把 TX+RX 相加，因为那会把同一份跨机数据重复计数。</p>
|
||||
|
||||
<h3>4.3 平台、拐点与自动选点</h3>
|
||||
<p><code>362-392</code> 比较相邻并发点。只有 Rail Mean 与 Input TPS 增益同时低于 5%，当前点才是平台候选。随后选择平台前一点、平台点和最高稳定点；本 Run 得到 <code>4 16 64</code>。</p>
|
||||
<pre><code>if bandwidth_gain < 5% and input_tps_gain < 5%:
|
||||
plateau_c = current_concurrency</code></pre>
|
||||
|
||||
<h3>4.4 线性通信强度与饱和曲线</h3>
|
||||
<p><code>352-360</code> 用过原点线性斜率拟合 <code>rail_gbps/input_tps</code>，再还原为 bytes/token。<code>318-350</code> 用双曲线 <code>B(C)=B∞×C/(K+C)</code> 拟合并发饱和曲线；<code>394-447</code> 组合两者，判断模型计算或网络谁先到平台。</p>
|
||||
<pre><code>required_input_tps
|
||||
= target_rail_gbps / linear_gbps_per_input_tps
|
||||
|
||||
if fitted_bandwidth_asymptote < 360:
|
||||
verdict = COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION</code></pre>
|
||||
<p>本 Run 的 RMSE 为 0.277 Gbit/s，五个 Scout 点与饱和曲线贴合良好；拟合上限 80.32 Gbit/s，与 C=16/32/64 的 79.90/79.93/79.97 一致。</p>
|
||||
|
||||
<h2>5. 输出文件如何阅读</h2>
|
||||
<table>
|
||||
<thead><tr><th>输出</th><th>用途</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>rdma_case_metrics.csv</code></td><td>每次重复的 benchmark + GPU + Rail 对齐数据，是审计主表</td></tr>
|
||||
<tr><td><code>rdma_demand_model.json</code></td><td>完整拟合参数、平台点、目标 TPS 和最终 verdict</td></tr>
|
||||
<tr><td><code>rdma_demand_report.md</code></td><td>面向人的 Scout/Confirm 摘要</td></tr>
|
||||
<tr><td><code>recommendation.env</code></td><td>Shell 可直接 source 的 Confirm 并发列表</td></tr>
|
||||
<tr><td><code>{scout,confirm}/case_windows.csv</code></td><td>每个正式 benchmark 的精确起止时间</td></tr>
|
||||
<tr><td><code>{scout,confirm}/{head,worker}/rdma.csv</code></td><td>原始 HCA counter 时间序列，仅保留在服务器完整结果中</td></tr>
|
||||
<tr><td><code>commands/*.cmd.txt</code></td><td>实际传给 Phase 2 的完整可复现命令</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>6. 测试与验收门槛</h2>
|
||||
<ul>
|
||||
<li><code>python3 -m unittest test_rdma_demand_model.py</code>：4/4 通过。</li>
|
||||
<li><code>bash -n run_rdma_demand_modeling.sh</code> 与 Python compile：通过。</li>
|
||||
<li><code>DRY_RUN=1 ... all</code>：展开 5 个 Scout 和 3×2 个 Confirm，不启动服务。</li>
|
||||
<li>正式 Run：Scout 5/5、Confirm 6/6；两个阶段各 18/18 个采集器正常启停，共保存 72 条 STARTED/STOPPED 生命周期事件。</li>
|
||||
<li>11 个测量结果全部 <code>COMPLETED</code>，<code>rdma_error_delta=0</code>。</li>
|
||||
<li>结束后 Head/Worker 均无实验容器，16 张 GPU 为 0 MiB / 0%。</li>
|
||||
</ul>
|
||||
|
||||
<h2>7. 行号索引</h2>
|
||||
<table>
|
||||
<thead><tr><th>功能</th><th>文件与行</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>配置与路径</td><td><code>config.env:3-36</code></td></tr>
|
||||
<tr><td>校验与场景生成</td><td><code>run_rdma_demand_modeling.sh:42-100</code></td></tr>
|
||||
<tr><td>Phase 2 调用</td><td><code>run_rdma_demand_modeling.sh:103-134</code></td></tr>
|
||||
<tr><td>Scout / Confirm</td><td><code>run_rdma_demand_modeling.sh:136-180</code></td></tr>
|
||||
<tr><td>唯一 all 与清理</td><td><code>run_rdma_demand_modeling.sh:206-264</code></td></tr>
|
||||
<tr><td>HCA interval rate</td><td><code>rdma_demand_model.py:117-147</code></td></tr>
|
||||
<tr><td>Case 对齐汇总</td><td><code>rdma_demand_model.py:171-281</code></td></tr>
|
||||
<tr><td>饱和拟合</td><td><code>rdma_demand_model.py:318-350</code></td></tr>
|
||||
<tr><td>平台选点</td><td><code>rdma_demand_model.py:362-392</code></td></tr>
|
||||
<tr><td>需求模型与 verdict</td><td><code>rdma_demand_model.py:394-447</code></td></tr>
|
||||
<tr><td>报告输出</td><td><code>rdma_demand_model.py:449-517</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>8. 复用时必须重新标定的边界</h2>
|
||||
<p>这套代码可复用，但 <code>3.332 MB/token/rail</code> 不是通用常数。换模型、量化、TP/EP、节点切分、backend、Prefill/Decode 形状或 Prefix Cache 策略后，都必须重新跑 Scout。代码输出的是“当前部署实现的经验模型”，不是由参数量单独推导出的理论通信量。</p>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@ -1,166 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Phase 2.5：DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模</title>
|
||||
<style>
|
||||
:root { --ink:#18202a; --muted:#5b6570; --line:#d8dde3; --paper:#fff; --page:#f3f5f7; --blue:#1769aa; --green:#16734a; --amber:#9a5a00; --code:#f0f3f6; }
|
||||
* { box-sizing:border-box; letter-spacing:0; }
|
||||
body { margin:0; background:var(--page); color:var(--ink); font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","PingFang SC",sans-serif; font-size:16px; line-height:1.68; }
|
||||
header { background:#202a34; color:#fff; border-bottom:5px solid #34a17a; }
|
||||
.header-inner, main { width:min(1120px,calc(100% - 32px)); margin:0 auto; }
|
||||
.header-inner { padding:36px 0 30px; }
|
||||
h1 { margin:4px 0 12px; font-size:clamp(27px,4vw,42px); line-height:1.2; }
|
||||
h2 { margin:38px 0 12px; padding-bottom:7px; border-bottom:2px solid var(--line); font-size:24px; }
|
||||
h3 { margin:26px 0 8px; font-size:19px; }
|
||||
.eyebrow { margin:0; color:#8fd8bd; font-size:13px; font-weight:700; }
|
||||
.meta { display:flex; flex-wrap:wrap; gap:8px 22px; color:#d7dee5; font-size:14px; }
|
||||
main { margin-top:24px; margin-bottom:48px; padding:30px 38px 42px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
|
||||
a { color:var(--blue); }
|
||||
.back { display:inline-block; margin:0 16px 10px 0; font-weight:650; }
|
||||
code { padding:1px 5px; background:var(--code); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; font-size:.92em; overflow-wrap:anywhere; }
|
||||
pre { overflow-x:auto; padding:14px 16px; background:#202a34; color:#f4f7fa; border-radius:5px; line-height:1.5; }
|
||||
pre code { padding:0; background:transparent; color:inherit; }
|
||||
table { width:100%; margin:14px 0 22px; border-collapse:collapse; font-size:14px; }
|
||||
th,td { padding:10px 11px; border:1px solid var(--line); text-align:left; vertical-align:top; }
|
||||
th { background:#edf1f4; }
|
||||
.status,.decision,.warning { padding:14px 16px; }
|
||||
.status { border-left:4px solid var(--green); background:#eef8f3; }
|
||||
.decision { border-left:4px solid var(--blue); background:#edf5fb; }
|
||||
.warning { border-left:4px solid var(--amber); background:#fff7e7; }
|
||||
.metric { font-size:20px; font-weight:750; color:var(--green); }
|
||||
li+li { margin-top:5px; }
|
||||
@media (max-width:720px) { main{padding:22px 18px 30px} table{display:block;overflow-x:auto;white-space:nowrap} }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">FINAL EXPERIMENT RECORD / PHASE 2.5</p>
|
||||
<h1>DeepSeek-V4-Pro 双机 Pro6000D SGLang：RDMA 需求建模与并发拐点</h1>
|
||||
<div class="meta">
|
||||
<span>Run：dsv4pro-phase2_5-20260801-130007</span>
|
||||
<span>拓扑：TP16 / EP2 / 双 Rail RoCE</span>
|
||||
<span>完成：2026-08-01 15:14:11 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
<main>
|
||||
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||
<a class="back" href="./phase2_5_code.html">打开 Phase 2.5 代码详解</a>
|
||||
|
||||
<p class="status"><strong>阶段已完成。</strong>正式 Run 用时 2 小时 14 分 04 秒；Scout 5/5、Confirm 6/6 成功，两阶段各 18/18 个采集器正常启停。所有测量窗 RDMA 错误增量为 0，结束后两节点容器和 16 张 GPU 均已清理。</p>
|
||||
|
||||
<h2>1. 要回答的问题</h2>
|
||||
<p>Phase 2 只看到代表负载约 83.5 Gbit/s/rail，不能判断继续增加并发是否会逼近 400G。Phase 2.5 专门回答三个问题：</p>
|
||||
<ol>
|
||||
<li>固定模型、TP/EP 和输入形状后，Input TPS 与每 Rail RDMA 带宽是什么关系？</li>
|
||||
<li>并发增加到哪里后，模型吞吐和 RDMA 带宽不再增长？</li>
|
||||
<li>要达到 400G，需要怎样的 Input TPS；当前瓶颈先出现在模型计算还是网络？</li>
|
||||
</ol>
|
||||
|
||||
<h2>2. 实验设计</h2>
|
||||
<table>
|
||||
<thead><tr><th>阶段</th><th>请求形状</th><th>并发</th><th>重复</th><th>目的</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Scout</td><td>64K → 1</td><td>1 / 4 / 16 / 32 / 64</td><td>1</td><td>隔离 Prefill，找吞吐与带宽平台</td></tr>
|
||||
<tr><td>Confirm</td><td>64K → 1K</td><td>自动选择 4 / 16 / 64</td><td>2</td><td>验证真实长输出不会推翻需求模型</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>服务参数沿用 Phase 1/2：SGLang nightly、TP16、EP2、双 Rail <code>mlx5_0/mlx5_3</code>、<code>NET/IB + GDRDMA</code>。每个 Case 使用冷 Prefix，并按 benchmark 正式测量窗口切片 HCA Counter。</p>
|
||||
|
||||
<h2>3. 实际启动命令</h2>
|
||||
<p>只在 Head <code>174.1.51.5</code> 执行，不需要 <code>source</code> 或 <code>conda activate</code>：</p>
|
||||
<pre><code>cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||
|
||||
RUN_ID=dsv4pro-phase2_5-20260801-130007
|
||||
tmux new-session -d -s dsv4pro-phase2_5 \
|
||||
"RUN_ID=${RUN_ID} bash run_rdma_demand_modeling.sh all \
|
||||
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||
|
||||
tmux attach -t dsv4pro-phase2_5</code></pre>
|
||||
<p>实际展开后的 Scout/Confirm 命令分别保存在结果目录的 <code>commands/scout.cmd.txt</code> 与 <code>commands/confirm.cmd.txt</code>。</p>
|
||||
|
||||
<h2>4. Scout 结果：并发 16 已进入平台</h2>
|
||||
<table>
|
||||
<thead><tr><th>C</th><th>Input TPS</th><th>Rail Mean</th><th>Rail P95</th><th>Rail Max</th><th>双 Rail 单向合计</th><th>MB/input-token/rail</th><th>GPU Util</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>1</td><td>2,709.64</td><td>70.97 Gbit/s</td><td>85.60</td><td>86.81</td><td>141.94</td><td>3.138</td><td>93.75%</td></tr>
|
||||
<tr><td>4</td><td>2,930.49</td><td>78.27 Gbit/s</td><td>86.82</td><td>88.86</td><td>156.55</td><td>3.298</td><td>97.45%</td></tr>
|
||||
<tr><td>16</td><td>2,983.77</td><td>79.90 Gbit/s</td><td>86.42</td><td>88.20</td><td>159.79</td><td>3.336</td><td>99.33%</td></tr>
|
||||
<tr><td>32</td><td>2,983.92</td><td>79.93 Gbit/s</td><td>86.23</td><td>88.22</td><td>159.86</td><td>3.344</td><td>99.49%</td></tr>
|
||||
<tr><td>64</td><td>2,991.28</td><td>79.97 Gbit/s</td><td>85.93</td><td>88.32</td><td>159.95</td><td>3.340</td><td>99.64%</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="decision"><strong>观察结论：</strong>C=16→32 的 Input TPS 只增长 0.005%，Rail Mean 只增长 0.040%；C=32→64 也仅增长 0.247% / 0.057%。并发 16 已是平台拐点，继续加到 64 只会增加排队和 TTFT，不会增加网络压力。</p>
|
||||
<p><strong>证据：</strong><code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv</code>；原始 HCA 数据位于同一 Run 的 <code>scout/head/rdma.csv</code> 与 <code>scout/worker/rdma.csv</code>，精确窗口位于 <code>scout/case_windows.csv</code>。</p>
|
||||
|
||||
<h2>5. Confirm 结果：加入 1K Decode 后仍由计算先饱和</h2>
|
||||
<table>
|
||||
<thead><tr><th>C</th><th>重复</th><th>Input TPS</th><th>Output TPS</th><th>Rail Mean</th><th>Rail P95</th><th>双 Rail单向合计</th><th>TTFT P95</th><th>TPOT P95</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>4</td><td>2</td><td>2,072.24</td><td>32.38</td><td>57.33 Gbit/s</td><td>86.28</td><td>114.66</td><td>86.36 s</td><td>95.63 ms</td></tr>
|
||||
<tr><td>16</td><td>2</td><td>2,582.13</td><td>40.35</td><td>71.64 Gbit/s</td><td>86.30</td><td>143.29</td><td>336.85 s</td><td>356.24 ms</td></tr>
|
||||
<tr><td>64</td><td>2</td><td>2,588.96</td><td>40.45</td><td>72.19 Gbit/s</td><td>86.69</td><td>144.38</td><td>1,506.50 s</td><td>418.58 ms</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>C=16→64 的 Input TPS 仅增长 0.26%，Rail Mean 仅增长 0.76%，但 TTFT P95 从 336.85 秒升至 1,506.50 秒。对于 64K→1K，<strong>最大有意义并发仍约为 16</strong>；C=64 是容量压力点，不是推荐服务点。</p>
|
||||
<p><strong>证据：</strong>同一 Run 的 <code>confirm/bench_summary.csv</code>、<code>confirm/case_rdma_summary.csv</code>、<code>confirm/case_windows.csv</code>；两轮逐点数据在顶层 <code>rdma_case_metrics.csv</code>。</p>
|
||||
|
||||
<h2>6. 400G 能否被模型负载打满</h2>
|
||||
<p>Scout 的线性比例为：</p>
|
||||
<pre><code>每 Rail 带宽（Gbit/s）
|
||||
≈ Input TPS × 3.332 MB/input-token/rail × 8 ÷ 1e9</code></pre>
|
||||
<table>
|
||||
<thead><tr><th>目标口径</th><th>需要的 Input TPS</th><th>当前约 2,991 TPS 的差距</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>单 Rail 400G</td><td class="metric">15,006 tok/s</td><td>约 5.02×</td></tr>
|
||||
<tr><td>单 Rail 360G（90% 实用线）</td><td>13,505 tok/s</td><td>约 4.51×</td></tr>
|
||||
<tr><td>双 Rail 单向合计 400G</td><td>7,503 tok/s</td><td>约 2.51×</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>拟合得到当前模型负载的单 Rail 渐近上限约 <strong>80.32 Gbit/s</strong>，即物理 400G 的约 20.1%。瞬时 Max 也只有 89.53 Gbit/s。结论不是“网络只能跑 80G”，而是<strong>当前 DSV4-Pro TP16/EP2 实现最多只能产生约 80G/rail 的持续 RDMA 流量</strong>；Phase 2 的 NCCL microbenchmark 已证明链路本身能达到更高通信带宽。</p>
|
||||
<p class="warning"><strong>口径提醒：</strong>400G 是每条 Rail 的线速；双 Rail 单向总量是两条 Rail 的 TX 之和。不要把 TX 与 RX 相加后声称打满，也不要把 NCCL <code>busbw GB/s</code> 与 HCA <code>Gbit/s</code> 直接比较。</p>
|
||||
|
||||
<h2>7. 一套可复用的 RDMA 需求评估方法</h2>
|
||||
<ol>
|
||||
<li><strong>固定部署变量。</strong>记录模型版本、精度/量化、TP/EP/PP/DP、节点数、Attention/MoE backend、chunked prefill 和网卡拓扑。任一项变化都要重新标定。</li>
|
||||
<li><strong>先选 Prefill Scout。</strong>固定 ISL，OSL=1，取稀疏并发点如 1/4/16/32/64；每点清 Prefix Cache，并保证请求文本实际达到目标 token 数。</li>
|
||||
<li><strong>对齐正式测量窗。</strong>从 benchmark 的 main-run 起止时间切片 Head/Worker 的 <code>mlx5_*</code> HCA Counter，不能用整个进程寿命，也不能只看 <code>sar eth*</code>。</li>
|
||||
<li><strong>计算通信强度。</strong><code>bytes_per_input_token_per_rail = rail_xmit_bytes / total_input_tokens</code>。这是该模型与并行策略下“每处理一个输入 token，要在一条 Rail 发送多少字节”。</li>
|
||||
<li><strong>找并发平台。</strong>同时观察 Input TPS 和 Rail Mean；连续一点的增益都低于阈值（本实验 5%）时，记为拐点。最大 C 不等于最大有效 C。</li>
|
||||
<li><strong>推导目标吞吐。</strong><code>required_input_tps = target_rail_gbps × 1e9 / (bytes_per_token × 8)</code>。若模型的实测/拟合 TPS 上限远低于该值，网络不会先饱和。</li>
|
||||
<li><strong>用业务 OSL 复测。</strong>在平台前、拐点、最高压力点各重复至少两次，确认 Decode、KV Cache 和调度没有改变结论。</li>
|
||||
<li><strong>最后做链路对照。</strong>模型负载未打满时，用 NCCL microbenchmark 验证网络能力，把“模型产流量不足”与“网络本身跑不满”分开。</li>
|
||||
</ol>
|
||||
|
||||
<h3>7.1 哪些变量会改变 bytes/token 与平台</h3>
|
||||
<table>
|
||||
<thead><tr><th>变量</th><th>可能改变的原因</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>模型架构与层数</td><td>每 token 触发的 TP collective、MoE dispatch/combine 和激活尺寸不同</td></tr>
|
||||
<tr><td>TP / EP / PP / DP</td><td>通信参与 rank、跨机边界、collective 类型和频率改变</td></tr>
|
||||
<tr><td>Prefill / Decode、ISL / OSL</td><td>计算强度、chunk 调度、KV 访问和 collective 消息粒度不同</td></tr>
|
||||
<tr><td>并发与 batch</td><td>决定 kernel/batch 效率和 Input TPS；超过平台后只增加排队</td></tr>
|
||||
<tr><td>量化与 backend</td><td>改变计算速度；通信字节可能不同比例变化，因此会移动“计算先饱和还是网络先饱和”的边界</td></tr>
|
||||
<tr><td>Prefix Cache</td><td>命中会绕过大量 Prefill，必须单独作为另一类业务场景建模</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>8. 最终结论</h2>
|
||||
<p class="decision"><strong>在两台 Pro6000D、DSV4-Pro、SGLang TP16/EP2 的当前实现中，RDMA 不是吞吐瓶颈。</strong>64K Prefill 在 C=16 已达到约 3K input tok/s 和 80 Gbit/s/rail 的平台；继续增加并发到 64 不会显著增加吞吐或带宽，只会令 TTFT 急剧上升。要打满单 Rail 400G，模型侧 Input TPS 需提高到约 15K，约为当前上限 5 倍。因此后续优化应先看 GPU Kernel、MoE/Attention 执行和 rank 同步，而不是扩容计算网。</p>
|
||||
|
||||
<h2>9. 证据与清理</h2>
|
||||
<ul>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_report.md">自动 RDMA 需求报告</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_model.json">机器可读模型</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv">全部逐点指标</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/run_manifest.txt">Run Manifest</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/head_server_cmd.txt">Scout Head 服务命令</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_server_cmd.txt">Worker 服务命令</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_nccl_transport.log">Confirm Head NCCL 路径</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_nccl_transport.log">Worker NCCL 路径</a></li>
|
||||
</ul>
|
||||
<p>Worker 在最后一个 Case 完成后随 Head 主动关闭进程组出现 Gloo peer-close Traceback；它发生在测量结束与结果落盘之后，不是实验失败。最终 tmux、服务容器和 GPU 进程均已退出。</p>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@ -1,435 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="color-scheme" content="light">
|
||||
<title>Phase 2 Code：DSV4-Pro 双机 Pro6000D SGLang 硬件竞争归因</title>
|
||||
<style>
|
||||
:root {
|
||||
--canvas: #eef3f4;
|
||||
--paper: #ffffff;
|
||||
--ink: #182126;
|
||||
--muted: #5a6970;
|
||||
--line: #d4dee1;
|
||||
--navy: #17363d;
|
||||
--teal: #087c72;
|
||||
--teal-soft: #e8f5f3;
|
||||
--amber: #a64c14;
|
||||
--amber-soft: #fff1e7;
|
||||
--code-bg: #17252b;
|
||||
--code-ink: #eaf2f3;
|
||||
}
|
||||
* { box-sizing: border-box; letter-spacing: 0; }
|
||||
html { scroll-behavior: smooth; }
|
||||
body {
|
||||
margin: 0;
|
||||
color: var(--ink);
|
||||
background: var(--canvas);
|
||||
font-family: "PingFang SC", "Microsoft YaHei", "Noto Sans CJK SC", Arial, sans-serif;
|
||||
font-size: 16px;
|
||||
line-height: 1.72;
|
||||
}
|
||||
header { color: #f6fbfb; background: var(--navy); border-bottom: 5px solid #d2692b; }
|
||||
.header-inner, main { width: min(100% - 36px, 1120px); margin: 0 auto; }
|
||||
.header-inner { padding: 34px 0 30px; }
|
||||
.eyebrow { margin: 0 0 6px; color: #9edbd5; font-size: 13px; font-weight: 700; }
|
||||
h1 { margin: 0; font-size: clamp(28px, 4vw, 42px); line-height: 1.25; }
|
||||
.meta { margin-top: 15px; color: #d6e5e7; font-size: 14px; }
|
||||
main {
|
||||
margin-top: 30px;
|
||||
margin-bottom: 70px;
|
||||
padding: 38px 48px 58px;
|
||||
background: var(--paper);
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 6px;
|
||||
box-shadow: 0 12px 30px rgba(27, 45, 51, 0.07);
|
||||
}
|
||||
h2 {
|
||||
margin: 46px 0 15px;
|
||||
padding-bottom: 8px;
|
||||
font-size: 25px;
|
||||
line-height: 1.35;
|
||||
border-bottom: 2px solid #adbbc0;
|
||||
}
|
||||
h2:first-of-type { margin-top: 18px; }
|
||||
h3 { margin: 29px 0 10px; color: #21454d; font-size: 19px; }
|
||||
h4 { margin: 22px 0 8px; font-size: 16px; }
|
||||
p, ul, ol { margin-top: 0; margin-bottom: 16px; }
|
||||
li + li { margin-top: 5px; }
|
||||
a { color: var(--teal); text-underline-offset: 3px; }
|
||||
code {
|
||||
padding: 2px 5px;
|
||||
color: #85380d;
|
||||
background: var(--amber-soft);
|
||||
border-radius: 3px;
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
pre {
|
||||
margin: 14px 0 22px;
|
||||
padding: 16px 18px;
|
||||
overflow: auto;
|
||||
color: var(--code-ink);
|
||||
background: var(--code-bg);
|
||||
border-radius: 5px;
|
||||
font: 13px/1.62 "SFMono-Regular", Consolas, monospace;
|
||||
}
|
||||
pre code { padding: 0; color: inherit; background: transparent; }
|
||||
table { width: 100%; margin: 16px 0 26px; border-collapse: collapse; font-size: 14px; }
|
||||
th, td {
|
||||
padding: 9px 11px;
|
||||
vertical-align: top;
|
||||
text-align: left;
|
||||
border: 1px solid var(--line);
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
th { color: #153b41; background: #eaf2f2; }
|
||||
tbody tr:nth-child(even) { background: #fafcfc; }
|
||||
.callout { margin: 18px 0 26px; padding: 14px 18px; background: var(--teal-soft); border-left: 4px solid var(--teal); }
|
||||
.warning { margin: 18px 0 26px; padding: 14px 18px; background: var(--amber-soft); border-left: 4px solid var(--amber); }
|
||||
.toc { columns: 2; column-gap: 38px; margin: 16px 0 24px; padding-left: 22px; }
|
||||
.toc li { break-inside: avoid; }
|
||||
.path { font-family: "SFMono-Regular", Consolas, monospace; font-size: 13px; }
|
||||
footer { margin-top: 48px; padding-top: 18px; color: var(--muted); border-top: 1px solid var(--line); }
|
||||
@media (max-width: 760px) {
|
||||
main { padding: 28px 20px 42px; }
|
||||
.toc { columns: 1; }
|
||||
table { display: block; overflow-x: auto; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">Standalone Code Walkthrough / Phase 2</p>
|
||||
<h1>DSV4-Pro 双机 Pro6000D SGLang 硬件竞争归因：代码详解</h1>
|
||||
<div class="meta">
|
||||
行号基线：<code>39fc2ba565a3</code>
|
||||
完成时间：2026-07-31 17:22:20 CST
|
||||
唯一入口：<code>run_hardware_contention_attribution.sh all</code>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<p>
|
||||
<a href="./推理优化计划.html">返回推理优化主计划</a> ·
|
||||
<a href="./phase2_exp.html">打开 Phase 2 实验档案</a>
|
||||
</p>
|
||||
<div class="callout">
|
||||
<strong>文档边界：</strong>本文只解释提交 <code>39fc2ba565a3</code> 的 Phase 2
|
||||
代码和文件调用关系。Phase 1 负责模型服务与请求；Phase 2 负责通信基线、
|
||||
两节点监控、精确时间切片和逐指标报告。最终正式 Run
|
||||
<code>dsv4pro-phase2-20260731-163620</code> 使用提交
|
||||
<code>5f24b7d22f98108f6cc234edba6768d55ea0a962</code>，代码树包含本页所述修复。
|
||||
</div>
|
||||
|
||||
<h2 id="read">1. 阅读导航</h2>
|
||||
<ul class="toc">
|
||||
<li><a href="#flow">总体控制流</a></li>
|
||||
<li><a href="#files">文件职责与调用关系</a></li>
|
||||
<li><a href="#config">配置来源</a></li>
|
||||
<li><a href="#communication">通信微基准</a></li>
|
||||
<li><a href="#collectors">采集器实现</a></li>
|
||||
<li><a href="#alignment">精确测量窗口</a></li>
|
||||
<li><a href="#report">逐指标报告</a></li>
|
||||
<li><a href="#index">函数行号索引</a></li>
|
||||
</ul>
|
||||
|
||||
<h2 id="flow">2. 总体控制流</h2>
|
||||
<pre><code>main "$@" → run_all
|
||||
├─ validate_config
|
||||
├─ preflight_node_tools
|
||||
│ └─ 两节点 dcgmi discovery -l 必须成功
|
||||
├─ preflight_clock_sync + preflight_gpus_idle
|
||||
├─ run_communication_baseline
|
||||
│ ├─ 两节点 CUDA P2P 全矩阵
|
||||
│ ├─ 两节点各自 8-rank AllReduce
|
||||
│ └─ 16-rank AllReduce，CROSS_NIC=0/1/2
|
||||
├─ start_service → Phase 1 start
|
||||
├─ capture_static_snapshots before
|
||||
├─ start_collectors → Head/Worker 同时采集
|
||||
├─ idle → fixed cases → mixed A/B → cooldown
|
||||
├─ check_collectors + stop_collectors
|
||||
├─ capture_static_snapshots after
|
||||
├─ stop_service → Phase 1 stop
|
||||
├─ summarize_results
|
||||
│ └─ 按正式 benchmark 窗口生成第 5 节逐项数据表
|
||||
└─ finish_manifest</code></pre>
|
||||
<p>
|
||||
<code>all</code> 是唯一正式入口。<code>communication</code>、<code>summarize</code>
|
||||
和 <code>stop</code> 是排错/恢复 action，不需要在正常执行前手工调用。
|
||||
</p>
|
||||
|
||||
<h2 id="files">3. 文件职责与调用关系</h2>
|
||||
<table>
|
||||
<thead><tr><th>文件</th><th>行数</th><th>职责</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="path">config.env</td><td>61</td><td>节点、Case、分层采样周期、通信尺寸、NCCL 选择和 fail-closed 策略。</td></tr>
|
||||
<tr><td class="path">run_hardware_contention_attribution.sh</td><td>1036</td><td>唯一 Shell 编排器：预检、通信文件分发、通信基线、Phase 1 委托、采集器、Case 和清理。</td></tr>
|
||||
<tr><td class="path">communication_baseline.py</td><td>227</td><td>CUDA P2P 全矩阵及 PyTorch/NCCL AllReduce 正确性、延迟和带宽测试。</td></tr>
|
||||
<tr><td class="path">hardware_contention_attribution.py</td><td>1480</td><td>解析所有原始采集器，按 Case 切片，聚合通信并生成 CSV/JSON/report.md。</td></tr>
|
||||
<tr><td class="path">tests/test_hardware_contention_attribution.py</td><td>322</td><td>9 项纯 Python 单元测试，覆盖 worker 无仓库依赖、精确窗口、解析器、RDMA 单位和通信聚合。</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<pre><code>用户
|
||||
└─ Phase2/run_hardware_contention_attribution.sh all
|
||||
├─ source Phase2/config.env
|
||||
├─ docker/torchrun → Phase2/communication_baseline.py
|
||||
├─ env ... bash Phase1/run_quick_map.sh start/fixed/mixed/stop
|
||||
│ └─ Phase1/quick_map_results.py 写 benchmark meta
|
||||
├─ Shell 采集 Head/Worker 原始时间序列
|
||||
└─ Phase2/hardware_contention_attribution.py summarize
|
||||
├─ 读取 Phase1 bench/cases/*/meta.json
|
||||
├─ 读取 Head/Worker 原始监控
|
||||
├─ 读取 communication/COMM_RESULT
|
||||
└─ 输出逐 Case、逐节点、逐指标表和 report.md</code></pre>
|
||||
|
||||
<h3>3.1 Phase 1 与 Phase 2 的边界</h3>
|
||||
<table>
|
||||
<thead><tr><th>问题</th><th>由哪个文件负责</th><th>证据</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>模型路径、镜像、TP16、EP、显存比例</td><td>Phase 1 <code>config.env</code> + <code>run_quick_map.sh</code></td><td><code>service/head_server_cmd.txt</code>、<code>worker_server_cmd.txt</code></td></tr>
|
||||
<tr><td>ISL/OSL/C、random 请求和 mixed A/B</td><td>Phase 1 场景表与 benchmark 函数</td><td><code>bench/*/bench_cmd.txt</code>、<code>bench.json</code></td></tr>
|
||||
<tr><td>通信基线、监控周期、Case 选择</td><td>Phase 2 <code>config.env</code></td><td>Phase 2 <code>manifest.json</code></td></tr>
|
||||
<tr><td>硬件归因和数值报告</td><td>Phase 2 Python 汇总器</td><td><code>case_*_summary.csv</code>、<code>report.md</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2 id="config">4. 配置来源</h2>
|
||||
<table>
|
||||
<thead><tr><th>行号</th><th>配置组</th><th>关键变量</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>config.env:L3-L16</code></td><td>入口与节点</td><td><code>PHASE1_ENTRY</code>、Head/Worker、端口和容器名。</td></tr>
|
||||
<tr><td><code>L18-L21</code></td><td>诊断 Case</td><td>五个 fixed Case、mixed A/B 开关。</td></tr>
|
||||
<tr><td><code>L23-L39</code></td><td>采样与严格性</td><td>GPU/DCGM/RDMA 1 秒；CPU/进程/网络/NUMA/perf 5 秒；精确窗口和采集器 fail-closed。</td></tr>
|
||||
<tr><td><code>L40-L55</code></td><td>通信基线</td><td>镜像、消息尺寸、迭代次数、P2P 大小、CROSS_NIC 列表、Socket/HCA。</td></tr>
|
||||
<tr><td><code>L57-L61</code></td><td>路径与模式</td><td><code>RESULT_BASE</code>、Runtime、Dry-run、是否允许部分采集器。</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
<code>MEM_FRACTION_STATIC</code> 不在 Phase 2 重复定义。它仍来自 Phase 1，
|
||||
最终展开为 SGLang 的 <code>--mem-fraction-static</code>。判断某次 Run 的真实值，
|
||||
应读取 <code>service/head_server_cmd.txt</code>，不能只看默认配置。
|
||||
</p>
|
||||
|
||||
<h2 id="communication">5. 通信微基准</h2>
|
||||
<h3>5.1 Shell 如何编排</h3>
|
||||
<p>
|
||||
<code>run_hardware_contention_attribution.sh:L281-L513</code> 负责源码暂存、
|
||||
Docker 命令、两节点同步和清理。所有命令先写入 <code>commands/*.txt</code>：
|
||||
</p>
|
||||
<ul>
|
||||
<li><code>L281-L319</code>：从 Head 将当次通信脚本暂存到两节点并校验 SHA256。</li>
|
||||
<li><code>L321-L374</code>：构造容器命令，Head/Worker 各跑一次 P2P。</li>
|
||||
<li><code>L376-L404</code>：Head/Worker 各跑一次 8-rank AllReduce。</li>
|
||||
<li><code>L406-L467</code>：每个 CROSS_NIC 值先启动 Worker rank，再运行 Head rank。</li>
|
||||
<li><code>L469-L513</code>：只清理本实验前缀的通信容器和本次 `/tmp` 暂存目录。</li>
|
||||
</ul>
|
||||
<p>
|
||||
Docker 使用和 SGLang 一致的 CUDA 13 nightly 镜像，并显式透传
|
||||
<code>rdma_cm</code>、<code>uverbs0</code>、<code>uverbs3</code>。
|
||||
<code>NCCL_DEBUG=INFO</code> 只在微基准中打开，用于证明 NET/IB/GDRDMA 路径。
|
||||
Worker 不要求存在 Git 仓库；容器只读挂载自动分发的
|
||||
<code>/tmp/.../<RUN_ID>/communication_baseline.py</code>。结果目录同时保存
|
||||
当次源码副本和 SHA256，避免两个节点 checkout 不一致造成版本漂移。
|
||||
</p>
|
||||
|
||||
<h3>5.2 P2P 代码</h3>
|
||||
<p>
|
||||
<code>communication_baseline.py:L45-L106</code> 遍历所有源 GPU 和目标 GPU，
|
||||
先调用 <code>torch.cuda.can_device_access_peer</code>，再对 256 MiB FP16 Tensor
|
||||
做预热和 CUDA Event 计时。输出包括方向、P50/P95 latency 和 GB/s。
|
||||
汇总器按拓扑拆成同 PCIe Switch 的 PIX 与跨 NUMA 的 SYS。
|
||||
</p>
|
||||
|
||||
<h3>5.3 AllReduce 代码</h3>
|
||||
<p>
|
||||
<code>communication_baseline.py:L107-L198</code> 初始化 NCCL process group，
|
||||
对 1 MiB、64 MiB、1 GiB 分别预热和重复测量。每轮先把各 rank latency
|
||||
gather 到 rank 0，使用最慢 rank 作为 collective 完成时间，并检查归约结果：
|
||||
</p>
|
||||
<pre><code>algbw = message_bytes / latency
|
||||
busbw = algbw × 2 × (world_size - 1) / world_size
|
||||
wrong_values = count(output != expected_sum)</code></pre>
|
||||
<p>
|
||||
这样不会用某个提前返回 rank 的时间美化结果；<code>wrong_values=0</code>
|
||||
才算正确完成。
|
||||
</p>
|
||||
|
||||
<h2 id="collectors">6. 两节点采集器</h2>
|
||||
<h3>6.1 启动前门禁</h3>
|
||||
<p>
|
||||
Shell <code>L67-L199</code> 完成配置、工具、时钟和 GPU 空闲检查。
|
||||
<code>preflight_node_tools</code> 不只检查 <code>dcgmi</code> 文件存在，
|
||||
还实际运行 <code>dcgmi discovery -l</code>；两节点任一 Host Engine 不可用即退出。
|
||||
</p>
|
||||
|
||||
<h3>6.2 采集器包装</h3>
|
||||
<p>
|
||||
<code>start_stream_collector</code> 位于 Shell <code>L517-L551</code>。
|
||||
它保存完整命令、PID、唯一进程 tag 和日志；<code>check_collectors</code> 在
|
||||
<code>L722-L740</code> 检查采集器是否提前退出，默认不允许部分成功。
|
||||
</p>
|
||||
|
||||
<table>
|
||||
<thead><tr><th>采集器</th><th>Shell 位置</th><th>周期</th><th>输出</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>nvidia-smi</code></td><td><code>L552-L565</code></td><td>1 秒</td><td><code>gpu_samples.csv</code></td></tr>
|
||||
<tr><td>RDMA HCA counters</td><td><code>L566-L592</code></td><td>1 秒</td><td><code>rdma.csv</code></td></tr>
|
||||
<tr><td>DCGM</td><td><code>L645-L655</code></td><td>1 秒</td><td><code>dcgm_dmon.log</code></td></tr>
|
||||
<tr><td><code>mpstat</code></td><td><code>L656-L663</code></td><td>5 秒</td><td><code>mpstat.log</code></td></tr>
|
||||
<tr><td><code>pidstat -durw</code></td><td><code>L664-L671</code></td><td>5 秒，进程级</td><td><code>pidstat.log</code></td></tr>
|
||||
<tr><td><code>sar -n DEV,EDEV</code></td><td><code>L672-L678</code></td><td>5 秒</td><td><code>sar_net.log</code></td></tr>
|
||||
<tr><td><code>perf stat</code></td><td><code>L680-L689</code></td><td>5 秒</td><td><code>perf_stat.log</code></td></tr>
|
||||
<tr><td><code>numastat</code></td><td><code>L618-L644</code></td><td>5 秒</td><td><code>numa_samples.csv</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
CPU、进程、perf 和 sar 的每行均由 Shell 增加
|
||||
<code>wall_time_ns TAB node TAB payload</code>。NUMA 直接转成结构化 CSV，
|
||||
避免旧版线程级 1 秒日志过大，也让所有指标能按 Case 切片。
|
||||
</p>
|
||||
|
||||
<h2 id="alignment">7. 精确测量窗口</h2>
|
||||
<h3>7.1 Phase 1 如何标记主测量</h3>
|
||||
<p>
|
||||
Phase 1 <code>run_quick_map.sh:L547-L564</code> 每 100 ms 观察 bench 日志；
|
||||
发现 <code>Starting main benchmark run</code> 后调用
|
||||
<code>quick_map_results.py mark-measurement-start</code>。
|
||||
<code>quick_map_results.py:L340-L385</code> 用这个起点和
|
||||
<code>bench.json.duration</code> 生成：
|
||||
</p>
|
||||
<pre><code>measurement_started_at
|
||||
measurement_ended_at
|
||||
measurement_duration_s
|
||||
measurement_window_source = bench_main_marker_plus_duration</code></pre>
|
||||
|
||||
<h3>7.2 Phase 2 如何使用</h3>
|
||||
<p>
|
||||
<code>hardware_contention_attribution.py:L509-L560</code> 优先读取上述字段。
|
||||
只有兼容旧结果时才可能使用进程级窗口；正式配置
|
||||
<code>REQUIRE_PRECISE_WINDOWS=1</code> 会拒绝任何 fallback。
|
||||
<code>L561-L841</code> 对 GPU、DCGM、CPU、进程、perf、NUMA、netdev 和 RDMA
|
||||
使用同一个 <code>started_ns ≤ sample ≤ ended_ns</code> 条件。
|
||||
</p>
|
||||
|
||||
<h2 id="report">8. 逐指标报告</h2>
|
||||
<p>
|
||||
Python <code>summarize</code> 位于
|
||||
<code>hardware_contention_attribution.py:L1036-L1378</code>。
|
||||
它不只生成一个抽象结论，而是按 Phase 2 第 5 节依次写出：
|
||||
</p>
|
||||
<table>
|
||||
<thead><tr><th>指标</th><th>解析函数</th><th>Case 汇总文件</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>GPU</td><td><code>summarize_gpu_rows L377-L413</code></td><td><code>case_gpu_summary.csv</code>、<code>case_gpu_node_summary.csv</code></td></tr>
|
||||
<tr><td>DCGM</td><td><code>parse_dcgm L167-L192</code></td><td><code>case_dcgm_summary.csv</code></td></tr>
|
||||
<tr><td>CPU</td><td><code>parse_mpstat L193-L222</code></td><td><code>case_cpu_summary.csv</code></td></tr>
|
||||
<tr><td>进程</td><td><code>parse_pidstat L223-L289</code></td><td><code>case_process_summary.csv</code></td></tr>
|
||||
<tr><td>perf</td><td><code>parse_perf L290-L310</code></td><td><code>case_perf_summary.csv</code></td></tr>
|
||||
<tr><td>NUMA</td><td>结构化 CSV + <code>summarize_case_metrics</code></td><td><code>case_numa_summary.csv</code></td></tr>
|
||||
<tr><td>Linux netdev</td><td><code>parse_sar_net L311-L358</code></td><td><code>case_netdev_summary.csv</code></td></tr>
|
||||
<tr><td>RDMA</td><td><code>summarize_rdma_rows L424-L484</code></td><td><code>case_rdma_summary.csv</code></td></tr>
|
||||
<tr><td>P2P/NCCL</td><td><code>load_communication_rows</code> + <code>aggregate_communication_rows L842-L928</code></td><td><code>communication_summary.csv</code>、<code>communication_aggregate.csv</code></td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
<code>report.md</code> 对每组都打印有效样本数、Mean/P95/Max、Head/Worker
|
||||
或 Case 间比较和源文件。解析不到的值保留为 <code>-</code>，不会被写成 0。
|
||||
</p>
|
||||
|
||||
<h2 id="outputs">9. 结果目录</h2>
|
||||
<pre><code>results/<RUN_ID>/
|
||||
manifest.json
|
||||
commands/
|
||||
communication/
|
||||
service/
|
||||
bench/<phase1-sub-run>/
|
||||
head/
|
||||
gpu_samples.csv
|
||||
dcgm_dmon.log
|
||||
mpstat.log
|
||||
pidstat.log
|
||||
perf_stat.log
|
||||
sar_net.log
|
||||
numa_samples.csv
|
||||
rdma.csv
|
||||
collector_commands/
|
||||
worker/
|
||||
...同上...
|
||||
case_windows.csv
|
||||
bench_summary.csv
|
||||
case_gpu_summary.csv
|
||||
case_gpu_node_summary.csv
|
||||
case_dcgm_summary.csv
|
||||
case_cpu_summary.csv
|
||||
case_process_summary.csv
|
||||
case_perf_summary.csv
|
||||
case_numa_summary.csv
|
||||
case_netdev_summary.csv
|
||||
case_rdma_summary.csv
|
||||
communication_summary.csv
|
||||
communication_aggregate.csv
|
||||
summary.json
|
||||
report.md</code></pre>
|
||||
|
||||
<h2 id="index">10. 函数行号索引</h2>
|
||||
<h3>10.1 Shell 编排器</h3>
|
||||
<table>
|
||||
<thead><tr><th>行号</th><th>函数组</th><th>职责</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>L26-L66</td><td>日志、远端执行、命令证据</td><td>基础设施。</td></tr>
|
||||
<tr><td>L67-L199</td><td>配置、工具、时钟、GPU 空闲门禁</td><td>正式运行前 fail-fast。</td></tr>
|
||||
<tr><td>L200-L268</td><td>Manifest、marker、Phase 1 委托</td><td>运行身份与复用边界。</td></tr>
|
||||
<tr><td>L281-L513</td><td>通信基线</td><td>按 Run 分发源码、P2P、8/16-rank AllReduce、CROSS_NIC A/B 与清理。</td></tr>
|
||||
<tr><td>L448-L516</td><td>服务和静态快照</td><td>启停 Phase 1 双机服务并保存环境。</td></tr>
|
||||
<tr><td>L517-L710</td><td>采集命令与启动</td><td>两节点分层采样。</td></tr>
|
||||
<tr><td>L711-L772</td><td>采集器检查和停止</td><td>fail-closed 与残留清理。</td></tr>
|
||||
<tr><td>L782-L835</td><td>fixed/mixed Case</td><td>代表负载编排。</td></tr>
|
||||
<tr><td>L836-L858</td><td>汇总、Manifest、trap</td><td>结果收口。</td></tr>
|
||||
<tr><td>L859-L928</td><td><code>run_all</code></td><td>完整状态机。</td></tr>
|
||||
<tr><td>L929-L968</td><td>辅助 action 与 main</td><td><code>communication/all/summarize/stop</code> 分发。</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>10.2 Python 文件</h3>
|
||||
<table>
|
||||
<thead><tr><th>文件/行号</th><th>职责</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>communication_baseline.py:L20-L44</code></td><td>尺寸解析、分位数和 JSON 结果协议。</td></tr>
|
||||
<tr><td><code>L45-L106</code></td><td>CUDA P2P 全矩阵。</td></tr>
|
||||
<tr><td><code>L107-L198</code></td><td>NCCL AllReduce 与正确性。</td></tr>
|
||||
<tr><td><code>hardware_contention_attribution.py:L76-L166</code></td><td>时间、CSV、数字统计基础函数。</td></tr>
|
||||
<tr><td><code>L167-L358</code></td><td>DCGM、mpstat、pidstat、perf、sar 解析器。</td></tr>
|
||||
<tr><td><code>L359-L508</code></td><td>通信、GPU、RDMA、bench 读取与汇总。</td></tr>
|
||||
<tr><td><code>L509-L841</code></td><td>精确窗口和全部 Case 指标切片。</td></tr>
|
||||
<tr><td><code>L842-L1035</code></td><td>通信聚合、CSV、Marker、Manifest。</td></tr>
|
||||
<tr><td><code>L1036-L1378</code></td><td>全部输出表和逐指标 <code>report.md</code>。</td></tr>
|
||||
<tr><td><code>L1379-L1480</code></td><td>CLI 子命令。</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2 id="evidence">11. 最终 Run 证据</h2>
|
||||
<p>
|
||||
正式 Run 完成 8/8 benchmark、8/8 精确测量窗口和 18/18 采集器启停。
|
||||
代码产生的各类输出与实验结论一一对应：
|
||||
</p>
|
||||
<ul>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/manifest.json"><code>manifest.json</code></a>：Run 身份、提交、时间和最终状态。</li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/bench_summary.csv"><code>bench_summary.csv</code></a>：8 个端到端结果。</li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/case_windows.csv"><code>case_windows.csv</code></a>：正式 benchmark 精确时间窗。</li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/collector_status.csv"><code>collector_status.csv</code></a>：两节点采集器生命周期。</li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/report.md"><code>report.md</code></a>：逐指标自动报告。</li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/analysis.md"><code>analysis.md</code></a>：阶段归因与 Phase 3 入口。</li>
|
||||
</ul>
|
||||
|
||||
<footer>
|
||||
本文只描述提交 <code>39fc2ba565a3</code>。Nsight Systems、SGLang Profiler 和
|
||||
Kernel Timeline 属于 Phase 3，不加入 Phase 2，避免重复采集和职责混淆。
|
||||
</footer>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@ -1,812 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Phase 2：DeepSeek-V4-Pro 双机 Pro6000D SGLang 硬件与资源竞争归因</title>
|
||||
<style>
|
||||
:root {
|
||||
color-scheme: light;
|
||||
--ink: #18202a;
|
||||
--muted: #5b6570;
|
||||
--line: #d8dde3;
|
||||
--paper: #ffffff;
|
||||
--page: #f3f5f7;
|
||||
--blue: #1769aa;
|
||||
--green: #16734a;
|
||||
--amber: #9a5a00;
|
||||
--red: #a13232;
|
||||
--code: #f0f3f6;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0;
|
||||
background: var(--page);
|
||||
color: var(--ink);
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
font-size: 16px;
|
||||
line-height: 1.65;
|
||||
}
|
||||
header {
|
||||
background: #202a34;
|
||||
color: #fff;
|
||||
border-bottom: 5px solid #34a17a;
|
||||
}
|
||||
.header-inner, main {
|
||||
width: min(1120px, calc(100% - 32px));
|
||||
margin: 0 auto;
|
||||
}
|
||||
.header-inner { padding: 38px 0 32px; }
|
||||
h1, h2, h3 { letter-spacing: 0; }
|
||||
h1 {
|
||||
margin: 4px 0 12px;
|
||||
font-size: clamp(26px, 4vw, 42px);
|
||||
line-height: 1.2;
|
||||
}
|
||||
h2 {
|
||||
margin: 38px 0 12px;
|
||||
padding-bottom: 7px;
|
||||
border-bottom: 2px solid var(--line);
|
||||
font-size: 24px;
|
||||
}
|
||||
h3 { margin: 26px 0 8px; font-size: 19px; }
|
||||
.eyebrow {
|
||||
margin: 0;
|
||||
color: #8fd8bd;
|
||||
font-size: 13px;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
.meta {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px 22px;
|
||||
color: #d7dee5;
|
||||
font-size: 14px;
|
||||
}
|
||||
main {
|
||||
margin-top: 24px;
|
||||
margin-bottom: 48px;
|
||||
padding: 30px 38px 42px;
|
||||
background: var(--paper);
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 6px;
|
||||
}
|
||||
.status {
|
||||
padding: 14px 16px;
|
||||
border-left: 4px solid var(--amber);
|
||||
background: #fff7e7;
|
||||
}
|
||||
.decision {
|
||||
padding: 14px 16px;
|
||||
border-left: 4px solid var(--green);
|
||||
background: #eef8f3;
|
||||
}
|
||||
a { color: var(--blue); }
|
||||
.back {
|
||||
display: inline-block;
|
||||
margin-bottom: 10px;
|
||||
font-weight: 650;
|
||||
}
|
||||
code {
|
||||
padding: 1px 5px;
|
||||
background: var(--code);
|
||||
border-radius: 3px;
|
||||
font-family: "SFMono-Regular", Consolas, monospace;
|
||||
font-size: 0.92em;
|
||||
}
|
||||
pre {
|
||||
overflow-x: auto;
|
||||
padding: 14px 16px;
|
||||
background: #202a34;
|
||||
color: #f4f7fa;
|
||||
border-radius: 5px;
|
||||
line-height: 1.5;
|
||||
}
|
||||
pre code { padding: 0; background: transparent; color: inherit; }
|
||||
table {
|
||||
width: 100%;
|
||||
margin: 14px 0 22px;
|
||||
border-collapse: collapse;
|
||||
font-size: 14px;
|
||||
}
|
||||
th, td {
|
||||
padding: 10px 11px;
|
||||
border: 1px solid var(--line);
|
||||
text-align: left;
|
||||
vertical-align: top;
|
||||
}
|
||||
th { background: #edf1f4; }
|
||||
.pass { color: var(--green); font-weight: 700; }
|
||||
.pending { color: var(--amber); font-weight: 700; }
|
||||
.fail { color: var(--red); font-weight: 700; }
|
||||
li + li { margin-top: 5px; }
|
||||
@media (max-width: 720px) {
|
||||
main { padding: 22px 18px 30px; }
|
||||
table { display: block; overflow-x: auto; white-space: nowrap; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="header-inner">
|
||||
<p class="eyebrow">Design, Implementation & Result Record</p>
|
||||
<h1>Phase 2：DeepSeek-V4-Pro 双机 Pro6000D SGLang 硬件与资源竞争归因</h1>
|
||||
<div class="meta">
|
||||
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||
<span>拓扑：SGLang TP16 / EP2</span>
|
||||
<span>更新：2026-07-31 17:22:20 CST</span>
|
||||
</div>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<main>
|
||||
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||
<a class="back" href="./phase2_code.html">打开 Phase 2 代码详解</a>
|
||||
|
||||
<p class="status">
|
||||
<strong>当前状态：Phase 2 已完成。</strong>
|
||||
最终 Run <code>dsv4pro-phase2-20260731-163620</code> 在 28 分 44 秒内完成
|
||||
8/8 个 benchmark，正式测量窗口 8/8 精确，18/18 个采集器正常启停。
|
||||
Head/Worker 的 DCGM、CPU、NUMA、双 Rail RDMA 和通信微基准证据均有效；
|
||||
实验结束后两节点容器、服务端口和 16 张 GPU 已清理。
|
||||
</p>
|
||||
|
||||
<h2>1. Phase 1 交接结果</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>代表负载</th><th>关键结果</th><th>Phase 2 用途</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>128K → 1，C=1</td><td>Input TPS 2,710.16；TTFT P95 48.344 s</td><td>纯长 Prefill 的计算、显存与通信归因</td></tr>
|
||||
<tr><td>32K → 1，C=16</td><td>Input TPS 3,112.77；TTFT P95 162.087 s</td><td>并发 Prefill 的排队、Chunk 调度与节点均衡</td></tr>
|
||||
<tr><td>1K → 1K，C=32</td><td>Output TPS 461.68；TPOT P95 63.31 ms</td><td>普通 Decode 的 GPU、CPU 与通信基线</td></tr>
|
||||
<tr><td>1K → 4K，C=16</td><td>Output TPS 310.02；TPOT P95 50.33 ms</td><td>持续 Decode、KV 增长和稳态资源占用</td></tr>
|
||||
<tr><td>128K → 1K，C=1</td><td>TTFT P95 49.326 s；TPOT P95 32.24 ms</td><td>分离长 Prefill 与长上下文 Decode 成本</td></tr>
|
||||
<tr><td>1K → 1K，C=32 + 128K 注入</td><td>Output TPS -24.08%；TPOT P95 +66.55%</td><td>Prefill 干扰 Decode 时的硬件资源竞争</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
最终基线已由 Head 与 Worker 日志证明使用
|
||||
<code>mlx5_0/mlx5_3</code> 双 Rail <code>NET/IB + GDRDMA</code>，
|
||||
正式测量请求为冷 Prefix。正式矩阵 12/12、长 Decode 补测 2/2 均成功。
|
||||
Phase 2 保持相同服务配置和请求口径。
|
||||
</p>
|
||||
|
||||
<h2>2. 本阶段的边界</h2>
|
||||
<ul>
|
||||
<li>只测试 SGLang，不测试 vLLM。</li>
|
||||
<li>保留模型、镜像、TP16、EP2、显存比例和已验证的双 Rail NCCL 配置。</li>
|
||||
<li>模型端到端 Case 不启用 Nsight Systems、PyTorch Profiler、NCCL DEBUG 或投机解码；独立通信基线临时启用 NCCL INFO 以保存实际路径证据。</li>
|
||||
<li>不调参，不尝试优化；先获得足以区分瓶颈类别的硬件证据。</li>
|
||||
<li>只重放五个固定代表负载和一组混合 A/B，不重复 Phase 1 全矩阵。</li>
|
||||
<li>采集器从请求开始前启动，到请求结束后停止，不能中途补采后声称完整。</li>
|
||||
</ul>
|
||||
|
||||
<h2>3. 待验证假设</h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>假设</th><th>预期硬件表现</th><th>后续方向</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>DSV4/NSA Prefill Kernel 计算受限</td><td>GPU 持续忙、高功耗和稳定频率；双 Rail 流量不高</td><td>Phase 3 捕获 Kernel 与 Attention/Indexer 时间线</td></tr>
|
||||
<tr><td>权重或激活显存带宽受限</td><td>GPU Memory Utilization 高，SM 指标未必饱和；功耗可能低于纯计算</td><td>补 DCGM/Profiler 的 DRAM Active，再看 Kernel</td></tr>
|
||||
<tr><td>TP16 跨机通信受限</td><td>RoCE 吞吐高或两条 Rail 明显失衡，GPU 出现等待</td><td>NCCL_CROSS_NIC 0/1/2 快速 A/B，随后看 NCCL Timeline</td></tr>
|
||||
<tr><td>CPU Scheduler 或 Kernel Launch 受限</td><td>GPU 利用率锯齿或有空洞，单 CPU 核持续满载</td><td>定位 Scheduler/Tokenizer 线程与 launch gap</td></tr>
|
||||
<tr><td>频率、功耗或温度限制</td><td>P-state、SM Clock 或 Power 持续异常，可能出现节流原因</td><td>修正电源、散热或 Clock Policy 后复测</td></tr>
|
||||
<tr><td>节点或 Rank 不均衡</td><td>两节点或不同 GPU 的利用率、功耗、网络流量存在固定偏差</td><td>检查 NUMA、GPU-NIC 亲和与慢 Rank</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h2>4. 诊断 Run</h2>
|
||||
<ol>
|
||||
<li>确认 16 张 GPU 空闲，先跑两节点 PCIe P2P、单机 8 rank AllReduce 和双机 16 rank AllReduce；双机分别测试 <code>NCCL_CROSS_NIC=0/1/2</code>。</li>
|
||||
<li>保存两节点静态快照：GPU/NIC/NUMA 拓扑、驱动、CUDA、镜像与服务命令。</li>
|
||||
<li>复用 Phase 1 已验证的 <code>run_quick_map.sh start</code> 启动同配置双机服务。</li>
|
||||
<li>在 Head 和 Worker 同时启动 GPU、CPU、网卡与 RDMA 采样，先记录 15 秒空闲基线。</li>
|
||||
<li>依次重放 <code>128K → 1, C=1</code>、<code>32K → 1, C=16</code> 与 <code>1K → 1K, C=32</code>。</li>
|
||||
<li>重放 <code>1K → 4K, C=16</code> 和 <code>128K → 1K, C=1</code>，观察持续与长上下文 Decode。</li>
|
||||
<li>重放 <code>1K → 1K, C=32</code> Control 与 128K Prefill 注入 Treatment，保留相同注入时序。</li>
|
||||
<li>请求结束后继续采样 15 秒，再停止采集器和服务。</li>
|
||||
<li>按时间戳将请求、GPU、CPU 和双 Rail 指标对齐，生成摘要与判定。</li>
|
||||
</ol>
|
||||
<pre><code>idle 15s
|
||||
│ 128K→1 C1 │ 32K→1 C16 │ 1K→1K C32
|
||||
│ 1K→4K C16 │ 128K→1K C1
|
||||
│ Decode Control │ Decode + Prefill
|
||||
cooldown 15s
|
||||
|
||||
Head 与 Worker 的所有采集器覆盖完整诊断窗口。</code></pre>
|
||||
<p>
|
||||
Phase 1 中服务加载约 5 分 30 秒；通信基线、五个固定负载、混合 A/B、
|
||||
静态快照、采样和清理组成一次完整 Phase 2 Run。
|
||||
</p>
|
||||
|
||||
<h3>4.1 你只需要运行的入口</h3>
|
||||
<p class="decision">
|
||||
<strong>操作规则：先在 Worker <code>.7</code> 做一次 DCGM 准备，再只在
|
||||
Head <code>.5</code> 执行 Phase 2 的 <code>all</code>。</strong>
|
||||
不要手工执行 Phase 1 的 <code>start</code> 或 <code>stop</code>。
|
||||
Phase 2 会在内部复用它们，并负责异常退出时的采集器、Head、Worker 清理。
|
||||
</p>
|
||||
<pre><code class="language-bash"># [仅在 Worker 174.1.51.7 执行一次]
|
||||
# 不需要 source、conda activate，也不要在 .7 运行 Phase 2 的 all
|
||||
systemctl start nvidia-dcgm
|
||||
systemctl is-active nvidia-dcgm
|
||||
dcgmi discovery -l
|
||||
|
||||
# 预期：第二条输出 active，第三条列出本机 8 张 GPU
|
||||
|
||||
# [以下仅在 Head 174.1.51.5 执行]
|
||||
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
|
||||
|
||||
# 第一次先展开全部命令，不启动服务、不占用 GPU、不发送请求
|
||||
DRY_RUN=1 RUN_ID=dsv4pro-phase2-dryrun-$(date +%Y%m%d-%H%M%S) \
|
||||
bash run_hardware_contention_attribution.sh all
|
||||
|
||||
# 正式实验：仍然只有同一个 all 入口，tmux 只负责断线后继续运行
|
||||
RUN_ID=dsv4pro-phase2-$(date +%Y%m%d-%H%M%S)
|
||||
tmux new-session -d -s dsv4pro-phase2 \
|
||||
"RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
|
||||
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||
|
||||
tmux attach -t dsv4pro-phase2</code></pre>
|
||||
<p>
|
||||
<code>.7</code> 的三条命令只负责让 Worker DCGM Host Engine 可用；
|
||||
SGLang Worker、其余采集器和结果回收仍由 <code>.5</code> 的唯一入口通过 SSH 管理。
|
||||
若需要机器重启后自动启动 DCGM，应由运维另行决定是否执行
|
||||
<code>systemctl enable nvidia-dcgm</code>。
|
||||
</p>
|
||||
<p><strong><code>all</code> 内部执行顺序：</strong></p>
|
||||
<pre><code>配置、工具、DCGM 与 GPU 空闲门禁
|
||||
→ 通信基线：两节点 P2P、两组单机 8-rank AllReduce、
|
||||
三组双机 16-rank NCCL_CROSS_NIC A/B
|
||||
→ Phase 1 start：启动同配置 TP16 服务
|
||||
→ 两节点静态快照
|
||||
→ 启动两节点采集器并记录 15 秒 idle
|
||||
→ 五个固定 Case
|
||||
→ 混合 Prefill/Decode A/B
|
||||
→ 15 秒 cooldown
|
||||
→ 停止采集器并保存后快照
|
||||
→ Phase 1 stop：停止 Head/Worker
|
||||
→ 生成按第 5 节逐项对应的 CSV、JSON 与 report.md</code></pre>
|
||||
<p>
|
||||
Phase 1 的作用是提供已经验证过的双机 Docker 服务和 Benchmark 实现，
|
||||
不是第二个用户入口。实际展开的服务、Benchmark 和采集命令都会写入
|
||||
<code>results/<RUN_ID>/service/</code>、<code>commands/</code> 和
|
||||
<code>head|worker/collector_commands/</code>，不依赖跨文档猜测。
|
||||
</p>
|
||||
|
||||
<h2>5. 采集指标</h2>
|
||||
<p>
|
||||
本节记录正式实现使用的命令，而不是建议性伪代码。命令由
|
||||
<code>run_hardware_contention_attribution.sh</code> 在 Head 和 Worker 同时启动；
|
||||
每条展开后的命令会另外保存在
|
||||
<code>results/<RUN_ID>/head|worker/collector_commands/</code>。
|
||||
</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>层级</th><th>连续采样</th><th>静态或前后快照</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>GPU</td><td>利用率、Memory Utilization、显存、功耗、SM/Memory Clock、温度、P-state</td><td><code>nvidia-smi topo -m</code>、Compute Process</td></tr>
|
||||
<tr><td>CPU</td><td>每核利用率、上下文切换、服务进程 CPU/内存</td><td>NUMA 拓扑、容器 PID 与 CPU Affinity</td></tr>
|
||||
<tr><td>Network</td><td><code>eth0/eth3</code> RX/TX</td><td><code>ethtool -S</code> 错误计数前后差</td></tr>
|
||||
<tr><td>RDMA</td><td><code>mlx5_0/mlx5_3</code> port_xmit/recv_data 差分</td><td>Port State、GID 与错误计数</td></tr>
|
||||
<tr><td>DCGM</td><td>SM Active、DRAM Active、Tensor Active、PCIe</td><td>工具版本与可用 Field</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>5.1 时间对齐与 Case Marker</h3>
|
||||
<pre><code class="language-bash"># 每条 GPU/RDMA 样本写入相同格式的宿主机墙钟时间
|
||||
date +%s%N
|
||||
|
||||
# 实验前检查两节点秒级时钟差
|
||||
date +%s
|
||||
|
||||
# Case 开始、结束和服务状态由 Python 写入 markers.csv
|
||||
python3 hardware_contention_attribution.py marker \
|
||||
--path markers.csv \
|
||||
--node head \
|
||||
--event case_start \
|
||||
--case-id long_prefill_latency_128k_c1</code></pre>
|
||||
<table>
|
||||
<thead><tr><th>数据</th><th>含义</th><th>为什么需要</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>wall_time_ns</code></td><td>Unix Epoch 纳秒时间</td><td>把 GPU、CPU、RDMA 与 Benchmark 放到同一时间轴</td></tr>
|
||||
<tr><td><code>case_start/case_end</code></td><td>一个 Case 的编排边界</td><td>从整段连续采样中切出对应负载</td></tr>
|
||||
<tr><td><code>CLOCK_SKEW_TOLERANCE_S=2</code></td><td>两节点允许的最大秒级时钟差</td><td>避免 Head/Worker 的同一时刻被错位比较</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
最终实现由 Phase 1 监听 <code>bench.log</code> 中的
|
||||
<code>Starting main benchmark run</code>，立刻写入
|
||||
<code>measurement_start.json</code>；再使用 SGLang <code>bench.json</code>
|
||||
的正式 benchmark duration 计算结束时间。Phase 2 优先读取
|
||||
<code>measurement_started_at/measurement_ended_at</code>，不会把数据生成和
|
||||
Warm-up 混入硬件均值。<code>REQUIRE_PRECISE_WINDOWS=1</code> 时，任何 Case
|
||||
缺少精确窗口都会让汇总失败，而不是悄悄回退。
|
||||
</p>
|
||||
|
||||
<h3>5.2 GPU 基础状态：<code>nvidia-smi</code></h3>
|
||||
<pre><code class="language-bash">nvidia-smi \
|
||||
--query-gpu=index,timestamp,utilization.gpu,utilization.memory,\
|
||||
memory.used,memory.total,power.draw,temperature.gpu,\
|
||||
clocks.sm,clocks.mem,pstate \
|
||||
--format=csv,noheader,nounits</code></pre>
|
||||
<p>脚本每秒运行一次，并在每行前加入 <code>wall_time_ns</code> 和节点角色。</p>
|
||||
<table>
|
||||
<thead><tr><th>字段</th><th>代表什么</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>utilization.gpu</code></td><td>采样周期内至少有一个 Kernel 在执行的时间比例</td></tr>
|
||||
<tr><td><code>utilization.memory</code></td><td>采样周期内显存控制器处于忙碌状态的时间比例</td></tr>
|
||||
<tr><td><code>memory.used/total</code></td><td>当前总显存分配量与设备显存容量</td></tr>
|
||||
<tr><td><code>power.draw</code></td><td>GPU 当前功耗，用于比较不同负载的能耗状态</td></tr>
|
||||
<tr><td><code>clocks.sm/clocks.mem</code></td><td>SM 与显存当前工作频率</td></tr>
|
||||
<tr><td><code>pstate</code></td><td>GPU 性能状态，P0 通常是最高性能态</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
原始输出为 <code>head|worker/gpu_samples.csv</code>；
|
||||
<code>gpu_summary.csv</code> 汇总整段运行，
|
||||
<code>case_gpu_summary.csv</code> 按节点、Case 和 GPU 汇总平均值与峰值。
|
||||
</p>
|
||||
|
||||
<h3>5.3 GPU Profiling Counter：DCGM</h3>
|
||||
<pre><code class="language-bash">DCGM_FIELD_IDS=1001,1002,1003,1004,1005,1009,1010
|
||||
|
||||
dcgmi dmon \
|
||||
-e 1001,1002,1003,1004,1005,1009,1010 \
|
||||
-d 1000</code></pre>
|
||||
<table>
|
||||
<thead><tr><th>Field ID</th><th>Field Tag</th><th>含义</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>1001</td><td><code>gr_engine_active</code></td><td>Graphics/Compute Engine 活跃比例，接近整体 GPU 执行忙碌度</td></tr>
|
||||
<tr><td>1002</td><td><code>sm_active</code></td><td>SM 至少有一个 Warp 活跃的比例</td></tr>
|
||||
<tr><td>1003</td><td><code>sm_occupancy</code></td><td>活跃 Warp 相对硬件可容纳 Warp 的比例</td></tr>
|
||||
<tr><td>1004</td><td><code>tensor_active</code></td><td>Tensor Core 指令活跃比例</td></tr>
|
||||
<tr><td>1005</td><td><code>dram_active</code></td><td>设备显存接口活跃比例；Pro6000D 为 GDDR7，用于判断设备显存带宽压力</td></tr>
|
||||
<tr><td>1009</td><td><code>pcie_tx_bytes</code></td><td>GPU 经 PCIe 发出的字节速率</td></tr>
|
||||
<tr><td>1010</td><td><code>pcie_rx_bytes</code></td><td>GPU 经 PCIe 接收的字节速率</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
<code>sm_active</code> 高而 <code>sm_occupancy</code> 低，表示 SM 经常有工作，
|
||||
但同时驻留的 Warp 不多；后续通过 Kernel Timeline 区分小 Kernel、
|
||||
寄存器/共享内存约束和同步。DCGM 是 NVIDIA Data Center GPU Manager：
|
||||
<code>nvidia-dcgm</code>/<code>nv-hostengine</code> 是后台 Host Engine，
|
||||
<code>dcgmi</code> 是客户端，Field ID 是指标编号。最终代码在两节点预检
|
||||
<code>dcgmi discovery -l</code>，任一 Host Engine 不可用即 fail-fast；
|
||||
正式结果必须同时包含 Head 和 Worker 的 <code>case_dcgm_summary.csv</code>。
|
||||
</p>
|
||||
|
||||
<h3>5.4 CPU、进程与 Kernel Launch 侧证据</h3>
|
||||
<pre><code class="language-bash"># 全部逻辑 CPU，每 5 秒输出一次
|
||||
mpstat -P ALL 5
|
||||
|
||||
# 找到容器内进程对应的宿主 PID
|
||||
docker top <container> -eo pid,ppid,psr,pcpu,pmem,stat,comm,args
|
||||
|
||||
# 最终命令：进程级 CPU、I/O、缺页、上下文切换，不展开全部线程
|
||||
pidstat -durw -p "<comma-separated-host-pids>" 5
|
||||
|
||||
# 每 5 秒输出一次硬件/软件计数器增量
|
||||
perf stat -p "<comma-separated-host-pids>" -I 5000 \
|
||||
-e cycles,instructions,cache-misses,context-switches,\
|
||||
cpu-migrations,page-faults
|
||||
|
||||
# mpstat/pidstat/perf 每行都由包装器增加：
|
||||
# wall_time_ns TAB node TAB 原始输出</code></pre>
|
||||
<table>
|
||||
<thead><tr><th>命令/字段</th><th>回答的问题</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>mpstat -P ALL</code></td><td>整机是否 CPU 饱和，是否只有少量核心接近 100%，是否存在 I/O Wait</td></tr>
|
||||
<tr><td><code>docker top</code></td><td>把容器进程映射为宿主 PID、CPU 核 <code>PSR</code> 和进程状态</td></tr>
|
||||
<tr><td><code>pidstat -u</code></td><td>服务进程的用户态、内核态 CPU 时间</td></tr>
|
||||
<tr><td><code>pidstat -d</code></td><td>进程块设备 I/O</td></tr>
|
||||
<tr><td><code>pidstat -r</code></td><td>内存和 Page Fault 行为</td></tr>
|
||||
<tr><td><code>pidstat -w</code></td><td>主动/被动上下文切换，辅助发现线程阻塞或调度抖动</td></tr>
|
||||
<tr><td><code>perf cycles/instructions</code></td><td>CPU 周期与指令执行量，可计算近似 IPC</td></tr>
|
||||
<tr><td><code>cache-misses/migrations</code></td><td>CPU Cache 压力和线程跨核迁移</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
最终实现使用进程级 5 秒采样，避免首轮线程级 1 秒采样产生数百 MB 日志。
|
||||
<code>case_cpu_summary.csv</code>、<code>case_process_summary.csv</code> 和
|
||||
<code>case_perf_summary.csv</code> 都按正式测量窗口切片；只有先发现异常进程，
|
||||
才在后续短窗口单独开启线程级采样。
|
||||
</p>
|
||||
|
||||
<h3>5.5 NUMA 与 CPU/内存亲和</h3>
|
||||
<pre><code class="language-bash"># 静态 NUMA 节点、CPU 和内存布局
|
||||
numactl --hardware
|
||||
numastat -m
|
||||
|
||||
# 每 5 秒按容器宿主 PID 查看本地/远端 NUMA 内存
|
||||
numastat -p <host-pid>
|
||||
# 解析为：
|
||||
# wall_time_ns,node,node0_mib,node1_mib,total_mib,process_count
|
||||
|
||||
# 同时保存 GPU、CPU、NIC 的拓扑关系
|
||||
nvidia-smi topo -m</code></pre>
|
||||
<p>
|
||||
NUMA 是多路 CPU 机器的“本地内存”结构。进程长期从远端 NUMA Node 取内存，
|
||||
或 GPU/NIC 对应的 CPU 线程被调度到另一侧，可能增加 Host 侧延迟。
|
||||
最终采集器把 <code>numastat -p</code> 解析为
|
||||
<code>numa_samples.csv</code>，再按正式测量窗口生成
|
||||
<code>case_numa_summary.csv</code>。这样可以直接比较 Node0/Node1 MiB，
|
||||
而不是依靠人工阅读不断刷新的文本。
|
||||
</p>
|
||||
|
||||
<h3>5.6 普通网卡统计与 RDMA 数据面</h3>
|
||||
<pre><code class="language-bash"># Linux netdev 层，每 5 秒采样吞吐与错误
|
||||
sar -n DEV,EDEV 5
|
||||
|
||||
# Case 前后保存物理端口状态和驱动计数器
|
||||
ethtool eth0
|
||||
ethtool eth3
|
||||
ethtool -S eth0
|
||||
ethtool -S eth3
|
||||
|
||||
# RDMA 设备与端口状态
|
||||
ibdev2netdev
|
||||
ibstat
|
||||
rdma link show</code></pre>
|
||||
<p>
|
||||
<code>sar</code> 记录 Linux 普通网络栈中的 <code>eth0/eth3</code> 流量；
|
||||
GDRDMA 数据量由 <code>mlx5_0/mlx5_3</code> HCA 的 sysfs Counter 记录：
|
||||
</p>
|
||||
<pre><code class="language-bash">for hca in mlx5_0 mlx5_3; do
|
||||
base="/sys/class/infiniband/${hca}/ports/1"
|
||||
cat "${base}/counters/port_xmit_data"
|
||||
cat "${base}/counters/port_rcv_data"
|
||||
cat "${base}/counters/port_xmit_wait"
|
||||
cat "${base}/counters/port_xmit_discards"
|
||||
cat "${base}/counters/port_rcv_errors"
|
||||
cat "${base}/hw_counters/req_transport_retries_exceeded"
|
||||
cat "${base}/hw_counters/req_rnr_retries_exceeded"
|
||||
done</code></pre>
|
||||
<table>
|
||||
<thead><tr><th>Counter</th><th>含义</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>port_xmit_data/port_rcv_data</code></td><td>HCA 发送/接收数据累计量；IB Counter 单位是 4 Octets，脚本用 <code>delta × 4 × 8 / seconds</code> 换算 Gbit/s</td></tr>
|
||||
<tr><td><code>port_xmit_wait</code></td><td>端口因缺少发送 Credit 等原因等待的时间，持续增长可能指向拥塞</td></tr>
|
||||
<tr><td><code>port_xmit_discards/port_rcv_errors</code></td><td>发送丢弃和接收错误增量</td></tr>
|
||||
<tr><td><code>req_transport_retries_exceeded</code></td><td>RDMA Transport 重试耗尽</td></tr>
|
||||
<tr><td><code>req_rnr_retries_exceeded</code></td><td>Receiver Not Ready 重试耗尽</td></tr>
|
||||
<tr><td><code>roce_adp_retrans*</code></td><td>RoCE 自适应重传及超时相关计数</td></tr>
|
||||
<tr><td><code>np_ecn_marked* / *cnp*</code></td><td>ECN 标记和拥塞通知包，用于辅助判断 RoCE 拥塞</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
原始数据为 <code>head|worker/rdma.csv</code>；
|
||||
<code>case_rdma_summary.csv</code> 按 Case、节点和 HCA 计算吞吐及错误增量。
|
||||
它说明双 Rail 的实际流量、均衡性和错误增量；
|
||||
<code>case_netdev_summary.csv</code> 同时保留 Linux netdev 层的
|
||||
<code>eth0/eth3</code> RX/TX 与错误。Phase 3 再把 NCCL Collective
|
||||
放到请求 Timeline 中分析持续时间和计算重叠。
|
||||
</p>
|
||||
|
||||
<h3>5.7 机内 PCIe 与 NCCL 通信基线</h3>
|
||||
<pre><code class="language-bash"># 由 all 入口自动执行；不需要用户手工运行 torchrun
|
||||
# 每个节点：所有 GPU 源/目标对，FP16 256 MiB CUDA P2P copy
|
||||
python3 communication_baseline.py p2p \
|
||||
--size 256M --warmup 3 --iterations 10
|
||||
|
||||
# 每个节点：8 rank NCCL AllReduce
|
||||
torchrun --standalone --nproc-per-node=8 \
|
||||
communication_baseline.py all-reduce \
|
||||
--sizes 1M,64M,1G --repetitions 3 --warmup 5 --iterations 10
|
||||
|
||||
# 双节点：16 rank；分别设置 NCCL_CROSS_NIC=0、1、2
|
||||
torchrun --nnodes=2 --nproc-per-node=8 \
|
||||
--master-addr 10.101.0.11 --node-rank <0-or-1> \
|
||||
communication_baseline.py all-reduce \
|
||||
--sizes 1M,64M,1G --repetitions 3 --warmup 5 --iterations 10</code></pre>
|
||||
<p>
|
||||
P2P 结果按 <code>same_pcie_switch</code>（PIX）和
|
||||
<code>cross_numa_sys</code>（SYS）分别汇总，不用一个平均值掩盖跨 CPU 路径。
|
||||
AllReduce 同时报告 P50/P95 latency、<code>algbw</code>、
|
||||
<code>busbw</code>、正确性错误数和实际 NCCL 路径。1 MiB、64 MiB、1 GiB
|
||||
分别覆盖小消息延迟、中等消息和大消息带宽；双机 A/B 直接给出
|
||||
<code>NCCL_CROSS_NIC=0/1/2</code> 的数值比较。
|
||||
</p>
|
||||
|
||||
<h3>5.8 静态快照与结果关系</h3>
|
||||
<pre><code class="language-bash">nvidia-smi
|
||||
nvidia-smi topo -m
|
||||
lscpu
|
||||
numactl --hardware
|
||||
ip -details link show eth0
|
||||
ip -details link show eth3
|
||||
docker inspect <container>
|
||||
docker top <container> -eo pid,ppid,psr,pcpu,pmem,stat,comm,args</code></pre>
|
||||
<table>
|
||||
<thead><tr><th>结果文件</th><th>内容</th><th>主要用途</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><code>static_before.log / static_after.log</code></td><td>GPU、CPU、NUMA、NIC、RDMA、容器前后快照</td><td>证明运行环境，并比较错误计数和清理状态</td></tr>
|
||||
<tr><td><code>collector_status.csv</code></td><td>每个采集器的启动、停止或提前退出状态</td><td>防止把缺失采集器当作 0 值</td></tr>
|
||||
<tr><td><code>case_windows.csv</code></td><td>每个 Benchmark Case 的起止时间</td><td>从连续硬件日志中切片</td></tr>
|
||||
<tr><td><code>bench_summary.csv</code></td><td>TPS、TTFT、TPOT、ITL、E2E</td><td>把硬件现象与用户侧性能对应</td></tr>
|
||||
<tr><td><code>case_gpu_summary.csv</code></td><td>每 Case、节点、GPU 的利用率、显存、功耗、频率</td><td>比较负载与节点/GPU 不均衡</td></tr>
|
||||
<tr><td><code>case_dcgm_summary.csv</code></td><td>每 Case、节点、GPU 的 SM/Tensor/显存接口/PCIe 指标</td><td>区分计算、设备显存和 PCIe 活跃度</td></tr>
|
||||
<tr><td><code>case_cpu_summary.csv</code></td><td>整机与逐核 CPU 利用率、I/O Wait</td><td>识别整机饱和和少数热点核</td></tr>
|
||||
<tr><td><code>case_process_summary.csv</code></td><td>服务进程 CPU、I/O、缺页、内存与上下文切换</td><td>定位 Host 进程开销与阻塞</td></tr>
|
||||
<tr><td><code>case_perf_summary.csv</code></td><td>cycles、instructions、cache miss、迁移与缺页</td><td>计算 IPC 并判断 Cache/调度压力</td></tr>
|
||||
<tr><td><code>case_numa_summary.csv</code></td><td>Node0/Node1 进程内存分布</td><td>识别跨 NUMA 放置</td></tr>
|
||||
<tr><td><code>case_netdev_summary.csv</code></td><td><code>eth0/eth3</code> 吞吐与错误</td><td>与 RDMA HCA Counter 做分层核对</td></tr>
|
||||
<tr><td><code>case_rdma_summary.csv</code></td><td>每 Case、节点、Rail 的吞吐和错误增量</td><td>判断双 Rail 使用、均衡和数据面错误</td></tr>
|
||||
<tr><td><code>communication_summary.csv</code></td><td>每次 P2P/AllReduce 原始测量</td><td>保留每条 GPU 对、消息尺寸、CROSS_NIC 和重复实验</td></tr>
|
||||
<tr><td><code>communication_aggregate.csv</code></td><td>PIX/SYS P2P 与单/双机 AllReduce 聚合</td><td>提供 P50/P95、algbw、busbw 和正确性比较</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>5.9 最终结果如何逐项汇报</h3>
|
||||
<p class="decision">
|
||||
最终 <code>report.md</code> 的章节顺序与本节一一对应。每一项必须同时给出
|
||||
<strong>原始文件、有效样本数、Head/Worker 数值、Case 间变化和解释</strong>；
|
||||
不能只写“GPU 较忙”“网络未饱和”这类抽象结论。
|
||||
</p>
|
||||
<table>
|
||||
<thead><tr><th>第 5 节指标</th><th>报告中的数值</th><th>最小分析动作</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>5.1 时间窗</td><td>窗口来源、开始/结束、duration、采样数</td><td>确认全部为 <code>bench_main_marker_plus_duration</code></td></tr>
|
||||
<tr><td>5.2 GPU</td><td>利用率/显存/功耗/频率的 Mean、P95、Max</td><td>比较两节点、8 卡离散度和不同 Case</td></tr>
|
||||
<tr><td>5.3 DCGM</td><td>SM Active/Occupancy、Tensor/DRAM Active、PCIe TX/RX</td><td>比较计算、设备显存和 PCIe 哪一侧随负载上升</td></tr>
|
||||
<tr><td>5.4 CPU/进程/perf</td><td>整机/热点核、进程 CPU/I/O/缺页/切换、IPC/Cache miss</td><td>区分整机容量、单线程热点和 Host 调度开销</td></tr>
|
||||
<tr><td>5.5 NUMA</td><td>Node0/Node1 MiB 与比例</td><td>比较服务内存是否偏离 GPU/NIC 所在 NUMA</td></tr>
|
||||
<tr><td>5.6 Network/RDMA</td><td>eth0/eth3、mlx5_0/mlx5_3 Gbit/s 与错误增量</td><td>计算双 Rail 均衡比例并核对丢弃/重试</td></tr>
|
||||
<tr><td>5.7 Communication</td><td>PIX/SYS P2P、8/16 rank AllReduce P50/P95、algbw/busbw</td><td>比较跨 NUMA 损失与 CROSS_NIC 0/1/2</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
某个采集器无数据时报告显示 <code>-</code> 并附失败状态，不会把缺失值写成
|
||||
<code>0</code>。正式 Run 默认 <code>ALLOW_PARTIAL_COLLECTORS=0</code>，
|
||||
因此必需采集器提前退出会让 Run 失败。
|
||||
</p>
|
||||
|
||||
<h2>6. 精简代码设计</h2>
|
||||
<p>已新增目录：</p>
|
||||
<pre><code>/data/hzy/sskj/experiments/pro6000/
|
||||
dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/</code></pre>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>文件</th><th>职责</th><th>当前状态</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td><code>run_hardware_contention_attribution.sh</code></td><td>唯一 Shell 入口；按 Run 分发通信代码、通信基线、服务启停、双节点采集器、Case 编排、门禁和 Trap 清理</td><td class="pass">已实现</td></tr>
|
||||
<tr><td><code>config.env</code></td><td>Phase 1 相对路径、节点、代表 Case、分层采样周期、通信基线与 fail-closed 策略</td><td class="pass">已实现</td></tr>
|
||||
<tr><td><code>communication_baseline.py</code></td><td>CUDA P2P 全矩阵与 PyTorch/NCCL 8/16-rank AllReduce 微基准</td><td class="pass">已实现</td></tr>
|
||||
<tr><td><code>hardware_contention_attribution.py</code></td><td>精确窗口、全部采集器解析、逐 Case 汇总、通信聚合和逐指标报告</td><td class="pass">已实现</td></tr>
|
||||
<tr><td><code>tests/test_hardware_contention_attribution.py</code></td><td>Worker 无仓库依赖、GPU/RDMA、精确窗口、DCGM/CPU 解析、通信聚合和结果生成测试</td><td class="pass">9/9 通过</td></tr>
|
||||
<tr><td><code>README.md</code></td><td>唯一入口、范围和结果目录说明</td><td class="pass">已实现</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="decision">
|
||||
Phase 2 不复制双机 Docker 启停实现。唯一入口在内部调用 Phase 1 的
|
||||
<code>run_quick_map.sh start/fixed/mixed/stop</code>，只新增通信基线、硬件采集、时间对齐和代表负载编排。
|
||||
顶层仍只保留一个 Shell 文件，不创建额外 tmux、launch、start 或 stop 脚本。
|
||||
</p>
|
||||
|
||||
<h2>7. 结果结构</h2>
|
||||
<pre><code>results/<RUN_ID>/
|
||||
manifest.json
|
||||
run.log
|
||||
bench/
|
||||
bench_cmd.txt
|
||||
bench.log
|
||||
bench.jsonl
|
||||
service/
|
||||
head_server_cmd.txt
|
||||
worker_server_cmd.txt
|
||||
head_server.log
|
||||
worker_server.log
|
||||
communication/
|
||||
communication_baseline.py
|
||||
communication_baseline.sha256
|
||||
p2p_head.log
|
||||
p2p_worker.log
|
||||
allreduce_head_8gpu.log
|
||||
allreduce_worker_8gpu.log
|
||||
allreduce_two_node_x0.log
|
||||
allreduce_two_node_x1.log
|
||||
allreduce_two_node_x2.log
|
||||
head/
|
||||
gpu_samples.csv
|
||||
dcgm_dmon.log
|
||||
mpstat.log
|
||||
pidstat.log
|
||||
sar_net.log
|
||||
perf_stat.log
|
||||
docker_top.log
|
||||
numa_samples.csv
|
||||
rdma.csv
|
||||
static_before.log
|
||||
static_after.log
|
||||
collector_commands/
|
||||
worker/
|
||||
...
|
||||
collector_status.csv
|
||||
markers.csv
|
||||
bench_summary.csv
|
||||
gpu_summary.csv
|
||||
rdma_summary.csv
|
||||
case_windows.csv
|
||||
communication_summary.csv
|
||||
communication_aggregate.csv
|
||||
case_gpu_summary.csv
|
||||
case_gpu_node_summary.csv
|
||||
case_dcgm_summary.csv
|
||||
case_cpu_summary.csv
|
||||
case_process_summary.csv
|
||||
case_perf_summary.csv
|
||||
case_numa_summary.csv
|
||||
case_netdev_summary.csv
|
||||
case_rdma_summary.csv
|
||||
summary.json
|
||||
report.md</code></pre>
|
||||
|
||||
<h2>8. 最终验收</h2>
|
||||
<ul>
|
||||
<li class="pass">最终 Run <code>dsv4pro-phase2-20260731-163620</code> 状态为 <code>COMPLETED</code>，8/8 benchmark 成功，0 失败、0 OOM。</li>
|
||||
<li class="pass">8/8 Case 使用正式 benchmark 精确时间窗；18 个采集器全部记录 <code>STARTED</code> 与 <code>STOPPED</code>。</li>
|
||||
<li class="pass">Head 与 Worker 的 DCGM、GPU、CPU、进程、NUMA、网卡和 HCA Counter 均有有效样本。</li>
|
||||
<li class="pass">P2P、8-rank 和 16-rank AllReduce 全部完成，所有正确性检查均为 <code>wrong_values=0</code>。</li>
|
||||
<li class="pass">Run 结束后 Head/Worker 无相关容器、无 GPU 计算进程，端口 <code>30002/30003</code> 已释放。</li>
|
||||
</ul>
|
||||
|
||||
<h2>9. 正式运行</h2>
|
||||
<p class="decision">
|
||||
<strong>Run：<code>dsv4pro-phase2-20260731-163620</code>。</strong>
|
||||
运行时间为 16:36:20 至 17:05:04 CST，总用时 28 分 44 秒。
|
||||
Manifest 记录代码提交 <code>5f24b7d22f98108f6cc234edba6768d55ea0a962</code>，
|
||||
<code>git_dirty=false</code>。
|
||||
</p>
|
||||
<pre><code class="language-bash"># 仅在 Head 174.1.51.5 执行
|
||||
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
|
||||
|
||||
RUN_ID=dsv4pro-phase2-20260731-163620
|
||||
tmux new-session -d -s dsv4pro-phase2 \
|
||||
"RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
|
||||
2>&1 | tee /data/hzy/${RUN_ID}.log"</code></pre>
|
||||
|
||||
<h2>10. 端到端结果</h2>
|
||||
<h3>10.1 五个代表负载</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>Case</th><th>Input TPS</th><th>Output TPS</th><th>TTFT P95</th><th>TPOT P95</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>128K → 1, C=1</td><td>2,641.38</td><td>0.02</td><td>49.610 s</td><td>—</td></tr>
|
||||
<tr><td>32K → 1, C=16</td><td>3,115.45</td><td>0.10</td><td>161.938 s</td><td>—</td></tr>
|
||||
<tr><td>1K → 1K, C=32</td><td>448.95</td><td>448.95</td><td>10.167 s</td><td>65.36 ms</td></tr>
|
||||
<tr><td>1K → 4K, C=16</td><td>79.39</td><td>317.56</td><td>1.724 s</td><td>49.98 ms</td></tr>
|
||||
<tr><td>128K → 1K, C=1</td><td>1,614.99</td><td>12.62</td><td>48.279 s</td><td>32.11 ms</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<h3>10.2 混合 Prefill/Decode A/B</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>Decode 指标</th><th>Control</th><th>注入 128K Prefill</th><th>变化</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr><td>Output TPS</td><td>453.55</td><td>344.89</td><td>-23.96%</td></tr>
|
||||
<tr><td>TTFT P95</td><td>9.443 s</td><td>9.892 s</td><td>+4.76%</td></tr>
|
||||
<tr><td>TPOT P95</td><td>66.24 ms</td><td>110.45 ms</td><td>+66.75%</td></tr>
|
||||
<tr><td>E2E P95</td><td>72.270 s</td><td>118.076 s</td><td>+63.38%</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
这是 Phase 2 最关键的现象：长 Prefill 与 Decode 共存时，首 token 延迟只增加
|
||||
4.76%，但 Decode 单 token 成本增加 66.75%，最终令 Output TPS 下降 23.96%。
|
||||
问题主要发生在持续 Decode 阶段，而不是只表现为 Prefill 请求排队。
|
||||
</p>
|
||||
|
||||
<h2>11. 第 5 节指标逐项结果</h2>
|
||||
<h3>11.1 GPU 基础状态与 DCGM</h3>
|
||||
<p><strong>服务器证据路径：</strong><br>
|
||||
汇总：<code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/case_gpu_summary.csv</code>、
|
||||
<code>case_gpu_node_summary.csv</code>、<code>case_dcgm_summary.csv</code><br>
|
||||
原始：同一 Run 目录下的 <code>head/gpu_samples.csv</code>、<code>worker/gpu_samples.csv</code>、
|
||||
<code>head/dcgm_dmon.log</code>、<code>worker/dcgm_dmon.log</code>
|
||||
</p>
|
||||
<ul>
|
||||
<li>各 Case GPU Util Mean 大多为 94%–99%，P95 为 100%；SM Clock 约 2.39–2.42 GHz，未见降频。</li>
|
||||
<li>每卡显存稳定在约 83,000–83,364 MiB。Decode 功耗约 216–258 W，Prefill 功耗约 293–307 W。</li>
|
||||
<li>普通 Decode 的 SM Active 约 0.523–0.525、DRAM Active 约 0.415–0.417；128K Prefill 的 SM Active 升至 0.683–0.686。</li>
|
||||
<li>32K C16 Prefill 的 SM Active 约 0.713–0.715、DRAM Active 约 0.440，是本轮最重的并发 Prefill 计算负载。</li>
|
||||
<li>混合 Treatment 中，Decode 背景 SM Active 约 0.578–0.581；注入 Prefill 窗口升至 0.720–0.723，证明两类工作确实争用同一 GPU 执行资源。</li>
|
||||
</ul>
|
||||
|
||||
<h3>11.2 CPU、进程与 NUMA</h3>
|
||||
<p><strong>服务器证据路径：</strong><br>
|
||||
汇总：<code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/case_cpu_summary.csv</code>、
|
||||
<code>case_process_summary.csv</code>、<code>case_perf_summary.csv</code>、<code>case_numa_summary.csv</code><br>
|
||||
原始：同一 Run 目录下 Head/Worker 各自的 <code>mpstat.log</code>、<code>pidstat.log</code>、
|
||||
<code>perf_stat.log</code>、<code>numa_samples.csv</code> 和 <code>docker_top.log</code>
|
||||
</p>
|
||||
<ul>
|
||||
<li>整机 CPU Active Mean 约 9.6%–10.4%，P95 约 10%–11.4%；没有全机 CPU 饱和。</li>
|
||||
<li>服务进程峰值约 1,210%–1,226%，相当于约 12 个 CPU Core；热点 Core 数量最多 12–13 个。</li>
|
||||
<li><code>perf</code> 观察到 IPC 约 2.7–3.1，未出现明显 Host 侧停摆。</li>
|
||||
<li>Head/Worker 的 NUMA 不均衡约 21.1% / 13.2%，跨 Case 基本稳定；它是拓扑基线，但不像混合性能退化的直接诱因。</li>
|
||||
</ul>
|
||||
|
||||
<h3>11.3 双 Rail RDMA</h3>
|
||||
<p><strong>服务器证据路径：</strong><br>
|
||||
汇总：<code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/case_rdma_summary.csv</code>、
|
||||
<code>rdma_summary.csv</code>、<code>case_netdev_summary.csv</code><br>
|
||||
原始：同一 Run 目录下的 <code>head/rdma.csv</code>、<code>worker/rdma.csv</code>、
|
||||
<code>head/sar_net.log</code>、<code>worker/sar_net.log</code>；端口/HCA 静态状态在两端
|
||||
<code>static_before.log</code> 与 <code>static_after.log</code>
|
||||
</p>
|
||||
<ul>
|
||||
<li>RoCE 绕过普通 Linux Socket 数据路径，因此 <code>sar</code> 的 <code>eth0/eth3</code> 流量接近 0；实际流量必须看 <code>mlx5_0/mlx5_3</code> HCA Counter。</li>
|
||||
<li>普通 Decode 每 Rail 约 36.7–36.8 Gbit/s；128K Prefill 每 Rail约 70.5–71.5 Gbit/s。</li>
|
||||
<li>最高点 32K C16 Prefill 每 Rail 约 83.0–83.45 Gbit/s，仅约占单条 400G Rail 的 20.9%。</li>
|
||||
<li>两条 Rail 流量对称，<code>port_xmit_wait</code>、丢弃、错误和 Retry Exceeded 增量均为 0。</li>
|
||||
</ul>
|
||||
|
||||
<h3>11.4 PCIe 与 NCCL 通信基线</h3>
|
||||
<p><strong>服务器证据路径：</strong><br>
|
||||
汇总：<code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/communication_aggregate.csv</code>、
|
||||
<code>communication_summary.csv</code><br>
|
||||
原始：同一 Run 目录的 <code>communication/p2p_head.log</code>、<code>p2p_worker.log</code>、
|
||||
<code>allreduce_head_8gpu.log</code>、<code>allreduce_worker_8gpu.log</code> 和
|
||||
<code>allreduce_16gpu_crossnic{0,1,2}_{head,worker}.log</code>；实际执行命令在 <code>commands/communication_*.cmd.txt</code>
|
||||
</p>
|
||||
<table>
|
||||
<thead><tr><th>测试</th><th>结果</th><th>解释</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>Head PCIe P2P 256 MiB</td><td>同 Switch 53.61 GB/s；跨 NUMA 52.40 GB/s</td><td>跨 NUMA 损失约 2.3%</td></tr>
|
||||
<tr><td>Worker PCIe P2P 256 MiB</td><td>同 Switch 53.50 GB/s；跨 NUMA 52.32 GB/s</td><td>两节点表现对称</td></tr>
|
||||
<tr><td>Head / Worker 8-GPU AllReduce 1 GiB</td><td>busbw 39.76 / 39.75 GB/s</td><td>节点内基线一致</td></tr>
|
||||
<tr><td>16-GPU AllReduce，CROSS_NIC=0</td><td>busbw 39.345 GB/s；51.169 ms</td><td>正确性 0 错误</td></tr>
|
||||
<tr><td>16-GPU AllReduce，CROSS_NIC=1</td><td>busbw 39.685 GB/s；50.732 ms</td><td>本轮数值最好</td></tr>
|
||||
<tr><td>16-GPU AllReduce，CROSS_NIC=2</td><td>busbw 39.530 GB/s；50.931 ms</td><td>正确性 0 错误</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>
|
||||
<code>NCCL_CROSS_NIC=1</code> 比 0 仅高 0.86%，比 2 仅高 0.39%。
|
||||
差异小于 1%，不足以把它当成主要调优旋钮；保留当前值即可，Phase 3 不再重复测试。
|
||||
NCCL 日志明确证明跨机路径使用 <code>mlx5_0,mlx5_3</code> 和
|
||||
<code>NET/IB/.../GDRDMA</code>。
|
||||
</p>
|
||||
|
||||
<h2>12. 结论与 Phase 3 入口</h2>
|
||||
<p class="decision">
|
||||
<strong>Phase 2 已把范围明显缩小：</strong>混合 Prefill/Decode 退化真实且稳定，
|
||||
但不是由整机 CPU 饱和、GPU 降频、双 Rail 原始带宽饱和、Rail 失衡、
|
||||
PCIe 跨 NUMA 带宽崩塌或 <code>NCCL_CROSS_NIC</code> 选择造成。
|
||||
Phase 3 应只捕获 Control 与 Treatment 的短时间线，定位 Attention/Indexer、
|
||||
MoE、NCCL Collective、Scheduler gap 和慢 Rank 同步之间的串行与重叠关系。
|
||||
</p>
|
||||
<ul>
|
||||
<li>不重复 Phase 2 的长时间 DCGM、CPU、RDMA 和通信微基准。</li>
|
||||
<li>优先对比混合 Control 与注入 128K Prefill 的 Treatment。</li>
|
||||
<li>再用单独 128K Prefill 作为 Kernel 对照，解释 SM Active 与 Tensor Active 的来源。</li>
|
||||
</ul>
|
||||
|
||||
<h2>13. 证据与清理说明</h2>
|
||||
<ul>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/report.md">自动生成逐指标报告</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/analysis.md">阶段归因摘要</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/bench_summary.csv">端到端 benchmark 汇总</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/communication_aggregate.csv">通信微基准汇总</a></li>
|
||||
<li><a href="./results/dsv4pro-phase2-20260731-163620/service/head_server_cmd.txt">Head 实际服务命令</a> /
|
||||
<a href="./results/dsv4pro-phase2-20260731-163620/service/worker_server_cmd.txt">Worker 实际服务命令</a></li>
|
||||
</ul>
|
||||
<p>
|
||||
Worker 日志在所有 benchmark 完成后的编排关闭阶段出现 Gloo
|
||||
<code>Connection closed by peer</code>；时间与 Head 主动退出进程组一致，
|
||||
未影响 8/8 结果。NCCL 日志中的可选 mlx5 symbol 探测提示同样未影响
|
||||
Collective，全部正确性检查为 0 错误。
|
||||
</p>
|
||||
|
||||
<p><a class="back" href="./phase1_exp.html">返回 Phase 1 实验档案</a></p>
|
||||
<p><a class="back" href="./phase2_5_exp.html">继续 Phase 2.5 RDMA 需求建模</a></p>
|
||||
<p><a class="back" href="./推理优化计划.html">返回推理优化主计划</a></p>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@ -1,13 +0,0 @@
|
||||
suite,case_id,role,stage,isl,osl,concurrency,planned_repetitions,completed_repetitions,statuses,total_tps_mean,total_tps_cv_pct,output_tps_mean,output_tps_cv_pct,ttft_p95_mean_ms,tpot_p95_mean_ms,itl_p95_mean_ms,e2e_p95_mean_ms
|
||||
fixed,balanced_32k_to_1k_c8,,balanced,32768,1024,8,1,1,COMPLETED,2101.686722481554,,63.68747643883498,,82083.64989476977,109.32309288714016,43.84165157971437,128588.8987001963
|
||||
fixed,decode_latency_1k_to_1k_c1,,decode_latency,1024,1024,1,1,1,COMPLETED,62.82990203738489,,31.414951018692445,,363.2723209448159,31.468245639333286,31.659404194215313,32555.28760998277
|
||||
fixed,decode_throughput_1k_to_1k_c16,,decode_throughput,1024,1024,16,1,1,COMPLETED,590.5818337765435,,295.29091688827174,,4950.267374995747,50.01878498413362,49.8873026604997,55443.82167501317
|
||||
fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1024,1024,32,1,1,COMPLETED,923.3692532200665,,461.68462661003326,,8021.852347906679,63.30661669023356,62.237933481810614,70933.18627287517
|
||||
fixed,decode_throughput_1k_to_1k_c64,,decode_throughput,1024,1024,64,1,1,COMPLETED,1294.833542313111,,647.4167711565555,,12716.491229846724,93.44459645826957,87.80462378927041,101163.1970004586
|
||||
fixed,long_prefill_latency_128k_c1,,prefill_latency,131072,1,1,1,1,COMPLETED,2710.1829473156417,,0.020676897204730506,,48344.461318978574,0.0,0.0,48344.52949295519
|
||||
fixed,mid_prefill_latency_32k_c1,,prefill_latency,32768,1,1,1,1,COMPLETED,2652.8390294198,,0.08095575175988892,,12334.68782599084,0.0,0.0,12334.764264000114
|
||||
fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,32768,1,16,1,1,COMPLETED,3112.8645802960464,,0.09499418902914482,,162087.46085499297,0.0,0.0,162087.50443853205
|
||||
fixed,short_prefill_latency_1k_c1,,prefill_latency,1024,1,1,1,1,COMPLETED,1971.5883577957954,,1.9235008368739468,,501.89953204244375,0.0,0.0,501.94522901438177
|
||||
mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1024,1024,32,1,1,COMPLETED,911.3699336194111,,455.68496680970554,,9442.636363586644,65.88327712923635,61.753195300116204,72007.63789927005
|
||||
mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1024,1024,32,1,1,COMPLETED,691.8932985710949,,345.94664928554744,,10193.896457596566,109.7299457727384,61.61956858995836,117629.63820034638
|
||||
mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,131072,1,1,1,1,COMPLETED,2859.7533639992707,,0.021818020217735695,,45752.13837000774,0.0,0.0,45752.21362197772
|
||||
|
@ -1,31 +0,0 @@
|
||||
# DeepSeek-V4-Pro Pro6000D Two-Node SGLang Quick Map
|
||||
|
||||
Profiler: disabled. Speculative decoding: disabled.
|
||||
|
||||
## Aggregate results
|
||||
|
||||
| Case | Suite / role | Stage | ISL | OSL | C | Reps | Total TPS | CV | Output TPS | TTFT P95 | TPOT P95 | E2E P95 | Status |
|
||||
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| balanced_32k_to_1k_c8 | fixed / - | balanced | 32768 | 1024 | 8 | 1/1 | 2101.69 | -% | 63.69 | 82083.65 ms | 109.32 ms | 128588.90 ms | COMPLETED |
|
||||
| decode_latency_1k_to_1k_c1 | fixed / - | decode_latency | 1024 | 1024 | 1 | 1/1 | 62.83 | -% | 31.41 | 363.27 ms | 31.47 ms | 32555.29 ms | COMPLETED |
|
||||
| decode_throughput_1k_to_1k_c16 | fixed / - | decode_throughput | 1024 | 1024 | 16 | 1/1 | 590.58 | -% | 295.29 | 4950.27 ms | 50.02 ms | 55443.82 ms | COMPLETED |
|
||||
| decode_throughput_1k_to_1k_c32 | fixed / - | decode_throughput | 1024 | 1024 | 32 | 1/1 | 923.37 | -% | 461.68 | 8021.85 ms | 63.31 ms | 70933.19 ms | COMPLETED |
|
||||
| decode_throughput_1k_to_1k_c64 | fixed / - | decode_throughput | 1024 | 1024 | 64 | 1/1 | 1294.83 | -% | 647.42 | 12716.49 ms | 93.44 ms | 101163.20 ms | COMPLETED |
|
||||
| long_prefill_latency_128k_c1 | fixed / - | prefill_latency | 131072 | 1 | 1 | 1/1 | 2710.18 | -% | 0.02 | 48344.46 ms | 0.00 ms | 48344.53 ms | COMPLETED |
|
||||
| mid_prefill_latency_32k_c1 | fixed / - | prefill_latency | 32768 | 1 | 1 | 1/1 | 2652.84 | -% | 0.08 | 12334.69 ms | 0.00 ms | 12334.76 ms | COMPLETED |
|
||||
| mid_prefill_throughput_32k_c16 | fixed / - | prefill_throughput | 32768 | 1 | 16 | 1/1 | 3112.86 | -% | 0.09 | 162087.46 ms | 0.00 ms | 162087.50 ms | COMPLETED |
|
||||
| short_prefill_latency_1k_c1 | fixed / - | prefill_latency | 1024 | 1 | 1 | 1/1 | 1971.59 | -% | 1.92 | 501.90 ms | 0.00 ms | 501.95 ms | COMPLETED |
|
||||
| decode_control_1k_to_1k_c32 | mixed / control | mixed_interference | 1024 | 1024 | 32 | 1/1 | 911.37 | -% | 455.68 | 9442.64 ms | 65.88 ms | 72007.64 ms | COMPLETED |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | mixed / decode_background | mixed_interference | 1024 | 1024 | 32 | 1/1 | 691.89 | -% | 345.95 | 10193.90 ms | 109.73 ms | 117629.64 ms | COMPLETED |
|
||||
| long_prefill_injection_128k_to_1_c1 | mixed / prefill_injection | mixed_interference | 131072 | 1 | 1 | 1/1 | 2859.75 | -% | 0.02 | 45752.14 ms | 0.00 ms | 45752.21 ms | COMPLETED |
|
||||
|
||||
## Mixed-interference A/B
|
||||
|
||||
| Metric | Control | With 128K prefill | Change |
|
||||
|---|---:|---:|---:|
|
||||
| Output TPS | 455.68 | 345.95 | -24.08% |
|
||||
| TTFT P95 (ms) | 9442.64 | 10193.90 | 7.96% |
|
||||
| TPOT P95 (ms) | 65.88 | 109.73 | 66.55% |
|
||||
| E2E P95 (ms) | 72007.64 | 117629.64 | 63.36% |
|
||||
|
||||
The fixed map is descriptive and never stops on SLO. With one repetition, CV is intentionally unavailable.
|
||||
@ -1,53 +0,0 @@
|
||||
[2026-07-30 22:09:49] Starting worker node=10.101.0.13 rank=1 container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker
|
||||
[2026-07-30 22:09:55] Starting head node=10.101.0.11 rank=0 container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head
|
||||
[2026-07-30 22:09:56] Waiting for SGLang health at 10.101.0.11:30002
|
||||
[2026-07-30 22:10:55] Still starting: checks=12/600
|
||||
[2026-07-30 22:12:00] Still starting: checks=24/600
|
||||
[2026-07-30 22:13:05] Still starting: checks=36/600
|
||||
[2026-07-30 22:14:09] Still starting: checks=48/600
|
||||
[2026-07-30 22:15:14] Still starting: checks=60/600
|
||||
[2026-07-30 22:15:20] SGLang is healthy after 61 checks
|
||||
[2026-07-30 22:15:20] Verified worker NCCL transport: NET/IB with mlx5_0 and mlx5_3
|
||||
[2026-07-30 22:15:21] Verified head NCCL transport: NET/IB with mlx5_0 and mlx5_3
|
||||
[2026-07-30 22:15:21] Validating scenario file: /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv
|
||||
short_prefill_latency_1k_c1: stage=prefill_latency isl=1024 osl=1 c=1 prompts=1 warmup=1
|
||||
mid_prefill_latency_32k_c1: stage=prefill_latency isl=32768 osl=1 c=1 prompts=1 warmup=0
|
||||
long_prefill_latency_128k_c1: stage=prefill_latency isl=131072 osl=1 c=1 prompts=1 warmup=0
|
||||
mid_prefill_throughput_32k_c16: stage=prefill_throughput isl=32768 osl=1 c=16 prompts=16 warmup=0
|
||||
decode_latency_1k_to_1k_c1: stage=decode_latency isl=1024 osl=1024 c=1 prompts=1 warmup=1
|
||||
decode_throughput_1k_to_1k_c16: stage=decode_throughput isl=1024 osl=1024 c=16 prompts=16 warmup=1
|
||||
decode_throughput_1k_to_1k_c32: stage=decode_throughput isl=1024 osl=1024 c=32 prompts=32 warmup=1
|
||||
decode_throughput_1k_to_1k_c64: stage=decode_throughput isl=1024 osl=1024 c=64 prompts=64 warmup=1
|
||||
balanced_32k_to_1k_c8: stage=balanced isl=32768 osl=1024 c=8 prompts=8 warmup=0
|
||||
[2026-07-30 22:15:21] START case=short_prefill_latency_1k_c1 rep=1 isl=1024 osl=1 c=1
|
||||
[2026-07-30 22:15:55] DONE case=short_prefill_latency_1k_c1 rep=1 elapsed=34s
|
||||
[2026-07-30 22:16:00] START case=mid_prefill_latency_32k_c1 rep=1 isl=32768 osl=1 c=1
|
||||
[2026-07-30 22:16:46] DONE case=mid_prefill_latency_32k_c1 rep=1 elapsed=46s
|
||||
[2026-07-30 22:16:51] START case=long_prefill_latency_128k_c1 rep=1 isl=131072 osl=1 c=1
|
||||
[2026-07-30 22:18:13] DONE case=long_prefill_latency_128k_c1 rep=1 elapsed=81s
|
||||
[2026-07-30 22:18:18] START case=mid_prefill_throughput_32k_c16 rep=1 isl=32768 osl=1 c=16
|
||||
[2026-07-30 22:21:39] DONE case=mid_prefill_throughput_32k_c16 rep=1 elapsed=201s
|
||||
[2026-07-30 22:21:44] START case=decode_latency_1k_to_1k_c1 rep=1 isl=1024 osl=1024 c=1
|
||||
[2026-07-30 22:22:51] DONE case=decode_latency_1k_to_1k_c1 rep=1 elapsed=66s
|
||||
[2026-07-30 22:22:56] START case=decode_throughput_1k_to_1k_c16 rep=1 isl=1024 osl=1024 c=16
|
||||
[2026-07-30 22:24:25] DONE case=decode_throughput_1k_to_1k_c16 rep=1 elapsed=89s
|
||||
[2026-07-30 22:24:30] START case=decode_throughput_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||
[2026-07-30 22:26:15] DONE case=decode_throughput_1k_to_1k_c32 rep=1 elapsed=105s
|
||||
[2026-07-30 22:26:20] START case=decode_throughput_1k_to_1k_c64 rep=1 isl=1024 osl=1024 c=64
|
||||
[2026-07-30 22:28:36] DONE case=decode_throughput_1k_to_1k_c64 rep=1 elapsed=136s
|
||||
[2026-07-30 22:28:41] START case=balanced_32k_to_1k_c8 rep=1 isl=32768 osl=1024 c=8
|
||||
[2026-07-30 22:31:23] DONE case=balanced_32k_to_1k_c8 rep=1 elapsed=161s
|
||||
[2026-07-30 22:31:28] Fixed quick map complete: completed=9/9 failed=0
|
||||
[2026-07-30 22:31:28] START case=decode_control_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||
[2026-07-30 22:34:26] DONE case=decode_control_1k_to_1k_c32 rep=1 elapsed=178s
|
||||
[2026-07-30 22:34:26] START mixed background rep=1; injection delay=10s
|
||||
[2026-07-30 22:34:26] START case=decode_with_128k_prefill_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||
[2026-07-30 22:35:06] START case=long_prefill_injection_128k_to_1_c1 rep=1 isl=131072 osl=1 c=1
|
||||
[2026-07-30 22:36:24] DONE case=long_prefill_injection_128k_to_1_c1 rep=1 elapsed=78s
|
||||
[2026-07-30 22:38:09] DONE case=decode_with_128k_prefill_1k_to_1k_c32 rep=1 elapsed=223s
|
||||
[2026-07-30 22:38:14] Mixed-interference A/B complete: failed_repetitions=0
|
||||
[2026-07-30 22:38:20] Stopped head container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head on 10.101.0.11
|
||||
[2026-07-30 22:38:22] Stopped worker container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker on 10.101.0.13
|
||||
summarized 12 case runs into /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916
|
||||
[2026-07-30 22:38:22] Quick performance map complete: status=COMPLETED
|
||||
[2026-07-30 22:38:22] Results: /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916
|
||||
@ -1,43 +0,0 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"workflow_stage": "quick_performance_map",
|
||||
"run_id": "dsv4pro-phase1-full-20260730-220916",
|
||||
"status": "COMPLETED",
|
||||
"started_at": "2026-07-30T22:09:47+08:00",
|
||||
"updated_at": "2026-07-30T22:38:22+08:00",
|
||||
"suites": [
|
||||
"fixed",
|
||||
"mixed"
|
||||
],
|
||||
"engine": "sglang",
|
||||
"model_name": "DeepSeek-V4-Pro",
|
||||
"model_path": "/data/hf_models/DeepSeek-V4-Pro",
|
||||
"docker_image": "lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45",
|
||||
"head_node": "10.101.0.11",
|
||||
"worker_node": "10.101.0.13",
|
||||
"head_ip": "10.101.0.11",
|
||||
"sglang_port": 30002,
|
||||
"dist_init_port": 20002,
|
||||
"tp_size": 16,
|
||||
"ep_size": 2,
|
||||
"nnodes": 2,
|
||||
"mem_fraction_static": 0.9,
|
||||
"cuda_graph_max_bs_decode": 64,
|
||||
"max_running_requests": 256,
|
||||
"nccl_socket_ifname": "eth0",
|
||||
"nccl_ib_hca": "=mlx5_0:1,mlx5_3:1",
|
||||
"nccl_cross_nic": "1",
|
||||
"enable_rdma": true,
|
||||
"require_nccl_ib": true,
|
||||
"rdma_device_paths": "/dev/infiniband/rdma_cm,/dev/infiniband/uverbs0,/dev/infiniband/uverbs3",
|
||||
"git_commit": "75182c6ededf94511c6678552aaf489b9fc6fba6",
|
||||
"git_dirty": false,
|
||||
"scenario_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv",
|
||||
"case_ids": "",
|
||||
"notes": [
|
||||
"The fixed quick map does not stop on SLO.",
|
||||
"Profiler is disabled; these results are eligible for performance comparison.",
|
||||
"Speculative decoding is not enabled."
|
||||
],
|
||||
"ended_at": "2026-07-30T22:38:22+08:00"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@ -1 +0,0 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
File diff suppressed because one or more lines are too long
@ -1 +0,0 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -1,13 +0,0 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,balanced_32k_to_1k_c8,,balanced,1,32768,1024,8,8,0,COMPLETED,,0,2026-07-30T22:28:41+0800,2026-07-30T22:31:22+0800,161.0,8,0,128.62811431800947,7.996993535425267,,262144,8192,0.06219480120979978,2037.9992460427193,63.68747643883498,2101.686722481554,,128579.77483438299,128578.26417451724,128588.8987001963,128589.40238882786,49665.467494261975,49563.69844998699,82083.64989476977,83746.94262376754,77.14008537646238,77.23808966229743,109.32309288714016,112.17132956471158,77.14005810044779,43.40646349010058,43.84165157971437,45.188912986195646,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1,1024,1024,32,64,1,COMPLETED,,0,2026-07-30T22:31:28+0800,2026-07-30T22:34:26+0800,178.0,64,0,143.81865712802391,31.98259215252821,,65536,65536,0.44500485040010307,455.68496680970554,455.68496680970554,911.3699336194111,,71870.21023202942,71867.41446750239,72007.63789927005,72012.73389439622,7370.452653370194,6938.839260517852,9442.636363586644,9443.245556704933,63.04961640142642,63.471088219448355,65.88327712923635,67.80910156527888,63.049588087913975,60.98580302204937,61.753195300116204,63.56921844591854,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,decode_latency_1k_to_1k_c1,,decode_latency,1,1024,1024,1,1,1,COMPLETED,,0,2026-07-30T22:21:44+0800,2026-07-30T22:22:50+0800,66.0,1,0,32.59594450396253,0.9987527008467322,,1024,1024,0.03067866310419184,31.414951018692445,31.414951018692445,62.82990203738489,,32555.28760998277,32555.28760998277,32555.28760998277,32555.28760998277,363.2723209448159,363.2723209448159,363.2723209448159,363.2723209448159,31.468245639333286,31.468245639333286,31.468245639333286,31.468245639333286,31.468210095789587,31.49253799347207,31.659404194215313,32.3411487112753,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c16,,decode_throughput,1,1024,1024,16,16,1,COMPLETED,,0,2026-07-30T22:22:56+0800,2026-07-30T22:24:25+0800,89.0,16,0,55.48426674498478,15.986115557256861,,16384,16384,0.28837003602370287,295.29091688827174,295.29091688827174,590.5818337765435,,55436.11873718692,55437.53465998452,55443.82167501317,55447.12010782387,4706.02819519263,4801.126986538293,4950.267374995747,4950.924708603998,49.58953132159755,49.49641186020903,50.01878498413362,50.915034893211796,49.58950283761395,49.32670452399179,49.8873026604997,52.25673661392648,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1,1024,1024,32,32,1,COMPLETED,,0,2026-07-30T22:24:30+0800,2026-07-30T22:26:15+0800,105.0,32,0,70.97485623596003,31.975931620502816,,32768,32768,0.4508638931738606,461.68462661003326,461.68462661003326,923.3692532200665,,70921.47343050237,70921.31606096518,70933.18627287517,70935.98751158977,6650.527383468216,6178.440199000761,8021.852347906679,8023.242657405207,62.825949215087164,63.29258629570958,63.30661669023356,65.29997366317116,62.82592070965087,61.42424049903639,62.237933481810614,64.6031521842815,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c64,,decode_throughput,1,1024,1024,64,64,1,COMPLETED,,0,2026-07-30T22:26:20+0800,2026-07-30T22:28:36+0800,136.0,64,0,101.22691119497176,63.94603747793439,,65536,65536,0.6322429405825737,647.4167711565555,647.4167711565555,1294.833542313111,,101141.56026639376,101148.19569600513,101163.1970004586,101167.44335912867,7978.397482784203,6505.179281026358,12716.491229846724,12719.49685954547,91.06858532122146,92.52174521165054,93.44459645826957,94.39186767905878,91.06855870121748,86.24391999910586,87.80462378927041,90.7129056059057,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,1024,1024,32,64,0,COMPLETED,,0,2026-07-30T22:34:26+0800,2026-07-30T22:38:09+0800,223.0,64,0,189.43961485201726,31.98654958092739,,65536,65536,0.3378385246929174,345.94664928554744,345.94664928554744,691.8932985710949,,94679.9942664975,94677.34680901049,117629.63820034638,117636.8834006862,7749.917391879535,7705.110649490962,10193.896457596566,10196.455221220385,84.97563721859039,86.47172644327328,109.7299457727384,110.56891953740994,84.97560842059728,60.77706499490887,61.61956858995836,64.70431641500909,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,131072,1,1,1,0,COMPLETED,,0,2026-07-30T22:35:06+0800,2026-07-30T22:36:24+0800,78.0,1,0,45.83367280900711,0.9982227218104723,,131072,1,0.021818020217735695,2859.731545979053,0.021818020217735695,2859.7533639992707,,45752.21362197772,45752.21362197772,45752.21362197772,45752.21362197772,45752.13837000774,45752.13837000774,45752.13837000774,45752.13837000774,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,long_prefill_latency_128k_c1,,prefill_latency,1,131072,1,1,1,0,COMPLETED,,0,2026-07-30T22:16:51+0800,2026-07-30T22:18:12+0800,81.0,1,0,48.363155752944294,0.9996148667368967,,131072,1,0.020676897204730506,2710.162270418437,0.020676897204730506,2710.1829473156417,,48344.52949295519,48344.52949295519,48344.52949295519,48344.52949295519,48344.461318978574,48344.461318978574,48344.461318978574,48344.461318978574,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,mid_prefill_latency_32k_c1,,prefill_latency,1,32768,1,1,1,0,COMPLETED,,0,2026-07-30T22:16:00+0800,2026-07-30T22:16:46+0800,46.0,1,0,12.352426828991156,0.9985701137731423,,32768,1,0.08095575175988892,2652.75807366804,0.08095575175988892,2652.8390294198,,12334.764264000114,12334.764264000114,12334.764264000114,12334.764264000114,12334.68782599084,12334.68782599084,12334.68782599084,12334.68782599084,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,1,32768,1,16,16,0,COMPLETED,,0,2026-07-30T22:18:18+0800,2026-07-30T22:21:39+0800,201.0,16,0,168.4313552599633,8.771898294558559,,524288,16,0.09499418902914482,3112.7695861070174,0.09499418902914482,3112.8645802960464,,92341.41987219118,92749.04042950948,162087.50443853205,167117.798359727,92341.37187193119,92748.99350100895,162087.46085499297,167117.75140058598,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.log
|
||||
dsv4pro-phase1-full-20260730-220916,fixed,short_prefill_latency_1k_c1,,prefill_latency,1,1024,1,1,1,1,COMPLETED,,0,2026-07-30T22:15:21+0800,2026-07-30T22:15:55+0800,34.0,1,0,0.5198853989713825,0.9654920680740482,,1024,1,1.9235008368739468,1969.6648569589215,1.9235008368739468,1971.5883577957954,,501.94522901438177,501.94522901438177,501.94522901438177,501.94522901438177,501.89953204244375,501.89953204244375,501.89953204244375,501.89953204244375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.log
|
||||
|
@ -1,12 +0,0 @@
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "balanced_32k_to_1k_c8", "role": "", "stage": "balanced", "repetition": 1, "isl": 32768, "osl": 1024, "concurrency": 8, "num_prompts": 8, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:28:41+0800", "ended_at": "2026-07-30T22:31:22+0800", "elapsed_s": 161.0, "completed": 8, "failed": 0, "duration_s": 128.62811431800947, "actual_concurrency": 7.996993535425267, "peak_concurrent_requests": null, "total_input_tokens": 262144, "total_output_tokens": 8192, "request_throughput": 0.06219480120979978, "input_token_throughput": 2037.9992460427193, "output_token_throughput": 63.68747643883498, "total_token_throughput": 2101.686722481554, "peak_output_token_throughput": null, "e2e_mean_ms": 128579.77483438299, "e2e_p50_ms": 128578.26417451724, "e2e_p95_ms": 128588.8987001963, "e2e_p99_ms": 128589.40238882786, "ttft_mean_ms": 49665.467494261975, "ttft_p50_ms": 49563.69844998699, "ttft_p95_ms": 82083.64989476977, "ttft_p99_ms": 83746.94262376754, "tpot_mean_ms": 77.14008537646238, "tpot_p50_ms": 77.23808966229743, "tpot_p95_ms": 109.32309288714016, "tpot_p99_ms": 112.17132956471158, "itl_mean_ms": 77.14005810044779, "itl_p50_ms": 43.40646349010058, "itl_p95_ms": 43.84165157971437, "itl_p99_ms": 45.188912986195646, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "decode_control_1k_to_1k_c32", "role": "control", "stage": "mixed_interference", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 64, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:31:28+0800", "ended_at": "2026-07-30T22:34:26+0800", "elapsed_s": 178.0, "completed": 64, "failed": 0, "duration_s": 143.81865712802391, "actual_concurrency": 31.98259215252821, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.44500485040010307, "input_token_throughput": 455.68496680970554, "output_token_throughput": 455.68496680970554, "total_token_throughput": 911.3699336194111, "peak_output_token_throughput": null, "e2e_mean_ms": 71870.21023202942, "e2e_p50_ms": 71867.41446750239, "e2e_p95_ms": 72007.63789927005, "e2e_p99_ms": 72012.73389439622, "ttft_mean_ms": 7370.452653370194, "ttft_p50_ms": 6938.839260517852, "ttft_p95_ms": 9442.636363586644, "ttft_p99_ms": 9443.245556704933, "tpot_mean_ms": 63.04961640142642, "tpot_p50_ms": 63.471088219448355, "tpot_p95_ms": 65.88327712923635, "tpot_p99_ms": 67.80910156527888, "itl_mean_ms": 63.049588087913975, "itl_p50_ms": 60.98580302204937, "itl_p95_ms": 61.753195300116204, "itl_p99_ms": 63.56921844591854, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_latency_1k_to_1k_c1", "role": "", "stage": "decode_latency", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 1, "num_prompts": 1, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:21:44+0800", "ended_at": "2026-07-30T22:22:50+0800", "elapsed_s": 66.0, "completed": 1, "failed": 0, "duration_s": 32.59594450396253, "actual_concurrency": 0.9987527008467322, "peak_concurrent_requests": null, "total_input_tokens": 1024, "total_output_tokens": 1024, "request_throughput": 0.03067866310419184, "input_token_throughput": 31.414951018692445, "output_token_throughput": 31.414951018692445, "total_token_throughput": 62.82990203738489, "peak_output_token_throughput": null, "e2e_mean_ms": 32555.28760998277, "e2e_p50_ms": 32555.28760998277, "e2e_p95_ms": 32555.28760998277, "e2e_p99_ms": 32555.28760998277, "ttft_mean_ms": 363.2723209448159, "ttft_p50_ms": 363.2723209448159, "ttft_p95_ms": 363.2723209448159, "ttft_p99_ms": 363.2723209448159, "tpot_mean_ms": 31.468245639333286, "tpot_p50_ms": 31.468245639333286, "tpot_p95_ms": 31.468245639333286, "tpot_p99_ms": 31.468245639333286, "itl_mean_ms": 31.468210095789587, "itl_p50_ms": 31.49253799347207, "itl_p95_ms": 31.659404194215313, "itl_p99_ms": 32.3411487112753, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c16", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 16, "num_prompts": 16, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:22:56+0800", "ended_at": "2026-07-30T22:24:25+0800", "elapsed_s": 89.0, "completed": 16, "failed": 0, "duration_s": 55.48426674498478, "actual_concurrency": 15.986115557256861, "peak_concurrent_requests": null, "total_input_tokens": 16384, "total_output_tokens": 16384, "request_throughput": 0.28837003602370287, "input_token_throughput": 295.29091688827174, "output_token_throughput": 295.29091688827174, "total_token_throughput": 590.5818337765435, "peak_output_token_throughput": null, "e2e_mean_ms": 55436.11873718692, "e2e_p50_ms": 55437.53465998452, "e2e_p95_ms": 55443.82167501317, "e2e_p99_ms": 55447.12010782387, "ttft_mean_ms": 4706.02819519263, "ttft_p50_ms": 4801.126986538293, "ttft_p95_ms": 4950.267374995747, "ttft_p99_ms": 4950.924708603998, "tpot_mean_ms": 49.58953132159755, "tpot_p50_ms": 49.49641186020903, "tpot_p95_ms": 50.01878498413362, "tpot_p99_ms": 50.915034893211796, "itl_mean_ms": 49.58950283761395, "itl_p50_ms": 49.32670452399179, "itl_p95_ms": 49.8873026604997, "itl_p99_ms": 52.25673661392648, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c32", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 32, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:24:30+0800", "ended_at": "2026-07-30T22:26:15+0800", "elapsed_s": 105.0, "completed": 32, "failed": 0, "duration_s": 70.97485623596003, "actual_concurrency": 31.975931620502816, "peak_concurrent_requests": null, "total_input_tokens": 32768, "total_output_tokens": 32768, "request_throughput": 0.4508638931738606, "input_token_throughput": 461.68462661003326, "output_token_throughput": 461.68462661003326, "total_token_throughput": 923.3692532200665, "peak_output_token_throughput": null, "e2e_mean_ms": 70921.47343050237, "e2e_p50_ms": 70921.31606096518, "e2e_p95_ms": 70933.18627287517, "e2e_p99_ms": 70935.98751158977, "ttft_mean_ms": 6650.527383468216, "ttft_p50_ms": 6178.440199000761, "ttft_p95_ms": 8021.852347906679, "ttft_p99_ms": 8023.242657405207, "tpot_mean_ms": 62.825949215087164, "tpot_p50_ms": 63.29258629570958, "tpot_p95_ms": 63.30661669023356, "tpot_p99_ms": 65.29997366317116, "itl_mean_ms": 62.82592070965087, "itl_p50_ms": 61.42424049903639, "itl_p95_ms": 62.237933481810614, "itl_p99_ms": 64.6031521842815, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c64", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 64, "num_prompts": 64, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:26:20+0800", "ended_at": "2026-07-30T22:28:36+0800", "elapsed_s": 136.0, "completed": 64, "failed": 0, "duration_s": 101.22691119497176, "actual_concurrency": 63.94603747793439, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.6322429405825737, "input_token_throughput": 647.4167711565555, "output_token_throughput": 647.4167711565555, "total_token_throughput": 1294.833542313111, "peak_output_token_throughput": null, "e2e_mean_ms": 101141.56026639376, "e2e_p50_ms": 101148.19569600513, "e2e_p95_ms": 101163.1970004586, "e2e_p99_ms": 101167.44335912867, "ttft_mean_ms": 7978.397482784203, "ttft_p50_ms": 6505.179281026358, "ttft_p95_ms": 12716.491229846724, "ttft_p99_ms": 12719.49685954547, "tpot_mean_ms": 91.06858532122146, "tpot_p50_ms": 92.52174521165054, "tpot_p95_ms": 93.44459645826957, "tpot_p99_ms": 94.39186767905878, "itl_mean_ms": 91.06855870121748, "itl_p50_ms": 86.24391999910586, "itl_p95_ms": 87.80462378927041, "itl_p99_ms": 90.7129056059057, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "decode_with_128k_prefill_1k_to_1k_c32", "role": "decode_background", "stage": "mixed_interference", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 64, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:34:26+0800", "ended_at": "2026-07-30T22:38:09+0800", "elapsed_s": 223.0, "completed": 64, "failed": 0, "duration_s": 189.43961485201726, "actual_concurrency": 31.98654958092739, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.3378385246929174, "input_token_throughput": 345.94664928554744, "output_token_throughput": 345.94664928554744, "total_token_throughput": 691.8932985710949, "peak_output_token_throughput": null, "e2e_mean_ms": 94679.9942664975, "e2e_p50_ms": 94677.34680901049, "e2e_p95_ms": 117629.63820034638, "e2e_p99_ms": 117636.8834006862, "ttft_mean_ms": 7749.917391879535, "ttft_p50_ms": 7705.110649490962, "ttft_p95_ms": 10193.896457596566, "ttft_p99_ms": 10196.455221220385, "tpot_mean_ms": 84.97563721859039, "tpot_p50_ms": 86.47172644327328, "tpot_p95_ms": 109.7299457727384, "tpot_p99_ms": 110.56891953740994, "itl_mean_ms": 84.97560842059728, "itl_p50_ms": 60.77706499490887, "itl_p95_ms": 61.61956858995836, "itl_p99_ms": 64.70431641500909, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "long_prefill_injection_128k_to_1_c1", "role": "prefill_injection", "stage": "mixed_interference", "repetition": 1, "isl": 131072, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:35:06+0800", "ended_at": "2026-07-30T22:36:24+0800", "elapsed_s": 78.0, "completed": 1, "failed": 0, "duration_s": 45.83367280900711, "actual_concurrency": 0.9982227218104723, "peak_concurrent_requests": null, "total_input_tokens": 131072, "total_output_tokens": 1, "request_throughput": 0.021818020217735695, "input_token_throughput": 2859.731545979053, "output_token_throughput": 0.021818020217735695, "total_token_throughput": 2859.7533639992707, "peak_output_token_throughput": null, "e2e_mean_ms": 45752.21362197772, "e2e_p50_ms": 45752.21362197772, "e2e_p95_ms": 45752.21362197772, "e2e_p99_ms": 45752.21362197772, "ttft_mean_ms": 45752.13837000774, "ttft_p50_ms": 45752.13837000774, "ttft_p95_ms": 45752.13837000774, "ttft_p99_ms": 45752.13837000774, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "long_prefill_latency_128k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 131072, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:16:51+0800", "ended_at": "2026-07-30T22:18:12+0800", "elapsed_s": 81.0, "completed": 1, "failed": 0, "duration_s": 48.363155752944294, "actual_concurrency": 0.9996148667368967, "peak_concurrent_requests": null, "total_input_tokens": 131072, "total_output_tokens": 1, "request_throughput": 0.020676897204730506, "input_token_throughput": 2710.162270418437, "output_token_throughput": 0.020676897204730506, "total_token_throughput": 2710.1829473156417, "peak_output_token_throughput": null, "e2e_mean_ms": 48344.52949295519, "e2e_p50_ms": 48344.52949295519, "e2e_p95_ms": 48344.52949295519, "e2e_p99_ms": 48344.52949295519, "ttft_mean_ms": 48344.461318978574, "ttft_p50_ms": 48344.461318978574, "ttft_p95_ms": 48344.461318978574, "ttft_p99_ms": 48344.461318978574, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "mid_prefill_latency_32k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 32768, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:16:00+0800", "ended_at": "2026-07-30T22:16:46+0800", "elapsed_s": 46.0, "completed": 1, "failed": 0, "duration_s": 12.352426828991156, "actual_concurrency": 0.9985701137731423, "peak_concurrent_requests": null, "total_input_tokens": 32768, "total_output_tokens": 1, "request_throughput": 0.08095575175988892, "input_token_throughput": 2652.75807366804, "output_token_throughput": 0.08095575175988892, "total_token_throughput": 2652.8390294198, "peak_output_token_throughput": null, "e2e_mean_ms": 12334.764264000114, "e2e_p50_ms": 12334.764264000114, "e2e_p95_ms": 12334.764264000114, "e2e_p99_ms": 12334.764264000114, "ttft_mean_ms": 12334.68782599084, "ttft_p50_ms": 12334.68782599084, "ttft_p95_ms": 12334.68782599084, "ttft_p99_ms": 12334.68782599084, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "mid_prefill_throughput_32k_c16", "role": "", "stage": "prefill_throughput", "repetition": 1, "isl": 32768, "osl": 1, "concurrency": 16, "num_prompts": 16, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:18:18+0800", "ended_at": "2026-07-30T22:21:39+0800", "elapsed_s": 201.0, "completed": 16, "failed": 0, "duration_s": 168.4313552599633, "actual_concurrency": 8.771898294558559, "peak_concurrent_requests": null, "total_input_tokens": 524288, "total_output_tokens": 16, "request_throughput": 0.09499418902914482, "input_token_throughput": 3112.7695861070174, "output_token_throughput": 0.09499418902914482, "total_token_throughput": 3112.8645802960464, "peak_output_token_throughput": null, "e2e_mean_ms": 92341.41987219118, "e2e_p50_ms": 92749.04042950948, "e2e_p95_ms": 162087.50443853205, "e2e_p99_ms": 167117.798359727, "ttft_mean_ms": 92341.37187193119, "ttft_p50_ms": 92748.99350100895, "ttft_p95_ms": 162087.46085499297, "ttft_p99_ms": 167117.75140058598, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.log"}
|
||||
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "short_prefill_latency_1k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 1024, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:15:21+0800", "ended_at": "2026-07-30T22:15:55+0800", "elapsed_s": 34.0, "completed": 1, "failed": 0, "duration_s": 0.5198853989713825, "actual_concurrency": 0.9654920680740482, "peak_concurrent_requests": null, "total_input_tokens": 1024, "total_output_tokens": 1, "request_throughput": 1.9235008368739468, "input_token_throughput": 1969.6648569589215, "output_token_throughput": 1.9235008368739468, "total_token_throughput": 1971.5883577957954, "peak_output_token_throughput": null, "e2e_mean_ms": 501.94522901438177, "e2e_p50_ms": 501.94522901438177, "e2e_p95_ms": 501.94522901438177, "e2e_p99_ms": 501.94522901438177, "ttft_mean_ms": 501.89953204244375, "ttft_p50_ms": 501.89953204244375, "ttft_p95_ms": 501.89953204244375, "ttft_p99_ms": 501.89953204244375, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.log"}
|
||||
@ -1,12 +0,0 @@
|
||||
# DeepSeek-V4-Pro Pro6000D Two-Node SGLang Quick Map
|
||||
|
||||
Profiler: disabled. Speculative decoding: disabled.
|
||||
|
||||
## Aggregate results
|
||||
|
||||
| Case | Suite / role | Stage | ISL | OSL | C | Reps | Total TPS | CV | Output TPS | TTFT P95 | TPOT P95 | E2E P95 | Status |
|
||||
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| long_context_decode_128k_to_1k_c1 | fixed / - | long_context_decode | 131072 | 1024 | 1 | 1/1 | 1604.09 | -% | 12.43 | 49325.72 ms | 32.24 ms | 82312.21 ms | COMPLETED |
|
||||
| long_output_decode_1k_to_4k_c16 | fixed / - | long_output_decode | 1024 | 4096 | 16 | 1/1 | 387.52 | -% | 310.02 | 6241.01 ms | 50.33 ms | 211363.56 ms | COMPLETED |
|
||||
|
||||
The fixed map is descriptive and never stops on SLO. With one repetition, CV is intentionally unavailable.
|
||||
@ -1,42 +0,0 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"workflow_stage": "quick_performance_map",
|
||||
"run_id": "dsv4pro-phase1-long-decode-20260730-234236",
|
||||
"status": "COMPLETED",
|
||||
"started_at": "2026-07-30T23:48:13+08:00",
|
||||
"updated_at": "2026-07-30T23:54:25+08:00",
|
||||
"suites": [
|
||||
"fixed"
|
||||
],
|
||||
"engine": "sglang",
|
||||
"model_name": "DeepSeek-V4-Pro",
|
||||
"model_path": "/data/hf_models/DeepSeek-V4-Pro",
|
||||
"docker_image": "lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45",
|
||||
"head_node": "10.101.0.11",
|
||||
"worker_node": "10.101.0.13",
|
||||
"head_ip": "10.101.0.11",
|
||||
"sglang_port": 30002,
|
||||
"dist_init_port": 20002,
|
||||
"tp_size": 16,
|
||||
"ep_size": 2,
|
||||
"nnodes": 2,
|
||||
"mem_fraction_static": 0.9,
|
||||
"cuda_graph_max_bs_decode": 64,
|
||||
"max_running_requests": 256,
|
||||
"nccl_socket_ifname": "eth0",
|
||||
"nccl_ib_hca": "=mlx5_0:1,mlx5_3:1",
|
||||
"nccl_cross_nic": "1",
|
||||
"enable_rdma": true,
|
||||
"require_nccl_ib": true,
|
||||
"rdma_device_paths": "/dev/infiniband/rdma_cm,/dev/infiniband/uverbs0,/dev/infiniband/uverbs3",
|
||||
"git_commit": "06b017483cb1cfc6aace3c60e94576fb667ec9bc",
|
||||
"git_dirty": false,
|
||||
"scenario_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv",
|
||||
"case_ids": "long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1",
|
||||
"notes": [
|
||||
"The fixed quick map does not stop on SLO.",
|
||||
"Profiler is disabled; these results are eligible for performance comparison.",
|
||||
"Speculative decoding is not enabled."
|
||||
],
|
||||
"ended_at": "2026-07-30T23:54:25+08:00"
|
||||
}
|
||||
@ -1,3 +0,0 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log
|
||||
dsv4pro-phase1-long-decode-20260730-234236,fixed,long_context_decode_128k_to_1k_c1,,long_context_decode,1,131072,1024,1,1,0,COMPLETED,,0,2026-07-30T23:52:24+0800,2026-07-30T23:54:19+0800,115.0,1,0,82.34937581099803,0.9995487118435447,,131072,1024,0.012143382875118852,1591.6574802075781,12.434824064121704,1604.0923042716997,,82312.21251300303,82312.21251300303,82312.21251300303,82312.21251300303,49325.72139299009,49325.72139299009,49325.72139299009,49325.72139299009,32.244859354851364,32.244859354851364,32.244859354851364,32.244859354851364,32.2448224535841,32.23047900246456,32.468517863890156,32.707680857274674,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-long-decode-20260730-234236/cases/long_context_decode_128k_to_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-long-decode-20260730-234236/cases/long_context_decode_128k_to_1k_c1/rep1/bench.log
|
||||
dsv4pro-phase1-long-decode-20260730-234236,fixed,long_output_decode_1k_to_4k_c16,,long_output_decode,1,1024,4096,16,16,1,COMPLETED,,0,2026-07-30T23:48:13+0800,2026-07-30T23:52:19+0800,246.0,16,0,211.3935806370573,15.996984262440824,,16384,65536,0.07568820184502423,77.50471868930481,310.01887475721924,387.52359344652405,,211353.73641450133,211353.21495501557,211363.55966723931,211364.98273107863,5862.642711690569,5970.386928500375,6241.009955512709,6241.542907894473,50.18097526320165,50.15488793663095,50.32745551037411,50.57979576238554,50.18097181628469,50.06324249552563,50.968476399430074,52.59070861677173,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-long-decode-20260730-234236/cases/long_output_decode_1k_to_4k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-long-decode-20260730-234236/cases/long_output_decode_1k_to_4k_c16/rep1/bench.log
|
||||
|
@ -1,138 +0,0 @@
|
||||
# Phase 2 Hardware Attribution Analysis
|
||||
|
||||
## Run
|
||||
|
||||
- Run ID: `dsv4pro-phase2-20260731-130125`
|
||||
- Status: `COMPLETED`
|
||||
- Time: `2026-07-31 13:01:25` to `13:27:51 CST`
|
||||
- Duration: 26 minutes 26 seconds
|
||||
- Topology: two Pro6000D nodes, SGLang TP16 / EP2
|
||||
- Bench result: 8 completed, 0 failed, no OOM
|
||||
|
||||
The command was run only on `174.1.51.5`:
|
||||
|
||||
```bash
|
||||
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
|
||||
|
||||
RUN_ID=dsv4pro-phase2-20260731-130125
|
||||
tmux new-session -d -s dsv4pro-phase2 \
|
||||
"RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
|
||||
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||
```
|
||||
|
||||
The entry script started the Worker and its collectors on `174.1.51.7` over SSH.
|
||||
The full Phase 2 command must not be run a second time on the Worker.
|
||||
|
||||
## Endpoint Results
|
||||
|
||||
| Case | Input TPS | Output TPS | TTFT P95 | TPOT P95 | E2E P95 |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| 128K -> 1, C=1 | 2,618.53 | 0.02 | 50.036 s | - | 50.036 s |
|
||||
| 32K -> 1, C=16 | 3,116.20 | 0.10 | 161.899 s | - | 161.899 s |
|
||||
| 1K -> 1K, C=32 | 447.41 | 447.41 | 10.144 s | 65.63 ms | 73.189 s |
|
||||
| 1K -> 4K, C=16 | 79.35 | 317.41 | 1.727 s | 50.01 ms | 206.427 s |
|
||||
| 128K -> 1K, C=1 | 1,610.89 | 12.59 | 48.489 s | 32.12 ms | 81.343 s |
|
||||
|
||||
## Mixed Prefill/Decode
|
||||
|
||||
| Decode metric | Control | With one 128K Prefill | Change |
|
||||
|---|---:|---:|---:|
|
||||
| Output TPS | 454.39 | 345.20 | -24.03% |
|
||||
| TTFT P95 | 9.437 s | 9.869 s | +4.58% |
|
||||
| TPOT P95 | 66.17 ms | 110.36 ms | +66.79% |
|
||||
| E2E P95 | 72.225 s | 117.941 s | +63.30% |
|
||||
|
||||
This reproduces the Phase 1 contention result. Prefill mainly damages ongoing
|
||||
Decode progress rather than its first-token latency. Global token-level ITL P95
|
||||
stayed near 62 ms because a small number of synchronized long stalls can remain
|
||||
below the top 5% of all token gaps. Per-request TPOT and E2E expose the impact
|
||||
more clearly.
|
||||
|
||||
## Hardware Evidence
|
||||
|
||||
### GPU
|
||||
|
||||
- During the 128K Prefill injection, average GPU utilization was 99.78% on the
|
||||
Head and 99.34% on the Worker; average power was about 274 W on both nodes.
|
||||
- During 32K -> 1, C=16 Prefill, average GPU utilization was about 82% on both
|
||||
nodes and power was about 265 W.
|
||||
- GPU clocks remained around 2.38 to 2.41 GHz. There is no evidence of a
|
||||
frequency collapse.
|
||||
- GPU memory stayed near 83.2 to 83.4 GiB of 85,651 MiB per GPU, leaving only
|
||||
about 2.3 GiB of headroom.
|
||||
- Head DCGM data is valid. During the injection window, `GRACT=0.995`,
|
||||
`SMACT=0.648`, `SMOCC=0.215`, `Tensor Active=0.108`, and
|
||||
`DRAM Active=0.412`.
|
||||
|
||||
### CPU
|
||||
|
||||
- Whole-machine CPU utilization stayed around 8% to 11% across 128 logical
|
||||
CPUs, with zero average I/O wait.
|
||||
- A few individual cores were saturated. During the injection window, 5 Head
|
||||
cores and 4 Worker cores averaged above 80%.
|
||||
- Therefore the machine is not globally CPU-bound, but scheduler, affinity, or
|
||||
NUMA hot threads remain plausible contributors.
|
||||
|
||||
### Inter-node RDMA
|
||||
|
||||
- NCCL logs on both nodes verified `NET/IB + GDRDMA` over
|
||||
`mlx5_0,mlx5_3`.
|
||||
- Both rails carried nearly identical traffic and all recorded RDMA error
|
||||
deltas were zero.
|
||||
- The largest average traffic occurred in 32K -> 1, C=16 Prefill:
|
||||
about 140.0 Gbit/s total transmit on the Head and 139.3 Gbit/s on the Worker,
|
||||
or about 70 Gbit/s per 400G rail.
|
||||
- This is only about 17.5% of each rail's nominal line rate. Raw RoCE bandwidth
|
||||
saturation is therefore not the primary ceiling, although collective
|
||||
latency and synchronization overhead are not ruled out.
|
||||
|
||||
### Intra-node GPU Communication
|
||||
|
||||
- RTX Pro 6000D has no NVLink. `nvidia-smi nvlink --status` reports no links.
|
||||
- `nvidia-smi topo -m` shows GPU0-3 and GPU4-7 connected through local PCIe
|
||||
switches (`PIX`), while traffic between the two four-GPU groups crosses the
|
||||
host bridge and CPU/NUMA interconnect (`SYS`).
|
||||
- NCCL logs show intra-node transfers through `P2P/IPC`; on this machine that
|
||||
means CUDA peer-to-peer over PCIe, not NVLink.
|
||||
- Phase 2 collected DCGM PCIe counters but did not run a dedicated GPU-to-GPU
|
||||
bandwidth/latency microbenchmark. A later short diagnostic should run
|
||||
`p2pBandwidthLatencyTest`, 8-GPU `all_reduce_perf`, and 16-GPU
|
||||
`all_reduce_perf`.
|
||||
|
||||
## Preliminary Attribution
|
||||
|
||||
The first successful Phase 2 run supports three conclusions:
|
||||
|
||||
1. Prefill/Decode resource contention is real and reproducible.
|
||||
2. Raw dual-rail RoCE bandwidth, whole-machine CPU capacity, and clock
|
||||
throttling do not explain the observed slowdown.
|
||||
3. The next useful target is the TP16 execution timeline: GPU kernels,
|
||||
scheduler gaps, PCIe/RDMA collectives, and synchronization between ranks.
|
||||
|
||||
This is a bottleneck direction, not a final kernel-level root cause. Phase 3
|
||||
should capture short Prefill, Decode, and mixed timelines rather than repeat the
|
||||
full benchmark matrix.
|
||||
|
||||
## Collection Limitations
|
||||
|
||||
- Worker DCGM exited because `nvidia-dcgm` hostengine was inactive on
|
||||
`174.1.51.7`. Before another hardware run, execute on the Worker:
|
||||
|
||||
```bash
|
||||
systemctl start nvidia-dcgm
|
||||
systemctl is-active nvidia-dcgm
|
||||
dcgmi discovery -l
|
||||
```
|
||||
|
||||
The main Phase 2 entry still runs only on the Head.
|
||||
|
||||
- Current Case windows include benchmark-client startup, tokenizer/data
|
||||
preparation, warm-up, the measured run, and client exit. Hardware averages
|
||||
are therefore diluted outside the true main benchmark interval. Future
|
||||
profiling must add explicit main-run start/end markers.
|
||||
- `pidstat -durwt` at one-second intervals generated about 850 MB on the Head
|
||||
and 735 MB on the Worker. The next run should use process-level sampling
|
||||
and/or a five-second interval.
|
||||
- Worker teardown logged a Gloo connection reset after the Head stopped. It
|
||||
occurred after all eight benchmark rows completed and is a cleanup artifact,
|
||||
not a benchmark failure.
|
||||
@ -1,20 +0,0 @@
|
||||
{
|
||||
"run_id": "dsv4pro-phase2-20260731-130125",
|
||||
"status": "COMPLETED",
|
||||
"created_at": "2026-07-31T13:01:25+08:00",
|
||||
"completed_at": "2026-07-31T13:27:51+08:00",
|
||||
"experiment": "dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution",
|
||||
"git_commit": "ca1f2f63375cd6e2bd249a91cbdbe7576aafad7e",
|
||||
"git_dirty": false,
|
||||
"phase1_entry": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh",
|
||||
"head_node": "10.101.0.11",
|
||||
"worker_node": "10.101.0.13",
|
||||
"fixed_case_ids": "long_prefill_latency_128k_c1,mid_prefill_throughput_32k_c16,decode_throughput_1k_to_1k_c32,long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1",
|
||||
"run_mixed_case": true,
|
||||
"sample_interval_s": 1,
|
||||
"numastat_interval_s": 5,
|
||||
"clock_skew_tolerance_s": 2,
|
||||
"idle_baseline_s": 15,
|
||||
"post_run_cooldown_s": 15,
|
||||
"dry_run": false
|
||||
}
|
||||
@ -1,34 +0,0 @@
|
||||
# Phase 2 Hardware Attribution
|
||||
|
||||
- Generated: `2026-07-31T13:27:51+08:00`
|
||||
- Bench rows: `8`
|
||||
- Failed bench rows: `0`
|
||||
- GPU summary rows: `16`
|
||||
- RDMA summary rows: `4`
|
||||
- Case windows: `8`
|
||||
- Collector status counts: `{"EXITED_BEFORE_STOP": 1, "STARTED": 18, "STOPPED": 17}`
|
||||
|
||||
## Bench
|
||||
|
||||
| Run | Case | Status | Input TPS | Output TPS | TTFT P95 (ms) | TPOT P95 (ms) |
|
||||
|---|---|---|---:|---:|---:|---:|
|
||||
| fixed_decode_throughput_1k_to_1k_c32 | decode_throughput_1k_to_1k_c32 | COMPLETED | 447.41274460374615 | 447.41274460374615 | 10144.207949569682 | 65.62598442892012 |
|
||||
| fixed_long_context_decode_128k_to_1k_c1 | long_context_decode_128k_to_1k_c1 | COMPLETED | 1610.8926328335153 | 12.585098694011839 | 48488.74244495528 | 32.11543163640164 |
|
||||
| fixed_long_output_decode_1k_to_4k_c16 | long_output_decode_1k_to_4k_c16 | COMPLETED | 79.35259394563253 | 317.41037578253014 | 1726.5846392256208 | 50.00692894707874 |
|
||||
| fixed_long_prefill_latency_128k_c1 | long_prefill_latency_128k_c1 | COMPLETED | 2618.53322910782 | 0.019977823098051604 | 50036.02714801673 | 0.0 |
|
||||
| fixed_mid_prefill_throughput_32k_c16 | mid_prefill_throughput_32k_c16 | COMPLETED | 3116.2035105433815 | 0.0950989840864069 | 161899.06251549837 | 0.0 |
|
||||
| mixed_prefill_decode_interference | decode_control_1k_to_1k_c32 | COMPLETED | 454.3923063408163 | 454.3923063408163 | 9436.576361441985 | 66.170705189718 |
|
||||
| mixed_prefill_decode_interference | decode_with_128k_prefill_1k_to_1k_c32 | COMPLETED | 345.2031620939666 | 345.2031620939666 | 9868.98645934416 | 110.36397890971698 |
|
||||
| mixed_prefill_decode_interference | long_prefill_injection_128k_to_1_c1 | COMPLETED | 2844.468616501081 | 0.021701573307045603 | 45999.09703899175 | 0.0 |
|
||||
|
||||
## Machine-readable summaries
|
||||
|
||||
- `gpu_summary.csv`
|
||||
- `rdma_summary.csv`
|
||||
- `bench_summary.csv`
|
||||
- `case_windows.csv`
|
||||
- `case_gpu_summary.csv`
|
||||
- `case_rdma_summary.csv`
|
||||
- `summary.json`
|
||||
|
||||
The final bottleneck conclusion is written only after aligning these files with `markers.csv`, raw DCGM/sysstat logs, and SGLang service logs.
|
||||
@ -1 +0,0 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -1 +0,0 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -1,45 +0,0 @@
|
||||
{
|
||||
"generated_at": "2026-07-31T13:27:51+08:00",
|
||||
"bench_rows": 8,
|
||||
"failed_bench_rows": 0,
|
||||
"gpu_summary_rows": 16,
|
||||
"rdma_summary_rows": 4,
|
||||
"case_windows": 8,
|
||||
"case_gpu_summary_rows": 128,
|
||||
"case_rdma_summary_rows": 32,
|
||||
"collector_status_counts": {
|
||||
"STARTED": 18,
|
||||
"STOPPED": 17,
|
||||
"EXITED_BEFORE_STOP": 1
|
||||
},
|
||||
"collector_files": {
|
||||
"head": {
|
||||
"dcgm_dmon.log": 1638191,
|
||||
"docker_top.log": 11692641,
|
||||
"gpu_samples.csv": 726449,
|
||||
"markers.csv": 1860,
|
||||
"mpstat.log": 15587271,
|
||||
"numastat.log": 3280062,
|
||||
"perf_stat.log": 744719,
|
||||
"pidstat.log": 850254976,
|
||||
"rdma.csv": 230202,
|
||||
"sar_net.log": 2118579,
|
||||
"static_after.log": 1195680,
|
||||
"static_before.log": 1306492
|
||||
},
|
||||
"worker": {
|
||||
"dcgm_dmon.log": 243,
|
||||
"docker_top.log": 11989768,
|
||||
"gpu_samples.csv": 758155,
|
||||
"markers.csv": 1902,
|
||||
"mpstat.log": 15587260,
|
||||
"numastat.log": 3579303,
|
||||
"perf_stat.log": 747768,
|
||||
"pidstat.log": 734994477,
|
||||
"rdma.csv": 242603,
|
||||
"sar_net.log": 2383072,
|
||||
"static_after.log": 1195267,
|
||||
"static_before.log": 1306098
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,51 +0,0 @@
|
||||
# Phase 2 Hardware Attribution Summary
|
||||
|
||||
## Run
|
||||
|
||||
- Run ID: `dsv4pro-phase2-20260731-163620`
|
||||
- Status: `COMPLETED`
|
||||
- Duration: 2026-07-31 16:36:20 to 17:05:04 CST, 28m44s
|
||||
- Benchmarks: 8/8 successful
|
||||
- Precise benchmark windows: 8/8
|
||||
- Collectors: 18 started, 18 stopped
|
||||
- Run commit: `5f24b7d22f98108f6cc234edba6768d55ea0a962`
|
||||
- Git dirty: `false`
|
||||
|
||||
## Main Result
|
||||
|
||||
Injecting one 128K prefill request into the `1K -> 1K, C=32` decode workload caused:
|
||||
|
||||
| Metric | Control | Treatment | Delta |
|
||||
|---|---:|---:|---:|
|
||||
| Output TPS | 453.55 | 344.89 | -23.96% |
|
||||
| TTFT P95 | 9.443 s | 9.892 s | +4.76% |
|
||||
| TPOT P95 | 66.24 ms | 110.45 ms | +66.75% |
|
||||
| E2E P95 | 72.270 s | 118.076 s | +63.38% |
|
||||
|
||||
The dominant impact is sustained decode slowdown, not only first-token queueing.
|
||||
|
||||
## Attribution
|
||||
|
||||
- GPU utilization is generally 94-99%, with stable 2.39-2.42 GHz clocks. There is no evidence of frequency collapse.
|
||||
- Whole-host CPU active is about 9.6-10.4%; CPU capacity is not globally saturated.
|
||||
- Head and Worker show similar GPU, CPU and communication behavior. There is no obvious slow-node imbalance.
|
||||
- The highest measured RDMA traffic is about 83.5 Gbit/s per 400G rail. Both rails are balanced, with zero wait/error/discard/retry deltas.
|
||||
- PCIe P2P bandwidth is about 53.5 GB/s within a switch and 52.3-52.4 GB/s across NUMA, only about a 2.3% difference.
|
||||
- 16-GPU 1 GiB AllReduce bus bandwidth is 39.345/39.685/39.530 GB/s for `NCCL_CROSS_NIC=0/1/2`. The spread is below 1%.
|
||||
|
||||
The evidence excludes raw rail saturation, whole-host CPU saturation, GPU clock throttling, severe PCIe cross-NUMA collapse, and `NCCL_CROSS_NIC` selection as the primary cause.
|
||||
|
||||
## Phase 3 Target
|
||||
|
||||
Capture short Control/Treatment timelines and identify:
|
||||
|
||||
1. Attention/indexer and MoE kernel serialization.
|
||||
2. NCCL collective placement and compute/communication overlap.
|
||||
3. Scheduler or CPU launch gaps.
|
||||
4. Slow-rank synchronization around injected prefill.
|
||||
|
||||
Do not repeat Phase 2's long-running DCGM, CPU, RDMA or communication microbenchmarks.
|
||||
|
||||
## Cleanup
|
||||
|
||||
After the run, both nodes had no related Docker containers, no GPU compute processes, and no listeners on the Phase 2 service ports. Worker Gloo `Connection closed by peer` tracebacks occurred during orchestrated shutdown after all benchmarks completed and did not affect the results.
|
||||
@ -1,9 +0,0 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||
fixed_decode_throughput_1k_to_1k_c32,fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1,1024,1024,32,32,1,COMPLETED,,0,2026-07-31T16:49:35+0800,2026-07-31T16:51:24+0800,109.0,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.98790900100721,bench_main_marker_plus_duration,32,0,72.98790900100721,31.973027693348868,,32768,32768,0.43842878139663394,448.95107215015315,448.95107215015315,897.9021443003063,,72926.38861777596,72926.03143348242,72939.17302195623,72943.4486198437,8317.972617309351,8423.306765995221,10166.788510262268,10167.903182043228,63.155831867513776,63.05236280694347,65.35758210354297,67.36198566667542,63.155803331197355,61.26605049939826,62.53751848998945,65.70577920065261,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log,fixed_decode_throughput_1k_to_1k_c32
|
||||
fixed_long_context_decode_128k_to_1k_c1,fixed,long_context_decode_128k_to_1k_c1,,long_context_decode,1,131072,1024,1,1,0,COMPLETED,,0,2026-07-31T16:55:45+0800,2026-07-31T16:57:39+0800,114.0,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.1598442849936,bench_main_marker_plus_duration,1,0,81.1598442849936,0.9995611458686556,,131072,1024,0.012321364201839645,1614.985848663526,12.617076942683797,1627.6029256062097,,81124.22695202986,81124.22695202986,81124.22695202986,81124.22695202986,48279.420554987155,48279.420554987155,48279.420554987155,48279.420554987155,32.106360114411245,32.106360114411245,32.106360114411245,32.106360114411245,32.10634595800085,32.09696803241968,32.28448228328489,32.53332086256705,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/bench.log,fixed_long_context_decode_128k_to_1k_c1
|
||||
fixed_long_output_decode_1k_to_4k_c16,fixed,long_output_decode_1k_to_4k_c16,,long_output_decode,1,1024,4096,16,16,1,COMPLETED,,0,2026-07-31T16:51:33+0800,2026-07-31T16:55:36+0800,243.0,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074682698585,bench_main_marker_plus_duration,16,0,206.37074682698585,15.99550577130734,,16384,65536,0.07753036826199912,79.3910971002871,317.5643884011484,396.95548550143553,,206312.77949375362,206311.27990750247,206327.66740498482,206328.1800113793,1697.1331631830253,1714.9575955118053,1723.5681610036409,1724.2807810165687,49.96719080111614,49.96277789426234,49.98305734157135,50.02649938860528,49.96718302440562,49.99568348284811,50.960358188604005,53.57307427912019,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/bench.log,fixed_long_output_decode_1k_to_4k_c16
|
||||
fixed_long_prefill_latency_128k_c1,fixed,long_prefill_latency_128k_c1,,prefill_latency,1,131072,1,1,1,0,COMPLETED,,0,2026-07-31T16:44:30+0800,2026-07-31T16:45:54+0800,84.0,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567456041,bench_main_marker_plus_duration,1,0,49.622567456041,0.9997500965858968,,131072,1,0.02015212132838284,2641.3788467537956,0.02015212132838284,2641.398998875124,,49610.16660701716,49610.16660701716,49610.16660701716,49610.16660701716,49610.14223104576,49610.14223104576,49610.14223104576,49610.14223104576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/bench.log,fixed_long_prefill_latency_128k_c1
|
||||
fixed_mid_prefill_throughput_32k_c16,fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,1,32768,1,16,16,0,COMPLETED,,0,2026-07-31T16:46:03+0800,2026-07-31T16:49:26+0800,203.0,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.2866359280306,bench_main_marker_plus_duration,16,0,168.2866359280306,8.764467555247327,,524288,16,0.09507587998159613,3115.446435236942,0.09507587998159613,3115.5415111169236,,92183.92253580896,92587.60345654446,161938.1903584872,166972.64813809306,92183.87291699764,92587.55126551841,161938.13879448862,166972.5966172962,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/bench.log,fixed_mid_prefill_throughput_32k_c16
|
||||
mixed_prefill_decode_interference,mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1,1024,1024,32,64,1,COMPLETED,,0,2026-07-31T16:57:48+0800,2026-07-31T17:00:49+0800,181.0,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.4964013720164,bench_main_marker_plus_duration,64,0,144.4964013720164,31.983334150914263,,65536,65536,0.4429176048144437,453.54762732999035,453.54762732999035,907.0952546599807,,72210.57326071605,72215.36969349836,72269.92528844276,72271.96539751603,7325.20390184618,6846.627883001929,9442.68154159945,9444.044730461319,63.426558513069274,63.83972124386403,66.23843437823616,68.16205457722045,63.42653002822952,61.320325010456145,62.66533228044864,65.59292251302395,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/bench.log,mixed_prefill_decode_interference
|
||||
mixed_prefill_decode_interference,mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,1024,1024,32,64,0,COMPLETED,,0,2026-07-31T17:00:49+0800,2026-07-31T17:04:32+0800,223.0,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.02009954402456,bench_main_marker_plus_duration,64,0,190.02009954402456,31.986201956249488,,65536,65536,0.3368064754916742,344.8898309034744,344.8898309034744,689.7796618069488,,94969.08249627813,94968.0481730029,118076.02923363738,118079.88882219535,7606.184888919415,7407.497464475455,9891.837346865213,9893.77336708887,85.3987268889137,86.89664765445467,110.451060616212,111.28696239483178,85.39869778190757,61.08463852433488,61.97826337593142,65.3158738068305,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log,mixed_prefill_decode_interference
|
||||
mixed_prefill_decode_interference,mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,131072,1,1,1,0,COMPLETED,,0,2026-07-31T17:01:29+0800,2026-07-31T17:02:47+0800,78.0,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.06182585703209,bench_main_marker_plus_duration,1,0,46.06182585703209,0.9982293388832256,,131072,1,0.02170995138368649,2845.5667477625557,0.02170995138368649,2845.5884577139395,,45980.265973019414,45980.265973019414,45980.265973019414,45980.265973019414,45980.19455798203,45980.19455798203,45980.19455798203,45980.19455798203,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log,mixed_prefill_decode_interference
|
||||
|
@ -1,17 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,cpu_active_pct_mean,cpu_active_pct_p95,cpu_active_pct_max,iowait_pct_mean,iowait_pct_p95,iowait_pct_max,hot_cores_ge80_mean,hot_cores_ge80_p95,hot_cores_ge80_max
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,15,10.130666666666666,10.418000000000003,10.459999999999994,0.0,0.0,0.0,10.466666666666667,12.0,12.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,15,9.843333333333332,10.085999999999997,10.099999999999994,0.0,0.0,0.0,10.4,12.0,12.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,17,9.899411764705883,10.292000000000003,10.299999999999997,0.0,0.0,0.0,10.588235294117647,12.0,12.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,17,9.651176470588235,10.089999999999996,10.129999999999995,0.0,0.0,0.0,10.352941176470589,12.0,12.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,41,10.361707317073172,10.519999999999996,10.620000000000005,0.0,0.0,0.0,11.024390243902438,12.0,12.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,41,9.994390243902439,10.090000000000003,10.159999999999997,0.0,0.0,0.0,11.414634146341463,12.0,12.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,10,10.003,10.272999999999996,10.299999999999997,0.0,0.0,0.0,10.1,12.0,12.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,10,9.688,10.180500000000004,10.230000000000004,0.0,0.0,0.0,10.2,12.0,12.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,34,9.985588235294117,10.267000000000003,10.319999999999993,0.0,0.0,0.0,11.058823529411764,12.0,12.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,34,9.84235294117647,10.1135,10.14,0.0,0.0,0.0,11.205882352941176,12.0,12.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,29,10.262758620689656,10.436,10.489999999999995,0.0,0.0,0.0,10.620689655172415,12.0,12.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,29,9.976551724137929,10.108000000000004,10.120000000000005,0.0,0.0,0.0,11.241379310344827,12.0,12.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,38,10.341052631578947,11.3725,12.36,0.0,0.0,0.0,10.868421052631579,12.149999999999999,13.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,38,9.868684210526316,10.091500000000002,10.14,0.0,0.0,0.0,10.868421052631579,12.0,12.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,10,10.212,10.630499999999998,10.950000000000003,0.0,0.0,0.0,11.6,12.549999999999999,13.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,9,10.035555555555556,10.120000000000001,10.14,0.0,0.0,0.0,11.666666666666666,12.0,12.0
|
||||
|
@ -1,17 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,gpus,samples,gr_engine_active_mean,gr_engine_active_p95,gr_engine_active_max,sm_active_mean,sm_active_p95,sm_active_max,sm_occupancy_mean,sm_occupancy_p95,sm_occupancy_max,tensor_active_mean,tensor_active_p95,tensor_active_max,dram_active_mean,dram_active_p95,dram_active_max,pcie_tx_bytes_per_s_mean,pcie_tx_bytes_per_s_p95,pcie_tx_bytes_per_s_max,pcie_rx_bytes_per_s_mean,pcie_rx_bytes_per_s_p95,pcie_rx_bytes_per_s_max
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,8,584,0.9710171232876712,0.999,1.0,0.5252157534246575,0.7,0.707,0.14078253424657533,0.291,0.294,0.05895205479452054,0.11499999999999999,0.12,0.41691609589041095,0.499,0.508,5205379813.392123,14778814130.95,15202540154.0,5200970877.481164,14816565450.150002,15254205047.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,8,576,0.9785503472222222,0.999,0.999,0.5226267361111111,0.6872499999999999,0.698,0.14059895833333333,0.28824999999999995,0.293,0.05846874999999999,0.113,0.117,0.4153368055555555,0.495,0.548,5236209935.171875,14647937823.25,16403294625.0,5231738841.786458,14714002991.25,16357346290.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,8,648,0.9613811728395062,0.9990000000000001,0.999,0.5782654320987655,0.76,0.768,0.21059722222222224,0.282,0.284,0.09468981481481482,0.162,0.168,0.3985987654320987,0.44500000000000006,0.479,6763589500.305555,13241413192.75,14068752272.0,6821390469.66821,13261233168.150002,14425674313.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,8,648,0.965466049382716,0.9990000000000001,1.0,0.5789398148148148,0.76,0.768,0.21097993827160494,0.281,0.285,0.09508796296296296,0.162,0.166,0.4005216049382716,0.44600000000000006,0.47,6780236898.856482,13234577683.95,13889246455.0,6837638258.785494,13289110201.45,13879127175.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,8,1648,0.9890279126213591,0.996,0.997,0.4903646844660194,0.5,0.54,0.13298240291262137,0.135,0.202,0.04484769417475728,0.048,0.102,0.41422026699029124,0.427,0.469,2620560854.958738,2632307969.6499996,11954964997.0,2634422729.365898,2682732685.6,10279631228.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,8,1640,0.9897713414634146,0.996,1.0,0.4890847560975609,0.498,0.683,0.13288231707317075,0.135,0.268,0.044690243902439024,0.047,0.129,0.4142451219512195,0.427,0.468,2618598082.628049,2632407781.75,14221424532.0,2633369965.667683,2681936017.75,13880734362.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,8,400,0.9295425,0.999,0.999,0.6828425,0.761,0.769,0.25086,0.282,0.284,0.1358075,0.163,0.168,0.37432499999999996,0.44904999999999995,0.505,10536527950.9525,13519030283.25,15163087193.0,10581948383.175,13517471298.849998,15273700217.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,8,392,0.9359923469387755,0.9989999999999999,1.0,0.6856326530612245,0.76,0.765,0.250795918367347,0.282,0.285,0.13560204081632654,0.162,0.165,0.37558673469387754,0.45324999999999993,0.485,10617939301.517857,13498405363.0,13917911496.0,10632410279.880102,13503375869.85,14629076948.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,8,1344,0.9804598214285715,0.999,1.0,0.7134583333333333,0.739,0.744,0.2776354166666667,0.287,0.288,0.1272514880952381,0.138,0.141,0.4404389880952381,0.467,0.509,12664097012.861607,14053446776.599998,15685065820.0,12719741431.203125,14071304248.15,15724306505.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,8,1344,0.9839747023809524,0.999,1.0,0.7149203869047619,0.739,0.743,0.27822619047619046,0.287,0.289,0.12745982142857143,0.137,0.14,0.44101190476190477,0.467,0.514,12698564254.736607,14063118317.15,15508567216.0,12754602448.509672,14079354109.05,15319006836.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,8,1152,0.9889574652777777,0.999,1.0,0.538584201388889,0.699,0.705,0.1456640625,0.292,0.295,0.060828125,0.11499999999999999,0.121,0.4266675347222222,0.5,0.558,5451444350.41493,14941958481.6,16367702095.0,5456132661.967882,14993495237.7,16348072773.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,8,1152,0.9896744791666667,0.999,1.0,0.5361883680555555,0.697,0.704,0.1450859375,0.291,0.295,0.06014930555555556,0.114,0.119,0.4251788194444444,0.5,0.55,5464990127.207465,14978142467.55,16515378482.0,5460547526.944445,14998798408.1,15300436571.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,8,1520,0.9768513157894737,0.999,1.0,0.5805618421052632,0.748,0.763,0.17383618421052632,0.291,0.295,0.08014671052631578,0.15404999999999996,0.168,0.4163407894736842,0.499,0.51,6865359213.189474,14886672661.75,15342998825.0,6868476165.281579,14922047967.4,15374888508.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,8,1520,0.978421052631579,0.999,1.0,0.578371052631579,0.748,0.763,0.17389407894736844,0.291,0.295,0.0797078947368421,0.154,0.175,0.4157486842105263,0.498,0.511,6864820605.633553,14934618195.95,15291598078.0,6872189917.845395,14950386000.25,15356103677.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,8,368,0.9974673913043478,0.9990000000000001,1.0,0.7229048913043478,0.756,0.763,0.2642554347826087,0.285,0.288,0.13833152173913044,0.16,0.165,0.40672282608695653,0.4545999999999999,0.48,11151168359.230978,13665801629.75,14505291121.0,11197667600.644022,13730586781.449997,14645337945.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,8,368,0.9973179347826087,0.9990000000000001,0.999,0.7197690217391305,0.757,0.763,0.2623777173913044,0.285,0.288,0.13687228260869566,0.15964999999999996,0.166,0.40697826086956523,0.45654999999999984,0.486,11068919985.380434,13568556767.3,14853005782.0,11109834217.782608,13585937423.0,14369923387.0
|
||||
|
@ -1,17 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,gpu_util_pct_mean,gpu_util_pct_p95,gpu_util_pct_max,memory_util_pct_mean,memory_util_pct_p95,memory_util_pct_max,memory_used_mib_mean,memory_used_mib_p95,memory_used_mib_max,power_w_mean,power_w_p95,power_w_max,temperature_c_mean,temperature_c_p95,temperature_c_max,sm_clock_mhz_mean,sm_clock_mhz_p95,sm_clock_mhz_max,memory_clock_mhz_mean,memory_clock_mhz_p95,memory_clock_mhz_max,gpus
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,408,97.09313725490196,100.0,100.0,50.85539215686274,61.0,63.0,83300.0,83409.0,83409.0,230.21735294117644,299.2755,312.84,43.713235294117645,45.0,46.0,2402.950980392157,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,432,97.72222222222223,100.0,100.0,51.21527777777778,60.44999999999999,64.0,83300.0,83409.0,83409.0,221.8317592592593,289.236,307.91,45.844907407407405,47.0,49.0,2418.5486111111113,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,456,97.70394736842105,100.0,100.0,50.24122807017544,57.0,59.0,83219.06140350878,83453.0,83453.0,257.11598684210526,325.23,342.99,46.23026315789474,52.0,54.0,2412.627192982456,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,472,97.43008474576271,100.0,100.0,49.84110169491525,57.0,58.0,83221.32203389831,83453.0,83453.0,257.90165254237286,327.982,372.66,48.74576271186441,54.0,57.0,2417.531779661017,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,1184,98.82516891891892,100.0,100.0,50.710304054054056,53.0,59.0,83300.0,83409.0,83409.0,220.82934966216217,242.39100000000002,275.9,43.5304054054054,45.0,45.0,2402.7592905405404,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,1216,99.2467105263158,100.0,100.0,50.49753289473684,53.0,54.0,83300.0,83409.0,83409.0,215.73606907894737,236.6025,270.17,45.59950657894737,47.0,48.0,2412.5715460526317,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,288,94.44444444444444,100.0,100.0,46.74305555555556,56.0,58.0,83028.21527777778,83408.2,83443.0,293.3987847222222,330.7185,368.5,44.83680555555556,51.0,53.0,2392.215277777778,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,288,94.85069444444444,100.0,100.0,46.420138888888886,55.0,57.0,83057.76388888889,83408.2,83443.0,297.9095486111111,335.86899999999997,363.48,46.53819444444444,52.0,54.0,2398.0243055555557,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,952,98.20588235294117,100.0,100.0,53.52205882352941,58.0,59.0,83231.06722689075,83405.0,83409.0,299.4595693277311,316.97,332.42,50.438025210084035,55.0,56.0,2417.686974789916,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,968,99.07438016528926,100.0,100.0,53.974173553719005,58.0,59.0,83231.94008264462,83405.0,83409.0,302.563285123967,322.293,339.05,53.75929752066116,59.0,61.0,2419.3326446280994,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,824,98.52427184466019,100.0,100.0,51.5376213592233,61.0,65.0,83364.0,83473.0,83473.0,231.9156432038835,297.50299999999993,324.46,44.297330097087375,46.0,50.0,2405.38713592233,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,856,98.60280373831776,100.0,100.0,51.441588785046726,61.0,64.0,83364.0,83473.0,83473.0,227.7883528037383,293.6475,322.45,46.43691588785047,49.0,52.0,2418.589953271028,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,1072,99.60914179104478,100.0,100.0,51.59794776119403,60.0,65.0,83259.71082089552,83433.0,83433.0,248.31210820895524,315.4865,376.49,46.649253731343286,52.0,55.0,2411.4869402985073,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,1112,98.92086330935251,100.0,100.0,51.085431654676256,60.0,64.0,83256.90647482015,83433.0,83433.0,247.74912769784174,321.8905,373.73,49.64118705035971,55.0,59.0,2420.7724820143885,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,264,99.95075757575758,100.0,100.0,49.45075757575758,57.0,59.0,83281.49242424243,83413.0,83433.0,301.19901515151514,332.613,376.49,49.92424242424242,54.0,55.0,2421.1136363636365,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,272,99.88602941176471,100.0,100.0,49.39705882352941,57.0,59.0,83282.94117647059,83413.0,83433.0,307.1604411764706,340.2845,373.73,52.86764705882353,57.0,59.0,2421.4595588235293,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||
|
@ -1,33 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,iface,samples,rx_gbps_mean,rx_gbps_p95,rx_gbps_max,tx_gbps_mean,tx_gbps_p95,tx_gbps_max,ifutil_pct_mean,ifutil_pct_p95,ifutil_pct_max,rx_errors_s_mean,rx_errors_s_p95,rx_errors_s_max,tx_errors_s_mean,tx_errors_s_p95,tx_errors_s_max,rx_drops_s_mean,rx_drops_s_p95,rx_drops_s_max,tx_drops_s_mean,tx_drops_s_p95,tx_drops_s_max
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,eth0,30,0.00015682133333333332,0.00016183999999999998,0.00016464,5.1274666666666665e-05,5.4591999999999995e-05,5.504e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,eth3,30,3.0933333333333335e-07,8.8e-07,8.8e-07,4.2666666666666665e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,eth0,30,5.1695999999999994e-05,5.4959999999999996e-05,5.608e-05,0.000156736,0.000161872,0.00016287999999999999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,eth3,30,3.0933333333333335e-07,8.8e-07,8.8e-07,4.2666666666666665e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,eth0,34,0.00015644235294117647,0.00015915199999999997,0.0001596,5.2367058823529417e-05,5.536e-05,5.568e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,eth3,34,3.435294117647059e-07,8.8e-07,8.8e-07,5.6470588235294123e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,eth0,34,5.264470588235294e-05,5.5552000000000006e-05,5.568e-05,0.0001562070588235294,0.00016,0.00016,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,eth3,34,3.435294117647059e-07,8.8e-07,8.8e-07,5.6470588235294123e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,eth0,82,0.0001566419512195122,0.00015984000000000001,0.00016272,4.933853658536585e-05,5.2560000000000005e-05,5.44e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,eth3,82,3.3365853658536586e-07,8.8e-07,8.8e-07,5.463414634146342e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,eth0,82,4.958634146341463e-05,5.4e-05,5.456e-05,0.00015623219512195122,0.00016128,0.00016287999999999999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,eth3,82,3.35609756097561e-07,8.8e-07,9.6e-07,5.6585365853658546e-08,3.2e-07,4.0000000000000003e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,eth0,20,0.00021578399999999998,0.0004841319999999994,0.00074848,5.9391999999999996e-05,8.969599999999994e-05,0.0001172,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,eth3,20,3.1200000000000004e-07,6.279999999999994e-07,8.8e-07,4e-08,2.1199999999999975e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,eth0,20,5.9592e-05,8.761999999999995e-05,0.00011336,0.000215264,0.00047957599999999935,0.00074,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,eth3,20,2.96e-07,6.279999999999994e-07,8.8e-07,3.2e-08,1.7599999999999967e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,eth0,68,0.0001568870588235294,0.000160908,0.00016112,5.124470588235294e-05,5.5707999999999996e-05,5.592e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,eth3,68,3.223529411764706e-07,8.8e-07,8.8e-07,4.705882352941176e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,eth0,68,5.149411764705883e-05,5.6268000000000004e-05,5.7520000000000005e-05,0.00015664235294117647,0.00016011200000000001,0.00016063999999999998,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,eth3,68,3.223529411764706e-07,8.8e-07,8.8e-07,4.705882352941176e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,eth0,58,0.00015664000000000002,0.000159888,0.00016,5.07751724137931e-05,5.4496e-05,5.456e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,eth3,58,3.3931034482758625e-07,8.8e-07,8.8e-07,5.517241379310345e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,eth0,58,5.0982068965517244e-05,5.4528000000000004e-05,5.568e-05,0.00015630896551724138,0.000160848,0.00016168,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,eth3,58,3.3931034482758625e-07,8.8e-07,8.8e-07,5.517241379310345e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,eth0,76,0.00015644631578947367,0.000160012,0.00016056,5.221263157894737e-05,5.579599999999999e-05,5.616e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,eth3,76,3.452631578947368e-07,8.8e-07,9.6e-07,6.105263157894737e-08,3.2e-07,4.0000000000000003e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,eth0,76,5.24042105263158e-05,5.641199999999999e-05,5.664e-05,0.0001559515789473684,0.00016033200000000004,0.00016424,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,eth3,76,3.431578947368421e-07,8.8e-07,8.8e-07,5.8947368421052637e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,eth0,20,0.000156048,0.000158632,0.00015928,5.1064000000000004e-05,5.358e-05,5.376e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,eth3,20,2.96e-07,6.279999999999994e-07,8.8e-07,3.2e-08,1.7599999999999967e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,eth0,18,5.1271111111111107e-05,5.4688e-05,5.52e-05,0.00015478222222222221,0.00016222399999999998,0.00016424,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,eth3,18,2.9333333333333337e-07,6.559999999999998e-07,8.8e-07,3.5555555555555554e-08,1.919999999999999e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
|
@ -1,17 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,node0_mib_mean,node0_mib_p95,node0_mib_max,node1_mib_mean,node1_mib_p95,node1_mib_max,total_mib_mean,total_mib_p95,total_mib_max,processes_mean,processes_p95,processes_max,numa_imbalance_pct_mean,numa_imbalance_pct_p95,numa_imbalance_pct_max
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,13,17941.21923076923,17941.472,17941.61,27565.280769230772,27565.394,27565.4,45506.51384615385,45506.69,45506.69,19.0,19.0,19.0,21.148755918904936,21.149743646540415,21.150655911887284
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,13,19029.48769230769,19029.536,19029.56,24814.318461538463,24814.381999999998,24814.4,43843.8,43843.854,43843.86,18.0,18.0,18.0,13.194182003167738,13.194329421344158,13.194370476132915
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,13,17946.50153846154,17948.994,17949.0,27569.42769230769,27571.648,27571.66,45515.943076923075,45520.61,45520.61,19.0,19.0,19.0,21.141880372077402,21.145764039410054,21.145782042670753
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,14,19037.215,19039.1035,19039.11,24816.07357142857,24818.423499999997,24818.43,43853.295,43857.54,43857.54,18.0,18.0,18.0,13.177706662062775,13.17809858730606,13.178146962903641
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,35,17941.668285714288,17942.12,17942.13,27565.19857142857,27566.617,27570.09,45506.88085714286,45506.89,45506.91,19.0,19.0,19.0,21.147417939082427,21.15369613423204,21.16893559148322
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,37,19029.40054054054,19029.452,19029.6,24814.58216216216,24814.65,24814.66,43843.96270270271,43843.97,43843.97,18.0,18.0,18.0,13.19493327010614,13.195223881414025,13.195246689567572
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,8,17869.37125,17877.1655,17878.8,27486.31625,27493.9225,27494.43,45355.7125,45371.1115,45373.25,19.0,19.0,19.0,21.203382762054954,21.207110348436384,21.207680920856813
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,8,18970.3375,18981.5545,18984.05,24731.71375,24743.9825,24745.68,43702.05125,43712.965,43715.31,18.0,18.0,18.0,13.183305003875798,13.21223924964188,13.21285380339291
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,28,17940.274285714284,17940.69,17940.72,27563.883928571428,27564.22,27564.23,45504.17285714286,45504.46,45504.46,19.0,19.0,19.0,21.148850835716523,21.149789612547433,21.149814841959135
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,30,19032.02333333333,19037.573,19037.75,24805.59,24810.55,24811.49,43837.58133333333,43841.73,43841.73,18.0,18.0,18.0,13.17036295064116,13.184651928653366,13.219751626067305
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,24,17946.96125,17951.6865,17951.9,27573.83541666667,27579.2765,27588.36,45520.80666666667,45520.8485,45520.85,19.0,19.0,19.0,21.148294318886816,21.17226656215764,21.212158491133064
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,26,19039.328461538462,19039.4575,19039.46,24818.277692307693,24818.565,24818.74,43857.62038461539,43857.65,43857.65,18.0,18.0,18.0,13.176613733571411,13.177963869896237,13.178761906998588
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,32,17955.7271875,17958.81,17958.83,27573.1746875,27577.874499999998,27578.04,45528.9121875,45535.71,45535.71,19.0,19.0,19.0,21.123824304701028,21.129656288079975,21.129778209910725
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,34,19042.789411764705,19046.17,19046.17,24823.40088235294,24827.86,24827.86,43866.178529411765,43873.26,43873.26,18.0,18.0,18.0,13.177832389566726,13.183982855860464,13.185098115743662
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,8,17951.77,17953.336,17953.98,27569.775,27571.0035,27571.7,45521.55625,45524.051,45525.71,19.0,19.0,19.0,21.128462660935995,21.129739766090218,21.129778209910725
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,8,19038.4325,19039.3115,19039.56,24819.52625,24820.626,24820.99,43857.942500000005,43858.878000000004,43859.48,18.0,18.0,18.0,13.181406637484468,13.184049863000734,13.184128037997716
|
||||
|
@ -1,17 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,ipc,cache_misses_mean,cache_misses_p95,cache_misses_max,context_switches_mean,context_switches_p95,context_switches_max,cpu_migrations_mean,cpu_migrations_p95,cpu_migrations_max,page_faults_mean,page_faults_p95,page_faults_max
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,84,3.0487843541376685,6727288.428571428,15567285.149999999,18088446.0,41240.92857142857,47221.399999999994,58274.0,934.9285714285714,1975.849999999999,4063.0,46455.857142857145,161971.65,174542.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,84,3.0681062936795103,4913412.214285715,13142249.399999999,14049236.0,39682.0,45243.25,55120.0,188.57142857142858,247.7,249.0,28090.428571428572,113948.55,114568.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,96,2.8870888885724733,6295623.4375,12277537.0,21573640.0,44877.375,72712.0,151042.0,1255.1875,3534.25,9802.0,51319.875,105487.75,183145.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,96,2.9020975501844197,4051617.5625,7033576.5,10195935.0,43686.6875,70482.5,149831.0,177.5,304.0,364.0,12704.625,37867.0,65872.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,246,3.0979852154775953,5103378.804878049,5773629.0,14670415.0,43749.9756097561,44169.0,78897.0,998.9268292682926,1151.0,6568.0,18694.80487804878,28750.0,152065.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,246,3.1168786636888095,2962364.2682926827,2913232.0,13101053.0,40973.46341463415,41420.0,67313.0,165.17073170731706,209.0,213.0,11960.536585365853,14773.0,135313.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,60,2.727435494137309,9447284.2,31640896.59999995,53106505.0,50995.8,134631.5499999998,217846.0,2372.3,12092.349999999977,21775.0,108507.0,295694.8499999996,444909.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,60,2.7489508087655112,7473178.3,22657424.999999966,37078962.0,48210.5,120052.39999999983,191525.0,296.6,871.5499999999988,1394.0,107749.9,291126.8499999996,447550.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,204,2.8023469789284796,4623449.029411765,16572071.899999999,19670078.0,39421.294117647056,35324.499999999985,263082.0,861.4117647058823,365.9499999999998,20998.0,39086.14705882353,123447.0999999998,215308.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,204,2.8098589749897895,4072417.588235294,9610375.949999992,16711381.0,39129.382352941175,45578.64999999992,229122.0,164.7058823529412,246.5999999999997,423.0,36854.5,114042.04999999983,216180.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,174,3.0424513549285557,5137320.0,7469347.399999995,13520315.0,42398.8275862069,43356.2,91980.0,970.7586206896551,1053.6,5258.0,29120.41379310345,88587.9999999999,167311.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,174,3.0719083836401526,3223721.2413793104,5225881.199999996,9479153.0,40484.58620689655,39252.8,101396.0,167.3448275862069,197.2,236.0,16289.206896551725,65153.199999999895,95763.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,228,2.980275800109699,5456416.5,10599088.849999998,17028720.0,41223.05263157895,44802.899999999994,127813.0,1211.421052631579,1458.199999999999,15346.0,36016.86842105263,119273.24999999994,173922.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,228,3.0050223544930565,3784084.789473684,6404245.849999995,13894635.0,39834.92105263158,46921.44999999995,97372.0,167.5,262.15000000000003,299.0,22792.86842105263,79725.24999999994,157546.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,54,2.808104894473792,5020429.555555556,9669376.799999999,10387880.0,33399.666666666664,37769.399999999994,41129.0,338.8888888888889,811.1999999999997,1178.0,43405.88888888889,91275.39999999998,112743.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,60,2.8298366086279145,3857952.9,5373284.599999999,5873843.0,33911.7,40936.75,42415.0,130.4,172.39999999999998,185.0,24117.8,52089.24999999995,73750.0
|
||||
|
@ -1,17 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,process_cpu_pct_mean,process_cpu_pct_p95,process_cpu_pct_max,process_wait_pct_mean,process_wait_pct_p95,process_wait_pct_max,process_read_kib_s_mean,process_read_kib_s_p95,process_read_kib_s_max,process_write_kib_s_mean,process_write_kib_s_p95,process_write_kib_s_max,process_major_faults_s_mean,process_major_faults_s_p95,process_major_faults_s_max,process_rss_kib_mean,process_rss_kib_p95,process_rss_kib_max,process_voluntary_switches_s_mean,process_voluntary_switches_s_p95,process_voluntary_switches_s_max,process_involuntary_switches_s_mean,process_involuntary_switches_s_p95,process_involuntary_switches_s_max
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,1140,1193.5866666666666,1223.7400000000002,1225.0000000000002,0.013333333333333334,0.05999999999999979,0.2,0.0,0.0,0.0,174.50666666666666,1036.1599999999996,1273.6000000000001,0.0,0.0,0.0,46588476.53333333,46588620.0,46588620.0,277.09333333333336,711.1799999999982,1938.0000000000002,13.32,31.619999999999997,34.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,1080,1183.24,1210.06,1210.2,0.013333333333333334,0.05999999999999979,0.2,0.0,0.0,0.0,174.50666666666666,1036.1599999999996,1273.6000000000001,0.0,0.0,0.0,44887948.266666666,44888007.2,44888024.0,230.50666666666666,625.8599999999982,1744.6,10.986666666666666,31.319999999999997,31.599999999999998
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,1292,1173.3882352941175,1222.08,1222.4,0.011764705882352941,0.03999999999999986,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46598011.52941176,46602992.8,46602996.0,581.7294117647059,3130.9999999999986,5014.2,12.835294117647058,15.639999999999997,16.599999999999998
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,1224,1160.7176470588236,1214.16,1215.6000000000001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44897810.35294118,44902216.0,44902216.0,545.2352941176471,3440.1999999999994,4371.400000000001,8.105882352941176,11.359999999999996,12.799999999999999
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,3116,1220.1268292682926,1225.6,1225.8,0.024390243902439025,0.2,0.2,0.0,0.0,0.0,9.990243902439024,0.0,409.59999999999997,0.0,0.0,0.0,46588905.26829268,46588920.0,46588920.0,225.06829268292682,227.6,647.8000000000001,7.86829268292683,10.200000000000001,28.2
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,2952,1206.5365853658536,1211.0,1211.4,0.01951219512195122,0.2,0.2,0.0,0.0,0.0,9.990243902439024,0.0,409.59999999999997,0.0,0.0,0.0,44888251.90243903,44888308.0,44888316.0,176.34634146341463,176.2,602.6,5.590243902439025,7.400000000000001,18.200000000000003
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,1129,1147.0800000000002,1211.02,1211.1999999999998,0.02,0.10999999999999979,0.2,0.0,0.0,0.0,297.30909090909097,1635.2000000000003,3270.4000000000005,0.0,0.0,0.0,46439570.4,46454466.6,46456380.0,1940.6666666666672,11299.580000000002,13680.2,16.5,35.83999999999999,41.99999999999999
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,720,1141.52,1210.62,1210.8,0.06000000000000001,0.2,0.2,0.0,0.0,0.0,327.0400000000001,1798.7199999999968,3270.4000000000005,0.0,0.0,0.0,44743050.0,44757778.6,44759728.0,872.5400000000006,4761.229999999995,8583.800000000007,14.919999999999998,27.009999999999973,37.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,2584,1184.6000000000001,1210.47,1211.0,0.03529411764705883,0.2,0.2,0.0,0.0,0.0,34.4470588235294,100.79999999999939,883.1999999999999,0.0,0.0,0.0,46582295.52941176,46586356.0,46586356.0,377.5411764705882,1431.4499999999923,8289.0,10.841176470588234,15.229999999999995,22.4
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,2448,1182.3235294117646,1210.0,1210.2,0.011764705882352941,0.06999999999999958,0.2,0.0,0.0,0.0,34.44705882352941,271.3599999999999,620.8000000000001,0.0,0.0,0.0,44881817.88235294,44885672.0,44885672.0,348.7588235294118,1094.469999999994,8625.2,11.129411764705884,18.909999999999997,20.999999999999996
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,2204,1214.6758620689654,1220.12,1220.4,0.013793103448275864,0.11999999999999958,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46603182.06896552,46603218.4,46603220.0,214.6758620689655,201.43999999999997,1574.0,9.682758620689656,14.839999999999996,15.4
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,2088,1204.3999999999999,1210.1200000000001,1210.4,0.020689655172413796,0.2,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44902315.862068966,44902356.0,44902356.0,170.98620689655175,146.44,1531.9999999999998,6.931034482758621,11.84,12.6
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,2888,1200.7473684210527,1223.2599999999998,1223.6,0.021052631578947368,0.2,0.2,0.0,0.0,0.0,11.452631578947367,0.0,435.19999999999993,0.0,0.0,0.0,46611446.84210526,46618484.0,46618484.0,281.61,426.2999999999979,4255.0,8.715789473684211,17.47999999999999,27.599999999999998
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,2736,1189.6052631578948,1210.43,1211.4,0.021052631578947368,0.2,0.2,0.0,0.0,0.0,11.452631578947367,0.0,435.19999999999993,0.0,0.0,0.0,44911026.94736842,44918376.0,44918376.0,249.65263157894734,289.9599999999987,4796.0,6.526315789473684,11.919999999999998,18.8
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,749,1210.66,1217.6599999999999,1223.6,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46604025.6,46607275.2,46610400.0,34.617999999999995,109.58999999999983,175.2,8.1,10.79,11.6
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,648,1207.4,1210.92,1211.4,0.022222222222222223,0.11999999999999994,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44902633.333333336,44903531.2,44904240.0,22.133333333333336,88.47999999999996,137.60000000000002,9.244444444444445,12.280000000000001,12.600000000000001
|
||||
|
@ -1,33 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.344697344,bench_main_marker_plus_duration,head,mlx5_0,66,327229799692.0,327241990572.0,36.69282364341205,36.69419062713517,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.344697344,bench_main_marker_plus_duration,head,mlx5_3,66,327393677184.0,327405760832.0,36.71119950013029,36.712554459750265,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.149780736,bench_main_marker_plus_duration,worker,mlx5_0,68,327312761288.0,327298289132.0,36.80267266064959,36.801045427975055,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.149780736,bench_main_marker_plus_duration,worker,mlx5_3,68,327195798972.0,327191949064.0,36.78952155156224,36.789088672308345,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.07872896,bench_main_marker_plus_duration,head,mlx5_0,75,444257636548.0,444258274080.0,43.83469176160109,43.834754666583265,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.07872896,bench_main_marker_plus_duration,head,mlx5_3,75,445471520584.0,445472014020.0,43.95446512771776,43.95451381481548,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,80.561727488,bench_main_marker_plus_duration,worker,mlx5_0,77,450403115384.0,450403670936.0,44.72626190406251,44.7263170718964,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,80.561727488,bench_main_marker_plus_duration,worker,mlx5_3,77,444176313384.0,444176585108.0,44.10792342556576,44.10795040850254,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,205.0068928,bench_main_marker_plus_duration,head,mlx5_0,188,484528783604.0,484522438956.0,18.907804590812276,18.907557003117507,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,205.0068928,bench_main_marker_plus_duration,head,mlx5_3,188,484292016064.0,484262756928.0,18.898565192594344,18.897423411043476,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,204.622719744,bench_main_marker_plus_duration,worker,mlx5_0,194,483378692900.0,483384132588.0,18.898339089803788,18.89855176171067,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,204.622719744,bench_main_marker_plus_duration,worker,mlx5_3,194,483484778080.0,483489710268.0,18.902486632369254,18.902679462882155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.080653568,bench_main_marker_plus_duration,head,mlx5_0,45,428992803940.0,428992820060.0,71.37886398874002,71.3788666708999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.080653568,bench_main_marker_plus_duration,head,mlx5_3,45,429439114384.0,429439126660.0,71.45312428445231,71.45312632702023,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.829253632,bench_main_marker_plus_duration,worker,mlx5_0,47,430547106324.0,430552482856.0,70.53920742984172,70.54008830048383,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.829253632,bench_main_marker_plus_duration,worker,mlx5_3,47,430553276516.0,430553408016.0,70.54021833073266,70.54023987519466,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,167.623034624,bench_main_marker_plus_duration,head,mlx5_0,154,1738843459984.0,1738965693932.0,82.98828207636016,82.99411583057059,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,167.623034624,bench_main_marker_plus_duration,head,mlx5_3,154,1739263067700.0,1739364620460.0,83.00830833191347,83.01315505290157,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,166.729046016,bench_main_marker_plus_duration,worker,mlx5_0,158,1739033672624.0,1739033174332.0,83.44238579555551,83.4423618864881,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,166.729046016,bench_main_marker_plus_duration,worker,mlx5_3,158,1739243327096.0,1739243022428.0,83.45244544512514,83.45243082653253,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,142.715803392,bench_main_marker_plus_duration,head,mlx5_0,131,678101307872.0,678090133532.0,38.011280699416155,38.010654316647916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,142.715803392,bench_main_marker_plus_duration,head,mlx5_3,131,677949641072.0,677941962640.0,38.00277894718437,38.00234852914697,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.266830336,bench_main_marker_plus_duration,worker,mlx5_0,137,680623741960.0,680629335892.0,37.74249370419051,37.742803903394964,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.266830336,bench_main_marker_plus_duration,worker,mlx5_3,137,680750304048.0,680766879432.0,37.74951192661656,37.75043107810614,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,188.527344896,bench_main_marker_plus_duration,head,mlx5_0,173,1102872843476.0,1102880032984.0,46.79948552118605,46.79979060193723,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,188.527344896,bench_main_marker_plus_duration,head,mlx5_3,173,1103015616220.0,1103029812428.0,46.80554396301384,46.806146367211824,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,189.088602368,bench_main_marker_plus_duration,worker,mlx5_0,179,1102994734488.0,1102998788292.0,46.66573111969494,46.665902628879486,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,189.088602368,bench_main_marker_plus_duration,worker,mlx5_3,179,1103130615912.0,1103144127560.0,46.67148001930277,46.67205167292254,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.866369024,bench_main_marker_plus_duration,head,mlx5_0,42,415875487184.0,415864521032.0,74.15362486080193,74.15166951612152,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.866369024,bench_main_marker_plus_duration,head,mlx5_3,42,416276974024.0,416281015316.0,74.22521288519235,74.22593347695637,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.584492544,bench_main_marker_plus_duration,worker,mlx5_0,43,415451144696.0,415441323744.0,74.5463045090838,74.54454229061909,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.584492544,bench_main_marker_plus_duration,worker,mlx5_3,43,415600251668.0,415656009484.0,74.57305945700259,74.58306433768075,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
|
@ -1,9 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,1785487808318962176,1785487881306871040,72.987908864,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/meta.json
|
||||
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,1785488175173981952,1785488256333826048,81.159844096,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/meta.json
|
||||
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,1785487926490086144,1785488132860833024,206.37074688,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/meta.json
|
||||
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,1785487502091363840,1785487551713931008,49.622567168,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/meta.json
|
||||
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,1785487595012883968,1785487763299520000,168.286636032,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/meta.json
|
||||
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,1785488301482129152,1785488445978530048,144.496400896,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/meta.json
|
||||
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,1785488479288048896,1785488669308148992,190.020100096,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/meta.json
|
||||
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,1785488519428990976,1785488565490817024,46.061826048,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/meta.json
|
||||
|
@ -1,37 +0,0 @@
|
||||
wall_time_ns,node,collector,status,pid
|
||||
1785487450093977528,head,gpu_samples.csv,STARTED,1795123
|
||||
1785487450108679448,head,rdma.csv,STARTED,1795134
|
||||
1785487450121779856,head,dcgm_dmon.log,STARTED,1795147
|
||||
1785487450134856077,head,mpstat.log,STARTED,1795163
|
||||
1785487450147239051,head,pidstat.log,STARTED,1795181
|
||||
1785487450160535144,head,sar_net.log,STARTED,1795207
|
||||
1785487450175160784,head,perf_stat.log,STARTED,1795238
|
||||
1785487450193437502,head,docker_top.log,STARTED,1795276
|
||||
1785487450214351210,head,numa_samples.csv,STARTED,1795348
|
||||
1785487450236070096,worker,gpu_samples.csv,STARTED,1795394
|
||||
1785487450254195207,worker,rdma.csv,STARTED,1795451
|
||||
1785487450270545415,worker,dcgm_dmon.log,STARTED,1795504
|
||||
1785487450285967202,worker,mpstat.log,STARTED,1795552
|
||||
1785487450301984460,worker,pidstat.log,STARTED,1795567
|
||||
1785487450317255696,worker,sar_net.log,STARTED,1795577
|
||||
1785487450331874012,worker,perf_stat.log,STARTED,1795584
|
||||
1785487450348717530,worker,docker_top.log,STARTED,1795593
|
||||
1785487450364742164,worker,numa_samples.csv,STARTED,1795604
|
||||
1785488689789962654,head,gpu_samples.csv,STOPPED,1795123
|
||||
1785488689794788834,head,rdma.csv,STOPPED,1795134
|
||||
1785488689801075236,head,dcgm_dmon.log,STOPPED,1795147
|
||||
1785488689805422095,head,mpstat.log,STOPPED,1795163
|
||||
1785488689810734986,head,pidstat.log,STOPPED,1795181
|
||||
1785488689814680689,head,sar_net.log,STOPPED,1795207
|
||||
1785488689951029262,head,perf_stat.log,STOPPED,1795238
|
||||
1785488689955936272,head,docker_top.log,STOPPED,1795276
|
||||
1785488689960978121,head,numa_samples.csv,STOPPED,1795348
|
||||
1785488689965181539,worker,gpu_samples.csv,STOPPED,1795394
|
||||
1785488689969473724,worker,rdma.csv,STOPPED,1795451
|
||||
1785488689973568170,worker,dcgm_dmon.log,STOPPED,1795504
|
||||
1785488689977772276,worker,mpstat.log,STOPPED,1795552
|
||||
1785488689982041873,worker,pidstat.log,STOPPED,1795567
|
||||
1785488689986056700,worker,sar_net.log,STOPPED,1795577
|
||||
1785488689990039780,worker,perf_stat.log,STOPPED,1795584
|
||||
1785488689994084643,worker,docker_top.log,STOPPED,1795593
|
||||
1785488689998332256,worker,numa_samples.csv,STOPPED,1795604
|
||||
|
@ -1,20 +0,0 @@
|
||||
test,scope,nccl_cross_nic,size_bytes,repetitions,mean_ms,algbw_GBps_mean,busbw_GBps_mean,busbw_GBps_min,wrong_values,path_class,samples,bandwidth_GBps_mean,bandwidth_GBps_p05,bandwidth_GBps_min
|
||||
all_reduce,head_8gpu,2,1048576,3,1.1792767981688181,0.9351587648306614,1.6365278384536577,1.1619881842280209,0,,,,,
|
||||
all_reduce,head_8gpu,2,1073741824,3,47.2596331278483,22.72016498475108,39.76028872331439,39.67252737812032,0,,,,,
|
||||
all_reduce,head_8gpu,2,67108864,3,3.107903989156087,21.596044891582753,37.79307856026982,37.253524087802575,0,,,,,
|
||||
all_reduce,two_node_16gpu,0,1048576,3,1.174121594429016,0.9225129006791467,1.7297116887734,1.3203079074091486,0,,,,,
|
||||
all_reduce,two_node_16gpu,0,1073741824,3,51.1693105061849,20.984136487918786,39.34525591484772,39.296690886871644,0,,,,,
|
||||
all_reduce,two_node_16gpu,0,67108864,3,3.4025983730951945,19.72330661126499,36.98119989612186,36.724996463125755,0,,,,,
|
||||
all_reduce,two_node_16gpu,1,1048576,3,1.2971573332945505,0.877370746063573,1.6450701488691994,1.0568849765163506,0,,,,,
|
||||
all_reduce,two_node_16gpu,1,1073741824,3,50.73157666524251,21.165255754284047,39.68485453928258,39.59303458131569,0,,,,,
|
||||
all_reduce,two_node_16gpu,1,67108864,3,3.3778613169988,19.86842574374279,37.25329826951773,36.858805194578224,0,,,,,
|
||||
all_reduce,two_node_16gpu,2,1048576,3,1.1436874707539877,0.936877857865813,1.7566459834983992,1.413443823860872,0,,,,,
|
||||
all_reduce,two_node_16gpu,2,1073741824,3,50.931247075398765,21.082601839476187,39.52987844901785,39.33647232214203,0,,,,,
|
||||
all_reduce,two_node_16gpu,2,67108864,3,3.4341418504714967,19.544645327030477,36.64620998818214,36.14018010573151,0,,,,,
|
||||
all_reduce,worker_8gpu,2,1048576,3,0.9693984071413676,1.0921968526479142,1.9113444921338498,1.664030821168365,0,,,,,
|
||||
all_reduce,worker_8gpu,2,1073741824,3,47.27112070719401,22.714924401498006,39.75111770262151,39.55926639799348,0,,,,,
|
||||
all_reduce,worker_8gpu,2,67108864,3,3.0737973292668657,21.837161295289466,38.215032266756566,37.43531449904809,0,,,,,
|
||||
p2p_copy,head,,268435456,,,,,,,cross_numa_sys,32,52.397332123279426,52.19168649638452,52.18129785806688
|
||||
p2p_copy,head,,268435456,,,,,,,same_pcie_switch,24,53.6126013471359,53.36177601211484,53.35073474580275
|
||||
p2p_copy,worker,,268435456,,,,,,,cross_numa_sys,32,52.31572860916867,52.14858683841258,52.07724663835188
|
||||
p2p_copy,worker,,268435456,,,,,,,same_pcie_switch,24,53.50030938041238,53.28731762907832,53.221028060678236
|
||||
|
@ -1,17 +0,0 @@
|
||||
node,gpu,samples,gpu_util_pct_mean,gpu_util_pct_p95,gpu_util_pct_max,memory_util_pct_mean,memory_util_pct_p95,memory_util_pct_max,memory_used_mib_mean,memory_used_mib_p95,memory_used_mib_max,power_w_mean,power_w_p95,power_w_max,temperature_c_mean,temperature_c_p95,temperature_c_max,sm_clock_mhz_mean,sm_clock_mhz_p95,sm_clock_mhz_max,memory_clock_mhz_mean,memory_clock_mhz_p95,memory_clock_mhz_max
|
||||
head,0,885,73.19774011299435,100.0,100.0,38.57514124293785,58.0,64.0,83114.45536723164,83361.0,83361.0,205.47479096045197,307.69199999999995,333.9,43.47570621468927,52.0,55.0,2393.25197740113,2422.0,2430.0,12481.0,12481.0,12481.0
|
||||
head,1,885,73.50056497175142,100.0,100.0,37.15819209039548,56.0,62.0,83080.49830508475,83327.0,83327.0,206.62670056497177,313.188,350.0,43.737853107344634,53.0,56.0,2378.2225988700566,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
head,2,885,73.52542372881356,100.0,100.0,38.364971751412426,57.0,63.0,83048.5209039548,83295.0,83295.0,210.91588700564972,317.204,371.02,42.911864406779664,52.0,55.0,2391.7344632768363,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
head,3,885,73.10960451977401,100.0,100.0,38.39322033898305,58.0,65.0,83146.58870056497,83393.0,83393.0,204.69710734463277,310.6,340.24,43.44858757062147,52.0,55.0,2388.697175141243,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
head,4,885,73.17062146892656,100.0,100.0,39.108474576271185,58.0,65.0,83194.63389830508,83441.0,83441.0,207.94813559322034,316.352,339.4,44.054237288135596,53.0,56.0,2402.890395480226,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
head,5,885,73.44519774011299,100.0,100.0,38.371751412429376,57.0,63.0,83080.63389830508,83327.0,83327.0,209.06979661016948,317.16999999999996,376.49,43.85084745762712,53.0,55.0,2399.3717514124296,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
head,6,885,73.67570621468927,100.0,100.0,39.31412429378531,58.0,65.0,83048.65423728814,83295.0,83295.0,205.60972881355931,310.78,342.99,43.70395480225989,52.0,55.0,2405.515254237288,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
head,7,885,73.06553672316385,100.0,100.0,35.74124293785311,54.0,60.0,83227.83163841808,83473.0,83473.0,206.52320903954802,314.84799999999996,336.9,43.929943502824855,53.0,56.0,2388.154802259887,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
worker,0,913,73.23986856516977,100.0,100.0,37.39539978094195,56.0,62.0,83195.78422782038,83441.0,83441.0,210.65017524644028,326.81399999999996,347.67,46.10624315443592,56.0,60.0,2398.8400876232204,2422.0,2430.0,12481.0,12481.0,12481.0
|
||||
worker,1,913,73.35158817086528,100.0,100.0,38.92880613362541,58.0,64.0,83081.95947426069,83327.0,83327.0,202.77134720700985,310.3,335.13,45.61226725082147,55.0,58.0,2398.3099671412924,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
worker,2,913,73.34720700985761,100.0,100.0,38.708652792990144,58.0,64.0,83050.44140197152,83295.0,83295.0,203.5297152245345,312.52,373.73,46.55859802847755,57.0,60.0,2400.135815991238,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
worker,3,913,72.96276013143483,100.0,100.0,37.24096385542169,57.0,63.0,83229.27382256297,83473.0,83473.0,207.60696604600219,318.69399999999996,359.03,45.467688937568454,55.0,59.0,2409.73713033954,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
worker,4,913,72.84008762322016,100.0,100.0,36.52464403066813,55.0,63.0,83117.25410733845,83361.0,83361.0,204.54877327491786,314.376,347.28,45.67798466593647,56.0,59.0,2392.5739320920043,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
worker,5,913,73.26944140197152,100.0,100.0,38.52245345016429,58.0,64.0,83083.19058050384,83327.0,83327.0,204.3163417305586,315.148,356.12,46.33077765607886,56.0,60.0,2385.7513691128147,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
worker,6,913,73.3274917853231,100.0,100.0,38.77984665936473,58.0,64.0,83052.13253012048,83295.0,83295.0,206.93734939759034,319.012,355.56,46.1829134720701,56.0,59.0,2401.7447973713033,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
worker,7,913,73.27929901423877,100.0,100.0,37.28806133625411,57.0,64.0,83150.0230010953,83393.0,83393.0,206.63869660460023,320.18,353.19,47.00766703176342,57.0,61.0,2406.594742606791,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||
|
@ -1,26 +0,0 @@
|
||||
{
|
||||
"run_id": "dsv4pro-phase2-20260731-163620",
|
||||
"status": "COMPLETED",
|
||||
"created_at": "2026-07-31T16:36:20+08:00",
|
||||
"completed_at": "2026-07-31T17:05:04+08:00",
|
||||
"experiment": "dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution",
|
||||
"git_commit": "5f24b7d22f98108f6cc234edba6768d55ea0a962",
|
||||
"git_dirty": false,
|
||||
"phase1_entry": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh",
|
||||
"head_node": "10.101.0.11",
|
||||
"worker_node": "10.101.0.13",
|
||||
"fixed_case_ids": "long_prefill_latency_128k_c1,mid_prefill_throughput_32k_c16,decode_throughput_1k_to_1k_c32,long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1",
|
||||
"run_mixed_case": true,
|
||||
"sample_interval_s": 1,
|
||||
"cpu_sample_interval_s": 5,
|
||||
"process_sample_interval_s": 5,
|
||||
"net_sample_interval_s": 5,
|
||||
"perf_interval_ms": 5000,
|
||||
"numastat_interval_s": 5,
|
||||
"clock_skew_tolerance_s": 2,
|
||||
"idle_baseline_s": 15,
|
||||
"post_run_cooldown_s": 15,
|
||||
"require_precise_windows": true,
|
||||
"run_communication_baseline": true,
|
||||
"dry_run": false
|
||||
}
|
||||
@ -1,5 +0,0 @@
|
||||
node,hca,samples,duration_s,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||
head,mlx5_0,1131,1239.614615552,5264991004888.0,5264992389656.0,33.97824413384156,33.978253070606144,0.0,0.0,0.0,0.0,,,,,,
|
||||
head,mlx5_3,1130,1238.507734784,5266283609308.0,5266274185868.0,34.01696064644414,34.0168997768041,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
worker,mlx5_0,1168,1238.553399296,5272091713408.0,5272078266720.0,34.05322187257931,34.05313501842828,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
worker,mlx5_3,1168,1238.553399296,5264840174196.0,5264861659552.0,34.00638310589474,34.00652188299398,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
|
@ -1,253 +0,0 @@
|
||||
# Phase 2 Hardware Attribution
|
||||
|
||||
- Generated: `2026-07-31T17:05:04+08:00`
|
||||
- Bench rows: `8`
|
||||
- Failed bench rows: `0`
|
||||
- GPU summary rows: `16`
|
||||
- RDMA summary rows: `4`
|
||||
- Case windows: `8`
|
||||
- Precise main-run windows: `8/8`
|
||||
- Collector status counts: `{"STARTED": 18, "STOPPED": 18}`
|
||||
|
||||
## 1. Benchmark results
|
||||
|
||||
| Run | Case | Status | Input TPS | Output TPS | TTFT P95 (ms) | TPOT P95 (ms) |
|
||||
|---|---|---|---:|---:|---:|---:|
|
||||
| fixed_decode_throughput_1k_to_1k_c32 | decode_throughput_1k_to_1k_c32 | COMPLETED | 448.95107215015315 | 448.95107215015315 | 10166.788510262268 | 65.35758210354297 |
|
||||
| fixed_long_context_decode_128k_to_1k_c1 | long_context_decode_128k_to_1k_c1 | COMPLETED | 1614.985848663526 | 12.617076942683797 | 48279.420554987155 | 32.106360114411245 |
|
||||
| fixed_long_output_decode_1k_to_4k_c16 | long_output_decode_1k_to_4k_c16 | COMPLETED | 79.3910971002871 | 317.5643884011484 | 1723.5681610036409 | 49.98305734157135 |
|
||||
| fixed_long_prefill_latency_128k_c1 | long_prefill_latency_128k_c1 | COMPLETED | 2641.3788467537956 | 0.02015212132838284 | 49610.14223104576 | 0.0 |
|
||||
| fixed_mid_prefill_throughput_32k_c16 | mid_prefill_throughput_32k_c16 | COMPLETED | 3115.446435236942 | 0.09507587998159613 | 161938.13879448862 | 0.0 |
|
||||
| mixed_prefill_decode_interference | decode_control_1k_to_1k_c32 | COMPLETED | 453.54762732999035 | 453.54762732999035 | 9442.68154159945 | 66.23843437823616 |
|
||||
| mixed_prefill_decode_interference | decode_with_128k_prefill_1k_to_1k_c32 | COMPLETED | 344.8898309034744 | 344.8898309034744 | 9891.837346865213 | 110.451060616212 |
|
||||
| mixed_prefill_decode_interference | long_prefill_injection_128k_to_1_c1 | COMPLETED | 2845.5667477625557 | 0.02170995138368649 | 45980.19455798203 | 0.0 |
|
||||
|
||||
## 2. Measurement-window validity
|
||||
|
||||
| Case | Role | Duration (s) | Window source |
|
||||
|---|---|---:|---|
|
||||
| decode_throughput_1k_to_1k_c32 | - | 72.99 | bench_main_marker_plus_duration |
|
||||
| long_context_decode_128k_to_1k_c1 | - | 81.16 | bench_main_marker_plus_duration |
|
||||
| long_output_decode_1k_to_4k_c16 | - | 206.37 | bench_main_marker_plus_duration |
|
||||
| long_prefill_latency_128k_c1 | - | 49.62 | bench_main_marker_plus_duration |
|
||||
| mid_prefill_throughput_32k_c16 | - | 168.29 | bench_main_marker_plus_duration |
|
||||
| decode_control_1k_to_1k_c32 | control | 144.50 | bench_main_marker_plus_duration |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | decode_background | 190.02 | bench_main_marker_plus_duration |
|
||||
| long_prefill_injection_128k_to_1_c1 | prefill_injection | 46.06 | bench_main_marker_plus_duration |
|
||||
|
||||
## 3. GPU basic state (`nvidia-smi`)
|
||||
|
||||
Data: `case_gpu_node_summary.csv`; raw: `head|worker/gpu_samples.csv`.
|
||||
|
||||
| Case | Node | Samples | GPU util mean/p95 (%) | Memory used mean (MiB) | Power mean (W) | SM clock mean (MHz) |
|
||||
|---|---|---:|---:|---:|---:|---:|
|
||||
| decode_throughput_1k_to_1k_c32 | head | 408 | 97.09/100.00 | 83300.00 | 230.22 | 2402.95 |
|
||||
| decode_throughput_1k_to_1k_c32 | worker | 432 | 97.72/100.00 | 83300.00 | 221.83 | 2418.55 |
|
||||
| long_context_decode_128k_to_1k_c1 | head | 456 | 97.70/100.00 | 83219.06 | 257.12 | 2412.63 |
|
||||
| long_context_decode_128k_to_1k_c1 | worker | 472 | 97.43/100.00 | 83221.32 | 257.90 | 2417.53 |
|
||||
| long_output_decode_1k_to_4k_c16 | head | 1184 | 98.83/100.00 | 83300.00 | 220.83 | 2402.76 |
|
||||
| long_output_decode_1k_to_4k_c16 | worker | 1216 | 99.25/100.00 | 83300.00 | 215.74 | 2412.57 |
|
||||
| long_prefill_latency_128k_c1 | head | 288 | 94.44/100.00 | 83028.22 | 293.40 | 2392.22 |
|
||||
| long_prefill_latency_128k_c1 | worker | 288 | 94.85/100.00 | 83057.76 | 297.91 | 2398.02 |
|
||||
| mid_prefill_throughput_32k_c16 | head | 952 | 98.21/100.00 | 83231.07 | 299.46 | 2417.69 |
|
||||
| mid_prefill_throughput_32k_c16 | worker | 968 | 99.07/100.00 | 83231.94 | 302.56 | 2419.33 |
|
||||
| decode_control_1k_to_1k_c32 | head | 824 | 98.52/100.00 | 83364.00 | 231.92 | 2405.39 |
|
||||
| decode_control_1k_to_1k_c32 | worker | 856 | 98.60/100.00 | 83364.00 | 227.79 | 2418.59 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | head | 1072 | 99.61/100.00 | 83259.71 | 248.31 | 2411.49 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | worker | 1112 | 98.92/100.00 | 83256.91 | 247.75 | 2420.77 |
|
||||
| long_prefill_injection_128k_to_1_c1 | head | 264 | 99.95/100.00 | 83281.49 | 301.20 | 2421.11 |
|
||||
| long_prefill_injection_128k_to_1_c1 | worker | 272 | 99.89/100.00 | 83282.94 | 307.16 | 2421.46 |
|
||||
|
||||
## 4. GPU profiling counters (DCGM)
|
||||
|
||||
Data: `case_dcgm_summary.csv`; raw: `head|worker/dcgm_dmon.log`.
|
||||
|
||||
| Case | Node | Samples | GR active | SM active | SM occupancy | Tensor active | DRAM active | PCIe TX/RX mean (GB/s) |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| decode_throughput_1k_to_1k_c32 | head | 584 | 0.971 | 0.525 | 0.141 | 0.059 | 0.417 | 5.205/5.201 |
|
||||
| decode_throughput_1k_to_1k_c32 | worker | 576 | 0.979 | 0.523 | 0.141 | 0.058 | 0.415 | 5.236/5.232 |
|
||||
| long_context_decode_128k_to_1k_c1 | head | 648 | 0.961 | 0.578 | 0.211 | 0.095 | 0.399 | 6.764/6.821 |
|
||||
| long_context_decode_128k_to_1k_c1 | worker | 648 | 0.965 | 0.579 | 0.211 | 0.095 | 0.401 | 6.780/6.838 |
|
||||
| long_output_decode_1k_to_4k_c16 | head | 1648 | 0.989 | 0.490 | 0.133 | 0.045 | 0.414 | 2.621/2.634 |
|
||||
| long_output_decode_1k_to_4k_c16 | worker | 1640 | 0.990 | 0.489 | 0.133 | 0.045 | 0.414 | 2.619/2.633 |
|
||||
| long_prefill_latency_128k_c1 | head | 400 | 0.930 | 0.683 | 0.251 | 0.136 | 0.374 | 10.537/10.582 |
|
||||
| long_prefill_latency_128k_c1 | worker | 392 | 0.936 | 0.686 | 0.251 | 0.136 | 0.376 | 10.618/10.632 |
|
||||
| mid_prefill_throughput_32k_c16 | head | 1344 | 0.980 | 0.713 | 0.278 | 0.127 | 0.440 | 12.664/12.720 |
|
||||
| mid_prefill_throughput_32k_c16 | worker | 1344 | 0.984 | 0.715 | 0.278 | 0.127 | 0.441 | 12.699/12.755 |
|
||||
| decode_control_1k_to_1k_c32 | head | 1152 | 0.989 | 0.539 | 0.146 | 0.061 | 0.427 | 5.451/5.456 |
|
||||
| decode_control_1k_to_1k_c32 | worker | 1152 | 0.990 | 0.536 | 0.145 | 0.060 | 0.425 | 5.465/5.461 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | head | 1520 | 0.977 | 0.581 | 0.174 | 0.080 | 0.416 | 6.865/6.868 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | worker | 1520 | 0.978 | 0.578 | 0.174 | 0.080 | 0.416 | 6.865/6.872 |
|
||||
| long_prefill_injection_128k_to_1_c1 | head | 368 | 0.997 | 0.723 | 0.264 | 0.138 | 0.407 | 11.151/11.198 |
|
||||
| long_prefill_injection_128k_to_1_c1 | worker | 368 | 0.997 | 0.720 | 0.262 | 0.137 | 0.407 | 11.069/11.110 |
|
||||
|
||||
## 5. CPU, process, and `perf`
|
||||
|
||||
Data: `case_cpu_summary.csv`, `case_process_summary.csv`, `case_perf_summary.csv`; raw: `mpstat.log`, `pidstat.log`, `perf_stat.log`.
|
||||
|
||||
| Case | Node | Samples CPU/process/perf | CPU active mean/p95 (%) | Hot cores max | Process CPU max (%) | Process wait max (%) | IPC | Context switches mean/interval |
|
||||
|---|---|---|---:|---:|---:|---:|---:|---:|
|
||||
| decode_throughput_1k_to_1k_c32 | head | 15/1140/84 | 10.13/10.42 | 12 | 1225.00 | 0.20 | 3.049 | 41240.93 |
|
||||
| decode_throughput_1k_to_1k_c32 | worker | 15/1080/84 | 9.84/10.09 | 12 | 1210.20 | 0.20 | 3.068 | 39682.00 |
|
||||
| long_context_decode_128k_to_1k_c1 | head | 17/1292/96 | 9.90/10.29 | 12 | 1222.40 | 0.20 | 2.887 | 44877.38 |
|
||||
| long_context_decode_128k_to_1k_c1 | worker | 17/1224/96 | 9.65/10.09 | 12 | 1215.60 | 0.00 | 2.902 | 43686.69 |
|
||||
| long_output_decode_1k_to_4k_c16 | head | 41/3116/246 | 10.36/10.52 | 12 | 1225.80 | 0.20 | 3.098 | 43749.98 |
|
||||
| long_output_decode_1k_to_4k_c16 | worker | 41/2952/246 | 9.99/10.09 | 12 | 1211.40 | 0.20 | 3.117 | 40973.46 |
|
||||
| long_prefill_latency_128k_c1 | head | 10/1129/60 | 10.00/10.27 | 12 | 1211.20 | 0.20 | 2.727 | 50995.80 |
|
||||
| long_prefill_latency_128k_c1 | worker | 10/720/60 | 9.69/10.18 | 12 | 1210.80 | 0.20 | 2.749 | 48210.50 |
|
||||
| mid_prefill_throughput_32k_c16 | head | 34/2584/204 | 9.99/10.27 | 12 | 1211.00 | 0.20 | 2.802 | 39421.29 |
|
||||
| mid_prefill_throughput_32k_c16 | worker | 34/2448/204 | 9.84/10.11 | 12 | 1210.20 | 0.20 | 2.810 | 39129.38 |
|
||||
| decode_control_1k_to_1k_c32 | head | 29/2204/174 | 10.26/10.44 | 12 | 1220.40 | 0.20 | 3.042 | 42398.83 |
|
||||
| decode_control_1k_to_1k_c32 | worker | 29/2088/174 | 9.98/10.11 | 12 | 1210.40 | 0.20 | 3.072 | 40484.59 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | head | 38/2888/228 | 10.34/11.37 | 13 | 1223.60 | 0.20 | 2.980 | 41223.05 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | worker | 38/2736/228 | 9.87/10.09 | 12 | 1211.40 | 0.20 | 3.005 | 39834.92 |
|
||||
| long_prefill_injection_128k_to_1_c1 | head | 10/749/54 | 10.21/10.63 | 13 | 1223.60 | 0.00 | 2.808 | 33399.67 |
|
||||
| long_prefill_injection_128k_to_1_c1 | worker | 9/648/60 | 10.04/10.12 | 12 | 1211.40 | 0.20 | 2.830 | 33911.70 |
|
||||
|
||||
## 6. NUMA memory placement
|
||||
|
||||
Data/raw: `case_numa_summary.csv`, `head|worker/numa_samples.csv`.
|
||||
|
||||
| Case | Node | Samples | Node 0 mean (MiB) | Node 1 mean (MiB) | Total mean (MiB) | Imbalance mean/max (%) |
|
||||
|---|---|---:|---:|---:|---:|---:|
|
||||
| decode_throughput_1k_to_1k_c32 | head | 13 | 17941.22 | 27565.28 | 45506.51 | 21.15/21.15 |
|
||||
| decode_throughput_1k_to_1k_c32 | worker | 13 | 19029.49 | 24814.32 | 43843.80 | 13.19/13.19 |
|
||||
| long_context_decode_128k_to_1k_c1 | head | 13 | 17946.50 | 27569.43 | 45515.94 | 21.14/21.15 |
|
||||
| long_context_decode_128k_to_1k_c1 | worker | 14 | 19037.22 | 24816.07 | 43853.29 | 13.18/13.18 |
|
||||
| long_output_decode_1k_to_4k_c16 | head | 35 | 17941.67 | 27565.20 | 45506.88 | 21.15/21.17 |
|
||||
| long_output_decode_1k_to_4k_c16 | worker | 37 | 19029.40 | 24814.58 | 43843.96 | 13.19/13.20 |
|
||||
| long_prefill_latency_128k_c1 | head | 8 | 17869.37 | 27486.32 | 45355.71 | 21.20/21.21 |
|
||||
| long_prefill_latency_128k_c1 | worker | 8 | 18970.34 | 24731.71 | 43702.05 | 13.18/13.21 |
|
||||
| mid_prefill_throughput_32k_c16 | head | 28 | 17940.27 | 27563.88 | 45504.17 | 21.15/21.15 |
|
||||
| mid_prefill_throughput_32k_c16 | worker | 30 | 19032.02 | 24805.59 | 43837.58 | 13.17/13.22 |
|
||||
| decode_control_1k_to_1k_c32 | head | 24 | 17946.96 | 27573.84 | 45520.81 | 21.15/21.21 |
|
||||
| decode_control_1k_to_1k_c32 | worker | 26 | 19039.33 | 24818.28 | 43857.62 | 13.18/13.18 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | head | 32 | 17955.73 | 27573.17 | 45528.91 | 21.12/21.13 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | worker | 34 | 19042.79 | 24823.40 | 43866.18 | 13.18/13.19 |
|
||||
| long_prefill_injection_128k_to_1_c1 | head | 8 | 17951.77 | 27569.78 | 45521.56 | 21.13/21.13 |
|
||||
| long_prefill_injection_128k_to_1_c1 | worker | 8 | 19038.43 | 24819.53 | 43857.94 | 13.18/13.18 |
|
||||
|
||||
## 7. Linux netdev and RDMA data path
|
||||
|
||||
Netdev data: `case_netdev_summary.csv`; RDMA data: `case_rdma_summary.csv`; raw: `sar_net.log`, `rdma.csv`.
|
||||
|
||||
### Linux interfaces
|
||||
|
||||
| Case | Node | Interface | Samples | RX mean/max (Gbit/s) | TX mean/max (Gbit/s) | Util max (%) | RX/TX error max (/s) |
|
||||
|---|---|---|---:|---:|---:|---:|---:|
|
||||
| decode_throughput_1k_to_1k_c32 | head | eth0 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_throughput_1k_to_1k_c32 | head | eth3 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_throughput_1k_to_1k_c32 | worker | eth0 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_throughput_1k_to_1k_c32 | worker | eth3 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_context_decode_128k_to_1k_c1 | head | eth0 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_context_decode_128k_to_1k_c1 | head | eth3 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_context_decode_128k_to_1k_c1 | worker | eth0 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_context_decode_128k_to_1k_c1 | worker | eth3 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_output_decode_1k_to_4k_c16 | head | eth0 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_output_decode_1k_to_4k_c16 | head | eth3 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_output_decode_1k_to_4k_c16 | worker | eth0 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_output_decode_1k_to_4k_c16 | worker | eth3 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_prefill_latency_128k_c1 | head | eth0 | 20 | 0.000/0.001 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_prefill_latency_128k_c1 | head | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_prefill_latency_128k_c1 | worker | eth0 | 20 | 0.000/0.000 | 0.000/0.001 | 0.000 | 0.00/0.00 |
|
||||
| long_prefill_latency_128k_c1 | worker | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| mid_prefill_throughput_32k_c16 | head | eth0 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| mid_prefill_throughput_32k_c16 | head | eth3 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| mid_prefill_throughput_32k_c16 | worker | eth0 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| mid_prefill_throughput_32k_c16 | worker | eth3 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_control_1k_to_1k_c32 | head | eth0 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_control_1k_to_1k_c32 | head | eth3 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_control_1k_to_1k_c32 | worker | eth0 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_control_1k_to_1k_c32 | worker | eth3 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | head | eth0 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | head | eth3 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | worker | eth0 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | worker | eth3 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_prefill_injection_128k_to_1_c1 | head | eth0 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_prefill_injection_128k_to_1_c1 | head | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_prefill_injection_128k_to_1_c1 | worker | eth0 | 18 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
| long_prefill_injection_128k_to_1_c1 | worker | eth3 | 18 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||
|
||||
### RDMA HCAs
|
||||
|
||||
| Case | Node | HCA | Samples | TX/RX (Gbit/s) | Wait delta | Discard/error delta | Retry exceeded delta |
|
||||
|---|---|---|---:|---:|---:|---:|---:|
|
||||
| decode_throughput_1k_to_1k_c32 | head | mlx5_0 | 66 | 36.69/36.69 | 0 | 0/0 | 0 |
|
||||
| decode_throughput_1k_to_1k_c32 | head | mlx5_3 | 66 | 36.71/36.71 | 0 | 0/0 | 0 |
|
||||
| decode_throughput_1k_to_1k_c32 | worker | mlx5_0 | 68 | 36.80/36.80 | 0 | 0/0 | 0 |
|
||||
| decode_throughput_1k_to_1k_c32 | worker | mlx5_3 | 68 | 36.79/36.79 | 0 | 0/0 | 0 |
|
||||
| long_context_decode_128k_to_1k_c1 | head | mlx5_0 | 75 | 43.83/43.83 | 0 | 0/0 | 0 |
|
||||
| long_context_decode_128k_to_1k_c1 | head | mlx5_3 | 75 | 43.95/43.95 | 0 | 0/0 | 0 |
|
||||
| long_context_decode_128k_to_1k_c1 | worker | mlx5_0 | 77 | 44.73/44.73 | 0 | 0/0 | 0 |
|
||||
| long_context_decode_128k_to_1k_c1 | worker | mlx5_3 | 77 | 44.11/44.11 | 0 | 0/0 | 0 |
|
||||
| long_output_decode_1k_to_4k_c16 | head | mlx5_0 | 188 | 18.91/18.91 | 0 | 0/0 | 0 |
|
||||
| long_output_decode_1k_to_4k_c16 | head | mlx5_3 | 188 | 18.90/18.90 | 0 | 0/0 | 0 |
|
||||
| long_output_decode_1k_to_4k_c16 | worker | mlx5_0 | 194 | 18.90/18.90 | 0 | 0/0 | 0 |
|
||||
| long_output_decode_1k_to_4k_c16 | worker | mlx5_3 | 194 | 18.90/18.90 | 0 | 0/0 | 0 |
|
||||
| long_prefill_latency_128k_c1 | head | mlx5_0 | 45 | 71.38/71.38 | 0 | 0/0 | 0 |
|
||||
| long_prefill_latency_128k_c1 | head | mlx5_3 | 45 | 71.45/71.45 | 0 | 0/0 | 0 |
|
||||
| long_prefill_latency_128k_c1 | worker | mlx5_0 | 47 | 70.54/70.54 | 0 | 0/0 | 0 |
|
||||
| long_prefill_latency_128k_c1 | worker | mlx5_3 | 47 | 70.54/70.54 | 0 | 0/0 | 0 |
|
||||
| mid_prefill_throughput_32k_c16 | head | mlx5_0 | 154 | 82.99/82.99 | 0 | 0/0 | 0 |
|
||||
| mid_prefill_throughput_32k_c16 | head | mlx5_3 | 154 | 83.01/83.01 | 0 | 0/0 | 0 |
|
||||
| mid_prefill_throughput_32k_c16 | worker | mlx5_0 | 158 | 83.44/83.44 | 0 | 0/0 | 0 |
|
||||
| mid_prefill_throughput_32k_c16 | worker | mlx5_3 | 158 | 83.45/83.45 | 0 | 0/0 | 0 |
|
||||
| decode_control_1k_to_1k_c32 | head | mlx5_0 | 131 | 38.01/38.01 | 0 | 0/0 | 0 |
|
||||
| decode_control_1k_to_1k_c32 | head | mlx5_3 | 131 | 38.00/38.00 | 0 | 0/0 | 0 |
|
||||
| decode_control_1k_to_1k_c32 | worker | mlx5_0 | 137 | 37.74/37.74 | 0 | 0/0 | 0 |
|
||||
| decode_control_1k_to_1k_c32 | worker | mlx5_3 | 137 | 37.75/37.75 | 0 | 0/0 | 0 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | head | mlx5_0 | 173 | 46.80/46.80 | 0 | 0/0 | 0 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | head | mlx5_3 | 173 | 46.81/46.81 | 0 | 0/0 | 0 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | worker | mlx5_0 | 179 | 46.67/46.67 | 0 | 0/0 | 0 |
|
||||
| decode_with_128k_prefill_1k_to_1k_c32 | worker | mlx5_3 | 179 | 46.67/46.67 | 0 | 0/0 | 0 |
|
||||
| long_prefill_injection_128k_to_1_c1 | head | mlx5_0 | 42 | 74.15/74.15 | 0 | 0/0 | 0 |
|
||||
| long_prefill_injection_128k_to_1_c1 | head | mlx5_3 | 42 | 74.23/74.23 | 0 | 0/0 | 0 |
|
||||
| long_prefill_injection_128k_to_1_c1 | worker | mlx5_0 | 43 | 74.55/74.54 | 0 | 0/0 | 0 |
|
||||
| long_prefill_injection_128k_to_1_c1 | worker | mlx5_3 | 43 | 74.57/74.58 | 0 | 0/0 | 0 |
|
||||
|
||||
## 8. PCIe P2P and NCCL communication baseline
|
||||
|
||||
Data: `communication_aggregate.csv`; raw: `communication/*.log`.
|
||||
|
||||
| Test | Scope | Path/CROSS_NIC | Samples/repetitions | Size (MiB) | Mean latency (ms) | Bandwidth / busbw (GB/s) | Minimum | Wrong values |
|
||||
|---|---|---|---:|---:|---:|---:|---:|---:|
|
||||
| all_reduce | head_8gpu | 2 | 3 | 1 | 1.18 | 1.64 | 1.16 | 0 |
|
||||
| all_reduce | head_8gpu | 2 | 3 | 1024 | 47.26 | 39.76 | 39.67 | 0 |
|
||||
| all_reduce | head_8gpu | 2 | 3 | 64 | 3.11 | 37.79 | 37.25 | 0 |
|
||||
| all_reduce | two_node_16gpu | 0 | 3 | 1 | 1.17 | 1.73 | 1.32 | 0 |
|
||||
| all_reduce | two_node_16gpu | 0 | 3 | 1024 | 51.17 | 39.35 | 39.30 | 0 |
|
||||
| all_reduce | two_node_16gpu | 0 | 3 | 64 | 3.40 | 36.98 | 36.72 | 0 |
|
||||
| all_reduce | two_node_16gpu | 1 | 3 | 1 | 1.30 | 1.65 | 1.06 | 0 |
|
||||
| all_reduce | two_node_16gpu | 1 | 3 | 1024 | 50.73 | 39.68 | 39.59 | 0 |
|
||||
| all_reduce | two_node_16gpu | 1 | 3 | 64 | 3.38 | 37.25 | 36.86 | 0 |
|
||||
| all_reduce | two_node_16gpu | 2 | 3 | 1 | 1.14 | 1.76 | 1.41 | 0 |
|
||||
| all_reduce | two_node_16gpu | 2 | 3 | 1024 | 50.93 | 39.53 | 39.34 | 0 |
|
||||
| all_reduce | two_node_16gpu | 2 | 3 | 64 | 3.43 | 36.65 | 36.14 | 0 |
|
||||
| all_reduce | worker_8gpu | 2 | 3 | 1 | 0.97 | 1.91 | 1.66 | 0 |
|
||||
| all_reduce | worker_8gpu | 2 | 3 | 1024 | 47.27 | 39.75 | 39.56 | 0 |
|
||||
| all_reduce | worker_8gpu | 2 | 3 | 64 | 3.07 | 38.22 | 37.44 | 0 |
|
||||
| p2p_copy | head | cross_numa_sys | 32 | 256 | - | 52.40 | 52.18 | - |
|
||||
| p2p_copy | head | same_pcie_switch | 24 | 256 | - | 53.61 | 53.35 | - |
|
||||
| p2p_copy | worker | cross_numa_sys | 32 | 256 | - | 52.32 | 52.08 | - |
|
||||
| p2p_copy | worker | same_pcie_switch | 24 | 256 | - | 53.50 | 53.22 | - |
|
||||
|
||||
## 9. Machine-readable summaries
|
||||
|
||||
- `gpu_summary.csv`
|
||||
- `rdma_summary.csv`
|
||||
- `bench_summary.csv`
|
||||
- `case_windows.csv`
|
||||
- `case_gpu_summary.csv`
|
||||
- `case_gpu_node_summary.csv`
|
||||
- `case_dcgm_summary.csv`
|
||||
- `case_cpu_summary.csv`
|
||||
- `case_process_summary.csv`
|
||||
- `case_perf_summary.csv`
|
||||
- `case_numa_summary.csv`
|
||||
- `case_netdev_summary.csv`
|
||||
- `case_rdma_summary.csv`
|
||||
- `communication_summary.csv`
|
||||
- `communication_aggregate.csv`
|
||||
- `summary.json`
|
||||
|
||||
Each conclusion must cite the corresponding table above and its raw file; missing samples are reported as `-`, never interpreted as zero.
|
||||
File diff suppressed because one or more lines are too long
@ -1 +0,0 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
File diff suppressed because one or more lines are too long
@ -1 +0,0 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -1,55 +0,0 @@
|
||||
{
|
||||
"generated_at": "2026-07-31T17:05:04+08:00",
|
||||
"bench_rows": 8,
|
||||
"failed_bench_rows": 0,
|
||||
"gpu_summary_rows": 16,
|
||||
"rdma_summary_rows": 4,
|
||||
"case_windows": 8,
|
||||
"case_gpu_summary_rows": 128,
|
||||
"case_rdma_summary_rows": 32,
|
||||
"case_metric_rows": {
|
||||
"gpu_node": 16,
|
||||
"dcgm": 16,
|
||||
"cpu": 16,
|
||||
"process": 16,
|
||||
"perf": 16,
|
||||
"numa": 16,
|
||||
"netdev": 32
|
||||
},
|
||||
"communication_aggregate_rows": 19,
|
||||
"precise_windows": 8,
|
||||
"collector_status_counts": {
|
||||
"STARTED": 18,
|
||||
"STOPPED": 18
|
||||
},
|
||||
"collector_files": {
|
||||
"head": {
|
||||
"dcgm_dmon.log": 1642103,
|
||||
"docker_top.log": 2636664,
|
||||
"gpu_samples.csv": 727301,
|
||||
"markers.csv": 1860,
|
||||
"mpstat.log": 3923940,
|
||||
"numa_samples.csv": 11356,
|
||||
"perf_stat.log": 188193,
|
||||
"pidstat.log": 3070380,
|
||||
"rdma.csv": 230893,
|
||||
"sar_net.log": 534606,
|
||||
"static_after.log": 1195708,
|
||||
"static_before.log": 1306541
|
||||
},
|
||||
"worker": {
|
||||
"dcgm_dmon.log": 1613150,
|
||||
"docker_top.log": 2592658,
|
||||
"gpu_samples.csv": 764832,
|
||||
"markers.csv": 1902,
|
||||
"mpstat.log": 3988645,
|
||||
"numa_samples.csv": 12614,
|
||||
"perf_stat.log": 191154,
|
||||
"pidstat.log": 3024655,
|
||||
"rdma.csv": 243227,
|
||||
"sar_net.log": 609685,
|
||||
"static_after.log": 1195276,
|
||||
"static_before.log": 1306123
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1 +0,0 @@
|
||||
env DRY_RUN=0 RUN_ID=confirm RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/confirm.tsv FIXED_CASE_IDS=rdma_confirm_i65536_o1024_c4\,rdma_confirm_i65536_o1024_c16\,rdma_confirm_i65536_o1024_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=2 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
|
||||
@ -1 +0,0 @@
|
||||
env DRY_RUN=0 RUN_ID=scout RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/scout.tsv FIXED_CASE_IDS=rdma_scout_i65536_o1_c1\,rdma_scout_i65536_o1_c4\,rdma_scout_i65536_o1_c16\,rdma_scout_i65536_o1_c32\,rdma_scout_i65536_o1_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=1 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
|
||||
@ -1,7 +0,0 @@
|
||||
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||
fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:03:43+0800,2026-08-01T14:11:03+0800,440.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,406.0844679859583,bench_main_marker_plus_duration,16,0,406.0844679859583,15.997723352218085,,1048576,16384,0.03940066971621592,2582.1622905219265,40.3462857894051,2622.5085763113316,,406026.68602950143,406024.73604350234,406053.3737935184,406053.5528299224,189372.86670175308,189251.9370575028,336778.7128902273,348624.74093402264,211.7828145921294,211.89852223703926,356.41907673483706,369.22852823690624,211.7827865053078,53.14174899831414,53.806967759737745,55.23117201111745,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c16
|
||||
fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:11:03+0800,2026-08-01T14:18:24+0800,441.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,406.09330375201534,bench_main_marker_plus_duration,16,0,406.09330375201534,15.997232835777202,,1048576,16384,0.03939981243761298,2582.1061079114043,40.34540793611569,2622.45151584752,,406023.07082318654,406021.80599197163,406049.04031673505,406056.0000617319,189594.6209833055,189467.45998901315,336924.2522477143,348614.9598911492,211.56251206244485,211.68662201320043,356.06784952074946,368.88221031313344,211.56248251307755,53.15867651370354,54.19228565006051,55.67694299679715,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c16
|
||||
fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T13:58:15+0800,2026-08-01T14:00:56+0800,161.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.71871886099689,bench_main_marker_plus_duration,4,0,127.71871886099689,3.9987289259054917,,262144,4096,0.031318823393095684,2052.5104098899187,32.07047515452998,2084.5808850444487,,127678.13387226488,127678.37510351092,127682.15249372879,127682.41568037542,58779.63522000937,59286.87346848892,87520.75181677355,89912.75672257761,67.34946104814811,66.85386279083285,96.14249014501952,98.71897348003505,67.3494309836352,36.35076849604957,36.65076588222291,37.22598696593195,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c4
|
||||
fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T14:00:57+0800,2026-08-01T14:03:36+0800,159.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,125.30951252899831,bench_main_marker_plus_duration,4,0,125.30951252899831,3.9986913595171454,,262144,4096,0.03192096050229504,2091.9720674784076,32.68706355435012,2124.6591310327576,,125268.51625375275,125267.95494600083,125274.71757131862,125275.15292868251,56944.67479274317,57454.862895508995,85207.63279019155,87584.75151082734,66.7877238133036,66.2884575273625,95.11951526871552,97.68299949875195,66.78769251954503,36.23565100133419,36.565644716029055,37.02181185770315,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c4
|
||||
fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:18:31+0800,2026-08-01T14:46:05+0800,1654.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.8975476099877,bench_main_marker_plus_duration,64,0,1619.8975476099877,40.3918658202939,,4194304,65536,0.039508671455442484,2589.2402925038787,40.456879570373104,2629.6971720742517,,1022354.4435263402,950470.9995904996,1619639.9038552889,1619650.3429244931,785995.8297684161,774078.3500669932,1506265.7001026294,1560787.9135349835,231.04458822866482,223.80146996136563,418.57784627036636,439.7380822666596,231.0445591993489,56.0161369794514,56.77989884279668,58.514122637570836,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c64
|
||||
fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:46:06+0800,2026-08-01T15:13:40+0800,1654.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1620.2422464910196,bench_main_marker_plus_duration,64,0,1620.2422464910196,40.39704233770943,,4194304,65536,0.03950026617229964,2588.689443867829,40.44827256043483,2629.137716428264,,1022703.0410756742,950775.2832530241,1620010.8873196847,1620019.4079715456,786335.3449273454,774403.2157540205,1506724.735757988,1561196.831805848,231.05346642065376,223.83287831180922,418.5783493891426,439.7222371669007,231.0534379278651,56.055732507957146,56.75308680802118,58.11881933768746,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c64
|
||||
|
@ -1,25 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_0,371,3626828296476.0,3626837993952.0,71.66006119953522,71.66025280544177,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_3,371,3626914718800.0,3626921347028.0,71.66176876011448,71.66189972281393,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_0,383,3626134447036.0,3626135801124.0,71.58004520955629,71.58007193930987,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_3,383,3626207403788.0,3626212421504.0,71.58148537888337,71.58158442880648,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_0,371,3625059758564.0,3625068124448.0,71.59790115790838,71.59806639097366,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_3,371,3625136689400.0,3625142126056.0,71.59942060496823,71.59952798337986,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_0,382,3625042688652.0,3625045908572.0,71.73367967522776,71.73374339219049,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_3,382,3625122942520.0,3625131203508.0,71.73526776832192,71.73543123979464,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_0,117,894188173332.0,894192013996.0,56.331515043976715,56.33175699575861,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_3,117,908654087048.0,908656485748.0,57.24282975425881,57.24298086608198,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_0,121,908912097324.0,908914949500.0,57.10271925252004,57.102898441471176,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_3,121,894353999288.0,894353927048.0,56.188101670195095,56.188097131691926,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_0,115,894195520320.0,894197433708.0,57.32326114869037,57.32338380826259,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_3,115,908661063200.0,908663318872.0,58.25058864399148,58.250733246007684,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_0,118,908220840268.0,908222493124.0,58.56932688787363,58.5694334772174,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_3,118,893986155676.0,893986644792.0,57.65136084035526,57.651392382457814,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_0,1479,14610070965300.0,14610072489408.0,72.21898772698583,72.21899526079844,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_3,1479,14610085802320.0,14610095102220.0,72.21906106780447,72.21910703810576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_0,1527,14611153636236.0,14611146819600.0,72.16844452340027,72.1684108541891,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_3,1527,14611226802652.0,14611236350532.0,72.16880591214243,72.16885307170732,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_0,1480,14610569508844.0,14610573106108.0,72.18614224806339,72.18616002099213,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_3,1480,14610597458160.0,14610605842440.0,72.18628033667787,72.18632176071877,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_0,1527,14610266497856.0,14610260234736.0,72.18231262718308,72.18228168411397,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_3,1527,14610323772088.0,14610331114336.0,72.18259559166701,72.18263186618948,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||
|
@ -1,7 +0,0 @@
|
||||
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,1785564254150768896,1785564660235236864,406.084467968,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,1785564694981654016,1785565101074958080,406.093304064,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,1785563926377015040,1785564054095733760,127.71871872,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,1785564088360896000,1785564213670408960,125.30951296,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1785565142582675968,1785566762480224000,1619.897548032,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/meta.json
|
||||
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1785566797465815040,1785568417708060928,1620.242245888,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/meta.json
|
||||
|
@ -1,37 +0,0 @@
|
||||
wall_time_ns,node,collector,status,pid
|
||||
1785563885833567643,head,gpu_samples.csv,STARTED,2581028
|
||||
1785563885848767676,head,rdma.csv,STARTED,2581039
|
||||
1785563885862486752,head,dcgm_dmon.log,STARTED,2581053
|
||||
1785563885876598354,head,mpstat.log,STARTED,2581071
|
||||
1785563885891181421,head,pidstat.log,STARTED,2581090
|
||||
1785563885906513050,head,sar_net.log,STARTED,2581117
|
||||
1785563885921729008,head,perf_stat.log,STARTED,2581144
|
||||
1785563885938632713,head,docker_top.log,STARTED,2581179
|
||||
1785563885959612586,head,numa_samples.csv,STARTED,2581251
|
||||
1785563885980492840,worker,gpu_samples.csv,STARTED,2581294
|
||||
1785563885999012597,worker,rdma.csv,STARTED,2581356
|
||||
1785563886014677475,worker,dcgm_dmon.log,STARTED,2581408
|
||||
1785563886031239484,worker,mpstat.log,STARTED,2581455
|
||||
1785563886046314326,worker,pidstat.log,STARTED,2581464
|
||||
1785563886060934631,worker,sar_net.log,STARTED,2581478
|
||||
1785563886075086128,worker,perf_stat.log,STARTED,2581486
|
||||
1785563886090622653,worker,docker_top.log,STARTED,2581497
|
||||
1785563886106525841,worker,numa_samples.csv,STARTED,2581509
|
||||
1785568431501511663,head,gpu_samples.csv,STOPPED,2581028
|
||||
1785568431506303146,head,rdma.csv,STOPPED,2581039
|
||||
1785568431513026775,head,dcgm_dmon.log,STOPPED,2581053
|
||||
1785568431518181488,head,mpstat.log,STOPPED,2581071
|
||||
1785568431523384390,head,pidstat.log,STOPPED,2581090
|
||||
1785568431528223936,head,sar_net.log,STOPPED,2581117
|
||||
1785568431608439580,head,perf_stat.log,STOPPED,2581144
|
||||
1785568431612888149,head,docker_top.log,STOPPED,2581179
|
||||
1785568431617027570,head,numa_samples.csv,STOPPED,2581251
|
||||
1785568431620791766,worker,gpu_samples.csv,STOPPED,2581294
|
||||
1785568431624694474,worker,rdma.csv,STOPPED,2581356
|
||||
1785568431628587426,worker,dcgm_dmon.log,STOPPED,2581408
|
||||
1785568431632294659,worker,mpstat.log,STOPPED,2581455
|
||||
1785568431636153536,worker,pidstat.log,STOPPED,2581464
|
||||
1785568431639870020,worker,sar_net.log,STOPPED,2581478
|
||||
1785568431643739953,worker,perf_stat.log,STOPPED,2581486
|
||||
1785568431647558030,worker,docker_top.log,STOPPED,2581497
|
||||
1785568431651292267,worker,numa_samples.csv,STOPPED,2581509
|
||||
|
File diff suppressed because one or more lines are too long
@ -1 +0,0 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
File diff suppressed because one or more lines are too long
@ -1 +0,0 @@
|
||||
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||
@ -1,12 +0,0 @@
|
||||
stage,case_id,repetition,status,isl,osl,concurrency,completed,duration_s,input_tps,output_tps,total_tps,ttft_p95_ms,tpot_p95_ms,gpu_util_mean_pct,rail_xmit_avg_gbps,rail_xmit_p95_gbps,rail_xmit_max_gbps,dual_rail_xmit_avg_gbps,rail_imbalance_pct,xmit_bytes_per_input_token_per_rail,xmit_bytes_per_total_token_per_rail,rdma_error_delta,measurement_started_at,measurement_ended_at,source_result_dir
|
||||
scout,rdma_scout_i65536_o1_c1,1,COMPLETED,65536,1,1,1,24.1862671909621,2709.6368150803128,0.04134577659729481,2709.67816085691,24167.605170048773,0.0,93.74897875816994,70.9712746848062,85.5972372945909,86.80566056858429,141.9425493696124,0.5207119247408836,3137732.762954712,3137684.8856828967,0.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c4,1,COMPLETED,65536,1,4,4,89.45394851703895,2930.491100122512,0.044715745546302976,2930.535815868058,86526.69315790116,0.0,97.4494891826923,78.27464182068744,86.81919268342193,88.85894624954848,156.5492836413749,0.2703571902588796,3298012.1388320923,3297961.815928407,0.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c16,1,COMPLETED,65536,1,16,16,351.42641380499117,2983.771164628114,0.045528734811830354,2983.816693362926,336717.4506017618,0.0,99.33137833282255,79.89685137869246,86.416508483831,88.20167661610235,159.79370275738495,0.1149304623247205,3335858.008945465,3335807.108568442,0.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c32,1,COMPLETED,65536,1,32,32,702.8175486620166,2983.9209393567885,0.04553102019282209,2983.9664703769813,671139.2772838357,0.0,99.49393723047987,79.92852806348517,86.22587499587476,88.21644913628985,159.85705612697035,0.10091204027725086,3343929.3806581497,3343878.357123648,0.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
scout,rdma_scout_i65536_o1_c64,1,COMPLETED,65536,1,64,64,1402.17612864502,2991.2818470623424,0.04564333873080967,2991.327490401073,1337131.1103800863,0.0,99.63928846991146,79.9743866369171,85.93278751515446,88.32212536165416,159.9487732738342,0.03496630506355109,3339597.901974201,3339546.9445318105,0.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||
confirm,rdma_confirm_i65536_o1024_c4,1,COMPLETED,65536,1024,4,4,127.71871886099689,2052.5104098899187,32.07047515452998,2084.5808850444487,87520.75181677355,96.14249014501952,97.83383458646617,56.71629143023766,86.06750356786138,88.99885476939444,113.43258286047534,1.8596562953363254,3439052.922241211,3386144.415745192,0.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c4,2,COMPLETED,65536,1024,4,4,125.30951252899831,2091.9720674784076,32.68706355435012,2124.6591310327576,85207.63279019155,95.11951526871552,98.63248207885304,57.94863438022769,86.4894823881113,88.5335715781298,115.89726876045536,2.1502935358359734,3438056.544746399,3385163.3671349157,0.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c16,1,COMPLETED,65536,1024,16,16,406.0844679859583,2582.1622905219265,40.3462857894051,2622.5085763113316,336778.7128902273,356.41907673483706,99.38883591065292,71.62084013702234,86.26345231858909,88.9399300384103,143.24168027404468,0.11410582506689998,3458520.1421022415,3405312.139916053,0.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c16,2,COMPLETED,65536,1024,16,16,406.09330375201534,2582.1061079114043,40.34540793611569,2622.45151584752,336924.2522477143,356.06784952074946,99.48923731257209,71.66656730160658,86.33499788736604,89.31414888004788,143.33313460321313,0.19167460586668467,3457155.7233657837,3403968.712237079,0.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c64,1,COMPLETED,65536,1024,64,64,1619.8975476099877,2589.2402925038787,40.456879570373104,2629.6971720742517,1506265.7001026294,418.57784627036636,99.6722631148773,72.19382480758325,86.71108451788668,89.82468221963666,144.38764961516648,0.070112013789426,3483446.6699664593,3429855.182736206,0.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
confirm,rdma_confirm_i65536_o1024_c64,2,COMPLETED,65536,1024,64,64,1620.2422464910196,2588.689443867829,40.44827256043483,2629.137716428264,1506724.735757988,418.5783493891426,99.67225809785734,72.18433270089784,86.67873733217695,89.23672463088391,144.36866540179568,0.005496635275729702,3483400.1801579,3429809.4081554706,0.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||
|
@ -1,205 +0,0 @@
|
||||
{
|
||||
"generated_at": "2026-08-01T15:14:11.558999+08:00",
|
||||
"formula": "rail_gbps ~= input_tps * bytes_per_input_token_per_rail * 8 / 1e9",
|
||||
"plateau_gain_pct": 5.0,
|
||||
"target_rail_gbps": 400.0,
|
||||
"practical_rail_gbps": 360.0,
|
||||
"scout": [
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 1,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2709.6368150803128,
|
||||
"output_tps": 0.04134577659729481,
|
||||
"total_tps": 2709.67816085691,
|
||||
"ttft_p95_ms": 24167.605170048773,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 93.74897875816994,
|
||||
"rail_xmit_avg_gbps": 70.9712746848062,
|
||||
"rail_xmit_p95_gbps": 85.5972372945909,
|
||||
"rail_xmit_max_gbps": 86.80566056858429,
|
||||
"dual_rail_xmit_avg_gbps": 141.9425493696124,
|
||||
"rail_imbalance_pct": 0.5207119247408836,
|
||||
"xmit_bytes_per_input_token_per_rail": 3137732.762954712,
|
||||
"xmit_bytes_per_total_token_per_rail": 3137684.8856828967,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 4,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2930.491100122512,
|
||||
"output_tps": 0.044715745546302976,
|
||||
"total_tps": 2930.535815868058,
|
||||
"ttft_p95_ms": 86526.69315790116,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 97.4494891826923,
|
||||
"rail_xmit_avg_gbps": 78.27464182068744,
|
||||
"rail_xmit_p95_gbps": 86.81919268342193,
|
||||
"rail_xmit_max_gbps": 88.85894624954848,
|
||||
"dual_rail_xmit_avg_gbps": 156.5492836413749,
|
||||
"rail_imbalance_pct": 0.2703571902588796,
|
||||
"xmit_bytes_per_input_token_per_rail": 3298012.1388320923,
|
||||
"xmit_bytes_per_total_token_per_rail": 3297961.815928407,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 16,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2983.771164628114,
|
||||
"output_tps": 0.045528734811830354,
|
||||
"total_tps": 2983.816693362926,
|
||||
"ttft_p95_ms": 336717.4506017618,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.33137833282255,
|
||||
"rail_xmit_avg_gbps": 79.89685137869246,
|
||||
"rail_xmit_p95_gbps": 86.416508483831,
|
||||
"rail_xmit_max_gbps": 88.20167661610235,
|
||||
"dual_rail_xmit_avg_gbps": 159.79370275738495,
|
||||
"rail_imbalance_pct": 0.1149304623247205,
|
||||
"xmit_bytes_per_input_token_per_rail": 3335858.008945465,
|
||||
"xmit_bytes_per_total_token_per_rail": 3335807.108568442,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 32,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2983.9209393567885,
|
||||
"output_tps": 0.04553102019282209,
|
||||
"total_tps": 2983.9664703769813,
|
||||
"ttft_p95_ms": 671139.2772838357,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.49393723047987,
|
||||
"rail_xmit_avg_gbps": 79.92852806348517,
|
||||
"rail_xmit_p95_gbps": 86.22587499587476,
|
||||
"rail_xmit_max_gbps": 88.21644913628985,
|
||||
"dual_rail_xmit_avg_gbps": 159.85705612697035,
|
||||
"rail_imbalance_pct": 0.10091204027725086,
|
||||
"xmit_bytes_per_input_token_per_rail": 3343929.3806581497,
|
||||
"xmit_bytes_per_total_token_per_rail": 3343878.357123648,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "scout",
|
||||
"concurrency": 64,
|
||||
"repetitions": 1,
|
||||
"input_tps": 2991.2818470623424,
|
||||
"output_tps": 0.04564333873080967,
|
||||
"total_tps": 2991.327490401073,
|
||||
"ttft_p95_ms": 1337131.1103800863,
|
||||
"tpot_p95_ms": 0.0,
|
||||
"gpu_util_mean_pct": 99.63928846991146,
|
||||
"rail_xmit_avg_gbps": 79.9743866369171,
|
||||
"rail_xmit_p95_gbps": 85.93278751515446,
|
||||
"rail_xmit_max_gbps": 88.32212536165416,
|
||||
"dual_rail_xmit_avg_gbps": 159.9487732738342,
|
||||
"rail_imbalance_pct": 0.03496630506355109,
|
||||
"xmit_bytes_per_input_token_per_rail": 3339597.901974201,
|
||||
"xmit_bytes_per_total_token_per_rail": 3339546.9445318105,
|
||||
"rdma_error_delta": 0.0
|
||||
}
|
||||
],
|
||||
"confirm": [
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 4,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2072.2412386841634,
|
||||
"output_tps": 32.37876935444005,
|
||||
"total_tps": 2104.6200080386034,
|
||||
"ttft_p95_ms": 86364.19230348256,
|
||||
"tpot_p95_ms": 95.63100270686752,
|
||||
"gpu_util_mean_pct": 98.2331583326596,
|
||||
"rail_xmit_avg_gbps": 57.332462905232674,
|
||||
"rail_xmit_p95_gbps": 86.27849297798633,
|
||||
"rail_xmit_max_gbps": 88.76621317376211,
|
||||
"dual_rail_xmit_avg_gbps": 114.66492581046535,
|
||||
"rail_imbalance_pct": 2.0049749155861494,
|
||||
"xmit_bytes_per_input_token_per_rail": 3438554.733493805,
|
||||
"xmit_bytes_per_total_token_per_rail": 3385653.891440054,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 16,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2582.1341992166654,
|
||||
"output_tps": 40.345846862760396,
|
||||
"total_tps": 2622.480046079426,
|
||||
"ttft_p95_ms": 336851.4825689708,
|
||||
"tpot_p95_ms": 356.24346312779323,
|
||||
"gpu_util_mean_pct": 99.4390366116125,
|
||||
"rail_xmit_avg_gbps": 71.64370371931446,
|
||||
"rail_xmit_p95_gbps": 86.29922510297757,
|
||||
"rail_xmit_max_gbps": 89.12703945922908,
|
||||
"dual_rail_xmit_avg_gbps": 143.28740743862892,
|
||||
"rail_imbalance_pct": 0.15289021546679232,
|
||||
"xmit_bytes_per_input_token_per_rail": 3457837.9327340126,
|
||||
"xmit_bytes_per_total_token_per_rail": 3404640.426076566,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
{
|
||||
"stage": "confirm",
|
||||
"concurrency": 64,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2588.964868185854,
|
||||
"output_tps": 40.45257606540397,
|
||||
"total_tps": 2629.4174442512576,
|
||||
"ttft_p95_ms": 1506495.2179303085,
|
||||
"tpot_p95_ms": 418.5780978297545,
|
||||
"gpu_util_mean_pct": 99.67226060636732,
|
||||
"rail_xmit_avg_gbps": 72.18907875424054,
|
||||
"rail_xmit_p95_gbps": 86.69491092503182,
|
||||
"rail_xmit_max_gbps": 89.53070342526028,
|
||||
"dual_rail_xmit_avg_gbps": 144.37815750848108,
|
||||
"rail_imbalance_pct": 0.03780432453257785,
|
||||
"xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
|
||||
"xmit_bytes_per_total_token_per_rail": 3429832.295445838,
|
||||
"rdma_error_delta": 0.0
|
||||
}
|
||||
],
|
||||
"selected_confirm_concurrencies": [
|
||||
4,
|
||||
16,
|
||||
64
|
||||
],
|
||||
"observed_plateau_concurrency": 16,
|
||||
"best_scout_bandwidth_concurrency": 64,
|
||||
"linear_gbps_per_input_tps": 0.026656368057740153,
|
||||
"modeled_bytes_per_input_token_per_rail": 3332046.007217519,
|
||||
"required_input_tps_for_400g_per_rail": 15005.795205617025,
|
||||
"required_input_tps_for_360g_per_rail": 13505.215685055322,
|
||||
"required_input_tps_for_400g_dual_rail_aggregate": 7502.897602808513,
|
||||
"bandwidth_saturation_fit": {
|
||||
"asymptote": 80.32102454669293,
|
||||
"half_saturation_c": 0.12589254117941673,
|
||||
"rmse": 0.27652726609534695
|
||||
},
|
||||
"input_tps_saturation_fit": {
|
||||
"asymptote": 3003.9418335884134,
|
||||
"half_saturation_c": 0.11220184543019636,
|
||||
"rmse": 7.680357081457988
|
||||
},
|
||||
"best_confirm_point": {
|
||||
"stage": "confirm",
|
||||
"concurrency": 64,
|
||||
"repetitions": 2,
|
||||
"input_tps": 2588.964868185854,
|
||||
"output_tps": 40.45257606540397,
|
||||
"total_tps": 2629.4174442512576,
|
||||
"ttft_p95_ms": 1506495.2179303085,
|
||||
"tpot_p95_ms": 418.5780978297545,
|
||||
"gpu_util_mean_pct": 99.67226060636732,
|
||||
"rail_xmit_avg_gbps": 72.18907875424054,
|
||||
"rail_xmit_p95_gbps": 86.69491092503182,
|
||||
"rail_xmit_max_gbps": 89.53070342526028,
|
||||
"dual_rail_xmit_avg_gbps": 144.37815750848108,
|
||||
"rail_imbalance_pct": 0.03780432453257785,
|
||||
"xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
|
||||
"xmit_bytes_per_total_token_per_rail": 3429832.295445838,
|
||||
"rdma_error_delta": 0.0
|
||||
},
|
||||
"verdict": "COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION"
|
||||
}
|
||||
@ -1,37 +0,0 @@
|
||||
# Phase 2.5 RDMA Demand Model
|
||||
|
||||
- Generated: `2026-08-01T15:14:11.558999+08:00`
|
||||
- Verdict: `COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION`
|
||||
- Model: `rail_gbps ~= input_tps x 3.332 MB/token/rail x 8`
|
||||
- Required Input TPS for 400G per rail: `15006`
|
||||
- Required Input TPS for 360G per rail: `13505`
|
||||
- Required Input TPS for 400G dual-rail aggregate: `7503`
|
||||
- Fitted rail bandwidth ceiling: `80.32 Gbit/s`
|
||||
- Observed plateau concurrency: `16`
|
||||
|
||||
## Scout: 64K -> 1
|
||||
|
||||
| C | Input TPS | Rail avg | Rail P95 | Dual-rail avg | MB/input-token/rail | GPU util |
|
||||
|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 1 | 2709.64 | 70.97 | 85.60 | 141.94 | 3.138 | 93.75 |
|
||||
| 4 | 2930.49 | 78.27 | 86.82 | 156.55 | 3.298 | 97.45 |
|
||||
| 16 | 2983.77 | 79.90 | 86.42 | 159.79 | 3.336 | 99.33 |
|
||||
| 32 | 2983.92 | 79.93 | 86.23 | 159.86 | 3.344 | 99.49 |
|
||||
| 64 | 2991.28 | 79.97 | 85.93 | 159.95 | 3.340 | 99.64 |
|
||||
|
||||
## Confirm: 64K -> 1K
|
||||
|
||||
| C | Reps | Input TPS | Output TPS | Rail avg | Rail P95 | Dual-rail avg | TTFT P95 ms | TPOT P95 ms |
|
||||
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| 4 | 2 | 2072.24 | 32.38 | 57.33 | 86.28 | 114.66 | 86364.19 | 95.63 |
|
||||
| 16 | 2 | 2582.13 | 40.35 | 71.64 | 86.30 | 143.29 | 336851.48 | 356.24 |
|
||||
| 64 | 2 | 2588.96 | 40.45 | 72.19 | 86.69 | 144.38 | 1506495.22 | 418.58 |
|
||||
|
||||
## Evidence
|
||||
|
||||
- Per-case metrics: `rdma_case_metrics.csv`
|
||||
- Machine-readable model: `rdma_demand_model.json`
|
||||
- Scout HCA source: `scout/{head,worker}/rdma.csv`
|
||||
- Confirm HCA source: `confirm/{head,worker}/rdma.csv`
|
||||
- Exact windows: `scout/case_windows.csv`, `confirm/case_windows.csv`
|
||||
- Benchmark summaries: `scout/bench_summary.csv`, `confirm/bench_summary.csv`
|
||||
@ -1 +0,0 @@
|
||||
CONFIRM_CONCURRENCIES='4 16 64'
|
||||
@ -1,13 +0,0 @@
|
||||
run_id=dsv4pro-phase2_5-20260801-130007
|
||||
experiment=dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||
created_at=2026-08-01T13:00:07+0800
|
||||
git_commit=c5fa700c50c0cc1b9e20d4d2d3f03bd485ba0dbf
|
||||
git_dirty=0
|
||||
isl=65536
|
||||
scout_osl=1
|
||||
confirm_osl=1024
|
||||
scout_concurrencies=1 4 16 32 64
|
||||
scout_repetitions=1
|
||||
confirm_repetitions=2
|
||||
target_rail_gbps=400
|
||||
practical_rail_gbps=360
|
||||
@ -1,4 +0,0 @@
|
||||
# case_id stage isl osl concurrency prompt_multiplier min_prompts warmup_requests note
|
||||
rdma_confirm_i65536_o1024_c4 rdma_confirm 65536 1024 4 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
rdma_confirm_i65536_o1024_c16 rdma_confirm 65536 1024 16 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
rdma_confirm_i65536_o1024_c64 rdma_confirm 65536 1024 64 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||
|
@ -1,6 +0,0 @@
|
||||
# case_id stage isl osl concurrency prompt_multiplier min_prompts warmup_requests note
|
||||
rdma_scout_i65536_o1_c1 rdma_scout 65536 1 1 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c4 rdma_scout 65536 1 4 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c16 rdma_scout 65536 1 16 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c32 rdma_scout 65536 1 32 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
rdma_scout_i65536_o1_c64 rdma_scout 65536 1 64 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user