Compare commits
44 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
35512db505 | ||
|
|
81d17407bc | ||
|
|
9652bfdb9d | ||
|
|
7984c25586 | ||
|
|
9e56401384 | ||
|
|
5ed30006a5 | ||
|
|
1e8c36b7d1 | ||
|
|
8189942353 | ||
|
|
1aa6c0f911 | ||
|
|
783c9325ae | ||
|
|
56d286bb4e | ||
|
|
41e2b000a4 | ||
|
|
82b7d91ac0 | ||
|
|
bc491eeeed | ||
|
|
4628d49755 | ||
|
|
e1719bd575 | ||
|
|
c5fa700c50 | ||
|
|
119701a417 | ||
|
|
405608ad23 | ||
|
|
771b868757 | ||
|
|
3b7291e8a0 | ||
|
|
1dc5612e3a | ||
|
|
5096661ce3 | ||
|
|
7be3062c51 | ||
|
|
5f24b7d22f | ||
|
|
e9c5f85500 | ||
|
|
39fc2ba565 | ||
|
|
66d1db8581 | ||
|
|
4892c0b14d | ||
|
|
30664faa41 | ||
|
|
a583c337ba | ||
|
|
72bae06576 | ||
|
|
337195254a | ||
|
|
3964b3d210 | ||
|
|
ca1f2f6337 | ||
|
|
daa4221128 | ||
|
|
451782457d | ||
|
|
ae85217225 | ||
|
|
06b017483c | ||
|
|
25418ec174 | ||
|
|
75182c6ede | ||
|
|
0d3dd86519 | ||
|
|
595bdde5d7 | ||
|
|
d5d96bd7e4 |
1
.gitattributes
vendored
Normal file
1
.gitattributes
vendored
Normal file
@ -0,0 +1 @@
|
|||||||
|
*.nsys-rep filter=lfs diff=lfs merge=lfs -text
|
||||||
137
README.md
137
README.md
@ -1,5 +1,137 @@
|
|||||||
# sskj — 多平台大模型推理性能基准测试项目
|
# sskj — 多平台大模型推理性能基准测试项目
|
||||||
|
|
||||||
|
> **更新（2026-08-27 13:53:26 CST）**
|
||||||
|
>
|
||||||
|
> 完成 Kimi-K3 八节点标准 PD 第一阶段。P 组 601-604 使用 PP8×TP4×EP4、FlashInfer MXFP4、Chunk 8K，D 组 605-608 使用 PP1×TP32×EP32、Marlin，通过 Mooncake 0.3.12.post1 和 4 Rail RDMA 传输；统一 P/D `page_size=64` 后，16K→1 与 16K→512 的 C1/C8 共 91/91 请求成功。代表结果：16K→1 C8 Input TPS 6364.31、TTFT P50/P95 20.555/21.345 秒；16K→512 C8 TPOT P50/P95 63.20/66.55 ms。详见 `experiments/pro6000/kimi3_pro6000_pd_pp8_standard/README.md`。
|
||||||
|
>
|
||||||
|
> **更新（2026-08-03 10:02:07 CST）**
|
||||||
|
>
|
||||||
|
> 将 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 正式 Run `dsv4pro-phase3-20260802-003213` 的六份原始 `.nsys-rep` 纳入 GitLab `hzy` 分支，并通过 Git LFS 保存；普通 Git 仅保存六份报告的 SHA256 清单。Bench 文本、请求响应、服务日志、环境快照不随原始 Trace 上传；约 2.44 GB 的临时 `.sqlite` 也不上传，因为它可由 `.nsys-rep` 重新导出且不构成独立证据。
|
||||||
|
>
|
||||||
|
> **更新（2026-08-02 01:25:30 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 时间线 Profiling。正式 Run `dsv4pro-phase3-20260802-003213` 在 21 分 03 秒内完成 Decode Control、Decode+128K Prefill Mixed、独立 128K Prefill 三段 Capture；Head/Worker 各生成 3 份 Nsight Systems 报告及 Stats，合计 6/6 校验通过。Control 以 GEMM/MoE（Head 44.7%）和 NCCL（31.9%）为主；Mixed 中 NSA/MLA 升至 29.7%，NCCL 为 23.1%，说明 Chunked Prefill 的 Sparse Attention/MLA 计算进入同一 TP16 GPU 关键路径。Prefill Trace 漏采 3 个 Rank 的 CUDA 活动，因此暂不判断固定慢 Rank，也不将 NCCL 单独定性为根因。实验结束后两节点容器和 16 张 GPU 均已清理。
|
||||||
|
>
|
||||||
|
> **更新（2026-08-02 00:35:00 CST）**
|
||||||
|
>
|
||||||
|
> 修复 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 混合 Capture 时序。两轮失败 Run 证明当前镜像的 `/start_profile` 会阻塞 API 事件循环：Profile 激活后发送的 128K 请求只能在捕获结束后进入 Scheduler，因此异步调用仍无法覆盖注入。最终实现先稳定 Decode 背景，再通过原生 `/generate` + `input_ids` 送入 128K 请求；确认首个 8K Chunk Prefill 已开始后才捕获 32 个 Mixed Step。Control Trace 单独提供纯 Decode 基线，Mixed Trace 捕获真实 Prefill+Decode Treatment。混合背景缩为单波 32 请求，Profiling 专用 watchdog 提升至 1800 秒，异常退出会自动将 manifest 标记为 `FAILED`。
|
||||||
|
>
|
||||||
|
> **更新（2026-08-01 19:34:34 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 正式时间线代码交付。正式 Control 与混合背景统一为 `1K -> 1K, C=32`；所有 `/start_profile` 请求显式记录 `start_step` 和 `num_steps`，Control 在活跃 Decode 后跳过 2 Step，混合场景先保留 2 个纯 Decode Step 再注入 `128K -> 1`，长 Prefill 从首个 Chunk 开始捕获。继续沿用已通过的双节点 PyTorch/Nsight smoke、16 Rank/6 份 Nsight 报告 fail-closed 门禁和单服务三段 Capture Range；正式结果尚未生成，因此不创建 `phase3_exp.html` 或 `phase3_code.html`。
|
||||||
|
>
|
||||||
|
> **更新（2026-08-01 15:24:52 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模。正式 Run `dsv4pro-phase2_5-20260801-130007` 完成 Scout 5/5 与 Confirm 6/6；64K Prefill 在 C=16 已进入约 2,984 input tok/s、79.90 Gbit/s/rail 的平台，C=32/64 不再显著增长。拟合通信强度为 3.332 MB/input-token/rail，单 Rail 400G 需约 15,006 input tok/s，约为当前平台的 5 倍，因此当前是模型计算/实现吞吐先饱和，不是 RDMA 先饱和。新增 `phase2_5_exp.html`、`phase2_5_code.html`、精简证据集和可复用的模型部署 RDMA 需求评估流程；实验结束后双节点容器与 16 张 GPU 已清理。
|
||||||
|
>
|
||||||
|
> **更新（2026-08-01 02:40:00 CST）**
|
||||||
|
>
|
||||||
|
> 新增 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2.5 RDMA 需求建模唯一入口。实验保持现有 TP16/EP2 服务参数不变，先以 `64K -> 1` 的 C=1/4/16/32/64 建立 Input TPS 与每 Rail HCA 带宽关系，再自动选择平台前、拐点和最大稳定并发，对 `64K -> 1K` 重复确认。结果将给出每 Token 跨机字节数、400G 所需 Token TPS、并发饱和曲线和“模型计算先饱和还是 RDMA 先饱和”的机器可读结论；正式结果尚未生成，因此暂不创建 Phase 2.5 HTML 档案。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 19:02:00 CST）**
|
||||||
|
>
|
||||||
|
> Phase 3 Nsight smoke 已在 Head/Worker 各生成一份可解析报告；首份时间线确认 NCCL AllReduce、Sparse MLA、CUTLASS GEMM 与 MoE kernel 均可见。根据 smoke 修正正式 capture：Decode 对照和混合负载必须等服务实际进入 `Decode batch` 后才触发；每段等待本节点全部 8 个 rank 完成，而非任一 rank；双节点近同步停止以减少 Gloo 断链噪声。`nsys stats` 改为可写临时 SQLite、保留 Kernel/API/NVTX 高价值汇总并删除临时库，结果与报告数量均 fail-closed。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 18:53:00 CST）**
|
||||||
|
>
|
||||||
|
> Phase 3 PyTorch Profiler 首轮 smoke 已证明双节点 16 个 rank 均可完成 trace 写盘，Head/Worker 分别生成 8 份、约 1.5/1.4 GB 压缩 trace。修复当前 OpenSSH 不接受 `scp remote:/path/.` 导致 Worker 结果未回收的问题，改为 SSH tar 流式传输；正式入口不再忽略回收失败，并新增至少 16 份 PyTorch rank trace、至少 2 份 Nsight 节点报告的结果门禁。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 18:43:00 CST）**
|
||||||
|
>
|
||||||
|
> 为 Phase 2 实验档案第 11 节的 GPU/DCGM、CPU/进程/NUMA、双 Rail RDMA、PCIe/NCCL 四组结果补充服务器证据路径。每组同时标明结构化汇总 CSV、Head/Worker 原始采样日志、通信原始输出及实际命令文件，便于从结论直接追溯最终 Run `dsv4pro-phase2-20260731-163620` 的证据。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 18:38:00 CST）**
|
||||||
|
>
|
||||||
|
> 新增 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 3 时间线分析唯一入口。代码提供双节点环境审计、PyTorch Profiler smoke、Nsight Systems smoke 与三段正式 capture range，覆盖 Decode 对照、Decode 背景叠加 128K Prefill、独立 128K Prefill；沿用 Phase 1 已验证的 TP16/EP2、CUDA Graph 与双 Rail NET/IB 配置，不重复 Phase 2 的硬件采样和通信微基准。阶段尚未产出正式结果，因此按档案门禁暂不创建 `phase3_exp.html` 或 `phase3_code.html`。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 17:22:20 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2 正式收口。最终 Run `dsv4pro-phase2-20260731-163620` 在 28 分 44 秒内完成 8/8 benchmark、8/8 精确测量窗口和 18/18 采集器启停；混合 Prefill 令 Decode Output TPS 下降 23.96%、TPOT P95 增加 66.75%。GPU/CPU/NUMA、双 Rail RDMA、PCIe P2P、8/16-GPU AllReduce 与 `NCCL_CROSS_NIC=0/1/2` 数据均已归档；证据排除原始 Rail 带宽饱和、整机 CPU 饱和、GPU 降频和 CROSS_NIC 选择作为首要原因。实验结束后两节点容器、端口和 16 张 GPU 均已清理。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 17:08:30 CST）**
|
||||||
|
>
|
||||||
|
> 修正阶段档案门禁的适用范围：规则禁止在 Phase 尚无阶段结果时提前创建 HTML，但不追溯撤下已经完成正式 Run 和结果汇总的档案。Phase 2 已有正式 Run `dsv4pro-phase2-20260731-130125` 的 8/8 结果，因此恢复 `phase2_exp.html`、`phase2_code.html` 及主计划、Phase 1/2 页面之间的双向导航。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 17:01:56 CST）**
|
||||||
|
>
|
||||||
|
> 固定阶段档案生成门禁：某个 Phase 在实验结束、结果汇总并完成汇报确认前，不创建或维护 `phaseN_exp.html` 与 `phaseN_code.html`；进行中只维护代码、原始结果和主计划状态。阶段确认完成后再一次性生成两份最终 HTML。Phase 2 尚待最终正式复跑，因此暂时撤下其两份 HTML 及导航；已完成的 Phase 1 档案继续保留。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 16:46:05 CST）**
|
||||||
|
>
|
||||||
|
> 统一 DeepSeek-V4-Pro 推理优化档案命名与导航：Phase 1/2 实验页分别更名为 `phase1_exp.html`、`phase2_exp.html`，代码页保持 `phase1_code.html`、`phase2_code.html`。主计划中的入口统一为“打开 Phase N 实验档案 / 代码详解”，并为实验页与代码页补齐双向链接。Phase 1 状态同步为固定点 11/11、混合 A/B 3/3、阶段总结果 14/14。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 16:27:58 CST）**
|
||||||
|
>
|
||||||
|
> 完成 Phase 2 Worker 通信代码分发的双节点真机 smoke test。Run `dsv4pro-phase2-stage-smoke-20260731-162326` 依次通过 Head/Worker P2P、两组单机 8-rank AllReduce 和一组双机 16-rank AllReduce，全部 `wrong_values=0`。两节点暂存文件 SHA256 一致；结束后 `/tmp` 暂存、通信容器和 GPU 进程均已清理。本次使用 1 MiB、单次迭代，仅验证执行链路，不作为正式性能数据。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 16:18:35 CST）**
|
||||||
|
>
|
||||||
|
> 修复 Phase 2 双节点通信基线在 Worker 启动后立即退出的问题。唯一入口仍只在 Head `174.1.51.5` 执行；脚本现在按 `RUN_ID` 将 `communication_baseline.py` 自动暂存到 Head/Worker 的 `/tmp`，校验 SHA256 后只读挂载进通信容器，并在结果目录保存当次源码与哈希。Worker `174.1.51.7` 不再依赖同路径 Git 工作树，异常退出也会清理暂存文件。新增回归测试后 Phase 2 单元测试为 9/9。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 15:59:49 CST）**
|
||||||
|
>
|
||||||
|
> 在 DeepSeek-V4-Pro 双机 Pro6000D 推理优化主计划和 Phase 2 实施档案中增加 `phase2_code.html` 的直接入口，便于从阶段状态、实验命令与结果页面跳转到对应代码调用关系和逐行实现说明。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 15:33:02 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2 最终采集代码。提交 `30664faa41f8` 新增正式 benchmark 精确时间窗、双节点 DCGM fail-closed 门禁、进程级 5 秒 Host 采样、机内 PCIe P2P 全矩阵、单机 8-rank 与双机 16-rank AllReduce、`NCCL_CROSS_NIC=0/1/2` A/B，以及与 Phase 2 第 5 节逐项对应的结构化 CSV 和 `report.md`。通信/NCCL 术语档案同步补充 DCGM、Host Engine、SM、Warp、NUMA、NIC、PIX/SYS、algbw/busbw 与 Linux 监控工具；最终双机复跑前不进入 Phase 3。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 14:15:00 CST）**
|
||||||
|
>
|
||||||
|
> 恢复并完善 Phase 2 档案中的采集命令与指标解释。第 5 节现按实际实现记录时间 Marker、`nvidia-smi`、DCGM Field 1001–1005/1009/1010、`mpstat`、`pidstat`、`perf`、`numastat`、`sar`、`ethtool` 和 `mlx5_0/mlx5_3` HCA Counter，并逐项说明字段含义、分析方法及对应原始/汇总文件。同步记录首轮线程级 1 秒 `pidstat` 日志过重，后续应改为进程级 5 秒采样。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 13:59:12 CST）**
|
||||||
|
>
|
||||||
|
> 补充 Phase 2 双节点执行边界：正式实验前仅需在 Worker `174.1.51.7` 启动并验证 `nvidia-dcgm` Host Engine；完整 `run_hardware_contention_attribution.sh all` 入口仍然只在 Head `174.1.51.5` 执行，由其通过 SSH 管理 Worker 服务与采集器。文档明确列出两台节点分别需要运行的命令，避免在 Worker 重复启动整套实验。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 13:40:03 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 的 Phase 2 首轮硬件归因。正式 Run `dsv4pro-phase2-20260731-130125` 在 26 分 26 秒内完成 8/8 个 benchmark，无 OOM；混合负载下 Decode Output TPS 下降 24.03%，TPOT P95 增加 66.79%。两端双 Rail `NET/IB + GDRDMA` 流量均衡、错误增量为 0，最高每 Rail 平均约 70 Gbit/s，原始 RoCE 带宽未饱和；整机 CPU 与 GPU 频率也未见全局瓶颈。档案补充了 6000D 无 NVLink、机内 PCIe P2P/IPC、跨机 GDRDMA 的真实通信路径，并记录 Worker DCGM Host Engine 未启动、Case 时间窗过宽和 `pidstat` 日志过大的采集限制。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 13:11:40 CST）**
|
||||||
|
>
|
||||||
|
> 新增 Phase 1 与 Phase 2 的独立代码详解 HTML 档案，行号固定到提交 `ca1f2f63375c`。文档从唯一入口展开到配置来源、文件调用关系、双机服务与 RDMA 门禁、benchmark 请求生成、混合 Prefill/Decode 时序、两节点采集器、Case 时间窗切片和结构化结果，并为 `MEM_FRACTION_STATIC` 等关键变量记录“默认值定义 → Shell 传递 → 服务参数 → Run 证据”的完整追踪路径。代码档案保持独立，不加入主计划 HTML 或阶段介绍 HTML 的导航。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 11:57:13 CST）**
|
||||||
|
>
|
||||||
|
> 实现 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 的 Phase 2 硬件与资源竞争归因。新增唯一入口 `run_hardware_contention_attribution.sh`，内部复用 Phase 1 的双机服务与 benchmark，不要求用户手工启动 Phase 1；默认重放长/并发 Prefill、普通/持续/长上下文 Decode 和混合 Prefill/Decode A/B。Head 与 Worker 在同一诊断窗口采集 GPU、DCGM、CPU、进程、NUMA、`eth0/eth3` 和 `mlx5_0/mlx5_3` RDMA 数据，并保存 Case marker、完整命令、Manifest 和结构化汇总。正式执行只需运行 Phase 2 的 `all` 入口。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 10:45:24 CST）**
|
||||||
|
>
|
||||||
|
> 为 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 的 Phase 2 硬件归因同步宿主机监控环境：`174.1.51.5` 与 `174.1.51.7` 均安装 `sysstat 12.5.2`，`pidstat`、`mpstat`、`sar -n DEV,EDEV` 的 1 秒实时采样验证通过。两节点的 sysstat service/timer 保持 `disabled/inactive`，正式实验由编排脚本显式启停采集，避免后台周期任务干扰 Case。Phase 2 必须同时保存 Head/Worker 的 sysstat、GPU、SGLang 与 RDMA 时间序列；`docker top` 用于 PID 映射，`perf` 用于 CPU 硬件事件，`numastat` 用于 NUMA 亲和，`mlx5_0/mlx5_3` HCA 计数器用于 RoCE 数据面，均不能由 sysstat 完整替代。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-31 00:11:25 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 的长 Decode 补测，Run `dsv4pro-phase1-long-decode-20260730-234236` 结果 2/2 成功。`1K → 4K, C=16` 生成 65,536 个目标输出 token，Output TPS 为 310.02，TPOT P95 为 50.33 ms；相比 `1K → 1K, C=16`，Output TPS 增加 4.99%，TPOT P95 仅增加 0.62%。`128K → 1K, C=1` 生成完整 1,024 token，TTFT P95 为 49.326 s，TPOT P95 为 32.24 ms；其 TTFT 与 `128K → 1` 纯 Prefill 仅差 2.03%，TPOT P95 相比 1K Context 仅增加 2.47%。Phase 1 至此共完成 11 个固定点和 3 个混合结果，14/14 成功；两节点容器和 16 张 GPU 已清理。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-30 23:38:42 CST）**
|
||||||
|
>
|
||||||
|
> 为 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 Phase 1 增加两个最小长 Decode 补充场景：`1K → 4K, C=16` 用于观察持续长输出、KV 增长与稳态 Decode，`128K → 1K, C=1` 用于观察长上下文上的 Decode Attention。继续复用唯一 `run_quick_map.sh` 入口和 `CASE_IDS` 过滤，不新增启动脚本；场景验证单测由 9 点更新为 11 点，Shell、3 个 Python 单测和两点 Dry-run 已通过。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-30 23:06:01 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang TP16 Phase 1 正式 quick-map。Head 与 Worker 均通过 `mlx5_0/mlx5_3` 双 Rail `NET/IB + GDRDMA` 门禁；9 个固定点和 3 个混合 A/B 结果共 12/12 成功，总用时 28 分 36 秒。32K/128K 单请求 Prefill 输入吞吐为 2,652.76/2,710.16 token/s；在 C=32 Decode 中注入一个 128K Prefill 后，Output TPS 下降 24.08%，TPOT P95 增加 66.55%。阶段 HTML 已重写为只保留成功结果，并补充正式汇总、运行清单和 Phase 2 三个诊断负载；两节点容器和 16 张 GPU 已清理。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-30 18:40:53 CST）**
|
||||||
|
>
|
||||||
|
> 为 DeepSeek-V4-Pro 双机 TP16 quick-map 的唯一入口新增 `CASE_IDS` 场景过滤和未知 Case 预检，可在完整九点实验前先跑 1K/32K Prefill 与 C1/C32 Decode 四点 Sanity；运行清单会记录实际过滤条件。同步精简阶段档案：正文只保留最终成功 Run 与有效结论，历史失败压缩到末尾经验教训。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-30 17:54:30 CST）**
|
||||||
|
>
|
||||||
|
> 为 DeepSeek-V4-Pro 双机 TP16 quick-map 加入 RDMA fail-closed 启动保护。唯一 Shell 入口现在只允许计算网 `eth0/eth3` 与其 RDMA HCA `mlx5_0/mlx5_3`，在两端预检并透传 `rdma_cm/uverbs0/uverbs3`，服务健康后必须从两端 NCCL INFO 日志证明 `NET/IB` 和两条 HCA 均已启用，否则 benchmark 不会开始。运行清单新增 RDMA 开关、强制校验和设备路径；语法、结果解析器单测、完整 dry-run 及非法网卡/HCA 负例均已通过，真机 NET/IB 验证与 Phase 1 重跑尚未执行。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-30 17:45:18 CST）**
|
||||||
|
>
|
||||||
|
> 修正 DeepSeek-V4-Pro 双机 TP16 quick-map 的 NCCL Socket 网卡错误。控制组确认服务容器未暴露 `/dev/infiniband`，NCCL 实际回退 `NET/Socket`；旧 quick-map 又误选低速非计算网，导致冷 1K/32K Prefill 比 `eth0` 计算网 Socket 控制组慢约 10.9 倍/13.25 倍。默认 `NCCL_SOCKET_IFNAME` 已改为 `eth0`；旧约 65 token/s 结果降级为事故证据，Phase 2 暂停并等待修正后的 Phase 1。新增双机通信/NCCL 术语 HTML、网络审计报告，并保留原 `/data/qqt/sskj` TP16 脚本不变。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-30 16:38:41 CST）**
|
||||||
|
>
|
||||||
|
> 完成 DeepSeek-V4-Pro 双机 TP16 新旧脚本 TTFT 口径审计。确认旧产物受到 16 条 Warm-up、跨 Case 固定 Seed 递增长度、未清 Prefix Cache 及前一轮残留服务状态影响；同配置冷请求稳定复现约 16 秒/1K。新增 Phase 1 结果、脚本审计和 Phase 2 设计 HTML 档案，后续 Cold/Warm Prefix 指标分开报告。
|
||||||
|
>
|
||||||
|
> **更新（2026-07-30 14:33:52 CST）**
|
||||||
|
>
|
||||||
|
> 新增独立的 `dsv4pro_pro6000d_2node_sglang_tp16_quick_map` 快速性能地图与混合干扰 A/B。实验只保留一个 Shell 入口；旧 TP16 全量脚本保持不变。首轮真机验证已确认双机 TP16 服务可用，并据实测耗时将快速矩阵缩为一波请求，同时修正 Warm-up 污染 Prefix Cache 和混合负载注入时序。
|
||||||
|
>
|
||||||
> 历史更新见 `git log`。项目目的与工作流见下方。
|
> 历史更新见 `git log`。项目目的与工作流见下方。
|
||||||
> **项目目的**：当新显卡（GPU/NPU）到货时，用最短时间完成大模型在该平台上的推理性能评估与部署配置选型。
|
> **项目目的**：当新显卡（GPU/NPU）到货时，用最短时间完成大模型在该平台上的推理性能评估与部署配置选型。
|
||||||
> 当前模型：DeepSeek-V4-Flash（FP8 / INT8）；后续接入 GLM5.2，**完全复用**本项目的实验与报告流程。
|
> 当前模型：DeepSeek-V4-Flash（FP8 / INT8）；后续接入 GLM5.2，**完全复用**本项目的实验与报告流程。
|
||||||
@ -44,8 +176,11 @@
|
|||||||
| `experiments/p800/dsv4_p800_sglang_tp_dp_matrix/` | P800 + SGLang（INT8；TP2/DP4 启动 OOM 无数据，见 config.env 注释） |
|
| `experiments/p800/dsv4_p800_sglang_tp_dp_matrix/` | P800 + SGLang（INT8；TP2/DP4 启动 OOM 无数据，见 config.env 注释） |
|
||||||
| `experiments/pro6000/dsv4_pro6000_vllm_tp_dp_matrix/` | RTX 6000D + vLLM |
|
| `experiments/pro6000/dsv4_pro6000_vllm_tp_dp_matrix/` | RTX 6000D + vLLM |
|
||||||
| `experiments/pro6000/dsv4_pro6000_sglang_tp_dp_matrix/` | RTX 6000D + SGLang |
|
| `experiments/pro6000/dsv4_pro6000_sglang_tp_dp_matrix/` | RTX 6000D + SGLang |
|
||||||
|
| `experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/` | 双机 RTX 6000D + DeepSeek-V4-Pro，SGLang TP16 快速性能地图与混合干扰 A/B |
|
||||||
|
| `experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/` | 双机 RTX 6000D + DeepSeek-V4-Pro，SGLang TP16 硬件与资源竞争归因 |
|
||||||
|
| `experiments/pro6000/kimi3_pro6000_pd_pp8_standard/` | Kimi-K3 八节点标准 PD：P=PP8/TP4/EP4、D=PP1/TP32/EP32、Mooncake 4 Rail RDMA，含 16K→1/512 C1/C8 原始证据 |
|
||||||
|
|
||||||
每个目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_concurrency.sh` 从 C=1 指数倍增搜饱和点；`run_adaptive_concurrency_add16.sh` 从 C=16 线性 +16 步进、带 TTFT SLO 停止与回退（当前主力用法，见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`）。
|
TP/DP matrix 目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_concurrency.sh` 从 C=1 指数倍增搜饱和点；`run_adaptive_concurrency_add16.sh` 从 C=16 线性 +16 步进、带 TTFT SLO 停止与回退（当前主力用法，见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`）。专项 quick-map 的入口以对应目录 README 为准。
|
||||||
|
|
||||||
### 其他实验（H200 老形态 & 专项）
|
### 其他实验（H200 老形态 & 专项）
|
||||||
|
|
||||||
|
|||||||
816
docs/dsv4pro_pro6000d_2node_sglang/6000D双机通信与NCCL术语入门.html
Normal file
816
docs/dsv4pro_pro6000d_2node_sglang/6000D双机通信与NCCL术语入门.html
Normal file
@ -0,0 +1,816 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>6000D 双机通信、NCCL 与 Profiling 术语入门</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
color-scheme: light;
|
||||||
|
--ink: #17202a;
|
||||||
|
--muted: #5f6b76;
|
||||||
|
--line: #d9dee3;
|
||||||
|
--panel: #f5f7f8;
|
||||||
|
--cyan: #087b83;
|
||||||
|
--cyan-soft: #e5f4f3;
|
||||||
|
--orange: #a64b17;
|
||||||
|
--orange-soft: #fff0e5;
|
||||||
|
--green: #287a45;
|
||||||
|
--green-soft: #eaf6ee;
|
||||||
|
--red: #a73535;
|
||||||
|
--red-soft: #fdecec;
|
||||||
|
--code: #f0f2f4;
|
||||||
|
--max: 1160px;
|
||||||
|
}
|
||||||
|
|
||||||
|
* { box-sizing: border-box; }
|
||||||
|
|
||||||
|
html { scroll-behavior: smooth; }
|
||||||
|
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
color: var(--ink);
|
||||||
|
background: #fff;
|
||||||
|
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", "PingFang SC",
|
||||||
|
"Hiragino Sans GB", "Microsoft YaHei", sans-serif;
|
||||||
|
font-size: 16px;
|
||||||
|
line-height: 1.72;
|
||||||
|
}
|
||||||
|
|
||||||
|
header {
|
||||||
|
color: #fff;
|
||||||
|
background: #172f35;
|
||||||
|
border-bottom: 5px solid #e57932;
|
||||||
|
}
|
||||||
|
|
||||||
|
.header-inner {
|
||||||
|
max-width: var(--max);
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 46px 28px 40px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.eyebrow {
|
||||||
|
margin: 0 0 8px;
|
||||||
|
color: #9ed8d5;
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 700;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
h1, h2, h3 { letter-spacing: 0; }
|
||||||
|
|
||||||
|
header h1 {
|
||||||
|
max-width: 850px;
|
||||||
|
margin: 0;
|
||||||
|
font-size: clamp(32px, 5vw, 54px);
|
||||||
|
line-height: 1.12;
|
||||||
|
}
|
||||||
|
|
||||||
|
.header-meta {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 10px 22px;
|
||||||
|
margin-top: 22px;
|
||||||
|
color: #d4e4e5;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.layout {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 240px minmax(0, 1fr);
|
||||||
|
gap: 38px;
|
||||||
|
max-width: var(--max);
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 34px 28px 70px;
|
||||||
|
}
|
||||||
|
|
||||||
|
nav {
|
||||||
|
position: sticky;
|
||||||
|
top: 20px;
|
||||||
|
align-self: start;
|
||||||
|
padding-right: 20px;
|
||||||
|
border-right: 1px solid var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
|
nav strong {
|
||||||
|
display: block;
|
||||||
|
margin-bottom: 10px;
|
||||||
|
font-size: 13px;
|
||||||
|
color: var(--muted);
|
||||||
|
}
|
||||||
|
|
||||||
|
nav a {
|
||||||
|
display: block;
|
||||||
|
padding: 6px 0;
|
||||||
|
color: #40515a;
|
||||||
|
font-size: 14px;
|
||||||
|
text-decoration: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
nav a:hover { color: var(--cyan); }
|
||||||
|
|
||||||
|
main { min-width: 0; }
|
||||||
|
|
||||||
|
h2 {
|
||||||
|
margin: 48px 0 16px;
|
||||||
|
padding-top: 8px;
|
||||||
|
font-size: 28px;
|
||||||
|
line-height: 1.25;
|
||||||
|
border-top: 2px solid var(--ink);
|
||||||
|
}
|
||||||
|
|
||||||
|
h2:first-child { margin-top: 0; }
|
||||||
|
|
||||||
|
h3 {
|
||||||
|
margin: 30px 0 10px;
|
||||||
|
font-size: 20px;
|
||||||
|
line-height: 1.35;
|
||||||
|
}
|
||||||
|
|
||||||
|
p { margin: 10px 0; }
|
||||||
|
|
||||||
|
a { color: var(--cyan); }
|
||||||
|
|
||||||
|
code {
|
||||||
|
padding: 2px 5px;
|
||||||
|
border-radius: 3px;
|
||||||
|
background: var(--code);
|
||||||
|
font-family: "SFMono-Regular", Consolas, monospace;
|
||||||
|
font-size: .92em;
|
||||||
|
}
|
||||||
|
|
||||||
|
pre {
|
||||||
|
overflow-x: auto;
|
||||||
|
margin: 14px 0;
|
||||||
|
padding: 16px 18px;
|
||||||
|
color: #e8f1f1;
|
||||||
|
background: #1c292d;
|
||||||
|
border-left: 4px solid #4cb3ae;
|
||||||
|
border-radius: 4px;
|
||||||
|
line-height: 1.55;
|
||||||
|
}
|
||||||
|
|
||||||
|
pre code {
|
||||||
|
padding: 0;
|
||||||
|
color: inherit;
|
||||||
|
background: transparent;
|
||||||
|
}
|
||||||
|
|
||||||
|
table {
|
||||||
|
width: 100%;
|
||||||
|
margin: 16px 0 24px;
|
||||||
|
border-collapse: collapse;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
th, td {
|
||||||
|
padding: 11px 12px;
|
||||||
|
text-align: left;
|
||||||
|
vertical-align: top;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
|
th {
|
||||||
|
color: #26363e;
|
||||||
|
background: #eef1f3;
|
||||||
|
}
|
||||||
|
|
||||||
|
ul, ol { padding-left: 24px; }
|
||||||
|
|
||||||
|
.note, .warning, .finding, .good {
|
||||||
|
margin: 18px 0;
|
||||||
|
padding: 15px 18px;
|
||||||
|
border-left: 4px solid;
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.note { background: var(--cyan-soft); border-color: var(--cyan); }
|
||||||
|
.warning { background: var(--orange-soft); border-color: var(--orange); }
|
||||||
|
.finding { background: var(--red-soft); border-color: var(--red); }
|
||||||
|
.good { background: var(--green-soft); border-color: var(--green); }
|
||||||
|
|
||||||
|
.path {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: repeat(5, minmax(0, 1fr));
|
||||||
|
gap: 8px;
|
||||||
|
margin: 18px 0 24px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.path div {
|
||||||
|
min-height: 108px;
|
||||||
|
padding: 12px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-top: 4px solid var(--cyan);
|
||||||
|
background: var(--panel);
|
||||||
|
}
|
||||||
|
|
||||||
|
.path b { display: block; margin-bottom: 5px; }
|
||||||
|
|
||||||
|
.split {
|
||||||
|
display: grid;
|
||||||
|
grid-template-columns: 1fr 1fr;
|
||||||
|
gap: 18px;
|
||||||
|
margin: 18px 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.split section {
|
||||||
|
padding: 16px 18px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 4px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.split h3 { margin-top: 0; }
|
||||||
|
|
||||||
|
.metric {
|
||||||
|
font-variant-numeric: tabular-nums;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
|
footer {
|
||||||
|
padding: 26px 28px 40px;
|
||||||
|
color: var(--muted);
|
||||||
|
text-align: center;
|
||||||
|
border-top: 1px solid var(--line);
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (max-width: 900px) {
|
||||||
|
.layout { grid-template-columns: 1fr; }
|
||||||
|
nav {
|
||||||
|
position: static;
|
||||||
|
padding: 0 0 18px;
|
||||||
|
border-right: 0;
|
||||||
|
border-bottom: 1px solid var(--line);
|
||||||
|
}
|
||||||
|
nav a { display: inline-block; margin-right: 14px; }
|
||||||
|
.path { grid-template-columns: 1fr; }
|
||||||
|
.split { grid-template-columns: 1fr; }
|
||||||
|
}
|
||||||
|
|
||||||
|
@media print {
|
||||||
|
nav { display: none; }
|
||||||
|
.layout { display: block; max-width: none; }
|
||||||
|
header { color: #000; background: #fff; border-bottom-color: #000; }
|
||||||
|
.header-meta, .eyebrow { color: #333; }
|
||||||
|
pre { color: #000; background: #f4f4f4; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<div class="header-inner">
|
||||||
|
<p class="eyebrow">Two-node communication primer</p>
|
||||||
|
<h1>6000D 双机通信、NCCL 与 Profiling 术语入门</h1>
|
||||||
|
<div class="header-meta">
|
||||||
|
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||||
|
<span>规模：16 GPU / TP16</span>
|
||||||
|
<span>版本：2026-07-31 15:25 CST</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<div class="layout">
|
||||||
|
<nav aria-label="目录">
|
||||||
|
<strong>阅读顺序</strong>
|
||||||
|
<a href="#mental-model">1. 一张总图</a>
|
||||||
|
<a href="#names">2. 设备名字</a>
|
||||||
|
<a href="#glossary">3. 核心术语</a>
|
||||||
|
<a href="#bootstrap">4. Bootstrap</a>
|
||||||
|
<a href="#parameters">5. NCCL 参数</a>
|
||||||
|
<a href="#logs">6. 日志怎么读</a>
|
||||||
|
<a href="#incident">7. 本次事故</a>
|
||||||
|
<a href="#checklist">8. 排查清单</a>
|
||||||
|
<a href="#sources">9. 官方资料</a>
|
||||||
|
</nav>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<h2 id="mental-model">1. 先建立一张总图</h2>
|
||||||
|
<p>
|
||||||
|
SGLang 不会自己搬运 16 张 GPU 之间的 Tensor。模型代码发起 TP/MoE 通信，
|
||||||
|
NCCL 决定用什么算法、经过哪条链路把数据送到其他 rank。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div class="path" aria-label="通信路径">
|
||||||
|
<div><b>SGLang</b>执行模型层、TP16 和 EP2</div>
|
||||||
|
<div><b>Collective</b>AllReduce、AllGather、ReduceScatter、AllToAll</div>
|
||||||
|
<div><b>NCCL</b>构造 rank、ring/tree 和 channel</div>
|
||||||
|
<div><b>Transport</b>机内 P2P/IPC；跨机 NET/IB 或 NET/Socket</div>
|
||||||
|
<div><b>硬件</b>GPU、PCIe、HCA、网卡、光模块、交换机</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="note">
|
||||||
|
<strong>最重要的区分：</strong>
|
||||||
|
<code>NCCL bootstrap</code> 是“启动时要完成的一件事”；
|
||||||
|
<code>NCCL_SOCKET_IFNAME</code> 是“选择 IP 网卡的一个参数”；
|
||||||
|
<code>NET/IB</code> 和 <code>NET/Socket</code> 才是 NCCL 实际搬运数据的传输后端。
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h3>两类跨机路径</h3>
|
||||||
|
<div class="split">
|
||||||
|
<section>
|
||||||
|
<h3>理想路径：RDMA</h3>
|
||||||
|
<p><code>GPU → HCA → RoCE 网络 → HCA → GPU</code></p>
|
||||||
|
<p>日志应出现 <code>NET/IB</code>，支持时还会出现 <code>GDRDMA</code>。</p>
|
||||||
|
</section>
|
||||||
|
<section>
|
||||||
|
<h3>回退路径：TCP Socket</h3>
|
||||||
|
<p><code>GPU/CPU → Linux Socket → ethX → TCP/IP → ethX</code></p>
|
||||||
|
<p>日志会出现 <code>Using network Socket</code>。这不是报错，但性能通常低得多。</p>
|
||||||
|
</section>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2 id="names">2. eth0 和 mlx5_0 不是同一个设备</h2>
|
||||||
|
<div class="note">
|
||||||
|
<strong>400G 是物理 Ethernet 端口的标称链路速率。</strong>
|
||||||
|
<code>eth0</code> 是该端口的 Linux netdev/IP 入口；
|
||||||
|
<code>mlx5_0</code> 是映射到该端口的 RDMA Verbs/HCA 入口。
|
||||||
|
二者相关联，但不相等，也不代表 TCP 或 RDMA 应用一定能跑到 400G。
|
||||||
|
</div>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>名字</th><th>属于哪一层</th><th>负责什么</th><th>本机实例</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td><code>eth0</code></td>
|
||||||
|
<td>Linux IP 网卡接口</td>
|
||||||
|
<td>配置 IP、TCP/UDP、路由；由 <code>NCCL_SOCKET_IFNAME</code> 选择</td>
|
||||||
|
<td>400 Gbit/s 计算网</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>eth3</code></td>
|
||||||
|
<td>Linux IP 网卡接口</td>
|
||||||
|
<td>第二条计算网 Rail</td>
|
||||||
|
<td>400 Gbit/s 计算网</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>mlx5_0</code></td>
|
||||||
|
<td>RDMA HCA / Verbs 设备</td>
|
||||||
|
<td>供 <code>NET/IB</code> 使用；由 <code>NCCL_IB_HCA</code> 选择</td>
|
||||||
|
<td>对应 <code>eth0</code>，挂 switch 1</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>mlx5_3</code></td>
|
||||||
|
<td>RDMA HCA / Verbs 设备</td>
|
||||||
|
<td>第二条 RDMA Rail</td>
|
||||||
|
<td>对应 <code>eth3</code>，挂 switch 2</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>/dev/infiniband/uverbs0</code></td>
|
||||||
|
<td>Linux 字符设备</td>
|
||||||
|
<td>容器进程访问 RDMA Verbs 的入口</td>
|
||||||
|
<td>对应 <code>mlx5_0</code></td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<pre><code>mlx5_0 port 1 ==> eth0 (Up)
|
||||||
|
mlx5_3 port 1 ==> eth3 (Up)</code></pre>
|
||||||
|
|
||||||
|
<p>
|
||||||
|
同一条物理端口可以同时暴露 Linux IP 接口和 RDMA HCA。
|
||||||
|
<code>eth0</code> 是 IP/Socket 世界的入口，<code>mlx5_0</code> 是 RDMA
|
||||||
|
Verbs 世界的入口。<code>ibdev2netdev</code> 输出的是映射关系，不是等号。
|
||||||
|
本项目部署时只把 <code>eth0/eth3</code> 作为节点间计算网。
|
||||||
|
</p>
|
||||||
|
<pre><code>同一条 400G 物理 Ethernet 端口
|
||||||
|
├── eth0 -> Linux netdev -> IP / TCP Socket
|
||||||
|
└── mlx5_0 -> RDMA HCA -> RoCE / Verbs / GDRDMA</code></pre>
|
||||||
|
<p>
|
||||||
|
<code>400 Gbit/s = 50 GB/s</code> 只是单方向理论线速。协议开销、PCIe、
|
||||||
|
CPU、Socket 线程、消息大小和 collective 算法都会让实际
|
||||||
|
<code>algbw/busbw</code> 低于或采用不同统计口径。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="glossary">3. 核心术语字典</h2>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>术语</th><th>通俗解释</th><th>在本项目中的意义</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td><strong>NCCL</strong></td>
|
||||||
|
<td>NVIDIA 的多 GPU 通信库，负责高效实现 collective 和点对点通信。</td>
|
||||||
|
<td>SGLang TP16 每层跨 GPU 通信最终大量落到 NCCL。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>rank</strong></td>
|
||||||
|
<td>一个通信参与者的编号。TP16 communicator 有 rank 0–15。</td>
|
||||||
|
<td>两台机器各 8 个 GPU rank，共 16 个。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>collective</strong></td>
|
||||||
|
<td>一组 rank 共同参与的通信操作。</td>
|
||||||
|
<td>TP 常见 AllReduce、AllGather、ReduceScatter；MoE 还可能有 AllToAll。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>RDMA</strong></td>
|
||||||
|
<td>远端直接内存访问。网卡可直接读写远端内存，减少 CPU 和内核数据拷贝。</td>
|
||||||
|
<td>双机 TP16 希望使用的高速数据路径。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>IB</strong></td>
|
||||||
|
<td>InfiniBand。既是一套高速网络体系，也常被 NCCL 用作 Verbs/RDMA 后端的统称。</td>
|
||||||
|
<td>NCCL 日志里的 <code>NET/IB</code> 也可承载 RoCE，不代表交换机一定是原生 IB。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>RoCE</strong></td>
|
||||||
|
<td>RDMA over Converged Ethernet，在以太网上承载 RDMA。</td>
|
||||||
|
<td>本项目的 400G 计算网类型。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>HCA</strong></td>
|
||||||
|
<td>Host Channel Adapter，提供 RDMA 能力的适配器。</td>
|
||||||
|
<td><code>mlx5_0</code>、<code>mlx5_3</code>。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>GDRDMA</strong></td>
|
||||||
|
<td>GPUDirect RDMA，让 HCA 直接访问 GPU 显存，减少经 CPU 内存中转。</td>
|
||||||
|
<td>跨机 GPU 通信的理想路径，日志可见 <code>via NET/IB/.../GDRDMA</code>。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Socket / TCP</strong></td>
|
||||||
|
<td>普通 IP 网络编程路径。NCCL 找不到 RDMA 时会使用。</td>
|
||||||
|
<td>本次脚本实际发生的回退路径。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Rail</strong></td>
|
||||||
|
<td>一条相对独立的网络通道，通常由一张 HCA 和一套交换路径组成。</td>
|
||||||
|
<td><code>mlx5_0/switch 1</code> 与 <code>mlx5_3/switch 2</code> 是双 Rail。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>ring / tree</strong></td>
|
||||||
|
<td>NCCL 对 collective 的通信拓扑组织方式。</td>
|
||||||
|
<td><code>NCCL_CROSS_NIC</code> 决定同一 ring/tree 能否跨不同 NIC。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>PFC / ECN</strong></td>
|
||||||
|
<td>RoCE 网络控制拥塞和丢包的机制。</td>
|
||||||
|
<td>RDMA 出现 retry、pause 或吞吐抖动时由运维检查。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>NIC</strong></td>
|
||||||
|
<td>Network Interface Card，网卡的统称。它可以暴露普通 IP 接口，也可以提供 RDMA 能力。</td>
|
||||||
|
<td><code>eth0/eth3</code> 是 Linux netdev 名；对应的 RDMA HCA 名是 <code>mlx5_0/mlx5_3</code>。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>NUMA</strong></td>
|
||||||
|
<td>Non-Uniform Memory Access。双路 CPU 机器中，每个 CPU 访问本地内存更快，访问另一侧内存更慢。</td>
|
||||||
|
<td>服务线程、GPU 和 NIC 若跨 NUMA 节点配合，可能增加 Host 侧延迟和 PCIe 路径长度。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>CUDA P2P / IPC</strong></td>
|
||||||
|
<td>P2P 让同机 GPU 直接互访显存；IPC 让不同进程共享可访问的 GPU 内存句柄。</td>
|
||||||
|
<td>6000D 无 NVLink，单机 8 卡的 NCCL P2P/IPC 实际经过 PCIe。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>PIX / SYS</strong></td>
|
||||||
|
<td>NVIDIA 拓扑标签。PIX 表示 GPU 间只跨一个 PCIe Switch；SYS 表示还要跨 CPU/NUMA 互联。</td>
|
||||||
|
<td>GPU0–3、GPU4–7 各自多为 PIX，两组之间为 SYS；P2P 微基准会分别汇总这两类路径。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>AllReduce</strong></td>
|
||||||
|
<td>所有 rank 先归约数据，再让每个 rank 都拿到相同结果的 collective。</td>
|
||||||
|
<td>TP16 高频使用；Phase 2 分别测单机 8 rank 和双机 16 rank。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>algbw / busbw</strong></td>
|
||||||
|
<td><code>algbw</code> 是有效数据量除以操作时间；<code>busbw</code> 再按 collective 的理论链路流量换算，便于比较硬件通信效率。</td>
|
||||||
|
<td>AllReduce 使用 <code>busbw = algbw × 2 × (N-1) / N</code>。两者单位通常为 GB/s，不能与 400 Gbit/s 直接混用。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>DCGM</strong></td>
|
||||||
|
<td>NVIDIA Data Center GPU Manager，一套 GPU 健康、遥测和诊断框架。它比 <code>nvidia-smi</code> 提供更细的 GPU 活跃度计数器。</td>
|
||||||
|
<td>Phase 2 用它采集 SM、Tensor、设备显存接口和 PCIe 活跃度；它不是 Nsight Timeline。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>DCGM Host Engine</strong></td>
|
||||||
|
<td>DCGM 的后台服务，负责连接驱动、维护 GPU 清单并提供指标。systemd 服务通常叫 <code>nvidia-dcgm</code>，底层进程是 <code>nv-hostengine</code>。</td>
|
||||||
|
<td>两节点都必须运行；否则 <code>dcgmi dmon</code> 客户端存在也无法采集。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>dcgmi / Field ID</strong></td>
|
||||||
|
<td><code>dcgmi</code> 是 DCGM 命令行客户端；Field ID 是某个遥测指标的数字编号。</td>
|
||||||
|
<td>Phase 2 使用 1001–1005、1009、1010，并把缺失样本保留为 <code>-</code>，不会当成 0。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>SM</strong></td>
|
||||||
|
<td>Streaming Multiprocessor，GPU 执行 CUDA Warp、Tensor Core 指令和大部分计算的基本处理单元。</td>
|
||||||
|
<td><code>sm_active</code> 高说明 SM 经常在工作，但不等于每个 SM 都满负载。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Warp</strong></td>
|
||||||
|
<td>NVIDIA GPU 同步执行的一组线程，通常包含 32 个 CUDA 线程。</td>
|
||||||
|
<td><code>sm_occupancy</code> 反映活跃 Warp 相对硬件可容纳 Warp 的比例。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>SM Active / Occupancy</strong></td>
|
||||||
|
<td>前者回答“SM 有多少时间在工作”，后者回答“工作时驻留了多少 Warp”。</td>
|
||||||
|
<td>Active 高、Occupancy 低可能来自小 Kernel、资源约束或同步，必须结合后续 Timeline 判断。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>Tensor Active</strong></td>
|
||||||
|
<td>Tensor Core 管线处于活跃状态的时间比例。</td>
|
||||||
|
<td>用于判断矩阵计算单元是否被充分使用；它不是模型总 FLOPS 利用率。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>DRAM Active</strong></td>
|
||||||
|
<td>DCGM 的历史字段名，表示 GPU 设备显存接口活跃比例，不限定显存必须是主机 DRAM 或 HBM。</td>
|
||||||
|
<td>Pro6000D 使用 GDDR7；该指标仍用于观察设备显存带宽压力。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>测量窗口 / Epoch</strong></td>
|
||||||
|
<td>Epoch 是统一的 Unix 时间基准；测量窗口是正式 benchmark 开始到结束的精确时间段。</td>
|
||||||
|
<td>Phase 2 用 <code>Starting main benchmark run</code> 加 benchmark duration 切片，排除数据准备和 Warm-up。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>mpstat</strong></td>
|
||||||
|
<td>查看整机和每个逻辑 CPU 的利用率、I/O Wait 等。</td>
|
||||||
|
<td>回答是否整机 CPU 饱和，或只有少数核心成为热点。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>pidstat</strong></td>
|
||||||
|
<td>按进程统计 CPU、内存、I/O、缺页和上下文切换。</td>
|
||||||
|
<td>Phase 2 使用进程级 5 秒采样，避免旧版线程级 1 秒采样产生数百 MB 日志。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>sar</strong></td>
|
||||||
|
<td>sysstat 套件中的系统活动记录工具，可采集网卡吞吐和错误。</td>
|
||||||
|
<td>Phase 2 只看计算网 <code>eth0/eth3</code>，与 HCA RDMA Counter 分层比较。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>perf stat</strong></td>
|
||||||
|
<td>Linux 性能计数器工具，统计 CPU cycles、instructions、cache miss、迁移和缺页。</td>
|
||||||
|
<td>用于判断 Host 进程是否受 CPU 执行、Cache 或调度开销限制，不提供 GPU Kernel 时间线。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>numastat</strong></td>
|
||||||
|
<td>查看系统或进程在各 NUMA 节点上的内存分布。</td>
|
||||||
|
<td>Phase 2 每 5 秒保存结构化 Node0/Node1 MiB，寻找跨 NUMA 内存放置。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>HCA Counter</strong></td>
|
||||||
|
<td>网卡硬件维护的发送、接收、等待、丢弃和错误累计计数器。</td>
|
||||||
|
<td>Phase 2.5 用 <code>mlx5_0/mlx5_3</code> 的 counter 差值计算正式 benchmark 窗口内的 RDMA Gbit/s。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>bytes/input-token/rail</strong></td>
|
||||||
|
<td>模型每处理一个输入 token，平均要在一条 Rail 上发送的字节数。</td>
|
||||||
|
<td>当前 DSV4-Pro TP16/EP2 Scout 拟合为约 3.332 MB/token/rail；换模型或并行策略必须重新标定。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>带宽平台 / 拐点</strong></td>
|
||||||
|
<td>继续增加并发后，吞吐与网络带宽都几乎不再增长的位置。</td>
|
||||||
|
<td>Phase 2.5 以相邻点的 Input TPS 和 Rail Mean 增益同时低于 5% 判断，当前拐点为 C=16。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><strong>渐近线 / 饱和上限</strong></td>
|
||||||
|
<td>饱和曲线在并发继续增大时逼近、但不会明显超过的预测上限。</td>
|
||||||
|
<td>当前 64K Prefill 的拟合上限约 80.32 Gbit/s/rail，表示模型产流量上限，不表示网卡硬件只能跑 80G。</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2 id="bootstrap">4. NCCL bootstrap 到底是什么</h2>
|
||||||
|
<p>
|
||||||
|
NCCL 本身不是进程启动器。SGLang 先启动各个 worker，NCCL communicator 初始化时，
|
||||||
|
rank 之间需要交换地址、唯一 ID、拓扑和连接信息，这段“先认识彼此”的过程就是 bootstrap。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<ol>
|
||||||
|
<li>每个 rank 启动并获得自己的 rank ID。</li>
|
||||||
|
<li>通过 IP Socket 交换 NCCL unique ID 和连接信息。</li>
|
||||||
|
<li>NCCL 探测 GPU、PCIe、HCA 和节点拓扑。</li>
|
||||||
|
<li>构造 ring/tree/channel。</li>
|
||||||
|
<li>选择真正的数据传输后端：P2P、SHM、NET/IB 或 NET/Socket。</li>
|
||||||
|
</ol>
|
||||||
|
|
||||||
|
<div class="warning">
|
||||||
|
<strong>容易误解的地方：</strong>
|
||||||
|
<code>NCCL_SOCKET_IFNAME</code> 不保证“只用于 bootstrap”。
|
||||||
|
RDMA 正常时它主要承担 bootstrap；RDMA 失败并回退 Socket 后，它也会决定大块 Tensor
|
||||||
|
数据走哪张 IP 网卡。
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2 id="parameters">5. 常见 NCCL 参数</h2>
|
||||||
|
|
||||||
|
<h3>NCCL_SOCKET_IFNAME</h3>
|
||||||
|
<p>筛选 NCCL 可使用的 Linux IP 接口。精确指定接口时可写：</p>
|
||||||
|
<pre><code>NCCL_SOCKET_IFNAME="=eth0"</code></pre>
|
||||||
|
<ul>
|
||||||
|
<li>RDMA 正常：主要影响 bootstrap/OOB IP 连接。</li>
|
||||||
|
<li>RDMA 不可用：决定 <code>NET/Socket</code> 的数据网卡。</li>
|
||||||
|
<li>部署时只允许使用计算网 <code>eth0/eth3</code>；Socket 回退时不能落到其他接口。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h3>NCCL_IB_HCA</h3>
|
||||||
|
<p>筛选 NCCL 的 RDMA HCA。推荐使用精确匹配：</p>
|
||||||
|
<pre><code>NCCL_IB_HCA="=mlx5_0:1,mlx5_3:1"</code></pre>
|
||||||
|
<p>
|
||||||
|
这个变量只是“允许选择谁”，不会自动把宿主机 RDMA 设备送进容器。
|
||||||
|
容器还必须看到 <code>/dev/infiniband/rdma_cm</code>、
|
||||||
|
<code>uverbs0</code> 和 <code>uverbs3</code>。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>NCCL_CROSS_NIC</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>值</th><th>行为</th><th>适用直觉</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td><code>0</code></td>
|
||||||
|
<td>尽量让同一 ring/tree 在不同节点使用对应的同一条 Rail。</td>
|
||||||
|
<td>每张 NIC 接不同交换机、跨 Rail 代价高的 rail-optimized 网络。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>1</code></td>
|
||||||
|
<td>允许同一 ring/tree 在不同节点使用不同 NIC。</td>
|
||||||
|
<td>所有 NIC 进入同一网络 Fabric，跨 NIC 不构成额外问题。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>2</code></td>
|
||||||
|
<td>优先对应同一 NIC，但必要时允许跨 NIC。</td>
|
||||||
|
<td>NCCL 默认的折中策略。</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<div class="note">
|
||||||
|
本项目的 <code>mlx5_0</code> 和 <code>mlx5_3</code> 分挂 switch 1/2，
|
||||||
|
拓扑直觉上更偏向 <code>0</code> 或默认 <code>2</code>。最终值必须在
|
||||||
|
<strong>真正启用 NET/IB 后</strong>用 all_reduce 和 SGLang 端到端 A/B 决定。
|
||||||
|
当 NCCL 使用 NET/Socket 时，这个参数不参与路径选择。
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h3>NCCL_DEBUG 与 NCCL_DEBUG_SUBSYS</h3>
|
||||||
|
<pre><code>NCCL_DEBUG=INFO
|
||||||
|
NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING</code></pre>
|
||||||
|
<p>用于确认实际路径，诊断完成后应关闭，正式性能数据不要长期带 INFO 日志。</p>
|
||||||
|
|
||||||
|
<h2 id="logs">6. NCCL 日志速查</h2>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>日志</th><th>含义</th><th>判断</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td><code>Bootstrap: Using eth0:...</code></td>
|
||||||
|
<td>初始化控制连接选择 eth0。</td>
|
||||||
|
<td>只说明 bootstrap，尚不能证明数据走 RDMA。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>NET/IB : Using ... mlx5_0 ...</code></td>
|
||||||
|
<td>NCCL 已识别 RDMA HCA。</td>
|
||||||
|
<td>RDMA 数据后端可用。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>via NET/IB/.../GDRDMA</code></td>
|
||||||
|
<td>跨机边通过 GPUDirect RDMA。</td>
|
||||||
|
<td>理想证据。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>NET/IB : No device found</code></td>
|
||||||
|
<td>容器没有可用 RDMA 设备或驱动/权限不完整。</td>
|
||||||
|
<td>继续看是否回退 Socket。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>NET/Socket : Using 非计算网...</code></td>
|
||||||
|
<td>跨机数据由普通 TCP Socket 传输。</td>
|
||||||
|
<td>若误入低速非计算网，性能会严重受限。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>via P2P/IPC</code></td>
|
||||||
|
<td>同机 GPU 通过 CUDA P2P/IPC。</td>
|
||||||
|
<td>机内路径，不代表跨机路径。</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2 id="incident">7. 2026-07-30 Prefill 变慢事故复盘</h2>
|
||||||
|
<h3>已观测事实</h3>
|
||||||
|
<ul>
|
||||||
|
<li>宿主机存在 <code>/dev/infiniband</code>，两条 400G Rail 均 Up。</li>
|
||||||
|
<li>原脚本容器内不存在 <code>/dev/infiniband</code>。</li>
|
||||||
|
<li>NCCL INFO 明确打印 <code>NET/IB : No device found</code> 和 <code>Using network Socket</code>。</li>
|
||||||
|
<li>原脚本选择 400G 计算网 <code>eth0</code>；quick-map 曾误选低速非计算网。</li>
|
||||||
|
<li>部署规定只有 <code>eth0/eth3</code> 用于节点间通信，两者均为 400G。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>冷缓存 Shape</th><th>原脚本网络：eth0 Socket</th><th>quick-map：错误的非计算网</th><th>差异</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td>1K → 1, C=1</td>
|
||||||
|
<td class="metric">TTFT 1.458s / 693.1 input tok/s</td>
|
||||||
|
<td class="metric">TTFT 15.88–16.04s / 约 64 tok/s</td>
|
||||||
|
<td>约 10.9×</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>32K → 1, C=1</td>
|
||||||
|
<td class="metric">TTFT 38.062s / 860.5 input tok/s</td>
|
||||||
|
<td class="metric">TTFT 504.44s / 64.96 tok/s</td>
|
||||||
|
<td>约 13.25×</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<div class="finding">
|
||||||
|
<strong>根因判断：</strong>
|
||||||
|
quick-map 没有把 RDMA 设备透传进容器，却把 <code>NCCL_SOCKET_IFNAME</code>
|
||||||
|
设成低速非计算网。NCCL 回退 NET/Socket 后，TP16 跨机数据没有进入规定的
|
||||||
|
<code>eth0/eth3</code> 计算网。<code>NCCL_CROSS_NIC=1</code> 在没有 NET/IB
|
||||||
|
的情况下不是致因。
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h3>为什么旧日志还会比 38 秒更短</h3>
|
||||||
|
<p>
|
||||||
|
旧矩阵脚本还有第二个独立因素：<code>warmup_requests=16</code>、固定
|
||||||
|
<code>seed=42</code>、ISL/OSL/C 升序运行，而且从不 flush Prefix Cache。
|
||||||
|
因此旧日志混入缓存命中，不能直接与冷 Prefill 比较。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div class="good">
|
||||||
|
quick-map 现在只允许 <code>eth0/eth3</code> 和
|
||||||
|
<code>mlx5_0/mlx5_3</code>，并会透传精确 RDMA 设备、强制检查两端
|
||||||
|
<code>NET/IB</code> 日志。代码与 dry-run 已通过；下一步是真机启动验证，
|
||||||
|
在拿到运行时证据前不进入 Kernel 归因。
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2 id="checklist">8. 从宿主机到 NCCL 的排查清单</h2>
|
||||||
|
<ol>
|
||||||
|
<li>
|
||||||
|
<strong>宿主机链路：</strong>
|
||||||
|
<code>ethtool eth0</code>、<code>ethtool eth3</code>。
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>HCA 映射：</strong>
|
||||||
|
<code>ibdev2netdev</code>，确认 <code>mlx5_0→eth0</code>、
|
||||||
|
<code>mlx5_3→eth3</code>。
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>宿主机设备：</strong>
|
||||||
|
<code>ls -l /dev/infiniband</code>。
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>容器设备：</strong>
|
||||||
|
<code>docker exec CONTAINER ls -l /dev/infiniband</code>。
|
||||||
|
宿主机有、容器没有，NCCL 仍然用不了 RDMA。
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>运行时证据：</strong>
|
||||||
|
用一次 <code>NCCL_DEBUG=INFO</code> 启动，搜索
|
||||||
|
<code>NET/IB</code>、<code>NET/Socket</code>、<code>GDRDMA</code>。
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>硬件计数器：</strong>
|
||||||
|
同时观察 eth0/eth3 流量和 RDMA 端口计数；不能只看环境变量。
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<strong>端到端 A/B：</strong>
|
||||||
|
冷缓存、同一 prompt、同一模型参数，仅改变一个网络变量。
|
||||||
|
</li>
|
||||||
|
</ol>
|
||||||
|
|
||||||
|
<h3>最小 RDMA 设备透传验证</h3>
|
||||||
|
<pre><code>docker run --rm \
|
||||||
|
--device=/dev/infiniband/rdma_cm \
|
||||||
|
--device=/dev/infiniband/uverbs0 \
|
||||||
|
--device=/dev/infiniband/uverbs3 \
|
||||||
|
IMAGE \
|
||||||
|
ls -l /dev/infiniband</code></pre>
|
||||||
|
<p>
|
||||||
|
能看到设备只是第一关。最终仍必须从 NCCL INFO 中看到 <code>NET/IB</code>，
|
||||||
|
并通过通信基准与 SGLang 结果确认。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="sources">9. 官方资料</h2>
|
||||||
|
<ul>
|
||||||
|
<li>
|
||||||
|
<a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/setup.html">
|
||||||
|
NVIDIA NCCL Setup：bootstrap 与通信安全边界
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html">
|
||||||
|
NVIDIA NCCL Environment Variables：SOCKET_IFNAME、IB_HCA、CROSS_NIC
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
<li>
|
||||||
|
<a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting/networking_troubleshooting.html">
|
||||||
|
NVIDIA NCCL Networking Troubleshooting
|
||||||
|
</a>
|
||||||
|
</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<p>
|
||||||
|
<a href="./推理优化计划.html">返回推理优化主计划</a>
|
||||||
|
</p>
|
||||||
|
</main>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<footer>
|
||||||
|
本页以两台 6000D 的真实设备映射和 2026-07-30 实测日志为例。
|
||||||
|
</footer>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
597
docs/dsv4pro_pro6000d_2node_sglang/phase1_code.html
Normal file
597
docs/dsv4pro_pro6000d_2node_sglang/phase1_code.html
Normal file
@ -0,0 +1,597 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<meta name="color-scheme" content="light">
|
||||||
|
<title>Phase 1 Code：DSV4-Pro 双机 Pro6000D SGLang 快速性能地图</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
--canvas: #eef3f4;
|
||||||
|
--paper: #ffffff;
|
||||||
|
--ink: #182126;
|
||||||
|
--muted: #5a6970;
|
||||||
|
--line: #d4dee1;
|
||||||
|
--navy: #17363d;
|
||||||
|
--teal: #087c72;
|
||||||
|
--teal-soft: #e8f5f3;
|
||||||
|
--amber: #a64c14;
|
||||||
|
--amber-soft: #fff1e7;
|
||||||
|
--code-bg: #17252b;
|
||||||
|
--code-ink: #eaf2f3;
|
||||||
|
}
|
||||||
|
|
||||||
|
* { box-sizing: border-box; letter-spacing: 0; }
|
||||||
|
html { scroll-behavior: smooth; }
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--canvas);
|
||||||
|
font-family: "PingFang SC", "Microsoft YaHei", "Noto Sans CJK SC", Arial, sans-serif;
|
||||||
|
font-size: 16px;
|
||||||
|
line-height: 1.72;
|
||||||
|
}
|
||||||
|
header {
|
||||||
|
color: #f6fbfb;
|
||||||
|
background: var(--navy);
|
||||||
|
border-bottom: 5px solid #d2692b;
|
||||||
|
}
|
||||||
|
.header-inner, main { width: min(100% - 36px, 1120px); margin: 0 auto; }
|
||||||
|
.header-inner { padding: 34px 0 30px; }
|
||||||
|
.eyebrow { margin: 0 0 6px; color: #9edbd5; font-size: 13px; font-weight: 700; }
|
||||||
|
h1 { margin: 0; font-size: clamp(28px, 4vw, 42px); line-height: 1.25; }
|
||||||
|
.meta { margin-top: 15px; color: #d6e5e7; font-size: 14px; }
|
||||||
|
main {
|
||||||
|
margin-top: 30px;
|
||||||
|
margin-bottom: 70px;
|
||||||
|
padding: 38px 48px 58px;
|
||||||
|
background: var(--paper);
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 6px;
|
||||||
|
box-shadow: 0 12px 30px rgba(27, 45, 51, 0.07);
|
||||||
|
}
|
||||||
|
h2 {
|
||||||
|
margin: 46px 0 15px;
|
||||||
|
padding-bottom: 8px;
|
||||||
|
font-size: 25px;
|
||||||
|
line-height: 1.35;
|
||||||
|
border-bottom: 2px solid #adbbc0;
|
||||||
|
}
|
||||||
|
h2:first-of-type { margin-top: 18px; }
|
||||||
|
h3 { margin: 29px 0 10px; color: #21454d; font-size: 19px; }
|
||||||
|
h4 { margin: 22px 0 8px; font-size: 16px; }
|
||||||
|
p, ul, ol { margin-top: 0; margin-bottom: 16px; }
|
||||||
|
li + li { margin-top: 5px; }
|
||||||
|
a { color: var(--teal); text-underline-offset: 3px; }
|
||||||
|
code {
|
||||||
|
padding: 2px 5px;
|
||||||
|
color: #85380d;
|
||||||
|
background: var(--amber-soft);
|
||||||
|
border-radius: 3px;
|
||||||
|
font-family: "SFMono-Regular", Consolas, monospace;
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
pre {
|
||||||
|
margin: 14px 0 22px;
|
||||||
|
padding: 16px 18px;
|
||||||
|
overflow: auto;
|
||||||
|
color: var(--code-ink);
|
||||||
|
background: var(--code-bg);
|
||||||
|
border-radius: 5px;
|
||||||
|
font: 13px/1.62 "SFMono-Regular", Consolas, monospace;
|
||||||
|
}
|
||||||
|
pre code { padding: 0; color: inherit; background: transparent; }
|
||||||
|
table { width: 100%; margin: 16px 0 26px; border-collapse: collapse; font-size: 14px; }
|
||||||
|
th, td {
|
||||||
|
padding: 9px 11px;
|
||||||
|
vertical-align: top;
|
||||||
|
text-align: left;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
th { color: #153b41; background: #eaf2f2; }
|
||||||
|
tbody tr:nth-child(even) { background: #fafcfc; }
|
||||||
|
.callout {
|
||||||
|
margin: 18px 0 26px;
|
||||||
|
padding: 14px 18px;
|
||||||
|
background: var(--teal-soft);
|
||||||
|
border-left: 4px solid var(--teal);
|
||||||
|
}
|
||||||
|
.warning {
|
||||||
|
margin: 18px 0 26px;
|
||||||
|
padding: 14px 18px;
|
||||||
|
background: var(--amber-soft);
|
||||||
|
border-left: 4px solid var(--amber);
|
||||||
|
}
|
||||||
|
.toc {
|
||||||
|
columns: 2;
|
||||||
|
column-gap: 38px;
|
||||||
|
margin: 16px 0 24px;
|
||||||
|
padding-left: 22px;
|
||||||
|
}
|
||||||
|
.toc li { break-inside: avoid; }
|
||||||
|
.path { font-family: "SFMono-Regular", Consolas, monospace; font-size: 13px; }
|
||||||
|
.nowrap { white-space: nowrap; }
|
||||||
|
footer { margin-top: 48px; padding-top: 18px; color: var(--muted); border-top: 1px solid var(--line); }
|
||||||
|
@media (max-width: 760px) {
|
||||||
|
main { padding: 28px 20px 42px; }
|
||||||
|
.toc { columns: 1; }
|
||||||
|
table { display: block; overflow-x: auto; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<div class="header-inner">
|
||||||
|
<p class="eyebrow">Standalone Code Walkthrough / Phase 1</p>
|
||||||
|
<h1>DSV4-Pro 双机 Pro6000D SGLang 快速性能地图：代码详解</h1>
|
||||||
|
<div class="meta">
|
||||||
|
行号基线：<code>ca1f2f63375c</code>
|
||||||
|
生成时间：2026-07-31 13:04:21 CST
|
||||||
|
入口：<code>run_quick_map.sh</code>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<p>
|
||||||
|
<a href="./推理优化计划.html">返回推理优化主计划</a> ·
|
||||||
|
<a href="./phase1_exp.html">打开 Phase 1 实验档案</a>
|
||||||
|
</p>
|
||||||
|
<div class="callout">
|
||||||
|
<strong>文档边界：</strong>这是一份独立代码档案，只解释 Phase 1 实现，不承担阶段结论展示。
|
||||||
|
下文的行号均绑定提交 <code>ca1f2f63375c</code>。代码变更后应先更新基线提交，再重新核对行号。
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2 id="read">1. 阅读方法</h2>
|
||||||
|
<ul class="toc">
|
||||||
|
<li><a href="#flow">总体控制流</a></li>
|
||||||
|
<li><a href="#files">文件职责</a></li>
|
||||||
|
<li><a href="#config">配置与场景</a></li>
|
||||||
|
<li><a href="#service">双机服务启动</a></li>
|
||||||
|
<li><a href="#bench">Benchmark 生成</a></li>
|
||||||
|
<li><a href="#mixed">混合 Prefill/Decode</a></li>
|
||||||
|
<li><a href="#results">指标解析与汇总</a></li>
|
||||||
|
<li><a href="#artifacts">结果目录与数据契约</a></li>
|
||||||
|
<li><a href="#index">函数行号索引</a></li>
|
||||||
|
</ul>
|
||||||
|
<p>
|
||||||
|
行号写法例如
|
||||||
|
<code>run_quick_map.sh:L212-L268</code>。它表示该提交中，从第 212 行到第 268 行的完整函数段，
|
||||||
|
不是当前编辑器自动漂移后的行号。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="flow">2. 总体控制流</h2>
|
||||||
|
<pre><code>main "$@"
|
||||||
|
└─ ACTION=all → run_all
|
||||||
|
├─ 校验场景与客户端
|
||||||
|
├─ start_service
|
||||||
|
│ ├─ Worker 节点先启动
|
||||||
|
│ ├─ Head 节点后启动
|
||||||
|
│ ├─ 等待 /health
|
||||||
|
│ └─ 从两端日志验证 NET/IB + 两条 HCA
|
||||||
|
├─ run_fixed_suite
|
||||||
|
│ └─ TSV 每一行 → run_bench_case
|
||||||
|
├─ run_mixed_suite
|
||||||
|
│ └─ control → decode background + long prefill injection
|
||||||
|
├─ stop_service
|
||||||
|
├─ summarize_results
|
||||||
|
└─ complete_manifest</code></pre>
|
||||||
|
<p>
|
||||||
|
Shell 负责生命周期、远端执行、容器和失败策略；Python 负责结果读取、指标补算、聚合与报告。
|
||||||
|
这条分工是理解代码的第一把钥匙。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="files">3. 文件职责</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>文件</th><th>行数</th><th>职责</th><th>主要输出</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td class="path">run_quick_map.sh</td><td>957</td>
|
||||||
|
<td>唯一入口，管理双机服务、固定场景、混合场景、失败恢复与清理。</td>
|
||||||
|
<td><code>run.log</code>、服务日志、每个 Case 的命令与原始结果。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td class="path">config.env</td><td>78</td>
|
||||||
|
<td>模型、节点、SGLang、NCCL/RDMA、benchmark、超时和路径配置。</td>
|
||||||
|
<td>被 Shell 直接 <code>source</code>，自身不产生输出。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td class="path">quick_map_scenarios.tsv</td><td>12</td>
|
||||||
|
<td>固定性能地图的声明式场景表，一行对应一个 Case。</td>
|
||||||
|
<td>输入给 <code>run_fixed_suite</code>。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td class="path">quick_map_results.py</td><td>716</td>
|
||||||
|
<td>校验 bench JSON、补算百分位、生成 meta/manifest、聚合重复实验。</td>
|
||||||
|
<td><code>summary.csv</code>、<code>summary.jsonl</code>、<code>aggregate.csv</code>、<code>report.md</code>。</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>3.1 文件之间如何调用</h3>
|
||||||
|
<pre><code>用户
|
||||||
|
└─ bash run_quick_map.sh all
|
||||||
|
├─ source config.env
|
||||||
|
│ ├─ 给 Shell 提供模型、节点、服务、NCCL 和 benchmark 变量
|
||||||
|
│ └─ 计算 SCENARIO_FILE / RESULT_BASE / RUNTIME_BASE
|
||||||
|
├─ 读取 quick_map_scenarios.tsv
|
||||||
|
│ └─ 每一行变成一次 run_bench_case 调用
|
||||||
|
├─ 调用 quick_map_results.py
|
||||||
|
│ ├─ validate-scenarios：启动前校验 TSV
|
||||||
|
│ ├─ write-case / mark-case-failed：维护 Case 状态
|
||||||
|
│ ├─ write-manifest / complete-manifest：维护 Run 状态
|
||||||
|
│ ├─ check-bench：验证 bench.json
|
||||||
|
│ └─ summarize：生成 CSV、JSONL 和报告
|
||||||
|
└─ tests/test_quick_map_results.py
|
||||||
|
└─ 只测试 Python 解析和聚合，不启动模型</code></pre>
|
||||||
|
<p>
|
||||||
|
<code>run_quick_map.sh:L6-L16</code> 是关系的起点：先定位自身目录，再
|
||||||
|
<code>source config.env</code>，随后把结果工具固定为同目录下的
|
||||||
|
<code>quick_map_results.py</code>。Shell 与 Python 之间不是 import 关系，
|
||||||
|
而是 Shell 通过 Python CLI 子命令交换 JSON/CSV 文件。
|
||||||
|
</p>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>上游文件</th><th>下游文件</th><th>连接点</th><th>传递内容</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td><code>config.env</code></td><td><code>run_quick_map.sh</code></td>
|
||||||
|
<td><code>run_quick_map.sh:L8</code></td><td>Shell 变量，允许调用命令中的环境变量覆盖默认值。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>quick_map_scenarios.tsv</code></td><td><code>run_fixed_suite</code></td>
|
||||||
|
<td><code>run_quick_map.sh:L695-L729</code></td><td>Case ID、ISL、OSL、C、请求数规则和 Warm-up。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>run_quick_map.sh</code></td><td><code>quick_map_results.py</code></td>
|
||||||
|
<td><code>RESULT_TOOL</code>，<code>run_quick_map.sh:L13</code></td><td>命令行参数、bench JSON、meta 和 Manifest 路径。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>quick_map_results.py</code></td><td>结果目录</td>
|
||||||
|
<td><code>quick_map_results.py:L307-L601</code></td><td>结构化 Case、Run、汇总和报告。</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>tests/test_quick_map_results.py</code></td><td><code>quick_map_results.py</code></td>
|
||||||
|
<td>Python 单元测试</td><td>用合成数据验证字段兼容、百分位和聚合。</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2 id="config">4. 配置与场景</h2>
|
||||||
|
<h3>4.1 配置分区</h3>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>代码范围</th><th>配置组</th><th>影响</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>config.env:L4-L6</code></td><td>实验与模型</td><td>实验名、模型名和两节点都能看到的模型路径。</td></tr>
|
||||||
|
<tr><td><code>config.env:L8-L18</code></td><td>节点与并行</td><td>Head/Worker 地址、TP16、EP2、双节点 rank。</td></tr>
|
||||||
|
<tr><td><code>config.env:L20-L22</code></td><td>镜像与缓存</td><td>SGLang 镜像、宿主机缓存目录和容器挂载。</td></tr>
|
||||||
|
<tr><td><code>config.env:L24-L35</code></td><td>NCCL/RDMA</td><td>限定 <code>eth0/eth3</code>、<code>mlx5_0/mlx5_3</code> 以及设备透传。</td></tr>
|
||||||
|
<tr><td><code>config.env:L37-L41</code></td><td>服务容量</td><td>显存比例、CUDA Graph Decode BS、活跃请求上限。</td></tr>
|
||||||
|
<tr><td><code>config.env:L43-L61</code></td><td>压测</td><td>随机数据生成、请求率、重复次数、混合注入和超时。</td></tr>
|
||||||
|
<tr><td><code>config.env:L67-L78</code></td><td>运行控制</td><td>相对路径、Case 过滤、Dry-run、断点续跑。</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>4.2 具体值在哪里看</h3>
|
||||||
|
<p>
|
||||||
|
配置采用 <code>VAR="${VAR:-default}"</code>。含义是：启动命令已经提供
|
||||||
|
<code>VAR</code> 时使用外部值，否则使用 <code>config.env</code> 里的默认值。
|
||||||
|
所以应区分“代码默认值”和“某次 Run 的实际值”。
|
||||||
|
</p>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>变量</th><th>当前默认值</th><th>默认值定义</th><th>传入服务</th><th>Run 后证据</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td><code>MEM_FRACTION_STATIC</code></td><td><code>0.9</code></td>
|
||||||
|
<td><code>config.env:L38</code></td><td><code>run_quick_map.sh:L258</code> → <code>--mem-fraction-static</code></td>
|
||||||
|
<td><code>server/head_server_cmd.txt</code>；<code>run_manifest.json</code> 的 <code>mem_fraction_static</code></td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>CUDA_GRAPH_MAX_BS_DECODE</code></td><td><code>64</code></td>
|
||||||
|
<td><code>config.env:L39</code></td><td><code>run_quick_map.sh:L259</code></td>
|
||||||
|
<td>服务命令；Manifest 的 <code>cuda_graph_max_bs_decode</code></td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>MAX_RUNNING_REQUESTS</code></td><td><code>256</code></td>
|
||||||
|
<td><code>config.env:L40</code></td><td><code>run_quick_map.sh:L260</code></td>
|
||||||
|
<td>服务命令；Manifest 的 <code>max_running_requests</code></td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>TP_SIZE / EP_SIZE / NNODES</code></td><td><code>16 / 2 / 2</code></td>
|
||||||
|
<td><code>config.env:L15-L17</code></td><td><code>run_quick_map.sh:L250-L253</code></td>
|
||||||
|
<td>服务命令；Manifest 的 <code>tp_size/ep_size/nnodes</code></td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>NCCL_SOCKET_IFNAME</code></td><td><code>eth0</code></td>
|
||||||
|
<td><code>config.env:L26</code></td><td><code>run_quick_map.sh:L231</code></td>
|
||||||
|
<td>服务命令；Manifest 的同名小写字段；NCCL 服务日志</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>NCCL_IB_HCA</code></td><td><code>=mlx5_0:1,mlx5_3:1</code></td>
|
||||||
|
<td><code>config.env:L27</code></td><td><code>run_quick_map.sh:L232</code></td>
|
||||||
|
<td>服务命令；Manifest；两节点 NCCL 日志</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>以 <code>MEM_FRACTION_STATIC</code> 为例，三个查看层级是：</p>
|
||||||
|
<pre><code># 1. 看仓库默认值
|
||||||
|
grep '^MEM_FRACTION_STATIC=' config.env
|
||||||
|
|
||||||
|
# 2. 看本次命令实际覆盖后的值
|
||||||
|
source ./config.env
|
||||||
|
printf '%s\n' "${MEM_FRACTION_STATIC}"
|
||||||
|
|
||||||
|
# 3. 看已经执行的 Run 最终用了什么
|
||||||
|
grep -- '--mem-fraction-static' results/<RUN_ID>/server/head_server_cmd.txt
|
||||||
|
python3 -c 'import json; print(json.load(open(
|
||||||
|
"results/<RUN_ID>/run_manifest.json"))["mem_fraction_static"])'</code></pre>
|
||||||
|
<p>
|
||||||
|
第 3 层最可信，因为 <code>start_service_node</code> 在
|
||||||
|
<code>run_quick_map.sh:L269-L291</code> 先展开命令，再写入
|
||||||
|
<code><role>_server_cmd.txt</code>；<code>write_run_manifest</code>
|
||||||
|
在 <code>L641-L676</code> 另存一份结构化配置。二者不一致时，应以实际容器命令和服务日志继续核查。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>4.3 TSV 如何变成请求</h3>
|
||||||
|
<p>
|
||||||
|
<code>quick_map_scenarios.tsv:L1</code> 定义列：
|
||||||
|
<code>case_id, stage, isl, osl, concurrency, multiplier, minimum, warmup, note</code>。
|
||||||
|
<code>run_fixed_suite</code> 在 <code>run_quick_map.sh:L695-L736</code> 中逐行读取。
|
||||||
|
</p>
|
||||||
|
<pre><code>num_prompts = concurrency × multiplier
|
||||||
|
num_prompts = max(num_prompts, minimum)</code></pre>
|
||||||
|
<p>
|
||||||
|
计算位于 <code>run_quick_map.sh:L700-L718</code>。因此场景表不直接写死总请求数，
|
||||||
|
而是让总请求数随并发扩大，同时允许 <code>minimum</code> 给低并发 Case 提供最小样本量。
|
||||||
|
<code>CASE_IDS</code> 的过滤发生在 <code>L69-L84</code> 与 <code>L703-L705</code>。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>4.4 11 个固定场景</h3>
|
||||||
|
<p>
|
||||||
|
<code>quick_map_scenarios.tsv:L2-L12</code> 覆盖冷 Prefill、并发 Prefill、短 Decode、
|
||||||
|
长 Decode 与长上下文 Decode。长 Prefill 和长 Decode Case 默认不做额外 Warm-up，
|
||||||
|
避免昂贵预热和 Prefix Cache 污染；短 Decode Case保留一次 Warm-up。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="service">5. 双机服务启动</h2>
|
||||||
|
<h3>5.1 参数校验与 RDMA 门禁</h3>
|
||||||
|
<p>
|
||||||
|
<code>validate_network_config</code> 位于 <code>run_quick_map.sh:L85-L140</code>。
|
||||||
|
它不接受任意网卡，而是把计算网约束为 <code>eth0/eth3</code>，把 RDMA HCA 约束为
|
||||||
|
<code>mlx5_0/mlx5_3</code>。开启 RDMA 时，两条 rail 和必需设备路径都必须存在。
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
<code>preflight_rdma_devices_on_node</code> 在 <code>L141-L156</code> 逐节点检查
|
||||||
|
<code>/dev/infiniband/rdma_cm</code>、<code>uverbs0</code>、<code>uverbs3</code>。
|
||||||
|
这是宿主机设备存在性检查，不能证明 NCCL 最终真的用了 IB，所以后面还有日志门禁。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.2 Docker 与 SGLang 命令展开</h3>
|
||||||
|
<p>
|
||||||
|
<code>build_server_command</code> 位于 <code>run_quick_map.sh:L212-L268</code>。
|
||||||
|
关键部分如下：
|
||||||
|
</p>
|
||||||
|
<pre><code>docker run --rm --network host --ipc host --shm-size 20g
|
||||||
|
--device /dev/infiniband/rdma_cm
|
||||||
|
--device /dev/infiniband/uverbs0
|
||||||
|
--device /dev/infiniband/uverbs3
|
||||||
|
-e NCCL_SOCKET_IFNAME=eth0,eth3
|
||||||
|
-e NCCL_IB_HCA=mlx5_0,mlx5_3
|
||||||
|
-e NCCL_CROSS_NIC=...
|
||||||
|
IMAGE python3 -m sglang.launch_server
|
||||||
|
--model-path ...
|
||||||
|
--tp-size 16 --ep-size 2 --nnodes 2 --node-rank ...
|
||||||
|
--dist-init-addr HEAD_IP:DIST_PORT
|
||||||
|
--mem-fraction-static ...
|
||||||
|
--cuda-graph-max-bs-decode ...
|
||||||
|
--max-running-requests ...</code></pre>
|
||||||
|
<ul>
|
||||||
|
<li><code>--network host</code> 让容器直接使用宿主机网络栈，避免额外端口映射。</li>
|
||||||
|
<li><code>--device</code> 把宿主机 RDMA 字符设备暴露给容器。只有环境变量而没有设备透传时，NCCL 仍可能找不到 IB。</li>
|
||||||
|
<li><code>--node-rank</code> 区分 Head 为 0、Worker 为 1；其余模型和并行参数保持一致。</li>
|
||||||
|
<li>完整展开命令会保存到结果目录，便于复现，而不是只留在终端历史中。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h3>5.3 为什么 Worker 先启动</h3>
|
||||||
|
<p>
|
||||||
|
<code>start_service</code> 位于 <code>run_quick_map.sh:L334-L388</code>。
|
||||||
|
它先调用 Worker 的 <code>start_service_node</code>，再启动 Head，随后轮询 Head 的
|
||||||
|
<code>/health</code>。这样 Worker 已经等待分布式 rendezvous，Head 启动后两端更容易同步进入初始化。
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
健康检查成功还不够。<code>verify_nccl_transport_node</code>
|
||||||
|
在 <code>L293-L325</code> 从服务日志拒绝 <code>NET/IB : No device found</code>，
|
||||||
|
并要求看到 <code>NET/IB</code> 及两条 HCA；<code>L326-L333</code> 对两节点都执行。
|
||||||
|
因而脚本采用 fail-closed：无法证明走 RDMA 就不开始 benchmark。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.4 停止与证据保存</h3>
|
||||||
|
<p>
|
||||||
|
<code>stop_service_node</code> 位于 <code>run_quick_map.sh:L389-L410</code>。
|
||||||
|
删除容器前先保存 <code>docker inspect</code> 和最终日志，再执行强制移除。
|
||||||
|
<code>cleanup</code> 在 <code>L849-L853</code> 配合 <code>trap</code>，保证异常退出也尝试清理两端服务。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="bench">6. Benchmark 请求生成与 Case 生命周期</h2>
|
||||||
|
<h3>6.1 命令生成</h3>
|
||||||
|
<p>
|
||||||
|
<code>prepare_bench_command</code> 位于 <code>run_quick_map.sh:L416-L464</code>。
|
||||||
|
它在 benchmark 客户端容器中运行 <code>python3 -m sglang.benchmark.serving</code>，
|
||||||
|
使用 <code>random</code> 数据集并显式传入 ISL、OSL、并发、请求数、请求率、Warm-up 与 Seed。
|
||||||
|
</p>
|
||||||
|
<pre><code>--dataset-name random
|
||||||
|
--random-input-len ISL
|
||||||
|
--random-output-len OSL
|
||||||
|
--num-prompts N
|
||||||
|
--max-concurrency C
|
||||||
|
--request-rate REQUEST_RATE
|
||||||
|
--warmup-requests W
|
||||||
|
--seed SEED
|
||||||
|
--output-file bench.json
|
||||||
|
--output-details</code></pre>
|
||||||
|
<div class="warning">
|
||||||
|
<strong>OSL 语义：</strong>随机 benchmark 会把目标输出长度传给服务端，并使用忽略 EOS 的生成设置，
|
||||||
|
目标是生成足量 token。是否真正达到 OSL 仍以 <code>bench.json</code> 中的成功请求数和
|
||||||
|
<code>total_output_tokens</code> 为准，不能只看命令参数。
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h3>6.2 单个 Case 的完整流程</h3>
|
||||||
|
<p><code>run_bench_case</code> 位于 <code>run_quick_map.sh:L545-L632</code>，顺序是：</p>
|
||||||
|
<ol>
|
||||||
|
<li>根据 suite、case、repetition 创建稳定结果目录。</li>
|
||||||
|
<li>若 <code>RESUME=1</code>，由 <code>case_already_completed</code> 检查 meta 和 bench 是否完整。</li>
|
||||||
|
<li>保存展开后的命令与 Case 元数据。</li>
|
||||||
|
<li>记录开始时间，使用 <code>timeout</code> 执行 benchmark。</li>
|
||||||
|
<li>调用 Python <code>check-bench</code> 校验 JSON，不把“进程退出码为 0”误当成有效结果。</li>
|
||||||
|
<li>失败时由 <code>detect_error_type</code> 区分超时、OOM、服务失活、传输错误和普通 benchmark 失败。</li>
|
||||||
|
<li>写入最终 <code>meta.json</code>，供后续汇总和 Phase 2 时间窗使用。</li>
|
||||||
|
</ol>
|
||||||
|
<p>
|
||||||
|
<code>case_already_completed</code> 在 <code>L511-L525</code> 同时要求 meta 状态为完成、
|
||||||
|
bench 文件存在且可解析。它避免只凭目录存在就跳过半成品。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="mixed">7. 混合 Prefill/Decode A/B</h2>
|
||||||
|
<h3>7.1 一次 repetition 的三个角色</h3>
|
||||||
|
<p><code>run_mixed_repetition</code> 位于 <code>run_quick_map.sh:L755-L832</code>：</p>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>角色</th><th>Shape</th><th>作用</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>control</td><td>1K → 1K，C=32</td><td>单独运行 Decode 背景，建立无注入基线。</td></tr>
|
||||||
|
<tr><td>decode_background</td><td>1K → 1K，C=32</td><td>混合组中的持续 Decode 请求流。</td></tr>
|
||||||
|
<tr><td>prefill_injection</td><td>128K → 1，C=1</td><td>在 Decode 正式测量期间注入一次长 Prefill。</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>7.2 “背景”在代码里是什么</h3>
|
||||||
|
<p>
|
||||||
|
背景不是 SGLang 特殊模式。它只是 Shell 把一个正常 benchmark 放到后台进程运行：
|
||||||
|
<code>run_quick_map.sh:L783-L792</code> 的子 Shell 加 <code>&</code>。
|
||||||
|
<code>background_pid=$!</code> 保存该进程 PID，主脚本随后还能并行发起长 Prefill。
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
<code>wait_for_bench_main</code> 位于 <code>L738-L753</code>，轮询背景日志中的
|
||||||
|
<code>Starting main benchmark run</code>。看到它以后再等待配置的注入延迟，避免把 Warm-up 阶段误当正式混合阶段。
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
注入前还会在 <code>L803-L816</code> 用 <code>kill -0</code> 检查背景进程是否仍存活。
|
||||||
|
若背景已经正常结束，Case 被重写为
|
||||||
|
<code>BACKGROUND_FINISHED_BEFORE_INJECTION</code>，防止生成一个实际上没有重叠的“混合成功”结果。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>7.3 A/B 对比来自哪里</h3>
|
||||||
|
<p>
|
||||||
|
Shell 只负责产生 control、background 和 injection 三份原始记录。
|
||||||
|
Python 在 <code>quick_map_results.py:L439-L580</code> 聚合同一 Case 的重复实验，
|
||||||
|
并在报告阶段计算 percentage change。主要观察 background 相对 control 的
|
||||||
|
Output TPS、TTFT P95、TPOT P95 与 E2E P95 变化。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="results">8. 指标解析与汇总</h2>
|
||||||
|
<h3>8.1 为什么需要 Python 补算</h3>
|
||||||
|
<p>
|
||||||
|
SGLang 版本变化可能导致字段名或原始明细形态不同。
|
||||||
|
<code>quick_map_results.py:L106-L125</code> 既支持单个 JSON 对象，也能从混合日志中寻找首个合法 JSON 行。
|
||||||
|
<code>L152-L157</code> 用多个候选字段名读取同一指标。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>8.2 延迟百分位</h3>
|
||||||
|
<p>
|
||||||
|
<code>latency_stats</code> 位于 <code>quick_map_results.py:L208-L233</code>。
|
||||||
|
优先读取 benchmark 已给出的 mean/P50/P95/P99；缺失时才从请求级数组补算：
|
||||||
|
</p>
|
||||||
|
<ul>
|
||||||
|
<li>E2E：优先 <code>request_latencies</code>，否则用 TTFT 加该请求所有 ITL。</li>
|
||||||
|
<li>TTFT：来自 <code>ttfts</code>。</li>
|
||||||
|
<li>TPOT：优先 <code>tpots</code>，否则取每请求 ITL 平均值。</li>
|
||||||
|
<li>ITL：展开所有请求的逐 token 间隔。</li>
|
||||||
|
</ul>
|
||||||
|
<p>
|
||||||
|
<code>percentile_ms</code> 在 <code>L128-L140</code> 使用线性插值，并把秒转换为毫秒。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>8.3 吞吐与完成状态</h3>
|
||||||
|
<p>
|
||||||
|
<code>compute_metrics</code> 位于 <code>quick_map_results.py:L236-L272</code>。
|
||||||
|
Total TPS 优先读取 benchmark 自带字段，缺失时才使用 Input TPS + Output TPS。
|
||||||
|
完成数优先读取 <code>completed</code> 或 <code>successful_requests</code>；
|
||||||
|
失败数缺失时才由尝试数减完成数。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>8.4 重复实验聚合</h3>
|
||||||
|
<p>
|
||||||
|
<code>aggregate_rows</code> 位于 <code>quick_map_results.py:L439-L481</code>。
|
||||||
|
它按 suite/case/role 聚合 repetition，输出均值、离散程度和成功状态。
|
||||||
|
<code>write_report</code> 在 <code>L493-L580</code> 生成面向人的 Markdown 报告，
|
||||||
|
<code>write_csv</code> 与 <code>summarize</code> 在 <code>L581-L601</code> 生成机器可读汇总。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="artifacts">9. 结果目录与数据契约</h2>
|
||||||
|
<pre><code>results/<RUN_ID>/
|
||||||
|
run.log
|
||||||
|
manifest.json
|
||||||
|
server/
|
||||||
|
head_command.txt
|
||||||
|
worker_command.txt
|
||||||
|
*.log
|
||||||
|
*.inspect.json
|
||||||
|
cases/
|
||||||
|
<case_id>/rep<N>/
|
||||||
|
bench_cmd.txt
|
||||||
|
bench.log
|
||||||
|
bench.json
|
||||||
|
meta.json
|
||||||
|
summary.csv
|
||||||
|
summary.jsonl
|
||||||
|
aggregate.csv
|
||||||
|
report.md</code></pre>
|
||||||
|
<p>
|
||||||
|
<code>meta.json</code> 的结构由 <code>quick_map_results.py:L307-L330</code> 写入，
|
||||||
|
包含 shape、并发、请求数、Warm-up、开始结束时间、退出码与错误分类。
|
||||||
|
<code>manifest.json</code> 由 <code>L333-L382</code> 维护，记录模型、镜像、并行参数、
|
||||||
|
NCCL/RDMA 参数和 Git 状态。两者共同保证结果可追溯。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="index">10. 函数行号索引</h2>
|
||||||
|
<h3>10.1 run_quick_map.sh</h3>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>行号</th><th>函数</th><th>一句话职责</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>L27-L68</td><td><code>log</code> 到 <code>case_selected</code></td><td>日志、时间、命令打印、节点执行和 Case 过滤基础函数。</td></tr>
|
||||||
|
<tr><td>L69-L140</td><td><code>validate_case_filter</code> / <code>validate_network_config</code></td><td>运行前拒绝未知 Case 和非计算网配置。</td></tr>
|
||||||
|
<tr><td>L141-L211</td><td>RDMA、健康和客户端预检</td><td>检查设备、服务、GPU 占用和 benchmark 客户端。</td></tr>
|
||||||
|
<tr><td>L212-L268</td><td><code>build_server_command</code></td><td>构造每个节点的完整 Docker + SGLang 命令。</td></tr>
|
||||||
|
<tr><td>L269-L388</td><td>启动与 NCCL 验证</td><td>启动节点、等待健康、从日志证明 NET/IB 双 HCA。</td></tr>
|
||||||
|
<tr><td>L389-L415</td><td>停止服务</td><td>保存日志与 inspect 后删除两端容器。</td></tr>
|
||||||
|
<tr><td>L416-L510</td><td>bench 命令与 meta 参数</td><td>构造请求并准备结果元数据。</td></tr>
|
||||||
|
<tr><td>L511-L632</td><td>断点续跑、错误分类、单 Case</td><td>执行并验证一个 benchmark Case。</td></tr>
|
||||||
|
<tr><td>L633-L694</td><td>失败标记、Manifest、汇总、日志</td><td>Run 级元数据和结果收口。</td></tr>
|
||||||
|
<tr><td>L695-L737</td><td><code>run_fixed_suite</code></td><td>遍历 TSV 与 repetition。</td></tr>
|
||||||
|
<tr><td>L738-L847</td><td>混合 A/B</td><td>确保 Decode 与长 Prefill 在时间上真实重叠。</td></tr>
|
||||||
|
<tr><td>L849-L932</td><td>清理、独立 suite、all</td><td>管理完整生命周期与最终状态。</td></tr>
|
||||||
|
<tr><td>L933-L957</td><td><code>main</code></td><td>分发 <code>all/start/fixed/mixed/stop</code>。</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>10.2 quick_map_results.py</h3>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>行号</th><th>函数组</th><th>职责</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>L94-L125</td><td>JSON I/O</td><td>可靠读取原始 benchmark 输出。</td></tr>
|
||||||
|
<tr><td>L128-L207</td><td>百分位与请求级 fallback</td><td>从明细恢复 E2E、TTFT、TPOT、ITL。</td></tr>
|
||||||
|
<tr><td>L208-L272</td><td><code>latency_stats</code> / <code>compute_metrics</code></td><td>统一指标字段与单位。</td></tr>
|
||||||
|
<tr><td>L275-L306</td><td><code>parse_scenarios</code></td><td>校验 TSV schema、类型和重复 Case。</td></tr>
|
||||||
|
<tr><td>L307-L404</td><td>Case、Manifest、失败状态</td><td>维护机器可读运行状态。</td></tr>
|
||||||
|
<tr><td>L405-L492</td><td>行构造与聚合</td><td>把每次 repetition 合并为 Case 统计。</td></tr>
|
||||||
|
<tr><td>L493-L601</td><td>报告与汇总</td><td>输出 Markdown、CSV、JSONL。</td></tr>
|
||||||
|
<tr><td>L602-L716</td><td>CLI</td><td>定义 Shell 调用的子命令和参数。</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<footer>
|
||||||
|
本文只描述提交 <code>ca1f2f63375c</code> 的实现。维护时应同时更新提交基线、行号索引与关键控制流，
|
||||||
|
不应只改文字结论。
|
||||||
|
</footer>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
853
docs/dsv4pro_pro6000d_2node_sglang/phase1_exp.html
Normal file
853
docs/dsv4pro_pro6000d_2node_sglang/phase1_exp.html
Normal file
@ -0,0 +1,853 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<meta name="color-scheme" content="light">
|
||||||
|
<title>Phase 1：DeepSeek-V4-Pro 双机 Pro6000D SGLang 快速性能地图</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
--canvas: #f3f6f7;
|
||||||
|
--paper: #ffffff;
|
||||||
|
--ink: #172126;
|
||||||
|
--muted: #5d6b71;
|
||||||
|
--line: #d8e0e3;
|
||||||
|
--teal: #087e75;
|
||||||
|
--teal-soft: #e8f5f3;
|
||||||
|
--orange: #b65318;
|
||||||
|
--orange-soft: #fff1e8;
|
||||||
|
--code-bg: #18262b;
|
||||||
|
--code-ink: #eaf2f3;
|
||||||
|
}
|
||||||
|
|
||||||
|
* {
|
||||||
|
box-sizing: border-box;
|
||||||
|
letter-spacing: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
html {
|
||||||
|
scroll-behavior: smooth;
|
||||||
|
}
|
||||||
|
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--canvas);
|
||||||
|
font-family: "PingFang SC", "Microsoft YaHei", "Noto Sans CJK SC",
|
||||||
|
Arial, sans-serif;
|
||||||
|
font-size: 16px;
|
||||||
|
line-height: 1.72;
|
||||||
|
}
|
||||||
|
|
||||||
|
a {
|
||||||
|
color: var(--teal);
|
||||||
|
text-underline-offset: 3px;
|
||||||
|
}
|
||||||
|
|
||||||
|
header {
|
||||||
|
color: #f7fbfb;
|
||||||
|
background: #17343a;
|
||||||
|
border-bottom: 5px solid #d76a2a;
|
||||||
|
}
|
||||||
|
|
||||||
|
.header-inner,
|
||||||
|
main {
|
||||||
|
width: min(100% - 40px, 1020px);
|
||||||
|
margin: 0 auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
.header-inner {
|
||||||
|
padding: 34px 0 30px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.eyebrow {
|
||||||
|
margin: 0 0 6px;
|
||||||
|
color: #9edbd5;
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 700;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
|
||||||
|
h1 {
|
||||||
|
max-width: 900px;
|
||||||
|
margin: 0;
|
||||||
|
font-size: clamp(28px, 4vw, 42px);
|
||||||
|
line-height: 1.25;
|
||||||
|
}
|
||||||
|
|
||||||
|
.meta {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 8px 20px;
|
||||||
|
margin-top: 16px;
|
||||||
|
color: #d4e4e6;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
main {
|
||||||
|
margin-top: 32px;
|
||||||
|
margin-bottom: 72px;
|
||||||
|
padding: 40px 50px 60px;
|
||||||
|
background: var(--paper);
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 6px;
|
||||||
|
box-shadow: 0 12px 34px rgba(28, 43, 49, 0.07);
|
||||||
|
}
|
||||||
|
|
||||||
|
.back {
|
||||||
|
display: inline-block;
|
||||||
|
margin-bottom: 20px;
|
||||||
|
font-weight: 700;
|
||||||
|
text-decoration: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.status {
|
||||||
|
margin: 0 0 28px;
|
||||||
|
padding: 14px 18px;
|
||||||
|
background: var(--teal-soft);
|
||||||
|
border-left: 4px solid var(--teal);
|
||||||
|
}
|
||||||
|
|
||||||
|
.status strong {
|
||||||
|
color: #075e58;
|
||||||
|
}
|
||||||
|
|
||||||
|
.decision {
|
||||||
|
padding: 14px 18px;
|
||||||
|
background: var(--teal-soft);
|
||||||
|
border-left: 4px solid var(--teal);
|
||||||
|
}
|
||||||
|
|
||||||
|
h2 {
|
||||||
|
margin: 48px 0 16px;
|
||||||
|
padding-bottom: 9px;
|
||||||
|
font-size: 25px;
|
||||||
|
line-height: 1.35;
|
||||||
|
border-bottom: 2px solid #afbcc1;
|
||||||
|
}
|
||||||
|
|
||||||
|
h2:first-of-type {
|
||||||
|
margin-top: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
h3 {
|
||||||
|
margin: 30px 0 10px;
|
||||||
|
color: #21434a;
|
||||||
|
font-size: 19px;
|
||||||
|
}
|
||||||
|
|
||||||
|
p,
|
||||||
|
ul,
|
||||||
|
ol {
|
||||||
|
margin-top: 0;
|
||||||
|
margin-bottom: 16px;
|
||||||
|
}
|
||||||
|
|
||||||
|
li + li {
|
||||||
|
margin-top: 5px;
|
||||||
|
}
|
||||||
|
|
||||||
|
table {
|
||||||
|
width: 100%;
|
||||||
|
margin: 18px 0 28px;
|
||||||
|
border-collapse: collapse;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
|
||||||
|
th,
|
||||||
|
td {
|
||||||
|
padding: 10px 12px;
|
||||||
|
vertical-align: top;
|
||||||
|
text-align: left;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
|
||||||
|
th {
|
||||||
|
color: #15393e;
|
||||||
|
background: #eaf2f2;
|
||||||
|
}
|
||||||
|
|
||||||
|
tbody tr:nth-child(even) {
|
||||||
|
background: #fafcfc;
|
||||||
|
}
|
||||||
|
|
||||||
|
code {
|
||||||
|
padding: 2px 5px;
|
||||||
|
color: #85380d;
|
||||||
|
background: var(--orange-soft);
|
||||||
|
border-radius: 3px;
|
||||||
|
font-family: "SFMono-Regular", Consolas, monospace;
|
||||||
|
font-size: 0.9em;
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
|
||||||
|
pre {
|
||||||
|
margin: 18px 0 24px;
|
||||||
|
padding: 16px 18px;
|
||||||
|
overflow: auto;
|
||||||
|
color: var(--code-ink);
|
||||||
|
background: var(--code-bg);
|
||||||
|
border-radius: 6px;
|
||||||
|
line-height: 1.55;
|
||||||
|
}
|
||||||
|
|
||||||
|
pre code {
|
||||||
|
padding: 0;
|
||||||
|
color: inherit;
|
||||||
|
background: transparent;
|
||||||
|
white-space: pre;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pending {
|
||||||
|
color: var(--orange);
|
||||||
|
font-weight: 700;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pass {
|
||||||
|
color: var(--teal);
|
||||||
|
font-weight: 700;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (max-width: 680px) {
|
||||||
|
.header-inner,
|
||||||
|
main {
|
||||||
|
width: min(100% - 24px, 1020px);
|
||||||
|
}
|
||||||
|
|
||||||
|
main {
|
||||||
|
padding: 28px 18px 44px;
|
||||||
|
}
|
||||||
|
|
||||||
|
table {
|
||||||
|
display: block;
|
||||||
|
overflow-x: auto;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<div class="header-inner">
|
||||||
|
<p class="eyebrow">Implementation & Result Record</p>
|
||||||
|
<h1>Phase 1：DeepSeek-V4-Pro 双机 Pro6000D SGLang 快速性能地图</h1>
|
||||||
|
<div class="meta">
|
||||||
|
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||||
|
<span>拓扑：SGLang TP16 / EP2</span>
|
||||||
|
<span>更新：2026-07-31 16:46:05 CST</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||||
|
<a class="back" href="./phase1_code.html">打开 Phase 1 代码详解</a>
|
||||||
|
|
||||||
|
<p class="status">
|
||||||
|
<strong>阶段状态：已完成。</strong>
|
||||||
|
正式 Run <code>dsv4pro-phase1-full-20260730-220916</code> 在双 Rail
|
||||||
|
<code>NET/IB + GDRDMA</code> 下完成 9 个固定点和 3 个混合 A/B 结果，
|
||||||
|
共 12/12 成功，用时 28 分 36 秒。补充 Run
|
||||||
|
<code>dsv4pro-phase1-long-decode-20260730-234236</code> 完成长输出与
|
||||||
|
长上下文 Decode 2/2。两个 Run 合计 11 个固定点和 3 个混合结果，
|
||||||
|
14/14 成功；服务、容器与 16 张 GPU 已清理。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>1. 目标与边界</h2>
|
||||||
|
<p>
|
||||||
|
用数小时以内、可重复的小矩阵替代约一天以上的全量扫描，先回答
|
||||||
|
Prefill、Decode、长上下文和混合干扰各自是否存在明显异常，再决定后续
|
||||||
|
Timeline 和 Kernel Profiling 的捕获对象。该阶段不要求为了“跑满表格”
|
||||||
|
而浪费算力；一旦出现稳定、可复现且足以改变调查方向的异常，就可以提前结束。
|
||||||
|
</p>
|
||||||
|
<ul>
|
||||||
|
<li>只测试 SGLang，不测试 vLLM。</li>
|
||||||
|
<li>使用双机 16 卡完整实例，不做 PD 分离。</li>
|
||||||
|
<li>不启用 MTP、EAGLE、DSpark 或其他投机解码。</li>
|
||||||
|
<li>本轮不启用 Profiler；正文只记录最终有效 Run，失败尝试仅在末尾总结经验。</li>
|
||||||
|
<li>不修改或调用旧的全天全量 Benchmark 脚本。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2>2. 精简实现</h2>
|
||||||
|
<p>实验代码位于：</p>
|
||||||
|
<pre><code>/data/hzy/sskj/experiments/pro6000/
|
||||||
|
dsv4pro_pro6000d_2node_sglang_tp16_quick_map/</code></pre>
|
||||||
|
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>文件</th>
|
||||||
|
<th>职责</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td><code>run_quick_map.sh</code></td>
|
||||||
|
<td>唯一 Shell 入口：双机服务启停、固定矩阵、混合 A/B、错误处理与清理</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>config.env</code></td>
|
||||||
|
<td>节点、模型、镜像、并行与容量参数</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>quick_map_scenarios.tsv</code></td>
|
||||||
|
<td>十一个固定工作负载点</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>quick_map_results.py</code></td>
|
||||||
|
<td>验证 Bench JSON，生成 CSV、JSONL 和 Markdown 汇总</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td><code>tests/test_quick_map_results.py</code></td>
|
||||||
|
<td>结果解析回归测试</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<p>单入口的操作面：</p>
|
||||||
|
<pre><code class="language-bash">bash run_quick_map.sh all
|
||||||
|
|
||||||
|
# 仅排障时使用同一个入口
|
||||||
|
bash run_quick_map.sh start
|
||||||
|
bash run_quick_map.sh fixed
|
||||||
|
bash run_quick_map.sh mixed
|
||||||
|
bash run_quick_map.sh stop</code></pre>
|
||||||
|
|
||||||
|
<h2>3. 服务配置</h2>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>配置项</th>
|
||||||
|
<th>当前值</th>
|
||||||
|
<th>说明</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr>
|
||||||
|
<td>镜像</td>
|
||||||
|
<td><code>lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45</code></td>
|
||||||
|
<td>沿用已验证可加载 DSV4-Pro 的版本</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>模型</td>
|
||||||
|
<td><code>/data/hf_models/DeepSeek-V4-Pro</code></td>
|
||||||
|
<td>两台节点均有本地权重</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>并行</td>
|
||||||
|
<td><code>TP=16, EP=2, nnodes=2</code></td>
|
||||||
|
<td>每台 8 卡，共 16 Rank</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>显存比例</td>
|
||||||
|
<td><code>0.9</code></td>
|
||||||
|
<td>保持已知基线，不在本阶段调参</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>活跃请求上限</td>
|
||||||
|
<td><code>256</code></td>
|
||||||
|
<td>覆盖本轮最大并发 64</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>CUDA Graph Decode BS</td>
|
||||||
|
<td><code>64</code></td>
|
||||||
|
<td>覆盖固定矩阵中的 Decode C64</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>NCCL Socket 接口</td>
|
||||||
|
<td><code>eth0</code></td>
|
||||||
|
<td>RDMA 失败回退时也承载跨机 Tensor，不只是 bootstrap</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>RoCE HCA</td>
|
||||||
|
<td><code>mlx5_0,mlx5_3</code></td>
|
||||||
|
<td>启动器只透传对应的 <code>uverbs0/uverbs3</code> 与 <code>rdma_cm</code></td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>传输后端门禁</td>
|
||||||
|
<td><code>REQUIRE_NCCL_IB=1</code></td>
|
||||||
|
<td>两端日志未证明 <code>NET/IB + mlx5_0 + mlx5_3</code> 时禁止开始 benchmark</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>代码分支</td>
|
||||||
|
<td><code>hzy</code></td>
|
||||||
|
<td>从该维护分支向中央仓库 <code>main</code> 提交合并请求</td>
|
||||||
|
</tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2>4. 固定快速矩阵</h2>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Case ID</th>
|
||||||
|
<th>ISL</th>
|
||||||
|
<th>OSL</th>
|
||||||
|
<th>C</th>
|
||||||
|
<th>目的</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>short_prefill_latency_1k_c1</code></td><td>1K</td><td>1</td><td>1</td><td>最小 TTFT</td></tr>
|
||||||
|
<tr><td><code>mid_prefill_latency_32k_c1</code></td><td>32K</td><td>1</td><td>1</td><td>中长 Prefill</td></tr>
|
||||||
|
<tr><td><code>long_prefill_latency_128k_c1</code></td><td>128K</td><td>1</td><td>1</td><td>长上下文 Prefill</td></tr>
|
||||||
|
<tr><td><code>mid_prefill_throughput_32k_c16</code></td><td>32K</td><td>1</td><td>16</td><td>Prefill 输入吞吐</td></tr>
|
||||||
|
<tr><td><code>decode_latency_1k_to_1k_c1</code></td><td>1K</td><td>1K</td><td>1</td><td>单请求 TPOT</td></tr>
|
||||||
|
<tr><td><code>decode_throughput_1k_to_1k_c16</code></td><td>1K</td><td>1K</td><td>16</td><td>Decode 吞吐</td></tr>
|
||||||
|
<tr><td><code>decode_throughput_1k_to_1k_c32</code></td><td>1K</td><td>1K</td><td>32</td><td>Decode 吞吐</td></tr>
|
||||||
|
<tr><td><code>decode_throughput_1k_to_1k_c64</code></td><td>1K</td><td>1K</td><td>64</td><td>Decode 高并发</td></tr>
|
||||||
|
<tr><td><code>long_output_decode_1k_to_4k_c16</code></td><td>1K</td><td>4K</td><td>16</td><td>持续 Decode 与 KV 增长</td></tr>
|
||||||
|
<tr><td><code>long_context_decode_128k_to_1k_c1</code></td><td>128K</td><td>1K</td><td>1</td><td>长上下文上的 Decode 成本</td></tr>
|
||||||
|
<tr><td><code>balanced_32k_to_1k_c8</code></td><td>32K</td><td>1K</td><td>8</td><td>综合压力</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
快速 Run 使用一次重复和一波测量请求，即 <code>num_prompts=C</code>。
|
||||||
|
32K/128K Prefill 与 128K 长上下文 Decode 不做昂贵的同形状 Warm-up；
|
||||||
|
短 Prefill、普通 Decode 与 1K → 4K 长输出 Decode 使用一个 Warm-up，
|
||||||
|
并在正式计时前清空 Prefix Cache。固定矩阵不做 SLO 截断或自适应并发搜索。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>5. SGLang Benchmark 与 Prefix Cache</h2>
|
||||||
|
<h3>5.1 <code>random</code> 如何生成 ISL</h3>
|
||||||
|
<p>
|
||||||
|
当前镜像的实现位于
|
||||||
|
<code>/sgl-workspace/sglang/python/sglang/benchmark/datasets/random.py</code>。
|
||||||
|
<code>dataset-name=random</code> 会读取 ShareGPT，打乱样本后取每条会话的首轮用户文本：
|
||||||
|
文本过长就截断，过短就重复其 token，直到达到目标 ISL。
|
||||||
|
<code>random-range-ratio=1.0</code> 使每条请求都使用精确的目标长度。
|
||||||
|
本机数据集共有 94,145 行，其中 92,886 行可用、71,904 个不同首轮文本，
|
||||||
|
因此不存在此前“两条数据只能形成两个并发请求”的问题。
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
<code>random-ids</code> 则直接构造随机整数 token id，不读取 ShareGPT。
|
||||||
|
当前源码同时警告这种方式可能触发 NaN，因此本阶段继续使用
|
||||||
|
<code>random + 大规模 ShareGPT</code>，并通过清缓存隔离不同测试点。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.2 OSL 为什么能达到指定长度</h3>
|
||||||
|
<p>
|
||||||
|
SGLang 原生请求函数位于
|
||||||
|
<code>/sgl-workspace/sglang/python/sglang/benchmark/serving.py</code>。
|
||||||
|
它将目标 OSL 写入 <code>max_new_tokens</code>，并默认设置
|
||||||
|
<code>ignore_eos=True</code>。因此模型即使提前生成 EOS，也会继续生成到指定 OSL；
|
||||||
|
只有请求失败、超时或触及上下文限制时，实际输出才可能不足。
|
||||||
|
</p>
|
||||||
|
<pre><code class="language-python">sampling_params = {
|
||||||
|
"max_new_tokens": request_func_input.output_len,
|
||||||
|
"ignore_eos": not args.disable_ignore_eos,
|
||||||
|
}</code></pre>
|
||||||
|
|
||||||
|
<h3>5.3 为什么 Warm-up 会污染 Prefix Cache</h3>
|
||||||
|
<p>
|
||||||
|
SGLang benchmark 的 Warm-up 直接复用 <code>input_requests[0]</code>，
|
||||||
|
而正式测量随后仍会遍历包含该请求的完整列表。因此，只要服务启用了 Prefix Cache，
|
||||||
|
第一条正式请求就可能命中刚刚 Warm-up 的前缀。第一次 Run 的服务日志实际出现
|
||||||
|
<code>#cached-token: 768</code>，证明该污染在当前环境真实发生。
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
修复方式是在每个隔离测试点传入 <code>--flush-cache</code>。benchmark 会先完成
|
||||||
|
Warm-up，再调用服务端 <code>/flush_cache</code>，最后才启动计时。这样保留 Kernel
|
||||||
|
和执行路径预热，同时不把 Warm-up 的 KV 前缀带入测量。混合干扰中的长 Prefill
|
||||||
|
注入不会清缓存，避免在 Decode 背景运行时改变其服务状态；背景与注入使用不同随机种子。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.4 如何单独测试 Prefix Caching</h3>
|
||||||
|
<ol>
|
||||||
|
<li>调用 <code>/flush_cache</code>，发送固定长 Prompt P，记录 Cold TTFT 和 <code>#cached-token</code>。</li>
|
||||||
|
<li>不清缓存，原样重发 P，记录 Warm TTFT；预期 cached token 明显增加、TTFT 降低。</li>
|
||||||
|
<li>再次清缓存，发送同长度但内容不同的 Prompt Q，排除长度、JIT 和偶然波动造成的假提升。</li>
|
||||||
|
</ol>
|
||||||
|
<p>
|
||||||
|
三组请求保持 OSL、采样参数和并发一致，各重复至少 3 次。Prefix Cache 是生产优化能力，
|
||||||
|
不是“坏东西”；这里只是在无缓存性能基线中隔离它，后续会把缓存命中场景作为单独 A/B。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>6. 混合干扰实现</h2>
|
||||||
|
<p>
|
||||||
|
这里的“背景”不是 SGLang 后台线程，而是<strong>先启动并持续运行的一批
|
||||||
|
Decode 基准流量</strong>。它既在实验期间占用 GPU，也是我们希望观察是否
|
||||||
|
变慢的对象。混合 A/B 的问题非常具体：同样一批 Decode 请求，在没有长
|
||||||
|
Prefill 干扰和有长 Prefill 干扰时，性能会相差多少？
|
||||||
|
</p>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>组别</th><th>运行内容</th><th>作用</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>A：Control</td><td>仅运行 64 条 <code>1K → 1K, C=32</code> Decode</td><td>建立无干扰基线</td></tr>
|
||||||
|
<tr><td>B：Treatment</td><td>运行相同 Decode，并在正式测量开始 10 秒后注入一条 <code>128K → 1</code> Prefill</td><td>测量 Prefill 对 Decode 的干扰</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<ol>
|
||||||
|
<li>先完成 A 组，仅运行 Decode，保存对照指标。</li>
|
||||||
|
<li>启动 B 组的 Decode 基准流量，并从日志确认它已进入正式测量，而不只是完成客户端初始化。</li>
|
||||||
|
<li>正式测量开始 10 秒后，并行提交一个 <code>128K → 1</code> 长 Prefill。</li>
|
||||||
|
<li>等待两类请求都结束，分别保存 Decode 流量和长 Prefill 请求的结果。</li>
|
||||||
|
<li>用 A、B 两组 Decode 的 Output TPS、TTFT P95、TPOT P95 与 E2E P95 计算变化率；长 Prefill 自身的 TTFT 单独报告。</li>
|
||||||
|
</ol>
|
||||||
|
<pre><code>A：Decode ───────────────────────────────→ 结束
|
||||||
|
|
||||||
|
B：Decode ───────────────────────────────→ 结束
|
||||||
|
正式测量 + 10 秒
|
||||||
|
└─ 128K Prefill ─→ 结束
|
||||||
|
共同占用同一服务</code></pre>
|
||||||
|
<pre><code class="language-bash">(
|
||||||
|
run_bench_case ... 1024 1024 32 64
|
||||||
|
) &
|
||||||
|
background_pid=$!
|
||||||
|
|
||||||
|
# 实际代码先从 bench.log 确认正式测量已经开始。
|
||||||
|
sleep 10
|
||||||
|
run_bench_case ... 131072 1 1 1
|
||||||
|
wait "${background_pid}"</code></pre>
|
||||||
|
<p>
|
||||||
|
<code>&</code> 让 Decode benchmark 与后续 Prefill 并行；
|
||||||
|
<code>$!</code> 取得该 Decode benchmark 的进程号；
|
||||||
|
<code>wait</code> 等待它完成。总请求数 64、并发 32，表示最多同时有
|
||||||
|
32 条请求在途，通常形成约两波请求。如果 Decode 流量在注入前已经结束，
|
||||||
|
两类请求没有发生重叠，结果会被明确改写为
|
||||||
|
<code>BACKGROUND_FINISHED_BEFORE_INJECTION</code>，避免生成虚假的“混合成功”。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>7. 结果与可追溯性</h2>
|
||||||
|
<pre><code>results/<RUN_ID>/
|
||||||
|
run_manifest.json
|
||||||
|
run.log
|
||||||
|
summary.csv
|
||||||
|
summary.jsonl
|
||||||
|
aggregate.csv
|
||||||
|
report.md
|
||||||
|
cases/<case_id>/rep1/
|
||||||
|
bench_cmd.txt
|
||||||
|
bench.jsonl
|
||||||
|
bench.log
|
||||||
|
meta.json
|
||||||
|
server/
|
||||||
|
head_server_cmd.txt
|
||||||
|
worker_server_cmd.txt
|
||||||
|
head_server.log
|
||||||
|
worker_server.log</code></pre>
|
||||||
|
<p>
|
||||||
|
汇总保留 Request/Input/Output/Total TPS，以及 E2E、TTFT、TPOT、ITL 的
|
||||||
|
Mean、P50、P95、P99。断点续跑前会重新解析原始 Bench JSON，不能只凭文件存在就跳过。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>8. 已完成验证</h2>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>检查</th><th>结果</th><th>证据</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>Shell 语法</td><td class="pass">通过</td><td><code>bash -n run_quick_map.sh</code></td></tr>
|
||||||
|
<tr><td>Python 单测</td><td class="pass">3/3 通过</td><td>场景唯一性、百分位回退、失败结果汇总</td></tr>
|
||||||
|
<tr><td>完整 Dry-run</td><td class="pass">通过</td><td>服务、十一个固定点、混合 A/B、清理均展开成功</td></tr>
|
||||||
|
<tr><td>真实旧 Bench JSON 解析</td><td class="pass">通过</td><td>成功解析 P50/P95/P99 与吞吐字段</td></tr>
|
||||||
|
<tr><td>项目精简</td><td class="pass">通过</td><td>实验目录顶层仅保留一个 Shell 入口</td></tr>
|
||||||
|
<tr><td>双 Rail 传输门禁</td><td class="pass">通过</td><td>Head 与 Worker 均识别 <code>mlx5_0/mlx5_3</code>，跨节点 Channel 使用 <code>NET/IB/*/GDRDMA</code></td></tr>
|
||||||
|
<tr><td>四点 Sanity</td><td class="pass">4/4 通过</td><td>1K/32K Prefill 与 C1/C32 Decode 均恢复到合理量级</td></tr>
|
||||||
|
<tr><td>冷 Prefix 口径</td><td class="pass">通过</td><td>正式测量请求的 Head 日志显示 <code>#cached-token: 0</code></td></tr>
|
||||||
|
<tr><td>完整真机 Run</td><td class="pass">12/12 通过</td><td>固定矩阵 9/9，混合 A/B 3/3，运行期失败 0</td></tr>
|
||||||
|
<tr><td>长 Decode 补测</td><td class="pass">2/2 通过</td><td>1K → 4K C16 与 128K → 1K C1 均生成完整目标 OSL</td></tr>
|
||||||
|
<tr><td>资源清理</td><td class="pass">通过</td><td>两节点相关容器与计算进程为 0，16 张 GPU 显存占用为 0</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2>9. 最终真机结果</h2>
|
||||||
|
<p>
|
||||||
|
本节只使用正式成功 Run。Profiler 与投机解码均关闭，每个 Case 只做一次快速测量，
|
||||||
|
所以它适合决定下一步 Profile 对象，不作为需要统计置信度的最终容量认证。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>9.1 执行摘要</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>项目</th><th>结果</th><th>证据</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>Run ID</td><td><code>dsv4pro-phase1-full-20260730-220916</code></td><td><code>COMPLETED</code></td></tr>
|
||||||
|
<tr><td>运行时间</td><td>28 分 36 秒</td><td>22:09:47 至 22:38:22 CST</td></tr>
|
||||||
|
<tr><td>网络路径</td><td>双 Rail <code>NET/IB + GDRDMA</code></td><td><code>mlx5_0</code> 与 <code>mlx5_3</code></td></tr>
|
||||||
|
<tr><td>正式 Run 完整性</td><td>12/12 成功</td><td>固定点 9/9；混合 A/B 3/3</td></tr>
|
||||||
|
<tr><td>补充 Run</td><td><code>dsv4pro-phase1-long-decode-20260730-234236</code></td><td>固定点 2/2；约 12 分钟含服务启动与清理</td></tr>
|
||||||
|
<tr><td>阶段合计</td><td>14/14 成功</td><td>固定点 11/11；混合 A/B 3/3</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>9.2 Prefill</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>场景</th><th>Input TPS</th><th>TTFT P95</th><th>观察</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>1K → 1，C=1</td><td>1,969.66 tok/s</td><td>0.502 s</td><td>短请求固定开销占比更高</td></tr>
|
||||||
|
<tr><td>32K → 1，C=1</td><td>2,652.76 tok/s</td><td>12.335 s</td><td>单请求吞吐进入稳定区间</td></tr>
|
||||||
|
<tr><td>128K → 1，C=1</td><td>2,710.16 tok/s</td><td>48.344 s</td><td>长 Prefill 代表点</td></tr>
|
||||||
|
<tr><td>32K → 1，C=16</td><td>3,112.77 tok/s</td><td>162.087 s</td><td>聚合吞吐仅比 C=1 高 17.3%，排队时延显著增加</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>9.3 Decode</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>1K → 1K</th><th>Output TPS</th><th>TTFT P95</th><th>TPOT P95</th><th>E2E P95</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>C=1</td><td>31.41 tok/s</td><td>0.363 s</td><td>31.47 ms</td><td>32.555 s</td></tr>
|
||||||
|
<tr><td>C=16</td><td>295.29 tok/s</td><td>4.950 s</td><td>50.02 ms</td><td>55.444 s</td></tr>
|
||||||
|
<tr><td>C=32</td><td>461.68 tok/s</td><td>8.022 s</td><td>63.31 ms</td><td>70.933 s</td></tr>
|
||||||
|
<tr><td>C=64</td><td>647.42 tok/s</td><td>12.716 s</td><td>93.44 ms</td><td>101.163 s</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
Decode 吞吐到 C=64 仍在上升，但增益递减且 TPOT 明显变差。综合场景
|
||||||
|
<code>32K → 1K，C=8</code> 的 Input/Output TPS 为
|
||||||
|
<code>2,038.00 / 63.69</code>，TTFT P95 为 <code>82.084 s</code>，
|
||||||
|
说明 Prefill 与 Decode 同时存在时干扰很强。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>9.4 长 Decode 补测</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>场景</th><th>Output TPS</th><th>TTFT P95</th><th>TPOT P95</th><th>E2E P95</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>1K → 4K，C=16</td><td>310.02 tok/s</td><td>6.241 s</td><td>50.33 ms</td><td>211.364 s</td></tr>
|
||||||
|
<tr><td>128K → 1K，C=1</td><td>12.43 tok/s</td><td>49.326 s</td><td>32.24 ms</td><td>82.312 s</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
<code>1K → 4K，C=16</code> 相比 <code>1K → 1K，C=16</code>，
|
||||||
|
Output TPS 增加 4.99%，TPOT P95 只增加 0.62%。较长 Decode 没有出现
|
||||||
|
稳态吞吐塌陷；吞吐略升是固定启动和 Prefill 成本被更多输出 token 摊薄。
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
<code>128K → 1K，C=1</code> 的 TTFT 只比 <code>128K → 1</code>
|
||||||
|
纯 Prefill 高 2.03%，而 TPOT P95 只比 <code>1K → 1K，C=1</code>
|
||||||
|
高 2.47%。因此这次长上下文请求的主要新增成本在 Prefill，而不是每个 Decode
|
||||||
|
token。表中的 12.43 Output TPS 是把 49 秒 Prefill 也计入总时长的端到端值，
|
||||||
|
不能把它误读为纯 Decode 速率。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>9.5 混合 Prefill/Decode A/B</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>指标</th><th>A：仅 Decode</th><th>B：注入 128K Prefill</th><th>变化</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>Output TPS</td><td>455.68 tok/s</td><td>345.95 tok/s</td><td>-24.08%</td></tr>
|
||||||
|
<tr><td>TTFT P95</td><td>9.443 s</td><td>10.194 s</td><td>+7.96%</td></tr>
|
||||||
|
<tr><td>TPOT P95</td><td>65.88 ms</td><td>109.73 ms</td><td>+66.55%</td></tr>
|
||||||
|
<tr><td>E2E P95</td><td>72.008 s</td><td>117.630 s</td><td>+63.36%</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p class="decision">
|
||||||
|
Phase 2 重放五个固定代表点：
|
||||||
|
<code>128K → 1，C=1</code>、<code>32K → 1，C=16</code>、
|
||||||
|
<code>1K → 1K，C=32</code>、<code>1K → 4K，C=16</code>、
|
||||||
|
<code>128K → 1K，C=1</code>，再执行有无 128K 注入的混合 A/B。
|
||||||
|
目标是区分计算、显存带宽、调度排队、跨机通信和节点不均衡。
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
完整产物：
|
||||||
|
<a href="./results/dsv4pro-phase1-full-20260730-220916/report.md">报告</a>、
|
||||||
|
<a href="./results/dsv4pro-phase1-full-20260730-220916/summary.csv">逐点汇总</a>、
|
||||||
|
<a href="./results/dsv4pro-phase1-full-20260730-220916/aggregate.csv">聚合表</a>、
|
||||||
|
<a href="./results/dsv4pro-phase1-full-20260730-220916/run_manifest.json">运行清单</a>；
|
||||||
|
长 Decode 补测的
|
||||||
|
<a href="./results/dsv4pro-phase1-long-decode-20260730-234236/report.md">报告</a>、
|
||||||
|
<a href="./results/dsv4pro-phase1-long-decode-20260730-234236/summary.csv">逐点汇总</a>
|
||||||
|
和
|
||||||
|
<a href="./results/dsv4pro-phase1-long-decode-20260730-234236/run_manifest.json">运行清单</a>。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>10. 经验教训</h2>
|
||||||
|
<ul>
|
||||||
|
<li>启动参数不等于实际传输路径；开始性能测试前必须由 NCCL 日志证明 <code>NET/IB</code>。</li>
|
||||||
|
<li>Warm-up、固定随机种子和跨 Case Prefix Cache 会改变 TTFT，冷缓存与热缓存必须分开报告。</li>
|
||||||
|
<li>先跑四点 Sanity 再启动完整矩阵，可以在几分钟内验证环境、口径和数量级。</li>
|
||||||
|
</ul>
|
||||||
|
<p>
|
||||||
|
历史排查细节保存在
|
||||||
|
<a href="./results/script-audit-20260730/report.md">TTFT 脚本口径审计报告</a>
|
||||||
|
与
|
||||||
|
<a href="./results/network-path-audit-20260730/report.md">TP16 网络路径审计报告</a>，
|
||||||
|
不作为本阶段最终结果。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>11. 实验复现命令</h2>
|
||||||
|
<p class="decision">
|
||||||
|
本节记录的是本阶段<strong>实际执行过</strong>的命令。长命令同时由程序原样保存到
|
||||||
|
<code>results/<RUN_ID>/server/*_server_cmd.txt</code> 和每个 Case 的
|
||||||
|
<code>bench_cmd.txt</code>；这些落盘文件是最终证据，正文中的换行仅用于阅读。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>11.1 实际执行：完整 Phase 1</h3>
|
||||||
|
<p><strong>执行位置：</strong><code>174.1.51.5</code>；脚本通过 SSH 启动 <code>.7</code> Worker。</p>
|
||||||
|
<pre><code class="language-bash">cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||||
|
|
||||||
|
tmux new-session -d -s dsv4pro-phase1-full \
|
||||||
|
"RUN_ID=dsv4pro-phase1-full-20260730-220916 bash run_quick_map.sh all \
|
||||||
|
2>&1 | tee /data/hzy/dsv4pro_phase1_full_20260730-220916.log"
|
||||||
|
|
||||||
|
tmux attach -t dsv4pro-phase1-full</code></pre>
|
||||||
|
<p>
|
||||||
|
<code>all</code> 的真实顺序是：
|
||||||
|
<code>Worker 启动 → Head 启动 → /health → NET/IB 门禁 → fixed → mixed → stop → summarize</code>。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>11.2 实际执行：Worker 服务</h3>
|
||||||
|
<details>
|
||||||
|
<summary>展开 174.1.51.7 的完整 docker run</summary>
|
||||||
|
<pre><code class="language-bash">docker run -d \
|
||||||
|
--name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker \
|
||||||
|
--gpus all \
|
||||||
|
--network host \
|
||||||
|
--ipc host \
|
||||||
|
--shm-size 20g \
|
||||||
|
--ulimit memlock=-1 \
|
||||||
|
--ulimit stack=67108864 \
|
||||||
|
-v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro \
|
||||||
|
-v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache \
|
||||||
|
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||||
|
-e PYTHONUNBUFFERED=1 \
|
||||||
|
-e HF_HUB_OFFLINE=1 \
|
||||||
|
-e TRANSFORMERS_OFFLINE=1 \
|
||||||
|
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||||
|
-e NCCL_SOCKET_IFNAME=eth0 \
|
||||||
|
-e 'NCCL_IB_HCA==mlx5_0:1,mlx5_3:1' \
|
||||||
|
-e NCCL_CROSS_NIC=1 \
|
||||||
|
-e NCCL_DEBUG=INFO \
|
||||||
|
-e SGLANG_SHARED_EXPERT_TP1=1 \
|
||||||
|
--device /dev/infiniband/rdma_cm \
|
||||||
|
--device /dev/infiniband/uverbs0 \
|
||||||
|
--device /dev/infiniband/uverbs3 \
|
||||||
|
--entrypoint python3 \
|
||||||
|
lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 \
|
||||||
|
-m sglang.launch_server \
|
||||||
|
--model-path /data/hf_models/DeepSeek-V4-Pro \
|
||||||
|
--tp-size 16 \
|
||||||
|
--ep-size 2 \
|
||||||
|
--nnodes 2 \
|
||||||
|
--node-rank 1 \
|
||||||
|
--dist-init-addr 10.101.0.11:20002 \
|
||||||
|
--trust-remote-code \
|
||||||
|
--host 0.0.0.0 \
|
||||||
|
--port 30002 \
|
||||||
|
--mem-fraction-static 0.9 \
|
||||||
|
--cuda-graph-max-bs-decode 64 \
|
||||||
|
--max-running-requests 256</code></pre>
|
||||||
|
</details>
|
||||||
|
|
||||||
|
<h3>11.3 实际执行：Head 服务</h3>
|
||||||
|
<details>
|
||||||
|
<summary>展开 174.1.51.5 的完整 docker run</summary>
|
||||||
|
<pre><code class="language-bash">docker run -d \
|
||||||
|
--name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head \
|
||||||
|
--gpus all \
|
||||||
|
--network host \
|
||||||
|
--ipc host \
|
||||||
|
--shm-size 20g \
|
||||||
|
--ulimit memlock=-1 \
|
||||||
|
--ulimit stack=67108864 \
|
||||||
|
-v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro \
|
||||||
|
-v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache \
|
||||||
|
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||||
|
-e PYTHONUNBUFFERED=1 \
|
||||||
|
-e HF_HUB_OFFLINE=1 \
|
||||||
|
-e TRANSFORMERS_OFFLINE=1 \
|
||||||
|
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
|
||||||
|
-e NCCL_SOCKET_IFNAME=eth0 \
|
||||||
|
-e 'NCCL_IB_HCA==mlx5_0:1,mlx5_3:1' \
|
||||||
|
-e NCCL_CROSS_NIC=1 \
|
||||||
|
-e NCCL_DEBUG=INFO \
|
||||||
|
-e SGLANG_SHARED_EXPERT_TP1=1 \
|
||||||
|
--device /dev/infiniband/rdma_cm \
|
||||||
|
--device /dev/infiniband/uverbs0 \
|
||||||
|
--device /dev/infiniband/uverbs3 \
|
||||||
|
--entrypoint python3 \
|
||||||
|
lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 \
|
||||||
|
-m sglang.launch_server \
|
||||||
|
--model-path /data/hf_models/DeepSeek-V4-Pro \
|
||||||
|
--tp-size 16 \
|
||||||
|
--ep-size 2 \
|
||||||
|
--nnodes 2 \
|
||||||
|
--node-rank 0 \
|
||||||
|
--dist-init-addr 10.101.0.11:20002 \
|
||||||
|
--trust-remote-code \
|
||||||
|
--host 0.0.0.0 \
|
||||||
|
--port 30002 \
|
||||||
|
--mem-fraction-static 0.9 \
|
||||||
|
--cuda-graph-max-bs-decode 64 \
|
||||||
|
--max-running-requests 256</code></pre>
|
||||||
|
</details>
|
||||||
|
<p>
|
||||||
|
<code>NCCL_IB_HCA==...</code> 的两个等号不是笔误：第一个是环境变量赋值分隔符，
|
||||||
|
第二个是 NCCL HCA 列表的“精确匹配”前缀。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>11.4 实际执行：代表 Benchmark</h3>
|
||||||
|
<p>以下是正式 Run 的 <code>128K → 1, C=1</code> 冷 Prefix 命令：</p>
|
||||||
|
<details>
|
||||||
|
<summary>展开完整 sglang.benchmark.serving 命令</summary>
|
||||||
|
<pre><code class="language-bash">timeout --signal=TERM --kill-after=30s 7200s \
|
||||||
|
docker run --rm \
|
||||||
|
--network host \
|
||||||
|
-v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro \
|
||||||
|
-v /data/yy/sskj/dataset/ShareGPT_V3_unfiltered_cleaned_split.json:/data/yy/sskj/dataset/ShareGPT_V3_unfiltered_cleaned_split.json:ro \
|
||||||
|
-v /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1:/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1 \
|
||||||
|
-e PYTHONUNBUFFERED=1 \
|
||||||
|
-e HF_HUB_OFFLINE=1 \
|
||||||
|
-e TRANSFORMERS_OFFLINE=1 \
|
||||||
|
--entrypoint python3 \
|
||||||
|
lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 \
|
||||||
|
-m sglang.benchmark.serving \
|
||||||
|
--backend sglang \
|
||||||
|
--host 10.101.0.11 \
|
||||||
|
--port 30002 \
|
||||||
|
--dataset-name random \
|
||||||
|
--dataset-path /data/yy/sskj/dataset/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||||
|
--random-input-len 131072 \
|
||||||
|
--random-output-len 1 \
|
||||||
|
--random-range-ratio 1.0 \
|
||||||
|
--num-prompts 1 \
|
||||||
|
--max-concurrency 1 \
|
||||||
|
--request-rate 10000 \
|
||||||
|
--output-file /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl \
|
||||||
|
--output-details \
|
||||||
|
--disable-tqdm \
|
||||||
|
--warmup-requests 0 \
|
||||||
|
--seed 42 \
|
||||||
|
--flush-cache</code></pre>
|
||||||
|
</details>
|
||||||
|
<p>
|
||||||
|
其余固定点使用同一命令模板，只替换 ISL、OSL、并发、请求数、Warm-up、Seed
|
||||||
|
和输出目录。每个点的最终展开命令保存在自己的 <code>bench_cmd.txt</code>。
|
||||||
|
混合 A/B 的并行启动顺序和两条请求命令见第 6 节及相应 Case 目录。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>11.5 实际执行：长 Decode 补测</h3>
|
||||||
|
<pre><code class="language-bash">cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||||
|
|
||||||
|
export CASE_IDS="long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1"
|
||||||
|
export RUN_ID="dsv4pro-phase1-long-decode-20260730-234236"
|
||||||
|
trap 'bash run_quick_map.sh stop' EXIT INT TERM
|
||||||
|
bash run_quick_map.sh start
|
||||||
|
bash run_quick_map.sh fixed</code></pre>
|
||||||
|
|
||||||
|
<h3>11.6 停止、清理与检查</h3>
|
||||||
|
<pre><code class="language-bash">cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||||
|
bash run_quick_map.sh stop
|
||||||
|
|
||||||
|
curl -fsS http://10.101.0.11:30002/health || true
|
||||||
|
docker ps --filter name=dsv4pro_pro6000d_2node_sglang_tp16_quick_map
|
||||||
|
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv</code></pre>
|
||||||
|
|
||||||
|
<p>
|
||||||
|
下一阶段：
|
||||||
|
<a class="back" href="./phase2_exp.html">打开 Phase 2 实验档案</a>
|
||||||
|
</p>
|
||||||
|
<p><a class="back" href="./推理优化计划.html">返回推理优化主计划</a></p>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
167
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_code.html
Normal file
167
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_code.html
Normal file
@ -0,0 +1,167 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>Phase 2.5 Code：DSV4-Pro 双机 SGLang RDMA 需求建模</title>
|
||||||
|
<style>
|
||||||
|
:root { --canvas:#eef3f4; --paper:#fff; --ink:#182126; --muted:#5a6970; --line:#d4dee1; --navy:#17363d; --teal:#087c72; --teal-soft:#e8f5f3; --amber:#a64c14; --amber-soft:#fff1e7; --code-bg:#17252b; --code-ink:#eaf2f3; }
|
||||||
|
* { box-sizing:border-box; letter-spacing:0; }
|
||||||
|
body { margin:0; color:var(--ink); background:var(--canvas); font-family:"PingFang SC","Microsoft YaHei",Arial,sans-serif; font-size:16px; line-height:1.72; }
|
||||||
|
header { color:#f6fbfb; background:var(--navy); border-bottom:5px solid #d2692b; }
|
||||||
|
.header-inner,main { width:min(100% - 36px,1120px); margin:0 auto; }
|
||||||
|
.header-inner { padding:34px 0 30px; }
|
||||||
|
h1 { margin:0; font-size:clamp(28px,4vw,42px); line-height:1.25; }
|
||||||
|
h2 { margin:42px 0 15px; padding-bottom:8px; font-size:25px; border-bottom:2px solid #adbbc0; }
|
||||||
|
h3 { margin:28px 0 10px; color:#21454d; font-size:19px; }
|
||||||
|
.eyebrow { margin:0 0 6px; color:#9edbd5; font-size:13px; font-weight:700; }
|
||||||
|
.meta { margin-top:15px; color:#d6e5e7; font-size:14px; }
|
||||||
|
main { margin-top:30px; margin-bottom:70px; padding:38px 48px 58px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
|
||||||
|
a { color:var(--teal); }
|
||||||
|
code { padding:2px 5px; color:#85380d; background:var(--amber-soft); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; overflow-wrap:anywhere; }
|
||||||
|
pre { margin:14px 0 22px; padding:16px 18px; overflow:auto; color:var(--code-ink); background:var(--code-bg); border-radius:5px; font:13px/1.62 "SFMono-Regular",Consolas,monospace; }
|
||||||
|
pre code { padding:0; color:inherit; background:transparent; }
|
||||||
|
table { width:100%; margin:16px 0 26px; border-collapse:collapse; font-size:14px; }
|
||||||
|
th,td { padding:9px 11px; vertical-align:top; text-align:left; border:1px solid var(--line); overflow-wrap:anywhere; }
|
||||||
|
th { color:#153b41; background:#eaf2f2; }
|
||||||
|
.callout { margin:18px 0 26px; padding:14px 18px; background:var(--teal-soft); border-left:4px solid var(--teal); }
|
||||||
|
.warning { margin:18px 0 26px; padding:14px 18px; background:var(--amber-soft); border-left:4px solid var(--amber); }
|
||||||
|
li+li { margin-top:5px; }
|
||||||
|
@media(max-width:760px){main{padding:28px 20px 42px}table{display:block;overflow-x:auto}}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<div class="header-inner">
|
||||||
|
<p class="eyebrow">STANDALONE CODE WALKTHROUGH / PHASE 2.5</p>
|
||||||
|
<h1>DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模：代码详解</h1>
|
||||||
|
<div class="meta">实现提交：<code>c5fa700c50c0</code> 正式 Run：<code>dsv4pro-phase2_5-20260801-130007</code> 唯一入口：<code>run_rdma_demand_modeling.sh all</code></div>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
<main>
|
||||||
|
<p><a href="./推理优化计划.html">返回推理优化主计划</a> · <a href="./phase2_5_exp.html">打开 Phase 2.5 实验档案</a></p>
|
||||||
|
<div class="callout"><strong>边界：</strong>Phase 2.5 不复制模型服务和采集器。它复用 Phase 1 的双机 TP16 服务/benchmark 与 Phase 2 的精确窗口、GPU/RDMA 采集能力，只新增“并发 Scout → 自动选点 → 业务 OSL Confirm → 需求拟合”这一层编排和分析。</div>
|
||||||
|
|
||||||
|
<h2>1. 文件职责与调用关系</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>文件</th><th>职责</th><th>调用关系</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>config.env</code></td><td>定义 ISL/OSL、Scout 并发、重复次数、平台阈值和 400G 目标</td><td>被唯一入口 source</td></tr>
|
||||||
|
<tr><td><code>run_rdma_demand_modeling.sh</code></td><td>生成场景、调用 Phase 2、串行运行 Scout/Confirm、清理服务</td><td>唯一人工入口</td></tr>
|
||||||
|
<tr><td><code>rdma_demand_model.py</code></td><td>对齐 benchmark/HCA 窗口，计算 bytes/token，拟合平台并生成报告</td><td>Scout 后选点；Confirm 后最终汇总</td></tr>
|
||||||
|
<tr><td><code>test_rdma_demand_model.py</code></td><td>覆盖 HCA counter 单位、平台选择、线性换算和拟合输出</td><td>本地/CI 回归测试</td></tr>
|
||||||
|
<tr><td>Phase 2 <code>run_hardware_contention_attribution.sh</code></td><td>启动服务、采集 Head/Worker、切正式测量窗</td><td>由 Phase 2.5 以环境变量调用</td></tr>
|
||||||
|
<tr><td>Phase 1 <code>run_quick_map.sh</code></td><td>双机服务启停与 SGLang benchmark</td><td>由 Phase 2 内部复用</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<pre><code>run_rdma_demand_modeling.sh all
|
||||||
|
├─ validate_config + write_manifest
|
||||||
|
├─ run_scout
|
||||||
|
│ ├─ write_scenario_file(64K→1, C=1/4/16/32/64)
|
||||||
|
│ ├─ Phase 2 all（服务 + 18 个采集器 + 5 Case）
|
||||||
|
│ └─ rdma_demand_model.py scout → recommendation.env
|
||||||
|
├─ run_confirm
|
||||||
|
│ ├─ 读取自动选择的 C=4/16/64
|
||||||
|
│ ├─ Phase 2 all（重启服务 + 18 个采集器 + 每点 2 次）
|
||||||
|
│ └─ rdma_demand_model.py final
|
||||||
|
└─ Phase 2 stop → 双节点清理</code></pre>
|
||||||
|
|
||||||
|
<h2>2. 配置层</h2>
|
||||||
|
<p><code>config.env:5-7</code> 通过相对路径找到 Phase 1/2，不依赖执行命令所在目录。<code>config.env:10-16</code> 定义 64K Scout 与 1K Confirm；<code>18-21</code> 定义 5% 平台阈值、400G 物理目标和 360G 实用目标。</p>
|
||||||
|
<pre><code>ISL=65536
|
||||||
|
SCOUT_OSL=1
|
||||||
|
CONFIRM_OSL=1024
|
||||||
|
SCOUT_CONCURRENCIES="1 4 16 32 64"
|
||||||
|
SCOUT_REPETITIONS=1
|
||||||
|
CONFIRM_REPETITIONS=2
|
||||||
|
PLATEAU_GAIN_PCT=5
|
||||||
|
TARGET_RAIL_GBPS=400
|
||||||
|
PRACTICAL_RAIL_GBPS=360</code></pre>
|
||||||
|
<p><code>SAMPLE_INTERVAL_S=1</code> 只决定 HCA/GPU 时间序列分辨率；<code>SCENARIO_TIMEOUT_S=7200</code> 是单个 benchmark 的保护上限，不是期望耗时。</p>
|
||||||
|
|
||||||
|
<h2>3. Shell 唯一入口</h2>
|
||||||
|
<h3>3.1 参数检查与场景生成</h3>
|
||||||
|
<p><code>run_rdma_demand_modeling.sh:42-71</code> fail-fast 检查依赖脚本、整数参数和并发列表。<code>73-100</code> 生成 Phase 2 能读取的 TSV，并为每个形状生成稳定的 case id。</p>
|
||||||
|
<h3>3.2 复用 Phase 2，而不是复制采集代码</h3>
|
||||||
|
<p><code>103-134</code> 构造一个数组命令，把场景、Case、重复次数和采样周期作为环境变量传给 Phase 2。它显式关闭 mixed case 与通信 microbenchmark，因为 Phase 2.5 只测模型 RDMA 需求，不重复已完成的硬件基线。</p>
|
||||||
|
<pre><code>RUN_MIXED_CASE=0
|
||||||
|
RUN_COMMUNICATION_BASELINE=0
|
||||||
|
SCENARIO_FILE=.../scout.tsv
|
||||||
|
FIXED_CASE_IDS=rdma_scout_...
|
||||||
|
bash run_hardware_contention_attribution.sh all</code></pre>
|
||||||
|
<h3>3.3 两阶段控制流</h3>
|
||||||
|
<p><code>136-152</code> 跑完 Scout 后立即调用 Python，并写出 <code>recommendation.env</code>；<code>154-180</code> 读取推荐并发，生成 64K→1K Confirm。<code>217-225</code> 的 <code>run_all</code> 严格串行执行，异常信号触发 stop 清理。</p>
|
||||||
|
<div class="warning"><strong>为什么服务会启动两次：</strong>Scout 结束后 Phase 2 会清理服务；Confirm 使用全新的 Prefix Cache、采集器和服务生命周期，避免 Scout 状态污染确认结果。</div>
|
||||||
|
|
||||||
|
<h2>4. Python 如何从计数器变成需求模型</h2>
|
||||||
|
<h3>4.1 精确时间窗与 HCA 单位</h3>
|
||||||
|
<p><code>rdma_demand_model.py:87-115</code> 以 <code>(case_id,repetition)</code> 对齐 benchmark、窗口和 RDMA 汇总。<code>117-147</code> 在正式窗口内计算相邻 HCA counter 的速率；IB <code>port_*_data</code> 单位是 4-octet，因此必须乘 4，再乘 8 转为 bit/s。</p>
|
||||||
|
<pre><code>gbps = (counter_delta × 4 bytes × 8 bits) / duration_s / 1e9</code></pre>
|
||||||
|
<h3>4.2 单 Case 指标</h3>
|
||||||
|
<p><code>171-281</code> 汇总四条观测边（Head/Worker × 两个 HCA）的 Rail Mean/P95/Max、双 Rail 单向合计、Rail 不均衡、错误计数和 GPU 利用率。通信强度按每条 Rail 平均发送字节计算：</p>
|
||||||
|
<pre><code>bytes_per_input_token_per_rail
|
||||||
|
= mean(head/worker × mlx5_0/mlx5_3 xmit_bytes)
|
||||||
|
/ total_input_tokens</code></pre>
|
||||||
|
<p>这里不把 TX+RX 相加，因为那会把同一份跨机数据重复计数。</p>
|
||||||
|
|
||||||
|
<h3>4.3 平台、拐点与自动选点</h3>
|
||||||
|
<p><code>362-392</code> 比较相邻并发点。只有 Rail Mean 与 Input TPS 增益同时低于 5%，当前点才是平台候选。随后选择平台前一点、平台点和最高稳定点；本 Run 得到 <code>4 16 64</code>。</p>
|
||||||
|
<pre><code>if bandwidth_gain < 5% and input_tps_gain < 5%:
|
||||||
|
plateau_c = current_concurrency</code></pre>
|
||||||
|
|
||||||
|
<h3>4.4 线性通信强度与饱和曲线</h3>
|
||||||
|
<p><code>352-360</code> 用过原点线性斜率拟合 <code>rail_gbps/input_tps</code>，再还原为 bytes/token。<code>318-350</code> 用双曲线 <code>B(C)=B∞×C/(K+C)</code> 拟合并发饱和曲线；<code>394-447</code> 组合两者，判断模型计算或网络谁先到平台。</p>
|
||||||
|
<pre><code>required_input_tps
|
||||||
|
= target_rail_gbps / linear_gbps_per_input_tps
|
||||||
|
|
||||||
|
if fitted_bandwidth_asymptote < 360:
|
||||||
|
verdict = COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION</code></pre>
|
||||||
|
<p>本 Run 的 RMSE 为 0.277 Gbit/s，五个 Scout 点与饱和曲线贴合良好；拟合上限 80.32 Gbit/s，与 C=16/32/64 的 79.90/79.93/79.97 一致。</p>
|
||||||
|
|
||||||
|
<h2>5. 输出文件如何阅读</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>输出</th><th>用途</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>rdma_case_metrics.csv</code></td><td>每次重复的 benchmark + GPU + Rail 对齐数据，是审计主表</td></tr>
|
||||||
|
<tr><td><code>rdma_demand_model.json</code></td><td>完整拟合参数、平台点、目标 TPS 和最终 verdict</td></tr>
|
||||||
|
<tr><td><code>rdma_demand_report.md</code></td><td>面向人的 Scout/Confirm 摘要</td></tr>
|
||||||
|
<tr><td><code>recommendation.env</code></td><td>Shell 可直接 source 的 Confirm 并发列表</td></tr>
|
||||||
|
<tr><td><code>{scout,confirm}/case_windows.csv</code></td><td>每个正式 benchmark 的精确起止时间</td></tr>
|
||||||
|
<tr><td><code>{scout,confirm}/{head,worker}/rdma.csv</code></td><td>原始 HCA counter 时间序列，仅保留在服务器完整结果中</td></tr>
|
||||||
|
<tr><td><code>commands/*.cmd.txt</code></td><td>实际传给 Phase 2 的完整可复现命令</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2>6. 测试与验收门槛</h2>
|
||||||
|
<ul>
|
||||||
|
<li><code>python3 -m unittest test_rdma_demand_model.py</code>：4/4 通过。</li>
|
||||||
|
<li><code>bash -n run_rdma_demand_modeling.sh</code> 与 Python compile：通过。</li>
|
||||||
|
<li><code>DRY_RUN=1 ... all</code>：展开 5 个 Scout 和 3×2 个 Confirm，不启动服务。</li>
|
||||||
|
<li>正式 Run：Scout 5/5、Confirm 6/6；两个阶段各 18/18 个采集器正常启停，共保存 72 条 STARTED/STOPPED 生命周期事件。</li>
|
||||||
|
<li>11 个测量结果全部 <code>COMPLETED</code>，<code>rdma_error_delta=0</code>。</li>
|
||||||
|
<li>结束后 Head/Worker 均无实验容器，16 张 GPU 为 0 MiB / 0%。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2>7. 行号索引</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>功能</th><th>文件与行</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>配置与路径</td><td><code>config.env:3-36</code></td></tr>
|
||||||
|
<tr><td>校验与场景生成</td><td><code>run_rdma_demand_modeling.sh:42-100</code></td></tr>
|
||||||
|
<tr><td>Phase 2 调用</td><td><code>run_rdma_demand_modeling.sh:103-134</code></td></tr>
|
||||||
|
<tr><td>Scout / Confirm</td><td><code>run_rdma_demand_modeling.sh:136-180</code></td></tr>
|
||||||
|
<tr><td>唯一 all 与清理</td><td><code>run_rdma_demand_modeling.sh:206-264</code></td></tr>
|
||||||
|
<tr><td>HCA interval rate</td><td><code>rdma_demand_model.py:117-147</code></td></tr>
|
||||||
|
<tr><td>Case 对齐汇总</td><td><code>rdma_demand_model.py:171-281</code></td></tr>
|
||||||
|
<tr><td>饱和拟合</td><td><code>rdma_demand_model.py:318-350</code></td></tr>
|
||||||
|
<tr><td>平台选点</td><td><code>rdma_demand_model.py:362-392</code></td></tr>
|
||||||
|
<tr><td>需求模型与 verdict</td><td><code>rdma_demand_model.py:394-447</code></td></tr>
|
||||||
|
<tr><td>报告输出</td><td><code>rdma_demand_model.py:449-517</code></td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2>8. 复用时必须重新标定的边界</h2>
|
||||||
|
<p>这套代码可复用，但 <code>3.332 MB/token/rail</code> 不是通用常数。换模型、量化、TP/EP、节点切分、backend、Prefill/Decode 形状或 Prefix Cache 策略后，都必须重新跑 Scout。代码输出的是“当前部署实现的经验模型”，不是由参数量单独推导出的理论通信量。</p>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
166
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_exp.html
Normal file
166
docs/dsv4pro_pro6000d_2node_sglang/phase2_5_exp.html
Normal file
@ -0,0 +1,166 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>Phase 2.5：DSV4-Pro 双机 Pro6000D SGLang RDMA 需求建模</title>
|
||||||
|
<style>
|
||||||
|
:root { --ink:#18202a; --muted:#5b6570; --line:#d8dde3; --paper:#fff; --page:#f3f5f7; --blue:#1769aa; --green:#16734a; --amber:#9a5a00; --code:#f0f3f6; }
|
||||||
|
* { box-sizing:border-box; letter-spacing:0; }
|
||||||
|
body { margin:0; background:var(--page); color:var(--ink); font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","PingFang SC",sans-serif; font-size:16px; line-height:1.68; }
|
||||||
|
header { background:#202a34; color:#fff; border-bottom:5px solid #34a17a; }
|
||||||
|
.header-inner, main { width:min(1120px,calc(100% - 32px)); margin:0 auto; }
|
||||||
|
.header-inner { padding:36px 0 30px; }
|
||||||
|
h1 { margin:4px 0 12px; font-size:clamp(27px,4vw,42px); line-height:1.2; }
|
||||||
|
h2 { margin:38px 0 12px; padding-bottom:7px; border-bottom:2px solid var(--line); font-size:24px; }
|
||||||
|
h3 { margin:26px 0 8px; font-size:19px; }
|
||||||
|
.eyebrow { margin:0; color:#8fd8bd; font-size:13px; font-weight:700; }
|
||||||
|
.meta { display:flex; flex-wrap:wrap; gap:8px 22px; color:#d7dee5; font-size:14px; }
|
||||||
|
main { margin-top:24px; margin-bottom:48px; padding:30px 38px 42px; background:var(--paper); border:1px solid var(--line); border-radius:6px; }
|
||||||
|
a { color:var(--blue); }
|
||||||
|
.back { display:inline-block; margin:0 16px 10px 0; font-weight:650; }
|
||||||
|
code { padding:1px 5px; background:var(--code); border-radius:3px; font-family:"SFMono-Regular",Consolas,monospace; font-size:.92em; overflow-wrap:anywhere; }
|
||||||
|
pre { overflow-x:auto; padding:14px 16px; background:#202a34; color:#f4f7fa; border-radius:5px; line-height:1.5; }
|
||||||
|
pre code { padding:0; background:transparent; color:inherit; }
|
||||||
|
table { width:100%; margin:14px 0 22px; border-collapse:collapse; font-size:14px; }
|
||||||
|
th,td { padding:10px 11px; border:1px solid var(--line); text-align:left; vertical-align:top; }
|
||||||
|
th { background:#edf1f4; }
|
||||||
|
.status,.decision,.warning { padding:14px 16px; }
|
||||||
|
.status { border-left:4px solid var(--green); background:#eef8f3; }
|
||||||
|
.decision { border-left:4px solid var(--blue); background:#edf5fb; }
|
||||||
|
.warning { border-left:4px solid var(--amber); background:#fff7e7; }
|
||||||
|
.metric { font-size:20px; font-weight:750; color:var(--green); }
|
||||||
|
li+li { margin-top:5px; }
|
||||||
|
@media (max-width:720px) { main{padding:22px 18px 30px} table{display:block;overflow-x:auto;white-space:nowrap} }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<div class="header-inner">
|
||||||
|
<p class="eyebrow">FINAL EXPERIMENT RECORD / PHASE 2.5</p>
|
||||||
|
<h1>DeepSeek-V4-Pro 双机 Pro6000D SGLang：RDMA 需求建模与并发拐点</h1>
|
||||||
|
<div class="meta">
|
||||||
|
<span>Run：dsv4pro-phase2_5-20260801-130007</span>
|
||||||
|
<span>拓扑：TP16 / EP2 / 双 Rail RoCE</span>
|
||||||
|
<span>完成：2026-08-01 15:14:11 CST</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
<main>
|
||||||
|
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||||
|
<a class="back" href="./phase2_5_code.html">打开 Phase 2.5 代码详解</a>
|
||||||
|
|
||||||
|
<p class="status"><strong>阶段已完成。</strong>正式 Run 用时 2 小时 14 分 04 秒；Scout 5/5、Confirm 6/6 成功，两阶段各 18/18 个采集器正常启停。所有测量窗 RDMA 错误增量为 0，结束后两节点容器和 16 张 GPU 均已清理。</p>
|
||||||
|
|
||||||
|
<h2>1. 要回答的问题</h2>
|
||||||
|
<p>Phase 2 只看到代表负载约 83.5 Gbit/s/rail，不能判断继续增加并发是否会逼近 400G。Phase 2.5 专门回答三个问题：</p>
|
||||||
|
<ol>
|
||||||
|
<li>固定模型、TP/EP 和输入形状后，Input TPS 与每 Rail RDMA 带宽是什么关系？</li>
|
||||||
|
<li>并发增加到哪里后，模型吞吐和 RDMA 带宽不再增长？</li>
|
||||||
|
<li>要达到 400G，需要怎样的 Input TPS；当前瓶颈先出现在模型计算还是网络？</li>
|
||||||
|
</ol>
|
||||||
|
|
||||||
|
<h2>2. 实验设计</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>阶段</th><th>请求形状</th><th>并发</th><th>重复</th><th>目的</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>Scout</td><td>64K → 1</td><td>1 / 4 / 16 / 32 / 64</td><td>1</td><td>隔离 Prefill，找吞吐与带宽平台</td></tr>
|
||||||
|
<tr><td>Confirm</td><td>64K → 1K</td><td>自动选择 4 / 16 / 64</td><td>2</td><td>验证真实长输出不会推翻需求模型</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>服务参数沿用 Phase 1/2：SGLang nightly、TP16、EP2、双 Rail <code>mlx5_0/mlx5_3</code>、<code>NET/IB + GDRDMA</code>。每个 Case 使用冷 Prefix，并按 benchmark 正式测量窗口切片 HCA Counter。</p>
|
||||||
|
|
||||||
|
<h2>3. 实际启动命令</h2>
|
||||||
|
<p>只在 Head <code>174.1.51.5</code> 执行，不需要 <code>source</code> 或 <code>conda activate</code>：</p>
|
||||||
|
<pre><code>cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||||
|
|
||||||
|
RUN_ID=dsv4pro-phase2_5-20260801-130007
|
||||||
|
tmux new-session -d -s dsv4pro-phase2_5 \
|
||||||
|
"RUN_ID=${RUN_ID} bash run_rdma_demand_modeling.sh all \
|
||||||
|
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||||
|
|
||||||
|
tmux attach -t dsv4pro-phase2_5</code></pre>
|
||||||
|
<p>实际展开后的 Scout/Confirm 命令分别保存在结果目录的 <code>commands/scout.cmd.txt</code> 与 <code>commands/confirm.cmd.txt</code>。</p>
|
||||||
|
|
||||||
|
<h2>4. Scout 结果：并发 16 已进入平台</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>C</th><th>Input TPS</th><th>Rail Mean</th><th>Rail P95</th><th>Rail Max</th><th>双 Rail 单向合计</th><th>MB/input-token/rail</th><th>GPU Util</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>1</td><td>2,709.64</td><td>70.97 Gbit/s</td><td>85.60</td><td>86.81</td><td>141.94</td><td>3.138</td><td>93.75%</td></tr>
|
||||||
|
<tr><td>4</td><td>2,930.49</td><td>78.27 Gbit/s</td><td>86.82</td><td>88.86</td><td>156.55</td><td>3.298</td><td>97.45%</td></tr>
|
||||||
|
<tr><td>16</td><td>2,983.77</td><td>79.90 Gbit/s</td><td>86.42</td><td>88.20</td><td>159.79</td><td>3.336</td><td>99.33%</td></tr>
|
||||||
|
<tr><td>32</td><td>2,983.92</td><td>79.93 Gbit/s</td><td>86.23</td><td>88.22</td><td>159.86</td><td>3.344</td><td>99.49%</td></tr>
|
||||||
|
<tr><td>64</td><td>2,991.28</td><td>79.97 Gbit/s</td><td>85.93</td><td>88.32</td><td>159.95</td><td>3.340</td><td>99.64%</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p class="decision"><strong>观察结论：</strong>C=16→32 的 Input TPS 只增长 0.005%，Rail Mean 只增长 0.040%；C=32→64 也仅增长 0.247% / 0.057%。并发 16 已是平台拐点，继续加到 64 只会增加排队和 TTFT，不会增加网络压力。</p>
|
||||||
|
<p><strong>证据：</strong><code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv</code>；原始 HCA 数据位于同一 Run 的 <code>scout/head/rdma.csv</code> 与 <code>scout/worker/rdma.csv</code>，精确窗口位于 <code>scout/case_windows.csv</code>。</p>
|
||||||
|
|
||||||
|
<h2>5. Confirm 结果：加入 1K Decode 后仍由计算先饱和</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>C</th><th>重复</th><th>Input TPS</th><th>Output TPS</th><th>Rail Mean</th><th>Rail P95</th><th>双 Rail单向合计</th><th>TTFT P95</th><th>TPOT P95</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>4</td><td>2</td><td>2,072.24</td><td>32.38</td><td>57.33 Gbit/s</td><td>86.28</td><td>114.66</td><td>86.36 s</td><td>95.63 ms</td></tr>
|
||||||
|
<tr><td>16</td><td>2</td><td>2,582.13</td><td>40.35</td><td>71.64 Gbit/s</td><td>86.30</td><td>143.29</td><td>336.85 s</td><td>356.24 ms</td></tr>
|
||||||
|
<tr><td>64</td><td>2</td><td>2,588.96</td><td>40.45</td><td>72.19 Gbit/s</td><td>86.69</td><td>144.38</td><td>1,506.50 s</td><td>418.58 ms</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>C=16→64 的 Input TPS 仅增长 0.26%，Rail Mean 仅增长 0.76%，但 TTFT P95 从 336.85 秒升至 1,506.50 秒。对于 64K→1K，<strong>最大有意义并发仍约为 16</strong>；C=64 是容量压力点，不是推荐服务点。</p>
|
||||||
|
<p><strong>证据：</strong>同一 Run 的 <code>confirm/bench_summary.csv</code>、<code>confirm/case_rdma_summary.csv</code>、<code>confirm/case_windows.csv</code>；两轮逐点数据在顶层 <code>rdma_case_metrics.csv</code>。</p>
|
||||||
|
|
||||||
|
<h2>6. 400G 能否被模型负载打满</h2>
|
||||||
|
<p>Scout 的线性比例为：</p>
|
||||||
|
<pre><code>每 Rail 带宽（Gbit/s）
|
||||||
|
≈ Input TPS × 3.332 MB/input-token/rail × 8 ÷ 1e9</code></pre>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>目标口径</th><th>需要的 Input TPS</th><th>当前约 2,991 TPS 的差距</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>单 Rail 400G</td><td class="metric">15,006 tok/s</td><td>约 5.02×</td></tr>
|
||||||
|
<tr><td>单 Rail 360G（90% 实用线）</td><td>13,505 tok/s</td><td>约 4.51×</td></tr>
|
||||||
|
<tr><td>双 Rail 单向合计 400G</td><td>7,503 tok/s</td><td>约 2.51×</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>拟合得到当前模型负载的单 Rail 渐近上限约 <strong>80.32 Gbit/s</strong>，即物理 400G 的约 20.1%。瞬时 Max 也只有 89.53 Gbit/s。结论不是“网络只能跑 80G”，而是<strong>当前 DSV4-Pro TP16/EP2 实现最多只能产生约 80G/rail 的持续 RDMA 流量</strong>；Phase 2 的 NCCL microbenchmark 已证明链路本身能达到更高通信带宽。</p>
|
||||||
|
<p class="warning"><strong>口径提醒：</strong>400G 是每条 Rail 的线速；双 Rail 单向总量是两条 Rail 的 TX 之和。不要把 TX 与 RX 相加后声称打满，也不要把 NCCL <code>busbw GB/s</code> 与 HCA <code>Gbit/s</code> 直接比较。</p>
|
||||||
|
|
||||||
|
<h2>7. 一套可复用的 RDMA 需求评估方法</h2>
|
||||||
|
<ol>
|
||||||
|
<li><strong>固定部署变量。</strong>记录模型版本、精度/量化、TP/EP/PP/DP、节点数、Attention/MoE backend、chunked prefill 和网卡拓扑。任一项变化都要重新标定。</li>
|
||||||
|
<li><strong>先选 Prefill Scout。</strong>固定 ISL，OSL=1，取稀疏并发点如 1/4/16/32/64；每点清 Prefix Cache，并保证请求文本实际达到目标 token 数。</li>
|
||||||
|
<li><strong>对齐正式测量窗。</strong>从 benchmark 的 main-run 起止时间切片 Head/Worker 的 <code>mlx5_*</code> HCA Counter，不能用整个进程寿命，也不能只看 <code>sar eth*</code>。</li>
|
||||||
|
<li><strong>计算通信强度。</strong><code>bytes_per_input_token_per_rail = rail_xmit_bytes / total_input_tokens</code>。这是该模型与并行策略下“每处理一个输入 token，要在一条 Rail 发送多少字节”。</li>
|
||||||
|
<li><strong>找并发平台。</strong>同时观察 Input TPS 和 Rail Mean；连续一点的增益都低于阈值（本实验 5%）时，记为拐点。最大 C 不等于最大有效 C。</li>
|
||||||
|
<li><strong>推导目标吞吐。</strong><code>required_input_tps = target_rail_gbps × 1e9 / (bytes_per_token × 8)</code>。若模型的实测/拟合 TPS 上限远低于该值，网络不会先饱和。</li>
|
||||||
|
<li><strong>用业务 OSL 复测。</strong>在平台前、拐点、最高压力点各重复至少两次，确认 Decode、KV Cache 和调度没有改变结论。</li>
|
||||||
|
<li><strong>最后做链路对照。</strong>模型负载未打满时，用 NCCL microbenchmark 验证网络能力，把“模型产流量不足”与“网络本身跑不满”分开。</li>
|
||||||
|
</ol>
|
||||||
|
|
||||||
|
<h3>7.1 哪些变量会改变 bytes/token 与平台</h3>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>变量</th><th>可能改变的原因</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>模型架构与层数</td><td>每 token 触发的 TP collective、MoE dispatch/combine 和激活尺寸不同</td></tr>
|
||||||
|
<tr><td>TP / EP / PP / DP</td><td>通信参与 rank、跨机边界、collective 类型和频率改变</td></tr>
|
||||||
|
<tr><td>Prefill / Decode、ISL / OSL</td><td>计算强度、chunk 调度、KV 访问和 collective 消息粒度不同</td></tr>
|
||||||
|
<tr><td>并发与 batch</td><td>决定 kernel/batch 效率和 Input TPS；超过平台后只增加排队</td></tr>
|
||||||
|
<tr><td>量化与 backend</td><td>改变计算速度；通信字节可能不同比例变化，因此会移动“计算先饱和还是网络先饱和”的边界</td></tr>
|
||||||
|
<tr><td>Prefix Cache</td><td>命中会绕过大量 Prefill，必须单独作为另一类业务场景建模</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2>8. 最终结论</h2>
|
||||||
|
<p class="decision"><strong>在两台 Pro6000D、DSV4-Pro、SGLang TP16/EP2 的当前实现中，RDMA 不是吞吐瓶颈。</strong>64K Prefill 在 C=16 已达到约 3K input tok/s 和 80 Gbit/s/rail 的平台；继续增加并发到 64 不会显著增加吞吐或带宽，只会令 TTFT 急剧上升。要打满单 Rail 400G，模型侧 Input TPS 需提高到约 15K，约为当前上限 5 倍。因此后续优化应先看 GPU Kernel、MoE/Attention 执行和 rank 同步，而不是扩容计算网。</p>
|
||||||
|
|
||||||
|
<h2>9. 证据与清理</h2>
|
||||||
|
<ul>
|
||||||
|
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_report.md">自动 RDMA 需求报告</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_demand_model.json">机器可读模型</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/rdma_case_metrics.csv">全部逐点指标</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/run_manifest.txt">Run Manifest</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/head_server_cmd.txt">Scout Head 服务命令</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/scout/service/worker_server_cmd.txt">Worker 服务命令</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/head_nccl_transport.log">Confirm Head NCCL 路径</a> / <a href="./results/dsv4pro-phase2_5-20260801-130007/confirm/service/worker_nccl_transport.log">Worker NCCL 路径</a></li>
|
||||||
|
</ul>
|
||||||
|
<p>Worker 在最后一个 Case 完成后随 Head 主动关闭进程组出现 Gloo peer-close Traceback；它发生在测量结束与结果落盘之后，不是实验失败。最终 tmux、服务容器和 GPU 进程均已退出。</p>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
435
docs/dsv4pro_pro6000d_2node_sglang/phase2_code.html
Normal file
435
docs/dsv4pro_pro6000d_2node_sglang/phase2_code.html
Normal file
@ -0,0 +1,435 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<meta name="color-scheme" content="light">
|
||||||
|
<title>Phase 2 Code：DSV4-Pro 双机 Pro6000D SGLang 硬件竞争归因</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
--canvas: #eef3f4;
|
||||||
|
--paper: #ffffff;
|
||||||
|
--ink: #182126;
|
||||||
|
--muted: #5a6970;
|
||||||
|
--line: #d4dee1;
|
||||||
|
--navy: #17363d;
|
||||||
|
--teal: #087c72;
|
||||||
|
--teal-soft: #e8f5f3;
|
||||||
|
--amber: #a64c14;
|
||||||
|
--amber-soft: #fff1e7;
|
||||||
|
--code-bg: #17252b;
|
||||||
|
--code-ink: #eaf2f3;
|
||||||
|
}
|
||||||
|
* { box-sizing: border-box; letter-spacing: 0; }
|
||||||
|
html { scroll-behavior: smooth; }
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
color: var(--ink);
|
||||||
|
background: var(--canvas);
|
||||||
|
font-family: "PingFang SC", "Microsoft YaHei", "Noto Sans CJK SC", Arial, sans-serif;
|
||||||
|
font-size: 16px;
|
||||||
|
line-height: 1.72;
|
||||||
|
}
|
||||||
|
header { color: #f6fbfb; background: var(--navy); border-bottom: 5px solid #d2692b; }
|
||||||
|
.header-inner, main { width: min(100% - 36px, 1120px); margin: 0 auto; }
|
||||||
|
.header-inner { padding: 34px 0 30px; }
|
||||||
|
.eyebrow { margin: 0 0 6px; color: #9edbd5; font-size: 13px; font-weight: 700; }
|
||||||
|
h1 { margin: 0; font-size: clamp(28px, 4vw, 42px); line-height: 1.25; }
|
||||||
|
.meta { margin-top: 15px; color: #d6e5e7; font-size: 14px; }
|
||||||
|
main {
|
||||||
|
margin-top: 30px;
|
||||||
|
margin-bottom: 70px;
|
||||||
|
padding: 38px 48px 58px;
|
||||||
|
background: var(--paper);
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 6px;
|
||||||
|
box-shadow: 0 12px 30px rgba(27, 45, 51, 0.07);
|
||||||
|
}
|
||||||
|
h2 {
|
||||||
|
margin: 46px 0 15px;
|
||||||
|
padding-bottom: 8px;
|
||||||
|
font-size: 25px;
|
||||||
|
line-height: 1.35;
|
||||||
|
border-bottom: 2px solid #adbbc0;
|
||||||
|
}
|
||||||
|
h2:first-of-type { margin-top: 18px; }
|
||||||
|
h3 { margin: 29px 0 10px; color: #21454d; font-size: 19px; }
|
||||||
|
h4 { margin: 22px 0 8px; font-size: 16px; }
|
||||||
|
p, ul, ol { margin-top: 0; margin-bottom: 16px; }
|
||||||
|
li + li { margin-top: 5px; }
|
||||||
|
a { color: var(--teal); text-underline-offset: 3px; }
|
||||||
|
code {
|
||||||
|
padding: 2px 5px;
|
||||||
|
color: #85380d;
|
||||||
|
background: var(--amber-soft);
|
||||||
|
border-radius: 3px;
|
||||||
|
font-family: "SFMono-Regular", Consolas, monospace;
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
pre {
|
||||||
|
margin: 14px 0 22px;
|
||||||
|
padding: 16px 18px;
|
||||||
|
overflow: auto;
|
||||||
|
color: var(--code-ink);
|
||||||
|
background: var(--code-bg);
|
||||||
|
border-radius: 5px;
|
||||||
|
font: 13px/1.62 "SFMono-Regular", Consolas, monospace;
|
||||||
|
}
|
||||||
|
pre code { padding: 0; color: inherit; background: transparent; }
|
||||||
|
table { width: 100%; margin: 16px 0 26px; border-collapse: collapse; font-size: 14px; }
|
||||||
|
th, td {
|
||||||
|
padding: 9px 11px;
|
||||||
|
vertical-align: top;
|
||||||
|
text-align: left;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
th { color: #153b41; background: #eaf2f2; }
|
||||||
|
tbody tr:nth-child(even) { background: #fafcfc; }
|
||||||
|
.callout { margin: 18px 0 26px; padding: 14px 18px; background: var(--teal-soft); border-left: 4px solid var(--teal); }
|
||||||
|
.warning { margin: 18px 0 26px; padding: 14px 18px; background: var(--amber-soft); border-left: 4px solid var(--amber); }
|
||||||
|
.toc { columns: 2; column-gap: 38px; margin: 16px 0 24px; padding-left: 22px; }
|
||||||
|
.toc li { break-inside: avoid; }
|
||||||
|
.path { font-family: "SFMono-Regular", Consolas, monospace; font-size: 13px; }
|
||||||
|
footer { margin-top: 48px; padding-top: 18px; color: var(--muted); border-top: 1px solid var(--line); }
|
||||||
|
@media (max-width: 760px) {
|
||||||
|
main { padding: 28px 20px 42px; }
|
||||||
|
.toc { columns: 1; }
|
||||||
|
table { display: block; overflow-x: auto; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<div class="header-inner">
|
||||||
|
<p class="eyebrow">Standalone Code Walkthrough / Phase 2</p>
|
||||||
|
<h1>DSV4-Pro 双机 Pro6000D SGLang 硬件竞争归因：代码详解</h1>
|
||||||
|
<div class="meta">
|
||||||
|
行号基线：<code>39fc2ba565a3</code>
|
||||||
|
完成时间：2026-07-31 17:22:20 CST
|
||||||
|
唯一入口：<code>run_hardware_contention_attribution.sh all</code>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<p>
|
||||||
|
<a href="./推理优化计划.html">返回推理优化主计划</a> ·
|
||||||
|
<a href="./phase2_exp.html">打开 Phase 2 实验档案</a>
|
||||||
|
</p>
|
||||||
|
<div class="callout">
|
||||||
|
<strong>文档边界：</strong>本文只解释提交 <code>39fc2ba565a3</code> 的 Phase 2
|
||||||
|
代码和文件调用关系。Phase 1 负责模型服务与请求；Phase 2 负责通信基线、
|
||||||
|
两节点监控、精确时间切片和逐指标报告。最终正式 Run
|
||||||
|
<code>dsv4pro-phase2-20260731-163620</code> 使用提交
|
||||||
|
<code>5f24b7d22f98108f6cc234edba6768d55ea0a962</code>，代码树包含本页所述修复。
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<h2 id="read">1. 阅读导航</h2>
|
||||||
|
<ul class="toc">
|
||||||
|
<li><a href="#flow">总体控制流</a></li>
|
||||||
|
<li><a href="#files">文件职责与调用关系</a></li>
|
||||||
|
<li><a href="#config">配置来源</a></li>
|
||||||
|
<li><a href="#communication">通信微基准</a></li>
|
||||||
|
<li><a href="#collectors">采集器实现</a></li>
|
||||||
|
<li><a href="#alignment">精确测量窗口</a></li>
|
||||||
|
<li><a href="#report">逐指标报告</a></li>
|
||||||
|
<li><a href="#index">函数行号索引</a></li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2 id="flow">2. 总体控制流</h2>
|
||||||
|
<pre><code>main "$@" → run_all
|
||||||
|
├─ validate_config
|
||||||
|
├─ preflight_node_tools
|
||||||
|
│ └─ 两节点 dcgmi discovery -l 必须成功
|
||||||
|
├─ preflight_clock_sync + preflight_gpus_idle
|
||||||
|
├─ run_communication_baseline
|
||||||
|
│ ├─ 两节点 CUDA P2P 全矩阵
|
||||||
|
│ ├─ 两节点各自 8-rank AllReduce
|
||||||
|
│ └─ 16-rank AllReduce，CROSS_NIC=0/1/2
|
||||||
|
├─ start_service → Phase 1 start
|
||||||
|
├─ capture_static_snapshots before
|
||||||
|
├─ start_collectors → Head/Worker 同时采集
|
||||||
|
├─ idle → fixed cases → mixed A/B → cooldown
|
||||||
|
├─ check_collectors + stop_collectors
|
||||||
|
├─ capture_static_snapshots after
|
||||||
|
├─ stop_service → Phase 1 stop
|
||||||
|
├─ summarize_results
|
||||||
|
│ └─ 按正式 benchmark 窗口生成第 5 节逐项数据表
|
||||||
|
└─ finish_manifest</code></pre>
|
||||||
|
<p>
|
||||||
|
<code>all</code> 是唯一正式入口。<code>communication</code>、<code>summarize</code>
|
||||||
|
和 <code>stop</code> 是排错/恢复 action，不需要在正常执行前手工调用。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="files">3. 文件职责与调用关系</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>文件</th><th>行数</th><th>职责</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td class="path">config.env</td><td>61</td><td>节点、Case、分层采样周期、通信尺寸、NCCL 选择和 fail-closed 策略。</td></tr>
|
||||||
|
<tr><td class="path">run_hardware_contention_attribution.sh</td><td>1036</td><td>唯一 Shell 编排器：预检、通信文件分发、通信基线、Phase 1 委托、采集器、Case 和清理。</td></tr>
|
||||||
|
<tr><td class="path">communication_baseline.py</td><td>227</td><td>CUDA P2P 全矩阵及 PyTorch/NCCL AllReduce 正确性、延迟和带宽测试。</td></tr>
|
||||||
|
<tr><td class="path">hardware_contention_attribution.py</td><td>1480</td><td>解析所有原始采集器，按 Case 切片，聚合通信并生成 CSV/JSON/report.md。</td></tr>
|
||||||
|
<tr><td class="path">tests/test_hardware_contention_attribution.py</td><td>322</td><td>9 项纯 Python 单元测试，覆盖 worker 无仓库依赖、精确窗口、解析器、RDMA 单位和通信聚合。</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<pre><code>用户
|
||||||
|
└─ Phase2/run_hardware_contention_attribution.sh all
|
||||||
|
├─ source Phase2/config.env
|
||||||
|
├─ docker/torchrun → Phase2/communication_baseline.py
|
||||||
|
├─ env ... bash Phase1/run_quick_map.sh start/fixed/mixed/stop
|
||||||
|
│ └─ Phase1/quick_map_results.py 写 benchmark meta
|
||||||
|
├─ Shell 采集 Head/Worker 原始时间序列
|
||||||
|
└─ Phase2/hardware_contention_attribution.py summarize
|
||||||
|
├─ 读取 Phase1 bench/cases/*/meta.json
|
||||||
|
├─ 读取 Head/Worker 原始监控
|
||||||
|
├─ 读取 communication/COMM_RESULT
|
||||||
|
└─ 输出逐 Case、逐节点、逐指标表和 report.md</code></pre>
|
||||||
|
|
||||||
|
<h3>3.1 Phase 1 与 Phase 2 的边界</h3>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>问题</th><th>由哪个文件负责</th><th>证据</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>模型路径、镜像、TP16、EP、显存比例</td><td>Phase 1 <code>config.env</code> + <code>run_quick_map.sh</code></td><td><code>service/head_server_cmd.txt</code>、<code>worker_server_cmd.txt</code></td></tr>
|
||||||
|
<tr><td>ISL/OSL/C、random 请求和 mixed A/B</td><td>Phase 1 场景表与 benchmark 函数</td><td><code>bench/*/bench_cmd.txt</code>、<code>bench.json</code></td></tr>
|
||||||
|
<tr><td>通信基线、监控周期、Case 选择</td><td>Phase 2 <code>config.env</code></td><td>Phase 2 <code>manifest.json</code></td></tr>
|
||||||
|
<tr><td>硬件归因和数值报告</td><td>Phase 2 Python 汇总器</td><td><code>case_*_summary.csv</code>、<code>report.md</code></td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2 id="config">4. 配置来源</h2>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>行号</th><th>配置组</th><th>关键变量</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>config.env:L3-L16</code></td><td>入口与节点</td><td><code>PHASE1_ENTRY</code>、Head/Worker、端口和容器名。</td></tr>
|
||||||
|
<tr><td><code>L18-L21</code></td><td>诊断 Case</td><td>五个 fixed Case、mixed A/B 开关。</td></tr>
|
||||||
|
<tr><td><code>L23-L39</code></td><td>采样与严格性</td><td>GPU/DCGM/RDMA 1 秒；CPU/进程/网络/NUMA/perf 5 秒；精确窗口和采集器 fail-closed。</td></tr>
|
||||||
|
<tr><td><code>L40-L55</code></td><td>通信基线</td><td>镜像、消息尺寸、迭代次数、P2P 大小、CROSS_NIC 列表、Socket/HCA。</td></tr>
|
||||||
|
<tr><td><code>L57-L61</code></td><td>路径与模式</td><td><code>RESULT_BASE</code>、Runtime、Dry-run、是否允许部分采集器。</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
<code>MEM_FRACTION_STATIC</code> 不在 Phase 2 重复定义。它仍来自 Phase 1，
|
||||||
|
最终展开为 SGLang 的 <code>--mem-fraction-static</code>。判断某次 Run 的真实值，
|
||||||
|
应读取 <code>service/head_server_cmd.txt</code>，不能只看默认配置。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="communication">5. 通信微基准</h2>
|
||||||
|
<h3>5.1 Shell 如何编排</h3>
|
||||||
|
<p>
|
||||||
|
<code>run_hardware_contention_attribution.sh:L281-L513</code> 负责源码暂存、
|
||||||
|
Docker 命令、两节点同步和清理。所有命令先写入 <code>commands/*.txt</code>：
|
||||||
|
</p>
|
||||||
|
<ul>
|
||||||
|
<li><code>L281-L319</code>：从 Head 将当次通信脚本暂存到两节点并校验 SHA256。</li>
|
||||||
|
<li><code>L321-L374</code>：构造容器命令，Head/Worker 各跑一次 P2P。</li>
|
||||||
|
<li><code>L376-L404</code>：Head/Worker 各跑一次 8-rank AllReduce。</li>
|
||||||
|
<li><code>L406-L467</code>：每个 CROSS_NIC 值先启动 Worker rank，再运行 Head rank。</li>
|
||||||
|
<li><code>L469-L513</code>：只清理本实验前缀的通信容器和本次 `/tmp` 暂存目录。</li>
|
||||||
|
</ul>
|
||||||
|
<p>
|
||||||
|
Docker 使用和 SGLang 一致的 CUDA 13 nightly 镜像，并显式透传
|
||||||
|
<code>rdma_cm</code>、<code>uverbs0</code>、<code>uverbs3</code>。
|
||||||
|
<code>NCCL_DEBUG=INFO</code> 只在微基准中打开，用于证明 NET/IB/GDRDMA 路径。
|
||||||
|
Worker 不要求存在 Git 仓库；容器只读挂载自动分发的
|
||||||
|
<code>/tmp/.../<RUN_ID>/communication_baseline.py</code>。结果目录同时保存
|
||||||
|
当次源码副本和 SHA256，避免两个节点 checkout 不一致造成版本漂移。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.2 P2P 代码</h3>
|
||||||
|
<p>
|
||||||
|
<code>communication_baseline.py:L45-L106</code> 遍历所有源 GPU 和目标 GPU，
|
||||||
|
先调用 <code>torch.cuda.can_device_access_peer</code>，再对 256 MiB FP16 Tensor
|
||||||
|
做预热和 CUDA Event 计时。输出包括方向、P50/P95 latency 和 GB/s。
|
||||||
|
汇总器按拓扑拆成同 PCIe Switch 的 PIX 与跨 NUMA 的 SYS。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.3 AllReduce 代码</h3>
|
||||||
|
<p>
|
||||||
|
<code>communication_baseline.py:L107-L198</code> 初始化 NCCL process group，
|
||||||
|
对 1 MiB、64 MiB、1 GiB 分别预热和重复测量。每轮先把各 rank latency
|
||||||
|
gather 到 rank 0，使用最慢 rank 作为 collective 完成时间，并检查归约结果：
|
||||||
|
</p>
|
||||||
|
<pre><code>algbw = message_bytes / latency
|
||||||
|
busbw = algbw × 2 × (world_size - 1) / world_size
|
||||||
|
wrong_values = count(output != expected_sum)</code></pre>
|
||||||
|
<p>
|
||||||
|
这样不会用某个提前返回 rank 的时间美化结果；<code>wrong_values=0</code>
|
||||||
|
才算正确完成。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="collectors">6. 两节点采集器</h2>
|
||||||
|
<h3>6.1 启动前门禁</h3>
|
||||||
|
<p>
|
||||||
|
Shell <code>L67-L199</code> 完成配置、工具、时钟和 GPU 空闲检查。
|
||||||
|
<code>preflight_node_tools</code> 不只检查 <code>dcgmi</code> 文件存在，
|
||||||
|
还实际运行 <code>dcgmi discovery -l</code>；两节点任一 Host Engine 不可用即退出。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>6.2 采集器包装</h3>
|
||||||
|
<p>
|
||||||
|
<code>start_stream_collector</code> 位于 Shell <code>L517-L551</code>。
|
||||||
|
它保存完整命令、PID、唯一进程 tag 和日志；<code>check_collectors</code> 在
|
||||||
|
<code>L722-L740</code> 检查采集器是否提前退出，默认不允许部分成功。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>采集器</th><th>Shell 位置</th><th>周期</th><th>输出</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>nvidia-smi</code></td><td><code>L552-L565</code></td><td>1 秒</td><td><code>gpu_samples.csv</code></td></tr>
|
||||||
|
<tr><td>RDMA HCA counters</td><td><code>L566-L592</code></td><td>1 秒</td><td><code>rdma.csv</code></td></tr>
|
||||||
|
<tr><td>DCGM</td><td><code>L645-L655</code></td><td>1 秒</td><td><code>dcgm_dmon.log</code></td></tr>
|
||||||
|
<tr><td><code>mpstat</code></td><td><code>L656-L663</code></td><td>5 秒</td><td><code>mpstat.log</code></td></tr>
|
||||||
|
<tr><td><code>pidstat -durw</code></td><td><code>L664-L671</code></td><td>5 秒，进程级</td><td><code>pidstat.log</code></td></tr>
|
||||||
|
<tr><td><code>sar -n DEV,EDEV</code></td><td><code>L672-L678</code></td><td>5 秒</td><td><code>sar_net.log</code></td></tr>
|
||||||
|
<tr><td><code>perf stat</code></td><td><code>L680-L689</code></td><td>5 秒</td><td><code>perf_stat.log</code></td></tr>
|
||||||
|
<tr><td><code>numastat</code></td><td><code>L618-L644</code></td><td>5 秒</td><td><code>numa_samples.csv</code></td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
CPU、进程、perf 和 sar 的每行均由 Shell 增加
|
||||||
|
<code>wall_time_ns TAB node TAB payload</code>。NUMA 直接转成结构化 CSV，
|
||||||
|
避免旧版线程级 1 秒日志过大，也让所有指标能按 Case 切片。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="alignment">7. 精确测量窗口</h2>
|
||||||
|
<h3>7.1 Phase 1 如何标记主测量</h3>
|
||||||
|
<p>
|
||||||
|
Phase 1 <code>run_quick_map.sh:L547-L564</code> 每 100 ms 观察 bench 日志；
|
||||||
|
发现 <code>Starting main benchmark run</code> 后调用
|
||||||
|
<code>quick_map_results.py mark-measurement-start</code>。
|
||||||
|
<code>quick_map_results.py:L340-L385</code> 用这个起点和
|
||||||
|
<code>bench.json.duration</code> 生成：
|
||||||
|
</p>
|
||||||
|
<pre><code>measurement_started_at
|
||||||
|
measurement_ended_at
|
||||||
|
measurement_duration_s
|
||||||
|
measurement_window_source = bench_main_marker_plus_duration</code></pre>
|
||||||
|
|
||||||
|
<h3>7.2 Phase 2 如何使用</h3>
|
||||||
|
<p>
|
||||||
|
<code>hardware_contention_attribution.py:L509-L560</code> 优先读取上述字段。
|
||||||
|
只有兼容旧结果时才可能使用进程级窗口；正式配置
|
||||||
|
<code>REQUIRE_PRECISE_WINDOWS=1</code> 会拒绝任何 fallback。
|
||||||
|
<code>L561-L841</code> 对 GPU、DCGM、CPU、进程、perf、NUMA、netdev 和 RDMA
|
||||||
|
使用同一个 <code>started_ns ≤ sample ≤ ended_ns</code> 条件。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="report">8. 逐指标报告</h2>
|
||||||
|
<p>
|
||||||
|
Python <code>summarize</code> 位于
|
||||||
|
<code>hardware_contention_attribution.py:L1036-L1378</code>。
|
||||||
|
它不只生成一个抽象结论，而是按 Phase 2 第 5 节依次写出：
|
||||||
|
</p>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>指标</th><th>解析函数</th><th>Case 汇总文件</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>GPU</td><td><code>summarize_gpu_rows L377-L413</code></td><td><code>case_gpu_summary.csv</code>、<code>case_gpu_node_summary.csv</code></td></tr>
|
||||||
|
<tr><td>DCGM</td><td><code>parse_dcgm L167-L192</code></td><td><code>case_dcgm_summary.csv</code></td></tr>
|
||||||
|
<tr><td>CPU</td><td><code>parse_mpstat L193-L222</code></td><td><code>case_cpu_summary.csv</code></td></tr>
|
||||||
|
<tr><td>进程</td><td><code>parse_pidstat L223-L289</code></td><td><code>case_process_summary.csv</code></td></tr>
|
||||||
|
<tr><td>perf</td><td><code>parse_perf L290-L310</code></td><td><code>case_perf_summary.csv</code></td></tr>
|
||||||
|
<tr><td>NUMA</td><td>结构化 CSV + <code>summarize_case_metrics</code></td><td><code>case_numa_summary.csv</code></td></tr>
|
||||||
|
<tr><td>Linux netdev</td><td><code>parse_sar_net L311-L358</code></td><td><code>case_netdev_summary.csv</code></td></tr>
|
||||||
|
<tr><td>RDMA</td><td><code>summarize_rdma_rows L424-L484</code></td><td><code>case_rdma_summary.csv</code></td></tr>
|
||||||
|
<tr><td>P2P/NCCL</td><td><code>load_communication_rows</code> + <code>aggregate_communication_rows L842-L928</code></td><td><code>communication_summary.csv</code>、<code>communication_aggregate.csv</code></td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
<code>report.md</code> 对每组都打印有效样本数、Mean/P95/Max、Head/Worker
|
||||||
|
或 Case 间比较和源文件。解析不到的值保留为 <code>-</code>，不会被写成 0。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2 id="outputs">9. 结果目录</h2>
|
||||||
|
<pre><code>results/<RUN_ID>/
|
||||||
|
manifest.json
|
||||||
|
commands/
|
||||||
|
communication/
|
||||||
|
service/
|
||||||
|
bench/<phase1-sub-run>/
|
||||||
|
head/
|
||||||
|
gpu_samples.csv
|
||||||
|
dcgm_dmon.log
|
||||||
|
mpstat.log
|
||||||
|
pidstat.log
|
||||||
|
perf_stat.log
|
||||||
|
sar_net.log
|
||||||
|
numa_samples.csv
|
||||||
|
rdma.csv
|
||||||
|
collector_commands/
|
||||||
|
worker/
|
||||||
|
...同上...
|
||||||
|
case_windows.csv
|
||||||
|
bench_summary.csv
|
||||||
|
case_gpu_summary.csv
|
||||||
|
case_gpu_node_summary.csv
|
||||||
|
case_dcgm_summary.csv
|
||||||
|
case_cpu_summary.csv
|
||||||
|
case_process_summary.csv
|
||||||
|
case_perf_summary.csv
|
||||||
|
case_numa_summary.csv
|
||||||
|
case_netdev_summary.csv
|
||||||
|
case_rdma_summary.csv
|
||||||
|
communication_summary.csv
|
||||||
|
communication_aggregate.csv
|
||||||
|
summary.json
|
||||||
|
report.md</code></pre>
|
||||||
|
|
||||||
|
<h2 id="index">10. 函数行号索引</h2>
|
||||||
|
<h3>10.1 Shell 编排器</h3>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>行号</th><th>函数组</th><th>职责</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>L26-L66</td><td>日志、远端执行、命令证据</td><td>基础设施。</td></tr>
|
||||||
|
<tr><td>L67-L199</td><td>配置、工具、时钟、GPU 空闲门禁</td><td>正式运行前 fail-fast。</td></tr>
|
||||||
|
<tr><td>L200-L268</td><td>Manifest、marker、Phase 1 委托</td><td>运行身份与复用边界。</td></tr>
|
||||||
|
<tr><td>L281-L513</td><td>通信基线</td><td>按 Run 分发源码、P2P、8/16-rank AllReduce、CROSS_NIC A/B 与清理。</td></tr>
|
||||||
|
<tr><td>L448-L516</td><td>服务和静态快照</td><td>启停 Phase 1 双机服务并保存环境。</td></tr>
|
||||||
|
<tr><td>L517-L710</td><td>采集命令与启动</td><td>两节点分层采样。</td></tr>
|
||||||
|
<tr><td>L711-L772</td><td>采集器检查和停止</td><td>fail-closed 与残留清理。</td></tr>
|
||||||
|
<tr><td>L782-L835</td><td>fixed/mixed Case</td><td>代表负载编排。</td></tr>
|
||||||
|
<tr><td>L836-L858</td><td>汇总、Manifest、trap</td><td>结果收口。</td></tr>
|
||||||
|
<tr><td>L859-L928</td><td><code>run_all</code></td><td>完整状态机。</td></tr>
|
||||||
|
<tr><td>L929-L968</td><td>辅助 action 与 main</td><td><code>communication/all/summarize/stop</code> 分发。</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>10.2 Python 文件</h3>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>文件/行号</th><th>职责</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>communication_baseline.py:L20-L44</code></td><td>尺寸解析、分位数和 JSON 结果协议。</td></tr>
|
||||||
|
<tr><td><code>L45-L106</code></td><td>CUDA P2P 全矩阵。</td></tr>
|
||||||
|
<tr><td><code>L107-L198</code></td><td>NCCL AllReduce 与正确性。</td></tr>
|
||||||
|
<tr><td><code>hardware_contention_attribution.py:L76-L166</code></td><td>时间、CSV、数字统计基础函数。</td></tr>
|
||||||
|
<tr><td><code>L167-L358</code></td><td>DCGM、mpstat、pidstat、perf、sar 解析器。</td></tr>
|
||||||
|
<tr><td><code>L359-L508</code></td><td>通信、GPU、RDMA、bench 读取与汇总。</td></tr>
|
||||||
|
<tr><td><code>L509-L841</code></td><td>精确窗口和全部 Case 指标切片。</td></tr>
|
||||||
|
<tr><td><code>L842-L1035</code></td><td>通信聚合、CSV、Marker、Manifest。</td></tr>
|
||||||
|
<tr><td><code>L1036-L1378</code></td><td>全部输出表和逐指标 <code>report.md</code>。</td></tr>
|
||||||
|
<tr><td><code>L1379-L1480</code></td><td>CLI 子命令。</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2 id="evidence">11. 最终 Run 证据</h2>
|
||||||
|
<p>
|
||||||
|
正式 Run 完成 8/8 benchmark、8/8 精确测量窗口和 18/18 采集器启停。
|
||||||
|
代码产生的各类输出与实验结论一一对应：
|
||||||
|
</p>
|
||||||
|
<ul>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/manifest.json"><code>manifest.json</code></a>：Run 身份、提交、时间和最终状态。</li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/bench_summary.csv"><code>bench_summary.csv</code></a>：8 个端到端结果。</li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/case_windows.csv"><code>case_windows.csv</code></a>：正式 benchmark 精确时间窗。</li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/collector_status.csv"><code>collector_status.csv</code></a>：两节点采集器生命周期。</li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/report.md"><code>report.md</code></a>：逐指标自动报告。</li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/analysis.md"><code>analysis.md</code></a>：阶段归因与 Phase 3 入口。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<footer>
|
||||||
|
本文只描述提交 <code>39fc2ba565a3</code>。Nsight Systems、SGLang Profiler 和
|
||||||
|
Kernel Timeline 属于 Phase 3，不加入 Phase 2，避免重复采集和职责混淆。
|
||||||
|
</footer>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
812
docs/dsv4pro_pro6000d_2node_sglang/phase2_exp.html
Normal file
812
docs/dsv4pro_pro6000d_2node_sglang/phase2_exp.html
Normal file
@ -0,0 +1,812 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>Phase 2：DeepSeek-V4-Pro 双机 Pro6000D SGLang 硬件与资源竞争归因</title>
|
||||||
|
<style>
|
||||||
|
:root {
|
||||||
|
color-scheme: light;
|
||||||
|
--ink: #18202a;
|
||||||
|
--muted: #5b6570;
|
||||||
|
--line: #d8dde3;
|
||||||
|
--paper: #ffffff;
|
||||||
|
--page: #f3f5f7;
|
||||||
|
--blue: #1769aa;
|
||||||
|
--green: #16734a;
|
||||||
|
--amber: #9a5a00;
|
||||||
|
--red: #a13232;
|
||||||
|
--code: #f0f3f6;
|
||||||
|
}
|
||||||
|
* { box-sizing: border-box; }
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
background: var(--page);
|
||||||
|
color: var(--ink);
|
||||||
|
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||||
|
font-size: 16px;
|
||||||
|
line-height: 1.65;
|
||||||
|
}
|
||||||
|
header {
|
||||||
|
background: #202a34;
|
||||||
|
color: #fff;
|
||||||
|
border-bottom: 5px solid #34a17a;
|
||||||
|
}
|
||||||
|
.header-inner, main {
|
||||||
|
width: min(1120px, calc(100% - 32px));
|
||||||
|
margin: 0 auto;
|
||||||
|
}
|
||||||
|
.header-inner { padding: 38px 0 32px; }
|
||||||
|
h1, h2, h3 { letter-spacing: 0; }
|
||||||
|
h1 {
|
||||||
|
margin: 4px 0 12px;
|
||||||
|
font-size: clamp(26px, 4vw, 42px);
|
||||||
|
line-height: 1.2;
|
||||||
|
}
|
||||||
|
h2 {
|
||||||
|
margin: 38px 0 12px;
|
||||||
|
padding-bottom: 7px;
|
||||||
|
border-bottom: 2px solid var(--line);
|
||||||
|
font-size: 24px;
|
||||||
|
}
|
||||||
|
h3 { margin: 26px 0 8px; font-size: 19px; }
|
||||||
|
.eyebrow {
|
||||||
|
margin: 0;
|
||||||
|
color: #8fd8bd;
|
||||||
|
font-size: 13px;
|
||||||
|
font-weight: 700;
|
||||||
|
text-transform: uppercase;
|
||||||
|
}
|
||||||
|
.meta {
|
||||||
|
display: flex;
|
||||||
|
flex-wrap: wrap;
|
||||||
|
gap: 8px 22px;
|
||||||
|
color: #d7dee5;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
main {
|
||||||
|
margin-top: 24px;
|
||||||
|
margin-bottom: 48px;
|
||||||
|
padding: 30px 38px 42px;
|
||||||
|
background: var(--paper);
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
border-radius: 6px;
|
||||||
|
}
|
||||||
|
.status {
|
||||||
|
padding: 14px 16px;
|
||||||
|
border-left: 4px solid var(--amber);
|
||||||
|
background: #fff7e7;
|
||||||
|
}
|
||||||
|
.decision {
|
||||||
|
padding: 14px 16px;
|
||||||
|
border-left: 4px solid var(--green);
|
||||||
|
background: #eef8f3;
|
||||||
|
}
|
||||||
|
a { color: var(--blue); }
|
||||||
|
.back {
|
||||||
|
display: inline-block;
|
||||||
|
margin-bottom: 10px;
|
||||||
|
font-weight: 650;
|
||||||
|
}
|
||||||
|
code {
|
||||||
|
padding: 1px 5px;
|
||||||
|
background: var(--code);
|
||||||
|
border-radius: 3px;
|
||||||
|
font-family: "SFMono-Regular", Consolas, monospace;
|
||||||
|
font-size: 0.92em;
|
||||||
|
}
|
||||||
|
pre {
|
||||||
|
overflow-x: auto;
|
||||||
|
padding: 14px 16px;
|
||||||
|
background: #202a34;
|
||||||
|
color: #f4f7fa;
|
||||||
|
border-radius: 5px;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
pre code { padding: 0; background: transparent; color: inherit; }
|
||||||
|
table {
|
||||||
|
width: 100%;
|
||||||
|
margin: 14px 0 22px;
|
||||||
|
border-collapse: collapse;
|
||||||
|
font-size: 14px;
|
||||||
|
}
|
||||||
|
th, td {
|
||||||
|
padding: 10px 11px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
text-align: left;
|
||||||
|
vertical-align: top;
|
||||||
|
}
|
||||||
|
th { background: #edf1f4; }
|
||||||
|
.pass { color: var(--green); font-weight: 700; }
|
||||||
|
.pending { color: var(--amber); font-weight: 700; }
|
||||||
|
.fail { color: var(--red); font-weight: 700; }
|
||||||
|
li + li { margin-top: 5px; }
|
||||||
|
@media (max-width: 720px) {
|
||||||
|
main { padding: 22px 18px 30px; }
|
||||||
|
table { display: block; overflow-x: auto; white-space: nowrap; }
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header>
|
||||||
|
<div class="header-inner">
|
||||||
|
<p class="eyebrow">Design, Implementation & Result Record</p>
|
||||||
|
<h1>Phase 2：DeepSeek-V4-Pro 双机 Pro6000D SGLang 硬件与资源竞争归因</h1>
|
||||||
|
<div class="meta">
|
||||||
|
<span>节点：174.1.51.5 + 174.1.51.7</span>
|
||||||
|
<span>拓扑：SGLang TP16 / EP2</span>
|
||||||
|
<span>更新：2026-07-31 17:22:20 CST</span>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<a class="back" href="./推理优化计划.html">返回推理优化主计划</a>
|
||||||
|
<a class="back" href="./phase2_code.html">打开 Phase 2 代码详解</a>
|
||||||
|
|
||||||
|
<p class="status">
|
||||||
|
<strong>当前状态：Phase 2 已完成。</strong>
|
||||||
|
最终 Run <code>dsv4pro-phase2-20260731-163620</code> 在 28 分 44 秒内完成
|
||||||
|
8/8 个 benchmark，正式测量窗口 8/8 精确，18/18 个采集器正常启停。
|
||||||
|
Head/Worker 的 DCGM、CPU、NUMA、双 Rail RDMA 和通信微基准证据均有效；
|
||||||
|
实验结束后两节点容器、服务端口和 16 张 GPU 已清理。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>1. Phase 1 交接结果</h2>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>代表负载</th><th>关键结果</th><th>Phase 2 用途</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>128K → 1，C=1</td><td>Input TPS 2,710.16；TTFT P95 48.344 s</td><td>纯长 Prefill 的计算、显存与通信归因</td></tr>
|
||||||
|
<tr><td>32K → 1，C=16</td><td>Input TPS 3,112.77；TTFT P95 162.087 s</td><td>并发 Prefill 的排队、Chunk 调度与节点均衡</td></tr>
|
||||||
|
<tr><td>1K → 1K，C=32</td><td>Output TPS 461.68；TPOT P95 63.31 ms</td><td>普通 Decode 的 GPU、CPU 与通信基线</td></tr>
|
||||||
|
<tr><td>1K → 4K，C=16</td><td>Output TPS 310.02；TPOT P95 50.33 ms</td><td>持续 Decode、KV 增长和稳态资源占用</td></tr>
|
||||||
|
<tr><td>128K → 1K，C=1</td><td>TTFT P95 49.326 s；TPOT P95 32.24 ms</td><td>分离长 Prefill 与长上下文 Decode 成本</td></tr>
|
||||||
|
<tr><td>1K → 1K，C=32 + 128K 注入</td><td>Output TPS -24.08%；TPOT P95 +66.55%</td><td>Prefill 干扰 Decode 时的硬件资源竞争</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
最终基线已由 Head 与 Worker 日志证明使用
|
||||||
|
<code>mlx5_0/mlx5_3</code> 双 Rail <code>NET/IB + GDRDMA</code>，
|
||||||
|
正式测量请求为冷 Prefix。正式矩阵 12/12、长 Decode 补测 2/2 均成功。
|
||||||
|
Phase 2 保持相同服务配置和请求口径。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>2. 本阶段的边界</h2>
|
||||||
|
<ul>
|
||||||
|
<li>只测试 SGLang，不测试 vLLM。</li>
|
||||||
|
<li>保留模型、镜像、TP16、EP2、显存比例和已验证的双 Rail NCCL 配置。</li>
|
||||||
|
<li>模型端到端 Case 不启用 Nsight Systems、PyTorch Profiler、NCCL DEBUG 或投机解码；独立通信基线临时启用 NCCL INFO 以保存实际路径证据。</li>
|
||||||
|
<li>不调参，不尝试优化；先获得足以区分瓶颈类别的硬件证据。</li>
|
||||||
|
<li>只重放五个固定代表负载和一组混合 A/B，不重复 Phase 1 全矩阵。</li>
|
||||||
|
<li>采集器从请求开始前启动，到请求结束后停止，不能中途补采后声称完整。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2>3. 待验证假设</h2>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>假设</th><th>预期硬件表现</th><th>后续方向</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>DSV4/NSA Prefill Kernel 计算受限</td><td>GPU 持续忙、高功耗和稳定频率；双 Rail 流量不高</td><td>Phase 3 捕获 Kernel 与 Attention/Indexer 时间线</td></tr>
|
||||||
|
<tr><td>权重或激活显存带宽受限</td><td>GPU Memory Utilization 高，SM 指标未必饱和；功耗可能低于纯计算</td><td>补 DCGM/Profiler 的 DRAM Active，再看 Kernel</td></tr>
|
||||||
|
<tr><td>TP16 跨机通信受限</td><td>RoCE 吞吐高或两条 Rail 明显失衡，GPU 出现等待</td><td>NCCL_CROSS_NIC 0/1/2 快速 A/B，随后看 NCCL Timeline</td></tr>
|
||||||
|
<tr><td>CPU Scheduler 或 Kernel Launch 受限</td><td>GPU 利用率锯齿或有空洞，单 CPU 核持续满载</td><td>定位 Scheduler/Tokenizer 线程与 launch gap</td></tr>
|
||||||
|
<tr><td>频率、功耗或温度限制</td><td>P-state、SM Clock 或 Power 持续异常，可能出现节流原因</td><td>修正电源、散热或 Clock Policy 后复测</td></tr>
|
||||||
|
<tr><td>节点或 Rank 不均衡</td><td>两节点或不同 GPU 的利用率、功耗、网络流量存在固定偏差</td><td>检查 NUMA、GPU-NIC 亲和与慢 Rank</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h2>4. 诊断 Run</h2>
|
||||||
|
<ol>
|
||||||
|
<li>确认 16 张 GPU 空闲，先跑两节点 PCIe P2P、单机 8 rank AllReduce 和双机 16 rank AllReduce；双机分别测试 <code>NCCL_CROSS_NIC=0/1/2</code>。</li>
|
||||||
|
<li>保存两节点静态快照：GPU/NIC/NUMA 拓扑、驱动、CUDA、镜像与服务命令。</li>
|
||||||
|
<li>复用 Phase 1 已验证的 <code>run_quick_map.sh start</code> 启动同配置双机服务。</li>
|
||||||
|
<li>在 Head 和 Worker 同时启动 GPU、CPU、网卡与 RDMA 采样，先记录 15 秒空闲基线。</li>
|
||||||
|
<li>依次重放 <code>128K → 1, C=1</code>、<code>32K → 1, C=16</code> 与 <code>1K → 1K, C=32</code>。</li>
|
||||||
|
<li>重放 <code>1K → 4K, C=16</code> 和 <code>128K → 1K, C=1</code>，观察持续与长上下文 Decode。</li>
|
||||||
|
<li>重放 <code>1K → 1K, C=32</code> Control 与 128K Prefill 注入 Treatment，保留相同注入时序。</li>
|
||||||
|
<li>请求结束后继续采样 15 秒，再停止采集器和服务。</li>
|
||||||
|
<li>按时间戳将请求、GPU、CPU 和双 Rail 指标对齐，生成摘要与判定。</li>
|
||||||
|
</ol>
|
||||||
|
<pre><code>idle 15s
|
||||||
|
│ 128K→1 C1 │ 32K→1 C16 │ 1K→1K C32
|
||||||
|
│ 1K→4K C16 │ 128K→1K C1
|
||||||
|
│ Decode Control │ Decode + Prefill
|
||||||
|
cooldown 15s
|
||||||
|
|
||||||
|
Head 与 Worker 的所有采集器覆盖完整诊断窗口。</code></pre>
|
||||||
|
<p>
|
||||||
|
Phase 1 中服务加载约 5 分 30 秒；通信基线、五个固定负载、混合 A/B、
|
||||||
|
静态快照、采样和清理组成一次完整 Phase 2 Run。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>4.1 你只需要运行的入口</h3>
|
||||||
|
<p class="decision">
|
||||||
|
<strong>操作规则：先在 Worker <code>.7</code> 做一次 DCGM 准备，再只在
|
||||||
|
Head <code>.5</code> 执行 Phase 2 的 <code>all</code>。</strong>
|
||||||
|
不要手工执行 Phase 1 的 <code>start</code> 或 <code>stop</code>。
|
||||||
|
Phase 2 会在内部复用它们，并负责异常退出时的采集器、Head、Worker 清理。
|
||||||
|
</p>
|
||||||
|
<pre><code class="language-bash"># [仅在 Worker 174.1.51.7 执行一次]
|
||||||
|
# 不需要 source、conda activate，也不要在 .7 运行 Phase 2 的 all
|
||||||
|
systemctl start nvidia-dcgm
|
||||||
|
systemctl is-active nvidia-dcgm
|
||||||
|
dcgmi discovery -l
|
||||||
|
|
||||||
|
# 预期：第二条输出 active，第三条列出本机 8 张 GPU
|
||||||
|
|
||||||
|
# [以下仅在 Head 174.1.51.5 执行]
|
||||||
|
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
|
||||||
|
|
||||||
|
# 第一次先展开全部命令，不启动服务、不占用 GPU、不发送请求
|
||||||
|
DRY_RUN=1 RUN_ID=dsv4pro-phase2-dryrun-$(date +%Y%m%d-%H%M%S) \
|
||||||
|
bash run_hardware_contention_attribution.sh all
|
||||||
|
|
||||||
|
# 正式实验：仍然只有同一个 all 入口，tmux 只负责断线后继续运行
|
||||||
|
RUN_ID=dsv4pro-phase2-$(date +%Y%m%d-%H%M%S)
|
||||||
|
tmux new-session -d -s dsv4pro-phase2 \
|
||||||
|
"RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
|
||||||
|
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||||
|
|
||||||
|
tmux attach -t dsv4pro-phase2</code></pre>
|
||||||
|
<p>
|
||||||
|
<code>.7</code> 的三条命令只负责让 Worker DCGM Host Engine 可用；
|
||||||
|
SGLang Worker、其余采集器和结果回收仍由 <code>.5</code> 的唯一入口通过 SSH 管理。
|
||||||
|
若需要机器重启后自动启动 DCGM，应由运维另行决定是否执行
|
||||||
|
<code>systemctl enable nvidia-dcgm</code>。
|
||||||
|
</p>
|
||||||
|
<p><strong><code>all</code> 内部执行顺序：</strong></p>
|
||||||
|
<pre><code>配置、工具、DCGM 与 GPU 空闲门禁
|
||||||
|
→ 通信基线：两节点 P2P、两组单机 8-rank AllReduce、
|
||||||
|
三组双机 16-rank NCCL_CROSS_NIC A/B
|
||||||
|
→ Phase 1 start：启动同配置 TP16 服务
|
||||||
|
→ 两节点静态快照
|
||||||
|
→ 启动两节点采集器并记录 15 秒 idle
|
||||||
|
→ 五个固定 Case
|
||||||
|
→ 混合 Prefill/Decode A/B
|
||||||
|
→ 15 秒 cooldown
|
||||||
|
→ 停止采集器并保存后快照
|
||||||
|
→ Phase 1 stop：停止 Head/Worker
|
||||||
|
→ 生成按第 5 节逐项对应的 CSV、JSON 与 report.md</code></pre>
|
||||||
|
<p>
|
||||||
|
Phase 1 的作用是提供已经验证过的双机 Docker 服务和 Benchmark 实现，
|
||||||
|
不是第二个用户入口。实际展开的服务、Benchmark 和采集命令都会写入
|
||||||
|
<code>results/<RUN_ID>/service/</code>、<code>commands/</code> 和
|
||||||
|
<code>head|worker/collector_commands/</code>，不依赖跨文档猜测。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>5. 采集指标</h2>
|
||||||
|
<p>
|
||||||
|
本节记录正式实现使用的命令，而不是建议性伪代码。命令由
|
||||||
|
<code>run_hardware_contention_attribution.sh</code> 在 Head 和 Worker 同时启动；
|
||||||
|
每条展开后的命令会另外保存在
|
||||||
|
<code>results/<RUN_ID>/head|worker/collector_commands/</code>。
|
||||||
|
</p>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>层级</th><th>连续采样</th><th>静态或前后快照</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>GPU</td><td>利用率、Memory Utilization、显存、功耗、SM/Memory Clock、温度、P-state</td><td><code>nvidia-smi topo -m</code>、Compute Process</td></tr>
|
||||||
|
<tr><td>CPU</td><td>每核利用率、上下文切换、服务进程 CPU/内存</td><td>NUMA 拓扑、容器 PID 与 CPU Affinity</td></tr>
|
||||||
|
<tr><td>Network</td><td><code>eth0/eth3</code> RX/TX</td><td><code>ethtool -S</code> 错误计数前后差</td></tr>
|
||||||
|
<tr><td>RDMA</td><td><code>mlx5_0/mlx5_3</code> port_xmit/recv_data 差分</td><td>Port State、GID 与错误计数</td></tr>
|
||||||
|
<tr><td>DCGM</td><td>SM Active、DRAM Active、Tensor Active、PCIe</td><td>工具版本与可用 Field</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>5.1 时间对齐与 Case Marker</h3>
|
||||||
|
<pre><code class="language-bash"># 每条 GPU/RDMA 样本写入相同格式的宿主机墙钟时间
|
||||||
|
date +%s%N
|
||||||
|
|
||||||
|
# 实验前检查两节点秒级时钟差
|
||||||
|
date +%s
|
||||||
|
|
||||||
|
# Case 开始、结束和服务状态由 Python 写入 markers.csv
|
||||||
|
python3 hardware_contention_attribution.py marker \
|
||||||
|
--path markers.csv \
|
||||||
|
--node head \
|
||||||
|
--event case_start \
|
||||||
|
--case-id long_prefill_latency_128k_c1</code></pre>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>数据</th><th>含义</th><th>为什么需要</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>wall_time_ns</code></td><td>Unix Epoch 纳秒时间</td><td>把 GPU、CPU、RDMA 与 Benchmark 放到同一时间轴</td></tr>
|
||||||
|
<tr><td><code>case_start/case_end</code></td><td>一个 Case 的编排边界</td><td>从整段连续采样中切出对应负载</td></tr>
|
||||||
|
<tr><td><code>CLOCK_SKEW_TOLERANCE_S=2</code></td><td>两节点允许的最大秒级时钟差</td><td>避免 Head/Worker 的同一时刻被错位比较</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
最终实现由 Phase 1 监听 <code>bench.log</code> 中的
|
||||||
|
<code>Starting main benchmark run</code>，立刻写入
|
||||||
|
<code>measurement_start.json</code>；再使用 SGLang <code>bench.json</code>
|
||||||
|
的正式 benchmark duration 计算结束时间。Phase 2 优先读取
|
||||||
|
<code>measurement_started_at/measurement_ended_at</code>，不会把数据生成和
|
||||||
|
Warm-up 混入硬件均值。<code>REQUIRE_PRECISE_WINDOWS=1</code> 时，任何 Case
|
||||||
|
缺少精确窗口都会让汇总失败，而不是悄悄回退。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.2 GPU 基础状态：<code>nvidia-smi</code></h3>
|
||||||
|
<pre><code class="language-bash">nvidia-smi \
|
||||||
|
--query-gpu=index,timestamp,utilization.gpu,utilization.memory,\
|
||||||
|
memory.used,memory.total,power.draw,temperature.gpu,\
|
||||||
|
clocks.sm,clocks.mem,pstate \
|
||||||
|
--format=csv,noheader,nounits</code></pre>
|
||||||
|
<p>脚本每秒运行一次，并在每行前加入 <code>wall_time_ns</code> 和节点角色。</p>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>字段</th><th>代表什么</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>utilization.gpu</code></td><td>采样周期内至少有一个 Kernel 在执行的时间比例</td></tr>
|
||||||
|
<tr><td><code>utilization.memory</code></td><td>采样周期内显存控制器处于忙碌状态的时间比例</td></tr>
|
||||||
|
<tr><td><code>memory.used/total</code></td><td>当前总显存分配量与设备显存容量</td></tr>
|
||||||
|
<tr><td><code>power.draw</code></td><td>GPU 当前功耗，用于比较不同负载的能耗状态</td></tr>
|
||||||
|
<tr><td><code>clocks.sm/clocks.mem</code></td><td>SM 与显存当前工作频率</td></tr>
|
||||||
|
<tr><td><code>pstate</code></td><td>GPU 性能状态，P0 通常是最高性能态</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
原始输出为 <code>head|worker/gpu_samples.csv</code>；
|
||||||
|
<code>gpu_summary.csv</code> 汇总整段运行，
|
||||||
|
<code>case_gpu_summary.csv</code> 按节点、Case 和 GPU 汇总平均值与峰值。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.3 GPU Profiling Counter：DCGM</h3>
|
||||||
|
<pre><code class="language-bash">DCGM_FIELD_IDS=1001,1002,1003,1004,1005,1009,1010
|
||||||
|
|
||||||
|
dcgmi dmon \
|
||||||
|
-e 1001,1002,1003,1004,1005,1009,1010 \
|
||||||
|
-d 1000</code></pre>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>Field ID</th><th>Field Tag</th><th>含义</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>1001</td><td><code>gr_engine_active</code></td><td>Graphics/Compute Engine 活跃比例，接近整体 GPU 执行忙碌度</td></tr>
|
||||||
|
<tr><td>1002</td><td><code>sm_active</code></td><td>SM 至少有一个 Warp 活跃的比例</td></tr>
|
||||||
|
<tr><td>1003</td><td><code>sm_occupancy</code></td><td>活跃 Warp 相对硬件可容纳 Warp 的比例</td></tr>
|
||||||
|
<tr><td>1004</td><td><code>tensor_active</code></td><td>Tensor Core 指令活跃比例</td></tr>
|
||||||
|
<tr><td>1005</td><td><code>dram_active</code></td><td>设备显存接口活跃比例；Pro6000D 为 GDDR7，用于判断设备显存带宽压力</td></tr>
|
||||||
|
<tr><td>1009</td><td><code>pcie_tx_bytes</code></td><td>GPU 经 PCIe 发出的字节速率</td></tr>
|
||||||
|
<tr><td>1010</td><td><code>pcie_rx_bytes</code></td><td>GPU 经 PCIe 接收的字节速率</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
<code>sm_active</code> 高而 <code>sm_occupancy</code> 低，表示 SM 经常有工作，
|
||||||
|
但同时驻留的 Warp 不多；后续通过 Kernel Timeline 区分小 Kernel、
|
||||||
|
寄存器/共享内存约束和同步。DCGM 是 NVIDIA Data Center GPU Manager：
|
||||||
|
<code>nvidia-dcgm</code>/<code>nv-hostengine</code> 是后台 Host Engine，
|
||||||
|
<code>dcgmi</code> 是客户端，Field ID 是指标编号。最终代码在两节点预检
|
||||||
|
<code>dcgmi discovery -l</code>，任一 Host Engine 不可用即 fail-fast；
|
||||||
|
正式结果必须同时包含 Head 和 Worker 的 <code>case_dcgm_summary.csv</code>。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.4 CPU、进程与 Kernel Launch 侧证据</h3>
|
||||||
|
<pre><code class="language-bash"># 全部逻辑 CPU，每 5 秒输出一次
|
||||||
|
mpstat -P ALL 5
|
||||||
|
|
||||||
|
# 找到容器内进程对应的宿主 PID
|
||||||
|
docker top <container> -eo pid,ppid,psr,pcpu,pmem,stat,comm,args
|
||||||
|
|
||||||
|
# 最终命令：进程级 CPU、I/O、缺页、上下文切换，不展开全部线程
|
||||||
|
pidstat -durw -p "<comma-separated-host-pids>" 5
|
||||||
|
|
||||||
|
# 每 5 秒输出一次硬件/软件计数器增量
|
||||||
|
perf stat -p "<comma-separated-host-pids>" -I 5000 \
|
||||||
|
-e cycles,instructions,cache-misses,context-switches,\
|
||||||
|
cpu-migrations,page-faults
|
||||||
|
|
||||||
|
# mpstat/pidstat/perf 每行都由包装器增加：
|
||||||
|
# wall_time_ns TAB node TAB 原始输出</code></pre>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>命令/字段</th><th>回答的问题</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>mpstat -P ALL</code></td><td>整机是否 CPU 饱和，是否只有少量核心接近 100%，是否存在 I/O Wait</td></tr>
|
||||||
|
<tr><td><code>docker top</code></td><td>把容器进程映射为宿主 PID、CPU 核 <code>PSR</code> 和进程状态</td></tr>
|
||||||
|
<tr><td><code>pidstat -u</code></td><td>服务进程的用户态、内核态 CPU 时间</td></tr>
|
||||||
|
<tr><td><code>pidstat -d</code></td><td>进程块设备 I/O</td></tr>
|
||||||
|
<tr><td><code>pidstat -r</code></td><td>内存和 Page Fault 行为</td></tr>
|
||||||
|
<tr><td><code>pidstat -w</code></td><td>主动/被动上下文切换，辅助发现线程阻塞或调度抖动</td></tr>
|
||||||
|
<tr><td><code>perf cycles/instructions</code></td><td>CPU 周期与指令执行量，可计算近似 IPC</td></tr>
|
||||||
|
<tr><td><code>cache-misses/migrations</code></td><td>CPU Cache 压力和线程跨核迁移</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
最终实现使用进程级 5 秒采样，避免首轮线程级 1 秒采样产生数百 MB 日志。
|
||||||
|
<code>case_cpu_summary.csv</code>、<code>case_process_summary.csv</code> 和
|
||||||
|
<code>case_perf_summary.csv</code> 都按正式测量窗口切片；只有先发现异常进程，
|
||||||
|
才在后续短窗口单独开启线程级采样。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.5 NUMA 与 CPU/内存亲和</h3>
|
||||||
|
<pre><code class="language-bash"># 静态 NUMA 节点、CPU 和内存布局
|
||||||
|
numactl --hardware
|
||||||
|
numastat -m
|
||||||
|
|
||||||
|
# 每 5 秒按容器宿主 PID 查看本地/远端 NUMA 内存
|
||||||
|
numastat -p <host-pid>
|
||||||
|
# 解析为：
|
||||||
|
# wall_time_ns,node,node0_mib,node1_mib,total_mib,process_count
|
||||||
|
|
||||||
|
# 同时保存 GPU、CPU、NIC 的拓扑关系
|
||||||
|
nvidia-smi topo -m</code></pre>
|
||||||
|
<p>
|
||||||
|
NUMA 是多路 CPU 机器的“本地内存”结构。进程长期从远端 NUMA Node 取内存，
|
||||||
|
或 GPU/NIC 对应的 CPU 线程被调度到另一侧，可能增加 Host 侧延迟。
|
||||||
|
最终采集器把 <code>numastat -p</code> 解析为
|
||||||
|
<code>numa_samples.csv</code>，再按正式测量窗口生成
|
||||||
|
<code>case_numa_summary.csv</code>。这样可以直接比较 Node0/Node1 MiB，
|
||||||
|
而不是依靠人工阅读不断刷新的文本。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.6 普通网卡统计与 RDMA 数据面</h3>
|
||||||
|
<pre><code class="language-bash"># Linux netdev 层，每 5 秒采样吞吐与错误
|
||||||
|
sar -n DEV,EDEV 5
|
||||||
|
|
||||||
|
# Case 前后保存物理端口状态和驱动计数器
|
||||||
|
ethtool eth0
|
||||||
|
ethtool eth3
|
||||||
|
ethtool -S eth0
|
||||||
|
ethtool -S eth3
|
||||||
|
|
||||||
|
# RDMA 设备与端口状态
|
||||||
|
ibdev2netdev
|
||||||
|
ibstat
|
||||||
|
rdma link show</code></pre>
|
||||||
|
<p>
|
||||||
|
<code>sar</code> 记录 Linux 普通网络栈中的 <code>eth0/eth3</code> 流量；
|
||||||
|
GDRDMA 数据量由 <code>mlx5_0/mlx5_3</code> HCA 的 sysfs Counter 记录：
|
||||||
|
</p>
|
||||||
|
<pre><code class="language-bash">for hca in mlx5_0 mlx5_3; do
|
||||||
|
base="/sys/class/infiniband/${hca}/ports/1"
|
||||||
|
cat "${base}/counters/port_xmit_data"
|
||||||
|
cat "${base}/counters/port_rcv_data"
|
||||||
|
cat "${base}/counters/port_xmit_wait"
|
||||||
|
cat "${base}/counters/port_xmit_discards"
|
||||||
|
cat "${base}/counters/port_rcv_errors"
|
||||||
|
cat "${base}/hw_counters/req_transport_retries_exceeded"
|
||||||
|
cat "${base}/hw_counters/req_rnr_retries_exceeded"
|
||||||
|
done</code></pre>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>Counter</th><th>含义</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>port_xmit_data/port_rcv_data</code></td><td>HCA 发送/接收数据累计量；IB Counter 单位是 4 Octets，脚本用 <code>delta × 4 × 8 / seconds</code> 换算 Gbit/s</td></tr>
|
||||||
|
<tr><td><code>port_xmit_wait</code></td><td>端口因缺少发送 Credit 等原因等待的时间，持续增长可能指向拥塞</td></tr>
|
||||||
|
<tr><td><code>port_xmit_discards/port_rcv_errors</code></td><td>发送丢弃和接收错误增量</td></tr>
|
||||||
|
<tr><td><code>req_transport_retries_exceeded</code></td><td>RDMA Transport 重试耗尽</td></tr>
|
||||||
|
<tr><td><code>req_rnr_retries_exceeded</code></td><td>Receiver Not Ready 重试耗尽</td></tr>
|
||||||
|
<tr><td><code>roce_adp_retrans*</code></td><td>RoCE 自适应重传及超时相关计数</td></tr>
|
||||||
|
<tr><td><code>np_ecn_marked* / *cnp*</code></td><td>ECN 标记和拥塞通知包，用于辅助判断 RoCE 拥塞</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
原始数据为 <code>head|worker/rdma.csv</code>；
|
||||||
|
<code>case_rdma_summary.csv</code> 按 Case、节点和 HCA 计算吞吐及错误增量。
|
||||||
|
它说明双 Rail 的实际流量、均衡性和错误增量；
|
||||||
|
<code>case_netdev_summary.csv</code> 同时保留 Linux netdev 层的
|
||||||
|
<code>eth0/eth3</code> RX/TX 与错误。Phase 3 再把 NCCL Collective
|
||||||
|
放到请求 Timeline 中分析持续时间和计算重叠。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.7 机内 PCIe 与 NCCL 通信基线</h3>
|
||||||
|
<pre><code class="language-bash"># 由 all 入口自动执行；不需要用户手工运行 torchrun
|
||||||
|
# 每个节点：所有 GPU 源/目标对，FP16 256 MiB CUDA P2P copy
|
||||||
|
python3 communication_baseline.py p2p \
|
||||||
|
--size 256M --warmup 3 --iterations 10
|
||||||
|
|
||||||
|
# 每个节点：8 rank NCCL AllReduce
|
||||||
|
torchrun --standalone --nproc-per-node=8 \
|
||||||
|
communication_baseline.py all-reduce \
|
||||||
|
--sizes 1M,64M,1G --repetitions 3 --warmup 5 --iterations 10
|
||||||
|
|
||||||
|
# 双节点：16 rank；分别设置 NCCL_CROSS_NIC=0、1、2
|
||||||
|
torchrun --nnodes=2 --nproc-per-node=8 \
|
||||||
|
--master-addr 10.101.0.11 --node-rank <0-or-1> \
|
||||||
|
communication_baseline.py all-reduce \
|
||||||
|
--sizes 1M,64M,1G --repetitions 3 --warmup 5 --iterations 10</code></pre>
|
||||||
|
<p>
|
||||||
|
P2P 结果按 <code>same_pcie_switch</code>（PIX）和
|
||||||
|
<code>cross_numa_sys</code>（SYS）分别汇总，不用一个平均值掩盖跨 CPU 路径。
|
||||||
|
AllReduce 同时报告 P50/P95 latency、<code>algbw</code>、
|
||||||
|
<code>busbw</code>、正确性错误数和实际 NCCL 路径。1 MiB、64 MiB、1 GiB
|
||||||
|
分别覆盖小消息延迟、中等消息和大消息带宽；双机 A/B 直接给出
|
||||||
|
<code>NCCL_CROSS_NIC=0/1/2</code> 的数值比较。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h3>5.8 静态快照与结果关系</h3>
|
||||||
|
<pre><code class="language-bash">nvidia-smi
|
||||||
|
nvidia-smi topo -m
|
||||||
|
lscpu
|
||||||
|
numactl --hardware
|
||||||
|
ip -details link show eth0
|
||||||
|
ip -details link show eth3
|
||||||
|
docker inspect <container>
|
||||||
|
docker top <container> -eo pid,ppid,psr,pcpu,pmem,stat,comm,args</code></pre>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>结果文件</th><th>内容</th><th>主要用途</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>static_before.log / static_after.log</code></td><td>GPU、CPU、NUMA、NIC、RDMA、容器前后快照</td><td>证明运行环境，并比较错误计数和清理状态</td></tr>
|
||||||
|
<tr><td><code>collector_status.csv</code></td><td>每个采集器的启动、停止或提前退出状态</td><td>防止把缺失采集器当作 0 值</td></tr>
|
||||||
|
<tr><td><code>case_windows.csv</code></td><td>每个 Benchmark Case 的起止时间</td><td>从连续硬件日志中切片</td></tr>
|
||||||
|
<tr><td><code>bench_summary.csv</code></td><td>TPS、TTFT、TPOT、ITL、E2E</td><td>把硬件现象与用户侧性能对应</td></tr>
|
||||||
|
<tr><td><code>case_gpu_summary.csv</code></td><td>每 Case、节点、GPU 的利用率、显存、功耗、频率</td><td>比较负载与节点/GPU 不均衡</td></tr>
|
||||||
|
<tr><td><code>case_dcgm_summary.csv</code></td><td>每 Case、节点、GPU 的 SM/Tensor/显存接口/PCIe 指标</td><td>区分计算、设备显存和 PCIe 活跃度</td></tr>
|
||||||
|
<tr><td><code>case_cpu_summary.csv</code></td><td>整机与逐核 CPU 利用率、I/O Wait</td><td>识别整机饱和和少数热点核</td></tr>
|
||||||
|
<tr><td><code>case_process_summary.csv</code></td><td>服务进程 CPU、I/O、缺页、内存与上下文切换</td><td>定位 Host 进程开销与阻塞</td></tr>
|
||||||
|
<tr><td><code>case_perf_summary.csv</code></td><td>cycles、instructions、cache miss、迁移与缺页</td><td>计算 IPC 并判断 Cache/调度压力</td></tr>
|
||||||
|
<tr><td><code>case_numa_summary.csv</code></td><td>Node0/Node1 进程内存分布</td><td>识别跨 NUMA 放置</td></tr>
|
||||||
|
<tr><td><code>case_netdev_summary.csv</code></td><td><code>eth0/eth3</code> 吞吐与错误</td><td>与 RDMA HCA Counter 做分层核对</td></tr>
|
||||||
|
<tr><td><code>case_rdma_summary.csv</code></td><td>每 Case、节点、Rail 的吞吐和错误增量</td><td>判断双 Rail 使用、均衡和数据面错误</td></tr>
|
||||||
|
<tr><td><code>communication_summary.csv</code></td><td>每次 P2P/AllReduce 原始测量</td><td>保留每条 GPU 对、消息尺寸、CROSS_NIC 和重复实验</td></tr>
|
||||||
|
<tr><td><code>communication_aggregate.csv</code></td><td>PIX/SYS P2P 与单/双机 AllReduce 聚合</td><td>提供 P50/P95、algbw、busbw 和正确性比较</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>5.9 最终结果如何逐项汇报</h3>
|
||||||
|
<p class="decision">
|
||||||
|
最终 <code>report.md</code> 的章节顺序与本节一一对应。每一项必须同时给出
|
||||||
|
<strong>原始文件、有效样本数、Head/Worker 数值、Case 间变化和解释</strong>；
|
||||||
|
不能只写“GPU 较忙”“网络未饱和”这类抽象结论。
|
||||||
|
</p>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>第 5 节指标</th><th>报告中的数值</th><th>最小分析动作</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>5.1 时间窗</td><td>窗口来源、开始/结束、duration、采样数</td><td>确认全部为 <code>bench_main_marker_plus_duration</code></td></tr>
|
||||||
|
<tr><td>5.2 GPU</td><td>利用率/显存/功耗/频率的 Mean、P95、Max</td><td>比较两节点、8 卡离散度和不同 Case</td></tr>
|
||||||
|
<tr><td>5.3 DCGM</td><td>SM Active/Occupancy、Tensor/DRAM Active、PCIe TX/RX</td><td>比较计算、设备显存和 PCIe 哪一侧随负载上升</td></tr>
|
||||||
|
<tr><td>5.4 CPU/进程/perf</td><td>整机/热点核、进程 CPU/I/O/缺页/切换、IPC/Cache miss</td><td>区分整机容量、单线程热点和 Host 调度开销</td></tr>
|
||||||
|
<tr><td>5.5 NUMA</td><td>Node0/Node1 MiB 与比例</td><td>比较服务内存是否偏离 GPU/NIC 所在 NUMA</td></tr>
|
||||||
|
<tr><td>5.6 Network/RDMA</td><td>eth0/eth3、mlx5_0/mlx5_3 Gbit/s 与错误增量</td><td>计算双 Rail 均衡比例并核对丢弃/重试</td></tr>
|
||||||
|
<tr><td>5.7 Communication</td><td>PIX/SYS P2P、8/16 rank AllReduce P50/P95、algbw/busbw</td><td>比较跨 NUMA 损失与 CROSS_NIC 0/1/2</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
某个采集器无数据时报告显示 <code>-</code> 并附失败状态，不会把缺失值写成
|
||||||
|
<code>0</code>。正式 Run 默认 <code>ALLOW_PARTIAL_COLLECTORS=0</code>，
|
||||||
|
因此必需采集器提前退出会让 Run 失败。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>6. 精简代码设计</h2>
|
||||||
|
<p>已新增目录：</p>
|
||||||
|
<pre><code>/data/hzy/sskj/experiments/pro6000/
|
||||||
|
dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/</code></pre>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>文件</th><th>职责</th><th>当前状态</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td><code>run_hardware_contention_attribution.sh</code></td><td>唯一 Shell 入口；按 Run 分发通信代码、通信基线、服务启停、双节点采集器、Case 编排、门禁和 Trap 清理</td><td class="pass">已实现</td></tr>
|
||||||
|
<tr><td><code>config.env</code></td><td>Phase 1 相对路径、节点、代表 Case、分层采样周期、通信基线与 fail-closed 策略</td><td class="pass">已实现</td></tr>
|
||||||
|
<tr><td><code>communication_baseline.py</code></td><td>CUDA P2P 全矩阵与 PyTorch/NCCL 8/16-rank AllReduce 微基准</td><td class="pass">已实现</td></tr>
|
||||||
|
<tr><td><code>hardware_contention_attribution.py</code></td><td>精确窗口、全部采集器解析、逐 Case 汇总、通信聚合和逐指标报告</td><td class="pass">已实现</td></tr>
|
||||||
|
<tr><td><code>tests/test_hardware_contention_attribution.py</code></td><td>Worker 无仓库依赖、GPU/RDMA、精确窗口、DCGM/CPU 解析、通信聚合和结果生成测试</td><td class="pass">9/9 通过</td></tr>
|
||||||
|
<tr><td><code>README.md</code></td><td>唯一入口、范围和结果目录说明</td><td class="pass">已实现</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p class="decision">
|
||||||
|
Phase 2 不复制双机 Docker 启停实现。唯一入口在内部调用 Phase 1 的
|
||||||
|
<code>run_quick_map.sh start/fixed/mixed/stop</code>，只新增通信基线、硬件采集、时间对齐和代表负载编排。
|
||||||
|
顶层仍只保留一个 Shell 文件，不创建额外 tmux、launch、start 或 stop 脚本。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>7. 结果结构</h2>
|
||||||
|
<pre><code>results/<RUN_ID>/
|
||||||
|
manifest.json
|
||||||
|
run.log
|
||||||
|
bench/
|
||||||
|
bench_cmd.txt
|
||||||
|
bench.log
|
||||||
|
bench.jsonl
|
||||||
|
service/
|
||||||
|
head_server_cmd.txt
|
||||||
|
worker_server_cmd.txt
|
||||||
|
head_server.log
|
||||||
|
worker_server.log
|
||||||
|
communication/
|
||||||
|
communication_baseline.py
|
||||||
|
communication_baseline.sha256
|
||||||
|
p2p_head.log
|
||||||
|
p2p_worker.log
|
||||||
|
allreduce_head_8gpu.log
|
||||||
|
allreduce_worker_8gpu.log
|
||||||
|
allreduce_two_node_x0.log
|
||||||
|
allreduce_two_node_x1.log
|
||||||
|
allreduce_two_node_x2.log
|
||||||
|
head/
|
||||||
|
gpu_samples.csv
|
||||||
|
dcgm_dmon.log
|
||||||
|
mpstat.log
|
||||||
|
pidstat.log
|
||||||
|
sar_net.log
|
||||||
|
perf_stat.log
|
||||||
|
docker_top.log
|
||||||
|
numa_samples.csv
|
||||||
|
rdma.csv
|
||||||
|
static_before.log
|
||||||
|
static_after.log
|
||||||
|
collector_commands/
|
||||||
|
worker/
|
||||||
|
...
|
||||||
|
collector_status.csv
|
||||||
|
markers.csv
|
||||||
|
bench_summary.csv
|
||||||
|
gpu_summary.csv
|
||||||
|
rdma_summary.csv
|
||||||
|
case_windows.csv
|
||||||
|
communication_summary.csv
|
||||||
|
communication_aggregate.csv
|
||||||
|
case_gpu_summary.csv
|
||||||
|
case_gpu_node_summary.csv
|
||||||
|
case_dcgm_summary.csv
|
||||||
|
case_cpu_summary.csv
|
||||||
|
case_process_summary.csv
|
||||||
|
case_perf_summary.csv
|
||||||
|
case_numa_summary.csv
|
||||||
|
case_netdev_summary.csv
|
||||||
|
case_rdma_summary.csv
|
||||||
|
summary.json
|
||||||
|
report.md</code></pre>
|
||||||
|
|
||||||
|
<h2>8. 最终验收</h2>
|
||||||
|
<ul>
|
||||||
|
<li class="pass">最终 Run <code>dsv4pro-phase2-20260731-163620</code> 状态为 <code>COMPLETED</code>，8/8 benchmark 成功，0 失败、0 OOM。</li>
|
||||||
|
<li class="pass">8/8 Case 使用正式 benchmark 精确时间窗；18 个采集器全部记录 <code>STARTED</code> 与 <code>STOPPED</code>。</li>
|
||||||
|
<li class="pass">Head 与 Worker 的 DCGM、GPU、CPU、进程、NUMA、网卡和 HCA Counter 均有有效样本。</li>
|
||||||
|
<li class="pass">P2P、8-rank 和 16-rank AllReduce 全部完成，所有正确性检查均为 <code>wrong_values=0</code>。</li>
|
||||||
|
<li class="pass">Run 结束后 Head/Worker 无相关容器、无 GPU 计算进程，端口 <code>30002/30003</code> 已释放。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2>9. 正式运行</h2>
|
||||||
|
<p class="decision">
|
||||||
|
<strong>Run：<code>dsv4pro-phase2-20260731-163620</code>。</strong>
|
||||||
|
运行时间为 16:36:20 至 17:05:04 CST，总用时 28 分 44 秒。
|
||||||
|
Manifest 记录代码提交 <code>5f24b7d22f98108f6cc234edba6768d55ea0a962</code>，
|
||||||
|
<code>git_dirty=false</code>。
|
||||||
|
</p>
|
||||||
|
<pre><code class="language-bash"># 仅在 Head 174.1.51.5 执行
|
||||||
|
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
|
||||||
|
|
||||||
|
RUN_ID=dsv4pro-phase2-20260731-163620
|
||||||
|
tmux new-session -d -s dsv4pro-phase2 \
|
||||||
|
"RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
|
||||||
|
2>&1 | tee /data/hzy/${RUN_ID}.log"</code></pre>
|
||||||
|
|
||||||
|
<h2>10. 端到端结果</h2>
|
||||||
|
<h3>10.1 五个代表负载</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>Case</th><th>Input TPS</th><th>Output TPS</th><th>TTFT P95</th><th>TPOT P95</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>128K → 1, C=1</td><td>2,641.38</td><td>0.02</td><td>49.610 s</td><td>—</td></tr>
|
||||||
|
<tr><td>32K → 1, C=16</td><td>3,115.45</td><td>0.10</td><td>161.938 s</td><td>—</td></tr>
|
||||||
|
<tr><td>1K → 1K, C=32</td><td>448.95</td><td>448.95</td><td>10.167 s</td><td>65.36 ms</td></tr>
|
||||||
|
<tr><td>1K → 4K, C=16</td><td>79.39</td><td>317.56</td><td>1.724 s</td><td>49.98 ms</td></tr>
|
||||||
|
<tr><td>128K → 1K, C=1</td><td>1,614.99</td><td>12.62</td><td>48.279 s</td><td>32.11 ms</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
|
||||||
|
<h3>10.2 混合 Prefill/Decode A/B</h3>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>Decode 指标</th><th>Control</th><th>注入 128K Prefill</th><th>变化</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>Output TPS</td><td>453.55</td><td>344.89</td><td>-23.96%</td></tr>
|
||||||
|
<tr><td>TTFT P95</td><td>9.443 s</td><td>9.892 s</td><td>+4.76%</td></tr>
|
||||||
|
<tr><td>TPOT P95</td><td>66.24 ms</td><td>110.45 ms</td><td>+66.75%</td></tr>
|
||||||
|
<tr><td>E2E P95</td><td>72.270 s</td><td>118.076 s</td><td>+63.38%</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
这是 Phase 2 最关键的现象：长 Prefill 与 Decode 共存时，首 token 延迟只增加
|
||||||
|
4.76%，但 Decode 单 token 成本增加 66.75%，最终令 Output TPS 下降 23.96%。
|
||||||
|
问题主要发生在持续 Decode 阶段，而不是只表现为 Prefill 请求排队。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>11. 第 5 节指标逐项结果</h2>
|
||||||
|
<h3>11.1 GPU 基础状态与 DCGM</h3>
|
||||||
|
<p><strong>服务器证据路径：</strong><br>
|
||||||
|
汇总：<code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/case_gpu_summary.csv</code>、
|
||||||
|
<code>case_gpu_node_summary.csv</code>、<code>case_dcgm_summary.csv</code><br>
|
||||||
|
原始：同一 Run 目录下的 <code>head/gpu_samples.csv</code>、<code>worker/gpu_samples.csv</code>、
|
||||||
|
<code>head/dcgm_dmon.log</code>、<code>worker/dcgm_dmon.log</code>
|
||||||
|
</p>
|
||||||
|
<ul>
|
||||||
|
<li>各 Case GPU Util Mean 大多为 94%–99%，P95 为 100%；SM Clock 约 2.39–2.42 GHz，未见降频。</li>
|
||||||
|
<li>每卡显存稳定在约 83,000–83,364 MiB。Decode 功耗约 216–258 W，Prefill 功耗约 293–307 W。</li>
|
||||||
|
<li>普通 Decode 的 SM Active 约 0.523–0.525、DRAM Active 约 0.415–0.417；128K Prefill 的 SM Active 升至 0.683–0.686。</li>
|
||||||
|
<li>32K C16 Prefill 的 SM Active 约 0.713–0.715、DRAM Active 约 0.440，是本轮最重的并发 Prefill 计算负载。</li>
|
||||||
|
<li>混合 Treatment 中，Decode 背景 SM Active 约 0.578–0.581；注入 Prefill 窗口升至 0.720–0.723，证明两类工作确实争用同一 GPU 执行资源。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h3>11.2 CPU、进程与 NUMA</h3>
|
||||||
|
<p><strong>服务器证据路径：</strong><br>
|
||||||
|
汇总：<code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/case_cpu_summary.csv</code>、
|
||||||
|
<code>case_process_summary.csv</code>、<code>case_perf_summary.csv</code>、<code>case_numa_summary.csv</code><br>
|
||||||
|
原始：同一 Run 目录下 Head/Worker 各自的 <code>mpstat.log</code>、<code>pidstat.log</code>、
|
||||||
|
<code>perf_stat.log</code>、<code>numa_samples.csv</code> 和 <code>docker_top.log</code>
|
||||||
|
</p>
|
||||||
|
<ul>
|
||||||
|
<li>整机 CPU Active Mean 约 9.6%–10.4%，P95 约 10%–11.4%；没有全机 CPU 饱和。</li>
|
||||||
|
<li>服务进程峰值约 1,210%–1,226%，相当于约 12 个 CPU Core；热点 Core 数量最多 12–13 个。</li>
|
||||||
|
<li><code>perf</code> 观察到 IPC 约 2.7–3.1，未出现明显 Host 侧停摆。</li>
|
||||||
|
<li>Head/Worker 的 NUMA 不均衡约 21.1% / 13.2%，跨 Case 基本稳定；它是拓扑基线，但不像混合性能退化的直接诱因。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h3>11.3 双 Rail RDMA</h3>
|
||||||
|
<p><strong>服务器证据路径：</strong><br>
|
||||||
|
汇总：<code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/case_rdma_summary.csv</code>、
|
||||||
|
<code>rdma_summary.csv</code>、<code>case_netdev_summary.csv</code><br>
|
||||||
|
原始：同一 Run 目录下的 <code>head/rdma.csv</code>、<code>worker/rdma.csv</code>、
|
||||||
|
<code>head/sar_net.log</code>、<code>worker/sar_net.log</code>；端口/HCA 静态状态在两端
|
||||||
|
<code>static_before.log</code> 与 <code>static_after.log</code>
|
||||||
|
</p>
|
||||||
|
<ul>
|
||||||
|
<li>RoCE 绕过普通 Linux Socket 数据路径，因此 <code>sar</code> 的 <code>eth0/eth3</code> 流量接近 0；实际流量必须看 <code>mlx5_0/mlx5_3</code> HCA Counter。</li>
|
||||||
|
<li>普通 Decode 每 Rail 约 36.7–36.8 Gbit/s；128K Prefill 每 Rail约 70.5–71.5 Gbit/s。</li>
|
||||||
|
<li>最高点 32K C16 Prefill 每 Rail 约 83.0–83.45 Gbit/s，仅约占单条 400G Rail 的 20.9%。</li>
|
||||||
|
<li>两条 Rail 流量对称，<code>port_xmit_wait</code>、丢弃、错误和 Retry Exceeded 增量均为 0。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h3>11.4 PCIe 与 NCCL 通信基线</h3>
|
||||||
|
<p><strong>服务器证据路径：</strong><br>
|
||||||
|
汇总：<code>/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/communication_aggregate.csv</code>、
|
||||||
|
<code>communication_summary.csv</code><br>
|
||||||
|
原始：同一 Run 目录的 <code>communication/p2p_head.log</code>、<code>p2p_worker.log</code>、
|
||||||
|
<code>allreduce_head_8gpu.log</code>、<code>allreduce_worker_8gpu.log</code> 和
|
||||||
|
<code>allreduce_16gpu_crossnic{0,1,2}_{head,worker}.log</code>；实际执行命令在 <code>commands/communication_*.cmd.txt</code>
|
||||||
|
</p>
|
||||||
|
<table>
|
||||||
|
<thead><tr><th>测试</th><th>结果</th><th>解释</th></tr></thead>
|
||||||
|
<tbody>
|
||||||
|
<tr><td>Head PCIe P2P 256 MiB</td><td>同 Switch 53.61 GB/s；跨 NUMA 52.40 GB/s</td><td>跨 NUMA 损失约 2.3%</td></tr>
|
||||||
|
<tr><td>Worker PCIe P2P 256 MiB</td><td>同 Switch 53.50 GB/s；跨 NUMA 52.32 GB/s</td><td>两节点表现对称</td></tr>
|
||||||
|
<tr><td>Head / Worker 8-GPU AllReduce 1 GiB</td><td>busbw 39.76 / 39.75 GB/s</td><td>节点内基线一致</td></tr>
|
||||||
|
<tr><td>16-GPU AllReduce，CROSS_NIC=0</td><td>busbw 39.345 GB/s；51.169 ms</td><td>正确性 0 错误</td></tr>
|
||||||
|
<tr><td>16-GPU AllReduce，CROSS_NIC=1</td><td>busbw 39.685 GB/s；50.732 ms</td><td>本轮数值最好</td></tr>
|
||||||
|
<tr><td>16-GPU AllReduce，CROSS_NIC=2</td><td>busbw 39.530 GB/s；50.931 ms</td><td>正确性 0 错误</td></tr>
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
<p>
|
||||||
|
<code>NCCL_CROSS_NIC=1</code> 比 0 仅高 0.86%，比 2 仅高 0.39%。
|
||||||
|
差异小于 1%，不足以把它当成主要调优旋钮；保留当前值即可，Phase 3 不再重复测试。
|
||||||
|
NCCL 日志明确证明跨机路径使用 <code>mlx5_0,mlx5_3</code> 和
|
||||||
|
<code>NET/IB/.../GDRDMA</code>。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h2>12. 结论与 Phase 3 入口</h2>
|
||||||
|
<p class="decision">
|
||||||
|
<strong>Phase 2 已把范围明显缩小：</strong>混合 Prefill/Decode 退化真实且稳定，
|
||||||
|
但不是由整机 CPU 饱和、GPU 降频、双 Rail 原始带宽饱和、Rail 失衡、
|
||||||
|
PCIe 跨 NUMA 带宽崩塌或 <code>NCCL_CROSS_NIC</code> 选择造成。
|
||||||
|
Phase 3 应只捕获 Control 与 Treatment 的短时间线，定位 Attention/Indexer、
|
||||||
|
MoE、NCCL Collective、Scheduler gap 和慢 Rank 同步之间的串行与重叠关系。
|
||||||
|
</p>
|
||||||
|
<ul>
|
||||||
|
<li>不重复 Phase 2 的长时间 DCGM、CPU、RDMA 和通信微基准。</li>
|
||||||
|
<li>优先对比混合 Control 与注入 128K Prefill 的 Treatment。</li>
|
||||||
|
<li>再用单独 128K Prefill 作为 Kernel 对照，解释 SM Active 与 Tensor Active 的来源。</li>
|
||||||
|
</ul>
|
||||||
|
|
||||||
|
<h2>13. 证据与清理说明</h2>
|
||||||
|
<ul>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/report.md">自动生成逐指标报告</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/analysis.md">阶段归因摘要</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/bench_summary.csv">端到端 benchmark 汇总</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/communication_aggregate.csv">通信微基准汇总</a></li>
|
||||||
|
<li><a href="./results/dsv4pro-phase2-20260731-163620/service/head_server_cmd.txt">Head 实际服务命令</a> /
|
||||||
|
<a href="./results/dsv4pro-phase2-20260731-163620/service/worker_server_cmd.txt">Worker 实际服务命令</a></li>
|
||||||
|
</ul>
|
||||||
|
<p>
|
||||||
|
Worker 日志在所有 benchmark 完成后的编排关闭阶段出现 Gloo
|
||||||
|
<code>Connection closed by peer</code>；时间与 Head 主动退出进程组一致，
|
||||||
|
未影响 8/8 结果。NCCL 日志中的可选 mlx5 symbol 探测提示同样未影响
|
||||||
|
Collective，全部正确性检查为 0 错误。
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<p><a class="back" href="./phase1_exp.html">返回 Phase 1 实验档案</a></p>
|
||||||
|
<p><a class="back" href="./phase2_5_exp.html">继续 Phase 2.5 RDMA 需求建模</a></p>
|
||||||
|
<p><a class="back" href="./推理优化计划.html">返回推理优化主计划</a></p>
|
||||||
|
</main>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@ -0,0 +1,13 @@
|
|||||||
|
suite,case_id,role,stage,isl,osl,concurrency,planned_repetitions,completed_repetitions,statuses,total_tps_mean,total_tps_cv_pct,output_tps_mean,output_tps_cv_pct,ttft_p95_mean_ms,tpot_p95_mean_ms,itl_p95_mean_ms,e2e_p95_mean_ms
|
||||||
|
fixed,balanced_32k_to_1k_c8,,balanced,32768,1024,8,1,1,COMPLETED,2101.686722481554,,63.68747643883498,,82083.64989476977,109.32309288714016,43.84165157971437,128588.8987001963
|
||||||
|
fixed,decode_latency_1k_to_1k_c1,,decode_latency,1024,1024,1,1,1,COMPLETED,62.82990203738489,,31.414951018692445,,363.2723209448159,31.468245639333286,31.659404194215313,32555.28760998277
|
||||||
|
fixed,decode_throughput_1k_to_1k_c16,,decode_throughput,1024,1024,16,1,1,COMPLETED,590.5818337765435,,295.29091688827174,,4950.267374995747,50.01878498413362,49.8873026604997,55443.82167501317
|
||||||
|
fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1024,1024,32,1,1,COMPLETED,923.3692532200665,,461.68462661003326,,8021.852347906679,63.30661669023356,62.237933481810614,70933.18627287517
|
||||||
|
fixed,decode_throughput_1k_to_1k_c64,,decode_throughput,1024,1024,64,1,1,COMPLETED,1294.833542313111,,647.4167711565555,,12716.491229846724,93.44459645826957,87.80462378927041,101163.1970004586
|
||||||
|
fixed,long_prefill_latency_128k_c1,,prefill_latency,131072,1,1,1,1,COMPLETED,2710.1829473156417,,0.020676897204730506,,48344.461318978574,0.0,0.0,48344.52949295519
|
||||||
|
fixed,mid_prefill_latency_32k_c1,,prefill_latency,32768,1,1,1,1,COMPLETED,2652.8390294198,,0.08095575175988892,,12334.68782599084,0.0,0.0,12334.764264000114
|
||||||
|
fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,32768,1,16,1,1,COMPLETED,3112.8645802960464,,0.09499418902914482,,162087.46085499297,0.0,0.0,162087.50443853205
|
||||||
|
fixed,short_prefill_latency_1k_c1,,prefill_latency,1024,1,1,1,1,COMPLETED,1971.5883577957954,,1.9235008368739468,,501.89953204244375,0.0,0.0,501.94522901438177
|
||||||
|
mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1024,1024,32,1,1,COMPLETED,911.3699336194111,,455.68496680970554,,9442.636363586644,65.88327712923635,61.753195300116204,72007.63789927005
|
||||||
|
mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1024,1024,32,1,1,COMPLETED,691.8932985710949,,345.94664928554744,,10193.896457596566,109.7299457727384,61.61956858995836,117629.63820034638
|
||||||
|
mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,131072,1,1,1,1,COMPLETED,2859.7533639992707,,0.021818020217735695,,45752.13837000774,0.0,0.0,45752.21362197772
|
||||||
|
@ -0,0 +1,31 @@
|
|||||||
|
# DeepSeek-V4-Pro Pro6000D Two-Node SGLang Quick Map
|
||||||
|
|
||||||
|
Profiler: disabled. Speculative decoding: disabled.
|
||||||
|
|
||||||
|
## Aggregate results
|
||||||
|
|
||||||
|
| Case | Suite / role | Stage | ISL | OSL | C | Reps | Total TPS | CV | Output TPS | TTFT P95 | TPOT P95 | E2E P95 | Status |
|
||||||
|
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||||
|
| balanced_32k_to_1k_c8 | fixed / - | balanced | 32768 | 1024 | 8 | 1/1 | 2101.69 | -% | 63.69 | 82083.65 ms | 109.32 ms | 128588.90 ms | COMPLETED |
|
||||||
|
| decode_latency_1k_to_1k_c1 | fixed / - | decode_latency | 1024 | 1024 | 1 | 1/1 | 62.83 | -% | 31.41 | 363.27 ms | 31.47 ms | 32555.29 ms | COMPLETED |
|
||||||
|
| decode_throughput_1k_to_1k_c16 | fixed / - | decode_throughput | 1024 | 1024 | 16 | 1/1 | 590.58 | -% | 295.29 | 4950.27 ms | 50.02 ms | 55443.82 ms | COMPLETED |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | fixed / - | decode_throughput | 1024 | 1024 | 32 | 1/1 | 923.37 | -% | 461.68 | 8021.85 ms | 63.31 ms | 70933.19 ms | COMPLETED |
|
||||||
|
| decode_throughput_1k_to_1k_c64 | fixed / - | decode_throughput | 1024 | 1024 | 64 | 1/1 | 1294.83 | -% | 647.42 | 12716.49 ms | 93.44 ms | 101163.20 ms | COMPLETED |
|
||||||
|
| long_prefill_latency_128k_c1 | fixed / - | prefill_latency | 131072 | 1 | 1 | 1/1 | 2710.18 | -% | 0.02 | 48344.46 ms | 0.00 ms | 48344.53 ms | COMPLETED |
|
||||||
|
| mid_prefill_latency_32k_c1 | fixed / - | prefill_latency | 32768 | 1 | 1 | 1/1 | 2652.84 | -% | 0.08 | 12334.69 ms | 0.00 ms | 12334.76 ms | COMPLETED |
|
||||||
|
| mid_prefill_throughput_32k_c16 | fixed / - | prefill_throughput | 32768 | 1 | 16 | 1/1 | 3112.86 | -% | 0.09 | 162087.46 ms | 0.00 ms | 162087.50 ms | COMPLETED |
|
||||||
|
| short_prefill_latency_1k_c1 | fixed / - | prefill_latency | 1024 | 1 | 1 | 1/1 | 1971.59 | -% | 1.92 | 501.90 ms | 0.00 ms | 501.95 ms | COMPLETED |
|
||||||
|
| decode_control_1k_to_1k_c32 | mixed / control | mixed_interference | 1024 | 1024 | 32 | 1/1 | 911.37 | -% | 455.68 | 9442.64 ms | 65.88 ms | 72007.64 ms | COMPLETED |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | mixed / decode_background | mixed_interference | 1024 | 1024 | 32 | 1/1 | 691.89 | -% | 345.95 | 10193.90 ms | 109.73 ms | 117629.64 ms | COMPLETED |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | mixed / prefill_injection | mixed_interference | 131072 | 1 | 1 | 1/1 | 2859.75 | -% | 0.02 | 45752.14 ms | 0.00 ms | 45752.21 ms | COMPLETED |
|
||||||
|
|
||||||
|
## Mixed-interference A/B
|
||||||
|
|
||||||
|
| Metric | Control | With 128K prefill | Change |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| Output TPS | 455.68 | 345.95 | -24.08% |
|
||||||
|
| TTFT P95 (ms) | 9442.64 | 10193.90 | 7.96% |
|
||||||
|
| TPOT P95 (ms) | 65.88 | 109.73 | 66.55% |
|
||||||
|
| E2E P95 (ms) | 72007.64 | 117629.64 | 63.36% |
|
||||||
|
|
||||||
|
The fixed map is descriptive and never stops on SLO. With one repetition, CV is intentionally unavailable.
|
||||||
@ -0,0 +1,53 @@
|
|||||||
|
[2026-07-30 22:09:49] Starting worker node=10.101.0.13 rank=1 container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker
|
||||||
|
[2026-07-30 22:09:55] Starting head node=10.101.0.11 rank=0 container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head
|
||||||
|
[2026-07-30 22:09:56] Waiting for SGLang health at 10.101.0.11:30002
|
||||||
|
[2026-07-30 22:10:55] Still starting: checks=12/600
|
||||||
|
[2026-07-30 22:12:00] Still starting: checks=24/600
|
||||||
|
[2026-07-30 22:13:05] Still starting: checks=36/600
|
||||||
|
[2026-07-30 22:14:09] Still starting: checks=48/600
|
||||||
|
[2026-07-30 22:15:14] Still starting: checks=60/600
|
||||||
|
[2026-07-30 22:15:20] SGLang is healthy after 61 checks
|
||||||
|
[2026-07-30 22:15:20] Verified worker NCCL transport: NET/IB with mlx5_0 and mlx5_3
|
||||||
|
[2026-07-30 22:15:21] Verified head NCCL transport: NET/IB with mlx5_0 and mlx5_3
|
||||||
|
[2026-07-30 22:15:21] Validating scenario file: /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv
|
||||||
|
short_prefill_latency_1k_c1: stage=prefill_latency isl=1024 osl=1 c=1 prompts=1 warmup=1
|
||||||
|
mid_prefill_latency_32k_c1: stage=prefill_latency isl=32768 osl=1 c=1 prompts=1 warmup=0
|
||||||
|
long_prefill_latency_128k_c1: stage=prefill_latency isl=131072 osl=1 c=1 prompts=1 warmup=0
|
||||||
|
mid_prefill_throughput_32k_c16: stage=prefill_throughput isl=32768 osl=1 c=16 prompts=16 warmup=0
|
||||||
|
decode_latency_1k_to_1k_c1: stage=decode_latency isl=1024 osl=1024 c=1 prompts=1 warmup=1
|
||||||
|
decode_throughput_1k_to_1k_c16: stage=decode_throughput isl=1024 osl=1024 c=16 prompts=16 warmup=1
|
||||||
|
decode_throughput_1k_to_1k_c32: stage=decode_throughput isl=1024 osl=1024 c=32 prompts=32 warmup=1
|
||||||
|
decode_throughput_1k_to_1k_c64: stage=decode_throughput isl=1024 osl=1024 c=64 prompts=64 warmup=1
|
||||||
|
balanced_32k_to_1k_c8: stage=balanced isl=32768 osl=1024 c=8 prompts=8 warmup=0
|
||||||
|
[2026-07-30 22:15:21] START case=short_prefill_latency_1k_c1 rep=1 isl=1024 osl=1 c=1
|
||||||
|
[2026-07-30 22:15:55] DONE case=short_prefill_latency_1k_c1 rep=1 elapsed=34s
|
||||||
|
[2026-07-30 22:16:00] START case=mid_prefill_latency_32k_c1 rep=1 isl=32768 osl=1 c=1
|
||||||
|
[2026-07-30 22:16:46] DONE case=mid_prefill_latency_32k_c1 rep=1 elapsed=46s
|
||||||
|
[2026-07-30 22:16:51] START case=long_prefill_latency_128k_c1 rep=1 isl=131072 osl=1 c=1
|
||||||
|
[2026-07-30 22:18:13] DONE case=long_prefill_latency_128k_c1 rep=1 elapsed=81s
|
||||||
|
[2026-07-30 22:18:18] START case=mid_prefill_throughput_32k_c16 rep=1 isl=32768 osl=1 c=16
|
||||||
|
[2026-07-30 22:21:39] DONE case=mid_prefill_throughput_32k_c16 rep=1 elapsed=201s
|
||||||
|
[2026-07-30 22:21:44] START case=decode_latency_1k_to_1k_c1 rep=1 isl=1024 osl=1024 c=1
|
||||||
|
[2026-07-30 22:22:51] DONE case=decode_latency_1k_to_1k_c1 rep=1 elapsed=66s
|
||||||
|
[2026-07-30 22:22:56] START case=decode_throughput_1k_to_1k_c16 rep=1 isl=1024 osl=1024 c=16
|
||||||
|
[2026-07-30 22:24:25] DONE case=decode_throughput_1k_to_1k_c16 rep=1 elapsed=89s
|
||||||
|
[2026-07-30 22:24:30] START case=decode_throughput_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||||
|
[2026-07-30 22:26:15] DONE case=decode_throughput_1k_to_1k_c32 rep=1 elapsed=105s
|
||||||
|
[2026-07-30 22:26:20] START case=decode_throughput_1k_to_1k_c64 rep=1 isl=1024 osl=1024 c=64
|
||||||
|
[2026-07-30 22:28:36] DONE case=decode_throughput_1k_to_1k_c64 rep=1 elapsed=136s
|
||||||
|
[2026-07-30 22:28:41] START case=balanced_32k_to_1k_c8 rep=1 isl=32768 osl=1024 c=8
|
||||||
|
[2026-07-30 22:31:23] DONE case=balanced_32k_to_1k_c8 rep=1 elapsed=161s
|
||||||
|
[2026-07-30 22:31:28] Fixed quick map complete: completed=9/9 failed=0
|
||||||
|
[2026-07-30 22:31:28] START case=decode_control_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||||
|
[2026-07-30 22:34:26] DONE case=decode_control_1k_to_1k_c32 rep=1 elapsed=178s
|
||||||
|
[2026-07-30 22:34:26] START mixed background rep=1; injection delay=10s
|
||||||
|
[2026-07-30 22:34:26] START case=decode_with_128k_prefill_1k_to_1k_c32 rep=1 isl=1024 osl=1024 c=32
|
||||||
|
[2026-07-30 22:35:06] START case=long_prefill_injection_128k_to_1_c1 rep=1 isl=131072 osl=1 c=1
|
||||||
|
[2026-07-30 22:36:24] DONE case=long_prefill_injection_128k_to_1_c1 rep=1 elapsed=78s
|
||||||
|
[2026-07-30 22:38:09] DONE case=decode_with_128k_prefill_1k_to_1k_c32 rep=1 elapsed=223s
|
||||||
|
[2026-07-30 22:38:14] Mixed-interference A/B complete: failed_repetitions=0
|
||||||
|
[2026-07-30 22:38:20] Stopped head container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head on 10.101.0.11
|
||||||
|
[2026-07-30 22:38:22] Stopped worker container=dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker on 10.101.0.13
|
||||||
|
summarized 12 case runs into /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916
|
||||||
|
[2026-07-30 22:38:22] Quick performance map complete: status=COMPLETED
|
||||||
|
[2026-07-30 22:38:22] Results: /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916
|
||||||
@ -0,0 +1,43 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"workflow_stage": "quick_performance_map",
|
||||||
|
"run_id": "dsv4pro-phase1-full-20260730-220916",
|
||||||
|
"status": "COMPLETED",
|
||||||
|
"started_at": "2026-07-30T22:09:47+08:00",
|
||||||
|
"updated_at": "2026-07-30T22:38:22+08:00",
|
||||||
|
"suites": [
|
||||||
|
"fixed",
|
||||||
|
"mixed"
|
||||||
|
],
|
||||||
|
"engine": "sglang",
|
||||||
|
"model_name": "DeepSeek-V4-Pro",
|
||||||
|
"model_path": "/data/hf_models/DeepSeek-V4-Pro",
|
||||||
|
"docker_image": "lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45",
|
||||||
|
"head_node": "10.101.0.11",
|
||||||
|
"worker_node": "10.101.0.13",
|
||||||
|
"head_ip": "10.101.0.11",
|
||||||
|
"sglang_port": 30002,
|
||||||
|
"dist_init_port": 20002,
|
||||||
|
"tp_size": 16,
|
||||||
|
"ep_size": 2,
|
||||||
|
"nnodes": 2,
|
||||||
|
"mem_fraction_static": 0.9,
|
||||||
|
"cuda_graph_max_bs_decode": 64,
|
||||||
|
"max_running_requests": 256,
|
||||||
|
"nccl_socket_ifname": "eth0",
|
||||||
|
"nccl_ib_hca": "=mlx5_0:1,mlx5_3:1",
|
||||||
|
"nccl_cross_nic": "1",
|
||||||
|
"enable_rdma": true,
|
||||||
|
"require_nccl_ib": true,
|
||||||
|
"rdma_device_paths": "/dev/infiniband/rdma_cm,/dev/infiniband/uverbs0,/dev/infiniband/uverbs3",
|
||||||
|
"git_commit": "75182c6ededf94511c6678552aaf489b9fc6fba6",
|
||||||
|
"git_dirty": false,
|
||||||
|
"scenario_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv",
|
||||||
|
"case_ids": "",
|
||||||
|
"notes": [
|
||||||
|
"The fixed quick map does not stop on SLO.",
|
||||||
|
"Profiler is disabled; these results are eligible for performance comparison.",
|
||||||
|
"Speculative decoding is not enabled."
|
||||||
|
],
|
||||||
|
"ended_at": "2026-07-30T22:38:22+08:00"
|
||||||
|
}
|
||||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
@ -0,0 +1,13 @@
|
|||||||
|
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,balanced_32k_to_1k_c8,,balanced,1,32768,1024,8,8,0,COMPLETED,,0,2026-07-30T22:28:41+0800,2026-07-30T22:31:22+0800,161.0,8,0,128.62811431800947,7.996993535425267,,262144,8192,0.06219480120979978,2037.9992460427193,63.68747643883498,2101.686722481554,,128579.77483438299,128578.26417451724,128588.8987001963,128589.40238882786,49665.467494261975,49563.69844998699,82083.64989476977,83746.94262376754,77.14008537646238,77.23808966229743,109.32309288714016,112.17132956471158,77.14005810044779,43.40646349010058,43.84165157971437,45.188912986195646,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1,1024,1024,32,64,1,COMPLETED,,0,2026-07-30T22:31:28+0800,2026-07-30T22:34:26+0800,178.0,64,0,143.81865712802391,31.98259215252821,,65536,65536,0.44500485040010307,455.68496680970554,455.68496680970554,911.3699336194111,,71870.21023202942,71867.41446750239,72007.63789927005,72012.73389439622,7370.452653370194,6938.839260517852,9442.636363586644,9443.245556704933,63.04961640142642,63.471088219448355,65.88327712923635,67.80910156527888,63.049588087913975,60.98580302204937,61.753195300116204,63.56921844591854,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,decode_latency_1k_to_1k_c1,,decode_latency,1,1024,1024,1,1,1,COMPLETED,,0,2026-07-30T22:21:44+0800,2026-07-30T22:22:50+0800,66.0,1,0,32.59594450396253,0.9987527008467322,,1024,1024,0.03067866310419184,31.414951018692445,31.414951018692445,62.82990203738489,,32555.28760998277,32555.28760998277,32555.28760998277,32555.28760998277,363.2723209448159,363.2723209448159,363.2723209448159,363.2723209448159,31.468245639333286,31.468245639333286,31.468245639333286,31.468245639333286,31.468210095789587,31.49253799347207,31.659404194215313,32.3411487112753,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c16,,decode_throughput,1,1024,1024,16,16,1,COMPLETED,,0,2026-07-30T22:22:56+0800,2026-07-30T22:24:25+0800,89.0,16,0,55.48426674498478,15.986115557256861,,16384,16384,0.28837003602370287,295.29091688827174,295.29091688827174,590.5818337765435,,55436.11873718692,55437.53465998452,55443.82167501317,55447.12010782387,4706.02819519263,4801.126986538293,4950.267374995747,4950.924708603998,49.58953132159755,49.49641186020903,50.01878498413362,50.915034893211796,49.58950283761395,49.32670452399179,49.8873026604997,52.25673661392648,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1,1024,1024,32,32,1,COMPLETED,,0,2026-07-30T22:24:30+0800,2026-07-30T22:26:15+0800,105.0,32,0,70.97485623596003,31.975931620502816,,32768,32768,0.4508638931738606,461.68462661003326,461.68462661003326,923.3692532200665,,70921.47343050237,70921.31606096518,70933.18627287517,70935.98751158977,6650.527383468216,6178.440199000761,8021.852347906679,8023.242657405207,62.825949215087164,63.29258629570958,63.30661669023356,65.29997366317116,62.82592070965087,61.42424049903639,62.237933481810614,64.6031521842815,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,decode_throughput_1k_to_1k_c64,,decode_throughput,1,1024,1024,64,64,1,COMPLETED,,0,2026-07-30T22:26:20+0800,2026-07-30T22:28:36+0800,136.0,64,0,101.22691119497176,63.94603747793439,,65536,65536,0.6322429405825737,647.4167711565555,647.4167711565555,1294.833542313111,,101141.56026639376,101148.19569600513,101163.1970004586,101167.44335912867,7978.397482784203,6505.179281026358,12716.491229846724,12719.49685954547,91.06858532122146,92.52174521165054,93.44459645826957,94.39186767905878,91.06855870121748,86.24391999910586,87.80462378927041,90.7129056059057,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,1024,1024,32,64,0,COMPLETED,,0,2026-07-30T22:34:26+0800,2026-07-30T22:38:09+0800,223.0,64,0,189.43961485201726,31.98654958092739,,65536,65536,0.3378385246929174,345.94664928554744,345.94664928554744,691.8932985710949,,94679.9942664975,94677.34680901049,117629.63820034638,117636.8834006862,7749.917391879535,7705.110649490962,10193.896457596566,10196.455221220385,84.97563721859039,86.47172644327328,109.7299457727384,110.56891953740994,84.97560842059728,60.77706499490887,61.61956858995836,64.70431641500909,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,131072,1,1,1,0,COMPLETED,,0,2026-07-30T22:35:06+0800,2026-07-30T22:36:24+0800,78.0,1,0,45.83367280900711,0.9982227218104723,,131072,1,0.021818020217735695,2859.731545979053,0.021818020217735695,2859.7533639992707,,45752.21362197772,45752.21362197772,45752.21362197772,45752.21362197772,45752.13837000774,45752.13837000774,45752.13837000774,45752.13837000774,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,long_prefill_latency_128k_c1,,prefill_latency,1,131072,1,1,1,0,COMPLETED,,0,2026-07-30T22:16:51+0800,2026-07-30T22:18:12+0800,81.0,1,0,48.363155752944294,0.9996148667368967,,131072,1,0.020676897204730506,2710.162270418437,0.020676897204730506,2710.1829473156417,,48344.52949295519,48344.52949295519,48344.52949295519,48344.52949295519,48344.461318978574,48344.461318978574,48344.461318978574,48344.461318978574,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,mid_prefill_latency_32k_c1,,prefill_latency,1,32768,1,1,1,0,COMPLETED,,0,2026-07-30T22:16:00+0800,2026-07-30T22:16:46+0800,46.0,1,0,12.352426828991156,0.9985701137731423,,32768,1,0.08095575175988892,2652.75807366804,0.08095575175988892,2652.8390294198,,12334.764264000114,12334.764264000114,12334.764264000114,12334.764264000114,12334.68782599084,12334.68782599084,12334.68782599084,12334.68782599084,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,1,32768,1,16,16,0,COMPLETED,,0,2026-07-30T22:18:18+0800,2026-07-30T22:21:39+0800,201.0,16,0,168.4313552599633,8.771898294558559,,524288,16,0.09499418902914482,3112.7695861070174,0.09499418902914482,3112.8645802960464,,92341.41987219118,92749.04042950948,162087.50443853205,167117.798359727,92341.37187193119,92748.99350100895,162087.46085499297,167117.75140058598,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.log
|
||||||
|
dsv4pro-phase1-full-20260730-220916,fixed,short_prefill_latency_1k_c1,,prefill_latency,1,1024,1,1,1,1,COMPLETED,,0,2026-07-30T22:15:21+0800,2026-07-30T22:15:55+0800,34.0,1,0,0.5198853989713825,0.9654920680740482,,1024,1,1.9235008368739468,1969.6648569589215,1.9235008368739468,1971.5883577957954,,501.94522901438177,501.94522901438177,501.94522901438177,501.94522901438177,501.89953204244375,501.89953204244375,501.89953204244375,501.89953204244375,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.log
|
||||||
|
@ -0,0 +1,12 @@
|
|||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "balanced_32k_to_1k_c8", "role": "", "stage": "balanced", "repetition": 1, "isl": 32768, "osl": 1024, "concurrency": 8, "num_prompts": 8, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:28:41+0800", "ended_at": "2026-07-30T22:31:22+0800", "elapsed_s": 161.0, "completed": 8, "failed": 0, "duration_s": 128.62811431800947, "actual_concurrency": 7.996993535425267, "peak_concurrent_requests": null, "total_input_tokens": 262144, "total_output_tokens": 8192, "request_throughput": 0.06219480120979978, "input_token_throughput": 2037.9992460427193, "output_token_throughput": 63.68747643883498, "total_token_throughput": 2101.686722481554, "peak_output_token_throughput": null, "e2e_mean_ms": 128579.77483438299, "e2e_p50_ms": 128578.26417451724, "e2e_p95_ms": 128588.8987001963, "e2e_p99_ms": 128589.40238882786, "ttft_mean_ms": 49665.467494261975, "ttft_p50_ms": 49563.69844998699, "ttft_p95_ms": 82083.64989476977, "ttft_p99_ms": 83746.94262376754, "tpot_mean_ms": 77.14008537646238, "tpot_p50_ms": 77.23808966229743, "tpot_p95_ms": 109.32309288714016, "tpot_p99_ms": 112.17132956471158, "itl_mean_ms": 77.14005810044779, "itl_p50_ms": 43.40646349010058, "itl_p95_ms": 43.84165157971437, "itl_p99_ms": 45.188912986195646, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/balanced_32k_to_1k_c8/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "decode_control_1k_to_1k_c32", "role": "control", "stage": "mixed_interference", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 64, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:31:28+0800", "ended_at": "2026-07-30T22:34:26+0800", "elapsed_s": 178.0, "completed": 64, "failed": 0, "duration_s": 143.81865712802391, "actual_concurrency": 31.98259215252821, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.44500485040010307, "input_token_throughput": 455.68496680970554, "output_token_throughput": 455.68496680970554, "total_token_throughput": 911.3699336194111, "peak_output_token_throughput": null, "e2e_mean_ms": 71870.21023202942, "e2e_p50_ms": 71867.41446750239, "e2e_p95_ms": 72007.63789927005, "e2e_p99_ms": 72012.73389439622, "ttft_mean_ms": 7370.452653370194, "ttft_p50_ms": 6938.839260517852, "ttft_p95_ms": 9442.636363586644, "ttft_p99_ms": 9443.245556704933, "tpot_mean_ms": 63.04961640142642, "tpot_p50_ms": 63.471088219448355, "tpot_p95_ms": 65.88327712923635, "tpot_p99_ms": 67.80910156527888, "itl_mean_ms": 63.049588087913975, "itl_p50_ms": 60.98580302204937, "itl_p95_ms": 61.753195300116204, "itl_p99_ms": 63.56921844591854, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_control_1k_to_1k_c32/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_latency_1k_to_1k_c1", "role": "", "stage": "decode_latency", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 1, "num_prompts": 1, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:21:44+0800", "ended_at": "2026-07-30T22:22:50+0800", "elapsed_s": 66.0, "completed": 1, "failed": 0, "duration_s": 32.59594450396253, "actual_concurrency": 0.9987527008467322, "peak_concurrent_requests": null, "total_input_tokens": 1024, "total_output_tokens": 1024, "request_throughput": 0.03067866310419184, "input_token_throughput": 31.414951018692445, "output_token_throughput": 31.414951018692445, "total_token_throughput": 62.82990203738489, "peak_output_token_throughput": null, "e2e_mean_ms": 32555.28760998277, "e2e_p50_ms": 32555.28760998277, "e2e_p95_ms": 32555.28760998277, "e2e_p99_ms": 32555.28760998277, "ttft_mean_ms": 363.2723209448159, "ttft_p50_ms": 363.2723209448159, "ttft_p95_ms": 363.2723209448159, "ttft_p99_ms": 363.2723209448159, "tpot_mean_ms": 31.468245639333286, "tpot_p50_ms": 31.468245639333286, "tpot_p95_ms": 31.468245639333286, "tpot_p99_ms": 31.468245639333286, "itl_mean_ms": 31.468210095789587, "itl_p50_ms": 31.49253799347207, "itl_p95_ms": 31.659404194215313, "itl_p99_ms": 32.3411487112753, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_latency_1k_to_1k_c1/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c16", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 16, "num_prompts": 16, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:22:56+0800", "ended_at": "2026-07-30T22:24:25+0800", "elapsed_s": 89.0, "completed": 16, "failed": 0, "duration_s": 55.48426674498478, "actual_concurrency": 15.986115557256861, "peak_concurrent_requests": null, "total_input_tokens": 16384, "total_output_tokens": 16384, "request_throughput": 0.28837003602370287, "input_token_throughput": 295.29091688827174, "output_token_throughput": 295.29091688827174, "total_token_throughput": 590.5818337765435, "peak_output_token_throughput": null, "e2e_mean_ms": 55436.11873718692, "e2e_p50_ms": 55437.53465998452, "e2e_p95_ms": 55443.82167501317, "e2e_p99_ms": 55447.12010782387, "ttft_mean_ms": 4706.02819519263, "ttft_p50_ms": 4801.126986538293, "ttft_p95_ms": 4950.267374995747, "ttft_p99_ms": 4950.924708603998, "tpot_mean_ms": 49.58953132159755, "tpot_p50_ms": 49.49641186020903, "tpot_p95_ms": 50.01878498413362, "tpot_p99_ms": 50.915034893211796, "itl_mean_ms": 49.58950283761395, "itl_p50_ms": 49.32670452399179, "itl_p95_ms": 49.8873026604997, "itl_p99_ms": 52.25673661392648, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c16/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c32", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 32, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:24:30+0800", "ended_at": "2026-07-30T22:26:15+0800", "elapsed_s": 105.0, "completed": 32, "failed": 0, "duration_s": 70.97485623596003, "actual_concurrency": 31.975931620502816, "peak_concurrent_requests": null, "total_input_tokens": 32768, "total_output_tokens": 32768, "request_throughput": 0.4508638931738606, "input_token_throughput": 461.68462661003326, "output_token_throughput": 461.68462661003326, "total_token_throughput": 923.3692532200665, "peak_output_token_throughput": null, "e2e_mean_ms": 70921.47343050237, "e2e_p50_ms": 70921.31606096518, "e2e_p95_ms": 70933.18627287517, "e2e_p99_ms": 70935.98751158977, "ttft_mean_ms": 6650.527383468216, "ttft_p50_ms": 6178.440199000761, "ttft_p95_ms": 8021.852347906679, "ttft_p99_ms": 8023.242657405207, "tpot_mean_ms": 62.825949215087164, "tpot_p50_ms": 63.29258629570958, "tpot_p95_ms": 63.30661669023356, "tpot_p99_ms": 65.29997366317116, "itl_mean_ms": 62.82592070965087, "itl_p50_ms": 61.42424049903639, "itl_p95_ms": 62.237933481810614, "itl_p99_ms": 64.6031521842815, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "decode_throughput_1k_to_1k_c64", "role": "", "stage": "decode_throughput", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 64, "num_prompts": 64, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:26:20+0800", "ended_at": "2026-07-30T22:28:36+0800", "elapsed_s": 136.0, "completed": 64, "failed": 0, "duration_s": 101.22691119497176, "actual_concurrency": 63.94603747793439, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.6322429405825737, "input_token_throughput": 647.4167711565555, "output_token_throughput": 647.4167711565555, "total_token_throughput": 1294.833542313111, "peak_output_token_throughput": null, "e2e_mean_ms": 101141.56026639376, "e2e_p50_ms": 101148.19569600513, "e2e_p95_ms": 101163.1970004586, "e2e_p99_ms": 101167.44335912867, "ttft_mean_ms": 7978.397482784203, "ttft_p50_ms": 6505.179281026358, "ttft_p95_ms": 12716.491229846724, "ttft_p99_ms": 12719.49685954547, "tpot_mean_ms": 91.06858532122146, "tpot_p50_ms": 92.52174521165054, "tpot_p95_ms": 93.44459645826957, "tpot_p99_ms": 94.39186767905878, "itl_mean_ms": 91.06855870121748, "itl_p50_ms": 86.24391999910586, "itl_p95_ms": 87.80462378927041, "itl_p99_ms": 90.7129056059057, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_throughput_1k_to_1k_c64/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "decode_with_128k_prefill_1k_to_1k_c32", "role": "decode_background", "stage": "mixed_interference", "repetition": 1, "isl": 1024, "osl": 1024, "concurrency": 32, "num_prompts": 64, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:34:26+0800", "ended_at": "2026-07-30T22:38:09+0800", "elapsed_s": 223.0, "completed": 64, "failed": 0, "duration_s": 189.43961485201726, "actual_concurrency": 31.98654958092739, "peak_concurrent_requests": null, "total_input_tokens": 65536, "total_output_tokens": 65536, "request_throughput": 0.3378385246929174, "input_token_throughput": 345.94664928554744, "output_token_throughput": 345.94664928554744, "total_token_throughput": 691.8932985710949, "peak_output_token_throughput": null, "e2e_mean_ms": 94679.9942664975, "e2e_p50_ms": 94677.34680901049, "e2e_p95_ms": 117629.63820034638, "e2e_p99_ms": 117636.8834006862, "ttft_mean_ms": 7749.917391879535, "ttft_p50_ms": 7705.110649490962, "ttft_p95_ms": 10193.896457596566, "ttft_p99_ms": 10196.455221220385, "tpot_mean_ms": 84.97563721859039, "tpot_p50_ms": 86.47172644327328, "tpot_p95_ms": 109.7299457727384, "tpot_p99_ms": 110.56891953740994, "itl_mean_ms": 84.97560842059728, "itl_p50_ms": 60.77706499490887, "itl_p95_ms": 61.61956858995836, "itl_p99_ms": 64.70431641500909, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "mixed", "case_id": "long_prefill_injection_128k_to_1_c1", "role": "prefill_injection", "stage": "mixed_interference", "repetition": 1, "isl": 131072, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:35:06+0800", "ended_at": "2026-07-30T22:36:24+0800", "elapsed_s": 78.0, "completed": 1, "failed": 0, "duration_s": 45.83367280900711, "actual_concurrency": 0.9982227218104723, "peak_concurrent_requests": null, "total_input_tokens": 131072, "total_output_tokens": 1, "request_throughput": 0.021818020217735695, "input_token_throughput": 2859.731545979053, "output_token_throughput": 0.021818020217735695, "total_token_throughput": 2859.7533639992707, "peak_output_token_throughput": null, "e2e_mean_ms": 45752.21362197772, "e2e_p50_ms": 45752.21362197772, "e2e_p95_ms": 45752.21362197772, "e2e_p99_ms": 45752.21362197772, "ttft_mean_ms": 45752.13837000774, "ttft_p50_ms": 45752.13837000774, "ttft_p95_ms": 45752.13837000774, "ttft_p99_ms": 45752.13837000774, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "long_prefill_latency_128k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 131072, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:16:51+0800", "ended_at": "2026-07-30T22:18:12+0800", "elapsed_s": 81.0, "completed": 1, "failed": 0, "duration_s": 48.363155752944294, "actual_concurrency": 0.9996148667368967, "peak_concurrent_requests": null, "total_input_tokens": 131072, "total_output_tokens": 1, "request_throughput": 0.020676897204730506, "input_token_throughput": 2710.162270418437, "output_token_throughput": 0.020676897204730506, "total_token_throughput": 2710.1829473156417, "peak_output_token_throughput": null, "e2e_mean_ms": 48344.52949295519, "e2e_p50_ms": 48344.52949295519, "e2e_p95_ms": 48344.52949295519, "e2e_p99_ms": 48344.52949295519, "ttft_mean_ms": 48344.461318978574, "ttft_p50_ms": 48344.461318978574, "ttft_p95_ms": 48344.461318978574, "ttft_p99_ms": 48344.461318978574, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/long_prefill_latency_128k_c1/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "mid_prefill_latency_32k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 32768, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:16:00+0800", "ended_at": "2026-07-30T22:16:46+0800", "elapsed_s": 46.0, "completed": 1, "failed": 0, "duration_s": 12.352426828991156, "actual_concurrency": 0.9985701137731423, "peak_concurrent_requests": null, "total_input_tokens": 32768, "total_output_tokens": 1, "request_throughput": 0.08095575175988892, "input_token_throughput": 2652.75807366804, "output_token_throughput": 0.08095575175988892, "total_token_throughput": 2652.8390294198, "peak_output_token_throughput": null, "e2e_mean_ms": 12334.764264000114, "e2e_p50_ms": 12334.764264000114, "e2e_p95_ms": 12334.764264000114, "e2e_p99_ms": 12334.764264000114, "ttft_mean_ms": 12334.68782599084, "ttft_p50_ms": 12334.68782599084, "ttft_p95_ms": 12334.68782599084, "ttft_p99_ms": 12334.68782599084, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_latency_32k_c1/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "mid_prefill_throughput_32k_c16", "role": "", "stage": "prefill_throughput", "repetition": 1, "isl": 32768, "osl": 1, "concurrency": 16, "num_prompts": 16, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:18:18+0800", "ended_at": "2026-07-30T22:21:39+0800", "elapsed_s": 201.0, "completed": 16, "failed": 0, "duration_s": 168.4313552599633, "actual_concurrency": 8.771898294558559, "peak_concurrent_requests": null, "total_input_tokens": 524288, "total_output_tokens": 16, "request_throughput": 0.09499418902914482, "input_token_throughput": 3112.7695861070174, "output_token_throughput": 0.09499418902914482, "total_token_throughput": 3112.8645802960464, "peak_output_token_throughput": null, "e2e_mean_ms": 92341.41987219118, "e2e_p50_ms": 92749.04042950948, "e2e_p95_ms": 162087.50443853205, "e2e_p99_ms": 167117.798359727, "ttft_mean_ms": 92341.37187193119, "ttft_p50_ms": 92748.99350100895, "ttft_p95_ms": 162087.46085499297, "ttft_p99_ms": 167117.75140058598, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/mid_prefill_throughput_32k_c16/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-phase1-full-20260730-220916", "suite": "fixed", "case_id": "short_prefill_latency_1k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 1024, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T22:15:21+0800", "ended_at": "2026-07-30T22:15:55+0800", "elapsed_s": 34.0, "completed": 1, "failed": 0, "duration_s": 0.5198853989713825, "actual_concurrency": 0.9654920680740482, "peak_concurrent_requests": null, "total_input_tokens": 1024, "total_output_tokens": 1, "request_throughput": 1.9235008368739468, "input_token_throughput": 1969.6648569589215, "output_token_throughput": 1.9235008368739468, "total_token_throughput": 1971.5883577957954, "peak_output_token_throughput": null, "e2e_mean_ms": 501.94522901438177, "e2e_p50_ms": 501.94522901438177, "e2e_p95_ms": 501.94522901438177, "e2e_p99_ms": 501.94522901438177, "ttft_mean_ms": 501.89953204244375, "ttft_p50_ms": 501.89953204244375, "ttft_p95_ms": 501.89953204244375, "ttft_p99_ms": 501.89953204244375, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-full-20260730-220916/cases/short_prefill_latency_1k_c1/rep1/bench.log"}
|
||||||
@ -0,0 +1,12 @@
|
|||||||
|
# DeepSeek-V4-Pro Pro6000D Two-Node SGLang Quick Map
|
||||||
|
|
||||||
|
Profiler: disabled. Speculative decoding: disabled.
|
||||||
|
|
||||||
|
## Aggregate results
|
||||||
|
|
||||||
|
| Case | Suite / role | Stage | ISL | OSL | C | Reps | Total TPS | CV | Output TPS | TTFT P95 | TPOT P95 | E2E P95 | Status |
|
||||||
|
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||||
|
| long_context_decode_128k_to_1k_c1 | fixed / - | long_context_decode | 131072 | 1024 | 1 | 1/1 | 1604.09 | -% | 12.43 | 49325.72 ms | 32.24 ms | 82312.21 ms | COMPLETED |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | fixed / - | long_output_decode | 1024 | 4096 | 16 | 1/1 | 387.52 | -% | 310.02 | 6241.01 ms | 50.33 ms | 211363.56 ms | COMPLETED |
|
||||||
|
|
||||||
|
The fixed map is descriptive and never stops on SLO. With one repetition, CV is intentionally unavailable.
|
||||||
@ -0,0 +1,42 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"workflow_stage": "quick_performance_map",
|
||||||
|
"run_id": "dsv4pro-phase1-long-decode-20260730-234236",
|
||||||
|
"status": "COMPLETED",
|
||||||
|
"started_at": "2026-07-30T23:48:13+08:00",
|
||||||
|
"updated_at": "2026-07-30T23:54:25+08:00",
|
||||||
|
"suites": [
|
||||||
|
"fixed"
|
||||||
|
],
|
||||||
|
"engine": "sglang",
|
||||||
|
"model_name": "DeepSeek-V4-Pro",
|
||||||
|
"model_path": "/data/hf_models/DeepSeek-V4-Pro",
|
||||||
|
"docker_image": "lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45",
|
||||||
|
"head_node": "10.101.0.11",
|
||||||
|
"worker_node": "10.101.0.13",
|
||||||
|
"head_ip": "10.101.0.11",
|
||||||
|
"sglang_port": 30002,
|
||||||
|
"dist_init_port": 20002,
|
||||||
|
"tp_size": 16,
|
||||||
|
"ep_size": 2,
|
||||||
|
"nnodes": 2,
|
||||||
|
"mem_fraction_static": 0.9,
|
||||||
|
"cuda_graph_max_bs_decode": 64,
|
||||||
|
"max_running_requests": 256,
|
||||||
|
"nccl_socket_ifname": "eth0",
|
||||||
|
"nccl_ib_hca": "=mlx5_0:1,mlx5_3:1",
|
||||||
|
"nccl_cross_nic": "1",
|
||||||
|
"enable_rdma": true,
|
||||||
|
"require_nccl_ib": true,
|
||||||
|
"rdma_device_paths": "/dev/infiniband/rdma_cm,/dev/infiniband/uverbs0,/dev/infiniband/uverbs3",
|
||||||
|
"git_commit": "06b017483cb1cfc6aace3c60e94576fb667ec9bc",
|
||||||
|
"git_dirty": false,
|
||||||
|
"scenario_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv",
|
||||||
|
"case_ids": "long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1",
|
||||||
|
"notes": [
|
||||||
|
"The fixed quick map does not stop on SLO.",
|
||||||
|
"Profiler is disabled; these results are eligible for performance comparison.",
|
||||||
|
"Speculative decoding is not enabled."
|
||||||
|
],
|
||||||
|
"ended_at": "2026-07-30T23:54:25+08:00"
|
||||||
|
}
|
||||||
@ -0,0 +1,3 @@
|
|||||||
|
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log
|
||||||
|
dsv4pro-phase1-long-decode-20260730-234236,fixed,long_context_decode_128k_to_1k_c1,,long_context_decode,1,131072,1024,1,1,0,COMPLETED,,0,2026-07-30T23:52:24+0800,2026-07-30T23:54:19+0800,115.0,1,0,82.34937581099803,0.9995487118435447,,131072,1024,0.012143382875118852,1591.6574802075781,12.434824064121704,1604.0923042716997,,82312.21251300303,82312.21251300303,82312.21251300303,82312.21251300303,49325.72139299009,49325.72139299009,49325.72139299009,49325.72139299009,32.244859354851364,32.244859354851364,32.244859354851364,32.244859354851364,32.2448224535841,32.23047900246456,32.468517863890156,32.707680857274674,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-long-decode-20260730-234236/cases/long_context_decode_128k_to_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-long-decode-20260730-234236/cases/long_context_decode_128k_to_1k_c1/rep1/bench.log
|
||||||
|
dsv4pro-phase1-long-decode-20260730-234236,fixed,long_output_decode_1k_to_4k_c16,,long_output_decode,1,1024,4096,16,16,1,COMPLETED,,0,2026-07-30T23:48:13+0800,2026-07-30T23:52:19+0800,246.0,16,0,211.3935806370573,15.996984262440824,,16384,65536,0.07568820184502423,77.50471868930481,310.01887475721924,387.52359344652405,,211353.73641450133,211353.21495501557,211363.55966723931,211364.98273107863,5862.642711690569,5970.386928500375,6241.009955512709,6241.542907894473,50.18097526320165,50.15488793663095,50.32745551037411,50.57979576238554,50.18097181628469,50.06324249552563,50.968476399430074,52.59070861677173,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-long-decode-20260730-234236/cases/long_output_decode_1k_to_4k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-phase1-long-decode-20260730-234236/cases/long_output_decode_1k_to_4k_c16/rep1/bench.log
|
||||||
|
@ -0,0 +1,138 @@
|
|||||||
|
# Phase 2 Hardware Attribution Analysis
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
- Run ID: `dsv4pro-phase2-20260731-130125`
|
||||||
|
- Status: `COMPLETED`
|
||||||
|
- Time: `2026-07-31 13:01:25` to `13:27:51 CST`
|
||||||
|
- Duration: 26 minutes 26 seconds
|
||||||
|
- Topology: two Pro6000D nodes, SGLang TP16 / EP2
|
||||||
|
- Bench result: 8 completed, 0 failed, no OOM
|
||||||
|
|
||||||
|
The command was run only on `174.1.51.5`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
|
||||||
|
|
||||||
|
RUN_ID=dsv4pro-phase2-20260731-130125
|
||||||
|
tmux new-session -d -s dsv4pro-phase2 \
|
||||||
|
"RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
|
||||||
|
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||||
|
```
|
||||||
|
|
||||||
|
The entry script started the Worker and its collectors on `174.1.51.7` over SSH.
|
||||||
|
The full Phase 2 command must not be run a second time on the Worker.
|
||||||
|
|
||||||
|
## Endpoint Results
|
||||||
|
|
||||||
|
| Case | Input TPS | Output TPS | TTFT P95 | TPOT P95 | E2E P95 |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| 128K -> 1, C=1 | 2,618.53 | 0.02 | 50.036 s | - | 50.036 s |
|
||||||
|
| 32K -> 1, C=16 | 3,116.20 | 0.10 | 161.899 s | - | 161.899 s |
|
||||||
|
| 1K -> 1K, C=32 | 447.41 | 447.41 | 10.144 s | 65.63 ms | 73.189 s |
|
||||||
|
| 1K -> 4K, C=16 | 79.35 | 317.41 | 1.727 s | 50.01 ms | 206.427 s |
|
||||||
|
| 128K -> 1K, C=1 | 1,610.89 | 12.59 | 48.489 s | 32.12 ms | 81.343 s |
|
||||||
|
|
||||||
|
## Mixed Prefill/Decode
|
||||||
|
|
||||||
|
| Decode metric | Control | With one 128K Prefill | Change |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| Output TPS | 454.39 | 345.20 | -24.03% |
|
||||||
|
| TTFT P95 | 9.437 s | 9.869 s | +4.58% |
|
||||||
|
| TPOT P95 | 66.17 ms | 110.36 ms | +66.79% |
|
||||||
|
| E2E P95 | 72.225 s | 117.941 s | +63.30% |
|
||||||
|
|
||||||
|
This reproduces the Phase 1 contention result. Prefill mainly damages ongoing
|
||||||
|
Decode progress rather than its first-token latency. Global token-level ITL P95
|
||||||
|
stayed near 62 ms because a small number of synchronized long stalls can remain
|
||||||
|
below the top 5% of all token gaps. Per-request TPOT and E2E expose the impact
|
||||||
|
more clearly.
|
||||||
|
|
||||||
|
## Hardware Evidence
|
||||||
|
|
||||||
|
### GPU
|
||||||
|
|
||||||
|
- During the 128K Prefill injection, average GPU utilization was 99.78% on the
|
||||||
|
Head and 99.34% on the Worker; average power was about 274 W on both nodes.
|
||||||
|
- During 32K -> 1, C=16 Prefill, average GPU utilization was about 82% on both
|
||||||
|
nodes and power was about 265 W.
|
||||||
|
- GPU clocks remained around 2.38 to 2.41 GHz. There is no evidence of a
|
||||||
|
frequency collapse.
|
||||||
|
- GPU memory stayed near 83.2 to 83.4 GiB of 85,651 MiB per GPU, leaving only
|
||||||
|
about 2.3 GiB of headroom.
|
||||||
|
- Head DCGM data is valid. During the injection window, `GRACT=0.995`,
|
||||||
|
`SMACT=0.648`, `SMOCC=0.215`, `Tensor Active=0.108`, and
|
||||||
|
`DRAM Active=0.412`.
|
||||||
|
|
||||||
|
### CPU
|
||||||
|
|
||||||
|
- Whole-machine CPU utilization stayed around 8% to 11% across 128 logical
|
||||||
|
CPUs, with zero average I/O wait.
|
||||||
|
- A few individual cores were saturated. During the injection window, 5 Head
|
||||||
|
cores and 4 Worker cores averaged above 80%.
|
||||||
|
- Therefore the machine is not globally CPU-bound, but scheduler, affinity, or
|
||||||
|
NUMA hot threads remain plausible contributors.
|
||||||
|
|
||||||
|
### Inter-node RDMA
|
||||||
|
|
||||||
|
- NCCL logs on both nodes verified `NET/IB + GDRDMA` over
|
||||||
|
`mlx5_0,mlx5_3`.
|
||||||
|
- Both rails carried nearly identical traffic and all recorded RDMA error
|
||||||
|
deltas were zero.
|
||||||
|
- The largest average traffic occurred in 32K -> 1, C=16 Prefill:
|
||||||
|
about 140.0 Gbit/s total transmit on the Head and 139.3 Gbit/s on the Worker,
|
||||||
|
or about 70 Gbit/s per 400G rail.
|
||||||
|
- This is only about 17.5% of each rail's nominal line rate. Raw RoCE bandwidth
|
||||||
|
saturation is therefore not the primary ceiling, although collective
|
||||||
|
latency and synchronization overhead are not ruled out.
|
||||||
|
|
||||||
|
### Intra-node GPU Communication
|
||||||
|
|
||||||
|
- RTX Pro 6000D has no NVLink. `nvidia-smi nvlink --status` reports no links.
|
||||||
|
- `nvidia-smi topo -m` shows GPU0-3 and GPU4-7 connected through local PCIe
|
||||||
|
switches (`PIX`), while traffic between the two four-GPU groups crosses the
|
||||||
|
host bridge and CPU/NUMA interconnect (`SYS`).
|
||||||
|
- NCCL logs show intra-node transfers through `P2P/IPC`; on this machine that
|
||||||
|
means CUDA peer-to-peer over PCIe, not NVLink.
|
||||||
|
- Phase 2 collected DCGM PCIe counters but did not run a dedicated GPU-to-GPU
|
||||||
|
bandwidth/latency microbenchmark. A later short diagnostic should run
|
||||||
|
`p2pBandwidthLatencyTest`, 8-GPU `all_reduce_perf`, and 16-GPU
|
||||||
|
`all_reduce_perf`.
|
||||||
|
|
||||||
|
## Preliminary Attribution
|
||||||
|
|
||||||
|
The first successful Phase 2 run supports three conclusions:
|
||||||
|
|
||||||
|
1. Prefill/Decode resource contention is real and reproducible.
|
||||||
|
2. Raw dual-rail RoCE bandwidth, whole-machine CPU capacity, and clock
|
||||||
|
throttling do not explain the observed slowdown.
|
||||||
|
3. The next useful target is the TP16 execution timeline: GPU kernels,
|
||||||
|
scheduler gaps, PCIe/RDMA collectives, and synchronization between ranks.
|
||||||
|
|
||||||
|
This is a bottleneck direction, not a final kernel-level root cause. Phase 3
|
||||||
|
should capture short Prefill, Decode, and mixed timelines rather than repeat the
|
||||||
|
full benchmark matrix.
|
||||||
|
|
||||||
|
## Collection Limitations
|
||||||
|
|
||||||
|
- Worker DCGM exited because `nvidia-dcgm` hostengine was inactive on
|
||||||
|
`174.1.51.7`. Before another hardware run, execute on the Worker:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl start nvidia-dcgm
|
||||||
|
systemctl is-active nvidia-dcgm
|
||||||
|
dcgmi discovery -l
|
||||||
|
```
|
||||||
|
|
||||||
|
The main Phase 2 entry still runs only on the Head.
|
||||||
|
|
||||||
|
- Current Case windows include benchmark-client startup, tokenizer/data
|
||||||
|
preparation, warm-up, the measured run, and client exit. Hardware averages
|
||||||
|
are therefore diluted outside the true main benchmark interval. Future
|
||||||
|
profiling must add explicit main-run start/end markers.
|
||||||
|
- `pidstat -durwt` at one-second intervals generated about 850 MB on the Head
|
||||||
|
and 735 MB on the Worker. The next run should use process-level sampling
|
||||||
|
and/or a five-second interval.
|
||||||
|
- Worker teardown logged a Gloo connection reset after the Head stopped. It
|
||||||
|
occurred after all eight benchmark rows completed and is a cleanup artifact,
|
||||||
|
not a benchmark failure.
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"run_id": "dsv4pro-phase2-20260731-130125",
|
||||||
|
"status": "COMPLETED",
|
||||||
|
"created_at": "2026-07-31T13:01:25+08:00",
|
||||||
|
"completed_at": "2026-07-31T13:27:51+08:00",
|
||||||
|
"experiment": "dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution",
|
||||||
|
"git_commit": "ca1f2f63375cd6e2bd249a91cbdbe7576aafad7e",
|
||||||
|
"git_dirty": false,
|
||||||
|
"phase1_entry": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh",
|
||||||
|
"head_node": "10.101.0.11",
|
||||||
|
"worker_node": "10.101.0.13",
|
||||||
|
"fixed_case_ids": "long_prefill_latency_128k_c1,mid_prefill_throughput_32k_c16,decode_throughput_1k_to_1k_c32,long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1",
|
||||||
|
"run_mixed_case": true,
|
||||||
|
"sample_interval_s": 1,
|
||||||
|
"numastat_interval_s": 5,
|
||||||
|
"clock_skew_tolerance_s": 2,
|
||||||
|
"idle_baseline_s": 15,
|
||||||
|
"post_run_cooldown_s": 15,
|
||||||
|
"dry_run": false
|
||||||
|
}
|
||||||
@ -0,0 +1,34 @@
|
|||||||
|
# Phase 2 Hardware Attribution
|
||||||
|
|
||||||
|
- Generated: `2026-07-31T13:27:51+08:00`
|
||||||
|
- Bench rows: `8`
|
||||||
|
- Failed bench rows: `0`
|
||||||
|
- GPU summary rows: `16`
|
||||||
|
- RDMA summary rows: `4`
|
||||||
|
- Case windows: `8`
|
||||||
|
- Collector status counts: `{"EXITED_BEFORE_STOP": 1, "STARTED": 18, "STOPPED": 17}`
|
||||||
|
|
||||||
|
## Bench
|
||||||
|
|
||||||
|
| Run | Case | Status | Input TPS | Output TPS | TTFT P95 (ms) | TPOT P95 (ms) |
|
||||||
|
|---|---|---|---:|---:|---:|---:|
|
||||||
|
| fixed_decode_throughput_1k_to_1k_c32 | decode_throughput_1k_to_1k_c32 | COMPLETED | 447.41274460374615 | 447.41274460374615 | 10144.207949569682 | 65.62598442892012 |
|
||||||
|
| fixed_long_context_decode_128k_to_1k_c1 | long_context_decode_128k_to_1k_c1 | COMPLETED | 1610.8926328335153 | 12.585098694011839 | 48488.74244495528 | 32.11543163640164 |
|
||||||
|
| fixed_long_output_decode_1k_to_4k_c16 | long_output_decode_1k_to_4k_c16 | COMPLETED | 79.35259394563253 | 317.41037578253014 | 1726.5846392256208 | 50.00692894707874 |
|
||||||
|
| fixed_long_prefill_latency_128k_c1 | long_prefill_latency_128k_c1 | COMPLETED | 2618.53322910782 | 0.019977823098051604 | 50036.02714801673 | 0.0 |
|
||||||
|
| fixed_mid_prefill_throughput_32k_c16 | mid_prefill_throughput_32k_c16 | COMPLETED | 3116.2035105433815 | 0.0950989840864069 | 161899.06251549837 | 0.0 |
|
||||||
|
| mixed_prefill_decode_interference | decode_control_1k_to_1k_c32 | COMPLETED | 454.3923063408163 | 454.3923063408163 | 9436.576361441985 | 66.170705189718 |
|
||||||
|
| mixed_prefill_decode_interference | decode_with_128k_prefill_1k_to_1k_c32 | COMPLETED | 345.2031620939666 | 345.2031620939666 | 9868.98645934416 | 110.36397890971698 |
|
||||||
|
| mixed_prefill_decode_interference | long_prefill_injection_128k_to_1_c1 | COMPLETED | 2844.468616501081 | 0.021701573307045603 | 45999.09703899175 | 0.0 |
|
||||||
|
|
||||||
|
## Machine-readable summaries
|
||||||
|
|
||||||
|
- `gpu_summary.csv`
|
||||||
|
- `rdma_summary.csv`
|
||||||
|
- `bench_summary.csv`
|
||||||
|
- `case_windows.csv`
|
||||||
|
- `case_gpu_summary.csv`
|
||||||
|
- `case_rdma_summary.csv`
|
||||||
|
- `summary.json`
|
||||||
|
|
||||||
|
The final bottleneck conclusion is written only after aligning these files with `markers.csv`, raw DCGM/sysstat logs, and SGLang service logs.
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
@ -0,0 +1,45 @@
|
|||||||
|
{
|
||||||
|
"generated_at": "2026-07-31T13:27:51+08:00",
|
||||||
|
"bench_rows": 8,
|
||||||
|
"failed_bench_rows": 0,
|
||||||
|
"gpu_summary_rows": 16,
|
||||||
|
"rdma_summary_rows": 4,
|
||||||
|
"case_windows": 8,
|
||||||
|
"case_gpu_summary_rows": 128,
|
||||||
|
"case_rdma_summary_rows": 32,
|
||||||
|
"collector_status_counts": {
|
||||||
|
"STARTED": 18,
|
||||||
|
"STOPPED": 17,
|
||||||
|
"EXITED_BEFORE_STOP": 1
|
||||||
|
},
|
||||||
|
"collector_files": {
|
||||||
|
"head": {
|
||||||
|
"dcgm_dmon.log": 1638191,
|
||||||
|
"docker_top.log": 11692641,
|
||||||
|
"gpu_samples.csv": 726449,
|
||||||
|
"markers.csv": 1860,
|
||||||
|
"mpstat.log": 15587271,
|
||||||
|
"numastat.log": 3280062,
|
||||||
|
"perf_stat.log": 744719,
|
||||||
|
"pidstat.log": 850254976,
|
||||||
|
"rdma.csv": 230202,
|
||||||
|
"sar_net.log": 2118579,
|
||||||
|
"static_after.log": 1195680,
|
||||||
|
"static_before.log": 1306492
|
||||||
|
},
|
||||||
|
"worker": {
|
||||||
|
"dcgm_dmon.log": 243,
|
||||||
|
"docker_top.log": 11989768,
|
||||||
|
"gpu_samples.csv": 758155,
|
||||||
|
"markers.csv": 1902,
|
||||||
|
"mpstat.log": 15587260,
|
||||||
|
"numastat.log": 3579303,
|
||||||
|
"perf_stat.log": 747768,
|
||||||
|
"pidstat.log": 734994477,
|
||||||
|
"rdma.csv": 242603,
|
||||||
|
"sar_net.log": 2383072,
|
||||||
|
"static_after.log": 1195267,
|
||||||
|
"static_before.log": 1306098
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,51 @@
|
|||||||
|
# Phase 2 Hardware Attribution Summary
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
- Run ID: `dsv4pro-phase2-20260731-163620`
|
||||||
|
- Status: `COMPLETED`
|
||||||
|
- Duration: 2026-07-31 16:36:20 to 17:05:04 CST, 28m44s
|
||||||
|
- Benchmarks: 8/8 successful
|
||||||
|
- Precise benchmark windows: 8/8
|
||||||
|
- Collectors: 18 started, 18 stopped
|
||||||
|
- Run commit: `5f24b7d22f98108f6cc234edba6768d55ea0a962`
|
||||||
|
- Git dirty: `false`
|
||||||
|
|
||||||
|
## Main Result
|
||||||
|
|
||||||
|
Injecting one 128K prefill request into the `1K -> 1K, C=32` decode workload caused:
|
||||||
|
|
||||||
|
| Metric | Control | Treatment | Delta |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| Output TPS | 453.55 | 344.89 | -23.96% |
|
||||||
|
| TTFT P95 | 9.443 s | 9.892 s | +4.76% |
|
||||||
|
| TPOT P95 | 66.24 ms | 110.45 ms | +66.75% |
|
||||||
|
| E2E P95 | 72.270 s | 118.076 s | +63.38% |
|
||||||
|
|
||||||
|
The dominant impact is sustained decode slowdown, not only first-token queueing.
|
||||||
|
|
||||||
|
## Attribution
|
||||||
|
|
||||||
|
- GPU utilization is generally 94-99%, with stable 2.39-2.42 GHz clocks. There is no evidence of frequency collapse.
|
||||||
|
- Whole-host CPU active is about 9.6-10.4%; CPU capacity is not globally saturated.
|
||||||
|
- Head and Worker show similar GPU, CPU and communication behavior. There is no obvious slow-node imbalance.
|
||||||
|
- The highest measured RDMA traffic is about 83.5 Gbit/s per 400G rail. Both rails are balanced, with zero wait/error/discard/retry deltas.
|
||||||
|
- PCIe P2P bandwidth is about 53.5 GB/s within a switch and 52.3-52.4 GB/s across NUMA, only about a 2.3% difference.
|
||||||
|
- 16-GPU 1 GiB AllReduce bus bandwidth is 39.345/39.685/39.530 GB/s for `NCCL_CROSS_NIC=0/1/2`. The spread is below 1%.
|
||||||
|
|
||||||
|
The evidence excludes raw rail saturation, whole-host CPU saturation, GPU clock throttling, severe PCIe cross-NUMA collapse, and `NCCL_CROSS_NIC` selection as the primary cause.
|
||||||
|
|
||||||
|
## Phase 3 Target
|
||||||
|
|
||||||
|
Capture short Control/Treatment timelines and identify:
|
||||||
|
|
||||||
|
1. Attention/indexer and MoE kernel serialization.
|
||||||
|
2. NCCL collective placement and compute/communication overlap.
|
||||||
|
3. Scheduler or CPU launch gaps.
|
||||||
|
4. Slow-rank synchronization around injected prefill.
|
||||||
|
|
||||||
|
Do not repeat Phase 2's long-running DCGM, CPU, RDMA or communication microbenchmarks.
|
||||||
|
|
||||||
|
## Cleanup
|
||||||
|
|
||||||
|
After the run, both nodes had no related Docker containers, no GPU compute processes, and no listeners on the Phase 2 service ports. Worker Gloo `Connection closed by peer` tracebacks occurred during orchestrated shutdown after all benchmarks completed and did not affect the results.
|
||||||
@ -0,0 +1,9 @@
|
|||||||
|
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1,1024,1024,32,32,1,COMPLETED,,0,2026-07-31T16:49:35+0800,2026-07-31T16:51:24+0800,109.0,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.98790900100721,bench_main_marker_plus_duration,32,0,72.98790900100721,31.973027693348868,,32768,32768,0.43842878139663394,448.95107215015315,448.95107215015315,897.9021443003063,,72926.38861777596,72926.03143348242,72939.17302195623,72943.4486198437,8317.972617309351,8423.306765995221,10166.788510262268,10167.903182043228,63.155831867513776,63.05236280694347,65.35758210354297,67.36198566667542,63.155803331197355,61.26605049939826,62.53751848998945,65.70577920065261,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log,fixed_decode_throughput_1k_to_1k_c32
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,fixed,long_context_decode_128k_to_1k_c1,,long_context_decode,1,131072,1024,1,1,0,COMPLETED,,0,2026-07-31T16:55:45+0800,2026-07-31T16:57:39+0800,114.0,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.1598442849936,bench_main_marker_plus_duration,1,0,81.1598442849936,0.9995611458686556,,131072,1024,0.012321364201839645,1614.985848663526,12.617076942683797,1627.6029256062097,,81124.22695202986,81124.22695202986,81124.22695202986,81124.22695202986,48279.420554987155,48279.420554987155,48279.420554987155,48279.420554987155,32.106360114411245,32.106360114411245,32.106360114411245,32.106360114411245,32.10634595800085,32.09696803241968,32.28448228328489,32.53332086256705,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/bench.log,fixed_long_context_decode_128k_to_1k_c1
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,fixed,long_output_decode_1k_to_4k_c16,,long_output_decode,1,1024,4096,16,16,1,COMPLETED,,0,2026-07-31T16:51:33+0800,2026-07-31T16:55:36+0800,243.0,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074682698585,bench_main_marker_plus_duration,16,0,206.37074682698585,15.99550577130734,,16384,65536,0.07753036826199912,79.3910971002871,317.5643884011484,396.95548550143553,,206312.77949375362,206311.27990750247,206327.66740498482,206328.1800113793,1697.1331631830253,1714.9575955118053,1723.5681610036409,1724.2807810165687,49.96719080111614,49.96277789426234,49.98305734157135,50.02649938860528,49.96718302440562,49.99568348284811,50.960358188604005,53.57307427912019,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/bench.log,fixed_long_output_decode_1k_to_4k_c16
|
||||||
|
fixed_long_prefill_latency_128k_c1,fixed,long_prefill_latency_128k_c1,,prefill_latency,1,131072,1,1,1,0,COMPLETED,,0,2026-07-31T16:44:30+0800,2026-07-31T16:45:54+0800,84.0,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567456041,bench_main_marker_plus_duration,1,0,49.622567456041,0.9997500965858968,,131072,1,0.02015212132838284,2641.3788467537956,0.02015212132838284,2641.398998875124,,49610.16660701716,49610.16660701716,49610.16660701716,49610.16660701716,49610.14223104576,49610.14223104576,49610.14223104576,49610.14223104576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/bench.log,fixed_long_prefill_latency_128k_c1
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,1,32768,1,16,16,0,COMPLETED,,0,2026-07-31T16:46:03+0800,2026-07-31T16:49:26+0800,203.0,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.2866359280306,bench_main_marker_plus_duration,16,0,168.2866359280306,8.764467555247327,,524288,16,0.09507587998159613,3115.446435236942,0.09507587998159613,3115.5415111169236,,92183.92253580896,92587.60345654446,161938.1903584872,166972.64813809306,92183.87291699764,92587.55126551841,161938.13879448862,166972.5966172962,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/bench.log,fixed_mid_prefill_throughput_32k_c16
|
||||||
|
mixed_prefill_decode_interference,mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1,1024,1024,32,64,1,COMPLETED,,0,2026-07-31T16:57:48+0800,2026-07-31T17:00:49+0800,181.0,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.4964013720164,bench_main_marker_plus_duration,64,0,144.4964013720164,31.983334150914263,,65536,65536,0.4429176048144437,453.54762732999035,453.54762732999035,907.0952546599807,,72210.57326071605,72215.36969349836,72269.92528844276,72271.96539751603,7325.20390184618,6846.627883001929,9442.68154159945,9444.044730461319,63.426558513069274,63.83972124386403,66.23843437823616,68.16205457722045,63.42653002822952,61.320325010456145,62.66533228044864,65.59292251302395,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/bench.log,mixed_prefill_decode_interference
|
||||||
|
mixed_prefill_decode_interference,mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,1024,1024,32,64,0,COMPLETED,,0,2026-07-31T17:00:49+0800,2026-07-31T17:04:32+0800,223.0,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.02009954402456,bench_main_marker_plus_duration,64,0,190.02009954402456,31.986201956249488,,65536,65536,0.3368064754916742,344.8898309034744,344.8898309034744,689.7796618069488,,94969.08249627813,94968.0481730029,118076.02923363738,118079.88882219535,7606.184888919415,7407.497464475455,9891.837346865213,9893.77336708887,85.3987268889137,86.89664765445467,110.451060616212,111.28696239483178,85.39869778190757,61.08463852433488,61.97826337593142,65.3158738068305,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log,mixed_prefill_decode_interference
|
||||||
|
mixed_prefill_decode_interference,mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,131072,1,1,1,0,COMPLETED,,0,2026-07-31T17:01:29+0800,2026-07-31T17:02:47+0800,78.0,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.06182585703209,bench_main_marker_plus_duration,1,0,46.06182585703209,0.9982293388832256,,131072,1,0.02170995138368649,2845.5667477625557,0.02170995138368649,2845.5884577139395,,45980.265973019414,45980.265973019414,45980.265973019414,45980.265973019414,45980.19455798203,45980.19455798203,45980.19455798203,45980.19455798203,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log,mixed_prefill_decode_interference
|
||||||
|
@ -0,0 +1,17 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,cpu_active_pct_mean,cpu_active_pct_p95,cpu_active_pct_max,iowait_pct_mean,iowait_pct_p95,iowait_pct_max,hot_cores_ge80_mean,hot_cores_ge80_p95,hot_cores_ge80_max
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,15,10.130666666666666,10.418000000000003,10.459999999999994,0.0,0.0,0.0,10.466666666666667,12.0,12.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,15,9.843333333333332,10.085999999999997,10.099999999999994,0.0,0.0,0.0,10.4,12.0,12.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,17,9.899411764705883,10.292000000000003,10.299999999999997,0.0,0.0,0.0,10.588235294117647,12.0,12.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,17,9.651176470588235,10.089999999999996,10.129999999999995,0.0,0.0,0.0,10.352941176470589,12.0,12.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,41,10.361707317073172,10.519999999999996,10.620000000000005,0.0,0.0,0.0,11.024390243902438,12.0,12.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,41,9.994390243902439,10.090000000000003,10.159999999999997,0.0,0.0,0.0,11.414634146341463,12.0,12.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,10,10.003,10.272999999999996,10.299999999999997,0.0,0.0,0.0,10.1,12.0,12.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,10,9.688,10.180500000000004,10.230000000000004,0.0,0.0,0.0,10.2,12.0,12.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,34,9.985588235294117,10.267000000000003,10.319999999999993,0.0,0.0,0.0,11.058823529411764,12.0,12.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,34,9.84235294117647,10.1135,10.14,0.0,0.0,0.0,11.205882352941176,12.0,12.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,29,10.262758620689656,10.436,10.489999999999995,0.0,0.0,0.0,10.620689655172415,12.0,12.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,29,9.976551724137929,10.108000000000004,10.120000000000005,0.0,0.0,0.0,11.241379310344827,12.0,12.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,38,10.341052631578947,11.3725,12.36,0.0,0.0,0.0,10.868421052631579,12.149999999999999,13.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,38,9.868684210526316,10.091500000000002,10.14,0.0,0.0,0.0,10.868421052631579,12.0,12.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,10,10.212,10.630499999999998,10.950000000000003,0.0,0.0,0.0,11.6,12.549999999999999,13.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,9,10.035555555555556,10.120000000000001,10.14,0.0,0.0,0.0,11.666666666666666,12.0,12.0
|
||||||
|
@ -0,0 +1,17 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,gpus,samples,gr_engine_active_mean,gr_engine_active_p95,gr_engine_active_max,sm_active_mean,sm_active_p95,sm_active_max,sm_occupancy_mean,sm_occupancy_p95,sm_occupancy_max,tensor_active_mean,tensor_active_p95,tensor_active_max,dram_active_mean,dram_active_p95,dram_active_max,pcie_tx_bytes_per_s_mean,pcie_tx_bytes_per_s_p95,pcie_tx_bytes_per_s_max,pcie_rx_bytes_per_s_mean,pcie_rx_bytes_per_s_p95,pcie_rx_bytes_per_s_max
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,8,584,0.9710171232876712,0.999,1.0,0.5252157534246575,0.7,0.707,0.14078253424657533,0.291,0.294,0.05895205479452054,0.11499999999999999,0.12,0.41691609589041095,0.499,0.508,5205379813.392123,14778814130.95,15202540154.0,5200970877.481164,14816565450.150002,15254205047.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,8,576,0.9785503472222222,0.999,0.999,0.5226267361111111,0.6872499999999999,0.698,0.14059895833333333,0.28824999999999995,0.293,0.05846874999999999,0.113,0.117,0.4153368055555555,0.495,0.548,5236209935.171875,14647937823.25,16403294625.0,5231738841.786458,14714002991.25,16357346290.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,8,648,0.9613811728395062,0.9990000000000001,0.999,0.5782654320987655,0.76,0.768,0.21059722222222224,0.282,0.284,0.09468981481481482,0.162,0.168,0.3985987654320987,0.44500000000000006,0.479,6763589500.305555,13241413192.75,14068752272.0,6821390469.66821,13261233168.150002,14425674313.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,8,648,0.965466049382716,0.9990000000000001,1.0,0.5789398148148148,0.76,0.768,0.21097993827160494,0.281,0.285,0.09508796296296296,0.162,0.166,0.4005216049382716,0.44600000000000006,0.47,6780236898.856482,13234577683.95,13889246455.0,6837638258.785494,13289110201.45,13879127175.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,8,1648,0.9890279126213591,0.996,0.997,0.4903646844660194,0.5,0.54,0.13298240291262137,0.135,0.202,0.04484769417475728,0.048,0.102,0.41422026699029124,0.427,0.469,2620560854.958738,2632307969.6499996,11954964997.0,2634422729.365898,2682732685.6,10279631228.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,8,1640,0.9897713414634146,0.996,1.0,0.4890847560975609,0.498,0.683,0.13288231707317075,0.135,0.268,0.044690243902439024,0.047,0.129,0.4142451219512195,0.427,0.468,2618598082.628049,2632407781.75,14221424532.0,2633369965.667683,2681936017.75,13880734362.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,8,400,0.9295425,0.999,0.999,0.6828425,0.761,0.769,0.25086,0.282,0.284,0.1358075,0.163,0.168,0.37432499999999996,0.44904999999999995,0.505,10536527950.9525,13519030283.25,15163087193.0,10581948383.175,13517471298.849998,15273700217.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,8,392,0.9359923469387755,0.9989999999999999,1.0,0.6856326530612245,0.76,0.765,0.250795918367347,0.282,0.285,0.13560204081632654,0.162,0.165,0.37558673469387754,0.45324999999999993,0.485,10617939301.517857,13498405363.0,13917911496.0,10632410279.880102,13503375869.85,14629076948.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,8,1344,0.9804598214285715,0.999,1.0,0.7134583333333333,0.739,0.744,0.2776354166666667,0.287,0.288,0.1272514880952381,0.138,0.141,0.4404389880952381,0.467,0.509,12664097012.861607,14053446776.599998,15685065820.0,12719741431.203125,14071304248.15,15724306505.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,8,1344,0.9839747023809524,0.999,1.0,0.7149203869047619,0.739,0.743,0.27822619047619046,0.287,0.289,0.12745982142857143,0.137,0.14,0.44101190476190477,0.467,0.514,12698564254.736607,14063118317.15,15508567216.0,12754602448.509672,14079354109.05,15319006836.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,8,1152,0.9889574652777777,0.999,1.0,0.538584201388889,0.699,0.705,0.1456640625,0.292,0.295,0.060828125,0.11499999999999999,0.121,0.4266675347222222,0.5,0.558,5451444350.41493,14941958481.6,16367702095.0,5456132661.967882,14993495237.7,16348072773.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,8,1152,0.9896744791666667,0.999,1.0,0.5361883680555555,0.697,0.704,0.1450859375,0.291,0.295,0.06014930555555556,0.114,0.119,0.4251788194444444,0.5,0.55,5464990127.207465,14978142467.55,16515378482.0,5460547526.944445,14998798408.1,15300436571.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,8,1520,0.9768513157894737,0.999,1.0,0.5805618421052632,0.748,0.763,0.17383618421052632,0.291,0.295,0.08014671052631578,0.15404999999999996,0.168,0.4163407894736842,0.499,0.51,6865359213.189474,14886672661.75,15342998825.0,6868476165.281579,14922047967.4,15374888508.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,8,1520,0.978421052631579,0.999,1.0,0.578371052631579,0.748,0.763,0.17389407894736844,0.291,0.295,0.0797078947368421,0.154,0.175,0.4157486842105263,0.498,0.511,6864820605.633553,14934618195.95,15291598078.0,6872189917.845395,14950386000.25,15356103677.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,8,368,0.9974673913043478,0.9990000000000001,1.0,0.7229048913043478,0.756,0.763,0.2642554347826087,0.285,0.288,0.13833152173913044,0.16,0.165,0.40672282608695653,0.4545999999999999,0.48,11151168359.230978,13665801629.75,14505291121.0,11197667600.644022,13730586781.449997,14645337945.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,8,368,0.9973179347826087,0.9990000000000001,0.999,0.7197690217391305,0.757,0.763,0.2623777173913044,0.285,0.288,0.13687228260869566,0.15964999999999996,0.166,0.40697826086956523,0.45654999999999984,0.486,11068919985.380434,13568556767.3,14853005782.0,11109834217.782608,13585937423.0,14369923387.0
|
||||||
|
@ -0,0 +1,17 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,gpu_util_pct_mean,gpu_util_pct_p95,gpu_util_pct_max,memory_util_pct_mean,memory_util_pct_p95,memory_util_pct_max,memory_used_mib_mean,memory_used_mib_p95,memory_used_mib_max,power_w_mean,power_w_p95,power_w_max,temperature_c_mean,temperature_c_p95,temperature_c_max,sm_clock_mhz_mean,sm_clock_mhz_p95,sm_clock_mhz_max,memory_clock_mhz_mean,memory_clock_mhz_p95,memory_clock_mhz_max,gpus
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,408,97.09313725490196,100.0,100.0,50.85539215686274,61.0,63.0,83300.0,83409.0,83409.0,230.21735294117644,299.2755,312.84,43.713235294117645,45.0,46.0,2402.950980392157,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,432,97.72222222222223,100.0,100.0,51.21527777777778,60.44999999999999,64.0,83300.0,83409.0,83409.0,221.8317592592593,289.236,307.91,45.844907407407405,47.0,49.0,2418.5486111111113,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,456,97.70394736842105,100.0,100.0,50.24122807017544,57.0,59.0,83219.06140350878,83453.0,83453.0,257.11598684210526,325.23,342.99,46.23026315789474,52.0,54.0,2412.627192982456,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,472,97.43008474576271,100.0,100.0,49.84110169491525,57.0,58.0,83221.32203389831,83453.0,83453.0,257.90165254237286,327.982,372.66,48.74576271186441,54.0,57.0,2417.531779661017,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,1184,98.82516891891892,100.0,100.0,50.710304054054056,53.0,59.0,83300.0,83409.0,83409.0,220.82934966216217,242.39100000000002,275.9,43.5304054054054,45.0,45.0,2402.7592905405404,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,1216,99.2467105263158,100.0,100.0,50.49753289473684,53.0,54.0,83300.0,83409.0,83409.0,215.73606907894737,236.6025,270.17,45.59950657894737,47.0,48.0,2412.5715460526317,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,288,94.44444444444444,100.0,100.0,46.74305555555556,56.0,58.0,83028.21527777778,83408.2,83443.0,293.3987847222222,330.7185,368.5,44.83680555555556,51.0,53.0,2392.215277777778,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,288,94.85069444444444,100.0,100.0,46.420138888888886,55.0,57.0,83057.76388888889,83408.2,83443.0,297.9095486111111,335.86899999999997,363.48,46.53819444444444,52.0,54.0,2398.0243055555557,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,952,98.20588235294117,100.0,100.0,53.52205882352941,58.0,59.0,83231.06722689075,83405.0,83409.0,299.4595693277311,316.97,332.42,50.438025210084035,55.0,56.0,2417.686974789916,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,968,99.07438016528926,100.0,100.0,53.974173553719005,58.0,59.0,83231.94008264462,83405.0,83409.0,302.563285123967,322.293,339.05,53.75929752066116,59.0,61.0,2419.3326446280994,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,824,98.52427184466019,100.0,100.0,51.5376213592233,61.0,65.0,83364.0,83473.0,83473.0,231.9156432038835,297.50299999999993,324.46,44.297330097087375,46.0,50.0,2405.38713592233,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,856,98.60280373831776,100.0,100.0,51.441588785046726,61.0,64.0,83364.0,83473.0,83473.0,227.7883528037383,293.6475,322.45,46.43691588785047,49.0,52.0,2418.589953271028,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,1072,99.60914179104478,100.0,100.0,51.59794776119403,60.0,65.0,83259.71082089552,83433.0,83433.0,248.31210820895524,315.4865,376.49,46.649253731343286,52.0,55.0,2411.4869402985073,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,1112,98.92086330935251,100.0,100.0,51.085431654676256,60.0,64.0,83256.90647482015,83433.0,83433.0,247.74912769784174,321.8905,373.73,49.64118705035971,55.0,59.0,2420.7724820143885,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,264,99.95075757575758,100.0,100.0,49.45075757575758,57.0,59.0,83281.49242424243,83413.0,83433.0,301.19901515151514,332.613,376.49,49.92424242424242,54.0,55.0,2421.1136363636365,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,272,99.88602941176471,100.0,100.0,49.39705882352941,57.0,59.0,83282.94117647059,83413.0,83433.0,307.1604411764706,340.2845,373.73,52.86764705882353,57.0,59.0,2421.4595588235293,2422.0,2422.0,12481.0,12481.0,12481.0,8
|
||||||
|
@ -0,0 +1,33 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,iface,samples,rx_gbps_mean,rx_gbps_p95,rx_gbps_max,tx_gbps_mean,tx_gbps_p95,tx_gbps_max,ifutil_pct_mean,ifutil_pct_p95,ifutil_pct_max,rx_errors_s_mean,rx_errors_s_p95,rx_errors_s_max,tx_errors_s_mean,tx_errors_s_p95,tx_errors_s_max,rx_drops_s_mean,rx_drops_s_p95,rx_drops_s_max,tx_drops_s_mean,tx_drops_s_p95,tx_drops_s_max
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,eth0,30,0.00015682133333333332,0.00016183999999999998,0.00016464,5.1274666666666665e-05,5.4591999999999995e-05,5.504e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,eth3,30,3.0933333333333335e-07,8.8e-07,8.8e-07,4.2666666666666665e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,eth0,30,5.1695999999999994e-05,5.4959999999999996e-05,5.608e-05,0.000156736,0.000161872,0.00016287999999999999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,eth3,30,3.0933333333333335e-07,8.8e-07,8.8e-07,4.2666666666666665e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,eth0,34,0.00015644235294117647,0.00015915199999999997,0.0001596,5.2367058823529417e-05,5.536e-05,5.568e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,eth3,34,3.435294117647059e-07,8.8e-07,8.8e-07,5.6470588235294123e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,eth0,34,5.264470588235294e-05,5.5552000000000006e-05,5.568e-05,0.0001562070588235294,0.00016,0.00016,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,eth3,34,3.435294117647059e-07,8.8e-07,8.8e-07,5.6470588235294123e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,eth0,82,0.0001566419512195122,0.00015984000000000001,0.00016272,4.933853658536585e-05,5.2560000000000005e-05,5.44e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,eth3,82,3.3365853658536586e-07,8.8e-07,8.8e-07,5.463414634146342e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,eth0,82,4.958634146341463e-05,5.4e-05,5.456e-05,0.00015623219512195122,0.00016128,0.00016287999999999999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,eth3,82,3.35609756097561e-07,8.8e-07,9.6e-07,5.6585365853658546e-08,3.2e-07,4.0000000000000003e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,eth0,20,0.00021578399999999998,0.0004841319999999994,0.00074848,5.9391999999999996e-05,8.969599999999994e-05,0.0001172,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,eth3,20,3.1200000000000004e-07,6.279999999999994e-07,8.8e-07,4e-08,2.1199999999999975e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,eth0,20,5.9592e-05,8.761999999999995e-05,0.00011336,0.000215264,0.00047957599999999935,0.00074,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,eth3,20,2.96e-07,6.279999999999994e-07,8.8e-07,3.2e-08,1.7599999999999967e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,eth0,68,0.0001568870588235294,0.000160908,0.00016112,5.124470588235294e-05,5.5707999999999996e-05,5.592e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,eth3,68,3.223529411764706e-07,8.8e-07,8.8e-07,4.705882352941176e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,eth0,68,5.149411764705883e-05,5.6268000000000004e-05,5.7520000000000005e-05,0.00015664235294117647,0.00016011200000000001,0.00016063999999999998,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,eth3,68,3.223529411764706e-07,8.8e-07,8.8e-07,4.705882352941176e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,eth0,58,0.00015664000000000002,0.000159888,0.00016,5.07751724137931e-05,5.4496e-05,5.456e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,eth3,58,3.3931034482758625e-07,8.8e-07,8.8e-07,5.517241379310345e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,eth0,58,5.0982068965517244e-05,5.4528000000000004e-05,5.568e-05,0.00015630896551724138,0.000160848,0.00016168,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,eth3,58,3.3931034482758625e-07,8.8e-07,8.8e-07,5.517241379310345e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,eth0,76,0.00015644631578947367,0.000160012,0.00016056,5.221263157894737e-05,5.579599999999999e-05,5.616e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,eth3,76,3.452631578947368e-07,8.8e-07,9.6e-07,6.105263157894737e-08,3.2e-07,4.0000000000000003e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,eth0,76,5.24042105263158e-05,5.641199999999999e-05,5.664e-05,0.0001559515789473684,0.00016033200000000004,0.00016424,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,eth3,76,3.431578947368421e-07,8.8e-07,8.8e-07,5.8947368421052637e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,eth0,20,0.000156048,0.000158632,0.00015928,5.1064000000000004e-05,5.358e-05,5.376e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,eth3,20,2.96e-07,6.279999999999994e-07,8.8e-07,3.2e-08,1.7599999999999967e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,eth0,18,5.1271111111111107e-05,5.4688e-05,5.52e-05,0.00015478222222222221,0.00016222399999999998,0.00016424,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,eth3,18,2.9333333333333337e-07,6.559999999999998e-07,8.8e-07,3.5555555555555554e-08,1.919999999999999e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
@ -0,0 +1,17 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,node0_mib_mean,node0_mib_p95,node0_mib_max,node1_mib_mean,node1_mib_p95,node1_mib_max,total_mib_mean,total_mib_p95,total_mib_max,processes_mean,processes_p95,processes_max,numa_imbalance_pct_mean,numa_imbalance_pct_p95,numa_imbalance_pct_max
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,13,17941.21923076923,17941.472,17941.61,27565.280769230772,27565.394,27565.4,45506.51384615385,45506.69,45506.69,19.0,19.0,19.0,21.148755918904936,21.149743646540415,21.150655911887284
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,13,19029.48769230769,19029.536,19029.56,24814.318461538463,24814.381999999998,24814.4,43843.8,43843.854,43843.86,18.0,18.0,18.0,13.194182003167738,13.194329421344158,13.194370476132915
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,13,17946.50153846154,17948.994,17949.0,27569.42769230769,27571.648,27571.66,45515.943076923075,45520.61,45520.61,19.0,19.0,19.0,21.141880372077402,21.145764039410054,21.145782042670753
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,14,19037.215,19039.1035,19039.11,24816.07357142857,24818.423499999997,24818.43,43853.295,43857.54,43857.54,18.0,18.0,18.0,13.177706662062775,13.17809858730606,13.178146962903641
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,35,17941.668285714288,17942.12,17942.13,27565.19857142857,27566.617,27570.09,45506.88085714286,45506.89,45506.91,19.0,19.0,19.0,21.147417939082427,21.15369613423204,21.16893559148322
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,37,19029.40054054054,19029.452,19029.6,24814.58216216216,24814.65,24814.66,43843.96270270271,43843.97,43843.97,18.0,18.0,18.0,13.19493327010614,13.195223881414025,13.195246689567572
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,8,17869.37125,17877.1655,17878.8,27486.31625,27493.9225,27494.43,45355.7125,45371.1115,45373.25,19.0,19.0,19.0,21.203382762054954,21.207110348436384,21.207680920856813
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,8,18970.3375,18981.5545,18984.05,24731.71375,24743.9825,24745.68,43702.05125,43712.965,43715.31,18.0,18.0,18.0,13.183305003875798,13.21223924964188,13.21285380339291
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,28,17940.274285714284,17940.69,17940.72,27563.883928571428,27564.22,27564.23,45504.17285714286,45504.46,45504.46,19.0,19.0,19.0,21.148850835716523,21.149789612547433,21.149814841959135
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,30,19032.02333333333,19037.573,19037.75,24805.59,24810.55,24811.49,43837.58133333333,43841.73,43841.73,18.0,18.0,18.0,13.17036295064116,13.184651928653366,13.219751626067305
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,24,17946.96125,17951.6865,17951.9,27573.83541666667,27579.2765,27588.36,45520.80666666667,45520.8485,45520.85,19.0,19.0,19.0,21.148294318886816,21.17226656215764,21.212158491133064
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,26,19039.328461538462,19039.4575,19039.46,24818.277692307693,24818.565,24818.74,43857.62038461539,43857.65,43857.65,18.0,18.0,18.0,13.176613733571411,13.177963869896237,13.178761906998588
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,32,17955.7271875,17958.81,17958.83,27573.1746875,27577.874499999998,27578.04,45528.9121875,45535.71,45535.71,19.0,19.0,19.0,21.123824304701028,21.129656288079975,21.129778209910725
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,34,19042.789411764705,19046.17,19046.17,24823.40088235294,24827.86,24827.86,43866.178529411765,43873.26,43873.26,18.0,18.0,18.0,13.177832389566726,13.183982855860464,13.185098115743662
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,8,17951.77,17953.336,17953.98,27569.775,27571.0035,27571.7,45521.55625,45524.051,45525.71,19.0,19.0,19.0,21.128462660935995,21.129739766090218,21.129778209910725
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,8,19038.4325,19039.3115,19039.56,24819.52625,24820.626,24820.99,43857.942500000005,43858.878000000004,43859.48,18.0,18.0,18.0,13.181406637484468,13.184049863000734,13.184128037997716
|
||||||
|
@ -0,0 +1,17 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,ipc,cache_misses_mean,cache_misses_p95,cache_misses_max,context_switches_mean,context_switches_p95,context_switches_max,cpu_migrations_mean,cpu_migrations_p95,cpu_migrations_max,page_faults_mean,page_faults_p95,page_faults_max
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,84,3.0487843541376685,6727288.428571428,15567285.149999999,18088446.0,41240.92857142857,47221.399999999994,58274.0,934.9285714285714,1975.849999999999,4063.0,46455.857142857145,161971.65,174542.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,84,3.0681062936795103,4913412.214285715,13142249.399999999,14049236.0,39682.0,45243.25,55120.0,188.57142857142858,247.7,249.0,28090.428571428572,113948.55,114568.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,96,2.8870888885724733,6295623.4375,12277537.0,21573640.0,44877.375,72712.0,151042.0,1255.1875,3534.25,9802.0,51319.875,105487.75,183145.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,96,2.9020975501844197,4051617.5625,7033576.5,10195935.0,43686.6875,70482.5,149831.0,177.5,304.0,364.0,12704.625,37867.0,65872.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,246,3.0979852154775953,5103378.804878049,5773629.0,14670415.0,43749.9756097561,44169.0,78897.0,998.9268292682926,1151.0,6568.0,18694.80487804878,28750.0,152065.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,246,3.1168786636888095,2962364.2682926827,2913232.0,13101053.0,40973.46341463415,41420.0,67313.0,165.17073170731706,209.0,213.0,11960.536585365853,14773.0,135313.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,60,2.727435494137309,9447284.2,31640896.59999995,53106505.0,50995.8,134631.5499999998,217846.0,2372.3,12092.349999999977,21775.0,108507.0,295694.8499999996,444909.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,60,2.7489508087655112,7473178.3,22657424.999999966,37078962.0,48210.5,120052.39999999983,191525.0,296.6,871.5499999999988,1394.0,107749.9,291126.8499999996,447550.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,204,2.8023469789284796,4623449.029411765,16572071.899999999,19670078.0,39421.294117647056,35324.499999999985,263082.0,861.4117647058823,365.9499999999998,20998.0,39086.14705882353,123447.0999999998,215308.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,204,2.8098589749897895,4072417.588235294,9610375.949999992,16711381.0,39129.382352941175,45578.64999999992,229122.0,164.7058823529412,246.5999999999997,423.0,36854.5,114042.04999999983,216180.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,174,3.0424513549285557,5137320.0,7469347.399999995,13520315.0,42398.8275862069,43356.2,91980.0,970.7586206896551,1053.6,5258.0,29120.41379310345,88587.9999999999,167311.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,174,3.0719083836401526,3223721.2413793104,5225881.199999996,9479153.0,40484.58620689655,39252.8,101396.0,167.3448275862069,197.2,236.0,16289.206896551725,65153.199999999895,95763.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,228,2.980275800109699,5456416.5,10599088.849999998,17028720.0,41223.05263157895,44802.899999999994,127813.0,1211.421052631579,1458.199999999999,15346.0,36016.86842105263,119273.24999999994,173922.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,228,3.0050223544930565,3784084.789473684,6404245.849999995,13894635.0,39834.92105263158,46921.44999999995,97372.0,167.5,262.15000000000003,299.0,22792.86842105263,79725.24999999994,157546.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,54,2.808104894473792,5020429.555555556,9669376.799999999,10387880.0,33399.666666666664,37769.399999999994,41129.0,338.8888888888889,811.1999999999997,1178.0,43405.88888888889,91275.39999999998,112743.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,60,2.8298366086279145,3857952.9,5373284.599999999,5873843.0,33911.7,40936.75,42415.0,130.4,172.39999999999998,185.0,24117.8,52089.24999999995,73750.0
|
||||||
|
@ -0,0 +1,17 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,process_cpu_pct_mean,process_cpu_pct_p95,process_cpu_pct_max,process_wait_pct_mean,process_wait_pct_p95,process_wait_pct_max,process_read_kib_s_mean,process_read_kib_s_p95,process_read_kib_s_max,process_write_kib_s_mean,process_write_kib_s_p95,process_write_kib_s_max,process_major_faults_s_mean,process_major_faults_s_p95,process_major_faults_s_max,process_rss_kib_mean,process_rss_kib_p95,process_rss_kib_max,process_voluntary_switches_s_mean,process_voluntary_switches_s_p95,process_voluntary_switches_s_max,process_involuntary_switches_s_mean,process_involuntary_switches_s_p95,process_involuntary_switches_s_max
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,1140,1193.5866666666666,1223.7400000000002,1225.0000000000002,0.013333333333333334,0.05999999999999979,0.2,0.0,0.0,0.0,174.50666666666666,1036.1599999999996,1273.6000000000001,0.0,0.0,0.0,46588476.53333333,46588620.0,46588620.0,277.09333333333336,711.1799999999982,1938.0000000000002,13.32,31.619999999999997,34.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,1080,1183.24,1210.06,1210.2,0.013333333333333334,0.05999999999999979,0.2,0.0,0.0,0.0,174.50666666666666,1036.1599999999996,1273.6000000000001,0.0,0.0,0.0,44887948.266666666,44888007.2,44888024.0,230.50666666666666,625.8599999999982,1744.6,10.986666666666666,31.319999999999997,31.599999999999998
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,1292,1173.3882352941175,1222.08,1222.4,0.011764705882352941,0.03999999999999986,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46598011.52941176,46602992.8,46602996.0,581.7294117647059,3130.9999999999986,5014.2,12.835294117647058,15.639999999999997,16.599999999999998
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,1224,1160.7176470588236,1214.16,1215.6000000000001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44897810.35294118,44902216.0,44902216.0,545.2352941176471,3440.1999999999994,4371.400000000001,8.105882352941176,11.359999999999996,12.799999999999999
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,3116,1220.1268292682926,1225.6,1225.8,0.024390243902439025,0.2,0.2,0.0,0.0,0.0,9.990243902439024,0.0,409.59999999999997,0.0,0.0,0.0,46588905.26829268,46588920.0,46588920.0,225.06829268292682,227.6,647.8000000000001,7.86829268292683,10.200000000000001,28.2
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,2952,1206.5365853658536,1211.0,1211.4,0.01951219512195122,0.2,0.2,0.0,0.0,0.0,9.990243902439024,0.0,409.59999999999997,0.0,0.0,0.0,44888251.90243903,44888308.0,44888316.0,176.34634146341463,176.2,602.6,5.590243902439025,7.400000000000001,18.200000000000003
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,1129,1147.0800000000002,1211.02,1211.1999999999998,0.02,0.10999999999999979,0.2,0.0,0.0,0.0,297.30909090909097,1635.2000000000003,3270.4000000000005,0.0,0.0,0.0,46439570.4,46454466.6,46456380.0,1940.6666666666672,11299.580000000002,13680.2,16.5,35.83999999999999,41.99999999999999
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,720,1141.52,1210.62,1210.8,0.06000000000000001,0.2,0.2,0.0,0.0,0.0,327.0400000000001,1798.7199999999968,3270.4000000000005,0.0,0.0,0.0,44743050.0,44757778.6,44759728.0,872.5400000000006,4761.229999999995,8583.800000000007,14.919999999999998,27.009999999999973,37.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,2584,1184.6000000000001,1210.47,1211.0,0.03529411764705883,0.2,0.2,0.0,0.0,0.0,34.4470588235294,100.79999999999939,883.1999999999999,0.0,0.0,0.0,46582295.52941176,46586356.0,46586356.0,377.5411764705882,1431.4499999999923,8289.0,10.841176470588234,15.229999999999995,22.4
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,2448,1182.3235294117646,1210.0,1210.2,0.011764705882352941,0.06999999999999958,0.2,0.0,0.0,0.0,34.44705882352941,271.3599999999999,620.8000000000001,0.0,0.0,0.0,44881817.88235294,44885672.0,44885672.0,348.7588235294118,1094.469999999994,8625.2,11.129411764705884,18.909999999999997,20.999999999999996
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,2204,1214.6758620689654,1220.12,1220.4,0.013793103448275864,0.11999999999999958,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46603182.06896552,46603218.4,46603220.0,214.6758620689655,201.43999999999997,1574.0,9.682758620689656,14.839999999999996,15.4
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,2088,1204.3999999999999,1210.1200000000001,1210.4,0.020689655172413796,0.2,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44902315.862068966,44902356.0,44902356.0,170.98620689655175,146.44,1531.9999999999998,6.931034482758621,11.84,12.6
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,2888,1200.7473684210527,1223.2599999999998,1223.6,0.021052631578947368,0.2,0.2,0.0,0.0,0.0,11.452631578947367,0.0,435.19999999999993,0.0,0.0,0.0,46611446.84210526,46618484.0,46618484.0,281.61,426.2999999999979,4255.0,8.715789473684211,17.47999999999999,27.599999999999998
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,2736,1189.6052631578948,1210.43,1211.4,0.021052631578947368,0.2,0.2,0.0,0.0,0.0,11.452631578947367,0.0,435.19999999999993,0.0,0.0,0.0,44911026.94736842,44918376.0,44918376.0,249.65263157894734,289.9599999999987,4796.0,6.526315789473684,11.919999999999998,18.8
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,749,1210.66,1217.6599999999999,1223.6,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46604025.6,46607275.2,46610400.0,34.617999999999995,109.58999999999983,175.2,8.1,10.79,11.6
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,648,1207.4,1210.92,1211.4,0.022222222222222223,0.11999999999999994,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44902633.333333336,44903531.2,44904240.0,22.133333333333336,88.47999999999996,137.60000000000002,9.244444444444445,12.280000000000001,12.600000000000001
|
||||||
|
@ -0,0 +1,33 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.344697344,bench_main_marker_plus_duration,head,mlx5_0,66,327229799692.0,327241990572.0,36.69282364341205,36.69419062713517,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.344697344,bench_main_marker_plus_duration,head,mlx5_3,66,327393677184.0,327405760832.0,36.71119950013029,36.712554459750265,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.149780736,bench_main_marker_plus_duration,worker,mlx5_0,68,327312761288.0,327298289132.0,36.80267266064959,36.801045427975055,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.149780736,bench_main_marker_plus_duration,worker,mlx5_3,68,327195798972.0,327191949064.0,36.78952155156224,36.789088672308345,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.07872896,bench_main_marker_plus_duration,head,mlx5_0,75,444257636548.0,444258274080.0,43.83469176160109,43.834754666583265,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.07872896,bench_main_marker_plus_duration,head,mlx5_3,75,445471520584.0,445472014020.0,43.95446512771776,43.95451381481548,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,80.561727488,bench_main_marker_plus_duration,worker,mlx5_0,77,450403115384.0,450403670936.0,44.72626190406251,44.7263170718964,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,80.561727488,bench_main_marker_plus_duration,worker,mlx5_3,77,444176313384.0,444176585108.0,44.10792342556576,44.10795040850254,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,205.0068928,bench_main_marker_plus_duration,head,mlx5_0,188,484528783604.0,484522438956.0,18.907804590812276,18.907557003117507,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,205.0068928,bench_main_marker_plus_duration,head,mlx5_3,188,484292016064.0,484262756928.0,18.898565192594344,18.897423411043476,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,204.622719744,bench_main_marker_plus_duration,worker,mlx5_0,194,483378692900.0,483384132588.0,18.898339089803788,18.89855176171067,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,204.622719744,bench_main_marker_plus_duration,worker,mlx5_3,194,483484778080.0,483489710268.0,18.902486632369254,18.902679462882155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.080653568,bench_main_marker_plus_duration,head,mlx5_0,45,428992803940.0,428992820060.0,71.37886398874002,71.3788666708999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.080653568,bench_main_marker_plus_duration,head,mlx5_3,45,429439114384.0,429439126660.0,71.45312428445231,71.45312632702023,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.829253632,bench_main_marker_plus_duration,worker,mlx5_0,47,430547106324.0,430552482856.0,70.53920742984172,70.54008830048383,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.829253632,bench_main_marker_plus_duration,worker,mlx5_3,47,430553276516.0,430553408016.0,70.54021833073266,70.54023987519466,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,167.623034624,bench_main_marker_plus_duration,head,mlx5_0,154,1738843459984.0,1738965693932.0,82.98828207636016,82.99411583057059,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,167.623034624,bench_main_marker_plus_duration,head,mlx5_3,154,1739263067700.0,1739364620460.0,83.00830833191347,83.01315505290157,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,166.729046016,bench_main_marker_plus_duration,worker,mlx5_0,158,1739033672624.0,1739033174332.0,83.44238579555551,83.4423618864881,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,166.729046016,bench_main_marker_plus_duration,worker,mlx5_3,158,1739243327096.0,1739243022428.0,83.45244544512514,83.45243082653253,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,142.715803392,bench_main_marker_plus_duration,head,mlx5_0,131,678101307872.0,678090133532.0,38.011280699416155,38.010654316647916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,142.715803392,bench_main_marker_plus_duration,head,mlx5_3,131,677949641072.0,677941962640.0,38.00277894718437,38.00234852914697,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.266830336,bench_main_marker_plus_duration,worker,mlx5_0,137,680623741960.0,680629335892.0,37.74249370419051,37.742803903394964,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.266830336,bench_main_marker_plus_duration,worker,mlx5_3,137,680750304048.0,680766879432.0,37.74951192661656,37.75043107810614,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,188.527344896,bench_main_marker_plus_duration,head,mlx5_0,173,1102872843476.0,1102880032984.0,46.79948552118605,46.79979060193723,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,188.527344896,bench_main_marker_plus_duration,head,mlx5_3,173,1103015616220.0,1103029812428.0,46.80554396301384,46.806146367211824,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,189.088602368,bench_main_marker_plus_duration,worker,mlx5_0,179,1102994734488.0,1102998788292.0,46.66573111969494,46.665902628879486,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,189.088602368,bench_main_marker_plus_duration,worker,mlx5_3,179,1103130615912.0,1103144127560.0,46.67148001930277,46.67205167292254,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.866369024,bench_main_marker_plus_duration,head,mlx5_0,42,415875487184.0,415864521032.0,74.15362486080193,74.15166951612152,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.866369024,bench_main_marker_plus_duration,head,mlx5_3,42,416276974024.0,416281015316.0,74.22521288519235,74.22593347695637,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.584492544,bench_main_marker_plus_duration,worker,mlx5_0,43,415451144696.0,415441323744.0,74.5463045090838,74.54454229061909,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.584492544,bench_main_marker_plus_duration,worker,mlx5_3,43,415600251668.0,415656009484.0,74.57305945700259,74.58306433768075,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
@ -0,0 +1,9 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||||
|
fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,1785487808318962176,1785487881306871040,72.987908864,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/meta.json
|
||||||
|
fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,1785488175173981952,1785488256333826048,81.159844096,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/meta.json
|
||||||
|
fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,1785487926490086144,1785488132860833024,206.37074688,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/meta.json
|
||||||
|
fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,1785487502091363840,1785487551713931008,49.622567168,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/meta.json
|
||||||
|
fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,1785487595012883968,1785487763299520000,168.286636032,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/meta.json
|
||||||
|
mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,1785488301482129152,1785488445978530048,144.496400896,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/meta.json
|
||||||
|
mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,1785488479288048896,1785488669308148992,190.020100096,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/meta.json
|
||||||
|
mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,1785488519428990976,1785488565490817024,46.061826048,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/meta.json
|
||||||
|
@ -0,0 +1,37 @@
|
|||||||
|
wall_time_ns,node,collector,status,pid
|
||||||
|
1785487450093977528,head,gpu_samples.csv,STARTED,1795123
|
||||||
|
1785487450108679448,head,rdma.csv,STARTED,1795134
|
||||||
|
1785487450121779856,head,dcgm_dmon.log,STARTED,1795147
|
||||||
|
1785487450134856077,head,mpstat.log,STARTED,1795163
|
||||||
|
1785487450147239051,head,pidstat.log,STARTED,1795181
|
||||||
|
1785487450160535144,head,sar_net.log,STARTED,1795207
|
||||||
|
1785487450175160784,head,perf_stat.log,STARTED,1795238
|
||||||
|
1785487450193437502,head,docker_top.log,STARTED,1795276
|
||||||
|
1785487450214351210,head,numa_samples.csv,STARTED,1795348
|
||||||
|
1785487450236070096,worker,gpu_samples.csv,STARTED,1795394
|
||||||
|
1785487450254195207,worker,rdma.csv,STARTED,1795451
|
||||||
|
1785487450270545415,worker,dcgm_dmon.log,STARTED,1795504
|
||||||
|
1785487450285967202,worker,mpstat.log,STARTED,1795552
|
||||||
|
1785487450301984460,worker,pidstat.log,STARTED,1795567
|
||||||
|
1785487450317255696,worker,sar_net.log,STARTED,1795577
|
||||||
|
1785487450331874012,worker,perf_stat.log,STARTED,1795584
|
||||||
|
1785487450348717530,worker,docker_top.log,STARTED,1795593
|
||||||
|
1785487450364742164,worker,numa_samples.csv,STARTED,1795604
|
||||||
|
1785488689789962654,head,gpu_samples.csv,STOPPED,1795123
|
||||||
|
1785488689794788834,head,rdma.csv,STOPPED,1795134
|
||||||
|
1785488689801075236,head,dcgm_dmon.log,STOPPED,1795147
|
||||||
|
1785488689805422095,head,mpstat.log,STOPPED,1795163
|
||||||
|
1785488689810734986,head,pidstat.log,STOPPED,1795181
|
||||||
|
1785488689814680689,head,sar_net.log,STOPPED,1795207
|
||||||
|
1785488689951029262,head,perf_stat.log,STOPPED,1795238
|
||||||
|
1785488689955936272,head,docker_top.log,STOPPED,1795276
|
||||||
|
1785488689960978121,head,numa_samples.csv,STOPPED,1795348
|
||||||
|
1785488689965181539,worker,gpu_samples.csv,STOPPED,1795394
|
||||||
|
1785488689969473724,worker,rdma.csv,STOPPED,1795451
|
||||||
|
1785488689973568170,worker,dcgm_dmon.log,STOPPED,1795504
|
||||||
|
1785488689977772276,worker,mpstat.log,STOPPED,1795552
|
||||||
|
1785488689982041873,worker,pidstat.log,STOPPED,1795567
|
||||||
|
1785488689986056700,worker,sar_net.log,STOPPED,1795577
|
||||||
|
1785488689990039780,worker,perf_stat.log,STOPPED,1795584
|
||||||
|
1785488689994084643,worker,docker_top.log,STOPPED,1795593
|
||||||
|
1785488689998332256,worker,numa_samples.csv,STOPPED,1795604
|
||||||
|
@ -0,0 +1,20 @@
|
|||||||
|
test,scope,nccl_cross_nic,size_bytes,repetitions,mean_ms,algbw_GBps_mean,busbw_GBps_mean,busbw_GBps_min,wrong_values,path_class,samples,bandwidth_GBps_mean,bandwidth_GBps_p05,bandwidth_GBps_min
|
||||||
|
all_reduce,head_8gpu,2,1048576,3,1.1792767981688181,0.9351587648306614,1.6365278384536577,1.1619881842280209,0,,,,,
|
||||||
|
all_reduce,head_8gpu,2,1073741824,3,47.2596331278483,22.72016498475108,39.76028872331439,39.67252737812032,0,,,,,
|
||||||
|
all_reduce,head_8gpu,2,67108864,3,3.107903989156087,21.596044891582753,37.79307856026982,37.253524087802575,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,0,1048576,3,1.174121594429016,0.9225129006791467,1.7297116887734,1.3203079074091486,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,0,1073741824,3,51.1693105061849,20.984136487918786,39.34525591484772,39.296690886871644,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,0,67108864,3,3.4025983730951945,19.72330661126499,36.98119989612186,36.724996463125755,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,1,1048576,3,1.2971573332945505,0.877370746063573,1.6450701488691994,1.0568849765163506,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,1,1073741824,3,50.73157666524251,21.165255754284047,39.68485453928258,39.59303458131569,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,1,67108864,3,3.3778613169988,19.86842574374279,37.25329826951773,36.858805194578224,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,2,1048576,3,1.1436874707539877,0.936877857865813,1.7566459834983992,1.413443823860872,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,2,1073741824,3,50.931247075398765,21.082601839476187,39.52987844901785,39.33647232214203,0,,,,,
|
||||||
|
all_reduce,two_node_16gpu,2,67108864,3,3.4341418504714967,19.544645327030477,36.64620998818214,36.14018010573151,0,,,,,
|
||||||
|
all_reduce,worker_8gpu,2,1048576,3,0.9693984071413676,1.0921968526479142,1.9113444921338498,1.664030821168365,0,,,,,
|
||||||
|
all_reduce,worker_8gpu,2,1073741824,3,47.27112070719401,22.714924401498006,39.75111770262151,39.55926639799348,0,,,,,
|
||||||
|
all_reduce,worker_8gpu,2,67108864,3,3.0737973292668657,21.837161295289466,38.215032266756566,37.43531449904809,0,,,,,
|
||||||
|
p2p_copy,head,,268435456,,,,,,,cross_numa_sys,32,52.397332123279426,52.19168649638452,52.18129785806688
|
||||||
|
p2p_copy,head,,268435456,,,,,,,same_pcie_switch,24,53.6126013471359,53.36177601211484,53.35073474580275
|
||||||
|
p2p_copy,worker,,268435456,,,,,,,cross_numa_sys,32,52.31572860916867,52.14858683841258,52.07724663835188
|
||||||
|
p2p_copy,worker,,268435456,,,,,,,same_pcie_switch,24,53.50030938041238,53.28731762907832,53.221028060678236
|
||||||
|
@ -0,0 +1,17 @@
|
|||||||
|
node,gpu,samples,gpu_util_pct_mean,gpu_util_pct_p95,gpu_util_pct_max,memory_util_pct_mean,memory_util_pct_p95,memory_util_pct_max,memory_used_mib_mean,memory_used_mib_p95,memory_used_mib_max,power_w_mean,power_w_p95,power_w_max,temperature_c_mean,temperature_c_p95,temperature_c_max,sm_clock_mhz_mean,sm_clock_mhz_p95,sm_clock_mhz_max,memory_clock_mhz_mean,memory_clock_mhz_p95,memory_clock_mhz_max
|
||||||
|
head,0,885,73.19774011299435,100.0,100.0,38.57514124293785,58.0,64.0,83114.45536723164,83361.0,83361.0,205.47479096045197,307.69199999999995,333.9,43.47570621468927,52.0,55.0,2393.25197740113,2422.0,2430.0,12481.0,12481.0,12481.0
|
||||||
|
head,1,885,73.50056497175142,100.0,100.0,37.15819209039548,56.0,62.0,83080.49830508475,83327.0,83327.0,206.62670056497177,313.188,350.0,43.737853107344634,53.0,56.0,2378.2225988700566,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
head,2,885,73.52542372881356,100.0,100.0,38.364971751412426,57.0,63.0,83048.5209039548,83295.0,83295.0,210.91588700564972,317.204,371.02,42.911864406779664,52.0,55.0,2391.7344632768363,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
head,3,885,73.10960451977401,100.0,100.0,38.39322033898305,58.0,65.0,83146.58870056497,83393.0,83393.0,204.69710734463277,310.6,340.24,43.44858757062147,52.0,55.0,2388.697175141243,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
head,4,885,73.17062146892656,100.0,100.0,39.108474576271185,58.0,65.0,83194.63389830508,83441.0,83441.0,207.94813559322034,316.352,339.4,44.054237288135596,53.0,56.0,2402.890395480226,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
head,5,885,73.44519774011299,100.0,100.0,38.371751412429376,57.0,63.0,83080.63389830508,83327.0,83327.0,209.06979661016948,317.16999999999996,376.49,43.85084745762712,53.0,55.0,2399.3717514124296,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
head,6,885,73.67570621468927,100.0,100.0,39.31412429378531,58.0,65.0,83048.65423728814,83295.0,83295.0,205.60972881355931,310.78,342.99,43.70395480225989,52.0,55.0,2405.515254237288,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
head,7,885,73.06553672316385,100.0,100.0,35.74124293785311,54.0,60.0,83227.83163841808,83473.0,83473.0,206.52320903954802,314.84799999999996,336.9,43.929943502824855,53.0,56.0,2388.154802259887,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
worker,0,913,73.23986856516977,100.0,100.0,37.39539978094195,56.0,62.0,83195.78422782038,83441.0,83441.0,210.65017524644028,326.81399999999996,347.67,46.10624315443592,56.0,60.0,2398.8400876232204,2422.0,2430.0,12481.0,12481.0,12481.0
|
||||||
|
worker,1,913,73.35158817086528,100.0,100.0,38.92880613362541,58.0,64.0,83081.95947426069,83327.0,83327.0,202.77134720700985,310.3,335.13,45.61226725082147,55.0,58.0,2398.3099671412924,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
worker,2,913,73.34720700985761,100.0,100.0,38.708652792990144,58.0,64.0,83050.44140197152,83295.0,83295.0,203.5297152245345,312.52,373.73,46.55859802847755,57.0,60.0,2400.135815991238,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
worker,3,913,72.96276013143483,100.0,100.0,37.24096385542169,57.0,63.0,83229.27382256297,83473.0,83473.0,207.60696604600219,318.69399999999996,359.03,45.467688937568454,55.0,59.0,2409.73713033954,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
worker,4,913,72.84008762322016,100.0,100.0,36.52464403066813,55.0,63.0,83117.25410733845,83361.0,83361.0,204.54877327491786,314.376,347.28,45.67798466593647,56.0,59.0,2392.5739320920043,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
worker,5,913,73.26944140197152,100.0,100.0,38.52245345016429,58.0,64.0,83083.19058050384,83327.0,83327.0,204.3163417305586,315.148,356.12,46.33077765607886,56.0,60.0,2385.7513691128147,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
worker,6,913,73.3274917853231,100.0,100.0,38.77984665936473,58.0,64.0,83052.13253012048,83295.0,83295.0,206.93734939759034,319.012,355.56,46.1829134720701,56.0,59.0,2401.7447973713033,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
worker,7,913,73.27929901423877,100.0,100.0,37.28806133625411,57.0,64.0,83150.0230010953,83393.0,83393.0,206.63869660460023,320.18,353.19,47.00766703176342,57.0,61.0,2406.594742606791,2422.0,2422.0,12481.0,12481.0,12481.0
|
||||||
|
@ -0,0 +1,26 @@
|
|||||||
|
{
|
||||||
|
"run_id": "dsv4pro-phase2-20260731-163620",
|
||||||
|
"status": "COMPLETED",
|
||||||
|
"created_at": "2026-07-31T16:36:20+08:00",
|
||||||
|
"completed_at": "2026-07-31T17:05:04+08:00",
|
||||||
|
"experiment": "dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution",
|
||||||
|
"git_commit": "5f24b7d22f98108f6cc234edba6768d55ea0a962",
|
||||||
|
"git_dirty": false,
|
||||||
|
"phase1_entry": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh",
|
||||||
|
"head_node": "10.101.0.11",
|
||||||
|
"worker_node": "10.101.0.13",
|
||||||
|
"fixed_case_ids": "long_prefill_latency_128k_c1,mid_prefill_throughput_32k_c16,decode_throughput_1k_to_1k_c32,long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1",
|
||||||
|
"run_mixed_case": true,
|
||||||
|
"sample_interval_s": 1,
|
||||||
|
"cpu_sample_interval_s": 5,
|
||||||
|
"process_sample_interval_s": 5,
|
||||||
|
"net_sample_interval_s": 5,
|
||||||
|
"perf_interval_ms": 5000,
|
||||||
|
"numastat_interval_s": 5,
|
||||||
|
"clock_skew_tolerance_s": 2,
|
||||||
|
"idle_baseline_s": 15,
|
||||||
|
"post_run_cooldown_s": 15,
|
||||||
|
"require_precise_windows": true,
|
||||||
|
"run_communication_baseline": true,
|
||||||
|
"dry_run": false
|
||||||
|
}
|
||||||
@ -0,0 +1,5 @@
|
|||||||
|
node,hca,samples,duration_s,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||||
|
head,mlx5_0,1131,1239.614615552,5264991004888.0,5264992389656.0,33.97824413384156,33.978253070606144,0.0,0.0,0.0,0.0,,,,,,
|
||||||
|
head,mlx5_3,1130,1238.507734784,5266283609308.0,5266274185868.0,34.01696064644414,34.0168997768041,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
worker,mlx5_0,1168,1238.553399296,5272091713408.0,5272078266720.0,34.05322187257931,34.05313501842828,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
worker,mlx5_3,1168,1238.553399296,5264840174196.0,5264861659552.0,34.00638310589474,34.00652188299398,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
@ -0,0 +1,253 @@
|
|||||||
|
# Phase 2 Hardware Attribution
|
||||||
|
|
||||||
|
- Generated: `2026-07-31T17:05:04+08:00`
|
||||||
|
- Bench rows: `8`
|
||||||
|
- Failed bench rows: `0`
|
||||||
|
- GPU summary rows: `16`
|
||||||
|
- RDMA summary rows: `4`
|
||||||
|
- Case windows: `8`
|
||||||
|
- Precise main-run windows: `8/8`
|
||||||
|
- Collector status counts: `{"STARTED": 18, "STOPPED": 18}`
|
||||||
|
|
||||||
|
## 1. Benchmark results
|
||||||
|
|
||||||
|
| Run | Case | Status | Input TPS | Output TPS | TTFT P95 (ms) | TPOT P95 (ms) |
|
||||||
|
|---|---|---|---:|---:|---:|---:|
|
||||||
|
| fixed_decode_throughput_1k_to_1k_c32 | decode_throughput_1k_to_1k_c32 | COMPLETED | 448.95107215015315 | 448.95107215015315 | 10166.788510262268 | 65.35758210354297 |
|
||||||
|
| fixed_long_context_decode_128k_to_1k_c1 | long_context_decode_128k_to_1k_c1 | COMPLETED | 1614.985848663526 | 12.617076942683797 | 48279.420554987155 | 32.106360114411245 |
|
||||||
|
| fixed_long_output_decode_1k_to_4k_c16 | long_output_decode_1k_to_4k_c16 | COMPLETED | 79.3910971002871 | 317.5643884011484 | 1723.5681610036409 | 49.98305734157135 |
|
||||||
|
| fixed_long_prefill_latency_128k_c1 | long_prefill_latency_128k_c1 | COMPLETED | 2641.3788467537956 | 0.02015212132838284 | 49610.14223104576 | 0.0 |
|
||||||
|
| fixed_mid_prefill_throughput_32k_c16 | mid_prefill_throughput_32k_c16 | COMPLETED | 3115.446435236942 | 0.09507587998159613 | 161938.13879448862 | 0.0 |
|
||||||
|
| mixed_prefill_decode_interference | decode_control_1k_to_1k_c32 | COMPLETED | 453.54762732999035 | 453.54762732999035 | 9442.68154159945 | 66.23843437823616 |
|
||||||
|
| mixed_prefill_decode_interference | decode_with_128k_prefill_1k_to_1k_c32 | COMPLETED | 344.8898309034744 | 344.8898309034744 | 9891.837346865213 | 110.451060616212 |
|
||||||
|
| mixed_prefill_decode_interference | long_prefill_injection_128k_to_1_c1 | COMPLETED | 2845.5667477625557 | 0.02170995138368649 | 45980.19455798203 | 0.0 |
|
||||||
|
|
||||||
|
## 2. Measurement-window validity
|
||||||
|
|
||||||
|
| Case | Role | Duration (s) | Window source |
|
||||||
|
|---|---|---:|---|
|
||||||
|
| decode_throughput_1k_to_1k_c32 | - | 72.99 | bench_main_marker_plus_duration |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | - | 81.16 | bench_main_marker_plus_duration |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | - | 206.37 | bench_main_marker_plus_duration |
|
||||||
|
| long_prefill_latency_128k_c1 | - | 49.62 | bench_main_marker_plus_duration |
|
||||||
|
| mid_prefill_throughput_32k_c16 | - | 168.29 | bench_main_marker_plus_duration |
|
||||||
|
| decode_control_1k_to_1k_c32 | control | 144.50 | bench_main_marker_plus_duration |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | decode_background | 190.02 | bench_main_marker_plus_duration |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | prefill_injection | 46.06 | bench_main_marker_plus_duration |
|
||||||
|
|
||||||
|
## 3. GPU basic state (`nvidia-smi`)
|
||||||
|
|
||||||
|
Data: `case_gpu_node_summary.csv`; raw: `head|worker/gpu_samples.csv`.
|
||||||
|
|
||||||
|
| Case | Node | Samples | GPU util mean/p95 (%) | Memory used mean (MiB) | Power mean (W) | SM clock mean (MHz) |
|
||||||
|
|---|---|---:|---:|---:|---:|---:|
|
||||||
|
| decode_throughput_1k_to_1k_c32 | head | 408 | 97.09/100.00 | 83300.00 | 230.22 | 2402.95 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | worker | 432 | 97.72/100.00 | 83300.00 | 221.83 | 2418.55 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | head | 456 | 97.70/100.00 | 83219.06 | 257.12 | 2412.63 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | worker | 472 | 97.43/100.00 | 83221.32 | 257.90 | 2417.53 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | head | 1184 | 98.83/100.00 | 83300.00 | 220.83 | 2402.76 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | worker | 1216 | 99.25/100.00 | 83300.00 | 215.74 | 2412.57 |
|
||||||
|
| long_prefill_latency_128k_c1 | head | 288 | 94.44/100.00 | 83028.22 | 293.40 | 2392.22 |
|
||||||
|
| long_prefill_latency_128k_c1 | worker | 288 | 94.85/100.00 | 83057.76 | 297.91 | 2398.02 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | head | 952 | 98.21/100.00 | 83231.07 | 299.46 | 2417.69 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | worker | 968 | 99.07/100.00 | 83231.94 | 302.56 | 2419.33 |
|
||||||
|
| decode_control_1k_to_1k_c32 | head | 824 | 98.52/100.00 | 83364.00 | 231.92 | 2405.39 |
|
||||||
|
| decode_control_1k_to_1k_c32 | worker | 856 | 98.60/100.00 | 83364.00 | 227.79 | 2418.59 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | head | 1072 | 99.61/100.00 | 83259.71 | 248.31 | 2411.49 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | worker | 1112 | 98.92/100.00 | 83256.91 | 247.75 | 2420.77 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | head | 264 | 99.95/100.00 | 83281.49 | 301.20 | 2421.11 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | worker | 272 | 99.89/100.00 | 83282.94 | 307.16 | 2421.46 |
|
||||||
|
|
||||||
|
## 4. GPU profiling counters (DCGM)
|
||||||
|
|
||||||
|
Data: `case_dcgm_summary.csv`; raw: `head|worker/dcgm_dmon.log`.
|
||||||
|
|
||||||
|
| Case | Node | Samples | GR active | SM active | SM occupancy | Tensor active | DRAM active | PCIe TX/RX mean (GB/s) |
|
||||||
|
|---|---|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| decode_throughput_1k_to_1k_c32 | head | 584 | 0.971 | 0.525 | 0.141 | 0.059 | 0.417 | 5.205/5.201 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | worker | 576 | 0.979 | 0.523 | 0.141 | 0.058 | 0.415 | 5.236/5.232 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | head | 648 | 0.961 | 0.578 | 0.211 | 0.095 | 0.399 | 6.764/6.821 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | worker | 648 | 0.965 | 0.579 | 0.211 | 0.095 | 0.401 | 6.780/6.838 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | head | 1648 | 0.989 | 0.490 | 0.133 | 0.045 | 0.414 | 2.621/2.634 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | worker | 1640 | 0.990 | 0.489 | 0.133 | 0.045 | 0.414 | 2.619/2.633 |
|
||||||
|
| long_prefill_latency_128k_c1 | head | 400 | 0.930 | 0.683 | 0.251 | 0.136 | 0.374 | 10.537/10.582 |
|
||||||
|
| long_prefill_latency_128k_c1 | worker | 392 | 0.936 | 0.686 | 0.251 | 0.136 | 0.376 | 10.618/10.632 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | head | 1344 | 0.980 | 0.713 | 0.278 | 0.127 | 0.440 | 12.664/12.720 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | worker | 1344 | 0.984 | 0.715 | 0.278 | 0.127 | 0.441 | 12.699/12.755 |
|
||||||
|
| decode_control_1k_to_1k_c32 | head | 1152 | 0.989 | 0.539 | 0.146 | 0.061 | 0.427 | 5.451/5.456 |
|
||||||
|
| decode_control_1k_to_1k_c32 | worker | 1152 | 0.990 | 0.536 | 0.145 | 0.060 | 0.425 | 5.465/5.461 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | head | 1520 | 0.977 | 0.581 | 0.174 | 0.080 | 0.416 | 6.865/6.868 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | worker | 1520 | 0.978 | 0.578 | 0.174 | 0.080 | 0.416 | 6.865/6.872 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | head | 368 | 0.997 | 0.723 | 0.264 | 0.138 | 0.407 | 11.151/11.198 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | worker | 368 | 0.997 | 0.720 | 0.262 | 0.137 | 0.407 | 11.069/11.110 |
|
||||||
|
|
||||||
|
## 5. CPU, process, and `perf`
|
||||||
|
|
||||||
|
Data: `case_cpu_summary.csv`, `case_process_summary.csv`, `case_perf_summary.csv`; raw: `mpstat.log`, `pidstat.log`, `perf_stat.log`.
|
||||||
|
|
||||||
|
| Case | Node | Samples CPU/process/perf | CPU active mean/p95 (%) | Hot cores max | Process CPU max (%) | Process wait max (%) | IPC | Context switches mean/interval |
|
||||||
|
|---|---|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| decode_throughput_1k_to_1k_c32 | head | 15/1140/84 | 10.13/10.42 | 12 | 1225.00 | 0.20 | 3.049 | 41240.93 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | worker | 15/1080/84 | 9.84/10.09 | 12 | 1210.20 | 0.20 | 3.068 | 39682.00 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | head | 17/1292/96 | 9.90/10.29 | 12 | 1222.40 | 0.20 | 2.887 | 44877.38 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | worker | 17/1224/96 | 9.65/10.09 | 12 | 1215.60 | 0.00 | 2.902 | 43686.69 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | head | 41/3116/246 | 10.36/10.52 | 12 | 1225.80 | 0.20 | 3.098 | 43749.98 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | worker | 41/2952/246 | 9.99/10.09 | 12 | 1211.40 | 0.20 | 3.117 | 40973.46 |
|
||||||
|
| long_prefill_latency_128k_c1 | head | 10/1129/60 | 10.00/10.27 | 12 | 1211.20 | 0.20 | 2.727 | 50995.80 |
|
||||||
|
| long_prefill_latency_128k_c1 | worker | 10/720/60 | 9.69/10.18 | 12 | 1210.80 | 0.20 | 2.749 | 48210.50 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | head | 34/2584/204 | 9.99/10.27 | 12 | 1211.00 | 0.20 | 2.802 | 39421.29 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | worker | 34/2448/204 | 9.84/10.11 | 12 | 1210.20 | 0.20 | 2.810 | 39129.38 |
|
||||||
|
| decode_control_1k_to_1k_c32 | head | 29/2204/174 | 10.26/10.44 | 12 | 1220.40 | 0.20 | 3.042 | 42398.83 |
|
||||||
|
| decode_control_1k_to_1k_c32 | worker | 29/2088/174 | 9.98/10.11 | 12 | 1210.40 | 0.20 | 3.072 | 40484.59 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | head | 38/2888/228 | 10.34/11.37 | 13 | 1223.60 | 0.20 | 2.980 | 41223.05 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | worker | 38/2736/228 | 9.87/10.09 | 12 | 1211.40 | 0.20 | 3.005 | 39834.92 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | head | 10/749/54 | 10.21/10.63 | 13 | 1223.60 | 0.00 | 2.808 | 33399.67 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | worker | 9/648/60 | 10.04/10.12 | 12 | 1211.40 | 0.20 | 2.830 | 33911.70 |
|
||||||
|
|
||||||
|
## 6. NUMA memory placement
|
||||||
|
|
||||||
|
Data/raw: `case_numa_summary.csv`, `head|worker/numa_samples.csv`.
|
||||||
|
|
||||||
|
| Case | Node | Samples | Node 0 mean (MiB) | Node 1 mean (MiB) | Total mean (MiB) | Imbalance mean/max (%) |
|
||||||
|
|---|---|---:|---:|---:|---:|---:|
|
||||||
|
| decode_throughput_1k_to_1k_c32 | head | 13 | 17941.22 | 27565.28 | 45506.51 | 21.15/21.15 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | worker | 13 | 19029.49 | 24814.32 | 43843.80 | 13.19/13.19 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | head | 13 | 17946.50 | 27569.43 | 45515.94 | 21.14/21.15 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | worker | 14 | 19037.22 | 24816.07 | 43853.29 | 13.18/13.18 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | head | 35 | 17941.67 | 27565.20 | 45506.88 | 21.15/21.17 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | worker | 37 | 19029.40 | 24814.58 | 43843.96 | 13.19/13.20 |
|
||||||
|
| long_prefill_latency_128k_c1 | head | 8 | 17869.37 | 27486.32 | 45355.71 | 21.20/21.21 |
|
||||||
|
| long_prefill_latency_128k_c1 | worker | 8 | 18970.34 | 24731.71 | 43702.05 | 13.18/13.21 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | head | 28 | 17940.27 | 27563.88 | 45504.17 | 21.15/21.15 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | worker | 30 | 19032.02 | 24805.59 | 43837.58 | 13.17/13.22 |
|
||||||
|
| decode_control_1k_to_1k_c32 | head | 24 | 17946.96 | 27573.84 | 45520.81 | 21.15/21.21 |
|
||||||
|
| decode_control_1k_to_1k_c32 | worker | 26 | 19039.33 | 24818.28 | 43857.62 | 13.18/13.18 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | head | 32 | 17955.73 | 27573.17 | 45528.91 | 21.12/21.13 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | worker | 34 | 19042.79 | 24823.40 | 43866.18 | 13.18/13.19 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | head | 8 | 17951.77 | 27569.78 | 45521.56 | 21.13/21.13 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | worker | 8 | 19038.43 | 24819.53 | 43857.94 | 13.18/13.18 |
|
||||||
|
|
||||||
|
## 7. Linux netdev and RDMA data path
|
||||||
|
|
||||||
|
Netdev data: `case_netdev_summary.csv`; RDMA data: `case_rdma_summary.csv`; raw: `sar_net.log`, `rdma.csv`.
|
||||||
|
|
||||||
|
### Linux interfaces
|
||||||
|
|
||||||
|
| Case | Node | Interface | Samples | RX mean/max (Gbit/s) | TX mean/max (Gbit/s) | Util max (%) | RX/TX error max (/s) |
|
||||||
|
|---|---|---|---:|---:|---:|---:|---:|
|
||||||
|
| decode_throughput_1k_to_1k_c32 | head | eth0 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | head | eth3 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | worker | eth0 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | worker | eth3 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | head | eth0 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | head | eth3 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | worker | eth0 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | worker | eth3 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | head | eth0 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | head | eth3 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | worker | eth0 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | worker | eth3 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_prefill_latency_128k_c1 | head | eth0 | 20 | 0.000/0.001 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_prefill_latency_128k_c1 | head | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_prefill_latency_128k_c1 | worker | eth0 | 20 | 0.000/0.000 | 0.000/0.001 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_prefill_latency_128k_c1 | worker | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | head | eth0 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | head | eth3 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | worker | eth0 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | worker | eth3 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_control_1k_to_1k_c32 | head | eth0 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_control_1k_to_1k_c32 | head | eth3 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_control_1k_to_1k_c32 | worker | eth0 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_control_1k_to_1k_c32 | worker | eth3 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | head | eth0 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | head | eth3 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | worker | eth0 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | worker | eth3 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | head | eth0 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | head | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | worker | eth0 | 18 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | worker | eth3 | 18 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 |
|
||||||
|
|
||||||
|
### RDMA HCAs
|
||||||
|
|
||||||
|
| Case | Node | HCA | Samples | TX/RX (Gbit/s) | Wait delta | Discard/error delta | Retry exceeded delta |
|
||||||
|
|---|---|---|---:|---:|---:|---:|---:|
|
||||||
|
| decode_throughput_1k_to_1k_c32 | head | mlx5_0 | 66 | 36.69/36.69 | 0 | 0/0 | 0 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | head | mlx5_3 | 66 | 36.71/36.71 | 0 | 0/0 | 0 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | worker | mlx5_0 | 68 | 36.80/36.80 | 0 | 0/0 | 0 |
|
||||||
|
| decode_throughput_1k_to_1k_c32 | worker | mlx5_3 | 68 | 36.79/36.79 | 0 | 0/0 | 0 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | head | mlx5_0 | 75 | 43.83/43.83 | 0 | 0/0 | 0 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | head | mlx5_3 | 75 | 43.95/43.95 | 0 | 0/0 | 0 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | worker | mlx5_0 | 77 | 44.73/44.73 | 0 | 0/0 | 0 |
|
||||||
|
| long_context_decode_128k_to_1k_c1 | worker | mlx5_3 | 77 | 44.11/44.11 | 0 | 0/0 | 0 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | head | mlx5_0 | 188 | 18.91/18.91 | 0 | 0/0 | 0 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | head | mlx5_3 | 188 | 18.90/18.90 | 0 | 0/0 | 0 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | worker | mlx5_0 | 194 | 18.90/18.90 | 0 | 0/0 | 0 |
|
||||||
|
| long_output_decode_1k_to_4k_c16 | worker | mlx5_3 | 194 | 18.90/18.90 | 0 | 0/0 | 0 |
|
||||||
|
| long_prefill_latency_128k_c1 | head | mlx5_0 | 45 | 71.38/71.38 | 0 | 0/0 | 0 |
|
||||||
|
| long_prefill_latency_128k_c1 | head | mlx5_3 | 45 | 71.45/71.45 | 0 | 0/0 | 0 |
|
||||||
|
| long_prefill_latency_128k_c1 | worker | mlx5_0 | 47 | 70.54/70.54 | 0 | 0/0 | 0 |
|
||||||
|
| long_prefill_latency_128k_c1 | worker | mlx5_3 | 47 | 70.54/70.54 | 0 | 0/0 | 0 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | head | mlx5_0 | 154 | 82.99/82.99 | 0 | 0/0 | 0 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | head | mlx5_3 | 154 | 83.01/83.01 | 0 | 0/0 | 0 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | worker | mlx5_0 | 158 | 83.44/83.44 | 0 | 0/0 | 0 |
|
||||||
|
| mid_prefill_throughput_32k_c16 | worker | mlx5_3 | 158 | 83.45/83.45 | 0 | 0/0 | 0 |
|
||||||
|
| decode_control_1k_to_1k_c32 | head | mlx5_0 | 131 | 38.01/38.01 | 0 | 0/0 | 0 |
|
||||||
|
| decode_control_1k_to_1k_c32 | head | mlx5_3 | 131 | 38.00/38.00 | 0 | 0/0 | 0 |
|
||||||
|
| decode_control_1k_to_1k_c32 | worker | mlx5_0 | 137 | 37.74/37.74 | 0 | 0/0 | 0 |
|
||||||
|
| decode_control_1k_to_1k_c32 | worker | mlx5_3 | 137 | 37.75/37.75 | 0 | 0/0 | 0 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | head | mlx5_0 | 173 | 46.80/46.80 | 0 | 0/0 | 0 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | head | mlx5_3 | 173 | 46.81/46.81 | 0 | 0/0 | 0 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | worker | mlx5_0 | 179 | 46.67/46.67 | 0 | 0/0 | 0 |
|
||||||
|
| decode_with_128k_prefill_1k_to_1k_c32 | worker | mlx5_3 | 179 | 46.67/46.67 | 0 | 0/0 | 0 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | head | mlx5_0 | 42 | 74.15/74.15 | 0 | 0/0 | 0 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | head | mlx5_3 | 42 | 74.23/74.23 | 0 | 0/0 | 0 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | worker | mlx5_0 | 43 | 74.55/74.54 | 0 | 0/0 | 0 |
|
||||||
|
| long_prefill_injection_128k_to_1_c1 | worker | mlx5_3 | 43 | 74.57/74.58 | 0 | 0/0 | 0 |
|
||||||
|
|
||||||
|
## 8. PCIe P2P and NCCL communication baseline
|
||||||
|
|
||||||
|
Data: `communication_aggregate.csv`; raw: `communication/*.log`.
|
||||||
|
|
||||||
|
| Test | Scope | Path/CROSS_NIC | Samples/repetitions | Size (MiB) | Mean latency (ms) | Bandwidth / busbw (GB/s) | Minimum | Wrong values |
|
||||||
|
|---|---|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| all_reduce | head_8gpu | 2 | 3 | 1 | 1.18 | 1.64 | 1.16 | 0 |
|
||||||
|
| all_reduce | head_8gpu | 2 | 3 | 1024 | 47.26 | 39.76 | 39.67 | 0 |
|
||||||
|
| all_reduce | head_8gpu | 2 | 3 | 64 | 3.11 | 37.79 | 37.25 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 0 | 3 | 1 | 1.17 | 1.73 | 1.32 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 0 | 3 | 1024 | 51.17 | 39.35 | 39.30 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 0 | 3 | 64 | 3.40 | 36.98 | 36.72 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 1 | 3 | 1 | 1.30 | 1.65 | 1.06 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 1 | 3 | 1024 | 50.73 | 39.68 | 39.59 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 1 | 3 | 64 | 3.38 | 37.25 | 36.86 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 2 | 3 | 1 | 1.14 | 1.76 | 1.41 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 2 | 3 | 1024 | 50.93 | 39.53 | 39.34 | 0 |
|
||||||
|
| all_reduce | two_node_16gpu | 2 | 3 | 64 | 3.43 | 36.65 | 36.14 | 0 |
|
||||||
|
| all_reduce | worker_8gpu | 2 | 3 | 1 | 0.97 | 1.91 | 1.66 | 0 |
|
||||||
|
| all_reduce | worker_8gpu | 2 | 3 | 1024 | 47.27 | 39.75 | 39.56 | 0 |
|
||||||
|
| all_reduce | worker_8gpu | 2 | 3 | 64 | 3.07 | 38.22 | 37.44 | 0 |
|
||||||
|
| p2p_copy | head | cross_numa_sys | 32 | 256 | - | 52.40 | 52.18 | - |
|
||||||
|
| p2p_copy | head | same_pcie_switch | 24 | 256 | - | 53.61 | 53.35 | - |
|
||||||
|
| p2p_copy | worker | cross_numa_sys | 32 | 256 | - | 52.32 | 52.08 | - |
|
||||||
|
| p2p_copy | worker | same_pcie_switch | 24 | 256 | - | 53.50 | 53.22 | - |
|
||||||
|
|
||||||
|
## 9. Machine-readable summaries
|
||||||
|
|
||||||
|
- `gpu_summary.csv`
|
||||||
|
- `rdma_summary.csv`
|
||||||
|
- `bench_summary.csv`
|
||||||
|
- `case_windows.csv`
|
||||||
|
- `case_gpu_summary.csv`
|
||||||
|
- `case_gpu_node_summary.csv`
|
||||||
|
- `case_dcgm_summary.csv`
|
||||||
|
- `case_cpu_summary.csv`
|
||||||
|
- `case_process_summary.csv`
|
||||||
|
- `case_perf_summary.csv`
|
||||||
|
- `case_numa_summary.csv`
|
||||||
|
- `case_netdev_summary.csv`
|
||||||
|
- `case_rdma_summary.csv`
|
||||||
|
- `communication_summary.csv`
|
||||||
|
- `communication_aggregate.csv`
|
||||||
|
- `summary.json`
|
||||||
|
|
||||||
|
Each conclusion must cite the corresponding table above and its raw file; missing samples are reported as `-`, never interpreted as zero.
|
||||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
@ -0,0 +1,55 @@
|
|||||||
|
{
|
||||||
|
"generated_at": "2026-07-31T17:05:04+08:00",
|
||||||
|
"bench_rows": 8,
|
||||||
|
"failed_bench_rows": 0,
|
||||||
|
"gpu_summary_rows": 16,
|
||||||
|
"rdma_summary_rows": 4,
|
||||||
|
"case_windows": 8,
|
||||||
|
"case_gpu_summary_rows": 128,
|
||||||
|
"case_rdma_summary_rows": 32,
|
||||||
|
"case_metric_rows": {
|
||||||
|
"gpu_node": 16,
|
||||||
|
"dcgm": 16,
|
||||||
|
"cpu": 16,
|
||||||
|
"process": 16,
|
||||||
|
"perf": 16,
|
||||||
|
"numa": 16,
|
||||||
|
"netdev": 32
|
||||||
|
},
|
||||||
|
"communication_aggregate_rows": 19,
|
||||||
|
"precise_windows": 8,
|
||||||
|
"collector_status_counts": {
|
||||||
|
"STARTED": 18,
|
||||||
|
"STOPPED": 18
|
||||||
|
},
|
||||||
|
"collector_files": {
|
||||||
|
"head": {
|
||||||
|
"dcgm_dmon.log": 1642103,
|
||||||
|
"docker_top.log": 2636664,
|
||||||
|
"gpu_samples.csv": 727301,
|
||||||
|
"markers.csv": 1860,
|
||||||
|
"mpstat.log": 3923940,
|
||||||
|
"numa_samples.csv": 11356,
|
||||||
|
"perf_stat.log": 188193,
|
||||||
|
"pidstat.log": 3070380,
|
||||||
|
"rdma.csv": 230893,
|
||||||
|
"sar_net.log": 534606,
|
||||||
|
"static_after.log": 1195708,
|
||||||
|
"static_before.log": 1306541
|
||||||
|
},
|
||||||
|
"worker": {
|
||||||
|
"dcgm_dmon.log": 1613150,
|
||||||
|
"docker_top.log": 2592658,
|
||||||
|
"gpu_samples.csv": 764832,
|
||||||
|
"markers.csv": 1902,
|
||||||
|
"mpstat.log": 3988645,
|
||||||
|
"numa_samples.csv": 12614,
|
||||||
|
"perf_stat.log": 191154,
|
||||||
|
"pidstat.log": 3024655,
|
||||||
|
"rdma.csv": 243227,
|
||||||
|
"sar_net.log": 609685,
|
||||||
|
"static_after.log": 1195276,
|
||||||
|
"static_before.log": 1306123
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1 @@
|
|||||||
|
env DRY_RUN=0 RUN_ID=confirm RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/confirm.tsv FIXED_CASE_IDS=rdma_confirm_i65536_o1024_c4\,rdma_confirm_i65536_o1024_c16\,rdma_confirm_i65536_o1024_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=2 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
|
||||||
@ -0,0 +1 @@
|
|||||||
|
env DRY_RUN=0 RUN_ID=scout RESULT_BASE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007 SCENARIO_FILE=/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scenarios/scout.tsv FIXED_CASE_IDS=rdma_scout_i65536_o1_c1\,rdma_scout_i65536_o1_c4\,rdma_scout_i65536_o1_c16\,rdma_scout_i65536_o1_c32\,rdma_scout_i65536_o1_c64 RUN_MIXED_CASE=0 RUN_COMMUNICATION_BASELINE=0 NUM_REPETITIONS=1 SAMPLE_INTERVAL_S=1 IDLE_BASELINE_S=5 POST_RUN_COOLDOWN_S=5 CASE_COOLDOWN_S=3 COLLECTOR_MAX_SECONDS=14400 SCENARIO_TIMEOUT_S=7200 bash /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/run_hardware_contention_attribution.sh all
|
||||||
@ -0,0 +1,7 @@
|
|||||||
|
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:03:43+0800,2026-08-01T14:11:03+0800,440.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,406.0844679859583,bench_main_marker_plus_duration,16,0,406.0844679859583,15.997723352218085,,1048576,16384,0.03940066971621592,2582.1622905219265,40.3462857894051,2622.5085763113316,,406026.68602950143,406024.73604350234,406053.3737935184,406053.5528299224,189372.86670175308,189251.9370575028,336778.7128902273,348624.74093402264,211.7828145921294,211.89852223703926,356.41907673483706,369.22852823690624,211.7827865053078,53.14174899831414,53.806967759737745,55.23117201111745,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c16
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,fixed,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,65536,1024,16,16,0,COMPLETED,,0,2026-08-01T14:11:03+0800,2026-08-01T14:18:24+0800,441.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,406.09330375201534,bench_main_marker_plus_duration,16,0,406.09330375201534,15.997232835777202,,1048576,16384,0.03939981243761298,2582.1061079114043,40.34540793611569,2622.45151584752,,406023.07082318654,406021.80599197163,406049.04031673505,406056.0000617319,189594.6209833055,189467.45998901315,336924.2522477143,348614.9598911492,211.56251206244485,211.68662201320043,356.06784952074946,368.88221031313344,211.56248251307755,53.15867651370354,54.19228565006051,55.67694299679715,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c16
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T13:58:15+0800,2026-08-01T14:00:56+0800,161.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.71871886099689,bench_main_marker_plus_duration,4,0,127.71871886099689,3.9987289259054917,,262144,4096,0.031318823393095684,2052.5104098899187,32.07047515452998,2084.5808850444487,,127678.13387226488,127678.37510351092,127682.15249372879,127682.41568037542,58779.63522000937,59286.87346848892,87520.75181677355,89912.75672257761,67.34946104814811,66.85386279083285,96.14249014501952,98.71897348003505,67.3494309836352,36.35076849604957,36.65076588222291,37.22598696593195,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c4
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,fixed,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,65536,1024,4,4,0,COMPLETED,,0,2026-08-01T14:00:57+0800,2026-08-01T14:03:36+0800,159.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,125.30951252899831,bench_main_marker_plus_duration,4,0,125.30951252899831,3.9986913595171454,,262144,4096,0.03192096050229504,2091.9720674784076,32.68706355435012,2124.6591310327576,,125268.51625375275,125267.95494600083,125274.71757131862,125275.15292868251,56944.67479274317,57454.862895508995,85207.63279019155,87584.75151082734,66.7877238133036,66.2884575273625,95.11951526871552,97.68299949875195,66.78769251954503,36.23565100133419,36.565644716029055,37.02181185770315,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c4
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:18:31+0800,2026-08-01T14:46:05+0800,1654.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.8975476099877,bench_main_marker_plus_duration,64,0,1619.8975476099877,40.3918658202939,,4194304,65536,0.039508671455442484,2589.2402925038787,40.456879570373104,2629.6971720742517,,1022354.4435263402,950470.9995904996,1619639.9038552889,1619650.3429244931,785995.8297684161,774078.3500669932,1506265.7001026294,1560787.9135349835,231.04458822866482,223.80146996136563,418.57784627036636,439.7380822666596,231.0445591993489,56.0161369794514,56.77989884279668,58.514122637570836,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/bench.log,fixed_rdma_confirm_i65536_o1024_c64
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,fixed,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,65536,1024,64,64,0,COMPLETED,,0,2026-08-01T14:46:06+0800,2026-08-01T15:13:40+0800,1654.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1620.2422464910196,bench_main_marker_plus_duration,64,0,1620.2422464910196,40.39704233770943,,4194304,65536,0.03950026617229964,2588.689443867829,40.44827256043483,2629.137716428264,,1022703.0410756742,950775.2832530241,1620010.8873196847,1620019.4079715456,786335.3449273454,774403.2157540205,1506724.735757988,1561196.831805848,231.05346642065376,223.83287831180922,418.5783493891426,439.7222371669007,231.0534379278651,56.055732507957146,56.75308680802118,58.11881933768746,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/bench.log,fixed_rdma_confirm_i65536_o1024_c64
|
||||||
|
@ -0,0 +1,25 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_0,371,3626828296476.0,3626837993952.0,71.66006119953522,71.66025280544177,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,404.892570368,bench_main_marker_plus_duration,head,mlx5_3,371,3626914718800.0,3626921347028.0,71.66176876011448,71.66189972281393,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_0,383,3626134447036.0,3626135801124.0,71.58004520955629,71.58007193930987,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,405.267634176,bench_main_marker_plus_duration,worker,mlx5_3,383,3626207403788.0,3626212421504.0,71.58148537888337,71.58158442880648,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_0,371,3625059758564.0,3625068124448.0,71.59790115790838,71.59806639097366,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,405.0464832,bench_main_marker_plus_duration,head,mlx5_3,371,3625136689400.0,3625142126056.0,71.59942060496823,71.59952798337986,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_0,382,3625042688652.0,3625045908572.0,71.73367967522776,71.73374339219049,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,404.277901824,bench_main_marker_plus_duration,worker,mlx5_3,382,3625122942520.0,3625131203508.0,71.73526776832192,71.73543123979464,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_0,117,894188173332.0,894192013996.0,56.331515043976715,56.33175699575861,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,126.9894016,bench_main_marker_plus_duration,head,mlx5_3,117,908654087048.0,908656485748.0,57.24282975425881,57.24298086608198,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_0,121,908912097324.0,908914949500.0,57.10271925252004,57.102898441471176,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,127.337136896,bench_main_marker_plus_duration,worker,mlx5_3,121,894353999288.0,894353927048.0,56.188101670195095,56.188097131691926,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_0,115,894195520320.0,894197433708.0,57.32326114869037,57.32338380826259,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.79339136,bench_main_marker_plus_duration,head,mlx5_3,115,908661063200.0,908663318872.0,58.25058864399148,58.250733246007684,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_0,118,908220840268.0,908222493124.0,58.56932688787363,58.5694334772174,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,124.054127104,bench_main_marker_plus_duration,worker,mlx5_3,118,893986155676.0,893986644792.0,57.65136084035526,57.651392382457814,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_0,1479,14610070965300.0,14610072489408.0,72.21898772698583,72.21899526079844,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1618.418803712,bench_main_marker_plus_duration,head,mlx5_3,1479,14610085802320.0,14610095102220.0,72.21906106780447,72.21910703810576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_0,1527,14611153636236.0,14611146819600.0,72.16844452340027,72.1684108541891,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1619.672280064,bench_main_marker_plus_duration,worker,mlx5_3,1527,14611226802652.0,14611236350532.0,72.16880591214243,72.16885307170732,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_0,1480,14610569508844.0,14610573106108.0,72.18614224806339,72.18616002099213,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.210452736,bench_main_marker_plus_duration,head,mlx5_3,1480,14610597458160.0,14610605842440.0,72.18628033667787,72.18632176071877,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_0,1527,14610266497856.0,14610260234736.0,72.18231262718308,72.18228168411397,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1619.262776832,bench_main_marker_plus_duration,worker,mlx5_3,1527,14610323772088.0,14610331114336.0,72.18259559166701,72.18263186618948,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
@ -0,0 +1,7 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,1,COMPLETED,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,1785564254150768896,1785564660235236864,406.084467968,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep1/meta.json
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c16,rdma_confirm_i65536_o1024_c16,,rdma_confirm,2,COMPLETED,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,1785564694981654016,1785565101074958080,406.093304064,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c16/cases/rdma_confirm_i65536_o1024_c16/rep2/meta.json
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,1,COMPLETED,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,1785563926377015040,1785564054095733760,127.71871872,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep1/meta.json
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c4,rdma_confirm_i65536_o1024_c4,,rdma_confirm,2,COMPLETED,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,1785564088360896000,1785564213670408960,125.30951296,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c4/cases/rdma_confirm_i65536_o1024_c4/rep2/meta.json
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,1,COMPLETED,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,1785565142582675968,1785566762480224000,1619.897548032,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep1/meta.json
|
||||||
|
fixed_rdma_confirm_i65536_o1024_c64,rdma_confirm_i65536_o1024_c64,,rdma_confirm,2,COMPLETED,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,1785566797465815040,1785568417708060928,1620.242245888,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm/bench/fixed_rdma_confirm_i65536_o1024_c64/cases/rdma_confirm_i65536_o1024_c64/rep2/meta.json
|
||||||
|
@ -0,0 +1,37 @@
|
|||||||
|
wall_time_ns,node,collector,status,pid
|
||||||
|
1785563885833567643,head,gpu_samples.csv,STARTED,2581028
|
||||||
|
1785563885848767676,head,rdma.csv,STARTED,2581039
|
||||||
|
1785563885862486752,head,dcgm_dmon.log,STARTED,2581053
|
||||||
|
1785563885876598354,head,mpstat.log,STARTED,2581071
|
||||||
|
1785563885891181421,head,pidstat.log,STARTED,2581090
|
||||||
|
1785563885906513050,head,sar_net.log,STARTED,2581117
|
||||||
|
1785563885921729008,head,perf_stat.log,STARTED,2581144
|
||||||
|
1785563885938632713,head,docker_top.log,STARTED,2581179
|
||||||
|
1785563885959612586,head,numa_samples.csv,STARTED,2581251
|
||||||
|
1785563885980492840,worker,gpu_samples.csv,STARTED,2581294
|
||||||
|
1785563885999012597,worker,rdma.csv,STARTED,2581356
|
||||||
|
1785563886014677475,worker,dcgm_dmon.log,STARTED,2581408
|
||||||
|
1785563886031239484,worker,mpstat.log,STARTED,2581455
|
||||||
|
1785563886046314326,worker,pidstat.log,STARTED,2581464
|
||||||
|
1785563886060934631,worker,sar_net.log,STARTED,2581478
|
||||||
|
1785563886075086128,worker,perf_stat.log,STARTED,2581486
|
||||||
|
1785563886090622653,worker,docker_top.log,STARTED,2581497
|
||||||
|
1785563886106525841,worker,numa_samples.csv,STARTED,2581509
|
||||||
|
1785568431501511663,head,gpu_samples.csv,STOPPED,2581028
|
||||||
|
1785568431506303146,head,rdma.csv,STOPPED,2581039
|
||||||
|
1785568431513026775,head,dcgm_dmon.log,STOPPED,2581053
|
||||||
|
1785568431518181488,head,mpstat.log,STOPPED,2581071
|
||||||
|
1785568431523384390,head,pidstat.log,STOPPED,2581090
|
||||||
|
1785568431528223936,head,sar_net.log,STOPPED,2581117
|
||||||
|
1785568431608439580,head,perf_stat.log,STOPPED,2581144
|
||||||
|
1785568431612888149,head,docker_top.log,STOPPED,2581179
|
||||||
|
1785568431617027570,head,numa_samples.csv,STOPPED,2581251
|
||||||
|
1785568431620791766,worker,gpu_samples.csv,STOPPED,2581294
|
||||||
|
1785568431624694474,worker,rdma.csv,STOPPED,2581356
|
||||||
|
1785568431628587426,worker,dcgm_dmon.log,STOPPED,2581408
|
||||||
|
1785568431632294659,worker,mpstat.log,STOPPED,2581455
|
||||||
|
1785568431636153536,worker,pidstat.log,STOPPED,2581464
|
||||||
|
1785568431639870020,worker,sar_net.log,STOPPED,2581478
|
||||||
|
1785568431643739953,worker,perf_stat.log,STOPPED,2581486
|
||||||
|
1785568431647558030,worker,docker_top.log,STOPPED,2581497
|
||||||
|
1785568431651292267,worker,numa_samples.csv,STOPPED,2581509
|
||||||
|
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
@ -0,0 +1,12 @@
|
|||||||
|
stage,case_id,repetition,status,isl,osl,concurrency,completed,duration_s,input_tps,output_tps,total_tps,ttft_p95_ms,tpot_p95_ms,gpu_util_mean_pct,rail_xmit_avg_gbps,rail_xmit_p95_gbps,rail_xmit_max_gbps,dual_rail_xmit_avg_gbps,rail_imbalance_pct,xmit_bytes_per_input_token_per_rail,xmit_bytes_per_total_token_per_rail,rdma_error_delta,measurement_started_at,measurement_ended_at,source_result_dir
|
||||||
|
scout,rdma_scout_i65536_o1_c1,1,COMPLETED,65536,1,1,1,24.1862671909621,2709.6368150803128,0.04134577659729481,2709.67816085691,24167.605170048773,0.0,93.74897875816994,70.9712746848062,85.5972372945909,86.80566056858429,141.9425493696124,0.5207119247408836,3137732.762954712,3137684.8856828967,0.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||||
|
scout,rdma_scout_i65536_o1_c4,1,COMPLETED,65536,1,4,4,89.45394851703895,2930.491100122512,0.044715745546302976,2930.535815868058,86526.69315790116,0.0,97.4494891826923,78.27464182068744,86.81919268342193,88.85894624954848,156.5492836413749,0.2703571902588796,3298012.1388320923,3297961.815928407,0.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||||
|
scout,rdma_scout_i65536_o1_c16,1,COMPLETED,65536,1,16,16,351.42641380499117,2983.771164628114,0.045528734811830354,2983.816693362926,336717.4506017618,0.0,99.33137833282255,79.89685137869246,86.416508483831,88.20167661610235,159.79370275738495,0.1149304623247205,3335858.008945465,3335807.108568442,0.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||||
|
scout,rdma_scout_i65536_o1_c32,1,COMPLETED,65536,1,32,32,702.8175486620166,2983.9209393567885,0.04553102019282209,2983.9664703769813,671139.2772838357,0.0,99.49393723047987,79.92852806348517,86.22587499587476,88.21644913628985,159.85705612697035,0.10091204027725086,3343929.3806581497,3343878.357123648,0.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||||
|
scout,rdma_scout_i65536_o1_c64,1,COMPLETED,65536,1,64,64,1402.17612864502,2991.2818470623424,0.04564333873080967,2991.327490401073,1337131.1103800863,0.0,99.63928846991146,79.9743866369171,85.93278751515446,88.32212536165416,159.9487732738342,0.03496630506355109,3339597.901974201,3339546.9445318105,0.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout
|
||||||
|
confirm,rdma_confirm_i65536_o1024_c4,1,COMPLETED,65536,1024,4,4,127.71871886099689,2052.5104098899187,32.07047515452998,2084.5808850444487,87520.75181677355,96.14249014501952,97.83383458646617,56.71629143023766,86.06750356786138,88.99885476939444,113.43258286047534,1.8596562953363254,3439052.922241211,3386144.415745192,0.0,2026-08-01T13:58:46.377015+08:00,2026-08-01T14:00:54.095734+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||||
|
confirm,rdma_confirm_i65536_o1024_c4,2,COMPLETED,65536,1024,4,4,125.30951252899831,2091.9720674784076,32.68706355435012,2124.6591310327576,85207.63279019155,95.11951526871552,98.63248207885304,57.94863438022769,86.4894823881113,88.5335715781298,115.89726876045536,2.1502935358359734,3438056.544746399,3385163.3671349157,0.0,2026-08-01T14:01:28.360896+08:00,2026-08-01T14:03:33.670409+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||||
|
confirm,rdma_confirm_i65536_o1024_c16,1,COMPLETED,65536,1024,16,16,406.0844679859583,2582.1622905219265,40.3462857894051,2622.5085763113316,336778.7128902273,356.41907673483706,99.38883591065292,71.62084013702234,86.26345231858909,88.9399300384103,143.24168027404468,0.11410582506689998,3458520.1421022415,3405312.139916053,0.0,2026-08-01T14:04:14.150769+08:00,2026-08-01T14:11:00.235237+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||||
|
confirm,rdma_confirm_i65536_o1024_c16,2,COMPLETED,65536,1024,16,16,406.09330375201534,2582.1061079114043,40.34540793611569,2622.45151584752,336924.2522477143,356.06784952074946,99.48923731257209,71.66656730160658,86.33499788736604,89.31414888004788,143.33313460321313,0.19167460586668467,3457155.7233657837,3403968.712237079,0.0,2026-08-01T14:11:34.981654+08:00,2026-08-01T14:18:21.074958+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||||
|
confirm,rdma_confirm_i65536_o1024_c64,1,COMPLETED,65536,1024,64,64,1619.8975476099877,2589.2402925038787,40.456879570373104,2629.6971720742517,1506265.7001026294,418.57784627036636,99.6722631148773,72.19382480758325,86.71108451788668,89.82468221963666,144.38764961516648,0.070112013789426,3483446.6699664593,3429855.182736206,0.0,2026-08-01T14:19:02.582676+08:00,2026-08-01T14:46:02.480224+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||||
|
confirm,rdma_confirm_i65536_o1024_c64,2,COMPLETED,65536,1024,64,64,1620.2422464910196,2588.689443867829,40.44827256043483,2629.137716428264,1506724.735757988,418.5783493891426,99.67225809785734,72.18433270089784,86.67873733217695,89.23672463088391,144.36866540179568,0.005496635275729702,3483400.1801579,3429809.4081554706,0.0,2026-08-01T14:46:37.465815+08:00,2026-08-01T15:13:37.708061+08:00,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/confirm
|
||||||
|
@ -0,0 +1,205 @@
|
|||||||
|
{
|
||||||
|
"generated_at": "2026-08-01T15:14:11.558999+08:00",
|
||||||
|
"formula": "rail_gbps ~= input_tps * bytes_per_input_token_per_rail * 8 / 1e9",
|
||||||
|
"plateau_gain_pct": 5.0,
|
||||||
|
"target_rail_gbps": 400.0,
|
||||||
|
"practical_rail_gbps": 360.0,
|
||||||
|
"scout": [
|
||||||
|
{
|
||||||
|
"stage": "scout",
|
||||||
|
"concurrency": 1,
|
||||||
|
"repetitions": 1,
|
||||||
|
"input_tps": 2709.6368150803128,
|
||||||
|
"output_tps": 0.04134577659729481,
|
||||||
|
"total_tps": 2709.67816085691,
|
||||||
|
"ttft_p95_ms": 24167.605170048773,
|
||||||
|
"tpot_p95_ms": 0.0,
|
||||||
|
"gpu_util_mean_pct": 93.74897875816994,
|
||||||
|
"rail_xmit_avg_gbps": 70.9712746848062,
|
||||||
|
"rail_xmit_p95_gbps": 85.5972372945909,
|
||||||
|
"rail_xmit_max_gbps": 86.80566056858429,
|
||||||
|
"dual_rail_xmit_avg_gbps": 141.9425493696124,
|
||||||
|
"rail_imbalance_pct": 0.5207119247408836,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3137732.762954712,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3137684.8856828967,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "scout",
|
||||||
|
"concurrency": 4,
|
||||||
|
"repetitions": 1,
|
||||||
|
"input_tps": 2930.491100122512,
|
||||||
|
"output_tps": 0.044715745546302976,
|
||||||
|
"total_tps": 2930.535815868058,
|
||||||
|
"ttft_p95_ms": 86526.69315790116,
|
||||||
|
"tpot_p95_ms": 0.0,
|
||||||
|
"gpu_util_mean_pct": 97.4494891826923,
|
||||||
|
"rail_xmit_avg_gbps": 78.27464182068744,
|
||||||
|
"rail_xmit_p95_gbps": 86.81919268342193,
|
||||||
|
"rail_xmit_max_gbps": 88.85894624954848,
|
||||||
|
"dual_rail_xmit_avg_gbps": 156.5492836413749,
|
||||||
|
"rail_imbalance_pct": 0.2703571902588796,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3298012.1388320923,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3297961.815928407,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "scout",
|
||||||
|
"concurrency": 16,
|
||||||
|
"repetitions": 1,
|
||||||
|
"input_tps": 2983.771164628114,
|
||||||
|
"output_tps": 0.045528734811830354,
|
||||||
|
"total_tps": 2983.816693362926,
|
||||||
|
"ttft_p95_ms": 336717.4506017618,
|
||||||
|
"tpot_p95_ms": 0.0,
|
||||||
|
"gpu_util_mean_pct": 99.33137833282255,
|
||||||
|
"rail_xmit_avg_gbps": 79.89685137869246,
|
||||||
|
"rail_xmit_p95_gbps": 86.416508483831,
|
||||||
|
"rail_xmit_max_gbps": 88.20167661610235,
|
||||||
|
"dual_rail_xmit_avg_gbps": 159.79370275738495,
|
||||||
|
"rail_imbalance_pct": 0.1149304623247205,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3335858.008945465,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3335807.108568442,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "scout",
|
||||||
|
"concurrency": 32,
|
||||||
|
"repetitions": 1,
|
||||||
|
"input_tps": 2983.9209393567885,
|
||||||
|
"output_tps": 0.04553102019282209,
|
||||||
|
"total_tps": 2983.9664703769813,
|
||||||
|
"ttft_p95_ms": 671139.2772838357,
|
||||||
|
"tpot_p95_ms": 0.0,
|
||||||
|
"gpu_util_mean_pct": 99.49393723047987,
|
||||||
|
"rail_xmit_avg_gbps": 79.92852806348517,
|
||||||
|
"rail_xmit_p95_gbps": 86.22587499587476,
|
||||||
|
"rail_xmit_max_gbps": 88.21644913628985,
|
||||||
|
"dual_rail_xmit_avg_gbps": 159.85705612697035,
|
||||||
|
"rail_imbalance_pct": 0.10091204027725086,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3343929.3806581497,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3343878.357123648,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "scout",
|
||||||
|
"concurrency": 64,
|
||||||
|
"repetitions": 1,
|
||||||
|
"input_tps": 2991.2818470623424,
|
||||||
|
"output_tps": 0.04564333873080967,
|
||||||
|
"total_tps": 2991.327490401073,
|
||||||
|
"ttft_p95_ms": 1337131.1103800863,
|
||||||
|
"tpot_p95_ms": 0.0,
|
||||||
|
"gpu_util_mean_pct": 99.63928846991146,
|
||||||
|
"rail_xmit_avg_gbps": 79.9743866369171,
|
||||||
|
"rail_xmit_p95_gbps": 85.93278751515446,
|
||||||
|
"rail_xmit_max_gbps": 88.32212536165416,
|
||||||
|
"dual_rail_xmit_avg_gbps": 159.9487732738342,
|
||||||
|
"rail_imbalance_pct": 0.03496630506355109,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3339597.901974201,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3339546.9445318105,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"confirm": [
|
||||||
|
{
|
||||||
|
"stage": "confirm",
|
||||||
|
"concurrency": 4,
|
||||||
|
"repetitions": 2,
|
||||||
|
"input_tps": 2072.2412386841634,
|
||||||
|
"output_tps": 32.37876935444005,
|
||||||
|
"total_tps": 2104.6200080386034,
|
||||||
|
"ttft_p95_ms": 86364.19230348256,
|
||||||
|
"tpot_p95_ms": 95.63100270686752,
|
||||||
|
"gpu_util_mean_pct": 98.2331583326596,
|
||||||
|
"rail_xmit_avg_gbps": 57.332462905232674,
|
||||||
|
"rail_xmit_p95_gbps": 86.27849297798633,
|
||||||
|
"rail_xmit_max_gbps": 88.76621317376211,
|
||||||
|
"dual_rail_xmit_avg_gbps": 114.66492581046535,
|
||||||
|
"rail_imbalance_pct": 2.0049749155861494,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3438554.733493805,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3385653.891440054,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "confirm",
|
||||||
|
"concurrency": 16,
|
||||||
|
"repetitions": 2,
|
||||||
|
"input_tps": 2582.1341992166654,
|
||||||
|
"output_tps": 40.345846862760396,
|
||||||
|
"total_tps": 2622.480046079426,
|
||||||
|
"ttft_p95_ms": 336851.4825689708,
|
||||||
|
"tpot_p95_ms": 356.24346312779323,
|
||||||
|
"gpu_util_mean_pct": 99.4390366116125,
|
||||||
|
"rail_xmit_avg_gbps": 71.64370371931446,
|
||||||
|
"rail_xmit_p95_gbps": 86.29922510297757,
|
||||||
|
"rail_xmit_max_gbps": 89.12703945922908,
|
||||||
|
"dual_rail_xmit_avg_gbps": 143.28740743862892,
|
||||||
|
"rail_imbalance_pct": 0.15289021546679232,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3457837.9327340126,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3404640.426076566,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"stage": "confirm",
|
||||||
|
"concurrency": 64,
|
||||||
|
"repetitions": 2,
|
||||||
|
"input_tps": 2588.964868185854,
|
||||||
|
"output_tps": 40.45257606540397,
|
||||||
|
"total_tps": 2629.4174442512576,
|
||||||
|
"ttft_p95_ms": 1506495.2179303085,
|
||||||
|
"tpot_p95_ms": 418.5780978297545,
|
||||||
|
"gpu_util_mean_pct": 99.67226060636732,
|
||||||
|
"rail_xmit_avg_gbps": 72.18907875424054,
|
||||||
|
"rail_xmit_p95_gbps": 86.69491092503182,
|
||||||
|
"rail_xmit_max_gbps": 89.53070342526028,
|
||||||
|
"dual_rail_xmit_avg_gbps": 144.37815750848108,
|
||||||
|
"rail_imbalance_pct": 0.03780432453257785,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3429832.295445838,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"selected_confirm_concurrencies": [
|
||||||
|
4,
|
||||||
|
16,
|
||||||
|
64
|
||||||
|
],
|
||||||
|
"observed_plateau_concurrency": 16,
|
||||||
|
"best_scout_bandwidth_concurrency": 64,
|
||||||
|
"linear_gbps_per_input_tps": 0.026656368057740153,
|
||||||
|
"modeled_bytes_per_input_token_per_rail": 3332046.007217519,
|
||||||
|
"required_input_tps_for_400g_per_rail": 15005.795205617025,
|
||||||
|
"required_input_tps_for_360g_per_rail": 13505.215685055322,
|
||||||
|
"required_input_tps_for_400g_dual_rail_aggregate": 7502.897602808513,
|
||||||
|
"bandwidth_saturation_fit": {
|
||||||
|
"asymptote": 80.32102454669293,
|
||||||
|
"half_saturation_c": 0.12589254117941673,
|
||||||
|
"rmse": 0.27652726609534695
|
||||||
|
},
|
||||||
|
"input_tps_saturation_fit": {
|
||||||
|
"asymptote": 3003.9418335884134,
|
||||||
|
"half_saturation_c": 0.11220184543019636,
|
||||||
|
"rmse": 7.680357081457988
|
||||||
|
},
|
||||||
|
"best_confirm_point": {
|
||||||
|
"stage": "confirm",
|
||||||
|
"concurrency": 64,
|
||||||
|
"repetitions": 2,
|
||||||
|
"input_tps": 2588.964868185854,
|
||||||
|
"output_tps": 40.45257606540397,
|
||||||
|
"total_tps": 2629.4174442512576,
|
||||||
|
"ttft_p95_ms": 1506495.2179303085,
|
||||||
|
"tpot_p95_ms": 418.5780978297545,
|
||||||
|
"gpu_util_mean_pct": 99.67226060636732,
|
||||||
|
"rail_xmit_avg_gbps": 72.18907875424054,
|
||||||
|
"rail_xmit_p95_gbps": 86.69491092503182,
|
||||||
|
"rail_xmit_max_gbps": 89.53070342526028,
|
||||||
|
"dual_rail_xmit_avg_gbps": 144.37815750848108,
|
||||||
|
"rail_imbalance_pct": 0.03780432453257785,
|
||||||
|
"xmit_bytes_per_input_token_per_rail": 3483423.4250621796,
|
||||||
|
"xmit_bytes_per_total_token_per_rail": 3429832.295445838,
|
||||||
|
"rdma_error_delta": 0.0
|
||||||
|
},
|
||||||
|
"verdict": "COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION"
|
||||||
|
}
|
||||||
@ -0,0 +1,37 @@
|
|||||||
|
# Phase 2.5 RDMA Demand Model
|
||||||
|
|
||||||
|
- Generated: `2026-08-01T15:14:11.558999+08:00`
|
||||||
|
- Verdict: `COMPUTE_OR_MODEL_THROUGHPUT_LIMITED_BEFORE_RDMA_SATURATION`
|
||||||
|
- Model: `rail_gbps ~= input_tps x 3.332 MB/token/rail x 8`
|
||||||
|
- Required Input TPS for 400G per rail: `15006`
|
||||||
|
- Required Input TPS for 360G per rail: `13505`
|
||||||
|
- Required Input TPS for 400G dual-rail aggregate: `7503`
|
||||||
|
- Fitted rail bandwidth ceiling: `80.32 Gbit/s`
|
||||||
|
- Observed plateau concurrency: `16`
|
||||||
|
|
||||||
|
## Scout: 64K -> 1
|
||||||
|
|
||||||
|
| C | Input TPS | Rail avg | Rail P95 | Dual-rail avg | MB/input-token/rail | GPU util |
|
||||||
|
|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| 1 | 2709.64 | 70.97 | 85.60 | 141.94 | 3.138 | 93.75 |
|
||||||
|
| 4 | 2930.49 | 78.27 | 86.82 | 156.55 | 3.298 | 97.45 |
|
||||||
|
| 16 | 2983.77 | 79.90 | 86.42 | 159.79 | 3.336 | 99.33 |
|
||||||
|
| 32 | 2983.92 | 79.93 | 86.23 | 159.86 | 3.344 | 99.49 |
|
||||||
|
| 64 | 2991.28 | 79.97 | 85.93 | 159.95 | 3.340 | 99.64 |
|
||||||
|
|
||||||
|
## Confirm: 64K -> 1K
|
||||||
|
|
||||||
|
| C | Reps | Input TPS | Output TPS | Rail avg | Rail P95 | Dual-rail avg | TTFT P95 ms | TPOT P95 ms |
|
||||||
|
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| 4 | 2 | 2072.24 | 32.38 | 57.33 | 86.28 | 114.66 | 86364.19 | 95.63 |
|
||||||
|
| 16 | 2 | 2582.13 | 40.35 | 71.64 | 86.30 | 143.29 | 336851.48 | 356.24 |
|
||||||
|
| 64 | 2 | 2588.96 | 40.45 | 72.19 | 86.69 | 144.38 | 1506495.22 | 418.58 |
|
||||||
|
|
||||||
|
## Evidence
|
||||||
|
|
||||||
|
- Per-case metrics: `rdma_case_metrics.csv`
|
||||||
|
- Machine-readable model: `rdma_demand_model.json`
|
||||||
|
- Scout HCA source: `scout/{head,worker}/rdma.csv`
|
||||||
|
- Confirm HCA source: `confirm/{head,worker}/rdma.csv`
|
||||||
|
- Exact windows: `scout/case_windows.csv`, `confirm/case_windows.csv`
|
||||||
|
- Benchmark summaries: `scout/bench_summary.csv`, `confirm/bench_summary.csv`
|
||||||
@ -0,0 +1 @@
|
|||||||
|
CONFIRM_CONCURRENCIES='4 16 64'
|
||||||
@ -0,0 +1,13 @@
|
|||||||
|
run_id=dsv4pro-phase2_5-20260801-130007
|
||||||
|
experiment=dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling
|
||||||
|
created_at=2026-08-01T13:00:07+0800
|
||||||
|
git_commit=c5fa700c50c0cc1b9e20d4d2d3f03bd485ba0dbf
|
||||||
|
git_dirty=0
|
||||||
|
isl=65536
|
||||||
|
scout_osl=1
|
||||||
|
confirm_osl=1024
|
||||||
|
scout_concurrencies=1 4 16 32 64
|
||||||
|
scout_repetitions=1
|
||||||
|
confirm_repetitions=2
|
||||||
|
target_rail_gbps=400
|
||||||
|
practical_rail_gbps=360
|
||||||
@ -0,0 +1,4 @@
|
|||||||
|
# case_id stage isl osl concurrency prompt_multiplier min_prompts warmup_requests note
|
||||||
|
rdma_confirm_i65536_o1024_c4 rdma_confirm 65536 1024 4 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||||
|
rdma_confirm_i65536_o1024_c16 rdma_confirm 65536 1024 16 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||||
|
rdma_confirm_i65536_o1024_c64 rdma_confirm 65536 1024 64 1 1 0 Phase 2.5 confirm: correlate model token rate with dual-rail RDMA
|
||||||
|
@ -0,0 +1,6 @@
|
|||||||
|
# case_id stage isl osl concurrency prompt_multiplier min_prompts warmup_requests note
|
||||||
|
rdma_scout_i65536_o1_c1 rdma_scout 65536 1 1 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||||
|
rdma_scout_i65536_o1_c4 rdma_scout 65536 1 4 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||||
|
rdma_scout_i65536_o1_c16 rdma_scout 65536 1 16 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||||
|
rdma_scout_i65536_o1_c32 rdma_scout 65536 1 32 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||||
|
rdma_scout_i65536_o1_c64 rdma_scout 65536 1 64 1 1 0 Phase 2.5 scout: correlate model token rate with dual-rail RDMA
|
||||||
|
@ -0,0 +1,6 @@
|
|||||||
|
run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run
|
||||||
|
fixed_rdma_scout_i65536_o1_c1,fixed,rdma_scout_i65536_o1_c1,,rdma_scout,1,65536,1,1,1,0,COMPLETED,,0,2026-08-01T13:05:54+0800,2026-08-01T13:06:53+0800,59.0,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,24.1862671909621,bench_main_marker_plus_duration,1,0,24.1862671909621,0.9992315809294645,,65536,1,0.04134577659729481,2709.6368150803128,0.04134577659729481,2709.67816085691,,24167.682002007496,24167.682002007496,24167.682002007496,24167.682002007496,24167.605170048773,24167.605170048773,24167.605170048773,24167.605170048773,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/bench.log,fixed_rdma_scout_i65536_o1_c1
|
||||||
|
fixed_rdma_scout_i65536_o1_c16,fixed,rdma_scout_i65536_o1_c16,,rdma_scout,1,65536,1,16,16,0,COMPLETED,,0,2026-08-01T13:09:09+0800,2026-08-01T13:15:34+0800,385.0,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,351.42641380499117,bench_main_marker_plus_duration,16,0,351.42641380499117,8.628873239905923,,1048576,16,0.045528734811830354,2983.771164628114,0.045528734811830354,2983.816693362926,,189525.8736173746,189769.315014506,336717.50638925005,348428.8343418157,189525.84591050618,189769.29299000767,336717.4506017618,348428.77078671294,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/bench.log,fixed_rdma_scout_i65536_o1_c16
|
||||||
|
fixed_rdma_scout_i65536_o1_c32,fixed,rdma_scout_i65536_o1_c32,,rdma_scout,1,65536,1,32,32,0,COMPLETED,,0,2026-08-01T13:15:40+0800,2026-08-01T13:27:58+0800,738.0,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.8175486620166,bench_main_marker_plus_duration,32,0,702.8175486620166,16.63411701870453,,2097152,32,0.04553102019282209,2983.9209393567885,0.04553102019282209,2983.9664703769813,,365335.9171013453,365369.92930600536,671139.3307480408,696643.7062339701,365335.8620414692,365369.8761375272,671139.2772838357,696643.6536653765,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/bench.log,fixed_rdma_scout_i65536_o1_c32
|
||||||
|
fixed_rdma_scout_i65536_o1_c4,fixed,rdma_scout_i65536_o1_c4,,rdma_scout,1,65536,1,4,4,0,COMPLETED,,0,2026-08-01T13:06:59+0800,2026-08-01T13:09:02+0800,123.0,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,89.45394851703895,bench_main_marker_plus_duration,4,0,89.45394851703895,2.5881014457616573,,262144,4,0.044715745546302976,2930.491100122512,0.044715745546302976,2930.535815868058,,57878.973371509346,58521.736774011515,86526.744662947,88841.4666661853,57878.91561826109,58521.6828455159,86526.69315790116,88841.41537077782,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/bench.log,fixed_rdma_scout_i65536_o1_c4
|
||||||
|
fixed_rdma_scout_i65536_o1_c64,fixed,rdma_scout_i65536_o1_c64,,rdma_scout,1,65536,1,64,64,0,COMPLETED,,0,2026-08-01T13:28:04+0800,2026-08-01T13:52:01+0800,1437.0,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1402.17612864502,bench_main_marker_plus_duration,64,0,1402.17612864502,32.732392604632125,,4194304,64,0.04564333873080967,2991.2818470623424,0.04564333873080967,2991.327490401073,,717134.0553695618,717609.0070659702,1337131.160498035,1390460.9022671543,717134.0051560618,717608.9713889814,1337131.1103800863,1390460.8547802733,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/bench.log,fixed_rdma_scout_i65536_o1_c64
|
||||||
|
@ -0,0 +1,21 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta
|
||||||
|
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.006126848,bench_main_marker_plus_duration,head,mlx5_0,22,203533427700.0,203651710996.0,70.77538224308064,70.81651330239592,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.006126848,bench_main_marker_plus_duration,head,mlx5_3,22,204100301340.0,204100542584.0,70.97250317308169,70.97258706169157,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.352313088,bench_main_marker_plus_duration,worker,mlx5_0,23,207229229632.0,207304460944.0,70.99227518955745,71.01804781832155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,23.352313088,bench_main_marker_plus_duration,worker,mlx5_3,23,207674858740.0,207674728976.0,71.14493813350504,71.14489367915073,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,350.585191424,bench_main_marker_plus_duration,head,mlx5_0,321,3499319374812.0,3499321894556.0,79.850934047694,79.85099154570729,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,350.585191424,bench_main_marker_plus_duration,head,mlx5_3,321,3499670238252.0,3499687020088.0,79.85894039704549,79.85932334159443,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,349.898471936,bench_main_marker_plus_duration,worker,mlx5_0,331,3496131787252.0,3496130977288.0,79.93477120166395,79.93475268282916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,349.898471936,bench_main_marker_plus_duration,worker,mlx5_3,331,3496481190036.0,3496480578468.0,79.94275986836642,79.94274588561318,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,701.184411136,bench_main_marker_plus_duration,head,mlx5_0,641,7008726918884.0,7008731821472.0,79.96443511947507,79.96449105440941,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,701.184411136,bench_main_marker_plus_duration,head,mlx5_3,641,7009132360248.0,7009232426556.0,79.9690609081556,79.97020259135803,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.616064768,bench_main_marker_plus_duration,worker,mlx5_0,663,7016359452164.0,7016357965840.0,79.88840339972316,79.88838647640956,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,702.616064768,bench_main_marker_plus_duration,worker,mlx5_3,663,7016694022728.0,7016692492508.0,79.89221282658687,79.89219540347258,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.716937728,bench_main_marker_plus_duration,head,mlx5_0,82,866863327600.0,866884788736.0,78.16891338226691,78.17084862814438,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.716937728,bench_main_marker_plus_duration,head,mlx5_3,82,867282690684.0,867282482924.0,78.20672921268122,78.20671047804,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.006032896,bench_main_marker_plus_duration,worker,mlx5_0,84,861825370952.0,861825497804.0,78.34239018322312,78.34240171443258,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,88.006032896,bench_main_marker_plus_duration,worker,mlx5_3,84,862244987252.0,862244674092.0,78.38053450457852,78.38050603743919,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.218045952,bench_main_marker_plus_duration,head,mlx5_0,1280,14005295392512.0,14005304292920.0,79.96069096010922,79.96074177537255,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.218045952,bench_main_marker_plus_duration,head,mlx5_3,1280,14005533397520.0,14005553945192.0,79.96204980648541,79.96216711968765,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.136956672,bench_main_marker_plus_duration,worker,mlx5_0,1321,14008943976736.0,14008941122316.0,79.9861507329604,79.98613443522456,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1401.136956672,bench_main_marker_plus_duration,worker,mlx5_3,1321,14009382587800.0,14009379963032.0,79.98865504811339,79.98864006160981,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
|
||||||
|
@ -0,0 +1,6 @@
|
|||||||
|
phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path
|
||||||
|
fixed_rdma_scout_i65536_o1_c1,rdma_scout_i65536_o1_c1,,rdma_scout,1,COMPLETED,2026-08-01T13:06:26.217761+08:00,2026-08-01T13:06:50.404028+08:00,1785560786217761024,1785560810404027904,24.18626688,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c1/cases/rdma_scout_i65536_o1_c1/rep1/meta.json
|
||||||
|
fixed_rdma_scout_i65536_o1_c16,rdma_scout_i65536_o1_c16,,rdma_scout,1,COMPLETED,2026-08-01T13:09:39.903596+08:00,2026-08-01T13:15:31.330010+08:00,1785560979903596032,1785561331330009856,351.426413824,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c16/cases/rdma_scout_i65536_o1_c16/rep1/meta.json
|
||||||
|
fixed_rdma_scout_i65536_o1_c32,rdma_scout_i65536_o1_c32,,rdma_scout,1,COMPLETED,2026-08-01T13:16:12.166763+08:00,2026-08-01T13:27:54.984312+08:00,1785561372166763008,1785562074984312064,702.817549056,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c32/cases/rdma_scout_i65536_o1_c32/rep1/meta.json
|
||||||
|
fixed_rdma_scout_i65536_o1_c4,rdma_scout_i65536_o1_c4,,rdma_scout,1,COMPLETED,2026-08-01T13:07:30.521113+08:00,2026-08-01T13:08:59.975062+08:00,1785560850521112832,1785560939975062016,89.453949184,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c4/cases/rdma_scout_i65536_o1_c4/rep1/meta.json
|
||||||
|
fixed_rdma_scout_i65536_o1_c64,rdma_scout_i65536_o1_c64,,rdma_scout,1,COMPLETED,2026-08-01T13:28:36.631968+08:00,2026-08-01T13:51:58.808097+08:00,1785562116631968000,1785563518808096768,1402.176128768,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_rdma_demand_modeling/results/dsv4pro-phase2_5-20260801-130007/scout/bench/fixed_rdma_scout_i65536_o1_c64/cases/rdma_scout_i65536_o1_c64/rep1/meta.json
|
||||||
|
@ -0,0 +1,37 @@
|
|||||||
|
wall_time_ns,node,collector,status,pid
|
||||||
|
1785560744455556276,head,gpu_samples.csv,STARTED,2168337
|
||||||
|
1785560744470800548,head,rdma.csv,STARTED,2168348
|
||||||
|
1785560744485157848,head,dcgm_dmon.log,STARTED,2168362
|
||||||
|
1785560744499919157,head,mpstat.log,STARTED,2168380
|
||||||
|
1785560744514378735,head,pidstat.log,STARTED,2168400
|
||||||
|
1785560744529055413,head,sar_net.log,STARTED,2168428
|
||||||
|
1785560744543220536,head,perf_stat.log,STARTED,2168451
|
||||||
|
1785560744559383367,head,docker_top.log,STARTED,2168486
|
||||||
|
1785560744579614511,head,numa_samples.csv,STARTED,2168561
|
||||||
|
1785560744602206477,worker,gpu_samples.csv,STARTED,2168604
|
||||||
|
1785560744619402981,worker,rdma.csv,STARTED,2168664
|
||||||
|
1785560744634896197,worker,dcgm_dmon.log,STARTED,2168717
|
||||||
|
1785560744650285593,worker,mpstat.log,STARTED,2168763
|
||||||
|
1785560744666005300,worker,pidstat.log,STARTED,2168775
|
||||||
|
1785560744681017748,worker,sar_net.log,STARTED,2168787
|
||||||
|
1785560744694432153,worker,perf_stat.log,STARTED,2168796
|
||||||
|
1785560744709522681,worker,docker_top.log,STARTED,2168808
|
||||||
|
1785560744723983222,worker,numa_samples.csv,STARTED,2168818
|
||||||
|
1785563532345901412,head,gpu_samples.csv,STOPPED,2168337
|
||||||
|
1785563532350635898,head,rdma.csv,STOPPED,2168348
|
||||||
|
1785563532357352405,head,dcgm_dmon.log,STOPPED,2168362
|
||||||
|
1785563532362419425,head,mpstat.log,STOPPED,2168380
|
||||||
|
1785563532367417490,head,pidstat.log,STOPPED,2168400
|
||||||
|
1785563532372197961,head,sar_net.log,STOPPED,2168428
|
||||||
|
1785563532541034088,head,perf_stat.log,STOPPED,2168451
|
||||||
|
1785563532545727806,head,docker_top.log,STOPPED,2168486
|
||||||
|
1785563532550309298,head,numa_samples.csv,STOPPED,2168561
|
||||||
|
1785563532554473337,worker,gpu_samples.csv,STOPPED,2168604
|
||||||
|
1785563532558689660,worker,rdma.csv,STOPPED,2168664
|
||||||
|
1785563532562663636,worker,dcgm_dmon.log,STOPPED,2168717
|
||||||
|
1785563532566892350,worker,mpstat.log,STOPPED,2168763
|
||||||
|
1785563532571076626,worker,pidstat.log,STOPPED,2168775
|
||||||
|
1785563532575164431,worker,sar_net.log,STOPPED,2168787
|
||||||
|
1785563532579199145,worker,perf_stat.log,STOPPED,2168796
|
||||||
|
1785563532583371937,worker,docker_top.log,STOPPED,2168808
|
||||||
|
1785563532587359055,worker,numa_samples.csv,STOPPED,2168818
|
||||||
|
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256
|
||||||
@ -0,0 +1,104 @@
|
|||||||
|
# DeepSeek-V4-Pro TP16 网络路径审计
|
||||||
|
|
||||||
|
- 时间：2026-07-30
|
||||||
|
- 节点：`174.1.51.5 + 174.1.51.7`
|
||||||
|
- 模型：DeepSeek-V4-Pro
|
||||||
|
- 引擎：SGLang nightly
|
||||||
|
- 拓扑：TP16 / EP2 / 2 nodes
|
||||||
|
|
||||||
|
## 结论
|
||||||
|
|
||||||
|
Phase 1 quick-map 的约 65 input token/s 不是可直接归因给模型、Kernel 或 GPU
|
||||||
|
的性能基线。它同时受两个独立因素影响：
|
||||||
|
|
||||||
|
1. quick-map 容器没有 RDMA 设备，NCCL 回退 `NET/Socket`。
|
||||||
|
2. quick-map 的 `NCCL_SOCKET_IFNAME` 误选低速非计算网，Socket 数据没有进入
|
||||||
|
部署规定的 `eth0/eth3` 计算网。
|
||||||
|
|
||||||
|
`NCCL_CROSS_NIC=1` 不是本次 10 倍以上退化的原因。该参数只有在 NCCL
|
||||||
|
真正使用多个 RDMA HCA 时才影响 ring/tree 的 NIC 选择；本次实际后端为 Socket。
|
||||||
|
|
||||||
|
## 设备关系
|
||||||
|
|
||||||
|
部署时只使用两条节点间计算网：
|
||||||
|
|
||||||
|
| 物理端口 | Linux netdev/IP 入口 | RDMA Verbs/HCA 入口 | 状态 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 400G Rail 1 | `eth0` | `mlx5_0` | Up |
|
||||||
|
| 400G Rail 2 | `eth3` | `mlx5_3` | Up |
|
||||||
|
|
||||||
|
`eth0` 与 `mlx5_0` 不是同一个软件设备。它们是同一条 400G 物理 Ethernet
|
||||||
|
端口的两种入口：前者服务 IP/TCP Socket，后者服务 RoCE/RDMA Verbs。
|
||||||
|
|
||||||
|
## 运行时证据
|
||||||
|
|
||||||
|
宿主机存在：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/dev/infiniband/rdma_cm
|
||||||
|
/dev/infiniband/uverbs0
|
||||||
|
/dev/infiniband/uverbs3
|
||||||
|
```
|
||||||
|
|
||||||
|
按原脚本启动的容器内：
|
||||||
|
|
||||||
|
```text
|
||||||
|
ls: cannot access '/dev/infiniband': No such file or directory
|
||||||
|
```
|
||||||
|
|
||||||
|
NCCL INFO：
|
||||||
|
|
||||||
|
```text
|
||||||
|
NCCL_SOCKET_IFNAME set by environment to eth0
|
||||||
|
Bootstrap: Using eth0:10.101.0.11
|
||||||
|
NET/IB : No device found.
|
||||||
|
Failed to initialize NET plugin IB
|
||||||
|
NET/Socket : Using [0]eth0:10.101.0.11
|
||||||
|
Using network Socket
|
||||||
|
```
|
||||||
|
|
||||||
|
## 冷缓存对照
|
||||||
|
|
||||||
|
共同条件：
|
||||||
|
|
||||||
|
- 同一模型、镜像、TP16/EP2 和 serving 参数。
|
||||||
|
- `random` 数据集，`seed=42`。
|
||||||
|
- `warmup_requests=0`。
|
||||||
|
- 测量前 `--flush-cache`。
|
||||||
|
- `num_prompts=1`、`max_concurrency=1`。
|
||||||
|
|
||||||
|
| Shape | 原脚本网络：`eth0` Socket | quick-map 错误网络 | 退化 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| 1K -> 1 | TTFT 1.458s；693.1 input tok/s | TTFT 15.88-16.04s；约 64 tok/s | 约 10.9x |
|
||||||
|
| 32K -> 1 | TTFT 38.062s；860.5 input tok/s | TTFT 504.44s；64.96 tok/s | 约 13.25x |
|
||||||
|
|
||||||
|
## 旧矩阵为什么还能更快
|
||||||
|
|
||||||
|
旧矩阵结果还有 Prefix Cache 污染：
|
||||||
|
|
||||||
|
- 每个 case 固定 16 条相同首个 prompt 的 warm-up。
|
||||||
|
- benchmark 默认 `seed=42`，每次使用相同 ShareGPT shuffle 顺序。
|
||||||
|
- ISL、OSL 和并发升序运行。
|
||||||
|
- 从不传 `--flush-cache`。
|
||||||
|
- 17:40 的失败 Run 已执行过首个 1K case，18:01 正式 Run 复用同一服务。
|
||||||
|
|
||||||
|
所以旧 1K 的 0.455s、32K 的 32.03s 和 128K 的 130.44s 不是完整冷
|
||||||
|
Prefill，不能与任一冷缓存结果直接比较。
|
||||||
|
|
||||||
|
## 修复顺序
|
||||||
|
|
||||||
|
1. 当前 Socket baseline 默认改为 `NCCL_SOCKET_IFNAME=eth0`。
|
||||||
|
2. 重跑 Phase 1 的冷 1K/32K/128K 代表点。
|
||||||
|
3. 单独给容器透传 `rdma_cm`、`uverbs0`、`uverbs3`。
|
||||||
|
4. 用 `NCCL_DEBUG=INFO` 确认出现 `NET/IB`，不能只看环境变量。
|
||||||
|
5. 真正启用双 Rail RDMA 后，再比较 `NCCL_CROSS_NIC=0/1/2`。
|
||||||
|
6. 以修正后的端到端结果决定是否进入 Phase 2 硬件归因。
|
||||||
|
|
||||||
|
## 产物
|
||||||
|
|
||||||
|
- `head_server.log`
|
||||||
|
- `worker_server.log`
|
||||||
|
- `oldscript_network_cold_1k_o1.log`
|
||||||
|
- `oldscript_network_cold_1k_o1.jsonl`
|
||||||
|
- `oldscript_network_cold_32k_o1.log`
|
||||||
|
- `oldscript_network_cold_32k_o1.jsonl`
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
# DeepSeek-V4-Pro Pro6000D Two-Node SGLang Quick Map
|
||||||
|
|
||||||
|
Profiler: disabled. Speculative decoding: disabled.
|
||||||
|
|
||||||
|
## Aggregate results
|
||||||
|
|
||||||
|
| Case | Suite / role | Stage | ISL | OSL | C | Reps | Total TPS | CV | Output TPS | TTFT P95 | TPOT P95 | E2E P95 | Status |
|
||||||
|
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||||
|
| long_prefill_latency_128k_c1 | fixed / - | prefill_latency | 131072 | 1 | 1 | 1/1 | 65.20 | -% | 0.00 | 2010382.63 ms | 0.00 ms | 2010382.70 ms | COMPLETED |
|
||||||
|
| mid_prefill_latency_32k_c1 | fixed / - | prefill_latency | 32768 | 1 | 1 | 1/1 | 64.96 | -% | 0.00 | 504435.18 ms | 0.00 ms | 504435.25 ms | COMPLETED |
|
||||||
|
| mid_prefill_throughput_32k_c16 | fixed / - | prefill_throughput | 32768 | 1 | 16 | 0/1 | - | -% | - | - ms | - ms | - ms | ABORTED |
|
||||||
|
| short_prefill_latency_1k_c1 | fixed / - | prefill_latency | 1024 | 1 | 1 | 1/1 | 64.44 | -% | 0.06 | 15882.76 ms | 0.00 ms | 15882.82 ms | COMPLETED |
|
||||||
|
|
||||||
|
## Failed or incomplete cases
|
||||||
|
|
||||||
|
| Case | Repetition | Status | Error | Exit code |
|
||||||
|
|---|---:|---|---|---:|
|
||||||
|
| mid_prefill_throughput_32k_c16 | 1 | ABORTED | EARLY_STOP_FOR_PHASE2 | 143 |
|
||||||
|
|
||||||
|
The fixed map is descriptive and never stops on SLO. With one repetition, CV is intentionally unavailable.
|
||||||
@ -0,0 +1,40 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"workflow_stage": "quick_performance_map",
|
||||||
|
"run_id": "dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625",
|
||||||
|
"status": "ABORTED_EARLY_FOR_PHASE2",
|
||||||
|
"started_at": "2026-07-30T14:36:26+08:00",
|
||||||
|
"updated_at": "2026-07-30T15:44:11+08:00",
|
||||||
|
"suites": [
|
||||||
|
"fixed",
|
||||||
|
"mixed"
|
||||||
|
],
|
||||||
|
"engine": "sglang",
|
||||||
|
"model_name": "DeepSeek-V4-Pro",
|
||||||
|
"model_path": "/data/hf_models/DeepSeek-V4-Pro",
|
||||||
|
"docker_image": "lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45",
|
||||||
|
"head_node": "10.101.0.11",
|
||||||
|
"worker_node": "10.101.0.13",
|
||||||
|
"head_ip": "10.101.0.11",
|
||||||
|
"sglang_port": 30002,
|
||||||
|
"dist_init_port": 20002,
|
||||||
|
"tp_size": 16,
|
||||||
|
"ep_size": 2,
|
||||||
|
"nnodes": 2,
|
||||||
|
"mem_fraction_static": 0.9,
|
||||||
|
"cuda_graph_max_bs_decode": 64,
|
||||||
|
"max_running_requests": 256,
|
||||||
|
"nccl_socket_ifname": "eth1",
|
||||||
|
"nccl_ib_hca": "mlx5_0,mlx5_3",
|
||||||
|
"nccl_cross_nic": "1",
|
||||||
|
"git_commit": "d5d96bd6f60f7fcdf0e118070ace468698949428",
|
||||||
|
"git_dirty": false,
|
||||||
|
"scenario_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_scenarios.tsv",
|
||||||
|
"notes": [
|
||||||
|
"The fixed quick map does not stop on SLO.",
|
||||||
|
"Profiler is disabled; these results are eligible for performance comparison.",
|
||||||
|
"Speculative decoding is not enabled.",
|
||||||
|
"Source tree was committed unchanged during model initialization."
|
||||||
|
],
|
||||||
|
"ended_at": "2026-07-30T15:44:11+08:00"
|
||||||
|
}
|
||||||
@ -0,0 +1,4 @@
|
|||||||
|
{"run_id": "dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625", "suite": "fixed", "case_id": "long_prefill_latency_128k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 131072, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T14:52:19+0800", "ended_at": "2026-07-30T15:26:24+0800", "elapsed_s": 2045.0, "completed": 1, "failed": 0, "duration_s": 2010.4037326959951, "actual_concurrency": 0.999989539343442, "peak_concurrent_requests": null, "total_input_tokens": 131072, "total_output_tokens": 1, "request_throughput": 0.0004974125265172375, "input_token_throughput": 65.19685467566735, "output_token_throughput": 0.0004974125265172375, "total_token_throughput": 65.19735208819388, "peak_output_token_throughput": null, "e2e_mean_ms": 2010382.7025530045, "e2e_p50_ms": 2010382.7025530045, "e2e_p95_ms": 2010382.7025530045, "e2e_p99_ms": 2010382.7025530045, "ttft_mean_ms": 2010382.6258230256, "ttft_p50_ms": 2010382.6258230256, "ttft_p95_ms": 2010382.6258230256, "ttft_p99_ms": 2010382.6258230256, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625/cases/long_prefill_latency_128k_c1/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625", "suite": "fixed", "case_id": "mid_prefill_latency_32k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 32768, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 0, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T14:43:15+0800", "ended_at": "2026-07-30T14:52:13+0800", "elapsed_s": 538.0, "completed": 1, "failed": 0, "duration_s": 504.4536426469858, "actual_concurrency": 0.9999635371827135, "peak_concurrent_requests": null, "total_input_tokens": 32768, "total_output_tokens": 1, "request_throughput": 0.001982342707949866, "input_token_throughput": 64.95740585410121, "output_token_throughput": 0.001982342707949866, "total_token_throughput": 64.95938819680917, "peak_output_token_throughput": null, "e2e_mean_ms": 504435.24884598446, "e2e_p50_ms": 504435.24884598446, "e2e_p95_ms": 504435.24884598446, "e2e_p99_ms": 504435.24884598446, "ttft_mean_ms": 504435.1755149837, "ttft_p50_ms": 504435.1755149837, "ttft_p95_ms": 504435.1755149837, "ttft_p99_ms": 504435.1755149837, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625/cases/mid_prefill_latency_32k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625/cases/mid_prefill_latency_32k_c1/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625", "suite": "fixed", "case_id": "mid_prefill_throughput_32k_c16", "role": "", "stage": "prefill_throughput", "repetition": 1, "isl": 32768, "osl": 1, "concurrency": 16, "num_prompts": 16, "warmup_requests": 0, "status": "ABORTED", "error_type": "EARLY_STOP_FOR_PHASE2", "exit_code": 143, "started_at": "2026-07-30T15:26:30+0800", "ended_at": "2026-07-30T15:41:52+0800", "elapsed_s": 922.0, "completed": null, "failed": null, "duration_s": null, "actual_concurrency": null, "peak_concurrent_requests": null, "total_input_tokens": null, "total_output_tokens": null, "request_throughput": null, "input_token_throughput": null, "output_token_throughput": null, "total_token_throughput": null, "peak_output_token_throughput": null, "e2e_mean_ms": null, "e2e_p50_ms": null, "e2e_p95_ms": null, "e2e_p99_ms": null, "ttft_mean_ms": null, "ttft_p50_ms": null, "ttft_p95_ms": null, "ttft_p99_ms": null, "tpot_mean_ms": null, "tpot_p50_ms": null, "tpot_p95_ms": null, "tpot_p99_ms": null, "itl_mean_ms": null, "itl_p50_ms": null, "itl_p95_ms": null, "itl_p99_ms": null, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625/cases/mid_prefill_throughput_32k_c16/rep1/bench.log"}
|
||||||
|
{"run_id": "dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625", "suite": "fixed", "case_id": "short_prefill_latency_1k_c1", "role": "", "stage": "prefill_latency", "repetition": 1, "isl": 1024, "osl": 1, "concurrency": 1, "num_prompts": 1, "warmup_requests": 1, "status": "COMPLETED", "error_type": "", "exit_code": 0, "started_at": "2026-07-30T14:42:04+0800", "ended_at": "2026-07-30T14:43:10+0800", "elapsed_s": 66.0, "completed": 1, "failed": 0, "duration_s": 15.905719314003363, "actual_concurrency": 0.9985603391737933, "peak_concurrent_requests": null, "total_input_tokens": 1024, "total_output_tokens": 1, "request_throughput": 0.06287046692189532, "input_token_throughput": 64.37935812802081, "output_token_throughput": 0.06287046692189532, "total_token_throughput": 64.4422285949427, "peak_output_token_throughput": null, "e2e_mean_ms": 15882.820472994354, "e2e_p50_ms": 15882.820472994354, "e2e_p95_ms": 15882.820472994354, "e2e_p99_ms": 15882.820472994354, "ttft_mean_ms": 15882.760226988466, "ttft_p50_ms": 15882.760226988466, "ttft_p95_ms": 15882.760226988466, "ttft_p99_ms": 15882.760226988466, "tpot_mean_ms": 0.0, "tpot_p50_ms": 0.0, "tpot_p95_ms": 0.0, "tpot_p99_ms": 0.0, "itl_mean_ms": 0.0, "itl_p50_ms": 0.0, "itl_p95_ms": 0.0, "itl_p99_ms": 0.0, "bench_file": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625/cases/short_prefill_latency_1k_c1/rep1/bench.jsonl", "bench_log": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/results/dsv4pro-pro6000d-2node-sglang-quick-v2-20260730-143625/cases/short_prefill_latency_1k_c1/rep1/bench.log"}
|
||||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -0,0 +1,54 @@
|
|||||||
|
# DSV4-Pro TP16 TTFT 脚本口径审计
|
||||||
|
|
||||||
|
时间：2026-07-30
|
||||||
|
机器：174.1.51.5 + 174.1.51.7
|
||||||
|
引擎：SGLang TP16 / EP2
|
||||||
|
|
||||||
|
## 结论
|
||||||
|
|
||||||
|
旧脚本的短 TTFT 不是同口径下的更高冷 Prefill 性能。旧结果同时受到以下状态影响：
|
||||||
|
|
||||||
|
- 每个 Shape 固定执行 16 条同 Prompt Warm-up。
|
||||||
|
- Benchmark 不传 `--flush-cache`。
|
||||||
|
- Seed 固定为 42，并按递增 ISL 重复选取同一批 Prompt。
|
||||||
|
- 17:40 的失败 Run 已执行 1K Case，18:01 的正式 Run 未重启服务。
|
||||||
|
- `C=1` 仍至少执行 10 条正式请求，聚合均值混合了不同缓存状态。
|
||||||
|
|
||||||
|
新旧 `server_info` 的关键运行参数一致：同一镜像、TP16、EP2、8K Chunk、FlashInfer
|
||||||
|
MXFP4 MoE、相同的调度和 CUDA Graph 配置。明显的服务启动参数回归已排除。
|
||||||
|
|
||||||
|
## 最小复现
|
||||||
|
|
||||||
|
| Case | Mean TTFT | P95 TTFT | 说明 |
|
||||||
|
|---|---:|---:|---|
|
||||||
|
| 1K→1，首次冷 Cache | 16.04 s | 16.04 s | `warmup=0`，测量前 flush |
|
||||||
|
| 1K→1，原样再次冷 Cache | 15.90 s | 15.90 s | 再次 flush，排除一次性 JIT 主导 |
|
||||||
|
| 1K→128，冷 Cache | 15.79 s | 15.79 s | 排除 OSL=1 特殊慢路径 |
|
||||||
|
| 旧命令语义复现 | 14.60 s | 15.97 s | 10 prompts、16 warmup、不 flush |
|
||||||
|
| 2026-07-28 旧产物 | 0.455 s | 0.513 s | 服务和 Prompt 已被前一轮预热 |
|
||||||
|
|
||||||
|
旧命令语义复现的逐请求 TTFT 为：
|
||||||
|
|
||||||
|
```text
|
||||||
|
4.13, 15.81, 15.70, 15.82, 15.81,
|
||||||
|
15.82, 16.07, 15.42, 15.63, 15.84 seconds
|
||||||
|
```
|
||||||
|
|
||||||
|
它无法复现旧产物约 0.45 秒的结果。
|
||||||
|
|
||||||
|
## 递增长度污染
|
||||||
|
|
||||||
|
旧 32K 产物中，第一条 TTFT 约 0.67 秒，其余 9 条平均约 35.51 秒。旧 128K
|
||||||
|
产物中，第一条约 0.96 秒，其余 9 条平均约 144.83 秒。由于脚本此前已经以相同
|
||||||
|
Seed 跑过 16K、64K，这些请求会继承上一档 Prompt 前缀。
|
||||||
|
|
||||||
|
旧报告仍使用完整 ISL 计算 Input TPS，即使服务实际只需计算新增后缀，所以旧
|
||||||
|
Input TPS 也会被高估。
|
||||||
|
|
||||||
|
## 后续口径
|
||||||
|
|
||||||
|
- Phase 2 继续归因清 Prefix Cache 后的完整冷 Prefill。
|
||||||
|
- Warm Prefix / Prefix Cache 收益单独设计 A/B。
|
||||||
|
- Cold 与 Warm 数据必须分列，不再直接比较。
|
||||||
|
|
||||||
|
原始 JSON 和日志就在本目录。
|
||||||
1336
docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html
Normal file
1336
docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html
Normal file
File diff suppressed because it is too large
Load Diff
772
docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md
Normal file
772
docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md
Normal file
@ -0,0 +1,772 @@
|
|||||||
|
# 6000D 双机 DeepSeek-V4-Pro 推理优化计划
|
||||||
|
|
||||||
|
> 适用环境：`174.1.51.5 + 174.1.51.7`，每台 8 张 RTX PRO 6000 Blackwell Server Edition
|
||||||
|
> 当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例
|
||||||
|
> 当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离
|
||||||
|
> 计划版本：2026-07-31 12:26:00 CST
|
||||||
|
|
||||||
|
## 阶段档案与实验命令规范
|
||||||
|
|
||||||
|
每个 Phase HTML 的正文只保留最终成功实验、有效结果和结论；失败尝试压缩到
|
||||||
|
末尾的经验教训。阶段完成时删除“暂不能下结论”等过渡内容。
|
||||||
|
|
||||||
|
每个 Phase HTML 必须包含“实验复现命令”，并明确区分：
|
||||||
|
|
||||||
|
| 标签 | 含义 | 必须记录的内容 |
|
||||||
|
|---|---|---|
|
||||||
|
| 实际执行命令 | 该次有效 Run 真正运行过 | 执行节点、工作目录、tmux/入口、完整 Docker 服务命令、Benchmark/Profile/监控命令、停止清理命令、Run ID 与命令证据路径 |
|
||||||
|
| 复现命令 | 根据实际 Run 整理，可重新执行 | 与实际参数等价；允许为可读性换行，但不得省略影响结果的参数 |
|
||||||
|
| 计划或示例命令 | 尚未在当前阶段运行 | 必须显式标注“未执行”，真机完成后替换为实际命令，不能作为结果证据 |
|
||||||
|
|
||||||
|
长命令必须同时原样保存到结果目录的 `*_cmd.txt` 或 Manifest。HTML 负责教学、
|
||||||
|
解释和索引，落盘命令文件负责精确审计。
|
||||||
|
|
||||||
|
## 0. 双机通信前置知识与当前修正
|
||||||
|
|
||||||
|
完整术语、设备映射、日志判读和 2026-07-30 网络事故复盘见：
|
||||||
|
|
||||||
|
- [6000D 双机通信与 NCCL 术语入门](./6000D双机通信与NCCL术语入门.html)
|
||||||
|
|
||||||
|
本项目部署时只使用两条节点间计算网：
|
||||||
|
|
||||||
|
| 物理端口 | Linux netdev/IP 入口 | RDMA Verbs/HCA 入口 | 交换路径 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 400G Rail 1 | `eth0` | `mlx5_0` | switch 1 |
|
||||||
|
| 400G Rail 2 | `eth3` | `mlx5_3` | switch 2 |
|
||||||
|
|
||||||
|
`eth0` 与 `mlx5_0` 不是同一个软件设备。它们是同一条 400G 物理 Ethernet
|
||||||
|
端口的两种入口：前者服务 IP/TCP Socket，后者服务 RoCE/RDMA Verbs。
|
||||||
|
400G 指物理链路的标称线速，不专属于 TCP 或 RDMA；400 Gbit/s 约等于
|
||||||
|
50 GB/s 单向理论上限，不能直接当作 NCCL 或模型端到端可达到的吞吐。
|
||||||
|
|
||||||
|
当前唯一启动入口只允许 `eth0/eth3` 与 `mlx5_0/mlx5_3`，在两端预检并
|
||||||
|
透传 `rdma_cm/uverbs0/uverbs3`，并要求 NCCL INFO 证明两条 HCA 的
|
||||||
|
`NET/IB + GDRDMA` 已启用，否则不开始 benchmark。修正后的 Phase 1
|
||||||
|
正式矩阵 12/12、长 Decode 补测 2/2 均成功。
|
||||||
|
|
||||||
|
## 1. 目标与原则
|
||||||
|
|
||||||
|
### 1.1 最终目标
|
||||||
|
|
||||||
|
在不做 PD 分离的前提下，定位 DeepSeek-V4-Pro 在双机 6000D 上的端到端瓶颈，并提高：
|
||||||
|
|
||||||
|
- 满足 TTFT、TPOT 等 SLO 时的最大吞吐。
|
||||||
|
- 长上下文 Prefill 性能。
|
||||||
|
- Decode 输出吞吐和单请求 TPOT。
|
||||||
|
- 混合流量下的稳定性与 P95/P99 时延。
|
||||||
|
- 16 张 GPU、PCIe 和双 Rail 计算网的有效利用率。
|
||||||
|
|
||||||
|
### 1.2 核心原则
|
||||||
|
|
||||||
|
1. 先找关键路径，再调参数。
|
||||||
|
2. 先用端到端指标确认问题，再用 Timeline 找到阶段，最后才用 Kernel Profiler。
|
||||||
|
3. Prefill、Decode 和混合干扰必须分别测试。
|
||||||
|
4. 一次只改变一个变量，每项优化都要保留可复现的 A/B 结果。
|
||||||
|
5. 单算子更快不代表服务吞吐更高，最终结论必须回到真实请求和 SLO。
|
||||||
|
6. Profiling Run 只用于定位，不能与无 Profiler 的正式性能结果直接比较。
|
||||||
|
|
||||||
|
## 2. 当前最值得验证的瓶颈假设
|
||||||
|
|
||||||
|
| 编号 | 假设 | 为什么值得优先检查 |
|
||||||
|
|---|---|---|
|
||||||
|
| H1 | TP16 每层跨机通信暴露过多 | 两台机器没有跨机 NVLink，TP Collective 需要经过 RoCE |
|
||||||
|
| H2 | NSA Indexer 或 Sparse Attention Kernel 效率不足 | DSV4-Pro 的稀疏注意力路径复杂，Indexer 可能抵消稀疏收益 |
|
||||||
|
| H3 | MoE Grouped GEMM 或路由负载不均 | Decode 小 Batch 容易 Memory-bound，热门专家可能制造慢 Rank |
|
||||||
|
| H4 | 长 Prefill 干扰在线 Decode | 统一实例中 Prefill 与 Decode 竞争计算、显存带宽和调度预算 |
|
||||||
|
| H5 | CPU Scheduler、Metadata 或 Kernel Launch 产生 GPU 空洞 | 小 Batch Decode 对 CPU 和 Launch 开销特别敏感 |
|
||||||
|
| H6 | KV Cache 容量、碎片或 Preemption 限制并发 | 大模型权重占用高，剩余 HBM 决定上下文与并发容量 |
|
||||||
|
| H7 | 当前并行拓扑并非最优 | 使用 16 张卡不等于只能采用单一 TP16 拓扑 |
|
||||||
|
|
||||||
|
## 3. Profiling 总体流程
|
||||||
|
|
||||||
|
```text
|
||||||
|
端到端性能地图
|
||||||
|
↓
|
||||||
|
服务内部指标与硬件计数器
|
||||||
|
↓
|
||||||
|
Nsight Systems 时间线
|
||||||
|
↓
|
||||||
|
确定 1-3 个主要瓶颈
|
||||||
|
↓
|
||||||
|
Nsight Compute 或专项 Microbenchmark
|
||||||
|
↓
|
||||||
|
提出优化并做单变量 A/B
|
||||||
|
↓
|
||||||
|
回到完整 Benchmark 和 SLO 验证
|
||||||
|
```
|
||||||
|
|
||||||
|
不要直接对完整服务运行长时间 Nsight Compute。它的开销很高，也会生成巨大的报告。应先用 Nsight Systems 找到占关键路径的 Kernel，再构造小型复现。
|
||||||
|
|
||||||
|
## 4. Phase 0：冻结可复现环境
|
||||||
|
|
||||||
|
正式测试前，每个 Run 必须保存以下信息：
|
||||||
|
|
||||||
|
- 两台机器的 GPU、Driver、CUDA、NCCL 版本。
|
||||||
|
- vLLM 或 SGLang 的镜像名、镜像 ID、Git Commit 和 Python 包版本。
|
||||||
|
- 模型目录、权重文件校验信息和模型配置。
|
||||||
|
- 完整 Docker Run 与服务启动命令。
|
||||||
|
- 完整 Benchmark 命令。
|
||||||
|
- TP、DP、PP、EP 拓扑。
|
||||||
|
- Attention、NSA、Indexer、MoE、GEMM 和通信 Backend。
|
||||||
|
- `NCCL_SOCKET_IFNAME`、`NCCL_IB_HCA`、`NCCL_CROSS_NIC` 等通信变量。
|
||||||
|
- GPU Memory Fraction、Context Limit、Active Request Limit、KV Cache Dtype。
|
||||||
|
- CUDA Graph、Chunked Prefill、Prefix Cache 和投机解码状态。
|
||||||
|
- 运行前后的 `nvidia-smi`、容器列表和网络状态。
|
||||||
|
|
||||||
|
建议每次运行生成：
|
||||||
|
|
||||||
|
```text
|
||||||
|
results/<RUN_ID>/
|
||||||
|
run_manifest.txt
|
||||||
|
server_cmd.txt
|
||||||
|
bench_cmd.txt
|
||||||
|
summary.csv
|
||||||
|
requests.jsonl
|
||||||
|
server/
|
||||||
|
hardware/
|
||||||
|
profiles/
|
||||||
|
notes.md
|
||||||
|
```
|
||||||
|
|
||||||
|
### 基线约束
|
||||||
|
|
||||||
|
- 初始基线不启用 MTP、EAGLE、DSpark 等投机解码。
|
||||||
|
- 初始基线使用唯一随机 Prompt，避免 Prefix Cache 影响。
|
||||||
|
- 服务启动完成后做固定 Warm-up。
|
||||||
|
- 每个测试点至少重复 3 次。
|
||||||
|
- 正式结果使用无 Profiler 运行。
|
||||||
|
- Profiling 只捕获预热后的少量 Engine Step。
|
||||||
|
|
||||||
|
## 5. Phase 1：建立阶段化性能地图
|
||||||
|
|
||||||
|
详细结果见 [Phase 1 实验档案](./phase1_exp.html)，实现说明见
|
||||||
|
[Phase 1 代码详解](./phase1_code.html)。
|
||||||
|
|
||||||
|
### 5.1 第一轮最小矩阵
|
||||||
|
|
||||||
|
| 场景 | ISL | OSL | 并发 | 主要目标 |
|
||||||
|
|---|---:|---:|---:|---|
|
||||||
|
| P1 短 Prefill 延迟底线 | 1K | 1 | 1 | 固定开销与最小 TTFT |
|
||||||
|
| P2 中长 Prefill | 32K | 1 | 1 | NSA、Indexer、Attention |
|
||||||
|
| P3 长 Prefill | 128K | 1 | 1 | 长上下文计算和显存压力 |
|
||||||
|
| P4 Prefill 吞吐 | 32K | 1 | 逐步加并发 | Chunked Prefill 与输入 TPS |
|
||||||
|
| D1 Decode 延迟底线 | 1K | 1K | 1 | 单请求 TPOT |
|
||||||
|
| D2 Decode 吞吐 | 1K | 1K | 16/32/64 | MoE、Batch 与通信 |
|
||||||
|
| M1 混合负载 | Decode C=32 时注入 128K Prefill | | | Prefill 对在线 Decode 的干扰 |
|
||||||
|
|
||||||
|
长度应以当前已验证的服务容量为上限。如果 128K 不可用，先降到 64K，但必须在 Manifest 中记录原因。
|
||||||
|
|
||||||
|
### 5.2 并发搜索
|
||||||
|
|
||||||
|
沿用 Add-16 加退化回退策略：
|
||||||
|
|
||||||
|
```text
|
||||||
|
C = 1（延迟基线）
|
||||||
|
C = 16, 32, 48, 64, ...
|
||||||
|
```
|
||||||
|
|
||||||
|
停止条件需要区分：
|
||||||
|
|
||||||
|
- `OOM`
|
||||||
|
- `ENGINE_CRASH`
|
||||||
|
- `TTFT_SLO_EXCEEDED`
|
||||||
|
- `TPOT_SLO_EXCEEDED`
|
||||||
|
- `TPS_SATURATED`
|
||||||
|
- `TPS_REGRESSION`
|
||||||
|
- `MAX_CONCURRENCY_REACHED`
|
||||||
|
|
||||||
|
不能把“最大成功并发”“最高 TPS 并发”和“满足 SLO 的最大并发”混为同一个值。
|
||||||
|
|
||||||
|
### 5.3 每个 Case 必须记录
|
||||||
|
|
||||||
|
#### 请求层
|
||||||
|
|
||||||
|
- 实际成功、失败和超时请求数。
|
||||||
|
- 实际输入、输出和总 token 数。
|
||||||
|
- Request Throughput。
|
||||||
|
- Input、Output 和 Total TPS。
|
||||||
|
- P50/P95/P99 TTFT。
|
||||||
|
- P50/P95/P99 TPOT。
|
||||||
|
- P50/P95/P99 ITL。
|
||||||
|
- P50/P95/P99 E2E。
|
||||||
|
- Queue Time 与 Service Time，若框架支持。
|
||||||
|
|
||||||
|
#### Scheduler 层
|
||||||
|
|
||||||
|
- Running、Waiting Request 数。
|
||||||
|
- 每轮 Batch Sequence 数。
|
||||||
|
- 每轮 Prefill、Decode Token 数。
|
||||||
|
- Chunked Prefill 次数和 Chunk 大小。
|
||||||
|
- Forward Step 时间。
|
||||||
|
- Scheduler/Metadata 准备时间。
|
||||||
|
- Preemption、Recompute、Retract 次数。
|
||||||
|
- Prefix Cache Hit Tokens。
|
||||||
|
|
||||||
|
#### 显存层
|
||||||
|
|
||||||
|
- 权重占用。
|
||||||
|
- KV Cache 总量、已用量和峰值。
|
||||||
|
- CUDA Graph 占用。
|
||||||
|
- Workspace 与临时 Tensor 峰值。
|
||||||
|
- Reserved/Allocated 差异与碎片。
|
||||||
|
|
||||||
|
## 6. Phase 2：同步采集硬件指标
|
||||||
|
|
||||||
|
Phase 2 已有正式阶段结果，详见 [Phase 2 实验档案](./phase2_exp.html) 和
|
||||||
|
[Phase 2 代码详解](./phase2_code.html)。
|
||||||
|
最终 Run `dsv4pro-phase2-20260731-163620` 在 28 分 44 秒内完成 8/8
|
||||||
|
benchmark，正式测量窗口 8/8 精确，18/18 个采集器正常启停。两节点
|
||||||
|
DCGM、CPU、NUMA、双 Rail RDMA 和通信微基准数据均有效，实验后容器、
|
||||||
|
端口与 16 张 GPU 已清理。
|
||||||
|
|
||||||
|
最终归因：
|
||||||
|
|
||||||
|
- 混合 Prefill/Decode 令 Output TPS 下降 23.96%、TPOT P95 增加 66.75%。
|
||||||
|
- GPU 未降频、整机 CPU 未饱和，双 Rail 最高约 83.5 Gbit/s/rail 且零错误。
|
||||||
|
- PCIe 跨 NUMA P2P 损失约 2.3%，16-GPU AllReduce busbw 约 39.3-39.7 GB/s。
|
||||||
|
- `NCCL_CROSS_NIC=0/1/2` 差异小于 1%，不再作为主要调优方向。
|
||||||
|
- Phase 3 只需捕获混合 Control/Treatment 的短 Timeline，定位 Kernel、
|
||||||
|
Collective、Scheduler gap 与慢 Rank 同步。
|
||||||
|
|
||||||
|
Phase 2 只提供一个用户入口：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
|
||||||
|
RUN_ID=dsv4pro-phase2-$(date +%Y%m%d-%H%M%S)
|
||||||
|
tmux new-session -d -s dsv4pro-phase2 \
|
||||||
|
"RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
|
||||||
|
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||||
|
```
|
||||||
|
|
||||||
|
`all` 会内部完成 TP16 服务启停、两节点采集、五个固定 Case、混合 A/B、
|
||||||
|
结果汇总和异常清理。不要手工并行执行 Phase 1 的 `start/stop`。
|
||||||
|
|
||||||
|
### 6.1 GPU
|
||||||
|
|
||||||
|
测试期间持续记录：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nvidia-smi \
|
||||||
|
--query-gpu=index,timestamp,utilization.gpu,utilization.memory,\
|
||||||
|
memory.used,memory.total,power.draw,temperature.gpu,clocks.sm,clocks.mem,pstate \
|
||||||
|
--format=csv,noheader,nounits
|
||||||
|
```
|
||||||
|
|
||||||
|
重点观察：
|
||||||
|
|
||||||
|
- SM Utilization。
|
||||||
|
- HBM Utilization。
|
||||||
|
- 显存占用。
|
||||||
|
- GPU Clock、Memory Clock。
|
||||||
|
- Power 与温度降频。
|
||||||
|
- PCIe RX/TX。
|
||||||
|
|
||||||
|
如果有 DCGM，增加：
|
||||||
|
|
||||||
|
- Tensor Core Active。
|
||||||
|
- DRAM Active。
|
||||||
|
- SM Active。
|
||||||
|
- PCIe Throughput。
|
||||||
|
- GPU Stall 与 XID。
|
||||||
|
|
||||||
|
### 6.2 CPU
|
||||||
|
|
||||||
|
记录服务主进程与 Worker 线程：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pidstat -t -p <PID> 1
|
||||||
|
mpstat -P ALL 1
|
||||||
|
numastat -p <PID>
|
||||||
|
```
|
||||||
|
|
||||||
|
需要发现：
|
||||||
|
|
||||||
|
- 单个 Scheduler Thread 是否满核。
|
||||||
|
- Tokenizer、HTTP Frontend 或 Python 线程是否阻塞。
|
||||||
|
- Worker 是否跨 NUMA 访问。
|
||||||
|
- CPU 空洞是否对应 GPU 空洞。
|
||||||
|
|
||||||
|
### 6.3 网络
|
||||||
|
|
||||||
|
Phase 1 正式 Run 已确认容器内可见 RDMA 设备，NCCL 同时识别
|
||||||
|
`mlx5_0/mlx5_3`，跨节点 Channel 使用 `NET/IB + GDRDMA`。Phase 2
|
||||||
|
保留相同 fail-closed 门禁，并同步采集两条 Rail 的流量和错误计数。
|
||||||
|
|
||||||
|
当前拓扑中需要分别观察两条 Compute Rail，确认：
|
||||||
|
|
||||||
|
- 两条 Rail 是否同时有流量。
|
||||||
|
- 带宽是否均衡。
|
||||||
|
- 是否有丢包、重传、PFC Pause 或错误计数。
|
||||||
|
- 慢 Rank 是否固定绑定某个 NIC 或 NUMA 节点。
|
||||||
|
|
||||||
|
基础监控可以使用：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sar -n DEV 1
|
||||||
|
ethtool -S eth0
|
||||||
|
ethtool -S eth3
|
||||||
|
```
|
||||||
|
|
||||||
|
通信调试 Run 可以临时开启：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
NCCL_DEBUG=INFO
|
||||||
|
NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING
|
||||||
|
```
|
||||||
|
|
||||||
|
该日志开销较高，不应在正式性能结果中长期启用。
|
||||||
|
|
||||||
|
## 7. Phase 3：Nsight Systems 时间线
|
||||||
|
|
||||||
|
### 7.1 捕获策略
|
||||||
|
|
||||||
|
- 只捕获预热后的 10 到 32 个 Engine Step；混合场景窗口略长，用于覆盖 Prefill 注入前后。
|
||||||
|
- Prefill、Decode 和混合干扰分别生成报告。
|
||||||
|
- 两台机器分别保存原始报告。
|
||||||
|
- 优先保留所有 Rank；文件过大时至少保留代表 Rank 和跨机通信相关 Rank。
|
||||||
|
- 报告必须和对应 Benchmark Case ID 绑定。
|
||||||
|
|
||||||
|
### 7.2 vLLM
|
||||||
|
|
||||||
|
当前版本支持时，使用 CUDA Profiler 动态 Capture：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||||
|
|
||||||
|
nsys profile \
|
||||||
|
--trace=cuda,nvtx,nccl \
|
||||||
|
--trace-fork-before-exec=true \
|
||||||
|
--cuda-graph-trace=node \
|
||||||
|
--capture-range=cudaProfilerApi \
|
||||||
|
--capture-range-end=repeat \
|
||||||
|
-o /data/profile/dsv4_tp16 \
|
||||||
|
vllm serve ... \
|
||||||
|
--profiler-config.profiler cuda
|
||||||
|
```
|
||||||
|
|
||||||
|
压测端使用支持 Profile Trigger 的 Bench：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
vllm bench serve ... --profile
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7.3 SGLang
|
||||||
|
|
||||||
|
服务启动前设置：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export SGLANG_TORCH_PROFILER_DIR=/data/profile/sglang
|
||||||
|
```
|
||||||
|
|
||||||
|
Profiling 专用 Run 可增加：
|
||||||
|
|
||||||
|
```text
|
||||||
|
--enable-layerwise-nvtx-marker
|
||||||
|
```
|
||||||
|
|
||||||
|
捕获预热后的 10 个 Step：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://127.0.0.1:30000/start_profile \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d '{
|
||||||
|
"output_dir": "/data/profile/sglang",
|
||||||
|
"start_step": 5,
|
||||||
|
"num_steps": 10,
|
||||||
|
"activities": ["CPU", "GPU"]
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
多机 Trace 自动合并要求两台机器能访问同一个共享输出目录。没有共享目录时分别保存，再在本地汇总。
|
||||||
|
|
||||||
|
当前实现位于：
|
||||||
|
|
||||||
|
```text
|
||||||
|
experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
|
||||||
|
```
|
||||||
|
|
||||||
|
正式场景为：
|
||||||
|
|
||||||
|
- Decode Control：`1K -> 1K, C=32`，确认 Decode 活跃后跳过 2 Step，捕获 16 Step。
|
||||||
|
- Mixed Treatment：`1K -> 1K, C=32` 背景先捕获 2 个纯 Decode Step，再注入 `128K -> 1`，总计捕获 32 Step。
|
||||||
|
- Long Prefill：`128K -> 1, C=1`，从首个 Chunk 开始捕获 16 Step。
|
||||||
|
|
||||||
|
双节点 PyTorch/Nsight smoke 已通过。正式 Run 尚未执行，因此暂不生成 Phase 3 的 `exp/code` HTML。正式命令只在 Head 执行：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
|
||||||
|
RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
|
||||||
|
tmux new-session -d -s dsv4pro-phase3 \
|
||||||
|
"RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
|
||||||
|
2>&1 | tee /data/hzy/${RUN_ID}.log"
|
||||||
|
tmux attach -t dsv4pro-phase3
|
||||||
|
```
|
||||||
|
|
||||||
|
三段范围复用一次模型加载，因此 Nsight 使用 `--capture-range-end=repeat:3:defer`，而不是单段示例中的 `stop`。
|
||||||
|
|
||||||
|
### 7.4 时间线必须回答的问题
|
||||||
|
|
||||||
|
1. Prefill 和 Decode 各自的 Top Kernel 是什么？
|
||||||
|
2. NCCL 在关键路径上的暴露时间是多少？
|
||||||
|
3. 通信与计算重叠了多少？
|
||||||
|
4. 每层之间是否存在 CPU 或同步空洞？
|
||||||
|
5. CUDA Graph 是否覆盖常见 Decode Batch？
|
||||||
|
6. 16 个 Rank 是否同时结束？
|
||||||
|
7. 是否存在固定慢 Rank？
|
||||||
|
8. MoE Expert Token 是否严重不均衡？
|
||||||
|
9. NSA Indexer 的成本占 Sparse Attention 总成本多少？
|
||||||
|
10. 长 Prefill 到来时，Decode Kernel 为什么被延迟？
|
||||||
|
|
||||||
|
### 7.5 报告分析
|
||||||
|
|
||||||
|
```bash
|
||||||
|
nsys stats <REPORT>.nsys-rep
|
||||||
|
```
|
||||||
|
|
||||||
|
重点查看：
|
||||||
|
|
||||||
|
- CUDA GPU Kernel Summary。
|
||||||
|
- NCCL Summary。
|
||||||
|
- NCCL GPU Time Utilization。
|
||||||
|
- Communication/Compute Overlap。
|
||||||
|
- NCCL Straggler。
|
||||||
|
- CUDA API Summary。
|
||||||
|
- OS Runtime 和 CPU Thread Timeline。
|
||||||
|
|
||||||
|
## 8. 证据到优化方向的映射
|
||||||
|
|
||||||
|
| 观察到的证据 | 更可能的根因 | 下一项 A/B |
|
||||||
|
|---|---|---|
|
||||||
|
| Decode 中 NCCL 占比高，且通信未被计算覆盖 | TP16 通信受限 | TP8+PP2、NCCL 拓扑与算法 |
|
||||||
|
| C=1 很慢，并发增加后 TPS 明显改善 | MoE/权重读取 Memory-bound | Batch、MoE Backend、MTP |
|
||||||
|
| GPU 利用率呈锯齿，Kernel 间有明显空洞 | CPU Scheduler 或 Launch 开销 | CUDA Graph、异步调度 |
|
||||||
|
| 一个或少数 Rank 长期最慢 | Expert、NIC 或 NUMA 不均衡 | EPLB、Affinity、Rank Mapping |
|
||||||
|
| NSA Indexer 时间接近或超过 Attention | 稀疏索引收益不足 | Indexer Backend、Top-K、融合 |
|
||||||
|
| 长 ISL 的 Attention 时间异常增长 | Prefill Kernel 或 Chunking 问题 | Prefill Backend、Chunk Size |
|
||||||
|
| KV Cache 长期接近满并发生重算 | HBM 容量不足 | FP8 KV、并发和 Context 上限 |
|
||||||
|
| 注入长 Prefill 后 Decode TPOT 暴涨 | Prefill/Decode 相互干扰 | Chunked Prefill 与 Scheduler |
|
||||||
|
| GPU 利用率低但 CPU 单核满载 | Host 端瓶颈 | Frontend、Tokenizer、Scheduler |
|
||||||
|
| 两条 Rail 流量明显不均 | NIC Mapping 或 NCCL 拓扑 | HCA、CROSS_NIC、NUMA Affinity |
|
||||||
|
|
||||||
|
## 9. Phase 4：优先级最高的拓扑实验
|
||||||
|
|
||||||
|
### 9.1 A：TP16 基线
|
||||||
|
|
||||||
|
当前方案用于建立所有后续实验的对照。
|
||||||
|
|
||||||
|
风险是每层 TP Collective 都可能跨越两台机器，Decode 小消息通信尤其容易被延迟支配。
|
||||||
|
|
||||||
|
### 9.2 B：TP8 + PP2
|
||||||
|
|
||||||
|
逻辑上：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Node 5: Pipeline Stage 0, TP8
|
||||||
|
Node 7: Pipeline Stage 1, TP8
|
||||||
|
```
|
||||||
|
|
||||||
|
理想情况下，每卡权重占用与 TP16 接近：
|
||||||
|
|
||||||
|
```text
|
||||||
|
TP16:
|
||||||
|
每卡权重约为 W / 16
|
||||||
|
|
||||||
|
TP8 + PP2:
|
||||||
|
每个 Stage 保存 W / 2
|
||||||
|
Stage 内由 8 卡切分
|
||||||
|
每卡权重约为 (W / 2) / 8 = W / 16
|
||||||
|
```
|
||||||
|
|
||||||
|
潜在收益：
|
||||||
|
|
||||||
|
- 每层 TP Collective 留在单机。
|
||||||
|
- 跨机主要传输 Pipeline Stage 边界激活。
|
||||||
|
- 避免每层都进行跨机 AllReduce。
|
||||||
|
|
||||||
|
潜在代价：
|
||||||
|
|
||||||
|
- Pipeline Bubble。
|
||||||
|
- 低并发延迟可能变差。
|
||||||
|
- KV Cache、Hybrid Cache 和 DSV4-Pro 模型实现可能暂不支持 PP。
|
||||||
|
- 两个 Stage 的计算量可能不均衡。
|
||||||
|
|
||||||
|
测试顺序：
|
||||||
|
|
||||||
|
1. 先做加载与单请求 Smoke Test。
|
||||||
|
2. 对比 C=1 Decode 延迟。
|
||||||
|
3. 对比 C=16/32/64 吞吐。
|
||||||
|
4. 观察跨机网络流量是否显著下降。
|
||||||
|
5. 观察两个 Pipeline Stage 是否负载均衡。
|
||||||
|
|
||||||
|
### 9.3 C：Attention TP8/DP2 + MoE EP16
|
||||||
|
|
||||||
|
目标是：
|
||||||
|
|
||||||
|
- Attention 在节点内使用 TP8。
|
||||||
|
- 两个 Attention DP Group 并行处理请求。
|
||||||
|
- MoE Expert 在 16 张卡上分布。
|
||||||
|
|
||||||
|
这接近“Attention DP + MoE EP”的思路。Expert 权重通常占模型大头，因此即使 Attention 权重复制两份，也有机会放入显存。
|
||||||
|
|
||||||
|
必须先验证：
|
||||||
|
|
||||||
|
- 当前 vLLM/SGLang 版本是否支持 DSV4-Pro 的该拓扑。
|
||||||
|
- Expert 权重、非 Expert 权重和 KV Cache 的实际显存占用。
|
||||||
|
- All-to-All 是否比当前 TP16 AllReduce 更划算。
|
||||||
|
- Expert 负载是否均衡。
|
||||||
|
|
||||||
|
## 10. Phase 5：通信专项
|
||||||
|
|
||||||
|
### 10.1 不只测 1 GiB 大消息
|
||||||
|
|
||||||
|
之前的 1 GiB `all_reduce_perf` 主要说明大消息带宽。Decode 中的 Collective 往往更小，可能由延迟主导。
|
||||||
|
|
||||||
|
需要覆盖真实消息尺度：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
all_reduce_perf -b 8K -e 64M -f 2 -g 8
|
||||||
|
all_gather_perf -b 8K -e 64M -f 2 -g 8
|
||||||
|
reduce_scatter_perf -b 8K -e 64M -f 2 -g 8
|
||||||
|
```
|
||||||
|
|
||||||
|
若启用 EP，还要测试 All-to-All。
|
||||||
|
|
||||||
|
### 10.2 通信优化顺序
|
||||||
|
|
||||||
|
1. 确认两条 Rail 都在工作。
|
||||||
|
2. 确认 Rank、GPU、NIC 和 NUMA Affinity。
|
||||||
|
3. 对照实际模型消息大小。
|
||||||
|
4. 查看 NCCL 自动选择的 Algorithm、Protocol 和 Channel。
|
||||||
|
5. 只有自动选择明显不合理时，才 A/B `Ring/Tree`、`Simple/LL128` 等设置。
|
||||||
|
6. 观察模型端到端结果，而不只看 nccl-tests 峰值。
|
||||||
|
|
||||||
|
## 11. Phase 6：Kernel 专项
|
||||||
|
|
||||||
|
从 Nsight Systems 中选累计占关键路径最高的 1 到 3 个 Kernel，再使用 Nsight Compute。
|
||||||
|
|
||||||
|
DSV4-Pro 的优先怀疑对象：
|
||||||
|
|
||||||
|
- NSA Indexer/Top-K。
|
||||||
|
- Sparse MLA/Attention Prefill。
|
||||||
|
- Sparse MLA/Attention Decode。
|
||||||
|
- MoE Gate、Dispatch、Grouped GEMM、Combine。
|
||||||
|
- FP8 Quant/Dequant 与 Scale Packing。
|
||||||
|
- RMSNorm、Rope、KV Cache Store 等碎片化小算子。
|
||||||
|
- NCCL Collective Kernel。
|
||||||
|
|
||||||
|
需要分析：
|
||||||
|
|
||||||
|
- SM 和 Tensor Core 利用率。
|
||||||
|
- DRAM 吞吐与 L2 Hit Rate。
|
||||||
|
- Occupancy。
|
||||||
|
- Register 与 Shared Memory 压力。
|
||||||
|
- Warp Stall 原因。
|
||||||
|
- Kernel Shape 与 Batch/Token 数。
|
||||||
|
- 小 Kernel Launch 次数。
|
||||||
|
|
||||||
|
优化优先顺序：
|
||||||
|
|
||||||
|
1. 切换已有高性能 Backend。
|
||||||
|
2. 调整 Backend 的 Shape/Workspace/Tile 配置。
|
||||||
|
3. 消除无用 Copy、Cast 和临时 Tensor。
|
||||||
|
4. 融合相邻的 Memory-bound 小算子。
|
||||||
|
5. 现有 Backend 不覆盖关键 Shape 时，再开发新 Kernel 或提交 PR。
|
||||||
|
|
||||||
|
## 12. Phase 7：Scheduler 与统一实例干扰
|
||||||
|
|
||||||
|
### 12.1 混合干扰实验
|
||||||
|
|
||||||
|
先建立稳定 Decode 背景流量：
|
||||||
|
|
||||||
|
```text
|
||||||
|
ISL=1K
|
||||||
|
OSL=1K
|
||||||
|
C=32
|
||||||
|
```
|
||||||
|
|
||||||
|
运行稳定后，周期性注入一个长 Prefill：
|
||||||
|
|
||||||
|
```text
|
||||||
|
ISL=128K
|
||||||
|
OSL=1
|
||||||
|
C=1
|
||||||
|
```
|
||||||
|
|
||||||
|
比较注入前后：
|
||||||
|
|
||||||
|
- Decode P50/P95/P99 TPOT。
|
||||||
|
- Decode Output TPS。
|
||||||
|
- 长请求 TTFT。
|
||||||
|
- 每轮 Prefill Chunk。
|
||||||
|
- Scheduler Queue。
|
||||||
|
- GPU Timeline。
|
||||||
|
|
||||||
|
### 12.2 可调方向
|
||||||
|
|
||||||
|
- Chunked Prefill Size。
|
||||||
|
- Max Prefill Tokens。
|
||||||
|
- Max Batched Tokens。
|
||||||
|
- Max Running Requests/Max Num Seqs。
|
||||||
|
- Prefill 与 Decode 调度优先级。
|
||||||
|
- CUDA Graph Batch Coverage。
|
||||||
|
- 双 Batch Overlap 或框架已有的通算重叠能力。
|
||||||
|
|
||||||
|
调优目标不是单独最大化 Prefill TPS，而是减少长 Prefill 对 Decode SLO 的破坏。
|
||||||
|
|
||||||
|
## 13. Phase 8：显存与缓存
|
||||||
|
|
||||||
|
当前初始值应保持固定，只在发现明确证据后调整：
|
||||||
|
|
||||||
|
- GPU Memory Fraction。
|
||||||
|
- Max Context Length。
|
||||||
|
- Active Request Limit。
|
||||||
|
- KV Cache Dtype。
|
||||||
|
- Page/Block Size。
|
||||||
|
- CUDA Graph Capture Size。
|
||||||
|
|
||||||
|
若 KV Cache 是瓶颈，优先顺序：
|
||||||
|
|
||||||
|
1. 确认权重和 Workspace 的真实占用。
|
||||||
|
2. 检查 Allocated/Reserved 差值与碎片。
|
||||||
|
3. 使用 FP8 KV Cache，前提是当前 Kernel 支持且精度可接受。
|
||||||
|
4. 根据业务上限设置 Context Length，不为不会出现的极端长度预留容量。
|
||||||
|
5. 设置合理的 Active Request Limit，避免运行时 OOM。
|
||||||
|
6. 再考虑 CPU/L3 KV Offload。
|
||||||
|
|
||||||
|
Prefix Cache 单独做第二阶段测试：
|
||||||
|
|
||||||
|
| 命中率 | 用途 |
|
||||||
|
|---:|---|
|
||||||
|
| 0% | 纯计算基线 |
|
||||||
|
| 20% | 低复用业务 |
|
||||||
|
| 50% | 中等公共前缀 |
|
||||||
|
| 80% | Agent/Coding 高复用 |
|
||||||
|
|
||||||
|
Mooncake 或三级缓存只有在 Prefix 可复用时才有明显价值。随机独立 Prompt 不适合评价它。
|
||||||
|
|
||||||
|
## 14. Phase 9：MTP 与模型级优化
|
||||||
|
|
||||||
|
当 TP16 Baseline、并行拓扑、通信、Backend 和 Scheduler 已稳定后，再测试：
|
||||||
|
|
||||||
|
- 原生 MTP。
|
||||||
|
- DSpark。
|
||||||
|
- EAGLE。
|
||||||
|
- KV Cache 量化。
|
||||||
|
- 更低比特权重量化。
|
||||||
|
- Sparse Attention 算法或 Indexer 优化。
|
||||||
|
|
||||||
|
投机解码至少记录：
|
||||||
|
|
||||||
|
- Accept Rate。
|
||||||
|
- Mean Accept Length。
|
||||||
|
- Target Forward TPS。
|
||||||
|
- Draft/MTP 开销。
|
||||||
|
- CPU 调度气泡。
|
||||||
|
- 不同并发下的净收益。
|
||||||
|
|
||||||
|
不能只看 Accept Length，也不能只看 C=1。
|
||||||
|
|
||||||
|
## 15. 里程碑与交付物
|
||||||
|
|
||||||
|
### M1：可信 Baseline
|
||||||
|
|
||||||
|
完成条件：
|
||||||
|
|
||||||
|
- 七组最小矩阵均有 3 次重复。
|
||||||
|
- 同一 Case 的关键 TPS 变异系数尽量不超过 3%。
|
||||||
|
- 所有环境、命令和日志可追溯。
|
||||||
|
|
||||||
|
交付：
|
||||||
|
|
||||||
|
- Baseline Summary。
|
||||||
|
- SLO Frontier。
|
||||||
|
- GPU/CPU/Network Timeline。
|
||||||
|
|
||||||
|
### M2：瓶颈报告
|
||||||
|
|
||||||
|
完成条件：
|
||||||
|
|
||||||
|
- Prefill、Decode、混合三类 Profile 完成。
|
||||||
|
- 找出累计贡献最高的 1 到 3 个瓶颈。
|
||||||
|
- 每个判断都有 Trace、计数器或日志证据。
|
||||||
|
|
||||||
|
交付：
|
||||||
|
|
||||||
|
- `.nsys-rep` 或 Torch Trace。
|
||||||
|
- Kernel/NCCL Summary。
|
||||||
|
- Bottleneck Evidence Table。
|
||||||
|
|
||||||
|
### M3：并行拓扑 A/B
|
||||||
|
|
||||||
|
完成条件：
|
||||||
|
|
||||||
|
- TP16 保留基线。
|
||||||
|
- TP8+PP2 完成可行性与性能验证。
|
||||||
|
- Attention DP + MoE EP 完成支持性和显存评估。
|
||||||
|
|
||||||
|
交付：
|
||||||
|
|
||||||
|
- 每种拓扑的显存、通信、TTFT、TPOT 和 TPS 对比。
|
||||||
|
- 推荐拓扑与不推荐拓扑的证据。
|
||||||
|
|
||||||
|
### M4：首轮优化闭环
|
||||||
|
|
||||||
|
完成条件：
|
||||||
|
|
||||||
|
- 至少一项优化通过完整 Benchmark。
|
||||||
|
- 结果在无 Profiler 环境下可复现。
|
||||||
|
- 正确性无回归。
|
||||||
|
- 满足 SLO 的吞吐有明确改善。
|
||||||
|
|
||||||
|
期望目标：
|
||||||
|
|
||||||
|
- 首轮争取获得至少 10% 的 SLO 内吞吐提升，或显著降低 P95/P99 长尾。
|
||||||
|
- 若无法提升，也必须形成排除结论，说明瓶颈为什么不在该方向。
|
||||||
|
|
||||||
|
## 16. 实验纪律
|
||||||
|
|
||||||
|
每次实验都必须回答：
|
||||||
|
|
||||||
|
1. 改了什么？
|
||||||
|
2. 为什么认为它会影响当前瓶颈？
|
||||||
|
3. 除该变量外，还有什么发生了变化？
|
||||||
|
4. 端到端指标如何变化？
|
||||||
|
5. Profile 证据如何变化？
|
||||||
|
6. 是否引入精度、稳定性或显存风险？
|
||||||
|
7. 是否值得保留？
|
||||||
|
|
||||||
|
禁止以下做法：
|
||||||
|
|
||||||
|
- 同时修改多个参数后只报告最终 TPS。
|
||||||
|
- 用 Profiling Run 和普通 Run 直接比较性能。
|
||||||
|
- 只看平均值，不看 P95/P99。
|
||||||
|
- 用配置 ISL/OSL 估算 TPS，而不核对实际 token 数。
|
||||||
|
- 用 1 GiB NCCL 带宽代表 Decode 小消息性能。
|
||||||
|
- 因单个 Kernel 更快就宣称端到端优化成功。
|
||||||
|
- OOM 后不重启服务继续测试。
|
||||||
|
|
||||||
|
## 17. 首轮执行建议
|
||||||
|
|
||||||
|
建议直接按以下顺序推进：
|
||||||
|
|
||||||
|
1. 固化当前 TP16 服务命令和 Manifest。
|
||||||
|
2. 跑 P1、P2、P3、D1、D2、M1。
|
||||||
|
3. 同步采集 GPU、CPU 和双 Rail 数据。
|
||||||
|
4. 对 P3、D2、M1 各捕获 5 到 10 个 Engine Step。
|
||||||
|
5. 输出 NCCL、NSA/Attention、MoE、CPU Gap 四项时间占比。
|
||||||
|
6. 根据最大暴露时间选择第一个优化方向。
|
||||||
|
7. 优先做 TP16 与 TP8+PP2 的可行性和性能对比。
|
||||||
|
8. 回到完整 Benchmark 验证 SLO 内吞吐。
|
||||||
|
|
||||||
|
最重要的判定标准是：
|
||||||
|
|
||||||
|
> 优化暴露在关键路径上的时间，而不是只优化看起来最慢的单个算子。
|
||||||
|
|
||||||
|
## 18. 参考资料
|
||||||
|
|
||||||
|
- [vLLM Profiling](https://docs.vllm.ai/en/stable/contributing/profiling/)
|
||||||
|
- [SGLang Benchmark and Profiling](https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/benchmark_and_profiling.md)
|
||||||
|
- [SGLang Server Arguments](https://github.com/sgl-project/sglang/blob/main/docs/advanced_features/server_arguments.md)
|
||||||
|
- [NVIDIA Nsight Systems User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html)
|
||||||
|
- [NVIDIA Nsight Systems Analysis Guide](https://docs.nvidia.com/nsight-systems/AnalysisGuide/index.html)
|
||||||
|
- [腾讯混元 Hy3 Preview AI Infra 精读笔记](../hy3_infra_article/Hy3_Preview_AI_Infra_精读笔记.md)
|
||||||
659
docs/hy3_infra_article/Hy3_Preview_AI_Infra_精读笔记.md
Normal file
659
docs/hy3_infra_article/Hy3_Preview_AI_Infra_精读笔记.md
Normal file
@ -0,0 +1,659 @@
|
|||||||
|
# Hy3 Preview AI Infra 推理优化精读笔记
|
||||||
|
|
||||||
|
> 原文：[腾讯混元 AI Infra 如何优化 Hy3 Preview：一次大模型推理性能提升的技术拆解](https://zhuanlan.zhihu.com/p/2053138680768943935)
|
||||||
|
> 作者：混元 AI Infra 推理团队
|
||||||
|
> 发布时间：2026-06-26
|
||||||
|
> 整理时间：2026-07-29
|
||||||
|
> 用途：推理优化学习、实验设计与工程路线参考
|
||||||
|
|
||||||
|
这是一份基于原文及配图整理的技术学习笔记，不是逐字转载。重点是解释每项优化在解决什么瓶颈、为什么有效、依赖什么条件，以及如何映射到我们当前的 vLLM、SGLang、DeepSeek-V4-Flash 和 Kimi-K3 实验。
|
||||||
|
|
||||||
|
## 1. 一页结论
|
||||||
|
|
||||||
|
这篇文章最值得学习的并不是某一个算子，而是它展示了一套完整的推理优化方法：
|
||||||
|
|
||||||
|
1. 先用真实业务数据和明确 SLO 定义目标，而不是只看固定长度随机请求。
|
||||||
|
2. 将 Prefill 和 Decode 分开分析，因为二者的瓶颈、并行策略和优化目标不同。
|
||||||
|
3. 从算子、融合、并行、缓存、调度、量化和稀疏算法六个层级逐层消除瓶颈。
|
||||||
|
4. 不是寻找一个对所有场景都最好的配置，而是围绕业务分布寻找吞吐、时延、容量之间的 Pareto 前沿。
|
||||||
|
5. 单算子加速不等于端到端等比例加速，必须回到真实流量和 SLO 重新测量。
|
||||||
|
|
||||||
|
文章的最终测试口径很有参考价值：
|
||||||
|
|
||||||
|
- 5000 条真实请求。
|
||||||
|
- 最大输入约 192K，平均输入约 68K。
|
||||||
|
- 最大输出约 64K，平均输出约 0.9K。
|
||||||
|
- 理论 Prefix Cache 命中率约 80%。
|
||||||
|
- 硬件为 96 GB Hopper 架构 GPU，结果图标注为 H20。
|
||||||
|
- SLO 为 TTFT 不超过 4 秒、TPOP 不超过 50 毫秒。
|
||||||
|
- 总体测试精度标注为 W8A8C8。
|
||||||
|
|
||||||
|
图中报告的最终单卡吞吐约为：
|
||||||
|
|
||||||
|
- 输入：287.8 万 token/min/GPU，约 47,967 token/s/GPU。
|
||||||
|
- 输出：8.6 万 token/min/GPU，约 1,433 token/s/GPU。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
这里需要特别注意：输入吞吐远高于输出吞吐并不奇怪。文章的真实流量平均输入约 68K、平均输出约 0.9K，输入 token 数本来就比输出多很多；同时 Prefix Cache 命中也会改变 Prefill 的实际计算量。不能只用这两个柱子的比例推断 Prefill 和 Decode 的硬件速度。
|
||||||
|
|
||||||
|
## 2. 模型与问题背景
|
||||||
|
|
||||||
|
Hy3 Preview 是一个 GQA + MoE 模型。官方仓库给出的主要规格包括：
|
||||||
|
|
||||||
|
- 总参数量约 295B，单 token 激活参数约 21B。
|
||||||
|
- 另有约 3.8B 的 MTP 层参数。
|
||||||
|
- 80 层主模型。
|
||||||
|
- 192 个专家，每个 token 激活 8 个专家。
|
||||||
|
- 64 个 Attention Head、8 个 KV Head，Head Dim 为 128。
|
||||||
|
- 原生上下文上限 256K。
|
||||||
|
|
||||||
|
官方模型仓库：[Tencent-Hunyuan/Hy3-preview](https://github.com/Tencent-Hunyuan/Hy3-preview)
|
||||||
|
|
||||||
|
它在 Hopper 96 GB GPU 上主要面对四类矛盾：
|
||||||
|
|
||||||
|
| 矛盾 | 表现 |
|
||||||
|
|---|---|
|
||||||
|
| 长上下文与 TTFT | Prefill 计算量大，混合长度请求造成长尾 |
|
||||||
|
| MoE 与通信 | Expert Dispatch/Combine、TP AllReduce 和跨节点流量较重 |
|
||||||
|
| 权重与 KV Cache | 权重挤压 HBM，限制长上下文和并发容量 |
|
||||||
|
| MTP 与异步调度 | 每轮实际接受 token 数不固定，CPU 无法按传统方式提前准备 |
|
||||||
|
|
||||||
|
文章的优化可以整理成六层：
|
||||||
|
|
||||||
|
| 层级 | 代表技术 | 主要目标 |
|
||||||
|
|---|---|---|
|
||||||
|
| 算子 | 动态 Attention、Router GEMM、FusedMoE | 提高单个热点算子的效率 |
|
||||||
|
| 融合 | Rope/Norm/Quant/KV、AllReduce/Norm/Add、Sampler、GEMM/RS | 减少 Kernel Launch、HBM 往返和通信等待 |
|
||||||
|
| 并行 | Prefill TPSP、Decode Attention-DP + MoE-EP | 为不同阶段选择合适的数据切分 |
|
||||||
|
| 缓存 | GPU、CPU、KVStore 三级缓存 | 扩大 Prefix Cache 容量并支持跨实例复用 |
|
||||||
|
| 调度 | MTP 异步流水 | 隐藏 CPU 调度开销 |
|
||||||
|
| 模型压缩 | W4A8、Attention FP8、Stem 稀疏注意力 | 降低权重、访存和长上下文计算成本 |
|
||||||
|
|
||||||
|
## 3. 算子优化
|
||||||
|
|
||||||
|
### 3.1 Attention：动态切分和负载均衡
|
||||||
|
|
||||||
|
#### 问题
|
||||||
|
|
||||||
|
线上 Batch 中常同时存在长、短请求。静态 Split-KV 必须预先固定切分粒度：
|
||||||
|
|
||||||
|
- 切得太少，长序列不能充分占满 SM。
|
||||||
|
- 切得太多，短序列会承担额外调度、归约和 Kernel 开销。
|
||||||
|
- 长短请求混合时，不同 CTA 的工作量不均，最慢 CTA 决定整次 Kernel 的结束时间。
|
||||||
|
|
||||||
|
#### 方案
|
||||||
|
|
||||||
|
文章采用统一 Tile 粒度加贪心装桶：
|
||||||
|
|
||||||
|
1. 将所有请求拆成统一大小的 Attention Tile。
|
||||||
|
2. 把不同请求产生的 Tile 汇总成一条任务流。
|
||||||
|
3. 根据全局 Tile 数量，为每个 CTA 分配相同或接近的任务预算。
|
||||||
|
4. 每轮推理前生成任务映射表，Attention Kernel 按表领取任务。
|
||||||
|
5. 最后由 Combine Kernel 合并 Split-KV 的局部结果。
|
||||||
|
|
||||||
|
配图中的例子把长度为 1024、5120、2048 的三个请求按 512 token 拆成 2、10、4 个 Tile，再给 4 个 CTA 各分配 4 个 Tile。长请求可以跨 CTA 执行，不再让某个 CTA 单独拖住整批请求。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#### 收益
|
||||||
|
|
||||||
|
- 单 Batch 长文本场景，单算子最高约 2.95 倍加速。
|
||||||
|
- 混合长度 Batch 场景约 1.59 到 1.76 倍加速。
|
||||||
|
|
||||||
|
#### 对我们的启发
|
||||||
|
|
||||||
|
我们当前固定 ISL/OSL Grid 适合测容量边界，但不能验证这种负载均衡优化。要增加一个混合长度测试：
|
||||||
|
|
||||||
|
- 同一 Batch 同时放入 1K、4K、16K、64K、128K 请求。
|
||||||
|
- 保持总 token 数近似相同，对比固定长度 Batch。
|
||||||
|
- 观察 P95/P99 TTFT、GPU SM Occupancy、Attention Kernel 尾部空转时间。
|
||||||
|
|
||||||
|
### 3.2 Router GEMM：用两路 BF16 重构 FP32
|
||||||
|
|
||||||
|
#### 问题
|
||||||
|
|
||||||
|
MoE Router 和稀疏 Attention 的打分对精度敏感，可能需要 FP32 权重。直接执行 BF16 激活乘 FP32 权重会遇到：
|
||||||
|
|
||||||
|
- Tensor Core 路径利用不足。
|
||||||
|
- 激活转成 FP32/TF32 会增加类型转换。
|
||||||
|
- 小 M Shape 下，CUDA Core 路径尤其低效。
|
||||||
|
|
||||||
|
#### 方案
|
||||||
|
|
||||||
|
离线把 FP32 权重拆成高位 BF16 与低位 BF16 残差：
|
||||||
|
|
||||||
|
```text
|
||||||
|
W ≈ W_high + scale * W_low
|
||||||
|
scale = 1 / 256
|
||||||
|
```
|
||||||
|
|
||||||
|
推理时执行两路 BF16 GEMM：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Y = X * W_high^T + scale * (X * W_low^T)
|
||||||
|
```
|
||||||
|
|
||||||
|
两路计算被放进同一个 Kernel：
|
||||||
|
|
||||||
|
- X 只从 HBM 读取一次。
|
||||||
|
- 两路结果分别在寄存器中累加。
|
||||||
|
- Epilogue 中完成修正。
|
||||||
|
- 最终只写回一次结果。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#### 收益
|
||||||
|
|
||||||
|
在 N=192、K=4096、M=2 到 4096 的测试范围内，相比 FP32 cuBLAS 路径约快 2.86 到 3.22 倍。
|
||||||
|
|
||||||
|
#### 对我们的启发
|
||||||
|
|
||||||
|
这不是简单的 `dtype` 开关，而是数值表示、Kernel 实现与模型精度共同设计。它提醒我们：
|
||||||
|
|
||||||
|
- Router 往往是小矩阵，不能用大 GEMM 的经验判断性能。
|
||||||
|
- 看 GPU 利用率时，要单独检查 Router、Indexer 和 Expert GEMM 的 Shape。
|
||||||
|
- 对 DeepSeek/Kimi 的稀疏路由，需要区分“精度敏感的小算子”和“吞吐主导的大算子”。
|
||||||
|
|
||||||
|
### 3.3 FusedMoE：重排完整专家执行链
|
||||||
|
|
||||||
|
文章不是只替换 Grouped GEMM，而是重构了整个 MoE 数据通路：
|
||||||
|
|
||||||
|
1. 在共享内存中分块统计路由结果，并为每个专家预留连续输出区间。
|
||||||
|
2. Gate-Up GEMM 直接按路由索引读取原始输入，省略显式 Gather。
|
||||||
|
3. 取消部分 Warp Specialization，以提高 SM 驻留密度。
|
||||||
|
4. 激活量化结果按专家连续写入，供 Down GEMM 顺序读取。
|
||||||
|
5. 末端直接完成 Top-K 加权聚合，减少中间 HBM 往返。
|
||||||
|
6. 用 PDL 串联阶段，降低频繁 Kernel Launch 形成的空隙。
|
||||||
|
|
||||||
|
报告的单算子收益：
|
||||||
|
|
||||||
|
- TP=8、EP=1：相比 vLLM CUTLASS、vLLM Triton 和 SGLang 路径约快 1.5 到 1.6 倍。
|
||||||
|
- TP=1、EP=8：约快 1.2 到 1.5 倍。
|
||||||
|
|
||||||
|
开源实现：[Tencent/hpc-ops](https://github.com/Tencent/hpc-ops)
|
||||||
|
|
||||||
|
这里有一个很重要的实验原则：同一个 MoE Kernel 在 TP8/EP1 与 TP1/EP8 下的收益不同，因为每卡 Expert 数、每个 Expert 收到的 token 数、通信方式和矩阵 Shape 都变了。比较 MoE Backend 时必须固定完整的 TP/DP/EP 拓扑。
|
||||||
|
|
||||||
|
## 4. 算子融合
|
||||||
|
|
||||||
|
### 4.1 Rope + Norm + Hadamard + Quant + Store KV
|
||||||
|
|
||||||
|
QKV Projection 之后通常存在一串算术强度很低的 Element-wise 操作。若每一步都是独立 Kernel，就会反复：
|
||||||
|
|
||||||
|
- 从 HBM 读数据。
|
||||||
|
- 写回中间结果。
|
||||||
|
- 发起新的 Kernel。
|
||||||
|
|
||||||
|
文章把 Rope、RMSNorm、Hadamard、量化和 KV Cache 写入融合为一个 Kernel。中间值尽量停留在寄存器中，最后直接以低比特格式写入 KV Cache。
|
||||||
|
|
||||||
|
报告的融合算子加速约 5 倍。它体现的是典型原则：
|
||||||
|
|
||||||
|
> 对访存受限的小算子，减少一次 HBM 往返往往比减少几次算术操作更重要。
|
||||||
|
|
||||||
|
### 4.2 AllReduce + Norm + Add
|
||||||
|
|
||||||
|
TP 路径通常按以下顺序执行：
|
||||||
|
|
||||||
|
```text
|
||||||
|
AllReduce -> Residual Add -> RMSNorm
|
||||||
|
```
|
||||||
|
|
||||||
|
拆开执行会产生通信等待和中间 Tensor 读写。文章把它融合为：
|
||||||
|
|
||||||
|
```text
|
||||||
|
RMSNorm(AllReduce(x) + residual, weight)
|
||||||
|
```
|
||||||
|
|
||||||
|
提供两类实现：
|
||||||
|
|
||||||
|
- Prefill 高吞吐路径：利用 NVSwitch 多播，面向较大的 token Batch。
|
||||||
|
- Decode 低延迟路径：使用 Lamport P2P，并用 PDL 让两个 Kernel 重叠。
|
||||||
|
|
||||||
|
覆盖约 8K 到 32K token 的场景，相比 NCCL 和 FlashInfer 同类路径最高约快 1.68 倍。
|
||||||
|
|
||||||
|
这说明通信优化不能只看 NCCL Bandwidth。对于小消息和 Decode，Kernel Launch、同步点与后处理往往和网络带宽同样重要。
|
||||||
|
|
||||||
|
### 4.3 Sampler 融合
|
||||||
|
|
||||||
|
常规采样可能包含重复惩罚、温度缩放、Top-K、Top-P、Softmax 和随机采样等十余个 Kernel。文章将其压缩成两个核心 CUDA Kernel，并根据简单温度采样或完整采样选择专用路径。
|
||||||
|
|
||||||
|
关键设计：
|
||||||
|
|
||||||
|
- 全局词表尽量只读取一次。
|
||||||
|
- 重复惩罚掩码留在 GPU 内处理。
|
||||||
|
- 单请求可拆给多个 CTA。
|
||||||
|
- Max Top-K 不超过 64 时使用局部堆归并。
|
||||||
|
- Top-K 与 Softmax 的 max/sum 归约融合。
|
||||||
|
|
||||||
|
下图直观展示了融合前后的 profiler 时间线：融合前有大量碎片化 Kernel，融合后主体工作集中到少数长 Kernel。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
文章报告相较 vLLM 与 FlashInfer 的采样路径分别约有 5.5 倍和 2.5 倍单算子提升。端到端收益仍取决于输出长度、Batch 和模型主体计算占比。
|
||||||
|
|
||||||
|
### 4.4 GEMM + ReduceScatter 细粒度重叠
|
||||||
|
|
||||||
|
传统执行顺序是完整 GEMM 结束后再开始 ReduceScatter。文章将 SM 分成两类角色：
|
||||||
|
|
||||||
|
- 计算 SM：执行 GEMM。
|
||||||
|
- 通信 SM：搬运已经完成的输出 Tile。
|
||||||
|
|
||||||
|
计算 SM 每生成一个 Tile，就写入本地 Buffer 并通知通信 SM；通信不必等待整个矩阵完成。
|
||||||
|
|
||||||
|
此外，GEMM 内部又划分为三级 Warp 流水：
|
||||||
|
|
||||||
|
```text
|
||||||
|
Load Warp -> MMA Warp -> Epilogue Warp
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
在 M 为 8K、16K、32K、64K 的四组 Shape 上，通信覆盖率约从 76.5% 增长到 84.8%，端到端相较串行路径约快 1.68 到 1.81 倍。
|
||||||
|
|
||||||
|
这个方向对多机 TP/EP 特别重要。我们以后跑 NCCL Test 只能知道通信上限，真正的模型吞吐还取决于能否把通信藏在计算后面。
|
||||||
|
|
||||||
|
## 5. Prefill 与 Decode 的并行策略
|
||||||
|
|
||||||
|
### 5.1 Prefill：TPSP
|
||||||
|
|
||||||
|
文章认为 Hy3 Preview 的纯 TP8 Prefill 有三个问题：
|
||||||
|
|
||||||
|
1. Norm、Router 等 token-wise 算子在各 TP Rank 重复计算。
|
||||||
|
2. 频繁 AllReduce 交换完整激活。
|
||||||
|
3. MoE Grouped GEMM 沿 Hidden 维切得过窄，Shape 不利于 Tensor Core。
|
||||||
|
|
||||||
|
因此，它没有让整层始终使用同一种并行方式，而是在不同模块切换布局。配图给出的一层时间线包含：
|
||||||
|
|
||||||
|
- Attention 使用 TP8。
|
||||||
|
- Routed Expert 使用 TP4 + SP2。
|
||||||
|
- Shared Expert 沿 token/sequence 维使用 SP8。
|
||||||
|
- AG + QKV 和 RS + O Projection 做通信计算融合。
|
||||||
|
- Shared Expert 与通信使用多 Stream 重叠。
|
||||||
|
- AllGather 通信采用 FP8，图中说明可比 BF16 减少约 50% 通信带宽。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
端到端 Prefill TTFT：
|
||||||
|
|
||||||
|
| 输入长度 | 优化前 | 优化后 | 降幅 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| 16K | 764 ms | 536 ms | 29.9% |
|
||||||
|
| 32K | 1885 ms | 1424 ms | 24.5% |
|
||||||
|
|
||||||
|
#### 对我们的启发
|
||||||
|
|
||||||
|
“TP 越小通信越少，所以一定更快”是不完整的。TP 改变的不只是通信量，还会改变：
|
||||||
|
|
||||||
|
- 每卡权重与 KV Cache 容量。
|
||||||
|
- GEMM 的 M/N/K Shape。
|
||||||
|
- 是否存在重复 token-wise 计算。
|
||||||
|
- Batch 在 DP Rank 之间的分散程度。
|
||||||
|
- 是否能使用特定融合算子。
|
||||||
|
|
||||||
|
因此，TP2/DP4、TP4/DP2、TP8/DP1 必须端到端实测，不能只用通信直觉排序。
|
||||||
|
|
||||||
|
### 5.2 Decode：Attention DP + MoE EP
|
||||||
|
|
||||||
|
Decode 阶段通常 Batch 较小，单 token GEMM 更偏 Memory-bound。文章采用 Attention DP 与 MoE EP 的混合并行：
|
||||||
|
|
||||||
|
- Attention 权重在 DP Rank 上复制，让请求可以分开执行。
|
||||||
|
- Expert 权重按 EP Rank 分布，减少每卡权重占用。
|
||||||
|
- 多节点请求汇聚到 Expert 后形成更大的 Grouped GEMM Batch。
|
||||||
|
- 使用异步 EPLB，根据真实专家负载重排权重。
|
||||||
|
- Shared Expert 计算与 Dispatch/Combine 尽量重叠。
|
||||||
|
- 长序列 Attention 使用 DPTP 混合方式缓解 DP Rank 负载不均。
|
||||||
|
|
||||||
|
报告的端到端吞吐提升约为 15.7% 到 44.7%。
|
||||||
|
|
||||||
|
这和我们之前 Custom DP 的现象能够对应：
|
||||||
|
|
||||||
|
- 短上下文、高并发时，独立实例容易各自形成稳定 Batch，Custom DP 可能反超。
|
||||||
|
- 低并发时，请求被分散后每个实例 Batch 太小，GPU 利用率下降。
|
||||||
|
- 长上下文时，Prefill 和 KV Cache 压力成为主导，简单 Round Robin 无法替代全局调度与混合并行。
|
||||||
|
|
||||||
|
## 6. GPU、CPU、KVStore 三级缓存
|
||||||
|
|
||||||
|
文章把 Prefix Cache 扩展成三级：
|
||||||
|
|
||||||
|
| 层级 | 介质 | 特点 | 复用范围 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| L1 | GPU HBM | 延迟最低、容量最小 | GPU 进程 |
|
||||||
|
| L2 | CPU DRAM | 容量更大、回载较快 | 实例内部 |
|
||||||
|
| L3 | 本地盘或共享 KVStore | 容量最大、延迟最高 | 本机或跨实例 |
|
||||||
|
|
||||||
|
完整请求流程：
|
||||||
|
|
||||||
|
1. Scheduler 先查 L1 GPU Prefix Cache。
|
||||||
|
2. 对未命中部分查询 L2/L3。
|
||||||
|
3. 命中的完整 KV Block 按需加载回 GPU。
|
||||||
|
4. 跳过已经命中的 Prefix Prefill。
|
||||||
|
5. 新生成的完整 KV Block 异步下沉到 L2/L3。
|
||||||
|
6. L3 连续读取失败时降级到 CPU-only，避免外部存储故障拖垮服务。
|
||||||
|
|
||||||
|
配图中的 L3 Backend 可以是：
|
||||||
|
|
||||||
|
- HoverDB 本地磁盘：本机持久化缓存。
|
||||||
|
- NitroFS 共享存储：支持跨实例复用。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#### 与 Mooncake 的关系
|
||||||
|
|
||||||
|
这正是 Mooncake/HiCache 一类系统的价值所在。即使不开 PD 分离，多级缓存仍能在以下场景产生价值：
|
||||||
|
|
||||||
|
- 多轮 Agent 对话存在长公共前缀。
|
||||||
|
- Coding 请求反复携带同一仓库上下文。
|
||||||
|
- 实例扩缩容、迁移或重启后仍希望复用 Prefix。
|
||||||
|
- GPU HBM 不足，希望把冷 KV 下沉到 CPU、SSD 或远端存储。
|
||||||
|
|
||||||
|
但如果测试流量全部是独立随机 token，几乎没有共享前缀，L2/L3 缓存只会增加查找和搬运开销。因此必须显式设计 0%、20%、50%、80% 命中率的测试组。
|
||||||
|
|
||||||
|
## 7. MTP 与异步调度
|
||||||
|
|
||||||
|
### 7.1 传统异步调度为什么失效
|
||||||
|
|
||||||
|
普通 Decode 每轮稳定生成一个 token，CPU 可以在 GPU 执行第 N 轮时提前准备第 N+1 轮。
|
||||||
|
|
||||||
|
MTP 会一次草拟多个 token，但实际接受长度是动态的。下一轮的：
|
||||||
|
|
||||||
|
- Sequence Length。
|
||||||
|
- Position ID。
|
||||||
|
- KV Cache Block 映射。
|
||||||
|
- 输入 token 布局。
|
||||||
|
|
||||||
|
都依赖本轮验证结果。若 CPU 必须等待 GPU 把接受长度拷回，就会重新出现同步气泡。
|
||||||
|
|
||||||
|
### 7.2 文章的方案
|
||||||
|
|
||||||
|
CPU 暂时不等待真实接受长度，而是：
|
||||||
|
|
||||||
|
1. 按最大可能接受长度插入 Placeholder。
|
||||||
|
2. 提前准备并 Launch 下一轮。
|
||||||
|
3. 真实接受长度继续保留在 GPU。
|
||||||
|
4. 下一轮正式计算前，再由 GPU 修正 Position、KV 映射等关键状态。
|
||||||
|
|
||||||
|
这样 CPU 可以提前一整轮，而不是只和很短的 MTP Layer Forward 重叠。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
报告结果：
|
||||||
|
|
||||||
|
- 每轮减少约 5 到 10 ms 的 CPU 气泡。
|
||||||
|
- 端到端性能提升约 10% 到 20%。
|
||||||
|
|
||||||
|
#### 对我们的启发
|
||||||
|
|
||||||
|
投机解码测试不能只记录 Accept Length。至少要同时记录：
|
||||||
|
|
||||||
|
- Target Model Decode TPS。
|
||||||
|
- Draft/MTP 接受长度和接受率。
|
||||||
|
- 每轮 CPU 调度时间。
|
||||||
|
- GPU 间隙和 Kernel Launch 间隔。
|
||||||
|
- 不同 Batch 下的收益。
|
||||||
|
|
||||||
|
小 Batch 时 CPU 气泡占比高，MTP 调度优化可能很重要；大 Batch 时 Target Forward 本身更重，收益比例可能下降。
|
||||||
|
|
||||||
|
## 8. W4A8、Attention FP8 与精度恢复
|
||||||
|
|
||||||
|
文章的压缩链路是：
|
||||||
|
|
||||||
|
1. SmoothQuant 风格的激活平滑，抑制少数通道的离群值。
|
||||||
|
2. Attention 的 Query/Key 在量化前做 Hadamard 正交旋转，把离群值打散。
|
||||||
|
3. 使用 GPTQ 做逐层权重重建，根据二阶信息补偿低比特权重误差。
|
||||||
|
4. 做轻量级 QAT，仅更新量化相关参数，使模型适应任务分布。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
报告称：
|
||||||
|
|
||||||
|
- 多领域评测与 BF16 基线的差距控制在约 1% 以内。
|
||||||
|
- 端到端吞吐提升超过 28%。
|
||||||
|
|
||||||
|
需要区分两种口径：
|
||||||
|
|
||||||
|
- 文章开头的总体线上结果标注为 W8A8C8。
|
||||||
|
- 量化章节进一步讨论的是 W4A8 + Attention FP8 路线。
|
||||||
|
|
||||||
|
二者不能当成同一套权重和同一组最终吞吐数据。
|
||||||
|
|
||||||
|
开源工具：[Tencent/AngelSlim](https://github.com/tencent/AngelSlim)
|
||||||
|
|
||||||
|
#### 对我们的路线判断
|
||||||
|
|
||||||
|
这部分不适合当前最先做，因为它可能涉及 Calibration、GPTQ 重建和 QAT。优先级应该低于：
|
||||||
|
|
||||||
|
- 正确部署和基线测量。
|
||||||
|
- TP/DP/EP 与 Scheduler 调优。
|
||||||
|
- Prefix Cache 和多级缓存。
|
||||||
|
- Backend 与已有 Kernel 的选择。
|
||||||
|
|
||||||
|
当系统参数已稳定，并且确实被权重容量或 HBM 带宽限制时，再进入量化训练与精度评估。
|
||||||
|
|
||||||
|
## 9. Stem 稀疏注意力
|
||||||
|
|
||||||
|
Stem 的目标是在长上下文 Prefill 中，只计算最有价值的一部分 Attention Block。
|
||||||
|
|
||||||
|
### 9.1 Token Position Decay
|
||||||
|
|
||||||
|
普通 Uniform Top-K 对不同 Query 位置使用相同预算。Stem 认为：
|
||||||
|
|
||||||
|
- 序列头部 token 会参与更多后续因果聚合，误差可能逐层传播。
|
||||||
|
- 序列尾部 token 的影响范围较小，可以更激进地稀疏。
|
||||||
|
|
||||||
|
因此 Top-K 预算从头部的 `k_start` 逐渐衰减到尾部：
|
||||||
|
|
||||||
|
```text
|
||||||
|
k_end = mu * k_start
|
||||||
|
```
|
||||||
|
|
||||||
|
在总计算预算近似不变时，把更多预算留给影响更大的早期位置。
|
||||||
|
|
||||||
|
### 9.2 Output-Aware Metric
|
||||||
|
|
||||||
|
仅按 `QK^T` 选 token，只衡量注意力路由概率，没有衡量 Value 实际携带的信息强度。Stem 加入 Value 向量模长：
|
||||||
|
|
||||||
|
```text
|
||||||
|
M(i, j) = QK^T + beta * max(0, log(||V_j||_2))
|
||||||
|
```
|
||||||
|
|
||||||
|
然后基于该分数做 Top-K，并交给 Block Sparse Flash Attention 计算。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### 9.3 性能与精度
|
||||||
|
|
||||||
|
文章给出的长上下文 Prefill 加速：
|
||||||
|
|
||||||
|
| 长度 | FA3 BF16 | FA3 FP8 | Stem |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| 16K | 1.27x | 1.45x | 1.50x |
|
||||||
|
| 32K | 1.36x | 1.73x | 1.96x |
|
||||||
|
| 64K | 1.42x | 2.02x | 2.68x |
|
||||||
|
| 128K | 1.47x | 2.29x | 3.62x |
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
效果随长度增长而放大，符合稠密 Attention 计算复杂度快速增长的直觉。
|
||||||
|
|
||||||
|
配图还比较了 BF16 与“FP8-W8A8 + Stem”的多个任务分数。后者在不同任务上有小幅升降，例如 LongBench v2 和 SWE-bench Verified 约下降 2 个绝对分，Terminal-Bench 基本持平，ClawEval 略有提升。不能只看平均值，需要为实际业务单独设精度门槛。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
#### 对我们的意义
|
||||||
|
|
||||||
|
你以前做过稀疏注意力基模工作，这一块很适合作为中后期深入方向，但需要把“算法”和“系统”同时验证：
|
||||||
|
|
||||||
|
- 稀疏索引本身的计算是否抵消节省。
|
||||||
|
- Indexer 在 Prefill/Decode 的 Shape 是否覆盖。
|
||||||
|
- Block Pattern 能否被现有 Kernel 高效执行。
|
||||||
|
- 稀疏 KV 的布局是否引入额外 Gather。
|
||||||
|
- 128K 以上是否仍保持精度。
|
||||||
|
- Chunked Prefill 是否改变选块逻辑或数值结果。
|
||||||
|
|
||||||
|
## 10. 如何正确理解文章中的加速数字
|
||||||
|
|
||||||
|
### 10.1 不要把所有加速比相乘
|
||||||
|
|
||||||
|
例如 Attention 2.95x、融合算子 5x、FusedMoE 1.6x 并不意味着端到端能快几十倍。Amdahl 定律决定了：
|
||||||
|
|
||||||
|
```text
|
||||||
|
总体收益 = 1 / (未优化部分 + 优化部分 / 加速比)
|
||||||
|
```
|
||||||
|
|
||||||
|
而且不同优化可能覆盖同一段时间，收益会重叠。
|
||||||
|
|
||||||
|
### 10.2 固定长度 Grid 与真实数据各有用途
|
||||||
|
|
||||||
|
| 方法 | 适合回答的问题 | 不适合回答的问题 |
|
||||||
|
|---|---|---|
|
||||||
|
| 固定 ISL/OSL/C Grid | 容量边界、Shape 性能、OOM 点、参数敏感度 | 真实 P95/P99、缓存收益、混合长度长尾 |
|
||||||
|
| 真实 Trace | 线上吞吐、SLO 达标率、Prefix Cache、调度效果 | 精确定位某个 Shape 的 Kernel 问题 |
|
||||||
|
|
||||||
|
正确做法不是二选一，而是：
|
||||||
|
|
||||||
|
1. 用 Grid 画出系统性能和容量地图。
|
||||||
|
2. 用真实 Trace 验证业务加权结果。
|
||||||
|
3. 对真实 Trace 暴露出的热点 Shape 再回到 Microbenchmark 和 Profiler。
|
||||||
|
|
||||||
|
### 10.3 文章没有完全披露的变量
|
||||||
|
|
||||||
|
做横向对比时还需要确认：
|
||||||
|
|
||||||
|
- 总 GPU 数与节点数。
|
||||||
|
- vLLM/SGLang 的具体版本和 Baseline 参数。
|
||||||
|
- Cache 命中是按请求、token 还是 block 计算。
|
||||||
|
- 输入 TPM 是否统计逻辑输入 token，还是实际执行 Prefill 的 token。
|
||||||
|
- MTP 接受率和平均接受长度。
|
||||||
|
- 量化精度数据与总体 W8A8C8 吞吐是否来自同一配置。
|
||||||
|
- 各单算子收益对应的 Batch、并行拓扑和频率锁定条件。
|
||||||
|
|
||||||
|
因此，这篇文章非常适合作为优化地图，但不能直接把数字当成我们的性能目标。
|
||||||
|
|
||||||
|
## 11. 映射到我们当前的工程路线
|
||||||
|
|
||||||
|
### 阶段 A：建立可信 Baseline
|
||||||
|
|
||||||
|
- 固定代码、镜像、模型权重和驱动版本。
|
||||||
|
- 保留 TP2/DP4、TP4/DP2、TP8/DP1 的 Shape Grid。
|
||||||
|
- 同时记录 TTFT、TPOT、ITL、E2E、请求吞吐、输入/输出/总 TPS。
|
||||||
|
- 记录实际成功请求的 Prompt/Output token，避免只用配置长度估算 TPS。
|
||||||
|
- 增加 GPU、HBM、PCIe/NVLink/RDMA、CPU 利用率和服务日志。
|
||||||
|
|
||||||
|
### 阶段 B：调度、缓存与并行
|
||||||
|
|
||||||
|
- 比较原生 DP 与 Custom DP。
|
||||||
|
- 对短上下文高并发和长上下文分别选择路由策略。
|
||||||
|
- 测试 Prefix Cache 命中率 0%、20%、50%、80%。
|
||||||
|
- 测试 GPU-only、GPU+CPU、GPU+CPU+Mooncake/KVStore。
|
||||||
|
- 对 MoE 分别测试 TP 主导、EP 主导和 Attention DP + MoE EP。
|
||||||
|
|
||||||
|
### 阶段 C：Profiler 驱动的 Kernel 优化
|
||||||
|
|
||||||
|
- 用 Nsight Systems 找 GPU 空洞、CPU 调度气泡和通信等待。
|
||||||
|
- 用 Nsight Compute 找热点 Kernel 的访存、Occupancy 和 Tensor Core 利用率。
|
||||||
|
- 先尝试已有 Backend：FlashInfer、FlashMLA、DeepGEMM、CUTLASS、Marlin、HPC-Ops。
|
||||||
|
- 只有现有 Backend 不覆盖关键 Shape 时，才值得自己写算子或提交 PR。
|
||||||
|
|
||||||
|
### 阶段 D：模型相关优化
|
||||||
|
|
||||||
|
- MTP/DSpark/EAGLE。
|
||||||
|
- W4A8、Attention FP8、KV Cache 量化。
|
||||||
|
- Stem/DSA 等稀疏 Attention。
|
||||||
|
- 精度回归、Calibration 和必要的轻量训练。
|
||||||
|
|
||||||
|
## 12. 建议补充的实验矩阵
|
||||||
|
|
||||||
|
### 12.1 真实流量 Baseline
|
||||||
|
|
||||||
|
先构建一个与文章接近但适合当前模型的 Trace：
|
||||||
|
|
||||||
|
- 输入长度按 1K、4K、16K、64K、128K、192K 分桶。
|
||||||
|
- 输出长度以 1K 左右为中心，同时保留短输出和长输出尾部。
|
||||||
|
- 混合长度请求一起进入服务。
|
||||||
|
- 每次测试至少数百到数千请求，保证 P95/P99 有意义。
|
||||||
|
|
||||||
|
### 12.2 Prefix Cache
|
||||||
|
|
||||||
|
| 变量 | 建议取值 |
|
||||||
|
|---|---|
|
||||||
|
| 前缀命中率 | 0%、20%、50%、80% |
|
||||||
|
| 前缀长度 | 4K、16K、64K、128K |
|
||||||
|
| Cache 层级 | GPU、GPU+CPU、GPU+CPU+L3 |
|
||||||
|
| 实例范围 | 单实例、跨实例 |
|
||||||
|
|
||||||
|
### 12.3 并行策略
|
||||||
|
|
||||||
|
| 阶段 | 候选策略 | 重点指标 |
|
||||||
|
|---|---|---|
|
||||||
|
| Prefill | TP8、TP/SP 混合、Chunked Prefill | TTFT、输入 TPS、通信时间 |
|
||||||
|
| Decode | TP、DP、Attention DP + MoE EP | TPOT、输出 TPS、负载均衡 |
|
||||||
|
| 多机 | TP/EP、DP/EP、PD 分离 | 网络流量、跨机长尾、容错 |
|
||||||
|
|
||||||
|
### 12.4 SLO 吞吐边界
|
||||||
|
|
||||||
|
不要只找 Total TPS 最大点。每个 Shape 应同时输出：
|
||||||
|
|
||||||
|
- 最大成功并发。
|
||||||
|
- Total TPS 峰值并发。
|
||||||
|
- 满足 TTFT SLO 的最大并发。
|
||||||
|
- 满足 TPOT SLO 的最大并发。
|
||||||
|
- 同时满足全部 SLO 的最大并发。
|
||||||
|
|
||||||
|
这些并发可能不是同一个点。
|
||||||
|
|
||||||
|
## 13. 阅读文章时需要记住的十个问题
|
||||||
|
|
||||||
|
1. 当前瓶颈属于 Prefill 还是 Decode？
|
||||||
|
2. 是 Compute-bound、Memory-bound、Communication-bound，还是 CPU-bound？
|
||||||
|
3. 优化改变了计算量，还是只改变了数据搬运和重叠？
|
||||||
|
4. 收益对应什么 Batch、Shape、TP/DP/EP？
|
||||||
|
5. 单算子收益在端到端占比是多少？
|
||||||
|
6. 是否依赖 NVLink/NVSwitch、RDMA、特定 GPU 架构？
|
||||||
|
7. 是否需要新权重、Calibration、QAT 或模型结构支持？
|
||||||
|
8. 是否改变数值结果或精度？
|
||||||
|
9. 对低并发、长上下文和混合长度是否仍成立？
|
||||||
|
10. 最终是否提高了满足 SLO 的吞吐，而不只是无约束峰值 TPS？
|
||||||
|
|
||||||
|
## 14. 术语速查
|
||||||
|
|
||||||
|
| 术语 | 含义 |
|
||||||
|
|---|---|
|
||||||
|
| CTA | CUDA Thread Block，Kernel 调度到 SM 的基本工作单元 |
|
||||||
|
| Tile | 对矩阵或序列任务做的固定粒度切块 |
|
||||||
|
| Split-KV | 将长 Attention 的 KV 维拆给多个 CTA，再合并局部结果 |
|
||||||
|
| PDL | Programmatic Dependent Launch，用于减少依赖 Kernel 之间的启动气泡 |
|
||||||
|
| SP | Sequence Parallel，沿 token/sequence 维切分 |
|
||||||
|
| TPSP | Tensor Parallel 与 Sequence Parallel 的混合布局 |
|
||||||
|
| EPLB | Expert Parallel Load Balancing，专家并行负载均衡 |
|
||||||
|
| TPOP | Time Per Output Token，与 TPOT 接近，衡量连续吐字速度 |
|
||||||
|
| Prefix Cache | 复用相同前缀已经生成的 KV Cache |
|
||||||
|
| W8A8C8 | 权重、激活和缓存均使用 8-bit 的总体精度标记，具体格式需看实现 |
|
||||||
|
| W4A8 | 4-bit 权重、8-bit 激活 |
|
||||||
|
| MTP | Multi-Token Prediction，一轮提出或预测多个后续 token |
|
||||||
|
| OAM | Output-Aware Metric，用 Value 强度修正稀疏 Attention 选块分数 |
|
||||||
|
|
||||||
|
## 15. 延伸资料
|
||||||
|
|
||||||
|
- [原始知乎文章](https://zhuanlan.zhihu.com/p/2053138680768943935)
|
||||||
|
- [Hy3 Preview 官方仓库](https://github.com/Tencent-Hunyuan/Hy3-preview)
|
||||||
|
- [HPC-Ops](https://github.com/Tencent/hpc-ops)
|
||||||
|
- [AngelSlim](https://github.com/tencent/AngelSlim)
|
||||||
|
- [腾讯混元 AI Infra 新开源：HPC-Ops 推理核心算子全面升级](https://developer.cloud.tencent.com/article/2688857)
|
||||||
|
- [Mooncake](https://github.com/kvcache-ai/Mooncake)
|
||||||
|
|
||||||
|
## 16. Mentor 结论
|
||||||
|
|
||||||
|
这篇文章可以作为我们后续推理优化工作的总地图，但学习顺序不要反过来。
|
||||||
|
|
||||||
|
当前最值得优先复刻的是：
|
||||||
|
|
||||||
|
1. 真实流量 + SLO 的 Benchmark 方法。
|
||||||
|
2. Prefill/Decode 分阶段分析。
|
||||||
|
3. TP/DP/EP 与混合长度负载的系统实验。
|
||||||
|
4. Prefix Cache 和多级缓存。
|
||||||
|
5. Profiler 驱动的 Backend 与融合优化。
|
||||||
|
|
||||||
|
量化、MTP 和稀疏 Attention 很有价值，但更依赖模型结构、精度评估和训练支持。等 Baseline、调度、并行与缓存做扎实之后，再进入这些方向，收益会更容易被正确测量，也更容易形成有说服力的技术成果。
|
||||||
@ -0,0 +1,154 @@
|
|||||||
|
# DeepSeek-V4-Flash 在 B300 上的文本性能报告
|
||||||
|
|
||||||
|
## 1. 结论摘要
|
||||||
|
|
||||||
|
本轮在一台 8 卡 NVIDIA B300 服务器上测试 DeepSeek-V4-Flash 的 Low-Latency 与 Balanced 两套部署方案；两套服务均以 TP4 启动，实际通信域为 4 个 GPU rank。两种模式合计完成 85/118 个终态测试点，完成率 72.03%；其中 83 个 PASS、1 个 TIMEOUT_PASS、1 个真实 OOM。因机器回收，Balanced 的部分长上下文点未执行。
|
||||||
|
|
||||||
|
- **交互式长输入长输出优先 Low-Latency**：`16K -> 512` 的全部并发点上，Low-Latency 的 Input/Output TPS 和 TTFT P95 均优于 Balanced。C=8 时吞吐高 100.2%，TTFT P95 低 41.2%；C=256 时吞吐仍高 5.3%，TTFT P95 低 23.0%。
|
||||||
|
- **Balanced 的优势集中在高并发短输出**：`1K -> 128` 从 C=8 开始反超，C=256 达到 36,265 input tok/s，比 Low-Latency 高 21.2%，TPOT P95 也低 39.8%。
|
||||||
|
- **长输出仍明显偏向 Low-Latency**：`1K -> 4K, C=256` 的 Output TPS 为 21,132，是 Balanced 9,764 的 2.16 倍。
|
||||||
|
- **64K Prefill 吞吐 Balanced 略高**：`64K -> 1` 在 C=8 至 C=128 约为 65K input tok/s，比 Low-Latency 的约 59K 高约 11%。
|
||||||
|
- **已确认的容量边界**：Low-Latency 可完成 `256K -> 1, C=4`，但 `512K -> 1, C=1` 在申请约 2.25-3.00 GiB 连续显存时 OOM。Balanced 仅测试到 `64K -> 1, C=128`，未触及其容量边界。
|
||||||
|
|
||||||
|
## 2. 测试环境与服务配置
|
||||||
|
|
||||||
|
| 项目 | Low-Latency | Balanced |
|
||||||
|
|---|---|---|
|
||||||
|
| 硬件 | 单机 8 x NVIDIA B300；服务使用 4 GPU ranks | 同左 |
|
||||||
|
| 模型 | DeepSeek-V4-Flash-0731 | 同左 |
|
||||||
|
| 镜像 | `registry.sy.com/lmsysorg/sglang:v0.5.18-cu130-dev4` | 同左 |
|
||||||
|
| SGLang commit | `71de97b264b04dcd514cf904003028aefe9775c8` | 同左 |
|
||||||
|
| FlashInfer | 0.6.17 | 0.6.17 |
|
||||||
|
| 并行 | TP4 / DP1 / EP1 | TP4 / DP4 / EP4 + DP Attention |
|
||||||
|
| MoE | `flashinfer_mxfp4`，无 A2A | runner `auto` + DeepEP A2A |
|
||||||
|
| 投机解码 | DSpark，1 step，6 draft tokens | 关闭 |
|
||||||
|
| KV Cache | FP8 E4M3 | FP8 E4M3 |
|
||||||
|
| `mem-fraction-static` | 0.903 | 0.862 |
|
||||||
|
| Chunk Prefill | 16K | 4K |
|
||||||
|
| `max-prefill-tokens` | 16K | 16K |
|
||||||
|
| 活跃请求 | 全局 256 | 全局 256；每个 DP worker 64 |
|
||||||
|
|
||||||
|
两套配置同时改变了 DP Attention、EP/DeepEP、MoE runner、Chunk Prefill 和投机解码，因此结果用于比较完整部署方案，不用于把差异归因到某一个参数。
|
||||||
|
|
||||||
|
## 3. 主场景：16K 输入、512 输出
|
||||||
|
|
||||||
|
| 并发 | 模式 | Input TPS | Output TPS | TTFT P95 | TPOT P95 |
|
||||||
|
|---:|---|---:|---:|---:|---:|
|
||||||
|
| 1 | Low-Latency | 13,129 | 410 | 0.27 s | 3.07 ms |
|
||||||
|
| 1 | Balanced | 2,518 | 79 | 1.07 s | 11.26 ms |
|
||||||
|
| 8 | Low-Latency | 33,937 | 1,061 | 1.52 s | 10.23 ms |
|
||||||
|
| 8 | Balanced | 16,955 | 530 | 2.58 s | 12.69 ms |
|
||||||
|
| 32 | Low-Latency | 49,778 | 1,556 | 6.23 s | 31.87 ms |
|
||||||
|
| 32 | Balanced | 36,100 | 1,128 | 8.09 s | 25.54 ms |
|
||||||
|
| 64 | Low-Latency | 55,645 | 1,739 | 12.09 s | 57.79 ms |
|
||||||
|
| 64 | Balanced | 45,337 | 1,417 | 16.04 s | 41.92 ms |
|
||||||
|
| 128 | Low-Latency | 59,108 | 1,847 | 24.71 s | 114.28 ms |
|
||||||
|
| 128 | Balanced | 52,116 | 1,629 | 31.96 s | 73.50 ms |
|
||||||
|
| 256 | Low-Latency | **60,486** | **1,890** | **48.39 s** | 226.23 ms |
|
||||||
|
| 256 | Balanced | 57,436 | 1,795 | 62.85 s | **134.28 ms** |
|
||||||
|
|
||||||
|
Low-Latency 在该场景中始终提供更高的系统吞吐和更低的首 token 延迟。Balanced 从 C=32 起拥有更低的 TPOT P95，说明其 Decode 尾延迟更平稳，但不足以抵消 Prefill 和排队阶段的损失。两种模式在 C=128 之后吞吐增幅都已明显收窄，继续提高并发主要抬升 TTFT。
|
||||||
|
|
||||||
|
## 4. 短输入场景
|
||||||
|
|
||||||
|
### 4.1 `1K -> 128`
|
||||||
|
|
||||||
|
| 并发 | 模式 | Input TPS | Output TPS | TTFT P95 | TPOT P95 |
|
||||||
|
|---:|---|---:|---:|---:|---:|
|
||||||
|
| 1 | Low-Latency | 2,042 | 255 | 0.18 s | 3.53 ms |
|
||||||
|
| 1 | Balanced | 643 | 80 | 0.34 s | 10.62 ms |
|
||||||
|
| 8 | Low-Latency | 3,579 | 447 | 3.39 s | 31.29 ms |
|
||||||
|
| 8 | Balanced | 4,392 | 549 | 0.65 s | 10.73 ms |
|
||||||
|
| 32 | Low-Latency | 7,185 | 898 | 1.71 s | 49.89 ms |
|
||||||
|
| 32 | Balanced | 13,757 | 1,720 | 0.94 s | 12.48 ms |
|
||||||
|
| 128 | Low-Latency | 20,917 | 2,615 | 2.10 s | 68.93 ms |
|
||||||
|
| 128 | Balanced | 29,473 | 3,684 | 2.84 s | 27.68 ms |
|
||||||
|
| 256 | Low-Latency | 29,913 | 3,739 | 3.54 s | 99.14 ms |
|
||||||
|
| 256 | Balanced | **36,265** | **4,533** | 4.51 s | **59.65 ms** |
|
||||||
|
|
||||||
|
短输入下，C=1 仍是 Low-Latency 更快；从 C=8 起，Balanced 的 DP4 能被有效填充，吞吐和 TPOT 均明显占优。C=128/256 时 Balanced 的 TTFT P95 略高，但整体仍更适合短请求高并发吞吐。
|
||||||
|
|
||||||
|
### 4.2 `1K -> 4K`
|
||||||
|
|
||||||
|
| 并发 | 模式 | Output TPS | TTFT P95 | TPOT P95 |
|
||||||
|
|---:|---|---:|---:|---:|
|
||||||
|
| 1 | Low-Latency | 605 | 0.18 s | 2.28 ms |
|
||||||
|
| 1 | Balanced | 88 | 0.37 s | 14.15 ms |
|
||||||
|
| 8 | Low-Latency | 2,837 | 0.45 s | 3.52 ms |
|
||||||
|
| 8 | Balanced | 752 | 0.80 s | 10.62 ms |
|
||||||
|
| 32 | Low-Latency | 6,717 | 0.75 s | 6.34 ms |
|
||||||
|
| 32 | Balanced | 2,527 | 0.92 s | 12.59 ms |
|
||||||
|
| 128 | Low-Latency | 12,869 | 2.19 s | 14.14 ms |
|
||||||
|
| 128 | Balanced | 7,653 | 3.16 s | 16.57 ms |
|
||||||
|
| 256 | Low-Latency | **21,132** | **3.66 s** | **17.31 ms** |
|
||||||
|
| 256 | Balanced | 9,764 | 4.39 s | 26.51 ms |
|
||||||
|
|
||||||
|
长 Decode 充分体现了 Low-Latency 配方中 DSpark 和单副本执行路径的优势。Balanced 虽可利用 DP4，但本轮配置下没有投机解码，Output TPS 在全部并发点均落后。
|
||||||
|
|
||||||
|
## 5. 长上下文与容量
|
||||||
|
|
||||||
|
### 5.1 `64K -> 1`
|
||||||
|
|
||||||
|
| 并发 | Low-Latency Input TPS | Balanced Input TPS | Low-Latency TTFT P95 | Balanced TTFT P95 |
|
||||||
|
|---:|---:|---:|---:|---:|
|
||||||
|
| 1 | 57,080 | 17,118 | 1.18 s | 3.87 s |
|
||||||
|
| 4 | 58,914 | 61,303 | 4.44 s | 4.41 s |
|
||||||
|
| 8 | 58,819 | 65,564 | 8.88 s | 8.36 s |
|
||||||
|
| 16 | 58,871 | **65,865** | 17.76 s | **15.82 s** |
|
||||||
|
| 32 | 58,822 | 65,505 | 35.52 s | 32.07 s |
|
||||||
|
| 64 | 58,852 | 65,617 | 71.23 s | 64.08 s |
|
||||||
|
| 128 | 58,702 | 65,401 | 142.74 s | 128.67 s |
|
||||||
|
|
||||||
|
Balanced 在 C>=4 后进入约 65K input tok/s 平台，较 Low-Latency 高约 11%，表明 DP Attention + EP4/DeepEP 的 Prefill 吞吐在足够 batch 下有效。但绝对 TTFT 仍随并发近似线性增长，不能只依据 Input TPS 判断在线体验。
|
||||||
|
|
||||||
|
### 5.2 Low-Latency 已测边界
|
||||||
|
|
||||||
|
| 场景 | 并发 | Input TPS | TTFT P95 | 状态 |
|
||||||
|
|---|---:|---:|---:|---|
|
||||||
|
| 128K -> 1 | 1 | 50,299 | 2.61 s | PASS |
|
||||||
|
| 128K -> 1 | 4 | 51,297 | 10.31 s | PASS |
|
||||||
|
| 128K -> 1 | 64 | 51,257 | 163.69 s | PASS |
|
||||||
|
| 256K -> 1 | 1 | 39,876 | 6.61 s | PASS |
|
||||||
|
| 256K -> 1 | 4 | 40,372 | 26.03 s | PASS |
|
||||||
|
| 512K -> 1 | 1 | - | - | OOM |
|
||||||
|
|
||||||
|
`512K -> 1, C=1` 的服务日志记录了多 rank 显存分配失败，属于真实容量问题。Balanced 因机器回收只完成到 `64K -> 1`，不能据此比较 128K 以上容量。
|
||||||
|
|
||||||
|
## 6. 完成范围
|
||||||
|
|
||||||
|
| 模式 | 终态/计划 | PASS | TIMEOUT_PASS | FAIL | 未执行 |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| Low-Latency | 54/59 | 53 | 0 | 1 | 5 |
|
||||||
|
| Balanced | 31/59 | 30 | 1 | 0 | 28 |
|
||||||
|
| 合计 | 85/118 | 83 | 1 | 1 | 33 |
|
||||||
|
|
||||||
|
Balanced 的 `1K -> 4K, C=1` 超过 30 分钟限制后以 TIMEOUT_PASS 保留了完整指标。其余 33 个点没有原始性能数据，不能插值或推断为成功；主要缺口是 Balanced 的 `64K -> 512` 以及 128K 以上场景。
|
||||||
|
|
||||||
|
## 7. 部署建议
|
||||||
|
|
||||||
|
1. 以长 Prompt、长输出和交互体验为主时，优先 Low-Latency 配方。
|
||||||
|
2. 以短请求、高并发吞吐为主时，Balanced 在 C>=8 后更有优势。
|
||||||
|
3. 纯 Prefill 压力下，Balanced 在 batch 充足时可获得约 11% 的 64K 输入吞吐提升。
|
||||||
|
4. 线上并发上限不能只按峰值 TPS 设置。`16K -> 512` 从 C=128 增加到 C=256 的吞吐收益很小，但 TTFT P95 接近翻倍。
|
||||||
|
5. FP8 KV 日志提示未提供缩放因子、默认使用 1.0。正式质量验收应补充精度测试，性能结果本身不代表生成质量已验证。
|
||||||
|
|
||||||
|
## 8. 原始证据
|
||||||
|
|
||||||
|
服务器原始目录：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/data/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427/
|
||||||
|
```
|
||||||
|
|
||||||
|
关键目录与文件：
|
||||||
|
|
||||||
|
```text
|
||||||
|
dsv4-low-latency/
|
||||||
|
dsv4-balanced/
|
||||||
|
final-summary/summary.csv
|
||||||
|
final-summary/summary.md
|
||||||
|
final-summary/comparison.md
|
||||||
|
final-summary/progress.json
|
||||||
|
```
|
||||||
|
|
||||||
|
每个点保留 benchmark JSON、终端日志、服务尾日志、GPU 状态和 `.done/.failed` 标记。完整服务命令位于对应模式的 `server_command.txt`。
|
||||||
@ -0,0 +1,176 @@
|
|||||||
|
# GLM-5.3 在 B300 上的全量文本性能阶段报告
|
||||||
|
|
||||||
|
## 1. 结论摘要
|
||||||
|
|
||||||
|
本轮在单台 8 卡 B300 上完成 GLM-5.3 的 Low-Latency 与 High-Throughput 两套官方风格配置测试。两套配置代表完整部署方案，不是只改变一个参数的 A/B：Low-Latency 使用 TP8；High-Throughput 使用 DP8、DP Attention 与 DeepEP。
|
||||||
|
|
||||||
|
- **低并发优先 Low-Latency**：`16K -> 512, C=1` 达到 7,882 input tok/s，TTFT P95 为 0.61 s，明显优于 High-Throughput 的 1,754 tok/s 和 1.71 s。
|
||||||
|
- **高并发优先 High-Throughput**：`16K -> 512, C=128` 达到 31,889 input tok/s，比 Low-Latency 的 23,920 tok/s 高 **33.3%**；TTFT P95 也由 63.41 s 降至 30.89 s。
|
||||||
|
- **High-Throughput 的甜点位约为 C=128**：继续升到 C=256 后，input TPS 降至 25,722（下降 19.3%），TPOT P95 升至 435 ms。
|
||||||
|
- **长上下文边界可运行**：两种模式都完成了接近 1M 输入、OSL=1、C=1/2/4 的测试，但请求延迟达到分钟级，只能说明容量边界可达，不代表交互性能可接受。
|
||||||
|
- 当前结果足以判断两种模式的吞吐/延迟取舍；17 个被误标的 OSL=1 结果已从原始 JSON 恢复，未实际运行的点仍将在队列末尾补跑。
|
||||||
|
|
||||||
|
## 2. 测试环境与配置
|
||||||
|
|
||||||
|
| 项目 | Low-Latency | High-Throughput |
|
||||||
|
|---|---|---|
|
||||||
|
| 硬件 | 单机 8 x NVIDIA B300 | 单机 8 x NVIDIA B300 |
|
||||||
|
| 模型 | GLM-5.3 | GLM-5.3 |
|
||||||
|
| 镜像 | `registry.sy.com/lmsysorg/sglang:v0.5.18-cu130-dev4` | 同左 |
|
||||||
|
| 并行 | TP8 | TP8 + DP8 + DP Attention |
|
||||||
|
| MoE 通信 | 默认路径 | DeepEP A2A |
|
||||||
|
| 投机解码 | EAGLE，5 steps，6 draft tokens | EAGLE，1 step，2 draft tokens |
|
||||||
|
| `mem-fraction-static` | 0.80 | 0.85 |
|
||||||
|
| Chunk Prefill | 默认 | 32K |
|
||||||
|
| 最大活跃请求 | 默认 | 256 |
|
||||||
|
|
||||||
|
因此，下文比较回答的是“官方 Low-Latency 配方与 High-Throughput 配方谁更适合该负载”，不能把差异单独归因于 DP Attention、DeepEP 或投机参数中的某一项。
|
||||||
|
|
||||||
|
## 3. 主场景：16K 输入、512 输出
|
||||||
|
|
||||||
|
| 并发 | 模式 | Input TPS | Output TPS | TTFT P95 | TPOT P95 |
|
||||||
|
|---:|---|---:|---:|---:|---:|
|
||||||
|
| 1 | Low-Latency | 7,882 | 246 | 0.61 s | 4.36 ms |
|
||||||
|
| 1 | High-Throughput | 1,754 | 55 | 1.71 s | 16.40 ms |
|
||||||
|
| 8 | Low-Latency | 16,384 | 512 | 3.89 s | 22.46 ms |
|
||||||
|
| 8 | High-Throughput | 7,602 | 238 | 2.95 s | 38.86 ms |
|
||||||
|
| 32 | Low-Latency | 21,303 | 666 | 15.13 s | 69.03 ms |
|
||||||
|
| 32 | High-Throughput | 15,478 | 484 | 10.71 s | 82.99 ms |
|
||||||
|
| 64 | Low-Latency | 23,503 | 734 | 29.49 s | 105.62 ms |
|
||||||
|
| 64 | High-Throughput | 22,864 | 715 | 17.18 s | 115.88 ms |
|
||||||
|
| 128 | Low-Latency | 23,920 | 748 | 63.41 s | 101.14 ms |
|
||||||
|
| 128 | High-Throughput | **31,889** | **997** | **30.89 s** | 165.33 ms |
|
||||||
|
| 256 | Low-Latency | 24,021 | 751 | 144.38 s | 102.36 ms |
|
||||||
|
| 256 | High-Throughput | 25,722 | 804 | 58.21 s | 435.35 ms |
|
||||||
|
|
||||||
|
趋势很清楚：Low-Latency 很快达到较高单请求效率，但约在 C=64 后进入 24K input tok/s 平台；High-Throughput 需要足够请求填满 8 个 DP rank，在 C=128 达到峰值。C=256 对两者都没有实际价值：Low-Latency 吞吐几乎不再增长，High-Throughput 已发生吞吐回退。
|
||||||
|
|
||||||
|
## 4. 短输入与长输出
|
||||||
|
|
||||||
|
### 4.1 `1K -> 128`
|
||||||
|
|
||||||
|
| 并发 | 模式 | Input TPS | Output TPS | TTFT P95 | TPOT P95 |
|
||||||
|
|---:|---|---:|---:|---:|---:|
|
||||||
|
| 1 | Low-Latency | 2,006 | 251 | 70 ms | 5.55 ms |
|
||||||
|
| 1 | High-Throughput | 441 | 55 | 354 ms | 17.25 ms |
|
||||||
|
| 32 | Low-Latency | 11,946 | 1,493 | 1.20 s | 31.01 ms |
|
||||||
|
| 32 | High-Throughput | 5,513 | 689 | 1.35 s | 55.76 ms |
|
||||||
|
| 128 | Low-Latency | 14,125 | 1,766 | 6.33 s | 42.03 ms |
|
||||||
|
| 128 | High-Throughput | 15,193 | 1,899 | 2.57 s | 87.74 ms |
|
||||||
|
| 256 | Low-Latency | 13,944 | 1,743 | 15.82 s | 43.56 ms |
|
||||||
|
| 256 | High-Throughput | 14,522 | 1,815 | 4.59 s | 192.57 ms |
|
||||||
|
|
||||||
|
短请求下 Low-Latency 的单请求和中低并发效率显著更高。High-Throughput 到 C=128 才在总吞吐上略微反超，但 TPOT 更差，适合批处理而非交互请求。
|
||||||
|
|
||||||
|
### 4.2 `1K -> 4K`
|
||||||
|
|
||||||
|
| 并发 | 模式 | Output TPS | TTFT P95 | TPOT P95 |
|
||||||
|
|---:|---|---:|---:|---:|
|
||||||
|
| 1 | Low-Latency | 436 | 71 ms | 3.32 ms |
|
||||||
|
| 1 | High-Throughput | 61 | 723 ms | 21.05 ms |
|
||||||
|
| 32 | Low-Latency | 4,767 | 0.89 s | 9.33 ms |
|
||||||
|
| 32 | High-Throughput | 1,834 | 0.96 s | 18.18 ms |
|
||||||
|
| 128 | Low-Latency | 6,123 | 56.05 s | 11.08 ms |
|
||||||
|
| 128 | High-Throughput | 5,687 | 2.49 s | 23.05 ms |
|
||||||
|
| 256 | Low-Latency | 6,105 | 140.39 s | 11.26 ms |
|
||||||
|
| 256 | High-Throughput | **10,613** | 4.54 s | 22.36 ms |
|
||||||
|
|
||||||
|
长输出进一步放大两套配置的差异：Low-Latency 在 C<=128 有更低 TPOT；High-Throughput 在 C=256 将 8 个 DP rank 填满后，Output TPS 比 Low-Latency 高 73.8%，但单 token 延迟仍约为后者两倍。
|
||||||
|
|
||||||
|
## 5. 长上下文观察
|
||||||
|
|
||||||
|
### 5.1 `64K/128K -> 512`
|
||||||
|
|
||||||
|
| 场景 | 并发 | 模式 | Input TPS | TTFT P95 | TPOT P95 |
|
||||||
|
|---|---:|---|---:|---:|---:|
|
||||||
|
| 64K -> 512 | 8 | Low-Latency | 22,427 | 13.07 s | 61.58 ms |
|
||||||
|
| 64K -> 512 | 8 | High-Throughput | 13,287 | 9.93 s | 96.89 ms |
|
||||||
|
| 64K -> 512 | 32 | Low-Latency | 23,917 | 62.69 s | 243.87 ms |
|
||||||
|
| 64K -> 512 | 32 | High-Throughput | 25,109 | 29.46 s | 206.58 ms |
|
||||||
|
| 64K -> 512 | 64 | Low-Latency | 24,220 | 128.26 s | 323.81 ms |
|
||||||
|
| 64K -> 512 | 64 | High-Throughput | **33,438** | **57.34 s** | 329.77 ms |
|
||||||
|
| 128K -> 512 | 8 | Low-Latency | 21,132 | 28.73 s | 145.04 ms |
|
||||||
|
| 128K -> 512 | 8 | High-Throughput | 13,503 | 20.22 s | 171.97 ms |
|
||||||
|
| 128K -> 512 | 32 | Low-Latency | 21,788 | 142.63 s | 339.34 ms |
|
||||||
|
| 128K -> 512 | 32 | High-Throughput | **27,504** | **59.91 s** | 401.75 ms |
|
||||||
|
|
||||||
|
长输入下，Low-Latency 的 input TPS 较早饱和；High-Throughput 在高并发下仍能利用 DP 扩展，并明显缩短排队造成的 TTFT 长尾。不过 64K/128K 高并发的绝对 TTFT 已达到几十至上百秒，生产侧仍需依靠准入控制或请求分层。
|
||||||
|
|
||||||
|
### 5.2 上下文边界
|
||||||
|
|
||||||
|
| 输入长度 | 模式 | 已完成并发 | C=1 Input TPS | 最高并发 TTFT P95 |
|
||||||
|
|---|---|---:|---:|---:|
|
||||||
|
| 256K | Low-Latency | 1/2/4 | 17,457 | 59.69 s（C=4） |
|
||||||
|
| 256K | High-Throughput | 1 | 9,205 | 29.44 s（C=1） |
|
||||||
|
| 512K | Low-Latency | 1/2/4 | 12,421 | 168.09 s（C=4） |
|
||||||
|
| 512K | High-Throughput | 1/2/4 | 7,533 | 86.78 s（C=4） |
|
||||||
|
| 约 1M | Low-Latency | 1/2/4 | 7,897 | 509.51 s（C=4） |
|
||||||
|
| 约 1M | High-Throughput | 1/2/4 | 5,330 | 225.40 s（C=4） |
|
||||||
|
|
||||||
|
这些点输出仅 1 token，主要验证容量和 Prefill，不应拿 Output TPS 或 TPOT 做模式比较。
|
||||||
|
|
||||||
|
## 6. 显存状态
|
||||||
|
|
||||||
|
- Low-Latency：每卡约使用 258.7-267.0 GiB，最少剩余约 7.1 GiB。
|
||||||
|
- High-Throughput：每卡约使用 252.1-268.9 GiB，最少剩余约 5.3 GiB。
|
||||||
|
- High-Throughput 的卡间显存分布更不均匀，且最紧张卡余量更低；继续扩大上下文或并发时应重点监控最满的 rank，而不是只看全机平均值。
|
||||||
|
|
||||||
|
## 7. 关于首轮“失败”
|
||||||
|
|
||||||
|
首轮统计曾显示 `81 PASS + 2 TIMEOUT_PASS + 17 FAIL`。审计 17 个失败标记后确认：
|
||||||
|
|
||||||
|
- 17 个点全部是 `OSL=1`；
|
||||||
|
- benchmark 返回码全部为 `rc=0`，请求均已完成并生成有效 JSON；
|
||||||
|
- OSL=1 只有首 token，没有相邻输出 token，因而 TPOT 按定义为 0；
|
||||||
|
- 旧校验逻辑错误地要求 TPOT 必须大于 0，产生了假失败；
|
||||||
|
- 假失败又触发了容量短路，导致部分后续高并发点未执行。
|
||||||
|
|
||||||
|
校验已改为：仅当 OSL=1 时接受 TPOT=0；容量短路只由明确的 OOM/显存错误触发。因此，这 17 个标记不能解读为 GLM-5.3 服务故障或容量不足。
|
||||||
|
|
||||||
|
17 个原始 JSON 已全部恢复如下；TPOT=0 是单输出 token 的正确数学结果：
|
||||||
|
|
||||||
|
| 模式 | 测试点 | 完成请求 | Input TPS | TTFT P95 |
|
||||||
|
|---|---|---:|---:|---:|
|
||||||
|
| High-Throughput | 16K -> 1, C=1 | 64 | 10,266 | 1.64 s |
|
||||||
|
| High-Throughput | 64K -> 1, C=16 | 80 | 56,023 | 19.48 s |
|
||||||
|
| High-Throughput | 64K -> 1, C=32 | 160 | 56,600 | 36.91 s |
|
||||||
|
| High-Throughput | 64K -> 1, C=64 | 320 | 56,555 | 74.26 s |
|
||||||
|
| High-Throughput | 64K -> 1, C=128 | 640 | 56,357 | 153.39 s |
|
||||||
|
| High-Throughput | 128K -> 1, C=1 | 64 | 10,101 | 13.35 s |
|
||||||
|
| High-Throughput | 256K -> 1, C=1 | 16 | 9,205 | 29.44 s |
|
||||||
|
| Low-Latency | 16K -> 1, C=1 | 64 | 26,687 | 0.61 s |
|
||||||
|
| Low-Latency | 64K -> 1, C=4 | 64 | 25,330 | 10.32 s |
|
||||||
|
| Low-Latency | 64K -> 1, C=16 | 80 | 25,368 | 41.26 s |
|
||||||
|
| Low-Latency | 64K -> 1, C=32 | 160 | 25,377 | 82.51 s |
|
||||||
|
| Low-Latency | 64K -> 1, C=64 | 320 | 25,399 | 164.95 s |
|
||||||
|
| Low-Latency | 64K -> 1, C=128 | 640 | 25,412 | 329.93 s |
|
||||||
|
| Low-Latency | 128K -> 1, C=8 | 64 | 22,556 | 46.45 s |
|
||||||
|
| Low-Latency | 128K -> 1, C=16 | 80 | 22,548 | 92.88 s |
|
||||||
|
| Low-Latency | 128K -> 1, C=32 | 160 | 22,550 | 185.81 s |
|
||||||
|
| Low-Latency | 256K -> 1, C=4 | 16 | 17,585 | 59.69 s |
|
||||||
|
|
||||||
|
这里的“恢复”是重新解释保留下来的原始 JSON，没有重算或猜测指标。错误短路造成的未运行点没有数据可恢复，必须实际补跑；补跑会在 DeepSeek-V4 两个模式结束后自动进行。
|
||||||
|
|
||||||
|
## 8. 建议
|
||||||
|
|
||||||
|
1. 在线交互或低并发服务采用 Low-Latency，并对高并发设置排队上限。
|
||||||
|
2. 离线批处理采用 High-Throughput，当前推荐运行区间为 C=64-128；不要直接采用 C=256。
|
||||||
|
3. 若要解释收益来源，下一步需对 DP Attention、DeepEP 和投机参数做单变量消融；本轮完整配方比较不能完成单项归因。
|
||||||
|
4. 生产容量应同时设吞吐和延迟 SLO。只追求峰值 TPS 会掩盖 C=128/256 时已经很高的 TTFT 与 TPOT。
|
||||||
|
|
||||||
|
## 9. 原始结果
|
||||||
|
|
||||||
|
服务器原始结果目录：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/data/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427/
|
||||||
|
```
|
||||||
|
|
||||||
|
关键子目录：
|
||||||
|
|
||||||
|
```text
|
||||||
|
glm53-low-latency/
|
||||||
|
glm53-high-throughput/
|
||||||
|
```
|
||||||
|
|
||||||
|
每个测试点包含 benchmark `.json`、终端 `.log`、服务尾日志和状态标记；两套服务的完整启动命令分别保存在各自目录的 `server_command.txt`。
|
||||||
@ -0,0 +1,65 @@
|
|||||||
|
# B300 DeepSeek-V4-Flash / GLM-5.3 SGLang text matrix
|
||||||
|
|
||||||
|
This directory archives the test code and all files written by the B300
|
||||||
|
text-matrix run through its manual cutoff and cleanup.
|
||||||
|
|
||||||
|
## Snapshot
|
||||||
|
|
||||||
|
- Snapshot time: `2026-09-10 02:43:55 UTC` (`2026-09-10 10:43:55 Asia/Shanghai`)
|
||||||
|
- Run ID: `b300-dsv4-glm53-dev4-20260909-084427`
|
||||||
|
- B300 code path: `/data/b300-deepseek-v4-glm53`
|
||||||
|
- B300 result path: `/data/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427`
|
||||||
|
- Main log: `/data/b300-dsv4-glm53-dev4-20260909-084427.log`
|
||||||
|
- Follow-up log: `/data/b300-dsv4-glm53-dev4-20260909-084427-followup.log`
|
||||||
|
- Source archive SHA-256: `8676f28c6b9df7676ad93d5a4a71777bb504539b0ea6e3b129ccec69f5a563fd`
|
||||||
|
|
||||||
|
Incremental synchronization:
|
||||||
|
|
||||||
|
- Delta time: `2026-09-10 07:23:10 UTC` (`2026-09-10 15:23:10 Asia/Shanghai`)
|
||||||
|
- Delta archive SHA-256: `c1bc269866152024afbdefbb15265832a773bfbf07ea2375e318dc252b25bdbb`
|
||||||
|
- At this point DeepSeek-V4-Flash had produced 53 Low-Latency and 18
|
||||||
|
Balanced point JSON files.
|
||||||
|
|
||||||
|
Final synchronization:
|
||||||
|
|
||||||
|
- Stop time: `2026-09-10 07:46:07 UTC` (`2026-09-10 15:46:07 Asia/Shanghai`)
|
||||||
|
- Final delta archive SHA-256: `f6ba8dbce5db850d43e7583a40eb8fc7cc95aab91c08fc5352fa69ec1d7d1117`
|
||||||
|
- Final status: 185/236 points reached a terminal state (78.39%): 164 pass,
|
||||||
|
3 timeout-pass, 18 fail markers, and 51 pending.
|
||||||
|
- Mode progress: DSV4 Low-Latency 54/59, DSV4 Balanced 31/59,
|
||||||
|
GLM-5.3 Low-Latency 54/59, and GLM-5.3 High-Throughput 46/59.
|
||||||
|
- Both benchmark tmux sessions were stopped, the test container was removed,
|
||||||
|
and `nvidia-smi` reported no remaining compute process.
|
||||||
|
|
||||||
|
The 17 GLM-5.3 fail markers on OSL=1 are validation false negatives: the
|
||||||
|
benchmark command returned zero and produced complete JSON, while TPOT is
|
||||||
|
mathematically zero because a one-token response has no inter-token interval.
|
||||||
|
The DSV4 Low-Latency `512k_1_c1` failure is a real GPU OOM. Raw status markers
|
||||||
|
are preserved exactly as emitted; use `final-summary/` together with the point
|
||||||
|
JSON and logs when interpreting them.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
- `GLM-5.3_B300全量文本性能阶段报告.md`: interim GLM-5.3 analysis.
|
||||||
|
- `snapshot-20260910-024355/b300-deepseek-v4-glm53/`: runner,
|
||||||
|
summarizer, model patch, and plan copied from the B300 host.
|
||||||
|
- `snapshot-20260910-024355/b300-dsv4-glm53-text-matrix/`: raw run
|
||||||
|
outputs, point logs, service logs, commands, status markers, and summaries.
|
||||||
|
- `snapshot-20260910-024355/*.log`: top-level orchestration logs.
|
||||||
|
- `snapshot-20260910-024355/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427/final-summary/`:
|
||||||
|
final `summary.md`, `summary.csv`, `comparison.md`, and `progress.json`.
|
||||||
|
|
||||||
|
The repository normally excludes raw benchmark logs. They are intentionally
|
||||||
|
included here because this snapshot is an explicit full-run archive.
|
||||||
|
|
||||||
|
## Compressed JSON
|
||||||
|
|
||||||
|
Raw benchmark JSON files larger than 5 MiB are stored losslessly as
|
||||||
|
`*.json.gz` to keep every individual Git object below common GitLab limits.
|
||||||
|
Restore one file with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
gzip -dk path/to/point.json.gz
|
||||||
|
```
|
||||||
|
|
||||||
|
Small JSON files remain uncompressed and can be inspected directly.
|
||||||
@ -0,0 +1,125 @@
|
|||||||
|
# B300 DeepSeek-V4-Flash / GLM-5.3 全量性能测试计划
|
||||||
|
|
||||||
|
## 1. 目标
|
||||||
|
|
||||||
|
在单台 8x NVIDIA B300 上，按 SGLang 官方 cookbook 的 B300 配置部署以下本地 checkpoint，并复用 Kimi-K3 文本全量矩阵测量延迟、吞吐、容量边界与 1M 上下文可用性。
|
||||||
|
|
||||||
|
| 测试对象 | B300 本地路径 | checkpoint 事实 |
|
||||||
|
|---|---|---|
|
||||||
|
| DeepSeek-V4-Flash-0731 | `/data/model/deepseek-ai--DeepSeek-V4-Flash-0731` | 约 156 GiB；`deepseek_v4`；FP4 experts + FP8/UE8M0 元数据；1M context；内置 DSpark draft head |
|
||||||
|
| GLM-5.3 | `/data/model/ZhipuAI--GLM-5.3` | 约 704 GiB；`glm_moe_dsa`；FP8；78 层；256 experts、top-8；1M context；内置 MTP layer |
|
||||||
|
|
||||||
|
本轮只做单实例文本 serving，不测试多模态、PD 分离、HiCache、DFlash2 或自定义优化。
|
||||||
|
|
||||||
|
## 2. 镜像门槛
|
||||||
|
|
||||||
|
DeepSeek-V4-Flash 与 GLM-5.3 均可由现有 SGLang v0.5.18 镜像识别和加载。镜像选择的门槛不是版本号新旧，而是目标 checkpoint、B300 kernel 路径及必要 hotfix 是否通过代表点验证。正式启动前必须记录：
|
||||||
|
|
||||||
|
- 镜像完整 tag、image ID 和 RepoDigest；
|
||||||
|
- SGLang commit/version；
|
||||||
|
- PyTorch、CUDA、FlashInfer、DeepGEMM、DeepEP 版本；
|
||||||
|
- `sglang serve --help` 是否包含本计划使用的参数。
|
||||||
|
|
||||||
|
当前基线镜像 `registry.sy.com/lmsysorg/sglang:latest-cu130` 实测为 SGLang v0.5.18，源码已包含 `DeepseekV4ForCausalLM` 与 `GlmMoeDsaForCausalLM`。Harbor 新增的 `v0.5.18-cu130-dev4` 使用相同 SGLang commit，并叠加 FlashInfer、Mooncake 与 GLM gate hotfix；它不是 GLM-5.3 的首次支持版本。正式结果固定使用验证通过且记录了 digest 的镜像，不用 tag 名替代真实版本证据。
|
||||||
|
|
||||||
|
`lmsysorg/sglang:v0.5.19` 已于 2026-09-10 导入 B300，Image ID 为 `sha256:e6238090791a938ab86dd21a9a6394192dad15237e815df557cf83524d54b813`。离线源码审计确认它同时包含两个模型类和 `SWATokenToKVPoolAllocator.clear_full_to_swa_mapping()`，组件版本为 SGLang 0.5.19、FlashInfer 0.6.18、DeepGEMM 0.1.7、DeepEP 0.1.2。为了不把不同镜像混入同一组长矩阵，进行中的 `dev4` Run 保持不变；矩阵结束后再用 v0.5.19 做独立模型加载与代表点 smoke。
|
||||||
|
|
||||||
|
## 3. 官方 B300 配置
|
||||||
|
|
||||||
|
### 3.1 DeepSeek-V4-Flash-0731
|
||||||
|
|
||||||
|
| Recipe | 关键参数 | 目的 |
|
||||||
|
|---|---|---|
|
||||||
|
| Low Latency | `TP4`、`flashinfer_mxfp4`、`DSPARK`、`swa-full-tokens-ratio=0.1` | 官方已验证的低并发交互配置 |
|
||||||
|
| Balanced | `TP4/DP4`、DP Attention、DeepEP、每 rank dispatch 上限 1024、dispatch/combine 各 96 SM | 官方给出的多请求候选配置；0731 对应单元目前未标记 verified，需由本次 B300 实测确认 |
|
||||||
|
|
||||||
|
Balanced 保持 cookbook 行为，不额外叠加 DSpark。两个 recipe 都使用 `deepseek-v4` reasoning parser 和 `deepseekv4` tool parser。
|
||||||
|
|
||||||
|
### 3.2 GLM-5.3 FP8
|
||||||
|
|
||||||
|
| Recipe | 关键参数 | 目的 |
|
||||||
|
|---|---|---|
|
||||||
|
| Low Latency | `TP8`、EAGLE `5/1/6`、`mem-fraction-static=0.8` | 官方已验证的单用户低延迟配置 |
|
||||||
|
| High Throughput | `TP8/DP8`、DP Attention、DeepEP、EAGLE `1/1/2`、chunk 32K、`max-running-requests=256`、显存比例 0.85 | 官方已验证的持续批处理配置 |
|
||||||
|
|
||||||
|
GLM-5.3 在 Blackwell 上由框架自动选择 FP8 KV cache 与 TRT-LLM DSA 配对，本轮不手工覆盖；parser 使用 `glm45` / `glm47`。
|
||||||
|
|
||||||
|
SGLang 当前 cookbook 的 GLM-5.3 FP8 B300 部署表提供 Low Latency 与 High Throughput 两个已验证点，因此不人为构造第三个 Balanced 配置。
|
||||||
|
|
||||||
|
## 4. 全量文本矩阵
|
||||||
|
|
||||||
|
每个 recipe 完整执行以下矩阵，和现有 Kimi-K3 全量脚本保持相同的输入、输出与并发点：
|
||||||
|
|
||||||
|
| Shape | ISL | OSL | 并发 C |
|
||||||
|
|---|---:|---:|---|
|
||||||
|
| `16k_512` | 16,384 | 512 | 1 / 8 / 32 / 64 / 128 / 256 |
|
||||||
|
| `16k_1` | 16,384 | 1 | 1 / 8 / 32 / 64 / 128 / 256 |
|
||||||
|
| `1k_128` | 1,024 | 128 | 1 / 8 / 32 / 64 / 128 / 256 |
|
||||||
|
| `1k_4k` | 1,024 | 4,096 | 1 / 8 / 32 / 64 / 128 / 256 |
|
||||||
|
| `64k_1` | 65,536 | 1 | 1 / 4 / 8 / 16 / 32 / 64 / 128 |
|
||||||
|
| `64k_512` | 65,536 | 512 | 1 / 4 / 8 / 16 / 32 / 64 |
|
||||||
|
| `128k_1` | 131,072 | 1 | 1 / 2 / 4 / 8 / 16 / 32 / 64 |
|
||||||
|
| `128k_512` | 131,072 | 512 | 1 / 2 / 4 / 8 / 16 / 32 |
|
||||||
|
| `256k_1` | 262,144 | 1 | 1 / 2 / 4 |
|
||||||
|
| `512k_1` | 524,288 | 1 | 1 / 2 / 4 |
|
||||||
|
| `1m_boundary_1` | 1,047,552 | 1 | 1 / 2 / 4 |
|
||||||
|
|
||||||
|
共 59 points/recipe，两个模型各 2 个 recipe，理论总计 236 points。
|
||||||
|
|
||||||
|
## 5. 测试口径
|
||||||
|
|
||||||
|
- `sglang.bench_serving`，dataset=`random-ids`，固定 seed=`20260904`；
|
||||||
|
- `random-range-ratio=1.0`，保证输入长度固定；输出应达到指定 OSL；
|
||||||
|
- 单点前执行 `--flush-cache`，避免前缀缓存污染跨点比较；
|
||||||
|
- 短/中上下文请求数为 `max(64, 5*C)`，更长上下文按现有 Kimi 脚本递减；
|
||||||
|
- 单点墙钟时间超过 1,800 秒记为 `TIMEOUT_PASS`，终止该点后继续矩阵；
|
||||||
|
- 保存 `output-details` 原始 JSON、console log、该点 server tail、GPU 状态；
|
||||||
|
- 服务启动失败、EngineDead、CUDA OOM、输出长度不足或请求错误都不得记为 PASS；
|
||||||
|
- 每个 recipe 独立启动服务并进行一次小 warm-up，测量点本身仍清空 prefix cache。
|
||||||
|
|
||||||
|
## 6. 执行顺序
|
||||||
|
|
||||||
|
1. 镜像、模型完整性、GPU 空闲和磁盘空间审计。
|
||||||
|
2. DeepSeek-V4-Flash Low Latency，优先获得可用基线。
|
||||||
|
3. DeepSeek-V4-Flash Balanced。
|
||||||
|
4. GLM-5.3 Low Latency。
|
||||||
|
5. GLM-5.3 High Throughput。
|
||||||
|
6. 生成按模型/recipe/shape/C 展开的汇总 CSV 与 Markdown。
|
||||||
|
7. 归档镜像 digest、server command、server info、启动/退出日志和失败证据。
|
||||||
|
8. 停止测试容器并确认 8 张 GPU 无残留进程。
|
||||||
|
|
||||||
|
脚本支持 `.done` 断点续跑；网络或会话中断后复用同一个 `RUN_ID` 即可继续。
|
||||||
|
|
||||||
|
## 7. 验收输出
|
||||||
|
|
||||||
|
每个成功点至少汇总：
|
||||||
|
|
||||||
|
- Benchmark duration、Request/Input/Output/Total token throughput；
|
||||||
|
- TTFT mean/P50/P95/P99；
|
||||||
|
- TPOT mean/P50/P95/P99；
|
||||||
|
- E2E mean/P50/P95/P99；
|
||||||
|
- 实际完成请求数、输入/输出 token 数和错误数。
|
||||||
|
|
||||||
|
最终报告同时回答：
|
||||||
|
|
||||||
|
1. 两个模型在 B300 上各 recipe 的吞吐平台和延迟拐点；
|
||||||
|
2. 16K->512 主场景在 C=8/32/128/256 下的变化；
|
||||||
|
3. 长输入容量边界以及 1M 点是否能完成；
|
||||||
|
4. 低延迟 recipe 与批处理 recipe 的收益和代价；
|
||||||
|
5. 所有 timeout、OOM、服务退出和结果无效点的直接日志路径。
|
||||||
|
|
||||||
|
## 8. 依据
|
||||||
|
|
||||||
|
- [SGLang DeepSeek-V4 cookbook](https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4)
|
||||||
|
- [SGLang GLM-5.3 cookbook](https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3)
|
||||||
|
- [SGLang v0.5.19 release](https://github.com/sgl-project/sglang/releases/tag/v0.5.19)
|
||||||
|
|
||||||
|
## 9. 首轮 dev4 审计记录
|
||||||
|
|
||||||
|
Run ID：`b300-dsv4-glm53-dev4-20260909-084427`。
|
||||||
|
|
||||||
|
- DeepSeek-V4-Flash 的 Low Latency 与 Balanced 均完成模型加载、内存池分配和 CUDA Graph 捕获，但在 warm-up 释放 SWA KV 映射时退出。直接错误为 `SWATokenToKVPoolAllocator` 缺少 `clear_full_to_swa_mapping()`；`dev4` 引入了方法调用，却未同步上游已有的方法实现。
|
||||||
|
- 该错误不是 OOM、B300 不兼容或模型不受支持。修复采用上游同名方法的原实现，并通过只读文件覆盖显式启用，不修改 checkpoint，也不把临时补丁固化为默认行为。
|
||||||
|
- GLM-5.3 Low Latency 已由 `dev4` 正确识别并加载：主模型约 89.09 GiB/卡，EAGLE draft 约 1.77 GiB/卡；当前继续完成 warm-up 和代表点验证。
|
||||||
|
- 后续容器挂载宿主机 `/data/b300-sglang-cache` 到 `/root/.cache`，复用 DeepGEMM/FlashInfer JIT 产物，减少各 profile 重复启动时间。
|
||||||
|
- GLM-5.3 Low Latency 已完成全部已调度点；自动队列继续执行 High Throughput，随后补跑 DeepSeek-V4-Flash 两个 profile 和旧校验逻辑漏掉的 GLM 点。补跑仍使用同一 Run ID 和同一 `dev4` 镜像，避免跨版本拼接性能数据。
|
||||||
@ -0,0 +1 @@
|
|||||||
|
b300-dsv4-glm53-dev4-20260909-084427
|
||||||
@ -0,0 +1,466 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
: "${IMAGE:?Set IMAGE to the validated SGLang image tag or digest}"
|
||||||
|
|
||||||
|
RESULT_BASE="${RESULT_BASE:-/data/b300-dsv4-glm53-text-matrix}"
|
||||||
|
CACHE_HOST="${CACHE_HOST:-/data/b300-sglang-cache}"
|
||||||
|
SWA_PATCH_HOST="${SWA_PATCH_HOST:-}"
|
||||||
|
RUN_ID="${RUN_ID:-b300-dsv4-glm53-$(date +%Y%m%d-%H%M%S)}"
|
||||||
|
RUN_DIR="${RESULT_BASE}/${RUN_ID}"
|
||||||
|
CONTAINER="${CONTAINER:-b300-dsv4-glm53-matrix}"
|
||||||
|
PORT="${PORT:-30000}"
|
||||||
|
SEED="${SEED:-20260904}"
|
||||||
|
PROFILES="${PROFILES:-dsv4-low-latency dsv4-balanced glm53-low-latency glm53-high-throughput}"
|
||||||
|
STARTUP_TIMEOUT="${STARTUP_TIMEOUT:-3600}"
|
||||||
|
MAX_POINT_SECONDS="${MAX_POINT_SECONDS:-1800}"
|
||||||
|
|
||||||
|
DSV4_MODEL_HOST="${DSV4_MODEL_HOST:-/data/model/deepseek-ai--DeepSeek-V4-Flash-0731}"
|
||||||
|
GLM53_MODEL_HOST="${GLM53_MODEL_HOST:-/data/model/ZhipuAI--GLM-5.3}"
|
||||||
|
MODEL_CONTAINER="/models"
|
||||||
|
|
||||||
|
mkdir -p "${RUN_DIR}" "${CACHE_HOST}"
|
||||||
|
DOCKER_PATCH_ARGS=()
|
||||||
|
if [[ -n "${SWA_PATCH_HOST}" ]]; then
|
||||||
|
[[ -f "${SWA_PATCH_HOST}" ]] || {
|
||||||
|
printf 'SWA patch file not found: %s\n' "${SWA_PATCH_HOST}" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
DOCKER_PATCH_ARGS=(
|
||||||
|
-v "${SWA_PATCH_HOST}:/sgl-workspace/sglang/python/sglang/srt/mem_cache/allocator/swa.py:ro"
|
||||||
|
)
|
||||||
|
fi
|
||||||
|
exec > >(tee -a "${RUN_DIR}/matrix.log") 2>&1
|
||||||
|
|
||||||
|
log() {
|
||||||
|
printf '[%s] %s\n' "$(date -Is)" "$*"
|
||||||
|
}
|
||||||
|
|
||||||
|
server_healthy() {
|
||||||
|
curl --fail --silent --max-time 5 \
|
||||||
|
"http://127.0.0.1:${PORT}/health" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
set_profile() {
|
||||||
|
local profile="$1"
|
||||||
|
SERVER_ENV=()
|
||||||
|
SERVER_ARGS=()
|
||||||
|
|
||||||
|
case "${profile}" in
|
||||||
|
dsv4-low-latency)
|
||||||
|
MODEL_HOST="${DSV4_MODEL_HOST}"
|
||||||
|
MODEL_KEY="dsv4-flash-0731"
|
||||||
|
SERVER_ARGS=(
|
||||||
|
--trust-remote-code
|
||||||
|
--model-path "${MODEL_CONTAINER}"
|
||||||
|
--tp 4
|
||||||
|
--moe-runner-backend flashinfer_mxfp4
|
||||||
|
--speculative-algorithm DSPARK
|
||||||
|
--swa-full-tokens-ratio 0.1
|
||||||
|
--reasoning-parser deepseek-v4
|
||||||
|
--tool-call-parser deepseekv4
|
||||||
|
)
|
||||||
|
;;
|
||||||
|
dsv4-balanced)
|
||||||
|
MODEL_HOST="${DSV4_MODEL_HOST}"
|
||||||
|
MODEL_KEY="dsv4-flash-0731"
|
||||||
|
SERVER_ENV=(
|
||||||
|
-e SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=1024
|
||||||
|
)
|
||||||
|
SERVER_ARGS=(
|
||||||
|
--trust-remote-code
|
||||||
|
--model-path "${MODEL_CONTAINER}"
|
||||||
|
--tp 4
|
||||||
|
--dp 4
|
||||||
|
--enable-dp-attention
|
||||||
|
--moe-a2a-backend deepep
|
||||||
|
--deepep-config '{"normal_dispatch":{"num_sms":96},"normal_combine":{"num_sms":96}}'
|
||||||
|
--reasoning-parser deepseek-v4
|
||||||
|
--tool-call-parser deepseekv4
|
||||||
|
)
|
||||||
|
;;
|
||||||
|
glm53-low-latency)
|
||||||
|
MODEL_HOST="${GLM53_MODEL_HOST}"
|
||||||
|
MODEL_KEY="glm-5.3"
|
||||||
|
SERVER_ARGS=(
|
||||||
|
--model-path "${MODEL_CONTAINER}"
|
||||||
|
--tp 8
|
||||||
|
--speculative-algorithm EAGLE
|
||||||
|
--speculative-num-steps 5
|
||||||
|
--speculative-eagle-topk 1
|
||||||
|
--speculative-num-draft-tokens 6
|
||||||
|
--mem-fraction-static 0.8
|
||||||
|
--reasoning-parser glm45
|
||||||
|
--tool-call-parser glm47
|
||||||
|
)
|
||||||
|
;;
|
||||||
|
glm53-high-throughput)
|
||||||
|
MODEL_HOST="${GLM53_MODEL_HOST}"
|
||||||
|
MODEL_KEY="glm-5.3"
|
||||||
|
SERVER_ARGS=(
|
||||||
|
--model-path "${MODEL_CONTAINER}"
|
||||||
|
--tp 8
|
||||||
|
--dp 8
|
||||||
|
--enable-dp-attention
|
||||||
|
--moe-a2a-backend deepep
|
||||||
|
--speculative-algorithm EAGLE
|
||||||
|
--speculative-num-steps 1
|
||||||
|
--speculative-eagle-topk 1
|
||||||
|
--speculative-num-draft-tokens 2
|
||||||
|
--mem-fraction-static 0.85
|
||||||
|
--chunked-prefill-size 32768
|
||||||
|
--max-running-requests 256
|
||||||
|
--reasoning-parser glm45
|
||||||
|
--tool-call-parser glm47
|
||||||
|
)
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
log "ERROR: unknown profile=${profile}"
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
SERVER_ARGS+=(--host 0.0.0.0 --port "${PORT}")
|
||||||
|
}
|
||||||
|
|
||||||
|
capture_server_state() {
|
||||||
|
local profile="$1"
|
||||||
|
local profile_dir="${RUN_DIR}/${profile}"
|
||||||
|
sudo docker logs "${CONTAINER}" >"${profile_dir}/server.log" 2>&1 || true
|
||||||
|
sudo docker inspect "${CONTAINER}" >"${profile_dir}/container.inspect.json" 2>/dev/null || true
|
||||||
|
curl --silent --max-time 10 "http://127.0.0.1:${PORT}/get_server_info" \
|
||||||
|
>"${profile_dir}/server_info.json" 2>/dev/null || true
|
||||||
|
nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free,utilization.gpu,power.draw \
|
||||||
|
--format=csv >"${profile_dir}/gpu_state.csv" 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_server() {
|
||||||
|
local profile="${1:-unknown}"
|
||||||
|
local profile_dir="${RUN_DIR}/${profile}"
|
||||||
|
if sudo docker inspect "${CONTAINER}" >/dev/null 2>&1; then
|
||||||
|
capture_server_state "${profile}"
|
||||||
|
sudo docker stop --time 60 "${CONTAINER}" >/dev/null 2>&1 || true
|
||||||
|
sudo docker logs "${CONTAINER}" >"${profile_dir}/server.final.log" 2>&1 || true
|
||||||
|
sudo docker rm "${CONTAINER}" >/dev/null 2>&1 || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
write_server_command() {
|
||||||
|
local profile_dir="$1"
|
||||||
|
{
|
||||||
|
printf 'sudo docker run --name %q --gpus all --ipc=host --ulimit memlock=-1 ' "${CONTAINER}"
|
||||||
|
if ((${#SERVER_ENV[@]})); then
|
||||||
|
printf '%q ' "${SERVER_ENV[@]}"
|
||||||
|
fi
|
||||||
|
if ((${#DOCKER_PATCH_ARGS[@]})); then
|
||||||
|
printf '%q ' "${DOCKER_PATCH_ARGS[@]}"
|
||||||
|
fi
|
||||||
|
printf -- '-p %q:%q -v %q:%q:ro -v %q:/root/.cache %q sglang serve ' \
|
||||||
|
"${PORT}" "${PORT}" "${MODEL_HOST}" "${MODEL_CONTAINER}" "${CACHE_HOST}" "${IMAGE}"
|
||||||
|
printf '%q ' "${SERVER_ARGS[@]}"
|
||||||
|
printf '\n'
|
||||||
|
} >"${profile_dir}/server_command.txt"
|
||||||
|
}
|
||||||
|
|
||||||
|
start_server() {
|
||||||
|
local profile="$1"
|
||||||
|
local profile_dir="${RUN_DIR}/${profile}"
|
||||||
|
mkdir -p "${profile_dir}/points"
|
||||||
|
stop_server "${profile}"
|
||||||
|
write_server_command "${profile_dir}"
|
||||||
|
|
||||||
|
log "starting profile=${profile} model=${MODEL_KEY}"
|
||||||
|
sudo docker run -d \
|
||||||
|
--name "${CONTAINER}" \
|
||||||
|
--gpus all \
|
||||||
|
--ipc=host \
|
||||||
|
--ulimit memlock=-1 \
|
||||||
|
-p "${PORT}:${PORT}" \
|
||||||
|
-v "${MODEL_HOST}:${MODEL_CONTAINER}:ro" \
|
||||||
|
-v "${CACHE_HOST}:/root/.cache" \
|
||||||
|
-v "${profile_dir}:/results" \
|
||||||
|
"${DOCKER_PATCH_ARGS[@]}" \
|
||||||
|
"${SERVER_ENV[@]}" \
|
||||||
|
"${IMAGE}" \
|
||||||
|
sglang serve "${SERVER_ARGS[@]}" >/dev/null
|
||||||
|
|
||||||
|
local waited=0
|
||||||
|
until server_healthy; do
|
||||||
|
if ! sudo docker inspect -f '{{.State.Running}}' "${CONTAINER}" 2>/dev/null | grep -qx true; then
|
||||||
|
capture_server_state "${profile}"
|
||||||
|
log "ERROR: server exited during startup profile=${profile}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
if (( waited >= STARTUP_TIMEOUT )); then
|
||||||
|
capture_server_state "${profile}"
|
||||||
|
log "ERROR: startup timeout profile=${profile} waited=${waited}s"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
if (( waited % 60 == 0 )); then
|
||||||
|
log "waiting for server profile=${profile} elapsed=${waited}s"
|
||||||
|
fi
|
||||||
|
sleep 10
|
||||||
|
waited=$((waited + 10))
|
||||||
|
done
|
||||||
|
|
||||||
|
log "server healthy profile=${profile} elapsed=${waited}s"
|
||||||
|
rm -f "${profile_dir}/startup.failed"
|
||||||
|
capture_server_state "${profile}"
|
||||||
|
|
||||||
|
sudo docker exec "${CONTAINER}" python3 -m sglang.bench_serving \
|
||||||
|
--backend sglang \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "${PORT}" \
|
||||||
|
--model "${MODEL_CONTAINER}" \
|
||||||
|
--tokenizer "${MODEL_CONTAINER}" \
|
||||||
|
--dataset-name random-ids \
|
||||||
|
--tokenize-prompt \
|
||||||
|
--random-input-len 128 \
|
||||||
|
--random-output-len 8 \
|
||||||
|
--random-range-ratio 1.0 \
|
||||||
|
--num-prompts 1 \
|
||||||
|
--max-concurrency 1 \
|
||||||
|
--seed "${SEED}" \
|
||||||
|
--disable-tqdm >"${profile_dir}/warmup.log" 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
num_prompts() {
|
||||||
|
local isl="$1"
|
||||||
|
local concurrency="$2"
|
||||||
|
local prompts
|
||||||
|
|
||||||
|
if (( isl >= 1047552 )); then
|
||||||
|
prompts=$((concurrency > 4 ? concurrency : 4))
|
||||||
|
elif (( isl >= 524288 )); then
|
||||||
|
prompts=$((concurrency * 2))
|
||||||
|
(( prompts < 8 )) && prompts=8
|
||||||
|
elif (( isl >= 262144 )); then
|
||||||
|
prompts=$((concurrency * 3))
|
||||||
|
(( prompts < 16 )) && prompts=16
|
||||||
|
else
|
||||||
|
prompts=$((concurrency * 5))
|
||||||
|
(( prompts < 64 )) && prompts=64
|
||||||
|
fi
|
||||||
|
printf '%s\n' "${prompts}"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_point() {
|
||||||
|
local profile="$1"
|
||||||
|
local shape="$2"
|
||||||
|
local isl="$3"
|
||||||
|
local osl="$4"
|
||||||
|
local concurrency="$5"
|
||||||
|
local profile_dir="${RUN_DIR}/${profile}"
|
||||||
|
local point="${shape}_c${concurrency}"
|
||||||
|
local output="${profile_dir}/points/${point}.json"
|
||||||
|
local console="${profile_dir}/points/${point}.log"
|
||||||
|
local done_file="${profile_dir}/points/${point}.done"
|
||||||
|
local failed_file="${profile_dir}/points/${point}.failed"
|
||||||
|
local timeout_file="${profile_dir}/points/${point}.timeout-pass"
|
||||||
|
local prompts
|
||||||
|
prompts="$(num_prompts "${isl}" "${concurrency}")"
|
||||||
|
|
||||||
|
if [[ -s "${done_file}" ]]; then
|
||||||
|
log "skip completed profile=${profile} point=${point}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if ! server_healthy; then
|
||||||
|
log "ERROR: unhealthy before profile=${profile} point=${point}"
|
||||||
|
return 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
rm -f "${output}" "${failed_file}" "${timeout_file}"
|
||||||
|
log "run profile=${profile} shape=${shape} isl=${isl} osl=${osl} c=${concurrency} prompts=${prompts}"
|
||||||
|
printf '%s,%s,%s,%s,%s,%s,%s\n' \
|
||||||
|
"$(date -Is)" "${profile}" "${shape}" "${isl}" "${osl}" "${concurrency}" "${prompts}" \
|
||||||
|
>>"${RUN_DIR}/started_points.csv"
|
||||||
|
|
||||||
|
set +e
|
||||||
|
timeout --signal=TERM --kill-after=120 "${MAX_POINT_SECONDS}" \
|
||||||
|
sudo docker exec "${CONTAINER}" python3 -m sglang.bench_serving \
|
||||||
|
--backend sglang \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "${PORT}" \
|
||||||
|
--model "${MODEL_CONTAINER}" \
|
||||||
|
--tokenizer "${MODEL_CONTAINER}" \
|
||||||
|
--dataset-name random-ids \
|
||||||
|
--tokenize-prompt \
|
||||||
|
--random-input-len "${isl}" \
|
||||||
|
--random-output-len "${osl}" \
|
||||||
|
--random-range-ratio 1.0 \
|
||||||
|
--num-prompts "${prompts}" \
|
||||||
|
--max-concurrency "${concurrency}" \
|
||||||
|
--seed "${SEED}" \
|
||||||
|
--disable-tqdm \
|
||||||
|
--flush-cache \
|
||||||
|
--output-details \
|
||||||
|
--output-file "/results/points/${point}.json" \
|
||||||
|
>"${console}" 2>&1
|
||||||
|
local rc=$?
|
||||||
|
set -e
|
||||||
|
|
||||||
|
sudo docker logs --since 5m "${CONTAINER}" \
|
||||||
|
>"${profile_dir}/points/${point}.server-tail.log" 2>&1 || true
|
||||||
|
nvidia-smi --query-gpu=index,memory.used,memory.free,utilization.gpu,power.draw \
|
||||||
|
--format=csv >"${profile_dir}/points/${point}.gpu-after.csv" 2>/dev/null || true
|
||||||
|
|
||||||
|
if (( rc == 0 )) && [[ -s "${output}" ]] && \
|
||||||
|
jq -e --argjson expected "${prompts}" --argjson osl "${osl}" '
|
||||||
|
(.completed == $expected) and
|
||||||
|
(.errors | type == "array" and length == $expected and all(. == "")) and
|
||||||
|
(.ttfts | type == "array" and length == $expected and
|
||||||
|
(if $osl == 1 then all(. >= 0) else all(. > 0) end)) and
|
||||||
|
(.output_lens | type == "array" and length == $expected and all(. == $osl))
|
||||||
|
' "${output}" >/dev/null 2>&1; then
|
||||||
|
local zero_ttft_count
|
||||||
|
zero_ttft_count="$(jq '[.ttfts[] | select(. == 0)] | length' "${output}")"
|
||||||
|
if (( zero_ttft_count > 0 )); then
|
||||||
|
log "warning profile=${profile} point=${point} zero_ttft_count=${zero_ttft_count}"
|
||||||
|
fi
|
||||||
|
printf '%s\n' "$(date -Is)" >"${done_file}"
|
||||||
|
printf '%s,%s,%s,%s,%s\n' \
|
||||||
|
"$(date -Is)" "${profile}" "${point}" PASS "${rc}" >>"${RUN_DIR}/point_status.csv"
|
||||||
|
log "pass profile=${profile} point=${point}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if (( rc == 124 || rc == 137 )); then
|
||||||
|
[[ -s "${output}" ]] && jq -e . "${output}" >/dev/null 2>&1 || rm -f "${output}"
|
||||||
|
printf 'timestamp=%s\nrc=%s\ntimeout_seconds=%s\n' \
|
||||||
|
"$(date -Is)" "${rc}" "${MAX_POINT_SECONDS}" >"${timeout_file}"
|
||||||
|
printf 'TIMEOUT_PASS\n' >"${done_file}"
|
||||||
|
printf '%s,%s,%s,%s,%s\n' \
|
||||||
|
"$(date -Is)" "${profile}" "${point}" TIMEOUT_PASS "${rc}" >>"${RUN_DIR}/point_status.csv"
|
||||||
|
log "timeout pass profile=${profile} point=${point}"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
printf 'timestamp=%s\nrc=%s\n' "$(date -Is)" "${rc}" >"${failed_file}"
|
||||||
|
printf '%s,%s,%s,%s,%s\n' \
|
||||||
|
"$(date -Is)" "${profile}" "${point}" FAIL "${rc}" >>"${RUN_DIR}/point_status.csv"
|
||||||
|
log "failed profile=${profile} point=${point} rc=${rc}"
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
run_shape() {
|
||||||
|
local profile="$1"
|
||||||
|
local shape="$2"
|
||||||
|
local isl="$3"
|
||||||
|
local osl="$4"
|
||||||
|
shift 4
|
||||||
|
local min_failed=0
|
||||||
|
local concurrency
|
||||||
|
|
||||||
|
for concurrency in "$@"; do
|
||||||
|
if (( min_failed == 1 )); then
|
||||||
|
log "capacity short-circuit profile=${profile} shape=${shape} c=${concurrency}"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if ! run_point "${profile}" "${shape}" "${isl}" "${osl}" "${concurrency}"; then
|
||||||
|
if (( concurrency <= 2 )) && grep -qiE \
|
||||||
|
'out of memory|cuda error: out of memory|not enough memory|insufficient.*memory|cannot allocate' \
|
||||||
|
"${RUN_DIR}/${profile}/points/${shape}_c${concurrency}.log" \
|
||||||
|
"${RUN_DIR}/${profile}/points/${shape}_c${concurrency}.server-tail.log" 2>/dev/null; then
|
||||||
|
min_failed=1
|
||||||
|
log "capacity short-circuit armed profile=${profile} shape=${shape} after c=${concurrency}"
|
||||||
|
fi
|
||||||
|
if ! server_healthy; then
|
||||||
|
log "server failure aborts profile=${profile} shape=${shape}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
run_matrix() {
|
||||||
|
local profile="$1"
|
||||||
|
run_shape "${profile}" 16k_512 16384 512 1 8 32 64 128 256
|
||||||
|
run_shape "${profile}" 16k_1 16384 1 1 8 32 64 128 256
|
||||||
|
run_shape "${profile}" 1k_128 1024 128 1 8 32 64 128 256
|
||||||
|
run_shape "${profile}" 1k_4k 1024 4096 1 8 32 64 128 256
|
||||||
|
run_shape "${profile}" 64k_1 65536 1 1 4 8 16 32 64 128
|
||||||
|
run_shape "${profile}" 64k_512 65536 512 1 4 8 16 32 64
|
||||||
|
run_shape "${profile}" 128k_1 131072 1 1 2 4 8 16 32 64
|
||||||
|
run_shape "${profile}" 128k_512 131072 512 1 2 4 8 16 32
|
||||||
|
run_shape "${profile}" 256k_1 262144 1 1 2 4
|
||||||
|
run_shape "${profile}" 512k_1 524288 1 1 2 4
|
||||||
|
run_shape "${profile}" 1m_boundary_1 1047552 1 1 2 4
|
||||||
|
}
|
||||||
|
|
||||||
|
audit() {
|
||||||
|
sudo docker image inspect "${IMAGE}" >"${RUN_DIR}/image.inspect.json"
|
||||||
|
printf '%s\n' "${IMAGE}" >"${RUN_DIR}/image.txt"
|
||||||
|
sudo docker run --rm "${IMAGE}" python3 -c '
|
||||||
|
import importlib.metadata as m
|
||||||
|
import torch
|
||||||
|
for package in ("sglang", "flashinfer-python", "sgl-kernel", "sgl-deep-gemm", "sgl-deep-ep"):
|
||||||
|
try:
|
||||||
|
print(f"{package}={m.version(package)}")
|
||||||
|
except m.PackageNotFoundError:
|
||||||
|
print(f"{package}=NOT_INSTALLED")
|
||||||
|
print(f"torch={torch.__version__}")
|
||||||
|
print(f"torch_cuda={torch.version.cuda}")
|
||||||
|
' >"${RUN_DIR}/image_versions.txt"
|
||||||
|
nvidia-smi -q >"${RUN_DIR}/nvidia-smi-q.txt"
|
||||||
|
cp "${BASH_SOURCE[0]}" "${RUN_DIR}/run_full_text_matrix.sh"
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
[[ -d "${DSV4_MODEL_HOST}" ]] || { log "ERROR: missing ${DSV4_MODEL_HOST}"; exit 1; }
|
||||||
|
[[ -d "${GLM53_MODEL_HOST}" ]] || { log "ERROR: missing ${GLM53_MODEL_HOST}"; exit 1; }
|
||||||
|
sudo docker image inspect "${IMAGE}" >/dev/null 2>&1 || { log "ERROR: missing image ${IMAGE}"; exit 1; }
|
||||||
|
|
||||||
|
[[ -e "${RUN_DIR}/started_points.csv" ]] || \
|
||||||
|
printf 'timestamp,profile,shape,isl,osl,concurrency,num_prompts\n' >"${RUN_DIR}/started_points.csv"
|
||||||
|
[[ -e "${RUN_DIR}/point_status.csv" ]] || \
|
||||||
|
printf 'timestamp,profile,point,status,rc\n' >"${RUN_DIR}/point_status.csv"
|
||||||
|
audit
|
||||||
|
|
||||||
|
local profile
|
||||||
|
for profile in ${PROFILES}; do
|
||||||
|
CURRENT_PROFILE="${profile}"
|
||||||
|
set_profile "${profile}"
|
||||||
|
if [[ ! -d "${MODEL_HOST}" ]]; then
|
||||||
|
log "ERROR: model missing profile=${profile} path=${MODEL_HOST}"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
mkdir -p "${RUN_DIR}/${profile}/points"
|
||||||
|
if [[ -s "${RUN_DIR}/${profile}/profile.done" ]]; then
|
||||||
|
log "skip completed profile=${profile}"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if ! start_server "${profile}"; then
|
||||||
|
printf '%s\n' "$(date -Is)" >"${RUN_DIR}/${profile}/startup.failed"
|
||||||
|
stop_server "${profile}"
|
||||||
|
CURRENT_PROFILE=""
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if run_matrix "${profile}"; then
|
||||||
|
printf '%s\n' "$(date -Is)" >"${RUN_DIR}/${profile}/profile.done"
|
||||||
|
fi
|
||||||
|
stop_server "${profile}"
|
||||||
|
CURRENT_PROFILE=""
|
||||||
|
sleep 30
|
||||||
|
done
|
||||||
|
|
||||||
|
log "matrix finished run_dir=${RUN_DIR}"
|
||||||
|
}
|
||||||
|
|
||||||
|
cleanup() {
|
||||||
|
local rc=$?
|
||||||
|
trap - EXIT INT TERM
|
||||||
|
if [[ -n "${CURRENT_PROFILE:-}" ]]; then
|
||||||
|
set_profile "${CURRENT_PROFILE}" || true
|
||||||
|
log "cleanup profile=${CURRENT_PROFILE} rc=${rc}"
|
||||||
|
stop_server "${CURRENT_PROFILE}"
|
||||||
|
fi
|
||||||
|
exit "${rc}"
|
||||||
|
}
|
||||||
|
|
||||||
|
CURRENT_PROFILE=""
|
||||||
|
MODEL_HOST=""
|
||||||
|
MODEL_KEY=""
|
||||||
|
SERVER_ENV=()
|
||||||
|
SERVER_ARGS=()
|
||||||
|
trap cleanup EXIT INT TERM
|
||||||
|
main "$@"
|
||||||
@ -0,0 +1,497 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Summarize one B300 DeepSeek-V4-Flash and GLM-5.3 matrix run."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import csv
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
POINT_RE = re.compile(r"^(?P<shape>.+)_c(?P<concurrency>\d+)$")
|
||||||
|
MODES = (
|
||||||
|
"dsv4-low-latency",
|
||||||
|
"dsv4-balanced",
|
||||||
|
"glm53-low-latency",
|
||||||
|
"glm53-high-throughput",
|
||||||
|
)
|
||||||
|
MATRIX = (
|
||||||
|
("16k_512", (1, 8, 32, 64, 128, 256)),
|
||||||
|
("16k_1", (1, 8, 32, 64, 128, 256)),
|
||||||
|
("1k_128", (1, 8, 32, 64, 128, 256)),
|
||||||
|
("1k_4k", (1, 8, 32, 64, 128, 256)),
|
||||||
|
("64k_1", (1, 4, 8, 16, 32, 64, 128)),
|
||||||
|
("64k_512", (1, 4, 8, 16, 32, 64)),
|
||||||
|
("128k_1", (1, 2, 4, 8, 16, 32, 64)),
|
||||||
|
("128k_512", (1, 2, 4, 8, 16, 32)),
|
||||||
|
("256k_1", (1, 2, 4)),
|
||||||
|
("512k_1", (1, 2, 4)),
|
||||||
|
("1m_boundary_1", (1, 2, 4)),
|
||||||
|
)
|
||||||
|
SHAPE_ORDER = {shape: index for index, (shape, _) in enumerate(MATRIX)}
|
||||||
|
ERROR_RE = re.compile(
|
||||||
|
r"CUDA out of memory|Traceback|FAILED|EngineDeadError|NCCL.*error|"
|
||||||
|
r"CUDA error|context length|\bError:",
|
||||||
|
re.IGNORECASE,
|
||||||
|
)
|
||||||
|
FIELDS = (
|
||||||
|
"mode",
|
||||||
|
"shape",
|
||||||
|
"concurrency",
|
||||||
|
"status",
|
||||||
|
"completed",
|
||||||
|
"duration_s",
|
||||||
|
"request_throughput",
|
||||||
|
"input_throughput",
|
||||||
|
"output_throughput",
|
||||||
|
"total_token_throughput",
|
||||||
|
"ttft_mean_ms",
|
||||||
|
"ttft_p50_ms",
|
||||||
|
"ttft_p95_ms",
|
||||||
|
"ttft_p99_ms",
|
||||||
|
"tpot_mean_ms",
|
||||||
|
"tpot_p50_ms",
|
||||||
|
"tpot_p95_ms",
|
||||||
|
"tpot_p99_ms",
|
||||||
|
"e2e_mean_ms",
|
||||||
|
"e2e_p95_ms",
|
||||||
|
"source",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def first(data: dict, *keys: str):
|
||||||
|
for key in keys:
|
||||||
|
if data.get(key) is not None:
|
||||||
|
return data[key]
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def load_point(mode: str, path: Path) -> dict:
|
||||||
|
match = POINT_RE.match(path.stem)
|
||||||
|
if not match:
|
||||||
|
raise ValueError(f"unexpected point name: {path.name}")
|
||||||
|
data = json.loads(path.read_text())
|
||||||
|
done = path.with_suffix(".done").exists()
|
||||||
|
failed = path.with_suffix(".failed").exists()
|
||||||
|
timeout_pass = path.with_suffix(".timeout-pass").exists()
|
||||||
|
ttfts = data.get("ttfts")
|
||||||
|
output_lens = data.get("output_lens")
|
||||||
|
completed = first(data, "completed", "completed_requests")
|
||||||
|
errors = data.get("errors")
|
||||||
|
allow_zero_ttft = (
|
||||||
|
isinstance(output_lens, list)
|
||||||
|
and len(output_lens) == completed
|
||||||
|
and all(value == 1 for value in output_lens)
|
||||||
|
)
|
||||||
|
invalid_result = bool(completed) and (
|
||||||
|
not isinstance(ttfts, list)
|
||||||
|
or len(ttfts) != completed
|
||||||
|
or any(
|
||||||
|
value is None
|
||||||
|
or not isinstance(value, (int, float))
|
||||||
|
or not math.isfinite(value)
|
||||||
|
or (value < 0 if allow_zero_ttft else value <= 0)
|
||||||
|
for value in ttfts
|
||||||
|
)
|
||||||
|
or not isinstance(errors, list)
|
||||||
|
or len(errors) != completed
|
||||||
|
or any(bool(value) for value in errors)
|
||||||
|
)
|
||||||
|
status = (
|
||||||
|
"TIMEOUT_PASS"
|
||||||
|
if timeout_pass
|
||||||
|
else "INVALID"
|
||||||
|
if invalid_result
|
||||||
|
else "PASS"
|
||||||
|
if done
|
||||||
|
else "FAIL"
|
||||||
|
if failed
|
||||||
|
else "INCOMPLETE"
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"mode": mode,
|
||||||
|
"shape": match.group("shape"),
|
||||||
|
"concurrency": int(match.group("concurrency")),
|
||||||
|
"status": status,
|
||||||
|
"completed": completed,
|
||||||
|
"duration_s": first(data, "benchmark_duration", "duration"),
|
||||||
|
"request_throughput": first(data, "request_throughput"),
|
||||||
|
"input_throughput": first(data, "input_throughput"),
|
||||||
|
"output_throughput": first(data, "output_throughput"),
|
||||||
|
"total_token_throughput": first(
|
||||||
|
data, "total_token_throughput", "total_throughput"
|
||||||
|
),
|
||||||
|
"ttft_mean_ms": first(data, "mean_ttft_ms"),
|
||||||
|
"ttft_p50_ms": first(data, "median_ttft_ms", "p50_ttft_ms"),
|
||||||
|
"ttft_p95_ms": first(data, "p95_ttft_ms"),
|
||||||
|
"ttft_p99_ms": first(data, "p99_ttft_ms"),
|
||||||
|
"tpot_mean_ms": first(data, "mean_tpot_ms"),
|
||||||
|
"tpot_p50_ms": first(data, "median_tpot_ms", "p50_tpot_ms"),
|
||||||
|
"tpot_p95_ms": first(data, "p95_tpot_ms"),
|
||||||
|
"tpot_p99_ms": first(data, "p99_tpot_ms"),
|
||||||
|
"e2e_mean_ms": first(data, "mean_e2el_ms", "mean_e2e_latency_ms"),
|
||||||
|
"e2e_p95_ms": first(data, "p95_e2el_ms", "p95_e2e_latency_ms"),
|
||||||
|
"source": str(path),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_failed_marker(mode: str, path: Path) -> dict:
|
||||||
|
match = POINT_RE.match(path.stem)
|
||||||
|
if not match:
|
||||||
|
raise ValueError(f"unexpected failure marker: {path.name}")
|
||||||
|
row = {field: "" for field in FIELDS}
|
||||||
|
row.update(
|
||||||
|
{
|
||||||
|
"mode": mode,
|
||||||
|
"shape": match.group("shape"),
|
||||||
|
"concurrency": int(match.group("concurrency")),
|
||||||
|
"status": "FAIL",
|
||||||
|
"source": str(path),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def load_timeout_pass_marker(mode: str, path: Path) -> dict:
|
||||||
|
match = POINT_RE.match(path.stem)
|
||||||
|
if not match:
|
||||||
|
raise ValueError(f"unexpected timeout marker: {path.name}")
|
||||||
|
row = {field: "" for field in FIELDS}
|
||||||
|
row.update(
|
||||||
|
{
|
||||||
|
"mode": mode,
|
||||||
|
"shape": match.group("shape"),
|
||||||
|
"concurrency": int(match.group("concurrency")),
|
||||||
|
"status": "TIMEOUT_PASS",
|
||||||
|
"source": str(path),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return row
|
||||||
|
|
||||||
|
|
||||||
|
def fmt(value, digits: int = 2) -> str:
|
||||||
|
if value == "" or value is None:
|
||||||
|
return "-"
|
||||||
|
if isinstance(value, (float, int)):
|
||||||
|
return f"{value:.{digits}f}"
|
||||||
|
return str(value)
|
||||||
|
|
||||||
|
|
||||||
|
def build_progress(rows: list[dict]) -> dict:
|
||||||
|
observed = {
|
||||||
|
(row["mode"], row["shape"], row["concurrency"]): row["status"]
|
||||||
|
for row in rows
|
||||||
|
}
|
||||||
|
per_mode = {}
|
||||||
|
for mode in MODES:
|
||||||
|
expected = sum(len(concurrencies) for _, concurrencies in MATRIX)
|
||||||
|
statuses = [
|
||||||
|
observed.get((mode, shape, concurrency), "PENDING")
|
||||||
|
for shape, concurrencies in MATRIX
|
||||||
|
for concurrency in concurrencies
|
||||||
|
]
|
||||||
|
passed = statuses.count("PASS")
|
||||||
|
timeout_passed = statuses.count("TIMEOUT_PASS")
|
||||||
|
failed = statuses.count("FAIL")
|
||||||
|
invalid = statuses.count("INVALID")
|
||||||
|
per_mode[mode] = {
|
||||||
|
"expected": expected,
|
||||||
|
"pass": passed,
|
||||||
|
"timeout_pass": timeout_passed,
|
||||||
|
"fail": failed,
|
||||||
|
"invalid": invalid,
|
||||||
|
"pending": expected - passed - timeout_passed - failed - invalid,
|
||||||
|
"finished_percent": round(
|
||||||
|
100 * (passed + timeout_passed + failed + invalid) / expected, 2
|
||||||
|
),
|
||||||
|
}
|
||||||
|
expected_total = sum(item["expected"] for item in per_mode.values())
|
||||||
|
pass_total = sum(item["pass"] for item in per_mode.values())
|
||||||
|
timeout_pass_total = sum(item["timeout_pass"] for item in per_mode.values())
|
||||||
|
fail_total = sum(item["fail"] for item in per_mode.values())
|
||||||
|
invalid_total = sum(item["invalid"] for item in per_mode.values())
|
||||||
|
return {
|
||||||
|
"expected": expected_total,
|
||||||
|
"pass": pass_total,
|
||||||
|
"timeout_pass": timeout_pass_total,
|
||||||
|
"fail": fail_total,
|
||||||
|
"invalid": invalid_total,
|
||||||
|
"pending": (
|
||||||
|
expected_total
|
||||||
|
- pass_total
|
||||||
|
- timeout_pass_total
|
||||||
|
- fail_total
|
||||||
|
- invalid_total
|
||||||
|
),
|
||||||
|
"finished_percent": round(
|
||||||
|
100
|
||||||
|
* (pass_total + timeout_pass_total + fail_total + invalid_total)
|
||||||
|
/ expected_total,
|
||||||
|
2,
|
||||||
|
),
|
||||||
|
"modes": per_mode,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def write_markdown(
|
||||||
|
run_dir: Path, rows: list[dict], progress: dict, output: Path
|
||||||
|
) -> None:
|
||||||
|
lines = [
|
||||||
|
f"# B300 DeepSeek-V4-Flash / GLM-5.3 Text Matrix: {run_dir.name}",
|
||||||
|
"",
|
||||||
|
"## Progress",
|
||||||
|
"",
|
||||||
|
"| Mode | Finished | Expected | Pass | Timeout pass | Invalid | Fail | Progress |",
|
||||||
|
"|---|---:|---:|---:|---:|---:|---:|---:|",
|
||||||
|
]
|
||||||
|
for mode in MODES:
|
||||||
|
item = progress["modes"][mode]
|
||||||
|
lines.append(
|
||||||
|
f"| {mode} | "
|
||||||
|
f"{item['pass'] + item['timeout_pass'] + item['invalid'] + item['fail']} | "
|
||||||
|
f"{item['expected']} | {item['pass']} | {item['timeout_pass']} | "
|
||||||
|
f"{item['invalid']} | {item['fail']} | {item['finished_percent']:.2f}% |"
|
||||||
|
)
|
||||||
|
lines.extend(
|
||||||
|
[
|
||||||
|
"",
|
||||||
|
"## Results",
|
||||||
|
"",
|
||||||
|
"| Mode | Shape | C | Status | Input TPS | Output TPS | TTFT P95 (ms) | TPOT P95 (ms) |",
|
||||||
|
"|---|---|---:|---|---:|---:|---:|---:|",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
for row in rows:
|
||||||
|
lines.append(
|
||||||
|
"| {mode} | {shape} | {c} | {status} | {itps} | {otps} | {ttft} | {tpot} |".format(
|
||||||
|
mode=row["mode"],
|
||||||
|
shape=row["shape"],
|
||||||
|
c=row["concurrency"],
|
||||||
|
status=row["status"],
|
||||||
|
itps=fmt(row["input_throughput"]),
|
||||||
|
otps=fmt(row["output_throughput"]),
|
||||||
|
ttft=fmt(row["ttft_p95_ms"]),
|
||||||
|
tpot=fmt(row["tpot_p95_ms"]),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
output.write_text("\n".join(lines) + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
def load_server_config(run_dir: Path, mode: str) -> dict:
|
||||||
|
path = run_dir / mode / "server_info.json"
|
||||||
|
if not path.is_file():
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
return json.loads(path.read_text())
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def scan_mode_errors(run_dir: Path, mode: str) -> tuple[int, list[str], str]:
|
||||||
|
mode_dir = run_dir / mode
|
||||||
|
candidates = (mode_dir / "server.final.log", mode_dir / "server.log")
|
||||||
|
path = next((candidate for candidate in candidates if candidate.is_file()), None)
|
||||||
|
if path is None:
|
||||||
|
return 0, [], "-"
|
||||||
|
|
||||||
|
count = 0
|
||||||
|
samples = []
|
||||||
|
seen = set()
|
||||||
|
with path.open(errors="replace") as handle:
|
||||||
|
for line in handle:
|
||||||
|
if not ERROR_RE.search(line):
|
||||||
|
continue
|
||||||
|
count += 1
|
||||||
|
sample = line.strip()
|
||||||
|
if sample and sample not in seen and len(samples) < 5:
|
||||||
|
samples.append(sample[:300])
|
||||||
|
seen.add(sample)
|
||||||
|
return count, samples, str(path)
|
||||||
|
|
||||||
|
|
||||||
|
def best_row(rows: list[dict], field: str) -> dict | None:
|
||||||
|
candidates = [
|
||||||
|
row
|
||||||
|
for row in rows
|
||||||
|
if row["status"] == "PASS"
|
||||||
|
and isinstance(row[field], (int, float))
|
||||||
|
and math.isfinite(row[field])
|
||||||
|
]
|
||||||
|
return max(candidates, key=lambda row: row[field], default=None)
|
||||||
|
|
||||||
|
|
||||||
|
def write_comparison(
|
||||||
|
run_dir: Path, rows: list[dict], progress: dict, output: Path
|
||||||
|
) -> None:
|
||||||
|
lines = [
|
||||||
|
f"# B300 DeepSeek-V4-Flash / GLM-5.3 Comparison: {run_dir.name}",
|
||||||
|
"",
|
||||||
|
"## Run Status",
|
||||||
|
"",
|
||||||
|
f"- Finished: {progress['finished_percent']:.2f}%",
|
||||||
|
f"- Pass: {progress['pass']}",
|
||||||
|
f"- Timeout pass: {progress['timeout_pass']}",
|
||||||
|
f"- Invalid: {progress['invalid']}",
|
||||||
|
f"- Fail: {progress['fail']}",
|
||||||
|
"",
|
||||||
|
"## Service Configuration And Capacity",
|
||||||
|
"",
|
||||||
|
"| Profile | TP | DP | EP | Max running | Max tokens/rank | Chunk | KV dtype | Attention | MoE runner | MoE A2A | Speculative |",
|
||||||
|
"|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---|",
|
||||||
|
]
|
||||||
|
for mode in MODES:
|
||||||
|
config = load_server_config(run_dir, mode)
|
||||||
|
lines.append(
|
||||||
|
"| {mode} | {tp} | {dp} | {ep} | {running} | {tokens} | {chunk} | "
|
||||||
|
"{kv} | {attention} | {moe} | {a2a} | {spec} |".format(
|
||||||
|
mode=mode,
|
||||||
|
tp=fmt(config.get("tp_size"), 0),
|
||||||
|
dp=fmt(config.get("dp_size"), 0),
|
||||||
|
ep=fmt(config.get("ep_size"), 0),
|
||||||
|
running=fmt(config.get("max_running_requests"), 0),
|
||||||
|
tokens=fmt(config.get("max_total_num_tokens"), 0),
|
||||||
|
chunk=fmt(config.get("chunked_prefill_size"), 0),
|
||||||
|
kv=config.get("kv_cache_dtype", "-"),
|
||||||
|
attention=config.get("attention_backend", "-"),
|
||||||
|
moe=config.get("moe_runner_backend", "-"),
|
||||||
|
a2a=config.get("moe_a2a_backend", "-"),
|
||||||
|
spec=config.get("speculative_algorithm", "-"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
lines.extend(
|
||||||
|
[
|
||||||
|
"",
|
||||||
|
"## Throughput Peaks And Their Latency",
|
||||||
|
"",
|
||||||
|
"Only validated PASS points are considered.",
|
||||||
|
"",
|
||||||
|
"| Shape | Mode | Peak input TPS @ C | TTFT P95 (ms) | Peak output TPS @ C | TPOT P95 (ms) |",
|
||||||
|
"|---|---|---:|---:|---:|---:|",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
for shape, _ in MATRIX:
|
||||||
|
for mode in MODES:
|
||||||
|
group = [
|
||||||
|
row for row in rows if row["shape"] == shape and row["mode"] == mode
|
||||||
|
]
|
||||||
|
input_peak = best_row(group, "input_throughput")
|
||||||
|
output_peak = best_row(group, "output_throughput")
|
||||||
|
if input_peak is None and output_peak is None:
|
||||||
|
continue
|
||||||
|
lines.append(
|
||||||
|
"| {shape} | {mode} | {input_tps} @ C{input_c} | {ttft} | "
|
||||||
|
"{output_tps} @ C{output_c} | {tpot} |".format(
|
||||||
|
shape=shape,
|
||||||
|
mode=mode,
|
||||||
|
input_tps=fmt(input_peak["input_throughput"]),
|
||||||
|
input_c=input_peak["concurrency"],
|
||||||
|
ttft=fmt(input_peak["ttft_p95_ms"]),
|
||||||
|
output_tps=fmt(output_peak["output_throughput"]),
|
||||||
|
output_c=output_peak["concurrency"],
|
||||||
|
tpot=fmt(output_peak["tpot_p95_ms"]),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
boundary_rows = [row for row in rows if row["status"] != "PASS"]
|
||||||
|
lines.extend(
|
||||||
|
[
|
||||||
|
"",
|
||||||
|
"## Capacity And Failure Boundaries",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
if not boundary_rows:
|
||||||
|
lines.append("No timeout, invalid, or failed point has been observed yet.")
|
||||||
|
else:
|
||||||
|
lines.extend(
|
||||||
|
[
|
||||||
|
"| Mode | Shape | C | Status | Evidence |",
|
||||||
|
"|---|---|---:|---|---|",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
for row in boundary_rows:
|
||||||
|
lines.append(
|
||||||
|
f"| {row['mode']} | {row['shape']} | {row['concurrency']} | "
|
||||||
|
f"{row['status']} | `{row['source']}` |"
|
||||||
|
)
|
||||||
|
|
||||||
|
lines.extend(["", "## Service Error Summary", ""])
|
||||||
|
for mode in MODES:
|
||||||
|
count, samples, source = scan_mode_errors(run_dir, mode)
|
||||||
|
lines.append(f"### {mode}")
|
||||||
|
lines.append("")
|
||||||
|
lines.append(f"- Matched lines: {count}")
|
||||||
|
lines.append(f"- Source: `{source}`")
|
||||||
|
for sample in samples:
|
||||||
|
lines.append(f"- `{sample}`")
|
||||||
|
lines.append("")
|
||||||
|
|
||||||
|
output.write_text("\n".join(lines) + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("run_dir", type=Path)
|
||||||
|
parser.add_argument("--output-dir", type=Path)
|
||||||
|
args = parser.parse_args()
|
||||||
|
run_dir = args.run_dir.resolve()
|
||||||
|
output_dir = (args.output_dir or run_dir).resolve()
|
||||||
|
output_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
for mode in MODES:
|
||||||
|
points_dir = run_dir / mode / "points"
|
||||||
|
if not points_dir.is_dir():
|
||||||
|
continue
|
||||||
|
loaded_stems = set()
|
||||||
|
for path in points_dir.glob("*.json"):
|
||||||
|
try:
|
||||||
|
rows.append(load_point(mode, path))
|
||||||
|
loaded_stems.add(path.stem)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
pass
|
||||||
|
for path in points_dir.glob("*.timeout-pass"):
|
||||||
|
if path.stem not in loaded_stems:
|
||||||
|
rows.append(load_timeout_pass_marker(mode, path))
|
||||||
|
loaded_stems.add(path.stem)
|
||||||
|
for path in points_dir.glob("*.failed"):
|
||||||
|
if path.stem not in loaded_stems:
|
||||||
|
rows.append(load_failed_marker(mode, path))
|
||||||
|
mode_order = {name: index for index, name in enumerate(MODES)}
|
||||||
|
rows.sort(
|
||||||
|
key=lambda row: (
|
||||||
|
mode_order[row["mode"]],
|
||||||
|
SHAPE_ORDER.get(row["shape"], len(SHAPE_ORDER)),
|
||||||
|
row["concurrency"],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
csv_path = output_dir / "summary.csv"
|
||||||
|
with csv_path.open("w", newline="") as handle:
|
||||||
|
writer = csv.DictWriter(handle, fieldnames=FIELDS)
|
||||||
|
writer.writeheader()
|
||||||
|
writer.writerows(rows)
|
||||||
|
progress = build_progress(rows)
|
||||||
|
(output_dir / "progress.json").write_text(
|
||||||
|
json.dumps(progress, indent=2, sort_keys=True) + "\n"
|
||||||
|
)
|
||||||
|
write_markdown(run_dir, rows, progress, output_dir / "summary.md")
|
||||||
|
write_comparison(run_dir, rows, progress, output_dir / "comparison.md")
|
||||||
|
|
||||||
|
passes = sum(row["status"] == "PASS" for row in rows)
|
||||||
|
failures = sum(row["status"] == "FAIL" for row in rows)
|
||||||
|
timeout_passes = sum(row["status"] == "TIMEOUT_PASS" for row in rows)
|
||||||
|
invalids = sum(row["status"] == "INVALID" for row in rows)
|
||||||
|
print(
|
||||||
|
f"rows={len(rows)} pass={passes} timeout_pass={timeout_passes} "
|
||||||
|
f"invalid={invalids} fail={failures} "
|
||||||
|
f"progress={progress['finished_percent']:.2f}% output={output_dir}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@ -0,0 +1,542 @@
|
|||||||
|
import torch
|
||||||
|
|
||||||
|
from sglang.srt.mem_cache.allocator.base import BaseTokenToKVPoolAllocator
|
||||||
|
from sglang.srt.mem_cache.allocator.paged import PagedTokenToKVPoolAllocator
|
||||||
|
from sglang.srt.mem_cache.allocator.token import TokenToKVPoolAllocator
|
||||||
|
from sglang.srt.mem_cache.base_swa_memory_pool import BaseSWAKVPool
|
||||||
|
from sglang.srt.utils import is_npu
|
||||||
|
from sglang.srt.utils.common import get_num_new_pages
|
||||||
|
|
||||||
|
_is_npu = is_npu()
|
||||||
|
|
||||||
|
if _is_npu:
|
||||||
|
import torch_npu
|
||||||
|
|
||||||
|
from sglang.srt.hardware_backend.npu.allocator_npu import (
|
||||||
|
NPUPagedTokenToKVPoolAllocator,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SWATokenToKVPoolAllocator(BaseTokenToKVPoolAllocator):
|
||||||
|
"""Allocator for SWA hybrid KV cache."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
size: int,
|
||||||
|
size_swa: int,
|
||||||
|
page_size: int,
|
||||||
|
dtype: torch.dtype,
|
||||||
|
device: str,
|
||||||
|
kvcache: BaseSWAKVPool,
|
||||||
|
need_sort: bool,
|
||||||
|
):
|
||||||
|
assert isinstance(kvcache, BaseSWAKVPool)
|
||||||
|
self._size_full = size
|
||||||
|
self._size_swa = size_swa
|
||||||
|
self.dtype = dtype
|
||||||
|
self.device = device
|
||||||
|
self.page_size = page_size
|
||||||
|
|
||||||
|
full_kv_pool = getattr(kvcache, "full_kv_pool", None)
|
||||||
|
swa_kv_pool = getattr(kvcache, "swa_kv_pool", None)
|
||||||
|
|
||||||
|
if page_size == 1:
|
||||||
|
self.full_attn_allocator = TokenToKVPoolAllocator(
|
||||||
|
size,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
full_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
self.swa_attn_allocator = TokenToKVPoolAllocator(
|
||||||
|
size_swa,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
swa_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
if _is_npu:
|
||||||
|
PagedTokenToKVPoolAllocatorClass = NPUPagedTokenToKVPoolAllocator
|
||||||
|
else:
|
||||||
|
PagedTokenToKVPoolAllocatorClass = PagedTokenToKVPoolAllocator
|
||||||
|
self.full_attn_allocator = PagedTokenToKVPoolAllocatorClass(
|
||||||
|
size,
|
||||||
|
page_size,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
full_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
self.swa_attn_allocator = PagedTokenToKVPoolAllocatorClass(
|
||||||
|
size_swa,
|
||||||
|
page_size,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
swa_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
# Note: append one more item of value -1 in the end so -1 maps to -1.
|
||||||
|
# It is needed for the last_loc in alloc_extend, where the first full_last_loc
|
||||||
|
# is -1, and we need to map it to swa_last_loc -1 as well.
|
||||||
|
self.full_to_swa_index_mapping = torch.cat(
|
||||||
|
[
|
||||||
|
torch.zeros(
|
||||||
|
size + self.page_size,
|
||||||
|
dtype=torch.int64,
|
||||||
|
device=device,
|
||||||
|
),
|
||||||
|
torch.tensor([-1], dtype=torch.int64, device=device),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
self.need_sort = need_sort
|
||||||
|
self.free_pages = None
|
||||||
|
self.release_pages = None
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
self.free_group = []
|
||||||
|
self.swa_free_group = []
|
||||||
|
|
||||||
|
self._kvcache = kvcache
|
||||||
|
self.clear()
|
||||||
|
self._kvcache.register_mapping(self.full_to_swa_index_mapping)
|
||||||
|
|
||||||
|
def available_size(self):
|
||||||
|
return min(
|
||||||
|
self.full_attn_allocator.available_size(),
|
||||||
|
self.swa_attn_allocator.available_size(),
|
||||||
|
)
|
||||||
|
|
||||||
|
def full_available_size(self):
|
||||||
|
return self.full_attn_allocator.available_size()
|
||||||
|
|
||||||
|
def swa_available_size(self):
|
||||||
|
return self.swa_attn_allocator.available_size()
|
||||||
|
|
||||||
|
# Slot-conservation views for the leak invariant. On the non-shared allocator
|
||||||
|
# the static budget IS physical (conserve == physical); the shared composite
|
||||||
|
# overrides these with the static-cap view.
|
||||||
|
def _conserve_full_available_size(self):
|
||||||
|
return self.full_available_size()
|
||||||
|
|
||||||
|
def _conserve_swa_available_size(self):
|
||||||
|
return self.swa_available_size()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def size(self):
|
||||||
|
return min(self._size_full, self._size_swa)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def size_swa(self):
|
||||||
|
return self._size_swa
|
||||||
|
|
||||||
|
@property
|
||||||
|
def size_full(self):
|
||||||
|
return self._size_full
|
||||||
|
|
||||||
|
def debug_print(self) -> str:
|
||||||
|
msg = ""
|
||||||
|
msg += f"#swa-available-size: {self.swa_attn_allocator.available_size()}, "
|
||||||
|
msg += (
|
||||||
|
f"#full-attn-available-size: {self.full_attn_allocator.available_size()}, "
|
||||||
|
)
|
||||||
|
return msg
|
||||||
|
|
||||||
|
def get_kvcache(self):
|
||||||
|
return self._kvcache
|
||||||
|
|
||||||
|
def translate_loc_from_full_to_swa(self, kv_indices: torch.Tensor):
|
||||||
|
assert self._kvcache.full_to_swa_index_mapping is not None
|
||||||
|
return self._kvcache.translate_loc_from_full_to_swa(kv_indices)
|
||||||
|
|
||||||
|
def alloc(self, need_size: int):
|
||||||
|
assert self.page_size == 1
|
||||||
|
if need_size > self.full_attn_allocator.available_size():
|
||||||
|
return None
|
||||||
|
if need_size > self.swa_attn_allocator.available_size():
|
||||||
|
return None
|
||||||
|
|
||||||
|
alloc_full_indices = self.full_attn_allocator.alloc(need_size)
|
||||||
|
alloc_swa_indices = self.swa_attn_allocator.alloc(need_size)
|
||||||
|
assert alloc_full_indices is not None
|
||||||
|
assert alloc_swa_indices is not None
|
||||||
|
|
||||||
|
self.set_full_to_swa_mapping(alloc_full_indices, alloc_swa_indices)
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
def new_pages_available(self, num_full_pages: int, num_swa_pages: int) -> bool:
|
||||||
|
return (
|
||||||
|
num_full_pages
|
||||||
|
<= self.full_attn_allocator.available_size() // self.page_size
|
||||||
|
and num_swa_pages
|
||||||
|
<= self.swa_attn_allocator.available_size() // self.page_size
|
||||||
|
)
|
||||||
|
|
||||||
|
def alloc_extend(
|
||||||
|
self,
|
||||||
|
prefix_lens: torch.Tensor,
|
||||||
|
prefix_lens_cpu: torch.Tensor,
|
||||||
|
seq_lens: torch.Tensor,
|
||||||
|
seq_lens_cpu: torch.Tensor,
|
||||||
|
last_loc: torch.Tensor, # last_loc for full layers
|
||||||
|
extend_num_tokens: int,
|
||||||
|
):
|
||||||
|
assert self.page_size > 1
|
||||||
|
|
||||||
|
num_new_pages = get_num_new_pages(
|
||||||
|
seq_lens=seq_lens_cpu, page_size=self.page_size, prefix_lens=prefix_lens_cpu
|
||||||
|
)
|
||||||
|
if not self.new_pages_available(num_new_pages, num_new_pages):
|
||||||
|
return None
|
||||||
|
|
||||||
|
swa_last_loc = self.translate_loc_from_full_to_swa(last_loc)
|
||||||
|
|
||||||
|
alloc_full_indices = self.full_attn_allocator.alloc_extend(
|
||||||
|
prefix_lens,
|
||||||
|
prefix_lens_cpu,
|
||||||
|
seq_lens,
|
||||||
|
seq_lens_cpu,
|
||||||
|
last_loc,
|
||||||
|
extend_num_tokens,
|
||||||
|
num_new_pages=num_new_pages,
|
||||||
|
)
|
||||||
|
alloc_swa_indices = self.swa_attn_allocator.alloc_extend(
|
||||||
|
prefix_lens,
|
||||||
|
prefix_lens_cpu,
|
||||||
|
seq_lens,
|
||||||
|
seq_lens_cpu,
|
||||||
|
swa_last_loc,
|
||||||
|
extend_num_tokens,
|
||||||
|
num_new_pages=num_new_pages,
|
||||||
|
)
|
||||||
|
assert alloc_full_indices is not None
|
||||||
|
assert alloc_swa_indices is not None
|
||||||
|
|
||||||
|
self.set_full_to_swa_mapping(alloc_full_indices, alloc_swa_indices)
|
||||||
|
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
def alloc_extend_swa_tail(
|
||||||
|
self,
|
||||||
|
prefix_lens: torch.Tensor,
|
||||||
|
prefix_lens_cpu: torch.Tensor,
|
||||||
|
seq_lens: torch.Tensor,
|
||||||
|
seq_lens_cpu: torch.Tensor,
|
||||||
|
last_loc: torch.Tensor, # last_loc for full layers
|
||||||
|
extend_num_tokens: int,
|
||||||
|
swa_tail_len: int,
|
||||||
|
):
|
||||||
|
"""Allocate full KV for the whole extend and SWA KV only for the tail.
|
||||||
|
|
||||||
|
This is used by disaggregated decode preallocation: decode receives full
|
||||||
|
prompt KV for full-attention layers, but only the sliding-window state is
|
||||||
|
transferred for SWA layers.
|
||||||
|
"""
|
||||||
|
assert self.page_size > 1
|
||||||
|
assert len(seq_lens_cpu) == 1, "SWA tail allocation currently supports bs=1"
|
||||||
|
assert len(prefix_lens_cpu) == 1
|
||||||
|
assert 0 <= swa_tail_len <= extend_num_tokens
|
||||||
|
|
||||||
|
num_full_pages = get_num_new_pages(
|
||||||
|
seq_lens=seq_lens_cpu, page_size=self.page_size, prefix_lens=prefix_lens_cpu
|
||||||
|
)
|
||||||
|
num_swa_pages = (swa_tail_len + self.page_size - 1) // self.page_size
|
||||||
|
if not self.new_pages_available(num_full_pages, num_swa_pages):
|
||||||
|
return None
|
||||||
|
|
||||||
|
alloc_full_indices = self.full_attn_allocator.alloc_extend(
|
||||||
|
prefix_lens,
|
||||||
|
prefix_lens_cpu,
|
||||||
|
seq_lens,
|
||||||
|
seq_lens_cpu,
|
||||||
|
last_loc,
|
||||||
|
extend_num_tokens,
|
||||||
|
num_new_pages=num_full_pages,
|
||||||
|
)
|
||||||
|
assert alloc_full_indices is not None
|
||||||
|
|
||||||
|
if swa_tail_len == 0:
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
device = self.device
|
||||||
|
swa_prefix_lens = torch.zeros((1,), dtype=torch.int64, device=device)
|
||||||
|
swa_prefix_lens_cpu = torch.zeros((1,), dtype=torch.int64)
|
||||||
|
swa_seq_lens = torch.tensor([swa_tail_len], dtype=torch.int64, device=device)
|
||||||
|
swa_seq_lens_cpu = torch.tensor([swa_tail_len], dtype=torch.int64)
|
||||||
|
swa_last_loc = torch.tensor([-1], dtype=torch.int64, device=device)
|
||||||
|
|
||||||
|
alloc_swa_indices = self.swa_attn_allocator.alloc_extend(
|
||||||
|
swa_prefix_lens,
|
||||||
|
swa_prefix_lens_cpu,
|
||||||
|
swa_seq_lens,
|
||||||
|
swa_seq_lens_cpu,
|
||||||
|
swa_last_loc,
|
||||||
|
swa_tail_len,
|
||||||
|
num_new_pages=num_swa_pages,
|
||||||
|
)
|
||||||
|
assert alloc_swa_indices is not None
|
||||||
|
|
||||||
|
self.set_full_to_swa_mapping(
|
||||||
|
alloc_full_indices[-swa_tail_len:], alloc_swa_indices
|
||||||
|
)
|
||||||
|
if swa_tail_len < extend_num_tokens:
|
||||||
|
self.full_to_swa_index_mapping[
|
||||||
|
alloc_full_indices[:-swa_tail_len].to(torch.int64)
|
||||||
|
] = 0
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
def alloc_decode(
|
||||||
|
self,
|
||||||
|
seq_lens: torch.Tensor,
|
||||||
|
seq_lens_cpu: torch.Tensor,
|
||||||
|
last_loc: torch.Tensor, # last_loc for full layers
|
||||||
|
):
|
||||||
|
assert self.page_size > 1
|
||||||
|
swa_last_loc = self.translate_loc_from_full_to_swa(last_loc)
|
||||||
|
|
||||||
|
alloc_full_indices = self.full_attn_allocator.alloc_decode(
|
||||||
|
seq_lens, seq_lens_cpu, last_loc
|
||||||
|
)
|
||||||
|
alloc_swa_indices = self.swa_attn_allocator.alloc_decode(
|
||||||
|
seq_lens, seq_lens_cpu, swa_last_loc
|
||||||
|
)
|
||||||
|
|
||||||
|
if alloc_full_indices is None or alloc_swa_indices is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if _is_npu:
|
||||||
|
indices_2d = alloc_full_indices.to(torch.int64).unsqueeze(-1)
|
||||||
|
torch_npu.npu_scatter_nd_update_(
|
||||||
|
self.full_to_swa_index_mapping,
|
||||||
|
indices_2d,
|
||||||
|
alloc_swa_indices.to(torch.int64),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.full_to_swa_index_mapping[alloc_full_indices] = alloc_swa_indices
|
||||||
|
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
def free(self, free_index: torch.Tensor):
|
||||||
|
if free_index.numel() == 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
# NOTE: the API is not idempotent.
|
||||||
|
if self.is_not_in_free_group:
|
||||||
|
self.full_attn_allocator.free(free_index)
|
||||||
|
self.free_swa(free_index)
|
||||||
|
else:
|
||||||
|
self.free_group.append(self._copy_for_free_group(free_index))
|
||||||
|
assert (
|
||||||
|
self.full_attn_allocator.available_size() <= self.full_attn_allocator.size
|
||||||
|
)
|
||||||
|
assert self.swa_attn_allocator.available_size() <= self.swa_attn_allocator.size
|
||||||
|
|
||||||
|
def set_full_to_swa_mapping(
|
||||||
|
self, full_indices: torch.Tensor, swa_indices: torch.Tensor
|
||||||
|
) -> None:
|
||||||
|
"""Write full_to_swa_index_mapping[full_indices[i]] = swa_indices[i].
|
||||||
|
|
||||||
|
Used by HiCache load-back path to rebuild the mapping after FULL and SWA device alloc.
|
||||||
|
"""
|
||||||
|
if full_indices.numel() == 0:
|
||||||
|
return
|
||||||
|
assert full_indices.numel() == swa_indices.numel()
|
||||||
|
full_indices = full_indices.to(torch.int64)
|
||||||
|
swa_indices = swa_indices.to(self.full_to_swa_index_mapping.dtype)
|
||||||
|
self.full_to_swa_index_mapping[full_indices] = swa_indices
|
||||||
|
|
||||||
|
def clear_full_to_swa_mapping(self, full_indices: torch.Tensor) -> None:
|
||||||
|
if full_indices.numel() == 0:
|
||||||
|
return
|
||||||
|
full_indices = full_indices.to(torch.int64)
|
||||||
|
if _is_npu:
|
||||||
|
# NPU: aclnnIndexFill is unoptimized; direct assignment avoids the overhead.
|
||||||
|
self.full_to_swa_index_mapping[full_indices] = 0
|
||||||
|
else:
|
||||||
|
# CUDA: index_fill_ passes the 0 as a kernel argument; mapping[idx] = 0
|
||||||
|
# copies a host-resident scalar and blocks until the stream drains.
|
||||||
|
self.full_to_swa_index_mapping.index_fill_(0, full_indices, 0)
|
||||||
|
|
||||||
|
def free_swa(self, free_index: torch.Tensor):
|
||||||
|
if free_index.numel() == 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
if self.page_size == 1:
|
||||||
|
mapping_indices = free_index
|
||||||
|
else:
|
||||||
|
mapping_indices = self._expand_to_full_pages(free_index)
|
||||||
|
|
||||||
|
swa_indices = self.full_to_swa_index_mapping[mapping_indices]
|
||||||
|
swa_indices = swa_indices[swa_indices > 0]
|
||||||
|
self.clear_full_to_swa_mapping(mapping_indices)
|
||||||
|
|
||||||
|
if not self.is_not_in_free_group:
|
||||||
|
# Resolve ownership now. A cache action later in this group may
|
||||||
|
# install a new mapping for the same full index.
|
||||||
|
self.swa_free_group.append(swa_indices)
|
||||||
|
return
|
||||||
|
|
||||||
|
self.swa_attn_allocator.free(swa_indices)
|
||||||
|
|
||||||
|
def free_group_begin(self):
|
||||||
|
super().free_group_begin()
|
||||||
|
self.swa_free_group = []
|
||||||
|
|
||||||
|
def free_group_end(self):
|
||||||
|
super().free_group_end()
|
||||||
|
if self.swa_free_group:
|
||||||
|
swa_free_group = self.swa_free_group
|
||||||
|
self.swa_free_group = []
|
||||||
|
self.swa_attn_allocator.free(torch.cat(swa_free_group))
|
||||||
|
|
||||||
|
def _expand_to_full_pages(self, indices: torch.Tensor) -> torch.Tensor:
|
||||||
|
pages = torch.unique(indices // self.page_size)
|
||||||
|
page_offsets = torch.arange(
|
||||||
|
self.page_size, dtype=indices.dtype, device=indices.device
|
||||||
|
)
|
||||||
|
return (pages[:, None] * self.page_size + page_offsets[None, :]).reshape(-1)
|
||||||
|
|
||||||
|
def resize(self, config) -> None:
|
||||||
|
size_full = int(config.full_max_total_num_tokens)
|
||||||
|
size_swa = int(config.swa_max_total_num_tokens)
|
||||||
|
self._size_full = size_full
|
||||||
|
self._size_swa = size_swa
|
||||||
|
for alloc, sz in (
|
||||||
|
(self.full_attn_allocator, size_full),
|
||||||
|
(self.swa_attn_allocator, size_swa),
|
||||||
|
):
|
||||||
|
alloc.size = int(sz)
|
||||||
|
if self.page_size > 1:
|
||||||
|
alloc.num_pages = int(sz) // self.page_size
|
||||||
|
self.clear()
|
||||||
|
|
||||||
|
def clear(self):
|
||||||
|
self.swa_attn_allocator.clear()
|
||||||
|
self.full_attn_allocator.clear()
|
||||||
|
# Note: the last item is -1, we don't clear it, see the comment in __init__
|
||||||
|
self.full_to_swa_index_mapping[:-1].fill_(0)
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
self.free_group = []
|
||||||
|
self.swa_free_group = []
|
||||||
|
|
||||||
|
def get_cpu_copy(self, indices, mamba_indices=None):
|
||||||
|
return self._kvcache.get_cpu_copy(indices, mamba_indices=mamba_indices)
|
||||||
|
|
||||||
|
def load_cpu_copy(self, kv_cache_cpu, indices, mamba_indices=None):
|
||||||
|
return self._kvcache.load_cpu_copy(
|
||||||
|
kv_cache_cpu, indices, mamba_indices=mamba_indices
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class PureSWATokenToKVPoolAllocator(SWATokenToKVPoolAllocator):
|
||||||
|
"""Single-pool allocator for models whose every layer is sliding-window attention."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
size_swa: int,
|
||||||
|
page_size: int,
|
||||||
|
dtype: torch.dtype,
|
||||||
|
device: str,
|
||||||
|
kvcache: BaseSWAKVPool,
|
||||||
|
need_sort: bool,
|
||||||
|
):
|
||||||
|
assert page_size == 1
|
||||||
|
assert isinstance(kvcache, BaseSWAKVPool)
|
||||||
|
|
||||||
|
self.page_size = page_size
|
||||||
|
self.dtype = dtype
|
||||||
|
self.device = device
|
||||||
|
self.need_sort = need_sort
|
||||||
|
self._size_full = self._size_swa = size_swa
|
||||||
|
|
||||||
|
self.swa_attn_allocator = TokenToKVPoolAllocator(
|
||||||
|
size_swa,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
kvcache.swa_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
self.full_attn_allocator = self.swa_attn_allocator
|
||||||
|
|
||||||
|
self.full_to_swa_index_mapping = torch.cat(
|
||||||
|
[
|
||||||
|
torch.arange(size_swa + page_size, dtype=torch.int64, device=device),
|
||||||
|
torch.tensor([-1], dtype=torch.int64, device=device),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
self.free_pages = None
|
||||||
|
self.release_pages = None
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
self.free_group = []
|
||||||
|
|
||||||
|
self._kvcache = kvcache
|
||||||
|
self.swa_attn_allocator.clear()
|
||||||
|
self._kvcache.register_mapping(self.full_to_swa_index_mapping)
|
||||||
|
|
||||||
|
def available_size(self):
|
||||||
|
return self.swa_attn_allocator.available_size()
|
||||||
|
|
||||||
|
def full_available_size(self):
|
||||||
|
return self.swa_attn_allocator.available_size()
|
||||||
|
|
||||||
|
def swa_available_size(self):
|
||||||
|
return self.swa_attn_allocator.available_size()
|
||||||
|
|
||||||
|
def new_pages_available(self, num_full_pages: int, num_swa_pages: int) -> bool:
|
||||||
|
avail = self.swa_attn_allocator.available_size() // self.page_size
|
||||||
|
return num_full_pages <= avail and num_swa_pages <= avail
|
||||||
|
|
||||||
|
def translate_loc_from_full_to_swa(self, kv_indices: torch.Tensor):
|
||||||
|
return kv_indices
|
||||||
|
|
||||||
|
def alloc(self, need_size: int):
|
||||||
|
assert self.page_size == 1
|
||||||
|
return self.swa_attn_allocator.alloc(need_size)
|
||||||
|
|
||||||
|
def alloc_extend(self, *args, **kwargs):
|
||||||
|
raise NotImplementedError(
|
||||||
|
"PureSWATokenToKVPoolAllocator does not support page_size > 1."
|
||||||
|
)
|
||||||
|
|
||||||
|
def alloc_decode(self, *args, **kwargs):
|
||||||
|
raise NotImplementedError(
|
||||||
|
"PureSWATokenToKVPoolAllocator does not support page_size > 1."
|
||||||
|
)
|
||||||
|
|
||||||
|
def alloc_extend_swa_tail(self, *args, **kwargs):
|
||||||
|
raise NotImplementedError(
|
||||||
|
"PureSWATokenToKVPoolAllocator does not support page_size > 1."
|
||||||
|
)
|
||||||
|
|
||||||
|
def free(self, free_index: torch.Tensor):
|
||||||
|
if free_index.numel() == 0:
|
||||||
|
return
|
||||||
|
if self.is_not_in_free_group:
|
||||||
|
self.swa_attn_allocator.free(free_index[free_index > 0])
|
||||||
|
else:
|
||||||
|
self.free_group.append(self._copy_for_free_group(free_index))
|
||||||
|
assert self.swa_attn_allocator.available_size() <= self.swa_attn_allocator.size
|
||||||
|
|
||||||
|
def free_swa(self, free_index: torch.Tensor):
|
||||||
|
if free_index.numel() == 0:
|
||||||
|
return
|
||||||
|
if self.is_not_in_free_group:
|
||||||
|
self.swa_attn_allocator.free(free_index[free_index > 0])
|
||||||
|
else:
|
||||||
|
self.free_group.append(self._copy_for_free_group(free_index))
|
||||||
|
|
||||||
|
def free_group_begin(self):
|
||||||
|
self.is_not_in_free_group = False
|
||||||
|
self.free_group = []
|
||||||
|
|
||||||
|
def free_group_end(self):
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
if self.free_group:
|
||||||
|
self.free(torch.cat(self.free_group))
|
||||||
|
self.free_group = []
|
||||||
|
|
||||||
|
def clear(self):
|
||||||
|
self.swa_attn_allocator.clear()
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
self.free_group = []
|
||||||
@ -0,0 +1,530 @@
|
|||||||
|
import torch
|
||||||
|
|
||||||
|
from sglang.srt.mem_cache.allocator.base import BaseTokenToKVPoolAllocator
|
||||||
|
from sglang.srt.mem_cache.allocator.paged import PagedTokenToKVPoolAllocator
|
||||||
|
from sglang.srt.mem_cache.allocator.token import TokenToKVPoolAllocator
|
||||||
|
from sglang.srt.mem_cache.base_swa_memory_pool import BaseSWAKVPool
|
||||||
|
from sglang.srt.utils import is_npu
|
||||||
|
from sglang.srt.utils.common import get_num_new_pages
|
||||||
|
|
||||||
|
_is_npu = is_npu()
|
||||||
|
|
||||||
|
if _is_npu:
|
||||||
|
import torch_npu
|
||||||
|
|
||||||
|
from sglang.srt.hardware_backend.npu.allocator_npu import (
|
||||||
|
NPUPagedTokenToKVPoolAllocator,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SWATokenToKVPoolAllocator(BaseTokenToKVPoolAllocator):
|
||||||
|
"""Allocator for SWA hybrid KV cache."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
size: int,
|
||||||
|
size_swa: int,
|
||||||
|
page_size: int,
|
||||||
|
dtype: torch.dtype,
|
||||||
|
device: str,
|
||||||
|
kvcache: BaseSWAKVPool,
|
||||||
|
need_sort: bool,
|
||||||
|
):
|
||||||
|
assert isinstance(kvcache, BaseSWAKVPool)
|
||||||
|
self._size_full = size
|
||||||
|
self._size_swa = size_swa
|
||||||
|
self.dtype = dtype
|
||||||
|
self.device = device
|
||||||
|
self.page_size = page_size
|
||||||
|
|
||||||
|
full_kv_pool = getattr(kvcache, "full_kv_pool", None)
|
||||||
|
swa_kv_pool = getattr(kvcache, "swa_kv_pool", None)
|
||||||
|
|
||||||
|
if page_size == 1:
|
||||||
|
self.full_attn_allocator = TokenToKVPoolAllocator(
|
||||||
|
size,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
full_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
self.swa_attn_allocator = TokenToKVPoolAllocator(
|
||||||
|
size_swa,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
swa_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
if _is_npu:
|
||||||
|
PagedTokenToKVPoolAllocatorClass = NPUPagedTokenToKVPoolAllocator
|
||||||
|
else:
|
||||||
|
PagedTokenToKVPoolAllocatorClass = PagedTokenToKVPoolAllocator
|
||||||
|
self.full_attn_allocator = PagedTokenToKVPoolAllocatorClass(
|
||||||
|
size,
|
||||||
|
page_size,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
full_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
self.swa_attn_allocator = PagedTokenToKVPoolAllocatorClass(
|
||||||
|
size_swa,
|
||||||
|
page_size,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
swa_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
# Note: append one more item of value -1 in the end so -1 maps to -1.
|
||||||
|
# It is needed for the last_loc in alloc_extend, where the first full_last_loc
|
||||||
|
# is -1, and we need to map it to swa_last_loc -1 as well.
|
||||||
|
self.full_to_swa_index_mapping = torch.cat(
|
||||||
|
[
|
||||||
|
torch.zeros(
|
||||||
|
size + self.page_size,
|
||||||
|
dtype=torch.int64,
|
||||||
|
device=device,
|
||||||
|
),
|
||||||
|
torch.tensor([-1], dtype=torch.int64, device=device),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
self.need_sort = need_sort
|
||||||
|
self.free_pages = None
|
||||||
|
self.release_pages = None
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
self.free_group = []
|
||||||
|
self.swa_free_group = []
|
||||||
|
|
||||||
|
self._kvcache = kvcache
|
||||||
|
self.clear()
|
||||||
|
self._kvcache.register_mapping(self.full_to_swa_index_mapping)
|
||||||
|
|
||||||
|
def available_size(self):
|
||||||
|
return min(
|
||||||
|
self.full_attn_allocator.available_size(),
|
||||||
|
self.swa_attn_allocator.available_size(),
|
||||||
|
)
|
||||||
|
|
||||||
|
def full_available_size(self):
|
||||||
|
return self.full_attn_allocator.available_size()
|
||||||
|
|
||||||
|
def swa_available_size(self):
|
||||||
|
return self.swa_attn_allocator.available_size()
|
||||||
|
|
||||||
|
# Slot-conservation views for the leak invariant. On the non-shared allocator
|
||||||
|
# the static budget IS physical (conserve == physical); the shared composite
|
||||||
|
# overrides these with the static-cap view.
|
||||||
|
def _conserve_full_available_size(self):
|
||||||
|
return self.full_available_size()
|
||||||
|
|
||||||
|
def _conserve_swa_available_size(self):
|
||||||
|
return self.swa_available_size()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def size(self):
|
||||||
|
return min(self._size_full, self._size_swa)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def size_swa(self):
|
||||||
|
return self._size_swa
|
||||||
|
|
||||||
|
@property
|
||||||
|
def size_full(self):
|
||||||
|
return self._size_full
|
||||||
|
|
||||||
|
def debug_print(self) -> str:
|
||||||
|
msg = ""
|
||||||
|
msg += f"#swa-available-size: {self.swa_attn_allocator.available_size()}, "
|
||||||
|
msg += (
|
||||||
|
f"#full-attn-available-size: {self.full_attn_allocator.available_size()}, "
|
||||||
|
)
|
||||||
|
return msg
|
||||||
|
|
||||||
|
def get_kvcache(self):
|
||||||
|
return self._kvcache
|
||||||
|
|
||||||
|
def translate_loc_from_full_to_swa(self, kv_indices: torch.Tensor):
|
||||||
|
assert self._kvcache.full_to_swa_index_mapping is not None
|
||||||
|
return self._kvcache.translate_loc_from_full_to_swa(kv_indices)
|
||||||
|
|
||||||
|
def alloc(self, need_size: int):
|
||||||
|
assert self.page_size == 1
|
||||||
|
if need_size > self.full_attn_allocator.available_size():
|
||||||
|
return None
|
||||||
|
if need_size > self.swa_attn_allocator.available_size():
|
||||||
|
return None
|
||||||
|
|
||||||
|
alloc_full_indices = self.full_attn_allocator.alloc(need_size)
|
||||||
|
alloc_swa_indices = self.swa_attn_allocator.alloc(need_size)
|
||||||
|
assert alloc_full_indices is not None
|
||||||
|
assert alloc_swa_indices is not None
|
||||||
|
|
||||||
|
self.set_full_to_swa_mapping(alloc_full_indices, alloc_swa_indices)
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
def new_pages_available(self, num_full_pages: int, num_swa_pages: int) -> bool:
|
||||||
|
return (
|
||||||
|
num_full_pages
|
||||||
|
<= self.full_attn_allocator.available_size() // self.page_size
|
||||||
|
and num_swa_pages
|
||||||
|
<= self.swa_attn_allocator.available_size() // self.page_size
|
||||||
|
)
|
||||||
|
|
||||||
|
def alloc_extend(
|
||||||
|
self,
|
||||||
|
prefix_lens: torch.Tensor,
|
||||||
|
prefix_lens_cpu: torch.Tensor,
|
||||||
|
seq_lens: torch.Tensor,
|
||||||
|
seq_lens_cpu: torch.Tensor,
|
||||||
|
last_loc: torch.Tensor, # last_loc for full layers
|
||||||
|
extend_num_tokens: int,
|
||||||
|
):
|
||||||
|
assert self.page_size > 1
|
||||||
|
|
||||||
|
num_new_pages = get_num_new_pages(
|
||||||
|
seq_lens=seq_lens_cpu, page_size=self.page_size, prefix_lens=prefix_lens_cpu
|
||||||
|
)
|
||||||
|
if not self.new_pages_available(num_new_pages, num_new_pages):
|
||||||
|
return None
|
||||||
|
|
||||||
|
swa_last_loc = self.translate_loc_from_full_to_swa(last_loc)
|
||||||
|
|
||||||
|
alloc_full_indices = self.full_attn_allocator.alloc_extend(
|
||||||
|
prefix_lens,
|
||||||
|
prefix_lens_cpu,
|
||||||
|
seq_lens,
|
||||||
|
seq_lens_cpu,
|
||||||
|
last_loc,
|
||||||
|
extend_num_tokens,
|
||||||
|
num_new_pages=num_new_pages,
|
||||||
|
)
|
||||||
|
alloc_swa_indices = self.swa_attn_allocator.alloc_extend(
|
||||||
|
prefix_lens,
|
||||||
|
prefix_lens_cpu,
|
||||||
|
seq_lens,
|
||||||
|
seq_lens_cpu,
|
||||||
|
swa_last_loc,
|
||||||
|
extend_num_tokens,
|
||||||
|
num_new_pages=num_new_pages,
|
||||||
|
)
|
||||||
|
assert alloc_full_indices is not None
|
||||||
|
assert alloc_swa_indices is not None
|
||||||
|
|
||||||
|
self.set_full_to_swa_mapping(alloc_full_indices, alloc_swa_indices)
|
||||||
|
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
def alloc_extend_swa_tail(
|
||||||
|
self,
|
||||||
|
prefix_lens: torch.Tensor,
|
||||||
|
prefix_lens_cpu: torch.Tensor,
|
||||||
|
seq_lens: torch.Tensor,
|
||||||
|
seq_lens_cpu: torch.Tensor,
|
||||||
|
last_loc: torch.Tensor, # last_loc for full layers
|
||||||
|
extend_num_tokens: int,
|
||||||
|
swa_tail_len: int,
|
||||||
|
):
|
||||||
|
"""Allocate full KV for the whole extend and SWA KV only for the tail.
|
||||||
|
|
||||||
|
This is used by disaggregated decode preallocation: decode receives full
|
||||||
|
prompt KV for full-attention layers, but only the sliding-window state is
|
||||||
|
transferred for SWA layers.
|
||||||
|
"""
|
||||||
|
assert self.page_size > 1
|
||||||
|
assert len(seq_lens_cpu) == 1, "SWA tail allocation currently supports bs=1"
|
||||||
|
assert len(prefix_lens_cpu) == 1
|
||||||
|
assert 0 <= swa_tail_len <= extend_num_tokens
|
||||||
|
|
||||||
|
num_full_pages = get_num_new_pages(
|
||||||
|
seq_lens=seq_lens_cpu, page_size=self.page_size, prefix_lens=prefix_lens_cpu
|
||||||
|
)
|
||||||
|
num_swa_pages = (swa_tail_len + self.page_size - 1) // self.page_size
|
||||||
|
if not self.new_pages_available(num_full_pages, num_swa_pages):
|
||||||
|
return None
|
||||||
|
|
||||||
|
alloc_full_indices = self.full_attn_allocator.alloc_extend(
|
||||||
|
prefix_lens,
|
||||||
|
prefix_lens_cpu,
|
||||||
|
seq_lens,
|
||||||
|
seq_lens_cpu,
|
||||||
|
last_loc,
|
||||||
|
extend_num_tokens,
|
||||||
|
num_new_pages=num_full_pages,
|
||||||
|
)
|
||||||
|
assert alloc_full_indices is not None
|
||||||
|
|
||||||
|
if swa_tail_len == 0:
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
device = self.device
|
||||||
|
swa_prefix_lens = torch.zeros((1,), dtype=torch.int64, device=device)
|
||||||
|
swa_prefix_lens_cpu = torch.zeros((1,), dtype=torch.int64)
|
||||||
|
swa_seq_lens = torch.tensor([swa_tail_len], dtype=torch.int64, device=device)
|
||||||
|
swa_seq_lens_cpu = torch.tensor([swa_tail_len], dtype=torch.int64)
|
||||||
|
swa_last_loc = torch.tensor([-1], dtype=torch.int64, device=device)
|
||||||
|
|
||||||
|
alloc_swa_indices = self.swa_attn_allocator.alloc_extend(
|
||||||
|
swa_prefix_lens,
|
||||||
|
swa_prefix_lens_cpu,
|
||||||
|
swa_seq_lens,
|
||||||
|
swa_seq_lens_cpu,
|
||||||
|
swa_last_loc,
|
||||||
|
swa_tail_len,
|
||||||
|
num_new_pages=num_swa_pages,
|
||||||
|
)
|
||||||
|
assert alloc_swa_indices is not None
|
||||||
|
|
||||||
|
self.set_full_to_swa_mapping(
|
||||||
|
alloc_full_indices[-swa_tail_len:], alloc_swa_indices
|
||||||
|
)
|
||||||
|
if swa_tail_len < extend_num_tokens:
|
||||||
|
self.full_to_swa_index_mapping[
|
||||||
|
alloc_full_indices[:-swa_tail_len].to(torch.int64)
|
||||||
|
] = 0
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
def alloc_decode(
|
||||||
|
self,
|
||||||
|
seq_lens: torch.Tensor,
|
||||||
|
seq_lens_cpu: torch.Tensor,
|
||||||
|
last_loc: torch.Tensor, # last_loc for full layers
|
||||||
|
):
|
||||||
|
assert self.page_size > 1
|
||||||
|
swa_last_loc = self.translate_loc_from_full_to_swa(last_loc)
|
||||||
|
|
||||||
|
alloc_full_indices = self.full_attn_allocator.alloc_decode(
|
||||||
|
seq_lens, seq_lens_cpu, last_loc
|
||||||
|
)
|
||||||
|
alloc_swa_indices = self.swa_attn_allocator.alloc_decode(
|
||||||
|
seq_lens, seq_lens_cpu, swa_last_loc
|
||||||
|
)
|
||||||
|
|
||||||
|
if alloc_full_indices is None or alloc_swa_indices is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if _is_npu:
|
||||||
|
indices_2d = alloc_full_indices.to(torch.int64).unsqueeze(-1)
|
||||||
|
torch_npu.npu_scatter_nd_update_(
|
||||||
|
self.full_to_swa_index_mapping,
|
||||||
|
indices_2d,
|
||||||
|
alloc_swa_indices.to(torch.int64),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.full_to_swa_index_mapping[alloc_full_indices] = alloc_swa_indices
|
||||||
|
|
||||||
|
return alloc_full_indices
|
||||||
|
|
||||||
|
def free(self, free_index: torch.Tensor):
|
||||||
|
if free_index.numel() == 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
# NOTE: the API is not idempotent.
|
||||||
|
if self.is_not_in_free_group:
|
||||||
|
self.full_attn_allocator.free(free_index)
|
||||||
|
self.free_swa(free_index)
|
||||||
|
else:
|
||||||
|
self.free_group.append(self._copy_for_free_group(free_index))
|
||||||
|
assert (
|
||||||
|
self.full_attn_allocator.available_size() <= self.full_attn_allocator.size
|
||||||
|
)
|
||||||
|
assert self.swa_attn_allocator.available_size() <= self.swa_attn_allocator.size
|
||||||
|
|
||||||
|
def set_full_to_swa_mapping(
|
||||||
|
self, full_indices: torch.Tensor, swa_indices: torch.Tensor
|
||||||
|
) -> None:
|
||||||
|
"""Write full_to_swa_index_mapping[full_indices[i]] = swa_indices[i].
|
||||||
|
|
||||||
|
Used by HiCache load-back path to rebuild the mapping after FULL and SWA device alloc.
|
||||||
|
"""
|
||||||
|
if full_indices.numel() == 0:
|
||||||
|
return
|
||||||
|
assert full_indices.numel() == swa_indices.numel()
|
||||||
|
full_indices = full_indices.to(torch.int64)
|
||||||
|
swa_indices = swa_indices.to(self.full_to_swa_index_mapping.dtype)
|
||||||
|
self.full_to_swa_index_mapping[full_indices] = swa_indices
|
||||||
|
|
||||||
|
def free_swa(self, free_index: torch.Tensor):
|
||||||
|
if free_index.numel() == 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
if self.page_size == 1:
|
||||||
|
mapping_indices = free_index
|
||||||
|
else:
|
||||||
|
mapping_indices = self._expand_to_full_pages(free_index)
|
||||||
|
|
||||||
|
swa_indices = self.full_to_swa_index_mapping[mapping_indices]
|
||||||
|
swa_indices = swa_indices[swa_indices > 0]
|
||||||
|
self.clear_full_to_swa_mapping(mapping_indices)
|
||||||
|
|
||||||
|
if not self.is_not_in_free_group:
|
||||||
|
# Resolve ownership now. A cache action later in this group may
|
||||||
|
# install a new mapping for the same full index.
|
||||||
|
self.swa_free_group.append(swa_indices)
|
||||||
|
return
|
||||||
|
|
||||||
|
self.swa_attn_allocator.free(swa_indices)
|
||||||
|
|
||||||
|
def free_group_begin(self):
|
||||||
|
super().free_group_begin()
|
||||||
|
self.swa_free_group = []
|
||||||
|
|
||||||
|
def free_group_end(self):
|
||||||
|
super().free_group_end()
|
||||||
|
if self.swa_free_group:
|
||||||
|
swa_free_group = self.swa_free_group
|
||||||
|
self.swa_free_group = []
|
||||||
|
self.swa_attn_allocator.free(torch.cat(swa_free_group))
|
||||||
|
|
||||||
|
def _expand_to_full_pages(self, indices: torch.Tensor) -> torch.Tensor:
|
||||||
|
pages = torch.unique(indices // self.page_size)
|
||||||
|
page_offsets = torch.arange(
|
||||||
|
self.page_size, dtype=indices.dtype, device=indices.device
|
||||||
|
)
|
||||||
|
return (pages[:, None] * self.page_size + page_offsets[None, :]).reshape(-1)
|
||||||
|
|
||||||
|
def resize(self, config) -> None:
|
||||||
|
size_full = int(config.full_max_total_num_tokens)
|
||||||
|
size_swa = int(config.swa_max_total_num_tokens)
|
||||||
|
self._size_full = size_full
|
||||||
|
self._size_swa = size_swa
|
||||||
|
for alloc, sz in (
|
||||||
|
(self.full_attn_allocator, size_full),
|
||||||
|
(self.swa_attn_allocator, size_swa),
|
||||||
|
):
|
||||||
|
alloc.size = int(sz)
|
||||||
|
if self.page_size > 1:
|
||||||
|
alloc.num_pages = int(sz) // self.page_size
|
||||||
|
self.clear()
|
||||||
|
|
||||||
|
def clear(self):
|
||||||
|
self.swa_attn_allocator.clear()
|
||||||
|
self.full_attn_allocator.clear()
|
||||||
|
# Note: the last item is -1, we don't clear it, see the comment in __init__
|
||||||
|
self.full_to_swa_index_mapping[:-1].fill_(0)
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
self.free_group = []
|
||||||
|
self.swa_free_group = []
|
||||||
|
|
||||||
|
def get_cpu_copy(self, indices, mamba_indices=None):
|
||||||
|
return self._kvcache.get_cpu_copy(indices, mamba_indices=mamba_indices)
|
||||||
|
|
||||||
|
def load_cpu_copy(self, kv_cache_cpu, indices, mamba_indices=None):
|
||||||
|
return self._kvcache.load_cpu_copy(
|
||||||
|
kv_cache_cpu, indices, mamba_indices=mamba_indices
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class PureSWATokenToKVPoolAllocator(SWATokenToKVPoolAllocator):
|
||||||
|
"""Single-pool allocator for models whose every layer is sliding-window attention."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
size_swa: int,
|
||||||
|
page_size: int,
|
||||||
|
dtype: torch.dtype,
|
||||||
|
device: str,
|
||||||
|
kvcache: BaseSWAKVPool,
|
||||||
|
need_sort: bool,
|
||||||
|
):
|
||||||
|
assert page_size == 1
|
||||||
|
assert isinstance(kvcache, BaseSWAKVPool)
|
||||||
|
|
||||||
|
self.page_size = page_size
|
||||||
|
self.dtype = dtype
|
||||||
|
self.device = device
|
||||||
|
self.need_sort = need_sort
|
||||||
|
self._size_full = self._size_swa = size_swa
|
||||||
|
|
||||||
|
self.swa_attn_allocator = TokenToKVPoolAllocator(
|
||||||
|
size_swa,
|
||||||
|
dtype,
|
||||||
|
device,
|
||||||
|
kvcache.swa_kv_pool,
|
||||||
|
need_sort,
|
||||||
|
)
|
||||||
|
self.full_attn_allocator = self.swa_attn_allocator
|
||||||
|
|
||||||
|
self.full_to_swa_index_mapping = torch.cat(
|
||||||
|
[
|
||||||
|
torch.arange(size_swa + page_size, dtype=torch.int64, device=device),
|
||||||
|
torch.tensor([-1], dtype=torch.int64, device=device),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
self.free_pages = None
|
||||||
|
self.release_pages = None
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
self.free_group = []
|
||||||
|
|
||||||
|
self._kvcache = kvcache
|
||||||
|
self.swa_attn_allocator.clear()
|
||||||
|
self._kvcache.register_mapping(self.full_to_swa_index_mapping)
|
||||||
|
|
||||||
|
def available_size(self):
|
||||||
|
return self.swa_attn_allocator.available_size()
|
||||||
|
|
||||||
|
def full_available_size(self):
|
||||||
|
return self.swa_attn_allocator.available_size()
|
||||||
|
|
||||||
|
def swa_available_size(self):
|
||||||
|
return self.swa_attn_allocator.available_size()
|
||||||
|
|
||||||
|
def new_pages_available(self, num_full_pages: int, num_swa_pages: int) -> bool:
|
||||||
|
avail = self.swa_attn_allocator.available_size() // self.page_size
|
||||||
|
return num_full_pages <= avail and num_swa_pages <= avail
|
||||||
|
|
||||||
|
def translate_loc_from_full_to_swa(self, kv_indices: torch.Tensor):
|
||||||
|
return kv_indices
|
||||||
|
|
||||||
|
def alloc(self, need_size: int):
|
||||||
|
assert self.page_size == 1
|
||||||
|
return self.swa_attn_allocator.alloc(need_size)
|
||||||
|
|
||||||
|
def alloc_extend(self, *args, **kwargs):
|
||||||
|
raise NotImplementedError(
|
||||||
|
"PureSWATokenToKVPoolAllocator does not support page_size > 1."
|
||||||
|
)
|
||||||
|
|
||||||
|
def alloc_decode(self, *args, **kwargs):
|
||||||
|
raise NotImplementedError(
|
||||||
|
"PureSWATokenToKVPoolAllocator does not support page_size > 1."
|
||||||
|
)
|
||||||
|
|
||||||
|
def alloc_extend_swa_tail(self, *args, **kwargs):
|
||||||
|
raise NotImplementedError(
|
||||||
|
"PureSWATokenToKVPoolAllocator does not support page_size > 1."
|
||||||
|
)
|
||||||
|
|
||||||
|
def free(self, free_index: torch.Tensor):
|
||||||
|
if free_index.numel() == 0:
|
||||||
|
return
|
||||||
|
if self.is_not_in_free_group:
|
||||||
|
self.swa_attn_allocator.free(free_index[free_index > 0])
|
||||||
|
else:
|
||||||
|
self.free_group.append(self._copy_for_free_group(free_index))
|
||||||
|
assert self.swa_attn_allocator.available_size() <= self.swa_attn_allocator.size
|
||||||
|
|
||||||
|
def free_swa(self, free_index: torch.Tensor):
|
||||||
|
if free_index.numel() == 0:
|
||||||
|
return
|
||||||
|
if self.is_not_in_free_group:
|
||||||
|
self.swa_attn_allocator.free(free_index[free_index > 0])
|
||||||
|
else:
|
||||||
|
self.free_group.append(self._copy_for_free_group(free_index))
|
||||||
|
|
||||||
|
def free_group_begin(self):
|
||||||
|
self.is_not_in_free_group = False
|
||||||
|
self.free_group = []
|
||||||
|
|
||||||
|
def free_group_end(self):
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
if self.free_group:
|
||||||
|
self.free(torch.cat(self.free_group))
|
||||||
|
self.free_group = []
|
||||||
|
|
||||||
|
def clear(self):
|
||||||
|
self.swa_attn_allocator.clear()
|
||||||
|
self.is_not_in_free_group = True
|
||||||
|
self.free_group = []
|
||||||
@ -0,0 +1,231 @@
|
|||||||
|
[2026-09-10T02:02:32+00:00] starting profile=dsv4-low-latency model=dsv4-flash-0731
|
||||||
|
[2026-09-10T02:02:34+00:00] waiting for server profile=dsv4-low-latency elapsed=0s
|
||||||
|
[2026-09-10T02:03:34+00:00] waiting for server profile=dsv4-low-latency elapsed=60s
|
||||||
|
[2026-09-10T02:04:35+00:00] waiting for server profile=dsv4-low-latency elapsed=120s
|
||||||
|
[2026-09-10T02:05:35+00:00] waiting for server profile=dsv4-low-latency elapsed=180s
|
||||||
|
[2026-09-10T02:06:36+00:00] waiting for server profile=dsv4-low-latency elapsed=240s
|
||||||
|
[2026-09-10T02:07:36+00:00] waiting for server profile=dsv4-low-latency elapsed=300s
|
||||||
|
[2026-09-10T02:08:37+00:00] waiting for server profile=dsv4-low-latency elapsed=360s
|
||||||
|
[2026-09-10T02:09:38+00:00] waiting for server profile=dsv4-low-latency elapsed=420s
|
||||||
|
[2026-09-10T02:10:38+00:00] waiting for server profile=dsv4-low-latency elapsed=480s
|
||||||
|
[2026-09-10T02:11:39+00:00] waiting for server profile=dsv4-low-latency elapsed=540s
|
||||||
|
[2026-09-10T02:12:39+00:00] waiting for server profile=dsv4-low-latency elapsed=600s
|
||||||
|
[2026-09-10T02:13:40+00:00] waiting for server profile=dsv4-low-latency elapsed=660s
|
||||||
|
[2026-09-10T02:14:40+00:00] waiting for server profile=dsv4-low-latency elapsed=720s
|
||||||
|
[2026-09-10T02:15:41+00:00] waiting for server profile=dsv4-low-latency elapsed=780s
|
||||||
|
[2026-09-10T02:16:41+00:00] waiting for server profile=dsv4-low-latency elapsed=840s
|
||||||
|
[2026-09-10T02:17:42+00:00] waiting for server profile=dsv4-low-latency elapsed=900s
|
||||||
|
[2026-09-10T02:18:43+00:00] waiting for server profile=dsv4-low-latency elapsed=960s
|
||||||
|
[2026-09-10T02:19:43+00:00] waiting for server profile=dsv4-low-latency elapsed=1020s
|
||||||
|
[2026-09-10T02:20:39+00:00] server healthy profile=dsv4-low-latency elapsed=1070s
|
||||||
|
[2026-09-10T02:20:55+00:00] run profile=dsv4-low-latency shape=16k_512 isl=16384 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-10T02:22:34+00:00] pass profile=dsv4-low-latency point=16k_512_c1
|
||||||
|
[2026-09-10T02:22:35+00:00] run profile=dsv4-low-latency shape=16k_512 isl=16384 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-10T02:23:19+00:00] pass profile=dsv4-low-latency point=16k_512_c8
|
||||||
|
[2026-09-10T02:23:20+00:00] run profile=dsv4-low-latency shape=16k_512 isl=16384 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-10T02:24:27+00:00] pass profile=dsv4-low-latency point=16k_512_c32
|
||||||
|
[2026-09-10T02:24:28+00:00] run profile=dsv4-low-latency shape=16k_512 isl=16384 osl=512 c=64 prompts=320
|
||||||
|
[2026-09-10T02:26:17+00:00] pass profile=dsv4-low-latency point=16k_512_c64
|
||||||
|
[2026-09-10T02:26:18+00:00] run profile=dsv4-low-latency shape=16k_512 isl=16384 osl=512 c=128 prompts=640
|
||||||
|
[2026-09-10T02:29:34+00:00] pass profile=dsv4-low-latency point=16k_512_c128
|
||||||
|
[2026-09-10T02:29:35+00:00] run profile=dsv4-low-latency shape=16k_512 isl=16384 osl=512 c=256 prompts=1280
|
||||||
|
[2026-09-10T02:35:44+00:00] pass profile=dsv4-low-latency point=16k_512_c256
|
||||||
|
[2026-09-10T02:35:45+00:00] run profile=dsv4-low-latency shape=16k_1 isl=16384 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-10T02:36:16+00:00] pass profile=dsv4-low-latency point=16k_1_c1
|
||||||
|
[2026-09-10T02:36:17+00:00] run profile=dsv4-low-latency shape=16k_1 isl=16384 osl=1 c=8 prompts=64
|
||||||
|
[2026-09-10T02:36:46+00:00] pass profile=dsv4-low-latency point=16k_1_c8
|
||||||
|
[2026-09-10T02:36:47+00:00] run profile=dsv4-low-latency shape=16k_1 isl=16384 osl=1 c=32 prompts=160
|
||||||
|
[2026-09-10T02:37:40+00:00] warning profile=dsv4-low-latency point=16k_1_c32 zero_ttft_count=2
|
||||||
|
[2026-09-10T02:37:40+00:00] pass profile=dsv4-low-latency point=16k_1_c32
|
||||||
|
[2026-09-10T02:37:41+00:00] run profile=dsv4-low-latency shape=16k_1 isl=16384 osl=1 c=64 prompts=320
|
||||||
|
[2026-09-10T02:39:14+00:00] warning profile=dsv4-low-latency point=16k_1_c64 zero_ttft_count=2
|
||||||
|
[2026-09-10T02:39:14+00:00] pass profile=dsv4-low-latency point=16k_1_c64
|
||||||
|
[2026-09-10T02:39:16+00:00] run profile=dsv4-low-latency shape=16k_1 isl=16384 osl=1 c=128 prompts=640
|
||||||
|
[2026-09-10T02:42:10+00:00] warning profile=dsv4-low-latency point=16k_1_c128 zero_ttft_count=4
|
||||||
|
[2026-09-10T02:42:10+00:00] pass profile=dsv4-low-latency point=16k_1_c128
|
||||||
|
[2026-09-10T02:42:11+00:00] run profile=dsv4-low-latency shape=16k_1 isl=16384 osl=1 c=256 prompts=1280
|
||||||
|
[2026-09-10T02:47:47+00:00] warning profile=dsv4-low-latency point=16k_1_c256 zero_ttft_count=7
|
||||||
|
[2026-09-10T02:47:47+00:00] pass profile=dsv4-low-latency point=16k_1_c256
|
||||||
|
[2026-09-10T02:47:48+00:00] run profile=dsv4-low-latency shape=1k_128 isl=1024 osl=128 c=1 prompts=64
|
||||||
|
[2026-09-10T02:48:36+00:00] pass profile=dsv4-low-latency point=1k_128_c1
|
||||||
|
[2026-09-10T02:48:37+00:00] run profile=dsv4-low-latency shape=1k_128 isl=1024 osl=128 c=8 prompts=64
|
||||||
|
[2026-09-10T02:49:08+00:00] pass profile=dsv4-low-latency point=1k_128_c8
|
||||||
|
[2026-09-10T02:49:09+00:00] run profile=dsv4-low-latency shape=1k_128 isl=1024 osl=128 c=32 prompts=160
|
||||||
|
[2026-09-10T02:49:45+00:00] pass profile=dsv4-low-latency point=1k_128_c32
|
||||||
|
[2026-09-10T02:49:46+00:00] run profile=dsv4-low-latency shape=1k_128 isl=1024 osl=128 c=64 prompts=320
|
||||||
|
[2026-09-10T02:50:28+00:00] pass profile=dsv4-low-latency point=1k_128_c64
|
||||||
|
[2026-09-10T02:50:29+00:00] run profile=dsv4-low-latency shape=1k_128 isl=1024 osl=128 c=128 prompts=640
|
||||||
|
[2026-09-10T02:51:13+00:00] pass profile=dsv4-low-latency point=1k_128_c128
|
||||||
|
[2026-09-10T02:51:14+00:00] run profile=dsv4-low-latency shape=1k_128 isl=1024 osl=128 c=256 prompts=1280
|
||||||
|
[2026-09-10T02:52:12+00:00] pass profile=dsv4-low-latency point=1k_128_c256
|
||||||
|
[2026-09-10T02:52:13+00:00] run profile=dsv4-low-latency shape=1k_4k isl=1024 osl=4096 c=1 prompts=64
|
||||||
|
[2026-09-10T02:59:40+00:00] pass profile=dsv4-low-latency point=1k_4k_c1
|
||||||
|
[2026-09-10T02:59:41+00:00] run profile=dsv4-low-latency shape=1k_4k isl=1024 osl=4096 c=8 prompts=64
|
||||||
|
[2026-09-10T03:01:28+00:00] pass profile=dsv4-low-latency point=1k_4k_c8
|
||||||
|
[2026-09-10T03:01:29+00:00] run profile=dsv4-low-latency shape=1k_4k isl=1024 osl=4096 c=32 prompts=160
|
||||||
|
[2026-09-10T03:03:21+00:00] pass profile=dsv4-low-latency point=1k_4k_c32
|
||||||
|
[2026-09-10T03:03:22+00:00] run profile=dsv4-low-latency shape=1k_4k isl=1024 osl=4096 c=64 prompts=320
|
||||||
|
[2026-09-10T03:05:53+00:00] pass profile=dsv4-low-latency point=1k_4k_c64
|
||||||
|
[2026-09-10T03:05:54+00:00] run profile=dsv4-low-latency shape=1k_4k isl=1024 osl=4096 c=128 prompts=640
|
||||||
|
[2026-09-10T03:09:39+00:00] pass profile=dsv4-low-latency point=1k_4k_c128
|
||||||
|
[2026-09-10T03:09:40+00:00] run profile=dsv4-low-latency shape=1k_4k isl=1024 osl=4096 c=256 prompts=1280
|
||||||
|
[2026-09-10T03:14:18+00:00] pass profile=dsv4-low-latency point=1k_4k_c256
|
||||||
|
[2026-09-10T03:14:19+00:00] run profile=dsv4-low-latency shape=64k_1 isl=65536 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-10T03:15:49+00:00] warning profile=dsv4-low-latency point=64k_1_c1 zero_ttft_count=1
|
||||||
|
[2026-09-10T03:15:49+00:00] pass profile=dsv4-low-latency point=64k_1_c1
|
||||||
|
[2026-09-10T03:15:50+00:00] run profile=dsv4-low-latency shape=64k_1 isl=65536 osl=1 c=4 prompts=64
|
||||||
|
[2026-09-10T03:17:15+00:00] warning profile=dsv4-low-latency point=64k_1_c4 zero_ttft_count=1
|
||||||
|
[2026-09-10T03:17:15+00:00] pass profile=dsv4-low-latency point=64k_1_c4
|
||||||
|
[2026-09-10T03:17:16+00:00] run profile=dsv4-low-latency shape=64k_1 isl=65536 osl=1 c=8 prompts=64
|
||||||
|
[2026-09-10T03:18:42+00:00] warning profile=dsv4-low-latency point=64k_1_c8 zero_ttft_count=1
|
||||||
|
[2026-09-10T03:18:42+00:00] pass profile=dsv4-low-latency point=64k_1_c8
|
||||||
|
[2026-09-10T03:18:43+00:00] run profile=dsv4-low-latency shape=64k_1 isl=65536 osl=1 c=16 prompts=80
|
||||||
|
[2026-09-10T03:20:27+00:00] warning profile=dsv4-low-latency point=64k_1_c16 zero_ttft_count=1
|
||||||
|
[2026-09-10T03:20:27+00:00] pass profile=dsv4-low-latency point=64k_1_c16
|
||||||
|
[2026-09-10T03:20:28+00:00] run profile=dsv4-low-latency shape=64k_1 isl=65536 osl=1 c=32 prompts=160
|
||||||
|
[2026-09-10T03:23:43+00:00] warning profile=dsv4-low-latency point=64k_1_c32 zero_ttft_count=1
|
||||||
|
[2026-09-10T03:23:43+00:00] pass profile=dsv4-low-latency point=64k_1_c32
|
||||||
|
[2026-09-10T03:23:44+00:00] run profile=dsv4-low-latency shape=64k_1 isl=65536 osl=1 c=64 prompts=320
|
||||||
|
[2026-09-10T03:30:02+00:00] warning profile=dsv4-low-latency point=64k_1_c64 zero_ttft_count=3
|
||||||
|
[2026-09-10T03:30:02+00:00] pass profile=dsv4-low-latency point=64k_1_c64
|
||||||
|
[2026-09-10T03:30:03+00:00] run profile=dsv4-low-latency shape=64k_1 isl=65536 osl=1 c=128 prompts=640
|
||||||
|
[2026-09-10T03:42:28+00:00] warning profile=dsv4-low-latency point=64k_1_c128 zero_ttft_count=5
|
||||||
|
[2026-09-10T03:42:28+00:00] pass profile=dsv4-low-latency point=64k_1_c128
|
||||||
|
[2026-09-10T03:42:29+00:00] run profile=dsv4-low-latency shape=64k_512 isl=65536 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-10T03:44:52+00:00] pass profile=dsv4-low-latency point=64k_512_c1
|
||||||
|
[2026-09-10T03:44:53+00:00] run profile=dsv4-low-latency shape=64k_512 isl=65536 osl=512 c=4 prompts=64
|
||||||
|
[2026-09-10T03:46:37+00:00] pass profile=dsv4-low-latency point=64k_512_c4
|
||||||
|
[2026-09-10T03:46:38+00:00] run profile=dsv4-low-latency shape=64k_512 isl=65536 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-10T03:48:14+00:00] pass profile=dsv4-low-latency point=64k_512_c8
|
||||||
|
[2026-09-10T03:48:15+00:00] run profile=dsv4-low-latency shape=64k_512 isl=65536 osl=512 c=16 prompts=80
|
||||||
|
[2026-09-10T03:50:06+00:00] pass profile=dsv4-low-latency point=64k_512_c16
|
||||||
|
[2026-09-10T03:50:07+00:00] run profile=dsv4-low-latency shape=64k_512 isl=65536 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-10T03:53:33+00:00] pass profile=dsv4-low-latency point=64k_512_c32
|
||||||
|
[2026-09-10T03:53:34+00:00] run profile=dsv4-low-latency shape=64k_512 isl=65536 osl=512 c=64 prompts=320
|
||||||
|
[2026-09-10T04:00:06+00:00] pass profile=dsv4-low-latency point=64k_512_c64
|
||||||
|
[2026-09-10T04:00:07+00:00] run profile=dsv4-low-latency shape=128k_1 isl=131072 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-10T04:03:11+00:00] pass profile=dsv4-low-latency point=128k_1_c1
|
||||||
|
[2026-09-10T04:03:12+00:00] run profile=dsv4-low-latency shape=128k_1 isl=131072 osl=1 c=2 prompts=64
|
||||||
|
[2026-09-10T04:06:11+00:00] pass profile=dsv4-low-latency point=128k_1_c2
|
||||||
|
[2026-09-10T04:06:12+00:00] run profile=dsv4-low-latency shape=128k_1 isl=131072 osl=1 c=4 prompts=64
|
||||||
|
[2026-09-10T04:09:12+00:00] pass profile=dsv4-low-latency point=128k_1_c4
|
||||||
|
[2026-09-10T04:09:13+00:00] run profile=dsv4-low-latency shape=128k_1 isl=131072 osl=1 c=8 prompts=64
|
||||||
|
[2026-09-10T04:12:12+00:00] pass profile=dsv4-low-latency point=128k_1_c8
|
||||||
|
[2026-09-10T04:12:13+00:00] run profile=dsv4-low-latency shape=128k_1 isl=131072 osl=1 c=16 prompts=80
|
||||||
|
[2026-09-10T04:15:55+00:00] pass profile=dsv4-low-latency point=128k_1_c16
|
||||||
|
[2026-09-10T04:15:56+00:00] run profile=dsv4-low-latency shape=128k_1 isl=131072 osl=1 c=32 prompts=160
|
||||||
|
[2026-09-10T04:23:05+00:00] pass profile=dsv4-low-latency point=128k_1_c32
|
||||||
|
[2026-09-10T04:23:06+00:00] run profile=dsv4-low-latency shape=128k_1 isl=131072 osl=1 c=64 prompts=320
|
||||||
|
[2026-09-10T04:37:16+00:00] warning profile=dsv4-low-latency point=128k_1_c64 zero_ttft_count=2
|
||||||
|
[2026-09-10T04:37:16+00:00] pass profile=dsv4-low-latency point=128k_1_c64
|
||||||
|
[2026-09-10T04:37:17+00:00] run profile=dsv4-low-latency shape=128k_512 isl=131072 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-10T04:42:26+00:00] pass profile=dsv4-low-latency point=128k_512_c1
|
||||||
|
[2026-09-10T04:42:27+00:00] run profile=dsv4-low-latency shape=128k_512 isl=131072 osl=512 c=2 prompts=64
|
||||||
|
[2026-09-10T04:46:39+00:00] pass profile=dsv4-low-latency point=128k_512_c2
|
||||||
|
[2026-09-10T04:46:40+00:00] run profile=dsv4-low-latency shape=128k_512 isl=131072 osl=512 c=4 prompts=64
|
||||||
|
[2026-09-10T04:50:19+00:00] pass profile=dsv4-low-latency point=128k_512_c4
|
||||||
|
[2026-09-10T04:50:21+00:00] run profile=dsv4-low-latency shape=128k_512 isl=131072 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-10T04:53:45+00:00] pass profile=dsv4-low-latency point=128k_512_c8
|
||||||
|
[2026-09-10T04:53:46+00:00] run profile=dsv4-low-latency shape=128k_512 isl=131072 osl=512 c=16 prompts=80
|
||||||
|
[2026-09-10T04:57:47+00:00] pass profile=dsv4-low-latency point=128k_512_c16
|
||||||
|
[2026-09-10T04:57:48+00:00] run profile=dsv4-low-latency shape=128k_512 isl=131072 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-10T05:05:25+00:00] pass profile=dsv4-low-latency point=128k_512_c32
|
||||||
|
[2026-09-10T05:05:26+00:00] run profile=dsv4-low-latency shape=256k_1 isl=262144 osl=1 c=1 prompts=16
|
||||||
|
[2026-09-10T05:07:29+00:00] pass profile=dsv4-low-latency point=256k_1_c1
|
||||||
|
[2026-09-10T05:07:31+00:00] run profile=dsv4-low-latency shape=256k_1 isl=262144 osl=1 c=2 prompts=16
|
||||||
|
[2026-09-10T05:09:29+00:00] pass profile=dsv4-low-latency point=256k_1_c2
|
||||||
|
[2026-09-10T05:09:30+00:00] run profile=dsv4-low-latency shape=256k_1 isl=262144 osl=1 c=4 prompts=16
|
||||||
|
[2026-09-10T05:11:28+00:00] pass profile=dsv4-low-latency point=256k_1_c4
|
||||||
|
[2026-09-10T05:11:29+00:00] run profile=dsv4-low-latency shape=512k_1 isl=524288 osl=1 c=1 prompts=8
|
||||||
|
[2026-09-10T05:12:50+00:00] failed profile=dsv4-low-latency point=512k_1_c1 rc=1
|
||||||
|
[2026-09-10T05:12:50+00:00] capacity short-circuit armed profile=dsv4-low-latency shape=512k_1 after c=1
|
||||||
|
[2026-09-10T05:12:50+00:00] server failure aborts profile=dsv4-low-latency shape=512k_1
|
||||||
|
[2026-09-10T05:12:50+00:00] ERROR: unhealthy before profile=dsv4-low-latency point=1m_boundary_1_c1
|
||||||
|
[2026-09-10T05:12:50+00:00] server failure aborts profile=dsv4-low-latency shape=1m_boundary_1
|
||||||
|
[2026-09-10T05:13:22+00:00] starting profile=dsv4-balanced model=dsv4-flash-0731
|
||||||
|
[2026-09-10T05:13:23+00:00] waiting for server profile=dsv4-balanced elapsed=0s
|
||||||
|
[2026-09-10T05:14:24+00:00] waiting for server profile=dsv4-balanced elapsed=60s
|
||||||
|
[2026-09-10T05:15:25+00:00] waiting for server profile=dsv4-balanced elapsed=120s
|
||||||
|
[2026-09-10T05:16:25+00:00] waiting for server profile=dsv4-balanced elapsed=180s
|
||||||
|
[2026-09-10T05:17:26+00:00] waiting for server profile=dsv4-balanced elapsed=240s
|
||||||
|
[2026-09-10T05:18:26+00:00] waiting for server profile=dsv4-balanced elapsed=300s
|
||||||
|
[2026-09-10T05:19:27+00:00] waiting for server profile=dsv4-balanced elapsed=360s
|
||||||
|
[2026-09-10T05:20:27+00:00] waiting for server profile=dsv4-balanced elapsed=420s
|
||||||
|
[2026-09-10T05:21:28+00:00] waiting for server profile=dsv4-balanced elapsed=480s
|
||||||
|
[2026-09-10T05:22:29+00:00] waiting for server profile=dsv4-balanced elapsed=540s
|
||||||
|
[2026-09-10T05:23:00+00:00] server healthy profile=dsv4-balanced elapsed=570s
|
||||||
|
[2026-09-10T05:23:18+00:00] run profile=dsv4-balanced shape=16k_512 isl=16384 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-10T05:30:32+00:00] pass profile=dsv4-balanced point=16k_512_c1
|
||||||
|
[2026-09-10T05:30:33+00:00] run profile=dsv4-balanced shape=16k_512 isl=16384 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-10T05:31:50+00:00] pass profile=dsv4-balanced point=16k_512_c8
|
||||||
|
[2026-09-10T05:31:51+00:00] run profile=dsv4-balanced shape=16k_512 isl=16384 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-10T05:33:19+00:00] pass profile=dsv4-balanced point=16k_512_c32
|
||||||
|
[2026-09-10T05:33:20+00:00] run profile=dsv4-balanced shape=16k_512 isl=16384 osl=512 c=64 prompts=320
|
||||||
|
[2026-09-10T05:35:32+00:00] pass profile=dsv4-balanced point=16k_512_c64
|
||||||
|
[2026-09-10T05:35:33+00:00] run profile=dsv4-balanced shape=16k_512 isl=16384 osl=512 c=128 prompts=640
|
||||||
|
[2026-09-10T05:39:14+00:00] pass profile=dsv4-balanced point=16k_512_c128
|
||||||
|
[2026-09-10T05:39:15+00:00] run profile=dsv4-balanced shape=16k_512 isl=16384 osl=512 c=256 prompts=1280
|
||||||
|
[2026-09-10T05:45:44+00:00] pass profile=dsv4-balanced point=16k_512_c256
|
||||||
|
[2026-09-10T05:45:45+00:00] run profile=dsv4-balanced shape=16k_1 isl=16384 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-10T05:47:02+00:00] pass profile=dsv4-balanced point=16k_1_c1
|
||||||
|
[2026-09-10T05:47:03+00:00] run profile=dsv4-balanced shape=16k_1 isl=16384 osl=1 c=8 prompts=64
|
||||||
|
[2026-09-10T05:47:34+00:00] pass profile=dsv4-balanced point=16k_1_c8
|
||||||
|
[2026-09-10T05:47:35+00:00] run profile=dsv4-balanced shape=16k_1 isl=16384 osl=1 c=32 prompts=160
|
||||||
|
[2026-09-10T05:48:31+00:00] warning profile=dsv4-balanced point=16k_1_c32 zero_ttft_count=2
|
||||||
|
[2026-09-10T05:48:31+00:00] pass profile=dsv4-balanced point=16k_1_c32
|
||||||
|
[2026-09-10T05:48:32+00:00] run profile=dsv4-balanced shape=16k_1 isl=16384 osl=1 c=64 prompts=320
|
||||||
|
[2026-09-10T05:50:09+00:00] warning profile=dsv4-balanced point=16k_1_c64 zero_ttft_count=2
|
||||||
|
[2026-09-10T05:50:09+00:00] pass profile=dsv4-balanced point=16k_1_c64
|
||||||
|
[2026-09-10T05:50:10+00:00] run profile=dsv4-balanced shape=16k_1 isl=16384 osl=1 c=128 prompts=640
|
||||||
|
[2026-09-10T05:53:09+00:00] warning profile=dsv4-balanced point=16k_1_c128 zero_ttft_count=4
|
||||||
|
[2026-09-10T05:53:09+00:00] pass profile=dsv4-balanced point=16k_1_c128
|
||||||
|
[2026-09-10T05:53:10+00:00] run profile=dsv4-balanced shape=16k_1 isl=16384 osl=1 c=256 prompts=1280
|
||||||
|
[2026-09-10T05:58:59+00:00] warning profile=dsv4-balanced point=16k_1_c256 zero_ttft_count=7
|
||||||
|
[2026-09-10T05:58:59+00:00] pass profile=dsv4-balanced point=16k_1_c256
|
||||||
|
[2026-09-10T05:59:00+00:00] run profile=dsv4-balanced shape=1k_128 isl=1024 osl=128 c=1 prompts=64
|
||||||
|
[2026-09-10T06:00:58+00:00] pass profile=dsv4-balanced point=1k_128_c1
|
||||||
|
[2026-09-10T06:00:59+00:00] run profile=dsv4-balanced shape=1k_128 isl=1024 osl=128 c=8 prompts=64
|
||||||
|
[2026-09-10T06:01:28+00:00] pass profile=dsv4-balanced point=1k_128_c8
|
||||||
|
[2026-09-10T06:01:29+00:00] run profile=dsv4-balanced shape=1k_128 isl=1024 osl=128 c=32 prompts=160
|
||||||
|
[2026-09-10T06:01:55+00:00] pass profile=dsv4-balanced point=1k_128_c32
|
||||||
|
[2026-09-10T06:01:56+00:00] run profile=dsv4-balanced shape=1k_128 isl=1024 osl=128 c=64 prompts=320
|
||||||
|
[2026-09-10T06:02:25+00:00] pass profile=dsv4-balanced point=1k_128_c64
|
||||||
|
[2026-09-10T06:02:26+00:00] run profile=dsv4-balanced shape=1k_128 isl=1024 osl=128 c=128 prompts=640
|
||||||
|
[2026-09-10T06:03:03+00:00] pass profile=dsv4-balanced point=1k_128_c128
|
||||||
|
[2026-09-10T06:03:04+00:00] run profile=dsv4-balanced shape=1k_128 isl=1024 osl=128 c=256 prompts=1280
|
||||||
|
[2026-09-10T06:03:55+00:00] pass profile=dsv4-balanced point=1k_128_c256
|
||||||
|
[2026-09-10T06:03:56+00:00] run profile=dsv4-balanced shape=1k_4k isl=1024 osl=4096 c=1 prompts=64
|
||||||
|
run_full_text_matrix.sh: line 247: 1797795 Killed timeout --signal=TERM --kill-after=120 "${MAX_POINT_SECONDS}" sudo docker exec "${CONTAINER}" python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port "${PORT}" --model "${MODEL_CONTAINER}" --tokenizer "${MODEL_CONTAINER}" --dataset-name random-ids --tokenize-prompt --random-input-len "${isl}" --random-output-len "${osl}" --random-range-ratio 1.0 --num-prompts "${prompts}" --max-concurrency "${concurrency}" --seed "${SEED}" --disable-tqdm --flush-cache --output-details --output-file "/results/points/${point}.json" > "${console}" 2>&1
|
||||||
|
[2026-09-10T06:35:57+00:00] timeout pass profile=dsv4-balanced point=1k_4k_c1
|
||||||
|
[2026-09-10T06:35:58+00:00] run profile=dsv4-balanced shape=1k_4k isl=1024 osl=4096 c=8 prompts=64
|
||||||
|
[2026-09-10T06:42:24+00:00] pass profile=dsv4-balanced point=1k_4k_c8
|
||||||
|
[2026-09-10T06:42:25+00:00] run profile=dsv4-balanced shape=1k_4k isl=1024 osl=4096 c=32 prompts=160
|
||||||
|
[2026-09-10T06:47:31+00:00] pass profile=dsv4-balanced point=1k_4k_c32
|
||||||
|
[2026-09-10T06:47:32+00:00] run profile=dsv4-balanced shape=1k_4k isl=1024 osl=4096 c=64 prompts=320
|
||||||
|
[2026-09-10T06:53:11+00:00] pass profile=dsv4-balanced point=1k_4k_c64
|
||||||
|
[2026-09-10T06:53:12+00:00] run profile=dsv4-balanced shape=1k_4k isl=1024 osl=4096 c=128 prompts=640
|
||||||
|
[2026-09-10T06:59:43+00:00] pass profile=dsv4-balanced point=1k_4k_c128
|
||||||
|
[2026-09-10T06:59:44+00:00] run profile=dsv4-balanced shape=1k_4k isl=1024 osl=4096 c=256 prompts=1280
|
||||||
|
[2026-09-10T07:09:13+00:00] pass profile=dsv4-balanced point=1k_4k_c256
|
||||||
|
[2026-09-10T07:09:14+00:00] run profile=dsv4-balanced shape=64k_1 isl=65536 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-10T07:13:38+00:00] warning profile=dsv4-balanced point=64k_1_c1 zero_ttft_count=1
|
||||||
|
[2026-09-10T07:13:38+00:00] pass profile=dsv4-balanced point=64k_1_c1
|
||||||
|
[2026-09-10T07:13:39+00:00] run profile=dsv4-balanced shape=64k_1 isl=65536 osl=1 c=4 prompts=64
|
||||||
|
[2026-09-10T07:15:06+00:00] warning profile=dsv4-balanced point=64k_1_c4 zero_ttft_count=1
|
||||||
|
[2026-09-10T07:15:06+00:00] pass profile=dsv4-balanced point=64k_1_c4
|
||||||
|
[2026-09-10T07:15:07+00:00] run profile=dsv4-balanced shape=64k_1 isl=65536 osl=1 c=8 prompts=64
|
||||||
|
[2026-09-10T07:16:30+00:00] warning profile=dsv4-balanced point=64k_1_c8 zero_ttft_count=1
|
||||||
|
[2026-09-10T07:16:30+00:00] pass profile=dsv4-balanced point=64k_1_c8
|
||||||
|
[2026-09-10T07:16:31+00:00] run profile=dsv4-balanced shape=64k_1 isl=65536 osl=1 c=16 prompts=80
|
||||||
|
[2026-09-10T07:18:09+00:00] warning profile=dsv4-balanced point=64k_1_c16 zero_ttft_count=1
|
||||||
|
[2026-09-10T07:18:09+00:00] pass profile=dsv4-balanced point=64k_1_c16
|
||||||
|
[2026-09-10T07:18:10+00:00] run profile=dsv4-balanced shape=64k_1 isl=65536 osl=1 c=32 prompts=160
|
||||||
|
[2026-09-10T07:21:11+00:00] warning profile=dsv4-balanced point=64k_1_c32 zero_ttft_count=1
|
||||||
|
[2026-09-10T07:21:11+00:00] pass profile=dsv4-balanced point=64k_1_c32
|
||||||
|
[2026-09-10T07:21:12+00:00] run profile=dsv4-balanced shape=64k_1 isl=65536 osl=1 c=64 prompts=320
|
||||||
|
[2026-09-10T07:26:57+00:00] warning profile=dsv4-balanced point=64k_1_c64 zero_ttft_count=3
|
||||||
|
[2026-09-10T07:26:57+00:00] pass profile=dsv4-balanced point=64k_1_c64
|
||||||
|
[2026-09-10T07:26:58+00:00] run profile=dsv4-balanced shape=64k_1 isl=65536 osl=1 c=128 prompts=640
|
||||||
|
[2026-09-10T07:38:13+00:00] warning profile=dsv4-balanced point=64k_1_c128 zero_ttft_count=5
|
||||||
|
[2026-09-10T07:38:13+00:00] pass profile=dsv4-balanced point=64k_1_c128
|
||||||
|
[2026-09-10T07:38:14+00:00] run profile=dsv4-balanced shape=64k_512 isl=65536 osl=512 c=1 prompts=64
|
||||||
@ -0,0 +1,295 @@
|
|||||||
|
[2026-09-09T08:44:37+00:00] starting profile=dsv4-low-latency model=dsv4-flash-0731
|
||||||
|
[2026-09-09T08:44:38+00:00] waiting for server profile=dsv4-low-latency elapsed=0s
|
||||||
|
[2026-09-09T08:45:39+00:00] waiting for server profile=dsv4-low-latency elapsed=60s
|
||||||
|
[2026-09-09T08:46:39+00:00] waiting for server profile=dsv4-low-latency elapsed=120s
|
||||||
|
[2026-09-09T08:47:40+00:00] waiting for server profile=dsv4-low-latency elapsed=180s
|
||||||
|
[2026-09-09T08:48:41+00:00] waiting for server profile=dsv4-low-latency elapsed=240s
|
||||||
|
[2026-09-09T08:49:41+00:00] waiting for server profile=dsv4-low-latency elapsed=300s
|
||||||
|
[2026-09-09T08:50:42+00:00] waiting for server profile=dsv4-low-latency elapsed=360s
|
||||||
|
[2026-09-09T08:51:42+00:00] waiting for server profile=dsv4-low-latency elapsed=420s
|
||||||
|
[2026-09-09T08:52:43+00:00] waiting for server profile=dsv4-low-latency elapsed=480s
|
||||||
|
[2026-09-09T08:53:44+00:00] waiting for server profile=dsv4-low-latency elapsed=540s
|
||||||
|
[2026-09-09T08:54:44+00:00] waiting for server profile=dsv4-low-latency elapsed=600s
|
||||||
|
[2026-09-09T08:55:45+00:00] waiting for server profile=dsv4-low-latency elapsed=660s
|
||||||
|
[2026-09-09T08:56:45+00:00] waiting for server profile=dsv4-low-latency elapsed=720s
|
||||||
|
[2026-09-09T08:57:46+00:00] waiting for server profile=dsv4-low-latency elapsed=780s
|
||||||
|
[2026-09-09T08:58:46+00:00] waiting for server profile=dsv4-low-latency elapsed=840s
|
||||||
|
[2026-09-09T08:59:47+00:00] waiting for server profile=dsv4-low-latency elapsed=900s
|
||||||
|
[2026-09-09T09:00:48+00:00] waiting for server profile=dsv4-low-latency elapsed=960s
|
||||||
|
[2026-09-09T09:01:48+00:00] waiting for server profile=dsv4-low-latency elapsed=1020s
|
||||||
|
[2026-09-09T09:02:49+00:00] waiting for server profile=dsv4-low-latency elapsed=1080s
|
||||||
|
[2026-09-09T09:03:49+00:00] waiting for server profile=dsv4-low-latency elapsed=1140s
|
||||||
|
[2026-09-09T09:04:50+00:00] ERROR: server exited during startup profile=dsv4-low-latency
|
||||||
|
[2026-09-09T09:04:52+00:00] starting profile=dsv4-balanced model=dsv4-flash-0731
|
||||||
|
[2026-09-09T09:04:53+00:00] waiting for server profile=dsv4-balanced elapsed=0s
|
||||||
|
[2026-09-09T09:05:54+00:00] waiting for server profile=dsv4-balanced elapsed=60s
|
||||||
|
[2026-09-09T09:06:54+00:00] waiting for server profile=dsv4-balanced elapsed=120s
|
||||||
|
[2026-09-09T09:07:55+00:00] waiting for server profile=dsv4-balanced elapsed=180s
|
||||||
|
[2026-09-09T09:08:55+00:00] waiting for server profile=dsv4-balanced elapsed=240s
|
||||||
|
[2026-09-09T09:09:56+00:00] waiting for server profile=dsv4-balanced elapsed=300s
|
||||||
|
[2026-09-09T09:10:57+00:00] waiting for server profile=dsv4-balanced elapsed=360s
|
||||||
|
[2026-09-09T09:11:57+00:00] waiting for server profile=dsv4-balanced elapsed=420s
|
||||||
|
[2026-09-09T09:12:58+00:00] waiting for server profile=dsv4-balanced elapsed=480s
|
||||||
|
[2026-09-09T09:13:58+00:00] waiting for server profile=dsv4-balanced elapsed=540s
|
||||||
|
[2026-09-09T09:14:59+00:00] waiting for server profile=dsv4-balanced elapsed=600s
|
||||||
|
[2026-09-09T09:16:00+00:00] waiting for server profile=dsv4-balanced elapsed=660s
|
||||||
|
[2026-09-09T09:17:00+00:00] waiting for server profile=dsv4-balanced elapsed=720s
|
||||||
|
[2026-09-09T09:18:01+00:00] waiting for server profile=dsv4-balanced elapsed=780s
|
||||||
|
[2026-09-09T09:19:26+00:00] waiting for server profile=dsv4-balanced elapsed=840s
|
||||||
|
[2026-09-09T09:20:57+00:00] waiting for server profile=dsv4-balanced elapsed=900s
|
||||||
|
[2026-09-09T09:22:27+00:00] waiting for server profile=dsv4-balanced elapsed=960s
|
||||||
|
[2026-09-09T09:23:58+00:00] waiting for server profile=dsv4-balanced elapsed=1020s
|
||||||
|
[2026-09-09T09:25:28+00:00] waiting for server profile=dsv4-balanced elapsed=1080s
|
||||||
|
[2026-09-09T09:26:59+00:00] waiting for server profile=dsv4-balanced elapsed=1140s
|
||||||
|
[2026-09-09T09:27:55+00:00] ERROR: server exited during startup profile=dsv4-balanced
|
||||||
|
[2026-09-09T09:27:56+00:00] starting profile=glm53-low-latency model=glm-5.3
|
||||||
|
[2026-09-09T09:27:58+00:00] waiting for server profile=glm53-low-latency elapsed=0s
|
||||||
|
[2026-09-09T09:28:58+00:00] waiting for server profile=glm53-low-latency elapsed=60s
|
||||||
|
[2026-09-09T09:29:59+00:00] waiting for server profile=glm53-low-latency elapsed=120s
|
||||||
|
[2026-09-09T09:30:59+00:00] waiting for server profile=glm53-low-latency elapsed=180s
|
||||||
|
[2026-09-09T09:32:00+00:00] waiting for server profile=glm53-low-latency elapsed=240s
|
||||||
|
[2026-09-09T09:33:00+00:00] waiting for server profile=glm53-low-latency elapsed=300s
|
||||||
|
[2026-09-09T09:34:01+00:00] waiting for server profile=glm53-low-latency elapsed=360s
|
||||||
|
[2026-09-09T09:35:02+00:00] waiting for server profile=glm53-low-latency elapsed=420s
|
||||||
|
[2026-09-09T09:36:02+00:00] waiting for server profile=glm53-low-latency elapsed=480s
|
||||||
|
[2026-09-09T09:37:03+00:00] waiting for server profile=glm53-low-latency elapsed=540s
|
||||||
|
[2026-09-09T09:38:03+00:00] waiting for server profile=glm53-low-latency elapsed=600s
|
||||||
|
[2026-09-09T09:39:04+00:00] waiting for server profile=glm53-low-latency elapsed=660s
|
||||||
|
[2026-09-09T09:40:05+00:00] waiting for server profile=glm53-low-latency elapsed=720s
|
||||||
|
[2026-09-09T09:41:05+00:00] waiting for server profile=glm53-low-latency elapsed=780s
|
||||||
|
[2026-09-09T09:41:26+00:00] server healthy profile=glm53-low-latency elapsed=800s
|
||||||
|
[2026-09-09T09:41:43+00:00] run profile=glm53-low-latency shape=16k_512 isl=16384 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-09T09:44:14+00:00] pass profile=glm53-low-latency point=16k_512_c1
|
||||||
|
[2026-09-09T09:44:15+00:00] run profile=glm53-low-latency shape=16k_512 isl=16384 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-09T09:45:33+00:00] pass profile=glm53-low-latency point=16k_512_c8
|
||||||
|
[2026-09-09T09:45:34+00:00] run profile=glm53-low-latency shape=16k_512 isl=16384 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-09T09:47:53+00:00] pass profile=glm53-low-latency point=16k_512_c32
|
||||||
|
[2026-09-09T09:47:54+00:00] run profile=glm53-low-latency shape=16k_512 isl=16384 osl=512 c=64 prompts=320
|
||||||
|
[2026-09-09T09:51:54+00:00] pass profile=glm53-low-latency point=16k_512_c64
|
||||||
|
[2026-09-09T09:51:55+00:00] run profile=glm53-low-latency shape=16k_512 isl=16384 osl=512 c=128 prompts=640
|
||||||
|
[2026-09-09T09:59:33+00:00] pass profile=glm53-low-latency point=16k_512_c128
|
||||||
|
[2026-09-09T09:59:34+00:00] run profile=glm53-low-latency shape=16k_512 isl=16384 osl=512 c=256 prompts=1280
|
||||||
|
[2026-09-09T10:14:32+00:00] pass profile=glm53-low-latency point=16k_512_c256
|
||||||
|
[2026-09-09T10:14:33+00:00] run profile=glm53-low-latency shape=16k_1 isl=16384 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-09T10:15:27+00:00] failed profile=glm53-low-latency point=16k_1_c1 rc=0
|
||||||
|
[2026-09-09T10:15:28+00:00] capacity short-circuit profile=glm53-low-latency shape=16k_1 c=8
|
||||||
|
[2026-09-09T10:15:28+00:00] capacity short-circuit profile=glm53-low-latency shape=16k_1 c=32
|
||||||
|
[2026-09-09T10:15:28+00:00] capacity short-circuit profile=glm53-low-latency shape=16k_1 c=64
|
||||||
|
[2026-09-09T10:15:28+00:00] capacity short-circuit profile=glm53-low-latency shape=16k_1 c=128
|
||||||
|
[2026-09-09T10:15:28+00:00] capacity short-circuit profile=glm53-low-latency shape=16k_1 c=256
|
||||||
|
[2026-09-09T10:15:29+00:00] run profile=glm53-low-latency shape=1k_128 isl=1024 osl=128 c=1 prompts=64
|
||||||
|
[2026-09-09T10:16:17+00:00] pass profile=glm53-low-latency point=1k_128_c1
|
||||||
|
[2026-09-09T10:16:18+00:00] run profile=glm53-low-latency shape=1k_128 isl=1024 osl=128 c=8 prompts=64
|
||||||
|
[2026-09-09T10:16:43+00:00] pass profile=glm53-low-latency point=1k_128_c8
|
||||||
|
[2026-09-09T10:16:44+00:00] run profile=glm53-low-latency shape=1k_128 isl=1024 osl=128 c=32 prompts=160
|
||||||
|
[2026-09-09T10:17:12+00:00] pass profile=glm53-low-latency point=1k_128_c32
|
||||||
|
[2026-09-09T10:17:13+00:00] run profile=glm53-low-latency shape=1k_128 isl=1024 osl=128 c=64 prompts=320
|
||||||
|
[2026-09-09T10:17:51+00:00] pass profile=glm53-low-latency point=1k_128_c64
|
||||||
|
[2026-09-09T10:17:52+00:00] run profile=glm53-low-latency shape=1k_128 isl=1024 osl=128 c=128 prompts=640
|
||||||
|
[2026-09-09T10:18:54+00:00] pass profile=glm53-low-latency point=1k_128_c128
|
||||||
|
[2026-09-09T10:18:55+00:00] run profile=glm53-low-latency shape=1k_128 isl=1024 osl=128 c=256 prompts=1280
|
||||||
|
[2026-09-09T10:20:44+00:00] pass profile=glm53-low-latency point=1k_128_c256
|
||||||
|
[2026-09-09T10:20:45+00:00] run profile=glm53-low-latency shape=1k_4k isl=1024 osl=4096 c=1 prompts=64
|
||||||
|
[2026-09-09T10:31:01+00:00] pass profile=glm53-low-latency point=1k_4k_c1
|
||||||
|
[2026-09-09T10:31:02+00:00] run profile=glm53-low-latency shape=1k_4k isl=1024 osl=4096 c=8 prompts=64
|
||||||
|
[2026-09-09T10:33:34+00:00] pass profile=glm53-low-latency point=1k_4k_c8
|
||||||
|
[2026-09-09T10:33:35+00:00] run profile=glm53-low-latency shape=1k_4k isl=1024 osl=4096 c=32 prompts=160
|
||||||
|
[2026-09-09T10:36:09+00:00] pass profile=glm53-low-latency point=1k_4k_c32
|
||||||
|
[2026-09-09T10:36:10+00:00] run profile=glm53-low-latency shape=1k_4k isl=1024 osl=4096 c=64 prompts=320
|
||||||
|
[2026-09-09T10:40:13+00:00] pass profile=glm53-low-latency point=1k_4k_c64
|
||||||
|
[2026-09-09T10:40:14+00:00] run profile=glm53-low-latency shape=1k_4k isl=1024 osl=4096 c=128 prompts=640
|
||||||
|
[2026-09-09T10:47:43+00:00] pass profile=glm53-low-latency point=1k_4k_c128
|
||||||
|
[2026-09-09T10:47:44+00:00] run profile=glm53-low-latency shape=1k_4k isl=1024 osl=4096 c=256 prompts=1280
|
||||||
|
[2026-09-09T11:02:30+00:00] pass profile=glm53-low-latency point=1k_4k_c256
|
||||||
|
[2026-09-09T11:02:31+00:00] run profile=glm53-low-latency shape=64k_1 isl=65536 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-09T11:05:37+00:00] pass profile=glm53-low-latency point=64k_1_c1
|
||||||
|
[2026-09-09T11:05:38+00:00] run profile=glm53-low-latency shape=64k_1 isl=65536 osl=1 c=4 prompts=64
|
||||||
|
[2026-09-09T11:08:42+00:00] failed profile=glm53-low-latency point=64k_1_c4 rc=0
|
||||||
|
[2026-09-09T11:08:44+00:00] run profile=glm53-low-latency shape=64k_1 isl=65536 osl=1 c=8 prompts=64
|
||||||
|
[2026-09-09T11:11:48+00:00] pass profile=glm53-low-latency point=64k_1_c8
|
||||||
|
[2026-09-09T11:11:49+00:00] run profile=glm53-low-latency shape=64k_1 isl=65536 osl=1 c=16 prompts=80
|
||||||
|
[2026-09-09T11:15:35+00:00] failed profile=glm53-low-latency point=64k_1_c16 rc=0
|
||||||
|
[2026-09-09T11:15:37+00:00] run profile=glm53-low-latency shape=64k_1 isl=65536 osl=1 c=32 prompts=160
|
||||||
|
[2026-09-09T11:22:51+00:00] failed profile=glm53-low-latency point=64k_1_c32 rc=0
|
||||||
|
[2026-09-09T11:22:53+00:00] run profile=glm53-low-latency shape=64k_1 isl=65536 osl=1 c=64 prompts=320
|
||||||
|
[2026-09-09T11:37:04+00:00] failed profile=glm53-low-latency point=64k_1_c64 rc=0
|
||||||
|
[2026-09-09T11:37:06+00:00] run profile=glm53-low-latency shape=64k_1 isl=65536 osl=1 c=128 prompts=640
|
||||||
|
[2026-09-09T12:05:11+00:00] failed profile=glm53-low-latency point=64k_1_c128 rc=0
|
||||||
|
[2026-09-09T12:05:13+00:00] run profile=glm53-low-latency shape=64k_512 isl=65536 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-09T12:09:57+00:00] pass profile=glm53-low-latency point=64k_512_c1
|
||||||
|
[2026-09-09T12:09:58+00:00] run profile=glm53-low-latency shape=64k_512 isl=65536 osl=512 c=4 prompts=64
|
||||||
|
[2026-09-09T12:13:37+00:00] pass profile=glm53-low-latency point=64k_512_c4
|
||||||
|
[2026-09-09T12:13:38+00:00] run profile=glm53-low-latency shape=64k_512 isl=65536 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-09T12:17:04+00:00] pass profile=glm53-low-latency point=64k_512_c8
|
||||||
|
[2026-09-09T12:17:05+00:00] run profile=glm53-low-latency shape=64k_512 isl=65536 osl=512 c=16 prompts=80
|
||||||
|
[2026-09-09T12:21:10+00:00] pass profile=glm53-low-latency point=64k_512_c16
|
||||||
|
[2026-09-09T12:21:11+00:00] run profile=glm53-low-latency shape=64k_512 isl=65536 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-09T12:28:50+00:00] pass profile=glm53-low-latency point=64k_512_c32
|
||||||
|
[2026-09-09T12:28:51+00:00] run profile=glm53-low-latency shape=64k_512 isl=65536 osl=512 c=64 prompts=320
|
||||||
|
[2026-09-09T12:43:43+00:00] pass profile=glm53-low-latency point=64k_512_c64
|
||||||
|
[2026-09-09T12:43:44+00:00] run profile=glm53-low-latency shape=128k_1 isl=131072 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-09T12:50:24+00:00] pass profile=glm53-low-latency point=128k_1_c1
|
||||||
|
[2026-09-09T12:50:25+00:00] run profile=glm53-low-latency shape=128k_1 isl=131072 osl=1 c=2 prompts=64
|
||||||
|
[2026-09-09T12:57:01+00:00] pass profile=glm53-low-latency point=128k_1_c2
|
||||||
|
[2026-09-09T12:57:02+00:00] run profile=glm53-low-latency shape=128k_1 isl=131072 osl=1 c=4 prompts=64
|
||||||
|
[2026-09-09T13:03:37+00:00] pass profile=glm53-low-latency point=128k_1_c4
|
||||||
|
[2026-09-09T13:03:39+00:00] run profile=glm53-low-latency shape=128k_1 isl=131072 osl=1 c=8 prompts=64
|
||||||
|
[2026-09-09T13:10:14+00:00] failed profile=glm53-low-latency point=128k_1_c8 rc=0
|
||||||
|
[2026-09-09T13:10:16+00:00] run profile=glm53-low-latency shape=128k_1 isl=131072 osl=1 c=16 prompts=80
|
||||||
|
[2026-09-09T13:18:26+00:00] failed profile=glm53-low-latency point=128k_1_c16 rc=0
|
||||||
|
[2026-09-09T13:18:28+00:00] run profile=glm53-low-latency shape=128k_1 isl=131072 osl=1 c=32 prompts=160
|
||||||
|
[2026-09-09T13:34:27+00:00] failed profile=glm53-low-latency point=128k_1_c32 rc=0
|
||||||
|
[2026-09-09T13:34:29+00:00] run profile=glm53-low-latency shape=128k_1 isl=131072 osl=1 c=64 prompts=320
|
||||||
|
[2026-09-09T14:06:04+00:00] timeout pass profile=glm53-low-latency point=128k_1_c64
|
||||||
|
[2026-09-09T14:06:05+00:00] run profile=glm53-low-latency shape=128k_512 isl=131072 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-09T14:14:28+00:00] pass profile=glm53-low-latency point=128k_512_c1
|
||||||
|
[2026-09-09T14:14:29+00:00] run profile=glm53-low-latency shape=128k_512 isl=131072 osl=512 c=2 prompts=64
|
||||||
|
[2026-09-09T14:22:02+00:00] pass profile=glm53-low-latency point=128k_512_c2
|
||||||
|
[2026-09-09T14:22:03+00:00] run profile=glm53-low-latency shape=128k_512 isl=131072 osl=512 c=4 prompts=64
|
||||||
|
[2026-09-09T14:29:16+00:00] pass profile=glm53-low-latency point=128k_512_c4
|
||||||
|
[2026-09-09T14:29:17+00:00] run profile=glm53-low-latency shape=128k_512 isl=131072 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-09T14:36:18+00:00] pass profile=glm53-low-latency point=128k_512_c8
|
||||||
|
[2026-09-09T14:36:19+00:00] run profile=glm53-low-latency shape=128k_512 isl=131072 osl=512 c=16 prompts=80
|
||||||
|
[2026-09-09T14:44:48+00:00] pass profile=glm53-low-latency point=128k_512_c16
|
||||||
|
[2026-09-09T14:44:49+00:00] run profile=glm53-low-latency shape=128k_512 isl=131072 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-09T15:01:22+00:00] pass profile=glm53-low-latency point=128k_512_c32
|
||||||
|
[2026-09-09T15:01:23+00:00] run profile=glm53-low-latency shape=256k_1 isl=262144 osl=1 c=1 prompts=16
|
||||||
|
[2026-09-09T15:05:54+00:00] pass profile=glm53-low-latency point=256k_1_c1
|
||||||
|
[2026-09-09T15:05:55+00:00] run profile=glm53-low-latency shape=256k_1 isl=262144 osl=1 c=2 prompts=16
|
||||||
|
[2026-09-09T15:10:25+00:00] pass profile=glm53-low-latency point=256k_1_c2
|
||||||
|
[2026-09-09T15:10:26+00:00] run profile=glm53-low-latency shape=256k_1 isl=262144 osl=1 c=4 prompts=16
|
||||||
|
[2026-09-09T15:14:56+00:00] failed profile=glm53-low-latency point=256k_1_c4 rc=0
|
||||||
|
[2026-09-09T15:14:58+00:00] run profile=glm53-low-latency shape=512k_1 isl=524288 osl=1 c=1 prompts=8
|
||||||
|
[2026-09-09T15:21:35+00:00] pass profile=glm53-low-latency point=512k_1_c1
|
||||||
|
[2026-09-09T15:21:36+00:00] run profile=glm53-low-latency shape=512k_1 isl=524288 osl=1 c=2 prompts=8
|
||||||
|
[2026-09-09T15:28:10+00:00] pass profile=glm53-low-latency point=512k_1_c2
|
||||||
|
[2026-09-09T15:28:12+00:00] run profile=glm53-low-latency shape=512k_1 isl=524288 osl=1 c=4 prompts=8
|
||||||
|
[2026-09-09T15:34:46+00:00] pass profile=glm53-low-latency point=512k_1_c4
|
||||||
|
[2026-09-09T15:34:47+00:00] run profile=glm53-low-latency shape=1m_boundary_1 isl=1047552 osl=1 c=1 prompts=4
|
||||||
|
[2026-09-09T15:46:07+00:00] pass profile=glm53-low-latency point=1m_boundary_1_c1
|
||||||
|
[2026-09-09T15:46:08+00:00] run profile=glm53-low-latency shape=1m_boundary_1 isl=1047552 osl=1 c=2 prompts=4
|
||||||
|
[2026-09-09T15:57:31+00:00] pass profile=glm53-low-latency point=1m_boundary_1_c2
|
||||||
|
[2026-09-09T15:57:32+00:00] run profile=glm53-low-latency shape=1m_boundary_1 isl=1047552 osl=1 c=4 prompts=4
|
||||||
|
[2026-09-09T16:08:51+00:00] pass profile=glm53-low-latency point=1m_boundary_1_c4
|
||||||
|
[2026-09-09T16:09:36+00:00] starting profile=glm53-high-throughput model=glm-5.3
|
||||||
|
[2026-09-09T16:09:38+00:00] waiting for server profile=glm53-high-throughput elapsed=0s
|
||||||
|
[2026-09-09T16:10:38+00:00] waiting for server profile=glm53-high-throughput elapsed=60s
|
||||||
|
[2026-09-09T16:11:39+00:00] waiting for server profile=glm53-high-throughput elapsed=120s
|
||||||
|
[2026-09-09T16:12:39+00:00] waiting for server profile=glm53-high-throughput elapsed=180s
|
||||||
|
[2026-09-09T16:13:40+00:00] waiting for server profile=glm53-high-throughput elapsed=240s
|
||||||
|
[2026-09-09T16:14:41+00:00] waiting for server profile=glm53-high-throughput elapsed=300s
|
||||||
|
[2026-09-09T16:15:41+00:00] waiting for server profile=glm53-high-throughput elapsed=360s
|
||||||
|
[2026-09-09T16:16:42+00:00] waiting for server profile=glm53-high-throughput elapsed=420s
|
||||||
|
[2026-09-09T16:17:42+00:00] waiting for server profile=glm53-high-throughput elapsed=480s
|
||||||
|
[2026-09-09T16:18:43+00:00] waiting for server profile=glm53-high-throughput elapsed=540s
|
||||||
|
[2026-09-09T16:19:44+00:00] waiting for server profile=glm53-high-throughput elapsed=600s
|
||||||
|
[2026-09-09T16:20:44+00:00] waiting for server profile=glm53-high-throughput elapsed=660s
|
||||||
|
[2026-09-09T16:21:17+00:00] server healthy profile=glm53-high-throughput elapsed=690s
|
||||||
|
[2026-09-09T16:21:35+00:00] run profile=glm53-high-throughput shape=16k_512 isl=16384 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-09T16:31:55+00:00] pass profile=glm53-high-throughput point=16k_512_c1
|
||||||
|
[2026-09-09T16:31:56+00:00] run profile=glm53-high-throughput shape=16k_512 isl=16384 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-09T16:34:33+00:00] pass profile=glm53-high-throughput point=16k_512_c8
|
||||||
|
[2026-09-09T16:34:34+00:00] run profile=glm53-high-throughput shape=16k_512 isl=16384 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-09T16:37:43+00:00] pass profile=glm53-high-throughput point=16k_512_c32
|
||||||
|
[2026-09-09T16:37:44+00:00] run profile=glm53-high-throughput shape=16k_512 isl=16384 osl=512 c=64 prompts=320
|
||||||
|
[2026-09-09T16:41:55+00:00] pass profile=glm53-high-throughput point=16k_512_c64
|
||||||
|
[2026-09-09T16:41:56+00:00] run profile=glm53-high-throughput shape=16k_512 isl=16384 osl=512 c=128 prompts=640
|
||||||
|
[2026-09-09T16:47:50+00:00] pass profile=glm53-high-throughput point=16k_512_c128
|
||||||
|
[2026-09-09T16:47:51+00:00] run profile=glm53-high-throughput shape=16k_512 isl=16384 osl=512 c=256 prompts=1280
|
||||||
|
[2026-09-09T17:01:55+00:00] pass profile=glm53-high-throughput point=16k_512_c256
|
||||||
|
[2026-09-09T17:01:57+00:00] run profile=glm53-high-throughput shape=16k_1 isl=16384 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-09T17:03:58+00:00] failed profile=glm53-high-throughput point=16k_1_c1 rc=0
|
||||||
|
[2026-09-09T17:03:59+00:00] capacity short-circuit profile=glm53-high-throughput shape=16k_1 c=8
|
||||||
|
[2026-09-09T17:03:59+00:00] capacity short-circuit profile=glm53-high-throughput shape=16k_1 c=32
|
||||||
|
[2026-09-09T17:03:59+00:00] capacity short-circuit profile=glm53-high-throughput shape=16k_1 c=64
|
||||||
|
[2026-09-09T17:03:59+00:00] capacity short-circuit profile=glm53-high-throughput shape=16k_1 c=128
|
||||||
|
[2026-09-09T17:03:59+00:00] capacity short-circuit profile=glm53-high-throughput shape=16k_1 c=256
|
||||||
|
[2026-09-09T17:04:00+00:00] run profile=glm53-high-throughput shape=1k_128 isl=1024 osl=128 c=1 prompts=64
|
||||||
|
[2026-09-09T17:06:45+00:00] pass profile=glm53-high-throughput point=1k_128_c1
|
||||||
|
[2026-09-09T17:06:47+00:00] run profile=glm53-high-throughput shape=1k_128 isl=1024 osl=128 c=8 prompts=64
|
||||||
|
[2026-09-09T17:07:33+00:00] pass profile=glm53-high-throughput point=1k_128_c8
|
||||||
|
[2026-09-09T17:07:34+00:00] run profile=glm53-high-throughput shape=1k_128 isl=1024 osl=128 c=32 prompts=160
|
||||||
|
[2026-09-09T17:08:21+00:00] pass profile=glm53-high-throughput point=1k_128_c32
|
||||||
|
[2026-09-09T17:08:22+00:00] run profile=glm53-high-throughput shape=1k_128 isl=1024 osl=128 c=64 prompts=320
|
||||||
|
[2026-09-09T17:09:18+00:00] pass profile=glm53-high-throughput point=1k_128_c64
|
||||||
|
[2026-09-09T17:09:19+00:00] run profile=glm53-high-throughput shape=1k_128 isl=1024 osl=128 c=128 prompts=640
|
||||||
|
[2026-09-09T17:10:20+00:00] pass profile=glm53-high-throughput point=1k_128_c128
|
||||||
|
[2026-09-09T17:10:21+00:00] run profile=glm53-high-throughput shape=1k_128 isl=1024 osl=128 c=256 prompts=1280
|
||||||
|
[2026-09-09T17:12:11+00:00] pass profile=glm53-high-throughput point=1k_128_c256
|
||||||
|
[2026-09-09T17:12:12+00:00] run profile=glm53-high-throughput shape=1k_4k isl=1024 osl=4096 c=1 prompts=64
|
||||||
|
run_full_text_matrix.sh: line 227: 1729155 Killed timeout --signal=TERM --kill-after=120 "${MAX_POINT_SECONDS}" sudo docker exec "${CONTAINER}" python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port "${PORT}" --model "${MODEL_CONTAINER}" --tokenizer "${MODEL_CONTAINER}" --dataset-name random-ids --tokenize-prompt --random-input-len "${isl}" --random-output-len "${osl}" --random-range-ratio 1.0 --num-prompts "${prompts}" --max-concurrency "${concurrency}" --seed "${SEED}" --disable-tqdm --flush-cache --output-details --output-file "/results/points/${point}.json" > "${console}" 2>&1
|
||||||
|
[2026-09-09T17:44:12+00:00] timeout pass profile=glm53-high-throughput point=1k_4k_c1
|
||||||
|
[2026-09-09T17:44:13+00:00] run profile=glm53-high-throughput shape=1k_4k isl=1024 osl=4096 c=8 prompts=64
|
||||||
|
[2026-09-09T17:54:26+00:00] pass profile=glm53-high-throughput point=1k_4k_c8
|
||||||
|
[2026-09-09T17:54:27+00:00] run profile=glm53-high-throughput shape=1k_4k isl=1024 osl=4096 c=32 prompts=160
|
||||||
|
[2026-09-09T18:01:43+00:00] pass profile=glm53-high-throughput point=1k_4k_c32
|
||||||
|
[2026-09-09T18:01:44+00:00] run profile=glm53-high-throughput shape=1k_4k isl=1024 osl=4096 c=64 prompts=320
|
||||||
|
[2026-09-09T18:09:25+00:00] pass profile=glm53-high-throughput point=1k_4k_c64
|
||||||
|
[2026-09-09T18:09:26+00:00] run profile=glm53-high-throughput shape=1k_4k isl=1024 osl=4096 c=128 prompts=640
|
||||||
|
[2026-09-09T18:18:25+00:00] pass profile=glm53-high-throughput point=1k_4k_c128
|
||||||
|
[2026-09-09T18:18:26+00:00] run profile=glm53-high-throughput shape=1k_4k isl=1024 osl=4096 c=256 prompts=1280
|
||||||
|
[2026-09-09T18:28:01+00:00] pass profile=glm53-high-throughput point=1k_4k_c256
|
||||||
|
[2026-09-09T18:28:02+00:00] run profile=glm53-high-throughput shape=64k_1 isl=65536 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-09T18:35:08+00:00] pass profile=glm53-high-throughput point=64k_1_c1
|
||||||
|
[2026-09-09T18:35:09+00:00] run profile=glm53-high-throughput shape=64k_1 isl=65536 osl=1 c=4 prompts=64
|
||||||
|
[2026-09-09T18:37:47+00:00] pass profile=glm53-high-throughput point=64k_1_c4
|
||||||
|
[2026-09-09T18:37:48+00:00] run profile=glm53-high-throughput shape=64k_1 isl=65536 osl=1 c=8 prompts=64
|
||||||
|
[2026-09-09T18:39:31+00:00] pass profile=glm53-high-throughput point=64k_1_c8
|
||||||
|
[2026-09-09T18:39:32+00:00] run profile=glm53-high-throughput shape=64k_1 isl=65536 osl=1 c=16 prompts=80
|
||||||
|
[2026-09-09T18:41:31+00:00] failed profile=glm53-high-throughput point=64k_1_c16 rc=0
|
||||||
|
[2026-09-09T18:41:33+00:00] run profile=glm53-high-throughput shape=64k_1 isl=65536 osl=1 c=32 prompts=160
|
||||||
|
[2026-09-09T18:45:06+00:00] failed profile=glm53-high-throughput point=64k_1_c32 rc=0
|
||||||
|
[2026-09-09T18:45:09+00:00] run profile=glm53-high-throughput shape=64k_1 isl=65536 osl=1 c=64 prompts=320
|
||||||
|
[2026-09-09T18:51:53+00:00] failed profile=glm53-high-throughput point=64k_1_c64 rc=0
|
||||||
|
[2026-09-09T18:51:55+00:00] run profile=glm53-high-throughput shape=64k_1 isl=65536 osl=1 c=128 prompts=640
|
||||||
|
[2026-09-09T19:04:59+00:00] failed profile=glm53-high-throughput point=64k_1_c128 rc=0
|
||||||
|
[2026-09-09T19:05:01+00:00] run profile=glm53-high-throughput shape=64k_512 isl=65536 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-09T19:20:18+00:00] pass profile=glm53-high-throughput point=64k_512_c1
|
||||||
|
[2026-09-09T19:20:19+00:00] run profile=glm53-high-throughput shape=64k_512 isl=65536 osl=512 c=4 prompts=64
|
||||||
|
[2026-09-09T19:28:39+00:00] pass profile=glm53-high-throughput point=64k_512_c4
|
||||||
|
[2026-09-09T19:28:40+00:00] run profile=glm53-high-throughput shape=64k_512 isl=65536 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-09T19:34:23+00:00] pass profile=glm53-high-throughput point=64k_512_c8
|
||||||
|
[2026-09-09T19:34:24+00:00] run profile=glm53-high-throughput shape=64k_512 isl=65536 osl=512 c=16 prompts=80
|
||||||
|
[2026-09-09T19:39:29+00:00] pass profile=glm53-high-throughput point=64k_512_c16
|
||||||
|
[2026-09-09T19:39:30+00:00] run profile=glm53-high-throughput shape=64k_512 isl=65536 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-09T19:46:57+00:00] pass profile=glm53-high-throughput point=64k_512_c32
|
||||||
|
[2026-09-09T19:46:58+00:00] run profile=glm53-high-throughput shape=64k_512 isl=65536 osl=512 c=64 prompts=320
|
||||||
|
[2026-09-09T19:57:59+00:00] pass profile=glm53-high-throughput point=64k_512_c64
|
||||||
|
[2026-09-09T19:58:00+00:00] run profile=glm53-high-throughput shape=128k_1 isl=131072 osl=1 c=1 prompts=64
|
||||||
|
[2026-09-09T20:12:25+00:00] failed profile=glm53-high-throughput point=128k_1_c1 rc=0
|
||||||
|
[2026-09-09T20:12:26+00:00] capacity short-circuit profile=glm53-high-throughput shape=128k_1 c=2
|
||||||
|
[2026-09-09T20:12:26+00:00] capacity short-circuit profile=glm53-high-throughput shape=128k_1 c=4
|
||||||
|
[2026-09-09T20:12:26+00:00] capacity short-circuit profile=glm53-high-throughput shape=128k_1 c=8
|
||||||
|
[2026-09-09T20:12:26+00:00] capacity short-circuit profile=glm53-high-throughput shape=128k_1 c=16
|
||||||
|
[2026-09-09T20:12:26+00:00] capacity short-circuit profile=glm53-high-throughput shape=128k_1 c=32
|
||||||
|
[2026-09-09T20:12:26+00:00] capacity short-circuit profile=glm53-high-throughput shape=128k_1 c=64
|
||||||
|
[2026-09-09T20:12:27+00:00] run profile=glm53-high-throughput shape=128k_512 isl=131072 osl=512 c=1 prompts=64
|
||||||
|
[2026-09-09T20:35:02+00:00] pass profile=glm53-high-throughput point=128k_512_c1
|
||||||
|
[2026-09-09T20:35:03+00:00] run profile=glm53-high-throughput shape=128k_512 isl=131072 osl=512 c=2 prompts=64
|
||||||
|
[2026-09-09T20:53:24+00:00] pass profile=glm53-high-throughput point=128k_512_c2
|
||||||
|
[2026-09-09T20:53:25+00:00] run profile=glm53-high-throughput shape=128k_512 isl=131072 osl=512 c=4 prompts=64
|
||||||
|
[2026-09-09T21:08:00+00:00] pass profile=glm53-high-throughput point=128k_512_c4
|
||||||
|
[2026-09-09T21:08:01+00:00] run profile=glm53-high-throughput shape=128k_512 isl=131072 osl=512 c=8 prompts=64
|
||||||
|
[2026-09-09T21:18:56+00:00] pass profile=glm53-high-throughput point=128k_512_c8
|
||||||
|
[2026-09-09T21:18:57+00:00] run profile=glm53-high-throughput shape=128k_512 isl=131072 osl=512 c=16 prompts=80
|
||||||
|
[2026-09-09T21:28:19+00:00] pass profile=glm53-high-throughput point=128k_512_c16
|
||||||
|
[2026-09-09T21:28:20+00:00] run profile=glm53-high-throughput shape=128k_512 isl=131072 osl=512 c=32 prompts=160
|
||||||
|
[2026-09-09T21:41:42+00:00] pass profile=glm53-high-throughput point=128k_512_c32
|
||||||
|
[2026-09-09T21:41:43+00:00] run profile=glm53-high-throughput shape=256k_1 isl=262144 osl=1 c=1 prompts=16
|
||||||
|
[2026-09-09T21:49:54+00:00] failed profile=glm53-high-throughput point=256k_1_c1 rc=0
|
||||||
|
[2026-09-09T21:49:55+00:00] capacity short-circuit profile=glm53-high-throughput shape=256k_1 c=2
|
||||||
|
[2026-09-09T21:49:55+00:00] capacity short-circuit profile=glm53-high-throughput shape=256k_1 c=4
|
||||||
|
[2026-09-09T21:49:56+00:00] run profile=glm53-high-throughput shape=512k_1 isl=524288 osl=1 c=1 prompts=8
|
||||||
|
[2026-09-09T22:00:41+00:00] pass profile=glm53-high-throughput point=512k_1_c1
|
||||||
|
[2026-09-09T22:00:42+00:00] run profile=glm53-high-throughput shape=512k_1 isl=524288 osl=1 c=2 prompts=8
|
||||||
|
[2026-09-09T22:07:05+00:00] pass profile=glm53-high-throughput point=512k_1_c2
|
||||||
|
[2026-09-09T22:07:06+00:00] run profile=glm53-high-throughput shape=512k_1 isl=524288 osl=1 c=4 prompts=8
|
||||||
|
[2026-09-09T22:11:24+00:00] pass profile=glm53-high-throughput point=512k_1_c4
|
||||||
|
[2026-09-09T22:11:25+00:00] run profile=glm53-high-throughput shape=1m_boundary_1 isl=1047552 osl=1 c=1 prompts=4
|
||||||
|
[2026-09-09T22:28:06+00:00] pass profile=glm53-high-throughput point=1m_boundary_1_c1
|
||||||
|
[2026-09-09T22:28:07+00:00] run profile=glm53-high-throughput shape=1m_boundary_1 isl=1047552 osl=1 c=2 prompts=4
|
||||||
|
[2026-09-09T22:38:42+00:00] pass profile=glm53-high-throughput point=1m_boundary_1_c2
|
||||||
|
[2026-09-09T22:38:43+00:00] run profile=glm53-high-throughput shape=1m_boundary_1 isl=1047552 osl=1 c=4 prompts=4
|
||||||
|
[2026-09-09T22:46:03+00:00] pass profile=glm53-high-throughput point=1m_boundary_1_c4
|
||||||
|
[2026-09-09T22:46:47+00:00] matrix finished run_dir=/data/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427
|
||||||
|
run_full_text_matrix.sh: line 460: unexpected EOF while looking for matching `''
|
||||||
@ -0,0 +1,66 @@
|
|||||||
|
# B300 DeepSeek-V4-Flash / GLM-5.3 Comparison: b300-dsv4-glm53-dev4-20260909-084427
|
||||||
|
|
||||||
|
## Run Status
|
||||||
|
|
||||||
|
- Finished: 3.39%
|
||||||
|
- Pass: 7
|
||||||
|
- Timeout pass: 0
|
||||||
|
- Invalid: 1
|
||||||
|
- Fail: 0
|
||||||
|
|
||||||
|
## Service Configuration And Capacity
|
||||||
|
|
||||||
|
| Profile | TP | DP | EP | Max running | Max tokens/rank | Chunk | KV dtype | Attention | MoE runner | MoE A2A | Speculative |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---|---|---|---|---|
|
||||||
|
| dsv4-low-latency | - | - | - | - | - | - | - | - | - | - | - |
|
||||||
|
| dsv4-balanced | - | - | - | - | - | - | - | - | - | - | - |
|
||||||
|
| glm53-low-latency | 8 | 1 | 1 | 48 | 2696000 | 16384 | fp8_e4m3 | dsa | flashinfer_trtllm | none | EAGLE |
|
||||||
|
| glm53-high-throughput | - | - | - | - | - | - | - | - | - | - | - |
|
||||||
|
|
||||||
|
## Throughput Peaks And Their Latency
|
||||||
|
|
||||||
|
Only validated PASS points are considered.
|
||||||
|
|
||||||
|
| Shape | Mode | Peak input TPS @ C | TTFT P95 (ms) | Peak output TPS @ C | TPOT P95 (ms) |
|
||||||
|
|---|---|---:|---:|---:|---:|
|
||||||
|
| 16k_512 | glm53-low-latency | 24021.20 @ C256 | 144384.42 | 750.66 @ C256 | 102.36 |
|
||||||
|
| 1k_128 | glm53-low-latency | 2005.88 @ C1 | 70.19 | 250.73 @ C1 | 5.55 |
|
||||||
|
|
||||||
|
## Capacity And Failure Boundaries
|
||||||
|
|
||||||
|
| Mode | Shape | C | Status | Evidence |
|
||||||
|
|---|---|---:|---|---|
|
||||||
|
| glm53-low-latency | 16k_1 | 1 | INVALID | `/data/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427/glm53-low-latency/points/16k_1_c1.json` |
|
||||||
|
|
||||||
|
## Service Error Summary
|
||||||
|
|
||||||
|
### dsv4-low-latency
|
||||||
|
|
||||||
|
- Matched lines: 10
|
||||||
|
- Source: `/data/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427/dsv4-low-latency/server.final.log`
|
||||||
|
- `[2026-09-09 09:04:34 TP3] Scheduler hit an exception: Traceback (most recent call last):`
|
||||||
|
- `[2026-09-09 09:04:34] SIGQUIT received. signum=None, frame=None. It usually means one child failed.`
|
||||||
|
- `[2026-09-09 09:04:34 TP1] Scheduler hit an exception: Traceback (most recent call last):`
|
||||||
|
- `[2026-09-09 09:04:34 TP0] Scheduler hit an exception: Traceback (most recent call last):`
|
||||||
|
- `[2026-09-09 09:04:34 TP2] Scheduler hit an exception: Traceback (most recent call last):`
|
||||||
|
|
||||||
|
### dsv4-balanced
|
||||||
|
|
||||||
|
- Matched lines: 14
|
||||||
|
- Source: `/data/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427/dsv4-balanced/server.final.log`
|
||||||
|
- `/dvs/p4/build/sw/rel/gpgpu/toolkit/r13.0/main_nvshmem/src/host/transport/transport.cpp:nvshmemi_transport_init:282: init failed for transport: IBGDA`
|
||||||
|
- `[2026-09-09 09:18:18 DP0 TP0 EP0] Scheduler hit an exception: Traceback (most recent call last):`
|
||||||
|
- `[2026-09-09 09:18:19] SIGQUIT received. signum=None, frame=None. It usually means one child failed.`
|
||||||
|
- `[2026-09-09 09:27:44] Initialization failed. warmup error: Traceback (most recent call last):`
|
||||||
|
- `Traceback (most recent call last):`
|
||||||
|
|
||||||
|
### glm53-low-latency
|
||||||
|
|
||||||
|
- Matched lines: 0
|
||||||
|
- Source: `/data/b300-dsv4-glm53-text-matrix/b300-dsv4-glm53-dev4-20260909-084427/glm53-low-latency/server.log`
|
||||||
|
|
||||||
|
### glm53-high-throughput
|
||||||
|
|
||||||
|
- Matched lines: 0
|
||||||
|
- Source: `-`
|
||||||
|
|
||||||
@ -0,0 +1 @@
|
|||||||
|
2026-09-10T07:45:20+00:00
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user