Compare commits
45 Commits
main
...
hzy-kimi3-
| Author | SHA1 | Date | |
|---|---|---|---|
|
75ab8865de | ||
|
|
411802e4b1 | ||
|
|
c9305fb830 | ||
|
|
3fbfb84e12 | ||
|
|
0e33209ae5 | ||
|
|
91ab3fe32d | ||
|
|
c06ef5fd61 | ||
|
|
a58b931cc8 | ||
|
|
0433fcc3ee | ||
|
|
d790df39b2 | ||
|
|
96ffea6d37 | ||
|
|
d7381abe84 | ||
|
|
60f77cd4ef | ||
|
|
6fac5ad567 | ||
|
|
2a3b12fa78 | ||
|
|
08a35066d7 | ||
|
|
74ec19dd48 | ||
|
|
0fdcab9927 | ||
|
|
a5248ed80e | ||
|
|
fc336a3c7b | ||
|
|
63f2327a90 | ||
|
|
a90c898683 | ||
|
|
ab9a5422f6 | ||
|
|
7f67dfe6b3 | ||
|
|
e8ff3ce1e8 | ||
|
|
d28db48e4b | ||
|
|
39f692caae | ||
|
|
a9206ff105 | ||
|
|
a11c80b703 | ||
|
|
e01df16667 | ||
|
|
ec7b604a50 | ||
|
|
27b8be09cb | ||
|
|
be9d6bfe3a | ||
|
|
d0863501ca | ||
|
|
13944079fa | ||
|
|
e3974e2352 | ||
|
|
b50de8fe99 | ||
|
|
c14f8aa43a | ||
|
|
daeffd147b | ||
|
|
5454fb984e | ||
|
|
c8f30ab7dc | ||
|
|
6493798ad5 | ||
|
|
a1c18d736b | ||
|
|
dac1bb652d | ||
|
|
0684d269df |
38
README.md
38
README.md
@ -1,5 +1,39 @@
|
||||
# sskj — 多平台大模型推理性能基准测试项目
|
||||
|
||||
**更新（2026-08-31 18:22:43 CST）**：完成 Kimi-K3 PP8 + DFlash 八节点 PD 部署验证。P 601-604 为 TP4/PP8/EP4，D 605-608 为 TP32/PP1/EP4，均使用 FlashInfer MXFP4、BF16 KV、8K Chunk。固定 GSM8K 64 题 C1/C8 共 128/128 请求成功，答案检查全部通过且无输出截断；按 verify 步数加权的接受长度为 6.4211/6.5158，整体 Output TPS 为 33.38/117.56。修复评测容器的数据集路径，保留原失败与续跑证据；结果、完整服务日志及资源清理记录已归档。历史 5-shot 与前 5 题重叠，本轮作为部署回归而非独立准确率评测。详见 [实现与结果](experiments/pro6000/kimi3_pro6000_pd_dflash_validation/README.md)。
|
||||
|
||||
**更新（2026-08-31 17:50:51 CST）**：修复 Kimi 混合注意力缓存池在 PD 初始化时缺少 `end_layer` 的兼容问题，按完整注意力层偏移和本地层数确定传输范围，保留普通池与 layer-shard 路径。新增回归后 24 项 CPU 测试通过，八节点修复镜像摘要一致；P 组已完成预热和 Mooncake 注册，健康检查返回 200，D 组正在验证启动。当前 Run 为 `pd-dflash-kvbounds1-20260831-1740`，尚无 GSM8K 验收结果。详见 `experiments/pro6000/kimi3_pro6000_pd_dflash_validation/README.md`。
|
||||
|
||||
**更新（2026-08-31 17:05:00 CST）**：新增 Kimi-K3 PP8 + DFlash PD 适配验证入口。基于 SGLang PR #33863 固定源码，接通 PP 分段 hidden 投影、P 侧 prompt draft KV 生成、D 侧输入生命周期与 TP4→TP32 的 draft GQA KV 传输，保留 Kimi SM120 FlashInfer MXFP4 接入。修复 P 普通预热误带 DFlash verify metadata 的启动问题，23 项 CPU 回归通过；修复镜像已同步 601–608，服务级验证正在进行，尚无 GSM8K 结果。配置为 P TP4/PP8/EP4、D TP32/PP1/EP4、BF16 KV、8K Chunk，计划固定 64 题 C1/C8。详见 `experiments/pro6000/kimi3_pro6000_pd_dflash_validation/README.md`。
|
||||
|
||||
**更新（2026-08-27 13:53:26 CST）**：完成 Kimi-K3 八节点标准 PD 第一阶段。P 组 601-604 使用 PP8×TP4×EP4、FlashInfer MXFP4、Chunk 8K，D 组 605-608 使用 PP1×TP32×EP32、Marlin，通过 Mooncake 0.3.12.post1 和 4 Rail RDMA 传输；统一 P/D `page_size=64` 后，16K→1 与 16K→512 的 C1/C8 共 91/91 请求成功。代表结果：16K→1 C8 Input TPS 6364.31、TTFT P50/P95 20.555/21.345 秒；16K→512 C8 TPOT P50/P95 63.20/66.55 ms。详见 `experiments/pro6000/kimi3_pro6000_pd_pp8_standard/README.md`。
|
||||
|
||||
**更新（2026-08-20 10:13:01 CST）**：完成 Kimi-K3 Prefill TP Reduce Scatter 可行性审计并停止该方向。K3 的 MLA 输出门控仍依赖完整 7168 维 hidden，且 69/93 层为 KDA；保留或恢复 gate hidden 后，原 hidden All-Reduce 无法消除并新增 2112 维 latent All-Gather，估算通信量反增约 14.7%。研究原型仅保留为否决证据，不进入四节点实验或上游 PR；后续转向 MoE A2A 与 Pipeline Parallelism。详见 `experiments/pro6000/kimi3_pro6000_sglang_tp_reduce_scatter_prefill/README.md`。
|
||||
|
||||
**更新（2026-08-19 16:51:43 CST）**：完成 Kimi-K3 / RTX 6000D / SGLang Prefill MoE backend 交付文档收敛与 Chunk=4K 补测。四节点 TP32×EP4、16K→1、C=8/16 下 12/12 重复、480/480 请求成功；FlashInfer 相对 Marlin 提升约 9.8% Input TPS、降低约 9.0% TTFT。结合 4K/8K/16K 全部结果，最终推荐保持 `flashinfer_mxfp4 + chunked_prefill_size=8192`。
|
||||
|
||||
**更新（2026-08-19 14:23:52 CST）**：601 的 GitLab SSH key 已生效，内部 Kimi-K3 SM120 审计分支已推送至 `qqtang/sskj:hzy-kimi-k3-sm120-flashinfer-mxfp4`。远端已有普通分支 `hzy`，因此不能创建 `hzy/...` ref，最终采用无层级冲突的分支名；GitHub SGLang feature 分支保持不变。
|
||||
|
||||
**更新（2026-08-19 13:58:50 CST）**：用户同步 GitHub fork 后，将 Kimi-K3 SM120 SGLang 的 7 个提交无冲突重放到最新 `main` `593b1a9`，`range-diff` 逐提交确认补丁等价；最终 Draft HEAD `b95b534` 已推送至 `ACEEE-1222/sglang:hzy/kimi-k3-sm120-flashinfer-mxfp4`。在 601 统一验证镜像中重跑最终 HEAD 的语法与定向回归，结果为 `10 passed, 3 deselected`；代表性四机性能数据保留，不重复消耗 GPU。
|
||||
|
||||
**更新（2026-08-19 13:07:48 CST）**：完成 Kimi-K3 SM120 SGLang Draft PR 的最终代表点验收。基于 SGLang `300c87a` 与 FlashInfer #4460 `b525c51`，在 601-604 四节点 TP32×EP4 上完成 16K→1、C=8、Chunk=8K 的 Marlin/FlashInfer 各 3 次 A/B；240/240 请求成功。FlashInfer 的 Input TPS 中位数提升 10.96%，TTFT P50/P95 分别下降 9.91%/9.92%。公开 PR Checklist 已全部勾选，轻量汇总和原始证据索引已归档。
|
||||
|
||||
**更新（2026-08-19 10:45:18 CST）**：新增 Kimi-K3 SM120 SGLang Draft PR 的单点代表性验收入口。使用精确 SGLang `300c87a`、FlashInfer #4460 实现 `b525c51` 和统一镜像，在 601-604 四节点 TP32×EP4 上仅复测 16K→1、C=8、Chunk=8K 的 Marlin/FlashInfer A/B，各 3 次重复；完整命令、镜像身份、原始日志和汇总统一落入单个 Run 目录。
|
||||
|
||||
**更新（2026-08-18 23:00:42 CST）**：完成依赖 FlashInfer #4460 的 SGLang Draft PR 收敛。确认不提交任何 FlashInfer PR，也不 vendor 或 pin 未合并 kernel；在 601 GPU6 上从 #4460 源码构建 FlashInfer 0.6.18 wheel，SGLang 定向测试 8/8 通过。Draft 标题、依赖、兼容策略、测试和四机 EP4 数据见 `experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/SGLANG_DRAFT_PR.md`。
|
||||
|
||||
**更新（2026-08-18 22:21:40 CST）**：完成 Kimi-K3 SM120 FlashInfer MXFP4 去重审计。确认 CUTLASS SiTU kernel 已由未合并的 FlashInfer PR #4460 实现，不再提交平行 kernel；SGLang 贡献收敛为 Kimi gate/up 与 scale 布局、SiTU 4.0/25.0 参数映射、非连续输入和 SM120 attention-residual guard，并保留 601-604 EP4 的全部服务级结果。详见 `experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/UPSTREAM_DUPLICATION_AUDIT.md`。
|
||||
|
||||
**更新（2026-08-18 18:38:17 CST）**：完成 Kimi-K3 四节点 TP32×EP4 的真实 Prefill MoE backend 验收；24/24 重复均 40/40 成功，FlashInfer MXFP4 在 8K Chunk、C=8/16 下相对 Marlin 提升约 11% Input TPS、降低约 10% TTFT，并确认 8K 优于 16K。实验默认 EP 改为 4，补齐非连续 Prefill 输入与 FlashInfer 官方 SiTU 回归，详见 `experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/README.md`。
|
||||
|
||||
**更新（2026-08-18 13:55:30 CST）**：Kimi-K3 四节点 MoE backend 验收入口支持通过环境变量覆盖 EP size 及 backend/chunk/concurrency 子矩阵，用于先执行 TP32×EP4、16K Chunk、16K→1、C=16 的最大压力容量验证，同时保留默认 TP32×EP32 全矩阵。
|
||||
|
||||
**更新（2026-08-18 13:03:33 CST）**：Phase 5 候选镜像改为针对 Kimi 基础镜像原生 SGLang 源码应用最小兼容补丁，仅补齐 SM120 FlashInfer MXFP4 backend 分发、权重布局与 SiTU 激活映射，避免整体替换新版 Python 包造成 `sglang-kernel` 版本错配。
|
||||
|
||||
**更新（2026-08-18 12:24:19 CST）**：新增 Kimi-K3 / 601-604 / SGLang TP32×EP32 的真实 Prefill MoE backend 验收实验；固定比较 Marlin 与 FlashInfer MXFP4 在 16K→1、C=8/16、Chunk=8K/16K 下的 TTFT、E2E 与 Input TPS，并提供四节点一致候选镜像构建、原始证据和自动汇总，详见 `experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/README.md`。
|
||||
|
||||
**更新（2026-08-17 11:56:47 CST）**：完成 Kimi-K3 / RTX PRO 6000D / SGLang FlashInfer MXFP4 MoE Phase 2 correctness harness；SwiGLU 控制矩阵 8/8 通过，覆盖 edge routing、EP global ID、NaN/Inf、确定性、CUDA Graph 和 Kimi 真尺寸，SiTU 目标矩阵 8/8 稳定复现同一 C++ 缺口，详见 `experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/README.md`。
|
||||
|
||||
> 历史更新见 `git log`。项目目的与工作流见下方。
|
||||
> **项目目的**：当新显卡（GPU/NPU）到货时，用最短时间完成大模型在该平台上的推理性能评估与部署配置选型。
|
||||
> 当前模型：DeepSeek-V4-Flash（FP8 / INT8）；后续接入 GLM5.2，**完全复用**本项目的实验与报告流程。
|
||||
@ -45,6 +79,9 @@
|
||||
| `experiments/pro6000/dsv4_pro6000_vllm_tp_dp_matrix/` | RTX 6000D + vLLM |
|
||||
| `experiments/pro6000/dsv4_pro6000_sglang_tp_dp_matrix/` | RTX 6000D + SGLang |
|
||||
| `experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/` | RTX 6000D + SGLang，Kimi-K3（TP32×EP32，部署手册见 docs/KIMI_K3_DEPLOY.md） |
|
||||
| `experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/` | RTX 6000D + SGLang，Kimi-K3 真实 Prefill 的 Marlin/FlashInfer MXFP4 与 Chunk 8K/16K 对照 |
|
||||
| `experiments/pro6000/kimi3_pro6000_sglang_tp_reduce_scatter_prefill/` | Kimi-K3 Prefill TP Reduce Scatter 可行性审计；因 MLA gate 与 KDA 依赖完整 hidden，结论为不推荐继续 |
|
||||
| `experiments/pro6000/kimi3_pro6000_pd_pp8_standard/` | Kimi-K3 八节点标准 PD：P=PP8/TP4/EP4、D=PP1/TP32/EP32、Mooncake 4 Rail RDMA，含 16K→1/512 C1/C8 原始证据 |
|
||||
| `experiments/pro6000/kimi3_pro6000_pd_rdma/` | RTX 6000D + SGLang，Kimi-K3 PD 分离（MoonCake RDMA，8 节点，见 deploy_pd.sh + docs/KIMI_K3_DEPLOY.md 附录 B） |
|
||||
|
||||
每个目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_concurrency.sh` 从 C=1 指数倍增搜饱和点；`run_adaptive_concurrency_add16.sh` 从 C=16 线性 +16 步进、带 TTFT SLO 停止与回退（当前主力用法，见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`）。
|
||||
@ -60,7 +97,6 @@
|
||||
| `experiments/h200/dsv4_h200_vllm_tp{2,4,8}_custom_bench/` | 自定义压测客户端（多服务负载均衡） |
|
||||
| `experiments/h200/dsv4_h200_max_context_length/` `…/dsv4_h200_long_context_matrix/` `…/dsv4_h200_256k_4k_probe/` `…/dsv4_h200_64k_sglang_vs_vllm/` | 长上下文专项 |
|
||||
| `experiments/p800/dsv4_p800_sglang/` `…/dsv4_p800_max_context_length/` `…/dsv4_p800_long_context_matrix/` `…/dsv4_p800_256k_4k_probe/` | P800 baseline 与长上下文专项 |
|
||||
| `experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/` | RTX 6000D + SGLang，GLM-5.3-NVFP4 双场景压测标准（16k/512 高并发 + 128k/64k 90% 命中低并发；真实语料 run-id 窗口纪律 + 三套部署配置 A/B/CAR，基线与口径见目录 README；部署 profile 见 `deploy/profiles/pro6000/glm53_nvfp4_*`） |
|
||||
| `experiments/TEMPLATE/` | 老式固定场景实验模板 |
|
||||
|
||||
## 快速复现
|
||||
|
||||
@ -1,35 +0,0 @@
|
||||
# 现役部署状态页（live 核验于 2026-09-10，60.8 当日核验；其余机器 09-09 口径）
|
||||
|
||||
> 本页记录 174.1.60.x 集群"现在跑的是什么"。改动机器前后先读这页；
|
||||
> **任何变动（停/起/改配置）完成后必须更新本页**。核验命令：`docker ps` + `nvidia-smi`。
|
||||
|
||||
## 机器状态（2026-09-09 实测）
|
||||
|
||||
| 机器 | 在役 | 口径 / 归属 | 对应 profile |
|
||||
|---|---|---|---|
|
||||
| 60.1 (6000D-1) | `glm53-pp4`（Up，8 卡满载，:30000） | **方案 D 生产**。09-09/10 两轮实验窗口停机（凌晨 D 基线战役 ~7h + 日间 A 基线战役 ~9.5h）均恢复核验（health+生成冒烟）。**D 配置升级路径已交付未执行**：latest 镜像+开 autotune+持久缓存，16k/512 cc8-64 五点 +1.2~3.2%（A/B/A 确认），profile `..._tp2pp4_latest_autotune.env`——**仅适用 D 配置**：09-10 A 基线（TP8+EAGLE）战役证明 latest 与 autotune 在 A 配置上均负收益（eager 路径对代差/ tactic 敏感，图内不敏感），A 配置维持 nightly+autotune off；**IPC@TP8 真增益 +4.6~9.1% 已验证**（vs 同镜像基线全点胜、cc8/cc64 分布不重叠，收益随 TP 度单调升：TP2 −3.5%→TP8 +9.1%）但补丁包仅兼容 latest（sglang 4 文件依赖新版 runtime_context API），net vs 生产≈0；落地=移植回 nightly 或等上游 PR #34528 合入，未执行。同轮 A 基线画像：生产 A 掉图陷阱主导场景二（cc16+ 全程 eager，TPOT 168-197ms，平台 ~70-73；已发表 95/96 是 A-s2 图覆盖 bs16 变体）；日内漂移=eager 点 8.5h 后 −6~8% 而图内点不动（CPU 侧签名）。实验全量 `experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/`（REPORT.md + REPORT_A_BASELINE.md）。`dsv4_scan` 经授权保持停止，还原说明 `/root/dsv4_scan_restore_note.txt` | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp2pp4.env`（现役）；升级候选 `..._tp2pp4_latest_autotune.env`（仅 D 配置） |
|
||||
| 60.2 (6000D-2) | `glm53-nvfp4` 实验容器（09-08 晚 TP1PP8 phase，run_phase.sh 实验链进行中） | **他人实验进行中，勿动**（方案 F PD 链已拆除；GPU7 曾有外部裸金属任务）。动卡前仍先核实归属 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_{prefill→decode 侧}.env` |
|
||||
| 60.3 | 无容器，但 8 卡被外部裸金属训练占用（`/data/mas/larm`，09-08 晚实测） | 外部任务，勿动（此前台账漏记） | — |
|
||||
| 60.4 | `glm53-nvfp4`（Up，:30000，restart=unless-stopped） | **TP8+EAGLE+custom-AR 1stage（E7b 配方）在役**（09-09 下午部署：EAGLE 4/1/5/mem0.90/MRR16/chunk8192/ctxlen270336/fp8KV+hicache3/decode 图桶 1-8/双 parser；CAR 补丁三处全注入、8 rank `SSKJ_CAR_PATCH_ACTIVE` 确认，health 200、生成冒烟、质量门 7/7 见 `/root/qg_604_car.log`）。启动 `/root/deploy_glm53_604_exp.sh`（=60.7 实验版逐字拷贝，`RESTART=yes CAR_PATCH=1`），补丁 `/root/patches/`（md5 与仓库 car_patch 归档一致）。当日早间曾短暂部署 TP2PP4 D 配方复刻（deploy_s2_test_604.sh 留盘可切回）后被本方案替换；同日经授权清退外部 vllm 评测流水线（tmux `mas` 的 run_multiseed.sh 链，--resume 可续跑） | `experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/`（deploy_glm53_604_exp.sh + car_patch/ 补丁快照） |
|
||||
| 60.5 | `glm53-nvfp4`（Up 2d，09-09 只读核验） | **NVFP4 团队生产**（deploy_glm53_605.sh，md5 fcd9109b）。生产机铁律：不实验、不重启、不覆盖脚本。**交付升级路径已更新为 v3（09-09 hit90 场景优胜 TP4PP2-hicache@0.90，池 647,040/c4 并发独立文档/hit90 cc8 out +34% vs 现役）：脚本 `experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/scripts/deploy_glm53_605_v3.sh` + 补丁束（60.8:/root/glm53_r37_patch_bundle_v3.tar.gz，md5 6922e534，需 scp 至 60.5）+ 需 docker pull cu13 镜像；未执行、未落 60.5 磁盘（60.5:/root 仅有原脚本，核验过）。此前 v2（TP2PP4-hicache，冷缓存口径优胜）被 v3 取代，仍留仓库可作"容量优先 6 条文档/512k 单条最快"备选；回滚=原脚本** | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp8eagle.env`（方案 A 口径）；待切 `glm53_nvfp4_pro6000_sglang_tp4pp2_hicache.env`（容量优先备选 `..._tp2pp4_hicache.env`） |
|
||||
| 60.6 | 无容器，但 8 卡被外部裸金属实验占用（`/data/hzy/sparse-opd-*`，09-08 晚实测） | 外部任务，勿动（此前台账漏记） | — |
|
||||
| 60.7 | 基本空（4 卡仍有 `/home/user/dirA_exp` 外部小任务，09-08 晚实测） | 09-08 已拆除清空（方案 F 前身单机实验 + 场景一深优资产留盘），不再恢复 | — |
|
||||
| 60.8 | `glm53-nvfp4`（Up，:30000，restart=unless-stopped） | **TP4PP2-nomtp@0.90 hit90 优胜口径在役**（09-09 晚接替同日早间的 TP2PP4@0.85：同 r37 栈 cu13+9 挂载+de-GLOO，仅改 tp4/pp2 + memfrac 0.90；KV 池 **647,040**；实测实例原样保留切 restart 策略，未重建容器。启动 `bash /root/deploy_ppmtp_r37.sh '--tp 4 --pp-size 2 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length 1048576' nomtp 8192 0.90 1 1 0 0`）。hit90（90% 命中 i128k/o512）out cc1/2/3/4/8/16 = **28.8/47.5/63.9/76.2/106.4/125.7**（vs TP2PP4 基线 cc4 +14%/cc8 +15%）；并发独立 128k 文档 **4 条**（cap cc4 98.5 零排队）、512k 单条 ✓ TTFT 135.8s、质量门 **7/7**；同轮判决：DP attention 容量负收益判死、DCP 对 DSA 静默算错禁用、MTP@128k accept 2.07 判负。让步项（知情）：容量 4 条（TP2PP4 为 6）、512k 单条慢 33%。回滚 TP2PP4 = 同命令改 `'--tp 2 --pp-size 4'` + memfrac 0.85。**09-10 B300 对标战役**：停役（容器 rename 保全 `glm53-nvfp4-insvc`）→ 双臂 B300 场景矩阵（TP2PP4-D 口径 24 点 + E7b 配方）→ E7b 高并发调参复测（初测 MRR16+图1-8 掉图断崖 → MRR64+图桶 1-64，`deploy_glm53_e7b_hicc.sh`，三场景 c8-c64 共 10 点全 OK；报告正文采用复测值：掉图断崖已修复、16K 仍 TP2PP4 占优（E7b 被 prefill 墙封 ~100 tok/s 平台）、decode 密集 1K→4K E7b 全档反超（c64 out 826.5 tok/s 全场最高、超 TP2PP4 72%）、边界仅 TP2PP4 可达、与 B300 绝对差 4-5×；报告飞书 wiki `A7V3wZTQeifCB4krdi6cA834nW9`，已整文更新）→ TP2PP4 高并发调参复测（初测 C=64 受 MRR48 封顶 48 活跃+16 排队、decode 图默认覆盖 bs≤256 从未掉图 → MRR64，`deploy_glm53_pp4_mrr64.sh` 逐 token 仅改 MRR；三场景 c32/c64 共 6 点两轮对齐 ≤1.8%（pass-1 + v2 有序版）+ 全新实例 MRR48 对照归因；正文采用 v2 值：4.1/4.2 c64 out +27%/+19%（433/572）、三场景 c64 TTFT P95 全塌缩（20.0/337.9/134.8s → 7.2/7.3/89.2s）、16K c64 到池顶 ~59 活跃 + 3 次回退（确定性行为）、1K 短输入 c32 付 −14% MRR64 配置代价（276→266→228 = −3% 新鲜度 −14% 配置，三实例可复现、机制未定位）、16K c32 无代价；报告已再次整文更新）→ **原容器恢复并三次核验**（rename 回 + start，fired up/health 200/16K 抽测 ok、KV 池 647,040 与启动口径逐字一致、显存水位 77.2/82.3 GiB 与停役前一致） | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp4pp2_hicache.env`；实验全量 `experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/`；B300 对标 `experiments/pro6000/glm53_nvfp4_b300_equivalent_matrix/`；前史 `experiments/pro6000/glm53_nvfp4_128k_capacity_topology/`（TP2PP4 口径）+ r37 `experiments/pro6000/glm53_ppmtp_r37_verify_graph/` |
|
||||
|
||||
## 方案 A-F 一览（GLM-5.3-NVFP4 @ pro6000，2026-09-08 双场景报告口径）
|
||||
|
||||
| 方案 | 一句话 | profile / 脚本 |
|
||||
|---|---|---|
|
||||
| A | TP8 + EAGLE，基准；60.5 生产口径 | `glm53_nvfp4_pro6000_sglang_tp8eagle.env` |
|
||||
| B | TP4 PP2 | `glm53_nvfp4_pro6000_sglang_tp4pp2.env` |
|
||||
| C | TP4 PP2 + IndexCache(freq=4) | `glm53_nvfp4_pro6000_sglang_tp4pp2_index.env` |
|
||||
| D | TP2 PP4；60.1 生产在役 | `glm53_nvfp4_pro6000_sglang_tp2pp4.env` |
|
||||
| E | TP8 + DFlash2 投机 | `glm53_nvfp4_pro6000_sglang_tp8dflash2.env` |
|
||||
| F | PD 分离双机链（mc-master→prefill→decode→router） | `glm53_nvfp4_pro6000_pd_{master,prefill,decode,router}.env` + `deploy/PD_CHAIN.md` |
|
||||
|
||||
压测数据：飞书《GLM-5.3-NVFP4 双场景压测报告》（`SZUSdEqY1oRVxGxgILBcHqPJnEc`）。
|
||||
|
||||
## 防漂移
|
||||
|
||||
每个在役容器用 `deploy/verify_profile.sh <profile.env>` 定期核验（镜像 digest + 启动参数 +
|
||||
端口），发现不一致 = 容器被人手改过，先查清归属再处理。
|
||||
@ -1,96 +0,0 @@
|
||||
# 方案F：GLM-5.3-NVFP4 PD 分离完整链（双机 6000D）编排手册
|
||||
|
||||
> 2026-09-08 实测终态。四角色、两台机、启动顺序强制。吞吐换算口径：**链合计 ÷2 = 单机等效**（与单机方案 A-E 可比）。
|
||||
> 完整压测数据见飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 `SZUSdEqY1oRVxGxgILBcHqPJnEc` / wiki `NzbMwzmKviYidRkZYRrc8GYQnrf`）。
|
||||
|
||||
## 拓扑
|
||||
|
||||
| 角色 | 机器 | 容器 | 端口 | profile |
|
||||
|---|---|---|---|---|
|
||||
| 1. mc-master（Mooncake 元数据） | 174.1.60.1 | `mc-master` | 50051 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_master.env` |
|
||||
| 2. prefill（TP4 PP2 + DFLASH 草稿） | 174.1.60.1 | `glm53-pd-smoke-prefill` | 30000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_prefill.env` |
|
||||
| 3. decode（TP8 + DFLASH v5 配方） | 174.1.60.2 | `glm53-s1-decode` | 30000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_decode.env` |
|
||||
| 4. router（MiniLB） | 174.1.60.2 | `pd-smoke-router` | 31000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_router.env` |
|
||||
|
||||
镜像统一：`lmsysorg/sglang:nightly-dev-20260828-daf63171`
|
||||
（digest `sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399`）。
|
||||
|
||||
## 启动顺序（强制）
|
||||
|
||||
```
|
||||
mc-master (60.1) → prefill (60.1) → decode (60.2) → router (60.2)
|
||||
```
|
||||
|
||||
对应脚本（`experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/`）：
|
||||
|
||||
```bash
|
||||
# 60.1（注意：60.1 日常跑生产容器 glm53-pp4，先停它，见下节）
|
||||
bash /tmp/deploy_pd_smoke_master.sh # 判据 ss :50051
|
||||
bash /tmp/deploy_pd_probe.sh p1b_prefill_tp4pp2_cps8k_radix_launch.sh # 判据 :30000/health 200，~10-15min
|
||||
# 60.2
|
||||
bash /tmp/deploy_s1_decode.sh # 判据 :30000/health 200，~10min
|
||||
bash /tmp/deploy_pd_smoke_router.sh # 判据 ss :31000
|
||||
```
|
||||
|
||||
场景一（90% 命中）必须用 **radix 变体** launch（`p1b_prefill_tp4pp2_cps8k_radix_launch.sh`，
|
||||
与基线唯一差异 = 无 `--disable-radix-cache`）。场景二 0 命中，radix 开销可忽略，两场景共用同一部署。
|
||||
|
||||
## 基础设施依赖（缺一不可）
|
||||
|
||||
| 依赖 | 位置 | 说明 |
|
||||
|---|---|---|
|
||||
| sglang 补丁树 | `/data/sglang_patch_glm53` → 容器 `/sgl-workspace/sglang` | 与镜像原版差 **11 个文件**（10 改 + 1 新增 `dflash_pp.py`），清单与 diff 见 `platforms/patches/pro6000/glm53_pd_chain/`。无补丁则 DFlash+PD 冷启动接线缺失，decode 首请求 400 |
|
||||
| mooncake wheel | `/data/flashkda_deploy/wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl` | 容器内 launch 脚本 `pip install --no-deps` 自装 |
|
||||
| DFlash2 草稿权重 | `/data/hf_models/GLM-5.3-DFlash2` | 两角色都要挂 |
|
||||
| IB 设备 | mlx5_0–mlx5_3 + `--device /dev/infiniband` + `--ulimit memlock=-1` | RDMA KV 传输通道 |
|
||||
| 环境变量 | `MOONCAKE_MASTER=174.1.60.1:50051` `MOONCAKE_PROTOCOL=rdma` | prefill/decode 容器都要 |
|
||||
|
||||
## 质量门（经 router :31000，全链路含 KV transfer + 草稿）
|
||||
|
||||
60.2 上生成 router 口径的 gate 副本：
|
||||
|
||||
```bash
|
||||
sed 's/PORT=30000/PORT=31000/' /root/quality_gate_605.sh > /tmp/quality_gate_31000.sh
|
||||
```
|
||||
|
||||
判据：GSM8K×5 + 中文推理必须过。tool call 预期不过（链配方无 parser，与方案 B/D 同口径的已知配置缺口，记录放行）。
|
||||
2026-09-08 实测 **6/7**（核心项全过，DFlash 草稿无质量损失）。
|
||||
|
||||
## 压测
|
||||
|
||||
驱动脚本：`run_pd_s1.sh` / `run_pd_s2.sh`（同 scripts 目录），跑在 60.1（语料本地，
|
||||
`--url http://174.1.60.2:31000/generate` 走 router，`--container glm53-pd-smoke-prefill` 读 prefill 日志核命中率）。
|
||||
|
||||
- **场景一**：128k/64k × cc1-4，nreq8，shared-frac 0.9，canonical 窗口 9301-9308
|
||||
- **场景二**：cc8/16/32 canonical 9311-9313（nreq 16/32/32）；cc40/64 全新窗口 pool-override
|
||||
16,384,000 / 17,203,200（nreq 40/64 单轮满波）
|
||||
- 指标读取：命中率读 prefill 容器日志（PP2 日志计数 ×2，不影响比值）；accept 从 decode 日志读
|
||||
（router 转发后 usage 字段可能缺失）
|
||||
- 每点核验：ok/failed=0、retractions、s1 hit≈0.90、s2 hit=0
|
||||
|
||||
### 2026-09-08 实测判决（13/13 干净点）
|
||||
|
||||
- **场景一**：128k cc1 TTFT 3.91s = 六方案最低（跨机重叠把 prefill 与 RDMA 传输完全藏住），
|
||||
但 cc≥2 时 decode 侧排队，TTFT 堆到 31.0s；÷2 输出吞吐 21.2-23.5 tok/s，仅为 A 的 30-57%。
|
||||
根因：decode KV 池 214,336 token 按 16k 场景定容 → 128k 仅 1 驻留、64k 仅 3。
|
||||
- **场景二**：÷2 单机等效 74-85 tok/s = A 的 75-96%，但 TTFT 较 A 减半。
|
||||
判决：**两机买 TTFT、不买吞吐**。decode MRR12 在 cc16+ 饱和；DFlash accept 在 12-batch
|
||||
verify 下掉到 1.8（单机 EAGLE 2.84）。
|
||||
- **PD 机制验证**：跨机 prefill/decode 重叠成立、mooncake RDMA 传输被吸收——机制无罪，
|
||||
容量配置（decode 池按 16k 定容）是判决主因。
|
||||
|
||||
## 拆链与生产恢复（顺序固定）
|
||||
|
||||
```bash
|
||||
# 1. 拆链
|
||||
# 60.1: docker rm -f mc-master glm53-pd-smoke-prefill
|
||||
# 60.2: docker rm -f glm53-s1-decode pd-smoke-router
|
||||
# 2. 等双机显存排空（60.1 <2000MiB；60.2 GPU7 常有外部裸金属任务 main_v2.py，核实归属勿清）
|
||||
# 3. 恢复 60.1 生产：bash /tmp/deploy_glm53_pp4.sh → health 200 + docker inspect Args 核对
|
||||
```
|
||||
|
||||
## 60.1 生产停机窗口提示
|
||||
|
||||
60.1 日常跑生产 `glm53-pp4`（方案 D，TP2PP4，8 卡满载）。拉 prefill 角色前必须停生产；
|
||||
`deploy_glm53_pp4.sh` 生成的参数与生产容器 `docker inspect` Args 已核对逐字一致，
|
||||
恢复即完全复现。全程停机 ~2-2.5h（部署+13 点压测+质量门+恢复）。
|
||||
@ -1,4 +0,0 @@
|
||||
302f15789d891aa39620209286252fed ./README.md
|
||||
ce1962554abf73c6ad953aa0ac3f20d7 ./config.json
|
||||
5e170425c97cda8f798c74041979a569 ./configuration.json
|
||||
32c81842f12e56e6ac2a1feaafd5bfa7 ./model.safetensors
|
||||
@ -1,58 +0,0 @@
|
||||
434f8692a4914f39714df2c4493c92d7 ./.gitattributes
|
||||
245254422e0ca504a5f09e37719ca718 ./LICENSE
|
||||
02e0c207304666abdd93de6a3780c52a ./README.md
|
||||
e4c9e1000a513f680dd796f76d1097e0 ./chat_template.jinja
|
||||
d42bba3f99ce9b2705bb19c41930ea0c ./config.json
|
||||
5e170425c97cda8f798c74041979a569 ./configuration.json
|
||||
ee2560496cd373851033b3c05ee461fb ./generation_config.json
|
||||
027f9b7a7bb96f30962b5be7e4e4a5f1 ./hf_quant_config.json
|
||||
57cdec3085e0805c4ad5875f61c63b68 ./model-00030-of-00047.safetensors
|
||||
be93abf8f84e890b8dd1d4187ef540c2 ./model-00001-of-00047.safetensors
|
||||
a4a627a33e69e91491c9a3d3a7c9cba9 ./model-00031-of-00047.safetensors
|
||||
500a87d1fd1dd6a93400539227361f5d ./model-00002-of-00047.safetensors
|
||||
c732e4ee2f910af666418abd547d34cb ./model-00032-of-00047.safetensors
|
||||
68024a4367c3341402766fcc40d383f6 ./model-00003-of-00047.safetensors
|
||||
94188d8acbc7fd47ee99400507793d6a ./model-00033-of-00047.safetensors
|
||||
b5870fe5e8704e879895a7b5a3be5136 ./model-00004-of-00047.safetensors
|
||||
87b17b02643b1c14ad2e90900b3dfa89 ./model-00034-of-00047.safetensors
|
||||
f2f51d0be2c49ba6569d59d8d1db5236 ./model-00005-of-00047.safetensors
|
||||
915184c001e70fea709ffbb5242e4a8c ./model-00035-of-00047.safetensors
|
||||
fe2146b7c71775e51bad2322c52c34d1 ./model-00006-of-00047.safetensors
|
||||
56d618d4ba615cc19a0898d02d82bb52 ./model-00036-of-00047.safetensors
|
||||
03c0d5ea0471553cce658b70daed7b47 ./model-00007-of-00047.safetensors
|
||||
c76ac8dd7dbddbc4a636118ad8a5f713 ./model-00038-of-00047.safetensors
|
||||
49e246d6faa068641d2858e36b6bb5cd ./model-00008-of-00047.safetensors
|
||||
8f31262bc0a079c946a9ec1cdeb4051e ./model-00037-of-00047.safetensors
|
||||
b956d449de03a2db14fee14df7aaddac ./model-00009-of-00047.safetensors
|
||||
087456e9659f4d7c8c9a4af093b7c828 ./model-00039-of-00047.safetensors
|
||||
f94083fc155df999ab9ff27f6f67b978 ./model-00010-of-00047.safetensors
|
||||
bfd9606be27f5efbb34d85dc8fad3cea ./model-00040-of-00047.safetensors
|
||||
775a242af0663e5c65bc1b41a8930314 ./model-00011-of-00047.safetensors
|
||||
a4c70502c1470d1cd520f4f2b0f968d8 ./model-00041-of-00047.safetensors
|
||||
1d9d4ca43e411b977cded63b3447b8b8 ./model-00012-of-00047.safetensors
|
||||
f00a801a8de9a31b2ff686f6f8924d41 ./model-00042-of-00047.safetensors
|
||||
5feebc534bb09cee7b7e908aab8e2108 ./model-00013-of-00047.safetensors
|
||||
b58238047ce49362d39c36b54761c4cf ./model-00043-of-00047.safetensors
|
||||
c15c8fa6325af6ca302e10425b357199 ./model-00014-of-00047.safetensors
|
||||
fb652edaa0a450b7ce333829e363ee7d ./model.safetensors.index.json
|
||||
ce7e000a8d1c076b13346374f378e1f3 ./model-00015-of-00047.safetensors
|
||||
752f6cd2e6a4a2ea824d1b513530e0b0 ./tokenizer.json
|
||||
96ab98059044ac21eff43da3d9882689 ./model-00016-of-00047.safetensors
|
||||
3312b710133454ca7150bccad6381bd8 ./tokenizer_config.json
|
||||
82429b288fe2a97cb771e78bea60a0bd ./model-00017-of-00047.safetensors
|
||||
7605255172c381931a496b316faecdb7 ./model-00018-of-00047.safetensors
|
||||
e7fdbb1bd3e82f97831f4c1c69e0403a ./model-00019-of-00047.safetensors
|
||||
f5d355f9a737d7aa270d178a1bceeeec ./model-00020-of-00047.safetensors
|
||||
50b23a8dc4932840e5ae2062ca04b60a ./model-00021-of-00047.safetensors
|
||||
285172cbc5e900681efc5783f04a346a ./model-00022-of-00047.safetensors
|
||||
accdb4eb9e0831705bcd2d39e4f3bdcb ./model-00023-of-00047.safetensors
|
||||
9182f13eb2670db2a5312aaca2f9b1da ./model-00024-of-00047.safetensors
|
||||
198809340ec4795bdcfafdf7cd3074c7 ./model-00025-of-00047.safetensors
|
||||
da5e3a1afe95fe14c5d2d1f432f4a933 ./model-00026-of-00047.safetensors
|
||||
1d15d1696fd6d2cab3ed8726c3b25112 ./model-00027-of-00047.safetensors
|
||||
6147f7b5acb9a297eab276b16a58efe7 ./model-00028-of-00047.safetensors
|
||||
529b4048cf2f18d9eeaf8d1ce37b60cd ./model-00029-of-00047.safetensors
|
||||
ffe813a8385daf74f69f632d9437085c ./model-00044-of-00047.safetensors
|
||||
f67ddcaf1b68509fd6ab8d0399632a75 ./model-00045-of-00047.safetensors
|
||||
489dc1a568e671176a98e7cc23519c17 ./model-00046-of-00047.safetensors
|
||||
3a7abe2df1bd98e749ee8733370ea78b ./model-00047-of-00047.safetensors
|
||||
@ -1,15 +0,0 @@
|
||||
# 权重完整性清单（md5）
|
||||
|
||||
2026-09-08 在 174.1.60.1 `/data/hf_models/` 下对两个模型目录逐文件 `md5sum` 的原样输出。
|
||||
用于：新机器部署前核对权重传输完整性、或怀疑权重被改动时做漂移检测。
|
||||
|
||||
校验方法（在权重目录下）：
|
||||
|
||||
```bash
|
||||
md5sum -c /path/to/GLM-5.3-NVFP4.md5 # 清单内路径为 ./ 相对路径
|
||||
```
|
||||
|
||||
| 清单 | 模型 | 规模 | 用途 |
|
||||
|---|---|---|---|
|
||||
| `GLM-5.3-NVFP4.md5` | GLM-5.3-NVFP4 主模型 | 47 分片 + 配置共 55 文件 | 方案 A-F 全部部署的目标模型 |
|
||||
| `GLM-5.3-DFlash2.md5` | GLM-5.3-DFlash2 草稿模型 | 单分片 model.safetensors + 配置共 4 文件 | 方案 E 与方案 F 的 DFLASH 投机草稿 |
|
||||
@ -1,52 +0,0 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色3: decode 节点（方案F，跑在 6000D-2 = 174.1.60.2，8 卡）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> prefill -> [本角色] -> router。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行部署脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_s1_decode.sh
|
||||
# （launch 脚本同目录 s1_decode_launch.sh）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 配方 = 方案 E v5（TP8 + DFLASH + MRR12 + fa4 + window2048）+ PD decode flags；
|
||||
# KV 池 214,336 token（比单机 E 的 243,584 少，PD 传输缓冲占显存）
|
||||
# - 容量属性（实测判决的核心）: 池按 16k 场景定容 → 128k 仅容 1 个驻留（cc4 时 TTFT
|
||||
# 堆到 31.0s）、64k 容 3、16k 容 12（MRR12 上限）。长上下文负载该池就是瓶颈
|
||||
# - 必须挂补丁树 /data/sglang_patch_glm53:/sgl-workspace/sglang（唯一改动
|
||||
# python/sglang/srt/speculative/spec_info.py 接 build_dflash_family_disagg_draft_input，
|
||||
# DFlash PD 冷启动接线）+ 装 mooncake wheel（--no-deps，launch 脚本自装）
|
||||
# - --network host + --device /dev/infiniband + --ulimit memlock=-1，IB 设备 mlx5_0-3
|
||||
# - DFlash accept 在 12-batch verify 下掉到 1.8（单机 EAGLE 2.84）——decode 侧并发
|
||||
# verify 是 DFlash 的弱势区，场景二吞吐上限由此而来
|
||||
# - 60.2 平时空闲但 GPU7 常有外部裸金属任务（main_v2.py）：动卡前核实归属，勿清
|
||||
#
|
||||
# 实测成绩: 飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 场景二判决: "两机买 TTFT、不买吞吐"（÷2 单机等效 74-85 tok/s = A 的 75-96%，TTFT 减半）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_decode
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
ROLE=pd-decode
|
||||
NODE=174.1.60.2
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-s1-decode
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1200
|
||||
|
||||
TP=8
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=12
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma PYTHONUNBUFFERED=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --shm-size 64g --ulimit memlock=-1 --device /dev/infiniband --restart unless-stopped"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/flashkda_deploy/wheels:/mc_wheels:ro /data/sglang_patch_glm53:/sgl-workspace/sglang"
|
||||
|
||||
BOOTSTRAP="pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --speculative-draft-window-size 2048 --disaggregation-mode decode --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -1,29 +0,0 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色1: Mooncake 元数据服务 mc-master（方案F，双机 6000D-1 + 6000D-2）。
|
||||
# 四角色链之一，启动顺序强制: mc-master(60.1:50051) -> prefill(60.1:30000) -> decode(60.2:30000) -> router(60.2:31000)。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md；可执行脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_smoke_master.sh
|
||||
#
|
||||
# 关键点:
|
||||
# - 镜像自带的 mooncake_master 二进制（/opt/sglang/bin/mooncake_master），无需 GPU
|
||||
# --network host，监听 50051；prefill/decode 容器通过 MOONCAKE_MASTER=174.1.60.1:50051 注册
|
||||
# - 就绪判据: ss -tln | grep :50051（脚本 sleep 3 后检查）
|
||||
# - 拆链时必须先删本容器之外的角色再删它? 否——顺序无依赖，但生产恢复 60.1 时
|
||||
# 本容器与 glm53-pd-smoke-prefill 都要删干净、等显存排空再拉生产容器
|
||||
#
|
||||
# 实测背景（2026-09-08 方案F 双场景压测）: 13/13 干净点，质量门 6/7。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_master
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang-mooncake-master
|
||||
RUNTIME=docker
|
||||
ROLE=pd-master
|
||||
NODE=174.1.60.1
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=mc-master
|
||||
PORT=50051
|
||||
|
||||
DOCKER_FLAGS="--network host --restart unless-stopped"
|
||||
ENTRYPOINT="/opt/sglang/bin/mooncake_master"
|
||||
|
||||
READY_CHECK="ss -tln | grep :50051"
|
||||
@ -1,59 +0,0 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色2: prefill 节点（方案F，跑在 6000D-1 = 174.1.60.1，8 卡）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> [本角色] -> decode -> router。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行部署脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_probe.sh
|
||||
# （参数 = launch 脚本名；launch 脚本同目录 p1b_prefill_tp4pp2_cps8k_launch.sh 及其 radix 变体）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 拓扑 TP4 PP2（4卡/stage × 2 stage）+ DFLASH 草稿（草稿只跑 prefill 侧草稿 KV，
|
||||
# export SGLANG_DFLASH_PD_DRAFT_KV_TRANSFER=0 = 草稿 KV 不跨机传输）
|
||||
# - mem 0.78 低于单机方案（PD 模式下 mooncake 传输缓冲占显存）；cps 8192（非 D 方案的 16384）
|
||||
# - 基线 launch 脚本带 --disable-radix-cache；场景一（90% 命中）实测用 radix 变体
|
||||
# （p1b_prefill_tp4pp2_cps8k_radix_launch.sh，唯一差异 = 删掉该 flag）
|
||||
# - 必须挂补丁树 /data/sglang_patch_glm53:/sgl-workspace/sglang（含 1 个文件改动:
|
||||
# python/sglang/srt/speculative/spec_info.py 接 build_dflash_family_disagg_draft_input，
|
||||
# DFlash PD 冷启动接线；无此补丁 decode 首请求 400）
|
||||
# - 必须装 mooncake wheel /data/flashkda_deploy/wheels/mooncake_transfer_engine_cuda13-
|
||||
# 0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl（--no-deps，容器内 launch 脚本自装）
|
||||
# - --network host + --device /dev/infiniband + --ulimit memlock=-1，IB 设备 mlx5_0-3
|
||||
# - 前置硬检查: mc-master 50051 必须已监听（deploy_pd_probe.sh 自带）
|
||||
# - 60.1 是生产机（日常跑 glm53-pp4）：拉本角色前须停生产容器，测完等显存 <2000MiB
|
||||
# 再跑 deploy_glm53_pp4.sh 恢复
|
||||
#
|
||||
# 实测成绩: 飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 128k cc1 TTFT 3.91s = 六方案最低；跨机 prefill/decode 重叠使 RDMA 传输被完全吸收。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_prefill
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
ROLE=pd-prefill
|
||||
NODE=174.1.60.1
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-pd-smoke-prefill
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1500
|
||||
|
||||
TP=4
|
||||
PP=2
|
||||
MEM_FRACTION_STATIC=0.78
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma SGLANG_DFLASH_PD_DRAFT_KV_TRANSFER=0 PYTHONUNBUFFERED=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --shm-size 64g --ulimit memlock=-1 --device /dev/infiniband --restart unless-stopped"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/flashkda_deploy/wheels:/mc_wheels:ro /data/sglang_patch_glm53:/sgl-workspace/sglang"
|
||||
|
||||
BOOTSTRAP="pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
|
||||
# 场景一（90% 命中）变体: LAUNCH_ARGS 追加 --disable-radix-cache 删除（radix 开）。
|
||||
# 基线（本 profile 口径）= radix off。
|
||||
@ -1,35 +0,0 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色4: MiniLB 路由（方案F，跑在 6000D-2 = 174.1.60.2:31000）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> prefill -> decode -> [本角色]。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_smoke_router.sh
|
||||
#
|
||||
# 关键点:
|
||||
# - 镜像内自带 sglang_router（python3 -m sglang_router.launch_router），无需 GPU
|
||||
# - --prefill 参数形态: URL 后跟 prefill 侧 disaggregation-bootstrap-port（28800），
|
||||
# 这是 KV 传输握手口，漏掉则路由建立后首请求挂起；--decode 只有 URL
|
||||
# - bench 与质量门全部经 :31000/generate 打（--url http://174.1.60.2:31000/generate），
|
||||
# 即全链路含 KV transfer + DFlash 草稿
|
||||
# - usage/accept 指标 router 转发后可能缺失: 命中率读 prefill 容器日志
|
||||
# （docker logs glm53-pd-smoke-prefill，PP2 下日志计数 ×2 不影响比值），
|
||||
# accept 读 decode 容器日志
|
||||
# - 质量门脚本: 用 quality_gate_605.sh 副本 sed 's/PORT=30000/PORT=31000/' 生成
|
||||
# - 就绪判据: ss -tln | grep :31000
|
||||
#
|
||||
# 实测背景: 全链 13/13 干净点、质量门 6/7（tool call 为 parser 配置缺口，与 B/D 同口径）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_router
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang-router
|
||||
RUNTIME=docker
|
||||
ROLE=pd-router
|
||||
NODE=174.1.60.2
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=pd-smoke-router
|
||||
PORT=31000
|
||||
|
||||
DOCKER_FLAGS="--network host --restart unless-stopped --entrypoint python3"
|
||||
LAUNCH_ARGS="-m sglang_router.launch_router --pd-disaggregation --mini-lb --prefill http://174.1.60.1:30000 28800 --decode http://174.1.60.2:30000 --host 0.0.0.0 --port 31000"
|
||||
|
||||
READY_CHECK="ss -tln | grep :31000"
|
||||
@ -1,52 +0,0 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 方案 D（2026-09-08 双场景补测；6000D-1 现役生产容器 glm53-pp4 的原样配方）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_pp4.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 脚本头注释是早期 TP4PP2 版残留（写"TP4 PP2 + IndexCache"），实际配置 TP=2 PP=4，
|
||||
# 以脚本 docker run 段为准；md5 def3c64c5dc19e1d507080e3366c3762
|
||||
# - 串行 prefill 有效速率 6.5-6.7k tok/s 为各方案最高；KV 池 1,040,384 token
|
||||
# （A 的 3.76 倍，≈61.6 请求驻留，超过 MRR 48——池在高并发下不构成约束）
|
||||
# - 场景二（16k 独立输入 cc8-64）成立：cc32 起反超 TP4PP2（方案B），cc40/64 输出
|
||||
# 203.8/208.2 tok/s；TTFT p50 五档全档低于 B
|
||||
# - 场景一（90% 命中长上下文）8/8 全败：TP2 长上下文每卡 KV 读量翻倍 + PP4 低并发
|
||||
# 流水空泡，单请求 decode 仅 16-19 tok/s。生产态 radix off 前缀命中恒 0，实际表现
|
||||
# 比报告 D 行（radix-on 最好情况）更差——长上下文/共享前缀负载勿用
|
||||
# - DSA 实测：上下文长度不影响 TPOT（52.8ms 恒定），并发才是驱动
|
||||
# - 上生产须补 --tool-call-parser glm47 与 --reasoning-parser glm45
|
||||
# - bench 口径：场景二 cc40/64 用 nreq 80/128（与其他方案 40/64 单轮满波不同，
|
||||
# 已在报告表注声明）
|
||||
#
|
||||
# 实测成绩：飞书《GLM-5.3-NVFP4 双场景压测报告》方案 D 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 质量：GSM8K×5 + 中文推理通过（6/7，tool call 为 parser 配置缺口）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp2pp4
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-pp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=16384
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -1,44 +0,0 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 + hicache deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 2026-09-09 128k 低并发容量扩容实验优胜配置（60.8 现役；60.5 交付 deploy_glm53_605_v2.sh 待执行）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_128k_capacity_topology/scripts/deploy_glm53_605_v2.sh
|
||||
#
|
||||
# 与方案 D（glm53_nvfp4_pro6000_sglang_tp2pp4.env）的关键差异（勿混淆）：
|
||||
# - radix/hicache 保持开启（60.5 真实流量命中率 90%+，关 radix 不可接受）；D 为 16k 独立输入场景关了 radix
|
||||
# - cu13 镜像 + 9 补丁只读挂载（de-GLOO request_receiver / decode_cuda_graph_runner_fix 等 r37 栈遗产，
|
||||
# nomtp 下 spec 相关补丁为惰性，de-GLOO 为 PP 通用修复）；D 用旧镜像 20260828 无挂载
|
||||
# - --reasoning-parser glm45 --tool-call-parser glm47 齐备（质量门 7/7）；D 当时 6/7
|
||||
# - --context-length 1048576（模型原生 1M；D 未设）；chunk 8192（D 16384）；MRR 16（D 48）
|
||||
# - 实测（60.8，i128k/o512 冷缓存）：KV 池 909,632 token（A 的 3.29×）、并发上限 c6、
|
||||
# 单条上限 ~909k（900k 实跑通过）、c4 输入/输出 5,907/23.1 tok/s、TTFT p50 43.4s、
|
||||
# 512k 单条 TTFT 88.4s、90% 命中 c4 输入 17,625 tok/s
|
||||
# - memfrac 0.85 为验证档；PP0 stage 空闲 22GB 提示 0.88 有余量（未验证，改动须重跑质量门+容量冒烟）
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_128k_capacity_topology
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1800
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=16
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
CONTEXT_LENGTH=1048576
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 SGLANG_PP_DEGLOO=1 SGLANG_PP_SPEC_FORCE_EAGER_DRAFT=1 SGLANG_PP_FORCE_EAGER_VERIFY=0 SGLANG_PP_SPEC_DEBUG=0"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host --cap-add SYS_PTRACE -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models + 9 patch ro-mounts (full list in scripts/deploy_glm53_605_v2.sh; files live in /root + /root/sglang_patch2 on the host)"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --reasoning-parser glm45 --tool-call-parser glm47 --enable-hierarchical-cache --hicache-ratio 3 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length ${CONTEXT_LENGTH} --host 0.0.0.0 --port ${PORT}"
|
||||
@ -1,49 +0,0 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 + latest镜像 + autotune profile（方案 D 升级版）。
|
||||
# 2026-09-09 DSV4 优化点迁移实验优胜配置：五点(16k/512 cc8-64)全胜 +1.2~+3.2%，
|
||||
# A/B/A 回切确认因果成立。见 experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/。
|
||||
# 已入档未执行；60.1 生产仍跑 nightly 版（glm53-pp4，见同目录 tp2pp4.env）。
|
||||
#
|
||||
# 关键点（实测，勿随意改）:
|
||||
# - 相对 tp2pp4.env 仅两处变化：镜像 nightly-dev-20260828 → latest(d6e72886)；
|
||||
# 删 --disable-flashinfer-autotune。启动参数其余逐字相同
|
||||
# - 镜像中性实测：latest 与 nightly 同配置五点差 ≤0.7%（换镜像无风险无收益）
|
||||
# - autotune 增益钉在 tactic 缓存上：SGLANG_CACHE_DIR 与 flashinfer JIT 缓存
|
||||
# 必须挂宿主持久盘，否则重部署重抽签（增益消失/不可复现，DSV4 §12.6 同款教训）
|
||||
# - 代价：available_gpu_mem 15.64→11.79 GB（tactic 缓冲 ≈3.9GB/卡），
|
||||
# KV 池不缩（1,040,384）
|
||||
# - PCIe-IPC AllReduce 包在同一实验中五点全降（-0.4~-3.5%）判负勿叠用：
|
||||
# TP2 单对端 NCCL AR 同 switch P2P 已近最优
|
||||
# - 上生产须补 --tool-call-parser glm47 与 --reasoning-parser glm45（D 系既有缺口）
|
||||
# - 部署器：experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/scripts/deploy_glm53_exp.sh
|
||||
# （EXP_NAME=<名> EXP_IMAGE=lmsysorg/sglang:latest AUTOTUNE=1）
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp2pp4_latest_autotune
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:latest
|
||||
DOCKER_IMAGE_DIGEST=sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9
|
||||
CONTAINER_NAME=glm53-pp4-autotune
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=16384
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 SGLANG_CACHE_DIR=/root/.cache/sglang"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/glm53_exp/fi_jit_cache:/root/.cache/flashinfer /data/glm53_exp/sglang_cache:/root/.cache/sglang /data/glm53_exp/triton_cache:/root/.triton"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -1,47 +0,0 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=4 PP=2 + hicache deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 2026-09-09 hit90 场景实验优胜配置（60.8 现役；60.5 交付 deploy_glm53_605_v3.sh 待执行）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/scripts/deploy_glm53_605_v3.sh
|
||||
# 60.8 启动口径：bash /root/deploy_ppmtp_r37.sh '--tp 4 --pp-size 2 --disable-overlap-schedule
|
||||
# --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length 1048576' nomtp 8192 0.90 1 1 0 0
|
||||
#
|
||||
# 与 TP2PP4-hicache（glm53_nvfp4_pro6000_sglang_tp2pp4_hicache.env）的关键差异（勿混淆）：
|
||||
# - tp2/pp4 → tp4/pp2，memfrac 0.85 → 0.90；其余（cu13+9 挂载、chunk 8192、MRR 16、
|
||||
# hicache 3、ctx 1048576、双 parser）逐项同构
|
||||
# - 实测 KV 池 647,040（fp8 KV 18.52GB/rank，PP0 初始化后剩 10.94GB）
|
||||
# - 取胜依据（hit90=90% 命中 i128k/o512 主场景）：out cc1/2/3/4 = 28.8/47.5/63.9/76.2、
|
||||
# cc8/16 = 106.4/125.7（vs TP2PP4：cc4 +14%/cc8 +15%/cc16 +1%）；cap cc2/cc4 = 62.9/98.5
|
||||
# 零排队；质量门 7/7
|
||||
# - 让步项（知情选择）：并发独立 128k 文档 4 条（TP2PP4 为 6）；512k 单条 151.9s
|
||||
# （TP2PP4 为 114.4s，PP4 单条巨请求 prefill 流水更优）；无投机解码
|
||||
# - 判决背景：DP attention 对本模型容量负收益、DCP 对 DSA 静默算错、MTP@128k accept 2.07 判负
|
||||
# （见 experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/README.md）
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_hit90_dp_dcp_bench
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1800
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=4
|
||||
PP=2
|
||||
MEM_FRACTION_STATIC=0.90
|
||||
MAX_RUNNING_REQUESTS=16
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
CONTEXT_LENGTH=1048576
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 SGLANG_PP_DEGLOO=1 SGLANG_PP_SPEC_FORCE_EAGER_DRAFT=1 SGLANG_PP_FORCE_EAGER_VERIFY=0 SGLANG_PP_SPEC_DEBUG=0"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host --cap-add SYS_PTRACE -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models + 9 patch ro-mounts (full list in scripts/deploy_glm53_605_v3.sh; files live in /root + /root/sglang_patch2 on the host, bundle md5 6922e53439991bc13feee72f3760704f)"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --reasoning-parser glm45 --tool-call-parser glm47 --enable-hierarchical-cache --hicache-ratio 3 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length ${CONTEXT_LENGTH} --host 0.0.0.0 --port ${PORT}"
|
||||
@ -1,43 +0,0 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=4 PP=2 + IndexCache deployment profile (single RTX 6000D node).
|
||||
# 场景二（16k 独立输入、cc8-32、prefill 主导）最优配置：吞吐 +41~79%、cc32 TTFT 减半
|
||||
# （对比 TP8+EAGLE 生产配置，2026-09-07 真实语料基线）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_optimal.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - index_topk_freq=4 为模型原生默认（层轴索引复用省 75% indexer，无质量损失）
|
||||
# - 禁投机解码：PP2 与投机框架不兼容（已实测）
|
||||
# - 本文件为场景二形态（--disable-radix-cache，独立输入无前缀复用）；
|
||||
# 场景一 90% 命中对比须启用 radix（唯一差异：去掉 --disable-radix-cache，
|
||||
# 见 deploy_glm53_optimal_s1.sh）
|
||||
# - mem 0.85：0.90 下 cuda graph 捕获余量不足会运行时 OOM（KV 池 569,600 = TP8 的 2.06 倍）
|
||||
# - --disable-custom-all-reduce：TP4 over PCIe 用自定义 AR 在本栈无收益
|
||||
# - 已知缺口：未带 --tool-call-parser glm47 --reasoning-parser glm45，质量门 6/7
|
||||
# （tool call 失败纯属参数缺失，非模型问题）；上生产必须补 parser
|
||||
# - 场景一（90% 命中低并发）该配置全面劣于 TP8+EAGLE（输出吞吐 −25~−65%），勿混用选型
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp4pp2_indexcache
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=4
|
||||
DP=1
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --ipc=host --shm-size 64g --entrypoint '' -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models:ro"
|
||||
|
||||
BOOTSTRAP="exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size 2 --mem-fraction-static 0.85 --max-running-requests 48 --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size 16384 --json-model-override-args '{\"index_topk_freq\": 4}' --host 0.0.0.0 --port ${PORT}"
|
||||
@ -1,52 +0,0 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=8 + DFlash2 speculative decoding profile (single RTX 6000D node, 8 GPUs).
|
||||
# 方案 E（2026-09-08 场景一补测，6000D-2）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_tp8_dflash2.sh
|
||||
# （v5 终版底稿，含四轮 OOM 战役完整教训注释；md5 5bf2b47c9349e5855b963e571e35c096）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - v5 配方核心：MRR 48→12（verify CUDA graph 4.04→0.83GB，真正起作用的杠杆）+
|
||||
# --speculative-draft-window-size 2048 + mem0.85/cps8192。四轮 OOM 根因与推导见脚本头注释
|
||||
# - DFLASH block-diffusion 草稿 7 tokens/步，draft 权重 GLM-5.3-DFlash2，fa4 draft
|
||||
# attention（fa4 会把 draft KV 强制 bf16，fp8 需换 flashinfer/triton 后端，仅省 0.35GB 未用）
|
||||
# - KV 池 243,584 token；kv fp8_e4m3 由模型配置自动带出（无需显式 flag）
|
||||
# - 底稿（本 profile LAUNCH_ARGS）radix/AR 均为禁用；场景一实测变体共四处 delta：
|
||||
# ① 去 --disable-radix-cache（90% 命中前提）② 去 --disable-custom-all-reduce
|
||||
# （v1 CAR 与方案 A 一致开启）③ 加 --context-length 270336 ④ 加 --reasoning-parser
|
||||
# glm45 --tool-call-parser glm47（质量门 7/7 的前提）
|
||||
# - 判决：场景一 8 点全部低于方案 C、7 点低于 A——DFlash accept 低于 EAGLE（同语料
|
||||
# 2.53 vs 2.84）而每步墙钟相当，劣势全在接受率。投机栈选型维持 EAGLE3，勿用
|
||||
# DFlash2 替换（性能问题非质量问题）
|
||||
#
|
||||
# 实测成绩：飞书《GLM-5.3-NVFP4 双场景压测报告》方案 E 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 质量：变体配置下质量门 7/7（GSM8K×5、中文推理、tool call 全过），DFlash 草稿无质量损失。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp8dflash2
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-tp8-dflash2
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=900
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=8
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=12
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --speculative-draft-window-size 2048 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -1,44 +0,0 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=8 + EAGLE deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 生产标准配置（174.1.60.5 团队自用在役，2026-09-07 定稿）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_605.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - SM120 必需三项：--disable-shared-experts-fusion --moe-runner-backend
|
||||
# flashinfer_cutlass --disable-flashinfer-autotune
|
||||
# - EAGLE 4/1/5 为 cc1-2 均衡点（3/1/4、5/1/6 在 ±10% 噪声内；树式 topk>1 在 DSA 上不可用；
|
||||
# EAGLE3 本 nightly 不支持 GLM-5.3）；draft 模型自动从主权重加载
|
||||
# - KV fp8_e4m3 + hicache-ratio 3 是长上下文 90% 前缀命中场景的容量前提（KV 池 276,864 token）
|
||||
# - chunk 8192：16384 已 A/B 证伪（MoE 工作区 OOM 风险，且 AR 带宽受限无次数红利）
|
||||
# - 容器入口必须 python3 -m sglang.launch_server（镜像 entrypoint 无 shebang）
|
||||
# - 重新部署前必须等显存排空（docker rm -f 异步滞留数分钟，不等会把新 KV 池压小）
|
||||
# - 场景二高并发变体（16k 独立输入 cc8-32）：仅改 --max-running-requests 32 +
|
||||
# --cuda-graph-max-bs-decode 16 --cuda-graph-bs-decode 4 8 12 16（KV 池 276,864 ÷
|
||||
# 16,896/req = 16.4 驻留上限，decode 批自然 ≤16，图覆盖到 bs16 即可，bs24/32 纯耗显存）
|
||||
# - 上线前后过质量门 quality_gate_605.sh（PASS=7/7）
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp8eagle
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=8
|
||||
DP=1
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --ipc=host --shm-size 64g --entrypoint '' -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models:ro"
|
||||
|
||||
BOOTSTRAP="exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static 0.90 --max-running-requests 16 --chunked-prefill-size 8192 --max-prefill-tokens 16384 --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --speculative-algorithm EAGLE --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 --kv-cache-dtype fp8_e4m3 --enable-hierarchical-cache --hicache-ratio 3 --cuda-graph-max-bs-decode 8 --cuda-graph-bs-decode 1 2 3 4 6 8 --cuda-graph-max-bs-prefill 8 --context-length 270336 --reasoning-parser glm45 --tool-call-parser glm47 --host 0.0.0.0 --port ${PORT}"
|
||||
@ -1,91 +0,0 @@
|
||||
#!/bin/bash
|
||||
# verify_profile.sh —— 防漂移核验：运行中容器 vs 仓库 profile 声明
|
||||
# 在目标服务器上运行（需 docker 读权限，无需 GPU）。用法：
|
||||
# bash verify_profile.sh deploy/profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp2pp4.env
|
||||
# bash verify_profile.sh <profile.env> [container_name] # container_name 缺省取 profile 的 CONTAINER_NAME
|
||||
# 核验项：①容器在跑 ②镜像 registry digest ③launch 参数（token 级比对）④端口监听。
|
||||
# PD 链角色的参数在挂载的 launch 脚本内（docker Args 只有 bash /smoke_launch.sh），
|
||||
# 第③项自动跳过并提示改用库内脚本 md5 对比（scripts/ 目录各脚本头注释有 md5）。
|
||||
set -uo pipefail
|
||||
|
||||
PROFILE="$1"
|
||||
[ -r "$PROFILE" ] || { echo "FATAL: 无法读取 profile: $PROFILE"; exit 2; }
|
||||
# profile 是声明式清单：BOOTSTRAP 行引用 ${LAUNCH_ARGS}（定义在后），set -u 下直接
|
||||
# source 会炸；临时关 -u。source 完成后 LAUNCH_ARGS 已是全部变量展开后的实参串。
|
||||
set +u
|
||||
# shellcheck disable=SC1090
|
||||
source "$PROFILE"
|
||||
set -u
|
||||
CONT="${2:-${CONTAINER_NAME:-}}"
|
||||
|
||||
fail=0
|
||||
section() { printf '\n== %s ==\n' "$1"; }
|
||||
|
||||
section "容器状态"
|
||||
if [ -z "$CONT" ]; then echo "FATAL: profile 未定义 CONTAINER_NAME 且未显式传入"; exit 2; fi
|
||||
if docker ps --format '{{.Names}}' | grep -qx "$CONT"; then
|
||||
echo "OK $CONT 在跑"
|
||||
else
|
||||
echo "FAIL $CONT 未在运行"; docker ps -a --filter "name=$CONT" --format '{{.Names}} {{.Status}}' | head -3
|
||||
exit 1
|
||||
fi
|
||||
|
||||
section "镜像 digest"
|
||||
if [ -n "${DOCKER_IMAGE_DIGEST:-}" ]; then
|
||||
# RepoDigests 在 image 对象上：container 先取 image ID，再 image inspect
|
||||
img_id=$(docker inspect --format '{{.Image}}' "$CONT" 2>/dev/null)
|
||||
repo_digests=$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$img_id" 2>/dev/null)
|
||||
digest_hex="${DOCKER_IMAGE_DIGEST#sha256:}"
|
||||
if echo "$repo_digests" | grep -q "$digest_hex"; then
|
||||
echo "OK registry digest 一致: ${DOCKER_IMAGE_DIGEST}"
|
||||
elif [ "$img_id" = "sha256:${digest_hex}" ]; then
|
||||
echo "OK image ID 一致: ${DOCKER_IMAGE_DIGEST}"
|
||||
else
|
||||
echo "FAIL digest 不一致"
|
||||
echo " profile: ${DOCKER_IMAGE_DIGEST}"
|
||||
echo " 实际 image ID: ${img_id}"
|
||||
echo " 实际 RepoDigests: $(echo "$repo_digests" | head -2 | tr '\n' ' ')"
|
||||
fail=1
|
||||
fi
|
||||
else
|
||||
echo "SKIP profile 未定义 DOCKER_IMAGE_DIGEST"
|
||||
fi
|
||||
|
||||
section "启动参数"
|
||||
args=$(docker inspect --format '{{join .Args " "}}' "$CONT" 2>/dev/null)
|
||||
if echo "$args" | grep -q launch_server; then
|
||||
actual=$(echo "$args" | sed 's/.*launch_server //')
|
||||
expected="${LAUNCH_ARGS:-}"
|
||||
tr ' ' '\n' <<<"$expected" | sed '/^$/d' | sort > /tmp/vp_expected.$$
|
||||
tr ' ' '\n' <<<"$actual" | sed '/^$/d' | sort > /tmp/vp_actual.$$
|
||||
if diff -q /tmp/vp_expected.$$ /tmp/vp_actual.$$ >/dev/null; then
|
||||
echo "OK 参数一致（token 比对，共 $(wc -l < /tmp/vp_expected.$$) 项）"
|
||||
else
|
||||
echo "FAIL 参数有漂移（< profile 声明 / > 容器实际）:"
|
||||
diff /tmp/vp_expected.$$ /tmp/vp_actual.$$ | sed 's/^/ /'
|
||||
fail=1
|
||||
fi
|
||||
rm -f /tmp/vp_expected.$$ /tmp/vp_actual.$$
|
||||
else
|
||||
echo "SKIP docker Args 为 '$args' —— 参数在挂载的 launch 脚本内"
|
||||
echo " 改用库内脚本 md5 对比：experiments/.../scripts/ 各脚本头注释"
|
||||
fi
|
||||
|
||||
section "端口 ${PORT:-?}"
|
||||
if [ -n "${PORT:-}" ]; then
|
||||
if ss -tln | grep -q ":${PORT} "; then
|
||||
echo "OK :${PORT} 在监听"
|
||||
else
|
||||
echo "FAIL :${PORT} 未监听"; fail=1
|
||||
fi
|
||||
if [ -n "${HEALTH_PATH:-}" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:${PORT}${HEALTH_PATH}" 2>/dev/null)
|
||||
[ "$code" = "200" ] && echo "OK health ${HEALTH_PATH} -> 200" || { echo "FAIL health ${HEALTH_PATH} -> ${code}"; fail=1; }
|
||||
fi
|
||||
else
|
||||
echo "SKIP profile 未定义 PORT"
|
||||
fi
|
||||
|
||||
echo
|
||||
[ $fail -eq 0 ] && echo "VERDICT: PASS" || echo "VERDICT: DRIFT DETECTED"
|
||||
exit $fail
|
||||
85
docs/Kimi-K3_DP2_TP32_PP8_16K512_C8对比.md
Normal file
85
docs/Kimi-K3_DP2_TP32_PP8_16K512_C8对比.md
Normal file
@ -0,0 +1,85 @@
|
||||
# Kimi-K3 DP2 无投机：TP32 与 PP8 对比
|
||||
|
||||
## 测试目的
|
||||
|
||||
在 8 台 RTX 6000D 上部署两个独立四节点 SGLang 副本，通过 SGLang Router `round_robin` 组成 DP2，比较两种单副本并行策略：
|
||||
|
||||
- TP32：`TP32/PP1/EP4`
|
||||
- PP8：`TP4/PP8/EP4`
|
||||
|
||||
本轮不启用投机解码，重点观察长输入场景下 TTFT 与长 Decode 的取舍。
|
||||
|
||||
## 固定口径
|
||||
|
||||
| 项目 | 配置 |
|
||||
|---|---|
|
||||
| 数据集 | SPEED-Bench `/data/speedbench16k_complete.jsonl` |
|
||||
| 请求 | 40 条，16K 输入，512 输出 |
|
||||
| 客户端并发 | C=8 |
|
||||
| Chunked Prefill | 8K |
|
||||
| MoE | EP4，FlashInfer MXFP4，A2A=none |
|
||||
| KV Cache | BF16 |
|
||||
| Prefix Cache | 关闭 |
|
||||
| Router | 两副本 `round_robin` |
|
||||
| GPU | 601-608，共 64 卡 |
|
||||
|
||||
两种配置均在正式计时前分别预热两个副本，并由 benchmark 再执行一条标准 warmup。`Peak concurrent requests=12` 是 SGLang 报告的服务端峰值，可因排队高于客户端 `max_concurrency=8`，不表示客户端发出了 12 路并发。
|
||||
|
||||
## 结果
|
||||
|
||||
| 指标 | DP2 TP32/PP1/EP4 | DP2 TP4/PP8/EP4 | PP8 相对 TP32 |
|
||||
|---|---:|---:|---:|
|
||||
| 成功请求 | 40/40 | 40/40 | 相同 |
|
||||
| Benchmark duration | 238.18s | 370.30s | +55.47% |
|
||||
| Input TPS | 2797.64 | 1799.46 | -35.68% |
|
||||
| Output TPS | 85.99 | 55.31 | -35.68% |
|
||||
| Total TPS | 2883.63 | 1854.76 | -35.68% |
|
||||
| TTFT mean | 8.35s | 7.84s | -6.15% |
|
||||
| TTFT median | 5.30s | 7.03s | +32.62% |
|
||||
| TTFT p95 | 20.79s | 11.68s | -43.82% |
|
||||
| TPOT mean | 75.93ms | 126.86ms | +67.07% |
|
||||
| TPOT median | 79.95ms | 136.09ms | +70.22% |
|
||||
| TPOT p95 | 88.06ms | 145.13ms | +64.80% |
|
||||
| E2E mean | 47.15s | 72.66s | +54.10% |
|
||||
| E2E p95 | 59.24s | 82.83s | +39.82% |
|
||||
|
||||
## 初步结论
|
||||
|
||||
PP8 继续体现了 Prefill 优势：平均 TTFT 小幅下降 6.15%，TTFT p95 降低 43.82%，长尾更稳定。这来自 TP 通信域从 32 缩到 4，并由流水线分摊模型层。
|
||||
|
||||
但 512-token Decode 暴露了 PP8 的代价。每个 token 都要依次通过 8 个 Pipeline Stage，TPOT 增加 67.07%，最终使 Output TPS 降低 35.68%、E2E 平均时延增加 54.10%。因此：
|
||||
|
||||
- 以 P 节点 TTFT 为目标时，PP8 有价值，尤其能改善 TTFT 长尾。
|
||||
- 作为不做 PD 分离的完整 16K→512 服务，TP32 综合表现更好。
|
||||
- PP8 更适合作为 PD 分离中的 P 侧配置，由独立 D 侧承担长 Decode；不能用本轮 PP8 的 E2E/TPOT 直接否定其 P 侧价值。
|
||||
|
||||
TP32 的 TTFT median 更低但 p95 明显更高，说明该配置在 DP2 排队与长 Prefill 交错时波动更大。后续如继续优化单实例，应分别报告 median 与 p95，不能只看 mean。
|
||||
|
||||
## 数据质量与日志说明
|
||||
|
||||
两轮 Router 初始命令没有挂载模型目录，因此 worker tokenizer 注册出现错误。Router 使用固定 `round_robin`，不依赖 tokenizer 做 cache-aware 路由；40/40 请求成功，两副本 Prefill 数量接近均分，未发现请求级失败或重试，所以不影响本轮 A/B 结论。脚本已补上 Router 模型只读挂载，后续运行不会再产生该注册错误。
|
||||
|
||||
满负载期间 Router 偶有 `/health` 超时，这是模型服务忙于长 Prefill/Decode 时健康响应延迟；两轮均未出现 OOM、NCCL collective failure、Traceback 或 EngineDeadError。
|
||||
|
||||
## 原始结果
|
||||
|
||||
TP32：
|
||||
|
||||
```text
|
||||
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-tp32-final-20260902-123051/
|
||||
```
|
||||
|
||||
PP8：
|
||||
|
||||
```text
|
||||
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-pp8-final-20260902-130711/
|
||||
```
|
||||
|
||||
核心 benchmark 日志分别位于：
|
||||
|
||||
```text
|
||||
bench/tp32_speedbench16k_c8_o512.log
|
||||
bench/pp8_speedbench16k_c8_o512.log
|
||||
```
|
||||
|
||||
每个结果目录同时保存完整启动命令、环境 manifest、输入校验值、Router 日志和八节点服务日志。
|
||||
229
docs/Kimi-K3_PD-DFlash_16K512_C8性能归因.md
Normal file
229
docs/Kimi-K3_PD-DFlash_16K512_C8性能归因.md
Normal file
@ -0,0 +1,229 @@
|
||||
# Kimi-K3 PD+DFlash 16K→512、C=8 性能归因
|
||||
|
||||
更新时间：2026-09-02
|
||||
|
||||
## 1. 目标与结论
|
||||
|
||||
本实验在 8 台 RTX 6000D 上，用同一镜像、同一 SPEED-Bench 请求和同一服务参数，对比：
|
||||
|
||||
- 标准 PD，无投机解码；
|
||||
- PD+DFlash；
|
||||
- 历史 PP8 双实例无投机结果，用作同数据集拓扑参考。
|
||||
|
||||
核心结论：**当前 PD+DFlash 同时放大了 P→D 交接成本和 D 侧单步成本，平均接受长度 3.69 不足以覆盖这些开销。瓶颈不是 400G Rail 被打满。**
|
||||
|
||||
与严格同口径 PD 无投机相比，PD+DFlash：
|
||||
|
||||
- P→D 每请求交接数据约从 `85.76 MB` 增至 `183.54 MB`，增加 `114.0%`；
|
||||
- D 侧 transfer wait 窗口从 `15.24 s` 增至 `22.24 s`，增加 `7.00 s`；
|
||||
- Mean TTFT 增加 `7.40 s`，其中 transfer wait 增量相当于 TTFT 增量的 `94.5%`；
|
||||
- Mean TPOT 增加 `41.9%`，Input/Output TPS 均下降 `29.0%`；
|
||||
- Nsight 中 `_fwd_kernel` 占 GPU kernel 时间 `49.8%`，NCCL AllReduce 占 `31.9%`，两者远高于 MoE/GEMM。
|
||||
|
||||
这里的 `transfer wait` 是 D 请求进入后等待 P 完成计算、bootstrap/注册并交付数据的整段窗口，不等于纯 RDMA wire time。
|
||||
|
||||
## 2. 指标口径
|
||||
|
||||
设第 `i` 个请求的发出、首 token、结束时刻为 `s_i/f_i/e_i`，输入和输出 token 数为 `I_i/O_i`，整轮 benchmark 墙钟时间为 `D`：
|
||||
|
||||
| 指标 | 计算方式 | PD 场景中的含义 |
|
||||
|---|---|---|
|
||||
| TTFT | `f_i - s_i` | Router、P 排队/Prefill、P→D 交接、D 接入及首 token 的总和 |
|
||||
| TPOT | `(e_i - f_i) / (O_i - 1)` | 首 token 后平均每个输出 token 的耗时，主要由 D 侧决定 |
|
||||
| E2E | `e_i - s_i` | 完整请求延迟 |
|
||||
| Input TPS | `ΣI_i / D` | 整个系统墙钟吞吐，不是 P 的纯 Prefill 算力 |
|
||||
| Output TPS | `ΣO_i / D` | 整个系统墙钟吞吐，不等于 `1 / Mean TPOT` |
|
||||
|
||||
DP、Router 和 PD 不改变公式，只改变时间花在哪个阶段。Profiler 本身开销很大，因此性能数字只取无 profiler 的严格 A/B；Nsight/PyTorch Trace 只用于归因。
|
||||
|
||||
## 3. 严格 A/B 配置
|
||||
|
||||
| 项目 | P 组 | D 组 |
|
||||
|---|---|---|
|
||||
| 节点 | 601-604 | 605-608 |
|
||||
| 并行 | TP4 / PP8 / EP4 | TP32 / PP1 / EP4 |
|
||||
| MoE | FlashInfer MXFP4，A2A none | FlashInfer MXFP4，A2A none |
|
||||
| Chunk | 8192 | 8192 |
|
||||
| KV Cache | BF16 | BF16 |
|
||||
| 请求上限 / CUDA Graph BS | 8 / 8 | 8 / 8 |
|
||||
| 显存比例 | 0.84 | 0.86 |
|
||||
| 传输 | Mooncake，mlx5_0/1/2/3 | Mooncake，mlx5_0/1/2/3 |
|
||||
|
||||
请求固定为同一批 SPEED-Bench 数据：40 请求，C=8，输出 512，实际输入共 666,343 token，warmup=1，seed=42，temperature=0。DFlash 的 draft token 数为 16。
|
||||
|
||||
镜像：
|
||||
|
||||
```text
|
||||
local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1
|
||||
```
|
||||
|
||||
## 4. 端到端结果
|
||||
|
||||
| 指标 | PP8 双实例无投机（历史参考） | 严格 PD 无投机 | 严格 PD+DFlash | DFlash vs PD 无投机 |
|
||||
|---|---:|---:|---:|---:|
|
||||
| 成功请求 | 40 | 40 | 40 | - |
|
||||
| Benchmark duration | 335.95 s | 255.29 s | 359.53 s | +40.8% |
|
||||
| Input TPS | 1983.46 | **2610.18** | 1853.35 | -29.0% |
|
||||
| Output TPS | 60.96 | **80.22** | 56.96 | -29.0% |
|
||||
| Mean TTFT | **6.88 s** | 15.40 s | 22.80 s | +48.1% |
|
||||
| Median TTFT | **7.04 s** | 16.03 s | 23.07 s | +43.9% |
|
||||
| P95 TTFT | **10.64 s** | 22.18 s | 40.54 s | +82.8% |
|
||||
| Mean TPOT | 116.13 ms | **63.22 ms** | 89.70 ms | +41.9% |
|
||||
| P95 TPOT | 136.01 ms | **70.45 ms** | 144.76 ms | +105.5% |
|
||||
| Mean E2E | 66.22 s | **47.70 s** | 68.63 s | +43.9% |
|
||||
| DFlash 接受长度 | - | - | 3.69 | - |
|
||||
|
||||
PP8 双实例结果使用相同数据，但缺少本轮完整命令归档，只作为拓扑参考。严格单变量结论来自后两列。
|
||||
|
||||
## 5. TTFT 瀑布拆解
|
||||
|
||||
从 P 最后一个 PP stage（PP7/TP0）和 D TP0 日志中去重，排除 warmup 后取正式 40 请求：
|
||||
|
||||
| 阶段 | PD 无投机 | PD+DFlash | 变化 |
|
||||
|---|---:|---:|---:|
|
||||
| P bootstrap | 5.18 s | 6.67 s | +28.7% |
|
||||
| P forward | 9.51 s | 14.22 s | +49.5% |
|
||||
| P queue | 1.94 s | 0.93 s | -52.3% |
|
||||
| P→D payload / 请求 | 85.76 MB | 183.54 MB | **+114.0%** |
|
||||
| D bootstrap | 5.15 ms | 4.85 ms | 基本不变 |
|
||||
| D allocation wait | 12.67 ms | 2.78 ms | 可忽略 |
|
||||
| D queue | 0.41 ms | 0.30 ms | 可忽略 |
|
||||
| D transfer wait | 15.24 s | 22.24 s | **+45.9%** |
|
||||
| D forward | 32.35 s | 46.10 s | **+42.5%** |
|
||||
|
||||
### 5.1 为什么说 PD 交接冲散了收益
|
||||
|
||||
DFlash 需要向 D 节点交接额外的 draft/KDA 状态，使每请求 payload 增加约 `97.78 MB`。D 侧 transfer wait 随之增加 `6.998 s`，而 Mean TTFT 总共增加 `7.402 s`。两者不是可相加的独立阶段，但增量高度一致，说明 TTFT 回退主要发生在 P 计算与 P→D 交接链路，而不是 D admission 排队。
|
||||
|
||||
### 5.2 为什么不能把 22.24 秒叫作 RDMA 拷贝耗时
|
||||
|
||||
该时间戳从 D 开始等待传输到请求可进入 forward，覆盖：
|
||||
|
||||
```text
|
||||
P 侧计算/准备 → Mooncake bootstrap 与 buffer 注册 → 数据传输 → D 侧完成可用
|
||||
```
|
||||
|
||||
因此它是协议端到端等待窗口。纯线速传输 `183.54 MB` 即使只用一条 400G Rail，理论时间也只有毫秒级；当前几十秒来自依赖链、同步与大量小通信，而不是字节在线上连续搬运了几十秒。
|
||||
|
||||
## 6. RDMA 与 GPU 证据
|
||||
|
||||
严格 DFlash run 的 D 节点活跃 Rail：
|
||||
|
||||
- `mlx5_0` RX P95 约 `39.25-39.38 Gbit/s`；
|
||||
- `mlx5_2` TX P95 约 `39.14-39.23 Gbit/s`；
|
||||
- 峰值约 `42 Gbit/s`；
|
||||
- 单 Rail 标称为 400G，当前峰值约为其 10%。
|
||||
|
||||
所以当前不是 raw bandwidth saturation。需要优化的是交接字节量、通信调用方式、同步依赖和通信域，而不是先增加 Rail 带宽。
|
||||
|
||||
严格 A/B 中 D 节点 GPU Util P95 均为 100%，DFlash D 节点平均利用率约 74%，并非 GPU 长期空闲等待网络；但高利用率不能区分有效计算、通信 kernel 和等待慢 rank，需结合 Nsight。
|
||||
|
||||
## 7. Nsight Systems 归因
|
||||
|
||||
Nsight 只包裹 D0 节点的 8 个本地 scheduler，保持 CUDA Graph 开启。Profile run 为 8 请求，仅用于时间线归因；其吞吐和延迟不能与无 profiler 结果比较。
|
||||
|
||||
### 7.1 全局 GPU kernel
|
||||
|
||||
| Kernel 类别 | GPU 时间 | 占比 | 调用数 |
|
||||
|---|---:|---:|---:|
|
||||
| `_fwd_kernel` | 14.59 s | **49.8%** | 4,590 |
|
||||
| NCCL AllReduce Ring LL BF16 | 9.34 s | **31.9%** | 30,600 |
|
||||
| MoE/GEMM 主 kernel | 约 1.0 s | 约 3.4% | - |
|
||||
| NCCL AllGather | 0.53 s | 1.8% | - |
|
||||
| KDA update | 0.42 s | 1.4% | - |
|
||||
|
||||
`_fwd_kernel` 通过 NVTX 相关性确定处于 DFlash target verify/draft forward 路径，但仅凭导出的 kernel 名还不能把它进一步等同为某个单一源码函数。
|
||||
|
||||
### 7.2 DFlash NVTX 阶段与 kernel
|
||||
|
||||
本轮额外标记：draft prepare、target verify、accept、ReplaySSM commit 和 draft KV materialize。
|
||||
|
||||
| NVTX 阶段 | 主要 GPU 内容 |
|
||||
|---|---|
|
||||
| Target verify | `_fwd_kernel` 51.0%；AllReduce 30.6%；MoE/GEMM 13.0%；AllGather 1.8% |
|
||||
| Draft prepare/forward/sample | AllReduce 66.8%；`_fwd_kernel` 25.4%；MoE/GEMM 4.0%；AllGather 2.3% |
|
||||
| Materialize draft KV | 主要为 GEMM/数据变换，GPU 总量较小 |
|
||||
| ReplaySSM commit / accept | GPU 总量很小，不是主瓶颈 |
|
||||
|
||||
NVTX host range 平均每个 speculative cycle：
|
||||
|
||||
- target verify：约 18.07 ms；
|
||||
- draft prepare/forward/sample：约 9.26 ms；
|
||||
- materialize draft KV：约 1.44 ms；
|
||||
- ReplaySSM commit：约 0.52 ms；
|
||||
- accept：约 0.43 ms。
|
||||
|
||||
CUDA API 侧 `cudaEventSynchronize` 共约 `24.34 s`，占 CUDA API 时间 `82.9%`；`cudaGraphLaunch` 320 次、约 `2.89 s`。这说明 CUDA Graph 已生效，但每轮 speculative cycle 仍暴露出 target/draft/NCCL 依赖的同步等待。Graph 能减少 launch overhead，不能消除 collective 和状态依赖。
|
||||
|
||||
### 7.3 PyTorch Profiler 补充结果
|
||||
|
||||
PyTorch Profiler 已导出 P0 本地 PP0/PP1 的 8 份 trace，共约 `117 MiB`。但 P/D 的 `/stop_profile` 均返回 `Internal Server Error`，D trace 未落盘；profile 期间客户端流式 TTFT 也失真。因此本报告不从这轮生成新的性能结论，保留 P trace 供后续定位 Prefill operator stack。
|
||||
|
||||
实验入口已修正：P/D 的 `stop_profile` 改为并行请求并各自限时 180 秒，避免一侧导出异常让整轮实验永久挂起。
|
||||
|
||||
## 8. 两个问题的回答
|
||||
|
||||
### 8.1 PD 的收益为什么被传输冲散
|
||||
|
||||
不是单纯“网卡慢”，而是 DFlash 改变了 PD 协议要交接的内容：payload 翻倍、P 侧准备时间增加、D 侧必须等更多状态就绪。当前 D 侧 Rail 远未饱和，因此提升网卡带宽不会按比例降低 7 秒差额。
|
||||
|
||||
### 8.2 投机解码为什么收益很少，甚至变慢
|
||||
|
||||
平均接受长度 3.69 意味着一次 draft/verify 只推进约 3.69 个 token。每轮仍需：
|
||||
|
||||
```text
|
||||
draft forward/sample
|
||||
→ target verify
|
||||
→ 大量 TP32 AllReduce
|
||||
→ KDA/ReplaySSM 状态处理
|
||||
→ accept 与 KV materialize
|
||||
```
|
||||
|
||||
Nsight 显示 target verify 和 AllReduce 已占关键路径。接受长度不足以摊薄这些成本，最终 D forward 增加 42.5%，TPOT 增加 41.9%。
|
||||
|
||||
## 9. 后续优化顺序
|
||||
|
||||
1. **先减少 PD 交接内容**：确认哪些 draft/KDA buffer 必须由 P 传输，哪些可以由 D 本地重建或延迟生成；用 payload/request 和 TTFT waterfall 验收。
|
||||
2. **降低 D 侧通信固定成本**：针对 TP32 小消息 AllReduce 比较更小通信域或 per-op Tree；必须保持同一 SPEED-Bench 复测正确性、TPOT 与接受长度。
|
||||
3. **搜索 DFlash 的收益边界**：固定 PD 拓扑，扫 draft token 数和 tree 参数，目标是提高接受长度同时减少 verify 次数；不要只看接受长度，要看 `accepted tokens / speculative cycle cost`。
|
||||
4. **最后再调 Mooncake/RDMA**：只有 HCA 接近线速或纯传输微基准异常时，才把 Rail 带宽作为主优化项。
|
||||
|
||||
## 10. 原始证据
|
||||
|
||||
严格 PD 无投机：
|
||||
|
||||
```text
|
||||
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_pd_dflash_attribution/results/kimi3-pd-nospec-ab-pmem084-20260901-1808/
|
||||
```
|
||||
|
||||
严格 PD+DFlash：
|
||||
|
||||
```text
|
||||
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_pd_dflash_attribution/results/kimi3-pd-dflash-ab-pmem084-20260901-1823/
|
||||
```
|
||||
|
||||
Nsight：
|
||||
|
||||
```text
|
||||
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_pd_dflash_attribution/results/kimi3-pd-dflash-nsys-donly-20260902-100251/
|
||||
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_pd_dflash_attribution/results/kimi3-pd-dflash-nsys-donly-20260902-100251/profiles/node_5/nsys/d0.nsys-rep
|
||||
```
|
||||
|
||||
PyTorch Profiler（仅 P0 的 PP0/PP1 trace 有效）：
|
||||
|
||||
```text
|
||||
/data/hzy/kimi3_pd_dflash_attribution/kimi3-pd-dflash-torch-retry-20260902-103042/torch/
|
||||
```
|
||||
|
||||
PP8 双实例历史参考：
|
||||
|
||||
```text
|
||||
/data/cdflash_speedbench/matrix/bench_dual_16k_o512_c8.log
|
||||
/data/cdflash_speedbench/matrix/bench_dual_16k_o512_c8.jsonl
|
||||
```
|
||||
|
||||
本地归档：
|
||||
|
||||
```text
|
||||
/Users/hzy/Desktop/infra/docs/evidence/kimi_k3_pd_dflash_attribution/
|
||||
```
|
||||
119
docs/Kimi-K3_PP与DFlash迁移审计.md
Normal file
119
docs/Kimi-K3_PP与DFlash迁移审计.md
Normal file
@ -0,0 +1,119 @@
|
||||
# Kimi-K3：从 PP + DSpark 迁移到 PP + DFlash
|
||||
|
||||
更新：2026-08-31；PP8 + PD + DFlash 集成与 GSM8K 64 题 C1/C8 验证完成。
|
||||
|
||||
## 1. 决策与范围
|
||||
|
||||
**优先推进 DFlash 的 PP + PD 适配。** #33863 的分段投影设计能复用于 DFlash，但需要补齐 DFlash worker、调度入口和异构 TP 下的 draft KV 传输。
|
||||
|
||||
已实现 DFlash PP worker、Kimi 跨 stage capture、PD 输入衔接、异构 TP 的 draft KV 传输及 Kimi FlashInfer 布局/SiTU 接入。24 项 CPU 回归及 SM120 GPU 集成回归通过。Run `pd-dflash-kvbounds1-20260831-1740` 已完成跨组 smoke 与 GSM8K 64 题 C1/C8：两轮各 64/64 请求和答案检查通过，均无输出截断；按验证步数加权的接受长度为 6.4211/6.5158，整体 Output TPS 为 33.38/117.56。八节点实验容器已清理、GPU 已释放。详见 [实现、结果与实际命令](../experiments/pro6000/kimi3_pro6000_pd_dflash_validation/README.md)。
|
||||
|
||||
暂停 EAGLE3 baseline。KV cache 保持 BF16，chunk 保持 8192。FlashInfer 使用官方 0.6.18 加已合并 #4460 的显式 backport；原版 0.6.18 wheel 尚未包含所需 CUTLASS SiTU 接口。
|
||||
|
||||
已验证拓扑：P 使用 601–604 的 TP4/PP8/EP4；D 使用 605–608 的 TP32/PP1/EP4，两组均为 FlashInfer MXFP4。这里适配的是 **P 侧 PP 生成 draft 上下文**，D 侧投机执行仍为 PP1。
|
||||
|
||||
前期源码审计快照（PR 状态为该次审计记录，不表示实时状态）：
|
||||
|
||||
| 对象 | 版本 |
|
||||
|---|---|
|
||||
| SGLang main | `3139ceaeec50868a441d68dc231663f3777e0d93` |
|
||||
| PR #33863 head | `6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5` |
|
||||
| PR 状态 | Open；维护者要求拆分 DSv4 PP/PD、DSpark、Kimi/线性注意力工作 |
|
||||
|
||||
PR 公布的准确性数据来自 DSv4 Flash / H20，不能代替本次 Kimi-K3 / 6000D / P-TP4 与 D-TP32 验证。[PR #33863](https://github.com/sgl-project/sglang/pull/33863)
|
||||
|
||||
## 2. 为什么可以迁移
|
||||
|
||||
DSparkDraftModel 继承 DFlashDraftModel。普通 Kimi draft 的 prompt 上下文处理都包含：采集多个 target 层的 hidden，拼接后线性投影，再归一化，生成 draft KV。
|
||||
|
||||
#33863 将这一步改为：
|
||||
|
||||
```text
|
||||
各 P/PP stage：采集自己负责的 hidden → 使用对应权重列做局部投影
|
||||
PP stage 间：传递并累加投影结果 [token 数, 7168]
|
||||
最后一个 P/PP stage：统一 RMSNorm → 各 draft 层 K/V 投影、K norm、RoPE → 写 draft KV
|
||||
P → D：传 target KV、KDA 状态及 draft KV
|
||||
D：使用已有 DFlash proposer / verify / commit 流程
|
||||
```
|
||||
|
||||
依据是线性运算 `concat(h_i) × W^T = sum(h_i × W_i^T)`。归一化必须在求和后做。BF16 分段累加会改变舍入顺序，因此需要比较中间张量、logits 和 token 接受行为。
|
||||
|
||||
关键接口已经加在共用 DFlash 模型中：[project_target_hidden_partial](https://github.com/sgl-project/sglang/blob/6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5/python/sglang/srt/models/dflash.py#L683)。最后阶段的归一化及 KV 写入实现位于 [write_projected_context_kv](https://github.com/sgl-project/sglang/blob/6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5/python/sglang/srt/models/dspark.py#L751)。
|
||||
|
||||
这一路线保留 P 侧 PP8 的收益，不要求把各层原始 hidden 通过 PD 网络传给 D。
|
||||
|
||||
## 3. 需要补哪些代码
|
||||
|
||||
以下路径均相对于 SGLang 的 `python/sglang/srt/`。
|
||||
|
||||
| 模块 | 已有基础 | 本次增量 |
|
||||
|---|---|---|
|
||||
| `models/kimi_k3.py` | PR 已支持各 PP stage 局部采集 DSpark 特征 | 接通 DFlash capture setter，按 DFlash checkpoint 的层号及 Kimi residual 语义采集 |
|
||||
| `models/dflash.py` | PR 已有局部线性投影 | 复用投影；最后阶段归一化后复用 DFlash 的逐层 KV 写入路径 |
|
||||
| `speculative/dflash_worker_v2.py` | 非 PP prefill 的 hidden → draft KV | 接收、转发 PP proxy；仅最后阶段写完整 draft KV；处理非末 stage 没有 logits/next token 的情况 |
|
||||
| `managers/scheduler_pp_mixin.py` | PR 的成功交集、失败并集及一致释放机制 | 将目前 DSpark 专用的 proxy/input 衔接扩展到 DFlash，保持请求顺序和完成条件一致 |
|
||||
| `speculative/spec_info.py` | 当前 PD 输入分派只有 EAGLE、DSpark | 接入 DFlash 输入构造，并验证 overlap FutureMap 与 idle batch 生命周期 |
|
||||
| `disaggregation/prefill.py`、`utils.py`、`mooncake/conn.py` | 最后 PP stage 传 draft KV、按层号配对 | 加入 draft GQA 的 head 分片/复制与收发端独立 stride，保留 target MLA/KDA 的已有路径 |
|
||||
| `arg_groups/speculative_hook.py`、`validation_hook.py` | 当前 main 拒绝 PP + DFlash | 仅放开已实现的 PD-prefill + PP 组合；D 端 PP1 范围不变 |
|
||||
|
||||
关键证据：
|
||||
|
||||
- [Kimi 局部层采集](https://github.com/sgl-project/sglang/blob/6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5/python/sglang/srt/models/kimi_k3.py#L2912)。
|
||||
- [DSpark PP 上下文处理](https://github.com/sgl-project/sglang/blob/6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5/python/sglang/srt/speculative/dspark_components/dspark_worker_v2.py#L655)。
|
||||
- [PP scheduler 的 DSpark 专用入口](https://github.com/sgl-project/sglang/blob/6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5/python/sglang/srt/managers/scheduler_pp_mixin.py#L1391)。
|
||||
- [main 的 PD 输入分派](https://github.com/sgl-project/sglang/blob/3139ceaeec50868a441d68dc231663f3777e0d93/python/sglang/srt/speculative/spec_info.py#L174)：DFlash 仍返回 None。仓库里的 `dflash_disaggregation.py` 辅助函数尚未被这个入口调用，不能仅凭文件存在认定 PD 已支持。
|
||||
|
||||
相关 [#36140](https://github.com/sgl-project/sglang/issues/36140) 记录了输入缺失和 draft 状态缺失问题；[#36277](https://github.com/sgl-project/sglang/pull/36277) 是提前拒绝不支持配置的保护性改动，不是 DFlash PD 的实现。
|
||||
|
||||
## 4. 异构 TP 的明确风险
|
||||
|
||||
605 的 `/data/hf_models/Kimi-K3-DFlash/config.json` 为 6 层 draft、8 个 KV head、head_dim=128、BF16、4096 sliding window；target capture 层是 `[19,37,54,66,78,90]`。
|
||||
|
||||
[DFlashAttention](https://github.com/sgl-project/sglang/blob/6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5/python/sglang/srt/models/dflash.py#L142) 使用 `max(1, total_kv_heads // tp_size)`：
|
||||
|
||||
| draft 布局 | 每 rank KV heads | BF16、page=64 时单层 K 的页大小 |
|
||||
|---|---:|---:|
|
||||
| P TP4 | 2 | `64 × 2 × 128 × 2 = 32768 bytes` |
|
||||
| D TP32 | 1，按 head 复制到多个 rank | `64 × 1 × 128 × 2 = 16384 bytes` |
|
||||
|
||||
PR 当前 [Mooncake flat 传输分支](https://github.com/sgl-project/sglang/blob/6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5/python/sglang/srt/disaggregation/mooncake/conn.py#L686) 按 layer ID 配对后，对源偏移、目标偏移和复制字节数都使用源端 `item_len`。Kimi hybrid MLA 会进入这条路径。层号正确只能确认“哪层到哪层”，没有解决“该层哪些 head 到哪个 rank”。
|
||||
|
||||
### CPU 验证结果
|
||||
|
||||
从该 PR 源码直接用 AST 提取 `_send_kvcache_generic` 和 `build_transfer_entry_pairs`，使用 synthetic 地址、单页索引和记录型传输函数执行。未调用 GPU、Mooncake 或 RDMA。
|
||||
|
||||
[地址规划结果与源文件 SHA256](evidence/kimi_k3_pp_dflash_pr33863_transfer_plan.json)。
|
||||
|
||||
| 用例 | 地址/长度是否符合目标布局 |
|
||||
|---|---|
|
||||
| 收发双方均为 1 KV head | 符合 |
|
||||
| P 为 2 head、D 为 1 head | 不符合：目标偏移和复制长度均按源端大条目计算 |
|
||||
|
||||
具体用例：D buffer 起始地址设为 2000000，目标 page ID=2。正确目标地址为 `2000000 + 2×16384 = 2032768`，当前函数生成 `2065536`，复制 32768 bytes 而不是单 head 页的 16384 bytes。
|
||||
|
||||
这是原 PR 的特定布局地址规划复现，未在 GPU 上故意执行越界复制。当前实现已补齐每条目布局校验及 GQA head-aware 传输，CPU 测试覆盖全部 P-TP4→D-TP32 head 映射；修复版本的完整服务已通过跨组请求和 C1/C8 评测。原 flat-copy 失败证据保留用于说明修复动机。
|
||||
|
||||
## 5. 验收与结果口径
|
||||
|
||||
1. **局部数学与协议验证已通过。** 覆盖各 stage capture、feature 顺序、空 capture stage、单次 RMSNorm、BF16/FP32 分段投影与 TP4→TP32 传输规划。CPU 方法级回归不等于完整模型 logits 等价验证。
|
||||
2. **最小 PD 请求已通过。** P TP4/PP8、D TP32/PP1，经 Router 返回 128 token，服务日志有实际 RDMA ready ACK，D Decode 使用 CUDA Graph。
|
||||
3. **GSM8K 固定 64 题 C1/C8 已完成。** 两轮各 64 个响应均成功，独立解析最终答案也全部通过；总耗时 195.42/55.55 秒，无 512-token 截断。完整逐题输出、直方图、服务日志和资源清理证据见结果目录。客户端非流式，不把总耗时换算成 TTFT/ITL。
|
||||
4. **统计口径已复核。** 接口的每题等权接受长度为 6.9433/7.0070；直方图按 verify 步数加权为 6.4211/6.5158。两者权重和首尾 token 处理不同，不混用。5-shot 与前 5 题重叠，排除后剩余 59 题也均通过；这仍是历史口径部署回归，不是独立无泄漏的模型能力评测。本轮不重跑 no-spec，因此不宣称投机加速倍数。
|
||||
|
||||
真实实验代码继续维护在 601 的 `/data/hzy/sskj` 工作区，不改同事部署目录，不再创建分支。本轮核实当前检出名为 `hzy-kimi3-pd-pp8-standard`，保持现状；独立 SGLang 源码位于 `/data/hzy/src/sglang-kimi-pp-dflash-33863`。
|
||||
|
||||
## 6. DSpark 接受长度诊断的备用路径
|
||||
|
||||
DSpark 低接受长度的问题先独立于 PD 排查。PP + PD 适配本身不会自动修复 D-only 已存在的问题。
|
||||
|
||||
| 检查项 | 目的 |
|
||||
|---|---|
|
||||
| checkpoint、tokenizer、mask token、target_layer_ids、RoPE | 排除 draft/target 配置失配及错误 hidden 来源 |
|
||||
| 相同 token 前缀下，逐位置 draft token 与 target logits | 找到拒绝从第几个位置开始，区分首 token 错位与持续预测质量问题 |
|
||||
| verify 的 KDA/SSM 状态更新、回滚与 token 位置 | 判断是否首次可用、随后状态漂移 |
|
||||
| fused/replay 路径与参考执行比较 | 将 kernel/状态管理错误与 drafter 本身质量分开 |
|
||||
| 接受统计定义、实际 proposal 数、bonus token | 统一接受长度口径，防止指标解释错误 |
|
||||
|
||||
605 的 DSpark checkpoint 为 block=7、5 层、capture `[7,23,51,67,83]`；DFlash 为 block=16、6 层、另一组 capture 与 RoPE。二者的原始平均接受长度不能单独判断实现异常，需同时看每次实际提议数、逐位置接受比例和实际吞吐。
|
||||
|
||||
本轮不重跑 no-spec 性能基线，保留此前结果作为参考。当前尚未确定 DSpark 接受长度异常的根因。
|
||||
32
docs/evidence/kimi_k3_pp_dflash_pr33863_transfer_plan.json
Normal file
32
docs/evidence/kimi_k3_pp_dflash_pr33863_transfer_plan.json
Normal file
@ -0,0 +1,32 @@
|
||||
{
|
||||
"upstream_commit": "6465a6f3d3b6c8b7fee40fba0fdc09cf5e9ca1c5",
|
||||
"scope": "Extracted upstream functions, synthetic pointer metadata, no GPU or network I/O",
|
||||
"source_sha256": {
|
||||
"transfer": "2f5a73b256e040966e181a5695fb4fa97e9fc7fb5c4760832e180b0683820cea",
|
||||
"pairing": "47bc97bfb649d6564c870113d276a1043230271f2d8924e8d41744cea12c971a"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"case": "equal_tp_control",
|
||||
"source_page_stride_bytes": 16384,
|
||||
"destination_page_stride_bytes": 16384,
|
||||
"actual_source_address": 1016384,
|
||||
"actual_destination_address": 2032768,
|
||||
"expected_destination_address": 2032768,
|
||||
"actual_copy_bytes": 16384,
|
||||
"one_destination_head_page_bytes": 16384,
|
||||
"layout_matches": true
|
||||
},
|
||||
{
|
||||
"case": "P_TP4_to_D_TP32_DFlash",
|
||||
"source_page_stride_bytes": 32768,
|
||||
"destination_page_stride_bytes": 16384,
|
||||
"actual_source_address": 1032768,
|
||||
"actual_destination_address": 2065536,
|
||||
"expected_destination_address": 2032768,
|
||||
"actual_copy_bytes": 32768,
|
||||
"one_destination_head_page_bytes": 16384,
|
||||
"layout_matches": false
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -1,11 +0,0 @@
|
||||
# GLM-5.2 W4A8 P800 single-node smoke benchmark
|
||||
|
||||
- Host: `sskj-P800-4` (`gpu051`)
|
||||
- Image: `iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746`
|
||||
- Model: `/data2/GLM-5.2-w4a8`
|
||||
- Parallelism: TP=8, EP=8, DP=1
|
||||
- Quantization: `w4a8_int4`; KV cache: `int8`
|
||||
- Dataset: `/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json`
|
||||
- Benchmark: ShareGPT, 64 prompts, concurrency 8, output length 128, request rate unlimited
|
||||
|
||||
Run `bash start_glm52_w4a8.sh`, wait for `http://127.0.0.1:30000/health`, then run `bash run_bench.sh`.
|
||||
File diff suppressed because one or more lines are too long
@ -1,231 +0,0 @@
|
||||
[
|
||||
{
|
||||
"Id": "5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180",
|
||||
"Created": "2026-08-17T15:52:21.087868771Z",
|
||||
"Path": "bash",
|
||||
"Args": [
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"State": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 137722,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-08-17T15:52:21.126324465Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"Image": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"ResolvConfPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/resolv.conf",
|
||||
"HostnamePath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/hostname",
|
||||
"HostsPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/hosts",
|
||||
"LogPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180-json.log",
|
||||
"Name": "/glm52-w4a8-tp8",
|
||||
"RestartCount": 0,
|
||||
"Driver": "overlayfs",
|
||||
"Platform": "linux",
|
||||
"MountLabel": "",
|
||||
"ProcessLabel": "",
|
||||
"AppArmorProfile": "unconfined",
|
||||
"ExecIDs": null,
|
||||
"HostConfig": {
|
||||
"Binds": [
|
||||
"/data2/GLM-5.2-w4a8:/models:ro",
|
||||
"/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json:/workspace/ShareGPT_V3_unfiltered_cleaned_split.json:ro",
|
||||
"/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke:/workspace/run"
|
||||
],
|
||||
"ContainerIDFile": "",
|
||||
"LogConfig": {
|
||||
"Type": "json-file",
|
||||
"Config": {}
|
||||
},
|
||||
"NetworkMode": "host",
|
||||
"PortBindings": {},
|
||||
"RestartPolicy": {
|
||||
"Name": "no",
|
||||
"MaximumRetryCount": 0
|
||||
},
|
||||
"AutoRemove": false,
|
||||
"VolumeDriver": "",
|
||||
"VolumesFrom": null,
|
||||
"ConsoleSize": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"CapAdd": null,
|
||||
"CapDrop": null,
|
||||
"CgroupnsMode": "private",
|
||||
"Dns": null,
|
||||
"DnsOptions": [],
|
||||
"DnsSearch": [],
|
||||
"ExtraHosts": null,
|
||||
"GroupAdd": null,
|
||||
"IpcMode": "host",
|
||||
"Cgroup": "",
|
||||
"Links": null,
|
||||
"OomScoreAdj": 0,
|
||||
"PidMode": "",
|
||||
"Privileged": true,
|
||||
"PublishAllPorts": false,
|
||||
"ReadonlyRootfs": false,
|
||||
"SecurityOpt": [
|
||||
"seccomp=unconfined",
|
||||
"label=disable"
|
||||
],
|
||||
"UTSMode": "host",
|
||||
"UsernsMode": "",
|
||||
"ShmSize": 274877906944,
|
||||
"Runtime": "runc",
|
||||
"Isolation": "",
|
||||
"CpuShares": 0,
|
||||
"Memory": 0,
|
||||
"NanoCpus": 0,
|
||||
"CgroupParent": "",
|
||||
"BlkioWeight": 0,
|
||||
"BlkioWeightDevice": [],
|
||||
"BlkioDeviceReadBps": [],
|
||||
"BlkioDeviceWriteBps": [],
|
||||
"BlkioDeviceReadIOps": [],
|
||||
"BlkioDeviceWriteIOps": [],
|
||||
"CpuPeriod": 0,
|
||||
"CpuQuota": 0,
|
||||
"CpuRealtimePeriod": 0,
|
||||
"CpuRealtimeRuntime": 0,
|
||||
"CpusetCpus": "",
|
||||
"CpusetMems": "",
|
||||
"Devices": [],
|
||||
"DeviceCgroupRules": null,
|
||||
"DeviceRequests": null,
|
||||
"MemoryReservation": 0,
|
||||
"MemorySwap": 0,
|
||||
"MemorySwappiness": null,
|
||||
"OomKillDisable": null,
|
||||
"PidsLimit": null,
|
||||
"Ulimits": [
|
||||
{
|
||||
"Name": "core",
|
||||
"Hard": -1,
|
||||
"Soft": -1
|
||||
}
|
||||
],
|
||||
"CpuCount": 0,
|
||||
"CpuPercent": 0,
|
||||
"IOMaximumIOps": 0,
|
||||
"IOMaximumBandwidth": 0,
|
||||
"MaskedPaths": null,
|
||||
"ReadonlyPaths": null
|
||||
},
|
||||
"Storage": {
|
||||
"RootFS": {
|
||||
"Snapshot": {
|
||||
"Name": "overlayfs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data2/GLM-5.2-w4a8",
|
||||
"Destination": "/models",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Destination": "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke",
|
||||
"Destination": "/workspace/run",
|
||||
"Mode": "",
|
||||
"RW": true,
|
||||
"Propagation": "rprivate"
|
||||
}
|
||||
],
|
||||
"Config": {
|
||||
"Hostname": "gpu051",
|
||||
"Domainname": "",
|
||||
"User": "",
|
||||
"AttachStdin": false,
|
||||
"AttachStdout": false,
|
||||
"AttachStderr": false,
|
||||
"Tty": false,
|
||||
"OpenStdin": false,
|
||||
"StdinOnce": false,
|
||||
"Env": [
|
||||
"CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"CUDA_DISABLE_PRINTF=1",
|
||||
"GLOO_SOCKET_TIMEOUT=60000",
|
||||
"PATH=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/torch_xmlir/xre/bin/:/root/miniconda/bin:/root/.cargo/bin:/usr/local/go/bin:/usr/local/cuda-11.7/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||
"TZ=Asia/Shanghai",
|
||||
"LANG=zh_CN.UTF-8"
|
||||
],
|
||||
"Cmd": [
|
||||
"bash",
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"Image": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"Volumes": null,
|
||||
"WorkingDir": "/workspace/run",
|
||||
"Entrypoint": null,
|
||||
"Labels": {
|
||||
"autofix_time": "2025-12-03T10:19:26",
|
||||
"description": "infer_ubuntu2204_dev runtime + miniconda",
|
||||
"maintainer": "wenli <wenli01@baidu.com>",
|
||||
"org.opencontainers.image.ref.name": "ubuntu",
|
||||
"org.opencontainers.image.version": "22.04",
|
||||
"version": "infer_ubuntu2204_dev:miniconda_0.1"
|
||||
}
|
||||
},
|
||||
"NetworkSettings": {
|
||||
"SandboxID": "81c79e65bbac2dcf756cbff44c4a0ba01bd8be20cc29b1b7907701dfb0a30619",
|
||||
"SandboxKey": "/var/run/docker/netns/default",
|
||||
"Ports": {},
|
||||
"Networks": {
|
||||
"host": {
|
||||
"IPAMConfig": null,
|
||||
"Links": null,
|
||||
"Aliases": null,
|
||||
"DriverOpts": null,
|
||||
"GwPriority": 0,
|
||||
"NetworkID": "0e3fdd7bc7639e6dc911f6f09344de480a35fa47ce2e83ff21a8160e319f48ac",
|
||||
"EndpointID": "35fe2742654dbbedd58999151fd8a77234c95ea9204ebd61074cd27d27d85871",
|
||||
"Gateway": "",
|
||||
"IPAddress": "",
|
||||
"MacAddress": "",
|
||||
"IPPrefixLen": 0,
|
||||
"IPv6Gateway": "",
|
||||
"GlobalIPv6Address": "",
|
||||
"GlobalIPv6PrefixLen": 0,
|
||||
"DNSNames": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"ImageManifestDescriptor": {
|
||||
"mediaType": "application/vnd.oci.image.manifest.v1+json",
|
||||
"digest": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"size": 3632,
|
||||
"annotations": {
|
||||
"io.containerd.image.name": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"org.opencontainers.image.ref.name": "20260715_5746"
|
||||
},
|
||||
"platform": {
|
||||
"architecture": "amd64",
|
||||
"os": "linux"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@ -1,111 +0,0 @@
|
||||
Mon Aug 17 15:56:53 2026
|
||||
+-----------------------------------------------------------------------------+
|
||||
| XPU-SMI Driver Version: 5.19.0.0 XPU-RT Version: 5.19.0 |
|
||||
|-------------------------------+----------------------+----------------------+
|
||||
| XPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
|
||||
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | XPU-Util Compute M. |
|
||||
| | L3-Usage | SR-IOV M. |
|
||||
|===============================+======================+======================|
|
||||
| 0 P800 OAM N/A | 00000000:03:00.0 N/A | 0 |
|
||||
| N/A 42C N/A 89W / 400W | 87958MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 1 P800 OAM N/A | 00000000:05:00.0 N/A | 0 |
|
||||
| N/A 46C N/A 90W / 400W | 87958MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 2 P800 OAM N/A | 00000000:63:00.0 N/A | 0 |
|
||||
| N/A 41C N/A 90W / 400W | 87958MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 3 P800 OAM N/A | 00000000:65:00.0 N/A | 0 |
|
||||
| N/A 47C N/A 90W / 400W | 87958MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 4 P800 OAM N/A | 00000000:83:00.0 N/A | 0 |
|
||||
| N/A 41C N/A 86W / 400W | 88310MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 5 P800 OAM N/A | 00000000:85:00.0 N/A | 0 |
|
||||
| N/A 47C N/A 88W / 400W | 87958MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 6 P800 OAM N/A | 00000000:A3:00.0 N/A | 0 |
|
||||
| N/A 43C N/A 89W / 400W | 87914MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 7 P800 OAM N/A | 00000000:A5:00.0 N/A | 0 |
|
||||
| N/A 50C N/A 90W / 400W | 88292MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
|
||||
+-----------------------------------------------------------------------------+
|
||||
| Processes: |
|
||||
| XPU XI CI PID Type Process name XPU Memory |
|
||||
| ID ID Usage |
|
||||
|=============================================================================|
|
||||
| 0 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 0 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 0 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 0 N/A N/A 137943 C sglang::scheduler_TP3_EP3 87826MiB |
|
||||
| 0 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 0 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 0 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 0 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 1 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 1 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 1 N/A N/A 137942 C sglang::scheduler_TP2_EP2 87826MiB |
|
||||
| 1 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 1 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 1 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 1 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 1 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 2 N/A N/A 137940 C sglang::scheduler_TP0_EP0 87826MiB |
|
||||
| 2 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 2 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 2 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 2 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 2 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 2 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 2 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 3 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 3 N/A N/A 137941 C sglang::scheduler_TP1_EP1 87826MiB |
|
||||
| 3 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 3 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 3 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 3 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 3 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 3 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 4 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 4 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 4 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 4 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 4 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 4 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 4 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 4 N/A N/A 137947 C sglang::scheduler_TP7_EP7 88178MiB |
|
||||
| 5 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 5 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 5 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 5 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 5 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 5 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 5 N/A N/A 137946 C sglang::scheduler_TP6_EP6 87826MiB |
|
||||
| 5 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 6 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 6 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 6 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 6 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 6 N/A N/A 137944 C sglang::scheduler_TP4_EP4 87782MiB |
|
||||
| 6 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 6 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 6 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 7 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 7 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 7 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 7 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 7 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 7 N/A N/A 137945 C sglang::scheduler_TP5_EP5 88160MiB |
|
||||
| 7 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 7 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
+-----------------------------------------------------------------------------+
|
||||
@ -1,111 +0,0 @@
|
||||
Mon Aug 17 15:55:17 2026
|
||||
+-----------------------------------------------------------------------------+
|
||||
| XPU-SMI Driver Version: 5.19.0.0 XPU-RT Version: 5.19.0 |
|
||||
|-------------------------------+----------------------+----------------------+
|
||||
| XPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
|
||||
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | XPU-Util Compute M. |
|
||||
| | L3-Usage | SR-IOV M. |
|
||||
|===============================+======================+======================|
|
||||
| 0 P800 OAM N/A | 00000000:03:00.0 N/A | 0 |
|
||||
| N/A 39C N/A 88W / 400W | 85774MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 1 P800 OAM N/A | 00000000:05:00.0 N/A | 0 |
|
||||
| N/A 43C N/A 89W / 400W | 85774MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 2 P800 OAM N/A | 00000000:63:00.0 N/A | 0 |
|
||||
| N/A 38C N/A 90W / 400W | 85774MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 3 P800 OAM N/A | 00000000:65:00.0 N/A | 0 |
|
||||
| N/A 43C N/A 93W / 400W | 85774MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 4 P800 OAM N/A | 00000000:83:00.0 N/A | 0 |
|
||||
| N/A 39C N/A 86W / 400W | 85774MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 5 P800 OAM N/A | 00000000:85:00.0 N/A | 0 |
|
||||
| N/A 43C N/A 91W / 400W | 85774MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 6 P800 OAM N/A | 00000000:A3:00.0 N/A | 0 |
|
||||
| N/A 41C N/A 89W / 400W | 85774MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 7 P800 OAM N/A | 00000000:A5:00.0 N/A | 0 |
|
||||
| N/A 47C N/A 89W / 400W | 85772MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
|
||||
+-----------------------------------------------------------------------------+
|
||||
| Processes: |
|
||||
| XPU XI CI PID Type Process name XPU Memory |
|
||||
| ID ID Usage |
|
||||
|=============================================================================|
|
||||
| 0 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 0 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 0 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 0 N/A N/A 137943 C sglang::scheduler_TP3_EP3 85642MiB |
|
||||
| 0 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 0 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 0 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 0 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 1 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 1 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 1 N/A N/A 137942 C sglang::scheduler_TP2_EP2 85642MiB |
|
||||
| 1 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 1 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 1 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 1 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 1 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 2 N/A N/A 137940 C sglang::scheduler_TP0_EP0 85642MiB |
|
||||
| 2 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 2 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 2 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 2 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 2 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 2 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 2 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 3 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 3 N/A N/A 137941 C sglang::scheduler_TP1_EP1 85642MiB |
|
||||
| 3 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 3 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 3 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 3 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 3 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 3 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 4 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 4 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 4 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 4 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 4 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 4 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 4 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 4 N/A N/A 137947 C sglang::scheduler_TP7_EP7 85642MiB |
|
||||
| 5 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 5 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 5 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 5 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 5 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 5 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 5 N/A N/A 137946 C sglang::scheduler_TP6_EP6 85642MiB |
|
||||
| 5 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 6 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 6 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 6 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 6 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 6 N/A N/A 137944 C sglang::scheduler_TP4_EP4 85642MiB |
|
||||
| 6 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 6 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 6 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 7 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 7 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 7 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 7 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 7 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 7 N/A N/A 137945 C sglang::scheduler_TP5_EP5 85640MiB |
|
||||
| 7 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 7 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
+-----------------------------------------------------------------------------+
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -1,231 +0,0 @@
|
||||
[
|
||||
{
|
||||
"Id": "5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180",
|
||||
"Created": "2026-08-17T15:52:21.087868771Z",
|
||||
"Path": "bash",
|
||||
"Args": [
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"State": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 137722,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-08-17T15:52:21.126324465Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"Image": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"ResolvConfPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/resolv.conf",
|
||||
"HostnamePath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/hostname",
|
||||
"HostsPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/hosts",
|
||||
"LogPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180-json.log",
|
||||
"Name": "/glm52-w4a8-tp8",
|
||||
"RestartCount": 0,
|
||||
"Driver": "overlayfs",
|
||||
"Platform": "linux",
|
||||
"MountLabel": "",
|
||||
"ProcessLabel": "",
|
||||
"AppArmorProfile": "unconfined",
|
||||
"ExecIDs": null,
|
||||
"HostConfig": {
|
||||
"Binds": [
|
||||
"/data2/GLM-5.2-w4a8:/models:ro",
|
||||
"/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json:/workspace/ShareGPT_V3_unfiltered_cleaned_split.json:ro",
|
||||
"/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke:/workspace/run"
|
||||
],
|
||||
"ContainerIDFile": "",
|
||||
"LogConfig": {
|
||||
"Type": "json-file",
|
||||
"Config": {}
|
||||
},
|
||||
"NetworkMode": "host",
|
||||
"PortBindings": {},
|
||||
"RestartPolicy": {
|
||||
"Name": "no",
|
||||
"MaximumRetryCount": 0
|
||||
},
|
||||
"AutoRemove": false,
|
||||
"VolumeDriver": "",
|
||||
"VolumesFrom": null,
|
||||
"ConsoleSize": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"CapAdd": null,
|
||||
"CapDrop": null,
|
||||
"CgroupnsMode": "private",
|
||||
"Dns": null,
|
||||
"DnsOptions": [],
|
||||
"DnsSearch": [],
|
||||
"ExtraHosts": null,
|
||||
"GroupAdd": null,
|
||||
"IpcMode": "host",
|
||||
"Cgroup": "",
|
||||
"Links": null,
|
||||
"OomScoreAdj": 0,
|
||||
"PidMode": "",
|
||||
"Privileged": true,
|
||||
"PublishAllPorts": false,
|
||||
"ReadonlyRootfs": false,
|
||||
"SecurityOpt": [
|
||||
"seccomp=unconfined",
|
||||
"label=disable"
|
||||
],
|
||||
"UTSMode": "host",
|
||||
"UsernsMode": "",
|
||||
"ShmSize": 274877906944,
|
||||
"Runtime": "runc",
|
||||
"Isolation": "",
|
||||
"CpuShares": 0,
|
||||
"Memory": 0,
|
||||
"NanoCpus": 0,
|
||||
"CgroupParent": "",
|
||||
"BlkioWeight": 0,
|
||||
"BlkioWeightDevice": [],
|
||||
"BlkioDeviceReadBps": [],
|
||||
"BlkioDeviceWriteBps": [],
|
||||
"BlkioDeviceReadIOps": [],
|
||||
"BlkioDeviceWriteIOps": [],
|
||||
"CpuPeriod": 0,
|
||||
"CpuQuota": 0,
|
||||
"CpuRealtimePeriod": 0,
|
||||
"CpuRealtimeRuntime": 0,
|
||||
"CpusetCpus": "",
|
||||
"CpusetMems": "",
|
||||
"Devices": [],
|
||||
"DeviceCgroupRules": null,
|
||||
"DeviceRequests": null,
|
||||
"MemoryReservation": 0,
|
||||
"MemorySwap": 0,
|
||||
"MemorySwappiness": null,
|
||||
"OomKillDisable": null,
|
||||
"PidsLimit": null,
|
||||
"Ulimits": [
|
||||
{
|
||||
"Name": "core",
|
||||
"Hard": -1,
|
||||
"Soft": -1
|
||||
}
|
||||
],
|
||||
"CpuCount": 0,
|
||||
"CpuPercent": 0,
|
||||
"IOMaximumIOps": 0,
|
||||
"IOMaximumBandwidth": 0,
|
||||
"MaskedPaths": null,
|
||||
"ReadonlyPaths": null
|
||||
},
|
||||
"Storage": {
|
||||
"RootFS": {
|
||||
"Snapshot": {
|
||||
"Name": "overlayfs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data2/GLM-5.2-w4a8",
|
||||
"Destination": "/models",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Destination": "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke",
|
||||
"Destination": "/workspace/run",
|
||||
"Mode": "",
|
||||
"RW": true,
|
||||
"Propagation": "rprivate"
|
||||
}
|
||||
],
|
||||
"Config": {
|
||||
"Hostname": "gpu051",
|
||||
"Domainname": "",
|
||||
"User": "",
|
||||
"AttachStdin": false,
|
||||
"AttachStdout": false,
|
||||
"AttachStderr": false,
|
||||
"Tty": false,
|
||||
"OpenStdin": false,
|
||||
"StdinOnce": false,
|
||||
"Env": [
|
||||
"CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"CUDA_DISABLE_PRINTF=1",
|
||||
"GLOO_SOCKET_TIMEOUT=60000",
|
||||
"PATH=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/torch_xmlir/xre/bin/:/root/miniconda/bin:/root/.cargo/bin:/usr/local/go/bin:/usr/local/cuda-11.7/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||
"TZ=Asia/Shanghai",
|
||||
"LANG=zh_CN.UTF-8"
|
||||
],
|
||||
"Cmd": [
|
||||
"bash",
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"Image": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"Volumes": null,
|
||||
"WorkingDir": "/workspace/run",
|
||||
"Entrypoint": null,
|
||||
"Labels": {
|
||||
"autofix_time": "2025-12-03T10:19:26",
|
||||
"description": "infer_ubuntu2204_dev runtime + miniconda",
|
||||
"maintainer": "wenli <wenli01@baidu.com>",
|
||||
"org.opencontainers.image.ref.name": "ubuntu",
|
||||
"org.opencontainers.image.version": "22.04",
|
||||
"version": "infer_ubuntu2204_dev:miniconda_0.1"
|
||||
}
|
||||
},
|
||||
"NetworkSettings": {
|
||||
"SandboxID": "81c79e65bbac2dcf756cbff44c4a0ba01bd8be20cc29b1b7907701dfb0a30619",
|
||||
"SandboxKey": "/var/run/docker/netns/default",
|
||||
"Ports": {},
|
||||
"Networks": {
|
||||
"host": {
|
||||
"IPAMConfig": null,
|
||||
"Links": null,
|
||||
"Aliases": null,
|
||||
"DriverOpts": null,
|
||||
"GwPriority": 0,
|
||||
"NetworkID": "0e3fdd7bc7639e6dc911f6f09344de480a35fa47ce2e83ff21a8160e319f48ac",
|
||||
"EndpointID": "35fe2742654dbbedd58999151fd8a77234c95ea9204ebd61074cd27d27d85871",
|
||||
"Gateway": "",
|
||||
"IPAddress": "",
|
||||
"MacAddress": "",
|
||||
"IPPrefixLen": 0,
|
||||
"IPv6Gateway": "",
|
||||
"GlobalIPv6Address": "",
|
||||
"GlobalIPv6PrefixLen": 0,
|
||||
"DNSNames": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"ImageManifestDescriptor": {
|
||||
"mediaType": "application/vnd.oci.image.manifest.v1+json",
|
||||
"digest": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"size": 3632,
|
||||
"annotations": {
|
||||
"io.containerd.image.name": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"org.opencontainers.image.ref.name": "20260715_5746"
|
||||
},
|
||||
"platform": {
|
||||
"architecture": "amd64",
|
||||
"os": "linux"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@ -1,111 +0,0 @@
|
||||
Mon Aug 17 16:11:51 2026
|
||||
+-----------------------------------------------------------------------------+
|
||||
| XPU-SMI Driver Version: 5.19.0.0 XPU-RT Version: 5.19.0 |
|
||||
|-------------------------------+----------------------+----------------------+
|
||||
| XPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
|
||||
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | XPU-Util Compute M. |
|
||||
| | L3-Usage | SR-IOV M. |
|
||||
|===============================+======================+======================|
|
||||
| 0 P800 OAM N/A | 00000000:03:00.0 N/A | 0 |
|
||||
| N/A 41C N/A 89W / 400W | 87970MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 1 P800 OAM N/A | 00000000:05:00.0 N/A | 0 |
|
||||
| N/A 47C N/A 90W / 400W | 87970MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 2 P800 OAM N/A | 00000000:63:00.0 N/A | 0 |
|
||||
| N/A 40C N/A 90W / 400W | 87970MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 3 P800 OAM N/A | 00000000:65:00.0 N/A | 0 |
|
||||
| N/A 48C N/A 91W / 400W | 87968MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 4 P800 OAM N/A | 00000000:83:00.0 N/A | 0 |
|
||||
| N/A 41C N/A 86W / 400W | 88318MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 5 P800 OAM N/A | 00000000:85:00.0 N/A | 0 |
|
||||
| N/A 48C N/A 88W / 400W | 87970MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 6 P800 OAM N/A | 00000000:A3:00.0 N/A | 0 |
|
||||
| N/A 43C N/A 88W / 400W | 87922MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 7 P800 OAM N/A | 00000000:A5:00.0 N/A | 0 |
|
||||
| N/A 51C N/A 90W / 400W | 88306MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
|
||||
+-----------------------------------------------------------------------------+
|
||||
| Processes: |
|
||||
| XPU XI CI PID Type Process name XPU Memory |
|
||||
| ID ID Usage |
|
||||
|=============================================================================|
|
||||
| 0 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 0 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 0 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 0 N/A N/A 137943 C sglang::scheduler_TP3_EP3 87838MiB |
|
||||
| 0 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 0 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 0 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 0 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 1 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 1 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 1 N/A N/A 137942 C sglang::scheduler_TP2_EP2 87838MiB |
|
||||
| 1 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 1 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 1 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 1 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 1 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 2 N/A N/A 137940 C sglang::scheduler_TP0_EP0 87838MiB |
|
||||
| 2 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 2 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 2 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 2 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 2 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 2 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 2 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 3 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 3 N/A N/A 137941 C sglang::scheduler_TP1_EP1 87836MiB |
|
||||
| 3 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 3 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 3 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 3 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 3 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 3 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 4 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 4 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 4 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 4 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 4 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 4 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 4 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 4 N/A N/A 137947 C sglang::scheduler_TP7_EP7 88186MiB |
|
||||
| 5 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 5 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 5 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 5 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 5 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 5 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 5 N/A N/A 137946 C sglang::scheduler_TP6_EP6 87838MiB |
|
||||
| 5 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 6 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 6 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 6 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 6 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 6 N/A N/A 137944 C sglang::scheduler_TP4_EP4 87790MiB |
|
||||
| 6 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 6 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 6 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 7 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 7 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 7 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 7 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 7 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 7 N/A N/A 137945 C sglang::scheduler_TP5_EP5 88174MiB |
|
||||
| 7 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 7 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
+-----------------------------------------------------------------------------+
|
||||
@ -1,111 +0,0 @@
|
||||
Mon Aug 17 16:00:14 2026
|
||||
+-----------------------------------------------------------------------------+
|
||||
| XPU-SMI Driver Version: 5.19.0.0 XPU-RT Version: 5.19.0 |
|
||||
|-------------------------------+----------------------+----------------------+
|
||||
| XPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
|
||||
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | XPU-Util Compute M. |
|
||||
| | L3-Usage | SR-IOV M. |
|
||||
|===============================+======================+======================|
|
||||
| 0 P800 OAM N/A | 00000000:03:00.0 N/A | 0 |
|
||||
| N/A 39C N/A 89W / 400W | 87958MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 1 P800 OAM N/A | 00000000:05:00.0 N/A | 0 |
|
||||
| N/A 43C N/A 94W / 400W | 87958MiB / 98304MiB | 0% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 2 P800 OAM N/A | 00000000:63:00.0 N/A | 0 |
|
||||
| N/A 38C N/A 95W / 400W | 87958MiB / 98304MiB | 33% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 3 P800 OAM N/A | 00000000:65:00.0 N/A | 0 |
|
||||
| N/A 44C N/A 92W / 400W | 87958MiB / 98304MiB | 32% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 4 P800 OAM N/A | 00000000:83:00.0 N/A | 0 |
|
||||
| N/A 39C N/A 90W / 400W | 88310MiB / 98304MiB | 29% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 5 P800 OAM N/A | 00000000:85:00.0 N/A | 0 |
|
||||
| N/A 44C N/A 95W / 400W | 87958MiB / 98304MiB | 29% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 6 P800 OAM N/A | 00000000:A3:00.0 N/A | 0 |
|
||||
| N/A 41C N/A 92W / 400W | 87914MiB / 98304MiB | 30% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
| 7 P800 OAM N/A | 00000000:A5:00.0 N/A | 0 |
|
||||
| N/A 48C N/A 89W / 400W | 88292MiB / 98304MiB | 3% Default |
|
||||
| | 92MiB / 96MiB | Disabled |
|
||||
+-------------------------------+----------------------+----------------------+
|
||||
|
||||
+-----------------------------------------------------------------------------+
|
||||
| Processes: |
|
||||
| XPU XI CI PID Type Process name XPU Memory |
|
||||
| ID ID Usage |
|
||||
|=============================================================================|
|
||||
| 0 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 0 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 0 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 0 N/A N/A 137943 C sglang::scheduler_TP3_EP3 87826MiB |
|
||||
| 0 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 0 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 0 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 0 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 1 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 1 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 1 N/A N/A 137942 C sglang::scheduler_TP2_EP2 87826MiB |
|
||||
| 1 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 1 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 1 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 1 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 1 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 2 N/A N/A 137940 C sglang::scheduler_TP0_EP0 87826MiB |
|
||||
| 2 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 2 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 2 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 2 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 2 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 2 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 2 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 3 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 3 N/A N/A 137941 C sglang::scheduler_TP1_EP1 87826MiB |
|
||||
| 3 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 3 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 3 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 3 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 3 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 3 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 4 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 4 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 4 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 4 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 4 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 4 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 4 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 4 N/A N/A 137947 C sglang::scheduler_TP7_EP7 88178MiB |
|
||||
| 5 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 5 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 5 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 5 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 5 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 5 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 5 N/A N/A 137946 C sglang::scheduler_TP6_EP6 87826MiB |
|
||||
| 5 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 6 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 6 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 6 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 6 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 6 N/A N/A 137944 C sglang::scheduler_TP4_EP4 87782MiB |
|
||||
| 6 N/A N/A 137945 C sglang::scheduler_TP5_EP5 32MiB |
|
||||
| 6 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 6 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
| 7 N/A N/A 137940 C sglang::scheduler_TP0_EP0 32MiB |
|
||||
| 7 N/A N/A 137941 C sglang::scheduler_TP1_EP1 32MiB |
|
||||
| 7 N/A N/A 137942 C sglang::scheduler_TP2_EP2 32MiB |
|
||||
| 7 N/A N/A 137943 C sglang::scheduler_TP3_EP3 32MiB |
|
||||
| 7 N/A N/A 137944 C sglang::scheduler_TP4_EP4 32MiB |
|
||||
| 7 N/A N/A 137945 C sglang::scheduler_TP5_EP5 88160MiB |
|
||||
| 7 N/A N/A 137946 C sglang::scheduler_TP6_EP6 32MiB |
|
||||
| 7 N/A N/A 137947 C sglang::scheduler_TP7_EP7 32MiB |
|
||||
+-----------------------------------------------------------------------------+
|
||||
@ -1,231 +0,0 @@
|
||||
[
|
||||
{
|
||||
"Id": "5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180",
|
||||
"Created": "2026-08-17T15:52:21.087868771Z",
|
||||
"Path": "bash",
|
||||
"Args": [
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"State": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 137722,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-08-17T15:52:21.126324465Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"Image": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"ResolvConfPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/resolv.conf",
|
||||
"HostnamePath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/hostname",
|
||||
"HostsPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/hosts",
|
||||
"LogPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180-json.log",
|
||||
"Name": "/glm52-w4a8-tp8",
|
||||
"RestartCount": 0,
|
||||
"Driver": "overlayfs",
|
||||
"Platform": "linux",
|
||||
"MountLabel": "",
|
||||
"ProcessLabel": "",
|
||||
"AppArmorProfile": "unconfined",
|
||||
"ExecIDs": null,
|
||||
"HostConfig": {
|
||||
"Binds": [
|
||||
"/data2/GLM-5.2-w4a8:/models:ro",
|
||||
"/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json:/workspace/ShareGPT_V3_unfiltered_cleaned_split.json:ro",
|
||||
"/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke:/workspace/run"
|
||||
],
|
||||
"ContainerIDFile": "",
|
||||
"LogConfig": {
|
||||
"Type": "json-file",
|
||||
"Config": {}
|
||||
},
|
||||
"NetworkMode": "host",
|
||||
"PortBindings": {},
|
||||
"RestartPolicy": {
|
||||
"Name": "no",
|
||||
"MaximumRetryCount": 0
|
||||
},
|
||||
"AutoRemove": false,
|
||||
"VolumeDriver": "",
|
||||
"VolumesFrom": null,
|
||||
"ConsoleSize": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"CapAdd": null,
|
||||
"CapDrop": null,
|
||||
"CgroupnsMode": "private",
|
||||
"Dns": null,
|
||||
"DnsOptions": [],
|
||||
"DnsSearch": [],
|
||||
"ExtraHosts": null,
|
||||
"GroupAdd": null,
|
||||
"IpcMode": "host",
|
||||
"Cgroup": "",
|
||||
"Links": null,
|
||||
"OomScoreAdj": 0,
|
||||
"PidMode": "",
|
||||
"Privileged": true,
|
||||
"PublishAllPorts": false,
|
||||
"ReadonlyRootfs": false,
|
||||
"SecurityOpt": [
|
||||
"seccomp=unconfined",
|
||||
"label=disable"
|
||||
],
|
||||
"UTSMode": "host",
|
||||
"UsernsMode": "",
|
||||
"ShmSize": 274877906944,
|
||||
"Runtime": "runc",
|
||||
"Isolation": "",
|
||||
"CpuShares": 0,
|
||||
"Memory": 0,
|
||||
"NanoCpus": 0,
|
||||
"CgroupParent": "",
|
||||
"BlkioWeight": 0,
|
||||
"BlkioWeightDevice": [],
|
||||
"BlkioDeviceReadBps": [],
|
||||
"BlkioDeviceWriteBps": [],
|
||||
"BlkioDeviceReadIOps": [],
|
||||
"BlkioDeviceWriteIOps": [],
|
||||
"CpuPeriod": 0,
|
||||
"CpuQuota": 0,
|
||||
"CpuRealtimePeriod": 0,
|
||||
"CpuRealtimeRuntime": 0,
|
||||
"CpusetCpus": "",
|
||||
"CpusetMems": "",
|
||||
"Devices": [],
|
||||
"DeviceCgroupRules": null,
|
||||
"DeviceRequests": null,
|
||||
"MemoryReservation": 0,
|
||||
"MemorySwap": 0,
|
||||
"MemorySwappiness": null,
|
||||
"OomKillDisable": null,
|
||||
"PidsLimit": null,
|
||||
"Ulimits": [
|
||||
{
|
||||
"Name": "core",
|
||||
"Hard": -1,
|
||||
"Soft": -1
|
||||
}
|
||||
],
|
||||
"CpuCount": 0,
|
||||
"CpuPercent": 0,
|
||||
"IOMaximumIOps": 0,
|
||||
"IOMaximumBandwidth": 0,
|
||||
"MaskedPaths": null,
|
||||
"ReadonlyPaths": null
|
||||
},
|
||||
"Storage": {
|
||||
"RootFS": {
|
||||
"Snapshot": {
|
||||
"Name": "overlayfs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data2/GLM-5.2-w4a8",
|
||||
"Destination": "/models",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Destination": "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke",
|
||||
"Destination": "/workspace/run",
|
||||
"Mode": "",
|
||||
"RW": true,
|
||||
"Propagation": "rprivate"
|
||||
}
|
||||
],
|
||||
"Config": {
|
||||
"Hostname": "gpu051",
|
||||
"Domainname": "",
|
||||
"User": "",
|
||||
"AttachStdin": false,
|
||||
"AttachStdout": false,
|
||||
"AttachStderr": false,
|
||||
"Tty": false,
|
||||
"OpenStdin": false,
|
||||
"StdinOnce": false,
|
||||
"Env": [
|
||||
"CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"CUDA_DISABLE_PRINTF=1",
|
||||
"GLOO_SOCKET_TIMEOUT=60000",
|
||||
"PATH=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/torch_xmlir/xre/bin/:/root/miniconda/bin:/root/.cargo/bin:/usr/local/go/bin:/usr/local/cuda-11.7/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||
"TZ=Asia/Shanghai",
|
||||
"LANG=zh_CN.UTF-8"
|
||||
],
|
||||
"Cmd": [
|
||||
"bash",
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"Image": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"Volumes": null,
|
||||
"WorkingDir": "/workspace/run",
|
||||
"Entrypoint": null,
|
||||
"Labels": {
|
||||
"autofix_time": "2025-12-03T10:19:26",
|
||||
"description": "infer_ubuntu2204_dev runtime + miniconda",
|
||||
"maintainer": "wenli <wenli01@baidu.com>",
|
||||
"org.opencontainers.image.ref.name": "ubuntu",
|
||||
"org.opencontainers.image.version": "22.04",
|
||||
"version": "infer_ubuntu2204_dev:miniconda_0.1"
|
||||
}
|
||||
},
|
||||
"NetworkSettings": {
|
||||
"SandboxID": "81c79e65bbac2dcf756cbff44c4a0ba01bd8be20cc29b1b7907701dfb0a30619",
|
||||
"SandboxKey": "/var/run/docker/netns/default",
|
||||
"Ports": {},
|
||||
"Networks": {
|
||||
"host": {
|
||||
"IPAMConfig": null,
|
||||
"Links": null,
|
||||
"Aliases": null,
|
||||
"DriverOpts": null,
|
||||
"GwPriority": 0,
|
||||
"NetworkID": "0e3fdd7bc7639e6dc911f6f09344de480a35fa47ce2e83ff21a8160e319f48ac",
|
||||
"EndpointID": "35fe2742654dbbedd58999151fd8a77234c95ea9204ebd61074cd27d27d85871",
|
||||
"Gateway": "",
|
||||
"IPAddress": "",
|
||||
"MacAddress": "",
|
||||
"IPPrefixLen": 0,
|
||||
"IPv6Gateway": "",
|
||||
"GlobalIPv6Address": "",
|
||||
"GlobalIPv6PrefixLen": 0,
|
||||
"DNSNames": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"ImageManifestDescriptor": {
|
||||
"mediaType": "application/vnd.oci.image.manifest.v1+json",
|
||||
"digest": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"size": 3632,
|
||||
"annotations": {
|
||||
"io.containerd.image.name": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"org.opencontainers.image.ref.name": "20260715_5746"
|
||||
},
|
||||
"platform": {
|
||||
"architecture": "amd64",
|
||||
"os": "linux"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@ -1,231 +0,0 @@
|
||||
[
|
||||
{
|
||||
"Id": "5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180",
|
||||
"Created": "2026-08-17T15:52:21.087868771Z",
|
||||
"Path": "bash",
|
||||
"Args": [
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"State": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 194559,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-08-18T02:07:51.684912653Z",
|
||||
"FinishedAt": "2026-08-18T01:49:40.51417112Z"
|
||||
},
|
||||
"Image": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"ResolvConfPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/resolv.conf",
|
||||
"HostnamePath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/hostname",
|
||||
"HostsPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/hosts",
|
||||
"LogPath": "/data1/sora/evalscope/docker/images/containers/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180/5526e1da90d6b5fbd4304d4e407b198cf1a6b2ad76af31ee4b76077d35ece180-json.log",
|
||||
"Name": "/glm52-w4a8-tp8",
|
||||
"RestartCount": 0,
|
||||
"Driver": "overlayfs",
|
||||
"Platform": "linux",
|
||||
"MountLabel": "",
|
||||
"ProcessLabel": "",
|
||||
"AppArmorProfile": "unconfined",
|
||||
"ExecIDs": null,
|
||||
"HostConfig": {
|
||||
"Binds": [
|
||||
"/data2/GLM-5.2-w4a8:/models:ro",
|
||||
"/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json:/workspace/ShareGPT_V3_unfiltered_cleaned_split.json:ro",
|
||||
"/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke:/workspace/run"
|
||||
],
|
||||
"ContainerIDFile": "",
|
||||
"LogConfig": {
|
||||
"Type": "json-file",
|
||||
"Config": {}
|
||||
},
|
||||
"NetworkMode": "host",
|
||||
"PortBindings": {},
|
||||
"RestartPolicy": {
|
||||
"Name": "no",
|
||||
"MaximumRetryCount": 0
|
||||
},
|
||||
"AutoRemove": false,
|
||||
"VolumeDriver": "",
|
||||
"VolumesFrom": null,
|
||||
"ConsoleSize": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"CapAdd": null,
|
||||
"CapDrop": null,
|
||||
"CgroupnsMode": "private",
|
||||
"Dns": null,
|
||||
"DnsOptions": [],
|
||||
"DnsSearch": [],
|
||||
"ExtraHosts": null,
|
||||
"GroupAdd": null,
|
||||
"IpcMode": "host",
|
||||
"Cgroup": "",
|
||||
"Links": null,
|
||||
"OomScoreAdj": 0,
|
||||
"PidMode": "",
|
||||
"Privileged": true,
|
||||
"PublishAllPorts": false,
|
||||
"ReadonlyRootfs": false,
|
||||
"SecurityOpt": [
|
||||
"seccomp=unconfined",
|
||||
"label=disable"
|
||||
],
|
||||
"UTSMode": "host",
|
||||
"UsernsMode": "",
|
||||
"ShmSize": 274877906944,
|
||||
"Runtime": "runc",
|
||||
"Isolation": "",
|
||||
"CpuShares": 0,
|
||||
"Memory": 0,
|
||||
"NanoCpus": 0,
|
||||
"CgroupParent": "",
|
||||
"BlkioWeight": 0,
|
||||
"BlkioWeightDevice": [],
|
||||
"BlkioDeviceReadBps": [],
|
||||
"BlkioDeviceWriteBps": [],
|
||||
"BlkioDeviceReadIOps": [],
|
||||
"BlkioDeviceWriteIOps": [],
|
||||
"CpuPeriod": 0,
|
||||
"CpuQuota": 0,
|
||||
"CpuRealtimePeriod": 0,
|
||||
"CpuRealtimeRuntime": 0,
|
||||
"CpusetCpus": "",
|
||||
"CpusetMems": "",
|
||||
"Devices": [],
|
||||
"DeviceCgroupRules": null,
|
||||
"DeviceRequests": null,
|
||||
"MemoryReservation": 0,
|
||||
"MemorySwap": 0,
|
||||
"MemorySwappiness": null,
|
||||
"OomKillDisable": null,
|
||||
"PidsLimit": null,
|
||||
"Ulimits": [
|
||||
{
|
||||
"Name": "core",
|
||||
"Hard": -1,
|
||||
"Soft": -1
|
||||
}
|
||||
],
|
||||
"CpuCount": 0,
|
||||
"CpuPercent": 0,
|
||||
"IOMaximumIOps": 0,
|
||||
"IOMaximumBandwidth": 0,
|
||||
"MaskedPaths": null,
|
||||
"ReadonlyPaths": null
|
||||
},
|
||||
"Storage": {
|
||||
"RootFS": {
|
||||
"Snapshot": {
|
||||
"Name": "overlayfs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data2/GLM-5.2-w4a8",
|
||||
"Destination": "/models",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Destination": "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke",
|
||||
"Destination": "/workspace/run",
|
||||
"Mode": "",
|
||||
"RW": true,
|
||||
"Propagation": "rprivate"
|
||||
}
|
||||
],
|
||||
"Config": {
|
||||
"Hostname": "gpu051",
|
||||
"Domainname": "",
|
||||
"User": "",
|
||||
"AttachStdin": false,
|
||||
"AttachStdout": false,
|
||||
"AttachStderr": false,
|
||||
"Tty": false,
|
||||
"OpenStdin": false,
|
||||
"StdinOnce": false,
|
||||
"Env": [
|
||||
"CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"CUDA_DISABLE_PRINTF=1",
|
||||
"GLOO_SOCKET_TIMEOUT=60000",
|
||||
"PATH=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/torch_xmlir/xre/bin/:/root/miniconda/bin:/root/.cargo/bin:/usr/local/go/bin:/usr/local/cuda-11.7/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||
"TZ=Asia/Shanghai",
|
||||
"LANG=zh_CN.UTF-8"
|
||||
],
|
||||
"Cmd": [
|
||||
"bash",
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"Image": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"Volumes": null,
|
||||
"WorkingDir": "/workspace/run",
|
||||
"Entrypoint": null,
|
||||
"Labels": {
|
||||
"autofix_time": "2025-12-03T10:19:26",
|
||||
"description": "infer_ubuntu2204_dev runtime + miniconda",
|
||||
"maintainer": "wenli <wenli01@baidu.com>",
|
||||
"org.opencontainers.image.ref.name": "ubuntu",
|
||||
"org.opencontainers.image.version": "22.04",
|
||||
"version": "infer_ubuntu2204_dev:miniconda_0.1"
|
||||
}
|
||||
},
|
||||
"NetworkSettings": {
|
||||
"SandboxID": "02aa573ca306f134cdbd46e5e4f7df8d14de30e27b9ef68bc83e41ef6db028c3",
|
||||
"SandboxKey": "/var/run/docker/netns/default",
|
||||
"Ports": {},
|
||||
"Networks": {
|
||||
"host": {
|
||||
"IPAMConfig": null,
|
||||
"Links": null,
|
||||
"Aliases": null,
|
||||
"DriverOpts": null,
|
||||
"GwPriority": 0,
|
||||
"NetworkID": "0e3fdd7bc7639e6dc911f6f09344de480a35fa47ce2e83ff21a8160e319f48ac",
|
||||
"EndpointID": "9d8f2b98c63551f3a9576d7381f86190bf36916fceaee65c116c1cc1bd83053a",
|
||||
"Gateway": "",
|
||||
"IPAddress": "",
|
||||
"MacAddress": "",
|
||||
"IPPrefixLen": 0,
|
||||
"IPv6Gateway": "",
|
||||
"GlobalIPv6Address": "",
|
||||
"GlobalIPv6PrefixLen": 0,
|
||||
"DNSNames": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"ImageManifestDescriptor": {
|
||||
"mediaType": "application/vnd.oci.image.manifest.v1+json",
|
||||
"digest": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"size": 3632,
|
||||
"annotations": {
|
||||
"io.containerd.image.name": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"org.opencontainers.image.ref.name": "20260715_5746"
|
||||
},
|
||||
"platform": {
|
||||
"architecture": "amd64",
|
||||
"os": "linux"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@ -1,232 +0,0 @@
|
||||
[
|
||||
{
|
||||
"Id": "ec1f7a67284e4e9815888959d86a56b7749357017d8a4b0f04b409d0a1120f87",
|
||||
"Created": "2026-08-18T03:55:32.13494019Z",
|
||||
"Path": "bash",
|
||||
"Args": [
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"State": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 256057,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-08-18T03:55:32.172163502Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"Image": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"ResolvConfPath": "/data1/sora/evalscope/docker/images/containers/ec1f7a67284e4e9815888959d86a56b7749357017d8a4b0f04b409d0a1120f87/resolv.conf",
|
||||
"HostnamePath": "/data1/sora/evalscope/docker/images/containers/ec1f7a67284e4e9815888959d86a56b7749357017d8a4b0f04b409d0a1120f87/hostname",
|
||||
"HostsPath": "/data1/sora/evalscope/docker/images/containers/ec1f7a67284e4e9815888959d86a56b7749357017d8a4b0f04b409d0a1120f87/hosts",
|
||||
"LogPath": "/data1/sora/evalscope/docker/images/containers/ec1f7a67284e4e9815888959d86a56b7749357017d8a4b0f04b409d0a1120f87/ec1f7a67284e4e9815888959d86a56b7749357017d8a4b0f04b409d0a1120f87-json.log",
|
||||
"Name": "/glm52-w4a8-tp8",
|
||||
"RestartCount": 0,
|
||||
"Driver": "overlayfs",
|
||||
"Platform": "linux",
|
||||
"MountLabel": "",
|
||||
"ProcessLabel": "",
|
||||
"AppArmorProfile": "unconfined",
|
||||
"ExecIDs": null,
|
||||
"HostConfig": {
|
||||
"Binds": [
|
||||
"/data2/GLM-5.2-w4a8:/models:ro",
|
||||
"/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json:/workspace/ShareGPT_V3_unfiltered_cleaned_split.json:ro",
|
||||
"/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke:/workspace/run"
|
||||
],
|
||||
"ContainerIDFile": "",
|
||||
"LogConfig": {
|
||||
"Type": "json-file",
|
||||
"Config": {}
|
||||
},
|
||||
"NetworkMode": "host",
|
||||
"PortBindings": {},
|
||||
"RestartPolicy": {
|
||||
"Name": "no",
|
||||
"MaximumRetryCount": 0
|
||||
},
|
||||
"AutoRemove": false,
|
||||
"VolumeDriver": "",
|
||||
"VolumesFrom": null,
|
||||
"ConsoleSize": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"CapAdd": null,
|
||||
"CapDrop": null,
|
||||
"CgroupnsMode": "private",
|
||||
"Dns": null,
|
||||
"DnsOptions": [],
|
||||
"DnsSearch": [],
|
||||
"ExtraHosts": null,
|
||||
"GroupAdd": null,
|
||||
"IpcMode": "host",
|
||||
"Cgroup": "",
|
||||
"Links": null,
|
||||
"OomScoreAdj": 0,
|
||||
"PidMode": "",
|
||||
"Privileged": true,
|
||||
"PublishAllPorts": false,
|
||||
"ReadonlyRootfs": false,
|
||||
"SecurityOpt": [
|
||||
"seccomp=unconfined",
|
||||
"label=disable"
|
||||
],
|
||||
"UTSMode": "host",
|
||||
"UsernsMode": "",
|
||||
"ShmSize": 274877906944,
|
||||
"Runtime": "runc",
|
||||
"Isolation": "",
|
||||
"CpuShares": 0,
|
||||
"Memory": 0,
|
||||
"NanoCpus": 0,
|
||||
"CgroupParent": "",
|
||||
"BlkioWeight": 0,
|
||||
"BlkioWeightDevice": [],
|
||||
"BlkioDeviceReadBps": [],
|
||||
"BlkioDeviceWriteBps": [],
|
||||
"BlkioDeviceReadIOps": [],
|
||||
"BlkioDeviceWriteIOps": [],
|
||||
"CpuPeriod": 0,
|
||||
"CpuQuota": 0,
|
||||
"CpuRealtimePeriod": 0,
|
||||
"CpuRealtimeRuntime": 0,
|
||||
"CpusetCpus": "",
|
||||
"CpusetMems": "",
|
||||
"Devices": [],
|
||||
"DeviceCgroupRules": null,
|
||||
"DeviceRequests": null,
|
||||
"MemoryReservation": 0,
|
||||
"MemorySwap": 0,
|
||||
"MemorySwappiness": null,
|
||||
"OomKillDisable": null,
|
||||
"PidsLimit": null,
|
||||
"Ulimits": [
|
||||
{
|
||||
"Name": "core",
|
||||
"Hard": -1,
|
||||
"Soft": -1
|
||||
}
|
||||
],
|
||||
"CpuCount": 0,
|
||||
"CpuPercent": 0,
|
||||
"IOMaximumIOps": 0,
|
||||
"IOMaximumBandwidth": 0,
|
||||
"MaskedPaths": null,
|
||||
"ReadonlyPaths": null,
|
||||
"Init": true
|
||||
},
|
||||
"Storage": {
|
||||
"RootFS": {
|
||||
"Snapshot": {
|
||||
"Name": "overlayfs"
|
||||
}
|
||||
}
|
||||
},
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data2/GLM-5.2-w4a8",
|
||||
"Destination": "/models",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Destination": "/workspace/ShareGPT_V3_unfiltered_cleaned_split.json",
|
||||
"Mode": "ro",
|
||||
"RW": false,
|
||||
"Propagation": "rprivate"
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke",
|
||||
"Destination": "/workspace/run",
|
||||
"Mode": "",
|
||||
"RW": true,
|
||||
"Propagation": "rprivate"
|
||||
}
|
||||
],
|
||||
"Config": {
|
||||
"Hostname": "gpu051",
|
||||
"Domainname": "",
|
||||
"User": "",
|
||||
"AttachStdin": false,
|
||||
"AttachStdout": false,
|
||||
"AttachStderr": false,
|
||||
"Tty": false,
|
||||
"OpenStdin": false,
|
||||
"StdinOnce": false,
|
||||
"Env": [
|
||||
"CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7",
|
||||
"CUDA_DISABLE_PRINTF=1",
|
||||
"GLOO_SOCKET_TIMEOUT=60000",
|
||||
"PATH=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/torch_xmlir/xre/bin/:/root/miniconda/bin:/root/.cargo/bin:/usr/local/go/bin:/usr/local/cuda-11.7/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||
"TZ=Asia/Shanghai",
|
||||
"LANG=zh_CN.UTF-8"
|
||||
],
|
||||
"Cmd": [
|
||||
"bash",
|
||||
"-lc",
|
||||
"exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server --host 0.0.0.0 --port 30000 --model-path /models --served-model-name GLM-5.2-w4a8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --prefill-attention-backend klxdsa --decode-attention-backend klxdsa --trust-remote-code --disable-custom-all-reduce --disable-shared-experts-fusion --chunked-prefill-size 32768 --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 32768 --max-running-requests 64 --tp-size 8 --ep-size 8 --quantization w4a8_int4 --kv-cache-dtype int8 --cuda-graph-max-bs 8 --disable-cuda-graph-padding --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --json-model-override-args '{\"index_topk_freq\": 4}' --allow-auto-truncate --enable-metrics"
|
||||
],
|
||||
"Image": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"Volumes": null,
|
||||
"WorkingDir": "/workspace/run",
|
||||
"Entrypoint": null,
|
||||
"Labels": {
|
||||
"autofix_time": "2025-12-03T10:19:26",
|
||||
"description": "infer_ubuntu2204_dev runtime + miniconda",
|
||||
"maintainer": "wenli <wenli01@baidu.com>",
|
||||
"org.opencontainers.image.ref.name": "ubuntu",
|
||||
"org.opencontainers.image.version": "22.04",
|
||||
"version": "infer_ubuntu2204_dev:miniconda_0.1"
|
||||
}
|
||||
},
|
||||
"NetworkSettings": {
|
||||
"SandboxID": "40813e57fb14f319ce4dc5e1f06a130d63d9b1cd36916bc1ebc7c33eafe04cc2",
|
||||
"SandboxKey": "/var/run/docker/netns/default",
|
||||
"Ports": {},
|
||||
"Networks": {
|
||||
"host": {
|
||||
"IPAMConfig": null,
|
||||
"Links": null,
|
||||
"Aliases": null,
|
||||
"DriverOpts": null,
|
||||
"GwPriority": 0,
|
||||
"NetworkID": "0e3fdd7bc7639e6dc911f6f09344de480a35fa47ce2e83ff21a8160e319f48ac",
|
||||
"EndpointID": "5eb7dfb7d5c85d6d8c88dc3595bfa83deff49cff1c3bcc1ab946a9b88b4bc3d4",
|
||||
"Gateway": "",
|
||||
"IPAddress": "",
|
||||
"MacAddress": "",
|
||||
"IPPrefixLen": 0,
|
||||
"IPv6Gateway": "",
|
||||
"GlobalIPv6Address": "",
|
||||
"GlobalIPv6PrefixLen": 0,
|
||||
"DNSNames": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"ImageManifestDescriptor": {
|
||||
"mediaType": "application/vnd.oci.image.manifest.v1+json",
|
||||
"digest": "sha256:5079ea4ecbeff690457c801261cc4ef4f80cd29012b8bda5fd6742138b82069d",
|
||||
"size": 3632,
|
||||
"annotations": {
|
||||
"io.containerd.image.name": "iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746",
|
||||
"org.opencontainers.image.ref.name": "20260715_5746"
|
||||
},
|
||||
"platform": {
|
||||
"architecture": "amd64",
|
||||
"os": "linux"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@ -1,17 +0,0 @@
|
||||
8192 2048 64 c64_i8192_o2048 kv_cache_decode_oom
|
||||
8192 4096 64 c64_i8192_o4096 kv_cache_decode_oom
|
||||
16384 512 32 c32_i16384_o512 xpu_kernel_launch_timeout
|
||||
16384 2048 32 c32_i16384_o2048 out_of_memory
|
||||
16384 4096 32 c32_i16384_o4096 kv_cache_decode_oom
|
||||
32768 128 1 c1_i32768_o128 out_of_memory
|
||||
65536 128 1 c1_i65536_o128 out_of_memory
|
||||
65536 128 8 c8_i65536_o128 out_of_memory
|
||||
65536 256 1 c1_i65536_o256 out_of_memory
|
||||
65536 512 1 c1_i65536_o512 out_of_memory
|
||||
65536 1024 1 c1_i65536_o1024 out_of_memory
|
||||
65536 2048 1 c1_i65536_o2048 out_of_memory
|
||||
131072 128 1 c1_i131072_o128 out_of_memory
|
||||
131072 128 4 c4_i131072_o128 out_of_memory
|
||||
131072 256 1 c1_i131072_o256 out_of_memory
|
||||
131072 512 1 c1_i131072_o512 out_of_memory
|
||||
131072 1024 1 c1_i131072_o1024 out_of_memory
|
||||
|
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
kv_cache_decode_oom
|
||||
@ -1 +0,0 @@
|
||||
previous_failure
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1 +0,0 @@
|
||||
kv_cache_decode_oom
|
||||
@ -1 +0,0 @@
|
||||
previous_failure
|
||||
@ -1 +0,0 @@
|
||||
out_of_memory
|
||||
@ -1,93 +0,0 @@
|
||||
{
|
||||
"comment": "ISL/OSL matrix for dsv4_p800_sglang_tp_dp_matrix. Y=must test, P=optional (record skipped on failure), N=skip.",
|
||||
"mode": "Y",
|
||||
"notes": "P800 INT8 sustains at most ~131k input tokens, so ISL stops at 131072. Concurrency ranges match the H20 matrix; scenarios that exceed server capacity are recorded as failed/skipped and the run moves on.",
|
||||
"matrix": {
|
||||
"1024": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "Y"
|
||||
},
|
||||
"4096": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "Y"
|
||||
},
|
||||
"8192": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "Y"
|
||||
},
|
||||
"16384": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "P"
|
||||
},
|
||||
"32768": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "P"
|
||||
},
|
||||
"65536": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "P",
|
||||
"4096": "N"
|
||||
},
|
||||
"131072": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "P",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
}
|
||||
},
|
||||
"concurrency": {
|
||||
"1024": {
|
||||
"low": 1,
|
||||
"high": 128
|
||||
},
|
||||
"4096": {
|
||||
"low": 1,
|
||||
"high": 64
|
||||
},
|
||||
"8192": {
|
||||
"low": 1,
|
||||
"high": 64
|
||||
},
|
||||
"16384": {
|
||||
"low": 1,
|
||||
"high": 32
|
||||
},
|
||||
"32768": {
|
||||
"low": 1,
|
||||
"high": 16
|
||||
},
|
||||
"65536": {
|
||||
"low": 1,
|
||||
"high": 8
|
||||
},
|
||||
"131072": {
|
||||
"low": 1,
|
||||
"high": 4
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,79 +0,0 @@
|
||||
mark input_len output_len concurrency num_prompts
|
||||
Y 1024 128 1 5
|
||||
Y 1024 128 128 640
|
||||
Y 1024 256 1 5
|
||||
Y 1024 256 128 640
|
||||
Y 1024 512 1 5
|
||||
Y 1024 512 128 640
|
||||
Y 1024 1024 1 5
|
||||
Y 1024 1024 128 640
|
||||
Y 1024 2048 1 5
|
||||
Y 1024 2048 128 640
|
||||
Y 1024 4096 1 5
|
||||
Y 1024 4096 128 640
|
||||
Y 4096 128 1 5
|
||||
Y 4096 128 64 320
|
||||
Y 4096 256 1 5
|
||||
Y 4096 256 64 320
|
||||
Y 4096 512 1 5
|
||||
Y 4096 512 64 320
|
||||
Y 4096 1024 1 5
|
||||
Y 4096 1024 64 320
|
||||
Y 4096 2048 1 5
|
||||
Y 4096 2048 64 320
|
||||
Y 4096 4096 1 5
|
||||
Y 4096 4096 64 320
|
||||
Y 8192 128 1 5
|
||||
Y 8192 128 64 320
|
||||
Y 8192 256 1 5
|
||||
Y 8192 256 64 320
|
||||
Y 8192 512 1 5
|
||||
Y 8192 512 64 320
|
||||
Y 8192 1024 1 5
|
||||
Y 8192 1024 64 320
|
||||
Y 8192 2048 1 5
|
||||
Y 8192 2048 64 320
|
||||
Y 8192 4096 1 5
|
||||
Y 8192 4096 64 320
|
||||
Y 16384 128 1 5
|
||||
Y 16384 128 32 160
|
||||
Y 16384 256 1 5
|
||||
Y 16384 256 32 160
|
||||
Y 16384 512 1 5
|
||||
Y 16384 512 32 160
|
||||
Y 16384 1024 1 5
|
||||
Y 16384 1024 32 160
|
||||
Y 16384 2048 1 5
|
||||
Y 16384 2048 32 160
|
||||
P 16384 4096 1 5
|
||||
P 16384 4096 32 160
|
||||
Y 32768 128 1 5
|
||||
Y 32768 128 16 80
|
||||
Y 32768 256 1 5
|
||||
Y 32768 256 16 80
|
||||
Y 32768 512 1 5
|
||||
Y 32768 512 16 80
|
||||
Y 32768 1024 1 5
|
||||
Y 32768 1024 16 80
|
||||
Y 32768 2048 1 5
|
||||
Y 32768 2048 16 80
|
||||
P 32768 4096 1 5
|
||||
P 32768 4096 16 80
|
||||
Y 65536 128 1 5
|
||||
Y 65536 128 8 40
|
||||
Y 65536 256 1 5
|
||||
Y 65536 256 8 40
|
||||
Y 65536 512 1 5
|
||||
Y 65536 512 8 40
|
||||
Y 65536 1024 1 5
|
||||
Y 65536 1024 8 40
|
||||
P 65536 2048 1 5
|
||||
P 65536 2048 8 40
|
||||
Y 131072 128 1 5
|
||||
Y 131072 128 4 20
|
||||
Y 131072 256 1 5
|
||||
Y 131072 256 4 20
|
||||
Y 131072 512 1 5
|
||||
Y 131072 512 4 20
|
||||
P 131072 1024 1 5
|
||||
P 131072 1024 4 20
|
||||
|
@ -1,79 +0,0 @@
|
||||
mark input_len output_len concurrency num_prompts scenario status completed duration_s request_throughput input_throughput output_throughput total_token_throughput mean_ttft_ms p99_ttft_ms mean_tpot_ms p99_tpot_ms
|
||||
Y 1024 128 1 5 c1_i1024_o128 "completed rc=0 duration_s=39" 5 17.02591361099985 0.2936699970549406 300.7180769842592 37.5897596230324 281.2652568012709 297.18383764105965 24.568037981093877 24.614217375115466
|
||||
Y 1024 128 128 640 c128_i1024_o128 "completed rc=0 duration_s=365" 640 342.486345420999 1.8686876383736828 1913.5361416946512 239.1920177118314 41669.024385146906 45741.00833293076 183.7266033815086 187.1878648276444
|
||||
Y 1024 256 1 5 c1_i1024_o256 "completed rc=0 duration_s=55" 5 32.81497869300074 0.1523694422226297 156.0263088359728 39.0065772089932 278.39411059903796 294.2181943576725 24.633620097259097 24.658055767057437
|
||||
Y 1024 256 128 640 c128_i1024_o256 "completed rc=0 duration_s=597" 640 574.0754678120029 1.1148360030768392 1141.5920671506833 285.3980167876708 62527.62441754065 68956.42354271989 182.28569678729144 184.03258206838555
|
||||
Y 1024 512 1 5 c1_i1024_o512 "completed rc=0 duration_s=87" 5 65.10708004299886 0.07679656339522269 78.63968091670803 39.319840458354015 279.30358499943395 294.0611118795641 24.93063542035209 24.953084254637705
|
||||
Y 1024 512 128 640 c128_i1024_o512 "completed rc=0 duration_s=1065" 640 1042.2514380899993 0.6140552812983843 628.7926080495455 314.39630402477275 104649.0250889984 115665.911804529 182.58666003007582 184.05825803236687
|
||||
Y 1024 1024 1 5 c1_i1024_o1024 "completed rc=0 duration_s=153" 5 131.8887679899999 0.03791073399350513 38.82059160934925 38.82059160934925 281.0054517998651 295.132770438795 25.5066858453576 25.511918811260603
|
||||
Y 1024 1024 128 640 c128_i1024_o1024 "completed rc=0 duration_s=1983" 640 1956.396799541999 0.32713200111032015 334.98316913696783 334.98316913696783 186932.53216034532 207552.52155184935 180.55811861379217 181.34573211966844
|
||||
Y 1024 2048 1 5 c1_i1024_o2048 "completed rc=0 duration_s=290" 5 268.2550993270015 0.01863897466457861 19.086310056528497 38.17262011305699 284.6344647994556 294.75402423980995 26.06863725754749 26.112281927875404
|
||||
Y 1024 2048 128 640 c128_i1024_o2048 "completed rc=0 duration_s=3843" 640 3815.7466828769975 0.16772601883452407 171.75144328655264 343.5028865731053 354123.60505508434 393690.0640932885 181.0707906767069 181.93800417857398
|
||||
Y 1024 4096 1 5 c1_i1024_o4096 "completed rc=0 duration_s=560" 5 537.174628175002 0.009307960089230216 9.531351131371741 38.125404525486964 287.3897543991916 295.1975400408264 26.164411530549657 26.182873973002035
|
||||
Y 1024 4096 128 640 c128_i1024_o4096 "completed rc=0 duration_s=7630" 640 7595.250456256996 0.0842631857482416 86.2855022061994 345.1420088247976 694281.792547897 772382.6850643696 182.80351444481107 183.29835544574874
|
||||
Y 4096 128 1 5 c1_i4096_o128 "completed rc=0 duration_s=49" 5 23.562466644005326 0.21220189191321703 869.178949276537 27.16184216489178 1348.1614620017353 1376.2398038062383 26.4617152897642 26.536813717162214
|
||||
Y 4096 128 64 320 c64_i4096_o128 "completed rc=0 duration_s=539" 320 513.9221596100033 0.6226623896561228 2550.425148031479 79.70078587598371 52487.32021765952 81989.41991875166 395.8419637399347 719.1297852202871
|
||||
Y 4096 256 1 5 c1_i4096_o256 "completed rc=0 duration_s=65" 5 40.363449060001585 0.12387444870153036 507.38974188146835 31.711858867591772 1348.8969976024237 1376.5715911236475 26.354579003910104 26.36532255057268
|
||||
Y 4096 256 64 320 c64_i4096_o256 "completed rc=0 duration_s=650" 320 625.7480431649965 0.5113879356001801 2094.644984218338 130.91531151364612 51983.60868599389 80408.54887854264 286.82753786230273 448.62066245765556
|
||||
Y 4096 512 1 5 c1_i4096_o512 "completed rc=0 duration_s=99" 5 73.99429440300446 0.0675727776086068 276.77809708485347 34.597262135606684 1350.9769304000656 1376.4231815637322 26.310007427002912 26.334555245552153
|
||||
Y 4096 512 64 320 c64_i4096_o512 "completed rc=0 duration_s=880" 320 855.845617013998 0.37389920990243986 1531.4911637603936 191.4363954700492 51983.15175365303 80405.63672760378 233.1860918822221 314.11634894583517
|
||||
Y 4096 1024 1 5 c1_i4096_o1024 "completed rc=0 duration_s=165" 5 141.22379078199447 0.03540479951935607 145.01805883128247 36.25451470782062 1351.8061094000586 1379.9816434006789 26.28578575601136 26.32495529572055
|
||||
Y 4096 1024 64 320 c64_i4096_o1024 "completed rc=0 duration_s=1345" 320 1320.7957280260016 0.24227819125237235 992.3714713697171 248.09286784242929 51987.80489426904 80406.50598565124 207.37264829075608 247.74751702926565
|
||||
Y 4096 2048 1 5 c1_i4096_o2048 "completed rc=0 duration_s=299" 5 276.0557316109989 0.01811228468549133 74.18791807177249 37.09395903588624 1359.5692171991686 1377.9620103552588 26.305917187592286 26.323675634277176
|
||||
Y 4096 2048 64 320 c64_i4096_o2048 "completed rc=0 duration_s=2280" 320 2253.1854364559986 0.14202115583674427 581.7186543073045 290.85932715365226 51999.37922932156 80419.80255557828 194.7249239024334 215.13688206616618
|
||||
Y 4096 4096 1 5 c1_i4096_o4096 "completed rc=0 duration_s=570" 5 546.1440214710019 0.009155094267136421 37.49926611819078 37.49926611819078 1356.901797800674 1381.4528299999074 26.341520792234874 26.363355306765136
|
||||
Y 4096 4096 64 320 c64_i4096_o4096 "completed rc=0 duration_s=4137" 320 4106.892298439998 0.07791779689999466 319.1512961023781 319.1512961023781 52028.30068695314 80428.11315494073 187.86320509006177 197.8918995976436
|
||||
Y 8192 128 1 5 c1_i8192_o128 "completed rc=0 duration_s=58" 5 31.345879535998392 0.1595105983310462 1306.7108215279304 20.417356586373913 2901.3523344008718 2943.8824014391867 26.498316146440686 26.53415173224786
|
||||
Y 8192 128 64 320 c64_i8192_o128 "completed rc=0 duration_s=1027" 320 1001.4245687750008 0.3195447864749735 2617.7108908029827 40.901732668796605 102210.96021643761 179702.23191498037 772.015502173622 1472.9441806528812
|
||||
Y 8192 256 1 5 c1_i8192_o256 "completed rc=0 duration_s=75" 5 48.29129336799815 0.10353833271554946 848.1860216057812 26.505813175180663 2907.958858400525 2941.109013320529 26.463175448623264 26.479798227135436
|
||||
Y 8192 256 64 320 c64_i8192_o256 "completed rc=0 duration_s=1144" 320 1118.6142376200005 0.28606823446199137 2343.4709767126333 73.23346802226979 102227.62581890015 179676.9426185613 476.33626962411694 825.4012724687083
|
||||
Y 8192 512 1 5 c1_i8192_o512 "completed rc=0 duration_s=110" 5 82.16094908700325 0.06085616166270568 498.53367634088494 31.15835477130531 2904.7368975981954 2941.0467594809597 26.467229087280366 26.505994761013312
|
||||
Y 8192 512 64 320 c64_i8192_o512 "completed rc=0 duration_s=1380" 320 1352.1895057329966 0.23665321956964455 1938.6631747145282 121.16644841965801 102229.57628892828 179680.07008601096 329.1102014473765 503.40242396277046
|
||||
Y 8192 1024 1 5 c1_i8192_o1024 "completed rc=0 duration_s=174" 5 150.05084697799612 0.03332203783383651 272.9741339347887 34.12176674184859 2906.111534401134 2942.81814436079 26.49209734721495 26.533382691103906
|
||||
Y 8192 1024 64 320 c64_i8192_o1024 "completed rc=0 duration_s=1845" 320 1819.0112392769952 0.17591974864717758 1441.1345809176787 180.14182261470984 102239.29165678738 179664.01260017834 255.6465966483472 343.00634268297966
|
||||
Y 8192 2048 1 5 c1_i8192_o2048 "completed rc=0 duration_s=310" 5 285.48406155599514 0.017514112601411534 143.47561043076328 35.86890260769082 2907.4056262004888 2944.1087911635987 26.47123923947269 26.497441076600033
|
||||
Y 8192 2048 64 320 c64_i8192_o2048 "failed rc=1 duration_s=553" 0
|
||||
Y 8192 4096 1 5 c1_i8192_o4096 "completed rc=0 duration_s=584" 5 558.9306328860039 0.008945653907324436 73.28279680880178 36.64139840440089 2908.0996894001146 2962.2204470040742 26.58702289313796 26.61703713931074
|
||||
Y 8192 4096 64 320 c64_i8192_o4096 "failed rc=1 duration_s=719 reason=kv_cache_decode_oom" 0
|
||||
Y 16384 128 1 5 c1_i16384_o128 "completed rc=0 duration_s=76" 5 47.921742649996304 0.10433677332058344 1709.4536940844391 13.35510698503468 6168.107185598637 6217.074746359431 26.87305271969036 26.958400565712882
|
||||
Y 16384 128 32 160 c32_i16384_o128 "completed rc=0 duration_s=1104" 160 1080.5961769310015 0.1480664131668646 2425.92011332591 18.95250088535867 107607.05564736869 194812.25933963628 854.2913843793324 1609.9457858553585
|
||||
Y 16384 256 1 5 c1_i16384_o256 "completed rc=0 duration_s=93" 5 65.06508007799857 0.07684613611488852 1259.0470941063336 19.672610845411462 6165.073253607261 6211.129082086845 26.841394899609302 26.881746652878494
|
||||
Y 16384 256 32 160 c32_i16384_o256 "completed rc=0 duration_s=1214" 160 1191.412069234997 0.1342944260273741 2200.279876032497 34.37937306300777 107601.49405923122 194837.96724378655 512.4159163919377 888.8134474421043
|
||||
Y 16384 512 1 5 c1_i16384_o512 "completed rc=0 duration_s=127" 5 99.36944190099894 0.05031727968223334 824.398310313711 25.76244719730347 6175.6319466017885 6251.701926806127 26.801992351080585 26.82276618959694
|
||||
Y 16384 512 32 160 c32_i16384_o512 "failed rc=1 duration_s=560 reason=xpu_kernel_launch_timeout" 0
|
||||
Y 16384 1024 1 5 c1_i16384_o1024 "completed rc=0 duration_s=224" 5 171.96120902399707 0.029076324994332027 476.38650870713593 29.774156794195996 6928.77221259987 9792.07814124471 26.842475365785894 26.88422662580788
|
||||
Y 16384 1024 32 160 c32_i16384_o1024 "completed rc=0 duration_s=1882" 160 1853.834564181001 0.08630759350993428 1414.0636120667632 88.3789757541727 107601.72120737507 194817.03514240915 257.23059973136543 351.1609817709484
|
||||
Y 16384 2048 1 5 c1_i16384_o2048 "completed rc=0 duration_s=334" 5 305.36463945500145 0.016373867023122695 268.26943730684224 33.53367966335528 6173.947739601135 6222.383297723136 26.817756513432563 26.837409417542258
|
||||
Y 16384 2048 32 160 c32_i16384_o2048 "failed rc=137 duration_s=167 reason=out_of_memory" 0
|
||||
P 16384 4096 1 5 c1_i16384_o4096 "completed rc=0 duration_s=608" 5 580.1729553189944 0.008618119741984299 141.19927385267076 35.29981846316769 6178.636959602591 6234.941079312703 26.82611517767935 26.86785587242856
|
||||
P 16384 4096 32 160 c32_i16384_o4096 "failed rc=137 duration_s=903 reason=kv_cache_decode_oom" 0
|
||||
Y 32768 128 1 5 c1_i32768_o128 "failed rc=137 duration_s=29 reason=out_of_memory" 0
|
||||
Y 32768 128 16 80 c16_i32768_o128 "completed rc=0 duration_s=1225" 80 1168.7748753930064 0.06844774103575729 2242.895578259695 8.761310852576933 124317.03163680068 214679.87077928378 861.6436795331623 1645.4117601102328
|
||||
Y 32768 256 1 5 c1_i32768_o256 "completed rc=0 duration_s=134" 5 100.38509806500224 0.0498081896255394 1632.114757649675 12.750896544138087 13168.107665798743 13302.10535181046 27.085759039219543 27.132425062000003
|
||||
Y 32768 256 16 80 c16_i32768_o256 "completed rc=0 duration_s=1286" 80 1263.4621468810074 0.06331808214238047 2074.806915641523 16.2094290284494 122892.92924403744 210721.14786341554 508.9795435045581 891.4333398982063
|
||||
Y 32768 512 1 5 c1_i32768_o512 "completed rc=0 duration_s=170" 5 135.05323958600638 0.03702243659853738 1213.1512024608728 18.955487538451138 13167.90345320187 13303.137884192984 27.085512908023635 27.110341254001163
|
||||
Y 32768 512 16 80 c16_i32768_o512 "completed rc=0 duration_s=1505" 80 1482.1555456469941 0.053975441535104335 1768.6672682222988 27.63542606597342 122898.48796358638 210686.75379800465 339.5751358392154 530.3086735966258
|
||||
Y 32768 1024 1 5 c1_i32768_o1024 "completed rc=0 duration_s=238" 5 204.34472282300703 0.024468456688899888 801.7823887818715 25.055699649433485 13170.243986597052 13308.293756752973 27.073815555036006 27.08208811331908
|
||||
Y 32768 1024 16 80 c16_i32768_o1024 "completed rc=0 duration_s=1944" 80 1921.5543885630032 0.04163296156286601 1364.2288844919933 42.63215264037479 122883.37932371232 210621.58506435342 255.5393236051565 350.83823580940947
|
||||
Y 32768 2048 1 5 c1_i32768_o2048 "completed rc=0 duration_s=377" 5 342.98018497200974 0.014578101648665345 477.695234823466 29.855952176466626 13179.81927599758 13319.727632589638 27.07057905745223 27.088217141381012
|
||||
Y 32768 2048 16 80 c16_i32768_o2048 "completed rc=0 duration_s=2827" 80 2803.750278225998 0.02853321161348862 934.9762781507951 58.436017384424694 122899.37849756316 210673.82780143176 213.8925572432094 261.5382891697284
|
||||
P 32768 4096 1 5 c1_i32768_o4096 "completed rc=0 duration_s=655" 5 620.6336467729998 0.008056282520288137 263.98826562480167 32.99853320310021 13185.259522197885 13320.523270551348 27.091213466128522 27.119225907115602
|
||||
P 32768 4096 16 80 c16_i32768_o4096 "completed rc=0 duration_s=4598" 80 4574.679185062007 0.017487565086799776 573.032532764255 71.62906659553188 122919.97263983685 210664.48323895675 193.40684656185272 217.6030520283007
|
||||
Y 65536 128 1 5 c1_i65536_o128 "failed rc=137 duration_s=33 reason=out_of_memory" 0
|
||||
Y 65536 128 8 40 c8_i65536_o128 "failed rc=137 duration_s=29 reason=out_of_memory" 0
|
||||
Y 65536 256 1 5 c1_i65536_o256 "failed rc=137 duration_s=54 reason=out_of_memory" 0
|
||||
Y 65536 256 8 40 c8_i65536_o256 "skipped reason=strictly_above_failed_boundary boundary=c1_i32768_o128" 0
|
||||
Y 65536 512 1 5 c1_i65536_o512 "failed rc=137 duration_s=33 reason=out_of_memory" 0
|
||||
Y 65536 512 8 40 c8_i65536_o512 "skipped reason=strictly_above_failed_boundary boundary=c1_i32768_o128" 0
|
||||
Y 65536 1024 1 5 c1_i65536_o1024 "failed rc=137 duration_s=34 reason=out_of_memory" 0
|
||||
Y 65536 1024 8 40 c8_i65536_o1024 "skipped reason=strictly_above_failed_boundary boundary=c1_i32768_o128" 0
|
||||
P 65536 2048 1 5 c1_i65536_o2048 "failed rc=137 duration_s=33 reason=out_of_memory" 0
|
||||
P 65536 2048 8 40 c8_i65536_o2048 "skipped reason=strictly_above_failed_boundary boundary=c1_i32768_o128" 0
|
||||
Y 131072 128 1 5 c1_i131072_o128 "failed rc=137 duration_s=34 reason=out_of_memory" 0
|
||||
Y 131072 128 4 20 c4_i131072_o128 "failed rc=137 duration_s=34 reason=out_of_memory" 0
|
||||
Y 131072 256 1 5 c1_i131072_o256 "failed rc=137 duration_s=33 reason=out_of_memory" 0
|
||||
Y 131072 256 4 20 c4_i131072_o256 "skipped reason=strictly_above_failed_boundary boundary=c1_i32768_o128" 0
|
||||
Y 131072 512 1 5 c1_i131072_o512 "failed rc=137 duration_s=33 reason=out_of_memory" 0
|
||||
Y 131072 512 4 20 c4_i131072_o512 "skipped reason=strictly_above_failed_boundary boundary=c1_i32768_o128" 0
|
||||
P 131072 1024 1 5 c1_i131072_o1024 "failed rc=1 duration_s=28 reason=out_of_memory" 0
|
||||
P 131072 1024 4 20 c4_i131072_o1024 "skipped reason=strictly_above_failed_boundary boundary=c1_i32768_o128" 0
|
||||
|
@ -1,43 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
CONTAINER="glm52-w4a8-tp8"
|
||||
PORT="${PORT:-30000}"
|
||||
RUN_DIR="/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke"
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||
RESULT_DIR="${RUN_DIR}/results/${RUN_ID}"
|
||||
PYTHON="/root/miniconda/envs/python310_torch25_cuda/bin/python"
|
||||
|
||||
mkdir -p "${RESULT_DIR}"
|
||||
curl --fail --silent --show-error --max-time 10 "http://127.0.0.1:${PORT}/health" >/dev/null
|
||||
|
||||
docker inspect "${CONTAINER}" >"${RESULT_DIR}/container_inspect.json"
|
||||
docker logs "${CONTAINER}" >"${RESULT_DIR}/server_before_bench.log" 2>&1
|
||||
xpu-smi >"${RESULT_DIR}/xpu_before_bench.txt"
|
||||
|
||||
docker exec "${CONTAINER}" env \
|
||||
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
|
||||
"${PYTHON}" -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port "${PORT}" \
|
||||
--dataset-name sharegpt \
|
||||
--dataset-path /workspace/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||
--model /models \
|
||||
--served-model-name GLM-5.2-w4a8 \
|
||||
--tokenizer /models \
|
||||
--num-prompts 64 \
|
||||
--sharegpt-output-len 128 \
|
||||
--sharegpt-context-len 32768 \
|
||||
--request-rate inf \
|
||||
--max-concurrency 8 \
|
||||
--warmup-requests 2 \
|
||||
--seed 1 \
|
||||
--output-file "/workspace/run/results/${RUN_ID}/bench_sharegpt_c8_o128.jsonl" \
|
||||
--output-details \
|
||||
2>&1 | tee "${RESULT_DIR}/bench_console.log"
|
||||
|
||||
docker logs "${CONTAINER}" >"${RESULT_DIR}/server_after_bench.log" 2>&1
|
||||
xpu-smi >"${RESULT_DIR}/xpu_after_bench.txt"
|
||||
printf '%s\n' "${RESULT_DIR}"
|
||||
|
||||
@ -1,46 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
CONTAINER="glm52-w4a8-tp8"
|
||||
PORT="${PORT:-30000}"
|
||||
RUN_DIR="/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke"
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')-sharegpt-sweep}"
|
||||
RESULT_DIR="${RUN_DIR}/results/${RUN_ID}"
|
||||
PYTHON="/root/miniconda/envs/python310_torch25_cuda/bin/python"
|
||||
CONCURRENCIES="${CONCURRENCIES:-1 2 4 8 16 32}"
|
||||
NUM_PROMPTS="${NUM_PROMPTS:-64}"
|
||||
OUTPUT_LEN="${OUTPUT_LEN:-128}"
|
||||
|
||||
mkdir -p "${RESULT_DIR}"
|
||||
curl --fail --silent --show-error --max-time 10 "http://127.0.0.1:${PORT}/health" >/dev/null
|
||||
docker inspect "${CONTAINER}" >"${RESULT_DIR}/container_inspect.json"
|
||||
xpu-smi >"${RESULT_DIR}/xpu_before_sweep.txt"
|
||||
|
||||
for concurrency in ${CONCURRENCIES}; do
|
||||
echo "===== concurrency=${concurrency} =====" | tee -a "${RESULT_DIR}/sweep.log"
|
||||
docker exec "${CONTAINER}" env \
|
||||
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
|
||||
"${PYTHON}" -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port "${PORT}" \
|
||||
--dataset-name sharegpt \
|
||||
--dataset-path /workspace/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||
--model /models \
|
||||
--served-model-name GLM-5.2-w4a8 \
|
||||
--tokenizer /models \
|
||||
--num-prompts "${NUM_PROMPTS}" \
|
||||
--sharegpt-output-len "${OUTPUT_LEN}" \
|
||||
--sharegpt-context-len 32768 \
|
||||
--request-rate inf \
|
||||
--max-concurrency "${concurrency}" \
|
||||
--warmup-requests 2 \
|
||||
--seed 1 \
|
||||
--output-file "/workspace/run/results/${RUN_ID}/bench_sharegpt_c${concurrency}_o${OUTPUT_LEN}.jsonl" \
|
||||
--output-details \
|
||||
2>&1 | tee "${RESULT_DIR}/bench_c${concurrency}.log" | tee -a "${RESULT_DIR}/sweep.log"
|
||||
done
|
||||
|
||||
docker logs "${CONTAINER}" >"${RESULT_DIR}/server_after_sweep.log" 2>&1
|
||||
xpu-smi >"${RESULT_DIR}/xpu_after_sweep.txt"
|
||||
printf '%s\n' "${RESULT_DIR}"
|
||||
@ -1,234 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
CONTAINER="glm52-w4a8-tp8"
|
||||
PORT="${PORT:-30000}"
|
||||
RUN_DIR="/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke"
|
||||
REFERENCE_DIR="/data1/wxy/sskj/experiments/p800/dsv4_p800_sglang_tp_dp_matrix"
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')-full-matrix}"
|
||||
RESULT_DIR="${RUN_DIR}/results/${RUN_ID}"
|
||||
RAW_DIR="${RESULT_DIR}/raw_outputs"
|
||||
LOG_DIR="${RESULT_DIR}/logs"
|
||||
PYTHON="/root/miniconda/envs/python310_torch25_cuda/bin/python"
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-28800}"
|
||||
MATRIX_MODE="${MATRIX_MODE:-Y+P}"
|
||||
SERVICE_RECOVERY_TIMEOUT_S="${SERVICE_RECOVERY_TIMEOUT_S:-900}"
|
||||
FAILURE_DIR="${RESULT_DIR}/failures"
|
||||
FAILURE_BOUNDARIES_FILE="${RESULT_DIR}/failure_boundaries.tsv"
|
||||
|
||||
mkdir -p "${RAW_DIR}" "${LOG_DIR}" "${FAILURE_DIR}"
|
||||
exec > >(tee -a "${RESULT_DIR}/orchestrator.log") 2>&1
|
||||
|
||||
ensure_service() {
|
||||
if curl --fail --silent --show-error --max-time 10 \
|
||||
"http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ "$(docker inspect -f '{{.State.Running}}' "${CONTAINER}" 2>/dev/null || true)" == "true" ]]; then
|
||||
echo "service_recovery=restart container=${CONTAINER}"
|
||||
if ! docker restart "${CONTAINER}" >/dev/null; then
|
||||
echo "service_recovery=recreate reason=restart_failed"
|
||||
bash "${RUN_DIR}/start_glm52_w4a8.sh" >/dev/null
|
||||
fi
|
||||
else
|
||||
echo "service_recovery=start container=${CONTAINER}"
|
||||
if ! docker start "${CONTAINER}" >/dev/null; then
|
||||
echo "service_recovery=recreate reason=start_failed"
|
||||
bash "${RUN_DIR}/start_glm52_w4a8.sh" >/dev/null
|
||||
fi
|
||||
fi
|
||||
|
||||
local deadline=$(( $(date +%s) + SERVICE_RECOVERY_TIMEOUT_S ))
|
||||
until curl --fail --silent --show-error --max-time 10 \
|
||||
"http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; do
|
||||
if (( $(date +%s) >= deadline )); then
|
||||
echo "service_recovery_failed timeout_s=${SERVICE_RECOVERY_TIMEOUT_S}"
|
||||
return 1
|
||||
fi
|
||||
if [[ "$(docker inspect -f '{{.State.Running}}' "${CONTAINER}" 2>/dev/null || true)" != "true" ]]; then
|
||||
echo "service_recovery_failed container_exited"
|
||||
return 1
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
echo "service_recovery=healthy container=${CONTAINER}"
|
||||
}
|
||||
|
||||
add_failure_boundary() {
|
||||
local failed_isl="$1" failed_osl="$2" failed_concurrency="$3" failed_scenario="$4" reason="$5"
|
||||
touch "${FAILURE_BOUNDARIES_FILE}"
|
||||
if ! awk -F '\t' -v i="${failed_isl}" -v o="${failed_osl}" -v c="${failed_concurrency}" \
|
||||
'$1 == i && $2 == o && $3 == c { found=1 } END { exit !found }' \
|
||||
"${FAILURE_BOUNDARIES_FILE}"; then
|
||||
printf '%s\t%s\t%s\t%s\t%s\n' \
|
||||
"${failed_isl}" "${failed_osl}" "${failed_concurrency}" "${failed_scenario}" "${reason}" \
|
||||
>>"${FAILURE_BOUNDARIES_FILE}"
|
||||
fi
|
||||
}
|
||||
|
||||
strictly_above_failed_boundary() {
|
||||
local candidate_isl="$1" candidate_osl="$2" candidate_concurrency="$3"
|
||||
local failed_isl failed_osl failed_concurrency failed_scenario reason
|
||||
[[ -s "${FAILURE_BOUNDARIES_FILE}" ]] || return 1
|
||||
while IFS=$'\t' read -r failed_isl failed_osl failed_concurrency failed_scenario reason; do
|
||||
if (( candidate_isl > failed_isl && candidate_osl > failed_osl && candidate_concurrency > failed_concurrency )); then
|
||||
printf '%s' "${failed_scenario}"
|
||||
return 0
|
||||
fi
|
||||
done <"${FAILURE_BOUNDARIES_FILE}"
|
||||
return 1
|
||||
}
|
||||
|
||||
failure_reason_from_log() {
|
||||
local failure_log="$1"
|
||||
if grep -q "Decode out of memory" "${failure_log}" 2>/dev/null; then
|
||||
printf '%s' "kv_cache_decode_oom"
|
||||
elif grep -qi "launch timed out and was terminated" "${failure_log}" 2>/dev/null; then
|
||||
printf '%s' "xpu_kernel_launch_timeout"
|
||||
elif grep -qiE "out of memory|OOM" "${failure_log}" 2>/dev/null; then
|
||||
printf '%s' "out_of_memory"
|
||||
elif grep -qiE "Xid|KL_XID|device.*error" "${failure_log}" 2>/dev/null; then
|
||||
printf '%s' "xpu_device_error"
|
||||
elif grep -qiE "watchdog|timeout" "${failure_log}" 2>/dev/null; then
|
||||
printf '%s' "timeout_or_watchdog"
|
||||
else
|
||||
printf '%s' "bench_or_service_failure"
|
||||
fi
|
||||
}
|
||||
|
||||
echo "run_id=${RUN_ID} matrix_mode=${MATRIX_MODE} timeout=${SCENARIO_TIMEOUT_S}"
|
||||
ensure_service
|
||||
docker inspect "${CONTAINER}" >"${RESULT_DIR}/container_inspect.resume.$(date +%Y%m%d-%H%M%S).json"
|
||||
if [[ ! -s "${RESULT_DIR}/scenarios.tsv" ]]; then
|
||||
cp "${REFERENCE_DIR}/matrix.json" "${RESULT_DIR}/matrix.json"
|
||||
python3 "${REFERENCE_DIR}/generate_scenarios.py" \
|
||||
--matrix "${RESULT_DIR}/matrix.json" \
|
||||
--mode "${MATRIX_MODE}" \
|
||||
>"${RESULT_DIR}/scenarios.tsv"
|
||||
fi
|
||||
|
||||
scenario_count="$(tail -n +2 "${RESULT_DIR}/scenarios.tsv" | wc -l)"
|
||||
echo "scenario_count=${scenario_count}"
|
||||
|
||||
(
|
||||
while true; do
|
||||
date -u '+timestamp=%Y-%m-%dT%H:%M:%SZ'
|
||||
xpu-smi
|
||||
sleep 60
|
||||
done
|
||||
) >>"${RESULT_DIR}/xpu_monitor.log" 2>&1 &
|
||||
monitor_pid=$!
|
||||
trap 'kill "${monitor_pid}" >/dev/null 2>&1 || true' EXIT
|
||||
|
||||
index=0
|
||||
tail -n +2 "${RESULT_DIR}/scenarios.tsv" | while IFS=$'\t' read -r mark isl osl concurrency num_prompts; do
|
||||
index=$((index + 1))
|
||||
scenario="c${concurrency}_i${isl}_o${osl}"
|
||||
output_file="${RAW_DIR}/${scenario}.jsonl"
|
||||
status_file="${LOG_DIR}/${scenario}.status"
|
||||
detail_log="${LOG_DIR}/${scenario}.log"
|
||||
|
||||
if [[ -s "${output_file}" ]] && python3 "${RUN_DIR}/summarize_matrix.py" \
|
||||
--check-completed "${output_file}" "${num_prompts}"; then
|
||||
echo "[$index/$scenario_count] skip completed ${scenario}"
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ -s "${status_file}" ]] && grep -qE '^(failed|timeout)' "${status_file}"; then
|
||||
existing_reason="previous_failure"
|
||||
if [[ "${scenario}" == "c64_i8192_o2048" ]]; then
|
||||
existing_reason="kv_cache_decode_oom"
|
||||
fi
|
||||
printf '%s\n' "${existing_reason}" >"${FAILURE_DIR}/${scenario}.reason"
|
||||
add_failure_boundary "${isl}" "${osl}" "${concurrency}" "${scenario}" "${existing_reason}"
|
||||
echo "[$index/$scenario_count] skip recorded failure ${scenario} reason=${existing_reason}"
|
||||
continue
|
||||
fi
|
||||
|
||||
failed_boundary=""
|
||||
if failed_boundary="$(strictly_above_failed_boundary "${isl}" "${osl}" "${concurrency}")"; then
|
||||
printf 'skipped\treason=strictly_above_failed_boundary\tboundary=%s\n' \
|
||||
"${failed_boundary}" >"${status_file}"
|
||||
echo "[$index/$scenario_count] skip strictly-above ${scenario} boundary=${failed_boundary}"
|
||||
continue
|
||||
fi
|
||||
|
||||
if ! ensure_service; then
|
||||
echo "service_unrecoverable_before=${scenario}" | tee "${status_file}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[$index/$scenario_count] start mark=${mark} isl=${isl} osl=${osl} concurrency=${concurrency} prompts=${num_prompts}"
|
||||
started="$(date +%s)"
|
||||
started_iso="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
bench_rc=0
|
||||
timeout "${SCENARIO_TIMEOUT_S}" docker exec "${CONTAINER}" env \
|
||||
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
|
||||
"${PYTHON}" -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port "${PORT}" \
|
||||
--dataset-name random \
|
||||
--dataset-path /workspace/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||
--model /models \
|
||||
--served-model-name GLM-5.2-w4a8 \
|
||||
--tokenizer /models \
|
||||
--random-input-len "${isl}" \
|
||||
--random-output-len "${osl}" \
|
||||
--random-range-ratio 1.0 \
|
||||
--num-prompts "${num_prompts}" \
|
||||
--max-concurrency "${concurrency}" \
|
||||
--request-rate 10000 \
|
||||
--seed 1 \
|
||||
--flush-cache \
|
||||
--output-file "/workspace/run/results/${RUN_ID}/raw_outputs/${scenario}.jsonl" \
|
||||
--output-details \
|
||||
>"${detail_log}" 2>&1 || bench_rc=$?
|
||||
duration=$(( $(date +%s) - started ))
|
||||
|
||||
if [[ "${bench_rc}" -eq 0 ]]; then
|
||||
printf 'completed\trc=0\tduration_s=%s\n' "${duration}" >"${status_file}"
|
||||
echo "[$index/$scenario_count] completed ${scenario} duration_s=${duration}"
|
||||
elif [[ "${bench_rc}" -eq 124 ]]; then
|
||||
failure_log="${FAILURE_DIR}/${scenario}.server.log"
|
||||
docker logs --since "${started_iso}" "${CONTAINER}" >"${failure_log}" 2>&1 || true
|
||||
reason="$(failure_reason_from_log "${failure_log}")"
|
||||
printf 'timeout\trc=124\tduration_s=%s\treason=%s\n' "${duration}" "${reason}" >"${status_file}"
|
||||
printf '%s\n' "${reason}" >"${FAILURE_DIR}/${scenario}.reason"
|
||||
add_failure_boundary "${isl}" "${osl}" "${concurrency}" "${scenario}" "${reason}"
|
||||
echo "[$index/$scenario_count] timeout ${scenario} duration_s=${duration} reason=${reason}"
|
||||
else
|
||||
failure_log="${FAILURE_DIR}/${scenario}.server.log"
|
||||
docker logs --since "${started_iso}" "${CONTAINER}" >"${failure_log}" 2>&1 || true
|
||||
reason="$(failure_reason_from_log "${failure_log}")"
|
||||
printf 'failed\trc=%s\tduration_s=%s\treason=%s\n' \
|
||||
"${bench_rc}" "${duration}" "${reason}" >"${status_file}"
|
||||
printf '%s\n' "${reason}" >"${FAILURE_DIR}/${scenario}.reason"
|
||||
add_failure_boundary "${isl}" "${osl}" "${concurrency}" "${scenario}" "${reason}"
|
||||
echo "[$index/$scenario_count] failed ${scenario} rc=${bench_rc} duration_s=${duration} reason=${reason}"
|
||||
fi
|
||||
|
||||
python3 "${RUN_DIR}/summarize_matrix.py" \
|
||||
--scenarios "${RESULT_DIR}/scenarios.tsv" \
|
||||
--raw-dir "${RAW_DIR}" \
|
||||
--log-dir "${LOG_DIR}" \
|
||||
--output "${RESULT_DIR}/summary.tsv"
|
||||
|
||||
if ! curl --fail --silent --show-error --max-time 10 "http://127.0.0.1:${PORT}/health" >/dev/null; then
|
||||
echo "service_unhealthy_after=${scenario}"
|
||||
docker logs --tail 500 "${CONTAINER}" >"${FAILURE_DIR}/${scenario}.server_tail.log" 2>&1 || true
|
||||
if ! ensure_service; then
|
||||
echo "service_unrecoverable_after=${scenario}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
python3 "${RUN_DIR}/summarize_matrix.py" \
|
||||
--scenarios "${RESULT_DIR}/scenarios.tsv" \
|
||||
--raw-dir "${RAW_DIR}" \
|
||||
--log-dir "${LOG_DIR}" \
|
||||
--output "${RESULT_DIR}/summary.tsv"
|
||||
docker logs "${CONTAINER}" >"${RESULT_DIR}/server_after_matrix.log" 2>&1
|
||||
echo "matrix_complete result_dir=${RESULT_DIR}"
|
||||
@ -1,118 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
CONTAINER="glm52-w4a8-tp8"
|
||||
PORT="${PORT:-30000}"
|
||||
RUN_DIR="/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke"
|
||||
REFERENCE_DIR="/data1/wxy/sskj/experiments/p800/dsv4_p800_sglang_tp_dp_matrix"
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')-full-matrix}"
|
||||
RESULT_DIR="${RUN_DIR}/results/${RUN_ID}"
|
||||
RAW_DIR="${RESULT_DIR}/raw_outputs"
|
||||
LOG_DIR="${RESULT_DIR}/logs"
|
||||
PYTHON="/root/miniconda/envs/python310_torch25_cuda/bin/python"
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-28800}"
|
||||
MATRIX_MODE="${MATRIX_MODE:-Y+P}"
|
||||
|
||||
mkdir -p "${RAW_DIR}" "${LOG_DIR}"
|
||||
exec > >(tee -a "${RESULT_DIR}/orchestrator.log") 2>&1
|
||||
|
||||
echo "run_id=${RUN_ID} matrix_mode=${MATRIX_MODE} timeout=${SCENARIO_TIMEOUT_S}"
|
||||
curl --fail --silent --show-error --max-time 10 "http://127.0.0.1:${PORT}/health" >/dev/null
|
||||
docker inspect "${CONTAINER}" >"${RESULT_DIR}/container_inspect.json"
|
||||
cp "${REFERENCE_DIR}/matrix.json" "${RESULT_DIR}/matrix.json"
|
||||
python3 "${REFERENCE_DIR}/generate_scenarios.py" \
|
||||
--matrix "${RESULT_DIR}/matrix.json" \
|
||||
--mode "${MATRIX_MODE}" \
|
||||
>"${RESULT_DIR}/scenarios.tsv"
|
||||
|
||||
scenario_count="$(tail -n +2 "${RESULT_DIR}/scenarios.tsv" | wc -l)"
|
||||
echo "scenario_count=${scenario_count}"
|
||||
|
||||
(
|
||||
while true; do
|
||||
date -u '+timestamp=%Y-%m-%dT%H:%M:%SZ'
|
||||
xpu-smi
|
||||
sleep 60
|
||||
done
|
||||
) >"${RESULT_DIR}/xpu_monitor.log" 2>&1 &
|
||||
monitor_pid=$!
|
||||
trap 'kill "${monitor_pid}" >/dev/null 2>&1 || true' EXIT
|
||||
|
||||
index=0
|
||||
tail -n +2 "${RESULT_DIR}/scenarios.tsv" | while IFS=$'\t' read -r mark isl osl concurrency num_prompts; do
|
||||
index=$((index + 1))
|
||||
scenario="c${concurrency}_i${isl}_o${osl}"
|
||||
output_file="${RAW_DIR}/${scenario}.jsonl"
|
||||
status_file="${LOG_DIR}/${scenario}.status"
|
||||
detail_log="${LOG_DIR}/${scenario}.log"
|
||||
|
||||
if [[ -s "${output_file}" ]] && python3 "${RUN_DIR}/summarize_matrix.py" \
|
||||
--check-completed "${output_file}" "${num_prompts}"; then
|
||||
echo "[$index/$scenario_count] skip completed ${scenario}"
|
||||
continue
|
||||
fi
|
||||
|
||||
if ! curl --fail --silent --show-error --max-time 10 "http://127.0.0.1:${PORT}/health" >/dev/null; then
|
||||
echo "service_unhealthy_before=${scenario}" | tee "${status_file}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[$index/$scenario_count] start mark=${mark} isl=${isl} osl=${osl} concurrency=${concurrency} prompts=${num_prompts}"
|
||||
started="$(date +%s)"
|
||||
bench_rc=0
|
||||
timeout "${SCENARIO_TIMEOUT_S}" docker exec "${CONTAINER}" env \
|
||||
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
|
||||
"${PYTHON}" -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port "${PORT}" \
|
||||
--dataset-name random \
|
||||
--dataset-path /workspace/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||
--model /models \
|
||||
--served-model-name GLM-5.2-w4a8 \
|
||||
--tokenizer /models \
|
||||
--random-input-len "${isl}" \
|
||||
--random-output-len "${osl}" \
|
||||
--random-range-ratio 1.0 \
|
||||
--num-prompts "${num_prompts}" \
|
||||
--max-concurrency "${concurrency}" \
|
||||
--request-rate 10000 \
|
||||
--seed 1 \
|
||||
--flush-cache \
|
||||
--output-file "/workspace/run/results/${RUN_ID}/raw_outputs/${scenario}.jsonl" \
|
||||
--output-details \
|
||||
>"${detail_log}" 2>&1 || bench_rc=$?
|
||||
duration=$(( $(date +%s) - started ))
|
||||
|
||||
if [[ "${bench_rc}" -eq 0 ]]; then
|
||||
printf 'completed\trc=0\tduration_s=%s\n' "${duration}" >"${status_file}"
|
||||
echo "[$index/$scenario_count] completed ${scenario} duration_s=${duration}"
|
||||
elif [[ "${bench_rc}" -eq 124 ]]; then
|
||||
printf 'timeout\trc=124\tduration_s=%s\n' "${duration}" >"${status_file}"
|
||||
echo "[$index/$scenario_count] timeout ${scenario} duration_s=${duration}"
|
||||
else
|
||||
printf 'failed\trc=%s\tduration_s=%s\n' "${bench_rc}" "${duration}" >"${status_file}"
|
||||
echo "[$index/$scenario_count] failed ${scenario} rc=${bench_rc} duration_s=${duration}"
|
||||
fi
|
||||
|
||||
python3 "${RUN_DIR}/summarize_matrix.py" \
|
||||
--scenarios "${RESULT_DIR}/scenarios.tsv" \
|
||||
--raw-dir "${RAW_DIR}" \
|
||||
--log-dir "${LOG_DIR}" \
|
||||
--output "${RESULT_DIR}/summary.tsv"
|
||||
|
||||
if ! curl --fail --silent --show-error --max-time 10 "http://127.0.0.1:${PORT}/health" >/dev/null; then
|
||||
echo "service_unhealthy_after=${scenario}"
|
||||
docker logs --tail 500 "${CONTAINER}" >"${RESULT_DIR}/server_failure.log" 2>&1 || true
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
python3 "${RUN_DIR}/summarize_matrix.py" \
|
||||
--scenarios "${RESULT_DIR}/scenarios.tsv" \
|
||||
--raw-dir "${RAW_DIR}" \
|
||||
--log-dir "${LOG_DIR}" \
|
||||
--output "${RESULT_DIR}/summary.tsv"
|
||||
docker logs "${CONTAINER}" >"${RESULT_DIR}/server_after_matrix.log" 2>&1
|
||||
echo "matrix_complete result_dir=${RESULT_DIR}"
|
||||
|
||||
@ -1,65 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
IMAGE="iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746"
|
||||
MODEL_DIR="/data2/GLM-5.2-w4a8"
|
||||
DATASET="/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json"
|
||||
RUN_DIR="/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke"
|
||||
CONTAINER="glm52-w4a8-tp8"
|
||||
PORT="${PORT:-30000}"
|
||||
|
||||
test -f "${MODEL_DIR}/config.json"
|
||||
test -f "${DATASET}"
|
||||
mkdir -p "${RUN_DIR}/results"
|
||||
|
||||
docker rm -f "${CONTAINER}" >/dev/null 2>&1 || true
|
||||
|
||||
docker run -d \
|
||||
--init \
|
||||
--name "${CONTAINER}" \
|
||||
--privileged \
|
||||
--ulimit core=-1 \
|
||||
--security-opt seccomp=unconfined \
|
||||
--net=host --uts=host --ipc=host \
|
||||
--shm-size=256g \
|
||||
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||
-e XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||
-e CUDA_DISABLE_PRINTF=1 \
|
||||
-e GLOO_SOCKET_TIMEOUT=60000 \
|
||||
-v "${MODEL_DIR}:/models:ro" \
|
||||
-v "${DATASET}:/workspace/ShareGPT_V3_unfiltered_cleaned_split.json:ro" \
|
||||
-v "${RUN_DIR}:/workspace/run" \
|
||||
-w /workspace/run \
|
||||
"${IMAGE}" \
|
||||
bash -lc "exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server \
|
||||
--host 0.0.0.0 \
|
||||
--port ${PORT} \
|
||||
--model-path /models \
|
||||
--served-model-name GLM-5.2-w4a8 \
|
||||
--attention-backend nsa \
|
||||
--nsa-prefill klxdsa \
|
||||
--nsa-decode klxdsa \
|
||||
--prefill-attention-backend klxdsa \
|
||||
--decode-attention-backend klxdsa \
|
||||
--trust-remote-code \
|
||||
--disable-custom-all-reduce \
|
||||
--disable-shared-experts-fusion \
|
||||
--chunked-prefill-size 32768 \
|
||||
--page-size 64 \
|
||||
--mem-fraction-static 0.82 \
|
||||
--max-prefill-tokens 32768 \
|
||||
--max-running-requests 64 \
|
||||
--tp-size 8 \
|
||||
--ep-size 8 \
|
||||
--quantization w4a8_int4 \
|
||||
--kv-cache-dtype int8 \
|
||||
--cuda-graph-max-bs 8 \
|
||||
--disable-cuda-graph-padding \
|
||||
--watchdog-timeout 3000000 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--json-model-override-args '{\"index_topk_freq\": 4}' \
|
||||
--allow-auto-truncate \
|
||||
--enable-metrics"
|
||||
|
||||
echo "container=${CONTAINER} port=${PORT}"
|
||||
@ -1,64 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
IMAGE="iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260715_5746"
|
||||
MODEL_DIR="/data2/GLM-5.2-w4a8"
|
||||
DATASET="/data1/wxy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json"
|
||||
RUN_DIR="/data1/wxy/sskj/experiments/p800/glm52_w4a8_sglang_tp8_smoke"
|
||||
CONTAINER="glm52-w4a8-tp8"
|
||||
PORT="${PORT:-30000}"
|
||||
|
||||
test -f "${MODEL_DIR}/config.json"
|
||||
test -f "${DATASET}"
|
||||
mkdir -p "${RUN_DIR}/results"
|
||||
|
||||
docker rm -f "${CONTAINER}" >/dev/null 2>&1 || true
|
||||
|
||||
docker run -d \
|
||||
--name "${CONTAINER}" \
|
||||
--privileged \
|
||||
--ulimit core=-1 \
|
||||
--security-opt seccomp=unconfined \
|
||||
--net=host --uts=host --ipc=host \
|
||||
--shm-size=256g \
|
||||
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||
-e XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||
-e CUDA_DISABLE_PRINTF=1 \
|
||||
-e GLOO_SOCKET_TIMEOUT=60000 \
|
||||
-v "${MODEL_DIR}:/models:ro" \
|
||||
-v "${DATASET}:/workspace/ShareGPT_V3_unfiltered_cleaned_split.json:ro" \
|
||||
-v "${RUN_DIR}:/workspace/run" \
|
||||
-w /workspace/run \
|
||||
"${IMAGE}" \
|
||||
bash -lc "exec /root/miniconda/envs/python310_torch25_cuda/bin/python -m sglang.launch_server \
|
||||
--host 0.0.0.0 \
|
||||
--port ${PORT} \
|
||||
--model-path /models \
|
||||
--served-model-name GLM-5.2-w4a8 \
|
||||
--attention-backend nsa \
|
||||
--nsa-prefill klxdsa \
|
||||
--nsa-decode klxdsa \
|
||||
--prefill-attention-backend klxdsa \
|
||||
--decode-attention-backend klxdsa \
|
||||
--trust-remote-code \
|
||||
--disable-custom-all-reduce \
|
||||
--disable-shared-experts-fusion \
|
||||
--chunked-prefill-size 32768 \
|
||||
--page-size 64 \
|
||||
--mem-fraction-static 0.82 \
|
||||
--max-prefill-tokens 32768 \
|
||||
--max-running-requests 64 \
|
||||
--tp-size 8 \
|
||||
--ep-size 8 \
|
||||
--quantization w4a8_int4 \
|
||||
--kv-cache-dtype int8 \
|
||||
--cuda-graph-max-bs 8 \
|
||||
--disable-cuda-graph-padding \
|
||||
--watchdog-timeout 3000000 \
|
||||
--tool-call-parser glm47 \
|
||||
--reasoning-parser glm45 \
|
||||
--json-model-override-args '{\"index_topk_freq\": 4}' \
|
||||
--allow-auto-truncate \
|
||||
--enable-metrics"
|
||||
|
||||
echo "container=${CONTAINER} port=${PORT}"
|
||||
@ -1,71 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def read_first_json(path: Path):
|
||||
try:
|
||||
with path.open("r", encoding="utf-8") as stream:
|
||||
for line in stream:
|
||||
line = line.strip()
|
||||
if line:
|
||||
return json.loads(line)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--check-completed", nargs=2, metavar=("FILE", "EXPECTED"))
|
||||
parser.add_argument("--scenarios")
|
||||
parser.add_argument("--raw-dir")
|
||||
parser.add_argument("--log-dir")
|
||||
parser.add_argument("--output")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.check_completed:
|
||||
path, expected = args.check_completed
|
||||
data = read_first_json(Path(path)) or {}
|
||||
raise SystemExit(0 if int(data.get("completed", 0)) >= int(expected) else 1)
|
||||
|
||||
required = (args.scenarios, args.raw_dir, args.log_dir, args.output)
|
||||
if not all(required):
|
||||
parser.error("summary mode requires --scenarios, --raw-dir, --log-dir, and --output")
|
||||
|
||||
raw_dir = Path(args.raw_dir)
|
||||
log_dir = Path(args.log_dir)
|
||||
rows = []
|
||||
with Path(args.scenarios).open("r", encoding="utf-8", newline="") as stream:
|
||||
for scenario in csv.DictReader(stream, delimiter="\t"):
|
||||
name = f"c{scenario['concurrency']}_i{scenario['input_len']}_o{scenario['output_len']}"
|
||||
data = read_first_json(raw_dir / f"{name}.jsonl") or {}
|
||||
status_path = log_dir / f"{name}.status"
|
||||
status = status_path.read_text(encoding="utf-8").strip() if status_path.exists() else "pending"
|
||||
rows.append({
|
||||
**scenario,
|
||||
"scenario": name,
|
||||
"status": status,
|
||||
"completed": data.get("completed", 0),
|
||||
"duration_s": data.get("duration", ""),
|
||||
"request_throughput": data.get("request_throughput", ""),
|
||||
"input_throughput": data.get("input_throughput", ""),
|
||||
"output_throughput": data.get("output_throughput", ""),
|
||||
"total_token_throughput": data.get("total_token_throughput", ""),
|
||||
"mean_ttft_ms": data.get("mean_ttft_ms", ""),
|
||||
"p99_ttft_ms": data.get("p99_ttft_ms", ""),
|
||||
"mean_tpot_ms": data.get("mean_tpot_ms", ""),
|
||||
"p99_tpot_ms": data.get("p99_tpot_ms", ""),
|
||||
})
|
||||
|
||||
fields = list(rows[0]) if rows else []
|
||||
with Path(args.output).open("w", encoding="utf-8", newline="") as stream:
|
||||
writer = csv.DictWriter(stream, fieldnames=fields, delimiter="\t")
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -1,135 +0,0 @@
|
||||
# GLM-5.3-NVFP4 128k 低并发容量扩容与拓扑选型实验报告 — 2026-09-09 @ 174.1.60.8
|
||||
|
||||
## 一、结论速览
|
||||
|
||||
**优胜配置 = TP2PP4 nomtp**（radix/hicache 保持开启、`--context-length 1048576`、cu13 栈 9 挂载、memfrac 0.85、chunk 8192）。
|
||||
|
||||
| 维度 | 60.5 现役（TP8+EAGLE） | 优胜配置（TP2PP4 nomtp） | 提升 |
|
||||
|---|---|---|---|
|
||||
| KV 池（token） | 276,864 | **909,632** | 3.29× |
|
||||
| i128k 并发上限 | c2（c3 起排队） | **c6**（c7 起排队，实测） | 3× |
|
||||
| 单条请求上限 | 264k（ctx 270,336 参数所限） | **~909k**（900k 实跑通过） | 3.4× |
|
||||
| c4 输入吞吐 | 3,228 tok/s | **5,907 tok/s** | 1.83× |
|
||||
| c4 输出吞吐 | 12.6 tok/s | **23.1 tok/s** | 1.83× |
|
||||
| c4 TTFT p50 | 119.6s | **43.4s** | −64% |
|
||||
| c4 e2e p50 | 161.3s | **88.8s** | −45% |
|
||||
| 90% 命中真实流量形态（c4） | — | 输入 **17,625 tok/s**、TTFT 7.3s | — |
|
||||
|
||||
- 质量门 7/7 通过（GSM8K×5 + 中文推理 + tool call，PP4+radix+hicache+parser 新组合无 correctness 问题）。
|
||||
- **60.8 已按用户决定部署优胜配置留役**（restart=unless-stopped，替代 r37）。
|
||||
- **60.5 交付 `deploy_glm53_605_v2.sh`（只交未执行）**，原 `deploy_glm53_605.sh` 原样留盘即回滚路径。
|
||||
|
||||
## 二、问题诊断（60.5 生产日志实证）
|
||||
|
||||
现役容器 `glm53-nvfp4`（TP8 + EAGLE 4/1/5，memfrac 0.90，fp8 KV，hicache-ratio 3，`--context-length 270336`）：
|
||||
|
||||
1. **"单条最高 256k" 不是模型限制**：模型原生 `max_position_embeddings = 1,048,576`（1M）。上限来自部署参数 `--context-length 270336`，且 KV 池 276,864 也刚好只装得下一条 26.4 万请求。
|
||||
2. **排队是纯容量算术**：单条 16~22 万 token 请求占池 0.58~0.90；近期一万行日志 ~13% 的 decode 步有 `#queue-req>0`，09-08 有一次池满 retract。日志实例：一条 173k（95% 命中）请求在 160k 请求 decode 期间排队无法准入。
|
||||
3. **命中不省并发容量**：radix/hicache 只加速重复前缀的 prefill；不同文档的并发请求各自全额驻留 KV，照样排队。60.5 真实流量 12万~22万 token、命中率 90%+，仍然撞容量墙。
|
||||
4. **TP8 调参无解（结构性）**：每 token KV 57.3KB（MLA 44.9 + DSA indexer ~10），fp8_e4m3 已是断言封死的硬顶（FP4 启动即 AssertionError）；TP8 全系调参上限 314,944（60.6 实测 0.89+hicache4）< c3×128k 需求 394,752。
|
||||
|
||||
## 三、机理：为什么只有 PP 拓扑能解
|
||||
|
||||
GLM-5.3 是 MLA/DSA 家族，**KV 在 TP 组内每卡全量复制、只按 PP 分层切**：TP8 每卡存全部 78 层 KV（15.85GB→276,864 token）；PP2 每卡存半数层（池×2）；PP4 每卡存 1/4 层（池×4，实际因 hicache/图开销打折）。**KV 池容量 ≈ PP 度数倍增，与 TP 度数无关**。这就是 TP4PP2≈2.37×、TP2PP4≈3.29×（实测）的来源。另：MTP/EAGLE 的 draft KV + verify 图额外吃池（r37 同 memfrac 下 589,696→384,960，−35%）。
|
||||
|
||||
## 四、实验方法
|
||||
|
||||
- **平台**：60.8（8×RTX 6000D 85.6GB，与 60.5 同型）；四配置臂各部署后跑同窗口。
|
||||
- **场景**：i131072 / o512 / cc∈{1,2,3,4}，nreq=8/点，PG19 真实语料（bench_corpus.py，input_ids 直发 /generate，temp 0，ignore_eos，stream）。
|
||||
- **公平性协议**：每点独立语料窗口（cc1→base 10M，cc2→11,048,576，cc3→12,097,152，cc4→13,145,728），**所有臂用同一映射=同文本**；每点前 `POST /flush_cache`（实测连 host 层一起清，复跑命中率 0.0）；每臂先做一次不计分的 131k 形状预热（吸收内核首触成本，实测不做会污染首点：16k 首跑 1653 vs 复跑 4526 tok/s）。
|
||||
- **指标口径**（沿用 rev20 双场景协议）：输入吞吐 = 总输入 token/墙钟；输出吞吐 = 服务端 completion_tokens 总和/墙钟；TTFT = 发出→首个流式响应（含排队）；命中率从 TP0 Prefill 日志核算。queue-req 日志用于排队取证。
|
||||
- **配置臂**：
|
||||
- **arm0 r37**：60.8 在役原样（TP4PP2+EAGLE 3/1/4+verify 图，memfrac 0.88，池 384,960）
|
||||
- **arm1 A-mirror**：60.5 现役逐参数复刻（旧镜像 20260828，池 276,864，ctx 270,336）
|
||||
- **arm2 B'**：TP4PP2 nomtp，memfrac 0.88，ctx 524,288（池 589,696）
|
||||
- **arm3 TP2PP4**：TP2PP4 nomtp，memfrac 0.85，ctx 1048576，radix/hicache 开，含 glm45/glm47 parser（池 909,632）
|
||||
|
||||
## 五、主扫数据（i128k/o512，冷缓存）
|
||||
|
||||
**输入吞吐（tok/s）**
|
||||
|
||||
| cc | A-mirror（池276,864） | r37（池384,960） | B'（池589,696） | TP2PP4（池909,632） |
|
||||
|---|---|---|---|---|
|
||||
| 1 | 3,026 | **4,110** | 3,737 | 3,262 |
|
||||
| 2 | 3,142 | **4,754** | 4,619 | 4,635 |
|
||||
| 3 | 3,214 | 5,016 | 4,843 | **5,017** |
|
||||
| 4 | 3,228 | 5,061 | 5,256 | **5,907** |
|
||||
|
||||
**输出吞吐（tok/s）**
|
||||
|
||||
| cc | A-mirror | r37 | B' | TP2PP4 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | 11.8 | **16.1** | 14.6 | 12.7 |
|
||||
| 2 | 12.3 | **18.6** | 18.0 | 18.1 |
|
||||
| 3 | 12.6 | 19.6 | 18.9 | **19.6** |
|
||||
| 4 | 12.6 | 19.8 | 20.5 | **23.1** |
|
||||
|
||||
**TTFT p50 / max（秒）**（排队签名 = p50 跳升整请求时长倍数）
|
||||
|
||||
| cc | A-mirror | r37 | B' | TP2PP4 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | 37.8 | 21.1 | 21.0 | **15.5** |
|
||||
| 2 | 37.8 / 76.9 | 21.1 / 41.7 | 41.3 | 29.0 |
|
||||
| 3 | **77.2** / 120.3 | 43.6 / 73.8 | 41.3 / 61.6 | **29.6** / 43.4 |
|
||||
| 4 | **119.6** / 156.4 | 72.2 / 93.9 | 61.6 / 81.9 | **43.4** / 57.3 |
|
||||
|
||||
**e2e p50 / max（秒）**
|
||||
|
||||
| cc | A-mirror | r37 | B' | TP2PP4 |
|
||||
|---|---|---|---|---|
|
||||
| 1 | 43.8 | **31.7** | 35.0 | 40.2 |
|
||||
| 2 | 83.7 / 121.6 | **55.3** / 76.6 | 56.8 | 56.5 |
|
||||
| 3 | 121.7 / 163.3 | **75.2** / 105.1 | 80.0 | 75.8 |
|
||||
| 4 | 161.3 / 201.8 | 103.3 / 125.2 | 99.7 | **88.8** |
|
||||
|
||||
**排队证据与机理注解**：
|
||||
- A-mirror：池只容 2 条并发，c3/c4 排队（TTFT p50 77→120s）；c4 输入吞吐被排队锁死在 3,228（prefill 带宽根本没用满）。
|
||||
- r37：池 384,960 同样只容 2 条 + 1 条错峰补位，c3/c4 排队（TTFT p50 43.6→72.2s；日志 518 行 queue-req>0）。**60.8 原在役配置在本场景也不合格。**
|
||||
- B'：c3/c4 全并发准入、零容量排队（稳态 usage 精确停在 0.22/0.45/0.67/0.89；e2e p50≈max 波内同步）。日志中的 queue-req 行是 chunk 调度/波间 radix 逐出瞬态（秒级），非容量排队——判据：TTFT 缩放=纯 prefill 带宽分摊（cc2 p50=max=41.3≈2×21.0）。
|
||||
- TP2PP4：同上零容量排队，且 TTFT 全场最优（4 级流水 prefill 重叠最深：cc1 TTFT 15.5s、串行 prefill 等效 ~8.5k tok/s）；代价是 TPOT 最慢（cc1 48.4ms vs B' 27.3 vs r37 20.7 vs A 12.0），c1 短输出场景 e2e 吃亏（40.2s），c2 起被并发摊平、c3/c4 反超。
|
||||
- MTP/EAGLE 观察：accept 随并发上升（r37 2.91→3.64，A-mirror 3.31→4.26，verify batch 越大接受越高）；但 MTP 的 35% 池代价在本容量场景不划算——r37 全程吞吐被 B'/TP2PP4 压制或打平。
|
||||
|
||||
## 六、优胜者（TP2PP4）附加验证
|
||||
|
||||
| 验证项 | 结果 |
|
||||
|---|---|
|
||||
| 质量门 | **7/7**（GSM8K 72/3/60/63/10 + 鸡兔同笼 23 + tool call get_weather北京） |
|
||||
| 512k 单条（523,776+512 真实语料） | ✅ TTFT 88.4s，e2e 114.4s，TPOT 50.9ms（旧配置直接拒绝） |
|
||||
| 900k 单条（900,000+512） | ✅ TTFT 210.6s，e2e 237.8s —— **单条上限 ~909k 实证**（池减 512 后的余量） |
|
||||
| cc5 | ✅ in 5,747 / out 22.5，e2e p50≈max 107s，零容量排队 |
|
||||
| cc6 | ✅ in 5,849 / out 22.9，e2e p50≈max 122.7s，零容量排队 —— **并发上限 = c6** |
|
||||
| cc7 | ⚠️ 第 7 条排队（TTFT max 138.7s、e2e max 179.7s）—— 边界与池算术吻合（7×131,584=921,088 > 909,632） |
|
||||
| 90% 命中 c4（60.5 真实流量形态，实测命中 0.8999） | in **17,625** / out 68.9 tok/s，TTFT p50 **7.3s**，e2e 29.9s —— radix 去重+hicache 对重复查询流量再放大 ~3× |
|
||||
|
||||
## 七、方案对比与推荐
|
||||
|
||||
| 方案 | 池 | c4 in/out | 容量定位 | 判定 |
|
||||
|---|---|---|---|---|
|
||||
| A（TP8+EAGLE，60.5 现役） | 276,864 | 3,228/12.6 | c2、单条 264k | 本场景被全面支配，淘汰 |
|
||||
| r37（TP4PP2+MTP，60.8 昨日在役） | 384,960 | 5,061/19.8 | c2、单条 ~380k | c3 起排队，容量不合格；仅 c1 最优 |
|
||||
| B'（TP4PP2 nomtp） | 589,696 | 5,256/20.5 | c4、单条 ~588k（512k 单条会占 89% 池、堵死并发） | 强力备选：16k 短文本场景历史成绩好 |
|
||||
| **TP2PP4 nomtp（优胜）** | **909,632** | **5,907/23.1** | **c6、单条 ~909k、512k 单条+2 并发共存** | 用户需求（512k 单条+长上下文为主）下的正解 |
|
||||
|
||||
**推荐**：60.5 采用 TP2PP4 nomtp（`deploy_glm53_605_v2.sh`，参数与 60.8 现役完全一致）。决策依据：c3/c4/c5/c6 吞吐全场第一 + 唯一满足"512k 单条与并发共存" + 质量门 7/7 + TTFT 全档最优。若 60.5 未来 16k 短文本流量占比显著上升，再评估切 B'（其 s1 短文本历史成绩比 TP2PP4 好 27~48%）。
|
||||
|
||||
**留观旋钮**（未验证，报告只记录不推荐）：memfrac 0.85→0.88 或可再抬池（PP0 空闲 22GB 最大的 stage 不均匀提示有空间）；TP2PP4+MTP 可修 c1 短输出短板（池约降至 ~59 万，恰为 B' 水平）；hicache-ratio 3→4 扩 host 前缀池（60.6 在 TP8 验证过）。
|
||||
|
||||
**r37 角色变化说明**：r37 的 MTP 优势区间是 i16k/cc≤16（09-09 判决 +6%@cc16），本次让位给容量优先的 TP2PP4 是按用户明确选择执行；若 60.8 未来主要服务短文本低并发，可用 deploy_ppmtp_r37.sh mtp 模式一键切回。
|
||||
|
||||
## 八、已证伪 / 排除项（本轮+引用前判）
|
||||
|
||||
- TP8 任何调参：fp8 KV 断言封顶 + 上限 314,944 < c3 需求 394,752（60.6 数据）。
|
||||
- r37 现役直接顶上：池 384,960，c3 差 1 万 token 仍排队（本轮实测）。
|
||||
- MTP/EAGLE 换 KV 池：draft+verify 图 −35% 池，本场景不划算。
|
||||
- HiSparse 超池驻留：本 nightly 自旋不可用（60.6 前判）。
|
||||
- TP1PP8：池 2M 但吞吐 −12~33%（60.2 前判），仅当需要 >c6 且能接受慢时再议。
|
||||
- PD 分离/DP/EP：前判劣化或不可用，与本问题正交。
|
||||
|
||||
## 九、资产与复现
|
||||
|
||||
- **60.8**：容器 glm53-nvfp4:30000 = 优胜配置，restart=unless-stopped。启动命令 = `bash /root/deploy_ppmtp_r37.sh '--tp 2 --pp-size 4 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length 1048576' nomtp 8192 0.85`。实验日志全量：`/root/bench_logs/128kexp/`（26 点 SUMMARY 已提取为 all_summaries.json）。
|
||||
- **60.5**：交付 `deploy_glm53_605_v2.sh`（内嵌完整 docker run+前置检查+显存归零等待+健康门+回滚提示）；回滚 = `bash /root/deploy_glm53_605.sh`。执行前置检查会列出缺失的补丁/镜像清单（60.8 /root 均有）。
|
||||
- **压测复现**：`bash /root/arm_runner.sh <arm>`（c1-4 四点+形状预热）、`bash /root/val_runner.sh <arm>`（512k/900k/cc5-7/hit90）；语料窗口映射与 flush 协议见第四节；质量门 `bash /root/quality_gate_605.sh`。
|
||||
- **仓库归档**：`experiments/pro6000/glm53_nvfp4_128k_capacity_topology/`（README=本报告、scripts、results/20260909）；profile `deploy/profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp2pp4_hicache.env`；CURRENT.md 已更新 60.8 行。
|
||||
|
||||
*报告与数据：ZCode 实验 2026-09-09；压测窗口 run-id 9501-9507/9601-9604/9990-9991；全部冷缓存口径。*
|
||||
File diff suppressed because it is too large
Load Diff
@ -1,3 +0,0 @@
|
||||
e782a8767d4478711757669f83adb64b *scripts/arm_runner.sh
|
||||
f6f5a49937ac6e8a0157ba1eb11b8ee7 *scripts/deploy_glm53_605_v2.sh
|
||||
8eb75693f01f6055b15ecc3d62feff08 *scripts/val_runner.sh
|
||||
@ -1,36 +0,0 @@
|
||||
#!/bin/bash
|
||||
# 128k capacity experiment - per-arm bench runner (i131072/o512, cc 1-4, nreq 8)
|
||||
# usage: nohup bash arm_runner.sh <arm-name> >/dev/null 2>&1 &
|
||||
# per-cc-point corpus windows are distinct but the mapping is identical for every arm:
|
||||
# cc1 base 10000000, cc2 11048576, cc3 12097152, cc4 13145728 (all in consumed space, flushed)
|
||||
ARM=${1:?arm name required}
|
||||
URL=http://127.0.0.1:30000
|
||||
LOGD=/root/bench_logs/128kexp
|
||||
mkdir -p $LOGD
|
||||
echo "$(date +%T) [$ARM] start" >> $LOGD/${ARM}_progress.log
|
||||
|
||||
# wait until server idle (no running/queued requests in recent logs)
|
||||
for i in $(seq 1 60); do
|
||||
R=$(docker logs --since 20s glm53-nvfp4 2>&1 | grep -cE 'running-req: [1-9]|queue-req: [1-9]')
|
||||
if [ "$R" -eq 0 ]; then break; fi
|
||||
sleep 10
|
||||
done
|
||||
|
||||
flush() { curl -s -X POST $URL/flush_cache >/dev/null; sleep 2; }
|
||||
|
||||
# shape warmup (not scored): one 128k prefill to absorb kernel first-touch cost
|
||||
flush
|
||||
python3 /root/bench_corpus.py --input-len 131072 --output-len 64 --shared-frac 0 \
|
||||
--concurrency 1 --num-requests 1 --run-id 9991 --pool-override 17000000 \
|
||||
> $LOGD/${ARM}_warmup.log 2>&1
|
||||
echo "$(date +%T) [$ARM] warmup done" >> $LOGD/${ARM}_progress.log
|
||||
|
||||
declare -A BASE=( [1]=10000000 [2]=11048576 [3]=12097152 [4]=13145728 )
|
||||
for CC in 1 2 3 4; do
|
||||
flush
|
||||
python3 /root/bench_corpus.py --input-len 131072 --output-len 512 --shared-frac 0 \
|
||||
--concurrency $CC --num-requests 8 --run-id 950${CC} --pool-override ${BASE[$CC]} \
|
||||
> $LOGD/${ARM}_cc${CC}.log 2>&1
|
||||
echo "$(date +%T) [$ARM] cc$CC done rc=$?" >> $LOGD/${ARM}_progress.log
|
||||
done
|
||||
echo "ALL-DONE" >> $LOGD/${ARM}_progress.log
|
||||
@ -1,129 +0,0 @@
|
||||
#!/bin/bash
|
||||
# deploy_glm53_605_v2.sh — GLM-5.3-NVFP4 60.5 生产容量扩容配置（2026-09-09 交付）
|
||||
#
|
||||
# 目的：解决 60.5 现役 TP8+EAGLE 配置的 KV 池容量问题（池 276,864 token）：
|
||||
# - 单条请求上限 264k（--context-length 270336 所致，模型原生支持 1M）
|
||||
# - 并发>1 且总 token>~26 万时排队（i128k 下 c3 起必然排队）
|
||||
# 本配置 = 2026-09-09 在 60.8 同型机（8×RTX 6000D）四臂对拍优胜者：
|
||||
# TP2 PP4 nomtp + radix/hicache 开 + ctx 1M + cu13 栈 9 挂载
|
||||
# 实测池 909,632 token（3.29×）：i128k 并发上限 c2→c6，单条上限 264k→909k，
|
||||
# c4 输入/输出吞吐 5907/23.1 tok/s（现役口径 3228/12.6 的 1.83×/1.83×），
|
||||
# 质量门 7/7。完整数据：experiments/pro6000/glm53_nvfp4_128k_capacity_topology/
|
||||
#
|
||||
# ⚠️ 运维提示：
|
||||
# 1) 在维护窗口执行：会移除现役 glm53-nvfp4 容器，停机约 10-15 分钟（权重加载+健康）。
|
||||
# 2) 回滚 = 原脚本原样留盘：bash /root/deploy_glm53_605.sh（TP8+EAGLE 原配置）。
|
||||
# 3) 与 60.8 现役（本配置已留役）一致，脚本可互相对拍 md5。
|
||||
#
|
||||
# 用法: bash deploy_glm53_605_v2.sh # 部署到 30000 端口
|
||||
# ROLLBACK_ONLY=1 bash deploy_glm53_605_v2.sh # 仅回滚到原配置
|
||||
|
||||
set -u
|
||||
IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729
|
||||
PORT=30000
|
||||
|
||||
# ---- 前置检查（缺什么给什么清单，不猜） ----
|
||||
MISSING=0
|
||||
if [ -z "$(docker images -q $IMAGE 2>/dev/null)" ]; then
|
||||
echo "[MISS] 镜像不存在: $IMAGE"
|
||||
echo " 拉取: docker pull $IMAGE"
|
||||
echo " 或从 60.8 导: ssh 60.8 'docker save $IMAGE | gzip' | gunzip | docker load"
|
||||
MISSING=1
|
||||
fi
|
||||
MOUNTS=(
|
||||
"/root/sglang_patch2/layer_setup.py"
|
||||
"/root/sglang_patch2/validation_hook.py"
|
||||
"/root/eagle_worker_v2_mask.py"
|
||||
"/root/sglang_patch2/eagle_worker_common.py"
|
||||
"/root/sglang_patch2/deepseek_nextn.py"
|
||||
"/root/scheduler_pp_mixin_r35.py"
|
||||
"/root/request_receiver_degloo.py"
|
||||
"/root/sglang_patch2/deepseek_v2.py"
|
||||
"/root/decode_cuda_graph_runner_fix.py"
|
||||
)
|
||||
for f in "${MOUNTS[@]}"; do
|
||||
if [ ! -f "$f" ]; then
|
||||
echo "[MISS] 补丁文件缺失: $f"
|
||||
MISSING=1
|
||||
fi
|
||||
done
|
||||
if [ "$MISSING" = "1" ]; then
|
||||
echo "[STOP] 上述文件在 60.8 /root 均有（含 sglang_patch2/ 子目录），scp 全部补齐后重跑本脚本。"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "${ROLLBACK_ONLY:-0}" = "1" ]; then
|
||||
echo "[rollback] 恢复 60.5 原生产配置（TP8+EAGLE）..."
|
||||
exec bash /root/deploy_glm53_605.sh
|
||||
fi
|
||||
|
||||
# ---- 移除旧容器（等待显存归零，纪律：重部署前必等显存<1500MiB） ----
|
||||
docker update --restart=no glm53-nvfp4 >/dev/null 2>&1
|
||||
for i in 1 2 3 4 5 6 7 8; do
|
||||
docker rm -f glm53-nvfp4 >/dev/null 2>&1
|
||||
docker ps -a --format '{{.Names}}' | grep -q '^glm53-nvfp4$' || break
|
||||
sleep 5
|
||||
done
|
||||
if docker ps -a --format '{{.Names}}' | grep -q '^glm53-nvfp4$'; then
|
||||
echo "[ERROR] 旧容器删不掉（zombie 时重试即可）"; exit 1
|
||||
fi
|
||||
echo "[wait] 等待显存释放 (<1500MiB/卡)..."
|
||||
for i in $(seq 1 60); do
|
||||
MAXMI=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | sort -rn | head -1)
|
||||
[ "$MAXMI" -lt 1500 ] && break
|
||||
sleep 5
|
||||
done
|
||||
echo "[wait] 显存就绪 (${MAXMI}MiB max)"
|
||||
|
||||
# ---- 部署优胜配置（与 60.8 实验臂 arm3_TP2PP4 完全一致） ----
|
||||
docker run -d --name glm53-nvfp4 --gpus all --shm-size 64g --ipc=host --cap-add SYS_PTRACE \
|
||||
-v /root/sglang_patch2/layer_setup.py:/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner_components/layer_setup.py:ro \
|
||||
-v /root/sglang_patch2/validation_hook.py:/sgl-workspace/sglang/python/sglang/srt/arg_groups/validation_hook.py:ro \
|
||||
-v /root/eagle_worker_v2_mask.py:/sgl-workspace/sglang/python/sglang/srt/speculative/eagle_worker_v2.py:ro \
|
||||
-v /root/sglang_patch2/eagle_worker_common.py:/sgl-workspace/sglang/python/sglang/srt/speculative/eagle_worker_common.py:ro \
|
||||
-v /root/sglang_patch2/deepseek_nextn.py:/sgl-workspace/sglang/python/sglang/srt/models/deepseek_nextn.py:ro \
|
||||
-v /root/scheduler_pp_mixin_r35.py:/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_pp_mixin.py:ro \
|
||||
-v /root/request_receiver_degloo.py:/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_components/request_receiver.py:ro \
|
||||
-v /root/sglang_patch2/deepseek_v2.py:/sgl-workspace/sglang/python/sglang/srt/models/deepseek_v2.py:ro \
|
||||
-v /root/decode_cuda_graph_runner_fix.py:/sgl-workspace/sglang/python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py:ro \
|
||||
-e SGLANG_PP_DEGLOO=1 -e SGLANG_PP_SPEC_FORCE_EAGER_DRAFT=1 -e SGLANG_PP_FORCE_EAGER_VERIFY=0 -e SGLANG_PP_SPEC_DEBUG=0 \
|
||||
--restart no -p ${PORT}:${PORT} \
|
||||
-v /data/hf_models:/data/hf_models \
|
||||
$IMAGE \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /data/hf_models/GLM-5.3-NVFP4 \
|
||||
--tp 8 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--max-running-requests 16 \
|
||||
--chunked-prefill-size 8192 \
|
||||
--disable-shared-experts-fusion \
|
||||
--moe-runner-backend flashinfer_cutlass \
|
||||
--disable-flashinfer-autotune \
|
||||
--reasoning-parser glm45 --tool-call-parser glm47 \
|
||||
--enable-hierarchical-cache --hicache-ratio 3 \
|
||||
--tp 2 --pp-size 4 --disable-overlap-schedule --max-prefill-tokens 16384 \
|
||||
--disable-custom-all-reduce --context-length 1048576 \
|
||||
--host 0.0.0.0 --port ${PORT}
|
||||
|
||||
echo "[deploy] TP2PP4-nomtp 容量扩容配置启动，等待健康（最长 1800s）..."
|
||||
for i in $(seq 10 10 1800); do
|
||||
code=$(curl -s -o /dev/null -m3 -w '%{http_code}' http://127.0.0.1:${PORT}/health 2>/dev/null)
|
||||
if [ "$code" = "200" ]; then
|
||||
echo "[OK] healthy after ${i}s"
|
||||
docker update --restart=unless-stopped glm53-nvfp4 >/dev/null && echo "[OK] restart=unless-stopped 已设置"
|
||||
POOL=$(docker logs glm53-nvfp4 2>&1 | grep -m1 -oE 'max_total_num_tokens=[0-9]+')
|
||||
echo "[INFO] ${POOL} (60.8 同型机实测 909632；若显著低于此值请停下核查)"
|
||||
echo "[NEXT] 质量门: bash /root/quality_gate_605.sh 期望 PASS=7 FAIL=0"
|
||||
echo "[NEXT] 容量冒烟: 300k 单条请求应能正常完成（旧配置会直接拒绝）"
|
||||
exit 0
|
||||
fi
|
||||
if ! docker ps --format '{{.Names}}' | grep -q '^glm53-nvfp4$'; then
|
||||
echo "[DIED] 容器启动失败，最近错误："
|
||||
docker logs --tail 40 glm53-nvfp4 2>&1 | grep -iE 'error|assert|not support' | tail -8
|
||||
echo "[回滚] bash /root/deploy_glm53_605.sh"
|
||||
exit 1
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
echo "[TIMEOUT] 健康等待超时；回滚: bash /root/deploy_glm53_605.sh"
|
||||
exit 1
|
||||
@ -1,48 +0,0 @@
|
||||
#!/bin/bash
|
||||
# winner validation runner (TP2PP4): big singles, cc boundary, 90%-hit point
|
||||
# usage: nohup bash val_runner.sh <arm-name> >/dev/null 2>&1 &
|
||||
ARM=${1:?arm name required}
|
||||
URL=http://127.0.0.1:30000
|
||||
LOGD=/root/bench_logs/128kexp
|
||||
echo "$(date +%T) [$ARM] val start" >> $LOGD/${ARM}_progress.log
|
||||
|
||||
for i in $(seq 1 60); do
|
||||
R=$(docker logs --since 20s glm53-nvfp4 2>&1 | grep -cE 'running-req: [1-9]|queue-req: [1-9]')
|
||||
if [ "$R" -eq 0 ]; then break; fi
|
||||
sleep 10
|
||||
done
|
||||
|
||||
flush() { curl -s -X POST $URL/flush_cache >/dev/null; sleep 2; }
|
||||
|
||||
# 1) 512k single (real corpus): 523776+512 = 524288
|
||||
flush
|
||||
python3 /root/bench_corpus.py --input-len 523776 --output-len 512 --shared-frac 0 \
|
||||
--concurrency 1 --num-requests 1 --run-id 9601 --pool-override 10000000 \
|
||||
> $LOGD/${ARM}_single512k.log 2>&1
|
||||
echo "$(date +%T) [$ARM] single512k rc=$?" >> $LOGD/${ARM}_progress.log
|
||||
|
||||
# 2) 900k single (pool ceiling probe): 900000+512 = 900512 vs pool 909632
|
||||
flush
|
||||
python3 /root/bench_corpus.py --input-len 900000 --output-len 512 --shared-frac 0 \
|
||||
--concurrency 1 --num-requests 1 --run-id 9602 --pool-override 10000000 \
|
||||
> $LOGD/${ARM}_single900k.log 2>&1
|
||||
echo "$(date +%T) [$ARM] single900k rc=$?" >> $LOGD/${ARM}_progress.log
|
||||
|
||||
# 3-5) cc boundary: 5 / 6 fit, 7 exceeds pool (921088 > 909632) -> expect queuing
|
||||
declare -A BASEB=( [5]=10000000 [6]=11048576 [7]=12097152 )
|
||||
for CC in 5 6 7; do
|
||||
flush
|
||||
python3 /root/bench_corpus.py --input-len 131072 --output-len 512 --shared-frac 0 \
|
||||
--concurrency $CC --num-requests 8 --run-id 950${CC} --pool-override ${BASEB[$CC]} \
|
||||
> $LOGD/${ARM}_cc${CC}.log 2>&1
|
||||
echo "$(date +%T) [$ARM] cc$CC rc=$?" >> $LOGD/${ARM}_progress.log
|
||||
done
|
||||
|
||||
# 6) 90%-hit cc4 point (mirrors 60.5 real traffic: shared 117968 + unique 13104x8)
|
||||
flush
|
||||
python3 /root/bench_corpus.py --input-len 131072 --output-len 512 --shared-frac 0.9 \
|
||||
--concurrency 4 --num-requests 8 --run-id 9604 --pool-override 15000000 \
|
||||
> $LOGD/${ARM}_hit90_cc4.log 2>&1
|
||||
echo "$(date +%T) [$ARM] hit90_cc4 rc=$?" >> $LOGD/${ARM}_progress.log
|
||||
|
||||
echo "VAL-ALL-DONE" >> $LOGD/${ARM}_progress.log
|
||||
@ -1,82 +0,0 @@
|
||||
# GLM-5.3-NVFP4 双方案 B300 对标场景矩阵压测 — 60.8（6000D）
|
||||
|
||||
日期：2026-09-10 ｜ 机器：174.1.60.8（8×RTX 6000D，96GB GDDR7，无 NVLink）
|
||||
模型：GLM-5.3-NVFP4（modelopt）｜ 镜像：`nightly-dev-20260828-daf63171`（两臂同）
|
||||
对标基线：飞书《GLM 5.3 ｜ SGLang ｜ Low-latency & High-Throughput 测试结果》（B300 报告，wiki UPB2w4Y5yi65qwkMxJJcZko5nUc）
|
||||
完整报告：本目录 `REPORT.md`（= 飞书发布版 A7V3wZTQeifCB4krdi6cA834nW9；两臂高并发复测后整文更新，正文一律采用复测值）
|
||||
|
||||
## 目标
|
||||
|
||||
在 6000D 上复刻 B300 报告的全部场景（主场景 16K→512、4.1 短输入、4.2 长输出、5.1 长上下文、5.2 边界），对两套在役部署方案各跑一遍完整矩阵，产出对齐 B300 8 章结构的对标报告。两臂各完成一轮当日调参复测：E7b 初测暴露 MRR16 + decode 图 bs1-8 的高并发掉图断崖，按用户决策调参为 MRR64 + 图桶 1-64（`deploy_glm53_e7b_hicc.sh`，其余配方逐字不变）复测三场景 C=8/16/32/64 共 10 点；TP2PP4 初测的 C=64 点受 MRR48 封顶（48 活跃+16 排队；decode 图栈默认覆盖 bs≤256、本臂从未掉图），按用户决策调参为 MRR64（`deploy_glm53_pp4_mrr64.sh`，逐 token diff 仅 MRR 一处）复测三场景 C=32/C=64 共 6 点两轮（pass-1 + v2 有序版，全部点位对齐 ≤1.8%），另以全新实例跑原版 MRR48 的 4.1 C=32 单点做归因对照。**报告正文一律采用复测值**（C=1 与 5.1/5.2 点受池上限约束、与调参无关，沿用初测值）。测后 60.8 在役服务（TP4PP2@0.90）原容器恢复，当日三轮拆台均按 rename→start 流程保全恢复并逐轮验证。
|
||||
|
||||
## 实验臂
|
||||
|
||||
| 臂 | 方案 | 关键配置 | 质量门 |
|
||||
|---|---|---|---|
|
||||
| tp2pp4 | **D 生产口径**（deploy_glm53_pp4.sh） | TP2PP4、mem0.85、MRR48、cps16384、radix 关、KV fp8_e4m3 池 1,040,384、无投机、index_topk_freq=4（=原生默认，恒等）、ctx 1,048,576 | 6/7（仅 tool-call：无 parser，历史已知） |
|
||||
| e7b | **TP8+EAGLE3+AR 初测**（deploy_glm53_607_exp.sh + CAR 补丁注入） | TP8、EAGLE 4/1/5、mem0.90、MRR16、cps8192、radix 开+hicache×3、KV fp8_e4m3 GPU 池 276,480、decode 图 bs1-8、ctx 270,336、custom-AR 1stage（`SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage`，8 rank `SSKJ_CAR_PATCH_ACTIVE` 验证） | 7/7 |
|
||||
| e7b64 | **TP8+EAGLE3+AR 高并发复测**（deploy_glm53_e7b_hicc.sh，= e7b 仅改 MRR 16→64 + decode 图桶 1-64） | 其余配方与 e7b 逐字一致；图捕获后 avail 6.11 GB/卡，KV 池 276,480 不变 | 7/7 |
|
||||
| pp4mrr64 | **D 生产口径高并发复测**（deploy_glm53_pp4_mrr64.sh，= tp2pp4 仅改 MRR 48→64） | 其余配方与 tp2pp4 逐字一致；decode 图栈默认覆盖 bs≤256（无需扩图）；KV 池 1,040,384 不变；16K 场景活跃上限转为池 ~59 条 | 6/7（v2 实例复验，仅 tool-call 已知项） |
|
||||
| mrr48ctl | **归因对照**（原版 deploy_glm53_pp4.sh 全新实例，4.1 C=32 单点） | = tp2pp4 逐字一致（MRR48）；用于分解 4.1 C=32 锚点 −17%（−3% 实例新鲜度 + −14% MRR64 配置代价） | 未跑（配置同 tp2pp4 已验） |
|
||||
|
||||
## 场景矩阵与并发档位（用户裁决收敛：16K 封 64、64K/128K 封 8/4）
|
||||
|
||||
| B300 章节 | 场景 | 并发档位 | TP2PP4 活跃上限 | E7b 活跃上限 |
|
||||
|---|---|---|---|---|
|
||||
| §3 主场景 | 16K→512 | 1/8/16/32/64 | ~59（池 1,040,384÷17.4K；复测臂 MRR64 超配，c64 3 次回退） | 16（池 276,480÷17.4K；复测臂 MRR64 不再是约束） |
|
||||
| §4.1 | 1K→128 | 1/8/32/64 | 64（复测臂 MRR；初测 48） | 64（复测臂 MRR；初测 16） |
|
||||
| §4.2 | 1K→4K | 1/8/32/64 | 64（复测臂 MRR；初测 48） | ~52（池；复测臂 c64 有 12 条排队；初测 c64 中止 rc=143，复测已补齐） |
|
||||
| §5.1 | 64K→512 | 1/4/8 | 15（池） | 4（池） |
|
||||
| §5.1 | 128K→512 | 1/2/4 | 7（池） | 2（池） |
|
||||
| §5.2 | 256K→1 | 1/2/3 | 3（池） | 1（池贴边） |
|
||||
| §5.2 | 512K→1 | 1 | 1 | 结构性不可（ctx） |
|
||||
| §5.2 | 896K→1（代 B300"约1M"） | 1 | 1 | 结构性不可（ctx） |
|
||||
|
||||
测量协议：冷缓存（shared-frac 0）+ 每点 flush + 服务端命中核验 ≤0.01（超限重试一次）；nreq=max(8, 2×cc)；P95 nearest-rank 对齐 B300；语料耗尽（21.23M/21.30M）下用回收窗口（`--pool-override` 基址映射见 REPORT 附录 A；复测臂为全新容器实例，同基址文本对 hicache 宿主层重新成为处女文本，命中核验全 0.0）。有效测量点 57 个（tp2pp4 24 + pp4mrr64 复测 12 + mrr48ctl 对照 1 + e7b/e7b64 合计 20），全点 0 OOM；唯一 retraction 点 = TP2PP4 主场景 C=64（MRR64 对 16K 池超配，两轮各 3 次，其余全点 0）。
|
||||
|
||||
## 判决速览（详见 REPORT.md，两臂数值均为复测值）
|
||||
|
||||
- **掉图断崖已修复（E7b 复测核心判决）**：初测 MRR16 + 图 bs1-8 把 E7b 窗口封死 C≤8（16K c16 TPOT 298ms 断崖）；调参 MRR64 + 图桶 1-64 后 C=16 TPOT 降至 228.8ms，16K c16/32/64 输出 92.1/97.6/99.6 tok/s（较初测 +18~33%），窗口扩到 C=64。
|
||||
- **TP2PP4 C=64 档全面提升（pp4mrr64 复测核心判决）**：初测 C=64 损失全部来自 MRR48（48 活跃+16 排队；decode 图默认覆盖 bs≤256 从未掉图）。MRR64 后：4.1 c64 输出 341→433（+27%）、4.2 c64 482→572（+19%）、16K c64 TTFT 135→89s（−34%）；4.1/4.2 c64 TTFT 从 20.0/337.9s 塌缩到 7.2/7.3s。16K c64 到达池顶（活跃 ~59、吞吐 −5%、3 次回退为确定性行为，两轮一致）。
|
||||
- **MRR64 的 −14% 代价（1K 短输入 c32）与归因对照**：4.1 c32 锚点 276→228（−17%），全新实例 MRR48 对照 266.42 分解 = −3% 实例新鲜度 + −14% MRR64 配置本身（逐请求 TPOT p50 87→102ms 均匀抬高，三实例可复现、机制未定位）；4.2 c32 −6%、16K c32 持平。MRR 按负载形态选：1K 短输入为主且并发 ≤48 可保持 MRR48。
|
||||
- **分界负载形态化**：prefill 密集（16K 主场景）仍 TP2PP4 占优——c64 out 201 vs 99.6 = 2.0×，E7b 输出被 prefill 墙（chunk8192+TP8 无 PP 流水）封在 ~100 tok/s 平台；短输入 c32 两方案打平（228 vs 229）、c64 TP2PP4 1.6×（433 vs 272）且 TTFT/TPOT 双优。
|
||||
- **decode 密集（1K→4K）E7b 全档反超**：c8/32/64 = 402/676/**826.5 tok/s**，c64 为全场最高输出吞吐（超 TP2PP4 同点 572 达 45%），TPOT 85.1ms 仍优、TTFT 被 TP2PP4 MRR64 反超（7.30 vs 12.18s）；初测排队断崖（c32 TTFT 305s）消除为 6.19s。
|
||||
- **E7b C=1 优势不变**：16K out 49.6 vs 16.7=3.0×、TPOT 13.7 vs 50.3ms=3.7×；1K→4K out 135 vs 20.3=6.7×、TPOT 8.7ms。
|
||||
- **TP2PP4 甜点 c16+**：16K 爬至 c64 池顶（活跃 ~59 封顶）；边界 256K/512K/896K 只有 TP2PP4 可达（input 7,050/5,662/4,153 tok/s）；E7b ctx 270,336 结构性封顶。
|
||||
- **DSA 复现**：C=1 TPOT 对上下文不敏感（50.3/50.0/49.7ms @16/64/128K），并发才是驱动（c8: 70.9→161.8ms）。
|
||||
- **vs B300**：定性结构复现（LL/HT 分野一致），绝对差 4-5×，边界 prefill 差距收窄至 ~2×；主场景分界本机更靠前（C8 vs C64-128，容量上限而非算力），decode 密集场景调参后 E7b 全档无交叉（B300 报告未呈现该形态）。
|
||||
|
||||
## 关键坑位（复测必读）
|
||||
|
||||
1. **hicache 宿主层陷阱**：256K prewarm KV 占池 94.8% 触发宿主层下放，`flush_cache` 清不掉宿主层 → 同文本测量命中 0.9998。冷缓存复测**必须换该实例从未发过的文本**（本战役 E7b 256K C=1 用窗口 9,900,000 重测达标；e7b64 复测臂为全新容器实例，沿用同基址窗口即满足该条件，10 点命中全 0.0）。
|
||||
2. 语料已耗尽：回收窗口复用仅在 flush+命中核验协议下有效；TP2PP4 臂 radix 本来就关，零污染。
|
||||
3. 在役保全流程：`docker stop` → `docker rename glm53-nvfp4 glm53-nvfp4-insvc`（必须先改名，E7b 部署脚本会 rm -f 同名容器）→ 测毕 `rename` 回 + `start`。docker stop/rm 偶发 "zombie PID" 报错是收尾边界现象，容器终态 exited(137)、显存归零，稍等重试即可。
|
||||
|
||||
## 资产与 md5 台账（60.8 执行件 = 本目录 = 60.7 原件 三方一致）
|
||||
|
||||
```
|
||||
1e34dd8d813ccb7ac8243e8d1573a49e bench_corpus_v2.py (scripts/)
|
||||
4c126d067d33b5ea27c268f37561634c extract_summary.py (scripts/)
|
||||
5892b44610b2ce61f533f1721105625e run_b300_matrix.sh (scripts/)
|
||||
165db732fa3237a80a4d53a963a128e7 deploy_glm53_e7b_hicc.sh (scripts/，E7b 高并发版：MRR64+图桶1-64，其余与 607_exp 逐字一致)
|
||||
fe46da2eb22f5ae23eb963bb9d467922 run_retest_e7b64.sh (scripts/，10 点复测驱动，run-id 96xx)
|
||||
376bedfd42fa29ba9de8bfdf0c67c3b2 gen_retest_compare.py (scripts/，复测前后对比表生成器)
|
||||
533440efc04e89bf5bbec3e0efbb35ed deploy_glm53_pp4_mrr64.sh (scripts/，TP2PP4 高并发版：仅 MRR 48→64，其余与 deploy_glm53_pp4.sh 逐 token 一致)
|
||||
c9b6361d8679c41e59aced298e0f50bf run_retest_pp4mrr64.sh (scripts/，pass-1 6 点复测驱动，run-id 966x)
|
||||
7d8578169b2e7d3d1c9d461f53c44e87 run_retest_pp4mrr64_v2.sh (scripts/，v2 有序版：回退点 16K c64 排末位，run-id 967x，正文采用值)
|
||||
889c242859e46a0d41240a818f754bd1 launch_v2_after_pass1.sh (scripts/，pass-1→拆台→重部署→QG→v2 全自动衔接 wrapper)
|
||||
1ff8a21f12a79a3b2aaa8c2dafb4af32 control_mrr48_41c32.sh (scripts/，MRR48 归因对照 + 自动链接在役恢复)
|
||||
20cea216bce9aeda4ec4ecea4569a3de gen_retest_compare_pp4.py (scripts/，TP2PP4 复测前后对比表生成器)
|
||||
def3c64c5dc19e1d507080e3366c3762 deploy_glm53_pp4.sh (见 dual_scenario_bench/scripts/，md5 对照一致)
|
||||
21db641f1d997e26fcf1ddc97163b87e deploy_glm53_607_exp.sh (见 dual_scenario_bench/scripts/，md5 对照一致)
|
||||
a8fc9a504c041acc40831a848b4985d8 patches/custom_all_reduce.py
|
||||
65a4d22bd87ae343e7d68c4879c14f98 patches/custom_all_reduce_utils.py
|
||||
```
|
||||
|
||||
gen_report_tables.py 为本地表格生成器（md5 未入台账，60.8 侧执行件同源）。
|
||||
|
||||
## 原始数据
|
||||
|
||||
- 本目录 `results/{tp2pp4,e7b,e7b64,pp4mrr64}/`：all_results*.jsonl（逐点 SUMMARY + 命中核验）、status*.txt、server_facts*.txt（启动参数+池分配日志摘录）、gpu_inventory_idle/final.csv、vram_timeline.csv（30s 采样全矩阵；csv/log 属 gitignore 中间件，关键事实折叠于 provenance.md）；pp4mrr64/ 含 pass-1、v2（正文采用值）、mrr48ctl（归因对照）三套 jsonl + retest_compare_pp4_v2.md 前后对比表
|
||||
- 60.8 侧：`/root/bench_logs/b300eq_tp2pp4_20260910_1138/`、`/root/bench_logs/b300eq_e7b_20260910_1428/`（初测留档基线）、`/root/bench_logs/b300eq_e7b64_20260910_1732/`（E7b 复测，正文采用值）+ `/root/bench_logs/retest_compare.md`（前后对比）、`/root/bench_logs/b300eq_pp4mrr64_20260910_1927/`（TP2PP4 复测 pass-1）、`/root/bench_logs/b300eq_pp4mrr64v2_20260910_2021/`（TP2PP4 复测 v2，正文采用值）+ `/root/bench_logs/retest_compare_pp4_v2.md`、`/root/bench_logs/b300eq_mrr48ctl_20260910_2112/`（MRR48 对照）
|
||||
- E7b 256K C=1：all_results.jsonl 同 tag 共 4 条，最后一条为干净重测值（生成器 dict 载入后写覆盖，天然生效）
|
||||
@ -1,295 +0,0 @@
|
||||
# GLM-5.3-NVFP4 ｜ RTX 6000D ｜ SGLang 双方案 B300 对标场景压测报告
|
||||
|
||||
- 测试日期：2026-09-10（单日单机完成两臂；两臂均完成当日调参复测——E7b：MRR 64 + decode 图桶 ≤64，TP2PP4：MRR 48→64——正文一律采用复测值）
|
||||
- 测试机：174.1.60.8（6000D，8 卡）
|
||||
- 对标基线：飞书《GLM 5.3 ｜ SGLang ｜ Low-latency & High-Throughput 测试结果》（B300 报告，wiki UPB2w4Y5yi65qwkMxJJcZko5nUc）
|
||||
- 测后状态：60.8 在役服务（TP4PP2@0.90 口径，KV 池 647,040）已原容器恢复并终验（当日三轮拆台均按 rename→start 流程保全恢复；末轮恢复后 fired up + health 200 + 16K 单发抽测 ok，池 647,040 与显存水位 77.2/82.3 GiB 逐字一致）
|
||||
|
||||
## 1. 结论摘要
|
||||
|
||||
本轮在单台 8 卡 RTX 6000D 上，用 GLM-5.3-**NVFP4** 完整复刻 B300 报告的场景矩阵，测试了两套在役部署方案。两套方案代表完整部署形态，不是单参数 A/B：**TP2PP4** 为 D 生产口径（吞吐/长上下文形态），**TP8+EAGLE3+AR** 为 E7b 配方（低延迟形态，含 custom allreduce 1stage 补丁）。
|
||||
|
||||
- **低并发优先 E7b（TP8+EAGLE3+AR）**：主场景 `16K→512, C=1` 输出 49.6 tok/s、TPOT 13.7 ms，对 TP2PP4（16.7 tok/s、50.3 ms）分别是 **3.0×** 与 **3.7×**；长输出 `1K→4K, C=1` 输出 135 tok/s、TPOT 8.7 ms，对 TP2PP4（20.3、49.8 ms）是 **6.7×** 与 **5.7×**。
|
||||
- **prefill 密集场景（16K 输入）高并发优先 TP2PP4**：主场景 C=64 达到 6,419 input tok/s / 201 output tok/s，对 E7b（3,188 / 99.6）为 **2.0×**；短输入 C=32 两方案打平（228 vs 229 tok/s，TP2PP4 的 MRR64 配置代价见下条）、C=64 TP2PP4 拉开（433 vs 272，**+59%**）。
|
||||
- **E7b 初测的掉图断崖是配置产物，调参后已消除**：初测 MRR=16 + decode 图仅覆盖 bs 1–8，并发 >8 即掉图（主场景 C=16 TPOT P95 298 ms）。按决策调参为 **MRR=64 + decode 图桶 1–64**（其余配方逐字不变）后复测：C=16 TPOT P95 降至 228.8 ms，16K 场景 C=16/32/64 输出 92.1/97.6/99.6 tok/s（较初测 +18~33%），E7b 可用并发窗口从 C≤8 扩到 **C=64**。
|
||||
- **TP2PP4 初测的 C=64 损失全部来自 MRR=48（48 活跃+16 排队），调至 64 后 C=64 档全面提升**：decode 图为栈默认覆盖 bs≤256、本臂从未掉图，MRR 才是唯一瓶颈。MRR64 复测（其余配方逐字不变）：4.1 短输入 C=64 输出 341→433（+27%）、4.2 长输出 C=64 输出 482→572（+19%）、主场景 C=64 TTFT P95 135→89 s（−34%）；4.1/4.2 的 C=64 TTFT P95 分别从 20.0/337.9 s 塌缩到 7.2/7.3 s。
|
||||
- **MRR64 的代价与边界**：1K 短输入 C=32 锚点 276→228（−17%；全新实例 MRR48 对照实验定案：−3% 为实例新鲜度、−14% 为 MRR64 配置本身——逐请求 TPOT p50 87→102 ms 均匀抬高，三实例可复现、机制未定位）、1K→4K C=32 −6%、16K C=32 持平（−0.5%）；16K C=64 吞吐 −5%（201 vs 211，池超配 3 次回退）。MRR 按负载形态选，见建议 3。
|
||||
- **TP2PP4 甜点在 C=16 之后，MRR64 后主场景 C=64 到达 16K 池顶**：输出吞吐从 C=8 的 101 爬到 C=64 的 201 tok/s（活跃 ~59 条为池所封顶）；其 4.2 长输出 C=64 的 572 tok/s 已被 E7b 反超（826.5），但 16K 主场景仍是本机 prefill 吞吐之王。
|
||||
- **decode 密集负载（1K 进、长出）E7b 全并发档最优**：`1K→4K` C=8/32/64 输出 402/676/**826.5 tok/s**——C=64 为全场最高输出吞吐（超 TP2PP4 同点 572 达 45%），TTFT P95 12.2 s、TPOT P95 85.1 ms，EAGLE accept 3.8~4.0。
|
||||
- **长上下文与容量边界只有 TP2PP4 可达**：128K C=1 两方案输出打平（11.8 vs 11.9 tok/s）但 TP2PP4 TTFT 减半（18.2 s vs 37.6 s）；256K/512K/896K 边界 E7b 结构性不可测（ctx 270,336 封顶 + KV 池 276,480 贴边），TP2PP4 全部完成（256K C=1/2/3、512K/896K C=1）。
|
||||
- **DSA 特性在 6000D 复现**：TP2PP4 C=1 的 TPOT 对上下文长度不敏感（16K/64K/128K = 50.3/50.0/49.7 ms 恒定），并发才是 TPOT 驱动因子（16K 行 C=8→C=64：70.9→261.5 ms，末档含池超配回退的批扰动）。
|
||||
- **与 B300 的绝对差距约 4~5×**，边界 prefill 差距收窄到约 2×（256K C=1 input 7,050 vs 17,457；896K 4,153 vs 约1M 行 7,897）。硬件与量化口径不同（B300 报告未写明量化方式），绝对值仅量级可比，两份报告的结构性结论一致（见第 9 章）。
|
||||
- 全部有效测量点 **0 OOM**、冷缓存命中核验全部 ≤0.01（E7b 256K C=1 首测触 hicache 宿主层陷阱，用全新文本重测达标，见 5.2 注记；E7b64 复测 10 点、TP2PP4 MRR64 复测两轮 12 点与 MRR48 对照 1 点命中核验全部 0.0）；回退（retraction）仅出现在 TP2PP4 主场景 C=64 一点（MRR64 对 16K 池超配，两轮各 3 次，其余全部点 0 回退）。
|
||||
|
||||
## 2. 测试环境与配置
|
||||
|
||||
| 项目 | TP2PP4（D 生产口径） | TP8+EAGLE3+AR（E7b 配方） |
|
||||
|-|-|-|
|
||||
| 硬件 | 单机 8 × NVIDIA RTX 6000D（96 GB GDDR7，nvidia-smi 可见 85,651 MiB/卡） | 同左 |
|
||||
| 模型 | GLM-5.3-NVFP4（modelopt 量化，/data/hf_models/GLM-5.3-NVFP4） | 同左 |
|
||||
| 镜像 | `nightly-dev-20260828-daf63171` | 同左 |
|
||||
| 并行 | TP2 × PP4 | TP8 |
|
||||
| 投机解码 | 无 | EAGLE3，num_steps=4，topk=1，draft_tokens=5 |
|
||||
| `mem-fraction-static` | 0.85 | 0.90 |
|
||||
| 最大活跃请求（MRR） | 64（复测口径；初测 48，见演进注记） | 64 |
|
||||
| Chunk Prefill | 16,384 | 8,192 |
|
||||
| KV dtype | fp8_e4m3 | fp8_e4m3 |
|
||||
| KV 池（服务端实测） | **1,040,384 tokens**（12.6~13.4 GB/rank，无宿主层） | **276,480 tokens GPU**（15.8 GB/rank）+ 分层缓存 hicache×3 宿主层（write_through） |
|
||||
| radix cache | 关（`disable_radix_cache=True`） | 开（分层缓存） |
|
||||
| 上下文上限 | 1,048,576（config 原生） | 270,336（显存约束下的部署值） |
|
||||
| CUDA graph | 常规捕获（decode 图栈默认覆盖 bs≤256，MRR64 全程在图内） | decode/verify 图桶 bs 1–64（1,2,3,4,6,8,12,16,24,32,48,64） |
|
||||
| custom allreduce | — | 1stage 补丁注入（`SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage` 环境强制；8 rank `SSKJ_CAR_PATCH_ACTIVE` 日志验证全出现） |
|
||||
| `index_topk_freq` | 4（override，等于原生默认，恒等） | 原生默认 4 |
|
||||
| 质量门 | 6/7（仅 tool-call 失败：D 口径未配 parser，历史已知；其余全过） | **7/7** |
|
||||
|
||||
**E7b 臂配置演进注记**：E7b 初测为 MRR=16 + decode 图桶 bs 1–8，高并发点（C>8）出现 decode 掉图 + MRR 排队双击。按决策将 MRR 调至 64、decode 图桶扩至 1–64（部署脚本 `deploy_glm53_e7b_hicc.sh`，其余配方与在役 E7b 逐字一致），三场景 C=8/16/32/64 共 10 点全部重测，正文一律采用复测值；C=1 各点与 5.1/5.2 长上下文点受 KV 池上限约束（活跃 1~4 条），行为与该调参无关，沿用初测值。C=8 锚点前后偏差 ≤3%（16K 81.4→83.9、1K 152→151、1K→4K 412→402 tok/s），证明两轮环境无漂移。初测原始数据留档于服务器 `b300eq_e7b_20260910_1428/` 与库内 `results/e7b/`。
|
||||
|
||||
**TP2PP4 臂配置演进注记**：TP2PP4 初测为 MRR=48，主场景/4.1/4.2 的 C=64 点实际运行为 48 活跃+16 排队（decode 图为栈默认覆盖 bs≤256，本臂从未掉图，MRR 才是 C=64 的活跃上限）。按决策将 MRR 调至 64（部署脚本 `deploy_glm53_pp4_mrr64.sh`，与初测脚本逐 token diff 仅 MRR 一处），三场景 C=32/C=64 共 6 点重测两轮——pass-1 常规顺序，v2 全新实例并把易触发回退的主场景 C=64 排末位以排除状态污染；两轮全部点位对齐 ≤1.8%，证明回退后遗污染假设不成立、结果为 MRR64 的可复现行为。另以全新实例跑原版 MRR48 的 4.1 C=32 单点做归因对照（276 初测 → 266 对照 → 228 复测：−3% 实例新鲜度 + −14% MRR64 配置代价）。正文一律采用 v2 复测值；C≤16 各点与 5.x 长上下文行沿用初测值（MRR 在这些档位不构成约束）。初测原始数据留档于服务器 `b300eq_tp2pp4_20260910_1138/` 与库内 `results/tp2pp4/`，复测两轮与对照留档 `b300eq_pp4mrr64_20260910_1927/`、`b300eq_pp4mrr64v2_20260910_2021/`、`b300eq_mrr48ctl_20260910_2112/`。
|
||||
|
||||
因此，下文比较回答的是"两种部署形态谁更适合该负载"，不能把差异单独归因于 EAGLE、PP 流水、chunk、radix 或图覆盖中的某一项（与 B300 报告同款声明）。
|
||||
|
||||
**测量协议**（对齐 B300 口径）：
|
||||
|
||||
- 冷缓存：`--shared-frac 0`，每点前 `POST /flush_cache`，服务端 Prefill 日志核算命中率，>0.01 重测一次，仍超停点排查；全矩阵命中核验最终全部达标。
|
||||
- 指标：Input TPS / Output TPS / TTFT P95 / TPOT P95（P95 为 nearest-rank；Input TPS = 输入 token / 全程墙钟，与 B300 口径一致）。
|
||||
- 负载：PG19 真实语料 token 切片（`corpus_ids.json`），input_ids 直打 `/generate`，temperature=0、ignore_eos、流式；nreq = max(8, 2×并发)，边界行 nreq=并发。
|
||||
- 并发档位按决策收敛：16K/1K 类封顶 64；64K 封 8、128K 封 4（128K C=2 补一档）；超出活跃上限的档位是**排队观察点**（与 B300 C=256 同性质，保留为有效观察）。
|
||||
- 语料已耗尽（21.23M/21.30M），冷缓存口径下用**回收窗口**复用（窗口基址见附录 A，逐记录 `corpus_window` 字段留档）。
|
||||
|
||||
**两臂活跃上限**（MRR 与 KV 池决定，解释各行哪些并发是排队观察点）：
|
||||
|
||||
| 场景 | TP2PP4 活跃上限 | E7b 活跃上限 |
|
||||
|-|-|-|
|
||||
| 16K | ~59（池 1,040,384 ÷ 17.4K；MRR64 超配，C=64 有 3 次回退） | 16（池 276,480 ÷ 约17.4K；MRR64 不再是约束） |
|
||||
| 1K→128 | 64（MRR） | 64（MRR） |
|
||||
| 1K→4K | 64（MRR） | ~52（池；C=64 有 12 条排队） |
|
||||
| 64K | 15（池） | 4（池） |
|
||||
| 128K | 7（池） | 2（池） |
|
||||
| 256K | 3（池） | 1（池 262K KV / 276K 贴边） |
|
||||
| 512K / 896K | 1（池） | 结构性不可（ctx 270,336） |
|
||||
|
||||
## 3. 主场景：16K 输入、512 输出
|
||||
|
||||
B300 跑了 C=1/8/32/64/128/256；本机按 MRR 上限收敛为 C=1/8/16/32/64（C=16 为本机甜点档，B300 无此档；C=128/256 超出两臂 MRR）。
|
||||
|
||||
| 并发 | 方案 | Input TPS | Output TPS | TTFT P95 | TPOT P95 |
|
||||
|-|-|-|-|-|-|
|
||||
| 1 | TP2PP4 | 534 | 16.7 | 5.36 s | 50.3 ms |
|
||||
| 1 | TP8+EAGLE3+AR | 1,587 | 49.6 | **3.98 s** | **13.7 ms** |
|
||||
| 8 | TP2PP4 | 3,236 | **101** | **14.79 s** | **70.9 ms** |
|
||||
| 8 | TP8+EAGLE3+AR | 2,684 | 83.9 | 32.47 s | 136.1 ms |
|
||||
| 16 | TP2PP4 | 4,674 | **146** | **25.78 s** | **98.5 ms** |
|
||||
| 16 | TP8+EAGLE3+AR | 2,947 | 92.1 | 59.91 s | 228.8 ms |
|
||||
| 32 | TP2PP4 | **6,156** | **192** | **46.31 s** | **153.2 ms** |
|
||||
| 32 | TP8+EAGLE3+AR | 3,123 | 97.6 | 140.33 s | 213.6 ms |
|
||||
| 64 | TP2PP4 | **6,419** | **201** | **89.22 s** | 261.5 ms |
|
||||
| 64 | TP8+EAGLE3+AR | 3,188 | 99.6 | 293.21 s | **214.6 ms** |
|
||||
|
||||
趋势：
|
||||
|
||||
- **分界仍在 C=8，但差距显著收窄**：C=1 E7b 全指标占优；C=8 起 TP2PP4 输出/输入/TTFT 反超。输出吞吐差距 2.0×（201 vs 99.6）；TPOT P95 在 C=64 反被 E7b 反超（261.5 vs 214.6 ms——TP2PP4 该点活跃 ~59 条贴池顶 + 3 次回退，E7b 池限 16 活跃反而批更轻）。
|
||||
- **调参消除掉图断崖**：E7b 输出从 C=8 的 83.9 单调爬到 C=64 的 99.6 tok/s（+19%），C=16 TPOT P95 从初测 298 ms 降到 228.8 ms，C=32/64 稳定在 ~214 ms——decode 图全程覆盖运行批。EAGLE accept 随并发从 2.56 爬到 2.95。
|
||||
- **prefill 墙成为 E7b 的输出上限**：其 input TPS 从 C=8 的 2,684 到 C=64 仅 +19%（3,188），TP2PP4 同区间 +98%（3,236→6,419）——chunk 8192 + TP8 无 PP 流水 vs chunk 16384 + PP4 摊满。16K 场景 E7b 输出被 prefill 封死在 ~100 tok/s 平台，并发再高也不突破。
|
||||
- **两臂的 C=64 都撞各自的容量墙**：E7b 活跃上限 = KV 池（~16 条，C=32/64 为排队观察点，TTFT P95 140/293 s）；TP2PP4 MRR64 后活跃 ~59 条 = 16K 池顶（C=32→64 输出仅 +4%，3 次回退），但 TTFT P95 从 MRR48 排队态的 134.8 s 压到 89.2 s（−34%）——同样需要准入控制。
|
||||
|
||||
## 4. 短输入与长输出
|
||||
|
||||
### 4.1 `1K -> 128`
|
||||
|
||||
| 并发 | 方案 | Input TPS | Output TPS | TTFT P95 | TPOT P95 |
|
||||
|-|-|-|-|-|-|
|
||||
| 1 | TP2PP4 | 155 | 19.3 | 386 ms | 49.3 ms |
|
||||
| 1 | TP8+EAGLE3+AR | **518** | **64.7** | **326 ms** | **14.3 ms** |
|
||||
| 8 | TP2PP4 | 815 | 102 | **1.69 s** | 72.9 ms |
|
||||
| 8 | TP8+EAGLE3+AR | **1,211** | **151** | 2.17 s | **53.7 ms** |
|
||||
| 32 | TP2PP4 | 1,825 | 228 | **5.49 s** | **120.7 ms** |
|
||||
| 32 | TP8+EAGLE3+AR | **1,833** | **229** | 7.09 s | 150.6 ms |
|
||||
| 64 | TP2PP4 | **3,466** | **433** | **7.16 s** | **116.7 ms** |
|
||||
| 64 | TP8+EAGLE3+AR | 2,178 | 272 | 12.15 s | 292.7 ms |
|
||||
|
||||
短输入下 E7b 在 C=8 显著占优（151 vs 102，1.5×）；C=32 两方案打平（229 vs 228——TP2PP4 的 MRR64 配置代价吃掉了初测 MRR48 276 的领先，见 2 节演进注记）；C=64 TP2PP4 重新拉开（433 vs 272，**1.6×**）且 TTFT/TPOT 双优（7.16 s/116.7 ms vs 12.15 s/292.7 ms）——MRR64 后 64 条满飞无排队，而 E7b 池上限 ~52 活跃、TPOT 被大稳态批拖高。B300 同场景 Low-Latency 到 C=128 才被反超，本机在 C=8~64 之间，主因是容量/配置而非算力。
|
||||
|
||||
### 4.2 `1K -> 4K`
|
||||
|
||||
| 并发 | 方案 | Output TPS | TTFT P95 | TPOT P95 |
|
||||
|-|-|-|-|-|
|
||||
| 1 | TP2PP4 | 20.3 | 395 ms | 49.8 ms |
|
||||
| 1 | TP8+EAGLE3+AR | **135** | **320 ms** | **8.7 ms** |
|
||||
| 8 | TP2PP4 | 101 | 1.87 s | 79.4 ms |
|
||||
| 8 | TP8+EAGLE3+AR | **402** | **1.65 s** | **24.3 ms** |
|
||||
| 32 | TP2PP4 | 347 | **4.95 s** | 101.2 ms |
|
||||
| 32 | TP8+EAGLE3+AR | **676** | 6.19 s | **54.6 ms** |
|
||||
| 64 | TP2PP4 | 572 | **7.30 s** | 112.5 ms |
|
||||
| 64 | TP8+EAGLE3+AR | **826.5** | 12.18 s | **85.1 ms** |
|
||||
|
||||
长输出放大了两形态的差异——调参后 E7b 在本场景全并发档反超：
|
||||
|
||||
- **E7b 全档最优，C=64 = 826.5 tok/s 为全场最高输出吞吐**：C=1/8/32/64 输出 135/402/676/826.5，对 TP2PP4（20.3/101/347/572）为 6.7×/4.0×/1.9×/1.4×；C=64 TPOT 仍优（85.1 vs 112.5 ms），TTFT 被 TP2PP4 的 MRR64 反超（7.30 vs 12.18 s）。EAGLE accept 3.7~4.0——长输出让草稿模型进入"顺笔"状态，显著高于 4.1 短输出行（2.0~2.1）。
|
||||
- **初测的排队断崖已消除**：C=32 TTFT P95 从初测 304.75 s（nreq=64 / MRR=16 四波串行）降到 6.19 s；C=64 从未完成变为 12.18 s。MRR64 下 1K→4K 的池上限约 52 条活跃，C=64 仅 12 条排队，请求几乎全程满飞。
|
||||
- **TP2PP4 本场景被压，但 MRR64 修复了它的 C=64**：1K 短进长出下既无 prefill 墙可摊、也无投机解码加成；C=64 输出 482→572 tok/s（+19%）、TTFT P95 从 337.92 s 塌缩到 7.30 s（48 活跃+16 排队 → 64 满飞），不再只限离线批处理，但输出仍被 E7b 反超 45%。
|
||||
|
||||
## 5. 长上下文观察
|
||||
|
||||
### 5.1 `64K/128K -> 512`
|
||||
|
||||
（并发档位按用户指示收敛：64K 封 8、128K 封 4，另补 128K C=2；B300 同场景为 64K C=8/32/64、128K C=8/32。）
|
||||
|
||||
| 场景 | 并发 | 方案 | Input TPS | Output TPS | TTFT P95 | TPOT P95 |
|
||||
|-|-|-|-|-|-|-|
|
||||
| 64K→512 | 1 | TP2PP4 | 1,834 | 14.3 | **10.32 s** | 50.0 ms |
|
||||
| 64K→512 | 1 | TP8+EAGLE3+AR | **2,877** | **22.5** | 17.17 s | **13.5 ms** |
|
||||
| 64K→512 | 4 | TP2PP4 | **4,287** | **33.5** | **28.83 s** | **99.5 ms** |
|
||||
| 64K→512 | 4 | TP8+EAGLE3+AR | 3,292 | 25.7 | 68.87 s | 154.6 ms |
|
||||
| 64K→512 | 8 | TP2PP4 | **5,650** | **44.1** | **53.38 s** | **161.8 ms** |
|
||||
| 64K→512 | 8 | TP8+EAGLE3+AR | 3,280 | 25.6 | 149.45 s | 157.5 ms |
|
||||
| 128K→512 | 1 | TP2PP4 | 3,017 | 11.8 | **18.23 s** | 49.7 ms |
|
||||
| 128K→512 | 1 | TP8+EAGLE3+AR | **3,054** | **11.9** | 37.60 s | **12.3 ms** |
|
||||
| 128K→512 | 2 | TP2PP4 | **4,310** | **16.8** | **32.42 s** | **83.7 ms** |
|
||||
| 128K→512 | 2 | TP8+EAGLE3+AR | 3,145 | 12.3 | 75.81 s | 162.2 ms |
|
||||
| 128K→512 | 4 | TP2PP4 | **5,626** | **22.0** | **60.35 s** | **146.8 ms** |
|
||||
| 128K→512 | 4 | TP8+EAGLE3+AR | 3,133 | 12.2 | 159.50 s | 163.4 ms |
|
||||
|
||||
- 64K C=1 E7b 仍占优（22.5 vs 14.3 tok/s），但 128K C=1 两方案输出打平（11.8 vs 11.9）——prefill 逐渐成为长上下文的主导成本，E7b 的 decode 优势被稀释；其 TTFT 反而慢 2×（37.6 vs 18.2 s）。
|
||||
- C≥2 起 TP2PP4 全指标占优；E7b 的 output TPS 在 64K/128K 行几乎不随并发变化（22.5→25.6、11.9→12.2）——KV 池把活跃钉在 ~4/~2 条，并发收益被容量封死（与 MRR/图调参无关，故沿用初测值）。
|
||||
- DSA 的 TPOT 上下文不变性（TP2PP4 C=1：50.0 ms @64K ≈ 49.7 ms @128K ≈ 50.3 ms @16K）与并发驱动性（C=8：70.9 ms @16K → 161.8 ms @64K）在本组完整呈现。
|
||||
|
||||
### 5.2 上下文边界
|
||||
|
||||
（OSL=1，只验证容量与 prefill，不比较 Output TPS/TPOT——与 B300 同声明。B300 完成到约 1M；本机以 896K=917,504 tokens 对应 B300"约 1M"档。）
|
||||
|
||||
| 输入长度 | 方案 | 已完成并发 | C=1 Input TPS | 最高并发 TTFT P95 |
|
||||
|-|-|-|-|-|
|
||||
| 256K | TP2PP4 | 1/2/3 | **7,050** | 102.99 s（C=3） |
|
||||
| 256K | TP8+EAGLE3+AR | 1 | 2,867 | 91.45 s（C=1） |
|
||||
| 512K | TP2PP4 | 1 | **5,662** | 92.60 s（C=1） |
|
||||
| 512K | TP8+EAGLE3+AR | 结构性不可（ctx 270,336） | — | — |
|
||||
| 896K | TP2PP4 | 1 | **4,153** | 220.91 s（C=1） |
|
||||
| 896K | TP8+EAGLE3+AR | 结构性不可（ctx 270,336） | — | — |
|
||||
|
||||
- 256K C=1 两臂相差 2.5×（7,050 vs 2,867 tok/s）：TP2PP4 的 chunk 16384 + PP4 流水对超长 prefill 的摊满优势，在边界长度上比 128K 行（几乎打平）进一步放大；E7b 的 chunk 8192 代价随长度累积。
|
||||
- TP2PP4 边界 input TPS 随长度衰减平缓（7,050 → 5,662 → 4,153），896K 单条 220.9 s 完成、池 1,040,384 tokens 单条可容（KV 917,504 + 余量）。
|
||||
- **E7b 256K C=1 命中核验注记（hicache 宿主层陷阱）**：首测命中率 0.9998、重试仍超——根因是该点 nreq=1 测量文本与预热完全相同，256K 预热 KV（262,160 tokens）占池 94.8% 触发分层缓存宿主层下放，`flush_cache` 只清 GPU radix 树、清不掉宿主层。改用该服务实例从未发过的文本（窗口基址 9,900,000）无预热重测，命中 0.0，数据干净。此为分层缓存运维要点：**宿主层缓存不受 flush_cache 影响，冷测必须换文本**。
|
||||
|
||||
## 6. 显存状态
|
||||
|
||||
- **TP2PP4**：服务加载后空载 64.6 GiB/卡，矩阵峰值 **84.8 GiB/卡**（MRR64 复测矩阵，主场景 C=64 活跃 ~59 条贴池顶时；初测 MRR48 峰值 85.0 GiB、最紧张卡余量约 0.6 GiB）。mem 0.85 下 KV 池按卡容量贴满分配，属预期；MRR64 已把 16K 场景推到池顶，继续上调并发或上下文没有余量，扩容前必须先降 mem-fraction。
|
||||
- **TP8+EAGLE3+AR**：空载 77.5 GiB/卡（EAGLE 草稿权重 + mem 0.90 大池 + 13 档 decode 图，图捕获完成后余 6.11 GB/卡），复测矩阵峰值 **81.7 GiB/卡**（余量约 3.9 GiB；初测臂含 64K/128K/256K 长上下文的 hicache 传输，峰值 83.6 GiB/卡）。
|
||||
- 全矩阵 **0 OOM**；唯一出现回退（retraction）的点是 TP2PP4 主场景 C=64（MRR64 对 16K 池超配，两轮复测各 3 次；MRR48 初测同点 0 回退）——B300 未披露该指标，除该点外全部点 retractions_total=0。
|
||||
- 显存时间线逐 30 s 采样留档（vram_timeline.csv），可复核任一时刻的卡间分布。
|
||||
|
||||
## 7. 建议
|
||||
|
||||
1. **低并发交互/agent 长思考（C≤8）用 E7b**：主场景 C=1 TPOT 13.7 ms、长输出 C=8 输出 402 tok/s；该区间对 TP2PP4 的优势最大（3.0~6.7×）。
|
||||
2. **prefill 密集/长上下文（16K 主场景、输入 ≥64K）用 TP2PP4**：主场景 C=64 输出 201 tok/s（E7b 99.6）、128K C=1 TTFT 18.2 s、896K 可达；MRR64 后 16K 场景活跃 ~59 条为池封顶，1K 场景吞吐上限即 MRR=64。
|
||||
3. **MRR 按负载形态选（本轮单变量结论）**：需要 C=64 档或低 TTFT 时 MRR64 收益确定——4.1 C=64 +27%、4.2 C=64 +19%、三场景 C=64 TTFT P95 全部塌缩（20.0/337.9/134.8 s → 7.2/7.3/89.2 s）；但 1K 短输入 C≤32 档付出 ~14% 配置代价（4.1 C=32 276→228，全新实例 MRR48 对照定案），16K 场景无代价。以 1K 短输入为主且并发 ≤48 的生产口可保持 MRR48；两档并存可按场景路由。
|
||||
4. **负载形态分界线（调参后）**：decode 密集（短进长出，1K→4K）任何并发档 E7b 全优（C=64 输出 826 vs 572 tok/s）；prefill 密集（16K 主场景）C≥8 TP2PP4 全优（201 vs 100）；短输入短输出（1K→128）C=8 E7b、C=32 打平、C=64 TP2PP4（1.6×）。选型看输入/输出长度分布，不能只看并发。
|
||||
5. **E7b 剩余瓶颈在 prefill 墙与 KV 池，不再在图**：MRR64 + 图桶 1–64 已消掉掉图断崖（本报告即复测值）；16K 主场景输出封顶 ~100 tok/s 是 chunk 8192 + TP8 无 PP 流水所致。扩并发容量的唯一杠杆是 KV 池（hicache 宿主层只救命中场景，不增并发容量）。
|
||||
6. **边界与超长上下文只有 TP2PP4 口径可服务**：E7b 若要对标 B300 512K/约1M 行，需要 ctx ≥524,288 与池 ≥52 万 tokens 的部署形态，本版（ctx 270,336 / 池 276,480）结构性不可达。
|
||||
7. **不要把两臂差异单归因 EAGLE**：两臂同时差在并行拓扑、chunk、radix、MRR 与图覆盖；单变量消融未做（与 B300 报告建议 3 同款）。MRR64 在 1K 短输入的 −14% 代价同样只有三实例 A/B 事实、机制未定位。
|
||||
8. **生产容量同时设吞吐和延迟 SLO**：TP2PP4 主场景 C=64 输出最高但 TTFT P95 仍达 89 s；E7b 主场景 C=64 TTFT P95 293 s（池限 16 活跃的排队）。只看峰值 TPS 会掩盖排队长尾。
|
||||
9. **分层缓存运维**：宿主层缓存不受 `flush_cache` 影响，任何冷缓存测量/复测必须更换输入文本（见 5.2 注记）。
|
||||
|
||||
## 8. 原始结果与复现
|
||||
|
||||
- 服务器原始结果（60.8）：
|
||||
- TP2PP4 臂：`/root/bench_logs/b300eq_tp2pp4_20260910_1138/`（all_results.jsonl 24 点、status.txt、server_facts.txt、gpu_inventory、vram_timeline.csv）
|
||||
- E7b 臂初测（MRR16/图 1–8，留档基线）：`/root/bench_logs/b300eq_e7b_20260910_1428/`（all_results.jsonl 20 条：16 点 OK + 4.2 C=64 用户中止 + 256K C=1 干净重测；status.txt 含 4 个结构性跳过与 HIT_FAIL_FINAL 首测记录）
|
||||
- E7b 臂复测（MRR64/图 1–64，**正文采用值**）：`/root/bench_logs/b300eq_e7b64_20260910_1732/`（all_results.jsonl 10 点全 OK：16K/4.1/4.2 三场景 C=8/16/32/64，命中核验全 0.0、0 retraction）；前后对比表 `/root/bench_logs/retest_compare.md`
|
||||
- TP2PP4 MRR64 复测 pass-1：`/root/bench_logs/b300eq_pp4mrr64_20260910_1927/`（6 点全 OK，与 v2 对齐 ≤1.8%）
|
||||
- TP2PP4 MRR64 复测 v2（**正文采用值**，全新实例 + 回退点排末位）：`/root/bench_logs/b300eq_pp4mrr64v2_20260910_2021/`（6 点全 OK；16K C=64 3 次回退为 MRR64 池超配的确定性行为）；前后对比表 `/root/bench_logs/retest_compare_pp4_v2.md`
|
||||
- TP2PP4 MRR48 对照（归因实验，全新实例单点）：`/root/bench_logs/b300eq_mrr48ctl_20260910_2112/`（4.1 C=32 = 266.42 tok/s）
|
||||
- 资产 md5 台账：`/root/bench_logs/b300eq_md5_ledger.txt`
|
||||
- 本地镜像：`D:\sskj\b300eq\{tp2pp4,e7b,e7b64}\`（上述全部文件）、`D:\sskj\b300eq\pp4mrr64_mirror\`（TP2PP4 复测两轮 + 对照全量）、`D:\sskj\b300eq\report_tables.md`（表格生成器输出）、`D:\sskj\b300eq\retest_compare.md`（E7b64 前后对比）、`D:\sskj\b300eq\pp4mrr64_mirror\retest_compare_pp4_v2.md`（TP2PP4 前后对比）
|
||||
- 部署脚本：`/root/deploy_glm53_pp4.sh`（md5 def3c64c…，与库内 sskj main 副本一致）、`/root/deploy_glm53_pp4_mrr64.sh`（533440ef…，TP2PP4 高并发版：仅 MRR 48→64 一处差异）、`/root/deploy_glm53_607_exp.sh`（E7b 在役配方）、`/root/deploy_glm53_e7b_hicc.sh`（165db732…，E7b 高并发版：MRR64 + 图桶 1–64，其余与前者逐字一致）；复测驱动 `/root/run_retest_e7b64.sh`（fe46da2e…）、`/root/run_retest_pp4mrr64.sh`（c9b6361d…，pass-1）、`/root/run_retest_pp4mrr64_v2.sh`（7d857816…，v2 有序版）、`/root/launch_v2_after_pass1.sh`（889c2428…，衔接 wrapper）、`/root/control_mrr48_41c32.sh`（1ff8a21f…，对照 + 自动恢复链）；对比生成器 `/root/gen_retest_compare.py`、`/root/gen_retest_compare_pp4.py`；CAR 补丁：`/root/patches/custom_all_reduce.py`（a8fc9a50…）+ `custom_all_reduce_utils.py`（65a4d22b…），三处（60.7 原件/本地/库内）md5 一致
|
||||
- 测量工具：`/root/bench_corpus_v2.py`（md5 1e34dd8d…，p95 nearest-rank + 逐请求 dump）、`/root/extract_summary.py`（4c126d06…）、`/root/run_b300_matrix.sh`（5892b446…，矩阵驱动：alive/idle_wait/prewarm/flush/命中核验/重试/VRAM 采样）
|
||||
- 复现命令（单点示例）：
|
||||
|
||||
```bash
|
||||
# 冷缓存压测（E7b 256K C=1 干净版）
|
||||
python3 /root/bench_corpus_v2.py --input-len 262144 --output-len 1 --shared-frac 0 \
|
||||
--concurrency 1 --num-requests 1 --run-id 9551 --pool-override 9900000 \
|
||||
--dump-records $L/b52_256k_c1_v2_records.jsonl
|
||||
# 全矩阵：nohup bash /root/run_b300_matrix.sh <arm> > <progress.log> 2>&1 &
|
||||
# E7b 高并发复测（MRR64/图≤64，10 点）：nohup bash /root/run_retest_e7b64.sh > <progress.log> 2>&1 &
|
||||
# TP2PP4 MRR64 复测（v2 有序，回退点排末位）：nohup bash /root/run_retest_pp4mrr64_v2.sh > <progress.log> 2>&1 &
|
||||
```
|
||||
|
||||
## 9. 与 B300 对比观察
|
||||
|
||||
> **口径声明**：B300 报告未写明模型量化方式（若为原始 BF16 权重，则与本机 NVFP4 非同模型形态）；硬件为 8×B300（288 GB HBM3e）vs 本机 8×RTX 6000D（96 GB GDDR7）；镜像 v0.5.18-cu130-dev4 vs nightly-20260828-daf63171。**绝对值仅量级可比，本对比只对结构性结论负责**。
|
||||
|
||||
- **定性结构完全复现**：低延迟配方（B300 Low-Latency = TP8+EAGLE vs 本机 E7b = TP8+EAGLE3+AR）在 C=1 占优、吞吐配方（B300 High-Throughput = DP8+DeepEP vs 本机 TP2PP4 = D 生产口径）在高并发占优——两套硬件上"低延迟 vs 高吞吐"的分野方向一致。
|
||||
- **分界点本机更靠前，且调参后由负载形态决定**：B300 的交叉点在 C=64~128（主场景 HT C=128 反超 33%）；本机主场景交叉仍在 C=8 附近（容量上限所致：本臂 MRR64/池贴边 16 活跃 vs B300 配方 256/默认，先于算力撞墙），但 decode 密集场景（1K→4K）E7b 调参后全档占优、无交叉——这一形态差异 B300 报告未呈现。
|
||||
- **绝对差距 4~5×（主场景）**：C=1 输出 246 vs 49.6 tok/s（5.0×）、input 7,882 vs 1,587（5.0×）；吞吐侧峰值 997 vs 201（5.0×）、31,889 vs 6,419（5.0×）。与显存带宽硬件代差量级一致。
|
||||
- **边界 prefill 差距收窄到 ~2×**：256K C=1 input 7,050 vs 17,457（2.5×）→ 512K 5,662 vs 12,421（2.2×）→ 896K/约1M 4,153 vs 7,897（1.9×）。计算密集的超长 prefill 是 6000D 相对最能打的位置（PP 流水摊满 + 带宽占比下降）。
|
||||
- **TPOT 差距小于吞吐差距**：B300 LL C=1 4.36 ms vs E7b 13.7 ms（3.1×）；高并发侧 B300 HT C=128 165 ms vs TP2PP4 C=64 261.5 ms（1.6×，本机该点为池顶运行、含 3 次回退的批扰动；MRR48 排队态为 203.7 ms）——NVFP4 + DSA 把 decode 单步成本压得相对不差，差距主要在吞吐面。
|
||||
- **饱和形态不同**：B300 LL 在 C=64 后进入 24K input tok/s 平台、HT 在 C=128 达峰后 C=256 回退 19%；本机 TP2PP4 到 C=64 到达 16K 池顶（活跃 ~59 条封顶，C=32→64 输出仅 +4%），E7b 调参后在 16K 场景呈 ~100 tok/s 输出平台（池限 16 活跃 + prefill 墙）、在 1K→4K 场景爬到 826 tok/s 无回退。本机没有一档出现吞吐回退——"甜点=并发上限"由 MRR/池决定而非算力。
|
||||
- **EAGLE 配方差异**：B300 LL 为 5 steps/6 draft tokens，本机 E7b 为 4 steps/topk1/5 draft tokens；本机实测 accept 2.0~4.0（短输出 2.0、主场景 2.1~3.0、长输出 3.7~4.0，随 decode 深入上升）。B300 未披露 accept，无法直接对比投机效率。
|
||||
- **容量边界差距最大**：B300 两模式都完成约 1M 输入 C=1/2/4；本机仅 TP2PP4 可达 896K 且 C=1 单条（池 1,040,384 刚容一条），E7b 连 512K 都结构性不可测（ctx 270,336）。96 GB 卡上"上下文边界=显存边界"比 B300 严酷得多。
|
||||
|
||||
## 附录 A：语料窗口映射（回收窗口）
|
||||
|
||||
语料总量 21,296,780 tokens，此前场景一/二战役已消费至 21,235,008。冷缓存协议下回收复用：窗口基址 `--pool-override` 显式指定，每点窗口在基址上顺序推进（逐记录 `corpus_window.start/end` 留档），每点 flush + 命中核验 ≤0.01 保证冷。E7b 复测臂沿用与初测相同的窗口基址（2,300,000 / 4,500,000 / 4,700,000）——复测为全新容器实例，这些文本对 hicache 宿主层重新成为"处女文本"，冷缓存协议成立（复测 10 点命中核验全部 0.0）。TP2PP4 复测两轮（pass-1/v2）与 MRR48 对照同样沿用该三处基址（radix 关 + flush 即冷，13 次测量命中核验全部 0.0）。
|
||||
|
||||
| 场景 | 窗口基址 | 备注 |
|
||||
|-|-|-|
|
||||
| 主场景 16K→512 | 2,300,000 | |
|
||||
| 4.1 短输入 1K→128 | 4,500,000 | |
|
||||
| 4.2 长输出 1K→4K | 4,700,000 | |
|
||||
| 5.1 64K→512 | 5,000,000 | |
|
||||
| 5.1 128K→512 | 6,200,000 | |
|
||||
| 5.2 256K→1 | 8,400,000 | E7b 干净重测改用 9,900,000（实例首用文本，避 hicache 宿主层残留） |
|
||||
| 5.2 512K→1 | 9,300,000 | 仅 TP2PP4 |
|
||||
| 5.2 896K→1 | 9,900,000 | 仅 TP2PP4 |
|
||||
|
||||
## 附录 B：全量指标（含 mean/p95/max、回退、投机接受长度）
|
||||
|
||||
| 场景点 | 方案 | ok/nreq | wall(s) | out tok/s | in tok/s | TTFT mean/p95/max(s) | TPOT mean/p95/max(ms) | retractions | accept |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| b3_16k_c1 | TP2PP4 | 8/8 | 245.55 | 16.68 | 533.8 | 5.35/5.36/5.36 | 49.6/50.3/50.3 | 0 | None |
|
||||
| b3_16k_c1 | TP8+EAGLE3+AR | 8/8 | 82.58 | 49.6 | 1587.15 | 3.93/3.98/3.98 | 12.5/13.7/13.7 | 0 | 2.138 |
|
||||
| b3_16k_c8 | TP2PP4 | 16/16 | 81.0 | 101.13 | 3236.22 | 10.05/14.79/14.79 | 59.5/70.9/70.9 | 0 | None |
|
||||
| b3_16k_c8 | TP8+EAGLE3+AR | 16/16 | 97.66 | 83.88 | 2684.26 | 12.12/32.47/32.47 | 69.2/136.1/136.1 | 0 | 2.556 |
|
||||
| b3_16k_c16 | TP2PP4 | 32/32 | 112.16 | 146.08 | 4674.5 | 15.53/25.78/25.84 | 79.2/98.5/101.2 | 0 | None |
|
||||
| b3_16k_c16 | TP8+EAGLE3+AR | 32/32 | 177.89 | 92.1 | 2947.27 | 19.76/59.91/63.69 | 129.7/228.8/275.9 | 0 | 2.485 |
|
||||
| b3_16k_c32 | TP2PP4 | 64/64 | 170.33 | 192.38 | 6156.1 | 26.48/46.31/47.97 | 114.5/153.2/159.2 | 0 | None |
|
||||
| b3_16k_c32 | TP8+EAGLE3+AR | 64/64 | 335.79 | 97.58 | 3122.67 | 82.65/140.33/154.66 | 138.1/213.6/277.5 | 0 | 2.777 |
|
||||
| b3_16k_c64 | TP2PP4 | 128/128 | 326.73 | 200.58 | 6418.52 | 50.76/89.22/147.41 | 185.0/261.5/313.1 | 3 | None |
|
||||
| b3_16k_c64 | TP8+EAGLE3+AR | 128/128 | 657.79 | 99.63 | 3188.2 | 204.27/293.21/322.84 | 142.5/214.6/276.6 | 0 | 2.954 |
|
||||
| b41_1k_c1 | TP2PP4 | 8/8 | 52.95 | 19.34 | 154.7 | 0.38/0.39/0.39 | 49.1/49.3/49.3 | 0 | None |
|
||||
| b41_1k_c1 | TP8+EAGLE3+AR | 8/8 | 15.82 | 64.74 | 517.93 | 0.30/0.33/0.33 | 13.2/14.3/14.3 | 0 | 2.028 |
|
||||
| b41_1k_c8 | TP2PP4 | 16/16 | 20.1 | 101.91 | 815.3 | 1.31/1.69/1.69 | 68.6/72.9/72.9 | 0 | None |
|
||||
| b41_1k_c8 | TP8+EAGLE3+AR | 16/16 | 13.53 | 151.33 | 1210.61 | 1.25/2.17/2.17 | 42.1/53.7/53.7 | 0 | 1.992 |
|
||||
| b41_1k_c32 | TP2PP4 | 64/64 | 35.92 | 228.08 | 1824.68 | 4.60/5.49/5.52 | 104.1/120.7/137.5 | 0 | None |
|
||||
| b41_1k_c32（MRR48 对照·全新实例） | TP2PP4 | 64/64 | 30.75 | 266.42 | 2131.36 | 4.04/4.92/4.93 | 88.4/103.0/110.0 | 0 | None |
|
||||
| b41_1k_c32 | TP8+EAGLE3+AR | 64/64 | 35.75 | 229.18 | 1833.42 | 2.99/7.09/7.10 | 111.9/150.6/179.8 | 0 | 2.035 |
|
||||
| b41_1k_c64 | TP2PP4 | 128/128 | 37.82 | 433.2 | 3465.61 | 5.46/7.16/7.24 | 104.3/116.7/147.3 | 0 | None |
|
||||
| b41_1k_c64 | TP8+EAGLE3+AR | 128/128 | 60.18 | 272.24 | 2177.88 | 4.76/12.15/13.48 | 191.1/292.7/331.5 | 0 | 2.094 |
|
||||
| b42_1k4k_c1 | TP2PP4 | 8/8 | 1614.16 | 20.3 | 5.08 | 0.39/0.40/0.40 | 49.2/49.8/49.8 | 0 | None |
|
||||
| b42_1k4k_c1 | TP8+EAGLE3+AR | 8/8 | 242.18 | 135.31 | 33.83 | 0.31/0.32/0.32 | 7.3/8.7/8.7 | 0 | 3.701 |
|
||||
| b42_1k4k_c8 | TP2PP4 | 16/16 | 649.69 | 100.87 | 25.22 | 1.35/1.87/1.87 | 79.0/79.4/79.4 | 0 | None |
|
||||
| b42_1k4k_c8 | TP8+EAGLE3+AR | 16/16 | 162.84 | 402.47 | 100.62 | 0.95/1.65/1.65 | 18.5/24.3/24.3 | 0 | 3.951 |
|
||||
| b42_1k4k_c32 | TP2PP4 | 64/64 | 756.13 | 346.69 | 86.67 | 2.85/4.95/4.97 | 91.0/101.2/101.4 | 0 | None |
|
||||
| b42_1k4k_c32 | TP8+EAGLE3+AR | 64/64 | 387.9 | 675.81 | 168.95 | 2.58/6.19/6.20 | 43.0/54.6/60.4 | 0 | 3.82 |
|
||||
| b42_1k4k_c64 | TP2PP4 | 128/128 | 916.56 | 572.01 | 143.0 | 3.47/7.30/7.32 | 109.7/112.5/115.4 | 0 | None |
|
||||
| b42_1k4k_c64 | TP8+EAGLE3+AR | 128/128 | 634.35 | 826.5 | 206.62 | 4.36/12.18/12.50 | 69.9/85.1/101.5 | 0 | 3.878 |
|
||||
| b51_64k_c1 | TP2PP4 | 8/8 | 285.91 | 14.33 | 1833.72 | 10.28/10.32/10.32 | 49.8/50.0/50.0 | 0 | None |
|
||||
| b51_64k_c1 | TP8+EAGLE3+AR | 8/8 | 182.23 | 22.48 | 2877.06 | 17.04/17.17/17.17 | 11.2/13.5/13.5 | 0 | 2.468 |
|
||||
| b51_64k_c4 | TP2PP4 | 8/8 | 122.3 | 33.49 | 4286.94 | 19.55/28.83/28.83 | 81.4/99.5/99.5 | 0 | None |
|
||||
| b51_64k_c4 | TP8+EAGLE3+AR | 8/8 | 159.24 | 25.72 | 3292.45 | 30.55/68.87/68.87 | 94.9/154.6/154.6 | 0 | 2.438 |
|
||||
| b51_64k_c8 | TP2PP4 | 16/16 | 185.59 | 44.14 | 5650.11 | 31.83/53.38/53.38 | 119.2/161.8/161.8 | 0 | None |
|
||||
| b51_64k_c8 | TP8+EAGLE3+AR | 16/16 | 319.66 | 25.63 | 3280.31 | 90.92/149.45/149.45 | 105.8/157.5/157.5 | 0 | 2.442 |
|
||||
| b51_128k_c1 | TP2PP4 | 8/8 | 347.5 | 11.79 | 3017.45 | 18.21/18.23/18.23 | 49.4/49.7/49.7 | 0 | None |
|
||||
| b51_128k_c1 | TP8+EAGLE3+AR | 8/8 | 343.36 | 11.93 | 3053.83 | 37.59/37.60/37.60 | 10.4/12.3/12.3 | 0 | 2.728 |
|
||||
| b51_128k_c2 | TP2PP4 | 8/8 | 243.3 | 16.84 | 4309.84 | 25.27/32.42/32.42 | 69.6/83.7/83.7 | 0 | None |
|
||||
| b51_128k_c2 | TP8+EAGLE3+AR | 8/8 | 333.36 | 12.29 | 3145.44 | 48.21/75.81/75.81 | 68.0/162.2/162.2 | 0 | 2.769 |
|
||||
| b51_128k_c4 | TP2PP4 | 8/8 | 186.37 | 21.98 | 5626.35 | 39.29/60.35/60.35 | 105.4/146.8/146.8 | 0 | None |
|
||||
| b51_128k_c4 | TP8+EAGLE3+AR | 8/8 | 334.71 | 12.24 | 3132.76 | 110.14/159.50/159.50 | 68.7/163.4/163.4 | 0 | 2.595 |
|
||||
| b52_256k_c1 | TP2PP4 | 1/1 | 37.18 | 0.03 | 7050.23 | 37.18/37.18/37.18 | — | 0 | None |
|
||||
| b52_256k_c1 | TP8+EAGLE3+AR | 1/1 | 91.45 | 0.01 | 2866.59 | 91.45/91.45/91.45 | — | 0 | None |
|
||||
| b52_256k_c2 | TP2PP4 | 2/2 | 70.19 | 0.03 | 7469.22 | 53.69/70.12/70.12 | — | 0 | None |
|
||||
| b52_256k_c3 | TP2PP4 | 3/3 | 103.12 | 0.03 | 7626.22 | 70.12/102.99/102.99 | — | 0 | None |
|
||||
| b52_512k_c1 | TP2PP4 | 1/1 | 92.6 | 0.01 | 5662.03 | 92.60/92.60/92.60 | — | 0 | None |
|
||||
| b52_896k_c1 | TP2PP4 | 1/1 | 220.91 | 0.0 | 4153.24 | 220.91/220.91/220.91 | — | 0 | None |
|
||||
|
||||
注：TP2PP4 的 16K/4.1/4.2 场景 C=32/C=64 共 6 点为 MRR64 复测 v2 值（正文采用值；pass-1 两轮对齐 ≤1.8% 留档 `results/pp4mrr64/`），C=1/C=8/C=16 与 5.1/5.2 行沿用初测 MRR48 值（MRR 在这些档位不构成约束）；"MRR48 对照"行是 4.1 C=32 的全新实例归因实验（276 初测 → 266 对照 → 228 复测 = −3% 实例新鲜度 −14% MRR64 配置代价）。E7b 的 16K/4.1/4.2 场景 C=8~C=64 共 10 点为调参后（MRR64/decode 图 1–64）复测值，即正文采用值；C=1 各点与 5.1/5.2 行沿用初测值（活跃数受池上限约束，与调参无关）。E7b 256K/512K/896K C>1 为结构性跳过；256K C=1 为全新文本干净重测值（窗口 9,900,000）。初测全量原始数据留档于 `results/tp2pp4/`（MRR48）与 `results/e7b/`（MRR16/图 1–8）。
|
||||
@ -1,424 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
# Adapted from https://github.com/vllm-project/vllm/blob/v0.6.4.post1/vllm/distributed/device_communicators/custom_all_reduce.py
|
||||
|
||||
import ctypes
|
||||
import logging
|
||||
import os
|
||||
from contextlib import contextmanager
|
||||
from functools import partial
|
||||
from typing import Any, List, Optional, Union
|
||||
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
from torch.distributed import ProcessGroup
|
||||
|
||||
import sglang.srt.distributed.device_communicators.custom_all_reduce_ops as ops
|
||||
from sglang.srt.distributed.device_communicators.cuda_wrapper import CudaRTLibrary
|
||||
from sglang.srt.distributed.device_communicators.custom_all_reduce_utils import (
|
||||
can_use_custom_all_reduce_with_nvlink,
|
||||
is_weak_contiguous,
|
||||
)
|
||||
from sglang.srt.environ import envs
|
||||
from sglang.srt.model_executor.runner_backend_utils.tc_piecewise_cuda_graph import (
|
||||
is_in_tc_piecewise_cuda_graph,
|
||||
)
|
||||
from sglang.srt.utils import (
|
||||
get_bool_env_var,
|
||||
is_cuda,
|
||||
is_hip,
|
||||
is_musa,
|
||||
log_info_on_rank0,
|
||||
)
|
||||
|
||||
_is_cuda = is_cuda()
|
||||
_is_hip = is_hip()
|
||||
_is_musa = is_musa()
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
os.environ.setdefault("SGLANG_CUSTOM_ALLREDUCE_ALGO", "1stage") # SSKJ-PATCH: C++ dispatch no-ops when full_nvlink=False; force kernel launch
|
||||
|
||||
|
||||
class CustomAllreduce:
|
||||
_SUPPORTED_WORLD_SIZES = [2, 4, 6, 8]
|
||||
_MAX_CAR_SIZE = 8192 * 1024
|
||||
if _is_hip:
|
||||
# crossover is at 16MB buffer size for ROCm
|
||||
_MAX_CAR_SIZE = 2 * 8192 * 1024
|
||||
if _is_musa:
|
||||
# crossover is at 128MB buffer size for MUSA
|
||||
_MAX_CAR_SIZE = 16 * 8196 * 1024
|
||||
|
||||
# max_size: max supported allreduce size
|
||||
def __init__(
|
||||
self,
|
||||
group: ProcessGroup,
|
||||
device: Union[int, str, torch.device],
|
||||
max_size=_MAX_CAR_SIZE,
|
||||
) -> None:
|
||||
"""
|
||||
Args:
|
||||
group: the process group to work on. If None, it will use the
|
||||
default process group.
|
||||
device: the device to bind the CustomAllreduce to. If None,
|
||||
it will be bind to f"cuda:{local_rank}".
|
||||
It is the caller's responsibility to make sure each communicator
|
||||
is bind to a unique device, and all communicators in this group
|
||||
are in the same node.
|
||||
"""
|
||||
self._IS_CAPTURING = False
|
||||
self.disabled = True # This can be modified in-place by context manager in piecewise cuda graph runner
|
||||
self.original_disabled = True # To store the original state
|
||||
self.use_amd_deterministic_impl = _use_amd_deterministic_impl()
|
||||
|
||||
if not ops.IS_CUSTOM_AR_AVAILABLE:
|
||||
# disable because of missing custom allreduce library
|
||||
# e.g. in a non-cuda environment
|
||||
return
|
||||
|
||||
rank = dist.get_rank(group=group)
|
||||
world_size = dist.get_world_size(group=group)
|
||||
|
||||
if isinstance(device, int):
|
||||
device = torch.device(f"cuda:{device}")
|
||||
elif isinstance(device, str):
|
||||
device = torch.device(device)
|
||||
# now `device` is a `torch.device` object
|
||||
assert isinstance(device, torch.device)
|
||||
self.device = device
|
||||
full_nvlink = can_use_custom_all_reduce_with_nvlink(
|
||||
group=group,
|
||||
device=device,
|
||||
supported_world_size=self._SUPPORTED_WORLD_SIZES,
|
||||
cls_name="CustomAllreduce",
|
||||
)
|
||||
if full_nvlink is None:
|
||||
return # fail to get nvlink status
|
||||
|
||||
self.group = group
|
||||
self.max_size = max_size
|
||||
self.rank = rank
|
||||
self.world_size = world_size
|
||||
self.full_nvlink = full_nvlink
|
||||
|
||||
if not _is_hip:
|
||||
# Buffers memory are owned by this Python class and passed to C++.
|
||||
# Meta data composes of two parts: meta data for synchronization and a
|
||||
# temporary buffer for storing intermediate allreduce results.
|
||||
self.meta_ptrs = self.create_shared_buffer(
|
||||
ops.meta_size() + max_size, group=group
|
||||
)
|
||||
# This is a pre-registered IPC buffer. In eager mode, input tensors
|
||||
# are first copied into this buffer before allreduce is performed
|
||||
self.buffer_ptrs = self.create_shared_buffer(max_size, group=group)
|
||||
# This is a buffer for storing the tuples of pointers pointing to
|
||||
# IPC buffers from all ranks. Each registered tuple has size of
|
||||
# 8*world_size bytes where world_size is at most 8. Allocating 8MB
|
||||
# is enough for 131072 such tuples. The largest model I've seen only
|
||||
# needs less than 10000 of registered tuples.
|
||||
self.rank_data = torch.empty(
|
||||
max_size, dtype=torch.uint8, device=self.device
|
||||
)
|
||||
self._ptr = ops.init_custom_ar(
|
||||
self.meta_ptrs, self.rank_data, rank, self.full_nvlink
|
||||
)
|
||||
ops.register_buffer(self._ptr, self.buffer_ptrs)
|
||||
else:
|
||||
# meta data buffers need to be "uncached" for signal on MI200
|
||||
self.meta = ops.allocate_meta_buffer(ops.meta_size() + max_size)
|
||||
self.buffer = torch.empty(max_size, dtype=torch.uint8, device=self.device)
|
||||
handle = ops.get_meta_buffer_ipc_handle(self.meta)
|
||||
shard_data = (
|
||||
bytes(handle), # ipc handle to base ptr
|
||||
0, # offset of base ptr
|
||||
)
|
||||
handles, offsets = self._gather_ipc_meta(shard_data)
|
||||
self.rank_data = torch.empty(
|
||||
max_size, dtype=torch.uint8, device=self.device
|
||||
)
|
||||
self._ptr = ops.init_custom_ar(
|
||||
self.meta, self.rank_data, handles, offsets, rank, self.full_nvlink
|
||||
)
|
||||
self.register_buffer(self.buffer)
|
||||
|
||||
self.disabled = False
|
||||
self.original_disabled = False # Ensure original_disabled == disabled
|
||||
logger.warning(f"SSKJ_CAR_PATCH_ACTIVE ws={self.world_size} full_nvlink={self.full_nvlink}")
|
||||
self.tms_cudagraph = envs.SGLANG_MEMORY_SAVER_CUDA_GRAPH.get()
|
||||
|
||||
@staticmethod
|
||||
def create_shared_buffer(
|
||||
size_in_bytes: int, group: Optional[ProcessGroup] = None
|
||||
) -> List[int]:
|
||||
"""
|
||||
Creates a shared buffer and returns a list of pointers
|
||||
representing the buffer on all processes in the group.
|
||||
"""
|
||||
lib = CudaRTLibrary()
|
||||
pointer = lib.cudaMalloc(size_in_bytes)
|
||||
if _is_musa:
|
||||
lib.cudaMemset(pointer, 0, size_in_bytes)
|
||||
handle = lib.cudaIpcGetMemHandle(pointer)
|
||||
world_size = dist.get_world_size(group=group)
|
||||
rank = dist.get_rank(group=group)
|
||||
handles = [None] * world_size
|
||||
dist.all_gather_object(handles, handle, group=group)
|
||||
|
||||
pointers: List[int] = []
|
||||
for i, h in enumerate(handles):
|
||||
if i == rank:
|
||||
pointers.append(pointer.value) # type: ignore
|
||||
else:
|
||||
pointers.append(lib.cudaIpcOpenMemHandle(h).value) # type: ignore
|
||||
|
||||
return pointers
|
||||
|
||||
@staticmethod
|
||||
def free_shared_buffer(
|
||||
pointers: List[int], group: Optional[ProcessGroup] = None
|
||||
) -> None:
|
||||
rank = dist.get_rank(group=group)
|
||||
lib = CudaRTLibrary()
|
||||
lib.cudaFree(ctypes.c_void_p(pointers[rank]))
|
||||
|
||||
@contextmanager
|
||||
def capture(self):
|
||||
"""
|
||||
The main responsibility of this context manager is the
|
||||
`register_graph_buffers` call at the end of the context.
|
||||
It records all the buffer addresses used in the CUDA graph.
|
||||
"""
|
||||
try:
|
||||
self._IS_CAPTURING = True
|
||||
yield
|
||||
finally:
|
||||
self._IS_CAPTURING = False
|
||||
if not self.disabled:
|
||||
self.register_graph_buffers()
|
||||
|
||||
def _get_ipc_meta(self, inp: torch.Tensor):
|
||||
# _share_cuda_() doesn't accept meta buffer not allocated from
|
||||
# PyTorch cache allocator, use direct HIP call to get IPC handle
|
||||
handle = ops.get_meta_buffer_ipc_handle(inp)
|
||||
shard_data = (
|
||||
bytes(handle), # ipc handle to base ptr
|
||||
0, # offset of base ptr
|
||||
)
|
||||
return self._gather_ipc_meta(shard_data)
|
||||
|
||||
def _gather_ipc_meta(self, shard_data):
|
||||
# Note: don't use `[[None]] * self.world_size` here
|
||||
# because it will create a list of the same reference
|
||||
all_data: List[Optional[Any]] = [[None] for i in range(self.world_size)]
|
||||
all_data[self.rank][0] = shard_data
|
||||
|
||||
ranks = dist.get_process_group_ranks(group=self.group)
|
||||
ranks.sort()
|
||||
for i, rank in enumerate(ranks):
|
||||
dist.broadcast_object_list(
|
||||
all_data[i], src=rank, group=self.group, device="cpu"
|
||||
)
|
||||
|
||||
# we cannot directly use `dist.all_gather_object` here
|
||||
# because it is incompatible with `gloo` backend under inference mode.
|
||||
# see https://github.com/pytorch/pytorch/issues/126032 for details.
|
||||
|
||||
handles = []
|
||||
offsets = []
|
||||
for i in range(len(all_data)):
|
||||
handles.append(all_data[i][0][0]) # type: ignore
|
||||
offsets.append(all_data[i][0][1]) # type: ignore
|
||||
return handles, offsets
|
||||
|
||||
def register_buffer(self, inp: torch.Tensor):
|
||||
handles, offsets = self._get_ipc_meta(inp)
|
||||
ops.register_buffer(self._ptr, inp, handles, offsets)
|
||||
|
||||
def register_graph_buffers(self):
|
||||
if _is_hip:
|
||||
handle, offset = ops.get_graph_buffer_ipc_meta(self._ptr)
|
||||
handles, offsets = self._gather_ipc_meta((bytes(handle), offset))
|
||||
log_info_on_rank0(logger, f"Registering {len(offset)} cuda graph addresses")
|
||||
ops.register_graph_buffers(self._ptr, handles, offsets)
|
||||
else:
|
||||
handle, offset = ops.get_graph_buffer_ipc_meta(self._ptr)
|
||||
log_info_on_rank0(logger, f"Registering {len(offset)} cuda graph addresses")
|
||||
# We cannot directly use `dist.all_gather_object` here
|
||||
# because it is incompatible with `gloo` backend under inference mode.
|
||||
# see https://github.com/pytorch/pytorch/issues/126032 for details.
|
||||
all_data = [
|
||||
[None, None] for _ in range(dist.get_world_size(group=self.group))
|
||||
]
|
||||
all_data[self.rank] = [handle, offset]
|
||||
ranks = sorted(dist.get_process_group_ranks(group=self.group))
|
||||
for i, rank in enumerate(ranks):
|
||||
dist.broadcast_object_list(
|
||||
all_data[i], src=rank, group=self.group, device="cpu"
|
||||
)
|
||||
# Unpack list of tuples to tuple of lists.
|
||||
handles = [d[0] for d in all_data] # type: ignore
|
||||
offsets = [d[1] for d in all_data] # type: ignore
|
||||
ops.register_graph_buffers(self._ptr, handles, offsets)
|
||||
|
||||
def should_custom_ar(self, inp: torch.Tensor):
|
||||
if self.disabled:
|
||||
return False
|
||||
inp_size = inp.numel() * inp.element_size()
|
||||
# custom allreduce requires input byte size to be multiples of 16
|
||||
if inp_size % 16 != 0:
|
||||
return False
|
||||
if not is_weak_contiguous(inp):
|
||||
return False
|
||||
# for 4 or more non NVLink-capable GPUs, custom allreduce provides
|
||||
# little performance improvement over NCCL.
|
||||
if not _is_hip:
|
||||
if True:
|
||||
return inp_size <= self.max_size
|
||||
return False
|
||||
|
||||
if _is_hip:
|
||||
if self.use_amd_deterministic_impl:
|
||||
return True
|
||||
if self.full_nvlink:
|
||||
return inp_size <= self.max_size
|
||||
return False
|
||||
|
||||
return False
|
||||
|
||||
def _all_reduce_impl(self, inp: torch.Tensor, registered: bool):
|
||||
out = torch.empty_like(inp)
|
||||
if not _is_hip: # CUDA-like
|
||||
if registered:
|
||||
ops.all_reduce(self._ptr, inp, out, 0, 0)
|
||||
else:
|
||||
ops.all_reduce(
|
||||
self._ptr, inp, out, self.buffer_ptrs[self.rank], self.max_size
|
||||
)
|
||||
elif self.use_amd_deterministic_impl:
|
||||
inp_size = inp.numel() * inp.element_size()
|
||||
if inp_size < self.max_size:
|
||||
reg_buffer = self.buffer.view(inp.dtype)[: inp.numel()]
|
||||
ops.deterministic_all_reduce_unreg(self._ptr, inp, reg_buffer, out)
|
||||
else:
|
||||
self.register_buffer(inp)
|
||||
ops.deterministic_all_reduce_reg(self._ptr, inp, out)
|
||||
else: # normal AMD ROCm path
|
||||
if registered:
|
||||
ops.all_reduce_reg(self._ptr, inp, out)
|
||||
else:
|
||||
ops.all_reduce_unreg(self._ptr, inp, self.buffer, out)
|
||||
return out
|
||||
|
||||
def custom_all_reduce(self, input: torch.Tensor) -> Optional[torch.Tensor]:
|
||||
"""The main allreduce API that provides support for cuda graph."""
|
||||
# When custom allreduce is disabled, this will be None.
|
||||
if self.disabled or not self.should_custom_ar(input):
|
||||
return None
|
||||
if self._IS_CAPTURING:
|
||||
if torch.cuda.is_current_stream_capturing():
|
||||
return self._all_reduce_impl(input, registered=not self.tms_cudagraph)
|
||||
else:
|
||||
# Could be warmup OR piecewise cuda graph split op execution.
|
||||
# In piecewise cuda graph, split ops run eagerly outside the graph
|
||||
# but _IS_CAPTURING is still True. We need to do real all-reduce.
|
||||
if is_in_tc_piecewise_cuda_graph():
|
||||
# Split op execution - do real all-reduce
|
||||
return self._all_reduce_impl(input, registered=False)
|
||||
else:
|
||||
# True warmup - mimic the allocation pattern since custom
|
||||
# allreduce is out-of-place.
|
||||
return torch.zeros_like(input)
|
||||
else:
|
||||
return self._all_reduce_impl(input, registered=False)
|
||||
|
||||
def close(self):
|
||||
if not self.disabled and self._ptr:
|
||||
if ops is not None:
|
||||
ops.dispose(self._ptr)
|
||||
if _is_cuda:
|
||||
self.free_shared_buffer(self.meta_ptrs)
|
||||
self.free_shared_buffer(self.buffer_ptrs)
|
||||
self._ptr = 0
|
||||
|
||||
def __del__(self):
|
||||
self.close()
|
||||
|
||||
|
||||
def dispatch_custom_allreduce(
|
||||
group: ProcessGroup,
|
||||
device: torch.device,
|
||||
):
|
||||
"""Return the CustomAllreduce class to use (aiter on ROCm if enabled).
|
||||
|
||||
On AMD with 1-stage AR enabled, use sglang's CustomAllreduce.
|
||||
Otherwise use AiterCustomAllreduce if available.
|
||||
|
||||
On CUDA, the JIT-compiled v2 implementation is used by default.
|
||||
Set SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=0 to fall back to the legacy CustomAllreduce.
|
||||
Multi-node v2 is admitted only for a single NVLink clique (see
|
||||
can_use_custom_all_reduce_v2); other cross-node groups fall back to NCCL.
|
||||
"""
|
||||
if _is_cuda and envs.SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2.get():
|
||||
from .custom_all_reduce_v2 import (
|
||||
CustomAllReduceV2,
|
||||
can_use_custom_all_reduce_v2,
|
||||
)
|
||||
|
||||
if can_use_custom_all_reduce_v2(group=group, device=device):
|
||||
logger.debug("[AR] Using CustomAllReduceV2 (JIT-compiled)")
|
||||
return CustomAllReduceV2
|
||||
|
||||
if _is_cuda or _is_musa:
|
||||
return CustomAllreduce
|
||||
|
||||
assert _is_hip
|
||||
|
||||
if envs.SGLANG_USE_1STAGE_ALLREDUCE.is_set():
|
||||
if envs.SGLANG_USE_1STAGE_ALLREDUCE.get():
|
||||
logger.debug(
|
||||
"[AR] All-reduce: 1-stage kernel (SGLANG_USE_1STAGE_ALLREDUCE=1)"
|
||||
)
|
||||
else:
|
||||
logger.debug("[AR] All-reduce: default (SGLANG_USE_1STAGE_ALLREDUCE=0)")
|
||||
elif envs.SGLANG_ENABLE_DETERMINISTIC_INFERENCE.get():
|
||||
logger.debug(
|
||||
"[AR] All-reduce: 1-stage kernel (deterministic inference enabled)"
|
||||
)
|
||||
else:
|
||||
logger.debug("[AR] All-reduce: default")
|
||||
|
||||
# On AMD with 1-stage AR, use sglang's CustomAllreduce
|
||||
# (AiterCustomAllreduce doesn't have deterministic_all_reduce method)
|
||||
if _use_amd_deterministic_impl():
|
||||
return CustomAllreduce
|
||||
|
||||
if get_bool_env_var("SGLANG_USE_AITER_AR", default="true"):
|
||||
try:
|
||||
from aiter.dist.device_communicators.custom_all_reduce import (
|
||||
CustomAllreduce as AiterCustomAllreduce,
|
||||
)
|
||||
|
||||
logger.info("[AR] Using AiterCustomAllreduce (AMD default)")
|
||||
tms_cudagraph = envs.SGLANG_MEMORY_SAVER_CUDA_GRAPH.get()
|
||||
return partial(
|
||||
AiterCustomAllreduce,
|
||||
enable_register_for_capturing=not tms_cudagraph,
|
||||
)
|
||||
except ImportError as e:
|
||||
logger.warning(
|
||||
"[AR] Aiter custom all-reduce not available; "
|
||||
"falling back to sglang CustomAllreduce. Details: %s",
|
||||
e,
|
||||
)
|
||||
return CustomAllreduce
|
||||
|
||||
return CustomAllreduce
|
||||
|
||||
|
||||
def _use_amd_deterministic_impl() -> bool:
|
||||
if not _is_hip: # CUDA is always deterministic
|
||||
return False
|
||||
if envs.SGLANG_USE_1STAGE_ALLREDUCE.is_set():
|
||||
return envs.SGLANG_USE_1STAGE_ALLREDUCE.get()
|
||||
else:
|
||||
return envs.SGLANG_ENABLE_DETERMINISTIC_INFERENCE.get()
|
||||
@ -1,519 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
# Adapted from https://github.com/vllm-project/vllm/blob/v0.6.4.post1/vllm/distributed/device_communicators/custom_all_reduce_utils.py
|
||||
|
||||
import ctypes
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pickle
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from functools import wraps
|
||||
from itertools import product
|
||||
from typing import Callable, Dict, List, Optional, Sequence, TypeVar
|
||||
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
import torch.multiprocessing as mp
|
||||
from typing_extensions import ParamSpec
|
||||
|
||||
from sglang.srt.distributed.device_communicators.cuda_wrapper import CudaRTLibrary
|
||||
from sglang.srt.distributed.parallel_state import in_the_same_node_as
|
||||
from sglang.srt.environ import envs as sglang_envs
|
||||
from sglang.srt.utils import is_cuda, is_hip, is_musa
|
||||
from sglang.srt.utils.cuda_vmm_utils import _gpu_fabric_clique
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_is_cuda = is_cuda()
|
||||
_is_hip = is_hip()
|
||||
_is_musa = is_musa()
|
||||
|
||||
if _is_cuda:
|
||||
try:
|
||||
import pynvml
|
||||
except ImportError as e:
|
||||
logger.warning("Failed to import pynvml with %r", e)
|
||||
|
||||
if _is_musa:
|
||||
try:
|
||||
import pymtml as pynvml
|
||||
except ImportError as e:
|
||||
logger.warning("Failed to import pymtml with %r", e)
|
||||
|
||||
if _is_hip:
|
||||
try:
|
||||
from amdsmi import (
|
||||
AmdSmiException,
|
||||
amdsmi_get_processor_handles,
|
||||
amdsmi_init,
|
||||
amdsmi_shut_down,
|
||||
amdsmi_topo_get_link_type,
|
||||
)
|
||||
except ImportError as e:
|
||||
logger.warning("Failed to import amdsmi with %r", e)
|
||||
|
||||
_P = ParamSpec("_P")
|
||||
_R = TypeVar("_R")
|
||||
|
||||
|
||||
def update_environment_variables(envs: Dict[str, str]):
|
||||
for k, v in envs.items():
|
||||
if k in os.environ and os.environ[k] != v:
|
||||
logger.warning(
|
||||
"Overwriting environment variable %s " "from '%s' to '%s'",
|
||||
k,
|
||||
os.environ[k],
|
||||
v,
|
||||
)
|
||||
os.environ[k] = v
|
||||
|
||||
|
||||
def producer(
|
||||
batch_src: Sequence[int],
|
||||
producer_queue,
|
||||
consumer_queue,
|
||||
result_queue,
|
||||
cuda_visible_devices: Optional[str] = None,
|
||||
):
|
||||
if cuda_visible_devices is not None:
|
||||
update_environment_variables({"CUDA_VISIBLE_DEVICES": cuda_visible_devices})
|
||||
|
||||
lib = CudaRTLibrary()
|
||||
for i in batch_src:
|
||||
lib.cudaSetDevice(i)
|
||||
pointer = lib.cudaMalloc(1024)
|
||||
lib.cudaMemset(pointer, 1, 1024)
|
||||
lib.cudaDeviceSynchronize()
|
||||
handle = lib.cudaIpcGetMemHandle(pointer)
|
||||
producer_queue.put(handle)
|
||||
open_success = consumer_queue.get()
|
||||
if open_success:
|
||||
# use two queues to simulate barrier
|
||||
producer_queue.put(0)
|
||||
consumer_queue.get()
|
||||
# check if the memory is modified
|
||||
host_data = (ctypes.c_char * 1024)()
|
||||
lib.cudaMemcpy(host_data, pointer, 1024) # type: ignore
|
||||
for i in range(1024):
|
||||
if ord(host_data[i]) != 2:
|
||||
open_success = False
|
||||
break
|
||||
result_queue.put(open_success)
|
||||
lib.cudaDeviceReset()
|
||||
|
||||
|
||||
def consumer(
|
||||
batch_tgt: Sequence[int],
|
||||
producer_queue,
|
||||
consumer_queue,
|
||||
result_queue,
|
||||
cuda_visible_devices: Optional[str] = None,
|
||||
):
|
||||
if cuda_visible_devices is not None:
|
||||
update_environment_variables({"CUDA_VISIBLE_DEVICES": cuda_visible_devices})
|
||||
|
||||
lib = CudaRTLibrary()
|
||||
for j in batch_tgt:
|
||||
lib.cudaSetDevice(j)
|
||||
handle = producer_queue.get()
|
||||
open_success = False
|
||||
try:
|
||||
pointer = lib.cudaIpcOpenMemHandle(handle) # type: ignore
|
||||
open_success = True
|
||||
except RuntimeError:
|
||||
# cannot error out here, because the producer process
|
||||
# is still waiting for the response.
|
||||
pass
|
||||
consumer_queue.put(open_success)
|
||||
if open_success:
|
||||
# modify the memory
|
||||
lib.cudaMemset(pointer, 2, 1024)
|
||||
lib.cudaDeviceSynchronize()
|
||||
# use two queues to simulate barrier
|
||||
producer_queue.get()
|
||||
consumer_queue.put(0)
|
||||
# check if the memory is modified
|
||||
host_data = (ctypes.c_char * 1024)()
|
||||
lib.cudaMemcpy(host_data, pointer, 1024) # type: ignore
|
||||
for i in range(1024):
|
||||
if ord(host_data[i]) != 2:
|
||||
open_success = False
|
||||
break
|
||||
result_queue.put(open_success)
|
||||
lib.cudaDeviceReset()
|
||||
|
||||
|
||||
def can_actually_p2p(
|
||||
batch_src: Sequence[int],
|
||||
batch_tgt: Sequence[int],
|
||||
) -> Sequence[bool]:
|
||||
"""
|
||||
Usually, checking if P2P access is enabled can be done by
|
||||
`torch.cuda.can_device_access_peer(src, tgt)`. However, sometimes
|
||||
the driver might be broken, and `torch.cuda.can_device_access_peer(src, tgt)`
|
||||
returns `True` even if P2P access is not actually possible.
|
||||
See https://github.com/vllm-project/vllm/issues/2728 and
|
||||
https://forums.developer.nvidia.com/t/direct-gpu-gpu-communication-does-not-seem-to-work-properly/283264/10
|
||||
Therefore, we have to perform a real P2P access to check if it is actually
|
||||
possible.
|
||||
|
||||
Note on p2p and cuda IPC:
|
||||
Usually, one process uses one GPU:
|
||||
GPU src --> cuda context src --> tensor src --> process src
|
||||
|
||||
We need to combine p2p and cuda IPC, so that:
|
||||
GPU src --> cuda context src --> tensor src --> process src
|
||||
|shared|
|
||||
GPU tgt --> cuda context tgt --> tensor tgt --> process tgt
|
||||
That is to say, process src creates a tensor in GPU src, passes IPC handle to
|
||||
process tgt, and process tgt accesses the tensor in GPU tgt. Any operation on the
|
||||
tensor in process tgt will be reflected in the tensor in process src, because
|
||||
they are the same memory segment.
|
||||
It is important to note that process tgt accesses the tensor in GPU tgt, not
|
||||
GPU src. That's why we need p2p access.
|
||||
|
||||
The most time-consuming part is the process creation. To avoid creating
|
||||
processes for every pair of GPUs, we use batched testing. We create two
|
||||
processes for testing all pairs of GPUs in batch. The trick is to reset
|
||||
the device after each test (which is not available in PyTorch).
|
||||
""" # noqa
|
||||
cuda_visible_devices = os.environ.get("CUDA_VISIBLE_DEVICES", None)
|
||||
# pass the CUDA_VISIBLE_DEVICES to the child process
|
||||
# to make sure they see the same set of GPUs
|
||||
|
||||
# make sure the processes are spawned
|
||||
smp = mp.get_context("spawn")
|
||||
producer_queue = smp.Queue()
|
||||
consumer_queue = smp.Queue()
|
||||
result_queue = smp.Queue()
|
||||
p_src = smp.Process(
|
||||
target=producer,
|
||||
args=(
|
||||
batch_src,
|
||||
producer_queue,
|
||||
consumer_queue,
|
||||
result_queue,
|
||||
cuda_visible_devices,
|
||||
),
|
||||
)
|
||||
p_tgt = smp.Process(
|
||||
target=consumer,
|
||||
args=(
|
||||
batch_tgt,
|
||||
producer_queue,
|
||||
consumer_queue,
|
||||
result_queue,
|
||||
cuda_visible_devices,
|
||||
),
|
||||
)
|
||||
p_src.start()
|
||||
p_tgt.start()
|
||||
p_src.join()
|
||||
p_tgt.join()
|
||||
assert p_src.exitcode == 0 and p_tgt.exitcode == 0
|
||||
result: List[bool] = []
|
||||
for src, tgt in zip(batch_src, batch_tgt):
|
||||
a = result_queue.get()
|
||||
b = result_queue.get()
|
||||
if a != b:
|
||||
logger.warning(
|
||||
"Two processes do not agree on the P2P access"
|
||||
" status on %d -> %d, treat as disabled.",
|
||||
src,
|
||||
tgt,
|
||||
)
|
||||
result.append(False)
|
||||
else:
|
||||
result.append(a)
|
||||
return result
|
||||
|
||||
|
||||
# why do we need this cache?
|
||||
# we are testing peer-to-peer (p2p) access between GPUs,across processes.
|
||||
# if we test it every time, it will be very slow, because we need to create
|
||||
# N * N * 2 processes, where N is the world size. This is very slow.
|
||||
# to reduce the time, we use a cache file to store the p2p access status.
|
||||
# the cache file is generated by the master process if it does not exist.
|
||||
# then all the processes can read the cache file to check the p2p access status.
|
||||
# Note that the cache file is suffixed by the CUDA_VISIBLE_DEVICES, so that we
|
||||
# can have different cache files for different CUDA_VISIBLE_DEVICES settings,
|
||||
# e.g. used by different vllm engines. The device id in the cache file is a
|
||||
# **local** device id, i.e. from 0 to num_dev-1, where num_dev is the number
|
||||
# of visible devices in the vllm engine.
|
||||
_gpu_p2p_access_cache: Optional[Dict[str, bool]] = None
|
||||
|
||||
|
||||
def gpu_p2p_access_check(src: int, tgt: int) -> bool:
|
||||
"""Check if GPU src can access GPU tgt."""
|
||||
|
||||
# if the cache variable is already calculated,
|
||||
# read from the cache instead of checking it again
|
||||
global _gpu_p2p_access_cache
|
||||
if _gpu_p2p_access_cache is not None:
|
||||
return _gpu_p2p_access_cache[f"{src}->{tgt}"]
|
||||
|
||||
is_distributed = dist.is_initialized()
|
||||
|
||||
num_dev = torch.cuda.device_count()
|
||||
cuda_visible_devices = os.environ.get("CUDA_VISIBLE_DEVICES", None)
|
||||
if cuda_visible_devices is None:
|
||||
cuda_visible_devices = ",".join(str(i) for i in range(num_dev))
|
||||
|
||||
# VLLM_CACHE_ROOT -> SGLANG_CACHE_ROOT
|
||||
# "~/.cache/vllm" -> envs.SGLANG_CACHE_DIR
|
||||
SGLANG_CACHE_ROOT = os.path.expanduser(sglang_envs.SGLANG_CACHE_DIR.get())
|
||||
path = os.path.join(
|
||||
SGLANG_CACHE_ROOT, f"gpu_p2p_access_cache_for_{cuda_visible_devices}.json"
|
||||
)
|
||||
cache_dir = os.path.dirname(path)
|
||||
try:
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
except (FileExistsError, NotADirectoryError):
|
||||
if not os.path.isdir(cache_dir):
|
||||
# Path exists as a file (stale cache/lock). Remove and retry.
|
||||
try:
|
||||
os.remove(cache_dir)
|
||||
except OSError:
|
||||
pass
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
from sglang.srt.distributed.parallel_state import get_world_group
|
||||
|
||||
if (not is_distributed or get_world_group().local_rank == 0) and (
|
||||
not os.path.exists(path)
|
||||
):
|
||||
# only the local master process (with local_rank == 0) can
|
||||
# enter this block to calculate the cache
|
||||
logger.info("generating GPU P2P access cache in %s", path)
|
||||
cache: Dict[str, bool] = {}
|
||||
ids = list(range(num_dev))
|
||||
# batch of all pairs of GPUs
|
||||
batch_src, batch_tgt = zip(*list(product(ids, ids)))
|
||||
# NOTE: we use `subprocess` rather than `multiprocessing` here
|
||||
# because the caller might not have `if __name__ == "__main__":`,
|
||||
# in that case we cannot use spawn method in multiprocessing.
|
||||
# However, `can_actually_p2p` requires spawn method.
|
||||
# The fix is, we use `subprocess` to call the function,
|
||||
# where we have `if __name__ == "__main__":` in this file.
|
||||
|
||||
# use a temporary file to store the result
|
||||
# we don't use the output of the subprocess directly,
|
||||
# because the subprocess might produce logging output
|
||||
with tempfile.NamedTemporaryFile() as output_file:
|
||||
input_bytes = pickle.dumps((batch_src, batch_tgt, output_file.name))
|
||||
returned = subprocess.run(
|
||||
[sys.executable, __file__], input=input_bytes, capture_output=True
|
||||
)
|
||||
# check if the subprocess is successful
|
||||
try:
|
||||
returned.check_returncode()
|
||||
except Exception as e:
|
||||
# wrap raised exception to provide more information
|
||||
raise RuntimeError(
|
||||
f"Error happened when batch testing "
|
||||
f"peer-to-peer access from {batch_src} to {batch_tgt}:\n"
|
||||
f"{returned.stderr.decode()}"
|
||||
) from e
|
||||
with open(output_file.name, "rb") as f:
|
||||
result = pickle.load(f)
|
||||
for _i, _j, r in zip(batch_src, batch_tgt, result):
|
||||
cache[f"{_i}->{_j}"] = r
|
||||
with open(path, "w") as f:
|
||||
json.dump(cache, f, indent=4)
|
||||
if is_distributed:
|
||||
get_world_group().barrier()
|
||||
logger.info("reading GPU P2P access cache from %s", path)
|
||||
with open(path) as f:
|
||||
cache = json.load(f)
|
||||
_gpu_p2p_access_cache = cache
|
||||
return _gpu_p2p_access_cache[f"{src}->{tgt}"]
|
||||
|
||||
|
||||
def with_nvml_context(fn: Callable[_P, _R]) -> Callable[_P, _R]:
|
||||
@wraps(fn)
|
||||
def wrapper(*args: _P.args, **kwargs: _P.kwargs) -> _R:
|
||||
if _is_hip:
|
||||
try:
|
||||
amdsmi_init()
|
||||
return fn(*args, **kwargs)
|
||||
finally:
|
||||
amdsmi_shut_down()
|
||||
else:
|
||||
pynvml.nvmlInit()
|
||||
try:
|
||||
return fn(*args, **kwargs)
|
||||
finally:
|
||||
pynvml.nvmlShutdown()
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
@with_nvml_context
|
||||
def is_full_nvlink(physical_device_ids: List[int], world_size: int) -> bool:
|
||||
if _is_hip:
|
||||
"""
|
||||
query if the set of gpus are fully connected by xgmi (1 hop)
|
||||
"""
|
||||
handles = [amdsmi_get_processor_handles()[i] for i in physical_device_ids]
|
||||
for i, handle in enumerate(handles):
|
||||
for j, peer_handle in enumerate(handles):
|
||||
if i < j:
|
||||
try:
|
||||
link_type = amdsmi_topo_get_link_type(handle, peer_handle)
|
||||
# type is 2 for XGMI
|
||||
if link_type["hops"] != 1 or link_type["type"] != 2:
|
||||
return False
|
||||
except AmdSmiException as error:
|
||||
logger.error("AMD 1 hop XGMI detection failed.", exc_info=error)
|
||||
return False
|
||||
return True
|
||||
else:
|
||||
"""
|
||||
query if the set of gpus are fully connected by nvlink (1 hop)
|
||||
"""
|
||||
handles = [pynvml.nvmlDeviceGetHandleByIndex(i) for i in physical_device_ids]
|
||||
for i, handle in enumerate(handles):
|
||||
for j, peer_handle in enumerate(handles):
|
||||
if i < j:
|
||||
try:
|
||||
p2p_status = pynvml.nvmlDeviceGetP2PStatus(
|
||||
handle, peer_handle, pynvml.NVML_P2P_CAPS_INDEX_NVLINK
|
||||
)
|
||||
if p2p_status != pynvml.NVML_P2P_STATUS_OK:
|
||||
return False
|
||||
except pynvml.NVMLError:
|
||||
logger.exception(
|
||||
"NVLink detection failed. This is normal if your"
|
||||
" machine has no NVLink equipped."
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
@with_nvml_context
|
||||
def is_one_nvlink_clique(
|
||||
group: torch.distributed.ProcessGroup, device: torch.device
|
||||
) -> bool:
|
||||
"""True iff every rank's GPU is in the same NVLink fabric clique (one NVL72 /
|
||||
MNNVL domain). Such a clique shares a single NVLink address space even across
|
||||
nodes, so custom-AR v2's symm-mem storage + fabric peer VAs are valid group-wide."""
|
||||
if _is_hip:
|
||||
return False
|
||||
try:
|
||||
clique = _gpu_fabric_clique(device)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"GPU fabric clique query failed (%r); custom-AR stays intra-node.", e
|
||||
)
|
||||
clique = None
|
||||
# Always all-gather (every rank calls it once) so a failed query on any rank
|
||||
# resolves to a clean False rather than a collective mismatch.
|
||||
world_size = dist.get_world_size(group=group)
|
||||
gathered: List[object] = [None] * world_size
|
||||
dist.all_gather_object(gathered, clique, group=group)
|
||||
if any(c is None for c in gathered):
|
||||
return False
|
||||
return len(set(gathered)) == 1
|
||||
|
||||
|
||||
def is_weak_contiguous(inp: torch.Tensor):
|
||||
return inp.is_contiguous() or (
|
||||
inp.storage().nbytes() - inp.storage_offset() * inp.element_size()
|
||||
== inp.numel() * inp.element_size()
|
||||
)
|
||||
|
||||
|
||||
def can_p2p(rank: int, world_size: int) -> bool:
|
||||
# SGLANG_SKIP_P2P_CHECK can be set to False in sglang
|
||||
SGLANG_SKIP_P2P_CHECK = os.getenv("SGLANG_SKIP_P2P_CHECK", "0") == "1"
|
||||
for i in range(world_size):
|
||||
if i == rank:
|
||||
continue
|
||||
if SGLANG_SKIP_P2P_CHECK:
|
||||
logger.info("Skipping P2P check and trusting the driver's P2P report.")
|
||||
return torch.cuda.can_device_access_peer(rank, i)
|
||||
if not gpu_p2p_access_check(rank, i):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def can_use_custom_all_reduce_with_nvlink(
|
||||
group: torch.distributed.ProcessGroup,
|
||||
device: torch.device,
|
||||
supported_world_size: List[int],
|
||||
cls_name: str,
|
||||
) -> Optional[bool]: # None if fail; otherwise return whether NVLink is available
|
||||
assert (
|
||||
dist.get_backend(group) != dist.Backend.NCCL
|
||||
), f"{cls_name} should be attached to a non-NCCL group."
|
||||
|
||||
rank = dist.get_rank(group=group)
|
||||
world_size = dist.get_world_size(group=group)
|
||||
|
||||
# No need to initialize custom allreduce for single GPU case.
|
||||
if world_size == 1:
|
||||
return
|
||||
|
||||
# No need to initialize custom allreduce for multi-node case.
|
||||
if not all(in_the_same_node_as(group, source_rank=0)):
|
||||
logger.warning(
|
||||
f"{cls_name} is disabled because this process group" " spans across nodes."
|
||||
)
|
||||
return
|
||||
|
||||
# For not supported world size, we disable custom allreduce.
|
||||
if world_size not in supported_world_size:
|
||||
logger.warning(
|
||||
f"{cls_name} is disabled due to an unsupported world"
|
||||
f" size: {world_size}. Supported world sizes: {supported_world_size}. "
|
||||
"To silence this warning, specify disable_custom_all_reduce=True explicitly.",
|
||||
)
|
||||
return
|
||||
|
||||
cuda_visible_devices = os.environ.get("CUDA_VISIBLE_DEVICES", None)
|
||||
if cuda_visible_devices:
|
||||
device_ids = list(map(int, cuda_visible_devices.split(",")))
|
||||
else:
|
||||
device_ids = list(range(torch.cuda.device_count()))
|
||||
physical_device_id = device_ids[device.index]
|
||||
tensor = torch.tensor([physical_device_id], dtype=torch.int, device="cpu")
|
||||
gather_list = [
|
||||
torch.tensor([0], dtype=torch.int, device="cpu") for _ in range(world_size)
|
||||
]
|
||||
dist.all_gather(gather_list, tensor, group=group)
|
||||
physical_device_ids = [int(t) for t in gather_list]
|
||||
full_nvlink = is_full_nvlink(physical_device_ids, world_size)
|
||||
|
||||
# test nvlink first, this will filter out most of the cases
|
||||
# where custom allreduce is not supported
|
||||
# this checks hardware and driver support for NVLink
|
||||
if False:
|
||||
logger.warning(
|
||||
f"{cls_name} is disabled because it's not supported on"
|
||||
" more than two PCIe-only GPUs. To silence this warning, "
|
||||
"specify disable_custom_all_reduce=True explicitly."
|
||||
)
|
||||
return
|
||||
|
||||
# test P2P capability, this checks software/cudaruntime support
|
||||
# this is expensive to compute at the first time
|
||||
# then we cache the result
|
||||
# On AMD GPU, p2p is always enabled between XGMI connected GPUs
|
||||
if not _is_hip and not can_p2p(rank, world_size):
|
||||
logger.warning(
|
||||
f"{cls_name} is disabled because your platform lacks "
|
||||
"GPU P2P capability or P2P test failed. To silence this "
|
||||
"warning, specify disable_custom_all_reduce=True explicitly."
|
||||
)
|
||||
return
|
||||
|
||||
return full_nvlink
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
batch_src, batch_tgt, output_file = pickle.loads(sys.stdin.buffer.read())
|
||||
result = can_actually_p2p(batch_src, batch_tgt)
|
||||
with open(output_file, "wb") as f:
|
||||
f.write(pickle.dumps(result))
|
||||
@ -1,20 +0,0 @@
|
||||
{"tag": "b3_16k_c1", "arm": "e7b", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9501, "corpus_window": {"start": 2300000, "end": 2431072}, "ok": 8, "failed": 0, "wall_s": 82.58, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 49.6, "input_throughput_tok_s": 1587.15, "ttft_s": {"mean": 3.9252, "p50": 3.9266, "p95": 3.9787, "max": 3.9787, "min": 3.8899}, "tpot_s": {"mean": 0.0125, "p50": 0.0128, "p95": 0.0137, "max": 0.0137, "min": 0.0112}, "e2e_s": {"mean": 10.3227, "p50": 10.4215, "p95": 10.9871, "max": 10.9871, "min": 9.6445}, "per_req_out_tok_s_e2e": {"mean": 49.6617, "p50": 49.8845, "p95": 53.0871, "max": 53.0871, "min": 46.6001}, "per_req_decode_tok_s": {"mean": 80.2896, "p50": 80.4694, "p95": 89.5483, "max": 89.5483, "min": 72.9492}, "spec_accept_length_mean": 2.138, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 16, "new_tokens": 131072, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c8", "arm": "e7b", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9502, "corpus_window": {"start": 2300000, "end": 2562144}, "ok": 16, "failed": 0, "wall_s": 100.63, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 8192, "output_throughput_tok_s": 81.41, "input_throughput_tok_s": 2605.13, "ttft_s": {"mean": 11.7518, "p50": 6.5296, "p95": 32.5233, "max": 32.5233, "min": 3.9869}, "tpot_s": {"mean": 0.0716, "p50": 0.0731, "p95": 0.1351, "max": 0.1351, "min": 0.0266}, "e2e_s": {"mean": 48.3305, "p50": 50.9027, "p95": 78.2404, "max": 78.2404, "min": 17.5929}, "per_req_out_tok_s_e2e": {"mean": 12.9171, "p50": 10.4857, "p95": 29.1027, "max": 29.1027, "min": 6.5439}, "per_req_decode_tok_s": {"mean": 16.8775, "p50": 15.7396, "p95": 37.6553, "max": 37.6553, "min": 7.4162}, "spec_accept_length_mean": 2.375, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 32, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c16", "arm": "e7b", "summary": {"concurrency": 16, "num_requests": 32, "run_id": 9503, "corpus_window": {"start": 2300000, "end": 2824288}, "ok": 32, "failed": 0, "wall_s": 236.44, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 16384, "output_throughput_tok_s": 69.29, "input_throughput_tok_s": 2217.4, "ttft_s": {"mean": 23.5964, "p50": 13.173, "p95": 60.864, "max": 90.3879, "min": 4.2969}, "tpot_s": {"mean": 0.1733, "p50": 0.1784, "p95": 0.2978, "max": 0.3055, "min": 0.0668}, "e2e_s": {"mean": 112.1663, "p50": 105.5499, "p95": 170.4271, "max": 183.3467, "min": 52.7259}, "per_req_out_tok_s_e2e": {"mean": 5.0312, "p50": 4.8797, "p95": 8.0542, "max": 9.7106, "min": 2.7925}, "per_req_decode_tok_s": {"mean": 6.7269, "p50": 5.8271, "p95": 12.9785, "max": 15.0103, "min": 3.2802}, "spec_accept_length_mean": 2.517, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 64, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c32", "arm": "e7b", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9504, "corpus_window": {"start": 2300000, "end": 3348576}, "ok": 64, "failed": 0, "wall_s": 414.96, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 32768, "output_throughput_tok_s": 78.97, "input_throughput_tok_s": 2526.91, "ttft_s": {"mean": 100.2552, "p50": 109.5841, "p95": 169.6812, "max": 190.7286, "min": 5.2261}, "tpot_s": {"mean": 0.1698, "p50": 0.1728, "p95": 0.2599, "max": 0.3191, "min": 0.0328}, "e2e_s": {"mean": 187.0411, "p50": 190.6599, "p95": 271.5134, "max": 292.7775, "min": 86.5615}, "per_req_out_tok_s_e2e": {"mean": 2.9412, "p50": 2.6906, "p95": 4.5951, "max": 5.9149, "min": 1.7488}, "per_req_decode_tok_s": {"mean": 7.1251, "p50": 5.846, "p95": 15.414, "max": 30.5614, "min": 3.1404}, "spec_accept_length_mean": 2.767, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 1048576, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c64", "arm": "e7b", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9505, "corpus_window": {"start": 2300000, "end": 4397152}, "ok": 128, "failed": 0, "wall_s": 777.55, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 65536, "output_throughput_tok_s": 84.29, "input_throughput_tok_s": 2697.14, "ttft_s": {"mean": 240.0965, "p50": 287.3833, "p95": 345.6933, "max": 380.9581, "min": 5.2288}, "tpot_s": {"mean": 0.168, "p50": 0.1702, "p95": 0.2315, "max": 0.3119, "min": 0.0404}, "e2e_s": {"mean": 325.934, "p50": 364.8118, "p95": 432.439, "max": 474.7226, "min": 83.7008}, "per_req_out_tok_s_e2e": {"mean": 1.8253, "p50": 1.4064, "p95": 4.196, "max": 6.117, "min": 1.0785}, "per_req_decode_tok_s": {"mean": 6.6174, "p50": 5.8876, "p95": 12.5326, "max": 24.7977, "min": 3.2126}, "spec_accept_length_mean": 2.986, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 2097152, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c1", "arm": "e7b", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9511, "corpus_window": {"start": 4500000, "end": 4508192}, "ok": 8, "failed": 0, "wall_s": 15.82, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 1024, "output_throughput_tok_s": 64.74, "input_throughput_tok_s": 517.93, "ttft_s": {"mean": 0.3034, "p50": 0.3015, "p95": 0.3258, "max": 0.3258, "min": 0.2959}, "tpot_s": {"mean": 0.0132, "p50": 0.0135, "p95": 0.0143, "max": 0.0143, "min": 0.0119}, "e2e_s": {"mean": 1.9769, "p50": 2.0115, "p95": 2.1177, "max": 2.1177, "min": 1.8091}, "per_req_out_tok_s_e2e": {"mean": 64.9178, "p50": 66.8893, "p95": 70.753, "max": 70.753, "min": 60.442}, "per_req_decode_tok_s": {"mean": 76.7983, "p50": 79.4095, "p95": 84.9204, "max": 84.9204, "min": 70.4319}, "spec_accept_length_mean": 2.028, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 8, "new_tokens": 8192, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c8", "arm": "e7b", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9512, "corpus_window": {"start": 4500000, "end": 4516384}, "ok": 16, "failed": 0, "wall_s": 13.44, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 2048, "output_throughput_tok_s": 152.4, "input_throughput_tok_s": 1219.24, "ttft_s": {"mean": 1.21, "p50": 1.7505, "p95": 2.1601, "max": 2.1601, "min": 0.387}, "tpot_s": {"mean": 0.0414, "p50": 0.042, "p95": 0.0537, "max": 0.0537, "min": 0.0319}, "e2e_s": {"mean": 6.4682, "p50": 6.5412, "p95": 8.9777, "max": 8.9777, "min": 4.4513}, "per_req_out_tok_s_e2e": {"mean": 20.4798, "p50": 20.2955, "p95": 28.7556, "max": 28.7556, "min": 14.2576}, "per_req_decode_tok_s": {"mean": 24.8432, "p50": 24.8024, "p95": 31.5592, "max": 31.5592, "min": 18.7758}, "spec_accept_length_mean": 2.048, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 10, "new_tokens": 16384, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c32", "arm": "e7b", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9513, "corpus_window": {"start": 4500000, "end": 4565536}, "ok": 64, "failed": 0, "wall_s": 81.8, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 8192, "output_throughput_tok_s": 100.14, "input_throughput_tok_s": 801.15, "ttft_s": {"mean": 17.5622, "p50": 21.7567, "p95": 25.6003, "max": 27.8932, "min": 1.5226}, "tpot_s": {"mean": 0.1466, "p50": 0.1501, "p95": 0.1905, "max": 0.1952, "min": 0.088}, "e2e_s": {"mean": 36.1812, "p50": 40.1766, "p95": 46.8967, "max": 47.1947, "min": 18.2249}, "per_req_out_tok_s_e2e": {"mean": 3.8358, "p50": 3.1917, "p95": 6.6145, "max": 7.0234, "min": 2.7122}, "per_req_decode_tok_s": {"mean": 7.1041, "p50": 6.7208, "p95": 10.1791, "max": 11.4499, "min": 5.164}, "spec_accept_length_mean": 2.054, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 38, "new_tokens": 65536, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c64", "arm": "e7b", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9514, "corpus_window": {"start": 4500000, "end": 4631072}, "ok": 128, "failed": 0, "wall_s": 158.73, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 16384, "output_throughput_tok_s": 103.22, "input_throughput_tok_s": 825.75, "ttft_s": {"mean": 46.7369, "p50": 57.65, "p95": 65.8132, "max": 70.7061, "min": 1.5163}, "tpot_s": {"mean": 0.1445, "p50": 0.144, "p95": 0.1777, "max": 0.2125, "min": 0.0759}, "e2e_s": {"mean": 65.0893, "p50": 74.2949, "p95": 84.7332, "max": 89.7485, "min": 18.3747}, "per_req_out_tok_s_e2e": {"mean": 2.4029, "p50": 1.7238, "p95": 6.272, "max": 6.9661, "min": 1.4262}, "per_req_decode_tok_s": {"mean": 7.1513, "p50": 7.0024, "p95": 9.0245, "max": 13.2811, "min": 4.7422}, "spec_accept_length_mean": 2.087, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 85, "new_tokens": 131072, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c1", "arm": "e7b", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9521, "corpus_window": {"start": 4700000, "end": 4708192}, "ok": 8, "failed": 0, "wall_s": 242.18, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 32768, "output_throughput_tok_s": 135.31, "input_throughput_tok_s": 33.83, "ttft_s": {"mean": 0.3142, "p50": 0.3133, "p95": 0.3204, "max": 0.3204, "min": 0.3097}, "tpot_s": {"mean": 0.0073, "p50": 0.0078, "p95": 0.0087, "max": 0.0087, "min": 0.0059}, "e2e_s": {"mean": 30.2716, "p50": 32.1753, "p95": 35.9141, "max": 35.9141, "min": 24.509}, "per_req_out_tok_s_e2e": {"mean": 137.6058, "p50": 136.5088, "p95": 167.1221, "max": 167.1221, "min": 114.0498}, "per_req_decode_tok_s": {"mean": 139.1062, "p50": 137.9513, "p95": 169.3421, "max": 169.3421, "min": 115.0748}, "spec_accept_length_mean": 3.701, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 8, "new_tokens": 8192, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c8", "arm": "e7b", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9522, "corpus_window": {"start": 4700000, "end": 4716384}, "ok": 16, "failed": 0, "wall_s": 159.24, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 65536, "output_throughput_tok_s": 411.54, "input_throughput_tok_s": 102.89, "ttft_s": {"mean": 0.9638, "p50": 1.2709, "p95": 1.6805, "max": 1.6805, "min": 0.378}, "tpot_s": {"mean": 0.0181, "p50": 0.0175, "p95": 0.0225, "max": 0.0225, "min": 0.0153}, "e2e_s": {"mean": 75.1101, "p50": 73.2342, "p95": 93.9711, "max": 93.9711, "min": 64.3187}, "per_req_out_tok_s_e2e": {"mean": 55.3003, "p50": 58.2448, "p95": 63.6829, "max": 63.6829, "min": 43.5879}, "per_req_decode_tok_s": {"mean": 55.9977, "p50": 58.5719, "p95": 65.3811, "max": 65.3811, "min": 44.3769}, "spec_accept_length_mean": 4.003, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 10, "new_tokens": 16384, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c32", "arm": "e7b", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9523, "corpus_window": {"start": 4700000, "end": 4765536}, "ok": 64, "failed": 0, "wall_s": 1040.79, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 262144, "output_throughput_tok_s": 251.87, "input_throughput_tok_s": 62.97, "ttft_s": {"mean": 195.07, "p50": 240.7323, "p95": 304.7528, "max": 334.6933, "min": 1.3472}, "tpot_s": {"mean": 0.0607, "p50": 0.0581, "p95": 0.0746, "max": 0.0923, "min": 0.0465}, "e2e_s": {"mean": 443.6795, "p50": 478.2794, "p95": 587.5712, "max": 597.8883, "min": 219.3532}, "per_req_out_tok_s_e2e": {"mean": 10.0455, "p50": 8.6106, "p95": 17.4441, "max": 18.6731, "min": 6.8508}, "per_req_decode_tok_s": {"mean": 16.784, "p50": 17.4276, "p95": 20.0358, "max": 21.5315, "min": 10.8381}, "spec_accept_length_mean": 3.852, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 51, "new_tokens": 65536, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_64k_c1", "arm": "e7b", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9531, "corpus_window": {"start": 5000000, "end": 5524288}, "ok": 8, "failed": 0, "wall_s": 182.23, "input_len": 65536, "shared_len": 0, "unique_len": 65536, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 22.48, "input_throughput_tok_s": 2877.06, "ttft_s": {"mean": 17.0384, "p50": 17.022, "p95": 17.1703, "max": 17.1703, "min": 17.0028}, "tpot_s": {"mean": 0.0112, "p50": 0.0117, "p95": 0.0135, "max": 0.0135, "min": 0.008}, "e2e_s": {"mean": 22.7786, "p50": 23.0017, "p95": 23.9301, "max": 23.9301, "min": 21.118}, "per_req_out_tok_s_e2e": {"mean": 22.5108, "p50": 22.5486, "p95": 24.2447, "max": 24.2447, "min": 21.3957}, "per_req_decode_tok_s": {"mean": 91.4983, "p50": 90.0463, "p95": 125.007, "max": 125.007, "min": 74.0091}, "spec_accept_length_mean": 2.468, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 64, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_64k_c4", "arm": "e7b", "summary": {"concurrency": 4, "num_requests": 8, "run_id": 9532, "corpus_window": {"start": 5000000, "end": 5524288}, "ok": 8, "failed": 0, "wall_s": 159.24, "input_len": 65536, "shared_len": 0, "unique_len": 65536, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 25.72, "input_throughput_tok_s": 3292.45, "ttft_s": {"mean": 30.5492, "p50": 18.5951, "p95": 68.8675, "max": 68.8675, "min": 17.0513}, "tpot_s": {"mean": 0.0949, "p50": 0.1145, "p95": 0.1546, "max": 0.1546, "min": 0.02}, "e2e_s": {"mean": 79.0616, "p50": 80.1169, "p95": 131.9282, "max": 131.9282, "min": 27.2645}, "per_req_out_tok_s_e2e": {"mean": 8.1791, "p50": 6.6417, "p95": 18.779, "max": 18.779, "min": 3.8809}, "per_req_decode_tok_s": {"mean": 16.1959, "p50": 11.2172, "p95": 50.1581, "max": 50.1581, "min": 6.4814}, "spec_accept_length_mean": 2.438, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 64, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_64k_c8", "arm": "e7b", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9533, "corpus_window": {"start": 5000000, "end": 6048576}, "ok": 16, "failed": 0, "wall_s": 319.66, "input_len": 65536, "shared_len": 0, "unique_len": 65536, "output_len": 512, "output_tokens_total": 8192, "output_throughput_tok_s": 25.63, "input_throughput_tok_s": 3280.31, "ttft_s": {"mean": 90.9186, "p50": 96.5085, "p95": 149.4465, "max": 149.4465, "min": 18.5948}, "tpot_s": {"mean": 0.1058, "p50": 0.1221, "p95": 0.1575, "max": 0.1575, "min": 0.0172}, "e2e_s": {"mean": 144.9847, "p50": 155.6435, "p95": 211.845, "max": 211.845, "min": 75.7166}, "per_req_out_tok_s_e2e": {"mean": 3.7942, "p50": 3.6883, "p95": 6.7621, "max": 6.7621, "min": 2.4169}, "per_req_decode_tok_s": {"mean": 13.0455, "p50": 8.4248, "p95": 58.0944, "max": 58.0944, "min": 6.3604}, "spec_accept_length_mean": 2.442, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 1048576, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_128k_c1", "arm": "e7b", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9541, "corpus_window": {"start": 6200000, "end": 7248576}, "ok": 8, "failed": 0, "wall_s": 343.36, "input_len": 131072, "shared_len": 0, "unique_len": 131072, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 11.93, "input_throughput_tok_s": 3053.83, "ttft_s": {"mean": 37.5917, "p50": 37.5968, "p95": 37.6012, "max": 37.6012, "min": 37.5564}, "tpot_s": {"mean": 0.0104, "p50": 0.0104, "p95": 0.0123, "max": 0.0123, "min": 0.009}, "e2e_s": {"mean": 42.9201, "p50": 42.9071, "p95": 43.9018, "max": 43.9018, "min": 42.1336}, "per_req_out_tok_s_e2e": {"mean": 11.9312, "p50": 11.9694, "p95": 12.1518, "max": 12.1518, "min": 11.6624}, "per_req_decode_tok_s": {"mean": 97.1304, "p50": 98.8664, "p95": 111.8659, "max": 111.8659, "min": 81.2653}, "spec_accept_length_mean": 2.728, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 1048576, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_128k_c2", "arm": "e7b", "summary": {"concurrency": 2, "num_requests": 8, "run_id": 9542, "corpus_window": {"start": 6200000, "end": 7248576}, "ok": 8, "failed": 0, "wall_s": 333.36, "input_len": 131072, "shared_len": 0, "unique_len": 131072, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 12.29, "input_throughput_tok_s": 3145.44, "ttft_s": {"mean": 48.2081, "p50": 39.1403, "p95": 75.8082, "max": 75.8082, "min": 37.5834}, "tpot_s": {"mean": 0.068, "p50": 0.0858, "p95": 0.1622, "max": 0.1622, "min": 0.0099}, "e2e_s": {"mean": 82.9494, "p50": 84.1258, "p95": 122.0311, "max": 122.0311, "min": 42.6804}, "per_req_out_tok_s_e2e": {"mean": 7.0763, "p50": 6.2562, "p95": 11.9961, "max": 11.9961, "min": 4.1957}, "per_req_decode_tok_s": {"mean": 37.0402, "p50": 12.1918, "p95": 100.963, "max": 100.963, "min": 6.1768}, "spec_accept_length_mean": 2.769, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 1048576, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_128k_c4", "arm": "e7b", "summary": {"concurrency": 4, "num_requests": 8, "run_id": 9543, "corpus_window": {"start": 6200000, "end": 7248576}, "ok": 8, "failed": 0, "wall_s": 334.71, "input_len": 131072, "shared_len": 0, "unique_len": 131072, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 12.24, "input_throughput_tok_s": 3132.76, "ttft_s": {"mean": 110.1385, "p50": 120.8044, "p95": 159.4963, "max": 159.4963, "min": 39.1664}, "tpot_s": {"mean": 0.0687, "p50": 0.0152, "p95": 0.1634, "max": 0.1634, "min": 0.0097}, "e2e_s": {"mean": 145.2581, "p50": 130.4964, "p95": 204.1758, "max": 204.1758, "min": 83.563}, "per_req_out_tok_s_e2e": {"mean": 3.8065, "p50": 4.04, "p95": 6.1271, "max": 6.1271, "min": 2.5076}, "per_req_decode_tok_s": {"mean": 54.2853, "p50": 73.1655, "p95": 103.6085, "max": 103.6085, "min": 6.1313}, "spec_accept_length_mean": 2.595, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 1048576, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b52_256k_c1", "arm": "e7b", "summary": {"concurrency": 1, "num_requests": 1, "run_id": 9551, "corpus_window": {"start": 8400000, "end": 8662144}, "ok": 1, "failed": 0, "wall_s": 0.5, "input_len": 262144, "shared_len": 0, "unique_len": 262144, "output_len": 1, "output_tokens_total": 1, "output_throughput_tok_s": 2.01, "input_throughput_tok_s": 527883.82, "ttft_s": {"mean": 0.4955, "p50": 0.4955, "p95": 0.4955, "max": 0.4955, "min": 0.4955}, "tpot_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "e2e_s": {"mean": 0.4957, "p50": 0.4957, "p95": 0.4957, "max": 0.4957, "min": 0.4957}, "per_req_out_tok_s_e2e": {"mean": 2.0172, "p50": 2.0172, "p95": 2.0172, "max": 2.0172, "min": 2.0172}, "per_req_decode_tok_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 1, "new_tokens": 64, "cached_tokens": 262080, "hit_rate": 0.9998}}}
|
||||
{"tag": "b52_256k_c1", "arm": "e7b", "summary": {"concurrency": 1, "num_requests": 1, "run_id": 9551, "corpus_window": {"start": 9900000, "end": 10162144}, "ok": 1, "failed": 0, "wall_s": 91.45, "input_len": 262144, "shared_len": 0, "unique_len": 262144, "output_len": 1, "output_tokens_total": 1, "output_throughput_tok_s": 0.01, "input_throughput_tok_s": 2866.59, "ttft_s": {"mean": 91.4465, "p50": 91.4465, "p95": 91.4465, "max": 91.4465, "min": 91.4465}, "tpot_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "e2e_s": {"mean": 91.4468, "p50": 91.4468, "p95": 91.4468, "max": 91.4468, "min": 91.4468}, "per_req_out_tok_s_e2e": {"mean": 0.0109, "p50": 0.0109, "p95": 0.0109, "max": 0.0109, "min": 0.0109}, "per_req_decode_tok_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 32, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
@ -1,3 +0,0 @@
|
||||
a8f3909211adb92501cf007ab71d3b29 /root/corpus_ids.json
|
||||
1e34dd8d813ccb7ac8243e8d1573a49e /root/bench_corpus_v2.py
|
||||
4c126d067d33b5ea27c268f37561634c /root/extract_summary.py
|
||||
File diff suppressed because one or more lines are too long
@ -1,20 +0,0 @@
|
||||
b3_16k_c1 OK
|
||||
b3_16k_c8 OK
|
||||
b3_16k_c16 OK
|
||||
b3_16k_c32 OK
|
||||
b3_16k_c64 OK
|
||||
b41_1k_c1 OK
|
||||
b41_1k_c8 OK
|
||||
b41_1k_c32 OK
|
||||
b41_1k_c64 OK
|
||||
b42_1k4k_c1 OK
|
||||
b42_1k4k_c8 OK
|
||||
b42_1k4k_c32 OK
|
||||
b42_1k4k_c64 BENCH_FAIL rc=143
|
||||
b51_64k_c1 OK
|
||||
b51_64k_c4 OK
|
||||
b51_64k_c8 OK
|
||||
b51_128k_c1 OK
|
||||
b51_128k_c2 OK
|
||||
b51_128k_c4 OK
|
||||
b52_256k_c1 HIT_FAIL_FINAL
|
||||
@ -1,10 +0,0 @@
|
||||
{"tag": "b3_16k_c8", "arm": "e7b64", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9602, "corpus_window": {"start": 2300000, "end": 2562144}, "ok": 16, "failed": 0, "wall_s": 97.66, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 8192, "output_throughput_tok_s": 83.88, "input_throughput_tok_s": 2684.26, "ttft_s": {"mean": 12.1187, "p50": 8.3735, "p95": 32.4651, "max": 32.4651, "min": 3.9862}, "tpot_s": {"mean": 0.0692, "p50": 0.0688, "p95": 0.1361, "max": 0.1361, "min": 0.0264}, "e2e_s": {"mean": 47.483, "p50": 50.5394, "p95": 78.7413, "max": 78.7413, "min": 18.1949}, "per_req_out_tok_s_e2e": {"mean": 13.2102, "p50": 11.2272, "p95": 28.1397, "max": 28.1397, "min": 6.5023}, "per_req_decode_tok_s": {"mean": 18.5895, "p50": 14.9416, "p95": 37.9333, "max": 37.9333, "min": 7.3633}, "spec_accept_length_mean": 2.556, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 32, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c16", "arm": "e7b64", "summary": {"concurrency": 16, "num_requests": 32, "run_id": 9603, "corpus_window": {"start": 2300000, "end": 2824288}, "ok": 32, "failed": 0, "wall_s": 177.89, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 16384, "output_throughput_tok_s": 92.1, "input_throughput_tok_s": 2947.27, "ttft_s": {"mean": 19.7632, "p50": 7.8805, "p95": 59.91, "max": 63.6881, "min": 4.062}, "tpot_s": {"mean": 0.1297, "p50": 0.134, "p95": 0.2288, "max": 0.2759, "min": 0.0334}, "e2e_s": {"mean": 86.0176, "p50": 83.7193, "p95": 145.8411, "max": 150.1856, "min": 22.7897}, "per_req_out_tok_s_e2e": {"mean": 7.4324, "p50": 6.5209, "p95": 16.0562, "max": 22.4663, "min": 3.4091}, "per_req_decode_tok_s": {"mean": 10.0542, "p50": 7.5804, "p95": 27.3398, "max": 29.9645, "min": 3.6318}, "spec_accept_length_mean": 2.485, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 64, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c32", "arm": "e7b64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9604, "corpus_window": {"start": 2300000, "end": 3348576}, "ok": 64, "failed": 0, "wall_s": 335.79, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 32768, "output_throughput_tok_s": 97.58, "input_throughput_tok_s": 3122.67, "ttft_s": {"mean": 82.6459, "p50": 87.8342, "p95": 140.3273, "max": 154.661, "min": 5.2339}, "tpot_s": {"mean": 0.1381, "p50": 0.1443, "p95": 0.2136, "max": 0.2775, "min": 0.026}, "e2e_s": {"mean": 153.2127, "p50": 153.6215, "p95": 227.6964, "max": 243.4494, "min": 78.5488}, "per_req_out_tok_s_e2e": {"mean": 3.581, "p50": 3.3906, "p95": 5.2619, "max": 6.5182, "min": 2.1031}, "per_req_decode_tok_s": {"mean": 9.3726, "p50": 7.1484, "p95": 24.3778, "max": 38.5186, "min": 3.6109}, "spec_accept_length_mean": 2.777, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 1048576, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c64", "arm": "e7b64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9605, "corpus_window": {"start": 2300000, "end": 4397152}, "ok": 128, "failed": 0, "wall_s": 657.79, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 65536, "output_throughput_tok_s": 99.63, "input_throughput_tok_s": 3188.2, "ttft_s": {"mean": 204.2735, "p50": 244.6246, "p95": 293.2111, "max": 322.84, "min": 5.231}, "tpot_s": {"mean": 0.1425, "p50": 0.1465, "p95": 0.2146, "max": 0.2766, "min": 0.0297}, "e2e_s": {"mean": 277.1081, "p50": 309.1898, "p95": 368.9412, "max": 410.789, "min": 77.0243}, "per_req_out_tok_s_e2e": {"mean": 2.133, "p50": 1.6583, "p95": 4.7327, "max": 6.6473, "min": 1.2464}, "per_req_decode_tok_s": {"mean": 8.0518, "p50": 6.8676, "p95": 16.3762, "max": 33.7377, "min": 3.6219}, "spec_accept_length_mean": 2.954, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 2097152, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c8", "arm": "e7b64", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9612, "corpus_window": {"start": 4500000, "end": 4516384}, "ok": 16, "failed": 0, "wall_s": 13.53, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 2048, "output_throughput_tok_s": 151.33, "input_throughput_tok_s": 1210.61, "ttft_s": {"mean": 1.2544, "p50": 1.7591, "p95": 2.1654, "max": 2.1654, "min": 0.3904}, "tpot_s": {"mean": 0.0421, "p50": 0.041, "p95": 0.0537, "max": 0.0537, "min": 0.0336}, "e2e_s": {"mean": 6.6071, "p50": 6.6204, "p95": 8.6618, "max": 8.6618, "min": 4.8636}, "per_req_out_tok_s_e2e": {"mean": 19.981, "p50": 19.5317, "p95": 26.3178, "max": 26.3178, "min": 14.7775}, "per_req_decode_tok_s": {"mean": 24.481, "p50": 24.6734, "p95": 29.9848, "max": 29.9848, "min": 18.7591}, "spec_accept_length_mean": 1.992, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 9, "new_tokens": 16384, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c32", "arm": "e7b64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9613, "corpus_window": {"start": 4500000, "end": 4565536}, "ok": 64, "failed": 0, "wall_s": 35.75, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 8192, "output_throughput_tok_s": 229.18, "input_throughput_tok_s": 1833.42, "ttft_s": {"mean": 2.9919, "p50": 2.2946, "p95": 7.0943, "max": 7.1032, "min": 0.5556}, "tpot_s": {"mean": 0.1119, "p50": 0.1092, "p95": 0.1506, "max": 0.1798, "min": 0.0728}, "e2e_s": {"mean": 17.1982, "p50": 16.6865, "p95": 23.6063, "max": 25.9273, "min": 9.7993}, "per_req_out_tok_s_e2e": {"mean": 7.8468, "p50": 7.6792, "p95": 11.3781, "max": 13.0622, "min": 4.9369}, "per_req_decode_tok_s": {"mean": 9.4487, "p50": 9.2601, "p95": 12.987, "max": 13.852, "min": 5.6066}, "spec_accept_length_mean": 2.035, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 20, "new_tokens": 65536, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c64", "arm": "e7b64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9614, "corpus_window": {"start": 4500000, "end": 4631072}, "ok": 128, "failed": 0, "wall_s": 60.18, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 16384, "output_throughput_tok_s": 272.24, "input_throughput_tok_s": 2177.88, "ttft_s": {"mean": 4.7575, "p50": 2.3032, "p95": 12.1513, "max": 13.4791, "min": 0.6534}, "tpot_s": {"mean": 0.1911, "p50": 0.1882, "p95": 0.2927, "max": 0.3315, "min": 0.1063}, "e2e_s": {"mean": 29.0341, "p50": 28.6144, "p95": 42.031, "max": 44.989, "min": 14.7114}, "per_req_out_tok_s_e2e": {"mean": 4.7331, "p50": 4.4822, "p95": 7.5631, "max": 8.7007, "min": 2.8451}, "per_req_decode_tok_s": {"mean": 5.6253, "p50": 5.3567, "p95": 8.1286, "max": 9.481, "min": 3.0404}, "spec_accept_length_mean": 2.094, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 27, "new_tokens": 131072, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c8", "arm": "e7b64", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9622, "corpus_window": {"start": 4700000, "end": 4716384}, "ok": 16, "failed": 0, "wall_s": 162.84, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 65536, "output_throughput_tok_s": 402.47, "input_throughput_tok_s": 100.62, "ttft_s": {"mean": 0.9469, "p50": 1.3161, "p95": 1.6549, "max": 1.6549, "min": 0.3801}, "tpot_s": {"mean": 0.0185, "p50": 0.0181, "p95": 0.0243, "max": 0.0243, "min": 0.0149}, "e2e_s": {"mean": 76.5237, "p50": 75.395, "p95": 101.2464, "max": 101.2464, "min": 61.5722}, "per_req_out_tok_s_e2e": {"mean": 54.4322, "p50": 55.1122, "p95": 66.5236, "max": 66.5236, "min": 40.4557}, "per_req_decode_tok_s": {"mean": 55.085, "p50": 56.3616, "p95": 66.9683, "max": 66.9683, "min": 41.1259}, "spec_accept_length_mean": 3.951, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 10, "new_tokens": 16384, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c32", "arm": "e7b64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9623, "corpus_window": {"start": 4700000, "end": 4765536}, "ok": 64, "failed": 0, "wall_s": 387.9, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 262144, "output_throughput_tok_s": 675.81, "input_throughput_tok_s": 168.95, "ttft_s": {"mean": 2.5762, "p50": 1.5202, "p95": 6.1895, "max": 6.2023, "min": 0.4797}, "tpot_s": {"mean": 0.043, "p50": 0.042, "p95": 0.0546, "max": 0.0604, "min": 0.0317}, "e2e_s": {"mean": 178.7057, "p50": 174.8683, "p95": 227.0262, "max": 253.2517, "min": 130.3185}, "per_req_out_tok_s_e2e": {"mean": 23.3694, "p50": 23.4457, "p95": 28.0936, "max": 31.4307, "min": 16.1736}, "per_req_decode_tok_s": {"mean": 23.6957, "p50": 23.8798, "p95": 28.3919, "max": 31.568, "min": 16.5524}, "spec_accept_length_mean": 3.82, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 36, "new_tokens": 65536, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c64", "arm": "e7b64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9624, "corpus_window": {"start": 4700000, "end": 4831072}, "ok": 128, "failed": 0, "wall_s": 634.35, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 524288, "output_throughput_tok_s": 826.5, "input_throughput_tok_s": 206.62, "ttft_s": {"mean": 4.359, "p50": 2.4011, "p95": 12.1834, "max": 12.4977, "min": 0.536}, "tpot_s": {"mean": 0.0699, "p50": 0.0687, "p95": 0.0851, "max": 0.1015, "min": 0.0521}, "e2e_s": {"mean": 290.7916, "p50": 283.6317, "p95": 358.4567, "max": 420.1138, "min": 213.9936}, "per_req_out_tok_s_e2e": {"mean": 14.2904, "p50": 14.4588, "p95": 16.8051, "max": 19.1408, "min": 9.7497}, "per_req_decode_tok_s": {"mean": 14.5029, "p50": 14.5947, "p95": 16.9633, "max": 19.2102, "min": 9.8594}, "spec_accept_length_mean": 3.878, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 69, "new_tokens": 131072, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
@ -1,3 +0,0 @@
|
||||
a8f3909211adb92501cf007ab71d3b29 /root/corpus_ids.json
|
||||
1e34dd8d813ccb7ac8243e8d1573a49e /root/bench_corpus_v2.py
|
||||
4c126d067d33b5ea27c268f37561634c /root/extract_summary.py
|
||||
@ -1,27 +0,0 @@
|
||||
| 场景 | 并发 | Output TPS 旧→新 | Δ | TPOT P95 旧→新 | TTFT P95 旧→新 | accept 旧→新 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 主场景 16K→512 | 8 | 81.4 → **83.9** | +3% | 135.1 ms → 136.1 ms | 32.52 s → 32.47 s | 2.375 → 2.556 |
|
||||
| 主场景 16K→512 | 16 | 69.3 → **92.1** | +33% | 297.8 ms → 228.8 ms | 60.86 s → 59.91 s | 2.517 → 2.485 |
|
||||
| 主场景 16K→512 | 32 | 79.0 → **97.6** | +24% | 259.9 ms → 213.6 ms | 169.68 s → 140.33 s | 2.767 → 2.777 |
|
||||
| 主场景 16K→512 | 64 | 84.3 → **99.6** | +18% | 231.5 ms → 214.6 ms | 345.69 s → 293.21 s | 2.986 → 2.954 |
|
||||
| 4.1 短输入 1K→128 | 8 | 152 → **151** | -1% | 53.7 ms → 53.7 ms | 2.16 s → 2.17 s | 2.048 → 1.992 |
|
||||
| 4.1 短输入 1K→128 | 32 | 100 → **229** | +129% | 190.5 ms → 150.6 ms | 25.60 s → 7.09 s | 2.054 → 2.035 |
|
||||
| 4.1 短输入 1K→128 | 64 | 103 → **272** | +164% | 177.7 ms → 292.7 ms | 65.81 s → 12.15 s | 2.087 → 2.094 |
|
||||
| 4.2 长输出 1K→4K | 8 | 412 → **402** | -2% | 22.5 ms → 24.3 ms | 1.68 s → 1.65 s | 4.003 → 3.951 |
|
||||
| 4.2 长输出 1K→4K | 32 | 252 → **676** | +168% | 74.6 ms → 54.6 ms | 304.75 s → 6.19 s | 3.852 → 3.82 |
|
||||
| 4.2 长输出 1K→4K | 64 | (无旧数据) | | 826 (新) | 85.1 ms | 3.878 |
|
||||
|
||||
## 附录：E7b64 复测全量指标
|
||||
|
||||
| 场景点 | ok/nreq | wall(s) | out tok/s | in tok/s | TTFT mean/p95/max(s) | TPOT mean/p95/max(ms) | retractions | accept | hit |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| b3_16k_c8 | 16/16 | 97.66 | 83.88 | 2684.26 | 12.12/32.47/32.47 | 69.2/136.1/136.1 | 0 | 2.556 | 0.0 |
|
||||
| b3_16k_c16 | 32/32 | 177.89 | 92.1 | 2947.27 | 19.76/59.91/63.69 | 129.7/228.8/275.9 | 0 | 2.485 | 0.0 |
|
||||
| b3_16k_c32 | 64/64 | 335.79 | 97.58 | 3122.67 | 82.65/140.33/154.66 | 138.1/213.6/277.5 | 0 | 2.777 | 0.0 |
|
||||
| b3_16k_c64 | 128/128 | 657.79 | 99.63 | 3188.2 | 204.27/293.21/322.84 | 142.5/214.6/276.6 | 0 | 2.954 | 0.0 |
|
||||
| b41_1k_c8 | 16/16 | 13.53 | 151.33 | 1210.61 | 1.25/2.17/2.17 | 42.1/53.7/53.7 | 0 | 1.992 | 0.0 |
|
||||
| b41_1k_c32 | 64/64 | 35.75 | 229.18 | 1833.42 | 2.99/7.09/7.10 | 111.9/150.6/179.8 | 0 | 2.035 | 0.0 |
|
||||
| b41_1k_c64 | 128/128 | 60.18 | 272.24 | 2177.88 | 4.76/12.15/13.48 | 191.1/292.7/331.5 | 0 | 2.094 | 0.0 |
|
||||
| b42_1k4k_c8 | 16/16 | 162.84 | 402.47 | 100.62 | 0.95/1.65/1.65 | 18.5/24.3/24.3 | 0 | 3.951 | 0.0 |
|
||||
| b42_1k4k_c32 | 64/64 | 387.9 | 675.81 | 168.95 | 2.58/6.19/6.20 | 43.0/54.6/60.4 | 0 | 3.82 | 0.0 |
|
||||
| b42_1k4k_c64 | 128/128 | 634.35 | 826.5 | 206.62 | 4.36/12.18/12.50 | 69.9/85.1/101.5 | 0 | 3.878 | 0.0 |
|
||||
File diff suppressed because one or more lines are too long
@ -1,10 +0,0 @@
|
||||
b3_16k_c8 OK
|
||||
b3_16k_c16 OK
|
||||
b3_16k_c32 OK
|
||||
b3_16k_c64 OK
|
||||
b41_1k_c8 OK
|
||||
b41_1k_c32 OK
|
||||
b41_1k_c64 OK
|
||||
b42_1k4k_c8 OK
|
||||
b42_1k4k_c32 OK
|
||||
b42_1k4k_c64 OK
|
||||
@ -1,7 +0,0 @@
|
||||
1e34dd8d813ccb7ac8243e8d1573a49e bench_corpus_v2.py
|
||||
4c126d067d33b5ea27c268f37561634c extract_summary.py
|
||||
5892b44610b2ce61f533f1721105625e run_b300_matrix.sh
|
||||
def3c64c5dc19e1d507080e3366c3762 deploy_glm53_pp4.sh
|
||||
21db641f1d997e26fcf1ddc97163b87e deploy_glm53_607_exp.sh
|
||||
a8fc9a504c041acc40831a848b4985d8 patches/custom_all_reduce.py
|
||||
65a4d22bd87ae343e7d68c4879c14f98 patches/custom_all_reduce_utils.py
|
||||
@ -1 +0,0 @@
|
||||
{"tag": "b41_1k_c32", "arm": "pp4mrr48ctl", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9681, "corpus_window": {"start": 4500000, "end": 4565536}, "ok": 64, "failed": 0, "wall_s": 30.75, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 8192, "output_throughput_tok_s": 266.42, "input_throughput_tok_s": 2131.36, "ttft_s": {"mean": 4.0402, "p50": 4.0687, "p95": 4.916, "max": 4.9315, "min": 1.6735}, "tpot_s": {"mean": 0.0884, "p50": 0.0878, "p95": 0.103, "max": 0.11, "min": 0.0809}, "e2e_s": {"mean": 15.2685, "p50": 15.3453, "p95": 15.5575, "max": 15.6425, "min": 15.0932}, "per_req_out_tok_s_e2e": {"mean": 8.3842, "p50": 8.4491, "p95": 8.4775, "max": 8.4806, "min": 8.1828}, "per_req_decode_tok_s": {"mean": 11.465, "p50": 11.4809, "p95": 12.363, "max": 12.4651, "min": 9.1633}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 36, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
@ -1,6 +0,0 @@
|
||||
{"tag": "b3_16k_c32", "arm": "pp4mrr64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9661, "corpus_window": {"start": 2300000, "end": 3348576}, "ok": 64, "failed": 0, "wall_s": 170.68, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 32768, "output_throughput_tok_s": 191.98, "input_throughput_tok_s": 6143.37, "ttft_s": {"mean": 26.5048, "p50": 27.0902, "p95": 46.4142, "max": 47.902, "min": 5.226}, "tpot_s": {"mean": 0.1148, "p50": 0.1149, "p95": 0.1529, "max": 0.1584, "min": 0.0713}, "e2e_s": {"mean": 85.1644, "p50": 85.6885, "p95": 86.1924, "max": 86.3287, "min": 84.3109}, "per_req_out_tok_s_e2e": {"mean": 6.0124, "p50": 6.0573, "p95": 6.0716, "max": 6.0728, "min": 5.9308}, "per_req_decode_tok_s": {"mean": 9.1855, "p50": 8.7353, "p95": 13.0741, "max": 14.0525, "min": 6.3271}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 4194304, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c64", "arm": "pp4mrr64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9662, "corpus_window": {"start": 2300000, "end": 4397152}, "ok": 128, "failed": 0, "wall_s": 326.79, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 65536, "output_throughput_tok_s": 200.55, "input_throughput_tok_s": 6417.5, "ttft_s": {"mean": 50.7799, "p50": 49.8304, "p95": 89.0393, "max": 146.8059, "min": 5.3643}, "tpot_s": {"mean": 0.1847, "p50": 0.1857, "p95": 0.2618, "max": 0.313, "min": 0.0538}, "e2e_s": {"mean": 145.1904, "p50": 145.9843, "p95": 149.0379, "max": 286.567, "min": 39.4665}, "per_req_out_tok_s_e2e": {"mean": 3.6818, "p50": 3.6238, "p95": 3.6472, "max": 12.973, "min": 1.7867}, "per_req_decode_tok_s": {"mean": 6.0216, "p50": 5.4382, "p95": 9.4602, "max": 18.6186, "min": 3.201}, "spec_accept_length_mean": null, "retractions_total": 3, "cache_hit_from_logs": {"prefill_batches": 532, "new_tokens": 8589312, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c32", "arm": "pp4mrr64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9663, "corpus_window": {"start": 4500000, "end": 4565536}, "ok": 64, "failed": 0, "wall_s": 35.58, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 8192, "output_throughput_tok_s": 230.24, "input_throughput_tok_s": 1841.95, "ttft_s": {"mean": 4.4841, "p50": 4.6833, "p95": 5.3728, "max": 5.3824, "min": 0.4799}, "tpot_s": {"mean": 0.1035, "p50": 0.1018, "p95": 0.1191, "max": 0.1346, "min": 0.0962}, "e2e_s": {"mean": 17.6355, "p50": 17.7518, "p95": 18.0249, "max": 18.1344, "min": 17.3667}, "per_req_out_tok_s_e2e": {"mean": 7.2594, "p50": 7.2851, "p95": 7.3661, "max": 7.3704, "min": 7.0584}, "per_req_decode_tok_s": {"mean": 9.7882, "p50": 9.8997, "p95": 10.4804, "max": 10.4815, "min": 7.4906}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 32, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c64", "arm": "pp4mrr64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9664, "corpus_window": {"start": 4500000, "end": 4631072}, "ok": 128, "failed": 0, "wall_s": 38.17, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 16384, "output_throughput_tok_s": 429.24, "input_throughput_tok_s": 3433.95, "ttft_s": {"mean": 5.5975, "p50": 5.8971, "p95": 7.303, "max": 7.3229, "min": 0.3871}, "tpot_s": {"mean": 0.105, "p50": 0.1022, "p95": 0.1191, "max": 0.1498, "min": 0.0899}, "e2e_s": {"mean": 18.9379, "p50": 19.1349, "p95": 19.3038, "max": 19.5071, "min": 18.5527}, "per_req_out_tok_s_e2e": {"mean": 6.7607, "p50": 6.8239, "p95": 6.8916, "max": 6.8993, "min": 6.5617}, "per_req_decode_tok_s": {"mean": 9.6961, "p50": 9.8648, "p95": 10.9748, "max": 11.2171, "min": 6.7276}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 44, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c32", "arm": "pp4mrr64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9665, "corpus_window": {"start": 4700000, "end": 4765536}, "ok": 64, "failed": 0, "wall_s": 769.61, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 262144, "output_throughput_tok_s": 340.62, "input_throughput_tok_s": 85.15, "ttft_s": {"mean": 2.8816, "p50": 2.3118, "p95": 4.8777, "max": 4.8873, "min": 0.3856}, "tpot_s": {"mean": 0.0925, "p50": 0.0992, "p95": 0.1004, "max": 0.1005, "min": 0.0837}, "e2e_s": {"mean": 381.5542, "p50": 411.1625, "p95": 414.2572, "max": 415.1497, "min": 344.0864}, "per_req_out_tok_s_e2e": {"mean": 10.8071, "p50": 11.4442, "p95": 11.809, "max": 11.904, "min": 9.8663}, "per_req_decode_tok_s": {"mean": 10.8859, "p50": 11.4629, "p95": 11.8826, "max": 11.9505, "min": 9.948}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 60, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c64", "arm": "pp4mrr64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9666, "corpus_window": {"start": 4700000, "end": 4831072}, "ok": 128, "failed": 0, "wall_s": 917.54, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 524288, "output_throughput_tok_s": 571.4, "input_throughput_tok_s": 142.85, "ttft_s": {"mean": 3.3071, "p50": 1.6982, "p95": 7.249, "max": 7.27, "min": 0.3884}, "tpot_s": {"mean": 0.1102, "p50": 0.1104, "p95": 0.1133, "max": 0.1185, "min": 0.1}, "e2e_s": {"mean": 454.4814, "p50": 455.407, "p95": 465.1982, "max": 486.5096, "min": 415.9375}, "per_req_out_tok_s_e2e": {"mean": 9.0166, "p50": 8.9946, "p95": 9.3778, "max": 9.8476, "min": 8.4192}, "per_req_decode_tok_s": {"mean": 9.084, "p50": 9.0613, "p95": 9.467, "max": 9.9975, "min": 8.439}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 228, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
@ -1,6 +0,0 @@
|
||||
{"tag": "b41_1k_c32", "arm": "pp4mrr64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9671, "corpus_window": {"start": 4500000, "end": 4565536}, "ok": 64, "failed": 0, "wall_s": 35.92, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 8192, "output_throughput_tok_s": 228.08, "input_throughput_tok_s": 1824.68, "ttft_s": {"mean": 4.6017, "p50": 4.71, "p95": 5.4921, "max": 5.5164, "min": 0.4391}, "tpot_s": {"mean": 0.1041, "p50": 0.1035, "p95": 0.1207, "max": 0.1375, "min": 0.0971}, "e2e_s": {"mean": 17.8172, "p50": 17.8119, "p95": 18.0286, "max": 18.187, "min": 17.6927}, "per_req_out_tok_s_e2e": {"mean": 7.1844, "p50": 7.1868, "p95": 7.2269, "max": 7.2346, "min": 7.038}, "per_req_decode_tok_s": {"mean": 9.737, "p50": 9.7435, "p95": 10.3684, "max": 10.3817, "min": 7.3313}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 32, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c64", "arm": "pp4mrr64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9672, "corpus_window": {"start": 4500000, "end": 4631072}, "ok": 128, "failed": 0, "wall_s": 37.82, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 16384, "output_throughput_tok_s": 433.2, "input_throughput_tok_s": 3465.61, "ttft_s": {"mean": 5.4604, "p50": 5.8986, "p95": 7.1577, "max": 7.2407, "min": 0.3928}, "tpot_s": {"mean": 0.1043, "p50": 0.0999, "p95": 0.1167, "max": 0.1473, "min": 0.0898}, "e2e_s": {"mean": 18.7041, "p50": 18.7087, "p95": 19.063, "max": 19.1341, "min": 18.5506}, "per_req_out_tok_s_e2e": {"mean": 6.8437, "p50": 6.8424, "p95": 6.8924, "max": 6.9, "min": 6.6896}, "per_req_decode_tok_s": {"mean": 9.7799, "p50": 10.0872, "p95": 11.0923, "max": 11.2174, "min": 6.8403}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 44, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c32", "arm": "pp4mrr64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9673, "corpus_window": {"start": 4700000, "end": 4765536}, "ok": 64, "failed": 0, "wall_s": 756.13, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 262144, "output_throughput_tok_s": 346.69, "input_throughput_tok_s": 86.67, "ttft_s": {"mean": 2.8487, "p50": 2.2062, "p95": 4.9526, "max": 4.9666, "min": 0.3857}, "tpot_s": {"mean": 0.091, "p50": 0.1, "p95": 0.1012, "max": 0.1014, "min": 0.0798}, "e2e_s": {"mean": 375.3459, "p50": 414.2945, "p95": 416.7195, "max": 417.0268, "min": 328.3758}, "per_req_out_tok_s_e2e": {"mean": 11.0403, "p50": 12.0491, "p95": 12.4404, "max": 12.4735, "min": 9.8219}, "per_req_decode_tok_s": {"mean": 11.1193, "p50": 12.0715, "p95": 12.4793, "max": 12.5369, "min": 9.8601}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 68, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c64", "arm": "pp4mrr64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9674, "corpus_window": {"start": 4700000, "end": 4831072}, "ok": 128, "failed": 0, "wall_s": 916.56, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 524288, "output_throughput_tok_s": 572.01, "input_throughput_tok_s": 143.0, "ttft_s": {"mean": 3.4732, "p50": 2.1089, "p95": 7.2969, "max": 7.3202, "min": 0.387}, "tpot_s": {"mean": 0.1097, "p50": 0.1102, "p95": 0.1125, "max": 0.1154, "min": 0.1013}, "e2e_s": {"mean": 452.9018, "p50": 454.1397, "p95": 465.3856, "max": 473.7285, "min": 416.137}, "per_req_out_tok_s_e2e": {"mean": 9.0488, "p50": 9.0208, "p95": 9.4307, "max": 9.8429, "min": 8.6463}, "per_req_decode_tok_s": {"mean": 9.1186, "p50": 9.0811, "p95": 9.5525, "max": 9.8755, "min": 8.6658}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 196, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c32", "arm": "pp4mrr64", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9675, "corpus_window": {"start": 2300000, "end": 3348576}, "ok": 64, "failed": 0, "wall_s": 170.33, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 32768, "output_throughput_tok_s": 192.38, "input_throughput_tok_s": 6156.1, "ttft_s": {"mean": 26.475, "p50": 27.0154, "p95": 46.313, "max": 47.9669, "min": 5.2614}, "tpot_s": {"mean": 0.1145, "p50": 0.1156, "p95": 0.1532, "max": 0.1592, "min": 0.0695}, "e2e_s": {"mean": 84.9718, "p50": 86.193, "p95": 86.7916, "max": 86.8312, "min": 83.4017}, "per_req_out_tok_s_e2e": {"mean": 6.0273, "p50": 6.1226, "p95": 6.1376, "max": 6.139, "min": 5.8965}, "per_req_decode_tok_s": {"mean": 9.2192, "p50": 8.8289, "p95": 13.2592, "max": 14.4106, "min": 6.2935}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 4194304, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c64", "arm": "pp4mrr64", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9676, "corpus_window": {"start": 2300000, "end": 4397152}, "ok": 128, "failed": 0, "wall_s": 326.73, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 65536, "output_throughput_tok_s": 200.58, "input_throughput_tok_s": 6418.52, "ttft_s": {"mean": 50.7606, "p50": 49.8362, "p95": 89.2249, "max": 147.4077, "min": 5.3842}, "tpot_s": {"mean": 0.185, "p50": 0.1854, "p95": 0.2615, "max": 0.3131, "min": 0.0533}, "e2e_s": {"mean": 145.3021, "p50": 145.6727, "p95": 148.6968, "max": 286.8298, "min": 39.2402}, "per_req_out_tok_s_e2e": {"mean": 3.6798, "p50": 3.6103, "p95": 3.6332, "max": 13.0478, "min": 1.785}, "per_req_decode_tok_s": {"mean": 6.0154, "p50": 5.4114, "p95": 9.3655, "max": 18.8041, "min": 3.1997}, "spec_accept_length_mean": null, "retractions_total": 3, "cache_hit_from_logs": {"prefill_batches": 532, "new_tokens": 8589312, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
@ -1,19 +0,0 @@
|
||||
| 场景 | 并发 | 点型 | Output TPS 旧→新 | Δ | TPOT P95 旧→新 | TTFT P95 旧→新 | E7b64 同点 out 参考 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 主场景 16K→512 | 32 | 锚点 | 193 → **192** | -0% | 152.0 ms → 153.2 ms | 46.43 s → 46.31 s | 97.6 |
|
||||
| 主场景 16K→512 | 64 | 目标 | 211 → **201** | -5% | 203.7 ms → 261.5 ms | 134.78 s → 89.22 s | 99.6 |
|
||||
| 4.1 短输入 1K→128 | 32 | 锚点 | 276 → **228** | -17% | 98.5 ms → 120.7 ms | 4.87 s → 5.49 s | 229 |
|
||||
| 4.1 短输入 1K→128 | 64 | 目标 | 341 → **433** | +27% | 101.8 ms → 116.7 ms | 19.96 s → 7.16 s | 272 |
|
||||
| 4.2 长输出 1K→4K | 32 | 锚点 | 367 → **347** | -6% | 90.5 ms → 101.2 ms | 4.14 s → 4.95 s | 676 |
|
||||
| 4.2 长输出 1K→4K | 64 | 目标 | 482 → **572** | +19% | 95.5 ms → 112.5 ms | 337.92 s → 7.30 s | 826 |
|
||||
|
||||
## 附录：PP4MRR64 复测全量指标
|
||||
|
||||
| 场景点 | ok/nreq | wall(s) | out tok/s | in tok/s | TTFT mean/p95/max(s) | TPOT mean/p95/max(ms) | retractions | hit |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| b3_16k_c32 | 64/64 | 170.33 | 192.38 | 6156.1 | 26.48/46.31/47.97 | 114.5/153.2/159.2 | 0 | 0.0 |
|
||||
| b3_16k_c64 | 128/128 | 326.73 | 200.58 | 6418.52 | 50.76/89.22/147.41 | 185.0/261.5/313.1 | 3 | 0.0 |
|
||||
| b41_1k_c32 | 64/64 | 35.92 | 228.08 | 1824.68 | 4.60/5.49/5.52 | 104.1/120.7/137.5 | 0 | 0.0 |
|
||||
| b41_1k_c64 | 128/128 | 37.82 | 433.2 | 3465.61 | 5.46/7.16/7.24 | 104.3/116.7/147.3 | 0 | 0.0 |
|
||||
| b42_1k4k_c32 | 64/64 | 756.13 | 346.69 | 86.67 | 2.85/4.95/4.97 | 91.0/101.2/101.4 | 0 | 0.0 |
|
||||
| b42_1k4k_c64 | 128/128 | 916.56 | 572.01 | 143.0 | 3.47/7.30/7.32 | 109.7/112.5/115.4 | 0 | 0.0 |
|
||||
File diff suppressed because one or more lines are too long
@ -1,6 +0,0 @@
|
||||
b3_16k_c32 OK
|
||||
b3_16k_c64 OK
|
||||
b41_1k_c32 OK
|
||||
b41_1k_c64 OK
|
||||
b42_1k4k_c32 OK
|
||||
b42_1k4k_c64 OK
|
||||
@ -1,6 +0,0 @@
|
||||
b41_1k_c32 OK
|
||||
b41_1k_c64 OK
|
||||
b42_1k4k_c32 OK
|
||||
b42_1k4k_c64 OK
|
||||
b3_16k_c32 OK
|
||||
b3_16k_c64 OK
|
||||
@ -1,66 +0,0 @@
|
||||
# 数据来源与核验记录（provenance）
|
||||
|
||||
原始 csv/log 按仓库惯例不入库（.gitignore `*.csv`/`*.log`），完整文件在 60.8
|
||||
`/root/bench_logs/b300eq_{tp2pp4_20260910_1138,e7b_20260910_1428,e7b64_20260910_1732}/` 与本地镜像
|
||||
`D:/sskj/b300eq/`。本文件固化其中的关键事实。
|
||||
|
||||
## GPU 清单（nvidia-smi，8×RTX 6000D，总 85,651 MiB/卡）
|
||||
|
||||
### TP2PP4 臂（mem0.85，加载后空载 → 矩阵结束）
|
||||
|
||||
| GPU | 空载 MiB | 结束 MiB |
|
||||
|---|---|---|
|
||||
| 0/1 | 64,613 | 79,391 |
|
||||
| 2/3 | 70,867 | 81,751 |
|
||||
| 4/5 | 74,499 | 84,439 / 84,631 |
|
||||
| 6/7 | 75,361 | 84,491 |
|
||||
|
||||
vram_timeline.csv（30s 采样）全程峰值 **85,013 MiB**（主场景 C=64，最紧张卡余量 ~638 MiB）。
|
||||
|
||||
### E7b 臂（mem0.90 + EAGLE 草稿权重，空载更高）
|
||||
|
||||
| GPU | 空载 MiB | 结束 MiB |
|
||||
|---|---|---|
|
||||
| 0 | 77,861 | 83,477 |
|
||||
| 1/2/5/6 | 77,955 | 83,551 / 83,553 |
|
||||
| 3/4/7 | 77,859 | 83,477 / 83,479 |
|
||||
|
||||
全程峰值 **83,553 MiB**（余量 ~2.1 GiB）。
|
||||
|
||||
### E7b64 复测臂（MRR64 + decode 图桶 1–64，20260910_1732）
|
||||
|
||||
- 部署：`deploy_glm53_e7b_hicc.sh`（md5 165db732…），与 E7b 初测唯一差异 = MRR 16→64 + 图参数 `--cuda-graph-max-bs-decode 64 --cuda-graph-bs-decode "1 2 3 4 6 8 12 16 24 32 48 64"`；KV 池 276,480 不变，server_args 核验 `max_running_requests=64`、图桶 13 档
|
||||
- 启动计时（09:30:27）：load_weight=217.07 s；cuda_graph={prefill=94.48, target_verify=35.95, draft_decode=13.40, draft_extend=1.85}；捕获后 avail_gpu_mem=6.11 GB
|
||||
- 空载 79,317~79,411 MiB/卡；复测矩阵（16K/1K 场景，vram_timeline 30s 采样 116 帧）全程峰值 **83,627 MiB**（余量 ~3.9 GiB）
|
||||
- 10 点全部 OK、命中核验全 0.0（全新容器实例=回收窗口重新处女文本）、0 retraction；run-id 9601-9610
|
||||
- C=8 锚点 vs 初测偏差：16K 81.4→83.9（+3%）、1K 152→151（−1%）、1K→4K 412→402（−2%）→ 两轮环境无漂移
|
||||
- 前后对比表：`retest_compare.md`（本目录镜像 = 60.8 `/root/bench_logs/retest_compare.md`）
|
||||
|
||||
## 质量门判决
|
||||
|
||||
- TP2PP4 臂：`PASS=6 FAIL=1`（唯一失败 = tool-call，D 口径无 parser，历史已知；GSM8K×5 + 中文推理全过）
|
||||
- E7b 臂：`PASS=7 FAIL=0`（含 tool-call `get_weather{"city": "北京"}`）
|
||||
- E7b64 复测臂：`PASS=7 FAIL=0`（部署后以 "The server is fired up" 真就绪信号判定后跑门，7/7）
|
||||
|
||||
## 在役容器保全与恢复（60.8，TP4PP2-nomtp@0.90 口径）
|
||||
|
||||
- 停役流程：`docker stop glm53-nvfp4` → `docker rename glm53-nvfp4 glm53-nvfp4-insvc`（先改名，防 E7b 部署脚本 rm -f 同名容器）；inspect/启动命令/挂载/镜像归档于 60.8 `/root/bench_logs/b300eq_meta/`
|
||||
- 镜像：`lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729`（sha256:eb090e39…）
|
||||
- 停役前显存：82,221~82,395 MiB/卡
|
||||
- 恢复流程：E7b 测试容器 rm（显存排干 0 MiB）→ `docker rename glm53-nvfp4-insvc glm53-nvfp4 && docker start`
|
||||
- 恢复核验第一轮（09-10 午，初测后）：health 200（启动后 ~4 min）；16K/16tok 冷抽测 ok=1/1、wall 3.59 s；显存 GPU4-7 与停役前持平（82,2xx MiB）、GPU0-3 低 ~5 GiB（重启后 radix 池未回填，正常）；容器口径未变
|
||||
- 恢复核验第二轮（09-10 晚，e7b64 复测拆台后，`restore_insvc_e7b64.sh` 自动化）：fired up（start 后 ~4 min）→ health 200 → 16K/32tok 抽测 ok + C=4×16K/256tok 抽测 ok；KV 池 647,040 tokens（8 rank 一致）+ server_args 与归档启动命令逐字一致（TP4PP2/mem0.90/MRR16/cps8192/hicache×3/ctx 1,048,576）；显存 GPU4-7 82,2xx MiB 持平、GPU0-3 77.2 GiB = mem0.90 静态预算水位（较停役前热稳态多 ~5 GiB 余量，与第一轮同象）
|
||||
- 僵尸 PID 现象记录：`docker stop`/`rm` 偶发 "container PID xxx is zombie and can not be killed"，实为收尾边界现象（容器终态 exited 137、显存归零），等待 ~20s 重试即成功
|
||||
|
||||
## 执行资产 md5（60.8 = 本目录 = 60.7 原件，三方一致）
|
||||
|
||||
见 `md5_ledger.txt`。corpus 语料：`/root/corpus_ids.json`（21,296,780 tokens，消费至 21,235,008，回收窗口协议见 REPORT.md 附录 A）。
|
||||
|
||||
### PP4MRR64 复测臂（MRR 48→64；pass-1 20260910_1927 + v2 20260910_2021 + MRR48 对照 20260910_2112）
|
||||
|
||||
- 部署：`deploy_glm53_pp4_mrr64.sh`（md5 533440ef…），与 `deploy_glm53_pp4.sh`（def3c64c）逐 token diff 仅 `MRR=48→64` 一处；KV 池 1,040,384 / mem0.85 / cps16384 / radix 关全部不变；decode 图保持栈默认（max_bs 256、桶含 56/64）——本臂从未掉图，MRR 才是初测 C=64 点的活跃上限（48 活跃+16 排队）
|
||||
- pass-1（19:27–20:14，run-id 9661–9666）：6 点全 OK、命中核验全 0.0；16K c64 出现 3 次 retraction（64×17.4K≈1.11M tokens > 池 1,040,384，活跃封顶 ~59 条）
|
||||
- v2（20:21–21:07，run-id 9671–9676，**正文采用值**）：全新实例 + 易回退点 16K c64 排末位，用于排除"retraction 后遗污染后续点"假设；6 点与 pass-1 全部对齐 ≤1.8%（4.1 c32 228.08 vs 230.2、4.1 c64 433.2 vs 429.2、4.2 c32 346.69 vs 340.6、4.2 c64 572.01 vs 571.4、16K c32 192.38 vs 192.0、16K c64 200.58 vs 200.6）→ 污染假设否证，1K 锚点回退为 MRR64 部署的可复现行为；16K c64 再次 3 次 retraction（确定性行为）
|
||||
- v2 矩阵 VRAM 峰值 84,791 MiB（GPU2，16K c64 池顶运行时采样）；QG 复验 PASS=6 FAIL=1（tool-call 已知项）
|
||||
- MRR48 归因对照（21:12–21:14，run-id 9681）：原版 `deploy_glm53_pp4.sh` 全新实例跑 4.1 c32 单点 = **266.42 tok/s**（TPOT p50 87.8ms，复现初测 87ms 水平）→ −17% 锚点分解：275.6（初测，第 9 点热实例）→ 266.4（新鲜度 −3.3%）→ 228.1（MRR64 配置代价 −14.4%）；逐请求 TPOT p50 87→102ms 均匀抬高、三实例可复现、机制未定位
|
||||
- 恢复核验第三轮（21:13–21:17，`control_mrr48_41c32.sh` 自动链）：rm glm53-pp4 → VRAM 排干 → rename 回 + start → fired up → health 200 → 16K/32tok 抽测 ok；KV 池 647,040（8 rank 一致）；显存 GPU0-3 77.2 GiB / GPU4-7 82.3 GiB 与停役前一致
|
||||
@ -1,24 +0,0 @@
|
||||
{"tag": "b3_16k_c1", "arm": "tp2pp4", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9501, "corpus_window": {"start": 2300000, "end": 2431072}, "ok": 8, "failed": 0, "wall_s": 245.55, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 16.68, "input_throughput_tok_s": 533.8, "ttft_s": {"mean": 5.3493, "p50": 5.3474, "p95": 5.3628, "max": 5.3628, "min": 5.3438}, "tpot_s": {"mean": 0.0496, "p50": 0.0495, "p95": 0.0503, "max": 0.0503, "min": 0.0493}, "e2e_s": {"mean": 30.6931, "p50": 30.657, "p95": 31.0364, "max": 31.0364, "min": 30.5593}, "per_req_out_tok_s_e2e": {"mean": 16.6817, "p50": 16.7233, "p95": 16.7543, "max": 16.7543, "min": 16.4968}, "per_req_decode_tok_s": {"mean": 20.2031, "p50": 20.2689, "p95": 20.3081, "max": 20.3081, "min": 19.9299}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 32, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c8", "arm": "tp2pp4", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9502, "corpus_window": {"start": 2300000, "end": 2562144}, "ok": 16, "failed": 0, "wall_s": 81.0, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 8192, "output_throughput_tok_s": 101.13, "input_throughput_tok_s": 3236.22, "ttft_s": {"mean": 10.0539, "p50": 10.6041, "p95": 14.7948, "max": 14.7948, "min": 5.2557}, "tpot_s": {"mean": 0.0595, "p50": 0.0602, "p95": 0.0709, "max": 0.0709, "min": 0.0481}, "e2e_s": {"mean": 40.4682, "p50": 41.4728, "p95": 41.6589, "max": 41.6589, "min": 39.2731}, "per_req_out_tok_s_e2e": {"mean": 12.662, "p50": 13.0063, "p95": 13.0369, "max": 13.0369, "min": 12.2903}, "per_req_decode_tok_s": {"mean": 17.0379, "p50": 17.0019, "p95": 20.8395, "max": 20.8395, "min": 14.1226}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 64, "new_tokens": 1048576, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c16", "arm": "tp2pp4", "summary": {"concurrency": 16, "num_requests": 32, "run_id": 9503, "corpus_window": {"start": 2300000, "end": 2824288}, "ok": 32, "failed": 0, "wall_s": 112.16, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 16384, "output_throughput_tok_s": 146.08, "input_throughput_tok_s": 4674.5, "ttft_s": {"mean": 15.5283, "p50": 16.1378, "p95": 25.7815, "max": 25.8389, "min": 5.267}, "tpot_s": {"mean": 0.0792, "p50": 0.0797, "p95": 0.0985, "max": 0.1012, "min": 0.0571}, "e2e_s": {"mean": 55.9934, "p50": 56.8023, "p95": 57.0853, "max": 57.1347, "min": 54.9678}, "per_req_out_tok_s_e2e": {"mean": 9.1467, "p50": 9.2952, "p95": 9.3139, "max": 9.3145, "min": 8.9613}, "per_req_decode_tok_s": {"mean": 12.9884, "p50": 12.7194, "p95": 16.7725, "max": 17.5544, "min": 9.9045}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 2097152, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c32", "arm": "tp2pp4", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9504, "corpus_window": {"start": 2300000, "end": 3348576}, "ok": 64, "failed": 0, "wall_s": 169.54, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 32768, "output_throughput_tok_s": 193.27, "input_throughput_tok_s": 6184.73, "ttft_s": {"mean": 26.5164, "p50": 27.1445, "p95": 46.4256, "max": 47.8906, "min": 5.2618}, "tpot_s": {"mean": 0.1137, "p50": 0.114, "p95": 0.152, "max": 0.1573, "min": 0.07}, "e2e_s": {"mean": 84.6344, "p50": 85.322, "p95": 85.7436, "max": 85.8459, "min": 83.6335}, "per_req_out_tok_s_e2e": {"mean": 6.0502, "p50": 6.108, "p95": 6.1205, "max": 6.1219, "min": 5.9642}, "per_req_decode_tok_s": {"mean": 9.2806, "p50": 8.8387, "p95": 13.3026, "max": 14.3169, "min": 6.3708}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 4194304, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b3_16k_c64", "arm": "tp2pp4", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9505, "corpus_window": {"start": 2300000, "end": 4397152}, "ok": 128, "failed": 0, "wall_s": 310.79, "input_len": 16384, "shared_len": 0, "unique_len": 16384, "output_len": 512, "output_tokens_total": 65536, "output_throughput_tok_s": 210.87, "input_throughput_tok_s": 6747.91, "ttft_s": {"mean": 63.0522, "p50": 49.1362, "p95": 134.7794, "max": 139.9931, "min": 5.3652}, "tpot_s": {"mean": 0.1392, "p50": 0.1358, "p95": 0.2037, "max": 0.2143, "min": 0.0707}, "e2e_s": {"mean": 134.1654, "p50": 114.4018, "p95": 226.099, "max": 226.1544, "min": 84.0644}, "per_req_out_tok_s_e2e": {"mean": 4.1916, "p50": 4.476, "p95": 6.0862, "max": 6.0906, "min": 2.2639}, "per_req_decode_tok_s": {"mean": 7.7966, "p50": 7.3815, "p95": 11.9051, "max": 14.1753, "min": 4.6761}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 512, "new_tokens": 8388608, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c1", "arm": "tp2pp4", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9511, "corpus_window": {"start": 4500000, "end": 4508192}, "ok": 8, "failed": 0, "wall_s": 52.95, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 1024, "output_throughput_tok_s": 19.34, "input_throughput_tok_s": 154.7, "ttft_s": {"mean": 0.3848, "p50": 0.3852, "p95": 0.3865, "max": 0.3865, "min": 0.3825}, "tpot_s": {"mean": 0.0491, "p50": 0.0491, "p95": 0.0493, "max": 0.0493, "min": 0.0488}, "e2e_s": {"mean": 6.619, "p50": 6.62, "p95": 6.6447, "max": 6.6447, "min": 6.5863}, "per_req_out_tok_s_e2e": {"mean": 19.3385, "p50": 19.3379, "p95": 19.4343, "max": 19.4343, "min": 19.2635}, "per_req_decode_tok_s": {"mean": 20.5327, "p50": 20.5305, "p95": 20.6346, "max": 20.6346, "min": 20.4442}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 32, "new_tokens": 32768, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c8", "arm": "tp2pp4", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9512, "corpus_window": {"start": 4500000, "end": 4516384}, "ok": 16, "failed": 0, "wall_s": 20.1, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 2048, "output_throughput_tok_s": 101.91, "input_throughput_tok_s": 815.3, "ttft_s": {"mean": 1.3131, "p50": 1.2998, "p95": 1.6901, "max": 1.6901, "min": 0.6481}, "tpot_s": {"mean": 0.0686, "p50": 0.0684, "p95": 0.0729, "max": 0.0729, "min": 0.0669}, "e2e_s": {"mean": 10.024, "p50": 10.1096, "p95": 10.2887, "max": 10.2887, "min": 9.7907}, "per_req_out_tok_s_e2e": {"mean": 12.7741, "p50": 12.9718, "p95": 13.0736, "max": 13.0736, "min": 12.4409}, "per_req_decode_tok_s": {"mean": 14.7032, "p50": 14.7329, "p95": 15.0683, "max": 15.0683, "min": 13.8323}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 24, "new_tokens": 65536, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c32", "arm": "tp2pp4", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9513, "corpus_window": {"start": 4500000, "end": 4565536}, "ok": 64, "failed": 0, "wall_s": 29.73, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 8192, "output_throughput_tok_s": 275.55, "input_throughput_tok_s": 2204.4, "ttft_s": {"mean": 3.8882, "p50": 3.9615, "p95": 4.8674, "max": 4.8759, "min": 1.5565}, "tpot_s": {"mean": 0.0856, "p50": 0.0869, "p95": 0.0985, "max": 0.1068, "min": 0.0785}, "e2e_s": {"mean": 14.758, "p50": 14.8384, "p95": 15.0559, "max": 15.1157, "min": 14.5917}, "per_req_out_tok_s_e2e": {"mean": 8.6742, "p50": 8.7413, "p95": 8.7701, "max": 8.7721, "min": 8.468}, "per_req_decode_tok_s": {"mean": 11.8377, "p50": 11.6015, "p95": 12.822, "max": 12.8317, "min": 9.4403}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 36, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b41_1k_c64", "arm": "tp2pp4", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9514, "corpus_window": {"start": 4500000, "end": 4631072}, "ok": 128, "failed": 0, "wall_s": 48.11, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 128, "output_tokens_total": 16384, "output_throughput_tok_s": 340.53, "input_throughput_tok_s": 2724.25, "ttft_s": {"mean": 8.1393, "p50": 4.8485, "p95": 19.9553, "max": 20.0096, "min": 0.3836}, "tpot_s": {"mean": 0.0933, "p50": 0.0924, "p95": 0.1018, "max": 0.1251, "min": 0.0818}, "e2e_s": {"mean": 19.9938, "p50": 16.0958, "p95": 32.0105, "max": 32.0464, "min": 15.7372}, "per_req_out_tok_s_e2e": {"mean": 6.9903, "p50": 7.9525, "p95": 8.1268, "max": 8.1336, "min": 3.9942}, "per_req_decode_tok_s": {"mean": 10.8849, "p50": 10.9086, "p95": 12.2874, "max": 12.3245, "min": 8.0552}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 52, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c1", "arm": "tp2pp4", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9521, "corpus_window": {"start": 4700000, "end": 4708192}, "ok": 8, "failed": 0, "wall_s": 1614.16, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 32768, "output_throughput_tok_s": 20.3, "input_throughput_tok_s": 5.08, "ttft_s": {"mean": 0.3926, "p50": 0.394, "p95": 0.3952, "max": 0.3952, "min": 0.3852}, "tpot_s": {"mean": 0.0492, "p50": 0.0491, "p95": 0.0498, "max": 0.0498, "min": 0.049}, "e2e_s": {"mean": 201.7695, "p50": 201.6506, "p95": 204.4013, "max": 204.4013, "min": 200.9309}, "per_req_out_tok_s_e2e": {"mean": 20.3009, "p50": 20.3439, "p95": 20.3851, "max": 20.3851, "min": 20.039}, "per_req_decode_tok_s": {"mean": 20.3407, "p50": 20.3837, "p95": 20.4254, "max": 20.4254, "min": 20.078}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 32, "new_tokens": 32768, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c8", "arm": "tp2pp4", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9522, "corpus_window": {"start": 4700000, "end": 4716384}, "ok": 16, "failed": 0, "wall_s": 649.69, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 65536, "output_throughput_tok_s": 100.87, "input_throughput_tok_s": 25.22, "ttft_s": {"mean": 1.352, "p50": 1.4385, "p95": 1.8726, "max": 1.8726, "min": 0.4364}, "tpot_s": {"mean": 0.079, "p50": 0.0791, "p95": 0.0794, "max": 0.0794, "min": 0.0788}, "e2e_s": {"mean": 324.8285, "p50": 325.5653, "p95": 325.6331, "max": 325.6331, "min": 324.0447}, "per_req_out_tok_s_e2e": {"mean": 12.6098, "p50": 12.6389, "p95": 12.6402, "max": 12.6402, "min": 12.5786}, "per_req_decode_tok_s": {"mean": 12.6626, "p50": 12.6734, "p95": 12.7003, "max": 12.7003, "min": 12.5955}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 20, "new_tokens": 65536, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c32", "arm": "tp2pp4", "summary": {"concurrency": 32, "num_requests": 64, "run_id": 9523, "corpus_window": {"start": 4700000, "end": 4765536}, "ok": 64, "failed": 0, "wall_s": 714.14, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 262144, "output_throughput_tok_s": 367.08, "input_throughput_tok_s": 91.77, "ttft_s": {"mean": 1.9772, "p50": 1.1755, "p95": 4.1378, "max": 4.1467, "min": 0.3867}, "tpot_s": {"mean": 0.0861, "p50": 0.0887, "p95": 0.0905, "max": 0.0911, "min": 0.0805}, "e2e_s": {"mean": 354.393, "p50": 367.2171, "p95": 373.3199, "max": 374.0319, "min": 330.2086}, "per_req_out_tok_s_e2e": {"mean": 11.5833, "p50": 11.8878, "p95": 12.2953, "max": 12.4043, "min": 10.9509}, "per_req_decode_tok_s": {"mean": 11.6457, "p50": 11.9132, "p95": 12.3308, "max": 12.4283, "min": 10.9854}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 262144, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b42_1k4k_c64", "arm": "tp2pp4", "summary": {"concurrency": 64, "num_requests": 128, "run_id": 9524, "corpus_window": {"start": 4700000, "end": 4831072}, "ok": 128, "failed": 0, "wall_s": 1088.63, "input_len": 1024, "shared_len": 0, "unique_len": 1024, "output_len": 4096, "output_tokens_total": 524288, "output_throughput_tok_s": 481.6, "input_throughput_tok_s": 120.4, "ttft_s": {"mean": 80.7415, "p50": 4.7569, "p95": 337.9195, "max": 338.4726, "min": 0.3985}, "tpot_s": {"mean": 0.0912, "p50": 0.0914, "p95": 0.0955, "max": 0.0963, "min": 0.0852}, "e2e_s": {"mean": 454.4024, "p50": 385.4538, "p95": 718.7004, "max": 732.2026, "min": 351.072}, "per_req_out_tok_s_e2e": {"mean": 9.6298, "p50": 10.6531, "p95": 11.4339, "max": 11.6671, "min": 5.5941}, "per_req_decode_tok_s": {"mean": 10.9736, "p50": 10.9486, "p95": 11.4764, "max": 11.7362, "min": 10.3887}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 200, "new_tokens": 524288, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_64k_c1", "arm": "tp2pp4", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9531, "corpus_window": {"start": 5000000, "end": 5524288}, "ok": 8, "failed": 0, "wall_s": 285.91, "input_len": 65536, "shared_len": 0, "unique_len": 65536, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 14.33, "input_throughput_tok_s": 1833.72, "ttft_s": {"mean": 10.2811, "p50": 10.2794, "p95": 10.3216, "max": 10.3216, "min": 10.2634}, "tpot_s": {"mean": 0.0498, "p50": 0.0499, "p95": 0.05, "max": 0.05, "min": 0.0496}, "e2e_s": {"mean": 35.739, "p50": 35.7693, "p95": 35.838, "max": 35.838, "min": 35.6073}, "per_req_out_tok_s_e2e": {"mean": 14.3261, "p50": 14.3345, "p95": 14.3791, "max": 14.3791, "min": 14.2865}, "per_req_decode_tok_s": {"mean": 20.112, "p50": 20.1156, "p95": 20.2096, "max": 20.2096, "min": 20.052}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 2097152, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_64k_c4", "arm": "tp2pp4", "summary": {"concurrency": 4, "num_requests": 8, "run_id": 9532, "corpus_window": {"start": 5000000, "end": 5524288}, "ok": 8, "failed": 0, "wall_s": 122.3, "input_len": 65536, "shared_len": 0, "unique_len": 65536, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 33.49, "input_throughput_tok_s": 4286.94, "ttft_s": {"mean": 19.5548, "p50": 22.5526, "p95": 28.8262, "max": 28.8262, "min": 10.2823}, "tpot_s": {"mean": 0.0814, "p50": 0.0873, "p95": 0.0995, "max": 0.0995, "min": 0.0632}, "e2e_s": {"mean": 61.1314, "p50": 61.2078, "p95": 61.2548, "max": 61.2548, "min": 61.002}, "per_req_out_tok_s_e2e": {"mean": 8.3754, "p50": 8.3851, "p95": 8.3932, "max": 8.3932, "min": 8.3585}, "per_req_decode_tok_s": {"mean": 12.6679, "p50": 13.2834, "p95": 15.8466, "max": 15.8466, "min": 10.0675}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 2097152, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_64k_c8", "arm": "tp2pp4", "summary": {"concurrency": 8, "num_requests": 16, "run_id": 9533, "corpus_window": {"start": 5000000, "end": 6048576}, "ok": 16, "failed": 0, "wall_s": 185.59, "input_len": 65536, "shared_len": 0, "unique_len": 65536, "output_len": 512, "output_tokens_total": 8192, "output_throughput_tok_s": 44.14, "input_throughput_tok_s": 5650.11, "ttft_s": {"mean": 31.833, "p50": 34.8143, "p95": 53.3841, "max": 53.3841, "min": 10.2802}, "tpot_s": {"mean": 0.1192, "p50": 0.1246, "p95": 0.1618, "max": 0.1618, "min": 0.0765}, "e2e_s": {"mean": 92.7517, "p50": 92.8482, "p95": 92.9902, "max": 92.9902, "min": 92.4957}, "per_req_out_tok_s_e2e": {"mean": 5.5201, "p50": 5.5226, "p95": 5.5354, "max": 5.5354, "min": 5.506}, "per_req_decode_tok_s": {"mean": 8.9039, "p50": 8.8159, "p95": 13.0909, "max": 13.0909, "min": 6.1915}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 4194304, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_128k_c1", "arm": "tp2pp4", "summary": {"concurrency": 1, "num_requests": 8, "run_id": 9541, "corpus_window": {"start": 6200000, "end": 7248576}, "ok": 8, "failed": 0, "wall_s": 347.5, "input_len": 131072, "shared_len": 0, "unique_len": 131072, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 11.79, "input_throughput_tok_s": 3017.45, "ttft_s": {"mean": 18.2074, "p50": 18.2043, "p95": 18.2252, "max": 18.2252, "min": 18.1922}, "tpot_s": {"mean": 0.0494, "p50": 0.0496, "p95": 0.0497, "max": 0.0497, "min": 0.0485}, "e2e_s": {"mean": 43.4375, "p50": 43.5633, "p95": 43.6252, "max": 43.6252, "min": 43.0007}, "per_req_out_tok_s_e2e": {"mean": 11.7874, "p50": 11.7541, "p95": 11.9068, "max": 11.9068, "min": 11.7363}, "per_req_decode_tok_s": {"mean": 20.2951, "p50": 20.1994, "p95": 20.6444, "max": 20.6444, "min": 20.1578}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 4194304, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_128k_c2", "arm": "tp2pp4", "summary": {"concurrency": 2, "num_requests": 8, "run_id": 9542, "corpus_window": {"start": 6200000, "end": 7248576}, "ok": 8, "failed": 0, "wall_s": 243.3, "input_len": 131072, "shared_len": 0, "unique_len": 131072, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 16.84, "input_throughput_tok_s": 4309.84, "ttft_s": {"mean": 25.2658, "p50": 32.1952, "p95": 32.4188, "max": 32.4188, "min": 18.173}, "tpot_s": {"mean": 0.0696, "p50": 0.0831, "p95": 0.0837, "max": 0.0837, "min": 0.0556}, "e2e_s": {"mean": 60.8183, "p50": 60.7483, "p95": 61.2374, "max": 61.2374, "min": 60.6134}, "per_req_out_tok_s_e2e": {"mean": 8.4186, "p50": 8.4352, "p95": 8.447, "max": 8.447, "min": 8.3609}, "per_req_decode_tok_s": {"mean": 14.9825, "p50": 17.8153, "p95": 18.0089, "max": 18.0089, "min": 11.965}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 4194304, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b51_128k_c4", "arm": "tp2pp4", "summary": {"concurrency": 4, "num_requests": 8, "run_id": 9543, "corpus_window": {"start": 6200000, "end": 7248576}, "ok": 8, "failed": 0, "wall_s": 186.37, "input_len": 131072, "shared_len": 0, "unique_len": 131072, "output_len": 512, "output_tokens_total": 4096, "output_throughput_tok_s": 21.98, "input_throughput_tok_s": 5626.35, "ttft_s": {"mean": 39.2939, "p50": 46.1927, "p95": 60.3496, "max": 60.3496, "min": 18.2774}, "tpot_s": {"mean": 0.1054, "p50": 0.1187, "p95": 0.1468, "max": 0.1468, "min": 0.0638}, "e2e_s": {"mean": 93.1534, "p50": 93.2237, "p95": 93.3139, "max": 93.3139, "min": 92.97}, "per_req_out_tok_s_e2e": {"mean": 5.4963, "p50": 5.497, "p95": 5.5072, "max": 5.5072, "min": 5.4869}, "per_req_decode_tok_s": {"mean": 10.4415, "p50": 10.8799, "p95": 15.6959, "max": 15.6959, "min": 6.8234}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 256, "new_tokens": 4194304, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b52_256k_c1", "arm": "tp2pp4", "summary": {"concurrency": 1, "num_requests": 1, "run_id": 9551, "corpus_window": {"start": 8400000, "end": 8662144}, "ok": 1, "failed": 0, "wall_s": 37.18, "input_len": 262144, "shared_len": 0, "unique_len": 262144, "output_len": 1, "output_tokens_total": 1, "output_throughput_tok_s": 0.03, "input_throughput_tok_s": 7050.23, "ttft_s": {"mean": 37.1811, "p50": 37.1811, "p95": 37.1811, "max": 37.1811, "min": 37.1811}, "tpot_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "e2e_s": {"mean": 37.1814, "p50": 37.1814, "p95": 37.1814, "max": 37.1814, "min": 37.1814}, "per_req_out_tok_s_e2e": {"mean": 0.0269, "p50": 0.0269, "p95": 0.0269, "max": 0.0269, "min": 0.0269}, "per_req_decode_tok_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 64, "new_tokens": 1048576, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b52_256k_c2", "arm": "tp2pp4", "summary": {"concurrency": 2, "num_requests": 2, "run_id": 9552, "corpus_window": {"start": 8400000, "end": 8924288}, "ok": 2, "failed": 0, "wall_s": 70.19, "input_len": 262144, "shared_len": 0, "unique_len": 262144, "output_len": 1, "output_tokens_total": 2, "output_throughput_tok_s": 0.03, "input_throughput_tok_s": 7469.22, "ttft_s": {"mean": 53.6945, "p50": 70.1229, "p95": 70.1229, "max": 70.1229, "min": 37.2662}, "tpot_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "e2e_s": {"mean": 53.6948, "p50": 70.1232, "p95": 70.1232, "max": 70.1232, "min": 37.2664}, "per_req_out_tok_s_e2e": {"mean": 0.0205, "p50": 0.0268, "p95": 0.0268, "max": 0.0268, "min": 0.0143}, "per_req_decode_tok_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 2097152, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b52_256k_c3", "arm": "tp2pp4", "summary": {"concurrency": 3, "num_requests": 3, "run_id": 9553, "corpus_window": {"start": 8400000, "end": 9186432}, "ok": 3, "failed": 0, "wall_s": 103.12, "input_len": 262144, "shared_len": 0, "unique_len": 262144, "output_len": 1, "output_tokens_total": 3, "output_throughput_tok_s": 0.03, "input_throughput_tok_s": 7626.22, "ttft_s": {"mean": 70.1194, "p50": 70.1196, "p95": 102.986, "max": 102.986, "min": 37.2526}, "tpot_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "e2e_s": {"mean": 70.1196, "p50": 70.1198, "p95": 102.9863, "max": 102.9863, "min": 37.2529}, "per_req_out_tok_s_e2e": {"mean": 0.0169, "p50": 0.0143, "p95": 0.0268, "max": 0.0268, "min": 0.0097}, "per_req_decode_tok_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 192, "new_tokens": 3145728, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b52_512k_c1", "arm": "tp2pp4", "summary": {"concurrency": 1, "num_requests": 1, "run_id": 9554, "corpus_window": {"start": 9300000, "end": 9824288}, "ok": 1, "failed": 0, "wall_s": 92.6, "input_len": 524288, "shared_len": 0, "unique_len": 524288, "output_len": 1, "output_tokens_total": 1, "output_throughput_tok_s": 0.01, "input_throughput_tok_s": 5662.03, "ttft_s": {"mean": 92.5959, "p50": 92.5959, "p95": 92.5959, "max": 92.5959, "min": 92.5959}, "tpot_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "e2e_s": {"mean": 92.5962, "p50": 92.5962, "p95": 92.5962, "max": 92.5962, "min": 92.5962}, "per_req_out_tok_s_e2e": {"mean": 0.0108, "p50": 0.0108, "p95": 0.0108, "max": 0.0108, "min": 0.0108}, "per_req_decode_tok_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 128, "new_tokens": 2097152, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
{"tag": "b52_896k_c1", "arm": "tp2pp4", "summary": {"concurrency": 1, "num_requests": 1, "run_id": 9555, "corpus_window": {"start": 9900000, "end": 10817504}, "ok": 1, "failed": 0, "wall_s": 220.91, "input_len": 917504, "shared_len": 0, "unique_len": 917504, "output_len": 1, "output_tokens_total": 1, "output_throughput_tok_s": 0.0, "input_throughput_tok_s": 4153.24, "ttft_s": {"mean": 220.9117, "p50": 220.9117, "p95": 220.9117, "max": 220.9117, "min": 220.9117}, "tpot_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "e2e_s": {"mean": 220.912, "p50": 220.912, "p95": 220.912, "max": 220.912, "min": 220.912}, "per_req_out_tok_s_e2e": {"mean": 0.0045, "p50": 0.0045, "p95": 0.0045, "max": 0.0045, "min": 0.0045}, "per_req_decode_tok_s": {"mean": null, "p50": null, "p95": null, "max": null, "min": null}, "spec_accept_length_mean": null, "retractions_total": 0, "cache_hit_from_logs": {"prefill_batches": 224, "new_tokens": 3670016, "cached_tokens": 0, "hit_rate": 0.0}}}
|
||||
@ -1,3 +0,0 @@
|
||||
a8f3909211adb92501cf007ab71d3b29 /root/corpus_ids.json
|
||||
1e34dd8d813ccb7ac8243e8d1573a49e /root/bench_corpus_v2.py
|
||||
4c126d067d33b5ea27c268f37561634c /root/extract_summary.py
|
||||
File diff suppressed because one or more lines are too long
@ -1,24 +0,0 @@
|
||||
b3_16k_c1 OK
|
||||
b3_16k_c8 OK
|
||||
b3_16k_c16 OK
|
||||
b3_16k_c32 OK
|
||||
b3_16k_c64 OK
|
||||
b41_1k_c1 OK
|
||||
b41_1k_c8 OK
|
||||
b41_1k_c32 OK
|
||||
b41_1k_c64 OK
|
||||
b42_1k4k_c1 OK
|
||||
b42_1k4k_c8 OK
|
||||
b42_1k4k_c32 OK
|
||||
b42_1k4k_c64 OK
|
||||
b51_64k_c1 OK
|
||||
b51_64k_c4 OK
|
||||
b51_64k_c8 OK
|
||||
b51_128k_c1 OK
|
||||
b51_128k_c2 OK
|
||||
b51_128k_c4 OK
|
||||
b52_256k_c1 OK
|
||||
b52_256k_c2 OK
|
||||
b52_256k_c3 OK
|
||||
b52_512k_c1 OK
|
||||
b52_896k_c1 OK
|
||||
@ -1,319 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Real-corpus (PG19) benchmark for sglang GLM-5.3-NVFP4.
|
||||
|
||||
Same methodology as bench_hit90.py (input_ids direct to /generate, temp 0,
|
||||
ignore_eos, stream, server-side completion_tokens counting, hit-rate verified
|
||||
from scheduler logs), but prompts are token slices of REAL book text tokenized
|
||||
with the served model's own tokenizer, replacing random ids.
|
||||
|
||||
Corpus file: JSON {"ids": [flat token ids], "books": [[start, end), ...]}
|
||||
|
||||
Fixed token-offset layout into the flat id array:
|
||||
[0, 117968) shared prefix for 128k points (90% of 131072)
|
||||
[0, 58976) shared prefix for 64k points (same region, shorter cut)
|
||||
pool A [131072, +4*8*13104) 128k unique suffixes, run-ids 9301-9304
|
||||
pool B [550400, +4*8*6560) 64k unique suffixes, run-ids 9305-9308
|
||||
pool C [760320, 262144+524288+524288) 16k fully-unique prompts, run-ids 9311-9313
|
||||
spare [2071040, end) warmup slices / re-run margin
|
||||
|
||||
Each run-id maps to one non-overlapping window (one window = one bench point);
|
||||
re-running a point with fresh text = bump --pool-override past the spare base.
|
||||
|
||||
v2 changes (b300-equivalent campaign, 2026-09-10):
|
||||
- stats(): p95 added (nearest-rank) to match the B300 report metric contract
|
||||
(TTFT P95 / TPOT P95).
|
||||
- --dump-records PATH: per-request records (ttft/e2e/tpot/n_out/retractions/
|
||||
spec_accept_len) written as JSONL for post-hoc percentile checks.
|
||||
- With --shared-frac 0 + --pool-override, any input length is supported
|
||||
(1024 / 16384 / 65536 / 131072 / 262144 / 524288 / 917504).
|
||||
|
||||
Usage:
|
||||
python3 bench_corpus_v2.py --corpus /root/corpus_ids.json --input-len 16384 \
|
||||
--concurrency 64 --num-requests 128 --run-id 9505 --shared-frac 0 \
|
||||
--pool-override 2300000 --output-len 512 \
|
||||
--dump-records /root/bench_logs/xx/point_records.jsonl
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import statistics
|
||||
import subprocess
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
import requests
|
||||
|
||||
OUTPUT_LEN_DEFAULT = 512
|
||||
CORPUS_DEFAULT = "/root/corpus_ids.json"
|
||||
|
||||
# fixed pool layout (see docstring)
|
||||
S1_128K_RID0, S1_64K_RID0, S2_RID0 = 9301, 9305, 9311
|
||||
POOL_A_BASE, POOL_A_PER = 131072, 8 * 13104 # 128k suffix windows
|
||||
POOL_B_BASE = POOL_A_BASE + 4 * POOL_A_PER # 550400
|
||||
POOL_B_PER = 8 * 6560 # 64k suffix windows
|
||||
POOL_C_BASE = POOL_B_BASE + 4 * POOL_B_PER # 760320
|
||||
POOL_C_SIZES = [16 * 16384, 32 * 16384, 32 * 16384] # cc8 / cc16 / cc32
|
||||
SPARE_BASE = POOL_C_BASE + sum(POOL_C_SIZES) # 2071040
|
||||
|
||||
sess = requests.Session()
|
||||
sess.trust_env = False # bypass any proxy env on the host
|
||||
|
||||
|
||||
def split_lens(input_len, shared_frac):
|
||||
# unique suffix = (1 - shared_frac) of the prompt, page-16 aligned
|
||||
# (128k @0.9 -> 13104 unique; 16k @0.0 -> fully unique prompts)
|
||||
unique = round(input_len * (1.0 - shared_frac) / 16) * 16
|
||||
return input_len - unique, unique
|
||||
|
||||
|
||||
def pool_start_for(input_len, shared_frac, run_id, override):
|
||||
if override is not None:
|
||||
return override
|
||||
if shared_frac > 0:
|
||||
if input_len == 131072:
|
||||
idx = run_id - S1_128K_RID0
|
||||
if not 0 <= idx < 4:
|
||||
sys_exit_bad_runid(run_id, "128k points use run-ids 9301-9304")
|
||||
return POOL_A_BASE + idx * POOL_A_PER
|
||||
if input_len == 65536:
|
||||
idx = run_id - S1_64K_RID0
|
||||
if not 0 <= idx < 4:
|
||||
sys_exit_bad_runid(run_id, "64k points use run-ids 9305-9308")
|
||||
return POOL_B_BASE + idx * POOL_B_PER
|
||||
sys_exit_bad_runid(run_id, "shared-frac>0 supports 131072/65536 only")
|
||||
idx = run_id - S2_RID0
|
||||
if not 0 <= idx < 3:
|
||||
sys_exit_bad_runid(run_id, "16k unique points use run-ids 9311-9313")
|
||||
return POOL_C_BASE + sum(POOL_C_SIZES[:idx])
|
||||
|
||||
|
||||
def sys_exit_bad_runid(run_id, msg):
|
||||
raise SystemExit(f"[pool] run-id {run_id} outside expected set: {msg}")
|
||||
|
||||
|
||||
def build_prompts(ids, shared_len, unique_len, num_requests, pool_start):
|
||||
if shared_len:
|
||||
shared = ids[0:shared_len]
|
||||
else:
|
||||
shared = []
|
||||
end = pool_start + num_requests * unique_len
|
||||
if end > len(ids):
|
||||
raise SystemExit(
|
||||
f"[pool] window [{pool_start}, {end}) exceeds corpus ({len(ids)} ids); "
|
||||
f"use --pool-override or a larger corpus")
|
||||
prompts = []
|
||||
for i in range(num_requests):
|
||||
s = pool_start + i * unique_len
|
||||
prompts.append(shared + ids[s:s + unique_len])
|
||||
return shared, prompts, (pool_start, end)
|
||||
|
||||
|
||||
def warmup(url, ids, shared_len):
|
||||
# primes the radix cache with the shared prefix (same role as in bench_hit90);
|
||||
# warm slice comes from the spare region so it never collides with a pool window
|
||||
if len(ids) >= SPARE_BASE + 64:
|
||||
warm_slice = ids[SPARE_BASE:SPARE_BASE + 64]
|
||||
else:
|
||||
warm_slice = ids[-64:]
|
||||
payload = {
|
||||
"input_ids": ids[0:shared_len] + warm_slice if shared_len else warm_slice,
|
||||
"sampling_params": {"max_new_tokens": 8, "temperature": 0.0, "ignore_eos": True},
|
||||
}
|
||||
t0 = time.perf_counter()
|
||||
r = sess.post(url, json=payload, timeout=1800)
|
||||
dt = time.perf_counter() - t0
|
||||
print(f"[warmup] http={r.status_code} wall={dt:.2f}s", flush=True)
|
||||
|
||||
|
||||
def bench_one(url, prompt, output_len, idx, results):
|
||||
payload = {
|
||||
"input_ids": prompt,
|
||||
"sampling_params": {"max_new_tokens": output_len, "temperature": 0.0, "ignore_eos": True},
|
||||
"stream": True,
|
||||
}
|
||||
rec = {"idx": idx}
|
||||
t0 = time.perf_counter()
|
||||
first = last = None
|
||||
first_ct = None
|
||||
final_meta = None
|
||||
max_ct = 0
|
||||
try:
|
||||
with sess.post(url, json=payload, stream=True, timeout=3600) as resp:
|
||||
for raw in resp.iter_lines():
|
||||
if not raw or not raw.startswith(b"data:"):
|
||||
continue
|
||||
body = raw[5:].strip()
|
||||
if body == b"[DONE]":
|
||||
continue
|
||||
now = time.perf_counter()
|
||||
try:
|
||||
d = json.loads(body)
|
||||
except Exception:
|
||||
continue
|
||||
mi = d.get("meta_info") or {}
|
||||
ct = mi.get("completion_tokens") or 0
|
||||
if ct:
|
||||
max_ct = max(max_ct, ct)
|
||||
if first is None:
|
||||
first = now
|
||||
first_ct = ct
|
||||
last = now
|
||||
if mi.get("finish_reason"):
|
||||
final_meta = mi
|
||||
t_end = time.perf_counter()
|
||||
n_out = max_ct or ((final_meta or {}).get("completion_tokens") or 0)
|
||||
decode_span = (last - first) if (first and last and last > first) else 0.0
|
||||
rec.update(
|
||||
ok=n_out > 0,
|
||||
ttft=(first - t0) if first else None,
|
||||
e2e=t_end - t0,
|
||||
n_out=n_out,
|
||||
first_chunk_tokens=first_ct,
|
||||
decode_span=decode_span,
|
||||
tpot=(decode_span / (n_out - 1)) if n_out > 1 else None,
|
||||
per_req_decode_tok_s=(n_out / decode_span) if decode_span > 0 else None,
|
||||
retractions=(final_meta or {}).get("num_retractions"),
|
||||
spec_accept_len=(final_meta or {}).get("spec_accept_length"),
|
||||
)
|
||||
except Exception as e:
|
||||
rec.update(ok=False, error=repr(e))
|
||||
results[idx] = rec
|
||||
|
||||
|
||||
def verify_hit_rate(container, t_start, t_end):
|
||||
def rfc3339(epoch):
|
||||
return (datetime.datetime.fromtimestamp(epoch, tz=datetime.timezone.utc)
|
||||
.isoformat().replace("+00:00", "Z"))
|
||||
|
||||
try:
|
||||
# No margin before t_start: warmup's prefill lines end strictly before it,
|
||||
# and catching them would deflate the measured hit rate.
|
||||
p = subprocess.run(
|
||||
["docker", "logs", container, "--since", rfc3339(t_start), "--until", rfc3339(t_end + 2)],
|
||||
capture_output=True, text=True, timeout=120)
|
||||
text = p.stdout + p.stderr
|
||||
except Exception as e:
|
||||
return {"error": repr(e)}
|
||||
pat = re.compile(r"#new-token: (\d+).*?#cached-token: (\d+)")
|
||||
n_batches = new_tok = cached_tok = 0
|
||||
for line in text.splitlines():
|
||||
if "TP0]" not in line or "Prefill batch" not in line:
|
||||
continue
|
||||
m = pat.search(line)
|
||||
if m:
|
||||
n_batches += 1
|
||||
new_tok += int(m.group(1))
|
||||
cached_tok += int(m.group(2))
|
||||
total = new_tok + cached_tok
|
||||
return {
|
||||
"prefill_batches": n_batches,
|
||||
"new_tokens": new_tok,
|
||||
"cached_tokens": cached_tok,
|
||||
"hit_rate": round(cached_tok / total, 4) if total else None,
|
||||
}
|
||||
|
||||
|
||||
def stats(vals):
|
||||
vals = [v for v in vals if v is not None]
|
||||
if not vals:
|
||||
return {"mean": None, "p50": None, "p95": None, "max": None, "min": None}
|
||||
s = sorted(vals)
|
||||
# nearest-rank p95: smallest value >= 95th percentile
|
||||
p95_idx = max(0, math.ceil(0.95 * len(s)) - 1)
|
||||
return {
|
||||
"mean": round(statistics.fmean(vals), 4),
|
||||
"p50": round(s[len(s) // 2], 4),
|
||||
"p95": round(s[p95_idx], 4),
|
||||
"max": round(s[-1], 4),
|
||||
"min": round(s[0], 4),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--concurrency", type=int, required=True)
|
||||
ap.add_argument("--num-requests", type=int, required=True)
|
||||
ap.add_argument("--run-id", type=int, required=True)
|
||||
ap.add_argument("--input-len", type=int, required=True, help="token length of each prompt")
|
||||
ap.add_argument("--output-len", type=int, default=OUTPUT_LEN_DEFAULT)
|
||||
ap.add_argument("--shared-frac", type=float, default=0.9)
|
||||
ap.add_argument("--corpus", default=CORPUS_DEFAULT)
|
||||
ap.add_argument("--pool-override", type=int, default=None,
|
||||
help="explicit corpus offset for the unique-suffix window (re-runs)")
|
||||
ap.add_argument("--url", default="http://127.0.0.1:30000/generate")
|
||||
ap.add_argument("--container", default="glm53-nvfp4")
|
||||
ap.add_argument("--dump-records", default=None,
|
||||
help="write per-request records as JSONL to this path")
|
||||
args = ap.parse_args()
|
||||
|
||||
with open(args.corpus) as f:
|
||||
corpus = json.load(f)
|
||||
ids = corpus["ids"]
|
||||
|
||||
shared_len, unique_len = split_lens(args.input_len, args.shared_frac)
|
||||
pool_start = pool_start_for(args.input_len, args.shared_frac, args.run_id, args.pool_override)
|
||||
shared, prompts, window = build_prompts(ids, shared_len, unique_len, args.num_requests, pool_start)
|
||||
print(f"[pool] window={window} shared_len={shared_len} unique_len={unique_len} "
|
||||
f"corpus_total={len(ids)}", flush=True)
|
||||
warmup(args.url, ids, shared_len)
|
||||
|
||||
results = {}
|
||||
t_start = time.time()
|
||||
t0 = time.perf_counter()
|
||||
with ThreadPoolExecutor(max_workers=args.concurrency) as ex:
|
||||
futs = [ex.submit(bench_one, args.url, p, args.output_len, i, results)
|
||||
for i, p in enumerate(prompts)]
|
||||
for f in futs:
|
||||
f.result()
|
||||
wall = time.perf_counter() - t0
|
||||
t_end = time.time()
|
||||
|
||||
if args.dump_records:
|
||||
with open(args.dump_records, "w") as f:
|
||||
for i in sorted(results):
|
||||
f.write(json.dumps(results[i]) + "\n")
|
||||
|
||||
hit = verify_hit_rate(args.container, t_start, t_end)
|
||||
|
||||
ok = [r for r in results.values() if r.get("ok")]
|
||||
n_out_total = sum(r["n_out"] for r in ok)
|
||||
out_tps = [r["n_out"] / r["e2e"] for r in ok if r.get("e2e")]
|
||||
ttft = stats([r.get("ttft") for r in ok])
|
||||
tpot = stats([r.get("tpot") for r in ok])
|
||||
e2e = stats([r.get("e2e") for r in ok])
|
||||
dec = stats([r.get("per_req_decode_tok_s") for r in ok])
|
||||
spec = [v for v in (r.get("spec_accept_len") for r in ok) if v is not None]
|
||||
retr = sum(r.get("retractions") or 0 for r in ok)
|
||||
|
||||
summary = {
|
||||
"concurrency": args.concurrency,
|
||||
"num_requests": args.num_requests,
|
||||
"run_id": args.run_id,
|
||||
"corpus_window": {"start": window[0], "end": window[1]},
|
||||
"ok": len(ok),
|
||||
"failed": args.num_requests - len(ok),
|
||||
"wall_s": round(wall, 2),
|
||||
"input_len": args.input_len,
|
||||
"shared_len": shared_len,
|
||||
"unique_len": unique_len,
|
||||
"output_len": args.output_len,
|
||||
"output_tokens_total": n_out_total,
|
||||
"output_throughput_tok_s": round(n_out_total / wall, 2) if wall else None,
|
||||
"input_throughput_tok_s": round(args.input_len * len(ok) / wall, 2) if wall else None,
|
||||
"ttft_s": ttft,
|
||||
"tpot_s": tpot,
|
||||
"e2e_s": e2e,
|
||||
"per_req_out_tok_s_e2e": stats(out_tps),
|
||||
"per_req_decode_tok_s": dec,
|
||||
"spec_accept_length_mean": round(statistics.fmean(spec), 3) if spec else None,
|
||||
"retractions_total": retr,
|
||||
"cache_hit_from_logs": hit,
|
||||
}
|
||||
print("\n===== SUMMARY =====")
|
||||
print(json.dumps(summary, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user