[Docs] Record Kimi EP4 MoE backend acceptance
This commit is contained in:
Zhiyi Hong
parent
27b8be09cb
commit
ec7b604a50
@ -1,5 +1,7 @@
|
|||||||
# sskj — 多平台大模型推理性能基准测试项目
|
# sskj — 多平台大模型推理性能基准测试项目
|
||||||
|
|
||||||
|
**更新（2026-08-18 18:38:17 CST）**：完成 Kimi-K3 四节点 TP32×EP4 的真实 Prefill MoE backend 验收；24/24 重复均 40/40 成功，FlashInfer MXFP4 在 8K Chunk、C=8/16 下相对 Marlin 提升约 11% Input TPS、降低约 10% TTFT，并确认 8K 优于 16K。实验默认 EP 改为 4，补齐非连续 Prefill 输入与 FlashInfer 官方 SiTU 回归，详见 `experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/README.md`。
|
||||||
|
|
||||||
**更新（2026-08-18 13:55:30 CST）**：Kimi-K3 四节点 MoE backend 验收入口支持通过环境变量覆盖 EP size 及 backend/chunk/concurrency 子矩阵，用于先执行 TP32×EP4、16K Chunk、16K→1、C=16 的最大压力容量验证，同时保留默认 TP32×EP32 全矩阵。
|
**更新（2026-08-18 13:55:30 CST）**：Kimi-K3 四节点 MoE backend 验收入口支持通过环境变量覆盖 EP size 及 backend/chunk/concurrency 子矩阵，用于先执行 TP32×EP4、16K Chunk、16K→1、C=16 的最大压力容量验证，同时保留默认 TP32×EP32 全矩阵。
|
||||||
|
|
||||||
**更新（2026-08-18 13:03:33 CST）**：Phase 5 候选镜像改为针对 Kimi 基础镜像原生 SGLang 源码应用最小兼容补丁，仅补齐 SM120 FlashInfer MXFP4 backend 分发、权重布局与 SiTU 激活映射，避免整体替换新版 Python 包造成 `sglang-kernel` 版本错配。
|
**更新（2026-08-18 13:03:33 CST）**：Phase 5 候选镜像改为针对 Kimi 基础镜像原生 SGLang 源码应用最小兼容补丁，仅补齐 SM120 FlashInfer MXFP4 backend 分发、权重布局与 SiTU 激活映射，避免整体替换新版 Python 包造成 `sglang-kernel` 版本错配。
|
||||||
|
|||||||
@ -574,3 +574,103 @@ phase4_ncu_2026_2_1_ab_gpu6_raw.csv
|
|||||||
```text
|
```text
|
||||||
/Users/hzy/Desktop/infra/.phase3_work/phase4_results/
|
/Users/hzy/Desktop/infra/.phase3_work/phase4_results/
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Phase 5 四节点真实 Serving
|
||||||
|
|
||||||
|
状态：**完成。** 本阶段在 601-604 共 32 张 RTX PRO 6000D 上验证
|
||||||
|
TP32/EP4 Kimi-K3，使用真实 SGLang Scheduler、Chunked Prefill、CUDA
|
||||||
|
Graph、四节点 NCCL/RDMA 和 HTTP benchmark。
|
||||||
|
|
||||||
|
### 正式矩阵
|
||||||
|
|
||||||
|
```text
|
||||||
|
ISL=16384, OSL=1, requests=40
|
||||||
|
TP32, EP4, DP1
|
||||||
|
backend=marlin / flashinfer_mxfp4
|
||||||
|
chunked_prefill_size=8192 / 16384
|
||||||
|
concurrency=8 / 16
|
||||||
|
3 repeats per point
|
||||||
|
```
|
||||||
|
|
||||||
|
Prefix/Radix cache 已禁用；client 使用 `random-ids` 保证精确输入长度，且
|
||||||
|
tokenizer 从本地模型目录读取。24/24 repeat 均完成 40/40 请求，errors 为空。
|
||||||
|
|
||||||
|
### 三次重复中位数
|
||||||
|
|
||||||
|
| Chunk | C | Backend | Input TPS | TTFT P50 | TTFT P95 |
|
||||||
|
|---:|---:|---|---:|---:|---:|
|
||||||
|
| 8K | 8 | Marlin | 2935.02 | 43.51 s | 46.22 s |
|
||||||
|
| 8K | 8 | FlashInfer | 3257.96 | 39.19 s | 41.62 s |
|
||||||
|
| 8K | 16 | Marlin | 2934.90 | 87.09 s | 89.86 s |
|
||||||
|
| 8K | 16 | FlashInfer | 3260.14 | 78.39 s | 80.88 s |
|
||||||
|
| 16K | 8 | Marlin | 2812.91 | 45.47 s | 45.86 s |
|
||||||
|
| 16K | 8 | FlashInfer | 3027.50 | 42.27 s | 42.63 s |
|
||||||
|
| 16K | 16 | Marlin | 2812.33 | 90.97 s | 91.36 s |
|
||||||
|
| 16K | 16 | FlashInfer | 3027.34 | 84.52 s | 84.85 s |
|
||||||
|
|
||||||
|
FlashInfer 在四个场景均胜过 Marlin：8K chunk 的 Input TPS 提升约 11%，
|
||||||
|
TTFT P50 降低约 10%；16K chunk 的收益约为 +7.6% / -7.1%。同一后端内，
|
||||||
|
8K chunk 也优于 16K。因此本工作负载的推荐组合是：
|
||||||
|
|
||||||
|
```text
|
||||||
|
EP4 + flashinfer_mxfp4 + chunked_prefill_size=8192
|
||||||
|
```
|
||||||
|
|
||||||
|
已有 EP32 Marlin/8K/C8 两次结果仅作同口径对照：EP4 相对 EP32 的 Input
|
||||||
|
TPS +15.94%，TTFT P50 -13.88%。按要求没有补跑 EP32。
|
||||||
|
|
||||||
|
### 生产路径修复
|
||||||
|
|
||||||
|
真实服务额外发现并修复：旧 Kimi 镜像缺少通用 `gemm1_beta` 字段、
|
||||||
|
Prefill hidden states 可能非连续、benchmark tokenizer 必须固定本地路径。
|
||||||
|
这些问题均在正式矩阵前复现，并由最终 24 点结果覆盖。
|
||||||
|
|
||||||
|
### 证据
|
||||||
|
|
||||||
|
服务器原始目录：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/results/kimi3-ep4-moe-full-20260818-151349/
|
||||||
|
/data/hzy/kimi3-ep4-moe-full-20260818-151349.log
|
||||||
|
```
|
||||||
|
|
||||||
|
仓库内汇总：
|
||||||
|
|
||||||
|
```text
|
||||||
|
results/phase5_ep4_moe_backend_prefill_summary.json
|
||||||
|
results/phase5_ep4_moe_backend_prefill_results.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
日志扫描未发现 OOM、Traceback、NCCL error 或 EngineDeadError。实验结束后
|
||||||
|
601-604 的容器全部删除，32 张 GPU 均回到 0 MiB / 0% utilization。
|
||||||
|
|
||||||
|
### PR 级回归
|
||||||
|
|
||||||
|
真实 serving 发现 `mxfp8_quantize` 要求连续输入，因此 SGLang runner 在量化
|
||||||
|
边界显式 materialize contiguous tensor，并把原 Kimi test 改为构造非连续
|
||||||
|
输入。GPU6 定向测试结果：
|
||||||
|
|
||||||
|
```text
|
||||||
|
1 passed, 5 deselected in 20.22s
|
||||||
|
```
|
||||||
|
|
||||||
|
FlashInfer 官方 `tests/moe/test_trtllm_cutlass_fused_moe.py` 增加 SiTU 的
|
||||||
|
MXFP8×MXFP4 参数组，覆盖 FP16/BF16 与 autotune on/off：
|
||||||
|
|
||||||
|
```text
|
||||||
|
4 passed, 153 deselected in 8.50s
|
||||||
|
```
|
||||||
|
|
||||||
|
候选提交：
|
||||||
|
|
||||||
|
| 仓库 | 提交 | 内容 |
|
||||||
|
|---|---|---|
|
||||||
|
| SGLang | `8a2d501` | 非连续输入修复与回归 |
|
||||||
|
| FlashInfer | `e0bc653` | 官方 SiTU MXFP8×MXFP4 测试 |
|
||||||
|
|
||||||
|
服务器日志：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/data/hzy/phase5_sglang_noncontiguous_gpu6_v2.log
|
||||||
|
/data/hzy/phase5_flashinfer_official_situ_gpu6.log
|
||||||
|
```
|
||||||
|
|||||||
@ -2,6 +2,76 @@
|
|||||||
|
|
||||||
本文件记录可恢复的关键决策、失败尝试和证据路径。成功结论见 `README.md`。
|
本文件记录可恢复的关键决策、失败尝试和证据路径。成功结论见 `README.md`。
|
||||||
|
|
||||||
|
## 2026-08-18 Phase 5
|
||||||
|
|
||||||
|
### 真实 serving 暴露的兼容问题
|
||||||
|
|
||||||
|
- Kimi 基线镜像早于通用 `MoeRunnerConfig.gemm1_beta` 字段，首次
|
||||||
|
FlashInfer EP4 启动在权重处理阶段抛出 `AttributeError`。镜像兼容补丁
|
||||||
|
改为 `getattr(..., "gemm1_beta", None)`，SiTU 的 beta 仍从 Kimi 已有的
|
||||||
|
`gemm1_clamp_limit=25` 读取；没有改变新版本源码的通用字段语义。
|
||||||
|
- 第二次启动完成权重和 KV cache 初始化后，autotune 收到非连续 Prefill
|
||||||
|
hidden states，`mxfp8_quantize` 抛出 `RuntimeError: input must be contiguous`。
|
||||||
|
修复位于 FlashInfer runner 边界，在量化前执行 `x.contiguous()`；正式
|
||||||
|
24 点矩阵验证了该路径。
|
||||||
|
- benchmark 容器最初尝试访问 Hugging Face。入口增加
|
||||||
|
`--tokenizer /data/hf_models/Kimi-K3`，正式矩阵完全使用本地 tokenizer。
|
||||||
|
|
||||||
|
### EP4 正式矩阵
|
||||||
|
|
||||||
|
- Run ID：`kimi3-ep4-moe-full-20260818-151349`。
|
||||||
|
- 口径：TP32/EP4/DP1，16K 输入、1 token 输出、40 requests，
|
||||||
|
Marlin/FlashInfer MXFP4，chunk 8K/16K，C8/C16，每点三次。
|
||||||
|
- 结果：24/24 raw JSONL 均为 40/40 成功且 errors 为空；服务与 bench 日志
|
||||||
|
未发现 OOM、Traceback、NCCL error 或 EngineDeadError。
|
||||||
|
- 8K chunk：FlashInfer Input TPS 提升 11.00%-11.08%，TTFT P50 降低
|
||||||
|
9.92%-9.99%。
|
||||||
|
- 16K chunk：FlashInfer Input TPS 提升 7.63%-7.65%，TTFT P50 降低
|
||||||
|
7.04%-7.09%。
|
||||||
|
- 决策：该 16K/1 Prefill 工作负载采用 EP4 + FlashInfer MXFP4 + 8K chunk。
|
||||||
|
|
||||||
|
### EP32 证据处理
|
||||||
|
|
||||||
|
- 不补跑 EP32。仅复用已有 `kimi3-moe-prefill-20260818-130900` 中
|
||||||
|
Marlin/8K/C8 两次结果。
|
||||||
|
- EP32 中位数为 Input TPS 2531.43、TTFT P50 50.52 s；EP4 三次中位数为
|
||||||
|
2935.02、43.51 s，即 EP4 吞吐 +15.94%、TTFT P50 -13.88%。
|
||||||
|
- EP32 只有两次重复，因此只作为方向性同口径对照，不替代 EP4 正式矩阵。
|
||||||
|
|
||||||
|
### 原始证据与清理
|
||||||
|
|
||||||
|
```text
|
||||||
|
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/results/kimi3-ep4-moe-full-20260818-151349/
|
||||||
|
/data/hzy/kimi3-ep4-moe-full-20260818-151349.log
|
||||||
|
```
|
||||||
|
|
||||||
|
仓库内保留小型汇总：
|
||||||
|
|
||||||
|
```text
|
||||||
|
results/phase5_ep4_moe_backend_prefill_summary.json
|
||||||
|
results/phase5_ep4_moe_backend_prefill_results.csv
|
||||||
|
```
|
||||||
|
|
||||||
|
运行结束后 601-604 无实验容器，32 张 GPU 均为 0 MiB / 0% utilization。
|
||||||
|
|
||||||
|
### PR 回归补充
|
||||||
|
|
||||||
|
- SGLang runner 增加非连续输入回归。第一次测试把
|
||||||
|
`expected=torch.empty_like(noncontiguous_x)` 也创建成非连续 stride，底层
|
||||||
|
kernel 按连续输出 ABI 写入后被测试错误解读，导致数值错位。reference
|
||||||
|
输出改为 `memory_format=torch.contiguous_format` 后通过。
|
||||||
|
- SGLang 最终结果：`1 passed, 5 deselected`，提交 `8a2d501`。
|
||||||
|
- FlashInfer 官方 `test_trtllm_cutlass_fused_moe.py` 增加 SiTU
|
||||||
|
`(alpha=4, beta=25)` 的 MXFP8×MXFP4 case，覆盖 FP16/BF16 和 autotune
|
||||||
|
on/off；最终 `4 passed, 153 deselected`，提交 `e0bc653`。
|
||||||
|
- 证据日志：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/data/hzy/phase5_sglang_noncontiguous_gpu6.log
|
||||||
|
/data/hzy/phase5_sglang_noncontiguous_gpu6_v2.log
|
||||||
|
/data/hzy/phase5_flashinfer_official_situ_gpu6.log
|
||||||
|
```
|
||||||
|
|
||||||
## 2026-08-18 Phase 4
|
## 2026-08-18 Phase 4
|
||||||
|
|
||||||
### 对称 A/B
|
### 对称 A/B
|
||||||
|
|||||||
@ -0,0 +1,25 @@
|
|||||||
|
backend,chunked_prefill_size,concurrency,repeat,completed,duration,request_throughput,input_throughput,output_throughput,total_throughput,mean_ttft_ms,median_ttft_ms,p95_ttft_ms,p99_ttft_ms,mean_e2e_latency_ms,median_e2e_latency_ms,p95_e2e_latency_ms,p99_e2e_latency_ms
|
||||||
|
flashinfer_mxfp4,16384,16,1,40,216.50833019288257,0.1847503971988739,3026.95050770635,0.1847503971988739,3027.135258103549,72316.50786341052,84522.44199160486,84848.25946678175,91778.32209909335,72316.55178854708,84522.48549158685,84848.3054406708,91778.3653298742
|
||||||
|
flashinfer_mxfp4,16384,16,2,40,216.24680145387538,0.1849738342073552,3030.6112996533075,0.1849738342073552,3030.796273487515,72215.42132570758,84481.09910136554,84765.86677560116,91561.01098880405,72215.46733585419,84481.14997893572,84765.91166217111,91561.05765777174
|
||||||
|
flashinfer_mxfp4,16384,16,3,40,216.48059168993495,0.18477406998818618,3027.3383626864425,0.18477406998818618,3027.5231367564306,72312.7285073162,84540.76406499371,84853.01167063879,91778.99107307894,72312.77419594117,84540.81119748298,84853.05667960782,91779.0367784421
|
||||||
|
flashinfer_mxfp4,16384,8,1,40,216.53949848609045,0.18472380456985968,3026.514814072581,0.18472380456985968,3026.6995378771508,40339.51893080957,42272.18415902462,42653.667874005616,49590.42487496044,40339.55953667755,42272.22537947819,42653.713031823274,49590.473274949465
|
||||||
|
flashinfer_mxfp4,16384,8,2,40,216.4690438960679,0.18478392697666732,3027.4998595857173,0.18478392697666732,3027.684643512694,40325.29723601765,42272.719990462065,42599.55736350965,49530.63742398982,40325.33475726959,42272.74997299537,42599.60809347685,49530.707661700435
|
||||||
|
flashinfer_mxfp4,16384,8,3,40,216.46678952896036,0.18478585138644807,3027.531389115565,0.18478585138644807,3027.7161749669513,40328.56577691855,42263.43939546496,42628.12673794104,49566.40897058882,40328.609929577215,42263.489623437636,42628.16974768647,49566.45701683592
|
||||||
|
flashinfer_mxfp4,8192,16,1,40,201.0955079239793,0.1989104600741321,3258.9489778545803,0.1989104600741321,3259.147888314654,66464.03402339201,78428.61395492218,80900.29283260228,82248.80441807676,66464.07695321832,78428.6568548996,80900.33666997915,82248.85000464972
|
||||||
|
flashinfer_mxfp4,8192,16,2,40,201.02179174800403,0.1989834020091862,3260.144058518507,0.1989834020091862,3260.343041920516,66441.70982562937,78389.55687300768,80875.61333409976,82222.40279711317,66441.75368165597,78389.59934201557,80875.65720523708,82222.44497414678
|
||||||
|
flashinfer_mxfp4,8192,16,3,40,200.99849134706892,0.19900646881439046,3260.5219850549734,0.19900646881439046,3260.7209915237877,66426.99547338416,78381.78810151294,80856.50636442006,82182.5564461993,66427.0412968297,78381.83325249702,80856.55405281577,82182.60281610303
|
||||||
|
flashinfer_mxfp4,8192,8,1,40,201.29355681990273,0.19871475586169896,3255.7425600380757,0.19871475586169896,3255.9412747939377,37166.8226861686,39200.42008545715,41681.43365625292,43097.31771401828,37166.86569513404,39200.45927807223,41681.47955433233,43097.35930965515
|
||||||
|
flashinfer_mxfp4,8192,8,2,40,201.1566553460434,0.1988499954485188,3257.958325428532,0.1988499954485188,3258.157175423981,37137.24017466302,39192.07067997195,41617.94732188573,43050.63308592187,37137.28631501435,39192.11111799814,41617.99226633739,43050.676568434574
|
||||||
|
flashinfer_mxfp4,8192,8,3,40,201.02496831398457,0.198980257703726,3260.0925422178466,0.198980257703726,3260.29152247555,37112.54617974046,39172.589769004844,41604.221663821954,43020.16220120946,37112.590136541985,39172.621404984966,41604.265929351095,43020.20668042591
|
||||||
|
marlin,16384,16,1,40,232.999846650986,0.17167393272973525,2812.7057138439823,0.17167393272973525,2812.877387776712,77838.70012289844,90956.05235511903,91350.91751561266,98844.17919690021,77838.74413453159,90956.10008307267,91350.963561167,98844.22676438
|
||||||
|
marlin,16384,16,2,40,233.0310456538573,0.17165094842948833,2812.329139068737,0.17165094842948833,2812.5007900171663,77849.1269534803,90990.29910599347,91359.9561718176,98847.1738615795,77849.17135268915,90990.34174648114,91360.00232317018,98847.21923549892
|
||||||
|
marlin,16384,16,3,40,233.0339416109491,0.1716488152905216,2812.2941897199057,0.1716488152905216,2812.4658385351963,77849.66164607904,90972.34335646499,91361.77453174023,98851.06839299435,77849.70525740064,90972.38598857075,91361.8199098506,98851.1173977633
|
||||||
|
marlin,16384,8,1,40,233.02748973597772,0.17165356776284363,2812.37205422643,0.17165356776284363,2812.543707794193,43413.6843872373,45488.352137384936,45877.185943827484,53396.961687700816,43413.72910180944,45488.39849594515,45877.235028997515,53397.00680534588
|
||||||
|
marlin,16384,8,2,40,232.74746951600537,0.17186008545304213,2815.755640062642,0.17186008545304213,2815.927500148095,43359.47861029417,45474.66833144426,45731.592441652894,53163.52174516069,43359.524680202594,45474.7117330553,45731.640505709205,53163.57911798171
|
||||||
|
marlin,16384,8,3,40,232.98255807999521,0.1716866718677966,2812.9144318819795,0.1716866718677966,2813.086118553847,43401.67168469634,45470.05465638358,45861.754696990814,53373.858650114846,43401.71520576696,45470.09659849573,45861.829353275236,53373.90170200961
|
||||||
|
marlin,8192,16,1,40,223.29927809908986,0.17913179272460458,2934.8952919999215,0.17913179272460458,2935.074423792646,73815.6138905033,87082.98856834881,89829.93190474808,91355.88917817222,73815.65661169589,87083.03174446337,89829.9770831829,91355.93388687819
|
||||||
|
marlin,8192,16,2,40,223.2892873159144,0.17913980773921837,2935.0266099993537,0.17913980773921837,2935.205749807093,73812.15487574227,87086.07016585302,89856.62238476798,91390.91669618152,73812.19822114217,87086.11152344383,89856.66677479167,91390.96168792807
|
||||||
|
marlin,8192,16,3,40,223.33232814306393,0.1791052837383063,2934.4609687684106,0.1791052837383063,2934.640074052149,73832.87409407203,87089.25215096679,89883.80955031607,91406.37799005723,73832.91684102733,87089.2927980749,89883.8516859687,91406.42066749977
|
||||||
|
marlin,8192,8,1,40,223.2894644900225,0.17913966559666036,2935.0242811356834,0.17913966559666036,2935.20342080128,41227.18363396125,43507.58656300604,46237.23470707191,47776.81458188454,41227.227372379275,43507.62654503342,46237.28018605616,47776.86155497795
|
||||||
|
marlin,8192,8,2,40,223.2271411139518,0.17918968007380884,2935.843718329284,0.17918968007380884,2936.0229080093577,41215.50619005575,43508.31371441018,46201.26801881706,47751.084343302064,41215.55230935919,43508.363836910576,46201.314437552355,47751.12928038696
|
||||||
|
marlin,8192,8,3,40,223.2904916331172,0.17913884154871654,2935.0107799341718,0.17913884154871654,2935.189918775721,41230.31144623528,43496.72723049298,46221.18599918904,47788.806532013696,41230.35565857426,43496.76744209137,46221.24284039019,47788.8507199334
|
||||||
|
@ -0,0 +1,211 @@
|
|||||||
|
{
|
||||||
|
"run_id": "kimi3-ep4-moe-full-20260818-151349",
|
||||||
|
"summary": [
|
||||||
|
{
|
||||||
|
"backend": "flashinfer_mxfp4",
|
||||||
|
"chunked_prefill_size": 8192,
|
||||||
|
"concurrency": 8,
|
||||||
|
"repeats": 3,
|
||||||
|
"completed_each": [
|
||||||
|
40,
|
||||||
|
40,
|
||||||
|
40
|
||||||
|
],
|
||||||
|
"median_request_throughput": 0.1988499954485188,
|
||||||
|
"median_input_throughput": 3257.958325428532,
|
||||||
|
"median_total_throughput": 3258.157175423981,
|
||||||
|
"median_mean_ttft_ms": 37137.24017466302,
|
||||||
|
"median_median_ttft_ms": 39192.07067997195,
|
||||||
|
"median_p95_ttft_ms": 41617.94732188573,
|
||||||
|
"median_p99_ttft_ms": 43050.63308592187,
|
||||||
|
"median_mean_e2e_latency_ms": 37137.28631501435,
|
||||||
|
"median_median_e2e_latency_ms": 39192.11111799814,
|
||||||
|
"median_p95_e2e_latency_ms": 41617.99226633739
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"backend": "flashinfer_mxfp4",
|
||||||
|
"chunked_prefill_size": 8192,
|
||||||
|
"concurrency": 16,
|
||||||
|
"repeats": 3,
|
||||||
|
"completed_each": [
|
||||||
|
40,
|
||||||
|
40,
|
||||||
|
40
|
||||||
|
],
|
||||||
|
"median_request_throughput": 0.1989834020091862,
|
||||||
|
"median_input_throughput": 3260.144058518507,
|
||||||
|
"median_total_throughput": 3260.343041920516,
|
||||||
|
"median_mean_ttft_ms": 66441.70982562937,
|
||||||
|
"median_median_ttft_ms": 78389.55687300768,
|
||||||
|
"median_p95_ttft_ms": 80875.61333409976,
|
||||||
|
"median_p99_ttft_ms": 82222.40279711317,
|
||||||
|
"median_mean_e2e_latency_ms": 66441.75368165597,
|
||||||
|
"median_median_e2e_latency_ms": 78389.59934201557,
|
||||||
|
"median_p95_e2e_latency_ms": 80875.65720523708
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"backend": "flashinfer_mxfp4",
|
||||||
|
"chunked_prefill_size": 16384,
|
||||||
|
"concurrency": 8,
|
||||||
|
"repeats": 3,
|
||||||
|
"completed_each": [
|
||||||
|
40,
|
||||||
|
40,
|
||||||
|
40
|
||||||
|
],
|
||||||
|
"median_request_throughput": 0.18478392697666732,
|
||||||
|
"median_input_throughput": 3027.4998595857173,
|
||||||
|
"median_total_throughput": 3027.684643512694,
|
||||||
|
"median_mean_ttft_ms": 40328.56577691855,
|
||||||
|
"median_median_ttft_ms": 42272.18415902462,
|
||||||
|
"median_p95_ttft_ms": 42628.12673794104,
|
||||||
|
"median_p99_ttft_ms": 49566.40897058882,
|
||||||
|
"median_mean_e2e_latency_ms": 40328.609929577215,
|
||||||
|
"median_median_e2e_latency_ms": 42272.22537947819,
|
||||||
|
"median_p95_e2e_latency_ms": 42628.16974768647
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"backend": "flashinfer_mxfp4",
|
||||||
|
"chunked_prefill_size": 16384,
|
||||||
|
"concurrency": 16,
|
||||||
|
"repeats": 3,
|
||||||
|
"completed_each": [
|
||||||
|
40,
|
||||||
|
40,
|
||||||
|
40
|
||||||
|
],
|
||||||
|
"median_request_throughput": 0.18477406998818618,
|
||||||
|
"median_input_throughput": 3027.3383626864425,
|
||||||
|
"median_total_throughput": 3027.5231367564306,
|
||||||
|
"median_mean_ttft_ms": 72312.7285073162,
|
||||||
|
"median_median_ttft_ms": 84522.44199160486,
|
||||||
|
"median_p95_ttft_ms": 84848.25946678175,
|
||||||
|
"median_p99_ttft_ms": 91778.32209909335,
|
||||||
|
"median_mean_e2e_latency_ms": 72312.77419594117,
|
||||||
|
"median_median_e2e_latency_ms": 84522.48549158685,
|
||||||
|
"median_p95_e2e_latency_ms": 84848.3054406708
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"backend": "marlin",
|
||||||
|
"chunked_prefill_size": 8192,
|
||||||
|
"concurrency": 8,
|
||||||
|
"repeats": 3,
|
||||||
|
"completed_each": [
|
||||||
|
40,
|
||||||
|
40,
|
||||||
|
40
|
||||||
|
],
|
||||||
|
"median_request_throughput": 0.17913966559666036,
|
||||||
|
"median_input_throughput": 2935.0242811356834,
|
||||||
|
"median_total_throughput": 2935.20342080128,
|
||||||
|
"median_mean_ttft_ms": 41227.18363396125,
|
||||||
|
"median_median_ttft_ms": 43507.58656300604,
|
||||||
|
"median_p95_ttft_ms": 46221.18599918904,
|
||||||
|
"median_p99_ttft_ms": 47776.81458188454,
|
||||||
|
"median_mean_e2e_latency_ms": 41227.227372379275,
|
||||||
|
"median_median_e2e_latency_ms": 43507.62654503342,
|
||||||
|
"median_p95_e2e_latency_ms": 46221.24284039019
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"backend": "marlin",
|
||||||
|
"chunked_prefill_size": 8192,
|
||||||
|
"concurrency": 16,
|
||||||
|
"repeats": 3,
|
||||||
|
"completed_each": [
|
||||||
|
40,
|
||||||
|
40,
|
||||||
|
40
|
||||||
|
],
|
||||||
|
"median_request_throughput": 0.17913179272460458,
|
||||||
|
"median_input_throughput": 2934.8952919999215,
|
||||||
|
"median_total_throughput": 2935.074423792646,
|
||||||
|
"median_mean_ttft_ms": 73815.6138905033,
|
||||||
|
"median_median_ttft_ms": 87086.07016585302,
|
||||||
|
"median_p95_ttft_ms": 89856.62238476798,
|
||||||
|
"median_p99_ttft_ms": 91390.91669618152,
|
||||||
|
"median_mean_e2e_latency_ms": 73815.65661169589,
|
||||||
|
"median_median_e2e_latency_ms": 87086.11152344383,
|
||||||
|
"median_p95_e2e_latency_ms": 89856.66677479167
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"backend": "marlin",
|
||||||
|
"chunked_prefill_size": 16384,
|
||||||
|
"concurrency": 8,
|
||||||
|
"repeats": 3,
|
||||||
|
"completed_each": [
|
||||||
|
40,
|
||||||
|
40,
|
||||||
|
40
|
||||||
|
],
|
||||||
|
"median_request_throughput": 0.1716866718677966,
|
||||||
|
"median_input_throughput": 2812.9144318819795,
|
||||||
|
"median_total_throughput": 2813.086118553847,
|
||||||
|
"median_mean_ttft_ms": 43401.67168469634,
|
||||||
|
"median_median_ttft_ms": 45474.66833144426,
|
||||||
|
"median_p95_ttft_ms": 45861.754696990814,
|
||||||
|
"median_p99_ttft_ms": 53373.858650114846,
|
||||||
|
"median_mean_e2e_latency_ms": 43401.71520576696,
|
||||||
|
"median_median_e2e_latency_ms": 45474.7117330553,
|
||||||
|
"median_p95_e2e_latency_ms": 45861.829353275236
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"backend": "marlin",
|
||||||
|
"chunked_prefill_size": 16384,
|
||||||
|
"concurrency": 16,
|
||||||
|
"repeats": 3,
|
||||||
|
"completed_each": [
|
||||||
|
40,
|
||||||
|
40,
|
||||||
|
40
|
||||||
|
],
|
||||||
|
"median_request_throughput": 0.17165094842948833,
|
||||||
|
"median_input_throughput": 2812.329139068737,
|
||||||
|
"median_total_throughput": 2812.5007900171663,
|
||||||
|
"median_mean_ttft_ms": 77849.1269534803,
|
||||||
|
"median_median_ttft_ms": 90972.34335646499,
|
||||||
|
"median_p95_ttft_ms": 91359.9561718176,
|
||||||
|
"median_p99_ttft_ms": 98847.1738615795,
|
||||||
|
"median_mean_e2e_latency_ms": 77849.17135268915,
|
||||||
|
"median_median_e2e_latency_ms": 90972.38598857075,
|
||||||
|
"median_p95_e2e_latency_ms": 91360.00232317018
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"comparisons": [
|
||||||
|
{
|
||||||
|
"chunked_prefill_size": 8192,
|
||||||
|
"concurrency": 8,
|
||||||
|
"flashinfer_input_throughput_change_pct": 11.002772493857949,
|
||||||
|
"flashinfer_total_throughput_change_pct": 11.002772493857949,
|
||||||
|
"flashinfer_median_ttft_ms_change_pct": -9.918996257778002,
|
||||||
|
"flashinfer_p95_ttft_ms_change_pct": -9.959153097854456,
|
||||||
|
"flashinfer_median_e2e_latency_ms_change_pct": -9.918986094468796
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"chunked_prefill_size": 8192,
|
||||||
|
"concurrency": 16,
|
||||||
|
"flashinfer_input_throughput_change_pct": 11.082125055880665,
|
||||||
|
"flashinfer_total_throughput_change_pct": 11.082125055880665,
|
||||||
|
"flashinfer_median_ttft_ms_change_pct": -9.986112906786438,
|
||||||
|
"flashinfer_p95_ttft_ms_change_pct": -9.994821541601407,
|
||||||
|
"flashinfer_median_e2e_latency_ms_change_pct": -9.98610688810826
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"chunked_prefill_size": 16384,
|
||||||
|
"concurrency": 8,
|
||||||
|
"flashinfer_input_throughput_change_pct": 7.628579997727458,
|
||||||
|
"flashinfer_total_throughput_change_pct": 7.62857999772748,
|
||||||
|
"flashinfer_median_ttft_ms_change_pct": -7.042347508899216,
|
||||||
|
"flashinfer_p95_ttft_ms_change_pct": -7.050816045775821,
|
||||||
|
"flashinfer_median_e2e_latency_ms_change_pct": -7.0423455840166405
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"chunked_prefill_size": 16384,
|
||||||
|
"concurrency": 16,
|
||||||
|
"flashinfer_input_throughput_change_pct": 7.645236847665093,
|
||||||
|
"flashinfer_total_throughput_change_pct": 7.645236847665093,
|
||||||
|
"flashinfer_median_ttft_ms_change_pct": -7.089958471869751,
|
||||||
|
"flashinfer_p95_ttft_ms_change_pct": -7.127517325850641,
|
||||||
|
"flashinfer_median_e2e_latency_ms_change_pct": -7.089954195324976
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@ -1,4 +1,4 @@
|
|||||||
# Kimi-K3 / Pro6000D / SGLang TP32 EP32 MoE Backend Prefill
|
# Kimi-K3 / Pro6000D / SGLang TP32 EP4 MoE Backend Prefill
|
||||||
|
|
||||||
This experiment is the real-serving acceptance gate for Kimi-K3 FlashInfer
|
This experiment is the real-serving acceptance gate for Kimi-K3 FlashInfer
|
||||||
MXFP4 MoE on NVIDIA SM120. It does not modify the existing baseline experiment.
|
MXFP4 MoE on NVIDIA SM120. It does not modify the existing baseline experiment.
|
||||||
@ -8,7 +8,7 @@ MXFP4 MoE on NVIDIA SM120. It does not modify the existing baseline experiment.
|
|||||||
| Dimension | Values |
|
| Dimension | Values |
|
||||||
|---|---|
|
|---|---|
|
||||||
| Nodes | 601-604, 8 GPUs per node |
|
| Nodes | 601-604, 8 GPUs per node |
|
||||||
| Parallelism | TP32, EP32, DP1 |
|
| Parallelism | TP32, EP4, DP1 |
|
||||||
| MoE runner | `marlin`, `flashinfer_mxfp4` |
|
| MoE runner | `marlin`, `flashinfer_mxfp4` |
|
||||||
| Input / output | 16K / 1 token |
|
| Input / output | 16K / 1 token |
|
||||||
| Concurrency | 8, 16 |
|
| Concurrency | 8, 16 |
|
||||||
@ -16,7 +16,10 @@ MXFP4 MoE on NVIDIA SM120. It does not modify the existing baseline experiment.
|
|||||||
| Requests | 40 per repeat |
|
| Requests | 40 per repeat |
|
||||||
| Repeats | 3 |
|
| Repeats | 3 |
|
||||||
|
|
||||||
The eight configurations are identical except for MoE runner and chunk size.
|
The eight configurations are identical except for MoE runner, chunk size, and
|
||||||
|
benchmark concurrency. The directory retains its historical `tp32ep32` name so
|
||||||
|
existing evidence paths remain stable; the production comparison and the
|
||||||
|
default `EP_SIZE` are EP4.
|
||||||
The client uses SGLang `random-ids`, which produces the exact synthetic token
|
The client uses SGLang `random-ids`, which produces the exact synthetic token
|
||||||
length without depending on a ShareGPT file or text truncation. Radix cache is
|
length without depending on a ShareGPT file or text truncation. Radix cache is
|
||||||
disabled. `OSL=1` makes TTFT, E2E latency and input TPS the primary metrics;
|
disabled. `OSL=1` makes TTFT, E2E latency and input TPS the primary metrics;
|
||||||
@ -27,6 +30,35 @@ The defaults above can be narrowed for a capacity-first run with `EP_SIZE`,
|
|||||||
example, the TP32/EP4 maximum-pressure probe uses Marlin, 16K chunk, and C=16
|
example, the TP32/EP4 maximum-pressure probe uses Marlin, 16K chunk, and C=16
|
||||||
before spending time on the complete backend comparison.
|
before spending time on the complete backend comparison.
|
||||||
|
|
||||||
|
## Accepted EP4 result
|
||||||
|
|
||||||
|
Run ID: `kimi3-ep4-moe-full-20260818-151349`.
|
||||||
|
|
||||||
|
All 24 repeat files completed 40/40 requests with empty error arrays. No OOM,
|
||||||
|
Traceback, NCCL error, or EngineDeadError was found. Values below are medians
|
||||||
|
across three repeats.
|
||||||
|
|
||||||
|
| Chunk | C | Backend | Input TPS | TTFT P50 | TTFT P95 |
|
||||||
|
|---:|---:|---|---:|---:|---:|
|
||||||
|
| 8K | 8 | Marlin | 2935.02 | 43.51 s | 46.22 s |
|
||||||
|
| 8K | 8 | FlashInfer MXFP4 | 3257.96 | 39.19 s | 41.62 s |
|
||||||
|
| 8K | 16 | Marlin | 2934.90 | 87.09 s | 89.86 s |
|
||||||
|
| 8K | 16 | FlashInfer MXFP4 | 3260.14 | 78.39 s | 80.88 s |
|
||||||
|
| 16K | 8 | Marlin | 2812.91 | 45.47 s | 45.86 s |
|
||||||
|
| 16K | 8 | FlashInfer MXFP4 | 3027.50 | 42.27 s | 42.63 s |
|
||||||
|
| 16K | 16 | Marlin | 2812.33 | 90.97 s | 91.36 s |
|
||||||
|
| 16K | 16 | FlashInfer MXFP4 | 3027.34 | 84.52 s | 84.85 s |
|
||||||
|
|
||||||
|
FlashInfer improves Input TPS by 11.00%-11.08% with an 8K chunk and by
|
||||||
|
7.63%-7.65% with a 16K chunk. It reduces median TTFT by 9.92%-9.99% and
|
||||||
|
7.04%-7.09%, respectively. The accepted Prefill setting is therefore
|
||||||
|
`flashinfer_mxfp4`, EP4, and an 8K chunk for this 16K/1 workload.
|
||||||
|
|
||||||
|
The only matching existing EP32 evidence is Marlin/8K/C8 with two repeats.
|
||||||
|
Its median Input TPS is 2531.43 and TTFT P50/P95 are 50.52/53.64 seconds.
|
||||||
|
Against that evidence, EP4 raises Input TPS by 15.94% and lowers TTFT P50 by
|
||||||
|
13.88%. EP32 was not rerun.
|
||||||
|
|
||||||
## Unique entrypoint
|
## Unique entrypoint
|
||||||
|
|
||||||
Run on 601 only. The sudo password is supplied at runtime and is never written
|
Run on 601 only. The sudo password is supplied at runtime and is never written
|
||||||
|
|||||||
@ -10,7 +10,7 @@ HEAD_HOST="${HEAD_HOST:-174.1.60.1}"
|
|||||||
NODE_SSH_USER="${NODE_SSH_USER:-user}"
|
NODE_SSH_USER="${NODE_SSH_USER:-user}"
|
||||||
NODE_HOSTS=(174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4)
|
NODE_HOSTS=(174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4)
|
||||||
DIST_PORT="${DIST_PORT:-20000}"
|
DIST_PORT="${DIST_PORT:-20000}"
|
||||||
EP_SIZE="${EP_SIZE:-32}"
|
EP_SIZE="${EP_SIZE:-4}"
|
||||||
|
|
||||||
BASE_IMAGE="${BASE_IMAGE:-lmsysorg/sglang:kimi-k3-fiv617situ-warm}"
|
BASE_IMAGE="${BASE_IMAGE:-lmsysorg/sglang:kimi-k3-fiv617situ-warm}"
|
||||||
CANDIDATE_IMAGE="${CANDIDATE_IMAGE:-local/sglang:kimi-k3-sm120-flashinfer-mxfp4-phase5}"
|
CANDIDATE_IMAGE="${CANDIDATE_IMAGE:-local/sglang:kimi-k3-sm120-flashinfer-mxfp4-phase5}"
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user