diff --git a/README.md b/README.md index 6f11b1c..6480177 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # sskj — 多平台大模型推理性能基准测试项目 +**更新(2026-08-18 18:38:17 CST)**:完成 Kimi-K3 四节点 TP32×EP4 的真实 Prefill MoE backend 验收;24/24 重复均 40/40 成功,FlashInfer MXFP4 在 8K Chunk、C=8/16 下相对 Marlin 提升约 11% Input TPS、降低约 10% TTFT,并确认 8K 优于 16K。实验默认 EP 改为 4,补齐非连续 Prefill 输入与 FlashInfer 官方 SiTU 回归,详见 `experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/README.md`。 + **更新(2026-08-18 13:55:30 CST)**:Kimi-K3 四节点 MoE backend 验收入口支持通过环境变量覆盖 EP size 及 backend/chunk/concurrency 子矩阵,用于先执行 TP32×EP4、16K Chunk、16K→1、C=16 的最大压力容量验证,同时保留默认 TP32×EP32 全矩阵。 **更新(2026-08-18 13:03:33 CST)**:Phase 5 候选镜像改为针对 Kimi 基础镜像原生 SGLang 源码应用最小兼容补丁,仅补齐 SM120 FlashInfer MXFP4 backend 分发、权重布局与 SiTU 激活映射,避免整体替换新版 Python 包造成 `sglang-kernel` 版本错配。 diff --git a/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/README.md b/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/README.md index 9a8a881..badba6d 100644 --- a/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/README.md +++ b/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/README.md @@ -574,3 +574,103 @@ phase4_ncu_2026_2_1_ab_gpu6_raw.csv ```text /Users/hzy/Desktop/infra/.phase3_work/phase4_results/ ``` + +## Phase 5 四节点真实 Serving + +状态:**完成。** 本阶段在 601-604 共 32 张 RTX PRO 6000D 上验证 +TP32/EP4 Kimi-K3,使用真实 SGLang Scheduler、Chunked Prefill、CUDA +Graph、四节点 NCCL/RDMA 和 HTTP benchmark。 + +### 正式矩阵 + +```text +ISL=16384, OSL=1, requests=40 +TP32, EP4, DP1 +backend=marlin / flashinfer_mxfp4 +chunked_prefill_size=8192 / 16384 +concurrency=8 / 16 +3 repeats per point +``` + +Prefix/Radix cache 已禁用;client 使用 `random-ids` 保证精确输入长度,且 +tokenizer 从本地模型目录读取。24/24 repeat 均完成 40/40 请求,errors 为空。 + +### 三次重复中位数 + +| Chunk | C | Backend | Input TPS | TTFT P50 | TTFT P95 | +|---:|---:|---|---:|---:|---:| +| 8K | 8 | Marlin | 2935.02 | 43.51 s | 46.22 s | +| 8K | 8 | FlashInfer | 3257.96 | 39.19 s | 41.62 s | +| 8K | 16 | Marlin | 2934.90 | 87.09 s | 89.86 s | +| 8K | 16 | FlashInfer | 3260.14 | 78.39 s | 80.88 s | +| 16K | 8 | Marlin | 2812.91 | 45.47 s | 45.86 s | +| 16K | 8 | FlashInfer | 3027.50 | 42.27 s | 42.63 s | +| 16K | 16 | Marlin | 2812.33 | 90.97 s | 91.36 s | +| 16K | 16 | FlashInfer | 3027.34 | 84.52 s | 84.85 s | + +FlashInfer 在四个场景均胜过 Marlin:8K chunk 的 Input TPS 提升约 11%, +TTFT P50 降低约 10%;16K chunk 的收益约为 +7.6% / -7.1%。同一后端内, +8K chunk 也优于 16K。因此本工作负载的推荐组合是: + +```text +EP4 + flashinfer_mxfp4 + chunked_prefill_size=8192 +``` + +已有 EP32 Marlin/8K/C8 两次结果仅作同口径对照:EP4 相对 EP32 的 Input +TPS +15.94%,TTFT P50 -13.88%。按要求没有补跑 EP32。 + +### 生产路径修复 + +真实服务额外发现并修复:旧 Kimi 镜像缺少通用 `gemm1_beta` 字段、 +Prefill hidden states 可能非连续、benchmark tokenizer 必须固定本地路径。 +这些问题均在正式矩阵前复现,并由最终 24 点结果覆盖。 + +### 证据 + +服务器原始目录: + +```text +/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/results/kimi3-ep4-moe-full-20260818-151349/ +/data/hzy/kimi3-ep4-moe-full-20260818-151349.log +``` + +仓库内汇总: + +```text +results/phase5_ep4_moe_backend_prefill_summary.json +results/phase5_ep4_moe_backend_prefill_results.csv +``` + +日志扫描未发现 OOM、Traceback、NCCL error 或 EngineDeadError。实验结束后 +601-604 的容器全部删除,32 张 GPU 均回到 0 MiB / 0% utilization。 + +### PR 级回归 + +真实 serving 发现 `mxfp8_quantize` 要求连续输入,因此 SGLang runner 在量化 +边界显式 materialize contiguous tensor,并把原 Kimi test 改为构造非连续 +输入。GPU6 定向测试结果: + +```text +1 passed, 5 deselected in 20.22s +``` + +FlashInfer 官方 `tests/moe/test_trtllm_cutlass_fused_moe.py` 增加 SiTU 的 +MXFP8×MXFP4 参数组,覆盖 FP16/BF16 与 autotune on/off: + +```text +4 passed, 153 deselected in 8.50s +``` + +候选提交: + +| 仓库 | 提交 | 内容 | +|---|---|---| +| SGLang | `8a2d501` | 非连续输入修复与回归 | +| FlashInfer | `e0bc653` | 官方 SiTU MXFP8×MXFP4 测试 | + +服务器日志: + +```text +/data/hzy/phase5_sglang_noncontiguous_gpu6_v2.log +/data/hzy/phase5_flashinfer_official_situ_gpu6.log +``` diff --git a/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/decision_log.md b/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/decision_log.md index 061c387..6540491 100644 --- a/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/decision_log.md +++ b/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/decision_log.md @@ -2,6 +2,76 @@ 本文件记录可恢复的关键决策、失败尝试和证据路径。成功结论见 `README.md`。 +## 2026-08-18 Phase 5 + +### 真实 serving 暴露的兼容问题 + +- Kimi 基线镜像早于通用 `MoeRunnerConfig.gemm1_beta` 字段,首次 + FlashInfer EP4 启动在权重处理阶段抛出 `AttributeError`。镜像兼容补丁 + 改为 `getattr(..., "gemm1_beta", None)`,SiTU 的 beta 仍从 Kimi 已有的 + `gemm1_clamp_limit=25` 读取;没有改变新版本源码的通用字段语义。 +- 第二次启动完成权重和 KV cache 初始化后,autotune 收到非连续 Prefill + hidden states,`mxfp8_quantize` 抛出 `RuntimeError: input must be contiguous`。 + 修复位于 FlashInfer runner 边界,在量化前执行 `x.contiguous()`;正式 + 24 点矩阵验证了该路径。 +- benchmark 容器最初尝试访问 Hugging Face。入口增加 + `--tokenizer /data/hf_models/Kimi-K3`,正式矩阵完全使用本地 tokenizer。 + +### EP4 正式矩阵 + +- Run ID:`kimi3-ep4-moe-full-20260818-151349`。 +- 口径:TP32/EP4/DP1,16K 输入、1 token 输出、40 requests, + Marlin/FlashInfer MXFP4,chunk 8K/16K,C8/C16,每点三次。 +- 结果:24/24 raw JSONL 均为 40/40 成功且 errors 为空;服务与 bench 日志 + 未发现 OOM、Traceback、NCCL error 或 EngineDeadError。 +- 8K chunk:FlashInfer Input TPS 提升 11.00%-11.08%,TTFT P50 降低 + 9.92%-9.99%。 +- 16K chunk:FlashInfer Input TPS 提升 7.63%-7.65%,TTFT P50 降低 + 7.04%-7.09%。 +- 决策:该 16K/1 Prefill 工作负载采用 EP4 + FlashInfer MXFP4 + 8K chunk。 + +### EP32 证据处理 + +- 不补跑 EP32。仅复用已有 `kimi3-moe-prefill-20260818-130900` 中 + Marlin/8K/C8 两次结果。 +- EP32 中位数为 Input TPS 2531.43、TTFT P50 50.52 s;EP4 三次中位数为 + 2935.02、43.51 s,即 EP4 吞吐 +15.94%、TTFT P50 -13.88%。 +- EP32 只有两次重复,因此只作为方向性同口径对照,不替代 EP4 正式矩阵。 + +### 原始证据与清理 + +```text +/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/results/kimi3-ep4-moe-full-20260818-151349/ +/data/hzy/kimi3-ep4-moe-full-20260818-151349.log +``` + +仓库内保留小型汇总: + +```text +results/phase5_ep4_moe_backend_prefill_summary.json +results/phase5_ep4_moe_backend_prefill_results.csv +``` + +运行结束后 601-604 无实验容器,32 张 GPU 均为 0 MiB / 0% utilization。 + +### PR 回归补充 + +- SGLang runner 增加非连续输入回归。第一次测试把 + `expected=torch.empty_like(noncontiguous_x)` 也创建成非连续 stride,底层 + kernel 按连续输出 ABI 写入后被测试错误解读,导致数值错位。reference + 输出改为 `memory_format=torch.contiguous_format` 后通过。 +- SGLang 最终结果:`1 passed, 5 deselected`,提交 `8a2d501`。 +- FlashInfer 官方 `test_trtllm_cutlass_fused_moe.py` 增加 SiTU + `(alpha=4, beta=25)` 的 MXFP8×MXFP4 case,覆盖 FP16/BF16 和 autotune + on/off;最终 `4 passed, 153 deselected`,提交 `e0bc653`。 +- 证据日志: + +```text +/data/hzy/phase5_sglang_noncontiguous_gpu6.log +/data/hzy/phase5_sglang_noncontiguous_gpu6_v2.log +/data/hzy/phase5_flashinfer_official_situ_gpu6.log +``` + ## 2026-08-18 Phase 4 ### 对称 A/B diff --git a/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/results/phase5_ep4_moe_backend_prefill_results.csv b/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/results/phase5_ep4_moe_backend_prefill_results.csv new file mode 100644 index 0000000..50044c7 --- /dev/null +++ b/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/results/phase5_ep4_moe_backend_prefill_results.csv @@ -0,0 +1,25 @@ +backend,chunked_prefill_size,concurrency,repeat,completed,duration,request_throughput,input_throughput,output_throughput,total_throughput,mean_ttft_ms,median_ttft_ms,p95_ttft_ms,p99_ttft_ms,mean_e2e_latency_ms,median_e2e_latency_ms,p95_e2e_latency_ms,p99_e2e_latency_ms +flashinfer_mxfp4,16384,16,1,40,216.50833019288257,0.1847503971988739,3026.95050770635,0.1847503971988739,3027.135258103549,72316.50786341052,84522.44199160486,84848.25946678175,91778.32209909335,72316.55178854708,84522.48549158685,84848.3054406708,91778.3653298742 +flashinfer_mxfp4,16384,16,2,40,216.24680145387538,0.1849738342073552,3030.6112996533075,0.1849738342073552,3030.796273487515,72215.42132570758,84481.09910136554,84765.86677560116,91561.01098880405,72215.46733585419,84481.14997893572,84765.91166217111,91561.05765777174 +flashinfer_mxfp4,16384,16,3,40,216.48059168993495,0.18477406998818618,3027.3383626864425,0.18477406998818618,3027.5231367564306,72312.7285073162,84540.76406499371,84853.01167063879,91778.99107307894,72312.77419594117,84540.81119748298,84853.05667960782,91779.0367784421 +flashinfer_mxfp4,16384,8,1,40,216.53949848609045,0.18472380456985968,3026.514814072581,0.18472380456985968,3026.6995378771508,40339.51893080957,42272.18415902462,42653.667874005616,49590.42487496044,40339.55953667755,42272.22537947819,42653.713031823274,49590.473274949465 +flashinfer_mxfp4,16384,8,2,40,216.4690438960679,0.18478392697666732,3027.4998595857173,0.18478392697666732,3027.684643512694,40325.29723601765,42272.719990462065,42599.55736350965,49530.63742398982,40325.33475726959,42272.74997299537,42599.60809347685,49530.707661700435 +flashinfer_mxfp4,16384,8,3,40,216.46678952896036,0.18478585138644807,3027.531389115565,0.18478585138644807,3027.7161749669513,40328.56577691855,42263.43939546496,42628.12673794104,49566.40897058882,40328.609929577215,42263.489623437636,42628.16974768647,49566.45701683592 +flashinfer_mxfp4,8192,16,1,40,201.0955079239793,0.1989104600741321,3258.9489778545803,0.1989104600741321,3259.147888314654,66464.03402339201,78428.61395492218,80900.29283260228,82248.80441807676,66464.07695321832,78428.6568548996,80900.33666997915,82248.85000464972 +flashinfer_mxfp4,8192,16,2,40,201.02179174800403,0.1989834020091862,3260.144058518507,0.1989834020091862,3260.343041920516,66441.70982562937,78389.55687300768,80875.61333409976,82222.40279711317,66441.75368165597,78389.59934201557,80875.65720523708,82222.44497414678 +flashinfer_mxfp4,8192,16,3,40,200.99849134706892,0.19900646881439046,3260.5219850549734,0.19900646881439046,3260.7209915237877,66426.99547338416,78381.78810151294,80856.50636442006,82182.5564461993,66427.0412968297,78381.83325249702,80856.55405281577,82182.60281610303 +flashinfer_mxfp4,8192,8,1,40,201.29355681990273,0.19871475586169896,3255.7425600380757,0.19871475586169896,3255.9412747939377,37166.8226861686,39200.42008545715,41681.43365625292,43097.31771401828,37166.86569513404,39200.45927807223,41681.47955433233,43097.35930965515 +flashinfer_mxfp4,8192,8,2,40,201.1566553460434,0.1988499954485188,3257.958325428532,0.1988499954485188,3258.157175423981,37137.24017466302,39192.07067997195,41617.94732188573,43050.63308592187,37137.28631501435,39192.11111799814,41617.99226633739,43050.676568434574 +flashinfer_mxfp4,8192,8,3,40,201.02496831398457,0.198980257703726,3260.0925422178466,0.198980257703726,3260.29152247555,37112.54617974046,39172.589769004844,41604.221663821954,43020.16220120946,37112.590136541985,39172.621404984966,41604.265929351095,43020.20668042591 +marlin,16384,16,1,40,232.999846650986,0.17167393272973525,2812.7057138439823,0.17167393272973525,2812.877387776712,77838.70012289844,90956.05235511903,91350.91751561266,98844.17919690021,77838.74413453159,90956.10008307267,91350.963561167,98844.22676438 +marlin,16384,16,2,40,233.0310456538573,0.17165094842948833,2812.329139068737,0.17165094842948833,2812.5007900171663,77849.1269534803,90990.29910599347,91359.9561718176,98847.1738615795,77849.17135268915,90990.34174648114,91360.00232317018,98847.21923549892 +marlin,16384,16,3,40,233.0339416109491,0.1716488152905216,2812.2941897199057,0.1716488152905216,2812.4658385351963,77849.66164607904,90972.34335646499,91361.77453174023,98851.06839299435,77849.70525740064,90972.38598857075,91361.8199098506,98851.1173977633 +marlin,16384,8,1,40,233.02748973597772,0.17165356776284363,2812.37205422643,0.17165356776284363,2812.543707794193,43413.6843872373,45488.352137384936,45877.185943827484,53396.961687700816,43413.72910180944,45488.39849594515,45877.235028997515,53397.00680534588 +marlin,16384,8,2,40,232.74746951600537,0.17186008545304213,2815.755640062642,0.17186008545304213,2815.927500148095,43359.47861029417,45474.66833144426,45731.592441652894,53163.52174516069,43359.524680202594,45474.7117330553,45731.640505709205,53163.57911798171 +marlin,16384,8,3,40,232.98255807999521,0.1716866718677966,2812.9144318819795,0.1716866718677966,2813.086118553847,43401.67168469634,45470.05465638358,45861.754696990814,53373.858650114846,43401.71520576696,45470.09659849573,45861.829353275236,53373.90170200961 +marlin,8192,16,1,40,223.29927809908986,0.17913179272460458,2934.8952919999215,0.17913179272460458,2935.074423792646,73815.6138905033,87082.98856834881,89829.93190474808,91355.88917817222,73815.65661169589,87083.03174446337,89829.9770831829,91355.93388687819 +marlin,8192,16,2,40,223.2892873159144,0.17913980773921837,2935.0266099993537,0.17913980773921837,2935.205749807093,73812.15487574227,87086.07016585302,89856.62238476798,91390.91669618152,73812.19822114217,87086.11152344383,89856.66677479167,91390.96168792807 +marlin,8192,16,3,40,223.33232814306393,0.1791052837383063,2934.4609687684106,0.1791052837383063,2934.640074052149,73832.87409407203,87089.25215096679,89883.80955031607,91406.37799005723,73832.91684102733,87089.2927980749,89883.8516859687,91406.42066749977 +marlin,8192,8,1,40,223.2894644900225,0.17913966559666036,2935.0242811356834,0.17913966559666036,2935.20342080128,41227.18363396125,43507.58656300604,46237.23470707191,47776.81458188454,41227.227372379275,43507.62654503342,46237.28018605616,47776.86155497795 +marlin,8192,8,2,40,223.2271411139518,0.17918968007380884,2935.843718329284,0.17918968007380884,2936.0229080093577,41215.50619005575,43508.31371441018,46201.26801881706,47751.084343302064,41215.55230935919,43508.363836910576,46201.314437552355,47751.12928038696 +marlin,8192,8,3,40,223.2904916331172,0.17913884154871654,2935.0107799341718,0.17913884154871654,2935.189918775721,41230.31144623528,43496.72723049298,46221.18599918904,47788.806532013696,41230.35565857426,43496.76744209137,46221.24284039019,47788.8507199334 diff --git a/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/results/phase5_ep4_moe_backend_prefill_summary.json b/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/results/phase5_ep4_moe_backend_prefill_summary.json new file mode 100644 index 0000000..ac85d69 --- /dev/null +++ b/experiments/pro6000/kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/results/phase5_ep4_moe_backend_prefill_summary.json @@ -0,0 +1,211 @@ +{ + "run_id": "kimi3-ep4-moe-full-20260818-151349", + "summary": [ + { + "backend": "flashinfer_mxfp4", + "chunked_prefill_size": 8192, + "concurrency": 8, + "repeats": 3, + "completed_each": [ + 40, + 40, + 40 + ], + "median_request_throughput": 0.1988499954485188, + "median_input_throughput": 3257.958325428532, + "median_total_throughput": 3258.157175423981, + "median_mean_ttft_ms": 37137.24017466302, + "median_median_ttft_ms": 39192.07067997195, + "median_p95_ttft_ms": 41617.94732188573, + "median_p99_ttft_ms": 43050.63308592187, + "median_mean_e2e_latency_ms": 37137.28631501435, + "median_median_e2e_latency_ms": 39192.11111799814, + "median_p95_e2e_latency_ms": 41617.99226633739 + }, + { + "backend": "flashinfer_mxfp4", + "chunked_prefill_size": 8192, + "concurrency": 16, + "repeats": 3, + "completed_each": [ + 40, + 40, + 40 + ], + "median_request_throughput": 0.1989834020091862, + "median_input_throughput": 3260.144058518507, + "median_total_throughput": 3260.343041920516, + "median_mean_ttft_ms": 66441.70982562937, + "median_median_ttft_ms": 78389.55687300768, + "median_p95_ttft_ms": 80875.61333409976, + "median_p99_ttft_ms": 82222.40279711317, + "median_mean_e2e_latency_ms": 66441.75368165597, + "median_median_e2e_latency_ms": 78389.59934201557, + "median_p95_e2e_latency_ms": 80875.65720523708 + }, + { + "backend": "flashinfer_mxfp4", + "chunked_prefill_size": 16384, + "concurrency": 8, + "repeats": 3, + "completed_each": [ + 40, + 40, + 40 + ], + "median_request_throughput": 0.18478392697666732, + "median_input_throughput": 3027.4998595857173, + "median_total_throughput": 3027.684643512694, + "median_mean_ttft_ms": 40328.56577691855, + "median_median_ttft_ms": 42272.18415902462, + "median_p95_ttft_ms": 42628.12673794104, + "median_p99_ttft_ms": 49566.40897058882, + "median_mean_e2e_latency_ms": 40328.609929577215, + "median_median_e2e_latency_ms": 42272.22537947819, + "median_p95_e2e_latency_ms": 42628.16974768647 + }, + { + "backend": "flashinfer_mxfp4", + "chunked_prefill_size": 16384, + "concurrency": 16, + "repeats": 3, + "completed_each": [ + 40, + 40, + 40 + ], + "median_request_throughput": 0.18477406998818618, + "median_input_throughput": 3027.3383626864425, + "median_total_throughput": 3027.5231367564306, + "median_mean_ttft_ms": 72312.7285073162, + "median_median_ttft_ms": 84522.44199160486, + "median_p95_ttft_ms": 84848.25946678175, + "median_p99_ttft_ms": 91778.32209909335, + "median_mean_e2e_latency_ms": 72312.77419594117, + "median_median_e2e_latency_ms": 84522.48549158685, + "median_p95_e2e_latency_ms": 84848.3054406708 + }, + { + "backend": "marlin", + "chunked_prefill_size": 8192, + "concurrency": 8, + "repeats": 3, + "completed_each": [ + 40, + 40, + 40 + ], + "median_request_throughput": 0.17913966559666036, + "median_input_throughput": 2935.0242811356834, + "median_total_throughput": 2935.20342080128, + "median_mean_ttft_ms": 41227.18363396125, + "median_median_ttft_ms": 43507.58656300604, + "median_p95_ttft_ms": 46221.18599918904, + "median_p99_ttft_ms": 47776.81458188454, + "median_mean_e2e_latency_ms": 41227.227372379275, + "median_median_e2e_latency_ms": 43507.62654503342, + "median_p95_e2e_latency_ms": 46221.24284039019 + }, + { + "backend": "marlin", + "chunked_prefill_size": 8192, + "concurrency": 16, + "repeats": 3, + "completed_each": [ + 40, + 40, + 40 + ], + "median_request_throughput": 0.17913179272460458, + "median_input_throughput": 2934.8952919999215, + "median_total_throughput": 2935.074423792646, + "median_mean_ttft_ms": 73815.6138905033, + "median_median_ttft_ms": 87086.07016585302, + "median_p95_ttft_ms": 89856.62238476798, + "median_p99_ttft_ms": 91390.91669618152, + "median_mean_e2e_latency_ms": 73815.65661169589, + "median_median_e2e_latency_ms": 87086.11152344383, + "median_p95_e2e_latency_ms": 89856.66677479167 + }, + { + "backend": "marlin", + "chunked_prefill_size": 16384, + "concurrency": 8, + "repeats": 3, + "completed_each": [ + 40, + 40, + 40 + ], + "median_request_throughput": 0.1716866718677966, + "median_input_throughput": 2812.9144318819795, + "median_total_throughput": 2813.086118553847, + "median_mean_ttft_ms": 43401.67168469634, + "median_median_ttft_ms": 45474.66833144426, + "median_p95_ttft_ms": 45861.754696990814, + "median_p99_ttft_ms": 53373.858650114846, + "median_mean_e2e_latency_ms": 43401.71520576696, + "median_median_e2e_latency_ms": 45474.7117330553, + "median_p95_e2e_latency_ms": 45861.829353275236 + }, + { + "backend": "marlin", + "chunked_prefill_size": 16384, + "concurrency": 16, + "repeats": 3, + "completed_each": [ + 40, + 40, + 40 + ], + "median_request_throughput": 0.17165094842948833, + "median_input_throughput": 2812.329139068737, + "median_total_throughput": 2812.5007900171663, + "median_mean_ttft_ms": 77849.1269534803, + "median_median_ttft_ms": 90972.34335646499, + "median_p95_ttft_ms": 91359.9561718176, + "median_p99_ttft_ms": 98847.1738615795, + "median_mean_e2e_latency_ms": 77849.17135268915, + "median_median_e2e_latency_ms": 90972.38598857075, + "median_p95_e2e_latency_ms": 91360.00232317018 + } + ], + "comparisons": [ + { + "chunked_prefill_size": 8192, + "concurrency": 8, + "flashinfer_input_throughput_change_pct": 11.002772493857949, + "flashinfer_total_throughput_change_pct": 11.002772493857949, + "flashinfer_median_ttft_ms_change_pct": -9.918996257778002, + "flashinfer_p95_ttft_ms_change_pct": -9.959153097854456, + "flashinfer_median_e2e_latency_ms_change_pct": -9.918986094468796 + }, + { + "chunked_prefill_size": 8192, + "concurrency": 16, + "flashinfer_input_throughput_change_pct": 11.082125055880665, + "flashinfer_total_throughput_change_pct": 11.082125055880665, + "flashinfer_median_ttft_ms_change_pct": -9.986112906786438, + "flashinfer_p95_ttft_ms_change_pct": -9.994821541601407, + "flashinfer_median_e2e_latency_ms_change_pct": -9.98610688810826 + }, + { + "chunked_prefill_size": 16384, + "concurrency": 8, + "flashinfer_input_throughput_change_pct": 7.628579997727458, + "flashinfer_total_throughput_change_pct": 7.62857999772748, + "flashinfer_median_ttft_ms_change_pct": -7.042347508899216, + "flashinfer_p95_ttft_ms_change_pct": -7.050816045775821, + "flashinfer_median_e2e_latency_ms_change_pct": -7.0423455840166405 + }, + { + "chunked_prefill_size": 16384, + "concurrency": 16, + "flashinfer_input_throughput_change_pct": 7.645236847665093, + "flashinfer_total_throughput_change_pct": 7.645236847665093, + "flashinfer_median_ttft_ms_change_pct": -7.089958471869751, + "flashinfer_p95_ttft_ms_change_pct": -7.127517325850641, + "flashinfer_median_e2e_latency_ms_change_pct": -7.089954195324976 + } + ] +} \ No newline at end of file diff --git a/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/README.md b/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/README.md index 151eda1..d99800c 100644 --- a/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/README.md +++ b/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/README.md @@ -1,4 +1,4 @@ -# Kimi-K3 / Pro6000D / SGLang TP32 EP32 MoE Backend Prefill +# Kimi-K3 / Pro6000D / SGLang TP32 EP4 MoE Backend Prefill This experiment is the real-serving acceptance gate for Kimi-K3 FlashInfer MXFP4 MoE on NVIDIA SM120. It does not modify the existing baseline experiment. @@ -8,7 +8,7 @@ MXFP4 MoE on NVIDIA SM120. It does not modify the existing baseline experiment. | Dimension | Values | |---|---| | Nodes | 601-604, 8 GPUs per node | -| Parallelism | TP32, EP32, DP1 | +| Parallelism | TP32, EP4, DP1 | | MoE runner | `marlin`, `flashinfer_mxfp4` | | Input / output | 16K / 1 token | | Concurrency | 8, 16 | @@ -16,7 +16,10 @@ MXFP4 MoE on NVIDIA SM120. It does not modify the existing baseline experiment. | Requests | 40 per repeat | | Repeats | 3 | -The eight configurations are identical except for MoE runner and chunk size. +The eight configurations are identical except for MoE runner, chunk size, and +benchmark concurrency. The directory retains its historical `tp32ep32` name so +existing evidence paths remain stable; the production comparison and the +default `EP_SIZE` are EP4. The client uses SGLang `random-ids`, which produces the exact synthetic token length without depending on a ShareGPT file or text truncation. Radix cache is disabled. `OSL=1` makes TTFT, E2E latency and input TPS the primary metrics; @@ -27,6 +30,35 @@ The defaults above can be narrowed for a capacity-first run with `EP_SIZE`, example, the TP32/EP4 maximum-pressure probe uses Marlin, 16K chunk, and C=16 before spending time on the complete backend comparison. +## Accepted EP4 result + +Run ID: `kimi3-ep4-moe-full-20260818-151349`. + +All 24 repeat files completed 40/40 requests with empty error arrays. No OOM, +Traceback, NCCL error, or EngineDeadError was found. Values below are medians +across three repeats. + +| Chunk | C | Backend | Input TPS | TTFT P50 | TTFT P95 | +|---:|---:|---|---:|---:|---:| +| 8K | 8 | Marlin | 2935.02 | 43.51 s | 46.22 s | +| 8K | 8 | FlashInfer MXFP4 | 3257.96 | 39.19 s | 41.62 s | +| 8K | 16 | Marlin | 2934.90 | 87.09 s | 89.86 s | +| 8K | 16 | FlashInfer MXFP4 | 3260.14 | 78.39 s | 80.88 s | +| 16K | 8 | Marlin | 2812.91 | 45.47 s | 45.86 s | +| 16K | 8 | FlashInfer MXFP4 | 3027.50 | 42.27 s | 42.63 s | +| 16K | 16 | Marlin | 2812.33 | 90.97 s | 91.36 s | +| 16K | 16 | FlashInfer MXFP4 | 3027.34 | 84.52 s | 84.85 s | + +FlashInfer improves Input TPS by 11.00%-11.08% with an 8K chunk and by +7.63%-7.65% with a 16K chunk. It reduces median TTFT by 9.92%-9.99% and +7.04%-7.09%, respectively. The accepted Prefill setting is therefore +`flashinfer_mxfp4`, EP4, and an 8K chunk for this 16K/1 workload. + +The only matching existing EP32 evidence is Marlin/8K/C8 with two repeats. +Its median Input TPS is 2531.43 and TTFT P50/P95 are 50.52/53.64 seconds. +Against that evidence, EP4 raises Input TPS by 15.94% and lowers TTFT P50 by +13.88%. EP32 was not rerun. + ## Unique entrypoint Run on 601 only. The sudo password is supplied at runtime and is never written diff --git a/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/config.env b/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/config.env index a70c281..07c8e9b 100644 --- a/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/config.env +++ b/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill/config.env @@ -10,7 +10,7 @@ HEAD_HOST="${HEAD_HOST:-174.1.60.1}" NODE_SSH_USER="${NODE_SSH_USER:-user}" NODE_HOSTS=(174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4) DIST_PORT="${DIST_PORT:-20000}" -EP_SIZE="${EP_SIZE:-32}" +EP_SIZE="${EP_SIZE:-4}" BASE_IMAGE="${BASE_IMAGE:-lmsysorg/sglang:kimi-k3-fiv617situ-warm}" CANDIDATE_IMAGE="${CANDIDATE_IMAGE:-local/sglang:kimi-k3-sm120-flashinfer-mxfp4-phase5}"