Compare commits
38 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
97c208cfdb | ||
|
|
b9e8c0c4a2 | ||
|
|
2370d33c73 | ||
|
|
c5d91ceafe | ||
|
|
d063bd63c9 | ||
|
|
21fcca5d16 | ||
|
|
e3476aff86 | ||
|
|
f15003d2f5 | ||
|
|
ad1853f49b | ||
|
|
0d929948a5 | ||
|
|
92517e87f0 | ||
|
|
498e333730 | ||
|
|
0c89fd4fbe | ||
|
|
4d6dac010c | ||
|
|
911a9a2fb0 | ||
|
|
a3b8f1b2ab | ||
|
|
ffda226f5a | ||
|
|
123023b6ae | ||
|
|
5c749cda03 | ||
|
|
b3165a1d3c | ||
|
|
f0ab17c561 | ||
| 9cdbc1fd22 | |||
| ba20973bea | |||
| eeec56c2cf | |||
|
|
987f1db4b0 | ||
|
|
c69831f258 | ||
|
|
fe375e1307 | ||
|
|
ddf807d458 | ||
|
|
da0e1b4372 | ||
|
|
04dfa31583 | ||
|
|
d72dbff689 | ||
|
|
3bd04698bb | ||
|
|
d9a2e3b5e5 | ||
|
|
f33c5f1d3d | ||
|
|
ea8302561e | ||
|
|
6ba04325d3 | ||
|
|
9acf9fdfdb | ||
|
|
3761d75b00 |
14
.gitignore
vendored
14
.gitignore
vendored
@ -49,6 +49,9 @@ experiments/**/runtime/
|
||||
# 无关项目
|
||||
loomeval_yy/
|
||||
|
||||
# 部署层运行时产物
|
||||
deploy/logs/
|
||||
|
||||
.gitignore
|
||||
|
||||
dsv4_dspark_h20_sglang_tp_dp_matrix
|
||||
@ -73,14 +76,22 @@ build/
|
||||
sglang_sm120_cache/
|
||||
vllm_sm120_cache/
|
||||
*_sm120_cache/
|
||||
sglang_nightly_cu13_cache/
|
||||
|
||||
# 一次性 bench 输出
|
||||
bench-output/
|
||||
|
||||
# 单数 dataset/（可重新下载，与 datasets/ 同源）
|
||||
dataset/
|
||||
|
||||
# 逐请求原始日志（体积大；汇总见 results.json / report.md）
|
||||
experiments/**/raw_outputs/
|
||||
**/dummy_sharegpt.json
|
||||
|
||||
# 统一 bench CLI 的一次性测试输出（ops_ 前缀 run_id）
|
||||
experiments/pro6000/*/results/ops_*/
|
||||
experiments/910c/*/results/ops_*/
|
||||
|
||||
# Virtual environments (keep only docs under envs/)
|
||||
envs/*
|
||||
!envs/README.md
|
||||
@ -93,3 +104,6 @@ skills-lock.json
|
||||
|
||||
# 临时备份文件
|
||||
*.bak.glm52orig
|
||||
*.bak
|
||||
*.bak.*
|
||||
*.tmp
|
||||
|
||||
11
README.md
11
README.md
@ -44,6 +44,8 @@
|
||||
| `experiments/p800/dsv4_p800_sglang_tp_dp_matrix/` | P800 + SGLang（INT8；TP2/DP4 启动 OOM 无数据，见 config.env 注释） |
|
||||
| `experiments/pro6000/dsv4_pro6000_vllm_tp_dp_matrix/` | RTX 6000D + vLLM |
|
||||
| `experiments/pro6000/dsv4_pro6000_sglang_tp_dp_matrix/` | RTX 6000D + SGLang |
|
||||
| `experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/` | RTX 6000D + SGLang，Kimi-K3（TP32×EP32，部署手册见 docs/KIMI_K3_DEPLOY.md） |
|
||||
| `experiments/pro6000/kimi3_pro6000_pd_rdma/` | RTX 6000D + SGLang，Kimi-K3 PD 分离（MoonCake RDMA，8 节点，见 deploy_pd.sh + docs/KIMI_K3_DEPLOY.md 附录 B） |
|
||||
|
||||
每个目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_concurrency.sh` 从 C=1 指数倍增搜饱和点；`run_adaptive_concurrency_add16.sh` 从 C=16 线性 +16 步进、带 TTFT SLO 停止与回退（当前主力用法，见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`）。
|
||||
|
||||
@ -58,6 +60,7 @@
|
||||
| `experiments/h200/dsv4_h200_vllm_tp{2,4,8}_custom_bench/` | 自定义压测客户端（多服务负载均衡） |
|
||||
| `experiments/h200/dsv4_h200_max_context_length/` `…/dsv4_h200_long_context_matrix/` `…/dsv4_h200_256k_4k_probe/` `…/dsv4_h200_64k_sglang_vs_vllm/` | 长上下文专项 |
|
||||
| `experiments/p800/dsv4_p800_sglang/` `…/dsv4_p800_max_context_length/` `…/dsv4_p800_long_context_matrix/` `…/dsv4_p800_256k_4k_probe/` | P800 baseline 与长上下文专项 |
|
||||
| `experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/` | RTX 6000D + SGLang，GLM-5.3-NVFP4 双场景压测标准（16k/512 高并发 + 128k/64k 90% 命中低并发；真实语料 run-id 窗口纪律 + 三套部署配置 A/B/CAR，基线与口径见目录 README；部署 profile 见 `deploy/profiles/pro6000/glm53_nvfp4_*`） |
|
||||
| `experiments/TEMPLATE/` | 老式固定场景实验模板 |
|
||||
|
||||
## 快速复现
|
||||
@ -92,6 +95,14 @@ tmux new-session -d -s h20-vllm-adaptive \
|
||||
- 各平台的镜像 / venv / 模型路径见对应实验目录的 `config.env` 与 `platforms/<chip>.env`；**模型与数据集路径是机器相关的**，换机器时按实际路径调整。
|
||||
- 环境搭建规范见 `envs/README.md` 与 `envs/UV_ENV_SETUP.md`。
|
||||
|
||||
## 运维与部署分层
|
||||
|
||||
- 模型团队用 `deploy/profiles/<platform>/<model>-<engine>.env` 与
|
||||
`python -m sskj.deploy start|stop|status` 管理服务生命周期。
|
||||
- 运维只跑测试：`python -m sskj.bench run --url http://<host>:<port> --platform <platform> --experiment <name> --profile smoke|matrix`。
|
||||
- 运维手册见 [`ops/README.md`](ops/README.md)，统一测试层复用现有
|
||||
`scripts/common/parse_backend.py` 等指标口径。
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 提交内容：实验代码 + 最终产物（results.json / report.md / adaptive 的 jsonl 与 summary）；日志、raw_outputs、gpu_logs 不入库（`.gitignore` 已排除）。
|
||||
|
||||
35
deploy/CURRENT.md
Normal file
35
deploy/CURRENT.md
Normal file
@ -0,0 +1,35 @@
|
||||
# 现役部署状态页（live 核验于 2026-09-10，60.8 当日核验；其余机器 09-09 口径）
|
||||
|
||||
> 本页记录 174.1.60.x 集群"现在跑的是什么"。改动机器前后先读这页；
|
||||
> **任何变动（停/起/改配置）完成后必须更新本页**。核验命令：`docker ps` + `nvidia-smi`。
|
||||
|
||||
## 机器状态（2026-09-09 实测）
|
||||
|
||||
| 机器 | 在役 | 口径 / 归属 | 对应 profile |
|
||||
|---|---|---|---|
|
||||
| 60.1 (6000D-1) | `glm53-pp4`（Up，8 卡满载，:30000） | **方案 D 生产**。09-09/10 两轮实验窗口停机（凌晨 D 基线战役 ~7h + 日间 A 基线战役 ~9.5h）均恢复核验（health+生成冒烟）。**D 配置升级路径已交付未执行**：latest 镜像+开 autotune+持久缓存，16k/512 cc8-64 五点 +1.2~3.2%（A/B/A 确认），profile `..._tp2pp4_latest_autotune.env`——**仅适用 D 配置**：09-10 A 基线（TP8+EAGLE）战役证明 latest 与 autotune 在 A 配置上均负收益（eager 路径对代差/ tactic 敏感，图内不敏感），A 配置维持 nightly+autotune off；**IPC@TP8 真增益 +4.6~9.1% 已验证**（vs 同镜像基线全点胜、cc8/cc64 分布不重叠，收益随 TP 度单调升：TP2 −3.5%→TP8 +9.1%）但补丁包仅兼容 latest（sglang 4 文件依赖新版 runtime_context API），net vs 生产≈0；落地=移植回 nightly 或等上游 PR #34528 合入，未执行。同轮 A 基线画像：生产 A 掉图陷阱主导场景二（cc16+ 全程 eager，TPOT 168-197ms，平台 ~70-73；已发表 95/96 是 A-s2 图覆盖 bs16 变体）；日内漂移=eager 点 8.5h 后 −6~8% 而图内点不动（CPU 侧签名）。实验全量 `experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/`（REPORT.md + REPORT_A_BASELINE.md）。`dsv4_scan` 经授权保持停止，还原说明 `/root/dsv4_scan_restore_note.txt` | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp2pp4.env`（现役）；升级候选 `..._tp2pp4_latest_autotune.env`（仅 D 配置） |
|
||||
| 60.2 (6000D-2) | `glm53-nvfp4` 实验容器（09-08 晚 TP1PP8 phase，run_phase.sh 实验链进行中） | **他人实验进行中，勿动**（方案 F PD 链已拆除；GPU7 曾有外部裸金属任务）。动卡前仍先核实归属 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_{prefill→decode 侧}.env` |
|
||||
| 60.3 | 无容器，但 8 卡被外部裸金属训练占用（`/data/mas/larm`，09-08 晚实测） | 外部任务，勿动（此前台账漏记） | — |
|
||||
| 60.4 | `glm53-nvfp4`（Up，:30000，restart=unless-stopped） | **TP8+EAGLE+custom-AR 1stage（E7b 配方）在役**（09-09 下午部署：EAGLE 4/1/5/mem0.90/MRR16/chunk8192/ctxlen270336/fp8KV+hicache3/decode 图桶 1-8/双 parser；CAR 补丁三处全注入、8 rank `SSKJ_CAR_PATCH_ACTIVE` 确认，health 200、生成冒烟、质量门 7/7 见 `/root/qg_604_car.log`）。启动 `/root/deploy_glm53_604_exp.sh`（=60.7 实验版逐字拷贝，`RESTART=yes CAR_PATCH=1`），补丁 `/root/patches/`（md5 与仓库 car_patch 归档一致）。当日早间曾短暂部署 TP2PP4 D 配方复刻（deploy_s2_test_604.sh 留盘可切回）后被本方案替换；同日经授权清退外部 vllm 评测流水线（tmux `mas` 的 run_multiseed.sh 链，--resume 可续跑） | `experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/`（deploy_glm53_604_exp.sh + car_patch/ 补丁快照） |
|
||||
| 60.5 | `glm53-nvfp4`（Up 2d，09-09 只读核验） | **NVFP4 团队生产**（deploy_glm53_605.sh，md5 fcd9109b）。生产机铁律：不实验、不重启、不覆盖脚本。**交付升级路径已更新为 v3（09-09 hit90 场景优胜 TP4PP2-hicache@0.90，池 647,040/c4 并发独立文档/hit90 cc8 out +34% vs 现役）：脚本 `experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/scripts/deploy_glm53_605_v3.sh` + 补丁束（60.8:/root/glm53_r37_patch_bundle_v3.tar.gz，md5 6922e534，需 scp 至 60.5）+ 需 docker pull cu13 镜像；未执行、未落 60.5 磁盘（60.5:/root 仅有原脚本，核验过）。此前 v2（TP2PP4-hicache，冷缓存口径优胜）被 v3 取代，仍留仓库可作"容量优先 6 条文档/512k 单条最快"备选；回滚=原脚本** | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp8eagle.env`（方案 A 口径）；待切 `glm53_nvfp4_pro6000_sglang_tp4pp2_hicache.env`（容量优先备选 `..._tp2pp4_hicache.env`） |
|
||||
| 60.6 | 无容器，但 8 卡被外部裸金属实验占用（`/data/hzy/sparse-opd-*`，09-08 晚实测） | 外部任务，勿动（此前台账漏记） | — |
|
||||
| 60.7 | 基本空（4 卡仍有 `/home/user/dirA_exp` 外部小任务，09-08 晚实测） | 09-08 已拆除清空（方案 F 前身单机实验 + 场景一深优资产留盘），不再恢复 | — |
|
||||
| 60.8 | `glm53-nvfp4`（Up，:30000，restart=unless-stopped） | **TP4PP2-nomtp@0.90 hit90 优胜口径在役**（09-09 晚接替同日早间的 TP2PP4@0.85：同 r37 栈 cu13+9 挂载+de-GLOO，仅改 tp4/pp2 + memfrac 0.90；KV 池 **647,040**；实测实例原样保留切 restart 策略，未重建容器。启动 `bash /root/deploy_ppmtp_r37.sh '--tp 4 --pp-size 2 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length 1048576' nomtp 8192 0.90 1 1 0 0`）。hit90（90% 命中 i128k/o512）out cc1/2/3/4/8/16 = **28.8/47.5/63.9/76.2/106.4/125.7**（vs TP2PP4 基线 cc4 +14%/cc8 +15%）；并发独立 128k 文档 **4 条**（cap cc4 98.5 零排队）、512k 单条 ✓ TTFT 135.8s、质量门 **7/7**；同轮判决：DP attention 容量负收益判死、DCP 对 DSA 静默算错禁用、MTP@128k accept 2.07 判负。让步项（知情）：容量 4 条（TP2PP4 为 6）、512k 单条慢 33%。回滚 TP2PP4 = 同命令改 `'--tp 2 --pp-size 4'` + memfrac 0.85。**09-10 B300 对标战役**：停役（容器 rename 保全 `glm53-nvfp4-insvc`）→ 双臂 B300 场景矩阵（TP2PP4-D 口径 24 点 + E7b 配方）→ E7b 高并发调参复测（初测 MRR16+图1-8 掉图断崖 → MRR64+图桶 1-64，`deploy_glm53_e7b_hicc.sh`，三场景 c8-c64 共 10 点全 OK；报告正文采用复测值：掉图断崖已修复、16K 仍 TP2PP4 占优（E7b 被 prefill 墙封 ~100 tok/s 平台）、decode 密集 1K→4K E7b 全档反超（c64 out 826.5 tok/s 全场最高、超 TP2PP4 72%）、边界仅 TP2PP4 可达、与 B300 绝对差 4-5×；报告飞书 wiki `A7V3wZTQeifCB4krdi6cA834nW9`，已整文更新）→ TP2PP4 高并发调参复测（初测 C=64 受 MRR48 封顶 48 活跃+16 排队、decode 图默认覆盖 bs≤256 从未掉图 → MRR64，`deploy_glm53_pp4_mrr64.sh` 逐 token 仅改 MRR；三场景 c32/c64 共 6 点两轮对齐 ≤1.8%（pass-1 + v2 有序版）+ 全新实例 MRR48 对照归因；正文采用 v2 值：4.1/4.2 c64 out +27%/+19%（433/572）、三场景 c64 TTFT P95 全塌缩（20.0/337.9/134.8s → 7.2/7.3/89.2s）、16K c64 到池顶 ~59 活跃 + 3 次回退（确定性行为）、1K 短输入 c32 付 −14% MRR64 配置代价（276→266→228 = −3% 新鲜度 −14% 配置，三实例可复现、机制未定位）、16K c32 无代价；报告已再次整文更新）→ **原容器恢复并三次核验**（rename 回 + start，fired up/health 200/16K 抽测 ok、KV 池 647,040 与启动口径逐字一致、显存水位 77.2/82.3 GiB 与停役前一致） | `profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp4pp2_hicache.env`；实验全量 `experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/`；B300 对标 `experiments/pro6000/glm53_nvfp4_b300_equivalent_matrix/`；前史 `experiments/pro6000/glm53_nvfp4_128k_capacity_topology/`（TP2PP4 口径）+ r37 `experiments/pro6000/glm53_ppmtp_r37_verify_graph/` |
|
||||
|
||||
## 方案 A-F 一览（GLM-5.3-NVFP4 @ pro6000，2026-09-08 双场景报告口径）
|
||||
|
||||
| 方案 | 一句话 | profile / 脚本 |
|
||||
|---|---|---|
|
||||
| A | TP8 + EAGLE，基准；60.5 生产口径 | `glm53_nvfp4_pro6000_sglang_tp8eagle.env` |
|
||||
| B | TP4 PP2 | `glm53_nvfp4_pro6000_sglang_tp4pp2.env` |
|
||||
| C | TP4 PP2 + IndexCache(freq=4) | `glm53_nvfp4_pro6000_sglang_tp4pp2_index.env` |
|
||||
| D | TP2 PP4；60.1 生产在役 | `glm53_nvfp4_pro6000_sglang_tp2pp4.env` |
|
||||
| E | TP8 + DFlash2 投机 | `glm53_nvfp4_pro6000_sglang_tp8dflash2.env` |
|
||||
| F | PD 分离双机链（mc-master→prefill→decode→router） | `glm53_nvfp4_pro6000_pd_{master,prefill,decode,router}.env` + `deploy/PD_CHAIN.md` |
|
||||
|
||||
压测数据：飞书《GLM-5.3-NVFP4 双场景压测报告》（`SZUSdEqY1oRVxGxgILBcHqPJnEc`）。
|
||||
|
||||
## 防漂移
|
||||
|
||||
每个在役容器用 `deploy/verify_profile.sh <profile.env>` 定期核验（镜像 digest + 启动参数 +
|
||||
端口），发现不一致 = 容器被人手改过，先查清归属再处理。
|
||||
96
deploy/PD_CHAIN.md
Normal file
96
deploy/PD_CHAIN.md
Normal file
@ -0,0 +1,96 @@
|
||||
# 方案F：GLM-5.3-NVFP4 PD 分离完整链（双机 6000D）编排手册
|
||||
|
||||
> 2026-09-08 实测终态。四角色、两台机、启动顺序强制。吞吐换算口径：**链合计 ÷2 = 单机等效**（与单机方案 A-E 可比）。
|
||||
> 完整压测数据见飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 `SZUSdEqY1oRVxGxgILBcHqPJnEc` / wiki `NzbMwzmKviYidRkZYRrc8GYQnrf`）。
|
||||
|
||||
## 拓扑
|
||||
|
||||
| 角色 | 机器 | 容器 | 端口 | profile |
|
||||
|---|---|---|---|---|
|
||||
| 1. mc-master（Mooncake 元数据） | 174.1.60.1 | `mc-master` | 50051 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_master.env` |
|
||||
| 2. prefill（TP4 PP2 + DFLASH 草稿） | 174.1.60.1 | `glm53-pd-smoke-prefill` | 30000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_prefill.env` |
|
||||
| 3. decode（TP8 + DFLASH v5 配方） | 174.1.60.2 | `glm53-s1-decode` | 30000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_decode.env` |
|
||||
| 4. router（MiniLB） | 174.1.60.2 | `pd-smoke-router` | 31000 | `profiles/pro6000/glm53_nvfp4_pro6000_pd_router.env` |
|
||||
|
||||
镜像统一：`lmsysorg/sglang:nightly-dev-20260828-daf63171`
|
||||
（digest `sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399`）。
|
||||
|
||||
## 启动顺序（强制）
|
||||
|
||||
```
|
||||
mc-master (60.1) → prefill (60.1) → decode (60.2) → router (60.2)
|
||||
```
|
||||
|
||||
对应脚本（`experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/`）：
|
||||
|
||||
```bash
|
||||
# 60.1（注意：60.1 日常跑生产容器 glm53-pp4，先停它，见下节）
|
||||
bash /tmp/deploy_pd_smoke_master.sh # 判据 ss :50051
|
||||
bash /tmp/deploy_pd_probe.sh p1b_prefill_tp4pp2_cps8k_radix_launch.sh # 判据 :30000/health 200，~10-15min
|
||||
# 60.2
|
||||
bash /tmp/deploy_s1_decode.sh # 判据 :30000/health 200，~10min
|
||||
bash /tmp/deploy_pd_smoke_router.sh # 判据 ss :31000
|
||||
```
|
||||
|
||||
场景一（90% 命中）必须用 **radix 变体** launch（`p1b_prefill_tp4pp2_cps8k_radix_launch.sh`，
|
||||
与基线唯一差异 = 无 `--disable-radix-cache`）。场景二 0 命中，radix 开销可忽略，两场景共用同一部署。
|
||||
|
||||
## 基础设施依赖（缺一不可）
|
||||
|
||||
| 依赖 | 位置 | 说明 |
|
||||
|---|---|---|
|
||||
| sglang 补丁树 | `/data/sglang_patch_glm53` → 容器 `/sgl-workspace/sglang` | 与镜像原版差 **11 个文件**（10 改 + 1 新增 `dflash_pp.py`），清单与 diff 见 `platforms/patches/pro6000/glm53_pd_chain/`。无补丁则 DFlash+PD 冷启动接线缺失，decode 首请求 400 |
|
||||
| mooncake wheel | `/data/flashkda_deploy/wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl` | 容器内 launch 脚本 `pip install --no-deps` 自装 |
|
||||
| DFlash2 草稿权重 | `/data/hf_models/GLM-5.3-DFlash2` | 两角色都要挂 |
|
||||
| IB 设备 | mlx5_0–mlx5_3 + `--device /dev/infiniband` + `--ulimit memlock=-1` | RDMA KV 传输通道 |
|
||||
| 环境变量 | `MOONCAKE_MASTER=174.1.60.1:50051` `MOONCAKE_PROTOCOL=rdma` | prefill/decode 容器都要 |
|
||||
|
||||
## 质量门（经 router :31000，全链路含 KV transfer + 草稿）
|
||||
|
||||
60.2 上生成 router 口径的 gate 副本：
|
||||
|
||||
```bash
|
||||
sed 's/PORT=30000/PORT=31000/' /root/quality_gate_605.sh > /tmp/quality_gate_31000.sh
|
||||
```
|
||||
|
||||
判据：GSM8K×5 + 中文推理必须过。tool call 预期不过（链配方无 parser，与方案 B/D 同口径的已知配置缺口，记录放行）。
|
||||
2026-09-08 实测 **6/7**（核心项全过，DFlash 草稿无质量损失）。
|
||||
|
||||
## 压测
|
||||
|
||||
驱动脚本：`run_pd_s1.sh` / `run_pd_s2.sh`（同 scripts 目录），跑在 60.1（语料本地，
|
||||
`--url http://174.1.60.2:31000/generate` 走 router，`--container glm53-pd-smoke-prefill` 读 prefill 日志核命中率）。
|
||||
|
||||
- **场景一**：128k/64k × cc1-4，nreq8，shared-frac 0.9，canonical 窗口 9301-9308
|
||||
- **场景二**：cc8/16/32 canonical 9311-9313（nreq 16/32/32）；cc40/64 全新窗口 pool-override
|
||||
16,384,000 / 17,203,200（nreq 40/64 单轮满波）
|
||||
- 指标读取：命中率读 prefill 容器日志（PP2 日志计数 ×2，不影响比值）；accept 从 decode 日志读
|
||||
（router 转发后 usage 字段可能缺失）
|
||||
- 每点核验：ok/failed=0、retractions、s1 hit≈0.90、s2 hit=0
|
||||
|
||||
### 2026-09-08 实测判决（13/13 干净点）
|
||||
|
||||
- **场景一**：128k cc1 TTFT 3.91s = 六方案最低（跨机重叠把 prefill 与 RDMA 传输完全藏住），
|
||||
但 cc≥2 时 decode 侧排队，TTFT 堆到 31.0s；÷2 输出吞吐 21.2-23.5 tok/s，仅为 A 的 30-57%。
|
||||
根因：decode KV 池 214,336 token 按 16k 场景定容 → 128k 仅 1 驻留、64k 仅 3。
|
||||
- **场景二**：÷2 单机等效 74-85 tok/s = A 的 75-96%，但 TTFT 较 A 减半。
|
||||
判决：**两机买 TTFT、不买吞吐**。decode MRR12 在 cc16+ 饱和；DFlash accept 在 12-batch
|
||||
verify 下掉到 1.8（单机 EAGLE 2.84）。
|
||||
- **PD 机制验证**：跨机 prefill/decode 重叠成立、mooncake RDMA 传输被吸收——机制无罪，
|
||||
容量配置（decode 池按 16k 定容）是判决主因。
|
||||
|
||||
## 拆链与生产恢复（顺序固定）
|
||||
|
||||
```bash
|
||||
# 1. 拆链
|
||||
# 60.1: docker rm -f mc-master glm53-pd-smoke-prefill
|
||||
# 60.2: docker rm -f glm53-s1-decode pd-smoke-router
|
||||
# 2. 等双机显存排空（60.1 <2000MiB；60.2 GPU7 常有外部裸金属任务 main_v2.py，核实归属勿清）
|
||||
# 3. 恢复 60.1 生产：bash /tmp/deploy_glm53_pp4.sh → health 200 + docker inspect Args 核对
|
||||
```
|
||||
|
||||
## 60.1 生产停机窗口提示
|
||||
|
||||
60.1 日常跑生产 `glm53-pp4`（方案 D，TP2PP4，8 卡满载）。拉 prefill 角色前必须停生产；
|
||||
`deploy_glm53_pp4.sh` 生成的参数与生产容器 `docker inspect` Args 已核对逐字一致，
|
||||
恢复即完全复现。全程停机 ~2-2.5h（部署+13 点压测+质量门+恢复）。
|
||||
4
deploy/manifests/GLM-5.3-DFlash2.md5
Normal file
4
deploy/manifests/GLM-5.3-DFlash2.md5
Normal file
@ -0,0 +1,4 @@
|
||||
302f15789d891aa39620209286252fed ./README.md
|
||||
ce1962554abf73c6ad953aa0ac3f20d7 ./config.json
|
||||
5e170425c97cda8f798c74041979a569 ./configuration.json
|
||||
32c81842f12e56e6ac2a1feaafd5bfa7 ./model.safetensors
|
||||
58
deploy/manifests/GLM-5.3-NVFP4.md5
Normal file
58
deploy/manifests/GLM-5.3-NVFP4.md5
Normal file
@ -0,0 +1,58 @@
|
||||
434f8692a4914f39714df2c4493c92d7 ./.gitattributes
|
||||
245254422e0ca504a5f09e37719ca718 ./LICENSE
|
||||
02e0c207304666abdd93de6a3780c52a ./README.md
|
||||
e4c9e1000a513f680dd796f76d1097e0 ./chat_template.jinja
|
||||
d42bba3f99ce9b2705bb19c41930ea0c ./config.json
|
||||
5e170425c97cda8f798c74041979a569 ./configuration.json
|
||||
ee2560496cd373851033b3c05ee461fb ./generation_config.json
|
||||
027f9b7a7bb96f30962b5be7e4e4a5f1 ./hf_quant_config.json
|
||||
57cdec3085e0805c4ad5875f61c63b68 ./model-00030-of-00047.safetensors
|
||||
be93abf8f84e890b8dd1d4187ef540c2 ./model-00001-of-00047.safetensors
|
||||
a4a627a33e69e91491c9a3d3a7c9cba9 ./model-00031-of-00047.safetensors
|
||||
500a87d1fd1dd6a93400539227361f5d ./model-00002-of-00047.safetensors
|
||||
c732e4ee2f910af666418abd547d34cb ./model-00032-of-00047.safetensors
|
||||
68024a4367c3341402766fcc40d383f6 ./model-00003-of-00047.safetensors
|
||||
94188d8acbc7fd47ee99400507793d6a ./model-00033-of-00047.safetensors
|
||||
b5870fe5e8704e879895a7b5a3be5136 ./model-00004-of-00047.safetensors
|
||||
87b17b02643b1c14ad2e90900b3dfa89 ./model-00034-of-00047.safetensors
|
||||
f2f51d0be2c49ba6569d59d8d1db5236 ./model-00005-of-00047.safetensors
|
||||
915184c001e70fea709ffbb5242e4a8c ./model-00035-of-00047.safetensors
|
||||
fe2146b7c71775e51bad2322c52c34d1 ./model-00006-of-00047.safetensors
|
||||
56d618d4ba615cc19a0898d02d82bb52 ./model-00036-of-00047.safetensors
|
||||
03c0d5ea0471553cce658b70daed7b47 ./model-00007-of-00047.safetensors
|
||||
c76ac8dd7dbddbc4a636118ad8a5f713 ./model-00038-of-00047.safetensors
|
||||
49e246d6faa068641d2858e36b6bb5cd ./model-00008-of-00047.safetensors
|
||||
8f31262bc0a079c946a9ec1cdeb4051e ./model-00037-of-00047.safetensors
|
||||
b956d449de03a2db14fee14df7aaddac ./model-00009-of-00047.safetensors
|
||||
087456e9659f4d7c8c9a4af093b7c828 ./model-00039-of-00047.safetensors
|
||||
f94083fc155df999ab9ff27f6f67b978 ./model-00010-of-00047.safetensors
|
||||
bfd9606be27f5efbb34d85dc8fad3cea ./model-00040-of-00047.safetensors
|
||||
775a242af0663e5c65bc1b41a8930314 ./model-00011-of-00047.safetensors
|
||||
a4c70502c1470d1cd520f4f2b0f968d8 ./model-00041-of-00047.safetensors
|
||||
1d9d4ca43e411b977cded63b3447b8b8 ./model-00012-of-00047.safetensors
|
||||
f00a801a8de9a31b2ff686f6f8924d41 ./model-00042-of-00047.safetensors
|
||||
5feebc534bb09cee7b7e908aab8e2108 ./model-00013-of-00047.safetensors
|
||||
b58238047ce49362d39c36b54761c4cf ./model-00043-of-00047.safetensors
|
||||
c15c8fa6325af6ca302e10425b357199 ./model-00014-of-00047.safetensors
|
||||
fb652edaa0a450b7ce333829e363ee7d ./model.safetensors.index.json
|
||||
ce7e000a8d1c076b13346374f378e1f3 ./model-00015-of-00047.safetensors
|
||||
752f6cd2e6a4a2ea824d1b513530e0b0 ./tokenizer.json
|
||||
96ab98059044ac21eff43da3d9882689 ./model-00016-of-00047.safetensors
|
||||
3312b710133454ca7150bccad6381bd8 ./tokenizer_config.json
|
||||
82429b288fe2a97cb771e78bea60a0bd ./model-00017-of-00047.safetensors
|
||||
7605255172c381931a496b316faecdb7 ./model-00018-of-00047.safetensors
|
||||
e7fdbb1bd3e82f97831f4c1c69e0403a ./model-00019-of-00047.safetensors
|
||||
f5d355f9a737d7aa270d178a1bceeeec ./model-00020-of-00047.safetensors
|
||||
50b23a8dc4932840e5ae2062ca04b60a ./model-00021-of-00047.safetensors
|
||||
285172cbc5e900681efc5783f04a346a ./model-00022-of-00047.safetensors
|
||||
accdb4eb9e0831705bcd2d39e4f3bdcb ./model-00023-of-00047.safetensors
|
||||
9182f13eb2670db2a5312aaca2f9b1da ./model-00024-of-00047.safetensors
|
||||
198809340ec4795bdcfafdf7cd3074c7 ./model-00025-of-00047.safetensors
|
||||
da5e3a1afe95fe14c5d2d1f432f4a933 ./model-00026-of-00047.safetensors
|
||||
1d15d1696fd6d2cab3ed8726c3b25112 ./model-00027-of-00047.safetensors
|
||||
6147f7b5acb9a297eab276b16a58efe7 ./model-00028-of-00047.safetensors
|
||||
529b4048cf2f18d9eeaf8d1ce37b60cd ./model-00029-of-00047.safetensors
|
||||
ffe813a8385daf74f69f632d9437085c ./model-00044-of-00047.safetensors
|
||||
f67ddcaf1b68509fd6ab8d0399632a75 ./model-00045-of-00047.safetensors
|
||||
489dc1a568e671176a98e7cc23519c17 ./model-00046-of-00047.safetensors
|
||||
3a7abe2df1bd98e749ee8733370ea78b ./model-00047-of-00047.safetensors
|
||||
15
deploy/manifests/README.md
Normal file
15
deploy/manifests/README.md
Normal file
@ -0,0 +1,15 @@
|
||||
# 权重完整性清单（md5）
|
||||
|
||||
2026-09-08 在 174.1.60.1 `/data/hf_models/` 下对两个模型目录逐文件 `md5sum` 的原样输出。
|
||||
用于：新机器部署前核对权重传输完整性、或怀疑权重被改动时做漂移检测。
|
||||
|
||||
校验方法（在权重目录下）：
|
||||
|
||||
```bash
|
||||
md5sum -c /path/to/GLM-5.3-NVFP4.md5 # 清单内路径为 ./ 相对路径
|
||||
```
|
||||
|
||||
| 清单 | 模型 | 规模 | 用途 |
|
||||
|---|---|---|---|
|
||||
| `GLM-5.3-NVFP4.md5` | GLM-5.3-NVFP4 主模型 | 47 分片 + 配置共 55 文件 | 方案 A-F 全部部署的目标模型 |
|
||||
| `GLM-5.3-DFlash2.md5` | GLM-5.3-DFlash2 草稿模型 | 单分片 model.safetensors + 配置共 4 文件 | 方案 E 与方案 F 的 DFLASH 投机草稿 |
|
||||
31
deploy/profiles/910c/dsv4_910c_vllm_tp_dp_matrix.env
Normal file
31
deploy/profiles/910c/dsv4_910c_vllm_tp_dp_matrix.env
Normal file
@ -0,0 +1,31 @@
|
||||
# DeepSeek-V4-Flash w8a8-mtp vLLM-Ascend TPxDP profile on a single 910C node
|
||||
# (8x Ascend910 = 16 dies, 64GB HBM/die).
|
||||
# Launch via BOOTSTRAP (base64) so the JSON-bearing args survive the image
|
||||
# entrypoint; the container runs vllm serve in the foreground.
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||
|
||||
PLATFORM=910c
|
||||
EXPERIMENT=dsv4_910c_vllm_tp_dp_matrix
|
||||
MODEL_NAME=DeepSeek-V4-Flash
|
||||
ENGINE=vllm
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=local/vllm-ascend:0.23-a3-dsv4-sglang
|
||||
CONTAINER_NAME=vllm-ascend-dsv4-910c_tp${TP}_dp${DP}
|
||||
MODEL_PATH=/mnt/models/DeepSeek-V4-Flash-w8a8-mtp
|
||||
SERVED_MODEL_NAME=dsv4
|
||||
PORT=30052
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=/usr/local/python3.12.13/bin/python3
|
||||
|
||||
DEVICE_VARS="ASCEND_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTORCH_NPU_ALLOC_CONF=expandable_segments:True PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 OMP_PROC_BIND=false OMP_NUM_THREADS=10 HCCL_BUFFSIZE=1024 VLLM_ASCEND_ENABLE_FLASHCOMM1=1 TASK_QUEUE_ENABLE=1 HCCL_OP_EXPANSION_MODE=AIV LD_PRELOAD=/usr/lib64/libjemalloc.so.2"
|
||||
|
||||
DOCKER_FLAGS="--privileged --ipc host --shm-size 512g --network host --ulimit memlock=-1 --device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 --device /dev/davinci4 --device /dev/davinci5 --device /dev/davinci6 --device /dev/davinci7 --device /dev/davinci8 --device /dev/davinci9 --device /dev/davinci10 --device /dev/davinci11 --device /dev/davinci12 --device /dev/davinci13 --device /dev/davinci14 --device /dev/davinci15 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc"
|
||||
|
||||
VOLUMES="/usr/local/dcmi:/usr/local/dcmi /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool /usr/local/bin/npu-smi:/usr/local/bin/npu-smi /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info /etc/ascend_install.info:/etc/ascend_install.info /etc/hccn.conf:/etc/hccn.conf ${MODEL_PATH}:${MODEL_PATH}:ro /mnt:/mnt"
|
||||
|
||||
LAUNCH_ARGS="${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --block-size 128 --tensor-parallel-size ${TP} --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION:-0.9} --max-model-len ${MAX_MODEL_LEN:-1048576} --max-num-seqs ${MAX_NUM_SEQS:-64} --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS:-10240} --api-server-count 1 --host 0.0.0.0 --port ${PORT} --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --quantization ascend --safetensors-load-strategy prefetch --enable-expert-parallel --async-scheduling --model-loader-extra-config '{\"enable_multithread_load\": \"true\", \"num_threads\": 128}' --compilation-config '{\"cudagraph_mode\": \"FULL_DECODE_ONLY\"}' --additional-config '{\"ascend_compilation_config\":{\"enable_npugraph_ex\":true,\"enable_static_kernel\":false},\"enable_cpu_binding\": true,\"multistream_overlap_shared_expert\":true}'"
|
||||
DP_FLAG="--data-parallel-size ${DP}"
|
||||
|
||||
BOOTSTRAP="for sh in /usr/local/Ascend/ascend-toolkit/set_env.sh /usr/local/Ascend/cann-9.0.1/share/info/ascendnpu-ir/bin/set_env.sh /usr/local/Ascend/cann-9.0.0/share/info/ascendnpu-ir/bin/set_env.sh /usr/local/Ascend/nnal/atb/set_env.sh; do [ -f \$sh ] && source \$sh && break; done; exec vllm serve ${LAUNCH_ARGS}"
|
||||
31
deploy/profiles/910c/glm52_910c_vllm_tp_dp_matrix.env
Normal file
31
deploy/profiles/910c/glm52_910c_vllm_tp_dp_matrix.env
Normal file
@ -0,0 +1,31 @@
|
||||
# GLM-5.2 w4a8c8 vLLM-Ascend TPxDP profile on a single 910C node
|
||||
# (8x Ascend910 = 16 dies, 64GB HBM/die). GLM5.2-tuned image.
|
||||
# Launch via BOOTSTRAP (base64) so the JSON-bearing args survive the image
|
||||
# entrypoint; the container runs vllm serve in the foreground.
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||
|
||||
PLATFORM=910c
|
||||
EXPERIMENT=glm52_910c_vllm_tp_dp_matrix
|
||||
MODEL_NAME=GLM-5.2
|
||||
ENGINE=vllm
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=local/vllm-ascend:0.23-a3-20260718-sglang
|
||||
CONTAINER_NAME=vllm-ascend-glm52-910c_tp${TP}_dp${DP}
|
||||
MODEL_PATH=/mnt/models/GLM-5.2-w4a8c8
|
||||
SERVED_MODEL_NAME=glm-5.2
|
||||
PORT=30050
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=/usr/local/python3.12.13/bin/python3
|
||||
|
||||
DEVICE_VARS="ASCEND_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTORCH_NPU_ALLOC_CONF=expandable_segments:True PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 VLLM_ASCEND_ENABLE_FLASHCOMM1=1"
|
||||
|
||||
DOCKER_FLAGS="--privileged --ipc host --shm-size 512g --network host --ulimit memlock=-1 --device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 --device /dev/davinci4 --device /dev/davinci5 --device /dev/davinci6 --device /dev/davinci7 --device /dev/davinci8 --device /dev/davinci9 --device /dev/davinci10 --device /dev/davinci11 --device /dev/davinci12 --device /dev/davinci13 --device /dev/davinci14 --device /dev/davinci15 --device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc"
|
||||
|
||||
VOLUMES="/usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool /usr/local/bin/npu-smi:/usr/local/bin/npu-smi /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info /etc/ascend_install.info:/etc/ascend_install.info /etc/hccn.conf:/etc/hccn.conf ${MODEL_PATH}:${MODEL_PATH}:ro /usr/local/dcmi:/usr/local/dcmi /mnt:/mnt"
|
||||
|
||||
LAUNCH_ARGS="${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --trust-remote-code --block-size 128 --tensor-parallel-size ${TP} --enable-expert-parallel --gpu-memory-utilization ${GPU_MEMORY_UTILIZATION:-0.95} --max-model-len ${MAX_MODEL_LEN:-131072} --max-num-seqs ${MAX_NUM_SEQS:-256} --max-num-batched-tokens ${MAX_NUM_BATCHED_TOKENS:-8192} --api-server-count 1 --host 0.0.0.0 --port ${PORT} --compilation-config '{\"cudagraph_mode\": \"FULL_DECODE_ONLY\"}' --additional-config '{\"enable_dsa_cp\": true,\"enable_sparse_sfa_c8\": false, \"enable_sparse_li_c8\": true,\"enable_balance_scheduling\": true,\"multistream_overlap_shared_expert\":true}'"
|
||||
DP_FLAG="--data-parallel-size ${DP}"
|
||||
|
||||
BOOTSTRAP="for sh in /usr/local/Ascend/ascend-toolkit/set_env.sh /usr/local/Ascend/cann-9.0.1/share/info/ascendnpu-ir/bin/set_env.sh /usr/local/Ascend/cann-9.0.0/share/info/ascendnpu-ir/bin/set_env.sh /usr/local/Ascend/nnal/atb/set_env.sh; do [ -f \$sh ] && source \$sh && break; done; exec vllm serve ${LAUNCH_ARGS}"
|
||||
27
deploy/profiles/p800/dsv4_w8a8_int8_sglang.env
Normal file
27
deploy/profiles/p800/dsv4_w8a8_int8_sglang.env
Normal file
@ -0,0 +1,27 @@
|
||||
# P800 + SGLang-XPU + DeepSeek-V4-Flash-INT8 deployment profile.
|
||||
# Model-team only. Ops should NOT run `python -m sskj.deploy`; they only
|
||||
# run `python -m sskj.bench` against the URL this profile serves.
|
||||
|
||||
PLATFORM=p800
|
||||
MODEL_NAME=DeepSeek-V4-Flash-INT8
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202
|
||||
CONTAINER_NAME=sglang-dsv4-flash-tpdp
|
||||
MODEL_PATH=/data1/models/DeepSeek-V4-Flash-INT8
|
||||
SERVED_MODEL_NAME=deepseek-v4-flash-int8
|
||||
PORT=30014
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
CONTAINER_PYTHON=/root/miniconda/envs/python310_torch25_cuda/bin/python
|
||||
PATCH_ROOT=${ROOT_DIR}/platforms/patches/kunlun_p800
|
||||
|
||||
DEVICE_VARS="XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 CUDA_DEVICE_ORDER=OAM_ID"
|
||||
ENGINE_ENV="SGLANG_USE_TRANSFORMERS_V5_TOKENIZER=1 XMLIR_FORCE_USE_XPU_GRAPH=1 SGLANG_DSV4_MODE=2604 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_NSA_DUAL_STREAM=true SGLANG_NSA_QUANT_WQ_B_WK=false SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1 SGLANG_OPT_DEEPGEMM_HC_PRENORM=false SGLANG_OPT_USE_TILELANG_MHC_PRE=1 SGLANG_OPT_USE_TILELANG_MHC_POST=1 SGLANG_CLEAN_REQUEST_WHEN_RETRACT=1 SGLANG_SET_CPU_AFFINITY=1 SGLANG_OPT_USE_KLX_TOPK_KERNEL=1 XSGL_INTERTYPE_BFP16=1 ENABLE_FAST_BFP16_ATTN=1 XSGL_USE_DEEP_GEMM_BMM=1 XSGL_XDNN_QUANT=1 XSGL_FUSE_RMS_NORM_QUANT=1 XSGL_TRANSPOSE_MATMUL_WEIGHT=1 XINFER_QUANT_SDNN=1 XSGL_USE_MOE_SIGMOID_GROUP_TOPK_NORM=1 XSGL_EARLY_FIRST_TOKEN=1 XSGL_ENABLE_TGEMM_FP16=1 SGLANG_ENABLE_SPEC_V2=True SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 PYTHONDONTWRITEBYTECODE=1 XTORCH_OPS_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/xtorch_ops XPU_RUNTIME_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/xcudart/lib BKCL_TREE_THRESHOLD=1048576 CUDA_ENABLE_P2P_NO_UVA=1 NCCL_IB_GID_INDEX=3 IS_DSV4=1 MC_CUSTOM_TOPO_JSON=/workspace/nic_priority_matrix_test.json SGLANG_DSV4_FP4_EXPERTS=false SGLANG_APPLY_CONFIG_BACKUP=auto BKCL_ENABLE_XDR=1 BKCL_RDMA_NICS=eth1,eth1,eth3,eth3,eth5,eth5,eth7,eth7 BKCL_RDMA_VERBS=1 XSGL_INT8_LM_HEAD=1 SGLANG_P800_ALL_GATHER_FALLBACK=0"
|
||||
|
||||
DOCKER_FLAGS="--privileged --network host --ipc host --device /dev/xpu0:/dev/xpu0 --device /dev/xpu1:/dev/xpu1 --device /dev/xpu2:/dev/xpu2 --device /dev/xpu3:/dev/xpu3 --device /dev/xpu4:/dev/xpu4 --device /dev/xpu5:/dev/xpu5 --device /dev/xpu6:/dev/xpu6 --device /dev/xpu7:/dev/xpu7 --device /dev/xpuctrl:/dev/xpuctrl"
|
||||
VOLUMES="${MODEL_PATH}:/models:ro ${MODEL_PATH}:${MODEL_PATH}:ro ${PATCH_ROOT}/nic_priority_matrix_test.json:/workspace/nic_priority_matrix_test.json:ro"
|
||||
|
||||
LAUNCH_ARGS="--host 0.0.0.0 --port ${PORT} --model-path /models --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --page-size 64 --mem-fraction-static 0.8 --tensor-parallel-size ${TP} --disable-shared-experts-fusion --quantization w8a8_int8 --kv-cache-dtype float16 --disable-piecewise-cuda-graph --cuda-graph-max-bs 32 --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --constrained-json-disable-any-whitespace --enable-metrics --enable-request-time-stats-logging --context-length 65536"
|
||||
DP_FLAG="--dp-size ${DP}"
|
||||
BOOTSTRAP="cd /workspace; find /root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true; ${CONTAINER_PYTHON} -m pip install --upgrade safetensors -q; ${CONTAINER_PYTHON} -m pip install https://files.pythonhosted.org/packages/14/8b/2a1333a6455c6fad401c2285dee6f58016c55b1cb44cae3a31f8a9cc7d83/apache_tvm_ffi-0.1.0b2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl -q; ${CONTAINER_PYTHON} -c \"import torch; torch.float8_e8m0fnu = torch.uint8; import runpy, sys; sys.argv[0] = 'sglang.launch_server'; runpy.run_module('sglang.launch_server', run_name='__main__')\" ${LAUNCH_ARGS}"
|
||||
29
deploy/profiles/p800/glm52_w8a8_int8_dynamic.env
Normal file
29
deploy/profiles/p800/glm52_w8a8_int8_dynamic.env
Normal file
@ -0,0 +1,29 @@
|
||||
# P800 + SGLang-XPU + GLM-5.2-W8A8-INT8-Dynamic deployment profile.
|
||||
# Supports the 2-node deploy previously living in deploy_glm52_2nodes.sh.
|
||||
# Model-team only. Ops should not run `python -m sskj.deploy`.
|
||||
|
||||
PLATFORM=p800
|
||||
MODEL_NAME=GLM-5.2-W8A8-INT8-Dynamic
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202
|
||||
CONTAINER_NAME=glm52_deploy
|
||||
MODEL_PATH=/data1/models/GLM-5.2-W8A8-INT8-Dynamic
|
||||
SERVED_MODEL_NAME=glm52
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
NNODES=2
|
||||
MASTER_IP=10.100.11.2
|
||||
DIST_PORT=6311
|
||||
DP_SIZE=2
|
||||
|
||||
DEVICE_VARS="XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 CUDA_DEVICE_ORDER=OAM_ID"
|
||||
ENGINE_ENV="BKCL_ENABLE_XDR=1 BKCL_ENABLE_IBGDA=1 BKCL_SOCKET_IFNAME=eth0 BKCL_RDMA_NICS=eth1,eth2,eth3,eth4,eth5,eth6,eth7,eth8 BKCL_TREE_THRESHOLD=1048576 XSHMEM_MODE=1 XSHMEM_QP_NUM_PER_RANK=32 BKCL_RDMA_VERBS=1 NCCL_IB_GID_INDEX=3 CUDA_ENABLE_P2P_NO_UVA=1 SGLANG_USE_TRANSFORMERS_V5_TOKENIZER=1 XSGL_INTERTYPE_BFP16=1 ENABLE_FAST_BFP16_ATTN=1 XSGL_USE_DEEP_GEMM_BMM=1 XSGL_XDNN_QUANT=1 XSGL_FUSE_RMS_NORM_QUANT=1 XSGL_TRANSPOSE_MATMUL_WEIGHT=1 XINFER_QUANT_SDNN=1 XMLIR_FORCE_USE_XPU_GRAPH=1 XSGL_USE_MOE_SIGMOID_GROUP_TOPK_NORM=1 XSGL_INT8_LM_HEAD=1 SGLANG_CLEAN_REQUEST_WHEN_RETRACT=1 SGLANG_ENABLE_SPEC_V2=True SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 USE_FAST_BFP16_FC=1 XPUAPI_SDNN_BF16_ROUND_MODE=3 XMLIR_ENABLE_FAST_FC=true XSGL_FAST_SWIGLU=1 SGLANG_NSA_DUAL_STREAM=true SGLANG_NSA_QUANT_WQ_B_WK=false XPU_FLASH_ATTENTION_DECODER_USE_BALANCE=true"
|
||||
|
||||
DOCKER_FLAGS="--privileged --network host --security-opt label=disable --shm-size 64m --device /dev/xpu0:/dev/xpu0 --device /dev/xpu1:/dev/xpu1 --device /dev/xpu2:/dev/xpu2 --device /dev/xpu3:/dev/xpu3 --device /dev/xpu4:/dev/xpu4 --device /dev/xpu5:/dev/xpu5 --device /dev/xpu6:/dev/xpu6 --device /dev/xpu7:/dev/xpu7 --device /dev/xpuctrl:/dev/xpuctrl"
|
||||
VOLUMES="${MODEL_PATH}:/models:ro ${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--host 0.0.0.0 --port ${PORT} --model-path /models --dist-init-addr ${MASTER_IP}:${DIST_PORT} --nnodes ${NNODES} --node-rank ${NODE_RANK} --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --page-size 64 --mem-fraction-static 0.82 --max-prefill-tokens 65536 --chunked-prefill-size 16384 --max-running-requests 128 --tp-size 16 --ep-size 16 --dp-size ${DP_SIZE} --enable-dp-attention --moe-dense-tp-size 1 --enable-dp-lm-head --moe-a2a-backend deepep --deepep-mode auto --quantization w8a8_int8 --disable-shared-experts-fusion --kv-cache-dtype float16 --disable-radix-cache --watchdog-timeout 3000000 --tool-call-parser glm47 --reasoning-parser glm45 --enable-nsa-prefill-context-parallel --enable-single-batch-overlap"
|
||||
|
||||
BOOTSTRAP="source /root/miniconda/bin/activate python310_torch25_cuda; cd /workspace; pip install --no-index --find-links=/workspace/ /workspace/transformers-5.3.0-py3-none-any.whl 2>/dev/null || true; nohup python3 -m sglang.launch_server ${LAUNCH_ARGS} > /workspace/sglang_log_${NODE_RANK}.log 2>&1 & sleep infinity"
|
||||
34
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp16.env
Normal file
34
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp16.env
Normal file
@ -0,0 +1,34 @@
|
||||
# DeepSeek-V4-Pro SGLang multi-node TP=16 EP=2 deployment profile (2x RTX 6000D).
|
||||
# Nodes: 10.101.0.11 (rank 0) + 10.101.0.13 (rank 1), 16x RTX 6000D (85GB).
|
||||
# Key: SGLANG_SHARED_EXPERT_TP1=1 + --ep-size 2 -> TP=16 works with FP8 block=128.
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=dsv4_pro6000_sglang_tp16
|
||||
MODEL_NAME=DeepSeek-V4-Pro
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/DeepSeek-V4-Pro
|
||||
SERVED_MODEL_NAME=default
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=10.101.0.11
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||
NNODES=2
|
||||
NODE_HOSTS="10.101.0.11 10.101.0.13"
|
||||
NODE_SSH_USER=root
|
||||
MASTER_IP=10.101.0.11
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="NCCL_SOCKET_IFNAME=eth0 NCCL_DEBUG=WARN SGLANG_SHARED_EXPERT_TP1=1 PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --shm-size=20g"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --ep-size 2 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --host 0.0.0.0 --port ${PORT} --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256"
|
||||
32
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp16_eagle.env
Normal file
32
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp16_eagle.env
Normal file
@ -0,0 +1,32 @@
|
||||
# DeepSeek-V4-Pro SGLang multi-node TP=16 EP=2 + EAGLE speculative decoding
|
||||
# (2x RTX 6000D). Same topology as dsv4_pro6000_sglang_tp16, plus EAGLE flags.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=dsv4_pro6000_sglang_tp16_eagle
|
||||
MODEL_NAME=DeepSeek-V4-Pro
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/DeepSeek-V4-Pro
|
||||
SERVED_MODEL_NAME=default
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=10.101.0.11
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||
NNODES=2
|
||||
NODE_HOSTS="10.101.0.11 10.101.0.13"
|
||||
NODE_SSH_USER=root
|
||||
MASTER_IP=10.101.0.11
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="NCCL_SOCKET_IFNAME=eth0 NCCL_DEBUG=WARN SGLANG_SHARED_EXPERT_TP1=1 PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --shm-size=20g"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --ep-size 2 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --host 0.0.0.0 --port ${PORT} --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4"
|
||||
24
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp_dp_matrix.env
Normal file
24
deploy/profiles/pro6000/dsv4_pro6000_sglang_tp_dp_matrix.env
Normal file
@ -0,0 +1,24 @@
|
||||
# DeepSeek-V4-Flash SGLang TPxDP matrix profile on a single RTX 6000D node (8 GPUs).
|
||||
# TP=2/DP=4 is excluded because Marlin weight loading OOMs on this machine.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=dsv4_pro6000_sglang_tp_dp_matrix
|
||||
MODEL_NAME=DeepSeek-V4-Flash
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
|
||||
CONTAINER_NAME=${EXPERIMENT}_sglang_tp${TP}_dp${DP}
|
||||
MODEL_PATH=/data/6000D/DeepSeek-V4-Flash
|
||||
SERVED_MODEL_NAME=deepseek-v4-flash
|
||||
PORT=30031
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --privileged --ipc=host --network host --ulimit memlock=-1 --ulimit stack=67108864"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --trust-remote-code --tp-size ${TP} --moe-runner-backend auto --mem-fraction-static 0.9 --context-length 131072 --max-running-requests 64 --host 0.0.0.0 --port ${PORT}"
|
||||
24
deploy/profiles/pro6000/dsv4_pro6000_vllm_tp_dp_matrix.env
Normal file
24
deploy/profiles/pro6000/dsv4_pro6000_vllm_tp_dp_matrix.env
Normal file
@ -0,0 +1,24 @@
|
||||
# DeepSeek-V4-Flash vLLM TPxDP matrix profile on a single RTX 6000D node (8 GPUs).
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=dsv4_pro6000_vllm_tp_dp_matrix
|
||||
MODEL_NAME=DeepSeek-V4-Flash
|
||||
ENGINE=vllm
|
||||
SERVER_CMD=vllm serve
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=vllm-sm120-dsv4:0.25.1-fi0.6.14
|
||||
CONTAINER_NAME=${EXPERIMENT}_vllm_tp${TP}_dp${DP}
|
||||
MODEL_PATH=/data/6000D/DeepSeek-V4-Flash
|
||||
SERVED_MODEL_NAME=deepseek-v4-flash
|
||||
PORT=30030
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --privileged --ipc=host --network host --ulimit memlock=-1 --ulimit stack=67108864"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
|
||||
LAUNCH_ARGS="--model ${MODEL_PATH} --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --tensor-parallel-size ${TP} --gpu-memory-utilization 0.9 --max-model-len 131072 --max-num-seqs 128 --host 0.0.0.0 --port ${PORT}"
|
||||
DP_FLAG="--data-parallel-size ${DP}"
|
||||
37
deploy/profiles/pro6000/glm52_pro6000_sglang_tp16.env
Normal file
37
deploy/profiles/pro6000/glm52_pro6000_sglang_tp16.env
Normal file
@ -0,0 +1,37 @@
|
||||
# GLM-5.2-FP8 SGLang multi-node TP=16 deployment profile (2x RTX 6000D).
|
||||
# node0 = 174.1.51.5 (pro6000D.1, HTTP API), node1 = 174.1.51.7 (pro6000D.3,
|
||||
# compute-only). Model ~700GB fp8 so 2-node TP=16 is mandatory.
|
||||
# Requires the tilelang/dsa patch mounts below (experiment-dir sources).
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm52_pro6000_sglang_multinode_tp16
|
||||
MODEL_NAME=GLM-5.2-FP8
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
|
||||
CONTAINER_NAME=${EXPERIMENT}_sglang_tp${TP}_dp${DP}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/GLM-5.2-FP8
|
||||
SERVED_MODEL_NAME=GLM-5.2-FP8
|
||||
PORT=30031
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.51.5
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
TP=16
|
||||
DP=1
|
||||
|
||||
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||
NNODES=2
|
||||
NODE_HOSTS="174.1.51.5 174.1.51.7"
|
||||
NODE_SSH_USER=root
|
||||
MASTER_IP=174.1.51.5
|
||||
DIST_PORT=50000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:256 NCCL_IB_HCA=mlx5_0,mlx5_3 NCCL_MIN_NCHANNELS=8 NCCL_IB_QPS_PER_CONNECTION=4 NCCL_NET_GDR_LEVEL=PHB NCCL_SOCKET_IFNAME=eth1 NCCL_IB_RETRY_CNT=14 NCCL_IB_TIMEOUT=16"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --privileged --ipc=host --network host --ulimit memlock=-1 --ulimit stack=67108864"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro ${ROOT_DIR}/sglang_nightly_cu13_cache:/root/.cache"
|
||||
PATCH_MOUNTS="${ROOT_DIR}/experiments/pro6000/glm52_pro6000_sglang_multinode_tp16/tilelang_kernel_sm120.py:/sgl-workspace/sglang/python/sglang/kernels/ops/attention/dsa/tilelang_kernel.py:ro ${ROOT_DIR}/experiments/pro6000/glm52_pro6000_sglang_multinode_tp16/dsa_backend.py:/sgl-workspace/sglang/python/sglang/srt/layers/attention/dsa_backend.py:ro ${ROOT_DIR}/experiments/pro6000/glm52_pro6000_sglang_multinode_tp16/dsa_indexer.py:/sgl-workspace/sglang/python/sglang/srt/layers/attention/dsa/dsa_indexer.py:ro"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --trust-remote-code --tp-size ${TP} --dp-size ${DP} --quantization fp8 --kv-cache-dtype bfloat16 --moe-runner-backend auto --mem-fraction-static 0.80 --context-length 131072 --max-running-requests 64 --cuda-graph-backend-decode disabled --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --host 0.0.0.0 --port ${PORT} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --nnodes ${NNODES} --node-rank ${NODE_RANK}"
|
||||
52
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_decode.env
Normal file
52
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_decode.env
Normal file
@ -0,0 +1,52 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色3: decode 节点（方案F，跑在 6000D-2 = 174.1.60.2，8 卡）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> prefill -> [本角色] -> router。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行部署脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_s1_decode.sh
|
||||
# （launch 脚本同目录 s1_decode_launch.sh）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 配方 = 方案 E v5（TP8 + DFLASH + MRR12 + fa4 + window2048）+ PD decode flags；
|
||||
# KV 池 214,336 token（比单机 E 的 243,584 少，PD 传输缓冲占显存）
|
||||
# - 容量属性（实测判决的核心）: 池按 16k 场景定容 → 128k 仅容 1 个驻留（cc4 时 TTFT
|
||||
# 堆到 31.0s）、64k 容 3、16k 容 12（MRR12 上限）。长上下文负载该池就是瓶颈
|
||||
# - 必须挂补丁树 /data/sglang_patch_glm53:/sgl-workspace/sglang（唯一改动
|
||||
# python/sglang/srt/speculative/spec_info.py 接 build_dflash_family_disagg_draft_input，
|
||||
# DFlash PD 冷启动接线）+ 装 mooncake wheel（--no-deps，launch 脚本自装）
|
||||
# - --network host + --device /dev/infiniband + --ulimit memlock=-1，IB 设备 mlx5_0-3
|
||||
# - DFlash accept 在 12-batch verify 下掉到 1.8（单机 EAGLE 2.84）——decode 侧并发
|
||||
# verify 是 DFlash 的弱势区，场景二吞吐上限由此而来
|
||||
# - 60.2 平时空闲但 GPU7 常有外部裸金属任务（main_v2.py）：动卡前核实归属，勿清
|
||||
#
|
||||
# 实测成绩: 飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 场景二判决: "两机买 TTFT、不买吞吐"（÷2 单机等效 74-85 tok/s = A 的 75-96%，TTFT 减半）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_decode
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
ROLE=pd-decode
|
||||
NODE=174.1.60.2
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-s1-decode
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1200
|
||||
|
||||
TP=8
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=12
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma PYTHONUNBUFFERED=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --shm-size 64g --ulimit memlock=-1 --device /dev/infiniband --restart unless-stopped"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/flashkda_deploy/wheels:/mc_wheels:ro /data/sglang_patch_glm53:/sgl-workspace/sglang"
|
||||
|
||||
BOOTSTRAP="pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --speculative-draft-window-size 2048 --disaggregation-mode decode --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
29
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_master.env
Normal file
29
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_master.env
Normal file
@ -0,0 +1,29 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色1: Mooncake 元数据服务 mc-master（方案F，双机 6000D-1 + 6000D-2）。
|
||||
# 四角色链之一，启动顺序强制: mc-master(60.1:50051) -> prefill(60.1:30000) -> decode(60.2:30000) -> router(60.2:31000)。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md；可执行脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_smoke_master.sh
|
||||
#
|
||||
# 关键点:
|
||||
# - 镜像自带的 mooncake_master 二进制（/opt/sglang/bin/mooncake_master），无需 GPU
|
||||
# --network host，监听 50051；prefill/decode 容器通过 MOONCAKE_MASTER=174.1.60.1:50051 注册
|
||||
# - 就绪判据: ss -tln | grep :50051（脚本 sleep 3 后检查）
|
||||
# - 拆链时必须先删本容器之外的角色再删它? 否——顺序无依赖，但生产恢复 60.1 时
|
||||
# 本容器与 glm53-pd-smoke-prefill 都要删干净、等显存排空再拉生产容器
|
||||
#
|
||||
# 实测背景（2026-09-08 方案F 双场景压测）: 13/13 干净点，质量门 6/7。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_master
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang-mooncake-master
|
||||
RUNTIME=docker
|
||||
ROLE=pd-master
|
||||
NODE=174.1.60.1
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=mc-master
|
||||
PORT=50051
|
||||
|
||||
DOCKER_FLAGS="--network host --restart unless-stopped"
|
||||
ENTRYPOINT="/opt/sglang/bin/mooncake_master"
|
||||
|
||||
READY_CHECK="ss -tln | grep :50051"
|
||||
59
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_prefill.env
Normal file
59
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_prefill.env
Normal file
@ -0,0 +1,59 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色2: prefill 节点（方案F，跑在 6000D-1 = 174.1.60.1，8 卡）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> [本角色] -> decode -> router。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行部署脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_probe.sh
|
||||
# （参数 = launch 脚本名；launch 脚本同目录 p1b_prefill_tp4pp2_cps8k_launch.sh 及其 radix 变体）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 拓扑 TP4 PP2（4卡/stage × 2 stage）+ DFLASH 草稿（草稿只跑 prefill 侧草稿 KV，
|
||||
# export SGLANG_DFLASH_PD_DRAFT_KV_TRANSFER=0 = 草稿 KV 不跨机传输）
|
||||
# - mem 0.78 低于单机方案（PD 模式下 mooncake 传输缓冲占显存）；cps 8192（非 D 方案的 16384）
|
||||
# - 基线 launch 脚本带 --disable-radix-cache；场景一（90% 命中）实测用 radix 变体
|
||||
# （p1b_prefill_tp4pp2_cps8k_radix_launch.sh，唯一差异 = 删掉该 flag）
|
||||
# - 必须挂补丁树 /data/sglang_patch_glm53:/sgl-workspace/sglang（含 1 个文件改动:
|
||||
# python/sglang/srt/speculative/spec_info.py 接 build_dflash_family_disagg_draft_input，
|
||||
# DFlash PD 冷启动接线；无此补丁 decode 首请求 400）
|
||||
# - 必须装 mooncake wheel /data/flashkda_deploy/wheels/mooncake_transfer_engine_cuda13-
|
||||
# 0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl（--no-deps，容器内 launch 脚本自装）
|
||||
# - --network host + --device /dev/infiniband + --ulimit memlock=-1，IB 设备 mlx5_0-3
|
||||
# - 前置硬检查: mc-master 50051 必须已监听（deploy_pd_probe.sh 自带）
|
||||
# - 60.1 是生产机（日常跑 glm53-pp4）：拉本角色前须停生产容器，测完等显存 <2000MiB
|
||||
# 再跑 deploy_glm53_pp4.sh 恢复
|
||||
#
|
||||
# 实测成绩: 飞书《GLM-5.3-NVFP4 双场景压测报告》方案 F 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 128k cc1 TTFT 3.91s = 六方案最低；跨机 prefill/decode 重叠使 RDMA 传输被完全吸收。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_prefill
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
ROLE=pd-prefill
|
||||
NODE=174.1.60.1
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-pd-smoke-prefill
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1500
|
||||
|
||||
TP=4
|
||||
PP=2
|
||||
MEM_FRACTION_STATIC=0.78
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma SGLANG_DFLASH_PD_DRAFT_KV_TRANSFER=0 PYTHONUNBUFFERED=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --shm-size 64g --ulimit memlock=-1 --device /dev/infiniband --restart unless-stopped"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/flashkda_deploy/wheels:/mc_wheels:ro /data/sglang_patch_glm53:/sgl-workspace/sglang"
|
||||
|
||||
BOOTSTRAP="pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
|
||||
# 场景一（90% 命中）变体: LAUNCH_ARGS 追加 --disable-radix-cache 删除（radix 开）。
|
||||
# 基线（本 profile 口径）= radix off。
|
||||
35
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_router.env
Normal file
35
deploy/profiles/pro6000/glm53_nvfp4_pro6000_pd_router.env
Normal file
@ -0,0 +1,35 @@
|
||||
# GLM-5.3-NVFP4 PD 分离链 - 角色4: MiniLB 路由（方案F，跑在 6000D-2 = 174.1.60.2:31000）。
|
||||
# 四角色链之一，启动顺序强制: mc-master -> prefill -> decode -> [本角色]。
|
||||
# 完整链编排见 deploy/PD_CHAIN.md。
|
||||
# 可执行脚本: experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_pd_smoke_router.sh
|
||||
#
|
||||
# 关键点:
|
||||
# - 镜像内自带 sglang_router（python3 -m sglang_router.launch_router），无需 GPU
|
||||
# - --prefill 参数形态: URL 后跟 prefill 侧 disaggregation-bootstrap-port（28800），
|
||||
# 这是 KV 传输握手口，漏掉则路由建立后首请求挂起；--decode 只有 URL
|
||||
# - bench 与质量门全部经 :31000/generate 打（--url http://174.1.60.2:31000/generate），
|
||||
# 即全链路含 KV transfer + DFlash 草稿
|
||||
# - usage/accept 指标 router 转发后可能缺失: 命中率读 prefill 容器日志
|
||||
# （docker logs glm53-pd-smoke-prefill，PP2 下日志计数 ×2 不影响比值），
|
||||
# accept 读 decode 容器日志
|
||||
# - 质量门脚本: 用 quality_gate_605.sh 副本 sed 's/PORT=30000/PORT=31000/' 生成
|
||||
# - 就绪判据: ss -tln | grep :31000
|
||||
#
|
||||
# 实测背景: 全链 13/13 干净点、质量门 6/7（tool call 为 parser 配置缺口，与 B/D 同口径）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_pd_router
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang-router
|
||||
RUNTIME=docker
|
||||
ROLE=pd-router
|
||||
NODE=174.1.60.2
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=pd-smoke-router
|
||||
PORT=31000
|
||||
|
||||
DOCKER_FLAGS="--network host --restart unless-stopped --entrypoint python3"
|
||||
LAUNCH_ARGS="-m sglang_router.launch_router --pd-disaggregation --mini-lb --prefill http://174.1.60.1:30000 28800 --decode http://174.1.60.2:30000 --host 0.0.0.0 --port 31000"
|
||||
|
||||
READY_CHECK="ss -tln | grep :31000"
|
||||
@ -0,0 +1,52 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 方案 D（2026-09-08 双场景补测；6000D-1 现役生产容器 glm53-pp4 的原样配方）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_pp4.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 脚本头注释是早期 TP4PP2 版残留（写"TP4 PP2 + IndexCache"），实际配置 TP=2 PP=4，
|
||||
# 以脚本 docker run 段为准；md5 def3c64c5dc19e1d507080e3366c3762
|
||||
# - 串行 prefill 有效速率 6.5-6.7k tok/s 为各方案最高；KV 池 1,040,384 token
|
||||
# （A 的 3.76 倍，≈61.6 请求驻留，超过 MRR 48——池在高并发下不构成约束）
|
||||
# - 场景二（16k 独立输入 cc8-64）成立：cc32 起反超 TP4PP2（方案B），cc40/64 输出
|
||||
# 203.8/208.2 tok/s；TTFT p50 五档全档低于 B
|
||||
# - 场景一（90% 命中长上下文）8/8 全败：TP2 长上下文每卡 KV 读量翻倍 + PP4 低并发
|
||||
# 流水空泡，单请求 decode 仅 16-19 tok/s。生产态 radix off 前缀命中恒 0，实际表现
|
||||
# 比报告 D 行（radix-on 最好情况）更差——长上下文/共享前缀负载勿用
|
||||
# - DSA 实测：上下文长度不影响 TPOT（52.8ms 恒定），并发才是驱动
|
||||
# - 上生产须补 --tool-call-parser glm47 与 --reasoning-parser glm45
|
||||
# - bench 口径：场景二 cc40/64 用 nreq 80/128（与其他方案 40/64 单轮满波不同，
|
||||
# 已在报告表注声明）
|
||||
#
|
||||
# 实测成绩：飞书《GLM-5.3-NVFP4 双场景压测报告》方案 D 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 质量：GSM8K×5 + 中文推理通过（6/7，tool call 为 parser 配置缺口）。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp2pp4
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-pp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=16384
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -0,0 +1,44 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 + hicache deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 2026-09-09 128k 低并发容量扩容实验优胜配置（60.8 现役；60.5 交付 deploy_glm53_605_v2.sh 待执行）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_128k_capacity_topology/scripts/deploy_glm53_605_v2.sh
|
||||
#
|
||||
# 与方案 D（glm53_nvfp4_pro6000_sglang_tp2pp4.env）的关键差异（勿混淆）：
|
||||
# - radix/hicache 保持开启（60.5 真实流量命中率 90%+，关 radix 不可接受）；D 为 16k 独立输入场景关了 radix
|
||||
# - cu13 镜像 + 9 补丁只读挂载（de-GLOO request_receiver / decode_cuda_graph_runner_fix 等 r37 栈遗产，
|
||||
# nomtp 下 spec 相关补丁为惰性，de-GLOO 为 PP 通用修复）；D 用旧镜像 20260828 无挂载
|
||||
# - --reasoning-parser glm45 --tool-call-parser glm47 齐备（质量门 7/7）；D 当时 6/7
|
||||
# - --context-length 1048576（模型原生 1M；D 未设）；chunk 8192（D 16384）；MRR 16（D 48）
|
||||
# - 实测（60.8，i128k/o512 冷缓存）：KV 池 909,632 token（A 的 3.29×）、并发上限 c6、
|
||||
# 单条上限 ~909k（900k 实跑通过）、c4 输入/输出 5,907/23.1 tok/s、TTFT p50 43.4s、
|
||||
# 512k 单条 TTFT 88.4s、90% 命中 c4 输入 17,625 tok/s
|
||||
# - memfrac 0.85 为验证档；PP0 stage 空闲 22GB 提示 0.88 有余量（未验证，改动须重跑质量门+容量冒烟）
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_128k_capacity_topology
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1800
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=16
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
CONTEXT_LENGTH=1048576
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 SGLANG_PP_DEGLOO=1 SGLANG_PP_SPEC_FORCE_EAGER_DRAFT=1 SGLANG_PP_FORCE_EAGER_VERIFY=0 SGLANG_PP_SPEC_DEBUG=0"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host --cap-add SYS_PTRACE -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models + 9 patch ro-mounts (full list in scripts/deploy_glm53_605_v2.sh; files live in /root + /root/sglang_patch2 on the host)"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --reasoning-parser glm45 --tool-call-parser glm47 --enable-hierarchical-cache --hicache-ratio 3 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length ${CONTEXT_LENGTH} --host 0.0.0.0 --port ${PORT}"
|
||||
@ -0,0 +1,49 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=2 PP=4 + latest镜像 + autotune profile（方案 D 升级版）。
|
||||
# 2026-09-09 DSV4 优化点迁移实验优胜配置：五点(16k/512 cc8-64)全胜 +1.2~+3.2%，
|
||||
# A/B/A 回切确认因果成立。见 experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/。
|
||||
# 已入档未执行；60.1 生产仍跑 nightly 版（glm53-pp4，见同目录 tp2pp4.env）。
|
||||
#
|
||||
# 关键点（实测，勿随意改）:
|
||||
# - 相对 tp2pp4.env 仅两处变化：镜像 nightly-dev-20260828 → latest(d6e72886)；
|
||||
# 删 --disable-flashinfer-autotune。启动参数其余逐字相同
|
||||
# - 镜像中性实测：latest 与 nightly 同配置五点差 ≤0.7%（换镜像无风险无收益）
|
||||
# - autotune 增益钉在 tactic 缓存上：SGLANG_CACHE_DIR 与 flashinfer JIT 缓存
|
||||
# 必须挂宿主持久盘，否则重部署重抽签（增益消失/不可复现，DSV4 §12.6 同款教训）
|
||||
# - 代价：available_gpu_mem 15.64→11.79 GB（tactic 缓冲 ≈3.9GB/卡），
|
||||
# KV 池不缩（1,040,384）
|
||||
# - PCIe-IPC AllReduce 包在同一实验中五点全降（-0.4~-3.5%）判负勿叠用：
|
||||
# TP2 单对端 NCCL AR 同 switch P2P 已近最优
|
||||
# - 上生产须补 --tool-call-parser glm47 与 --reasoning-parser glm45（D 系既有缺口）
|
||||
# - 部署器：experiments/pro6000/glm53_nvfp4_dsv4_migration_bench/scripts/deploy_glm53_exp.sh
|
||||
# （EXP_NAME=<名> EXP_IMAGE=lmsysorg/sglang:latest AUTOTUNE=1）
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp2pp4_latest_autotune
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:latest
|
||||
DOCKER_IMAGE_DIGEST=sha256:d6e7288627be8b02be88e4bba38e73f6d50e2826869f753c13a4c4385ab3eda9
|
||||
CONTAINER_NAME=glm53-pp4-autotune
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=600
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=2
|
||||
PP=4
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=48
|
||||
CHUNKED_PREFILL_SIZE=16384
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 SGLANG_CACHE_DIR=/root/.cache/sglang"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models /data/glm53_exp/fi_jit_cache:/root/.cache/flashinfer /data/glm53_exp/sglang_cache:/root/.cache/sglang /data/glm53_exp/triton_cache:/root/.triton"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -0,0 +1,47 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=4 PP=2 + hicache deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 2026-09-09 hit90 场景实验优胜配置（60.8 现役；60.5 交付 deploy_glm53_605_v3.sh 待执行）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/scripts/deploy_glm53_605_v3.sh
|
||||
# 60.8 启动口径：bash /root/deploy_ppmtp_r37.sh '--tp 4 --pp-size 2 --disable-overlap-schedule
|
||||
# --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length 1048576' nomtp 8192 0.90 1 1 0 0
|
||||
#
|
||||
# 与 TP2PP4-hicache（glm53_nvfp4_pro6000_sglang_tp2pp4_hicache.env）的关键差异（勿混淆）：
|
||||
# - tp2/pp4 → tp4/pp2，memfrac 0.85 → 0.90；其余（cu13+9 挂载、chunk 8192、MRR 16、
|
||||
# hicache 3、ctx 1048576、双 parser）逐项同构
|
||||
# - 实测 KV 池 647,040（fp8 KV 18.52GB/rank，PP0 初始化后剩 10.94GB）
|
||||
# - 取胜依据（hit90=90% 命中 i128k/o512 主场景）：out cc1/2/3/4 = 28.8/47.5/63.9/76.2、
|
||||
# cc8/16 = 106.4/125.7（vs TP2PP4：cc4 +14%/cc8 +15%/cc16 +1%）；cap cc2/cc4 = 62.9/98.5
|
||||
# 零排队；质量门 7/7
|
||||
# - 让步项（知情选择）：并发独立 128k 文档 4 条（TP2PP4 为 6）；512k 单条 151.9s
|
||||
# （TP2PP4 为 114.4s，PP4 单条巨请求 prefill 流水更优）；无投机解码
|
||||
# - 判决背景：DP attention 对本模型容量负收益、DCP 对 DSA 静默算错、MTP@128k accept 2.07 判负
|
||||
# （见 experiments/pro6000/glm53_nvfp4_hit90_dp_dcp_bench/README.md）
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_hit90_dp_dcp_bench
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-cu13-20260901-07c8f729
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1800
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=4
|
||||
PP=2
|
||||
MEM_FRACTION_STATIC=0.90
|
||||
MAX_RUNNING_REQUESTS=16
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
CONTEXT_LENGTH=1048576
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 SGLANG_PP_DEGLOO=1 SGLANG_PP_SPEC_FORCE_EAGER_DRAFT=1 SGLANG_PP_FORCE_EAGER_VERIFY=0 SGLANG_PP_SPEC_DEBUG=0"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host --cap-add SYS_PTRACE -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models + 9 patch ro-mounts (full list in scripts/deploy_glm53_605_v3.sh; files live in /root + /root/sglang_patch2 on the host, bundle md5 6922e53439991bc13feee72f3760704f)"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size ${PP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --reasoning-parser glm45 --tool-call-parser glm47 --enable-hierarchical-cache --hicache-ratio 3 --disable-overlap-schedule --max-prefill-tokens 16384 --disable-custom-all-reduce --context-length ${CONTEXT_LENGTH} --host 0.0.0.0 --port ${PORT}"
|
||||
@ -0,0 +1,43 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=4 PP=2 + IndexCache deployment profile (single RTX 6000D node).
|
||||
# 场景二（16k 独立输入、cc8-32、prefill 主导）最优配置：吞吐 +41~79%、cc32 TTFT 减半
|
||||
# （对比 TP8+EAGLE 生产配置，2026-09-07 真实语料基线）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_optimal.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - index_topk_freq=4 为模型原生默认（层轴索引复用省 75% indexer，无质量损失）
|
||||
# - 禁投机解码：PP2 与投机框架不兼容（已实测）
|
||||
# - 本文件为场景二形态（--disable-radix-cache，独立输入无前缀复用）；
|
||||
# 场景一 90% 命中对比须启用 radix（唯一差异：去掉 --disable-radix-cache，
|
||||
# 见 deploy_glm53_optimal_s1.sh）
|
||||
# - mem 0.85：0.90 下 cuda graph 捕获余量不足会运行时 OOM（KV 池 569,600 = TP8 的 2.06 倍）
|
||||
# - --disable-custom-all-reduce：TP4 over PCIe 用自定义 AR 在本栈无收益
|
||||
# - 已知缺口：未带 --tool-call-parser glm47 --reasoning-parser glm45，质量门 6/7
|
||||
# （tool call 失败纯属参数缺失，非模型问题）；上生产必须补 parser
|
||||
# - 场景一（90% 命中低并发）该配置全面劣于 TP8+EAGLE（输出吞吐 −25~−65%），勿混用选型
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp4pp2_indexcache
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=4
|
||||
DP=1
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --ipc=host --shm-size 64g --entrypoint '' -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models:ro"
|
||||
|
||||
BOOTSTRAP="exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --pp-size 2 --mem-fraction-static 0.85 --max-running-requests 48 --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size 16384 --json-model-override-args '{\"index_topk_freq\": 4}' --host 0.0.0.0 --port ${PORT}"
|
||||
@ -0,0 +1,52 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=8 + DFlash2 speculative decoding profile (single RTX 6000D node, 8 GPUs).
|
||||
# 方案 E（2026-09-08 场景一补测，6000D-2）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_tp8_dflash2.sh
|
||||
# （v5 终版底稿，含四轮 OOM 战役完整教训注释；md5 5bf2b47c9349e5855b963e571e35c096）
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - v5 配方核心：MRR 48→12（verify CUDA graph 4.04→0.83GB，真正起作用的杠杆）+
|
||||
# --speculative-draft-window-size 2048 + mem0.85/cps8192。四轮 OOM 根因与推导见脚本头注释
|
||||
# - DFLASH block-diffusion 草稿 7 tokens/步，draft 权重 GLM-5.3-DFlash2，fa4 draft
|
||||
# attention（fa4 会把 draft KV 强制 bf16，fp8 需换 flashinfer/triton 后端，仅省 0.35GB 未用）
|
||||
# - KV 池 243,584 token；kv fp8_e4m3 由模型配置自动带出（无需显式 flag）
|
||||
# - 底稿（本 profile LAUNCH_ARGS）radix/AR 均为禁用；场景一实测变体共四处 delta：
|
||||
# ① 去 --disable-radix-cache（90% 命中前提）② 去 --disable-custom-all-reduce
|
||||
# （v1 CAR 与方案 A 一致开启）③ 加 --context-length 270336 ④ 加 --reasoning-parser
|
||||
# glm45 --tool-call-parser glm47（质量门 7/7 的前提）
|
||||
# - 判决：场景一 8 点全部低于方案 C、7 点低于 A——DFlash accept 低于 EAGLE（同语料
|
||||
# 2.53 vs 2.84）而每步墙钟相当，劣势全在接受率。投机栈选型维持 EAGLE3，勿用
|
||||
# DFlash2 替换（性能问题非质量问题）
|
||||
#
|
||||
# 实测成绩：飞书《GLM-5.3-NVFP4 双场景压测报告》方案 E 行（文档 SZUSdEqY1oRVxGxgILBcHqPJnEc）。
|
||||
# 质量：变体配置下质量门 7/7（GSM8K×5、中文推理、tool call 全过），DFlash 草稿无质量损失。
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp8dflash2
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
DOCKER_IMAGE_DIGEST=sha256:28e0d26073161e49ca56eba808d264a4223804a212020f1dfe1b2405b9f8a399
|
||||
CONTAINER_NAME=glm53-tp8-dflash2
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
DRAFT_MODEL_PATH=/data/hf_models/GLM-5.3-DFlash2
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=900
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=8
|
||||
MEM_FRACTION_STATIC=0.85
|
||||
MAX_RUNNING_REQUESTS=12
|
||||
CHUNKED_PREFILL_SIZE=8192
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --shm-size 64g --ipc=host -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models"
|
||||
|
||||
BOOTSTRAP="python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static ${MEM_FRACTION_STATIC} --max-running-requests ${MAX_RUNNING_REQUESTS} --disable-radix-cache --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --disable-custom-all-reduce --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} --speculative-algorithm DFLASH --speculative-draft-model-path ${DRAFT_MODEL_PATH} --speculative-draft-attention-backend fa4 --speculative-draft-window-size 2048 --host 0.0.0.0 --port ${PORT} --json-model-override-args {\"index_topk_freq\": 4}"
|
||||
@ -0,0 +1,44 @@
|
||||
# GLM-5.3-NVFP4 SGLang TP=8 + EAGLE deployment profile (single RTX 6000D node, 8 GPUs).
|
||||
# 生产标准配置（174.1.60.5 团队自用在役，2026-09-07 定稿）。
|
||||
# 可执行部署脚本：experiments/pro6000/glm53_nvfp4_pro6000d_sglang_dual_scenario_bench/scripts/deploy_glm53_605.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - SM120 必需三项：--disable-shared-experts-fusion --moe-runner-backend
|
||||
# flashinfer_cutlass --disable-flashinfer-autotune
|
||||
# - EAGLE 4/1/5 为 cc1-2 均衡点（3/1/4、5/1/6 在 ±10% 噪声内；树式 topk>1 在 DSA 上不可用；
|
||||
# EAGLE3 本 nightly 不支持 GLM-5.3）；draft 模型自动从主权重加载
|
||||
# - KV fp8_e4m3 + hicache-ratio 3 是长上下文 90% 前缀命中场景的容量前提（KV 池 276,864 token）
|
||||
# - chunk 8192：16384 已 A/B 证伪（MoE 工作区 OOM 风险，且 AR 带宽受限无次数红利）
|
||||
# - 容器入口必须 python3 -m sglang.launch_server（镜像 entrypoint 无 shebang）
|
||||
# - 重新部署前必须等显存排空（docker rm -f 异步滞留数分钟，不等会把新 KV 池压小）
|
||||
# - 场景二高并发变体（16k 独立输入 cc8-32）：仅改 --max-running-requests 32 +
|
||||
# --cuda-graph-max-bs-decode 16 --cuda-graph-bs-decode 4 8 12 16（KV 池 276,864 ÷
|
||||
# 16,896/req = 16.4 驻留上限，decode 批自然 ≤16，图覆盖到 bs16 即可，bs24/32 纯耗显存）
|
||||
# - 上线前后过质量门 quality_gate_605.sh（PASS=7/7）
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=glm53_nvfp4_pro6000_sglang_tp8eagle
|
||||
MODEL_NAME=GLM-5.3-NVFP4
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:nightly-dev-20260828-daf63171
|
||||
CONTAINER_NAME=glm53-nvfp4
|
||||
MODEL_PATH=/data/hf_models/GLM-5.3-NVFP4
|
||||
SERVED_MODEL_NAME=/data/hf_models/GLM-5.3-NVFP4
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
TP=8
|
||||
DP=1
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --ipc=host --shm-size 64g --entrypoint '' -p ${PORT}:${PORT}"
|
||||
VOLUMES="/data/hf_models:/data/hf_models:ro"
|
||||
|
||||
BOOTSTRAP="exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --tp-size ${TP} --mem-fraction-static 0.90 --max-running-requests 16 --chunked-prefill-size 8192 --max-prefill-tokens 16384 --disable-shared-experts-fusion --moe-runner-backend flashinfer_cutlass --disable-flashinfer-autotune --speculative-algorithm EAGLE --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 --kv-cache-dtype fp8_e4m3 --enable-hierarchical-cache --hicache-ratio 3 --cuda-graph-max-bs-decode 8 --cuda-graph-bs-decode 1 2 3 4 6 8 --cuda-graph-max-bs-prefill 8 --context-length 270336 --reasoning-parser glm45 --tool-call-parser glm47 --host 0.0.0.0 --port ${PORT}"
|
||||
54
deploy/profiles/pro6000/kimi3_pro6000_pd_decode.env
Normal file
54
deploy/profiles/pro6000/kimi3_pro6000_pd_decode.env
Normal file
@ -0,0 +1,54 @@
|
||||
# Kimi-K3 PD 分离 - D 组 (decode) deployment profile (4x RTX 6000D).
|
||||
# Nodes: 174.1.60.5~8 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||
#
|
||||
# 这是 PD (Prefill/Decode) 分离部署的 D 组 (decode) 端 profile。
|
||||
# 配套 P 组 profile: kimi3_pro6000_pd_prefill.env
|
||||
# 配套编排脚本: experiments/pro6000/kimi3_pro6000_pd_rdma/deploy_pd.sh
|
||||
#
|
||||
# 与 P 组相同的传输/网络配置（mooncake RDMA + 计算网），区别仅在:
|
||||
# - 节点是 174.1.60.5~8
|
||||
# - --disaggregation-mode decode --port 30000
|
||||
# - dist-init-addr 174.1.60.5:20000
|
||||
# - 必须 D 组在 P 组之后启动（见 deploy_pd.sh）
|
||||
#
|
||||
# Model-team only. Ops only run benchmark against the served router URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=kimi3_pro6000_pd_decode
|
||||
MODEL_NAME=Kimi-K3
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||
SERVED_MODEL_NAME=kimi-k3
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.60.5
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- 并行度（固定，勿改）----
|
||||
TP=32
|
||||
DP=1
|
||||
EP_SIZE=32
|
||||
|
||||
# ---- Multi-node topology (D 组 = decode 4 节点) ----
|
||||
NNODES=4
|
||||
NODE_HOSTS="174.1.60.5 174.1.60.6 174.1.60.7 174.1.60.8"
|
||||
NODE_SSH_USER=root
|
||||
LOCAL_NODE_RANK=0
|
||||
MASTER_IP=174.1.60.5
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma NCCL_SOCKET_IFNAME=bond1 GLOO_SOCKET_IFNAME=bond1 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1 SGLANG_ENABLE_REQUEST_HEADER_OVERRIDES=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro /tmp/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:ro /data/flashkda_deploy/wheels:/mc_wheels:ro"
|
||||
|
||||
# BOOTSTRAP: 打补丁 → 装 flashkda + mooncake wheel → 按 rank 设 SGLANG_HOST_IP → 启动 decode。
|
||||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && pip install /flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl --no-deps -q && pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && export SGLANG_HOST_IP=\"174.1.60.$((5 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --disaggregation-mode decode --host 0.0.0.0 --port ${PORT}"
|
||||
60
deploy/profiles/pro6000/kimi3_pro6000_pd_prefill.env
Normal file
60
deploy/profiles/pro6000/kimi3_pro6000_pd_prefill.env
Normal file
@ -0,0 +1,60 @@
|
||||
# Kimi-K3 PD 分离 - P 组 (prefill) deployment profile (4x RTX 6000D).
|
||||
# Nodes: 174.1.60.1~4 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||
#
|
||||
# 这是 PD (Prefill/Decode) 分离部署的 P 组 (prefill) 端 profile。
|
||||
# 配套 D 组 profile: kimi3_pro6000_pd_decode.env
|
||||
# 配套编排脚本: experiments/pro6000/kimi3_pro6000_pd_rdma/deploy_pd.sh
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - 传输后端 mooncake + MOONCAKE_PROTOCOL=rdma（计算网 mlx5_0~3，4 链路）
|
||||
# - 必须关掉 PYTORCH_CUDA_ALLOC_CONF=expandable_segments
|
||||
# （实测: expandable_segments 分配的 GPU 段 mooncake RDMA 注册失败
|
||||
# Bad address [14]，见飞书文档 PD 章节 & GitHub kvcache-ai/Mooncake#2511）
|
||||
# - --disaggregation-ib-device mlx5_0~3 让 mooncake 走计算网 RDMA
|
||||
# - NCCL/GLOO 走 bond1（计算网），NCCL_IB_HCA=mlx5_0..3
|
||||
# - P 组先启动，D 组后启动（见 deploy_pd.sh）
|
||||
# - mooncake master 需先在 174.1.60.1 运行（MOONCAKE_MASTER=174.1.60.1:50051）
|
||||
# - 容器挂载 /mc_wheels 并在 BOOTSTRAP 里 pip install mooncake 0.3.12.post1
|
||||
# （镜像自带 0.3.11.post1 无 dmabuf 修复，必须升级）
|
||||
#
|
||||
# Model-team only. Ops only run benchmark against the served router URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=kimi3_pro6000_pd_prefill
|
||||
MODEL_NAME=Kimi-K3
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||
SERVED_MODEL_NAME=kimi-k3
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.60.1
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- 并行度（固定，勿改）----
|
||||
TP=32
|
||||
DP=1
|
||||
EP_SIZE=32
|
||||
|
||||
# ---- Multi-node topology (P 组 = prefill 4 节点) ----
|
||||
NNODES=4
|
||||
NODE_HOSTS="174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4"
|
||||
NODE_SSH_USER=root
|
||||
LOCAL_NODE_RANK=0
|
||||
MASTER_IP=174.1.60.1
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma NCCL_SOCKET_IFNAME=bond1 GLOO_SOCKET_IFNAME=bond1 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro /tmp/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:ro /data/flashkda_deploy/wheels:/mc_wheels:ro"
|
||||
|
||||
# BOOTSTRAP: 打补丁 → 装 flashkda + mooncake wheel → 按 rank 设 SGLANG_HOST_IP → 启动 prefill。
|
||||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && pip install /flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl --no-deps -q && pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && export SGLANG_HOST_IP=\"174.1.60.$((1 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --disaggregation-mode prefill --linear-attn-prefill-backend flashkda --host 0.0.0.0 --port ${PORT}"
|
||||
54
deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env
Normal file
54
deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env
Normal file
@ -0,0 +1,54 @@
|
||||
# Kimi-K3 SGLang multi-node TP=32 EP=32 deployment profile (4x RTX 6000D).
|
||||
# Nodes: 174.1.60.5~8 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||
#
|
||||
# 关键点（实测踩坑，勿随意改）:
|
||||
# - MoE 后端必须 marlin（K3 的 MXFP4 缩放因子为 uint8，DeepGEMM 只接受 fp32/UE8M0）
|
||||
# - RoCE: NCCL_IB_HCA=mlx5_0..3（4 张独立卡, 10.100.21-24/24, RoCEv2 GID index 3）
|
||||
# 实测 32-rank 117MB allreduce 2.4ms；勿用 mlx5_bond_0（仅 4.5GB/s）
|
||||
# - 容器必须 --ulimit memlock=-1（否则 ibv_create_cq 报 Cannot allocate memory）
|
||||
# - 不要设 NCCL_ALGO=TREE（CUDA graph 捕获报 "NCCL error: invalid usage"）
|
||||
# - 首次请求含 ~110s Triton JIT 编译，属正常现象，预热一次后回落
|
||||
# - flashkda 可选后端（与 triton prefill 性能等价）见 docs/KIMI_K3_DEPLOY.md 附录
|
||||
#
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=kimi3_pro6000_sglang_tp32ep32
|
||||
MODEL_NAME=Kimi-K3
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||
SERVED_MODEL_NAME=kimi-k3
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.60.5
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- 并行度（固定，勿改）----
|
||||
TP=32
|
||||
DP=1
|
||||
EP_SIZE=32
|
||||
|
||||
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||
NNODES=4
|
||||
NODE_HOSTS="174.1.60.5 174.1.60.6 174.1.60.7 174.1.60.8"
|
||||
NODE_SSH_USER=root
|
||||
LOCAL_NODE_RANK=0
|
||||
MASTER_IP=174.1.60.5
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="NCCL_SOCKET_IFNAME=bond0 GLOO_SOCKET_IFNAME=bond0 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro"
|
||||
|
||||
# BOOTSTRAP 在容器内执行: 打 sm_120 补丁 → 按节点 rank 计算 SGLANG_HOST_IP → 启动 sglang。
|
||||
# SGLANG_HOST_IP 必须为本节点实际 IP（174.1.60.5~8 = 5 + NODE_RANK）。
|
||||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && export SGLANG_HOST_IP=\"174.1.60.$((5 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --host 0.0.0.0 --port ${PORT}"
|
||||
@ -0,0 +1,61 @@
|
||||
# Kimi-K3 SGLang multi-node TP=32 EP=32 deployment profile — 实例 B (DP=2 的第二个实例).
|
||||
# Nodes: 174.1.60.1~4 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||
#
|
||||
# DP=2 方案 B：两个独立实例（实例 A: .5-.8, 实例 B: .1-.4），各自 TP32×EP32，
|
||||
# 前面用 sglang_router（--worker-urls）做负载均衡。实例间零通信。
|
||||
#
|
||||
# 与实例 A（kimi3_pro6000_sglang_tp32ep32）的差异仅在于节点拓扑：
|
||||
# - NODE_HOSTS = .1-.4
|
||||
# - MASTER_IP = 174.1.60.1（从 .1 执行部署，LOCAL_NODE_RANK=0 本地起 rank0）
|
||||
# - SGLANG_HOST_IP = 174.1.60.$(1 + NODE_RANK)
|
||||
# 其余并行度/网络配置与实例 A 一致。
|
||||
#
|
||||
# 关键点（与实例 A 相同，勿随意改）:
|
||||
# - MoE 后端必须 marlin（K3 的 MXFP4 缩放因子为 uint8，DeepGEMM 只接受 fp32/UE8M0）
|
||||
# - RoCE: NCCL_IB_HCA=mlx5_0..3（4 张独立卡, 10.100.21-24/24, RoCEv2 GID index 3）
|
||||
# - 容器必须 --ulimit memlock=-1（否则 ibv_create_cq 报 Cannot allocate memory）
|
||||
# - 不要设 NCCL_ALGO=TREE
|
||||
# - --disable-radix-cache 关闭前缀缓存（bench 测量纯净）
|
||||
#
|
||||
# Model-team only. Ops only run `python -m sskj.bench` against the router URL.
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=kimi3_pro6000_sglang_tp32ep32_instB
|
||||
MODEL_NAME=Kimi-K3
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||
SERVED_MODEL_NAME=kimi-k3
|
||||
PORT=30000
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_HOST=174.1.60.1
|
||||
HEALTH_WAIT_S=2400
|
||||
CONTAINER_PYTHON=python3
|
||||
|
||||
# ---- 并行度（固定，勿改）----
|
||||
TP=32
|
||||
DP=1
|
||||
EP_SIZE=32
|
||||
|
||||
# ---- Multi-node topology (实例 B = .1-.4; rank 0 exposes the HTTP API) ----
|
||||
NNODES=4
|
||||
NODE_HOSTS="174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4"
|
||||
NODE_SSH_USER=root
|
||||
LOCAL_NODE_RANK=0
|
||||
MASTER_IP=174.1.60.1
|
||||
DIST_PORT=20000
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="NCCL_SOCKET_IFNAME=bond0 GLOO_SOCKET_IFNAME=bond0 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro"
|
||||
|
||||
# BOOTSTRAP 在容器内执行: 打 sm_120 补丁 → 按节点 rank 计算 SGLANG_HOST_IP → 启动 sglang。
|
||||
# SGLANG_HOST_IP 必须为本节点实际 IP（174.1.60.1~4 = 1 + NODE_RANK）。
|
||||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && export SGLANG_HOST_IP=\"174.1.60.$((1 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --host 0.0.0.0 --port ${PORT}"
|
||||
26
deploy/profiles/pro6000/qwen3_235b_pro6000_sglang_tp8.env
Normal file
26
deploy/profiles/pro6000/qwen3_235b_pro6000_sglang_tp8.env
Normal file
@ -0,0 +1,26 @@
|
||||
# Qwen3-235B-A22B SGLang TP=8 profile on a single RTX 6000D node (8 GPUs).
|
||||
# NVIDIA counterpart of p800/qwen3_235b_p800_sglang_tp8 (same workload for
|
||||
# comparability: isl=2048 osl=2048 c=16).
|
||||
|
||||
PLATFORM=pro6000
|
||||
EXPERIMENT=qwen3_235b_pro6000_sglang_tp8
|
||||
MODEL_NAME=Qwen3-235B-A22B
|
||||
ENGINE=sglang
|
||||
RUNTIME=docker
|
||||
DOCKER_IMAGE=sglang-sm120-dsv4:0.5.15.post1-fi0.6.14-sm120fix1
|
||||
CONTAINER_NAME=qwen3_235b_pro6000_sglang_tp8
|
||||
MODEL_PATH=/data/6000D/Qwen3-235B-A22B
|
||||
SERVED_MODEL_NAME=/data/6000D/Qwen3-235B-A22B
|
||||
PORT=30010
|
||||
HEALTH_PATH=/health
|
||||
HEALTH_WAIT_S=1800
|
||||
CONTAINER_PYTHON=python3
|
||||
TP=8
|
||||
|
||||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||
ENGINE_ENV="PYTHONUNBUFFERED=1 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True"
|
||||
|
||||
DOCKER_FLAGS="--gpus all --privileged --ipc=host --network host --ulimit memlock=-1 --ulimit stack=67108864"
|
||||
VOLUMES="/data:/data"
|
||||
|
||||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --trust-remote-code --tp-size ${TP} --mem-fraction-static 0.9 --context-length 8192 --max-running-requests 16 --host 0.0.0.0 --port ${PORT}"
|
||||
91
deploy/verify_profile.sh
Normal file
91
deploy/verify_profile.sh
Normal file
@ -0,0 +1,91 @@
|
||||
#!/bin/bash
|
||||
# verify_profile.sh —— 防漂移核验：运行中容器 vs 仓库 profile 声明
|
||||
# 在目标服务器上运行（需 docker 读权限，无需 GPU）。用法：
|
||||
# bash verify_profile.sh deploy/profiles/pro6000/glm53_nvfp4_pro6000_sglang_tp2pp4.env
|
||||
# bash verify_profile.sh <profile.env> [container_name] # container_name 缺省取 profile 的 CONTAINER_NAME
|
||||
# 核验项：①容器在跑 ②镜像 registry digest ③launch 参数（token 级比对）④端口监听。
|
||||
# PD 链角色的参数在挂载的 launch 脚本内（docker Args 只有 bash /smoke_launch.sh），
|
||||
# 第③项自动跳过并提示改用库内脚本 md5 对比（scripts/ 目录各脚本头注释有 md5）。
|
||||
set -uo pipefail
|
||||
|
||||
PROFILE="$1"
|
||||
[ -r "$PROFILE" ] || { echo "FATAL: 无法读取 profile: $PROFILE"; exit 2; }
|
||||
# profile 是声明式清单：BOOTSTRAP 行引用 ${LAUNCH_ARGS}（定义在后），set -u 下直接
|
||||
# source 会炸；临时关 -u。source 完成后 LAUNCH_ARGS 已是全部变量展开后的实参串。
|
||||
set +u
|
||||
# shellcheck disable=SC1090
|
||||
source "$PROFILE"
|
||||
set -u
|
||||
CONT="${2:-${CONTAINER_NAME:-}}"
|
||||
|
||||
fail=0
|
||||
section() { printf '\n== %s ==\n' "$1"; }
|
||||
|
||||
section "容器状态"
|
||||
if [ -z "$CONT" ]; then echo "FATAL: profile 未定义 CONTAINER_NAME 且未显式传入"; exit 2; fi
|
||||
if docker ps --format '{{.Names}}' | grep -qx "$CONT"; then
|
||||
echo "OK $CONT 在跑"
|
||||
else
|
||||
echo "FAIL $CONT 未在运行"; docker ps -a --filter "name=$CONT" --format '{{.Names}} {{.Status}}' | head -3
|
||||
exit 1
|
||||
fi
|
||||
|
||||
section "镜像 digest"
|
||||
if [ -n "${DOCKER_IMAGE_DIGEST:-}" ]; then
|
||||
# RepoDigests 在 image 对象上：container 先取 image ID，再 image inspect
|
||||
img_id=$(docker inspect --format '{{.Image}}' "$CONT" 2>/dev/null)
|
||||
repo_digests=$(docker image inspect --format '{{join .RepoDigests "\n"}}' "$img_id" 2>/dev/null)
|
||||
digest_hex="${DOCKER_IMAGE_DIGEST#sha256:}"
|
||||
if echo "$repo_digests" | grep -q "$digest_hex"; then
|
||||
echo "OK registry digest 一致: ${DOCKER_IMAGE_DIGEST}"
|
||||
elif [ "$img_id" = "sha256:${digest_hex}" ]; then
|
||||
echo "OK image ID 一致: ${DOCKER_IMAGE_DIGEST}"
|
||||
else
|
||||
echo "FAIL digest 不一致"
|
||||
echo " profile: ${DOCKER_IMAGE_DIGEST}"
|
||||
echo " 实际 image ID: ${img_id}"
|
||||
echo " 实际 RepoDigests: $(echo "$repo_digests" | head -2 | tr '\n' ' ')"
|
||||
fail=1
|
||||
fi
|
||||
else
|
||||
echo "SKIP profile 未定义 DOCKER_IMAGE_DIGEST"
|
||||
fi
|
||||
|
||||
section "启动参数"
|
||||
args=$(docker inspect --format '{{join .Args " "}}' "$CONT" 2>/dev/null)
|
||||
if echo "$args" | grep -q launch_server; then
|
||||
actual=$(echo "$args" | sed 's/.*launch_server //')
|
||||
expected="${LAUNCH_ARGS:-}"
|
||||
tr ' ' '\n' <<<"$expected" | sed '/^$/d' | sort > /tmp/vp_expected.$$
|
||||
tr ' ' '\n' <<<"$actual" | sed '/^$/d' | sort > /tmp/vp_actual.$$
|
||||
if diff -q /tmp/vp_expected.$$ /tmp/vp_actual.$$ >/dev/null; then
|
||||
echo "OK 参数一致（token 比对，共 $(wc -l < /tmp/vp_expected.$$) 项）"
|
||||
else
|
||||
echo "FAIL 参数有漂移（< profile 声明 / > 容器实际）:"
|
||||
diff /tmp/vp_expected.$$ /tmp/vp_actual.$$ | sed 's/^/ /'
|
||||
fail=1
|
||||
fi
|
||||
rm -f /tmp/vp_expected.$$ /tmp/vp_actual.$$
|
||||
else
|
||||
echo "SKIP docker Args 为 '$args' —— 参数在挂载的 launch 脚本内"
|
||||
echo " 改用库内脚本 md5 对比：experiments/.../scripts/ 各脚本头注释"
|
||||
fi
|
||||
|
||||
section "端口 ${PORT:-?}"
|
||||
if [ -n "${PORT:-}" ]; then
|
||||
if ss -tln | grep -q ":${PORT} "; then
|
||||
echo "OK :${PORT} 在监听"
|
||||
else
|
||||
echo "FAIL :${PORT} 未监听"; fail=1
|
||||
fi
|
||||
if [ -n "${HEALTH_PATH:-}" ]; then
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' "http://localhost:${PORT}${HEALTH_PATH}" 2>/dev/null)
|
||||
[ "$code" = "200" ] && echo "OK health ${HEALTH_PATH} -> 200" || { echo "FAIL health ${HEALTH_PATH} -> ${code}"; fail=1; }
|
||||
fi
|
||||
else
|
||||
echo "SKIP profile 未定义 PORT"
|
||||
fi
|
||||
|
||||
echo
|
||||
[ $fail -eq 0 ] && echo "VERDICT: PASS" || echo "VERDICT: DRIFT DETECTED"
|
||||
exit $fail
|
||||
270
docs/KIMI_K3_DEPLOY.md
Normal file
270
docs/KIMI_K3_DEPLOY.md
Normal file
@ -0,0 +1,270 @@
|
||||
# Kimi-K3 部署手册（运维版）
|
||||
|
||||
> 面向运维的一键部署手册：按顺序复制粘贴命令即可完成 Kimi-K3 在 4 节点
|
||||
> RTX 6000D 集群上的部署。**不需要理解 SGLang 引擎参数。**
|
||||
> 部署配置统一在 `deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env`，
|
||||
> 压测统一走 `python -m sskj.bench`（见 `ops/README.md`）。
|
||||
|
||||
## 0. 概述
|
||||
|
||||
| 项 | 值 |
|
||||
|---|---|
|
||||
| 模型 | Kimi-K3（Moonshot AI，2.8T 参数 MXFP4 量化，~1.5TB） |
|
||||
| 节点 | 6000D-5~8 = 174.1.60.5~8（每节点 8× RTX 6000D 85GB，共 32 卡） |
|
||||
| 镜像 | `lmsysorg/sglang:kimi-k3`（专用镜像，普通 sglang 镜像不识别该模型架构） |
|
||||
| 并行 | TP=32 × EP=32（跨 4 节点） |
|
||||
| 服务地址 | `http://174.1.60.5:30000`（OpenAI 兼容） |
|
||||
| 模型 ID | `kimi-k3` |
|
||||
| 部署方式 | `python -m sskj.deploy start --profile pro6000/kimi3_pro6000_sglang_tp32ep32` |
|
||||
|
||||
## 1. 前置检查（每步都过再继续）
|
||||
|
||||
```bash
|
||||
# 1.1 4 节点 8 卡全部空闲（应全部显示 0 MiB / 0%）
|
||||
for i in 5 6 7 8; do echo "== 174.1.60.$i"; ssh 174.1.60.$i 'nvidia-smi --query-gpu=index,memory.used --format=csv,noheader'; done
|
||||
# 注意：如显示被占用，联系模型团队确认（其他用户的作业需先让出）
|
||||
|
||||
# 1.2 磁盘空间（模型 1.5TB + 运行余量，/data 需 ≥ 2TB 可用）
|
||||
ssh 174.1.60.5 'df -h /data'
|
||||
|
||||
# 1.3 RoCE 网卡就绪（应输出 4 个 ACTIVE 且 link_layer=Ethernet）
|
||||
ssh 174.1.60.5 'cat /sys/class/infiniband/mlx5_*/ports/1/link_layer; cat /sys/class/infiniband/mlx5_*/ports/1/state'
|
||||
|
||||
# 1.4 节点互信（174.1.60.5 免密登录其余节点；若失败执行步骤 1.5）
|
||||
ssh 174.1.60.5 'for i in 6 7 8; do ssh -o StrictHostKeyChecking=no 174.1.60.$i hostname; done'
|
||||
|
||||
# 1.5 若 1.4 失败，在 174.1.60.5 上执行（把 .5 的 root 公钥发到各节点）
|
||||
ssh 174.1.60.5 'for i in 6 7 8; do ssh-copy-id -o StrictHostKeyChecking=no root@174.1.60.$i; done'
|
||||
```
|
||||
|
||||
## 2. 模型下载（仅首次，~1.5TB，需数小时）
|
||||
|
||||
模型权重放 `/data/hf_models/Kimi-K3`（4 节点同一路径，本集群已就绪则跳过本节）。
|
||||
|
||||
```bash
|
||||
# 在 174.1.60.5 上执行（国内走 ModelScope 最快；modelscope CLI 需先 pip install modelscope）
|
||||
pip3 install -q modelscope
|
||||
mkdir -p /data/hf_models
|
||||
nohup modelscope download --model moonshotai/Kimi-K3 \
|
||||
--local_dir /data/hf_models/Kimi-K3 > /data/hf_models/download_k3.log 2>&1 &
|
||||
# 查看进度
|
||||
tail -f /data/hf_models/download_k3.log
|
||||
```
|
||||
|
||||
下载完成后校验（96 个分片，共约 1.5TB）：
|
||||
|
||||
```bash
|
||||
ssh 174.1.60.5 'ls /data/hf_models/Kimi-K3/*.safetensors | wc -l; du -sh /data/hf_models/Kimi-K3'
|
||||
# 期望：96（safetensors 数量）、1.5T（总大小）
|
||||
```
|
||||
|
||||
**分发到其余 3 台**（.5 上执行；源盘 NVMe 读是瓶颈，聚合约 3.5GB/s）：
|
||||
|
||||
```bash
|
||||
ssh 174.1.60.5 'for i in 6 7 8; do rsync -aH --partial /data/hf_models/Kimi-K3/ 174.1.60.$i:/data/hf_models/Kimi-K3/ & done; wait'
|
||||
```
|
||||
|
||||
> 每节点需约 1.5T 空闲磁盘（`df -h /data` 确认）。
|
||||
|
||||
## 3. 镜像准备（已拉取则跳过）
|
||||
|
||||
```bash
|
||||
# 4 节点并行拉取（约 9.6GB；如 Docker Hub 直连失败，各机 daemon.json 需配镜像加速：
|
||||
# "registry-mirrors": ["https://docker.m.daocloud.io", "https://docker.xuanyuan.me"]）
|
||||
for i in 5 6 7 8; do ssh 174.1.60.$i 'docker pull lmsysorg/sglang:kimi-k3' & done; wait
|
||||
|
||||
# 验证镜像存在
|
||||
ssh 174.1.60.5 'docker images lmsysorg/sglang:kimi-k3 --format "{{.Repository}}:{{.Tag}} {{.Size}}"'
|
||||
```
|
||||
|
||||
## 4. 补丁与依赖分发
|
||||
|
||||
Kimi-K3 镜像在消费级 Blackwell（sm_120，RTX 6000D）上有一个内核不兼容点，
|
||||
需要打补丁（补丁源文件在本仓库 `platforms/patches/pro6000/kimi_k3/patch_k3_sm120.py`，
|
||||
修复 attn_res 融合内核误用数据中心 Blackwell 专属的 tcgen05 指令问题）。
|
||||
|
||||
```bash
|
||||
# 在 174.1.60.5 上，把补丁分发到 4 节点 /tmp（部署时容器自动挂载并执行）
|
||||
ssh 174.1.60.5 '
|
||||
for i in 5 6 7 8; do
|
||||
scp /data/yy/sskj/platforms/patches/pro6000/kimi_k3/patch_k3_sm120.py 174.1.60.$i:/tmp/patch_k3_sm120.py
|
||||
done
|
||||
for i in 5 6 7 8; do ssh 174.1.60.$i "sha256sum /tmp/patch_k3_sm120.py"; done
|
||||
'
|
||||
# 4 台输出的 sha256 必须一致（内容校验）
|
||||
```
|
||||
|
||||
## 5. 一键部署
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj
|
||||
|
||||
# 5.1 先预览要执行的命令（不实际启动）
|
||||
PYTHONPATH=src python3 -m sskj.deploy start \
|
||||
--profile pro6000/kimi3_pro6000_sglang_tp32ep32 --dry-run
|
||||
|
||||
# 5.2 正式启动（4 节点同时拉起，耗时 8~12 分钟：模型加载 + CUDA graph 捕获）
|
||||
PYTHONPATH=src python3 -m sskj.deploy start \
|
||||
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||
|
||||
# 5.3 查看状态
|
||||
PYTHONPATH=src python3 -m sskj.deploy status \
|
||||
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||
```
|
||||
|
||||
就绪判定（满足其一）：
|
||||
- 步骤 5.3 `status` 显示容器 Up 且健康检查通过
|
||||
- 或手动验证：`curl -s http://174.1.60.5:30000/health` 返回 `{"status":"ok"}`
|
||||
|
||||
## 6. 验证服务
|
||||
|
||||
```bash
|
||||
# 6.1 模型列表
|
||||
curl -s http://174.1.60.5:30000/v1/models | head -c 300
|
||||
# 应包含 "id": "kimi-k3"
|
||||
|
||||
# 6.2 推理冒烟（正确性：23×47 应算得 1081，输出可能带思考标签属正常）
|
||||
curl -s -m 300 http://174.1.60.5:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "kimi-k3", "messages": [{"role": "user", "content": "What is 23*47? Answer briefly."}], "max_tokens": 128, "temperature": 0.1}'
|
||||
|
||||
# 6.3 服务日志（确认无 ERROR）
|
||||
ssh 174.1.60.5 'docker logs kimi3_pro6000_sglang_tp32ep32_node0 2>&1 | tail -20'
|
||||
```
|
||||
|
||||
> ⚠️ **首次请求很慢（TTFT 可达 100+ 秒）是正常现象**：Triton/FlashKDA 内核首次
|
||||
> JIT 编译。发一个任意请求预热后，后续请求恢复正常（16K 输入 TTFT ~6s）。
|
||||
|
||||
## 7. 停止 / 重启
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj
|
||||
|
||||
# 停止（4 节点容器全部移除）
|
||||
PYTHONPATH=src python3 -m sskj.deploy stop \
|
||||
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||
|
||||
# 重启 = 再次执行 5.2 的 start 命令（幂等，会先清旧容器）
|
||||
|
||||
# 手动兜底（start/stop 异常时）：4 节点各执行
|
||||
for i in 5 6 7 8; do ssh 174.1.60.$i 'docker rm -f kimi3_pro6000_sglang_tp32ep32_node*'; done
|
||||
```
|
||||
|
||||
## 8. 性能自检（可选）
|
||||
|
||||
```bash
|
||||
# 稳态基准：16K 输入 TTFT / prefill 吞吐 / TPOT（先跑一次丢弃 JIT 冷启动）
|
||||
cd /data/yy/sskj
|
||||
scp -o ConnectTimeout=10 /data/flashkda_deploy/bench_16k_steady.py 174.1.60.5:/tmp/ 2>/dev/null || true
|
||||
ssh 174.1.60.5 'python3 /tmp/bench_16k_steady.py 3'
|
||||
# 参考值（稳态）：TTFT ~6s / prefill ~2700 tok/s / TPOT ~39ms / decode ~26 tok/s
|
||||
```
|
||||
|
||||
正式压测（矩阵 + 自适应并发搜索）走 bench 层：
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||
DRY_RUN=1 bash run_adaptive_concurrency_add16.sh # 先看计划
|
||||
bash run_adaptive_concurrency_add16.sh # 正式跑（放 tmux）
|
||||
```
|
||||
|
||||
## 9. 故障排查速查
|
||||
|
||||
| 现象 | 原因 | 处理 |
|
||||
|---|---|---|
|
||||
| 首次请求 TTFT 100+ 秒 | Triton/FlashKDA 内核首次 JIT 编译 | 正常现象，预热一次即可 |
|
||||
| `Not enough GPU memory for hybrid mamba state cache` | 某节点显存被其他作业占用 | 检查 4 节点 `nvidia-smi`，确认 8 卡全空闲后重启部署 |
|
||||
| `ibv_create_cq failed: Cannot allocate memory` | 容器 memlock 限制 | 部署配置已带 `--ulimit memlock=-1`，检查 profile 未被改动 |
|
||||
| `NCCL error: invalid usage`（graph 捕获时） | 误设了 `NCCL_ALGO=TREE` | 确认环境变量里没有 NCCL_ALGO |
|
||||
| 启动后 `/health` 一直不通 | 分布式初始化失败/节点未就绪 | 看 master 日志；确认 4 节点 GPU 全空闲后 `stop` + 重新 `start` |
|
||||
| 推理结果带 `<\|open\|>think` 标签 | K3 思考型模型正常输出格式 | 非故障；如需精简可调 chat template（联系模型团队） |
|
||||
| 并发压测时部分请求失败 | flashkda 后端已知问题（未修复） | 当前默认 triton 后端无此问题；如误用 flashkda 见附录 |
|
||||
|
||||
## 附录 A：flashkda 可选后端（默认不启用）
|
||||
|
||||
FlashKDA（MoonshotAI CUTLASS KDA 内核，支持 sm_120）可作为 KDA prefill 的备选后端。
|
||||
实测与默认 triton prefill **性能等价**（16K prefill 2693 vs 2695 tok/s），仅作备份用途。
|
||||
|
||||
启用方式（需模型团队协助）：
|
||||
1. 构建 wheel（脚本见 `/data/flashkda_deploy/01_build_flashkda.sh`，产物
|
||||
`flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl` 分发到 4 节点 `/tmp/`）
|
||||
2. profile 增加 wheel 挂载与安装步骤，并在 LAUNCH_ARGS 加
|
||||
`--linear-attn-prefill-backend flashkda`
|
||||
|
||||
已知问题：flashkda 后端在 4/8 并发压测下服务异常（纯 triton 无此问题），修复前不建议生产使用。
|
||||
|
||||
## 附录 B：PD 分离部署（MoonCake RDMA，8 节点）
|
||||
|
||||
非 PD 单组部署之外的另一种形态：**Prefill/Decode 分离**（P 组 4 节点做 prefill，D 组 4 节点做 decode），KV 传输走 **MoonCake RDMA**。适合长上下文、吞吐优先的场景。
|
||||
|
||||
### B.1 拓扑
|
||||
|
||||
```
|
||||
┌───────────────────┐
|
||||
client ──31000──▶ │ router (MiniLB) │ 174.1.60.5
|
||||
└───────┬───────────┘
|
||||
┌───────▼───────────┐
|
||||
P 组 (prefill) │ 174.1.60.1~4 │ TP32×EP32，4 节点 32 卡
|
||||
└── prefill 计算 │ 计算完产出 KV │
|
||||
└───────┬───────────┘
|
||||
┌───────▼───────────┐
|
||||
MoonCake RDMA│ mlx5_0~3 (RoCE) │ 4 链路并行
|
||||
└───────┬───────────┘
|
||||
┌───────▼───────────┐
|
||||
D 组 (decode) │ 174.1.60.5~8 │ TP32×EP32，4 节点 32 卡
|
||||
└── decode 生成 │ 接收 KV 逐 token │
|
||||
└───────────────────┘
|
||||
```
|
||||
|
||||
### B.2 运维准备（一次性，新环境才需执行）
|
||||
|
||||
已部署过的集群可跳过本节。步骤顺序：
|
||||
|
||||
1. **免密与文件就位**：确保 8 节点互信、docker 可用；把 `patch_k3_sm120.py`、`flash_kda-*.whl`、mooncake wheel 分发到 8 节点（路径见附录 B.2 下方说明）。
|
||||
2. **sskj 仓库**：`.5` 和 `.1` 都需 clone `/data/yy/sskj`（deploy_pd.sh 的 deploy 层命令依赖）。
|
||||
3. **拉镜像**：8 节点 `docker pull lmsysorg/sglang:kimi-k3`。
|
||||
4. **GPU 空闲**：`nvidia-smi` 全 0 MiB，否则联系模型团队确认。
|
||||
|
||||
详细命令见 `experiments/pro6000/kimi3_pro6000_pd_rdma/README.md` 的「运维一键部署」章节（含每步的可执行命令与验证）。
|
||||
|
||||
### B.3 一键部署
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj/experiments/pro6000/kimi3_pro6000_pd_rdma
|
||||
bash deploy_pd.sh start # mc-master → P 组 → D 组 → router
|
||||
bash deploy_pd.sh status
|
||||
bash deploy_pd.sh stop
|
||||
bash deploy_pd.sh restart
|
||||
```
|
||||
|
||||
### B.4 验证
|
||||
|
||||
```bash
|
||||
# 端到端（router 自动双发 P/D）
|
||||
curl -s http://174.1.60.5:31000/generate -H "Content-Type: application/json" \
|
||||
-d '{"text":"Hello","sampling_params":{"max_new_tokens":16}}'
|
||||
```
|
||||
|
||||
长输入验证（2026-08-11 实测）：1680 / 5280 tokens 输入均正常，P 组 prefill ~112 tok/s，D 组 decode ~26 tok/s。
|
||||
|
||||
### B.5 关键注意事项（务必遵守）
|
||||
|
||||
1. **禁止设置 `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`**
|
||||
expandable_segments 分配的 GPU 段，mooncake RDMA 注册报 `Bad address [14]`
|
||||
（GitHub kvcache-ai/Mooncake **#2511**）。移除后 KV 注册从 2376 条失败 → 0。
|
||||
2. **P 组必须先于 D 组启动**（prefill 需先注册 bootstrap）。
|
||||
3. **容器内必须升级 mooncake 到 0.3.12.post1**（镜像自带 0.3.11.post1 无 dmabuf 修复 #2035）。
|
||||
wheel 在 `/data/flashkda_deploy/wheels/`，profile 的 BOOTSTRAP 已含 pip install。
|
||||
4. NCCL/GLOO 走计算网（`bond1` + `NCCL_IB_HCA=mlx5_0..3`），不要走管理网 bond0。
|
||||
5. mc-master（mooncake 元数据服务）需先在 174.1.60.1 运行（deploy_pd.sh 自动处理）。
|
||||
|
||||
### B.6 与单组部署的取舍
|
||||
|
||||
| 维度 | 单组 TP32×EP32 | PD 分离（MoonCake RDMA） |
|
||||
|---|---|---|
|
||||
| 节点 | 174.1.60.5~8（4 节点） | P 174.1.60.1~4 + D 174.1.60.5~8（8 节点） |
|
||||
| KV 传输 | 无（本地） | MoonCake RDMA（计算网 4 链路） |
|
||||
| 传输后端 | - | mooncake（必须 0.3.12.post1） |
|
||||
| 适用 | 单组吞吐、简单 | 长上下文、PD 分离 |
|
||||
| profile | kimi3_pro6000_sglang_tp32ep32 | kimi3_pro6000_pd_prefill / _pd_decode |
|
||||
| 编排 | `python -m sskj.deploy` | `deploy_pd.sh` |
|
||||
@ -0,0 +1 @@
|
||||
{"timestamp": "2026-07-29T11:50:47+08:00", "engine": "vllm", "tp": 4, "dp": 2, "mark": "Y", "isl": 1024, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 48, "saturation_concurrency": null, "stop_probe_concurrency": 48, "best_tps_concurrency": 48, "best_total_tps": 5069.798064365613, "last_total_tps": 5069.798064365613}
|
||||
@ -0,0 +1,7 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
| vllm | 4 | 2 | 1024 | 128 | TTFT_SLO_EXCEEDED | None | 48 | 5069.798064365613 | 48 |
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,10 @@
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":32.54043653002009,"request_tps":2.458479618925709,"input_tps":2517.483129779926,"output_tps":314.68539122249075,"total_tps":2832.1685210024166,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1178.376264986582,"ttft_p95_ms":1744.3892850889824,"ttft_p99_ms":1881.6810713254374,"tpot_p50_ms":41.60951500012371,"tpot_p95_ms":105.56267855407246,"tpot_p99_ms":60.98298653290148,"e2e_p50_ms":6521.07125998009,"e2e_p95_ms":8387.368027481714,"e2e_p99_ms":8883.328670103796,"itl_p50_ms":49.741589988116175,"itl_p95_ms":397.93538753292523,"itl_p99_ms":434.0583304758184,"validation_errors":[],"timestamp":"2026-07-29T13:31:58+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c16_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c16_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":54.806936430046335,"request_tps":2.919338507530309,"input_tps":2989.4026317110365,"output_tps":373.67532896387956,"total_tps":3363.077960674916,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1552.6748099946417,"ttft_p95_ms":2799.1436289856147,"ttft_p99_ms":4081.1345711071,"tpot_p50_ms":74.8756598423631,"tpot_p95_ms":168.06996610903298,"tpot_p99_ms":94.46507637603501,"e2e_p50_ms":10864.70078502316,"e2e_p95_ms":12499.93150093942,"e2e_p99_ms":13733.595000598576,"itl_p50_ms":61.65496492758393,"itl_p95_ms":426.26925505464897,"itl_p99_ms":454.9201102578081,"validation_errors":[],"timestamp":"2026-07-29T13:44:25+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":2,"gain_pct":18.745687,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c32_i1024_o128_a2.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c32_i1024_o128_a2.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":56.52760181995109,"request_tps":4.245713461618911,"input_tps":4347.610584697765,"output_tps":543.4513230872207,"total_tps":4891.061907784986,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1677.9174199909903,"ttft_p95_ms":4625.563451409107,"ttft_p99_ms":5456.264773586299,"tpot_p50_ms":78.36960055122856,"tpot_p95_ms":163.22612201465745,"tpot_p99_ms":97.57853847849508,"e2e_p50_ms":11642.064979940187,"e2e_p95_ms":13726.657478511333,"e2e_p99_ms":14860.793732419837,"itl_p50_ms":63.670160016044974,"itl_p95_ms":437.71982495673,"itl_p99_ms":467.8959050215781,"validation_errors":[],"timestamp":"2026-07-29T13:45:44+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":45.434092,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c48_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c48_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":64.80073764000554,"request_tps":1.234553847896494,"input_tps":1264.1831402460098,"output_tps":316.04578506150244,"total_tps":1580.2289253075123,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1549.8129050247371,"ttft_p95_ms":2982.683588570216,"ttft_p99_ms":3445.983611260308,"tpot_p50_ms":43.74364400008584,"tpot_p95_ms":103.21268538372851,"tpot_p99_ms":61.282857058784366,"e2e_p50_ms":12898.023929970805,"e2e_p95_ms":15419.191638979826,"e2e_p99_ms":17197.13726384449,"itl_p50_ms":56.227469933219254,"itl_p95_ms":405.5930419708602,"itl_p99_ms":449.0739406796638,"validation_errors":[],"timestamp":"2026-07-29T13:47:09+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c16_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c16_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":92.86755778000224,"request_tps":1.7228836832236996,"input_tps":1764.2328916210683,"output_tps":441.0582229052671,"total_tps":2205.2911145263356,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1665.0969700422138,"ttft_p95_ms":5806.633543991484,"ttft_p99_ms":6654.958495234605,"tpot_p50_ms":59.006720019604856,"tpot_p95_ms":154.3634356120908,"tpot_p99_ms":85.91761102803582,"e2e_p50_ms":18778.37522997288,"e2e_p95_ms":21718.774544069303,"e2e_p99_ms":23658.405131483447,"itl_p50_ms":66.90277997404337,"itl_p95_ms":436.01773898117244,"itl_p99_ms":462.3979718168266,"validation_errors":[],"timestamp":"2026-07-29T13:49:05+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":39.555167,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c32_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c32_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":108.85908220999409,"request_tps":0.7348950439034235,"input_tps":752.5325249571057,"output_tps":376.26626247855285,"total_tps":1128.7987874356586,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1278.6862149951048,"ttft_p95_ms":2917.725130013423,"ttft_p99_ms":3352.0424618478864,"tpot_p50_ms":39.87036136009981,"tpot_p95_ms":87.56900736711123,"tpot_p99_ms":49.62658343923462,"e2e_p50_ms":21604.65751506854,"e2e_p95_ms":25136.398001946505,"e2e_p99_ms":26470.48831115825,"itl_p50_ms":56.55378999654204,"itl_p95_ms":379.1963019873945,"itl_p99_ms":450.32704603392625,"validation_errors":[],"timestamp":"2026-07-29T13:51:14+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c16_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c16_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":138.06450264994055,"request_tps":1.1588786178129828,"input_tps":1186.6917046404944,"output_tps":593.3458523202472,"total_tps":1780.0375569607418,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1695.9613200160675,"ttft_p95_ms":6016.787129081786,"ttft_p99_ms":6636.4030784810875,"tpot_p50_ms":48.92657508794765,"tpot_p95_ms":112.16928823729037,"tpot_p99_ms":63.561613842126874,"e2e_p50_ms":27838.76409503864,"e2e_p95_ms":31206.31007204647,"e2e_p99_ms":34183.30330188618,"itl_p50_ms":66.48826005402952,"itl_p95_ms":423.58878504019225,"itl_p99_ms":455.9304605936631,"validation_errors":[],"timestamp":"2026-07-29T13:53:54+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":57.693078,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c32_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c32_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":182.90077979990747,"request_tps":0.4373956201144664,"input_tps":447.8931149972136,"output_tps":447.8931149972136,"total_tps":895.7862299944272,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1284.1967650456354,"ttft_p95_ms":3063.6929089669125,"ttft_p99_ms":3512.141903876327,"tpot_p50_ms":33.90796327957099,"tpot_p95_ms":72.344788964252,"tpot_p99_ms":39.7340261776677,"e2e_p50_ms":35819.9144549435,"e2e_p95_ms":39835.132474428974,"e2e_p99_ms":42287.27937493239,"itl_p50_ms":56.46366497967392,"itl_p95_ms":58.916712971404195,"itl_p99_ms":419.9753348703962,"validation_errors":[],"timestamp":"2026-07-29T13:57:17+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c16_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c16_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":230.6752928699134,"request_tps":0.6936156794660714,"input_tps":710.2624557732571,"output_tps":710.2624557732571,"total_tps":1420.5249115465142,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1319.5603450294584,"ttft_p95_ms":2549.0444859780837,"ttft_p99_ms":3204.443412218243,"tpot_p50_ms":43.128577756587575,"tpot_p95_ms":95.46165655838493,"tpot_p99_ms":52.77950603142335,"e2e_p50_ms":45443.14777501859,"e2e_p95_ms":50835.57356991224,"e2e_p99_ms":55416.93294369965,"itl_p50_ms":64.92043007165194,"itl_p95_ms":410.08403194136906,"itl_p99_ms":455.18957001622766,"validation_errors":[],"timestamp":"2026-07-29T14:01:29+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":58.578561,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c32_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c32_i1024_o1024_a1.log"}
|
||||
{"timestamp":"2026-07-29T14:07:33+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"status":"FAILED","completed":0,"failed":240,"error_type":"benchmark failed and server restart failed","validation_errors":[],"gain_pct":null,"plateau_streak":0,"raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/raw_outputs/vllm_adaptive_c48_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-131331/tp4_dp2/logs/vllm_c48_i1024_o1024_a1.log"}
|
||||
@ -0,0 +1,4 @@
|
||||
{"timestamp":"2026-07-29T13:45:44+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":3,"search_cap":256,"max_successful_concurrency":48,"saturation_concurrency":null,"stop_probe_concurrency":48,"best_tps_concurrency":48,"best_total_tps":4891.061907784986,"last_total_tps":4891.061907784986}
|
||||
{"timestamp":"2026-07-29T13:49:05+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":2205.2911145263356,"last_total_tps":2205.2911145263356}
|
||||
{"timestamp":"2026-07-29T13:53:54+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":1780.0375569607418,"last_total_tps":1780.0375569607418}
|
||||
{"timestamp":"2026-07-29T14:07:33+08:00","engine":"vllm","tp":4,"dp":2,"mark":"Y","isl":1024,"osl":1024,"status":"FAILED","stop_reason":"BENCHMARK_FAILED","tested_points":3,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":48,"best_tps_concurrency":32,"best_total_tps":1420.5249115465142,"last_total_tps":1420.5249115465142}
|
||||
@ -0,0 +1,4 @@
|
||||
{"timestamp": "2026-07-29T13:45:44+08:00", "engine": "vllm", "tp": 4, "dp": 2, "mark": "Y", "isl": 1024, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 48, "saturation_concurrency": null, "stop_probe_concurrency": 48, "best_tps_concurrency": 48, "best_total_tps": 4891.061907784986, "last_total_tps": 4891.061907784986}
|
||||
{"timestamp": "2026-07-29T13:49:05+08:00", "engine": "vllm", "tp": 4, "dp": 2, "mark": "Y", "isl": 1024, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 2205.2911145263356, "last_total_tps": 2205.2911145263356}
|
||||
{"timestamp": "2026-07-29T13:53:54+08:00", "engine": "vllm", "tp": 4, "dp": 2, "mark": "Y", "isl": 1024, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 1780.0375569607418, "last_total_tps": 1780.0375569607418}
|
||||
{"timestamp": "2026-07-29T14:07:33+08:00", "engine": "vllm", "tp": 4, "dp": 2, "mark": "Y", "isl": 1024, "osl": 1024, "status": "FAILED", "stop_reason": "BENCHMARK_FAILED", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 48, "best_tps_concurrency": 32, "best_total_tps": 1420.5249115465142, "last_total_tps": 1420.5249115465142}
|
||||
@ -0,0 +1,10 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
| vllm | 4 | 2 | 1024 | 128 | TTFT_SLO_EXCEEDED | None | 48 | 4891.061907784986 | 48 |
|
||||
| vllm | 4 | 2 | 1024 | 256 | TTFT_SLO_EXCEEDED | None | 32 | 2205.2911145263356 | 32 |
|
||||
| vllm | 4 | 2 | 1024 | 512 | TTFT_SLO_EXCEEDED | None | 32 | 1780.0375569607418 | 32 |
|
||||
| vllm | 4 | 2 | 1024 | 1024 | BENCHMARK_FAILED | None | 32 | 1420.5249115465142 | 32 |
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260729-131331",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -0,0 +1,9 @@
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":55.30741832999047,"request_tps":1.4464605728418165,"input_tps":1481.17562659002,"output_tps":185.1469533237525,"total_tps":1666.3225799137726,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1662.2197499964386,"ttft_p95_ms":4819.2169445042955,"ttft_p99_ms":5303.001867405836,"tpot_p50_ms":69.14839937028981,"tpot_p95_ms":178.66644495692282,"tpot_p99_ms":106.0779383308969,"e2e_p50_ms":11560.410994978156,"e2e_p95_ms":13573.297397990245,"e2e_p99_ms":15121.684719539478,"itl_p50_ms":61.53274007374421,"itl_p95_ms":425.91814599581994,"itl_p99_ms":455.43587851687334,"validation_errors":[],"timestamp":"2026-07-29T13:44:25+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c16_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":56.53529765002895,"request_tps":0.7075225861127047,"input_tps":724.5031281794096,"output_tps":90.5628910224262,"total_tps":815.0660192018358,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1067.0159699511714,"ttft_p95_ms":1749.7485749830958,"ttft_p99_ms":1778.5544445144478,"tpot_p50_ms":81.06458795258746,"tpot_p95_ms":178.2429101647517,"tpot_p99_ms":108.41963742080505,"e2e_p50_ms":11601.409650000278,"e2e_p95_ms":12388.850279501636,"e2e_p99_ms":14515.659210094018,"itl_p50_ms":63.82342003053054,"itl_p95_ms":434.6582284895703,"itl_p99_ms":467.47689294978045,"validation_errors":[],"timestamp":"2026-07-29T13:45:43+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c8_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c8_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":64.80073764000554,"request_tps":1.234553847896494,"input_tps":1264.1831402460098,"output_tps":316.04578506150244,"total_tps":1580.2289253075123,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1549.8129050247371,"ttft_p95_ms":2982.683588570216,"ttft_p99_ms":3445.983611260308,"tpot_p50_ms":43.74364400008584,"tpot_p95_ms":103.21268538372851,"tpot_p99_ms":61.282857058784366,"e2e_p50_ms":12898.023929970805,"e2e_p95_ms":15419.191638979826,"e2e_p99_ms":17197.13726384449,"itl_p50_ms":56.227469933219254,"itl_p95_ms":405.5930419708602,"itl_p99_ms":449.0739406796638,"validation_errors":[],"timestamp":"2026-07-29T13:47:08+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c16_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":92.86755778000224,"request_tps":1.7228836832236996,"input_tps":1764.2328916210683,"output_tps":441.0582229052671,"total_tps":2205.2911145263356,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1665.0969700422138,"ttft_p95_ms":5806.633543991484,"ttft_p99_ms":6654.958495234605,"tpot_p50_ms":59.006720019604856,"tpot_p95_ms":154.3634356120908,"tpot_p99_ms":85.91761102803582,"e2e_p50_ms":18778.37522997288,"e2e_p95_ms":21718.774544069303,"e2e_p99_ms":23658.405131483447,"itl_p50_ms":66.90277997404337,"itl_p95_ms":436.01773898117244,"itl_p99_ms":462.3979718168266,"validation_errors":[],"timestamp":"2026-07-29T13:49:05+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":39.555167,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c32_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":108.85908220999409,"request_tps":0.7348950439034235,"input_tps":752.5325249571057,"output_tps":376.26626247855285,"total_tps":1128.7987874356586,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1278.6862149951048,"ttft_p95_ms":2917.725130013423,"ttft_p99_ms":3352.0424618478864,"tpot_p50_ms":39.87036136009981,"tpot_p95_ms":87.56900736711123,"tpot_p99_ms":49.62658343923462,"e2e_p50_ms":21604.65751506854,"e2e_p95_ms":25136.398001946505,"e2e_p99_ms":26470.48831115825,"itl_p50_ms":56.55378999654204,"itl_p95_ms":379.1963019873945,"itl_p99_ms":450.32704603392625,"validation_errors":[],"timestamp":"2026-07-29T13:51:13+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c16_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":138.06450264994055,"request_tps":1.1588786178129828,"input_tps":1186.6917046404944,"output_tps":593.3458523202472,"total_tps":1780.0375569607418,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1695.9613200160675,"ttft_p95_ms":6016.787129081786,"ttft_p99_ms":6636.4030784810875,"tpot_p50_ms":48.92657508794765,"tpot_p95_ms":112.16928823729037,"tpot_p99_ms":63.561613842126874,"e2e_p50_ms":27838.76409503864,"e2e_p95_ms":31206.31007204647,"e2e_p99_ms":34183.30330188618,"itl_p50_ms":66.48826005402952,"itl_p95_ms":423.58878504019225,"itl_p99_ms":455.9304605936631,"validation_errors":[],"timestamp":"2026-07-29T13:53:54+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":57.693078,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c32_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":181.1469587100437,"request_tps":0.44163037883541567,"input_tps":452.22950792746565,"output_tps":452.22950792746565,"total_tps":904.4590158549313,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1195.9480450605042,"ttft_p95_ms":1952.2496014775243,"ttft_p99_ms":1958.210011383053,"tpot_p50_ms":33.76438179858373,"tpot_p95_ms":73.23491950637455,"tpot_p99_ms":42.50489829753352,"e2e_p50_ms":35769.36641498469,"e2e_p95_ms":39653.583698987495,"e2e_p99_ms":44781.71492753317,"itl_p50_ms":56.47630000021309,"itl_p95_ms":58.98615997284651,"itl_p99_ms":420.2479849336669,"validation_errors":[],"timestamp":"2026-07-29T13:57:15+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c16_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":230.6752928699134,"request_tps":0.6936156794660714,"input_tps":710.2624557732571,"output_tps":710.2624557732571,"total_tps":1420.5249115465142,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1319.5603450294584,"ttft_p95_ms":2549.0444859780837,"ttft_p99_ms":3204.443412218243,"tpot_p50_ms":43.128577756587575,"tpot_p95_ms":95.46165655838493,"tpot_p99_ms":52.77950603142335,"e2e_p50_ms":45443.14777501859,"e2e_p95_ms":50835.57356991224,"e2e_p99_ms":55416.93294369965,"itl_p50_ms":64.92043007165194,"itl_p95_ms":410.08403194136906,"itl_p99_ms":455.18957001622766,"validation_errors":[],"timestamp":"2026-07-29T14:01:30+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":57.057964,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c32_i1024_o1024_a1.log"}
|
||||
{"timestamp":"2026-07-29T14:04:55+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"status":"FAILED","completed":0,"failed":240,"error_type":"benchmark failed and server restart failed","validation_errors":[],"gain_pct":null,"plateau_streak":0,"raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-133518/tp4_dp4/logs/vllm_c48_i1024_o1024_a1.log"}
|
||||
@ -0,0 +1,4 @@
|
||||
{"timestamp":"2026-07-29T13:45:43+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":1666.3225799137726,"last_total_tps":815.0660192018358}
|
||||
{"timestamp":"2026-07-29T13:49:05+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":2205.2911145263356,"last_total_tps":2205.2911145263356}
|
||||
{"timestamp":"2026-07-29T13:53:54+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":1780.0375569607418,"last_total_tps":1780.0375569607418}
|
||||
{"timestamp":"2026-07-29T14:04:55+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"status":"FAILED","stop_reason":"BENCHMARK_FAILED","tested_points":3,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":48,"best_tps_concurrency":32,"best_total_tps":1420.5249115465142,"last_total_tps":1420.5249115465142}
|
||||
@ -0,0 +1,4 @@
|
||||
{"timestamp": "2026-07-29T13:45:43+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 1666.3225799137726, "last_total_tps": 815.0660192018358}
|
||||
{"timestamp": "2026-07-29T13:49:05+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 2205.2911145263356, "last_total_tps": 2205.2911145263356}
|
||||
{"timestamp": "2026-07-29T13:53:54+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 1780.0375569607418, "last_total_tps": 1780.0375569607418}
|
||||
{"timestamp": "2026-07-29T14:04:55+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 1024, "status": "FAILED", "stop_reason": "BENCHMARK_FAILED", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 48, "best_tps_concurrency": 32, "best_total_tps": 1420.5249115465142, "last_total_tps": 1420.5249115465142}
|
||||
@ -0,0 +1,10 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
| vllm | 4 | 4 | 1024 | 128 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 1666.3225799137726 | 8 |
|
||||
| vllm | 4 | 4 | 1024 | 256 | TTFT_SLO_EXCEEDED | None | 32 | 2205.2911145263356 | 32 |
|
||||
| vllm | 4 | 4 | 1024 | 512 | TTFT_SLO_EXCEEDED | None | 32 | 1780.0375569607418 | 32 |
|
||||
| vllm | 4 | 4 | 1024 | 1024 | BENCHMARK_FAILED | None | 32 | 1420.5249115465142 | 32 |
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260729-133518",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -0,0 +1,27 @@
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":27.429868040024303,"request_tps":2.9165287956641994,"input_tps":2986.52548676014,"output_tps":373.3156858450175,"total_tps":3359.8411726051577,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1093.8590499572456,"ttft_p95_ms":1759.2374775151256,"ttft_p99_ms":1762.2657715308014,"tpot_p50_ms":33.71841196836097,"tpot_p95_ms":83.95337250660212,"tpot_p99_ms":47.9477285059429,"e2e_p50_ms":5314.494080026634,"e2e_p95_ms":6207.9514399927575,"e2e_p99_ms":7312.6661522616605,"itl_p50_ms":46.030119992792606,"itl_p95_ms":343.74248797539605,"itl_p99_ms":382.76815079618245,"validation_errors":[],"timestamp":"2026-07-29T15:19:44+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c16_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":34.63911426998675,"request_tps":4.619055751625638,"input_tps":4729.913089664653,"output_tps":591.2391362080816,"total_tps":5321.152225872735,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1337.287495029159,"ttft_p95_ms":1447.7295449643862,"ttft_p99_ms":2444.2991836590145,"tpot_p50_ms":47.38358625964313,"tpot_p95_ms":109.98766556023376,"tpot_p99_ms":68.23056443038901,"e2e_p50_ms":7351.7748650047,"e2e_p95_ms":8142.698241077596,"e2e_p99_ms":9750.27762743644,"itl_p50_ms":50.03189004492015,"itl_p95_ms":351.2585399905219,"itl_p99_ms":434.8205319675617,"validation_errors":[],"timestamp":"2026-07-29T15:20:40+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":58.375112,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c32_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":47.69245305994991,"request_tps":5.032242726083255,"input_tps":5153.016551509253,"output_tps":644.1270689386566,"total_tps":5797.14362044791,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1839.6676849806681,"ttft_p95_ms":3656.1040009721182,"ttft_p99_ms":4155.3374408965465,"tpot_p50_ms":58.08645271646651,"tpot_p95_ms":151.407381617355,"tpot_p99_ms":100.1738910166469,"e2e_p50_ms":9005.63708000118,"e2e_p95_ms":10426.58044747077,"e2e_p99_ms":15786.531327799887,"itl_p50_ms":54.28050004411489,"itl_p95_ms":433.08854599017644,"itl_p99_ms":892.5818619807259,"validation_errors":[],"timestamp":"2026-07-29T15:21:49+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":8.945269,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c48_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":47.56827617995441,"request_tps":6.727172512819587,"input_tps":6888.624653127257,"output_tps":861.0780816409072,"total_tps":7749.702734768164,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":2038.135209993925,"ttft_p95_ms":4377.097526926082,"ttft_p99_ms":4397.005372742424,"tpot_p50_ms":53.529223739756205,"tpot_p95_ms":179.73692057939843,"tpot_p99_ms":102.78985201579145,"e2e_p50_ms":8498.075664916541,"e2e_p95_ms":13536.689858639147,"e2e_p99_ms":16145.421241113223,"itl_p50_ms":56.602629949338734,"itl_p95_ms":431.50126992259175,"itl_p99_ms":1089.457456045784,"validation_errors":[],"timestamp":"2026-07-29T15:22:57+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":33.6814,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c64_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":40.86554054997396,"request_tps":1.9576395888406029,"input_tps":2004.6229389727773,"output_tps":501.15573474319433,"total_tps":2505.7786737159718,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":769.0737400553189,"ttft_p95_ms":1177.7077199774792,"ttft_p99_ms":1180.443820510991,"tpot_p50_ms":27.270053274582562,"tpot_p95_ms":67.59154465057141,"tpot_p99_ms":37.27452847649634,"e2e_p50_ms":7824.769744998775,"e2e_p95_ms":9573.475948948178,"e2e_p99_ms":10682.496609513179,"itl_p50_ms":46.50609998498112,"itl_p95_ms":49.80733394622803,"itl_p99_ms":373.2970328396186,"validation_errors":[],"timestamp":"2026-07-29T15:23:57+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c16_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":52.12185687001329,"request_tps":3.0697294687528887,"input_tps":3143.402976002958,"output_tps":785.8507440007395,"total_tps":3929.2537200036977,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1196.1936799925752,"ttft_p95_ms":3376.661240000976,"ttft_p99_ms":3380.586671734927,"tpot_p50_ms":35.8726734706365,"tpot_p95_ms":78.6635779527387,"tpot_p99_ms":46.01526287276588,"e2e_p50_ms":10441.197750042193,"e2e_p95_ms":12427.191755012607,"e2e_p99_ms":13624.868184163934,"itl_p50_ms":49.74326002411544,"itl_p95_ms":334.3674059840851,"itl_p99_ms":407.44652380468324,"validation_errors":[],"timestamp":"2026-07-29T15:25:07+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":56.807693,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c32_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":61.11259619996417,"request_tps":3.927177291154597,"input_tps":4021.429546142307,"output_tps":1005.3573865355768,"total_tps":5026.786932677884,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1682.9362750286236,"ttft_p95_ms":3655.209425964859,"ttft_p99_ms":4176.237343116663,"tpot_p50_ms":42.79642080381403,"tpot_p95_ms":97.01216287526222,"tpot_p99_ms":55.76026406467378,"e2e_p50_ms":12591.057910001837,"e2e_p95_ms":13484.819837001847,"e2e_p99_ms":15755.308343548093,"itl_p50_ms":54.07109501538798,"itl_p95_ms":383.3582650113385,"itl_p99_ms":485.93931600917136,"validation_errors":[],"timestamp":"2026-07-29T15:26:28+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":27.932358,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c48_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":69.76031606993638,"request_tps":4.58713517982333,"input_tps":4697.22642413909,"output_tps":1174.3066060347726,"total_tps":5871.533030173863,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1739.8199950694107,"ttft_p95_ms":4390.747790900059,"ttft_p99_ms":4398.318768192548,"tpot_p50_ms":49.31123337261013,"tpot_p95_ms":110.4132296527041,"tpot_p99_ms":64.62568813183474,"e2e_p50_ms":14140.667494968511,"e2e_p95_ms":15900.652925507167,"e2e_p99_ms":18541.150288403736,"itl_p50_ms":56.618770002387464,"itl_p95_ms":429.3662079842761,"itl_p99_ms":682.1354693896137,"validation_errors":[],"timestamp":"2026-07-29T15:27:58+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":16.804892,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c64_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":76.93152055994142,"request_tps":1.0398858545590266,"input_tps":1064.8431150684432,"output_tps":532.4215575342216,"total_tps":1597.2646726026649,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":797.3025100072846,"ttft_p95_ms":1710.0351795379538,"ttft_p99_ms":1711.955783565063,"tpot_p50_ms":27.44004125248936,"tpot_p95_ms":60.026195124685394,"tpot_p99_ms":36.330749135630256,"e2e_p50_ms":14963.520629971754,"e2e_p95_ms":16610.776779579464,"e2e_p99_ms":19154.3003785645,"itl_p50_ms":46.401569969020784,"itl_p95_ms":49.56071104388685,"itl_p99_ms":370.3472658456303,"validation_errors":[],"timestamp":"2026-07-29T15:29:33+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c16_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":86.17978268989827,"request_tps":1.8565839342590347,"input_tps":1901.1419486812515,"output_tps":950.5709743406258,"total_tps":2851.712923021877,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1069.8379300301895,"ttft_p95_ms":3097.6050424680575,"ttft_p99_ms":3101.0737438197248,"tpot_p50_ms":31.19583226035945,"tpot_p95_ms":67.55592335706577,"tpot_p99_ms":38.296944260106955,"e2e_p50_ms":17224.928409967106,"e2e_p95_ms":18853.40187245747,"e2e_p99_ms":20977.192446121473,"itl_p50_ms":49.64963998645544,"itl_p95_ms":54.12363398354499,"itl_p99_ms":384.22842201776814,"validation_errors":[],"timestamp":"2026-07-29T15:31:18+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":78.537281,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c32_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":98.26273655006662,"request_tps":2.4424314692041547,"input_tps":2501.0498244650544,"output_tps":1250.5249122325272,"total_tps":3751.5747366975816,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1200.0460950075649,"ttft_p95_ms":1507.3092064878438,"ttft_p99_ms":1921.8239739851558,"tpot_p50_ms":37.58711437387173,"tpot_p95_ms":79.74151093235606,"tpot_p99_ms":43.475471112041944,"e2e_p50_ms":20439.060499949846,"e2e_p95_ms":21665.912618360013,"e2e_p99_ms":23484.33577637421,"itl_p50_ms":53.869650000706315,"itl_p95_ms":63.63196996971965,"itl_p99_ms":452.2413359954953,"validation_errors":[],"timestamp":"2026-07-29T15:33:15+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":31.555133,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c48_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":107.90679003996775,"request_tps":2.965522372424152,"input_tps":3036.6949093623316,"output_tps":1518.3474546811658,"total_tps":4555.042364043497,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1563.633510028012,"ttft_p95_ms":3616.530516545766,"ttft_p99_ms":4644.858318382176,"tpot_p50_ms":38.541489031412794,"tpot_p95_ms":86.43771651228319,"tpot_p99_ms":47.55039217014325,"e2e_p50_ms":21336.186399974395,"e2e_p95_ms":24247.328175453007,"e2e_p99_ms":25892.04101533629,"itl_p50_ms":56.55575997661799,"itl_p95_ms":66.31242599105462,"itl_p99_ms":478.35591284791013,"validation_errors":[],"timestamp":"2026-07-29T15:35:23+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":21.41681,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c64_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":110.3160341300536,"request_tps":3.6259461569152553,"input_tps":3712.9688646812215,"output_tps":1856.4844323406107,"total_tps":5569.453297021832,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1403.9371499675326,"ttft_p95_ms":2510.4300710139796,"ttft_p99_ms":2518.426071854774,"tpot_p50_ms":41.21547844420404,"tpot_p95_ms":89.77746244500653,"tpot_p99_ms":51.967190794166946,"e2e_p50_ms":22674.031475035008,"e2e_p95_ms":24767.616361554243,"e2e_p99_ms":27746.995200325036,"itl_p50_ms":57.40386003162712,"itl_p95_ms":411.9906185776926,"itl_p99_ms":465.08525467594154,"validation_errors":[],"timestamp":"2026-07-29T15:37:35+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":22.270066,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c80_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c80_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":121.89236294000875,"request_tps":3.9379005248773424,"input_tps":4032.4101374743987,"output_tps":2016.2050687371993,"total_tps":6048.615206211598,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":2106.833159981761,"ttft_p95_ms":4551.948839047691,"ttft_p99_ms":5381.965280963341,"tpot_p50_ms":43.55823076325569,"tpot_p95_ms":97.94984290426783,"tpot_p99_ms":57.37930273150554,"e2e_p50_ms":24362.383104977198,"e2e_p95_ms":28297.589786956076,"e2e_p99_ms":31468.233178079827,"itl_p50_ms":59.21252991538495,"itl_p95_ms":419.0418620128183,"itl_p99_ms":515.566125633195,"validation_errors":[],"timestamp":"2026-07-29T15:39:59+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":8.603392,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c96_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c96_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":132.6631694799289,"request_tps":0.6030309716978645,"input_tps":617.5037150186132,"output_tps":617.5037150186132,"total_tps":1235.0074300372264,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":762.5481949653476,"ttft_p95_ms":872.9899199970532,"ttft_p99_ms":873.7511334801093,"tpot_p50_ms":24.66274077228187,"tpot_p95_ms":52.63744276924314,"tpot_p99_ms":28.256946610176904,"e2e_p50_ms":25766.039129986893,"e2e_p95_ms":28760.770210431656,"e2e_p99_ms":29670.084396642633,"itl_p50_ms":46.430449991021305,"itl_p95_ms":48.03663847851567,"itl_p99_ms":54.986870161956176,"validation_errors":[],"timestamp":"2026-07-29T15:42:30+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c16_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":154.44891405000817,"request_tps":1.035941242993748,"input_tps":1060.803832825598,"output_tps":1060.803832825598,"total_tps":2121.607665651196,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":923.415949975606,"ttft_p95_ms":3096.213627979159,"ttft_p99_ms":3101.1152469355147,"tpot_p50_ms":28.553689662722157,"tpot_p95_ms":61.84267141036579,"tpot_p99_ms":33.29320668926456,"e2e_p50_ms":30044.512339984067,"e2e_p95_ms":32925.68296153331,"e2e_p99_ms":35176.18757603457,"itl_p50_ms":49.744945019483566,"itl_p95_ms":52.63150456012226,"itl_p99_ms":366.2834746052977,"validation_errors":[],"timestamp":"2026-07-29T15:45:23+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":71.789061,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c32_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":177.49117886996828,"request_tps":1.3521798746732439,"input_tps":1384.6321916654017,"output_tps":1384.6321916654017,"total_tps":2769.2643833308034,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1281.6960000200197,"ttft_p95_ms":3652.8770364704537,"ttft_p99_ms":4106.781017213361,"tpot_p50_ms":34.470334980408786,"tpot_p95_ms":73.01177464079139,"tpot_p99_ms":40.637195729912285,"e2e_p50_ms":36537.76817995822,"e2e_p95_ms":39914.47326196939,"e2e_p99_ms":42900.80525587312,"itl_p50_ms":53.89436008408666,"itl_p95_ms":58.40135198086494,"itl_p99_ms":436.23598001431674,"validation_errors":[],"timestamp":"2026-07-29T15:48:41+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":30.526696,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c48_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":189.58463912003208,"request_tps":1.6879004622172886,"input_tps":1728.4100733105035,"output_tps":1728.4100733105035,"total_tps":3456.820146621007,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1566.7527500190772,"ttft_p95_ms":3578.948683937791,"ttft_p99_ms":4614.5977783657145,"tpot_p50_ms":35.29899754642964,"tpot_p95_ms":78.0709832116916,"tpot_p99_ms":42.36126952037513,"e2e_p50_ms":37652.671699994244,"e2e_p95_ms":41609.66412900016,"e2e_p99_ms":44898.80485399859,"itl_p50_ms":56.692379992455244,"itl_p95_ms":62.34837896190581,"itl_p99_ms":446.15354116773227,"validation_errors":[],"timestamp":"2026-07-29T15:52:11+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":24.828101,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c64_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":195.7285060400609,"request_tps":2.043647131900806,"input_tps":2092.694663066425,"output_tps":2092.694663066425,"total_tps":4185.38932613285,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1691.601489961613,"ttft_p95_ms":4760.785039013717,"ttft_p99_ms":4778.639976511477,"tpot_p50_ms":36.631798470164775,"tpot_p95_ms":79.23510517347911,"tpot_p99_ms":45.106590701541826,"e2e_p50_ms":39439.64098999277,"e2e_p95_ms":42716.15189855802,"e2e_p99_ms":47830.29220274184,"itl_p50_ms":57.442729943431914,"itl_p95_ms":64.276416960638,"itl_p99_ms":456.15205098642036,"validation_errors":[],"timestamp":"2026-07-29T15:55:48+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":21.076282,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c80_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c80_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":256.01750103000086,"request_tps":0.3124786378983731,"input_tps":319.97812520793406,"output_tps":639.9562504158681,"total_tps":959.9343756238022,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":745.6624199985527,"ttft_p95_ms":1726.9185199518686,"ttft_p99_ms":1729.429161795415,"tpot_p50_ms":24.304036555937337,"tpot_p95_ms":50.21535895119722,"tpot_p99_ms":27.05765399672059,"e2e_p50_ms":50528.25715497602,"e2e_p95_ms":53604.79665597086,"e2e_p99_ms":56169.4428748032,"itl_p50_ms":46.40620993450284,"itl_p95_ms":47.94248996768147,"itl_p99_ms":51.48361006285995,"validation_errors":[],"timestamp":"2026-07-29T16:00:22+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c16_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":284.02030898001976,"request_tps":0.5633399969692156,"input_tps":576.8601568964767,"output_tps":1153.7203137929534,"total_tps":1730.58047068943,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":956.3094000332057,"ttft_p95_ms":1641.1008650087754,"ttft_p99_ms":3387.646524802549,"tpot_p50_ms":26.790202689301594,"tpot_p95_ms":56.30136434768366,"tpot_p99_ms":30.73970322055356,"e2e_p50_ms":55538.329180039,"e2e_p95_ms":59555.68195696105,"e2e_p99_ms":63985.25801560491,"itl_p50_ms":49.59034000057727,"itl_p95_ms":51.891003496712074,"itl_p99_ms":335.9504084591754,"validation_errors":[],"timestamp":"2026-07-29T16:05:26+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":80.281123,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c32_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":321.1590207699919,"request_tps":0.7472933483997746,"input_tps":765.2283887613692,"output_tps":1530.4567775227383,"total_tps":2295.6851662841073,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1255.1998100243509,"ttft_p95_ms":3672.5916310038883,"ttft_p99_ms":4125.538569397759,"tpot_p50_ms":30.86073848069816,"tpot_p95_ms":64.86165093757563,"tpot_p99_ms":35.33933813787876,"e2e_p50_ms":64510.2212249767,"e2e_p95_ms":68744.7411134548,"e2e_p99_ms":74047.64789270467,"itl_p50_ms":53.87361004250124,"itl_p95_ms":56.458701455267146,"itl_p99_ms":412.30462671723217,"validation_errors":[],"timestamp":"2026-07-29T16:11:07+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":32.654055,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c48_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":341.30243706004694,"request_tps":0.9375848668308832,"input_tps":960.0869036348244,"output_tps":1920.1738072696487,"total_tps":2880.260710904473,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1592.148675059434,"ttft_p95_ms":3643.7436114996767,"ttft_p99_ms":4432.330028413562,"tpot_p50_ms":32.619219379564164,"tpot_p95_ms":68.740189177771,"tpot_p99_ms":37.695687518965336,"e2e_p50_ms":68344.5101249963,"e2e_p95_ms":72469.8367085366,"e2e_p99_ms":78325.09971081396,"itl_p50_ms":56.271799956448376,"itl_p95_ms":59.70556400716304,"itl_p99_ms":427.5339267589152,"validation_errors":[],"timestamp":"2026-07-29T16:17:10+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":25.464099,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c64_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":353.8395146699622,"request_tps":1.1304559932292841,"input_tps":1157.586937066787,"output_tps":2315.173874133574,"total_tps":3472.760811200361,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1761.9533149991184,"ttft_p95_ms":4731.049663998419,"ttft_p99_ms":4741.435335098067,"tpot_p50_ms":33.54784910601569,"tpot_p95_ms":70.36911139918652,"tpot_p99_ms":38.09025691656619,"e2e_p50_ms":70156.03414998623,"e2e_p95_ms":74791.51736353524,"e2e_p99_ms":80605.7601390325,"itl_p50_ms":57.39102000370622,"itl_p95_ms":61.02473601931706,"itl_p99_ms":436.2752813776023,"validation_errors":[],"timestamp":"2026-07-29T16:23:27+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":20.571058,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c80_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c80_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":497.58994103001896,"request_tps":0.16077495424123475,"input_tps":164.6335531430244,"output_tps":658.5342125720975,"total_tps":823.167765715122,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1050.8773350156844,"ttft_p95_ms":1694.0854515065437,"ttft_p99_ms":1695.969674247317,"tpot_p50_ms":23.69077457265726,"tpot_p95_ms":48.07563670752405,"tpot_p99_ms":25.435760589014286,"e2e_p50_ms":97780.65606002929,"e2e_p95_ms":100612.72085299715,"e2e_p99_ms":104642.1418758889,"itl_p50_ms":46.31447501014918,"itl_p95_ms":47.51379496883601,"itl_p99_ms":49.31975653744303,"validation_errors":[],"timestamp":"2026-07-29T16:32:03+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c16_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":540.6980647000019,"request_tps":0.2959137649008853,"input_tps":303.0156952585065,"output_tps":1212.062781034026,"total_tps":1515.0784762925327,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1025.6684549967758,"ttft_p95_ms":1125.8449920394924,"ttft_p99_ms":1205.0617976591457,"tpot_p50_ms":25.85648229424995,"tpot_p95_ms":52.6812972631419,"tpot_p99_ms":28.779375956243957,"e2e_p50_ms":106835.4650750407,"e2e_p95_ms":110308.46746150636,"e2e_p99_ms":118988.76709997421,"itl_p50_ms":49.66543009504676,"itl_p95_ms":51.547764032147825,"itl_p99_ms":55.002351833973066,"validation_errors":[],"timestamp":"2026-07-29T16:41:24+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":84.054641,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c32_i1024_o4096_a1.log"}
|
||||
{"timestamp":"2026-07-29T16:49:28+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"status":"FAILED","completed":0,"failed":240,"error_type":"benchmark failed and server restart failed","validation_errors":[],"gain_pct":null,"plateau_streak":0,"raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260729-151004/tp4_dp4/logs/vllm_c48_i1024_o4096_a1.log"}
|
||||
@ -0,0 +1,6 @@
|
||||
{"timestamp":"2026-07-29T15:22:57+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":4,"search_cap":256,"max_successful_concurrency":64,"saturation_concurrency":null,"stop_probe_concurrency":64,"best_tps_concurrency":64,"best_total_tps":7749.702734768164,"last_total_tps":7749.702734768164}
|
||||
{"timestamp":"2026-07-29T15:27:58+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":4,"search_cap":256,"max_successful_concurrency":64,"saturation_concurrency":null,"stop_probe_concurrency":64,"best_tps_concurrency":64,"best_total_tps":5871.533030173863,"last_total_tps":5871.533030173863}
|
||||
{"timestamp":"2026-07-29T15:39:59+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":6,"search_cap":256,"max_successful_concurrency":96,"saturation_concurrency":null,"stop_probe_concurrency":96,"best_tps_concurrency":96,"best_total_tps":6048.615206211598,"last_total_tps":6048.615206211598}
|
||||
{"timestamp":"2026-07-29T15:55:48+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":5,"search_cap":256,"max_successful_concurrency":80,"saturation_concurrency":null,"stop_probe_concurrency":80,"best_tps_concurrency":80,"best_total_tps":4185.38932613285,"last_total_tps":4185.38932613285}
|
||||
{"timestamp":"2026-07-29T16:23:27+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":5,"search_cap":256,"max_successful_concurrency":80,"saturation_concurrency":null,"stop_probe_concurrency":80,"best_tps_concurrency":80,"best_total_tps":3472.760811200361,"last_total_tps":3472.760811200361}
|
||||
{"timestamp":"2026-07-29T16:49:28+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"status":"FAILED","stop_reason":"BENCHMARK_FAILED","tested_points":3,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":48,"best_tps_concurrency":32,"best_total_tps":1515.0784762925327,"last_total_tps":1515.0784762925327}
|
||||
@ -0,0 +1,6 @@
|
||||
{"timestamp": "2026-07-29T15:22:57+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 4, "search_cap": 256, "max_successful_concurrency": 64, "saturation_concurrency": null, "stop_probe_concurrency": 64, "best_tps_concurrency": 64, "best_total_tps": 7749.702734768164, "last_total_tps": 7749.702734768164}
|
||||
{"timestamp": "2026-07-29T15:27:58+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 4, "search_cap": 256, "max_successful_concurrency": 64, "saturation_concurrency": null, "stop_probe_concurrency": 64, "best_tps_concurrency": 64, "best_total_tps": 5871.533030173863, "last_total_tps": 5871.533030173863}
|
||||
{"timestamp": "2026-07-29T15:39:59+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 6, "search_cap": 256, "max_successful_concurrency": 96, "saturation_concurrency": null, "stop_probe_concurrency": 96, "best_tps_concurrency": 96, "best_total_tps": 6048.615206211598, "last_total_tps": 6048.615206211598}
|
||||
{"timestamp": "2026-07-29T15:55:48+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 5, "search_cap": 256, "max_successful_concurrency": 80, "saturation_concurrency": null, "stop_probe_concurrency": 80, "best_tps_concurrency": 80, "best_total_tps": 4185.38932613285, "last_total_tps": 4185.38932613285}
|
||||
{"timestamp": "2026-07-29T16:23:27+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 5, "search_cap": 256, "max_successful_concurrency": 80, "saturation_concurrency": null, "stop_probe_concurrency": 80, "best_tps_concurrency": 80, "best_total_tps": 3472.760811200361, "last_total_tps": 3472.760811200361}
|
||||
{"timestamp": "2026-07-29T16:49:28+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 4096, "status": "FAILED", "stop_reason": "BENCHMARK_FAILED", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 48, "best_tps_concurrency": 32, "best_total_tps": 1515.0784762925327, "last_total_tps": 1515.0784762925327}
|
||||
@ -0,0 +1,12 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
| vllm | 4 | 4 | 1024 | 128 | TTFT_SLO_EXCEEDED | None | 64 | 7749.702734768164 | 64 |
|
||||
| vllm | 4 | 4 | 1024 | 256 | TTFT_SLO_EXCEEDED | None | 64 | 5871.533030173863 | 64 |
|
||||
| vllm | 4 | 4 | 1024 | 512 | TTFT_SLO_EXCEEDED | None | 96 | 6048.615206211598 | 96 |
|
||||
| vllm | 4 | 4 | 1024 | 1024 | TTFT_SLO_EXCEEDED | None | 80 | 4185.38932613285 | 80 |
|
||||
| vllm | 4 | 4 | 1024 | 2048 | TTFT_SLO_EXCEEDED | None | 80 | 3472.760811200361 | 80 |
|
||||
| vllm | 4 | 4 | 1024 | 4096 | BENCHMARK_FAILED | None | 32 | 1515.0784762925327 | 32 |
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260729-151004",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -0,0 +1,6 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260729-165133",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -0,0 +1,195 @@
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":24.962610360002145,"request_tps":3.2047930423248063,"input_tps":3281.7080753406017,"output_tps":410.2135094175752,"total_tps":3691.921584758177,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":709.8514149547555,"ttft_p95_ms":752.985320443986,"ttft_p99_ms":754.2274305340834,"tpot_p50_ms":33.68007940905697,"tpot_p95_ms":36.66902147678064,"tpot_p99_ms":36.67307469200922,"e2e_p50_ms":4988.339944975451,"e2e_p95_ms":5039.645334566012,"e2e_p99_ms":5041.137067310046,"itl_p50_ms":33.59546995488927,"itl_p95_ms":34.415798500413075,"itl_p99_ms":36.75535452668555,"validation_errors":[],"timestamp":"2026-07-30T10:46:45+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":41.61825796996709,"request_tps":3.8444665347468536,"input_tps":3936.733731580778,"output_tps":492.09171644759726,"total_tps":4428.825448028376,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":2846.1439500097185,"ttft_p95_ms":5484.800329525023,"ttft_p99_ms":5506.579634363297,"tpot_p50_ms":39.080454763501706,"tpot_p95_ms":66.40261593293106,"tpot_p99_ms":78.25863986737168,"e2e_p50_ms":7781.921124958899,"e2e_p95_ms":10297.615796938771,"e2e_p99_ms":10317.958215529798,"itl_p50_ms":37.68735006451607,"itl_p95_ms":39.37462598551065,"itl_p99_ms":47.02749760821462,"validation_errors":[],"timestamp":"2026-07-30T10:47:45+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":19.9599,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c32_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":46.465715310070664,"request_tps":1.7216995254705858,"input_tps":1763.0203140818799,"output_tps":440.75507852046997,"total_tps":2203.7753926023497,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":678.755929984618,"ttft_p95_ms":713.1266865588259,"ttft_p99_ms":714.0169177134521,"tpot_p50_ms":33.80685539222231,"tpot_p95_ms":35.30786731186798,"tpot_p99_ms":35.31035023151587,"e2e_p50_ms":9283.80921005737,"e2e_p95_ms":9359.541199967496,"e2e_p99_ms":9360.741281379014,"itl_p50_ms":33.70353003265336,"itl_p95_ms":34.473714540945366,"itl_p99_ms":35.84104406414554,"validation_errors":[],"timestamp":"2026-07-30T10:48:50+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":52.64006859995425,"request_tps":3.0395097167509975,"input_tps":3112.4579499530214,"output_tps":778.1144874882553,"total_tps":3890.5724374412766,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1082.6259650057182,"ttft_p95_ms":1287.5385194492992,"ttft_p99_ms":1288.8101247232407,"tpot_p50_ms":36.778934921284076,"tpot_p95_ms":39.60059185297794,"tpot_p99_ms":41.531685830550444,"e2e_p50_ms":10446.624720003456,"e2e_p95_ms":10914.39445542055,"e2e_p99_ms":10921.224330934929,"itl_p50_ms":36.83501994237304,"itl_p95_ms":38.39530595578253,"itl_p99_ms":39.77536081336439,"validation_errors":[],"timestamp":"2026-07-30T10:50:01+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":76.541241,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c32_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":67.99680423003156,"request_tps":3.529577642915184,"input_tps":3614.2875063451484,"output_tps":903.5718765862871,"total_tps":4517.859382931436,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1434.9654350080527,"ttft_p95_ms":3905.7994770701043,"ttft_p99_ms":4140.922463373278,"tpot_p50_ms":42.377190431351686,"tpot_p95_ms":52.385104164691164,"tpot_p99_ms":55.77287803427771,"e2e_p50_ms":14235.570800025016,"e2e_p95_ms":14769.038806995375,"e2e_p99_ms":14797.052594327834,"itl_p50_ms":39.90000009071082,"itl_p95_ms":41.9920379994437,"itl_p99_ms":47.0534152118488,"validation_errors":[],"timestamp":"2026-07-30T10:51:31+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":16.123256,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c48_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":63.94534658000339,"request_tps":5.004273447789374,"input_tps":5124.376010536319,"output_tps":1281.0940026340797,"total_tps":6405.4700131703985,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1432.0790949859656,"ttft_p95_ms":2180.5370969872456,"ttft_p99_ms":2192.7704340883065,"tpot_p50_ms":44.47418611756909,"tpot_p95_ms":47.430748060989366,"tpot_p99_ms":49.03998703535135,"e2e_p50_ms":12777.47420500964,"e2e_p95_ms":13041.395013529109,"e2e_p99_ms":13045.853094748454,"itl_p50_ms":41.04871005984023,"itl_p95_ms":43.449138558935374,"itl_p99_ms":48.212759535526864,"validation_errors":[],"timestamp":"2026-07-30T10:52:55+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":41.781084,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c64_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":71.45656451000832,"request_tps":5.597806202171605,"input_tps":5732.153551023724,"output_tps":1433.038387755931,"total_tps":7165.191938779654,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1783.4126849775203,"ttft_p95_ms":2420.5658505437896,"ttft_p99_ms":2427.367927165469,"tpot_p50_ms":47.20392815686543,"tpot_p95_ms":59.25872118611747,"tpot_p99_ms":63.04450487820248,"e2e_p50_ms":13832.971810013987,"e2e_p95_ms":16615.52544903825,"e2e_p99_ms":16624.56594434101,"itl_p50_ms":42.27589996298775,"itl_p95_ms":45.80430198693647,"itl_p99_ms":60.046807993203394,"validation_errors":[],"timestamp":"2026-07-30T10:54:27+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":11.860518,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c80_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c80_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":78.78559228009544,"request_tps":6.092484502667986,"input_tps":6238.704130732018,"output_tps":1559.6760326830045,"total_tps":7798.380163415022,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":2440.6090949778445,"ttft_p95_ms":4778.216223593336,"ttft_p99_ms":5201.717905879477,"tpot_p50_ms":50.51180796067724,"tpot_p95_ms":62.45406786468355,"tpot_p99_ms":62.49101296290937,"e2e_p50_ms":14997.570549952798,"e2e_p95_ms":17389.587355556432,"e2e_p99_ms":17409.369085555663,"itl_p50_ms":44.98150496510789,"itl_p95_ms":49.27675147773697,"itl_p99_ms":59.8470669891683,"validation_errors":[],"timestamp":"2026-07-30T10:56:09+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":8.837003,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c96_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c96_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":90.1738068300765,"request_tps":0.8871755869279422,"input_tps":908.4678010142128,"output_tps":454.2339005071064,"total_tps":1362.7017015213191,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":671.7311050160788,"ttft_p95_ms":703.9750784751959,"ttft_p99_ms":704.8587221384514,"tpot_p50_ms":33.999248571354904,"tpot_p95_ms":34.74717959392554,"tpot_p99_ms":34.74737834673122,"e2e_p50_ms":18040.702164988033,"e2e_p95_ms":18109.357712505152,"e2e_p99_ms":18110.867167402757,"itl_p50_ms":33.94128999207169,"itl_p95_ms":34.82261796598323,"itl_p99_ms":35.78069728682749,"validation_errors":[],"timestamp":"2026-07-30T10:57:56+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":112.79712073993869,"request_tps":1.4184759234137785,"input_tps":1452.5193455757092,"output_tps":726.2596727878546,"total_tps":2178.779018363564,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":2846.2033250252716,"ttft_p95_ms":3536.073226522422,"ttft_p99_ms":3581.8792629498057,"tpot_p50_ms":38.08177776902152,"tpot_p95_ms":43.3671966468261,"tpot_p99_ms":44.38292396235245,"e2e_p50_ms":22511.110524937976,"e2e_p95_ms":23029.565962415654,"e2e_p99_ms":23074.61284482153,"itl_p50_ms":37.98756003379822,"itl_p95_ms":39.49427702464163,"itl_p99_ms":40.641573769971735,"validation_errors":[],"timestamp":"2026-07-30T11:00:08+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":59.886717,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c32_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":110.92540055001155,"request_tps":2.1636162575026647,"input_tps":2215.5430476827287,"output_tps":1107.7715238413643,"total_tps":3323.314571524093,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1313.5625050053932,"ttft_p95_ms":1459.7667015856132,"ttft_p99_ms":1703.8911627500784,"tpot_p50_ms":40.73612287664721,"tpot_p95_ms":42.454030472601154,"tpot_p99_ms":43.088983990488494,"e2e_p50_ms":22099.866654956713,"e2e_p95_ms":22443.076764995934,"e2e_p99_ms":22581.501946741482,"itl_p50_ms":40.0854799663648,"itl_p95_ms":42.25488796364515,"itl_p99_ms":44.29845676757395,"validation_errors":[],"timestamp":"2026-07-30T11:02:19+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":52.531053,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c48_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":122.49123519007117,"request_tps":2.612431816067917,"input_tps":2675.130179653547,"output_tps":1337.5650898267736,"total_tps":4012.695269480321,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1511.6283299867064,"ttft_p95_ms":4546.506124048028,"ttft_p99_ms":4560.975359951844,"tpot_p50_ms":43.39158780831359,"tpot_p95_ms":45.691087408893,"tpot_p99_ms":47.48176463897642,"e2e_p50_ms":24556.27372005256,"e2e_p95_ms":26043.656755040865,"e2e_p99_ms":26056.165733097587,"itl_p50_ms":41.30862501915544,"itl_p95_ms":44.346624525496736,"itl_p99_ms":47.545906659215674,"validation_errors":[],"timestamp":"2026-07-30T11:04:44+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":20.743769,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c64_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":177.09918103995733,"request_tps":0.45172427975231744,"input_tps":462.56566246637306,"output_tps":462.56566246637306,"total_tps":925.1313249327461,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":660.6611149618402,"ttft_p95_ms":745.793501008302,"ttft_p99_ms":748.8232597487513,"tpot_p50_ms":34.033160909094484,"tpot_p95_ms":34.38772128057713,"tpot_p99_ms":34.387737513859626,"e2e_p50_ms":35478.359810018446,"e2e_p95_ms":35565.13049490168,"e2e_p99_ms":35566.87529996503,"itl_p50_ms":33.94699498312548,"itl_p95_ms":34.724274498876184,"itl_p99_ms":35.767262146109736,"validation_errors":[],"timestamp":"2026-07-30T11:07:59+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":208.07148279005196,"request_tps":0.7689665006205728,"input_tps":787.4216966354666,"output_tps":787.4216966354666,"total_tps":1574.8433932709331,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":2719.4474400603212,"ttft_p95_ms":3005.3534678998403,"ttft_p99_ms":3019.3223996041343,"tpot_p50_ms":38.15622919843143,"tpot_p95_ms":40.49341477474079,"tpot_p99_ms":40.70055095070017,"e2e_p50_ms":41755.35361003131,"e2e_p95_ms":42049.70321388683,"e2e_p99_ms":42058.79473610781,"itl_p50_ms":37.986360024660826,"itl_p95_ms":39.41656998358667,"itl_p99_ms":40.45925601385534,"validation_errors":[],"timestamp":"2026-07-30T11:11:46+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":70.229172,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c32_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":220.42723491997458,"request_tps":1.0887946767881531,"input_tps":1114.9257490310688,"output_tps":1114.9257490310688,"total_tps":2229.8514980621376,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1453.5930849961005,"ttft_p95_ms":3827.7609515644144,"ttft_p99_ms":4009.9602222605604,"tpot_p50_ms":40.69755665207089,"tpot_p95_ms":42.05324425812576,"tpot_p99_ms":43.95963924901445,"e2e_p50_ms":43440.68913999945,"e2e_p95_ms":45448.399627016624,"e2e_p99_ms":45603.72769060312,"itl_p50_ms":40.11997499037534,"itl_p95_ms":42.132937553105876,"itl_p99_ms":43.82550250738859,"validation_errors":[],"timestamp":"2026-07-30T11:15:46+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":41.591952,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c48_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":223.95933887991123,"request_tps":1.428830794020099,"input_tps":1463.1227330765814,"output_tps":1463.1227330765814,"total_tps":2926.245466153163,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1494.8182849329896,"ttft_p95_ms":2345.479809527751,"ttft_p99_ms":2351.653085714206,"tpot_p50_ms":42.07219630500296,"tpot_p95_ms":43.10143055773922,"tpot_p99_ms":43.7935732987337,"e2e_p50_ms":44557.927899993956,"e2e_p95_ms":45358.47529611783,"e2e_p99_ms":45363.123479170026,"itl_p50_ms":41.37649992480874,"itl_p95_ms":43.970240047201514,"itl_p99_ms":46.317603060742854,"validation_errors":[],"timestamp":"2026-07-30T11:19:51+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":31.230509,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c64_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":239.5007436099695,"request_tps":1.6701409522611166,"input_tps":1710.2243351153834,"output_tps":1710.2243351153834,"total_tps":3420.4486702307668,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":2217.1752299764194,"ttft_p95_ms":4908.534731552936,"ttft_p99_ms":4925.871107542189,"tpot_p50_ms":44.29601882701605,"tpot_p95_ms":46.98131075955656,"tpot_p99_ms":47.583943076455874,"e2e_p50_ms":47312.53871502122,"e2e_p95_ms":49530.01518048695,"e2e_p99_ms":50496.45484582754,"itl_p50_ms":42.81872999854386,"itl_p95_ms":46.13860999234021,"itl_p99_ms":48.97980097448454,"validation_errors":[],"timestamp":"2026-07-30T11:24:13+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":16.888645,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c80_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c80_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":358.50675915996544,"request_tps":0.22314781508569567,"input_tps":228.50336264775237,"output_tps":457.00672529550474,"total_tps":685.5100879432571,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1068.5522500425577,"ttft_p95_ms":1707.9935764486436,"ttft_p99_ms":1745.3203821973875,"tpot_p50_ms":34.51486036149488,"tpot_p95_ms":34.94280105104734,"tpot_p99_ms":34.95001614938431,"e2e_p50_ms":71757.1453199489,"e2e_p95_ms":71882.05817842971,"e2e_p99_ms":71904.52759812702,"itl_p50_ms":34.25346000585705,"itl_p95_ms":34.97642747242935,"itl_p99_ms":35.703756526345394,"validation_errors":[],"timestamp":"2026-07-30T11:30:29+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":384.4194031700026,"request_tps":0.4162120815978762,"input_tps":426.20117155622523,"output_tps":852.4023431124505,"total_tps":1278.6035146686756,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1022.4521649652161,"ttft_p95_ms":1232.2589335090015,"ttft_p99_ms":1234.9709724995773,"tpot_p50_ms":36.876661636536554,"tpot_p95_ms":37.82567980460496,"tpot_p99_ms":38.25827930206556,"e2e_p50_ms":76546.62462003762,"e2e_p95_ms":78654.29459794541,"e2e_p99_ms":78655.16937754815,"itl_p50_ms":36.9447999401018,"itl_p95_ms":38.53771497961134,"itl_p99_ms":39.664434366859496,"validation_errors":[],"timestamp":"2026-07-30T11:37:14+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":86.518556,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c32_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":419.37448687001597,"request_tps":0.5722808790568783,"input_tps":586.0156201542434,"output_tps":1172.0312403084868,"total_tps":1758.0468604627301,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1315.1120099937543,"ttft_p95_ms":1454.1658969072157,"ttft_p99_ms":1720.0785641581751,"tpot_p50_ms":40.3679338690807,"tpot_p95_ms":40.79241916438556,"tpot_p99_ms":40.98337998424732,"e2e_p50_ms":83912.25324000698,"e2e_p95_ms":84309.06991737429,"e2e_p99_ms":84390.3370636981,"itl_p50_ms":40.142419980838895,"itl_p95_ms":41.92410352407023,"itl_p99_ms":43.48092281841673,"validation_errors":[],"timestamp":"2026-07-30T11:44:35+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":37.497421,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c48_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":441.08992932003457,"request_tps":0.7254756427862644,"input_tps":742.8870582131348,"output_tps":1485.7741164262695,"total_tps":2228.661174639404,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":2181.9433349883184,"ttft_p95_ms":4245.517735474277,"ttft_p99_ms":4542.859570770525,"tpot_p50_ms":41.82524316804051,"tpot_p95_ms":42.78160746188964,"tpot_p99_ms":43.26345783478499,"e2e_p50_ms":88634.28587495582,"e2e_p95_ms":89400.5682979594,"e2e_p99_ms":89693.90406433726,"itl_p50_ms":41.36583500076085,"itl_p95_ms":43.86979704722762,"itl_p99_ms":45.918094201479114,"validation_errors":[],"timestamp":"2026-07-30T11:52:18+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":26.769156,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c64_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":704.8561073399615,"request_tps":0.11349834266444817,"input_tps":116.22230288839492,"output_tps":464.8892115535797,"total_tps":581.1115144419746,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":698.6136849736795,"ttft_p95_ms":1652.3392460367177,"ttft_p99_ms":1654.152709345799,"tpot_p50_ms":34.094657125767384,"tpot_p95_ms":34.75502712138098,"tpot_p99_ms":34.755082833886846,"e2e_p50_ms":140801.76149995532,"e2e_p95_ms":142651.93052358227,"e2e_p99_ms":142653.873361703,"itl_p50_ms":34.13494001142681,"itl_p95_ms":34.93209497537464,"itl_p99_ms":36.165602430701256,"validation_errors":[],"timestamp":"2026-07-30T12:04:22+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":777.1398469399428,"request_tps":0.20588315041367936,"input_tps":210.82434602360766,"output_tps":843.2973840944306,"total_tps":1054.1217301180384,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":2452.6138749788515,"ttft_p95_ms":2993.2009705051314,"ttft_p99_ms":3000.269160452299,"tpot_p50_ms":37.18889282296511,"tpot_p95_ms":38.241126748608984,"tpot_p99_ms":38.53800754942926,"e2e_p50_ms":154013.80452490412,"e2e_p95_ms":158156.82374943863,"e2e_p99_ms":158198.25743252411,"itl_p50_ms":37.19291999004781,"itl_p95_ms":38.780850009061396,"itl_p99_ms":39.82597494730726,"validation_errors":[],"timestamp":"2026-07-30T12:17:40+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":81.397495,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c32_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c32_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":835.8009583000094,"request_tps":0.2871497066576136,"input_tps":294.0412996173963,"output_tps":1176.1651984695852,"total_tps":1470.2064980869816,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1922.7989499922842,"ttft_p95_ms":3705.8609565196098,"ttft_p99_ms":4009.1262503457256,"tpot_p50_ms":40.18782855188349,"tpot_p95_ms":40.81626425006839,"tpot_p99_ms":41.24870290498645,"e2e_p50_ms":166305.07121991832,"e2e_p95_ms":169437.80057150288,"e2e_p99_ms":169488.26419219258,"itl_p50_ms":40.07479996653274,"itl_p95_ms":41.79860900621861,"itl_p99_ms":43.272592935245484,"validation_errors":[],"timestamp":"2026-07-30T12:32:01+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":39.472174,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c48_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c48_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":851.7108331400668,"request_tps":0.37571437106210304,"input_tps":384.7315159675935,"output_tps":1538.926063870374,"total_tps":1923.6575798379677,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1380.978614964988,"ttft_p95_ms":2124.868409067858,"ttft_p99_ms":2142.679725413909,"tpot_p50_ms":41.1785393870596,"tpot_p95_ms":41.48739134553099,"tpot_p99_ms":41.68296409689711,"e2e_p50_ms":170327.84509501653,"e2e_p95_ms":171204.4143594976,"e2e_p99_ms":171216.43861321034,"itl_p50_ms":41.04975005611777,"itl_p95_ms":43.238010024651885,"itl_p99_ms":45.25090117473155,"validation_errors":[],"timestamp":"2026-07-30T12:46:38+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":30.84268,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c64_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c64_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":883.6534969600616,"request_tps":0.4526661201207002,"input_tps":463.530107003597,"output_tps":1854.120428014388,"total_tps":2317.650535017985,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1400.2583849942312,"ttft_p95_ms":2580.223578499863,"ttft_p99_ms":2592.3107670852914,"tpot_p50_ms":42.81032853357559,"tpot_p95_ms":43.22701445615666,"tpot_p99_ms":43.3546415215423,"e2e_p50_ms":176705.7987099979,"e2e_p95_ms":178402.64165002157,"e2e_p99_ms":178425.7758970582,"itl_p50_ms":42.484229896217585,"itl_p95_ms":45.35900999326259,"itl_p99_ms":47.78166259173304,"validation_errors":[],"timestamp":"2026-07-30T13:01:49+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":20.481449,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c80_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c80_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":940.5808211599942,"request_tps":0.5103229719356049,"input_tps":522.5707232620595,"output_tps":2090.282893048238,"total_tps":2612.853616310297,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":2296.969619987067,"ttft_p95_ms":4581.845566473203,"ttft_p99_ms":5118.853249545209,"tpot_p50_ms":45.52073066056875,"tpot_p95_ms":46.02148144432146,"tpot_p99_ms":46.0299915233175,"e2e_p50_ms":188407.17022499302,"e2e_p95_ms":190304.82212102503,"e2e_p99_ms":191206.26581523218,"itl_p50_ms":44.92625989951193,"itl_p95_ms":48.64048153394833,"itl_p99_ms":51.66329512605443,"validation_errors":[],"timestamp":"2026-07-30T13:17:59+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":12.73717,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c96_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c96_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":41.055201450013556,"request_tps":1.948595967733915,"input_tps":7981.449083838116,"output_tps":249.42028386994113,"total_tps":8230.869367708057,"mean_input_tokens":4096.0,"mean_output_tokens":128.0,"ttft_p50_ms":2510.692475072574,"ttft_p95_ms":4203.240546019512,"ttft_p99_ms":5207.170735974796,"tpot_p50_ms":41.968488346649636,"tpot_p95_ms":58.79040832684455,"tpot_p99_ms":70.572533787285,"e2e_p50_ms":8372.248700005002,"e2e_p95_ms":9610.177823511183,"e2e_p99_ms":10288.82064819918,"itl_p50_ms":34.202979994006455,"itl_p95_ms":35.5585484940093,"itl_p99_ms":431.83940290822164,"validation_errors":[],"timestamp":"2026-07-30T13:19:01+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i4096_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i4096_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":30.232305060024373,"request_tps":1.3230879987676254,"input_tps":5419.368442952194,"output_tps":169.35526384225605,"total_tps":5588.72370679445,"mean_input_tokens":4096.0,"mean_output_tokens":128.0,"ttft_p50_ms":1544.8776800185442,"ttft_p95_ms":2173.3171354338992,"ttft_p99_ms":2173.727849703282,"tpot_p50_ms":36.14710633895969,"tpot_p95_ms":36.95531224435027,"tpot_p99_ms":36.99814611934052,"e2e_p50_ms":6045.780599932186,"e2e_p95_ms":6192.5701869418845,"e2e_p99_ms":6192.9126092244405,"itl_p50_ms":31.699344981461763,"itl_p95_ms":32.38536607823334,"itl_p99_ms":35.31587217352352,"validation_errors":[],"timestamp":"2026-07-30T13:19:49+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i4096_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i4096_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":63.71854564989917,"request_tps":1.255521437032777,"input_tps":5142.615806086254,"output_tps":321.4134878803909,"total_tps":5464.029293966645,"mean_input_tokens":4096.0,"mean_output_tokens":256.0,"ttft_p50_ms":2781.662334979046,"ttft_p95_ms":4242.499211517861,"ttft_p99_ms":4891.756241990952,"tpot_p50_ms":36.592636195778404,"tpot_p95_ms":47.62132065856884,"tpot_p99_ms":48.17489895114547,"e2e_p50_ms":12849.310714984313,"e2e_p95_ms":13660.720703122208,"e2e_p99_ms":13694.331035739742,"itl_p50_ms":34.367775020655245,"itl_p95_ms":35.13374304166064,"itl_p99_ms":37.32068863115275,"validation_errors":[],"timestamp":"2026-07-30T13:21:12+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i4096_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i4096_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":45.83052829990629,"request_tps":0.8727806875418843,"input_tps":3574.909696171558,"output_tps":223.43185601072238,"total_tps":3798.3415521822803,"mean_input_tokens":4096.0,"mean_output_tokens":256.0,"ttft_p50_ms":1021.5830550296232,"ttft_p95_ms":1071.4627314766406,"ttft_p99_ms":1071.9806603714824,"tpot_p50_ms":31.89495596075978,"tpot_p95_ms":31.990573741180445,"tpot_p99_ms":31.984385390660965,"e2e_p50_ms":9173.082380031701,"e2e_p95_ms":9191.798211133573,"e2e_p99_ms":9192.639958703658,"itl_p50_ms":31.93137003108859,"itl_p95_ms":32.415719982236624,"itl_p99_ms":33.180953157134354,"validation_errors":[],"timestamp":"2026-07-30T13:22:15+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i4096_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i4096_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":107.83041821990628,"request_tps":0.7419056822802104,"input_tps":3038.8456746197417,"output_tps":379.8557093274677,"total_tps":3418.7013839472093,"mean_input_tokens":4096.0,"mean_output_tokens":512.0,"ttft_p50_ms":2747.2549049998634,"ttft_p95_ms":4481.920609984081,"ttft_p99_ms":4928.765463944292,"tpot_p50_ms":36.54808985318173,"tpot_p95_ms":41.3145649883055,"tpot_p99_ms":42.01884003951161,"e2e_p50_ms":21596.268740016967,"e2e_p95_ms":22482.594995072577,"e2e_p99_ms":22487.757388980826,"itl_p50_ms":34.38545006792992,"itl_p95_ms":35.127569938777015,"itl_p99_ms":36.30931670893915,"validation_errors":[],"timestamp":"2026-07-30T13:24:25+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i4096_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i4096_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":92.77496810001321,"request_tps":0.43115078150044983,"input_tps":1765.9936010258425,"output_tps":220.7492001282303,"total_tps":1986.7428011540728,"mean_input_tokens":4096.0,"mean_output_tokens":512.0,"ttft_p50_ms":1608.7955050170422,"ttft_p95_ms":2913.7884064286477,"ttft_p99_ms":2916.4313952240627,"tpot_p50_ms":33.06101817023027,"tpot_p95_ms":36.53993914696641,"tpot_p99_ms":36.5406291606816,"e2e_p50_ms":18441.123349999543,"e2e_p95_ms":19266.472526959842,"e2e_p99_ms":19267.383589179954,"itl_p50_ms":31.964060035534203,"itl_p95_ms":32.50097903655842,"itl_p99_ms":33.20809917058796,"validation_errors":[],"timestamp":"2026-07-30T13:26:16+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":512,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i4096_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i4096_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":184.13008001004346,"request_tps":0.4344754534166083,"input_tps":1779.6114571944277,"output_tps":444.9028642986069,"total_tps":2224.5143214930345,"mean_input_tokens":4096.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1640.22743498208,"ttft_p95_ms":2056.9887309917244,"ttft_p99_ms":2092.594425928546,"tpot_p50_ms":34.38067138812961,"tpot_p95_ms":35.50210980296077,"tpot_p99_ms":35.50237170879534,"e2e_p50_ms":36809.20601997059,"e2e_p95_ms":36934.68486655038,"e2e_p99_ms":36955.49217458931,"itl_p50_ms":33.9932800270617,"itl_p95_ms":34.731985069811344,"itl_p99_ms":35.73291197884828,"validation_errors":[],"timestamp":"2026-07-30T13:29:42+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i4096_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i4096_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":213.15322700992692,"request_tps":0.7506337213114231,"input_tps":3074.595722491589,"output_tps":768.6489306228973,"total_tps":3843.2446531144865,"mean_input_tokens":4096.0,"mean_output_tokens":1024.0,"ttft_p50_ms":3453.3462449908257,"ttft_p95_ms":4671.423198003321,"ttft_p99_ms":5819.419012630824,"tpot_p50_ms":38.18655447705074,"tpot_p95_ms":40.530007066970306,"tpot_p99_ms":42.201600309470685,"e2e_p50_ms":42198.04165000096,"e2e_p95_ms":44413.51390057826,"e2e_p99_ms":44443.24820852722,"itl_p50_ms":36.811040015891194,"itl_p95_ms":38.23056002147496,"itl_p99_ms":39.56557541387156,"validation_errors":[],"timestamp":"2026-07-30T13:33:38+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":1024,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":72.767809,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c32_i4096_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c32_i4096_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":375.2508632900426,"request_tps":0.21319071540194062,"input_tps":873.2291702863488,"output_tps":436.6145851431744,"total_tps":1309.8437554295233,"mean_input_tokens":4096.0,"mean_output_tokens":2048.0,"ttft_p50_ms":3335.4090199572966,"ttft_p95_ms":5194.406287535095,"ttft_p99_ms":5226.852498111547,"tpot_p50_ms":35.07309808256127,"tpot_p95_ms":36.30783987253107,"tpot_p99_ms":36.94945948304047,"e2e_p50_ms":75602.44916001102,"e2e_p95_ms":76644.11962447337,"e2e_p99_ms":77515.14158260426,"itl_p50_ms":34.27060996182263,"itl_p95_ms":35.10243942728266,"itl_p99_ms":35.8640546305105,"validation_errors":[],"timestamp":"2026-07-30T13:40:13+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i4096_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i4096_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":332.0347738399869,"request_tps":0.12046930969729294,"input_tps":493.4422925201119,"output_tps":246.72114626005595,"total_tps":740.1634387801678,"mean_input_tokens":4096.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1020.5126099754125,"ttft_p95_ms":1077.6548505004032,"ttft_p99_ms":1078.5140120936558,"tpot_p50_ms":31.928142366869874,"tpot_p95_ms":32.08152069475563,"tpot_p99_ms":32.08074003084834,"e2e_p50_ms":66421.05887993239,"e2e_p95_ms":66680.33669446595,"e2e_p99_ms":66680.7550895412,"itl_p50_ms":31.939859967678785,"itl_p95_ms":32.421850017271936,"itl_p99_ms":32.849360415712006,"validation_errors":[],"timestamp":"2026-07-30T13:46:04+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":2048,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i4096_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i4096_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":706.5170238499995,"request_tps":0.1132315249306502,"input_tps":463.7963261159432,"output_tps":463.7963261159432,"total_tps":927.5926522318864,"mean_input_tokens":4096.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1639.9033750640228,"ttft_p95_ms":2058.579426078358,"ttft_p99_ms":2096.6388538689353,"tpot_p50_ms":34.08195242002579,"tpot_p95_ms":34.49599784382867,"tpot_p99_ms":34.49673053152434,"e2e_p50_ms":141183.28500003554,"e2e_p95_ms":141877.09997143247,"e2e_p99_ms":141898.9781339164,"itl_p50_ms":34.00721005164087,"itl_p95_ms":34.6460499567911,"itl_p99_ms":35.31962999841198,"validation_errors":[],"timestamp":"2026-07-30T13:58:11+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":4096,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i4096_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i4096_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":806.8527105699759,"request_tps":0.19830137260984473,"input_tps":812.242422209924,"output_tps":812.242422209924,"total_tps":1624.484844419848,"mean_input_tokens":4096.0,"mean_output_tokens":4096.0,"ttft_p50_ms":3717.4146799370646,"ttft_p95_ms":6021.254276507534,"ttft_p99_ms":6037.563333724393,"tpot_p50_ms":38.70771579731254,"tpot_p95_ms":39.36125388485806,"tpot_p99_ms":39.44576556182191,"e2e_p50_ms":162232.10595001,"e2e_p95_ms":163103.40704801492,"e2e_p99_ms":164285.2194803767,"itl_p50_ms":37.83732000738382,"itl_p95_ms":39.160358067601926,"itl_p99_ms":40.175691628828645,"validation_errors":[],"timestamp":"2026-07-30T14:12:03+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":4096,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":75.12912,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c32_i4096_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c32_i4096_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":56.43456280999817,"request_tps":1.417571006429891,"input_tps":11612.741684673667,"output_tps":181.44908882302605,"total_tps":11794.190773496694,"mean_input_tokens":8192.0,"mean_output_tokens":128.0,"ttft_p50_ms":4410.53065500455,"ttft_p95_ms":5877.966689999446,"ttft_p99_ms":6761.926427773428,"tpot_p50_ms":52.69881708668472,"tpot_p95_ms":76.54478422835409,"tpot_p99_ms":90.59493893430216,"e2e_p50_ms":11069.294879969675,"e2e_p95_ms":12872.659270430448,"e2e_p99_ms":13885.651854466409,"itl_p50_ms":34.114275011233985,"itl_p95_ms":35.18050498678349,"itl_p99_ms":1152.7843369520283,"validation_errors":[],"timestamp":"2026-07-30T14:13:22+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i8192_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i8192_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":30.27523225999903,"request_tps":1.321211994559981,"input_tps":10823.368659435364,"output_tps":169.11513530367756,"total_tps":10992.48379473904,"mean_input_tokens":8192.0,"mean_output_tokens":128.0,"ttft_p50_ms":1568.2697700103745,"ttft_p95_ms":2110.2318435674533,"ttft_p99_ms":2110.707156478893,"tpot_p50_ms":35.20471303160925,"tpot_p95_ms":39.82134561011326,"tpot_p99_ms":39.822415161536874,"e2e_p50_ms":6025.207720056642,"e2e_p95_ms":6127.16651004157,"e2e_p99_ms":6127.803050625371,"itl_p50_ms":31.76189004443586,"itl_p95_ms":32.31932002818211,"itl_p99_ms":34.25376053433865,"validation_errors":[],"timestamp":"2026-07-30T14:14:11+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i8192_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i8192_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":75.96618373005185,"request_tps":1.0531001568313927,"input_tps":8626.996484762769,"output_tps":269.5936401488365,"total_tps":8896.590124911605,"mean_input_tokens":8192.0,"mean_output_tokens":256.0,"ttft_p50_ms":4248.153389955405,"ttft_p95_ms":5938.547791051678,"ttft_p99_ms":6023.076416938563,"tpot_p50_ms":40.79689786258135,"tpot_p95_ms":53.188870084385265,"tpot_p99_ms":58.611405439035714,"e2e_p50_ms":14568.05786001496,"e2e_p95_ms":16862.43658301537,"e2e_p99_ms":17145.15267942682,"itl_p50_ms":34.111529937945306,"itl_p95_ms":35.01285403035581,"itl_p99_ms":38.003555145114646,"validation_errors":[],"timestamp":"2026-07-30T14:15:48+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i8192_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i8192_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":51.03208633000031,"request_tps":0.7838205896842814,"input_tps":6421.058270693633,"output_tps":200.65807095917603,"total_tps":6621.716341652809,"mean_input_tokens":8192.0,"mean_output_tokens":256.0,"ttft_p50_ms":1630.7841700036079,"ttft_p95_ms":2134.156580985291,"ttft_p99_ms":2135.7517868978903,"tpot_p50_ms":33.34912580401436,"tpot_p95_ms":35.9359589373629,"tpot_p99_ms":35.93609775237137,"e2e_p50_ms":10222.651210031472,"e2e_p95_ms":10249.125796032602,"e2e_p99_ms":10249.958701712312,"itl_p50_ms":31.979900028090924,"itl_p95_ms":32.54858597647399,"itl_p99_ms":33.9589983096812,"validation_errors":[],"timestamp":"2026-07-30T14:16:58+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i8192_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i8192_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":112.46329941996373,"request_tps":0.7113431707286274,"input_tps":5827.323254608916,"output_tps":364.20770341305723,"total_tps":6191.530958021973,"mean_input_tokens":8192.0,"mean_output_tokens":512.0,"ttft_p50_ms":3798.9035400096327,"ttft_p95_ms":5956.667397049024,"ttft_p99_ms":5966.748825442046,"tpot_p50_ms":36.19370206453748,"tpot_p95_ms":43.58348893431114,"tpot_p99_ms":43.64095074100941,"e2e_p50_ms":23257.443999988027,"e2e_p95_ms":23386.611185065703,"e2e_p99_ms":23387.88044301793,"itl_p50_ms":34.0805200394243,"itl_p95_ms":34.75821397732943,"itl_p99_ms":35.78455118462444,"validation_errors":[],"timestamp":"2026-07-30T14:19:12+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i8192_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i8192_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":92.06817094993312,"request_tps":0.43446067829187235,"input_tps":3559.1018765670183,"output_tps":222.44386728543864,"total_tps":3781.545743852457,"mean_input_tokens":8192.0,"mean_output_tokens":512.0,"ttft_p50_ms":1617.0370299369097,"ttft_p95_ms":2120.281267008977,"ttft_p99_ms":2120.872485007858,"tpot_p50_ms":32.75202892364391,"tpot_p95_ms":34.0880753023189,"tpot_p99_ms":34.08814192439727,"e2e_p50_ms":18406.319784990046,"e2e_p95_ms":18497.071871592198,"e2e_p99_ms":18497.472389419563,"itl_p50_ms":32.07732003647834,"itl_p95_ms":32.54115258459933,"itl_p99_ms":33.10289443470538,"validation_errors":[],"timestamp":"2026-07-30T14:21:03+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":512,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i8192_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i8192_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":195.57287145999726,"request_tps":0.4090546884278033,"input_tps":3350.9760076005646,"output_tps":418.87200095007057,"total_tps":3769.8480085506353,"mean_input_tokens":8192.0,"mean_output_tokens":1024.0,"ttft_p50_ms":3250.0134150031954,"ttft_p95_ms":5368.418099050176,"ttft_p99_ms":5969.804349015467,"tpot_p50_ms":35.14680699905051,"tpot_p95_ms":36.824488353222534,"tpot_p99_ms":38.83839585409816,"e2e_p50_ms":38723.409665049985,"e2e_p95_ms":40826.88306341297,"e2e_p99_ms":40850.94169894233,"itl_p50_ms":34.085749997757375,"itl_p95_ms":34.756814484717324,"itl_p99_ms":35.6551459012553,"validation_errors":[],"timestamp":"2026-07-30T14:24:41+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i8192_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i8192_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":174.02791091008112,"request_tps":0.22984818809132113,"input_tps":1882.9163568441027,"output_tps":235.36454460551283,"total_tps":2118.2809014496156,"mean_input_tokens":8192.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1610.9142000204884,"ttft_p95_ms":2092.5945066090208,"ttft_p99_ms":2094.137596510118,"tpot_p50_ms":32.417914770233594,"tpot_p95_ms":33.059512150545814,"tpot_p99_ms":33.06554270215889,"e2e_p50_ms":34785.434790013824,"e2e_p95_ms":34877.831318497185,"e2e_p99_ms":34878.388072148664,"itl_p50_ms":32.06026001134887,"itl_p95_ms":32.48985345708206,"itl_p99_ms":32.94086805428378,"validation_errors":[],"timestamp":"2026-07-30T14:27:54+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":1024,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i8192_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i8192_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":376.0009511800017,"request_tps":0.2127654192068835,"input_tps":1742.9743141427896,"output_tps":435.7435785356974,"total_tps":2178.717892678487,"mean_input_tokens":8192.0,"mean_output_tokens":2048.0,"ttft_p50_ms":4215.830480039585,"ttft_p95_ms":5938.5147360677365,"ttft_p99_ms":5943.39860582375,"tpot_p50_ms":34.59169691498266,"tpot_p95_ms":36.44177634122271,"tpot_p99_ms":36.456001626638034,"e2e_p50_ms":75630.93460997334,"e2e_p95_ms":75684.31272897287,"e2e_p99_ms":75710.1647824049,"itl_p50_ms":34.06839998206124,"itl_p95_ms":34.741365001536906,"itl_p99_ms":35.46995103824884,"validation_errors":[],"timestamp":"2026-07-30T14:34:33+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i8192_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i8192_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":347.30351269000676,"request_tps":0.11517303608645865,"input_tps":943.4975116202693,"output_tps":235.87437790506732,"total_tps":1179.3718895253367,"mean_input_tokens":8192.0,"mean_output_tokens":2048.0,"ttft_p50_ms":3261.7873449926265,"ttft_p95_ms":3916.709963430185,"ttft_p99_ms":3918.1230788037647,"tpot_p50_ms":32.31261887882882,"tpot_p95_ms":33.46904774058135,"tpot_p99_ms":33.469143586509546,"e2e_p50_ms":69489.51499001123,"e2e_p95_ms":69518.02462543128,"e2e_p99_ms":69518.32342785783,"itl_p50_ms":32.06896490883082,"itl_p95_ms":32.53280749777332,"itl_p99_ms":33.049817936262116,"validation_errors":[],"timestamp":"2026-07-30T14:40:39+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":2048,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i8192_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i8192_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":718.7134695600253,"request_tps":0.11131000515264253,"input_tps":911.8515622104476,"output_tps":455.9257811052238,"total_tps":1367.7773433156713,"mean_input_tokens":8192.0,"mean_output_tokens":4096.0,"ttft_p50_ms":3247.5171950645745,"ttft_p95_ms":5330.93454002519,"ttft_p99_ms":5884.174860934727,"tpot_p50_ms":34.33161741148189,"tpot_p95_ms":34.88633165523948,"tpot_p99_ms":35.285465796003905,"e2e_p50_ms":143252.7803099947,"e2e_p95_ms":145575.84549940075,"e2e_p99_ms":145601.06625951827,"itl_p50_ms":34.0665599796921,"itl_p95_ms":34.72334996331483,"itl_p99_ms":35.44296506443061,"validation_errors":[],"timestamp":"2026-07-30T14:53:00+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":4096,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i8192_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i8192_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":666.2166640399955,"request_tps":0.06004052759268515,"input_tps":491.8520020392767,"output_tps":245.92600101963836,"total_tps":737.7780030589151,"mean_input_tokens":8192.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1604.0335899451748,"ttft_p95_ms":2087.327370059211,"ttft_p99_ms":2087.992780810455,"tpot_p50_ms":32.1549358095272,"tpot_p95_ms":32.35003905201539,"tpot_p99_ms":32.350065979004135,"e2e_p50_ms":133274.52575002098,"e2e_p95_ms":133536.22307849582,"e2e_p99_ms":133537.003906233,"itl_p50_ms":32.0477900095284,"itl_p95_ms":32.53813702031039,"itl_p99_ms":33.10175247373991,"validation_errors":[],"timestamp":"2026-07-30T15:04:26+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":4096,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i8192_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i8192_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":78.54383354005404,"request_tps":1.0185395389340575,"input_tps":16687.751805895597,"output_tps":130.37306098355936,"total_tps":16818.124866879156,"mean_input_tokens":16384.0,"mean_output_tokens":128.0,"ttft_p50_ms":6874.020555056632,"ttft_p95_ms":9451.94394995924,"ttft_p99_ms":10353.470595278775,"tpot_p50_ms":72.06764787354867,"tpot_p95_ms":107.56382110043387,"tpot_p99_ms":108.73315689162219,"e2e_p50_ms":15984.258159936871,"e2e_p95_ms":16507.20967692905,"e2e_p99_ms":16598.339111658275,"itl_p50_ms":34.22969998791814,"itl_p95_ms":35.999584000092,"itl_p99_ms":1338.2082098466344,"validation_errors":[],"timestamp":"2026-07-30T15:06:11+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i16384_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i16384_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":61.316923509933986,"request_tps":0.6523484498291826,"input_tps":10688.077002001328,"output_tps":83.50060157813537,"total_tps":10771.577603579462,"mean_input_tokens":16384.0,"mean_output_tokens":128.0,"ttft_p50_ms":5333.7890800321475,"ttft_p95_ms":8643.30326595227,"ttft_p99_ms":9381.80954080075,"tpot_p50_ms":56.017472007674876,"tpot_p95_ms":81.30520194811979,"tpot_p99_ms":92.00828380428995,"e2e_p50_ms":13196.060694986954,"e2e_p95_ms":14456.196296971755,"e2e_p99_ms":15841.04720049072,"itl_p50_ms":32.041669939644635,"itl_p95_ms":33.5994349909015,"itl_p99_ms":1312.4838541750803,"validation_errors":[],"timestamp":"2026-07-30T15:07:34+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i16384_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i16384_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":30.153587660053745,"request_tps":0.16581774800296145,"input_tps":2716.7579832805204,"output_tps":21.224671744379066,"total_tps":2737.9826550248995,"mean_input_tokens":16384.0,"mean_output_tokens":128.0,"ttft_p50_ms":1926.8963900394738,"ttft_p95_ms":1934.0090099722147,"ttft_p99_ms":1934.414833970368,"tpot_p50_ms":32.209837637790365,"tpot_p95_ms":32.67882642497521,"tpot_p99_ms":32.69941468682786,"e2e_p50_ms":6025.165669969283,"e2e_p95_ms":6074.604873964563,"e2e_p99_ms":6076.719088810496,"itl_p50_ms":32.322490005753934,"itl_p95_ms":32.959625916555524,"itl_p99_ms":33.38210338493809,"validation_errors":[],"timestamp":"2026-07-30T15:08:23+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i16384_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i16384_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":113.08389432006516,"request_tps":0.7074393792415152,"input_tps":11590.686789492986,"output_tps":181.1044810858279,"total_tps":11771.791270578813,"mean_input_tokens":16384.0,"mean_output_tokens":256.0,"ttft_p50_ms":7087.325385014992,"ttft_p95_ms":11295.852046535578,"ttft_p99_ms":19893.243561796826,"tpot_p50_ms":53.80412580390625,"tpot_p95_ms":106.70584383311079,"tpot_p99_ms":115.45597858694109,"e2e_p50_ms":20855.918050045148,"e2e_p95_ms":32611.908536014384,"e2e_p99_ms":32624.76044652052,"itl_p50_ms":34.547719988040626,"itl_p95_ms":35.55087202694267,"itl_p99_ms":1239.0348487533622,"validation_errors":[],"timestamp":"2026-07-30T15:10:42+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i16384_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i16384_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":80.96910245006438,"request_tps":0.49401560335523015,"input_tps":8093.951645372091,"output_tps":126.46799445893892,"total_tps":8220.41963983103,"mean_input_tokens":16384.0,"mean_output_tokens":256.0,"ttft_p50_ms":5373.884794942569,"ttft_p95_ms":7795.870495005509,"ttft_p99_ms":9066.102823694235,"tpot_p50_ms":44.10620494106529,"tpot_p95_ms":56.782571534544495,"tpot_p99_ms":60.7626779786649,"e2e_p50_ms":17214.721900003497,"e2e_p95_ms":17827.14329601149,"e2e_p99_ms":19586.994164984204,"itl_p50_ms":32.124120043590665,"itl_p95_ms":32.7510500093922,"itl_p99_ms":45.046042413450756,"validation_errors":[],"timestamp":"2026-07-30T15:12:25+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i16384_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i16384_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":49.315239269984886,"request_tps":0.10138853778294833,"input_tps":1661.1498030358255,"output_tps":25.955465672434773,"total_tps":1687.1052687082602,"mean_input_tokens":16384.0,"mean_output_tokens":256.0,"ttft_p50_ms":1943.994740024209,"ttft_p95_ms":1945.280466042459,"ttft_p99_ms":1945.5244612507522,"tpot_p50_ms":32.254649568623044,"tpot_p95_ms":32.32996336494883,"tpot_p99_ms":32.341423224949956,"e2e_p50_ms":10144.916270044632,"e2e_p95_ms":10187.835760111922,"e2e_p99_ms":10192.270367210731,"itl_p50_ms":32.270890078507364,"itl_p95_ms":32.7179629704915,"itl_p99_ms":33.01828076597303,"validation_errors":[],"timestamp":"2026-07-30T15:13:33+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":256,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i16384_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i16384_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":152.78934546001256,"request_tps":0.5235967191242226,"input_tps":8578.608646131263,"output_tps":268.08152019160195,"total_tps":8846.690166322864,"mean_input_tokens":16384.0,"mean_output_tokens":512.0,"ttft_p50_ms":7106.789550045505,"ttft_p95_ms":9347.392498987028,"ttft_p99_ms":9572.804082031362,"tpot_p50_ms":47.65913015643386,"tpot_p95_ms":52.29339734546352,"tpot_p99_ms":55.31886173072754,"e2e_p50_ms":31046.419079997577,"e2e_p95_ms":34771.474287944155,"e2e_p99_ms":35013.140914095566,"itl_p50_ms":34.27530999761075,"itl_p95_ms":35.183938447153196,"itl_p99_ms":38.24577150866388,"validation_errors":[],"timestamp":"2026-07-30T15:16:32+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i16384_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i16384_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":116.66552497004159,"request_tps":0.34286049808005886,"input_tps":5617.426400543684,"output_tps":175.54457501699014,"total_tps":5792.970975560675,"mean_input_tokens":16384.0,"mean_output_tokens":512.0,"ttft_p50_ms":4725.479695014656,"ttft_p95_ms":6912.078658514655,"ttft_p99_ms":7688.496480534086,"tpot_p50_ms":37.123617446282424,"tpot_p95_ms":42.58101770653736,"tpot_p99_ms":44.4524020558964,"e2e_p50_ms":23718.792780069634,"e2e_p95_ms":25863.707117567537,"e2e_p99_ms":26382.790781827644,"itl_p50_ms":32.209435012191534,"itl_p95_ms":32.73179245297797,"itl_p99_ms":33.84482345427388,"validation_errors":[],"timestamp":"2026-07-30T15:18:50+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":512,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i16384_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i16384_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":91.6290129100671,"request_tps":0.054567869293838694,"input_tps":894.0399705102532,"output_tps":27.93874907844541,"total_tps":921.9787195886986,"mean_input_tokens":16384.0,"mean_output_tokens":512.0,"ttft_p50_ms":1938.2234700024128,"ttft_p95_ms":1941.4331679930913,"ttft_p99_ms":1941.7911455919966,"tpot_p50_ms":32.60511512719569,"tpot_p95_ms":32.75817168280555,"tpot_p99_ms":32.75956868735341,"e2e_p50_ms":18599.43729999941,"e2e_p95_ms":18664.182707923464,"e2e_p99_ms":18676.805637632497,"itl_p50_ms":32.661040080711246,"itl_p95_ms":33.07889602147043,"itl_p99_ms":33.32370156655088,"validation_errors":[],"timestamp":"2026-07-30T15:20:41+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":512,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i16384_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i16384_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":235.20039203006309,"request_tps":0.3401354874858137,"input_tps":5572.779826967571,"output_tps":348.2987391854732,"total_tps":5921.078566153044,"mean_input_tokens":16384.0,"mean_output_tokens":1024.0,"ttft_p50_ms":6960.378815070726,"ttft_p95_ms":9869.751720567001,"ttft_p99_ms":12456.567441336794,"tpot_p50_ms":38.860051974632356,"tpot_p95_ms":45.34512697652466,"tpot_p99_ms":47.532708697547434,"e2e_p50_ms":46719.239865022246,"e2e_p95_ms":51425.21330044838,"e2e_p99_ms":51426.03578419541,"itl_p50_ms":34.37917004339397,"itl_p95_ms":35.12194408103824,"itl_p99_ms":36.044206966180354,"validation_errors":[],"timestamp":"2026-07-30T15:25:02+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i16384_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i16384_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":197.51548019004986,"request_tps":0.20251577223978548,"input_tps":3318.0184123766453,"output_tps":207.37615077354033,"total_tps":3525.3945631501856,"mean_input_tokens":16384.0,"mean_output_tokens":1024.0,"ttft_p50_ms":5385.595969972201,"ttft_p95_ms":6330.596724478525,"ttft_p99_ms":7234.8148909944575,"tpot_p50_ms":33.931499677389326,"tpot_p95_ms":36.427321220861465,"tpot_p99_ms":36.79423793949954,"e2e_p50_ms":38932.60721000843,"e2e_p95_ms":40929.65717004844,"e2e_p99_ms":40954.95652089128,"itl_p50_ms":32.15734998229891,"itl_p95_ms":32.696440001018345,"itl_p99_ms":33.47471099114046,"validation_errors":[],"timestamp":"2026-07-30T15:28:41+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":1024,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i16384_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i16384_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":174.60446656006388,"request_tps":0.02863615174632432,"input_tps":469.17471021177766,"output_tps":29.323419388236104,"total_tps":498.4981296000138,"mean_input_tokens":16384.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1924.8342500068247,"ttft_p95_ms":1929.1099580703303,"ttft_p99_ms":1929.1597116878256,"tpot_p50_ms":32.57387517113876,"tpot_p95_ms":32.64516331769803,"tpot_p99_ms":32.6479203809182,"e2e_p50_ms":35225.39343999233,"e2e_p95_ms":35298.23711398058,"e2e_p99_ms":35308.31841240637,"itl_p50_ms":32.55492995958775,"itl_p95_ms":32.952270086389035,"itl_p99_ms":33.21169619448483,"validation_errors":[],"timestamp":"2026-07-30T15:31:54+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":1024,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i16384_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i16384_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":390.7010881199967,"request_tps":0.20476011568063374,"input_tps":3354.789735311503,"output_tps":419.3487169139379,"total_tps":3774.138452225441,"mean_input_tokens":16384.0,"mean_output_tokens":2048.0,"ttft_p50_ms":5130.671439983416,"ttft_p95_ms":8147.44449844584,"ttft_p99_ms":8748.909668202978,"tpot_p50_ms":35.777547022450165,"tpot_p95_ms":37.130453392038525,"tpot_p99_ms":37.13367618751768,"e2e_p50_ms":78087.10496500134,"e2e_p95_ms":78742.37950344685,"e2e_p99_ms":78784.66001468129,"itl_p50_ms":34.184560005087405,"itl_p95_ms":35.038357513258234,"itl_p99_ms":35.86694504483603,"validation_errors":[],"timestamp":"2026-07-30T15:38:50+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i16384_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i16384_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":363.82582417002413,"request_tps":0.10994271803341558,"input_tps":1801.301492259481,"output_tps":225.16268653243512,"total_tps":2026.464178791916,"mean_input_tokens":16384.0,"mean_output_tokens":2048.0,"ttft_p50_ms":4057.0984299993142,"ttft_p95_ms":7887.51998557127,"ttft_p99_ms":8360.044797008157,"tpot_p50_ms":33.1913771788086,"tpot_p95_ms":34.70254537765925,"tpot_p99_ms":35.13001048312027,"e2e_p50_ms":72020.48604004085,"e2e_p95_ms":74572.34151496086,"e2e_p99_ms":74573.9457025344,"itl_p50_ms":32.20250003505498,"itl_p95_ms":32.716152450302616,"itl_p99_ms":33.429031011182815,"validation_errors":[],"timestamp":"2026-07-30T15:45:15+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":2048,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i16384_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i16384_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":338.99795087997336,"request_tps":0.014749351690831651,"input_tps":241.65337810258578,"output_tps":30.206672262823222,"total_tps":271.860050365409,"mean_input_tokens":16384.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1925.9122299263254,"ttft_p95_ms":1938.0354379536584,"ttft_p99_ms":1939.75881554652,"tpot_p50_ms":32.30771121639953,"tpot_p95_ms":32.52749904345504,"tpot_p99_ms":32.56950669742221,"e2e_p50_ms":67924.71943993587,"e2e_p95_ms":68507.78883593157,"e2e_p99_ms":68598.11470841058,"itl_p50_ms":32.30682003777474,"itl_p95_ms":32.78064705664292,"itl_p99_ms":33.08233875548467,"validation_errors":[],"timestamp":"2026-07-30T15:51:14+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":2048,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i16384_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i16384_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":98.85014289990067,"request_tps":0.8093058609031144,"input_tps":26519.334450073253,"output_tps":103.59115019559864,"total_tps":26622.925600268853,"mean_input_tokens":32768.0,"mean_output_tokens":128.0,"ttft_p50_ms":8851.623984926846,"ttft_p95_ms":13193.119923002081,"ttft_p99_ms":15431.260468903924,"tpot_p50_ms":89.67701385831681,"tpot_p95_ms":130.33471855932595,"tpot_p99_ms":134.6741715011435,"e2e_p50_ms":19395.432725024875,"e2e_p95_ms":24578.836674086044,"e2e_p99_ms":28224.473414105596,"itl_p50_ms":34.52211007243022,"itl_p95_ms":37.014562002150335,"itl_p99_ms":1837.8846404585056,"validation_errors":[],"timestamp":"2026-07-30T15:53:20+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i32768_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i32768_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":75.02505228994414,"request_tps":0.5331552432034937,"input_tps":17470.431009292082,"output_tps":68.2438711300472,"total_tps":17538.674880422128,"mean_input_tokens":32768.0,"mean_output_tokens":128.0,"ttft_p50_ms":6087.901180027984,"ttft_p95_ms":10196.022363082726,"ttft_p99_ms":15548.96268702345,"tpot_p50_ms":61.076136141846796,"tpot_p95_ms":127.75958901510472,"tpot_p99_ms":133.99942048948154,"e2e_p50_ms":13035.684015019797,"e2e_p95_ms":20425.419829040762,"e2e_p99_ms":21874.157680025088,"itl_p50_ms":32.33880992047489,"itl_p95_ms":33.156920992769294,"itl_p99_ms":1950.7074855733672,"validation_errors":[],"timestamp":"2026-07-30T15:54:57+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i32768_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i32768_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":34.82678088999819,"request_tps":0.1435676761453407,"input_tps":4704.425611930524,"output_tps":18.376662546603608,"total_tps":4722.802274477127,"mean_input_tokens":32768.0,"mean_output_tokens":128.0,"ttft_p50_ms":2103.229279979132,"ttft_p95_ms":3998.993751988746,"ttft_p99_ms":4000.512782377191,"tpot_p50_ms":32.196829684892684,"tpot_p95_ms":32.6885164887681,"tpot_p99_ms":32.721275168353706,"e2e_p50_ms":6233.9529800228775,"e2e_p95_ms":8142.089882027358,"e2e_p99_ms":8154.445675555616,"itl_p50_ms":32.35683997627348,"itl_p95_ms":33.07641998399049,"itl_p99_ms":33.61886723199859,"validation_errors":[],"timestamp":"2026-07-30T15:55:50+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i32768_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i32768_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":105.41555208002683,"request_tps":0.7589013046126963,"input_tps":24867.677949548834,"output_tps":194.27873398085026,"total_tps":25061.956683529683,"mean_input_tokens":32768.0,"mean_output_tokens":256.0,"ttft_p50_ms":7708.562039944809,"ttft_p95_ms":11736.407803586912,"ttft_p99_ms":14006.2151482573,"tpot_p50_ms":55.65175462751558,"tpot_p95_ms":75.59189497842453,"tpot_p99_ms":82.48523461568054,"e2e_p50_ms":18821.767169982195,"e2e_p95_ms":27293.3901234006,"e2e_p99_ms":27341.844631694257,"itl_p50_ms":34.61895999498665,"itl_p95_ms":35.7133699581027,"itl_p99_ms":1246.5273199952208,"validation_errors":[],"timestamp":"2026-07-30T15:58:02+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i32768_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i32768_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":84.12943504995201,"request_tps":0.4754578463084879,"input_tps":15579.802707836532,"output_tps":121.71720865497291,"total_tps":15701.519916491505,"mean_input_tokens":32768.0,"mean_output_tokens":256.0,"ttft_p50_ms":5852.399714989588,"ttft_p95_ms":9705.782485537928,"ttft_p99_ms":11655.279470100066,"tpot_p50_ms":43.056235980589456,"tpot_p95_ms":64.86480446288125,"tpot_p99_ms":68.558204426978,"e2e_p50_ms":15908.27104501659,"e2e_p95_ms":20770.184373517986,"e2e_p99_ms":21657.57642097422,"itl_p50_ms":32.50819002278149,"itl_p95_ms":33.10221803840249,"itl_p99_ms":39.10895352717358,"validation_errors":[],"timestamp":"2026-07-30T15:59:48+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i32768_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i32768_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":50.495257050031796,"request_tps":0.0990192008537731,"input_tps":3244.661173576437,"output_tps":25.348915418565912,"total_tps":3270.0100889950027,"mean_input_tokens":32768.0,"mean_output_tokens":256.0,"ttft_p50_ms":2109.8566600121558,"ttft_p95_ms":2114.9296860210598,"ttft_p99_ms":2115.9110332280397,"tpot_p50_ms":32.63095745077247,"tpot_p95_ms":32.65437352964107,"tpot_p99_ms":32.65740065251057,"e2e_p50_ms":10424.659930053167,"e2e_p95_ms":10441.794936079532,"e2e_p99_ms":10443.548199618235,"itl_p50_ms":32.65573992393911,"itl_p95_ms":33.05169598897919,"itl_p99_ms":33.37943659396842,"validation_errors":[],"timestamp":"2026-07-30T16:00:57+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":256,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i32768_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i32768_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":166.99366545001976,"request_tps":0.4790600876052004,"input_tps":15697.840950647207,"output_tps":245.2787648538626,"total_tps":15943.11971550107,"mean_input_tokens":32768.0,"mean_output_tokens":512.0,"ttft_p50_ms":7914.2759799724445,"ttft_p95_ms":13996.461907075718,"ttft_p99_ms":17174.03191690913,"tpot_p50_ms":48.02012970646454,"tpot_p95_ms":59.30876912228597,"tpot_p99_ms":64.17564651898769,"e2e_p50_ms":33162.07304992713,"e2e_p95_ms":41142.39283248898,"e2e_p99_ms":42402.05851594683,"itl_p50_ms":34.622090053744614,"itl_p95_ms":35.5443800566718,"itl_p99_ms":69.24455601722006,"validation_errors":[],"timestamp":"2026-07-30T16:04:12+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i32768_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i32768_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":115.05969306000043,"request_tps":0.347645634506787,"input_tps":11391.652151518396,"output_tps":177.99456486747493,"total_tps":11569.64671638587,"mean_input_tokens":32768.0,"mean_output_tokens":512.0,"ttft_p50_ms":4204.793805023655,"ttft_p95_ms":7222.958084999117,"ttft_p99_ms":8325.086588173872,"tpot_p50_ms":35.855004295443386,"tpot_p95_ms":42.46631518017488,"tpot_p99_ms":43.90301907054602,"e2e_p50_ms":22618.837285030168,"e2e_p95_ms":25603.36778697674,"e2e_p99_ms":25660.158702358603,"itl_p50_ms":32.51846501370892,"itl_p95_ms":32.97453897539526,"itl_p99_ms":33.979088019113945,"validation_errors":[],"timestamp":"2026-07-30T16:06:29+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":512,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i32768_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i32768_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":92.28715785010718,"request_tps":0.05417871908159747,"input_tps":1775.328266865786,"output_tps":27.739504169777906,"total_tps":1803.067771035564,"mean_input_tokens":32768.0,"mean_output_tokens":512.0,"ttft_p50_ms":2107.017490081489,"ttft_p95_ms":2118.4106060070917,"ttft_p99_ms":2119.6295132068917,"tpot_p50_ms":32.63437090015108,"tpot_p95_ms":32.64904802345257,"tpot_p99_ms":32.65114072410466,"e2e_p50_ms":18783.7315100478,"e2e_p95_ms":18786.262421985157,"e2e_p99_ms":18786.44774395507,"itl_p50_ms":32.64355007559061,"itl_p95_ms":33.03115892922506,"itl_p99_ms":33.24903703760356,"validation_errors":[],"timestamp":"2026-07-30T16:08:20+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":512,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i32768_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i32768_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":265.29733160010073,"request_tps":0.30154845326747953,"input_tps":9881.13971666877,"output_tps":308.78561614589904,"total_tps":10189.925332814668,"mean_input_tokens":32768.0,"mean_output_tokens":1024.0,"ttft_p50_ms":8315.445540007204,"ttft_p95_ms":16534.218570508525,"ttft_p99_ms":22289.87379098543,"tpot_p50_ms":41.86792332846999,"tpot_p95_ms":53.57580177370097,"tpot_p99_ms":54.70089958820451,"e2e_p50_ms":53946.871265012305,"e2e_p95_ms":62164.320686040446,"e2e_p99_ms":65391.87750443115,"itl_p50_ms":34.58937001414597,"itl_p95_ms":35.499829961918294,"itl_p99_ms":36.96433433331548,"validation_errors":[],"timestamp":"2026-07-30T16:13:13+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i32768_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i32768_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":205.38162611995358,"request_tps":0.1947593889272155,"input_tps":6381.875656366998,"output_tps":199.43361426146868,"total_tps":6581.309270628466,"mean_input_tokens":32768.0,"mean_output_tokens":1024.0,"ttft_p50_ms":4774.620740034152,"ttft_p95_ms":8818.138360464944,"ttft_p99_ms":9376.210473002866,"tpot_p50_ms":34.706421256090856,"tpot_p95_ms":38.07852678087601,"tpot_p99_ms":38.95898172768304,"e2e_p50_ms":40861.442404973786,"e2e_p95_ms":42907.905896921875,"e2e_p99_ms":42935.60153797385,"itl_p50_ms":32.50852494966239,"itl_p95_ms":32.976125518325716,"itl_p99_ms":33.681507869623594,"validation_errors":[],"timestamp":"2026-07-30T16:17:01+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":1024,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i32768_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i32768_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":176.666273789946,"request_tps":0.028301949731191694,"input_tps":927.3982887916894,"output_tps":28.981196524740295,"total_tps":956.3794853164297,"mean_input_tokens":32768.0,"mean_output_tokens":1024.0,"ttft_p50_ms":2099.7722600586712,"ttft_p95_ms":2102.8554920107126,"ttft_p99_ms":2102.9376424103975,"tpot_p50_ms":32.80972172041858,"tpot_p95_ms":32.8244334252238,"tpot_p99_ms":32.82508109481359,"e2e_p50_ms":35642.18627009541,"e2e_p95_ms":35676.10430603381,"e2e_p99_ms":35678.51781039499,"itl_p50_ms":32.80535992234945,"itl_p95_ms":33.186648006085306,"itl_p99_ms":33.392088799737394,"validation_errors":[],"timestamp":"2026-07-30T16:20:17+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":1024,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i32768_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i32768_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":195.72141315997578,"request_tps":0.4087442386010713,"input_tps":26787.462420959808,"output_tps":52.319262540937125,"total_tps":26839.781683500743,"mean_input_tokens":65536.0,"mean_output_tokens":128.0,"ttft_p50_ms":15644.347299996298,"ttft_p95_ms":25142.435244534863,"ttft_p99_ms":29374.71614571285,"tpot_p50_ms":177.5789429524393,"tpot_p95_ms":261.61006811009827,"tpot_p99_ms":289.2081087611075,"e2e_p50_ms":38504.18606004678,"e2e_p95_ms":47884.66295648249,"e2e_p99_ms":54362.32540329451,"itl_p50_ms":35.10395996272564,"itl_p95_ms":1442.057252046652,"itl_p99_ms":2383.873494393192,"validation_errors":[],"timestamp":"2026-07-30T16:24:06+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":65536,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i65536_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i65536_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":89.23524214001372,"request_tps":0.4482533922778892,"input_tps":29376.734316323746,"output_tps":57.376434211569816,"total_tps":29434.110750535314,"mean_input_tokens":65536.0,"mean_output_tokens":128.0,"ttft_p50_ms":7049.079604970757,"ttft_p95_ms":17236.322649545033,"ttft_p99_ms":19710.26847048895,"tpot_p50_ms":63.157572165336894,"tpot_p95_ms":129.19235332253913,"tpot_p99_ms":144.86635605627706,"e2e_p50_ms":20118.624389986508,"e2e_p95_ms":24636.071049969178,"e2e_p99_ms":24909.897351927357,"itl_p50_ms":32.931419962551445,"itl_p95_ms":33.97480295971036,"itl_p99_ms":1889.7423148283265,"validation_errors":[],"timestamp":"2026-07-30T16:25:59+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":65536,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i65536_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i65536_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":40.09191425004974,"request_tps":0.12471342647336418,"input_tps":8173.219117358395,"output_tps":15.963318588590615,"total_tps":8189.182435946986,"mean_input_tokens":65536.0,"mean_output_tokens":128.0,"ttft_p50_ms":4697.579369996674,"ttft_p95_ms":4769.806634052657,"ttft_p99_ms":4779.943734849803,"tpot_p50_ms":32.945965590894104,"tpot_p95_ms":33.06239844138521,"tpot_p99_ms":33.07117123450116,"e2e_p50_ms":8844.770389958285,"e2e_p95_ms":8958.708780142479,"e2e_p99_ms":8977.97795563005,"itl_p50_ms":33.03731989581138,"itl_p95_ms":33.565240050666034,"itl_p99_ms":34.63177818572148,"validation_errors":[],"timestamp":"2026-07-30T16:26:59+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":65536,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i65536_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i65536_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":189.37725268991198,"request_tps":0.4224372191679891,"input_tps":27684.845595393333,"output_tps":108.14392810700521,"total_tps":27792.98952350034,"mean_input_tokens":65536.0,"mean_output_tokens":256.0,"ttft_p50_ms":12045.162599999458,"ttft_p95_ms":25760.4014329554,"ttft_p99_ms":33501.22540239127,"tpot_p50_ms":87.36160049014086,"tpot_p95_ms":148.25696377658886,"tpot_p99_ms":170.67473581063504,"e2e_p50_ms":34465.053265041206,"e2e_p95_ms":60720.90335303218,"e2e_p99_ms":67056.30021626012,"itl_p50_ms":35.531270084902644,"itl_p95_ms":37.702583987265804,"itl_p99_ms":1962.2510332381353,"validation_errors":[],"timestamp":"2026-07-30T16:30:39+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":65536,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i65536_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i65536_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":111.85941335000098,"request_tps":0.3575917198389245,"input_tps":23435.130951363757,"output_tps":91.54348027876468,"total_tps":23526.67443164252,"mean_input_tokens":65536.0,"mean_output_tokens":256.0,"ttft_p50_ms":6120.151620008983,"ttft_p95_ms":16741.536042513315,"ttft_p99_ms":25361.303362770697,"tpot_p50_ms":42.08146796091551,"tpot_p95_ms":125.8893844626579,"tpot_p99_ms":129.43661638726846,"e2e_p50_ms":15615.934719971847,"e2e_p95_ms":36745.726476499105,"e2e_p99_ms":37001.29330721451,"itl_p50_ms":33.11860002577305,"itl_p95_ms":33.71075796894729,"itl_p99_ms":1716.2374512711538,"validation_errors":[],"timestamp":"2026-07-30T16:32:55+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":65536,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i65536_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i65536_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":52.896072340081446,"request_tps":0.0945249765966329,"input_tps":6194.788866236934,"output_tps":24.198394008738024,"total_tps":6218.987260245673,"mean_input_tokens":65536.0,"mean_output_tokens":256.0,"ttft_p50_ms":2517.244450049475,"ttft_p95_ms":2545.6370279425755,"ttft_p99_ms":2546.1767975473776,"tpot_p50_ms":33.056614039392734,"tpot_p95_ms":33.14055269008831,"tpot_p99_ms":33.150101314426635,"e2e_p50_ms":10945.207500015385,"e2e_p95_ms":10982.504641916603,"e2e_p99_ms":10984.54615843948,"itl_p50_ms":33.14998000860214,"itl_p95_ms":33.58733697095886,"itl_p99_ms":33.93944130977616,"validation_errors":[],"timestamp":"2026-07-30T16:34:07+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":65536,"osl":256,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i65536_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i65536_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":546.8937829199713,"request_tps":0.14628069014217823,"input_tps":19173.302618315585,"output_tps":18.723928338198814,"total_tps":19192.026546653786,"mean_input_tokens":131072.0,"mean_output_tokens":128.0,"ttft_p50_ms":31485.49054999603,"ttft_p95_ms":66379.71984341857,"ttft_p99_ms":72091.88513923435,"tpot_p50_ms":639.2921336218863,"tpot_p95_ms":788.1524168361425,"tpot_p99_ms":853.8619441226928,"e2e_p50_ms":117644.95416998398,"e2e_p95_ms":139681.77636257024,"e2e_p99_ms":142008.94796826408,"itl_p50_ms":36.653870018199086,"itl_p95_ms":2592.090099980121,"itl_p99_ms":3119.810117005836,"validation_errors":[],"timestamp":"2026-07-30T16:44:02+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":131072,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c16_i131072_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c16_i131072_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":200.84793128992897,"request_tps":0.19915564847047892,"input_tps":26103.729156322614,"output_tps":25.491923004221302,"total_tps":26129.221079326835,"mean_input_tokens":131072.0,"mean_output_tokens":128.0,"ttft_p50_ms":14449.635569995735,"ttft_p95_ms":47948.62388498729,"ttft_p99_ms":58404.37893889379,"tpot_p50_ms":139.48554996039908,"tpot_p95_ms":358.7906236411072,"tpot_p99_ms":383.03330750926943,"e2e_p50_ms":47848.52211497491,"e2e_p95_ms":63481.25937407604,"e2e_p99_ms":69165.94623451935,"itl_p50_ms":34.08529004082084,"itl_p95_ms":1778.0194899532944,"itl_p99_ms":2800.35724400077,"validation_errors":[],"timestamp":"2026-07-30T16:47:50+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":131072,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c8_i131072_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c8_i131072_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":85.4150076599326,"request_tps":0.058537722315810954,"input_tps":7672.656339377973,"output_tps":7.492828456423802,"total_tps":7680.149167834397,"mean_input_tokens":131072.0,"mean_output_tokens":128.0,"ttft_p50_ms":19109.781460021622,"ttft_p95_ms":19234.430833999068,"ttft_p99_ms":19259.02652680874,"tpot_p50_ms":33.305770866059525,"tpot_p95_ms":33.62814503883225,"tpot_p99_ms":33.630413953104885,"e2e_p50_ms":23328.823940013535,"e2e_p95_ms":23493.557532015257,"e2e_p99_ms":23526.376488846727,"itl_p50_ms":33.79800997208804,"itl_p95_ms":34.39098694361746,"itl_p99_ms":37.17546101426704,"validation_errors":[],"timestamp":"2026-07-30T16:49:36+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":131072,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/raw_outputs/vllm_adaptive_c1_i131072_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp4_dp4/logs/vllm_c1_i131072_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":29.460983420023695,"request_tps":2.7154558576488843,"input_tps":2780.6267982324575,"output_tps":347.5783497790572,"total_tps":3128.2051480115147,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":878.6893150536343,"ttft_p95_ms":3398.7951504823286,"ttft_p99_ms":3400.879614218138,"tpot_p50_ms":35.55662818913033,"tpot_p95_ms":40.57930129439034,"tpot_p99_ms":58.213182087606334,"e2e_p50_ms":5404.702249972615,"e2e_p95_ms":7799.261582951294,"e2e_p99_ms":7801.3193831185345,"itl_p50_ms":35.238189972005785,"itl_p95_ms":36.08757600886747,"itl_p99_ms":37.04129274236038,"validation_errors":[],"timestamp":"2026-07-30T16:58:38+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":31.915944129927084,"request_tps":5.013168319528749,"input_tps":5133.484359197439,"output_tps":641.6855448996798,"total_tps":5775.169904097118,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1101.1904500192031,"ttft_p95_ms":1552.1230874815956,"ttft_p99_ms":1553.983459128067,"tpot_p50_ms":41.64054330739039,"tpot_p95_ms":46.85725371617325,"tpot_p99_ms":47.123799338848805,"e2e_p50_ms":6357.790054986253,"e2e_p95_ms":6664.46898203576,"e2e_p99_ms":6666.895103772404,"itl_p50_ms":38.025059970095754,"itl_p95_ms":40.05427295342088,"itl_p99_ms":45.419337851926684,"validation_errors":[],"timestamp":"2026-07-30T16:59:31+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":84.616086,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":43.12136841996107,"request_tps":5.565686080799396,"input_tps":5699.2625467385815,"output_tps":712.4078183423227,"total_tps":6411.670365080904,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1392.3956250655465,"ttft_p95_ms":3902.6259889127687,"ttft_p99_ms":4418.104510882404,"tpot_p50_ms":50.39436173190795,"tpot_p95_ms":72.36689143698648,"tpot_p99_ms":77.45665329452567,"e2e_p50_ms":7739.853755047079,"e2e_p95_ms":10301.313012011815,"e2e_p99_ms":13352.006720406935,"itl_p50_ms":41.76427499623969,"itl_p95_ms":45.139143959386274,"itl_p99_ms":352.91854838957084,"validation_errors":[],"timestamp":"2026-07-30T17:00:37+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":11.021329,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c48_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c48_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":50.157450670027174,"request_tps":6.3799095792407154,"input_tps":6533.027409142493,"output_tps":816.6284261428116,"total_tps":7349.655835285304,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1784.8464549751952,"ttft_p95_ms":2615.2132745133713,"ttft_p99_ms":5453.940354832448,"tpot_p50_ms":62.28339236218658,"tpot_p95_ms":81.58984925624488,"tpot_p99_ms":83.4631941716854,"e2e_p50_ms":9285.761455015745,"e2e_p95_ms":12231.235033919802,"e2e_p99_ms":12265.910378013505,"itl_p50_ms":44.42058998392895,"itl_p95_ms":49.53994051902554,"itl_p99_ms":747.1158270293381,"validation_errors":[],"timestamp":"2026-07-30T17:01:51+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":14.629346,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c64_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c64_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":54.3888013799442,"request_tps":7.354455142442236,"input_tps":7530.96206586085,"output_tps":941.3702582326063,"total_tps":8472.332324093457,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1828.1486100167967,"ttft_p95_ms":3366.0936975618824,"ttft_p99_ms":3403.4216033958196,"tpot_p50_ms":66.47177074859873,"tpot_p95_ms":92.58714904738252,"tpot_p99_ms":94.28000483710473,"e2e_p50_ms":10132.766149996314,"e2e_p95_ms":13171.993914607443,"e2e_p99_ms":13252.227063442115,"itl_p50_ms":46.0804000031203,"itl_p95_ms":55.49256398808211,"itl_p99_ms":766.1933508142829,"validation_errors":[],"timestamp":"2026-07-30T17:03:07+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":15.275225,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c80_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c80_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":58.20616724004503,"request_tps":8.246548823949478,"input_tps":8444.465995724266,"output_tps":1055.5582494655332,"total_tps":9500.0242451898,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":1993.3499000035226,"ttft_p95_ms":3893.8902609865186,"ttft_p99_ms":5226.33907989366,"tpot_p50_ms":68.73529370073008,"tpot_p95_ms":97.25648697211744,"tpot_p99_ms":99.85317158131441,"e2e_p50_ms":10895.80219500931,"e2e_p95_ms":13754.646894481268,"e2e_p99_ms":14190.52368654171,"itl_p50_ms":47.69021994434297,"itl_p95_ms":58.422823040746145,"itl_p99_ms":766.7439047840888,"validation_errors":[],"timestamp":"2026-07-30T17:04:27+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":12.129977,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c96_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c96_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":560,"failed":0,"duration_s":63.10225083003752,"request_tps":8.874485341391855,"input_tps":9087.47298958526,"output_tps":1135.9341236981575,"total_tps":10223.407113283418,"mean_input_tokens":1024.0,"mean_output_tokens":128.0,"ttft_p50_ms":2260.820494964719,"ttft_p95_ms":4129.537580587203,"ttft_p99_ms":5355.7953963801265,"tpot_p50_ms":76.86690846458077,"tpot_p95_ms":102.29804364122876,"tpot_p99_ms":107.85478665905924,"e2e_p50_ms":12342.24946500035,"e2e_p95_ms":15017.503727448639,"e2e_p99_ms":15393.233570028096,"itl_p50_ms":49.77301001781598,"itl_p95_ms":65.24340801988726,"itl_p99_ms":768.9476097049192,"validation_errors":[],"timestamp":"2026-07-30T17:05:53+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":128,"concurrency":112,"num_prompts":560,"warmup_requests":112,"attempt":1,"gain_pct":7.614537,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c112_i1024_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c112_i1024_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":49.41169161000289,"request_tps":1.6190500141429043,"input_tps":1657.907214482334,"output_tps":414.4768036205835,"total_tps":2072.3840181029173,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":850.1817650394514,"ttft_p95_ms":889.4849319767673,"ttft_p99_ms":892.0701451471541,"tpot_p50_ms":35.61881564708207,"tpot_p95_ms":37.66624448432898,"tpot_p99_ms":37.676653638979715,"e2e_p50_ms":9861.218680045567,"e2e_p95_ms":9945.12115206453,"e2e_p99_ms":9946.138536310755,"itl_p50_ms":35.36217496730387,"itl_p95_ms":36.232357495464385,"itl_p99_ms":37.04285494168289,"validation_errors":[],"timestamp":"2026-07-30T17:07:01+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":58.19271920993924,"request_tps":2.749484852611462,"input_tps":2815.4724890741372,"output_tps":703.8681222685343,"total_tps":3519.3406113426718,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1168.37755497545,"ttft_p95_ms":1596.2989830062725,"ttft_p99_ms":1619.3840006820392,"tpot_p50_ms":41.213425333319485,"tpot_p95_ms":44.079397113550925,"tpot_p99_ms":44.55544799862557,"e2e_p50_ms":11705.644779955037,"e2e_p95_ms":11849.58514156169,"e2e_p99_ms":11856.07835849747,"itl_p50_ms":38.28029497526586,"itl_p95_ms":40.215927991084754,"itl_p99_ms":45.35360595909879,"validation_errors":[],"timestamp":"2026-07-30T17:08:19+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":69.820872,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":66.1121864799643,"request_tps":3.6301930518170575,"input_tps":3717.317685060667,"output_tps":929.3294212651667,"total_tps":4646.647106325834,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1387.4094450147822,"ttft_p95_ms":2114.0821015462275,"ttft_p99_ms":2317.8717700461857,"tpot_p50_ms":46.66030807854315,"tpot_p95_ms":48.186511425552524,"tpot_p99_ms":50.498127676980275,"e2e_p50_ms":13350.713515013922,"e2e_p95_ms":13697.923088440439,"e2e_p99_ms":14133.520027038176,"itl_p50_ms":42.404719977639616,"itl_p95_ms":45.02618601545691,"itl_p99_ms":52.773564737290144,"validation_errors":[],"timestamp":"2026-07-30T17:09:45+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":32.031753,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c48_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c48_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":72.01412146002986,"request_tps":4.443572920314112,"input_tps":4550.21867040165,"output_tps":1137.5546676004126,"total_tps":5687.773338002063,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1484.9153149989434,"ttft_p95_ms":2578.561547008576,"ttft_p99_ms":2662.701966339955,"tpot_p50_ms":50.29389217642008,"tpot_p95_ms":52.752956763735675,"tpot_p99_ms":54.903920572739565,"e2e_p50_ms":14357.403889996931,"e2e_p95_ms":15030.426644021645,"e2e_p99_ms":15164.506495348178,"itl_p50_ms":44.524859986267984,"itl_p95_ms":48.66255004890263,"itl_p99_ms":376.58238799776876,"validation_errors":[],"timestamp":"2026-07-30T17:11:17+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":22.405967,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c64_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c64_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":79.77297266002279,"request_tps":5.014229590073367,"input_tps":5134.571100235128,"output_tps":1283.642775058782,"total_tps":6418.21387529391,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1883.6029850062914,"ttft_p95_ms":3274.3605329538696,"ttft_p99_ms":3476.4080859720707,"tpot_p50_ms":55.11133866658543,"tpot_p95_ms":60.57697374711507,"tpot_p99_ms":61.06053318589579,"e2e_p50_ms":16047.942585020792,"e2e_p95_ms":16763.24705852312,"e2e_p99_ms":17200.16366496333,"itl_p50_ms":45.99126998800784,"itl_p95_ms":51.178909529699006,"itl_p99_ms":465.85821044165624,"validation_errors":[],"timestamp":"2026-07-30T17:12:59+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":12.842293,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c80_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c80_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":84.2057262599701,"request_tps":5.7003249222990595,"input_tps":5837.132720434237,"output_tps":1459.2831801085592,"total_tps":7296.415900542796,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":1902.5928599876352,"ttft_p95_ms":3337.0093190053008,"ttft_p99_ms":3930.1692080206703,"tpot_p50_ms":58.06618654899591,"tpot_p95_ms":64.92597843289794,"tpot_p99_ms":65.41815331706083,"e2e_p50_ms":17060.130960016977,"e2e_p95_ms":17851.459694560617,"e2e_p99_ms":18222.311295612017,"itl_p50_ms":48.08686999604106,"itl_p95_ms":54.12583006545901,"itl_p99_ms":676.5198479872198,"validation_errors":[],"timestamp":"2026-07-30T17:14:46+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":13.682966,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c96_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c96_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":560,"failed":0,"duration_s":87.49347653996665,"request_tps":6.400477180080898,"input_tps":6554.08863240284,"output_tps":1638.52215810071,"total_tps":8192.61079050355,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":2321.5200949343853,"ttft_p95_ms":3820.996321493277,"ttft_p99_ms":4667.278068445157,"tpot_p50_ms":59.087989529293466,"tpot_p95_ms":65.35422016081273,"tpot_p99_ms":67.14207445692688,"e2e_p50_ms":17487.774459994398,"e2e_p95_ms":18282.0225338859,"e2e_p99_ms":19068.67347399122,"itl_p50_ms":49.867029942106456,"itl_p95_ms":57.069560047239065,"itl_p99_ms":624.0572449867614,"validation_errors":[],"timestamp":"2026-07-30T17:16:36+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":112,"num_prompts":560,"warmup_requests":112,"attempt":1,"gain_pct":12.282673,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c112_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c112_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":640,"failed":0,"duration_s":109.21708890004084,"request_tps":5.859888836496545,"input_tps":6000.526168572462,"output_tps":1500.1315421431154,"total_tps":7500.657710715577,"mean_input_tokens":1024.0,"mean_output_tokens":256.0,"ttft_p50_ms":2423.238629999105,"ttft_p95_ms":4808.490526990499,"ttft_p99_ms":16888.046733541414,"tpot_p50_ms":69.38380015667018,"tpot_p95_ms":76.24058201596343,"tpot_p99_ms":77.40038021974374,"e2e_p50_ms":20295.52326502744,"e2e_p95_ms":22614.711468451424,"e2e_p99_ms":36084.81017707498,"itl_p50_ms":51.12898501101881,"itl_p95_ms":61.132238002028295,"itl_p99_ms":743.1606936710887,"validation_errors":[],"timestamp":"2026-07-30T17:18:52+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"concurrency":128,"num_prompts":640,"warmup_requests":128,"attempt":1,"gain_pct":-8.446063,"plateau_streak":1,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c128_i1024_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c128_i1024_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":94.88916621997487,"request_tps":0.8430888708047201,"input_tps":863.3230037040333,"output_tps":431.66150185201667,"total_tps":1294.98450555605,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":841.7698499979451,"ttft_p95_ms":863.01319153863,"ttft_p99_ms":864.28612874588,"tpot_p50_ms":35.76848684940415,"tpot_p95_ms":36.745883743624965,"tpot_p99_ms":36.77229937240017,"e2e_p50_ms":18993.666849972215,"e2e_p95_ms":19125.605078943776,"e2e_p99_ms":19126.32367210812,"itl_p50_ms":35.487710032612085,"itl_p95_ms":36.30406806478277,"itl_p99_ms":36.85290674678981,"validation_errors":[],"timestamp":"2026-07-30T17:20:45+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":106.17220417002682,"request_tps":1.5069857619586762,"input_tps":1543.1534202456844,"output_tps":771.5767101228422,"total_tps":2314.7301303685267,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1118.9615999464877,"ttft_p95_ms":1515.831251454074,"ttft_p99_ms":1518.9875466842204,"tpot_p50_ms":39.204914569344425,"tpot_p95_ms":40.5413200520503,"tpot_p99_ms":41.53470395929126,"e2e_p50_ms":21113.11224498786,"e2e_p95_ms":21713.705797493458,"e2e_p99_ms":21717.30715890415,"itl_p50_ms":38.38119003921747,"itl_p95_ms":40.410694014281034,"itl_p99_ms":42.12473154300823,"validation_errors":[],"timestamp":"2026-07-30T17:22:50+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":78.745778,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":121.35073115990963,"request_tps":1.9777383927233252,"input_tps":2025.204114148685,"output_tps":1012.6020570743425,"total_tps":3037.8061712230274,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1504.315625061281,"ttft_p95_ms":2124.850040522868,"ttft_p99_ms":2496.482050899649,"tpot_p50_ms":44.478661634156445,"tpot_p95_ms":46.627289500978726,"tpot_p99_ms":47.22175794238281,"e2e_p50_ms":24315.77692006249,"e2e_p95_ms":24605.28770747478,"e2e_p99_ms":25260.78345368267,"itl_p50_ms":42.67951496876776,"itl_p95_ms":45.783932553604245,"itl_p99_ms":48.167404497508,"validation_errors":[],"timestamp":"2026-07-30T17:25:12+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":31.238028,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c48_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c48_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":130.1659644000465,"request_tps":2.4583999471361477,"input_tps":2517.4015458674153,"output_tps":1258.7007729337076,"total_tps":3776.102318801123,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1771.9492749893107,"ttft_p95_ms":2583.16012902651,"ttft_p99_ms":2918.0812802468436,"tpot_p50_ms":47.43789362041334,"tpot_p95_ms":49.73561719142703,"tpot_p99_ms":50.642861152391745,"e2e_p50_ms":26055.440124997403,"e2e_p95_ms":26752.471042890105,"e2e_p99_ms":26823.372467247536,"itl_p50_ms":44.567755016032606,"itl_p95_ms":48.67709648679011,"itl_p99_ms":52.5578124553431,"validation_errors":[],"timestamp":"2026-07-30T17:27:43+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":24.303596,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c64_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c64_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":137.95168298995122,"request_tps":2.899565930117265,"input_tps":2969.1555124400793,"output_tps":1484.5777562200396,"total_tps":4453.7332686601185,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1797.9747799690813,"ttft_p95_ms":3203.8436605012976,"ttft_p99_ms":3212.2730390599463,"tpot_p50_ms":50.61857036208741,"tpot_p95_ms":52.456841946963614,"tpot_p99_ms":53.418321622560185,"e2e_p50_ms":27802.067935001105,"e2e_p95_ms":28447.153918043477,"e2e_p99_ms":28730.936635655817,"itl_p50_ms":46.31271498510614,"itl_p95_ms":50.98592001013458,"itl_p99_ms":57.67704304889784,"validation_errors":[],"timestamp":"2026-07-30T17:30:23+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":17.945249,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c80_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c80_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":148.7925869700266,"request_tps":3.225967165264041,"input_tps":3303.390377230378,"output_tps":1651.695188615189,"total_tps":4955.085565845567,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":1878.736850048881,"ttft_p95_ms":3334.065314568579,"ttft_p99_ms":3861.431198110804,"tpot_p50_ms":54.4160427983555,"tpot_p95_ms":57.571814833548366,"tpot_p99_ms":58.32551064007321,"e2e_p50_ms":29724.27341499133,"e2e_p95_ms":30873.609018040584,"e2e_p99_ms":30991.819918294204,"itl_p50_ms":48.407569993287325,"itl_p95_ms":55.205402034334824,"itl_p99_ms":69.89601758308709,"validation_errors":[],"timestamp":"2026-07-30T17:33:15+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":11.2569,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c96_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c96_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":560,"failed":0,"duration_s":153.98801110009663,"request_tps":3.6366467493107884,"input_tps":3723.9262712942473,"output_tps":1861.9631356471236,"total_tps":5585.889406941371,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":2266.5420949924737,"ttft_p95_ms":3981.750191492029,"ttft_p99_ms":4466.720784116769,"tpot_p50_ms":55.80332691778033,"tpot_p95_ms":59.34395213001326,"tpot_p99_ms":59.90514719491519,"e2e_p50_ms":30812.476855004206,"e2e_p95_ms":31652.516745083263,"e2e_p99_ms":31666.862492602086,"itl_p50_ms":50.01038999762386,"itl_p95_ms":58.51889199111611,"itl_p99_ms":76.19726077653466,"validation_errors":[],"timestamp":"2026-07-30T17:36:13+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":112,"num_prompts":560,"warmup_requests":112,"attempt":1,"gain_pct":12.730433,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c112_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c112_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":640,"failed":0,"duration_s":186.84227379004005,"request_tps":3.425349023097343,"input_tps":3507.557399651679,"output_tps":1753.7786998258396,"total_tps":5261.3360994775185,"mean_input_tokens":1024.0,"mean_output_tokens":512.0,"ttft_p50_ms":2578.5184500273317,"ttft_p95_ms":4728.05582403671,"ttft_p99_ms":30009.161690098703,"tpot_p50_ms":60.47960188854468,"tpot_p95_ms":65.09254957341378,"tpot_p99_ms":66.1020098991693,"e2e_p50_ms":33611.5337399533,"e2e_p95_ms":35483.3014129661,"e2e_p99_ms":63459.399813563796,"itl_p50_ms":51.10505491029471,"itl_p95_ms":59.96067305095493,"itl_p99_ms":483.7041217600926,"validation_errors":[],"timestamp":"2026-07-30T17:39:47+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"concurrency":128,"num_prompts":640,"warmup_requests":128,"attempt":1,"gain_pct":-5.810235,"plateau_streak":1,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c128_i1024_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c128_i1024_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":185.96836816996802,"request_tps":0.4301806849586541,"input_tps":440.5050213976618,"output_tps":440.5050213976618,"total_tps":881.0100427953236,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":835.4597299476154,"ttft_p95_ms":866.9820990588049,"ttft_p99_ms":868.1455627235118,"tpot_p50_ms":35.62154360220222,"tpot_p95_ms":36.23448730107735,"tpot_p99_ms":36.37256872273512,"e2e_p50_ms":37271.65017504012,"e2e_p95_ms":37550.97916450467,"e2e_p99_ms":37551.42987567,"itl_p50_ms":35.55400995537639,"itl_p95_ms":36.30549005465582,"itl_p99_ms":36.85658688191324,"validation_errors":[],"timestamp":"2026-07-30T17:43:11+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":205.0389171600109,"request_tps":0.780339665348199,"input_tps":799.0678173165558,"output_tps":799.0678173165558,"total_tps":1598.1356346331115,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1119.4225249928422,"ttft_p95_ms":1517.148989456473,"ttft_p99_ms":1519.4997572153807,"tpot_p50_ms":38.7213549413172,"tpot_p95_ms":40.1605721553997,"tpot_p99_ms":40.767122635966984,"e2e_p50_ms":40725.20693502156,"e2e_p95_ms":42238.10811257573,"e2e_p99_ms":42240.892483928474,"itl_p50_ms":38.39373495429754,"itl_p95_ms":40.44664147077128,"itl_p99_ms":41.618181908270344,"validation_errors":[],"timestamp":"2026-07-30T17:46:55+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":81.398118,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":229.85885027004406,"request_tps":1.0441190309533084,"input_tps":1069.1778876961878,"output_tps":1069.1778876961878,"total_tps":2138.3557753923756,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1494.4963099551387,"ttft_p95_ms":2308.5877454897854,"ttft_p99_ms":2384.6869855804835,"tpot_p50_ms":43.76367966276671,"tpot_p95_ms":44.90133274531739,"tpot_p99_ms":45.358635103223044,"e2e_p50_ms":46404.42610002356,"e2e_p95_ms":47102.79659850639,"e2e_p99_ms":47734.11575757433,"itl_p50_ms":42.48265002388507,"itl_p95_ms":45.53857804276049,"itl_p99_ms":47.40080343559383,"validation_errors":[],"timestamp":"2026-07-30T17:51:06+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":33.803147,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c48_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c48_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":246.45591532008257,"request_tps":1.298406652501737,"input_tps":1329.5684121617787,"output_tps":1329.5684121617787,"total_tps":2659.1368243235574,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1806.3082000007853,"ttft_p95_ms":2526.034537469968,"ttft_p99_ms":2724.466053863289,"tpot_p50_ms":46.720948084114156,"tpot_p95_ms":47.53952720725408,"tpot_p99_ms":47.916805507852594,"e2e_p50_ms":49166.941564995795,"e2e_p95_ms":49997.85971805687,"e2e_p99_ms":50342.97233300749,"itl_p50_ms":44.845580006949604,"itl_p95_ms":48.89546998310834,"itl_p99_ms":51.74153596162796,"validation_errors":[],"timestamp":"2026-07-30T17:55:34+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":24.354275,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c64_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c64_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":257.9188281899551,"request_tps":1.5508755324578447,"input_tps":1588.096545236833,"output_tps":1588.096545236833,"total_tps":3176.193090473666,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1883.7554450146854,"ttft_p95_ms":3039.602126972738,"ttft_p99_ms":3297.8107565816026,"tpot_p50_ms":48.548012834808404,"tpot_p95_ms":49.961682404683934,"tpot_p99_ms":50.438205968220565,"e2e_p50_ms":51805.96031504683,"e2e_p95_ms":52447.22246610908,"e2e_p99_ms":52864.42535214126,"itl_p50_ms":46.38780001550913,"itl_p95_ms":51.564824534580104,"itl_p99_ms":55.45575613505206,"validation_errors":[],"timestamp":"2026-07-30T18:00:14+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":19.444515,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c80_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c80_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":269.40732045006007,"request_tps":1.7816887796446401,"input_tps":1824.4493103561115,"output_tps":1824.4493103561115,"total_tps":3648.898620712223,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":2175.7572600618005,"ttft_p95_ms":3243.282936979085,"ttft_p99_ms":3828.245573324384,"tpot_p50_ms":50.28743643211456,"tpot_p95_ms":52.897247548416736,"tpot_p99_ms":53.51440947601059,"e2e_p50_ms":53563.73057496967,"e2e_p95_ms":55708.33695547772,"e2e_p99_ms":55778.71477488428,"itl_p50_ms":48.018985020462424,"itl_p95_ms":54.06370699056424,"itl_p99_ms":58.4948574320879,"validation_errors":[],"timestamp":"2026-07-30T18:05:08+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":14.882771,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c96_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c96_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":560,"failed":0,"duration_s":321.8454230700154,"request_tps":1.7399657098065229,"input_tps":1781.7248868418794,"output_tps":1781.7248868418794,"total_tps":3563.449773683759,"mean_input_tokens":1024.0,"mean_output_tokens":1024.0,"ttft_p50_ms":2438.694660027977,"ttft_p95_ms":4060.113952012034,"ttft_p99_ms":25135.310975597968,"tpot_p50_ms":54.4445669012898,"tpot_p95_ms":56.425600073338764,"tpot_p99_ms":57.01093179217806,"e2e_p50_ms":58199.468194972724,"e2e_p95_ms":59436.64607596002,"e2e_p99_ms":81725.48296127534,"itl_p50_ms":50.29838997870684,"itl_p95_ms":57.491641503293046,"itl_p99_ms":63.27032491331916,"validation_errors":[],"timestamp":"2026-07-30T18:10:54+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":1024,"concurrency":112,"num_prompts":560,"warmup_requests":112,"attempt":1,"gain_pct":-2.341771,"plateau_streak":1,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c112_i1024_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c112_i1024_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":363.41527339001186,"request_tps":0.22013384097411115,"input_tps":225.41705315748982,"output_tps":450.83410631497964,"total_tps":676.2511594724695,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":827.0725350012071,"ttft_p95_ms":945.8547049667686,"ttft_p99_ms":948.7971463194117,"tpot_p50_ms":35.13168357107943,"tpot_p95_ms":35.464742614063134,"tpot_p99_ms":35.46483627016907,"e2e_p50_ms":72655.71826003725,"e2e_p95_ms":72971.9756670529,"e2e_p99_ms":72972.67414886621,"itl_p50_ms":35.089110024273396,"itl_p95_ms":35.87696992326528,"itl_p99_ms":36.51140281930566,"validation_errors":[],"timestamp":"2026-07-30T18:17:17+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":403.78660704998765,"request_tps":0.3962489027779776,"input_tps":405.75887644464905,"output_tps":811.5177528892981,"total_tps":1217.276629333947,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1129.151820030529,"ttft_p95_ms":1640.8297515299637,"ttft_p99_ms":1646.1413000395987,"tpot_p50_ms":38.62735733514289,"tpot_p95_ms":40.13289778602423,"tpot_p99_ms":40.46295216693939,"e2e_p50_ms":80185.28453999897,"e2e_p95_ms":83340.30482058295,"e2e_p99_ms":83342.20448904554,"itl_p50_ms":38.42246998101473,"itl_p95_ms":40.4511250089854,"itl_p99_ms":41.73968599643558,"validation_errors":[],"timestamp":"2026-07-30T18:24:21+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":80.003629,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":444.5343426699983,"request_tps":0.5398907957448068,"input_tps":552.8481748426822,"output_tps":1105.6963496853643,"total_tps":1658.5445245280466,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1348.5136150266044,"ttft_p95_ms":1989.7994124621619,"ttft_p99_ms":2228.0535577668343,"tpot_p50_ms":42.33409423059427,"tpot_p95_ms":43.99942943088016,"tpot_p99_ms":44.2166463750089,"e2e_p50_ms":88055.02112500835,"e2e_p95_ms":91187.3767255398,"e2e_p99_ms":91937.03203074634,"itl_p50_ms":42.151225032284856,"itl_p95_ms":45.12471946654841,"itl_p99_ms":46.93031929433346,"validation_errors":[],"timestamp":"2026-07-30T18:32:07+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":36.25042,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c48_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c48_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":474.65229592006654,"request_tps":0.6741777144039968,"input_tps":690.3579795496927,"output_tps":1380.7159590993854,"total_tps":2071.073938649078,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1810.9590049134567,"ttft_p95_ms":2602.070517092943,"ttft_p99_ms":2612.936951976735,"tpot_p50_ms":45.542126687873264,"tpot_p95_ms":46.18780581926873,"tpot_p99_ms":46.25326385617262,"e2e_p50_ms":94893.09813996078,"e2e_p95_ms":95882.09593704669,"e2e_p99_ms":95887.642695104,"itl_p50_ms":44.821809977293015,"itl_p95_ms":48.61332306172698,"itl_p99_ms":50.804860296193496,"validation_errors":[],"timestamp":"2026-07-30T18:40:26+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":24.872978,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c64_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c64_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":495.80488350009546,"request_tps":0.8067689797168426,"input_tps":826.1314352300468,"output_tps":1652.2628704600936,"total_tps":2478.3943056901403,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":2008.8086449541152,"ttft_p95_ms":3194.740683055716,"ttft_p99_ms":3305.9905665984843,"tpot_p50_ms":47.69460696384218,"tpot_p95_ms":48.053248701281014,"tpot_p99_ms":48.49505417270274,"e2e_p50_ms":99784.07042002073,"e2e_p95_ms":99975.26022213278,"e2e_p99_ms":100709.74791280227,"itl_p50_ms":46.37155000818893,"itl_p95_ms":52.24911097320728,"itl_p99_ms":56.048801887081936,"validation_errors":[],"timestamp":"2026-07-30T18:49:06+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":19.667109,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c80_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c80_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":522.8778748300392,"request_tps":0.9179963871219898,"input_tps":940.0283004129176,"output_tps":1880.0566008258352,"total_tps":2820.0849012387525,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":2228.2336550415494,"ttft_p95_ms":3267.843960499158,"ttft_p99_ms":3852.2644429909997,"tpot_p50_ms":50.15215750857159,"tpot_p95_ms":50.822256714926034,"tpot_p99_ms":51.00306731597275,"e2e_p50_ms":104947.31521001086,"e2e_p95_ms":105654.9767084536,"e2e_p99_ms":105805.0579646742,"itl_p50_ms":48.63614001078531,"itl_p95_ms":55.2436864527408,"itl_p99_ms":60.178559111664086,"validation_errors":[],"timestamp":"2026-07-30T18:58:15+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":13.786773,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c96_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c96_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":560,"failed":0,"duration_s":542.6957234999863,"request_tps":1.0318857800986783,"input_tps":1056.6510388210465,"output_tps":2113.302077642093,"total_tps":3169.95311646314,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":2369.367200008128,"ttft_p95_ms":3973.681426962139,"ttft_p99_ms":4461.373646893771,"tpot_p50_ms":51.95132623838705,"tpot_p95_ms":52.855420082400805,"tpot_p99_ms":53.18311556157782,"e2e_p50_ms":108480.08382995613,"e2e_p95_ms":110333.09909289237,"e2e_p99_ms":110505.0883439905,"itl_p50_ms":50.47688994091004,"itl_p95_ms":57.28942004498094,"itl_p99_ms":63.277881010435564,"validation_errors":[],"timestamp":"2026-07-30T19:07:46+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"concurrency":112,"num_prompts":560,"warmup_requests":112,"attempt":1,"gain_pct":12.406301,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c112_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c112_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":640,"failed":0,"duration_s":636.1103709900053,"request_tps":1.0061147077415844,"input_tps":1030.2614607273824,"output_tps":2060.522921454765,"total_tps":3090.784382182147,"mean_input_tokens":1024.0,"mean_output_tokens":2048.0,"ttft_p50_ms":2392.5693249912,"ttft_p95_ms":4778.494630515343,"ttft_p99_ms":109034.04671798577,"tpot_p50_ms":53.65953891302409,"tpot_p95_ms":54.59831643991903,"tpot_p99_ms":54.820281263657186,"e2e_p50_ms":112342.7104599541,"e2e_p95_ms":114418.14921348704,"e2e_p99_ms":221119.1702728963,"itl_p50_ms":51.49906000588089,"itl_p95_ms":58.824372419621795,"itl_p99_ms":65.96712442464197,"validation_errors":[],"timestamp":"2026-07-30T19:18:51+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"concurrency":128,"num_prompts":640,"warmup_requests":128,"attempt":1,"gain_pct":-2.497473,"plateau_streak":1,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c128_i1024_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c128_i1024_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":732.0481155500747,"request_tps":0.10928243417427076,"input_tps":111.90521259445326,"output_tps":447.62085037781304,"total_tps":559.5260629722663,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":837.1147249708883,"ttft_p95_ms":868.7463235168252,"ttft_p99_ms":870.2030535240192,"tpot_p50_ms":35.49683059219462,"tpot_p95_ms":36.05613480035928,"tpot_p99_ms":36.05642720079902,"e2e_p50_ms":146191.03480002377,"e2e_p95_ms":147981.3069489319,"e2e_p99_ms":147983.25984179508,"itl_p50_ms":35.5558599694632,"itl_p95_ms":36.43596548354253,"itl_p99_ms":37.39740308839828,"validation_errors":[],"timestamp":"2026-07-30T19:31:22+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":4096,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":793.3032704100478,"request_tps":0.20168831513488927,"input_tps":206.52883469812662,"output_tps":826.1153387925065,"total_tps":1032.644173490633,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1168.777129962109,"ttft_p95_ms":1571.894304495072,"ttft_p99_ms":1576.716513504507,"tpot_p50_ms":38.45428101464274,"tpot_p95_ms":38.65967455973458,"tpot_p99_ms":38.78837207866542,"e2e_p50_ms":158559.46133995894,"e2e_p95_ms":159337.2602981166,"e2e_p99_ms":159340.20692107966,"itl_p50_ms":38.2956099929288,"itl_p95_ms":39.91584497271106,"itl_p99_ms":41.182119038421654,"validation_errors":[],"timestamp":"2026-07-30T19:44:57+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":4096,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":84.556939,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":240,"failed":0,"duration_s":883.6818267900962,"request_tps":0.2715909648971518,"input_tps":278.10914805468343,"output_tps":1112.4365922187337,"total_tps":1390.545740273417,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1176.264879934024,"ttft_p95_ms":1864.8611619544681,"ttft_p99_ms":2231.3991517049726,"tpot_p50_ms":43.311135512827846,"tpot_p95_ms":43.60382752868566,"tpot_p99_ms":43.69422689457571,"e2e_p50_ms":178554.33170491597,"e2e_p95_ms":179641.5094519616,"e2e_p99_ms":180282.09005660727,"itl_p50_ms":42.71904000779614,"itl_p95_ms":45.72233099024742,"itl_p99_ms":47.554475523065776,"validation_errors":[],"timestamp":"2026-07-30T20:00:05+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":4096,"concurrency":48,"num_prompts":240,"warmup_requests":48,"attempt":1,"gain_pct":34.65875,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c48_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c48_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":320,"failed":0,"duration_s":934.4282526499592,"request_tps":0.34245539889500043,"input_tps":350.67432846848044,"output_tps":1402.6973138739218,"total_tps":1753.371642342402,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1498.0225199833512,"ttft_p95_ms":2508.4393200755585,"ttft_p99_ms":2712.4019101948943,"tpot_p50_ms":45.23317869718531,"tpot_p95_ms":45.74096175384286,"tpot_p99_ms":45.74302739910958,"e2e_p50_ms":186700.77540999046,"e2e_p95_ms":189092.14691446393,"e2e_p99_ms":189114.6837966342,"itl_p50_ms":44.85946008935571,"itl_p95_ms":48.66276695393026,"itl_p99_ms":51.60436860751351,"validation_errors":[],"timestamp":"2026-07-30T20:16:06+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":4096,"concurrency":64,"num_prompts":320,"warmup_requests":64,"attempt":1,"gain_pct":26.092339,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c64_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c64_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":400,"failed":0,"duration_s":977.936277610017,"request_tps":0.4090246053429594,"input_tps":418.84119587119045,"output_tps":1675.3647834847618,"total_tps":2094.2059793559524,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":2078.8239800021984,"ttft_p95_ms":3195.0550694949925,"ttft_p99_ms":3442.5215482653584,"tpot_p50_ms":47.25045570574919,"tpot_p95_ms":48.6153246151293,"tpot_p99_ms":48.62069525336449,"e2e_p50_ms":195428.9762399858,"e2e_p95_ms":201012.691797124,"e2e_p99_ms":201320.66702918848,"itl_p50_ms":46.70404002536088,"itl_p95_ms":51.87546897213906,"itl_p99_ms":55.87511052144691,"validation_errors":[],"timestamp":"2026-07-30T20:32:54+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":4096,"concurrency":80,"num_prompts":400,"warmup_requests":80,"attempt":1,"gain_pct":19.438796,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c80_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c80_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":480,"failed":0,"duration_s":1019.9992440899368,"request_tps":0.4705885840418101,"input_tps":481.88271005881353,"output_tps":1927.5308402352541,"total_tps":2409.4135502940676,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":2054.306434991304,"ttft_p95_ms":3258.2020850502886,"ttft_p99_ms":3843.9868689107243,"tpot_p50_ms":49.412939338208545,"tpot_p95_ms":50.0014282725052,"tpot_p99_ms":50.03059984656262,"e2e_p50_ms":204575.93864999944,"e2e_p95_ms":205527.76334254304,"e2e_p99_ms":205960.25738344062,"itl_p50_ms":48.70361997745931,"itl_p95_ms":55.46393000986427,"itl_p99_ms":60.13232319382951,"validation_errors":[],"timestamp":"2026-07-30T20:50:25+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":4096,"concurrency":96,"num_prompts":480,"warmup_requests":96,"attempt":1,"gain_pct":15.051412,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c96_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c96_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":560,"failed":0,"duration_s":1056.8817638199544,"request_tps":0.5298605947896733,"input_tps":542.5772490646254,"output_tps":2170.308996258502,"total_tps":2712.8862453231272,"mean_input_tokens":1024.0,"mean_output_tokens":4096.0,"ttft_p50_ms":2420.0328049482778,"ttft_p95_ms":4056.7046150390524,"ttft_p99_ms":4544.868242548546,"tpot_p50_ms":51.026092341891236,"tpot_p95_ms":52.11853481525975,"tpot_p99_ms":52.12119694534969,"e2e_p50_ms":212005.33948501106,"e2e_p95_ms":214710.29747353168,"e2e_p99_ms":214720.86298826733,"itl_p50_ms":50.371260032989085,"itl_p95_ms":57.89080495014787,"itl_p99_ms":63.87767498381436,"validation_errors":[],"timestamp":"2026-07-30T21:08:34+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":4096,"concurrency":112,"num_prompts":560,"warmup_requests":112,"attempt":1,"gain_pct":12.595293,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c112_i1024_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c112_i1024_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":39.336056040017866,"request_tps":2.0337575256302607,"input_tps":8330.270824981548,"output_tps":260.32096328067337,"total_tps":8590.591788262222,"mean_input_tokens":4096.0,"mean_output_tokens":128.0,"ttft_p50_ms":2187.1839000377804,"ttft_p95_ms":3003.6019804829257,"ttft_p99_ms":3339.2457283183467,"tpot_p50_ms":44.93544370075321,"tpot_p95_ms":54.5679621451615,"tpot_p99_ms":55.321238977957414,"e2e_p50_ms":7871.6948900255375,"e2e_p95_ms":7996.729450981364,"e2e_p99_ms":8632.437928903382,"itl_p50_ms":35.119950014632195,"itl_p95_ms":36.24007592443377,"itl_p99_ms":634.2014464666136,"validation_errors":[],"timestamp":"2026-07-30T21:09:34+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i4096_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i4096_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":56.971435270039365,"request_tps":2.808424945617303,"input_tps":11503.308577248474,"output_tps":359.4783930390148,"total_tps":11862.786970287489,"mean_input_tokens":4096.0,"mean_output_tokens":128.0,"ttft_p50_ms":2662.593584973365,"ttft_p95_ms":4998.433482984546,"ttft_p99_ms":5912.604085912462,"tpot_p50_ms":65.37763149648039,"tpot_p95_ms":82.30828790075505,"tpot_p99_ms":89.18234636074877,"e2e_p50_ms":11208.880015008617,"e2e_p95_ms":12666.432679549325,"e2e_p99_ms":13575.299556087703,"itl_p50_ms":39.214359945617616,"itl_p95_ms":42.801773943938294,"itl_p99_ms":843.2107960432768,"validation_errors":[],"timestamp":"2026-07-30T21:10:55+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":128,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":38.090451,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i4096_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i4096_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":61.03357877000235,"request_tps":1.3107538770005658,"input_tps":5368.847880194317,"output_tps":335.55299251214484,"total_tps":5704.400872706462,"mean_input_tokens":4096.0,"mean_output_tokens":256.0,"ttft_p50_ms":2444.1770049743354,"ttft_p95_ms":2700.8904109999985,"ttft_p99_ms":3334.2654269270133,"tpot_p50_ms":38.218388902069094,"tpot_p95_ms":44.18059531170144,"tpot_p99_ms":44.29181505711384,"e2e_p50_ms":12212.612530041952,"e2e_p95_ms":12286.362495570209,"e2e_p99_ms":13116.968461566139,"itl_p50_ms":35.008389968425035,"itl_p95_ms":35.80531192710623,"itl_p99_ms":37.73003819398581,"validation_errors":[],"timestamp":"2026-07-30T21:12:16+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i4096_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i4096_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":80.90403363003861,"request_tps":1.9776517043841693,"input_tps":8100.461381157557,"output_tps":506.27883632234733,"total_tps":8606.740217479904,"mean_input_tokens":4096.0,"mean_output_tokens":256.0,"ttft_p50_ms":2995.807719998993,"ttft_p95_ms":5009.970198449445,"ttft_p99_ms":5934.9386612314265,"tpot_p50_ms":51.55588864695792,"tpot_p95_ms":58.88034104928831,"tpot_p99_ms":61.21395612803891,"e2e_p50_ms":16199.68890491873,"e2e_p95_ms":17618.272747506853,"e2e_p99_ms":18533.91959931352,"itl_p50_ms":39.24104000907391,"itl_p95_ms":41.46764747565612,"itl_p99_ms":816.3382475904655,"validation_errors":[],"timestamp":"2026-07-30T21:14:00+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":256,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":50.878951,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i4096_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i4096_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":104.90250632003881,"request_tps":0.7626128565120673,"input_tps":3123.662260273428,"output_tps":390.4577825341785,"total_tps":3514.1200428076063,"mean_input_tokens":4096.0,"mean_output_tokens":512.0,"ttft_p50_ms":1674.9178700265475,"ttft_p95_ms":2772.2888349788263,"ttft_p99_ms":2774.0980246162508,"tpot_p50_ms":37.6895719177979,"tpot_p95_ms":40.21890570731731,"tpot_p99_ms":41.49467472219157,"e2e_p50_ms":20845.47930501867,"e2e_p95_ms":21682.41444664891,"e2e_p99_ms":21684.941132486565,"itl_p50_ms":35.22750502452254,"itl_p95_ms":36.113520024809986,"itl_p99_ms":37.345657092519005,"validation_errors":[],"timestamp":"2026-07-30T21:16:06+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i4096_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i4096_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":133.93535621999763,"request_tps":1.1946061481868124,"input_tps":4893.106782973184,"output_tps":611.638347871648,"total_tps":5504.745130844832,"mean_input_tokens":4096.0,"mean_output_tokens":512.0,"ttft_p50_ms":2622.418349958025,"ttft_p95_ms":4579.945103963837,"ttft_p99_ms":5591.665025679393,"tpot_p50_ms":47.24100883560321,"tpot_p95_ms":51.94528996465408,"tpot_p99_ms":53.15489596806406,"e2e_p50_ms":26854.97258498799,"e2e_p95_ms":28559.196461905954,"e2e_p99_ms":28708.27611604589,"itl_p50_ms":39.19039003085345,"itl_p95_ms":41.417163936421275,"itl_p99_ms":428.0206580180674,"validation_errors":[],"timestamp":"2026-07-30T21:18:46+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":512,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":56.646474,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i4096_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i4096_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":196.0227936200099,"request_tps":0.4081158038951325,"input_tps":1671.6423327544628,"output_tps":417.9105831886157,"total_tps":2089.5529159430785,"mean_input_tokens":4096.0,"mean_output_tokens":1024.0,"ttft_p50_ms":2444.5193100254983,"ttft_p95_ms":2706.0936175170336,"ttft_p99_ms":3341.2637254328,"tpot_p50_ms":35.9830183724332,"tpot_p95_ms":37.401647853834845,"tpot_p99_ms":37.512727336326925,"e2e_p50_ms":39174.00777002331,"e2e_p95_ms":39396.778271964286,"e2e_p99_ms":40235.04633786273,"itl_p50_ms":35.10927991010249,"itl_p95_ms":35.856240964494646,"itl_p99_ms":36.543497659731656,"validation_errors":[],"timestamp":"2026-07-30T21:22:22+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i4096_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i4096_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":230.33209268003702,"request_tps":0.6946491830049146,"input_tps":2845.2830535881303,"output_tps":711.3207633970326,"total_tps":3556.6038169851627,"mean_input_tokens":4096.0,"mean_output_tokens":1024.0,"ttft_p50_ms":2651.720354973804,"ttft_p95_ms":5000.882567971711,"ttft_p99_ms":5921.433662185445,"tpot_p50_ms":42.00083971653318,"tpot_p95_ms":44.164303625683004,"tpot_p99_ms":44.93874814821305,"e2e_p50_ms":45643.312564934604,"e2e_p95_ms":47716.07479954836,"e2e_p99_ms":48201.21978895506,"itl_p50_ms":38.68406999390572,"itl_p95_ms":40.80521653522737,"itl_p99_ms":42.75008956319654,"validation_errors":[],"timestamp":"2026-07-30T21:26:36+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":1024,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":70.208842,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i4096_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i4096_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":376.8889283799799,"request_tps":0.21226412870198164,"input_tps":869.4338711633168,"output_tps":434.7169355816584,"total_tps":1304.1508067449752,"mean_input_tokens":4096.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1849.1058700019494,"ttft_p95_ms":2742.2154580359343,"ttft_p99_ms":3425.2684384712484,"tpot_p50_ms":35.97076491208321,"tpot_p95_ms":36.864100734974066,"tpot_p99_ms":36.907939242011274,"e2e_p50_ms":75179.16733497987,"e2e_p95_ms":76372.81816203031,"e2e_p99_ms":77070.7779555954,"itl_p50_ms":35.299079958349466,"itl_p95_ms":36.13904002122581,"itl_p99_ms":36.69700000900776,"validation_errors":[],"timestamp":"2026-07-30T21:33:14+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i4096_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i4096_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":431.61511022003833,"request_tps":0.3707006455784916,"input_tps":1518.3898442895015,"output_tps":759.1949221447508,"total_tps":2277.5847664342523,"mean_input_tokens":4096.0,"mean_output_tokens":2048.0,"ttft_p50_ms":2822.2276300075464,"ttft_p95_ms":4870.664548972854,"ttft_p99_ms":5430.5884197191335,"tpot_p50_ms":41.14077140937246,"tpot_p95_ms":42.366088142396286,"tpot_p99_ms":42.58499559558724,"e2e_p50_ms":87409.90129997954,"e2e_p95_ms":88342.24459398536,"e2e_p99_ms":88660.3383070929,"itl_p50_ms":39.23968004528433,"itl_p95_ms":41.080660035368055,"itl_p99_ms":42.509977030567825,"validation_errors":[],"timestamp":"2026-07-30T21:40:50+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":2048,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":74.641211,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i4096_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i4096_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":738.473562899977,"request_tps":0.10833156936023673,"input_tps":443.72610809952965,"output_tps":443.72610809952965,"total_tps":887.4522161990593,"mean_input_tokens":4096.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1835.0517200306058,"ttft_p95_ms":2738.382391585037,"ttft_p99_ms":3442.7284442516975,"tpot_p50_ms":35.58650593528062,"tpot_p95_ms":36.35826872404146,"tpot_p99_ms":36.358388344599476,"e2e_p50_ms":146571.5758500155,"e2e_p95_ms":149721.1351894657,"e2e_p99_ms":150415.76542337425,"itl_p50_ms":35.32536991406232,"itl_p95_ms":36.19082202203572,"itl_p99_ms":36.724539604038,"validation_errors":[],"timestamp":"2026-07-30T21:53:30+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":4096,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i4096_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i4096_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":160,"failed":0,"duration_s":832.8275065999478,"request_tps":0.1921166132626989,"input_tps":786.9096479240147,"output_tps":786.9096479240147,"total_tps":1573.8192958480295,"mean_input_tokens":4096.0,"mean_output_tokens":4096.0,"ttft_p50_ms":2637.4701249878854,"ttft_p95_ms":5024.245622445596,"ttft_p99_ms":5937.885705957888,"tpot_p50_ms":40.216948840055295,"tpot_p95_ms":40.8455547902185,"tpot_p99_ms":41.032989944985495,"e2e_p50_ms":167201.79951994214,"e2e_p95_ms":168980.74854701987,"e2e_p99_ms":169755.26779308682,"itl_p50_ms":39.25154998432845,"itl_p95_ms":40.93364652362652,"itl_p99_ms":42.26434658048675,"validation_errors":[],"timestamp":"2026-07-30T22:07:49+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":4096,"concurrency":32,"num_prompts":160,"warmup_requests":32,"attempt":1,"gain_pct":77.3413,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c32_i4096_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c32_i4096_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":55.12346043006983,"request_tps":1.451287698120636,"input_tps":11888.94882300425,"output_tps":185.7648253594414,"total_tps":12074.713648363691,"mean_input_tokens":8192.0,"mean_output_tokens":128.0,"ttft_p50_ms":2813.928855000995,"ttft_p95_ms":5019.744277483551,"ttft_p99_ms":5569.114368426381,"tpot_p50_ms":64.09184838549828,"tpot_p95_ms":82.15539147471247,"tpot_p99_ms":83.08371602962761,"e2e_p50_ms":11175.221070006955,"e2e_p95_ms":12169.121659972006,"e2e_p99_ms":12198.23964005569,"itl_p50_ms":35.72136501315981,"itl_p95_ms":37.33663706225343,"itl_p99_ms":898.6520615196787,"validation_errors":[],"timestamp":"2026-07-30T22:09:07+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i8192_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i8192_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":41.45226079004351,"request_tps":0.9649654623809486,"input_tps":7904.997067824731,"output_tps":123.51557918476142,"total_tps":8028.512647009492,"mean_input_tokens":8192.0,"mean_output_tokens":128.0,"ttft_p50_ms":2219.2530449829064,"ttft_p95_ms":4199.5956530503445,"ttft_p99_ms":4918.302679391345,"tpot_p50_ms":45.612174527642615,"tpot_p95_ms":64.0157438028149,"tpot_p99_ms":68.22527053693537,"e2e_p50_ms":8231.357355020009,"e2e_p95_ms":9618.941819528114,"e2e_p99_ms":10381.7686297826,"itl_p50_ms":32.42974996101111,"itl_p95_ms":33.23681900510564,"itl_p99_ms":800.9760206076327,"validation_errors":[],"timestamp":"2026-07-30T22:10:10+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i8192_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i8192_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":24.12239391996991,"request_tps":0.20727627683174146,"input_tps":1698.007259805626,"output_tps":26.531363434462907,"total_tps":1724.538623240089,"mean_input_tokens":8192.0,"mean_output_tokens":128.0,"ttft_p50_ms":760.4200500063598,"ttft_p95_ms":763.9043580042197,"ttft_p99_ms":764.0871676104143,"tpot_p50_ms":32.01531889705413,"tpot_p95_ms":32.202741873861235,"tpot_p99_ms":32.204054255463824,"e2e_p50_ms":4824.785929988138,"e2e_p95_ms":4844.88150398247,"e2e_p99_ms":4848.08537164703,"itl_p50_ms":31.98761004023254,"itl_p95_ms":32.76310997316614,"itl_p99_ms":33.5049433959648,"validation_errors":[],"timestamp":"2026-07-30T22:10:51+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i8192_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i8192_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":79.76820316992234,"request_tps":1.002905880048268,"input_tps":8215.804969355411,"output_tps":256.7439052923566,"total_tps":8472.548874647768,"mean_input_tokens":8192.0,"mean_output_tokens":256.0,"ttft_p50_ms":3432.2469150065444,"ttft_p95_ms":5187.720279477068,"ttft_p99_ms":6183.963104182617,"tpot_p50_ms":48.20486480397555,"tpot_p95_ms":59.15395624497358,"tpot_p99_ms":62.60881791608956,"e2e_p50_ms":15743.238400027622,"e2e_p95_ms":17438.800724968314,"e2e_p99_ms":18435.745777687986,"itl_p50_ms":35.39586998522282,"itl_p95_ms":36.71266995370388,"itl_p99_ms":850.9319839626551,"validation_errors":[],"timestamp":"2026-07-30T22:12:34+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i8192_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i8192_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":64.00357100996189,"request_tps":0.6249651288015503,"input_tps":5119.7143351423,"output_tps":159.99107297319688,"total_tps":5279.7054081154965,"mean_input_tokens":8192.0,"mean_output_tokens":256.0,"ttft_p50_ms":2323.569164960645,"ttft_p95_ms":4639.848213479853,"ttft_p99_ms":4968.635169529589,"tpot_p50_ms":40.688779019732394,"tpot_p95_ms":50.070788702155994,"tpot_p99_ms":50.25366874124544,"e2e_p50_ms":12721.775530022569,"e2e_p95_ms":13717.668963485627,"e2e_p99_ms":14477.898470436921,"itl_p50_ms":32.64263999881223,"itl_p95_ms":33.321365976007655,"itl_p99_ms":65.461080899695,"validation_errors":[],"timestamp":"2026-07-30T22:13:59+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i8192_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i8192_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":44.84280625998508,"request_tps":0.11150060437813607,"input_tps":913.4129510656907,"output_tps":28.544154720802833,"total_tps":941.9571057864935,"mean_input_tokens":8192.0,"mean_output_tokens":256.0,"ttft_p50_ms":764.1480000456795,"ttft_p95_ms":768.026505992748,"ttft_p99_ms":768.052565199323,"tpot_p50_ms":32.23698690223197,"tpot_p95_ms":32.39465422718329,"tpot_p99_ms":32.40736331624509,"e2e_p50_ms":8963.383469963446,"e2e_p95_ms":9027.072053938173,"e2e_p99_ms":9031.611954844557,"itl_p50_ms":32.157110050320625,"itl_p95_ms":32.99977403366938,"itl_p99_ms":33.46499894512817,"validation_errors":[],"timestamp":"2026-07-30T22:15:01+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":256,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i8192_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i8192_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":127.1029921900481,"request_tps":0.6294108314962535,"input_tps":5156.133531617309,"output_tps":322.2583457260818,"total_tps":5478.391877343391,"mean_input_tokens":8192.0,"mean_output_tokens":512.0,"ttft_p50_ms":3450.0303200329654,"ttft_p95_ms":5189.618267561305,"ttft_p99_ms":6183.65977530018,"tpot_p50_ms":43.19568540125906,"tpot_p95_ms":48.18383052851959,"tpot_p99_ms":50.15244488916934,"e2e_p50_ms":25323.496214987244,"e2e_p95_ms":27339.611532050185,"e2e_p99_ms":27552.141045666063,"itl_p50_ms":35.82602500682697,"itl_p95_ms":36.72540552215651,"itl_p99_ms":39.80812013847758,"validation_errors":[],"timestamp":"2026-07-30T22:17:31+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i8192_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i8192_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":103.82739592995495,"request_tps":0.3852547744430111,"input_tps":3156.007112237147,"output_tps":197.25044451482168,"total_tps":3353.2575567519684,"mean_input_tokens":8192.0,"mean_output_tokens":512.0,"ttft_p50_ms":2615.7780999783427,"ttft_p95_ms":3609.4307590217795,"ttft_p99_ms":3882.7247278741556,"tpot_p50_ms":35.50797715268705,"tpot_p95_ms":39.09098276890199,"tpot_p99_ms":39.08470750266482,"e2e_p50_ms":20760.04894002108,"e2e_p95_ms":20803.108048369177,"e2e_p99_ms":21209.463782060193,"itl_p50_ms":32.741850009188056,"itl_p95_ms":33.308899006806314,"itl_p99_ms":34.04030823148787,"validation_errors":[],"timestamp":"2026-07-30T22:19:35+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":512,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i8192_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i8192_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":212.01370789995417,"request_tps":0.37733409217931657,"input_tps":3091.1208831329614,"output_tps":386.39011039162017,"total_tps":3477.5109935245814,"mean_input_tokens":8192.0,"mean_output_tokens":1024.0,"ttft_p50_ms":3217.2945700003766,"ttft_p95_ms":5198.933932511133,"ttft_p99_ms":6195.261666304427,"tpot_p50_ms":38.548036867023,"tpot_p95_ms":40.674115410044465,"tpot_p99_ms":41.567788681148286,"e2e_p50_ms":42195.522080000956,"e2e_p95_ms":44629.9099859898,"e2e_p99_ms":45626.029903986484,"itl_p50_ms":35.347559954971075,"itl_p95_ms":36.13748695934191,"itl_p99_ms":36.862350879237056,"validation_errors":[],"timestamp":"2026-07-30T22:23:30+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i8192_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i8192_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":181.82139122998342,"request_tps":0.2199961166802675,"input_tps":1802.2081878447514,"output_tps":225.27602348059392,"total_tps":2027.4842113253453,"mean_input_tokens":8192.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1897.2331350087188,"ttft_p95_ms":2498.2621494855252,"ttft_p99_ms":2947.1088344499003,"tpot_p50_ms":33.67825573797767,"tpot_p95_ms":34.766068662347664,"tpot_p99_ms":34.79911653343056,"e2e_p50_ms":36347.188025014475,"e2e_p95_ms":36452.67974042217,"e2e_p99_ms":37125.30793252052,"itl_p50_ms":32.74657001020387,"itl_p95_ms":33.23221002356149,"itl_p99_ms":33.59079642803408,"validation_errors":[],"timestamp":"2026-07-30T22:26:53+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":1024,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i8192_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i8192_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":398.5475814199308,"request_tps":0.20072885579929733,"input_tps":1644.3707867078438,"output_tps":411.09269667696094,"total_tps":2055.463483384805,"mean_input_tokens":8192.0,"mean_output_tokens":2048.0,"ttft_p50_ms":2955.541364965029,"ttft_p95_ms":5010.337519459426,"ttft_p99_ms":5566.39981483575,"tpot_p50_ms":37.53321514165545,"tpot_p95_ms":38.66489654591509,"tpot_p99_ms":38.980949183663924,"e2e_p50_ms":79836.69116999954,"e2e_p95_ms":80674.94884411572,"e2e_p99_ms":81454.14971042308,"itl_p50_ms":35.77953006606549,"itl_p95_ms":36.60918492823839,"itl_p99_ms":37.316652024164796,"validation_errors":[],"timestamp":"2026-07-30T22:33:55+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i8192_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i8192_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":353.1742680199677,"request_tps":0.11325853444605564,"input_tps":927.8139141820878,"output_tps":231.95347854552196,"total_tps":1159.7673927276098,"mean_input_tokens":8192.0,"mean_output_tokens":2048.0,"ttft_p50_ms":2296.5269449632615,"ttft_p95_ms":4218.431406922171,"ttft_p99_ms":4808.634921110934,"tpot_p50_ms":33.36470043478337,"tpot_p95_ms":34.54703166716327,"tpot_p99_ms":34.93756947424954,"e2e_p50_ms":70604.13083503954,"e2e_p95_ms":72411.30348497536,"e2e_p99_ms":73240.00013367739,"itl_p50_ms":32.57943002972752,"itl_p95_ms":33.151150099001825,"itl_p99_ms":33.59684348106385,"validation_errors":[],"timestamp":"2026-07-30T22:40:09+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":2048,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i8192_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i8192_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":333.40072796994355,"request_tps":0.014996967854403593,"input_tps":122.85516066327423,"output_tps":30.71379016581856,"total_tps":153.56895082909278,"mean_input_tokens":8192.0,"mean_output_tokens":2048.0,"ttft_p50_ms":762.1548100141808,"ttft_p95_ms":765.5721820658074,"ttft_p99_ms":766.2075644684955,"tpot_p50_ms":32.14388766978209,"tpot_p95_ms":32.34333988760462,"tpot_p99_ms":32.35730914667915,"e2e_p50_ms":66537.45558997616,"e2e_p95_ms":66969.81387995183,"e2e_p99_ms":66997.7350972686,"itl_p50_ms":32.17111004050821,"itl_p95_ms":32.93500803411007,"itl_p99_ms":33.282529131975025,"validation_errors":[],"timestamp":"2026-07-30T22:45:59+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":2048,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i8192_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i8192_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":769.5213686000789,"request_tps":0.10396072580224357,"input_tps":851.6462657719793,"output_tps":425.82313288598965,"total_tps":1277.469398657969,"mean_input_tokens":8192.0,"mean_output_tokens":4096.0,"ttft_p50_ms":3468.727884988766,"ttft_p95_ms":6034.324070031289,"ttft_p99_ms":6207.9662846145175,"tpot_p50_ms":36.657809081810875,"tpot_p95_ms":37.66410315862598,"tpot_p99_ms":38.072294059362186,"e2e_p50_ms":153269.9984300416,"e2e_p95_ms":156718.12042262751,"e2e_p99_ms":156721.14878461813,"itl_p50_ms":35.82233004271984,"itl_p95_ms":36.720580072142184,"itl_p99_ms":37.30247050523759,"validation_errors":[],"timestamp":"2026-07-30T22:59:14+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":4096,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i8192_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i8192_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":682.6778566800058,"request_tps":0.05859278956919406,"input_tps":479.99213215083773,"output_tps":239.99606607541887,"total_tps":719.9881982262566,"mean_input_tokens":8192.0,"mean_output_tokens":4096.0,"ttft_p50_ms":1901.6123450128362,"ttft_p95_ms":2725.8143440238196,"ttft_p99_ms":2946.7279067763593,"tpot_p50_ms":32.847931531132275,"tpot_p95_ms":33.26826429908937,"tpot_p99_ms":33.26740647333825,"e2e_p50_ms":136455.91557997977,"e2e_p95_ms":137119.01321701007,"e2e_p99_ms":137160.87164167664,"itl_p50_ms":32.60632499586791,"itl_p95_ms":33.0648300005123,"itl_p99_ms":33.433920005336404,"validation_errors":[],"timestamp":"2026-07-30T23:10:58+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":4096,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i8192_o4096_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i8192_o4096_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":80.06392871995922,"request_tps":0.999201529065819,"input_tps":16370.91785221438,"output_tps":127.89779572042484,"total_tps":16498.815647934804,"mean_input_tokens":16384.0,"mean_output_tokens":128.0,"ttft_p50_ms":5071.793364943005,"ttft_p95_ms":9515.620909980496,"ttft_p99_ms":11679.143614190398,"tpot_p50_ms":85.33555208650365,"tpot_p95_ms":112.78037994845191,"tpot_p99_ms":113.24017413759854,"e2e_p50_ms":15956.599555036519,"e2e_p95_ms":20232.89516950608,"e2e_p99_ms":22895.673688390525,"itl_p50_ms":35.25934007484466,"itl_p95_ms":865.0056715297978,"itl_p99_ms":909.3054034083616,"validation_errors":[],"timestamp":"2026-07-30T23:12:47+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i16384_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i16384_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":57.43199563003145,"request_tps":0.6964758852830776,"input_tps":11411.060904477943,"output_tps":89.14891331623393,"total_tps":11500.209817794177,"mean_input_tokens":16384.0,"mean_output_tokens":128.0,"ttft_p50_ms":3982.042574963998,"ttft_p95_ms":6398.342003475409,"ttft_p99_ms":6935.166891488479,"tpot_p50_ms":55.022600315434225,"tpot_p95_ms":80.96102403190835,"tpot_p99_ms":83.39030308728692,"e2e_p50_ms":11951.426215004176,"e2e_p95_ms":12652.63668905827,"e2e_p99_ms":13259.991616997868,"itl_p50_ms":32.64083998510614,"itl_p95_ms":33.80479299230501,"itl_p99_ms":1081.8545488466018,"validation_errors":[],"timestamp":"2026-07-30T23:14:08+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i16384_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i16384_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":26.807673510047607,"request_tps":0.18651376062626185,"input_tps":3055.841454100674,"output_tps":23.873761360161517,"total_tps":3079.7152154608357,"mean_input_tokens":16384.0,"mean_output_tokens":128.0,"ttft_p50_ms":1511.5274098934606,"ttft_p95_ms":1526.1669679777697,"ttft_p99_ms":1526.3533295923844,"tpot_p50_ms":32.069420708732046,"tpot_p95_ms":32.321668046849275,"tpot_p99_ms":32.338059142994595,"e2e_p50_ms":5591.621770057827,"e2e_p95_ms":5604.43296181038,"e2e_p99_ms":5605.725061893463,"itl_p50_ms":32.09253994282335,"itl_p95_ms":32.738758984487504,"itl_p99_ms":33.329377428162836,"validation_errors":[],"timestamp":"2026-07-30T23:14:53+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i16384_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i16384_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":114.31496227998286,"request_tps":0.6998209018698894,"input_tps":11465.865656236268,"output_tps":179.1541508786917,"total_tps":11645.01980711496,"mean_input_tokens":16384.0,"mean_output_tokens":256.0,"ttft_p50_ms":4956.666279991623,"ttft_p95_ms":8644.620970974209,"ttft_p99_ms":10855.138358973189,"tpot_p50_ms":71.74005311772264,"tpot_p95_ms":82.36015661779389,"tpot_p99_ms":87.61948602489085,"e2e_p50_ms":23546.152259979863,"e2e_p95_ms":26220.31920055742,"e2e_p99_ms":27159.162078545185,"itl_p50_ms":35.43451998848468,"itl_p95_ms":36.81143904104829,"itl_p99_ms":1010.954532804899,"validation_errors":[],"timestamp":"2026-07-30T23:17:18+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i16384_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i16384_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":74.15404910000507,"request_tps":0.5394176108448981,"input_tps":8837.818136082811,"output_tps":138.09090837629392,"total_tps":8975.909044459106,"mean_input_tokens":16384.0,"mean_output_tokens":256.0,"ttft_p50_ms":3426.246049988549,"ttft_p95_ms":6389.42795046023,"ttft_p99_ms":6966.060597798787,"tpot_p50_ms":43.26478168619431,"tpot_p95_ms":51.93020603361998,"tpot_p99_ms":58.052050200053586,"e2e_p50_ms":14640.961005003192,"e2e_p95_ms":16477.29348452412,"e2e_p99_ms":17710.0465189456,"itl_p50_ms":32.74336003232747,"itl_p95_ms":33.45224198419601,"itl_p99_ms":754.170122044161,"validation_errors":[],"timestamp":"2026-07-30T23:18:55+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i16384_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i16384_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":47.63255255995318,"request_tps":0.10497023004817348,"input_tps":1719.8322491092742,"output_tps":26.87237889233241,"total_tps":1746.7046280016066,"mean_input_tokens":16384.0,"mean_output_tokens":256.0,"ttft_p50_ms":1512.2578700538725,"ttft_p95_ms":1527.869722014293,"ttft_p99_ms":1528.7186644319445,"tpot_p50_ms":32.25084894117625,"tpot_p95_ms":32.508531113739544,"tpot_p99_ms":32.5194845754361,"e2e_p50_ms":9688.020610017702,"e2e_p95_ms":9805.267910053955,"e2e_p99_ms":9805.393357989378,"itl_p50_ms":32.24425995722413,"itl_p95_ms":33.100191969424486,"itl_p99_ms":33.44084400916472,"validation_errors":[],"timestamp":"2026-07-30T23:20:01+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":256,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i16384_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i16384_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":158.7260950099444,"request_tps":0.5040129034547716,"input_tps":8257.747410202977,"output_tps":258.05460656884304,"total_tps":8515.80201677182,"mean_input_tokens":16384.0,"mean_output_tokens":512.0,"ttft_p50_ms":4941.926164960023,"ttft_p95_ms":8647.874704474812,"ttft_p99_ms":10867.240724399442,"tpot_p50_ms":54.2915322897143,"tpot_p95_ms":57.658430265111,"tpot_p99_ms":59.86091890567575,"e2e_p50_ms":32235.890500014648,"e2e_p95_ms":34828.54786563548,"e2e_p99_ms":36046.235660676604,"itl_p50_ms":35.50728998379782,"itl_p95_ms":36.53704052558169,"itl_p99_ms":896.0690329025964,"validation_errors":[],"timestamp":"2026-07-30T23:23:10+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i16384_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i16384_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":119.25208306999411,"request_tps":0.33542391017624656,"input_tps":5495.585344327624,"output_tps":171.73704201023824,"total_tps":5667.322386337862,"mean_input_tokens":16384.0,"mean_output_tokens":512.0,"ttft_p50_ms":3870.0622700271197,"ttft_p95_ms":6401.98362050578,"ttft_p99_ms":6992.524760661181,"tpot_p50_ms":38.530051731914575,"tpot_p95_ms":43.2417386518913,"tpot_p99_ms":45.50062891655172,"e2e_p50_ms":23896.730630018283,"e2e_p95_ms":25190.164128923785,"e2e_p99_ms":26532.708877677796,"itl_p50_ms":32.75144996587187,"itl_p95_ms":33.31755397375673,"itl_p99_ms":35.00160082243383,"validation_errors":[],"timestamp":"2026-07-30T23:25:33+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":512,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i16384_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i16384_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":89.08638435998,"request_tps":0.05612529945985926,"input_tps":919.5569063503341,"output_tps":28.73615332344794,"total_tps":948.2930596737821,"mean_input_tokens":16384.0,"mean_output_tokens":512.0,"ttft_p50_ms":1509.3700600555167,"ttft_p95_ms":1522.362068039365,"ttft_p99_ms":1522.611165647395,"tpot_p50_ms":32.34824731899517,"tpot_p95_ms":32.56563354208847,"tpot_p99_ms":32.57069389597347,"e2e_p50_ms":18051.070960005745,"e2e_p95_ms":18142.053974047303,"e2e_p99_ms":18151.323675694875,"itl_p50_ms":32.29875001125038,"itl_p95_ms":33.09366600587964,"itl_p99_ms":33.40430604061112,"validation_errors":[],"timestamp":"2026-07-30T23:27:20+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":512,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i16384_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i16384_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":246.99122473003808,"request_tps":0.32389814693797386,"input_tps":5306.747239431764,"output_tps":331.67170246448524,"total_tps":5638.418941896249,"mean_input_tokens":16384.0,"mean_output_tokens":1024.0,"ttft_p50_ms":4818.220240005758,"ttft_p95_ms":9568.643386021718,"ttft_p99_ms":13441.089669321658,"tpot_p50_ms":43.67226698928133,"tpot_p95_ms":46.298606610916806,"tpot_p99_ms":47.94557331616281,"e2e_p50_ms":49055.76183996163,"e2e_p95_ms":55417.24518160336,"e2e_p99_ms":58254.314218657084,"itl_p50_ms":35.57665494736284,"itl_p95_ms":36.36986351921223,"itl_p99_ms":39.907804812537506,"validation_errors":[],"timestamp":"2026-07-30T23:31:57+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i16384_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i16384_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":200.36147830996197,"request_tps":0.199639173844183,"input_tps":3270.8882242630943,"output_tps":204.4305140164434,"total_tps":3475.3187382795377,"mean_input_tokens":16384.0,"mean_output_tokens":1024.0,"ttft_p50_ms":3390.7036099699326,"ttft_p95_ms":6387.547698966227,"ttft_p99_ms":6933.166983270785,"tpot_p50_ms":35.64114845551292,"tpot_p95_ms":37.6733207110671,"tpot_p99_ms":39.22120363706334,"e2e_p50_ms":39782.47305500554,"e2e_p95_ms":41815.376722003566,"e2e_p99_ms":43439.745107070776,"itl_p50_ms":32.75929996743798,"itl_p95_ms":33.33421198185533,"itl_p99_ms":33.94344194326549,"validation_errors":[],"timestamp":"2026-07-30T23:35:41+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":1024,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i16384_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i16384_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":171.93057628010865,"request_tps":0.02908150550169749,"input_tps":476.47138613981167,"output_tps":29.77946163373823,"total_tps":506.2508477735499,"mean_input_tokens":16384.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1511.7493600118905,"ttft_p95_ms":1524.2673779604957,"ttft_p99_ms":1524.7432675538585,"tpot_p50_ms":32.39681810356371,"tpot_p95_ms":32.516540351887315,"tpot_p99_ms":32.52176622682153,"e2e_p50_ms":34663.832849939354,"e2e_p95_ms":34766.634137998335,"e2e_p99_ms":34779.60900965147,"itl_p50_ms":32.29630005080253,"itl_p95_ms":33.16990796010941,"itl_p99_ms":33.52519875159487,"validation_errors":[],"timestamp":"2026-07-30T23:38:52+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":1024,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i16384_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i16384_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":430.1635066899471,"request_tps":0.1859757946823284,"input_tps":3047.0274200752683,"output_tps":380.87842750940854,"total_tps":3427.905847584677,"mean_input_tokens":16384.0,"mean_output_tokens":2048.0,"ttft_p50_ms":4829.289519984741,"ttft_p95_ms":8658.818910975238,"ttft_p99_ms":10867.971536610265,"tpot_p50_ms":39.64858473620151,"tpot_p95_ms":41.02853787646718,"tpot_p99_ms":41.614869596871344,"e2e_p50_ms":86180.7442199788,"e2e_p95_ms":89039.925284934,"e2e_p99_ms":90648.56255627818,"itl_p50_ms":35.70271993521601,"itl_p95_ms":36.746554949786514,"itl_p99_ms":37.673628027550876,"validation_errors":[],"timestamp":"2026-07-30T23:46:32+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":2048,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i16384_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i16384_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":370.4225523399655,"request_tps":0.10798478588120332,"input_tps":1769.2227318776352,"output_tps":221.1528414847044,"total_tps":1990.3755733623395,"mean_input_tokens":16384.0,"mean_output_tokens":2048.0,"ttft_p50_ms":4094.8109249584377,"ttft_p95_ms":6492.4128525308315,"ttft_p99_ms":7273.569084772607,"tpot_p50_ms":34.1700071690292,"tpot_p95_ms":35.32191478040292,"tpot_p99_ms":36.06176294423183,"e2e_p50_ms":73763.13411502633,"e2e_p95_ms":75996.02211294696,"e2e_p99_ms":76143.74665727606,"itl_p50_ms":32.89110993500799,"itl_p95_ms":33.468777989037335,"itl_p99_ms":33.919533685548224,"validation_errors":[],"timestamp":"2026-07-30T23:53:06+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":2048,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i16384_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i16384_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":337.8656215099618,"request_tps":0.014798782952981138,"input_tps":242.46325990164297,"output_tps":30.30790748770537,"total_tps":272.77116738934836,"mean_input_tokens":16384.0,"mean_output_tokens":2048.0,"ttft_p50_ms":1514.0022500418127,"ttft_p95_ms":1528.195677977055,"ttft_p99_ms":1528.2413515634835,"tpot_p50_ms":32.50193069856177,"tpot_p95_ms":32.538110549101255,"tpot_p99_ms":32.5426682458124,"e2e_p50_ms":67993.43759997282,"e2e_p95_ms":68122.30755605269,"e2e_p99_ms":68129.40275161993,"itl_p50_ms":32.28677005972713,"itl_p95_ms":33.2750630681403,"itl_p99_ms":33.612142354249954,"validation_errors":[],"timestamp":"2026-07-30T23:59:02+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":2048,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i16384_o2048_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i16384_o2048_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":112.1049548999872,"request_tps":0.7136169857199517,"input_tps":23383.801388071377,"output_tps":91.34297417215382,"total_tps":23475.14436224353,"mean_input_tokens":32768.0,"mean_output_tokens":128.0,"ttft_p50_ms":6264.532379922457,"ttft_p95_ms":11788.734544499308,"ttft_p99_ms":14713.95283750724,"tpot_p50_ms":124.17686322848789,"tpot_p95_ms":159.6013339885371,"tpot_p99_ms":169.82236752588128,"e2e_p50_ms":22536.923530045897,"e2e_p95_ms":26994.105547526855,"e2e_p99_ms":27848.871363776263,"itl_p50_ms":35.80653003882617,"itl_p95_ms":980.5753880064003,"itl_p99_ms":1270.582257756032,"validation_errors":[],"timestamp":"2026-07-31T00:01:28+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i32768_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i32768_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":64.24533890001476,"request_tps":0.6226132616757168,"input_tps":20401.79135858989,"output_tps":79.69449749449176,"total_tps":20481.485856084382,"mean_input_tokens":32768.0,"mean_output_tokens":128.0,"ttft_p50_ms":5632.310719927773,"ttft_p95_ms":8470.602866029365,"ttft_p99_ms":8926.933612974826,"tpot_p50_ms":58.833704016072396,"tpot_p95_ms":84.75787846276869,"tpot_p99_ms":88.91485174214944,"e2e_p50_ms":12564.81506000273,"e2e_p95_ms":15607.383756409396,"e2e_p99_ms":16647.468635570258,"itl_p50_ms":33.08084001764655,"itl_p95_ms":34.12081797141582,"itl_p99_ms":990.1363984996009,"validation_errors":[],"timestamp":"2026-07-31T00:02:57+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i32768_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i32768_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":31.958580569946207,"request_tps":0.15645250542516245,"input_tps":5126.635697771723,"output_tps":20.025920694420794,"total_tps":5146.661618466144,"mean_input_tokens":32768.0,"mean_output_tokens":128.0,"ttft_p50_ms":1700.462199980393,"ttft_p95_ms":3197.5915120681752,"ttft_p99_ms":3203.4451104840264,"tpot_p50_ms":32.293592677745,"tpot_p95_ms":32.60141223647053,"tpot_p99_ms":32.64599751468425,"e2e_p50_ms":5812.605910003185,"e2e_p95_ms":7335.79949408304,"e2e_p99_ms":7349.052497250959,"itl_p50_ms":32.38019999116659,"itl_p95_ms":33.330171974375844,"itl_p99_ms":33.678758293390274,"validation_errors":[],"timestamp":"2026-07-31T00:03:47+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i32768_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i32768_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":137.57226052996702,"request_tps":0.5815125788572312,"input_tps":19055.00418399375,"output_tps":148.86722018745118,"total_tps":19203.8714041812,"mean_input_tokens":32768.0,"mean_output_tokens":256.0,"ttft_p50_ms":7163.180360046681,"ttft_p95_ms":13484.687575988935,"ttft_p99_ms":15979.149309875203,"tpot_p50_ms":75.36448574506258,"tpot_p95_ms":107.55254979811461,"tpot_p99_ms":120.79527146426742,"e2e_p50_ms":25667.993129987735,"e2e_p95_ms":38331.40066248015,"e2e_p99_ms":40406.43265740829,"itl_p50_ms":35.994410049170256,"itl_p95_ms":38.20282751112235,"itl_p99_ms":1162.667948025046,"validation_errors":[],"timestamp":"2026-07-31T00:06:36+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i32768_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i32768_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":94.14696834993083,"request_tps":0.4248676372809554,"input_tps":13922.062738422346,"output_tps":108.76611514392458,"total_tps":14030.82885356627,"mean_input_tokens":32768.0,"mean_output_tokens":256.0,"ttft_p50_ms":4727.724235039204,"ttft_p95_ms":8862.108623445958,"ttft_p99_ms":11843.086503889645,"tpot_p50_ms":51.86683898015569,"tpot_p95_ms":70.85318262547291,"tpot_p99_ms":75.66460194433218,"e2e_p50_ms":18786.882395041175,"e2e_p95_ms":23080.302888399456,"e2e_p99_ms":24655.965517557925,"itl_p50_ms":33.23625004850328,"itl_p95_ms":33.96341904299334,"itl_p99_ms":1003.9056185772628,"validation_errors":[],"timestamp":"2026-07-31T00:08:35+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i32768_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i32768_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":48.75932599999942,"request_tps":0.1025444855410852,"input_tps":3360.1777022102797,"output_tps":26.25138829851781,"total_tps":3386.4290905087973,"mean_input_tokens":32768.0,"mean_output_tokens":256.0,"ttft_p50_ms":1687.9026000387967,"ttft_p95_ms":1698.452756064944,"ttft_p99_ms":1699.8131992807612,"tpot_p50_ms":32.54334796083105,"tpot_p95_ms":32.628604964645326,"tpot_p99_ms":32.643269976199264,"e2e_p50_ms":9984.490629984066,"e2e_p95_ms":10013.645359990187,"e2e_p99_ms":10016.024494720623,"itl_p50_ms":32.567410031333566,"itl_p95_ms":33.526631991844624,"itl_p99_ms":33.839276023209095,"validation_errors":[],"timestamp":"2026-07-31T00:09:42+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":256,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i32768_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i32768_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":175.88219494000077,"request_tps":0.45484990693509736,"input_tps":14904.52175044927,"output_tps":232.88315235076985,"total_tps":15137.40490280004,"mean_input_tokens":32768.0,"mean_output_tokens":512.0,"ttft_p50_ms":7282.497374981176,"ttft_p95_ms":11612.620593543397,"ttft_p99_ms":13513.67576189921,"tpot_p50_ms":56.28897392370058,"tpot_p95_ms":65.42581131187995,"tpot_p99_ms":75.70095716893613,"e2e_p50_ms":35064.28156502079,"e2e_p95_ms":41399.32452555512,"e2e_p99_ms":42509.273093467345,"itl_p50_ms":35.802689963020384,"itl_p95_ms":36.61189399426803,"itl_p99_ms":970.378255995456,"validation_errors":[],"timestamp":"2026-07-31T00:13:10+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":512,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i32768_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i32768_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":126.9079122300027,"request_tps":0.3151891737648764,"input_tps":10328.118845927469,"output_tps":161.3768569676167,"total_tps":10489.495702895085,"mean_input_tokens":32768.0,"mean_output_tokens":512.0,"ttft_p50_ms":5256.267985037994,"ttft_p95_ms":8118.761901563263,"ttft_p99_ms":10673.156017513247,"tpot_p50_ms":39.61883118381148,"tpot_p95_ms":47.53056359306476,"tpot_p99_ms":51.91366864771103,"e2e_p50_ms":23955.839005007874,"e2e_p95_ms":29538.508309487945,"e2e_p99_ms":32583.46117716399,"itl_p50_ms":33.26170000946149,"itl_p95_ms":33.84944249410182,"itl_p99_ms":35.313343023881316,"validation_errors":[],"timestamp":"2026-07-31T00:15:41+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":512,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i32768_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i32768_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":90.00884446001146,"request_tps":0.055550096548804904,"input_tps":1820.265563711239,"output_tps":28.44164943298811,"total_tps":1848.7072131442271,"mean_input_tokens":32768.0,"mean_output_tokens":512.0,"ttft_p50_ms":1683.036610018462,"ttft_p95_ms":1695.400124019943,"ttft_p99_ms":1695.6186248129234,"tpot_p50_ms":32.404827906061215,"tpot_p95_ms":32.43041478654228,"tpot_p99_ms":32.43132917428254,"e2e_p50_ms":18231.23622010462,"e2e_p95_ms":18265.08787793573,"e2e_p99_ms":18267.57699246984,"itl_p50_ms":32.38183003850281,"itl_p95_ms":33.2370329531841,"itl_p99_ms":33.52295583579689,"validation_errors":[],"timestamp":"2026-07-31T00:17:30+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":512,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i32768_o512_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i32768_o512_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":270.35564968991093,"request_tps":0.29590652198967315,"input_tps":9696.26491255761,"output_tps":303.0082785174253,"total_tps":9999.273191075035,"mean_input_tokens":32768.0,"mean_output_tokens":1024.0,"ttft_p50_ms":7577.583965030499,"ttft_p95_ms":13501.426437031478,"ttft_p99_ms":13699.20470859506,"tpot_p50_ms":46.30204676933822,"tpot_p95_ms":51.781796207237264,"tpot_p99_ms":53.38839308724021,"e2e_p50_ms":53080.063844972756,"e2e_p95_ms":60654.80733492295,"e2e_p99_ms":61512.3877670872,"itl_p50_ms":36.065314954612404,"itl_p95_ms":37.221492058597505,"itl_p99_ms":70.87700469419627,"validation_errors":[],"timestamp":"2026-07-31T00:22:33+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":1024,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i32768_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i32768_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":212.70969468005933,"request_tps":0.18804972693024055,"input_tps":6162.013452050122,"output_tps":192.56292037656632,"total_tps":6354.576372426689,"mean_input_tokens":32768.0,"mean_output_tokens":1024.0,"ttft_p50_ms":5012.015764950775,"ttft_p95_ms":7728.843197069361,"ttft_p99_ms":8776.574334638426,"tpot_p50_ms":36.952718264944046,"tpot_p95_ms":39.947256897609876,"tpot_p99_ms":41.310931024580185,"e2e_p50_ms":42896.914635028224,"e2e_p95_ms":44506.33266856893,"e2e_p99_ms":46681.43589362269,"itl_p50_ms":33.07045501424,"itl_p95_ms":33.55089705437422,"itl_p99_ms":34.27205766318366,"validation_errors":[],"timestamp":"2026-07-31T00:26:30+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":1024,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i32768_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i32768_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":174.01568911992945,"request_tps":0.028733041401537433,"input_tps":941.5243006455786,"output_tps":29.42263439517433,"total_tps":970.9469350407529,"mean_input_tokens":32768.0,"mean_output_tokens":1024.0,"ttft_p50_ms":1692.631799960509,"ttft_p95_ms":1699.893803964369,"ttft_p99_ms":1699.9569047568366,"tpot_p50_ms":32.60722347020371,"tpot_p95_ms":32.62421441444707,"tpot_p99_ms":32.62611614512908,"e2e_p50_ms":35021.91101992503,"e2e_p95_ms":35068.926035962075,"e2e_p99_ms":35075.0504328683,"itl_p50_ms":32.617680029943585,"itl_p95_ms":33.44215395627543,"itl_p99_ms":33.75597374280914,"validation_errors":[],"timestamp":"2026-07-31T00:29:42+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":1024,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i32768_o1024_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i32768_o1024_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":211.50950796005782,"request_tps":0.37823358756575365,"input_tps":24787.91639470923,"output_tps":48.41389920841647,"total_tps":24836.330293917646,"mean_input_tokens":65536.0,"mean_output_tokens":128.0,"ttft_p50_ms":13439.222099957988,"ttft_p95_ms":23662.92248601675,"ttft_p99_ms":30041.56156165176,"tpot_p50_ms":239.8931117720103,"tpot_p95_ms":293.0183988311875,"tpot_p99_ms":319.98206732691045,"e2e_p50_ms":43295.3115249984,"e2e_p95_ms":52366.503072017804,"e2e_p99_ms":53455.50217833021,"itl_p50_ms":36.63335501914844,"itl_p95_ms":1177.381918480387,"itl_p99_ms":1454.99622246949,"validation_errors":[],"timestamp":"2026-07-31T00:33:52+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":65536,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i65536_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i65536_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":114.19754190999083,"request_tps":0.3502702363902678,"input_tps":22955.31021207259,"output_tps":44.834590257954275,"total_tps":23000.144802330542,"mean_input_tokens":65536.0,"mean_output_tokens":128.0,"ttft_p50_ms":7280.968330043834,"ttft_p95_ms":22232.449827069646,"ttft_p99_ms":22954.471262221923,"tpot_p50_ms":109.959793464541,"tpot_p95_ms":202.05545632247043,"tpot_p99_ms":216.9044806639935,"e2e_p50_ms":25572.443289973307,"e2e_p95_ms":36189.7886640334,"e2e_p99_ms":37176.05814124341,"itl_p50_ms":33.53754506679252,"itl_p95_ms":1080.6545805069618,"itl_p99_ms":1507.1284884784832,"validation_errors":[],"timestamp":"2026-07-31T00:36:13+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":65536,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i65536_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i65536_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":40.68805445008911,"request_tps":0.12288619024862343,"input_tps":8053.469364133785,"output_tps":15.7294323518238,"total_tps":8069.198796485609,"mean_input_tokens":65536.0,"mean_output_tokens":128.0,"ttft_p50_ms":2137.6239099772647,"ttft_p95_ms":6749.5946100214505,"ttft_p99_ms":6762.26468204055,"tpot_p50_ms":32.93847015739866,"tpot_p95_ms":33.11761971709766,"tpot_p99_ms":33.127447228262746,"e2e_p50_ms":6337.40061998833,"e2e_p95_ms":10945.24920207914,"e2e_p99_ms":10967.393862032332,"itl_p50_ms":33.02799991797656,"itl_p95_ms":33.888378017582,"itl_p99_ms":34.38445155043155,"validation_errors":[],"timestamp":"2026-07-31T00:37:13+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":65536,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i65536_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i65536_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":263.9694500999758,"request_tps":0.30306537354872237,"input_tps":19861.69232088907,"output_tps":77.58473562847293,"total_tps":19939.27705651754,"mean_input_tokens":65536.0,"mean_output_tokens":256.0,"ttft_p50_ms":15504.716625029687,"ttft_p95_ms":29120.100851473395,"ttft_p99_ms":30213.12269011977,"tpot_p50_ms":153.25243300005025,"tpot_p95_ms":225.68189732746367,"tpot_p99_ms":234.74831964564964,"e2e_p50_ms":54026.44227998098,"e2e_p95_ms":81153.42926749725,"e2e_p99_ms":83404.89694684744,"itl_p50_ms":36.58359503606334,"itl_p95_ms":1141.297499940265,"itl_p99_ms":1297.0997788000325,"validation_errors":[],"timestamp":"2026-07-31T00:42:14+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":65536,"osl":256,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i65536_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i65536_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":130.22381440002937,"request_tps":0.30716347992330795,"input_tps":20130.26582025391,"output_tps":78.63385086036683,"total_tps":20208.899671114275,"mean_input_tokens":65536.0,"mean_output_tokens":256.0,"ttft_p50_ms":7582.968545029871,"ttft_p95_ms":19865.768244070932,"ttft_p99_ms":20443.736691303784,"tpot_p50_ms":64.36459898044738,"tpot_p95_ms":97.95557936085115,"tpot_p99_ms":109.40221276348822,"e2e_p50_ms":28245.96308002947,"e2e_p95_ms":32262.50145944068,"e2e_p99_ms":34825.481427304905,"itl_p50_ms":33.730900031514466,"itl_p95_ms":34.501585969701395,"itl_p99_ms":1297.218953981064,"validation_errors":[],"timestamp":"2026-07-31T00:44:51+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":65536,"osl":256,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i65536_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i65536_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":61.70777909993194,"request_tps":0.08102706130296487,"input_tps":5310.189489551106,"output_tps":20.742927693559007,"total_tps":5330.932417244665,"mean_input_tokens":65536.0,"mean_output_tokens":256.0,"ttft_p50_ms":2130.761460051872,"ttft_p95_ms":6733.86445196811,"ttft_p99_ms":6743.597090379335,"tpot_p50_ms":32.89880643133074,"tpot_p95_ms":32.91635636851994,"tpot_p99_ms":32.91768423082563,"e2e_p50_ms":10519.95710004121,"e2e_p95_ms":15127.535325940698,"e2e_p99_ms":15137.60656923987,"itl_p50_ms":33.003499964252114,"itl_p95_ms":33.76901700394228,"itl_p99_ms":34.13493921048939,"validation_errors":[],"timestamp":"2026-07-31T00:46:12+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":65536,"osl":256,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i65536_o256_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i65536_o256_a1.log"}
|
||||
{"status":"COMPLETED","completed":80,"failed":0,"duration_s":631.5253264300991,"request_tps":0.12667742155682946,"input_tps":16603.86299829675,"output_tps":16.21470995927417,"total_tps":16620.077708256023,"mean_input_tokens":131072.0,"mean_output_tokens":128.0,"ttft_p50_ms":33153.88028998859,"ttft_p95_ms":69720.63562800759,"ttft_p99_ms":83305.34909058241,"tpot_p50_ms":711.0364152756556,"tpot_p95_ms":970.6018806381086,"tpot_p99_ms":999.1084034823697,"e2e_p50_ms":134322.98798498232,"e2e_p95_ms":171230.69961137956,"e2e_p99_ms":171814.49480276322,"itl_p50_ms":44.97996997088194,"itl_p95_ms":1687.7042724518105,"itl_p99_ms":1949.6111030108295,"validation_errors":[],"timestamp":"2026-07-31T00:57:39+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":131072,"osl":128,"concurrency":16,"num_prompts":80,"warmup_requests":16,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c16_i131072_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c16_i131072_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":40,"failed":0,"duration_s":377.0665047799703,"request_tps":0.1060820823195134,"input_tps":13904.39069378326,"output_tps":13.578506536897715,"total_tps":13917.969200320158,"mean_input_tokens":131072.0,"mean_output_tokens":128.0,"ttft_p50_ms":33209.02477001073,"ttft_p95_ms":52673.88631201465,"ttft_p99_ms":64568.48465631366,"tpot_p50_ms":326.9828843309427,"tpot_p95_ms":462.71891824881624,"tpot_p99_ms":528.490675542509,"e2e_p50_ms":74623.44429997029,"e2e_p95_ms":108510.93104301252,"e2e_p99_ms":119493.88643381302,"itl_p50_ms":34.93518492905423,"itl_p95_ms":1526.4565850084182,"itl_p99_ms":1751.120554964291,"validation_errors":[],"timestamp":"2026-07-31T01:04:44+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":131072,"osl":128,"concurrency":8,"num_prompts":40,"warmup_requests":8,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c8_i131072_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c8_i131072_o128_a1.log"}
|
||||
{"status":"COMPLETED","completed":5,"failed":0,"duration_s":61.839070249930955,"request_tps":0.080855031936797,"input_tps":10597.830746019856,"output_tps":10.349444087910015,"total_tps":10608.180190107765,"mean_input_tokens":131072.0,"mean_output_tokens":128.0,"ttft_p50_ms":2963.989039999433,"ttft_p95_ms":15828.032030025497,"ttft_p99_ms":15843.257734039798,"tpot_p50_ms":33.55854763796069,"tpot_p95_ms":34.022308188842565,"tpot_p99_ms":34.04963902285777,"e2e_p50_ms":7232.197770033963,"e2e_p95_ms":20140.541708003726,"e2e_p99_ms":20165.897155944258,"itl_p50_ms":34.073940012604,"itl_p95_ms":34.8807190079242,"itl_p99_ms":35.5330960243009,"validation_errors":[],"timestamp":"2026-07-31T01:06:06+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":131072,"osl":128,"concurrency":1,"num_prompts":5,"warmup_requests":1,"attempt":1,"gain_pct":null,"plateau_streak":0,"error_type":"","raw_file":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/raw_outputs/vllm_adaptive_c1_i131072_o128_a1.jsonl","detail_log":"/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_results/adaptive_20260730-103812/tp8_dp2/logs/vllm_c1_i131072_o128_a1.log"}
|
||||
@ -0,0 +1,60 @@
|
||||
{"timestamp":"2026-07-30T10:47:45+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":4428.825448028376,"last_total_tps":4428.825448028376}
|
||||
{"timestamp":"2026-07-30T10:56:09+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":6,"search_cap":256,"max_successful_concurrency":96,"saturation_concurrency":null,"stop_probe_concurrency":96,"best_tps_concurrency":96,"best_total_tps":7798.380163415022,"last_total_tps":7798.380163415022}
|
||||
{"timestamp":"2026-07-30T11:04:44+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":4,"search_cap":256,"max_successful_concurrency":64,"saturation_concurrency":null,"stop_probe_concurrency":64,"best_tps_concurrency":64,"best_total_tps":4012.695269480321,"last_total_tps":4012.695269480321}
|
||||
{"timestamp":"2026-07-30T11:24:13+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":5,"search_cap":256,"max_successful_concurrency":80,"saturation_concurrency":null,"stop_probe_concurrency":80,"best_tps_concurrency":80,"best_total_tps":3420.4486702307668,"last_total_tps":3420.4486702307668}
|
||||
{"timestamp":"2026-07-30T11:52:18+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":4,"search_cap":256,"max_successful_concurrency":64,"saturation_concurrency":null,"stop_probe_concurrency":64,"best_tps_concurrency":64,"best_total_tps":2228.661174639404,"last_total_tps":2228.661174639404}
|
||||
{"timestamp":"2026-07-30T13:17:59+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":1024,"osl":4096,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":6,"search_cap":256,"max_successful_concurrency":96,"saturation_concurrency":null,"stop_probe_concurrency":96,"best_tps_concurrency":96,"best_total_tps":2612.853616310297,"last_total_tps":2612.853616310297}
|
||||
{"timestamp":"2026-07-30T13:19:49+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":8230.869367708057,"last_total_tps":5588.72370679445}
|
||||
{"timestamp":"2026-07-30T13:22:15+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":5464.029293966645,"last_total_tps":3798.3415521822803}
|
||||
{"timestamp":"2026-07-30T13:26:16+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":3418.7013839472093,"last_total_tps":1986.7428011540728}
|
||||
{"timestamp":"2026-07-30T13:33:38+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":3843.2446531144865,"last_total_tps":3843.2446531144865}
|
||||
{"timestamp":"2026-07-30T13:46:04+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":1309.8437554295233,"last_total_tps":740.1634387801678}
|
||||
{"timestamp":"2026-07-30T14:12:03+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":4096,"osl":4096,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":1624.484844419848,"last_total_tps":1624.484844419848}
|
||||
{"timestamp":"2026-07-30T14:14:11+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":11794.190773496694,"last_total_tps":10992.48379473904}
|
||||
{"timestamp":"2026-07-30T14:16:58+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":8896.590124911605,"last_total_tps":6621.716341652809}
|
||||
{"timestamp":"2026-07-30T14:21:03+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":6191.530958021973,"last_total_tps":3781.545743852457}
|
||||
{"timestamp":"2026-07-30T14:27:54+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":3769.8480085506353,"last_total_tps":2118.2809014496156}
|
||||
{"timestamp":"2026-07-30T14:40:39+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":2178.717892678487,"last_total_tps":1179.3718895253367}
|
||||
{"timestamp":"2026-07-30T15:04:26+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":8192,"osl":4096,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":1367.7773433156713,"last_total_tps":737.7780030589151}
|
||||
{"timestamp":"2026-07-30T15:08:23+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":16818.124866879156,"last_total_tps":2737.9826550248995}
|
||||
{"timestamp":"2026-07-30T15:13:33+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":11771.791270578813,"last_total_tps":1687.1052687082602}
|
||||
{"timestamp":"2026-07-30T15:20:41+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":8846.690166322864,"last_total_tps":921.9787195886986}
|
||||
{"timestamp":"2026-07-30T15:31:54+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":5921.078566153044,"last_total_tps":498.4981296000138}
|
||||
{"timestamp":"2026-07-30T15:51:14+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":16384,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":3774.138452225441,"last_total_tps":271.860050365409}
|
||||
{"timestamp":"2026-07-30T15:55:50+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":26622.925600268853,"last_total_tps":4722.802274477127}
|
||||
{"timestamp":"2026-07-30T16:00:57+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":25061.956683529683,"last_total_tps":3270.0100889950027}
|
||||
{"timestamp":"2026-07-30T16:08:20+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":15943.11971550107,"last_total_tps":1803.067771035564}
|
||||
{"timestamp":"2026-07-30T16:20:17+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":32768,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":10189.925332814668,"last_total_tps":956.3794853164297}
|
||||
{"timestamp":"2026-07-30T16:26:59+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":65536,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":8,"best_total_tps":29434.110750535314,"last_total_tps":8189.182435946986}
|
||||
{"timestamp":"2026-07-30T16:34:07+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":65536,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":27792.98952350034,"last_total_tps":6218.987260245673}
|
||||
{"timestamp":"2026-07-30T16:49:36+08:00","engine":"vllm","tp":4,"dp":4,"mark":"Y","isl":131072,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_GROUP_SKIP_THRESHOLD","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":8,"best_total_tps":26129.221079326835,"last_total_tps":7680.149167834397}
|
||||
{"timestamp":"2026-07-30T17:05:53+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":7,"search_cap":256,"max_successful_concurrency":112,"saturation_concurrency":null,"stop_probe_concurrency":112,"best_tps_concurrency":112,"best_total_tps":10223.407113283418,"last_total_tps":10223.407113283418}
|
||||
{"timestamp":"2026-07-30T17:18:52+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":8,"search_cap":256,"max_successful_concurrency":128,"saturation_concurrency":null,"stop_probe_concurrency":128,"best_tps_concurrency":112,"best_total_tps":8192.61079050355,"last_total_tps":7500.657710715577}
|
||||
{"timestamp":"2026-07-30T17:39:47+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":8,"search_cap":256,"max_successful_concurrency":128,"saturation_concurrency":null,"stop_probe_concurrency":128,"best_tps_concurrency":112,"best_total_tps":5585.889406941371,"last_total_tps":5261.3360994775185}
|
||||
{"timestamp":"2026-07-30T18:10:54+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":7,"search_cap":256,"max_successful_concurrency":112,"saturation_concurrency":null,"stop_probe_concurrency":112,"best_tps_concurrency":96,"best_total_tps":3648.898620712223,"last_total_tps":3563.449773683759}
|
||||
{"timestamp":"2026-07-30T19:18:51+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":8,"search_cap":256,"max_successful_concurrency":128,"saturation_concurrency":null,"stop_probe_concurrency":128,"best_tps_concurrency":112,"best_total_tps":3169.95311646314,"last_total_tps":3090.784382182147}
|
||||
{"timestamp":"2026-07-30T21:08:35+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":1024,"osl":4096,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":7,"search_cap":256,"max_successful_concurrency":112,"saturation_concurrency":null,"stop_probe_concurrency":112,"best_tps_concurrency":112,"best_total_tps":2712.8862453231272,"last_total_tps":2712.8862453231272}
|
||||
{"timestamp":"2026-07-30T21:10:55+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":11862.786970287489,"last_total_tps":11862.786970287489}
|
||||
{"timestamp":"2026-07-30T21:14:00+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":8606.740217479904,"last_total_tps":8606.740217479904}
|
||||
{"timestamp":"2026-07-30T21:18:46+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":5504.745130844832,"last_total_tps":5504.745130844832}
|
||||
{"timestamp":"2026-07-30T21:26:36+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":3556.6038169851627,"last_total_tps":3556.6038169851627}
|
||||
{"timestamp":"2026-07-30T21:40:50+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":2277.5847664342523,"last_total_tps":2277.5847664342523}
|
||||
{"timestamp":"2026-07-30T22:07:49+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":4096,"osl":4096,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":2,"search_cap":256,"max_successful_concurrency":32,"saturation_concurrency":null,"stop_probe_concurrency":32,"best_tps_concurrency":32,"best_total_tps":1573.8192958480295,"last_total_tps":1573.8192958480295}
|
||||
{"timestamp":"2026-07-30T22:10:51+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":12074.713648363691,"last_total_tps":1724.538623240089}
|
||||
{"timestamp":"2026-07-30T22:15:01+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":8472.548874647768,"last_total_tps":941.9571057864935}
|
||||
{"timestamp":"2026-07-30T22:19:35+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":5478.391877343391,"last_total_tps":3353.2575567519684}
|
||||
{"timestamp":"2026-07-30T22:26:53+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":3477.5109935245814,"last_total_tps":2027.4842113253453}
|
||||
{"timestamp":"2026-07-30T22:45:59+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":2055.463483384805,"last_total_tps":153.56895082909278}
|
||||
{"timestamp":"2026-07-30T23:10:58+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":8192,"osl":4096,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":2,"search_cap":256,"max_successful_concurrency":8,"saturation_concurrency":null,"stop_probe_concurrency":8,"best_tps_concurrency":16,"best_total_tps":1277.469398657969,"last_total_tps":719.9881982262566}
|
||||
{"timestamp":"2026-07-30T23:14:53+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":16498.815647934804,"last_total_tps":3079.7152154608357}
|
||||
{"timestamp":"2026-07-30T23:20:01+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":11645.01980711496,"last_total_tps":1746.7046280016066}
|
||||
{"timestamp":"2026-07-30T23:27:20+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":8515.80201677182,"last_total_tps":948.2930596737821}
|
||||
{"timestamp":"2026-07-30T23:38:52+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":5638.418941896249,"last_total_tps":506.2508477735499}
|
||||
{"timestamp":"2026-07-30T23:59:02+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":16384,"osl":2048,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":3427.905847584677,"last_total_tps":272.77116738934836}
|
||||
{"timestamp":"2026-07-31T00:03:47+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":23475.14436224353,"last_total_tps":5146.661618466144}
|
||||
{"timestamp":"2026-07-31T00:09:42+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":19203.8714041812,"last_total_tps":3386.4290905087973}
|
||||
{"timestamp":"2026-07-31T00:17:30+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":512,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":15137.40490280004,"last_total_tps":1848.7072131442271}
|
||||
{"timestamp":"2026-07-31T00:29:42+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":32768,"osl":1024,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_BACKOFF_FOUND","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":9999.273191075035,"last_total_tps":970.9469350407529}
|
||||
{"timestamp":"2026-07-31T00:37:13+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":65536,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":24836.330293917646,"last_total_tps":8069.198796485609}
|
||||
{"timestamp":"2026-07-31T00:46:12+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":65536,"osl":256,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_SLO_EXCEEDED","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":8,"best_total_tps":20208.899671114275,"last_total_tps":5330.932417244665}
|
||||
{"timestamp":"2026-07-31T01:06:06+08:00","engine":"vllm","tp":8,"dp":2,"mark":"Y","isl":131072,"osl":128,"status":"TTFT_BOUNDARY","stop_reason":"TTFT_GROUP_SKIP_THRESHOLD","tested_points":3,"search_cap":256,"max_successful_concurrency":1,"saturation_concurrency":null,"stop_probe_concurrency":1,"best_tps_concurrency":16,"best_total_tps":16620.077708256023,"last_total_tps":10608.180190107765}
|
||||
@ -0,0 +1,60 @@
|
||||
{"timestamp": "2026-07-30T10:47:45+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 4428.825448028376, "last_total_tps": 4428.825448028376}
|
||||
{"timestamp": "2026-07-30T10:56:09+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 6, "search_cap": 256, "max_successful_concurrency": 96, "saturation_concurrency": null, "stop_probe_concurrency": 96, "best_tps_concurrency": 96, "best_total_tps": 7798.380163415022, "last_total_tps": 7798.380163415022}
|
||||
{"timestamp": "2026-07-30T11:04:44+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 4, "search_cap": 256, "max_successful_concurrency": 64, "saturation_concurrency": null, "stop_probe_concurrency": 64, "best_tps_concurrency": 64, "best_total_tps": 4012.695269480321, "last_total_tps": 4012.695269480321}
|
||||
{"timestamp": "2026-07-30T11:24:13+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 5, "search_cap": 256, "max_successful_concurrency": 80, "saturation_concurrency": null, "stop_probe_concurrency": 80, "best_tps_concurrency": 80, "best_total_tps": 3420.4486702307668, "last_total_tps": 3420.4486702307668}
|
||||
{"timestamp": "2026-07-30T11:52:18+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 4, "search_cap": 256, "max_successful_concurrency": 64, "saturation_concurrency": null, "stop_probe_concurrency": 64, "best_tps_concurrency": 64, "best_total_tps": 2228.661174639404, "last_total_tps": 2228.661174639404}
|
||||
{"timestamp": "2026-07-30T13:17:59+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 1024, "osl": 4096, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 6, "search_cap": 256, "max_successful_concurrency": 96, "saturation_concurrency": null, "stop_probe_concurrency": 96, "best_tps_concurrency": 96, "best_total_tps": 2612.853616310297, "last_total_tps": 2612.853616310297}
|
||||
{"timestamp": "2026-07-30T13:19:49+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 4096, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 8230.869367708057, "last_total_tps": 5588.72370679445}
|
||||
{"timestamp": "2026-07-30T13:22:15+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 4096, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 5464.029293966645, "last_total_tps": 3798.3415521822803}
|
||||
{"timestamp": "2026-07-30T13:26:16+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 4096, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 3418.7013839472093, "last_total_tps": 1986.7428011540728}
|
||||
{"timestamp": "2026-07-30T13:33:38+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 4096, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 3843.2446531144865, "last_total_tps": 3843.2446531144865}
|
||||
{"timestamp": "2026-07-30T13:46:04+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 4096, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 1309.8437554295233, "last_total_tps": 740.1634387801678}
|
||||
{"timestamp": "2026-07-30T14:12:03+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 4096, "osl": 4096, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 1624.484844419848, "last_total_tps": 1624.484844419848}
|
||||
{"timestamp": "2026-07-30T14:14:11+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 8192, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 11794.190773496694, "last_total_tps": 10992.48379473904}
|
||||
{"timestamp": "2026-07-30T14:16:58+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 8192, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 8896.590124911605, "last_total_tps": 6621.716341652809}
|
||||
{"timestamp": "2026-07-30T14:21:03+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 8192, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 6191.530958021973, "last_total_tps": 3781.545743852457}
|
||||
{"timestamp": "2026-07-30T14:27:54+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 8192, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 3769.8480085506353, "last_total_tps": 2118.2809014496156}
|
||||
{"timestamp": "2026-07-30T14:40:39+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 8192, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 2178.717892678487, "last_total_tps": 1179.3718895253367}
|
||||
{"timestamp": "2026-07-30T15:04:26+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 8192, "osl": 4096, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 1367.7773433156713, "last_total_tps": 737.7780030589151}
|
||||
{"timestamp": "2026-07-30T15:08:23+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 16384, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 16818.124866879156, "last_total_tps": 2737.9826550248995}
|
||||
{"timestamp": "2026-07-30T15:13:33+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 16384, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 11771.791270578813, "last_total_tps": 1687.1052687082602}
|
||||
{"timestamp": "2026-07-30T15:20:41+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 16384, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 8846.690166322864, "last_total_tps": 921.9787195886986}
|
||||
{"timestamp": "2026-07-30T15:31:54+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 16384, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 5921.078566153044, "last_total_tps": 498.4981296000138}
|
||||
{"timestamp": "2026-07-30T15:51:14+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 16384, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 3774.138452225441, "last_total_tps": 271.860050365409}
|
||||
{"timestamp": "2026-07-30T15:55:50+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 32768, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 26622.925600268853, "last_total_tps": 4722.802274477127}
|
||||
{"timestamp": "2026-07-30T16:00:57+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 32768, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 25061.956683529683, "last_total_tps": 3270.0100889950027}
|
||||
{"timestamp": "2026-07-30T16:08:20+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 32768, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 15943.11971550107, "last_total_tps": 1803.067771035564}
|
||||
{"timestamp": "2026-07-30T16:20:17+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 32768, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 10189.925332814668, "last_total_tps": 956.3794853164297}
|
||||
{"timestamp": "2026-07-30T16:26:59+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 65536, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 8, "best_total_tps": 29434.110750535314, "last_total_tps": 8189.182435946986}
|
||||
{"timestamp": "2026-07-30T16:34:07+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 65536, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 27792.98952350034, "last_total_tps": 6218.987260245673}
|
||||
{"timestamp": "2026-07-30T16:49:36+08:00", "engine": "vllm", "tp": 4, "dp": 4, "mark": "Y", "isl": 131072, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_GROUP_SKIP_THRESHOLD", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 8, "best_total_tps": 26129.221079326835, "last_total_tps": 7680.149167834397}
|
||||
{"timestamp": "2026-07-30T17:05:53+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 1024, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 7, "search_cap": 256, "max_successful_concurrency": 112, "saturation_concurrency": null, "stop_probe_concurrency": 112, "best_tps_concurrency": 112, "best_total_tps": 10223.407113283418, "last_total_tps": 10223.407113283418}
|
||||
{"timestamp": "2026-07-30T17:18:52+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 1024, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 8, "search_cap": 256, "max_successful_concurrency": 128, "saturation_concurrency": null, "stop_probe_concurrency": 128, "best_tps_concurrency": 112, "best_total_tps": 8192.61079050355, "last_total_tps": 7500.657710715577}
|
||||
{"timestamp": "2026-07-30T17:39:47+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 1024, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 8, "search_cap": 256, "max_successful_concurrency": 128, "saturation_concurrency": null, "stop_probe_concurrency": 128, "best_tps_concurrency": 112, "best_total_tps": 5585.889406941371, "last_total_tps": 5261.3360994775185}
|
||||
{"timestamp": "2026-07-30T18:10:54+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 1024, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 7, "search_cap": 256, "max_successful_concurrency": 112, "saturation_concurrency": null, "stop_probe_concurrency": 112, "best_tps_concurrency": 96, "best_total_tps": 3648.898620712223, "last_total_tps": 3563.449773683759}
|
||||
{"timestamp": "2026-07-30T19:18:51+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 1024, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 8, "search_cap": 256, "max_successful_concurrency": 128, "saturation_concurrency": null, "stop_probe_concurrency": 128, "best_tps_concurrency": 112, "best_total_tps": 3169.95311646314, "last_total_tps": 3090.784382182147}
|
||||
{"timestamp": "2026-07-30T21:08:35+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 1024, "osl": 4096, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 7, "search_cap": 256, "max_successful_concurrency": 112, "saturation_concurrency": null, "stop_probe_concurrency": 112, "best_tps_concurrency": 112, "best_total_tps": 2712.8862453231272, "last_total_tps": 2712.8862453231272}
|
||||
{"timestamp": "2026-07-30T21:10:55+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 4096, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 11862.786970287489, "last_total_tps": 11862.786970287489}
|
||||
{"timestamp": "2026-07-30T21:14:00+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 4096, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 8606.740217479904, "last_total_tps": 8606.740217479904}
|
||||
{"timestamp": "2026-07-30T21:18:46+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 4096, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 5504.745130844832, "last_total_tps": 5504.745130844832}
|
||||
{"timestamp": "2026-07-30T21:26:36+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 4096, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 3556.6038169851627, "last_total_tps": 3556.6038169851627}
|
||||
{"timestamp": "2026-07-30T21:40:50+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 4096, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 2277.5847664342523, "last_total_tps": 2277.5847664342523}
|
||||
{"timestamp": "2026-07-30T22:07:49+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 4096, "osl": 4096, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 32, "saturation_concurrency": null, "stop_probe_concurrency": 32, "best_tps_concurrency": 32, "best_total_tps": 1573.8192958480295, "last_total_tps": 1573.8192958480295}
|
||||
{"timestamp": "2026-07-30T22:10:51+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 8192, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 12074.713648363691, "last_total_tps": 1724.538623240089}
|
||||
{"timestamp": "2026-07-30T22:15:01+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 8192, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 8472.548874647768, "last_total_tps": 941.9571057864935}
|
||||
{"timestamp": "2026-07-30T22:19:35+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 8192, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 5478.391877343391, "last_total_tps": 3353.2575567519684}
|
||||
{"timestamp": "2026-07-30T22:26:53+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 8192, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 3477.5109935245814, "last_total_tps": 2027.4842113253453}
|
||||
{"timestamp": "2026-07-30T22:45:59+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 8192, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 2055.463483384805, "last_total_tps": 153.56895082909278}
|
||||
{"timestamp": "2026-07-30T23:10:58+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 8192, "osl": 4096, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 2, "search_cap": 256, "max_successful_concurrency": 8, "saturation_concurrency": null, "stop_probe_concurrency": 8, "best_tps_concurrency": 16, "best_total_tps": 1277.469398657969, "last_total_tps": 719.9881982262566}
|
||||
{"timestamp": "2026-07-30T23:14:53+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 16384, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 16498.815647934804, "last_total_tps": 3079.7152154608357}
|
||||
{"timestamp": "2026-07-30T23:20:01+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 16384, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 11645.01980711496, "last_total_tps": 1746.7046280016066}
|
||||
{"timestamp": "2026-07-30T23:27:20+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 16384, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 8515.80201677182, "last_total_tps": 948.2930596737821}
|
||||
{"timestamp": "2026-07-30T23:38:52+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 16384, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 5638.418941896249, "last_total_tps": 506.2508477735499}
|
||||
{"timestamp": "2026-07-30T23:59:02+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 16384, "osl": 2048, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 3427.905847584677, "last_total_tps": 272.77116738934836}
|
||||
{"timestamp": "2026-07-31T00:03:47+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 32768, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 23475.14436224353, "last_total_tps": 5146.661618466144}
|
||||
{"timestamp": "2026-07-31T00:09:42+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 32768, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 19203.8714041812, "last_total_tps": 3386.4290905087973}
|
||||
{"timestamp": "2026-07-31T00:17:30+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 32768, "osl": 512, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 15137.40490280004, "last_total_tps": 1848.7072131442271}
|
||||
{"timestamp": "2026-07-31T00:29:42+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 32768, "osl": 1024, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_BACKOFF_FOUND", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 9999.273191075035, "last_total_tps": 970.9469350407529}
|
||||
{"timestamp": "2026-07-31T00:37:13+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 65536, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 24836.330293917646, "last_total_tps": 8069.198796485609}
|
||||
{"timestamp": "2026-07-31T00:46:12+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 65536, "osl": 256, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_SLO_EXCEEDED", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 8, "best_total_tps": 20208.899671114275, "last_total_tps": 5330.932417244665}
|
||||
{"timestamp": "2026-07-31T01:06:06+08:00", "engine": "vllm", "tp": 8, "dp": 2, "mark": "Y", "isl": 131072, "osl": 128, "status": "TTFT_BOUNDARY", "stop_reason": "TTFT_GROUP_SKIP_THRESHOLD", "tested_points": 3, "search_cap": 256, "max_successful_concurrency": 1, "saturation_concurrency": null, "stop_probe_concurrency": 1, "best_tps_concurrency": 16, "best_total_tps": 16620.077708256023, "last_total_tps": 10608.180190107765}
|
||||
@ -0,0 +1,66 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
| vllm | 4 | 4 | 1024 | 128 | TTFT_SLO_EXCEEDED | None | 32 | 4428.825448028376 | 32 |
|
||||
| vllm | 4 | 4 | 1024 | 256 | TTFT_SLO_EXCEEDED | None | 96 | 7798.380163415022 | 96 |
|
||||
| vllm | 4 | 4 | 1024 | 512 | TTFT_SLO_EXCEEDED | None | 64 | 4012.695269480321 | 64 |
|
||||
| vllm | 4 | 4 | 1024 | 1024 | TTFT_SLO_EXCEEDED | None | 80 | 3420.4486702307668 | 80 |
|
||||
| vllm | 4 | 4 | 1024 | 2048 | TTFT_SLO_EXCEEDED | None | 64 | 2228.661174639404 | 64 |
|
||||
| vllm | 4 | 4 | 1024 | 4096 | TTFT_SLO_EXCEEDED | None | 96 | 2612.853616310297 | 96 |
|
||||
| vllm | 4 | 4 | 4096 | 128 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 8230.869367708057 | 8 |
|
||||
| vllm | 4 | 4 | 4096 | 256 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 5464.029293966645 | 8 |
|
||||
| vllm | 4 | 4 | 4096 | 512 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 3418.7013839472093 | 8 |
|
||||
| vllm | 4 | 4 | 4096 | 1024 | TTFT_SLO_EXCEEDED | None | 32 | 3843.2446531144865 | 32 |
|
||||
| vllm | 4 | 4 | 4096 | 2048 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 1309.8437554295233 | 8 |
|
||||
| vllm | 4 | 4 | 4096 | 4096 | TTFT_SLO_EXCEEDED | None | 32 | 1624.484844419848 | 32 |
|
||||
| vllm | 4 | 4 | 8192 | 128 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 11794.190773496694 | 8 |
|
||||
| vllm | 4 | 4 | 8192 | 256 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 8896.590124911605 | 8 |
|
||||
| vllm | 4 | 4 | 8192 | 512 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 6191.530958021973 | 8 |
|
||||
| vllm | 4 | 4 | 8192 | 1024 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 3769.8480085506353 | 8 |
|
||||
| vllm | 4 | 4 | 8192 | 2048 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 2178.717892678487 | 8 |
|
||||
| vllm | 4 | 4 | 8192 | 4096 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 1367.7773433156713 | 8 |
|
||||
| vllm | 4 | 4 | 16384 | 128 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 16818.124866879156 | 1 |
|
||||
| vllm | 4 | 4 | 16384 | 256 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 11771.791270578813 | 1 |
|
||||
| vllm | 4 | 4 | 16384 | 512 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 8846.690166322864 | 1 |
|
||||
| vllm | 4 | 4 | 16384 | 1024 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 5921.078566153044 | 1 |
|
||||
| vllm | 4 | 4 | 16384 | 2048 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 3774.138452225441 | 1 |
|
||||
| vllm | 4 | 4 | 32768 | 128 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 26622.925600268853 | 1 |
|
||||
| vllm | 4 | 4 | 32768 | 256 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 25061.956683529683 | 1 |
|
||||
| vllm | 4 | 4 | 32768 | 512 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 15943.11971550107 | 1 |
|
||||
| vllm | 4 | 4 | 32768 | 1024 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 10189.925332814668 | 1 |
|
||||
| vllm | 4 | 4 | 65536 | 128 | TTFT_SLO_EXCEEDED | None | 8 | 29434.110750535314 | 1 |
|
||||
| vllm | 4 | 4 | 65536 | 256 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 27792.98952350034 | 1 |
|
||||
| vllm | 4 | 4 | 131072 | 128 | TTFT_GROUP_SKIP_THRESHOLD | None | 8 | 26129.221079326835 | 1 |
|
||||
| vllm | 8 | 2 | 1024 | 128 | TTFT_SLO_EXCEEDED | None | 112 | 10223.407113283418 | 112 |
|
||||
| vllm | 8 | 2 | 1024 | 256 | TTFT_SLO_EXCEEDED | None | 112 | 8192.61079050355 | 128 |
|
||||
| vllm | 8 | 2 | 1024 | 512 | TTFT_SLO_EXCEEDED | None | 112 | 5585.889406941371 | 128 |
|
||||
| vllm | 8 | 2 | 1024 | 1024 | TTFT_SLO_EXCEEDED | None | 96 | 3648.898620712223 | 112 |
|
||||
| vllm | 8 | 2 | 1024 | 2048 | TTFT_SLO_EXCEEDED | None | 112 | 3169.95311646314 | 128 |
|
||||
| vllm | 8 | 2 | 1024 | 4096 | TTFT_SLO_EXCEEDED | None | 112 | 2712.8862453231272 | 112 |
|
||||
| vllm | 8 | 2 | 4096 | 128 | TTFT_SLO_EXCEEDED | None | 32 | 11862.786970287489 | 32 |
|
||||
| vllm | 8 | 2 | 4096 | 256 | TTFT_SLO_EXCEEDED | None | 32 | 8606.740217479904 | 32 |
|
||||
| vllm | 8 | 2 | 4096 | 512 | TTFT_SLO_EXCEEDED | None | 32 | 5504.745130844832 | 32 |
|
||||
| vllm | 8 | 2 | 4096 | 1024 | TTFT_SLO_EXCEEDED | None | 32 | 3556.6038169851627 | 32 |
|
||||
| vllm | 8 | 2 | 4096 | 2048 | TTFT_SLO_EXCEEDED | None | 32 | 2277.5847664342523 | 32 |
|
||||
| vllm | 8 | 2 | 4096 | 4096 | TTFT_SLO_EXCEEDED | None | 32 | 1573.8192958480295 | 32 |
|
||||
| vllm | 8 | 2 | 8192 | 128 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 12074.713648363691 | 1 |
|
||||
| vllm | 8 | 2 | 8192 | 256 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 8472.548874647768 | 1 |
|
||||
| vllm | 8 | 2 | 8192 | 512 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 5478.391877343391 | 8 |
|
||||
| vllm | 8 | 2 | 8192 | 1024 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 3477.5109935245814 | 8 |
|
||||
| vllm | 8 | 2 | 8192 | 2048 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 2055.463483384805 | 1 |
|
||||
| vllm | 8 | 2 | 8192 | 4096 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 1277.469398657969 | 8 |
|
||||
| vllm | 8 | 2 | 16384 | 128 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 16498.815647934804 | 1 |
|
||||
| vllm | 8 | 2 | 16384 | 256 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 11645.01980711496 | 1 |
|
||||
| vllm | 8 | 2 | 16384 | 512 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 8515.80201677182 | 1 |
|
||||
| vllm | 8 | 2 | 16384 | 1024 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 5638.418941896249 | 1 |
|
||||
| vllm | 8 | 2 | 16384 | 2048 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 3427.905847584677 | 1 |
|
||||
| vllm | 8 | 2 | 32768 | 128 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 23475.14436224353 | 1 |
|
||||
| vllm | 8 | 2 | 32768 | 256 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 19203.8714041812 | 1 |
|
||||
| vllm | 8 | 2 | 32768 | 512 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 15137.40490280004 | 1 |
|
||||
| vllm | 8 | 2 | 32768 | 1024 | TTFT_SLO_BACKOFF_FOUND | None | 16 | 9999.273191075035 | 1 |
|
||||
| vllm | 8 | 2 | 65536 | 128 | TTFT_SLO_EXCEEDED | None | 16 | 24836.330293917646 | 1 |
|
||||
| vllm | 8 | 2 | 65536 | 256 | TTFT_SLO_EXCEEDED | None | 8 | 20208.899671114275 | 1 |
|
||||
| vllm | 8 | 2 | 131072 | 128 | TTFT_GROUP_SKIP_THRESHOLD | None | 16 | 16620.077708256023 | 1 |
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260730-103812",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -0,0 +1,6 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260731-082113",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -0,0 +1,6 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260731-083406",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -0,0 +1,6 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260731-083955",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -0,0 +1,6 @@
|
||||
# Adaptive concurrency search summary
|
||||
|
||||
| Engine | TP | DP | ISL | OSL | Stop | Saturation C | Best TPS C | Best Total TPS | Max successful C |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|---:|---:|
|
||||
|
||||
`Saturation C` is the first point in the final low-gain streak. `Best TPS C` is the tested point with the highest observed Total TPS.
|
||||
@ -0,0 +1,27 @@
|
||||
{
|
||||
"experiment": "dsv4_910c_vllm_tp_dp_matrix",
|
||||
"engine": "vllm",
|
||||
"run_id": "adaptive_20260731-084113",
|
||||
"model": "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp",
|
||||
"hardware": "8x Ascend910 (16 dies, 64GB HBM/die)",
|
||||
"matrix": "/mnt/yy/sskj/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json",
|
||||
"dataset": "random",
|
||||
"tokenize_prompt": false,
|
||||
"random_range_ratio": 1.0,
|
||||
"search": {
|
||||
"start_concurrency": 16,
|
||||
"max_concurrency": 256,
|
||||
"multiplier": 2,
|
||||
"initial_backoff_concurrencies": [
|
||||
8,
|
||||
1
|
||||
],
|
||||
"num_prompts_multiplier": 5,
|
||||
"min_tps_gain_pct": 2.0,
|
||||
"plateau_patience": 2,
|
||||
"warmup_max_requests": 0,
|
||||
"ttft_slo_ms": 4000.0,
|
||||
"enable_ttft_slo_stop": 1,
|
||||
"ttft_group_skip_ms": 8000.0
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,31 @@
|
||||
mark input_len output_len
|
||||
Y 1024 128
|
||||
Y 1024 256
|
||||
Y 1024 512
|
||||
Y 1024 1024
|
||||
Y 1024 2048
|
||||
Y 1024 4096
|
||||
Y 4096 128
|
||||
Y 4096 256
|
||||
Y 4096 512
|
||||
Y 4096 1024
|
||||
Y 4096 2048
|
||||
Y 4096 4096
|
||||
Y 8192 128
|
||||
Y 8192 256
|
||||
Y 8192 512
|
||||
Y 8192 1024
|
||||
Y 8192 2048
|
||||
Y 8192 4096
|
||||
Y 16384 128
|
||||
Y 16384 256
|
||||
Y 16384 512
|
||||
Y 16384 1024
|
||||
Y 16384 2048
|
||||
Y 32768 128
|
||||
Y 32768 256
|
||||
Y 32768 512
|
||||
Y 32768 1024
|
||||
Y 65536 128
|
||||
Y 65536 256
|
||||
Y 131072 128
|
||||
|
@ -31,12 +31,12 @@ CONTAINER_PYTHON="/usr/local/python3.12.13/bin/python3"
|
||||
USE_DOCKER="${USE_DOCKER:-1}"
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-local/vllm-ascend:0.23-a3-dsv4-sglang}"
|
||||
|
||||
# Benchmark client Docker image. vLLM's image does not include sglang.bench_serving;
|
||||
# reuse the vllm-ascend container itself for the client via `docker exec` (see
|
||||
# run_adaptive_concurrency_add16.sh), so this is only used if USE_DOCKER_CLIENT=1
|
||||
# with an external sglang image. Default off on 910c.
|
||||
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
|
||||
USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-0}"
|
||||
# Benchmark client Docker image. The vllm-ascend images built on this host
|
||||
# (local/vllm-ascend:0.23-a3-*-sglang, see envs/ASCEND_910C_ENV_SETUP.md §8.1)
|
||||
# carry sglang 0.5.2 bench_serving, so the client runs as a standalone container
|
||||
# (TORCH_DEVICE_BACKEND_AUTOLOAD=0 injected by the runner, no NPU needed).
|
||||
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-local/vllm-ascend:0.23-a3-dsv4-sglang}"
|
||||
USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-1}"
|
||||
|
||||
# Device selection. We mount all 8 cards (16 dies) via --device /dev/davinci0..15
|
||||
# in start_vllm_docker.sh. ASCEND_VISIBLE_DEVICES is kept for parity with the
|
||||
@ -90,18 +90,20 @@ API_SERVER_COUNT="${API_SERVER_COUNT:-1}"
|
||||
|
||||
# Per-TP parameter overrides (applied in start_vllm_docker.sh via case $TP).
|
||||
# DSV4-Flash w8a8 weight per die = ~280GiB / TP. Each die has 64GB HBM.
|
||||
# Defaults below are the verified-OK values (from the 2026-07-29..30 adaptive
|
||||
# runs' server_cmd.txt); 1048576-context OOMs on TP4/TP8 (KV cache budget).
|
||||
# TP=4: ~39 GiB/die weights -> 25 GiB for KV cache; cap context to avoid OOM
|
||||
# TP=8: ~35 GiB/die weights -> 29 GiB for KV cache; balanced
|
||||
# TP=16: ~17.5 GiB/die weights -> 46 GiB for KV cache; full context
|
||||
TP4_GPU_MEMORY_UTILIZATION="${TP4_GPU_MEMORY_UTILIZATION:-0.9}"
|
||||
TP4_MAX_MODEL_LEN="${TP4_MAX_MODEL_LEN:-1048576}"
|
||||
TP4_MAX_NUM_SEQS="${TP4_MAX_NUM_SEQS:-64}"
|
||||
TP4_MAX_MODEL_LEN="${TP4_MAX_MODEL_LEN:-32768}"
|
||||
TP4_MAX_NUM_SEQS="${TP4_MAX_NUM_SEQS:-128}"
|
||||
TP8_GPU_MEMORY_UTILIZATION="${TP8_GPU_MEMORY_UTILIZATION:-0.9}"
|
||||
TP8_MAX_MODEL_LEN="${TP8_MAX_MODEL_LEN:-1048576}"
|
||||
TP8_MAX_NUM_SEQS="${TP8_MAX_NUM_SEQS:-64}"
|
||||
TP8_MAX_MODEL_LEN="${TP8_MAX_MODEL_LEN:-65536}"
|
||||
TP8_MAX_NUM_SEQS="${TP8_MAX_NUM_SEQS:-256}"
|
||||
TP16_GPU_MEMORY_UTILIZATION="${TP16_GPU_MEMORY_UTILIZATION:-0.9}"
|
||||
TP16_MAX_MODEL_LEN="${TP16_MAX_MODEL_LEN:-1048576}"
|
||||
TP16_MAX_NUM_SEQS="${TP16_MAX_NUM_SEQS:-64}"
|
||||
TP16_MAX_MODEL_LEN="${TP16_MAX_MODEL_LEN:-131072}"
|
||||
TP16_MAX_NUM_SEQS="${TP16_MAX_NUM_SEQS:-256}"
|
||||
|
||||
# vLLM-Ascend-specific launch flags injected by start_vllm_docker.sh.
|
||||
# attention backend for 910C: use the fused/atb attention path. Adjust per image.
|
||||
|
||||
124
experiments/910c/dsv4_910c_vllm_tp_dp_matrix/launch_vllm.py
Executable file
124
experiments/910c/dsv4_910c_vllm_tp_dp_matrix/launch_vllm.py
Executable file
@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Drop-in replacement for the docker launch part of start_vllm_docker.sh.
|
||||
Called from bash with: python3 /this_script.py <TP> <DP>
|
||||
Reads config from environment variables set by the bash script.
|
||||
"""
|
||||
import subprocess, sys, os
|
||||
|
||||
tp = sys.argv[1]
|
||||
dp = sys.argv[2]
|
||||
|
||||
# These come from the bash script environment
|
||||
model_path = os.environ.get("MODEL_PATH", "/mnt/models/DeepSeek-V4-Flash-w8a8-mtp")
|
||||
served = os.environ.get("SERVED_MODEL_NAME", "dsv4")
|
||||
port = os.environ.get("VLLM_PORT", "30052")
|
||||
name = os.environ.get("CONTAINER_NAME", "vllm-ascend-dsv4-910c") + f"_tp{tp}_dp{dp}"
|
||||
image = os.environ.get("DOCKER_IMAGE", "local/vllm-ascend:0.23-a3-dsv4-sglang")
|
||||
log = os.environ.get("LAUNCH_LOG", "/tmp/vllm_launch.log")
|
||||
runtime_tmp = os.environ.get("RUNTIME_TMP", "/tmp")
|
||||
pid_file = os.environ.get("PID_FILE", f"/tmp/vllm_tp{tp}_dp{dp}.pid")
|
||||
|
||||
mem_util = os.environ.get("GPU_MEMORY_UTILIZATION", "0.9")
|
||||
max_len = os.environ.get("MAX_MODEL_LEN", "1048576")
|
||||
max_seqs = os.environ.get("MAX_NUM_SEQS", "64")
|
||||
max_tokens = os.environ.get("MAX_NUM_BATCHED_TOKENS", "10240")
|
||||
api_count = os.environ.get("API_SERVER_COUNT", "1")
|
||||
ascend_devices = os.environ.get("ASCEND_VISIBLE_DEVICES", "0,1,2,3,4,5,6,7")
|
||||
|
||||
# Build docker run command
|
||||
docker_cmd = [
|
||||
"docker", "run", "-d",
|
||||
"--name", name,
|
||||
"--privileged", "--ipc", "host", "--shm-size", "512g",
|
||||
"--network", "host", "--ulimit", "memlock=-1",
|
||||
# 16 dies
|
||||
"--device", "/dev/davinci0", "--device", "/dev/davinci1",
|
||||
"--device", "/dev/davinci2", "--device", "/dev/davinci3",
|
||||
"--device", "/dev/davinci4", "--device", "/dev/davinci5",
|
||||
"--device", "/dev/davinci6", "--device", "/dev/davinci7",
|
||||
"--device", "/dev/davinci8", "--device", "/dev/davinci9",
|
||||
"--device", "/dev/davinci10", "--device", "/dev/davinci11",
|
||||
"--device", "/dev/davinci12", "--device", "/dev/davinci13",
|
||||
"--device", "/dev/davinci14", "--device", "/dev/davinci15",
|
||||
"--device", "/dev/davinci_manager", "--device", "/dev/devmm_svm",
|
||||
"--device", "/dev/hisi_hdc",
|
||||
# Driver mounts
|
||||
"-v", "/usr/local/dcmi:/usr/local/dcmi",
|
||||
"-v", "/usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool",
|
||||
"-v", "/usr/local/bin/npu-smi:/usr/local/bin/npu-smi",
|
||||
"-v", "/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64",
|
||||
"-v", "/usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info",
|
||||
"-v", "/etc/ascend_install.info:/etc/ascend_install.info",
|
||||
"-v", "/etc/hccn.conf:/etc/hccn.conf",
|
||||
# Model + data mounts
|
||||
"-v", f"{model_path}:{model_path}:ro",
|
||||
"-v", f"{runtime_tmp}:/tmp",
|
||||
"-v", "/mnt:/mnt",
|
||||
# Env vars
|
||||
"-e", f"ASCEND_VISIBLE_DEVICES={ascend_devices}",
|
||||
"-e", "PYTORCH_NPU_ALLOC_CONF=expandable_segments:True",
|
||||
"-e", "PYTHONUNBUFFERED=1",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-e", "OMP_PROC_BIND=false",
|
||||
"-e", "OMP_NUM_THREADS=10",
|
||||
"-e", "HCCL_BUFFSIZE=1024",
|
||||
"-e", "VLLM_ASCEND_ENABLE_FLASHCOMM1=1",
|
||||
"-e", "TASK_QUEUE_ENABLE=1",
|
||||
"-e", "HCCL_OP_EXPANSION_MODE=AIV",
|
||||
"-e", "LD_PRELOAD=/usr/lib64/libjemalloc.so.2",
|
||||
# Image + command
|
||||
image, "sleep", "infinity"
|
||||
]
|
||||
|
||||
print(f"=== Starting container {name} ===", flush=True)
|
||||
result = subprocess.run(docker_cmd, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
print(result.stderr.strip(), flush=True)
|
||||
sys.exit(1)
|
||||
print(f"Container started: {result.stdout.strip()}", flush=True)
|
||||
|
||||
# Now build vllm serve command (no shell - direct subprocess list)
|
||||
vllm_cmd = [
|
||||
"python3", "/serve.py",
|
||||
model_path,
|
||||
"--served-model-name", served,
|
||||
"--block-size", "128",
|
||||
"--tensor-parallel-size", tp,
|
||||
"--gpu-memory-utilization", mem_util,
|
||||
"--max-model-len", max_len,
|
||||
"--max-num-seqs", max_seqs,
|
||||
"--max-num-batched-tokens", max_tokens,
|
||||
"--api-server-count", api_count,
|
||||
"--host", "0.0.0.0",
|
||||
"--port", port,
|
||||
]
|
||||
if int(dp) > 1:
|
||||
vllm_cmd += ["--data-parallel-size", dp]
|
||||
|
||||
vllm_cmd += [
|
||||
"--tokenizer-mode", "deepseek_v4",
|
||||
"--tool-call-parser", "deepseek_v4",
|
||||
"--enable-auto-tool-choice",
|
||||
"--reasoning-parser", "deepseek_v4",
|
||||
"--quantization", "ascend",
|
||||
"--safetensors-load-strategy", "prefetch",
|
||||
"--enable-expert-parallel",
|
||||
"--async-scheduling",
|
||||
"--model-loader-extra-config", '{"enable_multithread_load": "true", "num_threads": 128}',
|
||||
"--compilation-config", '{"cudagraph_mode": "FULL_DECODE_ONLY"}',
|
||||
"--additional-config", '{"ascend_compilation_config":{"enable_npugraph_ex":true,"enable_static_kernel":false},"enable_cpu_binding": true,"multistream_overlap_shared_expert":true}',
|
||||
]
|
||||
|
||||
print(f"=== Launching vllm serve (log={log}) ===", flush=True)
|
||||
with open(log, "w") as lf:
|
||||
lf.write(f"CMD: vllm serve " + " ".join(vllm_cmd[2:]) + "\n")
|
||||
lf.flush()
|
||||
p = subprocess.Popen(
|
||||
["docker", "exec", name] + vllm_cmd,
|
||||
stdout=lf, stderr=lf
|
||||
)
|
||||
# Write PID file
|
||||
with open(pid_file, "w") as pf:
|
||||
pf.write(str(p.pid))
|
||||
print(f"PID: {p.pid}", flush=True)
|
||||
@ -15,6 +15,10 @@ source "${SCRIPT_DIR}/config.env"
|
||||
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||
|
||||
DEPLOY_PROFILE="${DEPLOY_PROFILE:-910c/dsv4_910c_vllm_tp_dp_matrix}"
|
||||
|
||||
ENGINE="vllm"
|
||||
ENGINE_PORT="$VLLM_PORT"
|
||||
@ -38,20 +42,9 @@ engine_is_healthy() {
|
||||
engine_stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
|
||||
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping vllm server pid=${pid} tp=${tp} dp=${dp}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
log "stopping vllm server tp=${tp} dp=${dp} via deploy profile"
|
||||
bash "${SCRIPT_DIR}/stop_vllm_docker.sh" "$tp" "$dp" >> "${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1 || true
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
sleep 2
|
||||
}
|
||||
@ -59,23 +52,7 @@ engine_stop_server() {
|
||||
engine_build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local -a args=(
|
||||
vllm serve "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$tp"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--no-enable-flashinfer-autotune
|
||||
--host 0.0.0.0
|
||||
--port "$ENGINE_PORT"
|
||||
)
|
||||
if (( dp > 1 )); then
|
||||
args+=(--data-parallel-size "$dp")
|
||||
fi
|
||||
printf '%q ' "${args[@]}"
|
||||
deploy_render_args "$DEPLOY_PROFILE" "$tp" "$dp" "$ENGINE_PORT" "$MODEL_PATH"
|
||||
}
|
||||
|
||||
engine_start_server() {
|
||||
|
||||
@ -14,6 +14,15 @@ set -Eeuo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# Auto-detect docker group access; re-exec under sg docker if needed.
|
||||
if ! docker ps >/dev/null 2>&1; then
|
||||
if sg docker -c "docker ps" >/dev/null 2>&1; then
|
||||
exec sg docker -c "bash ${BASH_SOURCE[0]} ${*:-}"
|
||||
fi
|
||||
echo "ERROR: Cannot access docker. Run 'sudo usermod -aG docker $USER' and re-login." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
@ -24,6 +33,10 @@ source "${SCRIPT_DIR}/config.env"
|
||||
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||
|
||||
DEPLOY_PROFILE="${DEPLOY_PROFILE:-910c/dsv4_910c_vllm_tp_dp_matrix}"
|
||||
|
||||
ENGINE="vllm"
|
||||
ENGINE_PORT="$VLLM_PORT"
|
||||
@ -52,21 +65,9 @@ engine_is_healthy() {
|
||||
engine_stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping vllm-ascend server pid=${pid} tp=${tp} dp=${dp}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
# Also remove the container by name in case the host process is gone but the
|
||||
# container lingers (e.g. orphaned by a kill -9).
|
||||
docker rm -f "${CONTAINER_NAME}_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
|
||||
log "stopping vllm-ascend server tp=${tp} dp=${dp} via deploy profile"
|
||||
bash "${SCRIPT_DIR}/stop_vllm_docker.sh" "$tp" "$dp" >> "${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1 || true
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
sleep 2
|
||||
}
|
||||
@ -96,23 +97,8 @@ engine_build_server_args() {
|
||||
;;
|
||||
esac
|
||||
log "server args tp=${tp} dp=${dp} gpu_mem_util=${mem_util} max_model_len=${max_len} max_num_seqs=${max_seqs}" >&2
|
||||
local -a args=(
|
||||
vllm serve "$MODEL_PATH"
|
||||
--served-model-name "$SERVED_MODEL_NAME"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$tp"
|
||||
--gpu-memory-utilization "$mem_util"
|
||||
--max-model-len "$max_len"
|
||||
--max-num-seqs "$max_seqs"
|
||||
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
|
||||
--api-server-count "$API_SERVER_COUNT"
|
||||
--host 0.0.0.0
|
||||
--port "$ENGINE_PORT"
|
||||
)
|
||||
if (( dp > 1 )); then
|
||||
args+=(--data-parallel-size "$dp")
|
||||
fi
|
||||
printf '%q ' "${args[@]}"
|
||||
export GPU_MEMORY_UTILIZATION="$mem_util" MAX_MODEL_LEN="$max_len" MAX_NUM_SEQS="$max_seqs"
|
||||
deploy_render_args "$DEPLOY_PROFILE" "$tp" "$dp" "$ENGINE_PORT" "$MODEL_PATH"
|
||||
}
|
||||
|
||||
engine_start_server() {
|
||||
|
||||
@ -5,12 +5,23 @@ set -Eeuo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# Auto-detect docker group access; re-exec under sg docker if needed.
|
||||
if ! docker ps >/dev/null 2>&1; then
|
||||
if sg docker -c "docker ps" >/dev/null 2>&1; then
|
||||
exec sg docker -c "bash ${BASH_SOURCE[0]} ${*:-}"
|
||||
fi
|
||||
echo "ERROR: Cannot access docker. Run 'sudo usermod -aG docker $USER' and re-login." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||
|
||||
# Export variables used inside functions that are called via bash -c subshells.
|
||||
export DATASET_PATH MODEL_PATH RESULT_BASE \
|
||||
@ -53,21 +64,8 @@ stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping vllm server pid=${pid} (tp=${tp}, dp=${dp})"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
|
||||
# Fallback: remove any Docker container started by this experiment.
|
||||
docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
log "stopping vllm server tp=${tp} dp=${dp} via deploy profile"
|
||||
bash "${SCRIPT_DIR}/stop_vllm_docker.sh" "$tp" "$dp" >> "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1 || true
|
||||
|
||||
# Fallback: kill any vllm serve processes for this model.
|
||||
pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
@ -79,25 +77,11 @@ build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
|
||||
local args=(
|
||||
"vllm serve" "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$tp"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--no-enable-flashinfer-autotune
|
||||
--host 0.0.0.0
|
||||
--port "$VLLM_PORT"
|
||||
)
|
||||
if [[ "$dp" -gt 1 ]]; then
|
||||
args+=(
|
||||
--data-parallel-size "$dp"
|
||||
)
|
||||
fi
|
||||
printf '%s ' "${args[@]}"
|
||||
deploy_render_args \
|
||||
"${DEPLOY_PROFILE:-910c/dsv4_910c_vllm_tp_dp_matrix}" \
|
||||
"$tp" "$dp" \
|
||||
"${VLLM_PORT:-30052}" \
|
||||
"$MODEL_PATH"
|
||||
}
|
||||
|
||||
start_server() {
|
||||
@ -140,6 +124,7 @@ run_bench_serving() {
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
-e HF_DATASETS_OFFLINE=1 \
|
||||
-e TORCH_DEVICE_BACKEND_AUTOLOAD=0 \
|
||||
"${DOCKER_CLIENT_IMAGE}" \
|
||||
python -m sglang.bench_serving "$@"
|
||||
else
|
||||
|
||||
@ -1,25 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start vLLM-Ascend server in Docker for a given TP×DP configuration on 910C.
|
||||
# Start vLLM-Ascend server for a given TP×DP configuration on 910C through the
|
||||
# shared deployment layer (profile: deploy/profiles/910c/dsv4_910c_vllm_tp_dp_matrix.env).
|
||||
# Usage: start_vllm_docker.sh <TP> <DP>
|
||||
#
|
||||
# DSV4-Flash-specific: mounts host driver libs + all 16 davinci dies explicitly
|
||||
# via --device (NOT relying on Ascend Docker Runtime injection, which was
|
||||
# unreliable here), and injects the DSV4-required vllm serve flags
|
||||
# (tokenizer-mode, expert-parallel, quantization ascend, MTP, ...).
|
||||
#
|
||||
# Synced from glm52 fix (98cdb67): TP=16 device mount, configurable health
|
||||
# timeout, driver mounts, container-name double-suffix fix.
|
||||
# Per-TP overrides (GPU_MEMORY_UTILIZATION / MAX_MODEL_LEN / MAX_NUM_SEQS) are
|
||||
# exported so the profile LAUNCH_ARGS template picks them up.
|
||||
set -e
|
||||
|
||||
TP="${1}"
|
||||
DP="${2}"
|
||||
if [[ -z "$TP" || -z "$DP" ]]; then
|
||||
echo "Usage: $0 <TP> <DP>"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Auto-detect docker group access; re-exec under sg docker if needed.
|
||||
if ! docker ps >/dev/null 2>&1; then
|
||||
if sg docker -c "docker ps" >/dev/null 2>&1; then
|
||||
exec sg docker -c "bash ${BASH_SOURCE[0]} ${1:-} ${2:-}"
|
||||
fi
|
||||
echo "ERROR: Cannot access docker. Run 'sudo usermod -aG docker $USER' and re-login." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||
|
||||
# Per-TP parameter overrides (synced from glm52 fix 6c81183).
|
||||
# DSV4-Flash w8a8 weight per die = ~280GiB / TP; each die has 64GB HBM.
|
||||
# Per-TP parameter overrides (synced from the pre-decoupling launcher).
|
||||
case "$TP" in
|
||||
4)
|
||||
GPU_MEMORY_UTILIZATION="${TP4_GPU_MEMORY_UTILIZATION:-0.9}"
|
||||
@ -37,167 +51,16 @@ case "$TP" in
|
||||
MAX_NUM_SEQS="${TP16_MAX_NUM_SEQS:-256}"
|
||||
;;
|
||||
esac
|
||||
export GPU_MEMORY_UTILIZATION MAX_MODEL_LEN MAX_NUM_SEQS
|
||||
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||
|
||||
IMAGE="${DOCKER_IMAGE:-quay.io/ascend/vllm-ascend:v0.23.0rc1-a3-openeuler}"
|
||||
PORT="${VLLM_PORT:-30052}"
|
||||
# Fixed container name (no env override) to avoid the double-suffix bug where
|
||||
# CONTAINER_NAME already carries _tpX_dpY and the launcher appends it again.
|
||||
NAME="${CONTAINER_NAME}_tp${TP}_dp${DP}"
|
||||
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${TP}_dp${DP}.pid"
|
||||
|
||||
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_vllm_docker_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
# Clean up any stale container with the same name.
|
||||
docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||
|
||||
# ---- vLLM-Ascend launch args ----
|
||||
# Base args common to GLM-5.2 and DSV4-Flash.
|
||||
SERVER_ARGS=(
|
||||
"$MODEL_PATH"
|
||||
--served-model-name "$SERVED_MODEL_NAME"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$TP"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
|
||||
--api-server-count "$API_SERVER_COUNT"
|
||||
--host 0.0.0.0
|
||||
--port "$PORT"
|
||||
)
|
||||
|
||||
if [[ "$DP" -gt 1 ]]; then
|
||||
SERVER_ARGS+=(--data-parallel-size "$DP")
|
||||
fi
|
||||
|
||||
# ---- DSV4-Flash-specific args (REQUIRED; GLM-5.2 does not use these) ----
|
||||
# Verified working in the manual bring-up on 910c.1.
|
||||
SERVER_ARGS+=(
|
||||
--tokenizer-mode "$DSV4_TOKENIZER_MODE"
|
||||
--tool-call-parser "$DSV4_TOOL_CALL_PARSER"
|
||||
--enable-auto-tool-choice
|
||||
--reasoning-parser "$DSV4_REASONING_PARSER"
|
||||
--quantization "$DSV4_QUANTIZATION"
|
||||
--safetensors-load-strategy "$DSV4_SAFETENSORS_LOAD_STRATEGY"
|
||||
)
|
||||
|
||||
if [[ "${DSV4_ENABLE_EXPERT_PARALLEL:-1}" == "1" ]]; then
|
||||
SERVER_ARGS+=(--enable-expert-parallel)
|
||||
fi
|
||||
|
||||
if [[ "${DSV4_ENABLE_ASYNC_SCHEDULING:-1}" == "1" ]]; then
|
||||
SERVER_ARGS+=(--async-scheduling)
|
||||
fi
|
||||
|
||||
# JSON-bearing args. Quoted as single strings so vllm parses each as one argv.
|
||||
SERVER_ARGS+=(--model-loader-extra-config "$DSV4_MODEL_LOADER_EXTRA_CONFIG")
|
||||
if [[ "${DSV4_ENABLE_MTP:-0}" == "1" ]]; then
|
||||
SERVER_ARGS+=(--speculative-config "$DSV4_SPEC_CONFIG")
|
||||
fi
|
||||
SERVER_ARGS+=(--compilation-config "$DSV4_COMPILATION_CONFIG")
|
||||
SERVER_ARGS+=(--additional-config "$DSV4_ADDITIONAL_CONFIG")
|
||||
|
||||
SERVER_ARGS_STR="vllm serve ${SERVER_ARGS[*]}"
|
||||
|
||||
echo "=== Starting vLLM-Ascend server in Docker (TP=${TP}, DP=${DP}) ==="
|
||||
echo "Image: $IMAGE"
|
||||
echo "Model: $MODEL_PATH"
|
||||
echo "Container name: $NAME"
|
||||
echo "Host port: $PORT"
|
||||
echo "ASCEND_VISIBLE_DEVICES: ${ASCEND_VISIBLE_DEVICES}"
|
||||
echo "Command: $SERVER_ARGS_STR"
|
||||
echo "Log: $LOG"
|
||||
|
||||
# ---- Docker run ----
|
||||
# Mount host driver libs + all 16 dies explicitly (synced from glm52 fix).
|
||||
# --privileged + --shm-size 512g for the 280GiB DSV4 weight load.
|
||||
DEVICE_ARGS=(
|
||||
--device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3
|
||||
--device /dev/davinci4 --device /dev/davinci5 --device /dev/davinci6 --device /dev/davinci7
|
||||
--device /dev/davinci8 --device /dev/davinci9 --device /dev/davinci10 --device /dev/davinci11
|
||||
--device /dev/davinci12 --device /dev/davinci13 --device /dev/davinci14 --device /dev/davinci15
|
||||
--device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc
|
||||
)
|
||||
|
||||
VOL_ARGS=(
|
||||
-v /usr/local/dcmi:/usr/local/dcmi
|
||||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi
|
||||
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info
|
||||
-v /etc/hccn.conf:/etc/hccn.conf
|
||||
-v "${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||
-v "${RUNTIME_BASE}/tmp:/tmp"
|
||||
-v /mnt:/mnt
|
||||
)
|
||||
|
||||
ENV_ARGS=(
|
||||
-e ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES}"
|
||||
-e PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
-e PYTHONUNBUFFERED=1
|
||||
-e HF_HUB_OFFLINE=1
|
||||
-e TRANSFORMERS_OFFLINE=1
|
||||
-e OMP_PROC_BIND=false
|
||||
-e OMP_NUM_THREADS=10
|
||||
-e HCCL_BUFFSIZE=1024
|
||||
-e VLLM_ASCEND_ENABLE_FLASHCOMM1=1
|
||||
-e TASK_QUEUE_ENABLE=1
|
||||
-e "HCCL_OP_EXPANSION_MODE=AIV"
|
||||
-e "LD_PRELOAD=/usr/lib64/libjemalloc.so.2"
|
||||
)
|
||||
|
||||
if [[ "${DRY_RUN:-0}" == "1" ]]; then
|
||||
echo "[DRY-RUN] docker run --rm --name \"$NAME\" --privileged --ipc host --shm-size 512g \\"
|
||||
echo " --network host ${DEVICE_ARGS[*]} ${VOL_ARGS[*]} ${ENV_ARGS[*]} \\"
|
||||
echo " \"$IMAGE\" $SERVER_ARGS_STR"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
nohup docker run --rm \
|
||||
--name "$NAME" \
|
||||
--privileged \
|
||||
--ipc host \
|
||||
--shm-size 512g \
|
||||
--network host \
|
||||
--ulimit memlock=-1 \
|
||||
"${DEVICE_ARGS[@]}" \
|
||||
"${VOL_ARGS[@]}" \
|
||||
"${ENV_ARGS[@]}" \
|
||||
"$IMAGE" \
|
||||
vllm serve "${SERVER_ARGS[@]}" \
|
||||
> "$LOG" 2>&1 &
|
||||
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
echo "Waiting for health on port ${PORT}..."
|
||||
|
||||
# Health-check budget. DSV4 weight load (280 GiB) + graph compile takes much
|
||||
# longer than GLM-5.2; TP=16 compiles 16 graphs (~60min). Allow overrides.
|
||||
# Synced from glm52 fix: 480 x 5s = 40min default.
|
||||
HEALTH_MAX_RETRIES="${HEALTH_MAX_RETRIES:-480}"
|
||||
HEALTH_RETRY_INTERVAL_S="${HEALTH_RETRY_INTERVAL_S:-5}"
|
||||
|
||||
for i in $(seq 1 "$HEALTH_MAX_RETRIES"); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
|
||||
echo "vLLM-Ascend server is ready at http://127.0.0.1:${PORT}"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
if ! kill -0 $PID 2>/dev/null; then
|
||||
echo "ERROR: Docker vLLM-Ascend server exited early"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/$HEALTH_MAX_RETRIES)"
|
||||
sleep "$HEALTH_RETRY_INTERVAL_S"
|
||||
done
|
||||
|
||||
echo "ERROR: vLLM-Ascend server not healthy after $HEALTH_MAX_RETRIES retries"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
log "starting vLLM-Ascend server tp=${TP} dp=${DP} via deploy profile"
|
||||
deploy_start \
|
||||
"${DEPLOY_PROFILE:-910c/dsv4_910c_vllm_tp_dp_matrix}" \
|
||||
"$TP" "$DP" \
|
||||
"${RUNTIME_BASE}/logs" \
|
||||
"${VLLM_PORT:-30052}" \
|
||||
"$MODEL_PATH" \
|
||||
""
|
||||
|
||||
39
experiments/910c/dsv4_910c_vllm_tp_dp_matrix/stop_vllm_docker.sh
Executable file
39
experiments/910c/dsv4_910c_vllm_tp_dp_matrix/stop_vllm_docker.sh
Executable file
@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stop vLLM-Ascend server for a given TP×DP configuration on 910C through the
|
||||
# shared deployment layer.
|
||||
# Usage: stop_vllm_docker.sh <TP> <DP>
|
||||
set -e
|
||||
|
||||
TP="${1}"
|
||||
DP="${2}"
|
||||
if [[ -z "$TP" || -z "$DP" ]]; then
|
||||
echo "Usage: $0 <TP> <DP>"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Auto-detect docker group access; re-exec under sg docker if needed.
|
||||
if ! docker ps >/dev/null 2>&1; then
|
||||
if sg docker -c "docker ps" >/dev/null 2>&1; then
|
||||
exec sg docker -c "bash ${BASH_SOURCE[0]} ${1:-} ${2:-}"
|
||||
fi
|
||||
echo "ERROR: Cannot access docker. Run 'sudo usermod -aG docker $USER' and re-login." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||
|
||||
log "stopping vLLM-Ascend server tp=${TP} dp=${DP} via deploy profile"
|
||||
deploy_stop \
|
||||
"${DEPLOY_PROFILE:-910c/dsv4_910c_vllm_tp_dp_matrix}" \
|
||||
"$TP" "$DP" \
|
||||
"${VLLM_PORT:-30052}" \
|
||||
"$MODEL_PATH" \
|
||||
""
|
||||
@ -29,12 +29,12 @@ CONTAINER_PYTHON="/usr/local/python3.12.13/bin/python3"
|
||||
USE_DOCKER="${USE_DOCKER:-1}"
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-local/vllm-ascend:0.23-a3-20260718-sglang}"
|
||||
|
||||
# Benchmark client Docker image. vLLM's image does not include sglang.bench_serving;
|
||||
# reuse the vllm-ascend container itself for the client via `docker exec` (see
|
||||
# run_adaptive_concurrency_add16.sh), so this is only used if USE_DOCKER_CLIENT=1
|
||||
# with an external sglang image. Default off on 910c.
|
||||
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
|
||||
USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-0}"
|
||||
# Benchmark client Docker image. The vllm-ascend images built on this host
|
||||
# (local/vllm-ascend:0.23-a3-*-sglang, see envs/ASCEND_910C_ENV_SETUP.md §8.1)
|
||||
# carry sglang 0.5.2 bench_serving, so the client runs as a standalone container
|
||||
# (TORCH_DEVICE_BACKEND_AUTOLOAD=0 injected by the runner, no NPU needed).
|
||||
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-local/vllm-ascend:0.23-a3-20260718-sglang}"
|
||||
USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-1}"
|
||||
|
||||
# Device selection. ASCEND_VISIBLE_DEVICES selects NPU cards 0..7; the Ascend
|
||||
# Docker Runtime (default runtime on this host) injects the matching dies.
|
||||
|
||||
@ -15,6 +15,10 @@ source "${SCRIPT_DIR}/config.env"
|
||||
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||
|
||||
DEPLOY_PROFILE="${DEPLOY_PROFILE:-910c/glm52_910c_vllm_tp_dp_matrix}"
|
||||
|
||||
ENGINE="vllm"
|
||||
ENGINE_PORT="$VLLM_PORT"
|
||||
@ -38,20 +42,9 @@ engine_is_healthy() {
|
||||
engine_stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
|
||||
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping vllm server pid=${pid} tp=${tp} dp=${dp}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
log "stopping vllm server tp=${tp} dp=${dp} via deploy profile"
|
||||
bash "${SCRIPT_DIR}/stop_vllm_docker.sh" "$tp" "$dp" >> "${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1 || true
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
sleep 2
|
||||
}
|
||||
@ -59,23 +52,7 @@ engine_stop_server() {
|
||||
engine_build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local -a args=(
|
||||
vllm serve "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$tp"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--no-enable-flashinfer-autotune
|
||||
--host 0.0.0.0
|
||||
--port "$ENGINE_PORT"
|
||||
)
|
||||
if (( dp > 1 )); then
|
||||
args+=(--data-parallel-size "$dp")
|
||||
fi
|
||||
printf '%q ' "${args[@]}"
|
||||
deploy_render_args "$DEPLOY_PROFILE" "$tp" "$dp" "$ENGINE_PORT" "$MODEL_PATH"
|
||||
}
|
||||
|
||||
engine_start_server() {
|
||||
|
||||
@ -24,6 +24,10 @@ source "${SCRIPT_DIR}/config.env"
|
||||
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||
|
||||
DEPLOY_PROFILE="${DEPLOY_PROFILE:-910c/glm52_910c_vllm_tp_dp_matrix}"
|
||||
|
||||
ENGINE="vllm"
|
||||
ENGINE_PORT="$VLLM_PORT"
|
||||
@ -52,21 +56,9 @@ engine_is_healthy() {
|
||||
engine_stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping vllm-ascend server pid=${pid} tp=${tp} dp=${dp}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
# Also remove the container by name in case the host process is gone but the
|
||||
# container lingers (e.g. orphaned by a kill -9).
|
||||
docker rm -f "${CONTAINER_NAME}_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
|
||||
log "stopping vllm-ascend server tp=${tp} dp=${dp} via deploy profile"
|
||||
bash "${SCRIPT_DIR}/stop_vllm_docker.sh" "$tp" "$dp" >> "${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1 || true
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
sleep 2
|
||||
}
|
||||
@ -97,40 +89,8 @@ engine_build_server_args() {
|
||||
;;
|
||||
esac
|
||||
log "server args tp=${tp} dp=${dp} gpu_mem_util=${mem_util} max_model_len=${max_len} max_num_seqs=${max_seqs}" >&2
|
||||
# MoE / DSA-CP args required by GLM-5.2 on vllm-ascend (must match
|
||||
# start_vllm_docker.sh). --enable-expert-parallel is mandatory for
|
||||
# FlashComm v1 (VLLM_ASCEND_ENABLE_FLASHCOMM1=1); without it MoE
|
||||
# MoeDistributeDispatchV2 tiling fails at engine init.
|
||||
local compilation_config='{"cudagraph_mode": "FULL_DECODE_ONLY"}'
|
||||
local additional_config='{"enable_dsa_cp": true,"enable_sparse_sfa_c8": false, "enable_sparse_li_c8": true,"enable_balance_scheduling": true,"multistream_overlap_shared_expert":true}'
|
||||
local -a args=(
|
||||
vllm serve "$MODEL_PATH"
|
||||
--served-model-name "$SERVED_MODEL_NAME"
|
||||
--trust-remote-code
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$tp"
|
||||
--enable-expert-parallel
|
||||
--gpu-memory-utilization "$mem_util"
|
||||
--max-model-len "$max_len"
|
||||
--max-num-seqs "$max_seqs"
|
||||
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS"
|
||||
--api-server-count "$API_SERVER_COUNT"
|
||||
--compilation-config "$compilation_config"
|
||||
--additional-config "$additional_config"
|
||||
--host 0.0.0.0
|
||||
--port "$ENGINE_PORT"
|
||||
)
|
||||
if (( dp > 1 )); then
|
||||
args+=(--data-parallel-size "$dp")
|
||||
fi
|
||||
# Emit args space-separated. JSON config strings are single-quoted
|
||||
# verbatim (not %q-escaped) so the recorded command stays re-runnable.
|
||||
local out=""
|
||||
local a
|
||||
for a in "${args[@]}"; do
|
||||
out+="'$a' "
|
||||
done
|
||||
printf '%s' "$out"
|
||||
export GPU_MEMORY_UTILIZATION="$mem_util" MAX_MODEL_LEN="$max_len" MAX_NUM_SEQS="$max_seqs"
|
||||
deploy_render_args "$DEPLOY_PROFILE" "$tp" "$dp" "$ENGINE_PORT" "$MODEL_PATH"
|
||||
}
|
||||
|
||||
engine_start_server() {
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user