Merge branch 'main' of https://gitee.com/yy-fighting/sskj
# Conflicts: # experiments/dsv4_p800_long_context_matrix/README.md # experiments/dsv4_p800_long_context_matrix/config.env # experiments/dsv4_p800_long_context_matrix/run_bench.sh
This commit is contained in:
yy-fighting
commit
18ded8e15d
15
.gitignore
vendored
15
.gitignore
vendored
@ -1,5 +1,11 @@
|
||||
# 环境/依赖/临时文件
|
||||
envs/
|
||||
# envs/ 目录下的虚拟环境子目录（具体环境文件不提交，但文档保留）
|
||||
envs/modelscope-hub/
|
||||
envs/modelscope-upload/
|
||||
envs/sglang/
|
||||
envs/sglang-src/
|
||||
envs/vllm/
|
||||
# 其他通用忽略
|
||||
deps/
|
||||
tmp/
|
||||
*.pid
|
||||
@ -16,7 +22,7 @@ __pycache__/
|
||||
# 中间日志
|
||||
logs/
|
||||
*.log
|
||||
|
||||
*.csv
|
||||
# 数据集（可重新下载）
|
||||
datasets/
|
||||
|
||||
@ -29,6 +35,11 @@ experiments/*/results/*/raw_outputs/
|
||||
experiments/*/results/*/logs/
|
||||
experiments/*/results/*/*/raw_outputs/
|
||||
experiments/*/results/*/*/logs/
|
||||
experiments/*/results/*/*.tsv
|
||||
experiments/*/results/*/*/*.tsv
|
||||
|
||||
# 运行时目录（pid 文件、缓存、临时文件）
|
||||
experiments/*/runtime/
|
||||
|
||||
# 无关项目
|
||||
loomeval_yy/
|
||||
|
||||
@ -83,6 +83,12 @@
|
||||
- This is required for cross-platform reproduction: when the same experiment is run on H200 and P800, the only differences should be model paths, ports, and device IDs.
|
||||
- If an experiment uses `start_server.sh`, that script should be self-contained and its command line should be reproducible from `results.json` alone.
|
||||
|
||||
10. **Version control: commit code and final artifacts only.**
|
||||
- Always commit the experiment code (`config.env`, `run_bench.sh`, `start_*.sh`, parsers, etc.) together with the run's final artifacts.
|
||||
- Final artifacts to commit: `results.json` (structured data) and `report.md` / `comparison.md` (human-readable summaries).
|
||||
- Do **not** commit intermediate logs, per-request JSONL raw outputs, or GPU sampling CSVs. These are already ignored by `.gitignore` (`raw_outputs/`, `logs/`, `gpu_logs/`).
|
||||
- `scenarios.tsv` and `skipped_after_oom.csv` may be committed if they are useful for reproducing the test plan, but they are optional.
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
### Experiment result directories
|
||||
@ -340,3 +346,4 @@ See `docs/H200_QUICKSTART.md` for a concrete H200 porting walkthrough.
|
||||
- [ ] `<run>/README.md` exists and documents provenance (or the report itself covers provenance).
|
||||
- [ ] Scripts either live under `experiments/<name>/` or in `scripts/` (or `scripts/<group>/`).
|
||||
- [ ] Script path references updated after moving.
|
||||
- [ ] Commits include the experiment code plus final `results.json` and `.md` reports; logs/raw outputs are left ignored.
|
||||
|
||||
82
README.md
82
README.md
@ -7,7 +7,7 @@
|
||||
|
||||
| 目录/文件 | 说明 |
|
||||
|---|---|
|
||||
| `platforms/` | 芯片/加速器平台配置（`*.env`），如 P800、H200 |
|
||||
| `platforms/` | 芯片/加速器平台配置（`*.env`），如 P800、H200、RTX 6000D |
|
||||
| `scripts/common/` | 跨实验复用的 orchestration 组件（server 启停、health check、元数据生成、结果解析） |
|
||||
| `scripts/` | 仅保留仍在使用的 legacy DSpark 脚本与 SLO 标准 |
|
||||
| `experiments/` | 以实验为单位的自包含目录（脚本 + 配置 + 结果） |
|
||||
@ -30,6 +30,19 @@
|
||||
| DSV4 H200 SGLang vs vLLM | `experiments/dsv4_h200_sglang_vs_vllm/run_bench.sh` | NVIDIA H200 上 SGLang 与 vLLM 控制变量对比（TP=8，最长 200k 上下文） |
|
||||
| DSV4 H200 vLLM DSpark vs default | `experiments/dsv4_h200_vllm_dspark_vs_default/run_bench.sh` | vLLM 开启 DSpark 投机解码 vs 默认配置，验证 TTFT 差异 |
|
||||
| DSV4 H200 max context length | `experiments/dsv4_h200_max_context_length/run_bench.sh` | 探索 SGLang / vLLM 在 H200 上能支持的最大输入长度 |
|
||||
| DSV4 H200 SGLang TP/DP Matrix | `experiments/dsv4_h200_sglang_tp_dp_matrix/run_bench.sh` | H200 上 SGLang 不同 TP×DP 配置的固定并发矩阵测试 |
|
||||
| DSV4 H200 SGLang TP/DP Adaptive | `experiments/dsv4_h200_sglang_tp_dp_matrix/run_adaptive_concurrency.sh` | H200 上 SGLang 不同 TP×DP 配置的自适应并发饱和点搜索 |
|
||||
| DSV4 H200 vLLM TP/DP Matrix | `experiments/dsv4_h200_vllm_tp_dp_matrix/run_bench.sh` | H200 上 vLLM 不同 TP×DP 配置的固定并发矩阵测试 |
|
||||
| DSV4 H200 vLLM TP/DP Adaptive | `experiments/dsv4_h200_vllm_tp_dp_matrix/run_adaptive_concurrency.sh` | H200 上 vLLM 不同 TP×DP 配置的自适应并发饱和点搜索 |
|
||||
| DSV4 H200 64k SGLang vs vLLM | `experiments/dsv4_h200_64k_sglang_vs_vllm/run_bench.sh` | H200 上 64k 上下文 SGLang 与 vLLM 对比 |
|
||||
| DSV4 H200 256k 4k Probe | `experiments/dsv4_h200_256k_4k_probe/run_bench.sh` | H200 上 256k 输入/4k 输出长上下文探测 |
|
||||
| DSV4 H200 vLLM TP2 Custom Bench | `experiments/dsv4_h200_vllm_tp2_custom_bench/run_bench.sh` | H200 上 vLLM TP=2 自定义压测客户端（多服务负载均衡） |
|
||||
| DSV4 H200 vLLM TP4 Custom Bench | `experiments/dsv4_h200_vllm_tp4_custom_bench/run_bench.sh` | H200 上 vLLM TP=4 自定义压测客户端 |
|
||||
| DSV4 H200 vLLM TP8 Custom Bench | `experiments/dsv4_h200_vllm_tp8_custom_bench/run_bench.sh` | H200 上 vLLM TP=8 自定义压测客户端 |
|
||||
| DSV4 H200 vLLM MTP vs Default | `experiments/dsv4_h200_vllm_mtp_vs_default/run_bench.sh` | H200 上 vLLM MTP（Multi-Token Prediction）vs 默认配置对比 |
|
||||
| DSV4 H200 Long Context Matrix | `experiments/dsv4_h200_long_context_matrix/run_bench.sh` | H200 上长上下文矩阵测试（多后端对比） |
|
||||
| **DSV4 RTX 6000D SGLang TP/DP Adaptive** | `experiments/dsv4_pro6000_sglang_tp_dp_matrix/run_adaptive_concurrency.sh` | **RTX 6000D 上 SGLang 自适应并发饱和点搜索** |
|
||||
| **DSV4 RTX 6000D vLLM TP/DP Adaptive** | `experiments/dsv4_pro6000_vllm_tp_dp_matrix/run_adaptive_concurrency.sh` | **RTX 6000D 上 vLLM 自适应并发饱和点搜索** |
|
||||
|
||||
### 旧结构（scripts/ + bench_results/）
|
||||
|
||||
@ -72,10 +85,72 @@ bash experiments/dsv4_h200_sglang_vs_vllm/run_bench.sh
|
||||
bash experiments/dsv4_h200_vllm_dspark_vs_default/run_bench.sh
|
||||
```
|
||||
|
||||
### H200 max context length
|
||||
### H200 SGLang TP/DP Matrix
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_max_context_length/run_bench.sh
|
||||
bash experiments/dsv4_h200_sglang_tp_dp_matrix/run_bench.sh
|
||||
```
|
||||
|
||||
### H200 SGLang TP/DP 自适应并发搜索
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_sglang_tp_dp_matrix/run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
### H200 vLLM TP/DP Matrix
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_vllm_tp_dp_matrix/run_bench.sh
|
||||
```
|
||||
|
||||
### H200 vLLM TP/DP 自适应并发搜索
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_vllm_tp_dp_matrix/run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
### H200 64k SGLang vs vLLM
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_64k_sglang_vs_vllm/run_bench.sh
|
||||
```
|
||||
|
||||
### H200 256k 4k Probe
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_256k_4k_probe/run_bench.sh
|
||||
```
|
||||
|
||||
### H200 vLLM TP2/TP4/TP8 Custom Bench
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_vllm_tp2_custom_bench/run_bench.sh
|
||||
bash experiments/dsv4_h200_vllm_tp4_custom_bench/run_bench.sh
|
||||
bash experiments/dsv4_h200_vllm_tp8_custom_bench/run_bench.sh
|
||||
```
|
||||
|
||||
### H200 vLLM MTP vs Default
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_vllm_mtp_vs_default/run_bench.sh
|
||||
```
|
||||
|
||||
### H200 Long Context Matrix
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_h200_long_context_matrix/run_bench.sh
|
||||
```
|
||||
|
||||
### RTX 6000D SGLang TP/DP 自适应并发搜索
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_pro6000_sglang_tp_dp_matrix/run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
### RTX 6000D vLLM TP/DP 自适应并发搜索
|
||||
|
||||
```bash
|
||||
bash experiments/dsv4_pro6000_vllm_tp_dp_matrix/run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
### DSpark grid / spec-tokens（旧结构）
|
||||
@ -109,6 +184,7 @@ python3 experiments/dsv4_h200_dspark/parse_results.py \
|
||||
## 环境要求
|
||||
|
||||
- **NVIDIA H200**：Python env `/data/user1/yy/envs/vllm-dspark`（服务端）、`/data/user1/yy/envs/sglang`（压测客户端）；模型 `/data/models/DeepSeek-V4-Flash`、`/data/models/DeepSeek-V4-Flash-DSpark`。
|
||||
- **NVIDIA RTX 6000D**：Python env `/root/.miniconda3/envs/sglang`（压测客户端）；模型 `/data/hf_models/DeepSeek-V4-Flash`；Docker 镜像 `lmsysorg/sglang:latest`、`vllm/vllm-openai:latest`。
|
||||
- **Kunlun P800**：Docker 镜像 `iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202`；模型 `/data1/models/DeepSeek-V4-Flash-INT8`；压测客户端在容器内运行。
|
||||
|
||||
平台相关常量见 `platforms/*.env`。
|
||||
|
||||
@ -155,14 +155,31 @@ git commit -m "results: <experiment_name> run <run_id>"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
## 7. 命名约定
|
||||
## 7. 场景设计规范
|
||||
|
||||
### 7.1 请求数 = 并发数 × 5
|
||||
|
||||
每个 scenario 的 `num_prompts` 必须是对应 `concurrency` 的 **5 倍**，即每个并发档位至少跑满 5 轮请求，保证 P95/P99 等尾延迟指标有统计意义。
|
||||
|
||||
```bash
|
||||
# 格式：concurrency input_len output_len num_prompts
|
||||
declare -a SCENARIOS=(
|
||||
"1 512 256 5" # 1 × 5
|
||||
"32 512 256 160" # 32 × 5
|
||||
"128 512 256 640" # 128 × 5
|
||||
)
|
||||
```
|
||||
|
||||
例外：对于超长上下文（如 128k/256k）或探针类实验，若执行成本过高，可在 `config.env` 中显式注释说明原因，并保留不小于 `concurrency × 2` 的最低样本量。
|
||||
|
||||
## 8. 命名约定
|
||||
|
||||
- 实验目录：`{model}_{chip}_{scenario}_{backend_vs_backend}`，例如 `dsv4_h200_sglang_vs_vllm`
|
||||
- scenario 名称：`c{concurrency}_i{input_len}_o{output_len}`
|
||||
- raw 输出：`{backend}_{label}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl`
|
||||
- run_id：`YYYYMMDD-HHMMSS`
|
||||
|
||||
## 8. 常见问题
|
||||
## 9. 常见问题
|
||||
|
||||
### 8.1 新芯片上 server 起不来
|
||||
|
||||
|
||||
42
envs/README.md
Normal file
42
envs/README.md
Normal file
@ -0,0 +1,42 @@
|
||||
# envs 目录说明
|
||||
|
||||
本目录用于存放**环境搭建相关的文档和指南**，不包括具体的虚拟环境或容器镜像文件。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
envs/
|
||||
├── README.md # 本说明文件
|
||||
├── UV_ENV_SETUP.md # uv 虚拟环境搭建规范
|
||||
├── SM120_DSV4_DEPLOYMENT_GUIDE.md # SM120 (RTX 6000D) 部署 DSV4 指南
|
||||
└── <future guides> # 其他平台/环境的部署文档
|
||||
```
|
||||
|
||||
## 使用规范
|
||||
|
||||
1. **文档放这里**：与建立环境、部署模型、修复平台兼容性相关的文档，应放到 `envs/` 目录下。
|
||||
2. **环境不放这里**：具体的 uv venv、conda env、Docker 镜像构建产物等，**不应**提交到 git。请在对应环境的 `.gitignore` 中排除。
|
||||
3. **命名规范**：`{平台}_{模型或框架}_DEPLOYMENT_GUIDE.md` 或 `{框架}_{平台}_FIX.md`。
|
||||
4. **内容要求**：
|
||||
- 环境信息（GPU、CUDA、驱动版本）
|
||||
- 问题现象与错误日志
|
||||
- 根因分析
|
||||
- 解决步骤（含具体命令或代码修改）
|
||||
- 验证结果
|
||||
|
||||
## 现有文档
|
||||
|
||||
| 文档 | 说明 |
|
||||
|---|---|
|
||||
| `UV_ENV_SETUP.md` | uv 虚拟环境搭建规范，含 vLLM 和 SGLang 的标准安装命令 |
|
||||
| `SM120_DSV4_DEPLOYMENT_GUIDE.md` | RTX 6000D (SM120) 上部署 DeepSeek-V4-Flash 的完整指南，含 vLLM 和 SGLang 的适配步骤 |
|
||||
|
||||
## 排除的环境目录（已加入 .gitignore）
|
||||
|
||||
以下子目录包含具体的虚拟环境文件，不应提交：
|
||||
|
||||
- `modelscope-hub/`
|
||||
- `modelscope-upload/`
|
||||
- `sglang/`
|
||||
- `sglang-src/`
|
||||
- `vllm/`
|
||||
235
envs/SM120_DSV4_DEPLOYMENT_GUIDE.md
Normal file
235
envs/SM120_DSV4_DEPLOYMENT_GUIDE.md
Normal file
@ -0,0 +1,235 @@
|
||||
# SM120 (RTX 6000D) 部署 DeepSeek-V4-Flash 问题总结
|
||||
|
||||
## 环境信息
|
||||
|
||||
- **GPU**: NVIDIA RTX 6000D (SM120, 48GB x 8)
|
||||
- **模型**: DeepSeek-V4-Flash (FP8 量化)
|
||||
- **模型路径**: `/data/hf_models/DeepSeek-V4-Flash`
|
||||
- **sglang 版本**: 0.5.15
|
||||
- **vllm 版本**: 0.11.0+cu128 (with SM120 patches)
|
||||
- **flashinfer**: 0.6.14
|
||||
- **torch**: 2.8.0+cu128
|
||||
|
||||
---
|
||||
|
||||
## 问题概述
|
||||
|
||||
sglang 和 vllm 在 SM120 上部署 DSV4 时，都遇到了**算子不支持**的问题。根本原因是：
|
||||
|
||||
1. **flashinfer** 的某些算子尚未支持 SM120
|
||||
2. **sgl_kernel** 的 `flash_mla_sparse_fwd` 只支持 SM90a 和 SM100f，不支持 SM120
|
||||
|
||||
---
|
||||
|
||||
## vllm 部署问题与修复
|
||||
|
||||
### 问题：flashinfer API 参数名不匹配
|
||||
|
||||
vllm 使用 flashinfer 的 `BatchDecodeWithPagedKVCacheWrapper` 时，调用参数名与 flashinfer 0.6.14 实际接口不一致。
|
||||
|
||||
**错误信息**:
|
||||
```
|
||||
TypeError: compute_kv_indptr() got an unexpected keyword argument 'page_size'
|
||||
```
|
||||
|
||||
**根本原因**: vllm 代码中使用的参数名是 `page_size`，但 flashinfer 0.6.14 实际需要的是 `kv_layout` 或其他参数。
|
||||
|
||||
### 修复方法
|
||||
|
||||
修改文件：`/data/yy/sskj/envs/vllm/lib/python3.12/site-packages/vllm/attention/ops/flashinfer_sm120.py`
|
||||
|
||||
将 `page_size` 参数改为 `kv_layout`:
|
||||
|
||||
```python
|
||||
# 修改前
|
||||
indptr = flashinfer.compute_kv_indptr(
|
||||
num_pages,
|
||||
page_size=page_size, # 错误
|
||||
...
|
||||
)
|
||||
|
||||
# 修改后
|
||||
indptr = flashinfer.compute_kv_indptr(
|
||||
num_pages,
|
||||
kv_layout=page_size, # 正确
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
### vllm 部署命令
|
||||
|
||||
```bash
|
||||
source /data/yy/sskj/envs/vllm/bin/activate
|
||||
python -m vllm.entrypoints.openai.api_server \
|
||||
--model /data/hf_models/DeepSeek-V4-Flash \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 65536 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
**状态**: ✅ 已修复，可成功部署
|
||||
|
||||
---
|
||||
|
||||
## sglang 部署问题与修复
|
||||
|
||||
### 问题1：flashinfer 版本过低
|
||||
|
||||
sglang 0.5.15 默认安装的 flashinfer 0.6.12 缺少 SM120 支持。需要升级到 0.6.14。
|
||||
|
||||
**修复**:
|
||||
```bash
|
||||
cp /data/yy/sskj/envs/vllm/lib/python3.12/site-packages/flashinfer* \
|
||||
/data/yy/sskj/envs/sglang/lib/python3.12/site-packages/
|
||||
```
|
||||
|
||||
### 问题2：sgl_kernel flash_mla_sparse_fwd 不支持 SM120
|
||||
|
||||
sglang 的 DSV4 attention backend 在 prefill 阶段有两个路径：
|
||||
|
||||
1. **sparse prefill**: 使用 `sgl_kernel.flash_mla.flash_mla_sparse_fwd`，**只支持 SM90a 和 SM100f**
|
||||
2. **非 sparse prefill**: 使用 `flash_mla_with_kvcache_sm120`，**支持 SM120**
|
||||
|
||||
**触发 sparse prefill 的条件** (`deepseek_v4_backend.py:1405`):
|
||||
```python
|
||||
if forward_batch.forward_mode.is_extend_without_speculative() and (
|
||||
q.shape[0] > _LARGE_INDEXER_QUERY_THRESHOLD # 11673
|
||||
or envs.SGLANG_OPT_FLASHMLA_SPARSE_PREFILL.get()
|
||||
):
|
||||
return self._forward_prefill_sparse(...)
|
||||
```
|
||||
|
||||
warmup 阶段的 token 数超过 11673 阈值，导致触发 sparse prefill，然后报错：
|
||||
```
|
||||
RuntimeError: flash_mla_sparse_fwd only supports SM90a and SM100f
|
||||
```
|
||||
|
||||
### 修复方法
|
||||
|
||||
修改文件：`/data/yy/sskj/envs/sglang/lib/python3.12/site-packages/sglang/srt/layers/attention/deepseek_v4_backend.py`
|
||||
|
||||
在 sparse prefill 条件判断中加入 `not _is_sm120`:
|
||||
|
||||
```python
|
||||
# 修改前 (第1405行)
|
||||
if forward_batch.forward_mode.is_extend_without_speculative() and (
|
||||
q.shape[0] > _LARGE_INDEXER_QUERY_THRESHOLD
|
||||
or envs.SGLANG_OPT_FLASHMLA_SPARSE_PREFILL.get()
|
||||
):
|
||||
|
||||
# 修改后
|
||||
if forward_batch.forward_mode.is_extend_without_speculative() and not _is_sm120 and (
|
||||
q.shape[0] > _LARGE_INDEXER_QUERY_THRESHOLD
|
||||
or envs.SGLANG_OPT_FLASHMLA_SPARSE_PREFILL.get()
|
||||
):
|
||||
```
|
||||
|
||||
这样 SM120 上就会走下面的 `flash_mla_with_kvcache_sm120` 路径，而不是 `flash_mla_sparse_fwd`。
|
||||
|
||||
### sglang 部署命令
|
||||
|
||||
```bash
|
||||
source /data/yy/sskj/envs/sglang/bin/activate
|
||||
python -m sglang.launch_server \
|
||||
--model-path /data/hf_models/DeepSeek-V4-Flash \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--mem-fraction-static 0.78 \
|
||||
--cuda-graph-backend-decode disabled \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**状态**: ✅ 已修复，可成功部署
|
||||
|
||||
---
|
||||
|
||||
## prefill 阶段算子选项分析
|
||||
|
||||
| 算子 | 来源 | 支持架构 | 说明 |
|
||||
|------|------|----------|------|
|
||||
| `flash_mla_sparse_fwd` | `sgl_kernel.flash_mla` | SM90a, SM100f | sparse prefill，性能更优，**不支持 SM120** |
|
||||
| `flash_mla_with_kvcache` | `sgl_kernel.flash_mla` | 非 SM120 GPU | 标准 prefill |
|
||||
| `flash_mla_with_kvcache_sm120` | `sglang.srt.layers.attention.flash_mla_sm120` | SM120 | SM120 专用路径 |
|
||||
|
||||
**结论**: 对于 SM120，prefill 阶段目前只有一个可用选项：`flash_mla_with_kvcache_sm120`。
|
||||
|
||||
sparse prefill 的优化（`flash_mla_sparse_fwd`）目前不支持 SM120，需要等待 `sgl_kernel` 更新。
|
||||
|
||||
---
|
||||
|
||||
## 是否会出现在 B300/B200 上
|
||||
|
||||
- **B200**: SM100，应该支持 `flash_mla_sparse_fwd`（支持 SM100f）
|
||||
- **B300**: 需要确认具体的 SM 版本。如果是 SM120 或更高，可能会遇到同样的问题
|
||||
|
||||
建议在这些 GPU 上部署前先检查：
|
||||
```python
|
||||
import torch
|
||||
print(torch.cuda.get_device_capability()) # 查看 SM 版本
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 相关 GitHub Issue
|
||||
|
||||
### vllm
|
||||
- [vllm-project/vllm#XXXX](https://github.com/vllm-project/vllm/issues) - flashinfer SM120 支持
|
||||
|
||||
### sglang
|
||||
- [sgl-project/sglang#XXXX](https://github.com/sgl-project/sglang/issues) - sgl_kernel SM120 支持
|
||||
|
||||
建议关注以下 PR/Issue：
|
||||
- flashinfer 官方 SM120 支持进度
|
||||
- sgl_kernel 官方 SM120 支持进度
|
||||
|
||||
---
|
||||
|
||||
## 长期解决方案
|
||||
|
||||
1. **等待官方更新**: 等待 flashinfer 和 sgl_kernel 官方支持 SM120
|
||||
2. **从源码编译**: 从 sglang 源码编译 sgl_kernel，添加 SM120 支持
|
||||
- 仓库: https://gitee.com/yy-fighting/sglang.git
|
||||
- 需要修改 CUDA 编译选项，添加 `sm120` 架构
|
||||
3. **使用 triton 算子**: 如果 CUDA 算子不支持，可以尝试使用 triton 实现的替代算子
|
||||
|
||||
---
|
||||
|
||||
## 当前部署状态
|
||||
|
||||
| 框架 | 状态 | 修改文件 |
|
||||
|------|------|----------|
|
||||
| vllm | ✅ 可部署 | `vllm/attention/ops/flashinfer_sm120.py` |
|
||||
| sglang | ✅ 可部署 | `sglang/srt/layers/attention/deepseek_v4_backend.py` |
|
||||
|
||||
---
|
||||
|
||||
## 测试验证
|
||||
|
||||
### vllm API 测试
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "DeepSeek-V4-Flash", "prompt": "Hello", "max_tokens": 10}'
|
||||
```
|
||||
|
||||
### sglang API 测试
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:30000/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"text": "Hello, how are you?", "sampling_params": {"max_new_tokens": 10}}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 备注
|
||||
|
||||
- 以上修改都是临时 workaround，建议跟踪上游官方修复
|
||||
- 如果升级 sglang 或 vllm 版本，可能需要重新应用这些修改
|
||||
- 性能方面：SM120 上禁用 sparse prefill 可能会有轻微性能损失，但可以正常运行
|
||||
|
||||
---
|
||||
|
||||
*文档生成时间: 2026-07-14*
|
||||
164
envs/UV_ENV_SETUP.md
Normal file
164
envs/UV_ENV_SETUP.md
Normal file
@ -0,0 +1,164 @@
|
||||
# UV 虚拟环境搭建规范
|
||||
|
||||
本文档说明如何使用 [uv](https://docs.astral.sh/uv/) 在本项目下搭建 vLLM 和 SGLang 的虚拟环境。
|
||||
|
||||
## 前置要求
|
||||
|
||||
- Python 3.10+
|
||||
- CUDA 驱动已正确安装
|
||||
|
||||
## 通用设置
|
||||
|
||||
### 1. 安装 uv
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
```
|
||||
|
||||
### 2. 设置 uv cache 目录
|
||||
|
||||
将 uv 的缓存目录设置到 `envs/` 下，避免占用用户主目录空间：
|
||||
|
||||
```bash
|
||||
export UV_CACHE_DIR="/data/yy/sskj/envs/.uv_cache"
|
||||
# 建议写入 ~/.bashrc 或 ~/.zshrc 持久化
|
||||
```
|
||||
|
||||
> **注意**：`envs/.uv_cache/` 已加入 `.gitignore`，不会被提交。
|
||||
|
||||
### 3. 创建虚拟环境
|
||||
|
||||
```bash
|
||||
cd /data/yy/sskj/envs
|
||||
|
||||
# vLLM 环境
|
||||
uv venv vllm --python 3.12
|
||||
|
||||
# SGLang 环境
|
||||
uv venv sglang --python 3.12
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## vLLM 环境安装
|
||||
|
||||
```bash
|
||||
source /data/yy/sskj/envs/vllm/bin/activate
|
||||
|
||||
# 安装 vLLM（自动匹配 PyTorch CUDA 后端）
|
||||
uv pip install vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
> `--torch-backend=auto` 让 uv 自动选择匹配当前 CUDA 版本的 PyTorch 后端。
|
||||
|
||||
---
|
||||
|
||||
## SGLang 环境安装
|
||||
|
||||
### 标准 CUDA 版本（如 cu128）
|
||||
|
||||
```bash
|
||||
source /data/yy/sskj/envs/sglang/bin/activate
|
||||
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
|
||||
# 安装 SGLang（允许预发布版本）
|
||||
uv pip install --prerelease=allow sglang
|
||||
```
|
||||
|
||||
### CUDA 12.9 (cu129) 特殊处理
|
||||
|
||||
如果目标平台使用 CUDA 12.9，需要强制重新安装匹配 cu129 的 PyTorch 和 SGLang 内核：
|
||||
|
||||
```bash
|
||||
source /data/yy/sskj/envs/sglang/bin/activate
|
||||
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
|
||||
# 1. 安装 SGLang
|
||||
uv pip install --prerelease=allow sglang
|
||||
|
||||
# 2. 强制重新安装 cu129 版 PyTorch
|
||||
uv pip install --force-reinstall \
|
||||
torch==2.11.0 torchaudio==2.11.0 torchvision \
|
||||
--index-url https://download.pytorch.org/whl/cu129
|
||||
|
||||
# 3. 强制重新安装 cu129 版 SGLang 内核
|
||||
uv pip install --force-reinstall sglang-kernel \
|
||||
--index-url https://docs.sglang.ai/whl/cu129/
|
||||
|
||||
# 4. 强制重新安装 cu129 版 deep gemm（无依赖）
|
||||
uv pip install --force-reinstall sgl-deep-gemm \
|
||||
--index-url https://docs.sglang.ai/whl/cu129/ \
|
||||
--no-deps
|
||||
```
|
||||
|
||||
> `--force-reinstall` 确保覆盖默认安装的 CUDA 版本。
|
||||
> `--no-deps` 避免 `sgl-deep-gemm` 拉取不兼容的依赖。
|
||||
|
||||
---
|
||||
|
||||
## 验证安装
|
||||
|
||||
### vLLM
|
||||
|
||||
```bash
|
||||
source /data/yy/sskj/envs/vllm/bin/activate
|
||||
python -c "import vllm; print(vllm.__version__)"
|
||||
```
|
||||
|
||||
### SGLang
|
||||
|
||||
```bash
|
||||
source /data/yy/sskj/envs/sglang/bin/activate
|
||||
python -c "import sglang; print(sglang.__version__)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 环境目录结构（参考）
|
||||
|
||||
```
|
||||
envs/
|
||||
├── .uv_cache/ # uv 缓存（已加入 .gitignore）
|
||||
├── vllm/ # vLLM 虚拟环境（已加入 .gitignore）
|
||||
│ ├── bin/
|
||||
│ ├── lib/
|
||||
│ └── ...
|
||||
├── sglang/ # SGLang 虚拟环境（已加入 .gitignore）
|
||||
│ ├── bin/
|
||||
│ ├── lib/
|
||||
│ └── ...
|
||||
├── README.md # 本目录说明
|
||||
├── UV_ENV_SETUP.md # 本文档
|
||||
└── SM120_DSV4_DEPLOYMENT_GUIDE.md # 平台特定部署指南
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 常见问题
|
||||
|
||||
### uv cache 占用过大
|
||||
|
||||
```bash
|
||||
# 清理 uv 缓存
|
||||
uv cache clean
|
||||
```
|
||||
|
||||
### 切换 CUDA 版本后 PyTorch 不匹配
|
||||
|
||||
```bash
|
||||
# 先卸载再重新安装对应 CUDA 版本的 PyTorch
|
||||
uv pip uninstall torch torchaudio torchvision
|
||||
uv pip install torch==<version> --index-url https://download.pytorch.org/whl/cu<version>
|
||||
```
|
||||
|
||||
### 环境激活脚本路径
|
||||
|
||||
| 环境 | 激活命令 |
|
||||
|---|---|
|
||||
| vLLM | `source /data/yy/sskj/envs/vllm/bin/activate` |
|
||||
| SGLang | `source /data/yy/sskj/envs/sglang/bin/activate` |
|
||||
128
experiments/ADAPTIVE_CONCURRENCY_USAGE.md
Normal file
128
experiments/ADAPTIVE_CONCURRENCY_USAGE.md
Normal file
@ -0,0 +1,128 @@
|
||||
# 自适应并发 Benchmark 使用说明
|
||||
|
||||
这套脚本对每个固定的 `(TP, DP, ISL, OSL)` 依次测试
|
||||
`C=1,2,4,8,...`，直到 Total TPS 连续两次增长不足 2%，或达到并发上限、
|
||||
发生 OOM。每个正式测点发送 `C * 5` 个请求，并先用相同并发 C warm-up。
|
||||
|
||||
## 1. 进入目录
|
||||
|
||||
vLLM：
|
||||
|
||||
```bash
|
||||
cd /data3/yy/sskj/experiments/dsv4_h200_vllm_tp_dp_matrix
|
||||
```
|
||||
|
||||
SGLang：
|
||||
|
||||
```bash
|
||||
cd /data3/yy/sskj/experiments/dsv4_h200_sglang_tp_dp_matrix
|
||||
```
|
||||
|
||||
两个目录的入口都是：
|
||||
|
||||
```bash
|
||||
bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
## 2. 先 dry-run
|
||||
|
||||
只打印 TP/DP、ISL/OSL 和并发搜索计划，不加载模型：
|
||||
|
||||
```bash
|
||||
DRY_RUN=1 bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
## 3. 单组合 smoke
|
||||
|
||||
只测 `TP=8, DP=1, ISL=1K, OSL=128`，并发搜索到 8：
|
||||
|
||||
```bash
|
||||
RUN_ID=smoke-$(date +%Y%m%d-%H%M%S) \
|
||||
TP_LIST="8" \
|
||||
ISL_LIST="1024" \
|
||||
OSL_LIST="128" \
|
||||
GRID_LIMIT=1 \
|
||||
SEARCH_MAX_CONCURRENCY=8 \
|
||||
RESULT_BASE=/data3/yy/sskj_adaptive_results/smoke \
|
||||
bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
## 4. 正式运行
|
||||
|
||||
长任务放进 tmux。vLLM 示例：
|
||||
|
||||
```bash
|
||||
tmux new-session -d -s vllm-adaptive \
|
||||
"cd /data3/yy/sskj/experiments/dsv4_h200_vllm_tp_dp_matrix && \
|
||||
RUN_ID=vllm-\$(date +%Y%m%d-%H%M%S) \
|
||||
RESULT_BASE=/data3/yy/sskj_adaptive_results/vllm \
|
||||
bash run_adaptive_concurrency.sh"
|
||||
```
|
||||
|
||||
SGLang 示例：
|
||||
|
||||
```bash
|
||||
tmux new-session -d -s sglang-adaptive \
|
||||
"cd /data3/yy/sskj/experiments/dsv4_h200_sglang_tp_dp_matrix && \
|
||||
RUN_ID=sglang-\$(date +%Y%m%d-%H%M%S) \
|
||||
RESULT_BASE=/data3/yy/sskj_adaptive_results/sglang \
|
||||
bash run_adaptive_concurrency.sh"
|
||||
```
|
||||
|
||||
查看任务：
|
||||
|
||||
```bash
|
||||
tmux ls
|
||||
tmux attach -t vllm-adaptive
|
||||
tmux attach -t sglang-adaptive
|
||||
```
|
||||
|
||||
从 tmux 退出但保持任务运行：按 `Ctrl-b`，松开，再按 `d`。
|
||||
|
||||
## 5. 常用覆盖参数
|
||||
|
||||
无需修改脚本，可以把变量写在命令前：
|
||||
|
||||
```bash
|
||||
MODEL_PATH=/path/to/model \
|
||||
DATASET_PATH=/path/to/ShareGPT.json \
|
||||
DOCKER_IMAGE=your/server:image \
|
||||
DOCKER_CLIENT_IMAGE=lmsysorg/sglang:latest \
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
|
||||
SEARCH_MAX_CONCURRENCY=512 \
|
||||
TPS_MIN_GAIN_PCT=2.0 \
|
||||
PLATEAU_PATIENCE=2 \
|
||||
bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
- vLLM 端口用 `VLLM_PORT`，SGLang 端口用 `SGLANG_PORT`。
|
||||
- `TP_LIST="8"` 只跑 TP=8；DP 使用 `config.env` 中对应的 TP/DP 组合。
|
||||
- `ISL_LIST="1024 4096"`、`OSL_LIST="128 256"` 用来筛选长度。
|
||||
- `GRID_LIMIT=1` 表示每组 TP/DP 只跑一个 ISL/OSL shape。
|
||||
- `BENCH_WARMUP_MAX_REQUESTS=0` 表示每个测点用完整并发 C warm-up；正数表示上限。
|
||||
- 正式基线保持 `BENCH_DATASET_NAME=random` 和 `RANDOM_RANGE_RATIO=1.0`。
|
||||
- `random-ids` 只用于检查链路，不与正式 ShareGPT-backed 结果混用。
|
||||
|
||||
换机器或模型时，至少检查 `MODEL_PATH`、`DATASET_PATH`、Docker 镜像、GPU 编号、
|
||||
端口、模型最大上下文和显存参数。需要永久修改时再编辑对应目录的 `config.env`。
|
||||
|
||||
## 6. 结果
|
||||
|
||||
每次运行在 `${RESULT_BASE}/${RUN_ID}/` 下生成：
|
||||
|
||||
```text
|
||||
adaptive_points.csv 每个并发测点的 TPS、TTFT、TPOT 等
|
||||
adaptive_summary.csv 每个 ISL/OSL/TP/DP 的饱和并发和最佳 TPS 并发
|
||||
adaptive_summary.md 可直接阅读的汇总表
|
||||
run_manifest.json 本次模型、数据集和搜索参数
|
||||
logs/ 编排与服务日志
|
||||
tp*_dp*/raw_outputs/ sglang.bench_serving 原始 JSON
|
||||
tp*_dp*/gpu_logs/ nvidia-smi 采样
|
||||
```
|
||||
|
||||
快速查看：
|
||||
|
||||
```bash
|
||||
column -s, -t adaptive_summary.csv | less -S
|
||||
tail -f logs/orchestrator.log
|
||||
```
|
||||
@ -22,9 +22,10 @@ MAX_NUM_SEQS="${MAX_NUM_SEQS:-256}"
|
||||
MAX_RUNNING="${MAX_RUNNING:-256}"
|
||||
|
||||
# Scenarios: "concurrency input_len output_len num_prompts".
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
declare -a SCENARIOS=(
|
||||
"1 512 256 32"
|
||||
"32 512 256 128"
|
||||
"1 512 256 5"
|
||||
"32 512 256 160"
|
||||
)
|
||||
|
||||
VENV_CLIENT="${VENV_CLIENT:-$VENV_SGLANG}"
|
||||
|
||||
@ -22,8 +22,9 @@ MAX_NUM_SEQS=8
|
||||
MAX_RUNNING=8
|
||||
|
||||
# "concurrency input_len output_len num_prompts"
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
declare -a SCENARIOS=(
|
||||
"2 262144 4096 4"
|
||||
"2 262144 4096 10"
|
||||
)
|
||||
|
||||
VENV_CLIENT="${VENV_CLIENT:-$VENV_SGLANG}"
|
||||
|
||||
@ -23,9 +23,10 @@ MAX_NUM_SEQS=64
|
||||
MAX_RUNNING=64
|
||||
|
||||
# "concurrency input_len output_len num_prompts"
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
declare -a SCENARIOS=(
|
||||
"25 65536 256 100"
|
||||
"8 65536 1024 100"
|
||||
"25 65536 256 125"
|
||||
"8 65536 1024 40"
|
||||
)
|
||||
|
||||
VENV_CLIENT="${VENV_CLIENT:-$VENV_SGLANG}"
|
||||
|
||||
100
experiments/dsv4_h200_long_context_matrix/compare.py
Executable file
100
experiments/dsv4_h200_long_context_matrix/compare.py
Executable file
@ -0,0 +1,100 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate a side-by-side comparison of SGLang, vLLM, and DSpark results.
|
||||
|
||||
Usage:
|
||||
python3 compare.py --sglang <sglang_root> --vllm <vllm_root> --dspark <dspark_root> \
|
||||
[--output comparison.md]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_result(result_root: Path) -> dict:
|
||||
path = result_root / "results.json"
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def slo_status(ttft_p95_ms: float, tpot_mean_ms: float, ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str:
|
||||
ttft_ok = ttft_p95_ms < ttft_limit_ms
|
||||
tpot_ok = tpot_mean_ms < tpot_limit_ms
|
||||
if ttft_ok and tpot_ok:
|
||||
return "✅"
|
||||
if ttft_ok or tpot_ok:
|
||||
return "⚠️"
|
||||
return "❌"
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--sglang", type=Path, required=True)
|
||||
parser.add_argument("--vllm", type=Path, required=True)
|
||||
parser.add_argument("--dspark", type=Path, required=True)
|
||||
parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md"))
|
||||
parser.add_argument("--ttft-limit", type=float, default=3000.0)
|
||||
parser.add_argument("--tpot-limit", type=float, default=50.0)
|
||||
args = parser.parse_args()
|
||||
|
||||
sglang_data = load_result(args.sglang)
|
||||
vllm_data = load_result(args.vllm)
|
||||
dspark_data = load_result(args.dspark)
|
||||
|
||||
model = sglang_data.get("metadata", {}).get("model", "unknown")
|
||||
hardware = sglang_data.get("metadata", {}).get("hardware", "unknown")
|
||||
|
||||
# Map engine names to display labels.
|
||||
engine_to_label = {
|
||||
"sglang": "sglang",
|
||||
"vllm": "vllm",
|
||||
"vllm-dspark": "dspark",
|
||||
}
|
||||
|
||||
by_scenario = defaultdict(dict)
|
||||
for data in (sglang_data, vllm_data, dspark_data):
|
||||
engine = data["metadata"]["engine"]
|
||||
label = engine_to_label.get(engine, engine)
|
||||
for s in data.get("scenarios", []):
|
||||
key = s["name"]
|
||||
by_scenario[key][label] = s
|
||||
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
f.write(f"# SGLang vs vLLM vs DSpark long-context matrix ({hardware})\n\n")
|
||||
f.write("## Summary\n\n")
|
||||
f.write(f"- Model: `{model}`\n")
|
||||
f.write(f"- DSpark model: `/data/models/DeepSeek-V4-Flash-DSpark`\n")
|
||||
f.write(f"- Hardware: {hardware}\n")
|
||||
f.write("- Benchmark client: `sglang.bench_serving`\n")
|
||||
f.write("- DSpark flags: `--spec-method dspark --spec-model <model> --spec-tokens 5`\n")
|
||||
f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n")
|
||||
|
||||
f.write("## Side-by-side results\n\n")
|
||||
f.write("| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | SLO |\n")
|
||||
f.write("|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n")
|
||||
|
||||
for scenario_name in sorted(by_scenario.keys()):
|
||||
for backend in ("sglang", "vllm", "dspark"):
|
||||
s = by_scenario[scenario_name].get(backend)
|
||||
if s is None:
|
||||
continue
|
||||
cfg = s["config"]
|
||||
m = s["metrics"]
|
||||
status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
|
||||
f.write(
|
||||
f"| {scenario_name} | {backend} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | "
|
||||
f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||
f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||
f"{m['e2e_ms']['p99']:.2f} | {status} |\n"
|
||||
)
|
||||
|
||||
f.write("\n## Notes\n\n")
|
||||
f.write("- SLO check uses TTFT P95 and TPOT mean.\n")
|
||||
f.write("- A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out.\n")
|
||||
|
||||
print(f"Wrote comparison to {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -10,9 +10,15 @@ SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
SGLANG_PORT="${SGLANG_PORT:-30006}"
|
||||
VLLM_PORT="${VLLM_PORT:-30005}"
|
||||
DSPARK_PORT="${DSPARK_PORT:-30013}"
|
||||
|
||||
VENV_SGLANG="${VENV_SGLANG:-/data/user1/yy/envs/sglang}"
|
||||
VENV_VLLM="${VENV_VLLM:-/data/user1/yy/envs/vllm}"
|
||||
VENV_VLLM_DSPARK="${VENV_VLLM_DSPARK:-/data/user1/yy/envs/vllm-dspark}"
|
||||
|
||||
# DSpark uses a separate model checkpoint and speculative decoding.
|
||||
DSPARK_MODEL_NAME="DeepSeek-V4-Flash-DSpark"
|
||||
DSPARK_MODEL_PATH="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||
|
||||
export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||
TP=8
|
||||
@ -26,37 +32,40 @@ declare -a CONTEXT_GROUPS=(
|
||||
)
|
||||
|
||||
# Scenarios per group: "concurrency input_len output_len num_prompts"
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
# NOTE: long-context scenarios are expensive; these counts follow the 5x rule.
|
||||
declare -a SCENARIOS_64K=(
|
||||
"25 65536 256 50"
|
||||
"25 65536 1024 50"
|
||||
"25 65536 4096 50"
|
||||
"25 65536 256 125"
|
||||
"25 65536 1024 125"
|
||||
"25 65536 4096 125"
|
||||
)
|
||||
|
||||
declare -a SCENARIOS_128K=(
|
||||
"10 131072 256 20"
|
||||
"5 131072 256 10"
|
||||
"2 131072 256 4"
|
||||
"10 131072 1024 20"
|
||||
"5 131072 1024 10"
|
||||
"2 131072 1024 4"
|
||||
"10 131072 4096 20"
|
||||
"5 131072 4096 10"
|
||||
"2 131072 4096 4"
|
||||
"10 131072 256 50"
|
||||
"5 131072 256 25"
|
||||
"2 131072 256 10"
|
||||
"10 131072 1024 50"
|
||||
"5 131072 1024 25"
|
||||
"2 131072 1024 10"
|
||||
"10 131072 4096 50"
|
||||
"5 131072 4096 25"
|
||||
"2 131072 4096 10"
|
||||
)
|
||||
|
||||
declare -a SCENARIOS_256K=(
|
||||
"10 262144 256 20"
|
||||
"5 262144 256 10"
|
||||
"2 262144 256 4"
|
||||
"10 262144 1024 20"
|
||||
"5 262144 1024 10"
|
||||
"2 262144 1024 4"
|
||||
"10 262144 4096 20"
|
||||
"5 262144 4096 10"
|
||||
"2 262144 4096 4"
|
||||
"10 262144 256 50"
|
||||
"5 262144 256 25"
|
||||
"2 262144 256 10"
|
||||
"10 262144 1024 50"
|
||||
"5 262144 1024 25"
|
||||
"2 262144 1024 10"
|
||||
"10 262144 4096 50"
|
||||
"5 262144 4096 25"
|
||||
"2 262144 4096 10"
|
||||
)
|
||||
|
||||
VENV_CLIENT="${VENV_CLIENT:-$VENV_SGLANG}"
|
||||
|
||||
SGLANG_START_SCRIPT="${SCRIPT_DIR:-.}/start_sglang.sh"
|
||||
VLLM_START_SCRIPT="${SCRIPT_DIR:-.}/start_vllm.sh"
|
||||
DSPARK_START_SCRIPT="${SCRIPT_DIR:-.}/start_dspark.sh"
|
||||
|
||||
@ -0,0 +1,83 @@
|
||||
# SGLang vs vLLM vs DSpark long-context matrix (8x NVIDIA H200 143GB)
|
||||
|
||||
## Summary
|
||||
|
||||
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||
- DSpark model: `/data/models/DeepSeek-V4-Flash-DSpark`
|
||||
- Hardware: 8x NVIDIA H200 143GB
|
||||
- Benchmark client: `sglang.bench_serving`
|
||||
- DSpark flags: `--spec-method dspark --spec-model <model> --spec-tokens 5`
|
||||
- SLO reference: TTFT P95 < 3000.0ms, TPOT mean < 50.0ms
|
||||
|
||||
## Side-by-side results
|
||||
|
||||
| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | SLO |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c10_i131072_o1024 | sglang | 10 | 131072 | 1024 | 0.68 | 319.74 | 4147.51 | 7409.82 | 25.69 | 49.89 | 58.92 | 26432.66 | ⚠️ |
|
||||
| c10_i131072_o1024 | vllm | 10 | 131072 | 1024 | 0.39 | 183.08 | 6596.15 | 6603.12 | 47.10 | 77.53 | 84.06 | 48426.64 | ⚠️ |
|
||||
| c10_i131072_o1024 | dspark | 10 | 131072 | 1024 | 0.66 | 310.66 | 11615.81 | 11676.24 | 24.58 | 46.24 | 46.56 | 29271.39 | ⚠️ |
|
||||
| c10_i131072_o256 | sglang | 10 | 131072 | 256 | 0.34 | 42.70 | 26046.62 | 26064.38 | 1374.50 | 1810.56 | 19553.62 | 58326.41 | ❌ |
|
||||
| c10_i131072_o256 | vllm | 10 | 131072 | 256 | 0.38 | 48.09 | 23250.03 | 24238.33 | 150.22 | 299.91 | 301.72 | 51439.08 | ❌ |
|
||||
| c10_i131072_o256 | dspark | 10 | 131072 | 256 | 0.40 | 51.33 | 24693.20 | 27039.11 | 59.88 | 78.22 | 316.95 | 36603.69 | ❌ |
|
||||
| c10_i131072_o4096 | sglang | 10 | 131072 | 4096 | 0.27 | 517.96 | 6177.42 | 6920.28 | 14.21 | 19.92 | 25.87 | 54286.63 | ⚠️ |
|
||||
| c10_i131072_o4096 | vllm | 10 | 131072 | 4096 | 0.23 | 445.81 | 2096.29 | 2168.81 | 18.57 | 21.47 | 21.61 | 67041.13 | ✅ |
|
||||
| c10_i131072_o4096 | dspark | 10 | 131072 | 4096 | 1.05 | 1998.23 | 1543.09 | 1546.01 | 3.34 | 4.81 | 4.84 | 14731.59 | ✅ |
|
||||
| c10_i262144_o1024 | sglang | 10 | 262144 | 1024 | 0.16 | 76.30 | 39446.96 | 43965.67 | 108.42 | 202.43 | 202.90 | 120180.87 | ❌ |
|
||||
| c10_i262144_o1024 | vllm | 10 | 262144 | 1024 | 0.18 | 84.63 | 34130.22 | 38144.40 | 98.87 | 142.57 | 266.08 | 108658.89 | ❌ |
|
||||
| c10_i262144_o1024 | dspark | 10 | 262144 | 1024 | 0.20 | 96.79 | 39321.30 | 40783.01 | 48.19 | 67.60 | 68.13 | 91123.07 | ⚠️ |
|
||||
| c10_i262144_o256 | sglang | 10 | 262144 | 256 | 0.15 | 18.70 | 58074.31 | 62652.59 | 2953.70 | 3625.59 | 41185.32 | 134628.33 | ❌ |
|
||||
| c10_i262144_o256 | vllm | 10 | 262144 | 256 | 0.18 | 22.76 | 50632.44 | 51883.02 | 229.20 | 339.27 | 344.95 | 110216.92 | ❌ |
|
||||
| c10_i262144_o256 | dspark | 10 | 262144 | 256 | 0.18 | 23.12 | 64654.53 | 67877.62 | 52.05 | 73.84 | 151.28 | 80196.26 | ❌ |
|
||||
| c10_i262144_o4096 | sglang | 10 | 262144 | 4096 | 0.14 | 272.46 | 18088.12 | 22579.09 | 28.00 | 43.87 | 45.07 | 120183.75 | ⚠️ |
|
||||
| c10_i262144_o4096 | vllm | 10 | 262144 | 4096 | 0.11 | 215.86 | 34100.32 | 38022.94 | 37.18 | 56.07 | 62.00 | 157944.21 | ⚠️ |
|
||||
| c10_i262144_o4096 | dspark | 10 | 262144 | 4096 | 0.18 | 349.94 | 33426.83 | 37512.71 | 20.67 | 35.94 | 44.44 | 103440.52 | ⚠️ |
|
||||
| c25_i65536_o1024 | sglang | 25 | 65536 | 1024 | 0.67 | 346.70 | 19996.43 | 24478.11 | 149.27 | 395.00 | 2037.09 | 69586.94 | ❌ |
|
||||
| c25_i65536_o1024 | vllm | 25 | 65536 | 1024 | 0.64 | 332.61 | 24861.20 | 28942.75 | 72.36 | 196.24 | 271.32 | 72689.84 | ❌ |
|
||||
| c25_i65536_o1024 | dspark | 25 | 65536 | 1024 | 0.80 | 412.74 | 26078.12 | 29135.60 | 42.82 | 56.19 | 133.89 | 60995.55 | ⚠️ |
|
||||
| c25_i65536_o256 | sglang | 25 | 65536 | 256 | 0.71 | 84.45 | 27219.06 | 31612.70 | 292.84 | 801.18 | 938.43 | 68854.26 | ❌ |
|
||||
| c25_i65536_o256 | vllm | 25 | 65536 | 256 | 0.81 | 95.60 | 24902.72 | 28974.45 | 180.78 | 273.59 | 278.16 | 60781.45 | ❌ |
|
||||
| c25_i65536_o256 | dspark | 25 | 65536 | 256 | 0.84 | 98.79 | 27635.11 | 29212.48 | 51.68 | 65.52 | 134.94 | 39437.04 | ❌ |
|
||||
| c25_i65536_o4096 | sglang | 25 | 65536 | 4096 | 0.41 | 950.01 | 18559.58 | 23249.71 | 23.78 | 42.50 | 82.84 | 96670.12 | ⚠️ |
|
||||
| c25_i65536_o4096 | vllm | 25 | 65536 | 4096 | 0.32 | 741.71 | 24847.97 | 28840.96 | 28.80 | 46.66 | 63.40 | 129654.04 | ⚠️ |
|
||||
| c25_i65536_o4096 | dspark | 25 | 65536 | 4096 | 0.61 | 1409.20 | 25048.60 | 29143.30 | 14.72 | 30.57 | 42.07 | 73496.43 | ⚠️ |
|
||||
| c2_i131072_o1024 | sglang | 2 | 131072 | 1024 | 0.60 | 136.10 | 479.99 | 482.72 | 8.56 | 9.72 | 9.79 | 5480.98 | ✅ |
|
||||
| c2_i131072_o1024 | vllm | 2 | 131072 | 1024 | 0.70 | 158.36 | 676.14 | 695.90 | 6.49 | 8.20 | 8.34 | 4803.24 | ✅ |
|
||||
| c2_i131072_o1024 | dspark | 2 | 131072 | 1024 | 2.29 | 514.96 | 679.81 | 691.56 | 1.10 | 1.60 | 1.63 | 1450.74 | ✅ |
|
||||
| c2_i131072_o256 | sglang | 2 | 131072 | 256 | 1.43 | 139.48 | 589.14 | 605.33 | 8.29 | 8.96 | 9.05 | 1750.12 | ✅ |
|
||||
| c2_i131072_o256 | vllm | 2 | 131072 | 256 | 1.51 | 147.12 | 684.63 | 700.28 | 7.08 | 9.27 | 9.45 | 1556.46 | ✅ |
|
||||
| c2_i131072_o256 | dspark | 2 | 131072 | 256 | 3.10 | 301.11 | 617.17 | 634.08 | 1.00 | 1.41 | 1.42 | 780.85 | ✅ |
|
||||
| c2_i131072_o4096 | sglang | 2 | 131072 | 4096 | 0.11 | 244.38 | 584.43 | 599.22 | 7.53 | 7.61 | 7.62 | 28360.35 | ✅ |
|
||||
| c2_i131072_o4096 | vllm | 2 | 131072 | 4096 | 0.12 | 262.22 | 671.34 | 688.15 | 7.09 | 7.17 | 7.18 | 26496.04 | ✅ |
|
||||
| c2_i131072_o4096 | dspark | 2 | 131072 | 4096 | 0.47 | 1076.96 | 540.43 | 541.94 | 1.76 | 2.03 | 2.08 | 6313.36 | ✅ |
|
||||
| c2_i262144_o1024 | sglang | 2 | 262144 | 1024 | 0.51 | 114.94 | 1007.07 | 1024.10 | 8.36 | 9.03 | 9.06 | 6240.65 | ✅ |
|
||||
| c2_i262144_o1024 | vllm | 2 | 262144 | 1024 | 0.19 | 43.14 | 8722.76 | 9009.70 | 20.17 | 52.24 | 57.91 | 13826.13 | ⚠️ |
|
||||
| c2_i262144_o1024 | dspark | 2 | 262144 | 1024 | 0.25 | 55.65 | 10466.90 | 10984.16 | 14.36 | 43.30 | 48.52 | 15361.85 | ⚠️ |
|
||||
| c2_i262144_o256 | sglang | 2 | 262144 | 256 | 0.29 | 28.44 | 9538.91 | 9980.78 | 14.10 | 26.98 | 29.16 | 11941.34 | ⚠️ |
|
||||
| c2_i262144_o256 | vllm | 2 | 262144 | 256 | 0.23 | 22.74 | 8625.18 | 8952.31 | 10.18 | 23.01 | 25.44 | 10378.60 | ⚠️ |
|
||||
| c2_i262144_o256 | dspark | 2 | 262144 | 256 | 0.26 | 25.25 | 10459.44 | 10978.82 | 17.58 | 45.28 | 48.94 | 14620.43 | ⚠️ |
|
||||
| c2_i262144_o4096 | sglang | 2 | 262144 | 4096 | 0.08 | 189.78 | 9453.57 | 9896.38 | 7.82 | 8.48 | 8.60 | 38723.14 | ⚠️ |
|
||||
| c2_i262144_o4096 | vllm | 2 | 262144 | 4096 | 0.08 | 180.76 | 8759.63 | 9062.80 | 8.63 | 11.53 | 12.01 | 35881.75 | ⚠️ |
|
||||
| c2_i262144_o4096 | dspark | 2 | 262144 | 4096 | 0.17 | 384.42 | 8709.00 | 8987.60 | 3.23 | 6.44 | 6.99 | 15113.78 | ⚠️ |
|
||||
| c5_i131072_o1024 | sglang | 5 | 131072 | 1024 | 0.87 | 321.54 | 1231.23 | 1298.36 | 9.03 | 10.47 | 10.71 | 7298.25 | ✅ |
|
||||
| c5_i131072_o1024 | vllm | 5 | 131072 | 1024 | 0.90 | 331.65 | 856.18 | 917.80 | 8.93 | 10.48 | 10.62 | 6989.27 | ✅ |
|
||||
| c5_i131072_o1024 | dspark | 5 | 131072 | 1024 | 2.75 | 1019.06 | 1080.05 | 1176.83 | 2.53 | 3.98 | 4.18 | 2383.07 | ✅ |
|
||||
| c5_i131072_o256 | sglang | 5 | 131072 | 256 | 1.05 | 146.97 | 3270.23 | 4685.65 | 19.91 | 40.81 | 46.10 | 7815.89 | ⚠️ |
|
||||
| c5_i131072_o256 | vllm | 5 | 131072 | 256 | 0.39 | 54.07 | 9384.23 | 9677.91 | 51.54 | 107.16 | 117.25 | 19860.69 | ❌ |
|
||||
| c5_i131072_o256 | dspark | 5 | 131072 | 256 | 0.44 | 60.98 | 10128.32 | 10501.28 | 37.60 | 60.86 | 61.78 | 17256.59 | ⚠️ |
|
||||
| c5_i131072_o4096 | sglang | 5 | 131072 | 4096 | 0.20 | 378.76 | 4667.56 | 6275.51 | 11.98 | 22.07 | 23.60 | 31176.81 | ⚠️ |
|
||||
| c5_i131072_o4096 | vllm | 5 | 131072 | 4096 | 0.23 | 447.85 | 1054.01 | 1104.66 | 8.48 | 9.13 | 9.14 | 28818.20 | ✅ |
|
||||
| c5_i131072_o4096 | dspark | 5 | 131072 | 4096 | 0.70 | 1335.09 | 1156.91 | 1249.38 | 3.20 | 4.19 | 4.21 | 11315.66 | ✅ |
|
||||
| c5_i262144_o1024 | sglang | 5 | 262144 | 1024 | 0.15 | 56.42 | 20522.02 | 25944.29 | 54.11 | 118.98 | 124.02 | 53444.22 | ❌ |
|
||||
| c5_i262144_o1024 | vllm | 5 | 262144 | 1024 | 0.16 | 60.32 | 22564.89 | 26639.92 | 47.49 | 100.69 | 109.86 | 53790.59 | ⚠️ |
|
||||
| c5_i262144_o1024 | dspark | 5 | 262144 | 1024 | 0.19 | 69.32 | 22995.12 | 26781.97 | 38.56 | 57.19 | 57.43 | 40847.11 | ⚠️ |
|
||||
| c5_i262144_o256 | sglang | 5 | 262144 | 256 | 0.16 | 21.94 | 25875.40 | 30596.95 | 97.05 | 231.75 | 252.98 | 43756.53 | ❌ |
|
||||
| c5_i262144_o256 | vllm | 5 | 262144 | 256 | 0.19 | 25.85 | 21986.99 | 26381.86 | 123.42 | 202.78 | 215.56 | 45761.90 | ❌ |
|
||||
| c5_i262144_o256 | dspark | 5 | 262144 | 256 | 0.19 | 26.70 | 29425.38 | 30576.66 | 41.74 | 60.42 | 62.17 | 37097.84 | ⚠️ |
|
||||
| c5_i262144_o4096 | sglang | 5 | 262144 | 4096 | 0.11 | 209.50 | 16539.50 | 20323.40 | 25.05 | 56.69 | 60.22 | 60383.57 | ⚠️ |
|
||||
| c5_i262144_o4096 | vllm | 5 | 262144 | 4096 | 0.11 | 205.12 | 22171.78 | 26533.69 | 25.69 | 63.32 | 68.33 | 64086.27 | ⚠️ |
|
||||
| c5_i262144_o4096 | dspark | 5 | 262144 | 4096 | 0.16 | 299.79 | 22250.95 | 26667.05 | 17.46 | 45.41 | 49.74 | 51820.15 | ⚠️ |
|
||||
|
||||
## Notes
|
||||
|
||||
- SLO check uses TTFT P95 and TPOT mean.
|
||||
- A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out.
|
||||
@ -0,0 +1,35 @@
|
||||
# 8x NVIDIA H200 143GB VLLM Benchmark Report
|
||||
|
||||
- Result root: `/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260709-020227/dspark`
|
||||
- Model: `/data/models/DeepSeek-V4-Flash-DSpark`
|
||||
- Backend: VLLM
|
||||
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||
|
||||
## Results
|
||||
|
||||
| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c10_i131072_o1024 | dspark | 10 | 131072 | 1024 | 30.45 | 20 | 0.66 | 48423.00 | 310.66 | 48733.65 | 4033.86 | 11615.81 | 11676.24 | 24.58 | 46.24 | 46.56 | 14685.68 | 28877.70 | 29271.39 | ⚠️ |
|
||||
| c10_i131072_o256 | dspark | 10 | 131072 | 256 | 49.62 | 20 | 0.40 | 29712.25 | 51.33 | 29763.58 | 16856.00 | 24693.20 | 27039.11 | 59.88 | 78.22 | 316.95 | 22010.51 | 32508.36 | 36603.69 | ❌ |
|
||||
| c10_i131072_o4096 | dspark | 10 | 131072 | 4096 | 19.08 | 20 | 1.05 | 77265.07 | 1998.23 | 79263.30 | 727.34 | 1543.09 | 1546.01 | 3.34 | 4.81 | 4.84 | 7316.24 | 12401.94 | 14731.59 | ✅ |
|
||||
| c2_i131072_o1024 | dspark | 2 | 131072 | 1024 | 1.75 | 4 | 2.29 | 138490.59 | 514.96 | 139005.55 | 447.72 | 679.81 | 691.56 | 1.10 | 1.60 | 1.63 | 794.56 | 1362.81 | 1450.74 | ✅ |
|
||||
| c2_i131072_o256 | dspark | 2 | 131072 | 256 | 1.29 | 4 | 3.10 | 187564.60 | 301.11 | 187865.71 | 419.01 | 617.17 | 634.08 | 1.00 | 1.41 | 1.42 | 536.04 | 747.52 | 780.85 | ✅ |
|
||||
| c2_i131072_o4096 | dspark | 2 | 131072 | 4096 | 8.44 | 4 | 0.47 | 28698.90 | 1076.96 | 29775.86 | 325.51 | 540.43 | 541.94 | 1.76 | 2.03 | 2.08 | 4193.26 | 6216.99 | 6313.36 | ✅ |
|
||||
| c5_i131072_o1024 | dspark | 5 | 131072 | 1024 | 3.63 | 10 | 2.75 | 203409.39 | 1019.06 | 204428.44 | 595.29 | 1080.05 | 1176.83 | 2.53 | 3.98 | 4.18 | 1507.07 | 2347.99 | 2383.07 | ✅ |
|
||||
| c5_i131072_o256 | dspark | 5 | 131072 | 256 | 22.89 | 10 | 0.44 | 32258.45 | 60.98 | 32319.42 | 5806.57 | 10128.32 | 10501.28 | 37.60 | 60.86 | 61.78 | 11223.11 | 16310.02 | 17256.59 | ⚠️ |
|
||||
| c5_i131072_o4096 | dspark | 5 | 131072 | 4096 | 14.28 | 10 | 0.70 | 51732.19 | 1335.09 | 53067.28 | 560.62 | 1156.91 | 1249.38 | 3.20 | 4.19 | 4.21 | 6029.71 | 10552.87 | 11315.66 | ✅ |
|
||||
| c10_i262144_o1024 | dspark | 10 | 262144 | 1024 | 97.73 | 20 | 0.20 | 28498.71 | 96.79 | 28595.50 | 24591.83 | 39321.30 | 40783.01 | 48.19 | 67.60 | 68.13 | 45387.50 | 70254.49 | 91123.07 | ⚠️ |
|
||||
| c10_i262144_o256 | dspark | 10 | 262144 | 256 | 110.17 | 20 | 0.18 | 25281.12 | 23.12 | 25304.24 | 42248.63 | 64654.53 | 67877.62 | 52.05 | 73.84 | 151.28 | 48223.85 | 78728.49 | 80196.26 | ❌ |
|
||||
| c10_i262144_o4096 | dspark | 10 | 262144 | 4096 | 108.97 | 20 | 0.18 | 25559.57 | 349.94 | 25909.50 | 13472.46 | 33426.83 | 37512.71 | 20.67 | 35.94 | 44.44 | 52210.57 | 100105.13 | 103440.52 | ⚠️ |
|
||||
| c2_i262144_o1024 | dspark | 2 | 262144 | 1024 | 16.19 | 4 | 0.25 | 31154.91 | 55.65 | 31210.55 | 5853.53 | 10466.90 | 10984.16 | 14.36 | 43.30 | 48.52 | 7871.57 | 14740.72 | 15361.85 | ⚠️ |
|
||||
| c2_i262144_o256 | dspark | 2 | 262144 | 256 | 15.41 | 4 | 0.26 | 32738.24 | 25.25 | 32763.49 | 5850.43 | 10459.44 | 10978.82 | 17.58 | 45.28 | 48.94 | 7672.82 | 14088.43 | 14620.43 | ⚠️ |
|
||||
| c2_i262144_o4096 | dspark | 2 | 262144 | 4096 | 23.65 | 4 | 0.17 | 21326.49 | 384.42 | 21710.90 | 5487.63 | 8709.00 | 8987.60 | 3.23 | 6.44 | 6.99 | 11596.58 | 14994.53 | 15113.78 | ⚠️ |
|
||||
| c5_i262144_o1024 | dspark | 5 | 262144 | 1024 | 53.37 | 10 | 0.19 | 26116.34 | 69.32 | 26185.66 | 13406.14 | 22995.12 | 26781.97 | 38.56 | 57.19 | 57.43 | 25295.54 | 38376.69 | 40847.11 | ⚠️ |
|
||||
| c5_i262144_o256 | dspark | 5 | 262144 | 256 | 52.29 | 10 | 0.19 | 26655.70 | 26.70 | 26682.40 | 17996.01 | 29425.38 | 30576.66 | 41.74 | 60.42 | 62.17 | 23507.98 | 35234.36 | 37097.84 | ⚠️ |
|
||||
| c5_i262144_o4096 | dspark | 5 | 262144 | 4096 | 63.58 | 10 | 0.16 | 21924.13 | 299.79 | 22223.92 | 10590.48 | 22250.95 | 26667.05 | 17.46 | 45.41 | 49.74 | 30807.30 | 48506.31 | 51820.15 | ⚠️ |
|
||||
| c25_i65536_o1024 | dspark | 25 | 65536 | 1024 | 62.71 | 50 | 0.80 | 29366.67 | 412.74 | 29779.41 | 11692.07 | 26078.12 | 29135.60 | 42.82 | 56.19 | 133.89 | 29981.59 | 57554.65 | 60995.55 | ⚠️ |
|
||||
| c25_i65536_o256 | dspark | 25 | 65536 | 256 | 59.86 | 50 | 0.84 | 30764.36 | 98.79 | 30863.14 | 19438.04 | 27635.11 | 29212.48 | 51.68 | 65.52 | 134.94 | 24957.48 | 37092.86 | 39437.04 | ❌ |
|
||||
| c25_i65536_o4096 | dspark | 25 | 65536 | 4096 | 82.31 | 50 | 0.61 | 22371.88 | 1409.20 | 23781.09 | 8084.05 | 25048.60 | 29143.30 | 14.72 | 30.57 | 42.07 | 38164.69 | 72245.56 | 73496.43 | ⚠️ |
|
||||
|
||||
SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -14,6 +14,13 @@ source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||
RESULT_BASE="${SCRIPT_DIR}/results"
|
||||
# Comma-separated list of backends to run. Default: sglang,vllm,dspark.
|
||||
BACKENDS="${BACKENDS:-sglang,vllm,dspark}"
|
||||
|
||||
backend_enabled() {
|
||||
local backend="$1"
|
||||
[[ ",${BACKENDS}," == *",${backend},"* ]]
|
||||
}
|
||||
|
||||
log_dir_global="${RESULT_BASE}/${RUN_ID}/logs"
|
||||
mkdir -p "$log_dir_global"
|
||||
@ -52,6 +59,8 @@ stop_server() {
|
||||
# Fallback cleanup.
|
||||
if [[ "$backend" == "sglang" ]]; then
|
||||
pkill -9 -f "sglang serve.*DeepSeek-V4-Flash" 2>/dev/null || true
|
||||
elif [[ "$backend" == "dspark" ]]; then
|
||||
pkill -9 -f "vllm serve.*DeepSeek-V4-Flash-DSpark" 2>/dev/null || true
|
||||
else
|
||||
pkill -9 -f "vllm serve.*DeepSeek-V4-Flash" 2>/dev/null || true
|
||||
fi
|
||||
@ -62,13 +71,16 @@ start_server() {
|
||||
local backend="$1"
|
||||
local max_model_len="$2"
|
||||
local max_slots="$3"
|
||||
local start_script
|
||||
local start_script port
|
||||
if [[ "$backend" == "sglang" ]]; then
|
||||
start_script="${SGLANG_START_SCRIPT}"
|
||||
local port="$SGLANG_PORT"
|
||||
port="$SGLANG_PORT"
|
||||
elif [[ "$backend" == "dspark" ]]; then
|
||||
start_script="${DSPARK_START_SCRIPT}"
|
||||
port="$DSPARK_PORT"
|
||||
else
|
||||
start_script="${VLLM_START_SCRIPT}"
|
||||
local port="$VLLM_PORT"
|
||||
port="$VLLM_PORT"
|
||||
fi
|
||||
|
||||
log "starting ${backend} server (max_model_len=${max_model_len}, slots=${max_slots}) with ${start_script}"
|
||||
@ -88,13 +100,15 @@ run_warmup() {
|
||||
local port
|
||||
if [[ "$backend" == "sglang" ]]; then
|
||||
port="$SGLANG_PORT"
|
||||
elif [[ "$backend" == "dspark" ]]; then
|
||||
port="$DSPARK_PORT"
|
||||
else
|
||||
port="$VLLM_PORT"
|
||||
fi
|
||||
|
||||
log "warming up ${backend} (input=${input_len}, output=${output_len}, num=1)"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/warmup.py" \
|
||||
--backend "$backend" \
|
||||
--backend vllm \
|
||||
--port "$port" \
|
||||
--input-len "$input_len" \
|
||||
--output-len "$output_len" \
|
||||
@ -142,6 +156,8 @@ run_group() {
|
||||
local port
|
||||
if [[ "$backend" == "sglang" ]]; then
|
||||
port="$SGLANG_PORT"
|
||||
elif [[ "$backend" == "dspark" ]]; then
|
||||
port="$DSPARK_PORT"
|
||||
else
|
||||
port="$VLLM_PORT"
|
||||
fi
|
||||
@ -161,9 +177,22 @@ run_group() {
|
||||
warmup_input_len="$(echo "$warmup_input" | awk '{print $2}')"
|
||||
run_warmup "$backend" "$warmup_input_len" 256
|
||||
|
||||
# Bench client backend: DSpark is served by vLLM API, so use vllm client.
|
||||
local client_backend="$backend"
|
||||
if [[ "$backend" == "dspark" ]]; then
|
||||
client_backend="vllm"
|
||||
fi
|
||||
|
||||
# Output filename prefix: DSpark raw files must start with "vllm_" so that
|
||||
# scripts/common/parse_backend.py (which only knows sglang/vllm) can parse them.
|
||||
local file_prefix="${backend}"
|
||||
if [[ "$backend" == "dspark" ]]; then
|
||||
file_prefix="vllm_dspark"
|
||||
fi
|
||||
|
||||
for scenario in "${scenarios_ref[@]}"; do
|
||||
read -r concurrency input_len output_len num_prompts <<< "$scenario"
|
||||
output_file="${raw_dir}/${backend}_${group_label}_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||
output_file="${raw_dir}/${file_prefix}_${group_label}_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||
detail_log="${phase_log_dir}/${backend}_${group_label}_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||
|
||||
if scenario_already_completed "$output_file" "$num_prompts"; then
|
||||
@ -174,7 +203,7 @@ run_group() {
|
||||
log "running ${backend} ${group_label} scenario: c=${concurrency} i=${input_len} o=${output_len} n=${num_prompts}"
|
||||
|
||||
"${VENV_CLIENT}/bin/python" -m sglang.bench_serving \
|
||||
--backend "$backend" \
|
||||
--backend "$client_backend" \
|
||||
--host 127.0.0.1 \
|
||||
--port "$port" \
|
||||
--dataset-name random \
|
||||
@ -200,8 +229,13 @@ run_group() {
|
||||
parse_backend() {
|
||||
local backend="$1"
|
||||
local result_root="${RESULT_BASE}/${RUN_ID}/${backend}"
|
||||
log "parsing ${backend} results in ${result_root}"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend "$backend" \
|
||||
# parse_backend.py only knows sglang/vllm; DSpark raw files use vllm client format.
|
||||
local parser_backend="$backend"
|
||||
if [[ "$backend" == "dspark" ]]; then
|
||||
parser_backend="vllm"
|
||||
fi
|
||||
log "parsing ${backend} results in ${result_root} (parser_backend=${parser_backend})"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend "$parser_backend" \
|
||||
>> "${result_root}/logs/parse.log" 2>&1 || {
|
||||
log "WARNING: parser failed for ${backend}; see ${result_root}/logs/parse.log"
|
||||
}
|
||||
@ -213,20 +247,28 @@ write_backend_metadata() {
|
||||
ensure_result_root "$result_root"
|
||||
local meta_json="${result_root}/results.json"
|
||||
|
||||
local env_path
|
||||
local env_path model_path engine_name
|
||||
if [[ "$backend" == "sglang" ]]; then
|
||||
env_path="$VENV_SGLANG"
|
||||
model_path="$MODEL_PATH"
|
||||
engine_name="sglang"
|
||||
elif [[ "$backend" == "dspark" ]]; then
|
||||
env_path="$VENV_VLLM_DSPARK"
|
||||
model_path="$DSPARK_MODEL_PATH"
|
||||
engine_name="vllm-dspark"
|
||||
else
|
||||
env_path="$VENV_VLLM"
|
||||
model_path="$MODEL_PATH"
|
||||
engine_name="vllm"
|
||||
fi
|
||||
|
||||
write_metadata_json \
|
||||
"$meta_json" \
|
||||
"${EXPERIMENT_NAME}_${backend}" \
|
||||
"$RUN_ID" \
|
||||
"$MODEL_PATH" \
|
||||
"$backend" \
|
||||
"$backend" \
|
||||
"$model_path" \
|
||||
"${backend}" \
|
||||
"$engine_name" \
|
||||
"$HARDWARE" \
|
||||
"$ACCELERATOR" \
|
||||
"$CHIP" \
|
||||
@ -254,11 +296,13 @@ write_backend_metadata() {
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Cleanup any leftovers.
|
||||
stop_server sglang
|
||||
stop_server vllm
|
||||
for backend in sglang vllm dspark; do
|
||||
backend_enabled "$backend" && stop_server "$backend"
|
||||
done
|
||||
|
||||
write_backend_metadata sglang
|
||||
write_backend_metadata vllm
|
||||
for backend in sglang vllm dspark; do
|
||||
backend_enabled "$backend" && write_backend_metadata "$backend"
|
||||
done
|
||||
|
||||
for group in "${CONTEXT_GROUPS[@]}"; do
|
||||
read -r group_label max_model_len max_slots <<< "$group"
|
||||
@ -270,24 +314,28 @@ for group in "${CONTEXT_GROUPS[@]}"; do
|
||||
continue
|
||||
fi
|
||||
|
||||
# Run SGLang for this group.
|
||||
run_group sglang "$group_label" "$max_model_len" "$max_slots" "$scenario_array_name"
|
||||
|
||||
# Run vLLM for this group.
|
||||
run_group vllm "$group_label" "$max_model_len" "$max_slots" "$scenario_array_name"
|
||||
for backend in sglang vllm dspark; do
|
||||
backend_enabled "$backend" && run_group "$backend" "$group_label" "$max_model_len" "$max_slots" "$scenario_array_name"
|
||||
done
|
||||
done
|
||||
|
||||
# Parse and compare.
|
||||
parse_backend sglang
|
||||
parse_backend vllm
|
||||
for backend in sglang vllm dspark; do
|
||||
backend_enabled "$backend" && parse_backend "$backend"
|
||||
done
|
||||
|
||||
log "generating comparison report"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/compare.py" \
|
||||
if backend_enabled sglang && backend_enabled vllm && backend_enabled dspark; then
|
||||
log "generating three-way comparison report"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/compare.py" \
|
||||
--sglang "${RESULT_BASE}/${RUN_ID}/sglang" \
|
||||
--vllm "${RESULT_BASE}/${RUN_ID}/vllm" \
|
||||
--dspark "${RESULT_BASE}/${RUN_ID}/dspark" \
|
||||
--output "${RESULT_BASE}/${RUN_ID}/comparison.md" \
|
||||
>> "${log_dir_global}/compare.log" 2>&1 || {
|
||||
log "WARNING: comparison script failed; see ${log_dir_global}/compare.log"
|
||||
}
|
||||
else
|
||||
log "skipping three-way comparison (not all backends enabled: BACKENDS=${BACKENDS})"
|
||||
fi
|
||||
|
||||
log "all results saved to ${RESULT_BASE}/${RUN_ID}"
|
||||
|
||||
72
experiments/dsv4_h200_long_context_matrix/start_dspark.sh
Executable file
72
experiments/dsv4_h200_long_context_matrix/start_dspark.sh
Executable file
@ -0,0 +1,72 @@
|
||||
#!/bin/bash
|
||||
# Start vLLM with DSpark speculative decoding for the long-context matrix.
|
||||
# Usage: start_dspark.sh <max_model_len> <max_num_seqs>
|
||||
set -e
|
||||
|
||||
MAX_MODEL_LEN="${1:-80000}"
|
||||
MAX_NUM_SEQS="${2:-64}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
cd /data/user1/yy
|
||||
mkdir -p logs tmp
|
||||
|
||||
VENV="${VENV_VLLM_DSPARK}"
|
||||
export PATH="$VENV/bin:$PATH"
|
||||
export PYTHONUNBUFFERED=1
|
||||
export TMPDIR=/data/user1/yy/tmp
|
||||
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}"
|
||||
|
||||
LOG="/data/user1/yy/logs/dsv4_h200_long_context_matrix_dspark_${MAX_MODEL_LEN}_$(date +%Y%m%d_%H%M%S).log"
|
||||
PID_FILE="/data/user1/yy/dsv4_h200_long_context_matrix_dspark.pid"
|
||||
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
echo "=== Starting DeepSeek-V4-Flash-DSpark server (TP=$TP, max_model_len=$MAX_MODEL_LEN, max_num_seqs=$MAX_NUM_SEQS) ==="
|
||||
echo "Model: $DSPARK_MODEL_PATH"
|
||||
echo "Port: $DSPARK_PORT"
|
||||
echo "Log: $LOG"
|
||||
|
||||
nohup vllm serve "$DSPARK_MODEL_PATH" \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size "$TP" \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-model-len "$MAX_MODEL_LEN" \
|
||||
--max-num-seqs "$MAX_NUM_SEQS" \
|
||||
--block-size 256 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--tokenizer-mode deepseek_v4 \
|
||||
--reasoning-parser deepseek_v4 \
|
||||
--spec-method dspark \
|
||||
--spec-model "$DSPARK_MODEL_PATH" \
|
||||
--spec-tokens 5 \
|
||||
--no-disable-hybrid-kv-cache-manager \
|
||||
--disable-uvicorn-access-log \
|
||||
--port "$DSPARK_PORT" \
|
||||
> "$LOG" 2>&1 &
|
||||
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
echo "Waiting for health..."
|
||||
|
||||
for i in $(seq 1 240); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${DSPARK_PORT}/health" >/dev/null 2>&1; then
|
||||
echo "DSpark server is ready at http://127.0.0.1:${DSPARK_PORT}"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
if ! kill -0 $PID 2>/dev/null; then
|
||||
echo "ERROR: DSpark server exited early"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/240)"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: DSpark server not healthy after 240 retries"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
100
experiments/dsv4_h200_max_context_length/kv_cache_analysis.md
Normal file
100
experiments/dsv4_h200_max_context_length/kv_cache_analysis.md
Normal file
@ -0,0 +1,100 @@
|
||||
# DeepSeek-V4-Flash 长上下文显存分析
|
||||
|
||||
> 针对 `dsv4_h200_max_context_length` 实验结果（8×H200，TP=8，output_len=1，concurrency=1）的显存复盘。
|
||||
|
||||
## 1. 关键模型参数
|
||||
|
||||
| 参数 | 数值 |
|
||||
|---|---|
|
||||
| 总参数量 | ~158B（safetensors 汇总） |
|
||||
| Checkpoint 大小 | ~149 GiB（fp8 权重 + scale） |
|
||||
| 层数 `num_hidden_layers` | 43 |
|
||||
| Hidden size | 4096 |
|
||||
| Attention heads | 64 |
|
||||
| KV heads | 1 |
|
||||
| 每层 KV `head_dim` | 512 |
|
||||
| KV cache dtype | fp8 |
|
||||
| 每层压缩比 | 2 层不压缩，21 层 4:1，20 层 128:1 |
|
||||
| `max_position_embeddings` | **1,048,576** |
|
||||
| H200 单卡显存 | 143 GiB |
|
||||
|
||||
## 2. 理论显存估算
|
||||
|
||||
### 2.1 权重
|
||||
|
||||
```
|
||||
149 GiB / 8 (TP) ≈ 18.6 GiB / GPU
|
||||
```
|
||||
|
||||
vLLM 实际启动日志：`Model loading took 20.07 GiB memory`，与理论值基本吻合（包含少量加载开销）。
|
||||
|
||||
### 2.2 KV cache
|
||||
|
||||
按压缩后的平均每层 KV 大小估算：
|
||||
|
||||
- 不压缩的 2 层：512 bytes/token
|
||||
- 4:1 压缩的 21 层：128 bytes/token
|
||||
- 128:1 压缩的 20 层：4 bytes/token
|
||||
|
||||
```
|
||||
平均每层 = (2×512 + 21×128 + 20×4) / 43 ≈ 88 bytes/token
|
||||
43 层 ≈ 3.8 KiB/token
|
||||
```
|
||||
|
||||
再考虑 indexer、alignment 等辅助缓存，vLLM 实际日志给出的数字是：
|
||||
|
||||
```
|
||||
Available KV cache memory: 101.39 GiB
|
||||
GPU KV cache size: 14,668,151 tokens
|
||||
=> 约 7.0 KiB / token / GPU
|
||||
```
|
||||
|
||||
因此 1M 上下文的 KV cache 约为：
|
||||
|
||||
```
|
||||
1,048,576 × 7.0 KiB ≈ 7.0 GiB / GPU
|
||||
```
|
||||
|
||||
### 2.3 1M 上下文单卡总占用
|
||||
|
||||
| 组件 | 估算 |
|
||||
|---|---|
|
||||
| 权重 | ~20 GiB |
|
||||
| KV cache（1M tokens） | ~7 GiB |
|
||||
| CUDA graph / 启动开销 | ~1 GiB |
|
||||
| 激活 / 临时 buffer | 数 GiB（prefill 峰值） |
|
||||
| **合计** | **~30 GiB / GPU** |
|
||||
|
||||
vLLM 可用显存按 `--gpu-memory-utilization 0.90` 计算：
|
||||
|
||||
```
|
||||
143 GiB × 0.9 ≈ 128.7 GiB
|
||||
```
|
||||
|
||||
**显存完全够用。**
|
||||
|
||||
## 3. 与实测结果对比
|
||||
|
||||
| 实测 target_len | max_model_len | 结果 |
|
||||
|---|---|---|
|
||||
| 917,504 | 918,528 | ✅ 通过 |
|
||||
| 1,048,576 | 1,049,600 | ❌ server failed to start |
|
||||
|
||||
vLLM 1M 失败日志的关键信息：
|
||||
|
||||
```text
|
||||
User-specified max_model_len (1049600) is greater than the derived max_model_len
|
||||
(max_position_embeddings=1048576 or model_max_length=None in model's config.json).
|
||||
```
|
||||
|
||||
这说明：
|
||||
|
||||
1. **917,504 通过是合理的**：此时 KV cache 约 `6.1 GiB/GPU`，加上权重和开销远不到显存上限。
|
||||
2. **1,048,576 失败是合理的**：实验里 `max_model_len = input_len + 1024 pad = 1,049,600`，超过了模型 config 声明的 `max_position_embeddings=1,048,576`。**这是位置编码/配置上限，不是显存上限**。
|
||||
3. 实验报告的最大长度 **917,504 是偏保守的**：理论上在 output_len=1 时，最大 input length 可以接近 `1,048,575`（只要 `max_model_len ≤ 1,048,576`）。实验因为采用了固定步长并加了 1024 的 pad，没有测到这个边界。
|
||||
|
||||
## 4. 结论
|
||||
|
||||
- **从显存角度看，1M 上下文在 8×H200 上完全可以放下**：单卡约 30 GiB，仅占可用显存的 1/4 左右。
|
||||
- **真正限制到不了 1M 的是 `max_position_embeddings=1048576`**，而不是显存。
|
||||
- `dsv4_h200_max_context_length` 的结果与理论估算一致；测出的 917,504 是“该实验配置下能跑通的最大值”，但不是“硬件/显存能支持的最大值”。
|
||||
@ -0,0 +1,36 @@
|
||||
# H200 Max Context Length Exploration Report
|
||||
|
||||
- Result root: `/data/user1/yy/experiments/dsv4_h200_max_context_length/results/20260708-143534`
|
||||
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||
- Hardware: 8x NVIDIA H200 143GB
|
||||
- TP: 8
|
||||
- Output len: 1, num prompts: 1, concurrency: 1
|
||||
|
||||
## Per-attempt results
|
||||
|
||||
| Backend | Target len | Max model/context len | Status | TTFT mean(ms) | TTFT P95(ms) | E2E mean(ms) | E2E P99(ms) | Error |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| sglang | 65536 | 66560 | ✅ pass | 521.23 | 521.23 | 521.25 | 521.25 | |
|
||||
| sglang | 131072 | 132096 | ✅ pass | 668.67 | 668.67 | 668.69 | 668.69 | |
|
||||
| sglang | 262144 | 263168 | ✅ pass | 1232.63 | 1232.63 | 1232.65 | 1232.65 | |
|
||||
| sglang | 393216 | 394240 | ✅ pass | 1496.49 | 1496.49 | 1496.53 | 1496.53 | |
|
||||
| sglang | 524288 | 525312 | ✅ pass | 1764.18 | 1764.18 | 1764.23 | 1764.23 | |
|
||||
| sglang | 655360 | 656384 | ✅ pass | 2129.19 | 2129.19 | 2129.24 | 2129.24 | |
|
||||
| sglang | 786432 | 787456 | ✅ pass | 2440.34 | 2440.34 | 2440.35 | 2440.35 | |
|
||||
| sglang | 917504 | 918528 | ✅ pass | 3149.58 | 3149.58 | 3149.61 | 3149.61 | |
|
||||
| sglang | 1048576 | 1049600 | ❌ fail | 0.00 | 0.00 | 0.00 | 0.00 | server failed to start |
|
||||
| vllm | 65536 | 66560 | ✅ pass | 594.47 | 594.47 | 594.53 | 594.53 | |
|
||||
| vllm | 131072 | 132096 | ✅ pass | 889.46 | 889.46 | 889.51 | 889.51 | |
|
||||
| vllm | 262144 | 263168 | ✅ pass | 1178.53 | 1178.53 | 1178.56 | 1178.56 | |
|
||||
| vllm | 393216 | 394240 | ✅ pass | 1371.19 | 1371.19 | 1371.24 | 1371.24 | |
|
||||
| vllm | 524288 | 525312 | ✅ pass | 2582.85 | 2582.85 | 2582.90 | 2582.90 | |
|
||||
| vllm | 655360 | 656384 | ✅ pass | 2203.55 | 2203.55 | 2203.59 | 2203.59 | |
|
||||
| vllm | 786432 | 787456 | ✅ pass | 3886.52 | 3886.52 | 3886.57 | 3886.57 | |
|
||||
| vllm | 917504 | 918528 | ✅ pass | 3715.61 | 3715.61 | 3715.64 | 3715.64 | |
|
||||
| vllm | 1048576 | 1049600 | ❌ fail | 0.00 | 0.00 | 0.00 | 0.00 | server failed to start |
|
||||
|
||||
## Maximum supported input length
|
||||
|
||||
- **sglang**: 917504 tokens
|
||||
- **vllm**: 917504 tokens
|
||||
|
||||
@ -0,0 +1,783 @@
|
||||
{
|
||||
"metadata": {
|
||||
"experiment": "dsv4_h200_max_context_length",
|
||||
"run_id": "20260708-143534",
|
||||
"timestamp": "2026-07-08T14:35:34+00:00",
|
||||
"model": "/data/models/DeepSeek-V4-Flash",
|
||||
"backend": "sglang+vllm",
|
||||
"engine": "sglang+vllm",
|
||||
"hardware": "8x NVIDIA H200 143GB",
|
||||
"accelerator": "NVIDIA H200",
|
||||
"chip": "nvidia_h200",
|
||||
"script": "experiments/dsv4_h200_max_context_length/run_bench.sh",
|
||||
"env": "/data/user1/yy/envs/sglang",
|
||||
"git_commit": "ef65885",
|
||||
"git_dirty": "dirty",
|
||||
"description": "H200 max context length exploration for DeepSeek-V4-Flash"
|
||||
},
|
||||
"config": {
|
||||
"tp": 8,
|
||||
"cuda_visible_devices": "0,1,2,3,4,5,6,7",
|
||||
"output_len": 1,
|
||||
"num_prompts": 1,
|
||||
"max_concurrency": 1,
|
||||
"request_rate": 10000,
|
||||
"context_pad": 1024
|
||||
},
|
||||
"scenarios": [
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 65536,
|
||||
"max_model_len": 66560,
|
||||
"success": true,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 66560 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 0.5241995020041941,
|
||||
"request_throughput": 1.9076706409995772,
|
||||
"input_token_throughput": 125021.10312854829,
|
||||
"output_token_throughput": 1.9076706409995772,
|
||||
"total_input_tokens": 65536,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 521.2510440032929,
|
||||
"p50": 521.2510440032929,
|
||||
"p90": 521.2510440032929,
|
||||
"p95": 521.2510440032929,
|
||||
"p99": 521.2510440032929
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 521.2282620050246,
|
||||
"p50": 521.2282620050246,
|
||||
"p90": 521.2282620050246,
|
||||
"p95": 521.2282620050246,
|
||||
"p99": 521.2282620050246
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 131072,
|
||||
"max_model_len": 132096,
|
||||
"success": true,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 132096 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 0.6736304189980729,
|
||||
"request_throughput": 1.484493532057763,
|
||||
"input_token_throughput": 194575.53623387511,
|
||||
"output_token_throughput": 1.484493532057763,
|
||||
"total_input_tokens": 131072,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 668.692392995581,
|
||||
"p50": 668.692392995581,
|
||||
"p90": 668.692392995581,
|
||||
"p95": 668.692392995581,
|
||||
"p99": 668.692392995581
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 668.666598998243,
|
||||
"p50": 668.666598998243,
|
||||
"p90": 668.666598998243,
|
||||
"p95": 668.666598998243,
|
||||
"p99": 668.666598998243
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 262144,
|
||||
"max_model_len": 263168,
|
||||
"success": true,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 263168 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 1.235619597006007,
|
||||
"request_throughput": 0.8093105697118029,
|
||||
"input_token_throughput": 212155.90998653087,
|
||||
"output_token_throughput": 0.8093105697118029,
|
||||
"total_input_tokens": 262144,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 1232.6548010023544,
|
||||
"p50": 1232.6548010023544,
|
||||
"p90": 1232.6548010023544,
|
||||
"p95": 1232.6548010023544,
|
||||
"p99": 1232.6548010023544
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 1232.628379992093,
|
||||
"p50": 1232.628379992093,
|
||||
"p90": 1232.628379992093,
|
||||
"p95": 1232.628379992093,
|
||||
"p99": 1232.628379992093
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 393216,
|
||||
"max_model_len": 394240,
|
||||
"success": true,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 394240 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 1.4997316709923325,
|
||||
"request_throughput": 0.6667859453406932,
|
||||
"input_token_throughput": 262190.90228308603,
|
||||
"output_token_throughput": 0.6667859453406932,
|
||||
"total_input_tokens": 393216,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 1496.5319500042824,
|
||||
"p50": 1496.5319500042824,
|
||||
"p90": 1496.5319500042824,
|
||||
"p95": 1496.5319500042824,
|
||||
"p99": 1496.5319500042824
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 1496.4891230047215,
|
||||
"p50": 1496.4891230047215,
|
||||
"p90": 1496.4891230047215,
|
||||
"p95": 1496.4891230047215,
|
||||
"p99": 1496.4891230047215
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 524288,
|
||||
"max_model_len": 525312,
|
||||
"success": true,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 525312 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 1.7669223830016563,
|
||||
"request_throughput": 0.5659558165204717,
|
||||
"input_token_throughput": 296723.84313188505,
|
||||
"output_token_throughput": 0.5659558165204717,
|
||||
"total_input_tokens": 524288,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 1764.228992004064,
|
||||
"p50": 1764.228992004064,
|
||||
"p90": 1764.228992004064,
|
||||
"p95": 1764.228992004064,
|
||||
"p99": 1764.228992004064
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 1764.184692001436,
|
||||
"p50": 1764.184692001436,
|
||||
"p90": 1764.184692001436,
|
||||
"p95": 1764.184692001436,
|
||||
"p99": 1764.184692001436
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 655360,
|
||||
"max_model_len": 656384,
|
||||
"success": true,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 656384 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 2.1324244450079277,
|
||||
"request_throughput": 0.4689497920271132,
|
||||
"input_token_throughput": 307330.93570288894,
|
||||
"output_token_throughput": 0.4689497920271132,
|
||||
"total_input_tokens": 655360,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 2129.2401870014146,
|
||||
"p50": 2129.2401870014146,
|
||||
"p90": 2129.2401870014146,
|
||||
"p95": 2129.2401870014146,
|
||||
"p99": 2129.2401870014146
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 2129.1923570097424,
|
||||
"p50": 2129.1923570097424,
|
||||
"p90": 2129.1923570097424,
|
||||
"p95": 2129.1923570097424,
|
||||
"p99": 2129.1923570097424
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 786432,
|
||||
"max_model_len": 787456,
|
||||
"success": true,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 787456 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 2.4429005489946576,
|
||||
"request_throughput": 0.409349451581865,
|
||||
"input_token_throughput": 321925.50790642924,
|
||||
"output_token_throughput": 0.409349451581865,
|
||||
"total_input_tokens": 786432,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 2440.3524239896797,
|
||||
"p50": 2440.3524239896797,
|
||||
"p90": 2440.3524239896797,
|
||||
"p95": 2440.3524239896797,
|
||||
"p99": 2440.3524239896797
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 2440.341288995114,
|
||||
"p50": 2440.341288995114,
|
||||
"p90": 2440.341288995114,
|
||||
"p95": 2440.341288995114,
|
||||
"p99": 2440.341288995114
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 917504,
|
||||
"max_model_len": 918528,
|
||||
"success": true,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 918528 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 3.152248802012764,
|
||||
"request_throughput": 0.31723384250681075,
|
||||
"input_token_throughput": 291063.3194353689,
|
||||
"output_token_throughput": 0.31723384250681075,
|
||||
"total_input_tokens": 917504,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 3149.605149999843,
|
||||
"p50": 3149.605149999843,
|
||||
"p90": 3149.605149999843,
|
||||
"p95": 3149.605149999843,
|
||||
"p99": 3149.605149999843
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 3149.579901000834,
|
||||
"p50": 3149.579901000834,
|
||||
"p90": 3149.579901000834,
|
||||
"p95": 3149.579901000834,
|
||||
"p99": 3149.579901000834
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "sglang",
|
||||
"target_len": 1048576,
|
||||
"max_model_len": 1049600,
|
||||
"success": false,
|
||||
"server_args": "sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash --tp 8 --moe-runner-backend marlin --context-length 1049600 --max-running-requests 1 --mem-fraction-static 0.88 --host 0.0.0.0 --port 30011",
|
||||
"metrics": null,
|
||||
"error": "server failed to start"
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 65536,
|
||||
"max_model_len": 66560,
|
||||
"success": true,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 66560 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 0.5968579519976629,
|
||||
"request_throughput": 1.675440524253777,
|
||||
"input_token_throughput": 109801.67019749554,
|
||||
"output_token_throughput": 1.675440524253777,
|
||||
"total_input_tokens": 65536,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 594.5277150021866,
|
||||
"p50": 594.5277150021866,
|
||||
"p90": 594.5277150021866,
|
||||
"p95": 594.5277150021866,
|
||||
"p99": 594.5277150021866
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 594.4739139958983,
|
||||
"p50": 594.4739139958983,
|
||||
"p90": 594.4739139958983,
|
||||
"p95": 594.4739139958983,
|
||||
"p99": 594.4739139958983
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 131072,
|
||||
"max_model_len": 132096,
|
||||
"success": true,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 132096 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 0.8928074569994351,
|
||||
"request_throughput": 1.1200623294084255,
|
||||
"input_token_throughput": 146808.80964022115,
|
||||
"output_token_throughput": 1.1200623294084255,
|
||||
"total_input_tokens": 131072,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 889.5100749941776,
|
||||
"p50": 889.5100749941776,
|
||||
"p90": 889.5100749941776,
|
||||
"p95": 889.5100749941776,
|
||||
"p99": 889.5100749941776
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 889.4585100060795,
|
||||
"p50": 889.4585100060795,
|
||||
"p90": 889.4585100060795,
|
||||
"p95": 889.4585100060795,
|
||||
"p99": 889.4585100060795
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 262144,
|
||||
"max_model_len": 263168,
|
||||
"success": true,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 263168 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 1.1815237270056969,
|
||||
"request_throughput": 0.8463647213706597,
|
||||
"input_token_throughput": 221869.4335189902,
|
||||
"output_token_throughput": 0.8463647213706597,
|
||||
"total_input_tokens": 262144,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 1178.564740010188,
|
||||
"p50": 1178.564740010188,
|
||||
"p90": 1178.564740010188,
|
||||
"p95": 1178.564740010188,
|
||||
"p99": 1178.564740010188
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 1178.5299420007505,
|
||||
"p50": 1178.5299420007505,
|
||||
"p90": 1178.5299420007505,
|
||||
"p95": 1178.5299420007505,
|
||||
"p99": 1178.5299420007505
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 393216,
|
||||
"max_model_len": 394240,
|
||||
"success": true,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 394240 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 1.3780717330082553,
|
||||
"request_throughput": 0.725651630497532,
|
||||
"input_token_throughput": 285337.83153771755,
|
||||
"output_token_throughput": 0.725651630497532,
|
||||
"total_input_tokens": 393216,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 1371.242612003698,
|
||||
"p50": 1371.242612003698,
|
||||
"p90": 1371.242612003698,
|
||||
"p95": 1371.242612003698,
|
||||
"p99": 1371.242612003698
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 1371.1869800026761,
|
||||
"p50": 1371.1869800026761,
|
||||
"p90": 1371.1869800026761,
|
||||
"p95": 1371.1869800026761,
|
||||
"p99": 1371.1869800026761
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 524288,
|
||||
"max_model_len": 525312,
|
||||
"success": true,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 525312 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 2.587135097011924,
|
||||
"request_throughput": 0.3865279401740462,
|
||||
"input_token_throughput": 202651.96069797032,
|
||||
"output_token_throughput": 0.3865279401740462,
|
||||
"total_input_tokens": 524288,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 2582.903844988323,
|
||||
"p50": 2582.903844988323,
|
||||
"p90": 2582.903844988323,
|
||||
"p95": 2582.903844988323,
|
||||
"p99": 2582.903844988323
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 2582.850620994577,
|
||||
"p50": 2582.850620994577,
|
||||
"p90": 2582.850620994577,
|
||||
"p95": 2582.850620994577,
|
||||
"p99": 2582.850620994577
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 655360,
|
||||
"max_model_len": 656384,
|
||||
"success": true,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 656384 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 2.206327235006029,
|
||||
"request_throughput": 0.453241923561383,
|
||||
"input_token_throughput": 297036.62702518795,
|
||||
"output_token_throughput": 0.453241923561383,
|
||||
"total_input_tokens": 655360,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 2203.5920649941545,
|
||||
"p50": 2203.5920649941545,
|
||||
"p90": 2203.5920649941545,
|
||||
"p95": 2203.5920649941545,
|
||||
"p99": 2203.5920649941545
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 2203.5506129905116,
|
||||
"p50": 2203.5506129905116,
|
||||
"p90": 2203.5506129905116,
|
||||
"p95": 2203.5506129905116,
|
||||
"p99": 2203.5506129905116
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 786432,
|
||||
"max_model_len": 787456,
|
||||
"success": true,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 787456 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 3.8894555929873604,
|
||||
"request_throughput": 0.25710539073977023,
|
||||
"input_token_throughput": 202195.90665025898,
|
||||
"output_token_throughput": 0.25710539073977023,
|
||||
"total_input_tokens": 786432,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 3886.5709000092465,
|
||||
"p50": 3886.5709000092465,
|
||||
"p90": 3886.5709000092465,
|
||||
"p95": 3886.5709000092465,
|
||||
"p99": 3886.5709000092465
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 3886.5172150108265,
|
||||
"p50": 3886.5172150108265,
|
||||
"p90": 3886.5172150108265,
|
||||
"p95": 3886.5172150108265,
|
||||
"p99": 3886.5172150108265
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 917504,
|
||||
"max_model_len": 918528,
|
||||
"success": true,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 918528 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": {
|
||||
"success_count": 1,
|
||||
"total_count": 1,
|
||||
"duration_s": 3.7180563760048244,
|
||||
"request_throughput": 0.26895772921940825,
|
||||
"input_token_throughput": 246769.79238972397,
|
||||
"output_token_throughput": 0.26895772921940825,
|
||||
"total_input_tokens": 917504,
|
||||
"total_output_tokens": 1,
|
||||
"e2e_ms": {
|
||||
"mean": 3715.635304004536,
|
||||
"p50": 3715.635304004536,
|
||||
"p90": 3715.635304004536,
|
||||
"p95": 3715.635304004536,
|
||||
"p99": 3715.635304004536
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 3715.6081900029676,
|
||||
"p50": 3715.6081900029676,
|
||||
"p90": 3715.6081900029676,
|
||||
"p95": 3715.6081900029676,
|
||||
"p99": 3715.6081900029676
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
}
|
||||
},
|
||||
"error": null
|
||||
},
|
||||
{
|
||||
"backend": "vllm",
|
||||
"target_len": 1048576,
|
||||
"max_model_len": 1049600,
|
||||
"success": false,
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 1049600 --max-num-seqs 1 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30012",
|
||||
"metrics": null,
|
||||
"error": "server failed to start"
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -15,14 +15,16 @@ ENGINE="sglang"
|
||||
VENV_SERVER="${VENV_SERVER:-/data/user1/yy/envs/sglang}"
|
||||
VENV_CLIENT="${VENV_CLIENT:-/data/user1/yy/envs/sglang}"
|
||||
|
||||
# Benchmark scenarios: each element is "concurrency input_len output_len".
|
||||
# Benchmark scenarios: each element is "concurrency input_len output_len num_prompts".
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
if [[ -z "${SCENARIOS:-}" ]]; then
|
||||
SCENARIOS=(
|
||||
"32 512 256"
|
||||
"128 512 256"
|
||||
"32 512 256 160"
|
||||
"128 512 256 640"
|
||||
)
|
||||
fi
|
||||
NUM_PROMPTS="${NUM_PROMPTS:-128}"
|
||||
# Legacy fallback: used only if SCENARIOS entries have 3 fields.
|
||||
NUM_PROMPTS="${NUM_PROMPTS:-160}"
|
||||
|
||||
# Server start script bundled with this experiment.
|
||||
SERVER_START_SCRIPT="${SCRIPT_DIR}/start_server.sh"
|
||||
|
||||
@ -60,8 +60,7 @@ data["config"] = {
|
||||
"tp": 8,
|
||||
"moe_runner_backend": "marlin",
|
||||
"port": 30006,
|
||||
"num_prompts": 128,
|
||||
"scenarios": ["32 512 256", "128 512 256"]
|
||||
"scenarios": ["32 512 256 160", "128 512 256 640"]
|
||||
}
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
@ -126,11 +125,15 @@ fi
|
||||
log "===== BENCHMARK START ====="
|
||||
|
||||
for scenario in "${SCENARIOS[@]}"; do
|
||||
read -r concurrency input_len output_len <<< "$scenario"
|
||||
read -r concurrency input_len output_len num_prompts <<< "$scenario"
|
||||
# Fallback for legacy 3-field scenarios.
|
||||
if [[ -z "${num_prompts:-}" ]]; then
|
||||
num_prompts="$NUM_PROMPTS"
|
||||
fi
|
||||
output_file="${RAW_DIR}/sglang_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||
detail_log="${LOG_DIR}/sglang_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||
|
||||
log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len}"
|
||||
log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len} num_prompts=${num_prompts}"
|
||||
|
||||
"${VENV_CLIENT}/bin/python" -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
@ -139,7 +142,7 @@ for scenario in "${SCENARIOS[@]}"; do
|
||||
--dataset-name random \
|
||||
--random-input-len "$input_len" \
|
||||
--random-output-len "$output_len" \
|
||||
--num-prompts "$NUM_PROMPTS" \
|
||||
--num-prompts "$num_prompts" \
|
||||
--max-concurrency "$concurrency" \
|
||||
--request-rate 10000 \
|
||||
--output-file "$output_file" \
|
||||
|
||||
234
experiments/dsv4_h200_sglang_tp_dp_matrix/README.md
Normal file
234
experiments/dsv4_h200_sglang_tp_dp_matrix/README.md
Normal file
@ -0,0 +1,234 @@
|
||||
# SGLang TP×DP Matrix Benchmark
|
||||
|
||||
Benchmark SGLang inference performance across different Tensor Parallelism (TP) × Data Parallelism (DP) configurations for large MoE models (e.g., DeepSeek-V4-Flash).
|
||||
|
||||
## What This Experiment Does
|
||||
|
||||
This experiment systematically tests how different parallel strategies affect serving performance:
|
||||
|
||||
| Config | TP | DP | GPUs per replica | Replicas |
|
||||
|--------|----|----|------------------|----------|
|
||||
| tp2_dp4 | 2 | 4 | 2 | 4 |
|
||||
| tp4_dp2 | 4 | 2 | 4 | 2 |
|
||||
| tp8_dp1 | 8 | 1 | 8 | 1 |
|
||||
|
||||
For each configuration, it runs a matrix of input/output sequence lengths and concurrency levels, measuring:
|
||||
- **TTFT** (Time To First Token)
|
||||
- **TPOT** (Time Per Output Token)
|
||||
- **Throughput** (requests/s, output tokens/s)
|
||||
- **GPU memory usage**
|
||||
- **SLO compliance** (TTFT P95 < 3000ms, TPOT mean < 50ms)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Prerequisites
|
||||
|
||||
- NVIDIA GPU server with Docker and NVIDIA Container Toolkit
|
||||
- Model weights mounted locally (e.g., `/data/models/DeepSeek-V4-Flash`)
|
||||
- ShareGPT-format dataset JSON for the random sampler (e.g., `/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json`)
|
||||
- SGLang Docker image pulled locally
|
||||
|
||||
### 2. Configure the Experiment
|
||||
|
||||
Edit `config.env` to match your environment:
|
||||
|
||||
```bash
|
||||
# Model settings
|
||||
MODEL_NAME="DeepSeek-V4-Flash"
|
||||
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
|
||||
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
# Docker image (change this if you use a custom or private registry image)
|
||||
DOCKER_IMAGE="lmsysorg/sglang:latest"
|
||||
|
||||
# Dataset path (change this to your local dataset)
|
||||
DATASET_PATH="/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"
|
||||
|
||||
# GPU devices to use
|
||||
CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||
|
||||
# TP×DP configurations to test
|
||||
PARALLEL_CONFIGS=(
|
||||
"2 4"
|
||||
"4 2"
|
||||
"8 1"
|
||||
)
|
||||
```
|
||||
|
||||
### 3. Run the Experiment
|
||||
|
||||
#### Option A: Persistent Container Mode (Recommended)
|
||||
|
||||
Starts a long-running container and reuses it across configurations. This avoids reloading the model and preserves DeepGEMM JIT cache, saving significant time.
|
||||
|
||||
```bash
|
||||
# Step 1: Start the persistent container (run once)
|
||||
bash start_sglang_container.sh
|
||||
|
||||
# Step 2: Run the benchmark
|
||||
CONTAINER_NAME="dsv4_h200_sglang_tp_dp_matrix_sglang_container" \
|
||||
bash run_bench.sh
|
||||
```
|
||||
|
||||
#### Option B: One-Container-Per-Config Mode
|
||||
|
||||
Each TP×DP configuration starts a fresh Docker container. Slower but simpler.
|
||||
|
||||
```bash
|
||||
# Just run the main script
|
||||
bash run_bench.sh
|
||||
```
|
||||
|
||||
### 4. View Results
|
||||
|
||||
Results are saved under `results/<run_id>/`:
|
||||
|
||||
```
|
||||
results/
|
||||
└── full_20260710-063109/
|
||||
├── tp2_dp4/
|
||||
│ ├── results.json # Parsed metrics per scenario
|
||||
│ ├── raw_outputs/ # Raw benchmark output files
|
||||
│ ├── gpu_logs/ # GPU memory CSV logs
|
||||
│ └── logs/ # Detailed logs per scenario
|
||||
├── tp4_dp2/
|
||||
│ └── ...
|
||||
├── tp8_dp1/
|
||||
│ └── ...
|
||||
├── comparison.md # Cross-config comparison report
|
||||
└── logs/
|
||||
└── orchestrator.log # Main experiment log
|
||||
```
|
||||
|
||||
Generate the comparison report manually:
|
||||
|
||||
```bash
|
||||
python3 compare.py --run-root results/<run_id> --output results/<run_id>/comparison.md
|
||||
```
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
### `config.env` — Main Configuration File
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `MODEL_NAME` | `DeepSeek-V4-Flash` | Human-readable model name |
|
||||
| `MODEL_PATH` | `/data/models/DeepSeek-V4-Flash` | **Local path to model weights** |
|
||||
| `SERVED_MODEL_NAME` | `deepseek-v4-flash` | Model name exposed by the server |
|
||||
| `DOCKER_IMAGE` | `lmsysorg/sglang:latest` | **SGLang Docker image** |
|
||||
| `DATASET_PATH` | `/data/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json` | **Local dataset for random sampler** |
|
||||
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3,4,5,6,7` | GPU devices to use |
|
||||
| `PARALLEL_CONFIGS` | `("2 4" "4 2" "8 1")` | TP×DP configurations to test |
|
||||
| `CONTEXT_LENGTH` | `1048576` | Max context length (tokens) |
|
||||
| `MAX_RUNNING_REQUESTS` | `128` | Max concurrent requests |
|
||||
| `MEM_FRACTION_STATIC` | `0.88` | Static memory fraction |
|
||||
| `MOE_RUNNER_BACKEND` | `marlin` | MoE backend (marlin, flashinfer, etc.) |
|
||||
| `USE_DOCKER` | `1` | Run server in Docker (1) or native venv (0) |
|
||||
| `USE_DOCKER_CLIENT` | `1` | Run benchmark client in Docker (1) or native (0) |
|
||||
| `SGLANG_PORT` | `30031` | Server port |
|
||||
| `MATRIX_FILE` | `matrix.json` | Scenario matrix definition |
|
||||
| `MATRIX_MODE` | `Y` | Test only Y (mandatory), or Y+P (include optional) |
|
||||
| `CONCURRENCY_SAMPLES` | `2` | Concurrency sampling density |
|
||||
| `SCENARIO_TIMEOUT_S` | `1800` | Per-scenario timeout (seconds) |
|
||||
| `DRY_RUN` | `0` | If 1, only print plan without running |
|
||||
| `GRID_LIMIT` | `0` | Limit scenarios per config (0 = all) |
|
||||
|
||||
### `matrix.json` — Test Scenario Matrix
|
||||
|
||||
Defines which (Input Sequence Length, Output Sequence Length) combinations to test:
|
||||
|
||||
```json
|
||||
{
|
||||
"mode": "Y",
|
||||
"matrix": {
|
||||
"1024": { "128": "Y", "256": "Y", ... },
|
||||
"4096": { "128": "Y", "256": "Y", ... }
|
||||
},
|
||||
"concurrency": {
|
||||
"1024": { "low": 1, "high": 128 },
|
||||
"4096": { "low": 1, "high": 64 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Mark meanings:
|
||||
- **Y** = Must test; failure stops the config
|
||||
- **P** = Optional; failure is recorded but continues
|
||||
- **N** = Skip
|
||||
|
||||
### `smoke_matrix.json` — Quick Verification
|
||||
|
||||
Small matrix for fast testing. Use it to verify the setup works:
|
||||
|
||||
```bash
|
||||
MATRIX_FILE="${PWD}/smoke_matrix.json" \
|
||||
GRID_LIMIT=4 \
|
||||
bash run_bench.sh
|
||||
```
|
||||
|
||||
## Script Reference
|
||||
|
||||
| Script | Purpose | When to Run |
|
||||
|--------|---------|-------------|
|
||||
| `run_bench.sh` | **Main experiment orchestrator** | Always run this |
|
||||
| `start_sglang_container.sh` | Start a persistent Docker container | Before run_bench.sh if using persistent mode |
|
||||
| `run_sglang_in_container.sh` | Start/restart server inside persistent container | Called by run_bench.sh |
|
||||
| `start_sglang_dp.sh` | Entry point for starting server | Called by run_bench.sh |
|
||||
| `start_sglang_docker.sh` | Start a fresh Docker container per config | Called by start_sglang_dp.sh |
|
||||
| `generate_scenarios.py` | Generate scenario TSV from matrix.json | Called by run_bench.sh |
|
||||
| `compare.py` | Generate cross-config comparison report | Called by run_bench.sh, or run manually |
|
||||
|
||||
## Migrating to a New Server
|
||||
|
||||
When moving this experiment to a different server, update these fields in `config.env`:
|
||||
|
||||
1. **Model path**: `MODEL_PATH` → your local model weights directory
|
||||
2. **Docker image**: `DOCKER_IMAGE` → your SGLang image (e.g., `myregistry/sglang:v0.4.0`)
|
||||
3. **Dataset path**: `DATASET_PATH` → your local ShareGPT-format JSON
|
||||
4. **GPU devices**: `CUDA_VISIBLE_DEVICES` → match your hardware
|
||||
5. **TP×DP configs**: `PARALLEL_CONFIGS` → adjust to your GPU count
|
||||
- Total GPUs must equal `TP × DP` for each config
|
||||
- Example: 8 GPUs → `(2 4)`, `(4 2)`, `(8 1)` are all valid
|
||||
|
||||
Optional adjustments:
|
||||
- `CONTEXT_LENGTH` → model's max context length
|
||||
- `MEM_FRACTION_STATIC` → adjust if you hit OOM
|
||||
- `MOE_RUNNER_BACKEND` → `marlin`, `flashinfer`, etc.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| `docker: invalid reference format` | `DOCKER_IMAGE` not exported to subshell | Already fixed; ensure `run_bench.sh` exports it |
|
||||
| Server health check timeout | Model loading takes >20 min | Increase wait loops in `start_sglang_docker.sh` or use persistent container mode |
|
||||
| `CUDA out of memory` | Config exceeds GPU memory | Reduce `MEM_FRACTION_STATIC` or test smaller ISL/DSL |
|
||||
| Benchmark client can't reach server | Network isolation | Ensure both server and client use `--network host` |
|
||||
| Stale container from previous run | Container name collision | `docker rm -f <container_name>` before restarting |
|
||||
|
||||
## Monitoring a Running Experiment
|
||||
|
||||
```bash
|
||||
# Watch the main log
|
||||
tail -f /tmp/full_experiment.log
|
||||
|
||||
# Check if the orchestrator is still running
|
||||
ps aux | grep run_bench | grep -v grep
|
||||
|
||||
# Check GPU utilization
|
||||
watch -n 1 nvidia-smi
|
||||
|
||||
# Check server health
|
||||
curl http://127.0.0.1:30031/health
|
||||
```
|
||||
|
||||
## Output Files
|
||||
|
||||
- `results/<run_id>/<config_label>/results.json` — Parsed metrics per scenario
|
||||
- `results/<run_id>/<config_label>/raw_outputs/` — Raw `sglang.bench_serving` JSONL output
|
||||
- `results/<run_id>/<config_label>/gpu_logs/` — GPU memory usage CSV
|
||||
- `results/<run_id>/comparison.md` — Side-by-side comparison across all configs
|
||||
- `results/<run_id>/skipped_after_oom.csv` — Record of OOM/failed scenarios
|
||||
|
||||
## License
|
||||
|
||||
Follows the project's existing license. Model weights and Docker images are subject to their respective licenses.
|
||||
@ -0,0 +1,109 @@
|
||||
# SGLang Adaptive Concurrency Search
|
||||
|
||||
`run_adaptive_concurrency.sh` keeps the existing ISL/DSL and TP/DP matrix, but
|
||||
searches concurrency at runtime instead of testing only low/high endpoints.
|
||||
|
||||
## Search rule
|
||||
|
||||
For every fixed `(TP, DP, ISL, DSL)`:
|
||||
|
||||
```text
|
||||
C = 1, 2, 4, 8, ... up to SEARCH_MAX_CONCURRENCY
|
||||
num_prompts = C * NUM_PROMPTS_MULTIPLIER
|
||||
```
|
||||
|
||||
The search stops when Total TPS grows by less than `TPS_MIN_GAIN_PCT` for
|
||||
`PLATEAU_PATIENCE` consecutive points. Defaults are 2% and two points.
|
||||
|
||||
- `saturation_concurrency`: first point in the final low-gain streak
|
||||
- `best_tps_concurrency`: tested point with the highest Total TPS
|
||||
- `stop_probe_concurrency`: last point tested to confirm the stop
|
||||
|
||||
OOM stops only the current shape. The service is restarted before the next
|
||||
shape. A non-OOM failure is retried once after a service restart.
|
||||
|
||||
Before every measured point, the client sends `C` concurrent warm-up requests.
|
||||
This keeps lazy kernel compilation and CUDA graph capture out of the measured
|
||||
TTFT/TPS. `BENCH_WARMUP_MAX_REQUESTS=0` means no cap; use a positive value only
|
||||
when deliberately limiting warm-up at very high concurrency.
|
||||
|
||||
## Dataset requirement
|
||||
|
||||
The default remains compatible with the fixed matrix baseline:
|
||||
|
||||
```text
|
||||
--dataset-name random --dataset-path DATASET_PATH
|
||||
--random-range-ratio 1.0
|
||||
```
|
||||
|
||||
In the installed SGLang client, the upstream default `0.0` samples each length
|
||||
uniformly from `1` to the requested maximum. This adaptive experiment sets
|
||||
`1.0` explicitly so every point represents the fixed ISL/DSL shown in the
|
||||
matrix. Override `RANDOM_RANGE_RATIO` only when a mixed-length workload is
|
||||
intentional.
|
||||
|
||||
`DATASET_PATH` must have at least
|
||||
`SEARCH_MAX_CONCURRENCY * NUM_PROMPTS_MULTIPLIER` usable two-turn
|
||||
conversations. The configured dataset is checked before model loading.
|
||||
|
||||
For setup smoke tests without ShareGPT, explicitly use:
|
||||
|
||||
```bash
|
||||
BENCH_DATASET_NAME=random-ids
|
||||
```
|
||||
|
||||
This sends generated token IDs with `--tokenize-prompt`; it is convenient for
|
||||
smoke tests but changes the token distribution, so do not mix it with random
|
||||
ShareGPT-backed baseline results.
|
||||
|
||||
## Commands
|
||||
|
||||
Print the plan without starting a server:
|
||||
|
||||
```bash
|
||||
DRY_RUN=1 bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
Run one small search:
|
||||
|
||||
```bash
|
||||
TP_LIST="8" \
|
||||
ISL_LIST="1024" \
|
||||
DSL_LIST="128" \
|
||||
GRID_LIMIT=1 \
|
||||
SEARCH_MAX_CONCURRENCY=8 \
|
||||
DATASET_PATH=/path/to/ShareGPT.json \
|
||||
bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
Persistent-container mode remains supported:
|
||||
|
||||
```bash
|
||||
bash start_sglang_container.sh
|
||||
CONTAINER_NAME="dsv4_h200_sglang_tp_dp_matrix_sglang_container" \
|
||||
DATASET_PATH=/path/to/ShareGPT.json \
|
||||
bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
Run long searches inside tmux.
|
||||
|
||||
## Outputs
|
||||
|
||||
Results are written to `adaptive_results/<run_id>/`:
|
||||
|
||||
```text
|
||||
adaptive_points.jsonl
|
||||
adaptive_points.csv
|
||||
adaptive_summary.jsonl
|
||||
adaptive_summary.csv
|
||||
adaptive_summary.md
|
||||
run_manifest.json
|
||||
tp*/raw_outputs/
|
||||
tp*/metrics/
|
||||
tp*/logs/
|
||||
tp*/gpu_logs/
|
||||
```
|
||||
|
||||
Every successful point is accepted only when `completed == num_prompts` and
|
||||
actual input/output lengths match the requested shape within configured
|
||||
tolerances. Invalid workloads never participate in saturation decisions.
|
||||
@ -0,0 +1,49 @@
|
||||
# Adaptive concurrency search settings.
|
||||
#
|
||||
# For each fixed (TP, DP, ISL, OSL), probe:
|
||||
# C = start, start * multiplier, ... up to max
|
||||
# and stop after Total TPS has less than TPS_MIN_GAIN_PCT meaningful growth for
|
||||
# PLATEAU_PATIENCE consecutive points.
|
||||
|
||||
SEARCH_START_CONCURRENCY="${SEARCH_START_CONCURRENCY:-1}"
|
||||
SEARCH_MAX_CONCURRENCY="${SEARCH_MAX_CONCURRENCY:-512}"
|
||||
SEARCH_MULTIPLIER="${SEARCH_MULTIPLIER:-2}"
|
||||
NUM_PROMPTS_MULTIPLIER="${NUM_PROMPTS_MULTIPLIER:-5}"
|
||||
|
||||
# A gain below 2% is treated as throughput saturation. Two consecutive
|
||||
# low-gain points prevent one noisy measurement from stopping the search.
|
||||
TPS_MIN_GAIN_PCT="${TPS_MIN_GAIN_PCT:-2.0}"
|
||||
PLATEAU_PATIENCE="${PLATEAU_PATIENCE:-2}"
|
||||
|
||||
# Keep the same random workload semantics as the fixed matrix baseline.
|
||||
# DATASET_PATH must contain at least SEARCH_MAX_CONCURRENCY times
|
||||
# NUM_PROMPTS_MULTIPLIER valid two-turn conversations. Set this explicitly to
|
||||
# random-ids to use generated token IDs without a ShareGPT seed dataset.
|
||||
BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}"
|
||||
# SGLang interprets 0.0 as Uniform[1, requested_len]. Use 1.0 for fixed
|
||||
# ISL/OSL points; lower values intentionally benchmark a length distribution.
|
||||
RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1.0}"
|
||||
# Before each measured point, warm up with the same concurrency so lazy kernel
|
||||
# compilation and CUDA graph capture are excluded from TTFT/TPS. 0 means no
|
||||
# cap; set a positive cap only when very high-concurrency warmup is impractical.
|
||||
BENCH_WARMUP_MAX_REQUESTS="${BENCH_WARMUP_MAX_REQUESTS:-0}"
|
||||
|
||||
# Reject a point if the completed request count or actual token lengths do not
|
||||
# match the requested workload.
|
||||
INPUT_LENGTH_TOLERANCE_PCT="${INPUT_LENGTH_TOLERANCE_PCT:-5.0}"
|
||||
OUTPUT_LENGTH_TOLERANCE_PCT="${OUTPUT_LENGTH_TOLERANCE_PCT:-10.0}"
|
||||
|
||||
MAX_POINT_RETRIES="${MAX_POINT_RETRIES:-1}"
|
||||
SERVER_RESTART_COOLDOWN_S="${SERVER_RESTART_COOLDOWN_S:-10}"
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||
|
||||
# Optional space-separated filters, useful for smoke tests:
|
||||
# TP_LIST="8" ISL_LIST="1024" OSL_LIST="128"
|
||||
TP_LIST="${TP_LIST:-}"
|
||||
ISL_LIST="${ISL_LIST:-}"
|
||||
OSL_LIST="${OSL_LIST:-}"
|
||||
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
# Counts ISL/OSL shapes per TP/DP config, not individual concurrency probes.
|
||||
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||
162
experiments/dsv4_h200_sglang_tp_dp_matrix/compare.py
Executable file
162
experiments/dsv4_h200_sglang_tp_dp_matrix/compare.py
Executable file
@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cross TP×DP configuration comparison for dsv4_h200_sglang_tp_dp_matrix.
|
||||
|
||||
Usage:
|
||||
python3 compare.py --run-root results/<run_id> [--output comparison.md]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_result(result_root: Path) -> dict:
|
||||
path = result_root / "results.json"
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def slo_status(ttft_p95_ms: float, tpot_mean_ms: float,
|
||||
ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str:
|
||||
ttft_ok = ttft_p95_ms < ttft_limit_ms
|
||||
tpot_ok = tpot_mean_ms < tpot_limit_ms
|
||||
if ttft_ok and tpot_ok:
|
||||
return "PASS"
|
||||
if ttft_ok or tpot_ok:
|
||||
return "PARTIAL"
|
||||
return "FAIL"
|
||||
|
||||
|
||||
def gpu_memory_str(gpu: dict | None) -> str:
|
||||
if not gpu:
|
||||
return "-"
|
||||
peak = gpu.get("peak_used_mb", 0)
|
||||
total = gpu.get("memory_total_mb", 0)
|
||||
if total:
|
||||
return f"{peak:.0f}/{total:.0f} ({100*peak/total:.1f}%)"
|
||||
return f"{peak:.0f}"
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--run-root", type=Path, required=True)
|
||||
parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md"))
|
||||
parser.add_argument("--ttft-limit", type=float, default=3000.0)
|
||||
parser.add_argument("--tpot-limit", type=float, default=50.0)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Discover configurations: tp*_dp* directories.
|
||||
configs = []
|
||||
for subdir in sorted(args.run_root.iterdir()):
|
||||
if not subdir.is_dir():
|
||||
continue
|
||||
name = subdir.name
|
||||
if not (name.startswith("tp") and "_dp" in name):
|
||||
continue
|
||||
results_json = subdir / "results.json"
|
||||
if not results_json.exists():
|
||||
continue
|
||||
configs.append((name, load_result(subdir)))
|
||||
|
||||
if not configs:
|
||||
print(f"No tp*_dp* results found under {args.run_root}")
|
||||
return
|
||||
|
||||
model = configs[0][1].get("metadata", {}).get("model", "unknown")
|
||||
hardware = configs[0][1].get("metadata", {}).get("hardware", "unknown")
|
||||
|
||||
# Group by scenario name.
|
||||
by_scenario: dict[str, dict[str, dict]] = defaultdict(dict)
|
||||
skipped: dict[str, dict[str, str]] = defaultdict(dict)
|
||||
for label, data in configs:
|
||||
for s in data.get("scenarios", []):
|
||||
key = s["name"]
|
||||
if s.get("status") == "skipped_oom":
|
||||
skipped[key][label] = s.get("note", "skipped")
|
||||
else:
|
||||
by_scenario[key][label] = s
|
||||
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
f.write(f"# SGLang TP×DP matrix comparison ({hardware})\n\n")
|
||||
f.write("## Summary\n\n")
|
||||
f.write(f"- Model: `{model}`\n")
|
||||
f.write(f"- Hardware: {hardware}\n")
|
||||
f.write("- Backend: SGLang (Docker)\n")
|
||||
f.write("- Benchmark client: `sglang.bench_serving`\n")
|
||||
f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n")
|
||||
|
||||
# Configuration overview.
|
||||
f.write("### Configurations\n\n")
|
||||
f.write("| Config | TP | DP | GPUs/replica | Notes |\n")
|
||||
f.write("|---|---:|---:|---:|---|\n")
|
||||
for label, data in configs:
|
||||
cfg = data.get("config", {})
|
||||
tp = cfg.get("tp", "?")
|
||||
dp = cfg.get("dp", "?")
|
||||
f.write(f"| {label} | {tp} | {dp} | {tp} | server args recorded per ISL in results.json |\n")
|
||||
f.write("\n")
|
||||
|
||||
# Side-by-side table.
|
||||
f.write("## Side-by-side results\n\n")
|
||||
headers = [
|
||||
"Scenario", "ISL", "DSL", "Config", "Conc", "Req/s", "OutTok/s",
|
||||
"TTFT P95(ms)", "TTFT P99(ms)", "TPOT Mean(ms)", "TPOT P95(ms)",
|
||||
"TPOT P99(ms)", "E2E P99(ms)", "Peak GPU mem", "SLO"
|
||||
]
|
||||
f.write("| " + " | ".join(headers) + " |\n")
|
||||
f.write("|" + "|".join(["---"] * len(headers)) + "|\n")
|
||||
|
||||
for scenario_name in sorted(by_scenario.keys(), key=lambda x: tuple(map(int, re.findall(r"\d+", x)))):
|
||||
_, isl, dsl = re.findall(r"\d+", scenario_name)
|
||||
for label, data in configs:
|
||||
s = by_scenario[scenario_name].get(label)
|
||||
if s is None:
|
||||
if scenario_name in skipped and label in skipped[scenario_name]:
|
||||
note = skipped[scenario_name][label]
|
||||
f.write(f"| {scenario_name} | {isl} | {dsl} | {label} | - | - | - | - | - | - | - | - | - | - | {note} |\n")
|
||||
continue
|
||||
cfg = s["config"]
|
||||
m = s["metrics"]
|
||||
status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
|
||||
gpu = m.get("gpu_memory")
|
||||
f.write(
|
||||
f"| {scenario_name} | {isl} | {dsl} | {label} | {cfg['concurrency']} | "
|
||||
f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||
f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||
f"{m['e2e_ms']['p99']:.2f} | {gpu_memory_str(gpu)} | {status} |\n"
|
||||
)
|
||||
|
||||
# Best throughput per ISL/DSL.
|
||||
f.write("\n## Best throughput per (ISL, DSL)\n\n")
|
||||
f.write("| ISL | DSL | Best Config | Concurrency | OutTok/s | TTFT P95(ms) | TPOT Mean(ms) | SLO |\n")
|
||||
f.write("|---:|---:|---|---:|---:|---:|---:|---:|\n")
|
||||
best_by_shape: dict[tuple[int, int], tuple[float, str, dict]] = {}
|
||||
for scenario_name, backends in by_scenario.items():
|
||||
_, isl, dsl = re.findall(r"\d+", scenario_name)
|
||||
isl_i, dsl_i = int(isl), int(dsl)
|
||||
for label, s in backends.items():
|
||||
m = s["metrics"]
|
||||
out_tok = m["output_token_throughput"]
|
||||
if (isl_i, dsl_i) not in best_by_shape or out_tok > best_by_shape[(isl_i, dsl_i)][0]:
|
||||
best_by_shape[(isl_i, dsl_i)] = (out_tok, label, s)
|
||||
for (isl_i, dsl_i), (out_tok, label, s) in sorted(best_by_shape.items()):
|
||||
m = s["metrics"]
|
||||
status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
|
||||
f.write(
|
||||
f"| {isl_i} | {dsl_i} | {label} | {s['config']['concurrency']} | "
|
||||
f"{out_tok:.2f} | {m['ttft_ms']['p95']:.2f} | {m['tpot_ms']['mean']:.2f} | {status} |\n"
|
||||
)
|
||||
|
||||
f.write("\n## Notes\n\n")
|
||||
f.write("- SLO check uses TTFT P95 and TPOT mean.\n")
|
||||
f.write("- A PARTIAL indicates one of the two metrics is out of target; FAIL indicates both are out.\n")
|
||||
f.write("- `Peak GPU mem` shows peak used / total MB and utilization percentage.\n")
|
||||
f.write("- Optional (P) combinations that failed are marked as skipped/OOM and do not break the run.\n")
|
||||
|
||||
print(f"Wrote comparison to {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
70
experiments/dsv4_h200_sglang_tp_dp_matrix/config.env
Normal file
70
experiments/dsv4_h200_sglang_tp_dp_matrix/config.env
Normal file
@ -0,0 +1,70 @@
|
||||
# TP×DP matrix experiment for DeepSeek-V4-Flash on H200 (8 GPUs) using SGLang.
|
||||
# Tests SGLang with three parallel configurations:
|
||||
# TP=2, DP=4 -> 2 GPUs per replica, 4 replicas
|
||||
# TP=4, DP=2 -> 4 GPUs per replica, 2 replicas
|
||||
# TP=8, DP=1 -> 8 GPUs, no data parallelism
|
||||
|
||||
EXPERIMENT="dsv4_h200_sglang_tp_dp_matrix"
|
||||
MODEL_NAME="DeepSeek-V4-Flash"
|
||||
MODEL_PATH="/data3/hf_models/DeepSeek-V4-Flash"
|
||||
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
SGLANG_PORT="${SGLANG_PORT:-30031}"
|
||||
|
||||
# Python interpreter for orchestration scripts (parse_backend.py, compare.py, etc.)
|
||||
# and the benchmark client. Defaults to the system python3 if the sglang venv
|
||||
# does not exist on the host.
|
||||
VENV_CLIENT="${VENV_CLIENT:-/data/user1/yy/envs/sglang}"
|
||||
|
||||
# Run the benchmark client natively (0) or inside Docker (1).
|
||||
USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-1}"
|
||||
|
||||
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
|
||||
|
||||
# Runtime working directory for logs, pid files, and tmp. Defaults to a local
|
||||
# directory under this experiment so the benchmark is self-contained.
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
|
||||
# Parallel configurations to test. Format: "TP DP"
|
||||
declare -a PARALLEL_CONFIGS=(
|
||||
"2 4"
|
||||
"4 2"
|
||||
"8 1"
|
||||
)
|
||||
|
||||
# SGLang server settings
|
||||
CONTEXT_LENGTH="${CONTEXT_LENGTH:-1048576}"
|
||||
MAX_RUNNING_REQUESTS="${MAX_RUNNING_REQUESTS:-128}"
|
||||
MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.88}"
|
||||
MOE_RUNNER_BACKEND="${MOE_RUNNER_BACKEND:-marlin}"
|
||||
|
||||
# Deployment switch. 0 = native sglang venv, 1 = Docker.
|
||||
USE_DOCKER="${USE_DOCKER:-1}"
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
# Dataset used by sglang.bench_serving --dataset-name random.
|
||||
# The random sampler needs a ShareGPT-style JSON file locally; it falls back to
|
||||
# downloading from HuggingFace, which usually fails on offline H200 nodes.
|
||||
DATASET_PATH="${DATASET_PATH:-/data3/yy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json}"
|
||||
|
||||
# Matrix and concurrency rules are defined in matrix.json by default.
|
||||
MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR:-.}/matrix.json}"
|
||||
MATRIX_MODE="${MATRIX_MODE:-Y}"
|
||||
|
||||
# Sampling density for concurrency.
|
||||
# 0 = use the default heuristic in generate_scenarios.py (6-8 points).
|
||||
# 2 = only test the low and high endpoints.
|
||||
export CONCURRENCY_SAMPLES="${CONCURRENCY_SAMPLES:-2}"
|
||||
|
||||
# Per-scenario timeout to avoid hangs (seconds).
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||
|
||||
# GPU memory sampling interval (seconds).
|
||||
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||
|
||||
# Dry-run mode: if 1, only log the server args and scenario plan without starting
|
||||
# any server or sending requests.
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
|
||||
# Per-config scenario limit for quick smoke tests. 0 = run all generated scenarios.
|
||||
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||
98
experiments/dsv4_h200_sglang_tp_dp_matrix/generate_scenarios.py
Executable file
98
experiments/dsv4_h200_sglang_tp_dp_matrix/generate_scenarios.py
Executable file
@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate the scenario list for the TP×DP matrix experiment.
|
||||
|
||||
Reads matrix.json and prints TSV lines:
|
||||
|
||||
mark input_len output_len concurrency num_prompts
|
||||
|
||||
mark is one of Y/P/N. The caller (run_bench.sh) decides how to treat each.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def sample_concurrency(low: int, high: int, target: int) -> list[int]:
|
||||
"""Return only the low and high concurrency values in [low, high].
|
||||
|
||||
For the TP×DP matrix we only need the two endpoints of the concurrency
|
||||
range (e.g. 1 and 128 for ISL=1024). The `target` argument is kept for
|
||||
API compatibility but is ignored.
|
||||
"""
|
||||
assert 1 <= low <= high, f"invalid concurrency range: {low}-{high}"
|
||||
if low == high:
|
||||
return [low]
|
||||
return [low, high]
|
||||
|
||||
|
||||
def generate_scenarios(matrix_path: Path, mode: str, target_samples: int) -> list[dict]:
|
||||
with open(matrix_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
matrix = data["matrix"]
|
||||
concurrency_cfg = data["concurrency"]
|
||||
|
||||
scenarios = []
|
||||
for isl_str in sorted(matrix.keys(), key=int):
|
||||
dsl_map = matrix[isl_str]
|
||||
low = concurrency_cfg[isl_str]["low"]
|
||||
high = concurrency_cfg[isl_str]["high"]
|
||||
concurrencies = sample_concurrency(low, high, target_samples)
|
||||
|
||||
for dsl_str in sorted(dsl_map.keys(), key=int):
|
||||
mark = dsl_map[dsl_str]
|
||||
if mode == "Y" and mark != "Y":
|
||||
continue
|
||||
if mode == "Y+P" and mark not in ("Y", "P"):
|
||||
continue
|
||||
# mode == "all" keeps everything, including N.
|
||||
|
||||
for conc in concurrencies:
|
||||
scenarios.append(
|
||||
{
|
||||
"mark": mark,
|
||||
"input_len": int(isl_str),
|
||||
"output_len": int(dsl_str),
|
||||
"concurrency": conc,
|
||||
"num_prompts": conc * 5,
|
||||
}
|
||||
)
|
||||
|
||||
return scenarios
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--matrix", type=Path, default=Path("matrix.json"))
|
||||
parser.add_argument("--mode", choices=["Y", "Y+P", "all"], default=None,
|
||||
help="Scenario selection mode. Defaults to matrix.mode.")
|
||||
parser.add_argument("--target-samples", type=int, default=0,
|
||||
help="Target number of concurrency samples. 0 = heuristic (6-8).")
|
||||
args = parser.parse_args()
|
||||
|
||||
with open(args.matrix, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
mode = args.mode if args.mode else data.get("mode", "Y+P")
|
||||
|
||||
target_samples = args.target_samples
|
||||
if target_samples <= 0:
|
||||
env_samples = os.getenv("CONCURRENCY_SAMPLES", "0")
|
||||
try:
|
||||
target_samples = int(env_samples)
|
||||
except ValueError:
|
||||
target_samples = 0
|
||||
if target_samples <= 0:
|
||||
target_samples = 7
|
||||
|
||||
scenarios = generate_scenarios(args.matrix, mode, target_samples)
|
||||
|
||||
print("mark\tinput_len\toutput_len\tconcurrency\tnum_prompts")
|
||||
for s in scenarios:
|
||||
print(f"{s['mark']}\t{s['input_len']}\t{s['output_len']}\t{s['concurrency']}\t{s['num_prompts']}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
81
experiments/dsv4_h200_sglang_tp_dp_matrix/matrix.json
Normal file
81
experiments/dsv4_h200_sglang_tp_dp_matrix/matrix.json
Normal file
@ -0,0 +1,81 @@
|
||||
{
|
||||
"comment": "ISL/DSL matrix for dsv4_h200_sglang_tp_dp_matrix. Y=must test, P=optional (record N on failure), N=skip.",
|
||||
"mode": "Y",
|
||||
"comment": "Only mandatory (Y) combinations are tested; 1M ISL is excluded per user request.",
|
||||
"matrix": {
|
||||
"1024": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "Y"
|
||||
},
|
||||
"4096": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "Y"
|
||||
},
|
||||
"16384": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "P"
|
||||
},
|
||||
"65536": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "P",
|
||||
"4096": "N"
|
||||
},
|
||||
"131072": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "P",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
},
|
||||
"262144": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "P",
|
||||
"1024": "N",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
},
|
||||
"524288": {
|
||||
"128": "Y",
|
||||
"256": "P",
|
||||
"512": "N",
|
||||
"1024": "N",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
},
|
||||
"1048576": {
|
||||
"128": "Y",
|
||||
"256": "P",
|
||||
"512": "N",
|
||||
"1024": "N",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
}
|
||||
},
|
||||
"concurrency": {
|
||||
"1024": { "low": 1, "high": 128 },
|
||||
"4096": { "low": 1, "high": 64 },
|
||||
"16384": { "low": 1, "high": 32 },
|
||||
"65536": { "low": 1, "high": 8 },
|
||||
"131072": { "low": 1, "high": 4 },
|
||||
"262144": { "low": 1, "high": 2 },
|
||||
"524288": { "low": 1, "high": 2 },
|
||||
"1048576": { "low": 1, "high": 2 }
|
||||
}
|
||||
}
|
||||
181
experiments/dsv4_h200_sglang_tp_dp_matrix/run_adaptive_concurrency.sh
Executable file
181
experiments/dsv4_h200_sglang_tp_dp_matrix/run_adaptive_concurrency.sh
Executable file
@ -0,0 +1,181 @@
|
||||
#!/usr/bin/env bash
|
||||
# Find the Total-TPS saturation concurrency for each SGLang TP/DP/ISL/OSL shape.
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/adaptive_bench_lib.sh"
|
||||
|
||||
ENGINE="sglang"
|
||||
ENGINE_PORT="$SGLANG_PORT"
|
||||
RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
|
||||
if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
|
||||
PYTHON="${VENV_CLIENT}/bin/python"
|
||||
else
|
||||
PYTHON="$(command -v python3)"
|
||||
fi
|
||||
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
engine_is_healthy() {
|
||||
curl --fail --silent --show-error --max-time 5 \
|
||||
"http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
engine_stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_sglang_tp${tp}_dp${dp}.pid"
|
||||
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||
if docker exec "$CONTAINER_NAME" kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping sglang in persistent container pid=${pid} tp=${tp} dp=${dp}"
|
||||
docker exec "$CONTAINER_NAME" kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
docker exec "$CONTAINER_NAME" kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
elif kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping sglang server pid=${pid} tp=${tp} dp=${dp}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
if [[ -z "${CONTAINER_NAME:-}" ]]; then
|
||||
docker rm -f "${EXPERIMENT}_sglang_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
fi
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
sleep 2
|
||||
}
|
||||
|
||||
engine_build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local -a args=(
|
||||
sglang serve --model-path "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--tp-size "$tp"
|
||||
--moe-runner-backend "$MOE_RUNNER_BACKEND"
|
||||
--context-length "$CONTEXT_LENGTH"
|
||||
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||
--mem-fraction-static "$MEM_FRACTION_STATIC"
|
||||
--host 0.0.0.0
|
||||
--port "$ENGINE_PORT"
|
||||
)
|
||||
if (( dp > 1 )); then
|
||||
args+=(--dp-size "$dp")
|
||||
fi
|
||||
printf '%q ' "${args[@]}"
|
||||
}
|
||||
|
||||
engine_start_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/sglang_tp${tp}_dp${dp}.server.outer.log"
|
||||
log "starting sglang server tp=${tp} dp=${dp}"
|
||||
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||
bash "${SCRIPT_DIR}/run_sglang_in_container.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||
else
|
||||
bash "${SCRIPT_DIR}/start_sglang_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||
fi
|
||||
if ! engine_is_healthy; then
|
||||
log "ERROR: sglang health check failed tp=${tp} dp=${dp}"
|
||||
return 1
|
||||
fi
|
||||
if [[ -z "${CONTAINER_NAME:-}" ]]; then
|
||||
ACTIVE_ENGINE_SERVER_LOG="$(
|
||||
find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
|
||||
-name "${EXPERIMENT}_sglang*tp${tp}_dp${dp}_*.log" \
|
||||
-printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
|
||||
)"
|
||||
fi
|
||||
log "sglang server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-container:/tmp/sglang_server.log}"
|
||||
}
|
||||
|
||||
engine_detect_oom() {
|
||||
local detail_log="$1"
|
||||
local tp="$2"
|
||||
local dp="$3"
|
||||
local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/sglang_tp${tp}_dp${dp}.server.outer.log"
|
||||
local -a logs=("$detail_log" "$outer_log")
|
||||
if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
|
||||
logs+=("$ACTIVE_ENGINE_SERVER_LOG")
|
||||
fi
|
||||
if grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||
docker exec "$CONTAINER_NAME" grep -Eiq "$pattern" /tmp/sglang_server.log 2>/dev/null
|
||||
return $?
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
engine_run_bench() {
|
||||
local isl="$1"
|
||||
local osl="$2"
|
||||
local concurrency="$3"
|
||||
local num_prompts="$4"
|
||||
local output_file="$5"
|
||||
local warmup_requests
|
||||
warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
|
||||
local -a bench_args=(
|
||||
--backend sglang
|
||||
--host 127.0.0.1
|
||||
--port "$ENGINE_PORT"
|
||||
--dataset-name "$BENCH_DATASET_NAME"
|
||||
--random-input-len "$isl"
|
||||
--random-output-len "$osl"
|
||||
--random-range-ratio "$RANDOM_RANGE_RATIO"
|
||||
--num-prompts "$num_prompts"
|
||||
--max-concurrency "$concurrency"
|
||||
--request-rate 10000
|
||||
--warmup-requests "$warmup_requests"
|
||||
--output-file "$output_file"
|
||||
--output-details
|
||||
--disable-tqdm
|
||||
)
|
||||
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||
bench_args+=(--dataset-path "$DATASET_PATH")
|
||||
else
|
||||
bench_args+=(--tokenize-prompt)
|
||||
fi
|
||||
|
||||
if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then
|
||||
local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}")
|
||||
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||
volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro")
|
||||
fi
|
||||
docker run --rm \
|
||||
--network host \
|
||||
"${volume_args[@]}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
"$DOCKER_IMAGE" \
|
||||
python -m sglang.bench_serving "${bench_args[@]}"
|
||||
else
|
||||
"$PYTHON" -m sglang.bench_serving "${bench_args[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
export -f engine_run_bench
|
||||
export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_IMAGE USE_DOCKER_CLIENT
|
||||
export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
|
||||
|
||||
adaptive_main "$@"
|
||||
184
experiments/dsv4_h200_sglang_tp_dp_matrix/run_adaptive_concurrency_add16.sh
Executable file
184
experiments/dsv4_h200_sglang_tp_dp_matrix/run_adaptive_concurrency_add16.sh
Executable file
@ -0,0 +1,184 @@
|
||||
#!/usr/bin/env bash
|
||||
# Find the Total-TPS saturation concurrency for each SGLang TP/DP/ISL/DSL shape.
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/adaptive_bench_lib.sh"
|
||||
|
||||
ENGINE="sglang"
|
||||
ENGINE_PORT="$SGLANG_PORT"
|
||||
RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
|
||||
if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
|
||||
PYTHON="${VENV_CLIENT}/bin/python"
|
||||
else
|
||||
PYTHON="$(command -v python3)"
|
||||
fi
|
||||
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
engine_is_healthy() {
|
||||
curl --fail --silent --show-error --max-time 5 \
|
||||
"http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
engine_stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_sglang_tp${tp}_dp${dp}.pid"
|
||||
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||
if docker exec "$CONTAINER_NAME" kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping sglang in persistent container pid=${pid} tp=${tp} dp=${dp}"
|
||||
docker exec "$CONTAINER_NAME" kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
docker exec "$CONTAINER_NAME" kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
elif kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping sglang server pid=${pid} tp=${tp} dp=${dp}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
if [[ -z "${CONTAINER_NAME:-}" ]]; then
|
||||
docker rm -f "${EXPERIMENT}_sglang_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
fi
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
sleep 2
|
||||
}
|
||||
|
||||
engine_build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local -a args=(
|
||||
sglang serve --model-path "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--tp-size "$tp"
|
||||
--moe-runner-backend "$MOE_RUNNER_BACKEND"
|
||||
--context-length "$CONTEXT_LENGTH"
|
||||
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||
--mem-fraction-static "$MEM_FRACTION_STATIC"
|
||||
--host 0.0.0.0
|
||||
--port "$ENGINE_PORT"
|
||||
)
|
||||
if (( dp > 1 )); then
|
||||
args+=(--dp-size "$dp")
|
||||
fi
|
||||
printf '%q ' "${args[@]}"
|
||||
}
|
||||
|
||||
engine_start_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/sglang_tp${tp}_dp${dp}.server.outer.log"
|
||||
log "starting sglang server tp=${tp} dp=${dp}"
|
||||
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||
bash "${SCRIPT_DIR}/run_sglang_in_container.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||
else
|
||||
bash "${SCRIPT_DIR}/start_sglang_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||
fi
|
||||
if ! engine_is_healthy; then
|
||||
log "ERROR: sglang health check failed tp=${tp} dp=${dp}"
|
||||
return 1
|
||||
fi
|
||||
if [[ -z "${CONTAINER_NAME:-}" ]]; then
|
||||
ACTIVE_ENGINE_SERVER_LOG="$(
|
||||
find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
|
||||
-name "${EXPERIMENT}_sglang*tp${tp}_dp${dp}_*.log" \
|
||||
-printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
|
||||
)"
|
||||
fi
|
||||
log "sglang server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-container:/tmp/sglang_server.log}"
|
||||
}
|
||||
|
||||
engine_detect_oom() {
|
||||
local detail_log="$1"
|
||||
local tp="$2"
|
||||
local dp="$3"
|
||||
local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/sglang_tp${tp}_dp${dp}.server.outer.log"
|
||||
local -a logs=("$detail_log" "$outer_log")
|
||||
if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
|
||||
logs+=("$ACTIVE_ENGINE_SERVER_LOG")
|
||||
fi
|
||||
if grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||
docker exec "$CONTAINER_NAME" grep -Eiq "$pattern" /tmp/sglang_server.log 2>/dev/null
|
||||
return $?
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
engine_run_bench() {
|
||||
local isl="$1"
|
||||
local dsl="$2"
|
||||
local concurrency="$3"
|
||||
local num_prompts="$4"
|
||||
local output_file="$5"
|
||||
local warmup_requests
|
||||
warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
|
||||
local -a bench_args=(
|
||||
--backend sglang
|
||||
--host 127.0.0.1
|
||||
--port "$ENGINE_PORT"
|
||||
--dataset-name "$BENCH_DATASET_NAME"
|
||||
--random-input-len "$isl"
|
||||
--random-output-len "$dsl"
|
||||
--random-range-ratio "$RANDOM_RANGE_RATIO"
|
||||
--num-prompts "$num_prompts"
|
||||
--max-concurrency "$concurrency"
|
||||
--request-rate 10000
|
||||
--warmup-requests "$warmup_requests"
|
||||
--output-file "$output_file"
|
||||
--output-details
|
||||
--disable-tqdm
|
||||
)
|
||||
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||
bench_args+=(--dataset-path "$DATASET_PATH")
|
||||
else
|
||||
bench_args+=(--tokenize-prompt)
|
||||
fi
|
||||
|
||||
if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then
|
||||
local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}")
|
||||
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||
volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro")
|
||||
fi
|
||||
docker run --rm \
|
||||
--network host \
|
||||
"${volume_args[@]}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
"$DOCKER_IMAGE" \
|
||||
python -m sglang.bench_serving "${bench_args[@]}"
|
||||
else
|
||||
"$PYTHON" -m sglang.bench_serving "${bench_args[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
export -f engine_run_bench
|
||||
export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_IMAGE USE_DOCKER_CLIENT
|
||||
export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
|
||||
|
||||
export SEARCH_START_CONCURRENCY=16
|
||||
export SEARCH_ADDEND=16
|
||||
|
||||
adaptive_main "$@"
|
||||
558
experiments/dsv4_h200_sglang_tp_dp_matrix/run_bench.sh
Executable file
558
experiments/dsv4_h200_sglang_tp_dp_matrix/run_bench.sh
Executable file
@ -0,0 +1,558 @@
|
||||
#!/usr/bin/env bash
|
||||
# TP×DP matrix benchmark for DeepSeek-V4-Flash on SGLang (Docker).
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||
RESULT_BASE="${SCRIPT_DIR}/results"
|
||||
MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR}/matrix.json}"
|
||||
MATRIX_MODE="${MATRIX_MODE:-Y}"
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||
|
||||
# Export variables used inside functions that are called via bash -c subshells.
|
||||
export DATASET_PATH MODEL_PATH RESULT_BASE DOCKER_IMAGE USE_DOCKER_CLIENT
|
||||
|
||||
if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
|
||||
PYTHON="${VENV_CLIENT}/bin/python"
|
||||
else
|
||||
PYTHON="$(command -v python3)"
|
||||
fi
|
||||
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
log_dir_global="${RESULT_BASE}/${RUN_ID}/logs"
|
||||
mkdir -p "$log_dir_global"
|
||||
log_init "${log_dir_global}/orchestrator.log"
|
||||
|
||||
log "experiment=${EXPERIMENT_NAME} run_id=${RUN_ID} platform=${PLATFORM} hardware=${HARDWARE}"
|
||||
log "matrix_mode=${MATRIX_MODE} matrix_file=${MATRIX_FILE} dry_run=${DRY_RUN} grid_limit=${GRID_LIMIT}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
is_server_healthy() {
|
||||
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${SGLANG_PORT}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_sglang_tp${tp}_dp${dp}.pid"
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
# In container-reuse mode, pid is the server PID inside the container.
|
||||
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||
if docker exec "$CONTAINER_NAME" kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping sglang server inside container (tp=${tp}, dp=${dp}, pid=${pid})"
|
||||
docker exec "$CONTAINER_NAME" kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
docker exec "$CONTAINER_NAME" kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
else
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping sglang server pid=${pid} (tp=${tp}, dp=${dp})"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
|
||||
# Fallback: remove any Docker container started by this experiment (legacy mode).
|
||||
if [[ -z "${CONTAINER_NAME:-}" ]]; then
|
||||
docker rm -f "${EXPERIMENT}_sglang_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
fi
|
||||
|
||||
# Fallback: kill any sglang serve processes for this model.
|
||||
pkill -9 -f "sglang serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
pkill -9 -f "sglang serve.*${MODEL_PATH}" 2>/dev/null || true
|
||||
sleep 2
|
||||
}
|
||||
|
||||
build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
|
||||
local args=(
|
||||
"sglang serve" "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--tp-size "$tp"
|
||||
--moe-runner-backend "$MOE_RUNNER_BACKEND"
|
||||
--context-length "$CONTEXT_LENGTH"
|
||||
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||
--mem-fraction-static "$MEM_FRACTION_STATIC"
|
||||
--host 0.0.0.0
|
||||
--port "$SGLANG_PORT"
|
||||
)
|
||||
if [[ "$dp" -gt 1 ]]; then
|
||||
args+=(
|
||||
--dp-size "$dp"
|
||||
)
|
||||
fi
|
||||
printf '%s ' "${args[@]}"
|
||||
}
|
||||
|
||||
start_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
|
||||
log "starting sglang server tp=${tp} dp=${dp}"
|
||||
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||
bash "${SCRIPT_DIR}/run_sglang_in_container.sh" "$tp" "$dp" \
|
||||
>> "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1
|
||||
else
|
||||
bash "${SCRIPT_DIR}/start_sglang_dp.sh" "$tp" "$dp" \
|
||||
>> "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1
|
||||
fi
|
||||
|
||||
if ! is_server_healthy; then
|
||||
log "error: sglang server tp=${tp} dp=${dp} failed health check on port ${SGLANG_PORT}"
|
||||
return 1
|
||||
fi
|
||||
log "sglang server tp=${tp} dp=${dp} is healthy on port ${SGLANG_PORT}"
|
||||
}
|
||||
|
||||
restart_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
log "restarting sglang server tp=${tp} dp=${dp} after non-OOM failure"
|
||||
stop_server "$tp" "$dp"
|
||||
sleep 10
|
||||
start_server "$tp" "$dp"
|
||||
}
|
||||
|
||||
run_bench_serving() {
|
||||
# Run sglang.bench_serving either natively or inside the SGLang Docker image.
|
||||
if [[ "${USE_DOCKER_CLIENT:-1}" == "1" ]]; then
|
||||
local vol_args=()
|
||||
vol_args+=("-v" "${MODEL_PATH}:${MODEL_PATH}:ro")
|
||||
if [[ -n "${DATASET_PATH:-}" ]]; then
|
||||
vol_args+=("-v" "${DATASET_PATH}:${DATASET_PATH}:ro")
|
||||
fi
|
||||
vol_args+=("-v" "${RESULT_BASE}:${RESULT_BASE}")
|
||||
docker run --rm \
|
||||
--network host \
|
||||
"${vol_args[@]}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
"${DOCKER_IMAGE}" \
|
||||
python -m sglang.bench_serving "$@"
|
||||
else
|
||||
"$PYTHON" -m sglang.bench_serving "$@"
|
||||
fi
|
||||
}
|
||||
export -f run_bench_serving
|
||||
|
||||
run_warmup() {
|
||||
local input_len="$1"
|
||||
local output_len="$2"
|
||||
|
||||
log "warming up (input=${input_len}, output=${output_len}, num=1)"
|
||||
bash -c '
|
||||
run_bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port "'"$SGLANG_PORT"'" \
|
||||
--dataset-name random \
|
||||
--dataset-path "'"$DATASET_PATH"'" \
|
||||
--random-input-len "'"$input_len"'" \
|
||||
--random-output-len "'"$output_len"'" \
|
||||
--num-prompts 1 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate 10000 \
|
||||
--output-file /dev/null \
|
||||
--output-details \
|
||||
>> "'"${log_dir_global}/warmup.log"'" 2>&1
|
||||
'
|
||||
log "warmup completed"
|
||||
}
|
||||
|
||||
scenario_already_completed() {
|
||||
local output_file="$1"
|
||||
local expected="$2"
|
||||
[[ -s "$output_file" ]] || return 1
|
||||
local completed
|
||||
completed="$("$PYTHON" -c "
|
||||
import json, sys
|
||||
path = sys.argv[1]
|
||||
try:
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
data = json.loads(line)
|
||||
print(data.get('completed', 0))
|
||||
break
|
||||
except Exception:
|
||||
print(0)
|
||||
" "$output_file")"
|
||||
[[ "${completed:-0}" -ge "$expected" ]]
|
||||
}
|
||||
|
||||
scenario_already_processed() {
|
||||
local result_root="$1"
|
||||
local scenario_name="$2"
|
||||
local json_path="${result_root}/results.json"
|
||||
[[ -f "$json_path" ]] || return 1
|
||||
"$PYTHON" -c "
|
||||
import json, sys
|
||||
path, name = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
for s in data.get('scenarios', []):
|
||||
if s.get('name') == name:
|
||||
if s.get('status') or s.get('metrics', {}).get('success', 0) > 0:
|
||||
sys.exit(0)
|
||||
except Exception:
|
||||
pass
|
||||
sys.exit(1)
|
||||
" "$json_path" "$scenario_name"
|
||||
}
|
||||
|
||||
detect_oom() {
|
||||
local detail_log="$1"
|
||||
local server_outer_log="$2"
|
||||
local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
|
||||
if grep -Eiq "$pattern" "$detail_log" "$server_outer_log" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
start_gpu_monitor() {
|
||||
local csv_path="$1"
|
||||
mkdir -p "$(dirname "$csv_path")"
|
||||
nvidia-smi \
|
||||
--query-gpu=timestamp,index,memory.used,memory.total,utilization.gpu \
|
||||
--format=csv \
|
||||
-l "$GPU_MEM_SAMPLE_INTERVAL_S" \
|
||||
> "$csv_path" 2>/dev/null &
|
||||
echo $!
|
||||
}
|
||||
|
||||
stop_gpu_monitor() {
|
||||
local pid="$1"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 1
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
|
||||
append_scenario_record() {
|
||||
local result_root="$1"
|
||||
local json_path="$result_root/results.json"
|
||||
shift
|
||||
local scenario_json
|
||||
scenario_json="$("$PYTHON" -c "
|
||||
import json, sys
|
||||
pairs = [a.split('=', 1) for a in sys.argv[1:]]
|
||||
d = {}
|
||||
for k, v in pairs:
|
||||
try:
|
||||
d[k] = json.loads(v)
|
||||
except json.JSONDecodeError:
|
||||
d[k] = v
|
||||
print(json.dumps(d, ensure_ascii=False))
|
||||
" "$@")"
|
||||
PYTHON="$PYTHON" append_scenario_to_json "$json_path" "$scenario_json"
|
||||
}
|
||||
|
||||
record_skipped_csv() {
|
||||
local csv_path="$1"
|
||||
shift
|
||||
# Args: key=value
|
||||
local row
|
||||
row="$("$PYTHON" -c "
|
||||
import csv, json, sys, io
|
||||
pairs = [a.split('=', 1) for a in sys.argv[1:]]
|
||||
d = {}
|
||||
for k, v in pairs:
|
||||
try:
|
||||
d[k] = json.loads(v)
|
||||
except json.JSONDecodeError:
|
||||
d[k] = v
|
||||
buf = io.StringIO()
|
||||
writer = csv.DictWriter(buf, fieldnames=['engine','tp','dp','mark','isl','dsl','concurrency','status','reason','detail_log'], extrasaction='ignore')
|
||||
writer.writerow(d)
|
||||
print(buf.getvalue().strip())
|
||||
" "$@")"
|
||||
echo "$row" >> "$csv_path"
|
||||
}
|
||||
|
||||
skip_remaining_scenarios() {
|
||||
local result_root="$1"
|
||||
local scenario_tsv="$2"
|
||||
local start_index="$3"
|
||||
local status="$4"
|
||||
local reason="$5"
|
||||
local tp="$6"
|
||||
local dp="$7"
|
||||
local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
|
||||
|
||||
local i=0
|
||||
tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl dsl conc num; do
|
||||
if (( i < start_index )); then
|
||||
i=$((i + 1))
|
||||
continue
|
||||
fi
|
||||
i=$((i + 1))
|
||||
local sname="c${conc}_i${isl}_o${dsl}"
|
||||
if scenario_already_processed "$result_root" "$sname"; then
|
||||
continue
|
||||
fi
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"${status}\"" \
|
||||
"note=\"${reason}\""
|
||||
record_skipped_csv "$skipped_csv" \
|
||||
"engine=sglang" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=${status}" "reason=${reason}"
|
||||
done
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-configuration runner
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
run_parallel_config() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local config_label="tp${tp}_dp${dp}"
|
||||
local result_root="${RESULT_BASE}/${RUN_ID}/${config_label}"
|
||||
local raw_dir="${result_root}/raw_outputs"
|
||||
local gpu_log_dir="${result_root}/gpu_logs"
|
||||
local phase_log_dir="${result_root}/logs"
|
||||
mkdir -p "$raw_dir" "$gpu_log_dir" "$phase_log_dir"
|
||||
|
||||
log "===== ${config_label} START ====="
|
||||
|
||||
# Generate scenario list for this config.
|
||||
local scenario_tsv="${result_root}/scenarios.tsv"
|
||||
"$PYTHON" "${SCRIPT_DIR}/generate_scenarios.py" \
|
||||
--matrix "$MATRIX_FILE" \
|
||||
--mode "$MATRIX_MODE" \
|
||||
> "$scenario_tsv"
|
||||
local total_scenarios
|
||||
total_scenarios="$(tail -n +2 "$scenario_tsv" | wc -l)"
|
||||
log "generated ${total_scenarios} scenarios for ${config_label}"
|
||||
|
||||
# Write metadata.
|
||||
ensure_result_root "$result_root"
|
||||
write_metadata_json \
|
||||
"${result_root}/results.json" \
|
||||
"${EXPERIMENT_NAME}_${config_label}" \
|
||||
"$RUN_ID" \
|
||||
"$MODEL_PATH" \
|
||||
"sglang" \
|
||||
"sglang" \
|
||||
"$HARDWARE" \
|
||||
"$ACCELERATOR" \
|
||||
"$CHIP" \
|
||||
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||
"$DOCKER_IMAGE" \
|
||||
"H200 SGLang TP×DP matrix for DeepSeek-V4-Flash"
|
||||
|
||||
local server_args_str
|
||||
server_args_str="$(build_server_args "$tp" "$dp")"
|
||||
jq --arg tp "$tp" --arg dp "$dp" --arg cuda "$CUDA_VISIBLE_DEVICES" --arg args "$server_args_str" \
|
||||
'.config = {
|
||||
"tp": ($tp | tonumber),
|
||||
"dp": ($dp | tonumber),
|
||||
"cuda_visible_devices": $cuda,
|
||||
"backend": "sglang",
|
||||
"server_start_script": "experiments/'${EXPERIMENT_NAME}'/start_sglang_dp.sh",
|
||||
"server_args": $args
|
||||
}' "${result_root}/results.json" > "${result_root}/results.json.tmp" && \
|
||||
mv "${result_root}/results.json.tmp" "${result_root}/results.json"
|
||||
|
||||
if [[ "$DRY_RUN" == "1" ]]; then
|
||||
log "DRY_RUN: would start server with args: ${server_args_str}"
|
||||
local line
|
||||
tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl dsl conc num; do
|
||||
log "DRY_RUN: ${config_label} scenario mark=${mark} c=${conc} i=${isl} o=${dsl} n=${num}"
|
||||
done
|
||||
log "===== ${config_label} DONE (dry run) ====="
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Initialize skipped_after_oom.csv for this run.
|
||||
local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
|
||||
if [[ ! -f "$skipped_csv" ]]; then
|
||||
echo "engine,tp,dp,mark,isl,dsl,concurrency,status,reason,detail_log" > "$skipped_csv"
|
||||
fi
|
||||
|
||||
# Start server once for this TP×DP config.
|
||||
if ! start_server "$tp" "$dp"; then
|
||||
log "ERROR: ${config_label} failed to start; skipping all scenarios"
|
||||
skip_remaining_scenarios "$result_root" "$scenario_tsv" 0 "SKIPPED_SERVICE_START_FAILED" "service failed to start" "$tp" "$dp"
|
||||
log "===== ${config_label} DONE ====="
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Warmup with a small prompt before the first scenario.
|
||||
run_warmup 1024 128 || true
|
||||
|
||||
# Read scenarios into an array so we can skip remaining entries on failure.
|
||||
local -a scenarios=()
|
||||
while IFS= read -r line; do
|
||||
scenarios+=("$line")
|
||||
done < <(tail -n +2 "$scenario_tsv")
|
||||
|
||||
local i mark isl dsl conc num
|
||||
local output_file detail_log gpu_csv sname bench_rc
|
||||
for (( i = 0; i < ${#scenarios[@]}; i++ )); do
|
||||
IFS=$'\t' read -r mark isl dsl conc num <<< "${scenarios[$i]}"
|
||||
|
||||
if [[ "$GRID_LIMIT" -gt 0 && "$i" -ge "$GRID_LIMIT" ]]; then
|
||||
log "GRID_LIMIT=${GRID_LIMIT} reached; skipping remaining scenarios"
|
||||
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$i" "SKIPPED_GRID_LIMIT" "GRID_LIMIT reached" "$tp" "$dp"
|
||||
break
|
||||
fi
|
||||
|
||||
sname="c${conc}_i${isl}_o${dsl}"
|
||||
output_file="${raw_dir}/sglang_main_${conc}_${isl}_${dsl}.jsonl"
|
||||
detail_log="${phase_log_dir}/sglang_${config_label}_${sname}.log"
|
||||
gpu_csv="${gpu_log_dir}/gpu_mem_${conc}_${isl}_${dsl}.csv"
|
||||
|
||||
if scenario_already_completed "$output_file" "$num" || scenario_already_processed "$result_root" "$sname"; then
|
||||
log "skipping already-processed ${config_label} scenario: ${sname}"
|
||||
continue
|
||||
fi
|
||||
|
||||
log "running ${config_label} scenario: mark=${mark} c=${conc} i=${isl} o=${dsl} n=${num}"
|
||||
|
||||
local gpu_pid
|
||||
gpu_pid="$(start_gpu_monitor "$gpu_csv")"
|
||||
|
||||
bench_rc=0
|
||||
timeout "$SCENARIO_TIMEOUT_S" bash -c '
|
||||
run_bench_serving \
|
||||
--backend sglang \
|
||||
--host 127.0.0.1 \
|
||||
--port "'"$SGLANG_PORT"'" \
|
||||
--dataset-name random \
|
||||
--dataset-path "'"$DATASET_PATH"'" \
|
||||
--random-input-len "'"$isl"'" \
|
||||
--random-output-len "'"$dsl"'" \
|
||||
--num-prompts "'"$num"'" \
|
||||
--max-concurrency "'"$conc"'" \
|
||||
--request-rate 10000 \
|
||||
--output-file "'"$output_file"'" \
|
||||
--output-details \
|
||||
> "'"$detail_log"'" 2>&1
|
||||
' || bench_rc=$?
|
||||
|
||||
stop_gpu_monitor "$gpu_pid"
|
||||
|
||||
if [[ "$bench_rc" -eq 0 ]]; then
|
||||
log "finished ${config_label} scenario: output=${output_file}"
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"completed\"" \
|
||||
"note=\"benchmark finished successfully\""
|
||||
continue
|
||||
fi
|
||||
|
||||
# Failure handling.
|
||||
if detect_oom "$detail_log" "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log"; then
|
||||
log "ERROR: ${config_label} scenario ${sname} triggered OOM; stopping config"
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"OOM\"" \
|
||||
"note=\"detected CUDA out-of-memory\""
|
||||
record_skipped_csv "$skipped_csv" \
|
||||
"engine=sglang" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=OOM" "reason=detected CUDA out-of-memory" "detail_log=${detail_log}"
|
||||
stop_server "$tp" "$dp"
|
||||
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_AFTER_OOM" "previous case OOM" "$tp" "$dp"
|
||||
break
|
||||
fi
|
||||
|
||||
log "ERROR: ${config_label} scenario ${sname} failed (rc=${bench_rc}); see ${detail_log}"
|
||||
if [[ "$mark" == "P" ]]; then
|
||||
log "optional (P) scenario failed; recording as skipped and continuing"
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"skipped_optional\"" \
|
||||
"note=\"optional scenario failed (rc=${bench_rc})\""
|
||||
record_skipped_csv "$skipped_csv" \
|
||||
"engine=sglang" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=skipped_optional" "reason=optional scenario failed (rc=${bench_rc})" "detail_log=${detail_log}"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Mandatory scenario failed but not OOM: try to restart the server.
|
||||
if restart_server "$tp" "$dp"; then
|
||||
run_warmup 1024 128 || true
|
||||
log "resuming ${config_label} after server restart"
|
||||
continue
|
||||
fi
|
||||
|
||||
log "ERROR: ${config_label} server restart failed; skipping remaining scenarios"
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"FAILED\"" \
|
||||
"note=\"scenario failed and server restart failed (rc=${bench_rc})\""
|
||||
record_skipped_csv "$skipped_csv" \
|
||||
"engine=sglang" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=FAILED" "reason=scenario failed and server restart failed" "detail_log=${detail_log}"
|
||||
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_RESTART_FAILED" "server restart failed" "$tp" "$dp"
|
||||
break
|
||||
done
|
||||
|
||||
stop_server "$tp" "$dp"
|
||||
|
||||
# Parse results.
|
||||
log "parsing ${config_label} results"
|
||||
"$PYTHON" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend sglang \
|
||||
>> "${phase_log_dir}/parse.log" 2>&1 || {
|
||||
log "WARNING: parser failed for ${config_label}; see ${phase_log_dir}/parse.log"
|
||||
}
|
||||
|
||||
log "===== ${config_label} DONE ====="
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Cleanup any leftovers.
|
||||
for cfg in "${PARALLEL_CONFIGS[@]}"; do
|
||||
read -r tp dp <<< "$cfg"
|
||||
stop_server "$tp" "$dp"
|
||||
done
|
||||
|
||||
# Run each parallel configuration.
|
||||
for cfg in "${PARALLEL_CONFIGS[@]}"; do
|
||||
read -r tp dp <<< "$cfg"
|
||||
run_parallel_config "$tp" "$dp"
|
||||
done
|
||||
|
||||
# Generate cross-configuration comparison.
|
||||
log "generating comparison report"
|
||||
"$PYTHON" "${SCRIPT_DIR}/compare.py" \
|
||||
--run-root "${RESULT_BASE}/${RUN_ID}" \
|
||||
--output "${RESULT_BASE}/${RUN_ID}/comparison.md" \
|
||||
>> "${log_dir_global}/compare.log" 2>&1 || {
|
||||
log "WARNING: comparison script failed; see ${log_dir_global}/compare.log"
|
||||
}
|
||||
|
||||
log "all results saved to ${RESULT_BASE}/${RUN_ID}"
|
||||
93
experiments/dsv4_h200_sglang_tp_dp_matrix/run_sglang_in_container.sh
Executable file
93
experiments/dsv4_h200_sglang_tp_dp_matrix/run_sglang_in_container.sh
Executable file
@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run SGLang server inside the already-running benchmark container.
|
||||
# Usage: run_sglang_in_container.sh <TP> <DP>
|
||||
set -e
|
||||
|
||||
TP="${1}"
|
||||
DP="${2}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
mkdir -p "${RUNTIME_BASE}/logs"
|
||||
|
||||
PORT="${SGLANG_PORT:-30031}"
|
||||
NAME="${EXPERIMENT}_sglang_container"
|
||||
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_sglang_tp${TP}_dp${DP}.pid"
|
||||
|
||||
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_sglang_in_container_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
# Build server args.
|
||||
SERVER_ARGS=(
|
||||
serve
|
||||
--model-path "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--tp-size "$TP"
|
||||
--moe-runner-backend "$MOE_RUNNER_BACKEND"
|
||||
--context-length "$CONTEXT_LENGTH"
|
||||
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||
--mem-fraction-static "$MEM_FRACTION_STATIC"
|
||||
--host 0.0.0.0
|
||||
--port "$PORT"
|
||||
)
|
||||
|
||||
if [[ "$DP" -gt 1 ]]; then
|
||||
SERVER_ARGS+=(
|
||||
--dp-size "$DP"
|
||||
)
|
||||
fi
|
||||
|
||||
SERVER_ARGS_STR="${SERVER_ARGS[*]}"
|
||||
|
||||
echo "=== Starting SGLang server inside container (TP=${TP}, DP=${DP}) ==="
|
||||
echo "Container: $NAME"
|
||||
echo "Command: sglang ${SERVER_ARGS_STR}"
|
||||
echo "Log: $LOG"
|
||||
|
||||
# Kill any existing sglang process inside container first.
|
||||
docker exec "$NAME" pkill -9 -f "sglang serve" 2>/dev/null || true
|
||||
sleep 3
|
||||
|
||||
# Start sglang serve inside the container in background.
|
||||
# We use nohup so it survives after docker exec returns.
|
||||
docker exec -d \
|
||||
-e CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
"$NAME" \
|
||||
bash -c "nohup sglang ${SERVER_ARGS_STR} > /tmp/sglang_server.log 2>&1 &"
|
||||
|
||||
# Wait for the server process to appear inside container.
|
||||
sleep 2
|
||||
SERVER_PID=$(docker exec "$NAME" pgrep -f "sglang serve" | head -n 1 || true)
|
||||
if [[ -z "$SERVER_PID" ]]; then
|
||||
echo "ERROR: sglang server process not found inside container"
|
||||
docker exec "$NAME" cat /tmp/sglang_server.log 2>/dev/null | tail -50 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Server PID inside container: $SERVER_PID"
|
||||
echo "$SERVER_PID" > "$PID_FILE"
|
||||
|
||||
echo "Waiting for health on port ${PORT}..."
|
||||
for i in $(seq 1 360); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
|
||||
echo "SGLang server is ready at http://127.0.0.1:${PORT}"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
# Check if server process is still alive.
|
||||
if ! docker exec "$NAME" kill -0 "$SERVER_PID" 2>/dev/null; then
|
||||
echo "ERROR: SGLang server exited early"
|
||||
docker exec "$NAME" cat /tmp/sglang_server.log 2>/dev/null | tail -200 || true
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/360)"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: SGLang server not healthy after 360 retries (30 mins)"
|
||||
docker exec "$NAME" cat /tmp/sglang_server.log 2>/dev/null | tail -200 || true
|
||||
exit 1
|
||||
18
experiments/dsv4_h200_sglang_tp_dp_matrix/smoke_matrix.json
Normal file
18
experiments/dsv4_h200_sglang_tp_dp_matrix/smoke_matrix.json
Normal file
@ -0,0 +1,18 @@
|
||||
{
|
||||
"comment": "Small smoke matrix for quickly verifying TP×DP server startup and bench path.",
|
||||
"mode": "Y",
|
||||
"matrix": {
|
||||
"1024": {
|
||||
"128": "Y",
|
||||
"256": "Y"
|
||||
},
|
||||
"4096": {
|
||||
"128": "Y",
|
||||
"256": "Y"
|
||||
}
|
||||
},
|
||||
"concurrency": {
|
||||
"1024": { "low": 1, "high": 8 },
|
||||
"4096": { "low": 1, "high": 4 }
|
||||
}
|
||||
}
|
||||
70
experiments/dsv4_h200_sglang_tp_dp_matrix/start_sglang_container.sh
Executable file
70
experiments/dsv4_h200_sglang_tp_dp_matrix/start_sglang_container.sh
Executable file
@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start a long-running Docker container for SGLang benchmarking.
|
||||
# The container stays alive (sleep infinity) so we can docker exec into it
|
||||
# to run different TP×DP configurations without losing DeepGEMM JIT cache.
|
||||
#
|
||||
# Usage: start_sglang_container.sh
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||
|
||||
IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:latest}"
|
||||
PORT="${SGLANG_PORT:-30031}"
|
||||
NAME="${EXPERIMENT}_sglang_container"
|
||||
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_container.pid"
|
||||
|
||||
# Persistent cache directory on host for DeepGEMM JIT kernels.
|
||||
CACHE_DIR="${RUNTIME_BASE}/cache"
|
||||
mkdir -p "${CACHE_DIR}/deep_gemm" "${CACHE_DIR}/tvm-ffi"
|
||||
|
||||
# Clean up any stale container.
|
||||
docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||
|
||||
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_container_$(date +%Y%m%d_%H%M%S).log"
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
echo "=== Starting SGLang benchmark container ==="
|
||||
echo "Image: $IMAGE"
|
||||
echo "Container name: $NAME"
|
||||
echo "Host port: $PORT"
|
||||
echo "Cache dir: $CACHE_DIR"
|
||||
echo "Log: $LOG"
|
||||
|
||||
# Start a long-running container.
|
||||
# We override the entrypoint to sleep infinity so the container stays alive.
|
||||
docker run -d \
|
||||
--name "$NAME" \
|
||||
--gpus all \
|
||||
--ipc host \
|
||||
--shm-size 16g \
|
||||
--entrypoint /bin/bash \
|
||||
-p "${PORT}:${PORT}" \
|
||||
-v "${MODEL_PATH}:${MODEL_PATH}:ro" \
|
||||
-v "${CACHE_DIR}/deep_gemm:/root/.cache/deep_gemm" \
|
||||
-v "${CACHE_DIR}/tvm-ffi:/root/.cache/tvm-ffi" \
|
||||
-v "${RUNTIME_BASE}/tmp:/tmp" \
|
||||
-e CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
"$IMAGE" \
|
||||
-c "sleep infinity" \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
# Wait a moment for container to be ready.
|
||||
sleep 2
|
||||
|
||||
if ! docker ps --filter "name=$NAME" --format '{{.Names}}' | grep -q "$NAME"; then
|
||||
echo "ERROR: Container failed to start"
|
||||
tail -50 "$LOG"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Record the container ID for later use.
|
||||
docker inspect -f '{{.Id}}' "$NAME" > "$PID_FILE"
|
||||
|
||||
echo "Container $NAME is running"
|
||||
echo "Log: $LOG"
|
||||
99
experiments/dsv4_h200_sglang_tp_dp_matrix/start_sglang_docker.sh
Executable file
99
experiments/dsv4_h200_sglang_tp_dp_matrix/start_sglang_docker.sh
Executable file
@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start SGLang server in Docker for a given TPxDP configuration.
|
||||
# Usage: start_sglang_docker.sh <TP> <DP>
|
||||
#
|
||||
# Uses the lmsysorg/sglang image and the same argument set as the
|
||||
# bare-metal start script. The container is removed automatically on stop.
|
||||
set -e
|
||||
|
||||
TP="${1}"
|
||||
DP="${2}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||
|
||||
IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:latest}"
|
||||
PORT="${SGLANG_PORT:-30031}"
|
||||
NAME="${EXPERIMENT}_sglang_tp${TP}_dp${DP}"
|
||||
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_sglang_tp${TP}_dp${DP}.pid"
|
||||
|
||||
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_sglang_docker_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
# Clean up any stale container with the same name.
|
||||
docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||
|
||||
SERVER_ARGS=(
|
||||
serve
|
||||
--model-path "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--tp-size "$TP"
|
||||
--moe-runner-backend "$MOE_RUNNER_BACKEND"
|
||||
--context-length "$CONTEXT_LENGTH"
|
||||
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||
--mem-fraction-static "$MEM_FRACTION_STATIC"
|
||||
--host 0.0.0.0
|
||||
--port "$PORT"
|
||||
)
|
||||
|
||||
if [[ "$DP" -gt 1 ]]; then
|
||||
SERVER_ARGS+=(
|
||||
--dp-size "$DP"
|
||||
)
|
||||
fi
|
||||
|
||||
SERVER_ARGS_STR="${SERVER_ARGS[*]}"
|
||||
|
||||
echo "=== Starting SGLang server in Docker (TP=${TP}, DP=${DP}) ==="
|
||||
echo "Image: $IMAGE"
|
||||
echo "Model: $MODEL_PATH"
|
||||
echo "Container name: $NAME"
|
||||
echo "Host port: $PORT"
|
||||
echo "Command: sglang ${SERVER_ARGS_STR}"
|
||||
echo "Log: $LOG"
|
||||
|
||||
# Run docker in the foreground so that killing the host process stops the
|
||||
# container (the --rm flag ensures cleanup). nohup lets us background it and
|
||||
# capture the host PID in the same way as the bare-metal start script.
|
||||
nohup docker run --rm \
|
||||
--name "$NAME" \
|
||||
--gpus all \
|
||||
--ipc host \
|
||||
--shm-size 16g \
|
||||
--entrypoint sglang \
|
||||
-p "${PORT}:${PORT}" \
|
||||
-v "${MODEL_PATH}:${MODEL_PATH}:ro" \
|
||||
-v "${RUNTIME_BASE}/tmp:/tmp" \
|
||||
-e CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
"$IMAGE" \
|
||||
"${SERVER_ARGS[@]}" \
|
||||
> "$LOG" 2>&1 &
|
||||
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
echo "Waiting for health on port ${PORT}..."
|
||||
|
||||
for i in $(seq 1 240); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
|
||||
echo "SGLang server is ready at http://127.0.0.1:${PORT}"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
if ! kill -0 $PID 2>/dev/null; then
|
||||
echo "ERROR: Docker SGLang server exited early"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/240)"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: Docker SGLang server not healthy after 240 retries"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
86
experiments/dsv4_h200_sglang_tp_dp_matrix/start_sglang_dp.sh
Executable file
86
experiments/dsv4_h200_sglang_tp_dp_matrix/start_sglang_dp.sh
Executable file
@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start SGLang server for a given TP×DP configuration.
|
||||
# Usage: start_sglang_dp.sh <TP> <DP>
|
||||
#
|
||||
# By default this delegates to the Docker start script because the experiment
|
||||
# is intended to run SGLang inside a container. Set USE_DOCKER=0 to use the
|
||||
# local VENV_CLIENT environment instead.
|
||||
set -e
|
||||
|
||||
TP="${1}"
|
||||
DP="${2}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
if [[ "${USE_DOCKER:-1}" == "1" ]]; then
|
||||
exec "${SCRIPT_DIR}/start_sglang_docker.sh" "$@"
|
||||
fi
|
||||
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||
|
||||
VENV="${VENV_CLIENT}"
|
||||
export PATH="$VENV/bin:$PATH"
|
||||
export PYTHONUNBUFFERED=1
|
||||
export TMPDIR="${RUNTIME_BASE}/tmp"
|
||||
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}"
|
||||
|
||||
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_sglang_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
|
||||
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_sglang_tp${TP}_dp${DP}.pid"
|
||||
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
SERVER_ARGS=(
|
||||
sglang serve
|
||||
--model-path "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--tp-size "$TP"
|
||||
--moe-runner-backend "$MOE_RUNNER_BACKEND"
|
||||
--context-length "$CONTEXT_LENGTH"
|
||||
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||
--mem-fraction-static "$MEM_FRACTION_STATIC"
|
||||
--host 0.0.0.0
|
||||
--port "$SGLANG_PORT"
|
||||
)
|
||||
|
||||
if [[ "$DP" -gt 1 ]]; then
|
||||
SERVER_ARGS+=(
|
||||
--dp-size "$DP"
|
||||
)
|
||||
fi
|
||||
|
||||
SERVER_ARGS_STR="${SERVER_ARGS[*]}"
|
||||
|
||||
echo "=== Starting SGLang server (TP=${TP}, DP=${DP}) ==="
|
||||
echo "Model: $MODEL_PATH"
|
||||
echo "Port: $SGLANG_PORT"
|
||||
echo "Command: $SERVER_ARGS_STR"
|
||||
echo "Log: $LOG"
|
||||
|
||||
nohup "${SERVER_ARGS[@]}" > "$LOG" 2>&1 &
|
||||
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
echo "Waiting for health on port ${SGLANG_PORT}..."
|
||||
|
||||
for i in $(seq 1 240); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${SGLANG_PORT}/health" >/dev/null 2>&1; then
|
||||
echo "SGLang server is ready at http://127.0.0.1:${SGLANG_PORT}"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
if ! kill -0 $PID 2>/dev/null; then
|
||||
echo "ERROR: SGLang server exited early"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/240)"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: SGLang server not healthy after 240 retries"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
@ -20,13 +20,14 @@ TP=8
|
||||
|
||||
# Phase 1: short-context throughput (max_model_len 32k).
|
||||
# Each element: "concurrency input_len output_len num_prompts"
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
if [[ -z "${PHASE1_SCENARIOS:-}" ]]; then
|
||||
declare -a PHASE1_SCENARIOS=(
|
||||
"1 512 256 32"
|
||||
"32 512 256 128"
|
||||
"128 512 256 128"
|
||||
"1 4000 512 32"
|
||||
"32 4000 512 64"
|
||||
"1 512 256 5"
|
||||
"32 512 256 160"
|
||||
"128 512 256 640"
|
||||
"1 4000 512 5"
|
||||
"32 4000 512 160"
|
||||
)
|
||||
fi
|
||||
PHASE1_MAX_MODEL_LEN=32768
|
||||
@ -34,6 +35,7 @@ PHASE1_MAX_NUM_SEQS=256
|
||||
PHASE1_MAX_RUNNING=256
|
||||
|
||||
# Phase 2: long context (max_model_len 210k to leave room for output + padding).
|
||||
# Exception to the 5x rule due to very long input lengths; kept at low counts.
|
||||
if [[ -z "${PHASE2_SCENARIOS:-}" ]]; then
|
||||
declare -a PHASE2_SCENARIOS=(
|
||||
"1 32768 1024 8"
|
||||
@ -51,8 +53,8 @@ SHAREGPT_DATASET="${SHAREGPT_DATASET:-/data/user1/yy/datasets/ShareGPT_filtered_
|
||||
SHAREGPT_CONTEXT_LEN=32768
|
||||
if [[ -z "${SHAREGPT_SCENARIOS:-}" ]]; then
|
||||
declare -a SHAREGPT_SCENARIOS=(
|
||||
"1 32768 256 32"
|
||||
"32 32768 256 128"
|
||||
"1 32768 256 5"
|
||||
"32 32768 256 160"
|
||||
)
|
||||
fi
|
||||
|
||||
|
||||
@ -15,17 +15,19 @@ ENGINE="vllm"
|
||||
VENV_SERVER="${VENV_SERVER:-/data/user1/yy/envs/vllm}"
|
||||
VENV_CLIENT="${VENV_CLIENT:-/data/user1/yy/envs/sglang}"
|
||||
|
||||
# Benchmark scenarios: each element is "concurrency input_len output_len".
|
||||
# Benchmark scenarios: each element is "concurrency input_len output_len num_prompts".
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
# Override via SCENARIOS env var as a bash array, e.g.:
|
||||
# SCENARIOS=("32 512 256" "128 512 256")
|
||||
# SCENARIOS=("32 512 256 160" "128 512 256 640")
|
||||
if [[ -z "${SCENARIOS:-}" ]]; then
|
||||
SCENARIOS=(
|
||||
"32 512 256"
|
||||
"128 512 256"
|
||||
"32 4000 512"
|
||||
"32 512 256 160"
|
||||
"128 512 256 640"
|
||||
"32 4000 512 160"
|
||||
)
|
||||
fi
|
||||
NUM_PROMPTS="${NUM_PROMPTS:-128}"
|
||||
# Legacy fallback: used only if SCENARIOS entries have 3 fields.
|
||||
NUM_PROMPTS="${NUM_PROMPTS:-160}"
|
||||
|
||||
# Server start script bundled with this experiment.
|
||||
SERVER_START_SCRIPT="${SCRIPT_DIR}/start_server.sh"
|
||||
|
||||
@ -62,8 +62,7 @@ data["config"] = {
|
||||
"block_size": 256,
|
||||
"max_num_seqs": 256,
|
||||
"port": 30005,
|
||||
"num_prompts": 128,
|
||||
"scenarios": ["32 512 256", "128 512 256", "32 4000 512"]
|
||||
"scenarios": ["32 512 256 160", "128 512 256 640", "32 4000 512 160"]
|
||||
}
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
@ -128,11 +127,15 @@ fi
|
||||
log "===== BENCHMARK START ====="
|
||||
|
||||
for scenario in "${SCENARIOS[@]}"; do
|
||||
read -r concurrency input_len output_len <<< "$scenario"
|
||||
read -r concurrency input_len output_len num_prompts <<< "$scenario"
|
||||
# Fallback for legacy 3-field scenarios.
|
||||
if [[ -z "${num_prompts:-}" ]]; then
|
||||
num_prompts="$NUM_PROMPTS"
|
||||
fi
|
||||
output_file="${RAW_DIR}/vllm_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||
detail_log="${LOG_DIR}/vllm_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||
|
||||
log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len}"
|
||||
log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len} num_prompts=${num_prompts}"
|
||||
|
||||
"${VENV_CLIENT}/bin/python" -m sglang.bench_serving \
|
||||
--backend vllm \
|
||||
@ -141,7 +144,7 @@ for scenario in "${SCENARIOS[@]}"; do
|
||||
--dataset-name random \
|
||||
--random-input-len "$input_len" \
|
||||
--random-output-len "$output_len" \
|
||||
--num-prompts "$NUM_PROMPTS" \
|
||||
--num-prompts "$num_prompts" \
|
||||
--max-concurrency "$concurrency" \
|
||||
--request-rate 10000 \
|
||||
--output-file "$output_file" \
|
||||
|
||||
@ -30,13 +30,14 @@ TP=8
|
||||
|
||||
# Scenarios focused on TTFT: low/high concurrency x short/medium input.
|
||||
# Each element: "concurrency input_len output_len num_prompts"
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
if [[ -z "${SCENARIOS:-}" ]]; then
|
||||
declare -a SCENARIOS=(
|
||||
"1 512 256 32"
|
||||
"32 512 256 128"
|
||||
"128 512 256 128"
|
||||
"1 4000 512 32"
|
||||
"32 4000 512 64"
|
||||
"1 512 256 5"
|
||||
"32 512 256 160"
|
||||
"128 512 256 640"
|
||||
"1 4000 512 5"
|
||||
"32 4000 512 160"
|
||||
)
|
||||
fi
|
||||
MAX_MODEL_LEN=32768
|
||||
|
||||
@ -0,0 +1,85 @@
|
||||
# DSpark TTFT 长于 vLLM baseline 的根因分析
|
||||
|
||||
## 数据回顾
|
||||
|
||||
| Scenario | Backend | Conc | Input | Output | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | Req/s |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c1_i512_o256 | vllm-dspark | 1 | 512 | 256 | 86.51 | 185.00 | 188.81 | 2.09 | 2.56 |
|
||||
| c1_i512_o256 | vllm-default | 1 | 512 | 256 | 49.50 | 68.50 | 71.09 | 6.66 | 0.99 |
|
||||
| c32_i512_o256 | vllm-dspark | 32 | 512 | 256 | 262.56 | 527.21 | 532.11 | 12.06 | 16.53 |
|
||||
| c32_i512_o256 | vllm-default | 32 | 512 | 256 | 122.62 | 272.00 | 304.80 | 15.77 | 12.98 |
|
||||
| c128_i512_o256 | vllm-dspark | 128 | 512 | 256 | 778.36 | 1071.85 | 1075.35 | 28.95 | 23.94 |
|
||||
| c128_i512_o256 | vllm-default | 128 | 512 | 256 | 617.72 | 767.18 | 770.89 | 18.31 | 29.99 |
|
||||
| c1_i4000_o512 | vllm-dspark | 1 | 4000 | 512 | 123.58 | 176.29 | 190.52 | 1.66 | 1.73 |
|
||||
| c1_i4000_o512 | vllm-default | 1 | 4000 | 512 | 131.50 | 191.53 | 268.69 | 6.68 | 0.50 |
|
||||
| c32_i4000_o512 | vllm-dspark | 32 | 4000 | 512 | 298.42 | 434.24 | 522.34 | 9.68 | 11.49 |
|
||||
| c32_i4000_o512 | vllm-default | 32 | 4000 | 512 | 382.74 | 681.98 | 684.75 | 16.43 | 5.38 |
|
||||
|
||||
## 核心结论
|
||||
|
||||
**DSpark 在短输入（i=512）场景下 TTFT 明显长于 vLLM baseline，是投机解码机制本身的正常 trade-off，但在当前配置下 overhead 偏大，有优化空间。**
|
||||
|
||||
## 为什么短输入场景 TTFT 会变差？
|
||||
|
||||
### 1. 投机解码的本质 overhead
|
||||
|
||||
DSpark 在生成第一个 token 时，除了执行一次目标模型（target model）的 prefill，还要：
|
||||
- 加载并执行 draft model；
|
||||
- 让 draft model 连续预测最多 `spec-tokens=5` 个 token；
|
||||
- 用 target model 对这 5 个 token 做并行验证（verification）；
|
||||
- 根据验证结果回退或接受。
|
||||
|
||||
对于 **短输入（512 tokens）**，baseline 的 prefill 本身非常快（P50 仅 44-65ms），draft model 的额外 forward + verification 的固定 overhead 占了 TTFT 的很大比例，导致 TTFT 反而变长。
|
||||
|
||||
### 2. 长输入场景反而受益
|
||||
|
||||
对于 **长输入（4000 tokens）**，prefill 是主要瓶颈。DSpark 通过 draft model 投机生成后续 token，能显著减少 target model 的 forward 次数，因此：
|
||||
- c1_i4000_o512：TTFT 与 baseline 接近（123ms vs 131ms），但 TPOT 从 6.68ms 降到 1.66ms；
|
||||
- c32_i4000_o512：TTFT 明显优于 baseline（298ms vs 383ms，-22%），TPOT 从 16.43ms 降到 9.68ms。
|
||||
|
||||
这符合投机解码的设计目标：**用 draft model 的额外计算换取 target model 更少的完整 forward，从而在长序列或高输出长度上获得端到端收益。**
|
||||
|
||||
### 3. 高并发短输入场景被放大
|
||||
|
||||
在 c128_i512_o256 下：
|
||||
- DSpark 的 Req/s 低于 baseline（23.94 vs 29.99），说明单位时间内处理的请求更少；
|
||||
- 同时 DSpark 的 mean TPOT 高于 baseline（28.95ms vs 18.31ms）。
|
||||
|
||||
这说明在并发较高、输入较短时，DSpark 的 draft-verification 流水线没有跑赢 baseline 的简单 decode，导致队列等待时间增加，进一步放大了 TTFT。
|
||||
|
||||
## 是代码缺陷还是正常现象？
|
||||
|
||||
### 属于正常现象的部分
|
||||
|
||||
1. **短输入下投机解码增加 TTFT 是预期行为**：只要 draft model 不是零成本，第一个 token 必然比 baseline 慢。
|
||||
2. **DSpark 的收益主要体现在 TPOT/E2E/吞吐量**：所有场景下 DSpark 的 TPOT 都显著低于 baseline，长输入场景 E2E 也大幅缩短。
|
||||
|
||||
### 可能存在优化空间的部分
|
||||
|
||||
1. **`spec-tokens=5` 对短输入可能过大**
|
||||
- 当前 DSpark 使用 `--spec-tokens 5`。
|
||||
- 对于 512 token 的短输入，prefill 后只需要 256 个输出 token，一次 spec 5 个 token 的 verification overhead 相对较高。
|
||||
- **建议**：对短输入场景尝试 `spec-tokens=1` 或 `spec-tokens=2`，看是否能降低 TTFT overhead 同时保持 TPOT 收益。
|
||||
|
||||
2. **DSpark 与 default 使用不同 checkpoint**
|
||||
- DSpark 模型路径是 `/data/models/DeepSeek-V4-Flash-DSpark`，而 baseline 是 `/data/models/DeepSeek-V4-Flash`。
|
||||
- 如果 DSpark checkpoint 的 prefill 路径没有被充分优化（例如 MLA/KV cache 格式差异），可能会额外增加 prefill 时间。
|
||||
- **建议**：确认两个 checkpoint 在 prefill 阶段是否使用相同的 kernel/量化路径，排除模型差异本身带来的 TTFT 偏移。
|
||||
|
||||
3. **draft model 与 target model 的调度/重叠**
|
||||
- 如果 draft model 的 forward 和 target model 的 verification 没有充分 overlap，TTFT 会进一步恶化。
|
||||
- **建议**：通过 vLLM 的 profile/profiling 工具查看 prefill 阶段 draft model 是否串行执行，以及是否可以通过 `--spec-max-model-len` 或调度参数优化。
|
||||
|
||||
4. **高并发下的 batching 策略**
|
||||
- c128_i512_o256 下 DSpark 吞吐量低于 baseline，可能是 speculative decoding 在高并发下的 batching 效率下降。
|
||||
- **建议**：测试不同 `max-num-seqs` 或 `--max-num-batched-tokens` 配置，观察并发吞吐和 TTFT 的变化。
|
||||
|
||||
## 建议的下一步
|
||||
|
||||
1. **参数调优实验**：固定模型和硬件，扫描 `spec-tokens ∈ {1, 2, 3, 5}`，重点观察短输入场景的 TTFT 和 TPOT 变化。
|
||||
2. **profiling**：在 c1_i512_o256 和 c32_i512_o256 下，用 vLLM 的 `--profile` 或 Nsight 抓取 prefill 时间线，确认 draft model 的 overhead 占比。
|
||||
3. **与 MTP 对比**：当前正在跑的 vLLM+MTP 实验会提供另一组投机解码数据，可以对比 DSpark 和官方 MTP 在 TTFT/TPOT 上的 trade-off 差异。
|
||||
|
||||
## 总结
|
||||
|
||||
DSpark 在短输入场景 TTFT 变差**不是代码 bug，而是投机解码机制的正常开销**。但当前 `spec-tokens=5` 的配置在短输入下 overhead 偏大，建议通过调参和 profiling 进一步优化，尤其是在 S1 实时交互层（输入短、延迟敏感）场景。
|
||||
74
experiments/dsv4_h200_vllm_mtp_vs_default/compare.py
Executable file
74
experiments/dsv4_h200_vllm_mtp_vs_default/compare.py
Executable file
@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate a side-by-side comparison of vLLM+MTP speculative decoding and vLLM default.
|
||||
|
||||
Usage:
|
||||
python3 compare.py --mtp <mtp_result_root> --default <default_result_root> \
|
||||
[--output comparison.md]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_result(result_root: Path) -> dict:
|
||||
path = result_root / "results.json"
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--mtp", type=Path, required=True)
|
||||
parser.add_argument("--default", type=Path, required=True)
|
||||
parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md"))
|
||||
args = parser.parse_args()
|
||||
|
||||
mtp_data = load_result(args.mtp)
|
||||
default_data = load_result(args.default)
|
||||
|
||||
by_scenario = defaultdict(dict)
|
||||
for data in (mtp_data, default_data):
|
||||
backend = data["metadata"]["engine"]
|
||||
for s in data.get("scenarios", []):
|
||||
key = s["name"]
|
||||
by_scenario[key][backend] = s
|
||||
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
f.write("# vLLM+MTP speculative decoding vs vLLM default on DeepSeek-V4-Flash (H200, TP=8)\n\n")
|
||||
f.write("## Summary\n\n")
|
||||
f.write("- Model: `/data/models/DeepSeek-V4-Flash`\n")
|
||||
f.write("- Hardware: 8x NVIDIA H200 143GB\n")
|
||||
f.write("- Tensor Parallelism: 8\n")
|
||||
f.write("- Benchmark client: `sglang.bench_serving --backend vllm`\n")
|
||||
f.write("- Default: no speculative decoding\n")
|
||||
f.write("- MTP: `--speculative-config '{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":1}'` (official DeepSeek-V4 recipe)\n\n")
|
||||
|
||||
f.write("## Side-by-side results\n\n")
|
||||
f.write("| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) |\n")
|
||||
f.write("|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n")
|
||||
|
||||
for scenario_name in sorted(by_scenario.keys()):
|
||||
for backend in ("vllm-mtp", "vllm-default"):
|
||||
s = by_scenario[scenario_name].get(backend)
|
||||
if s is None:
|
||||
continue
|
||||
cfg = s["config"]
|
||||
m = s["metrics"]
|
||||
f.write(
|
||||
f"| {scenario_name} | {backend} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | "
|
||||
f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||
f"{m['ttft_ms']['mean']:.2f} | {m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||
f"{m['e2e_ms']['mean']:.2f} |\n"
|
||||
)
|
||||
|
||||
f.write("\n## Notes\n\n")
|
||||
f.write("- TTFT mean/P95/P99 are the main focus for verifying MTP's impact on time-to-first-token.\n")
|
||||
f.write("- Mean TPOT and E2E are included to check whether speculative decoding pays back after first token.\n")
|
||||
|
||||
print(f"Wrote comparison to {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
35
experiments/dsv4_h200_vllm_mtp_vs_default/config.env
Normal file
35
experiments/dsv4_h200_vllm_mtp_vs_default/config.env
Normal file
@ -0,0 +1,35 @@
|
||||
# Configuration for vLLM MTP speculative decoding vs vLLM default comparison on H200.
|
||||
# The default-side results are reused from dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default.
|
||||
|
||||
EXPERIMENT="dsv4_h200_vllm_mtp_vs_default"
|
||||
MODEL_NAME="DeepSeek-V4-Flash"
|
||||
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
|
||||
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
MTP_PORT="${MTP_PORT:-30009}"
|
||||
DEFAULT_RESULT_ROOT="${DEFAULT_RESULT_ROOT:-/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default}"
|
||||
|
||||
VENV_VLLM="${VENV_VLLM:-/data/user1/yy/envs/vllm}"
|
||||
VENV_CLIENT="${VENV_CLIENT:-/data/user1/yy/envs/sglang}"
|
||||
|
||||
export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||
TP=8
|
||||
|
||||
MAX_MODEL_LEN=32768
|
||||
MAX_NUM_SEQS=256
|
||||
SPEC_METHOD="${SPEC_METHOD:-mtp}"
|
||||
SPEC_TOKENS="${SPEC_TOKENS:-1}"
|
||||
|
||||
# Scenarios must match the ones in the reused default result set.
|
||||
# Following docs/EXPERIMENT_GUIDE.md: num_prompts = concurrency * 5.
|
||||
# NOTE: if these scenarios are changed, the default-side results must be re-run
|
||||
# with the same scenarios (update DEFAULT_RESULT_ROOT accordingly).
|
||||
declare -a SCENARIOS=(
|
||||
"1 512 256 5"
|
||||
"32 512 256 160"
|
||||
"128 512 256 640"
|
||||
"1 4000 512 5"
|
||||
"32 4000 512 160"
|
||||
)
|
||||
|
||||
MTP_START_SCRIPT="${SCRIPT_DIR:-.}/start_mtp.sh"
|
||||
@ -0,0 +1,30 @@
|
||||
# vLLM+MTP speculative decoding vs vLLM default on DeepSeek-V4-Flash (H200, TP=8)
|
||||
|
||||
## Summary
|
||||
|
||||
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||
- Hardware: 8x NVIDIA H200 143GB
|
||||
- Tensor Parallelism: 8
|
||||
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||
- Default: no speculative decoding
|
||||
- MTP: `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'` (official DeepSeek-V4 recipe)
|
||||
|
||||
## Side-by-side results
|
||||
|
||||
| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c128_i512_o256 | vllm-mtp | 128 | 512 | 256 | 35.03 | 4625.47 | 771.59 | 1110.39 | 1114.20 | 18.16 | 111.99 | 2506.29 |
|
||||
| c128_i512_o256 | vllm-default | 128 | 512 | 256 | 29.99 | 3959.89 | 617.72 | 767.18 | 770.89 | 18.31 | 65.44 | 2695.91 |
|
||||
| c1_i4000_o512 | vllm-mtp | 1 | 4000 | 512 | 0.83 | 234.90 | 161.27 | 216.12 | 289.08 | 3.70 | 3.89 | 1197.52 |
|
||||
| c1_i4000_o512 | vllm-default | 1 | 4000 | 512 | 0.50 | 140.13 | 131.50 | 191.53 | 268.69 | 6.68 | 6.73 | 2008.09 |
|
||||
| c1_i512_o256 | vllm-mtp | 1 | 512 | 256 | 1.64 | 239.37 | 60.63 | 70.69 | 71.97 | 3.80 | 4.33 | 607.04 |
|
||||
| c1_i512_o256 | vllm-default | 1 | 512 | 256 | 0.99 | 143.82 | 49.50 | 68.50 | 71.09 | 6.66 | 6.71 | 1010.98 |
|
||||
| c32_i4000_o512 | vllm-mtp | 32 | 4000 | 512 | 6.32 | 1785.09 | 409.17 | 667.77 | 673.18 | 14.18 | 19.84 | 4431.73 |
|
||||
| c32_i4000_o512 | vllm-default | 32 | 4000 | 512 | 5.38 | 1518.49 | 382.74 | 681.98 | 684.75 | 16.43 | 25.49 | 4927.71 |
|
||||
| c32_i512_o256 | vllm-mtp | 32 | 512 | 256 | 12.56 | 1657.98 | 279.91 | 761.67 | 766.29 | 16.73 | 37.56 | 2332.50 |
|
||||
| c32_i512_o256 | vllm-default | 32 | 512 | 256 | 12.98 | 1713.20 | 122.62 | 272.00 | 304.80 | 15.77 | 23.26 | 2154.65 |
|
||||
|
||||
## Notes
|
||||
|
||||
- TTFT mean/P95/P99 are the main focus for verifying MTP's impact on time-to-first-token.
|
||||
- Mean TPOT and E2E are included to check whether speculative decoding pays back after first token.
|
||||
@ -0,0 +1,19 @@
|
||||
# 8x NVIDIA H200 143GB VLLM Benchmark Report
|
||||
|
||||
- Result root: `/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp`
|
||||
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||
- Backend: VLLM
|
||||
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||
|
||||
## Results
|
||||
|
||||
| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c128_i512_o256 | mtp | 128 | 512 | 256 | 3.65 | 128 | 35.03 | 9647.26 | 4625.47 | 14272.73 | 771.59 | 1110.39 | 1114.20 | 18.16 | 32.29 | 111.99 | 2506.29 | 3456.20 | 3560.22 | ✅ |
|
||||
| c1_i4000_o512 | mtp | 1 | 4000 | 512 | 38.35 | 32 | 0.83 | 1694.86 | 234.90 | 1929.76 | 161.27 | 216.12 | 289.08 | 3.70 | 3.82 | 3.89 | 1197.52 | 2034.69 | 2078.68 | ✅ |
|
||||
| c1_i512_o256 | mtp | 1 | 512 | 256 | 19.46 | 32 | 1.64 | 419.94 | 239.37 | 659.31 | 60.63 | 70.69 | 71.97 | 3.80 | 4.11 | 4.33 | 607.04 | 1022.42 | 1023.52 | ✅ |
|
||||
| c32_i4000_o512 | mtp | 32 | 4000 | 512 | 10.12 | 64 | 6.32 | 13280.42 | 1785.09 | 15065.50 | 409.17 | 667.77 | 673.18 | 14.18 | 17.48 | 19.84 | 4431.73 | 8157.51 | 8520.71 | ✅ |
|
||||
| c32_i512_o256 | mtp | 32 | 512 | 256 | 10.19 | 128 | 12.56 | 3458.01 | 1657.98 | 5115.99 | 279.91 | 761.67 | 766.29 | 16.73 | 26.24 | 37.56 | 2332.50 | 4812.90 | 5073.77 | ✅ |
|
||||
|
||||
SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.
|
||||
|
||||
@ -0,0 +1,311 @@
|
||||
{
|
||||
"metadata": {
|
||||
"experiment": "dsv4_h200_vllm_mtp_vs_default_mtp",
|
||||
"run_id": "20260708-160349",
|
||||
"timestamp": "2026-07-08T16:03:51+00:00",
|
||||
"model": "/data/models/DeepSeek-V4-Flash",
|
||||
"backend": "vllm",
|
||||
"engine": "vllm-mtp",
|
||||
"hardware": "8x NVIDIA H200 143GB",
|
||||
"accelerator": "NVIDIA H200",
|
||||
"chip": "nvidia_h200",
|
||||
"script": "experiments/dsv4_h200_vllm_mtp_vs_default/run_bench.sh",
|
||||
"env": "/data/user1/yy/envs/vllm",
|
||||
"git_commit": "ca98fe0",
|
||||
"git_dirty": "dirty",
|
||||
"description": "H200 vLLM+MTP TP=8 benchmark for DeepSeek-V4-Flash"
|
||||
},
|
||||
"config": {
|
||||
"tp": 8,
|
||||
"cuda_visible_devices": "0,1,2,3,4,5,6,7",
|
||||
"max_model_len": 32768,
|
||||
"max_num_seqs": 256,
|
||||
"spec_method": "mtp",
|
||||
"spec_tokens": 1,
|
||||
"backend": "vllm",
|
||||
"server_start_script": "experiments/dsv4_h200_vllm_mtp_vs_default/start_mtp.sh",
|
||||
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 32768 --max-num-seqs 256 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --speculative-config {\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":1} --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30009"
|
||||
},
|
||||
"scenarios": [
|
||||
{
|
||||
"name": "c128_i512_o256",
|
||||
"config": {
|
||||
"phase": "mtp",
|
||||
"concurrency": 128,
|
||||
"input_len": 512,
|
||||
"output_len": 256,
|
||||
"dataset": "random",
|
||||
"num_prompts": 128
|
||||
},
|
||||
"metrics": {
|
||||
"success": 128,
|
||||
"failed": 0,
|
||||
"duration_s": 3.6536797620065045,
|
||||
"request_throughput": 35.03317431676217,
|
||||
"input_token_throughput": 9647.260377478384,
|
||||
"output_token_throughput": 4625.473796510006,
|
||||
"total_token_throughput": 14272.734173988389,
|
||||
"total_input_tokens": 35248,
|
||||
"total_output_tokens": 16900,
|
||||
"e2e_ms": {
|
||||
"mean": 2506.2933596329913,
|
||||
"p50": 2625.210165999306,
|
||||
"p90": 3417.3656311962986,
|
||||
"p95": 3456.20117305225,
|
||||
"p99": 3560.217050112697
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 771.5936086792681,
|
||||
"p50": 743.3261279948056,
|
||||
"p90": 964.7660584974801,
|
||||
"p95": 1110.3887211014808,
|
||||
"p99": 1114.1990597829863
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 18.15787222271614,
|
||||
"p50": 13.890511986549402,
|
||||
"p90": 25.0715809341214,
|
||||
"p95": 32.29210961239336,
|
||||
"p99": 111.98906169440306
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 25.75249892379593,
|
||||
"p50": 21.063708009023685,
|
||||
"p90": 26.390794408507645,
|
||||
"p95": 28.396654246898834,
|
||||
"p99": 255.2068907531792
|
||||
}
|
||||
},
|
||||
"slo_status": {
|
||||
"ttft_p95_ok": true,
|
||||
"tpot_mean_ok": true,
|
||||
"overall": "✅"
|
||||
},
|
||||
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_128_512_256.jsonl"
|
||||
},
|
||||
{
|
||||
"name": "c1_i4000_o512",
|
||||
"config": {
|
||||
"phase": "mtp",
|
||||
"concurrency": 1,
|
||||
"input_len": 4000,
|
||||
"output_len": 512,
|
||||
"dataset": "random",
|
||||
"num_prompts": 32
|
||||
},
|
||||
"metrics": {
|
||||
"success": 32,
|
||||
"failed": 0,
|
||||
"duration_s": 38.35236829900532,
|
||||
"request_throughput": 0.8343682911709505,
|
||||
"input_token_throughput": 1694.8627394591913,
|
||||
"output_token_throughput": 234.90074797372165,
|
||||
"total_token_throughput": 1929.7634874329128,
|
||||
"total_input_tokens": 65002,
|
||||
"total_output_tokens": 9009,
|
||||
"e2e_ms": {
|
||||
"mean": 1197.5197937804296,
|
||||
"p50": 1222.8141454979777,
|
||||
"p90": 1945.1575131970458,
|
||||
"p95": 2034.6854075956799,
|
||||
"p99": 2078.678046991845
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 161.27064584452455,
|
||||
"p50": 187.0977380021941,
|
||||
"p90": 207.2812115002307,
|
||||
"p95": 216.12448994783335,
|
||||
"p99": 289.0834562029342
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 3.698815812859111,
|
||||
"p50": 3.6894009238415832,
|
||||
"p90": 3.761323209835616,
|
||||
"p95": 3.8155798700943198,
|
||||
"p99": 3.8925407732209094
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 7.345683533679442,
|
||||
"p50": 7.343719502387103,
|
||||
"p90": 7.546291207836475,
|
||||
"p95": 7.646059650869574,
|
||||
"p99": 7.946166486799484
|
||||
}
|
||||
},
|
||||
"slo_status": {
|
||||
"ttft_p95_ok": true,
|
||||
"tpot_mean_ok": true,
|
||||
"overall": "✅"
|
||||
},
|
||||
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_1_4000_512.jsonl"
|
||||
},
|
||||
{
|
||||
"name": "c1_i512_o256",
|
||||
"config": {
|
||||
"phase": "mtp",
|
||||
"concurrency": 1,
|
||||
"input_len": 512,
|
||||
"output_len": 256,
|
||||
"dataset": "random",
|
||||
"num_prompts": 32
|
||||
},
|
||||
"metrics": {
|
||||
"success": 32,
|
||||
"failed": 0,
|
||||
"duration_s": 19.455183718004264,
|
||||
"request_throughput": 1.644805850401016,
|
||||
"input_token_throughput": 419.9394936805094,
|
||||
"output_token_throughput": 239.37065141617282,
|
||||
"total_token_throughput": 659.3101450966822,
|
||||
"total_input_tokens": 8170,
|
||||
"total_output_tokens": 4657,
|
||||
"e2e_ms": {
|
||||
"mean": 607.0424630620437,
|
||||
"p50": 709.1372035065433,
|
||||
"p90": 974.411873005738,
|
||||
"p95": 1022.4235178509844,
|
||||
"p99": 1023.5160801192977
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 60.633899186541385,
|
||||
"p50": 67.31531800323864,
|
||||
"p90": 70.1838248045533,
|
||||
"p95": 70.6854243442649,
|
||||
"p99": 71.96889203012688
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 3.7982499249353343,
|
||||
"p50": 3.7510736315107254,
|
||||
"p90": 3.9678993170763492,
|
||||
"p95": 4.107961743017731,
|
||||
"p99": 4.327418892014148
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 7.296974936146524,
|
||||
"p50": 7.262905994139146,
|
||||
"p90": 7.464885500667151,
|
||||
"p95": 7.645853242138401,
|
||||
"p99": 9.469499189435757
|
||||
}
|
||||
},
|
||||
"slo_status": {
|
||||
"ttft_p95_ok": true,
|
||||
"tpot_mean_ok": true,
|
||||
"overall": "✅"
|
||||
},
|
||||
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_1_512_256.jsonl"
|
||||
},
|
||||
{
|
||||
"name": "c32_i4000_o512",
|
||||
"config": {
|
||||
"phase": "mtp",
|
||||
"concurrency": 32,
|
||||
"input_len": 4000,
|
||||
"output_len": 512,
|
||||
"dataset": "random",
|
||||
"num_prompts": 64
|
||||
},
|
||||
"metrics": {
|
||||
"success": 64,
|
||||
"failed": 0,
|
||||
"duration_s": 10.122198707002099,
|
||||
"request_throughput": 6.322736971733974,
|
||||
"input_token_throughput": 13280.415045301297,
|
||||
"output_token_throughput": 1785.086474097831,
|
||||
"total_token_throughput": 15065.501519399128,
|
||||
"total_input_tokens": 134427,
|
||||
"total_output_tokens": 18069,
|
||||
"e2e_ms": {
|
||||
"mean": 4431.7270394060415,
|
||||
"p50": 4118.452171998797,
|
||||
"p90": 7965.46315389569,
|
||||
"p95": 8157.508349696581,
|
||||
"p99": 8520.712041212246
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 409.17025550083963,
|
||||
"p50": 412.25154899439076,
|
||||
"p90": 663.4122133953497,
|
||||
"p95": 667.7715254438226,
|
||||
"p99": 673.1776816866477
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 14.18404794198284,
|
||||
"p50": 15.168117225958857,
|
||||
"p90": 17.233481520907418,
|
||||
"p95": 17.483692691057605,
|
||||
"p99": 19.836913968959017
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 28.557059211652966,
|
||||
"p50": 13.894067000364885,
|
||||
"p90": 29.86279199831188,
|
||||
"p95": 174.40551639592738,
|
||||
"p99": 181.762090458069
|
||||
}
|
||||
},
|
||||
"slo_status": {
|
||||
"ttft_p95_ok": true,
|
||||
"tpot_mean_ok": true,
|
||||
"overall": "✅"
|
||||
},
|
||||
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_32_4000_512.jsonl"
|
||||
},
|
||||
{
|
||||
"name": "c32_i512_o256",
|
||||
"config": {
|
||||
"phase": "mtp",
|
||||
"concurrency": 32,
|
||||
"input_len": 512,
|
||||
"output_len": 256,
|
||||
"dataset": "random",
|
||||
"num_prompts": 128
|
||||
},
|
||||
"metrics": {
|
||||
"success": 128,
|
||||
"failed": 0,
|
||||
"duration_s": 10.193132108004647,
|
||||
"request_throughput": 12.55747484126904,
|
||||
"input_token_throughput": 3458.0146344144614,
|
||||
"output_token_throughput": 1657.9791001363028,
|
||||
"total_token_throughput": 5115.993734550764,
|
||||
"total_input_tokens": 35248,
|
||||
"total_output_tokens": 16900,
|
||||
"e2e_ms": {
|
||||
"mean": 2332.5013259607204,
|
||||
"p50": 2112.3746865050634,
|
||||
"p90": 4070.417974996962,
|
||||
"p95": 4812.899109590944,
|
||||
"p99": 5073.772265354783
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": 279.9133442812263,
|
||||
"p50": 206.422848496004,
|
||||
"p90": 663.5394259006716,
|
||||
"p95": 761.6650494928763,
|
||||
"p99": 766.2871305434965
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": 16.7286719523913,
|
||||
"p50": 16.23081167053897,
|
||||
"p90": 23.97215435499593,
|
||||
"p95": 26.244743178375007,
|
||||
"p99": 37.558756268073765
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": 30.42237174675025,
|
||||
"p50": 14.531328000884969,
|
||||
"p90": 61.01081449742196,
|
||||
"p95": 163.01492899947334,
|
||||
"p99": 182.57800715073245
|
||||
}
|
||||
},
|
||||
"slo_status": {
|
||||
"ttft_p95_ok": true,
|
||||
"tpot_mean_ok": true,
|
||||
"overall": "✅"
|
||||
},
|
||||
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_32_512_256.jsonl"
|
||||
}
|
||||
]
|
||||
}
|
||||
@ -0,0 +1,86 @@
|
||||
# 投机解码策略对比：vLLM+MTP vs vLLM+DSpark vs vLLM default
|
||||
|
||||
> 数据来自同一批 default baseline（`dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default`），MTP 与 DSpark 分别与之对比后合并。
|
||||
|
||||
## 配置差异
|
||||
|
||||
| 维度 | vLLM default | vLLM+MTP | vLLM+DSpark |
|
||||
|---|---|---|---|
|
||||
| 模型 checkpoint | `DeepSeek-V4-Flash` | `DeepSeek-V4-Flash` | `DeepSeek-V4-Flash-DSpark` |
|
||||
| 运行环境 | `envs/vllm` | `envs/vllm` | `envs/vllm-dspark` |
|
||||
| 投机方法 | 无 | MTP (`num_speculative_tokens=1`) | DSpark (`spec-tokens=5`) |
|
||||
| 启动参数 | 无投机 | `--speculative-config '{"method":"mtp","num_speculative_tokens":1}'` | `--spec-method dspark --spec-model <model> --spec-tokens 5` |
|
||||
|
||||
**注意**：DSpark 同时换了模型 checkpoint 和 vLLM 环境，因此不完全是"方法 vs 方法"的公平对比，而是"DSpark 方案整体" vs "官方 MTP 方案"的对比。
|
||||
|
||||
## 三向结果表
|
||||
|
||||
| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | Mean TTFT(ms) | P95 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c1_i512_o256 | vllm-default | 1 | 512 | 256 | 0.99 | 143.82 | 49.50 | 68.50 | 6.66 | 6.71 | 1010.98 |
|
||||
| c1_i512_o256 | vllm-mtp | 1 | 512 | 256 | 1.64 | 239.37 | 60.63 | 70.69 | 3.80 | 4.33 | 607.04 |
|
||||
| c1_i512_o256 | vllm-dspark | 1 | 512 | 256 | 2.56 | 372.62 | 86.51 | 185.00 | 2.09 | 3.83 | 389.62 |
|
||||
| c32_i512_o256 | vllm-default | 32 | 512 | 256 | 12.98 | 1713.20 | 122.62 | 272.00 | 15.77 | 23.26 | 2154.65 |
|
||||
| c32_i512_o256 | vllm-mtp | 32 | 512 | 256 | 12.56 | 1657.98 | 279.91 | 761.67 | 16.73 | 37.56 | 2332.50 |
|
||||
| c32_i512_o256 | vllm-dspark | 32 | 512 | 256 | 16.53 | 2182.83 | 262.56 | 527.21 | 12.06 | 28.16 | 1738.79 |
|
||||
| c128_i512_o256 | vllm-default | 128 | 512 | 256 | 29.99 | 3959.89 | 617.72 | 767.18 | 18.31 | 65.44 | 2695.91 |
|
||||
| c128_i512_o256 | vllm-mtp | 128 | 512 | 256 | 35.03 | 4625.47 | 771.59 | 1110.39 | 18.16 | 111.99 | 2506.29 |
|
||||
| c128_i512_o256 | vllm-dspark | 128 | 512 | 256 | 23.94 | 3160.81 | 778.36 | 1071.85 | 28.95 | 78.75 | 3632.34 |
|
||||
| c1_i4000_o512 | vllm-default | 1 | 4000 | 512 | 0.50 | 140.13 | 131.50 | 191.53 | 6.68 | 6.73 | 2008.09 |
|
||||
| c1_i4000_o512 | vllm-mtp | 1 | 4000 | 512 | 0.83 | 234.90 | 161.27 | 216.12 | 3.70 | 3.89 | 1197.52 |
|
||||
| c1_i4000_o512 | vllm-dspark | 1 | 4000 | 512 | 1.73 | 486.20 | 123.58 | 176.29 | 1.66 | 3.56 | 578.26 |
|
||||
| c32_i4000_o512 | vllm-default | 32 | 4000 | 512 | 5.38 | 1518.49 | 382.74 | 681.98 | 16.43 | 25.49 | 4927.71 |
|
||||
| c32_i4000_o512 | vllm-mtp | 32 | 4000 | 512 | 6.32 | 1785.09 | 409.17 | 667.77 | 14.18 | 19.84 | 4431.73 |
|
||||
| c32_i4000_o512 | vllm-dspark | 32 | 4000 | 512 | 11.49 | 3242.88 | 298.42 | 434.24 | 9.68 | 54.60 | 2476.82 |
|
||||
|
||||
## 分场景结论
|
||||
|
||||
### 短输入（512 tokens）
|
||||
|
||||
| 场景 | 最优 TTFT | 最优 TPOT | 最优吞吐 | 综合推荐 |
|
||||
|---|---|---|---|---|
|
||||
| c1_i512_o256 | default | DSpark | DSpark | DSpark（TTFT 可接受，TPOT/E2E 大幅提升） |
|
||||
| c32_i512_o256 | default | DSpark | DSpark | DSpark（TTFT 与 MTP 接近，TPOT 更好） |
|
||||
| c128_i512_o256 | default | MTP | MTP | default（如果 TTFT 优先）；MTP（如果吞吐优先） |
|
||||
|
||||
### 长输入（4000 tokens）
|
||||
|
||||
| 场景 | 最优 TTFT | 最优 TPOT | 最优吞吐 | 综合推荐 |
|
||||
|---|---|---|---|---|
|
||||
| c1_i4000_o512 | DSpark | DSpark | DSpark | DSpark |
|
||||
| c32_i4000_o512 | DSpark | DSpark | DSpark | DSpark |
|
||||
|
||||
## 总体判断
|
||||
|
||||
### DSpark 整体更优，但有前提
|
||||
|
||||
1. **长输入场景（4000 tokens）DSpark 全胜**：TTFT、TPOT、吞吐均优于 MTP 和 default。
|
||||
2. **低并发短输入 DSpark 也更好**：虽然 TTFT 比 default 慢，但 TPOT 和 E2E 提升明显，吞吐更高。
|
||||
3. **高并发短输入（c128_i512_o256）DSpark 反而不如 MTP**：
|
||||
- DSpark 的 TPOT（28.95ms）显著差于 MTP（18.16ms）和 default（18.31ms）
|
||||
- DSpark 吞吐（23.94 req/s）也低于 default（29.99）和 MTP（35.03）
|
||||
- 这说明 DSpark 的 `spec-tokens=5` 在高并发短输入下会造成较重的 verification/调度开销，整体效率下降。
|
||||
|
||||
### MTP 的定位
|
||||
|
||||
- **优势**：使用官方 recipe，部署简单，不需要换模型 checkpoint 和 vLLM 环境；在高并发短输入下吞吐表现最好。
|
||||
- **劣势**：长输入和低并发下收益不如 DSpark；在 c32_i512_o256 这类场景中 TTFT overhead 比 DSpark 还大（P95 761ms vs 527ms）。
|
||||
|
||||
### default 的定位
|
||||
|
||||
- 在 TTFT 敏感且输入短的场景（尤其是高并发）下，default 仍然是最稳的选择。
|
||||
|
||||
## 推荐策略
|
||||
|
||||
| 应用场景 | 推荐方案 |
|
||||
|---|---|
|
||||
| 长输入 / 高输出（如文档总结、代码生成） | **DSpark** |
|
||||
| 低并发交互（如单用户 Chat） | **DSpark**（如果可接受略高 TTFT） |
|
||||
| 高并发短输入 API（如 S0/S1 轻量层） | **default** 或 **MTP** |
|
||||
| 不想换模型 checkpoint，又想尝试投机解码 | **MTP** |
|
||||
|
||||
## 后续可验证的优化点
|
||||
|
||||
1. **给 DSpark 调小 `spec-tokens`**：在短输入高并发场景测试 `spec-tokens=1/2/3`，可能改善 c128_i512_o256 的 TPOT 和吞吐。
|
||||
2. **给 MTP 调大 `num_speculative_tokens`**：当前是 1，尝试 2-4，看能否在长输入下接近 DSpark 的收益。
|
||||
3. **统一模型 checkpoint 对比**：如果可能，用同一个模型分别跑 DSpark 方法和 MTP 方法，排除模型差异带来的偏差。
|
||||
216
experiments/dsv4_h200_vllm_mtp_vs_default/run_bench.sh
Executable file
216
experiments/dsv4_h200_vllm_mtp_vs_default/run_bench.sh
Executable file
@ -0,0 +1,216 @@
|
||||
#!/usr/bin/env bash
|
||||
# vLLM+MTP speculative decoding vs vLLM default comparison on H200.
|
||||
# Reuses the default-side results from dsv4_h200_vllm_dspark_vs_default.
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||
RESULT_BASE="${SCRIPT_DIR}/results"
|
||||
|
||||
log_dir_global="${RESULT_BASE}/${RUN_ID}/logs"
|
||||
mkdir -p "$log_dir_global"
|
||||
log_init "${log_dir_global}/orchestrator.log"
|
||||
|
||||
log "experiment=${EXPERIMENT_NAME}"
|
||||
log "run_id=${RUN_ID}"
|
||||
log "platform=${PLATFORM}"
|
||||
log "hardware=${HARDWARE}"
|
||||
log "model=${MODEL_PATH}"
|
||||
log "spec_method=${SPEC_METHOD}"
|
||||
log "spec_tokens=${SPEC_TOKENS}"
|
||||
log "default_result_root=${DEFAULT_RESULT_ROOT}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
is_server_healthy() {
|
||||
local port="$1"
|
||||
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${port}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
stop_server() {
|
||||
local pid_file="/data/user1/yy/${EXPERIMENT_NAME}_mtp.pid"
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping mtp server pid=${pid}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
pkill -9 -f "vllm serve.*DeepSeek-V4-Flash" 2>/dev/null || true
|
||||
sleep 2
|
||||
}
|
||||
|
||||
start_server() {
|
||||
log "starting mtp server with ${MTP_START_SCRIPT}"
|
||||
bash "${MTP_START_SCRIPT}" >> "${log_dir_global}/mtp.server.outer.log" 2>&1
|
||||
|
||||
if ! is_server_healthy "$MTP_PORT"; then
|
||||
log "error: mtp server failed to become healthy"
|
||||
return 1
|
||||
fi
|
||||
log "mtp server is healthy on port ${MTP_PORT}"
|
||||
}
|
||||
|
||||
run_warmup() {
|
||||
log "warming up mtp (input=4000, output=512, num=2)"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/warmup.py" \
|
||||
--backend vllm \
|
||||
--port "$MTP_PORT" \
|
||||
--input-len 4000 \
|
||||
--output-len 512 \
|
||||
--num 2 \
|
||||
--env-python "${VENV_CLIENT}/bin/python" \
|
||||
>> "${log_dir_global}/mtp.warmup.log" 2>&1
|
||||
log "warmup for mtp completed"
|
||||
}
|
||||
|
||||
scenario_already_completed() {
|
||||
local output_file="$1"
|
||||
local expected="$2"
|
||||
[[ -s "$output_file" ]] || return 1
|
||||
local completed
|
||||
completed="$(${VENV_CLIENT}/bin/python -c "
|
||||
import json, sys
|
||||
path = sys.argv[1]
|
||||
try:
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
data = json.loads(line)
|
||||
print(data.get('completed', 0))
|
||||
break
|
||||
except Exception:
|
||||
print(0)
|
||||
" "$output_file")"
|
||||
[[ "${completed:-0}" -ge "$expected" ]]
|
||||
}
|
||||
|
||||
run_benchmark() {
|
||||
local result_root="${RESULT_BASE}/${RUN_ID}/mtp"
|
||||
local raw_dir="${result_root}/raw_outputs"
|
||||
local bench_log_dir="${result_root}/logs"
|
||||
|
||||
mkdir -p "$raw_dir" "$bench_log_dir"
|
||||
|
||||
log "===== mtp BENCHMARK START ====="
|
||||
|
||||
stop_server
|
||||
start_server
|
||||
run_warmup
|
||||
|
||||
for scenario in "${SCENARIOS[@]}"; do
|
||||
read -r concurrency input_len output_len num_prompts <<< "$scenario"
|
||||
output_file="${raw_dir}/vllm_mtp_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||
detail_log="${bench_log_dir}/vllm_mtp_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||
|
||||
if scenario_already_completed "$output_file" "$num_prompts"; then
|
||||
log "skipping already-completed mtp scenario: c=${concurrency} i=${input_len} o=${output_len}"
|
||||
continue
|
||||
fi
|
||||
|
||||
log "running mtp scenario: c=${concurrency} i=${input_len} o=${output_len} n=${num_prompts}"
|
||||
|
||||
"${VENV_CLIENT}/bin/python" -m sglang.bench_serving \
|
||||
--backend vllm \
|
||||
--host 127.0.0.1 \
|
||||
--port "$MTP_PORT" \
|
||||
--dataset-name random \
|
||||
--random-input-len "$input_len" \
|
||||
--random-output-len "$output_len" \
|
||||
--num-prompts "$num_prompts" \
|
||||
--max-concurrency "$concurrency" \
|
||||
--request-rate 10000 \
|
||||
--output-file "$output_file" \
|
||||
--output-details \
|
||||
> "$detail_log" 2>&1 || {
|
||||
log "ERROR: mtp scenario c=${concurrency} i=${input_len} o=${output_len} failed; see ${detail_log}"
|
||||
continue
|
||||
}
|
||||
|
||||
log "finished mtp scenario: output=${output_file}"
|
||||
done
|
||||
|
||||
stop_server
|
||||
log "===== mtp BENCHMARK DONE ====="
|
||||
}
|
||||
|
||||
parse_mtp() {
|
||||
local result_root="${RESULT_BASE}/${RUN_ID}/mtp"
|
||||
log "parsing mtp results in ${result_root}"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend vllm \
|
||||
>> "${result_root}/logs/parse.log" 2>&1 || {
|
||||
log "WARNING: parser failed for mtp; see ${result_root}/logs/parse.log"
|
||||
}
|
||||
}
|
||||
|
||||
write_mtp_metadata() {
|
||||
local result_root="${RESULT_BASE}/${RUN_ID}/mtp"
|
||||
ensure_result_root "$result_root"
|
||||
local meta_json="${result_root}/results.json"
|
||||
|
||||
local server_args
|
||||
server_args="vllm serve $MODEL_PATH --trust-remote-code --tensor-parallel-size $TP --kv-cache-dtype fp8 --max-model-len $MAX_MODEL_LEN --max-num-seqs $MAX_NUM_SEQS --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --speculative-config {\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":$SPEC_TOKENS} --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port $MTP_PORT"
|
||||
|
||||
write_metadata_json \
|
||||
"$meta_json" \
|
||||
"${EXPERIMENT_NAME}_mtp" \
|
||||
"$RUN_ID" \
|
||||
"$MODEL_PATH" \
|
||||
"vllm" \
|
||||
"vllm-mtp" \
|
||||
"$HARDWARE" \
|
||||
"$ACCELERATOR" \
|
||||
"$CHIP" \
|
||||
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||
"$VENV_VLLM" \
|
||||
"H200 vLLM+MTP TP=8 benchmark for DeepSeek-V4-Flash"
|
||||
|
||||
jq --arg server_args "$server_args" \
|
||||
'.config = {
|
||||
"tp": 8,
|
||||
"cuda_visible_devices": "0,1,2,3,4,5,6,7",
|
||||
"max_model_len": 32768,
|
||||
"max_num_seqs": 256,
|
||||
"spec_method": "mtp",
|
||||
"spec_tokens": 1,
|
||||
"backend": "vllm",
|
||||
"server_start_script": "experiments/dsv4_h200_vllm_mtp_vs_default/start_mtp.sh",
|
||||
"server_args": $server_args
|
||||
}' "$meta_json" > "${meta_json}.tmp" && mv "${meta_json}.tmp" "$meta_json"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
stop_server
|
||||
write_mtp_metadata
|
||||
run_benchmark
|
||||
parse_mtp
|
||||
|
||||
log "generating comparison report"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/compare.py" \
|
||||
--mtp "${RESULT_BASE}/${RUN_ID}/mtp" \
|
||||
--default "$DEFAULT_RESULT_ROOT" \
|
||||
--output "${RESULT_BASE}/${RUN_ID}/comparison.md" \
|
||||
>> "${log_dir_global}/compare.log" 2>&1 || {
|
||||
log "WARNING: comparison script failed; see ${log_dir_global}/compare.log"
|
||||
}
|
||||
|
||||
log "all results saved to ${RESULT_BASE}/${RUN_ID}"
|
||||
66
experiments/dsv4_h200_vllm_mtp_vs_default/start_mtp.sh
Executable file
66
experiments/dsv4_h200_vllm_mtp_vs_default/start_mtp.sh
Executable file
@ -0,0 +1,66 @@
|
||||
#!/bin/bash
|
||||
# Start vLLM with MTP speculative decoding for the comparison experiment.
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
cd /data/user1/yy
|
||||
mkdir -p logs tmp
|
||||
|
||||
VENV="${VENV_VLLM}"
|
||||
export PATH="$VENV/bin:$PATH"
|
||||
export PYTHONUNBUFFERED=1
|
||||
export TMPDIR=/data/user1/yy/tmp
|
||||
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}"
|
||||
|
||||
LOG="/data/user1/yy/logs/dsv4_h200_vllm_mtp_vs_default_mtp_$(date +%Y%m%d_%H%M%S).log"
|
||||
PID_FILE="/data/user1/yy/dsv4_h200_vllm_mtp_vs_default_mtp.pid"
|
||||
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
echo "=== Starting vLLM+MTP server (TP=$TP, spec_method=$SPEC_METHOD, spec_tokens=$SPEC_TOKENS) ==="
|
||||
echo "Model: $MODEL_PATH"
|
||||
echo "Port: $MTP_PORT"
|
||||
echo "Log: $LOG"
|
||||
|
||||
nohup vllm serve "$MODEL_PATH" \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size "$TP" \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-model-len "$MAX_MODEL_LEN" \
|
||||
--max-num-seqs "$MAX_NUM_SEQS" \
|
||||
--block-size 256 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--tokenizer-mode deepseek_v4 \
|
||||
--reasoning-parser deepseek_v4 \
|
||||
--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$SPEC_TOKENS}" \
|
||||
--no-disable-hybrid-kv-cache-manager \
|
||||
--disable-uvicorn-access-log \
|
||||
--port "$MTP_PORT" \
|
||||
> "$LOG" 2>&1 &
|
||||
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
echo "Waiting for health..."
|
||||
|
||||
for i in $(seq 1 240); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${MTP_PORT}/health" >/dev/null 2>&1; then
|
||||
echo "vLLM+MTP server is ready at http://127.0.0.1:${MTP_PORT}"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
if ! kill -0 $PID 2>/dev/null; then
|
||||
echo "ERROR: vLLM+MTP server exited early"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/240)"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: vLLM+MTP server not healthy after 240 retries"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
271
experiments/dsv4_h200_vllm_tp2_custom_bench/README.md
Normal file
271
experiments/dsv4_h200_vllm_tp2_custom_bench/README.md
Normal file
@ -0,0 +1,271 @@
|
||||
# dsv4_h200_vllm_tp2_custom_bench
|
||||
|
||||
## 目的
|
||||
|
||||
当 vLLM 使用 **TP=2**（Tensor Parallel = 2）部署时，单张 H200 上可以同时运行 **4 个独立服务**（8 卡 / 2 = 4 服务），每个服务独占 2 张 GPU：
|
||||
|
||||
| 服务 | 端口 | GPU 对 |
|
||||
|------|------|--------|
|
||||
| 1 | 30005 | 0, 1 |
|
||||
| 2 | 30006 | 2, 3 |
|
||||
| 3 | 30007 | 4, 5 |
|
||||
| 4 | 30008 | 6, 7 |
|
||||
|
||||
此时 `sglang.bench_serving` 存在以下问题：
|
||||
|
||||
1. **低并发下无法测出真实吞吐**：并发数不大时，请求可能只落在一个服务上，其他 3 个服务空闲，bench 报告的吞吐只是单服务的 1/4。
|
||||
2. **TTFT 测量不够精确**：`sglang.bench_serving` 的计时粒度较粗，对于低延迟场景（TP=2 下 TTFT 可能 < 100ms）误差较大。
|
||||
3. **不支持多服务负载均衡**：只能测一个端口，无法反映整个 8 卡集群性能。
|
||||
|
||||
因此本实验使用 **自定义压测客户端 `bench_client.py`**，直接调用 OpenAI `/v1/chat/completions` API，通过 `aiohttp` 并发发送请求，并记录每个请求的精确时间戳。同时通过 **负载均衡** 将请求均匀分发到 4 个服务上。
|
||||
|
||||
## 与 sglang.bench_serving 的区别
|
||||
|
||||
| 特性 | sglang.bench_serving | bench_client.py (本实验) |
|
||||
|------|----------------------|--------------------------|
|
||||
| 协议 | 内部协议 | OpenAI API (`/v1/chat/completions`) |
|
||||
| 并发控制 | 进程级 | `asyncio.Semaphore` |
|
||||
| TTFT 测量 | 粗略 | 精确到 `time.perf_counter()` |
|
||||
| TPOT 测量 | 基于总时间估算 | 基于首 token 后每个 token 的间隔 |
|
||||
| 多服务支持 | ❌ 单端口 | ✅ 轮询/随机分发到多个端口 |
|
||||
| 低并发精度 | 一般 | 高（适合 TP=2 场景） |
|
||||
| 集群吞吐 | 只能测单服务 | 可测 4 服务总吞吐 |
|
||||
|
||||
## 文件说明
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `bench_client.py` | 自定义异步压测客户端，支持多服务负载均衡 |
|
||||
| `config.env` | 实验配置（场景、模型路径、端口列表、LB 策略等） |
|
||||
| `run_bench.sh` | 编排脚本：启动 4 个服务 → 运行压测 → 解析结果 |
|
||||
| `start_server.sh` | 同时启动 4 个 vLLM TP=2 服务（每服务 2 GPU） |
|
||||
| `parse_results.py` | 解析 bench_client.py 输出的 JSONL，生成 `results.json` + `report.md` |
|
||||
|
||||
## 快速运行
|
||||
|
||||
```bash
|
||||
# 完整流程（自动启动 4 个服务、压测、生成报告）
|
||||
bash experiments/dsv4_h200_vllm_tp2_custom_bench/run_bench.sh
|
||||
|
||||
# 跳过服务管理（已有 4 个服务在运行）
|
||||
SKIP_MANAGE_SERVER=1 bash experiments/dsv4_h200_vllm_tp2_custom_bench/run_bench.sh
|
||||
|
||||
# 单独运行压测客户端（多服务，带健康检查）
|
||||
python3 experiments/dsv4_h200_vllm_tp2_custom_bench/bench_client.py \
|
||||
--host 127.0.0.1 \
|
||||
--ports 30005,30006,30007,30008 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 32 --input-len 512 --output-len 256 --num-prompts 640 \
|
||||
--lb-strategy round_robin \
|
||||
--health-check-interval 5 \
|
||||
--health-max-failures 2 \
|
||||
--health-recovery-interval 10 \
|
||||
--request-max-retries 2 \
|
||||
--output-file /tmp/test.jsonl
|
||||
|
||||
# 单独运行压测客户端（单服务，向后兼容）
|
||||
python3 experiments/dsv4_h200_vllm_tp2_custom_bench/bench_client.py \
|
||||
--host 127.0.0.1 --port 30005 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 8 --input-len 512 --output-len 256 --num-prompts 160 \
|
||||
--output-file /tmp/test_single.jsonl
|
||||
|
||||
# 解析已有结果
|
||||
python3 experiments/dsv4_h200_vllm_tp2_custom_bench/parse_results.py \
|
||||
experiments/dsv4_h200_vllm_tp2_custom_bench/results/<RUN_ID>
|
||||
```
|
||||
|
||||
## 场景设计
|
||||
|
||||
本实验覆盖了从 **单并发** 到 **128 并发** 的梯度，以及不同输入/输出长度组合：
|
||||
|
||||
| 并发 | 输入长度 | 输出长度 | 请求数 | 说明 |
|
||||
|------|----------|----------|--------|------|
|
||||
| 1 | 512 | 256 | 20 | 单并发基线，测纯延迟 |
|
||||
| 4 | 512 | 256 | 80 | 低并发，4 服务各 1 req |
|
||||
| 8 | 512 | 256 | 160 | 中低并发 |
|
||||
| 16 | 512 | 256 | 320 | 中等并发 |
|
||||
| 32 | 512 | 256 | 640 | 中高并发 |
|
||||
| 64 | 512 | 256 | 1280 | 高并发 |
|
||||
| 128 | 512 | 256 | 2560 | 极限并发，测集群吞吐上限 |
|
||||
| 1 | 2048 | 512 | 20 | 长输入基线 |
|
||||
| 8 | 2048 | 512 | 160 | 长输入中并发 |
|
||||
| 32 | 2048 | 512 | 640 | 长输入高并发 |
|
||||
| 128 | 2048 | 512 | 2560 | 长输入极限并发 |
|
||||
| 1 | 4096 | 1024 | 20 | 超长输入基线 |
|
||||
| 8 | 4096 | 1024 | 160 | 超长输入中并发 |
|
||||
| 32 | 4096 | 1024 | 640 | 超长输入高并发 |
|
||||
| 128 | 4096 | 1024 | 2560 | 超长输入极限并发 |
|
||||
|
||||
## 健康检查与故障自动跳过
|
||||
|
||||
`bench_client.py` 内置了 **后台健康检查 + 请求级重试** 机制，确保某个服务挂了不会导致整个 benchmark 失败。
|
||||
|
||||
### 健康检查机制
|
||||
|
||||
| 参数 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `--health-check-interval` | 5.0 | 每 N 秒检查一次所有服务 |
|
||||
| `--health-max-failures` | 2 | 连续失败 N 次才标记为不健康 |
|
||||
| `--health-recovery-interval` | 10.0 | 不健康服务 N 秒后自动重试恢复 |
|
||||
| `--request-max-retries` | 2 | 单个请求失败最多重试 N 次 |
|
||||
|
||||
### 故障处理流程
|
||||
|
||||
```
|
||||
请求 → 负载均衡选择端口 → 发送请求
|
||||
↓
|
||||
成功 → 记录结果
|
||||
↓
|
||||
失败 → 换另一个端口重试（最多 --request-max-retries 次）
|
||||
↓
|
||||
全部失败 → 记录失败，继续下一个请求
|
||||
```
|
||||
|
||||
后台同时运行健康检查：
|
||||
- 每 5 秒对所有端口执行 `/health` 探测
|
||||
- 连续 2 次失败的服务被标记为 **unhealthy**，不再接收新请求
|
||||
- 10 秒后自动尝试恢复，如果恢复成功则重新加入负载均衡池
|
||||
|
||||
### 输出中的故障信息
|
||||
|
||||
汇总 JSON 包含以下字段：
|
||||
|
||||
```json
|
||||
{
|
||||
"completed": 638,
|
||||
"failed": 2,
|
||||
"retried": 5,
|
||||
"healthy_ports_at_end": [30005, 30006, 30007],
|
||||
"per_port_stats": {
|
||||
"30005": {"count": 160, "failed": 0, ...},
|
||||
"30006": {"count": 159, "failed": 1, ...},
|
||||
"30007": {"count": 160, "failed": 0, ...},
|
||||
"30008": {"count": 159, "failed": 1, ...}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `failed`: 最终失败的请求数（重试后仍失败）
|
||||
- `retried`: 至少重试过一次的请求数
|
||||
- `healthy_ports_at_end`: 压测结束时仍健康的服务端口
|
||||
- `per_port_stats[].failed`: 每个端口的失败数
|
||||
|
||||
### 场景间健康检查
|
||||
|
||||
`run_bench.sh` 在每个场景结束后会打印一次服务健康状态：
|
||||
|
||||
```
|
||||
[2026-07-10T08:45:00+00:00] service health check: healthy=[30005 30006 30007 30008] unhealthy=[]
|
||||
[2026-07-10T08:45:30+00:00] service health check: healthy=[30005 30006 30007] unhealthy=[30008]
|
||||
[2026-07-10T08:45:30+00:00] WARNING: 1 service(s) unhealthy: 30008
|
||||
```
|
||||
|
||||
这样即使某个服务在压测中途 OOM 崩溃，其他服务仍能继续工作，benchmark 不会中断。
|
||||
|
||||
### 自定义故障参数
|
||||
|
||||
```bash
|
||||
python3 bench_client.py \
|
||||
--ports 30005,30006,30007,30008 \
|
||||
--health-check-interval 3 \
|
||||
--health-max-failures 1 \
|
||||
--health-recovery-interval 5 \
|
||||
--request-max-retries 3 \
|
||||
...
|
||||
```
|
||||
|
||||
- 更敏感：间隔 3 秒、1 次失败即标记不健康、5 秒恢复
|
||||
- 更容错：单个请求最多重试 3 次
|
||||
|
||||
## 负载均衡策略
|
||||
|
||||
`bench_client.py` 支持三种负载均衡策略：
|
||||
|
||||
| 策略 | 说明 | 适用场景 |
|
||||
|------|------|----------|
|
||||
| `round_robin` | 请求按顺序轮询分配到各端口 | **默认**，最公平，推荐 |
|
||||
| `random` | 随机选择一个端口 | 简单，分布可能不均 |
|
||||
| `least_conn` | 选择当前连接数最少的端口 | 需要追踪 in-flight，当前为 random 占位 |
|
||||
|
||||
通过 `--lb-strategy` 参数选择：
|
||||
|
||||
```bash
|
||||
python3 bench_client.py --lb-strategy round_robin --ports 30005,30006,30007,30008 ...
|
||||
```
|
||||
|
||||
## 输出格式
|
||||
|
||||
### JSONL 原始数据 (`raw_outputs/vllm_*.jsonl`)
|
||||
|
||||
每行一个 JSON 对象，包含请求分发到的目标端口：
|
||||
|
||||
```json
|
||||
{
|
||||
"request_id": 0,
|
||||
"input_tokens": 512,
|
||||
"output_tokens": 256,
|
||||
"first_token_time_ms": 45.23,
|
||||
"e2e_latency_ms": 5234.56,
|
||||
"inter_token_latencies": [18.5, 19.2, ...],
|
||||
"tpot_ms": 19.8,
|
||||
"mean_itl_ms": 19.5,
|
||||
"success": true,
|
||||
"error": "",
|
||||
"target_port": 30005
|
||||
}
|
||||
```
|
||||
|
||||
最后一行是汇总数据，包含 **各端口统计**：
|
||||
|
||||
```json
|
||||
{
|
||||
"completed": 640,
|
||||
"failed": 0,
|
||||
"duration": 12.34,
|
||||
"request_throughput": 51.88,
|
||||
"mean_ttft_ms": 48.5,
|
||||
"p95_ttft_ms": 62.3,
|
||||
"per_port_stats": {
|
||||
"30005": {"count": 160, "mean_ttft_ms": 47.2, "p95_ttft_ms": 58.1, "mean_e2e_ms": 5200.0},
|
||||
"30006": {"count": 160, "mean_ttft_ms": 49.1, "p95_ttft_ms": 64.3, "mean_e2e_ms": 5250.0},
|
||||
"30007": {"count": 160, "mean_ttft_ms": 48.8, "p95_ttft_ms": 61.5, "mean_e2e_ms": 5230.0},
|
||||
"30008": {"count": 160, "mean_ttft_ms": 48.9, "p95_ttft_ms": 65.2, "mean_e2e_ms": 5270.0}
|
||||
},
|
||||
"lb_strategy": "round_robin",
|
||||
"num_services": 4
|
||||
}
|
||||
```
|
||||
|
||||
### 结构化结果 (`results.json`)
|
||||
|
||||
遵循仓库统一的 [JSON Schema](../../BENCHMARK_WORKFLOW.md#final-json-schema)。
|
||||
|
||||
### 人类可读报告 (`report.md`)
|
||||
|
||||
Markdown 表格，包含所有场景的 TTFT/TPOT/E2E 均值与分位数，以及 SLO 达标状态。
|
||||
|
||||
## 单独启动/停止 4 个服务
|
||||
|
||||
```bash
|
||||
# 启动 4 个服务
|
||||
bash experiments/dsv4_h200_vllm_tp2_custom_bench/start_server.sh
|
||||
|
||||
# 停止（run_bench.sh 会自动停止，但也可以手动）
|
||||
for port in 30005 30006 30007 30008; do
|
||||
pid_file="experiments/dsv4_h200_vllm_tp2_custom_bench/server_port${port}.pid"
|
||||
[[ -f "$pid_file" ]] && kill "$(cat "$pid_file")" 2>/dev/null || true
|
||||
rm -f "$pid_file"
|
||||
done
|
||||
pkill -9 -f "vllm serve.*DeepSeek-V4-Flash" 2>/dev/null || true
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- `bench_client.py` 优先使用 `aiohttp`，如果未安装则自动回退到 `urllib`（但会丢失 streaming 和精确 TTFT/ITL 测量）。
|
||||
- 默认 `temperature=0.0` 以保证输出长度稳定。
|
||||
- 输入 token 数通过空格分隔的单词数估算，实际 tokenizer 可能略有偏差。
|
||||
- 4 个服务共享同一模型文件，确保 `/data/models/DeepSeek-V4-Flash` 存在且可访问。
|
||||
- 每个服务占用约 2 张 GPU 的 90% 显存，确保无其他进程占用 GPU。
|
||||
- 健康检查依赖 `/health` 端点，确保 vLLM 服务已启用该端点（vLLM 默认启用）。
|
||||
- 如果某个服务频繁崩溃，检查 GPU 显存是否充足（OOM 是最常见原因）。
|
||||
652
experiments/dsv4_h200_vllm_tp2_custom_bench/bench_client.py
Executable file
652
experiments/dsv4_h200_vllm_tp2_custom_bench/bench_client.py
Executable file
@ -0,0 +1,652 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Custom benchmark client for vLLM / OpenAI-compatible API with multi-service load balancing and health-check failover.
|
||||
|
||||
Sends concurrent requests to multiple inference servers and records per-request
|
||||
latency metrics (TTFT, TPOT, ITL, E2E) with fine-grained timing.
|
||||
|
||||
Designed for TP=2 on 8-GPU setups where 4 independent services run on
|
||||
(0,1), (2,3), (4,5), (6,7). Requests are distributed across services via
|
||||
round-robin or random load balancing. Unhealthy services are automatically
|
||||
skipped with periodic retry.
|
||||
|
||||
Usage:
|
||||
# Single service (backward compatible)
|
||||
python3 bench_client.py \
|
||||
--host 127.0.0.1 --port 30005 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 64 --input-len 512 --output-len 256 --num-prompts 320 \
|
||||
--output-file results.jsonl
|
||||
|
||||
# Multi-service (4 services on ports 30005-30008)
|
||||
python3 bench_client.py \
|
||||
--host 127.0.0.1 \
|
||||
--ports 30005,30006,30007,30008 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 64 --input-len 512 --output-len 256 --num-prompts 320 \
|
||||
--lb-strategy round_robin \
|
||||
--output-file results.jsonl
|
||||
|
||||
The output JSONL contains one object per request with raw timing data.
|
||||
A final summary line ("completed" field) aggregates all requests.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Any
|
||||
|
||||
# Try aiohttp first; fall back to urllib for stdlib-only environments.
|
||||
try:
|
||||
import aiohttp
|
||||
HAS_AIOHTTP = True
|
||||
except ImportError:
|
||||
HAS_AIOHTTP = False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data structures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class RequestResult:
|
||||
request_id: int
|
||||
input_tokens: int = 0
|
||||
output_tokens: int = 0
|
||||
first_token_time_ms: float = 0.0 # TTFT
|
||||
e2e_latency_ms: float = 0.0 # total wall time
|
||||
inter_token_latencies: list[float] = field(default_factory=list)
|
||||
success: bool = False
|
||||
error: str = ""
|
||||
target_port: int = 0 # which service handled this request
|
||||
retry_count: int = 0 # how many times this request was retried
|
||||
|
||||
@property
|
||||
def tpot_ms(self) -> float:
|
||||
"""Mean TPOT = (e2e - ttft) / (output_tokens - 1) if >1 token."""
|
||||
if self.output_tokens <= 1:
|
||||
return 0.0
|
||||
return (self.e2e_latency_ms - self.first_token_time_ms) / (self.output_tokens - 1)
|
||||
|
||||
@property
|
||||
def mean_itl_ms(self) -> float:
|
||||
if not self.inter_token_latencies:
|
||||
return 0.0
|
||||
return sum(self.inter_token_latencies) / len(self.inter_token_latencies)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
d = asdict(self)
|
||||
d["tpot_ms"] = self.tpot_ms
|
||||
d["mean_itl_ms"] = self.mean_itl_ms
|
||||
return d
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Health checker
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class HealthChecker:
|
||||
"""Periodically check service health and maintain a healthy-port list."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
host: str,
|
||||
ports: list[int],
|
||||
check_interval: float = 5.0,
|
||||
max_failures: int = 2,
|
||||
recovery_interval: float = 10.0,
|
||||
):
|
||||
self.host = host
|
||||
self.all_ports = ports
|
||||
self.check_interval = check_interval
|
||||
self.max_failures = max_failures
|
||||
self.recovery_interval = recovery_interval
|
||||
|
||||
# port -> consecutive failure count
|
||||
self._failures: dict[int, int] = {p: 0 for p in ports}
|
||||
# port -> last healthy timestamp
|
||||
self._last_healthy: dict[int, float] = {p: time.monotonic() for p in ports}
|
||||
self._lock = asyncio.Lock()
|
||||
self._task: asyncio.Task | None = None
|
||||
|
||||
async def _check_one(self, port: int, session: aiohttp.ClientSession | None) -> bool:
|
||||
url = f"http://{self.host}:{port}/health"
|
||||
try:
|
||||
if HAS_AIOHTTP and session is not None:
|
||||
async with session.get(url, timeout=aiohttp.ClientTimeout(total=5)) as resp:
|
||||
return resp.status == 200
|
||||
else:
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
with urllib.request.urlopen(req, timeout=5) as resp:
|
||||
return resp.status == 200
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
async def _check_all(self, session: aiohttp.ClientSession | None) -> None:
|
||||
checks = [self._check_one(p, session) for p in self.all_ports]
|
||||
results = await asyncio.gather(*checks, return_exceptions=True)
|
||||
now = time.monotonic()
|
||||
async with self._lock:
|
||||
for port, ok in zip(self.all_ports, results):
|
||||
if isinstance(ok, Exception):
|
||||
ok = False
|
||||
if ok:
|
||||
self._failures[port] = 0
|
||||
self._last_healthy[port] = now
|
||||
else:
|
||||
self._failures[port] += 1
|
||||
|
||||
async def start(self, session: aiohttp.ClientSession | None) -> None:
|
||||
async def _loop() -> None:
|
||||
while True:
|
||||
await asyncio.sleep(self.check_interval)
|
||||
await self._check_all(session)
|
||||
self._task = asyncio.create_task(_loop())
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
@property
|
||||
async def healthy_ports(self) -> list[int]:
|
||||
now = time.monotonic()
|
||||
async with self._lock:
|
||||
healthy = []
|
||||
for p in self.all_ports:
|
||||
if self._failures[p] < self.max_failures:
|
||||
healthy.append(p)
|
||||
elif now - self._last_healthy[p] > self.recovery_interval:
|
||||
# Give it another chance after recovery_interval.
|
||||
self._failures[p] = max(0, self._failures[p] - 1)
|
||||
healthy.append(p)
|
||||
return healthy
|
||||
|
||||
async def is_healthy(self, port: int) -> bool:
|
||||
ports = await self.healthy_ports
|
||||
return port in ports
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Load balancer with failover
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class LoadBalancer:
|
||||
"""Distribute requests across healthy backend ports with auto-failover."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
ports: list[int],
|
||||
health_checker: HealthChecker,
|
||||
strategy: str = "round_robin",
|
||||
):
|
||||
self.all_ports = ports
|
||||
self.health_checker = health_checker
|
||||
self.strategy = strategy
|
||||
self._idx = 0
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
def _next_round_robin(self, healthy: list[int]) -> int:
|
||||
if not healthy:
|
||||
return self.all_ports[self._idx % len(self.all_ports)]
|
||||
for _ in range(len(self.all_ports)):
|
||||
port = self.all_ports[self._idx % len(self.all_ports)]
|
||||
self._idx += 1
|
||||
if port in healthy:
|
||||
return port
|
||||
# Fallback: all unhealthy, pick first healthy anyway.
|
||||
return healthy[0] if healthy else self.all_ports[0]
|
||||
|
||||
def _next_random(self, healthy: list[int]) -> int:
|
||||
if healthy:
|
||||
return random.choice(healthy)
|
||||
return random.choice(self.all_ports)
|
||||
|
||||
async def next_port(self) -> int:
|
||||
healthy = await self.health_checker.healthy_ports
|
||||
async with self._lock:
|
||||
if self.strategy == "round_robin":
|
||||
return self._next_round_robin(healthy)
|
||||
elif self.strategy in ("random", "least_conn"):
|
||||
return self._next_random(healthy)
|
||||
else:
|
||||
return self._next_round_robin(healthy)
|
||||
|
||||
@property
|
||||
def port_count(self) -> int:
|
||||
return len(self.all_ports)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Token helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
import uuid
|
||||
def make_prompt(token_len: int) -> str:
|
||||
prefix = str(uuid.uuid4())
|
||||
|
||||
words = [
|
||||
"the", "quick", "brown", "fox",
|
||||
"jumps", "over", "lazy", "dog",
|
||||
"benchmark", "performance", "latency"
|
||||
]
|
||||
|
||||
tokens = [prefix]
|
||||
|
||||
while len(tokens) < token_len:
|
||||
tokens.extend(words)
|
||||
|
||||
return " ".join(tokens[:token_len])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# aiohttp-based async request (preferred)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def send_request_aiohttp(
|
||||
session: aiohttp.ClientSession,
|
||||
host: str,
|
||||
port: int,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
) -> RequestResult:
|
||||
url = f"http://{host}:{port}/v1/chat/completions"
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"stream": True,
|
||||
}
|
||||
headers = {"Content-Type": "application/json"}
|
||||
|
||||
result = RequestResult(request_id=request_id, target_port=port)
|
||||
start_time = time.perf_counter()
|
||||
first_token_received = False
|
||||
last_token_time = 0.0
|
||||
|
||||
try:
|
||||
async with session.post(url, json=payload, headers=headers) as resp:
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
result.error = f"HTTP {resp.status}: {text[:200]}"
|
||||
return result
|
||||
|
||||
async for line in resp.content:
|
||||
line = line.decode("utf-8").strip()
|
||||
if not line or not line.startswith("data: "):
|
||||
continue
|
||||
data_str = line[len("data: "):]
|
||||
if data_str == "[DONE]":
|
||||
break
|
||||
try:
|
||||
chunk = json.loads(data_str)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
choices = chunk.get("choices", [])
|
||||
if not choices:
|
||||
continue
|
||||
|
||||
delta = choices[0].get("delta", {})
|
||||
content = delta.get("content", "")
|
||||
if content:
|
||||
now = time.perf_counter()
|
||||
if not first_token_received:
|
||||
result.first_token_time_ms = (now - start_time) * 1000
|
||||
first_token_received = True
|
||||
else:
|
||||
result.inter_token_latencies.append((now - last_token_time) * 1000)
|
||||
last_token_time = now
|
||||
result.output_tokens += 1
|
||||
|
||||
result.e2e_latency_ms = (time.perf_counter() - start_time) * 1000
|
||||
result.input_tokens = len(prompt.split()) # approximate
|
||||
result.success = True
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
result.error = "timeout"
|
||||
except Exception as e:
|
||||
result.error = str(e)[:200]
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# urllib-based synchronous request (fallback, no aiohttp)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def send_request_urllib(
|
||||
host: str,
|
||||
port: int,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
) -> RequestResult:
|
||||
"""Blocking fallback using stdlib urllib. Used when aiohttp is absent."""
|
||||
url = f"http://{host}:{port}/v1/chat/completions"
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"stream": False,
|
||||
}
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
headers = {"Content-Type": "application/json", "Content-Length": str(len(data))}
|
||||
|
||||
result = RequestResult(request_id=request_id, target_port=port)
|
||||
start_time = time.perf_counter()
|
||||
|
||||
try:
|
||||
req = urllib.request.Request(url, data=data, headers=headers, method="POST")
|
||||
with urllib.request.urlopen(req, timeout=600) as resp:
|
||||
body = json.loads(resp.read().decode("utf-8"))
|
||||
|
||||
choices = body.get("choices", [])
|
||||
if not choices:
|
||||
result.error = "no choices in response"
|
||||
return result
|
||||
|
||||
content = choices[0].get("message", {}).get("content", "")
|
||||
result.output_tokens = len(content.split()) if content else 0
|
||||
result.e2e_latency_ms = (time.perf_counter() - start_time) * 1000
|
||||
result.first_token_time_ms = result.e2e_latency_ms
|
||||
result.input_tokens = len(prompt.split())
|
||||
result.success = True
|
||||
|
||||
except Exception as e:
|
||||
result.error = str(e)[:200]
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Unified send_request wrapper with retry & failover
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def send_request_with_retry(
|
||||
session: aiohttp.ClientSession | None,
|
||||
host: str,
|
||||
lb: LoadBalancer,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
max_retries: int = 2,
|
||||
) -> RequestResult:
|
||||
"""Send a request, retrying on failure with a different port each time."""
|
||||
result = RequestResult(request_id=request_id)
|
||||
for attempt in range(max_retries + 1):
|
||||
port = await lb.next_port()
|
||||
if HAS_AIOHTTP and session is not None:
|
||||
result = await send_request_aiohttp(
|
||||
session, host, port, model, request_id, prompt, max_tokens, temperature
|
||||
)
|
||||
else:
|
||||
loop = asyncio.get_event_loop()
|
||||
result = await loop.run_in_executor(
|
||||
None, send_request_urllib,
|
||||
host, port, model, request_id, prompt, max_tokens, temperature
|
||||
)
|
||||
result.retry_count = attempt
|
||||
if result.success:
|
||||
return result
|
||||
# If failed, try another port on next iteration (unless last attempt).
|
||||
if attempt < max_retries:
|
||||
await asyncio.sleep(0.5 * (attempt + 1)) # exponential backoff-ish
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Benchmark orchestration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def run_benchmark(
|
||||
host: str,
|
||||
ports: list[int],
|
||||
lb: LoadBalancer,
|
||||
health_checker: HealthChecker,
|
||||
model: str,
|
||||
concurrency: int,
|
||||
input_len: int,
|
||||
output_len: int,
|
||||
num_prompts: int,
|
||||
temperature: float,
|
||||
output_file: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Run the benchmark and write a JSONL file."""
|
||||
prompts = [make_prompt(input_len) for _ in range(num_prompts)]
|
||||
results: list[RequestResult] = []
|
||||
semaphore = asyncio.Semaphore(concurrency)
|
||||
|
||||
start = time.perf_counter()
|
||||
|
||||
if HAS_AIOHTTP:
|
||||
connector = aiohttp.TCPConnector(limit=concurrency * 2)
|
||||
timeout = aiohttp.ClientTimeout(total=600, sock_read=300)
|
||||
async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
|
||||
await health_checker.start(session)
|
||||
|
||||
async def bounded_send(idx: int, prompt: str) -> RequestResult:
|
||||
async with semaphore:
|
||||
return await send_request_with_retry(
|
||||
session, host, lb, model, idx, prompt, output_len, temperature
|
||||
)
|
||||
print("Starting warmup...")
|
||||
warmup_per_service = 10
|
||||
warmup_requests = warmup_per_service * len(ports)
|
||||
|
||||
warmup_prompts = [
|
||||
make_prompt(input_len)
|
||||
for _ in range(warmup_requests)
|
||||
]
|
||||
|
||||
warmup_tasks = [
|
||||
send_request_with_retry(
|
||||
session=session,
|
||||
host=host,
|
||||
lb=lb,
|
||||
model=model,
|
||||
request_id=-1,
|
||||
prompt=p,
|
||||
max_tokens=output_len,
|
||||
temperature=temperature,
|
||||
)
|
||||
for p in warmup_prompts
|
||||
]
|
||||
|
||||
await asyncio.gather(*warmup_tasks)
|
||||
|
||||
print("Warmup finished.")
|
||||
|
||||
# 给 CUDA / Scheduler 一点时间稳定
|
||||
await asyncio.sleep(2)
|
||||
|
||||
tasks = [bounded_send(i, p) for i, p in enumerate(prompts)]
|
||||
results = await asyncio.gather(*tasks)
|
||||
await health_checker.stop()
|
||||
else:
|
||||
await health_checker.start(None)
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
async def bounded_send(idx: int, prompt: str) -> RequestResult:
|
||||
async with semaphore:
|
||||
return await send_request_with_retry(
|
||||
None, host, lb, model, idx, prompt, output_len, temperature
|
||||
)
|
||||
|
||||
tasks = [bounded_send(i, p) for i, p in enumerate(prompts)]
|
||||
results = await asyncio.gather(*tasks)
|
||||
await health_checker.stop()
|
||||
|
||||
duration = time.perf_counter() - start
|
||||
|
||||
# Write JSONL
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
for r in results:
|
||||
f.write(json.dumps(r.to_dict(), ensure_ascii=False) + "\n")
|
||||
|
||||
# Compute summary
|
||||
successful = [r for r in results if r.success]
|
||||
failed = len(results) - len(successful)
|
||||
retried = [r for r in results if r.retry_count > 0]
|
||||
|
||||
def percentile(values: list[float], p: float) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
s = sorted(values)
|
||||
k = (len(s) - 1) * p / 100.0
|
||||
f = int(k)
|
||||
c = min(f + 1, len(s) - 1)
|
||||
return s[f] + (k - f) * (s[c] - s[f])
|
||||
|
||||
ttfts = [r.first_token_time_ms for r in successful]
|
||||
tpots = [r.tpot_ms for r in successful if r.output_tokens > 1]
|
||||
e2es = [r.e2e_latency_ms for r in successful]
|
||||
itls = [r.mean_itl_ms for r in successful if r.inter_token_latencies]
|
||||
|
||||
total_input_tokens = sum(r.input_tokens for r in successful)
|
||||
total_output_tokens = sum(r.output_tokens for r in successful)
|
||||
|
||||
# Per-port breakdown
|
||||
per_port_stats: dict[int, dict[str, Any]] = {}
|
||||
for port in ports:
|
||||
port_results = [r for r in successful if r.target_port == port]
|
||||
port_failed = [r for r in results if r.target_port == port and not r.success]
|
||||
if port_results or port_failed:
|
||||
port_ttfts = [r.first_token_time_ms for r in port_results]
|
||||
port_e2es = [r.e2e_latency_ms for r in port_results]
|
||||
per_port_stats[port] = {
|
||||
"count": len(port_results),
|
||||
"failed": len(port_failed),
|
||||
"mean_ttft_ms": sum(port_ttfts) / len(port_ttfts) if port_ttfts else 0.0,
|
||||
"p95_ttft_ms": percentile(port_ttfts, 95) if port_ttfts else 0.0,
|
||||
"mean_e2e_ms": sum(port_e2es) / len(port_e2es) if port_e2es else 0.0,
|
||||
}
|
||||
|
||||
# Health-check stats
|
||||
healthy_at_end = await health_checker.healthy_ports
|
||||
|
||||
summary = {
|
||||
"completed": len(successful),
|
||||
"failed": failed,
|
||||
"retried": len(retried),
|
||||
"duration": duration,
|
||||
"request_throughput": len(successful) / duration if duration > 0 else 0.0,
|
||||
"input_throughput": total_input_tokens / duration if duration > 0 else 0.0,
|
||||
"output_throughput": total_output_tokens / duration if duration > 0 else 0.0,
|
||||
"total_throughput": (total_input_tokens + total_output_tokens) / duration if duration > 0 else 0.0,
|
||||
"total_input_tokens": total_input_tokens,
|
||||
"total_output_tokens": total_output_tokens,
|
||||
"mean_ttft_ms": sum(ttfts) / len(ttfts) if ttfts else 0.0,
|
||||
"median_ttft_ms": percentile(ttfts, 50),
|
||||
"p90_ttft_ms": percentile(ttfts, 90),
|
||||
"p95_ttft_ms": percentile(ttfts, 95),
|
||||
"p99_ttft_ms": percentile(ttfts, 99),
|
||||
"mean_tpot_ms": sum(tpots) / len(tpots) if tpots else 0.0,
|
||||
"median_tpot_ms": percentile(tpots, 50),
|
||||
"p90_tpot_ms": percentile(tpots, 90),
|
||||
"p95_tpot_ms": percentile(tpots, 95),
|
||||
"p99_tpot_ms": percentile(tpots, 99),
|
||||
"mean_e2e_latency_ms": sum(e2es) / len(e2es) if e2es else 0.0,
|
||||
"median_e2e_latency_ms": percentile(e2es, 50),
|
||||
"p90_e2e_latency_ms": percentile(e2es, 90),
|
||||
"p95_e2e_latency_ms": percentile(e2es, 95),
|
||||
"p99_e2e_latency_ms": percentile(e2es, 99),
|
||||
"mean_itl_ms": sum(itls) / len(itls) if itls else 0.0,
|
||||
"median_itl_ms": percentile(itls, 50),
|
||||
"p90_itl_ms": percentile(itls, 90),
|
||||
"p95_itl_ms": percentile(itls, 95),
|
||||
"p99_itl_ms": percentile(itls, 99),
|
||||
"per_port_stats": per_port_stats,
|
||||
"healthy_ports_at_end": healthy_at_end,
|
||||
"lb_strategy": lb.strategy,
|
||||
"num_services": lb.port_count,
|
||||
}
|
||||
|
||||
# Append summary as the last line
|
||||
with open(output_file, "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(summary, ensure_ascii=False) + "\n")
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Custom vLLM benchmark client with multi-service LB and health-check failover")
|
||||
parser.add_argument("--host", default="127.0.0.1")
|
||||
parser.add_argument("--port", type=int, default=30005,
|
||||
help="Single service port (backward compatible)")
|
||||
parser.add_argument("--ports", default="",
|
||||
help="Comma-separated list of ports for multi-service, e.g. 30005,30006,30007,30008")
|
||||
parser.add_argument("--model", default="deepseek-v4-flash")
|
||||
parser.add_argument("--concurrency", type=int, default=1)
|
||||
parser.add_argument("--input-len", type=int, default=512)
|
||||
parser.add_argument("--output-len", type=int, default=256)
|
||||
parser.add_argument("--num-prompts", type=int, default=10)
|
||||
parser.add_argument("--temperature", type=float, default=0.0)
|
||||
parser.add_argument("--lb-strategy", default="round_robin",
|
||||
choices=["round_robin", "random", "least_conn"],
|
||||
help="Load balancing strategy across services")
|
||||
parser.add_argument("--health-check-interval", type=float, default=5.0,
|
||||
help="Seconds between health checks (default: 5)")
|
||||
parser.add_argument("--health-max-failures", type=int, default=2,
|
||||
help="Consecutive failures before marking a port unhealthy (default: 2)")
|
||||
parser.add_argument("--health-recovery-interval", type=float, default=10.0,
|
||||
help="Seconds before retrying an unhealthy port (default: 10)")
|
||||
parser.add_argument("--request-max-retries", type=int, default=2,
|
||||
help="Max retries per request on failure (default: 2)")
|
||||
parser.add_argument("--output-file", required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Determine ports: --ports takes precedence, else fall back to --port.
|
||||
if args.ports:
|
||||
ports = [int(p.strip()) for p in args.ports.split(",")]
|
||||
else:
|
||||
ports = [args.port]
|
||||
|
||||
health_checker = HealthChecker(
|
||||
host=args.host,
|
||||
ports=ports,
|
||||
check_interval=args.health_check_interval,
|
||||
max_failures=args.health_max_failures,
|
||||
recovery_interval=args.health_recovery_interval,
|
||||
)
|
||||
lb = LoadBalancer(ports, health_checker=health_checker, strategy=args.lb_strategy)
|
||||
|
||||
summary = asyncio.run(run_benchmark(
|
||||
host=args.host,
|
||||
ports=ports,
|
||||
lb=lb,
|
||||
health_checker=health_checker,
|
||||
model=args.model,
|
||||
concurrency=args.concurrency,
|
||||
input_len=args.input_len,
|
||||
output_len=args.output_len,
|
||||
num_prompts=args.num_prompts,
|
||||
temperature=args.temperature,
|
||||
output_file=args.output_file,
|
||||
))
|
||||
|
||||
print(json.dumps(summary, indent=2, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
180
experiments/dsv4_h200_vllm_tp2_custom_bench/config.env
Normal file
180
experiments/dsv4_h200_vllm_tp2_custom_bench/config.env
Normal file
@ -0,0 +1,180 @@
|
||||
# Experiment configuration for dsv4_h200_vllm_tp2_custom_bench
|
||||
# Custom benchmark client for multi-service vLLM TP=2 on 8x H200.
|
||||
#
|
||||
# This experiment runs 4 independent vLLM services (TP=2 each) on GPU pairs:
|
||||
# Service 1: port 30005, GPUs 0,1
|
||||
# Service 2: port 30006, GPUs 2,3
|
||||
# Service 3: port 30007, GPUs 4,5
|
||||
# Service 4: port 30008, GPUs 6,7
|
||||
#
|
||||
# bench_client.py distributes requests across all 4 services via round-robin
|
||||
# load balancing, giving accurate cluster-wide throughput even at low
|
||||
# per-service concurrency.
|
||||
|
||||
EXPERIMENT="dsv4_h200_vllm_tp2_custom_bench"
|
||||
MODEL_NAME="DeepSeek-V4-Flash"
|
||||
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
|
||||
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
# Primary port (backward compatible); multi-service ports are in PORTS.
|
||||
PORT="30005"
|
||||
PORTS="30005,30006,30007,30008"
|
||||
|
||||
BACKEND="vllm"
|
||||
ENGINE="vllm"
|
||||
|
||||
# Native virtual environments on the host.
|
||||
VENV_SERVER="/data/user1/yy/envs/vllm"
|
||||
VENV_CLIENT="/data/user1/yy/envs/vllm"
|
||||
|
||||
# Load balancing strategy: round_robin | random | least_conn
|
||||
LB_STRATEGY="round_robin"
|
||||
|
||||
# Benchmark scenarios: "concurrency input_len output_len num_prompts".
|
||||
# For multi-service we include low-concurrency scenarios to show how
|
||||
# requests spread across 4 services.
|
||||
SCENARIOS=(
|
||||
"1 512 256 20"
|
||||
"2 512 256 40"
|
||||
"4 512 256 80"
|
||||
"8 512 256 160"
|
||||
"16 512 256 320"
|
||||
"32 512 256 640"
|
||||
"64 512 256 1280"
|
||||
"128 512 256 2560"
|
||||
|
||||
"1 1024 128 20"
|
||||
"1 1024 256 20"
|
||||
"1 1024 512 20"
|
||||
"1 1024 1024 20"
|
||||
"1 1024 2048 20"
|
||||
"1 1024 4096 20"
|
||||
|
||||
"8 1024 128 160"
|
||||
"8 1024 256 160"
|
||||
"8 1024 512 160"
|
||||
"8 1024 1024 160"
|
||||
"8 1024 2048 160"
|
||||
"8 1024 4096 160"
|
||||
|
||||
"32 1024 128 640"
|
||||
"32 1024 256 640"
|
||||
"32 1024 512 640"
|
||||
"32 1024 1024 640"
|
||||
"32 1024 2048 640"
|
||||
"32 1024 4096 640"
|
||||
|
||||
"128 1024 128 2560"
|
||||
"128 1024 256 2560"
|
||||
"128 1024 512 2560"
|
||||
"128 1024 1024 2560"
|
||||
"128 1024 2048 2560"
|
||||
"128 1024 4096 2560"
|
||||
|
||||
"1 2048 128 20"
|
||||
"1 2048 256 20"
|
||||
"1 2048 512 20"
|
||||
"1 2048 1024 20"
|
||||
"1 2048 2048 20"
|
||||
"1 2048 4096 20"
|
||||
|
||||
"8 2048 128 160"
|
||||
"8 2048 256 160"
|
||||
"8 2048 512 160"
|
||||
"8 2048 1024 160"
|
||||
"8 2048 2048 160"
|
||||
"8 2048 4096 160"
|
||||
|
||||
"32 2048 128 640"
|
||||
"32 2048 256 640"
|
||||
"32 2048 512 640"
|
||||
"32 2048 1024 640"
|
||||
"32 2048 2048 640"
|
||||
"32 2048 4096 640"
|
||||
|
||||
"128 2048 128 2560"
|
||||
"128 2048 256 2560"
|
||||
"128 2048 512 2560"
|
||||
"128 2048 1024 2560"
|
||||
"128 2048 2048 2560"
|
||||
"128 2048 4096 2560"
|
||||
|
||||
"1 4096 128 20"
|
||||
"1 4096 256 20"
|
||||
"1 4096 512 20"
|
||||
"1 4096 1024 20"
|
||||
"1 4096 2048 20"
|
||||
"1 4096 4096 20"
|
||||
|
||||
"8 4096 128 160"
|
||||
"8 4096 256 160"
|
||||
"8 4096 512 160"
|
||||
"8 4096 1024 160"
|
||||
"8 4096 2048 160"
|
||||
"8 4096 4096 160"
|
||||
|
||||
"32 4096 128 640"
|
||||
"32 4096 256 640"
|
||||
"32 4096 512 640"
|
||||
"32 4096 1024 640"
|
||||
"32 4096 2048 640"
|
||||
"32 4096 4096 640"
|
||||
|
||||
"128 4096 128 2560"
|
||||
"128 4096 256 2560"
|
||||
"128 4096 512 2560"
|
||||
"128 4096 1024 2560"
|
||||
"128 4096 2048 2560"
|
||||
"128 4096 4096 2560"
|
||||
|
||||
"1 16384 128 20"
|
||||
"1 16384 256 20"
|
||||
"1 16384 512 20"
|
||||
"1 16384 1024 20"
|
||||
"1 16384 2048 20"
|
||||
|
||||
"8 16384 128 160"
|
||||
"8 16384 256 160"
|
||||
"8 16384 512 160"
|
||||
"8 16384 1024 160"
|
||||
"8 16384 2048 160"
|
||||
|
||||
"32 16384 128 640"
|
||||
"32 16384 256 640"
|
||||
"32 16384 512 640"
|
||||
"32 16384 1024 640"
|
||||
"32 16384 2048 640"
|
||||
|
||||
"128 16384 128 2560"
|
||||
"128 16384 256 2560"
|
||||
"128 16384 512 2560"
|
||||
"128 16384 1024 2560"
|
||||
"128 16384 2048 2560"
|
||||
|
||||
|
||||
"1 65536 128 20"
|
||||
"1 65536 256 20"
|
||||
"1 65536 512 20"
|
||||
"1 65536 1024 20"
|
||||
|
||||
"8 65536 128 160"
|
||||
"8 65536 256 160"
|
||||
"8 65536 512 160"
|
||||
"8 65536 1024 160"
|
||||
"32 65536 1024 160"
|
||||
|
||||
"1 131072 128 20"
|
||||
"1 131072 256 20"
|
||||
"1 131072 512 20"
|
||||
|
||||
"8 131072 512 160"
|
||||
|
||||
"1 262144 256 20"
|
||||
|
||||
"8 262144 128 160"
|
||||
|
||||
"1 524288 128 20"
|
||||
)
|
||||
|
||||
# Server start script bundled with this experiment.
|
||||
SERVER_START_SCRIPT="${SCRIPT_DIR}/start_server.sh"
|
||||
206
experiments/dsv4_h200_vllm_tp2_custom_bench/parse_results.py
Executable file
206
experiments/dsv4_h200_vllm_tp2_custom_bench/parse_results.py
Executable file
@ -0,0 +1,206 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Parse custom bench_client.py JSONL outputs for vLLM TP=2 benchmark.
|
||||
|
||||
Reads JSONL files produced by bench_client.py (one request per line,
|
||||
last line is the summary) and generates:
|
||||
- results.json (appended scenarios, following the standard schema)
|
||||
- report.md (human-readable summary)
|
||||
|
||||
Usage:
|
||||
python3 parse_results.py <result_root>
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_jsonl(path: Path) -> tuple[dict | None, list[dict]]:
|
||||
"""Read the summary JSON (last line) and all per-request lines."""
|
||||
requests: list[dict] = []
|
||||
summary: dict | None = None
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
# The summary line has "completed" and "duration" keys.
|
||||
if "completed" in obj and "duration" in obj:
|
||||
summary = obj
|
||||
else:
|
||||
requests.append(obj)
|
||||
return summary, requests
|
||||
|
||||
|
||||
def compute_metrics(summary: dict, requests: list[dict]) -> dict:
|
||||
completed = summary.get("completed", 0)
|
||||
total = len(requests)
|
||||
failed = total - completed if total > 0 else 0
|
||||
duration_s = summary.get("duration", 0.0)
|
||||
|
||||
# Extract per-request metrics for percentiles.
|
||||
ttfts = [r["first_token_time_ms"] for r in requests if r.get("success")]
|
||||
tpots = [r["tpot_ms"] for r in requests if r.get("success") and r.get("output_tokens", 0) > 1]
|
||||
e2es = [r["e2e_latency_ms"] for r in requests if r.get("success")]
|
||||
itls = [r["mean_itl_ms"] for r in requests if r.get("success") and r.get("inter_token_latencies")]
|
||||
|
||||
def percentile(values: list[float], p: float) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
s = sorted(values)
|
||||
k = (len(s) - 1) * p / 100.0
|
||||
f = int(k)
|
||||
c = min(f + 1, len(s) - 1)
|
||||
return s[f] + (k - f) * (s[c] - s[f])
|
||||
|
||||
return {
|
||||
"success": completed,
|
||||
"failed": failed,
|
||||
"duration_s": duration_s,
|
||||
"request_throughput": summary.get("request_throughput", 0.0),
|
||||
"input_token_throughput": summary.get("input_throughput", 0.0),
|
||||
"output_token_throughput": summary.get("output_throughput", 0.0),
|
||||
"total_token_throughput": summary.get("total_throughput", 0.0),
|
||||
"total_input_tokens": summary.get("total_input_tokens", 0),
|
||||
"total_output_tokens": summary.get("total_output_tokens", 0),
|
||||
"e2e_ms": {
|
||||
"mean": summary.get("mean_e2e_latency_ms", 0.0),
|
||||
"p50": percentile(e2es, 50),
|
||||
"p90": percentile(e2es, 90),
|
||||
"p95": percentile(e2es, 95),
|
||||
"p99": percentile(e2es, 99),
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": summary.get("mean_ttft_ms", 0.0),
|
||||
"p50": percentile(ttfts, 50),
|
||||
"p90": percentile(ttfts, 90),
|
||||
"p95": percentile(ttfts, 95),
|
||||
"p99": percentile(ttfts, 99),
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": summary.get("mean_tpot_ms", 0.0),
|
||||
"p50": percentile(tpots, 50),
|
||||
"p90": percentile(tpots, 90),
|
||||
"p95": percentile(tpots, 95),
|
||||
"p99": percentile(tpots, 99),
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": summary.get("mean_itl_ms", 0.0),
|
||||
"p50": percentile(itls, 50),
|
||||
"p90": percentile(itls, 90),
|
||||
"p95": percentile(itls, 95),
|
||||
"p99": percentile(itls, 99),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def scenario_name(concurrency: int, input_len: int, output_len: int) -> str:
|
||||
return f"c{concurrency}_i{input_len}_o{output_len}"
|
||||
|
||||
|
||||
def slo_status(metrics: dict, ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> dict:
|
||||
ttft_ok = metrics["ttft_ms"]["p95"] < ttft_limit_ms
|
||||
tpot_ok = metrics["tpot_ms"]["mean"] < tpot_limit_ms
|
||||
if ttft_ok and tpot_ok:
|
||||
mark = "✅"
|
||||
elif ttft_ok or tpot_ok:
|
||||
mark = "⚠️"
|
||||
else:
|
||||
mark = "❌"
|
||||
return {
|
||||
"ttft_p95_ok": ttft_ok,
|
||||
"tpot_mean_ok": tpot_ok,
|
||||
"overall": mark,
|
||||
}
|
||||
|
||||
|
||||
def append_scenario(results_json: Path, scenario: dict) -> None:
|
||||
with open(results_json, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
data["scenarios"].append(scenario)
|
||||
with open(results_json, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
def generate_report(result_root: Path, scenarios: list[dict]) -> None:
|
||||
report_path = result_root / "report.md"
|
||||
with open(report_path, "w", encoding="utf-8") as f:
|
||||
f.write("# H200 vLLM TP=2 Custom Benchmark Report\n\n")
|
||||
f.write("- **Client**: `bench_client.py` (async OpenAI API, per-request timing)\n")
|
||||
f.write("- **Backend**: vLLM (TP=2, FP8 KV cache, no speculative decoding)\n")
|
||||
f.write(f"- **Result root**: `{result_root}`\n\n")
|
||||
|
||||
f.write("## Results\n\n")
|
||||
f.write("| Scenario | Concurrency | Input | Output | Duration(s) | Success | Failed | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |\n")
|
||||
f.write("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n")
|
||||
|
||||
for s in scenarios:
|
||||
cfg = s["config"]
|
||||
m = s["metrics"]
|
||||
slo = s.get("slo_status", {}).get("overall", "")
|
||||
f.write(
|
||||
f"| {s['name']} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | "
|
||||
f"{m['duration_s']:.2f} | {m['success']} | {m['failed']} | {m['request_throughput']:.2f} | "
|
||||
f"{m['input_token_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||
f"{m['total_token_throughput']:.2f} | "
|
||||
f"{m['ttft_ms']['mean']:.2f} | {m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||
f"{m['e2e_ms']['mean']:.2f} | {m['e2e_ms']['p95']:.2f} | {m['e2e_ms']['p99']:.2f} | {slo} |\n"
|
||||
)
|
||||
f.write("\n")
|
||||
f.write("SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.\n\n")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("results")
|
||||
raw_dir = result_root / "raw_outputs"
|
||||
results_json = result_root / "results.json"
|
||||
|
||||
if not raw_dir.exists():
|
||||
raise SystemExit(f"raw_outputs directory not found: {raw_dir}")
|
||||
|
||||
scenarios = []
|
||||
for jsonl_path in sorted(raw_dir.glob("vllm_*.jsonl")):
|
||||
parts = jsonl_path.stem.split("_")
|
||||
if len(parts) < 5:
|
||||
continue
|
||||
concurrency, input_len, output_len = int(parts[2]), int(parts[3]), int(parts[4])
|
||||
|
||||
summary, requests = parse_jsonl(jsonl_path)
|
||||
if summary is None:
|
||||
continue
|
||||
|
||||
metrics = compute_metrics(summary, requests)
|
||||
scenario = {
|
||||
"name": scenario_name(concurrency, input_len, output_len),
|
||||
"config": {
|
||||
"concurrency": concurrency,
|
||||
"input_len": input_len,
|
||||
"output_len": output_len,
|
||||
"dataset": "random",
|
||||
"num_prompts": metrics["success"] + metrics["failed"],
|
||||
},
|
||||
"metrics": metrics,
|
||||
"slo_status": slo_status(metrics),
|
||||
"raw_file": str(jsonl_path),
|
||||
}
|
||||
scenarios.append(scenario)
|
||||
|
||||
if not scenarios:
|
||||
print("No benchmark outputs found to parse")
|
||||
return
|
||||
|
||||
if results_json.exists():
|
||||
for s in scenarios:
|
||||
append_scenario(results_json, s)
|
||||
|
||||
generate_report(result_root, scenarios)
|
||||
print(f"Parsed {len(scenarios)} scenarios into {result_root}/report.md")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -0,0 +1,47 @@
|
||||
{
|
||||
"metadata": {
|
||||
"experiment": "dsv4_h200_vllm_tp2_custom_bench",
|
||||
"run_id": "20260710-104930",
|
||||
"timestamp": "2026-07-10T10:49:30+00:00",
|
||||
"model": "/data/models/DeepSeek-V4-Flash",
|
||||
"backend": "vllm",
|
||||
"engine": "vllm",
|
||||
"hardware": "8x NVIDIA H200 143GB",
|
||||
"accelerator": "NVIDIA H200",
|
||||
"chip": "nvidia_h200",
|
||||
"script": "experiments/dsv4_h200_vllm_tp2_custom_bench/run_bench.sh",
|
||||
"env": "/data/user1/yy/envs/vllm",
|
||||
"git_commit": "def1355",
|
||||
"git_dirty": "dirty",
|
||||
"description": "H200 4x vLLM TP=2 multi-service benchmark using bench_client.py"
|
||||
},
|
||||
"config": {
|
||||
"tp": 2,
|
||||
"num_services": 4,
|
||||
"ports": "30005,30006,30007,30008",
|
||||
"gpu_pairs": "0,1|2,3|4,5|6,7",
|
||||
"kv_cache_dtype": "fp8",
|
||||
"block_size": 256,
|
||||
"max_num_seqs": 256,
|
||||
"client": "bench_client.py",
|
||||
"lb_strategy": "round_robin",
|
||||
"scenarios": [
|
||||
"1 512 256 20",
|
||||
"4 512 256 80",
|
||||
"8 512 256 160",
|
||||
"16 512 256 320",
|
||||
"32 512 256 640",
|
||||
"64 512 256 1280",
|
||||
"128 512 256 2560",
|
||||
"1 2048 512 20",
|
||||
"8 2048 512 160",
|
||||
"32 2048 512 640",
|
||||
"128 2048 512 2560",
|
||||
"1 4096 1024 20",
|
||||
"8 4096 1024 160",
|
||||
"32 4096 1024 640",
|
||||
"128 4096 1024 2560"
|
||||
]
|
||||
},
|
||||
"scenarios": []
|
||||
}
|
||||
277
experiments/dsv4_h200_vllm_tp2_custom_bench/run_bench.sh
Executable file
277
experiments/dsv4_h200_vllm_tp2_custom_bench/run_bench.sh
Executable file
@ -0,0 +1,277 @@
|
||||
#!/usr/bin/env bash
|
||||
# Custom benchmark orchestrator for multi-service vLLM TP=2 on 8x H200.
|
||||
#
|
||||
# This runs 4 independent vLLM services (TP=2 each) on GPU pairs (0,1),(2,3),(4,5),(6,7).
|
||||
# bench_client.py distributes requests across all 4 services via round-robin LB,
|
||||
# giving accurate cluster-wide throughput even at low per-service concurrency.
|
||||
#
|
||||
# Usage:
|
||||
# bash run_bench.sh
|
||||
# SKIP_MANAGE_SERVER=1 bash run_bench.sh # skip server start/stop
|
||||
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||
RESULT_ROOT="${RESULT_ROOT:-${SCRIPT_DIR}/results/${RUN_ID}}"
|
||||
RAW_DIR="${RESULT_ROOT}/raw_outputs"
|
||||
LOG_DIR="${RESULT_ROOT}/logs"
|
||||
|
||||
ensure_result_root "$RESULT_ROOT"
|
||||
log_init "${LOG_DIR}/orchestrator.log"
|
||||
|
||||
log "experiment=${EXPERIMENT_NAME}"
|
||||
log "run_id=${RUN_ID}"
|
||||
log "result_root=${RESULT_ROOT}"
|
||||
log "platform=${PLATFORM}"
|
||||
log "chip=${CHIP}"
|
||||
log "accelerator=${ACCELERATOR}"
|
||||
log "engine=${ENGINE}"
|
||||
log "hardware=${HARDWARE}"
|
||||
log "model=${MODEL_PATH}"
|
||||
log "services=${PORTS}"
|
||||
log "lb_strategy=${LB_STRATEGY}"
|
||||
log "server_start_script=${SERVER_START_SCRIPT}"
|
||||
|
||||
# Write initial metadata for this run.
|
||||
METADATA_JSON="${RESULT_ROOT}/results.json"
|
||||
write_metadata_json \
|
||||
"$METADATA_JSON" \
|
||||
"$EXPERIMENT_NAME" \
|
||||
"$RUN_ID" \
|
||||
"$MODEL_PATH" \
|
||||
"$BACKEND" \
|
||||
"$ENGINE" \
|
||||
"$HARDWARE" \
|
||||
"$ACCELERATOR" \
|
||||
"$CHIP" \
|
||||
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||
"$VENV_SERVER" \
|
||||
"H200 4x vLLM TP=2 multi-service benchmark using bench_client.py"
|
||||
|
||||
# Update metadata with config.
|
||||
"${VENV_CLIENT}/bin/python" - "$METADATA_JSON" <<'PY'
|
||||
import json, sys
|
||||
path = sys.argv[1]
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
data["config"] = {
|
||||
"tp": 2,
|
||||
"num_services": 4,
|
||||
"ports": "30005,30006,30007,30008",
|
||||
"gpu_pairs": "0,1|2,3|4,5|6,7",
|
||||
"kv_cache_dtype": "fp8",
|
||||
"block_size": 256,
|
||||
"max_num_seqs": 256,
|
||||
"client": "bench_client.py",
|
||||
"lb_strategy": "round_robin",
|
||||
"scenarios": [
|
||||
"1 512 256 20", "4 512 256 80", "8 512 256 160",
|
||||
"16 512 256 320", "32 512 256 640", "64 512 256 1280", "128 512 256 2560",
|
||||
"1 2048 512 20", "8 2048 512 160", "32 2048 512 640", "128 2048 512 2560",
|
||||
"1 4096 1024 20", "8 4096 1024 160", "32 4096 1024 640", "128 4096 1024 2560"
|
||||
]
|
||||
}
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
PY
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server helpers (multi-service)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
is_server_healthy() {
|
||||
local port="$1"
|
||||
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${port}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
are_all_services_healthy() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if ! is_server_healthy "$p"; then
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
stop_all_servers() {
|
||||
log "stopping all vllm services"
|
||||
# Kill by PID files first.
|
||||
for pid_file in "${SCRIPT_DIR}"/server_port*.pid; do
|
||||
[[ -f "$pid_file" ]] || continue
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "killing pid=${pid}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 2
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
done
|
||||
# Fallback: kill any remaining vllm processes for this model.
|
||||
pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
sleep 2
|
||||
}
|
||||
|
||||
start_all_servers() {
|
||||
log "starting all vllm services with ${SERVER_START_SCRIPT}"
|
||||
if [[ ! -x "${SERVER_START_SCRIPT}" ]]; then
|
||||
log "error: start script not found or not executable: ${SERVER_START_SCRIPT}"
|
||||
exit 1
|
||||
fi
|
||||
bash "${SERVER_START_SCRIPT}" >> "${LOG_DIR}/server.outer.log" 2>&1
|
||||
|
||||
if ! are_all_services_healthy "$PORTS"; then
|
||||
log "error: not all vllm services became healthy"
|
||||
exit 1
|
||||
fi
|
||||
log "all vllm services are healthy"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pre-benchmark health check with detailed reporting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
check_services_detailed() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
local healthy_count=0
|
||||
local unhealthy_ports=""
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if is_server_healthy "$p"; then
|
||||
((healthy_count++))
|
||||
else
|
||||
unhealthy_ports="${unhealthy_ports}${p} "
|
||||
fi
|
||||
done
|
||||
echo "$healthy_count"
|
||||
if [[ -n "$unhealthy_ports" ]]; then
|
||||
echo "unhealthy: ${unhealthy_ports}" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
wait_for_all_services() {
|
||||
local ports="$1"
|
||||
local max_wait="${2:-240}"
|
||||
local interval="${3:-5}"
|
||||
local elapsed=0
|
||||
while (( elapsed < max_wait )); do
|
||||
local healthy_count
|
||||
healthy_count="$(check_services_detailed "$ports" | head -1)"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
if (( healthy_count == ${#PORT_LIST[@]} )); then
|
||||
return 0
|
||||
fi
|
||||
log "waiting for services... ${healthy_count}/${#PORT_LIST[@]} healthy after ${elapsed}s"
|
||||
sleep "$interval"
|
||||
((elapsed += interval))
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mid-benchmark health check (called between scenarios)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
check_and_report_services() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
local healthy=()
|
||||
local unhealthy=()
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if is_server_healthy "$p"; then
|
||||
healthy+=("$p")
|
||||
else
|
||||
unhealthy+=("$p")
|
||||
fi
|
||||
done
|
||||
log "service health check: healthy=[${healthy[*]}] unhealthy=[${unhealthy[*]}]"
|
||||
if [[ ${#unhealthy[@]} -gt 0 ]]; then
|
||||
log "WARNING: ${#unhealthy[@]} service(s) unhealthy: ${unhealthy[*]}"
|
||||
fi
|
||||
return ${#unhealthy[@]}
|
||||
}
|
||||
|
||||
on_exit() {
|
||||
local code=$?
|
||||
log "orchestrator exiting with code=${code}"
|
||||
if [[ -z "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||
stop_all_servers
|
||||
fi
|
||||
exit "$code"
|
||||
}
|
||||
trap on_exit EXIT
|
||||
|
||||
if [[ -n "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||
log "SKIP_MANAGE_SERVER is set, assuming services are already running"
|
||||
if ! wait_for_all_services "$PORTS"; then
|
||||
log "error: not all healthy services found on ports ${PORTS}"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
stop_all_servers
|
||||
start_all_servers
|
||||
fi
|
||||
|
||||
log "===== BENCHMARK START ====="
|
||||
|
||||
for scenario in "${SCENARIOS[@]}"; do
|
||||
read -r concurrency input_len output_len num_prompts <<< "$scenario"
|
||||
# Fallback for legacy 3-field scenarios.
|
||||
if [[ -z "${num_prompts:-}" ]]; then
|
||||
num_prompts="$NUM_PROMPTS"
|
||||
fi
|
||||
|
||||
output_file="${RAW_DIR}/vllm_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||
detail_log="${LOG_DIR}/vllm_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||
|
||||
log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len} num_prompts=${num_prompts}"
|
||||
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/bench_client.py" \
|
||||
--host 127.0.0.1 \
|
||||
--ports "$PORTS" \
|
||||
--model "$SERVED_MODEL_NAME" \
|
||||
--concurrency "$concurrency" \
|
||||
--input-len "$input_len" \
|
||||
--output-len "$output_len" \
|
||||
--num-prompts "$num_prompts" \
|
||||
--lb-strategy "$LB_STRATEGY" \
|
||||
--health-check-interval 5 \
|
||||
--health-max-failures 2 \
|
||||
--health-recovery-interval 10 \
|
||||
--request-max-retries 2 \
|
||||
--output-file "$output_file" \
|
||||
> "$detail_log" 2>&1 || {
|
||||
log "ERROR: scenario c=${concurrency} i=${input_len} o=${output_len} failed; see ${detail_log}"
|
||||
# After failure, check service health before continuing.
|
||||
check_and_report_services "$PORTS"
|
||||
continue
|
||||
}
|
||||
|
||||
log "finished scenario: output=${output_file}"
|
||||
|
||||
# Between scenarios, report service health so we can spot degradation early.
|
||||
check_and_report_services "$PORTS" || true
|
||||
done
|
||||
|
||||
log "===== BENCHMARK DONE ====="
|
||||
|
||||
log "parsing results"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/parse_results.py" "$RESULT_ROOT" >> "${LOG_DIR}/parse.log" 2>&1 || {
|
||||
log "WARNING: parser failed; see ${LOG_DIR}/parse.log"
|
||||
}
|
||||
|
||||
log "all results saved to ${RESULT_ROOT}"
|
||||
110
experiments/dsv4_h200_vllm_tp2_custom_bench/start_server.sh
Executable file
110
experiments/dsv4_h200_vllm_tp2_custom_bench/start_server.sh
Executable file
@ -0,0 +1,110 @@
|
||||
#!/bin/bash
|
||||
# Start 4 independent vLLM TP=2 services on 8x H200.
|
||||
# Each service runs on a pair of GPUs: (0,1), (2,3), (4,5), (6,7).
|
||||
# This maximizes GPU utilization for low-concurrency workloads.
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
VENV="${VENV_SERVER}"
|
||||
export PATH="$VENV/bin:$PATH"
|
||||
VLLM="$VENV/bin/vllm"
|
||||
|
||||
TP=2
|
||||
LOG_DIR="${SCRIPT_DIR}/logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
# 4 services, each on 2 GPUs.
|
||||
# Port and GPU pair mapping.
|
||||
SERVICES=(
|
||||
"30005:0,1"
|
||||
"30006:2,3"
|
||||
"30007:4,5"
|
||||
"30008:6,7"
|
||||
)
|
||||
|
||||
# Kill any existing vLLM processes for this model.
|
||||
pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
sleep 2
|
||||
|
||||
# Remove old PID files.
|
||||
rm -f "${SCRIPT_DIR}"/*.pid
|
||||
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
GPUS="${svc##*:}"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_$(date +%Y%m%d_%H%M%S).log"
|
||||
PID_FILE="${SCRIPT_DIR}/server_port${PORT}.pid"
|
||||
|
||||
echo "=== Starting service on port ${PORT}, GPUs ${GPUS} ==="
|
||||
echo "Log: ${LOG}"
|
||||
|
||||
CUDA_VISIBLE_DEVICES="${GPUS}" \
|
||||
nohup "$VLLM" serve "$MODEL_PATH" \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size "$TP" \
|
||||
--kv-cache-dtype fp8 \
|
||||
--block-size 256 \
|
||||
--served-model-name "$SERVED_MODEL_NAME" \
|
||||
--port "$PORT" \
|
||||
> "$LOG" 2>&1 &
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Waiting for all 4 services to become healthy..."
|
||||
|
||||
MAX_WAIT=240
|
||||
all_healthy=0
|
||||
for i in $(seq 1 $MAX_WAIT); do
|
||||
all_healthy=1
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
if ! curl -s "http://127.0.0.1:${PORT}/health" > /dev/null 2>&1; then
|
||||
all_healthy=0
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ "$all_healthy" -eq 1 ]]; then
|
||||
echo "All 4 services are healthy!"
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
echo " - http://127.0.0.1:${PORT}"
|
||||
done
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check if any process died early.
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
PID_FILE="${SCRIPT_DIR}/server_port${PORT}.pid"
|
||||
if [[ -f "$PID_FILE" ]]; then
|
||||
PID="$(cat "$PID_FILE")"
|
||||
if ! kill -0 "$PID" 2>/dev/null; then
|
||||
echo "ERROR: Service on port ${PORT} (PID ${PID}) exited early"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_*.log"
|
||||
tail -200 $LOG 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
if (( i % 10 == 0 )); then
|
||||
echo "Waiting... (${i}/${MAX_WAIT})"
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: Not all services became healthy after ${MAX_WAIT} retries"
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_*.log"
|
||||
echo "--- Last 50 lines of port ${PORT} ---"
|
||||
tail -50 $LOG 2>/dev/null || true
|
||||
done
|
||||
exit 1
|
||||
124
experiments/dsv4_h200_vllm_tp4_custom_bench/README.md
Normal file
124
experiments/dsv4_h200_vllm_tp4_custom_bench/README.md
Normal file
@ -0,0 +1,124 @@
|
||||
# dsv4_h200_vllm_tp4_custom_bench
|
||||
|
||||
## 目的
|
||||
|
||||
当 vLLM 使用 **TP=4**（Tensor Parallel = 4）部署时，单张 H200 上可以同时运行 **2 个独立服务**（8 卡 / 4 = 2 服务），每个服务独占 4 张 GPU：
|
||||
|
||||
| 服务 | 端口 | GPU 组 |
|
||||
|------|------|--------|
|
||||
| 1 | 30005 | 0, 1, 2, 3 |
|
||||
| 2 | 30006 | 4, 5, 6, 7 |
|
||||
|
||||
相比 TP=2 方案，TP=4 的每个服务拥有更大的 tensor-parallel 宽度，适合更高并发的单服务场景。本实验同样使用 **自定义压测客户端 `bench_client.py`**，通过负载均衡将请求均匀分发到 2 个服务上。
|
||||
|
||||
## 与 TP=2 方案的区别
|
||||
|
||||
| 特性 | TP=2 (4 服务) | TP=4 (2 服务) |
|
||||
|------|---------------|---------------|
|
||||
| 服务数量 | 4 | 2 |
|
||||
| 每服务 GPU | 2 | 4 |
|
||||
| 单服务吞吐 | 较低 | 较高 |
|
||||
| 集群总吞吐 | 相近 | 相近 |
|
||||
| 适用场景 | 低并发、低延迟 | 高并发、大 batch |
|
||||
|
||||
## 文件说明
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `bench_client.py` | 自定义异步压测客户端，支持多服务负载均衡（复用 TP=2 版本） |
|
||||
| `config.env` | 实验配置（2 服务、端口 30005/30006、场景等） |
|
||||
| `run_bench.sh` | 编排脚本：启动 2 个服务 → 运行压测 → 解析结果 |
|
||||
| `start_server.sh` | 同时启动 2 个 vLLM TP=4 服务（每服务 4 GPU） |
|
||||
| `parse_results.py` | 解析 bench_client.py 输出的 JSONL，生成 `results.json` + `report.md` |
|
||||
|
||||
## 快速运行
|
||||
|
||||
```bash
|
||||
# 完整流程（自动启动 2 个服务、压测、生成报告）
|
||||
bash experiments/dsv4_h200_vllm_tp4_custom_bench/run_bench.sh
|
||||
|
||||
# 跳过服务管理（已有 2 个服务在运行）
|
||||
SKIP_MANAGE_SERVER=1 bash experiments/dsv4_h200_vllm_tp4_custom_bench/run_bench.sh
|
||||
|
||||
# 单独运行压测客户端（多服务，带健康检查）
|
||||
python3 experiments/dsv4_h200_vllm_tp4_custom_bench/bench_client.py \
|
||||
--host 127.0.0.1 \
|
||||
--ports 30005,30006 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 32 --input-len 512 --output-len 256 --num-prompts 640 \
|
||||
--lb-strategy round_robin \
|
||||
--health-check-interval 5 \
|
||||
--health-max-failures 2 \
|
||||
--health-recovery-interval 10 \
|
||||
--request-max-retries 2 \
|
||||
--output-file /tmp/test.jsonl
|
||||
|
||||
# 单独运行压测客户端（单服务，向后兼容）
|
||||
python3 experiments/dsv4_h200_vllm_tp4_custom_bench/bench_client.py \
|
||||
--host 127.0.0.1 --port 30005 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 8 --input-len 512 --output-len 256 --num-prompts 160 \
|
||||
--output-file /tmp/test_single.jsonl
|
||||
|
||||
# 解析已有结果
|
||||
python3 experiments/dsv4_h200_vllm_tp4_custom_bench/parse_results.py \
|
||||
experiments/dsv4_h200_vllm_tp4_custom_bench/results/<RUN_ID>
|
||||
```
|
||||
|
||||
## 场景设计
|
||||
|
||||
本实验覆盖了从 **单并发** 到 **128 并发** 的梯度，以及不同输入/输出长度组合：
|
||||
|
||||
| 并发 | 输入长度 | 输出长度 | 请求数 | 说明 |
|
||||
|------|----------|----------|--------|------|
|
||||
| 1 | 512 | 256 | 20 | 单并发基线，测纯延迟 |
|
||||
| 2 | 512 | 256 | 40 | 低并发，2 服务各 1 req |
|
||||
| 4 | 512 | 256 | 80 | 中低并发 |
|
||||
| 8 | 512 | 256 | 160 | 中等并发 |
|
||||
| 16 | 512 | 256 | 320 | 中高并发 |
|
||||
| 32 | 512 | 256 | 640 | 高并发 |
|
||||
| 64 | 512 | 256 | 1280 | 很高并发 |
|
||||
| 128 | 512 | 256 | 2560 | 极限并发，测集群吞吐上限 |
|
||||
| 1 | 2048 | 512 | 20 | 长输入基线 |
|
||||
| 8 | 2048 | 512 | 160 | 长输入中并发 |
|
||||
| 32 | 2048 | 512 | 640 | 长输入高并发 |
|
||||
| 128 | 2048 | 512 | 2560 | 长输入极限并发 |
|
||||
| 1 | 4096 | 1024 | 20 | 超长输入基线 |
|
||||
| 8 | 4096 | 1024 | 160 | 超长输入中并发 |
|
||||
| 32 | 4096 | 1024 | 640 | 超长输入高并发 |
|
||||
| 128 | 4096 | 1024 | 2560 | 超长输入极限并发 |
|
||||
|
||||
## 健康检查与故障自动跳过
|
||||
|
||||
同 TP=2 版本，详见原 README。
|
||||
|
||||
## 负载均衡策略
|
||||
|
||||
同 TP=2 版本，支持 `round_robin`（默认）、`random`、`least_conn`。
|
||||
|
||||
## 输出格式
|
||||
|
||||
同 TP=2 版本，输出 JSONL 原始数据、`results.json` 结构化结果、`report.md` 人类可读报告。
|
||||
|
||||
## 单独启动/停止 2 个服务
|
||||
|
||||
```bash
|
||||
# 启动 2 个服务
|
||||
bash experiments/dsv4_h200_vllm_tp4_custom_bench/start_server.sh
|
||||
|
||||
# 停止（run_bench.sh 会自动停止，但也可以手动）
|
||||
for port in 30005 30006; do
|
||||
pid_file="experiments/dsv4_h200_vllm_tp4_custom_bench/server_port${port}.pid"
|
||||
[[ -f "$pid_file" ]] && kill "$(cat "$pid_file")" 2>/dev/null || true
|
||||
rm -f "$pid_file"
|
||||
done
|
||||
pkill -9 -f "vllm serve.*DeepSeek-V4-Flash" 2>/dev/null || true
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 2 个服务共享同一模型文件，确保 `/data/models/DeepSeek-V4-Flash` 存在且可访问。
|
||||
- 每个服务占用约 4 张 GPU 的 90% 显存，确保无其他进程占用 GPU。
|
||||
- TP=4 相比 TP=2 单服务延迟更低（通信更少），但服务数量减半，低并发下可能无法充分利用集群。
|
||||
- 健康检查依赖 `/health` 端点，确保 vLLM 服务已启用该端点（vLLM 默认启用）。
|
||||
- 如果某个服务频繁崩溃，检查 GPU 显存是否充足（OOM 是最常见原因）。
|
||||
652
experiments/dsv4_h200_vllm_tp4_custom_bench/bench_client.py
Executable file
652
experiments/dsv4_h200_vllm_tp4_custom_bench/bench_client.py
Executable file
@ -0,0 +1,652 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Custom benchmark client for vLLM / OpenAI-compatible API with multi-service load balancing and health-check failover.
|
||||
|
||||
Sends concurrent requests to multiple inference servers and records per-request
|
||||
latency metrics (TTFT, TPOT, ITL, E2E) with fine-grained timing.
|
||||
|
||||
Designed for TP=2 on 8-GPU setups where 4 independent services run on
|
||||
(0,1), (2,3), (4,5), (6,7). Requests are distributed across services via
|
||||
round-robin or random load balancing. Unhealthy services are automatically
|
||||
skipped with periodic retry.
|
||||
|
||||
Usage:
|
||||
# Single service (backward compatible)
|
||||
python3 bench_client.py \
|
||||
--host 127.0.0.1 --port 30005 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 64 --input-len 512 --output-len 256 --num-prompts 320 \
|
||||
--output-file results.jsonl
|
||||
|
||||
# Multi-service (4 services on ports 30005-30008)
|
||||
python3 bench_client.py \
|
||||
--host 127.0.0.1 \
|
||||
--ports 30005,30006,30007,30008 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 64 --input-len 512 --output-len 256 --num-prompts 320 \
|
||||
--lb-strategy round_robin \
|
||||
--output-file results.jsonl
|
||||
|
||||
The output JSONL contains one object per request with raw timing data.
|
||||
A final summary line ("completed" field) aggregates all requests.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Any
|
||||
|
||||
# Try aiohttp first; fall back to urllib for stdlib-only environments.
|
||||
try:
|
||||
import aiohttp
|
||||
HAS_AIOHTTP = True
|
||||
except ImportError:
|
||||
HAS_AIOHTTP = False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data structures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class RequestResult:
|
||||
request_id: int
|
||||
input_tokens: int = 0
|
||||
output_tokens: int = 0
|
||||
first_token_time_ms: float = 0.0 # TTFT
|
||||
e2e_latency_ms: float = 0.0 # total wall time
|
||||
inter_token_latencies: list[float] = field(default_factory=list)
|
||||
success: bool = False
|
||||
error: str = ""
|
||||
target_port: int = 0 # which service handled this request
|
||||
retry_count: int = 0 # how many times this request was retried
|
||||
|
||||
@property
|
||||
def tpot_ms(self) -> float:
|
||||
"""Mean TPOT = (e2e - ttft) / (output_tokens - 1) if >1 token."""
|
||||
if self.output_tokens <= 1:
|
||||
return 0.0
|
||||
return (self.e2e_latency_ms - self.first_token_time_ms) / (self.output_tokens - 1)
|
||||
|
||||
@property
|
||||
def mean_itl_ms(self) -> float:
|
||||
if not self.inter_token_latencies:
|
||||
return 0.0
|
||||
return sum(self.inter_token_latencies) / len(self.inter_token_latencies)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
d = asdict(self)
|
||||
d["tpot_ms"] = self.tpot_ms
|
||||
d["mean_itl_ms"] = self.mean_itl_ms
|
||||
return d
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Health checker
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class HealthChecker:
|
||||
"""Periodically check service health and maintain a healthy-port list."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
host: str,
|
||||
ports: list[int],
|
||||
check_interval: float = 5.0,
|
||||
max_failures: int = 2,
|
||||
recovery_interval: float = 10.0,
|
||||
):
|
||||
self.host = host
|
||||
self.all_ports = ports
|
||||
self.check_interval = check_interval
|
||||
self.max_failures = max_failures
|
||||
self.recovery_interval = recovery_interval
|
||||
|
||||
# port -> consecutive failure count
|
||||
self._failures: dict[int, int] = {p: 0 for p in ports}
|
||||
# port -> last healthy timestamp
|
||||
self._last_healthy: dict[int, float] = {p: time.monotonic() for p in ports}
|
||||
self._lock = asyncio.Lock()
|
||||
self._task: asyncio.Task | None = None
|
||||
|
||||
async def _check_one(self, port: int, session: aiohttp.ClientSession | None) -> bool:
|
||||
url = f"http://{self.host}:{port}/health"
|
||||
try:
|
||||
if HAS_AIOHTTP and session is not None:
|
||||
async with session.get(url, timeout=aiohttp.ClientTimeout(total=5)) as resp:
|
||||
return resp.status == 200
|
||||
else:
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
with urllib.request.urlopen(req, timeout=5) as resp:
|
||||
return resp.status == 200
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
async def _check_all(self, session: aiohttp.ClientSession | None) -> None:
|
||||
checks = [self._check_one(p, session) for p in self.all_ports]
|
||||
results = await asyncio.gather(*checks, return_exceptions=True)
|
||||
now = time.monotonic()
|
||||
async with self._lock:
|
||||
for port, ok in zip(self.all_ports, results):
|
||||
if isinstance(ok, Exception):
|
||||
ok = False
|
||||
if ok:
|
||||
self._failures[port] = 0
|
||||
self._last_healthy[port] = now
|
||||
else:
|
||||
self._failures[port] += 1
|
||||
|
||||
async def start(self, session: aiohttp.ClientSession | None) -> None:
|
||||
async def _loop() -> None:
|
||||
while True:
|
||||
await asyncio.sleep(self.check_interval)
|
||||
await self._check_all(session)
|
||||
self._task = asyncio.create_task(_loop())
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
@property
|
||||
async def healthy_ports(self) -> list[int]:
|
||||
now = time.monotonic()
|
||||
async with self._lock:
|
||||
healthy = []
|
||||
for p in self.all_ports:
|
||||
if self._failures[p] < self.max_failures:
|
||||
healthy.append(p)
|
||||
elif now - self._last_healthy[p] > self.recovery_interval:
|
||||
# Give it another chance after recovery_interval.
|
||||
self._failures[p] = max(0, self._failures[p] - 1)
|
||||
healthy.append(p)
|
||||
return healthy
|
||||
|
||||
async def is_healthy(self, port: int) -> bool:
|
||||
ports = await self.healthy_ports
|
||||
return port in ports
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Load balancer with failover
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class LoadBalancer:
|
||||
"""Distribute requests across healthy backend ports with auto-failover."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
ports: list[int],
|
||||
health_checker: HealthChecker,
|
||||
strategy: str = "round_robin",
|
||||
):
|
||||
self.all_ports = ports
|
||||
self.health_checker = health_checker
|
||||
self.strategy = strategy
|
||||
self._idx = 0
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
def _next_round_robin(self, healthy: list[int]) -> int:
|
||||
if not healthy:
|
||||
return self.all_ports[self._idx % len(self.all_ports)]
|
||||
for _ in range(len(self.all_ports)):
|
||||
port = self.all_ports[self._idx % len(self.all_ports)]
|
||||
self._idx += 1
|
||||
if port in healthy:
|
||||
return port
|
||||
# Fallback: all unhealthy, pick first healthy anyway.
|
||||
return healthy[0] if healthy else self.all_ports[0]
|
||||
|
||||
def _next_random(self, healthy: list[int]) -> int:
|
||||
if healthy:
|
||||
return random.choice(healthy)
|
||||
return random.choice(self.all_ports)
|
||||
|
||||
async def next_port(self) -> int:
|
||||
healthy = await self.health_checker.healthy_ports
|
||||
async with self._lock:
|
||||
if self.strategy == "round_robin":
|
||||
return self._next_round_robin(healthy)
|
||||
elif self.strategy in ("random", "least_conn"):
|
||||
return self._next_random(healthy)
|
||||
else:
|
||||
return self._next_round_robin(healthy)
|
||||
|
||||
@property
|
||||
def port_count(self) -> int:
|
||||
return len(self.all_ports)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Token helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
import uuid
|
||||
def make_prompt(token_len: int) -> str:
|
||||
prefix = str(uuid.uuid4())
|
||||
|
||||
words = [
|
||||
"the", "quick", "brown", "fox",
|
||||
"jumps", "over", "lazy", "dog",
|
||||
"benchmark", "performance", "latency"
|
||||
]
|
||||
|
||||
tokens = [prefix]
|
||||
|
||||
while len(tokens) < token_len:
|
||||
tokens.extend(words)
|
||||
|
||||
return " ".join(tokens[:token_len])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# aiohttp-based async request (preferred)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def send_request_aiohttp(
|
||||
session: aiohttp.ClientSession,
|
||||
host: str,
|
||||
port: int,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
) -> RequestResult:
|
||||
url = f"http://{host}:{port}/v1/chat/completions"
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"stream": True,
|
||||
}
|
||||
headers = {"Content-Type": "application/json"}
|
||||
|
||||
result = RequestResult(request_id=request_id, target_port=port)
|
||||
start_time = time.perf_counter()
|
||||
first_token_received = False
|
||||
last_token_time = 0.0
|
||||
|
||||
try:
|
||||
async with session.post(url, json=payload, headers=headers) as resp:
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
result.error = f"HTTP {resp.status}: {text[:200]}"
|
||||
return result
|
||||
|
||||
async for line in resp.content:
|
||||
line = line.decode("utf-8").strip()
|
||||
if not line or not line.startswith("data: "):
|
||||
continue
|
||||
data_str = line[len("data: "):]
|
||||
if data_str == "[DONE]":
|
||||
break
|
||||
try:
|
||||
chunk = json.loads(data_str)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
choices = chunk.get("choices", [])
|
||||
if not choices:
|
||||
continue
|
||||
|
||||
delta = choices[0].get("delta", {})
|
||||
content = delta.get("content", "")
|
||||
if content:
|
||||
now = time.perf_counter()
|
||||
if not first_token_received:
|
||||
result.first_token_time_ms = (now - start_time) * 1000
|
||||
first_token_received = True
|
||||
else:
|
||||
result.inter_token_latencies.append((now - last_token_time) * 1000)
|
||||
last_token_time = now
|
||||
result.output_tokens += 1
|
||||
|
||||
result.e2e_latency_ms = (time.perf_counter() - start_time) * 1000
|
||||
result.input_tokens = len(prompt.split()) # approximate
|
||||
result.success = True
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
result.error = "timeout"
|
||||
except Exception as e:
|
||||
result.error = str(e)[:200]
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# urllib-based synchronous request (fallback, no aiohttp)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def send_request_urllib(
|
||||
host: str,
|
||||
port: int,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
) -> RequestResult:
|
||||
"""Blocking fallback using stdlib urllib. Used when aiohttp is absent."""
|
||||
url = f"http://{host}:{port}/v1/chat/completions"
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"stream": False,
|
||||
}
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
headers = {"Content-Type": "application/json", "Content-Length": str(len(data))}
|
||||
|
||||
result = RequestResult(request_id=request_id, target_port=port)
|
||||
start_time = time.perf_counter()
|
||||
|
||||
try:
|
||||
req = urllib.request.Request(url, data=data, headers=headers, method="POST")
|
||||
with urllib.request.urlopen(req, timeout=600) as resp:
|
||||
body = json.loads(resp.read().decode("utf-8"))
|
||||
|
||||
choices = body.get("choices", [])
|
||||
if not choices:
|
||||
result.error = "no choices in response"
|
||||
return result
|
||||
|
||||
content = choices[0].get("message", {}).get("content", "")
|
||||
result.output_tokens = len(content.split()) if content else 0
|
||||
result.e2e_latency_ms = (time.perf_counter() - start_time) * 1000
|
||||
result.first_token_time_ms = result.e2e_latency_ms
|
||||
result.input_tokens = len(prompt.split())
|
||||
result.success = True
|
||||
|
||||
except Exception as e:
|
||||
result.error = str(e)[:200]
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Unified send_request wrapper with retry & failover
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def send_request_with_retry(
|
||||
session: aiohttp.ClientSession | None,
|
||||
host: str,
|
||||
lb: LoadBalancer,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
max_retries: int = 2,
|
||||
) -> RequestResult:
|
||||
"""Send a request, retrying on failure with a different port each time."""
|
||||
result = RequestResult(request_id=request_id)
|
||||
for attempt in range(max_retries + 1):
|
||||
port = await lb.next_port()
|
||||
if HAS_AIOHTTP and session is not None:
|
||||
result = await send_request_aiohttp(
|
||||
session, host, port, model, request_id, prompt, max_tokens, temperature
|
||||
)
|
||||
else:
|
||||
loop = asyncio.get_event_loop()
|
||||
result = await loop.run_in_executor(
|
||||
None, send_request_urllib,
|
||||
host, port, model, request_id, prompt, max_tokens, temperature
|
||||
)
|
||||
result.retry_count = attempt
|
||||
if result.success:
|
||||
return result
|
||||
# If failed, try another port on next iteration (unless last attempt).
|
||||
if attempt < max_retries:
|
||||
await asyncio.sleep(0.5 * (attempt + 1)) # exponential backoff-ish
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Benchmark orchestration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def run_benchmark(
|
||||
host: str,
|
||||
ports: list[int],
|
||||
lb: LoadBalancer,
|
||||
health_checker: HealthChecker,
|
||||
model: str,
|
||||
concurrency: int,
|
||||
input_len: int,
|
||||
output_len: int,
|
||||
num_prompts: int,
|
||||
temperature: float,
|
||||
output_file: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Run the benchmark and write a JSONL file."""
|
||||
prompts = [make_prompt(input_len) for _ in range(num_prompts)]
|
||||
results: list[RequestResult] = []
|
||||
semaphore = asyncio.Semaphore(concurrency)
|
||||
|
||||
start = time.perf_counter()
|
||||
|
||||
if HAS_AIOHTTP:
|
||||
connector = aiohttp.TCPConnector(limit=concurrency * 2)
|
||||
timeout = aiohttp.ClientTimeout(total=600, sock_read=300)
|
||||
async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
|
||||
await health_checker.start(session)
|
||||
|
||||
async def bounded_send(idx: int, prompt: str) -> RequestResult:
|
||||
async with semaphore:
|
||||
return await send_request_with_retry(
|
||||
session, host, lb, model, idx, prompt, output_len, temperature
|
||||
)
|
||||
print("Starting warmup...")
|
||||
warmup_per_service = 10
|
||||
warmup_requests = warmup_per_service * len(ports)
|
||||
|
||||
warmup_prompts = [
|
||||
make_prompt(input_len)
|
||||
for _ in range(warmup_requests)
|
||||
]
|
||||
|
||||
warmup_tasks = [
|
||||
send_request_with_retry(
|
||||
session=session,
|
||||
host=host,
|
||||
lb=lb,
|
||||
model=model,
|
||||
request_id=-1,
|
||||
prompt=p,
|
||||
max_tokens=output_len,
|
||||
temperature=temperature,
|
||||
)
|
||||
for p in warmup_prompts
|
||||
]
|
||||
|
||||
await asyncio.gather(*warmup_tasks)
|
||||
|
||||
print("Warmup finished.")
|
||||
|
||||
# 给 CUDA / Scheduler 一点时间稳定
|
||||
await asyncio.sleep(2)
|
||||
|
||||
tasks = [bounded_send(i, p) for i, p in enumerate(prompts)]
|
||||
results = await asyncio.gather(*tasks)
|
||||
await health_checker.stop()
|
||||
else:
|
||||
await health_checker.start(None)
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
async def bounded_send(idx: int, prompt: str) -> RequestResult:
|
||||
async with semaphore:
|
||||
return await send_request_with_retry(
|
||||
None, host, lb, model, idx, prompt, output_len, temperature
|
||||
)
|
||||
|
||||
tasks = [bounded_send(i, p) for i, p in enumerate(prompts)]
|
||||
results = await asyncio.gather(*tasks)
|
||||
await health_checker.stop()
|
||||
|
||||
duration = time.perf_counter() - start
|
||||
|
||||
# Write JSONL
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
for r in results:
|
||||
f.write(json.dumps(r.to_dict(), ensure_ascii=False) + "\n")
|
||||
|
||||
# Compute summary
|
||||
successful = [r for r in results if r.success]
|
||||
failed = len(results) - len(successful)
|
||||
retried = [r for r in results if r.retry_count > 0]
|
||||
|
||||
def percentile(values: list[float], p: float) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
s = sorted(values)
|
||||
k = (len(s) - 1) * p / 100.0
|
||||
f = int(k)
|
||||
c = min(f + 1, len(s) - 1)
|
||||
return s[f] + (k - f) * (s[c] - s[f])
|
||||
|
||||
ttfts = [r.first_token_time_ms for r in successful]
|
||||
tpots = [r.tpot_ms for r in successful if r.output_tokens > 1]
|
||||
e2es = [r.e2e_latency_ms for r in successful]
|
||||
itls = [r.mean_itl_ms for r in successful if r.inter_token_latencies]
|
||||
|
||||
total_input_tokens = sum(r.input_tokens for r in successful)
|
||||
total_output_tokens = sum(r.output_tokens for r in successful)
|
||||
|
||||
# Per-port breakdown
|
||||
per_port_stats: dict[int, dict[str, Any]] = {}
|
||||
for port in ports:
|
||||
port_results = [r for r in successful if r.target_port == port]
|
||||
port_failed = [r for r in results if r.target_port == port and not r.success]
|
||||
if port_results or port_failed:
|
||||
port_ttfts = [r.first_token_time_ms for r in port_results]
|
||||
port_e2es = [r.e2e_latency_ms for r in port_results]
|
||||
per_port_stats[port] = {
|
||||
"count": len(port_results),
|
||||
"failed": len(port_failed),
|
||||
"mean_ttft_ms": sum(port_ttfts) / len(port_ttfts) if port_ttfts else 0.0,
|
||||
"p95_ttft_ms": percentile(port_ttfts, 95) if port_ttfts else 0.0,
|
||||
"mean_e2e_ms": sum(port_e2es) / len(port_e2es) if port_e2es else 0.0,
|
||||
}
|
||||
|
||||
# Health-check stats
|
||||
healthy_at_end = await health_checker.healthy_ports
|
||||
|
||||
summary = {
|
||||
"completed": len(successful),
|
||||
"failed": failed,
|
||||
"retried": len(retried),
|
||||
"duration": duration,
|
||||
"request_throughput": len(successful) / duration if duration > 0 else 0.0,
|
||||
"input_throughput": total_input_tokens / duration if duration > 0 else 0.0,
|
||||
"output_throughput": total_output_tokens / duration if duration > 0 else 0.0,
|
||||
"total_throughput": (total_input_tokens + total_output_tokens) / duration if duration > 0 else 0.0,
|
||||
"total_input_tokens": total_input_tokens,
|
||||
"total_output_tokens": total_output_tokens,
|
||||
"mean_ttft_ms": sum(ttfts) / len(ttfts) if ttfts else 0.0,
|
||||
"median_ttft_ms": percentile(ttfts, 50),
|
||||
"p90_ttft_ms": percentile(ttfts, 90),
|
||||
"p95_ttft_ms": percentile(ttfts, 95),
|
||||
"p99_ttft_ms": percentile(ttfts, 99),
|
||||
"mean_tpot_ms": sum(tpots) / len(tpots) if tpots else 0.0,
|
||||
"median_tpot_ms": percentile(tpots, 50),
|
||||
"p90_tpot_ms": percentile(tpots, 90),
|
||||
"p95_tpot_ms": percentile(tpots, 95),
|
||||
"p99_tpot_ms": percentile(tpots, 99),
|
||||
"mean_e2e_latency_ms": sum(e2es) / len(e2es) if e2es else 0.0,
|
||||
"median_e2e_latency_ms": percentile(e2es, 50),
|
||||
"p90_e2e_latency_ms": percentile(e2es, 90),
|
||||
"p95_e2e_latency_ms": percentile(e2es, 95),
|
||||
"p99_e2e_latency_ms": percentile(e2es, 99),
|
||||
"mean_itl_ms": sum(itls) / len(itls) if itls else 0.0,
|
||||
"median_itl_ms": percentile(itls, 50),
|
||||
"p90_itl_ms": percentile(itls, 90),
|
||||
"p95_itl_ms": percentile(itls, 95),
|
||||
"p99_itl_ms": percentile(itls, 99),
|
||||
"per_port_stats": per_port_stats,
|
||||
"healthy_ports_at_end": healthy_at_end,
|
||||
"lb_strategy": lb.strategy,
|
||||
"num_services": lb.port_count,
|
||||
}
|
||||
|
||||
# Append summary as the last line
|
||||
with open(output_file, "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(summary, ensure_ascii=False) + "\n")
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Custom vLLM benchmark client with multi-service LB and health-check failover")
|
||||
parser.add_argument("--host", default="127.0.0.1")
|
||||
parser.add_argument("--port", type=int, default=30005,
|
||||
help="Single service port (backward compatible)")
|
||||
parser.add_argument("--ports", default="",
|
||||
help="Comma-separated list of ports for multi-service, e.g. 30005,30006,30007,30008")
|
||||
parser.add_argument("--model", default="deepseek-v4-flash")
|
||||
parser.add_argument("--concurrency", type=int, default=1)
|
||||
parser.add_argument("--input-len", type=int, default=512)
|
||||
parser.add_argument("--output-len", type=int, default=256)
|
||||
parser.add_argument("--num-prompts", type=int, default=10)
|
||||
parser.add_argument("--temperature", type=float, default=0.0)
|
||||
parser.add_argument("--lb-strategy", default="round_robin",
|
||||
choices=["round_robin", "random", "least_conn"],
|
||||
help="Load balancing strategy across services")
|
||||
parser.add_argument("--health-check-interval", type=float, default=5.0,
|
||||
help="Seconds between health checks (default: 5)")
|
||||
parser.add_argument("--health-max-failures", type=int, default=2,
|
||||
help="Consecutive failures before marking a port unhealthy (default: 2)")
|
||||
parser.add_argument("--health-recovery-interval", type=float, default=10.0,
|
||||
help="Seconds before retrying an unhealthy port (default: 10)")
|
||||
parser.add_argument("--request-max-retries", type=int, default=2,
|
||||
help="Max retries per request on failure (default: 2)")
|
||||
parser.add_argument("--output-file", required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Determine ports: --ports takes precedence, else fall back to --port.
|
||||
if args.ports:
|
||||
ports = [int(p.strip()) for p in args.ports.split(",")]
|
||||
else:
|
||||
ports = [args.port]
|
||||
|
||||
health_checker = HealthChecker(
|
||||
host=args.host,
|
||||
ports=ports,
|
||||
check_interval=args.health_check_interval,
|
||||
max_failures=args.health_max_failures,
|
||||
recovery_interval=args.health_recovery_interval,
|
||||
)
|
||||
lb = LoadBalancer(ports, health_checker=health_checker, strategy=args.lb_strategy)
|
||||
|
||||
summary = asyncio.run(run_benchmark(
|
||||
host=args.host,
|
||||
ports=ports,
|
||||
lb=lb,
|
||||
health_checker=health_checker,
|
||||
model=args.model,
|
||||
concurrency=args.concurrency,
|
||||
input_len=args.input_len,
|
||||
output_len=args.output_len,
|
||||
num_prompts=args.num_prompts,
|
||||
temperature=args.temperature,
|
||||
output_file=args.output_file,
|
||||
))
|
||||
|
||||
print(json.dumps(summary, indent=2, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
180
experiments/dsv4_h200_vllm_tp4_custom_bench/config.env
Normal file
180
experiments/dsv4_h200_vllm_tp4_custom_bench/config.env
Normal file
@ -0,0 +1,180 @@
|
||||
# Experiment configuration for dsv4_h200_vllm_tp2_custom_bench
|
||||
# Custom benchmark client for multi-service vLLM TP=2 on 8x H200.
|
||||
#
|
||||
# This experiment runs 4 independent vLLM services (TP=2 each) on GPU pairs:
|
||||
# Service 1: port 30005, GPUs 0,1
|
||||
# Service 2: port 30006, GPUs 2,3
|
||||
# Service 3: port 30007, GPUs 4,5
|
||||
# Service 4: port 30008, GPUs 6,7
|
||||
#
|
||||
# bench_client.py distributes requests across all 4 services via round-robin
|
||||
# load balancing, giving accurate cluster-wide throughput even at low
|
||||
# per-service concurrency.
|
||||
|
||||
EXPERIMENT="dsv4_h200_vllm_tp4_custom_bench"
|
||||
MODEL_NAME="DeepSeek-V4-Flash"
|
||||
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
|
||||
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
# Primary port (backward compatible); multi-service ports are in PORTS.
|
||||
PORT="30005"
|
||||
PORTS="30005,30006"
|
||||
|
||||
BACKEND="vllm"
|
||||
ENGINE="vllm"
|
||||
|
||||
# Native virtual environments on the host.
|
||||
VENV_SERVER="/data/user1/yy/envs/vllm"
|
||||
VENV_CLIENT="/data/user1/yy/envs/vllm"
|
||||
|
||||
# Load balancing strategy: round_robin | random | least_conn
|
||||
LB_STRATEGY="round_robin"
|
||||
|
||||
# Benchmark scenarios: "concurrency input_len output_len num_prompts".
|
||||
# For multi-service we include low-concurrency scenarios to show how
|
||||
# requests spread across 2 services.
|
||||
SCENARIOS=(
|
||||
"1 512 256 20"
|
||||
"2 512 256 40"
|
||||
"4 512 256 80"
|
||||
"8 512 256 160"
|
||||
"16 512 256 320"
|
||||
"32 512 256 640"
|
||||
"64 512 256 1280"
|
||||
"128 512 256 2560"
|
||||
|
||||
"1 1024 128 20"
|
||||
"1 1024 256 20"
|
||||
"1 1024 512 20"
|
||||
"1 1024 1024 20"
|
||||
"1 1024 2048 20"
|
||||
"1 1024 4096 20"
|
||||
|
||||
"8 1024 128 160"
|
||||
"8 1024 256 160"
|
||||
"8 1024 512 160"
|
||||
"8 1024 1024 160"
|
||||
"8 1024 2048 160"
|
||||
"8 1024 4096 160"
|
||||
|
||||
"32 1024 128 640"
|
||||
"32 1024 256 640"
|
||||
"32 1024 512 640"
|
||||
"32 1024 1024 640"
|
||||
"32 1024 2048 640"
|
||||
"32 1024 4096 640"
|
||||
|
||||
"128 1024 128 2560"
|
||||
"128 1024 256 2560"
|
||||
"128 1024 512 2560"
|
||||
"128 1024 1024 2560"
|
||||
"128 1024 2048 2560"
|
||||
"128 1024 4096 2560"
|
||||
|
||||
"1 2048 128 20"
|
||||
"1 2048 256 20"
|
||||
"1 2048 512 20"
|
||||
"1 2048 1024 20"
|
||||
"1 2048 2048 20"
|
||||
"1 2048 4096 20"
|
||||
|
||||
"8 2048 128 160"
|
||||
"8 2048 256 160"
|
||||
"8 2048 512 160"
|
||||
"8 2048 1024 160"
|
||||
"8 2048 2048 160"
|
||||
"8 2048 4096 160"
|
||||
|
||||
"32 2048 128 640"
|
||||
"32 2048 256 640"
|
||||
"32 2048 512 640"
|
||||
"32 2048 1024 640"
|
||||
"32 2048 2048 640"
|
||||
"32 2048 4096 640"
|
||||
|
||||
"128 2048 128 2560"
|
||||
"128 2048 256 2560"
|
||||
"128 2048 512 2560"
|
||||
"128 2048 1024 2560"
|
||||
"128 2048 2048 2560"
|
||||
"128 2048 4096 2560"
|
||||
|
||||
"1 4096 128 20"
|
||||
"1 4096 256 20"
|
||||
"1 4096 512 20"
|
||||
"1 4096 1024 20"
|
||||
"1 4096 2048 20"
|
||||
"1 4096 4096 20"
|
||||
|
||||
"8 4096 128 160"
|
||||
"8 4096 256 160"
|
||||
"8 4096 512 160"
|
||||
"8 4096 1024 160"
|
||||
"8 4096 2048 160"
|
||||
"8 4096 4096 160"
|
||||
|
||||
"32 4096 128 640"
|
||||
"32 4096 256 640"
|
||||
"32 4096 512 640"
|
||||
"32 4096 1024 640"
|
||||
"32 4096 2048 640"
|
||||
"32 4096 4096 640"
|
||||
|
||||
"128 4096 128 2560"
|
||||
"128 4096 256 2560"
|
||||
"128 4096 512 2560"
|
||||
"128 4096 1024 2560"
|
||||
"128 4096 2048 2560"
|
||||
"128 4096 4096 2560"
|
||||
|
||||
"1 16384 128 20"
|
||||
"1 16384 256 20"
|
||||
"1 16384 512 20"
|
||||
"1 16384 1024 20"
|
||||
"1 16384 2048 20"
|
||||
|
||||
"8 16384 128 160"
|
||||
"8 16384 256 160"
|
||||
"8 16384 512 160"
|
||||
"8 16384 1024 160"
|
||||
"8 16384 2048 160"
|
||||
|
||||
"32 16384 128 640"
|
||||
"32 16384 256 640"
|
||||
"32 16384 512 640"
|
||||
"32 16384 1024 640"
|
||||
"32 16384 2048 640"
|
||||
|
||||
"128 16384 128 2560"
|
||||
"128 16384 256 2560"
|
||||
"128 16384 512 2560"
|
||||
"128 16384 1024 2560"
|
||||
"128 16384 2048 2560"
|
||||
|
||||
|
||||
"1 65536 128 20"
|
||||
"1 65536 256 20"
|
||||
"1 65536 512 20"
|
||||
"1 65536 1024 20"
|
||||
|
||||
"8 65536 128 160"
|
||||
"8 65536 256 160"
|
||||
"8 65536 512 160"
|
||||
"8 65536 1024 160"
|
||||
"32 65536 1024 160"
|
||||
|
||||
"1 131072 128 20"
|
||||
"1 131072 256 20"
|
||||
"1 131072 512 20"
|
||||
|
||||
"8 131072 512 160"
|
||||
|
||||
"1 262144 256 20"
|
||||
|
||||
"8 262144 128 160"
|
||||
|
||||
"1 524288 128 20"
|
||||
)
|
||||
|
||||
# Server start script bundled with this experiment.
|
||||
SERVER_START_SCRIPT="${SCRIPT_DIR}/start_server.sh"
|
||||
206
experiments/dsv4_h200_vllm_tp4_custom_bench/parse_results.py
Executable file
206
experiments/dsv4_h200_vllm_tp4_custom_bench/parse_results.py
Executable file
@ -0,0 +1,206 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Parse custom bench_client.py JSONL outputs for vLLM TP=2 benchmark.
|
||||
|
||||
Reads JSONL files produced by bench_client.py (one request per line,
|
||||
last line is the summary) and generates:
|
||||
- results.json (appended scenarios, following the standard schema)
|
||||
- report.md (human-readable summary)
|
||||
|
||||
Usage:
|
||||
python3 parse_results.py <result_root>
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_jsonl(path: Path) -> tuple[dict | None, list[dict]]:
|
||||
"""Read the summary JSON (last line) and all per-request lines."""
|
||||
requests: list[dict] = []
|
||||
summary: dict | None = None
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
# The summary line has "completed" and "duration" keys.
|
||||
if "completed" in obj and "duration" in obj:
|
||||
summary = obj
|
||||
else:
|
||||
requests.append(obj)
|
||||
return summary, requests
|
||||
|
||||
|
||||
def compute_metrics(summary: dict, requests: list[dict]) -> dict:
|
||||
completed = summary.get("completed", 0)
|
||||
total = len(requests)
|
||||
failed = total - completed if total > 0 else 0
|
||||
duration_s = summary.get("duration", 0.0)
|
||||
|
||||
# Extract per-request metrics for percentiles.
|
||||
ttfts = [r["first_token_time_ms"] for r in requests if r.get("success")]
|
||||
tpots = [r["tpot_ms"] for r in requests if r.get("success") and r.get("output_tokens", 0) > 1]
|
||||
e2es = [r["e2e_latency_ms"] for r in requests if r.get("success")]
|
||||
itls = [r["mean_itl_ms"] for r in requests if r.get("success") and r.get("inter_token_latencies")]
|
||||
|
||||
def percentile(values: list[float], p: float) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
s = sorted(values)
|
||||
k = (len(s) - 1) * p / 100.0
|
||||
f = int(k)
|
||||
c = min(f + 1, len(s) - 1)
|
||||
return s[f] + (k - f) * (s[c] - s[f])
|
||||
|
||||
return {
|
||||
"success": completed,
|
||||
"failed": failed,
|
||||
"duration_s": duration_s,
|
||||
"request_throughput": summary.get("request_throughput", 0.0),
|
||||
"input_token_throughput": summary.get("input_throughput", 0.0),
|
||||
"output_token_throughput": summary.get("output_throughput", 0.0),
|
||||
"total_token_throughput": summary.get("total_throughput", 0.0),
|
||||
"total_input_tokens": summary.get("total_input_tokens", 0),
|
||||
"total_output_tokens": summary.get("total_output_tokens", 0),
|
||||
"e2e_ms": {
|
||||
"mean": summary.get("mean_e2e_latency_ms", 0.0),
|
||||
"p50": percentile(e2es, 50),
|
||||
"p90": percentile(e2es, 90),
|
||||
"p95": percentile(e2es, 95),
|
||||
"p99": percentile(e2es, 99),
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": summary.get("mean_ttft_ms", 0.0),
|
||||
"p50": percentile(ttfts, 50),
|
||||
"p90": percentile(ttfts, 90),
|
||||
"p95": percentile(ttfts, 95),
|
||||
"p99": percentile(ttfts, 99),
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": summary.get("mean_tpot_ms", 0.0),
|
||||
"p50": percentile(tpots, 50),
|
||||
"p90": percentile(tpots, 90),
|
||||
"p95": percentile(tpots, 95),
|
||||
"p99": percentile(tpots, 99),
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": summary.get("mean_itl_ms", 0.0),
|
||||
"p50": percentile(itls, 50),
|
||||
"p90": percentile(itls, 90),
|
||||
"p95": percentile(itls, 95),
|
||||
"p99": percentile(itls, 99),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def scenario_name(concurrency: int, input_len: int, output_len: int) -> str:
|
||||
return f"c{concurrency}_i{input_len}_o{output_len}"
|
||||
|
||||
|
||||
def slo_status(metrics: dict, ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> dict:
|
||||
ttft_ok = metrics["ttft_ms"]["p95"] < ttft_limit_ms
|
||||
tpot_ok = metrics["tpot_ms"]["mean"] < tpot_limit_ms
|
||||
if ttft_ok and tpot_ok:
|
||||
mark = "✅"
|
||||
elif ttft_ok or tpot_ok:
|
||||
mark = "⚠️"
|
||||
else:
|
||||
mark = "❌"
|
||||
return {
|
||||
"ttft_p95_ok": ttft_ok,
|
||||
"tpot_mean_ok": tpot_ok,
|
||||
"overall": mark,
|
||||
}
|
||||
|
||||
|
||||
def append_scenario(results_json: Path, scenario: dict) -> None:
|
||||
with open(results_json, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
data["scenarios"].append(scenario)
|
||||
with open(results_json, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
def generate_report(result_root: Path, scenarios: list[dict]) -> None:
|
||||
report_path = result_root / "report.md"
|
||||
with open(report_path, "w", encoding="utf-8") as f:
|
||||
f.write("# H200 vLLM TP=2 Custom Benchmark Report\n\n")
|
||||
f.write("- **Client**: `bench_client.py` (async OpenAI API, per-request timing)\n")
|
||||
f.write("- **Backend**: vLLM (TP=2, FP8 KV cache, no speculative decoding)\n")
|
||||
f.write(f"- **Result root**: `{result_root}`\n\n")
|
||||
|
||||
f.write("## Results\n\n")
|
||||
f.write("| Scenario | Concurrency | Input | Output | Duration(s) | Success | Failed | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |\n")
|
||||
f.write("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n")
|
||||
|
||||
for s in scenarios:
|
||||
cfg = s["config"]
|
||||
m = s["metrics"]
|
||||
slo = s.get("slo_status", {}).get("overall", "")
|
||||
f.write(
|
||||
f"| {s['name']} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | "
|
||||
f"{m['duration_s']:.2f} | {m['success']} | {m['failed']} | {m['request_throughput']:.2f} | "
|
||||
f"{m['input_token_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||
f"{m['total_token_throughput']:.2f} | "
|
||||
f"{m['ttft_ms']['mean']:.2f} | {m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||
f"{m['e2e_ms']['mean']:.2f} | {m['e2e_ms']['p95']:.2f} | {m['e2e_ms']['p99']:.2f} | {slo} |\n"
|
||||
)
|
||||
f.write("\n")
|
||||
f.write("SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.\n\n")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("results")
|
||||
raw_dir = result_root / "raw_outputs"
|
||||
results_json = result_root / "results.json"
|
||||
|
||||
if not raw_dir.exists():
|
||||
raise SystemExit(f"raw_outputs directory not found: {raw_dir}")
|
||||
|
||||
scenarios = []
|
||||
for jsonl_path in sorted(raw_dir.glob("vllm_*.jsonl")):
|
||||
parts = jsonl_path.stem.split("_")
|
||||
if len(parts) < 5:
|
||||
continue
|
||||
concurrency, input_len, output_len = int(parts[2]), int(parts[3]), int(parts[4])
|
||||
|
||||
summary, requests = parse_jsonl(jsonl_path)
|
||||
if summary is None:
|
||||
continue
|
||||
|
||||
metrics = compute_metrics(summary, requests)
|
||||
scenario = {
|
||||
"name": scenario_name(concurrency, input_len, output_len),
|
||||
"config": {
|
||||
"concurrency": concurrency,
|
||||
"input_len": input_len,
|
||||
"output_len": output_len,
|
||||
"dataset": "random",
|
||||
"num_prompts": metrics["success"] + metrics["failed"],
|
||||
},
|
||||
"metrics": metrics,
|
||||
"slo_status": slo_status(metrics),
|
||||
"raw_file": str(jsonl_path),
|
||||
}
|
||||
scenarios.append(scenario)
|
||||
|
||||
if not scenarios:
|
||||
print("No benchmark outputs found to parse")
|
||||
return
|
||||
|
||||
if results_json.exists():
|
||||
for s in scenarios:
|
||||
append_scenario(results_json, s)
|
||||
|
||||
generate_report(result_root, scenarios)
|
||||
print(f"Parsed {len(scenarios)} scenarios into {result_root}/report.md")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -0,0 +1,129 @@
|
||||
# H200 vLLM TP=2 Custom Benchmark Report
|
||||
|
||||
- **Client**: `bench_client.py` (async OpenAI API, per-request timing)
|
||||
- **Backend**: vLLM (TP=2, FP8 KV cache, no speculative decoding)
|
||||
- **Result root**: `/data/user1/qqt/sskj/experiments/dsv4_h200_vllm_tp4_custom_bench/results/20260711-133908`
|
||||
|
||||
## Results
|
||||
|
||||
| Scenario | Concurrency | Input | Output | Duration(s) | Success | Failed | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c128_i1024_o1024 | 128 | 1024 | 1024 | 172.39 | 2560 | 0 | 14.85 | 15206.17 | 3100.60 | 18306.76 | 267.16 | 478.80 | 2297.70 | 38.53 | 66.16 | 70.07 | 8265.00 | 13967.23 | 17536.46 | ✅ |
|
||||
| c128_i1024_o128 | 128 | 1024 | 128 | 98.84 | 2560 | 0 | 25.90 | 26522.07 | 3314.91 | 29836.98 | 900.39 | 1590.87 | 2307.93 | 30.16 | 35.98 | 37.09 | 4730.89 | 5530.64 | 6122.15 | ✅ |
|
||||
| c128_i1024_o2048 | 128 | 1024 | 2048 | 161.76 | 2560 | 0 | 15.83 | 16205.28 | 3297.74 | 19503.02 | 256.61 | 340.87 | 2315.99 | 36.07 | 39.27 | 40.30 | 7732.01 | 10616.59 | 11870.42 | ✅ |
|
||||
| c128_i1024_o256 | 128 | 1024 | 256 | 161.16 | 2560 | 0 | 15.89 | 16266.52 | 3250.72 | 19517.24 | 259.69 | 356.25 | 2333.30 | 36.68 | 40.81 | 51.99 | 7727.17 | 10133.95 | 11225.21 | ✅ |
|
||||
| c128_i1024_o4096 | 128 | 1024 | 4096 | 163.16 | 2560 | 0 | 15.69 | 16066.27 | 3281.03 | 19347.31 | 257.28 | 422.58 | 2303.92 | 36.09 | 39.21 | 40.35 | 7768.07 | 10730.88 | 12234.15 | ✅ |
|
||||
| c128_i1024_o512 | 128 | 1024 | 512 | 161.49 | 2560 | 0 | 15.85 | 16232.48 | 3275.92 | 19508.40 | 255.16 | 334.67 | 2297.21 | 36.27 | 39.39 | 40.57 | 7716.22 | 10581.99 | 11832.60 | ✅ |
|
||||
| c128_i16384_o1024 | 128 | 16384 | 1024 | 919.25 | 2560 | 0 | 2.78 | 45627.36 | 697.33 | 46324.69 | 2625.93 | 3630.11 | 33634.45 | 171.30 | 196.69 | 207.71 | 45001.23 | 90438.26 | 123707.69 | ❌ |
|
||||
| c128_i16384_o128 | 128 | 16384 | 128 | 859.37 | 2560 | 0 | 2.98 | 48806.72 | 380.95 | 49187.68 | 2833.19 | 3131.25 | 33643.84 | 311.88 | 328.04 | 328.10 | 42403.99 | 44239.06 | 73426.74 | ❌ |
|
||||
| c128_i16384_o2048 | 128 | 16384 | 2048 | 918.61 | 2560 | 0 | 2.79 | 45659.28 | 701.99 | 46361.27 | 2601.36 | 3607.43 | 33624.66 | 170.03 | 195.00 | 208.79 | 45030.71 | 92401.43 | 124284.89 | ❌ |
|
||||
| c128_i16384_o256 | 128 | 16384 | 256 | 897.62 | 2560 | 0 | 2.85 | 46726.87 | 602.42 | 47329.29 | 2804.71 | 4596.59 | 33617.37 | 197.06 | 224.66 | 236.73 | 44219.84 | 57755.16 | 76587.71 | ❌ |
|
||||
| c128_i16384_o512 | 128 | 16384 | 512 | 915.13 | 2560 | 0 | 2.80 | 45833.00 | 684.56 | 46517.56 | 2646.03 | 3737.57 | 33718.32 | 174.28 | 203.88 | 224.11 | 44914.54 | 88267.88 | 100313.27 | ❌ |
|
||||
| c128_i2048_o1024 | 128 | 2048 | 1024 | 204.30 | 2560 | 0 | 12.53 | 25663.20 | 2671.39 | 28334.59 | 372.84 | 493.00 | 4208.03 | 44.24 | 48.11 | 50.53 | 9762.62 | 15935.84 | 21541.33 | ✅ |
|
||||
| c128_i2048_o128 | 128 | 2048 | 128 | 153.29 | 2560 | 0 | 16.70 | 34202.45 | 2125.54 | 36327.99 | 791.19 | 1283.06 | 4199.42 | 52.51 | 58.15 | 59.97 | 7426.54 | 8226.85 | 10576.91 | ⚠️ |
|
||||
| c128_i2048_o2048 | 128 | 2048 | 2048 | 208.80 | 2560 | 0 | 12.26 | 25109.31 | 2631.24 | 27740.55 | 365.94 | 530.02 | 4211.56 | 44.40 | 48.85 | 54.00 | 9842.38 | 16285.06 | 20408.42 | ✅ |
|
||||
| c128_i2048_o256 | 128 | 2048 | 256 | 196.63 | 2560 | 0 | 13.02 | 26663.63 | 2605.54 | 29269.17 | 379.32 | 522.39 | 4205.72 | 45.73 | 50.01 | 51.66 | 9476.10 | 12538.01 | 13720.71 | ✅ |
|
||||
| c128_i2048_o4096 | 128 | 2048 | 4096 | 204.98 | 2560 | 0 | 12.49 | 25578.03 | 2696.32 | 28274.35 | 369.65 | 497.47 | 4203.66 | 44.06 | 47.90 | 49.49 | 9834.14 | 15705.61 | 22444.28 | ✅ |
|
||||
| c128_i2048_o512 | 128 | 2048 | 512 | 205.58 | 2560 | 0 | 12.45 | 25502.28 | 2682.24 | 28184.52 | 364.96 | 479.81 | 4198.31 | 44.32 | 48.38 | 49.46 | 9849.41 | 16273.27 | 21537.21 | ✅ |
|
||||
| c128_i4096_o1024 | 128 | 4096 | 1024 | 299.56 | 2560 | 0 | 8.55 | 35004.04 | 1823.49 | 36827.53 | 670.94 | 898.57 | 8472.05 | 64.95 | 71.72 | 78.31 | 14410.80 | 23823.73 | 34763.30 | ⚠️ |
|
||||
| c128_i4096_o128 | 128 | 4096 | 128 | 252.68 | 2560 | 0 | 10.13 | 41497.56 | 1283.45 | 42781.02 | 1110.40 | 1837.39 | 8464.33 | 89.25 | 99.28 | 101.99 | 12341.74 | 13909.51 | 19709.73 | ⚠️ |
|
||||
| c128_i4096_o2048 | 128 | 4096 | 2048 | 299.51 | 2560 | 0 | 8.55 | 35009.54 | 1841.89 | 36851.42 | 673.64 | 903.51 | 8456.42 | 64.31 | 71.20 | 77.98 | 14467.32 | 24917.88 | 37076.16 | ⚠️ |
|
||||
| c128_i4096_o256 | 128 | 4096 | 256 | 288.98 | 2560 | 0 | 8.86 | 36284.85 | 1743.96 | 38028.81 | 688.49 | 1029.32 | 8474.33 | 68.27 | 74.67 | 80.07 | 14052.64 | 18727.14 | 21844.01 | ⚠️ |
|
||||
| c128_i4096_o4096 | 128 | 4096 | 4096 | 297.69 | 2560 | 0 | 8.60 | 35223.60 | 1820.84 | 37044.43 | 679.88 | 979.68 | 8471.74 | 65.18 | 72.12 | 79.04 | 14368.53 | 24055.68 | 33864.98 | ⚠️ |
|
||||
| c128_i4096_o512 | 128 | 4096 | 512 | 296.90 | 2560 | 0 | 8.62 | 35317.89 | 1815.79 | 37133.68 | 666.19 | 888.77 | 8469.32 | 65.45 | 72.23 | 78.81 | 14377.25 | 24981.06 | 33674.28 | ⚠️ |
|
||||
| c128_i512_o256 | 128 | 512 | 256 | 155.29 | 2560 | 0 | 16.49 | 8440.72 | 3401.04 | 11841.77 | 216.12 | 439.65 | 1351.98 | 35.17 | 38.71 | 39.83 | 7433.74 | 9800.36 | 10039.98 | ✅ |
|
||||
| c16_i512_o256 | 16 | 512 | 256 | 60.82 | 320 | 0 | 5.26 | 2693.64 | 1089.67 | 3783.32 | 145.56 | 311.20 | 388.94 | 12.62 | 17.00 | 17.83 | 2748.60 | 4105.76 | 4640.23 | ✅ |
|
||||
| c1_i1024_o1024 | 1 | 1024 | 1024 | 52.34 | 20 | 0 | 0.38 | 391.29 | 85.14 | 476.42 | 115.81 | 118.26 | 140.50 | 6.99 | 7.00 | 7.00 | 1666.63 | 2301.13 | 2358.69 | ✅ |
|
||||
| c1_i1024_o128 | 1 | 1024 | 128 | 32.71 | 20 | 0 | 0.61 | 626.06 | 78.07 | 704.13 | 115.30 | 122.94 | 128.47 | 6.96 | 6.97 | 7.01 | 997.08 | 1004.84 | 1011.97 | ✅ |
|
||||
| c1_i1024_o2048 | 1 | 1024 | 2048 | 48.50 | 20 | 0 | 0.41 | 422.30 | 83.02 | 505.31 | 115.62 | 119.50 | 126.29 | 6.99 | 7.00 | 7.01 | 1516.37 | 1700.77 | 1778.35 | ✅ |
|
||||
| c1_i1024_o256 | 1 | 1024 | 256 | 48.48 | 20 | 0 | 0.41 | 422.40 | 81.26 | 503.66 | 114.81 | 120.58 | 125.23 | 6.99 | 7.01 | 7.01 | 1485.39 | 1837.55 | 1880.36 | ✅ |
|
||||
| c1_i1024_o4096 | 1 | 1024 | 4096 | 51.98 | 20 | 0 | 0.38 | 393.97 | 82.72 | 476.69 | 116.51 | 126.81 | 137.83 | 6.99 | 7.00 | 7.01 | 1612.52 | 2093.77 | 2163.97 | ✅ |
|
||||
| c1_i1024_o512 | 1 | 1024 | 512 | 49.68 | 20 | 0 | 0.40 | 412.23 | 78.90 | 491.13 | 132.68 | 174.48 | 175.39 | 7.00 | 7.02 | 7.02 | 1497.09 | 2052.55 | 2150.57 | ✅ |
|
||||
| c1_i131072_o128 | 1 | 131072 | 128 | 257.45 | 20 | 0 | 0.08 | 10182.44 | 8.90 | 10191.34 | 6471.48 | 6574.88 | 6590.68 | 7.21 | 7.25 | 7.32 | 7290.28 | 7408.33 | 7425.83 | ⚠️ |
|
||||
| c1_i131072_o256 | 1 | 131072 | 256 | 259.43 | 20 | 0 | 0.08 | 10104.51 | 13.11 | 10117.61 | 6523.80 | 6605.21 | 6683.36 | 7.25 | 7.28 | 7.30 | 7749.51 | 8208.24 | 8219.55 | ⚠️ |
|
||||
| c1_i131072_o512 | 1 | 131072 | 512 | 281.03 | 20 | 0 | 0.07 | 9328.01 | 15.40 | 9343.41 | 6520.17 | 6595.29 | 6639.39 | 7.26 | 7.27 | 7.29 | 8083.13 | 9930.97 | 10147.71 | ⚠️ |
|
||||
| c1_i16384_o1024 | 1 | 16384 | 1024 | 85.55 | 20 | 0 | 0.23 | 3830.19 | 66.12 | 3896.31 | 761.59 | 767.17 | 767.57 | 7.09 | 7.11 | 7.11 | 2761.83 | 4275.72 | 5529.53 | ✅ |
|
||||
| c1_i16384_o128 | 1 | 16384 | 128 | 54.06 | 20 | 0 | 0.37 | 6061.09 | 46.98 | 6108.07 | 758.98 | 762.04 | 764.60 | 7.03 | 7.04 | 7.04 | 1645.31 | 1649.67 | 1651.54 | ✅ |
|
||||
| c1_i16384_o2048 | 1 | 16384 | 2048 | 90.20 | 20 | 0 | 0.22 | 3632.96 | 62.71 | 3695.67 | 762.25 | 769.13 | 769.49 | 7.10 | 7.11 | 7.11 | 2762.54 | 3858.26 | 4063.08 | ✅ |
|
||||
| c1_i16384_o256 | 1 | 16384 | 256 | 73.13 | 20 | 0 | 0.27 | 4480.86 | 56.19 | 4537.04 | 757.24 | 769.18 | 772.94 | 7.09 | 7.11 | 7.11 | 2205.64 | 2558.68 | 2568.89 | ✅ |
|
||||
| c1_i16384_o512 | 1 | 16384 | 512 | 75.43 | 20 | 0 | 0.27 | 4344.23 | 53.96 | 4398.19 | 761.98 | 768.30 | 778.23 | 7.10 | 7.11 | 7.11 | 2198.96 | 2832.27 | 2889.46 | ✅ |
|
||||
| c1_i2048_o1024 | 1 | 2048 | 1024 | 52.15 | 20 | 0 | 0.38 | 785.38 | 81.93 | 867.31 | 135.15 | 140.23 | 150.82 | 7.07 | 7.09 | 7.10 | 1639.00 | 2263.67 | 3129.84 | ✅ |
|
||||
| c1_i2048_o128 | 1 | 2048 | 128 | 33.24 | 20 | 0 | 0.60 | 1232.22 | 74.97 | 1307.19 | 136.94 | 159.17 | 181.96 | 7.04 | 7.10 | 7.10 | 1006.62 | 1053.40 | 1075.84 | ✅ |
|
||||
| c1_i2048_o2048 | 1 | 2048 | 2048 | 51.48 | 20 | 0 | 0.39 | 795.71 | 84.60 | 880.31 | 134.55 | 137.97 | 148.69 | 7.07 | 7.09 | 7.10 | 1667.40 | 3179.96 | 3366.92 | ✅ |
|
||||
| c1_i2048_o256 | 1 | 2048 | 256 | 48.45 | 20 | 0 | 0.41 | 845.46 | 79.01 | 924.47 | 133.36 | 134.84 | 138.71 | 7.07 | 7.08 | 7.09 | 1479.65 | 1930.94 | 1931.75 | ✅ |
|
||||
| c1_i2048_o4096 | 1 | 2048 | 4096 | 52.65 | 20 | 0 | 0.38 | 777.92 | 77.54 | 855.46 | 134.26 | 137.78 | 149.15 | 7.08 | 7.09 | 7.10 | 1571.48 | 2033.68 | 2095.42 | ✅ |
|
||||
| c1_i2048_o512 | 1 | 2048 | 512 | 50.39 | 20 | 0 | 0.40 | 812.88 | 79.10 | 891.98 | 133.86 | 135.40 | 147.37 | 7.08 | 7.10 | 7.10 | 1537.04 | 2769.40 | 2894.64 | ✅ |
|
||||
| c1_i262144_o256 | 1 | 262144 | 256 | 556.90 | 20 | 0 | 0.04 | 9414.44 | 6.27 | 9420.71 | 15234.61 | 15316.53 | 15320.27 | 7.47 | 7.55 | 7.57 | 16530.72 | 17025.05 | 17032.91 | ⚠️ |
|
||||
| c1_i4096_o1024 | 1 | 4096 | 1024 | 50.77 | 20 | 0 | 0.39 | 1613.53 | 78.27 | 1691.81 | 223.98 | 228.84 | 229.98 | 7.08 | 7.09 | 7.09 | 1622.74 | 2090.63 | 2641.63 | ✅ |
|
||||
| c1_i4096_o128 | 1 | 4096 | 128 | 36.18 | 20 | 0 | 0.55 | 2264.03 | 70.17 | 2334.20 | 223.37 | 225.71 | 227.21 | 7.03 | 7.04 | 7.07 | 1108.62 | 1117.89 | 1119.93 | ✅ |
|
||||
| c1_i4096_o2048 | 1 | 4096 | 2048 | 54.09 | 20 | 0 | 0.37 | 1514.64 | 78.34 | 1592.98 | 223.30 | 226.85 | 227.97 | 7.08 | 7.09 | 7.09 | 1715.15 | 2736.41 | 2965.24 | ✅ |
|
||||
| c1_i4096_o256 | 1 | 4096 | 256 | 53.93 | 20 | 0 | 0.37 | 1519.00 | 77.17 | 1596.18 | 224.11 | 227.58 | 229.35 | 7.07 | 7.09 | 7.09 | 1687.85 | 2022.93 | 2023.17 | ✅ |
|
||||
| c1_i4096_o4096 | 1 | 4096 | 4096 | 50.60 | 20 | 0 | 0.40 | 1618.96 | 72.98 | 1691.94 | 223.43 | 228.43 | 229.73 | 7.08 | 7.09 | 7.09 | 1522.86 | 2146.74 | 2282.49 | ✅ |
|
||||
| c1_i4096_o512 | 1 | 4096 | 512 | 59.75 | 20 | 0 | 0.33 | 1371.10 | 81.14 | 1452.25 | 224.35 | 229.07 | 230.84 | 7.07 | 7.08 | 7.08 | 1931.53 | 2977.28 | 3648.36 | ✅ |
|
||||
| c1_i512_o256 | 1 | 512 | 256 | 53.26 | 20 | 0 | 0.38 | 192.26 | 80.81 | 273.07 | 167.28 | 203.13 | 205.33 | 6.99 | 7.02 | 7.03 | 1663.97 | 1951.39 | 1953.26 | ✅ |
|
||||
| c1_i524288_o128 | 1 | 524288 | 128 | 1513.75 | 20 | 0 | 0.01 | 6927.01 | 1.20 | 6928.22 | 40602.84 | 41121.70 | 41219.49 | 7.78 | 7.92 | 7.93 | 41303.57 | 41857.14 | 41958.45 | ⚠️ |
|
||||
| c1_i65536_o1024 | 1 | 65536 | 1024 | 152.81 | 20 | 0 | 0.13 | 8577.64 | 26.28 | 8603.92 | 2984.49 | 2994.56 | 3006.30 | 7.18 | 7.18 | 7.19 | 4418.27 | 5282.47 | 5593.15 | ✅ |
|
||||
| c1_i65536_o128 | 1 | 65536 | 128 | 134.86 | 20 | 0 | 0.15 | 9718.82 | 17.89 | 9736.71 | 2996.65 | 3058.30 | 3078.26 | 7.11 | 7.16 | 7.16 | 3847.83 | 3911.31 | 3932.26 | ⚠️ |
|
||||
| c1_i65536_o256 | 1 | 65536 | 256 | 153.56 | 20 | 0 | 0.13 | 8535.68 | 25.96 | 8561.64 | 2986.14 | 2994.62 | 3008.90 | 7.16 | 7.18 | 7.21 | 4407.24 | 4762.51 | 4778.90 | ✅ |
|
||||
| c1_i65536_o512 | 1 | 65536 | 512 | 157.16 | 20 | 0 | 0.13 | 8340.12 | 27.70 | 8367.82 | 2986.06 | 2996.34 | 3023.15 | 7.18 | 7.19 | 7.20 | 4541.31 | 6107.66 | 6492.71 | ✅ |
|
||||
| c2_i512_o256 | 2 | 512 | 256 | 44.94 | 40 | 0 | 0.89 | 455.73 | 188.32 | 644.06 | 123.41 | 177.91 | 187.26 | 7.06 | 7.48 | 7.68 | 1611.58 | 2020.21 | 2050.24 | ✅ |
|
||||
| c32_i1024_o1024 | 32 | 1024 | 1024 | 78.93 | 640 | 0 | 8.11 | 8303.55 | 1684.20 | 9987.75 | 154.93 | 217.27 | 815.30 | 16.71 | 18.61 | 18.94 | 3611.27 | 4919.69 | 5510.54 | ✅ |
|
||||
| c32_i1024_o128 | 32 | 1024 | 128 | 46.47 | 640 | 0 | 13.77 | 14103.72 | 1762.90 | 15866.62 | 408.25 | 714.24 | 757.98 | 13.41 | 16.99 | 17.83 | 2110.85 | 2431.17 | 2650.42 | ✅ |
|
||||
| c32_i1024_o2048 | 32 | 1024 | 2048 | 81.00 | 640 | 0 | 7.90 | 8090.92 | 1622.21 | 9713.13 | 162.95 | 334.37 | 753.35 | 17.45 | 23.66 | 25.29 | 3730.41 | 5542.76 | 6244.95 | ✅ |
|
||||
| c32_i1024_o256 | 32 | 1024 | 256 | 76.79 | 640 | 0 | 8.33 | 8534.95 | 1672.11 | 10207.06 | 154.72 | 212.02 | 753.62 | 16.97 | 19.11 | 20.00 | 3544.98 | 4746.50 | 4887.24 | ✅ |
|
||||
| c32_i1024_o4096 | 32 | 1024 | 4096 | 77.85 | 640 | 0 | 8.22 | 8418.72 | 1664.78 | 10083.50 | 154.28 | 214.19 | 756.53 | 16.85 | 18.82 | 19.49 | 3549.62 | 4954.17 | 5479.06 | ✅ |
|
||||
| c32_i1024_o512 | 32 | 1024 | 512 | 77.73 | 640 | 0 | 8.23 | 8431.17 | 1700.51 | 10131.68 | 155.17 | 216.90 | 751.85 | 16.62 | 18.26 | 18.67 | 3574.12 | 4831.87 | 5478.82 | ✅ |
|
||||
| c32_i16384_o1024 | 32 | 16384 | 1024 | 277.96 | 640 | 0 | 2.30 | 37723.90 | 563.76 | 38287.65 | 1266.27 | 1984.25 | 8777.02 | 48.75 | 60.64 | 65.85 | 13073.64 | 26643.75 | 35597.74 | ✅ |
|
||||
| c32_i16384_o128 | 32 | 16384 | 128 | 240.94 | 640 | 0 | 2.66 | 43520.72 | 338.90 | 43859.62 | 3738.21 | 4848.25 | 8794.30 | 61.61 | 81.93 | 82.76 | 11539.01 | 13150.69 | 16925.01 | ❌ |
|
||||
| c32_i16384_o2048 | 32 | 16384 | 2048 | 277.48 | 640 | 0 | 2.31 | 37789.10 | 576.94 | 38366.04 | 1268.99 | 2250.82 | 8796.37 | 47.57 | 58.28 | 64.83 | 13129.88 | 24681.93 | 38211.25 | ✅ |
|
||||
| c32_i16384_o256 | 32 | 16384 | 256 | 264.70 | 640 | 0 | 2.42 | 39614.37 | 508.66 | 40123.04 | 1298.03 | 1994.66 | 8761.72 | 54.06 | 64.54 | 70.67 | 12625.00 | 16418.68 | 19908.81 | ⚠️ |
|
||||
| c32_i16384_o512 | 32 | 16384 | 512 | 275.93 | 640 | 0 | 2.32 | 38001.51 | 561.14 | 38562.65 | 1268.67 | 1990.86 | 8787.03 | 48.89 | 60.15 | 66.01 | 13034.17 | 25370.34 | 27540.55 | ✅ |
|
||||
| c32_i2048_o1024 | 32 | 2048 | 1024 | 87.95 | 640 | 0 | 7.28 | 14903.68 | 1574.85 | 16478.54 | 200.89 | 270.09 | 1352.10 | 17.77 | 19.91 | 20.51 | 4023.46 | 6506.86 | 8623.71 | ✅ |
|
||||
| c32_i2048_o128 | 32 | 2048 | 128 | 59.24 | 640 | 0 | 10.80 | 22125.82 | 1375.94 | 23501.76 | 367.53 | 672.56 | 1352.00 | 18.71 | 21.55 | 22.09 | 2732.47 | 3094.92 | 3404.73 | ✅ |
|
||||
| c32_i2048_o2048 | 32 | 2048 | 2048 | 89.13 | 640 | 0 | 7.18 | 14705.49 | 1556.10 | 16261.59 | 199.91 | 291.10 | 1361.13 | 17.87 | 20.28 | 20.98 | 4056.53 | 6825.74 | 9339.78 | ✅ |
|
||||
| c32_i2048_o256 | 32 | 2048 | 256 | 83.40 | 640 | 0 | 7.67 | 15715.67 | 1544.12 | 17259.80 | 202.63 | 380.57 | 1355.98 | 18.19 | 20.18 | 20.73 | 3838.39 | 5110.33 | 5339.24 | ✅ |
|
||||
| c32_i2048_o4096 | 32 | 2048 | 4096 | 88.69 | 640 | 0 | 7.22 | 14778.01 | 1520.45 | 16298.46 | 199.19 | 288.11 | 1360.76 | 18.06 | 20.41 | 20.84 | 3982.98 | 6383.30 | 9405.39 | ✅ |
|
||||
| c32_i2048_o512 | 32 | 2048 | 512 | 90.84 | 640 | 0 | 7.05 | 14428.41 | 1530.32 | 15958.73 | 197.17 | 253.30 | 1349.46 | 17.88 | 20.25 | 21.02 | 4060.40 | 6700.48 | 9080.73 | ✅ |
|
||||
| c32_i4096_o1024 | 32 | 4096 | 1024 | 113.81 | 640 | 0 | 5.62 | 23032.89 | 1212.69 | 24245.58 | 339.67 | 542.37 | 2422.62 | 22.72 | 26.10 | 27.21 | 5215.72 | 8904.34 | 13457.62 | ✅ |
|
||||
| c32_i4096_o128 | 32 | 4096 | 128 | 85.66 | 640 | 0 | 7.47 | 30604.61 | 950.87 | 31555.48 | 556.18 | 1180.23 | 2449.13 | 27.29 | 31.52 | 34.82 | 4002.80 | 4613.66 | 5507.05 | ✅ |
|
||||
| c32_i4096_o2048 | 32 | 4096 | 2048 | 114.27 | 640 | 0 | 5.60 | 22939.90 | 1189.49 | 24129.39 | 341.11 | 550.60 | 2435.46 | 22.78 | 26.02 | 27.73 | 5165.78 | 9243.12 | 13018.90 | ✅ |
|
||||
| c32_i4096_o256 | 32 | 4096 | 256 | 106.21 | 640 | 0 | 6.03 | 24681.91 | 1178.61 | 25860.52 | 350.79 | 564.37 | 2447.99 | 23.55 | 26.71 | 28.32 | 4937.54 | 6734.16 | 7405.49 | ✅ |
|
||||
| c32_i4096_o4096 | 32 | 4096 | 4096 | 114.94 | 640 | 0 | 5.57 | 22806.49 | 1192.22 | 23998.71 | 341.86 | 550.55 | 2448.39 | 22.83 | 26.16 | 27.23 | 5203.63 | 8545.93 | 12625.46 | ✅ |
|
||||
| c32_i4096_o512 | 32 | 4096 | 512 | 112.20 | 640 | 0 | 5.70 | 23364.93 | 1200.40 | 24565.33 | 340.20 | 560.25 | 2385.06 | 22.99 | 26.29 | 27.73 | 5150.64 | 8529.09 | 12164.94 | ✅ |
|
||||
| c32_i512_o256 | 32 | 512 | 256 | 78.27 | 640 | 0 | 8.18 | 4186.51 | 1690.88 | 5877.39 | 149.00 | 289.93 | 473.49 | 16.81 | 19.27 | 22.36 | 3610.39 | 4786.53 | 5755.87 | ✅ |
|
||||
| c32_i65536_o1024 | 32 | 65536 | 1024 | 266.52 | 160 | 0 | 0.60 | 39343.10 | 131.31 | 39474.42 | 8656.40 | 33860.81 | 43057.06 | 171.14 | 244.27 | 281.95 | 46303.99 | 103438.83 | 127750.75 | ❌ |
|
||||
| c4_i512_o256 | 4 | 512 | 256 | 43.40 | 80 | 0 | 1.84 | 943.85 | 379.18 | 1323.03 | 118.07 | 125.54 | 241.46 | 7.83 | 8.29 | 8.68 | 1720.27 | 2165.16 | 2253.32 | ✅ |
|
||||
| c64_i512_o256 | 64 | 512 | 256 | 109.33 | 1280 | 0 | 11.71 | 5994.26 | 2402.36 | 8396.62 | 171.45 | 299.06 | 771.51 | 24.39 | 26.72 | 27.58 | 5151.60 | 6720.29 | 6878.42 | ✅ |
|
||||
| c8_i1024_o1024 | 8 | 1024 | 1024 | 47.80 | 160 | 0 | 3.35 | 3427.67 | 696.10 | 4123.77 | 129.12 | 192.22 | 281.04 | 9.27 | 9.95 | 10.24 | 2052.01 | 2788.77 | 3052.14 | ✅ |
|
||||
| c8_i1024_o128 | 8 | 1024 | 128 | 31.03 | 160 | 0 | 5.16 | 5279.38 | 659.92 | 5939.30 | 187.55 | 269.21 | 292.69 | 8.88 | 9.90 | 9.97 | 1315.49 | 1387.68 | 1458.94 | ✅ |
|
||||
| c8_i1024_o2048 | 8 | 1024 | 2048 | 47.63 | 160 | 0 | 3.36 | 3439.84 | 684.76 | 4124.59 | 139.64 | 191.21 | 283.85 | 9.49 | 10.93 | 11.77 | 2064.28 | 2867.60 | 3177.60 | ✅ |
|
||||
| c8_i1024_o256 | 8 | 1024 | 256 | 47.68 | 160 | 0 | 3.36 | 3436.34 | 684.94 | 4121.28 | 134.22 | 195.34 | 290.76 | 9.44 | 10.83 | 11.66 | 2050.72 | 2669.20 | 2870.35 | ✅ |
|
||||
| c8_i1024_o4096 | 8 | 1024 | 4096 | 48.02 | 160 | 0 | 3.33 | 3411.81 | 683.42 | 4095.23 | 132.59 | 194.92 | 285.39 | 9.39 | 10.45 | 11.05 | 2050.12 | 2647.51 | 2950.32 | ✅ |
|
||||
| c8_i1024_o512 | 8 | 1024 | 512 | 48.89 | 160 | 0 | 3.27 | 3351.35 | 699.85 | 4051.20 | 128.43 | 130.82 | 286.17 | 9.25 | 9.96 | 10.23 | 2097.31 | 2846.33 | 3059.68 | ✅ |
|
||||
| c8_i131072_o512 | 8 | 131072 | 512 | 593.71 | 160 | 0 | 0.27 | 35322.95 | 49.97 | 35372.92 | 8837.26 | 12856.65 | 21274.84 | 95.98 | 174.74 | 250.31 | 26223.81 | 43791.96 | 52427.10 | ❌ |
|
||||
| c8_i16384_o1024 | 8 | 16384 | 1024 | 117.03 | 160 | 0 | 1.37 | 22399.35 | 381.47 | 22780.82 | 880.55 | 1325.49 | 2519.24 | 15.14 | 18.88 | 22.89 | 5043.96 | 9664.45 | 11942.89 | ✅ |
|
||||
| c8_i16384_o128 | 8 | 16384 | 128 | 83.84 | 160 | 0 | 1.91 | 31265.56 | 241.22 | 31506.78 | 1766.89 | 2637.47 | 2676.54 | 14.95 | 23.25 | 26.08 | 3640.27 | 3713.62 | 5332.84 | ✅ |
|
||||
| c8_i16384_o2048 | 8 | 16384 | 2048 | 111.65 | 160 | 0 | 1.43 | 23479.38 | 353.00 | 23832.38 | 858.33 | 1105.07 | 2515.80 | 15.80 | 19.89 | 21.08 | 4778.32 | 8910.12 | 13501.67 | ✅ |
|
||||
| c8_i16384_o256 | 8 | 16384 | 256 | 101.77 | 160 | 0 | 1.57 | 25759.71 | 329.17 | 26088.88 | 867.77 | 1267.33 | 2511.92 | 17.06 | 21.93 | 23.13 | 4427.86 | 5958.68 | 6667.56 | ✅ |
|
||||
| c8_i16384_o512 | 8 | 16384 | 512 | 109.64 | 160 | 0 | 1.46 | 23908.84 | 355.69 | 24264.53 | 908.35 | 1361.76 | 2515.33 | 15.82 | 20.35 | 22.15 | 4755.66 | 8684.67 | 10122.99 | ✅ |
|
||||
| c8_i2048_o1024 | 8 | 2048 | 1024 | 53.11 | 160 | 0 | 3.01 | 6170.36 | 677.84 | 6848.20 | 150.15 | 153.04 | 410.55 | 9.49 | 10.24 | 10.65 | 2286.09 | 3870.04 | 5178.54 | ✅ |
|
||||
| c8_i2048_o128 | 8 | 2048 | 128 | 33.35 | 160 | 0 | 4.80 | 9826.22 | 612.40 | 10438.62 | 228.93 | 388.77 | 404.46 | 9.33 | 10.33 | 11.02 | 1411.23 | 1530.95 | 1595.88 | ✅ |
|
||||
| c8_i2048_o2048 | 8 | 2048 | 2048 | 51.12 | 160 | 0 | 3.13 | 6409.90 | 672.70 | 7082.60 | 150.70 | 165.44 | 418.00 | 9.56 | 10.25 | 10.68 | 2202.49 | 3406.00 | 4277.62 | ✅ |
|
||||
| c8_i2048_o256 | 8 | 2048 | 256 | 48.03 | 160 | 0 | 3.33 | 6822.11 | 658.85 | 7480.96 | 148.57 | 147.18 | 403.31 | 9.66 | 10.48 | 10.98 | 2051.77 | 2715.99 | 2745.12 | ✅ |
|
||||
| c8_i2048_o4096 | 8 | 2048 | 4096 | 53.12 | 160 | 0 | 3.01 | 6168.50 | 661.05 | 6829.55 | 150.99 | 219.64 | 421.18 | 9.58 | 10.44 | 10.58 | 2250.24 | 3788.73 | 4649.51 | ✅ |
|
||||
| c8_i2048_o512 | 8 | 2048 | 512 | 50.97 | 160 | 0 | 3.14 | 6428.69 | 669.04 | 7097.73 | 148.41 | 147.71 | 403.41 | 9.60 | 10.48 | 11.08 | 2190.12 | 3392.37 | 4269.13 | ✅ |
|
||||
| c8_i262144_o128 | 8 | 262144 | 128 | 1333.56 | 160 | 0 | 0.12 | 31451.94 | 12.61 | 31464.55 | 36128.45 | 57997.25 | 58592.27 | 215.95 | 392.14 | 446.94 | 59143.65 | 61777.27 | 74358.75 | ❌ |
|
||||
| c8_i4096_o1024 | 8 | 4096 | 1024 | 60.18 | 160 | 0 | 2.66 | 10890.64 | 578.38 | 11469.02 | 253.10 | 364.29 | 731.93 | 10.57 | 11.89 | 12.17 | 2550.26 | 4642.49 | 7535.89 | ✅ |
|
||||
| c8_i4096_o128 | 8 | 4096 | 128 | 40.35 | 160 | 0 | 3.96 | 16240.24 | 506.02 | 16746.26 | 376.51 | 695.40 | 727.26 | 10.62 | 11.99 | 13.73 | 1721.35 | 1910.10 | 2181.08 | ✅ |
|
||||
| c8_i4096_o2048 | 8 | 4096 | 2048 | 57.14 | 160 | 0 | 2.80 | 11470.28 | 593.22 | 12063.50 | 254.35 | 361.27 | 732.14 | 10.52 | 11.68 | 12.36 | 2478.80 | 3843.55 | 5407.45 | ✅ |
|
||||
| c8_i4096_o256 | 8 | 4096 | 256 | 54.73 | 160 | 0 | 2.92 | 11973.53 | 575.24 | 12548.76 | 252.76 | 366.29 | 732.21 | 10.63 | 12.02 | 12.59 | 2342.56 | 3122.99 | 3247.98 | ✅ |
|
||||
| c8_i4096_o4096 | 8 | 4096 | 4096 | 61.97 | 160 | 0 | 2.58 | 10574.93 | 602.60 | 11177.53 | 251.62 | 276.31 | 733.83 | 10.48 | 11.81 | 12.23 | 2715.50 | 3797.26 | 5487.12 | ✅ |
|
||||
| c8_i4096_o512 | 8 | 4096 | 512 | 57.59 | 160 | 0 | 2.78 | 11379.47 | 586.75 | 11966.22 | 255.02 | 376.23 | 733.95 | 10.54 | 11.70 | 12.17 | 2482.90 | 4048.32 | 5636.75 | ✅ |
|
||||
| c8_i512_o256 | 8 | 512 | 256 | 48.34 | 160 | 0 | 3.31 | 1694.53 | 687.00 | 2381.52 | 125.68 | 194.69 | 258.22 | 9.43 | 10.43 | 10.95 | 2075.15 | 2669.68 | 2871.40 | ✅ |
|
||||
| c8_i65536_o1024 | 8 | 65536 | 1024 | 298.66 | 160 | 0 | 0.54 | 35109.32 | 107.56 | 35216.88 | 3847.52 | 5884.53 | 9844.52 | 46.16 | 82.22 | 101.31 | 13001.39 | 21336.69 | 30533.51 | ⚠️ |
|
||||
| c8_i65536_o128 | 8 | 65536 | 128 | 274.11 | 160 | 0 | 0.58 | 38254.29 | 70.30 | 38324.59 | 7223.66 | 11101.75 | 11262.28 | 40.50 | 72.54 | 72.63 | 12089.84 | 12347.21 | 12881.32 | ⚠️ |
|
||||
| c8_i65536_o256 | 8 | 65536 | 256 | 291.01 | 160 | 0 | 0.55 | 36032.88 | 98.87 | 36131.75 | 3804.89 | 5718.15 | 9852.36 | 50.92 | 78.36 | 96.49 | 12843.24 | 18584.33 | 21293.29 | ❌ |
|
||||
| c8_i65536_o512 | 8 | 65536 | 512 | 297.04 | 160 | 0 | 0.54 | 35301.20 | 111.90 | 35413.10 | 3902.29 | 6055.87 | 9835.60 | 44.23 | 73.93 | 83.82 | 13139.03 | 22983.14 | 31590.46 | ⚠️ |
|
||||
|
||||
SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
278
experiments/dsv4_h200_vllm_tp4_custom_bench/run_bench.sh
Executable file
278
experiments/dsv4_h200_vllm_tp4_custom_bench/run_bench.sh
Executable file
@ -0,0 +1,278 @@
|
||||
#!/usr/bin/env bash
|
||||
# Custom benchmark orchestrator for multi-service vLLM TP=4 on 8x H200.
|
||||
#
|
||||
# This runs 2 independent vLLM services (TP=4 each) on GPU groups (0,1,2,3) and (4,5,6,7).
|
||||
# bench_client.py distributes requests across all 2 services via round-robin LB,
|
||||
# giving accurate cluster-wide throughput even at low per-service concurrency.
|
||||
#
|
||||
# Usage:
|
||||
# bash run_bench.sh
|
||||
# SKIP_MANAGE_SERVER=1 bash run_bench.sh # skip server start/stop
|
||||
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||
RESULT_ROOT="${RESULT_ROOT:-${SCRIPT_DIR}/results/${RUN_ID}}"
|
||||
RAW_DIR="${RESULT_ROOT}/raw_outputs"
|
||||
LOG_DIR="${RESULT_ROOT}/logs"
|
||||
|
||||
ensure_result_root "$RESULT_ROOT"
|
||||
log_init "${LOG_DIR}/orchestrator.log"
|
||||
|
||||
log "experiment=${EXPERIMENT_NAME}"
|
||||
log "run_id=${RUN_ID}"
|
||||
log "result_root=${RESULT_ROOT}"
|
||||
log "platform=${PLATFORM}"
|
||||
log "chip=${CHIP}"
|
||||
log "accelerator=${ACCELERATOR}"
|
||||
log "engine=${ENGINE}"
|
||||
log "hardware=${HARDWARE}"
|
||||
log "model=${MODEL_PATH}"
|
||||
log "services=${PORTS}"
|
||||
log "lb_strategy=${LB_STRATEGY}"
|
||||
log "server_start_script=${SERVER_START_SCRIPT}"
|
||||
|
||||
# Write initial metadata for this run.
|
||||
METADATA_JSON="${RESULT_ROOT}/results.json"
|
||||
write_metadata_json \
|
||||
"$METADATA_JSON" \
|
||||
"$EXPERIMENT_NAME" \
|
||||
"$RUN_ID" \
|
||||
"$MODEL_PATH" \
|
||||
"$BACKEND" \
|
||||
"$ENGINE" \
|
||||
"$HARDWARE" \
|
||||
"$ACCELERATOR" \
|
||||
"$CHIP" \
|
||||
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||
"$VENV_SERVER" \
|
||||
"H200 2x vLLM TP=4 multi-service benchmark using bench_client.py"
|
||||
|
||||
# Update metadata with config.
|
||||
"${VENV_CLIENT}/bin/python" - "$METADATA_JSON" <<'PY'
|
||||
import json, sys
|
||||
path = sys.argv[1]
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
data["config"] = {
|
||||
"tp": 4,
|
||||
"num_services": 2,
|
||||
"ports": "30005,30006",
|
||||
"gpu_groups": "0,1,2,3|4,5,6,7",
|
||||
"kv_cache_dtype": "fp8",
|
||||
"block_size": 256,
|
||||
"max_num_seqs": 256,
|
||||
"client": "bench_client.py",
|
||||
"lb_strategy": "round_robin",
|
||||
"scenarios": [
|
||||
"1 512 256 20", "2 512 256 40", "4 512 256 80",
|
||||
"8 512 256 160", "16 512 256 320", "32 512 256 640",
|
||||
"64 512 256 1280", "128 512 256 2560",
|
||||
"1 2048 512 20", "8 2048 512 160", "32 2048 512 640", "128 2048 512 2560",
|
||||
"1 4096 1024 20", "8 4096 1024 160", "32 4096 1024 640", "128 4096 1024 2560"
|
||||
]
|
||||
}
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
PY
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server helpers (multi-service)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
is_server_healthy() {
|
||||
local port="$1"
|
||||
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${port}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
are_all_services_healthy() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if ! is_server_healthy "$p"; then
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
stop_all_servers() {
|
||||
log "stopping all vllm services"
|
||||
# Kill by PID files first.
|
||||
for pid_file in "${SCRIPT_DIR}"/server_port*.pid; do
|
||||
[[ -f "$pid_file" ]] || continue
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "killing pid=${pid}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 2
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
done
|
||||
# Fallback: kill any remaining vllm processes for this model.
|
||||
pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
sleep 2
|
||||
}
|
||||
|
||||
start_all_servers() {
|
||||
log "starting all vllm services with ${SERVER_START_SCRIPT}"
|
||||
if [[ ! -x "${SERVER_START_SCRIPT}" ]]; then
|
||||
log "error: start script not found or not executable: ${SERVER_START_SCRIPT}"
|
||||
exit 1
|
||||
fi
|
||||
bash "${SERVER_START_SCRIPT}" >> "${LOG_DIR}/server.outer.log" 2>&1
|
||||
|
||||
if ! are_all_services_healthy "$PORTS"; then
|
||||
log "error: not all vllm services became healthy"
|
||||
exit 1
|
||||
fi
|
||||
log "all vllm services are healthy"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pre-benchmark health check with detailed reporting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
check_services_detailed() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
local healthy_count=0
|
||||
local unhealthy_ports=""
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if is_server_healthy "$p"; then
|
||||
((healthy_count++))
|
||||
else
|
||||
unhealthy_ports="${unhealthy_ports}${p} "
|
||||
fi
|
||||
done
|
||||
echo "$healthy_count"
|
||||
if [[ -n "$unhealthy_ports" ]]; then
|
||||
echo "unhealthy: ${unhealthy_ports}" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
wait_for_all_services() {
|
||||
local ports="$1"
|
||||
local max_wait="${2:-240}"
|
||||
local interval="${3:-5}"
|
||||
local elapsed=0
|
||||
while (( elapsed < max_wait )); do
|
||||
local healthy_count
|
||||
healthy_count="$(check_services_detailed "$ports" | head -1)"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
if (( healthy_count == ${#PORT_LIST[@]} )); then
|
||||
return 0
|
||||
fi
|
||||
log "waiting for services... ${healthy_count}/${#PORT_LIST[@]} healthy after ${elapsed}s"
|
||||
sleep "$interval"
|
||||
((elapsed += interval))
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mid-benchmark health check (called between scenarios)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
check_and_report_services() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
local healthy=()
|
||||
local unhealthy=()
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if is_server_healthy "$p"; then
|
||||
healthy+=("$p")
|
||||
else
|
||||
unhealthy+=("$p")
|
||||
fi
|
||||
done
|
||||
log "service health check: healthy=[${healthy[*]}] unhealthy=[${unhealthy[*]}]"
|
||||
if [[ ${#unhealthy[@]} -gt 0 ]]; then
|
||||
log "WARNING: ${#unhealthy[@]} service(s) unhealthy: ${unhealthy[*]}"
|
||||
fi
|
||||
return ${#unhealthy[@]}
|
||||
}
|
||||
|
||||
on_exit() {
|
||||
local code=$?
|
||||
log "orchestrator exiting with code=${code}"
|
||||
if [[ -z "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||
stop_all_servers
|
||||
fi
|
||||
exit "$code"
|
||||
}
|
||||
trap on_exit EXIT
|
||||
|
||||
if [[ -n "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||
log "SKIP_MANAGE_SERVER is set, assuming services are already running"
|
||||
if ! wait_for_all_services "$PORTS"; then
|
||||
log "error: not all healthy services found on ports ${PORTS}"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
stop_all_servers
|
||||
start_all_servers
|
||||
fi
|
||||
|
||||
log "===== BENCHMARK START ====="
|
||||
|
||||
for scenario in "${SCENARIOS[@]}"; do
|
||||
read -r concurrency input_len output_len num_prompts <<< "$scenario"
|
||||
# Fallback for legacy 3-field scenarios.
|
||||
if [[ -z "${num_prompts:-}" ]]; then
|
||||
num_prompts="$NUM_PROMPTS"
|
||||
fi
|
||||
|
||||
output_file="${RAW_DIR}/vllm_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||
detail_log="${LOG_DIR}/vllm_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||
|
||||
log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len} num_prompts=${num_prompts}"
|
||||
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/bench_client.py" \
|
||||
--host 127.0.0.1 \
|
||||
--ports "$PORTS" \
|
||||
--model "$SERVED_MODEL_NAME" \
|
||||
--concurrency "$concurrency" \
|
||||
--input-len "$input_len" \
|
||||
--output-len "$output_len" \
|
||||
--num-prompts "$num_prompts" \
|
||||
--lb-strategy "$LB_STRATEGY" \
|
||||
--health-check-interval 5 \
|
||||
--health-max-failures 2 \
|
||||
--health-recovery-interval 10 \
|
||||
--request-max-retries 2 \
|
||||
--output-file "$output_file" \
|
||||
> "$detail_log" 2>&1 || {
|
||||
log "ERROR: scenario c=${concurrency} i=${input_len} o=${output_len} failed; see ${detail_log}"
|
||||
# After failure, check service health before continuing.
|
||||
check_and_report_services "$PORTS"
|
||||
continue
|
||||
}
|
||||
|
||||
log "finished scenario: output=${output_file}"
|
||||
|
||||
# Between scenarios, report service health so we can spot degradation early.
|
||||
check_and_report_services "$PORTS" || true
|
||||
done
|
||||
|
||||
log "===== BENCHMARK DONE ====="
|
||||
|
||||
log "parsing results"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/parse_results.py" "$RESULT_ROOT" >> "${LOG_DIR}/parse.log" 2>&1 || {
|
||||
log "WARNING: parser failed; see ${LOG_DIR}/parse.log"
|
||||
}
|
||||
|
||||
log "all results saved to ${RESULT_ROOT}"
|
||||
108
experiments/dsv4_h200_vllm_tp4_custom_bench/start_server.sh
Executable file
108
experiments/dsv4_h200_vllm_tp4_custom_bench/start_server.sh
Executable file
@ -0,0 +1,108 @@
|
||||
#!/bin/bash
|
||||
# Start 2 independent vLLM TP=4 services on 8x H200.
|
||||
# Each service runs on 4 GPUs: (0,1,2,3) and (4,5,6,7).
|
||||
# This maximizes per-service throughput for higher-concurrency workloads.
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
VENV="${VENV_SERVER}"
|
||||
export PATH="$VENV/bin:$PATH"
|
||||
VLLM="$VENV/bin/vllm"
|
||||
|
||||
TP=4
|
||||
LOG_DIR="${SCRIPT_DIR}/logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
# 2 services, each on 4 GPUs.
|
||||
# Port and GPU mapping.
|
||||
SERVICES=(
|
||||
"30005:0,1,2,3"
|
||||
"30006:4,5,6,7"
|
||||
)
|
||||
|
||||
# Kill any existing vLLM processes for this model.
|
||||
pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
sleep 2
|
||||
|
||||
# Remove old PID files.
|
||||
rm -f "${SCRIPT_DIR}"/*.pid
|
||||
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
GPUS="${svc##*:}"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_$(date +%Y%m%d_%H%M%S).log"
|
||||
PID_FILE="${SCRIPT_DIR}/server_port${PORT}.pid"
|
||||
|
||||
echo "=== Starting service on port ${PORT}, GPUs ${GPUS} ==="
|
||||
echo "Log: ${LOG}"
|
||||
|
||||
CUDA_VISIBLE_DEVICES="${GPUS}" \
|
||||
nohup "$VLLM" serve "$MODEL_PATH" \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size "$TP" \
|
||||
--kv-cache-dtype fp8 \
|
||||
--block-size 256 \
|
||||
--served-model-name "$SERVED_MODEL_NAME" \
|
||||
--port "$PORT" \
|
||||
> "$LOG" 2>&1 &
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Waiting for all 2 services to become healthy..."
|
||||
|
||||
MAX_WAIT=240
|
||||
all_healthy=0
|
||||
for i in $(seq 1 $MAX_WAIT); do
|
||||
all_healthy=1
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
if ! curl -s "http://127.0.0.1:${PORT}/health" > /dev/null 2>&1; then
|
||||
all_healthy=0
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ "$all_healthy" -eq 1 ]]; then
|
||||
echo "All 2 services are healthy!"
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
echo " - http://127.0.0.1:${PORT}"
|
||||
done
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check if any process died early.
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
PID_FILE="${SCRIPT_DIR}/server_port${PORT}.pid"
|
||||
if [[ -f "$PID_FILE" ]]; then
|
||||
PID="$(cat "$PID_FILE")"
|
||||
if ! kill -0 "$PID" 2>/dev/null; then
|
||||
echo "ERROR: Service on port ${PORT} (PID ${PID}) exited early"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_*.log"
|
||||
tail -200 $LOG 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
if (( i % 10 == 0 )); then
|
||||
echo "Waiting... (${i}/${MAX_WAIT})"
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: Not all services became healthy after ${MAX_WAIT} retries"
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_*.log"
|
||||
echo "--- Last 50 lines of port ${PORT} ---"
|
||||
tail -50 $LOG 2>/dev/null || true
|
||||
done
|
||||
exit 1
|
||||
101
experiments/dsv4_h200_vllm_tp8_custom_bench/README.md
Normal file
101
experiments/dsv4_h200_vllm_tp8_custom_bench/README.md
Normal file
@ -0,0 +1,101 @@
|
||||
# dsv4_h200_vllm_tp8_custom_bench
|
||||
|
||||
## 目的
|
||||
|
||||
当 vLLM 使用 **TP=8**（Tensor Parallel = 8）部署时，单张 H200 上运行 **1 个服务**，独占全部 8 张 GPU：
|
||||
|
||||
| 服务 | 端口 | GPU 组 |
|
||||
|------|------|--------|
|
||||
| 1 | 30005 | 0, 1, 2, 3, 4, 5, 6, 7 |
|
||||
|
||||
TP=8 提供最大的单服务 tensor-parallel 宽度，适合极高并发、大 batch 场景。本实验使用 **自定义压测客户端 `bench_client.py`**，直接调用 OpenAI `/v1/chat/completions` API。
|
||||
|
||||
## 与多服务方案的区别
|
||||
|
||||
| 特性 | TP=2 (4 服务) | TP=4 (2 服务) | TP=8 (1 服务) |
|
||||
|------|---------------|---------------|---------------|
|
||||
| 服务数量 | 4 | 2 | 1 |
|
||||
| 每服务 GPU | 2 | 4 | 8 |
|
||||
| 单服务吞吐 | 较低 | 中等 | 最高 |
|
||||
| 集群总吞吐 | 相近 | 相近 | 相近 |
|
||||
| 适用场景 | 低并发、低延迟 | 中等并发 | 极高并发、大 batch |
|
||||
|
||||
## 文件说明
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `bench_client.py` | 自定义异步压测客户端（复用 TP=2/TP=4 版本） |
|
||||
| `config.env` | 实验配置（单服务、端口 30005） |
|
||||
| `run_bench.sh` | 编排脚本：启动 1 个服务 → 运行压测 → 解析结果 |
|
||||
| `start_server.sh` | 启动 1 个 vLLM TP=8 服务（占用全部 8 GPU） |
|
||||
| `parse_results.py` | 解析 bench_client.py 输出的 JSONL，生成 `results.json` + `report.md` |
|
||||
|
||||
## 快速运行
|
||||
|
||||
```bash
|
||||
# 完整流程（自动启动服务、压测、生成报告）
|
||||
bash experiments/dsv4_h200_vllm_tp8_custom_bench/run_bench.sh
|
||||
|
||||
# 跳过服务管理（已有服务在运行）
|
||||
SKIP_MANAGE_SERVER=1 bash experiments/dsv4_h200_vllm_tp8_custom_bench/run_bench.sh
|
||||
|
||||
# 单独运行压测客户端（单服务）
|
||||
python3 experiments/dsv4_h200_vllm_tp8_custom_bench/bench_client.py \
|
||||
--host 127.0.0.1 --port 30005 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 32 --input-len 512 --output-len 256 --num-prompts 640 \
|
||||
--output-file /tmp/test.jsonl
|
||||
|
||||
# 解析已有结果
|
||||
python3 experiments/dsv4_h200_vllm_tp8_custom_bench/parse_results.py \
|
||||
experiments/dsv4_h200_vllm_tp8_custom_bench/results/<RUN_ID>
|
||||
```
|
||||
|
||||
## 场景设计
|
||||
|
||||
本实验覆盖了从 **单并发** 到 **128 并发** 的梯度，以及不同输入/输出长度组合：
|
||||
|
||||
| 并发 | 输入长度 | 输出长度 | 请求数 | 说明 |
|
||||
|------|----------|----------|--------|------|
|
||||
| 1 | 512 | 256 | 20 | 单并发基线，测纯延迟 |
|
||||
| 2 | 512 | 256 | 40 | 低并发 |
|
||||
| 4 | 512 | 256 | 80 | 中低并发 |
|
||||
| 8 | 512 | 256 | 160 | 中等并发 |
|
||||
| 16 | 512 | 256 | 320 | 中高并发 |
|
||||
| 32 | 512 | 256 | 640 | 高并发 |
|
||||
| 64 | 512 | 256 | 1280 | 很高并发 |
|
||||
| 128 | 512 | 256 | 2560 | 极限并发，测吞吐上限 |
|
||||
| 1 | 2048 | 512 | 20 | 长输入基线 |
|
||||
| 8 | 2048 | 512 | 160 | 长输入中并发 |
|
||||
| 32 | 2048 | 512 | 640 | 长输入高并发 |
|
||||
| 128 | 2048 | 512 | 2560 | 长输入极限并发 |
|
||||
| 1 | 4096 | 1024 | 20 | 超长输入基线 |
|
||||
| 8 | 4096 | 1024 | 160 | 超长输入中并发 |
|
||||
| 32 | 4096 | 1024 | 640 | 超长输入高并发 |
|
||||
| 128 | 4096 | 1024 | 2560 | 超长输入极限并发 |
|
||||
|
||||
## 输出格式
|
||||
|
||||
同 TP=2/TP=4 版本，输出 JSONL 原始数据、`results.json` 结构化结果、`report.md` 人类可读报告。
|
||||
|
||||
## 单独启动/停止服务
|
||||
|
||||
```bash
|
||||
# 启动服务
|
||||
bash experiments/dsv4_h200_vllm_tp8_custom_bench/start_server.sh
|
||||
|
||||
# 停止（run_bench.sh 会自动停止，但也可以手动）
|
||||
port=30005
|
||||
pid_file="experiments/dsv4_h200_vllm_tp8_custom_bench/server_port${port}.pid"
|
||||
[[ -f "$pid_file" ]] && kill "$(cat "$pid_file")" 2>/dev/null || true
|
||||
rm -f "$pid_file"
|
||||
pkill -9 -f "vllm serve.*DeepSeek-V4-Flash" 2>/dev/null || true
|
||||
```
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 服务独占全部 8 张 GPU，确保无其他进程占用 GPU。
|
||||
- 模型文件路径：`/data/models/DeepSeek-V4-Flash`
|
||||
- TP=8 相比 TP=2/TP=4 单服务延迟最低，但无法通过多服务并行提升低并发下的集群利用率。
|
||||
- 健康检查依赖 `/health` 端点（vLLM 默认启用）。
|
||||
- 如果服务频繁崩溃，检查 GPU 显存是否充足（OOM 是最常见原因）。
|
||||
652
experiments/dsv4_h200_vllm_tp8_custom_bench/bench_client.py
Executable file
652
experiments/dsv4_h200_vllm_tp8_custom_bench/bench_client.py
Executable file
@ -0,0 +1,652 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Custom benchmark client for vLLM / OpenAI-compatible API with multi-service load balancing and health-check failover.
|
||||
|
||||
Sends concurrent requests to multiple inference servers and records per-request
|
||||
latency metrics (TTFT, TPOT, ITL, E2E) with fine-grained timing.
|
||||
|
||||
Designed for TP=2 on 8-GPU setups where 4 independent services run on
|
||||
(0,1), (2,3), (4,5), (6,7). Requests are distributed across services via
|
||||
round-robin or random load balancing. Unhealthy services are automatically
|
||||
skipped with periodic retry.
|
||||
|
||||
Usage:
|
||||
# Single service (backward compatible)
|
||||
python3 bench_client.py \
|
||||
--host 127.0.0.1 --port 30005 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 64 --input-len 512 --output-len 256 --num-prompts 320 \
|
||||
--output-file results.jsonl
|
||||
|
||||
# Multi-service (4 services on ports 30005-30008)
|
||||
python3 bench_client.py \
|
||||
--host 127.0.0.1 \
|
||||
--ports 30005,30006,30007,30008 \
|
||||
--model deepseek-v4-flash \
|
||||
--concurrency 64 --input-len 512 --output-len 256 --num-prompts 320 \
|
||||
--lb-strategy round_robin \
|
||||
--output-file results.jsonl
|
||||
|
||||
The output JSONL contains one object per request with raw timing data.
|
||||
A final summary line ("completed" field) aggregates all requests.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Any
|
||||
|
||||
# Try aiohttp first; fall back to urllib for stdlib-only environments.
|
||||
try:
|
||||
import aiohttp
|
||||
HAS_AIOHTTP = True
|
||||
except ImportError:
|
||||
HAS_AIOHTTP = False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data structures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class RequestResult:
|
||||
request_id: int
|
||||
input_tokens: int = 0
|
||||
output_tokens: int = 0
|
||||
first_token_time_ms: float = 0.0 # TTFT
|
||||
e2e_latency_ms: float = 0.0 # total wall time
|
||||
inter_token_latencies: list[float] = field(default_factory=list)
|
||||
success: bool = False
|
||||
error: str = ""
|
||||
target_port: int = 0 # which service handled this request
|
||||
retry_count: int = 0 # how many times this request was retried
|
||||
|
||||
@property
|
||||
def tpot_ms(self) -> float:
|
||||
"""Mean TPOT = (e2e - ttft) / (output_tokens - 1) if >1 token."""
|
||||
if self.output_tokens <= 1:
|
||||
return 0.0
|
||||
return (self.e2e_latency_ms - self.first_token_time_ms) / (self.output_tokens - 1)
|
||||
|
||||
@property
|
||||
def mean_itl_ms(self) -> float:
|
||||
if not self.inter_token_latencies:
|
||||
return 0.0
|
||||
return sum(self.inter_token_latencies) / len(self.inter_token_latencies)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
d = asdict(self)
|
||||
d["tpot_ms"] = self.tpot_ms
|
||||
d["mean_itl_ms"] = self.mean_itl_ms
|
||||
return d
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Health checker
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class HealthChecker:
|
||||
"""Periodically check service health and maintain a healthy-port list."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
host: str,
|
||||
ports: list[int],
|
||||
check_interval: float = 5.0,
|
||||
max_failures: int = 2,
|
||||
recovery_interval: float = 10.0,
|
||||
):
|
||||
self.host = host
|
||||
self.all_ports = ports
|
||||
self.check_interval = check_interval
|
||||
self.max_failures = max_failures
|
||||
self.recovery_interval = recovery_interval
|
||||
|
||||
# port -> consecutive failure count
|
||||
self._failures: dict[int, int] = {p: 0 for p in ports}
|
||||
# port -> last healthy timestamp
|
||||
self._last_healthy: dict[int, float] = {p: time.monotonic() for p in ports}
|
||||
self._lock = asyncio.Lock()
|
||||
self._task: asyncio.Task | None = None
|
||||
|
||||
async def _check_one(self, port: int, session: aiohttp.ClientSession | None) -> bool:
|
||||
url = f"http://{self.host}:{port}/health"
|
||||
try:
|
||||
if HAS_AIOHTTP and session is not None:
|
||||
async with session.get(url, timeout=aiohttp.ClientTimeout(total=5)) as resp:
|
||||
return resp.status == 200
|
||||
else:
|
||||
req = urllib.request.Request(url, method="GET")
|
||||
with urllib.request.urlopen(req, timeout=5) as resp:
|
||||
return resp.status == 200
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
async def _check_all(self, session: aiohttp.ClientSession | None) -> None:
|
||||
checks = [self._check_one(p, session) for p in self.all_ports]
|
||||
results = await asyncio.gather(*checks, return_exceptions=True)
|
||||
now = time.monotonic()
|
||||
async with self._lock:
|
||||
for port, ok in zip(self.all_ports, results):
|
||||
if isinstance(ok, Exception):
|
||||
ok = False
|
||||
if ok:
|
||||
self._failures[port] = 0
|
||||
self._last_healthy[port] = now
|
||||
else:
|
||||
self._failures[port] += 1
|
||||
|
||||
async def start(self, session: aiohttp.ClientSession | None) -> None:
|
||||
async def _loop() -> None:
|
||||
while True:
|
||||
await asyncio.sleep(self.check_interval)
|
||||
await self._check_all(session)
|
||||
self._task = asyncio.create_task(_loop())
|
||||
|
||||
async def stop(self) -> None:
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
try:
|
||||
await self._task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
|
||||
@property
|
||||
async def healthy_ports(self) -> list[int]:
|
||||
now = time.monotonic()
|
||||
async with self._lock:
|
||||
healthy = []
|
||||
for p in self.all_ports:
|
||||
if self._failures[p] < self.max_failures:
|
||||
healthy.append(p)
|
||||
elif now - self._last_healthy[p] > self.recovery_interval:
|
||||
# Give it another chance after recovery_interval.
|
||||
self._failures[p] = max(0, self._failures[p] - 1)
|
||||
healthy.append(p)
|
||||
return healthy
|
||||
|
||||
async def is_healthy(self, port: int) -> bool:
|
||||
ports = await self.healthy_ports
|
||||
return port in ports
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Load balancer with failover
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class LoadBalancer:
|
||||
"""Distribute requests across healthy backend ports with auto-failover."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
ports: list[int],
|
||||
health_checker: HealthChecker,
|
||||
strategy: str = "round_robin",
|
||||
):
|
||||
self.all_ports = ports
|
||||
self.health_checker = health_checker
|
||||
self.strategy = strategy
|
||||
self._idx = 0
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
def _next_round_robin(self, healthy: list[int]) -> int:
|
||||
if not healthy:
|
||||
return self.all_ports[self._idx % len(self.all_ports)]
|
||||
for _ in range(len(self.all_ports)):
|
||||
port = self.all_ports[self._idx % len(self.all_ports)]
|
||||
self._idx += 1
|
||||
if port in healthy:
|
||||
return port
|
||||
# Fallback: all unhealthy, pick first healthy anyway.
|
||||
return healthy[0] if healthy else self.all_ports[0]
|
||||
|
||||
def _next_random(self, healthy: list[int]) -> int:
|
||||
if healthy:
|
||||
return random.choice(healthy)
|
||||
return random.choice(self.all_ports)
|
||||
|
||||
async def next_port(self) -> int:
|
||||
healthy = await self.health_checker.healthy_ports
|
||||
async with self._lock:
|
||||
if self.strategy == "round_robin":
|
||||
return self._next_round_robin(healthy)
|
||||
elif self.strategy in ("random", "least_conn"):
|
||||
return self._next_random(healthy)
|
||||
else:
|
||||
return self._next_round_robin(healthy)
|
||||
|
||||
@property
|
||||
def port_count(self) -> int:
|
||||
return len(self.all_ports)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Token helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
import uuid
|
||||
def make_prompt(token_len: int) -> str:
|
||||
prefix = str(uuid.uuid4())
|
||||
|
||||
words = [
|
||||
"the", "quick", "brown", "fox",
|
||||
"jumps", "over", "lazy", "dog",
|
||||
"benchmark", "performance", "latency"
|
||||
]
|
||||
|
||||
tokens = [prefix]
|
||||
|
||||
while len(tokens) < token_len:
|
||||
tokens.extend(words)
|
||||
|
||||
return " ".join(tokens[:token_len])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# aiohttp-based async request (preferred)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def send_request_aiohttp(
|
||||
session: aiohttp.ClientSession,
|
||||
host: str,
|
||||
port: int,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
) -> RequestResult:
|
||||
url = f"http://{host}:{port}/v1/chat/completions"
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"stream": True,
|
||||
}
|
||||
headers = {"Content-Type": "application/json"}
|
||||
|
||||
result = RequestResult(request_id=request_id, target_port=port)
|
||||
start_time = time.perf_counter()
|
||||
first_token_received = False
|
||||
last_token_time = 0.0
|
||||
|
||||
try:
|
||||
async with session.post(url, json=payload, headers=headers) as resp:
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
result.error = f"HTTP {resp.status}: {text[:200]}"
|
||||
return result
|
||||
|
||||
async for line in resp.content:
|
||||
line = line.decode("utf-8").strip()
|
||||
if not line or not line.startswith("data: "):
|
||||
continue
|
||||
data_str = line[len("data: "):]
|
||||
if data_str == "[DONE]":
|
||||
break
|
||||
try:
|
||||
chunk = json.loads(data_str)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
choices = chunk.get("choices", [])
|
||||
if not choices:
|
||||
continue
|
||||
|
||||
delta = choices[0].get("delta", {})
|
||||
content = delta.get("content", "")
|
||||
if content:
|
||||
now = time.perf_counter()
|
||||
if not first_token_received:
|
||||
result.first_token_time_ms = (now - start_time) * 1000
|
||||
first_token_received = True
|
||||
else:
|
||||
result.inter_token_latencies.append((now - last_token_time) * 1000)
|
||||
last_token_time = now
|
||||
result.output_tokens += 1
|
||||
|
||||
result.e2e_latency_ms = (time.perf_counter() - start_time) * 1000
|
||||
result.input_tokens = len(prompt.split()) # approximate
|
||||
result.success = True
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
result.error = "timeout"
|
||||
except Exception as e:
|
||||
result.error = str(e)[:200]
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# urllib-based synchronous request (fallback, no aiohttp)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def send_request_urllib(
|
||||
host: str,
|
||||
port: int,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
) -> RequestResult:
|
||||
"""Blocking fallback using stdlib urllib. Used when aiohttp is absent."""
|
||||
url = f"http://{host}:{port}/v1/chat/completions"
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"stream": False,
|
||||
}
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
headers = {"Content-Type": "application/json", "Content-Length": str(len(data))}
|
||||
|
||||
result = RequestResult(request_id=request_id, target_port=port)
|
||||
start_time = time.perf_counter()
|
||||
|
||||
try:
|
||||
req = urllib.request.Request(url, data=data, headers=headers, method="POST")
|
||||
with urllib.request.urlopen(req, timeout=600) as resp:
|
||||
body = json.loads(resp.read().decode("utf-8"))
|
||||
|
||||
choices = body.get("choices", [])
|
||||
if not choices:
|
||||
result.error = "no choices in response"
|
||||
return result
|
||||
|
||||
content = choices[0].get("message", {}).get("content", "")
|
||||
result.output_tokens = len(content.split()) if content else 0
|
||||
result.e2e_latency_ms = (time.perf_counter() - start_time) * 1000
|
||||
result.first_token_time_ms = result.e2e_latency_ms
|
||||
result.input_tokens = len(prompt.split())
|
||||
result.success = True
|
||||
|
||||
except Exception as e:
|
||||
result.error = str(e)[:200]
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Unified send_request wrapper with retry & failover
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def send_request_with_retry(
|
||||
session: aiohttp.ClientSession | None,
|
||||
host: str,
|
||||
lb: LoadBalancer,
|
||||
model: str,
|
||||
request_id: int,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
temperature: float,
|
||||
max_retries: int = 2,
|
||||
) -> RequestResult:
|
||||
"""Send a request, retrying on failure with a different port each time."""
|
||||
result = RequestResult(request_id=request_id)
|
||||
for attempt in range(max_retries + 1):
|
||||
port = await lb.next_port()
|
||||
if HAS_AIOHTTP and session is not None:
|
||||
result = await send_request_aiohttp(
|
||||
session, host, port, model, request_id, prompt, max_tokens, temperature
|
||||
)
|
||||
else:
|
||||
loop = asyncio.get_event_loop()
|
||||
result = await loop.run_in_executor(
|
||||
None, send_request_urllib,
|
||||
host, port, model, request_id, prompt, max_tokens, temperature
|
||||
)
|
||||
result.retry_count = attempt
|
||||
if result.success:
|
||||
return result
|
||||
# If failed, try another port on next iteration (unless last attempt).
|
||||
if attempt < max_retries:
|
||||
await asyncio.sleep(0.5 * (attempt + 1)) # exponential backoff-ish
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Benchmark orchestration
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def run_benchmark(
|
||||
host: str,
|
||||
ports: list[int],
|
||||
lb: LoadBalancer,
|
||||
health_checker: HealthChecker,
|
||||
model: str,
|
||||
concurrency: int,
|
||||
input_len: int,
|
||||
output_len: int,
|
||||
num_prompts: int,
|
||||
temperature: float,
|
||||
output_file: str,
|
||||
) -> dict[str, Any]:
|
||||
"""Run the benchmark and write a JSONL file."""
|
||||
prompts = [make_prompt(input_len) for _ in range(num_prompts)]
|
||||
results: list[RequestResult] = []
|
||||
semaphore = asyncio.Semaphore(concurrency)
|
||||
|
||||
start = time.perf_counter()
|
||||
|
||||
if HAS_AIOHTTP:
|
||||
connector = aiohttp.TCPConnector(limit=concurrency * 2)
|
||||
timeout = aiohttp.ClientTimeout(total=600, sock_read=300)
|
||||
async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
|
||||
await health_checker.start(session)
|
||||
|
||||
async def bounded_send(idx: int, prompt: str) -> RequestResult:
|
||||
async with semaphore:
|
||||
return await send_request_with_retry(
|
||||
session, host, lb, model, idx, prompt, output_len, temperature
|
||||
)
|
||||
print("Starting warmup...")
|
||||
warmup_per_service = 10
|
||||
warmup_requests = warmup_per_service * len(ports)
|
||||
|
||||
warmup_prompts = [
|
||||
make_prompt(input_len)
|
||||
for _ in range(warmup_requests)
|
||||
]
|
||||
|
||||
warmup_tasks = [
|
||||
send_request_with_retry(
|
||||
session=session,
|
||||
host=host,
|
||||
lb=lb,
|
||||
model=model,
|
||||
request_id=-1,
|
||||
prompt=p,
|
||||
max_tokens=output_len,
|
||||
temperature=temperature,
|
||||
)
|
||||
for p in warmup_prompts
|
||||
]
|
||||
|
||||
await asyncio.gather(*warmup_tasks)
|
||||
|
||||
print("Warmup finished.")
|
||||
|
||||
# 给 CUDA / Scheduler 一点时间稳定
|
||||
await asyncio.sleep(2)
|
||||
|
||||
tasks = [bounded_send(i, p) for i, p in enumerate(prompts)]
|
||||
results = await asyncio.gather(*tasks)
|
||||
await health_checker.stop()
|
||||
else:
|
||||
await health_checker.start(None)
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
async def bounded_send(idx: int, prompt: str) -> RequestResult:
|
||||
async with semaphore:
|
||||
return await send_request_with_retry(
|
||||
None, host, lb, model, idx, prompt, output_len, temperature
|
||||
)
|
||||
|
||||
tasks = [bounded_send(i, p) for i, p in enumerate(prompts)]
|
||||
results = await asyncio.gather(*tasks)
|
||||
await health_checker.stop()
|
||||
|
||||
duration = time.perf_counter() - start
|
||||
|
||||
# Write JSONL
|
||||
with open(output_file, "w", encoding="utf-8") as f:
|
||||
for r in results:
|
||||
f.write(json.dumps(r.to_dict(), ensure_ascii=False) + "\n")
|
||||
|
||||
# Compute summary
|
||||
successful = [r for r in results if r.success]
|
||||
failed = len(results) - len(successful)
|
||||
retried = [r for r in results if r.retry_count > 0]
|
||||
|
||||
def percentile(values: list[float], p: float) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
s = sorted(values)
|
||||
k = (len(s) - 1) * p / 100.0
|
||||
f = int(k)
|
||||
c = min(f + 1, len(s) - 1)
|
||||
return s[f] + (k - f) * (s[c] - s[f])
|
||||
|
||||
ttfts = [r.first_token_time_ms for r in successful]
|
||||
tpots = [r.tpot_ms for r in successful if r.output_tokens > 1]
|
||||
e2es = [r.e2e_latency_ms for r in successful]
|
||||
itls = [r.mean_itl_ms for r in successful if r.inter_token_latencies]
|
||||
|
||||
total_input_tokens = sum(r.input_tokens for r in successful)
|
||||
total_output_tokens = sum(r.output_tokens for r in successful)
|
||||
|
||||
# Per-port breakdown
|
||||
per_port_stats: dict[int, dict[str, Any]] = {}
|
||||
for port in ports:
|
||||
port_results = [r for r in successful if r.target_port == port]
|
||||
port_failed = [r for r in results if r.target_port == port and not r.success]
|
||||
if port_results or port_failed:
|
||||
port_ttfts = [r.first_token_time_ms for r in port_results]
|
||||
port_e2es = [r.e2e_latency_ms for r in port_results]
|
||||
per_port_stats[port] = {
|
||||
"count": len(port_results),
|
||||
"failed": len(port_failed),
|
||||
"mean_ttft_ms": sum(port_ttfts) / len(port_ttfts) if port_ttfts else 0.0,
|
||||
"p95_ttft_ms": percentile(port_ttfts, 95) if port_ttfts else 0.0,
|
||||
"mean_e2e_ms": sum(port_e2es) / len(port_e2es) if port_e2es else 0.0,
|
||||
}
|
||||
|
||||
# Health-check stats
|
||||
healthy_at_end = await health_checker.healthy_ports
|
||||
|
||||
summary = {
|
||||
"completed": len(successful),
|
||||
"failed": failed,
|
||||
"retried": len(retried),
|
||||
"duration": duration,
|
||||
"request_throughput": len(successful) / duration if duration > 0 else 0.0,
|
||||
"input_throughput": total_input_tokens / duration if duration > 0 else 0.0,
|
||||
"output_throughput": total_output_tokens / duration if duration > 0 else 0.0,
|
||||
"total_throughput": (total_input_tokens + total_output_tokens) / duration if duration > 0 else 0.0,
|
||||
"total_input_tokens": total_input_tokens,
|
||||
"total_output_tokens": total_output_tokens,
|
||||
"mean_ttft_ms": sum(ttfts) / len(ttfts) if ttfts else 0.0,
|
||||
"median_ttft_ms": percentile(ttfts, 50),
|
||||
"p90_ttft_ms": percentile(ttfts, 90),
|
||||
"p95_ttft_ms": percentile(ttfts, 95),
|
||||
"p99_ttft_ms": percentile(ttfts, 99),
|
||||
"mean_tpot_ms": sum(tpots) / len(tpots) if tpots else 0.0,
|
||||
"median_tpot_ms": percentile(tpots, 50),
|
||||
"p90_tpot_ms": percentile(tpots, 90),
|
||||
"p95_tpot_ms": percentile(tpots, 95),
|
||||
"p99_tpot_ms": percentile(tpots, 99),
|
||||
"mean_e2e_latency_ms": sum(e2es) / len(e2es) if e2es else 0.0,
|
||||
"median_e2e_latency_ms": percentile(e2es, 50),
|
||||
"p90_e2e_latency_ms": percentile(e2es, 90),
|
||||
"p95_e2e_latency_ms": percentile(e2es, 95),
|
||||
"p99_e2e_latency_ms": percentile(e2es, 99),
|
||||
"mean_itl_ms": sum(itls) / len(itls) if itls else 0.0,
|
||||
"median_itl_ms": percentile(itls, 50),
|
||||
"p90_itl_ms": percentile(itls, 90),
|
||||
"p95_itl_ms": percentile(itls, 95),
|
||||
"p99_itl_ms": percentile(itls, 99),
|
||||
"per_port_stats": per_port_stats,
|
||||
"healthy_ports_at_end": healthy_at_end,
|
||||
"lb_strategy": lb.strategy,
|
||||
"num_services": lb.port_count,
|
||||
}
|
||||
|
||||
# Append summary as the last line
|
||||
with open(output_file, "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(summary, ensure_ascii=False) + "\n")
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Custom vLLM benchmark client with multi-service LB and health-check failover")
|
||||
parser.add_argument("--host", default="127.0.0.1")
|
||||
parser.add_argument("--port", type=int, default=30005,
|
||||
help="Single service port (backward compatible)")
|
||||
parser.add_argument("--ports", default="",
|
||||
help="Comma-separated list of ports for multi-service, e.g. 30005,30006,30007,30008")
|
||||
parser.add_argument("--model", default="deepseek-v4-flash")
|
||||
parser.add_argument("--concurrency", type=int, default=1)
|
||||
parser.add_argument("--input-len", type=int, default=512)
|
||||
parser.add_argument("--output-len", type=int, default=256)
|
||||
parser.add_argument("--num-prompts", type=int, default=10)
|
||||
parser.add_argument("--temperature", type=float, default=0.0)
|
||||
parser.add_argument("--lb-strategy", default="round_robin",
|
||||
choices=["round_robin", "random", "least_conn"],
|
||||
help="Load balancing strategy across services")
|
||||
parser.add_argument("--health-check-interval", type=float, default=5.0,
|
||||
help="Seconds between health checks (default: 5)")
|
||||
parser.add_argument("--health-max-failures", type=int, default=2,
|
||||
help="Consecutive failures before marking a port unhealthy (default: 2)")
|
||||
parser.add_argument("--health-recovery-interval", type=float, default=10.0,
|
||||
help="Seconds before retrying an unhealthy port (default: 10)")
|
||||
parser.add_argument("--request-max-retries", type=int, default=2,
|
||||
help="Max retries per request on failure (default: 2)")
|
||||
parser.add_argument("--output-file", required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Determine ports: --ports takes precedence, else fall back to --port.
|
||||
if args.ports:
|
||||
ports = [int(p.strip()) for p in args.ports.split(",")]
|
||||
else:
|
||||
ports = [args.port]
|
||||
|
||||
health_checker = HealthChecker(
|
||||
host=args.host,
|
||||
ports=ports,
|
||||
check_interval=args.health_check_interval,
|
||||
max_failures=args.health_max_failures,
|
||||
recovery_interval=args.health_recovery_interval,
|
||||
)
|
||||
lb = LoadBalancer(ports, health_checker=health_checker, strategy=args.lb_strategy)
|
||||
|
||||
summary = asyncio.run(run_benchmark(
|
||||
host=args.host,
|
||||
ports=ports,
|
||||
lb=lb,
|
||||
health_checker=health_checker,
|
||||
model=args.model,
|
||||
concurrency=args.concurrency,
|
||||
input_len=args.input_len,
|
||||
output_len=args.output_len,
|
||||
num_prompts=args.num_prompts,
|
||||
temperature=args.temperature,
|
||||
output_file=args.output_file,
|
||||
))
|
||||
|
||||
print(json.dumps(summary, indent=2, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
180
experiments/dsv4_h200_vllm_tp8_custom_bench/config.env
Normal file
180
experiments/dsv4_h200_vllm_tp8_custom_bench/config.env
Normal file
@ -0,0 +1,180 @@
|
||||
# Experiment configuration for dsv4_h200_vllm_tp2_custom_bench
|
||||
# Custom benchmark client for multi-service vLLM TP=2 on 8x H200.
|
||||
#
|
||||
# This experiment runs 4 independent vLLM services (TP=2 each) on GPU pairs:
|
||||
# Service 1: port 30005, GPUs 0,1
|
||||
# Service 2: port 30006, GPUs 2,3
|
||||
# Service 3: port 30007, GPUs 4,5
|
||||
# Service 4: port 30008, GPUs 6,7
|
||||
#
|
||||
# bench_client.py distributes requests across all 4 services via round-robin
|
||||
# load balancing, giving accurate cluster-wide throughput even at low
|
||||
# per-service concurrency.
|
||||
|
||||
EXPERIMENT="dsv4_h200_vllm_tp8_custom_bench"
|
||||
MODEL_NAME="DeepSeek-V4-Flash"
|
||||
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
|
||||
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
# Primary port (backward compatible); multi-service ports are in PORTS.
|
||||
PORT="30005"
|
||||
PORTS="30005"
|
||||
|
||||
BACKEND="vllm"
|
||||
ENGINE="vllm"
|
||||
|
||||
# Native virtual environments on the host.
|
||||
VENV_SERVER="/data/user1/yy/envs/vllm"
|
||||
VENV_CLIENT="/data/user1/yy/envs/vllm"
|
||||
|
||||
# Load balancing strategy: round_robin | random | least_conn
|
||||
LB_STRATEGY="round_robin"
|
||||
|
||||
# Benchmark scenarios: "concurrency input_len output_len num_prompts".
|
||||
# For single-service TP=8 we cover standard concurrency gradients.
|
||||
# concurrency input_len output_len num_prompts
|
||||
SCENARIOS=(
|
||||
"1 512 256 20"
|
||||
"2 512 256 40"
|
||||
"4 512 256 80"
|
||||
"8 512 256 160"
|
||||
"16 512 256 320"
|
||||
"32 512 256 640"
|
||||
"64 512 256 1280"
|
||||
"128 512 256 2560"
|
||||
|
||||
"1 1024 128 20"
|
||||
"1 1024 256 20"
|
||||
"1 1024 512 20"
|
||||
"1 1024 1024 20"
|
||||
"1 1024 2048 20"
|
||||
"1 1024 4096 20"
|
||||
|
||||
"8 1024 128 160"
|
||||
"8 1024 256 160"
|
||||
"8 1024 512 160"
|
||||
"8 1024 1024 160"
|
||||
"8 1024 2048 160"
|
||||
"8 1024 4096 160"
|
||||
|
||||
"32 1024 128 640"
|
||||
"32 1024 256 640"
|
||||
"32 1024 512 640"
|
||||
"32 1024 1024 640"
|
||||
"32 1024 2048 640"
|
||||
"32 1024 4096 640"
|
||||
|
||||
"128 1024 128 2560"
|
||||
"128 1024 256 2560"
|
||||
"128 1024 512 2560"
|
||||
"128 1024 1024 2560"
|
||||
"128 1024 2048 2560"
|
||||
"128 1024 4096 2560"
|
||||
|
||||
"1 2048 128 20"
|
||||
"1 2048 256 20"
|
||||
"1 2048 512 20"
|
||||
"1 2048 1024 20"
|
||||
"1 2048 2048 20"
|
||||
"1 2048 4096 20"
|
||||
|
||||
"8 2048 128 160"
|
||||
"8 2048 256 160"
|
||||
"8 2048 512 160"
|
||||
"8 2048 1024 160"
|
||||
"8 2048 2048 160"
|
||||
"8 2048 4096 160"
|
||||
|
||||
"32 2048 128 640"
|
||||
"32 2048 256 640"
|
||||
"32 2048 512 640"
|
||||
"32 2048 1024 640"
|
||||
"32 2048 2048 640"
|
||||
"32 2048 4096 640"
|
||||
|
||||
"128 2048 128 2560"
|
||||
"128 2048 256 2560"
|
||||
"128 2048 512 2560"
|
||||
"128 2048 1024 2560"
|
||||
"128 2048 2048 2560"
|
||||
"128 2048 4096 2560"
|
||||
|
||||
"1 4096 128 20"
|
||||
"1 4096 256 20"
|
||||
"1 4096 512 20"
|
||||
"1 4096 1024 20"
|
||||
"1 4096 2048 20"
|
||||
"1 4096 4096 20"
|
||||
|
||||
"8 4096 128 160"
|
||||
"8 4096 256 160"
|
||||
"8 4096 512 160"
|
||||
"8 4096 1024 160"
|
||||
"8 4096 2048 160"
|
||||
"8 4096 4096 160"
|
||||
|
||||
"32 4096 128 640"
|
||||
"32 4096 256 640"
|
||||
"32 4096 512 640"
|
||||
"32 4096 1024 640"
|
||||
"32 4096 2048 640"
|
||||
"32 4096 4096 640"
|
||||
|
||||
"128 4096 128 2560"
|
||||
"128 4096 256 2560"
|
||||
"128 4096 512 2560"
|
||||
"128 4096 1024 2560"
|
||||
"128 4096 2048 2560"
|
||||
"128 4096 4096 2560"
|
||||
|
||||
"1 16384 128 20"
|
||||
"1 16384 256 20"
|
||||
"1 16384 512 20"
|
||||
"1 16384 1024 20"
|
||||
"1 16384 2048 20"
|
||||
|
||||
"8 16384 128 160"
|
||||
"8 16384 256 160"
|
||||
"8 16384 512 160"
|
||||
"8 16384 1024 160"
|
||||
"8 16384 2048 160"
|
||||
|
||||
"32 16384 128 640"
|
||||
"32 16384 256 640"
|
||||
"32 16384 512 640"
|
||||
"32 16384 1024 640"
|
||||
"32 16384 2048 640"
|
||||
|
||||
"128 16384 128 2560"
|
||||
"128 16384 256 2560"
|
||||
"128 16384 512 2560"
|
||||
"128 16384 1024 2560"
|
||||
"128 16384 2048 2560"
|
||||
|
||||
|
||||
"1 65536 128 20"
|
||||
"1 65536 256 20"
|
||||
"1 65536 512 20"
|
||||
"1 65536 1024 20"
|
||||
|
||||
"8 65536 128 160"
|
||||
"8 65536 256 160"
|
||||
"8 65536 512 160"
|
||||
"8 65536 1024 160"
|
||||
"32 65536 1024 160"
|
||||
|
||||
"1 131072 128 20"
|
||||
"1 131072 256 20"
|
||||
"1 131072 512 20"
|
||||
|
||||
"8 131072 512 160"
|
||||
|
||||
"1 262144 256 20"
|
||||
|
||||
"8 262144 128 160"
|
||||
|
||||
"1 524288 128 20"
|
||||
)
|
||||
|
||||
# Server start script bundled with this experiment.
|
||||
SERVER_START_SCRIPT="${SCRIPT_DIR}/start_server.sh"
|
||||
206
experiments/dsv4_h200_vllm_tp8_custom_bench/parse_results.py
Executable file
206
experiments/dsv4_h200_vllm_tp8_custom_bench/parse_results.py
Executable file
@ -0,0 +1,206 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Parse custom bench_client.py JSONL outputs for vLLM TP=2 benchmark.
|
||||
|
||||
Reads JSONL files produced by bench_client.py (one request per line,
|
||||
last line is the summary) and generates:
|
||||
- results.json (appended scenarios, following the standard schema)
|
||||
- report.md (human-readable summary)
|
||||
|
||||
Usage:
|
||||
python3 parse_results.py <result_root>
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def parse_jsonl(path: Path) -> tuple[dict | None, list[dict]]:
|
||||
"""Read the summary JSON (last line) and all per-request lines."""
|
||||
requests: list[dict] = []
|
||||
summary: dict | None = None
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
# The summary line has "completed" and "duration" keys.
|
||||
if "completed" in obj and "duration" in obj:
|
||||
summary = obj
|
||||
else:
|
||||
requests.append(obj)
|
||||
return summary, requests
|
||||
|
||||
|
||||
def compute_metrics(summary: dict, requests: list[dict]) -> dict:
|
||||
completed = summary.get("completed", 0)
|
||||
total = len(requests)
|
||||
failed = total - completed if total > 0 else 0
|
||||
duration_s = summary.get("duration", 0.0)
|
||||
|
||||
# Extract per-request metrics for percentiles.
|
||||
ttfts = [r["first_token_time_ms"] for r in requests if r.get("success")]
|
||||
tpots = [r["tpot_ms"] for r in requests if r.get("success") and r.get("output_tokens", 0) > 1]
|
||||
e2es = [r["e2e_latency_ms"] for r in requests if r.get("success")]
|
||||
itls = [r["mean_itl_ms"] for r in requests if r.get("success") and r.get("inter_token_latencies")]
|
||||
|
||||
def percentile(values: list[float], p: float) -> float:
|
||||
if not values:
|
||||
return 0.0
|
||||
s = sorted(values)
|
||||
k = (len(s) - 1) * p / 100.0
|
||||
f = int(k)
|
||||
c = min(f + 1, len(s) - 1)
|
||||
return s[f] + (k - f) * (s[c] - s[f])
|
||||
|
||||
return {
|
||||
"success": completed,
|
||||
"failed": failed,
|
||||
"duration_s": duration_s,
|
||||
"request_throughput": summary.get("request_throughput", 0.0),
|
||||
"input_token_throughput": summary.get("input_throughput", 0.0),
|
||||
"output_token_throughput": summary.get("output_throughput", 0.0),
|
||||
"total_token_throughput": summary.get("total_throughput", 0.0),
|
||||
"total_input_tokens": summary.get("total_input_tokens", 0),
|
||||
"total_output_tokens": summary.get("total_output_tokens", 0),
|
||||
"e2e_ms": {
|
||||
"mean": summary.get("mean_e2e_latency_ms", 0.0),
|
||||
"p50": percentile(e2es, 50),
|
||||
"p90": percentile(e2es, 90),
|
||||
"p95": percentile(e2es, 95),
|
||||
"p99": percentile(e2es, 99),
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": summary.get("mean_ttft_ms", 0.0),
|
||||
"p50": percentile(ttfts, 50),
|
||||
"p90": percentile(ttfts, 90),
|
||||
"p95": percentile(ttfts, 95),
|
||||
"p99": percentile(ttfts, 99),
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": summary.get("mean_tpot_ms", 0.0),
|
||||
"p50": percentile(tpots, 50),
|
||||
"p90": percentile(tpots, 90),
|
||||
"p95": percentile(tpots, 95),
|
||||
"p99": percentile(tpots, 99),
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": summary.get("mean_itl_ms", 0.0),
|
||||
"p50": percentile(itls, 50),
|
||||
"p90": percentile(itls, 90),
|
||||
"p95": percentile(itls, 95),
|
||||
"p99": percentile(itls, 99),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def scenario_name(concurrency: int, input_len: int, output_len: int) -> str:
|
||||
return f"c{concurrency}_i{input_len}_o{output_len}"
|
||||
|
||||
|
||||
def slo_status(metrics: dict, ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> dict:
|
||||
ttft_ok = metrics["ttft_ms"]["p95"] < ttft_limit_ms
|
||||
tpot_ok = metrics["tpot_ms"]["mean"] < tpot_limit_ms
|
||||
if ttft_ok and tpot_ok:
|
||||
mark = "✅"
|
||||
elif ttft_ok or tpot_ok:
|
||||
mark = "⚠️"
|
||||
else:
|
||||
mark = "❌"
|
||||
return {
|
||||
"ttft_p95_ok": ttft_ok,
|
||||
"tpot_mean_ok": tpot_ok,
|
||||
"overall": mark,
|
||||
}
|
||||
|
||||
|
||||
def append_scenario(results_json: Path, scenario: dict) -> None:
|
||||
with open(results_json, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
data["scenarios"].append(scenario)
|
||||
with open(results_json, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
|
||||
|
||||
def generate_report(result_root: Path, scenarios: list[dict]) -> None:
|
||||
report_path = result_root / "report.md"
|
||||
with open(report_path, "w", encoding="utf-8") as f:
|
||||
f.write("# H200 vLLM TP=2 Custom Benchmark Report\n\n")
|
||||
f.write("- **Client**: `bench_client.py` (async OpenAI API, per-request timing)\n")
|
||||
f.write("- **Backend**: vLLM (TP=2, FP8 KV cache, no speculative decoding)\n")
|
||||
f.write(f"- **Result root**: `{result_root}`\n\n")
|
||||
|
||||
f.write("## Results\n\n")
|
||||
f.write("| Scenario | Concurrency | Input | Output | Duration(s) | Success | Failed | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |\n")
|
||||
f.write("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n")
|
||||
|
||||
for s in scenarios:
|
||||
cfg = s["config"]
|
||||
m = s["metrics"]
|
||||
slo = s.get("slo_status", {}).get("overall", "")
|
||||
f.write(
|
||||
f"| {s['name']} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | "
|
||||
f"{m['duration_s']:.2f} | {m['success']} | {m['failed']} | {m['request_throughput']:.2f} | "
|
||||
f"{m['input_token_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||
f"{m['total_token_throughput']:.2f} | "
|
||||
f"{m['ttft_ms']['mean']:.2f} | {m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||
f"{m['e2e_ms']['mean']:.2f} | {m['e2e_ms']['p95']:.2f} | {m['e2e_ms']['p99']:.2f} | {slo} |\n"
|
||||
)
|
||||
f.write("\n")
|
||||
f.write("SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.\n\n")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("results")
|
||||
raw_dir = result_root / "raw_outputs"
|
||||
results_json = result_root / "results.json"
|
||||
|
||||
if not raw_dir.exists():
|
||||
raise SystemExit(f"raw_outputs directory not found: {raw_dir}")
|
||||
|
||||
scenarios = []
|
||||
for jsonl_path in sorted(raw_dir.glob("vllm_*.jsonl")):
|
||||
parts = jsonl_path.stem.split("_")
|
||||
if len(parts) < 5:
|
||||
continue
|
||||
concurrency, input_len, output_len = int(parts[2]), int(parts[3]), int(parts[4])
|
||||
|
||||
summary, requests = parse_jsonl(jsonl_path)
|
||||
if summary is None:
|
||||
continue
|
||||
|
||||
metrics = compute_metrics(summary, requests)
|
||||
scenario = {
|
||||
"name": scenario_name(concurrency, input_len, output_len),
|
||||
"config": {
|
||||
"concurrency": concurrency,
|
||||
"input_len": input_len,
|
||||
"output_len": output_len,
|
||||
"dataset": "random",
|
||||
"num_prompts": metrics["success"] + metrics["failed"],
|
||||
},
|
||||
"metrics": metrics,
|
||||
"slo_status": slo_status(metrics),
|
||||
"raw_file": str(jsonl_path),
|
||||
}
|
||||
scenarios.append(scenario)
|
||||
|
||||
if not scenarios:
|
||||
print("No benchmark outputs found to parse")
|
||||
return
|
||||
|
||||
if results_json.exists():
|
||||
for s in scenarios:
|
||||
append_scenario(results_json, s)
|
||||
|
||||
generate_report(result_root, scenarios)
|
||||
print(f"Parsed {len(scenarios)} scenarios into {result_root}/report.md")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@ -0,0 +1,129 @@
|
||||
# H200 vLLM TP=2 Custom Benchmark Report
|
||||
|
||||
- **Client**: `bench_client.py` (async OpenAI API, per-request timing)
|
||||
- **Backend**: vLLM (TP=2, FP8 KV cache, no speculative decoding)
|
||||
- **Result root**: `/data/user1/qqt/sskj/experiments/dsv4_h200_vllm_tp8_custom_bench/results/20260711-193756`
|
||||
|
||||
## Results
|
||||
|
||||
| Scenario | Concurrency | Input | Output | Duration(s) | Success | Failed | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c128_i1024_o1024 | 128 | 1024 | 1024 | 206.75 | 2560 | 0 | 12.38 | 12679.34 | 2588.49 | 15267.83 | 355.51 | 541.00 | 3399.25 | 46.35 | 50.58 | 52.54 | 9995.61 | 13713.98 | 15284.95 | ✅ |
|
||||
| c128_i1024_o128 | 128 | 1024 | 128 | 133.01 | 2560 | 0 | 19.25 | 19708.67 | 2459.70 | 22168.36 | 1183.32 | 1479.29 | 3426.82 | 41.60 | 47.50 | 49.22 | 6458.71 | 7012.22 | 8759.52 | ✅ |
|
||||
| c128_i1024_o2048 | 128 | 1024 | 2048 | 208.80 | 2560 | 0 | 12.26 | 12554.51 | 2547.16 | 15101.67 | 351.46 | 493.93 | 3429.21 | 47.17 | 52.36 | 59.29 | 10106.58 | 13980.86 | 15725.90 | ✅ |
|
||||
| c128_i1024_o256 | 128 | 1024 | 256 | 205.11 | 2560 | 0 | 12.48 | 12780.63 | 2561.23 | 15341.86 | 351.22 | 456.94 | 3426.54 | 46.93 | 50.36 | 51.39 | 9935.31 | 12606.55 | 13028.92 | ✅ |
|
||||
| c128_i1024_o4096 | 128 | 1024 | 4096 | 208.63 | 2560 | 0 | 12.27 | 12564.96 | 2562.18 | 15127.14 | 352.13 | 475.05 | 3401.06 | 46.82 | 50.35 | 51.59 | 10079.33 | 13938.76 | 15626.71 | ✅ |
|
||||
| c128_i1024_o512 | 128 | 1024 | 512 | 210.21 | 2560 | 0 | 12.18 | 12470.38 | 2534.38 | 15004.76 | 355.66 | 519.46 | 3397.33 | 47.25 | 52.84 | 55.05 | 10138.54 | 13931.68 | 15398.92 | ✅ |
|
||||
| c128_i16384_o128 | 128 | 16384 | 128 | 1359.07 | 2560 | 0 | 1.88 | 30861.66 | 240.91 | 31102.57 | 34308.65 | 34726.75 | 53850.14 | 257.93 | 261.13 | 261.25 | 67037.51 | 67882.99 | 86482.77 | ❌ |
|
||||
| c128_i16384_o256 | 128 | 16384 | 256 | 1383.21 | 2560 | 0 | 1.85 | 30323.03 | 393.12 | 30716.14 | 13674.31 | 16619.00 | 53959.97 | 259.70 | 265.74 | 266.01 | 68522.28 | 82348.90 | 113143.30 | ❌ |
|
||||
| c128_i16384_o512 | 128 | 16384 | 512 | 1394.58 | 2560 | 0 | 1.84 | 30075.73 | 449.14 | 30524.87 | 6929.19 | 14006.30 | 53929.18 | 256.42 | 266.39 | 266.68 | 68997.86 | 134484.18 | 142208.84 | ❌ |
|
||||
| c128_i2048_o1024 | 128 | 2048 | 1024 | 274.18 | 2560 | 0 | 9.34 | 19121.84 | 2068.13 | 21189.97 | 541.55 | 764.67 | 6604.67 | 57.69 | 62.18 | 65.53 | 13253.98 | 21801.42 | 28048.56 | ⚠️ |
|
||||
| c128_i2048_o128 | 128 | 2048 | 128 | 212.99 | 2560 | 0 | 12.02 | 24615.40 | 1530.02 | 26145.42 | 1293.09 | 1997.49 | 6703.31 | 72.33 | 80.38 | 81.78 | 10431.32 | 11370.95 | 15884.14 | ⚠️ |
|
||||
| c128_i2048_o2048 | 128 | 2048 | 2048 | 272.06 | 2560 | 0 | 9.41 | 19270.69 | 2055.16 | 21325.85 | 542.66 | 696.46 | 6595.59 | 58.02 | 62.67 | 65.23 | 13143.51 | 21512.40 | 28583.36 | ⚠️ |
|
||||
| c128_i2048_o256 | 128 | 2048 | 256 | 263.83 | 2560 | 0 | 9.70 | 19872.28 | 1986.56 | 21858.84 | 552.74 | 766.94 | 6667.42 | 60.36 | 64.80 | 67.61 | 12841.30 | 16455.00 | 19770.97 | ⚠️ |
|
||||
| c128_i2048_o4096 | 128 | 2048 | 4096 | 274.18 | 2560 | 0 | 9.34 | 19121.82 | 2075.59 | 21197.41 | 538.85 | 772.70 | 6666.24 | 57.59 | 61.73 | 65.98 | 13271.96 | 22132.76 | 28679.85 | ⚠️ |
|
||||
| c128_i2048_o512 | 128 | 2048 | 512 | 271.80 | 2560 | 0 | 9.42 | 19289.73 | 2074.26 | 21363.99 | 544.24 | 780.74 | 6725.67 | 57.75 | 61.62 | 65.40 | 13185.12 | 21822.20 | 28335.44 | ⚠️ |
|
||||
| c128_i4096_o1024 | 128 | 4096 | 1024 | 424.83 | 2560 | 0 | 6.03 | 24682.53 | 1273.81 | 25956.34 | 1014.26 | 1488.91 | 13325.90 | 93.93 | 103.74 | 115.24 | 20697.67 | 35507.95 | 47170.14 | ⚠️ |
|
||||
| c128_i4096_o128 | 128 | 4096 | 128 | 371.42 | 2560 | 0 | 6.89 | 28231.71 | 873.71 | 29105.42 | 1687.74 | 2056.57 | 13344.42 | 132.08 | 142.11 | 145.58 | 18310.58 | 19462.37 | 30169.28 | ⚠️ |
|
||||
| c128_i4096_o2048 | 128 | 4096 | 2048 | 425.25 | 2560 | 0 | 6.02 | 24657.77 | 1288.03 | 25945.80 | 1019.46 | 1464.89 | 13332.86 | 93.13 | 101.52 | 111.69 | 20779.35 | 36358.76 | 50447.50 | ⚠️ |
|
||||
| c128_i4096_o256 | 128 | 4096 | 256 | 413.80 | 2560 | 0 | 6.19 | 25340.14 | 1217.79 | 26557.93 | 1065.75 | 1554.31 | 13333.89 | 98.37 | 108.53 | 114.97 | 20311.65 | 26976.60 | 33054.57 | ⚠️ |
|
||||
| c128_i4096_o4096 | 128 | 4096 | 4096 | 423.76 | 2560 | 0 | 6.04 | 24744.50 | 1282.56 | 26027.06 | 1040.84 | 1941.31 | 13328.94 | 93.34 | 101.81 | 111.13 | 20723.12 | 35255.54 | 48374.31 | ⚠️ |
|
||||
| c128_i4096_o512 | 128 | 4096 | 512 | 423.70 | 2560 | 0 | 6.04 | 24748.17 | 1274.42 | 26022.58 | 1032.52 | 1562.56 | 13323.34 | 93.86 | 103.68 | 112.51 | 20669.59 | 35396.55 | 49034.77 | ⚠️ |
|
||||
| c128_i512_o256 | 128 | 512 | 256 | 198.05 | 2560 | 0 | 12.93 | 6618.16 | 2673.83 | 9291.99 | 290.66 | 501.00 | 1944.87 | 45.16 | 52.68 | 57.19 | 9585.92 | 13135.34 | 14237.26 | ✅ |
|
||||
| c16_i512_o256 | 16 | 512 | 256 | 70.00 | 320 | 0 | 4.57 | 2340.54 | 954.25 | 3294.79 | 155.99 | 299.17 | 421.01 | 14.77 | 16.43 | 17.44 | 3227.91 | 4110.18 | 4253.24 | ✅ |
|
||||
| c1_i1024_o1024 | 1 | 1024 | 1024 | 41.00 | 20 | 0 | 0.49 | 499.49 | 102.02 | 601.51 | 113.27 | 116.29 | 137.79 | 6.73 | 6.74 | 6.74 | 1513.65 | 1949.27 | 1998.66 | ✅ |
|
||||
| c1_i1024_o128 | 1 | 1024 | 128 | 26.62 | 20 | 0 | 0.75 | 769.29 | 96.16 | 865.45 | 112.10 | 114.09 | 127.55 | 6.69 | 6.70 | 6.70 | 961.87 | 964.11 | 977.09 | ✅ |
|
||||
| c1_i1024_o2048 | 1 | 1024 | 2048 | 40.37 | 20 | 0 | 0.50 | 507.26 | 100.39 | 607.65 | 113.54 | 118.17 | 129.38 | 6.73 | 6.75 | 6.75 | 1470.10 | 1949.87 | 2093.40 | ✅ |
|
||||
| c1_i1024_o256 | 1 | 1024 | 256 | 39.66 | 20 | 0 | 0.50 | 516.37 | 101.41 | 617.77 | 112.03 | 113.19 | 128.94 | 6.73 | 6.74 | 6.74 | 1457.72 | 1819.60 | 1819.96 | ✅ |
|
||||
| c1_i1024_o4096 | 1 | 1024 | 4096 | 39.07 | 20 | 0 | 0.51 | 524.19 | 100.38 | 624.57 | 126.46 | 178.17 | 178.35 | 6.73 | 6.74 | 6.75 | 1439.54 | 1773.78 | 1831.83 | ✅ |
|
||||
| c1_i1024_o512 | 1 | 1024 | 512 | 38.86 | 20 | 0 | 0.51 | 527.06 | 98.62 | 625.68 | 113.09 | 114.42 | 136.00 | 6.73 | 6.74 | 6.75 | 1396.05 | 1797.45 | 1820.39 | ✅ |
|
||||
| c1_i16384_o1024 | 1 | 16384 | 1024 | 60.02 | 20 | 0 | 0.33 | 5459.87 | 74.28 | 5534.15 | 621.87 | 633.10 | 637.21 | 6.82 | 6.83 | 6.85 | 2136.32 | 2962.66 | 3211.92 | ✅ |
|
||||
| c1_i16384_o128 | 1 | 16384 | 128 | 41.67 | 20 | 0 | 0.48 | 7862.85 | 60.88 | 7923.72 | 628.49 | 631.58 | 636.36 | 6.77 | 6.81 | 6.81 | 1480.09 | 1482.52 | 1482.98 | ✅ |
|
||||
| c1_i16384_o2048 | 1 | 16384 | 2048 | 63.80 | 20 | 0 | 0.31 | 5136.27 | 80.76 | 5217.03 | 630.15 | 636.36 | 636.98 | 6.82 | 6.83 | 6.85 | 2381.45 | 4577.47 | 5209.66 | ✅ |
|
||||
| c1_i16384_o256 | 1 | 16384 | 256 | 56.44 | 20 | 0 | 0.35 | 5805.62 | 73.65 | 5879.27 | 630.48 | 638.61 | 639.04 | 6.82 | 6.85 | 6.85 | 2040.24 | 2358.86 | 2362.47 | ✅ |
|
||||
| c1_i16384_o512 | 1 | 16384 | 512 | 70.93 | 20 | 0 | 0.28 | 4619.51 | 85.87 | 4705.38 | 621.45 | 629.96 | 645.42 | 6.82 | 6.85 | 6.85 | 2692.18 | 4094.39 | 4094.98 | ✅ |
|
||||
| c1_i2048_o1024 | 1 | 2048 | 1024 | 43.34 | 20 | 0 | 0.46 | 945.05 | 102.00 | 1047.06 | 123.15 | 127.95 | 148.69 | 6.80 | 6.81 | 6.81 | 1619.33 | 2410.40 | 2903.39 | ✅ |
|
||||
| c1_i2048_o128 | 1 | 2048 | 128 | 27.02 | 20 | 0 | 0.74 | 1515.96 | 94.12 | 1610.08 | 122.85 | 126.65 | 147.47 | 6.77 | 6.77 | 6.82 | 976.42 | 983.68 | 1006.22 | ✅ |
|
||||
| c1_i2048_o2048 | 1 | 2048 | 2048 | 43.35 | 20 | 0 | 0.46 | 944.86 | 101.45 | 1046.31 | 123.06 | 128.19 | 147.54 | 6.80 | 6.83 | 6.84 | 1611.88 | 2108.88 | 2496.01 | ✅ |
|
||||
| c1_i2048_o256 | 1 | 2048 | 256 | 37.22 | 20 | 0 | 0.54 | 1100.59 | 96.68 | 1197.27 | 122.80 | 125.69 | 145.02 | 6.81 | 6.82 | 6.83 | 1340.69 | 1621.05 | 1775.55 | ✅ |
|
||||
| c1_i2048_o4096 | 1 | 2048 | 4096 | 40.11 | 20 | 0 | 0.50 | 1021.08 | 101.61 | 1122.69 | 123.10 | 127.51 | 146.72 | 6.80 | 6.82 | 6.82 | 1502.66 | 2076.61 | 2137.65 | ✅ |
|
||||
| c1_i2048_o512 | 1 | 2048 | 512 | 37.02 | 20 | 0 | 0.54 | 1106.44 | 96.52 | 1202.96 | 122.80 | 127.33 | 146.10 | 6.81 | 6.82 | 6.82 | 1331.89 | 1558.94 | 1578.90 | ✅ |
|
||||
| c1_i4096_o1024 | 1 | 4096 | 1024 | 43.12 | 20 | 0 | 0.46 | 1899.75 | 99.83 | 1999.59 | 186.35 | 190.63 | 190.95 | 6.80 | 6.81 | 6.81 | 1644.18 | 2109.89 | 2311.93 | ✅ |
|
||||
| c1_i4096_o128 | 1 | 4096 | 128 | 28.91 | 20 | 0 | 0.69 | 2833.86 | 88.56 | 2922.42 | 185.86 | 190.38 | 191.25 | 6.75 | 6.76 | 6.76 | 1043.43 | 1045.36 | 1047.58 | ✅ |
|
||||
| c1_i4096_o2048 | 1 | 4096 | 2048 | 42.94 | 20 | 0 | 0.47 | 1907.88 | 96.51 | 2004.39 | 186.77 | 193.30 | 193.74 | 6.80 | 6.81 | 6.81 | 1589.25 | 2097.82 | 2452.99 | ✅ |
|
||||
| c1_i4096_o256 | 1 | 4096 | 256 | 40.43 | 20 | 0 | 0.49 | 2026.29 | 93.65 | 2119.94 | 189.05 | 207.59 | 207.83 | 6.80 | 6.81 | 6.82 | 1469.51 | 1913.76 | 1913.87 | ✅ |
|
||||
| c1_i4096_o4096 | 1 | 4096 | 4096 | 43.12 | 20 | 0 | 0.46 | 1899.82 | 96.29 | 1996.11 | 187.93 | 205.29 | 207.29 | 6.80 | 6.82 | 6.82 | 1593.63 | 2383.61 | 2729.94 | ✅ |
|
||||
| c1_i4096_o512 | 1 | 4096 | 512 | 41.37 | 20 | 0 | 0.48 | 1980.16 | 89.19 | 2069.35 | 186.00 | 187.35 | 190.66 | 6.81 | 6.81 | 6.82 | 1434.80 | 2102.19 | 2396.89 | ✅ |
|
||||
| c1_i512_o256 | 1 | 512 | 256 | 39.88 | 20 | 0 | 0.50 | 256.78 | 96.80 | 353.58 | 118.18 | 176.60 | 184.48 | 6.72 | 6.75 | 6.75 | 1408.44 | 1822.85 | 1871.30 | ✅ |
|
||||
| c2_i512_o256 | 2 | 512 | 256 | 41.58 | 40 | 0 | 0.96 | 492.58 | 196.05 | 688.62 | 173.55 | 211.08 | 357.86 | 7.61 | 8.16 | 8.40 | 1706.70 | 2108.24 | 2165.31 | ✅ |
|
||||
| c32_i1024_o1024 | 32 | 1024 | 1024 | 101.71 | 640 | 0 | 6.29 | 6443.13 | 1317.59 | 7760.73 | 197.93 | 311.27 | 984.94 | 22.03 | 23.78 | 24.21 | 4796.09 | 6445.23 | 7146.16 | ✅ |
|
||||
| c32_i1024_o128 | 32 | 1024 | 128 | 53.75 | 640 | 0 | 11.91 | 12192.47 | 1523.87 | 13716.35 | 681.10 | 1046.63 | 1091.53 | 14.34 | 16.98 | 18.60 | 2502.18 | 2579.44 | 2641.32 | ✅ |
|
||||
| c32_i1024_o2048 | 32 | 1024 | 2048 | 101.76 | 640 | 0 | 6.29 | 6440.44 | 1316.74 | 7757.18 | 204.19 | 335.85 | 1145.74 | 21.89 | 24.02 | 24.53 | 4767.74 | 6630.19 | 7109.51 | ✅ |
|
||||
| c32_i1024_o256 | 32 | 1024 | 256 | 99.60 | 640 | 0 | 6.43 | 6579.98 | 1309.14 | 7889.12 | 200.93 | 331.50 | 921.88 | 22.15 | 24.50 | 25.32 | 4694.16 | 6056.40 | 6274.46 | ✅ |
|
||||
| c32_i1024_o4096 | 32 | 1024 | 4096 | 103.10 | 640 | 0 | 6.21 | 6356.26 | 1309.63 | 7665.88 | 205.97 | 316.61 | 1157.52 | 22.04 | 24.05 | 24.79 | 4833.69 | 6458.13 | 7314.04 | ✅ |
|
||||
| c32_i1024_o512 | 32 | 1024 | 512 | 103.06 | 640 | 0 | 6.21 | 6359.18 | 1304.32 | 7663.50 | 196.61 | 340.27 | 945.16 | 22.38 | 25.38 | 27.97 | 4872.21 | 6788.10 | 7689.83 | ✅ |
|
||||
| c32_i16384_o1024 | 32 | 16384 | 1024 | 398.20 | 640 | 0 | 1.61 | 26333.15 | 402.85 | 26735.99 | 1772.26 | 3387.53 | 13518.46 | 70.46 | 85.49 | 101.84 | 19184.30 | 36508.49 | 54198.78 | ❌ |
|
||||
| c32_i16384_o128 | 32 | 16384 | 128 | 359.20 | 640 | 0 | 1.78 | 29191.93 | 227.55 | 29419.48 | 3438.60 | 3983.92 | 13516.26 | 111.13 | 127.95 | 130.66 | 17518.50 | 19731.45 | 27923.40 | ❌ |
|
||||
| c32_i16384_o2048 | 32 | 16384 | 2048 | 398.63 | 640 | 0 | 1.61 | 26304.81 | 403.52 | 26708.32 | 1715.04 | 2963.25 | 13514.42 | 70.47 | 86.65 | 95.98 | 19175.98 | 37536.35 | 50504.81 | ⚠️ |
|
||||
| c32_i16384_o256 | 32 | 16384 | 256 | 383.92 | 640 | 0 | 1.67 | 27312.05 | 348.16 | 27660.21 | 1826.81 | 3388.69 | 13517.47 | 80.79 | 92.93 | 101.63 | 18646.62 | 24405.54 | 29695.09 | ❌ |
|
||||
| c32_i16384_o512 | 32 | 16384 | 512 | 393.37 | 640 | 0 | 1.63 | 26656.12 | 383.78 | 27039.90 | 1747.04 | 3400.82 | 13513.48 | 73.78 | 90.30 | 96.96 | 19000.89 | 36338.55 | 41160.95 | ❌ |
|
||||
| c32_i2048_o1024 | 32 | 2048 | 1024 | 112.12 | 640 | 0 | 5.71 | 11690.22 | 1270.45 | 12960.67 | 246.53 | 373.10 | 1896.26 | 22.21 | 24.25 | 25.25 | 5165.11 | 8407.83 | 11262.12 | ✅ |
|
||||
| c32_i2048_o128 | 32 | 2048 | 128 | 73.91 | 640 | 0 | 8.66 | 17733.36 | 1105.41 | 18838.77 | 904.68 | 1464.61 | 1844.76 | 20.43 | 26.38 | 27.06 | 3492.80 | 3746.52 | 4348.11 | ✅ |
|
||||
| c32_i2048_o2048 | 32 | 2048 | 2048 | 111.79 | 640 | 0 | 5.73 | 11725.08 | 1282.42 | 13007.50 | 246.50 | 442.18 | 1847.04 | 22.17 | 24.28 | 25.62 | 5185.73 | 8810.37 | 10858.07 | ✅ |
|
||||
| c32_i2048_o256 | 32 | 2048 | 256 | 103.61 | 640 | 0 | 6.18 | 12650.11 | 1258.11 | 13908.22 | 240.98 | 367.88 | 1848.70 | 22.80 | 24.74 | 25.47 | 4859.84 | 6298.45 | 6687.99 | ✅ |
|
||||
| c32_i2048_o4096 | 32 | 2048 | 4096 | 106.84 | 640 | 0 | 5.99 | 12268.59 | 1282.15 | 13550.74 | 250.26 | 439.47 | 1837.79 | 22.45 | 24.65 | 25.84 | 5036.07 | 8060.13 | 10159.85 | ✅ |
|
||||
| c32_i2048_o512 | 32 | 2048 | 512 | 111.06 | 640 | 0 | 5.76 | 11802.41 | 1281.65 | 13084.06 | 242.26 | 437.24 | 1893.17 | 22.15 | 24.51 | 25.55 | 5136.64 | 8816.12 | 11310.11 | ✅ |
|
||||
| c32_i4096_o1024 | 32 | 4096 | 1024 | 141.80 | 640 | 0 | 4.51 | 18486.39 | 947.32 | 19433.71 | 432.85 | 813.49 | 3483.06 | 30.04 | 33.92 | 36.32 | 6702.72 | 11605.60 | 14590.75 | ✅ |
|
||||
| c32_i4096_o128 | 32 | 4096 | 128 | 114.34 | 640 | 0 | 5.60 | 22925.90 | 710.98 | 23636.88 | 1053.81 | 1909.50 | 3485.20 | 35.10 | 41.98 | 43.26 | 5478.89 | 6106.50 | 7458.03 | ✅ |
|
||||
| c32_i4096_o2048 | 32 | 4096 | 2048 | 144.45 | 640 | 0 | 4.43 | 18147.38 | 939.67 | 19087.05 | 434.39 | 747.31 | 3483.20 | 29.88 | 33.53 | 35.93 | 6700.04 | 10714.15 | 14733.26 | ✅ |
|
||||
| c32_i4096_o256 | 32 | 4096 | 256 | 137.28 | 640 | 0 | 4.66 | 19095.92 | 921.51 | 20017.42 | 441.00 | 731.75 | 3478.68 | 30.98 | 34.42 | 36.58 | 6528.50 | 8638.60 | 10034.50 | ✅ |
|
||||
| c32_i4096_o4096 | 32 | 4096 | 4096 | 143.01 | 640 | 0 | 4.48 | 18330.54 | 961.50 | 19292.04 | 429.61 | 749.16 | 3469.34 | 29.60 | 33.17 | 35.01 | 6743.49 | 10500.86 | 14281.37 | ✅ |
|
||||
| c32_i4096_o512 | 32 | 4096 | 512 | 143.19 | 640 | 0 | 4.47 | 18307.18 | 929.29 | 19236.47 | 430.74 | 691.68 | 3532.95 | 30.57 | 34.15 | 36.96 | 6735.35 | 10401.62 | 14892.06 | ✅ |
|
||||
| c32_i512_o256 | 32 | 512 | 256 | 98.73 | 640 | 0 | 6.48 | 3319.01 | 1330.82 | 4649.83 | 186.98 | 378.34 | 678.28 | 21.82 | 23.82 | 24.54 | 4649.84 | 6039.51 | 6163.96 | ✅ |
|
||||
| c4_i512_o256 | 4 | 512 | 256 | 46.00 | 80 | 0 | 1.74 | 890.42 | 360.47 | 1250.89 | 164.64 | 347.47 | 374.55 | 8.92 | 10.51 | 11.09 | 1997.44 | 2523.64 | 2675.43 | ✅ |
|
||||
| c64_i512_o256 | 64 | 512 | 256 | 140.30 | 1280 | 0 | 9.12 | 4671.07 | 1901.53 | 6572.60 | 229.06 | 453.78 | 1147.81 | 31.26 | 35.35 | 36.92 | 6712.71 | 8741.37 | 9237.35 | ✅ |
|
||||
| c8_i1024_o1024 | 8 | 1024 | 1024 | 54.01 | 160 | 0 | 2.96 | 3033.30 | 621.77 | 3655.07 | 144.65 | 218.00 | 394.00 | 10.80 | 11.82 | 12.24 | 2395.48 | 3214.61 | 3506.80 | ✅ |
|
||||
| c8_i1024_o128 | 8 | 1024 | 128 | 30.89 | 160 | 0 | 5.18 | 5304.66 | 662.82 | 5967.48 | 307.64 | 345.56 | 389.57 | 8.30 | 9.77 | 9.80 | 1360.75 | 1376.96 | 1412.43 | ✅ |
|
||||
| c8_i1024_o2048 | 8 | 1024 | 2048 | 53.44 | 160 | 0 | 2.99 | 3065.61 | 627.08 | 3692.69 | 147.14 | 284.31 | 375.40 | 10.76 | 11.60 | 12.31 | 2391.87 | 3190.58 | 3558.55 | ✅ |
|
||||
| c8_i1024_o256 | 8 | 1024 | 256 | 52.32 | 160 | 0 | 3.06 | 3131.63 | 620.55 | 3752.18 | 144.06 | 216.85 | 383.28 | 10.89 | 11.75 | 12.29 | 2345.43 | 2965.11 | 3042.16 | ✅ |
|
||||
| c8_i1024_o4096 | 8 | 1024 | 4096 | 55.32 | 160 | 0 | 2.89 | 2961.58 | 621.49 | 3583.07 | 144.43 | 220.03 | 380.43 | 10.73 | 11.56 | 12.09 | 2438.42 | 3382.06 | 3756.38 | ✅ |
|
||||
| c8_i1024_o512 | 8 | 1024 | 512 | 54.03 | 160 | 0 | 2.96 | 3032.14 | 632.21 | 3664.35 | 144.40 | 216.10 | 375.86 | 10.74 | 11.48 | 11.76 | 2425.82 | 3208.35 | 3616.77 | ✅ |
|
||||
| c8_i16384_o1024 | 8 | 16384 | 1024 | 138.97 | 160 | 0 | 1.15 | 18863.47 | 287.59 | 19151.06 | 967.72 | 1600.40 | 3663.33 | 21.47 | 27.32 | 29.59 | 6298.76 | 12516.84 | 14588.60 | ✅ |
|
||||
| c8_i16384_o128 | 8 | 16384 | 128 | 111.93 | 160 | 0 | 1.43 | 23420.44 | 181.33 | 23601.76 | 2556.64 | 4119.31 | 4136.10 | 20.74 | 33.97 | 34.00 | 5167.54 | 5417.75 | 5933.67 | ⚠️ |
|
||||
| c8_i16384_o2048 | 8 | 16384 | 2048 | 136.26 | 160 | 0 | 1.17 | 19238.95 | 275.01 | 19513.96 | 961.94 | 1572.87 | 3673.56 | 22.52 | 28.43 | 30.68 | 6163.97 | 11684.40 | 14272.99 | ✅ |
|
||||
| c8_i16384_o256 | 8 | 16384 | 256 | 127.82 | 160 | 0 | 1.25 | 20509.06 | 253.69 | 20762.74 | 1027.09 | 1938.52 | 3668.72 | 24.19 | 29.35 | 30.47 | 5890.64 | 7584.77 | 7970.70 | ✅ |
|
||||
| c8_i16384_o512 | 8 | 16384 | 512 | 136.61 | 160 | 0 | 1.17 | 19188.83 | 286.09 | 19474.92 | 969.45 | 1904.35 | 3667.98 | 21.65 | 26.89 | 29.25 | 6250.80 | 10966.73 | 12572.72 | ✅ |
|
||||
| c8_i2048_o1024 | 8 | 2048 | 1024 | 59.22 | 160 | 0 | 2.70 | 5533.53 | 606.82 | 6140.34 | 155.89 | 244.40 | 621.82 | 10.83 | 11.86 | 12.11 | 2572.18 | 4486.64 | 5823.30 | ✅ |
|
||||
| c8_i2048_o128 | 8 | 2048 | 128 | 36.79 | 160 | 0 | 4.35 | 8907.76 | 554.37 | 9462.13 | 337.85 | 507.02 | 613.47 | 10.27 | 12.09 | 12.51 | 1636.65 | 1709.00 | 1726.45 | ✅ |
|
||||
| c8_i2048_o2048 | 8 | 2048 | 2048 | 57.55 | 160 | 0 | 2.78 | 5693.86 | 626.97 | 6320.83 | 156.06 | 275.62 | 603.04 | 10.76 | 11.60 | 11.99 | 2571.87 | 4101.50 | 5201.48 | ✅ |
|
||||
| c8_i2048_o256 | 8 | 2048 | 256 | 52.19 | 160 | 0 | 3.07 | 6279.15 | 604.00 | 6883.15 | 158.73 | 268.03 | 611.54 | 11.13 | 12.15 | 12.45 | 2334.26 | 3004.26 | 3124.59 | ✅ |
|
||||
| c8_i2048_o4096 | 8 | 2048 | 4096 | 59.22 | 160 | 0 | 2.70 | 5533.71 | 614.83 | 6148.54 | 164.46 | 304.53 | 685.46 | 10.84 | 11.77 | 11.98 | 2624.68 | 4094.22 | 5661.19 | ✅ |
|
||||
| c8_i2048_o512 | 8 | 2048 | 512 | 57.76 | 160 | 0 | 2.77 | 5672.85 | 605.80 | 6278.66 | 154.18 | 248.11 | 617.88 | 10.86 | 11.81 | 12.00 | 2521.32 | 3808.31 | 4671.03 | ✅ |
|
||||
| c8_i4096_o1024 | 8 | 4096 | 1024 | 64.92 | 160 | 0 | 2.46 | 10094.27 | 518.13 | 10612.40 | 256.33 | 442.48 | 1016.15 | 12.36 | 13.77 | 14.46 | 2846.88 | 4602.44 | 5170.71 | ✅ |
|
||||
| c8_i4096_o128 | 8 | 4096 | 128 | 46.47 | 160 | 0 | 3.44 | 14102.60 | 440.04 | 14542.64 | 707.03 | 1047.28 | 1071.50 | 10.92 | 15.03 | 15.06 | 2091.88 | 2145.56 | 2225.52 | ✅ |
|
||||
| c8_i4096_o2048 | 8 | 4096 | 2048 | 66.29 | 160 | 0 | 2.41 | 9886.22 | 513.33 | 10399.56 | 253.25 | 423.32 | 1022.59 | 12.39 | 14.05 | 14.66 | 2865.36 | 4853.71 | 6028.24 | ✅ |
|
||||
| c8_i4096_o256 | 8 | 4096 | 256 | 62.26 | 160 | 0 | 2.57 | 10526.27 | 516.50 | 11042.77 | 256.86 | 457.13 | 1013.55 | 12.63 | 14.05 | 14.57 | 2789.27 | 3625.36 | 3833.62 | ✅ |
|
||||
| c8_i4096_o4096 | 8 | 4096 | 4096 | 64.28 | 160 | 0 | 2.49 | 10195.14 | 543.84 | 10738.99 | 248.19 | 462.57 | 1014.75 | 12.27 | 13.68 | 14.32 | 2921.13 | 5570.16 | 8112.52 | ✅ |
|
||||
| c8_i4096_o512 | 8 | 4096 | 512 | 65.25 | 160 | 0 | 2.45 | 10043.48 | 538.74 | 10582.22 | 247.69 | 442.69 | 1015.54 | 12.25 | 13.71 | 14.06 | 2931.06 | 5124.89 | 6513.32 | ✅ |
|
||||
| c8_i512_o256 | 8 | 512 | 256 | 52.73 | 160 | 0 | 3.03 | 1553.68 | 629.48 | 2183.16 | 135.86 | 271.88 | 289.07 | 10.80 | 11.89 | 12.07 | 2369.82 | 3039.64 | 3126.47 | ✅ |
|
||||
| c128_i16384_o1024 | 128 | 16384 | 1024 | 1401.24 | 2560 | 0 | 1.83 | 29932.73 | 453.17 | 30385.90 | 6827.03 | 13719.55 | 53845.73 | 254.25 | 266.59 | 267.01 | 69085.30 | 135447.23 | 185844.63 | ❌ |
|
||||
| c128_i16384_o2048 | 128 | 16384 | 2048 | 1400.62 | 2560 | 0 | 1.83 | 29945.97 | 468.45 | 30414.41 | 5874.33 | 13475.14 | 53934.11 | 249.38 | 266.26 | 266.71 | 69277.40 | 141505.20 | 189264.63 | ❌ |
|
||||
| c1_i131072_o128 | 1 | 131072 | 128 | 182.19 | 20 | 0 | 0.11 | 14388.29 | 12.72 | 14401.01 | 5392.41 | 5419.10 | 5420.54 | 6.96 | 6.97 | 6.98 | 6192.49 | 6221.42 | 6221.43 | ⚠️ |
|
||||
| c1_i131072_o256 | 1 | 131072 | 256 | 195.04 | 20 | 0 | 0.10 | 13440.21 | 19.01 | 13459.22 | 5397.54 | 5431.66 | 5440.22 | 7.00 | 7.03 | 7.03 | 6686.72 | 7103.29 | 7119.59 | ⚠️ |
|
||||
| c1_i131072_o512 | 1 | 131072 | 512 | 201.48 | 20 | 0 | 0.10 | 13010.69 | 20.19 | 13030.87 | 5500.61 | 5555.72 | 5562.25 | 7.00 | 7.02 | 7.03 | 6917.34 | 7364.34 | 8109.96 | ⚠️ |
|
||||
| c1_i262144_o256 | 1 | 262144 | 256 | 426.96 | 20 | 0 | 0.05 | 12279.65 | 8.19 | 12287.84 | 13218.70 | 13504.54 | 13586.50 | 7.22 | 7.32 | 7.36 | 14470.75 | 14834.79 | 14835.15 | ⚠️ |
|
||||
| c1_i524288_o128 | 1 | 524288 | 128 | 1098.28 | 20 | 0 | 0.02 | 9547.46 | 1.62 | 9549.08 | 36239.34 | 36595.44 | 37102.25 | 9.37 | 9.80 | 37.00 | 36907.34 | 37308.40 | 37807.76 | ⚠️ |
|
||||
| c1_i65536_o1024 | 1 | 65536 | 1024 | 114.83 | 20 | 0 | 0.17 | 11414.14 | 39.37 | 11453.51 | 2462.78 | 2475.26 | 2495.15 | 6.92 | 6.93 | 6.93 | 4020.21 | 5428.03 | 7058.64 | ✅ |
|
||||
| c1_i65536_o128 | 1 | 65536 | 128 | 96.37 | 20 | 0 | 0.21 | 13600.71 | 24.89 | 13625.61 | 2497.10 | 2528.43 | 2558.93 | 6.87 | 6.87 | 6.88 | 3313.69 | 3344.68 | 3373.65 | ✅ |
|
||||
| c1_i65536_o256 | 1 | 65536 | 256 | 105.96 | 20 | 0 | 0.19 | 12370.49 | 32.66 | 12403.15 | 2460.46 | 2472.33 | 2483.17 | 6.92 | 6.93 | 6.93 | 3650.21 | 4159.14 | 4159.75 | ✅ |
|
||||
| c1_i65536_o512 | 1 | 65536 | 512 | 108.81 | 20 | 0 | 0.18 | 12045.94 | 34.57 | 12080.52 | 2461.82 | 2468.77 | 2498.35 | 6.92 | 6.93 | 6.93 | 3756.97 | 4315.50 | 4593.51 | ✅ |
|
||||
| c32_i65536_o1024 | 32 | 65536 | 1024 | 395.45 | 160 | 0 | 0.40 | 26515.91 | 89.22 | 26605.13 | 18530.86 | 54451.46 | 68783.28 | 252.00 | 283.71 | 284.41 | 72731.85 | 117936.92 | 151338.20 | ❌ |
|
||||
| c8_i131072_o512 | 8 | 131072 | 512 | 887.91 | 160 | 0 | 0.18 | 23619.01 | 33.13 | 23652.14 | 12820.61 | 25478.95 | 32945.62 | 156.62 | 223.19 | 294.51 | 41479.53 | 66809.84 | 79279.57 | ❌ |
|
||||
| c8_i262144_o128 | 8 | 262144 | 128 | 2121.21 | 160 | 0 | 0.08 | 19773.16 | 9.48 | 19782.64 | 51391.65 | 61587.85 | 80661.75 | 382.04 | 388.68 | 389.24 | 99110.05 | 110875.08 | 129961.05 | ❌ |
|
||||
| c8_i65536_o1024 | 8 | 65536 | 1024 | 422.41 | 160 | 0 | 0.38 | 24823.70 | 76.89 | 24900.59 | 4665.85 | 8696.15 | 14818.98 | 73.86 | 99.00 | 113.60 | 19699.15 | 35998.21 | 56237.93 | ❌ |
|
||||
| c8_i65536_o128 | 8 | 65536 | 128 | 402.35 | 160 | 0 | 0.40 | 26061.00 | 46.48 | 26107.48 | 8659.91 | 13193.45 | 15765.71 | 87.87 | 143.78 | 166.11 | 18805.63 | 21397.73 | 22483.11 | ❌ |
|
||||
| c8_i65536_o256 | 8 | 65536 | 256 | 416.75 | 160 | 0 | 0.38 | 25160.80 | 67.25 | 25228.06 | 5113.03 | 10698.95 | 14785.06 | 82.53 | 113.13 | 132.88 | 19445.90 | 27248.32 | 32918.89 | ❌ |
|
||||
| c8_i65536_o512 | 8 | 65536 | 512 | 422.66 | 160 | 0 | 0.38 | 24809.03 | 76.23 | 24885.26 | 4532.43 | 9015.11 | 14789.72 | 75.30 | 108.22 | 126.16 | 19689.27 | 34072.42 | 42942.27 | ❌ |
|
||||
|
||||
SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
271
experiments/dsv4_h200_vllm_tp8_custom_bench/run_bench.sh
Executable file
271
experiments/dsv4_h200_vllm_tp8_custom_bench/run_bench.sh
Executable file
@ -0,0 +1,271 @@
|
||||
#!/usr/bin/env bash
|
||||
# Custom benchmark orchestrator for single-service vLLM TP=8 on 8x H200.
|
||||
#
|
||||
# This runs 1 vLLM service (TP=8) on all 8 GPUs (0,1,2,3,4,5,6,7).
|
||||
# bench_client.py sends requests to the single service.
|
||||
#
|
||||
# Usage:
|
||||
# bash run_bench.sh
|
||||
# SKIP_MANAGE_SERVER=1 bash run_bench.sh # skip server start/stop
|
||||
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||
RESULT_ROOT="${RESULT_ROOT:-${SCRIPT_DIR}/results/${RUN_ID}}"
|
||||
RAW_DIR="${RESULT_ROOT}/raw_outputs"
|
||||
LOG_DIR="${RESULT_ROOT}/logs"
|
||||
|
||||
ensure_result_root "$RESULT_ROOT"
|
||||
log_init "${LOG_DIR}/orchestrator.log"
|
||||
|
||||
log "experiment=${EXPERIMENT_NAME}"
|
||||
log "run_id=${RUN_ID}"
|
||||
log "result_root=${RESULT_ROOT}"
|
||||
log "platform=${PLATFORM}"
|
||||
log "chip=${CHIP}"
|
||||
log "accelerator=${ACCELERATOR}"
|
||||
log "engine=${ENGINE}"
|
||||
log "hardware=${HARDWARE}"
|
||||
log "model=${MODEL_PATH}"
|
||||
log "services=${PORTS}"
|
||||
log "lb_strategy=${LB_STRATEGY}"
|
||||
log "server_start_script=${SERVER_START_SCRIPT}"
|
||||
|
||||
# Write initial metadata for this run.
|
||||
METADATA_JSON="${RESULT_ROOT}/results.json"
|
||||
write_metadata_json \
|
||||
"$METADATA_JSON" \
|
||||
"$EXPERIMENT_NAME" \
|
||||
"$RUN_ID" \
|
||||
"$MODEL_PATH" \
|
||||
"$BACKEND" \
|
||||
"$ENGINE" \
|
||||
"$HARDWARE" \
|
||||
"$ACCELERATOR" \
|
||||
"$CHIP" \
|
||||
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||
"$VENV_SERVER" \
|
||||
"H200 1x vLLM TP=8 single-service benchmark using bench_client.py"
|
||||
|
||||
# Update metadata with config.
|
||||
"${VENV_CLIENT}/bin/python" - "$METADATA_JSON" <<'PY'
|
||||
import json, sys
|
||||
path = sys.argv[1]
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
data["config"] = {
|
||||
"tp": 8,
|
||||
"num_services": 1,
|
||||
"ports": "30005",
|
||||
"gpu_groups": "0,1,2,3,4,5,6,7",
|
||||
"kv_cache_dtype": "fp8",
|
||||
"block_size": 256,
|
||||
"client": "bench_client.py",
|
||||
"lb_strategy": "round_robin",
|
||||
"scenarios": [
|
||||
"1 512 256 20", "2 512 256 40", "4 512 256 80",
|
||||
"8 512 256 160", "16 512 256 320", "32 512 256 640",
|
||||
"64 512 256 1280", "128 512 256 2560",
|
||||
"1 2048 512 20", "8 2048 512 160", "32 2048 512 640", "128 2048 512 2560",
|
||||
"1 4096 1024 20", "8 4096 1024 160", "32 4096 1024 640", "128 4096 1024 2560"
|
||||
]
|
||||
}
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||
PY
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server helpers (single-service)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
is_server_healthy() {
|
||||
local port="$1"
|
||||
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${port}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
are_all_services_healthy() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if ! is_server_healthy "$p"; then
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
stop_all_servers() {
|
||||
log "stopping all vllm services"
|
||||
# Kill by PID files first.
|
||||
for pid_file in "${SCRIPT_DIR}"/server_port*.pid; do
|
||||
[[ -f "$pid_file" ]] || continue
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "killing pid=${pid}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 2
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
done
|
||||
# Fallback: kill any remaining vllm processes for this model.
|
||||
pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
sleep 2
|
||||
}
|
||||
|
||||
start_all_servers() {
|
||||
log "starting all vllm services with ${SERVER_START_SCRIPT}"
|
||||
if [[ ! -x "${SERVER_START_SCRIPT}" ]]; then
|
||||
log "error: start script not found or not executable: ${SERVER_START_SCRIPT}"
|
||||
exit 1
|
||||
fi
|
||||
bash "${SERVER_START_SCRIPT}" >> "${LOG_DIR}/server.outer.log" 2>&1
|
||||
|
||||
if ! are_all_services_healthy "$PORTS"; then
|
||||
log "error: not all vllm services became healthy"
|
||||
exit 1
|
||||
fi
|
||||
log "all vllm services are healthy"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pre-benchmark health check with detailed reporting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
check_services_detailed() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
local healthy_count=0
|
||||
local unhealthy_ports=""
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if is_server_healthy "$p"; then
|
||||
((healthy_count++))
|
||||
else
|
||||
unhealthy_ports="${unhealthy_ports}${p} "
|
||||
fi
|
||||
done
|
||||
echo "$healthy_count"
|
||||
if [[ -n "$unhealthy_ports" ]]; then
|
||||
echo "unhealthy: ${unhealthy_ports}" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
wait_for_all_services() {
|
||||
local ports="$1"
|
||||
local max_wait="${2:-240}"
|
||||
local interval="${3:-5}"
|
||||
local elapsed=0
|
||||
while (( elapsed < max_wait )); do
|
||||
local healthy_count
|
||||
healthy_count="$(check_services_detailed "$ports" | head -1)"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
if (( healthy_count == ${#PORT_LIST[@]} )); then
|
||||
return 0
|
||||
fi
|
||||
log "waiting for services... ${healthy_count}/${#PORT_LIST[@]} healthy after ${elapsed}s"
|
||||
sleep "$interval"
|
||||
((elapsed += interval))
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Mid-benchmark health check (called between scenarios)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
check_and_report_services() {
|
||||
local ports="$1"
|
||||
IFS=',' read -ra PORT_LIST <<< "$ports"
|
||||
local healthy=()
|
||||
local unhealthy=()
|
||||
for p in "${PORT_LIST[@]}"; do
|
||||
if is_server_healthy "$p"; then
|
||||
healthy+=("$p")
|
||||
else
|
||||
unhealthy+=("$p")
|
||||
fi
|
||||
done
|
||||
log "service health check: healthy=[${healthy[*]}] unhealthy=[${unhealthy[*]}]"
|
||||
if [[ ${#unhealthy[@]} -gt 0 ]]; then
|
||||
log "WARNING: ${#unhealthy[@]} service(s) unhealthy: ${unhealthy[*]}"
|
||||
fi
|
||||
return ${#unhealthy[@]}
|
||||
}
|
||||
|
||||
on_exit() {
|
||||
local code=$?
|
||||
log "orchestrator exiting with code=${code}"
|
||||
if [[ -z "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||
stop_all_servers
|
||||
fi
|
||||
exit "$code"
|
||||
}
|
||||
trap on_exit EXIT
|
||||
|
||||
if [[ -n "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||
log "SKIP_MANAGE_SERVER is set, assuming services are already running"
|
||||
if ! wait_for_all_services "$PORTS"; then
|
||||
log "error: not all healthy services found on ports ${PORTS}"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
stop_all_servers
|
||||
start_all_servers
|
||||
fi
|
||||
|
||||
log "===== BENCHMARK START ====="
|
||||
|
||||
for scenario in "${SCENARIOS[@]}"; do
|
||||
read -r concurrency input_len output_len num_prompts <<< "$scenario"
|
||||
# Fallback for legacy 3-field scenarios.
|
||||
if [[ -z "${num_prompts:-}" ]]; then
|
||||
num_prompts="$NUM_PROMPTS"
|
||||
fi
|
||||
|
||||
output_file="${RAW_DIR}/vllm_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||
detail_log="${LOG_DIR}/vllm_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||
|
||||
log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len} num_prompts=${num_prompts}"
|
||||
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/bench_client.py" \
|
||||
--host 127.0.0.1 \
|
||||
--port "$PORT" \
|
||||
--model "$SERVED_MODEL_NAME" \
|
||||
--concurrency "$concurrency" \
|
||||
--input-len "$input_len" \
|
||||
--output-len "$output_len" \
|
||||
--num-prompts "$num_prompts" \
|
||||
--output-file "$output_file" \
|
||||
> "$detail_log" 2>&1 || {
|
||||
log "ERROR: scenario c=${concurrency} i=${input_len} o=${output_len} failed; see ${detail_log}"
|
||||
# After failure, check service health before continuing.
|
||||
check_and_report_services "$PORTS"
|
||||
continue
|
||||
}
|
||||
|
||||
log "finished scenario: output=${output_file}"
|
||||
|
||||
# Between scenarios, report service health so we can spot degradation early.
|
||||
check_and_report_services "$PORTS" || true
|
||||
done
|
||||
|
||||
log "===== BENCHMARK DONE ====="
|
||||
|
||||
log "parsing results"
|
||||
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/parse_results.py" "$RESULT_ROOT" >> "${LOG_DIR}/parse.log" 2>&1 || {
|
||||
log "WARNING: parser failed; see ${LOG_DIR}/parse.log"
|
||||
}
|
||||
|
||||
log "all results saved to ${RESULT_ROOT}"
|
||||
107
experiments/dsv4_h200_vllm_tp8_custom_bench/start_server.sh
Executable file
107
experiments/dsv4_h200_vllm_tp8_custom_bench/start_server.sh
Executable file
@ -0,0 +1,107 @@
|
||||
#!/bin/bash
|
||||
# Start 1 vLLM TP=8 service on 8x H200.
|
||||
# Uses all 8 GPUs: (0,1,2,3,4,5,6,7).
|
||||
# This maximizes single-service throughput for highest-concurrency workloads.
|
||||
set -e
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
VENV="${VENV_SERVER}"
|
||||
export PATH="$VENV/bin:$PATH"
|
||||
VLLM="$VENV/bin/vllm"
|
||||
|
||||
TP=8
|
||||
LOG_DIR="${SCRIPT_DIR}/logs"
|
||||
mkdir -p "$LOG_DIR"
|
||||
|
||||
# 1 service on all 8 GPUs.
|
||||
# Port and GPU mapping.
|
||||
SERVICES=(
|
||||
"30005:0,1,2,3,4,5,6,7"
|
||||
)
|
||||
|
||||
# Kill any existing vLLM processes for this model.
|
||||
pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
sleep 2
|
||||
|
||||
# Remove old PID files.
|
||||
rm -f "${SCRIPT_DIR}"/*.pid
|
||||
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
GPUS="${svc##*:}"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_$(date +%Y%m%d_%H%M%S).log"
|
||||
PID_FILE="${SCRIPT_DIR}/server_port${PORT}.pid"
|
||||
|
||||
echo "=== Starting service on port ${PORT}, GPUs ${GPUS} ==="
|
||||
echo "Log: ${LOG}"
|
||||
|
||||
CUDA_VISIBLE_DEVICES="${GPUS}" \
|
||||
nohup "$VLLM" serve "$MODEL_PATH" \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size "$TP" \
|
||||
--kv-cache-dtype fp8 \
|
||||
--block-size 256 \
|
||||
--served-model-name "$SERVED_MODEL_NAME" \
|
||||
--port "$PORT" \
|
||||
> "$LOG" 2>&1 &
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Waiting for service to become healthy..."
|
||||
|
||||
MAX_WAIT=240
|
||||
all_healthy=0
|
||||
for i in $(seq 1 $MAX_WAIT); do
|
||||
all_healthy=1
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
if ! curl -s "http://127.0.0.1:${PORT}/health" > /dev/null 2>&1; then
|
||||
all_healthy=0
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ "$all_healthy" -eq 1 ]]; then
|
||||
echo "Service is healthy!"
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
echo " - http://127.0.0.1:${PORT}"
|
||||
done
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Check if any process died early.
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
PID_FILE="${SCRIPT_DIR}/server_port${PORT}.pid"
|
||||
if [[ -f "$PID_FILE" ]]; then
|
||||
PID="$(cat "$PID_FILE")"
|
||||
if ! kill -0 "$PID" 2>/dev/null; then
|
||||
echo "ERROR: Service on port ${PORT} (PID ${PID}) exited early"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_*.log"
|
||||
tail -200 $LOG 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
if (( i % 10 == 0 )); then
|
||||
echo "Waiting... (${i}/${MAX_WAIT})"
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: Service did not become healthy after ${MAX_WAIT} retries"
|
||||
for svc in "${SERVICES[@]}"; do
|
||||
PORT="${svc%%:*}"
|
||||
LOG="${LOG_DIR}/server_port${PORT}_*.log"
|
||||
echo "--- Last 50 lines of port ${PORT} ---"
|
||||
tail -50 $LOG 2>/dev/null || true
|
||||
done
|
||||
exit 1
|
||||
@ -0,0 +1,115 @@
|
||||
# vLLM Adaptive Concurrency Search
|
||||
|
||||
`run_adaptive_concurrency.sh` keeps the existing ISL/DSL and TP/DP matrix, but
|
||||
searches concurrency at runtime instead of testing only low/high endpoints.
|
||||
|
||||
## Search rule
|
||||
|
||||
For every fixed `(TP, DP, ISL, DSL)`:
|
||||
|
||||
```text
|
||||
C = 1, 2, 4, 8, ... up to SEARCH_MAX_CONCURRENCY
|
||||
num_prompts = C * NUM_PROMPTS_MULTIPLIER
|
||||
```
|
||||
|
||||
The search stops when Total TPS grows by less than `TPS_MIN_GAIN_PCT` for
|
||||
`PLATEAU_PATIENCE` consecutive points. Defaults are 2% and two points.
|
||||
|
||||
- `saturation_concurrency`: first point in the final low-gain streak
|
||||
- `best_tps_concurrency`: tested point with the highest Total TPS
|
||||
- `stop_probe_concurrency`: last point tested to confirm the stop
|
||||
|
||||
OOM stops only the current shape. The service is restarted before the next
|
||||
shape. A non-OOM failure is retried once after a service restart.
|
||||
|
||||
Before every measured point, the client sends `C` concurrent warm-up requests.
|
||||
This keeps lazy kernel compilation and CUDA graph capture out of the measured
|
||||
TTFT/TPS. `BENCH_WARMUP_MAX_REQUESTS=0` means no cap; use a positive value only
|
||||
when deliberately limiting warm-up at very high concurrency.
|
||||
|
||||
## Dataset requirement
|
||||
|
||||
The default remains compatible with the fixed matrix baseline:
|
||||
|
||||
```text
|
||||
--dataset-name random --dataset-path DATASET_PATH
|
||||
--random-range-ratio 1.0
|
||||
```
|
||||
|
||||
In the installed SGLang client, the upstream default `0.0` samples each length
|
||||
uniformly from `1` to the requested maximum. This adaptive experiment sets
|
||||
`1.0` explicitly so every point represents the fixed ISL/DSL shown in the
|
||||
matrix. Override `RANDOM_RANGE_RATIO` only when a mixed-length workload is
|
||||
intentional.
|
||||
|
||||
`DATASET_PATH` must have at least
|
||||
`SEARCH_MAX_CONCURRENCY * NUM_PROMPTS_MULTIPLIER` usable two-turn
|
||||
conversations. The current `runtime/dummy_sharegpt.json` has only two and is
|
||||
not valid for a real adaptive run.
|
||||
|
||||
For setup smoke tests without ShareGPT, explicitly use:
|
||||
|
||||
```bash
|
||||
BENCH_DATASET_NAME=random-ids
|
||||
```
|
||||
|
||||
This sends generated token IDs with `--tokenize-prompt`; it is convenient for
|
||||
smoke tests but changes the token distribution, so do not mix it with random
|
||||
ShareGPT-backed baseline results.
|
||||
|
||||
## Commands
|
||||
|
||||
Print the plan without starting a server:
|
||||
|
||||
```bash
|
||||
DRY_RUN=1 bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
Run one small search:
|
||||
|
||||
```bash
|
||||
TP_LIST="8" \
|
||||
ISL_LIST="1024" \
|
||||
DSL_LIST="128" \
|
||||
GRID_LIMIT=1 \
|
||||
SEARCH_MAX_CONCURRENCY=8 \
|
||||
DATASET_PATH=/path/to/ShareGPT.json \
|
||||
bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
Run the full search in tmux:
|
||||
|
||||
```bash
|
||||
tmux new -s vllm-adaptive
|
||||
DATASET_PATH=/path/to/ShareGPT.json bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
Tune the stop rule when needed:
|
||||
|
||||
```bash
|
||||
TPS_MIN_GAIN_PCT=3.0 \
|
||||
PLATEAU_PATIENCE=2 \
|
||||
SEARCH_MAX_CONCURRENCY=512 \
|
||||
bash run_adaptive_concurrency.sh
|
||||
```
|
||||
|
||||
## Outputs
|
||||
|
||||
Results are written to `adaptive_results/<run_id>/`:
|
||||
|
||||
```text
|
||||
adaptive_points.jsonl
|
||||
adaptive_points.csv
|
||||
adaptive_summary.jsonl
|
||||
adaptive_summary.csv
|
||||
adaptive_summary.md
|
||||
run_manifest.json
|
||||
tp*/raw_outputs/
|
||||
tp*/metrics/
|
||||
tp*/logs/
|
||||
tp*/gpu_logs/
|
||||
```
|
||||
|
||||
Every successful point is accepted only when `completed == num_prompts` and
|
||||
actual input/output lengths match the requested shape within configured
|
||||
tolerances. Invalid workloads never participate in saturation decisions.
|
||||
49
experiments/dsv4_h200_vllm_tp_dp_matrix/adaptive_config.env
Normal file
49
experiments/dsv4_h200_vllm_tp_dp_matrix/adaptive_config.env
Normal file
@ -0,0 +1,49 @@
|
||||
# Adaptive concurrency search settings.
|
||||
#
|
||||
# For each fixed (TP, DP, ISL, OSL), probe:
|
||||
# C = start, start * multiplier, ... up to max
|
||||
# and stop after Total TPS has less than TPS_MIN_GAIN_PCT meaningful growth for
|
||||
# PLATEAU_PATIENCE consecutive points.
|
||||
|
||||
SEARCH_START_CONCURRENCY="${SEARCH_START_CONCURRENCY:-1}"
|
||||
SEARCH_MAX_CONCURRENCY="${SEARCH_MAX_CONCURRENCY:-512}"
|
||||
SEARCH_MULTIPLIER="${SEARCH_MULTIPLIER:-2}"
|
||||
NUM_PROMPTS_MULTIPLIER="${NUM_PROMPTS_MULTIPLIER:-5}"
|
||||
|
||||
# A gain below 2% is treated as throughput saturation. Two consecutive
|
||||
# low-gain points prevent one noisy measurement from stopping the search.
|
||||
TPS_MIN_GAIN_PCT="${TPS_MIN_GAIN_PCT:-2.0}"
|
||||
PLATEAU_PATIENCE="${PLATEAU_PATIENCE:-2}"
|
||||
|
||||
# Keep the same random workload semantics as the fixed matrix baseline.
|
||||
# DATASET_PATH must contain at least SEARCH_MAX_CONCURRENCY times
|
||||
# NUM_PROMPTS_MULTIPLIER valid two-turn conversations. Set this explicitly to
|
||||
# random-ids to use generated token IDs without a ShareGPT seed dataset.
|
||||
BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}"
|
||||
# SGLang interprets 0.0 as Uniform[1, requested_len]. Use 1.0 for fixed
|
||||
# ISL/OSL points; lower values intentionally benchmark a length distribution.
|
||||
RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1.0}"
|
||||
# Before each measured point, warm up with the same concurrency so lazy kernel
|
||||
# compilation and CUDA graph capture are excluded from TTFT/TPS. 0 means no
|
||||
# cap; set a positive cap only when very high-concurrency warmup is impractical.
|
||||
BENCH_WARMUP_MAX_REQUESTS="${BENCH_WARMUP_MAX_REQUESTS:-0}"
|
||||
|
||||
# Reject a point if the completed request count or actual token lengths do not
|
||||
# match the requested workload.
|
||||
INPUT_LENGTH_TOLERANCE_PCT="${INPUT_LENGTH_TOLERANCE_PCT:-5.0}"
|
||||
OUTPUT_LENGTH_TOLERANCE_PCT="${OUTPUT_LENGTH_TOLERANCE_PCT:-10.0}"
|
||||
|
||||
MAX_POINT_RETRIES="${MAX_POINT_RETRIES:-1}"
|
||||
SERVER_RESTART_COOLDOWN_S="${SERVER_RESTART_COOLDOWN_S:-10}"
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||
|
||||
# Optional space-separated filters, useful for smoke tests:
|
||||
# TP_LIST="8" ISL_LIST="1024" OSL_LIST="128"
|
||||
TP_LIST="${TP_LIST:-}"
|
||||
ISL_LIST="${ISL_LIST:-}"
|
||||
OSL_LIST="${OSL_LIST:-}"
|
||||
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
# Counts ISL/OSL shapes per TP/DP config, not individual concurrency probes.
|
||||
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||
163
experiments/dsv4_h200_vllm_tp_dp_matrix/compare.py
Executable file
163
experiments/dsv4_h200_vllm_tp_dp_matrix/compare.py
Executable file
@ -0,0 +1,163 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cross TP×DP configuration comparison for dsv4_h200_vllm_tp_dp_matrix.
|
||||
|
||||
Usage:
|
||||
python3 compare.py --run-root results/<run_id> [--output comparison.md]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_result(result_root: Path) -> dict:
|
||||
path = result_root / "results.json"
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def slo_status(ttft_p95_ms: float, tpot_mean_ms: float,
|
||||
ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str:
|
||||
ttft_ok = ttft_p95_ms < ttft_limit_ms
|
||||
tpot_ok = tpot_mean_ms < tpot_limit_ms
|
||||
if ttft_ok and tpot_ok:
|
||||
return "PASS"
|
||||
if ttft_ok or tpot_ok:
|
||||
return "PARTIAL"
|
||||
return "FAIL"
|
||||
|
||||
|
||||
def gpu_memory_str(gpu: dict | None) -> str:
|
||||
if not gpu:
|
||||
return "-"
|
||||
peak = gpu.get("peak_used_mb", 0)
|
||||
total = gpu.get("memory_total_mb", 0)
|
||||
if total:
|
||||
return f"{peak:.0f}/{total:.0f} ({100*peak/total:.1f}%)"
|
||||
return f"{peak:.0f}"
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--run-root", type=Path, required=True)
|
||||
parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md"))
|
||||
parser.add_argument("--ttft-limit", type=float, default=3000.0)
|
||||
parser.add_argument("--tpot-limit", type=float, default=50.0)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Discover configurations: tp*_dp* directories.
|
||||
configs = []
|
||||
for subdir in sorted(args.run_root.iterdir()):
|
||||
if not subdir.is_dir():
|
||||
continue
|
||||
name = subdir.name
|
||||
if not (name.startswith("tp") and "_dp" in name):
|
||||
continue
|
||||
results_json = subdir / "results.json"
|
||||
if not results_json.exists():
|
||||
continue
|
||||
configs.append((name, load_result(subdir)))
|
||||
|
||||
if not configs:
|
||||
print(f"No tp*_dp* results found under {args.run_root}")
|
||||
return
|
||||
|
||||
model = configs[0][1].get("metadata", {}).get("model", "unknown")
|
||||
hardware = configs[0][1].get("metadata", {}).get("hardware", "unknown")
|
||||
|
||||
# Group by scenario name.
|
||||
by_scenario: dict[str, dict[str, dict]] = defaultdict(dict)
|
||||
skipped: dict[str, dict[str, str]] = defaultdict(dict)
|
||||
for label, data in configs:
|
||||
for s in data.get("scenarios", []):
|
||||
key = s["name"]
|
||||
if s.get("status") == "skipped_oom":
|
||||
skipped[key][label] = s.get("note", "skipped")
|
||||
else:
|
||||
by_scenario[key][label] = s
|
||||
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
f.write(f"# vLLM TP×DP matrix comparison ({hardware})\n\n")
|
||||
f.write("## Summary\n\n")
|
||||
f.write(f"- Model: `{model}`\n")
|
||||
f.write(f"- Hardware: {hardware}\n")
|
||||
f.write("- Backend: vLLM (Docker)\n")
|
||||
f.write("- Benchmark client: `sglang.bench_serving`\n")
|
||||
f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n")
|
||||
|
||||
# Configuration overview.
|
||||
f.write("### Configurations\n\n")
|
||||
f.write("| Config | TP | DP | GPUs/replica | Notes |\n")
|
||||
f.write("|---|---:|---:|---:|---|\n")
|
||||
for label, data in configs:
|
||||
cfg = data.get("config", {})
|
||||
tp = cfg.get("tp", "?")
|
||||
dp = cfg.get("dp", "?")
|
||||
f.write(f"| {label} | {tp} | {dp} | {tp} | server args recorded per ISL in results.json |\n")
|
||||
f.write("\n")
|
||||
|
||||
# Side-by-side table.
|
||||
f.write("## Side-by-side results\n\n")
|
||||
headers = [
|
||||
"Scenario", "ISL", "DSL", "Config", "Conc", "Req/s", "OutTok/s",
|
||||
"TTFT P95(ms)", "TTFT P99(ms)", "TPOT Mean(ms)", "TPOT P95(ms)",
|
||||
"TPOT P99(ms)", "E2E P99(ms)", "Peak GPU mem", "SLO"
|
||||
]
|
||||
f.write("| " + " | ".join(headers) + " |\n")
|
||||
f.write("|" + "|".join(["---"] * len(headers)) + "|\n")
|
||||
|
||||
for scenario_name in sorted(by_scenario.keys(), key=lambda x: tuple(map(int, re.findall(r"\d+", x)))):
|
||||
_, isl, dsl = re.findall(r"\d+", scenario_name)
|
||||
# cfg_part not used; just for readability.
|
||||
for label, data in configs:
|
||||
s = by_scenario[scenario_name].get(label)
|
||||
if s is None:
|
||||
if scenario_name in skipped and label in skipped[scenario_name]:
|
||||
note = skipped[scenario_name][label]
|
||||
f.write(f"| {scenario_name} | {isl} | {dsl} | {label} | - | - | - | - | - | - | - | - | - | - | {note} |\n")
|
||||
continue
|
||||
cfg = s["config"]
|
||||
m = s["metrics"]
|
||||
status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
|
||||
gpu = m.get("gpu_memory")
|
||||
f.write(
|
||||
f"| {scenario_name} | {isl} | {dsl} | {label} | {cfg['concurrency']} | "
|
||||
f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||
f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||
f"{m['e2e_ms']['p99']:.2f} | {gpu_memory_str(gpu)} | {status} |\n"
|
||||
)
|
||||
|
||||
# Best throughput per ISL/DSL.
|
||||
f.write("\n## Best throughput per (ISL, DSL)\n\n")
|
||||
f.write("| ISL | DSL | Best Config | Concurrency | OutTok/s | TTFT P95(ms) | TPOT Mean(ms) | SLO |\n")
|
||||
f.write("|---:|---:|---|---:|---:|---:|---:|---:|\n")
|
||||
best_by_shape: dict[tuple[int, int], tuple[float, str, dict]] = {}
|
||||
for scenario_name, backends in by_scenario.items():
|
||||
_, isl, dsl = re.findall(r"\d+", scenario_name)
|
||||
isl_i, dsl_i = int(isl), int(dsl)
|
||||
for label, s in backends.items():
|
||||
m = s["metrics"]
|
||||
out_tok = m["output_token_throughput"]
|
||||
if (isl_i, dsl_i) not in best_by_shape or out_tok > best_by_shape[(isl_i, dsl_i)][0]:
|
||||
best_by_shape[(isl_i, dsl_i)] = (out_tok, label, s)
|
||||
for (isl_i, dsl_i), (out_tok, label, s) in sorted(best_by_shape.items()):
|
||||
m = s["metrics"]
|
||||
status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
|
||||
f.write(
|
||||
f"| {isl_i} | {dsl_i} | {label} | {s['config']['concurrency']} | "
|
||||
f"{out_tok:.2f} | {m['ttft_ms']['p95']:.2f} | {m['tpot_ms']['mean']:.2f} | {status} |\n"
|
||||
)
|
||||
|
||||
f.write("\n## Notes\n\n")
|
||||
f.write("- SLO check uses TTFT P95 and TPOT mean.\n")
|
||||
f.write("- A PARTIAL indicates one of the two metrics is out of target; FAIL indicates both are out.\n")
|
||||
f.write("- `Peak GPU mem` shows peak used / total MB and utilization percentage.\n")
|
||||
f.write("- Optional (P) combinations that failed are marked as skipped/OOM and do not break the run.\n")
|
||||
|
||||
print(f"Wrote comparison to {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
76
experiments/dsv4_h200_vllm_tp_dp_matrix/config.env
Normal file
76
experiments/dsv4_h200_vllm_tp_dp_matrix/config.env
Normal file
@ -0,0 +1,76 @@
|
||||
# TP×DP matrix experiment for DeepSeek-V4-Flash on H200 (8 GPUs) using vLLM.
|
||||
# Tests vLLM with three parallel configurations:
|
||||
# TP=2, DP=4 -> 2 GPUs per replica, 4 replicas
|
||||
# TP=4, DP=2 -> 4 GPUs per replica, 2 replicas
|
||||
# TP=8, DP=1 -> 8 GPUs, no data parallelism
|
||||
|
||||
EXPERIMENT="dsv4_h200_vllm_tp_dp_matrix"
|
||||
MODEL_NAME="DeepSeek-V4-Flash"
|
||||
MODEL_PATH="/data3/hf_models/DeepSeek-V4-Flash"
|
||||
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
VLLM_PORT="${VLLM_PORT:-30030}"
|
||||
|
||||
# Python interpreter for orchestration scripts (parse_backend.py, compare.py, etc.)
|
||||
# and the benchmark client. Defaults to the system python3 if the sglang venv
|
||||
# does not exist on the host.
|
||||
VENV_CLIENT="${VENV_CLIENT:-/data/user1/yy/envs/sglang}"
|
||||
|
||||
# Run the benchmark client natively (0) or inside Docker (1).
|
||||
USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-1}"
|
||||
|
||||
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
|
||||
|
||||
# Runtime working directory for logs, pid files, and tmp. Defaults to a local
|
||||
# directory under this experiment so the benchmark is self-contained.
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
|
||||
# Parallel configurations to test. Format: "TP DP"
|
||||
declare -a PARALLEL_CONFIGS=(
|
||||
"2 4"
|
||||
"4 2"
|
||||
"8 1"
|
||||
)
|
||||
|
||||
# vLLM server settings
|
||||
MAX_MODEL_LEN="${MAX_MODEL_LEN:-1048576}"
|
||||
MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}"
|
||||
GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.9}"
|
||||
KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}"
|
||||
BLOCK_SIZE="${BLOCK_SIZE:-256}"
|
||||
|
||||
# Deployment switch. 0 = native vllm venv, 1 = Docker.
|
||||
USE_DOCKER="${USE_DOCKER:-1}"
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
|
||||
|
||||
# Benchmark client Docker image. vLLM's image does not include sglang.bench_serving,
|
||||
# so the client runs inside the SGLang image and targets the vLLM backend via the
|
||||
# OpenAI-compatible API.
|
||||
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
# Dataset used by sglang.bench_serving --dataset-name random.
|
||||
# The random sampler needs a ShareGPT-style JSON file locally; it falls back to
|
||||
# downloading from HuggingFace, which usually fails on offline H200 nodes.
|
||||
DATASET_PATH="${DATASET_PATH:-/data3/yy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json}"
|
||||
|
||||
# Matrix and concurrency rules are defined in matrix.json by default.
|
||||
MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR:-.}/matrix.json}"
|
||||
MATRIX_MODE="${MATRIX_MODE:-Y}"
|
||||
|
||||
# Sampling density for concurrency.
|
||||
# 0 = use the default heuristic in generate_scenarios.py (6-8 points).
|
||||
# 2 = only test the low and high endpoints.
|
||||
export CONCURRENCY_SAMPLES="${CONCURRENCY_SAMPLES:-2}"
|
||||
|
||||
# Per-scenario timeout to avoid hangs (seconds).
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||
|
||||
# GPU memory sampling interval (seconds).
|
||||
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||
|
||||
# Dry-run mode: if 1, only log the server args and scenario plan without starting
|
||||
# any server or sending requests.
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
|
||||
# Per-config scenario limit for quick smoke tests. 0 = run all generated scenarios.
|
||||
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||
98
experiments/dsv4_h200_vllm_tp_dp_matrix/generate_scenarios.py
Executable file
98
experiments/dsv4_h200_vllm_tp_dp_matrix/generate_scenarios.py
Executable file
@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate the scenario list for the TP×DP matrix experiment.
|
||||
|
||||
Reads matrix.json and prints TSV lines:
|
||||
|
||||
mark input_len output_len concurrency num_prompts
|
||||
|
||||
mark is one of Y/P/N. The caller (run_bench.sh) decides how to treat each.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def sample_concurrency(low: int, high: int, target: int) -> list[int]:
|
||||
"""Return only the low and high concurrency values in [low, high].
|
||||
|
||||
For the TP×DP matrix we only need the two endpoints of the concurrency
|
||||
range (e.g. 1 and 128 for ISL=1024). The `target` argument is kept for
|
||||
API compatibility but is ignored.
|
||||
"""
|
||||
assert 1 <= low <= high, f"invalid concurrency range: {low}-{high}"
|
||||
if low == high:
|
||||
return [low]
|
||||
return [low, high]
|
||||
|
||||
|
||||
def generate_scenarios(matrix_path: Path, mode: str, target_samples: int) -> list[dict]:
|
||||
with open(matrix_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
matrix = data["matrix"]
|
||||
concurrency_cfg = data["concurrency"]
|
||||
|
||||
scenarios = []
|
||||
for isl_str in sorted(matrix.keys(), key=int):
|
||||
dsl_map = matrix[isl_str]
|
||||
low = concurrency_cfg[isl_str]["low"]
|
||||
high = concurrency_cfg[isl_str]["high"]
|
||||
concurrencies = sample_concurrency(low, high, target_samples)
|
||||
|
||||
for dsl_str in sorted(dsl_map.keys(), key=int):
|
||||
mark = dsl_map[dsl_str]
|
||||
if mode == "Y" and mark != "Y":
|
||||
continue
|
||||
if mode == "Y+P" and mark not in ("Y", "P"):
|
||||
continue
|
||||
# mode == "all" keeps everything, including N.
|
||||
|
||||
for conc in concurrencies:
|
||||
scenarios.append(
|
||||
{
|
||||
"mark": mark,
|
||||
"input_len": int(isl_str),
|
||||
"output_len": int(dsl_str),
|
||||
"concurrency": conc,
|
||||
"num_prompts": conc * 5,
|
||||
}
|
||||
)
|
||||
|
||||
return scenarios
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--matrix", type=Path, default=Path("matrix.json"))
|
||||
parser.add_argument("--mode", choices=["Y", "Y+P", "all"], default=None,
|
||||
help="Scenario selection mode. Defaults to matrix.mode.")
|
||||
parser.add_argument("--target-samples", type=int, default=0,
|
||||
help="Target number of concurrency samples. 0 = heuristic (6-8).")
|
||||
args = parser.parse_args()
|
||||
|
||||
with open(args.matrix, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
mode = args.mode if args.mode else data.get("mode", "Y+P")
|
||||
|
||||
target_samples = args.target_samples
|
||||
if target_samples <= 0:
|
||||
env_samples = os.getenv("CONCURRENCY_SAMPLES", "0")
|
||||
try:
|
||||
target_samples = int(env_samples)
|
||||
except ValueError:
|
||||
target_samples = 0
|
||||
if target_samples <= 0:
|
||||
target_samples = 7
|
||||
|
||||
scenarios = generate_scenarios(args.matrix, mode, target_samples)
|
||||
|
||||
print("mark\tinput_len\toutput_len\tconcurrency\tnum_prompts")
|
||||
for s in scenarios:
|
||||
print(f"{s['mark']}\t{s['input_len']}\t{s['output_len']}\t{s['concurrency']}\t{s['num_prompts']}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
81
experiments/dsv4_h200_vllm_tp_dp_matrix/matrix.json
Normal file
81
experiments/dsv4_h200_vllm_tp_dp_matrix/matrix.json
Normal file
@ -0,0 +1,81 @@
|
||||
{
|
||||
"comment": "ISL/DSL matrix for dsv4_h200_vllm_tp_dp_matrix. Y=must test, P=optional (record N on failure), N=skip.",
|
||||
"mode": "Y",
|
||||
"comment": "Only mandatory (Y) combinations are tested; 1M ISL is excluded per user request.",
|
||||
"matrix": {
|
||||
"1024": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "Y"
|
||||
},
|
||||
"4096": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "Y"
|
||||
},
|
||||
"16384": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "Y",
|
||||
"4096": "P"
|
||||
},
|
||||
"65536": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "Y",
|
||||
"2048": "P",
|
||||
"4096": "N"
|
||||
},
|
||||
"131072": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "Y",
|
||||
"1024": "P",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
},
|
||||
"262144": {
|
||||
"128": "Y",
|
||||
"256": "Y",
|
||||
"512": "P",
|
||||
"1024": "N",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
},
|
||||
"524288": {
|
||||
"128": "Y",
|
||||
"256": "P",
|
||||
"512": "N",
|
||||
"1024": "N",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
},
|
||||
"1048576": {
|
||||
"128": "Y",
|
||||
"256": "P",
|
||||
"512": "N",
|
||||
"1024": "N",
|
||||
"2048": "N",
|
||||
"4096": "N"
|
||||
}
|
||||
},
|
||||
"concurrency": {
|
||||
"1024": { "low": 1, "high": 128 },
|
||||
"4096": { "low": 1, "high": 64 },
|
||||
"16384": { "low": 1, "high": 32 },
|
||||
"65536": { "low": 1, "high": 8 },
|
||||
"131072": { "low": 1, "high": 4 },
|
||||
"262144": { "low": 1, "high": 2 },
|
||||
"524288": { "low": 1, "high": 2 },
|
||||
"1048576": { "low": 1, "high": 2 }
|
||||
}
|
||||
}
|
||||
@ -0,0 +1,230 @@
|
||||
# vLLM TP×DP matrix comparison (8x NVIDIA H200 143GB)
|
||||
|
||||
## Summary
|
||||
|
||||
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||
- Hardware: 8x NVIDIA H200 143GB
|
||||
- Backend: vLLM
|
||||
- Benchmark client: `sglang.bench_serving`
|
||||
- SLO reference: TTFT P95 < 3000.0ms, TPOT mean < 50.0ms
|
||||
|
||||
### Configurations
|
||||
|
||||
| Config | TP | DP | GPUs/replica | Notes |
|
||||
|---|---:|---:|---:|---|
|
||||
| tp2_dp4 | 2 | 4 | 2 | server args recorded per ISL in results.json |
|
||||
| tp4_dp2 | 4 | 2 | 4 | server args recorded per ISL in results.json |
|
||||
| tp8_dp1 | 8 | 1 | 8 | server args recorded per ISL in results.json |
|
||||
|
||||
## Side-by-side results
|
||||
|
||||
| Scenario | ISL | DSL | Config | Conc | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | Peak GPU mem | SLO |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| c1_i1024_o128 | 1024 | 128 | tp2_dp4 | 1 | 1.05 | 79.65 | 319.08 | 355.57 | 10.56 | 10.99 | 11.00 | 1406.84 | 138264/143771 (96.2%) | ✅ |
|
||||
| c1_i1024_o128 | 1024 | 128 | tp4_dp2 | 1 | 1.34 | 101.85 | 302.85 | 345.83 | 8.17 | 8.32 | 8.35 | 1055.00 | 138396/143771 (96.3%) | ✅ |
|
||||
| c1_i1024_o128 | 1024 | 128 | tp8_dp1 | 1 | 1.61 | 121.80 | 274.43 | 314.97 | 6.74 | 6.78 | 6.79 | 871.45 | 131900/143771 (91.7%) | ✅ |
|
||||
| c1_i1024_o256 | 1024 | 256 | tp2_dp4 | 1 | 0.84 | 84.85 | 107.93 | 108.38 | 11.01 | 11.57 | 11.61 | 2082.56 | 139322/143771 (96.9%) | ✅ |
|
||||
| c1_i1024_o256 | 1024 | 256 | tp4_dp2 | 1 | 1.12 | 113.80 | 88.52 | 90.59 | 8.12 | 8.21 | 8.23 | 1572.36 | 139454/143771 (97.0%) | ✅ |
|
||||
| c1_i1024_o256 | 1024 | 256 | tp8_dp1 | 1 | 1.40 | 141.77 | 44.63 | 45.58 | 6.71 | 6.72 | 6.72 | 1281.54 | 132930/143771 (92.5%) | ✅ |
|
||||
| c1_i1024_o512 | 1024 | 512 | tp2_dp4 | 1 | 0.79 | 79.92 | 113.71 | 115.02 | 11.50 | 11.85 | 11.86 | 2303.60 | 139322/143771 (96.9%) | ✅ |
|
||||
| c1_i1024_o512 | 1024 | 512 | tp4_dp2 | 1 | 1.09 | 110.67 | 187.00 | 208.64 | 8.12 | 8.17 | 8.18 | 1572.17 | 139454/143771 (97.0%) | ✅ |
|
||||
| c1_i1024_o512 | 1024 | 512 | tp8_dp1 | 1 | 1.36 | 137.55 | 65.04 | 65.69 | 6.74 | 6.78 | 6.79 | 1305.74 | 132930/143771 (92.5%) | ✅ |
|
||||
| c1_i1024_o1024 | 1024 | 1024 | tp2_dp4 | 1 | 0.27 | 83.58 | 100.14 | 100.52 | 11.71 | 11.90 | 11.93 | 8200.76 | 139322/143771 (96.9%) | ✅ |
|
||||
| c1_i1024_o1024 | 1024 | 1024 | tp4_dp2 | 1 | 0.39 | 120.49 | 90.47 | 90.97 | 8.11 | 8.19 | 8.21 | 5705.34 | 139456/143771 (97.0%) | ✅ |
|
||||
| c1_i1024_o1024 | 1024 | 1024 | tp8_dp1 | 1 | 0.48 | 146.08 | 60.67 | 60.68 | 6.71 | 6.73 | 6.73 | 4727.62 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i1024_o2048 | 1024 | 2048 | tp2_dp4 | 1 | 0.08 | 84.74 | 105.11 | 105.45 | 11.70 | 11.81 | 11.81 | 20192.89 | 139322/143771 (96.9%) | ✅ |
|
||||
| c1_i1024_o2048 | 1024 | 2048 | tp4_dp2 | 1 | 0.11 | 122.91 | 84.75 | 84.77 | 8.08 | 8.09 | 8.09 | 13978.61 | 139456/143771 (97.0%) | ✅ |
|
||||
| c1_i1024_o2048 | 1024 | 2048 | tp8_dp1 | 1 | 0.13 | 148.32 | 60.19 | 62.81 | 6.71 | 6.72 | 6.72 | 11586.89 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i1024_o4096 | 1024 | 4096 | tp2_dp4 | 1 | 0.04 | 86.04 | 114.24 | 116.37 | 11.59 | 11.84 | 11.86 | 43455.08 | 139322/143771 (96.9%) | ✅ |
|
||||
| c1_i1024_o4096 | 1024 | 4096 | tp4_dp2 | 1 | 0.05 | 122.97 | 82.10 | 82.14 | 8.09 | 8.12 | 8.13 | 30482.21 | 139456/143771 (97.0%) | ✅ |
|
||||
| c1_i1024_o4096 | 1024 | 4096 | tp8_dp1 | 1 | 0.06 | 148.14 | 46.60 | 47.48 | 6.73 | 6.76 | 6.76 | 25332.98 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i4096_o128 | 4096 | 128 | tp2_dp4 | 1 | 0.92 | 69.56 | 353.45 | 371.20 | 11.63 | 11.95 | 12.00 | 1608.68 | 139322/143771 (96.9%) | ✅ |
|
||||
| c1_i4096_o128 | 4096 | 128 | tp4_dp2 | 1 | 1.44 | 109.20 | 96.00 | 96.62 | 8.16 | 8.23 | 8.24 | 1054.23 | 139456/143771 (97.0%) | ✅ |
|
||||
| c1_i4096_o128 | 4096 | 128 | tp8_dp1 | 1 | 1.62 | 122.84 | 148.51 | 155.20 | 6.72 | 6.74 | 6.74 | 918.13 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i4096_o256 | 4096 | 256 | tp2_dp4 | 1 | 0.77 | 78.01 | 208.90 | 226.63 | 11.61 | 11.87 | 11.89 | 2296.91 | 139324/143771 (96.9%) | ✅ |
|
||||
| c1_i4096_o256 | 4096 | 256 | tp4_dp2 | 1 | 1.11 | 112.37 | 91.50 | 91.90 | 8.16 | 8.23 | 8.24 | 1604.71 | 139456/143771 (97.0%) | ✅ |
|
||||
| c1_i4096_o256 | 4096 | 256 | tp8_dp1 | 1 | 1.37 | 138.57 | 67.18 | 67.54 | 6.73 | 6.77 | 6.78 | 1320.30 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i4096_o512 | 4096 | 512 | tp2_dp4 | 1 | 0.77 | 78.30 | 210.04 | 232.13 | 11.63 | 11.95 | 11.98 | 2378.47 | 139324/143771 (96.9%) | ✅ |
|
||||
| c1_i4096_o512 | 4096 | 512 | tp4_dp2 | 1 | 1.10 | 111.43 | 104.51 | 107.11 | 8.16 | 8.24 | 8.25 | 1621.06 | 139456/143771 (97.0%) | ✅ |
|
||||
| c1_i4096_o512 | 4096 | 512 | tp8_dp1 | 1 | 1.37 | 138.55 | 64.63 | 65.10 | 6.73 | 6.78 | 6.78 | 1320.67 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i4096_o1024 | 4096 | 1024 | tp2_dp4 | 1 | 0.27 | 83.60 | 113.91 | 115.64 | 11.55 | 11.86 | 11.87 | 8360.74 | 139324/143771 (96.9%) | ✅ |
|
||||
| c1_i4096_o1024 | 4096 | 1024 | tp4_dp2 | 1 | 0.39 | 119.41 | 92.40 | 93.46 | 8.15 | 8.23 | 8.24 | 5771.30 | 139456/143771 (97.0%) | ✅ |
|
||||
| c1_i4096_o1024 | 4096 | 1024 | tp8_dp1 | 1 | 0.47 | 144.76 | 66.58 | 66.81 | 6.73 | 6.78 | 6.79 | 4791.74 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i4096_o2048 | 4096 | 2048 | tp2_dp4 | 1 | 0.08 | 86.30 | 122.31 | 124.03 | 11.54 | 11.75 | 11.78 | 19528.19 | 139324/143771 (96.9%) | ✅ |
|
||||
| c1_i4096_o2048 | 4096 | 2048 | tp4_dp2 | 1 | 0.11 | 121.84 | 105.61 | 108.34 | 8.13 | 8.16 | 8.16 | 14152.06 | 139824/143771 (97.3%) | ✅ |
|
||||
| c1_i4096_o2048 | 4096 | 2048 | tp8_dp1 | 1 | 0.13 | 146.91 | 67.22 | 67.42 | 6.75 | 6.79 | 6.80 | 11755.04 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i4096_o4096 | 4096 | 4096 | tp2_dp4 | 1 | 0.04 | 85.60 | 130.43 | 132.68 | 11.66 | 11.87 | 11.89 | 43588.26 | 139324/143771 (96.9%) | ✅ |
|
||||
| c1_i4096_o4096 | 4096 | 4096 | tp4_dp2 | 1 | 0.05 | 122.26 | 110.67 | 114.17 | 8.14 | 8.17 | 8.17 | 30710.72 | 139824/143771 (97.3%) | ✅ |
|
||||
| c1_i4096_o4096 | 4096 | 4096 | tp8_dp1 | 1 | 0.06 | 147.18 | 66.70 | 66.85 | 6.77 | 6.80 | 6.80 | 25525.53 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i16384_o128 | 16384 | 128 | tp2_dp4 | 1 | 0.88 | 66.35 | 599.10 | 618.22 | 11.00 | 11.48 | 11.54 | 1948.72 | 139324/143771 (96.9%) | ✅ |
|
||||
| c1_i16384_o128 | 16384 | 128 | tp4_dp2 | 1 | 1.14 | 86.24 | 512.76 | 518.98 | 8.11 | 8.23 | 8.25 | 1441.62 | 139824/143771 (97.3%) | ✅ |
|
||||
| c1_i16384_o128 | 16384 | 128 | tp8_dp1 | 1 | 1.34 | 101.29 | 433.34 | 440.01 | 6.72 | 6.80 | 6.80 | 1242.36 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i16384_o256 | 16384 | 256 | tp2_dp4 | 1 | 0.71 | 71.69 | 448.96 | 486.54 | 11.52 | 11.66 | 11.67 | 2621.34 | 139324/143771 (96.9%) | ✅ |
|
||||
| c1_i16384_o256 | 16384 | 256 | tp4_dp2 | 1 | 0.98 | 99.42 | 407.06 | 440.10 | 8.14 | 8.23 | 8.24 | 1937.46 | 139824/143771 (97.3%) | ✅ |
|
||||
| c1_i16384_o256 | 16384 | 256 | tp8_dp1 | 1 | 1.31 | 132.55 | 122.35 | 122.83 | 6.73 | 6.76 | 6.76 | 1372.37 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i16384_o512 | 16384 | 512 | tp2_dp4 | 1 | 0.77 | 77.70 | 188.54 | 190.96 | 11.55 | 11.73 | 11.74 | 2356.50 | 139684/143771 (97.2%) | ✅ |
|
||||
| c1_i16384_o512 | 16384 | 512 | tp4_dp2 | 1 | 1.07 | 108.69 | 155.92 | 157.11 | 8.16 | 8.23 | 8.25 | 1661.52 | 140338/143771 (97.6%) | ✅ |
|
||||
| c1_i16384_o512 | 16384 | 512 | tp8_dp1 | 1 | 1.31 | 132.98 | 118.65 | 119.23 | 6.73 | 6.76 | 6.76 | 1365.21 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i16384_o1024 | 16384 | 1024 | tp2_dp4 | 1 | 0.27 | 83.88 | 322.92 | 349.35 | 11.31 | 11.64 | 11.64 | 7915.88 | 139684/143771 (97.2%) | ✅ |
|
||||
| c1_i16384_o1024 | 16384 | 1024 | tp4_dp2 | 1 | 0.38 | 117.41 | 158.31 | 161.09 | 8.16 | 8.23 | 8.24 | 5837.98 | 140854/143771 (98.0%) | ✅ |
|
||||
| c1_i16384_o1024 | 16384 | 1024 | tp8_dp1 | 1 | 0.46 | 141.13 | 132.08 | 134.06 | 6.75 | 6.80 | 6.81 | 4873.19 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i16384_o2048 | 16384 | 2048 | tp2_dp4 | 1 | 0.08 | 85.62 | 249.46 | 261.08 | 11.54 | 11.64 | 11.65 | 20210.50 | 139684/143771 (97.2%) | ✅ |
|
||||
| c1_i16384_o2048 | 16384 | 2048 | tp4_dp2 | 1 | 0.11 | 121.16 | 155.57 | 156.44 | 8.14 | 8.18 | 8.19 | 14233.84 | 140854/143771 (98.0%) | ✅ |
|
||||
| c1_i16384_o2048 | 16384 | 2048 | tp8_dp1 | 1 | 0.13 | 145.70 | 132.65 | 135.00 | 6.77 | 6.82 | 6.82 | 11867.20 | 132932/143771 (92.5%) | ✅ |
|
||||
| c1_i65536_o128 | 65536 | 128 | tp2_dp4 | 1 | 0.44 | 33.25 | 2811.65 | 2892.88 | 11.47 | 12.08 | 12.21 | 4233.91 | 139706/143771 (97.2%) | ✅ |
|
||||
| c1_i65536_o128 | 65536 | 128 | tp4_dp2 | 1 | 0.56 | 42.54 | 2211.56 | 2223.60 | 7.89 | 8.16 | 8.19 | 3092.42 | 140854/143771 (98.0%) | ✅ |
|
||||
| c1_i65536_o128 | 65536 | 128 | tp8_dp1 | 1 | 0.84 | 63.38 | 1783.81 | 1903.90 | 6.67 | 6.75 | 6.75 | 2686.50 | 133260/143771 (92.7%) | ✅ |
|
||||
| c1_i65536_o256 | 65536 | 256 | tp2_dp4 | 1 | 0.48 | 48.96 | 2181.49 | 2314.91 | 11.55 | 11.95 | 11.96 | 3709.96 | 139708/143771 (97.2%) | ✅ |
|
||||
| c1_i65536_o256 | 65536 | 256 | tp4_dp2 | 1 | 1.01 | 102.18 | 274.65 | 283.52 | 7.97 | 8.23 | 8.25 | 1647.86 | 140854/143771 (98.0%) | ✅ |
|
||||
| c1_i65536_o256 | 65536 | 256 | tp8_dp1 | 1 | 1.19 | 120.56 | 254.47 | 256.20 | 6.74 | 6.79 | 6.80 | 1352.80 | 133780/143771 (93.1%) | ✅ |
|
||||
| c1_i65536_o512 | 65536 | 512 | tp2_dp4 | 1 | 0.62 | 63.19 | 1276.37 | 1453.96 | 11.06 | 11.68 | 11.78 | 2624.42 | 139708/143771 (97.2%) | ✅ |
|
||||
| c1_i65536_o512 | 65536 | 512 | tp4_dp2 | 1 | 0.98 | 99.45 | 326.62 | 330.25 | 7.91 | 8.17 | 8.18 | 1655.09 | 140854/143771 (98.0%) | ✅ |
|
||||
| c1_i65536_o512 | 65536 | 512 | tp8_dp1 | 1 | 1.18 | 119.95 | 288.97 | 290.20 | 6.54 | 6.74 | 6.75 | 1356.99 | 133780/143771 (93.1%) | ✅ |
|
||||
| c1_i65536_o1024 | 65536 | 1024 | tp2_dp4 | 1 | 0.27 | 83.13 | 375.58 | 383.90 | 11.12 | 11.57 | 11.58 | 8213.89 | 139708/143771 (97.2%) | ✅ |
|
||||
| c1_i65536_o1024 | 65536 | 1024 | tp4_dp2 | 1 | 0.37 | 112.31 | 358.50 | 364.11 | 8.07 | 8.26 | 8.26 | 5842.34 | 140854/143771 (98.0%) | ✅ |
|
||||
| c1_i65536_o1024 | 65536 | 1024 | tp8_dp1 | 1 | 0.45 | 136.44 | 254.42 | 254.84 | 6.76 | 6.85 | 6.86 | 4841.93 | 133780/143771 (93.1%) | ✅ |
|
||||
| c1_i131072_o128 | 131072 | 128 | tp2_dp4 | 1 | 0.32 | 24.40 | 3831.71 | 3847.18 | 11.06 | 11.65 | 11.68 | 5071.55 | 140228/143771 (97.5%) | ⚠️ |
|
||||
| c1_i131072_o128 | 131072 | 128 | tp4_dp2 | 1 | 0.38 | 28.62 | 3498.43 | 3510.98 | 7.74 | 8.18 | 8.22 | 4353.06 | 141374/143771 (98.3%) | ⚠️ |
|
||||
| c1_i131072_o128 | 131072 | 128 | tp8_dp1 | 1 | 0.55 | 42.03 | 3042.40 | 3072.01 | 6.34 | 6.70 | 6.72 | 3832.09 | 133780/143771 (93.1%) | ⚠️ |
|
||||
| c1_i131072_o256 | 131072 | 256 | tp2_dp4 | 1 | 0.46 | 46.84 | 3143.30 | 3630.53 | 11.11 | 11.90 | 11.97 | 4950.01 | 140228/143771 (97.5%) | ⚠️ |
|
||||
| c1_i131072_o256 | 131072 | 256 | tp4_dp2 | 1 | 0.61 | 61.94 | 2612.01 | 3033.01 | 7.81 | 8.23 | 8.24 | 3829.62 | 141374/143771 (98.3%) | ✅ |
|
||||
| c1_i131072_o256 | 131072 | 256 | tp8_dp1 | 1 | 1.03 | 104.48 | 526.13 | 531.79 | 6.38 | 6.74 | 6.75 | 1365.14 | 134302/143771 (93.4%) | ✅ |
|
||||
| c1_i131072_o512 | 131072 | 512 | tp2_dp4 | 1 | 0.67 | 68.11 | 574.50 | 577.45 | 10.89 | 11.54 | 11.54 | 2318.95 | 140228/143771 (97.5%) | ✅ |
|
||||
| c1_i131072_o512 | 131072 | 512 | tp4_dp2 | 1 | 0.89 | 90.55 | 555.04 | 573.87 | 7.81 | 8.24 | 8.27 | 1649.69 | 141374/143771 (98.3%) | ✅ |
|
||||
| c1_i131072_o512 | 131072 | 512 | tp8_dp1 | 1 | 1.03 | 104.09 | 528.58 | 529.52 | 6.38 | 6.74 | 6.75 | 1365.66 | 134302/143771 (93.4%) | ✅ |
|
||||
| c1_i262144_o128 | 262144 | 128 | tp2_dp4 | 1 | 0.15 | 11.35 | 13444.27 | 14788.12 | 9.88 | 11.09 | 11.15 | 16032.30 | 140744/143771 (97.9%) | ⚠️ |
|
||||
| c1_i262144_o128 | 262144 | 128 | tp4_dp2 | 1 | 0.20 | 15.35 | 8221.07 | 8613.48 | 6.70 | 7.64 | 7.66 | 9509.11 | 141374/143771 (98.3%) | ⚠️ |
|
||||
| c1_i262144_o128 | 262144 | 128 | tp8_dp1 | 1 | 0.23 | 17.26 | 7547.16 | 7933.13 | 5.31 | 6.25 | 6.26 | 8667.71 | 134304/143771 (93.4%) | ⚠️ |
|
||||
| c1_i262144_o256 | 262144 | 256 | tp2_dp4 | 1 | 0.18 | 18.19 | 8943.43 | 9385.72 | 10.07 | 11.43 | 11.50 | 10671.30 | 141258/143771 (98.3%) | ⚠️ |
|
||||
| c1_i262144_o256 | 262144 | 256 | tp4_dp2 | 1 | 0.71 | 72.25 | 962.12 | 1036.06 | 6.92 | 7.88 | 7.93 | 2080.08 | 141374/143771 (98.3%) | ✅ |
|
||||
| c1_i262144_o256 | 262144 | 256 | tp8_dp1 | 1 | 0.91 | 92.31 | 741.06 | 802.19 | 5.45 | 6.50 | 6.55 | 1664.47 | 134304/143771 (93.4%) | ✅ |
|
||||
| c1_i524288_o128 | 524288 | 128 | tp2_dp4 | 1 | 0.13 | 9.76 | 17139.22 | 19091.80 | 9.70 | 10.39 | 10.40 | 20099.12 | 141258/143771 (98.3%) | ⚠️ |
|
||||
| c1_i524288_o128 | 524288 | 128 | tp4_dp2 | 1 | 0.21 | 16.00 | 14806.51 | 17551.38 | 6.63 | 7.42 | 7.46 | 18281.61 | 141374/143771 (98.3%) | ⚠️ |
|
||||
| c1_i524288_o128 | 524288 | 128 | tp8_dp1 | 1 | 0.23 | 17.65 | 13982.40 | 16622.77 | 5.18 | 6.10 | 6.17 | 17210.26 | 134304/143771 (93.4%) | ⚠️ |
|
||||
| c1_i1048576_o128 | 1048576 | 128 | tp2_dp4 | 1 | 0.06 | 4.85 | 46081.66 | 51394.85 | 8.95 | 10.50 | 10.51 | 51849.45 | 141258/143771 (98.3%) | ⚠️ |
|
||||
| c1_i1048576_o128 | 1048576 | 128 | tp4_dp2 | 1 | 0.06 | 4.88 | 43228.08 | 48170.56 | 6.14 | 7.51 | 7.56 | 48471.96 | 141374/143771 (98.3%) | ⚠️ |
|
||||
| c1_i1048576_o128 | 1048576 | 128 | tp8_dp1 | 1 | 0.10 | 7.26 | 38265.96 | 45702.81 | 4.67 | 6.04 | 6.08 | 45912.47 | 134310/143771 (93.4%) | ⚠️ |
|
||||
| c2_i262144_o128 | 262144 | 128 | tp2_dp4 | 2 | 0.27 | 20.65 | 9292.65 | 9433.23 | 24.30 | 74.40 | 75.57 | 10323.72 | 140744/143771 (97.9%) | ⚠️ |
|
||||
| c2_i262144_o128 | 262144 | 128 | tp4_dp2 | 2 | 0.33 | 24.58 | 7994.94 | 8377.33 | 25.23 | 70.28 | 72.19 | 9106.67 | 141374/143771 (98.3%) | ⚠️ |
|
||||
| c2_i262144_o128 | 262144 | 128 | tp8_dp1 | 2 | 0.58 | 43.70 | 6405.29 | 6864.92 | 12.04 | 40.84 | 63.02 | 7624.83 | 134304/143771 (93.4%) | ⚠️ |
|
||||
| c2_i262144_o256 | 262144 | 256 | tp2_dp4 | 2 | 0.30 | 42.07 | 9153.02 | 10642.68 | 24.92 | 66.10 | 74.81 | 12104.02 | 141258/143771 (98.3%) | ⚠️ |
|
||||
| c2_i262144_o256 | 262144 | 256 | tp4_dp2 | 2 | 0.48 | 66.52 | 6399.54 | 6424.61 | 10.75 | 22.96 | 32.38 | 8063.51 | 141374/143771 (98.3%) | ⚠️ |
|
||||
| c2_i262144_o256 | 262144 | 256 | tp8_dp1 | 2 | 1.23 | 171.18 | 872.87 | 960.16 | 6.69 | 7.16 | 7.25 | 2191.39 | 134304/143771 (93.4%) | ✅ |
|
||||
| c2_i524288_o128 | 524288 | 128 | tp2_dp4 | 2 | 0.69 | 52.24 | 4695.87 | 6463.46 | 15.77 | 44.14 | 65.40 | 8060.58 | 141258/143771 (98.3%) | ⚠️ |
|
||||
| c2_i524288_o128 | 524288 | 128 | tp4_dp2 | 2 | 0.64 | 48.43 | 6241.67 | 9313.98 | 44.54 | 213.86 | 348.24 | 10806.98 | 141374/143771 (98.3%) | ⚠️ |
|
||||
| c2_i524288_o128 | 524288 | 128 | tp8_dp1 | 2 | 1.76 | 133.35 | 1167.12 | 1232.75 | 5.64 | 6.90 | 6.94 | 1708.86 | 134306/143771 (93.4%) | ✅ |
|
||||
| c2_i1048576_o128 | 1048576 | 128 | tp2_dp4 | 2 | 0.06 | 4.16 | 61414.70 | 65340.99 | 117.07 | 500.43 | 531.18 | 66294.91 | 141780/143771 (98.6%) | ❌ |
|
||||
| c2_i1048576_o128 | 1048576 | 128 | tp4_dp2 | 2 | 0.06 | 4.56 | 51577.89 | 52987.83 | 277.94 | 766.41 | 858.05 | 56546.05 | 141374/143771 (98.3%) | ❌ |
|
||||
| c2_i1048576_o128 | 1048576 | 128 | tp8_dp1 | 2 | 0.08 | 6.16 | 59936.57 | 60383.98 | 15.73 | 56.69 | 68.50 | 66073.78 | 134312/143771 (93.4%) | ⚠️ |
|
||||
| c4_i131072_o128 | 131072 | 128 | tp2_dp4 | 4 | 0.74 | 41.97 | 5381.79 | 6485.26 | 63.46 | 234.23 | 254.36 | 9800.84 | 140228/143771 (97.5%) | ❌ |
|
||||
| c4_i131072_o128 | 131072 | 128 | tp4_dp2 | 4 | 0.61 | 34.68 | 7688.92 | 7746.95 | 55.15 | 151.40 | 157.05 | 13546.32 | 141374/143771 (98.3%) | ❌ |
|
||||
| c4_i131072_o128 | 131072 | 128 | tp8_dp1 | 4 | 0.82 | 46.76 | 5665.45 | 7563.45 | 46.92 | 130.55 | 135.39 | 9263.16 | 134302/143771 (93.4%) | ⚠️ |
|
||||
| c4_i131072_o256 | 131072 | 256 | tp2_dp4 | 4 | 0.51 | 64.78 | 4431.12 | 6485.73 | 38.02 | 82.59 | 93.31 | 21326.10 | 140228/143771 (97.5%) | ⚠️ |
|
||||
| c4_i131072_o256 | 131072 | 256 | tp4_dp2 | 4 | 0.79 | 101.20 | 6019.33 | 6092.09 | 20.62 | 43.90 | 46.90 | 10875.06 | 141374/143771 (98.3%) | ⚠️ |
|
||||
| c4_i131072_o256 | 131072 | 256 | tp8_dp1 | 4 | 2.40 | 306.01 | 826.11 | 873.84 | 7.11 | 8.42 | 8.69 | 2602.99 | 134302/143771 (93.4%) | ✅ |
|
||||
| c4_i131072_o512 | 131072 | 512 | tp2_dp4 | 4 | 0.46 | 136.39 | 5162.58 | 5751.13 | 24.26 | 43.94 | 51.73 | 14391.10 | 140228/143771 (97.5%) | ⚠️ |
|
||||
| c4_i131072_o512 | 131072 | 512 | tp4_dp2 | 4 | 1.03 | 302.02 | 724.28 | 1360.67 | 9.94 | 13.97 | 16.16 | 5861.60 | 141374/143771 (98.3%) | ✅ |
|
||||
| c4_i131072_o512 | 131072 | 512 | tp8_dp1 | 4 | 1.30 | 382.40 | 581.21 | 686.73 | 7.67 | 8.27 | 8.30 | 4365.16 | 134302/143771 (93.4%) | ✅ |
|
||||
| c8_i65536_o128 | 65536 | 128 | tp2_dp4 | 8 | 0.86 | 55.23 | 5855.87 | 7061.54 | 115.39 | 327.74 | 399.92 | 18685.86 | 139708/143771 (97.2%) | ❌ |
|
||||
| c8_i65536_o128 | 65536 | 128 | tp4_dp2 | 8 | 0.94 | 60.49 | 5620.08 | 5766.46 | 118.91 | 301.63 | 309.58 | 16549.56 | 140854/143771 (98.0%) | ❌ |
|
||||
| c8_i65536_o128 | 65536 | 128 | tp8_dp1 | 8 | 1.01 | 64.77 | 7445.34 | 8645.53 | 88.42 | 248.75 | 263.80 | 16484.17 | 133780/143771 (93.1%) | ❌ |
|
||||
| c8_i65536_o256 | 65536 | 256 | tp2_dp4 | 8 | 0.90 | 117.73 | 4029.43 | 4806.74 | 54.45 | 101.30 | 122.71 | 18052.15 | 139708/143771 (97.2%) | ❌ |
|
||||
| c8_i65536_o256 | 65536 | 256 | tp4_dp2 | 8 | 1.31 | 171.79 | 3353.05 | 4860.71 | 44.63 | 113.59 | 141.58 | 15303.64 | 140854/143771 (98.0%) | ⚠️ |
|
||||
| c8_i65536_o256 | 65536 | 256 | tp8_dp1 | 8 | 4.63 | 608.09 | 749.11 | 864.77 | 9.14 | 10.21 | 10.30 | 2998.37 | 133780/143771 (93.1%) | ✅ |
|
||||
| c8_i65536_o512 | 65536 | 512 | tp2_dp4 | 8 | 0.98 | 272.84 | 3168.19 | 5671.91 | 24.56 | 51.52 | 85.56 | 18250.24 | 139708/143771 (97.2%) | ⚠️ |
|
||||
| c8_i65536_o512 | 65536 | 512 | tp4_dp2 | 8 | 1.47 | 410.24 | 1809.22 | 2638.39 | 17.90 | 23.23 | 79.99 | 9676.38 | 140854/143771 (98.0%) | ✅ |
|
||||
| c8_i65536_o512 | 65536 | 512 | tp8_dp1 | 8 | 2.53 | 705.69 | 661.39 | 679.12 | 9.10 | 9.63 | 10.39 | 5200.71 | 133780/143771 (93.1%) | ✅ |
|
||||
| c8_i65536_o1024 | 65536 | 1024 | tp2_dp4 | 8 | 0.64 | 351.24 | 2850.09 | 2967.29 | 19.43 | 29.44 | 36.68 | 21507.37 | 139708/143771 (97.2%) | ✅ |
|
||||
| c8_i65536_o1024 | 65536 | 1024 | tp4_dp2 | 8 | 0.97 | 528.38 | 2177.55 | 2783.15 | 13.13 | 20.42 | 32.65 | 13878.42 | 140854/143771 (98.0%) | ✅ |
|
||||
| c8_i65536_o1024 | 65536 | 1024 | tp8_dp1 | 8 | 1.34 | 734.78 | 671.08 | 707.77 | 8.91 | 9.20 | 10.49 | 9476.99 | 133780/143771 (93.1%) | ✅ |
|
||||
| c32_i16384_o128 | 16384 | 128 | tp2_dp4 | 32 | 4.31 | 284.72 | 3298.27 | 4051.80 | 107.21 | 186.59 | 376.91 | 15245.93 | 139324/143771 (96.9%) | ❌ |
|
||||
| c32_i16384_o128 | 16384 | 128 | tp4_dp2 | 32 | 5.16 | 340.76 | 3479.51 | 3760.82 | 85.23 | 160.17 | 235.33 | 12999.09 | 139824/143771 (97.3%) | ❌ |
|
||||
| c32_i16384_o128 | 16384 | 128 | tp8_dp1 | 32 | 5.39 | 356.17 | 3789.36 | 4532.55 | 78.86 | 150.73 | 226.61 | 11830.35 | 132932/143771 (92.5%) | ❌ |
|
||||
| c32_i16384_o256 | 16384 | 256 | tp2_dp4 | 32 | 4.49 | 623.03 | 1913.91 | 2384.97 | 47.34 | 71.67 | 147.97 | 13936.33 | 139684/143771 (97.2%) | ✅ |
|
||||
| c32_i16384_o256 | 16384 | 256 | tp4_dp2 | 32 | 5.41 | 751.48 | 1701.48 | 2572.97 | 38.88 | 55.35 | 126.86 | 11645.26 | 140338/143771 (97.6%) | ✅ |
|
||||
| c32_i16384_o256 | 16384 | 256 | tp8_dp1 | 32 | 12.84 | 1782.54 | 636.48 | 644.96 | 14.84 | 16.94 | 20.78 | 4262.99 | 132932/143771 (92.5%) | ✅ |
|
||||
| c32_i16384_o512 | 16384 | 512 | tp2_dp4 | 32 | 3.77 | 952.48 | 1520.07 | 2259.12 | 30.21 | 45.07 | 57.04 | 18769.12 | 139684/143771 (97.2%) | ✅ |
|
||||
| c32_i16384_o512 | 16384 | 512 | tp4_dp2 | 32 | 5.44 | 1373.00 | 965.45 | 1010.01 | 20.31 | 25.61 | 28.12 | 12002.66 | 140854/143771 (98.0%) | ✅ |
|
||||
| c32_i16384_o512 | 16384 | 512 | tp8_dp1 | 32 | 7.83 | 1977.19 | 640.86 | 739.06 | 14.04 | 15.70 | 18.53 | 7504.82 | 132932/143771 (92.5%) | ✅ |
|
||||
| c32_i16384_o1024 | 16384 | 1024 | tp2_dp4 | 32 | 2.52 | 1337.54 | 1871.97 | 2404.37 | 22.95 | 27.86 | 72.28 | 22977.81 | 139684/143771 (97.2%) | ✅ |
|
||||
| c32_i16384_o1024 | 16384 | 1024 | tp4_dp2 | 32 | 3.21 | 1704.78 | 1131.20 | 1660.57 | 17.14 | 21.79 | 37.19 | 18824.49 | 140854/143771 (98.0%) | ✅ |
|
||||
| c32_i16384_o1024 | 16384 | 1024 | tp8_dp1 | 32 | 4.00 | 2122.35 | 789.94 | 950.40 | 13.38 | 14.23 | 22.02 | 13797.93 | 132932/143771 (92.5%) | ✅ |
|
||||
| c32_i16384_o2048 | 16384 | 2048 | tp2_dp4 | 32 | 1.56 | 1785.47 | 1089.13 | 1111.26 | 15.83 | 17.47 | 19.70 | 33430.94 | 139684/143771 (97.2%) | ✅ |
|
||||
| c32_i16384_o2048 | 16384 | 2048 | tp4_dp2 | 32 | 1.73 | 1982.72 | 1143.92 | 2172.49 | 14.60 | 16.68 | 25.49 | 30322.94 | 140854/143771 (98.0%) | ✅ |
|
||||
| c32_i16384_o2048 | 16384 | 2048 | tp8_dp1 | 32 | 1.94 | 2226.25 | 628.02 | 806.24 | 12.88 | 13.39 | 14.85 | 26578.49 | 132932/143771 (92.5%) | ✅ |
|
||||
| c64_i4096_o128 | 4096 | 128 | tp2_dp4 | 64 | 11.02 | 721.10 | 1790.01 | 2034.19 | 83.97 | 111.62 | 188.61 | 12237.14 | 139324/143771 (96.9%) | ⚠️ |
|
||||
| c64_i4096_o128 | 4096 | 128 | tp4_dp2 | 64 | 10.43 | 682.69 | 1904.20 | 2262.19 | 88.50 | 123.52 | 173.68 | 12463.74 | 139456/143771 (97.0%) | ⚠️ |
|
||||
| c64_i4096_o128 | 4096 | 128 | tp8_dp1 | 64 | 11.81 | 772.92 | 2978.27 | 3725.28 | 76.32 | 108.07 | 233.29 | 11989.51 | 132932/143771 (92.5%) | ⚠️ |
|
||||
| c64_i4096_o256 | 4096 | 256 | tp2_dp4 | 64 | 8.50 | 1101.01 | 1290.52 | 1667.29 | 55.00 | 72.09 | 89.93 | 15631.13 | 139324/143771 (96.9%) | ⚠️ |
|
||||
| c64_i4096_o256 | 4096 | 256 | tp4_dp2 | 64 | 10.23 | 1323.77 | 1198.05 | 1234.72 | 45.94 | 59.77 | 82.58 | 13085.09 | 139456/143771 (97.0%) | ✅ |
|
||||
| c64_i4096_o256 | 4096 | 256 | tp8_dp1 | 64 | 19.45 | 2518.43 | 804.28 | 815.33 | 22.25 | 25.88 | 28.53 | 6644.63 | 132932/143771 (92.5%) | ✅ |
|
||||
| c64_i4096_o512 | 4096 | 512 | tp2_dp4 | 64 | 6.47 | 1712.46 | 1099.58 | 1280.63 | 32.92 | 40.95 | 51.15 | 18838.52 | 139324/143771 (96.9%) | ✅ |
|
||||
| c64_i4096_o512 | 4096 | 512 | tp4_dp2 | 64 | 8.10 | 2143.08 | 946.13 | 974.08 | 26.25 | 32.55 | 35.60 | 15173.06 | 139456/143771 (97.0%) | ✅ |
|
||||
| c64_i4096_o512 | 4096 | 512 | tp8_dp1 | 64 | 11.71 | 3100.03 | 584.91 | 601.95 | 17.83 | 18.78 | 19.95 | 9668.74 | 132932/143771 (92.5%) | ✅ |
|
||||
| c64_i4096_o1024 | 4096 | 1024 | tp2_dp4 | 64 | 4.39 | 2351.38 | 1083.22 | 1200.04 | 24.41 | 28.13 | 32.24 | 26088.66 | 139324/143771 (96.9%) | ✅ |
|
||||
| c64_i4096_o1024 | 4096 | 1024 | tp4_dp2 | 64 | 5.45 | 2915.21 | 1023.51 | 1212.54 | 19.18 | 21.78 | 24.49 | 21920.34 | 139824/143771 (97.3%) | ✅ |
|
||||
| c64_i4096_o1024 | 4096 | 1024 | tp8_dp1 | 64 | 6.14 | 3283.93 | 789.79 | 800.08 | 17.06 | 17.95 | 19.33 | 17906.82 | 132932/143771 (92.5%) | ✅ |
|
||||
| c64_i4096_o2048 | 4096 | 2048 | tp2_dp4 | 64 | 2.73 | 2846.07 | 1149.71 | 1194.86 | 19.80 | 22.43 | 24.96 | 41227.42 | 139324/143771 (96.9%) | ✅ |
|
||||
| c64_i4096_o2048 | 4096 | 2048 | tp4_dp2 | 64 | 3.04 | 3165.95 | 856.07 | 869.36 | 17.64 | 18.77 | 20.69 | 37188.62 | 139824/143771 (97.3%) | ✅ |
|
||||
| c64_i4096_o2048 | 4096 | 2048 | tp8_dp1 | 64 | 3.28 | 3408.88 | 570.75 | 582.03 | 16.56 | 17.06 | 18.07 | 34336.54 | 132932/143771 (92.5%) | ✅ |
|
||||
| c64_i4096_o4096 | 4096 | 4096 | tp2_dp4 | 64 | 1.49 | 3050.95 | 900.93 | 999.74 | 17.93 | 19.28 | 20.04 | 75036.72 | 139324/143771 (96.9%) | ✅ |
|
||||
| c64_i4096_o4096 | 4096 | 4096 | tp4_dp2 | 64 | 1.62 | 3323.06 | 727.11 | 753.66 | 16.53 | 17.22 | 17.64 | 68039.71 | 139824/143771 (97.3%) | ✅ |
|
||||
| c64_i4096_o4096 | 4096 | 4096 | tp8_dp1 | 64 | 1.67 | 3408.94 | 576.41 | 599.00 | 16.44 | 16.79 | 16.98 | 68183.67 | 132932/143771 (92.5%) | ✅ |
|
||||
| c128_i1024_o128 | 1024 | 128 | tp2_dp4 | 128 | 14.09 | 919.53 | 1787.94 | 1815.68 | 131.99 | 159.76 | 191.60 | 19385.13 | 139322/143771 (96.9%) | ⚠️ |
|
||||
| c128_i1024_o128 | 1024 | 128 | tp4_dp2 | 128 | 15.24 | 994.43 | 1659.18 | 1917.69 | 122.83 | 156.63 | 189.20 | 18386.47 | 139454/143771 (97.0%) | ⚠️ |
|
||||
| c128_i1024_o128 | 1024 | 128 | tp8_dp1 | 128 | 25.21 | 1644.87 | 1647.93 | 2127.36 | 71.51 | 108.98 | 146.42 | 11987.25 | 132930/143771 (92.5%) | ⚠️ |
|
||||
| c128_i1024_o256 | 1024 | 256 | tp2_dp4 | 128 | 12.51 | 1654.84 | 1302.99 | 1324.67 | 72.57 | 92.16 | 105.85 | 20487.34 | 139322/143771 (96.9%) | ⚠️ |
|
||||
| c128_i1024_o256 | 1024 | 256 | tp4_dp2 | 128 | 15.10 | 1996.42 | 938.43 | 1171.74 | 60.05 | 74.97 | 96.47 | 16952.04 | 139454/143771 (97.0%) | ⚠️ |
|
||||
| c128_i1024_o256 | 1024 | 256 | tp8_dp1 | 128 | 23.31 | 3082.56 | 725.46 | 741.75 | 37.61 | 47.17 | 56.15 | 10662.58 | 132930/143771 (92.5%) | ✅ |
|
||||
| c128_i1024_o512 | 1024 | 512 | tp2_dp4 | 128 | 9.80 | 2483.34 | 1236.35 | 1486.37 | 47.72 | 59.20 | 70.21 | 26017.94 | 139322/143771 (96.9%) | ✅ |
|
||||
| c128_i1024_o512 | 1024 | 512 | tp4_dp2 | 128 | 11.78 | 2986.46 | 996.03 | 1011.18 | 39.13 | 48.34 | 56.02 | 22283.80 | 139456/143771 (97.0%) | ✅ |
|
||||
| c128_i1024_o512 | 1024 | 512 | tp8_dp1 | 128 | 17.09 | 4330.93 | 760.28 | 777.02 | 26.29 | 29.37 | 34.23 | 14073.81 | 132932/143771 (92.5%) | ✅ |
|
||||
| c128_i1024_o1024 | 1024 | 1024 | tp2_dp4 | 128 | 7.08 | 3578.74 | 1147.11 | 1259.64 | 32.79 | 39.48 | 50.72 | 34693.91 | 139322/143771 (96.9%) | ✅ |
|
||||
| c128_i1024_o1024 | 1024 | 1024 | tp4_dp2 | 128 | 8.62 | 4359.31 | 719.08 | 837.92 | 26.31 | 30.84 | 33.28 | 27367.26 | 139456/143771 (97.0%) | ✅ |
|
||||
| c128_i1024_o1024 | 1024 | 1024 | tp8_dp1 | 128 | 9.35 | 4727.69 | 724.06 | 738.02 | 24.27 | 25.99 | 28.67 | 25355.91 | 132932/143771 (92.5%) | ✅ |
|
||||
| c128_i1024_o2048 | 1024 | 2048 | tp2_dp4 | 128 | 4.40 | 4467.90 | 990.93 | 1038.67 | 25.24 | 28.25 | 31.35 | 52638.14 | 139322/143771 (96.9%) | ✅ |
|
||||
| c128_i1024_o2048 | 1024 | 2048 | tp4_dp2 | 128 | 4.83 | 4909.69 | 979.55 | 1002.42 | 22.76 | 24.25 | 25.28 | 47303.18 | 139456/143771 (97.0%) | ✅ |
|
||||
| c128_i1024_o2048 | 1024 | 2048 | tp8_dp1 | 128 | 4.80 | 4871.12 | 746.05 | 761.60 | 23.37 | 24.26 | 24.89 | 48201.59 | 132932/143771 (92.5%) | ✅ |
|
||||
| c128_i1024_o4096 | 1024 | 4096 | tp2_dp4 | 128 | 2.42 | 4961.16 | 901.88 | 946.01 | 22.33 | 23.78 | 25.83 | 91949.11 | 139322/143771 (96.9%) | ✅ |
|
||||
| c128_i1024_o4096 | 1024 | 4096 | tp4_dp2 | 128 | 2.49 | 5110.60 | 737.03 | 754.71 | 21.93 | 23.01 | 23.26 | 90427.39 | 139456/143771 (97.0%) | ✅ |
|
||||
| c128_i1024_o4096 | 1024 | 4096 | tp8_dp1 | 128 | 2.33 | 4775.68 | 736.09 | 756.63 | 23.99 | 25.92 | 26.62 | 97316.79 | 132932/143771 (92.5%) | ✅ |
|
||||
|
||||
## Best throughput per (ISL, DSL)
|
||||
|
||||
| ISL | DSL | Best Config | Concurrency | OutTok/s | TTFT P95(ms) | TPOT Mean(ms) | SLO |
|
||||
|---:|---:|---|---:|---:|---:|---:|---:|
|
||||
| 1024 | 128 | tp8_dp1 | 128 | 1644.87 | 1647.93 | 71.51 | ⚠️ |
|
||||
| 1024 | 256 | tp8_dp1 | 128 | 3082.56 | 725.46 | 37.61 | ✅ |
|
||||
| 1024 | 512 | tp8_dp1 | 128 | 4330.93 | 760.28 | 26.29 | ✅ |
|
||||
| 1024 | 1024 | tp8_dp1 | 128 | 4727.69 | 724.06 | 24.27 | ✅ |
|
||||
| 1024 | 2048 | tp4_dp2 | 128 | 4909.69 | 979.55 | 22.76 | ✅ |
|
||||
| 1024 | 4096 | tp4_dp2 | 128 | 5110.60 | 737.03 | 21.93 | ✅ |
|
||||
| 4096 | 128 | tp8_dp1 | 64 | 772.92 | 2978.27 | 76.32 | ⚠️ |
|
||||
| 4096 | 256 | tp8_dp1 | 64 | 2518.43 | 804.28 | 22.25 | ✅ |
|
||||
| 4096 | 512 | tp8_dp1 | 64 | 3100.03 | 584.91 | 17.83 | ✅ |
|
||||
| 4096 | 1024 | tp8_dp1 | 64 | 3283.93 | 789.79 | 17.06 | ✅ |
|
||||
| 4096 | 2048 | tp8_dp1 | 64 | 3408.88 | 570.75 | 16.56 | ✅ |
|
||||
| 4096 | 4096 | tp8_dp1 | 64 | 3408.94 | 576.41 | 16.44 | ✅ |
|
||||
| 16384 | 128 | tp8_dp1 | 32 | 356.17 | 3789.36 | 78.86 | ❌ |
|
||||
| 16384 | 256 | tp8_dp1 | 32 | 1782.54 | 636.48 | 14.84 | ✅ |
|
||||
| 16384 | 512 | tp8_dp1 | 32 | 1977.19 | 640.86 | 14.04 | ✅ |
|
||||
| 16384 | 1024 | tp8_dp1 | 32 | 2122.35 | 789.94 | 13.38 | ✅ |
|
||||
| 16384 | 2048 | tp8_dp1 | 32 | 2226.25 | 628.02 | 12.88 | ✅ |
|
||||
| 65536 | 128 | tp8_dp1 | 8 | 64.77 | 7445.34 | 88.42 | ❌ |
|
||||
| 65536 | 256 | tp8_dp1 | 8 | 608.09 | 749.11 | 9.14 | ✅ |
|
||||
| 65536 | 512 | tp8_dp1 | 8 | 705.69 | 661.39 | 9.10 | ✅ |
|
||||
| 65536 | 1024 | tp8_dp1 | 8 | 734.78 | 671.08 | 8.91 | ✅ |
|
||||
| 131072 | 128 | tp8_dp1 | 4 | 46.76 | 5665.45 | 46.92 | ⚠️ |
|
||||
| 131072 | 256 | tp8_dp1 | 4 | 306.01 | 826.11 | 7.11 | ✅ |
|
||||
| 131072 | 512 | tp8_dp1 | 4 | 382.40 | 581.21 | 7.67 | ✅ |
|
||||
| 262144 | 128 | tp8_dp1 | 2 | 43.70 | 6405.29 | 12.04 | ⚠️ |
|
||||
| 262144 | 256 | tp8_dp1 | 2 | 171.18 | 872.87 | 6.69 | ✅ |
|
||||
| 524288 | 128 | tp8_dp1 | 2 | 133.35 | 1167.12 | 5.64 | ✅ |
|
||||
| 1048576 | 128 | tp8_dp1 | 1 | 7.26 | 38265.96 | 4.67 | ⚠️ |
|
||||
|
||||
## Notes
|
||||
|
||||
- SLO check uses TTFT P95 and TPOT mean.
|
||||
- A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out.
|
||||
- `Peak GPU mem` shows peak used / total MB and utilization percentage.
|
||||
- Optional (P) combinations that failed are marked as skipped/OOM and do not break the run.
|
||||
@ -0,0 +1 @@
|
||||
engine,tp,dp,mark,isl,dsl,concurrency,status,reason,detail_log
|
||||
|
@ -0,0 +1,70 @@
|
||||
# 8x NVIDIA H200 143GB VLLM Benchmark Report
|
||||
|
||||
- Result root: `/data/user1/yy/experiments/dsv4_h200_vllm_tp_dp_matrix/results/20260709-152419_matrix/tp2_dp4`
|
||||
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||
- Backend: VLLM
|
||||
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||
|
||||
## Results
|
||||
|
||||
| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Peak GPU mem | SLO |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c1_i1024_o128 | main | 1 | 1024 | 128 | 4.76 | 5 | 1.05 | 373.68 | 79.65 | 453.33 | 159.97 | 319.08 | 355.57 | 10.56 | 10.99 | 11.00 | 950.14 | 1366.62 | 1406.84 | 138264/143771 MiB (96.2%) | ✅ |
|
||||
| c128_i1024_o128 | main | 128 | 1024 | 128 | 45.42 | 640 | 14.09 | 7372.62 | 919.53 | 8292.15 | 682.86 | 1787.94 | 1815.68 | 131.99 | 159.76 | 191.60 | 8857.59 | 17800.52 | 19385.13 | 139322/143771 MiB (96.9%) | ⚠️ |
|
||||
| c1_i1024_o256 | main | 1 | 1024 | 256 | 5.98 | 5 | 0.84 | 297.55 | 84.85 | 382.39 | 96.69 | 107.93 | 108.38 | 11.01 | 11.57 | 11.61 | 1193.49 | 1982.06 | 2082.56 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c128_i1024_o256 | main | 128 | 1024 | 256 | 51.15 | 640 | 12.51 | 6546.63 | 1654.84 | 8201.47 | 455.67 | 1302.99 | 1324.67 | 72.57 | 92.16 | 105.85 | 9769.30 | 18928.98 | 20487.34 | 139322/143771 MiB (96.9%) | ⚠️ |
|
||||
| c1_i1024_o512 | main | 1 | 1024 | 512 | 6.34 | 5 | 0.79 | 280.27 | 79.92 | 360.18 | 95.33 | 113.71 | 115.02 | 11.50 | 11.85 | 11.86 | 1267.01 | 2173.11 | 2303.60 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c128_i1024_o512 | main | 128 | 1024 | 512 | 65.32 | 640 | 9.80 | 5126.38 | 2483.34 | 7609.72 | 392.20 | 1236.35 | 1486.37 | 47.72 | 59.20 | 70.21 | 12165.11 | 24348.08 | 26017.94 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c1_i1024_o1024 | main | 1 | 1024 | 1024 | 18.32 | 5 | 0.27 | 97.06 | 83.58 | 180.64 | 98.32 | 100.14 | 100.52 | 11.71 | 11.90 | 11.93 | 3661.85 | 8044.41 | 8200.76 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c128_i1024_o1024 | main | 128 | 1024 | 1024 | 90.39 | 640 | 7.08 | 3704.45 | 3578.74 | 7283.20 | 340.83 | 1147.11 | 1259.64 | 32.79 | 39.48 | 50.72 | 16393.11 | 32588.31 | 34693.91 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c1_i1024_o2048 | main | 1 | 1024 | 2048 | 66.40 | 5 | 0.08 | 26.78 | 84.74 | 111.51 | 95.99 | 105.11 | 105.45 | 11.70 | 11.81 | 11.81 | 13278.26 | 20062.55 | 20192.89 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c128_i1024_o2048 | main | 128 | 1024 | 2048 | 145.52 | 640 | 4.40 | 2301.16 | 4467.90 | 6769.06 | 282.02 | 990.93 | 1038.67 | 25.24 | 28.25 | 31.35 | 25564.98 | 48968.45 | 52638.14 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c1_i1024_o4096 | main | 1 | 1024 | 4096 | 136.81 | 5 | 0.04 | 13.00 | 86.04 | 99.04 | 98.28 | 114.24 | 116.37 | 11.59 | 11.84 | 11.86 | 27359.51 | 42290.37 | 43455.08 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c128_i1024_o4096 | main | 128 | 1024 | 4096 | 264.80 | 640 | 2.42 | 1264.57 | 4961.16 | 6225.74 | 265.10 | 901.88 | 946.01 | 22.33 | 23.78 | 25.83 | 45632.76 | 86442.33 | 91949.11 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c1_i4096_o128 | main | 1 | 4096 | 128 | 5.45 | 5 | 0.92 | 1829.89 | 69.56 | 1899.45 | 221.18 | 353.45 | 371.20 | 11.63 | 11.95 | 12.00 | 1088.26 | 1560.46 | 1608.68 | 139322/143771 MiB (96.9%) | ✅ |
|
||||
| c64_i4096_o128 | main | 64 | 4096 | 128 | 29.05 | 320 | 11.02 | 23259.85 | 721.10 | 23980.95 | 566.58 | 1790.01 | 2034.19 | 83.97 | 111.62 | 188.61 | 5607.28 | 11212.93 | 12237.14 | 139324/143771 MiB (96.9%) | ⚠️ |
|
||||
| c1_i4096_o256 | main | 1 | 4096 | 256 | 6.50 | 5 | 0.77 | 1534.00 | 78.01 | 1612.01 | 128.46 | 208.90 | 226.63 | 11.61 | 11.87 | 11.89 | 1298.45 | 2189.81 | 2296.91 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c64_i4096_o256 | main | 64 | 4096 | 256 | 37.63 | 320 | 8.50 | 17956.97 | 1101.01 | 19057.98 | 447.33 | 1290.52 | 1667.29 | 55.00 | 72.09 | 89.93 | 7132.99 | 14073.09 | 15631.13 | 139324/143771 MiB (96.9%) | ⚠️ |
|
||||
| c1_i4096_o512 | main | 1 | 4096 | 512 | 6.48 | 5 | 0.77 | 1539.75 | 78.30 | 1618.05 | 124.97 | 210.04 | 232.13 | 11.63 | 11.95 | 11.98 | 1293.49 | 2240.30 | 2378.47 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c64_i4096_o512 | main | 64 | 4096 | 512 | 49.46 | 320 | 6.47 | 13661.57 | 1712.46 | 15374.03 | 363.21 | 1099.58 | 1280.63 | 32.92 | 40.95 | 51.15 | 9074.10 | 17834.26 | 18838.52 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c1_i4096_o1024 | main | 1 | 4096 | 1024 | 18.31 | 5 | 0.27 | 544.41 | 83.60 | 628.01 | 103.66 | 113.91 | 115.64 | 11.55 | 11.86 | 11.87 | 3661.10 | 8164.68 | 8360.74 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c64_i4096_o1024 | main | 64 | 4096 | 1024 | 72.82 | 320 | 4.39 | 9278.68 | 2351.38 | 11630.06 | 336.61 | 1083.22 | 1200.04 | 24.41 | 28.13 | 32.24 | 13050.59 | 25157.60 | 26088.66 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c1_i4096_o2048 | main | 1 | 4096 | 2048 | 65.21 | 5 | 0.08 | 152.90 | 86.30 | 239.20 | 107.97 | 122.31 | 124.03 | 11.54 | 11.75 | 11.78 | 13039.42 | 19438.63 | 19528.19 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c64_i4096_o2048 | main | 64 | 4096 | 2048 | 117.01 | 320 | 2.73 | 5774.33 | 2846.07 | 8620.40 | 297.64 | 1149.71 | 1194.86 | 19.80 | 22.43 | 24.96 | 20535.82 | 39745.31 | 41227.42 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c1_i4096_o4096 | main | 1 | 4096 | 4096 | 137.51 | 5 | 0.04 | 72.50 | 85.60 | 158.10 | 114.72 | 130.43 | 132.68 | 11.66 | 11.87 | 11.89 | 27500.53 | 42270.91 | 43588.26 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c64_i4096_o4096 | main | 64 | 4096 | 4096 | 214.54 | 320 | 1.49 | 3149.26 | 3050.95 | 6200.21 | 274.73 | 900.93 | 999.74 | 17.93 | 19.28 | 20.04 | 36687.19 | 72324.60 | 75036.72 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c1_i16384_o128 | main | 1 | 16384 | 128 | 5.71 | 5 | 0.88 | 7481.41 | 66.35 | 7547.76 | 320.96 | 599.10 | 618.22 | 11.00 | 11.48 | 11.54 | 1140.76 | 1856.53 | 1948.72 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c32_i16384_o128 | main | 32 | 16384 | 128 | 37.11 | 160 | 4.31 | 34791.25 | 284.72 | 35075.97 | 956.28 | 3298.27 | 4051.80 | 107.21 | 186.59 | 376.91 | 7222.82 | 13755.15 | 15245.93 | 139324/143771 MiB (96.9%) | ❌ |
|
||||
| c1_i16384_o256 | main | 1 | 16384 | 256 | 7.07 | 5 | 0.71 | 6043.42 | 71.69 | 6115.12 | 256.48 | 448.96 | 486.54 | 11.52 | 11.66 | 11.67 | 1412.57 | 2447.12 | 2621.34 | 139324/143771 MiB (96.9%) | ✅ |
|
||||
| c32_i16384_o256 | main | 32 | 16384 | 256 | 35.65 | 160 | 4.49 | 36211.60 | 623.03 | 36834.63 | 680.08 | 1913.91 | 2384.97 | 47.34 | 71.67 | 147.97 | 6749.38 | 12780.12 | 13936.33 | 139684/143771 MiB (97.2%) | ✅ |
|
||||
| c1_i16384_o512 | main | 1 | 16384 | 512 | 6.52 | 5 | 0.77 | 6550.19 | 77.70 | 6627.89 | 139.05 | 188.54 | 190.96 | 11.55 | 11.73 | 11.74 | 1303.13 | 2227.30 | 2356.50 | 139684/143771 MiB (97.2%) | ✅ |
|
||||
| c32_i16384_o512 | main | 32 | 16384 | 512 | 42.41 | 160 | 3.77 | 30447.27 | 952.48 | 31399.75 | 538.18 | 1520.07 | 2259.12 | 30.21 | 45.07 | 57.04 | 7937.47 | 16413.95 | 18769.12 | 139684/143771 MiB (97.2%) | ✅ |
|
||||
| c1_i16384_o1024 | main | 1 | 16384 | 1024 | 18.25 | 5 | 0.27 | 2341.54 | 83.88 | 2425.42 | 188.47 | 322.92 | 349.35 | 11.31 | 11.64 | 11.64 | 3648.93 | 7809.61 | 7915.88 | 139684/143771 MiB (97.2%) | ✅ |
|
||||
| c32_i16384_o1024 | main | 32 | 16384 | 1024 | 63.50 | 160 | 2.52 | 20332.49 | 1337.54 | 21670.03 | 485.63 | 1871.97 | 2404.37 | 22.95 | 27.86 | 72.28 | 11657.91 | 21919.16 | 22977.81 | 139684/143771 MiB (97.2%) | ✅ |
|
||||
| c1_i16384_o2048 | main | 1 | 16384 | 2048 | 65.72 | 5 | 0.08 | 650.30 | 85.62 | 735.91 | 175.36 | 249.46 | 261.08 | 11.54 | 11.64 | 11.65 | 13142.27 | 19985.51 | 20210.50 | 139684/143771 MiB (97.2%) | ✅ |
|
||||
| c32_i16384_o2048 | main | 32 | 16384 | 2048 | 102.63 | 160 | 1.56 | 12580.64 | 1785.47 | 14366.11 | 336.45 | 1089.13 | 1111.26 | 15.83 | 17.47 | 19.70 | 18450.05 | 31359.53 | 33430.94 | 139684/143771 MiB (97.2%) | ✅ |
|
||||
| c1_i65536_o128 | main | 1 | 65536 | 128 | 11.40 | 5 | 0.44 | 15249.82 | 33.25 | 15283.07 | 1429.50 | 2811.65 | 2892.88 | 11.47 | 12.08 | 12.21 | 2278.14 | 4059.40 | 4233.91 | 139706/143771 MiB (97.2%) | ✅ |
|
||||
| c8_i65536_o128 | main | 8 | 65536 | 128 | 46.50 | 40 | 0.86 | 31649.77 | 55.23 | 31705.00 | 2642.31 | 5855.87 | 7061.54 | 115.39 | 327.74 | 399.92 | 9193.45 | 17992.56 | 18685.86 | 139708/143771 MiB (97.2%) | ❌ |
|
||||
| c1_i65536_o256 | main | 1 | 65536 | 256 | 10.35 | 5 | 0.48 | 16785.56 | 48.96 | 16834.52 | 902.47 | 2181.49 | 2314.91 | 11.55 | 11.95 | 11.96 | 2069.19 | 3540.82 | 3709.96 | 139708/143771 MiB (97.2%) | ✅ |
|
||||
| c8_i65536_o256 | main | 8 | 65536 | 256 | 44.64 | 40 | 0.90 | 32962.95 | 117.73 | 33080.69 | 1800.38 | 4029.43 | 4806.74 | 54.45 | 101.30 | 122.71 | 8621.68 | 16550.72 | 18052.15 | 139708/143771 MiB (97.2%) | ❌ |
|
||||
| c1_i65536_o512 | main | 1 | 65536 | 512 | 8.02 | 5 | 0.62 | 21662.85 | 63.19 | 21726.04 | 501.88 | 1276.37 | 1453.96 | 11.06 | 11.68 | 11.78 | 1602.98 | 2564.65 | 2624.42 | 139708/143771 MiB (97.2%) | ✅ |
|
||||
| c8_i65536_o512 | main | 8 | 65536 | 512 | 40.84 | 40 | 0.98 | 36028.75 | 272.84 | 36301.59 | 1260.91 | 3168.19 | 5671.91 | 24.56 | 51.52 | 85.56 | 7629.47 | 14481.52 | 18250.24 | 139708/143771 MiB (97.2%) | ⚠️ |
|
||||
| c1_i65536_o1024 | main | 1 | 65536 | 1024 | 18.42 | 5 | 0.27 | 9437.05 | 83.13 | 9520.17 | 255.57 | 375.58 | 383.90 | 11.12 | 11.57 | 11.58 | 3681.79 | 8015.79 | 8213.89 | 139708/143771 MiB (97.2%) | ✅ |
|
||||
| c8_i65536_o1024 | main | 8 | 65536 | 1024 | 62.34 | 40 | 0.64 | 23606.18 | 351.24 | 23957.42 | 962.35 | 2850.09 | 2967.29 | 19.43 | 29.44 | 36.68 | 11157.95 | 21105.82 | 21507.37 | 139708/143771 MiB (97.2%) | ✅ |
|
||||
| c1_i131072_o128 | main | 1 | 131072 | 128 | 15.53 | 5 | 0.32 | 23847.78 | 24.40 | 23872.18 | 2286.42 | 3831.71 | 3847.18 | 11.06 | 11.65 | 11.68 | 3104.64 | 4990.35 | 5071.55 | 140228/143771 MiB (97.5%) | ⚠️ |
|
||||
| c4_i131072_o128 | main | 4 | 131072 | 128 | 27.14 | 20 | 0.74 | 54328.24 | 41.97 | 54370.21 | 2648.47 | 5381.79 | 6485.26 | 63.46 | 234.23 | 254.36 | 5273.33 | 9530.16 | 9800.84 | 140228/143771 MiB (97.5%) | ❌ |
|
||||
| c1_i131072_o256 | main | 1 | 131072 | 256 | 10.82 | 5 | 0.46 | 34222.74 | 46.84 | 34269.59 | 1053.51 | 3143.30 | 3630.53 | 11.11 | 11.90 | 11.97 | 2162.71 | 4514.33 | 4950.01 | 140228/143771 MiB (97.5%) | ⚠️ |
|
||||
| c4_i131072_o256 | main | 4 | 131072 | 256 | 39.32 | 20 | 0.51 | 37498.25 | 64.78 | 37563.03 | 2168.98 | 4431.12 | 6485.73 | 38.02 | 82.59 | 93.31 | 7621.59 | 19293.47 | 21326.10 | 140228/143771 MiB (97.5%) | ⚠️ |
|
||||
| c1_i131072_o512 | main | 1 | 131072 | 512 | 7.44 | 5 | 0.67 | 49763.22 | 68.11 | 49831.33 | 391.27 | 574.50 | 577.45 | 10.89 | 11.54 | 11.54 | 1487.11 | 2243.60 | 2318.95 | 140228/143771 MiB (97.5%) | ✅ |
|
||||
| c4_i131072_o512 | main | 4 | 131072 | 512 | 43.08 | 20 | 0.46 | 34227.86 | 136.39 | 34364.24 | 1588.49 | 5162.58 | 5751.13 | 24.26 | 43.94 | 51.73 | 8028.29 | 13969.64 | 14391.10 | 140228/143771 MiB (97.5%) | ⚠️ |
|
||||
| c1_i262144_o128 | main | 1 | 262144 | 128 | 33.40 | 5 | 0.15 | 22865.42 | 11.35 | 22876.77 | 5896.15 | 13444.27 | 14788.12 | 9.88 | 11.09 | 11.15 | 6677.66 | 14582.28 | 16032.30 | 140744/143771 MiB (97.9%) | ⚠️ |
|
||||
| c2_i262144_o128 | main | 2 | 262144 | 128 | 36.61 | 10 | 0.27 | 38069.90 | 20.65 | 38090.55 | 4480.80 | 9292.65 | 9433.23 | 24.30 | 74.40 | 75.57 | 6665.65 | 10320.34 | 10323.72 | 140744/143771 MiB (97.9%) | ⚠️ |
|
||||
| c1_i262144_o256 | main | 1 | 262144 | 256 | 27.87 | 5 | 0.18 | 27404.52 | 18.19 | 27422.71 | 4494.81 | 8943.43 | 9385.72 | 10.07 | 11.43 | 11.50 | 5571.33 | 10356.69 | 10671.30 | 141258/143771 MiB (98.3%) | ⚠️ |
|
||||
| c2_i262144_o256 | main | 2 | 262144 | 256 | 33.18 | 10 | 0.30 | 42010.29 | 42.07 | 42052.36 | 2785.85 | 9153.02 | 10642.68 | 24.92 | 66.10 | 74.81 | 6437.85 | 12049.75 | 12104.02 | 141258/143771 MiB (98.3%) | ⚠️ |
|
||||
| c1_i524288_o128 | main | 1 | 524288 | 128 | 38.85 | 5 | 0.13 | 26404.42 | 9.76 | 26414.18 | 7011.27 | 17139.22 | 19091.80 | 9.70 | 10.39 | 10.40 | 7768.26 | 18082.00 | 20099.12 | 141258/143771 MiB (98.3%) | ⚠️ |
|
||||
| c2_i524288_o128 | main | 2 | 524288 | 128 | 14.47 | 10 | 0.69 | 114432.83 | 52.24 | 114485.07 | 1445.03 | 4695.87 | 6463.46 | 15.77 | 44.14 | 65.40 | 2876.67 | 7604.53 | 8060.58 | 141258/143771 MiB (98.3%) | ⚠️ |
|
||||
| c1_i1048576_o128 | main | 1 | 1048576 | 128 | 78.14 | 5 | 0.06 | 19836.89 | 4.85 | 19841.74 | 14906.36 | 46081.66 | 51394.85 | 8.95 | 10.50 | 10.51 | 15626.56 | 46632.30 | 51849.45 | 141258/143771 MiB (98.3%) | ⚠️ |
|
||||
| c2_i1048576_o128 | main | 2 | 1048576 | 128 | 181.75 | 10 | 0.06 | 17765.33 | 4.16 | 17769.49 | 24258.31 | 61414.70 | 65340.99 | 117.07 | 500.43 | 531.18 | 30965.88 | 62664.75 | 66294.91 | 141780/143771 MiB (98.6%) | ❌ |
|
||||
|
||||
SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,57 @@
|
||||
mark input_len output_len concurrency num_prompts
|
||||
Y 1024 128 1 5
|
||||
Y 1024 128 128 640
|
||||
Y 1024 256 1 5
|
||||
Y 1024 256 128 640
|
||||
Y 1024 512 1 5
|
||||
Y 1024 512 128 640
|
||||
Y 1024 1024 1 5
|
||||
Y 1024 1024 128 640
|
||||
Y 1024 2048 1 5
|
||||
Y 1024 2048 128 640
|
||||
Y 1024 4096 1 5
|
||||
Y 1024 4096 128 640
|
||||
Y 4096 128 1 5
|
||||
Y 4096 128 64 320
|
||||
Y 4096 256 1 5
|
||||
Y 4096 256 64 320
|
||||
Y 4096 512 1 5
|
||||
Y 4096 512 64 320
|
||||
Y 4096 1024 1 5
|
||||
Y 4096 1024 64 320
|
||||
Y 4096 2048 1 5
|
||||
Y 4096 2048 64 320
|
||||
Y 4096 4096 1 5
|
||||
Y 4096 4096 64 320
|
||||
Y 16384 128 1 5
|
||||
Y 16384 128 32 160
|
||||
Y 16384 256 1 5
|
||||
Y 16384 256 32 160
|
||||
Y 16384 512 1 5
|
||||
Y 16384 512 32 160
|
||||
Y 16384 1024 1 5
|
||||
Y 16384 1024 32 160
|
||||
Y 16384 2048 1 5
|
||||
Y 16384 2048 32 160
|
||||
Y 65536 128 1 5
|
||||
Y 65536 128 8 40
|
||||
Y 65536 256 1 5
|
||||
Y 65536 256 8 40
|
||||
Y 65536 512 1 5
|
||||
Y 65536 512 8 40
|
||||
Y 65536 1024 1 5
|
||||
Y 65536 1024 8 40
|
||||
Y 131072 128 1 5
|
||||
Y 131072 128 4 20
|
||||
Y 131072 256 1 5
|
||||
Y 131072 256 4 20
|
||||
Y 131072 512 1 5
|
||||
Y 131072 512 4 20
|
||||
Y 262144 128 1 5
|
||||
Y 262144 128 2 10
|
||||
Y 262144 256 1 5
|
||||
Y 262144 256 2 10
|
||||
Y 524288 128 1 5
|
||||
Y 524288 128 2 10
|
||||
Y 1048576 128 1 5
|
||||
Y 1048576 128 2 10
|
||||
|
@ -0,0 +1,70 @@
|
||||
# 8x NVIDIA H200 143GB VLLM Benchmark Report
|
||||
|
||||
- Result root: `/data/user1/yy/experiments/dsv4_h200_vllm_tp_dp_matrix/results/20260709-152419_matrix/tp4_dp2`
|
||||
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||
- Backend: VLLM
|
||||
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||
|
||||
## Results
|
||||
|
||||
| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Peak GPU mem | SLO |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c1_i1024_o128 | main | 1 | 1024 | 128 | 3.72 | 5 | 1.34 | 477.80 | 101.85 | 579.65 | 134.63 | 302.85 | 345.83 | 8.17 | 8.32 | 8.35 | 742.89 | 1030.20 | 1055.00 | 138396/143771 MiB (96.3%) | ✅ |
|
||||
| c128_i1024_o128 | main | 128 | 1024 | 128 | 42.00 | 640 | 15.24 | 7973.13 | 994.43 | 8967.56 | 615.83 | 1659.18 | 1917.69 | 122.83 | 156.63 | 189.20 | 8188.50 | 16737.02 | 18386.47 | 139454/143771 MiB (97.0%) | ⚠️ |
|
||||
| c1_i1024_o256 | main | 1 | 1024 | 256 | 4.46 | 5 | 1.12 | 399.09 | 113.80 | 512.89 | 76.65 | 88.52 | 90.59 | 8.12 | 8.21 | 8.23 | 889.64 | 1486.72 | 1572.36 | 139454/143771 MiB (97.0%) | ✅ |
|
||||
| c128_i1024_o256 | main | 128 | 1024 | 256 | 42.40 | 640 | 15.10 | 7897.93 | 1996.42 | 9894.36 | 381.79 | 938.43 | 1171.74 | 60.05 | 74.97 | 96.47 | 8045.59 | 15999.15 | 16952.04 | 139454/143771 MiB (97.0%) | ⚠️ |
|
||||
| c1_i1024_o512 | main | 1 | 1024 | 512 | 4.58 | 5 | 1.09 | 388.12 | 110.67 | 498.80 | 101.05 | 187.00 | 208.64 | 8.12 | 8.17 | 8.18 | 914.85 | 1486.90 | 1572.17 | 139454/143771 MiB (97.0%) | ✅ |
|
||||
| c128_i1024_o512 | main | 128 | 1024 | 512 | 54.32 | 640 | 11.78 | 6164.96 | 2986.46 | 9151.42 | 316.84 | 996.03 | 1011.18 | 39.13 | 48.34 | 56.02 | 10050.16 | 20325.81 | 22283.80 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c1_i1024_o1024 | main | 1 | 1024 | 1024 | 12.71 | 5 | 0.39 | 139.93 | 120.49 | 260.41 | 75.65 | 90.47 | 90.97 | 8.11 | 8.19 | 8.21 | 2539.90 | 5592.97 | 5705.34 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c128_i1024_o1024 | main | 128 | 1024 | 1024 | 74.21 | 640 | 8.62 | 4512.43 | 4359.31 | 8871.74 | 224.26 | 719.08 | 837.92 | 26.31 | 30.84 | 33.28 | 13282.76 | 25759.40 | 27367.26 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c1_i1024_o2048 | main | 1 | 1024 | 2048 | 45.78 | 5 | 0.11 | 38.84 | 122.91 | 161.75 | 77.63 | 84.75 | 84.77 | 8.08 | 8.09 | 8.09 | 9154.40 | 13865.78 | 13978.61 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c128_i1024_o2048 | main | 128 | 1024 | 2048 | 132.42 | 640 | 4.83 | 2528.71 | 4909.69 | 7438.40 | 249.17 | 979.55 | 1002.42 | 22.76 | 24.25 | 25.28 | 23085.59 | 43805.99 | 47303.18 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c1_i1024_o4096 | main | 1 | 1024 | 4096 | 95.72 | 5 | 0.05 | 18.57 | 122.97 | 141.55 | 77.88 | 82.10 | 82.14 | 8.09 | 8.12 | 8.13 | 19143.13 | 29602.89 | 30482.21 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c128_i1024_o4096 | main | 128 | 1024 | 4096 | 257.05 | 640 | 2.49 | 1302.66 | 5110.60 | 6413.26 | 197.57 | 737.03 | 754.71 | 21.93 | 23.01 | 23.26 | 44808.43 | 84504.69 | 90427.39 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c1_i4096_o128 | main | 1 | 4096 | 128 | 3.47 | 5 | 1.44 | 2872.58 | 109.20 | 2981.78 | 83.99 | 96.00 | 96.62 | 8.16 | 8.23 | 8.24 | 692.79 | 1028.22 | 1054.23 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c64_i4096_o128 | main | 64 | 4096 | 128 | 30.68 | 320 | 10.43 | 22020.97 | 682.69 | 22703.66 | 584.11 | 1904.20 | 2262.19 | 88.50 | 123.52 | 173.68 | 5949.69 | 11822.35 | 12463.74 | 139456/143771 MiB (97.0%) | ⚠️ |
|
||||
| c1_i4096_o256 | main | 1 | 4096 | 256 | 4.51 | 5 | 1.11 | 2209.73 | 112.37 | 2322.10 | 83.55 | 91.50 | 91.90 | 8.16 | 8.23 | 8.24 | 900.98 | 1513.84 | 1604.71 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c64_i4096_o256 | main | 64 | 4096 | 256 | 31.29 | 320 | 10.23 | 21590.10 | 1323.77 | 22913.87 | 346.30 | 1198.05 | 1234.72 | 45.94 | 59.77 | 82.58 | 5923.11 | 11845.66 | 13085.09 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c1_i4096_o512 | main | 1 | 4096 | 512 | 4.55 | 5 | 1.10 | 2191.28 | 111.43 | 2302.71 | 91.00 | 104.51 | 107.11 | 8.16 | 8.24 | 8.25 | 908.65 | 1527.91 | 1621.06 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c64_i4096_o512 | main | 64 | 4096 | 512 | 39.52 | 320 | 8.10 | 17097.00 | 2143.08 | 19240.08 | 277.89 | 946.13 | 974.08 | 26.25 | 32.55 | 35.60 | 7228.62 | 14145.53 | 15173.06 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c1_i4096_o1024 | main | 1 | 4096 | 1024 | 12.82 | 5 | 0.39 | 777.58 | 119.41 | 896.98 | 82.49 | 92.40 | 93.46 | 8.15 | 8.23 | 8.24 | 2563.02 | 5648.98 | 5771.30 | 139456/143771 MiB (97.0%) | ✅ |
|
||||
| c64_i4096_o1024 | main | 64 | 4096 | 1024 | 58.73 | 320 | 5.45 | 11503.60 | 2915.21 | 14418.81 | 261.52 | 1023.51 | 1212.54 | 19.18 | 21.78 | 24.49 | 10436.37 | 20083.65 | 21920.34 | 139824/143771 MiB (97.3%) | ✅ |
|
||||
| c1_i4096_o2048 | main | 1 | 4096 | 2048 | 46.18 | 5 | 0.11 | 215.88 | 121.84 | 337.72 | 89.17 | 105.61 | 108.34 | 8.13 | 8.16 | 8.16 | 9235.07 | 14025.34 | 14152.06 | 139824/143771 MiB (97.3%) | ✅ |
|
||||
| c64_i4096_o2048 | main | 64 | 4096 | 2048 | 105.18 | 320 | 3.04 | 6423.33 | 3165.95 | 9589.28 | 219.53 | 856.07 | 869.36 | 17.64 | 18.77 | 20.69 | 18415.93 | 35312.14 | 37188.62 | 139824/143771 MiB (97.3%) | ✅ |
|
||||
| c1_i4096_o4096 | main | 1 | 4096 | 4096 | 96.28 | 5 | 0.05 | 103.55 | 122.26 | 225.82 | 92.47 | 110.67 | 114.17 | 8.14 | 8.17 | 8.17 | 19254.34 | 29794.95 | 30710.72 | 139824/143771 MiB (97.3%) | ✅ |
|
||||
| c64_i4096_o4096 | main | 64 | 4096 | 4096 | 196.97 | 320 | 1.62 | 3430.14 | 3323.06 | 6753.20 | 191.83 | 727.11 | 753.66 | 16.53 | 17.22 | 17.64 | 33830.38 | 65711.11 | 68039.71 | 139824/143771 MiB (97.3%) | ✅ |
|
||||
| c1_i16384_o128 | main | 1 | 16384 | 128 | 4.39 | 5 | 1.14 | 9724.92 | 86.24 | 9811.16 | 272.22 | 512.76 | 518.98 | 8.11 | 8.23 | 8.25 | 877.50 | 1371.53 | 1441.62 | 139824/143771 MiB (97.3%) | ✅ |
|
||||
| c32_i16384_o128 | main | 32 | 16384 | 128 | 31.01 | 160 | 5.16 | 41639.17 | 340.76 | 41979.93 | 908.42 | 3479.51 | 3760.82 | 85.23 | 160.17 | 235.33 | 6030.16 | 11373.02 | 12999.09 | 139824/143771 MiB (97.3%) | ❌ |
|
||||
| c1_i16384_o256 | main | 1 | 16384 | 256 | 5.10 | 5 | 0.98 | 8380.37 | 99.42 | 8479.79 | 203.25 | 407.06 | 440.10 | 8.14 | 8.23 | 8.24 | 1018.57 | 1804.68 | 1937.46 | 139824/143771 MiB (97.3%) | ✅ |
|
||||
| c32_i16384_o256 | main | 32 | 16384 | 256 | 29.56 | 160 | 5.41 | 43677.33 | 751.48 | 44428.80 | 597.59 | 1701.48 | 2572.97 | 38.88 | 55.35 | 126.86 | 5607.62 | 10467.65 | 11645.26 | 140338/143771 MiB (97.6%) | ✅ |
|
||||
| c1_i16384_o512 | main | 1 | 16384 | 512 | 4.66 | 5 | 1.07 | 9162.19 | 108.69 | 9270.88 | 114.10 | 155.92 | 157.11 | 8.16 | 8.23 | 8.25 | 931.42 | 1574.32 | 1661.52 | 140338/143771 MiB (97.6%) | ✅ |
|
||||
| c32_i16384_o512 | main | 32 | 16384 | 512 | 29.42 | 160 | 5.44 | 43889.80 | 1373.00 | 45262.81 | 313.79 | 965.45 | 1010.01 | 20.31 | 25.61 | 28.12 | 5422.34 | 10895.57 | 12002.66 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c1_i16384_o1024 | main | 1 | 16384 | 1024 | 13.04 | 5 | 0.38 | 3277.50 | 117.41 | 3394.91 | 114.24 | 158.31 | 161.09 | 8.16 | 8.23 | 8.24 | 2606.54 | 5717.71 | 5837.98 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c32_i16384_o1024 | main | 32 | 16384 | 1024 | 49.82 | 160 | 3.21 | 25915.11 | 1704.78 | 27619.89 | 303.13 | 1131.20 | 1660.57 | 17.14 | 21.79 | 37.19 | 9072.32 | 16361.33 | 18824.49 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c1_i16384_o2048 | main | 1 | 16384 | 2048 | 46.44 | 5 | 0.11 | 920.20 | 121.16 | 1041.36 | 117.87 | 155.57 | 156.44 | 8.14 | 8.18 | 8.19 | 9287.27 | 14109.44 | 14233.84 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c32_i16384_o2048 | main | 32 | 16384 | 2048 | 92.42 | 160 | 1.73 | 13970.52 | 1982.72 | 15953.24 | 323.18 | 1143.92 | 2172.49 | 14.60 | 16.68 | 25.49 | 16690.84 | 28308.53 | 30322.94 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c1_i65536_o128 | main | 1 | 65536 | 128 | 8.91 | 5 | 0.56 | 19510.10 | 42.54 | 19552.65 | 1193.16 | 2211.56 | 2223.60 | 7.89 | 8.16 | 8.19 | 1780.28 | 3036.84 | 3092.42 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c8_i65536_o128 | main | 8 | 65536 | 128 | 42.45 | 40 | 0.94 | 34665.79 | 60.49 | 34726.28 | 2187.88 | 5620.08 | 5766.46 | 118.91 | 301.63 | 309.58 | 8428.69 | 16394.59 | 16549.56 | 140854/143771 MiB (98.0%) | ❌ |
|
||||
| c1_i65536_o256 | main | 1 | 65536 | 256 | 4.96 | 5 | 1.01 | 35030.52 | 102.18 | 35132.70 | 190.89 | 274.65 | 283.52 | 7.97 | 8.23 | 8.25 | 990.97 | 1580.31 | 1647.86 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c8_i65536_o256 | main | 8 | 65536 | 256 | 30.60 | 40 | 1.31 | 48097.46 | 171.79 | 48269.25 | 1187.02 | 3353.05 | 4860.71 | 44.63 | 113.59 | 141.58 | 5938.64 | 14918.88 | 15303.64 | 140854/143771 MiB (98.0%) | ⚠️ |
|
||||
| c1_i65536_o512 | main | 1 | 65536 | 512 | 5.10 | 5 | 0.98 | 34094.52 | 99.45 | 34193.97 | 221.62 | 326.62 | 330.25 | 7.91 | 8.17 | 8.18 | 1018.20 | 1589.95 | 1655.09 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c8_i65536_o512 | main | 8 | 65536 | 512 | 27.16 | 40 | 1.47 | 54172.15 | 410.24 | 54582.39 | 493.38 | 1809.22 | 2638.39 | 17.90 | 23.23 | 79.99 | 4993.77 | 8917.36 | 9676.38 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c1_i65536_o1024 | main | 1 | 65536 | 1024 | 13.63 | 5 | 0.37 | 12750.53 | 112.31 | 12862.85 | 229.69 | 358.50 | 364.11 | 8.07 | 8.26 | 8.26 | 2724.95 | 5751.59 | 5842.34 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c8_i65536_o1024 | main | 8 | 65536 | 1024 | 41.44 | 40 | 0.97 | 35510.70 | 528.38 | 36039.07 | 576.47 | 2177.55 | 2783.15 | 13.13 | 20.42 | 32.65 | 7344.83 | 12591.94 | 13878.42 | 140854/143771 MiB (98.0%) | ✅ |
|
||||
| c1_i131072_o128 | main | 1 | 131072 | 128 | 13.24 | 5 | 0.38 | 27974.16 | 28.62 | 28002.78 | 2074.69 | 3498.43 | 3510.98 | 7.74 | 8.18 | 8.22 | 2646.89 | 4297.81 | 4353.06 | 141374/143771 MiB (98.3%) | ⚠️ |
|
||||
| c4_i131072_o128 | main | 4 | 131072 | 128 | 32.85 | 20 | 0.61 | 44885.93 | 34.68 | 44920.61 | 2993.02 | 7688.92 | 7746.95 | 55.15 | 151.40 | 157.05 | 6305.32 | 12451.58 | 13546.32 | 141374/143771 MiB (98.3%) | ❌ |
|
||||
| c1_i131072_o256 | main | 1 | 131072 | 256 | 8.19 | 5 | 0.61 | 45250.75 | 61.94 | 45312.69 | 851.06 | 2612.01 | 3033.01 | 7.81 | 8.23 | 8.24 | 1636.29 | 3468.17 | 3829.62 | 141374/143771 MiB (98.3%) | ✅ |
|
||||
| c4_i131072_o256 | main | 4 | 131072 | 256 | 25.17 | 20 | 0.79 | 58583.41 | 101.20 | 58684.61 | 1578.82 | 6019.33 | 6092.09 | 20.62 | 43.90 | 46.90 | 4833.02 | 10204.22 | 10875.06 | 141374/143771 MiB (98.3%) | ⚠️ |
|
||||
| c1_i131072_o512 | main | 1 | 131072 | 512 | 5.60 | 5 | 0.89 | 66152.93 | 90.55 | 66243.48 | 333.30 | 555.04 | 573.87 | 7.81 | 8.24 | 8.27 | 1118.43 | 1626.84 | 1649.69 | 141374/143771 MiB (98.3%) | ✅ |
|
||||
| c4_i131072_o512 | main | 4 | 131072 | 512 | 19.45 | 20 | 1.03 | 75794.92 | 302.02 | 76096.94 | 489.33 | 724.28 | 1360.67 | 9.94 | 13.97 | 16.16 | 3412.52 | 5788.93 | 5861.60 | 141374/143771 MiB (98.3%) | ✅ |
|
||||
| c1_i262144_o128 | main | 1 | 262144 | 128 | 24.68 | 5 | 0.20 | 30936.55 | 15.35 | 30951.90 | 4393.17 | 8221.07 | 8613.48 | 6.70 | 7.64 | 7.66 | 4935.30 | 9036.75 | 9509.11 | 141374/143771 MiB (98.3%) | ⚠️ |
|
||||
| c2_i262144_o128 | main | 2 | 262144 | 128 | 30.76 | 10 | 0.33 | 45318.61 | 24.58 | 45343.19 | 3243.14 | 7994.94 | 8377.33 | 25.23 | 70.28 | 72.19 | 5572.85 | 9102.02 | 9106.67 | 141374/143771 MiB (98.3%) | ⚠️ |
|
||||
| c1_i262144_o256 | main | 1 | 262144 | 256 | 7.02 | 5 | 0.71 | 108820.99 | 72.25 | 108893.24 | 643.55 | 962.12 | 1036.06 | 6.92 | 7.88 | 7.93 | 1401.77 | 2062.13 | 2080.08 | 141374/143771 MiB (98.3%) | ✅ |
|
||||
| c2_i262144_o256 | main | 2 | 262144 | 256 | 20.99 | 10 | 0.48 | 66416.34 | 66.52 | 66482.86 | 1808.18 | 6399.54 | 6424.61 | 10.75 | 22.96 | 32.38 | 3491.75 | 7972.48 | 8063.51 | 141374/143771 MiB (98.3%) | ⚠️ |
|
||||
| c1_i524288_o128 | main | 1 | 524288 | 128 | 23.69 | 5 | 0.21 | 43291.93 | 16.00 | 43307.93 | 4207.11 | 14806.51 | 17551.38 | 6.63 | 7.42 | 7.46 | 4736.94 | 15565.54 | 18281.61 | 141374/143771 MiB (98.3%) | ⚠️ |
|
||||
| c2_i524288_o128 | main | 2 | 524288 | 128 | 15.61 | 10 | 0.64 | 106086.70 | 48.43 | 106135.13 | 1630.83 | 6241.67 | 9313.98 | 44.54 | 213.86 | 348.24 | 3078.79 | 10277.55 | 10806.98 | 141374/143771 MiB (98.3%) | ⚠️ |
|
||||
| c1_i1048576_o128 | main | 1 | 1048576 | 128 | 77.70 | 5 | 0.06 | 19948.43 | 4.88 | 19953.31 | 15035.42 | 43228.08 | 48170.56 | 6.14 | 7.51 | 7.56 | 15539.21 | 43599.43 | 48471.96 | 141374/143771 MiB (98.3%) | ⚠️ |
|
||||
| c2_i1048576_o128 | main | 2 | 1048576 | 128 | 165.96 | 10 | 0.06 | 19455.73 | 4.56 | 19460.29 | 19476.11 | 51577.89 | 52987.83 | 277.94 | 766.41 | 858.05 | 31427.72 | 55532.93 | 56546.05 | 141374/143771 MiB (98.3%) | ❌ |
|
||||
|
||||
SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,57 @@
|
||||
mark input_len output_len concurrency num_prompts
|
||||
Y 1024 128 1 5
|
||||
Y 1024 128 128 640
|
||||
Y 1024 256 1 5
|
||||
Y 1024 256 128 640
|
||||
Y 1024 512 1 5
|
||||
Y 1024 512 128 640
|
||||
Y 1024 1024 1 5
|
||||
Y 1024 1024 128 640
|
||||
Y 1024 2048 1 5
|
||||
Y 1024 2048 128 640
|
||||
Y 1024 4096 1 5
|
||||
Y 1024 4096 128 640
|
||||
Y 4096 128 1 5
|
||||
Y 4096 128 64 320
|
||||
Y 4096 256 1 5
|
||||
Y 4096 256 64 320
|
||||
Y 4096 512 1 5
|
||||
Y 4096 512 64 320
|
||||
Y 4096 1024 1 5
|
||||
Y 4096 1024 64 320
|
||||
Y 4096 2048 1 5
|
||||
Y 4096 2048 64 320
|
||||
Y 4096 4096 1 5
|
||||
Y 4096 4096 64 320
|
||||
Y 16384 128 1 5
|
||||
Y 16384 128 32 160
|
||||
Y 16384 256 1 5
|
||||
Y 16384 256 32 160
|
||||
Y 16384 512 1 5
|
||||
Y 16384 512 32 160
|
||||
Y 16384 1024 1 5
|
||||
Y 16384 1024 32 160
|
||||
Y 16384 2048 1 5
|
||||
Y 16384 2048 32 160
|
||||
Y 65536 128 1 5
|
||||
Y 65536 128 8 40
|
||||
Y 65536 256 1 5
|
||||
Y 65536 256 8 40
|
||||
Y 65536 512 1 5
|
||||
Y 65536 512 8 40
|
||||
Y 65536 1024 1 5
|
||||
Y 65536 1024 8 40
|
||||
Y 131072 128 1 5
|
||||
Y 131072 128 4 20
|
||||
Y 131072 256 1 5
|
||||
Y 131072 256 4 20
|
||||
Y 131072 512 1 5
|
||||
Y 131072 512 4 20
|
||||
Y 262144 128 1 5
|
||||
Y 262144 128 2 10
|
||||
Y 262144 256 1 5
|
||||
Y 262144 256 2 10
|
||||
Y 524288 128 1 5
|
||||
Y 524288 128 2 10
|
||||
Y 1048576 128 1 5
|
||||
Y 1048576 128 2 10
|
||||
|
@ -0,0 +1,70 @@
|
||||
# 8x NVIDIA H200 143GB VLLM Benchmark Report
|
||||
|
||||
- Result root: `/data/user1/yy/experiments/dsv4_h200_vllm_tp_dp_matrix/results/20260709-152419_matrix/tp8_dp1`
|
||||
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||
- Backend: VLLM
|
||||
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||
|
||||
## Results
|
||||
|
||||
| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Peak GPU mem | SLO |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| c1_i1024_o128 | main | 1 | 1024 | 128 | 3.11 | 5 | 1.61 | 571.42 | 121.80 | 693.22 | 117.30 | 274.43 | 314.97 | 6.74 | 6.78 | 6.79 | 620.97 | 851.31 | 871.45 | 131900/143771 MiB (91.7%) | ✅ |
|
||||
| c128_i1024_o128 | main | 128 | 1024 | 128 | 25.39 | 640 | 25.21 | 13188.24 | 1644.87 | 14833.11 | 486.29 | 1647.93 | 2127.36 | 71.51 | 108.98 | 146.42 | 4876.35 | 10467.31 | 11987.25 | 132930/143771 MiB (92.5%) | ⚠️ |
|
||||
| c1_i1024_o256 | main | 1 | 1024 | 256 | 3.58 | 5 | 1.40 | 497.17 | 141.77 | 638.94 | 40.11 | 44.63 | 45.58 | 6.71 | 6.72 | 6.72 | 713.89 | 1209.48 | 1281.54 | 132930/143771 MiB (92.5%) | ✅ |
|
||||
| c128_i1024_o256 | main | 128 | 1024 | 256 | 27.46 | 640 | 23.31 | 12194.74 | 3082.56 | 15277.30 | 258.20 | 725.46 | 741.75 | 37.61 | 47.17 | 56.15 | 5081.28 | 9930.41 | 10662.58 | 132930/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i1024_o512 | main | 1 | 1024 | 512 | 3.69 | 5 | 1.36 | 482.38 | 137.55 | 619.93 | 60.69 | 65.04 | 65.69 | 6.74 | 6.78 | 6.79 | 735.69 | 1233.35 | 1305.74 | 132930/143771 MiB (92.5%) | ✅ |
|
||||
| c128_i1024_o512 | main | 128 | 1024 | 512 | 37.45 | 640 | 17.09 | 8940.37 | 4330.93 | 13271.30 | 211.37 | 760.28 | 777.02 | 26.29 | 29.37 | 34.23 | 6728.56 | 13132.89 | 14073.81 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i1024_o1024 | main | 1 | 1024 | 1024 | 10.48 | 5 | 0.48 | 169.65 | 146.08 | 315.74 | 48.51 | 60.67 | 60.68 | 6.71 | 6.73 | 6.73 | 2094.65 | 4631.94 | 4727.62 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c128_i1024_o1024 | main | 128 | 1024 | 1024 | 68.42 | 640 | 9.35 | 4893.76 | 4727.69 | 9621.45 | 198.71 | 724.06 | 738.02 | 24.27 | 25.99 | 28.67 | 12260.92 | 23755.36 | 25355.91 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i1024_o2048 | main | 1 | 1024 | 2048 | 37.94 | 5 | 0.13 | 46.87 | 148.32 | 195.19 | 46.74 | 60.19 | 62.81 | 6.71 | 6.72 | 6.72 | 7585.93 | 11492.67 | 11586.89 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c128_i1024_o2048 | main | 128 | 1024 | 2048 | 133.47 | 640 | 4.80 | 2508.84 | 4871.12 | 7379.96 | 198.26 | 746.05 | 761.60 | 23.37 | 24.26 | 24.89 | 23639.36 | 44752.06 | 48201.59 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i1024_o4096 | main | 1 | 1024 | 4096 | 79.46 | 5 | 0.06 | 22.38 | 148.14 | 170.51 | 42.13 | 46.60 | 47.48 | 6.73 | 6.76 | 6.76 | 15890.77 | 24601.03 | 25332.98 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c128_i1024_o4096 | main | 128 | 1024 | 4096 | 275.08 | 640 | 2.33 | 1217.30 | 4775.68 | 5992.98 | 218.44 | 736.09 | 756.63 | 23.99 | 25.92 | 26.62 | 48923.09 | 92762.87 | 97316.79 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i4096_o128 | main | 1 | 4096 | 128 | 3.09 | 5 | 1.62 | 3231.34 | 122.84 | 3354.18 | 113.28 | 148.51 | 155.20 | 6.72 | 6.74 | 6.74 | 615.65 | 898.15 | 918.13 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c64_i4096_o128 | main | 64 | 4096 | 128 | 27.10 | 320 | 11.81 | 24931.54 | 772.92 | 25704.46 | 705.87 | 2978.27 | 3725.28 | 76.32 | 108.07 | 233.29 | 5252.69 | 10435.23 | 11989.51 | 132932/143771 MiB (92.5%) | ⚠️ |
|
||||
| c1_i4096_o256 | main | 1 | 4096 | 256 | 3.66 | 5 | 1.37 | 2724.98 | 138.57 | 2863.55 | 54.71 | 67.18 | 67.54 | 6.73 | 6.77 | 6.78 | 731.14 | 1243.27 | 1320.30 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c64_i4096_o256 | main | 64 | 4096 | 256 | 16.45 | 320 | 19.45 | 41074.35 | 2518.43 | 43592.78 | 216.47 | 804.28 | 815.33 | 22.25 | 25.88 | 28.53 | 2990.66 | 5802.25 | 6644.63 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i4096_o512 | main | 1 | 4096 | 512 | 3.66 | 5 | 1.37 | 2724.45 | 138.55 | 2863.00 | 53.85 | 64.63 | 65.10 | 6.73 | 6.78 | 6.78 | 730.54 | 1243.77 | 1320.67 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c64_i4096_o512 | main | 64 | 4096 | 512 | 27.32 | 320 | 11.71 | 24731.28 | 3100.03 | 27831.30 | 174.83 | 584.91 | 601.95 | 17.83 | 18.78 | 19.95 | 4860.53 | 9051.41 | 9668.74 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i4096_o1024 | main | 1 | 4096 | 1024 | 10.58 | 5 | 0.47 | 942.72 | 144.76 | 1087.48 | 54.50 | 66.58 | 66.81 | 6.73 | 6.78 | 6.79 | 2113.75 | 4687.47 | 4791.74 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c64_i4096_o1024 | main | 64 | 4096 | 1024 | 52.14 | 320 | 6.14 | 12958.58 | 3283.93 | 16242.51 | 197.14 | 789.79 | 800.08 | 17.06 | 17.95 | 19.33 | 9239.51 | 17164.77 | 17906.82 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i4096_o2048 | main | 1 | 4096 | 2048 | 38.30 | 5 | 0.13 | 260.29 | 146.91 | 407.20 | 55.00 | 67.22 | 67.42 | 6.75 | 6.79 | 6.80 | 7659.17 | 11647.12 | 11755.04 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c64_i4096_o2048 | main | 64 | 4096 | 2048 | 97.69 | 320 | 3.28 | 6916.20 | 3408.88 | 10325.08 | 162.14 | 570.75 | 582.03 | 16.56 | 17.06 | 18.07 | 17265.87 | 32918.48 | 34336.54 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i4096_o4096 | main | 1 | 4096 | 4096 | 79.98 | 5 | 0.06 | 124.66 | 147.18 | 271.85 | 55.41 | 66.70 | 66.85 | 6.77 | 6.80 | 6.80 | 15993.58 | 24761.64 | 25525.53 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c64_i4096_o4096 | main | 64 | 4096 | 4096 | 192.01 | 320 | 1.67 | 3518.79 | 3408.94 | 6927.73 | 163.81 | 576.41 | 599.00 | 16.44 | 16.79 | 16.98 | 33560.76 | 65515.45 | 68183.67 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i16384_o128 | main | 1 | 16384 | 128 | 3.74 | 5 | 1.34 | 11422.24 | 101.29 | 11523.54 | 244.24 | 433.34 | 440.01 | 6.72 | 6.80 | 6.80 | 746.96 | 1184.46 | 1242.36 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c32_i16384_o128 | main | 32 | 16384 | 128 | 29.67 | 160 | 5.39 | 43522.36 | 356.17 | 43878.53 | 996.18 | 3789.36 | 4532.55 | 78.86 | 150.73 | 226.61 | 5789.68 | 10661.97 | 11830.35 | 132932/143771 MiB (92.5%) | ❌ |
|
||||
| c1_i16384_o256 | main | 1 | 16384 | 256 | 3.82 | 5 | 1.31 | 11173.76 | 132.55 | 11306.32 | 87.44 | 122.35 | 122.83 | 6.73 | 6.76 | 6.76 | 763.51 | 1299.07 | 1372.37 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c32_i16384_o256 | main | 32 | 16384 | 256 | 12.46 | 160 | 12.84 | 103604.79 | 1782.54 | 105387.34 | 200.10 | 636.48 | 644.96 | 14.84 | 16.94 | 20.78 | 2229.45 | 3832.96 | 4262.99 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i16384_o512 | main | 1 | 16384 | 512 | 3.81 | 5 | 1.31 | 11209.67 | 132.98 | 11342.65 | 85.20 | 118.65 | 119.23 | 6.73 | 6.76 | 6.76 | 761.16 | 1293.05 | 1365.21 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c32_i16384_o512 | main | 32 | 16384 | 512 | 20.43 | 160 | 7.83 | 63203.51 | 1977.19 | 65180.70 | 190.69 | 640.86 | 739.06 | 14.04 | 15.70 | 18.53 | 3673.45 | 7038.21 | 7504.82 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i16384_o1024 | main | 1 | 16384 | 1024 | 10.85 | 5 | 0.46 | 3939.68 | 141.13 | 4080.81 | 96.12 | 132.08 | 134.06 | 6.75 | 6.80 | 6.81 | 2168.26 | 4767.81 | 4873.19 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c32_i16384_o1024 | main | 32 | 16384 | 1024 | 40.02 | 160 | 4.00 | 32262.84 | 2122.35 | 34385.19 | 226.08 | 789.94 | 950.40 | 13.38 | 14.23 | 22.02 | 7223.57 | 13034.14 | 13797.93 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i16384_o2048 | main | 1 | 16384 | 2048 | 38.62 | 5 | 0.13 | 1106.58 | 145.70 | 1252.28 | 92.60 | 132.65 | 135.00 | 6.77 | 6.82 | 6.82 | 7722.67 | 11757.03 | 11867.20 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c32_i16384_o2048 | main | 32 | 16384 | 2048 | 82.31 | 160 | 1.94 | 15686.47 | 2226.25 | 17912.73 | 187.67 | 628.02 | 806.24 | 12.88 | 13.39 | 14.85 | 14894.46 | 25291.54 | 26578.49 | 132932/143771 MiB (92.5%) | ✅ |
|
||||
| c1_i65536_o128 | main | 1 | 65536 | 128 | 5.98 | 5 | 0.84 | 29066.05 | 63.38 | 29129.43 | 697.62 | 1783.81 | 1903.90 | 6.67 | 6.75 | 6.75 | 1194.51 | 2550.76 | 2686.50 | 133260/143771 MiB (92.7%) | ✅ |
|
||||
| c8_i65536_o128 | main | 8 | 65536 | 128 | 39.65 | 40 | 1.01 | 37118.27 | 64.77 | 37183.05 | 3209.81 | 7445.34 | 8645.53 | 88.42 | 248.75 | 263.80 | 7821.82 | 14289.92 | 16484.17 | 133780/143771 MiB (93.1%) | ❌ |
|
||||
| c1_i65536_o256 | main | 1 | 65536 | 256 | 4.21 | 5 | 1.19 | 41330.67 | 120.56 | 41451.23 | 161.62 | 254.47 | 256.20 | 6.74 | 6.79 | 6.80 | 839.71 | 1306.35 | 1352.80 | 133780/143771 MiB (93.1%) | ✅ |
|
||||
| c8_i65536_o256 | main | 8 | 65536 | 256 | 8.64 | 40 | 4.63 | 170252.44 | 608.09 | 170860.53 | 356.04 | 749.11 | 864.77 | 9.14 | 10.21 | 10.30 | 1581.50 | 2865.09 | 2998.37 | 133780/143771 MiB (93.1%) | ✅ |
|
||||
| c1_i65536_o512 | main | 1 | 65536 | 512 | 4.23 | 5 | 1.18 | 41120.06 | 119.95 | 41240.01 | 184.79 | 288.97 | 290.20 | 6.54 | 6.74 | 6.75 | 843.98 | 1310.36 | 1356.99 | 133780/143771 MiB (93.1%) | ✅ |
|
||||
| c8_i65536_o512 | main | 8 | 65536 | 512 | 15.79 | 40 | 2.53 | 93186.96 | 705.69 | 93892.65 | 274.50 | 661.39 | 679.12 | 9.10 | 9.63 | 10.39 | 2813.45 | 5054.25 | 5200.71 | 133780/143771 MiB (93.1%) | ✅ |
|
||||
| c1_i65536_o1024 | main | 1 | 65536 | 1024 | 11.22 | 5 | 0.45 | 15489.07 | 136.44 | 15625.51 | 161.17 | 254.42 | 254.84 | 6.76 | 6.85 | 6.86 | 2242.90 | 4765.81 | 4841.93 | 133780/143771 MiB (93.1%) | ✅ |
|
||||
| c8_i65536_o1024 | main | 8 | 65536 | 1024 | 29.80 | 40 | 1.34 | 49382.35 | 734.78 | 50117.12 | 281.66 | 671.08 | 707.77 | 8.91 | 9.20 | 10.49 | 5146.09 | 8994.81 | 9476.99 | 133780/143771 MiB (93.1%) | ✅ |
|
||||
| c1_i131072_o128 | main | 1 | 131072 | 128 | 9.02 | 5 | 0.55 | 41083.10 | 42.03 | 41125.13 | 1332.75 | 3042.40 | 3072.01 | 6.34 | 6.70 | 6.72 | 1801.84 | 3782.08 | 3832.09 | 133780/143771 MiB (93.1%) | ⚠️ |
|
||||
| c4_i131072_o128 | main | 4 | 131072 | 128 | 24.36 | 20 | 0.82 | 60525.08 | 46.76 | 60571.83 | 2268.91 | 5665.45 | 7563.45 | 46.92 | 130.55 | 135.39 | 4786.17 | 9054.74 | 9263.16 | 134302/143771 MiB (93.4%) | ⚠️ |
|
||||
| c1_i131072_o256 | main | 1 | 131072 | 256 | 4.85 | 5 | 1.03 | 76336.33 | 104.48 | 76440.81 | 324.28 | 526.13 | 531.79 | 6.38 | 6.74 | 6.75 | 969.07 | 1348.49 | 1365.14 | 134302/143771 MiB (93.4%) | ✅ |
|
||||
| c4_i131072_o256 | main | 4 | 131072 | 256 | 8.32 | 20 | 2.40 | 177141.68 | 306.01 | 177447.69 | 542.06 | 826.11 | 873.84 | 7.11 | 8.42 | 8.69 | 1532.39 | 2573.00 | 2602.99 | 134302/143771 MiB (93.4%) | ✅ |
|
||||
| c1_i131072_o512 | main | 1 | 131072 | 512 | 4.87 | 5 | 1.03 | 76048.83 | 104.09 | 76152.92 | 327.91 | 528.58 | 529.52 | 6.38 | 6.74 | 6.75 | 972.74 | 1353.22 | 1365.66 | 134302/143771 MiB (93.4%) | ✅ |
|
||||
| c4_i131072_o512 | main | 4 | 131072 | 512 | 15.36 | 20 | 1.30 | 95969.05 | 382.40 | 96351.45 | 384.13 | 581.21 | 686.73 | 7.67 | 8.27 | 8.30 | 2674.27 | 3889.94 | 4365.16 | 134302/143771 MiB (93.4%) | ✅ |
|
||||
| c1_i262144_o128 | main | 1 | 262144 | 128 | 21.96 | 5 | 0.23 | 34775.53 | 17.26 | 34792.78 | 3949.96 | 7547.16 | 7933.13 | 5.31 | 6.25 | 6.26 | 4389.70 | 8216.37 | 8667.71 | 134304/143771 MiB (93.4%) | ⚠️ |
|
||||
| c2_i262144_o128 | main | 2 | 262144 | 128 | 17.30 | 10 | 0.58 | 80576.29 | 43.70 | 80619.99 | 1811.47 | 6405.29 | 6864.92 | 12.04 | 40.84 | 63.02 | 2911.30 | 7493.52 | 7624.83 | 134304/143771 MiB (93.4%) | ⚠️ |
|
||||
| c1_i262144_o256 | main | 1 | 262144 | 256 | 5.49 | 5 | 0.91 | 139043.08 | 92.31 | 139135.39 | 483.52 | 741.06 | 802.19 | 5.45 | 6.50 | 6.55 | 1096.55 | 1644.68 | 1664.47 | 134304/143771 MiB (93.4%) | ✅ |
|
||||
| c2_i262144_o256 | main | 2 | 262144 | 256 | 8.16 | 10 | 1.23 | 170921.30 | 171.18 | 171092.48 | 588.20 | 872.87 | 960.16 | 6.69 | 7.16 | 7.25 | 1528.35 | 2101.10 | 2191.39 | 134304/143771 MiB (93.4%) | ✅ |
|
||||
| c1_i524288_o128 | main | 1 | 524288 | 128 | 21.47 | 5 | 0.23 | 47769.55 | 17.65 | 47787.20 | 3868.11 | 13982.40 | 16622.77 | 5.18 | 6.10 | 6.17 | 4292.93 | 14596.89 | 17210.26 | 134304/143771 MiB (93.4%) | ⚠️ |
|
||||
| c2_i524288_o128 | main | 2 | 524288 | 128 | 5.67 | 10 | 1.76 | 292108.80 | 133.35 | 292242.15 | 645.47 | 1167.12 | 1232.75 | 5.64 | 6.90 | 6.94 | 1126.39 | 1682.93 | 1708.86 | 134306/143771 MiB (93.4%) | ✅ |
|
||||
| c1_i1048576_o128 | main | 1 | 1048576 | 128 | 52.22 | 5 | 0.10 | 29684.37 | 7.26 | 29691.63 | 10048.05 | 38265.96 | 45702.81 | 4.67 | 6.04 | 6.08 | 10442.12 | 38536.13 | 45912.47 | 134310/143771 MiB (93.4%) | ⚠️ |
|
||||
| c2_i1048576_o128 | main | 2 | 1048576 | 128 | 122.76 | 10 | 0.08 | 26303.08 | 6.16 | 26309.24 | 23067.17 | 59936.57 | 60383.98 | 15.73 | 56.69 | 68.50 | 24549.49 | 63918.14 | 66073.78 | 134312/143771 MiB (93.4%) | ⚠️ |
|
||||
|
||||
SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,57 @@
|
||||
mark input_len output_len concurrency num_prompts
|
||||
Y 1024 128 1 5
|
||||
Y 1024 128 128 640
|
||||
Y 1024 256 1 5
|
||||
Y 1024 256 128 640
|
||||
Y 1024 512 1 5
|
||||
Y 1024 512 128 640
|
||||
Y 1024 1024 1 5
|
||||
Y 1024 1024 128 640
|
||||
Y 1024 2048 1 5
|
||||
Y 1024 2048 128 640
|
||||
Y 1024 4096 1 5
|
||||
Y 1024 4096 128 640
|
||||
Y 4096 128 1 5
|
||||
Y 4096 128 64 320
|
||||
Y 4096 256 1 5
|
||||
Y 4096 256 64 320
|
||||
Y 4096 512 1 5
|
||||
Y 4096 512 64 320
|
||||
Y 4096 1024 1 5
|
||||
Y 4096 1024 64 320
|
||||
Y 4096 2048 1 5
|
||||
Y 4096 2048 64 320
|
||||
Y 4096 4096 1 5
|
||||
Y 4096 4096 64 320
|
||||
Y 16384 128 1 5
|
||||
Y 16384 128 32 160
|
||||
Y 16384 256 1 5
|
||||
Y 16384 256 32 160
|
||||
Y 16384 512 1 5
|
||||
Y 16384 512 32 160
|
||||
Y 16384 1024 1 5
|
||||
Y 16384 1024 32 160
|
||||
Y 16384 2048 1 5
|
||||
Y 16384 2048 32 160
|
||||
Y 65536 128 1 5
|
||||
Y 65536 128 8 40
|
||||
Y 65536 256 1 5
|
||||
Y 65536 256 8 40
|
||||
Y 65536 512 1 5
|
||||
Y 65536 512 8 40
|
||||
Y 65536 1024 1 5
|
||||
Y 65536 1024 8 40
|
||||
Y 131072 128 1 5
|
||||
Y 131072 128 4 20
|
||||
Y 131072 256 1 5
|
||||
Y 131072 256 4 20
|
||||
Y 131072 512 1 5
|
||||
Y 131072 512 4 20
|
||||
Y 262144 128 1 5
|
||||
Y 262144 128 2 10
|
||||
Y 262144 256 1 5
|
||||
Y 262144 256 2 10
|
||||
Y 524288 128 1 5
|
||||
Y 524288 128 2 10
|
||||
Y 1048576 128 1 5
|
||||
Y 1048576 128 2 10
|
||||
|
164
experiments/dsv4_h200_vllm_tp_dp_matrix/run_adaptive_concurrency.sh
Executable file
164
experiments/dsv4_h200_vllm_tp_dp_matrix/run_adaptive_concurrency.sh
Executable file
@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env bash
|
||||
# Find the Total-TPS saturation concurrency for each vLLM TP/DP/ISL/OSL shape.
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/adaptive_bench_lib.sh"
|
||||
|
||||
ENGINE="vllm"
|
||||
ENGINE_PORT="$VLLM_PORT"
|
||||
RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
|
||||
if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
|
||||
PYTHON="${VENV_CLIENT}/bin/python"
|
||||
else
|
||||
PYTHON="$(command -v python3)"
|
||||
fi
|
||||
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
|
||||
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
engine_is_healthy() {
|
||||
curl --fail --silent --show-error --max-time 5 \
|
||||
"http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
engine_stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
|
||||
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping vllm server pid=${pid} tp=${tp} dp=${dp}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
sleep 2
|
||||
}
|
||||
|
||||
engine_build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local -a args=(
|
||||
vllm serve "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$tp"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--no-enable-flashinfer-autotune
|
||||
--host 0.0.0.0
|
||||
--port "$ENGINE_PORT"
|
||||
)
|
||||
if (( dp > 1 )); then
|
||||
args+=(--data-parallel-size "$dp")
|
||||
fi
|
||||
printf '%q ' "${args[@]}"
|
||||
}
|
||||
|
||||
engine_start_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
|
||||
log "starting vllm server tp=${tp} dp=${dp}"
|
||||
bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||
if ! engine_is_healthy; then
|
||||
log "ERROR: vllm health check failed tp=${tp} dp=${dp}"
|
||||
return 1
|
||||
fi
|
||||
ACTIVE_ENGINE_SERVER_LOG="$(
|
||||
find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
|
||||
-name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \
|
||||
-printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
|
||||
)"
|
||||
log "vllm server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}"
|
||||
}
|
||||
|
||||
engine_detect_oom() {
|
||||
local detail_log="$1"
|
||||
local tp="$2"
|
||||
local dp="$3"
|
||||
local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
|
||||
local -a logs=("$detail_log" "$outer_log")
|
||||
if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
|
||||
logs+=("$ACTIVE_ENGINE_SERVER_LOG")
|
||||
fi
|
||||
grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null
|
||||
}
|
||||
|
||||
engine_run_bench() {
|
||||
local isl="$1"
|
||||
local osl="$2"
|
||||
local concurrency="$3"
|
||||
local num_prompts="$4"
|
||||
local output_file="$5"
|
||||
local warmup_requests
|
||||
warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
|
||||
local -a bench_args=(
|
||||
--backend vllm
|
||||
--host 127.0.0.1
|
||||
--port "$ENGINE_PORT"
|
||||
--dataset-name "$BENCH_DATASET_NAME"
|
||||
--random-input-len "$isl"
|
||||
--random-output-len "$osl"
|
||||
--random-range-ratio "$RANDOM_RANGE_RATIO"
|
||||
--num-prompts "$num_prompts"
|
||||
--max-concurrency "$concurrency"
|
||||
--request-rate 10000
|
||||
--warmup-requests "$warmup_requests"
|
||||
--output-file "$output_file"
|
||||
--output-details
|
||||
--disable-tqdm
|
||||
)
|
||||
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||
bench_args+=(--dataset-path "$DATASET_PATH")
|
||||
else
|
||||
bench_args+=(--tokenize-prompt)
|
||||
fi
|
||||
|
||||
if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then
|
||||
local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}")
|
||||
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||
volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro")
|
||||
fi
|
||||
docker run --rm \
|
||||
--network host \
|
||||
"${volume_args[@]}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
"$DOCKER_CLIENT_IMAGE" \
|
||||
python -m sglang.bench_serving "${bench_args[@]}"
|
||||
else
|
||||
"$PYTHON" -m sglang.bench_serving "${bench_args[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
export -f engine_run_bench
|
||||
export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT
|
||||
export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
|
||||
|
||||
adaptive_main "$@"
|
||||
167
experiments/dsv4_h200_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh
Executable file
167
experiments/dsv4_h200_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh
Executable file
@ -0,0 +1,167 @@
|
||||
#!/usr/bin/env bash
|
||||
# Find the Total-TPS saturation concurrency for each vLLM TP/DP/ISL/DSL shape.
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/adaptive_bench_lib.sh"
|
||||
|
||||
ENGINE="vllm"
|
||||
ENGINE_PORT="$VLLM_PORT"
|
||||
RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
|
||||
if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
|
||||
PYTHON="${VENV_CLIENT}/bin/python"
|
||||
else
|
||||
PYTHON="$(command -v python3)"
|
||||
fi
|
||||
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
|
||||
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
engine_is_healthy() {
|
||||
curl --fail --silent --show-error --max-time 5 \
|
||||
"http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
engine_stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
|
||||
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping vllm server pid=${pid} tp=${tp} dp=${dp}"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
ACTIVE_ENGINE_SERVER_LOG=""
|
||||
sleep 2
|
||||
}
|
||||
|
||||
engine_build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local -a args=(
|
||||
vllm serve "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$tp"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--no-enable-flashinfer-autotune
|
||||
--host 0.0.0.0
|
||||
--port "$ENGINE_PORT"
|
||||
)
|
||||
if (( dp > 1 )); then
|
||||
args+=(--data-parallel-size "$dp")
|
||||
fi
|
||||
printf '%q ' "${args[@]}"
|
||||
}
|
||||
|
||||
engine_start_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
|
||||
log "starting vllm server tp=${tp} dp=${dp}"
|
||||
bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||
if ! engine_is_healthy; then
|
||||
log "ERROR: vllm health check failed tp=${tp} dp=${dp}"
|
||||
return 1
|
||||
fi
|
||||
ACTIVE_ENGINE_SERVER_LOG="$(
|
||||
find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
|
||||
-name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \
|
||||
-printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
|
||||
)"
|
||||
log "vllm server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}"
|
||||
}
|
||||
|
||||
engine_detect_oom() {
|
||||
local detail_log="$1"
|
||||
local tp="$2"
|
||||
local dp="$3"
|
||||
local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
|
||||
local -a logs=("$detail_log" "$outer_log")
|
||||
if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
|
||||
logs+=("$ACTIVE_ENGINE_SERVER_LOG")
|
||||
fi
|
||||
grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null
|
||||
}
|
||||
|
||||
engine_run_bench() {
|
||||
local isl="$1"
|
||||
local dsl="$2"
|
||||
local concurrency="$3"
|
||||
local num_prompts="$4"
|
||||
local output_file="$5"
|
||||
local warmup_requests
|
||||
warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
|
||||
local -a bench_args=(
|
||||
--backend vllm
|
||||
--host 127.0.0.1
|
||||
--port "$ENGINE_PORT"
|
||||
--dataset-name "$BENCH_DATASET_NAME"
|
||||
--random-input-len "$isl"
|
||||
--random-output-len "$dsl"
|
||||
--random-range-ratio "$RANDOM_RANGE_RATIO"
|
||||
--num-prompts "$num_prompts"
|
||||
--max-concurrency "$concurrency"
|
||||
--request-rate 10000
|
||||
--warmup-requests "$warmup_requests"
|
||||
--output-file "$output_file"
|
||||
--output-details
|
||||
--disable-tqdm
|
||||
)
|
||||
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||
bench_args+=(--dataset-path "$DATASET_PATH")
|
||||
else
|
||||
bench_args+=(--tokenize-prompt)
|
||||
fi
|
||||
|
||||
if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then
|
||||
local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}")
|
||||
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||
volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro")
|
||||
fi
|
||||
docker run --rm \
|
||||
--network host \
|
||||
"${volume_args[@]}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
"$DOCKER_CLIENT_IMAGE" \
|
||||
python -m sglang.bench_serving "${bench_args[@]}"
|
||||
else
|
||||
"$PYTHON" -m sglang.bench_serving "${bench_args[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
export -f engine_run_bench
|
||||
export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT
|
||||
export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
|
||||
|
||||
export SEARCH_START_CONCURRENCY=16
|
||||
export SEARCH_ADDEND=16
|
||||
|
||||
adaptive_main "$@"
|
||||
548
experiments/dsv4_h200_vllm_tp_dp_matrix/run_bench.sh
Executable file
548
experiments/dsv4_h200_vllm_tp_dp_matrix/run_bench.sh
Executable file
@ -0,0 +1,548 @@
|
||||
#!/usr/bin/env bash
|
||||
# TP×DP matrix benchmark for DeepSeek-V4-Flash on vLLM (Docker).
|
||||
set -Eeuo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
# Export variables used inside functions that are called via bash -c subshells.
|
||||
export DATASET_PATH MODEL_PATH RESULT_BASE \
|
||||
DOCKER_IMAGE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT
|
||||
|
||||
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||
RESULT_BASE="${SCRIPT_DIR}/results"
|
||||
MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR}/matrix.json}"
|
||||
MATRIX_MODE="${MATRIX_MODE:-Y}"
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||
|
||||
if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
|
||||
PYTHON="${VENV_CLIENT}/bin/python"
|
||||
else
|
||||
PYTHON="$(command -v python3)"
|
||||
fi
|
||||
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
|
||||
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
log_dir_global="${RESULT_BASE}/${RUN_ID}/logs"
|
||||
mkdir -p "$log_dir_global"
|
||||
log_init "${log_dir_global}/orchestrator.log"
|
||||
|
||||
log "experiment=${EXPERIMENT_NAME} run_id=${RUN_ID} platform=${PLATFORM} hardware=${HARDWARE}"
|
||||
log "matrix_mode=${MATRIX_MODE} matrix_file=${MATRIX_FILE} dry_run=${DRY_RUN} grid_limit=${GRID_LIMIT}"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
is_server_healthy() {
|
||||
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${VLLM_PORT}/health" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
stop_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
|
||||
local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
|
||||
if [[ -f "$pid_file" ]]; then
|
||||
local pid
|
||||
pid="$(cat "$pid_file")"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
log "stopping vllm server pid=${pid} (tp=${tp}, dp=${dp})"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
rm -f "$pid_file"
|
||||
fi
|
||||
|
||||
# Fallback: remove any Docker container started by this experiment.
|
||||
docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
|
||||
|
||||
# Fallback: kill any vllm serve processes for this model.
|
||||
pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
|
||||
pkill -9 -f "vllm serve.*${MODEL_PATH}" 2>/dev/null || true
|
||||
sleep 2
|
||||
}
|
||||
|
||||
build_server_args() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
|
||||
local args=(
|
||||
"vllm serve" "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$tp"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--no-enable-flashinfer-autotune
|
||||
--host 0.0.0.0
|
||||
--port "$VLLM_PORT"
|
||||
)
|
||||
if [[ "$dp" -gt 1 ]]; then
|
||||
args+=(
|
||||
--data-parallel-size "$dp"
|
||||
)
|
||||
fi
|
||||
printf '%s ' "${args[@]}"
|
||||
}
|
||||
|
||||
start_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
|
||||
log "starting vllm server tp=${tp} dp=${dp}"
|
||||
bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" \
|
||||
>> "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1
|
||||
|
||||
if ! is_server_healthy; then
|
||||
log "error: vllm server tp=${tp} dp=${dp} failed health check on port ${VLLM_PORT}"
|
||||
return 1
|
||||
fi
|
||||
log "vllm server tp=${tp} dp=${dp} is healthy on port ${VLLM_PORT}"
|
||||
}
|
||||
|
||||
restart_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
log "restarting vllm server tp=${tp} dp=${dp} after non-OOM failure"
|
||||
stop_server "$tp" "$dp"
|
||||
sleep 10
|
||||
start_server "$tp" "$dp"
|
||||
}
|
||||
|
||||
run_bench_serving() {
|
||||
# Run sglang.bench_serving either natively or inside the SGLang Docker image.
|
||||
if [[ "${USE_DOCKER_CLIENT:-1}" == "1" ]]; then
|
||||
local vol_args=()
|
||||
vol_args+=("-v" "${MODEL_PATH}:${MODEL_PATH}:ro")
|
||||
if [[ -n "${DATASET_PATH:-}" && -f "${DATASET_PATH}" ]]; then
|
||||
vol_args+=("-v" "${DATASET_PATH}:${DATASET_PATH}:ro")
|
||||
fi
|
||||
vol_args+=("-v" "${RESULT_BASE}:${RESULT_BASE}")
|
||||
docker run --rm \
|
||||
--network host \
|
||||
"${vol_args[@]}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
-e HF_DATASETS_OFFLINE=1 \
|
||||
"${DOCKER_CLIENT_IMAGE}" \
|
||||
python -m sglang.bench_serving "$@"
|
||||
else
|
||||
"$PYTHON" -m sglang.bench_serving "$@"
|
||||
fi
|
||||
}
|
||||
export -f run_bench_serving
|
||||
|
||||
run_warmup() {
|
||||
local input_len="$1"
|
||||
local output_len="$2"
|
||||
|
||||
log "warming up (input=${input_len}, output=${output_len}, num=1)"
|
||||
bash -c '
|
||||
run_bench_serving \
|
||||
--backend vllm \
|
||||
--host 127.0.0.1 \
|
||||
--port "'"$VLLM_PORT"'" \
|
||||
--dataset-name random \
|
||||
--dataset-path "'"$DATASET_PATH"'" \
|
||||
--random-input-len "'"$input_len"'" \
|
||||
--random-output-len "'"$output_len"'" \
|
||||
--num-prompts 1 \
|
||||
--max-concurrency 1 \
|
||||
--request-rate 10000 \
|
||||
--output-file /dev/null \
|
||||
--output-details \
|
||||
>> "'"${log_dir_global}/warmup.log"'" 2>&1
|
||||
'
|
||||
log "warmup completed"
|
||||
}
|
||||
|
||||
scenario_already_completed() {
|
||||
local output_file="$1"
|
||||
local expected="$2"
|
||||
[[ -s "$output_file" ]] || return 1
|
||||
local completed
|
||||
completed="$("$PYTHON" -c "
|
||||
import json, sys
|
||||
path = sys.argv[1]
|
||||
try:
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
data = json.loads(line)
|
||||
print(data.get('completed', 0))
|
||||
break
|
||||
except Exception:
|
||||
print(0)
|
||||
" "$output_file")"
|
||||
[[ "${completed:-0}" -ge "$expected" ]]
|
||||
}
|
||||
|
||||
scenario_already_processed() {
|
||||
local result_root="$1"
|
||||
local scenario_name="$2"
|
||||
local json_path="${result_root}/results.json"
|
||||
[[ -f "$json_path" ]] || return 1
|
||||
"$PYTHON" -c "
|
||||
import json, sys
|
||||
path, name = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
with open(path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
for s in data.get('scenarios', []):
|
||||
if s.get('name') == name:
|
||||
if s.get('status') or s.get('metrics', {}).get('success', 0) > 0:
|
||||
sys.exit(0)
|
||||
except Exception:
|
||||
pass
|
||||
sys.exit(1)
|
||||
" "$json_path" "$scenario_name"
|
||||
}
|
||||
|
||||
detect_oom() {
|
||||
local detail_log="$1"
|
||||
local server_outer_log="$2"
|
||||
local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
|
||||
if grep -Eiq "$pattern" "$detail_log" "$server_outer_log" 2>/dev/null; then
|
||||
return 0
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
start_gpu_monitor() {
|
||||
local csv_path="$1"
|
||||
mkdir -p "$(dirname "$csv_path")"
|
||||
nvidia-smi \
|
||||
--query-gpu=timestamp,index,memory.used,memory.total,utilization.gpu \
|
||||
--format=csv \
|
||||
-l "$GPU_MEM_SAMPLE_INTERVAL_S" \
|
||||
> "$csv_path" 2>/dev/null &
|
||||
echo $!
|
||||
}
|
||||
|
||||
stop_gpu_monitor() {
|
||||
local pid="$1"
|
||||
if kill -0 "$pid" 2>/dev/null; then
|
||||
kill "$pid" 2>/dev/null || true
|
||||
sleep 1
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
fi
|
||||
}
|
||||
|
||||
append_scenario_record() {
|
||||
local result_root="$1"
|
||||
local json_path="$result_root/results.json"
|
||||
shift
|
||||
local scenario_json
|
||||
scenario_json="$("$PYTHON" -c "
|
||||
import json, sys
|
||||
pairs = [a.split('=', 1) for a in sys.argv[1:]]
|
||||
d = {}
|
||||
for k, v in pairs:
|
||||
try:
|
||||
d[k] = json.loads(v)
|
||||
except json.JSONDecodeError:
|
||||
d[k] = v
|
||||
print(json.dumps(d, ensure_ascii=False))
|
||||
" "$@")"
|
||||
PYTHON="$PYTHON" append_scenario_to_json "$json_path" "$scenario_json"
|
||||
}
|
||||
|
||||
record_skipped_csv() {
|
||||
local csv_path="$1"
|
||||
shift
|
||||
# Args: key=value
|
||||
local row
|
||||
row="$("$PYTHON" -c "
|
||||
import csv, json, sys, io
|
||||
pairs = [a.split('=', 1) for a in sys.argv[1:]]
|
||||
d = {}
|
||||
for k, v in pairs:
|
||||
try:
|
||||
d[k] = json.loads(v)
|
||||
except json.JSONDecodeError:
|
||||
d[k] = v
|
||||
buf = io.StringIO()
|
||||
writer = csv.DictWriter(buf, fieldnames=['engine','tp','dp','mark','isl','dsl','concurrency','status','reason','detail_log'], extrasaction='ignore')
|
||||
writer.writerow(d)
|
||||
print(buf.getvalue().strip())
|
||||
" "$@")"
|
||||
echo "$row" >> "$csv_path"
|
||||
}
|
||||
|
||||
skip_remaining_scenarios() {
|
||||
local result_root="$1"
|
||||
local scenario_tsv="$2"
|
||||
local start_index="$3"
|
||||
local status="$4"
|
||||
local reason="$5"
|
||||
local tp="$6"
|
||||
local dp="$7"
|
||||
local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
|
||||
|
||||
local i=0
|
||||
tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl dsl conc num; do
|
||||
if (( i < start_index )); then
|
||||
i=$((i + 1))
|
||||
continue
|
||||
fi
|
||||
i=$((i + 1))
|
||||
local sname="c${conc}_i${isl}_o${dsl}"
|
||||
if scenario_already_processed "$result_root" "$sname"; then
|
||||
continue
|
||||
fi
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"${status}\"" \
|
||||
"note=\"${reason}\""
|
||||
record_skipped_csv "$skipped_csv" \
|
||||
"engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=${status}" "reason=${reason}"
|
||||
done
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Per-configuration runner
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
run_parallel_config() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local config_label="tp${tp}_dp${dp}"
|
||||
local result_root="${RESULT_BASE}/${RUN_ID}/${config_label}"
|
||||
local raw_dir="${result_root}/raw_outputs"
|
||||
local gpu_log_dir="${result_root}/gpu_logs"
|
||||
local phase_log_dir="${result_root}/logs"
|
||||
mkdir -p "$raw_dir" "$gpu_log_dir" "$phase_log_dir"
|
||||
|
||||
log "===== ${config_label} START ====="
|
||||
|
||||
# Generate scenario list for this config.
|
||||
local scenario_tsv="${result_root}/scenarios.tsv"
|
||||
"$PYTHON" "${SCRIPT_DIR}/generate_scenarios.py" \
|
||||
--matrix "$MATRIX_FILE" \
|
||||
--mode "$MATRIX_MODE" \
|
||||
> "$scenario_tsv"
|
||||
local total_scenarios
|
||||
total_scenarios="$(tail -n +2 "$scenario_tsv" | wc -l)"
|
||||
log "generated ${total_scenarios} scenarios for ${config_label}"
|
||||
|
||||
# Write metadata.
|
||||
ensure_result_root "$result_root"
|
||||
write_metadata_json \
|
||||
"${result_root}/results.json" \
|
||||
"${EXPERIMENT_NAME}_${config_label}" \
|
||||
"$RUN_ID" \
|
||||
"$MODEL_PATH" \
|
||||
"vllm" \
|
||||
"vllm" \
|
||||
"$HARDWARE" \
|
||||
"$ACCELERATOR" \
|
||||
"$CHIP" \
|
||||
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||
"$DOCKER_IMAGE" \
|
||||
"H200 vLLM TP×DP matrix for DeepSeek-V4-Flash"
|
||||
|
||||
local server_args_str
|
||||
server_args_str="$(build_server_args "$tp" "$dp")"
|
||||
jq --arg tp "$tp" --arg dp "$dp" --arg cuda "$CUDA_VISIBLE_DEVICES" --arg args "$server_args_str" \
|
||||
'.config = {
|
||||
"tp": ($tp | tonumber),
|
||||
"dp": ($dp | tonumber),
|
||||
"cuda_visible_devices": $cuda,
|
||||
"backend": "vllm",
|
||||
"server_start_script": "experiments/'${EXPERIMENT_NAME}'/start_vllm_dp.sh",
|
||||
"server_args": $args
|
||||
}' "${result_root}/results.json" > "${result_root}/results.json.tmp" && \
|
||||
mv "${result_root}/results.json.tmp" "${result_root}/results.json"
|
||||
|
||||
if [[ "$DRY_RUN" == "1" ]]; then
|
||||
log "DRY_RUN: would start server with args: ${server_args_str}"
|
||||
local line
|
||||
tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl dsl conc num; do
|
||||
log "DRY_RUN: ${config_label} scenario mark=${mark} c=${conc} i=${isl} o=${dsl} n=${num}"
|
||||
done
|
||||
log "===== ${config_label} DONE (dry run) ====="
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Initialize skipped_after_oom.csv for this run.
|
||||
local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
|
||||
if [[ ! -f "$skipped_csv" ]]; then
|
||||
echo "engine,tp,dp,mark,isl,dsl,concurrency,status,reason,detail_log" > "$skipped_csv"
|
||||
fi
|
||||
|
||||
# Start server once for this TP×DP config.
|
||||
if ! start_server "$tp" "$dp"; then
|
||||
log "ERROR: ${config_label} failed to start; skipping all scenarios"
|
||||
skip_remaining_scenarios "$result_root" "$scenario_tsv" 0 "SKIPPED_SERVICE_START_FAILED" "service failed to start" "$tp" "$dp"
|
||||
log "===== ${config_label} DONE ====="
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Warmup with a small prompt before the first scenario.
|
||||
run_warmup 1024 128 || true
|
||||
|
||||
# Read scenarios into an array so we can skip remaining entries on failure.
|
||||
local -a scenarios=()
|
||||
while IFS= read -r line; do
|
||||
scenarios+=("$line")
|
||||
done < <(tail -n +2 "$scenario_tsv")
|
||||
|
||||
local i mark isl dsl conc num
|
||||
local output_file detail_log gpu_csv sname bench_rc
|
||||
for (( i = 0; i < ${#scenarios[@]}; i++ )); do
|
||||
IFS=$'\t' read -r mark isl dsl conc num <<< "${scenarios[$i]}"
|
||||
|
||||
if [[ "$GRID_LIMIT" -gt 0 && "$i" -ge "$GRID_LIMIT" ]]; then
|
||||
log "GRID_LIMIT=${GRID_LIMIT} reached; skipping remaining scenarios"
|
||||
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$i" "SKIPPED_GRID_LIMIT" "GRID_LIMIT reached" "$tp" "$dp"
|
||||
break
|
||||
fi
|
||||
|
||||
sname="c${conc}_i${isl}_o${dsl}"
|
||||
output_file="${raw_dir}/vllm_main_${conc}_${isl}_${dsl}.jsonl"
|
||||
detail_log="${phase_log_dir}/vllm_${config_label}_${sname}.log"
|
||||
gpu_csv="${gpu_log_dir}/gpu_mem_${conc}_${isl}_${dsl}.csv"
|
||||
|
||||
if scenario_already_completed "$output_file" "$num" || scenario_already_processed "$result_root" "$sname"; then
|
||||
log "skipping already-processed ${config_label} scenario: ${sname}"
|
||||
continue
|
||||
fi
|
||||
|
||||
log "running ${config_label} scenario: mark=${mark} c=${conc} i=${isl} o=${dsl} n=${num}"
|
||||
|
||||
local gpu_pid
|
||||
gpu_pid="$(start_gpu_monitor "$gpu_csv")"
|
||||
|
||||
bench_rc=0
|
||||
timeout "$SCENARIO_TIMEOUT_S" bash -c '
|
||||
run_bench_serving \
|
||||
--backend vllm \
|
||||
--host 127.0.0.1 \
|
||||
--port "'"$VLLM_PORT"'" \
|
||||
--dataset-name random \
|
||||
--dataset-path "'"$DATASET_PATH"'" \
|
||||
--random-input-len "'"$isl"'" \
|
||||
--random-output-len "'"$dsl"'" \
|
||||
--num-prompts "'"$num"'" \
|
||||
--max-concurrency "'"$conc"'" \
|
||||
--request-rate 10000 \
|
||||
--output-file "'"$output_file"'" \
|
||||
--output-details \
|
||||
> "'"$detail_log"'" 2>&1
|
||||
' || bench_rc=$?
|
||||
|
||||
stop_gpu_monitor "$gpu_pid"
|
||||
|
||||
if [[ "$bench_rc" -eq 0 ]]; then
|
||||
log "finished ${config_label} scenario: output=${output_file}"
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"completed\"" \
|
||||
"note=\"benchmark finished successfully\""
|
||||
continue
|
||||
fi
|
||||
|
||||
# Failure handling.
|
||||
if detect_oom "$detail_log" "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log"; then
|
||||
log "ERROR: ${config_label} scenario ${sname} triggered OOM; stopping config"
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"OOM\"" \
|
||||
"note=\"detected CUDA out-of-memory\""
|
||||
record_skipped_csv "$skipped_csv" \
|
||||
"engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=OOM" "reason=detected CUDA out-of-memory" "detail_log=${detail_log}"
|
||||
stop_server "$tp" "$dp"
|
||||
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_AFTER_OOM" "previous case OOM" "$tp" "$dp"
|
||||
break
|
||||
fi
|
||||
|
||||
log "ERROR: ${config_label} scenario ${sname} failed (rc=${bench_rc}); see ${detail_log}"
|
||||
if [[ "$mark" == "P" ]]; then
|
||||
log "optional (P) scenario failed; recording as skipped and continuing"
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"skipped_optional\"" \
|
||||
"note=\"optional scenario failed (rc=${bench_rc})\""
|
||||
record_skipped_csv "$skipped_csv" \
|
||||
"engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=skipped_optional" "reason=optional scenario failed (rc=${bench_rc})" "detail_log=${detail_log}"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Mandatory scenario failed but not OOM: try to restart the server.
|
||||
if restart_server "$tp" "$dp"; then
|
||||
run_warmup 1024 128 || true
|
||||
log "resuming ${config_label} after server restart"
|
||||
continue
|
||||
fi
|
||||
|
||||
log "ERROR: ${config_label} server restart failed; skipping remaining scenarios"
|
||||
append_scenario_record "$result_root" \
|
||||
"name=${sname}" \
|
||||
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||
"status=\"FAILED\"" \
|
||||
"note=\"scenario failed and server restart failed (rc=${bench_rc})\""
|
||||
record_skipped_csv "$skipped_csv" \
|
||||
"engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=FAILED" "reason=scenario failed and server restart failed" "detail_log=${detail_log}"
|
||||
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_RESTART_FAILED" "server restart failed" "$tp" "$dp"
|
||||
break
|
||||
done
|
||||
|
||||
stop_server "$tp" "$dp"
|
||||
|
||||
# Parse results.
|
||||
log "parsing ${config_label} results"
|
||||
"$PYTHON" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend vllm \
|
||||
>> "${phase_log_dir}/parse.log" 2>&1 || {
|
||||
log "WARNING: parser failed for ${config_label}; see ${phase_log_dir}/parse.log"
|
||||
}
|
||||
|
||||
log "===== ${config_label} DONE ====="
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Cleanup any leftovers.
|
||||
for cfg in "${PARALLEL_CONFIGS[@]}"; do
|
||||
read -r tp dp <<< "$cfg"
|
||||
stop_server "$tp" "$dp"
|
||||
done
|
||||
|
||||
# Run each parallel configuration.
|
||||
for cfg in "${PARALLEL_CONFIGS[@]}"; do
|
||||
read -r tp dp <<< "$cfg"
|
||||
run_parallel_config "$tp" "$dp"
|
||||
done
|
||||
|
||||
# Generate cross-configuration comparison.
|
||||
log "generating comparison report"
|
||||
"$PYTHON" "${SCRIPT_DIR}/compare.py" \
|
||||
--run-root "${RESULT_BASE}/${RUN_ID}" \
|
||||
--output "${RESULT_BASE}/${RUN_ID}/comparison.md" \
|
||||
>> "${log_dir_global}/compare.log" 2>&1 || {
|
||||
log "WARNING: comparison script failed; see ${log_dir_global}/compare.log"
|
||||
}
|
||||
|
||||
log "all results saved to ${RESULT_BASE}/${RUN_ID}"
|
||||
18
experiments/dsv4_h200_vllm_tp_dp_matrix/smoke_matrix.json
Normal file
18
experiments/dsv4_h200_vllm_tp_dp_matrix/smoke_matrix.json
Normal file
@ -0,0 +1,18 @@
|
||||
{
|
||||
"comment": "Small smoke matrix for quickly verifying TP×DP server startup and bench path.",
|
||||
"mode": "Y",
|
||||
"matrix": {
|
||||
"1024": {
|
||||
"128": "Y",
|
||||
"256": "Y"
|
||||
},
|
||||
"4096": {
|
||||
"128": "Y",
|
||||
"256": "Y"
|
||||
}
|
||||
},
|
||||
"concurrency": {
|
||||
"1024": { "low": 1, "high": 8 },
|
||||
"4096": { "low": 1, "high": 4 }
|
||||
}
|
||||
}
|
||||
103
experiments/dsv4_h200_vllm_tp_dp_matrix/start_vllm_docker.sh
Executable file
103
experiments/dsv4_h200_vllm_tp_dp_matrix/start_vllm_docker.sh
Executable file
@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start vLLM server in Docker for a given TPxDP configuration.
|
||||
# Usage: start_vllm_docker.sh <TP> <DP>
|
||||
#
|
||||
# Uses the vllm/vllm-openai image and the same argument set as the
|
||||
# bare-metal start script. The container is removed automatically on stop.
|
||||
set -e
|
||||
|
||||
TP="${1}"
|
||||
DP="${2}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||
|
||||
IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
|
||||
PORT="${VLLM_PORT:-30030}"
|
||||
NAME="${EXPERIMENT}_vllm_tp${TP}_dp${DP}"
|
||||
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${TP}_dp${DP}.pid"
|
||||
|
||||
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_vllm_docker_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
# Clean up any stale container with the same name.
|
||||
docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||
|
||||
SERVER_ARGS=(
|
||||
serve "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$TP"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--no-enable-flashinfer-autotune
|
||||
--host 0.0.0.0
|
||||
--port "$PORT"
|
||||
)
|
||||
|
||||
if [[ "$DP" -gt 1 ]]; then
|
||||
SERVER_ARGS+=(
|
||||
--data-parallel-size "$DP"
|
||||
)
|
||||
fi
|
||||
|
||||
SERVER_ARGS_STR="${SERVER_ARGS[*]}"
|
||||
|
||||
echo "=== Starting vLLM server in Docker (TP=${TP}, DP=${DP}) ==="
|
||||
echo "Image: $IMAGE"
|
||||
echo "Model: $MODEL_PATH"
|
||||
echo "Container name: $NAME"
|
||||
echo "Host port: $PORT"
|
||||
echo "Command: vllm ${SERVER_ARGS_STR}"
|
||||
echo "Log: $LOG"
|
||||
|
||||
# Run docker in the foreground so that killing the host process stops the
|
||||
# container (the --rm flag ensures cleanup). nohup lets us background it and
|
||||
# capture the host PID in the same way as the bare-metal start script.
|
||||
nohup docker run --rm \
|
||||
--name "$NAME" \
|
||||
--gpus all \
|
||||
--ipc host \
|
||||
--shm-size 16g \
|
||||
--ulimit memlock=-1 \
|
||||
--entrypoint vllm \
|
||||
-p "${PORT}:${PORT}" \
|
||||
-v "${MODEL_PATH}:${MODEL_PATH}:ro" \
|
||||
-v "${RUNTIME_BASE}/tmp:/tmp" \
|
||||
-e CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}" \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
"$IMAGE" \
|
||||
"${SERVER_ARGS[@]}" \
|
||||
> "$LOG" 2>&1 &
|
||||
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
echo "Waiting for health on port ${PORT}..."
|
||||
|
||||
for i in $(seq 1 240); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
|
||||
echo "vLLM server is ready at http://127.0.0.1:${PORT}"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
if ! kill -0 $PID 2>/dev/null; then
|
||||
echo "ERROR: Docker vLLM server exited early"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/240)"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: Docker vLLM server not healthy after 240 retries"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
87
experiments/dsv4_h200_vllm_tp_dp_matrix/start_vllm_dp.sh
Executable file
87
experiments/dsv4_h200_vllm_tp_dp_matrix/start_vllm_dp.sh
Executable file
@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start vLLM server for a given TP×DP configuration.
|
||||
# Usage: start_vllm_dp.sh <TP> <DP>
|
||||
#
|
||||
# By default this delegates to the Docker start script because the experiment
|
||||
# is intended to run vLLM inside a container. Set USE_DOCKER=0 to use the
|
||||
# local VENV_VLLM environment instead.
|
||||
set -e
|
||||
|
||||
TP="${1}"
|
||||
DP="${2}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${SCRIPT_DIR}/config.env"
|
||||
|
||||
if [[ "${USE_DOCKER:-1}" == "1" ]]; then
|
||||
exec "${SCRIPT_DIR}/start_vllm_docker.sh" "$@"
|
||||
fi
|
||||
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||
|
||||
VENV="${VENV_VLLM}"
|
||||
export PATH="$VENV/bin:$PATH"
|
||||
export PYTHONUNBUFFERED=1
|
||||
export TMPDIR="${RUNTIME_BASE}/tmp"
|
||||
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}"
|
||||
|
||||
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_vllm_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
|
||||
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${TP}_dp${DP}.pid"
|
||||
|
||||
rm -f "$PID_FILE"
|
||||
|
||||
SERVER_ARGS=(
|
||||
vllm serve "$MODEL_PATH"
|
||||
--trust-remote-code
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$TP"
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--no-enable-flashinfer-autotune
|
||||
--host 0.0.0.0
|
||||
--port "$VLLM_PORT"
|
||||
)
|
||||
|
||||
if [[ "$DP" -gt 1 ]]; then
|
||||
SERVER_ARGS+=(
|
||||
--data-parallel-size "$DP"
|
||||
)
|
||||
fi
|
||||
|
||||
SERVER_ARGS_STR="${SERVER_ARGS[*]}"
|
||||
|
||||
echo "=== Starting vLLM server (TP=${TP}, DP=${DP}) ==="
|
||||
echo "Model: $MODEL_PATH"
|
||||
echo "Port: $VLLM_PORT"
|
||||
echo "Command: $SERVER_ARGS_STR"
|
||||
echo "Log: $LOG"
|
||||
|
||||
nohup "${SERVER_ARGS[@]}" > "$LOG" 2>&1 &
|
||||
|
||||
PID=$!
|
||||
echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
echo "Waiting for health on port ${VLLM_PORT}..."
|
||||
|
||||
for i in $(seq 1 240); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${VLLM_PORT}/health" >/dev/null 2>&1; then
|
||||
echo "vLLM server is ready at http://127.0.0.1:${VLLM_PORT}"
|
||||
echo "Log: $LOG"
|
||||
exit 0
|
||||
fi
|
||||
if ! kill -0 $PID 2>/dev/null; then
|
||||
echo "ERROR: vLLM server exited early"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/240)"
|
||||
sleep 5
|
||||
done
|
||||
|
||||
echo "ERROR: vLLM server not healthy after 240 retries"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
49
experiments/dsv4_h20_sglang_tp_dp_matrix/adaptive_config.env
Normal file
49
experiments/dsv4_h20_sglang_tp_dp_matrix/adaptive_config.env
Normal file
@ -0,0 +1,49 @@
|
||||
# Adaptive concurrency search settings.
|
||||
#
|
||||
# For each fixed (TP, DP, ISL, OSL), probe:
|
||||
# C = start, start * multiplier, ... up to max
|
||||
# and stop after Total TPS has less than TPS_MIN_GAIN_PCT meaningful growth for
|
||||
# PLATEAU_PATIENCE consecutive points.
|
||||
|
||||
SEARCH_START_CONCURRENCY="${SEARCH_START_CONCURRENCY:-1}"
|
||||
SEARCH_MAX_CONCURRENCY="${SEARCH_MAX_CONCURRENCY:-512}"
|
||||
SEARCH_MULTIPLIER="${SEARCH_MULTIPLIER:-2}"
|
||||
NUM_PROMPTS_MULTIPLIER="${NUM_PROMPTS_MULTIPLIER:-5}"
|
||||
|
||||
# A gain below 2% is treated as throughput saturation. Two consecutive
|
||||
# low-gain points prevent one noisy measurement from stopping the search.
|
||||
TPS_MIN_GAIN_PCT="${TPS_MIN_GAIN_PCT:-2.0}"
|
||||
PLATEAU_PATIENCE="${PLATEAU_PATIENCE:-2}"
|
||||
|
||||
# Keep the same random workload semantics as the fixed matrix baseline.
|
||||
# DATASET_PATH must contain at least SEARCH_MAX_CONCURRENCY times
|
||||
# NUM_PROMPTS_MULTIPLIER valid two-turn conversations. Set this explicitly to
|
||||
# random-ids to use generated token IDs without a ShareGPT seed dataset.
|
||||
BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}"
|
||||
# SGLang interprets 0.0 as Uniform[1, requested_len]. Use 1.0 for fixed
|
||||
# ISL/OSL points; lower values intentionally benchmark a length distribution.
|
||||
RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1.0}"
|
||||
# Before each measured point, warm up with the same concurrency so lazy kernel
|
||||
# compilation and CUDA graph capture are excluded from TTFT/TPS. 0 means no
|
||||
# cap; set a positive cap only when very high-concurrency warmup is impractical.
|
||||
BENCH_WARMUP_MAX_REQUESTS="${BENCH_WARMUP_MAX_REQUESTS:-0}"
|
||||
|
||||
# Reject a point if the completed request count or actual token lengths do not
|
||||
# match the requested workload.
|
||||
INPUT_LENGTH_TOLERANCE_PCT="${INPUT_LENGTH_TOLERANCE_PCT:-5.0}"
|
||||
OUTPUT_LENGTH_TOLERANCE_PCT="${OUTPUT_LENGTH_TOLERANCE_PCT:-10.0}"
|
||||
|
||||
MAX_POINT_RETRIES="${MAX_POINT_RETRIES:-1}"
|
||||
SERVER_RESTART_COOLDOWN_S="${SERVER_RESTART_COOLDOWN_S:-10}"
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||
|
||||
# Optional space-separated filters, useful for smoke tests:
|
||||
# TP_LIST="8" ISL_LIST="1024" OSL_LIST="128"
|
||||
TP_LIST="${TP_LIST:-}"
|
||||
ISL_LIST="${ISL_LIST:-}"
|
||||
OSL_LIST="${OSL_LIST:-}"
|
||||
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
# Counts ISL/OSL shapes per TP/DP config, not individual concurrency probes.
|
||||
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||
162
experiments/dsv4_h20_sglang_tp_dp_matrix/compare.py
Normal file
162
experiments/dsv4_h20_sglang_tp_dp_matrix/compare.py
Normal file
@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cross TP×DP configuration comparison for dsv4_h20_sglang_tp_dp_matrix.
|
||||
|
||||
Usage:
|
||||
python3 compare.py --run-root results/<run_id> [--output comparison.md]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def load_result(result_root: Path) -> dict:
|
||||
path = result_root / "results.json"
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def slo_status(ttft_p95_ms: float, tpot_mean_ms: float,
|
||||
ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str:
|
||||
ttft_ok = ttft_p95_ms < ttft_limit_ms
|
||||
tpot_ok = tpot_mean_ms < tpot_limit_ms
|
||||
if ttft_ok and tpot_ok:
|
||||
return "PASS"
|
||||
if ttft_ok or tpot_ok:
|
||||
return "PARTIAL"
|
||||
return "FAIL"
|
||||
|
||||
|
||||
def gpu_memory_str(gpu: dict | None) -> str:
|
||||
if not gpu:
|
||||
return "-"
|
||||
peak = gpu.get("peak_used_mb", 0)
|
||||
total = gpu.get("memory_total_mb", 0)
|
||||
if total:
|
||||
return f"{peak:.0f}/{total:.0f} ({100*peak/total:.1f}%)"
|
||||
return f"{peak:.0f}"
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--run-root", type=Path, required=True)
|
||||
parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md"))
|
||||
parser.add_argument("--ttft-limit", type=float, default=3000.0)
|
||||
parser.add_argument("--tpot-limit", type=float, default=50.0)
|
||||
args = parser.parse_args()
|
||||
|
||||
# Discover configurations: tp*_dp* directories.
|
||||
configs = []
|
||||
for subdir in sorted(args.run_root.iterdir()):
|
||||
if not subdir.is_dir():
|
||||
continue
|
||||
name = subdir.name
|
||||
if not (name.startswith("tp") and "_dp" in name):
|
||||
continue
|
||||
results_json = subdir / "results.json"
|
||||
if not results_json.exists():
|
||||
continue
|
||||
configs.append((name, load_result(subdir)))
|
||||
|
||||
if not configs:
|
||||
print(f"No tp*_dp* results found under {args.run_root}")
|
||||
return
|
||||
|
||||
model = configs[0][1].get("metadata", {}).get("model", "unknown")
|
||||
hardware = configs[0][1].get("metadata", {}).get("hardware", "unknown")
|
||||
|
||||
# Group by scenario name.
|
||||
by_scenario: dict[str, dict[str, dict]] = defaultdict(dict)
|
||||
skipped: dict[str, dict[str, str]] = defaultdict(dict)
|
||||
for label, data in configs:
|
||||
for s in data.get("scenarios", []):
|
||||
key = s["name"]
|
||||
if s.get("status") == "skipped_oom":
|
||||
skipped[key][label] = s.get("note", "skipped")
|
||||
else:
|
||||
by_scenario[key][label] = s
|
||||
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
f.write(f"# SGLang TP×DP matrix comparison ({hardware})\n\n")
|
||||
f.write("## Summary\n\n")
|
||||
f.write(f"- Model: `{model}`\n")
|
||||
f.write(f"- Hardware: {hardware}\n")
|
||||
f.write("- Backend: SGLang (Docker)\n")
|
||||
f.write("- Benchmark client: `sglang.bench_serving`\n")
|
||||
f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n")
|
||||
|
||||
# Configuration overview.
|
||||
f.write("### Configurations\n\n")
|
||||
f.write("| Config | TP | DP | GPUs/replica | Notes |\n")
|
||||
f.write("|---|---:|---:|---:|---|\n")
|
||||
for label, data in configs:
|
||||
cfg = data.get("config", {})
|
||||
tp = cfg.get("tp", "?")
|
||||
dp = cfg.get("dp", "?")
|
||||
f.write(f"| {label} | {tp} | {dp} | {tp} | server args recorded per ISL in results.json |\n")
|
||||
f.write("\n")
|
||||
|
||||
# Side-by-side table.
|
||||
f.write("## Side-by-side results\n\n")
|
||||
headers = [
|
||||
"Scenario", "ISL", "OSL", "Config", "Conc", "Req/s", "OutTok/s",
|
||||
"TTFT P95(ms)", "TTFT P99(ms)", "TPOT Mean(ms)", "TPOT P95(ms)",
|
||||
"TPOT P99(ms)", "E2E P99(ms)", "Peak GPU mem", "SLO"
|
||||
]
|
||||
f.write("| " + " | ".join(headers) + " |\n")
|
||||
f.write("|" + "|".join(["---"] * len(headers)) + "|\n")
|
||||
|
||||
for scenario_name in sorted(by_scenario.keys(), key=lambda x: tuple(map(int, re.findall(r"\d+", x)))):
|
||||
_, isl, osl = re.findall(r"\d+", scenario_name)
|
||||
for label, data in configs:
|
||||
s = by_scenario[scenario_name].get(label)
|
||||
if s is None:
|
||||
if scenario_name in skipped and label in skipped[scenario_name]:
|
||||
note = skipped[scenario_name][label]
|
||||
f.write(f"| {scenario_name} | {isl} | {osl} | {label} | - | - | - | - | - | - | - | - | - | - | {note} |\n")
|
||||
continue
|
||||
cfg = s["config"]
|
||||
m = s["metrics"]
|
||||
status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
|
||||
gpu = m.get("gpu_memory")
|
||||
f.write(
|
||||
f"| {scenario_name} | {isl} | {osl} | {label} | {cfg['concurrency']} | "
|
||||
f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||
f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||
f"{m['e2e_ms']['p99']:.2f} | {gpu_memory_str(gpu)} | {status} |\n"
|
||||
)
|
||||
|
||||
# Best throughput per ISL/OSL.
|
||||
f.write("\n## Best throughput per (ISL, OSL)\n\n")
|
||||
f.write("| ISL | OSL | Best Config | Concurrency | OutTok/s | TTFT P95(ms) | TPOT Mean(ms) | SLO |\n")
|
||||
f.write("|---:|---:|---|---:|---:|---:|---:|---:|\n")
|
||||
best_by_shape: dict[tuple[int, int], tuple[float, str, dict]] = {}
|
||||
for scenario_name, backends in by_scenario.items():
|
||||
_, isl, osl = re.findall(r"\d+", scenario_name)
|
||||
isl_i, osl_i = int(isl), int(osl)
|
||||
for label, s in backends.items():
|
||||
m = s["metrics"]
|
||||
out_tok = m["output_token_throughput"]
|
||||
if (isl_i, osl_i) not in best_by_shape or out_tok > best_by_shape[(isl_i, osl_i)][0]:
|
||||
best_by_shape[(isl_i, osl_i)] = (out_tok, label, s)
|
||||
for (isl_i, osl_i), (out_tok, label, s) in sorted(best_by_shape.items()):
|
||||
m = s["metrics"]
|
||||
status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
|
||||
f.write(
|
||||
f"| {isl_i} | {osl_i} | {label} | {s['config']['concurrency']} | "
|
||||
f"{out_tok:.2f} | {m['ttft_ms']['p95']:.2f} | {m['tpot_ms']['mean']:.2f} | {status} |\n"
|
||||
)
|
||||
|
||||
f.write("\n## Notes\n\n")
|
||||
f.write("- SLO check uses TTFT P95 and TPOT mean.\n")
|
||||
f.write("- A PARTIAL indicates one of the two metrics is out of target; FAIL indicates both are out.\n")
|
||||
f.write("- `Peak GPU mem` shows peak used / total MB and utilization percentage.\n")
|
||||
f.write("- Optional (P) combinations that failed are marked as skipped/OOM and do not break the run.\n")
|
||||
|
||||
print(f"Wrote comparison to {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
70
experiments/dsv4_h20_sglang_tp_dp_matrix/config.env
Normal file
70
experiments/dsv4_h20_sglang_tp_dp_matrix/config.env
Normal file
@ -0,0 +1,70 @@
|
||||
# TP×DP matrix experiment for DeepSeek-V4-Flash on H20 (8 GPUs) using SGLang.
|
||||
# Tests SGLang with three parallel configurations:
|
||||
# TP=2, DP=4 -> 2 GPUs per replica, 4 replicas
|
||||
# TP=4, DP=2 -> 4 GPUs per replica, 2 replicas
|
||||
# TP=8, DP=1 -> 8 GPUs, no data parallelism
|
||||
|
||||
EXPERIMENT="dsv4_h20_sglang_tp_dp_matrix"
|
||||
MODEL_NAME="DeepSeek-V4-Flash"
|
||||
MODEL_PATH="/data1/hf_models/DeepSeek-V4-Flash"
|
||||
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||
|
||||
SGLANG_PORT="${SGLANG_PORT:-30031}"
|
||||
|
||||
# Python interpreter for orchestration scripts (parse_backend.py, compare.py, etc.)
|
||||
# and the benchmark client. Defaults to the system python3 if the sglang venv
|
||||
# does not exist on the host.
|
||||
VENV_CLIENT="${VENV_CLIENT:-/data1/yy/sskj/envs/sglang}"
|
||||
|
||||
# Run the benchmark client natively (0) or inside Docker (1).
|
||||
USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-1}"
|
||||
|
||||
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
|
||||
|
||||
# Runtime working directory for logs, pid files, and tmp. Defaults to a local
|
||||
# directory under this experiment so the benchmark is self-contained.
|
||||
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||
|
||||
# Parallel configurations to test. Format: "TP DP"
|
||||
declare -a PARALLEL_CONFIGS=(
|
||||
"2 4"
|
||||
"4 2"
|
||||
"8 1"
|
||||
)
|
||||
|
||||
# SGLang server settings
|
||||
CONTEXT_LENGTH="${CONTEXT_LENGTH:-1048576}"
|
||||
MAX_RUNNING_REQUESTS="${MAX_RUNNING_REQUESTS:-128}"
|
||||
MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.88}"
|
||||
MOE_RUNNER_BACKEND="${MOE_RUNNER_BACKEND:-marlin}"
|
||||
|
||||
# Deployment switch. 0 = native sglang venv, 1 = Docker.
|
||||
USE_DOCKER="${USE_DOCKER:-1}"
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:latest}"
|
||||
|
||||
# Dataset used by sglang.bench_serving --dataset-name random.
|
||||
# The random sampler needs a ShareGPT-style JSON file locally; it falls back to
|
||||
# downloading from HuggingFace, which usually fails on offline H20 nodes.
|
||||
DATASET_PATH="${DATASET_PATH:-/data1/yy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json}"
|
||||
|
||||
# Matrix and concurrency rules are defined in matrix.json by default.
|
||||
MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR:-.}/matrix.json}"
|
||||
MATRIX_MODE="${MATRIX_MODE:-Y}"
|
||||
|
||||
# Sampling density for concurrency.
|
||||
# 0 = use the default heuristic in generate_scenarios.py (6-8 points).
|
||||
# 2 = only test the low and high endpoints.
|
||||
export CONCURRENCY_SAMPLES="${CONCURRENCY_SAMPLES:-2}"
|
||||
|
||||
# Per-scenario timeout to avoid hangs (seconds).
|
||||
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||
|
||||
# GPU memory sampling interval (seconds).
|
||||
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||
|
||||
# Dry-run mode: if 1, only log the server args and scenario plan without starting
|
||||
# any server or sending requests.
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
|
||||
# Per-config scenario limit for quick smoke tests. 0 = run all generated scenarios.
|
||||
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
x
Reference in New Issue
Block a user