Add vLLM+MTP speculative decoding benchmark vs default on H200
- Experiment: experiments/dsv4_h200_vllm_mtp_vs_default/
- Use official DeepSeek-V4 recipe: --speculative-config '{method: mtp, num_speculative_tokens: 1}'
- Reuse default-side results from dsv4_h200_vllm_dspark_vs_default
- Add TTFT analysis for DSpark vs default trade-offs
This commit is contained in:
yy-fighting
parent
a56c995091
commit
7d2ed7f05d
@ -0,0 +1,85 @@
|
|||||||
|
# DSpark TTFT 长于 vLLM baseline 的根因分析
|
||||||
|
|
||||||
|
## 数据回顾
|
||||||
|
|
||||||
|
| Scenario | Backend | Conc | Input | Output | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | Req/s |
|
||||||
|
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| c1_i512_o256 | vllm-dspark | 1 | 512 | 256 | 86.51 | 185.00 | 188.81 | 2.09 | 2.56 |
|
||||||
|
| c1_i512_o256 | vllm-default | 1 | 512 | 256 | 49.50 | 68.50 | 71.09 | 6.66 | 0.99 |
|
||||||
|
| c32_i512_o256 | vllm-dspark | 32 | 512 | 256 | 262.56 | 527.21 | 532.11 | 12.06 | 16.53 |
|
||||||
|
| c32_i512_o256 | vllm-default | 32 | 512 | 256 | 122.62 | 272.00 | 304.80 | 15.77 | 12.98 |
|
||||||
|
| c128_i512_o256 | vllm-dspark | 128 | 512 | 256 | 778.36 | 1071.85 | 1075.35 | 28.95 | 23.94 |
|
||||||
|
| c128_i512_o256 | vllm-default | 128 | 512 | 256 | 617.72 | 767.18 | 770.89 | 18.31 | 29.99 |
|
||||||
|
| c1_i4000_o512 | vllm-dspark | 1 | 4000 | 512 | 123.58 | 176.29 | 190.52 | 1.66 | 1.73 |
|
||||||
|
| c1_i4000_o512 | vllm-default | 1 | 4000 | 512 | 131.50 | 191.53 | 268.69 | 6.68 | 0.50 |
|
||||||
|
| c32_i4000_o512 | vllm-dspark | 32 | 4000 | 512 | 298.42 | 434.24 | 522.34 | 9.68 | 11.49 |
|
||||||
|
| c32_i4000_o512 | vllm-default | 32 | 4000 | 512 | 382.74 | 681.98 | 684.75 | 16.43 | 5.38 |
|
||||||
|
|
||||||
|
## 核心结论
|
||||||
|
|
||||||
|
**DSpark 在短输入（i=512）场景下 TTFT 明显长于 vLLM baseline，是投机解码机制本身的正常 trade-off，但在当前配置下 overhead 偏大，有优化空间。**
|
||||||
|
|
||||||
|
## 为什么短输入场景 TTFT 会变差？
|
||||||
|
|
||||||
|
### 1. 投机解码的本质 overhead
|
||||||
|
|
||||||
|
DSpark 在生成第一个 token 时，除了执行一次目标模型（target model）的 prefill，还要：
|
||||||
|
- 加载并执行 draft model；
|
||||||
|
- 让 draft model 连续预测最多 `spec-tokens=5` 个 token；
|
||||||
|
- 用 target model 对这 5 个 token 做并行验证（verification）；
|
||||||
|
- 根据验证结果回退或接受。
|
||||||
|
|
||||||
|
对于 **短输入（512 tokens）**，baseline 的 prefill 本身非常快（P50 仅 44-65ms），draft model 的额外 forward + verification 的固定 overhead 占了 TTFT 的很大比例，导致 TTFT 反而变长。
|
||||||
|
|
||||||
|
### 2. 长输入场景反而受益
|
||||||
|
|
||||||
|
对于 **长输入（4000 tokens）**，prefill 是主要瓶颈。DSpark 通过 draft model 投机生成后续 token，能显著减少 target model 的 forward 次数，因此：
|
||||||
|
- c1_i4000_o512：TTFT 与 baseline 接近（123ms vs 131ms），但 TPOT 从 6.68ms 降到 1.66ms；
|
||||||
|
- c32_i4000_o512：TTFT 明显优于 baseline（298ms vs 383ms，-22%），TPOT 从 16.43ms 降到 9.68ms。
|
||||||
|
|
||||||
|
这符合投机解码的设计目标：**用 draft model 的额外计算换取 target model 更少的完整 forward，从而在长序列或高输出长度上获得端到端收益。**
|
||||||
|
|
||||||
|
### 3. 高并发短输入场景被放大
|
||||||
|
|
||||||
|
在 c128_i512_o256 下：
|
||||||
|
- DSpark 的 Req/s 低于 baseline（23.94 vs 29.99），说明单位时间内处理的请求更少；
|
||||||
|
- 同时 DSpark 的 mean TPOT 高于 baseline（28.95ms vs 18.31ms）。
|
||||||
|
|
||||||
|
这说明在并发较高、输入较短时，DSpark 的 draft-verification 流水线没有跑赢 baseline 的简单 decode，导致队列等待时间增加，进一步放大了 TTFT。
|
||||||
|
|
||||||
|
## 是代码缺陷还是正常现象？
|
||||||
|
|
||||||
|
### 属于正常现象的部分
|
||||||
|
|
||||||
|
1. **短输入下投机解码增加 TTFT 是预期行为**：只要 draft model 不是零成本，第一个 token 必然比 baseline 慢。
|
||||||
|
2. **DSpark 的收益主要体现在 TPOT/E2E/吞吐量**：所有场景下 DSpark 的 TPOT 都显著低于 baseline，长输入场景 E2E 也大幅缩短。
|
||||||
|
|
||||||
|
### 可能存在优化空间的部分
|
||||||
|
|
||||||
|
1. **`spec-tokens=5` 对短输入可能过大**
|
||||||
|
- 当前 DSpark 使用 `--spec-tokens 5`。
|
||||||
|
- 对于 512 token 的短输入，prefill 后只需要 256 个输出 token，一次 spec 5 个 token 的 verification overhead 相对较高。
|
||||||
|
- **建议**：对短输入场景尝试 `spec-tokens=1` 或 `spec-tokens=2`，看是否能降低 TTFT overhead 同时保持 TPOT 收益。
|
||||||
|
|
||||||
|
2. **DSpark 与 default 使用不同 checkpoint**
|
||||||
|
- DSpark 模型路径是 `/data/models/DeepSeek-V4-Flash-DSpark`，而 baseline 是 `/data/models/DeepSeek-V4-Flash`。
|
||||||
|
- 如果 DSpark checkpoint 的 prefill 路径没有被充分优化（例如 MLA/KV cache 格式差异），可能会额外增加 prefill 时间。
|
||||||
|
- **建议**：确认两个 checkpoint 在 prefill 阶段是否使用相同的 kernel/量化路径，排除模型差异本身带来的 TTFT 偏移。
|
||||||
|
|
||||||
|
3. **draft model 与 target model 的调度/重叠**
|
||||||
|
- 如果 draft model 的 forward 和 target model 的 verification 没有充分 overlap，TTFT 会进一步恶化。
|
||||||
|
- **建议**：通过 vLLM 的 profile/profiling 工具查看 prefill 阶段 draft model 是否串行执行，以及是否可以通过 `--spec-max-model-len` 或调度参数优化。
|
||||||
|
|
||||||
|
4. **高并发下的 batching 策略**
|
||||||
|
- c128_i512_o256 下 DSpark 吞吐量低于 baseline，可能是 speculative decoding 在高并发下的 batching 效率下降。
|
||||||
|
- **建议**：测试不同 `max-num-seqs` 或 `--max-num-batched-tokens` 配置，观察并发吞吐和 TTFT 的变化。
|
||||||
|
|
||||||
|
## 建议的下一步
|
||||||
|
|
||||||
|
1. **参数调优实验**：固定模型和硬件，扫描 `spec-tokens ∈ {1, 2, 3, 5}`，重点观察短输入场景的 TTFT 和 TPOT 变化。
|
||||||
|
2. **profiling**：在 c1_i512_o256 和 c32_i512_o256 下，用 vLLM 的 `--profile` 或 Nsight 抓取 prefill 时间线，确认 draft model 的 overhead 占比。
|
||||||
|
3. **与 MTP 对比**：当前正在跑的 vLLM+MTP 实验会提供另一组投机解码数据，可以对比 DSpark 和官方 MTP 在 TTFT/TPOT 上的 trade-off 差异。
|
||||||
|
|
||||||
|
## 总结
|
||||||
|
|
||||||
|
DSpark 在短输入场景 TTFT 变差**不是代码 bug，而是投机解码机制的正常开销**。但当前 `spec-tokens=5` 的配置在短输入下 overhead 偏大，建议通过调参和 profiling 进一步优化，尤其是在 S1 实时交互层（输入短、延迟敏感）场景。
|
||||||
74
experiments/dsv4_h200_vllm_mtp_vs_default/compare.py
Executable file
74
experiments/dsv4_h200_vllm_mtp_vs_default/compare.py
Executable file
@ -0,0 +1,74 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Generate a side-by-side comparison of vLLM+MTP speculative decoding and vLLM default.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 compare.py --mtp <mtp_result_root> --default <default_result_root> \
|
||||||
|
[--output comparison.md]
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from collections import defaultdict
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def load_result(result_root: Path) -> dict:
|
||||||
|
path = result_root / "results.json"
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
return json.load(f)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--mtp", type=Path, required=True)
|
||||||
|
parser.add_argument("--default", type=Path, required=True)
|
||||||
|
parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md"))
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
mtp_data = load_result(args.mtp)
|
||||||
|
default_data = load_result(args.default)
|
||||||
|
|
||||||
|
by_scenario = defaultdict(dict)
|
||||||
|
for data in (mtp_data, default_data):
|
||||||
|
backend = data["metadata"]["engine"]
|
||||||
|
for s in data.get("scenarios", []):
|
||||||
|
key = s["name"]
|
||||||
|
by_scenario[key][backend] = s
|
||||||
|
|
||||||
|
with open(args.output, "w", encoding="utf-8") as f:
|
||||||
|
f.write("# vLLM+MTP speculative decoding vs vLLM default on DeepSeek-V4-Flash (H200, TP=8)\n\n")
|
||||||
|
f.write("## Summary\n\n")
|
||||||
|
f.write("- Model: `/data/models/DeepSeek-V4-Flash`\n")
|
||||||
|
f.write("- Hardware: 8x NVIDIA H200 143GB\n")
|
||||||
|
f.write("- Tensor Parallelism: 8\n")
|
||||||
|
f.write("- Benchmark client: `sglang.bench_serving --backend vllm`\n")
|
||||||
|
f.write("- Default: no speculative decoding\n")
|
||||||
|
f.write("- MTP: `--speculative-config '{\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":1}'` (official DeepSeek-V4 recipe)\n\n")
|
||||||
|
|
||||||
|
f.write("## Side-by-side results\n\n")
|
||||||
|
f.write("| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) |\n")
|
||||||
|
f.write("|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n")
|
||||||
|
|
||||||
|
for scenario_name in sorted(by_scenario.keys()):
|
||||||
|
for backend in ("vllm-mtp", "vllm-default"):
|
||||||
|
s = by_scenario[scenario_name].get(backend)
|
||||||
|
if s is None:
|
||||||
|
continue
|
||||||
|
cfg = s["config"]
|
||||||
|
m = s["metrics"]
|
||||||
|
f.write(
|
||||||
|
f"| {scenario_name} | {backend} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | "
|
||||||
|
f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
|
||||||
|
f"{m['ttft_ms']['mean']:.2f} | {m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
|
||||||
|
f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p99']:.2f} | "
|
||||||
|
f"{m['e2e_ms']['mean']:.2f} |\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
f.write("\n## Notes\n\n")
|
||||||
|
f.write("- TTFT mean/P95/P99 are the main focus for verifying MTP's impact on time-to-first-token.\n")
|
||||||
|
f.write("- Mean TPOT and E2E are included to check whether speculative decoding pays back after first token.\n")
|
||||||
|
|
||||||
|
print(f"Wrote comparison to {args.output}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
32
experiments/dsv4_h200_vllm_mtp_vs_default/config.env
Normal file
32
experiments/dsv4_h200_vllm_mtp_vs_default/config.env
Normal file
@ -0,0 +1,32 @@
|
|||||||
|
# Configuration for vLLM MTP speculative decoding vs vLLM default comparison on H200.
|
||||||
|
# The default-side results are reused from dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default.
|
||||||
|
|
||||||
|
EXPERIMENT="dsv4_h200_vllm_mtp_vs_default"
|
||||||
|
MODEL_NAME="DeepSeek-V4-Flash"
|
||||||
|
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
|
||||||
|
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||||
|
|
||||||
|
MTP_PORT="${MTP_PORT:-30009}"
|
||||||
|
DEFAULT_RESULT_ROOT="${DEFAULT_RESULT_ROOT:-/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default}"
|
||||||
|
|
||||||
|
VENV_VLLM="${VENV_VLLM:-/data/user1/yy/envs/vllm}"
|
||||||
|
VENV_CLIENT="${VENV_CLIENT:-/data/user1/yy/envs/sglang}"
|
||||||
|
|
||||||
|
export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||||
|
TP=8
|
||||||
|
|
||||||
|
MAX_MODEL_LEN=32768
|
||||||
|
MAX_NUM_SEQS=256
|
||||||
|
SPEC_METHOD="${SPEC_METHOD:-mtp}"
|
||||||
|
SPEC_TOKENS="${SPEC_TOKENS:-1}"
|
||||||
|
|
||||||
|
# Scenarios must match the ones in the reused default result set.
|
||||||
|
declare -a SCENARIOS=(
|
||||||
|
"1 512 256 32"
|
||||||
|
"32 512 256 128"
|
||||||
|
"128 512 256 128"
|
||||||
|
"1 4000 512 32"
|
||||||
|
"32 4000 512 64"
|
||||||
|
)
|
||||||
|
|
||||||
|
MTP_START_SCRIPT="${SCRIPT_DIR:-.}/start_mtp.sh"
|
||||||
@ -0,0 +1,30 @@
|
|||||||
|
# vLLM+MTP speculative decoding vs vLLM default on DeepSeek-V4-Flash (H200, TP=8)
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||||
|
- Hardware: 8x NVIDIA H200 143GB
|
||||||
|
- Tensor Parallelism: 8
|
||||||
|
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||||
|
- Default: no speculative decoding
|
||||||
|
- MTP: `--speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":1}'` (official DeepSeek-V4 recipe)
|
||||||
|
|
||||||
|
## Side-by-side results
|
||||||
|
|
||||||
|
| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) |
|
||||||
|
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| c128_i512_o256 | vllm-mtp | 128 | 512 | 256 | 35.03 | 4625.47 | 771.59 | 1110.39 | 1114.20 | 18.16 | 111.99 | 2506.29 |
|
||||||
|
| c128_i512_o256 | vllm-default | 128 | 512 | 256 | 29.99 | 3959.89 | 617.72 | 767.18 | 770.89 | 18.31 | 65.44 | 2695.91 |
|
||||||
|
| c1_i4000_o512 | vllm-mtp | 1 | 4000 | 512 | 0.83 | 234.90 | 161.27 | 216.12 | 289.08 | 3.70 | 3.89 | 1197.52 |
|
||||||
|
| c1_i4000_o512 | vllm-default | 1 | 4000 | 512 | 0.50 | 140.13 | 131.50 | 191.53 | 268.69 | 6.68 | 6.73 | 2008.09 |
|
||||||
|
| c1_i512_o256 | vllm-mtp | 1 | 512 | 256 | 1.64 | 239.37 | 60.63 | 70.69 | 71.97 | 3.80 | 4.33 | 607.04 |
|
||||||
|
| c1_i512_o256 | vllm-default | 1 | 512 | 256 | 0.99 | 143.82 | 49.50 | 68.50 | 71.09 | 6.66 | 6.71 | 1010.98 |
|
||||||
|
| c32_i4000_o512 | vllm-mtp | 32 | 4000 | 512 | 6.32 | 1785.09 | 409.17 | 667.77 | 673.18 | 14.18 | 19.84 | 4431.73 |
|
||||||
|
| c32_i4000_o512 | vllm-default | 32 | 4000 | 512 | 5.38 | 1518.49 | 382.74 | 681.98 | 684.75 | 16.43 | 25.49 | 4927.71 |
|
||||||
|
| c32_i512_o256 | vllm-mtp | 32 | 512 | 256 | 12.56 | 1657.98 | 279.91 | 761.67 | 766.29 | 16.73 | 37.56 | 2332.50 |
|
||||||
|
| c32_i512_o256 | vllm-default | 32 | 512 | 256 | 12.98 | 1713.20 | 122.62 | 272.00 | 304.80 | 15.77 | 23.26 | 2154.65 |
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- TTFT mean/P95/P99 are the main focus for verifying MTP's impact on time-to-first-token.
|
||||||
|
- Mean TPOT and E2E are included to check whether speculative decoding pays back after first token.
|
||||||
@ -0,0 +1,19 @@
|
|||||||
|
# 8x NVIDIA H200 143GB VLLM Benchmark Report
|
||||||
|
|
||||||
|
- Result root: `/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp`
|
||||||
|
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||||
|
- Backend: VLLM
|
||||||
|
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| c128_i512_o256 | mtp | 128 | 512 | 256 | 3.65 | 128 | 35.03 | 9647.26 | 4625.47 | 14272.73 | 771.59 | 1110.39 | 1114.20 | 18.16 | 32.29 | 111.99 | 2506.29 | 3456.20 | 3560.22 | ✅ |
|
||||||
|
| c1_i4000_o512 | mtp | 1 | 4000 | 512 | 38.35 | 32 | 0.83 | 1694.86 | 234.90 | 1929.76 | 161.27 | 216.12 | 289.08 | 3.70 | 3.82 | 3.89 | 1197.52 | 2034.69 | 2078.68 | ✅ |
|
||||||
|
| c1_i512_o256 | mtp | 1 | 512 | 256 | 19.46 | 32 | 1.64 | 419.94 | 239.37 | 659.31 | 60.63 | 70.69 | 71.97 | 3.80 | 4.11 | 4.33 | 607.04 | 1022.42 | 1023.52 | ✅ |
|
||||||
|
| c32_i4000_o512 | mtp | 32 | 4000 | 512 | 10.12 | 64 | 6.32 | 13280.42 | 1785.09 | 15065.50 | 409.17 | 667.77 | 673.18 | 14.18 | 17.48 | 19.84 | 4431.73 | 8157.51 | 8520.71 | ✅ |
|
||||||
|
| c32_i512_o256 | mtp | 32 | 512 | 256 | 10.19 | 128 | 12.56 | 3458.01 | 1657.98 | 5115.99 | 279.91 | 761.67 | 766.29 | 16.73 | 26.24 | 37.56 | 2332.50 | 4812.90 | 5073.77 | ✅ |
|
||||||
|
|
||||||
|
SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.
|
||||||
|
|
||||||
@ -0,0 +1,311 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_h200_vllm_mtp_vs_default_mtp",
|
||||||
|
"run_id": "20260708-160349",
|
||||||
|
"timestamp": "2026-07-08T16:03:51+00:00",
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash",
|
||||||
|
"backend": "vllm",
|
||||||
|
"engine": "vllm-mtp",
|
||||||
|
"hardware": "8x NVIDIA H200 143GB",
|
||||||
|
"accelerator": "NVIDIA H200",
|
||||||
|
"chip": "nvidia_h200",
|
||||||
|
"script": "experiments/dsv4_h200_vllm_mtp_vs_default/run_bench.sh",
|
||||||
|
"env": "/data/user1/yy/envs/vllm",
|
||||||
|
"git_commit": "ca98fe0",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "H200 vLLM+MTP TP=8 benchmark for DeepSeek-V4-Flash"
|
||||||
|
},
|
||||||
|
"config": {
|
||||||
|
"tp": 8,
|
||||||
|
"cuda_visible_devices": "0,1,2,3,4,5,6,7",
|
||||||
|
"max_model_len": 32768,
|
||||||
|
"max_num_seqs": 256,
|
||||||
|
"spec_method": "mtp",
|
||||||
|
"spec_tokens": 1,
|
||||||
|
"backend": "vllm",
|
||||||
|
"server_start_script": "experiments/dsv4_h200_vllm_mtp_vs_default/start_mtp.sh",
|
||||||
|
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 32768 --max-num-seqs 256 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --speculative-config {\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":1} --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30009"
|
||||||
|
},
|
||||||
|
"scenarios": [
|
||||||
|
{
|
||||||
|
"name": "c128_i512_o256",
|
||||||
|
"config": {
|
||||||
|
"phase": "mtp",
|
||||||
|
"concurrency": 128,
|
||||||
|
"input_len": 512,
|
||||||
|
"output_len": 256,
|
||||||
|
"dataset": "random",
|
||||||
|
"num_prompts": 128
|
||||||
|
},
|
||||||
|
"metrics": {
|
||||||
|
"success": 128,
|
||||||
|
"failed": 0,
|
||||||
|
"duration_s": 3.6536797620065045,
|
||||||
|
"request_throughput": 35.03317431676217,
|
||||||
|
"input_token_throughput": 9647.260377478384,
|
||||||
|
"output_token_throughput": 4625.473796510006,
|
||||||
|
"total_token_throughput": 14272.734173988389,
|
||||||
|
"total_input_tokens": 35248,
|
||||||
|
"total_output_tokens": 16900,
|
||||||
|
"e2e_ms": {
|
||||||
|
"mean": 2506.2933596329913,
|
||||||
|
"p50": 2625.210165999306,
|
||||||
|
"p90": 3417.3656311962986,
|
||||||
|
"p95": 3456.20117305225,
|
||||||
|
"p99": 3560.217050112697
|
||||||
|
},
|
||||||
|
"ttft_ms": {
|
||||||
|
"mean": 771.5936086792681,
|
||||||
|
"p50": 743.3261279948056,
|
||||||
|
"p90": 964.7660584974801,
|
||||||
|
"p95": 1110.3887211014808,
|
||||||
|
"p99": 1114.1990597829863
|
||||||
|
},
|
||||||
|
"tpot_ms": {
|
||||||
|
"mean": 18.15787222271614,
|
||||||
|
"p50": 13.890511986549402,
|
||||||
|
"p90": 25.0715809341214,
|
||||||
|
"p95": 32.29210961239336,
|
||||||
|
"p99": 111.98906169440306
|
||||||
|
},
|
||||||
|
"itl_ms": {
|
||||||
|
"mean": 25.75249892379593,
|
||||||
|
"p50": 21.063708009023685,
|
||||||
|
"p90": 26.390794408507645,
|
||||||
|
"p95": 28.396654246898834,
|
||||||
|
"p99": 255.2068907531792
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"slo_status": {
|
||||||
|
"ttft_p95_ok": true,
|
||||||
|
"tpot_mean_ok": true,
|
||||||
|
"overall": "✅"
|
||||||
|
},
|
||||||
|
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_128_512_256.jsonl"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "c1_i4000_o512",
|
||||||
|
"config": {
|
||||||
|
"phase": "mtp",
|
||||||
|
"concurrency": 1,
|
||||||
|
"input_len": 4000,
|
||||||
|
"output_len": 512,
|
||||||
|
"dataset": "random",
|
||||||
|
"num_prompts": 32
|
||||||
|
},
|
||||||
|
"metrics": {
|
||||||
|
"success": 32,
|
||||||
|
"failed": 0,
|
||||||
|
"duration_s": 38.35236829900532,
|
||||||
|
"request_throughput": 0.8343682911709505,
|
||||||
|
"input_token_throughput": 1694.8627394591913,
|
||||||
|
"output_token_throughput": 234.90074797372165,
|
||||||
|
"total_token_throughput": 1929.7634874329128,
|
||||||
|
"total_input_tokens": 65002,
|
||||||
|
"total_output_tokens": 9009,
|
||||||
|
"e2e_ms": {
|
||||||
|
"mean": 1197.5197937804296,
|
||||||
|
"p50": 1222.8141454979777,
|
||||||
|
"p90": 1945.1575131970458,
|
||||||
|
"p95": 2034.6854075956799,
|
||||||
|
"p99": 2078.678046991845
|
||||||
|
},
|
||||||
|
"ttft_ms": {
|
||||||
|
"mean": 161.27064584452455,
|
||||||
|
"p50": 187.0977380021941,
|
||||||
|
"p90": 207.2812115002307,
|
||||||
|
"p95": 216.12448994783335,
|
||||||
|
"p99": 289.0834562029342
|
||||||
|
},
|
||||||
|
"tpot_ms": {
|
||||||
|
"mean": 3.698815812859111,
|
||||||
|
"p50": 3.6894009238415832,
|
||||||
|
"p90": 3.761323209835616,
|
||||||
|
"p95": 3.8155798700943198,
|
||||||
|
"p99": 3.8925407732209094
|
||||||
|
},
|
||||||
|
"itl_ms": {
|
||||||
|
"mean": 7.345683533679442,
|
||||||
|
"p50": 7.343719502387103,
|
||||||
|
"p90": 7.546291207836475,
|
||||||
|
"p95": 7.646059650869574,
|
||||||
|
"p99": 7.946166486799484
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"slo_status": {
|
||||||
|
"ttft_p95_ok": true,
|
||||||
|
"tpot_mean_ok": true,
|
||||||
|
"overall": "✅"
|
||||||
|
},
|
||||||
|
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_1_4000_512.jsonl"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "c1_i512_o256",
|
||||||
|
"config": {
|
||||||
|
"phase": "mtp",
|
||||||
|
"concurrency": 1,
|
||||||
|
"input_len": 512,
|
||||||
|
"output_len": 256,
|
||||||
|
"dataset": "random",
|
||||||
|
"num_prompts": 32
|
||||||
|
},
|
||||||
|
"metrics": {
|
||||||
|
"success": 32,
|
||||||
|
"failed": 0,
|
||||||
|
"duration_s": 19.455183718004264,
|
||||||
|
"request_throughput": 1.644805850401016,
|
||||||
|
"input_token_throughput": 419.9394936805094,
|
||||||
|
"output_token_throughput": 239.37065141617282,
|
||||||
|
"total_token_throughput": 659.3101450966822,
|
||||||
|
"total_input_tokens": 8170,
|
||||||
|
"total_output_tokens": 4657,
|
||||||
|
"e2e_ms": {
|
||||||
|
"mean": 607.0424630620437,
|
||||||
|
"p50": 709.1372035065433,
|
||||||
|
"p90": 974.411873005738,
|
||||||
|
"p95": 1022.4235178509844,
|
||||||
|
"p99": 1023.5160801192977
|
||||||
|
},
|
||||||
|
"ttft_ms": {
|
||||||
|
"mean": 60.633899186541385,
|
||||||
|
"p50": 67.31531800323864,
|
||||||
|
"p90": 70.1838248045533,
|
||||||
|
"p95": 70.6854243442649,
|
||||||
|
"p99": 71.96889203012688
|
||||||
|
},
|
||||||
|
"tpot_ms": {
|
||||||
|
"mean": 3.7982499249353343,
|
||||||
|
"p50": 3.7510736315107254,
|
||||||
|
"p90": 3.9678993170763492,
|
||||||
|
"p95": 4.107961743017731,
|
||||||
|
"p99": 4.327418892014148
|
||||||
|
},
|
||||||
|
"itl_ms": {
|
||||||
|
"mean": 7.296974936146524,
|
||||||
|
"p50": 7.262905994139146,
|
||||||
|
"p90": 7.464885500667151,
|
||||||
|
"p95": 7.645853242138401,
|
||||||
|
"p99": 9.469499189435757
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"slo_status": {
|
||||||
|
"ttft_p95_ok": true,
|
||||||
|
"tpot_mean_ok": true,
|
||||||
|
"overall": "✅"
|
||||||
|
},
|
||||||
|
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_1_512_256.jsonl"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "c32_i4000_o512",
|
||||||
|
"config": {
|
||||||
|
"phase": "mtp",
|
||||||
|
"concurrency": 32,
|
||||||
|
"input_len": 4000,
|
||||||
|
"output_len": 512,
|
||||||
|
"dataset": "random",
|
||||||
|
"num_prompts": 64
|
||||||
|
},
|
||||||
|
"metrics": {
|
||||||
|
"success": 64,
|
||||||
|
"failed": 0,
|
||||||
|
"duration_s": 10.122198707002099,
|
||||||
|
"request_throughput": 6.322736971733974,
|
||||||
|
"input_token_throughput": 13280.415045301297,
|
||||||
|
"output_token_throughput": 1785.086474097831,
|
||||||
|
"total_token_throughput": 15065.501519399128,
|
||||||
|
"total_input_tokens": 134427,
|
||||||
|
"total_output_tokens": 18069,
|
||||||
|
"e2e_ms": {
|
||||||
|
"mean": 4431.7270394060415,
|
||||||
|
"p50": 4118.452171998797,
|
||||||
|
"p90": 7965.46315389569,
|
||||||
|
"p95": 8157.508349696581,
|
||||||
|
"p99": 8520.712041212246
|
||||||
|
},
|
||||||
|
"ttft_ms": {
|
||||||
|
"mean": 409.17025550083963,
|
||||||
|
"p50": 412.25154899439076,
|
||||||
|
"p90": 663.4122133953497,
|
||||||
|
"p95": 667.7715254438226,
|
||||||
|
"p99": 673.1776816866477
|
||||||
|
},
|
||||||
|
"tpot_ms": {
|
||||||
|
"mean": 14.18404794198284,
|
||||||
|
"p50": 15.168117225958857,
|
||||||
|
"p90": 17.233481520907418,
|
||||||
|
"p95": 17.483692691057605,
|
||||||
|
"p99": 19.836913968959017
|
||||||
|
},
|
||||||
|
"itl_ms": {
|
||||||
|
"mean": 28.557059211652966,
|
||||||
|
"p50": 13.894067000364885,
|
||||||
|
"p90": 29.86279199831188,
|
||||||
|
"p95": 174.40551639592738,
|
||||||
|
"p99": 181.762090458069
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"slo_status": {
|
||||||
|
"ttft_p95_ok": true,
|
||||||
|
"tpot_mean_ok": true,
|
||||||
|
"overall": "✅"
|
||||||
|
},
|
||||||
|
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_32_4000_512.jsonl"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "c32_i512_o256",
|
||||||
|
"config": {
|
||||||
|
"phase": "mtp",
|
||||||
|
"concurrency": 32,
|
||||||
|
"input_len": 512,
|
||||||
|
"output_len": 256,
|
||||||
|
"dataset": "random",
|
||||||
|
"num_prompts": 128
|
||||||
|
},
|
||||||
|
"metrics": {
|
||||||
|
"success": 128,
|
||||||
|
"failed": 0,
|
||||||
|
"duration_s": 10.193132108004647,
|
||||||
|
"request_throughput": 12.55747484126904,
|
||||||
|
"input_token_throughput": 3458.0146344144614,
|
||||||
|
"output_token_throughput": 1657.9791001363028,
|
||||||
|
"total_token_throughput": 5115.993734550764,
|
||||||
|
"total_input_tokens": 35248,
|
||||||
|
"total_output_tokens": 16900,
|
||||||
|
"e2e_ms": {
|
||||||
|
"mean": 2332.5013259607204,
|
||||||
|
"p50": 2112.3746865050634,
|
||||||
|
"p90": 4070.417974996962,
|
||||||
|
"p95": 4812.899109590944,
|
||||||
|
"p99": 5073.772265354783
|
||||||
|
},
|
||||||
|
"ttft_ms": {
|
||||||
|
"mean": 279.9133442812263,
|
||||||
|
"p50": 206.422848496004,
|
||||||
|
"p90": 663.5394259006716,
|
||||||
|
"p95": 761.6650494928763,
|
||||||
|
"p99": 766.2871305434965
|
||||||
|
},
|
||||||
|
"tpot_ms": {
|
||||||
|
"mean": 16.7286719523913,
|
||||||
|
"p50": 16.23081167053897,
|
||||||
|
"p90": 23.97215435499593,
|
||||||
|
"p95": 26.244743178375007,
|
||||||
|
"p99": 37.558756268073765
|
||||||
|
},
|
||||||
|
"itl_ms": {
|
||||||
|
"mean": 30.42237174675025,
|
||||||
|
"p50": 14.531328000884969,
|
||||||
|
"p90": 61.01081449742196,
|
||||||
|
"p95": 163.01492899947334,
|
||||||
|
"p99": 182.57800715073245
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"slo_status": {
|
||||||
|
"ttft_p95_ok": true,
|
||||||
|
"tpot_mean_ok": true,
|
||||||
|
"overall": "✅"
|
||||||
|
},
|
||||||
|
"raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/mtp/raw_outputs/vllm_mtp_0708_32_512_256.jsonl"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
216
experiments/dsv4_h200_vllm_mtp_vs_default/run_bench.sh
Executable file
216
experiments/dsv4_h200_vllm_mtp_vs_default/run_bench.sh
Executable file
@ -0,0 +1,216 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# vLLM+MTP speculative decoding vs vLLM default comparison on H200.
|
||||||
|
# Reuses the default-side results from dsv4_h200_vllm_dspark_vs_default.
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
RESULT_BASE="${SCRIPT_DIR}/results"
|
||||||
|
|
||||||
|
log_dir_global="${RESULT_BASE}/${RUN_ID}/logs"
|
||||||
|
mkdir -p "$log_dir_global"
|
||||||
|
log_init "${log_dir_global}/orchestrator.log"
|
||||||
|
|
||||||
|
log "experiment=${EXPERIMENT_NAME}"
|
||||||
|
log "run_id=${RUN_ID}"
|
||||||
|
log "platform=${PLATFORM}"
|
||||||
|
log "hardware=${HARDWARE}"
|
||||||
|
log "model=${MODEL_PATH}"
|
||||||
|
log "spec_method=${SPEC_METHOD}"
|
||||||
|
log "spec_tokens=${SPEC_TOKENS}"
|
||||||
|
log "default_result_root=${DEFAULT_RESULT_ROOT}"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
is_server_healthy() {
|
||||||
|
local port="$1"
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${port}/health" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_server() {
|
||||||
|
local pid_file="/data/user1/yy/${EXPERIMENT_NAME}_mtp.pid"
|
||||||
|
if [[ -f "$pid_file" ]]; then
|
||||||
|
local pid
|
||||||
|
pid="$(cat "$pid_file")"
|
||||||
|
if kill -0 "$pid" 2>/dev/null; then
|
||||||
|
log "stopping mtp server pid=${pid}"
|
||||||
|
kill "$pid" 2>/dev/null || true
|
||||||
|
sleep 5
|
||||||
|
kill -9 "$pid" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
rm -f "$pid_file"
|
||||||
|
fi
|
||||||
|
pkill -9 -f "vllm serve.*DeepSeek-V4-Flash" 2>/dev/null || true
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
|
||||||
|
start_server() {
|
||||||
|
log "starting mtp server with ${MTP_START_SCRIPT}"
|
||||||
|
bash "${MTP_START_SCRIPT}" >> "${log_dir_global}/mtp.server.outer.log" 2>&1
|
||||||
|
|
||||||
|
if ! is_server_healthy "$MTP_PORT"; then
|
||||||
|
log "error: mtp server failed to become healthy"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
log "mtp server is healthy on port ${MTP_PORT}"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_warmup() {
|
||||||
|
log "warming up mtp (input=4000, output=512, num=2)"
|
||||||
|
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/warmup.py" \
|
||||||
|
--backend vllm \
|
||||||
|
--port "$MTP_PORT" \
|
||||||
|
--input-len 4000 \
|
||||||
|
--output-len 512 \
|
||||||
|
--num 2 \
|
||||||
|
--env-python "${VENV_CLIENT}/bin/python" \
|
||||||
|
>> "${log_dir_global}/mtp.warmup.log" 2>&1
|
||||||
|
log "warmup for mtp completed"
|
||||||
|
}
|
||||||
|
|
||||||
|
scenario_already_completed() {
|
||||||
|
local output_file="$1"
|
||||||
|
local expected="$2"
|
||||||
|
[[ -s "$output_file" ]] || return 1
|
||||||
|
local completed
|
||||||
|
completed="$(${VENV_CLIENT}/bin/python -c "
|
||||||
|
import json, sys
|
||||||
|
path = sys.argv[1]
|
||||||
|
try:
|
||||||
|
with open(path, 'r', encoding='utf-8') as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if line:
|
||||||
|
data = json.loads(line)
|
||||||
|
print(data.get('completed', 0))
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
print(0)
|
||||||
|
" "$output_file")"
|
||||||
|
[[ "${completed:-0}" -ge "$expected" ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
run_benchmark() {
|
||||||
|
local result_root="${RESULT_BASE}/${RUN_ID}/mtp"
|
||||||
|
local raw_dir="${result_root}/raw_outputs"
|
||||||
|
local bench_log_dir="${result_root}/logs"
|
||||||
|
|
||||||
|
mkdir -p "$raw_dir" "$bench_log_dir"
|
||||||
|
|
||||||
|
log "===== mtp BENCHMARK START ====="
|
||||||
|
|
||||||
|
stop_server
|
||||||
|
start_server
|
||||||
|
run_warmup
|
||||||
|
|
||||||
|
for scenario in "${SCENARIOS[@]}"; do
|
||||||
|
read -r concurrency input_len output_len num_prompts <<< "$scenario"
|
||||||
|
output_file="${raw_dir}/vllm_mtp_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||||
|
detail_log="${bench_log_dir}/vllm_mtp_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||||
|
|
||||||
|
if scenario_already_completed "$output_file" "$num_prompts"; then
|
||||||
|
log "skipping already-completed mtp scenario: c=${concurrency} i=${input_len} o=${output_len}"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "running mtp scenario: c=${concurrency} i=${input_len} o=${output_len} n=${num_prompts}"
|
||||||
|
|
||||||
|
"${VENV_CLIENT}/bin/python" -m sglang.bench_serving \
|
||||||
|
--backend vllm \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "$MTP_PORT" \
|
||||||
|
--dataset-name random \
|
||||||
|
--random-input-len "$input_len" \
|
||||||
|
--random-output-len "$output_len" \
|
||||||
|
--num-prompts "$num_prompts" \
|
||||||
|
--max-concurrency "$concurrency" \
|
||||||
|
--request-rate 10000 \
|
||||||
|
--output-file "$output_file" \
|
||||||
|
--output-details \
|
||||||
|
> "$detail_log" 2>&1 || {
|
||||||
|
log "ERROR: mtp scenario c=${concurrency} i=${input_len} o=${output_len} failed; see ${detail_log}"
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
log "finished mtp scenario: output=${output_file}"
|
||||||
|
done
|
||||||
|
|
||||||
|
stop_server
|
||||||
|
log "===== mtp BENCHMARK DONE ====="
|
||||||
|
}
|
||||||
|
|
||||||
|
parse_mtp() {
|
||||||
|
local result_root="${RESULT_BASE}/${RUN_ID}/mtp"
|
||||||
|
log "parsing mtp results in ${result_root}"
|
||||||
|
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend vllm \
|
||||||
|
>> "${result_root}/logs/parse.log" 2>&1 || {
|
||||||
|
log "WARNING: parser failed for mtp; see ${result_root}/logs/parse.log"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
write_mtp_metadata() {
|
||||||
|
local result_root="${RESULT_BASE}/${RUN_ID}/mtp"
|
||||||
|
ensure_result_root "$result_root"
|
||||||
|
local meta_json="${result_root}/results.json"
|
||||||
|
|
||||||
|
local server_args
|
||||||
|
server_args="vllm serve $MODEL_PATH --trust-remote-code --tensor-parallel-size $TP --kv-cache-dtype fp8 --max-model-len $MAX_MODEL_LEN --max-num-seqs $MAX_NUM_SEQS --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --speculative-config {\\\"method\\\":\\\"mtp\\\",\\\"num_speculative_tokens\\\":$SPEC_TOKENS} --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port $MTP_PORT"
|
||||||
|
|
||||||
|
write_metadata_json \
|
||||||
|
"$meta_json" \
|
||||||
|
"${EXPERIMENT_NAME}_mtp" \
|
||||||
|
"$RUN_ID" \
|
||||||
|
"$MODEL_PATH" \
|
||||||
|
"vllm" \
|
||||||
|
"vllm-mtp" \
|
||||||
|
"$HARDWARE" \
|
||||||
|
"$ACCELERATOR" \
|
||||||
|
"$CHIP" \
|
||||||
|
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||||
|
"$VENV_VLLM" \
|
||||||
|
"H200 vLLM+MTP TP=8 benchmark for DeepSeek-V4-Flash"
|
||||||
|
|
||||||
|
jq --arg server_args "$server_args" \
|
||||||
|
'.config = {
|
||||||
|
"tp": 8,
|
||||||
|
"cuda_visible_devices": "0,1,2,3,4,5,6,7",
|
||||||
|
"max_model_len": 32768,
|
||||||
|
"max_num_seqs": 256,
|
||||||
|
"spec_method": "mtp",
|
||||||
|
"spec_tokens": 1,
|
||||||
|
"backend": "vllm",
|
||||||
|
"server_start_script": "experiments/dsv4_h200_vllm_mtp_vs_default/start_mtp.sh",
|
||||||
|
"server_args": $server_args
|
||||||
|
}' "$meta_json" > "${meta_json}.tmp" && mv "${meta_json}.tmp" "$meta_json"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
stop_server
|
||||||
|
write_mtp_metadata
|
||||||
|
run_benchmark
|
||||||
|
parse_mtp
|
||||||
|
|
||||||
|
log "generating comparison report"
|
||||||
|
"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/compare.py" \
|
||||||
|
--mtp "${RESULT_BASE}/${RUN_ID}/mtp" \
|
||||||
|
--default "$DEFAULT_RESULT_ROOT" \
|
||||||
|
--output "${RESULT_BASE}/${RUN_ID}/comparison.md" \
|
||||||
|
>> "${log_dir_global}/compare.log" 2>&1 || {
|
||||||
|
log "WARNING: comparison script failed; see ${log_dir_global}/compare.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
log "all results saved to ${RESULT_BASE}/${RUN_ID}"
|
||||||
66
experiments/dsv4_h200_vllm_mtp_vs_default/start_mtp.sh
Executable file
66
experiments/dsv4_h200_vllm_mtp_vs_default/start_mtp.sh
Executable file
@ -0,0 +1,66 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Start vLLM with MTP speculative decoding for the comparison experiment.
|
||||||
|
set -e
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p logs tmp
|
||||||
|
|
||||||
|
VENV="${VENV_VLLM}"
|
||||||
|
export PATH="$VENV/bin:$PATH"
|
||||||
|
export PYTHONUNBUFFERED=1
|
||||||
|
export TMPDIR=/data/user1/yy/tmp
|
||||||
|
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}"
|
||||||
|
|
||||||
|
LOG="/data/user1/yy/logs/dsv4_h200_vllm_mtp_vs_default_mtp_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
PID_FILE="/data/user1/yy/dsv4_h200_vllm_mtp_vs_default_mtp.pid"
|
||||||
|
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
|
||||||
|
echo "=== Starting vLLM+MTP server (TP=$TP, spec_method=$SPEC_METHOD, spec_tokens=$SPEC_TOKENS) ==="
|
||||||
|
echo "Model: $MODEL_PATH"
|
||||||
|
echo "Port: $MTP_PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
|
||||||
|
nohup vllm serve "$MODEL_PATH" \
|
||||||
|
--trust-remote-code \
|
||||||
|
--tensor-parallel-size "$TP" \
|
||||||
|
--kv-cache-dtype fp8 \
|
||||||
|
--max-model-len "$MAX_MODEL_LEN" \
|
||||||
|
--max-num-seqs "$MAX_NUM_SEQS" \
|
||||||
|
--block-size 256 \
|
||||||
|
--gpu-memory-utilization 0.90 \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--speculative-config "{\"method\":\"mtp\",\"num_speculative_tokens\":$SPEC_TOKENS}" \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--port "$MTP_PORT" \
|
||||||
|
> "$LOG" 2>&1 &
|
||||||
|
|
||||||
|
PID=$!
|
||||||
|
echo $PID > "$PID_FILE"
|
||||||
|
echo "PID: $PID"
|
||||||
|
echo "Waiting for health..."
|
||||||
|
|
||||||
|
for i in $(seq 1 240); do
|
||||||
|
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${MTP_PORT}/health" >/dev/null 2>&1; then
|
||||||
|
echo "vLLM+MTP server is ready at http://127.0.0.1:${MTP_PORT}"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PID 2>/dev/null; then
|
||||||
|
echo "ERROR: vLLM+MTP server exited early"
|
||||||
|
tail -200 "$LOG"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting... ($i/240)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "ERROR: vLLM+MTP server not healthy after 240 retries"
|
||||||
|
tail -200 "$LOG"
|
||||||
|
exit 1
|
||||||
Loading…
x
Reference in New Issue
Block a user