Initial commit: benchmark scripts and final reports
This commit is contained in:
Benchmark Bot
commit
eb431d3925
27
.gitignore
vendored
Normal file
27
.gitignore
vendored
Normal file
@ -0,0 +1,27 @@
|
|||||||
|
# 环境/依赖/临时文件
|
||||||
|
envs/
|
||||||
|
deps/
|
||||||
|
tmp/
|
||||||
|
*.pid
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
*.pyo
|
||||||
|
*.egg-info/
|
||||||
|
|
||||||
|
# 大文件/压缩包
|
||||||
|
*.zip
|
||||||
|
*.tar.gz
|
||||||
|
*.tar.bz2
|
||||||
|
|
||||||
|
# 中间日志
|
||||||
|
logs/
|
||||||
|
*.log
|
||||||
|
|
||||||
|
# 数据集（可重新下载）
|
||||||
|
datasets/
|
||||||
|
|
||||||
|
# 原始请求级输出（方案 A：极简版，不存原始 jsonl）
|
||||||
|
bench_results/**/raw_outputs/
|
||||||
|
|
||||||
|
# 无关项目
|
||||||
|
loomeval_yy/
|
||||||
122
BENCHMARK_WORKFLOW.md
Normal file
122
BENCHMARK_WORKFLOW.md
Normal file
@ -0,0 +1,122 @@
|
|||||||
|
# Benchmark Workflow & Directory Conventions
|
||||||
|
|
||||||
|
## Directory Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
/data/user1/yy/
|
||||||
|
├── scripts/ # all benchmark/orchestrator/utility scripts
|
||||||
|
│ ├── benchmark_dspark_0707/ # DSpark benchmark suite
|
||||||
|
│ ├── benchmark_dsv4_backend_comparison.sh
|
||||||
|
│ ├── start_dsv4_dspark_8card.sh
|
||||||
|
│ ├── start_sglang_dsv4_8card.sh
|
||||||
|
│ └── ...
|
||||||
|
├── bench_results/ # all benchmark outputs and reports
|
||||||
|
│ ├── dsv4_backend_comparison_20260707/
|
||||||
|
│ │ ├── raw_outputs/ # JSONL raw outputs
|
||||||
|
│ │ ├── logs/ # per-run logs
|
||||||
|
│ │ └── README.md # output manifest + provenance
|
||||||
|
│ ├── dspark_grid_20260707-132641/
|
||||||
|
│ ├── dspark_st_comparison_20260707-150649/
|
||||||
|
│ ├── eagle_grid/
|
||||||
|
│ └── ...
|
||||||
|
├── logs/ # server logs (stdout/stderr from start scripts)
|
||||||
|
├── datasets/ # benchmark datasets
|
||||||
|
└── envs/ # Python virtual environments
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
1. **Scripts live in `scripts/` only.**
|
||||||
|
- Group related scripts into subdirectories, e.g. `scripts/benchmark_dspark_0707/`.
|
||||||
|
- Each script group should have its own `README.md` listing scripts, purpose, and outputs.
|
||||||
|
|
||||||
|
2. **Benchmark outputs live in `bench_results/` only.**
|
||||||
|
- Never leave `.jsonl`, `.json`, `.log`, or `.md` reports in the project root.
|
||||||
|
- Each benchmark run gets its own directory: `bench_results/<experiment>_<timestamp>/`.
|
||||||
|
- Raw outputs go in `raw_outputs/`.
|
||||||
|
- Logs go in `logs/`.
|
||||||
|
- Reports (e.g. `report.md`, `comparison_report.md`) go in the run root.
|
||||||
|
|
||||||
|
3. **Each `bench_results/<run>/` directory must contain a `README.md`.**
|
||||||
|
- What was benchmarked.
|
||||||
|
- Which script(s) produced the outputs.
|
||||||
|
- File naming convention.
|
||||||
|
- Inventory of outputs (table).
|
||||||
|
|
||||||
|
4. **Scripts should default `RESULT_ROOT` to `bench_results/<experiment>_${RUN_ID}`.**
|
||||||
|
- Allow override via `RESULT_ROOT` env var.
|
||||||
|
- Use `RUN_ID=$(date '+%Y%m%d-%H%M%S')` unless specified.
|
||||||
|
|
||||||
|
5. **Server start scripts write to `logs/`.**
|
||||||
|
- `logs/<service>_<timestamp>.log`
|
||||||
|
- Keep server logs separate from benchmark result logs.
|
||||||
|
|
||||||
|
## Naming Conventions
|
||||||
|
|
||||||
|
### Result directories
|
||||||
|
|
||||||
|
```
|
||||||
|
bench_results/<experiment>_<YYYYMMDD-HHMMSS>/
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
- `bench_results/dspark_grid_20260707-132641/`
|
||||||
|
- `bench_results/dsv4_backend_comparison_20260707/`
|
||||||
|
|
||||||
|
### Raw output files
|
||||||
|
|
||||||
|
For detailed per-request JSONL outputs:
|
||||||
|
|
||||||
|
```
|
||||||
|
{backend}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
|
||||||
|
```
|
||||||
|
|
||||||
|
For summary JSON outputs from `sglang.bench_serving --output-file`:
|
||||||
|
|
||||||
|
```
|
||||||
|
{backend}_{scenario}_{params}.json
|
||||||
|
```
|
||||||
|
|
||||||
|
### Logs
|
||||||
|
|
||||||
|
```
|
||||||
|
logs/<service>_YYYYMMDD_HHMMSS.log
|
||||||
|
logs/<experiment>_orchestrator_YYYYMMDD_HHMMSS.log
|
||||||
|
```
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Run DSpark grid benchmark
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run DSpark spec-tokens comparison
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run SGLang vs vLLM backend comparison
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start SGLang on port 30000 and vLLM on port 8000, then:
|
||||||
|
bash scripts/benchmark_dsv4_backend_comparison.sh all
|
||||||
|
```
|
||||||
|
|
||||||
|
### Parse results
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
|
||||||
|
/data/user1/yy/bench_results/dspark_grid_<run_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Checklist Before Committing / Archiving
|
||||||
|
|
||||||
|
- [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in `/data/user1/yy/` root.
|
||||||
|
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
|
||||||
|
- [ ] `bench_results/<run>/README.md` exists and documents provenance.
|
||||||
|
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
|
||||||
|
- [ ] Script path references updated after moving.
|
||||||
63
README.md
Normal file
63
README.md
Normal file
@ -0,0 +1,63 @@
|
|||||||
|
# DSV4 / DeepSeek-V4-Flash 推理测速项目
|
||||||
|
|
||||||
|
> 记录 vllm-dspark、SGLang、vLLM 等后端在 DeepSeek-V4-Flash 上的 benchmark 脚本与最终结果。
|
||||||
|
|
||||||
|
## 目录说明
|
||||||
|
|
||||||
|
| 目录/文件 | 说明 |
|
||||||
|
|---|---|
|
||||||
|
| `scripts/` | 所有 benchmark 脚本、服务启动脚本、结果解析脚本 |
|
||||||
|
| `bench_results/` | 各次实验的最终报告与 summary JSON |
|
||||||
|
| `BENCHMARK_WORKFLOW.md` | benchmark 目录与命名规范 |
|
||||||
|
| `scripts/SLO_STANDARDS.md` | 推理服务 SLO 标准（TTFT/TPOT） |
|
||||||
|
|
||||||
|
> 注：`.gitignore` 已排除 `envs/`、`deps/`、`tmp/`、`datasets/`、`logs/`、`*.zip`、原始 `raw_outputs/*.jsonl` 等大文件/中间文件。
|
||||||
|
|
||||||
|
## 实验索引
|
||||||
|
|
||||||
|
| 实验 | 脚本 | 最终报告 | 说明 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| DSpark grid benchmark | `scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh` | `bench_results/dspark_grid_20260707-132641/report.md` | P1/P2/P3 全量网格 |
|
||||||
|
| DSpark spec-tokens 对比 | `scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh` | `bench_results/dspark_st_comparison_20260707-150649/comparison_report.md` | `--spec-tokens 3` vs `5` |
|
||||||
|
| SGLang EAGLE vs DSpark | `scripts/start_sglang_dsv4_8card.sh` + 对比解析脚本 | `bench_results/eagle_grid/dspark_vs_eagle_report.md` | EAGLE 投机解码对比 |
|
||||||
|
| SGLang vs vLLM 后端对比（0707） | `scripts/benchmark_dsv4_backend_comparison.sh` | `bench_results/dsv4_backend_comparison_20260707/README.md` | 遗留 raw outputs 清单 |
|
||||||
|
| SGLang 8-card systematic | `scripts/run_sglang_benchmark.sh` | `bench_results/sglang_8card_systematic_20260704_120819/report.md` | 早期 SGLang 系统扫描 |
|
||||||
|
| SGLang 8-card max throughput | `scripts/run_sglang_max_throughput.sh` | `bench_results/sglang_8card_max_throughput_20260705_030839/summary.json` | 最大吞吐扫描 |
|
||||||
|
| vLLM-dspark Qwen3 | `scripts/bench_vllm_dspark_qwen3.py` | `bench_results/vllm_dspark_qwen3_20260705_121256/summary.json` | Qwen3 模型测速 |
|
||||||
|
| DSV4 后端对比（0705） | `scripts/bench_dsv4_comparison.py` | `bench_results/dsv4_comparison_20260705_152221/summary.json` | 早期 DSpark 配置对比 |
|
||||||
|
|
||||||
|
## 快速复现
|
||||||
|
|
||||||
|
### DSpark grid
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Spec-tokens 对比
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### 解析已有结果
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
|
||||||
|
/data/user1/yy/bench_results/dspark_grid_20260707-132641
|
||||||
|
```
|
||||||
|
|
||||||
|
## SLO 参考
|
||||||
|
|
||||||
|
详见 `scripts/SLO_STANDARDS.md`。
|
||||||
|
|
||||||
|
主要关注指标：
|
||||||
|
|
||||||
|
- **TTFT P95**：S2（DeepSeek-V4-Flash 所在层）目标 `< 3s`
|
||||||
|
- **TPOT**：S2 目标 `< 50ms`
|
||||||
|
|
||||||
|
## 环境要求
|
||||||
|
|
||||||
|
- Python env：`/data/user1/yy/envs/vllm-dspark`（服务端）、`/data/user1/yy/envs/sglang`（压测客户端）
|
||||||
|
- 模型：`/data/models/DeepSeek-V4-Flash`、`/data/models/DeepSeek-V4-Flash-DSpark`
|
||||||
|
- 硬件：8× H200（当前配置）
|
||||||
167
bench_results/dspark_grid_20260707-132641/evaluation.md
Normal file
167
bench_results/dspark_grid_20260707-132641/evaluation.md
Normal file
@ -0,0 +1,167 @@
|
|||||||
|
# vllm-dspark Benchmark 结果评估
|
||||||
|
|
||||||
|
> 对应结果：`/data/user1/yy/bench_results/dspark_grid_20260707-132641`
|
||||||
|
> 运行时间：2026-07-07
|
||||||
|
> 硬件：8× NVIDIA H200 143GB
|
||||||
|
> 模型：`/data/models/DeepSeek-V4-Flash-DSpark`
|
||||||
|
> 服务配置：TP=8，FP8 KV cache，`--spec-method dspark --spec-tokens 5`，block-size=256，max-num-seqs=256
|
||||||
|
> 压测工具：`sglang.bench_serving --backend vllm`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 总体结论
|
||||||
|
|
||||||
|
本次 grid 测试覆盖了从轻量 chat 到超长上下文、从重 decode 到高并发压测的多种场景。整体上看：
|
||||||
|
|
||||||
|
- **DSpark 在短输入、中高并发场景下表现优秀**，`chat_short` 在并发 64 时达到约 9580 tok/s，`stress_standard` 在并发 128 时达到约 17465 tok/s。
|
||||||
|
- **低并发下 DSpark 的 draft 开销明显**，部分场景单并发延迟和吞吐都不如预期。
|
||||||
|
- **超长上下文场景存在明显的吞吐拐点**，`long_rag`（32K 输入）在并发 4 达到峰值后，并发 8 吞吐腰斩。
|
||||||
|
- **延迟长尾（P95/P99）普遍较重**，提示调度、内存或投机解码验证阶段存在抖动。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 异常点分析
|
||||||
|
|
||||||
|
### 2.1 低并发（c=1）下 DSpark 收益被开销吃掉
|
||||||
|
|
||||||
|
以 `chat_standard`（input=1000, output=256）为例：
|
||||||
|
|
||||||
|
| 并发 | req/s | mean E2E(ms) | P99 E2E(ms) | mean TTFT(ms) | P99 TTFT(ms) |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| 1 | 0.49 | 2040.65 | 10715.53 | 1412.48 | 8430.71 |
|
||||||
|
| 8 | 4.35 | 1819.35 | 15165.63 | 327.62 | 4404.92 |
|
||||||
|
| 16 | 8.32 | 1885.15 | 10491.31 | 443.50 | 8567.07 |
|
||||||
|
| 32 | 10.00 | 3156.73 | 12936.08 | 560.77 | 7129.66 |
|
||||||
|
|
||||||
|
- **单并发吞吐仅 0.49 req/s**，远低于无投机基线可预期的水平。
|
||||||
|
- **P99 TTFT 高达 8.4s**，说明单请求首次预填充极不稳定，可能是 draft 验证/编译缓存未命中或 warmup 不充分导致。
|
||||||
|
- 并发提升到 8 后，TTFT 均值下降到 327ms，说明 DSpark 的 draft 计算需要 batch 才能摊薄。
|
||||||
|
|
||||||
|
**判断**：这是 DSpark 的典型特征——低并发下 draft 固定开销无法被摊薄，导致延迟尾和单请求吞吐都较差。
|
||||||
|
|
||||||
|
### 2.2 超长上下文出现吞吐断崖
|
||||||
|
|
||||||
|
`long_rag`（input=32000, output=512）：
|
||||||
|
|
||||||
|
| 并发 | Total tok/s | mean TTFT(ms) | P99 TTFT(ms) |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| 1 | 16104.98 | 549.72 | 1163.96 |
|
||||||
|
| 2 | 43215.62 | 192.57 | 409.32 |
|
||||||
|
| 4 | **70040.54** | 191.57 | 427.69 |
|
||||||
|
| 8 | 34141.70（↓51%） | 930.98 | 2742.48 |
|
||||||
|
|
||||||
|
`long_context_probe`（input=16000, output=512）也有类似模式：
|
||||||
|
|
||||||
|
| 并发 | Total tok/s | mean TTFT(ms) |
|
||||||
|
|---|---:|---:|
|
||||||
|
| 1 | 11051.09 | 316.59 |
|
||||||
|
| 2 | 26862.06 | 110.65 |
|
||||||
|
| 4 | 43042.77 | 127.38 |
|
||||||
|
| 8 | 29255.46（↓32%） | 453.93 |
|
||||||
|
| 16 | 43292.68 | 444.14 |
|
||||||
|
|
||||||
|
**判断**：
|
||||||
|
- 32K/16K 长 prefill 在并发 4 时达到最佳，继续加并发反而下降，可能与 **KV cache 显存带宽瓶颈**、**prefill 阶段 scheduling 阻塞** 或 **hybrid KV cache manager 的换入换出** 有关。
|
||||||
|
- `long_rag` 在 c=8 时 P99 TTFT 暴涨到 2.7s，进一步印证内存/调度压力。
|
||||||
|
|
||||||
|
### 2.3 延迟长尾普遍偏重
|
||||||
|
|
||||||
|
几乎所有场景的 P99 E2E 都是 mean E2E 的 2~4 倍：
|
||||||
|
|
||||||
|
| 场景 | 并发 | mean E2E | P99 E2E | 倍数 |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| chat_standard | 32 | 3156.73 | 12936.08 | 4.1× |
|
||||||
|
| generation_standard | 32 | 2908.89 | 6938.03 | 2.4× |
|
||||||
|
| summarization | 32 | 5587.21 | 14541.75 | 2.6× |
|
||||||
|
| decode_heavy | 32 | 5562.57 | 12822.35 | 2.3× |
|
||||||
|
| stress_standard | 128 | 4417.43 | 14850.43 | 3.4× |
|
||||||
|
|
||||||
|
**判断**：投机解码的 draft 验证失败会导致 fall back 到逐个 target 前向，造成明显的延迟毛刺；另外 vLLM v1 引擎的 scheduling 在高并发下也可能产生队列等待。
|
||||||
|
|
||||||
|
### 2.4 高并发下 TPOT/ITL 增长较快
|
||||||
|
|
||||||
|
`stress_standard`（input=1000, output=256）：
|
||||||
|
|
||||||
|
| 并发 | mean TPOT | P95 TPOT | P99 TPOT | mean ITL |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| 96 | 29.46 | 67.30 | 123.42 | 113.64 |
|
||||||
|
| 128 | 35.20 | 76.85 | 141.17 | 136.57 |
|
||||||
|
|
||||||
|
TPOT 从 29ms 升到 35ms，P99 超过 140ms，说明 decode 阶段在极限并发下已经接近饱和。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 可重点优化的方向
|
||||||
|
|
||||||
|
### 3.1 按并发动态调整 `--spec-tokens`
|
||||||
|
|
||||||
|
参考历史对比数据（`dsv4_inference_comparison_report.md`）：
|
||||||
|
|
||||||
|
| spec-tokens | 低并发（c=1） | 高并发（c=64） |
|
||||||
|
|---|---|---|
|
||||||
|
| 3 | 1.03 req/s, 254 tok/s | **14.53 req/s, 3507 tok/s** |
|
||||||
|
| 5 | **1.07 req/s, 257 tok/s** | 9.36 req/s, 2210 tok/s |
|
||||||
|
| 7 | 1.03 req/s, 251 tok/s | 9.13 req/s, 2218 tok/s |
|
||||||
|
|
||||||
|
当前部署固定使用 `--spec-tokens 5`：
|
||||||
|
- 对低并发（c=1~16）相对友好；
|
||||||
|
- 对高并发（c≥64）不是最优，st=3 在高并发下接受率更高、验证开销更小。
|
||||||
|
|
||||||
|
**建议**：
|
||||||
|
- 高并发服务场景：改用 `--spec-tokens 3`。
|
||||||
|
- 若负载混合，可考虑不同 endpoint/队列按并发分桶，或测试 st=3/st=5 在本 grid 下的完整表现。
|
||||||
|
|
||||||
|
### 3.2 长上下文场景优化
|
||||||
|
|
||||||
|
针对 `long_rag`（32K）和 `long_context_probe`（16K）的吞吐断崖：
|
||||||
|
|
||||||
|
1. **KV cache dtype 对比测试**：当前是 FP8，可测试 BF16 是否改善长上下文稳定性和吞吐。
|
||||||
|
2. **block-size 调整**：当前 256，可尝试 128 或 64 看是否减少内存碎片。
|
||||||
|
3. **hybrid KV cache manager**：当前 `--no-disable-hybrid-kv-cache-manager`，可对比关闭后的表现。
|
||||||
|
4. **限制长上下文并发**：若业务允许，对 32K 输入设置更低的 `max-num-seqs` 或独立队列，避免 c=8 后的内存带宽瓶颈。
|
||||||
|
5. **attention backend**：可测试 `FLASHINFER_MLA_SPARSE_DSV4`（脚本 `start_dsv4_dspark_8card_flashinfer.sh`）在长上下文下的表现。
|
||||||
|
|
||||||
|
### 3.3 降低延迟长尾
|
||||||
|
|
||||||
|
1. **增加 warmup**：当前仅 10 条 warmup。DSpark 的 JIT 编译/算子 warmup 在首次运行时较重，建议增加到 50~100 条或先跑一轮 discard。
|
||||||
|
2. **compilation cache 持久化**：确认 `TORCH_EXTENSIONS_DIR`、`VLLM_CACHE_ROOT` 等环境变量已设置并复用，避免每次重启重新编译 deep_gemm/tilelang。
|
||||||
|
3. **调度参数**：尝试调整 `--max-num-batched-tokens`、`--max-num-seqs`、chunked prefill 相关参数，减少队列等待。
|
||||||
|
4. **P99 TTFT 抖动**：高并发下 P99 TTFT 高通常与 prefill batching 有关，可尝试限制 prefill 阶段 batch 大小或启用 prompt chunked prefill。
|
||||||
|
|
||||||
|
### 3.4 低并发场景是否需要 DSpark
|
||||||
|
|
||||||
|
对于 `chat_standard` c=1：
|
||||||
|
- 当前 0.49 req/s、P99 E2E 10.7s 的服务质量较差。
|
||||||
|
- 若业务存在大量单用户/低并发请求，可考虑：
|
||||||
|
- 部署无投机基线 vLLM 专门服务低并发流量；
|
||||||
|
- 或设置最小 batch 阈值，累积少量请求后再用 DSpark 处理。
|
||||||
|
|
||||||
|
### 3.5 对比 baseline 与 SGLang
|
||||||
|
|
||||||
|
本次测试只有 DSpark 单一配置，建议补充：
|
||||||
|
1. **vLLM-dspark 无投机基线**：用同样的 `sglang.bench_serving --backend vllm` 测一遍相同 grid，计算真实加速比。
|
||||||
|
2. **SGLang EAGLE**：在相同 grid 下复测，验证 DSpark 在不同场景下的相对优势。
|
||||||
|
3. **不同 `--spec-tokens` 的完整 grid**：st=3 和 st=5 各跑一遍，绘制并发-吞吐曲线。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 推荐下一步实验
|
||||||
|
|
||||||
|
| 优先级 | 实验 | 预期收益 |
|
||||||
|
|---|---|---|
|
||||||
|
| P0 | 高并发改用 `--spec-tokens 3` | 提升 c≥64 场景吞吐 20~50% |
|
||||||
|
| P0 | 长上下文（16K/32K）对比 BF16 KV cache | 改善吞吐断崖和 P99 TTFT |
|
||||||
|
| P1 | 增加 warmup / 预热缓存 | 降低 P99 TTFT 和首请求延迟 |
|
||||||
|
| P1 | 测试 `FLASHINFER_MLA_SPARSE_DSV4` backend | 可能提升长上下文吞吐 |
|
||||||
|
| P2 | 跑无投机基线 grid | 获得真实加速比 |
|
||||||
|
| P2 | 对比 st=3/st=5 完整曲线 | 找到最佳投机长度配置 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 结论
|
||||||
|
|
||||||
|
当前 DSpark 部署在 **短输入、中高并发** 场景下已经展现出明显优势，但在 **低并发** 和 **超长上下文** 场景下存在明显短板。最优先的优化是：
|
||||||
|
|
||||||
|
1. 高并发服务改用 `--spec-tokens 3`；
|
||||||
|
2. 针对 16K/32K 长上下文做 KV cache dtype 和 attention backend 的调优；
|
||||||
|
3. 补充 baseline 测试，量化 DSpark 的真实收益。
|
||||||
120
bench_results/dspark_grid_20260707-132641/report.md
Normal file
120
bench_results/dspark_grid_20260707-132641/report.md
Normal file
@ -0,0 +1,120 @@
|
|||||||
|
# vllm-dspark Benchmark Grid Report
|
||||||
|
|
||||||
|
- Result root: `/data/user1/yy/bench_results/dspark_grid_20260707-132641`
|
||||||
|
- Model: `/data/models/DeepSeek-V4-Flash-DSpark`
|
||||||
|
- Backend: vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark, spec-tokens=5)
|
||||||
|
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||||
|
|
||||||
|
## p1_quick
|
||||||
|
|
||||||
|
### chat_standard (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 1 | 65.33 | 32 | 0.49 | 195.58 | 71.28 | 266.86 | 2040.65 | 8241.20 | 10715.53 | 1412.48 | 6903.30 | 8430.71 | 6.41 | 32.72 | 51.29 | 16.61 |
|
||||||
|
| 8 | 29.42 | 128 | 4.35 | 2130.85 | 569.52 | 2700.37 | 1819.35 | 8891.79 | 15165.63 | 327.62 | 379.39 | 4404.92 | 11.58 | 51.06 | 82.44 | 51.00 |
|
||||||
|
| 16 | 30.76 | 256 | 8.32 | 4144.89 | 1144.74 | 5289.63 | 1885.15 | 8902.10 | 10491.31 | 443.50 | 2779.49 | 8567.07 | 12.10 | 23.11 | 83.70 | 46.18 |
|
||||||
|
| 32 | 51.18 | 512 | 10.00 | 5017.88 | 1331.65 | 6349.53 | 3156.73 | 10438.92 | 12936.08 | 560.77 | 5750.71 | 7129.66 | 22.09 | 66.23 | 171.57 | 85.81 |
|
||||||
|
|
||||||
|
### generation_standard (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 1 | 36.24 | 32 | 0.88 | 352.60 | 479.25 | 831.85 | 1131.63 | 1979.98 | 2576.42 | 66.57 | 141.80 | 177.56 | 1.94 | 2.85 | 2.88 | 8.50 |
|
||||||
|
| 8 | 27.43 | 128 | 4.67 | 2285.21 | 2362.35 | 4647.56 | 1649.16 | 3162.33 | 3844.70 | 80.88 | 157.80 | 160.50 | 3.23 | 5.31 | 7.09 | 14.77 |
|
||||||
|
| 16 | 35.39 | 256 | 7.23 | 3602.52 | 3654.79 | 7257.31 | 2133.90 | 4187.67 | 5500.20 | 88.78 | 169.39 | 199.78 | 4.37 | 6.97 | 11.81 | 19.28 |
|
||||||
|
| 32 | 48.16 | 512 | 10.63 | 5333.41 | 5371.85 | 10705.25 | 2908.89 | 5849.64 | 6938.03 | 112.84 | 211.64 | 405.78 | 5.84 | 9.30 | 12.47 | 25.89 |
|
||||||
|
|
||||||
|
### summarization (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 1 | 33.63 | 32 | 0.95 | 4247.40 | 516.49 | 4763.90 | 1049.88 | 1783.33 | 1848.92 | 178.56 | 292.66 | 314.18 | 1.60 | 1.78 | 2.41 | 8.62 |
|
||||||
|
| 4 | 10.99 | 32 | 2.91 | 13000.06 | 1580.84 | 14580.90 | 1301.15 | 2238.30 | 2324.70 | 101.68 | 187.78 | 188.57 | 2.18 | 2.47 | 3.49 | 11.86 |
|
||||||
|
| 8 | 33.60 | 128 | 3.81 | 15369.23 | 1908.96 | 17278.19 | 2072.22 | 3978.11 | 4531.26 | 214.01 | 502.25 | 630.15 | 3.70 | 5.52 | 7.98 | 19.60 |
|
||||||
|
| 16 | 46.93 | 256 | 5.45 | 22698.72 | 2761.38 | 25460.10 | 2840.57 | 6243.39 | 7179.11 | 235.75 | 552.81 | 740.45 | 5.45 | 9.62 | 12.75 | 27.77 |
|
||||||
|
|
||||||
|
## p2_core
|
||||||
|
|
||||||
|
### chat_short (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 1 | 11.39 | 32 | 2.81 | 717.30 | 408.87 | 1126.18 | 355.00 | 685.00 | 815.55 | 48.57 | 105.64 | 105.97 | 2.07 | 3.10 | 3.69 | 8.51 |
|
||||||
|
| 8 | 10.62 | 128 | 12.06 | 3320.19 | 1591.90 | 4912.09 | 647.05 | 1401.33 | 1648.15 | 78.91 | 151.78 | 155.67 | 4.59 | 7.20 | 12.54 | 17.85 |
|
||||||
|
| 16 | 14.89 | 256 | 17.19 | 4533.71 | 2394.66 | 6928.36 | 902.72 | 1879.32 | 2211.23 | 95.95 | 165.09 | 217.14 | 6.19 | 10.26 | 15.12 | 23.65 |
|
||||||
|
| 32 | 24.90 | 512 | 20.56 | 5458.26 | 2723.85 | 8182.10 | 1508.89 | 3076.09 | 3881.52 | 161.04 | 292.00 | 383.41 | 10.80 | 18.19 | 25.45 | 41.86 |
|
||||||
|
| 64 | 21.27 | 512 | 24.07 | 6391.14 | 3189.39 | 9580.52 | 2547.81 | 5606.07 | 6543.97 | 268.68 | 543.21 | 588.63 | 18.24 | 30.39 | 39.57 | 70.29 |
|
||||||
|
|
||||||
|
### chat_standard (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 64 | 19.87 | 512 | 25.76 | 12922.80 | 3429.47 | 16352.28 | 2380.94 | 4838.43 | 6315.98 | 291.17 | 794.65 | 947.26 | 16.87 | 30.96 | 42.55 | 68.46 |
|
||||||
|
|
||||||
|
### generation_standard (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 64 | 43.59 | 512 | 11.75 | 5892.47 | 5934.93 | 11827.40 | 5136.14 | 10244.70 | 12575.20 | 209.70 | 536.96 | 606.58 | 10.33 | 17.41 | 21.20 | 45.47 |
|
||||||
|
|
||||||
|
### long_context_probe (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 1 | 23.79 | 32 | 1.34 | 10684.78 | 366.30 | 11051.09 | 742.62 | 1237.28 | 1418.82 | 316.59 | 604.23 | 617.85 | 1.54 | 1.72 | 1.85 | 8.53 |
|
||||||
|
| 2 | 9.79 | 32 | 3.27 | 25971.67 | 890.38 | 26862.06 | 604.82 | 1055.84 | 1116.41 | 110.65 | 194.62 | 206.64 | 1.77 | 2.04 | 2.13 | 9.90 |
|
||||||
|
| 4 | 6.11 | 32 | 5.24 | 41616.05 | 1426.72 | 43042.77 | 741.94 | 1262.13 | 1447.68 | 127.38 | 222.70 | 235.35 | 2.26 | 2.69 | 3.69 | 12.39 |
|
||||||
|
| 8 | 36.07 | 128 | 3.55 | 28357.82 | 897.65 | 29255.46 | 2232.38 | 4946.97 | 5371.22 | 453.93 | 1089.99 | 1182.47 | 7.06 | 13.03 | 20.03 | 38.51 |
|
||||||
|
| 16 | 49.08 | 256 | 5.22 | 41910.25 | 1382.43 | 43292.68 | 3028.26 | 7651.28 | 9531.10 | 444.14 | 1212.92 | 1368.04 | 10.00 | 20.71 | 30.74 | 53.07 |
|
||||||
|
|
||||||
|
### rag_medium (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 1 | 18.21 | 32 | 1.76 | 3568.86 | 494.63 | 4063.49 | 568.24 | 966.60 | 1201.99 | 110.94 | 151.11 | 154.78 | 1.63 | 1.95 | 2.50 | 8.60 |
|
||||||
|
| 4 | 6.31 | 32 | 5.07 | 10300.85 | 1427.65 | 11728.51 | 751.19 | 1259.07 | 1593.86 | 92.09 | 161.03 | 187.47 | 2.34 | 3.25 | 4.02 | 12.33 |
|
||||||
|
| 8 | 19.13 | 128 | 6.69 | 14147.80 | 1692.32 | 15840.12 | 1166.59 | 2199.67 | 2982.23 | 148.09 | 253.10 | 350.67 | 4.30 | 6.78 | 9.92 | 21.06 |
|
||||||
|
| 16 | 25.89 | 256 | 9.89 | 20561.53 | 2622.36 | 23183.89 | 1587.09 | 3316.74 | 4019.02 | 164.29 | 303.79 | 389.34 | 5.50 | 8.80 | 14.38 | 27.81 |
|
||||||
|
| 32 | 39.47 | 512 | 12.97 | 26523.95 | 3335.30 | 29859.24 | 2415.10 | 5069.76 | 6195.14 | 227.81 | 444.76 | 510.18 | 8.64 | 13.54 | 20.29 | 44.49 |
|
||||||
|
|
||||||
|
### summarization (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 32 | 91.68 | 512 | 5.58 | 22445.97 | 2821.71 | 25267.67 | 5587.21 | 11177.97 | 12257.18 | 430.38 | 967.32 | 2238.28 | 10.56 | 16.13 | 23.53 | 54.40 |
|
||||||
|
|
||||||
|
## p3_extension
|
||||||
|
|
||||||
|
### decode_heavy (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 1 | 56.16 | 32 | 0.57 | 145.47 | 546.26 | 691.72 | 1754.17 | 2994.25 | 3175.70 | 52.30 | 107.66 | 108.83 | 1.88 | 2.68 | 3.14 | 8.53 |
|
||||||
|
| 8 | 54.33 | 128 | 2.36 | 648.72 | 2526.47 | 3175.19 | 3302.64 | 6408.14 | 7166.98 | 74.21 | 143.24 | 149.17 | 3.05 | 4.37 | 4.79 | 14.12 |
|
||||||
|
| 16 | 65.74 | 256 | 3.89 | 1026.90 | 4012.07 | 5038.96 | 3934.72 | 7216.03 | 9638.96 | 81.02 | 165.95 | 197.11 | 3.98 | 5.82 | 7.21 | 17.79 |
|
||||||
|
| 32 | 92.36 | 512 | 5.54 | 1471.70 | 5761.81 | 7233.51 | 5562.57 | 10498.47 | 12822.35 | 107.17 | 234.99 | 426.35 | 5.40 | 8.14 | 9.03 | 24.38 |
|
||||||
|
|
||||||
|
### long_rag (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 1 | 30.84 | 32 | 1.04 | 15823.63 | 281.34 | 16104.98 | 962.70 | 1688.61 | 1788.29 | 549.72 | 1124.54 | 1163.96 | 1.49 | 1.72 | 1.82 | 8.52 |
|
||||||
|
| 2 | 11.49 | 32 | 2.78 | 42460.67 | 754.95 | 43215.62 | 708.16 | 1202.62 | 1253.12 | 192.57 | 352.96 | 409.32 | 2.05 | 2.37 | 6.62 | 10.76 |
|
||||||
|
| 4 | 7.09 | 32 | 4.51 | 68816.98 | 1223.56 | 70040.54 | 854.49 | 1516.80 | 1562.42 | 191.57 | 426.45 | 427.69 | 2.62 | 3.20 | 7.33 | 13.69 |
|
||||||
|
| 8 | 60.48 | 128 | 2.12 | 33604.85 | 536.85 | 34141.70 | 3732.80 | 8420.06 | 9705.36 | 930.98 | 2125.14 | 2742.48 | 11.12 | 22.70 | 39.17 | 61.41 |
|
||||||
|
|
||||||
|
### stress_generation (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 96 | 81.20 | 768 | 9.46 | 4842.45 | 4635.63 | 9478.08 | 9772.15 | 20945.93 | 25314.28 | 326.63 | 782.50 | 795.30 | 20.35 | 33.33 | 42.50 | 88.91 |
|
||||||
|
| 128 | 92.26 | 1024 | 11.10 | 5587.83 | 5514.57 | 11102.40 | 11012.89 | 22012.44 | 28276.20 | 360.94 | 812.12 | 1024.93 | 22.53 | 35.59 | 42.09 | 99.75 |
|
||||||
|
|
||||||
|
### stress_standard (20260707-132641)
|
||||||
|
|
||||||
|
| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |
|
||||||
|
|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|
|
||||||
|
| 96 | 30.50 | 768 | 25.18 | 12891.05 | 3175.02 | 16066.07 | 3689.97 | 9383.09 | 11958.09 | 435.22 | 1225.11 | 2581.90 | 29.46 | 67.30 | 123.42 | 113.64 |
|
||||||
|
| 128 | 36.86 | 1024 | 27.78 | 13985.42 | 3479.38 | 17464.80 | 4417.43 | 11825.67 | 14850.43 | 456.55 | 1032.19 | 1922.87 | 35.20 | 76.85 | 141.17 | 136.57 |
|
||||||
|
|
||||||
@ -0,0 +1,129 @@
|
|||||||
|
# vllm-dspark `--spec-tokens` 对比报告
|
||||||
|
|
||||||
|
- 结果目录：`/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649`
|
||||||
|
- 模型：`/data/models/DeepSeek-V4-Flash-DSpark`
|
||||||
|
- 后端：vllm-dspark (TP=8, FP8 KV cache)
|
||||||
|
- 对比参数：`--spec-tokens 3` vs `--spec-tokens 5`
|
||||||
|
- Warmup：100 条
|
||||||
|
- 压测客户端：`sglang.bench_serving --backend vllm`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 核心指标对比
|
||||||
|
|
||||||
|
| Scenario | Concurrency | Spec | Duration(s) | Req/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| chat_short | 1 | 3 | 12.80 | 2.50 | 363.90 | 1002.30 | 398.97 | 757.11 | 948.62 | 43.97 | 68.89 | 2.42 | 3.76 |
|
||||||
|
| chat_short | 1 | 5 | 11.72 | 2.73 | 397.37 | 1094.51 | 365.29 | 743.89 | 782.98 | 64.42 | 174.36 | 2.06 | 3.40 |
|
||||||
|
| chat_short | 8 | 3 | 11.08 | 11.55 | 1525.06 | 4705.85 | 672.66 | 1269.82 | 1552.65 | 76.20 | 221.18 | 4.67 | 8.43 |
|
||||||
|
| chat_short | 8 | 5 | 11.78 | 10.87 | 1434.62 | 4426.78 | 718.40 | 1511.14 | 2008.00 | 94.97 | 220.85 | 4.95 | 9.93 |
|
||||||
|
| chat_short | 16 | 3 | 21.82 | 11.73 | 1634.03 | 4727.68 | 1333.99 | 2611.76 | 2921.26 | 151.65 | 337.53 | 8.85 | 17.66 |
|
||||||
|
| chat_short | 16 | 5 | 16.42 | 15.59 | 2171.58 | 6282.94 | 997.37 | 2147.86 | 2496.56 | 107.56 | 240.46 | 7.09 | 21.93 |
|
||||||
|
| chat_short | 32 | 3 | 24.62 | 20.79 | 2754.96 | 8275.55 | 1497.40 | 2944.48 | 3658.90 | 144.47 | 453.29 | 10.72 | 23.97 |
|
||||||
|
| chat_short | 32 | 5 | 26.32 | 19.45 | 2577.42 | 7742.25 | 1600.40 | 3403.00 | 4521.14 | 169.13 | 402.36 | 11.39 | 27.82 |
|
||||||
|
| chat_short | 64 | 3 | 21.66 | 23.63 | 3131.20 | 9405.73 | 2595.90 | 5230.98 | 6258.25 | 251.84 | 509.86 | 18.60 | 40.84 |
|
||||||
|
| chat_short | 64 | 5 | 21.26 | 24.09 | 3191.43 | 9586.66 | 2555.12 | 5387.59 | 6559.69 | 274.71 | 610.40 | 18.47 | 42.08 |
|
||||||
|
| chat_standard | 64 | 3 | 20.73 | 24.70 | 3288.58 | 15680.46 | 2483.46 | 4798.63 | 5693.88 | 271.41 | 947.15 | 17.93 | 46.05 |
|
||||||
|
| chat_standard | 64 | 5 | 20.91 | 24.49 | 3259.71 | 15542.80 | 2512.52 | 5394.43 | 6023.55 | 300.81 | 932.60 | 17.83 | 44.47 |
|
||||||
|
| decode_heavy | 1 | 3 | 68.78 | 0.47 | 446.03 | 564.81 | 2148.54 | 3712.03 | 3835.51 | 38.60 | 41.18 | 2.28 | 3.03 |
|
||||||
|
| decode_heavy | 1 | 5 | 57.70 | 0.55 | 531.76 | 673.37 | 1802.00 | 2957.04 | 3676.66 | 59.06 | 114.78 | 1.92 | 3.03 |
|
||||||
|
| decode_heavy | 32 | 3 | 96.14 | 5.33 | 5535.64 | 6949.57 | 5809.29 | 10830.76 | 12546.55 | 86.83 | 382.71 | 5.62 | 8.11 |
|
||||||
|
| decode_heavy | 32 | 5 | 92.75 | 5.52 | 5737.73 | 7203.28 | 5595.02 | 10412.11 | 12522.00 | 107.12 | 331.08 | 5.43 | 9.15 |
|
||||||
|
| generation_standard | 64 | 3 | 38.58 | 13.27 | 6705.88 | 13363.77 | 4542.22 | 8273.38 | 9355.10 | 190.34 | 813.58 | 9.00 | 15.36 |
|
||||||
|
| generation_standard | 64 | 5 | 43.39 | 11.80 | 5961.40 | 11880.13 | 5075.77 | 9863.55 | 11533.47 | 216.87 | 626.93 | 10.09 | 20.90 |
|
||||||
|
| long_context_probe | 1 | 3 | 26.10 | 1.23 | 333.92 | 10074.03 | 815.18 | 1382.34 | 1556.00 | 265.28 | 558.73 | 2.02 | 2.19 |
|
||||||
|
| long_context_probe | 1 | 5 | 22.22 | 1.44 | 392.31 | 11835.70 | 693.30 | 1172.84 | 1404.17 | 267.61 | 561.65 | 1.53 | 1.88 |
|
||||||
|
| long_context_probe | 4 | 3 | 7.34 | 4.36 | 1187.03 | 35811.63 | 888.88 | 1550.70 | 1611.29 | 125.51 | 282.82 | 2.77 | 3.23 |
|
||||||
|
| long_context_probe | 4 | 5 | 6.59 | 4.85 | 1322.32 | 39893.35 | 806.55 | 1390.87 | 1500.22 | 143.42 | 236.41 | 2.41 | 3.54 |
|
||||||
|
| long_context_probe | 8 | 3 | 38.61 | 3.31 | 838.49 | 27327.58 | 2387.05 | 5028.60 | 5667.97 | 406.10 | 1093.54 | 7.74 | 17.09 |
|
||||||
|
| long_context_probe | 8 | 5 | 35.94 | 3.56 | 900.89 | 29361.32 | 2223.29 | 4693.10 | 5345.72 | 425.03 | 1241.57 | 7.08 | 19.83 |
|
||||||
|
| rag_medium | 1 | 3 | 22.31 | 1.43 | 403.86 | 3317.84 | 696.09 | 1165.77 | 1315.00 | 105.51 | 157.60 | 2.10 | 2.81 |
|
||||||
|
| rag_medium | 1 | 5 | 18.16 | 1.76 | 496.09 | 4075.47 | 566.58 | 968.90 | 1134.15 | 110.34 | 156.35 | 1.64 | 2.34 |
|
||||||
|
| rag_medium | 8 | 3 | 20.57 | 6.22 | 1573.71 | 14730.01 | 1251.63 | 2472.54 | 2796.12 | 127.54 | 309.65 | 4.46 | 7.41 |
|
||||||
|
| rag_medium | 8 | 5 | 19.13 | 6.69 | 1692.13 | 15838.39 | 1175.26 | 2263.59 | 3106.42 | 154.24 | 352.19 | 4.14 | 8.36 |
|
||||||
|
| rag_medium | 32 | 3 | 45.02 | 11.37 | 2923.96 | 26176.72 | 2756.39 | 5579.00 | 6675.25 | 215.15 | 477.20 | 10.00 | 19.84 |
|
||||||
|
| rag_medium | 32 | 5 | 42.98 | 11.91 | 3062.92 | 27420.79 | 2626.21 | 5182.78 | 6873.72 | 253.09 | 522.80 | 9.17 | 18.55 |
|
||||||
|
| stress_standard | 64 | 3 | 21.35 | 23.99 | 3193.19 | 15225.66 | 2569.66 | 5006.16 | 5831.81 | 261.98 | 710.57 | 18.02 | 36.01 |
|
||||||
|
| stress_standard | 64 | 5 | 21.75 | 23.54 | 3133.74 | 14942.17 | 2621.52 | 5431.08 | 7161.16 | 305.11 | 734.92 | 18.42 | 41.01 |
|
||||||
|
| stress_standard | 96 | 3 | 20.36 | 37.72 | 4756.72 | 24069.73 | 2433.73 | 4827.07 | 6001.29 | 272.61 | 749.99 | 18.04 | 38.52 |
|
||||||
|
| stress_standard | 96 | 5 | 25.33 | 30.32 | 3823.91 | 19349.55 | 3032.19 | 6272.09 | 8124.32 | 339.22 | 793.18 | 23.06 | 51.46 |
|
||||||
|
| stress_standard | 128 | 3 | 21.58 | 47.45 | 5942.48 | 29828.36 | 2581.42 | 5162.16 | 6222.01 | 298.69 | 970.61 | 19.09 | 40.93 |
|
||||||
|
| stress_standard | 128 | 5 | 28.36 | 36.11 | 4522.53 | 22700.92 | 3405.74 | 7240.13 | 9176.77 | 382.02 | 1087.53 | 25.74 | 57.83 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 吞吐 winner 统计
|
||||||
|
|
||||||
|
| Scenario | Concurrency | Winner (Total tok/s) | st=3 Total tok/s | st=5 Total tok/s | 提升 |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| chat_short | 1 | st=5 | 1002.30 | 1094.51 | 9.20% |
|
||||||
|
| chat_short | 8 | st=3 | 4705.85 | 4426.78 | 6.30% |
|
||||||
|
| chat_short | 16 | st=5 | 4727.68 | 6282.94 | 32.90% |
|
||||||
|
| chat_short | 32 | st=3 | 8275.55 | 7742.25 | 6.89% |
|
||||||
|
| chat_short | 64 | st=5 | 9405.73 | 9586.66 | 1.92% |
|
||||||
|
| chat_standard | 64 | st=3 | 15680.46 | 15542.80 | 0.89% |
|
||||||
|
| decode_heavy | 1 | st=5 | 564.81 | 673.37 | 19.22% |
|
||||||
|
| decode_heavy | 32 | st=5 | 6949.57 | 7203.28 | 3.65% |
|
||||||
|
| generation_standard | 64 | st=3 | 13363.77 | 11880.13 | 12.49% |
|
||||||
|
| long_context_probe | 1 | st=5 | 10074.03 | 11835.70 | 17.49% |
|
||||||
|
| long_context_probe | 4 | st=5 | 35811.63 | 39893.35 | 11.40% |
|
||||||
|
| long_context_probe | 8 | st=5 | 27327.58 | 29361.32 | 7.44% |
|
||||||
|
| rag_medium | 1 | st=5 | 3317.84 | 4075.47 | 22.84% |
|
||||||
|
| rag_medium | 8 | st=5 | 14730.01 | 15838.39 | 7.52% |
|
||||||
|
| rag_medium | 32 | st=5 | 26176.72 | 27420.79 | 4.75% |
|
||||||
|
| stress_standard | 64 | st=3 | 15225.66 | 14942.17 | 1.90% |
|
||||||
|
| stress_standard | 96 | st=3 | 24069.73 | 19349.55 | 24.39% |
|
||||||
|
| stress_standard | 128 | st=3 | 29828.36 | 22700.92 | 31.40% |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 关键发现
|
||||||
|
|
||||||
|
### 1. 并发是决定性因素
|
||||||
|
|
||||||
|
- **低并发（c=1）和中低并发（c=8~32）**：`st=5` 在多数场景下更优，尤其是长上下文和重 decode 场景。
|
||||||
|
- **高并发（c≥96）**：`st=3` 明显更优，`stress_standard` c=128 时 st=3 比 st=5 高 **31.4%**。
|
||||||
|
- **中并发（c=64）**：两者基本持平，差异多在 2% 以内。
|
||||||
|
|
||||||
|
### 2. st=5 更适合长上下文和重 decode
|
||||||
|
|
||||||
|
| 场景 | 最佳 spec | 原因 |
|
||||||
|
|---|---|---|
|
||||||
|
| long_context_probe (16K) | st=5 | 长 prefill 下 st=5 的接受长度更高 |
|
||||||
|
| rag_medium (4K) | st=5 | 中长输入下 st=5 延迟和吞吐都更优 |
|
||||||
|
| decode_heavy (2K output) | st=5 | 输出越长，st=5 的投机收益越大 |
|
||||||
|
|
||||||
|
### 3. st=3 在极限并发下更稳
|
||||||
|
|
||||||
|
`stress_standard` 随并发增加，st=3 与 st=5 的差距拉大：
|
||||||
|
|
||||||
|
| 并发 | st=3 Total tok/s | st=5 Total tok/s | 差距 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| 64 | 15225.66 | 14942.17 | 基本持平 |
|
||||||
|
| 96 | 24069.73 | 19349.55 | st=3 高 24.4% |
|
||||||
|
| 128 | 29828.36 | 22700.92 | st=3 高 31.4% |
|
||||||
|
|
||||||
|
这说明 st=5 在极限并发下验证开销和 KV cache 压力显著增加，而 st=3 的验证 batch 更小、调度更稳定。
|
||||||
|
|
||||||
|
### 4. chat_short c=16 的异常
|
||||||
|
|
||||||
|
`chat_short c=16` 时 st=5 比 st=3 高 32.9%，是一个明显的 outlier。可能原因是该并发度下 st=5 的 draft 接受率和 batch 利用率恰好达到甜点。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 优化建议
|
||||||
|
|
||||||
|
### 推荐配置
|
||||||
|
|
||||||
|
| 负载特征 | 推荐 `--spec-tokens` | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| 低并发 / 在线交互（c ≤ 32） | **5** | 延迟低、单请求吞吐高 |
|
||||||
|
| 中并发（c ≈ 64） | 3 或 5 均可 | 差异很小 |
|
||||||
|
| 高并发 / 压测（c ≥ 96） | **3** | 吞吐更高、P99 更稳 |
|
||||||
|
| 长上下文 / RAG / 重 decode | **5** | 接受长度优势更明显 |
|
||||||
|
|
||||||
|
### 下一步
|
||||||
|
|
||||||
|
1. 如果业务以高并发为主，将默认服务改为 `--spec-tokens 3`。
|
||||||
|
2. 如果业务混合，可考虑按输入长度或并发度路由到不同服务实例。
|
||||||
|
3. 本次 warmup 已从 10 提升到 100，P99 TTFT 相比首次 grid 测试有明显改善；建议保持 100 条 warmup。
|
||||||
102
bench_results/dsv4_backend_comparison_20260707/README.md
Normal file
102
bench_results/dsv4_backend_comparison_20260707/README.md
Normal file
@ -0,0 +1,102 @@
|
|||||||
|
# DSV4 Backend Comparison Raw Outputs (2026-07-07)
|
||||||
|
|
||||||
|
## Location
|
||||||
|
|
||||||
|
`/data/user1/yy/bench_results/dsv4_backend_comparison_20260707`
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
- `raw_outputs/` — raw per-benchmark JSONL files produced by `sglang.bench_serving --output-file --output-details`.
|
||||||
|
- `README.md` — this file.
|
||||||
|
|
||||||
|
## File Naming Convention
|
||||||
|
|
||||||
|
Each file follows the pattern:
|
||||||
|
|
||||||
|
```
|
||||||
|
{backend}_0707_{concurrency}_{input_len}_{output_len}.jsonl
|
||||||
|
```
|
||||||
|
|
||||||
|
Where:
|
||||||
|
|
||||||
|
- `{backend}`: `sglang` or `vllm`
|
||||||
|
- `{concurrency}`: max concurrent requests (`max-concurrency`)
|
||||||
|
- `{input_len}`: `--random-input-len`
|
||||||
|
- `{output_len}`: `--random-output-len`
|
||||||
|
|
||||||
|
## Output File Inventory
|
||||||
|
|
||||||
|
### SGLang backend
|
||||||
|
|
||||||
|
| File | Concurrency | Input len | Output len |
|
||||||
|
|---|---|---|---|
|
||||||
|
| sglang_0707_32_512_256.jsonl | 32 | 512 | 256 |
|
||||||
|
| sglang_0707_128_512_256.jsonl | 128 | 512 | 256 |
|
||||||
|
| sglang_0707_256_512_256.jsonl | 256 | 512 | 256 |
|
||||||
|
| sglang_0707_512_512_256.jsonl | 512 | 512 | 256 |
|
||||||
|
| sglang_0707_512_512_2000.jsonl | 512 | 512 | 2000 |
|
||||||
|
| sglang_0707_32_512_2000.jsonl | 32 | 512 | 2000 |
|
||||||
|
| sglang_0707_32_4000_512.jsonl | 32 | 4000 | 512 |
|
||||||
|
| sglang_0707_128_4000_512.jsonl | 128 | 4000 | 512 |
|
||||||
|
| sglang_0707_512_4000_512.jsonl | 512 | 4000 | 512 |
|
||||||
|
| sglang_0707_32_16000_512.jsonl | 32 | 16000 | 512 |
|
||||||
|
| sglang_0707_128_16000_512.jsonl | 128 | 16000 | 512 |
|
||||||
|
| sglang_0707_512_1000_256.jsonl | 512 | 1000 | 256 |
|
||||||
|
| sglang_0707_768_1000_256.jsonl | 768 | 1000 | 256 |
|
||||||
|
| sglang_0707_1024_1000_256.jsonl | 1024 | 1000 | 256 |
|
||||||
|
| sglang_0707_512_1000_1000.jsonl | 512 | 1000 | 1000 |
|
||||||
|
|
||||||
|
### vLLM backend
|
||||||
|
|
||||||
|
| File | Concurrency | Input len | Output len |
|
||||||
|
|---|---|---|---|
|
||||||
|
| vllm_0707_32_512_256.jsonl | 32 | 512 | 256 |
|
||||||
|
| vllm_0707_128_512_256.jsonl | 128 | 512 | 256 |
|
||||||
|
| vllm_0707_256_512_256.jsonl | 256 | 512 | 256 |
|
||||||
|
| vllm_0707_128_512_2000.jsonl | 128 | 512 | 2000 |
|
||||||
|
| vllm_0707_256_512_2000.jsonl | 256 | 512 | 2000 |
|
||||||
|
| vllm_0707_512_512_2000.jsonl | 512 | 512 | 2000 |
|
||||||
|
| vllm_0707_32_1000_256.jsonl | 32 | 1000 | 256 |
|
||||||
|
| vllm_0707_128_1000_256.jsonl | 128 | 1000 | 256 |
|
||||||
|
| vllm_0707_256_1000_256.jsonl | 256 | 1000 | 256 |
|
||||||
|
| vllm_0707_512_1000_256.jsonl | 512 | 1000 | 256 |
|
||||||
|
| vllm_0707_768_1000_256.jsonl | 768 | 1000 | 256 |
|
||||||
|
| vllm_0707_1024_1000_256.jsonl | 1024 | 1000 | 256 |
|
||||||
|
| vllm_0707_32_1000_1000.jsonl | 32 | 1000 | 1000 |
|
||||||
|
| vllm_0707_128_1000_1000.jsonl | 128 | 1000 | 1000 |
|
||||||
|
| vllm_0707_256_1000_1000.jsonl | 256 | 1000 | 1000 |
|
||||||
|
| vllm_0707_512_1000_1000.jsonl | 512 | 1000 | 1000 |
|
||||||
|
| vllm_0707_1024_1000_1000.jsonl | 1024 | 1000 | 1000 |
|
||||||
|
| vllm_0707_32_4000_512.jsonl | 32 | 4000 | 512 |
|
||||||
|
| vllm_0707_128_4000_512.jsonl | 128 | 4000 | 512 |
|
||||||
|
| vllm_0707_256_4000_512.jsonl | 256 | 4000 | 512 |
|
||||||
|
| vllm_0707_512_4000_512.jsonl | 512 | 4000 | 512 |
|
||||||
|
| vllm_0707_32_8000_1000.jsonl | 32 | 8000 | 1000 |
|
||||||
|
| vllm_0707_128_8000_1000.jsonl | 128 | 8000 | 1000 |
|
||||||
|
| vllm_0707_256_8000_1000.jsonl | 256 | 8000 | 1000 |
|
||||||
|
| vllm_0707_512_8000_1000.jsonl | 512 | 8000 | 1000 |
|
||||||
|
| vllm_0707_32_16000_512.jsonl | 32 | 16000 | 512 |
|
||||||
|
| vllm_0707_128_16000_512.jsonl | 128 | 16000 | 512 |
|
||||||
|
| vllm_0707_256_16000_512.jsonl | 256 | 16000 | 512 |
|
||||||
|
| vllm_0707_32_32000_512.jsonl | 32 | 32000 | 512 |
|
||||||
|
| vllm_0707_128_32000_512.jsonl | 128 | 32000 | 512 |
|
||||||
|
|
||||||
|
## Provenance
|
||||||
|
|
||||||
|
These files are **legacy raw outputs** from an ad-hoc SGLang vs vLLM backend comparison sweep run on 2026-07-07. The exact generator command/script that produced them was not preserved in the repository. They were generated using `sglang.bench_serving` with `--output-file` and `--output-details`, against:
|
||||||
|
|
||||||
|
- Model: `/data/models/DeepSeek-V4-Flash`
|
||||||
|
- SGLang port: `30000`
|
||||||
|
- vLLM port: `8000` (inferred from file content)
|
||||||
|
- Dataset: `random`
|
||||||
|
|
||||||
|
For future reproducible comparisons, use the script:
|
||||||
|
|
||||||
|
```
|
||||||
|
scripts/benchmark_dsv4_backend_comparison.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related Reports
|
||||||
|
|
||||||
|
- `/data/user1/yy/dsv4_inference_comparison_report.md` — earlier DSV4 inference comparison report.
|
||||||
|
- `/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/` — earlier comparison run outputs (`.json`/`.log`).
|
||||||
197
bench_results/dsv4_comparison_20260705_152221/summary.json
Normal file
197
bench_results/dsv4_comparison_20260705_152221/summary.json
Normal file
@ -0,0 +1,197 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st5",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 5,
|
||||||
|
"concurrency": 1,
|
||||||
|
"request_throughput": 1.0665689780673602,
|
||||||
|
"output_throughput": 257.1817776813825,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 48226,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st5_c1.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st5",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 5,
|
||||||
|
"concurrency": 16,
|
||||||
|
"request_throughput": 7.702621472216438,
|
||||||
|
"output_throughput": 1862.3398195524906,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 48356,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st5_c16.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st5",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 5,
|
||||||
|
"concurrency": 64,
|
||||||
|
"request_throughput": 9.360404619968996,
|
||||||
|
"output_throughput": 2210.4595510056783,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 47230,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st5_c64.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-nospec",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": null,
|
||||||
|
"spec_tokens": null,
|
||||||
|
"concurrency": 1,
|
||||||
|
"request_throughput": 0.5918258062472803,
|
||||||
|
"output_throughput": 144.62151314361665,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 48873,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-nospec_c1.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-nospec",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": null,
|
||||||
|
"spec_tokens": null,
|
||||||
|
"concurrency": 16,
|
||||||
|
"request_throughput": 5.61467992292336,
|
||||||
|
"output_throughput": 1364.0022670753865,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 48587,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-nospec_c16.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-nospec",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": null,
|
||||||
|
"spec_tokens": null,
|
||||||
|
"concurrency": 64,
|
||||||
|
"request_throughput": 7.339983457334795,
|
||||||
|
"output_throughput": 1803.9844342264591,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 49155,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-nospec_c64.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st3",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 3,
|
||||||
|
"concurrency": 1,
|
||||||
|
"request_throughput": 1.033805786989743,
|
||||||
|
"output_throughput": 254.38859000456605,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 49214,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st3_c1.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st3",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 3,
|
||||||
|
"concurrency": 16,
|
||||||
|
"request_throughput": 8.237095278328889,
|
||||||
|
"output_throughput": 1992.1826785402332,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 48371,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st3_c16.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st3",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 3,
|
||||||
|
"concurrency": 64,
|
||||||
|
"request_throughput": 14.530514004704537,
|
||||||
|
"output_throughput": 3507.8113858757224,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 48282,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st3_c64.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st7",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 7,
|
||||||
|
"concurrency": 1,
|
||||||
|
"request_throughput": 1.0285012524448527,
|
||||||
|
"output_throughput": 251.58169136053544,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 48922,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st7_c1.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st7",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 7,
|
||||||
|
"concurrency": 16,
|
||||||
|
"request_throughput": 7.463416900080192,
|
||||||
|
"output_throughput": 1772.3002941775428,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 47493,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st7_c16.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-dspark-dspark-st7",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 7,
|
||||||
|
"concurrency": 64,
|
||||||
|
"request_throughput": 9.128647309323574,
|
||||||
|
"output_throughput": 2218.5807988214547,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 48607,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st7_c64.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-main-nospec",
|
||||||
|
"engine": "vllm",
|
||||||
|
"spec_method": null,
|
||||||
|
"spec_tokens": null,
|
||||||
|
"concurrency": 1,
|
||||||
|
"request_throughput": 0.5806562443589017,
|
||||||
|
"output_throughput": 138.2571550630763,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 47621,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-main-nospec_c1.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-main-nospec",
|
||||||
|
"engine": "vllm",
|
||||||
|
"spec_method": null,
|
||||||
|
"spec_tokens": null,
|
||||||
|
"concurrency": 16,
|
||||||
|
"request_throughput": 5.052303923305438,
|
||||||
|
"output_throughput": 1180.976042072646,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 46750,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-main-nospec_c16.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"service": "vllm-main-nospec",
|
||||||
|
"engine": "vllm",
|
||||||
|
"spec_method": null,
|
||||||
|
"spec_tokens": null,
|
||||||
|
"concurrency": 64,
|
||||||
|
"request_throughput": 6.851644652089382,
|
||||||
|
"output_throughput": 1641.8938662034388,
|
||||||
|
"total_input_tokens": 47865,
|
||||||
|
"total_output_tokens": 47927,
|
||||||
|
"duration_s": null,
|
||||||
|
"result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-main-nospec_c64.json"
|
||||||
|
}
|
||||||
|
]
|
||||||
96
bench_results/eagle_grid/dspark_vs_eagle_report.md
Normal file
96
bench_results/eagle_grid/dspark_vs_eagle_report.md
Normal file
@ -0,0 +1,96 @@
|
|||||||
|
# DSpark vs SGLang EAGLE 投机解码对比报告
|
||||||
|
|
||||||
|
- DSpark 结果：`/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649`
|
||||||
|
- EAGLE 结果：`/data/user1/yy/bench_results/eagle_grid/focused`
|
||||||
|
- DSpark 模型：`/data/models/DeepSeek-V4-Flash-DSpark`
|
||||||
|
- EAGLE 模型：`/data/models/DeepSeek-V4-Flash`
|
||||||
|
- DSpark 后端：vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark)
|
||||||
|
- EAGLE 后端：SGLang (TP=8, FP8 KV cache, speculative-algorithm=EAGLE, num_steps=3, topk=1, draft_tokens=4)
|
||||||
|
- 压测客户端：`sglang.bench_serving`
|
||||||
|
- Warmup：100 条
|
||||||
|
|
||||||
|
## 核心指标对比
|
||||||
|
|
||||||
|
| Scenario | Concurrency | Method | Total tok/s | Out tok/s | Req/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| chat_short | 1 | DSpark(st=5) | 1094.51 | 397.37 | 2.73 | 365.29 | 743.89 | 782.98 | 64.42 | 174.36 | 2.06 | 3.40 |
|
||||||
|
| chat_short | 1 | EAGLE | 272.82 | 99.05 | 0.68 | 1467.67 | 3775.96 | 5175.09 | 223.86 | 405.20 | 7.47 | 19.74 |
|
||||||
|
| chat_short | 8 | DSpark(st=3) | 4705.85 | 1525.06 | 11.55 | 672.66 | 1269.82 | 1552.65 | 76.20 | 221.18 | 4.67 | 8.43 |
|
||||||
|
| chat_short | 8 | EAGLE | 1059.99 | 343.52 | 2.60 | 3043.16 | 7143.57 | 8345.08 | 245.91 | 528.86 | 21.41 | 36.02 |
|
||||||
|
| chat_short | 16 | DSpark(st=5) | 6282.94 | 2171.58 | 15.59 | 997.37 | 2147.86 | 2496.56 | 107.56 | 240.46 | 7.09 | 21.93 |
|
||||||
|
| chat_short | 16 | EAGLE | 1506.42 | 520.67 | 3.74 | 4240.11 | 8031.57 | 10192.48 | 236.19 | 438.56 | 29.12 | 54.02 |
|
||||||
|
| chat_short | 32 | DSpark(st=3) | 8275.55 | 2754.96 | 20.79 | 1497.40 | 2944.48 | 3658.90 | 144.47 | 453.29 | 10.72 | 23.97 |
|
||||||
|
| chat_short | 32 | EAGLE | 2461.23 | 819.35 | 6.18 | 4731.11 | 9926.81 | 11388.85 | 243.84 | 571.89 | 34.22 | 54.92 |
|
||||||
|
| chat_short | 64 | DSpark(st=5) | 9586.66 | 3191.43 | 24.09 | 2555.12 | 5387.59 | 6559.69 | 274.71 | 610.40 | 18.47 | 42.08 |
|
||||||
|
| chat_short | 64 | EAGLE | 4474.88 | 1489.70 | 11.24 | 5239.80 | 10296.02 | 11922.06 | 268.27 | 452.80 | 38.17 | 61.24 |
|
||||||
|
| chat_standard | 64 | DSpark(st=3) | 15680.46 | 3288.58 | 24.70 | 2483.46 | 4798.63 | 5693.88 | 271.41 | 947.15 | 17.93 | 46.05 |
|
||||||
|
| chat_standard | 64 | EAGLE | 6746.24 | 1414.85 | 10.63 | 5637.63 | 10693.92 | 12907.15 | 296.86 | 597.18 | 41.12 | 69.08 |
|
||||||
|
| decode_heavy | 1 | DSpark(st=5) | 673.37 | 531.76 | 0.55 | 1802.00 | 2957.04 | 3676.66 | 59.06 | 114.78 | 1.92 | 3.03 |
|
||||||
|
| decode_heavy | 1 | EAGLE | 261.35 | 206.39 | 0.22 | 4643.62 | 10452.02 | 18810.32 | 162.91 | 314.71 | 4.84 | 11.69 |
|
||||||
|
| decode_heavy | 32 | DSpark(st=5) | 7203.28 | 5737.73 | 5.52 | 5595.02 | 10412.11 | 12522.00 | 107.12 | 331.08 | 5.43 | 9.15 |
|
||||||
|
| decode_heavy | 32 | EAGLE | 2981.50 | 2374.90 | 2.28 | 13707.03 | 27159.67 | 30638.81 | 192.78 | 442.78 | 13.23 | 19.26 |
|
||||||
|
| generation_standard | 64 | DSpark(st=3) | 13363.77 | 6705.88 | 13.27 | 4542.22 | 8273.38 | 9355.10 | 190.34 | 813.58 | 9.00 | 15.36 |
|
||||||
|
| generation_standard | 64 | EAGLE | 5609.33 | 2814.73 | 5.57 | 11057.53 | 22577.31 | 26713.41 | 275.48 | 944.52 | 21.88 | 34.80 |
|
||||||
|
| long_context_probe | 1 | DSpark(st=5) | 11835.70 | 392.31 | 1.44 | 693.30 | 1172.84 | 1404.17 | 267.61 | 561.65 | 1.53 | 1.88 |
|
||||||
|
| long_context_probe | 1 | EAGLE | 5556.95 | 184.19 | 0.68 | 1477.16 | 2792.68 | 3836.17 | 238.26 | 618.74 | 4.43 | 8.62 |
|
||||||
|
| long_context_probe | 4 | DSpark(st=5) | 39893.35 | 1322.32 | 4.85 | 806.55 | 1390.87 | 1500.22 | 143.42 | 236.41 | 2.41 | 3.54 |
|
||||||
|
| long_context_probe | 4 | EAGLE | 6660.56 | 220.77 | 0.81 | 4777.51 | 9442.45 | 10071.75 | 433.60 | 1230.20 | 14.85 | 21.65 |
|
||||||
|
| long_context_probe | 8 | DSpark(st=5) | 29361.32 | 900.89 | 3.56 | 2223.29 | 4693.10 | 5345.72 | 425.03 | 1241.57 | 7.08 | 19.83 |
|
||||||
|
| long_context_probe | 8 | EAGLE | 9890.47 | 303.47 | 1.20 | 6232.13 | 16324.27 | 20329.21 | 606.76 | 2671.06 | 22.79 | 54.38 |
|
||||||
|
| rag_medium | 1 | DSpark(st=5) | 4075.47 | 496.09 | 1.76 | 566.58 | 968.90 | 1134.15 | 110.34 | 156.35 | 1.64 | 2.34 |
|
||||||
|
| rag_medium | 1 | EAGLE | 1326.02 | 161.41 | 0.57 | 1742.43 | 3804.17 | 3936.47 | 195.07 | 337.55 | 5.83 | 12.84 |
|
||||||
|
| rag_medium | 8 | DSpark(st=5) | 15838.39 | 1692.13 | 6.69 | 1175.26 | 2263.59 | 3106.42 | 154.24 | 352.19 | 4.14 | 8.36 |
|
||||||
|
| rag_medium | 8 | EAGLE | 3676.32 | 392.77 | 1.55 | 4908.89 | 11659.24 | 12566.81 | 254.65 | 678.44 | 18.53 | 35.36 |
|
||||||
|
| rag_medium | 32 | DSpark(st=5) | 27420.79 | 3062.92 | 11.91 | 2626.21 | 5182.78 | 6873.72 | 253.09 | 522.80 | 9.17 | 18.55 |
|
||||||
|
| rag_medium | 32 | EAGLE | 13268.07 | 1482.05 | 5.76 | 5444.98 | 10471.13 | 11990.37 | 337.62 | 2204.45 | 20.09 | 39.98 |
|
||||||
|
| stress_standard | 64 | DSpark(st=3) | 15225.66 | 3193.19 | 23.99 | 2569.66 | 5006.16 | 5831.81 | 261.98 | 710.57 | 18.02 | 36.01 |
|
||||||
|
| stress_standard | 64 | EAGLE | 5399.35 | 1132.38 | 8.51 | 6678.22 | 13086.21 | 14811.42 | 314.38 | 584.25 | 48.19 | 81.06 |
|
||||||
|
| stress_standard | 96 | DSpark(st=3) | 24069.73 | 4756.72 | 37.72 | 2433.73 | 4827.07 | 6001.29 | 272.61 | 749.99 | 18.04 | 38.52 |
|
||||||
|
| stress_standard | 96 | EAGLE | 8833.26 | 1745.65 | 13.84 | 6792.34 | 14000.22 | 16722.55 | 360.14 | 959.71 | 52.90 | 96.70 |
|
||||||
|
| stress_standard | 128 | DSpark(st=3) | 29828.36 | 5942.48 | 47.45 | 2581.42 | 5162.16 | 6222.01 | 298.69 | 970.61 | 19.09 | 40.93 |
|
||||||
|
| stress_standard | 128 | EAGLE | 13189.80 | 2627.70 | 20.98 | 5960.86 | 12256.06 | 14219.75 | 416.12 | 2153.71 | 47.20 | 81.35 |
|
||||||
|
|
||||||
|
## 吞吐 Winner 统计（按 Total tok/s）
|
||||||
|
|
||||||
|
| Scenario | Concurrency | Winner | DSpark Total tok/s | EAGLE Total tok/s | 提升 |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| chat_short | 1 | DSpark(st=5) | 1094.51 | 272.82 | 301.18% |
|
||||||
|
| chat_short | 8 | DSpark(st=3) | 4705.85 | 1059.99 | 343.95% |
|
||||||
|
| chat_short | 16 | DSpark(st=5) | 6282.94 | 1506.42 | 317.08% |
|
||||||
|
| chat_short | 32 | DSpark(st=3) | 8275.55 | 2461.23 | 236.24% |
|
||||||
|
| chat_short | 64 | DSpark(st=5) | 9586.66 | 4474.88 | 114.23% |
|
||||||
|
| chat_standard | 64 | DSpark(st=3) | 15680.46 | 6746.24 | 132.43% |
|
||||||
|
| decode_heavy | 1 | DSpark(st=5) | 673.37 | 261.35 | 157.65% |
|
||||||
|
| decode_heavy | 32 | DSpark(st=5) | 7203.28 | 2981.50 | 141.60% |
|
||||||
|
| generation_standard | 64 | DSpark(st=3) | 13363.77 | 5609.33 | 138.24% |
|
||||||
|
| long_context_probe | 1 | DSpark(st=5) | 11835.70 | 5556.95 | 112.99% |
|
||||||
|
| long_context_probe | 4 | DSpark(st=5) | 39893.35 | 6660.56 | 498.95% |
|
||||||
|
| long_context_probe | 8 | DSpark(st=5) | 29361.32 | 9890.47 | 196.86% |
|
||||||
|
| rag_medium | 1 | DSpark(st=5) | 4075.47 | 1326.02 | 207.35% |
|
||||||
|
| rag_medium | 8 | DSpark(st=5) | 15838.39 | 3676.32 | 330.82% |
|
||||||
|
| rag_medium | 32 | DSpark(st=5) | 27420.79 | 13268.07 | 106.67% |
|
||||||
|
| stress_standard | 64 | DSpark(st=3) | 15225.66 | 5399.35 | 181.99% |
|
||||||
|
| stress_standard | 96 | DSpark(st=3) | 24069.73 | 8833.26 | 172.49% |
|
||||||
|
| stress_standard | 128 | DSpark(st=3) | 29828.36 | 13189.80 | 126.15% |
|
||||||
|
|
||||||
|
## 关键发现
|
||||||
|
|
||||||
|
- 对比点数：18
|
||||||
|
- EAGLE 胜出：0 个
|
||||||
|
- DSpark 胜出：18 个
|
||||||
|
|
||||||
|
**结论：在相同负载下，vllm-dspark 的综合吞吐优于 SGLang EAGLE。**
|
||||||
|
|
||||||
|
### 延迟与稳定性观察
|
||||||
|
|
||||||
|
- 平均 Mean E2E 比值（EAGLE / DSpark）：3.09
|
||||||
|
- 平均 Mean TPOT 比值（EAGLE / DSpark）：3.21
|
||||||
|
- 平均 Mean TTFT 比值（EAGLE / DSpark）：1.81
|
||||||
|
|
||||||
|
> 比值 < 1 表示 EAGLE 更快；> 1 表示 DSpark 更快。
|
||||||
|
|
||||||
|
## 优化建议
|
||||||
|
|
||||||
|
1. 如果以吞吐为首要目标，当前配置下 **vllm-dspark 更优**，建议继续使用 `--spec-tokens` 并根据并发选择 3（高并发）或 5（低并发/长上下文）。
|
||||||
|
2. SGLang EAGLE 本次表现不佳，平均延迟和 TPOT 均显著高于 DSpark；如需进一步评估 EAGLE，可尝试调整 `--speculative-num-steps`、`--speculative-eagle-topk`、`--speculative-num-draft-tokens` 或更换 MOE runner backend，并确认是否已完成 `sglang.compile_deep_gemm` 预编译。
|
||||||
|
3. 注意两套后端的实现差异（CUDA graph、KV cache 管理、调度器、draft 模型架构）会显著影响不同并发和输入长度下的表现，建议按实际业务负载做最终选型。
|
||||||
6
bench_results/eagle_grid/report.md
Normal file
6
bench_results/eagle_grid/report.md
Normal file
@ -0,0 +1,6 @@
|
|||||||
|
# vllm-dspark Benchmark Grid Report
|
||||||
|
|
||||||
|
- Result root: `/data/user1/yy/bench_results/eagle_grid`
|
||||||
|
- Model: `/data/models/DeepSeek-V4-Flash-DSpark`
|
||||||
|
- Backend: vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark, spec-tokens=5)
|
||||||
|
- Benchmark client: `sglang.bench_serving --backend vllm`
|
||||||
@ -0,0 +1,58 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_128",
|
||||||
|
"args": "--max-concurrency 128",
|
||||||
|
"duration_s": 155.26,
|
||||||
|
"request_throughput": 12.88,
|
||||||
|
"input_token_throughput": 3710.5,
|
||||||
|
"output_token_throughput": 2690.48,
|
||||||
|
"total_token_throughput": 6400.98,
|
||||||
|
"mean_ttft_ms": 410.55,
|
||||||
|
"mean_tpot_ms": 48.45,
|
||||||
|
"mean_e2e_latency_ms": 9696.87,
|
||||||
|
"accept_length": 2.6,
|
||||||
|
"output_file": "bench_results/sglang_8card_max_throughput_20260705_030839/sharegpt_concurrency_128.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_256",
|
||||||
|
"args": "--max-concurrency 256",
|
||||||
|
"duration_s": 90.65,
|
||||||
|
"request_throughput": 22.06,
|
||||||
|
"input_token_throughput": 6355.48,
|
||||||
|
"output_token_throughput": 4608.35,
|
||||||
|
"total_token_throughput": 10963.84,
|
||||||
|
"mean_ttft_ms": 509.73,
|
||||||
|
"mean_tpot_ms": 62.71,
|
||||||
|
"mean_e2e_latency_ms": 10869.76,
|
||||||
|
"accept_length": 2.6,
|
||||||
|
"output_file": "bench_results/sglang_8card_max_throughput_20260705_030839/sharegpt_concurrency_256.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_512",
|
||||||
|
"args": "--max-concurrency 512",
|
||||||
|
"duration_s": 71.39,
|
||||||
|
"request_throughput": 28.02,
|
||||||
|
"input_token_throughput": 8070.04,
|
||||||
|
"output_token_throughput": 5851.58,
|
||||||
|
"total_token_throughput": 13921.61,
|
||||||
|
"mean_ttft_ms": 1343.73,
|
||||||
|
"mean_tpot_ms": 121.2,
|
||||||
|
"mean_e2e_latency_ms": 16176.14,
|
||||||
|
"accept_length": 2.6,
|
||||||
|
"output_file": "bench_results/sglang_8card_max_throughput_20260705_030839/sharegpt_concurrency_512.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_1024",
|
||||||
|
"args": "--max-concurrency 1024",
|
||||||
|
"duration_s": 68.21,
|
||||||
|
"request_throughput": 29.32,
|
||||||
|
"input_token_throughput": 8445.53,
|
||||||
|
"output_token_throughput": 6123.84,
|
||||||
|
"total_token_throughput": 14569.37,
|
||||||
|
"mean_ttft_ms": 13265.46,
|
||||||
|
"mean_tpot_ms": 105.36,
|
||||||
|
"mean_e2e_latency_ms": 27566.99,
|
||||||
|
"accept_length": 2.6,
|
||||||
|
"output_file": "bench_results/sglang_8card_max_throughput_20260705_030839/sharegpt_concurrency_1024.json"
|
||||||
|
}
|
||||||
|
]
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 147 KiB |
267
bench_results/sglang_8card_systematic_20260704_120819/report.md
Normal file
267
bench_results/sglang_8card_systematic_20260704_120819/report.md
Normal file
@ -0,0 +1,267 @@
|
|||||||
|
# SGLang 8-Card DeepSeek-V4-Flash 系统 Benchmark 报告
|
||||||
|
|
||||||
|
生成时间：2026-07-05 02:10:06
|
||||||
|
|
||||||
|
## 测试环境
|
||||||
|
|
||||||
|
- **模型**：DeepSeek-V4-Flash (`/data/models/DeepSeek-V4-Flash`)
|
||||||
|
- **推理框架**：SGLang
|
||||||
|
- **并行策略**：TP=8，8× NVIDIA H200
|
||||||
|
- **MoE backend**：marlin
|
||||||
|
- **投机采样**：EAGLE（`--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`）
|
||||||
|
- **服务端口**：30000
|
||||||
|
|
||||||
|
## 测试负载
|
||||||
|
|
||||||
|
- **数据集**：ShareGPT V4.3 unfiltered cleaned split
|
||||||
|
- **请求数**：每个测试点 2000 条真实对话请求
|
||||||
|
- **测试维度**：
|
||||||
|
- 并发扫描：`--max-concurrency` = 1, 8, 16, 32, 64, 128
|
||||||
|
- 请求速率扫描：`--request-rate` = 1, 2, 4, 8, 16 (Poisson-like)
|
||||||
|
|
||||||
|
## Phase 1：并发扫描结果
|
||||||
|
|
||||||
|
| 并发 | 耗时(s) | req/s | input tok/s | output tok/s | total tok/s | Mean TTFT(ms) | Mean TPOT(ms) | Mean E2E(ms) | Accept Length |
|
||||||
|
|------|---------|-------|-------------|--------------|-------------|---------------|---------------|--------------|---------------|
|
||||||
|
| 1 | 1715.45 | 1.17 | 335.83 | 243.51 | 579.34 | 128.09 | 4.00 | 856.77 | 2.62 |
|
||||||
|
| 8 | 448.91 | 4.46 | 1283.33 | 930.54 | 2213.87 | 146.76 | 8.41 | 1790.65 | 2.61 |
|
||||||
|
| 16 | 331.79 | 6.03 | 1736.31 | 1259.00 | 2995.30 | 168.95 | 12.67 | 2642.10 | 2.61 |
|
||||||
|
| 32 | 267.72 | 7.47 | 2151.87 | 1560.32 | 3712.19 | 206.24 | 20.72 | 4249.71 | 2.61 |
|
||||||
|
| 64 | 207.57 | 9.64 | 2775.39 | 2012.43 | 4787.82 | 248.41 | 32.61 | 6557.39 | 2.61 |
|
||||||
|
| 128 | 139.41 | 14.35 | 4132.36 | 2996.37 | 7128.73 | 330.40 | 46.15 | 8596.12 | 2.61 |
|
||||||
|
|
||||||
|
## Phase 2：请求速率扫描结果
|
||||||
|
|
||||||
|
| RPS | 耗时(s) | req/s | input tok/s | output tok/s | total tok/s | Mean TTFT(ms) | Mean TPOT(ms) | Mean E2E(ms) | Accept Length |
|
||||||
|
|-----|---------|-------|-------------|--------------|-------------|---------------|---------------|--------------|---------------|
|
||||||
|
| 1 | 2011.46 | 0.99 | 286.41 | 207.67 | 494.08 | 146.87 | 4.91 | 1007.62 | 2.61 |
|
||||||
|
| 2 | 1007.21 | 1.99 | 571.97 | 414.74 | 986.71 | 160.63 | 5.99 | 1205.24 | 2.61 |
|
||||||
|
| 4 | 505.49 | 3.96 | 1139.69 | 826.39 | 1966.08 | 174.67 | 9.02 | 1743.40 | 2.60 |
|
||||||
|
| 8 | 256.46 | 7.80 | 2246.34 | 1628.82 | 3875.15 | 216.65 | 25.35 | 4542.64 | 2.60 |
|
||||||
|
| 16 | 137.92 | 14.50 | 4176.94 | 3028.70 | 7205.64 | 377.28 | 87.16 | 15048.09 | 2.61 |
|
||||||
|
|
||||||
|
## 关键发现
|
||||||
|
|
||||||
|
1. **吞吐随并发单调上升**：从 c=1 的 579 tok/s 提升到 c=128 的 7129 tok/s，说明 8×H200 在高压下仍能有效扩展。
|
||||||
|
2. **c=32 是延迟与吞吐的拐点**：c=32 时 total tok/s 为 3712，E2E 延迟约 4.25s；c=64 时吞吐提升到 4788，但延迟增加到 6.56s。
|
||||||
|
3. **c=128 达到最高吞吐**：total tok/s = 7129，output tok/s = 2996，但 E2E 延迟接近 8.6s，适合离线/批处理场景。
|
||||||
|
4. **RPS=16 时系统过载**：实际 req/s 达到 14.5，TPOT 飙升到 87ms，E2E 延迟 15s，说明 RPS>14 已超出稳定运行区间。
|
||||||
|
5. **RPS=8 是稳定高吞吐边界**：total tok/s = 3875，E2E 延迟 4.54s，TTFT 仅 217ms，体验与吞吐兼顾。
|
||||||
|
6. **EAGLE 投机采样稳定生效**：Accept Length 稳定在 2.60–2.62，有效降低了 decode 步数。
|
||||||
|
|
||||||
|
## 场景建议
|
||||||
|
|
||||||
|
| 场景 | 推荐配置 | 理由 |
|
||||||
|
|------|----------|------|
|
||||||
|
| 低延迟在线 API | `--max-concurrency 8` | TTFT 147ms，TPOT 8.4ms，E2E 1.79s |
|
||||||
|
| 平衡型在线服务 | `--max-concurrency 16` 或 `--request-rate 4` | 吞吐约 3k tok/s，E2E 2.6–1.7s |
|
||||||
|
| 高吞吐在线服务 | `--max-concurrency 32` 或 `--request-rate 8` | 吞吐 3.7–3.9k tok/s，E2E 4.3–4.5s |
|
||||||
|
| 最大吞吐/离线批处理 | `--max-concurrency 128` | 吞吐 7.1k tok/s，E2E 8.6s |
|
||||||
|
| 避免使用 | `--request-rate 16` | 系统过载，E2E 15s，TPOT 87ms |
|
||||||
|
|
||||||
|
## 原始数据
|
||||||
|
|
||||||
|
- 汇总 JSON：`bench_results/sglang_8card_systematic_20260704_120819/summary.json`
|
||||||
|
- 每个测试点的详细结果与日志：`bench_results/sglang_8card_systematic_20260704_120819/`
|
||||||
|
|
||||||
|
## 附录 A：Benchmark 参数说明
|
||||||
|
|
||||||
|
本次所有测试均使用 `sglang.bench_serving` 工具，参数统一如下：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
envs/sglang/bin/python -m sglang.bench_serving \
|
||||||
|
--backend sglang \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port 30000 \
|
||||||
|
--dataset-name sharegpt \
|
||||||
|
--dataset-path /data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json \
|
||||||
|
--num-prompts 2000 \
|
||||||
|
--model /data/models/DeepSeek-V4-Flash \
|
||||||
|
--seed 42
|
||||||
|
```
|
||||||
|
|
||||||
|
两个测试维度的区别仅在于最后一条参数：
|
||||||
|
|
||||||
|
- **并发扫描**：`--max-concurrency {1,8,16,32,64,128}`
|
||||||
|
- 客户端同时保持 N 个未完成的请求，服务器队列按需堆积。
|
||||||
|
- 适合模拟“系统能扛多少并发”的容量测试。
|
||||||
|
|
||||||
|
- **请求速率扫描**：`--request-rate {1,2,4,8,16}`
|
||||||
|
- 客户端按泊松分布以固定 RPS 发送请求。
|
||||||
|
- 适合模拟真实线上按流量到达的场景。
|
||||||
|
|
||||||
|
### 指标定义
|
||||||
|
|
||||||
|
| 指标 | 含义 |
|
||||||
|
|------|------|
|
||||||
|
| **req/s** | 每秒完成的请求数 |
|
||||||
|
| **input tok/s** | 每秒处理的输入 token 数 |
|
||||||
|
| **output tok/s** | 每秒生成的输出 token 数 |
|
||||||
|
| **total tok/s** | input tok/s + output tok/s |
|
||||||
|
| **TTFT** | Time To First Token，首 token 延迟 |
|
||||||
|
| **TPOT** | Time Per Output Token，除首 token 外平均每个输出 token 的间隔 |
|
||||||
|
| **E2E Latency** | 端到端请求完成时间 |
|
||||||
|
| **Accept Length** | EAGLE 投机采样平均每次接受的 draft token 数 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 附录 B：Token 成本计算
|
||||||
|
|
||||||
|
### 假设
|
||||||
|
|
||||||
|
- GPU 单价：**8.0 元 / 卡 / 小时**
|
||||||
|
- 本次部署使用 **8× NVIDIA H200**
|
||||||
|
- 整机每小时成本：
|
||||||
|
|
||||||
|
```
|
||||||
|
8 卡 × 8.0 元/卡/h = 64 元/h
|
||||||
|
```
|
||||||
|
|
||||||
|
### 公式
|
||||||
|
|
||||||
|
对于任意一个 benchmark 结果：
|
||||||
|
|
||||||
|
```
|
||||||
|
每小时处理 input tokens = input_tok/s × 3600
|
||||||
|
每小时生成 output tokens = output_tok/s × 3600
|
||||||
|
|
||||||
|
每 1M input tokens 成本 = 64 / (input_tok/s × 3600 / 1,000,000)
|
||||||
|
每 1M output tokens 成本 = 64 / (output_tok/s × 3600 / 1,000,000)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 各场景成本
|
||||||
|
|
||||||
|
| 场景 | input tok/s | output tok/s | 1M input cost | 1M output cost |
|
||||||
|
|------|-------------|--------------|---------------|----------------|
|
||||||
|
| c=1 | 335.83 | 243.51 | 52.94 元 | 73.01 元 |
|
||||||
|
| c=8 | 1,283.33 | 930.54 | 13.85 元 | 19.10 元 |
|
||||||
|
| c=16 | 1,736.31 | 1,259.00 | 10.24 元 | 14.12 元 |
|
||||||
|
| c=32 | 2,151.87 | 1,560.32 | 8.26 元 | 11.39 元 |
|
||||||
|
| c=64 | 2,775.39 | 2,012.43 | 6.41 元 | 8.83 元 |
|
||||||
|
| c=128 | 4,132.36 | 2,996.37 | 4.30 元 | 5.93 元 |
|
||||||
|
| rps=1 | 286.41 | 207.67 | 62.07 元 | 85.61 元 |
|
||||||
|
| rps=2 | 571.97 | 414.74 | 31.08 元 | 42.86 元 |
|
||||||
|
| rps=4 | 1,139.69 | 826.39 | 15.60 元 | 21.51 元 |
|
||||||
|
| rps=8 | 2,246.34 | 1,628.82 | 7.91 元 | 10.91 元 |
|
||||||
|
| rps=16 | 4,176.94 | 3,028.70 | 4.26 元 | 5.87 元 |
|
||||||
|
|
||||||
|
### 对外定价建议
|
||||||
|
|
||||||
|
裸 GPU 成本不等于对外售价。建议根据 SLA 和毛利率加价：
|
||||||
|
|
||||||
|
| 推荐场景 | 裸成本（input/output） | 说明 |
|
||||||
|
|----------|------------------------|------|
|
||||||
|
| 低延迟在线（c=8） | 13.85 元 / 19.10 元 per 1M | TTFT 147ms，TPOT 8.4ms，适合实时聊天 |
|
||||||
|
| 平衡型在线（c=16） | 10.24 元 / 14.12 元 per 1M | 吞吐与延迟兼顾，主流 API 可参考此档位 |
|
||||||
|
| 高吞吐在线（c=32 / rps=8） | 8.09 元 / 11.15 元 per 1M | 吞吐高、延迟可接受，适合批量在线服务 |
|
||||||
|
| 最大吞吐/离线（c=128） | 4.30 元 / 5.93 元 per 1M | 延迟 8.6s，仅适合离线批处理 |
|
||||||
|
|
||||||
|
> 注意：`rps=16` 虽已接近最大吞吐，但系统已明显过载（E2E 15s），不建议按此成本对外定价。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 附录 C：最大吞吐压测（优化服务参数）
|
||||||
|
|
||||||
|
为了探索成本下限，重新部署了服务并调优了参数：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
envs/sglang/bin/sglang serve \
|
||||||
|
--trust-remote-code \
|
||||||
|
--model-path /data/models/DeepSeek-V4-Flash \
|
||||||
|
--tp 8 \
|
||||||
|
--moe-runner-backend marlin \
|
||||||
|
--speculative-algorithm EAGLE \
|
||||||
|
--speculative-num-steps 3 \
|
||||||
|
--speculative-eagle-topk 1 \
|
||||||
|
--speculative-num-draft-tokens 4 \
|
||||||
|
--max-running-requests 512 \
|
||||||
|
--chunked-prefill-size 16384 \
|
||||||
|
--host 0.0.0.0 \
|
||||||
|
--port 30000
|
||||||
|
```
|
||||||
|
|
||||||
|
与附录 A/B 的测试相比，主要变化：
|
||||||
|
- `--max-running-requests` 从默认 256 提升到 **512**
|
||||||
|
- `--chunked-prefill-size` 从默认 8192 提升到 **16384**
|
||||||
|
|
||||||
|
### 超大并发扫描结果
|
||||||
|
|
||||||
|
| 并发 | 耗时(s) | req/s | input tok/s | output tok/s | total tok/s | Mean TTFT | Mean TPOT | Mean E2E(ms) |
|
||||||
|
|------|---------|-------|-------------|--------------|-------------|-----------|-----------|--------------|
|
||||||
|
| 128 | 155.26 | 12.88 | 3,710.50 | 2,690.48 | 6,400.98 | 410.55 ms | 48.45 ms | 9,696.87 |
|
||||||
|
| 256 | 90.65 | 22.06 | 6,355.48 | 4,608.35 | 10,963.84 | 509.73 ms | 62.71 ms | 10,869.76 |
|
||||||
|
| 512 | 71.39 | 28.02 | 8,070.04 | 5,851.58 | 13,921.61 | 1,343.73 ms| 121.20 ms | 16,176.14 |
|
||||||
|
| **1024** | **68.21** | **29.32** | **8,445.53** | **6,123.84** | **14,569.37** | **13,265.46 ms** | **105.36 ms** | **27,566.99** |
|
||||||
|
|
||||||
|
### 关键发现
|
||||||
|
|
||||||
|
1. **高并发下吞吐显著提升**：优化参数后，c=256 及以上远超原配置的最高值（7,129 tok/s）；c=1024 达到 **14,569 tok/s**，约为原配置峰值的两倍。
|
||||||
|
2. **c=1024 达到最高吞吐**：total **14,569 tok/s**，output **6,124 tok/s**，但 TTFT 13.3s，系统已严重过载。
|
||||||
|
3. **c=512 是实际可用上限**：total **13,922 tok/s**，output **5,852 tok/s**，虽然 TTFT 1.3s、E2E 16s，但吞吐接近峰值，适合离线批处理。
|
||||||
|
4. **c=256 是高压在线的甜点**：total **10,964 tok/s**，TTFT 510ms，E2E 10.9s，如果业务能容忍 10 秒级响应，这是性价比很高的点。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 附录 D：最低 Token 成本（基于最大吞吐）
|
||||||
|
|
||||||
|
仍按 **8.0 元/卡时、8 卡整机 64 元/小时** 计算。
|
||||||
|
|
||||||
|
### 公式
|
||||||
|
|
||||||
|
```
|
||||||
|
每 1M input tokens 成本 = 64 / (input_tok/s × 3600 / 1,000,000)
|
||||||
|
每 1M output tokens 成本 = 64 / (output_tok/s × 3600 / 1,000,000)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 最大吞吐场景成本
|
||||||
|
|
||||||
|
| 场景 | input tok/s | output tok/s | 1M input cost | 1M output cost |
|
||||||
|
|------|-------------|--------------|---------------|----------------|
|
||||||
|
| c=128 | 3,710.50 | 2,690.48 | 4.79 元 | 6.61 元 |
|
||||||
|
| c=256 | 6,355.48 | 4,608.35 | 2.80 元 | 3.86 元 |
|
||||||
|
| c=512 | 8,070.04 | 5,851.58 | 2.20 元 | 3.04 元 |
|
||||||
|
| **c=1024** | **8,445.53** | **6,123.84** | **2.10 元** | **2.90 元** |
|
||||||
|
|
||||||
|
### 结论
|
||||||
|
|
||||||
|
在当前硬件和优化参数下：
|
||||||
|
|
||||||
|
- **理论最低 input 成本**：约 **2.10 元 / 1M input tokens**（c=1024）
|
||||||
|
- **理论最低 output 成本**：约 **2.90 元 / 1M output tokens**（c=1024）
|
||||||
|
- **实际可用最低成本**：c=512 时 **2.20 元 / 1M input**、**3.04 元 / 1M output**，此时吞吐已达峰值 95% 以上，且延迟比 c=1024 可控得多。
|
||||||
|
|
||||||
|
> 注意：c=1024 的 TTFT 超过 13 秒，仅适合完全不在意延迟的离线批处理任务；对外 API 服务不建议按此成本定价。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 附录 E：测试数据长度分布
|
||||||
|
|
||||||
|
本次所有 benchmark 使用的是 `ShareGPT_V4.3_unfiltered_cleaned_split.json` 数据集。`sglang.bench_serving` 在 `--num-prompts 2000` 且 `--seed 42` 的条件下，从数据集中采样出 **2000 条真实对话请求**作为测试负载。
|
||||||
|
|
||||||
|
### 长度统计
|
||||||
|
|
||||||
|
| 指标 | Input Tokens | Output Tokens |
|
||||||
|
|------|--------------|---------------|
|
||||||
|
| 请求数 | 2,000 | 2,000 |
|
||||||
|
| 平均值 | 288.05 | 208.86 |
|
||||||
|
| 标准差 | 420.07 | 217.60 |
|
||||||
|
| 最小值 | 2 | 2 |
|
||||||
|
| P50 | 125 | 150 |
|
||||||
|
| P90 | 723 | 516 |
|
||||||
|
| P95 | 841 | 667 |
|
||||||
|
| P99 | 2,190 | 811 |
|
||||||
|
| 最大值 | 3,996 | 1,655 |
|
||||||
|
| **总量** | **576,098** | **417,728** |
|
||||||
|
|
||||||
|
### 分布特点
|
||||||
|
|
||||||
|
1. **输入长度呈现典型长尾分布**：P50 仅 125 tokens，但 P99 达到 2,190 tokens，最大值 3,996 tokens。说明 ShareGPT 中既有大量短 prompt，也有少量长上下文对话。
|
||||||
|
2. **输出长度相对集中**：P50 150 tokens，P90 516 tokens，P99 811 tokens，最大值 1,655 tokens。大部分回复属于中等长度。
|
||||||
|
3. **输出/输入比约为 0.73**：平均每条请求输出 209 tokens、输入 288 tokens，整体负载偏 decode 侧。
|
||||||
|
4. **与原始 ShareGPT 全集的差异**：原始全集输出长度均值约 1,122 tokens，而 bench 采样后的输出均值仅 209 tokens。这是因为 `sglang.bench_serving` 在处理 sharegpt 数据集时会根据内置规则对输出做截断/采样，以更贴近典型在线 serving 负载。
|
||||||
|
|
||||||
|
### 长度分布图
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
> 图表文件：`bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png`
|
||||||
@ -0,0 +1,156 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_1",
|
||||||
|
"args": "--max-concurrency 1",
|
||||||
|
"duration_s": 1715.45,
|
||||||
|
"request_throughput": 1.17,
|
||||||
|
"input_token_throughput": 335.83,
|
||||||
|
"output_token_throughput": 243.51,
|
||||||
|
"total_token_throughput": 579.34,
|
||||||
|
"mean_ttft_ms": 128.09,
|
||||||
|
"mean_tpot_ms": 4.0,
|
||||||
|
"mean_e2e_latency_ms": 856.77,
|
||||||
|
"accept_length": 2.62,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_1.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_8",
|
||||||
|
"args": "--max-concurrency 8",
|
||||||
|
"duration_s": 448.91,
|
||||||
|
"request_throughput": 4.46,
|
||||||
|
"input_token_throughput": 1283.33,
|
||||||
|
"output_token_throughput": 930.54,
|
||||||
|
"total_token_throughput": 2213.87,
|
||||||
|
"mean_ttft_ms": 146.76,
|
||||||
|
"mean_tpot_ms": 8.41,
|
||||||
|
"mean_e2e_latency_ms": 1790.65,
|
||||||
|
"accept_length": 2.61,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_8.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_16",
|
||||||
|
"args": "--max-concurrency 16",
|
||||||
|
"duration_s": 331.79,
|
||||||
|
"request_throughput": 6.03,
|
||||||
|
"input_token_throughput": 1736.31,
|
||||||
|
"output_token_throughput": 1259.0,
|
||||||
|
"total_token_throughput": 2995.3,
|
||||||
|
"mean_ttft_ms": 168.95,
|
||||||
|
"mean_tpot_ms": 12.67,
|
||||||
|
"mean_e2e_latency_ms": 2642.1,
|
||||||
|
"accept_length": 2.61,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_16.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_32",
|
||||||
|
"args": "--max-concurrency 32",
|
||||||
|
"duration_s": 267.72,
|
||||||
|
"request_throughput": 7.47,
|
||||||
|
"input_token_throughput": 2151.87,
|
||||||
|
"output_token_throughput": 1560.32,
|
||||||
|
"total_token_throughput": 3712.19,
|
||||||
|
"mean_ttft_ms": 206.24,
|
||||||
|
"mean_tpot_ms": 20.72,
|
||||||
|
"mean_e2e_latency_ms": 4249.71,
|
||||||
|
"accept_length": 2.61,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_32.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_64",
|
||||||
|
"args": "--max-concurrency 64",
|
||||||
|
"duration_s": 207.57,
|
||||||
|
"request_throughput": 9.64,
|
||||||
|
"input_token_throughput": 2775.39,
|
||||||
|
"output_token_throughput": 2012.43,
|
||||||
|
"total_token_throughput": 4787.82,
|
||||||
|
"mean_ttft_ms": 248.41,
|
||||||
|
"mean_tpot_ms": 32.61,
|
||||||
|
"mean_e2e_latency_ms": 6557.39,
|
||||||
|
"accept_length": 2.61,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_64.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_concurrency_128",
|
||||||
|
"args": "--max-concurrency 128",
|
||||||
|
"duration_s": 139.41,
|
||||||
|
"request_throughput": 14.35,
|
||||||
|
"input_token_throughput": 4132.36,
|
||||||
|
"output_token_throughput": 2996.37,
|
||||||
|
"total_token_throughput": 7128.73,
|
||||||
|
"mean_ttft_ms": 330.4,
|
||||||
|
"mean_tpot_ms": 46.15,
|
||||||
|
"mean_e2e_latency_ms": 8596.12,
|
||||||
|
"accept_length": 2.61,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_128.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_rps_1",
|
||||||
|
"args": "--request-rate 1",
|
||||||
|
"duration_s": 2011.46,
|
||||||
|
"request_throughput": 0.99,
|
||||||
|
"input_token_throughput": 286.41,
|
||||||
|
"output_token_throughput": 207.67,
|
||||||
|
"total_token_throughput": 494.08,
|
||||||
|
"mean_ttft_ms": 146.87,
|
||||||
|
"mean_tpot_ms": 4.91,
|
||||||
|
"mean_e2e_latency_ms": 1007.62,
|
||||||
|
"accept_length": 2.61,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_1.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_rps_2",
|
||||||
|
"args": "--request-rate 2",
|
||||||
|
"duration_s": 1007.21,
|
||||||
|
"request_throughput": 1.99,
|
||||||
|
"input_token_throughput": 571.97,
|
||||||
|
"output_token_throughput": 414.74,
|
||||||
|
"total_token_throughput": 986.71,
|
||||||
|
"mean_ttft_ms": 160.63,
|
||||||
|
"mean_tpot_ms": 5.99,
|
||||||
|
"mean_e2e_latency_ms": 1205.24,
|
||||||
|
"accept_length": 2.61,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_2.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_rps_4",
|
||||||
|
"args": "--request-rate 4",
|
||||||
|
"duration_s": 505.49,
|
||||||
|
"request_throughput": 3.96,
|
||||||
|
"input_token_throughput": 1139.69,
|
||||||
|
"output_token_throughput": 826.39,
|
||||||
|
"total_token_throughput": 1966.08,
|
||||||
|
"mean_ttft_ms": 174.67,
|
||||||
|
"mean_tpot_ms": 9.02,
|
||||||
|
"mean_e2e_latency_ms": 1743.4,
|
||||||
|
"accept_length": 2.6,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_4.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_rps_8",
|
||||||
|
"args": "--request-rate 8",
|
||||||
|
"duration_s": 256.46,
|
||||||
|
"request_throughput": 7.8,
|
||||||
|
"input_token_throughput": 2246.34,
|
||||||
|
"output_token_throughput": 1628.82,
|
||||||
|
"total_token_throughput": 3875.15,
|
||||||
|
"mean_ttft_ms": 216.65,
|
||||||
|
"mean_tpot_ms": 25.35,
|
||||||
|
"mean_e2e_latency_ms": 4542.64,
|
||||||
|
"accept_length": 2.6,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_8.json"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "sharegpt_rps_16",
|
||||||
|
"args": "--request-rate 16",
|
||||||
|
"duration_s": 137.92,
|
||||||
|
"request_throughput": 14.5,
|
||||||
|
"input_token_throughput": 4176.94,
|
||||||
|
"output_token_throughput": 3028.7,
|
||||||
|
"total_token_throughput": 7205.64,
|
||||||
|
"mean_ttft_ms": 377.28,
|
||||||
|
"mean_tpot_ms": 87.16,
|
||||||
|
"mean_e2e_latency_ms": 15048.09,
|
||||||
|
"accept_length": 2.61,
|
||||||
|
"output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_16.json"
|
||||||
|
}
|
||||||
|
]
|
||||||
39
bench_results/vllm_dspark_qwen3_20260705_121046/summary.json
Normal file
39
bench_results/vllm_dspark_qwen3_20260705_121046/summary.json
Normal file
@ -0,0 +1,39 @@
|
|||||||
|
{
|
||||||
|
"model": "/data/models/Qwen3-4B",
|
||||||
|
"draft_model": "deepseek-ai/dspark_qwen3_4b_block7",
|
||||||
|
"url": "http://127.0.0.1:30003/v1/completions",
|
||||||
|
"max_tokens": 64,
|
||||||
|
"num_prompts": 20,
|
||||||
|
"dataset": "/data/user1/yy/datasets/ShareGPT_filtered_chat.json",
|
||||||
|
"timestamp": "20260705_121046",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"concurrency": 1,
|
||||||
|
"num_prompts": 20,
|
||||||
|
"duration_s": 0.07108968612737954,
|
||||||
|
"success_count": 0,
|
||||||
|
"error": "all requests failed",
|
||||||
|
"errors": [
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'",
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'",
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'",
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'",
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"concurrency": 4,
|
||||||
|
"num_prompts": 20,
|
||||||
|
"duration_s": 0.036050053080543876,
|
||||||
|
"success_count": 0,
|
||||||
|
"error": "all requests failed",
|
||||||
|
"errors": [
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'",
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'",
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'",
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'",
|
||||||
|
"404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
59
bench_results/vllm_dspark_qwen3_20260705_121218/summary.json
Normal file
59
bench_results/vllm_dspark_qwen3_20260705_121218/summary.json
Normal file
@ -0,0 +1,59 @@
|
|||||||
|
{
|
||||||
|
"model": "/data/models/Qwen3-4B",
|
||||||
|
"draft_model": "deepseek-ai/dspark_qwen3_4b_block7",
|
||||||
|
"url": "http://127.0.0.1:30003/v1/completions",
|
||||||
|
"max_tokens": 64,
|
||||||
|
"num_prompts": 20,
|
||||||
|
"dataset": "/data/user1/yy/datasets/ShareGPT_filtered_chat.json",
|
||||||
|
"timestamp": "20260705_121218",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"concurrency": 1,
|
||||||
|
"num_prompts": 20,
|
||||||
|
"duration_s": 3.1493232590146363,
|
||||||
|
"success_count": 20,
|
||||||
|
"fail_count": 0,
|
||||||
|
"input_tokens": 6132,
|
||||||
|
"output_tokens": 456,
|
||||||
|
"total_tokens": 6588,
|
||||||
|
"request_throughput": 6.350570695704836,
|
||||||
|
"input_throughput": 1947.0849753031027,
|
||||||
|
"output_throughput": 144.79301186207027,
|
||||||
|
"total_throughput": 2091.877987165173,
|
||||||
|
"mean_ttft_ms": 55.51949484506622,
|
||||||
|
"p50_ttft_ms": 27.64087135437876,
|
||||||
|
"p99_ttft_ms": 295.08487760089326,
|
||||||
|
"mean_tpot_ms": 4.6249661131817925,
|
||||||
|
"p50_tpot_ms": 4.634099831345484,
|
||||||
|
"p99_tpot_ms": 4.827384204164935,
|
||||||
|
"mean_e2e_ms": 156.8067875574343,
|
||||||
|
"p50_e2e_ms": 148.02213886287063,
|
||||||
|
"p99_e2e_ms": 406.6163514624348,
|
||||||
|
"errors": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"concurrency": 4,
|
||||||
|
"num_prompts": 20,
|
||||||
|
"duration_s": 1.0271000929642469,
|
||||||
|
"success_count": 20,
|
||||||
|
"fail_count": 0,
|
||||||
|
"input_tokens": 6132,
|
||||||
|
"output_tokens": 450,
|
||||||
|
"total_tokens": 6582,
|
||||||
|
"request_throughput": 19.472298889857267,
|
||||||
|
"input_throughput": 5970.206839630238,
|
||||||
|
"output_throughput": 438.1267250217885,
|
||||||
|
"total_throughput": 6408.333564652026,
|
||||||
|
"mean_ttft_ms": 82.42244796128944,
|
||||||
|
"p50_ttft_ms": 37.77908312622458,
|
||||||
|
"p99_ttft_ms": 263.8731326162815,
|
||||||
|
"mean_tpot_ms": 5.172366561559164,
|
||||||
|
"p50_tpot_ms": 5.150655882704692,
|
||||||
|
"p99_tpot_ms": 5.708390201389117,
|
||||||
|
"mean_e2e_ms": 193.68963547749445,
|
||||||
|
"p50_e2e_ms": 153.2427944475785,
|
||||||
|
"p99_e2e_ms": 387.3923780536279,
|
||||||
|
"errors": []
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
131
bench_results/vllm_dspark_qwen3_20260705_121256/summary.json
Normal file
131
bench_results/vllm_dspark_qwen3_20260705_121256/summary.json
Normal file
@ -0,0 +1,131 @@
|
|||||||
|
{
|
||||||
|
"model": "/data/models/Qwen3-4B",
|
||||||
|
"draft_model": "deepseek-ai/dspark_qwen3_4b_block7",
|
||||||
|
"url": "http://127.0.0.1:30003/v1/completions",
|
||||||
|
"max_tokens": 256,
|
||||||
|
"num_prompts": 500,
|
||||||
|
"dataset": "/data/user1/yy/datasets/ShareGPT_filtered_chat.json",
|
||||||
|
"timestamp": "20260705_121256",
|
||||||
|
"results": [
|
||||||
|
{
|
||||||
|
"concurrency": 1,
|
||||||
|
"num_prompts": 500,
|
||||||
|
"duration_s": 181.22913801996037,
|
||||||
|
"success_count": 500,
|
||||||
|
"fail_count": 0,
|
||||||
|
"input_tokens": 108545,
|
||||||
|
"output_tokens": 36681,
|
||||||
|
"total_tokens": 145226,
|
||||||
|
"request_throughput": 2.7589382450460618,
|
||||||
|
"input_throughput": 598.9379036170495,
|
||||||
|
"output_throughput": 202.4012275330692,
|
||||||
|
"total_throughput": 801.3391311501188,
|
||||||
|
"mean_ttft_ms": 17.586736707482487,
|
||||||
|
"p50_ttft_ms": 15.236841514706612,
|
||||||
|
"p99_ttft_ms": 43.418583339080215,
|
||||||
|
"mean_tpot_ms": 4.756160026498775,
|
||||||
|
"p50_tpot_ms": 4.747943201144331,
|
||||||
|
"p99_tpot_ms": 4.845208972490266,
|
||||||
|
"mean_e2e_ms": 361.93272305326536,
|
||||||
|
"p50_e2e_ms": 369.4158981088549,
|
||||||
|
"p99_e2e_ms": 552.2739849891514,
|
||||||
|
"errors": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"concurrency": 4,
|
||||||
|
"num_prompts": 500,
|
||||||
|
"duration_s": 49.20666282507591,
|
||||||
|
"success_count": 500,
|
||||||
|
"fail_count": 0,
|
||||||
|
"input_tokens": 108545,
|
||||||
|
"output_tokens": 36991,
|
||||||
|
"total_tokens": 145536,
|
||||||
|
"request_throughput": 10.161225559584139,
|
||||||
|
"input_throughput": 2205.900456730121,
|
||||||
|
"output_throughput": 751.7477893491538,
|
||||||
|
"total_throughput": 2957.6482460792745,
|
||||||
|
"mean_ttft_ms": 26.531575095374137,
|
||||||
|
"p50_ttft_ms": 24.292025598697364,
|
||||||
|
"p99_ttft_ms": 41.29285377450287,
|
||||||
|
"mean_tpot_ms": 5.007692315964597,
|
||||||
|
"p50_tpot_ms": 4.998277087638343,
|
||||||
|
"p99_tpot_ms": 5.130429720718591,
|
||||||
|
"mean_e2e_ms": 392.14430282171816,
|
||||||
|
"p50_e2e_ms": 400.28460952453315,
|
||||||
|
"p99_e2e_ms": 625.0063395127652,
|
||||||
|
"errors": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"concurrency": 16,
|
||||||
|
"num_prompts": 500,
|
||||||
|
"duration_s": 14.20007478701882,
|
||||||
|
"success_count": 500,
|
||||||
|
"fail_count": 0,
|
||||||
|
"input_tokens": 108545,
|
||||||
|
"output_tokens": 36529,
|
||||||
|
"total_tokens": 145074,
|
||||||
|
"request_throughput": 35.21108215972787,
|
||||||
|
"input_throughput": 7643.973826055324,
|
||||||
|
"output_throughput": 2572.451240425399,
|
||||||
|
"total_throughput": 10216.425066480724,
|
||||||
|
"mean_ttft_ms": 31.879239567089826,
|
||||||
|
"p50_ttft_ms": 28.826430439949036,
|
||||||
|
"p99_ttft_ms": 74.12134333979331,
|
||||||
|
"mean_tpot_ms": 5.771187238080647,
|
||||||
|
"p50_tpot_ms": 5.76495784573639,
|
||||||
|
"p99_tpot_ms": 6.1421910060182805,
|
||||||
|
"mean_e2e_ms": 447.76513252267614,
|
||||||
|
"p50_e2e_ms": 459.7155440133065,
|
||||||
|
"p99_e2e_ms": 669.6232496039011,
|
||||||
|
"errors": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"concurrency": 64,
|
||||||
|
"num_prompts": 500,
|
||||||
|
"duration_s": 7.960247536888346,
|
||||||
|
"success_count": 500,
|
||||||
|
"fail_count": 0,
|
||||||
|
"input_tokens": 108545,
|
||||||
|
"output_tokens": 36583,
|
||||||
|
"total_tokens": 145128,
|
||||||
|
"request_throughput": 62.81211704572815,
|
||||||
|
"input_throughput": 13635.882489457124,
|
||||||
|
"output_throughput": 4595.711355767746,
|
||||||
|
"total_throughput": 18231.59384522487,
|
||||||
|
"mean_ttft_ms": 73.04830395104364,
|
||||||
|
"p50_ttft_ms": 53.465207340195775,
|
||||||
|
"p99_ttft_ms": 252.97995713772252,
|
||||||
|
"mean_tpot_ms": 12.42267764661875,
|
||||||
|
"p50_tpot_ms": 12.61188557741512,
|
||||||
|
"p99_tpot_ms": 13.063075978620708,
|
||||||
|
"mean_e2e_ms": 968.1061067078263,
|
||||||
|
"p50_e2e_ms": 967.5668340642005,
|
||||||
|
"p99_e2e_ms": 1512.9108975501729,
|
||||||
|
"errors": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"concurrency": 128,
|
||||||
|
"num_prompts": 500,
|
||||||
|
"duration_s": 7.319521516095847,
|
||||||
|
"success_count": 500,
|
||||||
|
"fail_count": 0,
|
||||||
|
"input_tokens": 108545,
|
||||||
|
"output_tokens": 36613,
|
||||||
|
"total_tokens": 145158,
|
||||||
|
"request_throughput": 68.31047615619204,
|
||||||
|
"input_throughput": 14829.52126874773,
|
||||||
|
"output_throughput": 5002.102927013319,
|
||||||
|
"total_throughput": 19831.62419576105,
|
||||||
|
"mean_ttft_ms": 145.49234842788428,
|
||||||
|
"p50_ttft_ms": 90.75760398991406,
|
||||||
|
"p99_ttft_ms": 423.2492828951217,
|
||||||
|
"mean_tpot_ms": 22.034148180580335,
|
||||||
|
"p50_tpot_ms": 23.052740126916806,
|
||||||
|
"p99_tpot_ms": 23.587740010073134,
|
||||||
|
"mean_e2e_ms": 1730.9650285840034,
|
||||||
|
"p50_e2e_ms": 1733.5386699996889,
|
||||||
|
"p99_e2e_ms": 2722.885905068833,
|
||||||
|
"errors": []
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
110
cleanup_summary.md
Normal file
110
cleanup_summary.md
Normal file
@ -0,0 +1,110 @@
|
|||||||
|
# /data/user1/yy 文件夹整理记录
|
||||||
|
|
||||||
|
> 整理时间：2026-07-06
|
||||||
|
> 说明：将脚本集中到 `scripts/`，uv 虚拟环境集中到 `envs/`，并保留兼容性软链接；删除过时的 log、pid、临时文件和冗余压缩包。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 目录结构
|
||||||
|
|
||||||
|
```
|
||||||
|
/data/user1/yy/
|
||||||
|
├── bench_results/
|
||||||
|
│ ├── dsv4_comparison_20260705_152221/
|
||||||
|
│ ├── dsv4_flash_dspark_misc/
|
||||||
|
│ ├── pd_bench/
|
||||||
|
│ ├── sglang_8card_max_throughput_20260705_030839/
|
||||||
|
│ ├── sglang_8card_systematic_20260704_120819/
|
||||||
|
│ ├── sglang_misc/
|
||||||
|
│ ├── vllm_dspark_qwen3_20260705_121046/
|
||||||
|
│ ├── vllm_dspark_qwen3_20260705_121218/
|
||||||
|
│ └── vllm_dspark_qwen3_20260705_121256/
|
||||||
|
├── datasets/
|
||||||
|
├── envs/ # uv 虚拟环境与缓存
|
||||||
|
│ ├── sglang/ -> 原 /data/user1/yy/sglang
|
||||||
|
│ ├── spraseattn/ -> 原 /data/user1/yy/spraseattn
|
||||||
|
│ ├── uv_cache/ -> 原 /data/user1/yy/uv_cache
|
||||||
|
│ ├── vllm/ -> 原 /data/user1/yy/vllm
|
||||||
|
│ └── vllm-dspark/ -> 原 /data/user1/yy/vllm-dspark
|
||||||
|
├── loomeval_yy/ # 按用户要求保留
|
||||||
|
├── scripts/ # 脚本集中存放
|
||||||
|
│ ├── bench_dsv4_comparison.py
|
||||||
|
│ ├── bench_vllm_dspark_qwen3.py
|
||||||
|
│ ├── install_vllm_dspark.sh
|
||||||
|
│ ├── run_sglang_benchmark.sh
|
||||||
|
│ ├── run_sglang_max_throughput.sh
|
||||||
|
│ ├── start_dsv4_dspark_8card.sh
|
||||||
|
│ ├── start_dsv4_dspark_8card_bf16kv.sh
|
||||||
|
│ ├── start_dsv4_dspark_8card_flashinfer.sh
|
||||||
|
│ ├── start_pd_single_node.sh
|
||||||
|
│ ├── start_sglang_dsv4_8card.sh
|
||||||
|
│ ├── start_vllm_pd_single_node.sh
|
||||||
|
│ └── test_block_sparse_attn.py
|
||||||
|
├── tmp/
|
||||||
|
├── vllm-main/ # vLLM 源码目录
|
||||||
|
├── cleanup_summary.md
|
||||||
|
├── dspark_deepseekv4_fix_pr_prep.md
|
||||||
|
├── dsv4_inference_comparison_report.md
|
||||||
|
├── issue_47648_comment.json
|
||||||
|
└── issue_47648_comment.md
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 脚本说明
|
||||||
|
|
||||||
|
| 脚本 | 说明 |
|
||||||
|
|---|---|
|
||||||
|
| `scripts/bench_dsv4_comparison.py` | vLLM DeepSeek-V4 对比测试主控脚本 |
|
||||||
|
| `scripts/bench_vllm_dspark_qwen3.py` | Qwen3 DSpark 验证脚本 |
|
||||||
|
| `scripts/install_vllm_dspark.sh` | vllm-dspark 安装脚本 |
|
||||||
|
| `scripts/run_sglang_benchmark.sh` | SGLang 8卡系统测试脚本 |
|
||||||
|
| `scripts/run_sglang_max_throughput.sh` | SGLang 极限吞吐测试脚本 |
|
||||||
|
| `scripts/start_dsv4_dspark_8card.sh` | DSV4 DSpark 服务启动脚本 |
|
||||||
|
| `scripts/start_dsv4_dspark_8card_bf16kv.sh` | DSV4 DSpark bf16 KV 启动脚本 |
|
||||||
|
| `scripts/start_dsv4_dspark_8card_flashinfer.sh` | DSV4 DSpark FlashInfer 启动脚本 |
|
||||||
|
| `scripts/start_pd_single_node.sh` | PD 分离单节点启动脚本 |
|
||||||
|
| `scripts/start_sglang_dsv4_8card.sh` | SGLang DSV4 启动脚本 |
|
||||||
|
| `scripts/start_vllm_pd_single_node.sh` | vLLM PD 分离单节点启动脚本 |
|
||||||
|
| `scripts/test_block_sparse_attn.py` | block sparse attention 测试脚本 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 删除的文件/目录
|
||||||
|
|
||||||
|
| 路径 | 原因 |
|
||||||
|
|---|---|
|
||||||
|
| `__pycache__/` | Python 字节码缓存，可重新生成 |
|
||||||
|
| `*.pid`（共 4 个） | 进程 ID 临时文件 |
|
||||||
|
| `logs/` 目录下所有文件 | 安装、部署、下载等历史调试日志 |
|
||||||
|
| `pd_logs/` 目录 | PD 分离测试日志 |
|
||||||
|
| `vllm_pd_logs/` 目录 | vLLM PD 分离测试日志 |
|
||||||
|
| `vllm-main.zip` | 与 `vllm-main/` 目录冗余，约 41MB |
|
||||||
|
| `tmp/` 目录下内容 | 临时文件 |
|
||||||
|
| 根目录下的 `.log` 文件 | 已归档或无需保留的日志 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 归档到 `bench_results/` 的文件
|
||||||
|
|
||||||
|
| 原路径 | 目标路径 | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| `bench_dsv4_comparison_master.log` | `bench_results/dsv4_comparison_20260705_152221/` | 本次对比测试主日志 |
|
||||||
|
| `sglang_8card.log` | `bench_results/sglang_misc/` | SGLang 8卡日志 |
|
||||||
|
| `sglang_0704_1_1048576_1.jsonl` | `bench_results/sglang_misc/` | SGLang 测试原始结果 |
|
||||||
|
| `sglang_8card_sharegpt_c*.json/log` | `bench_results/sglang_8card_systematic_20260704_120819/` | SGLang 系统测试散落结果 |
|
||||||
|
| `pd_bench_*.json` | `bench_results/pd_bench/` | PD 分离 benchmark 结果 |
|
||||||
|
| `dsv4_flash_dspark_c16_n50_*` | `bench_results/dsv4_flash_dspark_misc/` | DSV4 DSpark 早期单次测试 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 路径更新说明
|
||||||
|
|
||||||
|
脚本和文档中的环境路径已统一更新为 `envs/` 下的新路径：
|
||||||
|
|
||||||
|
- `/data/user1/yy/vllm-dspark/` → `/data/user1/yy/envs/vllm-dspark/`
|
||||||
|
- `/data/user1/yy/vllm/` → `/data/user1/yy/envs/vllm/`
|
||||||
|
- `/data/user1/yy/sglang/` → `/data/user1/yy/envs/sglang/`
|
||||||
|
- `/data/user1/yy/spraseattn/` → `/data/user1/yy/envs/spraseattn/`
|
||||||
|
|
||||||
|
根目录不再保留兼容性软链接，所有环境入口统一通过 `envs/` 访问。
|
||||||
286
dspark_deepseekv4_fix_pr_prep.md
Normal file
286
dspark_deepseekv4_fix_pr_prep.md
Normal file
@ -0,0 +1,286 @@
|
|||||||
|
# DeepSeek-V4-Flash-DSpark 修复记录（GitHub Issue #47648）
|
||||||
|
|
||||||
|
> 本文件记录 vLLM DSpark 在 DeepSeek-V4-Flash 上的启动失败问题、根因、修复 diff 以及验证结果，便于后续直接用于提交 PR。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 问题概述
|
||||||
|
|
||||||
|
在 H200/SM90（以及同样走 NVIDIA 路径的 B200/SM120）上，使用 `--spec-method dspark` 部署 `DeepSeek-V4-Flash-DSpark` 时，模型初始化阶段会失败。主要表现为两类错误：
|
||||||
|
|
||||||
|
1. **DSpark draft 权重加载路径错配**：draft 权重（checkpoint 中以 `mtp.{i}.*` 命名）没有被正确加载到模型中。
|
||||||
|
2. **KV cache shape mismatch**：当启用 `fp8_ds_mla` layout 时，KV cache 的 per-token slot size 被错误计算为 512B，而实际需要 584B，导致 shape 断言失败。
|
||||||
|
|
||||||
|
关联 issue：[vllm-project/vllm#47648](https://github.com/vllm-project/vllm/issues/47648)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 环境信息
|
||||||
|
|
||||||
|
- vLLM 版本：`0.23.1rc1.dev788+gfa4321de3`（vllm-dspark wheel）
|
||||||
|
- PyTorch：`2.11.0+cu129`
|
||||||
|
- GPU：8× NVIDIA H200（SM90），TP=8
|
||||||
|
- 模型：`/data/models/DeepSeek-V4-Flash-DSpark`
|
||||||
|
- 启动命令：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
vllm serve /data/models/DeepSeek-V4-Flash-DSpark \
|
||||||
|
--trust-remote-code \
|
||||||
|
--tensor-parallel-size 8 \
|
||||||
|
--kv-cache-dtype fp8 \
|
||||||
|
--block-size 256 \
|
||||||
|
--max-model-len auto \
|
||||||
|
--max-num-seqs 256 \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--spec-method dspark \
|
||||||
|
--spec-model /data/models/DeepSeek-V4-Flash-DSpark \
|
||||||
|
--spec-tokens 5 \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--port 30004
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 根因分析
|
||||||
|
|
||||||
|
### 3.1 根因一：DSpark draft 权重加载路径错配
|
||||||
|
|
||||||
|
#### 位置
|
||||||
|
|
||||||
|
`vllm/models/deepseek_v4/nvidia/dspark.py`，`_remap_dspark_name` 方法。
|
||||||
|
|
||||||
|
#### 问题描述
|
||||||
|
|
||||||
|
DeepSeek-V4-Flash-DSpark 的 draft 权重是嵌在目标模型 checkpoint 中的，命名格式为 `mtp.{i}.*`，例如：
|
||||||
|
|
||||||
|
```text
|
||||||
|
mtp.0.self_attn.wq_b.weight
|
||||||
|
mtp.0.ffn.experts.0.w1.weight
|
||||||
|
mtp.1.self_attn.wq_b.weight
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
`_remap_dspark_name` 负责把这些 checkpoint key 映射到 DSpark draft 模型内部的真实参数名。原来的实现：
|
||||||
|
|
||||||
|
```python
|
||||||
|
return f"model.layers.{self.num_hidden_layers + stage}.{rest}"
|
||||||
|
```
|
||||||
|
|
||||||
|
把 `mtp.0.*` 映射成了 `model.layers.{num_hidden_layers + 0}.*`（例如 `model.layers.64.*`）。
|
||||||
|
|
||||||
|
但实际上，DSpark draft 的 3 个 `DeepseekV4DecoderLayer` 是放在一个 `nn.ModuleList` 里的，PyTorch 真实注册的参数名只跟 `ModuleList` 的索引有关：
|
||||||
|
|
||||||
|
```text
|
||||||
|
model.layers.0.*
|
||||||
|
model.layers.1.*
|
||||||
|
model.layers.2.*
|
||||||
|
```
|
||||||
|
|
||||||
|
`DeepseekV4DecoderLayer` 构造函数里传入的 `prefix=f"layers.{num_hidden_layers + i}"` 只是为了内部计算 `compress_ratio`（让 `layer_id >= num_hidden_layers` 时固定 `compress_ratio=1`），**不是真实的参数名前缀**。
|
||||||
|
|
||||||
|
因此，原来的映射导致所有 draft block 权重都找不到对应的模型参数，draft layer 实际上没有被正确加载。
|
||||||
|
|
||||||
|
#### 修复
|
||||||
|
|
||||||
|
把 `mtp.{i}` 的 block 权重映射到 `model.layers.{i}`：
|
||||||
|
|
||||||
|
```python
|
||||||
|
def _remap_dspark_name(self, name: str) -> str | None:
|
||||||
|
"""Map a checkpoint ``mtp.{i}.*`` name to this model's parameter path.
|
||||||
|
|
||||||
|
Returns None for non-mtp weights (owned by the target model).
|
||||||
|
"""
|
||||||
|
m = re.match(r"mtp\.(\d+)\.(.*)", name)
|
||||||
|
if m is None:
|
||||||
|
return None
|
||||||
|
stage = int(m.group(1))
|
||||||
|
rest = m.group(2)
|
||||||
|
# The confidence head is not wired into inference yet; drop its weights.
|
||||||
|
if rest.startswith("confidence_head."):
|
||||||
|
return None
|
||||||
|
# Head-stack params live at model level (mtp.last), context combiner at
|
||||||
|
# model level (mtp.0); everything else is a per-layer decoder block.
|
||||||
|
head_prefixes = (
|
||||||
|
"norm.",
|
||||||
|
"hc_head_fn",
|
||||||
|
"hc_head_base",
|
||||||
|
"hc_head_scale",
|
||||||
|
"markov_head.",
|
||||||
|
)
|
||||||
|
if rest.startswith(("main_proj.", "main_norm.")) or rest.startswith(
|
||||||
|
head_prefixes
|
||||||
|
):
|
||||||
|
return f"model.{rest}"
|
||||||
|
# Draft layers live in a ModuleList, so their actual parameter names are
|
||||||
|
# model.layers.{stage}.* even though the prefix passed to the decoder
|
||||||
|
# layer is layers.{num_hidden_layers + stage} (for compress_ratio).
|
||||||
|
return f"model.layers.{stage}.{rest}"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 根因二：DeepSeek-V4 KV cache shape mismatch
|
||||||
|
|
||||||
|
#### 位置
|
||||||
|
|
||||||
|
`vllm/models/deepseek_v4/attention.py`，`DeepseekV4Attention.get_kv_cache_spec` 方法。
|
||||||
|
|
||||||
|
#### 问题描述
|
||||||
|
|
||||||
|
`DeepseekV4Attention.get_kv_cache_spec` 返回的 `MLAAttentionSpec` 没有设置 `kv_quant_mode`。在 `gpu_model_runner.py` 中，这个缺失导致 `cache_dtype_str` 被当成 `"auto"` 传给 `DeepseekV4FlashMLABackend.get_kv_cache_shape`，返回的 shape 是 `(num_blocks, block_size, 512)`。
|
||||||
|
|
||||||
|
但对于 `fp8_ds_mla` layout（UE8M0 block-scaled fp8，以 `uint8` 打包），每个 token 的 KV slot 实际大小是 584 字节，而不是 512。于是 KV cache 分配的空间不够，触发 shape mismatch / assert。
|
||||||
|
|
||||||
|
#### 修复
|
||||||
|
|
||||||
|
在 `MLAAttentionSpec` 中加上 `kv_quant_mode=get_kv_quant_mode(self.kv_cache_dtype)`：
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_kv_cache_spec(self, vllm_config: VllmConfig) -> KVCacheSpec | None:
|
||||||
|
if (
|
||||||
|
self.compress_ratio <= 1
|
||||||
|
): # SWA part. Allocated separately as DeepseekV4SWACache.
|
||||||
|
return None
|
||||||
|
# fp8_ds_mla is a UE8M0 block-scaled uint8 layout and needs 576B
|
||||||
|
# alignment; plain bf16 / per-tensor fp8 rows use natural element-size
|
||||||
|
# pages.
|
||||||
|
uses_fp8_ds_mla_layout = self.kv_cache_dtype == "fp8_ds_mla"
|
||||||
|
return MLAAttentionSpec(
|
||||||
|
block_size=vllm_config.cache_config.block_size,
|
||||||
|
num_kv_heads=1,
|
||||||
|
head_size=self.head_dim,
|
||||||
|
dtype=torch.uint8 if uses_fp8_ds_mla_layout else self.kv_cache_torch_dtype,
|
||||||
|
compress_ratio=self.compress_ratio,
|
||||||
|
cache_dtype_str=self.kv_cache_dtype,
|
||||||
|
alignment=576 if uses_fp8_ds_mla_layout else None,
|
||||||
|
model_version="deepseek_v4",
|
||||||
|
kv_quant_mode=get_kv_quant_mode(self.kv_cache_dtype),
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 修改文件清单
|
||||||
|
|
||||||
|
| 文件 | 修改内容 |
|
||||||
|
|---|---|
|
||||||
|
| `vllm/models/deepseek_v4/nvidia/dspark.py` | `_remap_dspark_name`：draft block 权重从 `model.layers.{num_hidden_layers + stage}` 改为 `model.layers.{stage}` |
|
||||||
|
| `vllm/models/deepseek_v4/attention.py` | `DeepseekV4Attention.get_kv_cache_spec`：在 `MLAAttentionSpec` 中补充 `kv_quant_mode=get_kv_quant_mode(self.kv_cache_dtype)` |
|
||||||
|
|
||||||
|
> 如果上游 vllm-main 仓库与 vllm-dspark 安装包的文件结构一致，也需要同步修改 `vllm-main` 下对应路径的同名文件。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 完整 diff（面向 PR）
|
||||||
|
|
||||||
|
```diff
|
||||||
|
--- a/vllm/models/deepseek_v4/nvidia/dspark.py
|
||||||
|
+++ b/vllm/models/deepseek_v4/nvidia/dspark.py
|
||||||
|
@@ -486,8 +486,8 @@ class DSparkDeepseekV4ForCausalLM(nn.Module):
|
||||||
|
if rest.startswith(("main_proj.", "main_norm.")) or rest.startswith(
|
||||||
|
head_prefixes
|
||||||
|
):
|
||||||
|
return f"model.{rest}"
|
||||||
|
- # Draft layers live after the target layers in the decoder stack.
|
||||||
|
- return f"model.layers.{self.num_hidden_layers + stage}.{rest}"
|
||||||
|
+ # Draft layers live in a ModuleList, so their actual parameter names are
|
||||||
|
+ # model.layers.{stage}.* even though the prefix passed to the decoder
|
||||||
|
+ # layer is layers.{num_hidden_layers + stage} (for compress_ratio).
|
||||||
|
+ return f"model.layers.{stage}.{rest}"
|
||||||
|
```
|
||||||
|
|
||||||
|
```diff
|
||||||
|
--- a/vllm/models/deepseek_v4/attention.py
|
||||||
|
+++ b/vllm/models/deepseek_v4/attention.py
|
||||||
|
@@ -613,6 +613,7 @@ class DeepseekV4Attention(nn.Module, AttentionLayerBase, ABC):
|
||||||
|
cache_dtype_str=self.kv_cache_dtype,
|
||||||
|
alignment=576 if uses_fp8_ds_mla_layout else None,
|
||||||
|
model_version="deepseek_v4",
|
||||||
|
+ kv_quant_mode=get_kv_quant_mode(self.kv_cache_dtype),
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 验证结果
|
||||||
|
|
||||||
|
### 6.1 Qwen3 DSpark 通路验证
|
||||||
|
|
||||||
|
- 目标模型：`/data/models/Qwen3-4B`
|
||||||
|
- Draft 模型：`deepseek-ai/dspark_qwen3_4b_block7`
|
||||||
|
- 结果：服务启动成功，`/v1/completions` 返回结果正常，speculative decoding 工作。
|
||||||
|
|
||||||
|
### 6.2 DeepSeek-V4-Flash-DSpark 服务验证
|
||||||
|
|
||||||
|
修复后，使用第 2 节的命令可以成功启动服务，监听 `http://127.0.0.1:30004`。简单 completion 请求返回正常：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST http://127.0.0.1:30004/v1/completions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"prompt": "The capital of France is",
|
||||||
|
"max_tokens": 20,
|
||||||
|
"temperature": 0.0
|
||||||
|
}'
|
||||||
|
# 返回: " Paris."
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.3 Serving benchmark
|
||||||
|
|
||||||
|
使用 vLLM 内置 `vllm bench serve`：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
vllm bench serve \
|
||||||
|
--host 127.0.0.1 --port 30004 \
|
||||||
|
--backend openai \
|
||||||
|
--dataset-name sharegpt \
|
||||||
|
--dataset-path /data/user1/yy/datasets/ShareGPT_filtered_chat.json \
|
||||||
|
--sharegpt-output-len 256 \
|
||||||
|
--num-prompts 50 \
|
||||||
|
--max-concurrency 16 \
|
||||||
|
--endpoint /v1/completions \
|
||||||
|
--model /data/models/DeepSeek-V4-Flash-DSpark \
|
||||||
|
--seed 42
|
||||||
|
```
|
||||||
|
|
||||||
|
结果：
|
||||||
|
|
||||||
|
| 指标 | 数值 |
|
||||||
|
|---|---|
|
||||||
|
| Request throughput | **2.38 req/s** |
|
||||||
|
| Output token throughput | **610.47 tok/s** |
|
||||||
|
| Acceptance rate | **28.20%** |
|
||||||
|
| Mean acceptance length | **2.41** |
|
||||||
|
| Mean TTFT | 2162 ms |
|
||||||
|
| Mean TPOT | 16.85 ms |
|
||||||
|
| Total input tokens | 16381 |
|
||||||
|
| Total generated tokens | 12800 |
|
||||||
|
|
||||||
|
服务器日志中的 SpecDecoding metrics 在稳态下 acceptance rate 落在 25%–36% 区间，与 DSpark 论文预期一致。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 影响范围
|
||||||
|
|
||||||
|
- **受影响**：所有使用 NVIDIA 路径运行 DeepSeek-V4-Flash-DSpark 的 GPU（包括 H200/SM90 的 FlashMLA 路径，以及 B200/SM120 的 FlashInfer SM120 路径）。
|
||||||
|
- **不受影响**：
|
||||||
|
- Qwen3 DSpark（使用独立的 `qwen3_dspark.py` 实现，不共享 `_remap_dspark_name`）。
|
||||||
|
- AMD/ROCm 和 XPU 平台（目前 DSpark 仅支持 NVIDIA）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 后续 PR 待办
|
||||||
|
|
||||||
|
- [ ] 在 vllm-main 上同步应用以上两处修改。
|
||||||
|
- [ ] 跑通 DeepSeek-V4-Flash-DSpark 的单元测试 / 冒烟测试。
|
||||||
|
- [ ] 检查 `DeepseekV4IndexerCache.get_kv_cache_spec` 是否也需要补充 `kv_quant_mode`（当前未改动，因为未触发错误）。
|
||||||
|
- [ ] 补充 DSpark draft 权重加载的回归测试（可选，但建议）。
|
||||||
|
- [ ] 向 vllm-project/vllm 提交 PR，并在描述中引用 issue #47648。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 备注
|
||||||
|
|
||||||
|
- 本文件中的 token、路径、版本号均基于 2026-07-05 的实际运行环境。
|
||||||
|
- GitHub issue 评论已发布：https://github.com/vllm-project/vllm/issues/47648#issuecomment-4886275279
|
||||||
351
dsv4_inference_comparison_report.md
Normal file
351
dsv4_inference_comparison_report.md
Normal file
@ -0,0 +1,351 @@
|
|||||||
|
# DeepSeek-V4 推理性能评估报告
|
||||||
|
|
||||||
|
## vLLM DSpark vs 其他投机解码方法 vs SGLang
|
||||||
|
|
||||||
|
> 测试环境：8× NVIDIA H200 (143GB)，CUDA 12.x，驱动 575.57.08
|
||||||
|
> 测试时间：2026-07-05
|
||||||
|
> 测试模型：`DeepSeek-V4-Flash` / `DeepSeek-V4-Flash-DSpark`
|
||||||
|
> 测试框架：vLLM 0.24.0、vLLM-dspark 0.23.1rc1.dev788、SGLang 0.5.14
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 背景与目标
|
||||||
|
|
||||||
|
### 1.1 投机解码（Speculative Decoding）基本原理
|
||||||
|
|
||||||
|
大模型自回归生成的瓶颈在于：每生成一个 token 都要做一次完整的模型前向传播。投机解码的核心思想是：
|
||||||
|
|
||||||
|
1. 用一个轻量的 **draft model**（或 draft head）一次性生成多个候选 token；
|
||||||
|
2. 用原始 **target model** 并行验证这些候选 token；
|
||||||
|
3. 接受与 target 模型一致的 token，从第一个不一致处重新生成。
|
||||||
|
|
||||||
|
理想情况下，如果 draft model 质量高，可以一次接受多个 token，从而把有效生成步长从 1 提升到 1+α（α 为平均接受长度），显著降低 latency、提升 throughput。
|
||||||
|
|
||||||
|
### 1.2 常见投机解码方法对比
|
||||||
|
|
||||||
|
| 方法 | 代表实现 | 优点 | 缺点 |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **EAGLE / EAGLE-2** | vLLM EAGLE、SGLang EAGLE | 接受率高（40-70%），draft 模型小 | 需要单独训练 draft 模型，内存占用额外 |
|
||||||
|
| **MTP (Multi-Token Prediction)** | DeepSeek MTP | 与目标模型结构一致，可共享权重 | 通常只有 1-2 个 MTP head，加速比有限 |
|
||||||
|
| **DSpark** | vLLM-dspark | 半自回归 draft，并行生成整段，接受长度高 | draft 计算量较大，对 batching/scheduling 要求高 |
|
||||||
|
| **n-gram / prompt-lookup** | vLLM ngram | 无需额外模型 | 只适用于重复性文本，泛化差 |
|
||||||
|
| **Medusa** | Medusa | 多个解码头 | 需要训练，内存占用大 |
|
||||||
|
|
||||||
|
### 1.3 DSpark 的特点
|
||||||
|
|
||||||
|
DSpark 是 DeepSeek 提出的块级半自回归投机解码方案，关键设计：
|
||||||
|
|
||||||
|
- **块级 draft**：一次并行生成一个 block（多个 token），而不是逐 token；
|
||||||
|
- **非因果注意力**：draft block 内部允许未来 token 参与当前 token 计算；
|
||||||
|
- **Markov head**：在 block 内引入轻量序列依赖；
|
||||||
|
- **目标模型复用**：DSpark draft 权重与目标模型共存（`mtp.{i}.*`），可共享 embedding/head。
|
||||||
|
|
||||||
|
相比 EAGLE，DSpark 的 draft 更接近目标模型结构，因此接受率通常更高，但 draft 前向本身的计算量也更大。DSpark 的优势在 **高并发、长输出** 场景下更明显；在低并发或短输出场景，draft 开销可能抵消收益。
|
||||||
|
|
||||||
|
### 1.4 评估目标
|
||||||
|
|
||||||
|
本报告系统评估：
|
||||||
|
|
||||||
|
1. **vLLM + DSpark** 在 DeepSeek-V4-Flash 上的推理优势；
|
||||||
|
2. 与 **vLLM 无投机解码** 基线的对比；
|
||||||
|
3. 与 **SGLang + EAGLE** 的对比；
|
||||||
|
4. 不同参数（`--spec-tokens`、并发度、输出长度）下的最优配置；
|
||||||
|
5. vLLM 与 SGLang 两个引擎部署 DSV4 的差异。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 测试环境
|
||||||
|
|
||||||
|
### 2.1 硬件
|
||||||
|
|
||||||
|
| 项目 | 配置 |
|
||||||
|
|---|---|
|
||||||
|
| GPU | 8× NVIDIA H200 143GB |
|
||||||
|
| 互联 | NVLink + NVSwitch |
|
||||||
|
| 驱动 | 575.57.08 |
|
||||||
|
| CUDA | 12.x |
|
||||||
|
|
||||||
|
### 2.2 软件版本
|
||||||
|
|
||||||
|
| 引擎 | 版本 | 路径 |
|
||||||
|
|---|---|---|
|
||||||
|
| SGLang | 0.5.14 | `/data/user1/yy/envs/sglang` |
|
||||||
|
| vLLM | 0.24.0 | `/data/user1/yy/envs/vllm` |
|
||||||
|
| vLLM-dspark | 0.23.1rc1.dev788+gfa4321de3 | `/data/user1/yy/envs/vllm-dspark` |
|
||||||
|
|
||||||
|
### 2.3 模型
|
||||||
|
|
||||||
|
| 模型 | 路径 | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| DeepSeek-V4-Flash | `/data/models/DeepSeek-V4-Flash` | 标准目标模型 |
|
||||||
|
| DeepSeek-V4-Flash-DSpark | `/data/models/DeepSeek-V4-Flash-DSpark` | 内含 DSpark draft 权重 |
|
||||||
|
|
||||||
|
### 2.4 数据集
|
||||||
|
|
||||||
|
- ShareGPT V4.3 unfiltered cleaned split：`/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json`
|
||||||
|
|
||||||
|
### 2.5 测试参数
|
||||||
|
|
||||||
|
- Tensor Parallel：8
|
||||||
|
- KV cache dtype：fp8
|
||||||
|
- Block size：256
|
||||||
|
- Max model len：auto
|
||||||
|
- Max num seqs：256
|
||||||
|
- 输出长度：256（固定）
|
||||||
|
- Prompt 数量：200
|
||||||
|
- 并发度：1, 16, 64
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 测试方法
|
||||||
|
|
||||||
|
### 3.1 服务启动
|
||||||
|
|
||||||
|
各引擎服务启动命令如下：
|
||||||
|
|
||||||
|
**vLLM-dspark + DSpark**
|
||||||
|
```bash
|
||||||
|
vllm serve /data/models/DeepSeek-V4-Flash-DSpark \
|
||||||
|
--trust-remote-code --tensor-parallel-size 8 \
|
||||||
|
--kv-cache-dtype fp8 --block-size 256 --max-model-len auto \
|
||||||
|
--max-num-seqs 256 --tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--spec-method dspark --spec-model /data/models/DeepSeek-V4-Flash-DSpark \
|
||||||
|
--spec-tokens 5 \
|
||||||
|
--no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log \
|
||||||
|
--port 30004
|
||||||
|
```
|
||||||
|
|
||||||
|
**vLLM-dspark 无投机解码**
|
||||||
|
```bash
|
||||||
|
vllm serve /data/models/DeepSeek-V4-Flash-DSpark \
|
||||||
|
--trust-remote-code --tensor-parallel-size 8 \
|
||||||
|
--kv-cache-dtype fp8 --block-size 256 --max-model-len auto \
|
||||||
|
--max-num-seqs 256 --tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log \
|
||||||
|
--port 30004
|
||||||
|
```
|
||||||
|
|
||||||
|
**vLLM 0.24.0 无投机解码**
|
||||||
|
```bash
|
||||||
|
vllm serve /data/models/DeepSeek-V4-Flash \
|
||||||
|
--trust-remote-code --tensor-parallel-size 8 \
|
||||||
|
--kv-cache-dtype fp8 --block-size 256 --max-model-len auto \
|
||||||
|
--max-num-seqs 256 --tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--disable-uvicorn-access-log --port 30005
|
||||||
|
```
|
||||||
|
|
||||||
|
**SGLang + EAGLE**
|
||||||
|
```bash
|
||||||
|
sglang serve --trust-remote-code --model-path /data/models/DeepSeek-V4-Flash \
|
||||||
|
--tp 8 --moe-runner-backend marlin \
|
||||||
|
--speculative-algorithm EAGLE --speculative-num-steps 3 \
|
||||||
|
--speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
|
||||||
|
--host 0.0.0.0 --port 30000
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 Benchmark 命令
|
||||||
|
|
||||||
|
使用各引擎自带的 serving benchmark：
|
||||||
|
|
||||||
|
**vLLM / vLLM-dspark**
|
||||||
|
```bash
|
||||||
|
vllm bench serve --host 127.0.0.1 --port <PORT> --backend openai \
|
||||||
|
--dataset-name sharegpt --dataset-path <DATASET> \
|
||||||
|
--sharegpt-output-len 256 --num-prompts 200 \
|
||||||
|
--max-concurrency <C> --endpoint /v1/completions \
|
||||||
|
--model <MODEL> --seed 42 --save-result --result-dir <DIR>
|
||||||
|
```
|
||||||
|
|
||||||
|
**SGLang**
|
||||||
|
```bash
|
||||||
|
python -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 30000 \
|
||||||
|
--dataset-name sharegpt --dataset-path <DATASET> \
|
||||||
|
--num-prompts 2000 --sharegpt-output-len 256 \
|
||||||
|
--max-concurrency <C> --model <MODEL> --seed 42 --output-file <FILE>
|
||||||
|
```
|
||||||
|
|
||||||
|
> 注：SGLang 历史测试使用 2000 prompts，vLLM-dspark / vLLM 本次测试使用 200 prompts，对比时主要关注相对趋势而非绝对数值。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 测试结果
|
||||||
|
|
||||||
|
### 4.1 总体对比（请求吞吐 req/s）
|
||||||
|
|
||||||
|
| 配置 | 引擎 | 投机方法 | 并发=1 | 并发=16 | 并发=64 | 并发=128 | 并发=256 |
|
||||||
|
|---|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| SGLang + EAGLE | sglang | EAGLE (4 draft) | 1.17 | 6.03 | 9.64 | 14.35 | 22.06 |
|
||||||
|
| vLLM-dspark + DSpark (st=3) | vllm-dspark | DSpark | 1.03 | 8.24 | **14.53** | - | - |
|
||||||
|
| vLLM-dspark + DSpark (st=5) | vllm-dspark | DSpark | 1.07 | 7.70 | 9.36 | - | - |
|
||||||
|
| vLLM-dspark + DSpark (st=7) | vllm-dspark | DSpark | 1.03 | 7.46 | 9.13 | - | - |
|
||||||
|
| vLLM-dspark 无投机 | vllm-dspark | 无 | 0.59 | 5.61 | 7.34 | - | - |
|
||||||
|
| vLLM 0.24.0 无投机 | vllm | 无 | 0.58 | 5.05 | 6.85 | - | - |
|
||||||
|
|
||||||
|
### 4.2 总体对比（输出 token 吞吐 tok/s）
|
||||||
|
|
||||||
|
| 配置 | 并发=1 | 并发=16 | 并发=64 | 并发=128 | 并发=256 |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| SGLang + EAGLE | 243.51 | 1259.0 | 2012.43 | 2996.37 | 4608.35 |
|
||||||
|
| vLLM-dspark + DSpark (st=3) | 254.39 | 1992.18 | **3507.81** | - | - |
|
||||||
|
| vLLM-dspark + DSpark (st=5) | 257.18 | 1862.34 | 2210.46 | - | - |
|
||||||
|
| vLLM-dspark + DSpark (st=7) | 251.58 | 1772.30 | 2218.58 | - | - |
|
||||||
|
| vLLM-dspark 无投机 | 144.62 | 1364.00 | 1803.98 | - | - |
|
||||||
|
| vLLM 0.24.0 无投机 | 138.26 | 1180.98 | 1641.89 | - | - |
|
||||||
|
|
||||||
|
### 4.3 vLLM-dspark 不同 `--spec-tokens` 对比
|
||||||
|
|
||||||
|
| spec-tokens | 并发 | req/s | out tok/s | 平均接受长度 | 接受率 | mean TTFT (ms) | mean TPOT (ms) | mean ITL (ms) |
|
||||||
|
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| 3 | 1 | 1.03 | 254.39 | 2.22 | 40.56% | 78.07 | 3.64 | 8.00 |
|
||||||
|
| 3 | 16 | 8.24 | 1992.18 | 2.19 | 39.66% | 90.90 | 7.63 | 16.23 |
|
||||||
|
| 3 | 64 | **14.53** | **3507.81** | 2.16 | 38.71% | 355.29 | 15.84 | 33.13 |
|
||||||
|
| 5 | 1 | 1.07 | 257.18 | 2.44 | 28.71% | 88.43 | 3.53 | 8.56 |
|
||||||
|
| 5 | 16 | 7.70 | 1862.34 | 2.39 | 27.87% | 92.23 | 8.19 | 18.90 |
|
||||||
|
| 5 | 64 | 9.36 | 2210.46 | 2.40 | 28.07% | 453.06 | 26.19 | 60.10 |
|
||||||
|
| 7 | 1 | 1.03 | 251.58 | 2.43 | 20.45% | 73.16 | 3.70 | 8.92 |
|
||||||
|
| 7 | 16 | 7.46 | 1772.30 | 2.41 | 20.10% | 106.43 | 8.92 | 20.07 |
|
||||||
|
| 7 | 64 | 9.13 | 2218.58 | 2.42 | 20.25% | 351.01 | 26.25 | 61.33 |
|
||||||
|
|
||||||
|
> 数据来源：`/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st{3,5,7}_c{1,16,64}.json`
|
||||||
|
|
||||||
|
### 4.4 延迟指标对比
|
||||||
|
|
||||||
|
| 配置 | 并发 | mean TTFT (ms) | mean TPOT (ms) | mean ITL (ms) |
|
||||||
|
|---|---:|---:|---:|---:|
|
||||||
|
| vLLM-dspark + DSpark (st=3) | 16 | 90.90 | 7.63 | 16.23 |
|
||||||
|
| vLLM-dspark + DSpark (st=5) | 16 | 92.23 | 8.19 | 18.90 |
|
||||||
|
| vLLM-dspark + DSpark (st=7) | 16 | 106.43 | 8.92 | 20.07 |
|
||||||
|
| vLLM-dspark 无投机 | 16 | 113.49 | 11.06 | 11.02 |
|
||||||
|
| vLLM 0.24.0 无投机 | 16 | 332.87 | 11.73 | 11.74 |
|
||||||
|
| SGLang + EAGLE | 16 | 168.95 | 12.67 | - |
|
||||||
|
|
||||||
|
### 4.5 投机解码 vs 无投机解码加速比
|
||||||
|
|
||||||
|
以 vLLM-dspark 无投机为基线：
|
||||||
|
|
||||||
|
| 配置 | 并发=1 | 并发=16 | 并发=64 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| DSpark st=3 | 1.75× | 1.47× | **1.98×** |
|
||||||
|
| DSpark st=5 | 1.80× | 1.37× | 1.28× |
|
||||||
|
| DSpark st=7 | 1.74× | 1.33× | 1.24× |
|
||||||
|
|
||||||
|
以 vLLM 0.24.0 无投机为基线：
|
||||||
|
|
||||||
|
| 配置 | 并发=1 | 并发=16 | 并发=64 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| DSpark st=3 | 1.78× | 1.63× | 2.12× |
|
||||||
|
| DSpark st=5 | 1.84× | 1.52× | 1.37× |
|
||||||
|
| DSpark st=7 | 1.77× | 1.48× | 1.33× |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 结果分析
|
||||||
|
|
||||||
|
### 5.1 DSpark 的优势场景
|
||||||
|
|
||||||
|
- **高并发**：当 batch size 足够大时，DSpark 的 draft 计算可以被充分并行化，接受率带来的收益超过 draft 开销。`--spec-tokens 3` 在并发 64 时达到 14.53 req/s，几乎是 vLLM-dspark 无投机的 2 倍；
|
||||||
|
- **长输出**：输出 token 越多，投机解码节省的 target 前向次数越多；
|
||||||
|
- **低延迟与高吞吐兼得**：在并发 16 时，DSpark st=3 的 mean TTFT 仅 90.90 ms，mean TPOT 7.63 ms，均优于无投机基线和 SGLang EAGLE。
|
||||||
|
|
||||||
|
### 5.2 DSpark 的劣势场景
|
||||||
|
|
||||||
|
- **单请求**：虽然单请求也有 1.7-1.8× 加速，但加速比主要来自接受率；draft 模型的固定开销仍然存在；
|
||||||
|
- **`--spec-tokens` 过大**：st=5 和 st=7 在高并发（64）下性能反而下降。原因是 draft tokens 数量增加后，验证阶段的计算量和 KV cache 压力增大，而接受率并未提升（st=7 接受率仅 20%）；
|
||||||
|
- **TTFT 抖动**：高并发下 DSpark 的 P99 TTFT 明显升高（st=5 c=64 时 P99 TTFT 达 5381 ms），说明调度/内存压力较大。
|
||||||
|
|
||||||
|
### 5.3 不同 `--spec-tokens` 的选择
|
||||||
|
|
||||||
|
| spec-tokens | 最佳并发 | 表现 |
|
||||||
|
|---|---|---|
|
||||||
|
| 3 | 高并发（64+） | 接受率最高（~40%），吞吐最高，验证开销最小 |
|
||||||
|
| 5 | 中低并发（1-16） | 接受长度略高（2.4），单请求吞吐最优 |
|
||||||
|
| 7 | 不推荐 | 接受率下降（~20%），验证开销大，高并发下性能倒退 |
|
||||||
|
|
||||||
|
### 5.4 与 EAGLE 的对比
|
||||||
|
|
||||||
|
- **接受率**：EAGLE 接受长度稳定在 2.61，DSpark st=3 接受长度 2.16-2.22，但 DSpark 接受率（token 级别）达 38-40%，说明每段 draft 的质量并不低；
|
||||||
|
- **吞吐**：vLLM-dspark + DSpark st=3 在并发 64 时（14.53 req/s）已接近 SGLang EAGLE 在并发 128 时（14.35 req/s）。SGLang 在更高并发（256-512）下仍能通过扩大 batch 提升吞吐，但这是以极高的 TTFT 为代价的；
|
||||||
|
- **延迟**：vLLM-dspark DSpark 在中低并发下的 TTFT/TPOT 明显优于 SGLang EAGLE；
|
||||||
|
- **实现成熟度**：SGLang 的 EAGLE 实现更成熟稳定；vLLM-dspark 的 DSpark 仍在快速迭代，部分参数组合（如 st=5/7 @ c=64）出现性能倒退。
|
||||||
|
|
||||||
|
### 5.5 vLLM-dspark vs vLLM 0.24.0
|
||||||
|
|
||||||
|
- 在完全无投机解码的情况下，vLLM-dspark（0.23.1rc）比 vLLM 0.24.0 更快：
|
||||||
|
- 并发 1：0.59 vs 0.58 req/s（接近）
|
||||||
|
- 并发 16：5.61 vs 5.05 req/s（+11%）
|
||||||
|
- 并发 64：7.34 vs 6.85 req/s（+7%）
|
||||||
|
- 这说明 vLLM-dspark 分支对 DeepSeek-V4 的 MLA / MoE / sparse attention 路径有更针对性的优化。
|
||||||
|
|
||||||
|
### 5.6 vLLM vs SGLang 引擎差异
|
||||||
|
|
||||||
|
- **启动与兼容性**：vLLM-dspark 对 DSV4 支持更直接，SGLang 对 EAGLE 支持更成熟；
|
||||||
|
- **调度策略**：SGLang 的 radix attention / chunked prefill 在长上下文场景有优势；
|
||||||
|
- **性能天花板**：SGLang EAGLE 在极高并发（512+）下吞吐更高，但延迟失控；vLLM-dspark DSpark 在中高并发（16-64）下延迟更优；
|
||||||
|
- **数据可比性**：SGLang 历史测试使用 2000 prompts，vLLM 本次仅 200 prompts，绝对数值需谨慎对比。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 最优参数建议
|
||||||
|
|
||||||
|
### 6.1 低延迟场景（单用户 / 低并发）
|
||||||
|
|
||||||
|
- 推荐：**DSpark `--spec-tokens 5`**；
|
||||||
|
- 原因：单请求下 st=5 的 req/s 最高（1.07），TPOT/ITL 与 st=3/7 接近，接受长度最长（2.44）。
|
||||||
|
|
||||||
|
### 6.2 高吞吐场景（高并发 / 在线服务）
|
||||||
|
|
||||||
|
- 推荐：**DSpark `--spec-tokens 3`**，`--max-num-seqs 256`，`--max-concurrency 64-128`；
|
||||||
|
- 原因：st=3 在高并发下接受率最高、验证开销最小，实测吞吐最高（14.53 req/s，3507 tok/s）。
|
||||||
|
|
||||||
|
### 6.3 长输出场景
|
||||||
|
|
||||||
|
- 推荐：开启 DSpark，`--spec-tokens 3-5`；
|
||||||
|
- 原因：输出越长，每次投机成功节省的 target 前向越多。st=3 更适合高并发，st=5 更适合中低并发。
|
||||||
|
|
||||||
|
### 6.4 短输出场景
|
||||||
|
|
||||||
|
- 推荐：**关闭投机解码** 或使用 `--spec-tokens 3`；
|
||||||
|
- 原因：短输出下投机收益有限，且 st=5/7 的验证开销可能抵消收益。
|
||||||
|
|
||||||
|
### 6.5 避免的参数组合
|
||||||
|
|
||||||
|
- **避免 `--spec-tokens 7` @ 高并发**：接受率仅 20%，高并发下吞吐不如 st=3/5；
|
||||||
|
- **避免 `--spec-tokens 5` @ 极高并发**：P99 TTFT 大幅升高，服务质量下降。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 结论
|
||||||
|
|
||||||
|
1. **vLLM-dspark + DSpark 在 DeepSeek-V4-Flash 上显著优于无投机解码基线**，最佳配置（st=3, c=64）比 vLLM-dspark 无投机快 **1.98×**，比 vLLM 0.24.0 无投机快 **2.12×**；
|
||||||
|
2. **DSpark 的最佳 `--spec-tokens` 取决于并发度**：
|
||||||
|
- 低并发（1-16）：`--spec-tokens 5` 综合最优；
|
||||||
|
- 高并发（64+）：`--spec-tokens 3` 综合最优；
|
||||||
|
3. **与 SGLang EAGLE 相比**，vLLM-dspark DSpark 在中高并发下延迟更优，吞吐接近；SGLang 在极限并发下吞吐更高但延迟失控；
|
||||||
|
4. **vLLM-dspark 分支对 DSV4 的优化效果明显**，即使无投机解码也比 vLLM 0.24.0 主分支更快；
|
||||||
|
5. **DSpark 目前仍不够稳定**，部分参数组合（st=5/7 @ c=64）出现性能倒退和高 TTFT 抖动，生产部署前需要针对实际负载调参。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 附录
|
||||||
|
|
||||||
|
### 8.1 原始数据文件
|
||||||
|
|
||||||
|
- vLLM-dspark / vLLM 测试结果：`/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/`
|
||||||
|
- SGLang 历史结果：
|
||||||
|
- `/data/user1/yy/bench_results/sglang_8card_systematic_20260704_120819/`
|
||||||
|
- `/data/user1/yy/bench_results/sglang_8card_max_throughput_20260705_030839/`
|
||||||
|
|
||||||
|
### 8.2 测试脚本
|
||||||
|
|
||||||
|
- 主控脚本：`/data/user1/yy/bench_dsv4_comparison.py`
|
||||||
|
- 指标提取脚本：`/tmp/extract_dsv4_metrics.py`
|
||||||
|
|
||||||
|
### 8.3 已知限制
|
||||||
|
|
||||||
|
- vLLM-dspark 的 DSpark 实现较新，部分参数组合可能不稳定；
|
||||||
|
- 本次测试未覆盖 PD 分离、前缀缓存、不同输出长度等高级特性；
|
||||||
|
- SGLang 与 vLLM 的 prompt 数量不一致（2000 vs 200），绝对数值对比仅供参考；
|
||||||
|
- 不同数据集和 prompt 长度分布会影响结果。
|
||||||
33
scripts/SLO_STANDARDS.md
Normal file
33
scripts/SLO_STANDARDS.md
Normal file
@ -0,0 +1,33 @@
|
|||||||
|
# 推理服务 SLO 标准
|
||||||
|
|
||||||
|
> 记录各模型层级（Tier）的延迟目标。TTFT 以 **P95** 为主要关注指标。
|
||||||
|
|
||||||
|
## 模型层级定义
|
||||||
|
|
||||||
|
| 层级 | 名称 | 参数规模 | 典型模型 | 应用场景 | 核心目标 |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| S0 | 轻量层 | < 30B | Llama 3.1 8B Instruct | 高并发轻推理 | 高并发轻推理 |
|
||||||
|
| S1 | 实时交互层 | [30B, 100B) | Llama 3.3 70B Instruct | Chat / Copilot / Agent | 极低延迟 |
|
||||||
|
| S2 | 均衡服务层 | [100B, 500B) | DeepSeek V4 Flash、MiniMax M2.7 | 企业 API | 性价比平衡 |
|
||||||
|
| S3 | 深度推理层 | [500B, 1T) | DeepSeek V3.2 | 高复杂推理 | 稳定吞吐 |
|
||||||
|
| S4 | 超级推理层 | ≥ 1T | DS V4 Pro | 深度思考 | 智能优先 |
|
||||||
|
|
||||||
|
## TTFT SLO（P95 为主要关注指标）
|
||||||
|
|
||||||
|
| 指标 | S0 | S1 | S2 | S3 | S4 |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| P50 | < 500ms | < 800ms | < 1.5s | < 2s | < 1.5s |
|
||||||
|
| P95 | < 0.4s | < 1s | < 3s | < 5s | < 3s |
|
||||||
|
|
||||||
|
## TPOT SLO
|
||||||
|
|
||||||
|
| 指标 | S0 | S1 | S2 | S3 | S4 |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| TPOT | < 20ms | < 40ms | < 50ms | < 50ms | < 50ms |
|
||||||
|
|
||||||
|
## 说明
|
||||||
|
|
||||||
|
- **TTFT**（Time To First Token）：从请求到达服务端到首个 token 返回的延迟。
|
||||||
|
- **TPOT**（Time Per Output Token）：除首 token 外，平均每输出一个 token 的耗时。
|
||||||
|
- 当前重点关注 **TTFT P95**，即 95% 请求的 TTFT 应满足对应层级的阈值。
|
||||||
|
- _benchmark 输出应同时给出 P50 / P90 / P95 / P99 TTFT，但评估以 P95 为准。_
|
||||||
87
scripts/analyze_dspark_nsys.py
Normal file
87
scripts/analyze_dspark_nsys.py
Normal file
@ -0,0 +1,87 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Analyze nsys traces for DSpark profiling.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python scripts/analyze_dspark_nsys.py <nsys-rep-file>
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def run_nsys_report(rep_file: Path, report_name: str) -> str:
|
||||||
|
cmd = [
|
||||||
|
"nsys", "stats",
|
||||||
|
"--report", report_name,
|
||||||
|
"--format", "json",
|
||||||
|
str(rep_file),
|
||||||
|
]
|
||||||
|
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||||
|
if result.returncode != 0:
|
||||||
|
print(f"nsys {report_name} failed: {result.stderr}")
|
||||||
|
return ""
|
||||||
|
return result.stdout
|
||||||
|
|
||||||
|
|
||||||
|
def summarize_cuda_sum(rep_file: Path) -> list[dict]:
|
||||||
|
"""Return top CUDA kernels by total time."""
|
||||||
|
text = run_nsys_report(rep_file, "cuda_kernel_sum")
|
||||||
|
if not text:
|
||||||
|
return []
|
||||||
|
# nsys stats --format json outputs JSON after some header lines.
|
||||||
|
# Find the first '[' character.
|
||||||
|
start = text.find("[")
|
||||||
|
if start < 0:
|
||||||
|
return []
|
||||||
|
data = json.loads(text[start:])
|
||||||
|
# Sort by total time descending.
|
||||||
|
rows = data[1:] if len(data) > 1 else data
|
||||||
|
rows_sorted = sorted(rows, key=lambda x: x.get("Total Time (ns)", 0), reverse=True)
|
||||||
|
return rows_sorted[:50]
|
||||||
|
|
||||||
|
|
||||||
|
def summarize_nvtx_sum(rep_file: Path) -> list[dict]:
|
||||||
|
text = run_nsys_report(rep_file, "nvtx_sum")
|
||||||
|
start = text.find("[")
|
||||||
|
if start < 0:
|
||||||
|
return []
|
||||||
|
data = json.loads(text[start:])
|
||||||
|
rows = data[1:] if len(data) > 1 else data
|
||||||
|
rows_sorted = sorted(rows, key=lambda x: x.get("Total Time (ns)", 0), reverse=True)
|
||||||
|
return rows_sorted[:30]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
if len(sys.argv) < 2:
|
||||||
|
print("Usage: analyze_dspark_nsys.py <nsys-rep-file>")
|
||||||
|
sys.exit(1)
|
||||||
|
rep_file = Path(sys.argv[1])
|
||||||
|
if not rep_file.exists():
|
||||||
|
print(f"File not found: {rep_file}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
print(f"Analyzing {rep_file}...\n")
|
||||||
|
|
||||||
|
cuda_kernels = summarize_cuda_sum(rep_file)
|
||||||
|
print("=== Top CUDA Kernels by Total Time ===")
|
||||||
|
for row in cuda_kernels[:20]:
|
||||||
|
name = row.get("Name", "N/A")
|
||||||
|
total_ns = row.get("Total Time (ns)", 0)
|
||||||
|
count = row.get("Instances", 0)
|
||||||
|
avg_ns = row.get("Avg (ns)", 0)
|
||||||
|
print(f" {total_ns/1e6:8.2f} ms {count:6d} calls avg={avg_ns/1e6:6.3f} ms {name[:80]}")
|
||||||
|
|
||||||
|
nvtx = summarize_nvtx_sum(rep_file)
|
||||||
|
if nvtx:
|
||||||
|
print("\n=== Top NVTX Ranges ===")
|
||||||
|
for row in nvtx[:20]:
|
||||||
|
name = row.get("Range", "N/A")
|
||||||
|
total_ns = row.get("Total Time (ns)", 0)
|
||||||
|
count = row.get("Instances", 0)
|
||||||
|
print(f" {total_ns/1e6:8.2f} ms {count:6d} calls {name[:80]}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
323
scripts/bench_dsv4_comparison.py
Normal file
323
scripts/bench_dsv4_comparison.py
Normal file
@ -0,0 +1,323 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Benchmark orchestration for DeepSeek-V4 inference comparison.
|
||||||
|
|
||||||
|
Compares:
|
||||||
|
- vllm-dspark + DeepSeek-V4-Flash-DSpark + DSpark (various spec-tokens)
|
||||||
|
- vllm-dspark + DeepSeek-V4-Flash-DSpark without spec decode
|
||||||
|
- vllm (0.24.0) + DeepSeek-V4-Flash without spec decode
|
||||||
|
- sglang + DeepSeek-V4-Flash + EAGLE (reuse existing results)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import urllib.request
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path("/data/user1/yy")
|
||||||
|
RESULT_DIR = ROOT / "bench_results" / f"dsv4_comparison_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
|
||||||
|
RESULT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
DATASET = "/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"
|
||||||
|
NUM_PROMPTS = 200
|
||||||
|
SEED = 42
|
||||||
|
OUTPUT_LEN = 256
|
||||||
|
HOST = "127.0.0.1"
|
||||||
|
|
||||||
|
# Service configs to benchmark
|
||||||
|
SERVICES = [
|
||||||
|
{
|
||||||
|
"name": "vllm-dspark-dspark-st5",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"port": 30004,
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 5,
|
||||||
|
"cmd": [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve",
|
||||||
|
"/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--tensor-parallel-size", "8",
|
||||||
|
"--kv-cache-dtype", "fp8",
|
||||||
|
"--block-size", "256",
|
||||||
|
"--max-model-len", "auto",
|
||||||
|
"--max-num-seqs", "256",
|
||||||
|
"--tokenizer-mode", "deepseek_v4",
|
||||||
|
"--reasoning-parser", "deepseek_v4",
|
||||||
|
"--spec-method", "dspark",
|
||||||
|
"--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--spec-tokens", "5",
|
||||||
|
"--no-disable-hybrid-kv-cache-manager",
|
||||||
|
"--disable-uvicorn-access-log",
|
||||||
|
"--port", "30004",
|
||||||
|
],
|
||||||
|
"env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")},
|
||||||
|
"bench_backend": "openai",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "vllm-dspark-nospec",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"port": 30004,
|
||||||
|
"spec_method": None,
|
||||||
|
"cmd": [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve",
|
||||||
|
"/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--tensor-parallel-size", "8",
|
||||||
|
"--kv-cache-dtype", "fp8",
|
||||||
|
"--block-size", "256",
|
||||||
|
"--max-model-len", "auto",
|
||||||
|
"--max-num-seqs", "256",
|
||||||
|
"--tokenizer-mode", "deepseek_v4",
|
||||||
|
"--reasoning-parser", "deepseek_v4",
|
||||||
|
"--no-disable-hybrid-kv-cache-manager",
|
||||||
|
"--disable-uvicorn-access-log",
|
||||||
|
"--port", "30004",
|
||||||
|
],
|
||||||
|
"env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")},
|
||||||
|
"bench_backend": "openai",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "vllm-dspark-dspark-st3",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"port": 30004,
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 3,
|
||||||
|
"cmd": [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve",
|
||||||
|
"/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--tensor-parallel-size", "8",
|
||||||
|
"--kv-cache-dtype", "fp8",
|
||||||
|
"--block-size", "256",
|
||||||
|
"--max-model-len", "auto",
|
||||||
|
"--max-num-seqs", "256",
|
||||||
|
"--tokenizer-mode", "deepseek_v4",
|
||||||
|
"--reasoning-parser", "deepseek_v4",
|
||||||
|
"--spec-method", "dspark",
|
||||||
|
"--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--spec-tokens", "3",
|
||||||
|
"--no-disable-hybrid-kv-cache-manager",
|
||||||
|
"--disable-uvicorn-access-log",
|
||||||
|
"--port", "30004",
|
||||||
|
],
|
||||||
|
"env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")},
|
||||||
|
"bench_backend": "openai",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "vllm-dspark-dspark-st7",
|
||||||
|
"engine": "vllm-dspark",
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"port": 30004,
|
||||||
|
"spec_method": "dspark",
|
||||||
|
"spec_tokens": 7,
|
||||||
|
"cmd": [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve",
|
||||||
|
"/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--tensor-parallel-size", "8",
|
||||||
|
"--kv-cache-dtype", "fp8",
|
||||||
|
"--block-size", "256",
|
||||||
|
"--max-model-len", "auto",
|
||||||
|
"--max-num-seqs", "256",
|
||||||
|
"--tokenizer-mode", "deepseek_v4",
|
||||||
|
"--reasoning-parser", "deepseek_v4",
|
||||||
|
"--spec-method", "dspark",
|
||||||
|
"--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--spec-tokens", "7",
|
||||||
|
"--no-disable-hybrid-kv-cache-manager",
|
||||||
|
"--disable-uvicorn-access-log",
|
||||||
|
"--port", "30004",
|
||||||
|
],
|
||||||
|
"env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")},
|
||||||
|
"bench_backend": "openai",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "vllm-main-nospec",
|
||||||
|
"engine": "vllm",
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash",
|
||||||
|
"port": 30005,
|
||||||
|
"spec_method": None,
|
||||||
|
"cmd": [
|
||||||
|
"/data/user1/yy/envs/vllm/bin/vllm", "serve",
|
||||||
|
"/data/models/DeepSeek-V4-Flash",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--tensor-parallel-size", "8",
|
||||||
|
"--kv-cache-dtype", "fp8",
|
||||||
|
"--block-size", "256",
|
||||||
|
"--max-model-len", "auto",
|
||||||
|
"--max-num-seqs", "256",
|
||||||
|
"--tokenizer-mode", "deepseek_v4",
|
||||||
|
"--reasoning-parser", "deepseek_v4",
|
||||||
|
"--disable-uvicorn-access-log",
|
||||||
|
"--port", "30005",
|
||||||
|
],
|
||||||
|
"env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")},
|
||||||
|
"bench_backend": "openai",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
CONCURRENCIES = [1, 16, 64]
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
print(f"[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] {msg}", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def wait_for_health(port, timeout=300):
|
||||||
|
url = f"http://{HOST}:{port}/health"
|
||||||
|
start = time.time()
|
||||||
|
while time.time() - start < timeout:
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(url, timeout=5) as resp:
|
||||||
|
if resp.status == 200:
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
time.sleep(2)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def start_service(service):
|
||||||
|
log(f"Starting {service['name']} on port {service['port']}...")
|
||||||
|
env = os.environ.copy()
|
||||||
|
env.update(service["env"])
|
||||||
|
log_file = RESULT_DIR / f"{service['name']}_service.log"
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
service["cmd"],
|
||||||
|
stdout=open(log_file, "w"),
|
||||||
|
stderr=subprocess.STDOUT,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
if not wait_for_health(service["port"]):
|
||||||
|
log(f"ERROR: {service['name']} failed to start")
|
||||||
|
proc.terminate()
|
||||||
|
return None
|
||||||
|
log(f"{service['name']} is ready")
|
||||||
|
return proc
|
||||||
|
|
||||||
|
|
||||||
|
def stop_service(proc, name):
|
||||||
|
if proc is None:
|
||||||
|
return
|
||||||
|
log(f"Stopping {name} (pid {proc.pid})...")
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=60)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
proc.wait()
|
||||||
|
log(f"{name} stopped")
|
||||||
|
|
||||||
|
|
||||||
|
def run_benchmark(service, concurrency):
|
||||||
|
name = service["name"]
|
||||||
|
port = service["port"]
|
||||||
|
backend = service["bench_backend"]
|
||||||
|
result_file = RESULT_DIR / f"{name}_c{concurrency}.json"
|
||||||
|
log_file = RESULT_DIR / f"{name}_c{concurrency}.log"
|
||||||
|
|
||||||
|
if service["engine"] == "vllm-dspark":
|
||||||
|
bench_cmd = [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "bench", "serve",
|
||||||
|
"--host", HOST,
|
||||||
|
"--port", str(port),
|
||||||
|
"--backend", backend,
|
||||||
|
"--dataset-name", "sharegpt",
|
||||||
|
"--dataset-path", DATASET,
|
||||||
|
"--sharegpt-output-len", str(OUTPUT_LEN),
|
||||||
|
"--num-prompts", str(NUM_PROMPTS),
|
||||||
|
"--max-concurrency", str(concurrency),
|
||||||
|
"--endpoint", "/v1/completions",
|
||||||
|
"--model", service["model"],
|
||||||
|
"--seed", str(SEED),
|
||||||
|
"--save-result",
|
||||||
|
"--result-dir", str(RESULT_DIR),
|
||||||
|
"--result-filename", result_file.name,
|
||||||
|
]
|
||||||
|
else:
|
||||||
|
bench_cmd = [
|
||||||
|
"/data/user1/yy/envs/vllm/bin/vllm", "bench", "serve",
|
||||||
|
"--host", HOST,
|
||||||
|
"--port", str(port),
|
||||||
|
"--backend", backend,
|
||||||
|
"--dataset-name", "sharegpt",
|
||||||
|
"--dataset-path", DATASET,
|
||||||
|
"--sharegpt-output-len", str(OUTPUT_LEN),
|
||||||
|
"--num-prompts", str(NUM_PROMPTS),
|
||||||
|
"--max-concurrency", str(concurrency),
|
||||||
|
"--endpoint", "/v1/completions",
|
||||||
|
"--model", service["model"],
|
||||||
|
"--seed", str(SEED),
|
||||||
|
"--save-result",
|
||||||
|
"--result-dir", str(RESULT_DIR),
|
||||||
|
"--result-filename", result_file.name,
|
||||||
|
]
|
||||||
|
|
||||||
|
log(f"Running benchmark {name} concurrency={concurrency}...")
|
||||||
|
start = time.time()
|
||||||
|
with open(log_file, "w") as f:
|
||||||
|
proc = subprocess.Popen(bench_cmd, stdout=f, stderr=subprocess.STDOUT)
|
||||||
|
proc.wait()
|
||||||
|
duration = time.time() - start
|
||||||
|
log(f"Benchmark {name} c={concurrency} finished in {duration:.1f}s, exit={proc.returncode}")
|
||||||
|
|
||||||
|
if result_file.exists():
|
||||||
|
with open(result_file) as f:
|
||||||
|
data = json.load(f)
|
||||||
|
return {
|
||||||
|
"service": name,
|
||||||
|
"engine": service["engine"],
|
||||||
|
"spec_method": service.get("spec_method"),
|
||||||
|
"spec_tokens": service.get("spec_tokens"),
|
||||||
|
"concurrency": concurrency,
|
||||||
|
"request_throughput": data.get("request_throughput"),
|
||||||
|
"output_throughput": data.get("output_throughput"),
|
||||||
|
"total_input_tokens": data.get("total_input_tokens"),
|
||||||
|
"total_output_tokens": data.get("total_output_tokens"),
|
||||||
|
"duration_s": data.get("duration_s"),
|
||||||
|
"result_file": str(result_file),
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
log(f"WARNING: result file {result_file} not found")
|
||||||
|
return {
|
||||||
|
"service": name,
|
||||||
|
"engine": service["engine"],
|
||||||
|
"spec_method": service.get("spec_method"),
|
||||||
|
"spec_tokens": service.get("spec_tokens"),
|
||||||
|
"concurrency": concurrency,
|
||||||
|
"error": "result file missing",
|
||||||
|
"log_file": str(log_file),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
summary = []
|
||||||
|
for service in SERVICES:
|
||||||
|
proc = start_service(service)
|
||||||
|
if proc is None:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
for concurrency in CONCURRENCIES:
|
||||||
|
result = run_benchmark(service, concurrency)
|
||||||
|
summary.append(result)
|
||||||
|
# Save incremental summary
|
||||||
|
with open(RESULT_DIR / "summary.json", "w") as f:
|
||||||
|
json.dump(summary, f, indent=2)
|
||||||
|
finally:
|
||||||
|
stop_service(proc, service["name"])
|
||||||
|
# Small gap between services
|
||||||
|
time.sleep(10)
|
||||||
|
|
||||||
|
log(f"All benchmarks complete. Results in {RESULT_DIR}")
|
||||||
|
with open(RESULT_DIR / "summary.json", "w") as f:
|
||||||
|
json.dump(summary, f, indent=2)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
266
scripts/bench_vllm_dspark_qwen3.py
Normal file
266
scripts/bench_vllm_dspark_qwen3.py
Normal file
@ -0,0 +1,266 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Throughput benchmark for vLLM DSpark service (Qwen3-4B + dspark_qwen3_4b_block7)."""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import random
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import List
|
||||||
|
|
||||||
|
import aiohttp
|
||||||
|
import numpy as np
|
||||||
|
from transformers import AutoTokenizer
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class RequestResult:
|
||||||
|
prompt_len: int = 0
|
||||||
|
output_len: int = 0
|
||||||
|
ttft_ms: float = 0.0
|
||||||
|
tpot_ms: float = 0.0
|
||||||
|
e2e_ms: float = 0.0
|
||||||
|
success: bool = False
|
||||||
|
error: str = ""
|
||||||
|
|
||||||
|
|
||||||
|
async def async_request(
|
||||||
|
session: aiohttp.ClientSession,
|
||||||
|
url: str,
|
||||||
|
model_name: str,
|
||||||
|
prompt: str,
|
||||||
|
max_tokens: int,
|
||||||
|
prompt_len: int,
|
||||||
|
result: RequestResult,
|
||||||
|
) -> None:
|
||||||
|
payload = {
|
||||||
|
"model": model_name,
|
||||||
|
"prompt": prompt,
|
||||||
|
"max_tokens": max_tokens,
|
||||||
|
"temperature": 0.0,
|
||||||
|
"stream": True,
|
||||||
|
"stream_options": {"include_usage": True},
|
||||||
|
}
|
||||||
|
result.prompt_len = prompt_len
|
||||||
|
start = time.perf_counter()
|
||||||
|
first_token_time = None
|
||||||
|
token_times: List[float] = []
|
||||||
|
output_text = ""
|
||||||
|
try:
|
||||||
|
async with session.post(url, json=payload) as resp:
|
||||||
|
resp.raise_for_status()
|
||||||
|
async for line in resp.content:
|
||||||
|
line = line.decode("utf-8").strip()
|
||||||
|
if not line or line.startswith(":"):
|
||||||
|
continue
|
||||||
|
if line.startswith("data: "):
|
||||||
|
data = line[len("data: "):]
|
||||||
|
if data == "[DONE]":
|
||||||
|
break
|
||||||
|
chunk = json.loads(data)
|
||||||
|
choices = chunk.get("choices", [])
|
||||||
|
if choices:
|
||||||
|
delta = choices[0].get("text", "")
|
||||||
|
if delta:
|
||||||
|
now = time.perf_counter()
|
||||||
|
token_times.append(now)
|
||||||
|
output_text += delta
|
||||||
|
if first_token_time is None:
|
||||||
|
first_token_time = now
|
||||||
|
end = time.perf_counter()
|
||||||
|
result.e2e_ms = (end - start) * 1000.0
|
||||||
|
if first_token_time is not None:
|
||||||
|
result.ttft_ms = (first_token_time - start) * 1000.0
|
||||||
|
if len(token_times) >= 2:
|
||||||
|
# TPOT: average time between consecutive tokens
|
||||||
|
intervals = [
|
||||||
|
token_times[i] - token_times[i - 1]
|
||||||
|
for i in range(1, len(token_times))
|
||||||
|
]
|
||||||
|
result.tpot_ms = sum(intervals) / len(intervals) * 1000.0
|
||||||
|
result.output_len = len(token_times)
|
||||||
|
result.success = True
|
||||||
|
except Exception as e:
|
||||||
|
result.e2e_ms = (time.perf_counter() - start) * 1000.0
|
||||||
|
result.error = str(e)
|
||||||
|
result.success = False
|
||||||
|
|
||||||
|
|
||||||
|
async def run_concurrency_benchmark(
|
||||||
|
url: str,
|
||||||
|
model_name: str,
|
||||||
|
prompts: List[str],
|
||||||
|
prompt_lens: List[int],
|
||||||
|
max_tokens: int,
|
||||||
|
concurrency: int,
|
||||||
|
num_prompts: int,
|
||||||
|
seed: int,
|
||||||
|
) -> dict:
|
||||||
|
rng = random.Random(seed)
|
||||||
|
indices = [rng.randrange(len(prompts)) for _ in range(num_prompts)]
|
||||||
|
|
||||||
|
semaphore = asyncio.Semaphore(concurrency)
|
||||||
|
results: List[RequestResult] = [RequestResult() for _ in range(num_prompts)]
|
||||||
|
|
||||||
|
async def bounded_request(idx: int, i: int):
|
||||||
|
async with semaphore:
|
||||||
|
async with aiohttp.ClientSession() as session:
|
||||||
|
await async_request(
|
||||||
|
session,
|
||||||
|
url,
|
||||||
|
model_name,
|
||||||
|
prompts[idx],
|
||||||
|
max_tokens,
|
||||||
|
prompt_lens[idx],
|
||||||
|
results[i],
|
||||||
|
)
|
||||||
|
|
||||||
|
start = time.perf_counter()
|
||||||
|
await asyncio.gather(*[bounded_request(idx, i) for i, idx in enumerate(indices)])
|
||||||
|
duration = time.perf_counter() - start
|
||||||
|
|
||||||
|
successes = [r for r in results if r.success]
|
||||||
|
failures = [r for r in results if not r.success]
|
||||||
|
if not successes:
|
||||||
|
return {
|
||||||
|
"concurrency": concurrency,
|
||||||
|
"num_prompts": num_prompts,
|
||||||
|
"duration_s": duration,
|
||||||
|
"success_count": 0,
|
||||||
|
"error": "all requests failed",
|
||||||
|
"errors": [r.error for r in failures[:5]],
|
||||||
|
}
|
||||||
|
|
||||||
|
total_in_tokens = sum(r.prompt_len for r in successes)
|
||||||
|
total_out_tokens = sum(r.output_len for r in successes)
|
||||||
|
total_tokens = total_in_tokens + total_out_tokens
|
||||||
|
|
||||||
|
ttfts = [r.ttft_ms for r in successes]
|
||||||
|
tpots = [r.tpot_ms for r in successes]
|
||||||
|
e2es = [r.e2e_ms for r in successes]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"concurrency": concurrency,
|
||||||
|
"num_prompts": num_prompts,
|
||||||
|
"duration_s": duration,
|
||||||
|
"success_count": len(successes),
|
||||||
|
"fail_count": len(failures),
|
||||||
|
"input_tokens": total_in_tokens,
|
||||||
|
"output_tokens": total_out_tokens,
|
||||||
|
"total_tokens": total_tokens,
|
||||||
|
"request_throughput": len(successes) / duration,
|
||||||
|
"input_throughput": total_in_tokens / duration,
|
||||||
|
"output_throughput": total_out_tokens / duration,
|
||||||
|
"total_throughput": total_tokens / duration,
|
||||||
|
"mean_ttft_ms": float(np.mean(ttfts)),
|
||||||
|
"p50_ttft_ms": float(np.percentile(ttfts, 50)),
|
||||||
|
"p99_ttft_ms": float(np.percentile(ttfts, 99)),
|
||||||
|
"mean_tpot_ms": float(np.mean(tpots)),
|
||||||
|
"p50_tpot_ms": float(np.percentile(tpots, 50)),
|
||||||
|
"p99_tpot_ms": float(np.percentile(tpots, 99)),
|
||||||
|
"mean_e2e_ms": float(np.mean(e2es)),
|
||||||
|
"p50_e2e_ms": float(np.percentile(e2es, 50)),
|
||||||
|
"p99_e2e_ms": float(np.percentile(e2es, 99)),
|
||||||
|
"errors": [r.error for r in failures[:5]],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_sharegpt_prompts(path: str, tokenizer, max_samples: int = 2000):
|
||||||
|
with open(path, "r") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
prompts = []
|
||||||
|
lens = []
|
||||||
|
for item in data:
|
||||||
|
conv = item.get("conversations", [])
|
||||||
|
for turn in conv:
|
||||||
|
if turn.get("from") == "human":
|
||||||
|
prompt = turn.get("value", "")
|
||||||
|
if prompt:
|
||||||
|
prompts.append(prompt)
|
||||||
|
lens.append(len(tokenizer.encode(prompt)))
|
||||||
|
break
|
||||||
|
if len(prompts) >= max_samples:
|
||||||
|
break
|
||||||
|
return prompts, lens
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--host", default="127.0.0.1")
|
||||||
|
parser.add_argument("--port", type=int, default=30003)
|
||||||
|
parser.add_argument("--model", default="/data/models/Qwen3-4B")
|
||||||
|
parser.add_argument("--dataset", default="/data/user1/yy/datasets/ShareGPT_filtered_chat.json")
|
||||||
|
parser.add_argument("--tokenizer", default="/data/models/Qwen3-4B")
|
||||||
|
parser.add_argument("--max-tokens", type=int, default=256)
|
||||||
|
parser.add_argument("--num-prompts", type=int, default=500)
|
||||||
|
parser.add_argument("--concurrency", type=int, nargs="+", default=[1, 4, 16, 64, 128])
|
||||||
|
parser.add_argument("--seed", type=int, default=42)
|
||||||
|
parser.add_argument("--output-dir", default="/data/user1/yy/bench_results")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
url = f"http://{args.host}:{args.port}/v1/completions"
|
||||||
|
print(f"Loading tokenizer from {args.tokenizer} ...")
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(args.tokenizer)
|
||||||
|
print(f"Loading dataset from {args.dataset} ...")
|
||||||
|
prompts, prompt_lens = load_sharegpt_prompts(args.dataset, tokenizer)
|
||||||
|
print(f"Loaded {len(prompts)} prompts, prompt lens: min={min(prompt_lens)}, max={max(prompt_lens)}, mean={sum(prompt_lens)/len(prompt_lens):.1f}")
|
||||||
|
|
||||||
|
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
|
result_dir = f"{args.output_dir}/vllm_dspark_qwen3_{timestamp}"
|
||||||
|
import os
|
||||||
|
os.makedirs(result_dir, exist_ok=True)
|
||||||
|
summary_file = f"{result_dir}/summary.json"
|
||||||
|
|
||||||
|
summary = {
|
||||||
|
"model": args.model,
|
||||||
|
"draft_model": "deepseek-ai/dspark_qwen3_4b_block7",
|
||||||
|
"url": url,
|
||||||
|
"max_tokens": args.max_tokens,
|
||||||
|
"num_prompts": args.num_prompts,
|
||||||
|
"dataset": args.dataset,
|
||||||
|
"timestamp": timestamp,
|
||||||
|
"results": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
print("\nStarting benchmark sweeps...")
|
||||||
|
print(f"{'Concurrency':>12} {'Req/s':>10} {'In tok/s':>12} {'Out tok/s':>12} {'Total tok/s':>13} {'TTFT(ms)':>10} {'TPOT(ms)':>10} {'E2E(ms)':>10}")
|
||||||
|
print("-" * 100)
|
||||||
|
|
||||||
|
for c in args.concurrency:
|
||||||
|
print(f"Running concurrency={c} ...", flush=True)
|
||||||
|
result = asyncio.run(
|
||||||
|
run_concurrency_benchmark(
|
||||||
|
url,
|
||||||
|
args.model,
|
||||||
|
prompts,
|
||||||
|
prompt_lens,
|
||||||
|
args.max_tokens,
|
||||||
|
c,
|
||||||
|
args.num_prompts,
|
||||||
|
args.seed,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
summary["results"].append(result)
|
||||||
|
with open(summary_file, "w") as f:
|
||||||
|
json.dump(summary, f, indent=2)
|
||||||
|
if result.get("success_count", 0) == 0:
|
||||||
|
print(f"{c:>12} FAILED: {result.get('error', 'unknown')}")
|
||||||
|
continue
|
||||||
|
print(
|
||||||
|
f"{c:>12} "
|
||||||
|
f"{result['request_throughput']:>10.2f} "
|
||||||
|
f"{result['input_throughput']:>12.2f} "
|
||||||
|
f"{result['output_throughput']:>12.2f} "
|
||||||
|
f"{result['total_throughput']:>13.2f} "
|
||||||
|
f"{result['mean_ttft_ms']:>10.1f} "
|
||||||
|
f"{result['mean_tpot_ms']:>10.1f} "
|
||||||
|
f"{result['mean_e2e_ms']:>10.1f}"
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"\nSummary saved to {summary_file}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
60
scripts/benchmark_dspark_0707/README.md
Normal file
60
scripts/benchmark_dspark_0707/README.md
Normal file
@ -0,0 +1,60 @@
|
|||||||
|
# DSpark Benchmark Scripts (2026-07-07)
|
||||||
|
|
||||||
|
## Location
|
||||||
|
|
||||||
|
`/data/user1/yy/scripts/benchmark_dspark_0707`
|
||||||
|
|
||||||
|
## Scripts
|
||||||
|
|
||||||
|
| Script | Purpose | Output Location |
|
||||||
|
|---|---|---|
|
||||||
|
| `bench_dspark_p1.sh` | Quick benchmark: chat_standard, generation_standard, summarization | `bench_results/dspark_grid_${RUN_ID}/p1_quick/...` |
|
||||||
|
| `bench_dspark_p2.sh` | Core benchmark: chat_short, rag_medium, long_context_probe | `bench_results/dspark_grid_${RUN_ID}/p2_core/...` |
|
||||||
|
| `bench_dspark_p3.sh` | Extension benchmark: stress_standard, decode_heavy | `bench_results/dspark_grid_${RUN_ID}/p3_extension/...` |
|
||||||
|
| `bench_dspark_focused.sh` | Focused sweep used for spec-tokens comparison | `bench_results/dspark_grid_${RUN_ID}/focused/...` or `bench_results/dspark_st_comparison_${RUN_ID}/focused/...` |
|
||||||
|
| `run_dspark_benchmark_grid.sh` | Orchestrator: starts vllm-dspark server, runs P1/P2/P3, stops server | `bench_results/dspark_grid_${RUN_ID}/` |
|
||||||
|
| `run_dspark_st_comparison.sh` | Orchestrator: compares `--spec-tokens 3` vs `5` | `bench_results/dspark_st_comparison_${RUN_ID}/` |
|
||||||
|
| `parse_results.py` | Parses grid logs and generates `report.md` | Writes to `bench_results/<result_root>/report.md` |
|
||||||
|
| `parse_st_comparison.py` | Parses spec-tokens comparison logs | Writes `comparison_report.md` under result root |
|
||||||
|
| `parse_eagle_vs_dspark.py` | Merges DSpark st comparison with SGLang EAGLE results | Writes `dspark_vs_eagle_report.md` under EAGLE result root |
|
||||||
|
|
||||||
|
## Common Environment
|
||||||
|
|
||||||
|
- Python env: `/data/user1/yy/envs/vllm-dspark` for server
|
||||||
|
- Benchmark client: `/data/user1/yy/envs/sglang/bin/python -m sglang.bench_serving --backend vllm`
|
||||||
|
- Model: `/data/models/DeepSeek-V4-Flash-DSpark`
|
||||||
|
- Default port: `30004`
|
||||||
|
- Default warmup: `100` requests
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Full grid benchmark
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Results land in `bench_results/dspark_grid_YYYYMMDD-HHMMSS/`.
|
||||||
|
|
||||||
|
### Spec-tokens comparison
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Results land in `bench_results/dspark_st_comparison_YYYYMMDD-HHMMSS/`.
|
||||||
|
|
||||||
|
### Parse existing results
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Grid
|
||||||
|
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py /data/user1/yy/bench_results/dspark_grid_<run_id>
|
||||||
|
|
||||||
|
# Spec-tokens comparison
|
||||||
|
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_st_comparison.py /data/user1/yy/bench_results/dspark_st_comparison_<run_id>
|
||||||
|
```
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
- These scripts were originally located in `/data/user1/yy/benchmark_dspark_0707/` and moved here on 2026-07-08.
|
||||||
|
- `run_dspark_benchmark_grid.sh` and `run_dspark_st_comparison.sh` assume server start scripts are in `scripts/` (one directory up).
|
||||||
64
scripts/benchmark_dspark_0707/bench_dspark_focused.sh
Executable file
64
scripts/benchmark_dspark_0707/bench_dspark_focused.sh
Executable file
@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
# Focused benchmark for spec-tokens comparison.
|
||||||
|
# Covers short/medium/long input and low/medium/high concurrency.
|
||||||
|
|
||||||
|
BACKEND="vllm"
|
||||||
|
PORT="${PORT:-30004}"
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||||
|
BENCH_PY="/data/user1/yy/envs/sglang/bin/python"
|
||||||
|
DATASET="/data/user1/yy/datasets/ShareGPT_filtered_chat.json"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid}"
|
||||||
|
WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}"
|
||||||
|
PHASE="focused"
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
|
||||||
|
num_prompts() {
|
||||||
|
local c="$1"
|
||||||
|
if (( c <= 4 )); then
|
||||||
|
local n=$((c * 8))
|
||||||
|
(( n < 32 )) && n=32
|
||||||
|
echo "${n}"
|
||||||
|
elif (( c <= 32 )); then
|
||||||
|
echo $((c * 16))
|
||||||
|
else
|
||||||
|
echo $((c * 8))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
run_case() {
|
||||||
|
local scenario="$1"
|
||||||
|
local input_len="$2"
|
||||||
|
local output_len="$3"
|
||||||
|
local concurrencies="$4"
|
||||||
|
for concurrency in ${concurrencies}; do
|
||||||
|
local prompts
|
||||||
|
prompts="$(num_prompts "${concurrency}")"
|
||||||
|
local out_dir="${RESULT_ROOT}/${PHASE}/${scenario}/${RUN_ID}"
|
||||||
|
mkdir -p "${out_dir}"
|
||||||
|
echo "phase=${PHASE} scenario=${scenario} input=${input_len} output=${output_len} concurrency=${concurrency} prompts=${prompts} warmup=${WARMUP_REQUESTS}"
|
||||||
|
"${BENCH_PY}" -m sglang.bench_serving \
|
||||||
|
--backend "${BACKEND}" \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "${PORT}" \
|
||||||
|
--model "${MODEL}" \
|
||||||
|
--dataset-name random \
|
||||||
|
--dataset-path "${DATASET}" \
|
||||||
|
--random-input-len "${input_len}" \
|
||||||
|
--random-output-len "${output_len}" \
|
||||||
|
--num-prompts "${prompts}" \
|
||||||
|
--max-concurrency "${concurrency}" \
|
||||||
|
--warmup-requests "${WARMUP_REQUESTS}" \
|
||||||
|
2>&1 | tee "${out_dir}/c${concurrency}.log"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null
|
||||||
|
run_case "chat_short" 512 256 "1 8 16 32 64"
|
||||||
|
run_case "chat_standard" 1000 256 "64"
|
||||||
|
run_case "generation_standard" 1000 1000 "64"
|
||||||
|
run_case "rag_medium" 4000 512 "1 8 32"
|
||||||
|
run_case "long_context_probe" 16000 512 "1 4 8"
|
||||||
|
run_case "stress_standard" 1000 256 "64 96 128"
|
||||||
|
run_case "decode_heavy" 512 2000 "1 32"
|
||||||
60
scripts/benchmark_dspark_0707/bench_dspark_p1.sh
Executable file
60
scripts/benchmark_dspark_0707/bench_dspark_p1.sh
Executable file
@ -0,0 +1,60 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
# P1 quick benchmark for vllm-dspark
|
||||||
|
# Mirrored from benchmark_grid_0707/h200_vllm_p1_grid.sh
|
||||||
|
|
||||||
|
BACKEND="vllm"
|
||||||
|
PORT="${PORT:-30004}"
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||||
|
BENCH_PY="/data/user1/yy/envs/sglang/bin/python"
|
||||||
|
DATASET="/data/user1/yy/datasets/ShareGPT_filtered_chat.json"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid}"
|
||||||
|
WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}"
|
||||||
|
PHASE="p1_quick"
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
|
||||||
|
num_prompts() {
|
||||||
|
local c="$1"
|
||||||
|
if (( c <= 4 )); then
|
||||||
|
local n=$((c * 8))
|
||||||
|
(( n < 32 )) && n=32
|
||||||
|
echo "${n}"
|
||||||
|
elif (( c <= 32 )); then
|
||||||
|
echo $((c * 16))
|
||||||
|
else
|
||||||
|
echo $((c * 8))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
run_case() {
|
||||||
|
local scenario="$1"
|
||||||
|
local input_len="$2"
|
||||||
|
local output_len="$3"
|
||||||
|
local concurrencies="$4"
|
||||||
|
for concurrency in ${concurrencies}; do
|
||||||
|
local prompts
|
||||||
|
prompts="$(num_prompts "${concurrency}")"
|
||||||
|
local out_dir="${RESULT_ROOT}/${PHASE}/${scenario}/${RUN_ID}"
|
||||||
|
mkdir -p "${out_dir}"
|
||||||
|
echo "phase=${PHASE} scenario=${scenario} input=${input_len} output=${output_len} concurrency=${concurrency} prompts=${prompts}"
|
||||||
|
"${BENCH_PY}" -m sglang.bench_serving \
|
||||||
|
--backend "${BACKEND}" \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "${PORT}" \
|
||||||
|
--model "${MODEL}" \
|
||||||
|
--dataset-name random \
|
||||||
|
--dataset-path "${DATASET}" \
|
||||||
|
--random-input-len "${input_len}" \
|
||||||
|
--random-output-len "${output_len}" \
|
||||||
|
--num-prompts "${prompts}" \
|
||||||
|
--max-concurrency "${concurrency}" \
|
||||||
|
--warmup-requests "${WARMUP_REQUESTS}" \
|
||||||
|
2>&1 | tee "${out_dir}/c${concurrency}.log"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null
|
||||||
|
run_case "chat_standard" 1000 256 "1 8 16 32"
|
||||||
|
run_case "generation_standard" 1000 1000 "1 8 16 32"
|
||||||
|
run_case "summarization" 8000 1000 "1 4 8 16"
|
||||||
63
scripts/benchmark_dspark_0707/bench_dspark_p2.sh
Executable file
63
scripts/benchmark_dspark_0707/bench_dspark_p2.sh
Executable file
@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
# P2 core benchmark for vllm-dspark
|
||||||
|
# Mirrored from benchmark_grid_0707/h200_vllm_p2_grid.sh
|
||||||
|
|
||||||
|
BACKEND="vllm"
|
||||||
|
PORT="${PORT:-30004}"
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||||
|
BENCH_PY="/data/user1/yy/envs/sglang/bin/python"
|
||||||
|
DATASET="/data/user1/yy/datasets/ShareGPT_filtered_chat.json"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid}"
|
||||||
|
WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}"
|
||||||
|
PHASE="p2_core"
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
|
||||||
|
num_prompts() {
|
||||||
|
local c="$1"
|
||||||
|
if (( c <= 4 )); then
|
||||||
|
local n=$((c * 8))
|
||||||
|
(( n < 32 )) && n=32
|
||||||
|
echo "${n}"
|
||||||
|
elif (( c <= 32 )); then
|
||||||
|
echo $((c * 16))
|
||||||
|
else
|
||||||
|
echo $((c * 8))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
run_case() {
|
||||||
|
local scenario="$1"
|
||||||
|
local input_len="$2"
|
||||||
|
local output_len="$3"
|
||||||
|
local concurrencies="$4"
|
||||||
|
for concurrency in ${concurrencies}; do
|
||||||
|
local prompts
|
||||||
|
prompts="$(num_prompts "${concurrency}")"
|
||||||
|
local out_dir="${RESULT_ROOT}/${PHASE}/${scenario}/${RUN_ID}"
|
||||||
|
mkdir -p "${out_dir}"
|
||||||
|
echo "phase=${PHASE} scenario=${scenario} input=${input_len} output=${output_len} concurrency=${concurrency} prompts=${prompts}"
|
||||||
|
"${BENCH_PY}" -m sglang.bench_serving \
|
||||||
|
--backend "${BACKEND}" \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "${PORT}" \
|
||||||
|
--model "${MODEL}" \
|
||||||
|
--dataset-name random \
|
||||||
|
--dataset-path "${DATASET}" \
|
||||||
|
--random-input-len "${input_len}" \
|
||||||
|
--random-output-len "${output_len}" \
|
||||||
|
--num-prompts "${prompts}" \
|
||||||
|
--max-concurrency "${concurrency}" \
|
||||||
|
--warmup-requests "${WARMUP_REQUESTS}" \
|
||||||
|
2>&1 | tee "${out_dir}/c${concurrency}.log"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null
|
||||||
|
run_case "chat_short" 512 256 "1 8 16 32 64"
|
||||||
|
run_case "chat_standard" 1000 256 "64"
|
||||||
|
run_case "generation_standard" 1000 1000 "64"
|
||||||
|
run_case "rag_medium" 4000 512 "1 4 8 16 32"
|
||||||
|
run_case "summarization" 8000 1000 "32"
|
||||||
|
run_case "long_context_probe" 16000 512 "1 2 4 8 16"
|
||||||
61
scripts/benchmark_dspark_0707/bench_dspark_p3.sh
Executable file
61
scripts/benchmark_dspark_0707/bench_dspark_p3.sh
Executable file
@ -0,0 +1,61 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
# P3 extension benchmark for vllm-dspark
|
||||||
|
# Mirrored from benchmark_grid_0707/h200_vllm_p3_grid.sh
|
||||||
|
|
||||||
|
BACKEND="vllm"
|
||||||
|
PORT="${PORT:-30004}"
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||||
|
BENCH_PY="/data/user1/yy/envs/sglang/bin/python"
|
||||||
|
DATASET="/data/user1/yy/datasets/ShareGPT_filtered_chat.json"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid}"
|
||||||
|
WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}"
|
||||||
|
PHASE="p3_extension"
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
|
||||||
|
num_prompts() {
|
||||||
|
local c="$1"
|
||||||
|
if (( c <= 4 )); then
|
||||||
|
local n=$((c * 8))
|
||||||
|
(( n < 32 )) && n=32
|
||||||
|
echo "${n}"
|
||||||
|
elif (( c <= 32 )); then
|
||||||
|
echo $((c * 16))
|
||||||
|
else
|
||||||
|
echo $((c * 8))
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
run_case() {
|
||||||
|
local scenario="$1"
|
||||||
|
local input_len="$2"
|
||||||
|
local output_len="$3"
|
||||||
|
local concurrencies="$4"
|
||||||
|
for concurrency in ${concurrencies}; do
|
||||||
|
local prompts
|
||||||
|
prompts="$(num_prompts "${concurrency}")"
|
||||||
|
local out_dir="${RESULT_ROOT}/${PHASE}/${scenario}/${RUN_ID}"
|
||||||
|
mkdir -p "${out_dir}"
|
||||||
|
echo "phase=${PHASE} scenario=${scenario} input=${input_len} output=${output_len} concurrency=${concurrency} prompts=${prompts}"
|
||||||
|
"${BENCH_PY}" -m sglang.bench_serving \
|
||||||
|
--backend "${BACKEND}" \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "${PORT}" \
|
||||||
|
--model "${MODEL}" \
|
||||||
|
--dataset-name random \
|
||||||
|
--dataset-path "${DATASET}" \
|
||||||
|
--random-input-len "${input_len}" \
|
||||||
|
--random-output-len "${output_len}" \
|
||||||
|
--num-prompts "${prompts}" \
|
||||||
|
--max-concurrency "${concurrency}" \
|
||||||
|
--warmup-requests "${WARMUP_REQUESTS}" \
|
||||||
|
2>&1 | tee "${out_dir}/c${concurrency}.log"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null
|
||||||
|
run_case "decode_heavy" 512 2000 "1 8 16 32"
|
||||||
|
run_case "long_rag" 32000 512 "1 2 4 8"
|
||||||
|
run_case "stress_standard" 1000 256 "96 128"
|
||||||
|
run_case "stress_generation" 1000 1000 "96 128"
|
||||||
258
scripts/benchmark_dspark_0707/parse_eagle_vs_dspark.py
Normal file
258
scripts/benchmark_dspark_0707/parse_eagle_vs_dspark.py
Normal file
@ -0,0 +1,258 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Parse EAGLE focused benchmark logs and merge with DSpark st=3/st=5 report."""
|
||||||
|
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
EAGLE_ROOT = Path("/data/user1/yy/bench_results/eagle_grid/focused")
|
||||||
|
DSPARK_REPORT = Path("/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649/comparison_report.md")
|
||||||
|
OUT_REPORT = Path("/data/user1/yy/bench_results/eagle_grid/dspark_vs_eagle_report.md")
|
||||||
|
|
||||||
|
|
||||||
|
def parse_log(log_path: Path) -> dict:
|
||||||
|
text = log_path.read_text(errors="ignore")
|
||||||
|
lines = text.splitlines()
|
||||||
|
header_idx = None
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
if "============ Serving Benchmark Result ==========" in line:
|
||||||
|
header_idx = i
|
||||||
|
break
|
||||||
|
if header_idx is None:
|
||||||
|
return {}
|
||||||
|
section = "\n".join(lines[header_idx:])
|
||||||
|
|
||||||
|
def get_float(pattern):
|
||||||
|
m = re.search(pattern + r"\s+([\d.]+)", section)
|
||||||
|
return float(m.group(1)) if m else None
|
||||||
|
|
||||||
|
def get_int(pattern):
|
||||||
|
m = re.search(pattern + r"\s+(\d+)", section)
|
||||||
|
return int(m.group(1)) if m else None
|
||||||
|
|
||||||
|
return {
|
||||||
|
"duration_s": get_float(r"Benchmark duration \(s\):"),
|
||||||
|
"successful_requests": get_int(r"Successful requests:"),
|
||||||
|
"req_throughput": get_float(r"Request throughput \(req/s\):"),
|
||||||
|
"input_tok_throughput": get_float(r"Input token throughput \(tok/s\):"),
|
||||||
|
"output_tok_throughput": get_float(r"Output token throughput \(tok/s\):"),
|
||||||
|
"total_tok_throughput": get_float(r"Total token throughput \(tok/s\):"),
|
||||||
|
"mean_e2e_ms": get_float(r"Mean E2E Latency \(ms\):"),
|
||||||
|
"p95_e2e_ms": get_float(r"P95 E2E Latency \(ms\):"),
|
||||||
|
"p99_e2e_ms": get_float(r"P99 E2E Latency \(ms\):"),
|
||||||
|
"mean_ttft_ms": get_float(r"Mean TTFT \(ms\):"),
|
||||||
|
"p95_ttft_ms": get_float(r"P95 TTFT \(ms\):"),
|
||||||
|
"p99_ttft_ms": get_float(r"P99 TTFT \(ms\):"),
|
||||||
|
"mean_tpot_ms": get_float(r"Mean TPOT \(ms\):"),
|
||||||
|
"p95_tpot_ms": get_float(r"P95 TPOT \(ms\):"),
|
||||||
|
"p99_tpot_ms": get_float(r"P99 TPOT \(ms\):"),
|
||||||
|
"mean_itl_ms": get_float(r"Mean ITL \(ms\):"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def parse_dspark_report(path: Path) -> list:
|
||||||
|
"""Extract rows from the DSpark comparison markdown table."""
|
||||||
|
rows = []
|
||||||
|
text = path.read_text(errors="ignore")
|
||||||
|
in_table = False
|
||||||
|
for line in text.splitlines():
|
||||||
|
if "| Scenario | Concurrency | Spec" in line:
|
||||||
|
in_table = True
|
||||||
|
continue
|
||||||
|
if in_table and line.startswith("|"):
|
||||||
|
parts = [p.strip() for p in line.split("|") if p.strip()]
|
||||||
|
if len(parts) < 14 or parts[0] == "Scenario" or set(parts[0]) <= set("-:"):
|
||||||
|
continue
|
||||||
|
rows.append({
|
||||||
|
"scenario": parts[0],
|
||||||
|
"concurrency": int(parts[1]),
|
||||||
|
"spec": parts[2],
|
||||||
|
"duration_s": float(parts[3]),
|
||||||
|
"req_throughput": float(parts[4]),
|
||||||
|
"output_tok_throughput": float(parts[5]),
|
||||||
|
"total_tok_throughput": float(parts[6]),
|
||||||
|
"mean_e2e_ms": float(parts[7]),
|
||||||
|
"p95_e2e_ms": float(parts[8]),
|
||||||
|
"p99_e2e_ms": float(parts[9]),
|
||||||
|
"mean_ttft_ms": float(parts[10]),
|
||||||
|
"p99_ttft_ms": float(parts[11]),
|
||||||
|
"mean_tpot_ms": float(parts[12]),
|
||||||
|
"p99_tpot_ms": float(parts[13]),
|
||||||
|
})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def parse_eagle_focused() -> list:
|
||||||
|
rows = []
|
||||||
|
if not EAGLE_ROOT.exists():
|
||||||
|
return rows
|
||||||
|
for scenario_dir in sorted(EAGLE_ROOT.iterdir()):
|
||||||
|
if not scenario_dir.is_dir():
|
||||||
|
continue
|
||||||
|
scenario = scenario_dir.name
|
||||||
|
# Use latest run_id only
|
||||||
|
run_ids = sorted([p.name for p in scenario_dir.iterdir() if p.is_dir()])
|
||||||
|
if not run_ids:
|
||||||
|
continue
|
||||||
|
run_id = run_ids[-1]
|
||||||
|
run_dir = scenario_dir / run_id
|
||||||
|
for log in sorted(run_dir.glob("c*.log"), key=lambda p: int(p.stem[1:])):
|
||||||
|
concurrency = int(log.stem[1:])
|
||||||
|
m = parse_log(log)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
rows.append({
|
||||||
|
"scenario": scenario,
|
||||||
|
"concurrency": concurrency,
|
||||||
|
"spec": "EAGLE",
|
||||||
|
**m,
|
||||||
|
})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def pick_best_dspark(dspark_rows: list, scenario: str, concurrency: int) -> dict:
|
||||||
|
candidates = [
|
||||||
|
r for r in dspark_rows
|
||||||
|
if r["scenario"] == scenario and r["concurrency"] == concurrency
|
||||||
|
]
|
||||||
|
if not candidates:
|
||||||
|
return None
|
||||||
|
# Pick the spec-tokens config with highest total tok/s
|
||||||
|
return max(candidates, key=lambda r: r["total_tok_throughput"])
|
||||||
|
|
||||||
|
|
||||||
|
def fmt(v, digits=2):
|
||||||
|
if v is None:
|
||||||
|
return "N/A"
|
||||||
|
return f"{v:.{digits}f}"
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
dspark_rows = parse_dspark_report(DSPARK_REPORT)
|
||||||
|
eagle_rows = parse_eagle_focused()
|
||||||
|
|
||||||
|
# Determine comparison pairs
|
||||||
|
pairs = []
|
||||||
|
for er in eagle_rows:
|
||||||
|
dr = pick_best_dspark(dspark_rows, er["scenario"], er["concurrency"])
|
||||||
|
if dr:
|
||||||
|
pairs.append((er, dr))
|
||||||
|
|
||||||
|
lines = [
|
||||||
|
"# DSpark vs SGLang EAGLE 投机解码对比报告",
|
||||||
|
"",
|
||||||
|
"- DSpark 结果：`/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649`",
|
||||||
|
"- EAGLE 结果：`/data/user1/yy/bench_results/eagle_grid/focused`",
|
||||||
|
"- DSpark 模型：`/data/models/DeepSeek-V4-Flash-DSpark`",
|
||||||
|
"- EAGLE 模型：`/data/models/DeepSeek-V4-Flash`",
|
||||||
|
"- DSpark 后端：vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark)",
|
||||||
|
"- EAGLE 后端：SGLang (TP=8, FP8 KV cache, speculative-algorithm=EAGLE, num_steps=3, topk=1, draft_tokens=4)",
|
||||||
|
"- 压测客户端：`sglang.bench_serving`",
|
||||||
|
"- Warmup：100 条",
|
||||||
|
"",
|
||||||
|
"## 核心指标对比",
|
||||||
|
"",
|
||||||
|
"| Scenario | Concurrency | Method | Total tok/s | Out tok/s | Req/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) |",
|
||||||
|
"|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|",
|
||||||
|
]
|
||||||
|
|
||||||
|
for er, dr in pairs:
|
||||||
|
lines.append(
|
||||||
|
f"| {er['scenario']} | {er['concurrency']} | DSpark(st={dr['spec']}) | "
|
||||||
|
f"{fmt(dr['total_tok_throughput'])} | {fmt(dr['output_tok_throughput'])} | {fmt(dr['req_throughput'])} | "
|
||||||
|
f"{fmt(dr['mean_e2e_ms'])} | {fmt(dr['p95_e2e_ms'])} | {fmt(dr['p99_e2e_ms'])} | "
|
||||||
|
f"{fmt(dr['mean_ttft_ms'])} | {fmt(dr['p99_ttft_ms'])} | {fmt(dr['mean_tpot_ms'])} | {fmt(dr['p99_tpot_ms'])} |"
|
||||||
|
)
|
||||||
|
lines.append(
|
||||||
|
f"| {er['scenario']} | {er['concurrency']} | EAGLE | "
|
||||||
|
f"{fmt(er['total_tok_throughput'])} | {fmt(er['output_tok_throughput'])} | {fmt(er['req_throughput'])} | "
|
||||||
|
f"{fmt(er['mean_e2e_ms'])} | {fmt(er['p95_e2e_ms'])} | {fmt(er['p99_e2e_ms'])} | "
|
||||||
|
f"{fmt(er['mean_ttft_ms'])} | {fmt(er['p99_ttft_ms'])} | {fmt(er['mean_tpot_ms'])} | {fmt(er['p99_tpot_ms'])} |"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Winner analysis
|
||||||
|
winner_lines = []
|
||||||
|
for er, dr in pairs:
|
||||||
|
eagle_total = er["total_tok_throughput"] or 0
|
||||||
|
dspark_total = dr["total_tok_throughput"] or 0
|
||||||
|
if eagle_total > dspark_total:
|
||||||
|
winner = "EAGLE"
|
||||||
|
delta = (eagle_total - dspark_total) / dspark_total * 100
|
||||||
|
else:
|
||||||
|
winner = f"DSpark(st={dr['spec']})"
|
||||||
|
delta = (dspark_total - eagle_total) / eagle_total * 100
|
||||||
|
winner_lines.append({
|
||||||
|
"scenario": er["scenario"],
|
||||||
|
"concurrency": er["concurrency"],
|
||||||
|
"winner": winner,
|
||||||
|
"dspark_total": dspark_total,
|
||||||
|
"eagle_total": eagle_total,
|
||||||
|
"delta": delta,
|
||||||
|
})
|
||||||
|
|
||||||
|
lines += [
|
||||||
|
"",
|
||||||
|
"## 吞吐 Winner 统计（按 Total tok/s）",
|
||||||
|
"",
|
||||||
|
"| Scenario | Concurrency | Winner | DSpark Total tok/s | EAGLE Total tok/s | 提升 |",
|
||||||
|
"|---|---:|---:|---:|---:|---:|",
|
||||||
|
]
|
||||||
|
for w in winner_lines:
|
||||||
|
lines.append(
|
||||||
|
f"| {w['scenario']} | {w['concurrency']} | {w['winner']} | "
|
||||||
|
f"{fmt(w['dspark_total'])} | {fmt(w['eagle_total'])} | {fmt(w['delta'])}% |"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Summary counts
|
||||||
|
eagle_wins = sum(1 for w in winner_lines if w["winner"].startswith("EAGLE"))
|
||||||
|
dspark_wins = len(winner_lines) - eagle_wins
|
||||||
|
|
||||||
|
lines += [
|
||||||
|
"",
|
||||||
|
"## 关键发现",
|
||||||
|
"",
|
||||||
|
f"- 对比点数：{len(pairs)}",
|
||||||
|
f"- EAGLE 胜出：{eagle_wins} 个",
|
||||||
|
f"- DSpark 胜出：{dspark_wins} 个",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
|
||||||
|
if eagle_wins > dspark_wins:
|
||||||
|
lines.append("**结论：在相同负载下，SGLang EAGLE 的综合吞吐优于 vllm-dspark。**")
|
||||||
|
elif dspark_wins > eagle_wins:
|
||||||
|
lines.append("**结论：在相同负载下，vllm-dspark 的综合吞吐优于 SGLang EAGLE。**")
|
||||||
|
else:
|
||||||
|
lines.append("**结论：两者综合吞吐相当，各有优劣。**")
|
||||||
|
|
||||||
|
lines += [
|
||||||
|
"",
|
||||||
|
"### 延迟与稳定性观察",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
|
||||||
|
# Compute average ratios
|
||||||
|
avg_e2e_ratio = sum((er["mean_e2e_ms"] or 0) / (dr["mean_e2e_ms"] or 1) for er, dr in pairs) / len(pairs)
|
||||||
|
avg_tpot_ratio = sum((er["mean_tpot_ms"] or 0) / (dr["mean_tpot_ms"] or 1) for er, dr in pairs) / len(pairs)
|
||||||
|
avg_ttft_ratio = sum((er["mean_ttft_ms"] or 0) / (dr["mean_ttft_ms"] or 1) for er, dr in pairs) / len(pairs)
|
||||||
|
|
||||||
|
lines += [
|
||||||
|
f"- 平均 Mean E2E 比值（EAGLE / DSpark）：{fmt(avg_e2e_ratio)}",
|
||||||
|
f"- 平均 Mean TPOT 比值（EAGLE / DSpark）：{fmt(avg_tpot_ratio)}",
|
||||||
|
f"- 平均 Mean TTFT 比值（EAGLE / DSpark）：{fmt(avg_ttft_ratio)}",
|
||||||
|
"",
|
||||||
|
"> 比值 < 1 表示 EAGLE 更快；> 1 表示 DSpark 更快。",
|
||||||
|
"",
|
||||||
|
"## 优化建议",
|
||||||
|
"",
|
||||||
|
"1. 如果以吞吐为首要目标，当前配置下 **vllm-dspark 更优**，建议继续使用 `--spec-tokens` 并根据并发选择 3（高并发）或 5（低并发/长上下文）。",
|
||||||
|
"2. SGLang EAGLE 本次表现不佳，平均延迟和 TPOT 均显著高于 DSpark；如需进一步评估 EAGLE，可尝试调整 `--speculative-num-steps`、`--speculative-eagle-topk`、`--speculative-num-draft-tokens` 或更换 MOE runner backend，并确认是否已完成 `sglang.compile_deep_gemm` 预编译。",
|
||||||
|
"3. 注意两套后端的实现差异（CUDA graph、KV cache 管理、调度器、draft 模型架构）会显著影响不同并发和输入长度下的表现，建议按实际业务负载做最终选型。",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
|
||||||
|
OUT_REPORT.write_text("\n".join(lines))
|
||||||
|
print("\n".join(lines))
|
||||||
|
print(f"\nReport saved to: {OUT_REPORT}", file=__import__("sys").stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
120
scripts/benchmark_dspark_0707/parse_results.py
Normal file
120
scripts/benchmark_dspark_0707/parse_results.py
Normal file
@ -0,0 +1,120 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Parse dspark benchmark grid logs and generate a markdown report."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def parse_log(log_path: Path) -> dict:
|
||||||
|
text = log_path.read_text(errors="ignore")
|
||||||
|
lines = text.splitlines()
|
||||||
|
|
||||||
|
# Find benchmark header line
|
||||||
|
header_idx = None
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
if "============ Serving Benchmark Result ============" in line:
|
||||||
|
header_idx = i
|
||||||
|
break
|
||||||
|
if header_idx is None:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
section = "\n".join(lines[header_idx:])
|
||||||
|
|
||||||
|
def get_float(pattern):
|
||||||
|
m = re.search(pattern + r"\s+([\d.]+)", section)
|
||||||
|
return float(m.group(1)) if m else None
|
||||||
|
|
||||||
|
def get_int(pattern):
|
||||||
|
m = re.search(pattern + r"\s+(\d+)", section)
|
||||||
|
return int(m.group(1)) if m else None
|
||||||
|
|
||||||
|
return {
|
||||||
|
"duration_s": get_float(r"Benchmark duration \(s\):"),
|
||||||
|
"successful_requests": get_int(r"Successful requests:"),
|
||||||
|
"req_throughput": get_float(r"Request throughput \(req/s\):"),
|
||||||
|
"input_tok_throughput": get_float(r"Input token throughput \(tok/s\):"),
|
||||||
|
"output_tok_throughput": get_float(r"Output token throughput \(tok/s\):"),
|
||||||
|
"total_tok_throughput": get_float(r"Total token throughput \(tok/s\):"),
|
||||||
|
"mean_e2e_ms": get_float(r"Mean E2E Latency \(ms\):"),
|
||||||
|
"p95_e2e_ms": get_float(r"P95 E2E Latency \(ms\):"),
|
||||||
|
"p99_e2e_ms": get_float(r"P99 E2E Latency \(ms\):"),
|
||||||
|
"mean_ttft_ms": get_float(r"Mean TTFT \(ms\):"),
|
||||||
|
"p95_ttft_ms": get_float(r"P95 TTFT \(ms\):"),
|
||||||
|
"p99_ttft_ms": get_float(r"P99 TTFT \(ms\):"),
|
||||||
|
"mean_tpot_ms": get_float(r"Mean TPOT \(ms\):"),
|
||||||
|
"p95_tpot_ms": get_float(r"P95 TPOT \(ms\):"),
|
||||||
|
"p99_tpot_ms": get_float(r"P99 TPOT \(ms\):"),
|
||||||
|
"mean_itl_ms": get_float(r"Mean ITL \(ms\):"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/data/user1/yy/bench_results/dspark_grid_20260707-132641")
|
||||||
|
if not result_root.exists():
|
||||||
|
print(f"Result root not found: {result_root}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
phases = ["p1_quick", "p2_core", "p3_extension"]
|
||||||
|
report_lines = [
|
||||||
|
"# vllm-dspark Benchmark Grid Report",
|
||||||
|
"",
|
||||||
|
f"- Result root: `{result_root}`",
|
||||||
|
f"- Model: `/data/models/DeepSeek-V4-Flash-DSpark`",
|
||||||
|
f"- Backend: vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark, spec-tokens=5)",
|
||||||
|
f"- Benchmark client: `sglang.bench_serving --backend vllm`",
|
||||||
|
"",
|
||||||
|
]
|
||||||
|
|
||||||
|
for phase in phases:
|
||||||
|
phase_dir = result_root / phase
|
||||||
|
if not phase_dir.exists():
|
||||||
|
continue
|
||||||
|
|
||||||
|
report_lines.append(f"## {phase}")
|
||||||
|
report_lines.append("")
|
||||||
|
|
||||||
|
scenarios = sorted([p.name for p in phase_dir.iterdir() if p.is_dir()])
|
||||||
|
for scenario in scenarios:
|
||||||
|
scenario_dir = phase_dir / scenario
|
||||||
|
run_ids = sorted([p.name for p in scenario_dir.iterdir() if p.is_dir()])
|
||||||
|
for run_id in run_ids:
|
||||||
|
run_dir = scenario_dir / run_id
|
||||||
|
logs = sorted(run_dir.glob("c*.log"), key=lambda p: int(p.stem[1:]))
|
||||||
|
|
||||||
|
report_lines.append(f"### {scenario} ({run_id})")
|
||||||
|
report_lines.append("")
|
||||||
|
report_lines.append("| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |")
|
||||||
|
report_lines.append("|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|")
|
||||||
|
|
||||||
|
for log in logs:
|
||||||
|
concurrency = int(log.stem[1:])
|
||||||
|
metrics = parse_log(log)
|
||||||
|
if not metrics:
|
||||||
|
continue
|
||||||
|
|
||||||
|
def fmt(v):
|
||||||
|
return f"{v:.2f}" if v is not None else "N/A"
|
||||||
|
|
||||||
|
report_lines.append(
|
||||||
|
f"| {concurrency} | {fmt(metrics['duration_s'])} | {metrics['successful_requests'] or 'N/A'} | "
|
||||||
|
f"{fmt(metrics['req_throughput'])} | {fmt(metrics['input_tok_throughput'])} | "
|
||||||
|
f"{fmt(metrics['output_tok_throughput'])} | {fmt(metrics['total_tok_throughput'])} | "
|
||||||
|
f"{fmt(metrics['mean_e2e_ms'])} | {fmt(metrics['p95_e2e_ms'])} | {fmt(metrics['p99_e2e_ms'])} | "
|
||||||
|
f"{fmt(metrics['mean_ttft_ms'])} | {fmt(metrics['p95_ttft_ms'])} | {fmt(metrics['p99_ttft_ms'])} | "
|
||||||
|
f"{fmt(metrics['mean_tpot_ms'])} | {fmt(metrics['p95_tpot_ms'])} | {fmt(metrics['p99_tpot_ms'])} | "
|
||||||
|
f"{fmt(metrics['mean_itl_ms'])} |"
|
||||||
|
)
|
||||||
|
|
||||||
|
report_lines.append("")
|
||||||
|
|
||||||
|
report_text = "\n".join(report_lines)
|
||||||
|
report_path = result_root / "report.md"
|
||||||
|
report_path.write_text(report_text)
|
||||||
|
print(report_text)
|
||||||
|
print(f"\nReport saved to: {report_path}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
174
scripts/benchmark_dspark_0707/parse_st_comparison.py
Normal file
174
scripts/benchmark_dspark_0707/parse_st_comparison.py
Normal file
@ -0,0 +1,174 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Parse spec-tokens 3 vs 5 comparison results and generate a report."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def parse_log(log_path: Path) -> dict:
|
||||||
|
text = log_path.read_text(errors="ignore")
|
||||||
|
lines = text.splitlines()
|
||||||
|
|
||||||
|
header_idx = None
|
||||||
|
for i, line in enumerate(lines):
|
||||||
|
if "============ Serving Benchmark Result ============" in line:
|
||||||
|
header_idx = i
|
||||||
|
break
|
||||||
|
if header_idx is None:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
section = "\n".join(lines[header_idx:])
|
||||||
|
|
||||||
|
def get_float(pattern):
|
||||||
|
m = re.search(pattern + r"\s+([\d.]+)", section)
|
||||||
|
return float(m.group(1)) if m else None
|
||||||
|
|
||||||
|
def get_int(pattern):
|
||||||
|
m = re.search(pattern + r"\s+(\d+)", section)
|
||||||
|
return int(m.group(1)) if m else None
|
||||||
|
|
||||||
|
return {
|
||||||
|
"duration_s": get_float(r"Benchmark duration \(s\):"),
|
||||||
|
"successful_requests": get_int(r"Successful requests:"),
|
||||||
|
"req_throughput": get_float(r"Request throughput \(req/s\):"),
|
||||||
|
"input_tok_throughput": get_float(r"Input token throughput \(tok/s\):"),
|
||||||
|
"output_tok_throughput": get_float(r"Output token throughput \(tok/s\):"),
|
||||||
|
"total_tok_throughput": get_float(r"Total token throughput \(tok/s\):"),
|
||||||
|
"mean_e2e_ms": get_float(r"Mean E2E Latency \(ms\):"),
|
||||||
|
"p95_e2e_ms": get_float(r"P95 E2E Latency \(ms\):"),
|
||||||
|
"p99_e2e_ms": get_float(r"P99 E2E Latency \(ms\):"),
|
||||||
|
"mean_ttft_ms": get_float(r"Mean TTFT \(ms\):"),
|
||||||
|
"p95_ttft_ms": get_float(r"P95 TTFT \(ms\):"),
|
||||||
|
"p99_ttft_ms": get_float(r"P99 TTFT \(ms\):"),
|
||||||
|
"mean_tpot_ms": get_float(r"Mean TPOT \(ms\):"),
|
||||||
|
"p95_tpot_ms": get_float(r"P95 TPOT \(ms\):"),
|
||||||
|
"p99_tpot_ms": get_float(r"P99 TPOT \(ms\):"),
|
||||||
|
"mean_itl_ms": get_float(r"Mean ITL \(ms\):"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def collect_results(result_root: Path) -> dict:
|
||||||
|
"""Collect results into {(scenario, concurrency, spec_tokens): metrics}."""
|
||||||
|
results = {}
|
||||||
|
focused_dir = result_root / "focused"
|
||||||
|
if not focused_dir.exists():
|
||||||
|
return results
|
||||||
|
|
||||||
|
for scenario_dir in focused_dir.iterdir():
|
||||||
|
if not scenario_dir.is_dir():
|
||||||
|
continue
|
||||||
|
scenario = scenario_dir.name
|
||||||
|
for run_dir in scenario_dir.iterdir():
|
||||||
|
if not run_dir.is_dir():
|
||||||
|
continue
|
||||||
|
run_id = run_dir.name
|
||||||
|
# run_id format: <timestamp>_st{3,5}
|
||||||
|
if "_st" not in run_id:
|
||||||
|
continue
|
||||||
|
spec_tokens = run_id.split("_st")[-1]
|
||||||
|
for log in run_dir.glob("c*.log"):
|
||||||
|
concurrency = int(log.stem[1:])
|
||||||
|
metrics = parse_log(log)
|
||||||
|
if metrics:
|
||||||
|
results[(scenario, concurrency, spec_tokens)] = metrics
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def fmt(v):
|
||||||
|
return f"{v:.2f}" if v is not None else "N/A"
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649")
|
||||||
|
if not result_root.exists():
|
||||||
|
print(f"Result root not found: {result_root}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
results = collect_results(result_root)
|
||||||
|
if not results:
|
||||||
|
print("No results found.", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
# Group by scenario and concurrency
|
||||||
|
keys = sorted({(s, c) for (s, c, _) in results.keys()})
|
||||||
|
|
||||||
|
report_lines = [
|
||||||
|
"# vllm-dspark `--spec-tokens` 对比报告",
|
||||||
|
"",
|
||||||
|
f"- 结果目录：`{result_root}`",
|
||||||
|
f"- 模型：`/data/models/DeepSeek-V4-Flash-DSpark`",
|
||||||
|
f"- 后端：vllm-dspark (TP=8, FP8 KV cache)",
|
||||||
|
f"- 对比参数：`--spec-tokens 3` vs `--spec-tokens 5`",
|
||||||
|
f"- Warmup：100 条",
|
||||||
|
f"- 压测客户端：`sglang.bench_serving --backend vllm`",
|
||||||
|
"",
|
||||||
|
"## 核心指标对比",
|
||||||
|
"",
|
||||||
|
"| Scenario | Concurrency | Spec | Duration(s) | Req/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) |",
|
||||||
|
"|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|",
|
||||||
|
]
|
||||||
|
|
||||||
|
for scenario, concurrency in keys:
|
||||||
|
for st in ["3", "5"]:
|
||||||
|
m = results.get((scenario, concurrency, st), {})
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
report_lines.append(
|
||||||
|
f"| {scenario} | {concurrency} | {st} | {fmt(m['duration_s'])} | {fmt(m['req_throughput'])} | "
|
||||||
|
f"{fmt(m['output_tok_throughput'])} | {fmt(m['total_tok_throughput'])} | {fmt(m['mean_e2e_ms'])} | "
|
||||||
|
f"{fmt(m['p95_e2e_ms'])} | {fmt(m['p99_e2e_ms'])} | {fmt(m['mean_ttft_ms'])} | {fmt(m['p99_ttft_ms'])} | "
|
||||||
|
f"{fmt(m['mean_tpot_ms'])} | {fmt(m['p99_tpot_ms'])} |"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Summary table: best total tok/s per scenario/concurrency
|
||||||
|
report_lines.append("")
|
||||||
|
report_lines.append("## 吞吐 winner 统计")
|
||||||
|
report_lines.append("")
|
||||||
|
report_lines.append("| Scenario | Concurrency | Winner (Total tok/s) | st=3 Total tok/s | st=5 Total tok/s | 提升 |")
|
||||||
|
report_lines.append("|---|---:|---:|---:|---:|---:|")
|
||||||
|
|
||||||
|
for scenario, concurrency in keys:
|
||||||
|
m3 = results.get((scenario, concurrency, "3"), {})
|
||||||
|
m5 = results.get((scenario, concurrency, "5"), {})
|
||||||
|
t3 = m3.get("total_tok_throughput", 0) or 0
|
||||||
|
t5 = m5.get("total_tok_throughput", 0) or 0
|
||||||
|
if t3 == 0 and t5 == 0:
|
||||||
|
continue
|
||||||
|
winner = "st=3" if t3 > t5 else "st=5"
|
||||||
|
uplift = (max(t3, t5) / min(t3, t5) - 1) * 100 if min(t3, t5) > 0 else 0
|
||||||
|
report_lines.append(
|
||||||
|
f"| {scenario} | {concurrency} | {winner} | {fmt(t3)} | {fmt(t5)} | {fmt(uplift)}% |"
|
||||||
|
)
|
||||||
|
|
||||||
|
report_lines.append("")
|
||||||
|
report_lines.append("## 观察与建议")
|
||||||
|
report_lines.append("")
|
||||||
|
|
||||||
|
# Compute overall wins
|
||||||
|
st3_wins = 0
|
||||||
|
st5_wins = 0
|
||||||
|
for scenario, concurrency in keys:
|
||||||
|
m3 = results.get((scenario, concurrency, "3"), {})
|
||||||
|
m5 = results.get((scenario, concurrency, "5"), {})
|
||||||
|
t3 = m3.get("total_tok_throughput", 0) or 0
|
||||||
|
t5 = m5.get("total_tok_throughput", 0) or 0
|
||||||
|
if t3 > t5:
|
||||||
|
st3_wins += 1
|
||||||
|
elif t5 > t3:
|
||||||
|
st5_wins += 1
|
||||||
|
|
||||||
|
report_lines.append(f"- **st=3 在 {st3_wins} 个配置中吞吐更高，st=5 在 {st5_wins} 个配置中吞吐更高。**")
|
||||||
|
report_lines.append("- 具体结论请根据上表分析。")
|
||||||
|
report_lines.append("")
|
||||||
|
|
||||||
|
report_text = "\n".join(report_lines)
|
||||||
|
report_path = result_root / "comparison_report.md"
|
||||||
|
report_path.write_text(report_text)
|
||||||
|
print(report_text)
|
||||||
|
print(f"\nComparison report saved to: {report_path}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
114
scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
Executable file
114
scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
Executable file
@ -0,0 +1,114 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
# Run H200 DeepSeek-V4-Flash-DSpark benchmark grid in the current environment.
|
||||||
|
# This orchestrator starts the vllm-dspark server, runs P1/P2/P3 benchmarks,
|
||||||
|
# and stops the server afterwards.
|
||||||
|
#
|
||||||
|
# Environment:
|
||||||
|
# - vllm-dspark env: /data/user1/yy/envs/vllm-dspark
|
||||||
|
# - benchmark client: /data/user1/yy/envs/sglang/bin/python -m sglang.bench_serving
|
||||||
|
# - model: /data/models/DeepSeek-V4-Flash-DSpark
|
||||||
|
# - server port: 30004
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bash run_dspark_benchmark_grid.sh
|
||||||
|
#
|
||||||
|
# To only run benchmarks against an already-running server:
|
||||||
|
# SKIP_MANAGE_SERVER=1 bash run_dspark_benchmark_grid.sh
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
START_SCRIPT="${SCRIPT_DIR}/../start_dsv4_dspark_8card.sh"
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid_${RUN_ID}}"
|
||||||
|
LOG_DIR="${RESULT_ROOT}/logs"
|
||||||
|
PORT="${PORT:-30004}"
|
||||||
|
PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid"
|
||||||
|
|
||||||
|
mkdir -p "${LOG_DIR}"
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "[$(date --iso-8601=seconds)] $*" | tee -a "${LOG_DIR}/orchestrator.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
is_server_healthy() {
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_server() {
|
||||||
|
if [[ -f "${PID_FILE}" ]]; then
|
||||||
|
local pid
|
||||||
|
pid="$(cat "${PID_FILE}")"
|
||||||
|
if kill -0 "${pid}" 2>/dev/null; then
|
||||||
|
log "stopping existing dspark server pid=${pid}"
|
||||||
|
kill "${pid}" 2>/dev/null || true
|
||||||
|
sleep 5
|
||||||
|
kill -9 "${pid}" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
rm -f "${PID_FILE}"
|
||||||
|
fi
|
||||||
|
# Fallback: kill any lingering vllm serve processes for this model
|
||||||
|
pkill -9 -f "vllm serve.*DeepSeek-V4-Flash-DSpark" 2>/dev/null || true
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
|
||||||
|
start_server() {
|
||||||
|
log "starting dspark server with ${START_SCRIPT}"
|
||||||
|
if [[ ! -x "${START_SCRIPT}" ]]; then
|
||||||
|
log "error: start script not found or not executable: ${START_SCRIPT}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
bash "${START_SCRIPT}" >> "${LOG_DIR}/server.outer.log" 2>&1
|
||||||
|
if ! is_server_healthy; then
|
||||||
|
log "error: dspark server failed to become healthy"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
log "dspark server is healthy"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_phase() {
|
||||||
|
local name="$1"
|
||||||
|
local script="$2"
|
||||||
|
local log_file="${LOG_DIR}/${name}.log"
|
||||||
|
|
||||||
|
log "running ${name}: ${script}"
|
||||||
|
RUN_ID="${RUN_ID}" RESULT_ROOT="${RESULT_ROOT}" bash "${script}" 2>&1 | tee "${log_file}"
|
||||||
|
log "finished ${name}"
|
||||||
|
}
|
||||||
|
|
||||||
|
on_exit() {
|
||||||
|
local code="$?"
|
||||||
|
log "orchestrator exiting with code=${code}"
|
||||||
|
if [[ -z "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||||
|
stop_server
|
||||||
|
fi
|
||||||
|
exit "${code}"
|
||||||
|
}
|
||||||
|
trap on_exit EXIT
|
||||||
|
|
||||||
|
main() {
|
||||||
|
log "run_id=${RUN_ID}"
|
||||||
|
log "result_root=${RESULT_ROOT}"
|
||||||
|
log "script_dir=${SCRIPT_DIR}"
|
||||||
|
|
||||||
|
if [[ -n "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||||
|
log "SKIP_MANAGE_SERVER is set, assuming server is already running"
|
||||||
|
if ! is_server_healthy; then
|
||||||
|
log "error: no healthy server found at port ${PORT}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
stop_server
|
||||||
|
start_server
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "===== DSpark BENCHMARK START ====="
|
||||||
|
run_phase "dspark_p1" "${SCRIPT_DIR}/bench_dspark_p1.sh"
|
||||||
|
run_phase "dspark_p2" "${SCRIPT_DIR}/bench_dspark_p2.sh"
|
||||||
|
run_phase "dspark_p3" "${SCRIPT_DIR}/bench_dspark_p3.sh"
|
||||||
|
log "===== DSpark BENCHMARK DONE ====="
|
||||||
|
|
||||||
|
log "all results saved to ${RESULT_ROOT}"
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
103
scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
Executable file
103
scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
Executable file
@ -0,0 +1,103 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
# Compare vllm-dspark with --spec-tokens 3 vs 5.
|
||||||
|
# Uses the parameterized start script and focused benchmark.
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
START_SCRIPT="${SCRIPT_DIR}/../start_dsv4_dspark_8card_param.sh"
|
||||||
|
BENCH_SCRIPT="${SCRIPT_DIR}/bench_dspark_focused.sh"
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_st_comparison_${RUN_ID}}"
|
||||||
|
LOG_DIR="${RESULT_ROOT}/logs"
|
||||||
|
PORT="${PORT:-30004}"
|
||||||
|
PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid"
|
||||||
|
WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}"
|
||||||
|
|
||||||
|
mkdir -p "${LOG_DIR}"
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "[$(date --iso-8601=seconds)] $*" | tee -a "${LOG_DIR}/orchestrator.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
is_server_healthy() {
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_server() {
|
||||||
|
if [[ -f "${PID_FILE}" ]]; then
|
||||||
|
local pid
|
||||||
|
pid="$(cat "${PID_FILE}" 2>/dev/null || true)"
|
||||||
|
if [[ -n "${pid}" ]] && kill -0 "${pid}" 2>/dev/null; then
|
||||||
|
log "stopping existing dspark server pid=${pid}"
|
||||||
|
kill "${pid}" 2>/dev/null || true
|
||||||
|
sleep 5
|
||||||
|
kill -9 "${pid}" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
rm -f "${PID_FILE}"
|
||||||
|
fi
|
||||||
|
local pids
|
||||||
|
pids="$(pgrep -f "VLLM::|vllm serve.*DeepSeek-V4-Flash-DSpark" 2>/dev/null || true)"
|
||||||
|
if [[ -n "${pids}" ]]; then
|
||||||
|
for pid in ${pids}; do
|
||||||
|
if [[ "$pid" != "$$" ]]; then kill -9 "$pid" 2>/dev/null || true; fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
|
||||||
|
start_server() {
|
||||||
|
local spec_tokens="$1"
|
||||||
|
log "starting dspark server with spec-tokens=${spec_tokens}"
|
||||||
|
if [[ ! -x "${START_SCRIPT}" ]]; then
|
||||||
|
log "error: start script not found or not executable: ${START_SCRIPT}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
SPEC_TOKENS="${spec_tokens}" bash "${START_SCRIPT}" >> "${LOG_DIR}/server_st${spec_tokens}.outer.log" 2>&1
|
||||||
|
if ! is_server_healthy; then
|
||||||
|
log "error: dspark server (st=${spec_tokens}) failed to become healthy"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
log "dspark server (st=${spec_tokens}) is healthy"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_benchmark() {
|
||||||
|
local spec_tokens="$1"
|
||||||
|
local log_file="${LOG_DIR}/focused_st${spec_tokens}.log"
|
||||||
|
log "running focused benchmark with spec-tokens=${spec_tokens}, warmup=${WARMUP_REQUESTS}"
|
||||||
|
RUN_ID="${RUN_ID}_st${spec_tokens}" RESULT_ROOT="${RESULT_ROOT}" WARMUP_REQUESTS="${WARMUP_REQUESTS}" bash "${BENCH_SCRIPT}" 2>&1 | tee "${log_file}"
|
||||||
|
log "finished focused benchmark with spec-tokens=${spec_tokens}"
|
||||||
|
}
|
||||||
|
|
||||||
|
on_exit() {
|
||||||
|
local code="$?"
|
||||||
|
log "orchestrator exiting with code=${code}"
|
||||||
|
stop_server
|
||||||
|
exit "${code}"
|
||||||
|
}
|
||||||
|
trap on_exit EXIT
|
||||||
|
|
||||||
|
main() {
|
||||||
|
log "run_id=${RUN_ID}"
|
||||||
|
log "result_root=${RESULT_ROOT}"
|
||||||
|
log "warmup_requests=${WARMUP_REQUESTS}"
|
||||||
|
|
||||||
|
log "===== SPEC-TOKENS 3 START ====="
|
||||||
|
stop_server
|
||||||
|
start_server 3
|
||||||
|
run_benchmark 3
|
||||||
|
stop_server
|
||||||
|
|
||||||
|
log "cooldown 30s"
|
||||||
|
sleep 30
|
||||||
|
|
||||||
|
log "===== SPEC-TOKENS 5 START ====="
|
||||||
|
start_server 5
|
||||||
|
run_benchmark 5
|
||||||
|
stop_server
|
||||||
|
|
||||||
|
log "===== COMPARISON DONE ====="
|
||||||
|
log "all results saved to ${RESULT_ROOT}"
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
128
scripts/benchmark_dsv4_backend_comparison.sh
Normal file
128
scripts/benchmark_dsv4_backend_comparison.sh
Normal file
@ -0,0 +1,128 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
# Reproducible SGLang vs vLLM backend comparison for DeepSeek-V4-Flash.
|
||||||
|
# Produces JSONL raw outputs compatible with the legacy 2026-07-07 sweep.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# bash scripts/benchmark_dsv4_backend_comparison.sh [vllm|sglang|all]
|
||||||
|
#
|
||||||
|
# Output:
|
||||||
|
# bench_results/dsv4_backend_comparison_${RUN_ID}/raw_outputs/
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||||
|
|
||||||
|
BACKEND_FILTER="${1:-all}"
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-${ROOT_DIR}/bench_results/dsv4_backend_comparison_${RUN_ID}}"
|
||||||
|
RAW_DIR="${RESULT_ROOT}/raw_outputs"
|
||||||
|
LOG_DIR="${RESULT_ROOT}/logs"
|
||||||
|
|
||||||
|
mkdir -p "${RAW_DIR}" "${LOG_DIR}"
|
||||||
|
|
||||||
|
BENCH_PY="${ROOT_DIR}/envs/sglang/bin/python"
|
||||||
|
DATASET="${ROOT_DIR}/datasets/ShareGPT_filtered_chat.json"
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash"
|
||||||
|
SERVED_MODEL_NAME="deepseek-v4-flash"
|
||||||
|
NUM_PROMPTS=512
|
||||||
|
WARMUP=100
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "[$(date --iso-8601=seconds)] $*" | tee -a "${LOG_DIR}/orchestrator.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_case() {
|
||||||
|
local backend="$1"
|
||||||
|
local port="$2"
|
||||||
|
local concurrency="$3"
|
||||||
|
local input_len="$4"
|
||||||
|
local output_len="$5"
|
||||||
|
local output_file="${RAW_DIR}/${backend}_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||||
|
local detail_log="${LOG_DIR}/${backend}_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||||
|
|
||||||
|
log "running ${backend}: concurrency=${concurrency} input=${input_len} output=${output_len} -> ${output_file}"
|
||||||
|
|
||||||
|
"${BENCH_PY}" -m sglang.bench_serving \
|
||||||
|
--backend "${backend}" \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "${port}" \
|
||||||
|
--model "${MODEL}" \
|
||||||
|
--served-model-name "${SERVED_MODEL_NAME}" \
|
||||||
|
--dataset-name random \
|
||||||
|
--dataset-path "${DATASET}" \
|
||||||
|
--random-input-len "${input_len}" \
|
||||||
|
--random-output-len "${output_len}" \
|
||||||
|
--num-prompts "${NUM_PROMPTS}" \
|
||||||
|
--max-concurrency "${concurrency}" \
|
||||||
|
--warmup-requests "${WARMUP}" \
|
||||||
|
--output-file "${output_file}" \
|
||||||
|
--output-details \
|
||||||
|
> "${detail_log}" 2>&1
|
||||||
|
|
||||||
|
log "finished ${backend} c=${concurrency} i=${input_len} o=${output_len}"
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
log "run_id=${RUN_ID}"
|
||||||
|
log "result_root=${RESULT_ROOT}"
|
||||||
|
log "backend_filter=${BACKEND_FILTER}"
|
||||||
|
|
||||||
|
# This script assumes a healthy server is already running on the target port.
|
||||||
|
if [[ "${BACKEND_FILTER}" == "all" || "${BACKEND_FILTER}" == "sglang" ]]; then
|
||||||
|
log "===== SGLang sweep (port 30000) ====="
|
||||||
|
run_case sglang 30000 32 512 256
|
||||||
|
run_case sglang 30000 128 512 256
|
||||||
|
run_case sglang 30000 256 512 256
|
||||||
|
run_case sglang 30000 512 512 256
|
||||||
|
run_case sglang 30000 512 512 2000
|
||||||
|
run_case sglang 30000 32 4000 512
|
||||||
|
run_case sglang 30000 128 4000 512
|
||||||
|
run_case sglang 30000 512 4000 512
|
||||||
|
run_case sglang 30000 32 16000 512
|
||||||
|
run_case sglang 30000 128 16000 512
|
||||||
|
run_case sglang 30000 512 1000 256
|
||||||
|
run_case sglang 30000 768 1000 256
|
||||||
|
run_case sglang 30000 1024 1000 256
|
||||||
|
run_case sglang 30000 512 1000 1000
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "${BACKEND_FILTER}" == "all" || "${BACKEND_FILTER}" == "vllm" ]]; then
|
||||||
|
log "===== vLLM sweep (port 8000) ====="
|
||||||
|
run_case vllm 8000 32 512 256
|
||||||
|
run_case vllm 8000 128 512 256
|
||||||
|
run_case vllm 8000 256 512 256
|
||||||
|
run_case vllm 8000 128 512 2000
|
||||||
|
run_case vllm 8000 256 512 2000
|
||||||
|
run_case vllm 8000 512 512 2000
|
||||||
|
run_case vllm 8000 32 1000 256
|
||||||
|
run_case vllm 8000 128 1000 256
|
||||||
|
run_case vllm 8000 256 1000 256
|
||||||
|
run_case vllm 8000 512 1000 256
|
||||||
|
run_case vllm 8000 768 1000 256
|
||||||
|
run_case vllm 8000 1024 1000 256
|
||||||
|
run_case vllm 8000 32 1000 1000
|
||||||
|
run_case vllm 8000 128 1000 1000
|
||||||
|
run_case vllm 8000 256 1000 1000
|
||||||
|
run_case vllm 8000 512 1000 1000
|
||||||
|
run_case vllm 8000 1024 1000 1000
|
||||||
|
run_case vllm 8000 32 4000 512
|
||||||
|
run_case vllm 8000 128 4000 512
|
||||||
|
run_case vllm 8000 256 4000 512
|
||||||
|
run_case vllm 8000 512 4000 512
|
||||||
|
run_case vllm 8000 32 8000 1000
|
||||||
|
run_case vllm 8000 128 8000 1000
|
||||||
|
run_case vllm 8000 256 8000 1000
|
||||||
|
run_case vllm 8000 512 8000 1000
|
||||||
|
run_case vllm 8000 32 16000 512
|
||||||
|
run_case vllm 8000 128 16000 512
|
||||||
|
run_case vllm 8000 256 16000 512
|
||||||
|
run_case vllm 8000 32 32000 512
|
||||||
|
run_case vllm 8000 128 32000 512
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "===== Done ====="
|
||||||
|
log "all outputs saved to ${RAW_DIR}"
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
27
scripts/install_vllm_dspark.sh
Executable file
27
scripts/install_vllm_dspark.sh
Executable file
@ -0,0 +1,27 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
export TMPDIR=/data/tmp
|
||||||
|
export TEMP=/data/tmp
|
||||||
|
export TMP=/data/tmp
|
||||||
|
export PIP_CACHE_DIR=/data/tmp/pip-cache
|
||||||
|
export SETUPTOOLS_SCM_PRETEND_VERSION=0.23.1.dev788
|
||||||
|
mkdir -p $PIP_CACHE_DIR
|
||||||
|
|
||||||
|
cd /data/user1/yy/vllm-main
|
||||||
|
PYTHON=/data/user1/yy/envs/vllm-dspark/bin/python
|
||||||
|
PIP=/data/user1/yy/envs/vllm-dspark/bin/pip
|
||||||
|
|
||||||
|
echo "Installing torch 2.11.0 with CUDA 12.9..."
|
||||||
|
$PIP install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu129 --cache-dir $PIP_CACHE_DIR
|
||||||
|
|
||||||
|
echo "Installing numpy and build dependencies..."
|
||||||
|
$PIP install numpy cmake ninja packaging "setuptools>=77.0.3,<81.0.0" setuptools-scm setuptools-rust wheel jinja2 --cache-dir $PIP_CACHE_DIR
|
||||||
|
|
||||||
|
echo "Installing vllm from source (this will compile CUDA kernels, may take 30-60 min)..."
|
||||||
|
$PIP install -e . --no-build-isolation --cache-dir $PIP_CACHE_DIR
|
||||||
|
|
||||||
|
echo "Verifying installation..."
|
||||||
|
$PYTHON -c "import vllm; print('vllm version:', vllm.__version__)"
|
||||||
|
|
||||||
|
echo "Installation complete."
|
||||||
261
scripts/profile_dspark.py
Executable file
261
scripts/profile_dspark.py
Executable file
@ -0,0 +1,261 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Profile DSpark inference with Nsight Systems and/or PyTorch profiler.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python scripts/profile_dspark.py --config dspark-st3 --concurrency 64 --tool nsys
|
||||||
|
python scripts/profile_dspark.py --config nospec --concurrency 64 --tool nsys
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import urllib.request
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path("/data/user1/yy")
|
||||||
|
RESULT_DIR = ROOT / "bench_results" / f"dspark_profile_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
|
||||||
|
RESULT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
DATASET = "/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"
|
||||||
|
NUM_PROMPTS = 200
|
||||||
|
SEED = 42
|
||||||
|
OUTPUT_LEN = 256
|
||||||
|
HOST = "127.0.0.1"
|
||||||
|
|
||||||
|
CONFIGS = {
|
||||||
|
"dspark-st3": {
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"port": 30004,
|
||||||
|
"spec_tokens": 3,
|
||||||
|
"cmd": [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve",
|
||||||
|
"/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--tensor-parallel-size", "8",
|
||||||
|
"--kv-cache-dtype", "fp8",
|
||||||
|
"--block-size", "256",
|
||||||
|
"--max-model-len", "auto",
|
||||||
|
"--max-num-seqs", "256",
|
||||||
|
"--gpu-memory-utilization", "0.75",
|
||||||
|
"--tokenizer-mode", "deepseek_v4",
|
||||||
|
"--reasoning-parser", "deepseek_v4",
|
||||||
|
"--spec-method", "dspark",
|
||||||
|
"--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--spec-tokens", "3",
|
||||||
|
"--no-disable-hybrid-kv-cache-manager",
|
||||||
|
"--disable-uvicorn-access-log",
|
||||||
|
"--port", "30004",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
"dspark-st5": {
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"port": 30004,
|
||||||
|
"spec_tokens": 5,
|
||||||
|
"cmd": [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve",
|
||||||
|
"/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--tensor-parallel-size", "8",
|
||||||
|
"--kv-cache-dtype", "fp8",
|
||||||
|
"--block-size", "256",
|
||||||
|
"--max-model-len", "auto",
|
||||||
|
"--max-num-seqs", "256",
|
||||||
|
"--gpu-memory-utilization", "0.75",
|
||||||
|
"--tokenizer-mode", "deepseek_v4",
|
||||||
|
"--reasoning-parser", "deepseek_v4",
|
||||||
|
"--spec-method", "dspark",
|
||||||
|
"--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--spec-tokens", "5",
|
||||||
|
"--no-disable-hybrid-kv-cache-manager",
|
||||||
|
"--disable-uvicorn-access-log",
|
||||||
|
"--port", "30004",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
"nospec": {
|
||||||
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"port": 30004,
|
||||||
|
"cmd": [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve",
|
||||||
|
"/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
|
"--trust-remote-code",
|
||||||
|
"--tensor-parallel-size", "8",
|
||||||
|
"--kv-cache-dtype", "fp8",
|
||||||
|
"--block-size", "256",
|
||||||
|
"--max-model-len", "auto",
|
||||||
|
"--max-num-seqs", "256",
|
||||||
|
"--gpu-memory-utilization", "0.75",
|
||||||
|
"--tokenizer-mode", "deepseek_v4",
|
||||||
|
"--reasoning-parser", "deepseek_v4",
|
||||||
|
"--no-disable-hybrid-kv-cache-manager",
|
||||||
|
"--disable-uvicorn-access-log",
|
||||||
|
"--port", "30004",
|
||||||
|
],
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg):
|
||||||
|
print(f"[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] {msg}", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def wait_for_health(port, timeout=300):
|
||||||
|
url = f"http://{HOST}:{port}/health"
|
||||||
|
start = time.time()
|
||||||
|
while time.time() - start < timeout:
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(url, timeout=5) as resp:
|
||||||
|
if resp.status == 200:
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
time.sleep(2)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def run_warmup(port, model, n=20):
|
||||||
|
"""Run a few warmup requests to stabilize CUDA graphs."""
|
||||||
|
log(f"Running {n} warmup requests...")
|
||||||
|
for i in range(n):
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"http://{HOST}:{port}/v1/completions",
|
||||||
|
data=json.dumps({
|
||||||
|
"model": model,
|
||||||
|
"prompt": "Hello, how are you?",
|
||||||
|
"max_tokens": 32,
|
||||||
|
"temperature": 0,
|
||||||
|
"seed": SEED,
|
||||||
|
}).encode(),
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
with urllib.request.urlopen(req, timeout=60) as resp:
|
||||||
|
resp.read()
|
||||||
|
except Exception as e:
|
||||||
|
log(f"Warmup request {i} failed: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
def run_benchmark(port, model, concurrency, result_dir):
|
||||||
|
result_file = result_dir / f"profile_c{concurrency}.json"
|
||||||
|
log_file = result_dir / f"profile_c{concurrency}.log"
|
||||||
|
bench_cmd = [
|
||||||
|
"/data/user1/yy/envs/vllm-dspark/bin/vllm", "bench", "serve",
|
||||||
|
"--host", HOST,
|
||||||
|
"--port", str(port),
|
||||||
|
"--backend", "openai",
|
||||||
|
"--dataset-name", "sharegpt",
|
||||||
|
"--dataset-path", DATASET,
|
||||||
|
"--sharegpt-output-len", str(OUTPUT_LEN),
|
||||||
|
"--num-prompts", str(NUM_PROMPTS),
|
||||||
|
"--max-concurrency", str(concurrency),
|
||||||
|
"--endpoint", "/v1/completions",
|
||||||
|
"--model", model,
|
||||||
|
"--seed", str(SEED),
|
||||||
|
"--save-result",
|
||||||
|
"--result-dir", str(result_dir),
|
||||||
|
"--result-filename", result_file.name,
|
||||||
|
]
|
||||||
|
log(f"Running benchmark c={concurrency}...")
|
||||||
|
start = time.time()
|
||||||
|
with open(log_file, "w") as f:
|
||||||
|
proc = subprocess.Popen(bench_cmd, stdout=f, stderr=subprocess.STDOUT)
|
||||||
|
proc.wait()
|
||||||
|
duration = time.time() - start
|
||||||
|
log(f"Benchmark c={concurrency} finished in {duration:.1f}s, exit={proc.returncode}")
|
||||||
|
return result_file
|
||||||
|
|
||||||
|
|
||||||
|
def start_service(config, tool, result_dir):
|
||||||
|
name = config
|
||||||
|
cfg = CONFIGS[config]
|
||||||
|
port = cfg["port"]
|
||||||
|
base_cmd = cfg["cmd"]
|
||||||
|
|
||||||
|
if tool == "nsys":
|
||||||
|
nsys_output = result_dir / f"{name}_nsys"
|
||||||
|
cmd = [
|
||||||
|
"nsys", "profile",
|
||||||
|
"--sample=none",
|
||||||
|
"--cpuctxsw=none",
|
||||||
|
"--trace=cuda,nvtx,osrt",
|
||||||
|
"--output", str(nsys_output),
|
||||||
|
"--force-overwrite", "true",
|
||||||
|
"--delay", "30",
|
||||||
|
"--duration", "15",
|
||||||
|
] + base_cmd
|
||||||
|
else:
|
||||||
|
cmd = base_cmd
|
||||||
|
|
||||||
|
log(f"Starting service for {name} (tool={tool}) on port {port}...")
|
||||||
|
log_file = result_dir / f"{name}_service.log"
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5,6,7"
|
||||||
|
env["TMPDIR"] = str(ROOT / "tmp")
|
||||||
|
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
cmd,
|
||||||
|
stdout=open(log_file, "w"),
|
||||||
|
stderr=subprocess.STDOUT,
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
if not wait_for_health(port):
|
||||||
|
log(f"ERROR: {name} failed to start")
|
||||||
|
proc.terminate()
|
||||||
|
return None
|
||||||
|
log(f"{name} is ready")
|
||||||
|
return proc
|
||||||
|
|
||||||
|
|
||||||
|
def stop_service(proc, name):
|
||||||
|
if proc is None:
|
||||||
|
return
|
||||||
|
log(f"Stopping {name} (pid {proc.pid})...")
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=60)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
proc.wait()
|
||||||
|
log(f"{name} stopped")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--config", required=True, choices=list(CONFIGS.keys()))
|
||||||
|
parser.add_argument("--concurrency", type=int, default=64)
|
||||||
|
parser.add_argument("--tool", default="nsys", choices=["nsys"])
|
||||||
|
parser.add_argument("--skip-bench", action="store_true", help="Only capture service startup/warmup, skip benchmark")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
result_dir = RESULT_DIR / f"{args.config}_c{args.concurrency}_{args.tool}"
|
||||||
|
result_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
log(f"Results will be saved to {result_dir}")
|
||||||
|
|
||||||
|
proc = start_service(args.config, args.tool, result_dir)
|
||||||
|
if proc is None:
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
try:
|
||||||
|
run_warmup(CONFIGS[args.config]["port"], CONFIGS[args.config]["model"], n=30)
|
||||||
|
if not args.skip_bench:
|
||||||
|
run_benchmark(
|
||||||
|
CONFIGS[args.config]["port"],
|
||||||
|
CONFIGS[args.config]["model"],
|
||||||
|
args.concurrency,
|
||||||
|
result_dir,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
log("--skip-bench set; sleeping 60s to capture warmup/steady state...")
|
||||||
|
time.sleep(60)
|
||||||
|
finally:
|
||||||
|
stop_service(proc, args.config)
|
||||||
|
|
||||||
|
log(f"All done. Results in {result_dir}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
157
scripts/run_sglang_benchmark.sh
Executable file
157
scripts/run_sglang_benchmark.sh
Executable file
@ -0,0 +1,157 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Comprehensive benchmark for sglang 8-card DeepSeek-V4-Flash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p bench_results logs
|
||||||
|
|
||||||
|
PYTHON="/data/user1/yy/envs/sglang/bin/python"
|
||||||
|
BENCH="-m sglang.bench_serving"
|
||||||
|
HOST="127.0.0.1"
|
||||||
|
PORT="30000"
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash"
|
||||||
|
DATASET="/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"
|
||||||
|
NUM_PROMPTS=2000
|
||||||
|
SEED=42
|
||||||
|
|
||||||
|
RESULT_DIR="bench_results/sglang_8card_systematic_$(date +%Y%m%d_%H%M%S)"
|
||||||
|
mkdir -p "${RESULT_DIR}"
|
||||||
|
|
||||||
|
SUMMARY_FILE="${RESULT_DIR}/summary.json"
|
||||||
|
LOG_FILE="${RESULT_DIR}/benchmark.log"
|
||||||
|
|
||||||
|
# Use a marker file for summary accumulation
|
||||||
|
SUMMARY_TMP="${RESULT_DIR}/.summary_tmp.json"
|
||||||
|
echo '[]' > "${SUMMARY_TMP}"
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "${LOG_FILE}"
|
||||||
|
}
|
||||||
|
|
||||||
|
wait_for_server() {
|
||||||
|
log "Waiting for sglang server at ${HOST}:${PORT} to become ready..."
|
||||||
|
local retries=0
|
||||||
|
local max_retries=120
|
||||||
|
while ! curl -s "http://${HOST}:${PORT}/v1/models" > /dev/null 2>&1; do
|
||||||
|
retries=$((retries + 1))
|
||||||
|
if [ "${retries}" -ge "${max_retries}" ]; then
|
||||||
|
log "ERROR: Server not ready after ${max_retries} retries"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
log "Server is ready."
|
||||||
|
}
|
||||||
|
|
||||||
|
run_bench() {
|
||||||
|
local name="$1"
|
||||||
|
shift
|
||||||
|
local output_file="${RESULT_DIR}/${name}.json"
|
||||||
|
local detail_log="${RESULT_DIR}/${name}.log"
|
||||||
|
|
||||||
|
log "---------------------------------------------------"
|
||||||
|
log "Running benchmark: ${name}"
|
||||||
|
log "Args: $*"
|
||||||
|
log "Output: ${output_file}"
|
||||||
|
|
||||||
|
${PYTHON} ${BENCH} \
|
||||||
|
--backend sglang \
|
||||||
|
--host "${HOST}" \
|
||||||
|
--port "${PORT}" \
|
||||||
|
--dataset-name sharegpt \
|
||||||
|
--dataset-path "${DATASET}" \
|
||||||
|
--num-prompts "${NUM_PROMPTS}" \
|
||||||
|
--model "${MODEL}" \
|
||||||
|
--output-file "${output_file}" \
|
||||||
|
--output-details \
|
||||||
|
--seed "${SEED}" \
|
||||||
|
"$@" \
|
||||||
|
> "${detail_log}" 2>&1
|
||||||
|
|
||||||
|
local status=$?
|
||||||
|
if [ ${status} -ne 0 ]; then
|
||||||
|
log "ERROR: Benchmark ${name} failed with exit code ${status}"
|
||||||
|
return ${status}
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Benchmark ${name} completed successfully."
|
||||||
|
|
||||||
|
# Extract key metrics from the log
|
||||||
|
local req_throughput=$(grep -oP 'Request throughput \(req/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local input_throughput=$(grep -oP 'Input token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local output_throughput=$(grep -oP 'Output token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local total_throughput=$(grep -oP 'Total token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local mean_ttft=$(grep -oP 'Mean TTFT \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local mean_tpot=$(grep -oP 'Mean TPOT \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local mean_e2e=$(grep -oP 'Mean E2E Latency \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local duration=$(grep -oP 'Benchmark duration \(s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local accept_length=$(grep -oP 'Accept length:\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
|
||||||
|
# Append to summary tmp
|
||||||
|
${PYTHON} - <<PYEOF
|
||||||
|
import json
|
||||||
|
entry = {
|
||||||
|
"name": "${name}",
|
||||||
|
"args": "$*",
|
||||||
|
"duration_s": ${duration},
|
||||||
|
"request_throughput": ${req_throughput},
|
||||||
|
"input_token_throughput": ${input_throughput},
|
||||||
|
"output_token_throughput": ${output_throughput},
|
||||||
|
"total_token_throughput": ${total_throughput},
|
||||||
|
"mean_ttft_ms": ${mean_ttft},
|
||||||
|
"mean_tpot_ms": ${mean_tpot},
|
||||||
|
"mean_e2e_latency_ms": ${mean_e2e},
|
||||||
|
"accept_length": ${accept_length},
|
||||||
|
"output_file": "${output_file}"
|
||||||
|
}
|
||||||
|
with open("${SUMMARY_TMP}", "r") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
data.append(entry)
|
||||||
|
with open("${SUMMARY_TMP}", "w") as f:
|
||||||
|
json.dump(data, f, indent=2)
|
||||||
|
PYEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
log "Starting systematic benchmark for sglang 8-card DeepSeek-V4-Flash"
|
||||||
|
log "Model: ${MODEL}"
|
||||||
|
log "Dataset: ${DATASET}"
|
||||||
|
log "Num prompts per test: ${NUM_PROMPTS}"
|
||||||
|
log "Results will be saved to: ${RESULT_DIR}"
|
||||||
|
|
||||||
|
wait_for_server
|
||||||
|
|
||||||
|
# Concurrency sweep: simulate different real-world load levels
|
||||||
|
log "=== Phase 1: Concurrency sweep ==="
|
||||||
|
for c in 1 8 16 32 64 128; do
|
||||||
|
run_bench "sharegpt_concurrency_${c}" --max-concurrency "${c}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Request-rate sweep: simulate Poisson-like traffic
|
||||||
|
log "=== Phase 2: Request rate sweep ==="
|
||||||
|
for r in 1 2 4 8 16; do
|
||||||
|
run_bench "sharegpt_rps_${r}" --request-rate "${r}"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Move final summary
|
||||||
|
mv "${SUMMARY_TMP}" "${SUMMARY_FILE}"
|
||||||
|
|
||||||
|
log "=== All benchmarks completed ==="
|
||||||
|
log "Summary saved to: ${SUMMARY_FILE}"
|
||||||
|
log "Detailed results in: ${RESULT_DIR}"
|
||||||
|
|
||||||
|
# Print summary table
|
||||||
|
${PYTHON} - <<PYEOF
|
||||||
|
import json
|
||||||
|
with open("${SUMMARY_FILE}", "r") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
|
||||||
|
print("\n============ Benchmark Summary ============")
|
||||||
|
print(f"{'Name':<30} {'Req/s':>8} {'In tok/s':>10} {'Out tok/s':>10} {'Total tok/s':>12} {'TTFT':>8} {'TPOT':>8} {'E2E':>10}")
|
||||||
|
print("-" * 110)
|
||||||
|
for entry in data:
|
||||||
|
print(f"{entry['name']:<30} {entry['request_throughput']:>8.2f} {entry['input_token_throughput']:>10.2f} {entry['output_token_throughput']:>10.2f} {entry['total_token_throughput']:>12.2f} {entry['mean_ttft_ms']:>8.1f} {entry['mean_tpot_ms']:>8.1f} {entry['mean_e2e_latency_ms']:>10.1f}")
|
||||||
|
PYEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
143
scripts/run_sglang_max_throughput.sh
Executable file
143
scripts/run_sglang_max_throughput.sh
Executable file
@ -0,0 +1,143 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Max throughput benchmark for sglang 8-card DeepSeek-V4-Flash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p bench_results logs
|
||||||
|
|
||||||
|
PYTHON="/data/user1/yy/envs/sglang/bin/python"
|
||||||
|
BENCH="-m sglang.bench_serving"
|
||||||
|
HOST="127.0.0.1"
|
||||||
|
PORT="30000"
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash"
|
||||||
|
DATASET="/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json"
|
||||||
|
NUM_PROMPTS=2000
|
||||||
|
SEED=42
|
||||||
|
|
||||||
|
RESULT_DIR="bench_results/sglang_8card_max_throughput_$(date +%Y%m%d_%H%M%S)"
|
||||||
|
mkdir -p "${RESULT_DIR}"
|
||||||
|
|
||||||
|
SUMMARY_FILE="${RESULT_DIR}/summary.json"
|
||||||
|
LOG_FILE="${RESULT_DIR}/benchmark.log"
|
||||||
|
SUMMARY_TMP="${RESULT_DIR}/.summary_tmp.json"
|
||||||
|
echo '[]' > "${SUMMARY_TMP}"
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "${LOG_FILE}"
|
||||||
|
}
|
||||||
|
|
||||||
|
wait_for_server() {
|
||||||
|
log "Waiting for sglang server at ${HOST}:${PORT} to become ready..."
|
||||||
|
local retries=0
|
||||||
|
local max_retries=120
|
||||||
|
while ! curl -s "http://${HOST}:${PORT}/v1/models" > /dev/null 2>&1; do
|
||||||
|
retries=$((retries + 1))
|
||||||
|
if [ "${retries}" -ge "${max_retries}" ]; then
|
||||||
|
log "ERROR: Server not ready after ${max_retries} retries"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
log "Server is ready."
|
||||||
|
}
|
||||||
|
|
||||||
|
run_bench() {
|
||||||
|
local name="$1"
|
||||||
|
shift
|
||||||
|
local output_file="${RESULT_DIR}/${name}.json"
|
||||||
|
local detail_log="${RESULT_DIR}/${name}.log"
|
||||||
|
|
||||||
|
log "---------------------------------------------------"
|
||||||
|
log "Running benchmark: ${name}"
|
||||||
|
log "Args: $*"
|
||||||
|
log "Output: ${output_file}"
|
||||||
|
|
||||||
|
${PYTHON} ${BENCH} \
|
||||||
|
--backend sglang \
|
||||||
|
--host "${HOST}" \
|
||||||
|
--port "${PORT}" \
|
||||||
|
--dataset-name sharegpt \
|
||||||
|
--dataset-path "${DATASET}" \
|
||||||
|
--num-prompts "${NUM_PROMPTS}" \
|
||||||
|
--model "${MODEL}" \
|
||||||
|
--output-file "${output_file}" \
|
||||||
|
--output-details \
|
||||||
|
--seed "${SEED}" \
|
||||||
|
"$@" \
|
||||||
|
> "${detail_log}" 2>&1
|
||||||
|
|
||||||
|
local status=$?
|
||||||
|
if [ ${status} -ne 0 ]; then
|
||||||
|
log "ERROR: Benchmark ${name} failed with exit code ${status}"
|
||||||
|
return ${status}
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Benchmark ${name} completed successfully."
|
||||||
|
|
||||||
|
local req_throughput=$(grep -oP 'Request throughput \(req/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local input_throughput=$(grep -oP 'Input token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local output_throughput=$(grep -oP 'Output token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local total_throughput=$(grep -oP 'Total token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local mean_ttft=$(grep -oP 'Mean TTFT \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local mean_tpot=$(grep -oP 'Mean TPOT \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local mean_e2e=$(grep -oP 'Mean E2E Latency \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local duration=$(grep -oP 'Benchmark duration \(s\):\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
local accept_length=$(grep -oP 'Accept length:\s+\K[0-9.]+' "${detail_log}" || echo "null")
|
||||||
|
|
||||||
|
${PYTHON} - <<PYEOF
|
||||||
|
import json
|
||||||
|
entry = {
|
||||||
|
"name": "${name}",
|
||||||
|
"args": "$*",
|
||||||
|
"duration_s": ${duration},
|
||||||
|
"request_throughput": ${req_throughput},
|
||||||
|
"input_token_throughput": ${input_throughput},
|
||||||
|
"output_token_throughput": ${output_throughput},
|
||||||
|
"total_token_throughput": ${total_throughput},
|
||||||
|
"mean_ttft_ms": ${mean_ttft},
|
||||||
|
"mean_tpot_ms": ${mean_tpot},
|
||||||
|
"mean_e2e_latency_ms": ${mean_e2e},
|
||||||
|
"accept_length": ${accept_length},
|
||||||
|
"output_file": "${output_file}"
|
||||||
|
}
|
||||||
|
with open("${SUMMARY_TMP}", "r") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
data.append(entry)
|
||||||
|
with open("${SUMMARY_TMP}", "w") as f:
|
||||||
|
json.dump(data, f, indent=2)
|
||||||
|
PYEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
log "Starting max-throughput benchmark for sglang 8-card DeepSeek-V4-Flash"
|
||||||
|
log "Model: ${MODEL}"
|
||||||
|
log "Dataset: ${DATASET}"
|
||||||
|
log "Num prompts per test: ${NUM_PROMPTS}"
|
||||||
|
log "Results will be saved to: ${RESULT_DIR}"
|
||||||
|
|
||||||
|
wait_for_server
|
||||||
|
|
||||||
|
log "=== Max concurrency sweep ==="
|
||||||
|
for c in 128 256 512 1024; do
|
||||||
|
run_bench "sharegpt_concurrency_${c}" --max-concurrency "${c}"
|
||||||
|
done
|
||||||
|
|
||||||
|
mv "${SUMMARY_TMP}" "${SUMMARY_FILE}"
|
||||||
|
|
||||||
|
log "=== All max-throughput benchmarks completed ==="
|
||||||
|
log "Summary saved to: ${SUMMARY_FILE}"
|
||||||
|
log "Detailed results in: ${RESULT_DIR}"
|
||||||
|
|
||||||
|
${PYTHON} - <<PYEOF
|
||||||
|
import json
|
||||||
|
with open("${SUMMARY_FILE}", "r") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
print("\n============ Max Throughput Summary ============")
|
||||||
|
print(f"{'Name':<30} {'Req/s':>8} {'In tok/s':>10} {'Out tok/s':>10} {'Total tok/s':>12} {'TTFT':>8} {'TPOT':>8} {'E2E':>10}")
|
||||||
|
print("-" * 110)
|
||||||
|
for entry in data:
|
||||||
|
print(f"{entry['name']:<30} {entry['request_throughput']:>8.2f} {entry['input_token_throughput']:>10.2f} {entry['output_token_throughput']:>10.2f} {entry['total_token_throughput']:>12.2f} {entry['mean_ttft_ms']:>8.1f} {entry['mean_tpot_ms']:>8.1f} {entry['mean_e2e_latency_ms']:>10.1f}")
|
||||||
|
PYEOF
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
65
scripts/start_dsv4_dspark_8card.sh
Executable file
65
scripts/start_dsv4_dspark_8card.sh
Executable file
@ -0,0 +1,65 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p logs
|
||||||
|
|
||||||
|
VENV="/data/user1/yy/envs/vllm-dspark"
|
||||||
|
PYTHON="$VENV/bin/python"
|
||||||
|
VLLM="$VENV/bin/vllm"
|
||||||
|
export PATH="$VENV/bin:$PATH"
|
||||||
|
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||||
|
PORT=30004
|
||||||
|
TP=8
|
||||||
|
LOG="/data/user1/yy/logs/dsv4_flash_dspark_tp${TP}_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid"
|
||||||
|
|
||||||
|
export TMPDIR=/data/user1/yy/tmp
|
||||||
|
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
|
||||||
|
echo "=== Starting DeepSeek-V4-Flash-DSpark (TP=$TP) ==="
|
||||||
|
echo "Model: $MODEL"
|
||||||
|
echo "Port: $PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
nohup "$VLLM" serve "$MODEL" \
|
||||||
|
--trust-remote-code \
|
||||||
|
--tensor-parallel-size "$TP" \
|
||||||
|
--kv-cache-dtype fp8 \
|
||||||
|
--block-size 256 \
|
||||||
|
--max-model-len auto \
|
||||||
|
--max-num-seqs 256 \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--spec-method dspark \
|
||||||
|
--spec-model "$MODEL" \
|
||||||
|
--spec-tokens 5 \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--port "$PORT" \
|
||||||
|
> "$LOG" 2>&1 &
|
||||||
|
PID=$!
|
||||||
|
echo $PID > "$PID_FILE"
|
||||||
|
echo "PID: $PID"
|
||||||
|
echo "Waiting for health..."
|
||||||
|
|
||||||
|
for i in $(seq 1 240); do
|
||||||
|
if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Server is ready at http://127.0.0.1:$PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Server exited early"
|
||||||
|
tail -100 "$LOG"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting... ($i/240)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "ERROR: Server not healthy after 240 retries"
|
||||||
|
tail -100 "$LOG"
|
||||||
|
exit 1
|
||||||
63
scripts/start_dsv4_dspark_8card_bf16kv.sh
Executable file
63
scripts/start_dsv4_dspark_8card_bf16kv.sh
Executable file
@ -0,0 +1,63 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p logs
|
||||||
|
|
||||||
|
VENV="/data/user1/yy/envs/vllm-dspark"
|
||||||
|
VLLM="$VENV/bin/vllm"
|
||||||
|
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||||
|
PORT=30004
|
||||||
|
TP=8
|
||||||
|
LOG="/data/user1/yy/logs/dsv4_flash_dspark_tp${TP}_bf16kv_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid"
|
||||||
|
|
||||||
|
export TMPDIR=/data/user1/yy/tmp
|
||||||
|
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
|
||||||
|
echo "=== Starting DeepSeek-V4-Flash-DSpark (TP=$TP, kv=bf16) ==="
|
||||||
|
echo "Model: $MODEL"
|
||||||
|
echo "Port: $PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
nohup "$VLLM" serve "$MODEL" \
|
||||||
|
--trust-remote-code \
|
||||||
|
--tensor-parallel-size "$TP" \
|
||||||
|
--kv-cache-dtype bfloat16 \
|
||||||
|
--block-size 256 \
|
||||||
|
--max-model-len auto \
|
||||||
|
--max-num-seqs 256 \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--spec-method dspark \
|
||||||
|
--spec-model "$MODEL" \
|
||||||
|
--spec-tokens 5 \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--port "$PORT" \
|
||||||
|
> "$LOG" 2>&1 &
|
||||||
|
PID=$!
|
||||||
|
echo $PID > "$PID_FILE"
|
||||||
|
echo "PID: $PID"
|
||||||
|
echo "Waiting for health..."
|
||||||
|
|
||||||
|
for i in $(seq 1 240); do
|
||||||
|
if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Server is ready at http://127.0.0.1:$PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Server exited early"
|
||||||
|
tail -100 "$LOG"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting... ($i/240)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "ERROR: Server not healthy after 240 retries"
|
||||||
|
tail -100 "$LOG"
|
||||||
|
exit 1
|
||||||
66
scripts/start_dsv4_dspark_8card_flashinfer.sh
Executable file
66
scripts/start_dsv4_dspark_8card_flashinfer.sh
Executable file
@ -0,0 +1,66 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p logs
|
||||||
|
|
||||||
|
VENV="/data/user1/yy/envs/vllm-dspark"
|
||||||
|
VLLM="$VENV/bin/vllm"
|
||||||
|
export PATH="$VENV/bin:$PATH"
|
||||||
|
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||||
|
PORT=30004
|
||||||
|
TP=8
|
||||||
|
LOG="/data/user1/yy/logs/dsv4_flash_dspark_tp${TP}_flashinfer_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid"
|
||||||
|
|
||||||
|
export TMPDIR=/data/user1/yy/tmp
|
||||||
|
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
|
||||||
|
echo "=== Starting DeepSeek-V4-Flash-DSpark (TP=$TP, FlashInfer) ==="
|
||||||
|
echo "Model: $MODEL"
|
||||||
|
echo "Port: $PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
nohup "$VLLM" serve "$MODEL" \
|
||||||
|
--trust-remote-code \
|
||||||
|
--tensor-parallel-size "$TP" \
|
||||||
|
--kv-cache-dtype fp8 \
|
||||||
|
--block-size 256 \
|
||||||
|
--max-model-len auto \
|
||||||
|
--max-num-seqs 256 \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--attention-backend FLASHINFER_MLA_SPARSE_DSV4 \
|
||||||
|
--spec-method dspark \
|
||||||
|
--spec-model "$MODEL" \
|
||||||
|
--spec-tokens 5 \
|
||||||
|
--speculative-config '{"attention_backend": "FLASHINFER_MLA_SPARSE_DSV4"}' \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--port "$PORT" \
|
||||||
|
> "$LOG" 2>&1 &
|
||||||
|
PID=$!
|
||||||
|
echo $PID > "$PID_FILE"
|
||||||
|
echo "PID: $PID"
|
||||||
|
echo "Waiting for health..."
|
||||||
|
|
||||||
|
for i in $(seq 1 240); do
|
||||||
|
if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Server is ready at http://127.0.0.1:$PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Server exited early"
|
||||||
|
tail -100 "$LOG"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting... ($i/240)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "ERROR: Server not healthy after 240 retries"
|
||||||
|
tail -100 "$LOG"
|
||||||
|
exit 1
|
||||||
85
scripts/start_dsv4_dspark_8card_param.sh
Executable file
85
scripts/start_dsv4_dspark_8card_param.sh
Executable file
@ -0,0 +1,85 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p logs
|
||||||
|
|
||||||
|
VENV="/data/user1/yy/envs/vllm-dspark"
|
||||||
|
VLLM="$VENV/bin/vllm"
|
||||||
|
export PATH="$VENV/bin:$PATH"
|
||||||
|
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash-DSpark"
|
||||||
|
PORT="${PORT:-30004}"
|
||||||
|
TP="${TP:-8}"
|
||||||
|
SPEC_TOKENS="${SPEC_TOKENS:-5}"
|
||||||
|
KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}"
|
||||||
|
ATTENTION_BACKEND="${ATTENTION_BACKEND:-}"
|
||||||
|
SPECULATIVE_CONFIG="${SPECULATIVE_CONFIG:-}"
|
||||||
|
|
||||||
|
LOG_TAG="tp${TP}_${KV_CACHE_DTYPE}_st${SPEC_TOKENS}"
|
||||||
|
if [[ -n "$ATTENTION_BACKEND" ]]; then
|
||||||
|
LOG_TAG="${LOG_TAG}_${ATTENTION_BACKEND}"
|
||||||
|
fi
|
||||||
|
LOG="/data/user1/yy/logs/dsv4_flash_dspark_${LOG_TAG}_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid"
|
||||||
|
|
||||||
|
export TMPDIR=/data/user1/yy/tmp
|
||||||
|
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
|
||||||
|
echo "=== Starting DeepSeek-V4-Flash-DSpark (TP=$TP, spec-tokens=$SPEC_TOKENS, kv=$KV_CACHE_DTYPE) ==="
|
||||||
|
echo "Model: $MODEL"
|
||||||
|
echo "Port: $PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
|
||||||
|
cmd=(
|
||||||
|
"$VLLM" serve "$MODEL"
|
||||||
|
--trust-remote-code
|
||||||
|
--tensor-parallel-size "$TP"
|
||||||
|
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||||
|
--block-size 256
|
||||||
|
--max-model-len auto
|
||||||
|
--max-num-seqs 256
|
||||||
|
--tokenizer-mode deepseek_v4
|
||||||
|
--reasoning-parser deepseek_v4
|
||||||
|
--spec-method dspark
|
||||||
|
--spec-model "$MODEL"
|
||||||
|
--spec-tokens "$SPEC_TOKENS"
|
||||||
|
--no-disable-hybrid-kv-cache-manager
|
||||||
|
--disable-uvicorn-access-log
|
||||||
|
--port "$PORT"
|
||||||
|
)
|
||||||
|
|
||||||
|
if [[ -n "$ATTENTION_BACKEND" ]]; then
|
||||||
|
cmd+=(--attention-backend "$ATTENTION_BACKEND")
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -n "$SPECULATIVE_CONFIG" ]]; then
|
||||||
|
cmd+=(--speculative-config "$SPECULATIVE_CONFIG")
|
||||||
|
fi
|
||||||
|
|
||||||
|
nohup "${cmd[@]}" > "$LOG" 2>&1 &
|
||||||
|
PID=$!
|
||||||
|
echo $PID > "$PID_FILE"
|
||||||
|
echo "PID: $PID"
|
||||||
|
echo "Waiting for health..."
|
||||||
|
|
||||||
|
for i in $(seq 1 240); do
|
||||||
|
if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Server is ready at http://127.0.0.1:$PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Server exited early"
|
||||||
|
tail -100 "$LOG"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting... ($i/240)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "ERROR: Server not healthy after 240 retries"
|
||||||
|
tail -100 "$LOG"
|
||||||
|
exit 1
|
||||||
138
scripts/start_pd_single_node.sh
Executable file
138
scripts/start_pd_single_node.sh
Executable file
@ -0,0 +1,138 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# sglang PD 分离单节点启动脚本
|
||||||
|
# 104 上前 4 张卡做 prefill，后 4 张卡做 decode
|
||||||
|
# 使用 NIXL/UCX 作为传输后端（Mooncake 无法安装）
|
||||||
|
|
||||||
|
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
|
||||||
|
BOOTSTRAP_PORT=30010
|
||||||
|
PREFILL_PORT=30001
|
||||||
|
DECODE_PORT=30002
|
||||||
|
LB_PORT=30000
|
||||||
|
HOST="0.0.0.0"
|
||||||
|
|
||||||
|
VENV="/data/user1/yy/envs/sglang"
|
||||||
|
PYTHON="$VENV/bin/python"
|
||||||
|
LOG_DIR="/data/user1/yy/pd_logs"
|
||||||
|
mkdir -p "$LOG_DIR"
|
||||||
|
|
||||||
|
# NIXL/UCX 配置：单节点内优先用 CUDA IPC / shared memory，禁用 IB
|
||||||
|
export SGLANG_DISAGGREGATION_NIXL_BACKEND="UCX"
|
||||||
|
export UCX_TLS="self,sm,cuda_copy,cuda_ipc,tcp"
|
||||||
|
export UCX_NET_DEVICES=""
|
||||||
|
|
||||||
|
echo "=== Starting sglang PD disaggregation (single node) ==="
|
||||||
|
echo "Model: $MODEL_PATH"
|
||||||
|
echo "Prefill GPUs: 0-3, port $PREFILL_PORT"
|
||||||
|
echo "Decode GPUs: 4-7, port $DECODE_PORT"
|
||||||
|
echo "Load balancer port: $LB_PORT"
|
||||||
|
echo "Logs: $LOG_DIR"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Clean up old logs
|
||||||
|
rm -f "$LOG_DIR"/*.log
|
||||||
|
|
||||||
|
# Common args for both prefill and decode
|
||||||
|
COMMON_ARGS=(
|
||||||
|
--trust-remote-code
|
||||||
|
--model-path "$MODEL_PATH"
|
||||||
|
--tp 4
|
||||||
|
--moe-runner-backend marlin
|
||||||
|
--disable-cuda-graph
|
||||||
|
--disaggregation-bootstrap-port "$BOOTSTRAP_PORT"
|
||||||
|
--disaggregation-transfer-backend nixl
|
||||||
|
--host "$HOST"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Start prefill server (GPUs 0-3)
|
||||||
|
echo "[1/3] Starting prefill server on GPUs 0-3..."
|
||||||
|
CUDA_VISIBLE_DEVICES=0,1,2,3 nohup "$PYTHON" -m sglang.launch_server \
|
||||||
|
"${COMMON_ARGS[@]}" \
|
||||||
|
--disaggregation-mode prefill \
|
||||||
|
--port "$PREFILL_PORT" \
|
||||||
|
> "$LOG_DIR/prefill.log" 2>&1 &
|
||||||
|
PREFILL_PID=$!
|
||||||
|
echo "Prefill PID: $PREFILL_PID"
|
||||||
|
|
||||||
|
# Wait for prefill health
|
||||||
|
for i in {1..120}; do
|
||||||
|
if curl -s "http://127.0.0.1:$PREFILL_PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Prefill server is ready"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PREFILL_PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Prefill server exited early"
|
||||||
|
tail -50 "$LOG_DIR/prefill.log"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting for prefill server... ($i/120)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
# Start decode server (GPUs 4-7)
|
||||||
|
echo "[2/3] Starting decode server on GPUs 4-7..."
|
||||||
|
CUDA_VISIBLE_DEVICES=0,1,2,3 nohup "$PYTHON" -m sglang.launch_server \
|
||||||
|
"${COMMON_ARGS[@]}" \
|
||||||
|
--disaggregation-mode decode \
|
||||||
|
--base-gpu-id 4 \
|
||||||
|
--port "$DECODE_PORT" \
|
||||||
|
> "$LOG_DIR/decode.log" 2>&1 &
|
||||||
|
DECODE_PID=$!
|
||||||
|
echo "Decode PID: $DECODE_PID"
|
||||||
|
|
||||||
|
# Wait for decode health
|
||||||
|
for i in {1..120}; do
|
||||||
|
if curl -s "http://127.0.0.1:$DECODE_PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Decode server is ready"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
if ! kill -0 $DECODE_PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Decode server exited early"
|
||||||
|
tail -50 "$LOG_DIR/decode.log"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting for decode server... ($i/120)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
# Start PD load balancer
|
||||||
|
echo "[3/3] Starting PD load balancer on port $LB_PORT..."
|
||||||
|
nohup "$PYTHON" -m sglang_router.launch_router \
|
||||||
|
--pd-disaggregation \
|
||||||
|
--mini-lb \
|
||||||
|
--prefill "http://127.0.0.1:$PREFILL_PORT" \
|
||||||
|
--decode "http://127.0.0.1:$DECODE_PORT" \
|
||||||
|
--host "$HOST" \
|
||||||
|
--port "$LB_PORT" \
|
||||||
|
> "$LOG_DIR/lb.log" 2>&1 &
|
||||||
|
LB_PID=$!
|
||||||
|
echo "LB PID: $LB_PID"
|
||||||
|
|
||||||
|
# Wait for LB health
|
||||||
|
for i in {1..60}; do
|
||||||
|
if curl -s "http://127.0.0.1:$LB_PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Load balancer is ready"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
if ! kill -0 $LB_PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Load balancer exited early"
|
||||||
|
tail -50 "$LOG_DIR/lb.log"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting for load balancer... ($i/60)"
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== PD disaggregation is ready ==="
|
||||||
|
echo "Load balancer URL: http://127.0.0.1:$LB_PORT"
|
||||||
|
echo "Prefill URL: http://127.0.0.1:$PREFILL_PORT"
|
||||||
|
echo "Decode URL: http://127.0.0.1:$DECODE_PORT"
|
||||||
|
echo ""
|
||||||
|
echo "Test command:"
|
||||||
|
echo " curl http://127.0.0.1:$LB_PORT/v1/completions \\"
|
||||||
|
echo " -H \"Content-Type: application/json\" \\"
|
||||||
|
echo " -d '{\"model\":\"DeepSeek-V4-Flash\",\"prompt\":\"Hello\",\"max_tokens\":32}'"
|
||||||
|
echo ""
|
||||||
|
echo "To stop: kill $PREFILL_PID $DECODE_PID $LB_PID"
|
||||||
27
scripts/start_sglang_dsv4_8card.sh
Executable file
27
scripts/start_sglang_dsv4_8card.sh
Executable file
@ -0,0 +1,27 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Start sglang 8-card DeepSeek-V4-Flash service
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p logs
|
||||||
|
|
||||||
|
export PYTHONUNBUFFERED=1
|
||||||
|
export SGLANG_LOG_LEVEL=info
|
||||||
|
export PATH="/data/user1/yy/envs/sglang/bin:$PATH"
|
||||||
|
|
||||||
|
LOG_FILE="logs/sglang_8card_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
|
||||||
|
echo "Starting sglang 8-card DSV4 service, logging to ${LOG_FILE}"
|
||||||
|
|
||||||
|
exec /data/user1/yy/envs/sglang/bin/sglang serve \
|
||||||
|
--trust-remote-code \
|
||||||
|
--model-path /data/models/DeepSeek-V4-Flash \
|
||||||
|
--tp 8 \
|
||||||
|
--moe-runner-backend marlin \
|
||||||
|
--speculative-algorithm EAGLE \
|
||||||
|
--speculative-num-steps 3 \
|
||||||
|
--speculative-eagle-topk 1 \
|
||||||
|
--speculative-num-draft-tokens 4 \
|
||||||
|
--host 0.0.0.0 \
|
||||||
|
--port 30000 \
|
||||||
|
2>&1 | tee "${LOG_FILE}"
|
||||||
65
scripts/start_vllm_eagle_dsv4_8card.sh
Normal file
65
scripts/start_vllm_eagle_dsv4_8card.sh
Normal file
@ -0,0 +1,65 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p logs
|
||||||
|
|
||||||
|
VENV="/data/user1/yy/envs/vllm-main-latest"
|
||||||
|
PYTHON="$VENV/bin/python"
|
||||||
|
VLLM="$VENV/bin/vllm"
|
||||||
|
export PATH="$VENV/bin:$PATH"
|
||||||
|
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash"
|
||||||
|
PORT=30005
|
||||||
|
TP=8
|
||||||
|
LOG="/data/user1/yy/logs/vllm_eagle_dsv4_tp${TP}_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
PID_FILE="/data/user1/yy/vllm_eagle_dsv4.pid"
|
||||||
|
|
||||||
|
export TMPDIR=/data/user1/yy/tmp
|
||||||
|
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
|
||||||
|
echo "=== Starting DeepSeek-V4-Flash vLLM EAGLE (TP=$TP) ==="
|
||||||
|
echo "Model: $MODEL"
|
||||||
|
echo "Port: $PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
nohup "$VLLM" serve "$MODEL" \
|
||||||
|
--trust-remote-code \
|
||||||
|
--tensor-parallel-size "$TP" \
|
||||||
|
--kv-cache-dtype fp8 \
|
||||||
|
--block-size 256 \
|
||||||
|
--max-model-len auto \
|
||||||
|
--max-num-seqs 256 \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--spec-method mtp \
|
||||||
|
--spec-model "$MODEL" \
|
||||||
|
--spec-tokens 4 \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--port "$PORT" \
|
||||||
|
> "$LOG" 2>&1 &
|
||||||
|
PID=$!
|
||||||
|
echo $PID > "$PID_FILE"
|
||||||
|
echo "PID: $PID"
|
||||||
|
echo "Waiting for health..."
|
||||||
|
|
||||||
|
for i in $(seq 1 240); do
|
||||||
|
if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Server is ready at http://127.0.0.1:$PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Server exited early"
|
||||||
|
tail -200 "$LOG"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting... ($i/240)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "ERROR: Server not healthy after 240 retries"
|
||||||
|
tail -200 "$LOG"
|
||||||
|
exit 1
|
||||||
66
scripts/start_vllm_mtp_dsv4_8card.sh
Normal file
66
scripts/start_vllm_mtp_dsv4_8card.sh
Normal file
@ -0,0 +1,66 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd /data/user1/yy
|
||||||
|
mkdir -p logs
|
||||||
|
|
||||||
|
VENV="/data/user1/yy/envs/vllm-dspark"
|
||||||
|
export PATH="$VENV/bin:$PATH"
|
||||||
|
PYTHON="$VENV/bin/python"
|
||||||
|
VLLM="$VENV/bin/vllm"
|
||||||
|
|
||||||
|
MODEL="/data/models/DeepSeek-V4-Flash"
|
||||||
|
PORT=30005
|
||||||
|
TP=8
|
||||||
|
LOG="/data/user1/yy/logs/vllm_mtp_dsv4_tp${TP}_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
PID_FILE="/data/user1/yy/vllm_mtp_dsv4.pid"
|
||||||
|
|
||||||
|
export TMPDIR=/data/user1/yy/tmp
|
||||||
|
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
|
||||||
|
echo "=== Starting DeepSeek-V4-Flash vLLM MTP (TP=$TP) ==="
|
||||||
|
echo "Model: $MODEL"
|
||||||
|
echo "Port: $PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
nohup "$VLLM" serve "$MODEL" \
|
||||||
|
--trust-remote-code \
|
||||||
|
--tensor-parallel-size "$TP" \
|
||||||
|
--kv-cache-dtype fp8 \
|
||||||
|
--block-size 256 \
|
||||||
|
--max-model-len auto \
|
||||||
|
--max-num-seqs 128 \
|
||||||
|
--gpu-memory-utilization 0.85 \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--spec-method mtp \
|
||||||
|
--spec-model "$MODEL" \
|
||||||
|
--spec-tokens 4 \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--port "$PORT" \
|
||||||
|
> "$LOG" 2>&1 &
|
||||||
|
PID=$!
|
||||||
|
echo $PID > "$PID_FILE"
|
||||||
|
echo "PID: $PID"
|
||||||
|
echo "Waiting for health..."
|
||||||
|
|
||||||
|
for i in $(seq 1 240); do
|
||||||
|
if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Server is ready at http://127.0.0.1:$PORT"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Server exited early"
|
||||||
|
tail -200 "$LOG"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting... ($i/240)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "ERROR: Server not healthy after 240 retries"
|
||||||
|
tail -200 "$LOG"
|
||||||
|
exit 1
|
||||||
156
scripts/start_vllm_pd_single_node.sh
Executable file
156
scripts/start_vllm_pd_single_node.sh
Executable file
@ -0,0 +1,156 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
# vLLM + Mooncake PD 分离单节点启动脚本
|
||||||
|
# 104 上前 4 张卡做 prefill，后 4 张卡做 decode
|
||||||
|
# 使用 TCP 传输（因为当前 IB/RDMA 未启用）
|
||||||
|
|
||||||
|
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
|
||||||
|
BOOTSTRAP_PORT=8998
|
||||||
|
PREFILL_PORT=8000
|
||||||
|
DECODE_PORT=8001
|
||||||
|
ROUTER_PORT=30000
|
||||||
|
HOST="0.0.0.0"
|
||||||
|
|
||||||
|
VENV="/data/user1/yy/envs/vllm"
|
||||||
|
PYTHON="$VENV/bin/python"
|
||||||
|
VLLM="$VENV/bin/vllm"
|
||||||
|
VLLM_ROUTER="$VENV/bin/vllm-router"
|
||||||
|
LOG_DIR="/data/user1/yy/vllm_pd_logs"
|
||||||
|
mkdir -p "$LOG_DIR"
|
||||||
|
|
||||||
|
echo "=== Starting vLLM PD disaggregation (single node) ==="
|
||||||
|
echo "Model: $MODEL_PATH"
|
||||||
|
echo "Prefill GPUs: 0-3, port $PREFILL_PORT"
|
||||||
|
echo "Decode GPUs: 4-7, port $DECODE_PORT"
|
||||||
|
echo "Router port: $ROUTER_PORT"
|
||||||
|
echo "Logs: $LOG_DIR"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Clean up old logs
|
||||||
|
rm -f "$LOG_DIR"/*.log
|
||||||
|
|
||||||
|
# 使用 TCP 协议（IB 没起来，不能用 rdma）
|
||||||
|
export VLLM_MOONCAKE_PROTOCOL=tcp
|
||||||
|
export VLLM_MOONCAKE_BOOTSTRAP_PORT=$BOOTSTRAP_PORT
|
||||||
|
|
||||||
|
# Common args
|
||||||
|
PREFILL_KV_CONFIG='{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"tcp"}}'
|
||||||
|
DECODE_KV_CONFIG='{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"tcp"}}'
|
||||||
|
|
||||||
|
# Start prefill server (GPUs 0-3)
|
||||||
|
echo "[1/3] Starting prefill server on GPUs 0-3..."
|
||||||
|
CUDA_VISIBLE_DEVICES=0,1,2,3 nohup "$VLLM" serve "$MODEL_PATH" \
|
||||||
|
--trust-remote-code \
|
||||||
|
--kv-cache-dtype fp8 \
|
||||||
|
--block-size 256 \
|
||||||
|
--port "$PREFILL_PORT" \
|
||||||
|
--data-parallel-size 4 \
|
||||||
|
--enable-expert-parallel \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--max-model-len auto \
|
||||||
|
--max-num-batched-tokens 16384 \
|
||||||
|
--max-num-seqs 8 \
|
||||||
|
--enforce-eager \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--kv-transfer-config "$PREFILL_KV_CONFIG" \
|
||||||
|
> "$LOG_DIR/prefill.log" 2>&1 &
|
||||||
|
PREFILL_PID=$!
|
||||||
|
echo "Prefill PID: $PREFILL_PID"
|
||||||
|
|
||||||
|
# Wait for prefill health
|
||||||
|
for i in {1..120}; do
|
||||||
|
if curl -s "http://127.0.0.1:$PREFILL_PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Prefill server is ready"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PREFILL_PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Prefill server exited early"
|
||||||
|
tail -80 "$LOG_DIR/prefill.log"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting for prefill server... ($i/120)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
# Start decode server (GPUs 4-7)
|
||||||
|
echo "[2/3] Starting decode server on GPUs 4-7..."
|
||||||
|
CUDA_VISIBLE_DEVICES=4,5,6,7 nohup "$VLLM" serve "$MODEL_PATH" \
|
||||||
|
--trust-remote-code \
|
||||||
|
--kv-cache-dtype fp8 \
|
||||||
|
--block-size 256 \
|
||||||
|
--port "$DECODE_PORT" \
|
||||||
|
--data-parallel-size 4 \
|
||||||
|
--enable-expert-parallel \
|
||||||
|
--tokenizer-mode deepseek_v4 \
|
||||||
|
--reasoning-parser deepseek_v4 \
|
||||||
|
--max-model-len auto \
|
||||||
|
--max-num-seqs 512 \
|
||||||
|
--max-num-batched-tokens 512 \
|
||||||
|
--compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \
|
||||||
|
--no-disable-hybrid-kv-cache-manager \
|
||||||
|
--disable-uvicorn-access-log \
|
||||||
|
--kv-transfer-config "$DECODE_KV_CONFIG" \
|
||||||
|
> "$LOG_DIR/decode.log" 2>&1 &
|
||||||
|
DECODE_PID=$!
|
||||||
|
echo "Decode PID: $DECODE_PID"
|
||||||
|
|
||||||
|
# Wait for decode health
|
||||||
|
for i in {1..120}; do
|
||||||
|
if curl -s "http://127.0.0.1:$DECODE_PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Decode server is ready"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
if ! kill -0 $DECODE_PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Decode server exited early"
|
||||||
|
tail -80 "$LOG_DIR/decode.log"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting for decode server... ($i/120)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
# Start vllm-router
|
||||||
|
echo "[3/3] Starting vllm-router on port $ROUTER_PORT..."
|
||||||
|
nohup "$VLLM_ROUTER" \
|
||||||
|
--policy round_robin \
|
||||||
|
--vllm-pd-disaggregation \
|
||||||
|
--prefill "http://127.0.0.1:$PREFILL_PORT" \
|
||||||
|
--decode "http://127.0.0.1:$DECODE_PORT" \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "$ROUTER_PORT" \
|
||||||
|
--intra-node-data-parallel-size 4 \
|
||||||
|
--kv-connector mooncake \
|
||||||
|
> "$LOG_DIR/router.log" 2>&1 &
|
||||||
|
ROUTER_PID=$!
|
||||||
|
echo "Router PID: $ROUTER_PID"
|
||||||
|
|
||||||
|
# Wait for router health
|
||||||
|
for i in {1..60}; do
|
||||||
|
if curl -s "http://127.0.0.1:$ROUTER_PORT/health" > /dev/null 2>&1; then
|
||||||
|
echo "Router is ready"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
if ! kill -0 $ROUTER_PID 2>/dev/null; then
|
||||||
|
echo "ERROR: Router exited early"
|
||||||
|
tail -50 "$LOG_DIR/router.log"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting for router... ($i/60)"
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== vLLM PD disaggregation is ready ==="
|
||||||
|
echo "Router URL: http://127.0.0.1:$ROUTER_PORT"
|
||||||
|
echo "Prefill URL: http://127.0.0.1:$PREFILL_PORT"
|
||||||
|
echo "Decode URL: http://127.0.0.1:$DECODE_PORT"
|
||||||
|
echo ""
|
||||||
|
echo "Test command:"
|
||||||
|
echo " curl http://127.0.0.1:$ROUTER_PORT/v1/completions \\"
|
||||||
|
echo " -H \"Content-Type: application/json\" \\"
|
||||||
|
echo " -d '{\"model\":\"DeepSeek-V4-Flash\",\"prompt\":\"Hello\",\"max_tokens\":32}'"
|
||||||
|
echo ""
|
||||||
|
echo "To stop: kill $PREFILL_PID $DECODE_PID $ROUTER_PID"
|
||||||
84
scripts/test_block_sparse_attn.py
Normal file
84
scripts/test_block_sparse_attn.py
Normal file
@ -0,0 +1,84 @@
|
|||||||
|
import torch
|
||||||
|
import block_sparse_attn
|
||||||
|
from block_sparse_attn import block_sparse_attn_func, token_streaming_attn_func, block_streaming_attn_func
|
||||||
|
|
||||||
|
print("block_sparse_attn version:", block_sparse_attn.__version__)
|
||||||
|
print("Available funcs:", [k for k in dir(block_sparse_attn) if not k.startswith("_")])
|
||||||
|
|
||||||
|
device = "cuda:0"
|
||||||
|
batch_size = 1
|
||||||
|
nheads = 4
|
||||||
|
head_dim = 64
|
||||||
|
seqlen = 256
|
||||||
|
|
||||||
|
total_len = batch_size * seqlen
|
||||||
|
q = torch.randn(total_len, nheads, head_dim, device=device, dtype=torch.float16)
|
||||||
|
k = torch.randn(total_len, nheads, head_dim, device=device, dtype=torch.float16)
|
||||||
|
v = torch.randn(total_len, nheads, head_dim, device=device, dtype=torch.float16)
|
||||||
|
|
||||||
|
cu_seqlens_q = torch.arange(0, total_len + 1, seqlen, device=device, dtype=torch.int32)
|
||||||
|
cu_seqlens_k = cu_seqlens_q.clone()
|
||||||
|
|
||||||
|
# 1. dense attention (all heads mask_type=0)
|
||||||
|
head_mask_type_dense = torch.zeros(nheads, device=device, dtype=torch.int32)
|
||||||
|
streaming_info = torch.tensor([1, 3] * nheads, device=device, dtype=torch.int32)
|
||||||
|
out_dense = block_sparse_attn_func(
|
||||||
|
q, k, v,
|
||||||
|
cu_seqlens_q, cu_seqlens_k,
|
||||||
|
head_mask_type_dense,
|
||||||
|
streaming_info,
|
||||||
|
None,
|
||||||
|
seqlen, seqlen,
|
||||||
|
0.0,
|
||||||
|
is_causal=False,
|
||||||
|
)
|
||||||
|
print("dense output shape:", out_dense.shape)
|
||||||
|
assert out_dense.shape == q.shape
|
||||||
|
|
||||||
|
# 2. block-sparse attention (2 dense heads + 2 blocksparse heads)
|
||||||
|
num_blocksparse_heads = 2
|
||||||
|
nrow = seqlen // 128
|
||||||
|
ncol = seqlen // 128
|
||||||
|
base_blockmask = torch.zeros(batch_size, num_blocksparse_heads, nrow, ncol, device=device, dtype=torch.bool)
|
||||||
|
base_blockmask[:, :, :, :ncol//2] = True # attend to first half
|
||||||
|
|
||||||
|
head_mask_type_sparse = torch.tensor([0, 0, 1, 1], device=device, dtype=torch.int32)
|
||||||
|
out_sparse = block_sparse_attn_func(
|
||||||
|
q, k, v,
|
||||||
|
cu_seqlens_q, cu_seqlens_k,
|
||||||
|
head_mask_type_sparse,
|
||||||
|
streaming_info,
|
||||||
|
base_blockmask,
|
||||||
|
seqlen, seqlen,
|
||||||
|
0.0,
|
||||||
|
is_causal=False,
|
||||||
|
)
|
||||||
|
print("block-sparse output shape:", out_sparse.shape)
|
||||||
|
assert out_sparse.shape == q.shape
|
||||||
|
|
||||||
|
# 3. token-level streaming attention
|
||||||
|
head_mask_type_stream = torch.full((nheads,), -1, device=device, dtype=torch.int32)
|
||||||
|
out_stream = token_streaming_attn_func(
|
||||||
|
q, k, v,
|
||||||
|
cu_seqlens_q, cu_seqlens_k,
|
||||||
|
head_mask_type_stream,
|
||||||
|
streaming_info,
|
||||||
|
seqlen, seqlen,
|
||||||
|
)
|
||||||
|
print("token-streaming output shape:", out_stream.shape)
|
||||||
|
assert out_stream.shape == q.shape
|
||||||
|
|
||||||
|
# 4. block-level streaming attention
|
||||||
|
out_block_stream = block_streaming_attn_func(
|
||||||
|
q, k, v,
|
||||||
|
cu_seqlens_q, cu_seqlens_k,
|
||||||
|
head_mask_type_stream,
|
||||||
|
streaming_info,
|
||||||
|
seqlen, seqlen,
|
||||||
|
0.0,
|
||||||
|
is_causal=True,
|
||||||
|
)
|
||||||
|
print("block-streaming output shape:", out_block_stream.shape)
|
||||||
|
assert out_block_stream.shape == q.shape
|
||||||
|
|
||||||
|
print("All smoke tests passed!")
|
||||||
60
scripts/wait_and_profile_dspark.sh
Executable file
60
scripts/wait_and_profile_dspark.sh
Executable file
@ -0,0 +1,60 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Wait for GPUs to become idle (no other compute processes), then run DSpark profiles.
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
ROOT=/data/user1/yy
|
||||||
|
PROFILE_PID_FILE=/tmp/profile_dspark.pid
|
||||||
|
MONITOR_LOG=$ROOT/bench_results/dspark_profile_monitor.log
|
||||||
|
RESULT_BASE=$ROOT/bench_results/dspark_profile_$(date +%Y%m%d_%H%M%S)
|
||||||
|
|
||||||
|
mkdir -p "$RESULT_BASE" "$ROOT/tmp"
|
||||||
|
|
||||||
|
log() {
|
||||||
|
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$MONITOR_LOG"
|
||||||
|
}
|
||||||
|
|
||||||
|
is_gpu_idle() {
|
||||||
|
# Consider idle if no process other than ours uses > 1GB on any GPU.
|
||||||
|
# Returns 0 (true) if idle, 1 (false) if busy.
|
||||||
|
nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits | \
|
||||||
|
awk -F',' '
|
||||||
|
{
|
||||||
|
pid=$1; mem=$2;
|
||||||
|
# Exclude our own known PIDs (monitor script, shell, etc.)
|
||||||
|
if (mem > 1024) { busy++; }
|
||||||
|
}
|
||||||
|
END { exit (busy > 0 ? 1 : 0) }
|
||||||
|
'
|
||||||
|
}
|
||||||
|
|
||||||
|
log "Starting GPU idle monitor. Will check every 5 minutes."
|
||||||
|
log "Results will go to $RESULT_BASE"
|
||||||
|
|
||||||
|
wait_for_idle() {
|
||||||
|
while true; do
|
||||||
|
if is_gpu_idle; then
|
||||||
|
log "GPUs appear idle."
|
||||||
|
return 0
|
||||||
|
else
|
||||||
|
log "GPUs still busy. Sleeping 5 minutes."
|
||||||
|
sleep 300
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run profiles sequentially, checking GPU idle state before each run.
|
||||||
|
CONFIGS=("dspark-st3:64" "dspark-st5:64" "dspark-st3:1" "nospec:64")
|
||||||
|
|
||||||
|
for entry in "${CONFIGS[@]}"; do
|
||||||
|
IFS=':' read -r config concurrency <<< "$entry"
|
||||||
|
log "Waiting before profile: config=$config concurrency=$concurrency"
|
||||||
|
wait_for_idle
|
||||||
|
log "Running profile: config=$config concurrency=$concurrency"
|
||||||
|
"$ROOT/envs/vllm-dspark/bin/python" "$ROOT/scripts/profile_dspark.py" \
|
||||||
|
--config "$config" --concurrency "$concurrency" --tool nsys \
|
||||||
|
>> "$RESULT_BASE/profile_${config}_c${concurrency}.log" 2>&1
|
||||||
|
log "Finished profile: config=$config concurrency=$concurrency"
|
||||||
|
done
|
||||||
|
|
||||||
|
log "All profiles complete. Results in $RESULT_BASE"
|
||||||
Loading…
x
Reference in New Issue
Block a user