diff --git a/experiments/legacy_bench_results/README.md b/experiments/legacy_bench_results/README.md deleted file mode 100644 index 6a9698c..0000000 --- a/experiments/legacy_bench_results/README.md +++ /dev/null @@ -1,25 +0,0 @@ -# Legacy Bench Results Archive - -This directory contains historical benchmark results that were originally stored under `bench_results/` before the `experiments/` layout was introduced. - -All legacy results have been moved here; `bench_results/` itself no longer exists. - -## Contents - -| Directory | Original location | Description | -|---|---|---| -| `dspark_grid_20260707-132641/` | `bench_results/dspark_grid_20260707-132641/` | P1/P2/P3 DSpark grid benchmark | -| `dspark_st_comparison_20260707-150649/` | `bench_results/dspark_st_comparison_20260707-150649/` | `--spec-tokens 3` vs `5` comparison | -| `dsv4_backend_comparison_20260707/` | `bench_results/dsv4_backend_comparison_20260707/` | SGLang vs vLLM backend comparison raw outputs manifest | -| `dsv4_comparison_20260705_152221/` | `bench_results/dsv4_comparison_20260705_152221/` | Early vLLM-dspark configuration comparison | -| `dsv4_flash_dspark_misc/` | `bench_results/dsv4_flash_dspark_misc/` | Early DSV4 DSpark one-off tests | -| `eagle_grid/` | `bench_results/eagle_grid/` | SGLang EAGLE vs DSpark comparison | -| `pd_bench/` | `bench_results/pd_bench/` | PD disaggregation benchmark outputs | -| `sglang_8card_max_throughput_20260705_030839/` | `bench_results/sglang_8card_max_throughput_20260705_030839/` | SGLang 8-card max throughput scan | -| `sglang_8card_systematic_20260704_120819/` | `bench_results/sglang_8card_systematic_20260704_120819/` | Early SGLang systematic scan | -| `sglang_misc/` | `bench_results/sglang_misc/` | Miscellaneous SGLang outputs | -| `vllm_dspark_qwen3_20260705_121046/` | `bench_results/vllm_dspark_qwen3_20260705_121046/` | Qwen3 DSpark tests | -| `vllm_dspark_qwen3_20260705_121218/` | `bench_results/vllm_dspark_qwen3_20260705_121218/` | Qwen3 DSpark tests | -| `vllm_dspark_qwen3_20260705_121256/` | `bench_results/vllm_dspark_qwen3_20260705_121256/` | Qwen3 DSpark tests | - -> Note: Some internal file paths (e.g. `result_file` in `summary.json`, hardcoded paths in old scripts) still reference the original `bench_results/` locations. These are kept as-is for historical accuracy. diff --git a/experiments/legacy_bench_results/dspark_grid_20260707-132641/evaluation.md b/experiments/legacy_bench_results/dspark_grid_20260707-132641/evaluation.md deleted file mode 100644 index d0663c9..0000000 --- a/experiments/legacy_bench_results/dspark_grid_20260707-132641/evaluation.md +++ /dev/null @@ -1,167 +0,0 @@ -# vllm-dspark Benchmark 结果评估 - -> 对应结果：`/data/user1/yy/experiments/legacy_bench_results/dspark_grid_20260707-132641` -> 运行时间：2026-07-07 -> 硬件：8× NVIDIA H200 143GB -> 模型：`/data/models/DeepSeek-V4-Flash-DSpark` -> 服务配置：TP=8，FP8 KV cache，`--spec-method dspark --spec-tokens 5`，block-size=256，max-num-seqs=256 -> 压测工具：`sglang.bench_serving --backend vllm` - ---- - -## 1. 总体结论 - -本次 grid 测试覆盖了从轻量 chat 到超长上下文、从重 decode 到高并发压测的多种场景。整体上看： - -- **DSpark 在短输入、中高并发场景下表现优秀**，`chat_short` 在并发 64 时达到约 9580 tok/s，`stress_standard` 在并发 128 时达到约 17465 tok/s。 -- **低并发下 DSpark 的 draft 开销明显**，部分场景单并发延迟和吞吐都不如预期。 -- **超长上下文场景存在明显的吞吐拐点**，`long_rag`（32K 输入）在并发 4 达到峰值后，并发 8 吞吐腰斩。 -- **延迟长尾（P95/P99）普遍较重**，提示调度、内存或投机解码验证阶段存在抖动。 - ---- - -## 2. 异常点分析 - -### 2.1 低并发（c=1）下 DSpark 收益被开销吃掉 - -以 `chat_standard`（input=1000, output=256）为例： - -| 并发 | req/s | mean E2E(ms) | P99 E2E(ms) | mean TTFT(ms) | P99 TTFT(ms) | -|---|---:|---:|---:|---:|---:| -| 1 | 0.49 | 2040.65 | 10715.53 | 1412.48 | 8430.71 | -| 8 | 4.35 | 1819.35 | 15165.63 | 327.62 | 4404.92 | -| 16 | 8.32 | 1885.15 | 10491.31 | 443.50 | 8567.07 | -| 32 | 10.00 | 3156.73 | 12936.08 | 560.77 | 7129.66 | - -- **单并发吞吐仅 0.49 req/s**，远低于无投机基线可预期的水平。 -- **P99 TTFT 高达 8.4s**，说明单请求首次预填充极不稳定，可能是 draft 验证/编译缓存未命中或 warmup 不充分导致。 -- 并发提升到 8 后，TTFT 均值下降到 327ms，说明 DSpark 的 draft 计算需要 batch 才能摊薄。 - -**判断**：这是 DSpark 的典型特征——低并发下 draft 固定开销无法被摊薄，导致延迟尾和单请求吞吐都较差。 - -### 2.2 超长上下文出现吞吐断崖 - -`long_rag`（input=32000, output=512）： - -| 并发 | Total tok/s | mean TTFT(ms) | P99 TTFT(ms) | -|---|---:|---:|---:| -| 1 | 16104.98 | 549.72 | 1163.96 | -| 2 | 43215.62 | 192.57 | 409.32 | -| 4 | **70040.54** | 191.57 | 427.69 | -| 8 | 34141.70（↓51%） | 930.98 | 2742.48 | - -`long_context_probe`（input=16000, output=512）也有类似模式： - -| 并发 | Total tok/s | mean TTFT(ms) | -|---|---:|---:| -| 1 | 11051.09 | 316.59 | -| 2 | 26862.06 | 110.65 | -| 4 | 43042.77 | 127.38 | -| 8 | 29255.46（↓32%） | 453.93 | -| 16 | 43292.68 | 444.14 | - -**判断**： -- 32K/16K 长 prefill 在并发 4 时达到最佳，继续加并发反而下降，可能与 **KV cache 显存带宽瓶颈**、**prefill 阶段 scheduling 阻塞** 或 **hybrid KV cache manager 的换入换出** 有关。 -- `long_rag` 在 c=8 时 P99 TTFT 暴涨到 2.7s，进一步印证内存/调度压力。 - -### 2.3 延迟长尾普遍偏重 - -几乎所有场景的 P99 E2E 都是 mean E2E 的 2~4 倍： - -| 场景 | 并发 | mean E2E | P99 E2E | 倍数 | -|---|---:|---:|---:|---:| -| chat_standard | 32 | 3156.73 | 12936.08 | 4.1× | -| generation_standard | 32 | 2908.89 | 6938.03 | 2.4× | -| summarization | 32 | 5587.21 | 14541.75 | 2.6× | -| decode_heavy | 32 | 5562.57 | 12822.35 | 2.3× | -| stress_standard | 128 | 4417.43 | 14850.43 | 3.4× | - -**判断**：投机解码的 draft 验证失败会导致 fall back 到逐个 target 前向，造成明显的延迟毛刺；另外 vLLM v1 引擎的 scheduling 在高并发下也可能产生队列等待。 - -### 2.4 高并发下 TPOT/ITL 增长较快 - -`stress_standard`（input=1000, output=256）： - -| 并发 | mean TPOT | P95 TPOT | P99 TPOT | mean ITL | -|---|---:|---:|---:|---:| -| 96 | 29.46 | 67.30 | 123.42 | 113.64 | -| 128 | 35.20 | 76.85 | 141.17 | 136.57 | - -TPOT 从 29ms 升到 35ms，P99 超过 140ms，说明 decode 阶段在极限并发下已经接近饱和。 - ---- - -## 3. 可重点优化的方向 - -### 3.1 按并发动态调整 `--spec-tokens` - -参考历史对比数据（`dsv4_inference_comparison_report.md`）： - -| spec-tokens | 低并发（c=1） | 高并发（c=64） | -|---|---|---| -| 3 | 1.03 req/s, 254 tok/s | **14.53 req/s, 3507 tok/s** | -| 5 | **1.07 req/s, 257 tok/s** | 9.36 req/s, 2210 tok/s | -| 7 | 1.03 req/s, 251 tok/s | 9.13 req/s, 2218 tok/s | - -当前部署固定使用 `--spec-tokens 5`： -- 对低并发（c=1~16）相对友好； -- 对高并发（c≥64）不是最优，st=3 在高并发下接受率更高、验证开销更小。 - -**建议**： -- 高并发服务场景：改用 `--spec-tokens 3`。 -- 若负载混合，可考虑不同 endpoint/队列按并发分桶，或测试 st=3/st=5 在本 grid 下的完整表现。 - -### 3.2 长上下文场景优化 - -针对 `long_rag`（32K）和 `long_context_probe`（16K）的吞吐断崖： - -1. **KV cache dtype 对比测试**：当前是 FP8，可测试 BF16 是否改善长上下文稳定性和吞吐。 -2. **block-size 调整**：当前 256，可尝试 128 或 64 看是否减少内存碎片。 -3. **hybrid KV cache manager**：当前 `--no-disable-hybrid-kv-cache-manager`，可对比关闭后的表现。 -4. **限制长上下文并发**：若业务允许，对 32K 输入设置更低的 `max-num-seqs` 或独立队列，避免 c=8 后的内存带宽瓶颈。 -5. **attention backend**：可测试 `FLASHINFER_MLA_SPARSE_DSV4`（脚本 `start_dsv4_dspark_8card_flashinfer.sh`）在长上下文下的表现。 - -### 3.3 降低延迟长尾 - -1. **增加 warmup**：当前仅 10 条 warmup。DSpark 的 JIT 编译/算子 warmup 在首次运行时较重，建议增加到 50~100 条或先跑一轮 discard。 -2. **compilation cache 持久化**：确认 `TORCH_EXTENSIONS_DIR`、`VLLM_CACHE_ROOT` 等环境变量已设置并复用，避免每次重启重新编译 deep_gemm/tilelang。 -3. **调度参数**：尝试调整 `--max-num-batched-tokens`、`--max-num-seqs`、chunked prefill 相关参数，减少队列等待。 -4. **P99 TTFT 抖动**：高并发下 P99 TTFT 高通常与 prefill batching 有关，可尝试限制 prefill 阶段 batch 大小或启用 prompt chunked prefill。 - -### 3.4 低并发场景是否需要 DSpark - -对于 `chat_standard` c=1： -- 当前 0.49 req/s、P99 E2E 10.7s 的服务质量较差。 -- 若业务存在大量单用户/低并发请求，可考虑： - - 部署无投机基线 vLLM 专门服务低并发流量； - - 或设置最小 batch 阈值，累积少量请求后再用 DSpark 处理。 - -### 3.5 对比 baseline 与 SGLang - -本次测试只有 DSpark 单一配置，建议补充： -1. **vLLM-dspark 无投机基线**：用同样的 `sglang.bench_serving --backend vllm` 测一遍相同 grid，计算真实加速比。 -2. **SGLang EAGLE**：在相同 grid 下复测，验证 DSpark 在不同场景下的相对优势。 -3. **不同 `--spec-tokens` 的完整 grid**：st=3 和 st=5 各跑一遍，绘制并发-吞吐曲线。 - ---- - -## 4. 推荐下一步实验 - -| 优先级 | 实验 | 预期收益 | -|---|---|---| -| P0 | 高并发改用 `--spec-tokens 3` | 提升 c≥64 场景吞吐 20~50% | -| P0 | 长上下文（16K/32K）对比 BF16 KV cache | 改善吞吐断崖和 P99 TTFT | -| P1 | 增加 warmup / 预热缓存 | 降低 P99 TTFT 和首请求延迟 | -| P1 | 测试 `FLASHINFER_MLA_SPARSE_DSV4` backend | 可能提升长上下文吞吐 | -| P2 | 跑无投机基线 grid | 获得真实加速比 | -| P2 | 对比 st=3/st=5 完整曲线 | 找到最佳投机长度配置 | - ---- - -## 5. 结论 - -当前 DSpark 部署在 **短输入、中高并发** 场景下已经展现出明显优势，但在 **低并发** 和 **超长上下文** 场景下存在明显短板。最优先的优化是： - -1. 高并发服务改用 `--spec-tokens 3`； -2. 针对 16K/32K 长上下文做 KV cache dtype 和 attention backend 的调优； -3. 补充 baseline 测试，量化 DSpark 的真实收益。 diff --git a/experiments/legacy_bench_results/dspark_grid_20260707-132641/report.md b/experiments/legacy_bench_results/dspark_grid_20260707-132641/report.md deleted file mode 100644 index 6b0ce76..0000000 --- a/experiments/legacy_bench_results/dspark_grid_20260707-132641/report.md +++ /dev/null @@ -1,120 +0,0 @@ -# vllm-dspark Benchmark Grid Report - -- Result root: `/data/user1/yy/experiments/legacy_bench_results/dspark_grid_20260707-132641` -- Model: `/data/models/DeepSeek-V4-Flash-DSpark` -- Backend: vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark, spec-tokens=5) -- Benchmark client: `sglang.bench_serving --backend vllm` - -## p1_quick - -### chat_standard (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 1 | 65.33 | 32 | 0.49 | 195.58 | 71.28 | 266.86 | 2040.65 | 8241.20 | 10715.53 | 1412.48 | 6903.30 | 8430.71 | 6.41 | 32.72 | 51.29 | 16.61 | -| 8 | 29.42 | 128 | 4.35 | 2130.85 | 569.52 | 2700.37 | 1819.35 | 8891.79 | 15165.63 | 327.62 | 379.39 | 4404.92 | 11.58 | 51.06 | 82.44 | 51.00 | -| 16 | 30.76 | 256 | 8.32 | 4144.89 | 1144.74 | 5289.63 | 1885.15 | 8902.10 | 10491.31 | 443.50 | 2779.49 | 8567.07 | 12.10 | 23.11 | 83.70 | 46.18 | -| 32 | 51.18 | 512 | 10.00 | 5017.88 | 1331.65 | 6349.53 | 3156.73 | 10438.92 | 12936.08 | 560.77 | 5750.71 | 7129.66 | 22.09 | 66.23 | 171.57 | 85.81 | - -### generation_standard (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 1 | 36.24 | 32 | 0.88 | 352.60 | 479.25 | 831.85 | 1131.63 | 1979.98 | 2576.42 | 66.57 | 141.80 | 177.56 | 1.94 | 2.85 | 2.88 | 8.50 | -| 8 | 27.43 | 128 | 4.67 | 2285.21 | 2362.35 | 4647.56 | 1649.16 | 3162.33 | 3844.70 | 80.88 | 157.80 | 160.50 | 3.23 | 5.31 | 7.09 | 14.77 | -| 16 | 35.39 | 256 | 7.23 | 3602.52 | 3654.79 | 7257.31 | 2133.90 | 4187.67 | 5500.20 | 88.78 | 169.39 | 199.78 | 4.37 | 6.97 | 11.81 | 19.28 | -| 32 | 48.16 | 512 | 10.63 | 5333.41 | 5371.85 | 10705.25 | 2908.89 | 5849.64 | 6938.03 | 112.84 | 211.64 | 405.78 | 5.84 | 9.30 | 12.47 | 25.89 | - -### summarization (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 1 | 33.63 | 32 | 0.95 | 4247.40 | 516.49 | 4763.90 | 1049.88 | 1783.33 | 1848.92 | 178.56 | 292.66 | 314.18 | 1.60 | 1.78 | 2.41 | 8.62 | -| 4 | 10.99 | 32 | 2.91 | 13000.06 | 1580.84 | 14580.90 | 1301.15 | 2238.30 | 2324.70 | 101.68 | 187.78 | 188.57 | 2.18 | 2.47 | 3.49 | 11.86 | -| 8 | 33.60 | 128 | 3.81 | 15369.23 | 1908.96 | 17278.19 | 2072.22 | 3978.11 | 4531.26 | 214.01 | 502.25 | 630.15 | 3.70 | 5.52 | 7.98 | 19.60 | -| 16 | 46.93 | 256 | 5.45 | 22698.72 | 2761.38 | 25460.10 | 2840.57 | 6243.39 | 7179.11 | 235.75 | 552.81 | 740.45 | 5.45 | 9.62 | 12.75 | 27.77 | - -## p2_core - -### chat_short (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 1 | 11.39 | 32 | 2.81 | 717.30 | 408.87 | 1126.18 | 355.00 | 685.00 | 815.55 | 48.57 | 105.64 | 105.97 | 2.07 | 3.10 | 3.69 | 8.51 | -| 8 | 10.62 | 128 | 12.06 | 3320.19 | 1591.90 | 4912.09 | 647.05 | 1401.33 | 1648.15 | 78.91 | 151.78 | 155.67 | 4.59 | 7.20 | 12.54 | 17.85 | -| 16 | 14.89 | 256 | 17.19 | 4533.71 | 2394.66 | 6928.36 | 902.72 | 1879.32 | 2211.23 | 95.95 | 165.09 | 217.14 | 6.19 | 10.26 | 15.12 | 23.65 | -| 32 | 24.90 | 512 | 20.56 | 5458.26 | 2723.85 | 8182.10 | 1508.89 | 3076.09 | 3881.52 | 161.04 | 292.00 | 383.41 | 10.80 | 18.19 | 25.45 | 41.86 | -| 64 | 21.27 | 512 | 24.07 | 6391.14 | 3189.39 | 9580.52 | 2547.81 | 5606.07 | 6543.97 | 268.68 | 543.21 | 588.63 | 18.24 | 30.39 | 39.57 | 70.29 | - -### chat_standard (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 64 | 19.87 | 512 | 25.76 | 12922.80 | 3429.47 | 16352.28 | 2380.94 | 4838.43 | 6315.98 | 291.17 | 794.65 | 947.26 | 16.87 | 30.96 | 42.55 | 68.46 | - -### generation_standard (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 64 | 43.59 | 512 | 11.75 | 5892.47 | 5934.93 | 11827.40 | 5136.14 | 10244.70 | 12575.20 | 209.70 | 536.96 | 606.58 | 10.33 | 17.41 | 21.20 | 45.47 | - -### long_context_probe (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 1 | 23.79 | 32 | 1.34 | 10684.78 | 366.30 | 11051.09 | 742.62 | 1237.28 | 1418.82 | 316.59 | 604.23 | 617.85 | 1.54 | 1.72 | 1.85 | 8.53 | -| 2 | 9.79 | 32 | 3.27 | 25971.67 | 890.38 | 26862.06 | 604.82 | 1055.84 | 1116.41 | 110.65 | 194.62 | 206.64 | 1.77 | 2.04 | 2.13 | 9.90 | -| 4 | 6.11 | 32 | 5.24 | 41616.05 | 1426.72 | 43042.77 | 741.94 | 1262.13 | 1447.68 | 127.38 | 222.70 | 235.35 | 2.26 | 2.69 | 3.69 | 12.39 | -| 8 | 36.07 | 128 | 3.55 | 28357.82 | 897.65 | 29255.46 | 2232.38 | 4946.97 | 5371.22 | 453.93 | 1089.99 | 1182.47 | 7.06 | 13.03 | 20.03 | 38.51 | -| 16 | 49.08 | 256 | 5.22 | 41910.25 | 1382.43 | 43292.68 | 3028.26 | 7651.28 | 9531.10 | 444.14 | 1212.92 | 1368.04 | 10.00 | 20.71 | 30.74 | 53.07 | - -### rag_medium (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 1 | 18.21 | 32 | 1.76 | 3568.86 | 494.63 | 4063.49 | 568.24 | 966.60 | 1201.99 | 110.94 | 151.11 | 154.78 | 1.63 | 1.95 | 2.50 | 8.60 | -| 4 | 6.31 | 32 | 5.07 | 10300.85 | 1427.65 | 11728.51 | 751.19 | 1259.07 | 1593.86 | 92.09 | 161.03 | 187.47 | 2.34 | 3.25 | 4.02 | 12.33 | -| 8 | 19.13 | 128 | 6.69 | 14147.80 | 1692.32 | 15840.12 | 1166.59 | 2199.67 | 2982.23 | 148.09 | 253.10 | 350.67 | 4.30 | 6.78 | 9.92 | 21.06 | -| 16 | 25.89 | 256 | 9.89 | 20561.53 | 2622.36 | 23183.89 | 1587.09 | 3316.74 | 4019.02 | 164.29 | 303.79 | 389.34 | 5.50 | 8.80 | 14.38 | 27.81 | -| 32 | 39.47 | 512 | 12.97 | 26523.95 | 3335.30 | 29859.24 | 2415.10 | 5069.76 | 6195.14 | 227.81 | 444.76 | 510.18 | 8.64 | 13.54 | 20.29 | 44.49 | - -### summarization (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 32 | 91.68 | 512 | 5.58 | 22445.97 | 2821.71 | 25267.67 | 5587.21 | 11177.97 | 12257.18 | 430.38 | 967.32 | 2238.28 | 10.56 | 16.13 | 23.53 | 54.40 | - -## p3_extension - -### decode_heavy (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 1 | 56.16 | 32 | 0.57 | 145.47 | 546.26 | 691.72 | 1754.17 | 2994.25 | 3175.70 | 52.30 | 107.66 | 108.83 | 1.88 | 2.68 | 3.14 | 8.53 | -| 8 | 54.33 | 128 | 2.36 | 648.72 | 2526.47 | 3175.19 | 3302.64 | 6408.14 | 7166.98 | 74.21 | 143.24 | 149.17 | 3.05 | 4.37 | 4.79 | 14.12 | -| 16 | 65.74 | 256 | 3.89 | 1026.90 | 4012.07 | 5038.96 | 3934.72 | 7216.03 | 9638.96 | 81.02 | 165.95 | 197.11 | 3.98 | 5.82 | 7.21 | 17.79 | -| 32 | 92.36 | 512 | 5.54 | 1471.70 | 5761.81 | 7233.51 | 5562.57 | 10498.47 | 12822.35 | 107.17 | 234.99 | 426.35 | 5.40 | 8.14 | 9.03 | 24.38 | - -### long_rag (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 1 | 30.84 | 32 | 1.04 | 15823.63 | 281.34 | 16104.98 | 962.70 | 1688.61 | 1788.29 | 549.72 | 1124.54 | 1163.96 | 1.49 | 1.72 | 1.82 | 8.52 | -| 2 | 11.49 | 32 | 2.78 | 42460.67 | 754.95 | 43215.62 | 708.16 | 1202.62 | 1253.12 | 192.57 | 352.96 | 409.32 | 2.05 | 2.37 | 6.62 | 10.76 | -| 4 | 7.09 | 32 | 4.51 | 68816.98 | 1223.56 | 70040.54 | 854.49 | 1516.80 | 1562.42 | 191.57 | 426.45 | 427.69 | 2.62 | 3.20 | 7.33 | 13.69 | -| 8 | 60.48 | 128 | 2.12 | 33604.85 | 536.85 | 34141.70 | 3732.80 | 8420.06 | 9705.36 | 930.98 | 2125.14 | 2742.48 | 11.12 | 22.70 | 39.17 | 61.41 | - -### stress_generation (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 96 | 81.20 | 768 | 9.46 | 4842.45 | 4635.63 | 9478.08 | 9772.15 | 20945.93 | 25314.28 | 326.63 | 782.50 | 795.30 | 20.35 | 33.33 | 42.50 | 88.91 | -| 128 | 92.26 | 1024 | 11.10 | 5587.83 | 5514.57 | 11102.40 | 11012.89 | 22012.44 | 28276.20 | 360.94 | 812.12 | 1024.93 | 22.53 | 35.59 | 42.09 | 99.75 | - -### stress_standard (20260707-132641) - -| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) | -|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------| -| 96 | 30.50 | 768 | 25.18 | 12891.05 | 3175.02 | 16066.07 | 3689.97 | 9383.09 | 11958.09 | 435.22 | 1225.11 | 2581.90 | 29.46 | 67.30 | 123.42 | 113.64 | -| 128 | 36.86 | 1024 | 27.78 | 13985.42 | 3479.38 | 17464.80 | 4417.43 | 11825.67 | 14850.43 | 456.55 | 1032.19 | 1922.87 | 35.20 | 76.85 | 141.17 | 136.57 | - diff --git a/experiments/legacy_bench_results/dspark_st_comparison_20260707-150649/comparison_report.md b/experiments/legacy_bench_results/dspark_st_comparison_20260707-150649/comparison_report.md deleted file mode 100644 index 0240608..0000000 --- a/experiments/legacy_bench_results/dspark_st_comparison_20260707-150649/comparison_report.md +++ /dev/null @@ -1,129 +0,0 @@ -# vllm-dspark `--spec-tokens` 对比报告 - -- 结果目录：`/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649` -- 模型：`/data/models/DeepSeek-V4-Flash-DSpark` -- 后端：vllm-dspark (TP=8, FP8 KV cache) -- 对比参数：`--spec-tokens 3` vs `--spec-tokens 5` -- Warmup：100 条 -- 压测客户端：`sglang.bench_serving --backend vllm` - ---- - -## 核心指标对比 - -| Scenario | Concurrency | Spec | Duration(s) | Req/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) | -|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| chat_short | 1 | 3 | 12.80 | 2.50 | 363.90 | 1002.30 | 398.97 | 757.11 | 948.62 | 43.97 | 68.89 | 2.42 | 3.76 | -| chat_short | 1 | 5 | 11.72 | 2.73 | 397.37 | 1094.51 | 365.29 | 743.89 | 782.98 | 64.42 | 174.36 | 2.06 | 3.40 | -| chat_short | 8 | 3 | 11.08 | 11.55 | 1525.06 | 4705.85 | 672.66 | 1269.82 | 1552.65 | 76.20 | 221.18 | 4.67 | 8.43 | -| chat_short | 8 | 5 | 11.78 | 10.87 | 1434.62 | 4426.78 | 718.40 | 1511.14 | 2008.00 | 94.97 | 220.85 | 4.95 | 9.93 | -| chat_short | 16 | 3 | 21.82 | 11.73 | 1634.03 | 4727.68 | 1333.99 | 2611.76 | 2921.26 | 151.65 | 337.53 | 8.85 | 17.66 | -| chat_short | 16 | 5 | 16.42 | 15.59 | 2171.58 | 6282.94 | 997.37 | 2147.86 | 2496.56 | 107.56 | 240.46 | 7.09 | 21.93 | -| chat_short | 32 | 3 | 24.62 | 20.79 | 2754.96 | 8275.55 | 1497.40 | 2944.48 | 3658.90 | 144.47 | 453.29 | 10.72 | 23.97 | -| chat_short | 32 | 5 | 26.32 | 19.45 | 2577.42 | 7742.25 | 1600.40 | 3403.00 | 4521.14 | 169.13 | 402.36 | 11.39 | 27.82 | -| chat_short | 64 | 3 | 21.66 | 23.63 | 3131.20 | 9405.73 | 2595.90 | 5230.98 | 6258.25 | 251.84 | 509.86 | 18.60 | 40.84 | -| chat_short | 64 | 5 | 21.26 | 24.09 | 3191.43 | 9586.66 | 2555.12 | 5387.59 | 6559.69 | 274.71 | 610.40 | 18.47 | 42.08 | -| chat_standard | 64 | 3 | 20.73 | 24.70 | 3288.58 | 15680.46 | 2483.46 | 4798.63 | 5693.88 | 271.41 | 947.15 | 17.93 | 46.05 | -| chat_standard | 64 | 5 | 20.91 | 24.49 | 3259.71 | 15542.80 | 2512.52 | 5394.43 | 6023.55 | 300.81 | 932.60 | 17.83 | 44.47 | -| decode_heavy | 1 | 3 | 68.78 | 0.47 | 446.03 | 564.81 | 2148.54 | 3712.03 | 3835.51 | 38.60 | 41.18 | 2.28 | 3.03 | -| decode_heavy | 1 | 5 | 57.70 | 0.55 | 531.76 | 673.37 | 1802.00 | 2957.04 | 3676.66 | 59.06 | 114.78 | 1.92 | 3.03 | -| decode_heavy | 32 | 3 | 96.14 | 5.33 | 5535.64 | 6949.57 | 5809.29 | 10830.76 | 12546.55 | 86.83 | 382.71 | 5.62 | 8.11 | -| decode_heavy | 32 | 5 | 92.75 | 5.52 | 5737.73 | 7203.28 | 5595.02 | 10412.11 | 12522.00 | 107.12 | 331.08 | 5.43 | 9.15 | -| generation_standard | 64 | 3 | 38.58 | 13.27 | 6705.88 | 13363.77 | 4542.22 | 8273.38 | 9355.10 | 190.34 | 813.58 | 9.00 | 15.36 | -| generation_standard | 64 | 5 | 43.39 | 11.80 | 5961.40 | 11880.13 | 5075.77 | 9863.55 | 11533.47 | 216.87 | 626.93 | 10.09 | 20.90 | -| long_context_probe | 1 | 3 | 26.10 | 1.23 | 333.92 | 10074.03 | 815.18 | 1382.34 | 1556.00 | 265.28 | 558.73 | 2.02 | 2.19 | -| long_context_probe | 1 | 5 | 22.22 | 1.44 | 392.31 | 11835.70 | 693.30 | 1172.84 | 1404.17 | 267.61 | 561.65 | 1.53 | 1.88 | -| long_context_probe | 4 | 3 | 7.34 | 4.36 | 1187.03 | 35811.63 | 888.88 | 1550.70 | 1611.29 | 125.51 | 282.82 | 2.77 | 3.23 | -| long_context_probe | 4 | 5 | 6.59 | 4.85 | 1322.32 | 39893.35 | 806.55 | 1390.87 | 1500.22 | 143.42 | 236.41 | 2.41 | 3.54 | -| long_context_probe | 8 | 3 | 38.61 | 3.31 | 838.49 | 27327.58 | 2387.05 | 5028.60 | 5667.97 | 406.10 | 1093.54 | 7.74 | 17.09 | -| long_context_probe | 8 | 5 | 35.94 | 3.56 | 900.89 | 29361.32 | 2223.29 | 4693.10 | 5345.72 | 425.03 | 1241.57 | 7.08 | 19.83 | -| rag_medium | 1 | 3 | 22.31 | 1.43 | 403.86 | 3317.84 | 696.09 | 1165.77 | 1315.00 | 105.51 | 157.60 | 2.10 | 2.81 | -| rag_medium | 1 | 5 | 18.16 | 1.76 | 496.09 | 4075.47 | 566.58 | 968.90 | 1134.15 | 110.34 | 156.35 | 1.64 | 2.34 | -| rag_medium | 8 | 3 | 20.57 | 6.22 | 1573.71 | 14730.01 | 1251.63 | 2472.54 | 2796.12 | 127.54 | 309.65 | 4.46 | 7.41 | -| rag_medium | 8 | 5 | 19.13 | 6.69 | 1692.13 | 15838.39 | 1175.26 | 2263.59 | 3106.42 | 154.24 | 352.19 | 4.14 | 8.36 | -| rag_medium | 32 | 3 | 45.02 | 11.37 | 2923.96 | 26176.72 | 2756.39 | 5579.00 | 6675.25 | 215.15 | 477.20 | 10.00 | 19.84 | -| rag_medium | 32 | 5 | 42.98 | 11.91 | 3062.92 | 27420.79 | 2626.21 | 5182.78 | 6873.72 | 253.09 | 522.80 | 9.17 | 18.55 | -| stress_standard | 64 | 3 | 21.35 | 23.99 | 3193.19 | 15225.66 | 2569.66 | 5006.16 | 5831.81 | 261.98 | 710.57 | 18.02 | 36.01 | -| stress_standard | 64 | 5 | 21.75 | 23.54 | 3133.74 | 14942.17 | 2621.52 | 5431.08 | 7161.16 | 305.11 | 734.92 | 18.42 | 41.01 | -| stress_standard | 96 | 3 | 20.36 | 37.72 | 4756.72 | 24069.73 | 2433.73 | 4827.07 | 6001.29 | 272.61 | 749.99 | 18.04 | 38.52 | -| stress_standard | 96 | 5 | 25.33 | 30.32 | 3823.91 | 19349.55 | 3032.19 | 6272.09 | 8124.32 | 339.22 | 793.18 | 23.06 | 51.46 | -| stress_standard | 128 | 3 | 21.58 | 47.45 | 5942.48 | 29828.36 | 2581.42 | 5162.16 | 6222.01 | 298.69 | 970.61 | 19.09 | 40.93 | -| stress_standard | 128 | 5 | 28.36 | 36.11 | 4522.53 | 22700.92 | 3405.74 | 7240.13 | 9176.77 | 382.02 | 1087.53 | 25.74 | 57.83 | - ---- - -## 吞吐 winner 统计 - -| Scenario | Concurrency | Winner (Total tok/s) | st=3 Total tok/s | st=5 Total tok/s | 提升 | -|---|---:|---:|---:|---:|---:| -| chat_short | 1 | st=5 | 1002.30 | 1094.51 | 9.20% | -| chat_short | 8 | st=3 | 4705.85 | 4426.78 | 6.30% | -| chat_short | 16 | st=5 | 4727.68 | 6282.94 | 32.90% | -| chat_short | 32 | st=3 | 8275.55 | 7742.25 | 6.89% | -| chat_short | 64 | st=5 | 9405.73 | 9586.66 | 1.92% | -| chat_standard | 64 | st=3 | 15680.46 | 15542.80 | 0.89% | -| decode_heavy | 1 | st=5 | 564.81 | 673.37 | 19.22% | -| decode_heavy | 32 | st=5 | 6949.57 | 7203.28 | 3.65% | -| generation_standard | 64 | st=3 | 13363.77 | 11880.13 | 12.49% | -| long_context_probe | 1 | st=5 | 10074.03 | 11835.70 | 17.49% | -| long_context_probe | 4 | st=5 | 35811.63 | 39893.35 | 11.40% | -| long_context_probe | 8 | st=5 | 27327.58 | 29361.32 | 7.44% | -| rag_medium | 1 | st=5 | 3317.84 | 4075.47 | 22.84% | -| rag_medium | 8 | st=5 | 14730.01 | 15838.39 | 7.52% | -| rag_medium | 32 | st=5 | 26176.72 | 27420.79 | 4.75% | -| stress_standard | 64 | st=3 | 15225.66 | 14942.17 | 1.90% | -| stress_standard | 96 | st=3 | 24069.73 | 19349.55 | 24.39% | -| stress_standard | 128 | st=3 | 29828.36 | 22700.92 | 31.40% | - ---- - -## 关键发现 - -### 1. 并发是决定性因素 - -- **低并发（c=1）和中低并发（c=8~32）**：`st=5` 在多数场景下更优，尤其是长上下文和重 decode 场景。 -- **高并发（c≥96）**：`st=3` 明显更优，`stress_standard` c=128 时 st=3 比 st=5 高 **31.4%**。 -- **中并发（c=64）**：两者基本持平，差异多在 2% 以内。 - -### 2. st=5 更适合长上下文和重 decode - -| 场景 | 最佳 spec | 原因 | -|---|---|---| -| long_context_probe (16K) | st=5 | 长 prefill 下 st=5 的接受长度更高 | -| rag_medium (4K) | st=5 | 中长输入下 st=5 延迟和吞吐都更优 | -| decode_heavy (2K output) | st=5 | 输出越长，st=5 的投机收益越大 | - -### 3. st=3 在极限并发下更稳 - -`stress_standard` 随并发增加，st=3 与 st=5 的差距拉大： - -| 并发 | st=3 Total tok/s | st=5 Total tok/s | 差距 | -|---|---:|---:|---:| -| 64 | 15225.66 | 14942.17 | 基本持平 | -| 96 | 24069.73 | 19349.55 | st=3 高 24.4% | -| 128 | 29828.36 | 22700.92 | st=3 高 31.4% | - -这说明 st=5 在极限并发下验证开销和 KV cache 压力显著增加，而 st=3 的验证 batch 更小、调度更稳定。 - -### 4. chat_short c=16 的异常 - -`chat_short c=16` 时 st=5 比 st=3 高 32.9%，是一个明显的 outlier。可能原因是该并发度下 st=5 的 draft 接受率和 batch 利用率恰好达到甜点。 - ---- - -## 优化建议 - -### 推荐配置 - -| 负载特征 | 推荐 `--spec-tokens` | 说明 | -|---|---|---| -| 低并发 / 在线交互（c ≤ 32） | **5** | 延迟低、单请求吞吐高 | -| 中并发（c ≈ 64） | 3 或 5 均可 | 差异很小 | -| 高并发 / 压测（c ≥ 96） | **3** | 吞吐更高、P99 更稳 | -| 长上下文 / RAG / 重 decode | **5** | 接受长度优势更明显 | - -### 下一步 - -1. 如果业务以高并发为主，将默认服务改为 `--spec-tokens 3`。 -2. 如果业务混合，可考虑按输入长度或并发度路由到不同服务实例。 -3. 本次 warmup 已从 10 提升到 100，P99 TTFT 相比首次 grid 测试有明显改善；建议保持 100 条 warmup。 diff --git a/experiments/legacy_bench_results/dsv4_backend_comparison_20260707/README.md b/experiments/legacy_bench_results/dsv4_backend_comparison_20260707/README.md deleted file mode 100644 index 0845e31..0000000 --- a/experiments/legacy_bench_results/dsv4_backend_comparison_20260707/README.md +++ /dev/null @@ -1,102 +0,0 @@ -# DSV4 Backend Comparison Raw Outputs (2026-07-07) - -## Location - -`/data/user1/yy/bench_results/dsv4_backend_comparison_20260707` - -## Contents - -- `raw_outputs/` — raw per-benchmark JSONL files produced by `sglang.bench_serving --output-file --output-details`. -- `README.md` — this file. - -## File Naming Convention - -Each file follows the pattern: - -``` -{backend}_0707_{concurrency}_{input_len}_{output_len}.jsonl -``` - -Where: - -- `{backend}`: `sglang` or `vllm` -- `{concurrency}`: max concurrent requests (`max-concurrency`) -- `{input_len}`: `--random-input-len` -- `{output_len}`: `--random-output-len` - -## Output File Inventory - -### SGLang backend - -| File | Concurrency | Input len | Output len | -|---|---|---|---| -| sglang_0707_32_512_256.jsonl | 32 | 512 | 256 | -| sglang_0707_128_512_256.jsonl | 128 | 512 | 256 | -| sglang_0707_256_512_256.jsonl | 256 | 512 | 256 | -| sglang_0707_512_512_256.jsonl | 512 | 512 | 256 | -| sglang_0707_512_512_2000.jsonl | 512 | 512 | 2000 | -| sglang_0707_32_512_2000.jsonl | 32 | 512 | 2000 | -| sglang_0707_32_4000_512.jsonl | 32 | 4000 | 512 | -| sglang_0707_128_4000_512.jsonl | 128 | 4000 | 512 | -| sglang_0707_512_4000_512.jsonl | 512 | 4000 | 512 | -| sglang_0707_32_16000_512.jsonl | 32 | 16000 | 512 | -| sglang_0707_128_16000_512.jsonl | 128 | 16000 | 512 | -| sglang_0707_512_1000_256.jsonl | 512 | 1000 | 256 | -| sglang_0707_768_1000_256.jsonl | 768 | 1000 | 256 | -| sglang_0707_1024_1000_256.jsonl | 1024 | 1000 | 256 | -| sglang_0707_512_1000_1000.jsonl | 512 | 1000 | 1000 | - -### vLLM backend - -| File | Concurrency | Input len | Output len | -|---|---|---|---| -| vllm_0707_32_512_256.jsonl | 32 | 512 | 256 | -| vllm_0707_128_512_256.jsonl | 128 | 512 | 256 | -| vllm_0707_256_512_256.jsonl | 256 | 512 | 256 | -| vllm_0707_128_512_2000.jsonl | 128 | 512 | 2000 | -| vllm_0707_256_512_2000.jsonl | 256 | 512 | 2000 | -| vllm_0707_512_512_2000.jsonl | 512 | 512 | 2000 | -| vllm_0707_32_1000_256.jsonl | 32 | 1000 | 256 | -| vllm_0707_128_1000_256.jsonl | 128 | 1000 | 256 | -| vllm_0707_256_1000_256.jsonl | 256 | 1000 | 256 | -| vllm_0707_512_1000_256.jsonl | 512 | 1000 | 256 | -| vllm_0707_768_1000_256.jsonl | 768 | 1000 | 256 | -| vllm_0707_1024_1000_256.jsonl | 1024 | 1000 | 256 | -| vllm_0707_32_1000_1000.jsonl | 32 | 1000 | 1000 | -| vllm_0707_128_1000_1000.jsonl | 128 | 1000 | 1000 | -| vllm_0707_256_1000_1000.jsonl | 256 | 1000 | 1000 | -| vllm_0707_512_1000_1000.jsonl | 512 | 1000 | 1000 | -| vllm_0707_1024_1000_1000.jsonl | 1024 | 1000 | 1000 | -| vllm_0707_32_4000_512.jsonl | 32 | 4000 | 512 | -| vllm_0707_128_4000_512.jsonl | 128 | 4000 | 512 | -| vllm_0707_256_4000_512.jsonl | 256 | 4000 | 512 | -| vllm_0707_512_4000_512.jsonl | 512 | 4000 | 512 | -| vllm_0707_32_8000_1000.jsonl | 32 | 8000 | 1000 | -| vllm_0707_128_8000_1000.jsonl | 128 | 8000 | 1000 | -| vllm_0707_256_8000_1000.jsonl | 256 | 8000 | 1000 | -| vllm_0707_512_8000_1000.jsonl | 512 | 8000 | 1000 | -| vllm_0707_32_16000_512.jsonl | 32 | 16000 | 512 | -| vllm_0707_128_16000_512.jsonl | 128 | 16000 | 512 | -| vllm_0707_256_16000_512.jsonl | 256 | 16000 | 512 | -| vllm_0707_32_32000_512.jsonl | 32 | 32000 | 512 | -| vllm_0707_128_32000_512.jsonl | 128 | 32000 | 512 | - -## Provenance - -These files are **legacy raw outputs** from an ad-hoc SGLang vs vLLM backend comparison sweep run on 2026-07-07. The exact generator command/script that produced them was not preserved in the repository. They were generated using `sglang.bench_serving` with `--output-file` and `--output-details`, against: - -- Model: `/data/models/DeepSeek-V4-Flash` -- SGLang port: `30000` -- vLLM port: `8000` (inferred from file content) -- Dataset: `random` - -For future reproducible comparisons, use the script: - -``` -scripts/benchmark_dsv4_backend_comparison.sh -``` - -## Related Reports - -- `/data/user1/yy/dsv4_inference_comparison_report.md` — earlier DSV4 inference comparison report. -- `/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/` — earlier comparison run outputs (`.json`/`.log`). diff --git a/experiments/legacy_bench_results/dsv4_comparison_20260705_152221/summary.json b/experiments/legacy_bench_results/dsv4_comparison_20260705_152221/summary.json deleted file mode 100644 index 98e8e23..0000000 --- a/experiments/legacy_bench_results/dsv4_comparison_20260705_152221/summary.json +++ /dev/null @@ -1,197 +0,0 @@ -[ - { - "service": "vllm-dspark-dspark-st5", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 5, - "concurrency": 1, - "request_throughput": 1.0665689780673602, - "output_throughput": 257.1817776813825, - "total_input_tokens": 47865, - "total_output_tokens": 48226, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st5_c1.json" - }, - { - "service": "vllm-dspark-dspark-st5", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 5, - "concurrency": 16, - "request_throughput": 7.702621472216438, - "output_throughput": 1862.3398195524906, - "total_input_tokens": 47865, - "total_output_tokens": 48356, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st5_c16.json" - }, - { - "service": "vllm-dspark-dspark-st5", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 5, - "concurrency": 64, - "request_throughput": 9.360404619968996, - "output_throughput": 2210.4595510056783, - "total_input_tokens": 47865, - "total_output_tokens": 47230, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st5_c64.json" - }, - { - "service": "vllm-dspark-nospec", - "engine": "vllm-dspark", - "spec_method": null, - "spec_tokens": null, - "concurrency": 1, - "request_throughput": 0.5918258062472803, - "output_throughput": 144.62151314361665, - "total_input_tokens": 47865, - "total_output_tokens": 48873, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-nospec_c1.json" - }, - { - "service": "vllm-dspark-nospec", - "engine": "vllm-dspark", - "spec_method": null, - "spec_tokens": null, - "concurrency": 16, - "request_throughput": 5.61467992292336, - "output_throughput": 1364.0022670753865, - "total_input_tokens": 47865, - "total_output_tokens": 48587, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-nospec_c16.json" - }, - { - "service": "vllm-dspark-nospec", - "engine": "vllm-dspark", - "spec_method": null, - "spec_tokens": null, - "concurrency": 64, - "request_throughput": 7.339983457334795, - "output_throughput": 1803.9844342264591, - "total_input_tokens": 47865, - "total_output_tokens": 49155, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-nospec_c64.json" - }, - { - "service": "vllm-dspark-dspark-st3", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 3, - "concurrency": 1, - "request_throughput": 1.033805786989743, - "output_throughput": 254.38859000456605, - "total_input_tokens": 47865, - "total_output_tokens": 49214, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st3_c1.json" - }, - { - "service": "vllm-dspark-dspark-st3", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 3, - "concurrency": 16, - "request_throughput": 8.237095278328889, - "output_throughput": 1992.1826785402332, - "total_input_tokens": 47865, - "total_output_tokens": 48371, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st3_c16.json" - }, - { - "service": "vllm-dspark-dspark-st3", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 3, - "concurrency": 64, - "request_throughput": 14.530514004704537, - "output_throughput": 3507.8113858757224, - "total_input_tokens": 47865, - "total_output_tokens": 48282, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st3_c64.json" - }, - { - "service": "vllm-dspark-dspark-st7", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 7, - "concurrency": 1, - "request_throughput": 1.0285012524448527, - "output_throughput": 251.58169136053544, - "total_input_tokens": 47865, - "total_output_tokens": 48922, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st7_c1.json" - }, - { - "service": "vllm-dspark-dspark-st7", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 7, - "concurrency": 16, - "request_throughput": 7.463416900080192, - "output_throughput": 1772.3002941775428, - "total_input_tokens": 47865, - "total_output_tokens": 47493, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st7_c16.json" - }, - { - "service": "vllm-dspark-dspark-st7", - "engine": "vllm-dspark", - "spec_method": "dspark", - "spec_tokens": 7, - "concurrency": 64, - "request_throughput": 9.128647309323574, - "output_throughput": 2218.5807988214547, - "total_input_tokens": 47865, - "total_output_tokens": 48607, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-dspark-dspark-st7_c64.json" - }, - { - "service": "vllm-main-nospec", - "engine": "vllm", - "spec_method": null, - "spec_tokens": null, - "concurrency": 1, - "request_throughput": 0.5806562443589017, - "output_throughput": 138.2571550630763, - "total_input_tokens": 47865, - "total_output_tokens": 47621, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-main-nospec_c1.json" - }, - { - "service": "vllm-main-nospec", - "engine": "vllm", - "spec_method": null, - "spec_tokens": null, - "concurrency": 16, - "request_throughput": 5.052303923305438, - "output_throughput": 1180.976042072646, - "total_input_tokens": 47865, - "total_output_tokens": 46750, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-main-nospec_c16.json" - }, - { - "service": "vllm-main-nospec", - "engine": "vllm", - "spec_method": null, - "spec_tokens": null, - "concurrency": 64, - "request_throughput": 6.851644652089382, - "output_throughput": 1641.8938662034388, - "total_input_tokens": 47865, - "total_output_tokens": 47927, - "duration_s": null, - "result_file": "/data/user1/yy/bench_results/dsv4_comparison_20260705_152221/vllm-main-nospec_c64.json" - } -] \ No newline at end of file diff --git a/experiments/legacy_bench_results/eagle_grid/dspark_vs_eagle_report.md b/experiments/legacy_bench_results/eagle_grid/dspark_vs_eagle_report.md deleted file mode 100644 index c4097e2..0000000 --- a/experiments/legacy_bench_results/eagle_grid/dspark_vs_eagle_report.md +++ /dev/null @@ -1,96 +0,0 @@ -# DSpark vs SGLang EAGLE 投机解码对比报告 - -- DSpark 结果：`/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649` -- EAGLE 结果：`/data/user1/yy/bench_results/eagle_grid/focused` -- DSpark 模型：`/data/models/DeepSeek-V4-Flash-DSpark` -- EAGLE 模型：`/data/models/DeepSeek-V4-Flash` -- DSpark 后端：vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark) -- EAGLE 后端：SGLang (TP=8, FP8 KV cache, speculative-algorithm=EAGLE, num_steps=3, topk=1, draft_tokens=4) -- 压测客户端：`sglang.bench_serving` -- Warmup：100 条 - -## 核心指标对比 - -| Scenario | Concurrency | Method | Total tok/s | Out tok/s | Req/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) | -|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| chat_short | 1 | DSpark(st=5) | 1094.51 | 397.37 | 2.73 | 365.29 | 743.89 | 782.98 | 64.42 | 174.36 | 2.06 | 3.40 | -| chat_short | 1 | EAGLE | 272.82 | 99.05 | 0.68 | 1467.67 | 3775.96 | 5175.09 | 223.86 | 405.20 | 7.47 | 19.74 | -| chat_short | 8 | DSpark(st=3) | 4705.85 | 1525.06 | 11.55 | 672.66 | 1269.82 | 1552.65 | 76.20 | 221.18 | 4.67 | 8.43 | -| chat_short | 8 | EAGLE | 1059.99 | 343.52 | 2.60 | 3043.16 | 7143.57 | 8345.08 | 245.91 | 528.86 | 21.41 | 36.02 | -| chat_short | 16 | DSpark(st=5) | 6282.94 | 2171.58 | 15.59 | 997.37 | 2147.86 | 2496.56 | 107.56 | 240.46 | 7.09 | 21.93 | -| chat_short | 16 | EAGLE | 1506.42 | 520.67 | 3.74 | 4240.11 | 8031.57 | 10192.48 | 236.19 | 438.56 | 29.12 | 54.02 | -| chat_short | 32 | DSpark(st=3) | 8275.55 | 2754.96 | 20.79 | 1497.40 | 2944.48 | 3658.90 | 144.47 | 453.29 | 10.72 | 23.97 | -| chat_short | 32 | EAGLE | 2461.23 | 819.35 | 6.18 | 4731.11 | 9926.81 | 11388.85 | 243.84 | 571.89 | 34.22 | 54.92 | -| chat_short | 64 | DSpark(st=5) | 9586.66 | 3191.43 | 24.09 | 2555.12 | 5387.59 | 6559.69 | 274.71 | 610.40 | 18.47 | 42.08 | -| chat_short | 64 | EAGLE | 4474.88 | 1489.70 | 11.24 | 5239.80 | 10296.02 | 11922.06 | 268.27 | 452.80 | 38.17 | 61.24 | -| chat_standard | 64 | DSpark(st=3) | 15680.46 | 3288.58 | 24.70 | 2483.46 | 4798.63 | 5693.88 | 271.41 | 947.15 | 17.93 | 46.05 | -| chat_standard | 64 | EAGLE | 6746.24 | 1414.85 | 10.63 | 5637.63 | 10693.92 | 12907.15 | 296.86 | 597.18 | 41.12 | 69.08 | -| decode_heavy | 1 | DSpark(st=5) | 673.37 | 531.76 | 0.55 | 1802.00 | 2957.04 | 3676.66 | 59.06 | 114.78 | 1.92 | 3.03 | -| decode_heavy | 1 | EAGLE | 261.35 | 206.39 | 0.22 | 4643.62 | 10452.02 | 18810.32 | 162.91 | 314.71 | 4.84 | 11.69 | -| decode_heavy | 32 | DSpark(st=5) | 7203.28 | 5737.73 | 5.52 | 5595.02 | 10412.11 | 12522.00 | 107.12 | 331.08 | 5.43 | 9.15 | -| decode_heavy | 32 | EAGLE | 2981.50 | 2374.90 | 2.28 | 13707.03 | 27159.67 | 30638.81 | 192.78 | 442.78 | 13.23 | 19.26 | -| generation_standard | 64 | DSpark(st=3) | 13363.77 | 6705.88 | 13.27 | 4542.22 | 8273.38 | 9355.10 | 190.34 | 813.58 | 9.00 | 15.36 | -| generation_standard | 64 | EAGLE | 5609.33 | 2814.73 | 5.57 | 11057.53 | 22577.31 | 26713.41 | 275.48 | 944.52 | 21.88 | 34.80 | -| long_context_probe | 1 | DSpark(st=5) | 11835.70 | 392.31 | 1.44 | 693.30 | 1172.84 | 1404.17 | 267.61 | 561.65 | 1.53 | 1.88 | -| long_context_probe | 1 | EAGLE | 5556.95 | 184.19 | 0.68 | 1477.16 | 2792.68 | 3836.17 | 238.26 | 618.74 | 4.43 | 8.62 | -| long_context_probe | 4 | DSpark(st=5) | 39893.35 | 1322.32 | 4.85 | 806.55 | 1390.87 | 1500.22 | 143.42 | 236.41 | 2.41 | 3.54 | -| long_context_probe | 4 | EAGLE | 6660.56 | 220.77 | 0.81 | 4777.51 | 9442.45 | 10071.75 | 433.60 | 1230.20 | 14.85 | 21.65 | -| long_context_probe | 8 | DSpark(st=5) | 29361.32 | 900.89 | 3.56 | 2223.29 | 4693.10 | 5345.72 | 425.03 | 1241.57 | 7.08 | 19.83 | -| long_context_probe | 8 | EAGLE | 9890.47 | 303.47 | 1.20 | 6232.13 | 16324.27 | 20329.21 | 606.76 | 2671.06 | 22.79 | 54.38 | -| rag_medium | 1 | DSpark(st=5) | 4075.47 | 496.09 | 1.76 | 566.58 | 968.90 | 1134.15 | 110.34 | 156.35 | 1.64 | 2.34 | -| rag_medium | 1 | EAGLE | 1326.02 | 161.41 | 0.57 | 1742.43 | 3804.17 | 3936.47 | 195.07 | 337.55 | 5.83 | 12.84 | -| rag_medium | 8 | DSpark(st=5) | 15838.39 | 1692.13 | 6.69 | 1175.26 | 2263.59 | 3106.42 | 154.24 | 352.19 | 4.14 | 8.36 | -| rag_medium | 8 | EAGLE | 3676.32 | 392.77 | 1.55 | 4908.89 | 11659.24 | 12566.81 | 254.65 | 678.44 | 18.53 | 35.36 | -| rag_medium | 32 | DSpark(st=5) | 27420.79 | 3062.92 | 11.91 | 2626.21 | 5182.78 | 6873.72 | 253.09 | 522.80 | 9.17 | 18.55 | -| rag_medium | 32 | EAGLE | 13268.07 | 1482.05 | 5.76 | 5444.98 | 10471.13 | 11990.37 | 337.62 | 2204.45 | 20.09 | 39.98 | -| stress_standard | 64 | DSpark(st=3) | 15225.66 | 3193.19 | 23.99 | 2569.66 | 5006.16 | 5831.81 | 261.98 | 710.57 | 18.02 | 36.01 | -| stress_standard | 64 | EAGLE | 5399.35 | 1132.38 | 8.51 | 6678.22 | 13086.21 | 14811.42 | 314.38 | 584.25 | 48.19 | 81.06 | -| stress_standard | 96 | DSpark(st=3) | 24069.73 | 4756.72 | 37.72 | 2433.73 | 4827.07 | 6001.29 | 272.61 | 749.99 | 18.04 | 38.52 | -| stress_standard | 96 | EAGLE | 8833.26 | 1745.65 | 13.84 | 6792.34 | 14000.22 | 16722.55 | 360.14 | 959.71 | 52.90 | 96.70 | -| stress_standard | 128 | DSpark(st=3) | 29828.36 | 5942.48 | 47.45 | 2581.42 | 5162.16 | 6222.01 | 298.69 | 970.61 | 19.09 | 40.93 | -| stress_standard | 128 | EAGLE | 13189.80 | 2627.70 | 20.98 | 5960.86 | 12256.06 | 14219.75 | 416.12 | 2153.71 | 47.20 | 81.35 | - -## 吞吐 Winner 统计（按 Total tok/s） - -| Scenario | Concurrency | Winner | DSpark Total tok/s | EAGLE Total tok/s | 提升 | -|---|---:|---:|---:|---:|---:| -| chat_short | 1 | DSpark(st=5) | 1094.51 | 272.82 | 301.18% | -| chat_short | 8 | DSpark(st=3) | 4705.85 | 1059.99 | 343.95% | -| chat_short | 16 | DSpark(st=5) | 6282.94 | 1506.42 | 317.08% | -| chat_short | 32 | DSpark(st=3) | 8275.55 | 2461.23 | 236.24% | -| chat_short | 64 | DSpark(st=5) | 9586.66 | 4474.88 | 114.23% | -| chat_standard | 64 | DSpark(st=3) | 15680.46 | 6746.24 | 132.43% | -| decode_heavy | 1 | DSpark(st=5) | 673.37 | 261.35 | 157.65% | -| decode_heavy | 32 | DSpark(st=5) | 7203.28 | 2981.50 | 141.60% | -| generation_standard | 64 | DSpark(st=3) | 13363.77 | 5609.33 | 138.24% | -| long_context_probe | 1 | DSpark(st=5) | 11835.70 | 5556.95 | 112.99% | -| long_context_probe | 4 | DSpark(st=5) | 39893.35 | 6660.56 | 498.95% | -| long_context_probe | 8 | DSpark(st=5) | 29361.32 | 9890.47 | 196.86% | -| rag_medium | 1 | DSpark(st=5) | 4075.47 | 1326.02 | 207.35% | -| rag_medium | 8 | DSpark(st=5) | 15838.39 | 3676.32 | 330.82% | -| rag_medium | 32 | DSpark(st=5) | 27420.79 | 13268.07 | 106.67% | -| stress_standard | 64 | DSpark(st=3) | 15225.66 | 5399.35 | 181.99% | -| stress_standard | 96 | DSpark(st=3) | 24069.73 | 8833.26 | 172.49% | -| stress_standard | 128 | DSpark(st=3) | 29828.36 | 13189.80 | 126.15% | - -## 关键发现 - -- 对比点数：18 -- EAGLE 胜出：0 个 -- DSpark 胜出：18 个 - -**结论：在相同负载下，vllm-dspark 的综合吞吐优于 SGLang EAGLE。** - -### 延迟与稳定性观察 - -- 平均 Mean E2E 比值（EAGLE / DSpark）：3.09 -- 平均 Mean TPOT 比值（EAGLE / DSpark）：3.21 -- 平均 Mean TTFT 比值（EAGLE / DSpark）：1.81 - -> 比值 < 1 表示 EAGLE 更快；> 1 表示 DSpark 更快。 - -## 优化建议 - -1. 如果以吞吐为首要目标，当前配置下 **vllm-dspark 更优**，建议继续使用 `--spec-tokens` 并根据并发选择 3（高并发）或 5（低并发/长上下文）。 -2. SGLang EAGLE 本次表现不佳，平均延迟和 TPOT 均显著高于 DSpark；如需进一步评估 EAGLE，可尝试调整 `--speculative-num-steps`、`--speculative-eagle-topk`、`--speculative-num-draft-tokens` 或更换 MOE runner backend，并确认是否已完成 `sglang.compile_deep_gemm` 预编译。 -3. 注意两套后端的实现差异（CUDA graph、KV cache 管理、调度器、draft 模型架构）会显著影响不同并发和输入长度下的表现，建议按实际业务负载做最终选型。 diff --git a/experiments/legacy_bench_results/eagle_grid/report.md b/experiments/legacy_bench_results/eagle_grid/report.md deleted file mode 100644 index c5ab5d7..0000000 --- a/experiments/legacy_bench_results/eagle_grid/report.md +++ /dev/null @@ -1,6 +0,0 @@ -# vllm-dspark Benchmark Grid Report - -- Result root: `/data/user1/yy/bench_results/eagle_grid` -- Model: `/data/models/DeepSeek-V4-Flash-DSpark` -- Backend: vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark, spec-tokens=5) -- Benchmark client: `sglang.bench_serving --backend vllm` diff --git a/experiments/legacy_bench_results/sglang_8card_max_throughput_20260705_030839/summary.json b/experiments/legacy_bench_results/sglang_8card_max_throughput_20260705_030839/summary.json deleted file mode 100644 index 001f511..0000000 --- a/experiments/legacy_bench_results/sglang_8card_max_throughput_20260705_030839/summary.json +++ /dev/null @@ -1,58 +0,0 @@ -[ - { - "name": "sharegpt_concurrency_128", - "args": "--max-concurrency 128", - "duration_s": 155.26, - "request_throughput": 12.88, - "input_token_throughput": 3710.5, - "output_token_throughput": 2690.48, - "total_token_throughput": 6400.98, - "mean_ttft_ms": 410.55, - "mean_tpot_ms": 48.45, - "mean_e2e_latency_ms": 9696.87, - "accept_length": 2.6, - "output_file": "bench_results/sglang_8card_max_throughput_20260705_030839/sharegpt_concurrency_128.json" - }, - { - "name": "sharegpt_concurrency_256", - "args": "--max-concurrency 256", - "duration_s": 90.65, - "request_throughput": 22.06, - "input_token_throughput": 6355.48, - "output_token_throughput": 4608.35, - "total_token_throughput": 10963.84, - "mean_ttft_ms": 509.73, - "mean_tpot_ms": 62.71, - "mean_e2e_latency_ms": 10869.76, - "accept_length": 2.6, - "output_file": "bench_results/sglang_8card_max_throughput_20260705_030839/sharegpt_concurrency_256.json" - }, - { - "name": "sharegpt_concurrency_512", - "args": "--max-concurrency 512", - "duration_s": 71.39, - "request_throughput": 28.02, - "input_token_throughput": 8070.04, - "output_token_throughput": 5851.58, - "total_token_throughput": 13921.61, - "mean_ttft_ms": 1343.73, - "mean_tpot_ms": 121.2, - "mean_e2e_latency_ms": 16176.14, - "accept_length": 2.6, - "output_file": "bench_results/sglang_8card_max_throughput_20260705_030839/sharegpt_concurrency_512.json" - }, - { - "name": "sharegpt_concurrency_1024", - "args": "--max-concurrency 1024", - "duration_s": 68.21, - "request_throughput": 29.32, - "input_token_throughput": 8445.53, - "output_token_throughput": 6123.84, - "total_token_throughput": 14569.37, - "mean_ttft_ms": 13265.46, - "mean_tpot_ms": 105.36, - "mean_e2e_latency_ms": 27566.99, - "accept_length": 2.6, - "output_file": "bench_results/sglang_8card_max_throughput_20260705_030839/sharegpt_concurrency_1024.json" - } -] \ No newline at end of file diff --git a/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png b/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png deleted file mode 100644 index a511adb..0000000 Binary files a/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png and /dev/null differ diff --git a/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/report.md b/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/report.md deleted file mode 100644 index ba05d12..0000000 --- a/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/report.md +++ /dev/null @@ -1,267 +0,0 @@ -# SGLang 8-Card DeepSeek-V4-Flash 系统 Benchmark 报告 - -生成时间：2026-07-05 02:10:06 - -## 测试环境 - -- **模型**：DeepSeek-V4-Flash (`/data/models/DeepSeek-V4-Flash`) -- **推理框架**：SGLang -- **并行策略**：TP=8，8× NVIDIA H200 -- **MoE backend**：marlin -- **投机采样**：EAGLE（`--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`） -- **服务端口**：30000 - -## 测试负载 - -- **数据集**：ShareGPT V4.3 unfiltered cleaned split -- **请求数**：每个测试点 2000 条真实对话请求 -- **测试维度**： - - 并发扫描：`--max-concurrency` = 1, 8, 16, 32, 64, 128 - - 请求速率扫描：`--request-rate` = 1, 2, 4, 8, 16 (Poisson-like) - -## Phase 1：并发扫描结果 - -| 并发 | 耗时(s) | req/s | input tok/s | output tok/s | total tok/s | Mean TTFT(ms) | Mean TPOT(ms) | Mean E2E(ms) | Accept Length | -|------|---------|-------|-------------|--------------|-------------|---------------|---------------|--------------|---------------| -| 1 | 1715.45 | 1.17 | 335.83 | 243.51 | 579.34 | 128.09 | 4.00 | 856.77 | 2.62 | -| 8 | 448.91 | 4.46 | 1283.33 | 930.54 | 2213.87 | 146.76 | 8.41 | 1790.65 | 2.61 | -| 16 | 331.79 | 6.03 | 1736.31 | 1259.00 | 2995.30 | 168.95 | 12.67 | 2642.10 | 2.61 | -| 32 | 267.72 | 7.47 | 2151.87 | 1560.32 | 3712.19 | 206.24 | 20.72 | 4249.71 | 2.61 | -| 64 | 207.57 | 9.64 | 2775.39 | 2012.43 | 4787.82 | 248.41 | 32.61 | 6557.39 | 2.61 | -| 128 | 139.41 | 14.35 | 4132.36 | 2996.37 | 7128.73 | 330.40 | 46.15 | 8596.12 | 2.61 | - -## Phase 2：请求速率扫描结果 - -| RPS | 耗时(s) | req/s | input tok/s | output tok/s | total tok/s | Mean TTFT(ms) | Mean TPOT(ms) | Mean E2E(ms) | Accept Length | -|-----|---------|-------|-------------|--------------|-------------|---------------|---------------|--------------|---------------| -| 1 | 2011.46 | 0.99 | 286.41 | 207.67 | 494.08 | 146.87 | 4.91 | 1007.62 | 2.61 | -| 2 | 1007.21 | 1.99 | 571.97 | 414.74 | 986.71 | 160.63 | 5.99 | 1205.24 | 2.61 | -| 4 | 505.49 | 3.96 | 1139.69 | 826.39 | 1966.08 | 174.67 | 9.02 | 1743.40 | 2.60 | -| 8 | 256.46 | 7.80 | 2246.34 | 1628.82 | 3875.15 | 216.65 | 25.35 | 4542.64 | 2.60 | -| 16 | 137.92 | 14.50 | 4176.94 | 3028.70 | 7205.64 | 377.28 | 87.16 | 15048.09 | 2.61 | - -## 关键发现 - -1. **吞吐随并发单调上升**：从 c=1 的 579 tok/s 提升到 c=128 的 7129 tok/s，说明 8×H200 在高压下仍能有效扩展。 -2. **c=32 是延迟与吞吐的拐点**：c=32 时 total tok/s 为 3712，E2E 延迟约 4.25s；c=64 时吞吐提升到 4788，但延迟增加到 6.56s。 -3. **c=128 达到最高吞吐**：total tok/s = 7129，output tok/s = 2996，但 E2E 延迟接近 8.6s，适合离线/批处理场景。 -4. **RPS=16 时系统过载**：实际 req/s 达到 14.5，TPOT 飙升到 87ms，E2E 延迟 15s，说明 RPS>14 已超出稳定运行区间。 -5. **RPS=8 是稳定高吞吐边界**：total tok/s = 3875，E2E 延迟 4.54s，TTFT 仅 217ms，体验与吞吐兼顾。 -6. **EAGLE 投机采样稳定生效**：Accept Length 稳定在 2.60–2.62，有效降低了 decode 步数。 - -## 场景建议 - -| 场景 | 推荐配置 | 理由 | -|------|----------|------| -| 低延迟在线 API | `--max-concurrency 8` | TTFT 147ms，TPOT 8.4ms，E2E 1.79s | -| 平衡型在线服务 | `--max-concurrency 16` 或 `--request-rate 4` | 吞吐约 3k tok/s，E2E 2.6–1.7s | -| 高吞吐在线服务 | `--max-concurrency 32` 或 `--request-rate 8` | 吞吐 3.7–3.9k tok/s，E2E 4.3–4.5s | -| 最大吞吐/离线批处理 | `--max-concurrency 128` | 吞吐 7.1k tok/s，E2E 8.6s | -| 避免使用 | `--request-rate 16` | 系统过载，E2E 15s，TPOT 87ms | - -## 原始数据 - -- 汇总 JSON：`bench_results/sglang_8card_systematic_20260704_120819/summary.json` -- 每个测试点的详细结果与日志：`bench_results/sglang_8card_systematic_20260704_120819/` - -## 附录 A：Benchmark 参数说明 - -本次所有测试均使用 `sglang.bench_serving` 工具，参数统一如下： - -```bash -envs/sglang/bin/python -m sglang.bench_serving \ - --backend sglang \ - --host 127.0.0.1 \ - --port 30000 \ - --dataset-name sharegpt \ - --dataset-path /data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json \ - --num-prompts 2000 \ - --model /data/models/DeepSeek-V4-Flash \ - --seed 42 -``` - -两个测试维度的区别仅在于最后一条参数： - -- **并发扫描**：`--max-concurrency {1,8,16,32,64,128}` - - 客户端同时保持 N 个未完成的请求，服务器队列按需堆积。 - - 适合模拟“系统能扛多少并发”的容量测试。 - -- **请求速率扫描**：`--request-rate {1,2,4,8,16}` - - 客户端按泊松分布以固定 RPS 发送请求。 - - 适合模拟真实线上按流量到达的场景。 - -### 指标定义 - -| 指标 | 含义 | -|------|------| -| **req/s** | 每秒完成的请求数 | -| **input tok/s** | 每秒处理的输入 token 数 | -| **output tok/s** | 每秒生成的输出 token 数 | -| **total tok/s** | input tok/s + output tok/s | -| **TTFT** | Time To First Token，首 token 延迟 | -| **TPOT** | Time Per Output Token，除首 token 外平均每个输出 token 的间隔 | -| **E2E Latency** | 端到端请求完成时间 | -| **Accept Length** | EAGLE 投机采样平均每次接受的 draft token 数 | - ---- - -## 附录 B：Token 成本计算 - -### 假设 - -- GPU 单价：**8.0 元 / 卡 / 小时** -- 本次部署使用 **8× NVIDIA H200** -- 整机每小时成本： - -``` -8 卡 × 8.0 元/卡/h = 64 元/h -``` - -### 公式 - -对于任意一个 benchmark 结果： - -``` -每小时处理 input tokens = input_tok/s × 3600 -每小时生成 output tokens = output_tok/s × 3600 - -每 1M input tokens 成本 = 64 / (input_tok/s × 3600 / 1,000,000) -每 1M output tokens 成本 = 64 / (output_tok/s × 3600 / 1,000,000) -``` - -### 各场景成本 - -| 场景 | input tok/s | output tok/s | 1M input cost | 1M output cost | -|------|-------------|--------------|---------------|----------------| -| c=1 | 335.83 | 243.51 | 52.94 元 | 73.01 元 | -| c=8 | 1,283.33 | 930.54 | 13.85 元 | 19.10 元 | -| c=16 | 1,736.31 | 1,259.00 | 10.24 元 | 14.12 元 | -| c=32 | 2,151.87 | 1,560.32 | 8.26 元 | 11.39 元 | -| c=64 | 2,775.39 | 2,012.43 | 6.41 元 | 8.83 元 | -| c=128 | 4,132.36 | 2,996.37 | 4.30 元 | 5.93 元 | -| rps=1 | 286.41 | 207.67 | 62.07 元 | 85.61 元 | -| rps=2 | 571.97 | 414.74 | 31.08 元 | 42.86 元 | -| rps=4 | 1,139.69 | 826.39 | 15.60 元 | 21.51 元 | -| rps=8 | 2,246.34 | 1,628.82 | 7.91 元 | 10.91 元 | -| rps=16 | 4,176.94 | 3,028.70 | 4.26 元 | 5.87 元 | - -### 对外定价建议 - -裸 GPU 成本不等于对外售价。建议根据 SLA 和毛利率加价： - -| 推荐场景 | 裸成本（input/output） | 说明 | -|----------|------------------------|------| -| 低延迟在线（c=8） | 13.85 元 / 19.10 元 per 1M | TTFT 147ms，TPOT 8.4ms，适合实时聊天 | -| 平衡型在线（c=16） | 10.24 元 / 14.12 元 per 1M | 吞吐与延迟兼顾，主流 API 可参考此档位 | -| 高吞吐在线（c=32 / rps=8） | 8.09 元 / 11.15 元 per 1M | 吞吐高、延迟可接受，适合批量在线服务 | -| 最大吞吐/离线（c=128） | 4.30 元 / 5.93 元 per 1M | 延迟 8.6s，仅适合离线批处理 | - -> 注意：`rps=16` 虽已接近最大吞吐，但系统已明显过载（E2E 15s），不建议按此成本对外定价。 - ---- - -## 附录 C：最大吞吐压测（优化服务参数） - -为了探索成本下限，重新部署了服务并调优了参数： - -```bash -envs/sglang/bin/sglang serve \ - --trust-remote-code \ - --model-path /data/models/DeepSeek-V4-Flash \ - --tp 8 \ - --moe-runner-backend marlin \ - --speculative-algorithm EAGLE \ - --speculative-num-steps 3 \ - --speculative-eagle-topk 1 \ - --speculative-num-draft-tokens 4 \ - --max-running-requests 512 \ - --chunked-prefill-size 16384 \ - --host 0.0.0.0 \ - --port 30000 -``` - -与附录 A/B 的测试相比，主要变化： -- `--max-running-requests` 从默认 256 提升到 **512** -- `--chunked-prefill-size` 从默认 8192 提升到 **16384** - -### 超大并发扫描结果 - -| 并发 | 耗时(s) | req/s | input tok/s | output tok/s | total tok/s | Mean TTFT | Mean TPOT | Mean E2E(ms) | -|------|---------|-------|-------------|--------------|-------------|-----------|-----------|--------------| -| 128 | 155.26 | 12.88 | 3,710.50 | 2,690.48 | 6,400.98 | 410.55 ms | 48.45 ms | 9,696.87 | -| 256 | 90.65 | 22.06 | 6,355.48 | 4,608.35 | 10,963.84 | 509.73 ms | 62.71 ms | 10,869.76 | -| 512 | 71.39 | 28.02 | 8,070.04 | 5,851.58 | 13,921.61 | 1,343.73 ms| 121.20 ms | 16,176.14 | -| **1024** | **68.21** | **29.32** | **8,445.53** | **6,123.84** | **14,569.37** | **13,265.46 ms** | **105.36 ms** | **27,566.99** | - -### 关键发现 - -1. **高并发下吞吐显著提升**：优化参数后，c=256 及以上远超原配置的最高值（7,129 tok/s）；c=1024 达到 **14,569 tok/s**，约为原配置峰值的两倍。 -2. **c=1024 达到最高吞吐**：total **14,569 tok/s**，output **6,124 tok/s**，但 TTFT 13.3s，系统已严重过载。 -3. **c=512 是实际可用上限**：total **13,922 tok/s**，output **5,852 tok/s**，虽然 TTFT 1.3s、E2E 16s，但吞吐接近峰值，适合离线批处理。 -4. **c=256 是高压在线的甜点**：total **10,964 tok/s**，TTFT 510ms，E2E 10.9s，如果业务能容忍 10 秒级响应，这是性价比很高的点。 - ---- - -## 附录 D：最低 Token 成本（基于最大吞吐） - -仍按 **8.0 元/卡时、8 卡整机 64 元/小时** 计算。 - -### 公式 - -``` -每 1M input tokens 成本 = 64 / (input_tok/s × 3600 / 1,000,000) -每 1M output tokens 成本 = 64 / (output_tok/s × 3600 / 1,000,000) -``` - -### 最大吞吐场景成本 - -| 场景 | input tok/s | output tok/s | 1M input cost | 1M output cost | -|------|-------------|--------------|---------------|----------------| -| c=128 | 3,710.50 | 2,690.48 | 4.79 元 | 6.61 元 | -| c=256 | 6,355.48 | 4,608.35 | 2.80 元 | 3.86 元 | -| c=512 | 8,070.04 | 5,851.58 | 2.20 元 | 3.04 元 | -| **c=1024** | **8,445.53** | **6,123.84** | **2.10 元** | **2.90 元** | - -### 结论 - -在当前硬件和优化参数下： - -- **理论最低 input 成本**：约 **2.10 元 / 1M input tokens**（c=1024） -- **理论最低 output 成本**：约 **2.90 元 / 1M output tokens**（c=1024） -- **实际可用最低成本**：c=512 时 **2.20 元 / 1M input**、**3.04 元 / 1M output**，此时吞吐已达峰值 95% 以上，且延迟比 c=1024 可控得多。 - -> 注意：c=1024 的 TTFT 超过 13 秒，仅适合完全不在意延迟的离线批处理任务；对外 API 服务不建议按此成本定价。 - ---- - -## 附录 E：测试数据长度分布 - -本次所有 benchmark 使用的是 `ShareGPT_V4.3_unfiltered_cleaned_split.json` 数据集。`sglang.bench_serving` 在 `--num-prompts 2000` 且 `--seed 42` 的条件下，从数据集中采样出 **2000 条真实对话请求**作为测试负载。 - -### 长度统计 - -| 指标 | Input Tokens | Output Tokens | -|------|--------------|---------------| -| 请求数 | 2,000 | 2,000 | -| 平均值 | 288.05 | 208.86 | -| 标准差 | 420.07 | 217.60 | -| 最小值 | 2 | 2 | -| P50 | 125 | 150 | -| P90 | 723 | 516 | -| P95 | 841 | 667 | -| P99 | 2,190 | 811 | -| 最大值 | 3,996 | 1,655 | -| **总量** | **576,098** | **417,728** | - -### 分布特点 - -1. **输入长度呈现典型长尾分布**：P50 仅 125 tokens，但 P99 达到 2,190 tokens，最大值 3,996 tokens。说明 ShareGPT 中既有大量短 prompt，也有少量长上下文对话。 -2. **输出长度相对集中**：P50 150 tokens，P90 516 tokens，P99 811 tokens，最大值 1,655 tokens。大部分回复属于中等长度。 -3. **输出/输入比约为 0.73**：平均每条请求输出 209 tokens、输入 288 tokens，整体负载偏 decode 侧。 -4. **与原始 ShareGPT 全集的差异**：原始全集输出长度均值约 1,122 tokens，而 bench 采样后的输出均值仅 209 tokens。这是因为 `sglang.bench_serving` 在处理 sharegpt 数据集时会根据内置规则对输出做截断/采样，以更贴近典型在线 serving 负载。 - -### 长度分布图 - -![长度分布](length_distribution.png) - -> 图表文件：`bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png` diff --git a/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/summary.json b/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/summary.json deleted file mode 100644 index af4117a..0000000 --- a/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/summary.json +++ /dev/null @@ -1,156 +0,0 @@ -[ - { - "name": "sharegpt_concurrency_1", - "args": "--max-concurrency 1", - "duration_s": 1715.45, - "request_throughput": 1.17, - "input_token_throughput": 335.83, - "output_token_throughput": 243.51, - "total_token_throughput": 579.34, - "mean_ttft_ms": 128.09, - "mean_tpot_ms": 4.0, - "mean_e2e_latency_ms": 856.77, - "accept_length": 2.62, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_1.json" - }, - { - "name": "sharegpt_concurrency_8", - "args": "--max-concurrency 8", - "duration_s": 448.91, - "request_throughput": 4.46, - "input_token_throughput": 1283.33, - "output_token_throughput": 930.54, - "total_token_throughput": 2213.87, - "mean_ttft_ms": 146.76, - "mean_tpot_ms": 8.41, - "mean_e2e_latency_ms": 1790.65, - "accept_length": 2.61, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_8.json" - }, - { - "name": "sharegpt_concurrency_16", - "args": "--max-concurrency 16", - "duration_s": 331.79, - "request_throughput": 6.03, - "input_token_throughput": 1736.31, - "output_token_throughput": 1259.0, - "total_token_throughput": 2995.3, - "mean_ttft_ms": 168.95, - "mean_tpot_ms": 12.67, - "mean_e2e_latency_ms": 2642.1, - "accept_length": 2.61, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_16.json" - }, - { - "name": "sharegpt_concurrency_32", - "args": "--max-concurrency 32", - "duration_s": 267.72, - "request_throughput": 7.47, - "input_token_throughput": 2151.87, - "output_token_throughput": 1560.32, - "total_token_throughput": 3712.19, - "mean_ttft_ms": 206.24, - "mean_tpot_ms": 20.72, - "mean_e2e_latency_ms": 4249.71, - "accept_length": 2.61, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_32.json" - }, - { - "name": "sharegpt_concurrency_64", - "args": "--max-concurrency 64", - "duration_s": 207.57, - "request_throughput": 9.64, - "input_token_throughput": 2775.39, - "output_token_throughput": 2012.43, - "total_token_throughput": 4787.82, - "mean_ttft_ms": 248.41, - "mean_tpot_ms": 32.61, - "mean_e2e_latency_ms": 6557.39, - "accept_length": 2.61, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_64.json" - }, - { - "name": "sharegpt_concurrency_128", - "args": "--max-concurrency 128", - "duration_s": 139.41, - "request_throughput": 14.35, - "input_token_throughput": 4132.36, - "output_token_throughput": 2996.37, - "total_token_throughput": 7128.73, - "mean_ttft_ms": 330.4, - "mean_tpot_ms": 46.15, - "mean_e2e_latency_ms": 8596.12, - "accept_length": 2.61, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_concurrency_128.json" - }, - { - "name": "sharegpt_rps_1", - "args": "--request-rate 1", - "duration_s": 2011.46, - "request_throughput": 0.99, - "input_token_throughput": 286.41, - "output_token_throughput": 207.67, - "total_token_throughput": 494.08, - "mean_ttft_ms": 146.87, - "mean_tpot_ms": 4.91, - "mean_e2e_latency_ms": 1007.62, - "accept_length": 2.61, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_1.json" - }, - { - "name": "sharegpt_rps_2", - "args": "--request-rate 2", - "duration_s": 1007.21, - "request_throughput": 1.99, - "input_token_throughput": 571.97, - "output_token_throughput": 414.74, - "total_token_throughput": 986.71, - "mean_ttft_ms": 160.63, - "mean_tpot_ms": 5.99, - "mean_e2e_latency_ms": 1205.24, - "accept_length": 2.61, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_2.json" - }, - { - "name": "sharegpt_rps_4", - "args": "--request-rate 4", - "duration_s": 505.49, - "request_throughput": 3.96, - "input_token_throughput": 1139.69, - "output_token_throughput": 826.39, - "total_token_throughput": 1966.08, - "mean_ttft_ms": 174.67, - "mean_tpot_ms": 9.02, - "mean_e2e_latency_ms": 1743.4, - "accept_length": 2.6, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_4.json" - }, - { - "name": "sharegpt_rps_8", - "args": "--request-rate 8", - "duration_s": 256.46, - "request_throughput": 7.8, - "input_token_throughput": 2246.34, - "output_token_throughput": 1628.82, - "total_token_throughput": 3875.15, - "mean_ttft_ms": 216.65, - "mean_tpot_ms": 25.35, - "mean_e2e_latency_ms": 4542.64, - "accept_length": 2.6, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_8.json" - }, - { - "name": "sharegpt_rps_16", - "args": "--request-rate 16", - "duration_s": 137.92, - "request_throughput": 14.5, - "input_token_throughput": 4176.94, - "output_token_throughput": 3028.7, - "total_token_throughput": 7205.64, - "mean_ttft_ms": 377.28, - "mean_tpot_ms": 87.16, - "mean_e2e_latency_ms": 15048.09, - "accept_length": 2.61, - "output_file": "bench_results/sglang_8card_systematic_20260704_120819/sharegpt_rps_16.json" - } -] \ No newline at end of file diff --git a/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121046/summary.json b/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121046/summary.json deleted file mode 100644 index 20dfa8b..0000000 --- a/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121046/summary.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "model": "/data/models/Qwen3-4B", - "draft_model": "deepseek-ai/dspark_qwen3_4b_block7", - "url": "http://127.0.0.1:30003/v1/completions", - "max_tokens": 64, - "num_prompts": 20, - "dataset": "/data/user1/yy/datasets/ShareGPT_filtered_chat.json", - "timestamp": "20260705_121046", - "results": [ - { - "concurrency": 1, - "num_prompts": 20, - "duration_s": 0.07108968612737954, - "success_count": 0, - "error": "all requests failed", - "errors": [ - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'", - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'", - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'", - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'", - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'" - ] - }, - { - "concurrency": 4, - "num_prompts": 20, - "duration_s": 0.036050053080543876, - "success_count": 0, - "error": "all requests failed", - "errors": [ - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'", - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'", - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'", - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'", - "404, message='Not Found', url='http://127.0.0.1:30003/v1/completions'" - ] - } - ] -} \ No newline at end of file diff --git a/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121218/summary.json b/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121218/summary.json deleted file mode 100644 index c719bcc..0000000 --- a/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121218/summary.json +++ /dev/null @@ -1,59 +0,0 @@ -{ - "model": "/data/models/Qwen3-4B", - "draft_model": "deepseek-ai/dspark_qwen3_4b_block7", - "url": "http://127.0.0.1:30003/v1/completions", - "max_tokens": 64, - "num_prompts": 20, - "dataset": "/data/user1/yy/datasets/ShareGPT_filtered_chat.json", - "timestamp": "20260705_121218", - "results": [ - { - "concurrency": 1, - "num_prompts": 20, - "duration_s": 3.1493232590146363, - "success_count": 20, - "fail_count": 0, - "input_tokens": 6132, - "output_tokens": 456, - "total_tokens": 6588, - "request_throughput": 6.350570695704836, - "input_throughput": 1947.0849753031027, - "output_throughput": 144.79301186207027, - "total_throughput": 2091.877987165173, - "mean_ttft_ms": 55.51949484506622, - "p50_ttft_ms": 27.64087135437876, - "p99_ttft_ms": 295.08487760089326, - "mean_tpot_ms": 4.6249661131817925, - "p50_tpot_ms": 4.634099831345484, - "p99_tpot_ms": 4.827384204164935, - "mean_e2e_ms": 156.8067875574343, - "p50_e2e_ms": 148.02213886287063, - "p99_e2e_ms": 406.6163514624348, - "errors": [] - }, - { - "concurrency": 4, - "num_prompts": 20, - "duration_s": 1.0271000929642469, - "success_count": 20, - "fail_count": 0, - "input_tokens": 6132, - "output_tokens": 450, - "total_tokens": 6582, - "request_throughput": 19.472298889857267, - "input_throughput": 5970.206839630238, - "output_throughput": 438.1267250217885, - "total_throughput": 6408.333564652026, - "mean_ttft_ms": 82.42244796128944, - "p50_ttft_ms": 37.77908312622458, - "p99_ttft_ms": 263.8731326162815, - "mean_tpot_ms": 5.172366561559164, - "p50_tpot_ms": 5.150655882704692, - "p99_tpot_ms": 5.708390201389117, - "mean_e2e_ms": 193.68963547749445, - "p50_e2e_ms": 153.2427944475785, - "p99_e2e_ms": 387.3923780536279, - "errors": [] - } - ] -} \ No newline at end of file diff --git a/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121256/summary.json b/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121256/summary.json deleted file mode 100644 index 61edc71..0000000 --- a/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121256/summary.json +++ /dev/null @@ -1,131 +0,0 @@ -{ - "model": "/data/models/Qwen3-4B", - "draft_model": "deepseek-ai/dspark_qwen3_4b_block7", - "url": "http://127.0.0.1:30003/v1/completions", - "max_tokens": 256, - "num_prompts": 500, - "dataset": "/data/user1/yy/datasets/ShareGPT_filtered_chat.json", - "timestamp": "20260705_121256", - "results": [ - { - "concurrency": 1, - "num_prompts": 500, - "duration_s": 181.22913801996037, - "success_count": 500, - "fail_count": 0, - "input_tokens": 108545, - "output_tokens": 36681, - "total_tokens": 145226, - "request_throughput": 2.7589382450460618, - "input_throughput": 598.9379036170495, - "output_throughput": 202.4012275330692, - "total_throughput": 801.3391311501188, - "mean_ttft_ms": 17.586736707482487, - "p50_ttft_ms": 15.236841514706612, - "p99_ttft_ms": 43.418583339080215, - "mean_tpot_ms": 4.756160026498775, - "p50_tpot_ms": 4.747943201144331, - "p99_tpot_ms": 4.845208972490266, - "mean_e2e_ms": 361.93272305326536, - "p50_e2e_ms": 369.4158981088549, - "p99_e2e_ms": 552.2739849891514, - "errors": [] - }, - { - "concurrency": 4, - "num_prompts": 500, - "duration_s": 49.20666282507591, - "success_count": 500, - "fail_count": 0, - "input_tokens": 108545, - "output_tokens": 36991, - "total_tokens": 145536, - "request_throughput": 10.161225559584139, - "input_throughput": 2205.900456730121, - "output_throughput": 751.7477893491538, - "total_throughput": 2957.6482460792745, - "mean_ttft_ms": 26.531575095374137, - "p50_ttft_ms": 24.292025598697364, - "p99_ttft_ms": 41.29285377450287, - "mean_tpot_ms": 5.007692315964597, - "p50_tpot_ms": 4.998277087638343, - "p99_tpot_ms": 5.130429720718591, - "mean_e2e_ms": 392.14430282171816, - "p50_e2e_ms": 400.28460952453315, - "p99_e2e_ms": 625.0063395127652, - "errors": [] - }, - { - "concurrency": 16, - "num_prompts": 500, - "duration_s": 14.20007478701882, - "success_count": 500, - "fail_count": 0, - "input_tokens": 108545, - "output_tokens": 36529, - "total_tokens": 145074, - "request_throughput": 35.21108215972787, - "input_throughput": 7643.973826055324, - "output_throughput": 2572.451240425399, - "total_throughput": 10216.425066480724, - "mean_ttft_ms": 31.879239567089826, - "p50_ttft_ms": 28.826430439949036, - "p99_ttft_ms": 74.12134333979331, - "mean_tpot_ms": 5.771187238080647, - "p50_tpot_ms": 5.76495784573639, - "p99_tpot_ms": 6.1421910060182805, - "mean_e2e_ms": 447.76513252267614, - "p50_e2e_ms": 459.7155440133065, - "p99_e2e_ms": 669.6232496039011, - "errors": [] - }, - { - "concurrency": 64, - "num_prompts": 500, - "duration_s": 7.960247536888346, - "success_count": 500, - "fail_count": 0, - "input_tokens": 108545, - "output_tokens": 36583, - "total_tokens": 145128, - "request_throughput": 62.81211704572815, - "input_throughput": 13635.882489457124, - "output_throughput": 4595.711355767746, - "total_throughput": 18231.59384522487, - "mean_ttft_ms": 73.04830395104364, - "p50_ttft_ms": 53.465207340195775, - "p99_ttft_ms": 252.97995713772252, - "mean_tpot_ms": 12.42267764661875, - "p50_tpot_ms": 12.61188557741512, - "p99_tpot_ms": 13.063075978620708, - "mean_e2e_ms": 968.1061067078263, - "p50_e2e_ms": 967.5668340642005, - "p99_e2e_ms": 1512.9108975501729, - "errors": [] - }, - { - "concurrency": 128, - "num_prompts": 500, - "duration_s": 7.319521516095847, - "success_count": 500, - "fail_count": 0, - "input_tokens": 108545, - "output_tokens": 36613, - "total_tokens": 145158, - "request_throughput": 68.31047615619204, - "input_throughput": 14829.52126874773, - "output_throughput": 5002.102927013319, - "total_throughput": 19831.62419576105, - "mean_ttft_ms": 145.49234842788428, - "p50_ttft_ms": 90.75760398991406, - "p99_ttft_ms": 423.2492828951217, - "mean_tpot_ms": 22.034148180580335, - "p50_tpot_ms": 23.052740126916806, - "p99_tpot_ms": 23.587740010073134, - "mean_e2e_ms": 1730.9650285840034, - "p50_e2e_ms": 1733.5386699996889, - "p99_e2e_ms": 2722.885905068833, - "errors": [] - } - ] -} \ No newline at end of file diff --git a/scripts/benchmark_dspark_0707/README.md b/scripts/benchmark_dspark_0707/README.md deleted file mode 100644 index 08cbf61..0000000 --- a/scripts/benchmark_dspark_0707/README.md +++ /dev/null @@ -1,60 +0,0 @@ -# DSpark Benchmark Scripts (2026-07-07) - -## Location - -`/data/user1/yy/scripts/benchmark_dspark_0707` - -## Scripts - -| Script | Purpose | Output Location | -|---|---|---| -| `bench_dspark_p1.sh` | Quick benchmark: chat_standard, generation_standard, summarization | `bench_results/dspark_grid_${RUN_ID}/p1_quick/...` | -| `bench_dspark_p2.sh` | Core benchmark: chat_short, rag_medium, long_context_probe | `bench_results/dspark_grid_${RUN_ID}/p2_core/...` | -| `bench_dspark_p3.sh` | Extension benchmark: stress_standard, decode_heavy | `bench_results/dspark_grid_${RUN_ID}/p3_extension/...` | -| `bench_dspark_focused.sh` | Focused sweep used for spec-tokens comparison | `bench_results/dspark_grid_${RUN_ID}/focused/...` or `bench_results/dspark_st_comparison_${RUN_ID}/focused/...` | -| `run_dspark_benchmark_grid.sh` | Orchestrator: starts vllm-dspark server, runs P1/P2/P3, stops server | `bench_results/dspark_grid_${RUN_ID}/` | -| `run_dspark_st_comparison.sh` | Orchestrator: compares `--spec-tokens 3` vs `5` | `bench_results/dspark_st_comparison_${RUN_ID}/` | -| `parse_results.py` | Parses grid logs and generates `report.md` | Writes to `bench_results//report.md` | -| `parse_st_comparison.py` | Parses spec-tokens comparison logs | Writes `comparison_report.md` under result root | -| `parse_eagle_vs_dspark.py` | Merges DSpark st comparison with SGLang EAGLE results | Writes `dspark_vs_eagle_report.md` under EAGLE result root | - -## Common Environment - -- Python env: `/data/user1/yy/envs/vllm-dspark` for server -- Benchmark client: `/data/user1/yy/envs/sglang/bin/python -m sglang.bench_serving --backend vllm` -- Model: `/data/models/DeepSeek-V4-Flash-DSpark` -- Default port: `30004` -- Default warmup: `100` requests - -## Usage - -### Full grid benchmark - -```bash -bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh -``` - -Results land in `bench_results/dspark_grid_YYYYMMDD-HHMMSS/`. - -### Spec-tokens comparison - -```bash -bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh -``` - -Results land in `bench_results/dspark_st_comparison_YYYYMMDD-HHMMSS/`. - -### Parse existing results - -```bash -# Grid -/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py /data/user1/yy/bench_results/dspark_grid_ - -# Spec-tokens comparison -/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_st_comparison.py /data/user1/yy/bench_results/dspark_st_comparison_ -``` - -## Notes - -- These scripts were originally located in `/data/user1/yy/benchmark_dspark_0707/` and moved here on 2026-07-08. -- `run_dspark_benchmark_grid.sh` and `run_dspark_st_comparison.sh` assume server start scripts are in `scripts/` (one directory up). diff --git a/scripts/benchmark_dspark_0707/bench_dspark_focused.sh b/scripts/benchmark_dspark_0707/bench_dspark_focused.sh deleted file mode 100755 index 0c4a124..0000000 --- a/scripts/benchmark_dspark_0707/bench_dspark_focused.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env bash -set -Eeuo pipefail - -# Focused benchmark for spec-tokens comparison. -# Covers short/medium/long input and low/medium/high concurrency. - -BACKEND="vllm" -PORT="${PORT:-30004}" -MODEL="/data/models/DeepSeek-V4-Flash-DSpark" -BENCH_PY="/data/user1/yy/envs/sglang/bin/python" -DATASET="/data/user1/yy/datasets/ShareGPT_filtered_chat.json" -RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid}" -WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}" -PHASE="focused" -RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" - -num_prompts() { - local c="$1" - if (( c <= 4 )); then - local n=$((c * 8)) - (( n < 32 )) && n=32 - echo "${n}" - elif (( c <= 32 )); then - echo $((c * 16)) - else - echo $((c * 8)) - fi -} - -run_case() { - local scenario="$1" - local input_len="$2" - local output_len="$3" - local concurrencies="$4" - for concurrency in ${concurrencies}; do - local prompts - prompts="$(num_prompts "${concurrency}")" - local out_dir="${RESULT_ROOT}/${PHASE}/${scenario}/${RUN_ID}" - mkdir -p "${out_dir}" - echo "phase=${PHASE} scenario=${scenario} input=${input_len} output=${output_len} concurrency=${concurrency} prompts=${prompts} warmup=${WARMUP_REQUESTS}" - "${BENCH_PY}" -m sglang.bench_serving \ - --backend "${BACKEND}" \ - --host 127.0.0.1 \ - --port "${PORT}" \ - --model "${MODEL}" \ - --dataset-name random \ - --dataset-path "${DATASET}" \ - --random-input-len "${input_len}" \ - --random-output-len "${output_len}" \ - --num-prompts "${prompts}" \ - --max-concurrency "${concurrency}" \ - --warmup-requests "${WARMUP_REQUESTS}" \ - 2>&1 | tee "${out_dir}/c${concurrency}.log" - done -} - -curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null -run_case "chat_short" 512 256 "1 8 16 32 64" -run_case "chat_standard" 1000 256 "64" -run_case "generation_standard" 1000 1000 "64" -run_case "rag_medium" 4000 512 "1 8 32" -run_case "long_context_probe" 16000 512 "1 4 8" -run_case "stress_standard" 1000 256 "64 96 128" -run_case "decode_heavy" 512 2000 "1 32" diff --git a/scripts/benchmark_dspark_0707/bench_dspark_p1.sh b/scripts/benchmark_dspark_0707/bench_dspark_p1.sh deleted file mode 100755 index 45505f6..0000000 --- a/scripts/benchmark_dspark_0707/bench_dspark_p1.sh +++ /dev/null @@ -1,60 +0,0 @@ -#!/usr/bin/env bash -set -Eeuo pipefail - -# P1 quick benchmark for vllm-dspark -# Mirrored from benchmark_grid_0707/h200_vllm_p1_grid.sh - -BACKEND="vllm" -PORT="${PORT:-30004}" -MODEL="/data/models/DeepSeek-V4-Flash-DSpark" -BENCH_PY="/data/user1/yy/envs/sglang/bin/python" -DATASET="/data/user1/yy/datasets/ShareGPT_filtered_chat.json" -RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid}" -WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}" -PHASE="p1_quick" -RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" - -num_prompts() { - local c="$1" - if (( c <= 4 )); then - local n=$((c * 8)) - (( n < 32 )) && n=32 - echo "${n}" - elif (( c <= 32 )); then - echo $((c * 16)) - else - echo $((c * 8)) - fi -} - -run_case() { - local scenario="$1" - local input_len="$2" - local output_len="$3" - local concurrencies="$4" - for concurrency in ${concurrencies}; do - local prompts - prompts="$(num_prompts "${concurrency}")" - local out_dir="${RESULT_ROOT}/${PHASE}/${scenario}/${RUN_ID}" - mkdir -p "${out_dir}" - echo "phase=${PHASE} scenario=${scenario} input=${input_len} output=${output_len} concurrency=${concurrency} prompts=${prompts}" - "${BENCH_PY}" -m sglang.bench_serving \ - --backend "${BACKEND}" \ - --host 127.0.0.1 \ - --port "${PORT}" \ - --model "${MODEL}" \ - --dataset-name random \ - --dataset-path "${DATASET}" \ - --random-input-len "${input_len}" \ - --random-output-len "${output_len}" \ - --num-prompts "${prompts}" \ - --max-concurrency "${concurrency}" \ - --warmup-requests "${WARMUP_REQUESTS}" \ - 2>&1 | tee "${out_dir}/c${concurrency}.log" - done -} - -curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null -run_case "chat_standard" 1000 256 "1 8 16 32" -run_case "generation_standard" 1000 1000 "1 8 16 32" -run_case "summarization" 8000 1000 "1 4 8 16" diff --git a/scripts/benchmark_dspark_0707/bench_dspark_p2.sh b/scripts/benchmark_dspark_0707/bench_dspark_p2.sh deleted file mode 100755 index 5a2be94..0000000 --- a/scripts/benchmark_dspark_0707/bench_dspark_p2.sh +++ /dev/null @@ -1,63 +0,0 @@ -#!/usr/bin/env bash -set -Eeuo pipefail - -# P2 core benchmark for vllm-dspark -# Mirrored from benchmark_grid_0707/h200_vllm_p2_grid.sh - -BACKEND="vllm" -PORT="${PORT:-30004}" -MODEL="/data/models/DeepSeek-V4-Flash-DSpark" -BENCH_PY="/data/user1/yy/envs/sglang/bin/python" -DATASET="/data/user1/yy/datasets/ShareGPT_filtered_chat.json" -RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid}" -WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}" -PHASE="p2_core" -RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" - -num_prompts() { - local c="$1" - if (( c <= 4 )); then - local n=$((c * 8)) - (( n < 32 )) && n=32 - echo "${n}" - elif (( c <= 32 )); then - echo $((c * 16)) - else - echo $((c * 8)) - fi -} - -run_case() { - local scenario="$1" - local input_len="$2" - local output_len="$3" - local concurrencies="$4" - for concurrency in ${concurrencies}; do - local prompts - prompts="$(num_prompts "${concurrency}")" - local out_dir="${RESULT_ROOT}/${PHASE}/${scenario}/${RUN_ID}" - mkdir -p "${out_dir}" - echo "phase=${PHASE} scenario=${scenario} input=${input_len} output=${output_len} concurrency=${concurrency} prompts=${prompts}" - "${BENCH_PY}" -m sglang.bench_serving \ - --backend "${BACKEND}" \ - --host 127.0.0.1 \ - --port "${PORT}" \ - --model "${MODEL}" \ - --dataset-name random \ - --dataset-path "${DATASET}" \ - --random-input-len "${input_len}" \ - --random-output-len "${output_len}" \ - --num-prompts "${prompts}" \ - --max-concurrency "${concurrency}" \ - --warmup-requests "${WARMUP_REQUESTS}" \ - 2>&1 | tee "${out_dir}/c${concurrency}.log" - done -} - -curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null -run_case "chat_short" 512 256 "1 8 16 32 64" -run_case "chat_standard" 1000 256 "64" -run_case "generation_standard" 1000 1000 "64" -run_case "rag_medium" 4000 512 "1 4 8 16 32" -run_case "summarization" 8000 1000 "32" -run_case "long_context_probe" 16000 512 "1 2 4 8 16" diff --git a/scripts/benchmark_dspark_0707/bench_dspark_p3.sh b/scripts/benchmark_dspark_0707/bench_dspark_p3.sh deleted file mode 100755 index b4df410..0000000 --- a/scripts/benchmark_dspark_0707/bench_dspark_p3.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/env bash -set -Eeuo pipefail - -# P3 extension benchmark for vllm-dspark -# Mirrored from benchmark_grid_0707/h200_vllm_p3_grid.sh - -BACKEND="vllm" -PORT="${PORT:-30004}" -MODEL="/data/models/DeepSeek-V4-Flash-DSpark" -BENCH_PY="/data/user1/yy/envs/sglang/bin/python" -DATASET="/data/user1/yy/datasets/ShareGPT_filtered_chat.json" -RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid}" -WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}" -PHASE="p3_extension" -RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" - -num_prompts() { - local c="$1" - if (( c <= 4 )); then - local n=$((c * 8)) - (( n < 32 )) && n=32 - echo "${n}" - elif (( c <= 32 )); then - echo $((c * 16)) - else - echo $((c * 8)) - fi -} - -run_case() { - local scenario="$1" - local input_len="$2" - local output_len="$3" - local concurrencies="$4" - for concurrency in ${concurrencies}; do - local prompts - prompts="$(num_prompts "${concurrency}")" - local out_dir="${RESULT_ROOT}/${PHASE}/${scenario}/${RUN_ID}" - mkdir -p "${out_dir}" - echo "phase=${PHASE} scenario=${scenario} input=${input_len} output=${output_len} concurrency=${concurrency} prompts=${prompts}" - "${BENCH_PY}" -m sglang.bench_serving \ - --backend "${BACKEND}" \ - --host 127.0.0.1 \ - --port "${PORT}" \ - --model "${MODEL}" \ - --dataset-name random \ - --dataset-path "${DATASET}" \ - --random-input-len "${input_len}" \ - --random-output-len "${output_len}" \ - --num-prompts "${prompts}" \ - --max-concurrency "${concurrency}" \ - --warmup-requests "${WARMUP_REQUESTS}" \ - 2>&1 | tee "${out_dir}/c${concurrency}.log" - done -} - -curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null -run_case "decode_heavy" 512 2000 "1 8 16 32" -run_case "long_rag" 32000 512 "1 2 4 8" -run_case "stress_standard" 1000 256 "96 128" -run_case "stress_generation" 1000 1000 "96 128" diff --git a/scripts/benchmark_dspark_0707/parse_eagle_vs_dspark.py b/scripts/benchmark_dspark_0707/parse_eagle_vs_dspark.py deleted file mode 100644 index 8bf10dc..0000000 --- a/scripts/benchmark_dspark_0707/parse_eagle_vs_dspark.py +++ /dev/null @@ -1,258 +0,0 @@ -#!/usr/bin/env python3 -"""Parse EAGLE focused benchmark logs and merge with DSpark st=3/st=5 report.""" - -import re -from pathlib import Path - -EAGLE_ROOT = Path("/data/user1/yy/bench_results/eagle_grid/focused") -DSPARK_REPORT = Path("/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649/comparison_report.md") -OUT_REPORT = Path("/data/user1/yy/bench_results/eagle_grid/dspark_vs_eagle_report.md") - - -def parse_log(log_path: Path) -> dict: - text = log_path.read_text(errors="ignore") - lines = text.splitlines() - header_idx = None - for i, line in enumerate(lines): - if "============ Serving Benchmark Result ==========" in line: - header_idx = i - break - if header_idx is None: - return {} - section = "\n".join(lines[header_idx:]) - - def get_float(pattern): - m = re.search(pattern + r"\s+([\d.]+)", section) - return float(m.group(1)) if m else None - - def get_int(pattern): - m = re.search(pattern + r"\s+(\d+)", section) - return int(m.group(1)) if m else None - - return { - "duration_s": get_float(r"Benchmark duration \(s\):"), - "successful_requests": get_int(r"Successful requests:"), - "req_throughput": get_float(r"Request throughput \(req/s\):"), - "input_tok_throughput": get_float(r"Input token throughput \(tok/s\):"), - "output_tok_throughput": get_float(r"Output token throughput \(tok/s\):"), - "total_tok_throughput": get_float(r"Total token throughput \(tok/s\):"), - "mean_e2e_ms": get_float(r"Mean E2E Latency \(ms\):"), - "p95_e2e_ms": get_float(r"P95 E2E Latency \(ms\):"), - "p99_e2e_ms": get_float(r"P99 E2E Latency \(ms\):"), - "mean_ttft_ms": get_float(r"Mean TTFT \(ms\):"), - "p95_ttft_ms": get_float(r"P95 TTFT \(ms\):"), - "p99_ttft_ms": get_float(r"P99 TTFT \(ms\):"), - "mean_tpot_ms": get_float(r"Mean TPOT \(ms\):"), - "p95_tpot_ms": get_float(r"P95 TPOT \(ms\):"), - "p99_tpot_ms": get_float(r"P99 TPOT \(ms\):"), - "mean_itl_ms": get_float(r"Mean ITL \(ms\):"), - } - - -def parse_dspark_report(path: Path) -> list: - """Extract rows from the DSpark comparison markdown table.""" - rows = [] - text = path.read_text(errors="ignore") - in_table = False - for line in text.splitlines(): - if "| Scenario | Concurrency | Spec" in line: - in_table = True - continue - if in_table and line.startswith("|"): - parts = [p.strip() for p in line.split("|") if p.strip()] - if len(parts) < 14 or parts[0] == "Scenario" or set(parts[0]) <= set("-:"): - continue - rows.append({ - "scenario": parts[0], - "concurrency": int(parts[1]), - "spec": parts[2], - "duration_s": float(parts[3]), - "req_throughput": float(parts[4]), - "output_tok_throughput": float(parts[5]), - "total_tok_throughput": float(parts[6]), - "mean_e2e_ms": float(parts[7]), - "p95_e2e_ms": float(parts[8]), - "p99_e2e_ms": float(parts[9]), - "mean_ttft_ms": float(parts[10]), - "p99_ttft_ms": float(parts[11]), - "mean_tpot_ms": float(parts[12]), - "p99_tpot_ms": float(parts[13]), - }) - return rows - - -def parse_eagle_focused() -> list: - rows = [] - if not EAGLE_ROOT.exists(): - return rows - for scenario_dir in sorted(EAGLE_ROOT.iterdir()): - if not scenario_dir.is_dir(): - continue - scenario = scenario_dir.name - # Use latest run_id only - run_ids = sorted([p.name for p in scenario_dir.iterdir() if p.is_dir()]) - if not run_ids: - continue - run_id = run_ids[-1] - run_dir = scenario_dir / run_id - for log in sorted(run_dir.glob("c*.log"), key=lambda p: int(p.stem[1:])): - concurrency = int(log.stem[1:]) - m = parse_log(log) - if not m: - continue - rows.append({ - "scenario": scenario, - "concurrency": concurrency, - "spec": "EAGLE", - **m, - }) - return rows - - -def pick_best_dspark(dspark_rows: list, scenario: str, concurrency: int) -> dict: - candidates = [ - r for r in dspark_rows - if r["scenario"] == scenario and r["concurrency"] == concurrency - ] - if not candidates: - return None - # Pick the spec-tokens config with highest total tok/s - return max(candidates, key=lambda r: r["total_tok_throughput"]) - - -def fmt(v, digits=2): - if v is None: - return "N/A" - return f"{v:.{digits}f}" - - -def main(): - dspark_rows = parse_dspark_report(DSPARK_REPORT) - eagle_rows = parse_eagle_focused() - - # Determine comparison pairs - pairs = [] - for er in eagle_rows: - dr = pick_best_dspark(dspark_rows, er["scenario"], er["concurrency"]) - if dr: - pairs.append((er, dr)) - - lines = [ - "# DSpark vs SGLang EAGLE 投机解码对比报告", - "", - "- DSpark 结果：`/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649`", - "- EAGLE 结果：`/data/user1/yy/bench_results/eagle_grid/focused`", - "- DSpark 模型：`/data/models/DeepSeek-V4-Flash-DSpark`", - "- EAGLE 模型：`/data/models/DeepSeek-V4-Flash`", - "- DSpark 后端：vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark)", - "- EAGLE 后端：SGLang (TP=8, FP8 KV cache, speculative-algorithm=EAGLE, num_steps=3, topk=1, draft_tokens=4)", - "- 压测客户端：`sglang.bench_serving`", - "- Warmup：100 条", - "", - "## 核心指标对比", - "", - "| Scenario | Concurrency | Method | Total tok/s | Out tok/s | Req/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) |", - "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|", - ] - - for er, dr in pairs: - lines.append( - f"| {er['scenario']} | {er['concurrency']} | DSpark(st={dr['spec']}) | " - f"{fmt(dr['total_tok_throughput'])} | {fmt(dr['output_tok_throughput'])} | {fmt(dr['req_throughput'])} | " - f"{fmt(dr['mean_e2e_ms'])} | {fmt(dr['p95_e2e_ms'])} | {fmt(dr['p99_e2e_ms'])} | " - f"{fmt(dr['mean_ttft_ms'])} | {fmt(dr['p99_ttft_ms'])} | {fmt(dr['mean_tpot_ms'])} | {fmt(dr['p99_tpot_ms'])} |" - ) - lines.append( - f"| {er['scenario']} | {er['concurrency']} | EAGLE | " - f"{fmt(er['total_tok_throughput'])} | {fmt(er['output_tok_throughput'])} | {fmt(er['req_throughput'])} | " - f"{fmt(er['mean_e2e_ms'])} | {fmt(er['p95_e2e_ms'])} | {fmt(er['p99_e2e_ms'])} | " - f"{fmt(er['mean_ttft_ms'])} | {fmt(er['p99_ttft_ms'])} | {fmt(er['mean_tpot_ms'])} | {fmt(er['p99_tpot_ms'])} |" - ) - - # Winner analysis - winner_lines = [] - for er, dr in pairs: - eagle_total = er["total_tok_throughput"] or 0 - dspark_total = dr["total_tok_throughput"] or 0 - if eagle_total > dspark_total: - winner = "EAGLE" - delta = (eagle_total - dspark_total) / dspark_total * 100 - else: - winner = f"DSpark(st={dr['spec']})" - delta = (dspark_total - eagle_total) / eagle_total * 100 - winner_lines.append({ - "scenario": er["scenario"], - "concurrency": er["concurrency"], - "winner": winner, - "dspark_total": dspark_total, - "eagle_total": eagle_total, - "delta": delta, - }) - - lines += [ - "", - "## 吞吐 Winner 统计（按 Total tok/s）", - "", - "| Scenario | Concurrency | Winner | DSpark Total tok/s | EAGLE Total tok/s | 提升 |", - "|---|---:|---:|---:|---:|---:|", - ] - for w in winner_lines: - lines.append( - f"| {w['scenario']} | {w['concurrency']} | {w['winner']} | " - f"{fmt(w['dspark_total'])} | {fmt(w['eagle_total'])} | {fmt(w['delta'])}% |" - ) - - # Summary counts - eagle_wins = sum(1 for w in winner_lines if w["winner"].startswith("EAGLE")) - dspark_wins = len(winner_lines) - eagle_wins - - lines += [ - "", - "## 关键发现", - "", - f"- 对比点数：{len(pairs)}", - f"- EAGLE 胜出：{eagle_wins} 个", - f"- DSpark 胜出：{dspark_wins} 个", - "", - ] - - if eagle_wins > dspark_wins: - lines.append("**结论：在相同负载下，SGLang EAGLE 的综合吞吐优于 vllm-dspark。**") - elif dspark_wins > eagle_wins: - lines.append("**结论：在相同负载下，vllm-dspark 的综合吞吐优于 SGLang EAGLE。**") - else: - lines.append("**结论：两者综合吞吐相当，各有优劣。**") - - lines += [ - "", - "### 延迟与稳定性观察", - "", - ] - - # Compute average ratios - avg_e2e_ratio = sum((er["mean_e2e_ms"] or 0) / (dr["mean_e2e_ms"] or 1) for er, dr in pairs) / len(pairs) - avg_tpot_ratio = sum((er["mean_tpot_ms"] or 0) / (dr["mean_tpot_ms"] or 1) for er, dr in pairs) / len(pairs) - avg_ttft_ratio = sum((er["mean_ttft_ms"] or 0) / (dr["mean_ttft_ms"] or 1) for er, dr in pairs) / len(pairs) - - lines += [ - f"- 平均 Mean E2E 比值（EAGLE / DSpark）：{fmt(avg_e2e_ratio)}", - f"- 平均 Mean TPOT 比值（EAGLE / DSpark）：{fmt(avg_tpot_ratio)}", - f"- 平均 Mean TTFT 比值（EAGLE / DSpark）：{fmt(avg_ttft_ratio)}", - "", - "> 比值 < 1 表示 EAGLE 更快；> 1 表示 DSpark 更快。", - "", - "## 优化建议", - "", - "1. 如果以吞吐为首要目标，当前配置下 **vllm-dspark 更优**，建议继续使用 `--spec-tokens` 并根据并发选择 3（高并发）或 5（低并发/长上下文）。", - "2. SGLang EAGLE 本次表现不佳，平均延迟和 TPOT 均显著高于 DSpark；如需进一步评估 EAGLE，可尝试调整 `--speculative-num-steps`、`--speculative-eagle-topk`、`--speculative-num-draft-tokens` 或更换 MOE runner backend，并确认是否已完成 `sglang.compile_deep_gemm` 预编译。", - "3. 注意两套后端的实现差异（CUDA graph、KV cache 管理、调度器、draft 模型架构）会显著影响不同并发和输入长度下的表现，建议按实际业务负载做最终选型。", - "", - ] - - OUT_REPORT.write_text("\n".join(lines)) - print("\n".join(lines)) - print(f"\nReport saved to: {OUT_REPORT}", file=__import__("sys").stderr) - - -if __name__ == "__main__": - main() diff --git a/scripts/benchmark_dspark_0707/parse_results.py b/scripts/benchmark_dspark_0707/parse_results.py deleted file mode 100644 index d85fefa..0000000 --- a/scripts/benchmark_dspark_0707/parse_results.py +++ /dev/null @@ -1,120 +0,0 @@ -#!/usr/bin/env python3 -"""Parse dspark benchmark grid logs and generate a markdown report.""" - -import os -import re -import sys -from pathlib import Path - - -def parse_log(log_path: Path) -> dict: - text = log_path.read_text(errors="ignore") - lines = text.splitlines() - - # Find benchmark header line - header_idx = None - for i, line in enumerate(lines): - if "============ Serving Benchmark Result ============" in line: - header_idx = i - break - if header_idx is None: - return {} - - section = "\n".join(lines[header_idx:]) - - def get_float(pattern): - m = re.search(pattern + r"\s+([\d.]+)", section) - return float(m.group(1)) if m else None - - def get_int(pattern): - m = re.search(pattern + r"\s+(\d+)", section) - return int(m.group(1)) if m else None - - return { - "duration_s": get_float(r"Benchmark duration \(s\):"), - "successful_requests": get_int(r"Successful requests:"), - "req_throughput": get_float(r"Request throughput \(req/s\):"), - "input_tok_throughput": get_float(r"Input token throughput \(tok/s\):"), - "output_tok_throughput": get_float(r"Output token throughput \(tok/s\):"), - "total_tok_throughput": get_float(r"Total token throughput \(tok/s\):"), - "mean_e2e_ms": get_float(r"Mean E2E Latency \(ms\):"), - "p95_e2e_ms": get_float(r"P95 E2E Latency \(ms\):"), - "p99_e2e_ms": get_float(r"P99 E2E Latency \(ms\):"), - "mean_ttft_ms": get_float(r"Mean TTFT \(ms\):"), - "p95_ttft_ms": get_float(r"P95 TTFT \(ms\):"), - "p99_ttft_ms": get_float(r"P99 TTFT \(ms\):"), - "mean_tpot_ms": get_float(r"Mean TPOT \(ms\):"), - "p95_tpot_ms": get_float(r"P95 TPOT \(ms\):"), - "p99_tpot_ms": get_float(r"P99 TPOT \(ms\):"), - "mean_itl_ms": get_float(r"Mean ITL \(ms\):"), - } - - -def main(): - result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/data/user1/yy/experiments/legacy_bench_results/dspark_grid_20260707-132641") - if not result_root.exists(): - print(f"Result root not found: {result_root}", file=sys.stderr) - sys.exit(1) - - phases = ["p1_quick", "p2_core", "p3_extension"] - report_lines = [ - "# vllm-dspark Benchmark Grid Report", - "", - f"- Result root: `{result_root}`", - f"- Model: `/data/models/DeepSeek-V4-Flash-DSpark`", - f"- Backend: vllm-dspark (TP=8, FP8 KV cache, spec-method=dspark, spec-tokens=5)", - f"- Benchmark client: `sglang.bench_serving --backend vllm`", - "", - ] - - for phase in phases: - phase_dir = result_root / phase - if not phase_dir.exists(): - continue - - report_lines.append(f"## {phase}") - report_lines.append("") - - scenarios = sorted([p.name for p in phase_dir.iterdir() if p.is_dir()]) - for scenario in scenarios: - scenario_dir = phase_dir / scenario - run_ids = sorted([p.name for p in scenario_dir.iterdir() if p.is_dir()]) - for run_id in run_ids: - run_dir = scenario_dir / run_id - logs = sorted(run_dir.glob("c*.log"), key=lambda p: int(p.stem[1:])) - - report_lines.append(f"### {scenario} ({run_id})") - report_lines.append("") - report_lines.append("| Concurrency | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean ITL(ms) |") - report_lines.append("|------------|------------|---------|------|---------|----------|------------|-------------|-------------|-------------|--------------|-------------|-------------|--------------|-------------|-------------|-------------|") - - for log in logs: - concurrency = int(log.stem[1:]) - metrics = parse_log(log) - if not metrics: - continue - - def fmt(v): - return f"{v:.2f}" if v is not None else "N/A" - - report_lines.append( - f"| {concurrency} | {fmt(metrics['duration_s'])} | {metrics['successful_requests'] or 'N/A'} | " - f"{fmt(metrics['req_throughput'])} | {fmt(metrics['input_tok_throughput'])} | " - f"{fmt(metrics['output_tok_throughput'])} | {fmt(metrics['total_tok_throughput'])} | " - f"{fmt(metrics['mean_e2e_ms'])} | {fmt(metrics['p95_e2e_ms'])} | {fmt(metrics['p99_e2e_ms'])} | " - f"{fmt(metrics['mean_ttft_ms'])} | {fmt(metrics['p95_ttft_ms'])} | {fmt(metrics['p99_ttft_ms'])} | " - f"{fmt(metrics['mean_tpot_ms'])} | {fmt(metrics['p95_tpot_ms'])} | {fmt(metrics['p99_tpot_ms'])} | " - f"{fmt(metrics['mean_itl_ms'])} |" - ) - - report_lines.append("") - - report_text = "\n".join(report_lines) - report_path = result_root / "report.md" - report_path.write_text(report_text) - print(report_text) - print(f"\nReport saved to: {report_path}", file=sys.stderr) - - -if __name__ == "__main__": - main() diff --git a/scripts/benchmark_dspark_0707/parse_st_comparison.py b/scripts/benchmark_dspark_0707/parse_st_comparison.py deleted file mode 100644 index d8aecaa..0000000 --- a/scripts/benchmark_dspark_0707/parse_st_comparison.py +++ /dev/null @@ -1,174 +0,0 @@ -#!/usr/bin/env python3 -"""Parse spec-tokens 3 vs 5 comparison results and generate a report.""" - -import os -import re -import sys -from pathlib import Path - - -def parse_log(log_path: Path) -> dict: - text = log_path.read_text(errors="ignore") - lines = text.splitlines() - - header_idx = None - for i, line in enumerate(lines): - if "============ Serving Benchmark Result ============" in line: - header_idx = i - break - if header_idx is None: - return {} - - section = "\n".join(lines[header_idx:]) - - def get_float(pattern): - m = re.search(pattern + r"\s+([\d.]+)", section) - return float(m.group(1)) if m else None - - def get_int(pattern): - m = re.search(pattern + r"\s+(\d+)", section) - return int(m.group(1)) if m else None - - return { - "duration_s": get_float(r"Benchmark duration \(s\):"), - "successful_requests": get_int(r"Successful requests:"), - "req_throughput": get_float(r"Request throughput \(req/s\):"), - "input_tok_throughput": get_float(r"Input token throughput \(tok/s\):"), - "output_tok_throughput": get_float(r"Output token throughput \(tok/s\):"), - "total_tok_throughput": get_float(r"Total token throughput \(tok/s\):"), - "mean_e2e_ms": get_float(r"Mean E2E Latency \(ms\):"), - "p95_e2e_ms": get_float(r"P95 E2E Latency \(ms\):"), - "p99_e2e_ms": get_float(r"P99 E2E Latency \(ms\):"), - "mean_ttft_ms": get_float(r"Mean TTFT \(ms\):"), - "p95_ttft_ms": get_float(r"P95 TTFT \(ms\):"), - "p99_ttft_ms": get_float(r"P99 TTFT \(ms\):"), - "mean_tpot_ms": get_float(r"Mean TPOT \(ms\):"), - "p95_tpot_ms": get_float(r"P95 TPOT \(ms\):"), - "p99_tpot_ms": get_float(r"P99 TPOT \(ms\):"), - "mean_itl_ms": get_float(r"Mean ITL \(ms\):"), - } - - -def collect_results(result_root: Path) -> dict: - """Collect results into {(scenario, concurrency, spec_tokens): metrics}.""" - results = {} - focused_dir = result_root / "focused" - if not focused_dir.exists(): - return results - - for scenario_dir in focused_dir.iterdir(): - if not scenario_dir.is_dir(): - continue - scenario = scenario_dir.name - for run_dir in scenario_dir.iterdir(): - if not run_dir.is_dir(): - continue - run_id = run_dir.name - # run_id format: _st{3,5} - if "_st" not in run_id: - continue - spec_tokens = run_id.split("_st")[-1] - for log in run_dir.glob("c*.log"): - concurrency = int(log.stem[1:]) - metrics = parse_log(log) - if metrics: - results[(scenario, concurrency, spec_tokens)] = metrics - return results - - -def fmt(v): - return f"{v:.2f}" if v is not None else "N/A" - - -def main(): - result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/data/user1/yy/bench_results/dspark_st_comparison_20260707-150649") - if not result_root.exists(): - print(f"Result root not found: {result_root}", file=sys.stderr) - sys.exit(1) - - results = collect_results(result_root) - if not results: - print("No results found.", file=sys.stderr) - sys.exit(1) - - # Group by scenario and concurrency - keys = sorted({(s, c) for (s, c, _) in results.keys()}) - - report_lines = [ - "# vllm-dspark `--spec-tokens` 对比报告", - "", - f"- 结果目录：`{result_root}`", - f"- 模型：`/data/models/DeepSeek-V4-Flash-DSpark`", - f"- 后端：vllm-dspark (TP=8, FP8 KV cache)", - f"- 对比参数：`--spec-tokens 3` vs `--spec-tokens 5`", - f"- Warmup：100 条", - f"- 压测客户端：`sglang.bench_serving --backend vllm`", - "", - "## 核心指标对比", - "", - "| Scenario | Concurrency | Spec | Duration(s) | Req/s | Out tok/s | Total tok/s | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | Mean TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) |", - "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|", - ] - - for scenario, concurrency in keys: - for st in ["3", "5"]: - m = results.get((scenario, concurrency, st), {}) - if not m: - continue - report_lines.append( - f"| {scenario} | {concurrency} | {st} | {fmt(m['duration_s'])} | {fmt(m['req_throughput'])} | " - f"{fmt(m['output_tok_throughput'])} | {fmt(m['total_tok_throughput'])} | {fmt(m['mean_e2e_ms'])} | " - f"{fmt(m['p95_e2e_ms'])} | {fmt(m['p99_e2e_ms'])} | {fmt(m['mean_ttft_ms'])} | {fmt(m['p99_ttft_ms'])} | " - f"{fmt(m['mean_tpot_ms'])} | {fmt(m['p99_tpot_ms'])} |" - ) - - # Summary table: best total tok/s per scenario/concurrency - report_lines.append("") - report_lines.append("## 吞吐 winner 统计") - report_lines.append("") - report_lines.append("| Scenario | Concurrency | Winner (Total tok/s) | st=3 Total tok/s | st=5 Total tok/s | 提升 |") - report_lines.append("|---|---:|---:|---:|---:|---:|") - - for scenario, concurrency in keys: - m3 = results.get((scenario, concurrency, "3"), {}) - m5 = results.get((scenario, concurrency, "5"), {}) - t3 = m3.get("total_tok_throughput", 0) or 0 - t5 = m5.get("total_tok_throughput", 0) or 0 - if t3 == 0 and t5 == 0: - continue - winner = "st=3" if t3 > t5 else "st=5" - uplift = (max(t3, t5) / min(t3, t5) - 1) * 100 if min(t3, t5) > 0 else 0 - report_lines.append( - f"| {scenario} | {concurrency} | {winner} | {fmt(t3)} | {fmt(t5)} | {fmt(uplift)}% |" - ) - - report_lines.append("") - report_lines.append("## 观察与建议") - report_lines.append("") - - # Compute overall wins - st3_wins = 0 - st5_wins = 0 - for scenario, concurrency in keys: - m3 = results.get((scenario, concurrency, "3"), {}) - m5 = results.get((scenario, concurrency, "5"), {}) - t3 = m3.get("total_tok_throughput", 0) or 0 - t5 = m5.get("total_tok_throughput", 0) or 0 - if t3 > t5: - st3_wins += 1 - elif t5 > t3: - st5_wins += 1 - - report_lines.append(f"- **st=3 在 {st3_wins} 个配置中吞吐更高，st=5 在 {st5_wins} 个配置中吞吐更高。**") - report_lines.append("- 具体结论请根据上表分析。") - report_lines.append("") - - report_text = "\n".join(report_lines) - report_path = result_root / "comparison_report.md" - report_path.write_text(report_text) - print(report_text) - print(f"\nComparison report saved to: {report_path}", file=sys.stderr) - - -if __name__ == "__main__": - main() diff --git a/scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh b/scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh deleted file mode 100755 index 4170d3c..0000000 --- a/scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh +++ /dev/null @@ -1,114 +0,0 @@ -#!/usr/bin/env bash -set -Eeuo pipefail - -# Run H200 DeepSeek-V4-Flash-DSpark benchmark grid in the current environment. -# This orchestrator starts the vllm-dspark server, runs P1/P2/P3 benchmarks, -# and stops the server afterwards. -# -# Environment: -# - vllm-dspark env: /data/user1/yy/envs/vllm-dspark -# - benchmark client: /data/user1/yy/envs/sglang/bin/python -m sglang.bench_serving -# - model: /data/models/DeepSeek-V4-Flash-DSpark -# - server port: 30004 -# -# Usage: -# bash run_dspark_benchmark_grid.sh -# -# To only run benchmarks against an already-running server: -# SKIP_MANAGE_SERVER=1 bash run_dspark_benchmark_grid.sh - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -START_SCRIPT="${SCRIPT_DIR}/../start_dsv4_dspark_8card.sh" -RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" -RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_grid_${RUN_ID}}" -LOG_DIR="${RESULT_ROOT}/logs" -PORT="${PORT:-30004}" -PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid" - -mkdir -p "${LOG_DIR}" - -log() { - echo "[$(date --iso-8601=seconds)] $*" | tee -a "${LOG_DIR}/orchestrator.log" -} - -is_server_healthy() { - curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1 -} - -stop_server() { - if [[ -f "${PID_FILE}" ]]; then - local pid - pid="$(cat "${PID_FILE}")" - if kill -0 "${pid}" 2>/dev/null; then - log "stopping existing dspark server pid=${pid}" - kill "${pid}" 2>/dev/null || true - sleep 5 - kill -9 "${pid}" 2>/dev/null || true - fi - rm -f "${PID_FILE}" - fi - # Fallback: kill any lingering vllm serve processes for this model - pkill -9 -f "vllm serve.*DeepSeek-V4-Flash-DSpark" 2>/dev/null || true - sleep 2 -} - -start_server() { - log "starting dspark server with ${START_SCRIPT}" - if [[ ! -x "${START_SCRIPT}" ]]; then - log "error: start script not found or not executable: ${START_SCRIPT}" - exit 1 - fi - bash "${START_SCRIPT}" >> "${LOG_DIR}/server.outer.log" 2>&1 - if ! is_server_healthy; then - log "error: dspark server failed to become healthy" - exit 1 - fi - log "dspark server is healthy" -} - -run_phase() { - local name="$1" - local script="$2" - local log_file="${LOG_DIR}/${name}.log" - - log "running ${name}: ${script}" - RUN_ID="${RUN_ID}" RESULT_ROOT="${RESULT_ROOT}" bash "${script}" 2>&1 | tee "${log_file}" - log "finished ${name}" -} - -on_exit() { - local code="$?" - log "orchestrator exiting with code=${code}" - if [[ -z "${SKIP_MANAGE_SERVER:-}" ]]; then - stop_server - fi - exit "${code}" -} -trap on_exit EXIT - -main() { - log "run_id=${RUN_ID}" - log "result_root=${RESULT_ROOT}" - log "script_dir=${SCRIPT_DIR}" - - if [[ -n "${SKIP_MANAGE_SERVER:-}" ]]; then - log "SKIP_MANAGE_SERVER is set, assuming server is already running" - if ! is_server_healthy; then - log "error: no healthy server found at port ${PORT}" - exit 1 - fi - else - stop_server - start_server - fi - - log "===== DSpark BENCHMARK START =====" - run_phase "dspark_p1" "${SCRIPT_DIR}/bench_dspark_p1.sh" - run_phase "dspark_p2" "${SCRIPT_DIR}/bench_dspark_p2.sh" - run_phase "dspark_p3" "${SCRIPT_DIR}/bench_dspark_p3.sh" - log "===== DSpark BENCHMARK DONE =====" - - log "all results saved to ${RESULT_ROOT}" -} - -main "$@" diff --git a/scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh b/scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh deleted file mode 100755 index 747c166..0000000 --- a/scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env bash -set -Eeuo pipefail - -# Compare vllm-dspark with --spec-tokens 3 vs 5. -# Uses the parameterized start script and focused benchmark. - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -START_SCRIPT="${SCRIPT_DIR}/../start_dsv4_dspark_8card_param.sh" -BENCH_SCRIPT="${SCRIPT_DIR}/bench_dspark_focused.sh" -RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" -RESULT_ROOT="${RESULT_ROOT:-/data/user1/yy/bench_results/dspark_st_comparison_${RUN_ID}}" -LOG_DIR="${RESULT_ROOT}/logs" -PORT="${PORT:-30004}" -PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid" -WARMUP_REQUESTS="${WARMUP_REQUESTS:-100}" - -mkdir -p "${LOG_DIR}" - -log() { - echo "[$(date --iso-8601=seconds)] $*" | tee -a "${LOG_DIR}/orchestrator.log" -} - -is_server_healthy() { - curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1 -} - -stop_server() { - if [[ -f "${PID_FILE}" ]]; then - local pid - pid="$(cat "${PID_FILE}" 2>/dev/null || true)" - if [[ -n "${pid}" ]] && kill -0 "${pid}" 2>/dev/null; then - log "stopping existing dspark server pid=${pid}" - kill "${pid}" 2>/dev/null || true - sleep 5 - kill -9 "${pid}" 2>/dev/null || true - fi - rm -f "${PID_FILE}" - fi - local pids - pids="$(pgrep -f "VLLM::|vllm serve.*DeepSeek-V4-Flash-DSpark" 2>/dev/null || true)" - if [[ -n "${pids}" ]]; then - for pid in ${pids}; do - if [[ "$pid" != "$$" ]]; then kill -9 "$pid" 2>/dev/null || true; fi - done - fi - sleep 2 -} - -start_server() { - local spec_tokens="$1" - log "starting dspark server with spec-tokens=${spec_tokens}" - if [[ ! -x "${START_SCRIPT}" ]]; then - log "error: start script not found or not executable: ${START_SCRIPT}" - exit 1 - fi - SPEC_TOKENS="${spec_tokens}" bash "${START_SCRIPT}" >> "${LOG_DIR}/server_st${spec_tokens}.outer.log" 2>&1 - if ! is_server_healthy; then - log "error: dspark server (st=${spec_tokens}) failed to become healthy" - exit 1 - fi - log "dspark server (st=${spec_tokens}) is healthy" -} - -run_benchmark() { - local spec_tokens="$1" - local log_file="${LOG_DIR}/focused_st${spec_tokens}.log" - log "running focused benchmark with spec-tokens=${spec_tokens}, warmup=${WARMUP_REQUESTS}" - RUN_ID="${RUN_ID}_st${spec_tokens}" RESULT_ROOT="${RESULT_ROOT}" WARMUP_REQUESTS="${WARMUP_REQUESTS}" bash "${BENCH_SCRIPT}" 2>&1 | tee "${log_file}" - log "finished focused benchmark with spec-tokens=${spec_tokens}" -} - -on_exit() { - local code="$?" - log "orchestrator exiting with code=${code}" - stop_server - exit "${code}" -} -trap on_exit EXIT - -main() { - log "run_id=${RUN_ID}" - log "result_root=${RESULT_ROOT}" - log "warmup_requests=${WARMUP_REQUESTS}" - - log "===== SPEC-TOKENS 3 START =====" - stop_server - start_server 3 - run_benchmark 3 - stop_server - - log "cooldown 30s" - sleep 30 - - log "===== SPEC-TOKENS 5 START =====" - start_server 5 - run_benchmark 5 - stop_server - - log "===== COMPARISON DONE =====" - log "all results saved to ${RESULT_ROOT}" -} - -main "$@" diff --git a/scripts/start_dsv4_dspark_8card.sh b/scripts/start_dsv4_dspark_8card.sh deleted file mode 100755 index 53bd04b..0000000 --- a/scripts/start_dsv4_dspark_8card.sh +++ /dev/null @@ -1,65 +0,0 @@ -#!/bin/bash -set -e - -cd /data/user1/yy -mkdir -p logs - -VENV="/data/user1/yy/envs/vllm-dspark" -PYTHON="$VENV/bin/python" -VLLM="$VENV/bin/vllm" -export PATH="$VENV/bin:$PATH" - -MODEL="/data/models/DeepSeek-V4-Flash-DSpark" -PORT=30004 -TP=8 -LOG="/data/user1/yy/logs/dsv4_flash_dspark_tp${TP}_$(date +%Y%m%d_%H%M%S).log" -PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid" - -export TMPDIR=/data/user1/yy/tmp -export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 - -echo "=== Starting DeepSeek-V4-Flash-DSpark (TP=$TP) ===" -echo "Model: $MODEL" -echo "Port: $PORT" -echo "Log: $LOG" - -rm -f "$PID_FILE" -nohup "$VLLM" serve "$MODEL" \ - --trust-remote-code \ - --tensor-parallel-size "$TP" \ - --kv-cache-dtype fp8 \ - --block-size 256 \ - --max-model-len auto \ - --max-num-seqs 256 \ - --tokenizer-mode deepseek_v4 \ - --reasoning-parser deepseek_v4 \ - --spec-method dspark \ - --spec-model "$MODEL" \ - --spec-tokens 5 \ - --no-disable-hybrid-kv-cache-manager \ - --disable-uvicorn-access-log \ - --port "$PORT" \ - > "$LOG" 2>&1 & -PID=$! -echo $PID > "$PID_FILE" -echo "PID: $PID" -echo "Waiting for health..." - -for i in $(seq 1 240); do - if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then - echo "Server is ready at http://127.0.0.1:$PORT" - echo "Log: $LOG" - exit 0 - fi - if ! kill -0 $PID 2>/dev/null; then - echo "ERROR: Server exited early" - tail -100 "$LOG" - exit 1 - fi - echo "Waiting... ($i/240)" - sleep 5 -done - -echo "ERROR: Server not healthy after 240 retries" -tail -100 "$LOG" -exit 1