964 B
964 B
SGLang vs vLLM on DeepSeek-V4-Flash (H200, TP=8)
Summary
- Model:
/data/models/DeepSeek-V4-Flash - Hardware: 8x NVIDIA H200 143GB
- Tensor Parallelism: 8
- Benchmark client:
sglang.bench_serving - SLO reference: S2 tier (TTFT P95 < 3s, TPOT < 50ms)
Side-by-side results
| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| c2_i262144_o4096 | sglang | 2 | 262144 | 4096 | 0.07 | 165.58 | 9734.97 | 10056.38 | 9.32 | 12.82 | 13.41 | 38983.24 | ⚠️ |
| c2_i262144_o4096 | vllm | 2 | 262144 | 4096 | 0.08 | 180.27 | 8742.36 | 9023.10 | 8.64 | 11.56 | 12.05 | 35853.05 | ⚠️ |
Notes
- SLO check uses TTFT P95 and TPOT mean (the same criteria as
docs/SLO_STANDARDS.md). - A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out.