diff --git a/BENCHMARK_WORKFLOW.md b/BENCHMARK_WORKFLOW.md index 4958067..e36f593 100644 --- a/BENCHMARK_WORKFLOW.md +++ b/BENCHMARK_WORKFLOW.md @@ -9,10 +9,9 @@ │ ├── nvidia_h200.env │ └── patches/kunlun_p800/ # runtime patches required by some images ├── scripts/ # shared benchmark/orchestrator/utility scripts -│ ├── common/ # reusable components (lib.sh, platform.sh, ...) -│ ├── analysis/ # cross-experiment comparison tools -│ ├── benchmark_dspark_0707/ # legacy DSpark benchmark suite -│ └── ... +│ ├── common/ # reusable components (lib.sh, platform.sh, parse_backend.py, ...) +│ ├── benchmark_dspark_0707/ # legacy DSpark benchmark suite (仍被 experiments/dsv4_h200_dspark/ 引用) +│ └── SLO_STANDARDS.md # 推理服务 SLO 标准 ├── experiments/ # experiment-centric directories (preferred) │ └── dsv4_p800_sglang/ │ ├── README.md @@ -38,7 +37,7 @@ 1. **Experiments are the primary organization unit.** - Each experiment lives under `experiments//` and contains its scripts, configuration, and results. - Shared orchestration code lives in `scripts/common/`; do not copy server start / health check logic into every experiment. - - Legacy experiments may remain under `scripts//` with outputs in `bench_results/`, but new work should use `experiments/`. + - 新实验必须放在 `experiments//` 下；`scripts/` 下仅保留 `scripts/common/` 和被现有实验引用的 legacy 脚本。 2. **Benchmark outputs live with their experiment.** - For `experiments//`, results go in `experiments//results//`. @@ -266,13 +265,7 @@ python3 experiments/dsv4_p800_sglang/parse_results.py \ # Legacy layout /data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \ - /data/user1/yy/bench_results/dspark_grid_ -``` - -### Cross-experiment comparison - -```bash -python3 scripts/analysis/compare_experiments.py + /data/user1/yy/experiments/legacy_bench_results/dspark_grid_ ``` ## Adding a New Platform or Experiment @@ -320,11 +313,11 @@ At minimum, `run_bench.sh` should: - `scripts/common/lib.sh`: logging, health checks, metadata JSON. - `scripts/common/platform.sh`: platform auto-detection and env loading. -- `scripts/common/server_docker.sh`: Docker-based server lifecycle. -- `scripts/common/bench_client_docker.sh`: run `sglang.bench_serving` inside a container. +- `scripts/common/warmup.py`: server 预热。 +- `scripts/common/parse_backend.py`: raw jsonl -> `results.json` + `report.md`。 +- `scripts/common/compare.py`: SGLang vs vLLM 横向对比表。 -Native (non-Docker) platforms can delegate to existing start scripts or add -new shared helpers under `scripts/common/`. +Docker/XPU 平台可以在 `scripts/common/` 下新增自己的 helper，但不要在实验目录复制通用逻辑。 ### 4. Example experiments diff --git a/README.md b/README.md index b135e7a..80832b5 100644 --- a/README.md +++ b/README.md @@ -8,13 +8,12 @@ | 目录/文件 | 说明 | |---|---| | `platforms/` | 芯片/加速器平台配置（`*.env`），如 P800、H200 | -| `scripts/common/` | 跨实验复用的 orchestration 组件（server 启停、health check、元数据生成） | -| `scripts/analysis/` | 跨实验对比分析工具 | -| `scripts/` | 旧版/legacy benchmark 脚本 | +| `scripts/common/` | 跨实验复用的 orchestration 组件（server 启停、health check、元数据生成、结果解析） | +| `scripts/` | 仅保留仍在使用的 legacy DSpark 脚本与 SLO 标准 | | `experiments/` | 以实验为单位的自包含目录（脚本 + 配置 + 结果） | | `experiments/legacy_bench_results/` | 历史归档的实验产物（只读），从原 `bench_results/` 迁移而来 | | `BENCHMARK_WORKFLOW.md` | benchmark 目录与命名规范 | -| `scripts/SLO_STANDARDS.md` | 推理服务 SLO 标准（TTFT/TPOT） | +| `docs/SLO_STANDARDS.md` | 推理服务 SLO 标准（TTFT/TPOT） | > 注：`.gitignore` 已排除 `envs/`、`deps/`、`tmp/`、`datasets/`、`logs/`、`*.zip`、原始 `raw_outputs/*.jsonl` 等大文件/中间文件。 @@ -34,16 +33,12 @@ ### 旧结构（scripts/ + bench_results/） +仅保留还在被 `experiments/dsv4_h200_dspark/` 引用的 legacy DSpark 脚本，其余旧脚本已删除。 + | 实验 | 脚本 | 最终报告 | 说明 | |---|---|---|---| | DSpark grid benchmark | `scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh` | `experiments/legacy_bench_results/dspark_grid_20260707-132641/report.md` | P1/P2/P3 全量网格 | | DSpark spec-tokens 对比 | `scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh` | `experiments/legacy_bench_results/dspark_st_comparison_20260707-150649/comparison_report.md` | `--spec-tokens 3` vs `5` | -| SGLang EAGLE vs DSpark | `scripts/start_sglang_dsv4_8card.sh` + 对比解析脚本 | `experiments/legacy_bench_results/eagle_grid/dspark_vs_eagle_report.md` | EAGLE 投机解码对比 | -| SGLang vs vLLM 后端对比（0707） | `scripts/benchmark_dsv4_backend_comparison.sh` | `experiments/legacy_bench_results/dsv4_backend_comparison_20260707/README.md` | 遗留 raw outputs 清单 | -| SGLang 8-card systematic | `scripts/run_sglang_benchmark.sh` | `experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/report.md` | 早期 SGLang 系统扫描 | -| SGLang 8-card max throughput | `scripts/run_sglang_max_throughput.sh` | `experiments/legacy_bench_results/sglang_8card_max_throughput_20260705_030839/summary.json` | 最大吞吐扫描 | -| vLLM-dspark Qwen3 | `scripts/bench_vllm_dspark_qwen3.py` | `experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121256/summary.json` | Qwen3 模型测速 | -| DSV4 后端对比（0705） | `scripts/bench_dsv4_comparison.py` | `experiments/legacy_bench_results/dsv4_comparison_20260705_152221/summary.json` | 早期 DSpark 配置对比 | ## 快速复现 @@ -83,15 +78,10 @@ bash experiments/dsv4_h200_vllm_dspark_vs_default/run_bench.sh bash experiments/dsv4_h200_max_context_length/run_bench.sh ``` -### DSpark grid（旧结构） +### DSpark grid / spec-tokens（旧结构） ```bash bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh -``` - -### Spec-tokens 对比 - -```bash bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh ``` @@ -99,9 +89,6 @@ bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh ```bash # 新结构 -python3 experiments/dsv4_p800_sglang/parse_results.py \ - experiments/dsv4_p800_sglang/results/ - python3 experiments/dsv4_h200_dspark/parse_results.py \ experiments/dsv4_h200_dspark/results/ @@ -112,7 +99,7 @@ python3 experiments/dsv4_h200_dspark/parse_results.py \ ## SLO 参考 -详见 `scripts/SLO_STANDARDS.md`。 +详见 `docs/SLO_STANDARDS.md`。 主要关注指标： diff --git a/docs/EXPERIMENT_GUIDE.md b/docs/EXPERIMENT_GUIDE.md new file mode 100644 index 0000000..9609f5e --- /dev/null +++ b/docs/EXPERIMENT_GUIDE.md @@ -0,0 +1,178 @@ +# 实验规范指南 + +本仓库用于统一记录和复现不同芯片、不同 backend（SGLang / vLLM）下的推理测速实验。 + +## 1. 目录结构 + +``` +experiments// +├── config.env # 实验参数唯一来源 +├── run_bench.sh # 编排入口 +├── start_sglang.sh # SGLang server 启动脚本 +├── start_vllm.sh # vLLM server 启动脚本 +└── results// + ├── comparison.md + ├── sglang/ + │ ├── results.json + │ ├── report.md + │ ├── raw_outputs/ # .gitignore 忽略 + │ └── logs/ # .gitignore 忽略 + └── vllm/ + ├── results.json + ├── report.md + ├── raw_outputs/ # .gitignore 忽略 + └── logs/ # .gitignore 忽略 +``` + +通用工具集中放在 `scripts/common/`，不要复制到每个实验： + +- `scripts/common/lib.sh`：日志、metadata、JSON 工具 +- `scripts/common/platform.sh`：平台自动检测 +- `scripts/common/warmup.py`：server 预热 +- `scripts/common/parse_backend.py`：解析 raw jsonl -> results.json + report.md +- `scripts/common/compare.py`：生成 SGLang vs vLLM 对比表 + +## 2. 新增一个实验 + +最快方式： + +```bash +cp -r experiments/TEMPLATE experiments/ +# 修改 config.env、start_*.sh +bash experiments//run_bench.sh +``` + +### 2.1 config.env 必备字段 + +```bash +EXPERIMENT="" +MODEL_NAME="DeepSeek-V4-Flash" +MODEL_PATH="/data/models/DeepSeek-V4-Flash" + +SGLANG_PORT="${SGLANG_PORT:-30006}" +VLLM_PORT="${VLLM_PORT:-30005}" + +VENV_SGLANG="${VENV_SGLANG:-/data/user1/yy/envs/sglang}" +VENV_VLLM="${VENV_VLLM:-/data/user1/yy/envs/vllm}" + +export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}" +TP="${TP:-8}" + +MAX_MODEL_LEN=... +MAX_NUM_SEQS=... +MAX_RUNNING=... + +# 场景："concurrency input_len output_len num_prompts" +declare -a SCENARIOS=( + "1 512 256 32" +) + +VENV_CLIENT="${VENV_CLIENT:-$VENV_SGLANG}" +SGLANG_START_SCRIPT="${SCRIPT_DIR:-.}/start_sglang.sh" +VLLM_START_SCRIPT="${SCRIPT_DIR:-.}/start_vllm.sh" +``` + +### 2.2 可复现性要求 + +每个 `results.json` 必须记录： + +- `metadata.model`：模型路径 +- `metadata.hardware` / `metadata.accelerator` / `metadata.chip` +- `metadata.env`：使用的虚拟环境路径 +- `metadata.git_commit` / `metadata.git_dirty` +- `config.server_args`：完整的 server 启动命令 +- `config.cuda_visible_devices` + +这些通过 `scripts/common/lib.sh` 中的 `write_metadata_json` 和 `jq` 注入。 + +## 3. 平台与硬件适配 + +### 3.1 芯片级默认配置 + +芯片相关默认值放到 `platforms/.env`，例如： + +- `platforms/nvidia_h200.env` +- `platforms/kunlun_p800.env` + +实验级覆盖通过 `config.env` 实现。 + +### 3.2 跨芯片注意事项 + +| 项 | H200 | P800 | 备注 | +|---|---|---|---| +| TP 大小 | 8 | 8/16 | 由 `TP` 控制 | +| KV cache dtype | fp8 | 可能不同 | 在 `start_vllm.sh` 调整 | +| MLA backend | flashinfer_mla | 可能不支持 | 在 `start_sglang.sh`/`start_vllm.sh` 调整 | +| 最大上下文 | 受显存限制 | 受显存限制 | 通过探针测试确定 | +| 设备变量 | `CUDA_VISIBLE_DEVICES` | 可能不同 | 在 `config.env` 设置 | + +如果某芯片不支持某个 feature，在 `start_*.sh` 里用条件判断，不要把条件写进通用脚本。 + +## 4. SLO 标准 + +默认使用 S2 层级： + +- TTFT P95 < 3000 ms +- TPOT mean < 50 ms + +`scripts/common/parse_backend.py` 和 `scripts/common/compare.py` 默认按这个标准打标。如果需要其他 tier，调用 `compare.py` 时传入 `--ttft-limit` 和 `--tpot-limit`。 + +## 5. 断点续测 + +`run_bench.sh` 必须实现 `scenario_already_completed()` 检查： + +- 如果某 scenario 的 `raw_outputs/*.jsonl` 中 `completed` 数量已达标，直接跳过 +- 重新执行脚本即可从中断处继续 + +## 6. Git 提交规范 + +### 6.1 必须提交 + +- 实验代码：`config.env`、`run_bench.sh`、`start_*.sh` +- 最终结果：`comparison.md`、`sglang/results.json`、`sglang/report.md`、`vllm/results.json`、`vllm/report.md` + +### 6.2 不要提交 + +- `raw_outputs/` +- `logs/` +- 中间 server 日志 + +这些已在 `.gitignore` 中忽略。 + +### 6.3 提交示例 + +```bash +git add experiments//config.env \ + experiments//run_bench.sh \ + experiments//start_*.sh \ + experiments//results//comparison.md \ + experiments//results//sglang/results.json \ + experiments//results//sglang/report.md \ + experiments//results//vllm/results.json \ + experiments//results//vllm/report.md + +git commit -m "results: run " +git push origin main +``` + +## 7. 命名约定 + +- 实验目录：`{model}_{chip}_{scenario}_{backend_vs_backend}`，例如 `dsv4_h200_sglang_vs_vllm` +- scenario 名称：`c{concurrency}_i{input_len}_o{output_len}` +- raw 输出：`{backend}_{label}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl` +- run_id：`YYYYMMDD-HHMMSS` + +## 8. 常见问题 + +### 8.1 新芯片上 server 起不来 + +1. 检查 `platforms/.env` 是否已定义 +2. 检查 `start_*.sh` 中的 backend-specific 参数是否被该芯片支持 +3. 先用最小场景（input=512, output=256, concurrency=1）验证通路 + +### 8.2 长上下文 OOM + +1. 降低并发 +2. 降低输出长度 +3. 调整 `mem-fraction-static` / `gpu-memory-utilization` +4. 记录能跑通的最大组合，更新矩阵 diff --git a/docs/H200_QUICKSTART.md b/docs/H200_QUICKSTART.md index 664dee2..dc83a90 100644 --- a/docs/H200_QUICKSTART.md +++ b/docs/H200_QUICKSTART.md @@ -97,13 +97,7 @@ For a concrete example, see `experiments/dsv4_h200_dspark/`. ## 5. Cross-platform comparison -After running experiments on both P800 and H200, compare them with: - -```bash -python3 scripts/analysis/compare_experiments.py \ - experiments/dsv4_p800_sglang/results/ \ - experiments/dsv4_h200_dspark/results/ -``` +跨平台对比脚本目前未统一提供。可分别读取各实验 `results//results.json` 中的结构化数据，按 scenario 聚合后生成对比表。 ## Notes diff --git a/scripts/SLO_STANDARDS.md b/docs/SLO_STANDARDS.md similarity index 100% rename from scripts/SLO_STANDARDS.md rename to docs/SLO_STANDARDS.md diff --git a/experiments/TEMPLATE/README.md b/experiments/TEMPLATE/README.md new file mode 100644 index 0000000..c6ee833 --- /dev/null +++ b/experiments/TEMPLATE/README.md @@ -0,0 +1,45 @@ +# TEMPLATE：SGLang vs vLLM 实验模板 + +这是一个最小可运行的实验模板。要新增一个实验： + +```bash +cp -r experiments/TEMPLATE experiments/ +# 修改 config.env 里的参数 +# 按需调整 start_sglang.sh / start_vllm.sh 的启动参数 +bash experiments//run_bench.sh +``` + +## 目录约定 + +- `config.env`：实验参数唯一来源 +- `start_sglang.sh` / `start_vllm.sh`：server 启动脚本 +- `run_bench.sh`：编排入口 +- `results//`：自动生成的结果目录 + - `sglang/`, `vllm/`：各自 backend 的结果 + - `comparison.md`：横向对比表 + +## 通用工具位置 + +不要在本目录下复制解析脚本，直接引用： + +- 预热：`../../scripts/common/warmup.py` +- 结果解析：`../../scripts/common/parse_backend.py` +- 对比报告：`../../scripts/common/compare.py` +- 公共函数：`../../scripts/common/lib.sh`, `../../scripts/common/platform.sh` + +## 提交规范 + +只提交最终文件： + +```bash +git add experiments//config.env \ + experiments//run_bench.sh \ + experiments//start_*.sh \ + experiments//results//comparison.md \ + experiments//results//sglang/results.json \ + experiments//results//sglang/report.md \ + experiments//results//vllm/results.json \ + experiments//results//vllm/report.md +``` + +`raw_outputs/` 和 `logs/` 已在 `.gitignore` 中忽略。 diff --git a/experiments/TEMPLATE/config.env b/experiments/TEMPLATE/config.env new file mode 100644 index 0000000..8d6df3c --- /dev/null +++ b/experiments/TEMPLATE/config.env @@ -0,0 +1,33 @@ +# Template configuration for a SGLang vs vLLM benchmark experiment. +# Copy this directory to experiments// and adjust the values. + +EXPERIMENT="TEMPLATE" +MODEL_NAME="DeepSeek-V4-Flash" +MODEL_PATH="/data/models/DeepSeek-V4-Flash" +SERVED_MODEL_NAME="deepseek-v4-flash" + +SGLANG_PORT="${SGLANG_PORT:-30006}" +VLLM_PORT="${VLLM_PORT:-30005}" + +VENV_SGLANG="${VENV_SGLANG:-/data/user1/yy/envs/sglang}" +VENV_VLLM="${VENV_VLLM:-/data/user1/yy/envs/vllm}" + +# Hardware: override in platforms/.env or set explicitly. +export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}" +TP="${TP:-8}" + +# Server limits. +MAX_MODEL_LEN="${MAX_MODEL_LEN:-32768}" +MAX_NUM_SEQS="${MAX_NUM_SEQS:-256}" +MAX_RUNNING="${MAX_RUNNING:-256}" + +# Scenarios: "concurrency input_len output_len num_prompts". +declare -a SCENARIOS=( + "1 512 256 32" + "32 512 256 128" +) + +VENV_CLIENT="${VENV_CLIENT:-$VENV_SGLANG}" + +SGLANG_START_SCRIPT="${SCRIPT_DIR:-.}/start_sglang.sh" +VLLM_START_SCRIPT="${SCRIPT_DIR:-.}/start_vllm.sh" diff --git a/experiments/TEMPLATE/run_bench.sh b/experiments/TEMPLATE/run_bench.sh new file mode 100755 index 0000000..04a6fdd --- /dev/null +++ b/experiments/TEMPLATE/run_bench.sh @@ -0,0 +1,265 @@ +#!/usr/bin/env bash +# Template orchestrator for a SGLang vs vLLM benchmark experiment. +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")" + +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../scripts/common/lib.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../scripts/common/platform.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" +RESULT_BASE="${SCRIPT_DIR}/results" + +log_dir_global="${RESULT_BASE}/${RUN_ID}/logs" +mkdir -p "$log_dir_global" +log_init "${log_dir_global}/orchestrator.log" + +log "experiment=${EXPERIMENT_NAME}" +log "run_id=${RUN_ID}" +log "platform=${PLATFORM}" +log "hardware=${HARDWARE}" +log "model=${MODEL_PATH}" +log "max_model_len=${MAX_MODEL_LEN}" +log "scenarios=${#SCENARIOS[@]}" + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +is_server_healthy() { + local port="$1" + curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${port}/health" >/dev/null 2>&1 +} + +stop_server() { + local backend="$1" + local pid_file="/data/user1/yy/${EXPERIMENT_NAME}_${backend}.pid" + if [[ -f "$pid_file" ]]; then + local pid + pid="$(cat "$pid_file")" + if kill -0 "$pid" 2>/dev/null; then + log "stopping ${backend} server pid=${pid}" + kill "$pid" 2>/dev/null || true + sleep 5 + kill -9 "$pid" 2>/dev/null || true + fi + rm -f "$pid_file" + fi + # Fallback cleanup. + if [[ "$backend" == "sglang" ]]; then + pkill -9 -f "sglang serve.*${MODEL_NAME}" 2>/dev/null || true + else + pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true + fi + sleep 2 +} + +start_server() { + local backend="$1" + local start_script + if [[ "$backend" == "sglang" ]]; then + start_script="${SGLANG_START_SCRIPT}" + local port="$SGLANG_PORT" + else + start_script="${VLLM_START_SCRIPT}" + local port="$VLLM_PORT" + fi + + log "starting ${backend} server with ${start_script}" + bash "${start_script}" >> "${log_dir_global}/${backend}.server.outer.log" 2>&1 + + if ! is_server_healthy "$port"; then + log "error: ${backend} server failed to become healthy" + return 1 + fi + log "${backend} server is healthy on port ${port}" +} + +run_warmup() { + local backend="$1" + local port + if [[ "$backend" == "sglang" ]]; then + port="$SGLANG_PORT" + else + port="$VLLM_PORT" + fi + + log "warming up ${backend}" + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/warmup.py" \ + --backend "$backend" \ + --port "$port" \ + --input-len 4000 \ + --output-len 512 \ + --num 2 \ + --env-python "${VENV_CLIENT}/bin/python" \ + >> "${log_dir_global}/${backend}.warmup.log" 2>&1 + log "warmup for ${backend} completed" +} + +scenario_already_completed() { + local output_file="$1" + local expected="$2" + [[ -s "$output_file" ]] || return 1 + local completed + completed="$(${VENV_CLIENT}/bin/python -c " +import json, sys +path = sys.argv[1] +try: + with open(path, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if line: + data = json.loads(line) + print(data.get('completed', 0)) + break +except Exception: + print(0) +" "$output_file")" + [[ "${completed:-0}" -ge "$expected" ]] +} + +run_benchmark() { + local backend="$1" + local result_root="${RESULT_BASE}/${RUN_ID}/${backend}" + local raw_dir="${result_root}/raw_outputs" + local phase_log_dir="${result_root}/logs" + + mkdir -p "$raw_dir" "$phase_log_dir" + + local port + if [[ "$backend" == "sglang" ]]; then + port="$SGLANG_PORT" + else + port="$VLLM_PORT" + fi + + log "===== ${backend} START (max_model_len=${MAX_MODEL_LEN}) =====" + + stop_server "$backend" + start_server "$backend" + run_warmup "$backend" + + for scenario in "${SCENARIOS[@]}"; do + read -r concurrency input_len output_len num_prompts <<< "$scenario" + output_file="${raw_dir}/${backend}_phase1_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl" + detail_log="${phase_log_dir}/${backend}_phase1_c${concurrency}_i${input_len}_o${output_len}.log" + + if scenario_already_completed "$output_file" "$num_prompts"; then + log "skipping already-completed ${backend} scenario: c=${concurrency} i=${input_len} o=${output_len}" + continue + fi + + log "running ${backend} scenario: c=${concurrency} i=${input_len} o=${output_len} n=${num_prompts}" + + "${VENV_CLIENT}/bin/python" -m sglang.bench_serving \ + --backend "$backend" \ + --host 127.0.0.1 \ + --port "$port" \ + --dataset-name random \ + --random-input-len "$input_len" \ + --random-output-len "$output_len" \ + --num-prompts "$num_prompts" \ + --max-concurrency "$concurrency" \ + --request-rate 10000 \ + --output-file "$output_file" \ + --output-details \ + > "$detail_log" 2>&1 || { + log "ERROR: ${backend} scenario c=${concurrency} i=${input_len} o=${output_len} failed; see ${detail_log}" + continue + } + + log "finished ${backend} scenario: output=${output_file}" + done + + stop_server "$backend" + log "===== ${backend} DONE =====" +} + +parse_backend() { + local backend="$1" + local result_root="${RESULT_BASE}/${RUN_ID}/${backend}" + log "parsing ${backend} results in ${result_root}" + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend "$backend" \ + >> "${result_root}/logs/parse.log" 2>&1 || { + log "WARNING: parser failed for ${backend}; see ${result_root}/logs/parse.log" + } +} + +write_backend_metadata() { + local backend="$1" + local result_root="${RESULT_BASE}/${RUN_ID}/${backend}" + ensure_result_root "$result_root" + local meta_json="${result_root}/results.json" + + local env_path + if [[ "$backend" == "sglang" ]]; then + env_path="$VENV_SGLANG" + else + env_path="$VENV_VLLM" + fi + + # Capture the exact server launch args for reproducibility. + local server_args + if [[ "$backend" == "sglang" ]]; then + server_args="sglang serve --trust-remote-code --model-path $MODEL_PATH --tp $TP --moe-runner-backend marlin --context-length $MAX_MODEL_LEN --max-running-requests $MAX_RUNNING --mem-fraction-static 0.88 --host 0.0.0.0 --port $SGLANG_PORT" + else + server_args="vllm serve $MODEL_PATH --trust-remote-code --tensor-parallel-size $TP --kv-cache-dtype fp8 --max-model-len $MAX_MODEL_LEN --max-num-seqs $MAX_NUM_SEQS --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port $VLLM_PORT" + fi + + write_metadata_json \ + "$meta_json" \ + "${EXPERIMENT_NAME}_${backend}" \ + "$RUN_ID" \ + "$MODEL_PATH" \ + "$backend" \ + "$backend" \ + "$HARDWARE" \ + "$ACCELERATOR" \ + "$CHIP" \ + "experiments/${EXPERIMENT_NAME}/run_bench.sh" \ + "$env_path" \ + "${HARDWARE} ${backend} TP=${TP} benchmark for ${MODEL_NAME}" + + jq --arg backend "$backend" \ + --arg server_args "$server_args" \ + '.config = { + "tp": ($TP | tonumber), + "cuda_visible_devices": $CUDA_VISIBLE_DEVICES, + "max_model_len": ($MAX_MODEL_LEN | tonumber), + "backend": $backend, + "server_start_script": "experiments/\($EXPERIMENT_NAME)/start_\($backend).sh", + "server_args": $server_args + }' "$meta_json" > "${meta_json}.tmp" && mv "${meta_json}.tmp" "$meta_json" +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +stop_server sglang +stop_server vllm + +write_backend_metadata sglang +write_backend_metadata vllm + +run_benchmark sglang +parse_backend sglang + +run_benchmark vllm +parse_backend vllm + +log "generating comparison report" +"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/compare.py" \ + --sglang "${RESULT_BASE}/${RUN_ID}/sglang" \ + --vllm "${RESULT_BASE}/${RUN_ID}/vllm" \ + --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \ + >> "${log_dir_global}/compare.log" 2>&1 || { + log "WARNING: comparison script failed; see ${log_dir_global}/compare.log" + } + +log "all results saved to ${RESULT_BASE}/${RUN_ID}" diff --git a/experiments/TEMPLATE/start_sglang.sh b/experiments/TEMPLATE/start_sglang.sh new file mode 100755 index 0000000..0bd63aa --- /dev/null +++ b/experiments/TEMPLATE/start_sglang.sh @@ -0,0 +1,62 @@ +#!/bin/bash +# Start SGLang server for this experiment. +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +cd /data/user1/yy +mkdir -p logs + +export PATH="${VENV_SGLANG}/bin:$PATH" +export PYTHONUNBUFFERED=1 +export SGLANG_LOG_LEVEL=info +export TMPDIR=/data/user1/yy/tmp +export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}" + +LOG="/data/user1/yy/logs/${EXPERIMENT}_sglang_$(date +%Y%m%d_%H%M%S).log" +PID_FILE="/data/user1/yy/${EXPERIMENT}_sglang.pid" + +rm -f "$PID_FILE" + +echo "=== Starting SGLang server (TP=$TP, max_model_len=$MAX_MODEL_LEN) ===" +echo "Model: $MODEL_PATH" +echo "Port: $SGLANG_PORT" +echo "Log: $LOG" + +nohup sglang serve \ + --trust-remote-code \ + --model-path "$MODEL_PATH" \ + --tp "$TP" \ + --moe-runner-backend marlin \ + --context-length "$MAX_MODEL_LEN" \ + --max-running-requests "$MAX_RUNNING" \ + --mem-fraction-static 0.88 \ + --host 0.0.0.0 \ + --port "$SGLANG_PORT" \ + > "$LOG" 2>&1 & + +PID=$! +echo $PID > "$PID_FILE" +echo "PID: $PID" +echo "Waiting for health..." + +for i in $(seq 1 240); do + if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${SGLANG_PORT}/health" >/dev/null 2>&1; then + echo "SGLang server is ready at http://127.0.0.1:${SGLANG_PORT}" + echo "Log: $LOG" + exit 0 + fi + if ! kill -0 $PID 2>/dev/null; then + echo "ERROR: SGLang server exited early" + tail -200 "$LOG" + exit 1 + fi + echo "Waiting... ($i/240)" + sleep 5 +done + +echo "ERROR: SGLang server not healthy after 240 retries" +tail -200 "$LOG" +exit 1 diff --git a/experiments/TEMPLATE/start_vllm.sh b/experiments/TEMPLATE/start_vllm.sh new file mode 100755 index 0000000..555b88b --- /dev/null +++ b/experiments/TEMPLATE/start_vllm.sh @@ -0,0 +1,65 @@ +#!/bin/bash +# Start vLLM server for this experiment. +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +cd /data/user1/yy +mkdir -p logs + +VENV="${VENV_VLLM}" +export PATH="$VENV/bin:$PATH" +export PYTHONUNBUFFERED=1 +export TMPDIR=/data/user1/yy/tmp +export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}" + +LOG="/data/user1/yy/logs/${EXPERIMENT}_vllm_$(date +%Y%m%d_%H%M%S).log" +PID_FILE="/data/user1/yy/${EXPERIMENT}_vllm.pid" + +rm -f "$PID_FILE" + +echo "=== Starting vLLM server (TP=$TP, max_model_len=$MAX_MODEL_LEN) ===" +echo "Model: $MODEL_PATH" +echo "Port: $VLLM_PORT" +echo "Log: $LOG" + +nohup vllm serve "$MODEL_PATH" \ + --trust-remote-code \ + --tensor-parallel-size "$TP" \ + --kv-cache-dtype fp8 \ + --max-model-len "$MAX_MODEL_LEN" \ + --max-num-seqs "$MAX_NUM_SEQS" \ + --block-size 256 \ + --gpu-memory-utilization 0.90 \ + --tokenizer-mode deepseek_v4 \ + --reasoning-parser deepseek_v4 \ + --no-disable-hybrid-kv-cache-manager \ + --disable-uvicorn-access-log \ + --port "$VLLM_PORT" \ + > "$LOG" 2>&1 & + +PID=$! +echo $PID > "$PID_FILE" +echo "PID: $PID" +echo "Waiting for health..." + +for i in $(seq 1 240); do + if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${VLLM_PORT}/health" >/dev/null 2>&1; then + echo "vLLM server is ready at http://127.0.0.1:${VLLM_PORT}" + echo "Log: $LOG" + exit 0 + fi + if ! kill -0 $PID 2>/dev/null; then + echo "ERROR: vLLM server exited early" + tail -200 "$LOG" + exit 1 + fi + echo "Waiting... ($i/240)" + sleep 5 +done + +echo "ERROR: vLLM server not healthy after 240 retries" +tail -200 "$LOG" +exit 1 diff --git a/experiments/dsv4_h200_256k_4k_probe/parse_backend.py b/experiments/dsv4_h200_256k_4k_probe/parse_backend.py deleted file mode 100755 index 935d870..0000000 --- a/experiments/dsv4_h200_256k_4k_probe/parse_backend.py +++ /dev/null @@ -1,219 +0,0 @@ -#!/usr/bin/env python3 -"""Parse raw sglang.bench_serving JSONL outputs for one backend. - -Reads JSONL files like {sglang|vllm}_phase1_MMDD_concurrency_inputlen_outputlen.jsonl -and generates results.json + report.md in the given result root. - -Usage: - python3 parse_backend.py [--backend sglang|vllm] -""" -import argparse -import json -import re -import sys -from pathlib import Path - - -def parse_jsonl(path: Path) -> dict | None: - with open(path, "r", encoding="utf-8") as f: - for line in f: - line = line.strip() - if not line: - continue - try: - return json.loads(line) - except json.JSONDecodeError: - continue - return None - - -def compute_metrics(data: dict) -> dict: - completed = data.get("completed", 0) - total = len(data.get("input_lens", [])) - failed = total - completed if total > 0 else 0 - duration_s = data.get("duration", 0.0) - - return { - "success": completed, - "failed": failed, - "duration_s": duration_s, - "request_throughput": data.get("request_throughput", 0.0), - "input_token_throughput": data.get("input_throughput", 0.0), - "output_token_throughput": data.get("output_throughput", 0.0), - "total_token_throughput": data.get("total_throughput", 0.0), - "total_input_tokens": data.get("total_input_tokens", 0), - "total_output_tokens": data.get("total_output_tokens", 0), - "e2e_ms": { - "mean": data.get("mean_e2e_latency_ms", 0.0), - "p50": data.get("median_e2e_latency_ms", 0.0), - "p90": data.get("p90_e2e_latency_ms", 0.0), - "p95": data.get("p95_e2e_latency_ms", 0.0), - "p99": data.get("p99_e2e_latency_ms", 0.0), - }, - "ttft_ms": { - "mean": data.get("mean_ttft_ms", 0.0), - "p50": data.get("median_ttft_ms", 0.0), - "p90": data.get("p90_ttft_ms", 0.0), - "p95": data.get("p95_ttft_ms", 0.0), - "p99": data.get("p99_ttft_ms", 0.0), - }, - "tpot_ms": { - "mean": data.get("mean_tpot_ms", 0.0), - "p50": data.get("median_tpot_ms", 0.0), - "p90": data.get("p90_tpot_ms", 0.0), - "p95": data.get("p95_tpot_ms", 0.0), - "p99": data.get("p99_tpot_ms", 0.0), - }, - "itl_ms": { - "mean": data.get("mean_itl_ms", 0.0), - "p50": data.get("median_itl_ms", 0.0), - "p90": data.get("p90_itl_ms", 0.0), - "p95": data.get("p95_itl_ms", 0.0), - "p99": data.get("p99_itl_ms", 0.0), - }, - } - - -def scenario_name(concurrency: int, input_len: int, output_len: int) -> str: - return f"c{concurrency}_i{input_len}_o{output_len}" - - -def slo_status(metrics: dict) -> dict: - """Check S2-tier SLO: TTFT P95 < 3s, TPOT mean < 50ms.""" - ttft_ok = metrics["ttft_ms"]["p95"] < 3000.0 - tpot_ok = metrics["tpot_ms"]["mean"] < 50.0 - if ttft_ok and tpot_ok: - mark = "✅" - elif ttft_ok or tpot_ok: - mark = "⚠️" - else: - mark = "❌" - return { - "ttft_p95_ok": ttft_ok, - "tpot_mean_ok": tpot_ok, - "overall": mark, - } - - -def append_scenario(results_json: Path, scenario: dict) -> None: - with open(results_json, "r", encoding="utf-8") as f: - data = json.load(f) - data["scenarios"].append(scenario) - with open(results_json, "w", encoding="utf-8") as f: - json.dump(data, f, indent=2, ensure_ascii=False) - - -def generate_report(result_root: Path, backend: str, scenarios: list[dict]) -> None: - report_path = result_root / "report.md" - with open(report_path, "w", encoding="utf-8") as f: - f.write(f"# H200 {backend.upper()} Comparison Benchmark Report\n\n") - f.write(f"- Result root: `{result_root}`\n") - f.write(f"- Model: `/data/models/DeepSeek-V4-Flash`\n") - f.write(f"- Backend: {backend.upper()} (TP=8)\n") - f.write(f"- Benchmark client: `sglang.bench_serving --backend {backend}`\n\n") - - f.write("## Results\n\n") - f.write("| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |\n") - f.write("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n") - - for s in scenarios: - cfg = s["config"] - m = s["metrics"] - slo = s.get("slo_status", {}).get("overall", "") - f.write( - f"| {s['name']} | {cfg['phase']} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | " - f"{m['duration_s']:.2f} | {m['success']} | {m['request_throughput']:.2f} | " - f"{m['input_token_throughput']:.2f} | {m['output_token_throughput']:.2f} | " - f"{m['total_token_throughput']:.2f} | " - f"{m['ttft_ms']['mean']:.2f} | {m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | " - f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | " - f"{m['e2e_ms']['mean']:.2f} | {m['e2e_ms']['p95']:.2f} | {m['e2e_ms']['p99']:.2f} | {slo} |\n" - ) - f.write("\n") - f.write("SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.\n\n") - - -def main() -> None: - parser = argparse.ArgumentParser() - parser.add_argument("result_root", type=Path) - parser.add_argument("--backend", default=None, choices=["sglang", "vllm"]) - args = parser.parse_args() - - result_root = args.result_root - raw_dir = result_root / "raw_outputs" - results_json = result_root / "results.json" - - if not raw_dir.exists(): - raise SystemExit(f"raw_outputs directory not found: {raw_dir}") - - backend = args.backend - if backend is None: - # Infer from filenames if not provided. - for p in raw_dir.iterdir(): - if p.name.startswith("sglang_"): - backend = "sglang" - break - if p.name.startswith("vllm_"): - backend = "vllm" - break - if backend is None: - raise SystemExit("Could not infer backend from raw outputs") - - scenarios = [] - for jsonl_path in sorted(raw_dir.glob(f"{backend}_*.jsonl")): - parts = jsonl_path.stem.split("_") - if len(parts) < 5: - continue - - label = parts[1] - if label == "sharegpt": - phase = "sharegpt" - dataset = "sharegpt" - else: - phase = label - dataset = "random" - - # The last three numeric tokens are: concurrency, input_len/output_ctx, output_len. - try: - concurrency, input_len, output_len = int(parts[-3]), int(parts[-2]), int(parts[-1]) - except ValueError: - continue - - data = parse_jsonl(jsonl_path) - if data is None: - continue - - metrics = compute_metrics(data) - scenario = { - "name": scenario_name(concurrency, input_len, output_len), - "config": { - "phase": phase, - "concurrency": concurrency, - "input_len": input_len, - "output_len": output_len, - "dataset": dataset, - "num_prompts": metrics["success"] + metrics["failed"], - }, - "metrics": metrics, - "slo_status": slo_status(metrics), - "raw_file": str(jsonl_path), - } - scenarios.append(scenario) - - if not scenarios: - print("No benchmark outputs found to parse") - return - - if results_json.exists(): - with open(results_json, "r", encoding="utf-8") as f: - data = json.load(f) - data["scenarios"] = scenarios - with open(results_json, "w", encoding="utf-8") as f: - json.dump(data, f, indent=2, ensure_ascii=False) - - generate_report(result_root, backend, scenarios) - print(f"Parsed {len(scenarios)} scenarios into {result_root}/report.md") - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_256k_4k_probe/results/20260708-094838/comparison.md b/experiments/dsv4_h200_256k_4k_probe/results/20260708-094838/comparison.md index f31e46b..f668e9a 100644 --- a/experiments/dsv4_h200_256k_4k_probe/results/20260708-094838/comparison.md +++ b/experiments/dsv4_h200_256k_4k_probe/results/20260708-094838/comparison.md @@ -17,5 +17,5 @@ ## Notes -- SLO check uses TTFT P95 and TPOT mean (the same criteria as `scripts/SLO_STANDARDS.md`). +- SLO check uses TTFT P95 and TPOT mean (the same criteria as `docs/SLO_STANDARDS.md`). - A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out. diff --git a/experiments/dsv4_h200_256k_4k_probe/run_bench.sh b/experiments/dsv4_h200_256k_4k_probe/run_bench.sh index c05cbc6..5b6b8ef 100644 --- a/experiments/dsv4_h200_256k_4k_probe/run_bench.sh +++ b/experiments/dsv4_h200_256k_4k_probe/run_bench.sh @@ -95,7 +95,7 @@ run_warmup() { num=1 log "warming up ${backend} (input=${input_len}, output=${output_len}, num=${num})" - "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/warmup.py" \ + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/warmup.py" \ --backend "$backend" \ --port "$port" \ --input-len "$input_len" \ @@ -189,7 +189,7 @@ parse_backend() { local backend="$1" local result_root="${RESULT_BASE}/${RUN_ID}/${backend}" log "parsing ${backend} results in ${result_root}" - "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/parse_backend.py" "$result_root" --backend "$backend" \ + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend "$backend" \ >> "${result_root}/logs/parse.log" 2>&1 || { log "WARNING: parser failed for ${backend}; see ${result_root}/logs/parse.log" } @@ -264,7 +264,7 @@ parse_backend vllm # Generate comparison. log "generating comparison report" -"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/compare.py" \ +"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/compare.py" \ --sglang "${RESULT_BASE}/${RUN_ID}/sglang" \ --vllm "${RESULT_BASE}/${RUN_ID}/vllm" \ --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \ diff --git a/experiments/dsv4_h200_64k_sglang_vs_vllm/compare.py b/experiments/dsv4_h200_64k_sglang_vs_vllm/compare.py deleted file mode 100755 index 0f4eb51..0000000 --- a/experiments/dsv4_h200_64k_sglang_vs_vllm/compare.py +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env python3 -"""Generate a side-by-side comparison of SGLang and vLLM results. - -Usage: - python3 compare.py --sglang --vllm \ - [--output comparison.md] -""" -import argparse -import json -from collections import defaultdict -from pathlib import Path - - -def load_result(result_root: Path) -> dict: - path = result_root / "results.json" - with open(path, "r", encoding="utf-8") as f: - return json.load(f) - - -def slo_status(ttft_p95_ms: float, tpot_mean_ms: float) -> str: - # S2 tier targets from scripts/SLO_STANDARDS.md. - ttft_ok = ttft_p95_ms < 3000.0 - tpot_ok = tpot_mean_ms < 50.0 - if ttft_ok and tpot_ok: - return "✅" - if ttft_ok or tpot_ok: - return "⚠️" - return "❌" - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--sglang", type=Path, required=True) - parser.add_argument("--vllm", type=Path, required=True) - parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md")) - args = parser.parse_args() - - sglang_data = load_result(args.sglang) - vllm_data = load_result(args.vllm) - - by_scenario = defaultdict(dict) - for data in (sglang_data, vllm_data): - backend = data["metadata"]["engine"] - for s in data.get("scenarios", []): - key = s["name"] - by_scenario[key][backend] = s - - with open(args.output, "w", encoding="utf-8") as f: - f.write("# SGLang vs vLLM on DeepSeek-V4-Flash (H200, TP=8)\n\n") - f.write("## Summary\n\n") - f.write("- Model: `/data/models/DeepSeek-V4-Flash`\n") - f.write("- Hardware: 8x NVIDIA H200 143GB\n") - f.write("- Tensor Parallelism: 8\n") - f.write("- Benchmark client: `sglang.bench_serving`\n") - f.write("- SLO reference: S2 tier (TTFT P95 < 3s, TPOT < 50ms)\n\n") - - f.write("## Side-by-side results\n\n") - f.write("| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | SLO |\n") - f.write("|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n") - - for scenario_name in sorted(by_scenario.keys()): - for backend in ("sglang", "vllm"): - s = by_scenario[scenario_name].get(backend) - if s is None: - continue - cfg = s["config"] - m = s["metrics"] - status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"]) - f.write( - f"| {scenario_name} | {backend} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | " - f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | " - f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | " - f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | " - f"{m['e2e_ms']['p99']:.2f} | {status} |\n" - ) - - f.write("\n## Notes\n\n") - f.write("- SLO check uses TTFT P95 and TPOT mean (the same criteria as `scripts/SLO_STANDARDS.md`).\n") - f.write("- A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out.\n") - - print(f"Wrote comparison to {args.output}") - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_64k_sglang_vs_vllm/parse_backend.py b/experiments/dsv4_h200_64k_sglang_vs_vllm/parse_backend.py deleted file mode 100755 index 935d870..0000000 --- a/experiments/dsv4_h200_64k_sglang_vs_vllm/parse_backend.py +++ /dev/null @@ -1,219 +0,0 @@ -#!/usr/bin/env python3 -"""Parse raw sglang.bench_serving JSONL outputs for one backend. - -Reads JSONL files like {sglang|vllm}_phase1_MMDD_concurrency_inputlen_outputlen.jsonl -and generates results.json + report.md in the given result root. - -Usage: - python3 parse_backend.py [--backend sglang|vllm] -""" -import argparse -import json -import re -import sys -from pathlib import Path - - -def parse_jsonl(path: Path) -> dict | None: - with open(path, "r", encoding="utf-8") as f: - for line in f: - line = line.strip() - if not line: - continue - try: - return json.loads(line) - except json.JSONDecodeError: - continue - return None - - -def compute_metrics(data: dict) -> dict: - completed = data.get("completed", 0) - total = len(data.get("input_lens", [])) - failed = total - completed if total > 0 else 0 - duration_s = data.get("duration", 0.0) - - return { - "success": completed, - "failed": failed, - "duration_s": duration_s, - "request_throughput": data.get("request_throughput", 0.0), - "input_token_throughput": data.get("input_throughput", 0.0), - "output_token_throughput": data.get("output_throughput", 0.0), - "total_token_throughput": data.get("total_throughput", 0.0), - "total_input_tokens": data.get("total_input_tokens", 0), - "total_output_tokens": data.get("total_output_tokens", 0), - "e2e_ms": { - "mean": data.get("mean_e2e_latency_ms", 0.0), - "p50": data.get("median_e2e_latency_ms", 0.0), - "p90": data.get("p90_e2e_latency_ms", 0.0), - "p95": data.get("p95_e2e_latency_ms", 0.0), - "p99": data.get("p99_e2e_latency_ms", 0.0), - }, - "ttft_ms": { - "mean": data.get("mean_ttft_ms", 0.0), - "p50": data.get("median_ttft_ms", 0.0), - "p90": data.get("p90_ttft_ms", 0.0), - "p95": data.get("p95_ttft_ms", 0.0), - "p99": data.get("p99_ttft_ms", 0.0), - }, - "tpot_ms": { - "mean": data.get("mean_tpot_ms", 0.0), - "p50": data.get("median_tpot_ms", 0.0), - "p90": data.get("p90_tpot_ms", 0.0), - "p95": data.get("p95_tpot_ms", 0.0), - "p99": data.get("p99_tpot_ms", 0.0), - }, - "itl_ms": { - "mean": data.get("mean_itl_ms", 0.0), - "p50": data.get("median_itl_ms", 0.0), - "p90": data.get("p90_itl_ms", 0.0), - "p95": data.get("p95_itl_ms", 0.0), - "p99": data.get("p99_itl_ms", 0.0), - }, - } - - -def scenario_name(concurrency: int, input_len: int, output_len: int) -> str: - return f"c{concurrency}_i{input_len}_o{output_len}" - - -def slo_status(metrics: dict) -> dict: - """Check S2-tier SLO: TTFT P95 < 3s, TPOT mean < 50ms.""" - ttft_ok = metrics["ttft_ms"]["p95"] < 3000.0 - tpot_ok = metrics["tpot_ms"]["mean"] < 50.0 - if ttft_ok and tpot_ok: - mark = "✅" - elif ttft_ok or tpot_ok: - mark = "⚠️" - else: - mark = "❌" - return { - "ttft_p95_ok": ttft_ok, - "tpot_mean_ok": tpot_ok, - "overall": mark, - } - - -def append_scenario(results_json: Path, scenario: dict) -> None: - with open(results_json, "r", encoding="utf-8") as f: - data = json.load(f) - data["scenarios"].append(scenario) - with open(results_json, "w", encoding="utf-8") as f: - json.dump(data, f, indent=2, ensure_ascii=False) - - -def generate_report(result_root: Path, backend: str, scenarios: list[dict]) -> None: - report_path = result_root / "report.md" - with open(report_path, "w", encoding="utf-8") as f: - f.write(f"# H200 {backend.upper()} Comparison Benchmark Report\n\n") - f.write(f"- Result root: `{result_root}`\n") - f.write(f"- Model: `/data/models/DeepSeek-V4-Flash`\n") - f.write(f"- Backend: {backend.upper()} (TP=8)\n") - f.write(f"- Benchmark client: `sglang.bench_serving --backend {backend}`\n\n") - - f.write("## Results\n\n") - f.write("| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |\n") - f.write("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n") - - for s in scenarios: - cfg = s["config"] - m = s["metrics"] - slo = s.get("slo_status", {}).get("overall", "") - f.write( - f"| {s['name']} | {cfg['phase']} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | " - f"{m['duration_s']:.2f} | {m['success']} | {m['request_throughput']:.2f} | " - f"{m['input_token_throughput']:.2f} | {m['output_token_throughput']:.2f} | " - f"{m['total_token_throughput']:.2f} | " - f"{m['ttft_ms']['mean']:.2f} | {m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | " - f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | " - f"{m['e2e_ms']['mean']:.2f} | {m['e2e_ms']['p95']:.2f} | {m['e2e_ms']['p99']:.2f} | {slo} |\n" - ) - f.write("\n") - f.write("SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.\n\n") - - -def main() -> None: - parser = argparse.ArgumentParser() - parser.add_argument("result_root", type=Path) - parser.add_argument("--backend", default=None, choices=["sglang", "vllm"]) - args = parser.parse_args() - - result_root = args.result_root - raw_dir = result_root / "raw_outputs" - results_json = result_root / "results.json" - - if not raw_dir.exists(): - raise SystemExit(f"raw_outputs directory not found: {raw_dir}") - - backend = args.backend - if backend is None: - # Infer from filenames if not provided. - for p in raw_dir.iterdir(): - if p.name.startswith("sglang_"): - backend = "sglang" - break - if p.name.startswith("vllm_"): - backend = "vllm" - break - if backend is None: - raise SystemExit("Could not infer backend from raw outputs") - - scenarios = [] - for jsonl_path in sorted(raw_dir.glob(f"{backend}_*.jsonl")): - parts = jsonl_path.stem.split("_") - if len(parts) < 5: - continue - - label = parts[1] - if label == "sharegpt": - phase = "sharegpt" - dataset = "sharegpt" - else: - phase = label - dataset = "random" - - # The last three numeric tokens are: concurrency, input_len/output_ctx, output_len. - try: - concurrency, input_len, output_len = int(parts[-3]), int(parts[-2]), int(parts[-1]) - except ValueError: - continue - - data = parse_jsonl(jsonl_path) - if data is None: - continue - - metrics = compute_metrics(data) - scenario = { - "name": scenario_name(concurrency, input_len, output_len), - "config": { - "phase": phase, - "concurrency": concurrency, - "input_len": input_len, - "output_len": output_len, - "dataset": dataset, - "num_prompts": metrics["success"] + metrics["failed"], - }, - "metrics": metrics, - "slo_status": slo_status(metrics), - "raw_file": str(jsonl_path), - } - scenarios.append(scenario) - - if not scenarios: - print("No benchmark outputs found to parse") - return - - if results_json.exists(): - with open(results_json, "r", encoding="utf-8") as f: - data = json.load(f) - data["scenarios"] = scenarios - with open(results_json, "w", encoding="utf-8") as f: - json.dump(data, f, indent=2, ensure_ascii=False) - - generate_report(result_root, backend, scenarios) - print(f"Parsed {len(scenarios)} scenarios into {result_root}/report.md") - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_64k_sglang_vs_vllm/results/20260708-092556/comparison.md b/experiments/dsv4_h200_64k_sglang_vs_vllm/results/20260708-092556/comparison.md index e73a8ce..00bdd81 100644 --- a/experiments/dsv4_h200_64k_sglang_vs_vllm/results/20260708-092556/comparison.md +++ b/experiments/dsv4_h200_64k_sglang_vs_vllm/results/20260708-092556/comparison.md @@ -19,5 +19,5 @@ ## Notes -- SLO check uses TTFT P95 and TPOT mean (the same criteria as `scripts/SLO_STANDARDS.md`). +- SLO check uses TTFT P95 and TPOT mean (the same criteria as `docs/SLO_STANDARDS.md`). - A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out. diff --git a/experiments/dsv4_h200_64k_sglang_vs_vllm/run_bench.sh b/experiments/dsv4_h200_64k_sglang_vs_vllm/run_bench.sh index eaf41b5..532b5fb 100644 --- a/experiments/dsv4_h200_64k_sglang_vs_vllm/run_bench.sh +++ b/experiments/dsv4_h200_64k_sglang_vs_vllm/run_bench.sh @@ -95,7 +95,7 @@ run_warmup() { num=1 log "warming up ${backend} (input=${input_len}, output=${output_len}, num=${num})" - "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/warmup.py" \ + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/warmup.py" \ --backend "$backend" \ --port "$port" \ --input-len "$input_len" \ @@ -189,7 +189,7 @@ parse_backend() { local backend="$1" local result_root="${RESULT_BASE}/${RUN_ID}/${backend}" log "parsing ${backend} results in ${result_root}" - "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/parse_backend.py" "$result_root" --backend "$backend" \ + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend "$backend" \ >> "${result_root}/logs/parse.log" 2>&1 || { log "WARNING: parser failed for ${backend}; see ${result_root}/logs/parse.log" } @@ -264,7 +264,7 @@ parse_backend vllm # Generate comparison. log "generating comparison report" -"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/compare.py" \ +"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/compare.py" \ --sglang "${RESULT_BASE}/${RUN_ID}/sglang" \ --vllm "${RESULT_BASE}/${RUN_ID}/vllm" \ --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \ diff --git a/experiments/dsv4_h200_64k_sglang_vs_vllm/warmup.py b/experiments/dsv4_h200_64k_sglang_vs_vllm/warmup.py deleted file mode 100755 index 75dbe5f..0000000 --- a/experiments/dsv4_h200_64k_sglang_vs_vllm/warmup.py +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env python3 -"""Send a small number of warmup requests to a running backend. - -Uses sglang.bench_serving with a single request so that the same code path -(prefill / decode kernels, CUDA graphs, etc.) is exercised before the real -benchmark begins. Discards the output. -""" -import argparse -import subprocess -import sys -import tempfile -from pathlib import Path - - -def run_warmup(backend: str, host: str, port: int, input_len: int, output_len: int, num: int, env_python: Path) -> None: - with tempfile.NamedTemporaryFile(mode="w", suffix=".jsonl", delete=True) as tmp: - cmd = [ - str(env_python), - "-m", - "sglang.bench_serving", - "--backend", - backend, - "--host", - host, - "--port", - str(port), - "--dataset-name", - "random", - "--random-input-len", - str(input_len), - "--random-output-len", - str(output_len), - "--num-prompts", - str(num), - "--max-concurrency", - "1", - "--request-rate", - "10000", - "--output-file", - tmp.name, - "--output-details", - ] - print(f"[warmup] {' '.join(cmd)}", flush=True) - result = subprocess.run(cmd, capture_output=True, text=True) - if result.returncode != 0: - print("[warmup] FAILED", file=sys.stderr) - print(result.stdout, file=sys.stderr) - print(result.stderr, file=sys.stderr) - sys.exit(1) - print(f"[warmup] OK: backend={backend} port={port} input={input_len} output={output_len} num={num}") - - -def main(): - parser = argparse.ArgumentParser(description="Warmup a serving backend.") - parser.add_argument("--backend", required=True, choices=["sglang", "vllm"]) - parser.add_argument("--port", type=int, required=True) - parser.add_argument("--input-len", type=int, required=True) - parser.add_argument("--output-len", type=int, required=True) - parser.add_argument("--num", type=int, default=1) - parser.add_argument("--host", default="127.0.0.1") - parser.add_argument("--env-python", default="/data/user1/yy/envs/sglang/bin/python") - args = parser.parse_args() - - run_warmup( - backend=args.backend, - host=args.host, - port=args.port, - input_len=args.input_len, - output_len=args.output_len, - num=args.num, - env_python=Path(args.env_python), - ) - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_long_context_matrix/compare.py b/experiments/dsv4_h200_long_context_matrix/compare.py deleted file mode 100755 index 0f4eb51..0000000 --- a/experiments/dsv4_h200_long_context_matrix/compare.py +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env python3 -"""Generate a side-by-side comparison of SGLang and vLLM results. - -Usage: - python3 compare.py --sglang --vllm \ - [--output comparison.md] -""" -import argparse -import json -from collections import defaultdict -from pathlib import Path - - -def load_result(result_root: Path) -> dict: - path = result_root / "results.json" - with open(path, "r", encoding="utf-8") as f: - return json.load(f) - - -def slo_status(ttft_p95_ms: float, tpot_mean_ms: float) -> str: - # S2 tier targets from scripts/SLO_STANDARDS.md. - ttft_ok = ttft_p95_ms < 3000.0 - tpot_ok = tpot_mean_ms < 50.0 - if ttft_ok and tpot_ok: - return "✅" - if ttft_ok or tpot_ok: - return "⚠️" - return "❌" - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--sglang", type=Path, required=True) - parser.add_argument("--vllm", type=Path, required=True) - parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md")) - args = parser.parse_args() - - sglang_data = load_result(args.sglang) - vllm_data = load_result(args.vllm) - - by_scenario = defaultdict(dict) - for data in (sglang_data, vllm_data): - backend = data["metadata"]["engine"] - for s in data.get("scenarios", []): - key = s["name"] - by_scenario[key][backend] = s - - with open(args.output, "w", encoding="utf-8") as f: - f.write("# SGLang vs vLLM on DeepSeek-V4-Flash (H200, TP=8)\n\n") - f.write("## Summary\n\n") - f.write("- Model: `/data/models/DeepSeek-V4-Flash`\n") - f.write("- Hardware: 8x NVIDIA H200 143GB\n") - f.write("- Tensor Parallelism: 8\n") - f.write("- Benchmark client: `sglang.bench_serving`\n") - f.write("- SLO reference: S2 tier (TTFT P95 < 3s, TPOT < 50ms)\n\n") - - f.write("## Side-by-side results\n\n") - f.write("| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | SLO |\n") - f.write("|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n") - - for scenario_name in sorted(by_scenario.keys()): - for backend in ("sglang", "vllm"): - s = by_scenario[scenario_name].get(backend) - if s is None: - continue - cfg = s["config"] - m = s["metrics"] - status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"]) - f.write( - f"| {scenario_name} | {backend} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | " - f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | " - f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | " - f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | " - f"{m['e2e_ms']['p99']:.2f} | {status} |\n" - ) - - f.write("\n## Notes\n\n") - f.write("- SLO check uses TTFT P95 and TPOT mean (the same criteria as `scripts/SLO_STANDARDS.md`).\n") - f.write("- A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out.\n") - - print(f"Wrote comparison to {args.output}") - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/comparison.md b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/comparison.md new file mode 100644 index 0000000..89ef577 --- /dev/null +++ b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/comparison.md @@ -0,0 +1,61 @@ +# SGLang vs vLLM on DeepSeek-V4-Flash (H200, TP=8) + +## Summary + +- Model: `/data/models/DeepSeek-V4-Flash` +- Hardware: 8x NVIDIA H200 143GB +- Tensor Parallelism: 8 +- Benchmark client: `sglang.bench_serving` +- SLO reference: S2 tier (TTFT P95 < 3s, TPOT < 50ms) + +## Side-by-side results + +| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | SLO | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| c10_i131072_o1024 | sglang | 10 | 131072 | 1024 | 0.68 | 319.74 | 4147.51 | 7409.82 | 25.69 | 49.89 | 58.92 | 26432.66 | ⚠️ | +| c10_i131072_o1024 | vllm | 10 | 131072 | 1024 | 0.39 | 183.08 | 6596.15 | 6603.12 | 47.10 | 77.53 | 84.06 | 48426.64 | ⚠️ | +| c10_i131072_o256 | sglang | 10 | 131072 | 256 | 0.34 | 42.70 | 26046.62 | 26064.38 | 1374.50 | 1810.56 | 19553.62 | 58326.41 | ❌ | +| c10_i131072_o256 | vllm | 10 | 131072 | 256 | 0.38 | 48.09 | 23250.03 | 24238.33 | 150.22 | 299.91 | 301.72 | 51439.08 | ❌ | +| c10_i131072_o4096 | sglang | 10 | 131072 | 4096 | 0.27 | 517.96 | 6177.42 | 6920.28 | 14.21 | 19.92 | 25.87 | 54286.63 | ⚠️ | +| c10_i131072_o4096 | vllm | 10 | 131072 | 4096 | 0.23 | 445.81 | 2096.29 | 2168.81 | 18.57 | 21.47 | 21.61 | 67041.13 | ✅ | +| c10_i262144_o1024 | sglang | 10 | 262144 | 1024 | 0.16 | 76.30 | 39446.96 | 43965.67 | 108.42 | 202.43 | 202.90 | 120180.87 | ❌ | +| c10_i262144_o1024 | vllm | 10 | 262144 | 1024 | 0.18 | 84.63 | 34130.22 | 38144.40 | 98.87 | 142.57 | 266.08 | 108658.89 | ❌ | +| c10_i262144_o256 | sglang | 10 | 262144 | 256 | 0.15 | 18.70 | 58074.31 | 62652.59 | 2953.70 | 3625.59 | 41185.32 | 134628.33 | ❌ | +| c10_i262144_o256 | vllm | 10 | 262144 | 256 | 0.18 | 22.76 | 50632.44 | 51883.02 | 229.20 | 339.27 | 344.95 | 110216.92 | ❌ | +| c10_i262144_o4096 | sglang | 10 | 262144 | 4096 | 0.14 | 272.46 | 18088.12 | 22579.09 | 28.00 | 43.87 | 45.07 | 120183.75 | ⚠️ | +| c10_i262144_o4096 | vllm | 10 | 262144 | 4096 | 0.11 | 215.86 | 34100.32 | 38022.94 | 37.18 | 56.07 | 62.00 | 157944.21 | ⚠️ | +| c25_i65536_o1024 | sglang | 25 | 65536 | 1024 | 0.67 | 346.70 | 19996.43 | 24478.11 | 149.27 | 395.00 | 2037.09 | 69586.94 | ❌ | +| c25_i65536_o1024 | vllm | 25 | 65536 | 1024 | 0.64 | 332.61 | 24861.20 | 28942.75 | 72.36 | 196.24 | 271.32 | 72689.84 | ❌ | +| c25_i65536_o256 | sglang | 25 | 65536 | 256 | 0.71 | 84.45 | 27219.06 | 31612.70 | 292.84 | 801.18 | 938.43 | 68854.26 | ❌ | +| c25_i65536_o256 | vllm | 25 | 65536 | 256 | 0.81 | 95.60 | 24902.72 | 28974.45 | 180.78 | 273.59 | 278.16 | 60781.45 | ❌ | +| c25_i65536_o4096 | sglang | 25 | 65536 | 4096 | 0.41 | 950.01 | 18559.58 | 23249.71 | 23.78 | 42.50 | 82.84 | 96670.12 | ⚠️ | +| c25_i65536_o4096 | vllm | 25 | 65536 | 4096 | 0.32 | 741.71 | 24847.97 | 28840.96 | 28.80 | 46.66 | 63.40 | 129654.04 | ⚠️ | +| c2_i131072_o1024 | sglang | 2 | 131072 | 1024 | 0.60 | 136.10 | 479.99 | 482.72 | 8.56 | 9.72 | 9.79 | 5480.98 | ✅ | +| c2_i131072_o1024 | vllm | 2 | 131072 | 1024 | 0.70 | 158.36 | 676.14 | 695.90 | 6.49 | 8.20 | 8.34 | 4803.24 | ✅ | +| c2_i131072_o256 | sglang | 2 | 131072 | 256 | 1.43 | 139.48 | 589.14 | 605.33 | 8.29 | 8.96 | 9.05 | 1750.12 | ✅ | +| c2_i131072_o256 | vllm | 2 | 131072 | 256 | 1.51 | 147.12 | 684.63 | 700.28 | 7.08 | 9.27 | 9.45 | 1556.46 | ✅ | +| c2_i131072_o4096 | sglang | 2 | 131072 | 4096 | 0.11 | 244.38 | 584.43 | 599.22 | 7.53 | 7.61 | 7.62 | 28360.35 | ✅ | +| c2_i131072_o4096 | vllm | 2 | 131072 | 4096 | 0.12 | 262.22 | 671.34 | 688.15 | 7.09 | 7.17 | 7.18 | 26496.04 | ✅ | +| c2_i262144_o1024 | sglang | 2 | 262144 | 1024 | 0.51 | 114.94 | 1007.07 | 1024.10 | 8.36 | 9.03 | 9.06 | 6240.65 | ✅ | +| c2_i262144_o1024 | vllm | 2 | 262144 | 1024 | 0.19 | 43.14 | 8722.76 | 9009.70 | 20.17 | 52.24 | 57.91 | 13826.13 | ⚠️ | +| c2_i262144_o256 | sglang | 2 | 262144 | 256 | 0.29 | 28.44 | 9538.91 | 9980.78 | 14.10 | 26.98 | 29.16 | 11941.34 | ⚠️ | +| c2_i262144_o256 | vllm | 2 | 262144 | 256 | 0.23 | 22.74 | 8625.18 | 8952.31 | 10.18 | 23.01 | 25.44 | 10378.60 | ⚠️ | +| c2_i262144_o4096 | sglang | 2 | 262144 | 4096 | 0.08 | 189.78 | 9453.57 | 9896.38 | 7.82 | 8.48 | 8.60 | 38723.14 | ⚠️ | +| c2_i262144_o4096 | vllm | 2 | 262144 | 4096 | 0.08 | 180.76 | 8759.63 | 9062.80 | 8.63 | 11.53 | 12.01 | 35881.75 | ⚠️ | +| c5_i131072_o1024 | sglang | 5 | 131072 | 1024 | 0.87 | 321.54 | 1231.23 | 1298.36 | 9.03 | 10.47 | 10.71 | 7298.25 | ✅ | +| c5_i131072_o1024 | vllm | 5 | 131072 | 1024 | 0.90 | 331.65 | 856.18 | 917.80 | 8.93 | 10.48 | 10.62 | 6989.27 | ✅ | +| c5_i131072_o256 | sglang | 5 | 131072 | 256 | 1.05 | 146.97 | 3270.23 | 4685.65 | 19.91 | 40.81 | 46.10 | 7815.89 | ⚠️ | +| c5_i131072_o256 | vllm | 5 | 131072 | 256 | 0.39 | 54.07 | 9384.23 | 9677.91 | 51.54 | 107.16 | 117.25 | 19860.69 | ❌ | +| c5_i131072_o4096 | sglang | 5 | 131072 | 4096 | 0.20 | 378.76 | 4667.56 | 6275.51 | 11.98 | 22.07 | 23.60 | 31176.81 | ⚠️ | +| c5_i131072_o4096 | vllm | 5 | 131072 | 4096 | 0.23 | 447.85 | 1054.01 | 1104.66 | 8.48 | 9.13 | 9.14 | 28818.20 | ✅ | +| c5_i262144_o1024 | sglang | 5 | 262144 | 1024 | 0.15 | 56.42 | 20522.02 | 25944.29 | 54.11 | 118.98 | 124.02 | 53444.22 | ❌ | +| c5_i262144_o1024 | vllm | 5 | 262144 | 1024 | 0.16 | 60.32 | 22564.89 | 26639.92 | 47.49 | 100.69 | 109.86 | 53790.59 | ⚠️ | +| c5_i262144_o256 | sglang | 5 | 262144 | 256 | 0.16 | 21.94 | 25875.40 | 30596.95 | 97.05 | 231.75 | 252.98 | 43756.53 | ❌ | +| c5_i262144_o256 | vllm | 5 | 262144 | 256 | 0.19 | 25.85 | 21986.99 | 26381.86 | 123.42 | 202.78 | 215.56 | 45761.90 | ❌ | +| c5_i262144_o4096 | sglang | 5 | 262144 | 4096 | 0.11 | 209.50 | 16539.50 | 20323.40 | 25.05 | 56.69 | 60.22 | 60383.57 | ⚠️ | +| c5_i262144_o4096 | vllm | 5 | 262144 | 4096 | 0.11 | 205.12 | 22171.78 | 26533.69 | 25.69 | 63.32 | 68.33 | 64086.27 | ⚠️ | + +## Notes + +- SLO check uses TTFT P95 and TPOT mean (the same criteria as `docs/SLO_STANDARDS.md`). +- A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out. diff --git a/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/report.md b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/report.md new file mode 100644 index 0000000..452fe6c --- /dev/null +++ b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/report.md @@ -0,0 +1,35 @@ +# H200 SGLANG Comparison Benchmark Report + +- Result root: `/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang` +- Model: `/data/models/DeepSeek-V4-Flash` +- Backend: SGLANG (TP=8) +- Benchmark client: `sglang.bench_serving --backend sglang` + +## Results + +| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| c10_i131072_o1024 | 128k | 10 | 131072 | 1024 | 29.58 | 20 | 0.68 | 49838.58 | 319.74 | 50158.31 | 1968.70 | 4147.51 | 7409.82 | 25.69 | 49.89 | 58.92 | 13268.98 | 25531.40 | 26432.66 | ⚠️ | +| c10_i131072_o256 | 128k | 10 | 131072 | 256 | 59.65 | 20 | 0.34 | 24716.54 | 42.70 | 24759.24 | 9130.87 | 26046.62 | 26064.38 | 1374.50 | 1810.56 | 19553.62 | 29314.12 | 58034.97 | 58326.41 | ❌ | +| c10_i131072_o4096 | 128k | 10 | 131072 | 4096 | 73.62 | 20 | 0.27 | 20027.92 | 517.96 | 20545.89 | 2624.32 | 6177.42 | 6920.28 | 14.21 | 19.92 | 25.87 | 28500.68 | 53466.72 | 54286.63 | ⚠️ | +| c2_i131072_o1024 | 128k | 2 | 131072 | 1024 | 6.62 | 4 | 0.60 | 36602.18 | 136.10 | 36738.28 | 363.59 | 479.99 | 482.72 | 8.56 | 9.72 | 9.79 | 2166.16 | 4983.98 | 5480.98 | ✅ | +| c2_i131072_o256 | 128k | 2 | 131072 | 256 | 2.79 | 4 | 1.43 | 86882.00 | 139.48 | 87021.48 | 423.87 | 589.14 | 605.33 | 8.29 | 8.96 | 9.05 | 1238.88 | 1727.94 | 1750.12 | ✅ | +| c2_i131072_o4096 | 128k | 2 | 131072 | 4096 | 37.21 | 4 | 0.11 | 6512.27 | 244.38 | 6756.65 | 417.65 | 584.43 | 599.22 | 7.53 | 7.61 | 7.62 | 17481.60 | 27763.29 | 28360.35 | ✅ | +| c5_i131072_o1024 | 128k | 5 | 131072 | 1024 | 11.51 | 10 | 0.87 | 64181.45 | 321.54 | 64502.99 | 747.50 | 1231.23 | 1298.36 | 9.03 | 10.47 | 10.71 | 4011.50 | 6687.19 | 7298.25 | ✅ | +| c5_i131072_o256 | 128k | 5 | 131072 | 256 | 9.50 | 10 | 1.05 | 77752.11 | 146.97 | 77899.07 | 1222.22 | 3270.23 | 4685.65 | 19.91 | 40.81 | 46.10 | 4366.18 | 7814.85 | 7815.89 | ⚠️ | +| c5_i131072_o4096 | 128k | 5 | 131072 | 4096 | 50.32 | 10 | 0.20 | 14676.36 | 378.76 | 15055.12 | 1452.15 | 4667.56 | 6275.51 | 11.98 | 22.07 | 23.60 | 19785.61 | 30849.59 | 31176.81 | ⚠️ | +| c10_i262144_o1024 | 256k | 10 | 262144 | 1024 | 123.96 | 20 | 0.16 | 22467.00 | 76.30 | 22543.30 | 12461.07 | 39446.96 | 43965.67 | 108.42 | 202.43 | 202.90 | 60222.02 | 119241.30 | 120180.87 | ❌ | +| c10_i262144_o256 | 256k | 10 | 262144 | 256 | 136.21 | 20 | 0.15 | 20447.06 | 18.70 | 20465.76 | 21481.91 | 58074.31 | 62652.59 | 2953.70 | 3625.59 | 41185.32 | 67539.67 | 134322.68 | 134628.33 | ❌ | +| c10_i262144_o4096 | 256k | 10 | 262144 | 4096 | 139.95 | 20 | 0.14 | 19900.66 | 272.46 | 20173.12 | 8041.92 | 18088.12 | 22579.09 | 28.00 | 43.87 | 45.07 | 61533.12 | 119183.37 | 120183.75 | ⚠️ | +| c2_i262144_o1024 | 256k | 2 | 262144 | 1024 | 7.84 | 4 | 0.51 | 64353.10 | 114.94 | 64468.04 | 817.04 | 1007.07 | 1024.10 | 8.36 | 9.03 | 9.06 | 2623.61 | 5714.71 | 6240.65 | ✅ | +| c2_i262144_o256 | 256k | 2 | 262144 | 256 | 13.68 | 4 | 0.29 | 36885.02 | 28.44 | 36913.47 | 4623.18 | 9538.91 | 9980.78 | 14.10 | 26.98 | 29.16 | 6497.61 | 11731.18 | 11941.34 | ⚠️ | +| c2_i262144_o4096 | 256k | 2 | 262144 | 4096 | 47.91 | 4 | 0.08 | 10528.36 | 189.78 | 10718.13 | 4475.72 | 9453.57 | 9896.38 | 7.82 | 8.48 | 8.60 | 22625.15 | 38039.50 | 38723.14 | ⚠️ | +| c5_i262144_o1024 | 256k | 5 | 262144 | 1024 | 65.58 | 10 | 0.15 | 21254.00 | 56.42 | 21310.42 | 9553.36 | 20522.02 | 25944.29 | 54.11 | 118.98 | 124.02 | 29705.24 | 53300.68 | 53444.22 | ❌ | +| c5_i262144_o256 | 256k | 5 | 262144 | 256 | 63.62 | 10 | 0.16 | 21909.37 | 21.94 | 21931.32 | 12937.52 | 25875.40 | 30596.95 | 97.05 | 231.75 | 252.98 | 27766.49 | 43745.44 | 43756.53 | ❌ | +| c5_i262144_o4096 | 256k | 5 | 262144 | 4096 | 90.98 | 10 | 0.11 | 15321.51 | 209.50 | 15531.02 | 6888.18 | 16539.50 | 20323.40 | 25.05 | 56.69 | 60.22 | 40104.55 | 58680.97 | 60383.57 | ⚠️ | +| c25_i65536_o1024 | 64k | 25 | 65536 | 1024 | 74.65 | 50 | 0.67 | 24668.22 | 346.70 | 25014.92 | 6120.96 | 19996.43 | 24478.11 | 149.27 | 395.00 | 2037.09 | 34945.80 | 68400.96 | 69586.94 | ❌ | +| c25_i65536_o256 | 64k | 25 | 65536 | 256 | 70.02 | 50 | 0.71 | 26299.26 | 84.45 | 26383.70 | 9330.86 | 27219.06 | 31612.70 | 292.84 | 801.18 | 938.43 | 34392.33 | 68802.89 | 68854.26 | ❌ | +| c25_i65536_o4096 | 64k | 25 | 65536 | 4096 | 122.10 | 50 | 0.41 | 15081.87 | 950.01 | 16031.88 | 5499.83 | 18559.58 | 23249.71 | 23.78 | 42.50 | 82.84 | 52152.37 | 93852.74 | 96670.12 | ⚠️ | + +SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail. + diff --git a/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/results.json b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/results.json new file mode 100644 index 0000000..0f3ed10 --- /dev/null +++ b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/results.json @@ -0,0 +1,1219 @@ +{ + "metadata": { + "experiment": "dsv4_h200_long_context_matrix_sglang", + "run_id": "20260708-100016", + "timestamp": "2026-07-08T10:00:20+00:00", + "model": "/data/models/DeepSeek-V4-Flash", + "backend": "sglang", + "engine": "sglang", + "hardware": "8x NVIDIA H200 143GB", + "accelerator": "NVIDIA H200", + "chip": "nvidia_h200", + "script": "experiments/dsv4_h200_long_context_matrix/run_bench.sh", + "env": "/data/user1/yy/envs/sglang", + "git_commit": "7e35945", + "git_dirty": "dirty", + "description": "H200 long-context matrix sglang TP=8 for DeepSeek-V4-Flash" + }, + "config": { + "tp": 8, + "cuda_visible_devices": "0,1,2,3,4,5,6,7", + "backend": "sglang", + "server_start_script": "experiments/dsv4_h200_long_context_matrix/start_sglang.sh", + "groups": [ + { + "label": "64k", + "max_model_len": 80000, + "max_slots": 25 + }, + { + "label": "128k", + "max_model_len": 140000, + "max_slots": 10 + }, + { + "label": "256k", + "max_model_len": 270000, + "max_slots": 10 + } + ] + }, + "scenarios": [ + { + "name": "c10_i131072_o1024", + "config": { + "phase": "128k", + "concurrency": 10, + "input_len": 131072, + "output_len": 1024, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 29.583529918003478, + "request_throughput": 0.6760518455854964, + "input_token_throughput": 49838.57585915508, + "output_token_throughput": 319.73872036966054, + "total_token_throughput": 50158.31457952473, + "total_input_tokens": 1474401, + "total_output_tokens": 9459, + "e2e_ms": { + "mean": 13268.980272101908, + "p50": 10510.840059003385, + "p90": 24816.737577588356, + "p95": 25531.396126995245, + "p99": 26432.655442199466 + }, + "ttft_ms": { + "mean": 1968.6975458011148, + "p50": 1759.2950304970145, + "p90": 3613.020290805435, + "p95": 4147.509501551397, + "p99": 7409.8210011149 + }, + "tpot_ms": { + "mean": 25.693756340936623, + "p50": 24.88176984022087, + "p90": 39.529624163622046, + "p95": 49.887355328486585, + "p99": 58.924154640674516 + }, + "itl_ms": { + "mean": 23.943748617228504, + "p50": 8.454066002741456, + "p90": 8.788931590970606, + "p95": 8.975919598015025, + "p99": 233.5498191238731 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_10_131072_1024.jsonl" + }, + { + "name": "c10_i131072_o256", + "config": { + "phase": "128k", + "concurrency": 10, + "input_len": 131072, + "output_len": 256, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 59.65240250399802, + "request_throughput": 0.335275683132118, + "input_token_throughput": 24716.540124283893, + "output_token_throughput": 42.697358246875226, + "total_token_throughput": 24759.237482530767, + "total_input_tokens": 1474401, + "total_output_tokens": 2547, + "e2e_ms": { + "mean": 29314.11689295128, + "p50": 26262.799380499928, + "p90": 57981.770187895745, + "p95": 58034.969827709574, + "p99": 58326.41254074915 + }, + "ttft_ms": { + "mean": 9130.866601550952, + "p50": 5061.937274498632, + "p90": 24214.829088801343, + "p95": 26046.623209959216, + "p99": 26064.37579079109 + }, + "tpot_ms": { + "mean": 1374.5048535422816, + "p50": 169.0106426934659, + "p90": 430.81965476299, + "p95": 1810.5569532731727, + "p99": 19553.62188185579 + }, + "itl_ms": { + "mean": 159.74055988326018, + "p50": 7.61685699399095, + "p90": 9.68485160265118, + "p95": 145.2174844002002, + "p99": 4701.002810652135 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_10_131072_256.jsonl" + }, + { + "name": "c10_i131072_o4096", + "config": { + "phase": "128k", + "concurrency": 10, + "input_len": 131072, + "output_len": 4096, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 73.61727057601092, + "request_throughput": 0.2716753805664352, + "input_token_throughput": 20027.92263912663, + "output_token_throughput": 517.962696818937, + "total_token_throughput": 20545.885335945568, + "total_input_tokens": 1474401, + "total_output_tokens": 38131, + "e2e_ms": { + "mean": 28500.67895909815, + "p50": 28112.17715300154, + "p90": 49829.967515188044, + "p95": 53466.72197574008, + "p99": 54286.62850075067 + }, + "ttft_ms": { + "mean": 2624.3248954968294, + "p50": 2136.155736996443, + "p90": 6003.281899908325, + "p95": 6177.423165745859, + "p99": 6920.282273945922 + }, + "tpot_ms": { + "mean": 14.213841209315783, + "p50": 13.928366367431902, + "p90": 19.11311534143368, + "p95": 19.916946141598892, + "p99": 25.871121799909396 + }, + "itl_ms": { + "mean": 13.579451054235527, + "p50": 8.536182998795994, + "p90": 8.77653200586792, + "p95": 8.853724495565984, + "p99": 14.320646900160352 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_10_131072_4096.jsonl" + }, + { + "name": "c2_i131072_o1024", + "config": { + "phase": "128k", + "concurrency": 2, + "input_len": 131072, + "output_len": 1024, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 6.6201243749965215, + "request_throughput": 0.6042182553408751, + "input_token_throughput": 36602.1824174757, + "output_token_throughput": 136.10016201553213, + "total_token_throughput": 36738.282579491235, + "total_input_tokens": 242311, + "total_output_tokens": 901, + "e2e_ms": { + "mean": 2166.1627940011385, + "p50": 1224.2758355059777, + "p90": 4362.741659799941, + "p95": 4983.983836899277, + "p99": 5480.977578578749 + }, + "ttft_ms": { + "mean": 363.5929425036011, + "p50": 374.64400700991973, + "p90": 476.5860987958149, + "p95": 479.9941893936193, + "p99": 482.7206618718628 + }, + "tpot_ms": { + "mean": 8.56378605699, + "p50": 8.469017351992196, + "p90": 9.6394401799744, + "p95": 9.723479477276998, + "p99": 9.790710915119075 + }, + "itl_ms": { + "mean": 8.038057003330794, + "p50": 7.442517002345994, + "p90": 7.628289199783467, + "p95": 7.714029011549428, + "p99": 15.245293956249952 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_2_131072_1024.jsonl" + }, + { + "name": "c2_i131072_o256", + "config": { + "phase": "128k", + "concurrency": 2, + "input_len": 131072, + "output_len": 256, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 2.78896649699891, + "request_throughput": 1.434223037209028, + "input_token_throughput": 86882.0045922892, + "output_token_throughput": 139.47819036857797, + "total_token_throughput": 87021.48278265777, + "total_input_tokens": 242311, + "total_output_tokens": 389, + "e2e_ms": { + "mean": 1238.8818739927956, + "p50": 1287.7310694893822, + "p90": 1700.2253821905472, + "p95": 1727.9445655905874, + "p99": 1750.1199123106196 + }, + "ttft_ms": { + "mean": 423.87411399613484, + "p50": 443.92389149288647, + "p90": 568.8948895971407, + "p95": 589.1363902985177, + "p99": 605.3295908596191 + }, + "tpot_ms": { + "mean": 8.294876356890017, + "p50": 8.305240008359462, + "p90": 8.8474133241449, + "p95": 8.959042992216144, + "p99": 9.048346726673142 + }, + "itl_ms": { + "mean": 8.46726296366761, + "p50": 7.438301996444352, + "p90": 7.678568997653201, + "p95": 7.913634003489277, + "p99": 38.00833222979695 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_2_131072_256.jsonl" + }, + { + "name": "c2_i131072_o4096", + "config": { + "phase": "128k", + "concurrency": 2, + "input_len": 131072, + "output_len": 4096, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 37.20838306899532, + "request_throughput": 0.10750265585534367, + "input_token_throughput": 6512.269010741045, + "output_token_throughput": 244.38041242316, + "total_token_throughput": 6756.6494231642055, + "total_input_tokens": 242311, + "total_output_tokens": 9093, + "e2e_ms": { + "mean": 17481.596304744016, + "p50": 16374.755681492388, + "p90": 27016.964761391864, + "p95": 27763.290148692613, + "p99": 28360.350458533212 + }, + "ttft_ms": { + "mean": 417.6458147485391, + "p50": 440.9555434976937, + "p90": 565.9416574999341, + "p95": 584.4300492499315, + "p99": 599.2207626499294 + }, + "tpot_ms": { + "mean": 7.533689034138554, + "p50": 7.528465619569621, + "p90": 7.59955080906869, + "p95": 7.610607248780574, + "p99": 7.619452400550082 + }, + "itl_ms": { + "mean": 7.509697526022953, + "p50": 7.4637350044213235, + "p90": 7.645775395212695, + "p95": 7.709391202661209, + "p99": 7.858362724073233 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_2_131072_4096.jsonl" + }, + { + "name": "c5_i131072_o1024", + "config": { + "phase": "128k", + "concurrency": 5, + "input_len": 131072, + "output_len": 1024, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 11.507078130991431, + "request_throughput": 0.8690303382113576, + "input_token_throughput": 64181.45350129543, + "output_token_throughput": 321.5412251382023, + "total_token_throughput": 64502.99472643363, + "total_input_tokens": 738541, + "total_output_tokens": 3700, + "e2e_ms": { + "mean": 4011.5025283041177, + "p50": 4134.981311006413, + "p90": 5923.378200098522, + "p95": 6687.19352155458, + "p99": 7298.245778719429 + }, + "ttft_ms": { + "mean": 747.4954129982507, + "p50": 600.3231319991755, + "p90": 1147.3169775985298, + "p95": 1231.2274228017484, + "p99": 1298.3557789643237 + }, + "tpot_ms": { + "mean": 9.02663269935308, + "p50": 9.102326783070463, + "p90": 10.184806440411771, + "p95": 10.474653736578562, + "p99": 10.706531573511999 + }, + "itl_ms": { + "mean": 8.845442765854395, + "p50": 7.658207003260031, + "p90": 8.203889403375797, + "p95": 8.270193445059704, + "p99": 40.88204233179569 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_5_131072_1024.jsonl" + }, + { + "name": "c5_i131072_o256", + "config": { + "phase": "128k", + "concurrency": 5, + "input_len": 131072, + "output_len": 256, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 9.498662344994955, + "request_throughput": 1.0527798164411235, + "input_token_throughput": 77752.10584142437, + "output_token_throughput": 146.96806237518084, + "total_token_throughput": 77899.07390379955, + "total_input_tokens": 738541, + "total_output_tokens": 1396, + "e2e_ms": { + "mean": 4366.1755084991455, + "p50": 3982.204109990562, + "p90": 7813.550987806229, + "p95": 7814.851387899398, + "p99": 7815.891707973933 + }, + "ttft_ms": { + "mean": 1222.2237519992632, + "p50": 929.061669499788, + "p90": 1500.9601024939898, + "p95": 3270.2325747450186, + "p99": 4685.650552545849 + }, + "tpot_ms": { + "mean": 19.906604901446816, + "p50": 10.306678985809755, + "p90": 34.20789826843462, + "p95": 40.81402125487341, + "p99": 46.098919644024484 + }, + "itl_ms": { + "mean": 22.68340019768527, + "p50": 7.854708004742861, + "p90": 8.238709502620623, + "p95": 8.335292746778578, + "p99": 209.5159812430211 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_5_131072_256.jsonl" + }, + { + "name": "c5_i131072_o4096", + "config": { + "phase": "128k", + "concurrency": 5, + "input_len": 131072, + "output_len": 4096, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 50.321813589995145, + "request_throughput": 0.19872097777469958, + "input_token_throughput": 14676.35896467044, + "output_token_throughput": 378.76218363857737, + "total_token_throughput": 15055.121148309017, + "total_input_tokens": 738541, + "total_output_tokens": 19060, + "e2e_ms": { + "mean": 19785.605015904002, + "p50": 22238.636635505827, + "p90": 30440.56655391032, + "p95": 30849.593009458476, + "p99": 31176.814173897 + }, + "ttft_ms": { + "mean": 1452.1458315022755, + "p50": 845.5761140066897, + "p90": 2657.630662203881, + "p95": 4667.562957607147, + "p99": 6275.508793929766 + }, + "tpot_ms": { + "mean": 11.983874961162693, + "p50": 8.97393289609107, + "p90": 20.14835106419254, + "p95": 22.065141436643415, + "p99": 23.598573734604123 + }, + "itl_ms": { + "mean": 9.62384469852828, + "p50": 7.92445049592061, + "p90": 8.225913498608861, + "p95": 8.297063296777196, + "p99": 8.573916231980547 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_128k_0708_5_131072_4096.jsonl" + }, + { + "name": "c10_i262144_o1024", + "config": { + "phase": "256k", + "concurrency": 10, + "input_len": 262144, + "output_len": 1024, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 123.96498416698887, + "request_throughput": 0.16133588153456868, + "input_token_throughput": 22466.997585771973, + "output_token_throughput": 76.30380517177426, + "total_token_throughput": 22543.30139094375, + "total_input_tokens": 2785121, + "total_output_tokens": 9459, + "e2e_ms": { + "mean": 60222.0201104501, + "p50": 47189.717301494966, + "p90": 118746.55429970153, + "p95": 119241.29611300159, + "p99": 120180.87091939495 + }, + "ttft_ms": { + "mean": 12461.068878801598, + "p50": 8218.190929001139, + "p90": 30000.515215401552, + "p95": 39446.964688052685, + "p99": 43965.67287040655 + }, + "tpot_ms": { + "mean": 108.41895635156409, + "p50": 110.26026996855646, + "p90": 183.6072472225656, + "p95": 202.4327935913202, + "p99": 202.90272963421216 + }, + "itl_ms": { + "mean": 101.19910211611595, + "p50": 8.39356699725613, + "p90": 9.216163592645898, + "p95": 10.460725796292536, + "p99": 459.108861363784 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_10_262144_1024.jsonl" + }, + { + "name": "c10_i262144_o256", + "config": { + "phase": "256k", + "concurrency": 10, + "input_len": 262144, + "output_len": 256, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 136.21132193600351, + "request_throughput": 0.14683067248548287, + "input_token_throughput": 20447.059469172025, + "output_token_throughput": 18.698886141026243, + "total_token_throughput": 20465.758355313053, + "total_input_tokens": 2785121, + "total_output_tokens": 2547, + "e2e_ms": { + "mean": 67539.66885109912, + "p50": 64579.87214499735, + "p90": 134268.22255290608, + "p95": 134322.67501935567, + "p99": 134628.3311102711 + }, + "ttft_ms": { + "mean": 21481.911876800586, + "p50": 16078.680619000806, + "p90": 47468.99928749774, + "p95": 58074.31215379911, + "p99": 62652.58622915774 + }, + "tpot_ms": { + "mean": 2953.6964707275465, + "p50": 376.45904176504143, + "p90": 1102.486299162998, + "p95": 3625.5861425419776, + "p99": 41185.3192885079 + }, + "itl_ms": { + "mean": 364.52494103166526, + "p50": 2.6113710046047345, + "p90": 9.080003603594378, + "p95": 12.315574500826166, + "p99": 9161.599309783333 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_10_262144_256.jsonl" + }, + { + "name": "c10_i262144_o4096", + "config": { + "phase": "256k", + "concurrency": 10, + "input_len": 262144, + "output_len": 4096, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 139.951198544004, + "request_throughput": 0.14290695762574354, + "input_token_throughput": 19900.65843647842, + "output_token_throughput": 272.45926006136136, + "total_token_throughput": 20173.117696539783, + "total_input_tokens": 2785121, + "total_output_tokens": 38131, + "e2e_ms": { + "mean": 61533.118652649864, + "p50": 57467.8910960065, + "p90": 115673.9482717938, + "p95": 119183.3668849511, + "p99": 120183.74748178918 + }, + "ttft_ms": { + "mean": 8041.92020519913, + "p50": 5042.6441470044665, + "p90": 17776.405719191825, + "p95": 18088.116007791545, + "p99": 22579.08617755019 + }, + "tpot_ms": { + "mean": 27.99621378762109, + "p50": 33.11629646517114, + "p90": 41.01358958532386, + "p95": 43.87014164002981, + "p99": 45.07000334233188 + }, + "itl_ms": { + "mean": 28.071248564246346, + "p50": 8.652534990687855, + "p90": 9.23243400757201, + "p95": 9.507564500381704, + "p99": 17.724962301144885 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_10_262144_4096.jsonl" + }, + { + "name": "c2_i262144_o1024", + "config": { + "phase": "256k", + "concurrency": 2, + "input_len": 262144, + "output_len": 1024, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 7.838861249998445, + "request_throughput": 0.510278198890278, + "input_token_throughput": 64353.09720529881, + "output_token_throughput": 114.94016430003514, + "total_token_throughput": 64468.03736959884, + "total_input_tokens": 504455, + "total_output_tokens": 901, + "e2e_ms": { + "mean": 2623.6137064988725, + "p50": 1710.9498594945762, + "p90": 5057.276670502325, + "p95": 5714.705932253127, + "p99": 6240.64934165377 + }, + "ttft_ms": { + "mean": 817.0428332487063, + "p50": 846.876464493107, + "p90": 985.784365501604, + "p95": 1007.0702317520044, + "p99": 1024.0989247523248 + }, + "tpot_ms": { + "mean": 8.356801650073102, + "p50": 8.322034111108326, + "p90": 8.987893284592078, + "p95": 9.02657565642891, + "p99": 9.057521553898379 + }, + "itl_ms": { + "mean": 8.055848182821979, + "p50": 7.6801530085504055, + "p90": 7.929283403791487, + "p95": 8.031782798934728, + "p99": 10.759145955671555 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_2_262144_1024.jsonl" + }, + { + "name": "c2_i262144_o256", + "config": { + "phase": "256k", + "concurrency": 2, + "input_len": 262144, + "output_len": 256, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 13.676418158007436, + "request_throughput": 0.29247423951117135, + "input_token_throughput": 36885.02312315199, + "output_token_throughput": 28.443119792461413, + "total_token_throughput": 36913.46624294445, + "total_input_tokens": 504455, + "total_output_tokens": 389, + "e2e_ms": { + "mean": 6497.607406749012, + "p50": 6174.236201499298, + "p90": 11468.486644695804, + "p95": 11731.182600344617, + "p99": 11941.339364863668 + }, + "ttft_ms": { + "mean": 4623.183049494401, + "p50": 3760.040304492577, + "p90": 8986.581812394434, + "p95": 9538.914146694877, + "p99": 9980.780014135233 + }, + "tpot_ms": { + "mean": 14.09713766259988, + "p50": 9.559062930339627, + "p90": 24.260197655032492, + "p95": 26.983417011029736, + "p99": 29.16199249582754 + }, + "itl_ms": { + "mean": 19.47422767535771, + "p50": 7.5968110031681135, + "p90": 8.036788206663914, + "p95": 8.225746598327532, + "p99": 13.369320277706462 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_2_262144_256.jsonl" + }, + { + "name": "c2_i262144_o4096", + "config": { + "phase": "256k", + "concurrency": 2, + "input_len": 262144, + "output_len": 4096, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 47.9139328859892, + "request_throughput": 0.08348302381100643, + "input_token_throughput": 10528.357194145312, + "output_token_throughput": 189.77778387837034, + "total_token_throughput": 10718.134978023681, + "total_input_tokens": 504455, + "total_output_tokens": 9093, + "e2e_ms": { + "mean": 22625.154106750415, + "p50": 21311.755373506458, + "p90": 37184.96222729591, + "p95": 38039.503609643, + "p99": 38723.13671552067 + }, + "ttft_ms": { + "mean": 4475.717700250243, + "p50": 3599.2469300035737, + "p90": 8900.0592248005, + "p95": 9453.57058790032, + "p99": 9896.37967838018 + }, + "tpot_ms": { + "mean": 7.816333810675103, + "p50": 7.564392993347875, + "p90": 8.315652633977185, + "p95": 8.476097631276604, + "p99": 8.604453629116142 + }, + "itl_ms": { + "mean": 7.987409411269122, + "p50": 7.520452010794543, + "p90": 7.734301398159005, + "p95": 7.81220099888742, + "p99": 8.031192004564218 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_2_262144_4096.jsonl" + }, + { + "name": "c5_i262144_o1024", + "config": { + "phase": "256k", + "concurrency": 5, + "input_len": 262144, + "output_len": 1024, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 65.58298989200557, + "request_throughput": 0.15247856214647784, + "input_token_throughput": 21254.002025453763, + "output_token_throughput": 56.417067994196806, + "total_token_throughput": 21310.41909344796, + "total_input_tokens": 1393901, + "total_output_tokens": 3700, + "e2e_ms": { + "mean": 29705.239315898507, + "p50": 26832.999823993305, + "p90": 53121.259530991665, + "p95": 53300.682289002, + "p99": 53444.22049541026 + }, + "ttft_ms": { + "mean": 9553.357254501316, + "p50": 7428.888360009296, + "p90": 13744.180342195608, + "p95": 20522.01899459686, + "p99": 25944.289916517886 + }, + "tpot_ms": { + "mean": 54.106340392946684, + "p50": 35.3284117471819, + "p90": 112.67091399955211, + "p95": 118.9769503805897, + "p99": 124.02177948541978 + }, + "itl_ms": { + "mean": 54.61205585257842, + "p50": 7.787503513100091, + "p90": 8.485531604674179, + "p95": 8.731119994627077, + "p99": 163.1971791438993 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_5_262144_1024.jsonl" + }, + { + "name": "c5_i262144_o256", + "config": { + "phase": "256k", + "concurrency": 5, + "input_len": 262144, + "output_len": 256, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 63.62121302900778, + "request_throughput": 0.15718027877652302, + "input_token_throughput": 21909.37477668742, + "output_token_throughput": 21.942366917202612, + "total_token_throughput": 21931.31714360462, + "total_input_tokens": 1393901, + "total_output_tokens": 1396, + "e2e_ms": { + "mean": 27766.488692698476, + "p50": 26212.031901995942, + "p90": 43731.58486139291, + "p95": 43745.44146119224, + "p99": 43756.5267410317 + }, + "ttft_ms": { + "mean": 12937.516684601724, + "p50": 11740.926471502462, + "p90": 19973.475571106243, + "p95": 25875.404617557782, + "p99": 30596.94785471904 + }, + "tpot_ms": { + "mean": 97.04540830096094, + "p50": 68.37296407113367, + "p90": 205.2091521978607, + "p95": 231.7474121596734, + "p99": 252.97802012912362 + }, + "itl_ms": { + "mean": 106.99087801515344, + "p50": 7.89495949720731, + "p90": 8.512704996974207, + "p95": 8.72298425383633, + "p99": 533.0643086941564 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_5_262144_256.jsonl" + }, + { + "name": "c5_i262144_o4096", + "config": { + "phase": "256k", + "concurrency": 5, + "input_len": 262144, + "output_len": 4096, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 90.97672484000213, + "request_throughput": 0.1099182237829146, + "input_token_throughput": 15321.512204922845, + "output_token_throughput": 209.50413453023523, + "total_token_throughput": 15531.01633945308, + "total_input_tokens": 1393901, + "total_output_tokens": 19060, + "e2e_ms": { + "mean": 40104.55356059829, + "p50": 41497.74021099438, + "p90": 56552.7266723031, + "p95": 58680.973652151435, + "p99": 60383.57123603011 + }, + "ttft_ms": { + "mean": 6888.178656999662, + "p50": 6021.056942001451, + "p90": 11809.612826703229, + "p95": 16539.496441348445, + "p99": 20323.403333064638 + }, + "tpot_ms": { + "mean": 25.054269337052965, + "p50": 16.602636430291934, + "p90": 52.2769187839349, + "p95": 56.69092042522937, + "p99": 60.22212173826497 + }, + "itl_ms": { + "mean": 17.436399203515855, + "p50": 8.002083493920509, + "p90": 8.508344892470632, + "p95": 8.67334350114106, + "p99": 9.995497615309418 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_256k_0708_5_262144_4096.jsonl" + }, + { + "name": "c25_i65536_o1024", + "config": { + "phase": "64k", + "concurrency": 25, + "input_len": 65536, + "output_len": 1024, + "dataset": "random", + "num_prompts": 50 + }, + "metrics": { + "success": 50, + "failed": 0, + "duration_s": 74.64872127999843, + "request_throughput": 0.6698038377972475, + "input_token_throughput": 24668.218938311584, + "output_token_throughput": 346.7038625206112, + "total_token_throughput": 25014.922800832195, + "total_input_tokens": 1841451, + "total_output_tokens": 25881, + "e2e_ms": { + "mean": 34945.79836199962, + "p50": 31994.75705599616, + "p90": 67992.56891410187, + "p95": 68400.96390364997, + "p99": 69586.94131211669 + }, + "ttft_ms": { + "mean": 6120.962128339743, + "p50": 3360.3478820004966, + "p90": 18518.40006079001, + "p95": 19996.429142699337, + "p99": 24478.108350212275 + }, + "tpot_ms": { + "mean": 149.26707781764028, + "p50": 63.565434038753835, + "p90": 123.77963635468883, + "p95": 394.9986222815072, + "p99": 2037.0942240442123 + }, + "itl_ms": { + "mean": 55.794990739112286, + "p50": 10.010411002440378, + "p90": 11.551048010005616, + "p95": 20.707134499389213, + "p99": 880.9660942002665 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_64k_0708_25_65536_1024.jsonl" + }, + { + "name": "c25_i65536_o256", + "config": { + "phase": "64k", + "concurrency": 25, + "input_len": 65536, + "output_len": 256, + "dataset": "random", + "num_prompts": 50 + }, + "metrics": { + "success": 50, + "failed": 0, + "duration_s": 70.01913154299837, + "request_throughput": 0.7140905477997149, + "input_token_throughput": 26299.255066726655, + "output_token_throughput": 84.44834818279429, + "total_token_throughput": 26383.70341490945, + "total_input_tokens": 1841451, + "total_output_tokens": 5913, + "e2e_ms": { + "mean": 34392.330781399796, + "p50": 31949.203033007507, + "p90": 68333.77209850296, + "p95": 68802.89003190192, + "p99": 68854.26202498449 + }, + "ttft_ms": { + "mean": 9330.859263940365, + "p50": 4773.252274004335, + "p90": 25854.001908602368, + "p95": 27219.0645494993, + "p99": 31612.69766841418 + }, + "tpot_ms": { + "mean": 292.8437435563451, + "p50": 237.7339640860819, + "p90": 632.4370414020669, + "p95": 801.176717005343, + "p99": 938.4264802414143 + }, + "itl_ms": { + "mean": 213.72542370732003, + "p50": 9.647318001952954, + "p90": 54.957336393999874, + "p95": 328.82969500496984, + "p99": 4943.480069027282 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_64k_0708_25_65536_256.jsonl" + }, + { + "name": "c25_i65536_o4096", + "config": { + "phase": "64k", + "concurrency": 25, + "input_len": 65536, + "output_len": 4096, + "dataset": "random", + "num_prompts": 50 + }, + "metrics": { + "success": 50, + "failed": 0, + "duration_s": 122.097007467004, + "request_throughput": 0.40951044613859355, + "input_token_throughput": 15081.868411047186, + "output_token_throughput": 950.0069035790776, + "total_token_throughput": 16031.875314626262, + "total_input_tokens": 1841451, + "total_output_tokens": 115993, + "e2e_ms": { + "mean": 52152.373436100024, + "p50": 49024.354140492505, + "p90": 89052.32497549441, + "p95": 93852.743980995, + "p99": 96670.1228921843 + }, + "ttft_ms": { + "mean": 5499.833113261266, + "p50": 2674.395349000406, + "p90": 17162.67022081156, + "p95": 18559.575018648313, + "p99": 23249.706619027413 + }, + "tpot_ms": { + "mean": 23.781828525346313, + "p50": 20.795121718049213, + "p90": 31.204594821803948, + "p95": 42.50058968497073, + "p99": 82.84478563792041 + }, + "itl_ms": { + "mean": 20.118731597388148, + "p50": 10.207813000306487, + "p90": 10.9851807996165, + "p95": 11.638686200603841, + "p99": 94.83414596616095 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/sglang/raw_outputs/sglang_64k_0708_25_65536_4096.jsonl" + } + ] +} \ No newline at end of file diff --git a/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/report.md b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/report.md new file mode 100644 index 0000000..d2f00d9 --- /dev/null +++ b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/report.md @@ -0,0 +1,35 @@ +# H200 VLLM Comparison Benchmark Report + +- Result root: `/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm` +- Model: `/data/models/DeepSeek-V4-Flash` +- Backend: VLLM (TP=8) +- Benchmark client: `sglang.bench_serving --backend vllm` + +## Results + +| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| c10_i131072_o1024 | 128k | 10 | 131072 | 1024 | 51.67 | 20 | 0.39 | 28537.46 | 183.08 | 28720.55 | 2870.79 | 6596.15 | 6603.12 | 47.10 | 77.53 | 84.06 | 24302.31 | 47488.34 | 48426.64 | ⚠️ | +| c10_i131072_o256 | 128k | 10 | 131072 | 256 | 52.96 | 20 | 0.38 | 27838.09 | 48.09 | 27886.18 | 10289.92 | 23250.03 | 24238.33 | 150.22 | 299.91 | 301.72 | 25961.91 | 51418.73 | 51439.08 | ❌ | +| c10_i131072_o4096 | 128k | 10 | 131072 | 4096 | 85.53 | 20 | 0.23 | 17238.00 | 445.81 | 17683.81 | 937.55 | 2096.29 | 2168.81 | 18.57 | 21.47 | 21.61 | 34484.49 | 65903.20 | 67041.13 | ✅ | +| c2_i131072_o1024 | 128k | 2 | 131072 | 1024 | 5.69 | 4 | 0.70 | 42587.41 | 158.36 | 42745.76 | 384.25 | 676.14 | 695.90 | 6.49 | 8.20 | 8.34 | 1937.38 | 4390.02 | 4803.24 | ✅ | +| c2_i131072_o256 | 128k | 2 | 131072 | 256 | 2.64 | 4 | 1.51 | 91644.87 | 147.12 | 91791.99 | 469.80 | 684.63 | 700.28 | 7.08 | 9.27 | 9.45 | 1208.21 | 1556.10 | 1556.46 | ✅ | +| c2_i131072_o4096 | 128k | 2 | 131072 | 4096 | 34.68 | 4 | 0.12 | 6987.68 | 262.22 | 7249.90 | 386.27 | 671.34 | 688.15 | 7.09 | 7.17 | 7.18 | 16427.81 | 25979.54 | 26496.04 | ✅ | +| c5_i131072_o1024 | 128k | 5 | 131072 | 1024 | 11.16 | 10 | 0.90 | 66199.25 | 331.65 | 66530.90 | 616.34 | 856.18 | 917.80 | 8.93 | 10.48 | 10.62 | 3905.82 | 6409.85 | 6989.27 | ✅ | +| c5_i131072_o256 | 128k | 5 | 131072 | 256 | 25.82 | 10 | 0.39 | 28605.35 | 54.07 | 28659.43 | 4405.54 | 9384.23 | 9677.91 | 51.54 | 107.16 | 117.25 | 12185.89 | 19857.41 | 19860.69 | ❌ | +| c5_i131072_o4096 | 128k | 5 | 131072 | 4096 | 42.56 | 10 | 0.23 | 17353.52 | 447.85 | 17801.37 | 615.76 | 1054.01 | 1104.66 | 8.48 | 9.13 | 9.14 | 16241.78 | 28711.60 | 28818.20 | ✅ | +| c10_i262144_o1024 | 256k | 10 | 262144 | 1024 | 111.77 | 20 | 0.18 | 24919.08 | 84.63 | 25003.71 | 15133.21 | 34130.22 | 38144.40 | 98.87 | 142.57 | 266.08 | 54087.22 | 107356.89 | 108658.89 | ❌ | +| c10_i262144_o256 | 256k | 10 | 262144 | 256 | 111.93 | 20 | 0.18 | 24883.63 | 22.76 | 24906.39 | 28922.71 | 50632.44 | 51883.02 | 229.20 | 339.27 | 344.95 | 55262.71 | 110121.03 | 110216.92 | ❌ | +| c10_i262144_o4096 | 256k | 10 | 262144 | 4096 | 176.65 | 20 | 0.11 | 15766.52 | 215.86 | 15982.38 | 11662.77 | 34100.32 | 38022.94 | 37.18 | 56.07 | 62.00 | 79905.41 | 156158.99 | 157944.21 | ⚠️ | +| c2_i262144_o1024 | 256k | 2 | 262144 | 1024 | 20.88 | 4 | 0.19 | 24156.04 | 43.14 | 24199.18 | 5483.04 | 8722.76 | 9009.70 | 20.17 | 52.24 | 57.91 | 9282.52 | 13247.86 | 13826.13 | ⚠️ | +| c2_i262144_o256 | 256k | 2 | 262144 | 256 | 17.11 | 4 | 0.23 | 29488.06 | 22.74 | 29510.80 | 6791.08 | 8625.18 | 8952.31 | 10.18 | 23.01 | 25.44 | 8281.06 | 10221.13 | 10378.60 | ⚠️ | +| c2_i262144_o4096 | 256k | 2 | 262144 | 4096 | 50.30 | 4 | 0.08 | 10027.95 | 180.76 | 10208.71 | 5468.77 | 8759.63 | 9062.80 | 8.63 | 11.53 | 12.01 | 23992.03 | 35306.58 | 35881.75 | ⚠️ | +| c5_i262144_o1024 | 256k | 5 | 262144 | 1024 | 61.34 | 10 | 0.16 | 22724.99 | 60.32 | 22785.31 | 11196.96 | 22564.89 | 26639.92 | 47.49 | 100.69 | 109.86 | 28871.28 | 47913.60 | 53790.59 | ⚠️ | +| c5_i262144_o256 | 256k | 5 | 262144 | 256 | 54.01 | 10 | 0.19 | 25809.33 | 25.85 | 25835.18 | 9094.42 | 21986.99 | 26381.86 | 123.42 | 202.78 | 215.56 | 26745.69 | 45719.33 | 45761.90 | ❌ | +| c5_i262144_o4096 | 256k | 5 | 262144 | 4096 | 92.92 | 10 | 0.11 | 15001.24 | 205.12 | 15206.37 | 9122.40 | 22171.78 | 26533.69 | 25.69 | 63.32 | 68.33 | 41465.64 | 61304.00 | 64086.27 | ⚠️ | +| c25_i65536_o1024 | 64k | 25 | 65536 | 1024 | 77.81 | 50 | 0.64 | 23665.15 | 332.61 | 23997.76 | 8358.86 | 24861.20 | 28942.75 | 72.36 | 196.24 | 271.32 | 36604.32 | 71227.08 | 72689.84 | ❌ | +| c25_i65536_o256 | 64k | 25 | 65536 | 256 | 61.85 | 50 | 0.81 | 29772.41 | 95.60 | 29868.01 | 11442.71 | 24902.72 | 28974.45 | 180.78 | 273.59 | 278.16 | 30201.94 | 60181.25 | 60781.45 | ❌ | +| c25_i65536_o4096 | 64k | 25 | 65536 | 4096 | 156.39 | 50 | 0.32 | 11775.02 | 741.71 | 12516.73 | 7847.53 | 24847.97 | 28840.96 | 28.80 | 46.66 | 63.40 | 69020.59 | 126999.05 | 129654.04 | ⚠️ | + +SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail. + diff --git a/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/results.json b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/results.json new file mode 100644 index 0000000..ee32d30 --- /dev/null +++ b/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/results.json @@ -0,0 +1,1219 @@ +{ + "metadata": { + "experiment": "dsv4_h200_long_context_matrix_vllm", + "run_id": "20260708-100016", + "timestamp": "2026-07-08T10:00:20+00:00", + "model": "/data/models/DeepSeek-V4-Flash", + "backend": "vllm", + "engine": "vllm", + "hardware": "8x NVIDIA H200 143GB", + "accelerator": "NVIDIA H200", + "chip": "nvidia_h200", + "script": "experiments/dsv4_h200_long_context_matrix/run_bench.sh", + "env": "/data/user1/yy/envs/vllm", + "git_commit": "7e35945", + "git_dirty": "dirty", + "description": "H200 long-context matrix vllm TP=8 for DeepSeek-V4-Flash" + }, + "config": { + "tp": 8, + "cuda_visible_devices": "0,1,2,3,4,5,6,7", + "backend": "vllm", + "server_start_script": "experiments/dsv4_h200_long_context_matrix/start_vllm.sh", + "groups": [ + { + "label": "64k", + "max_model_len": 80000, + "max_slots": 25 + }, + { + "label": "128k", + "max_model_len": 140000, + "max_slots": 10 + }, + { + "label": "256k", + "max_model_len": 270000, + "max_slots": 10 + } + ] + }, + "scenarios": [ + { + "name": "c10_i131072_o1024", + "config": { + "phase": "128k", + "concurrency": 10, + "input_len": 131072, + "output_len": 1024, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 51.66545137499634, + "request_throughput": 0.387105879610665, + "input_token_throughput": 28537.464800192207, + "output_token_throughput": 183.08172576186402, + "total_token_throughput": 28720.54652595407, + "total_input_tokens": 1474401, + "total_output_tokens": 9459, + "e2e_ms": { + "mean": 24302.307708252192, + "p50": 21800.77651650936, + "p90": 47116.329806011345, + "p95": 47488.33684020138, + "p99": 48426.63913604789 + }, + "ttft_ms": { + "mean": 2870.791056800226, + "p50": 1759.841219500231, + "p90": 5543.851142600763, + "p95": 6596.149501448235, + "p99": 6603.124833886832 + }, + "tpot_ms": { + "mean": 47.09948558432735, + "p50": 55.566536207784615, + "p90": 67.81654982985651, + "p95": 77.52663093973904, + "p99": 84.05732836300406 + }, + "itl_ms": { + "mean": 45.89674528407251, + "p50": 20.6618029915262, + "p90": 36.3844861916733, + "p95": 279.25614129489986, + "p99": 340.7204606453888 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_10_131072_1024.jsonl" + }, + { + "name": "c10_i131072_o256", + "config": { + "phase": "128k", + "concurrency": 10, + "input_len": 131072, + "output_len": 256, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 52.96343759499723, + "request_throughput": 0.3776189935580983, + "input_token_throughput": 27838.091086052682, + "output_token_throughput": 48.08977882962382, + "total_token_throughput": 27886.180864882306, + "total_input_tokens": 1474401, + "total_output_tokens": 2547, + "e2e_ms": { + "mean": 25961.91139175062, + "p50": 25894.207332996302, + "p90": 51272.70432260411, + "p95": 51418.72619945061, + "p99": 51439.07880868617 + }, + "ttft_ms": { + "mean": 10289.923211449786, + "p50": 7619.981570496748, + "p90": 21425.749930497845, + "p95": 23250.027229247647, + "p99": 24238.332692253316 + }, + "tpot_ms": { + "mean": 150.21826331221968, + "p50": 152.71475159090537, + "p90": 298.5961233597156, + "p95": 299.91164498010716, + "p99": 301.722881242001 + }, + "itl_ms": { + "mean": 127.77789348878402, + "p50": 21.68448800512124, + "p90": 324.1446094005369, + "p95": 342.3754072049635, + "p99": 391.54394332435913 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_10_131072_256.jsonl" + }, + { + "name": "c10_i131072_o4096", + "config": { + "phase": "128k", + "concurrency": 10, + "input_len": 131072, + "output_len": 4096, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 85.53201726201223, + "request_throughput": 0.23383056591233586, + "input_token_throughput": 17238.001010585696, + "output_token_throughput": 445.80966544016394, + "total_token_throughput": 17683.81067602586, + "total_input_tokens": 1474401, + "total_output_tokens": 38131, + "e2e_ms": { + "mean": 34484.490811148135, + "p50": 33977.14802798873, + "p90": 62483.23918669922, + "p95": 65903.20190555212, + "p99": 67041.12905950897 + }, + "ttft_ms": { + "mean": 937.5491100487125, + "p50": 642.0005089967162, + "p90": 1956.0433094957264, + "p95": 2096.2857415936014, + "p99": 2168.806972318125 + }, + "tpot_ms": { + "mean": 18.574484843485518, + "p50": 20.402039316506, + "p90": 21.315744186469686, + "p95": 21.47395290997979, + "p99": 21.612366181997732 + }, + "itl_ms": { + "mean": 17.75861549879622, + "p50": 20.105969000724144, + "p90": 20.685401002992876, + "p95": 20.930733007844537, + "p99": 35.77366020763292 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_10_131072_4096.jsonl" + }, + { + "name": "c2_i131072_o1024", + "config": { + "phase": "128k", + "concurrency": 2, + "input_len": 131072, + "output_len": 1024, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 5.689733478997368, + "request_throughput": 0.7030206273747766, + "input_token_throughput": 42587.40780995237, + "output_token_throughput": 158.35539631616842, + "total_token_throughput": 42745.76320626854, + "total_input_tokens": 242311, + "total_output_tokens": 901, + "e2e_ms": { + "mean": 1937.3816890001763, + "p50": 1117.7794850009377, + "p90": 3873.4886924983593, + "p95": 4390.017209249343, + "p99": 4803.240022650133 + }, + "ttft_ms": { + "mean": 384.2540892474062, + "p50": 357.03718150034547, + "p90": 651.4245806931286, + "p95": 676.1357038427377, + "p99": 695.9046023624251 + }, + "tpot_ms": { + "mean": 6.491317100270631, + "p50": 7.012631920858373, + "p90": 8.019041642945352, + "p95": 8.196621751066914, + "p99": 8.338685837564164 + }, + "itl_ms": { + "mean": 7.404440741348319, + "p50": 6.702701997710392, + "p90": 7.250566800939851, + "p95": 7.664964400464669, + "p99": 16.972954474331345 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_2_131072_1024.jsonl" + }, + { + "name": "c2_i131072_o256", + "config": { + "phase": "128k", + "concurrency": 2, + "input_len": 131072, + "output_len": 256, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 2.6440215219918173, + "request_throughput": 1.5128469896064558, + "input_token_throughput": 91644.86672463248, + "output_token_throughput": 147.12436973922783, + "total_token_throughput": 91791.9910943717, + "total_input_tokens": 242311, + "total_output_tokens": 389, + "e2e_ms": { + "mean": 1208.2100787556556, + "p50": 1315.4866860058974, + "p90": 1555.6610688057845, + "p95": 1556.102363406535, + "p99": 1556.4553990871354 + }, + "ttft_ms": { + "mean": 469.8006515027373, + "p50": 488.6462400027085, + "p90": 665.0697350021801, + "p95": 684.631172002264, + "p99": 700.2803216023312 + }, + "tpot_ms": { + "mean": 7.077280489734826, + "p50": 7.6205459385392995, + "p90": 9.047700073656017, + "p95": 9.269358677714516, + "p99": 9.446685560961315 + }, + "itl_ms": { + "mean": 8.949986227327662, + "p50": 7.033545000012964, + "p90": 7.379413201124407, + "p95": 10.905502697278283, + "p99": 37.00466673850295 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_2_131072_256.jsonl" + }, + { + "name": "c2_i131072_o4096", + "config": { + "phase": "128k", + "concurrency": 2, + "input_len": 131072, + "output_len": 4096, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 34.67690336800297, + "request_throughput": 0.11535055358175018, + "input_token_throughput": 6987.6769972368675, + "output_token_throughput": 262.2206459297136, + "total_token_throughput": 7249.8976431665815, + "total_input_tokens": 242311, + "total_output_tokens": 9093, + "e2e_ms": { + "mean": 16427.805564500886, + "p50": 15522.598966999794, + "p90": 25333.909562396002, + "p95": 25979.535298196424, + "p99": 26496.035886836762 + }, + "ttft_ms": { + "mean": 386.27199300026405, + "p50": 364.75162849819753, + "p90": 650.3169154995704, + "p95": 671.3353617502435, + "p99": 688.1501187507819 + }, + "tpot_ms": { + "mean": 7.0888771850629775, + "p50": 7.074092415464063, + "p90": 7.159303860780842, + "p95": 7.170332691124276, + "p99": 7.179155755399024 + }, + "itl_ms": { + "mean": 7.101148280325202, + "p50": 7.057993505441118, + "p90": 7.237646510475315, + "p95": 7.2992147543118335, + "p99": 7.477600647689542 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_2_131072_4096.jsonl" + }, + { + "name": "c5_i131072_o1024", + "config": { + "phase": "128k", + "concurrency": 5, + "input_len": 131072, + "output_len": 1024, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 11.156335463994765, + "request_throughput": 0.8963516767914834, + "input_token_throughput": 66199.2463729259, + "output_token_throughput": 331.6501204128489, + "total_token_throughput": 66530.89649333875, + "total_input_tokens": 738541, + "total_output_tokens": 3700, + "e2e_ms": { + "mean": 3905.8177351005725, + "p50": 4192.37918699946, + "p90": 5685.582281209644, + "p95": 6409.85399711062, + "p99": 6989.271369831405 + }, + "ttft_ms": { + "mean": 616.3354310003342, + "p50": 630.4351844955818, + "p90": 779.1622185060987, + "p95": 856.1830642516724, + "p99": 917.7997408481316 + }, + "tpot_ms": { + "mean": 8.930163709257268, + "p50": 8.916015531095471, + "p90": 10.314694953508035, + "p95": 10.484373031482733, + "p99": 10.620115493862492 + }, + "itl_ms": { + "mean": 9.45514735153596, + "p50": 7.74998300767038, + "p90": 8.76904659671709, + "p95": 9.120293395244516, + "p99": 27.098282215009426 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_5_131072_1024.jsonl" + }, + { + "name": "c5_i131072_o256", + "config": { + "phase": "128k", + "concurrency": 5, + "input_len": 131072, + "output_len": 256, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 25.818277914993814, + "request_throughput": 0.3873225020245273, + "input_token_throughput": 28605.354796769643, + "output_token_throughput": 54.07022128262401, + "total_token_throughput": 28659.425018052265, + "total_input_tokens": 738541, + "total_output_tokens": 1396, + "e2e_ms": { + "mean": 12185.8890471005, + "p50": 10999.203152496193, + "p90": 19853.3254096983, + "p95": 19857.414202844666, + "p99": 19860.68523736176 + }, + "ttft_ms": { + "mean": 4405.543919697811, + "p50": 4785.443460001261, + "p90": 9017.131690100357, + "p95": 9384.23135404664, + "p99": 9677.911085203668 + }, + "tpot_ms": { + "mean": 51.53614550049823, + "p50": 47.148800233655614, + "p90": 94.54653787075607, + "p95": 107.15865861780897, + "p99": 117.24835521545133 + }, + "itl_ms": { + "mean": 61.31056672891171, + "p50": 8.423997001955286, + "p90": 291.9380680017639, + "p95": 327.3017734027235, + "p99": 370.823923558346 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_5_131072_256.jsonl" + }, + { + "name": "c5_i131072_o4096", + "config": { + "phase": "128k", + "concurrency": 5, + "input_len": 131072, + "output_len": 4096, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 42.558574515001965, + "request_throughput": 0.2349702760010203, + "input_token_throughput": 17353.51826080695, + "output_token_throughput": 447.8533460579447, + "total_token_throughput": 17801.371606864897, + "total_input_tokens": 738541, + "total_output_tokens": 19060, + "e2e_ms": { + "mean": 16241.784594696945, + "p50": 16785.255688002508, + "p90": 28578.344115095388, + "p95": 28711.595474048954, + "p99": 28818.196561211807 + }, + "ttft_ms": { + "mean": 615.7584768981906, + "p50": 619.8361904971534, + "p90": 990.6822835037019, + "p95": 1054.0053227523456, + "p99": 1104.663754151261 + }, + "tpot_ms": { + "mean": 8.482415081791261, + "p50": 8.584914685349283, + "p90": 9.119316274995734, + "p95": 9.13225667612229, + "p99": 9.142608997023535 + }, + "itl_ms": { + "mean": 8.322323361523896, + "p50": 8.334478501637932, + "p90": 8.716492498933803, + "p95": 8.831801002088469, + "p99": 11.488471249322174 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_128k_0708_5_131072_4096.jsonl" + }, + { + "name": "c10_i262144_o1024", + "config": { + "phase": "256k", + "concurrency": 10, + "input_len": 262144, + "output_len": 1024, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 111.7666079490009, + "request_throughput": 0.1789443230586903, + "input_token_throughput": 24919.07959907713, + "output_token_throughput": 84.63171759060758, + "total_token_throughput": 25003.711316667737, + "total_input_tokens": 2785121, + "total_output_tokens": 9459, + "e2e_ms": { + "mean": 54087.21958720125, + "p50": 45854.67296499701, + "p90": 104581.33334430022, + "p95": 107356.88945585352, + "p99": 108658.89333357001 + }, + "ttft_ms": { + "mean": 15133.209349600656, + "p50": 15488.721631001681, + "p90": 25876.80724879175, + "p95": 34130.219954399945, + "p99": 38144.397810086804 + }, + "tpot_ms": { + "mean": 98.8651079269732, + "p50": 103.25214553801447, + "p90": 133.975633900669, + "p95": 142.56661943596396, + "p99": 266.08017341112605 + }, + "itl_ms": { + "mean": 85.27571747186897, + "p50": 20.101816502574366, + "p90": 335.60964649223024, + "p95": 394.74781924946, + "p99": 477.74120174944983 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_10_262144_1024.jsonl" + }, + { + "name": "c10_i262144_o256", + "config": { + "phase": "256k", + "concurrency": 10, + "input_len": 262144, + "output_len": 256, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 111.92582058999687, + "request_throughput": 0.17868977769895802, + "input_token_throughput": 24883.632617734984, + "output_token_throughput": 22.756143189962305, + "total_token_throughput": 24906.388760924947, + "total_input_tokens": 2785121, + "total_output_tokens": 2547, + "e2e_ms": { + "mean": 55262.713170899224, + "p50": 47144.762987496506, + "p90": 106810.52704010073, + "p95": 110121.03238129348, + "p99": 110216.92147946043 + }, + "ttft_ms": { + "mean": 28922.714126997744, + "p50": 30003.8248819983, + "p90": 49919.92798739375, + "p95": 50632.43951184195, + "p99": 51883.02472316703 + }, + "tpot_ms": { + "mean": 229.20485763975984, + "p50": 304.37904707435774, + "p90": 337.78385858514525, + "p95": 339.2687884048935, + "p99": 344.9480431705525 + }, + "itl_ms": { + "mean": 209.96384566004198, + "p50": 275.63429399742745, + "p90": 409.77780719404115, + "p95": 443.09554080828093, + "p99": 506.6882172867193 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_10_262144_256.jsonl" + }, + { + "name": "c10_i262144_o4096", + "config": { + "phase": "256k", + "concurrency": 10, + "input_len": 262144, + "output_len": 4096, + "dataset": "random", + "num_prompts": 20 + }, + "metrics": { + "success": 20, + "failed": 0, + "duration_s": 176.64780976901238, + "request_throughput": 0.11321963191138533, + "input_token_throughput": 15766.518722433471, + "output_token_throughput": 215.85888922065172, + "total_token_throughput": 15982.377611654123, + "total_input_tokens": 2785121, + "total_output_tokens": 38131, + "e2e_ms": { + "mean": 79905.41021660028, + "p50": 71006.16067700321, + "p90": 153299.27942080103, + "p95": 156158.9913786549, + "p99": 157944.20693174092 + }, + "ttft_ms": { + "mean": 11662.773168051353, + "p50": 8513.091607506794, + "p90": 25862.007495203594, + "p95": 34100.31528659311, + "p99": 38022.94094612385 + }, + "tpot_ms": { + "mean": 37.1794706933376, + "p50": 40.5456700184206, + "p90": 54.37848334570407, + "p95": 56.065247396465274, + "p99": 62.004285348212015 + }, + "itl_ms": { + "mean": 36.36695911897189, + "p50": 20.078994508367032, + "p90": 34.5691745984368, + "p95": 269.43711890198756, + "p99": 410.95061424697633 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_10_262144_4096.jsonl" + }, + { + "name": "c2_i262144_o1024", + "config": { + "phase": "256k", + "concurrency": 2, + "input_len": 262144, + "output_len": 1024, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 20.883182838006178, + "request_throughput": 0.19154168361348792, + "input_token_throughput": 24156.04000181051, + "output_token_throughput": 43.144764233938155, + "total_token_throughput": 24199.18476604445, + "total_input_tokens": 504455, + "total_output_tokens": 901, + "e2e_ms": { + "mean": 9282.52205724857, + "p50": 8126.615065506485, + "p90": 12525.01908979175, + "p95": 13247.856977889749, + "p99": 13826.12728836815 + }, + "ttft_ms": { + "mean": 5483.044165495812, + "p50": 6116.950868999993, + "p90": 8364.095199097937, + "p95": 8722.764758044648, + "p99": 9009.70040520202 + }, + "tpot_ms": { + "mean": 20.169965146698488, + "p50": 8.914781762485976, + "p90": 45.14487564022166, + "p95": 52.23905397577902, + "p99": 57.91439664422493 + }, + "itl_ms": { + "mean": 18.26621805286357, + "p50": 6.9415050020325, + "p90": 7.282738502544817, + "p95": 13.22578915351182, + "p99": 332.4238129777949 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_2_262144_1024.jsonl" + }, + { + "name": "c2_i262144_o256", + "config": { + "phase": "256k", + "concurrency": 2, + "input_len": 262144, + "output_len": 256, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 17.107093440004974, + "request_throughput": 0.23382113472566834, + "input_token_throughput": 29488.060129509257, + "output_token_throughput": 22.739105352071245, + "total_token_throughput": 29510.799234861326, + "total_input_tokens": 504455, + "total_output_tokens": 389, + "e2e_ms": { + "mean": 8281.064997001522, + "p50": 8014.565017496352, + "p90": 10024.288492804046, + "p95": 10221.127670405258, + "p99": 10378.59901248623 + }, + "ttft_ms": { + "mean": 6791.076173005422, + "p50": 6304.710036005417, + "p90": 8216.254670906346, + "p95": 8625.176349956018, + "p99": 8952.313693195756 + }, + "tpot_ms": { + "mean": 10.181760587403115, + "p50": 5.548914722857892, + "p90": 19.976080493794306, + "p95": 23.012455824028287, + "p99": 25.44155608821548 + }, + "itl_ms": { + "mean": 18.114836313073543, + "p50": 7.0091839879751205, + "p90": 7.488688002922572, + "p95": 11.860086003434832, + "p99": 312.7510143991091 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_2_262144_256.jsonl" + }, + { + "name": "c2_i262144_o4096", + "config": { + "phase": "256k", + "concurrency": 2, + "input_len": 262144, + "output_len": 4096, + "dataset": "random", + "num_prompts": 4 + }, + "metrics": { + "success": 4, + "failed": 0, + "duration_s": 50.30488854799478, + "request_throughput": 0.07951513491941621, + "input_token_throughput": 10027.951846443526, + "output_token_throughput": 180.75778045556288, + "total_token_throughput": 10208.709626899088, + "total_input_tokens": 504455, + "total_output_tokens": 9093, + "e2e_ms": { + "mean": 23992.02666349447, + "p50": 22834.540578493034, + "p90": 34587.61013499753, + "p95": 35306.576348497765, + "p99": 35881.74931929796 + }, + "ttft_ms": { + "mean": 5468.770269493689, + "p50": 6087.707849997969, + "p90": 8380.67385629256, + "p95": 8759.632129640522, + "p99": 9062.79874831889 + }, + "tpot_ms": { + "mean": 8.625470352547925, + "p50": 7.610760335213301, + "p90": 10.917184712848888, + "p95": 11.526029506868458, + "p99": 12.013105342084113 + }, + "itl_ms": { + "mean": 8.190675389343355, + "p50": 7.165019997046329, + "p90": 7.346970996877644, + "p95": 7.413411745801568, + "p99": 7.657706698955734 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_2_262144_4096.jsonl" + }, + { + "name": "c5_i262144_o1024", + "config": { + "phase": "256k", + "concurrency": 5, + "input_len": 262144, + "output_len": 1024, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 61.33781583599921, + "request_throughput": 0.16303156321603146, + "input_token_throughput": 22724.985899838946, + "output_token_throughput": 60.321678389931634, + "total_token_throughput": 22785.307578228876, + "total_input_tokens": 1393901, + "total_output_tokens": 3700, + "e2e_ms": { + "mean": 28871.281030599494, + "p50": 28017.1025669988, + "p90": 40567.3574001019, + "p95": 47913.59944454815, + "p99": 53790.59308010517 + }, + "ttft_ms": { + "mean": 11196.959846300888, + "p50": 10630.609530002403, + "p90": 17471.093674494474, + "p95": 22564.888229250926, + "p99": 26639.923873056105 + }, + "tpot_ms": { + "mean": 47.48773145747153, + "p50": 37.93084358310368, + "p90": 89.22286159943364, + "p95": 100.68561415719321, + "p99": 109.85581620340092 + }, + "itl_ms": { + "mean": 51.40852411926291, + "p50": 7.74506800371455, + "p90": 283.8695481987088, + "p95": 338.8318899014843, + "p99": 459.29262580379196 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_5_262144_1024.jsonl" + }, + { + "name": "c5_i262144_o256", + "config": { + "phase": "256k", + "concurrency": 5, + "input_len": 262144, + "output_len": 256, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 54.00763233398902, + "request_throughput": 0.1851590148992076, + "input_token_throughput": 25809.333602702038, + "output_token_throughput": 25.848198479929383, + "total_token_throughput": 25835.181801181967, + "total_input_tokens": 1393901, + "total_output_tokens": 1396, + "e2e_ms": { + "mean": 26745.69082800008, + "p50": 26546.488270505506, + "p90": 45666.111250901304, + "p95": 45719.32812445375, + "p99": 45761.90162329571 + }, + "ttft_ms": { + "mean": 9094.423362601083, + "p50": 6875.693890498951, + "p90": 16493.41224690724, + "p95": 21986.99414745059, + "p99": 26381.859667885295 + }, + "tpot_ms": { + "mean": 123.42273835337818, + "p50": 157.13877274814615, + "p90": 186.793076411668, + "p95": 202.77654827342073, + "p99": 215.56332576282296 + }, + "itl_ms": { + "mean": 156.4825442339936, + "p50": 14.127962000202388, + "p90": 379.0354756027227, + "p95": 433.3574508113088, + "p99": 501.85217722595576 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_5_262144_256.jsonl" + }, + { + "name": "c5_i262144_o4096", + "config": { + "phase": "256k", + "concurrency": 5, + "input_len": 262144, + "output_len": 4096, + "dataset": "random", + "num_prompts": 10 + }, + "metrics": { + "success": 10, + "failed": 0, + "duration_s": 92.91902663900692, + "request_throughput": 0.10762058495135003, + "input_token_throughput": 15001.244098427176, + "output_token_throughput": 205.12483491727315, + "total_token_throughput": 15206.368933344449, + "total_input_tokens": 1393901, + "total_output_tokens": 19060, + "e2e_ms": { + "mean": 41465.640329997404, + "p50": 42983.5961284989, + "p90": 57826.16482919402, + "p95": 61304.00343809668, + "p99": 64086.27432521884 + }, + "ttft_ms": { + "mean": 9122.39999499725, + "p50": 6438.129387992376, + "p90": 16719.39194359147, + "p95": 22171.777112298136, + "p99": 26533.685247263496 + }, + "tpot_ms": { + "mean": 25.69300724592341, + "p50": 15.639313094015739, + "p90": 57.04181744930917, + "p95": 63.31536574396504, + "p99": 68.33420437968975 + }, + "itl_ms": { + "mean": 17.320837186150214, + "p50": 8.466871004202403, + "p90": 8.95137139596045, + "p95": 9.216577408369629, + "p99": 342.54000560729764 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_256k_0708_5_262144_4096.jsonl" + }, + { + "name": "c25_i65536_o1024", + "config": { + "phase": "64k", + "concurrency": 25, + "input_len": 65536, + "output_len": 1024, + "dataset": "random", + "num_prompts": 50 + }, + "metrics": { + "success": 50, + "failed": 0, + "duration_s": 77.81277099800354, + "request_throughput": 0.6425680432493898, + "input_token_throughput": 23665.151316192638, + "output_token_throughput": 332.6060705467491, + "total_token_throughput": 23997.757386739388, + "total_input_tokens": 1841451, + "total_output_tokens": 25881, + "e2e_ms": { + "mean": 36604.3198038204, + "p50": 33663.05495699635, + "p90": 70952.4083481927, + "p95": 71227.07770220513, + "p99": 72689.83805696538 + }, + "ttft_ms": { + "mean": 8358.863178979955, + "p50": 4153.6215449959855, + "p90": 23624.027395700978, + "p95": 24861.19765224866, + "p99": 28942.75298933673 + }, + "tpot_ms": { + "mean": 72.35892965785095, + "p50": 66.58819170680694, + "p90": 123.35084275705259, + "p95": 196.2389562667242, + "p99": 271.3221725716027 + }, + "itl_ms": { + "mean": 56.14946871815307, + "p50": 12.636155493964907, + "p90": 266.132606593601, + "p95": 283.47483030302095, + "p99": 315.9706088724488 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_64k_0708_25_65536_1024.jsonl" + }, + { + "name": "c25_i65536_o256", + "config": { + "phase": "64k", + "concurrency": 25, + "input_len": 65536, + "output_len": 256, + "dataset": "random", + "num_prompts": 50 + }, + "metrics": { + "success": 50, + "failed": 0, + "duration_s": 61.850929373002145, + "request_throughput": 0.8083952902060181, + "input_token_throughput": 29772.406310903247, + "output_token_throughput": 95.6008270197637, + "total_token_throughput": 29868.00713792301, + "total_input_tokens": 1841451, + "total_output_tokens": 5913, + "e2e_ms": { + "mean": 30201.942331739992, + "p50": 29890.801813991857, + "p90": 54022.06084709906, + "p95": 60181.24526374595, + "p99": 60781.44726209328 + }, + "ttft_ms": { + "mean": 11442.71382926061, + "p50": 9054.179862498131, + "p90": 23664.933058293536, + "p95": 24902.72004000362, + "p99": 28974.451748033018 + }, + "tpot_ms": { + "mean": 180.7841084353183, + "p50": 205.89811879472109, + "p90": 271.62320932265123, + "p95": 273.59178299184475, + "p99": 278.1578257179677 + }, + "itl_ms": { + "mean": 164.32375486440372, + "p50": 234.953918006795, + "p90": 298.6645766053698, + "p95": 312.48340460369946, + "p99": 343.43004875729093 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": false, + "overall": "❌" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_64k_0708_25_65536_256.jsonl" + }, + { + "name": "c25_i65536_o4096", + "config": { + "phase": "64k", + "concurrency": 25, + "input_len": 65536, + "output_len": 4096, + "dataset": "random", + "num_prompts": 50 + }, + "metrics": { + "success": 50, + "failed": 0, + "duration_s": 156.38622992498858, + "request_throughput": 0.3197212441529075, + "input_token_throughput": 11775.020095332313, + "output_token_throughput": 741.7085254605639, + "total_token_throughput": 12516.728620792876, + "total_input_tokens": 1841451, + "total_output_tokens": 115993, + "e2e_ms": { + "mean": 69020.59092995944, + "p50": 63262.664559006225, + "p90": 121268.93448900519, + "p95": 126999.05028985013, + "p99": 129654.03733668412 + }, + "ttft_ms": { + "mean": 7847.533580679155, + "p50": 2710.7052505089086, + "p90": 21635.62821639644, + "p95": 24847.965374695188, + "p99": 28840.956312033664 + }, + "tpot_ms": { + "mean": 28.80194254340415, + "p50": 30.27874658957449, + "p90": 40.0271379392775, + "p95": 46.66429590270747, + "p99": 63.40445082082613 + }, + "itl_ms": { + "mean": 26.62544646323448, + "p50": 21.984036997309886, + "p90": 22.893277191906236, + "p95": 39.47063520317897, + "p99": 284.1834103205481 + } + }, + "slo_status": { + "ttft_p95_ok": false, + "tpot_mean_ok": true, + "overall": "⚠️" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_long_context_matrix/results/20260708-100016/vllm/raw_outputs/vllm_64k_0708_25_65536_4096.jsonl" + } + ] +} \ No newline at end of file diff --git a/experiments/dsv4_h200_long_context_matrix/run_bench.sh b/experiments/dsv4_h200_long_context_matrix/run_bench.sh index 4ff579d..5f9dc6f 100755 --- a/experiments/dsv4_h200_long_context_matrix/run_bench.sh +++ b/experiments/dsv4_h200_long_context_matrix/run_bench.sh @@ -93,7 +93,7 @@ run_warmup() { fi log "warming up ${backend} (input=${input_len}, output=${output_len}, num=1)" - "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/warmup.py" \ + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/warmup.py" \ --backend "$backend" \ --port "$port" \ --input-len "$input_len" \ @@ -201,7 +201,7 @@ parse_backend() { local backend="$1" local result_root="${RESULT_BASE}/${RUN_ID}/${backend}" log "parsing ${backend} results in ${result_root}" - "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/parse_backend.py" "$result_root" --backend "$backend" \ + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend "$backend" \ >> "${result_root}/logs/parse.log" 2>&1 || { log "WARNING: parser failed for ${backend}; see ${result_root}/logs/parse.log" } @@ -282,7 +282,7 @@ parse_backend sglang parse_backend vllm log "generating comparison report" -"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/compare.py" \ +"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/compare.py" \ --sglang "${RESULT_BASE}/${RUN_ID}/sglang" \ --vllm "${RESULT_BASE}/${RUN_ID}/vllm" \ --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \ diff --git a/experiments/dsv4_h200_long_context_matrix/warmup.py b/experiments/dsv4_h200_long_context_matrix/warmup.py deleted file mode 100755 index 75dbe5f..0000000 --- a/experiments/dsv4_h200_long_context_matrix/warmup.py +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env python3 -"""Send a small number of warmup requests to a running backend. - -Uses sglang.bench_serving with a single request so that the same code path -(prefill / decode kernels, CUDA graphs, etc.) is exercised before the real -benchmark begins. Discards the output. -""" -import argparse -import subprocess -import sys -import tempfile -from pathlib import Path - - -def run_warmup(backend: str, host: str, port: int, input_len: int, output_len: int, num: int, env_python: Path) -> None: - with tempfile.NamedTemporaryFile(mode="w", suffix=".jsonl", delete=True) as tmp: - cmd = [ - str(env_python), - "-m", - "sglang.bench_serving", - "--backend", - backend, - "--host", - host, - "--port", - str(port), - "--dataset-name", - "random", - "--random-input-len", - str(input_len), - "--random-output-len", - str(output_len), - "--num-prompts", - str(num), - "--max-concurrency", - "1", - "--request-rate", - "10000", - "--output-file", - tmp.name, - "--output-details", - ] - print(f"[warmup] {' '.join(cmd)}", flush=True) - result = subprocess.run(cmd, capture_output=True, text=True) - if result.returncode != 0: - print("[warmup] FAILED", file=sys.stderr) - print(result.stdout, file=sys.stderr) - print(result.stderr, file=sys.stderr) - sys.exit(1) - print(f"[warmup] OK: backend={backend} port={port} input={input_len} output={output_len} num={num}") - - -def main(): - parser = argparse.ArgumentParser(description="Warmup a serving backend.") - parser.add_argument("--backend", required=True, choices=["sglang", "vllm"]) - parser.add_argument("--port", type=int, required=True) - parser.add_argument("--input-len", type=int, required=True) - parser.add_argument("--output-len", type=int, required=True) - parser.add_argument("--num", type=int, default=1) - parser.add_argument("--host", default="127.0.0.1") - parser.add_argument("--env-python", default="/data/user1/yy/envs/sglang/bin/python") - args = parser.parse_args() - - run_warmup( - backend=args.backend, - host=args.host, - port=args.port, - input_len=args.input_len, - output_len=args.output_len, - num=args.num, - env_python=Path(args.env_python), - ) - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_sglang_vs_vllm/compare.py b/experiments/dsv4_h200_sglang_vs_vllm/compare.py deleted file mode 100755 index 0f4eb51..0000000 --- a/experiments/dsv4_h200_sglang_vs_vllm/compare.py +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env python3 -"""Generate a side-by-side comparison of SGLang and vLLM results. - -Usage: - python3 compare.py --sglang --vllm \ - [--output comparison.md] -""" -import argparse -import json -from collections import defaultdict -from pathlib import Path - - -def load_result(result_root: Path) -> dict: - path = result_root / "results.json" - with open(path, "r", encoding="utf-8") as f: - return json.load(f) - - -def slo_status(ttft_p95_ms: float, tpot_mean_ms: float) -> str: - # S2 tier targets from scripts/SLO_STANDARDS.md. - ttft_ok = ttft_p95_ms < 3000.0 - tpot_ok = tpot_mean_ms < 50.0 - if ttft_ok and tpot_ok: - return "✅" - if ttft_ok or tpot_ok: - return "⚠️" - return "❌" - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--sglang", type=Path, required=True) - parser.add_argument("--vllm", type=Path, required=True) - parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md")) - args = parser.parse_args() - - sglang_data = load_result(args.sglang) - vllm_data = load_result(args.vllm) - - by_scenario = defaultdict(dict) - for data in (sglang_data, vllm_data): - backend = data["metadata"]["engine"] - for s in data.get("scenarios", []): - key = s["name"] - by_scenario[key][backend] = s - - with open(args.output, "w", encoding="utf-8") as f: - f.write("# SGLang vs vLLM on DeepSeek-V4-Flash (H200, TP=8)\n\n") - f.write("## Summary\n\n") - f.write("- Model: `/data/models/DeepSeek-V4-Flash`\n") - f.write("- Hardware: 8x NVIDIA H200 143GB\n") - f.write("- Tensor Parallelism: 8\n") - f.write("- Benchmark client: `sglang.bench_serving`\n") - f.write("- SLO reference: S2 tier (TTFT P95 < 3s, TPOT < 50ms)\n\n") - - f.write("## Side-by-side results\n\n") - f.write("| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | SLO |\n") - f.write("|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n") - - for scenario_name in sorted(by_scenario.keys()): - for backend in ("sglang", "vllm"): - s = by_scenario[scenario_name].get(backend) - if s is None: - continue - cfg = s["config"] - m = s["metrics"] - status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"]) - f.write( - f"| {scenario_name} | {backend} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | " - f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | " - f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | " - f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | " - f"{m['e2e_ms']['p99']:.2f} | {status} |\n" - ) - - f.write("\n## Notes\n\n") - f.write("- SLO check uses TTFT P95 and TPOT mean (the same criteria as `scripts/SLO_STANDARDS.md`).\n") - f.write("- A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out.\n") - - print(f"Wrote comparison to {args.output}") - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_sglang_vs_vllm/parse_backend.py b/experiments/dsv4_h200_sglang_vs_vllm/parse_backend.py deleted file mode 100755 index d0c195f..0000000 --- a/experiments/dsv4_h200_sglang_vs_vllm/parse_backend.py +++ /dev/null @@ -1,221 +0,0 @@ -#!/usr/bin/env python3 -"""Parse raw sglang.bench_serving JSONL outputs for one backend. - -Reads JSONL files like {sglang|vllm}_phase1_MMDD_concurrency_inputlen_outputlen.jsonl -and generates results.json + report.md in the given result root. - -Usage: - python3 parse_backend.py [--backend sglang|vllm] -""" -import argparse -import json -import re -import sys -from pathlib import Path - - -def parse_jsonl(path: Path) -> dict | None: - with open(path, "r", encoding="utf-8") as f: - for line in f: - line = line.strip() - if not line: - continue - try: - return json.loads(line) - except json.JSONDecodeError: - continue - return None - - -def compute_metrics(data: dict) -> dict: - completed = data.get("completed", 0) - total = len(data.get("input_lens", [])) - failed = total - completed if total > 0 else 0 - duration_s = data.get("duration", 0.0) - - return { - "success": completed, - "failed": failed, - "duration_s": duration_s, - "request_throughput": data.get("request_throughput", 0.0), - "input_token_throughput": data.get("input_throughput", 0.0), - "output_token_throughput": data.get("output_throughput", 0.0), - "total_token_throughput": data.get("total_throughput", 0.0), - "total_input_tokens": data.get("total_input_tokens", 0), - "total_output_tokens": data.get("total_output_tokens", 0), - "e2e_ms": { - "mean": data.get("mean_e2e_latency_ms", 0.0), - "p50": data.get("median_e2e_latency_ms", 0.0), - "p90": data.get("p90_e2e_latency_ms", 0.0), - "p95": data.get("p95_e2e_latency_ms", 0.0), - "p99": data.get("p99_e2e_latency_ms", 0.0), - }, - "ttft_ms": { - "mean": data.get("mean_ttft_ms", 0.0), - "p50": data.get("median_ttft_ms", 0.0), - "p90": data.get("p90_ttft_ms", 0.0), - "p95": data.get("p95_ttft_ms", 0.0), - "p99": data.get("p99_ttft_ms", 0.0), - }, - "tpot_ms": { - "mean": data.get("mean_tpot_ms", 0.0), - "p50": data.get("median_tpot_ms", 0.0), - "p90": data.get("p90_tpot_ms", 0.0), - "p95": data.get("p95_tpot_ms", 0.0), - "p99": data.get("p99_tpot_ms", 0.0), - }, - "itl_ms": { - "mean": data.get("mean_itl_ms", 0.0), - "p50": data.get("median_itl_ms", 0.0), - "p90": data.get("p90_itl_ms", 0.0), - "p95": data.get("p95_itl_ms", 0.0), - "p99": data.get("p99_itl_ms", 0.0), - }, - } - - -def scenario_name(concurrency: int, input_len: int, output_len: int) -> str: - return f"c{concurrency}_i{input_len}_o{output_len}" - - -def slo_status(metrics: dict) -> dict: - """Check S2-tier SLO: TTFT P95 < 3s, TPOT mean < 50ms.""" - ttft_ok = metrics["ttft_ms"]["p95"] < 3000.0 - tpot_ok = metrics["tpot_ms"]["mean"] < 50.0 - if ttft_ok and tpot_ok: - mark = "✅" - elif ttft_ok or tpot_ok: - mark = "⚠️" - else: - mark = "❌" - return { - "ttft_p95_ok": ttft_ok, - "tpot_mean_ok": tpot_ok, - "overall": mark, - } - - -def append_scenario(results_json: Path, scenario: dict) -> None: - with open(results_json, "r", encoding="utf-8") as f: - data = json.load(f) - data["scenarios"].append(scenario) - with open(results_json, "w", encoding="utf-8") as f: - json.dump(data, f, indent=2, ensure_ascii=False) - - -def generate_report(result_root: Path, backend: str, scenarios: list[dict]) -> None: - report_path = result_root / "report.md" - with open(report_path, "w", encoding="utf-8") as f: - f.write(f"# H200 {backend.upper()} Comparison Benchmark Report\n\n") - f.write(f"- Result root: `{result_root}`\n") - f.write(f"- Model: `/data/models/DeepSeek-V4-Flash`\n") - f.write(f"- Backend: {backend.upper()} (TP=8)\n") - f.write(f"- Benchmark client: `sglang.bench_serving --backend {backend}`\n\n") - - f.write("## Results\n\n") - f.write("| Scenario | Phase | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO |\n") - f.write("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n") - - for s in scenarios: - cfg = s["config"] - m = s["metrics"] - slo = s.get("slo_status", {}).get("overall", "") - f.write( - f"| {s['name']} | {cfg['phase']} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | " - f"{m['duration_s']:.2f} | {m['success']} | {m['request_throughput']:.2f} | " - f"{m['input_token_throughput']:.2f} | {m['output_token_throughput']:.2f} | " - f"{m['total_token_throughput']:.2f} | " - f"{m['ttft_ms']['mean']:.2f} | {m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | " - f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | " - f"{m['e2e_ms']['mean']:.2f} | {m['e2e_ms']['p95']:.2f} | {m['e2e_ms']['p99']:.2f} | {slo} |\n" - ) - f.write("\n") - f.write("SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail.\n\n") - - -def main() -> None: - parser = argparse.ArgumentParser() - parser.add_argument("result_root", type=Path) - parser.add_argument("--backend", default=None, choices=["sglang", "vllm"]) - args = parser.parse_args() - - result_root = args.result_root - raw_dir = result_root / "raw_outputs" - results_json = result_root / "results.json" - - if not raw_dir.exists(): - raise SystemExit(f"raw_outputs directory not found: {raw_dir}") - - backend = args.backend - if backend is None: - # Infer from filenames if not provided. - for p in raw_dir.iterdir(): - if p.name.startswith("sglang_"): - backend = "sglang" - break - if p.name.startswith("vllm_"): - backend = "vllm" - break - if backend is None: - raise SystemExit("Could not infer backend from raw outputs") - - scenarios = [] - for jsonl_path in sorted(raw_dir.glob(f"{backend}_*.jsonl")): - parts = jsonl_path.stem.split("_") - if len(parts) < 5: - continue - - label = parts[1] - if label.startswith("phase"): - phase = label - dataset = "random" - elif label == "sharegpt": - phase = "sharegpt" - dataset = "sharegpt" - else: - continue - - # The last three numeric tokens are: concurrency, input_len/output_ctx, output_len. - try: - concurrency, input_len, output_len = int(parts[-3]), int(parts[-2]), int(parts[-1]) - except ValueError: - continue - - data = parse_jsonl(jsonl_path) - if data is None: - continue - - metrics = compute_metrics(data) - scenario = { - "name": scenario_name(concurrency, input_len, output_len), - "config": { - "phase": phase, - "concurrency": concurrency, - "input_len": input_len, - "output_len": output_len, - "dataset": dataset, - "num_prompts": metrics["success"] + metrics["failed"], - }, - "metrics": metrics, - "slo_status": slo_status(metrics), - "raw_file": str(jsonl_path), - } - scenarios.append(scenario) - - if not scenarios: - print("No benchmark outputs found to parse") - return - - if results_json.exists(): - with open(results_json, "r", encoding="utf-8") as f: - data = json.load(f) - data["scenarios"] = scenarios - with open(results_json, "w", encoding="utf-8") as f: - json.dump(data, f, indent=2, ensure_ascii=False) - - generate_report(result_root, backend, scenarios) - print(f"Parsed {len(scenarios)} scenarios into {result_root}/report.md") - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_sglang_vs_vllm/results/20260708-083346/comparison.md b/experiments/dsv4_h200_sglang_vs_vllm/results/20260708-083346/comparison.md index dfc98f0..8176b41 100644 --- a/experiments/dsv4_h200_sglang_vs_vllm/results/20260708-083346/comparison.md +++ b/experiments/dsv4_h200_sglang_vs_vllm/results/20260708-083346/comparison.md @@ -37,5 +37,5 @@ ## Notes -- SLO check uses TTFT P95 and TPOT mean (the same criteria as `scripts/SLO_STANDARDS.md`). +- SLO check uses TTFT P95 and TPOT mean (the same criteria as `docs/SLO_STANDARDS.md`). - A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out. diff --git a/experiments/dsv4_h200_sglang_vs_vllm/run_bench.sh b/experiments/dsv4_h200_sglang_vs_vllm/run_bench.sh index 56d738d..4299eed 100755 --- a/experiments/dsv4_h200_sglang_vs_vllm/run_bench.sh +++ b/experiments/dsv4_h200_sglang_vs_vllm/run_bench.sh @@ -100,7 +100,7 @@ run_warmup() { fi log "warming up ${backend} (phase=${phase}, input=${input_len}, output=${output_len}, num=${num})" - "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/warmup.py" \ + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/warmup.py" \ --backend "$backend" \ --port "$port" \ --input-len "$input_len" \ @@ -264,7 +264,7 @@ parse_backend() { local backend="$1" local result_root="${RESULT_BASE}/${RUN_ID}/${backend}" log "parsing ${backend} results in ${result_root}" - "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/parse_backend.py" "$result_root" --backend "$backend" \ + "${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/parse_backend.py" "$result_root" --backend "$backend" \ >> "${result_root}/logs/parse.log" 2>&1 || { log "WARNING: parser failed for ${backend}; see ${result_root}/logs/parse.log" } @@ -348,7 +348,7 @@ parse_backend vllm # Generate comparison. log "generating comparison report" -"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/compare.py" \ +"${VENV_CLIENT}/bin/python" "${SCRIPT_DIR}/../../scripts/common/compare.py" \ --sglang "${RESULT_BASE}/${RUN_ID}/sglang" \ --vllm "${RESULT_BASE}/${RUN_ID}/vllm" \ --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \ diff --git a/experiments/dsv4_h200_sglang_vs_vllm/warmup.py b/experiments/dsv4_h200_sglang_vs_vllm/warmup.py deleted file mode 100755 index 75dbe5f..0000000 --- a/experiments/dsv4_h200_sglang_vs_vllm/warmup.py +++ /dev/null @@ -1,76 +0,0 @@ -#!/usr/bin/env python3 -"""Send a small number of warmup requests to a running backend. - -Uses sglang.bench_serving with a single request so that the same code path -(prefill / decode kernels, CUDA graphs, etc.) is exercised before the real -benchmark begins. Discards the output. -""" -import argparse -import subprocess -import sys -import tempfile -from pathlib import Path - - -def run_warmup(backend: str, host: str, port: int, input_len: int, output_len: int, num: int, env_python: Path) -> None: - with tempfile.NamedTemporaryFile(mode="w", suffix=".jsonl", delete=True) as tmp: - cmd = [ - str(env_python), - "-m", - "sglang.bench_serving", - "--backend", - backend, - "--host", - host, - "--port", - str(port), - "--dataset-name", - "random", - "--random-input-len", - str(input_len), - "--random-output-len", - str(output_len), - "--num-prompts", - str(num), - "--max-concurrency", - "1", - "--request-rate", - "10000", - "--output-file", - tmp.name, - "--output-details", - ] - print(f"[warmup] {' '.join(cmd)}", flush=True) - result = subprocess.run(cmd, capture_output=True, text=True) - if result.returncode != 0: - print("[warmup] FAILED", file=sys.stderr) - print(result.stdout, file=sys.stderr) - print(result.stderr, file=sys.stderr) - sys.exit(1) - print(f"[warmup] OK: backend={backend} port={port} input={input_len} output={output_len} num={num}") - - -def main(): - parser = argparse.ArgumentParser(description="Warmup a serving backend.") - parser.add_argument("--backend", required=True, choices=["sglang", "vllm"]) - parser.add_argument("--port", type=int, required=True) - parser.add_argument("--input-len", type=int, required=True) - parser.add_argument("--output-len", type=int, required=True) - parser.add_argument("--num", type=int, default=1) - parser.add_argument("--host", default="127.0.0.1") - parser.add_argument("--env-python", default="/data/user1/yy/envs/sglang/bin/python") - args = parser.parse_args() - - run_warmup( - backend=args.backend, - host=args.host, - port=args.port, - input_len=args.input_len, - output_len=args.output_len, - num=args.num, - env_python=Path(args.env_python), - ) - - -if __name__ == "__main__": - main() diff --git a/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/comparison.md b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/comparison.md new file mode 100644 index 0000000..707388d --- /dev/null +++ b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/comparison.md @@ -0,0 +1,30 @@ +# vLLM+DSpark vs vLLM default on DeepSeek-V4-Flash (H200, TP=8) + +## Summary + +- DSpark model: `/data/models/DeepSeek-V4-Flash-DSpark` +- Default model: `/data/models/DeepSeek-V4-Flash` +- Hardware: 8x NVIDIA H200 143GB +- Tensor Parallelism: 8 +- Benchmark client: `sglang.bench_serving --backend vllm` +- DSpark flags: `--spec-method dspark --spec-model --spec-tokens 5` + +## Side-by-side results (TTFT focused) + +| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| c128_i512_o256 | vllm-dspark | 128 | 512 | 256 | 23.94 | 3160.81 | 778.36 | 1071.85 | 1075.35 | 28.95 | 78.75 | 3632.34 | +| c128_i512_o256 | vllm-default | 128 | 512 | 256 | 29.99 | 3959.89 | 617.72 | 767.18 | 770.89 | 18.31 | 65.44 | 2695.91 | +| c1_i4000_o512 | vllm-dspark | 1 | 4000 | 512 | 1.73 | 486.20 | 123.58 | 176.29 | 190.52 | 1.66 | 3.56 | 578.26 | +| c1_i4000_o512 | vllm-default | 1 | 4000 | 512 | 0.50 | 140.13 | 131.50 | 191.53 | 268.69 | 6.68 | 6.73 | 2008.09 | +| c1_i512_o256 | vllm-dspark | 1 | 512 | 256 | 2.56 | 372.62 | 86.51 | 185.00 | 188.81 | 2.09 | 3.83 | 389.62 | +| c1_i512_o256 | vllm-default | 1 | 512 | 256 | 0.99 | 143.82 | 49.50 | 68.50 | 71.09 | 6.66 | 6.71 | 1010.98 | +| c32_i4000_o512 | vllm-dspark | 32 | 4000 | 512 | 11.49 | 3242.88 | 298.42 | 434.24 | 522.34 | 9.68 | 54.60 | 2476.82 | +| c32_i4000_o512 | vllm-default | 32 | 4000 | 512 | 5.38 | 1518.49 | 382.74 | 681.98 | 684.75 | 16.43 | 25.49 | 4927.71 | +| c32_i512_o256 | vllm-dspark | 32 | 512 | 256 | 16.53 | 2182.83 | 262.56 | 527.21 | 532.11 | 12.06 | 28.16 | 1738.79 | +| c32_i512_o256 | vllm-default | 32 | 512 | 256 | 12.98 | 1713.20 | 122.62 | 272.00 | 304.80 | 15.77 | 23.26 | 2154.65 | + +## Notes + +- TTFT mean/P95/P99 are the main focus for verifying whether DSpark increases time-to-first-token. +- Mean TPOT and E2E are included to check whether speculative decoding pays back after first token. diff --git a/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/report.md b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/report.md new file mode 100644 index 0000000..90f024f --- /dev/null +++ b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/report.md @@ -0,0 +1,18 @@ +# H200 vLLM default Benchmark Report + +- Result root: `/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default` +- Backend: vLLM default (TP=8) +- Benchmark client: `sglang.bench_serving --backend vllm` + +## Results + +| Scenario | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| c128_i512_o256 | 128 | 512 | 256 | 4.27 | 128 | 29.99 | 8259.06 | 3959.89 | 12218.95 | 617.72 | 767.18 | 770.89 | 18.31 | 28.68 | 65.44 | 2695.91 | 4153.36 | 4205.54 | ✅ | +| c1_i4000_o512 | 1 | 4000 | 512 | 64.29 | 32 | 0.50 | 1011.09 | 140.13 | 1151.23 | 131.50 | 191.53 | 268.69 | 6.68 | 6.73 | 6.73 | 2008.09 | 3523.51 | 3599.32 | ✅ | +| c1_i512_o256 | 1 | 512 | 256 | 32.38 | 32 | 0.99 | 252.30 | 143.82 | 396.12 | 49.50 | 68.50 | 71.09 | 6.66 | 6.69 | 6.71 | 1010.98 | 1735.31 | 1759.78 | ✅ | +| c32_i4000_o512 | 32 | 4000 | 512 | 11.90 | 64 | 5.38 | 11297.00 | 1518.49 | 12815.49 | 382.74 | 681.98 | 684.75 | 16.43 | 20.90 | 25.49 | 4927.71 | 8743.37 | 9349.37 | ✅ | +| c32_i512_o256 | 32 | 512 | 256 | 9.86 | 128 | 12.98 | 3573.19 | 1713.20 | 5286.40 | 122.62 | 272.00 | 304.80 | 15.77 | 19.50 | 23.26 | 2154.65 | 4102.71 | 4275.05 | ✅ | + +SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail. + diff --git a/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/results.json b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/results.json new file mode 100644 index 0000000..44fc6e0 --- /dev/null +++ b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/results.json @@ -0,0 +1,304 @@ +{ + "metadata": { + "experiment": "dsv4_h200_vllm_dspark_vs_default_default", + "run_id": "20260708-142121", + "timestamp": "2026-07-08T14:21:25+00:00", + "model": "/data/models/DeepSeek-V4-Flash", + "backend": "vllm", + "engine": "vllm-default", + "hardware": "8x NVIDIA H200 143GB", + "accelerator": "NVIDIA H200", + "chip": "nvidia_h200", + "script": "experiments/dsv4_h200_vllm_dspark_vs_default/run_bench.sh", + "env": "/data/user1/yy/envs/vllm", + "git_commit": "1a6b173", + "git_dirty": "dirty", + "description": "H200 vllm-default TP=8 benchmark for TTFT comparison" + }, + "config": { + "tp": 8, + "cuda_visible_devices": "0,1,2,3,4,5,6,7", + "max_model_len": 32768, + "max_num_seqs": 256, + "backend": "default", + "server_start_script": "experiments/dsv4_h200_vllm_dspark_vs_default/start_default.sh", + "server_args": "vllm serve /data/models/DeepSeek-V4-Flash --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 32768 --max-num-seqs 256 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30004" + }, + "scenarios": [ + { + "name": "c128_i512_o256", + "config": { + "concurrency": 128, + "input_len": 512, + "output_len": 256, + "dataset": "random", + "num_prompts": 128 + }, + "metrics": { + "success": 128, + "failed": 0, + "duration_s": 4.267798511005822, + "request_throughput": 29.992043830071385, + "input_token_throughput": 8259.059069705907, + "output_token_throughput": 3959.8870369391125, + "total_token_throughput": 12218.94610664502, + "total_input_tokens": 35248, + "total_output_tokens": 16900, + "e2e_ms": { + "mean": 2695.9114590795252, + "p50": 2786.5102920040954, + "p90": 4094.3853520046105, + "p95": 4153.361671900348, + "p99": 4205.538639774604 + }, + "ttft_ms": { + "mean": 617.7214959924413, + "p50": 563.3532095089322, + "p90": 764.0312482006266, + "p95": 767.1767810017627, + "p99": 770.8853784023086 + }, + "tpot_ms": { + "mean": 18.314879062516017, + "p50": 15.941864871586276, + "p90": 23.19956007269117, + "p95": 28.68311282203317, + "p99": 65.43692430894995 + }, + "itl_ms": { + "mean": 15.86385086378528, + "p50": 14.933839993318543, + "p90": 17.340479693666566, + "p95": 18.231574493984226, + "p99": 21.184932928154012 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/raw_outputs/default_0708_128_512_256.jsonl" + }, + { + "name": "c1_i4000_o512", + "config": { + "concurrency": 1, + "input_len": 4000, + "output_len": 512, + "dataset": "random", + "num_prompts": 32 + }, + "metrics": { + "success": 32, + "failed": 0, + "duration_s": 64.28883961599786, + "request_throughput": 0.49775357886591887, + "input_token_throughput": 1011.0930666700768, + "output_token_throughput": 140.13318725009572, + "total_token_throughput": 1151.2262539201727, + "total_input_tokens": 65002, + "total_output_tokens": 9009, + "e2e_ms": { + "mean": 2008.0858264368544, + "p50": 2064.2024725020747, + "p90": 3374.621986798593, + "p95": 3523.513536003884, + "p99": 3599.319600486051 + }, + "ttft_ms": { + "mean": 131.49705237538, + "p50": 128.75337050354574, + "p90": 188.4824504086282, + "p95": 191.5273676990182, + "p99": 268.6876055170434 + }, + "tpot_ms": { + "mean": 6.678438385450706, + "p50": 6.647801321133651, + "p90": 6.729117710662969, + "p95": 6.730015678053446, + "p99": 6.730549147956696 + }, + "itl_ms": { + "mean": 6.689239366715674, + "p50": 6.692727998597547, + "p90": 6.8432597909122705, + "p95": 6.884838393307291, + "p99": 6.988287522690371 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/raw_outputs/default_0708_1_4000_512.jsonl" + }, + { + "name": "c1_i512_o256", + "config": { + "concurrency": 1, + "input_len": 512, + "output_len": 256, + "dataset": "random", + "num_prompts": 32 + }, + "metrics": { + "success": 32, + "failed": 0, + "duration_s": 32.381663249005214, + "request_throughput": 0.9882135995896709, + "input_token_throughput": 252.30328464523785, + "output_token_throughput": 143.81596041528428, + "total_token_throughput": 396.11924506052213, + "total_input_tokens": 8170, + "total_output_tokens": 4657, + "e2e_ms": { + "mean": 1010.982028125909, + "p50": 1195.345941996493, + "p90": 1688.3191696979338, + "p95": 1735.3133527984028, + "p99": 1759.781412960583 + }, + "ttft_ms": { + "mean": 49.50495168895941, + "p50": 44.729964502039365, + "p90": 65.38926539797103, + "p95": 68.496801002766, + "p99": 71.08588116068859 + }, + "tpot_ms": { + "mean": 6.659676006644546, + "p50": 6.654429546349644, + "p90": 6.684645557662856, + "p95": 6.69398312746385, + "p99": 6.711085371917718 + }, + "itl_ms": { + "mean": 6.652042307463047, + "p50": 6.637780999881215, + "p90": 6.787886397796684, + "p95": 6.841546000214294, + "p99": 7.143319878960029 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/raw_outputs/default_0708_1_512_256.jsonl" + }, + { + "name": "c32_i4000_o512", + "config": { + "concurrency": 32, + "input_len": 4000, + "output_len": 512, + "dataset": "random", + "num_prompts": 64 + }, + "metrics": { + "success": 64, + "failed": 0, + "duration_s": 11.899348942009965, + "request_throughput": 5.378445519321792, + "input_token_throughput": 11297.004622279226, + "output_token_throughput": 1518.4864388847727, + "total_token_throughput": 12815.491061163999, + "total_input_tokens": 134427, + "total_output_tokens": 18069, + "e2e_ms": { + "mean": 4927.714462639642, + "p50": 4734.132497498649, + "p90": 8425.151473002916, + "p95": 8743.374133249745, + "p99": 9349.369201347727 + }, + "ttft_ms": { + "mean": 382.74424496808024, + "p50": 390.3064074984286, + "p90": 678.5326104989508, + "p95": 681.9790261564776, + "p99": 684.7464643983403 + }, + "tpot_ms": { + "mean": 16.425838503257715, + "p50": 17.044603306082735, + "p90": 18.812656380989317, + "p95": 20.897049261926014, + "p99": 25.487691384272917 + }, + "itl_ms": { + "mean": 16.239184021720266, + "p50": 11.25292000506306, + "p90": 12.360760505544022, + "p95": 48.09924705259619, + "p99": 161.26204263797263 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/raw_outputs/default_0708_32_4000_512.jsonl" + }, + { + "name": "c32_i512_o256", + "config": { + "concurrency": 32, + "input_len": 512, + "output_len": 256, + "dataset": "random", + "num_prompts": 128 + }, + "metrics": { + "success": 128, + "failed": 0, + "duration_s": 9.864565095995204, + "request_throughput": 12.975736766333995, + "input_token_throughput": 3573.1935120292237, + "output_token_throughput": 1713.2027449300354, + "total_token_throughput": 5286.3962569592595, + "total_input_tokens": 35248, + "total_output_tokens": 16900, + "e2e_ms": { + "mean": 2154.6506077650065, + "p50": 2170.227252499899, + "p90": 3869.9233716019075, + "p95": 4102.707951706542, + "p99": 4275.054011319589 + }, + "ttft_ms": { + "mean": 122.61577058620787, + "p50": 76.25793749321019, + "p90": 266.052756505087, + "p95": 272.00344715092797, + "p99": 304.79746079756296 + }, + "tpot_ms": { + "mean": 15.771494928385172, + "p50": 16.114622466954064, + "p90": 17.704943566836604, + "p95": 19.49543390091434, + "p99": 23.258708736311746 + }, + "itl_ms": { + "mean": 15.513364437561801, + "p50": 11.447760996816214, + "p90": 26.607878498907667, + "p95": 30.321172751428094, + "p99": 100.95071544419625 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/default/raw_outputs/default_0708_32_512_256.jsonl" + } + ] +} \ No newline at end of file diff --git a/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/report.md b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/report.md new file mode 100644 index 0000000..a369c54 --- /dev/null +++ b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/report.md @@ -0,0 +1,18 @@ +# H200 vLLM+DSpark Benchmark Report + +- Result root: `/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark` +- Backend: vLLM+DSpark (TP=8) +- Benchmark client: `sglang.bench_serving --backend vllm` + +## Results + +| Scenario | Concurrency | Input | Output | Duration(s) | Success | Req/s | In tok/s | Out tok/s | Total tok/s | Mean TTFT(ms) | P95 TTFT(ms) | P99 TTFT(ms) | Mean TPOT(ms) | P95 TPOT(ms) | P99 TPOT(ms) | Mean E2E(ms) | P95 E2E(ms) | P99 E2E(ms) | SLO | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| c128_i512_o256 | 128 | 512 | 256 | 5.35 | 128 | 23.94 | 6592.44 | 3160.81 | 9753.24 | 778.36 | 1071.85 | 1075.35 | 28.95 | 69.00 | 78.75 | 3632.34 | 5022.66 | 5229.14 | ✅ | +| c1_i4000_o512 | 1 | 4000 | 512 | 18.53 | 32 | 1.73 | 3508.05 | 486.20 | 3994.25 | 123.58 | 176.29 | 190.52 | 1.66 | 2.30 | 3.56 | 578.26 | 964.23 | 1147.23 | ✅ | +| c1_i512_o256 | 1 | 512 | 256 | 12.50 | 32 | 2.56 | 653.70 | 372.62 | 1026.32 | 86.51 | 185.00 | 188.81 | 2.09 | 3.41 | 3.83 | 389.62 | 777.58 | 915.24 | ✅ | +| c32_i4000_o512 | 32 | 4000 | 512 | 5.57 | 64 | 11.49 | 24125.91 | 3242.88 | 27368.79 | 298.42 | 434.24 | 522.34 | 9.68 | 14.02 | 54.60 | 2476.82 | 4615.41 | 4710.26 | ✅ | +| c32_i512_o256 | 32 | 512 | 256 | 7.74 | 128 | 16.53 | 4552.68 | 2182.83 | 6735.51 | 262.56 | 527.21 | 532.11 | 12.06 | 22.73 | 28.16 | 1738.79 | 3830.80 | 5143.02 | ✅ | + +SLO: S2 tier — TTFT P95 < 3000ms, TPOT mean < 50ms. ✅ pass, ⚠️ partial, ❌ fail. + diff --git a/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/results.json b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/results.json new file mode 100644 index 0000000..7e03144 --- /dev/null +++ b/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/results.json @@ -0,0 +1,304 @@ +{ + "metadata": { + "experiment": "dsv4_h200_vllm_dspark_vs_default_dspark", + "run_id": "20260708-142121", + "timestamp": "2026-07-08T14:21:25+00:00", + "model": "/data/models/DeepSeek-V4-Flash-DSpark", + "backend": "vllm", + "engine": "vllm-dspark", + "hardware": "8x NVIDIA H200 143GB", + "accelerator": "NVIDIA H200", + "chip": "nvidia_h200", + "script": "experiments/dsv4_h200_vllm_dspark_vs_default/run_bench.sh", + "env": "/data/user1/yy/envs/vllm-dspark", + "git_commit": "1a6b173", + "git_dirty": "dirty", + "description": "H200 vllm-dspark TP=8 benchmark for TTFT comparison" + }, + "config": { + "tp": 8, + "cuda_visible_devices": "0,1,2,3,4,5,6,7", + "max_model_len": 32768, + "max_num_seqs": 256, + "backend": "dspark", + "server_start_script": "experiments/dsv4_h200_vllm_dspark_vs_default/start_dspark.sh", + "server_args": "vllm serve /data/models/DeepSeek-V4-Flash-DSpark --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len 32768 --max-num-seqs 256 --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --spec-method dspark --spec-model /data/models/DeepSeek-V4-Flash-DSpark --spec-tokens 5 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port 30007" + }, + "scenarios": [ + { + "name": "c128_i512_o256", + "config": { + "concurrency": 128, + "input_len": 512, + "output_len": 256, + "dataset": "random", + "num_prompts": 128 + }, + "metrics": { + "success": 128, + "failed": 0, + "duration_s": 5.346734229999129, + "request_throughput": 23.939847109255112, + "input_token_throughput": 6592.435397711127, + "output_token_throughput": 3160.807938643839, + "total_token_throughput": 9753.243336354966, + "total_input_tokens": 35248, + "total_output_tokens": 16900, + "e2e_ms": { + "mean": 3632.342373304141, + "p50": 4026.272206501744, + "p90": 4568.2280586028355, + "p95": 5022.660237102536, + "p99": 5229.135414937482 + }, + "ttft_ms": { + "mean": 778.3609630697583, + "p50": 860.7529874934698, + "p90": 1069.5359163975809, + "p95": 1071.847062408051, + "p99": 1075.3475757954584 + }, + "tpot_ms": { + "mean": 28.950386059739248, + "p50": 25.412556098403837, + "p90": 47.2479180524213, + "p95": 69.00331620594505, + "p99": 78.74700601221775 + }, + "itl_ms": { + "mean": 93.81264048253927, + "p50": 40.38110149849672, + "p90": 167.10851929528872, + "p95": 178.33137240304495, + "p99": 226.998560061038 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/raw_outputs/dspark_0708_128_512_256.jsonl" + }, + { + "name": "c1_i4000_o512", + "config": { + "concurrency": 1, + "input_len": 4000, + "output_len": 512, + "dataset": "random", + "num_prompts": 32 + }, + "metrics": { + "success": 32, + "failed": 0, + "duration_s": 18.529377975006355, + "request_throughput": 1.7269872762681893, + "input_token_throughput": 3508.050841624526, + "output_token_throughput": 486.2008866218786, + "total_token_throughput": 3994.2517282464046, + "total_input_tokens": 65002, + "total_output_tokens": 9009, + "e2e_ms": { + "mean": 578.2551125621467, + "p50": 591.3451755041024, + "p90": 892.7950516954298, + "p95": 964.232088495919, + "p99": 1147.2280103387313 + }, + "ttft_ms": { + "mean": 123.576120467078, + "p50": 116.67309649055824, + "p90": 165.92405038827567, + "p95": 176.29006390707215, + "p99": 190.51729560960666 + }, + "tpot_ms": { + "mean": 1.663761482756609, + "p50": 1.5393092801409114, + "p90": 1.721910965739694, + "p95": 2.2984976072946695, + "p99": 3.557179712464779 + }, + "itl_ms": { + "mean": 8.58824287840908, + "p50": 8.61248900037026, + "p90": 8.832874392101076, + "p95": 8.92088470282033, + "p99": 9.17154152484727 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/raw_outputs/dspark_0708_1_4000_512.jsonl" + }, + { + "name": "c1_i512_o256", + "config": { + "concurrency": 1, + "input_len": 512, + "output_len": 256, + "dataset": "random", + "num_prompts": 32 + }, + "metrics": { + "success": 32, + "failed": 0, + "duration_s": 12.498076534990105, + "request_throughput": 2.560393986259529, + "input_token_throughput": 653.7005896168861, + "output_token_throughput": 372.61733731283215, + "total_token_throughput": 1026.3179269297182, + "total_input_tokens": 8170, + "total_output_tokens": 4657, + "e2e_ms": { + "mean": 389.62139918930916, + "p50": 377.8842995016021, + "p90": 646.4735688015936, + "p95": 777.5833995023276, + "p99": 915.2371472108645 + }, + "ttft_ms": { + "mean": 86.50895553273585, + "p50": 65.89411200548057, + "p90": 180.50938470114488, + "p95": 184.9997164463275, + "p99": 188.8067441045132 + }, + "tpot_ms": { + "mean": 2.0946395465617687, + "p50": 1.810156284521167, + "p90": 3.2686956743572244, + "p95": 3.410268214368921, + "p99": 3.8341900713573533 + }, + "itl_ms": { + "mean": 8.582383006246063, + "p50": 8.498353497998323, + "p90": 8.866577497974504, + "p95": 9.046812405722449, + "p99": 11.975977418769617 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/raw_outputs/dspark_0708_1_512_256.jsonl" + }, + { + "name": "c32_i4000_o512", + "config": { + "concurrency": 32, + "input_len": 4000, + "output_len": 512, + "dataset": "random", + "num_prompts": 64 + }, + "metrics": { + "success": 64, + "failed": 0, + "duration_s": 5.57189327001106, + "request_throughput": 11.486221450877318, + "input_token_throughput": 24125.910796516957, + "output_token_throughput": 3242.883365560973, + "total_token_throughput": 27368.794162077927, + "total_input_tokens": 134427, + "total_output_tokens": 18069, + "e2e_ms": { + "mean": 2476.816028921121, + "p50": 2355.44883399416, + "p90": 4230.580038897461, + "p95": 4615.411171296, + "p99": 4710.255208882008 + }, + "ttft_ms": { + "mean": 298.4201381239018, + "p50": 329.9452685023425, + "p90": 424.3153512070421, + "p95": 434.23709184717154, + "p99": 522.339181041898 + }, + "tpot_ms": { + "mean": 9.682042273616121, + "p50": 7.969379892329501, + "p90": 10.570340490525057, + "p95": 14.016753123090474, + "p99": 54.6038999076698 + }, + "itl_ms": { + "mean": 42.37590053159988, + "p50": 21.612419492157642, + "p90": 113.86398609756725, + "p95": 132.79422069972497, + "p99": 167.44969745966955 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/raw_outputs/dspark_0708_32_4000_512.jsonl" + }, + { + "name": "c32_i512_o256", + "config": { + "concurrency": 32, + "input_len": 512, + "output_len": 256, + "dataset": "random", + "num_prompts": 128 + }, + "metrics": { + "success": 128, + "failed": 0, + "duration_s": 7.7422533049975755, + "request_throughput": 16.532654636522523, + "input_token_throughput": 4552.67977053239, + "output_token_throughput": 2182.8270574783646, + "total_token_throughput": 6735.506828010754, + "total_input_tokens": 35248, + "total_output_tokens": 16900, + "e2e_ms": { + "mean": 1738.7919132729621, + "p50": 1538.9780115001486, + "p90": 2865.2426461980212, + "p95": 3830.7956773496635, + "p99": 5143.022389483815 + }, + "ttft_ms": { + "mean": 262.5563681242511, + "p50": 225.63749099936103, + "p90": 521.6669989036745, + "p95": 527.2058025482693, + "p99": 532.1059286133095 + }, + "tpot_ms": { + "mean": 12.059512611469799, + "p50": 10.357537249850187, + "p90": 20.459189923756977, + "p95": 22.732079964678167, + "p99": 28.16360305609708 + }, + "itl_ms": { + "mean": 48.537332998996746, + "p50": 27.78467499592807, + "p90": 105.31790559180081, + "p95": 108.22369960369542, + "p99": 175.24975506879855 + } + }, + "slo_status": { + "ttft_p95_ok": true, + "tpot_mean_ok": true, + "overall": "✅" + }, + "raw_file": "/data/user1/yy/experiments/dsv4_h200_vllm_dspark_vs_default/results/20260708-142121/dspark/raw_outputs/dspark_0708_32_512_256.jsonl" + } + ] +} \ No newline at end of file diff --git a/scripts/analysis/compare_experiments.py b/scripts/analysis/compare_experiments.py deleted file mode 100644 index 0c4d4ee..0000000 --- a/scripts/analysis/compare_experiments.py +++ /dev/null @@ -1,77 +0,0 @@ -#!/usr/bin/env python3 -"""Load results.json from multiple experiments and produce a comparison report. - -Usage: - python3 scripts/analysis/compare_experiments.py - python3 scripts/analysis/compare_experiments.py --output comparison.md -""" - -import argparse -import json -from collections import defaultdict -from pathlib import Path - -REPO_ROOT = Path(__file__).resolve().parents[2] - - -def load_results(): - """Glob experiments/*/results/*/results.json and load them.""" - results = [] - for path in sorted(REPO_ROOT.glob("experiments/*/results/*/results.json")): - try: - with open(path, "r", encoding="utf-8") as f: - data = json.load(f) - data["_source"] = str(path.relative_to(REPO_ROOT)) - results.append(data) - except (json.JSONDecodeError, OSError) as e: - print(f"WARN: failed to load {path}: {e}") - return results - - -def main(): - parser = argparse.ArgumentParser(description="Compare benchmark experiments.") - parser.add_argument("--output", "-o", default="comparison_report.md", help="Output markdown file") - args = parser.parse_args() - - results = load_results() - if not results: - print("No experiments found under experiments/*/results/*/results.json") - return - - # Group by scenario name. - by_scenario = defaultdict(list) - for data in results: - meta = data.get("metadata", {}) - for scenario in data.get("scenarios", []): - key = scenario.get("name", "unknown") - by_scenario[key].append((meta, scenario)) - - output_path = Path(args.output) - with open(output_path, "w", encoding="utf-8") as f: - f.write("# Cross-Experiment Comparison\n\n") - f.write("| Experiment | Chip | Engine | Scenario | Conc | In/Out | Req/s | OutTok/s | TTFT p99 | TPOT p99 | E2E p99 |\n") - f.write("|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|\n") - - for scenario_name in sorted(by_scenario.keys()): - for meta, scenario in by_scenario[scenario_name]: - lat = scenario.get("latencies", {}) - f.write( - f"| {meta.get('experiment', '')} " - f"| {meta.get('chip', '')} " - f"| {meta.get('engine', '')} " - f"| {scenario_name} " - f"| {scenario.get('concurrency', '')} " - f"| {scenario.get('input_len', '')}/{scenario.get('output_len', '')} " - f"| {scenario.get('request_throughput') or ''} " - f"| {scenario.get('output_token_throughput') or ''} " - f"| {lat.get('ttft_ms', {}).get('p99') or ''} " - f"| {lat.get('tpot_ms', {}).get('p99') or ''} " - f"| {lat.get('e2e_ms', {}).get('p99') or ''} |\n" - ) - - print(f"Loaded {len(results)} experiment runs.") - print(f"Wrote {output_path}") - - -if __name__ == "__main__": - main() diff --git a/scripts/analyze_dspark_nsys.py b/scripts/analyze_dspark_nsys.py deleted file mode 100644 index bd8ffb9..0000000 --- a/scripts/analyze_dspark_nsys.py +++ /dev/null @@ -1,87 +0,0 @@ -#!/usr/bin/env python3 -"""Analyze nsys traces for DSpark profiling. - -Usage: - python scripts/analyze_dspark_nsys.py -""" - -import json -import subprocess -import sys -from pathlib import Path - - -def run_nsys_report(rep_file: Path, report_name: str) -> str: - cmd = [ - "nsys", "stats", - "--report", report_name, - "--format", "json", - str(rep_file), - ] - result = subprocess.run(cmd, capture_output=True, text=True) - if result.returncode != 0: - print(f"nsys {report_name} failed: {result.stderr}") - return "" - return result.stdout - - -def summarize_cuda_sum(rep_file: Path) -> list[dict]: - """Return top CUDA kernels by total time.""" - text = run_nsys_report(rep_file, "cuda_kernel_sum") - if not text: - return [] - # nsys stats --format json outputs JSON after some header lines. - # Find the first '[' character. - start = text.find("[") - if start < 0: - return [] - data = json.loads(text[start:]) - # Sort by total time descending. - rows = data[1:] if len(data) > 1 else data - rows_sorted = sorted(rows, key=lambda x: x.get("Total Time (ns)", 0), reverse=True) - return rows_sorted[:50] - - -def summarize_nvtx_sum(rep_file: Path) -> list[dict]: - text = run_nsys_report(rep_file, "nvtx_sum") - start = text.find("[") - if start < 0: - return [] - data = json.loads(text[start:]) - rows = data[1:] if len(data) > 1 else data - rows_sorted = sorted(rows, key=lambda x: x.get("Total Time (ns)", 0), reverse=True) - return rows_sorted[:30] - - -def main(): - if len(sys.argv) < 2: - print("Usage: analyze_dspark_nsys.py ") - sys.exit(1) - rep_file = Path(sys.argv[1]) - if not rep_file.exists(): - print(f"File not found: {rep_file}") - sys.exit(1) - - print(f"Analyzing {rep_file}...\n") - - cuda_kernels = summarize_cuda_sum(rep_file) - print("=== Top CUDA Kernels by Total Time ===") - for row in cuda_kernels[:20]: - name = row.get("Name", "N/A") - total_ns = row.get("Total Time (ns)", 0) - count = row.get("Instances", 0) - avg_ns = row.get("Avg (ns)", 0) - print(f" {total_ns/1e6:8.2f} ms {count:6d} calls avg={avg_ns/1e6:6.3f} ms {name[:80]}") - - nvtx = summarize_nvtx_sum(rep_file) - if nvtx: - print("\n=== Top NVTX Ranges ===") - for row in nvtx[:20]: - name = row.get("Range", "N/A") - total_ns = row.get("Total Time (ns)", 0) - count = row.get("Instances", 0) - print(f" {total_ns/1e6:8.2f} ms {count:6d} calls {name[:80]}") - - -if __name__ == "__main__": - main() diff --git a/scripts/bench_dsv4_comparison.py b/scripts/bench_dsv4_comparison.py deleted file mode 100644 index e9acf5d..0000000 --- a/scripts/bench_dsv4_comparison.py +++ /dev/null @@ -1,323 +0,0 @@ -#!/usr/bin/env python3 -"""Benchmark orchestration for DeepSeek-V4 inference comparison. - -Compares: -- vllm-dspark + DeepSeek-V4-Flash-DSpark + DSpark (various spec-tokens) -- vllm-dspark + DeepSeek-V4-Flash-DSpark without spec decode -- vllm (0.24.0) + DeepSeek-V4-Flash without spec decode -- sglang + DeepSeek-V4-Flash + EAGLE (reuse existing results) -""" - -import json -import os -import subprocess -import sys -import time -import urllib.request -from datetime import datetime -from pathlib import Path - -ROOT = Path("/data/user1/yy") -RESULT_DIR = ROOT / "bench_results" / f"dsv4_comparison_{datetime.now().strftime('%Y%m%d_%H%M%S')}" -RESULT_DIR.mkdir(parents=True, exist_ok=True) - -DATASET = "/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json" -NUM_PROMPTS = 200 -SEED = 42 -OUTPUT_LEN = 256 -HOST = "127.0.0.1" - -# Service configs to benchmark -SERVICES = [ - { - "name": "vllm-dspark-dspark-st5", - "engine": "vllm-dspark", - "model": "/data/models/DeepSeek-V4-Flash-DSpark", - "port": 30004, - "spec_method": "dspark", - "spec_tokens": 5, - "cmd": [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve", - "/data/models/DeepSeek-V4-Flash-DSpark", - "--trust-remote-code", - "--tensor-parallel-size", "8", - "--kv-cache-dtype", "fp8", - "--block-size", "256", - "--max-model-len", "auto", - "--max-num-seqs", "256", - "--tokenizer-mode", "deepseek_v4", - "--reasoning-parser", "deepseek_v4", - "--spec-method", "dspark", - "--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark", - "--spec-tokens", "5", - "--no-disable-hybrid-kv-cache-manager", - "--disable-uvicorn-access-log", - "--port", "30004", - ], - "env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")}, - "bench_backend": "openai", - }, - { - "name": "vllm-dspark-nospec", - "engine": "vllm-dspark", - "model": "/data/models/DeepSeek-V4-Flash-DSpark", - "port": 30004, - "spec_method": None, - "cmd": [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve", - "/data/models/DeepSeek-V4-Flash-DSpark", - "--trust-remote-code", - "--tensor-parallel-size", "8", - "--kv-cache-dtype", "fp8", - "--block-size", "256", - "--max-model-len", "auto", - "--max-num-seqs", "256", - "--tokenizer-mode", "deepseek_v4", - "--reasoning-parser", "deepseek_v4", - "--no-disable-hybrid-kv-cache-manager", - "--disable-uvicorn-access-log", - "--port", "30004", - ], - "env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")}, - "bench_backend": "openai", - }, - { - "name": "vllm-dspark-dspark-st3", - "engine": "vllm-dspark", - "model": "/data/models/DeepSeek-V4-Flash-DSpark", - "port": 30004, - "spec_method": "dspark", - "spec_tokens": 3, - "cmd": [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve", - "/data/models/DeepSeek-V4-Flash-DSpark", - "--trust-remote-code", - "--tensor-parallel-size", "8", - "--kv-cache-dtype", "fp8", - "--block-size", "256", - "--max-model-len", "auto", - "--max-num-seqs", "256", - "--tokenizer-mode", "deepseek_v4", - "--reasoning-parser", "deepseek_v4", - "--spec-method", "dspark", - "--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark", - "--spec-tokens", "3", - "--no-disable-hybrid-kv-cache-manager", - "--disable-uvicorn-access-log", - "--port", "30004", - ], - "env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")}, - "bench_backend": "openai", - }, - { - "name": "vllm-dspark-dspark-st7", - "engine": "vllm-dspark", - "model": "/data/models/DeepSeek-V4-Flash-DSpark", - "port": 30004, - "spec_method": "dspark", - "spec_tokens": 7, - "cmd": [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve", - "/data/models/DeepSeek-V4-Flash-DSpark", - "--trust-remote-code", - "--tensor-parallel-size", "8", - "--kv-cache-dtype", "fp8", - "--block-size", "256", - "--max-model-len", "auto", - "--max-num-seqs", "256", - "--tokenizer-mode", "deepseek_v4", - "--reasoning-parser", "deepseek_v4", - "--spec-method", "dspark", - "--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark", - "--spec-tokens", "7", - "--no-disable-hybrid-kv-cache-manager", - "--disable-uvicorn-access-log", - "--port", "30004", - ], - "env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")}, - "bench_backend": "openai", - }, - { - "name": "vllm-main-nospec", - "engine": "vllm", - "model": "/data/models/DeepSeek-V4-Flash", - "port": 30005, - "spec_method": None, - "cmd": [ - "/data/user1/yy/envs/vllm/bin/vllm", "serve", - "/data/models/DeepSeek-V4-Flash", - "--trust-remote-code", - "--tensor-parallel-size", "8", - "--kv-cache-dtype", "fp8", - "--block-size", "256", - "--max-model-len", "auto", - "--max-num-seqs", "256", - "--tokenizer-mode", "deepseek_v4", - "--reasoning-parser", "deepseek_v4", - "--disable-uvicorn-access-log", - "--port", "30005", - ], - "env": {"CUDA_VISIBLE_DEVICES": "0,1,2,3,4,5,6,7", "TMPDIR": str(ROOT / "tmp")}, - "bench_backend": "openai", - }, -] - -CONCURRENCIES = [1, 16, 64] - - -def log(msg): - print(f"[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] {msg}", flush=True) - - -def wait_for_health(port, timeout=300): - url = f"http://{HOST}:{port}/health" - start = time.time() - while time.time() - start < timeout: - try: - with urllib.request.urlopen(url, timeout=5) as resp: - if resp.status == 200: - return True - except Exception: - pass - time.sleep(2) - return False - - -def start_service(service): - log(f"Starting {service['name']} on port {service['port']}...") - env = os.environ.copy() - env.update(service["env"]) - log_file = RESULT_DIR / f"{service['name']}_service.log" - proc = subprocess.Popen( - service["cmd"], - stdout=open(log_file, "w"), - stderr=subprocess.STDOUT, - env=env, - ) - if not wait_for_health(service["port"]): - log(f"ERROR: {service['name']} failed to start") - proc.terminate() - return None - log(f"{service['name']} is ready") - return proc - - -def stop_service(proc, name): - if proc is None: - return - log(f"Stopping {name} (pid {proc.pid})...") - proc.terminate() - try: - proc.wait(timeout=60) - except subprocess.TimeoutExpired: - proc.kill() - proc.wait() - log(f"{name} stopped") - - -def run_benchmark(service, concurrency): - name = service["name"] - port = service["port"] - backend = service["bench_backend"] - result_file = RESULT_DIR / f"{name}_c{concurrency}.json" - log_file = RESULT_DIR / f"{name}_c{concurrency}.log" - - if service["engine"] == "vllm-dspark": - bench_cmd = [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "bench", "serve", - "--host", HOST, - "--port", str(port), - "--backend", backend, - "--dataset-name", "sharegpt", - "--dataset-path", DATASET, - "--sharegpt-output-len", str(OUTPUT_LEN), - "--num-prompts", str(NUM_PROMPTS), - "--max-concurrency", str(concurrency), - "--endpoint", "/v1/completions", - "--model", service["model"], - "--seed", str(SEED), - "--save-result", - "--result-dir", str(RESULT_DIR), - "--result-filename", result_file.name, - ] - else: - bench_cmd = [ - "/data/user1/yy/envs/vllm/bin/vllm", "bench", "serve", - "--host", HOST, - "--port", str(port), - "--backend", backend, - "--dataset-name", "sharegpt", - "--dataset-path", DATASET, - "--sharegpt-output-len", str(OUTPUT_LEN), - "--num-prompts", str(NUM_PROMPTS), - "--max-concurrency", str(concurrency), - "--endpoint", "/v1/completions", - "--model", service["model"], - "--seed", str(SEED), - "--save-result", - "--result-dir", str(RESULT_DIR), - "--result-filename", result_file.name, - ] - - log(f"Running benchmark {name} concurrency={concurrency}...") - start = time.time() - with open(log_file, "w") as f: - proc = subprocess.Popen(bench_cmd, stdout=f, stderr=subprocess.STDOUT) - proc.wait() - duration = time.time() - start - log(f"Benchmark {name} c={concurrency} finished in {duration:.1f}s, exit={proc.returncode}") - - if result_file.exists(): - with open(result_file) as f: - data = json.load(f) - return { - "service": name, - "engine": service["engine"], - "spec_method": service.get("spec_method"), - "spec_tokens": service.get("spec_tokens"), - "concurrency": concurrency, - "request_throughput": data.get("request_throughput"), - "output_throughput": data.get("output_throughput"), - "total_input_tokens": data.get("total_input_tokens"), - "total_output_tokens": data.get("total_output_tokens"), - "duration_s": data.get("duration_s"), - "result_file": str(result_file), - } - else: - log(f"WARNING: result file {result_file} not found") - return { - "service": name, - "engine": service["engine"], - "spec_method": service.get("spec_method"), - "spec_tokens": service.get("spec_tokens"), - "concurrency": concurrency, - "error": "result file missing", - "log_file": str(log_file), - } - - -def main(): - summary = [] - for service in SERVICES: - proc = start_service(service) - if proc is None: - continue - try: - for concurrency in CONCURRENCIES: - result = run_benchmark(service, concurrency) - summary.append(result) - # Save incremental summary - with open(RESULT_DIR / "summary.json", "w") as f: - json.dump(summary, f, indent=2) - finally: - stop_service(proc, service["name"]) - # Small gap between services - time.sleep(10) - - log(f"All benchmarks complete. Results in {RESULT_DIR}") - with open(RESULT_DIR / "summary.json", "w") as f: - json.dump(summary, f, indent=2) - - -if __name__ == "__main__": - main() diff --git a/scripts/bench_vllm_dspark_qwen3.py b/scripts/bench_vllm_dspark_qwen3.py deleted file mode 100644 index ff427d9..0000000 --- a/scripts/bench_vllm_dspark_qwen3.py +++ /dev/null @@ -1,266 +0,0 @@ -#!/usr/bin/env python3 -"""Throughput benchmark for vLLM DSpark service (Qwen3-4B + dspark_qwen3_4b_block7).""" - -import argparse -import asyncio -import json -import random -import time -from dataclasses import dataclass, field -from datetime import datetime -from typing import List - -import aiohttp -import numpy as np -from transformers import AutoTokenizer - - -@dataclass -class RequestResult: - prompt_len: int = 0 - output_len: int = 0 - ttft_ms: float = 0.0 - tpot_ms: float = 0.0 - e2e_ms: float = 0.0 - success: bool = False - error: str = "" - - -async def async_request( - session: aiohttp.ClientSession, - url: str, - model_name: str, - prompt: str, - max_tokens: int, - prompt_len: int, - result: RequestResult, -) -> None: - payload = { - "model": model_name, - "prompt": prompt, - "max_tokens": max_tokens, - "temperature": 0.0, - "stream": True, - "stream_options": {"include_usage": True}, - } - result.prompt_len = prompt_len - start = time.perf_counter() - first_token_time = None - token_times: List[float] = [] - output_text = "" - try: - async with session.post(url, json=payload) as resp: - resp.raise_for_status() - async for line in resp.content: - line = line.decode("utf-8").strip() - if not line or line.startswith(":"): - continue - if line.startswith("data: "): - data = line[len("data: "):] - if data == "[DONE]": - break - chunk = json.loads(data) - choices = chunk.get("choices", []) - if choices: - delta = choices[0].get("text", "") - if delta: - now = time.perf_counter() - token_times.append(now) - output_text += delta - if first_token_time is None: - first_token_time = now - end = time.perf_counter() - result.e2e_ms = (end - start) * 1000.0 - if first_token_time is not None: - result.ttft_ms = (first_token_time - start) * 1000.0 - if len(token_times) >= 2: - # TPOT: average time between consecutive tokens - intervals = [ - token_times[i] - token_times[i - 1] - for i in range(1, len(token_times)) - ] - result.tpot_ms = sum(intervals) / len(intervals) * 1000.0 - result.output_len = len(token_times) - result.success = True - except Exception as e: - result.e2e_ms = (time.perf_counter() - start) * 1000.0 - result.error = str(e) - result.success = False - - -async def run_concurrency_benchmark( - url: str, - model_name: str, - prompts: List[str], - prompt_lens: List[int], - max_tokens: int, - concurrency: int, - num_prompts: int, - seed: int, -) -> dict: - rng = random.Random(seed) - indices = [rng.randrange(len(prompts)) for _ in range(num_prompts)] - - semaphore = asyncio.Semaphore(concurrency) - results: List[RequestResult] = [RequestResult() for _ in range(num_prompts)] - - async def bounded_request(idx: int, i: int): - async with semaphore: - async with aiohttp.ClientSession() as session: - await async_request( - session, - url, - model_name, - prompts[idx], - max_tokens, - prompt_lens[idx], - results[i], - ) - - start = time.perf_counter() - await asyncio.gather(*[bounded_request(idx, i) for i, idx in enumerate(indices)]) - duration = time.perf_counter() - start - - successes = [r for r in results if r.success] - failures = [r for r in results if not r.success] - if not successes: - return { - "concurrency": concurrency, - "num_prompts": num_prompts, - "duration_s": duration, - "success_count": 0, - "error": "all requests failed", - "errors": [r.error for r in failures[:5]], - } - - total_in_tokens = sum(r.prompt_len for r in successes) - total_out_tokens = sum(r.output_len for r in successes) - total_tokens = total_in_tokens + total_out_tokens - - ttfts = [r.ttft_ms for r in successes] - tpots = [r.tpot_ms for r in successes] - e2es = [r.e2e_ms for r in successes] - - return { - "concurrency": concurrency, - "num_prompts": num_prompts, - "duration_s": duration, - "success_count": len(successes), - "fail_count": len(failures), - "input_tokens": total_in_tokens, - "output_tokens": total_out_tokens, - "total_tokens": total_tokens, - "request_throughput": len(successes) / duration, - "input_throughput": total_in_tokens / duration, - "output_throughput": total_out_tokens / duration, - "total_throughput": total_tokens / duration, - "mean_ttft_ms": float(np.mean(ttfts)), - "p50_ttft_ms": float(np.percentile(ttfts, 50)), - "p99_ttft_ms": float(np.percentile(ttfts, 99)), - "mean_tpot_ms": float(np.mean(tpots)), - "p50_tpot_ms": float(np.percentile(tpots, 50)), - "p99_tpot_ms": float(np.percentile(tpots, 99)), - "mean_e2e_ms": float(np.mean(e2es)), - "p50_e2e_ms": float(np.percentile(e2es, 50)), - "p99_e2e_ms": float(np.percentile(e2es, 99)), - "errors": [r.error for r in failures[:5]], - } - - -def load_sharegpt_prompts(path: str, tokenizer, max_samples: int = 2000): - with open(path, "r") as f: - data = json.load(f) - prompts = [] - lens = [] - for item in data: - conv = item.get("conversations", []) - for turn in conv: - if turn.get("from") == "human": - prompt = turn.get("value", "") - if prompt: - prompts.append(prompt) - lens.append(len(tokenizer.encode(prompt))) - break - if len(prompts) >= max_samples: - break - return prompts, lens - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--host", default="127.0.0.1") - parser.add_argument("--port", type=int, default=30003) - parser.add_argument("--model", default="/data/models/Qwen3-4B") - parser.add_argument("--dataset", default="/data/user1/yy/datasets/ShareGPT_filtered_chat.json") - parser.add_argument("--tokenizer", default="/data/models/Qwen3-4B") - parser.add_argument("--max-tokens", type=int, default=256) - parser.add_argument("--num-prompts", type=int, default=500) - parser.add_argument("--concurrency", type=int, nargs="+", default=[1, 4, 16, 64, 128]) - parser.add_argument("--seed", type=int, default=42) - parser.add_argument("--output-dir", default="/data/user1/yy/bench_results") - args = parser.parse_args() - - url = f"http://{args.host}:{args.port}/v1/completions" - print(f"Loading tokenizer from {args.tokenizer} ...") - tokenizer = AutoTokenizer.from_pretrained(args.tokenizer) - print(f"Loading dataset from {args.dataset} ...") - prompts, prompt_lens = load_sharegpt_prompts(args.dataset, tokenizer) - print(f"Loaded {len(prompts)} prompts, prompt lens: min={min(prompt_lens)}, max={max(prompt_lens)}, mean={sum(prompt_lens)/len(prompt_lens):.1f}") - - timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") - result_dir = f"{args.output_dir}/vllm_dspark_qwen3_{timestamp}" - import os - os.makedirs(result_dir, exist_ok=True) - summary_file = f"{result_dir}/summary.json" - - summary = { - "model": args.model, - "draft_model": "deepseek-ai/dspark_qwen3_4b_block7", - "url": url, - "max_tokens": args.max_tokens, - "num_prompts": args.num_prompts, - "dataset": args.dataset, - "timestamp": timestamp, - "results": [], - } - - print("\nStarting benchmark sweeps...") - print(f"{'Concurrency':>12} {'Req/s':>10} {'In tok/s':>12} {'Out tok/s':>12} {'Total tok/s':>13} {'TTFT(ms)':>10} {'TPOT(ms)':>10} {'E2E(ms)':>10}") - print("-" * 100) - - for c in args.concurrency: - print(f"Running concurrency={c} ...", flush=True) - result = asyncio.run( - run_concurrency_benchmark( - url, - args.model, - prompts, - prompt_lens, - args.max_tokens, - c, - args.num_prompts, - args.seed, - ) - ) - summary["results"].append(result) - with open(summary_file, "w") as f: - json.dump(summary, f, indent=2) - if result.get("success_count", 0) == 0: - print(f"{c:>12} FAILED: {result.get('error', 'unknown')}") - continue - print( - f"{c:>12} " - f"{result['request_throughput']:>10.2f} " - f"{result['input_throughput']:>12.2f} " - f"{result['output_throughput']:>12.2f} " - f"{result['total_throughput']:>13.2f} " - f"{result['mean_ttft_ms']:>10.1f} " - f"{result['mean_tpot_ms']:>10.1f} " - f"{result['mean_e2e_ms']:>10.1f}" - ) - - print(f"\nSummary saved to {summary_file}") - - -if __name__ == "__main__": - main() diff --git a/scripts/benchmark_dsv4_backend_comparison.sh b/scripts/benchmark_dsv4_backend_comparison.sh deleted file mode 100644 index 5e7561e..0000000 --- a/scripts/benchmark_dsv4_backend_comparison.sh +++ /dev/null @@ -1,128 +0,0 @@ -#!/usr/bin/env bash -set -Eeuo pipefail - -# Reproducible SGLang vs vLLM backend comparison for DeepSeek-V4-Flash. -# Produces JSONL raw outputs compatible with the legacy 2026-07-07 sweep. -# -# Usage: -# bash scripts/benchmark_dsv4_backend_comparison.sh [vllm|sglang|all] -# -# Output: -# bench_results/dsv4_backend_comparison_${RUN_ID}/raw_outputs/ - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -ROOT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" - -BACKEND_FILTER="${1:-all}" -RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" -RESULT_ROOT="${RESULT_ROOT:-${ROOT_DIR}/bench_results/dsv4_backend_comparison_${RUN_ID}}" -RAW_DIR="${RESULT_ROOT}/raw_outputs" -LOG_DIR="${RESULT_ROOT}/logs" - -mkdir -p "${RAW_DIR}" "${LOG_DIR}" - -BENCH_PY="${ROOT_DIR}/envs/sglang/bin/python" -DATASET="${ROOT_DIR}/datasets/ShareGPT_filtered_chat.json" -MODEL="/data/models/DeepSeek-V4-Flash" -SERVED_MODEL_NAME="deepseek-v4-flash" -NUM_PROMPTS=512 -WARMUP=100 - -log() { - echo "[$(date --iso-8601=seconds)] $*" | tee -a "${LOG_DIR}/orchestrator.log" -} - -run_case() { - local backend="$1" - local port="$2" - local concurrency="$3" - local input_len="$4" - local output_len="$5" - local output_file="${RAW_DIR}/${backend}_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl" - local detail_log="${LOG_DIR}/${backend}_c${concurrency}_i${input_len}_o${output_len}.log" - - log "running ${backend}: concurrency=${concurrency} input=${input_len} output=${output_len} -> ${output_file}" - - "${BENCH_PY}" -m sglang.bench_serving \ - --backend "${backend}" \ - --host 127.0.0.1 \ - --port "${port}" \ - --model "${MODEL}" \ - --served-model-name "${SERVED_MODEL_NAME}" \ - --dataset-name random \ - --dataset-path "${DATASET}" \ - --random-input-len "${input_len}" \ - --random-output-len "${output_len}" \ - --num-prompts "${NUM_PROMPTS}" \ - --max-concurrency "${concurrency}" \ - --warmup-requests "${WARMUP}" \ - --output-file "${output_file}" \ - --output-details \ - > "${detail_log}" 2>&1 - - log "finished ${backend} c=${concurrency} i=${input_len} o=${output_len}" -} - -main() { - log "run_id=${RUN_ID}" - log "result_root=${RESULT_ROOT}" - log "backend_filter=${BACKEND_FILTER}" - - # This script assumes a healthy server is already running on the target port. - if [[ "${BACKEND_FILTER}" == "all" || "${BACKEND_FILTER}" == "sglang" ]]; then - log "===== SGLang sweep (port 30000) =====" - run_case sglang 30000 32 512 256 - run_case sglang 30000 128 512 256 - run_case sglang 30000 256 512 256 - run_case sglang 30000 512 512 256 - run_case sglang 30000 512 512 2000 - run_case sglang 30000 32 4000 512 - run_case sglang 30000 128 4000 512 - run_case sglang 30000 512 4000 512 - run_case sglang 30000 32 16000 512 - run_case sglang 30000 128 16000 512 - run_case sglang 30000 512 1000 256 - run_case sglang 30000 768 1000 256 - run_case sglang 30000 1024 1000 256 - run_case sglang 30000 512 1000 1000 - fi - - if [[ "${BACKEND_FILTER}" == "all" || "${BACKEND_FILTER}" == "vllm" ]]; then - log "===== vLLM sweep (port 8000) =====" - run_case vllm 8000 32 512 256 - run_case vllm 8000 128 512 256 - run_case vllm 8000 256 512 256 - run_case vllm 8000 128 512 2000 - run_case vllm 8000 256 512 2000 - run_case vllm 8000 512 512 2000 - run_case vllm 8000 32 1000 256 - run_case vllm 8000 128 1000 256 - run_case vllm 8000 256 1000 256 - run_case vllm 8000 512 1000 256 - run_case vllm 8000 768 1000 256 - run_case vllm 8000 1024 1000 256 - run_case vllm 8000 32 1000 1000 - run_case vllm 8000 128 1000 1000 - run_case vllm 8000 256 1000 1000 - run_case vllm 8000 512 1000 1000 - run_case vllm 8000 1024 1000 1000 - run_case vllm 8000 32 4000 512 - run_case vllm 8000 128 4000 512 - run_case vllm 8000 256 4000 512 - run_case vllm 8000 512 4000 512 - run_case vllm 8000 32 8000 1000 - run_case vllm 8000 128 8000 1000 - run_case vllm 8000 256 8000 1000 - run_case vllm 8000 512 8000 1000 - run_case vllm 8000 32 16000 512 - run_case vllm 8000 128 16000 512 - run_case vllm 8000 256 16000 512 - run_case vllm 8000 32 32000 512 - run_case vllm 8000 128 32000 512 - fi - - log "===== Done =====" - log "all outputs saved to ${RAW_DIR}" -} - -main "$@" diff --git a/experiments/dsv4_h200_256k_4k_probe/compare.py b/scripts/common/compare.py similarity index 76% rename from experiments/dsv4_h200_256k_4k_probe/compare.py rename to scripts/common/compare.py index 0f4eb51..a29dfec 100755 --- a/experiments/dsv4_h200_256k_4k_probe/compare.py +++ b/scripts/common/compare.py @@ -17,10 +17,9 @@ def load_result(result_root: Path) -> dict: return json.load(f) -def slo_status(ttft_p95_ms: float, tpot_mean_ms: float) -> str: - # S2 tier targets from scripts/SLO_STANDARDS.md. - ttft_ok = ttft_p95_ms < 3000.0 - tpot_ok = tpot_mean_ms < 50.0 +def slo_status(ttft_p95_ms: float, tpot_mean_ms: float, ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str: + ttft_ok = ttft_p95_ms < ttft_limit_ms + tpot_ok = tpot_mean_ms < tpot_limit_ms if ttft_ok and tpot_ok: return "✅" if ttft_ok or tpot_ok: @@ -33,11 +32,16 @@ def main(): parser.add_argument("--sglang", type=Path, required=True) parser.add_argument("--vllm", type=Path, required=True) parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md")) + parser.add_argument("--ttft-limit", type=float, default=3000.0) + parser.add_argument("--tpot-limit", type=float, default=50.0) args = parser.parse_args() sglang_data = load_result(args.sglang) vllm_data = load_result(args.vllm) + model = sglang_data.get("metadata", {}).get("model", "unknown") + hardware = sglang_data.get("metadata", {}).get("hardware", "unknown") + by_scenario = defaultdict(dict) for data in (sglang_data, vllm_data): backend = data["metadata"]["engine"] @@ -46,13 +50,12 @@ def main(): by_scenario[key][backend] = s with open(args.output, "w", encoding="utf-8") as f: - f.write("# SGLang vs vLLM on DeepSeek-V4-Flash (H200, TP=8)\n\n") + f.write(f"# SGLang vs vLLM ({hardware})\n\n") f.write("## Summary\n\n") - f.write("- Model: `/data/models/DeepSeek-V4-Flash`\n") - f.write("- Hardware: 8x NVIDIA H200 143GB\n") - f.write("- Tensor Parallelism: 8\n") + f.write(f"- Model: `{model}`\n") + f.write(f"- Hardware: {hardware}\n") f.write("- Benchmark client: `sglang.bench_serving`\n") - f.write("- SLO reference: S2 tier (TTFT P95 < 3s, TPOT < 50ms)\n\n") + f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n") f.write("## Side-by-side results\n\n") f.write("| Scenario | Backend | Conc | Input | Output | Req/s | OutTok/s | TTFT P95(ms) | TTFT P99(ms) | TPOT Mean(ms) | TPOT P95(ms) | TPOT P99(ms) | E2E P99(ms) | SLO |\n") @@ -65,7 +68,7 @@ def main(): continue cfg = s["config"] m = s["metrics"] - status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"]) + status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit) f.write( f"| {scenario_name} | {backend} | {cfg['concurrency']} | {cfg['input_len']} | {cfg['output_len']} | " f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | " @@ -75,7 +78,7 @@ def main(): ) f.write("\n## Notes\n\n") - f.write("- SLO check uses TTFT P95 and TPOT mean (the same criteria as `scripts/SLO_STANDARDS.md`).\n") + f.write("- SLO check uses TTFT P95 and TPOT mean.\n") f.write("- A ⚠️ indicates one of the two metrics is out of target; ❌ indicates both are out.\n") print(f"Wrote comparison to {args.output}") diff --git a/experiments/dsv4_h200_long_context_matrix/parse_backend.py b/scripts/common/parse_backend.py similarity index 86% rename from experiments/dsv4_h200_long_context_matrix/parse_backend.py rename to scripts/common/parse_backend.py index 935d870..f07e6c6 100755 --- a/experiments/dsv4_h200_long_context_matrix/parse_backend.py +++ b/scripts/common/parse_backend.py @@ -1,16 +1,14 @@ #!/usr/bin/env python3 """Parse raw sglang.bench_serving JSONL outputs for one backend. -Reads JSONL files like {sglang|vllm}_phase1_MMDD_concurrency_inputlen_outputlen.jsonl -and generates results.json + report.md in the given result root. +Reads JSONL files like {backend}_{label}_MMDD_concurrency_inputlen_outputlen.jsonl +and updates results.json + report.md in the given result root. Usage: python3 parse_backend.py [--backend sglang|vllm] """ import argparse import json -import re -import sys from pathlib import Path @@ -78,10 +76,10 @@ def scenario_name(concurrency: int, input_len: int, output_len: int) -> str: return f"c{concurrency}_i{input_len}_o{output_len}" -def slo_status(metrics: dict) -> dict: - """Check S2-tier SLO: TTFT P95 < 3s, TPOT mean < 50ms.""" - ttft_ok = metrics["ttft_ms"]["p95"] < 3000.0 - tpot_ok = metrics["tpot_ms"]["mean"] < 50.0 +def slo_status(metrics: dict, ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> dict: + """Check SLO: TTFT P95 < limit, TPOT mean < limit. Defaults to S2 tier.""" + ttft_ok = metrics["ttft_ms"]["p95"] < ttft_limit_ms + tpot_ok = metrics["tpot_ms"]["mean"] < tpot_limit_ms if ttft_ok and tpot_ok: mark = "✅" elif ttft_ok or tpot_ok: @@ -95,21 +93,16 @@ def slo_status(metrics: dict) -> dict: } -def append_scenario(results_json: Path, scenario: dict) -> None: - with open(results_json, "r", encoding="utf-8") as f: - data = json.load(f) - data["scenarios"].append(scenario) - with open(results_json, "w", encoding="utf-8") as f: - json.dump(data, f, indent=2, ensure_ascii=False) - - -def generate_report(result_root: Path, backend: str, scenarios: list[dict]) -> None: +def generate_report(result_root: Path, backend: str, scenarios: list[dict], metadata: dict | None) -> None: report_path = result_root / "report.md" + model = metadata.get("model", "unknown") if metadata else "unknown" + hardware = metadata.get("hardware", "unknown") if metadata else "unknown" + with open(report_path, "w", encoding="utf-8") as f: - f.write(f"# H200 {backend.upper()} Comparison Benchmark Report\n\n") + f.write(f"# {hardware} {backend.upper()} Benchmark Report\n\n") f.write(f"- Result root: `{result_root}`\n") - f.write(f"- Model: `/data/models/DeepSeek-V4-Flash`\n") - f.write(f"- Backend: {backend.upper()} (TP=8)\n") + f.write(f"- Model: `{model}`\n") + f.write(f"- Backend: {backend.upper()}\n") f.write(f"- Benchmark client: `sglang.bench_serving --backend {backend}`\n\n") f.write("## Results\n\n") @@ -148,7 +141,6 @@ def main() -> None: backend = args.backend if backend is None: - # Infer from filenames if not provided. for p in raw_dir.iterdir(): if p.name.startswith("sglang_"): backend = "sglang" @@ -159,6 +151,14 @@ def main() -> None: if backend is None: raise SystemExit("Could not infer backend from raw outputs") + metadata = None + if results_json.exists(): + with open(results_json, "r", encoding="utf-8") as f: + try: + metadata = json.load(f).get("metadata") + except json.JSONDecodeError: + pass + scenarios = [] for jsonl_path in sorted(raw_dir.glob(f"{backend}_*.jsonl")): parts = jsonl_path.stem.split("_") @@ -173,7 +173,6 @@ def main() -> None: phase = label dataset = "random" - # The last three numeric tokens are: concurrency, input_len/output_ctx, output_len. try: concurrency, input_len, output_len = int(parts[-3]), int(parts[-2]), int(parts[-1]) except ValueError: @@ -211,7 +210,7 @@ def main() -> None: with open(results_json, "w", encoding="utf-8") as f: json.dump(data, f, indent=2, ensure_ascii=False) - generate_report(result_root, backend, scenarios) + generate_report(result_root, backend, scenarios, metadata) print(f"Parsed {len(scenarios)} scenarios into {result_root}/report.md") diff --git a/experiments/dsv4_h200_256k_4k_probe/warmup.py b/scripts/common/warmup.py similarity index 100% rename from experiments/dsv4_h200_256k_4k_probe/warmup.py rename to scripts/common/warmup.py diff --git a/scripts/install_vllm_dspark.sh b/scripts/install_vllm_dspark.sh deleted file mode 100755 index d1595f4..0000000 --- a/scripts/install_vllm_dspark.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/bin/bash -set -e - -export TMPDIR=/data/tmp -export TEMP=/data/tmp -export TMP=/data/tmp -export PIP_CACHE_DIR=/data/tmp/pip-cache -export SETUPTOOLS_SCM_PRETEND_VERSION=0.23.1.dev788 -mkdir -p $PIP_CACHE_DIR - -cd /data/user1/yy/vllm-main -PYTHON=/data/user1/yy/envs/vllm-dspark/bin/python -PIP=/data/user1/yy/envs/vllm-dspark/bin/pip - -echo "Installing torch 2.11.0 with CUDA 12.9..." -$PIP install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu129 --cache-dir $PIP_CACHE_DIR - -echo "Installing numpy and build dependencies..." -$PIP install numpy cmake ninja packaging "setuptools>=77.0.3,<81.0.0" setuptools-scm setuptools-rust wheel jinja2 --cache-dir $PIP_CACHE_DIR - -echo "Installing vllm from source (this will compile CUDA kernels, may take 30-60 min)..." -$PIP install -e . --no-build-isolation --cache-dir $PIP_CACHE_DIR - -echo "Verifying installation..." -$PYTHON -c "import vllm; print('vllm version:', vllm.__version__)" - -echo "Installation complete." diff --git a/scripts/profile_dspark.py b/scripts/profile_dspark.py deleted file mode 100755 index ce40591..0000000 --- a/scripts/profile_dspark.py +++ /dev/null @@ -1,261 +0,0 @@ -#!/usr/bin/env python3 -"""Profile DSpark inference with Nsight Systems and/or PyTorch profiler. - -Usage: - python scripts/profile_dspark.py --config dspark-st3 --concurrency 64 --tool nsys - python scripts/profile_dspark.py --config nospec --concurrency 64 --tool nsys -""" - -import argparse -import json -import os -import subprocess -import sys -import time -import urllib.request -from datetime import datetime -from pathlib import Path - -ROOT = Path("/data/user1/yy") -RESULT_DIR = ROOT / "bench_results" / f"dspark_profile_{datetime.now().strftime('%Y%m%d_%H%M%S')}" -RESULT_DIR.mkdir(parents=True, exist_ok=True) - -DATASET = "/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json" -NUM_PROMPTS = 200 -SEED = 42 -OUTPUT_LEN = 256 -HOST = "127.0.0.1" - -CONFIGS = { - "dspark-st3": { - "model": "/data/models/DeepSeek-V4-Flash-DSpark", - "port": 30004, - "spec_tokens": 3, - "cmd": [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve", - "/data/models/DeepSeek-V4-Flash-DSpark", - "--trust-remote-code", - "--tensor-parallel-size", "8", - "--kv-cache-dtype", "fp8", - "--block-size", "256", - "--max-model-len", "auto", - "--max-num-seqs", "256", - "--gpu-memory-utilization", "0.75", - "--tokenizer-mode", "deepseek_v4", - "--reasoning-parser", "deepseek_v4", - "--spec-method", "dspark", - "--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark", - "--spec-tokens", "3", - "--no-disable-hybrid-kv-cache-manager", - "--disable-uvicorn-access-log", - "--port", "30004", - ], - }, - "dspark-st5": { - "model": "/data/models/DeepSeek-V4-Flash-DSpark", - "port": 30004, - "spec_tokens": 5, - "cmd": [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve", - "/data/models/DeepSeek-V4-Flash-DSpark", - "--trust-remote-code", - "--tensor-parallel-size", "8", - "--kv-cache-dtype", "fp8", - "--block-size", "256", - "--max-model-len", "auto", - "--max-num-seqs", "256", - "--gpu-memory-utilization", "0.75", - "--tokenizer-mode", "deepseek_v4", - "--reasoning-parser", "deepseek_v4", - "--spec-method", "dspark", - "--spec-model", "/data/models/DeepSeek-V4-Flash-DSpark", - "--spec-tokens", "5", - "--no-disable-hybrid-kv-cache-manager", - "--disable-uvicorn-access-log", - "--port", "30004", - ], - }, - "nospec": { - "model": "/data/models/DeepSeek-V4-Flash-DSpark", - "port": 30004, - "cmd": [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "serve", - "/data/models/DeepSeek-V4-Flash-DSpark", - "--trust-remote-code", - "--tensor-parallel-size", "8", - "--kv-cache-dtype", "fp8", - "--block-size", "256", - "--max-model-len", "auto", - "--max-num-seqs", "256", - "--gpu-memory-utilization", "0.75", - "--tokenizer-mode", "deepseek_v4", - "--reasoning-parser", "deepseek_v4", - "--no-disable-hybrid-kv-cache-manager", - "--disable-uvicorn-access-log", - "--port", "30004", - ], - }, -} - - -def log(msg): - print(f"[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] {msg}", flush=True) - - -def wait_for_health(port, timeout=300): - url = f"http://{HOST}:{port}/health" - start = time.time() - while time.time() - start < timeout: - try: - with urllib.request.urlopen(url, timeout=5) as resp: - if resp.status == 200: - return True - except Exception: - pass - time.sleep(2) - return False - - -def run_warmup(port, model, n=20): - """Run a few warmup requests to stabilize CUDA graphs.""" - log(f"Running {n} warmup requests...") - for i in range(n): - try: - req = urllib.request.Request( - f"http://{HOST}:{port}/v1/completions", - data=json.dumps({ - "model": model, - "prompt": "Hello, how are you?", - "max_tokens": 32, - "temperature": 0, - "seed": SEED, - }).encode(), - headers={"Content-Type": "application/json"}, - method="POST", - ) - with urllib.request.urlopen(req, timeout=60) as resp: - resp.read() - except Exception as e: - log(f"Warmup request {i} failed: {e}") - - -def run_benchmark(port, model, concurrency, result_dir): - result_file = result_dir / f"profile_c{concurrency}.json" - log_file = result_dir / f"profile_c{concurrency}.log" - bench_cmd = [ - "/data/user1/yy/envs/vllm-dspark/bin/vllm", "bench", "serve", - "--host", HOST, - "--port", str(port), - "--backend", "openai", - "--dataset-name", "sharegpt", - "--dataset-path", DATASET, - "--sharegpt-output-len", str(OUTPUT_LEN), - "--num-prompts", str(NUM_PROMPTS), - "--max-concurrency", str(concurrency), - "--endpoint", "/v1/completions", - "--model", model, - "--seed", str(SEED), - "--save-result", - "--result-dir", str(result_dir), - "--result-filename", result_file.name, - ] - log(f"Running benchmark c={concurrency}...") - start = time.time() - with open(log_file, "w") as f: - proc = subprocess.Popen(bench_cmd, stdout=f, stderr=subprocess.STDOUT) - proc.wait() - duration = time.time() - start - log(f"Benchmark c={concurrency} finished in {duration:.1f}s, exit={proc.returncode}") - return result_file - - -def start_service(config, tool, result_dir): - name = config - cfg = CONFIGS[config] - port = cfg["port"] - base_cmd = cfg["cmd"] - - if tool == "nsys": - nsys_output = result_dir / f"{name}_nsys" - cmd = [ - "nsys", "profile", - "--sample=none", - "--cpuctxsw=none", - "--trace=cuda,nvtx,osrt", - "--output", str(nsys_output), - "--force-overwrite", "true", - "--delay", "30", - "--duration", "15", - ] + base_cmd - else: - cmd = base_cmd - - log(f"Starting service for {name} (tool={tool}) on port {port}...") - log_file = result_dir / f"{name}_service.log" - env = os.environ.copy() - env["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5,6,7" - env["TMPDIR"] = str(ROOT / "tmp") - - proc = subprocess.Popen( - cmd, - stdout=open(log_file, "w"), - stderr=subprocess.STDOUT, - env=env, - ) - if not wait_for_health(port): - log(f"ERROR: {name} failed to start") - proc.terminate() - return None - log(f"{name} is ready") - return proc - - -def stop_service(proc, name): - if proc is None: - return - log(f"Stopping {name} (pid {proc.pid})...") - proc.terminate() - try: - proc.wait(timeout=60) - except subprocess.TimeoutExpired: - proc.kill() - proc.wait() - log(f"{name} stopped") - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("--config", required=True, choices=list(CONFIGS.keys())) - parser.add_argument("--concurrency", type=int, default=64) - parser.add_argument("--tool", default="nsys", choices=["nsys"]) - parser.add_argument("--skip-bench", action="store_true", help="Only capture service startup/warmup, skip benchmark") - args = parser.parse_args() - - result_dir = RESULT_DIR / f"{args.config}_c{args.concurrency}_{args.tool}" - result_dir.mkdir(parents=True, exist_ok=True) - log(f"Results will be saved to {result_dir}") - - proc = start_service(args.config, args.tool, result_dir) - if proc is None: - sys.exit(1) - - try: - run_warmup(CONFIGS[args.config]["port"], CONFIGS[args.config]["model"], n=30) - if not args.skip_bench: - run_benchmark( - CONFIGS[args.config]["port"], - CONFIGS[args.config]["model"], - args.concurrency, - result_dir, - ) - else: - log("--skip-bench set; sleeping 60s to capture warmup/steady state...") - time.sleep(60) - finally: - stop_service(proc, args.config) - - log(f"All done. Results in {result_dir}") - - -if __name__ == "__main__": - main() diff --git a/scripts/run_sglang_benchmark.sh b/scripts/run_sglang_benchmark.sh deleted file mode 100755 index bb19967..0000000 --- a/scripts/run_sglang_benchmark.sh +++ /dev/null @@ -1,157 +0,0 @@ -#!/bin/bash -# Comprehensive benchmark for sglang 8-card DeepSeek-V4-Flash -set -e - -cd /data/user1/yy -mkdir -p bench_results logs - -PYTHON="/data/user1/yy/envs/sglang/bin/python" -BENCH="-m sglang.bench_serving" -HOST="127.0.0.1" -PORT="30000" -MODEL="/data/models/DeepSeek-V4-Flash" -DATASET="/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json" -NUM_PROMPTS=2000 -SEED=42 - -RESULT_DIR="bench_results/sglang_8card_systematic_$(date +%Y%m%d_%H%M%S)" -mkdir -p "${RESULT_DIR}" - -SUMMARY_FILE="${RESULT_DIR}/summary.json" -LOG_FILE="${RESULT_DIR}/benchmark.log" - -# Use a marker file for summary accumulation -SUMMARY_TMP="${RESULT_DIR}/.summary_tmp.json" -echo '[]' > "${SUMMARY_TMP}" - -log() { - echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "${LOG_FILE}" -} - -wait_for_server() { - log "Waiting for sglang server at ${HOST}:${PORT} to become ready..." - local retries=0 - local max_retries=120 - while ! curl -s "http://${HOST}:${PORT}/v1/models" > /dev/null 2>&1; do - retries=$((retries + 1)) - if [ "${retries}" -ge "${max_retries}" ]; then - log "ERROR: Server not ready after ${max_retries} retries" - exit 1 - fi - sleep 2 - done - log "Server is ready." -} - -run_bench() { - local name="$1" - shift - local output_file="${RESULT_DIR}/${name}.json" - local detail_log="${RESULT_DIR}/${name}.log" - - log "---------------------------------------------------" - log "Running benchmark: ${name}" - log "Args: $*" - log "Output: ${output_file}" - - ${PYTHON} ${BENCH} \ - --backend sglang \ - --host "${HOST}" \ - --port "${PORT}" \ - --dataset-name sharegpt \ - --dataset-path "${DATASET}" \ - --num-prompts "${NUM_PROMPTS}" \ - --model "${MODEL}" \ - --output-file "${output_file}" \ - --output-details \ - --seed "${SEED}" \ - "$@" \ - > "${detail_log}" 2>&1 - - local status=$? - if [ ${status} -ne 0 ]; then - log "ERROR: Benchmark ${name} failed with exit code ${status}" - return ${status} - fi - - log "Benchmark ${name} completed successfully." - - # Extract key metrics from the log - local req_throughput=$(grep -oP 'Request throughput \(req/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local input_throughput=$(grep -oP 'Input token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local output_throughput=$(grep -oP 'Output token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local total_throughput=$(grep -oP 'Total token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local mean_ttft=$(grep -oP 'Mean TTFT \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local mean_tpot=$(grep -oP 'Mean TPOT \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local mean_e2e=$(grep -oP 'Mean E2E Latency \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local duration=$(grep -oP 'Benchmark duration \(s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local accept_length=$(grep -oP 'Accept length:\s+\K[0-9.]+' "${detail_log}" || echo "null") - - # Append to summary tmp - ${PYTHON} - <8} {'In tok/s':>10} {'Out tok/s':>10} {'Total tok/s':>12} {'TTFT':>8} {'TPOT':>8} {'E2E':>10}") -print("-" * 110) -for entry in data: - print(f"{entry['name']:<30} {entry['request_throughput']:>8.2f} {entry['input_token_throughput']:>10.2f} {entry['output_token_throughput']:>10.2f} {entry['total_token_throughput']:>12.2f} {entry['mean_ttft_ms']:>8.1f} {entry['mean_tpot_ms']:>8.1f} {entry['mean_e2e_latency_ms']:>10.1f}") -PYEOF -} - -main "$@" diff --git a/scripts/run_sglang_max_throughput.sh b/scripts/run_sglang_max_throughput.sh deleted file mode 100755 index 7d1513b..0000000 --- a/scripts/run_sglang_max_throughput.sh +++ /dev/null @@ -1,143 +0,0 @@ -#!/bin/bash -# Max throughput benchmark for sglang 8-card DeepSeek-V4-Flash -set -e - -cd /data/user1/yy -mkdir -p bench_results logs - -PYTHON="/data/user1/yy/envs/sglang/bin/python" -BENCH="-m sglang.bench_serving" -HOST="127.0.0.1" -PORT="30000" -MODEL="/data/models/DeepSeek-V4-Flash" -DATASET="/data/user1/yy/datasets/ShareGPT_V4.3_unfiltered_cleaned_split.json" -NUM_PROMPTS=2000 -SEED=42 - -RESULT_DIR="bench_results/sglang_8card_max_throughput_$(date +%Y%m%d_%H%M%S)" -mkdir -p "${RESULT_DIR}" - -SUMMARY_FILE="${RESULT_DIR}/summary.json" -LOG_FILE="${RESULT_DIR}/benchmark.log" -SUMMARY_TMP="${RESULT_DIR}/.summary_tmp.json" -echo '[]' > "${SUMMARY_TMP}" - -log() { - echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "${LOG_FILE}" -} - -wait_for_server() { - log "Waiting for sglang server at ${HOST}:${PORT} to become ready..." - local retries=0 - local max_retries=120 - while ! curl -s "http://${HOST}:${PORT}/v1/models" > /dev/null 2>&1; do - retries=$((retries + 1)) - if [ "${retries}" -ge "${max_retries}" ]; then - log "ERROR: Server not ready after ${max_retries} retries" - exit 1 - fi - sleep 2 - done - log "Server is ready." -} - -run_bench() { - local name="$1" - shift - local output_file="${RESULT_DIR}/${name}.json" - local detail_log="${RESULT_DIR}/${name}.log" - - log "---------------------------------------------------" - log "Running benchmark: ${name}" - log "Args: $*" - log "Output: ${output_file}" - - ${PYTHON} ${BENCH} \ - --backend sglang \ - --host "${HOST}" \ - --port "${PORT}" \ - --dataset-name sharegpt \ - --dataset-path "${DATASET}" \ - --num-prompts "${NUM_PROMPTS}" \ - --model "${MODEL}" \ - --output-file "${output_file}" \ - --output-details \ - --seed "${SEED}" \ - "$@" \ - > "${detail_log}" 2>&1 - - local status=$? - if [ ${status} -ne 0 ]; then - log "ERROR: Benchmark ${name} failed with exit code ${status}" - return ${status} - fi - - log "Benchmark ${name} completed successfully." - - local req_throughput=$(grep -oP 'Request throughput \(req/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local input_throughput=$(grep -oP 'Input token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local output_throughput=$(grep -oP 'Output token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local total_throughput=$(grep -oP 'Total token throughput \(tok/s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local mean_ttft=$(grep -oP 'Mean TTFT \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local mean_tpot=$(grep -oP 'Mean TPOT \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local mean_e2e=$(grep -oP 'Mean E2E Latency \(ms\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local duration=$(grep -oP 'Benchmark duration \(s\):\s+\K[0-9.]+' "${detail_log}" || echo "null") - local accept_length=$(grep -oP 'Accept length:\s+\K[0-9.]+' "${detail_log}" || echo "null") - - ${PYTHON} - <8} {'In tok/s':>10} {'Out tok/s':>10} {'Total tok/s':>12} {'TTFT':>8} {'TPOT':>8} {'E2E':>10}") -print("-" * 110) -for entry in data: - print(f"{entry['name']:<30} {entry['request_throughput']:>8.2f} {entry['input_token_throughput']:>10.2f} {entry['output_token_throughput']:>10.2f} {entry['total_token_throughput']:>12.2f} {entry['mean_ttft_ms']:>8.1f} {entry['mean_tpot_ms']:>8.1f} {entry['mean_e2e_latency_ms']:>10.1f}") -PYEOF -} - -main "$@" diff --git a/scripts/start_dsv4_dspark_8card_bf16kv.sh b/scripts/start_dsv4_dspark_8card_bf16kv.sh deleted file mode 100755 index 3833a53..0000000 --- a/scripts/start_dsv4_dspark_8card_bf16kv.sh +++ /dev/null @@ -1,63 +0,0 @@ -#!/bin/bash -set -e - -cd /data/user1/yy -mkdir -p logs - -VENV="/data/user1/yy/envs/vllm-dspark" -VLLM="$VENV/bin/vllm" - -MODEL="/data/models/DeepSeek-V4-Flash-DSpark" -PORT=30004 -TP=8 -LOG="/data/user1/yy/logs/dsv4_flash_dspark_tp${TP}_bf16kv_$(date +%Y%m%d_%H%M%S).log" -PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid" - -export TMPDIR=/data/user1/yy/tmp -export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 - -echo "=== Starting DeepSeek-V4-Flash-DSpark (TP=$TP, kv=bf16) ===" -echo "Model: $MODEL" -echo "Port: $PORT" -echo "Log: $LOG" - -rm -f "$PID_FILE" -nohup "$VLLM" serve "$MODEL" \ - --trust-remote-code \ - --tensor-parallel-size "$TP" \ - --kv-cache-dtype bfloat16 \ - --block-size 256 \ - --max-model-len auto \ - --max-num-seqs 256 \ - --tokenizer-mode deepseek_v4 \ - --reasoning-parser deepseek_v4 \ - --spec-method dspark \ - --spec-model "$MODEL" \ - --spec-tokens 5 \ - --no-disable-hybrid-kv-cache-manager \ - --disable-uvicorn-access-log \ - --port "$PORT" \ - > "$LOG" 2>&1 & -PID=$! -echo $PID > "$PID_FILE" -echo "PID: $PID" -echo "Waiting for health..." - -for i in $(seq 1 240); do - if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then - echo "Server is ready at http://127.0.0.1:$PORT" - echo "Log: $LOG" - exit 0 - fi - if ! kill -0 $PID 2>/dev/null; then - echo "ERROR: Server exited early" - tail -100 "$LOG" - exit 1 - fi - echo "Waiting... ($i/240)" - sleep 5 -done - -echo "ERROR: Server not healthy after 240 retries" -tail -100 "$LOG" -exit 1 diff --git a/scripts/start_dsv4_dspark_8card_flashinfer.sh b/scripts/start_dsv4_dspark_8card_flashinfer.sh deleted file mode 100755 index 458e309..0000000 --- a/scripts/start_dsv4_dspark_8card_flashinfer.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/bin/bash -set -e - -cd /data/user1/yy -mkdir -p logs - -VENV="/data/user1/yy/envs/vllm-dspark" -VLLM="$VENV/bin/vllm" -export PATH="$VENV/bin:$PATH" - -MODEL="/data/models/DeepSeek-V4-Flash-DSpark" -PORT=30004 -TP=8 -LOG="/data/user1/yy/logs/dsv4_flash_dspark_tp${TP}_flashinfer_$(date +%Y%m%d_%H%M%S).log" -PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid" - -export TMPDIR=/data/user1/yy/tmp -export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 - -echo "=== Starting DeepSeek-V4-Flash-DSpark (TP=$TP, FlashInfer) ===" -echo "Model: $MODEL" -echo "Port: $PORT" -echo "Log: $LOG" - -rm -f "$PID_FILE" -nohup "$VLLM" serve "$MODEL" \ - --trust-remote-code \ - --tensor-parallel-size "$TP" \ - --kv-cache-dtype fp8 \ - --block-size 256 \ - --max-model-len auto \ - --max-num-seqs 256 \ - --tokenizer-mode deepseek_v4 \ - --reasoning-parser deepseek_v4 \ - --attention-backend FLASHINFER_MLA_SPARSE_DSV4 \ - --spec-method dspark \ - --spec-model "$MODEL" \ - --spec-tokens 5 \ - --speculative-config '{"attention_backend": "FLASHINFER_MLA_SPARSE_DSV4"}' \ - --no-disable-hybrid-kv-cache-manager \ - --disable-uvicorn-access-log \ - --port "$PORT" \ - > "$LOG" 2>&1 & -PID=$! -echo $PID > "$PID_FILE" -echo "PID: $PID" -echo "Waiting for health..." - -for i in $(seq 1 240); do - if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then - echo "Server is ready at http://127.0.0.1:$PORT" - echo "Log: $LOG" - exit 0 - fi - if ! kill -0 $PID 2>/dev/null; then - echo "ERROR: Server exited early" - tail -100 "$LOG" - exit 1 - fi - echo "Waiting... ($i/240)" - sleep 5 -done - -echo "ERROR: Server not healthy after 240 retries" -tail -100 "$LOG" -exit 1 diff --git a/scripts/start_dsv4_dspark_8card_param.sh b/scripts/start_dsv4_dspark_8card_param.sh deleted file mode 100755 index 6be3ac5..0000000 --- a/scripts/start_dsv4_dspark_8card_param.sh +++ /dev/null @@ -1,85 +0,0 @@ -#!/bin/bash -set -e - -cd /data/user1/yy -mkdir -p logs - -VENV="/data/user1/yy/envs/vllm-dspark" -VLLM="$VENV/bin/vllm" -export PATH="$VENV/bin:$PATH" - -MODEL="/data/models/DeepSeek-V4-Flash-DSpark" -PORT="${PORT:-30004}" -TP="${TP:-8}" -SPEC_TOKENS="${SPEC_TOKENS:-5}" -KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" -ATTENTION_BACKEND="${ATTENTION_BACKEND:-}" -SPECULATIVE_CONFIG="${SPECULATIVE_CONFIG:-}" - -LOG_TAG="tp${TP}_${KV_CACHE_DTYPE}_st${SPEC_TOKENS}" -if [[ -n "$ATTENTION_BACKEND" ]]; then - LOG_TAG="${LOG_TAG}_${ATTENTION_BACKEND}" -fi -LOG="/data/user1/yy/logs/dsv4_flash_dspark_${LOG_TAG}_$(date +%Y%m%d_%H%M%S).log" -PID_FILE="/data/user1/yy/dsv4_flash_dspark.pid" - -export TMPDIR=/data/user1/yy/tmp -export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 - -echo "=== Starting DeepSeek-V4-Flash-DSpark (TP=$TP, spec-tokens=$SPEC_TOKENS, kv=$KV_CACHE_DTYPE) ===" -echo "Model: $MODEL" -echo "Port: $PORT" -echo "Log: $LOG" - -rm -f "$PID_FILE" - -cmd=( - "$VLLM" serve "$MODEL" - --trust-remote-code - --tensor-parallel-size "$TP" - --kv-cache-dtype "$KV_CACHE_DTYPE" - --block-size 256 - --max-model-len auto - --max-num-seqs 256 - --tokenizer-mode deepseek_v4 - --reasoning-parser deepseek_v4 - --spec-method dspark - --spec-model "$MODEL" - --spec-tokens "$SPEC_TOKENS" - --no-disable-hybrid-kv-cache-manager - --disable-uvicorn-access-log - --port "$PORT" -) - -if [[ -n "$ATTENTION_BACKEND" ]]; then - cmd+=(--attention-backend "$ATTENTION_BACKEND") -fi - -if [[ -n "$SPECULATIVE_CONFIG" ]]; then - cmd+=(--speculative-config "$SPECULATIVE_CONFIG") -fi - -nohup "${cmd[@]}" > "$LOG" 2>&1 & -PID=$! -echo $PID > "$PID_FILE" -echo "PID: $PID" -echo "Waiting for health..." - -for i in $(seq 1 240); do - if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then - echo "Server is ready at http://127.0.0.1:$PORT" - echo "Log: $LOG" - exit 0 - fi - if ! kill -0 $PID 2>/dev/null; then - echo "ERROR: Server exited early" - tail -100 "$LOG" - exit 1 - fi - echo "Waiting... ($i/240)" - sleep 5 -done - -echo "ERROR: Server not healthy after 240 retries" -tail -100 "$LOG" -exit 1 diff --git a/scripts/start_pd_single_node.sh b/scripts/start_pd_single_node.sh deleted file mode 100755 index 3f3a2fd..0000000 --- a/scripts/start_pd_single_node.sh +++ /dev/null @@ -1,138 +0,0 @@ -#!/bin/bash -set -e - -# sglang PD 分离单节点启动脚本 -# 104 上前 4 张卡做 prefill，后 4 张卡做 decode -# 使用 NIXL/UCX 作为传输后端（Mooncake 无法安装） - -MODEL_PATH="/data/models/DeepSeek-V4-Flash" -BOOTSTRAP_PORT=30010 -PREFILL_PORT=30001 -DECODE_PORT=30002 -LB_PORT=30000 -HOST="0.0.0.0" - -VENV="/data/user1/yy/envs/sglang" -PYTHON="$VENV/bin/python" -LOG_DIR="/data/user1/yy/pd_logs" -mkdir -p "$LOG_DIR" - -# NIXL/UCX 配置：单节点内优先用 CUDA IPC / shared memory，禁用 IB -export SGLANG_DISAGGREGATION_NIXL_BACKEND="UCX" -export UCX_TLS="self,sm,cuda_copy,cuda_ipc,tcp" -export UCX_NET_DEVICES="" - -echo "=== Starting sglang PD disaggregation (single node) ===" -echo "Model: $MODEL_PATH" -echo "Prefill GPUs: 0-3, port $PREFILL_PORT" -echo "Decode GPUs: 4-7, port $DECODE_PORT" -echo "Load balancer port: $LB_PORT" -echo "Logs: $LOG_DIR" -echo "" - -# Clean up old logs -rm -f "$LOG_DIR"/*.log - -# Common args for both prefill and decode -COMMON_ARGS=( - --trust-remote-code - --model-path "$MODEL_PATH" - --tp 4 - --moe-runner-backend marlin - --disable-cuda-graph - --disaggregation-bootstrap-port "$BOOTSTRAP_PORT" - --disaggregation-transfer-backend nixl - --host "$HOST" -) - -# Start prefill server (GPUs 0-3) -echo "[1/3] Starting prefill server on GPUs 0-3..." -CUDA_VISIBLE_DEVICES=0,1,2,3 nohup "$PYTHON" -m sglang.launch_server \ - "${COMMON_ARGS[@]}" \ - --disaggregation-mode prefill \ - --port "$PREFILL_PORT" \ - > "$LOG_DIR/prefill.log" 2>&1 & -PREFILL_PID=$! -echo "Prefill PID: $PREFILL_PID" - -# Wait for prefill health -for i in {1..120}; do - if curl -s "http://127.0.0.1:$PREFILL_PORT/health" > /dev/null 2>&1; then - echo "Prefill server is ready" - break - fi - if ! kill -0 $PREFILL_PID 2>/dev/null; then - echo "ERROR: Prefill server exited early" - tail -50 "$LOG_DIR/prefill.log" - exit 1 - fi - echo "Waiting for prefill server... ($i/120)" - sleep 5 -done - -# Start decode server (GPUs 4-7) -echo "[2/3] Starting decode server on GPUs 4-7..." -CUDA_VISIBLE_DEVICES=0,1,2,3 nohup "$PYTHON" -m sglang.launch_server \ - "${COMMON_ARGS[@]}" \ - --disaggregation-mode decode \ - --base-gpu-id 4 \ - --port "$DECODE_PORT" \ - > "$LOG_DIR/decode.log" 2>&1 & -DECODE_PID=$! -echo "Decode PID: $DECODE_PID" - -# Wait for decode health -for i in {1..120}; do - if curl -s "http://127.0.0.1:$DECODE_PORT/health" > /dev/null 2>&1; then - echo "Decode server is ready" - break - fi - if ! kill -0 $DECODE_PID 2>/dev/null; then - echo "ERROR: Decode server exited early" - tail -50 "$LOG_DIR/decode.log" - exit 1 - fi - echo "Waiting for decode server... ($i/120)" - sleep 5 -done - -# Start PD load balancer -echo "[3/3] Starting PD load balancer on port $LB_PORT..." -nohup "$PYTHON" -m sglang_router.launch_router \ - --pd-disaggregation \ - --mini-lb \ - --prefill "http://127.0.0.1:$PREFILL_PORT" \ - --decode "http://127.0.0.1:$DECODE_PORT" \ - --host "$HOST" \ - --port "$LB_PORT" \ - > "$LOG_DIR/lb.log" 2>&1 & -LB_PID=$! -echo "LB PID: $LB_PID" - -# Wait for LB health -for i in {1..60}; do - if curl -s "http://127.0.0.1:$LB_PORT/health" > /dev/null 2>&1; then - echo "Load balancer is ready" - break - fi - if ! kill -0 $LB_PID 2>/dev/null; then - echo "ERROR: Load balancer exited early" - tail -50 "$LOG_DIR/lb.log" - exit 1 - fi - echo "Waiting for load balancer... ($i/60)" - sleep 2 -done - -echo "" -echo "=== PD disaggregation is ready ===" -echo "Load balancer URL: http://127.0.0.1:$LB_PORT" -echo "Prefill URL: http://127.0.0.1:$PREFILL_PORT" -echo "Decode URL: http://127.0.0.1:$DECODE_PORT" -echo "" -echo "Test command:" -echo " curl http://127.0.0.1:$LB_PORT/v1/completions \\" -echo " -H \"Content-Type: application/json\" \\" -echo " -d '{\"model\":\"DeepSeek-V4-Flash\",\"prompt\":\"Hello\",\"max_tokens\":32}'" -echo "" -echo "To stop: kill $PREFILL_PID $DECODE_PID $LB_PID" diff --git a/scripts/start_sglang_dsv4_8card.sh b/scripts/start_sglang_dsv4_8card.sh deleted file mode 100755 index 638ecbf..0000000 --- a/scripts/start_sglang_dsv4_8card.sh +++ /dev/null @@ -1,27 +0,0 @@ -#!/bin/bash -# Start sglang 8-card DeepSeek-V4-Flash service -set -e - -cd /data/user1/yy -mkdir -p logs - -export PYTHONUNBUFFERED=1 -export SGLANG_LOG_LEVEL=info -export PATH="/data/user1/yy/envs/sglang/bin:$PATH" - -LOG_FILE="logs/sglang_8card_$(date +%Y%m%d_%H%M%S).log" - -echo "Starting sglang 8-card DSV4 service, logging to ${LOG_FILE}" - -exec /data/user1/yy/envs/sglang/bin/sglang serve \ - --trust-remote-code \ - --model-path /data/models/DeepSeek-V4-Flash \ - --tp 8 \ - --moe-runner-backend marlin \ - --speculative-algorithm EAGLE \ - --speculative-num-steps 3 \ - --speculative-eagle-topk 1 \ - --speculative-num-draft-tokens 4 \ - --host 0.0.0.0 \ - --port 30000 \ - 2>&1 | tee "${LOG_FILE}" diff --git a/scripts/start_vllm_eagle_dsv4_8card.sh b/scripts/start_vllm_eagle_dsv4_8card.sh deleted file mode 100644 index 676998e..0000000 --- a/scripts/start_vllm_eagle_dsv4_8card.sh +++ /dev/null @@ -1,65 +0,0 @@ -#!/bin/bash -set -e - -cd /data/user1/yy -mkdir -p logs - -VENV="/data/user1/yy/envs/vllm-main-latest" -PYTHON="$VENV/bin/python" -VLLM="$VENV/bin/vllm" -export PATH="$VENV/bin:$PATH" - -MODEL="/data/models/DeepSeek-V4-Flash" -PORT=30005 -TP=8 -LOG="/data/user1/yy/logs/vllm_eagle_dsv4_tp${TP}_$(date +%Y%m%d_%H%M%S).log" -PID_FILE="/data/user1/yy/vllm_eagle_dsv4.pid" - -export TMPDIR=/data/user1/yy/tmp -export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 - -echo "=== Starting DeepSeek-V4-Flash vLLM EAGLE (TP=$TP) ===" -echo "Model: $MODEL" -echo "Port: $PORT" -echo "Log: $LOG" - -rm -f "$PID_FILE" -nohup "$VLLM" serve "$MODEL" \ - --trust-remote-code \ - --tensor-parallel-size "$TP" \ - --kv-cache-dtype fp8 \ - --block-size 256 \ - --max-model-len auto \ - --max-num-seqs 256 \ - --tokenizer-mode deepseek_v4 \ - --reasoning-parser deepseek_v4 \ - --spec-method mtp \ - --spec-model "$MODEL" \ - --spec-tokens 4 \ - --no-disable-hybrid-kv-cache-manager \ - --disable-uvicorn-access-log \ - --port "$PORT" \ - > "$LOG" 2>&1 & -PID=$! -echo $PID > "$PID_FILE" -echo "PID: $PID" -echo "Waiting for health..." - -for i in $(seq 1 240); do - if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then - echo "Server is ready at http://127.0.0.1:$PORT" - echo "Log: $LOG" - exit 0 - fi - if ! kill -0 $PID 2>/dev/null; then - echo "ERROR: Server exited early" - tail -200 "$LOG" - exit 1 - fi - echo "Waiting... ($i/240)" - sleep 5 -done - -echo "ERROR: Server not healthy after 240 retries" -tail -200 "$LOG" -exit 1 diff --git a/scripts/start_vllm_mtp_dsv4_8card.sh b/scripts/start_vllm_mtp_dsv4_8card.sh deleted file mode 100644 index 1c6a20c..0000000 --- a/scripts/start_vllm_mtp_dsv4_8card.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/bin/bash -set -e - -cd /data/user1/yy -mkdir -p logs - -VENV="/data/user1/yy/envs/vllm-dspark" -export PATH="$VENV/bin:$PATH" -PYTHON="$VENV/bin/python" -VLLM="$VENV/bin/vllm" - -MODEL="/data/models/DeepSeek-V4-Flash" -PORT=30005 -TP=8 -LOG="/data/user1/yy/logs/vllm_mtp_dsv4_tp${TP}_$(date +%Y%m%d_%H%M%S).log" -PID_FILE="/data/user1/yy/vllm_mtp_dsv4.pid" - -export TMPDIR=/data/user1/yy/tmp -export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 - -echo "=== Starting DeepSeek-V4-Flash vLLM MTP (TP=$TP) ===" -echo "Model: $MODEL" -echo "Port: $PORT" -echo "Log: $LOG" - -rm -f "$PID_FILE" -nohup "$VLLM" serve "$MODEL" \ - --trust-remote-code \ - --tensor-parallel-size "$TP" \ - --kv-cache-dtype fp8 \ - --block-size 256 \ - --max-model-len auto \ - --max-num-seqs 128 \ - --gpu-memory-utilization 0.85 \ - --tokenizer-mode deepseek_v4 \ - --reasoning-parser deepseek_v4 \ - --spec-method mtp \ - --spec-model "$MODEL" \ - --spec-tokens 4 \ - --no-disable-hybrid-kv-cache-manager \ - --disable-uvicorn-access-log \ - --port "$PORT" \ - > "$LOG" 2>&1 & -PID=$! -echo $PID > "$PID_FILE" -echo "PID: $PID" -echo "Waiting for health..." - -for i in $(seq 1 240); do - if curl -s "http://127.0.0.1:$PORT/health" > /dev/null 2>&1; then - echo "Server is ready at http://127.0.0.1:$PORT" - echo "Log: $LOG" - exit 0 - fi - if ! kill -0 $PID 2>/dev/null; then - echo "ERROR: Server exited early" - tail -200 "$LOG" - exit 1 - fi - echo "Waiting... ($i/240)" - sleep 5 -done - -echo "ERROR: Server not healthy after 240 retries" -tail -200 "$LOG" -exit 1 diff --git a/scripts/start_vllm_pd_single_node.sh b/scripts/start_vllm_pd_single_node.sh deleted file mode 100755 index 667282d..0000000 --- a/scripts/start_vllm_pd_single_node.sh +++ /dev/null @@ -1,156 +0,0 @@ -#!/bin/bash -set -e - -# vLLM + Mooncake PD 分离单节点启动脚本 -# 104 上前 4 张卡做 prefill，后 4 张卡做 decode -# 使用 TCP 传输（因为当前 IB/RDMA 未启用） - -MODEL_PATH="/data/models/DeepSeek-V4-Flash" -BOOTSTRAP_PORT=8998 -PREFILL_PORT=8000 -DECODE_PORT=8001 -ROUTER_PORT=30000 -HOST="0.0.0.0" - -VENV="/data/user1/yy/envs/vllm" -PYTHON="$VENV/bin/python" -VLLM="$VENV/bin/vllm" -VLLM_ROUTER="$VENV/bin/vllm-router" -LOG_DIR="/data/user1/yy/vllm_pd_logs" -mkdir -p "$LOG_DIR" - -echo "=== Starting vLLM PD disaggregation (single node) ===" -echo "Model: $MODEL_PATH" -echo "Prefill GPUs: 0-3, port $PREFILL_PORT" -echo "Decode GPUs: 4-7, port $DECODE_PORT" -echo "Router port: $ROUTER_PORT" -echo "Logs: $LOG_DIR" -echo "" - -# Clean up old logs -rm -f "$LOG_DIR"/*.log - -# 使用 TCP 协议（IB 没起来，不能用 rdma） -export VLLM_MOONCAKE_PROTOCOL=tcp -export VLLM_MOONCAKE_BOOTSTRAP_PORT=$BOOTSTRAP_PORT - -# Common args -PREFILL_KV_CONFIG='{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"tcp"}}' -DECODE_KV_CONFIG='{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"mooncake_protocol":"tcp"}}' - -# Start prefill server (GPUs 0-3) -echo "[1/3] Starting prefill server on GPUs 0-3..." -CUDA_VISIBLE_DEVICES=0,1,2,3 nohup "$VLLM" serve "$MODEL_PATH" \ - --trust-remote-code \ - --kv-cache-dtype fp8 \ - --block-size 256 \ - --port "$PREFILL_PORT" \ - --data-parallel-size 4 \ - --enable-expert-parallel \ - --tokenizer-mode deepseek_v4 \ - --reasoning-parser deepseek_v4 \ - --max-model-len auto \ - --max-num-batched-tokens 16384 \ - --max-num-seqs 8 \ - --enforce-eager \ - --no-disable-hybrid-kv-cache-manager \ - --disable-uvicorn-access-log \ - --kv-transfer-config "$PREFILL_KV_CONFIG" \ - > "$LOG_DIR/prefill.log" 2>&1 & -PREFILL_PID=$! -echo "Prefill PID: $PREFILL_PID" - -# Wait for prefill health -for i in {1..120}; do - if curl -s "http://127.0.0.1:$PREFILL_PORT/health" > /dev/null 2>&1; then - echo "Prefill server is ready" - break - fi - if ! kill -0 $PREFILL_PID 2>/dev/null; then - echo "ERROR: Prefill server exited early" - tail -80 "$LOG_DIR/prefill.log" - exit 1 - fi - echo "Waiting for prefill server... ($i/120)" - sleep 5 -done - -# Start decode server (GPUs 4-7) -echo "[2/3] Starting decode server on GPUs 4-7..." -CUDA_VISIBLE_DEVICES=4,5,6,7 nohup "$VLLM" serve "$MODEL_PATH" \ - --trust-remote-code \ - --kv-cache-dtype fp8 \ - --block-size 256 \ - --port "$DECODE_PORT" \ - --data-parallel-size 4 \ - --enable-expert-parallel \ - --tokenizer-mode deepseek_v4 \ - --reasoning-parser deepseek_v4 \ - --max-model-len auto \ - --max-num-seqs 512 \ - --max-num-batched-tokens 512 \ - --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","max_cudagraph_capture_size":512,"compile_ranges_endpoints":[512]}' \ - --no-disable-hybrid-kv-cache-manager \ - --disable-uvicorn-access-log \ - --kv-transfer-config "$DECODE_KV_CONFIG" \ - > "$LOG_DIR/decode.log" 2>&1 & -DECODE_PID=$! -echo "Decode PID: $DECODE_PID" - -# Wait for decode health -for i in {1..120}; do - if curl -s "http://127.0.0.1:$DECODE_PORT/health" > /dev/null 2>&1; then - echo "Decode server is ready" - break - fi - if ! kill -0 $DECODE_PID 2>/dev/null; then - echo "ERROR: Decode server exited early" - tail -80 "$LOG_DIR/decode.log" - exit 1 - fi - echo "Waiting for decode server... ($i/120)" - sleep 5 -done - -# Start vllm-router -echo "[3/3] Starting vllm-router on port $ROUTER_PORT..." -nohup "$VLLM_ROUTER" \ - --policy round_robin \ - --vllm-pd-disaggregation \ - --prefill "http://127.0.0.1:$PREFILL_PORT" \ - --decode "http://127.0.0.1:$DECODE_PORT" \ - --host 127.0.0.1 \ - --port "$ROUTER_PORT" \ - --intra-node-data-parallel-size 4 \ - --kv-connector mooncake \ - > "$LOG_DIR/router.log" 2>&1 & -ROUTER_PID=$! -echo "Router PID: $ROUTER_PID" - -# Wait for router health -for i in {1..60}; do - if curl -s "http://127.0.0.1:$ROUTER_PORT/health" > /dev/null 2>&1; then - echo "Router is ready" - break - fi - if ! kill -0 $ROUTER_PID 2>/dev/null; then - echo "ERROR: Router exited early" - tail -50 "$LOG_DIR/router.log" - exit 1 - fi - echo "Waiting for router... ($i/60)" - sleep 2 -done - -echo "" -echo "=== vLLM PD disaggregation is ready ===" -echo "Router URL: http://127.0.0.1:$ROUTER_PORT" -echo "Prefill URL: http://127.0.0.1:$PREFILL_PORT" -echo "Decode URL: http://127.0.0.1:$DECODE_PORT" -echo "" -echo "Test command:" -echo " curl http://127.0.0.1:$ROUTER_PORT/v1/completions \\" -echo " -H \"Content-Type: application/json\" \\" -echo " -d '{\"model\":\"DeepSeek-V4-Flash\",\"prompt\":\"Hello\",\"max_tokens\":32}'" -echo "" -echo "To stop: kill $PREFILL_PID $DECODE_PID $ROUTER_PID" diff --git a/scripts/wait_and_profile_dspark.sh b/scripts/wait_and_profile_dspark.sh deleted file mode 100755 index 944b5c2..0000000 --- a/scripts/wait_and_profile_dspark.sh +++ /dev/null @@ -1,60 +0,0 @@ -#!/bin/bash -# Wait for GPUs to become idle (no other compute processes), then run DSpark profiles. - -set -e - -ROOT=/data/user1/yy -PROFILE_PID_FILE=/tmp/profile_dspark.pid -MONITOR_LOG=$ROOT/bench_results/dspark_profile_monitor.log -RESULT_BASE=$ROOT/bench_results/dspark_profile_$(date +%Y%m%d_%H%M%S) - -mkdir -p "$RESULT_BASE" "$ROOT/tmp" - -log() { - echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$MONITOR_LOG" -} - -is_gpu_idle() { - # Consider idle if no process other than ours uses > 1GB on any GPU. - # Returns 0 (true) if idle, 1 (false) if busy. - nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader,nounits | \ - awk -F',' ' - { - pid=$1; mem=$2; - # Exclude our own known PIDs (monitor script, shell, etc.) - if (mem > 1024) { busy++; } - } - END { exit (busy > 0 ? 1 : 0) } - ' -} - -log "Starting GPU idle monitor. Will check every 5 minutes." -log "Results will go to $RESULT_BASE" - -wait_for_idle() { - while true; do - if is_gpu_idle; then - log "GPUs appear idle." - return 0 - else - log "GPUs still busy. Sleeping 5 minutes." - sleep 300 - fi - done -} - -# Run profiles sequentially, checking GPU idle state before each run. -CONFIGS=("dspark-st3:64" "dspark-st5:64" "dspark-st3:1" "nospec:64") - -for entry in "${CONFIGS[@]}"; do - IFS=':' read -r config concurrency <<< "$entry" - log "Waiting before profile: config=$config concurrency=$concurrency" - wait_for_idle - log "Running profile: config=$config concurrency=$concurrency" - "$ROOT/envs/vllm-dspark/bin/python" "$ROOT/scripts/profile_dspark.py" \ - --config "$config" --concurrency "$concurrency" --tool nsys \ - >> "$RESULT_BASE/profile_${config}_c${concurrency}.log" 2>&1 - log "Finished profile: config=$config concurrency=$concurrency" -done - -log "All profiles complete. Results in $RESULT_BASE"