docs(h200): H200 quick-start guide, porting workflow, and example experiment

- Update platforms/nvidia_h200.env with ENGINE and configurable venv/paths.
- Update platforms/README.md with H200 notes and platform-addition guide.
- Add docs/H200_QUICKSTART.md with concrete H200 usage steps.
- Add experiments/dsv4_h200_dspark/ as a migration wrapper around the
  existing scripts/benchmark_dspark_0707/ grid benchmark, writing results
  into the new experiments/<name>/results/<RUN_ID>/ layout.
- Update BENCHMARK_WORKFLOW.md with a 'Adding a New Platform or Experiment'
  section and mention the H200 wrapper in Quick Start.
- Update main README.md to index the H200 experiment and link to the guide.
This commit is contained in:
Quantong Qiu 2026-07-08 05:55:11 +00:00
parent f0d4c4753b
commit a5d69dfd9a
9 changed files with 374 additions and 7 deletions

View File

@ -218,12 +218,18 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
## Quick Start ## Quick Start
### Run P800 SGLang benchmark ### Run P800 SGLang benchmark (Kunlun P800, Docker)
```bash ```bash
bash experiments/dsv4_p800_sglang/run_bench.sh bash experiments/dsv4_p800_sglang/run_bench.sh
``` ```
### Run H200 DSpark benchmark (NVIDIA H200, native venv)
```bash
bash experiments/dsv4_h200_dspark/run_bench.sh
```
### Run legacy DSpark grid benchmark ### Run legacy DSpark grid benchmark
```bash ```bash
@ -254,6 +260,63 @@ python3 experiments/dsv4_p800_sglang/parse_results.py \
python3 scripts/analysis/compare_experiments.py python3 scripts/analysis/compare_experiments.py
``` ```
## Adding a New Platform or Experiment
### 1. Add or update a platform config
Create `platforms/<chip>.env` with identity and platform-wide paths:
```bash
CHIP="my_chip"
ACCELERATOR="My Accelerator"
HARDWARE="8x My Accelerator"
ENGINE="vllm-myengine"
DEFAULT_PORT="30000"
MODEL_ROOT="/data/models"
```
For Docker-based platforms, also set `DOCKER_IMAGE`, `CONTAINER_NAME`,
`CONTAINER_PYTHON`, and `PATCH_ROOT` (see `platforms/kunlun_p800.env`).
For host-native platforms, set the relevant venv paths (see
`platforms/nvidia_h200.env`).
### 2. Create an experiment directory
```
experiments/<name>/
├── README.md # Purpose and usage
├── config.env # Model, port, scenarios, engine overrides
├── start_server.sh # (optional) platform-specific server launch
├── run_bench.sh # Orchestrator
└── parse_results.py # Convert raw outputs to results.json + report.md
```
At minimum, `run_bench.sh` should:
1. Source `scripts/common/lib.sh` and `scripts/common/platform.sh`.
2. Read `config.env`.
3. Create `experiments/<name>/results/<RUN_ID>/`.
4. Call `write_metadata_json` to create `results.json`.
5. Run the benchmark scenarios.
6. Call `parse_results.py` to generate `report.md`.
### 3. Reuse shared helpers
- `scripts/common/lib.sh`: logging, health checks, metadata JSON.
- `scripts/common/platform.sh`: platform auto-detection and env loading.
- `scripts/common/server_docker.sh`: Docker-based server lifecycle.
- `scripts/common/bench_client_docker.sh`: run `sglang.bench_serving` inside a container.
Native (non-Docker) platforms can delegate to existing start scripts or add
new shared helpers under `scripts/common/`.
### 4. Example experiments
- Docker / XPU: `experiments/dsv4_p800_sglang/`
- Native / H200: `experiments/dsv4_h200_dspark/`
See `docs/H200_QUICKSTART.md` for a concrete H200 porting walkthrough.
## Checklist Before Committing / Archiving ## Checklist Before Committing / Archiving
- [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in the project root. - [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in the project root.

View File

@ -1,6 +1,7 @@
# DSV4 / DeepSeek-V4-Flash 推理测速项目 # DSV4 / DeepSeek-V4-Flash 推理测速项目
> 记录 vllm-dspark、SGLang、vLLM 等后端在 DeepSeek-V4-Flash 上的 benchmark 脚本与最终结果。 > 记录 vllm-dspark、SGLang、vLLM 等后端在 DeepSeek-V4-Flash 上的 benchmark 脚本与最终结果。
> H200 快速上手指南见 [`docs/H200_QUICKSTART.md`](docs/H200_QUICKSTART.md)。
## 目录说明 ## 目录说明
@ -24,6 +25,7 @@
| 实验 | 脚本 | 说明 | | 实验 | 脚本 | 说明 |
|---|---|---| |---|---|---|
| DSV4 P800 SGLang | `experiments/dsv4_p800_sglang/run_bench.sh` | Kunlun P800 + SGLang + DeepSeek-V4-Flash-INT8 | | DSV4 P800 SGLang | `experiments/dsv4_p800_sglang/run_bench.sh` | Kunlun P800 + SGLang + DeepSeek-V4-Flash-INT8 |
| DSV4 H200 DSpark | `experiments/dsv4_h200_dspark/run_bench.sh` | NVIDIA H200 + vllm-dspark + DeepSeek-V4-Flash-DSpark |
### 旧结构scripts/ + bench_results/ ### 旧结构scripts/ + bench_results/
@ -46,7 +48,13 @@
bash experiments/dsv4_p800_sglang/run_bench.sh bash experiments/dsv4_p800_sglang/run_bench.sh
``` ```
### DSpark grid ### H200 DSpark
```bash
bash experiments/dsv4_h200_dspark/run_bench.sh
```
### DSpark grid旧结构
```bash ```bash
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
@ -65,6 +73,9 @@ bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
python3 experiments/dsv4_p800_sglang/parse_results.py \ python3 experiments/dsv4_p800_sglang/parse_results.py \
experiments/dsv4_p800_sglang/results/<run_id> experiments/dsv4_p800_sglang/results/<run_id>
python3 experiments/dsv4_h200_dspark/parse_results.py \
experiments/dsv4_h200_dspark/results/<run_id>
# 旧结构 # 旧结构
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \ /data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
/data/user1/yy/bench_results/dspark_grid_20260707-132641 /data/user1/yy/bench_results/dspark_grid_20260707-132641

111
docs/H200_QUICKSTART.md Normal file
View File

@ -0,0 +1,111 @@
# H200 Quick Start Guide
This guide covers how to use this benchmark repository on an NVIDIA H200 machine.
## Current state
The H200 path is currently based on **native host virtual environments** (not Docker):
- Server engine: `vllm-dspark`
- Server env: `envs/vllm-dspark`
- Benchmark client env: `envs/sglang` (uses `sglang.bench_serving --backend vllm`)
- Default model: `/data/models/DeepSeek-V4-Flash-DSpark`
- Default port: `30004`
The legacy benchmark scripts live under `scripts/benchmark_dspark_0707/`. A migration wrapper experiment `experiments/dsv4_h200_dspark/` is also provided so the results follow the new `experiments/<name>/results/<RUN_ID>/` layout.
## 1. Pull and verify
```bash
git clone <repo-url> /data/user1/yy
cd /data/user1/yy
# Platform should auto-detect as nvidia_h200
source scripts/common/platform.sh
```
If auto-detection fails, set it explicitly:
```bash
PLATFORM=nvidia_h200 source scripts/common/platform.sh
```
## 2. Run the existing DSpark benchmark grid
This uses the legacy scripts and writes results to `bench_results/dspark_grid_<RUN_ID>/`.
```bash
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
```
To reuse an already-running server:
```bash
SKIP_MANAGE_SERVER=1 bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
```
## 3. Run the migration wrapper experiment
This produces results in the new experiment-centric layout:
```bash
bash experiments/dsv4_h200_dspark/run_bench.sh
```
Results land in `experiments/dsv4_h200_dspark/results/<RUN_ID>/`.
## 4. Create a new H200 experiment
To add a new H200 benchmark (for example a different model or engine), create:
```
experiments/<your_name>/
├── README.md # What this experiment measures
├── config.env # Model, port, scenarios, venv paths
├── start_server.sh # (optional) native server launch
├── run_bench.sh # Orchestrator: server → benchmark → stop
└── parse_results.py # Generate results.json + report.md
```
Minimum `config.env`:
```bash
EXPERIMENT="${EXPERIMENT:-<your_name>}"
MODEL_NAME="${MODEL_NAME:-DeepSeek-V4-Flash-DSpark}"
MODEL_PATH="${MODEL_PATH:-/data/models/DeepSeek-V4-Flash-DSpark}"
PORT="${PORT:-30004}"
BACKEND="${BACKEND:-vllm}"
ENGINE="${ENGINE:-vllm-dspark}"
# Native venvs
VENV_SERVER="${VENV_SERVER:-/data/user1/yy/envs/vllm-dspark}"
VENV_CLIENT="${VENV_CLIENT:-/data/user1/yy/envs/sglang}"
SCENARIOS=(
"32 512 256"
)
```
Then source the platform loader and shared helpers at the top of `run_bench.sh`:
```bash
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
```
For a concrete example, see `experiments/dsv4_h200_dspark/`.
## 5. Cross-platform comparison
After running experiments on both P800 and H200, compare them with:
```bash
python3 scripts/analysis/compare_experiments.py \
experiments/dsv4_p800_sglang/results/<run_id> \
experiments/dsv4_h200_dspark/results/<run_id>
```
## Notes
- `platforms/nvidia_h200.env` assumes the legacy directory layout under `/data/user1/yy`. If your H200 machine uses different paths, override `VENV_VLLM_DSPARK`, `VENV_SGLANG`, `MODEL_ROOT`, or `SERVER_START_SCRIPT`.
- Native server management helpers are not yet as mature as the Docker helpers in `scripts/common/server_docker.sh`. The H200 wrapper currently delegates to the existing `scripts/start_dsv4_dspark_8card.sh`.

View File

@ -0,0 +1,41 @@
# DSV4 H200 DSpark Benchmark
NVIDIA H200 + native `vllm-dspark` + `DeepSeek-V4-Flash-DSpark` benchmark experiment.
This is a migration wrapper: it reuses the proven legacy scripts under
`scripts/benchmark_dspark_0707/` and `scripts/start_dsv4_dspark_8card.sh`,
but stores results in the new `experiments/<name>/results/<RUN_ID>/` layout.
## Quick Start
```bash
# Run the full DSpark P1/P2/P3 grid
bash experiments/dsv4_h200_dspark/run_bench.sh
# Reuse an already-running server
SKIP_MANAGE_SERVER=1 bash experiments/dsv4_h200_dspark/run_bench.sh
```
Results land in `experiments/dsv4_h200_dspark/results/<RUN_ID>/`.
## Configuration
Edit `config.env` or override via environment variables:
```bash
MODEL_PATH=/data/models/DeepSeek-V4-Flash-DSpark \
PORT=30004 \
bash experiments/dsv4_h200_dspark/run_bench.sh
```
## Files
| File | Purpose |
|---|---|
| `config.env` | Experiment-level configuration (model, port, venv paths) |
| `run_bench.sh` | Orchestrator: metadata → legacy grid benchmark → parse |
| `parse_results.py` | Wraps the legacy parser to generate `report.md` |
## Platform
This experiment targets `platforms/nvidia_h200.env`.

View File

@ -0,0 +1,22 @@
# Experiment-level configuration for dsv4_h200_dspark
# All values can be overridden via environment variables.
EXPERIMENT="${EXPERIMENT:-dsv4_h200_dspark}"
MODEL_NAME="${MODEL_NAME:-DeepSeek-V4-Flash-DSpark}"
MODEL_PATH="${MODEL_PATH:-/data/models/DeepSeek-V4-Flash-DSpark}"
SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-deepseek-v4-flash-dspark}"
PORT="${PORT:-30004}"
BACKEND="${BACKEND:-vllm}"
ENGINE="${ENGINE:-vllm-dspark}"
# Native virtual environments on the host.
# Override if your H200 machine uses different paths.
VENV_SERVER="${VENV_SERVER:-/data/user1/yy/envs/vllm-dspark}"
VENV_CLIENT="${VENV_CLIENT:-/data/user1/yy/envs/sglang}"
# Legacy server start script.
SERVER_START_SCRIPT="${SERVER_START_SCRIPT:-${ROOT_DIR}/scripts/start_dsv4_dspark_8card.sh}"
# Legacy benchmark orchestrator.
LEGACY_GRID_SCRIPT="${LEGACY_GRID_SCRIPT:-${ROOT_DIR}/scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh}"
LEGACY_PARSE_SCRIPT="${LEGACY_PARSE_SCRIPT:-${ROOT_DIR}/scripts/benchmark_dspark_0707/parse_results.py}"

View File

@ -0,0 +1,36 @@
#!/usr/bin/env python3
"""Parse H200 DSpark benchmark outputs.
This is a thin wrapper around the legacy parser in
scripts/benchmark_dspark_0707/parse_results.py. It exists so that the
experiment keeps the same entry point as other experiments.
Usage:
python3 parse_results.py <result_root>
"""
import os
import subprocess
import sys
from pathlib import Path
def main() -> None:
result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("results")
repo_root = Path(__file__).resolve().parents[2]
legacy_parser = repo_root / "scripts" / "benchmark_dspark_0707" / "parse_results.py"
if not legacy_parser.exists():
raise SystemExit(f"legacy parser not found: {legacy_parser}")
venv_client = os.environ.get("VENV_CLIENT", "/data/user1/yy/envs/sglang")
python = Path(venv_client) / "bin" / "python"
if not python.exists():
python = Path(sys.executable)
subprocess.run([str(python), str(legacy_parser), str(result_root)], check=True)
print(f"Parsed {result_root}")
if __name__ == "__main__":
main()

View File

@ -0,0 +1,66 @@
#!/usr/bin/env bash
# H200 native vllm-dspark benchmark orchestrator.
# This wraps the legacy scripts/benchmark_dspark_0707/ grid benchmark so that
# results live under experiments/dsv4_h200_dspark/results/<RUN_ID>/.
set -Eeuo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
# shellcheck source=/dev/null
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
# shellcheck source=/dev/null
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
# shellcheck source=/dev/null
source "${SCRIPT_DIR}/config.env"
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
RESULT_ROOT="${RESULT_ROOT:-${SCRIPT_DIR}/results/${RUN_ID}}"
LOG_DIR="${RESULT_ROOT}/logs"
ensure_result_root "$RESULT_ROOT"
log_init "${LOG_DIR}/orchestrator.log"
log "experiment=${EXPERIMENT_NAME}"
log "run_id=${RUN_ID}"
log "result_root=${RESULT_ROOT}"
log "platform=${PLATFORM}"
log "chip=${CHIP}"
log "accelerator=${ACCELERATOR}"
log "engine=${ENGINE}"
log "hardware=${HARDWARE}"
log "model=${MODEL_PATH}"
log "server_start_script=${SERVER_START_SCRIPT}"
# Write initial metadata for this run.
METADATA_JSON="${RESULT_ROOT}/results.json"
write_metadata_json \
"$METADATA_JSON" \
"$EXPERIMENT_NAME" \
"$RUN_ID" \
"$MODEL_PATH" \
"$BACKEND" \
"$ENGINE" \
"$HARDWARE" \
"$ACCELERATOR" \
"$CHIP" \
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
"$VENV_SERVER" \
"H200 native vllm-dspark benchmark for DeepSeek-V4-Flash-DSpark"
log "starting legacy DSpark grid benchmark"
RUN_ID="$RUN_ID" \
RESULT_ROOT="$RESULT_ROOT" \
PORT="$PORT" \
MODEL_PATH="$MODEL_PATH" \
SERVER_START_SCRIPT="$SERVER_START_SCRIPT" \
SKIP_MANAGE_SERVER="${SKIP_MANAGE_SERVER:-}" \
bash "$LEGACY_GRID_SCRIPT" >> "${LOG_DIR}/legacy_grid.log" 2>&1
log "parsing results"
"$VENV_CLIENT/bin/python" "$LEGACY_PARSE_SCRIPT" "$RESULT_ROOT" >> "${LOG_DIR}/parse.log" 2>&1 || {
log "WARNING: legacy parser failed; see ${LOG_DIR}/parse.log"
}
log "all results saved to ${RESULT_ROOT}"

View File

@ -19,14 +19,15 @@ PLATFORM=kunlun_p800 bash experiments/dsv4_p800_sglang/run_bench.sh
| File | Chip/Accelerator | Engine | Notes | | File | Chip/Accelerator | Engine | Notes |
|---|---|---|---| |---|---|---|---|
| `kunlun_p800.env` | Kunlun P800 XPU | `sglang-xpu` | Docker-based SGLang serving image | | `kunlun_p800.env` | Kunlun P800 XPU | `sglang-xpu` | Docker-based SGLang serving image |
| `nvidia_h200.env` | NVIDIA H200 | native vllm/sglang | Host virtual environments | | `nvidia_h200.env` | NVIDIA H200 | `vllm-dspark` | Native host virtual environments |
## What belongs here ## What belongs here
- Chip/accelerator identity (`CHIP`, `ACCELERATOR`, `HARDWARE`). - Chip/accelerator identity (`CHIP`, `ACCELERATOR`, `HARDWARE`, `ENGINE`).
- Device selection environment variables. - Device selection environment variables.
- Platform-wide paths that rarely change (model root, default port). - Platform-wide paths that rarely change (model root, default port).
- Container image / interpreter paths for Docker-based platforms. - Container image / interpreter paths for Docker-based platforms.
- Native interpreter / venv paths for host-based platforms.
## What does NOT belong here ## What does NOT belong here
@ -34,3 +35,14 @@ PLATFORM=kunlun_p800 bash experiments/dsv4_p800_sglang/run_bench.sh
`experiments/<name>/config.env`. `experiments/<name>/config.env`.
- Engine-specific launch flags — those go in the experiment's - Engine-specific launch flags — those go in the experiment's
`start_server.sh` or `run_bench.sh`. `start_server.sh` or `run_bench.sh`.
## Adding a new platform
1. Create `platforms/<chip>.env` with at least `CHIP`, `ACCELERATOR`,
`HARDWARE`, `ENGINE`, `DEFAULT_PORT`, `MODEL_ROOT`.
2. If the platform runs inside Docker, set `DOCKER_IMAGE`, `CONTAINER_NAME`,
`CONTAINER_PYTHON`, and `PATCH_ROOT` (see `kunlun_p800.env`).
3. If the platform runs natively on the host, set the relevant venv paths
(see `nvidia_h200.env`).
4. Add a row to the table above and write a quick-start experiment under
`experiments/<name>/`.

View File

@ -4,6 +4,7 @@
CHIP="nvidia_h200" CHIP="nvidia_h200"
ACCELERATOR="NVIDIA H200" ACCELERATOR="NVIDIA H200"
HARDWARE="8x NVIDIA H200 143GB" HARDWARE="8x NVIDIA H200 143GB"
ENGINE="vllm-dspark"
# Device selection # Device selection
DEVICE_SELECT_ENV="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7" DEVICE_SELECT_ENV="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
@ -13,6 +14,10 @@ CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
DEFAULT_PORT="30004" DEFAULT_PORT="30004"
MODEL_ROOT="/data/models" MODEL_ROOT="/data/models"
# Virtual environments on the host (used by native H200 scripts) # Virtual environments on the host (used by native H200 scripts).
VENV_VLLM_DSPARK="/data/user1/yy/envs/vllm-dspark" # Override these if your H200 machine uses different paths.
VENV_SGLANG="/data/user1/yy/envs/sglang" VENV_VLLM_DSPARK="${VENV_VLLM_DSPARK:-/data/user1/yy/envs/vllm-dspark}"
VENV_SGLANG="${VENV_SGLANG:-/data/user1/yy/envs/sglang}"
# Default server start script for the legacy DSpark benchmark grid.
SERVER_START_SCRIPT="${SERVER_START_SCRIPT:-${ROOT_DIR}/scripts/start_dsv4_dspark_8card.sh}"