refactor: experiments/ + platforms/ layout and P800 SGLang support
- Restructure repo around experiments/<name>/ and platforms/<chip>.env. - Add shared scripts under scripts/common/ for platform/server/bench-client logic. - Add Kunlun P800 platform config and runtime patches. - Add dsv4_p800_sglang experiment with INT8 smoke-test support. - Update BENCHMARK_WORKFLOW.md and README.md with chip/engine recording rules. - Add scripts/analysis/compare_experiments.py for cross-experiment comparison. - Ignore experiments/*/results/ raw output directories by default.
This commit is contained in:
Quantong Qiu
parent
2c332ff712
commit
227ef603a6
3
.gitignore
vendored
3
.gitignore
vendored
@ -23,5 +23,8 @@ datasets/
|
|||||||
# 原始请求级输出（方案 A：极简版，不存原始 jsonl）
|
# 原始请求级输出（方案 A：极简版，不存原始 jsonl）
|
||||||
bench_results/**/raw_outputs/
|
bench_results/**/raw_outputs/
|
||||||
|
|
||||||
|
# 实验级原始结果目录（report.md / results.json 可单独保留，默认忽略整个目录）
|
||||||
|
experiments/*/results/
|
||||||
|
|
||||||
# 无关项目
|
# 无关项目
|
||||||
loomeval_yy/
|
loomeval_yy/
|
||||||
|
|||||||
@ -4,20 +4,29 @@
|
|||||||
|
|
||||||
```
|
```
|
||||||
/data/user1/yy/
|
/data/user1/yy/
|
||||||
├── scripts/ # all benchmark/orchestrator/utility scripts
|
├── platforms/ # chip/accelerator platform configs
|
||||||
│ ├── benchmark_dspark_0707/ # DSpark benchmark suite
|
│ ├── kunlun_p800.env
|
||||||
│ ├── benchmark_dsv4_backend_comparison.sh
|
│ ├── nvidia_h200.env
|
||||||
│ ├── start_dsv4_dspark_8card.sh
|
│ └── patches/kunlun_p800/ # runtime patches required by some images
|
||||||
│ ├── start_sglang_dsv4_8card.sh
|
├── scripts/ # shared benchmark/orchestrator/utility scripts
|
||||||
|
│ ├── common/ # reusable components (lib.sh, platform.sh, ...)
|
||||||
|
│ ├── analysis/ # cross-experiment comparison tools
|
||||||
|
│ ├── benchmark_dspark_0707/ # legacy DSpark benchmark suite
|
||||||
│ └── ...
|
│ └── ...
|
||||||
├── bench_results/ # all benchmark outputs and reports
|
├── experiments/ # experiment-centric directories (preferred)
|
||||||
│ ├── dsv4_backend_comparison_20260707/
|
│ └── dsv4_p800_sglang/
|
||||||
│ │ ├── raw_outputs/ # JSONL raw outputs
|
│ ├── README.md
|
||||||
│ │ ├── logs/ # per-run logs
|
│ ├── config.env # experiment-level configuration
|
||||||
│ │ └── README.md # output manifest + provenance
|
│ ├── start_server.sh
|
||||||
|
│ ├── run_bench.sh
|
||||||
|
│ ├── parse_results.py
|
||||||
|
│ └── results/
|
||||||
|
│ └── 20260708-XXXXXX/
|
||||||
|
│ ├── report.md
|
||||||
|
│ ├── results.json
|
||||||
|
│ └── logs/
|
||||||
|
├── bench_results/ # legacy benchmark outputs (read-only history)
|
||||||
│ ├── dspark_grid_20260707-132641/
|
│ ├── dspark_grid_20260707-132641/
|
||||||
│ ├── dspark_st_comparison_20260707-150649/
|
|
||||||
│ ├── eagle_grid/
|
|
||||||
│ └── ...
|
│ └── ...
|
||||||
├── logs/ # server logs (stdout/stderr from start scripts)
|
├── logs/ # server logs (stdout/stderr from start scripts)
|
||||||
├── datasets/ # benchmark datasets
|
├── datasets/ # benchmark datasets
|
||||||
@ -26,16 +35,16 @@
|
|||||||
|
|
||||||
## Rules
|
## Rules
|
||||||
|
|
||||||
1. **Scripts live in `scripts/` only.**
|
1. **Experiments are the primary organization unit.**
|
||||||
- Group related scripts into subdirectories, e.g. `scripts/benchmark_dspark_0707/`.
|
- Each experiment lives under `experiments/<experiment_name>/` and contains its scripts, configuration, and results.
|
||||||
- Each script group should have its own `README.md` listing scripts, purpose, and outputs.
|
- Shared orchestration code lives in `scripts/common/`; do not copy server start / health check logic into every experiment.
|
||||||
|
- Legacy experiments may remain under `scripts/<group>/` with outputs in `bench_results/`, but new work should use `experiments/`.
|
||||||
|
|
||||||
2. **Benchmark outputs live in `bench_results/` only.**
|
2. **Benchmark outputs live with their experiment.**
|
||||||
- Never leave `.jsonl`, `.json`, `.log`, or `.md` reports in the project root.
|
- For `experiments/<name>/`, results go in `experiments/<name>/results/<RUN_ID>/`.
|
||||||
- Each benchmark run gets its own directory: `bench_results/<experiment>_<timestamp>/`.
|
- Each run directory must contain `report.md` (human-readable) and `results.json` (structured data).
|
||||||
- Raw outputs go in `raw_outputs/`.
|
- Raw outputs go in `raw_outputs/`; logs go in `logs/`.
|
||||||
- Logs go in `logs/`.
|
- Legacy `bench_results/<experiment>_<timestamp>/` directories remain valid for archived runs.
|
||||||
- Reports (e.g. `report.md`, `comparison_report.md`) go in the run root.
|
|
||||||
|
|
||||||
3. **Each `bench_results/<run>/` directory must contain two final artifacts.**
|
3. **Each `bench_results/<run>/` directory must contain two final artifacts.**
|
||||||
- A Markdown report for human reading (e.g. `report.md`, `comparison_report.md`).
|
- A Markdown report for human reading (e.g. `report.md`, `comparison_report.md`).
|
||||||
@ -50,7 +59,9 @@
|
|||||||
- Keep the schema stable so downstream Python scripts can parse all experiments uniformly.
|
- Keep the schema stable so downstream Python scripts can parse all experiments uniformly.
|
||||||
- See [Final JSON Schema](#final-json-schema) below for the recommended structure.
|
- See [Final JSON Schema](#final-json-schema) below for the recommended structure.
|
||||||
|
|
||||||
5. **Scripts should default `RESULT_ROOT` to `bench_results/<experiment>_${RUN_ID}`.**
|
5. **Scripts should default `RESULT_ROOT` to the experiment's results directory.**
|
||||||
|
- For `experiments/<name>/run_bench.sh`, default to `experiments/<name>/results/${RUN_ID}/`.
|
||||||
|
- For legacy scripts, default to `bench_results/<experiment>_${RUN_ID}/`.
|
||||||
- Allow override via `RESULT_ROOT` env var.
|
- Allow override via `RESULT_ROOT` env var.
|
||||||
- Use `RUN_ID=$(date '+%Y%m%d-%H%M%S')` unless specified.
|
- Use `RUN_ID=$(date '+%Y%m%d-%H%M%S')` unless specified.
|
||||||
|
|
||||||
@ -63,9 +74,33 @@
|
|||||||
- Final reports and JSON outputs must include both the accelerator/chip family and the inference engine/backend used for the run.
|
- Final reports and JSON outputs must include both the accelerator/chip family and the inference engine/backend used for the run.
|
||||||
- Do not rely on the experiment name alone to identify the platform or engine.
|
- Do not rely on the experiment name alone to identify the platform or engine.
|
||||||
|
|
||||||
|
8. **Use platform configuration files for chip-specific constants.**
|
||||||
|
- Put per-platform settings in `platforms/<chip>.env` (e.g. `platforms/kunlun_p800.env`, `platforms/nvidia_h200.env`).
|
||||||
|
- Scripts load the platform file via `scripts/common/platform.sh`; the active platform is selected by the `PLATFORM` env var or auto-detected.
|
||||||
|
- Keep experiment scripts free of hardcoded device IDs, image names, or model root paths.
|
||||||
|
|
||||||
## Naming Conventions
|
## Naming Conventions
|
||||||
|
|
||||||
### Result directories
|
### Experiment result directories
|
||||||
|
|
||||||
|
For experiment-centric layout:
|
||||||
|
|
||||||
|
```
|
||||||
|
experiments/<experiment>/results/<YYYYMMDD-HHMMSS>/
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
|
||||||
|
- `experiments/dsv4_p800_sglang/results/20260708-120000/`
|
||||||
|
- `experiments/dspark_grid/results/20260707-132641/`
|
||||||
|
|
||||||
|
The `results.json` metadata already records `chip`/`accelerator` and `engine`, so the directory path does not need to encode them. If a single experiment must distinguish across platforms in its directory tree, use:
|
||||||
|
|
||||||
|
```
|
||||||
|
experiments/<experiment>/results/<chip>_<engine>_<YYYYMMDD-HHMMSS>/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Legacy result directories
|
||||||
|
|
||||||
```
|
```
|
||||||
bench_results/<experiment>_<YYYYMMDD-HHMMSS>/
|
bench_results/<experiment>_<YYYYMMDD-HHMMSS>/
|
||||||
@ -76,13 +111,6 @@ Examples:
|
|||||||
- `bench_results/dspark_grid_20260707-132641/`
|
- `bench_results/dspark_grid_20260707-132641/`
|
||||||
- `bench_results/dsv4_backend_comparison_20260707/`
|
- `bench_results/dsv4_backend_comparison_20260707/`
|
||||||
|
|
||||||
When the same experiment is repeated across chips or engines, include them in the directory name or organize by subdirectories so results are not confused:
|
|
||||||
|
|
||||||
```
|
|
||||||
bench_results/<experiment>_<chip>_<engine>_<YYYYMMDD-HHMMSS>/
|
|
||||||
bench_results/<experiment>/<chip>/<engine>/<YYYYMMDD-HHMMSS>/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Raw output files
|
### Raw output files
|
||||||
|
|
||||||
Include the accelerator and inference engine in raw output filenames so files from different platforms cannot overwrite each other.
|
Include the accelerator and inference engine in raw output filenames so files from different platforms cannot overwrite each other.
|
||||||
@ -190,7 +218,13 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
|
|||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
### Run DSpark grid benchmark
|
### Run P800 SGLang benchmark
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash experiments/dsv4_p800_sglang/run_bench.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run legacy DSpark grid benchmark
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
|
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
|
||||||
@ -202,28 +236,33 @@ bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
|
|||||||
bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
|
bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
### Run SGLang vs vLLM backend comparison
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Start SGLang on port 30000 and vLLM on port 8000, then:
|
|
||||||
bash scripts/benchmark_dsv4_backend_comparison.sh all
|
|
||||||
```
|
|
||||||
|
|
||||||
### Parse results
|
### Parse results
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# Experiment-centric layout
|
||||||
|
python3 experiments/dsv4_p800_sglang/parse_results.py \
|
||||||
|
experiments/dsv4_p800_sglang/results/<run_id>
|
||||||
|
|
||||||
|
# Legacy layout
|
||||||
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
|
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
|
||||||
/data/user1/yy/bench_results/dspark_grid_<run_id>
|
/data/user1/yy/bench_results/dspark_grid_<run_id>
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Cross-experiment comparison
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/analysis/compare_experiments.py
|
||||||
|
```
|
||||||
|
|
||||||
## Checklist Before Committing / Archiving
|
## Checklist Before Committing / Archiving
|
||||||
|
|
||||||
- [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in `/data/user1/yy/` root.
|
- [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in the project root.
|
||||||
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
|
- [ ] For `experiments/<name>/`, outputs live in `experiments/<name>/results/<timestamp>/`.
|
||||||
- [ ] `bench_results/<run>/report.md` (or equivalent human-readable `.md`) exists.
|
- [ ] For legacy runs, outputs live in `bench_results/<experiment>_<timestamp>/`.
|
||||||
- [ ] `bench_results/<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
|
- [ ] `<run>/report.md` (or equivalent human-readable `.md`) exists.
|
||||||
- [ ] `bench_results/<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
|
- [ ] `<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
|
||||||
|
- [ ] `<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
|
||||||
- [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide.
|
- [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide.
|
||||||
- [ ] `bench_results/<run>/README.md` exists and documents provenance (or the report itself covers provenance).
|
- [ ] `<run>/README.md` exists and documents provenance (or the report itself covers provenance).
|
||||||
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
|
- [ ] Scripts either live under `experiments/<name>/` or in `scripts/` (or `scripts/<group>/`).
|
||||||
- [ ] Script path references updated after moving.
|
- [ ] Script path references updated after moving.
|
||||||
|
|||||||
34
README.md
34
README.md
@ -6,8 +6,12 @@
|
|||||||
|
|
||||||
| 目录/文件 | 说明 |
|
| 目录/文件 | 说明 |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `scripts/` | 所有 benchmark 脚本、服务启动脚本、结果解析脚本 |
|
| `platforms/` | 芯片/加速器平台配置（`*.env`），如 P800、H200 |
|
||||||
| `bench_results/` | 各次实验的最终产物：人可读 `.md` 报告 + 结构化 `results.json`（含原始数据与分位统计） |
|
| `scripts/common/` | 跨实验复用的 orchestration 组件（server 启停、health check、元数据生成） |
|
||||||
|
| `scripts/analysis/` | 跨实验对比分析工具 |
|
||||||
|
| `scripts/` | 旧版/legacy benchmark 脚本 |
|
||||||
|
| `experiments/` | 以实验为单位的自包含目录（脚本 + 配置 + 结果） |
|
||||||
|
| `bench_results/` | 历史归档的实验产物（只读） |
|
||||||
| `BENCHMARK_WORKFLOW.md` | benchmark 目录与命名规范 |
|
| `BENCHMARK_WORKFLOW.md` | benchmark 目录与命名规范 |
|
||||||
| `scripts/SLO_STANDARDS.md` | 推理服务 SLO 标准（TTFT/TPOT） |
|
| `scripts/SLO_STANDARDS.md` | 推理服务 SLO 标准（TTFT/TPOT） |
|
||||||
|
|
||||||
@ -15,6 +19,14 @@
|
|||||||
|
|
||||||
## 实验索引
|
## 实验索引
|
||||||
|
|
||||||
|
### 新结构（experiments/）
|
||||||
|
|
||||||
|
| 实验 | 脚本 | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| DSV4 P800 SGLang | `experiments/dsv4_p800_sglang/run_bench.sh` | Kunlun P800 + SGLang + DeepSeek-V4-Flash-INT8 |
|
||||||
|
|
||||||
|
### 旧结构（scripts/ + bench_results/）
|
||||||
|
|
||||||
| 实验 | 脚本 | 最终报告 | 说明 |
|
| 实验 | 脚本 | 最终报告 | 说明 |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| DSpark grid benchmark | `scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh` | `bench_results/dspark_grid_20260707-132641/report.md` | P1/P2/P3 全量网格 |
|
| DSpark grid benchmark | `scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh` | `bench_results/dspark_grid_20260707-132641/report.md` | P1/P2/P3 全量网格 |
|
||||||
@ -28,6 +40,12 @@
|
|||||||
|
|
||||||
## 快速复现
|
## 快速复现
|
||||||
|
|
||||||
|
### P800 SGLang
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash experiments/dsv4_p800_sglang/run_bench.sh
|
||||||
|
```
|
||||||
|
|
||||||
### DSpark grid
|
### DSpark grid
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@ -43,6 +61,11 @@ bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
|
|||||||
### 解析已有结果
|
### 解析已有结果
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# 新结构
|
||||||
|
python3 experiments/dsv4_p800_sglang/parse_results.py \
|
||||||
|
experiments/dsv4_p800_sglang/results/<run_id>
|
||||||
|
|
||||||
|
# 旧结构
|
||||||
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
|
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
|
||||||
/data/user1/yy/bench_results/dspark_grid_20260707-132641
|
/data/user1/yy/bench_results/dspark_grid_20260707-132641
|
||||||
```
|
```
|
||||||
@ -58,6 +81,7 @@ bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
|
|||||||
|
|
||||||
## 环境要求
|
## 环境要求
|
||||||
|
|
||||||
- Python env：`/data/user1/yy/envs/vllm-dspark`（服务端）、`/data/user1/yy/envs/sglang`（压测客户端）
|
- **NVIDIA H200**：Python env `/data/user1/yy/envs/vllm-dspark`（服务端）、`/data/user1/yy/envs/sglang`（压测客户端）；模型 `/data/models/DeepSeek-V4-Flash`、`/data/models/DeepSeek-V4-Flash-DSpark`。
|
||||||
- 模型：`/data/models/DeepSeek-V4-Flash`、`/data/models/DeepSeek-V4-Flash-DSpark`
|
- **Kunlun P800**：Docker 镜像 `iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202`；模型 `/data1/models/DeepSeek-V4-Flash-INT8`；压测客户端在容器内运行。
|
||||||
- 硬件：8× H200（当前配置）
|
|
||||||
|
平台相关常量见 `platforms/*.env`。
|
||||||
|
|||||||
28
experiments/dsv4_p800_sglang/README.md
Normal file
28
experiments/dsv4_p800_sglang/README.md
Normal file
@ -0,0 +1,28 @@
|
|||||||
|
# DSV4 P800 SGLang Benchmark
|
||||||
|
|
||||||
|
Kunlun P800 XPU + SGLang + `DeepSeek-V4-Flash-INT8` benchmark experiment.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run all configured scenarios (starts server, runs benchmark, stops server)
|
||||||
|
bash experiments/dsv4_p800_sglang/run_bench.sh
|
||||||
|
|
||||||
|
# Reuse an already-running server
|
||||||
|
SKIP_MANAGE_SERVER=1 bash experiments/dsv4_p800_sglang/run_bench.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Results land in `experiments/dsv4_p800_sglang/results/<RUN_ID>/`.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `config.env` | Experiment-level configuration (model, port, scenarios) |
|
||||||
|
| `start_server.sh` | Start the P800 SGLang Docker container |
|
||||||
|
| `run_bench.sh` | Orchestrator: server → benchmark → stop server |
|
||||||
|
| `parse_results.py` | Parse logs and generate `results.json` + `report.md` |
|
||||||
|
|
||||||
|
## Platform
|
||||||
|
|
||||||
|
This experiment targets `platforms/kunlun_p800.env`.
|
||||||
35
experiments/dsv4_p800_sglang/config.env
Normal file
35
experiments/dsv4_p800_sglang/config.env
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
# Experiment-level configuration for dsv4_p800_sglang
|
||||||
|
# All values can be overridden via environment variables.
|
||||||
|
|
||||||
|
EXPERIMENT="${EXPERIMENT:-dsv4_p800_sglang}"
|
||||||
|
MODEL_NAME="${MODEL_NAME:-DeepSeek-V4-Flash-INT8}"
|
||||||
|
MODEL_PATH="${MODEL_PATH:-/data1/models/DeepSeek-V4-Flash-INT8}"
|
||||||
|
SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-deepseek-v4-flash-int8}"
|
||||||
|
PORT="${PORT:-30000}"
|
||||||
|
BACKEND="${BACKEND:-sglang}"
|
||||||
|
ENGINE="${ENGINE:-sglang-xpu}"
|
||||||
|
DATASET="${DATASET:-random}"
|
||||||
|
# Path to a local ShareGPT-style dataset inside the container.
|
||||||
|
# Required when HF Hub is offline. The repo ships a dummy dataset under
|
||||||
|
# platforms/patches/kunlun_p800/dummy_sharegpt.json; for managed mode it is
|
||||||
|
# mounted to /workspace/dummy_sharegpt.json.
|
||||||
|
DATASET_PATH="${DATASET_PATH:-/workspace/dummy_sharegpt.json}"
|
||||||
|
WARMUP="${WARMUP:-100}"
|
||||||
|
NUM_PROMPTS="${NUM_PROMPTS:-512}"
|
||||||
|
|
||||||
|
# Server launch mode for P800 SGLang.
|
||||||
|
# - "fp8" : non-INT8 DeepSeek-V4-Flash checkpoint
|
||||||
|
# - "w8a8_int8": DeepSeek-V4-Flash-INT8 checkpoint
|
||||||
|
SERVER_MODE="${SERVER_MODE:-w8a8_int8}"
|
||||||
|
|
||||||
|
# Scenario list: "concurrency input_len output_len" (one per line)
|
||||||
|
# Override example: SCENARIOS=("32 512 256")
|
||||||
|
if [[ -z "${SCENARIOS:-}" ]]; then
|
||||||
|
SCENARIOS=(
|
||||||
|
"32 512 256"
|
||||||
|
"128 512 256"
|
||||||
|
"256 512 256"
|
||||||
|
"512 512 256"
|
||||||
|
)
|
||||||
|
fi
|
||||||
|
|
||||||
313
experiments/dsv4_p800_sglang/parse_results.py
Executable file
313
experiments/dsv4_p800_sglang/parse_results.py
Executable file
@ -0,0 +1,313 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Parse P800 SGLang benchmark outputs and produce results.json + report.md.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 parse_results.py <result_root>
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from collections import OrderedDict
|
||||||
|
|
||||||
|
METRIC_PATTERNS = OrderedDict(
|
||||||
|
[
|
||||||
|
("successful_requests", [r"Successful requests:\s+(\d+)"]),
|
||||||
|
("benchmark_duration_s", [r"Benchmark duration \(s\):\s+([\d.]+)"]),
|
||||||
|
("request_throughput", [r"Request throughput \(req/s\):\s+([\d.]+)"]),
|
||||||
|
("input_token_throughput", [r"Input token throughput \(tok/s\):\s+([\d.]+)"]),
|
||||||
|
("output_token_throughput", [r"Output token throughput \(tok/s\):\s+([\d.]+)"]),
|
||||||
|
("total_token_throughput", [r"Total token throughput \(tok/s\):\s+([\d.]+)"]),
|
||||||
|
("ttft_mean", [r"Mean TTFT \(ms\):\s+([\d.]+)"]),
|
||||||
|
("ttft_p50", [r"Median TTFT \(ms\):\s+([\d.]+)", r"P50 TTFT \(ms\):\s+([\d.]+)"]),
|
||||||
|
("ttft_p90", [r"P90 TTFT \(ms\):\s+([\d.]+)"]),
|
||||||
|
("ttft_p99", [r"P99 TTFT \(ms\):\s+([\d.]+)"]),
|
||||||
|
("tpot_mean", [r"Mean TPOT \(ms\):\s+([\d.]+)"]),
|
||||||
|
("tpot_p50", [r"Median TPOT \(ms\):\s+([\d.]+)", r"P50 TPOT \(ms\):\s+([\d.]+)"]),
|
||||||
|
("tpot_p90", [r"P90 TPOT \(ms\):\s+([\d.]+)"]),
|
||||||
|
("tpot_p99", [r"P99 TPOT \(ms\):\s+([\d.]+)"]),
|
||||||
|
("e2e_mean", [r"Mean E2E \(ms\):\s+([\d.]+)"]),
|
||||||
|
("e2e_p50", [r"Median E2E \(ms\):\s+([\d.]+)", r"P50 E2E \(ms\):\s+([\d.]+)"]),
|
||||||
|
("e2e_p90", [r"P90 E2E \(ms\):\s+([\d.]+)"]),
|
||||||
|
("e2e_p99", [r"P99 E2E \(ms\):\s+([\d.]+)"]),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def find_float(text: str, patterns: list[str]) -> float | None:
|
||||||
|
for pat in patterns:
|
||||||
|
m = re.search(pat, text)
|
||||||
|
if m:
|
||||||
|
try:
|
||||||
|
return float(m.group(1))
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def parse_summary_log(log_text: str) -> dict:
|
||||||
|
result = {}
|
||||||
|
for key, patterns in METRIC_PATTERNS.items():
|
||||||
|
result[key] = find_float(log_text, patterns)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def percentile(values: list[float], p: float) -> float:
|
||||||
|
if not values:
|
||||||
|
return 0.0
|
||||||
|
sorted_values = sorted(values)
|
||||||
|
k = (len(sorted_values) - 1) * p / 100.0
|
||||||
|
f = int(k)
|
||||||
|
c = min(f + 1, len(sorted_values) - 1)
|
||||||
|
if f == c:
|
||||||
|
return sorted_values[f]
|
||||||
|
return sorted_values[f] * (c - k) + sorted_values[c] * (k - f)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_jsonl(jsonl_path: Path) -> tuple[list[dict], dict]:
|
||||||
|
"""Parse sglang.bench_serving --output-file output.
|
||||||
|
|
||||||
|
Newer sglang writes one aggregate JSON object. Older versions write one
|
||||||
|
JSON object per request. Returns (raw_requests, aggregates).
|
||||||
|
"""
|
||||||
|
text = jsonl_path.read_text(encoding="utf-8", errors="replace").strip()
|
||||||
|
if not text:
|
||||||
|
return [], {}
|
||||||
|
|
||||||
|
# Try single aggregate JSON object first.
|
||||||
|
try:
|
||||||
|
data = json.loads(text)
|
||||||
|
if isinstance(data, dict) and "mean_e2e_latency_ms" in data:
|
||||||
|
aggregates = {
|
||||||
|
"success": data.get("total_output_tokens", 0) > 0 and 1 or 0,
|
||||||
|
"failed": 0,
|
||||||
|
"input_tokens": data.get("total_input_tokens", 0),
|
||||||
|
"output_tokens": data.get("total_output_tokens", 0),
|
||||||
|
"latencies": {
|
||||||
|
"e2e_ms": {
|
||||||
|
"mean": data.get("mean_e2e_latency_ms"),
|
||||||
|
"p50": data.get("median_e2e_latency_ms"),
|
||||||
|
"p90": data.get("p90_e2e_latency_ms"),
|
||||||
|
"p95": None,
|
||||||
|
"p99": data.get("p99_e2e_latency_ms"),
|
||||||
|
},
|
||||||
|
"ttft_ms": {
|
||||||
|
"mean": data.get("mean_ttft_ms"),
|
||||||
|
"p50": data.get("median_ttft_ms"),
|
||||||
|
"p90": None,
|
||||||
|
"p95": None,
|
||||||
|
"p99": data.get("p99_ttft_ms"),
|
||||||
|
},
|
||||||
|
"tpot_ms": {
|
||||||
|
"mean": data.get("mean_tpot_ms"),
|
||||||
|
"p50": data.get("median_tpot_ms"),
|
||||||
|
"p90": None,
|
||||||
|
"p95": None,
|
||||||
|
"p99": data.get("p99_tpot_ms"),
|
||||||
|
},
|
||||||
|
"itl_ms": {
|
||||||
|
"mean": data.get("mean_itl_ms"),
|
||||||
|
"p50": data.get("median_itl_ms"),
|
||||||
|
"p90": None,
|
||||||
|
"p95": data.get("p95_itl_ms"),
|
||||||
|
"p99": data.get("p99_itl_ms"),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return [data], aggregates
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Fall back to JSONL per-request parsing.
|
||||||
|
raw_requests = []
|
||||||
|
ttfts = []
|
||||||
|
tpots = []
|
||||||
|
itls = []
|
||||||
|
e2es = []
|
||||||
|
input_tokens = []
|
||||||
|
output_tokens = []
|
||||||
|
success = 0
|
||||||
|
failed = 0
|
||||||
|
|
||||||
|
for line in text.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
req = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue
|
||||||
|
|
||||||
|
raw_requests.append(req)
|
||||||
|
|
||||||
|
ttft = req.get("ttft") or req.get("ttft_ms") or 0
|
||||||
|
tpot = req.get("tpot") or req.get("tpot_ms") or 0
|
||||||
|
itl = req.get("inter_token_latency") or req.get("itl") or req.get("itl_ms") or 0
|
||||||
|
e2e = req.get("e2e_latency") or req.get("e2e") or req.get("e2e_ms") or 0
|
||||||
|
in_tok = req.get("input_tokens") or req.get("prompt_tokens") or 0
|
||||||
|
out_tok = req.get("output_tokens") or req.get("completion_tokens") or 0
|
||||||
|
|
||||||
|
if ttft:
|
||||||
|
ttfts.append(float(ttft))
|
||||||
|
if tpot:
|
||||||
|
tpots.append(float(tpot))
|
||||||
|
if itl:
|
||||||
|
itls.append(float(itl))
|
||||||
|
if e2e:
|
||||||
|
e2es.append(float(e2e))
|
||||||
|
if in_tok:
|
||||||
|
input_tokens.append(int(in_tok))
|
||||||
|
if out_tok:
|
||||||
|
output_tokens.append(int(out_tok))
|
||||||
|
|
||||||
|
if req.get("success", True):
|
||||||
|
success += 1
|
||||||
|
else:
|
||||||
|
failed += 1
|
||||||
|
|
||||||
|
def latency_bucket(values: list[float]) -> dict:
|
||||||
|
if not values:
|
||||||
|
return {"mean": None, "p50": None, "p90": None, "p95": None, "p99": None}
|
||||||
|
return {
|
||||||
|
"mean": round(sum(values) / len(values), 2),
|
||||||
|
"p50": round(percentile(values, 50), 2),
|
||||||
|
"p90": round(percentile(values, 90), 2),
|
||||||
|
"p95": round(percentile(values, 95), 2),
|
||||||
|
"p99": round(percentile(values, 99), 2),
|
||||||
|
}
|
||||||
|
|
||||||
|
aggregates = {
|
||||||
|
"success": success,
|
||||||
|
"failed": failed,
|
||||||
|
"input_tokens": sum(input_tokens),
|
||||||
|
"output_tokens": sum(output_tokens),
|
||||||
|
"latencies": {
|
||||||
|
"e2e_ms": latency_bucket(e2es),
|
||||||
|
"ttft_ms": latency_bucket(ttfts),
|
||||||
|
"tpot_ms": latency_bucket(tpots),
|
||||||
|
"itl_ms": latency_bucket(itls),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
return raw_requests, aggregates
|
||||||
|
|
||||||
|
|
||||||
|
def parse_scenario(result_root: Path, raw_file: Path) -> dict | None:
|
||||||
|
"""Parse one raw output file into a scenario dict."""
|
||||||
|
# Filename: {chip}_{engine}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
|
||||||
|
parts = raw_file.stem.split("_")
|
||||||
|
if len(parts) < 6:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
concurrency = int(parts[-3])
|
||||||
|
input_len = int(parts[-2])
|
||||||
|
output_len = int(parts[-1])
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
log_file = result_root / "logs" / f"sglang_c{concurrency}_i{input_len}_o{output_len}.log"
|
||||||
|
summary = {}
|
||||||
|
if log_file.exists():
|
||||||
|
summary = parse_summary_log(log_file.read_text(encoding="utf-8", errors="replace"))
|
||||||
|
|
||||||
|
raw_requests, aggregates = parse_jsonl(raw_file)
|
||||||
|
|
||||||
|
# For newer sglang aggregate JSON, success/failed are not present in the
|
||||||
|
# raw output file. Override them from the human-readable summary log when
|
||||||
|
# it is available.
|
||||||
|
if summary.get("successful_requests") is not None:
|
||||||
|
aggregates["success"] = int(summary["successful_requests"])
|
||||||
|
# The summary log only reports successes; assume failures are zero
|
||||||
|
# unless the aggregate JSON already provided a non-zero failed count.
|
||||||
|
if not aggregates.get("failed"):
|
||||||
|
aggregates["failed"] = 0
|
||||||
|
|
||||||
|
scenario = {
|
||||||
|
"name": f"c{concurrency}_i{input_len}_o{output_len}",
|
||||||
|
"concurrency": concurrency,
|
||||||
|
"input_len": input_len,
|
||||||
|
"output_len": output_len,
|
||||||
|
"success": aggregates["success"],
|
||||||
|
"failed": aggregates["failed"],
|
||||||
|
"duration_s": summary.get("benchmark_duration_s"),
|
||||||
|
"request_throughput": summary.get("request_throughput"),
|
||||||
|
"input_token_throughput": summary.get("input_token_throughput"),
|
||||||
|
"output_token_throughput": summary.get("output_token_throughput"),
|
||||||
|
"total_token_throughput": summary.get("total_token_throughput"),
|
||||||
|
"accept_length": None,
|
||||||
|
"latencies": aggregates["latencies"],
|
||||||
|
"raw_requests": raw_requests[:100] if len(raw_requests) <= 100 else None,
|
||||||
|
}
|
||||||
|
|
||||||
|
return scenario
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("results")
|
||||||
|
raw_dir = result_root / "raw_outputs"
|
||||||
|
json_path = result_root / "results.json"
|
||||||
|
report_path = result_root / "report.md"
|
||||||
|
|
||||||
|
if not json_path.exists():
|
||||||
|
raise SystemExit(f"metadata results.json not found: {json_path}")
|
||||||
|
|
||||||
|
with open(json_path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
|
||||||
|
scenarios = []
|
||||||
|
if raw_dir.exists():
|
||||||
|
for raw_file in sorted(raw_dir.glob("*.jsonl")):
|
||||||
|
scenario = parse_scenario(result_root, raw_file)
|
||||||
|
if scenario:
|
||||||
|
scenarios.append(scenario)
|
||||||
|
|
||||||
|
data["scenarios"] = scenarios
|
||||||
|
|
||||||
|
with open(json_path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||||
|
|
||||||
|
# Generate report.md
|
||||||
|
with open(report_path, "w", encoding="utf-8") as f:
|
||||||
|
meta = data["metadata"]
|
||||||
|
f.write(f"# Benchmark Report: {meta['experiment']}\n\n")
|
||||||
|
f.write("## Metadata\n\n")
|
||||||
|
f.write(f"- **Run ID**: {meta['run_id']}\n")
|
||||||
|
f.write(f"- **Timestamp**: {meta['timestamp']}\n")
|
||||||
|
f.write(f"- **Chip/Accelerator**: {meta.get('accelerator', '')} / {meta.get('chip', '')}\n")
|
||||||
|
f.write(f"- **Engine/Backend**: {meta.get('engine', '')} / {meta.get('backend', '')}\n")
|
||||||
|
f.write(f"- **Hardware**: {meta.get('hardware', '')}\n")
|
||||||
|
f.write(f"- **Model**: {meta['model']}\n")
|
||||||
|
f.write(f"- **Git Commit**: {meta.get('git_commit', 'unknown')}\n")
|
||||||
|
f.write("\n## Results\n\n")
|
||||||
|
f.write(
|
||||||
|
"| Scenario | Conc | In/Out | Success | Failed | Req/s | OutTok/s | "
|
||||||
|
"TTFT p50 | TTFT p99 | TPOT p50 | TPOT p99 | E2E p99 |\n"
|
||||||
|
)
|
||||||
|
f.write(
|
||||||
|
"|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n"
|
||||||
|
)
|
||||||
|
for s in scenarios:
|
||||||
|
lat = s["latencies"]
|
||||||
|
f.write(
|
||||||
|
f"| {s['name']} "
|
||||||
|
f"| {s['concurrency']} "
|
||||||
|
f"| {s['input_len']}/{s['output_len']} "
|
||||||
|
f"| {s['success']} "
|
||||||
|
f"| {s['failed']} "
|
||||||
|
f"| {s.get('request_throughput') or ''} "
|
||||||
|
f"| {s.get('output_token_throughput') or ''} "
|
||||||
|
f"| {lat['ttft_ms']['p50'] or ''} "
|
||||||
|
f"| {lat['ttft_ms']['p99'] or ''} "
|
||||||
|
f"| {lat['tpot_ms']['p50'] or ''} "
|
||||||
|
f"| {lat['tpot_ms']['p99'] or ''} "
|
||||||
|
f"| {lat['e2e_ms']['p99'] or ''} |\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Updated {json_path}")
|
||||||
|
print(f"Wrote {report_path}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
112
experiments/dsv4_p800_sglang/run_bench.sh
Executable file
112
experiments/dsv4_p800_sglang/run_bench.sh
Executable file
@ -0,0 +1,112 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/server_docker.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/bench_client_docker.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-${SCRIPT_DIR}/results/${RUN_ID}}"
|
||||||
|
RAW_DIR="${RESULT_ROOT}/raw_outputs"
|
||||||
|
LOG_DIR="${RESULT_ROOT}/logs"
|
||||||
|
|
||||||
|
ensure_result_root "$RESULT_ROOT"
|
||||||
|
log_init "${LOG_DIR}/orchestrator.log"
|
||||||
|
|
||||||
|
log "experiment=${EXPERIMENT_NAME}"
|
||||||
|
log "run_id=${RUN_ID}"
|
||||||
|
log "result_root=${RESULT_ROOT}"
|
||||||
|
log "platform=${PLATFORM}"
|
||||||
|
log "chip=${CHIP}"
|
||||||
|
log "accelerator=${ACCELERATOR}"
|
||||||
|
log "engine=${ENGINE}"
|
||||||
|
log "hardware=${HARDWARE}"
|
||||||
|
|
||||||
|
# Write initial metadata for this run.
|
||||||
|
METADATA_JSON="${RESULT_ROOT}/results.json"
|
||||||
|
write_metadata_json \
|
||||||
|
"$METADATA_JSON" \
|
||||||
|
"$EXPERIMENT_NAME" \
|
||||||
|
"$RUN_ID" \
|
||||||
|
"$MODEL_PATH" \
|
||||||
|
"$BACKEND" \
|
||||||
|
"$ENGINE" \
|
||||||
|
"$HARDWARE" \
|
||||||
|
"$ACCELERATOR" \
|
||||||
|
"$CHIP" \
|
||||||
|
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||||
|
"" \
|
||||||
|
"Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
|
||||||
|
# Start server unless SKIP_MANAGE_SERVER is set.
|
||||||
|
if [[ -n "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||||
|
log "SKIP_MANAGE_SERVER is set, assuming server is already running on port ${PORT}"
|
||||||
|
if ! health_check 127.0.0.1 "${PORT}" 30; then
|
||||||
|
log "ERROR: no healthy server found at port ${PORT}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
docker_server_start "${MODEL_PATH}" "${PORT}" "${LOG_DIR}/server.outer.log" "${SERVER_MODE}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
on_exit() {
|
||||||
|
local code=$?
|
||||||
|
if [[ -z "${SKIP_MANAGE_SERVER:-}" ]]; then
|
||||||
|
docker_server_stop
|
||||||
|
fi
|
||||||
|
log "orchestrator exiting with code=${code}"
|
||||||
|
exit "$code"
|
||||||
|
}
|
||||||
|
trap on_exit EXIT
|
||||||
|
|
||||||
|
log "===== BENCHMARK START ====="
|
||||||
|
|
||||||
|
for scenario in "${SCENARIOS[@]}"; do
|
||||||
|
# Allow SCENARIOS to be passed as a string like "(32 512 256)" from env.
|
||||||
|
scenario="${scenario//[()]/}"
|
||||||
|
read -r concurrency input_len output_len <<< "$scenario"
|
||||||
|
|
||||||
|
tmp_output_file="/tmp/bench_outputs/${CHIP}_${ENGINE}_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||||
|
host_output_file="${RAW_DIR}/${CHIP}_${ENGINE}_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
|
||||||
|
detail_log="${LOG_DIR}/${BACKEND}_c${concurrency}_i${input_len}_o${output_len}.log"
|
||||||
|
|
||||||
|
log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len}"
|
||||||
|
|
||||||
|
run_random_case \
|
||||||
|
"$BACKEND" \
|
||||||
|
"$PORT" \
|
||||||
|
"$MODEL_PATH" \
|
||||||
|
"$tmp_output_file" \
|
||||||
|
"$concurrency" \
|
||||||
|
"$input_len" \
|
||||||
|
"$output_len" \
|
||||||
|
"$NUM_PROMPTS" \
|
||||||
|
"${DATASET_PATH:-}" \
|
||||||
|
> "$detail_log" 2>&1 || {
|
||||||
|
log "ERROR: scenario c=${concurrency} i=${input_len} o=${output_len} failed"
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
# Copy the JSONL from the container to the host result directory.
|
||||||
|
docker cp "${CONTAINER_NAME}:${tmp_output_file}" "$host_output_file" || {
|
||||||
|
log "ERROR: failed to copy ${tmp_output_file} from container"
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
log "finished scenario: output=${host_output_file}"
|
||||||
|
done
|
||||||
|
|
||||||
|
log "===== BENCHMARK DONE ====="
|
||||||
|
log "parsing results..."
|
||||||
|
"${SCRIPT_DIR}/parse_results.py" "$RESULT_ROOT"
|
||||||
|
|
||||||
|
log "all results saved to ${RESULT_ROOT}"
|
||||||
17
experiments/dsv4_p800_sglang/start_server.sh
Executable file
17
experiments/dsv4_p800_sglang/start_server.sh
Executable file
@ -0,0 +1,17 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/server_docker.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
|
||||||
|
log_init "${RESULT_ROOT:-/tmp/${EXPERIMENT_NAME}}/logs/start_server.log"
|
||||||
|
log "starting server for ${EXPERIMENT_NAME}"
|
||||||
|
log "model: ${MODEL_PATH}"
|
||||||
|
log "port: ${PORT}"
|
||||||
|
|
||||||
|
docker_server_start "${MODEL_PATH}" "${PORT}" "${LOG_DIR}/server.outer.log" "${SERVER_MODE}"
|
||||||
36
platforms/README.md
Normal file
36
platforms/README.md
Normal file
@ -0,0 +1,36 @@
|
|||||||
|
# Platform Configurations
|
||||||
|
|
||||||
|
Each `.env` file in this directory describes one accelerator platform.
|
||||||
|
They are meant to be sourced by benchmark scripts through
|
||||||
|
`scripts/common/platform.sh`, not executed directly.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Default platform for the current machine
|
||||||
|
bash experiments/dsv4_p800_sglang/run_bench.sh
|
||||||
|
|
||||||
|
# Explicitly select a platform
|
||||||
|
PLATFORM=kunlun_p800 bash experiments/dsv4_p800_sglang/run_bench.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current platforms
|
||||||
|
|
||||||
|
| File | Chip/Accelerator | Engine | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `kunlun_p800.env` | Kunlun P800 XPU | `sglang-xpu` | Docker-based SGLang serving image |
|
||||||
|
| `nvidia_h200.env` | NVIDIA H200 | native vllm/sglang | Host virtual environments |
|
||||||
|
|
||||||
|
## What belongs here
|
||||||
|
|
||||||
|
- Chip/accelerator identity (`CHIP`, `ACCELERATOR`, `HARDWARE`).
|
||||||
|
- Device selection environment variables.
|
||||||
|
- Platform-wide paths that rarely change (model root, default port).
|
||||||
|
- Container image / interpreter paths for Docker-based platforms.
|
||||||
|
|
||||||
|
## What does NOT belong here
|
||||||
|
|
||||||
|
- Specific model names or experiment scenarios — those go in
|
||||||
|
`experiments/<name>/config.env`.
|
||||||
|
- Engine-specific launch flags — those go in the experiment's
|
||||||
|
`start_server.sh` or `run_bench.sh`.
|
||||||
27
platforms/kunlun_p800.env
Normal file
27
platforms/kunlun_p800.env
Normal file
@ -0,0 +1,27 @@
|
|||||||
|
# Platform configuration for Kunlun P800 XPU
|
||||||
|
# Source this file via scripts/common/platform.sh
|
||||||
|
|
||||||
|
CHIP="kunlun_p800"
|
||||||
|
ACCELERATOR="Kunlun P800 XPU"
|
||||||
|
HARDWARE="8x Kunlun P800 XPU"
|
||||||
|
ENGINE="sglang-xpu"
|
||||||
|
|
||||||
|
# Container runtime
|
||||||
|
DOCKER_IMAGE="iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202"
|
||||||
|
CONTAINER_NAME="${CONTAINER_NAME:-sglang-dsv4-flash}"
|
||||||
|
|
||||||
|
# Device selection
|
||||||
|
DEVICE_SELECT_ENV="XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||||
|
CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||||
|
CUDA_DEVICE_ORDER="OAM_ID"
|
||||||
|
|
||||||
|
# Default serving port and model root on the host
|
||||||
|
DEFAULT_PORT="30000"
|
||||||
|
MODEL_ROOT="/data1/models"
|
||||||
|
|
||||||
|
# Python interpreter inside the container
|
||||||
|
CONTAINER_PYTHON="/root/miniconda/envs/python310_torch25_cuda/bin/python"
|
||||||
|
|
||||||
|
# Host mount points for patches required by the P800 SGLang image
|
||||||
|
# These are baked into the container start command in server_docker.sh.
|
||||||
|
PATCH_ROOT="${PATCH_ROOT:-/data1/yy/sskj/platforms/patches/kunlun_p800}"
|
||||||
18
platforms/nvidia_h200.env
Normal file
18
platforms/nvidia_h200.env
Normal file
@ -0,0 +1,18 @@
|
|||||||
|
# Platform configuration for NVIDIA H200
|
||||||
|
# Source this file via scripts/common/platform.sh
|
||||||
|
|
||||||
|
CHIP="nvidia_h200"
|
||||||
|
ACCELERATOR="NVIDIA H200"
|
||||||
|
HARDWARE="8x NVIDIA H200 143GB"
|
||||||
|
|
||||||
|
# Device selection
|
||||||
|
DEVICE_SELECT_ENV="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||||
|
CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||||
|
|
||||||
|
# Default serving port and model root on the host
|
||||||
|
DEFAULT_PORT="30004"
|
||||||
|
MODEL_ROOT="/data/models"
|
||||||
|
|
||||||
|
# Virtual environments on the host (used by native H200 scripts)
|
||||||
|
VENV_VLLM_DSPARK="/data/user1/yy/envs/vllm-dspark"
|
||||||
|
VENV_SGLANG="/data/user1/yy/envs/sglang"
|
||||||
114
platforms/patches/kunlun_p800/config_backup_small_fp8.json
Normal file
114
platforms/patches/kunlun_p800/config_backup_small_fp8.json
Normal file
@ -0,0 +1,114 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"DeepseekXYZForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 1,
|
||||||
|
"hc_eps": 1e-06,
|
||||||
|
"hc_mult": 4,
|
||||||
|
"hc_sinkhorn_iters": 20,
|
||||||
|
"head_dim": 512,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"index_head_dim": 128,
|
||||||
|
"index_n_heads": 64,
|
||||||
|
"index_topk": 512,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"max_position_embeddings": 1048576,
|
||||||
|
"model_type": "deepseek_ref",
|
||||||
|
"moe_intermediate_size": 2048,
|
||||||
|
"n_routed_experts": 256,
|
||||||
|
"n_shared_experts": 1,
|
||||||
|
"norm_topk_prob": true,
|
||||||
|
"num_attention_heads": 64,
|
||||||
|
"num_experts_per_tok": 6,
|
||||||
|
"num_hidden_layers": 43,
|
||||||
|
"num_hash_layers": 3,
|
||||||
|
"num_key_value_heads": 1,
|
||||||
|
"num_nextn_predict_layers": 1,
|
||||||
|
"o_groups": 8,
|
||||||
|
"o_lora_rank": 1024,
|
||||||
|
"q_lora_rank": 1024,
|
||||||
|
"qk_rope_head_dim": 64,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": {
|
||||||
|
"beta_fast": 32.0,
|
||||||
|
"beta_slow": 1.0,
|
||||||
|
"factor": 16.0,
|
||||||
|
"original_max_position_embeddings": 65536,
|
||||||
|
"type": "yarn"
|
||||||
|
},
|
||||||
|
"rope_theta": 10000,
|
||||||
|
"routed_scaling_factor": 1.5,
|
||||||
|
"scoring_func": "sqrtsoftplus",
|
||||||
|
"sliding_window": 128,
|
||||||
|
"swiglu_limit": 10.0,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"n_group": 8,
|
||||||
|
"topk_group": 8,
|
||||||
|
"topk_method": "noaux_tc",
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.57.1",
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 129280,
|
||||||
|
"compress_rope_theta": 160000,
|
||||||
|
"compress_ratios": [
|
||||||
|
0,
|
||||||
|
0,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
128,
|
||||||
|
4,
|
||||||
|
0
|
||||||
|
],
|
||||||
|
"quantization_config": {
|
||||||
|
"activation_scheme": "dynamic",
|
||||||
|
"fmt": "e4m3",
|
||||||
|
"quant_method": "fp8",
|
||||||
|
"scale_fmt": "ue8m0",
|
||||||
|
"weight_block_size": [
|
||||||
|
128,
|
||||||
|
128
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"expert_dtype": "fp4"
|
||||||
|
}
|
||||||
@ -0,0 +1,58 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"DeepseekXYZForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_bias": false,
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 1,
|
||||||
|
"hc_eps": 1e-06,
|
||||||
|
"hc_mult": 4,
|
||||||
|
"hc_sinkhorn_iters": 20,
|
||||||
|
"head_dim": 512,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 4096,
|
||||||
|
"index_head_dim": 128,
|
||||||
|
"index_n_heads": 64,
|
||||||
|
"index_topk": 512,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"max_position_embeddings": 1048576,
|
||||||
|
"model_type": "deepseek_ref",
|
||||||
|
"moe_intermediate_size": 2048,
|
||||||
|
"n_routed_experts": 256,
|
||||||
|
"n_shared_experts": 1,
|
||||||
|
"norm_topk_prob": true,
|
||||||
|
"num_attention_heads": 64,
|
||||||
|
"num_experts_per_tok": 6,
|
||||||
|
"num_hidden_layers": 43,
|
||||||
|
"num_hash_layers": 3,
|
||||||
|
"num_key_value_heads": 1,
|
||||||
|
"num_nextn_predict_layers": 1,
|
||||||
|
"o_groups": 8,
|
||||||
|
"o_lora_rank": 1024,
|
||||||
|
"q_lora_rank": 1024,
|
||||||
|
"qk_rope_head_dim": 64,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_scaling": {
|
||||||
|
"beta_fast": 32,
|
||||||
|
"beta_slow": 1,
|
||||||
|
"factor": 16,
|
||||||
|
"original_max_position_embeddings": 65536,
|
||||||
|
"type": "yarn"
|
||||||
|
},
|
||||||
|
"rope_theta": 10000,
|
||||||
|
"routed_scaling_factor": 1.5,
|
||||||
|
"scoring_func": "sqrtsoftplus",
|
||||||
|
"sliding_window": 128,
|
||||||
|
"swiglu_limit": 10.0,
|
||||||
|
"tie_word_embeddings": false,
|
||||||
|
"n_group": 8,
|
||||||
|
"topk_group": 8,
|
||||||
|
"topk_method": "noaux_tc",
|
||||||
|
"torch_dtype": "bfloat16",
|
||||||
|
"transformers_version": "4.57.1",
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 129280,
|
||||||
|
"compress_rope_theta": 160000,
|
||||||
|
"compress_ratios": [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 0]
|
||||||
|
}
|
||||||
1689
platforms/patches/kunlun_p800/fp8_utils.py.patched
Executable file
1689
platforms/patches/kunlun_p800/fp8_utils.py.patched
Executable file
File diff suppressed because it is too large
Load Diff
1451
platforms/patches/kunlun_p800/hf_transformers_utils.py.patched
Normal file
1451
platforms/patches/kunlun_p800/hf_transformers_utils.py.patched
Normal file
File diff suppressed because it is too large
Load Diff
98
platforms/patches/kunlun_p800/nic_priority_matrix_test.json
Normal file
98
platforms/patches/kunlun_p800/nic_priority_matrix_test.json
Normal file
@ -0,0 +1,98 @@
|
|||||||
|
{
|
||||||
|
"cpu:0": [
|
||||||
|
[
|
||||||
|
"mlx5_0"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cpu:1": [
|
||||||
|
[
|
||||||
|
"mlx5_0"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cpu:2": [
|
||||||
|
[
|
||||||
|
"mlx5_0"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cpu:3": [
|
||||||
|
[
|
||||||
|
"mlx5_0"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cpu:4": [
|
||||||
|
[
|
||||||
|
"mlx5_0"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cpu:5": [
|
||||||
|
[
|
||||||
|
"mlx5_0"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cpu:6": [
|
||||||
|
[
|
||||||
|
"mlx5_0"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cpu:7": [
|
||||||
|
[
|
||||||
|
"mlx5_0"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cuda:0": [
|
||||||
|
[
|
||||||
|
"mlx5_1"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cuda:1": [
|
||||||
|
[
|
||||||
|
"mlx5_1"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cuda:2": [
|
||||||
|
[
|
||||||
|
"mlx5_2"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cuda:3": [
|
||||||
|
[
|
||||||
|
"mlx5_2"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cuda:4": [
|
||||||
|
[
|
||||||
|
"mlx5_3"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cuda:5": [
|
||||||
|
[
|
||||||
|
"mlx5_3"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cuda:6": [
|
||||||
|
[
|
||||||
|
"mlx5_4"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
],
|
||||||
|
"cuda:7": [
|
||||||
|
[
|
||||||
|
"mlx5_4"
|
||||||
|
],
|
||||||
|
[]
|
||||||
|
]
|
||||||
|
}
|
||||||
2384
platforms/patches/kunlun_p800/parallel_state.py
Normal file
2384
platforms/patches/kunlun_p800/parallel_state.py
Normal file
File diff suppressed because it is too large
Load Diff
7
platforms/patches/kunlun_p800/sitecustomize_xpu.py
Normal file
7
platforms/patches/kunlun_p800/sitecustomize_xpu.py
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
import torch
|
||||||
|
# DeepSeek-V4-Flash safetensors contain F8_E8M0FN dtype metadata, but
|
||||||
|
# PyTorch 2.5.1 does not define this dtype. Alias it to uint8 so that
|
||||||
|
# safetensors deserialization can load the scales; sglang reinterprets
|
||||||
|
# them on P800 via its own FP8 kernels.
|
||||||
|
if not hasattr(torch, "float8_e8m0fnu"):
|
||||||
|
torch.float8_e8m0fnu = torch.uint8
|
||||||
680
platforms/patches/kunlun_p800/vocab_parallel_embedding.py
Normal file
680
platforms/patches/kunlun_p800/vocab_parallel_embedding.py
Normal file
@ -0,0 +1,680 @@
|
|||||||
|
# Adapted from https://github.com/vllm-project/vllm/blob/v0.6.3.post1/vllm/model_executor/layers/vocab_parallel_embedding.py
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import List, Optional, Sequence, Tuple
|
||||||
|
|
||||||
|
import torch
|
||||||
|
import torch.nn.functional as F
|
||||||
|
from torch.nn.parameter import Parameter, UninitializedParameter
|
||||||
|
|
||||||
|
from sglang.srt.distributed import (
|
||||||
|
divide,
|
||||||
|
get_tensor_model_parallel_rank,
|
||||||
|
get_tensor_model_parallel_world_size,
|
||||||
|
get_tp_group,
|
||||||
|
tensor_model_parallel_all_gather,
|
||||||
|
tensor_model_parallel_all_reduce,
|
||||||
|
)
|
||||||
|
from sglang.srt.distributed.device_communicators.pynccl_allocator import (
|
||||||
|
use_symmetric_memory,
|
||||||
|
)
|
||||||
|
from sglang.srt.layers.amx_utils import PackWeightMethod
|
||||||
|
from sglang.srt.layers.communicator import get_attn_tp_context
|
||||||
|
from sglang.srt.layers.dp_attention import (
|
||||||
|
attn_tp_all_reduce,
|
||||||
|
get_attention_tp_rank,
|
||||||
|
get_attention_tp_size,
|
||||||
|
is_allocation_symmetric,
|
||||||
|
)
|
||||||
|
from sglang.srt.layers.parameter import BasevLLMParameter
|
||||||
|
from sglang.srt.layers.quantization.base_config import (
|
||||||
|
QuantizationConfig,
|
||||||
|
QuantizeMethodBase,
|
||||||
|
method_has_implemented_embedding,
|
||||||
|
)
|
||||||
|
from sglang.srt.layers.quantization.unquant import UnquantizedEmbeddingMethod
|
||||||
|
from sglang.srt.utils import (
|
||||||
|
cpu_has_amx_support,
|
||||||
|
is_cpu,
|
||||||
|
is_npu,
|
||||||
|
set_weight_attrs,
|
||||||
|
)
|
||||||
|
|
||||||
|
DEFAULT_VOCAB_PADDING_SIZE = 64
|
||||||
|
|
||||||
|
_is_cpu_amx_available = cpu_has_amx_support()
|
||||||
|
_is_cpu = is_cpu()
|
||||||
|
_is_npu = is_npu()
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def pad_vocab_size(vocab_size: int, pad_to: int = DEFAULT_VOCAB_PADDING_SIZE) -> int:
|
||||||
|
"""Pad the vocab size to the given value."""
|
||||||
|
return ((vocab_size + pad_to - 1) // pad_to) * pad_to
|
||||||
|
|
||||||
|
|
||||||
|
def vocab_range_from_per_partition_vocab_size(
|
||||||
|
per_partition_vocab_size: int, rank: int, offset: int = 0
|
||||||
|
) -> Sequence[int]:
|
||||||
|
index_f = rank * per_partition_vocab_size
|
||||||
|
index_l = index_f + per_partition_vocab_size
|
||||||
|
return index_f + offset, index_l + offset
|
||||||
|
|
||||||
|
|
||||||
|
def vocab_range_from_global_vocab_size(
|
||||||
|
global_vocab_size: int, rank: int, world_size: int, offset: int = 0
|
||||||
|
) -> Sequence[int]:
|
||||||
|
per_partition_vocab_size = divide(global_vocab_size, world_size)
|
||||||
|
return vocab_range_from_per_partition_vocab_size(
|
||||||
|
per_partition_vocab_size, rank, offset=offset
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class VocabParallelEmbeddingShardIndices:
|
||||||
|
"""Indices for a shard of a vocab parallel embedding."""
|
||||||
|
|
||||||
|
padded_org_vocab_start_index: int
|
||||||
|
padded_org_vocab_end_index: int
|
||||||
|
padded_added_vocab_start_index: int
|
||||||
|
padded_added_vocab_end_index: int
|
||||||
|
|
||||||
|
org_vocab_start_index: int
|
||||||
|
org_vocab_end_index: int
|
||||||
|
added_vocab_start_index: int
|
||||||
|
added_vocab_end_index: int
|
||||||
|
|
||||||
|
@property
|
||||||
|
def num_org_elements(self) -> int:
|
||||||
|
return self.org_vocab_end_index - self.org_vocab_start_index
|
||||||
|
|
||||||
|
@property
|
||||||
|
def num_added_elements(self) -> int:
|
||||||
|
return self.added_vocab_end_index - self.added_vocab_start_index
|
||||||
|
|
||||||
|
@property
|
||||||
|
def num_org_elements_padded(self) -> int:
|
||||||
|
return self.padded_org_vocab_end_index - self.padded_org_vocab_start_index
|
||||||
|
|
||||||
|
@property
|
||||||
|
def num_added_elements_padded(self) -> int:
|
||||||
|
return self.padded_added_vocab_end_index - self.padded_added_vocab_start_index
|
||||||
|
|
||||||
|
@property
|
||||||
|
def num_org_vocab_padding(self) -> int:
|
||||||
|
return self.num_org_elements_padded - self.num_org_elements
|
||||||
|
|
||||||
|
@property
|
||||||
|
def num_added_vocab_padding(self) -> int:
|
||||||
|
return self.num_added_elements_padded - self.num_added_elements
|
||||||
|
|
||||||
|
@property
|
||||||
|
def num_elements_padded(self) -> int:
|
||||||
|
return self.num_org_elements_padded + self.num_added_elements_padded
|
||||||
|
|
||||||
|
def __post_init__(self):
|
||||||
|
# sanity checks
|
||||||
|
assert self.padded_org_vocab_start_index <= self.padded_org_vocab_end_index
|
||||||
|
assert self.padded_added_vocab_start_index <= self.padded_added_vocab_end_index
|
||||||
|
|
||||||
|
assert self.org_vocab_start_index <= self.org_vocab_end_index
|
||||||
|
assert self.added_vocab_start_index <= self.added_vocab_end_index
|
||||||
|
|
||||||
|
assert self.org_vocab_start_index <= self.padded_org_vocab_start_index
|
||||||
|
assert self.added_vocab_start_index <= self.padded_added_vocab_start_index
|
||||||
|
assert self.org_vocab_end_index <= self.padded_org_vocab_end_index
|
||||||
|
assert self.added_vocab_end_index <= self.padded_added_vocab_end_index
|
||||||
|
|
||||||
|
assert self.num_org_elements <= self.num_org_elements_padded
|
||||||
|
assert self.num_added_elements <= self.num_added_elements_padded
|
||||||
|
|
||||||
|
|
||||||
|
def get_masked_input_and_mask(
|
||||||
|
input_: torch.Tensor,
|
||||||
|
org_vocab_start_index: int,
|
||||||
|
org_vocab_end_index: int,
|
||||||
|
num_org_vocab_padding: int,
|
||||||
|
added_vocab_start_index: int,
|
||||||
|
added_vocab_end_index: int,
|
||||||
|
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||||
|
# torch.compile will fuse all of the pointwise ops below
|
||||||
|
# into a single kernel, making it very fast
|
||||||
|
org_vocab_mask = (input_ >= org_vocab_start_index) & (input_ < org_vocab_end_index)
|
||||||
|
added_vocab_mask = (input_ >= added_vocab_start_index) & (
|
||||||
|
input_ < added_vocab_end_index
|
||||||
|
)
|
||||||
|
added_offset = (
|
||||||
|
added_vocab_start_index
|
||||||
|
- (org_vocab_end_index - org_vocab_start_index)
|
||||||
|
- num_org_vocab_padding
|
||||||
|
)
|
||||||
|
valid_offset = (org_vocab_start_index * org_vocab_mask) + (
|
||||||
|
added_offset * added_vocab_mask
|
||||||
|
)
|
||||||
|
vocab_mask = org_vocab_mask | added_vocab_mask
|
||||||
|
input_ = vocab_mask * (input_ - valid_offset)
|
||||||
|
return input_, ~vocab_mask
|
||||||
|
|
||||||
|
|
||||||
|
class VocabParallelEmbedding(torch.nn.Module):
|
||||||
|
"""Embedding parallelized in the vocabulary dimension.
|
||||||
|
|
||||||
|
Adapted from torch.nn.Embedding, note that we pad the vocabulary size to
|
||||||
|
make sure it is divisible by the number of model parallel GPUs.
|
||||||
|
|
||||||
|
In order to support various loading methods, we ensure that LoRA-added
|
||||||
|
embeddings are always at the end of TP-sharded tensors. In other words,
|
||||||
|
we shard base embeddings and LoRA embeddings separately (both padded),
|
||||||
|
and place them in the same tensor.
|
||||||
|
In this example, we will have the original vocab size = 1010,
|
||||||
|
added vocab size = 16 and padding to 64. Therefore, the total
|
||||||
|
vocab size with padding will be 1088 (because we first pad 1010 to
|
||||||
|
1024, add 16, and then pad to 1088).
|
||||||
|
Therefore, the tensor format looks like the following:
|
||||||
|
TP1, rank 0 (no sharding):
|
||||||
|
|< --------BASE-------- >|< -BASE PADDING-- >|< -----LORA------ >|< -LORA PADDING-- >|
|
||||||
|
corresponding token_id: | 0 | 1 | ... | 1009 | -1 | ... | -1 | 1010 | ... | 1015 | -1 | ... | -1 |
|
||||||
|
index: | 0 | 1 | ... | 1009 | 1010 | ... | 1023 | 1024 | ... | 1039 | 1040 | ... | 1087 |
|
||||||
|
|
||||||
|
TP2, rank 0:
|
||||||
|
|< --------------------BASE--------------------- >|< -----LORA------ >|< -LORA PADDING- >|
|
||||||
|
corresponding token_id: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 1000 | ... | 1015 | -1 | ... | -1 |
|
||||||
|
index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 527 | 520 | ... | 543 |
|
||||||
|
TP2, rank 1:
|
||||||
|
|< -----------BASE----------- >|< -BASE PADDING- >|< -----------LORA PADDING----------- >|
|
||||||
|
corresponding token_id: | 512 | 513 | 514 | ... | 1009 | -1 | ... | -1 | -1 | ... | -1 | -1 | ... | -1 |
|
||||||
|
index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 519 | 520 | ... | 543 |
|
||||||
|
|
||||||
|
Args:
|
||||||
|
num_embeddings: vocabulary size.
|
||||||
|
embedding_dim: size of hidden state.
|
||||||
|
params_dtype: type of the parameters.
|
||||||
|
org_num_embeddings: original vocabulary size (without LoRA).
|
||||||
|
padding_size: padding size for the vocabulary.
|
||||||
|
quant_config: quant config for the layer
|
||||||
|
prefix: full name of the layer in the state dict
|
||||||
|
""" # noqa: E501
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
num_embeddings: int,
|
||||||
|
embedding_dim: int,
|
||||||
|
*,
|
||||||
|
params_dtype: Optional[torch.dtype] = None,
|
||||||
|
org_num_embeddings: Optional[int] = None,
|
||||||
|
padding_size: int = DEFAULT_VOCAB_PADDING_SIZE,
|
||||||
|
quant_config: Optional[QuantizationConfig] = None,
|
||||||
|
prefix: str = "",
|
||||||
|
enable_tp: bool = True,
|
||||||
|
use_attn_tp_group: bool = False,
|
||||||
|
use_presharded_weights: bool = False,
|
||||||
|
enable_over_encoding: bool = False,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.quant_config = quant_config
|
||||||
|
self.enable_over_encoding = enable_over_encoding
|
||||||
|
|
||||||
|
self.enable_tp = enable_tp
|
||||||
|
self.use_attn_tp_group = use_attn_tp_group
|
||||||
|
if self.enable_tp:
|
||||||
|
if use_attn_tp_group:
|
||||||
|
tp_rank = get_attention_tp_rank()
|
||||||
|
self.tp_size = get_attention_tp_size()
|
||||||
|
else:
|
||||||
|
tp_rank = get_tensor_model_parallel_rank()
|
||||||
|
self.tp_size = get_tensor_model_parallel_world_size()
|
||||||
|
else:
|
||||||
|
assert use_attn_tp_group is False
|
||||||
|
tp_rank = 0
|
||||||
|
self.tp_size = 1
|
||||||
|
|
||||||
|
self.num_embeddings = num_embeddings
|
||||||
|
self.org_vocab_size = org_num_embeddings or num_embeddings
|
||||||
|
|
||||||
|
# Support the case where the vocab size is not divisible by the TP size.
|
||||||
|
if (
|
||||||
|
_is_cpu
|
||||||
|
and pad_vocab_size(self.org_vocab_size, padding_size) % self.tp_size != 0
|
||||||
|
):
|
||||||
|
padding_size *= self.tp_size
|
||||||
|
self.padding_size = padding_size
|
||||||
|
|
||||||
|
num_added_embeddings = num_embeddings - self.org_vocab_size
|
||||||
|
self.use_presharded_weights = use_presharded_weights
|
||||||
|
if use_presharded_weights:
|
||||||
|
assert (
|
||||||
|
num_added_embeddings == 0
|
||||||
|
), "Lora is not supported with presharded weights."
|
||||||
|
|
||||||
|
self.org_vocab_size_padded = pad_vocab_size(
|
||||||
|
self.org_vocab_size, self.padding_size
|
||||||
|
)
|
||||||
|
self.num_embeddings_padded = pad_vocab_size(
|
||||||
|
self.org_vocab_size_padded + num_added_embeddings, self.padding_size
|
||||||
|
)
|
||||||
|
assert self.org_vocab_size_padded <= self.num_embeddings_padded
|
||||||
|
|
||||||
|
self.shard_indices = self._get_indices(
|
||||||
|
self.num_embeddings_padded,
|
||||||
|
self.org_vocab_size_padded,
|
||||||
|
self.num_embeddings,
|
||||||
|
self.org_vocab_size,
|
||||||
|
tp_rank,
|
||||||
|
self.tp_size,
|
||||||
|
)
|
||||||
|
self.embedding_dim = embedding_dim
|
||||||
|
|
||||||
|
quant_method = None
|
||||||
|
if quant_config is not None:
|
||||||
|
quant_method = quant_config.get_quant_method(self, prefix=prefix)
|
||||||
|
if quant_method is None:
|
||||||
|
quant_method = UnquantizedEmbeddingMethod()
|
||||||
|
|
||||||
|
# If we are making an embedding layer, then our quantization linear
|
||||||
|
# method must implement the embedding operation. If we are another
|
||||||
|
# layer type like ParallelLMHead, this is not important.
|
||||||
|
is_embedding_layer = type(self.__class__) is VocabParallelEmbedding
|
||||||
|
quant_method_implements_embedding = method_has_implemented_embedding(
|
||||||
|
type(quant_method)
|
||||||
|
)
|
||||||
|
if is_embedding_layer and not quant_method_implements_embedding:
|
||||||
|
raise NotImplementedError(
|
||||||
|
f"The class {type(quant_method).__name__} must implement "
|
||||||
|
"the 'embedding' method, see UnquantizedEmbeddingMethod."
|
||||||
|
)
|
||||||
|
|
||||||
|
self.quant_method: QuantizeMethodBase = quant_method
|
||||||
|
|
||||||
|
if params_dtype is None:
|
||||||
|
params_dtype = torch.get_default_dtype()
|
||||||
|
# Divide the weight matrix along the vocaburaly dimension.
|
||||||
|
self.num_added_embeddings = self.num_embeddings - self.org_vocab_size
|
||||||
|
self.num_embeddings_per_partition = divide(
|
||||||
|
self.num_embeddings_padded, self.tp_size
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
self.shard_indices.num_elements_padded == self.num_embeddings_per_partition
|
||||||
|
)
|
||||||
|
self.num_org_embeddings_per_partition = (
|
||||||
|
self.shard_indices.org_vocab_end_index
|
||||||
|
- self.shard_indices.org_vocab_start_index
|
||||||
|
)
|
||||||
|
self.num_added_embeddings_per_partition = (
|
||||||
|
self.shard_indices.added_vocab_end_index
|
||||||
|
- self.shard_indices.added_vocab_start_index
|
||||||
|
)
|
||||||
|
|
||||||
|
self.quant_method.create_weights(
|
||||||
|
self,
|
||||||
|
self.embedding_dim,
|
||||||
|
[self.num_embeddings_per_partition],
|
||||||
|
self.embedding_dim,
|
||||||
|
self.num_embeddings_padded,
|
||||||
|
params_dtype=params_dtype,
|
||||||
|
weight_loader=self.weight_loader,
|
||||||
|
host_tensor=(
|
||||||
|
True
|
||||||
|
if (
|
||||||
|
self.__class__ is VocabParallelEmbedding
|
||||||
|
and self.enable_over_encoding
|
||||||
|
)
|
||||||
|
else False
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _get_indices(
|
||||||
|
cls,
|
||||||
|
vocab_size_padded: int,
|
||||||
|
org_vocab_size_padded: int,
|
||||||
|
vocab_size: int,
|
||||||
|
org_vocab_size: int,
|
||||||
|
tp_rank: int,
|
||||||
|
tp_size: int,
|
||||||
|
) -> VocabParallelEmbeddingShardIndices:
|
||||||
|
"""Get start and end indices for vocab parallel embedding, following the
|
||||||
|
layout outlined in the class docstring, based on the given tp_rank and
|
||||||
|
tp_size."""
|
||||||
|
num_added_embeddings_padded = vocab_size_padded - org_vocab_size_padded
|
||||||
|
padded_org_vocab_start_index, padded_org_vocab_end_index = (
|
||||||
|
vocab_range_from_global_vocab_size(org_vocab_size_padded, tp_rank, tp_size)
|
||||||
|
)
|
||||||
|
padded_added_vocab_start_index, padded_added_vocab_end_index = (
|
||||||
|
vocab_range_from_global_vocab_size(
|
||||||
|
num_added_embeddings_padded, tp_rank, tp_size, offset=org_vocab_size
|
||||||
|
)
|
||||||
|
)
|
||||||
|
# remove padding
|
||||||
|
org_vocab_start_index = min(padded_org_vocab_start_index, org_vocab_size)
|
||||||
|
org_vocab_end_index = min(padded_org_vocab_end_index, org_vocab_size)
|
||||||
|
added_vocab_start_index = min(padded_added_vocab_start_index, vocab_size)
|
||||||
|
added_vocab_end_index = min(padded_added_vocab_end_index, vocab_size)
|
||||||
|
return VocabParallelEmbeddingShardIndices(
|
||||||
|
padded_org_vocab_start_index,
|
||||||
|
padded_org_vocab_end_index,
|
||||||
|
padded_added_vocab_start_index,
|
||||||
|
padded_added_vocab_end_index,
|
||||||
|
org_vocab_start_index,
|
||||||
|
org_vocab_end_index,
|
||||||
|
added_vocab_start_index,
|
||||||
|
added_vocab_end_index,
|
||||||
|
)
|
||||||
|
|
||||||
|
def get_sharded_to_full_mapping(self) -> Optional[List[int]]:
|
||||||
|
"""Get a mapping that can be used to reindex the gathered
|
||||||
|
logits for sampling.
|
||||||
|
|
||||||
|
During sampling, we gather logits from all ranks. The relationship
|
||||||
|
of index->token_id will follow the same format as outlined in the class
|
||||||
|
docstring. However, after the gather, we want to reindex the final
|
||||||
|
logits tensor to map index->token_id one-to-one (the index is always
|
||||||
|
equal the token_id it corresponds to). The indices returned by this
|
||||||
|
method allow us to do that.
|
||||||
|
"""
|
||||||
|
if self.tp_size < 2:
|
||||||
|
return None
|
||||||
|
|
||||||
|
base_embeddings: List[int] = []
|
||||||
|
added_embeddings: List[int] = []
|
||||||
|
padding: List[int] = []
|
||||||
|
for tp_rank in range(self.tp_size):
|
||||||
|
shard_indices = self._get_indices(
|
||||||
|
self.num_embeddings_padded,
|
||||||
|
self.org_vocab_size_padded,
|
||||||
|
self.num_embeddings,
|
||||||
|
self.org_vocab_size,
|
||||||
|
tp_rank,
|
||||||
|
self.tp_size,
|
||||||
|
)
|
||||||
|
range_start = self.num_embeddings_per_partition * tp_rank
|
||||||
|
range_end = self.num_embeddings_per_partition * (tp_rank + 1)
|
||||||
|
base_embeddings.extend(
|
||||||
|
range(range_start, range_start + shard_indices.num_org_elements)
|
||||||
|
)
|
||||||
|
padding.extend(
|
||||||
|
range(
|
||||||
|
range_start + shard_indices.num_org_elements,
|
||||||
|
range_start + shard_indices.num_org_elements_padded,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
added_embeddings.extend(
|
||||||
|
range(
|
||||||
|
range_start + shard_indices.num_org_elements_padded,
|
||||||
|
range_start
|
||||||
|
+ shard_indices.num_org_elements_padded
|
||||||
|
+ shard_indices.num_added_elements,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
padding.extend(
|
||||||
|
range(
|
||||||
|
range_start
|
||||||
|
+ shard_indices.num_org_elements_padded
|
||||||
|
+ shard_indices.num_added_elements,
|
||||||
|
range_start
|
||||||
|
+ shard_indices.num_org_elements_padded
|
||||||
|
+ shard_indices.num_added_elements_padded,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
range_start
|
||||||
|
+ shard_indices.num_org_elements_padded
|
||||||
|
+ shard_indices.num_added_elements_padded
|
||||||
|
== range_end
|
||||||
|
)
|
||||||
|
ret = base_embeddings + added_embeddings + padding
|
||||||
|
assert len(ret) == self.num_embeddings_padded
|
||||||
|
return ret
|
||||||
|
|
||||||
|
def weight_loader(self, param: Parameter, loaded_weight: torch.Tensor):
|
||||||
|
output_dim = getattr(param, "output_dim", None)
|
||||||
|
packed_dim = getattr(param, "packed_dim", None)
|
||||||
|
|
||||||
|
# If the parameter is a gguf weight, then load it directly.
|
||||||
|
if getattr(param, "is_gguf_weight_type", None):
|
||||||
|
param.data.copy_(loaded_weight)
|
||||||
|
param.weight_type = loaded_weight.item()
|
||||||
|
return
|
||||||
|
elif isinstance(param, UninitializedParameter):
|
||||||
|
shape = list(loaded_weight.shape)
|
||||||
|
if output_dim is not None:
|
||||||
|
shape[output_dim] = shape[output_dim] // self.tp_size
|
||||||
|
param.materialize(tuple(shape), dtype=loaded_weight.dtype)
|
||||||
|
|
||||||
|
# If parameter does not have output dim, then it should
|
||||||
|
# be copied onto all gpus (e.g. g_idx for act_order gptq).
|
||||||
|
if output_dim is None:
|
||||||
|
assert param.data.shape == loaded_weight.shape
|
||||||
|
param.data.copy_(loaded_weight)
|
||||||
|
return
|
||||||
|
|
||||||
|
# Shard indexes for loading the weight
|
||||||
|
start_idx = self.shard_indices.org_vocab_start_index
|
||||||
|
shard_size = self.shard_indices.org_vocab_end_index - start_idx
|
||||||
|
|
||||||
|
# If param packed on the same dim we are sharding on, then
|
||||||
|
# need to adjust offsets of loaded weight by pack_factor.
|
||||||
|
if packed_dim is not None and packed_dim == output_dim:
|
||||||
|
packed_factor = (
|
||||||
|
param.packed_factor
|
||||||
|
if isinstance(param, BasevLLMParameter)
|
||||||
|
else param.packed_factor
|
||||||
|
)
|
||||||
|
assert loaded_weight.shape[output_dim] == (
|
||||||
|
self.org_vocab_size // param.packed_factor
|
||||||
|
)
|
||||||
|
start_idx = start_idx // packed_factor
|
||||||
|
shard_size = shard_size // packed_factor
|
||||||
|
else:
|
||||||
|
assert loaded_weight.shape[output_dim] == (
|
||||||
|
self.org_vocab_size
|
||||||
|
// (self.tp_size if self.use_presharded_weights else 1)
|
||||||
|
), f"{self.org_vocab_size=} {self.use_presharded_weights=} {loaded_weight.shape[output_dim]=}"
|
||||||
|
|
||||||
|
# Copy the data.
|
||||||
|
if not self.use_presharded_weights:
|
||||||
|
loaded_weight = loaded_weight.narrow(output_dim, start_idx, shard_size)
|
||||||
|
param[: loaded_weight.shape[0]].data.copy_(loaded_weight)
|
||||||
|
param[loaded_weight.shape[0] :].data.fill_(0)
|
||||||
|
|
||||||
|
def forward(self, input_):
|
||||||
|
if self.tp_size > 1:
|
||||||
|
# Build the mask.
|
||||||
|
masked_input, input_mask = get_masked_input_and_mask(
|
||||||
|
input_,
|
||||||
|
self.shard_indices.org_vocab_start_index,
|
||||||
|
self.shard_indices.org_vocab_end_index,
|
||||||
|
self.shard_indices.num_org_vocab_padding,
|
||||||
|
self.shard_indices.added_vocab_start_index,
|
||||||
|
self.shard_indices.added_vocab_end_index,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
masked_input = input_
|
||||||
|
|
||||||
|
# Get the embeddings.
|
||||||
|
with use_symmetric_memory(
|
||||||
|
get_tp_group(), disabled=not is_allocation_symmetric()
|
||||||
|
):
|
||||||
|
output_parallel = self.quant_method.embedding(self, masked_input.long())
|
||||||
|
|
||||||
|
if self.tp_size > 1:
|
||||||
|
# Mask the output embedding.
|
||||||
|
output_parallel.masked_fill_(input_mask.unsqueeze(-1), 0)
|
||||||
|
if not get_attn_tp_context().input_scattered:
|
||||||
|
if self.use_attn_tp_group:
|
||||||
|
output_parallel = attn_tp_all_reduce(output_parallel)
|
||||||
|
else:
|
||||||
|
# Reduce across all the model parallel GPUs.
|
||||||
|
output_parallel = tensor_model_parallel_all_reduce(output_parallel)
|
||||||
|
return output_parallel
|
||||||
|
|
||||||
|
def extra_repr(self) -> str:
|
||||||
|
s = f"num_embeddings={self.num_embeddings_per_partition}"
|
||||||
|
s += f", embedding_dim={self.embedding_dim}"
|
||||||
|
s += f", org_vocab_size={self.org_vocab_size}"
|
||||||
|
s += f", num_embeddings_padded={self.num_embeddings_padded}"
|
||||||
|
if self.enable_tp:
|
||||||
|
s += f", tp_size={self.tp_size}"
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
class KLXVocabParallelEmbedding(torch.nn.Module):
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
num_embeddings: int,
|
||||||
|
embedding_dim: int,
|
||||||
|
params_dtype: Optional[torch.dtype] = None,
|
||||||
|
org_num_embeddings: Optional[int] = None,
|
||||||
|
padding_size: int = DEFAULT_VOCAB_PADDING_SIZE,
|
||||||
|
quant_config: Optional[QuantizationConfig] = None,
|
||||||
|
prefix: str = "",
|
||||||
|
enable_tp: bool = True,
|
||||||
|
use_attn_tp_group: bool = False,
|
||||||
|
use_presharded_weights: bool = False,
|
||||||
|
):
|
||||||
|
super().__init__()
|
||||||
|
self.quant_config = quant_config
|
||||||
|
|
||||||
|
self.enable_tp = enable_tp
|
||||||
|
self.use_attn_tp_group = use_attn_tp_group
|
||||||
|
if self.enable_tp:
|
||||||
|
if use_attn_tp_group:
|
||||||
|
self.tp_rank = get_attention_tp_rank()
|
||||||
|
self.tp_size = get_attention_tp_size()
|
||||||
|
else:
|
||||||
|
self.tp_rank = get_tensor_model_parallel_rank()
|
||||||
|
self.tp_size = get_tensor_model_parallel_world_size()
|
||||||
|
else:
|
||||||
|
assert use_attn_tp_group is False
|
||||||
|
self.tp_rank = 0
|
||||||
|
self.tp_size = 1
|
||||||
|
# vocal_size, embedding_dim -> vocal_size, embedding_dim // tp_size
|
||||||
|
# P800 workaround: keep the embedding weight replicated on every TP rank
|
||||||
|
# to avoid the unstable all_gather across the BKCL/XCCL stack.
|
||||||
|
self.local_dim = embedding_dim
|
||||||
|
|
||||||
|
self.num_embeddings = num_embeddings
|
||||||
|
self.padding_size = padding_size
|
||||||
|
self.org_vocab_size = org_num_embeddings or num_embeddings
|
||||||
|
num_added_embeddings = num_embeddings - self.org_vocab_size
|
||||||
|
self.use_presharded_weights = use_presharded_weights
|
||||||
|
if use_presharded_weights:
|
||||||
|
assert (
|
||||||
|
num_added_embeddings == 0
|
||||||
|
), "Lora is not supported with presharded weights."
|
||||||
|
|
||||||
|
self.embedding_dim = embedding_dim
|
||||||
|
|
||||||
|
"""Create weights for embedding layer."""
|
||||||
|
weight = Parameter(
|
||||||
|
torch.empty(
|
||||||
|
(self.num_embeddings, self.local_dim),
|
||||||
|
dtype=params_dtype,
|
||||||
|
),
|
||||||
|
requires_grad=False,
|
||||||
|
)
|
||||||
|
set_weight_attrs(
|
||||||
|
weight,
|
||||||
|
{"input_dim": 1, "output_dim": 0, "weight_loader": self.weight_loader},
|
||||||
|
)
|
||||||
|
self.register_parameter("weight", weight)
|
||||||
|
|
||||||
|
def weight_loader(self, param: Parameter, loaded_weight: torch.Tensor):
|
||||||
|
# With the replicated-weight workaround, every rank loads the full weight.
|
||||||
|
param.copy_(loaded_weight)
|
||||||
|
|
||||||
|
def forward(self, input_):
|
||||||
|
hidden_states = F.embedding(input_, self.weight)
|
||||||
|
# No all_gather needed because the full embedding weight is replicated.
|
||||||
|
return hidden_states
|
||||||
|
|
||||||
|
def extra_repr(self) -> str:
|
||||||
|
s = f"num_embeddings={self.num_embeddings}"
|
||||||
|
s += f", embedding_dim={self.embedding_dim}"
|
||||||
|
s += f", org_vocab_size={self.org_vocab_size}"
|
||||||
|
if self.enable_tp:
|
||||||
|
s += f", tp_size={self.tp_size}"
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
class ParallelLMHead(VocabParallelEmbedding):
|
||||||
|
"""Parallelized LM head.
|
||||||
|
|
||||||
|
Output logits weight matrices used in the Sampler. The weight and bias
|
||||||
|
tensors are padded to make sure they are divisible by the number of
|
||||||
|
model parallel GPUs.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
num_embeddings: vocabulary size.
|
||||||
|
embedding_dim: size of hidden state.
|
||||||
|
bias: whether to use bias.
|
||||||
|
params_dtype: type of the parameters.
|
||||||
|
org_num_embeddings: original vocabulary size (without LoRA).
|
||||||
|
padding_size: padding size for the vocabulary.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
num_embeddings: int,
|
||||||
|
embedding_dim: int,
|
||||||
|
*,
|
||||||
|
bias: bool = False,
|
||||||
|
params_dtype: Optional[torch.dtype] = None,
|
||||||
|
org_num_embeddings: Optional[int] = None,
|
||||||
|
padding_size: int = DEFAULT_VOCAB_PADDING_SIZE,
|
||||||
|
quant_config: Optional[QuantizationConfig] = None,
|
||||||
|
prefix: str = "",
|
||||||
|
use_attn_tp_group: bool = False,
|
||||||
|
use_presharded_weights: bool = False,
|
||||||
|
enable_over_encoding: bool = False,
|
||||||
|
):
|
||||||
|
super().__init__(
|
||||||
|
num_embeddings,
|
||||||
|
embedding_dim,
|
||||||
|
params_dtype=params_dtype,
|
||||||
|
org_num_embeddings=org_num_embeddings,
|
||||||
|
padding_size=padding_size,
|
||||||
|
quant_config=quant_config,
|
||||||
|
prefix=prefix,
|
||||||
|
use_attn_tp_group=use_attn_tp_group,
|
||||||
|
use_presharded_weights=use_presharded_weights,
|
||||||
|
enable_over_encoding=enable_over_encoding,
|
||||||
|
)
|
||||||
|
self.quant_config = quant_config
|
||||||
|
|
||||||
|
# We only support pack LMHead if it's not quantized.
|
||||||
|
if _is_cpu and _is_cpu_amx_available:
|
||||||
|
if hasattr(self, "weight") and self.weight.dtype in [
|
||||||
|
torch.bfloat16,
|
||||||
|
torch.float16,
|
||||||
|
]:
|
||||||
|
self.quant_method = PackWeightMethod(weight_names=["weight"])
|
||||||
|
|
||||||
|
if bias:
|
||||||
|
self.bias = Parameter(
|
||||||
|
torch.empty(self.num_embeddings_per_partition, dtype=params_dtype)
|
||||||
|
)
|
||||||
|
set_weight_attrs(
|
||||||
|
self.bias,
|
||||||
|
{
|
||||||
|
"output_dim": 0,
|
||||||
|
"weight_loader": self.weight_loader,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.register_parameter("bias", None)
|
||||||
|
|
||||||
|
def tie_weights(self, embed_tokens: VocabParallelEmbedding):
|
||||||
|
"""Tie the weights with word embeddings."""
|
||||||
|
# GGUF quantized embed_tokens.
|
||||||
|
if self.quant_config and self.quant_config.get_name() == "gguf":
|
||||||
|
return embed_tokens
|
||||||
|
else:
|
||||||
|
self.weight = embed_tokens.weight
|
||||||
|
return self
|
||||||
|
|
||||||
|
def forward(self, input_):
|
||||||
|
del input_
|
||||||
|
raise RuntimeError("LMHead's weights should be used in the sampler.")
|
||||||
77
scripts/analysis/compare_experiments.py
Normal file
77
scripts/analysis/compare_experiments.py
Normal file
@ -0,0 +1,77 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Load results.json from multiple experiments and produce a comparison report.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 scripts/analysis/compare_experiments.py
|
||||||
|
python3 scripts/analysis/compare_experiments.py --output comparison.md
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from collections import defaultdict
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||||
|
|
||||||
|
|
||||||
|
def load_results():
|
||||||
|
"""Glob experiments/*/results/*/results.json and load them."""
|
||||||
|
results = []
|
||||||
|
for path in sorted(REPO_ROOT.glob("experiments/*/results/*/results.json")):
|
||||||
|
try:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
data["_source"] = str(path.relative_to(REPO_ROOT))
|
||||||
|
results.append(data)
|
||||||
|
except (json.JSONDecodeError, OSError) as e:
|
||||||
|
print(f"WARN: failed to load {path}: {e}")
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description="Compare benchmark experiments.")
|
||||||
|
parser.add_argument("--output", "-o", default="comparison_report.md", help="Output markdown file")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
results = load_results()
|
||||||
|
if not results:
|
||||||
|
print("No experiments found under experiments/*/results/*/results.json")
|
||||||
|
return
|
||||||
|
|
||||||
|
# Group by scenario name.
|
||||||
|
by_scenario = defaultdict(list)
|
||||||
|
for data in results:
|
||||||
|
meta = data.get("metadata", {})
|
||||||
|
for scenario in data.get("scenarios", []):
|
||||||
|
key = scenario.get("name", "unknown")
|
||||||
|
by_scenario[key].append((meta, scenario))
|
||||||
|
|
||||||
|
output_path = Path(args.output)
|
||||||
|
with open(output_path, "w", encoding="utf-8") as f:
|
||||||
|
f.write("# Cross-Experiment Comparison\n\n")
|
||||||
|
f.write("| Experiment | Chip | Engine | Scenario | Conc | In/Out | Req/s | OutTok/s | TTFT p99 | TPOT p99 | E2E p99 |\n")
|
||||||
|
f.write("|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|\n")
|
||||||
|
|
||||||
|
for scenario_name in sorted(by_scenario.keys()):
|
||||||
|
for meta, scenario in by_scenario[scenario_name]:
|
||||||
|
lat = scenario.get("latencies", {})
|
||||||
|
f.write(
|
||||||
|
f"| {meta.get('experiment', '')} "
|
||||||
|
f"| {meta.get('chip', '')} "
|
||||||
|
f"| {meta.get('engine', '')} "
|
||||||
|
f"| {scenario_name} "
|
||||||
|
f"| {scenario.get('concurrency', '')} "
|
||||||
|
f"| {scenario.get('input_len', '')}/{scenario.get('output_len', '')} "
|
||||||
|
f"| {scenario.get('request_throughput') or ''} "
|
||||||
|
f"| {scenario.get('output_token_throughput') or ''} "
|
||||||
|
f"| {lat.get('ttft_ms', {}).get('p99') or ''} "
|
||||||
|
f"| {lat.get('tpot_ms', {}).get('p99') or ''} "
|
||||||
|
f"| {lat.get('e2e_ms', {}).get('p99') or ''} |\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Loaded {len(results)} experiment runs.")
|
||||||
|
print(f"Wrote {output_path}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
76
scripts/common/bench_client_docker.sh
Executable file
76
scripts/common/bench_client_docker.sh
Executable file
@ -0,0 +1,76 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Helpers to run sglang.bench_serving inside the P800 Docker container.
|
||||||
|
# Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/bench_client_docker.sh"
|
||||||
|
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
_BENCH_CLIENT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${_BENCH_CLIENT_DIR}/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${_BENCH_CLIENT_DIR}/platform.sh"
|
||||||
|
|
||||||
|
# Required platform variables: CONTAINER_NAME, CONTAINER_PYTHON
|
||||||
|
|
||||||
|
# Run sglang.bench_serving inside the running container.
|
||||||
|
# All arguments are forwarded to bench_serving.
|
||||||
|
run_bench_in_container() {
|
||||||
|
local container="${CONTAINER_NAME}"
|
||||||
|
|
||||||
|
if ! docker inspect "$container" >/dev/null 2>&1; then
|
||||||
|
log "ERROR: container ${container} is not running"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "running bench_serving in container ${container}"
|
||||||
|
docker exec "$container" \
|
||||||
|
env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
|
||||||
|
"${CONTAINER_PYTHON}" -m sglang.bench_serving "$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Convenience wrapper for a single random-dataset case.
|
||||||
|
# Args:
|
||||||
|
# $1: backend (e.g. sglang)
|
||||||
|
# $2: port
|
||||||
|
# $3: model path or served model name
|
||||||
|
# $4: output jsonl path (inside the container)
|
||||||
|
# $5: concurrency
|
||||||
|
# $6: input length
|
||||||
|
# $7: output length
|
||||||
|
# $8: num prompts (optional, default 512)
|
||||||
|
# $9: dataset path inside container (optional)
|
||||||
|
run_random_case() {
|
||||||
|
local backend="$1"
|
||||||
|
local port="$2"
|
||||||
|
local model="$3"
|
||||||
|
local output_file="$4"
|
||||||
|
local concurrency="$5"
|
||||||
|
local input_len="$6"
|
||||||
|
local output_len="$7"
|
||||||
|
local num_prompts="${8:-512}"
|
||||||
|
local dataset_path="${9:-}"
|
||||||
|
local warmup="${WARMUP:-100}"
|
||||||
|
|
||||||
|
# Ensure the output directory exists inside the container.
|
||||||
|
docker exec "${CONTAINER_NAME}" mkdir -p "$(dirname "$output_file")"
|
||||||
|
|
||||||
|
local args=(
|
||||||
|
--backend "$backend"
|
||||||
|
--host 127.0.0.1
|
||||||
|
--port "$port"
|
||||||
|
--model "$model"
|
||||||
|
--dataset-name random
|
||||||
|
--random-input-len "$input_len"
|
||||||
|
--random-output-len "$output_len"
|
||||||
|
--num-prompts "$num_prompts"
|
||||||
|
--max-concurrency "$concurrency"
|
||||||
|
--warmup-requests "$warmup"
|
||||||
|
--output-file "$output_file"
|
||||||
|
--output-details
|
||||||
|
)
|
||||||
|
if [[ -n "$dataset_path" ]]; then
|
||||||
|
args+=(--dataset-path "$dataset_path")
|
||||||
|
fi
|
||||||
|
|
||||||
|
run_bench_in_container "${args[@]}"
|
||||||
|
}
|
||||||
153
scripts/common/lib.sh
Executable file
153
scripts/common/lib.sh
Executable file
@ -0,0 +1,153 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Common helpers for benchmark orchestrators.
|
||||||
|
# Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/lib.sh"
|
||||||
|
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
# Resolve repository root relative to this file.
|
||||||
|
_COMMON_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ROOT_DIR="$(cd "${_COMMON_LIB_DIR}/../.." && pwd)"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Logging
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
LOG_FILE=""
|
||||||
|
|
||||||
|
log_init() {
|
||||||
|
local log_path="$1"
|
||||||
|
LOG_FILE="$log_path"
|
||||||
|
mkdir -p "$(dirname "$LOG_FILE")"
|
||||||
|
: > "$LOG_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
log() {
|
||||||
|
local msg="[$(date --iso-8601=seconds)] $*"
|
||||||
|
echo "$msg"
|
||||||
|
if [[ -n "${LOG_FILE:-}" ]]; then
|
||||||
|
echo "$msg" >> "$LOG_FILE"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Result directories
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
ensure_result_root() {
|
||||||
|
local result_root="$1"
|
||||||
|
mkdir -p "${result_root}/logs"
|
||||||
|
mkdir -p "${result_root}/raw_outputs"
|
||||||
|
echo "$result_root"
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Server health check
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
health_check() {
|
||||||
|
local host="${1:-127.0.0.1}"
|
||||||
|
local port="${2:-30000}"
|
||||||
|
local max_wait="${3:-120}"
|
||||||
|
|
||||||
|
for ((i = 1; i <= max_wait; i++)); do
|
||||||
|
if curl --fail --silent --show-error --max-time 5 "http://${host}:${port}/health" >/dev/null 2>&1; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# Git metadata
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
git_commit() {
|
||||||
|
cd "$ROOT_DIR" || return 1
|
||||||
|
git rev-parse --short HEAD 2>/dev/null || echo "unknown"
|
||||||
|
}
|
||||||
|
|
||||||
|
git_dirty() {
|
||||||
|
cd "$ROOT_DIR" || return 1
|
||||||
|
if [[ -n "$(git status --porcelain 2>/dev/null)" ]]; then
|
||||||
|
echo "dirty"
|
||||||
|
else
|
||||||
|
echo "clean"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# results.json metadata
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
write_metadata_json() {
|
||||||
|
local output_path="$1"
|
||||||
|
local experiment="$2"
|
||||||
|
local run_id="$3"
|
||||||
|
local model="$4"
|
||||||
|
local backend="$5"
|
||||||
|
local engine="$6"
|
||||||
|
local hardware="$7"
|
||||||
|
local accelerator="$8"
|
||||||
|
local chip="$9"
|
||||||
|
local script="${10}"
|
||||||
|
local env_path="${11:-}"
|
||||||
|
local description="${12:-}"
|
||||||
|
|
||||||
|
mkdir -p "$(dirname "$output_path")"
|
||||||
|
|
||||||
|
cat > "$output_path" <<EOF
|
||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "${experiment}",
|
||||||
|
"run_id": "${run_id}",
|
||||||
|
"timestamp": "$(date --iso-8601=seconds)",
|
||||||
|
"model": "${model}",
|
||||||
|
"backend": "${backend}",
|
||||||
|
"engine": "${engine}",
|
||||||
|
"hardware": "${hardware}",
|
||||||
|
"accelerator": "${accelerator}",
|
||||||
|
"chip": "${chip}",
|
||||||
|
"script": "${script}",
|
||||||
|
"env": "${env_path}",
|
||||||
|
"git_commit": "$(git_commit)",
|
||||||
|
"git_dirty": "$(git_dirty)",
|
||||||
|
"description": "${description}"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
# JSON append helper (naive but sufficient for small result JSONs)
|
||||||
|
# -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
append_scenario_to_json() {
|
||||||
|
local json_path="$1"
|
||||||
|
local scenario_json="$2"
|
||||||
|
|
||||||
|
if [[ ! -f "$json_path" ]]; then
|
||||||
|
echo "ERROR: $json_path not found" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Use Python for safe JSON manipulation.
|
||||||
|
"${PYTHON:-python3}" - "$json_path" "$scenario_json" <<'PY'
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
json_path = sys.argv[1]
|
||||||
|
scenario_json = sys.argv[2]
|
||||||
|
|
||||||
|
with open(json_path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
|
||||||
|
scenario = json.loads(scenario_json)
|
||||||
|
data["scenarios"].append(scenario)
|
||||||
|
|
||||||
|
with open(json_path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||||
|
PY
|
||||||
|
}
|
||||||
37
scripts/common/platform.sh
Executable file
37
scripts/common/platform.sh
Executable file
@ -0,0 +1,37 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Platform loader.
|
||||||
|
# Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/platform.sh"
|
||||||
|
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
_PLATFORM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ROOT_DIR="$(cd "${_PLATFORM_DIR}/../.." && pwd)"
|
||||||
|
|
||||||
|
# Auto-detect platform if not set.
|
||||||
|
if [[ -z "${PLATFORM:-}" ]]; then
|
||||||
|
if lspci 2>/dev/null | grep -qiE "XPU|Kunlun"; then
|
||||||
|
PLATFORM="kunlun_p800"
|
||||||
|
elif command -v nvidia-smi >/dev/null 2>&1; then
|
||||||
|
PLATFORM="nvidia_h200"
|
||||||
|
else
|
||||||
|
echo "ERROR: Could not auto-detect PLATFORM. Set PLATFORM env var explicitly." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Auto-detected platform: ${PLATFORM}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
PLATFORM_FILE="${ROOT_DIR}/platforms/${PLATFORM}.env"
|
||||||
|
if [[ ! -f "$PLATFORM_FILE" ]]; then
|
||||||
|
echo "ERROR: Platform config not found: ${PLATFORM_FILE}" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "$PLATFORM_FILE"
|
||||||
|
|
||||||
|
# Export key variables if not already set.
|
||||||
|
export CHIP="${CHIP:-$PLATFORM}"
|
||||||
|
export ACCELERATOR="${ACCELERATOR:-$PLATFORM}"
|
||||||
|
export HARDWARE="${HARDWARE:-$PLATFORM}"
|
||||||
|
export DEFAULT_PORT="${DEFAULT_PORT:-30000}"
|
||||||
|
export MODEL_ROOT="${MODEL_ROOT:-/data1/models}"
|
||||||
205
scripts/common/server_docker.sh
Executable file
205
scripts/common/server_docker.sh
Executable file
@ -0,0 +1,205 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Docker-based SGLang server lifecycle helpers for Kunlun P800.
|
||||||
|
# Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/server_docker.sh"
|
||||||
|
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
_DOCKER_COMMON_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${_DOCKER_COMMON_DIR}/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${_DOCKER_COMMON_DIR}/platform.sh"
|
||||||
|
|
||||||
|
# Required platform variables: DOCKER_IMAGE, CONTAINER_NAME, DEFAULT_PORT, PATCH_ROOT
|
||||||
|
|
||||||
|
docker_server_status() {
|
||||||
|
docker inspect -f '{{.State.Status}}' "$CONTAINER_NAME" 2>/dev/null || echo "missing"
|
||||||
|
}
|
||||||
|
|
||||||
|
docker_server_stop() {
|
||||||
|
local container="${1:-$CONTAINER_NAME}"
|
||||||
|
log "stopping container ${container}"
|
||||||
|
docker rm -f "$container" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Clean up leftover shared memory on the host.
|
||||||
|
ipcs -m 2>/dev/null | awk '$4 == 666 {print $2}' | while read -r shmid; do
|
||||||
|
ipcrm -m "$shmid" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Kill lingering sglang processes just in case.
|
||||||
|
pkill -9 -f 'sglang.launch_server' 2>/dev/null || true
|
||||||
|
pkill -9 -f 'multiprocessing.spawn' 2>/dev/null || true
|
||||||
|
pkill -9 -f 'multiprocessing.resource_tracker' 2>/dev/null || true
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
|
||||||
|
# Build device args for docker run (8 XPU devices).
|
||||||
|
_docker_device_args() {
|
||||||
|
local args=""
|
||||||
|
for i in 0 1 2 3 4 5 6 7; do
|
||||||
|
args="${args} --device /dev/xpu${i}:/dev/xpu${i}"
|
||||||
|
done
|
||||||
|
args="${args} --device /dev/xpuctrl:/dev/xpuctrl"
|
||||||
|
echo "$args"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Start the SGLang server container for DeepSeek-V4 on P800.
|
||||||
|
# Args:
|
||||||
|
# $1: model path on host (e.g. /data1/models/DeepSeek-V4-Flash-INT8)
|
||||||
|
# $2: port (optional, defaults to DEFAULT_PORT)
|
||||||
|
# $3: server log path (optional)
|
||||||
|
# $4: server mode (optional): "fp8" or "w8a8_int8" (default "fp8")
|
||||||
|
docker_server_start() {
|
||||||
|
local model_path="$1"
|
||||||
|
local port="${2:-$DEFAULT_PORT}"
|
||||||
|
local server_log="${3:-${RESULT_ROOT:-/tmp}/logs/server.outer.log}"
|
||||||
|
local mode="${4:-fp8}"
|
||||||
|
local container="${CONTAINER_NAME}"
|
||||||
|
local image="${DOCKER_IMAGE}"
|
||||||
|
|
||||||
|
mkdir -p "$(dirname "$server_log")"
|
||||||
|
|
||||||
|
log "starting container ${container} from ${image}"
|
||||||
|
log "model path: ${model_path} -> /models"
|
||||||
|
log "port: ${port}"
|
||||||
|
log "mode: ${mode}"
|
||||||
|
log "server log: ${server_log}"
|
||||||
|
|
||||||
|
docker_server_stop "$container"
|
||||||
|
|
||||||
|
local device_args
|
||||||
|
device_args="$(_docker_device_args)"
|
||||||
|
|
||||||
|
# Ensure patch files exist under PATCH_ROOT. Fall back to /tmp if the repo
|
||||||
|
# copy is not present (legacy path).
|
||||||
|
local patch_root="${PATCH_ROOT:-/tmp}"
|
||||||
|
|
||||||
|
# Common environment variables shared by all modes.
|
||||||
|
local env_args=(
|
||||||
|
-e XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
-e CUDA_DEVICE_ORDER=OAM_ID
|
||||||
|
-e SGLANG_USE_TRANSFORMERS_V5_TOKENIZER=1
|
||||||
|
-e XMLIR_FORCE_USE_XPU_GRAPH=1
|
||||||
|
-e SGLANG_DSV4_MODE=2604
|
||||||
|
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
||||||
|
-e SGLANG_NSA_DUAL_STREAM=true
|
||||||
|
-e SGLANG_NSA_QUANT_WQ_B_WK=false
|
||||||
|
-e SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1
|
||||||
|
-e SGLANG_OPT_DEEPGEMM_HC_PRENORM=false
|
||||||
|
-e SGLANG_OPT_USE_TILELANG_MHC_PRE=1
|
||||||
|
-e SGLANG_OPT_USE_TILELANG_MHC_POST=1
|
||||||
|
-e SGLANG_CLEAN_REQUEST_WHEN_RETRACT=1
|
||||||
|
-e SGLANG_SET_CPU_AFFINITY=1
|
||||||
|
-e SGLANG_OPT_USE_KLX_TOPK_KERNEL=1
|
||||||
|
-e XSGL_INTERTYPE_BFP16=1
|
||||||
|
-e ENABLE_FAST_BFP16_ATTN=1
|
||||||
|
-e XSGL_USE_DEEP_GEMM_BMM=1
|
||||||
|
-e XSGL_XDNN_QUANT=1
|
||||||
|
-e XSGL_FUSE_RMS_NORM_QUANT=1
|
||||||
|
-e XSGL_TRANSPOSE_MATMUL_WEIGHT=1
|
||||||
|
-e XINFER_QUANT_SDNN=1
|
||||||
|
-e XSGL_USE_MOE_SIGMOID_GROUP_TOPK_NORM=1
|
||||||
|
-e XSGL_EARLY_FIRST_TOKEN=1
|
||||||
|
-e USE_FAST_BFP16_MOE=1
|
||||||
|
-e XSGL_ENABLE_TGEMM_FP16=1
|
||||||
|
-e SGLANG_ENABLE_SPEC_V2=True
|
||||||
|
-e SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||||
|
-e PYTHONDONTWRITEBYTECODE=1
|
||||||
|
-e XTORCH_OPS_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/xtorch_ops
|
||||||
|
-e XPU_RUNTIME_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/xcudart/lib
|
||||||
|
-e BKCL_TREE_THRESHOLD=1048576
|
||||||
|
-e CUDA_ENABLE_P2P_NO_UVA=1
|
||||||
|
-e NCCL_IB_GID_INDEX=3
|
||||||
|
-e IS_DSV4=1
|
||||||
|
-e MC_CUSTOM_TOPO_JSON=/workspace/nic_priority_matrix_test.json
|
||||||
|
)
|
||||||
|
|
||||||
|
# Mode-specific environment variables.
|
||||||
|
if [[ "$mode" == "w8a8_int8" ]]; then
|
||||||
|
env_args+=(
|
||||||
|
-e SGLANG_DSV4_FP4_EXPERTS=false
|
||||||
|
-e SGLANG_APPLY_CONFIG_BACKUP=auto
|
||||||
|
-e BKCL_ENABLE_XDR=1
|
||||||
|
-e BKCL_RDMA_NICS=eth1,eth1,eth3,eth3,eth5,eth5,eth7,eth7
|
||||||
|
-e BKCL_RDMA_VERBS=1
|
||||||
|
-e XSGL_INT8_LM_HEAD=1
|
||||||
|
-e SGLANG_P800_ALL_GATHER_FALLBACK=0
|
||||||
|
)
|
||||||
|
else
|
||||||
|
env_args+=(
|
||||||
|
-e SGLANG_DSV4_FP4_EXPERTS=true
|
||||||
|
-e SGLANG_OPT_FUSE_WQA_WKV=false
|
||||||
|
-e SGLANG_DSV4_2604_SUBMODE=2604B
|
||||||
|
-e SGLANG_APPLY_CONFIG_BACKUP=small
|
||||||
|
-e BKCL_ENABLE_XDR=0
|
||||||
|
-e BKCL_FORCE_PCIE_P2P=1
|
||||||
|
-e BKCL_ENABLE_PCIE_P2P=1
|
||||||
|
-e BKCL_SOCKET_IFNAME=bond0
|
||||||
|
-e BKCL_NET_ENABLE_INTRA=1
|
||||||
|
-e XSHMEM_MODE=0
|
||||||
|
-e SGLANG_P800_ALL_GATHER_FALLBACK=1
|
||||||
|
)
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Mode-specific launch arguments.
|
||||||
|
local launch_args
|
||||||
|
if [[ "$mode" == "w8a8_int8" ]]; then
|
||||||
|
launch_args="--host 0.0.0.0 --port ${port} --model-path /models --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.8 --max-prefill-tokens 16384 --max-running-requests 64 --tensor-parallel-size 8 --ep-size 8 --disable-shared-experts-fusion --quantization w8a8_int8 --kv-cache-dtype float16 --disable-piecewise-cuda-graph --cuda-graph-max-bs 32 --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --constrained-json-disable-any-whitespace --enable-metrics --enable-request-time-stats-logging"
|
||||||
|
else
|
||||||
|
launch_args="--host 0.0.0.0 --port ${port} --model-path /models --quantization fp8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.85 --max-prefill-tokens 16384 --context-length 65536 --max-running-requests 8 --max-total-tokens 524288 --tensor-parallel-size 8 --ep-size 8 --moe-runner-backend deep_gemm --disable-shared-experts-fusion --kv-cache-dtype float16 --disable-piecewise-cuda-graph --disable-cuda-graph --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --enable-metrics --enable-request-time-stats-logging"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Build the full server bootstrap command and base64-encode it to avoid
|
||||||
|
# host-shell quoting hell.
|
||||||
|
local server_cmd
|
||||||
|
server_cmd=$(cat <<EOF
|
||||||
|
cd /workspace
|
||||||
|
find /root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
|
||||||
|
/root/miniconda/envs/python310_torch25_cuda/bin/pip install --upgrade safetensors -q
|
||||||
|
/root/miniconda/envs/python310_torch25_cuda/bin/pip install https://files.pythonhosted.org/packages/14/8b/2a1333a6455c6fad401c2285dee6f58016c55b1cb44cae3a31f8a9cc7d83/apache_tvm_ffi-0.1.0b2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl -q
|
||||||
|
/root/miniconda/envs/python310_torch25_cuda/bin/python -c "import torch; torch.float8_e8m0fnu = torch.uint8; import runpy, sys; sys.argv[0] = 'sglang.launch_server'; runpy.run_module('sglang.launch_server', run_name='__main__')" ${launch_args}
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
local server_cmd_b64
|
||||||
|
server_cmd_b64=$(printf '%s' "$server_cmd" | base64 -w0)
|
||||||
|
|
||||||
|
# Patch mounts. For w8a8_int8 the working container relies on patches baked
|
||||||
|
# into the image, so we only mount the runtime config, nic matrix, and a
|
||||||
|
# local dummy dataset for offline benchmarking.
|
||||||
|
local patch_mounts=(
|
||||||
|
-v "${patch_root}/nic_priority_matrix_test.json:/workspace/nic_priority_matrix_test.json:ro"
|
||||||
|
-v "${patch_root}/dummy_sharegpt.json:/workspace/dummy_sharegpt.json:ro"
|
||||||
|
)
|
||||||
|
if [[ "$mode" == "fp8" ]]; then
|
||||||
|
patch_mounts+=(
|
||||||
|
-v "${patch_root}/sitecustomize_xpu.py:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sitecustomize.py:ro"
|
||||||
|
-v "${patch_root}/hf_transformers_utils.py.patched:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/utils/hf_transformers_utils.py:ro"
|
||||||
|
-v "${patch_root}/fp8_utils.py.patched:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/layers/quantization/fp8_utils.py:ro"
|
||||||
|
-v "${patch_root}/config_backup_small_fp8.json:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/configs/config_backup_small.json:ro"
|
||||||
|
-v "${patch_root}/parallel_state.py:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/distributed/parallel_state.py:ro"
|
||||||
|
-v "${patch_root}/vocab_parallel_embedding.py:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/layers/vocab_parallel_embedding.py:ro"
|
||||||
|
)
|
||||||
|
fi
|
||||||
|
|
||||||
|
docker run -d \
|
||||||
|
--name "${container}" \
|
||||||
|
--privileged \
|
||||||
|
--network host \
|
||||||
|
--ipc host \
|
||||||
|
${device_args} \
|
||||||
|
-v "${model_path}:/models:ro" \
|
||||||
|
"${patch_mounts[@]}" \
|
||||||
|
"${env_args[@]}" \
|
||||||
|
"${image}" \
|
||||||
|
bash -c "echo '${server_cmd_b64}' | base64 -d | bash" \
|
||||||
|
>> "${server_log}" 2>&1
|
||||||
|
|
||||||
|
log "container ${container} started, waiting for health"
|
||||||
|
if health_check 127.0.0.1 "$port" 600; then
|
||||||
|
log "container ${container} is healthy"
|
||||||
|
else
|
||||||
|
log "ERROR: container ${container} failed health check"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
}
|
||||||
Loading…
x
Reference in New Issue
Block a user