qqtang/sskj
1
3
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
sskj/docs/EXPERIMENT_GUIDE.md
shishi 63ab41b65a docs: consolidate project docs (dedup, relocate, expand 910C client guide) 
Project-level documentation was scattered and duplicated across README.md,
BENCHMARK_WORKFLOW.md, and docs/EXPERIMENT_GUIDE.md (directory layout +
scripts/common component table repeated 3x). Reorganize into a clear
single-source-of-truth structure.

Changes:
- README.md: drop the 6 stale changelog entries at the top (latest was
  07-21; history lives in git log). Replace the duplicated directory-
  layout + scripts/common sections with a one-line link to
  docs/EXPERIMENT_GUIDE.md. (151 -> 99 lines)
- BENCHMARK_WORKFLOW.md -> docs/BENCHMARK_WORKFLOW.md: relocate into docs/.
  Replace its duplicated Directory Layout and Quick Start/Adding sections
  with links to EXPERIMENT_GUIDE / README / NEW_PLATFORM_GUIDE; keep the
  unique parts (Rules, Naming Conventions, Final JSON Schema, Checklist).
  (394 -> 224 lines)
- docs/EXPERIMENT_GUIDE.md: now the single authority for directory layout
  + component table + experiment conventions. Add a cross-link from the
  results.json field list to BENCHMARK_WORKFLOW's full JSON Schema and
  Naming Conventions.
- docs/H200_QUICKSTART.md: deleted (outdated, repeatedly references
  removed legacy scripts; H200 usage is covered by ADAPTIVE_CONCURRENCY_USAGE
  and experiment READMEs).
- docs/DSV4_INFERENCE_COMPARISON_REPORT.md -> experiments/h200/
  dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/: this is an
  experiment report, not a project doc; relocate next to its sibling
  report.md.
- envs/ASCEND_910C_ENV_SETUP.md §8: expand the vague "pip install sglang"
  note into a full sglang client image build guide -- pin sglang 0.5.2
  (not latest; >=0.5.16 deprecates bench_serving and breaks the parser),
  --no-deps minimal install loop, docker commit to a local image, with
  the exact commands used to build local/vllm-ascend:0.23-a3-dsv4-sglang.
- experiments/h200/dsv4_h200_vllm_tp2_custom_bench/README.md: fix the
  now-broken link to BENCHMARK_WORKFLOW.md (../../ -> ../../../docs/).
- .gitignore: ignore *.bak.glm52orig scratch backups.

Also includes the add16 adaptive_results produced by the dsv4 TP=4/DP=2
runs on 910c.1.
2026-07-29 11:52:42 +08:00

208 lines
7.6 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# 实验规范指南
本仓库用于统一记录和复现不同芯片、不同 backend（SGLang / vLLM）下的推理测速实验。
## 1. 目录结构
实验目录为三层结构：`experiments/<platform>/<experiment_name>/`，其中 platform ∈ {h20, h200, p800, pro6000}；另有 `experiments/TEMPLATE/` 作为新实验模板。
```
experiments/<platform>/<experiment_name>/
├── config.env # 实验参数唯一来源
├── run_bench.sh # 编排入口
├── start_sglang.sh # SGLang server 启动脚本
├── start_vllm.sh # vLLM server 启动脚本
└── results/<run_id>/
├── comparison.md
├── sglang/
│ ├── results.json
│ ├── report.md
│ ├── raw_outputs/ # .gitignore 忽略
│ └── logs/ # .gitignore 忽略
└── vllm/
├── results.json
├── report.md
├── raw_outputs/ # .gitignore 忽略
└── logs/ # .gitignore 忽略
```
上述是 TEMPLATE 式的固定场景对比实验布局（sglang vs vllm，老形态，仍可用）。当前主流实验形态是 TP/DP matrix + 自适应并发搜索：实验目录内由 `matrix.json` + `generate_scenarios.py` 生成场景，`adaptive_config.env` 配置搜索参数，`run_adaptive_concurrency.sh` 驱动，结果落盘 `adaptive_results/<run_id>/`（`adaptive_points.jsonl`、`adaptive_summary.{jsonl,md,csv}`、`run_manifest.json`、`tp*_dp*/` 等），用法见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`。
通用工具集中放在 `scripts/common/`，不要复制到每个实验。三层实验目录以 `../../../scripts/common` 引用（`experiments/TEMPLATE/` 为两层，用 `../../scripts/common`）：
- `scripts/common/lib.sh`：日志、metadata、JSON 工具
- `scripts/common/platform.sh`：平台自动检测
- `scripts/common/warmup.py`：server 预热
- `scripts/common/parse_backend.py`：解析 raw jsonl -> results.json + report.md
- `scripts/common/compare.py`：生成 SGLang vs vLLM 对比表
- `scripts/common/adaptive_bench_lib.sh` / `adaptive_concurrency.py`：自适应并发搜索
- `scripts/common/server_docker.sh` / `bench_client_docker.sh`：Docker server / client 管理
- `scripts/common/adaptive_heartbeat.sh`：自适应测试心跳保活
## 2. 新增一个实验
最快方式：
```bash
cp -r experiments/TEMPLATE experiments/<platform>/<your_experiment_name>
# 修改 config.env、start_*.sh
bash experiments/<platform>/<your_experiment_name>/run_bench.sh
```
### 2.1 config.env 必备字段
```bash
EXPERIMENT="<experiment_name>"
MODEL_NAME="DeepSeek-V4-Flash"
MODEL_PATH="/data/models/DeepSeek-V4-Flash"
SGLANG_PORT="${SGLANG_PORT:-30006}"
VLLM_PORT="${VLLM_PORT:-30005}"
# venv 路径按机器实际配置调整
VENV_SGLANG="${VENV_SGLANG:-/path/to/envs/sglang}"
VENV_VLLM="${VENV_VLLM:-/path/to/envs/vllm}"
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
TP="${TP:-8}"
MAX_MODEL_LEN=...
MAX_NUM_SEQS=...
MAX_RUNNING=...
# 场景："concurrency input_len output_len num_prompts"
declare -a SCENARIOS=(
"1 512 256 32"
)
VENV_CLIENT="${VENV_CLIENT:-$VENV_SGLANG}"
SGLANG_START_SCRIPT="${SCRIPT_DIR:-.}/start_sglang.sh"
VLLM_START_SCRIPT="${SCRIPT_DIR:-.}/start_vllm.sh"
```
### 2.2 可复现性要求
每个 `results.json` 必须记录：
- `metadata.model`：模型路径
- `metadata.hardware` / `metadata.accelerator` / `metadata.chip`
- `metadata.env`：使用的虚拟环境路径
- `metadata.git_commit` / `metadata.git_dirty`
- `config.server_args`：完整的 server 启动命令
- `config.cuda_visible_devices`
这些通过 `scripts/common/lib.sh` 中的 `write_metadata_json` 和 `jq` 注入。
> 完整的 `results.json` schema（含 scenarios/latencies/slo_status 等字段）见 [`BENCHMARK_WORKFLOW.md`](BENCHMARK_WORKFLOW.md) §Final JSON Schema；结果目录命名规范见同文 §Naming Conventions。
## 3. 平台与硬件适配
### 3.1 芯片级默认配置
芯片相关默认值放到 `platforms/<chip>.env`，现有：
- `platforms/nvidia_h20.env`
- `platforms/nvidia_h200.env`
- `platforms/nvidia_rtx6000d.env`
- `platforms/kunlun_p800.env`
实验级覆盖通过 `config.env` 实现。
### 3.2 跨芯片注意事项
| 项 | H200 | P800 | 备注 |
|---|---|---|---|
| TP 大小 | 8 | 8/16 | 由 `TP` 控制 |
| KV cache dtype | fp8 | 可能不同 | 在 `start_vllm.sh` 调整 |
| MLA backend | flashinfer_mla | 可能不支持 | 在 `start_sglang.sh`/`start_vllm.sh` 调整 |
| 最大上下文 | 受显存限制 | 受显存限制 | 通过探针测试确定 |
| 设备变量 | `CUDA_VISIBLE_DEVICES` | 可能不同 | 在 `config.env` 设置 |
如果某芯片不支持某个 feature，在 `start_*.sh` 里用条件判断，不要把条件写进通用脚本。
## 4. SLO 标准
默认使用 S2 层级（完整分级定义见 `docs/SLO_STANDARDS.md`）：
- TTFT P95 < 3000 ms
- TPOT mean < 50 ms
`scripts/common/parse_backend.py` 和 `scripts/common/compare.py` 默认按这个标准打标。如果需要其他 tier，调用 `compare.py` 时传入 `--ttft-limit` 和 `--tpot-limit`。
## 5. 断点续测
`run_bench.sh` 必须实现 `scenario_already_completed()` 检查：
- 如果某 scenario 的 `raw_outputs/*.jsonl` 中 `completed` 数量已达标，直接跳过
- 重新执行脚本即可从中断处继续
## 6. Git 提交规范
### 6.1 必须提交
- 实验代码：`config.env`、`run_bench.sh`、`start_*.sh`
- 最终结果：`comparison.md`、`sglang/results.json`、`sglang/report.md`、`vllm/results.json`、`vllm/report.md`
### 6.2 不要提交
- `raw_outputs/`
- `logs/`
- 中间 server 日志
这些已在 `.gitignore` 中忽略。
### 6.3 提交示例
```bash
git add experiments/<platform>/<name>/config.env \
experiments/<platform>/<name>/run_bench.sh \
experiments/<platform>/<name>/start_*.sh \
experiments/<platform>/<name>/results/<run_id>/comparison.md \
experiments/<platform>/<name>/results/<run_id>/sglang/results.json \
experiments/<platform>/<name>/results/<run_id>/sglang/report.md \
experiments/<platform>/<name>/results/<run_id>/vllm/results.json \
experiments/<platform>/<name>/results/<run_id>/vllm/report.md
git commit -m "results: <experiment_name> run <run_id>"
git push origin main
```
## 7. 场景设计规范
### 7.1 请求数 = 并发数 × 5
每个 scenario 的 `num_prompts` 必须是对应 `concurrency` 的 **5 倍**，即每个并发档位至少跑满 5 轮请求，保证 P95/P99 等尾延迟指标有统计意义。
```bash
# 格式：concurrency input_len output_len num_prompts
declare -a SCENARIOS=(
"1 512 256 5" # 1 × 5
"32 512 256 160" # 32 × 5
"128 512 256 640" # 128 × 5
)
```
例外：对于超长上下文（如 128k/256k）或探针类实验，若执行成本过高，可在 `config.env` 中显式注释说明原因，并保留不小于 `concurrency × 2` 的最低样本量。
## 8. 命名约定
- 实验目录：`experiments/<platform>/{model}_{chip}_{scenario}_{backend_vs_backend}`，例如 `experiments/h200/dsv4_h200_sglang_vs_vllm`
- scenario 名称：`c{concurrency}_i{input_len}_o{output_len}`
- raw 输出：`{backend}_{label}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl`
- run_id：`YYYYMMDD-HHMMSS`
## 9. 常见问题
### 8.1 新芯片上 server 起不来
1. 检查 `platforms/<chip>.env` 是否已定义
2. 检查 `start_*.sh` 中的 backend-specific 参数是否被该芯片支持
3. 先用最小场景（input=512, output=256, concurrency=1）验证通路
### 8.2 长上下文 OOM
1. 降低并发
2. 降低输出长度
3. 调整 `mem-fraction-static` / `gpu-memory-utilization`
4. 记录能跑通的最大组合，更新矩阵
Reference in New Issue View Git Blame Copy Permalink