docs: record chip/accelerator and inference engine in benchmark workflow

Update BENCHMARK_WORKFLOW.md to require scripts and final outputs to
capture the hardware platform (e.g. NVIDIA H200, Kunlun XPU) and the
inference engine/backend (e.g. vllm-dspark, sglang, vllm-xpu). This
includes directory/file naming conventions, the results.json metadata
schema, and the pre-archive checklist.
This commit is contained in:
Quantong Qiu 2026-07-08 03:52:23 +00:00
parent 7de88b995a
commit 2c332ff712

View File

@ -43,7 +43,8 @@
- The directory may also contain a `README.md` documenting provenance if the report alone does not cover it. - The directory may also contain a `README.md` documenting provenance if the report alone does not cover it.
4. **Final JSON must contain raw/structured data, not just summary numbers.** 4. **Final JSON must contain raw/structured data, not just summary numbers.**
- Metadata: experiment name, timestamp, model, backend, hardware, script path, environment/commit info. - Metadata: experiment name, timestamp, model, backend/inference engine, hardware/accelerator, script path, environment/commit info.
- Record the **chip/accelerator** (e.g. `NVIDIA H200`, `Kunlun XPU`) and the **inference engine** (e.g. `vllm-dspark`, `sglang`, `vllm-xpu`) explicitly. Do not infer them from directory names.
- Per-scenario/per-configuration results: all request latencies, TTFT, TPOT, ITL, token counts, throughput, accept length, success/failure counts. - Per-scenario/per-configuration results: all request latencies, TTFT, TPOT, ITL, token counts, throughput, accept length, success/failure counts.
- Include P50 / P90 / P95 / P99 percentiles where applicable. - Include P50 / P90 / P95 / P99 percentiles where applicable.
- Keep the schema stable so downstream Python scripts can parse all experiments uniformly. - Keep the schema stable so downstream Python scripts can parse all experiments uniformly.
@ -57,6 +58,11 @@
- `logs/<service>_<timestamp>.log` - `logs/<service>_<timestamp>.log`
- Keep server logs separate from benchmark result logs. - Keep server logs separate from benchmark result logs.
7. **Scripts and outputs must record chip/accelerator and inference engine.**
- Every benchmark script should capture or accept the platform and engine it is running on (e.g. via environment variables `CHIP`, `ACCELERATOR`, `ENGINE`, `BACKEND`, or auto-detection).
- Final reports and JSON outputs must include both the accelerator/chip family and the inference engine/backend used for the run.
- Do not rely on the experiment name alone to identify the platform or engine.
## Naming Conventions ## Naming Conventions
### Result directories ### Result directories
@ -70,18 +76,27 @@ Examples:
- `bench_results/dspark_grid_20260707-132641/` - `bench_results/dspark_grid_20260707-132641/`
- `bench_results/dsv4_backend_comparison_20260707/` - `bench_results/dsv4_backend_comparison_20260707/`
When the same experiment is repeated across chips or engines, include them in the directory name or organize by subdirectories so results are not confused:
```
bench_results/<experiment>_<chip>_<engine>_<YYYYMMDD-HHMMSS>/
bench_results/<experiment>/<chip>/<engine>/<YYYYMMDD-HHMMSS>/
```
### Raw output files ### Raw output files
Include the accelerator and inference engine in raw output filenames so files from different platforms cannot overwrite each other.
For detailed per-request JSONL outputs: For detailed per-request JSONL outputs:
``` ```
{backend}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl {chip}_{engine}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
``` ```
For summary JSON outputs from `sglang.bench_serving --output-file`: For summary JSON outputs from `sglang.bench_serving --output-file`:
``` ```
{backend}_{scenario}_{params}.json {chip}_{engine}_{scenario}_{params}.json
``` ```
### Logs ### Logs
@ -105,8 +120,11 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
"timestamp": "2026-07-07T13:26:41+08:00", "timestamp": "2026-07-07T13:26:41+08:00",
"model": "/data/models/DeepSeek-V4-Flash-DSpark", "model": "/data/models/DeepSeek-V4-Flash-DSpark",
"backend": "vllm-dspark", "backend": "vllm-dspark",
"script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh", "engine": "vllm-dspark",
"hardware": "8x NVIDIA H200 143GB", "hardware": "8x NVIDIA H200 143GB",
"accelerator": "NVIDIA H200",
"chip": "NVIDIA H200",
"script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh",
"env": "/data/user1/yy/envs/vllm-dspark", "env": "/data/user1/yy/envs/vllm-dspark",
"git_commit": "optional git sha", "git_commit": "optional git sha",
"description": "optional free-text note" "description": "optional free-text note"
@ -163,6 +181,11 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
- `raw_requests` is optional but recommended when the JSON size is manageable. If a single run produces millions of requests, store per-request data as `raw_outputs/*.jsonl` and keep only aggregated percentiles in `results.json`. - `raw_requests` is optional but recommended when the JSON size is manageable. If a single run produces millions of requests, store per-request data as `raw_outputs/*.jsonl` and keep only aggregated percentiles in `results.json`.
- Always include **P50 / P90 / P95 / P99** for TTFT, TPOT, E2E, and ITL. P95 is the primary SLO metric. - Always include **P50 / P90 / P95 / P99** for TTFT, TPOT, E2E, and ITL. P95 is the primary SLO metric.
- Keep field names snake_case and consistent across experiments. - Keep field names snake_case and consistent across experiments.
- Record hardware and engine information explicitly:
- `accelerator` / `chip`: the accelerator family, e.g. `NVIDIA H200`, `Kunlun XPU`, `AMD MI300X`.
- `engine` / `backend`: the inference engine or serving backend, e.g. `vllm-dspark`, `sglang`, `vllm-xpu`.
- `hardware`: a human-readable full hardware description, e.g. `8x NVIDIA H200 143GB`, `8x Kunlun XPU R480`.
- Keep at least one of `accelerator` or `chip`, and at least one of `engine` or `backend`, populated in every run.
- If a metric is not applicable (e.g. `accept_length` for non-speculative decoding), set it to `null` rather than omitting the key. - If a metric is not applicable (e.g. `accept_length` for non-speculative decoding), set it to `null` rather than omitting the key.
## Quick Start ## Quick Start
@ -199,6 +222,8 @@ bash scripts/benchmark_dsv4_backend_comparison.sh all
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`. - [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
- [ ] `bench_results/<run>/report.md` (or equivalent human-readable `.md`) exists. - [ ] `bench_results/<run>/report.md` (or equivalent human-readable `.md`) exists.
- [ ] `bench_results/<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema). - [ ] `bench_results/<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
- [ ] `bench_results/<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
- [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide.
- [ ] `bench_results/<run>/README.md` exists and documents provenance (or the report itself covers provenance). - [ ] `bench_results/<run>/README.md` exists and documents provenance (or the report itself covers provenance).
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`). - [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
- [ ] Script path references updated after moving. - [ ] Script path references updated after moving.