docs: record chip/accelerator and inference engine in benchmark workflow
Update BENCHMARK_WORKFLOW.md to require scripts and final outputs to capture the hardware platform (e.g. NVIDIA H200, Kunlun XPU) and the inference engine/backend (e.g. vllm-dspark, sglang, vllm-xpu). This includes directory/file naming conventions, the results.json metadata schema, and the pre-archive checklist.
This commit is contained in:
parent
7de88b995a
commit
2c332ff712
@ -43,7 +43,8 @@
|
|||||||
- The directory may also contain a `README.md` documenting provenance if the report alone does not cover it.
|
- The directory may also contain a `README.md` documenting provenance if the report alone does not cover it.
|
||||||
|
|
||||||
4. **Final JSON must contain raw/structured data, not just summary numbers.**
|
4. **Final JSON must contain raw/structured data, not just summary numbers.**
|
||||||
- Metadata: experiment name, timestamp, model, backend, hardware, script path, environment/commit info.
|
- Metadata: experiment name, timestamp, model, backend/inference engine, hardware/accelerator, script path, environment/commit info.
|
||||||
|
- Record the **chip/accelerator** (e.g. `NVIDIA H200`, `Kunlun XPU`) and the **inference engine** (e.g. `vllm-dspark`, `sglang`, `vllm-xpu`) explicitly. Do not infer them from directory names.
|
||||||
- Per-scenario/per-configuration results: all request latencies, TTFT, TPOT, ITL, token counts, throughput, accept length, success/failure counts.
|
- Per-scenario/per-configuration results: all request latencies, TTFT, TPOT, ITL, token counts, throughput, accept length, success/failure counts.
|
||||||
- Include P50 / P90 / P95 / P99 percentiles where applicable.
|
- Include P50 / P90 / P95 / P99 percentiles where applicable.
|
||||||
- Keep the schema stable so downstream Python scripts can parse all experiments uniformly.
|
- Keep the schema stable so downstream Python scripts can parse all experiments uniformly.
|
||||||
@ -57,6 +58,11 @@
|
|||||||
- `logs/<service>_<timestamp>.log`
|
- `logs/<service>_<timestamp>.log`
|
||||||
- Keep server logs separate from benchmark result logs.
|
- Keep server logs separate from benchmark result logs.
|
||||||
|
|
||||||
|
7. **Scripts and outputs must record chip/accelerator and inference engine.**
|
||||||
|
- Every benchmark script should capture or accept the platform and engine it is running on (e.g. via environment variables `CHIP`, `ACCELERATOR`, `ENGINE`, `BACKEND`, or auto-detection).
|
||||||
|
- Final reports and JSON outputs must include both the accelerator/chip family and the inference engine/backend used for the run.
|
||||||
|
- Do not rely on the experiment name alone to identify the platform or engine.
|
||||||
|
|
||||||
## Naming Conventions
|
## Naming Conventions
|
||||||
|
|
||||||
### Result directories
|
### Result directories
|
||||||
@ -70,18 +76,27 @@ Examples:
|
|||||||
- `bench_results/dspark_grid_20260707-132641/`
|
- `bench_results/dspark_grid_20260707-132641/`
|
||||||
- `bench_results/dsv4_backend_comparison_20260707/`
|
- `bench_results/dsv4_backend_comparison_20260707/`
|
||||||
|
|
||||||
|
When the same experiment is repeated across chips or engines, include them in the directory name or organize by subdirectories so results are not confused:
|
||||||
|
|
||||||
|
```
|
||||||
|
bench_results/<experiment>_<chip>_<engine>_<YYYYMMDD-HHMMSS>/
|
||||||
|
bench_results/<experiment>/<chip>/<engine>/<YYYYMMDD-HHMMSS>/
|
||||||
|
```
|
||||||
|
|
||||||
### Raw output files
|
### Raw output files
|
||||||
|
|
||||||
|
Include the accelerator and inference engine in raw output filenames so files from different platforms cannot overwrite each other.
|
||||||
|
|
||||||
For detailed per-request JSONL outputs:
|
For detailed per-request JSONL outputs:
|
||||||
|
|
||||||
```
|
```
|
||||||
{backend}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
|
{chip}_{engine}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
|
||||||
```
|
```
|
||||||
|
|
||||||
For summary JSON outputs from `sglang.bench_serving --output-file`:
|
For summary JSON outputs from `sglang.bench_serving --output-file`:
|
||||||
|
|
||||||
```
|
```
|
||||||
{backend}_{scenario}_{params}.json
|
{chip}_{engine}_{scenario}_{params}.json
|
||||||
```
|
```
|
||||||
|
|
||||||
### Logs
|
### Logs
|
||||||
@ -105,8 +120,11 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
|
|||||||
"timestamp": "2026-07-07T13:26:41+08:00",
|
"timestamp": "2026-07-07T13:26:41+08:00",
|
||||||
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||||
"backend": "vllm-dspark",
|
"backend": "vllm-dspark",
|
||||||
"script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh",
|
"engine": "vllm-dspark",
|
||||||
"hardware": "8x NVIDIA H200 143GB",
|
"hardware": "8x NVIDIA H200 143GB",
|
||||||
|
"accelerator": "NVIDIA H200",
|
||||||
|
"chip": "NVIDIA H200",
|
||||||
|
"script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh",
|
||||||
"env": "/data/user1/yy/envs/vllm-dspark",
|
"env": "/data/user1/yy/envs/vllm-dspark",
|
||||||
"git_commit": "optional git sha",
|
"git_commit": "optional git sha",
|
||||||
"description": "optional free-text note"
|
"description": "optional free-text note"
|
||||||
@ -163,6 +181,11 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
|
|||||||
- `raw_requests` is optional but recommended when the JSON size is manageable. If a single run produces millions of requests, store per-request data as `raw_outputs/*.jsonl` and keep only aggregated percentiles in `results.json`.
|
- `raw_requests` is optional but recommended when the JSON size is manageable. If a single run produces millions of requests, store per-request data as `raw_outputs/*.jsonl` and keep only aggregated percentiles in `results.json`.
|
||||||
- Always include **P50 / P90 / P95 / P99** for TTFT, TPOT, E2E, and ITL. P95 is the primary SLO metric.
|
- Always include **P50 / P90 / P95 / P99** for TTFT, TPOT, E2E, and ITL. P95 is the primary SLO metric.
|
||||||
- Keep field names snake_case and consistent across experiments.
|
- Keep field names snake_case and consistent across experiments.
|
||||||
|
- Record hardware and engine information explicitly:
|
||||||
|
- `accelerator` / `chip`: the accelerator family, e.g. `NVIDIA H200`, `Kunlun XPU`, `AMD MI300X`.
|
||||||
|
- `engine` / `backend`: the inference engine or serving backend, e.g. `vllm-dspark`, `sglang`, `vllm-xpu`.
|
||||||
|
- `hardware`: a human-readable full hardware description, e.g. `8x NVIDIA H200 143GB`, `8x Kunlun XPU R480`.
|
||||||
|
- Keep at least one of `accelerator` or `chip`, and at least one of `engine` or `backend`, populated in every run.
|
||||||
- If a metric is not applicable (e.g. `accept_length` for non-speculative decoding), set it to `null` rather than omitting the key.
|
- If a metric is not applicable (e.g. `accept_length` for non-speculative decoding), set it to `null` rather than omitting the key.
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
@ -199,6 +222,8 @@ bash scripts/benchmark_dsv4_backend_comparison.sh all
|
|||||||
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
|
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
|
||||||
- [ ] `bench_results/<run>/report.md` (or equivalent human-readable `.md`) exists.
|
- [ ] `bench_results/<run>/report.md` (or equivalent human-readable `.md`) exists.
|
||||||
- [ ] `bench_results/<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
|
- [ ] `bench_results/<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
|
||||||
|
- [ ] `bench_results/<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
|
||||||
|
- [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide.
|
||||||
- [ ] `bench_results/<run>/README.md` exists and documents provenance (or the report itself covers provenance).
|
- [ ] `bench_results/<run>/README.md` exists and documents provenance (or the report itself covers provenance).
|
||||||
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
|
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
|
||||||
- [ ] Script path references updated after moving.
|
- [ ] Script path references updated after moving.
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user