docs: record chip/accelerator and inference engine in benchmark workflow
Update BENCHMARK_WORKFLOW.md to require scripts and final outputs to capture the hardware platform (e.g. NVIDIA H200, Kunlun XPU) and the inference engine/backend (e.g. vllm-dspark, sglang, vllm-xpu). This includes directory/file naming conventions, the results.json metadata schema, and the pre-archive checklist.
This commit is contained in:
parent
7de88b995a
commit
2c332ff712
@ -43,7 +43,8 @@
|
||||
- The directory may also contain a `README.md` documenting provenance if the report alone does not cover it.
|
||||
|
||||
4. **Final JSON must contain raw/structured data, not just summary numbers.**
|
||||
- Metadata: experiment name, timestamp, model, backend, hardware, script path, environment/commit info.
|
||||
- Metadata: experiment name, timestamp, model, backend/inference engine, hardware/accelerator, script path, environment/commit info.
|
||||
- Record the **chip/accelerator** (e.g. `NVIDIA H200`, `Kunlun XPU`) and the **inference engine** (e.g. `vllm-dspark`, `sglang`, `vllm-xpu`) explicitly. Do not infer them from directory names.
|
||||
- Per-scenario/per-configuration results: all request latencies, TTFT, TPOT, ITL, token counts, throughput, accept length, success/failure counts.
|
||||
- Include P50 / P90 / P95 / P99 percentiles where applicable.
|
||||
- Keep the schema stable so downstream Python scripts can parse all experiments uniformly.
|
||||
@ -57,6 +58,11 @@
|
||||
- `logs/<service>_<timestamp>.log`
|
||||
- Keep server logs separate from benchmark result logs.
|
||||
|
||||
7. **Scripts and outputs must record chip/accelerator and inference engine.**
|
||||
- Every benchmark script should capture or accept the platform and engine it is running on (e.g. via environment variables `CHIP`, `ACCELERATOR`, `ENGINE`, `BACKEND`, or auto-detection).
|
||||
- Final reports and JSON outputs must include both the accelerator/chip family and the inference engine/backend used for the run.
|
||||
- Do not rely on the experiment name alone to identify the platform or engine.
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
### Result directories
|
||||
@ -70,18 +76,27 @@ Examples:
|
||||
- `bench_results/dspark_grid_20260707-132641/`
|
||||
- `bench_results/dsv4_backend_comparison_20260707/`
|
||||
|
||||
When the same experiment is repeated across chips or engines, include them in the directory name or organize by subdirectories so results are not confused:
|
||||
|
||||
```
|
||||
bench_results/<experiment>_<chip>_<engine>_<YYYYMMDD-HHMMSS>/
|
||||
bench_results/<experiment>/<chip>/<engine>/<YYYYMMDD-HHMMSS>/
|
||||
```
|
||||
|
||||
### Raw output files
|
||||
|
||||
Include the accelerator and inference engine in raw output filenames so files from different platforms cannot overwrite each other.
|
||||
|
||||
For detailed per-request JSONL outputs:
|
||||
|
||||
```
|
||||
{backend}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
|
||||
{chip}_{engine}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
|
||||
```
|
||||
|
||||
For summary JSON outputs from `sglang.bench_serving --output-file`:
|
||||
|
||||
```
|
||||
{backend}_{scenario}_{params}.json
|
||||
{chip}_{engine}_{scenario}_{params}.json
|
||||
```
|
||||
|
||||
### Logs
|
||||
@ -105,8 +120,11 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
|
||||
"timestamp": "2026-07-07T13:26:41+08:00",
|
||||
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
|
||||
"backend": "vllm-dspark",
|
||||
"script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh",
|
||||
"engine": "vllm-dspark",
|
||||
"hardware": "8x NVIDIA H200 143GB",
|
||||
"accelerator": "NVIDIA H200",
|
||||
"chip": "NVIDIA H200",
|
||||
"script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh",
|
||||
"env": "/data/user1/yy/envs/vllm-dspark",
|
||||
"git_commit": "optional git sha",
|
||||
"description": "optional free-text note"
|
||||
@ -163,6 +181,11 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
|
||||
- `raw_requests` is optional but recommended when the JSON size is manageable. If a single run produces millions of requests, store per-request data as `raw_outputs/*.jsonl` and keep only aggregated percentiles in `results.json`.
|
||||
- Always include **P50 / P90 / P95 / P99** for TTFT, TPOT, E2E, and ITL. P95 is the primary SLO metric.
|
||||
- Keep field names snake_case and consistent across experiments.
|
||||
- Record hardware and engine information explicitly:
|
||||
- `accelerator` / `chip`: the accelerator family, e.g. `NVIDIA H200`, `Kunlun XPU`, `AMD MI300X`.
|
||||
- `engine` / `backend`: the inference engine or serving backend, e.g. `vllm-dspark`, `sglang`, `vllm-xpu`.
|
||||
- `hardware`: a human-readable full hardware description, e.g. `8x NVIDIA H200 143GB`, `8x Kunlun XPU R480`.
|
||||
- Keep at least one of `accelerator` or `chip`, and at least one of `engine` or `backend`, populated in every run.
|
||||
- If a metric is not applicable (e.g. `accept_length` for non-speculative decoding), set it to `null` rather than omitting the key.
|
||||
|
||||
## Quick Start
|
||||
@ -199,6 +222,8 @@ bash scripts/benchmark_dsv4_backend_comparison.sh all
|
||||
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
|
||||
- [ ] `bench_results/<run>/report.md` (or equivalent human-readable `.md`) exists.
|
||||
- [ ] `bench_results/<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
|
||||
- [ ] `bench_results/<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
|
||||
- [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide.
|
||||
- [ ] `bench_results/<run>/README.md` exists and documents provenance (or the report itself covers provenance).
|
||||
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
|
||||
- [ ] Script path references updated after moving.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user