diff --git a/BENCHMARK_WORKFLOW.md b/BENCHMARK_WORKFLOW.md index f97606e..c9bb135 100644 --- a/BENCHMARK_WORKFLOW.md +++ b/BENCHMARK_WORKFLOW.md @@ -43,7 +43,8 @@ - The directory may also contain a `README.md` documenting provenance if the report alone does not cover it. 4. **Final JSON must contain raw/structured data, not just summary numbers.** - - Metadata: experiment name, timestamp, model, backend, hardware, script path, environment/commit info. + - Metadata: experiment name, timestamp, model, backend/inference engine, hardware/accelerator, script path, environment/commit info. + - Record the **chip/accelerator** (e.g. `NVIDIA H200`, `Kunlun XPU`) and the **inference engine** (e.g. `vllm-dspark`, `sglang`, `vllm-xpu`) explicitly. Do not infer them from directory names. - Per-scenario/per-configuration results: all request latencies, TTFT, TPOT, ITL, token counts, throughput, accept length, success/failure counts. - Include P50 / P90 / P95 / P99 percentiles where applicable. - Keep the schema stable so downstream Python scripts can parse all experiments uniformly. @@ -57,6 +58,11 @@ - `logs/_.log` - Keep server logs separate from benchmark result logs. +7. **Scripts and outputs must record chip/accelerator and inference engine.** + - Every benchmark script should capture or accept the platform and engine it is running on (e.g. via environment variables `CHIP`, `ACCELERATOR`, `ENGINE`, `BACKEND`, or auto-detection). + - Final reports and JSON outputs must include both the accelerator/chip family and the inference engine/backend used for the run. + - Do not rely on the experiment name alone to identify the platform or engine. + ## Naming Conventions ### Result directories @@ -70,18 +76,27 @@ Examples: - `bench_results/dspark_grid_20260707-132641/` - `bench_results/dsv4_backend_comparison_20260707/` +When the same experiment is repeated across chips or engines, include them in the directory name or organize by subdirectories so results are not confused: + +``` +bench_results/___/ +bench_results///// +``` + ### Raw output files +Include the accelerator and inference engine in raw output filenames so files from different platforms cannot overwrite each other. + For detailed per-request JSONL outputs: ``` -{backend}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl +{chip}_{engine}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl ``` For summary JSON outputs from `sglang.bench_serving --output-file`: ``` -{backend}_{scenario}_{params}.json +{chip}_{engine}_{scenario}_{params}.json ``` ### Logs @@ -105,8 +120,11 @@ The JSON file inside each `bench_results//` directory should follow a stabl "timestamp": "2026-07-07T13:26:41+08:00", "model": "/data/models/DeepSeek-V4-Flash-DSpark", "backend": "vllm-dspark", - "script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh", + "engine": "vllm-dspark", "hardware": "8x NVIDIA H200 143GB", + "accelerator": "NVIDIA H200", + "chip": "NVIDIA H200", + "script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh", "env": "/data/user1/yy/envs/vllm-dspark", "git_commit": "optional git sha", "description": "optional free-text note" @@ -163,6 +181,11 @@ The JSON file inside each `bench_results//` directory should follow a stabl - `raw_requests` is optional but recommended when the JSON size is manageable. If a single run produces millions of requests, store per-request data as `raw_outputs/*.jsonl` and keep only aggregated percentiles in `results.json`. - Always include **P50 / P90 / P95 / P99** for TTFT, TPOT, E2E, and ITL. P95 is the primary SLO metric. - Keep field names snake_case and consistent across experiments. +- Record hardware and engine information explicitly: + - `accelerator` / `chip`: the accelerator family, e.g. `NVIDIA H200`, `Kunlun XPU`, `AMD MI300X`. + - `engine` / `backend`: the inference engine or serving backend, e.g. `vllm-dspark`, `sglang`, `vllm-xpu`. + - `hardware`: a human-readable full hardware description, e.g. `8x NVIDIA H200 143GB`, `8x Kunlun XPU R480`. + - Keep at least one of `accelerator` or `chip`, and at least one of `engine` or `backend`, populated in every run. - If a metric is not applicable (e.g. `accept_length` for non-speculative decoding), set it to `null` rather than omitting the key. ## Quick Start @@ -199,6 +222,8 @@ bash scripts/benchmark_dsv4_backend_comparison.sh all - [ ] All outputs moved to `bench_results/_/`. - [ ] `bench_results//report.md` (or equivalent human-readable `.md`) exists. - [ ] `bench_results//results.json` exists and follows the [Final JSON Schema](#final-json-schema). +- [ ] `bench_results//results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used. +- [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide. - [ ] `bench_results//README.md` exists and documents provenance (or the report itself covers provenance). - [ ] Scripts moved to `scripts/` (or `scripts//`). - [ ] Script path references updated after moving.