qqtang/sskj
1
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
sskj/BENCHMARK_WORKFLOW.md
Quantong Qiu 2c332ff712 docs: record chip/accelerator and inference engine in benchmark workflow 
Update BENCHMARK_WORKFLOW.md to require scripts and final outputs to
capture the hardware platform (e.g. NVIDIA H200, Kunlun XPU) and the
inference engine/backend (e.g. vllm-dspark, sglang, vllm-xpu). This
includes directory/file naming conventions, the results.json metadata
schema, and the pre-archive checklist.
2026-07-08 03:52:23 +00:00

230 lines
9.1 KiB
Markdown
Raw Blame History

# Benchmark Workflow & Directory Conventions
## Directory Layout
```
/data/user1/yy/
├── scripts/ # all benchmark/orchestrator/utility scripts
│ ├── benchmark_dspark_0707/ # DSpark benchmark suite
│ ├── benchmark_dsv4_backend_comparison.sh
│ ├── start_dsv4_dspark_8card.sh
│ ├── start_sglang_dsv4_8card.sh
│ └── ...
├── bench_results/ # all benchmark outputs and reports
│ ├── dsv4_backend_comparison_20260707/
│ │ ├── raw_outputs/ # JSONL raw outputs
│ │ ├── logs/ # per-run logs
│ │ └── README.md # output manifest + provenance
│ ├── dspark_grid_20260707-132641/
│ ├── dspark_st_comparison_20260707-150649/
│ ├── eagle_grid/
│ └── ...
├── logs/ # server logs (stdout/stderr from start scripts)
├── datasets/ # benchmark datasets
└── envs/ # Python virtual environments
```
## Rules
1. **Scripts live in `scripts/` only.**
- Group related scripts into subdirectories, e.g. `scripts/benchmark_dspark_0707/`.
- Each script group should have its own `README.md` listing scripts, purpose, and outputs.
2. **Benchmark outputs live in `bench_results/` only.**
- Never leave `.jsonl`, `.json`, `.log`, or `.md` reports in the project root.
- Each benchmark run gets its own directory: `bench_results/<experiment>_<timestamp>/`.
- Raw outputs go in `raw_outputs/`.
- Logs go in `logs/`.
- Reports (e.g. `report.md`, `comparison_report.md`) go in the run root.
3. **Each `bench_results/<run>/` directory must contain two final artifacts.**
- A Markdown report for human reading (e.g. `report.md`, `comparison_report.md`).
- A JSON file with the complete structured result data for programmatic analysis (e.g. `results.json`).
- The directory may also contain a `README.md` documenting provenance if the report alone does not cover it.
4. **Final JSON must contain raw/structured data, not just summary numbers.**
- Metadata: experiment name, timestamp, model, backend/inference engine, hardware/accelerator, script path, environment/commit info.
- Record the **chip/accelerator** (e.g. `NVIDIA H200`, `Kunlun XPU`) and the **inference engine** (e.g. `vllm-dspark`, `sglang`, `vllm-xpu`) explicitly. Do not infer them from directory names.
- Per-scenario/per-configuration results: all request latencies, TTFT, TPOT, ITL, token counts, throughput, accept length, success/failure counts.
- Include P50 / P90 / P95 / P99 percentiles where applicable.
- Keep the schema stable so downstream Python scripts can parse all experiments uniformly.
- See [Final JSON Schema](#final-json-schema) below for the recommended structure.
5. **Scripts should default `RESULT_ROOT` to `bench_results/<experiment>_${RUN_ID}`.**
- Allow override via `RESULT_ROOT` env var.
- Use `RUN_ID=$(date '+%Y%m%d-%H%M%S')` unless specified.
6. **Server start scripts write to `logs/`.**
- `logs/<service>_<timestamp>.log`
- Keep server logs separate from benchmark result logs.
7. **Scripts and outputs must record chip/accelerator and inference engine.**
- Every benchmark script should capture or accept the platform and engine it is running on (e.g. via environment variables `CHIP`, `ACCELERATOR`, `ENGINE`, `BACKEND`, or auto-detection).
- Final reports and JSON outputs must include both the accelerator/chip family and the inference engine/backend used for the run.
- Do not rely on the experiment name alone to identify the platform or engine.
## Naming Conventions
### Result directories
```
bench_results/<experiment>_<YYYYMMDD-HHMMSS>/
```
Examples:
- `bench_results/dspark_grid_20260707-132641/`
- `bench_results/dsv4_backend_comparison_20260707/`
When the same experiment is repeated across chips or engines, include them in the directory name or organize by subdirectories so results are not confused:
```
bench_results/<experiment>_<chip>_<engine>_<YYYYMMDD-HHMMSS>/
bench_results/<experiment>/<chip>/<engine>/<YYYYMMDD-HHMMSS>/
```
### Raw output files
Include the accelerator and inference engine in raw output filenames so files from different platforms cannot overwrite each other.
For detailed per-request JSONL outputs:
```
{chip}_{engine}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
```
For summary JSON outputs from `sglang.bench_serving --output-file`:
```
{chip}_{engine}_{scenario}_{params}.json
```
### Logs
```
logs/<service>_YYYYMMDD_HHMMSS.log
logs/<experiment>_orchestrator_YYYYMMDD_HHMMSS.log
```
## Final JSON Schema
The JSON file inside each `bench_results/<run>/` directory should follow a stable schema so that downstream Python scripts can load every experiment the same way. The file is usually named `results.json`.
### Required top-level fields
```json
{
"metadata": {
"experiment": "dspark_grid",
"run_id": "20260707-132641",
"timestamp": "2026-07-07T13:26:41+08:00",
"model": "/data/models/DeepSeek-V4-Flash-DSpark",
"backend": "vllm-dspark",
"engine": "vllm-dspark",
"hardware": "8x NVIDIA H200 143GB",
"accelerator": "NVIDIA H200",
"chip": "NVIDIA H200",
"script": "scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh",
"env": "/data/user1/yy/envs/vllm-dspark",
"git_commit": "optional git sha",
"description": "optional free-text note"
},
"config": {
"tp": 8,
"kv_cache_dtype": "fp8",
"spec_method": "dspark",
"spec_tokens": 5,
"block_size": 256,
"max_num_seqs": 256,
"extra_args": "--no-disable-hybrid-kv-cache-manager"
},
"scenarios": [
{
"name": "chat_short",
"concurrency": 64,
"input_len": 1000,
"output_len": 256,
"duration_s": 21.27,
"success": 512,
"failed": 0,
"request_throughput": 24.07,
"input_token_throughput": 6391.14,
"output_token_throughput": 3189.39,
"total_token_throughput": 9580.52,
"accept_length": 3.2,
"latencies": {
"e2e_ms": { "mean": 2547.81, "p50": 2400.0, "p90": 4800.0, "p95": 5606.07, "p99": 6543.97 },
"ttft_ms": { "mean": 268.68, "p50": 240.0, "p90": 480.0, "p95": 543.21, "p99": 588.63 },
"tpot_ms": { "mean": 18.24, "p50": 16.0, "p90": 28.0, "p95": 30.39, "p99": 39.57 },
"itl_ms": { "mean": 70.29, "p50": 60.0, "p90": 110.0, "p95": 130.0, "p99": 160.0 }
},
"raw_requests": [
{
"request_id": "uuid-or-index",
"input_tokens": 1000,
"output_tokens": 256,
"e2e_ms": 2500.0,
"ttft_ms": 260.0,
"tpot_ms": 18.0,
"itl_ms": 70.0,
"accept_length": 3.0,
"success": true
}
]
}
]
}
```
### Notes
- `raw_requests` is optional but recommended when the JSON size is manageable. If a single run produces millions of requests, store per-request data as `raw_outputs/*.jsonl` and keep only aggregated percentiles in `results.json`.
- Always include **P50 / P90 / P95 / P99** for TTFT, TPOT, E2E, and ITL. P95 is the primary SLO metric.
- Keep field names snake_case and consistent across experiments.
- Record hardware and engine information explicitly:
- `accelerator` / `chip`: the accelerator family, e.g. `NVIDIA H200`, `Kunlun XPU`, `AMD MI300X`.
- `engine` / `backend`: the inference engine or serving backend, e.g. `vllm-dspark`, `sglang`, `vllm-xpu`.
- `hardware`: a human-readable full hardware description, e.g. `8x NVIDIA H200 143GB`, `8x Kunlun XPU R480`.
- Keep at least one of `accelerator` or `chip`, and at least one of `engine` or `backend`, populated in every run.
- If a metric is not applicable (e.g. `accept_length` for non-speculative decoding), set it to `null` rather than omitting the key.
## Quick Start
### Run DSpark grid benchmark
```bash
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
```
### Run DSpark spec-tokens comparison
```bash
bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
```
### Run SGLang vs vLLM backend comparison
```bash
# Start SGLang on port 30000 and vLLM on port 8000, then:
bash scripts/benchmark_dsv4_backend_comparison.sh all
```
### Parse results
```bash
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
/data/user1/yy/bench_results/dspark_grid_<run_id>
```
## Checklist Before Committing / Archiving
- [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in `/data/user1/yy/` root.
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
- [ ] `bench_results/<run>/report.md` (or equivalent human-readable `.md`) exists.
- [ ] `bench_results/<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
- [ ] `bench_results/<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
- [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide.
- [ ] `bench_results/<run>/README.md` exists and documents provenance (or the report itself covers provenance).
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
- [ ] Script path references updated after moving.
Reference in New Issue View Git Blame Copy Permalink