qqtang/sskj
1
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: require server launch args to be recorded in results.json

Browse Source 
- Add rule 9: exact server command/args must be saved in config.server_args
- Update Final JSON Schema example to include server_args
- Update checklist and new-experiment instructions
This commit is contained in:
yy-fighting 2026-07-08 08:22:33 +00:00
parent 162168da3c
commit b5b636236d
1 changed files with 11 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
BENCHMARK_WORKFLOW.md
View File

@ -79,6 +79,11 @@
- Scripts load the platform file via `scripts/common/platform.sh`; the active platform is selected by the `PLATFORM` env var or auto-detected. - Scripts load the platform file via `scripts/common/platform.sh`; the active platform is selected by the `PLATFORM` env var or auto-detected.
- Keep experiment scripts free of hardcoded device IDs, image names, or model root paths. - Keep experiment scripts free of hardcoded device IDs, image names, or model root paths.
9. **Record the exact server launch command/args for every run.**
- The exact command or full argument list used to start the server must be saved in `results.json` under `config.server_args` (or `config.phaseN_server_args` if the experiment starts the server in multiple phases).
- This is required for cross-platform reproduction: when the same experiment is run on H200 and P800, the only differences should be model paths, ports, and device IDs.
- If an experiment uses `start_server.sh`, that script should be self-contained and its command line should be reproducible from `results.json` alone.
## Naming Conventions ## Naming Conventions
### Experiment result directories ### Experiment result directories
@ -164,7 +169,8 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl
"spec_tokens": 5, "spec_tokens": 5,
"block_size": 256, "block_size": 256,
"max_num_seqs": 256, "max_num_seqs": 256,
"extra_args": "--no-disable-hybrid-kv-cache-manager" "extra_args": "--no-disable-hybrid-kv-cache-manager",
"server_args": "vllm serve /data/models/DeepSeek-V4-Flash-DSpark --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len auto --max-num-seqs 256 --spec-method dspark --spec-tokens 5 --no-disable-hybrid-kv-cache-manager --port 30004"
}, },
"scenarios": [ "scenarios": [
{ {
@ -297,8 +303,9 @@ At minimum, `run_bench.sh` should:
2. Read `config.env`. 2. Read `config.env`.
3. Create `experiments/<name>/results/<RUN_ID>/`. 3. Create `experiments/<name>/results/<RUN_ID>/`.
4. Call `write_metadata_json` to create `results.json`. 4. Call `write_metadata_json` to create `results.json`.
5. Run the benchmark scenarios. 5. Record the exact server launch command/args in `results.json` `config.server_args` (or `phaseN_server_args`).
6. Call `parse_results.py` to generate `report.md`. 6. Run the benchmark scenarios.
7. Call `parse_results.py` to generate `report.md`.
### 3. Reuse shared helpers ### 3. Reuse shared helpers
@ -325,6 +332,7 @@ See `docs/H200_QUICKSTART.md` for a concrete H200 porting walkthrough.
- [ ] `<run>/report.md` (or equivalent human-readable `.md`) exists. - [ ] `<run>/report.md` (or equivalent human-readable `.md`) exists.
- [ ] `<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema). - [ ] `<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
- [ ] `<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used. - [ ] `<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
- [ ] `<run>/results.json` `config` records the exact server launch command/args (`server_args` or `phaseN_server_args`).
- [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide. - [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide.
- [ ] `<run>/README.md` exists and documents provenance (or the report itself covers provenance). - [ ] `<run>/README.md` exists and documents provenance (or the report itself covers provenance).
- [ ] Scripts either live under `experiments/<name>/` or in `scripts/` (or `scripts/<group>/`). - [ ] Scripts either live under `experiments/<name>/` or in `scripts/` (or `scripts/<group>/`).