diff --git a/BENCHMARK_WORKFLOW.md b/BENCHMARK_WORKFLOW.md index 6a2f352..0553597 100644 --- a/BENCHMARK_WORKFLOW.md +++ b/BENCHMARK_WORKFLOW.md @@ -79,6 +79,11 @@ - Scripts load the platform file via `scripts/common/platform.sh`; the active platform is selected by the `PLATFORM` env var or auto-detected. - Keep experiment scripts free of hardcoded device IDs, image names, or model root paths. +9. **Record the exact server launch command/args for every run.** + - The exact command or full argument list used to start the server must be saved in `results.json` under `config.server_args` (or `config.phaseN_server_args` if the experiment starts the server in multiple phases). + - This is required for cross-platform reproduction: when the same experiment is run on H200 and P800, the only differences should be model paths, ports, and device IDs. + - If an experiment uses `start_server.sh`, that script should be self-contained and its command line should be reproducible from `results.json` alone. + ## Naming Conventions ### Experiment result directories @@ -164,7 +169,8 @@ The JSON file inside each `bench_results//` directory should follow a stabl "spec_tokens": 5, "block_size": 256, "max_num_seqs": 256, - "extra_args": "--no-disable-hybrid-kv-cache-manager" + "extra_args": "--no-disable-hybrid-kv-cache-manager", + "server_args": "vllm serve /data/models/DeepSeek-V4-Flash-DSpark --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len auto --max-num-seqs 256 --spec-method dspark --spec-tokens 5 --no-disable-hybrid-kv-cache-manager --port 30004" }, "scenarios": [ { @@ -297,8 +303,9 @@ At minimum, `run_bench.sh` should: 2. Read `config.env`. 3. Create `experiments//results//`. 4. Call `write_metadata_json` to create `results.json`. -5. Run the benchmark scenarios. -6. Call `parse_results.py` to generate `report.md`. +5. Record the exact server launch command/args in `results.json` `config.server_args` (or `phaseN_server_args`). +6. Run the benchmark scenarios. +7. Call `parse_results.py` to generate `report.md`. ### 3. Reuse shared helpers @@ -325,6 +332,7 @@ See `docs/H200_QUICKSTART.md` for a concrete H200 porting walkthrough. - [ ] `/report.md` (or equivalent human-readable `.md`) exists. - [ ] `/results.json` exists and follows the [Final JSON Schema](#final-json-schema). - [ ] `/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used. +- [ ] `/results.json` `config` records the exact server launch command/args (`server_args` or `phaseN_server_args`). - [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide. - [ ] `/README.md` exists and documents provenance (or the report itself covers provenance). - [ ] Scripts either live under `experiments//` or in `scripts/` (or `scripts//`).