qqtang/sskj
1
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add new-experiment quick start and forbid raw_requests in results.json

Browse Source
This commit is contained in:
yy-fighting 2026-07-22 04:08:51 +00:00
parent 3b0297516e
commit c234468cd6
1 changed files with 62 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

63
BENCHMARK_WORKFLOW.md
View File

@ -216,7 +216,7 @@ The JSON file inside each experiment run directory (`experiments/<platform>/<exp
### Notes
- `raw_requests` is optional but recommended when the JSON size is manageable. If a single run produces millions of requests, store per-request data as `raw_outputs/*.jsonl` and keep only aggregated percentiles in `results.json`.
- **Do NOT embed `raw_requests` in `results.json`.** Per-request data must live only in `raw_outputs/*.jsonl` (gitignored). `results.json` must stay small (metadata + config + per-scenario summary metrics/percentiles only). Embedding raw requests caused multi-MB `results.json` bloat historically and is no longer done by `scripts/common/parse_backend.py`.
- Always include **P50 / P90 / P95 / P99** for TTFT, TPOT, E2E, and ITL. P95 is the primary SLO metric.
- Include an `slo_status` object per scenario indicating whether the scenario meets the relevant SLO (e.g. S2 tier: TTFT P95 < 3000ms, TPOT mean < 50ms). Example:
```json
@ -260,6 +260,67 @@ python3 experiments/p800/dsv4_p800_sglang/parse_results.py \
## Adding a New Platform or Experiment
### Quick Start: New Experiment on an Existing Platform (e.g. a new P800 box)
Use this when the platform already exists (`platforms/<chip>.env` present) and you only want to test a new model/config on a machine.
**0. Machine setup (one-time, only on a brand-new machine)**
```bash
git clone https://git.meta-stone.net/qqtang/sskj.git /data1/yy/sskj
cd /data1/yy/sskj
git config credential.helper store # push once to save the token
# Verify platform auto-detection:
bash -c 'source scripts/common/platform.sh && echo "$PLATFORM | $HARDWARE | $ENGINE"'
# Ensure platform deps exist (see platforms/<chip>.env): docker image pulled, MODEL_ROOT has the model, datasets/ present.
```
If the platform is not auto-detected, run with `PLATFORM=<chip> ...`. Platform-wide paths/images live in `platforms/<chip>.env` - edit that file (not the experiment) if a path differs on this machine.
**1. Create the experiment from TEMPLATE**
```bash
cd /data1/yy/sskj
cp -r experiments/TEMPLATE experiments/p800/<your_experiment_name> # e.g. qwen3_8b_p800_sglang_tp4
cd experiments/p800/<your_experiment_name>
```
**2. Edit `config.env`** - the single source of experiment params (model, TP, ports, scenarios). All values use `${VAR:-default}`, so they can be overridden by env vars at run time.
**3. Edit `start_sglang.sh` / `start_server.sh`** - the deployment script (`docker run` + launch flags). This is where you change deployment params (TP, mem-fraction, attention backend, cuda graph, ...). Reference `config.env` variables (`$TP`, `$MEM_FRACTION_STATIC`, ...) rather than hardcoding.
**4. Run**
```bash
bash run_bench.sh # auto-detects platform
# or with overrides / explicit RUN_ID:
PLATFORM=kunlun_p800 RUN_ID=tp4_20260722 bash run_bench.sh
# quick config probe without editing files:
TP=4 MEM_FRACTION_STATIC=0.85 MAX_RUNNING=32 bash run_bench.sh
```
Results land in `results/<RUN_ID>/` (`report.md`, `results.json`; `raw_outputs/` and `logs/` are gitignored). Watch `logs/orchestrator.log` and `logs/server.log`.
**5. Commit code + final artifacts, push, open PR**
```bash
cd /data1/yy/sskj
git checkout -b feat/p800-<your_experiment_name>
git add experiments/p800/<your_experiment_name>/config.env \
experiments/p800/<your_experiment_name>/start_sglang.sh \
experiments/p800/<your_experiment_name>/run_bench.sh \
experiments/p800/<your_experiment_name>/README.md \
experiments/p800/<your_experiment_name>/results/<RUN_ID>/report.md \
experiments/p800/<your_experiment_name>/results/<RUN_ID>/results.json
git commit -m "feat(p800): add <your_experiment_name> experiment"
git push -u origin feat/p800-<your_experiment_name>
```
Do NOT `git add` `raw_outputs/` or `logs/` (already ignored). Run through the [Checklist](#checklist-before-committing--archiving) below before pushing.
> Adding a brand-new chip (not just a new experiment on an existing platform)? Follow steps 1-4 below instead.
### 1. Add or update a platform config
Create `platforms/<chip>.env` with identity and platform-wide paths: