diff --git a/BENCHMARK_WORKFLOW.md b/BENCHMARK_WORKFLOW.md index c1e8df2..b49e522 100644 --- a/BENCHMARK_WORKFLOW.md +++ b/BENCHMARK_WORKFLOW.md @@ -216,7 +216,7 @@ The JSON file inside each experiment run directory (`experiments//.env` present) and you only want to test a new model/config on a machine. + +**0. Machine setup (one-time, only on a brand-new machine)** + +```bash +git clone https://git.meta-stone.net/qqtang/sskj.git /data1/yy/sskj +cd /data1/yy/sskj +git config credential.helper store # push once to save the token +# Verify platform auto-detection: +bash -c 'source scripts/common/platform.sh && echo "$PLATFORM | $HARDWARE | $ENGINE"' +# Ensure platform deps exist (see platforms/.env): docker image pulled, MODEL_ROOT has the model, datasets/ present. +``` + +If the platform is not auto-detected, run with `PLATFORM= ...`. Platform-wide paths/images live in `platforms/.env` - edit that file (not the experiment) if a path differs on this machine. + +**1. Create the experiment from TEMPLATE** + +```bash +cd /data1/yy/sskj +cp -r experiments/TEMPLATE experiments/p800/ # e.g. qwen3_8b_p800_sglang_tp4 +cd experiments/p800/ +``` + +**2. Edit `config.env`** - the single source of experiment params (model, TP, ports, scenarios). All values use `${VAR:-default}`, so they can be overridden by env vars at run time. + +**3. Edit `start_sglang.sh` / `start_server.sh`** - the deployment script (`docker run` + launch flags). This is where you change deployment params (TP, mem-fraction, attention backend, cuda graph, ...). Reference `config.env` variables (`$TP`, `$MEM_FRACTION_STATIC`, ...) rather than hardcoding. + +**4. Run** + +```bash +bash run_bench.sh # auto-detects platform +# or with overrides / explicit RUN_ID: +PLATFORM=kunlun_p800 RUN_ID=tp4_20260722 bash run_bench.sh +# quick config probe without editing files: +TP=4 MEM_FRACTION_STATIC=0.85 MAX_RUNNING=32 bash run_bench.sh +``` + +Results land in `results//` (`report.md`, `results.json`; `raw_outputs/` and `logs/` are gitignored). Watch `logs/orchestrator.log` and `logs/server.log`. + +**5. Commit code + final artifacts, push, open PR** + +```bash +cd /data1/yy/sskj +git checkout -b feat/p800- +git add experiments/p800//config.env \ + experiments/p800//start_sglang.sh \ + experiments/p800//run_bench.sh \ + experiments/p800//README.md \ + experiments/p800//results//report.md \ + experiments/p800//results//results.json +git commit -m "feat(p800): add experiment" +git push -u origin feat/p800- +``` + +Do NOT `git add` `raw_outputs/` or `logs/` (already ignored). Run through the [Checklist](#checklist-before-committing--archiving) below before pushing. + +> Adding a brand-new chip (not just a new experiment on an existing platform)? Follow steps 1-4 below instead. + + ### 1. Add or update a platform config Create `platforms/.env` with identity and platform-wide paths: