123 lines
3.7 KiB
Markdown
123 lines
3.7 KiB
Markdown
# Benchmark Workflow & Directory Conventions
|
|
|
|
## Directory Layout
|
|
|
|
```
|
|
/data/user1/yy/
|
|
├── scripts/ # all benchmark/orchestrator/utility scripts
|
|
│ ├── benchmark_dspark_0707/ # DSpark benchmark suite
|
|
│ ├── benchmark_dsv4_backend_comparison.sh
|
|
│ ├── start_dsv4_dspark_8card.sh
|
|
│ ├── start_sglang_dsv4_8card.sh
|
|
│ └── ...
|
|
├── bench_results/ # all benchmark outputs and reports
|
|
│ ├── dsv4_backend_comparison_20260707/
|
|
│ │ ├── raw_outputs/ # JSONL raw outputs
|
|
│ │ ├── logs/ # per-run logs
|
|
│ │ └── README.md # output manifest + provenance
|
|
│ ├── dspark_grid_20260707-132641/
|
|
│ ├── dspark_st_comparison_20260707-150649/
|
|
│ ├── eagle_grid/
|
|
│ └── ...
|
|
├── logs/ # server logs (stdout/stderr from start scripts)
|
|
├── datasets/ # benchmark datasets
|
|
└── envs/ # Python virtual environments
|
|
```
|
|
|
|
## Rules
|
|
|
|
1. **Scripts live in `scripts/` only.**
|
|
- Group related scripts into subdirectories, e.g. `scripts/benchmark_dspark_0707/`.
|
|
- Each script group should have its own `README.md` listing scripts, purpose, and outputs.
|
|
|
|
2. **Benchmark outputs live in `bench_results/` only.**
|
|
- Never leave `.jsonl`, `.json`, `.log`, or `.md` reports in the project root.
|
|
- Each benchmark run gets its own directory: `bench_results/<experiment>_<timestamp>/`.
|
|
- Raw outputs go in `raw_outputs/`.
|
|
- Logs go in `logs/`.
|
|
- Reports (e.g. `report.md`, `comparison_report.md`) go in the run root.
|
|
|
|
3. **Each `bench_results/<run>/` directory must contain a `README.md`.**
|
|
- What was benchmarked.
|
|
- Which script(s) produced the outputs.
|
|
- File naming convention.
|
|
- Inventory of outputs (table).
|
|
|
|
4. **Scripts should default `RESULT_ROOT` to `bench_results/<experiment>_${RUN_ID}`.**
|
|
- Allow override via `RESULT_ROOT` env var.
|
|
- Use `RUN_ID=$(date '+%Y%m%d-%H%M%S')` unless specified.
|
|
|
|
5. **Server start scripts write to `logs/`.**
|
|
- `logs/<service>_<timestamp>.log`
|
|
- Keep server logs separate from benchmark result logs.
|
|
|
|
## Naming Conventions
|
|
|
|
### Result directories
|
|
|
|
```
|
|
bench_results/<experiment>_<YYYYMMDD-HHMMSS>/
|
|
```
|
|
|
|
Examples:
|
|
|
|
- `bench_results/dspark_grid_20260707-132641/`
|
|
- `bench_results/dsv4_backend_comparison_20260707/`
|
|
|
|
### Raw output files
|
|
|
|
For detailed per-request JSONL outputs:
|
|
|
|
```
|
|
{backend}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
|
|
```
|
|
|
|
For summary JSON outputs from `sglang.bench_serving --output-file`:
|
|
|
|
```
|
|
{backend}_{scenario}_{params}.json
|
|
```
|
|
|
|
### Logs
|
|
|
|
```
|
|
logs/<service>_YYYYMMDD_HHMMSS.log
|
|
logs/<experiment>_orchestrator_YYYYMMDD_HHMMSS.log
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
### Run DSpark grid benchmark
|
|
|
|
```bash
|
|
bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
|
|
```
|
|
|
|
### Run DSpark spec-tokens comparison
|
|
|
|
```bash
|
|
bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
|
|
```
|
|
|
|
### Run SGLang vs vLLM backend comparison
|
|
|
|
```bash
|
|
# Start SGLang on port 30000 and vLLM on port 8000, then:
|
|
bash scripts/benchmark_dsv4_backend_comparison.sh all
|
|
```
|
|
|
|
### Parse results
|
|
|
|
```bash
|
|
/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
|
|
/data/user1/yy/bench_results/dspark_grid_<run_id>
|
|
```
|
|
|
|
## Checklist Before Committing / Archiving
|
|
|
|
- [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in `/data/user1/yy/` root.
|
|
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
|
|
- [ ] `bench_results/<run>/README.md` exists and documents provenance.
|
|
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
|
|
- [ ] Script path references updated after moving.
|