sskj/BENCHMARK_WORKFLOW.md

Benchmark Workflow & Directory Conventions

Directory Layout

/data/user1/yy/
├── scripts/                    # all benchmark/orchestrator/utility scripts
│   ├── benchmark_dspark_0707/  # DSpark benchmark suite
│   ├── benchmark_dsv4_backend_comparison.sh
│   ├── start_dsv4_dspark_8card.sh
│   ├── start_sglang_dsv4_8card.sh
│   └── ...
├── bench_results/              # all benchmark outputs and reports
│   ├── dsv4_backend_comparison_20260707/
│   │   ├── raw_outputs/        # JSONL raw outputs
│   │   ├── logs/               # per-run logs
│   │   └── README.md           # output manifest + provenance
│   ├── dspark_grid_20260707-132641/
│   ├── dspark_st_comparison_20260707-150649/
│   ├── eagle_grid/
│   └── ...
├── logs/                       # server logs (stdout/stderr from start scripts)
├── datasets/                   # benchmark datasets
└── envs/                       # Python virtual environments

Rules

1. Scripts live in scripts/ only.

   • Group related scripts into subdirectories, e.g. scripts/benchmark_dspark_0707/.
   • Each script group should have its own README.md listing scripts, purpose, and outputs.

2. Benchmark outputs live in bench_results/ only.

   • Never leave .jsonl, .json, .log, or .md reports in the project root.
   • Each benchmark run gets its own directory: bench_results/<experiment>_<timestamp>/.
   • Raw outputs go in raw_outputs/.
   • Logs go in logs/.
   • Reports (e.g. report.md, comparison_report.md) go in the run root.

3. Each bench_results/<run>/ directory must contain a README.md.

   • What was benchmarked.
   • Which script(s) produced the outputs.
   • File naming convention.
   • Inventory of outputs (table).

4. Scripts should default RESULT_ROOT to bench_results/<experiment>_${RUN_ID}.

   • Allow override via RESULT_ROOT env var.
   • Use RUN_ID=$(date '+%Y%m%d-%H%M%S') unless specified.

5. Server start scripts write to logs/.

   • logs/<service>_<timestamp>.log
   • Keep server logs separate from benchmark result logs.

Naming Conventions

Result directories

bench_results/<experiment>_<YYYYMMDD-HHMMSS>/

Examples:

• bench_results/dspark_grid_20260707-132641/
• bench_results/dsv4_backend_comparison_20260707/

Raw output files

For detailed per-request JSONL outputs:

{backend}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl

For summary JSON outputs from sglang.bench_serving --output-file:

{backend}_{scenario}_{params}.json

Logs

logs/<service>_YYYYMMDD_HHMMSS.log
logs/<experiment>_orchestrator_YYYYMMDD_HHMMSS.log

Quick Start

Run DSpark grid benchmark

bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh

Run DSpark spec-tokens comparison

bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh

Run SGLang vs vLLM backend comparison

# Start SGLang on port 30000 and vLLM on port 8000, then:
bash scripts/benchmark_dsv4_backend_comparison.sh all

Parse results

/data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
  /data/user1/yy/bench_results/dspark_grid_<run_id>

Checklist Before Committing / Archiving

• No .jsonl, .json, .log, or .md files left in /data/user1/yy/ root.
• All outputs moved to bench_results/<experiment>_<timestamp>/.
• bench_results/<run>/README.md exists and documents provenance.
• Scripts moved to scripts/ (or scripts/<group>/).
• Script path references updated after moving.