qqtang/sskj
1
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
53 Commits 1 Branch 0 Tags
Commit Graph

15 Commits

Author SHA1 Message Date
yy-fighting
1482601ae5 Add num_prompts = 5×concurrency convention and update all experiments 
- Document the rule in docs/EXPERIMENT_GUIDE.md section 7
- Update all config.env SCENARIOS to follow the 5x rule
- Migrate dsv4_h200_sglang, dsv4_h200_vllm, dsv4_p800_sglang
  from global NUM_PROMPTS to per-scenario num_prompts
- Keep long-context phase2 as exception (low counts documented)
- Add legacy 3-field fallback in run_bench.sh loops
 2026-07-09 09:26:41 +00:00
yy-fighting
80959cf765 docs: move SLO_STANDARDS.md from scripts/ to docs/; update references 2026-07-08 11:06:49 +00:00
yy-fighting
f51697aed6 refactor(common): remove duplicated parse/compare/warmup from existing experiments 2026-07-08 10:49:27 +00:00
yy-fighting
a127cf7ef0 refactor(common): centralize parse/compare/warmup; add TEMPLATE and EXPERIMENT_GUIDE 2026-07-08 10:48:43 +00:00
yy-fighting
ee954d7409 fix(sglang-vs-vllm): use jq for metadata update; add results run 20260708-083346 2026-07-08 09:21:49 +00:00
yy-fighting
5eb605b7a2 feat: support resuming interrupted SGLang vs vLLM runs 
- run_bench.sh: skip scenarios whose raw jsonl already exists and has expected completed count
- parse_backend.py: replace scenarios list instead of appending, so resume+reparse stays clean
- README.md: document RUN_ID-based resume
 2026-07-08 08:42:21 +00:00
yy-fighting
fd0474099a chore: do not commit partial SGLang vs vLLM results while run is in progress 2026-07-08 08:36:13 +00:00
yy-fighting
19d336de08 feat: add max context length exploration experiment 
- New experiments/dsv4_h200_max_context_length/
- start_sglang.sh / start_vllm.sh accept target length as argument
- run_bench.sh tests a ladder of input lengths (default 64k -> 1M)
  with --random-range-ratio 1.0 for exact length
- extract_metrics.py + parse_results.py produce results.json + report.md
- README.md documents usage and control variables
- Root README updated with entry and quick command
 2026-07-08 08:35:56 +00:00
yy-fighting
b12039599d chore: remove partial SGLang vs vLLM run that failed mid-way 2026-07-08 08:33:37 +00:00
yy-fighting
0bb8adf2d9 feat: add SLO status field to per-backend reports and results.json 
- parse_backend.py for sglang_vs_vllm and dspark_vs_default:
  - compute slo_status per scenario (TTFT P95 < 3000ms, TPOT mean < 50ms)
  - append slo_status to results.json
  - add SLO column to report.md
- BENCHMARK_WORKFLOW.md documents slo_status in schema and checklist
 2026-07-08 08:28:37 +00:00
yy-fighting
162168da3c feat: record exact server launch args in results.json for reproducibility 
- sglang_vs_vllm/run_bench.sh: add phase1/phase2 server_args per backend
- dspark_vs_default/run_bench.sh: add server_args for dspark/default
- READMEs document where to find the recorded server args
 2026-07-08 08:21:55 +00:00
yy-fighting
5d1cd9c266 feat: add vLLM+DSpark vs vLLM-default TTFT comparison experiment 
- New experiments/dsv4_h200_vllm_dspark_vs_default/
- start_dspark.sh: vllm serve with --spec-method dspark --spec-tokens 5
- start_default.sh: plain vllm serve baseline
- Unified scenario matrix (512/4000 input, 1/32/128 concurrency)
- parse_backend.py + compare.py focused on TTFT differences
- README.md + root README entries
 2026-07-08 08:15:02 +00:00
yy-fighting
3cf8b8f0d7 fix: use --context-length instead of --max-model-len for SGLang server 2026-07-08 07:57:15 +00:00
yy-fighting
06b3398ad0 feat: add ShareGPT dataset scenarios to SGLang vs vLLM comparison 
- config.env: add SHAREGPT_DATASET, SHAREGPT_CONTEXT_LEN and SHAREGPT_SCENARIOS
- run_bench.sh: add run_sharegpt() and integrate it after Phase 1 for each backend
- parse_backend.py: support both phase{1|2} and sharegpt raw output filenames
- README.md: document ShareGPT scenarios
 2026-07-08 07:42:05 +00:00
yy-fighting
e6d9e99b6f feat: add SGLang vs vLLM controlled comparison experiment 
- New experiments/dsv4_h200_sglang_vs_vllm/ with TP=8 on all 8 H200 cards
- Phase1 short-context throughput + Phase2 long context up to 200k
- Unified scenario matrix, warmup, parsing, and side-by-side comparison report
- H200 SGLang vs vLLM entry added to root README
 2026-07-08 07:34:34 +00:00