From 0397b4a2b53cf3cb9cacd756f3ba37305041ad27 Mon Sep 17 00:00:00 2001 From: Quantong Qiu Date: Wed, 8 Jul 2026 06:36:19 +0000 Subject: [PATCH] refactor: move legacy bench_results into experiments/legacy_bench_results/ - Keep only bench_results/dspark_grid_20260707-132641/ in original location - Move all other historical results to experiments/legacy_bench_results/ - Add README.md explaining the archive contents - Update root README.md and BENCHMARK_WORKFLOW.md path references --- BENCHMARK_WORKFLOW.md | 4 +-- README.md | 19 +++++++------- experiments/legacy_bench_results/README.md | 24 ++++++++++++++++++ .../comparison_report.md | 0 .../README.md | 0 .../summary.json | 0 .../eagle_grid/dspark_vs_eagle_report.md | 0 .../eagle_grid/report.md | 0 .../summary.json | 0 .../length_distribution.png | Bin .../report.md | 0 .../summary.json | 0 .../summary.json | 0 .../summary.json | 0 .../summary.json | 0 15 files changed, 36 insertions(+), 11 deletions(-) create mode 100644 experiments/legacy_bench_results/README.md rename {bench_results => experiments/legacy_bench_results}/dspark_st_comparison_20260707-150649/comparison_report.md (100%) rename {bench_results => experiments/legacy_bench_results}/dsv4_backend_comparison_20260707/README.md (100%) rename {bench_results => experiments/legacy_bench_results}/dsv4_comparison_20260705_152221/summary.json (100%) rename {bench_results => experiments/legacy_bench_results}/eagle_grid/dspark_vs_eagle_report.md (100%) rename {bench_results => experiments/legacy_bench_results}/eagle_grid/report.md (100%) rename {bench_results => experiments/legacy_bench_results}/sglang_8card_max_throughput_20260705_030839/summary.json (100%) rename {bench_results => experiments/legacy_bench_results}/sglang_8card_systematic_20260704_120819/length_distribution.png (100%) rename {bench_results => experiments/legacy_bench_results}/sglang_8card_systematic_20260704_120819/report.md (100%) rename {bench_results => experiments/legacy_bench_results}/sglang_8card_systematic_20260704_120819/summary.json (100%) rename {bench_results => experiments/legacy_bench_results}/vllm_dspark_qwen3_20260705_121046/summary.json (100%) rename {bench_results => experiments/legacy_bench_results}/vllm_dspark_qwen3_20260705_121218/summary.json (100%) rename {bench_results => experiments/legacy_bench_results}/vllm_dspark_qwen3_20260705_121256/summary.json (100%) diff --git a/BENCHMARK_WORKFLOW.md b/BENCHMARK_WORKFLOW.md index b5ffe05..ff15ad0 100644 --- a/BENCHMARK_WORKFLOW.md +++ b/BENCHMARK_WORKFLOW.md @@ -108,8 +108,8 @@ bench_results/_/ Examples: -- `bench_results/dspark_grid_20260707-132641/` -- `bench_results/dsv4_backend_comparison_20260707/` +- `bench_results/dspark_grid_20260707-132641/` (legacy result kept in original location) +- `experiments/legacy_bench_results/dsv4_backend_comparison_20260707/` (legacy result moved under `experiments/`) ### Raw output files diff --git a/README.md b/README.md index 7b1c872..0be6c4f 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ | `scripts/analysis/` | 跨实验对比分析工具 | | `scripts/` | 旧版/legacy benchmark 脚本 | | `experiments/` | 以实验为单位的自包含目录（脚本 + 配置 + 结果） | -| `bench_results/` | 历史归档的实验产物（只读） | +| `bench_results/` | 历史归档的实验产物（只读），目前仅保留 `dspark_grid_20260707-132641` | | `BENCHMARK_WORKFLOW.md` | benchmark 目录与命名规范 | | `scripts/SLO_STANDARDS.md` | 推理服务 SLO 标准（TTFT/TPOT） | @@ -26,19 +26,20 @@ |---|---|---| | DSV4 P800 SGLang | `experiments/dsv4_p800_sglang/run_bench.sh` | Kunlun P800 + SGLang + DeepSeek-V4-Flash-INT8 | | DSV4 H200 DSpark | `experiments/dsv4_h200_dspark/run_bench.sh` | NVIDIA H200 + vllm-dspark + DeepSeek-V4-Flash-DSpark | +| DSV4 H200 vLLM baseline | `experiments/dsv4_h200_vllm/run_bench.sh` | NVIDIA H200 + vLLM + DeepSeek-V4-Flash | ### 旧结构（scripts/ + bench_results/） | 实验 | 脚本 | 最终报告 | 说明 | |---|---|---|---| -| DSpark grid benchmark | `scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh` | `bench_results/dspark_grid_20260707-132641/report.md` | P1/P2/P3 全量网格 | -| DSpark spec-tokens 对比 | `scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh` | `bench_results/dspark_st_comparison_20260707-150649/comparison_report.md` | `--spec-tokens 3` vs `5` | -| SGLang EAGLE vs DSpark | `scripts/start_sglang_dsv4_8card.sh` + 对比解析脚本 | `bench_results/eagle_grid/dspark_vs_eagle_report.md` | EAGLE 投机解码对比 | -| SGLang vs vLLM 后端对比（0707） | `scripts/benchmark_dsv4_backend_comparison.sh` | `bench_results/dsv4_backend_comparison_20260707/README.md` | 遗留 raw outputs 清单 | -| SGLang 8-card systematic | `scripts/run_sglang_benchmark.sh` | `bench_results/sglang_8card_systematic_20260704_120819/report.md` | 早期 SGLang 系统扫描 | -| SGLang 8-card max throughput | `scripts/run_sglang_max_throughput.sh` | `bench_results/sglang_8card_max_throughput_20260705_030839/summary.json` | 最大吞吐扫描 | -| vLLM-dspark Qwen3 | `scripts/bench_vllm_dspark_qwen3.py` | `bench_results/vllm_dspark_qwen3_20260705_121256/summary.json` | Qwen3 模型测速 | -| DSV4 后端对比（0705） | `scripts/bench_dsv4_comparison.py` | `bench_results/dsv4_comparison_20260705_152221/summary.json` | 早期 DSpark 配置对比 | +| DSpark grid benchmark | `scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh` | `bench_results/dspark_grid_20260707-132641/report.md` | P1/P2/P3 全量网格（唯一保留在 `bench_results/` 的结果） | +| DSpark spec-tokens 对比 | `scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh` | `experiments/legacy_bench_results/dspark_st_comparison_20260707-150649/comparison_report.md` | `--spec-tokens 3` vs `5` | +| SGLang EAGLE vs DSpark | `scripts/start_sglang_dsv4_8card.sh` + 对比解析脚本 | `experiments/legacy_bench_results/eagle_grid/dspark_vs_eagle_report.md` | EAGLE 投机解码对比 | +| SGLang vs vLLM 后端对比（0707） | `scripts/benchmark_dsv4_backend_comparison.sh` | `experiments/legacy_bench_results/dsv4_backend_comparison_20260707/README.md` | 遗留 raw outputs 清单 | +| SGLang 8-card systematic | `scripts/run_sglang_benchmark.sh` | `experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/report.md` | 早期 SGLang 系统扫描 | +| SGLang 8-card max throughput | `scripts/run_sglang_max_throughput.sh` | `experiments/legacy_bench_results/sglang_8card_max_throughput_20260705_030839/summary.json` | 最大吞吐扫描 | +| vLLM-dspark Qwen3 | `scripts/bench_vllm_dspark_qwen3.py` | `experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121256/summary.json` | Qwen3 模型测速 | +| DSV4 后端对比（0705） | `scripts/bench_dsv4_comparison.py` | `experiments/legacy_bench_results/dsv4_comparison_20260705_152221/summary.json` | 早期 DSpark 配置对比 | ## 快速复现 diff --git a/experiments/legacy_bench_results/README.md b/experiments/legacy_bench_results/README.md new file mode 100644 index 0000000..e5b6594 --- /dev/null +++ b/experiments/legacy_bench_results/README.md @@ -0,0 +1,24 @@ +# Legacy Bench Results Archive + +This directory contains historical benchmark results that were originally stored under `bench_results/` before the `experiments/` layout was introduced. + +Only `bench_results/dspark_grid_20260707-132641/` remains in the original location per project decision; everything else was moved here for archival. + +## Contents + +| Directory | Original location | Description | +|---|---|---| +| `dspark_st_comparison_20260707-150649/` | `bench_results/dspark_st_comparison_20260707-150649/` | `--spec-tokens 3` vs `5` comparison | +| `dsv4_backend_comparison_20260707/` | `bench_results/dsv4_backend_comparison_20260707/` | SGLang vs vLLM backend comparison raw outputs manifest | +| `dsv4_comparison_20260705_152221/` | `bench_results/dsv4_comparison_20260705_152221/` | Early vLLM-dspark configuration comparison | +| `dsv4_flash_dspark_misc/` | `bench_results/dsv4_flash_dspark_misc/` | Early DSV4 DSpark one-off tests | +| `eagle_grid/` | `bench_results/eagle_grid/` | SGLang EAGLE vs DSpark comparison | +| `pd_bench/` | `bench_results/pd_bench/` | PD disaggregation benchmark outputs | +| `sglang_8card_max_throughput_20260705_030839/` | `bench_results/sglang_8card_max_throughput_20260705_030839/` | SGLang 8-card max throughput scan | +| `sglang_8card_systematic_20260704_120819/` | `bench_results/sglang_8card_systematic_20260704_120819/` | Early SGLang systematic scan | +| `sglang_misc/` | `bench_results/sglang_misc/` | Miscellaneous SGLang outputs | +| `vllm_dspark_qwen3_20260705_121046/` | `bench_results/vllm_dspark_qwen3_20260705_121046/` | Qwen3 DSpark tests | +| `vllm_dspark_qwen3_20260705_121218/` | `bench_results/vllm_dspark_qwen3_20260705_121218/` | Qwen3 DSpark tests | +| `vllm_dspark_qwen3_20260705_121256/` | `bench_results/vllm_dspark_qwen3_20260705_121256/` | Qwen3 DSpark tests | + +> Note: Some internal file paths (e.g. `result_file` in `summary.json`, hardcoded paths in old scripts) still reference the original `bench_results/` locations. These are kept as-is for historical accuracy. diff --git a/bench_results/dspark_st_comparison_20260707-150649/comparison_report.md b/experiments/legacy_bench_results/dspark_st_comparison_20260707-150649/comparison_report.md similarity index 100% rename from bench_results/dspark_st_comparison_20260707-150649/comparison_report.md rename to experiments/legacy_bench_results/dspark_st_comparison_20260707-150649/comparison_report.md diff --git a/bench_results/dsv4_backend_comparison_20260707/README.md b/experiments/legacy_bench_results/dsv4_backend_comparison_20260707/README.md similarity index 100% rename from bench_results/dsv4_backend_comparison_20260707/README.md rename to experiments/legacy_bench_results/dsv4_backend_comparison_20260707/README.md diff --git a/bench_results/dsv4_comparison_20260705_152221/summary.json b/experiments/legacy_bench_results/dsv4_comparison_20260705_152221/summary.json similarity index 100% rename from bench_results/dsv4_comparison_20260705_152221/summary.json rename to experiments/legacy_bench_results/dsv4_comparison_20260705_152221/summary.json diff --git a/bench_results/eagle_grid/dspark_vs_eagle_report.md b/experiments/legacy_bench_results/eagle_grid/dspark_vs_eagle_report.md similarity index 100% rename from bench_results/eagle_grid/dspark_vs_eagle_report.md rename to experiments/legacy_bench_results/eagle_grid/dspark_vs_eagle_report.md diff --git a/bench_results/eagle_grid/report.md b/experiments/legacy_bench_results/eagle_grid/report.md similarity index 100% rename from bench_results/eagle_grid/report.md rename to experiments/legacy_bench_results/eagle_grid/report.md diff --git a/bench_results/sglang_8card_max_throughput_20260705_030839/summary.json b/experiments/legacy_bench_results/sglang_8card_max_throughput_20260705_030839/summary.json similarity index 100% rename from bench_results/sglang_8card_max_throughput_20260705_030839/summary.json rename to experiments/legacy_bench_results/sglang_8card_max_throughput_20260705_030839/summary.json diff --git a/bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png b/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png similarity index 100% rename from bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png rename to experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/length_distribution.png diff --git a/bench_results/sglang_8card_systematic_20260704_120819/report.md b/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/report.md similarity index 100% rename from bench_results/sglang_8card_systematic_20260704_120819/report.md rename to experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/report.md diff --git a/bench_results/sglang_8card_systematic_20260704_120819/summary.json b/experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/summary.json similarity index 100% rename from bench_results/sglang_8card_systematic_20260704_120819/summary.json rename to experiments/legacy_bench_results/sglang_8card_systematic_20260704_120819/summary.json diff --git a/bench_results/vllm_dspark_qwen3_20260705_121046/summary.json b/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121046/summary.json similarity index 100% rename from bench_results/vllm_dspark_qwen3_20260705_121046/summary.json rename to experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121046/summary.json diff --git a/bench_results/vllm_dspark_qwen3_20260705_121218/summary.json b/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121218/summary.json similarity index 100% rename from bench_results/vllm_dspark_qwen3_20260705_121218/summary.json rename to experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121218/summary.json diff --git a/bench_results/vllm_dspark_qwen3_20260705_121256/summary.json b/experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121256/summary.json similarity index 100% rename from bench_results/vllm_dspark_qwen3_20260705_121256/summary.json rename to experiments/legacy_bench_results/vllm_dspark_qwen3_20260705_121256/summary.json