qqtang/sskj
1
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: record exact server launch args in results.json for reproducibility

Browse Source 
- sglang_vs_vllm/run_bench.sh: add phase1/phase2 server_args per backend
- dspark_vs_default/run_bench.sh: add server_args for dspark/default
- READMEs document where to find the recorded server args
This commit is contained in:
yy-fighting 2026-07-08 08:21:55 +00:00
parent 69756c598c
commit 162168da3c
4 changed files with 32 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
experiments/dsv4_h200_sglang_vs_vllm/README.md
View File

@ -9,6 +9,7 @@
- 场景矩阵、并发、输入/输出长度、请求数对两个 backend 完全一致。
- 每个 phase 启动 server 后先做 warmup，再跑正式压测。
- 短上下文和长上下文分阶段启动 server，避免 `max-model-len` / `max-num-seqs` 显存冲突。
- 每次 server 启动的**完整命令行参数**会写入 `results/<RUN_ID>/<backend>/results.json` 的 `config.phase1_server_args` / `config.phase2_server_args`，方便在 P800 等其他平台复现时保持参数一致。
## 场景矩阵

19
experiments/dsv4_h200_sglang_vs_vllm/run_bench.sh
View File

@ -251,6 +251,16 @@ write_backend_metadata() {
env_path="$VENV_VLLM"
fi
# Capture the exact server launch args for reproducibility.
local phase1_args phase2_args
if [[ "$backend" == "sglang" ]]; then
phase1_args="sglang serve --trust-remote-code --model-path $MODEL_PATH --tp 8 --moe-runner-backend marlin --context-length $PHASE1_MAX_MODEL_LEN --max-running-requests $PHASE1_MAX_RUNNING --mem-fraction-static 0.88 --host 0.0.0.0 --port $SGLANG_PORT"
phase2_args="sglang serve --trust-remote-code --model-path $MODEL_PATH --tp 8 --moe-runner-backend marlin --context-length $PHASE2_MAX_MODEL_LEN --max-running-requests $PHASE2_MAX_RUNNING --mem-fraction-static 0.88 --host 0.0.0.0 --port $SGLANG_PORT"
else
phase1_args="vllm serve $MODEL_PATH --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len $PHASE1_MAX_MODEL_LEN --max-num-seqs $PHASE1_MAX_NUM_SEQS --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port $VLLM_PORT"
phase2_args="vllm serve $MODEL_PATH --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len $PHASE2_MAX_MODEL_LEN --max-num-seqs $PHASE2_MAX_NUM_SEQS --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port $VLLM_PORT"
fi
write_metadata_json \
"$meta_json" \
"${EXPERIMENT_NAME}_${backend}" \
@ -265,12 +275,12 @@ write_backend_metadata() {
"$env_path" \
"H200 ${backend} TP=8 comparison benchmark for DeepSeek-V4-Flash"
# Embed config.
"${VENV_CLIENT}/bin/python" - "$meta_json" "$backend" <<'PY'
# Embed config and server args.
"${VENV_CLIENT}/bin/python" - "$meta_json" "$backend" "$phase1_args" "$phase2_args" <<'PY'
import json
import sys
path, backend = sys.argv[1], sys.argv[2]
path, backend, phase1_args, phase2_args = sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4]
with open(path, "r", encoding="utf-8") as f:
data = json.load(f)
@ -280,6 +290,9 @@ data["config"] = {
"phase1_max_model_len": 32768,
"phase2_max_model_len": 210000,
"backend": backend,
"server_start_script": f"experiments/dsv4_h200_sglang_vs_vllm/start_{backend}.sh",
"phase1_server_args": phase1_args,
"phase2_server_args": phase2_args,
}
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)

2
experiments/dsv4_h200_vllm_dspark_vs_default/README.md
View File

@ -6,6 +6,8 @@
你之前的观察是：开启 DSpark 后 TTFT 比默认 vLLM 更长。这个实验把两个配置放在同一套场景下重测，确认现象并量化差异，为后续分析根因提供数据。
> 每次 server 启动的**完整命令行参数**会写入 `results/<RUN_ID>/<backend>/results.json` 的 `config.server_args`，方便在 P800 等其他平台复现时保持参数一致。
## 控制变量
| 维度 | DSpark | Default |

16
experiments/dsv4_h200_vllm_dspark_vs_default/run_bench.sh
View File

@ -178,6 +178,14 @@ write_backend_metadata() {
engine="vllm-default"
fi
# Capture exact server launch args for reproducibility.
local server_args
if [[ "$backend" == "dspark" ]]; then
server_args="vllm serve $model_path --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len $MAX_MODEL_LEN --max-num-seqs $MAX_NUM_SEQS --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --spec-method dspark --spec-model $model_path --spec-tokens 5 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port $DSPARK_PORT"
else
server_args="vllm serve $model_path --trust-remote-code --tensor-parallel-size 8 --kv-cache-dtype fp8 --max-model-len $MAX_MODEL_LEN --max-num-seqs $MAX_NUM_SEQS --block-size 256 --gpu-memory-utilization 0.90 --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --no-disable-hybrid-kv-cache-manager --disable-uvicorn-access-log --port $DEFAULT_PORT"
fi
write_metadata_json \
"$meta_json" \
"${EXPERIMENT_NAME}_${backend}" \
@ -192,12 +200,12 @@ write_backend_metadata() {
"$env_path" \
"H200 ${engine} TP=8 benchmark for TTFT comparison"
# Embed config.
"${VENV_CLIENT}/bin/python" - "$meta_json" "$backend" <<'PY'
# Embed config and server args.
"${VENV_CLIENT}/bin/python" - "$meta_json" "$backend" "$server_args" <<'PY'
import json
import sys
path, backend = sys.argv[1], sys.argv[2]
path, backend, server_args = sys.argv[1], sys.argv[2], sys.argv[3]
with open(path, "r", encoding="utf-8") as f:
data = json.load(f)
@ -207,6 +215,8 @@ data["config"] = {
"max_model_len": 32768,
"max_num_seqs": 256,
"backend": backend,
"server_start_script": f"experiments/dsv4_h200_vllm_dspark_vs_default/start_{backend}.sh",
"server_args": server_args,
}
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)