qqtang/sskj
1
2
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(h20): capture decode graphs through batch 128

Browse Source
This commit is contained in:
Quantong Qiu 2026-07-20 17:56:58 +08:00
parent ca965b6d26
commit 105267b1b8
4 changed files with 13 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
README.md
View File

@ -1,5 +1,8 @@
# sskj — 多平台大模型推理性能基准测试项目 # sskj — 多平台大模型推理性能基准测试项目
> **更新（2026-07-20 17:55:20 +0800）**
> - H20 SGLang TPxDP matrix 默认传入 `--cuda-graph-max-bs-decode 128`，使高并发 decode 在 batch 不超过 128 时持续使用 CUDA Graph；Docker 与 native 启动入口均已覆盖。
> **更新（2026-07-20 17:25:40 +0800）** > **更新（2026-07-20 17:25:40 +0800）**
> - RTX 6000D 的 SGLang DSV4-Flash 默认传入 `--cuda-graph-max-bs-decode 16`，避免 SM120 sparse MLA 在大 batch CUDA Graph 预捕获时 OOM。该参数仅限制预捕获图的 decode batch 上限，不限制服务的 `max-running-requests` 或 benchmark 并发；其他平台默认不受影响。 > - RTX 6000D 的 SGLang DSV4-Flash 默认传入 `--cuda-graph-max-bs-decode 16`，避免 SM120 sparse MLA 在大 batch CUDA Graph 预捕获时 OOM。该参数仅限制预捕获图的 decode batch 上限，不限制服务的 `max-running-requests` 或 benchmark 并发；其他平台默认不受影响。
> - H20、RTX 6000D、P800 的配置不再硬编码 sskj 仓库绝对路径：仓库内的环境、数据集、缓存和 P800 patch 均通过共享的 `${ROOT_DIR}` 推导，仓库迁移后无需逐个修改路径。 > - H20、RTX 6000D、P800 的配置不再硬编码 sskj 仓库绝对路径：仓库内的环境、数据集、缓存和 P800 patch 均通过共享的 `${ROOT_DIR}` 推导，仓库迁移后无需逐个修改路径。

2
experiments/h20/dsv4_h20_sglang_tp_dp_matrix/config.env
View File

@ -36,6 +36,8 @@ declare -a PARALLEL_CONFIGS=(
# Context length and scheduler capacity use SGLang's model/default settings. # Context length and scheduler capacity use SGLang's model/default settings.
MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.88}" MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.88}"
MOE_RUNNER_BACKEND="${MOE_RUNNER_BACKEND:-marlin}" MOE_RUNNER_BACKEND="${MOE_RUNNER_BACKEND:-marlin}"
# Keep high-concurrency decode on captured CUDA Graphs on H20.
CUDA_GRAPH_MAX_BS_DECODE="${CUDA_GRAPH_MAX_BS_DECODE:-128}"
# Deployment switch. 0 = native sglang venv, 1 = Docker. # Deployment switch. 0 = native sglang venv, 1 = Docker.
USE_DOCKER="${USE_DOCKER:-1}" USE_DOCKER="${USE_DOCKER:-1}"

4
experiments/h20/dsv4_h20_sglang_tp_dp_matrix/start_sglang_docker.sh
View File

@ -44,6 +44,10 @@ if [[ "$DP" -gt 1 ]]; then
) )
fi fi
if [[ -n "${CUDA_GRAPH_MAX_BS_DECODE:-}" ]]; then
SERVER_ARGS+=(--cuda-graph-max-bs-decode "$CUDA_GRAPH_MAX_BS_DECODE")
fi
SERVER_ARGS_STR="${SERVER_ARGS[*]}" SERVER_ARGS_STR="${SERVER_ARGS[*]}"
echo "=== Starting SGLang server in Docker (TP=${TP}, DP=${DP}) ===" echo "=== Starting SGLang server in Docker (TP=${TP}, DP=${DP}) ==="

4
experiments/h20/dsv4_h20_sglang_tp_dp_matrix/start_sglang_dp.sh
View File

@ -49,6 +49,10 @@ if [[ "$DP" -gt 1 ]]; then
) )
fi fi
if [[ -n "${CUDA_GRAPH_MAX_BS_DECODE:-}" ]]; then
SERVER_ARGS+=(--cuda-graph-max-bs-decode "$CUDA_GRAPH_MAX_BS_DECODE")
fi
SERVER_ARGS_STR="${SERVER_ARGS[*]}" SERVER_ARGS_STR="${SERVER_ARGS[*]}"
echo "=== Starting SGLang server (TP=${TP}, DP=${DP}) ===" echo "=== Starting SGLang server (TP=${TP}, DP=${DP}) ==="