feat(pro6000): Kimi-K3 TP32×EP32 部署 profile、sm_120 补丁与运维手册 - 4 节点 RoCE 部署 + bench 实验
This commit is contained in:
shishi
parent
d9a2e3b5e5
commit
3bd04698bb
@ -44,6 +44,7 @@
|
|||||||
| `experiments/p800/dsv4_p800_sglang_tp_dp_matrix/` | P800 + SGLang（INT8；TP2/DP4 启动 OOM 无数据，见 config.env 注释） |
|
| `experiments/p800/dsv4_p800_sglang_tp_dp_matrix/` | P800 + SGLang（INT8；TP2/DP4 启动 OOM 无数据，见 config.env 注释） |
|
||||||
| `experiments/pro6000/dsv4_pro6000_vllm_tp_dp_matrix/` | RTX 6000D + vLLM |
|
| `experiments/pro6000/dsv4_pro6000_vllm_tp_dp_matrix/` | RTX 6000D + vLLM |
|
||||||
| `experiments/pro6000/dsv4_pro6000_sglang_tp_dp_matrix/` | RTX 6000D + SGLang |
|
| `experiments/pro6000/dsv4_pro6000_sglang_tp_dp_matrix/` | RTX 6000D + SGLang |
|
||||||
|
| `experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/` | RTX 6000D + SGLang，Kimi-K3（TP32×EP32，部署手册见 docs/KIMI_K3_DEPLOY.md） |
|
||||||
|
|
||||||
每个目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_concurrency.sh` 从 C=1 指数倍增搜饱和点；`run_adaptive_concurrency_add16.sh` 从 C=16 线性 +16 步进、带 TTFT SLO 停止与回退（当前主力用法，见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`）。
|
每个目录内：`run_bench.sh` 跑固定并发矩阵；`run_adaptive_concurrency.sh` 从 C=1 指数倍增搜饱和点；`run_adaptive_concurrency_add16.sh` 从 C=16 线性 +16 步进、带 TTFT SLO 停止与回退（当前主力用法，见 `experiments/ADAPTIVE_CONCURRENCY_USAGE.md`）。
|
||||||
|
|
||||||
|
|||||||
54
deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env
Normal file
54
deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env
Normal file
@ -0,0 +1,54 @@
|
|||||||
|
# Kimi-K3 SGLang multi-node TP=32 EP=32 deployment profile (4x RTX 6000D).
|
||||||
|
# Nodes: 174.1.60.5~8 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||||||
|
#
|
||||||
|
# 关键点（实测踩坑，勿随意改）:
|
||||||
|
# - MoE 后端必须 marlin（K3 的 MXFP4 缩放因子为 uint8，DeepGEMM 只接受 fp32/UE8M0）
|
||||||
|
# - RoCE: NCCL_IB_HCA=mlx5_0..3（4 张独立卡, 10.100.21-24/24, RoCEv2 GID index 3）
|
||||||
|
# 实测 32-rank 117MB allreduce 2.4ms；勿用 mlx5_bond_0（仅 4.5GB/s）
|
||||||
|
# - 容器必须 --ulimit memlock=-1（否则 ibv_create_cq 报 Cannot allocate memory）
|
||||||
|
# - 不要设 NCCL_ALGO=TREE（CUDA graph 捕获报 "NCCL error: invalid usage"）
|
||||||
|
# - 首次请求含 ~110s Triton JIT 编译，属正常现象，预热一次后回落
|
||||||
|
# - flashkda 可选后端（与 triton prefill 性能等价）见 docs/KIMI_K3_DEPLOY.md 附录
|
||||||
|
#
|
||||||
|
# Model-team only. Ops only run `python -m sskj.bench` against the served URL.
|
||||||
|
|
||||||
|
PLATFORM=pro6000
|
||||||
|
EXPERIMENT=kimi3_pro6000_sglang_tp32ep32
|
||||||
|
MODEL_NAME=Kimi-K3
|
||||||
|
ENGINE=sglang
|
||||||
|
RUNTIME=docker
|
||||||
|
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||||||
|
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||||||
|
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||||
|
SERVED_MODEL_NAME=kimi-k3
|
||||||
|
PORT=30000
|
||||||
|
HEALTH_PATH=/health
|
||||||
|
HEALTH_HOST=174.1.60.5
|
||||||
|
HEALTH_WAIT_S=2400
|
||||||
|
CONTAINER_PYTHON=python3
|
||||||
|
|
||||||
|
# ---- 并行度（固定，勿改）----
|
||||||
|
TP=32
|
||||||
|
DP=1
|
||||||
|
EP_SIZE=32
|
||||||
|
|
||||||
|
# ---- Multi-node topology (rank order; rank 0 exposes the HTTP API) ----
|
||||||
|
NNODES=4
|
||||||
|
NODE_HOSTS="174.1.60.5 174.1.60.6 174.1.60.7 174.1.60.8"
|
||||||
|
NODE_SSH_USER=root
|
||||||
|
LOCAL_NODE_RANK=0
|
||||||
|
MASTER_IP=174.1.60.5
|
||||||
|
DIST_PORT=20000
|
||||||
|
|
||||||
|
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||||||
|
ENGINE_ENV="NCCL_SOCKET_IFNAME=bond0 GLOO_SOCKET_IFNAME=bond0 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1"
|
||||||
|
|
||||||
|
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||||||
|
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||||
|
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro"
|
||||||
|
|
||||||
|
# BOOTSTRAP 在容器内执行: 打 sm_120 补丁 → 按节点 rank 计算 SGLANG_HOST_IP → 启动 sglang。
|
||||||
|
# SGLANG_HOST_IP 必须为本节点实际 IP（174.1.60.5~8 = 5 + NODE_RANK）。
|
||||||
|
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && export SGLANG_HOST_IP=\"174.1.60.$((5 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||||||
|
|
||||||
|
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer_lazy --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --host 0.0.0.0 --port ${PORT}"
|
||||||
194
docs/KIMI_K3_DEPLOY.md
Normal file
194
docs/KIMI_K3_DEPLOY.md
Normal file
@ -0,0 +1,194 @@
|
|||||||
|
# Kimi-K3 部署手册（运维版）
|
||||||
|
|
||||||
|
> 面向运维的一键部署手册：按顺序复制粘贴命令即可完成 Kimi-K3 在 4 节点
|
||||||
|
> RTX 6000D 集群上的部署。**不需要理解 SGLang 引擎参数。**
|
||||||
|
> 部署配置统一在 `deploy/profiles/pro6000/kimi3_pro6000_sglang_tp32ep32.env`，
|
||||||
|
> 压测统一走 `python -m sskj.bench`（见 `ops/README.md`）。
|
||||||
|
|
||||||
|
## 0. 概述
|
||||||
|
|
||||||
|
| 项 | 值 |
|
||||||
|
|---|---|
|
||||||
|
| 模型 | Kimi-K3（Moonshot AI，2.8T 参数 MXFP4 量化，~1.5TB） |
|
||||||
|
| 节点 | 6000D-5~8 = 174.1.60.5~8（每节点 8× RTX 6000D 85GB，共 32 卡） |
|
||||||
|
| 镜像 | `lmsysorg/sglang:kimi-k3`（专用镜像，普通 sglang 镜像不识别该模型架构） |
|
||||||
|
| 并行 | TP=32 × EP=32（跨 4 节点） |
|
||||||
|
| 服务地址 | `http://174.1.60.5:30000`（OpenAI 兼容） |
|
||||||
|
| 模型 ID | `kimi-k3` |
|
||||||
|
| 部署方式 | `python -m sskj.deploy start --profile pro6000/kimi3_pro6000_sglang_tp32ep32` |
|
||||||
|
|
||||||
|
## 1. 前置检查（每步都过再继续）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1.1 4 节点 8 卡全部空闲（应全部显示 0 MiB / 0%）
|
||||||
|
for i in 5 6 7 8; do echo "== 174.1.60.$i"; ssh 174.1.60.$i 'nvidia-smi --query-gpu=index,memory.used --format=csv,noheader'; done
|
||||||
|
# 注意：如显示被占用，联系模型团队确认（其他用户的作业需先让出）
|
||||||
|
|
||||||
|
# 1.2 磁盘空间（模型 1.5TB + 运行余量，/data 需 ≥ 2TB 可用）
|
||||||
|
ssh 174.1.60.5 'df -h /data'
|
||||||
|
|
||||||
|
# 1.3 RoCE 网卡就绪（应输出 4 个 ACTIVE 且 link_layer=Ethernet）
|
||||||
|
ssh 174.1.60.5 'cat /sys/class/infiniband/mlx5_*/ports/1/link_layer; cat /sys/class/infiniband/mlx5_*/ports/1/state'
|
||||||
|
|
||||||
|
# 1.4 节点互信（174.1.60.5 免密登录其余节点；若失败执行步骤 1.5）
|
||||||
|
ssh 174.1.60.5 'for i in 6 7 8; do ssh -o StrictHostKeyChecking=no 174.1.60.$i hostname; done'
|
||||||
|
|
||||||
|
# 1.5 若 1.4 失败，在 174.1.60.5 上执行（把 .5 的 root 公钥发到各节点）
|
||||||
|
ssh 174.1.60.5 'for i in 6 7 8; do ssh-copy-id -o StrictHostKeyChecking=no root@174.1.60.$i; done'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. 模型下载（仅首次，~1.5TB，需数小时）
|
||||||
|
|
||||||
|
模型权重放 `/data/hf_models/Kimi-K3`（4 节点同一路径，本集群已就绪则跳过本节）。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 在 174.1.60.5 上执行（国内走 ModelScope 最快；modelscope CLI 需先 pip install modelscope）
|
||||||
|
pip3 install -q modelscope
|
||||||
|
mkdir -p /data/hf_models
|
||||||
|
nohup modelscope download --model moonshotai/Kimi-K3 \
|
||||||
|
--local_dir /data/hf_models/Kimi-K3 > /data/hf_models/download_k3.log 2>&1 &
|
||||||
|
# 查看进度
|
||||||
|
tail -f /data/hf_models/download_k3.log
|
||||||
|
```
|
||||||
|
|
||||||
|
下载完成后校验（96 个分片，共约 1.5TB）：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh 174.1.60.5 'ls /data/hf_models/Kimi-K3/*.safetensors | wc -l; du -sh /data/hf_models/Kimi-K3'
|
||||||
|
# 期望：96（safetensors 数量）、1.5T（总大小）
|
||||||
|
```
|
||||||
|
|
||||||
|
**分发到其余 3 台**（.5 上执行；源盘 NVMe 读是瓶颈，聚合约 3.5GB/s）：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh 174.1.60.5 'for i in 6 7 8; do rsync -aH --partial /data/hf_models/Kimi-K3/ 174.1.60.$i:/data/hf_models/Kimi-K3/ & done; wait'
|
||||||
|
```
|
||||||
|
|
||||||
|
> 每节点需约 1.5T 空闲磁盘（`df -h /data` 确认）。
|
||||||
|
|
||||||
|
## 3. 镜像准备（已拉取则跳过）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 4 节点并行拉取（约 9.6GB；如 Docker Hub 直连失败，各机 daemon.json 需配镜像加速：
|
||||||
|
# "registry-mirrors": ["https://docker.m.daocloud.io", "https://docker.xuanyuan.me"]）
|
||||||
|
for i in 5 6 7 8; do ssh 174.1.60.$i 'docker pull lmsysorg/sglang:kimi-k3' & done; wait
|
||||||
|
|
||||||
|
# 验证镜像存在
|
||||||
|
ssh 174.1.60.5 'docker images lmsysorg/sglang:kimi-k3 --format "{{.Repository}}:{{.Tag}} {{.Size}}"'
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. 补丁与依赖分发
|
||||||
|
|
||||||
|
Kimi-K3 镜像在消费级 Blackwell（sm_120，RTX 6000D）上有一个内核不兼容点，
|
||||||
|
需要打补丁（补丁源文件在本仓库 `platforms/patches/pro6000/kimi_k3/patch_k3_sm120.py`，
|
||||||
|
修复 attn_res 融合内核误用数据中心 Blackwell 专属的 tcgen05 指令问题）。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 在 174.1.60.5 上，把补丁分发到 4 节点 /tmp（部署时容器自动挂载并执行）
|
||||||
|
ssh 174.1.60.5 '
|
||||||
|
for i in 5 6 7 8; do
|
||||||
|
scp /data/yy/sskj/platforms/patches/pro6000/kimi_k3/patch_k3_sm120.py 174.1.60.$i:/tmp/patch_k3_sm120.py
|
||||||
|
done
|
||||||
|
for i in 5 6 7 8; do ssh 174.1.60.$i "sha256sum /tmp/patch_k3_sm120.py"; done
|
||||||
|
'
|
||||||
|
# 4 台输出的 sha256 必须一致（内容校验）
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. 一键部署
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /data/yy/sskj
|
||||||
|
|
||||||
|
# 5.1 先预览要执行的命令（不实际启动）
|
||||||
|
PYTHONPATH=src python3 -m sskj.deploy start \
|
||||||
|
--profile pro6000/kimi3_pro6000_sglang_tp32ep32 --dry-run
|
||||||
|
|
||||||
|
# 5.2 正式启动（4 节点同时拉起，耗时 8~12 分钟：模型加载 + CUDA graph 捕获）
|
||||||
|
PYTHONPATH=src python3 -m sskj.deploy start \
|
||||||
|
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||||
|
|
||||||
|
# 5.3 查看状态
|
||||||
|
PYTHONPATH=src python3 -m sskj.deploy status \
|
||||||
|
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||||
|
```
|
||||||
|
|
||||||
|
就绪判定（满足其一）：
|
||||||
|
- 步骤 5.3 `status` 显示容器 Up 且健康检查通过
|
||||||
|
- 或手动验证：`curl -s http://174.1.60.5:30000/health` 返回 `{"status":"ok"}`
|
||||||
|
|
||||||
|
## 6. 验证服务
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 6.1 模型列表
|
||||||
|
curl -s http://174.1.60.5:30000/v1/models | head -c 300
|
||||||
|
# 应包含 "id": "kimi-k3"
|
||||||
|
|
||||||
|
# 6.2 推理冒烟（正确性：23×47 应算得 1081，输出可能带思考标签属正常）
|
||||||
|
curl -s -m 300 http://174.1.60.5:30000/v1/chat/completions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"model": "kimi-k3", "messages": [{"role": "user", "content": "What is 23*47? Answer briefly."}], "max_tokens": 128, "temperature": 0.1}'
|
||||||
|
|
||||||
|
# 6.3 服务日志（确认无 ERROR）
|
||||||
|
ssh 174.1.60.5 'docker logs kimi3_pro6000_sglang_tp32ep32_node0 2>&1 | tail -20'
|
||||||
|
```
|
||||||
|
|
||||||
|
> ⚠️ **首次请求很慢（TTFT 可达 100+ 秒）是正常现象**：Triton/FlashKDA 内核首次
|
||||||
|
> JIT 编译。发一个任意请求预热后，后续请求恢复正常（16K 输入 TTFT ~6s）。
|
||||||
|
|
||||||
|
## 7. 停止 / 重启
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /data/yy/sskj
|
||||||
|
|
||||||
|
# 停止（4 节点容器全部移除）
|
||||||
|
PYTHONPATH=src python3 -m sskj.deploy stop \
|
||||||
|
--profile pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||||
|
|
||||||
|
# 重启 = 再次执行 5.2 的 start 命令（幂等，会先清旧容器）
|
||||||
|
|
||||||
|
# 手动兜底（start/stop 异常时）：4 节点各执行
|
||||||
|
for i in 5 6 7 8; do ssh 174.1.60.$i 'docker rm -f kimi3_pro6000_sglang_tp32ep32_node*'; done
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. 性能自检（可选）
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 稳态基准：16K 输入 TTFT / prefill 吞吐 / TPOT（先跑一次丢弃 JIT 冷启动）
|
||||||
|
cd /data/yy/sskj
|
||||||
|
scp -o ConnectTimeout=10 /data/flashkda_deploy/bench_16k_steady.py 174.1.60.5:/tmp/ 2>/dev/null || true
|
||||||
|
ssh 174.1.60.5 'python3 /tmp/bench_16k_steady.py 3'
|
||||||
|
# 参考值（稳态）：TTFT ~6s / prefill ~2700 tok/s / TPOT ~39ms / decode ~26 tok/s
|
||||||
|
```
|
||||||
|
|
||||||
|
正式压测（矩阵 + 自适应并发搜索）走 bench 层：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /data/yy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32
|
||||||
|
DRY_RUN=1 bash run_adaptive_concurrency_add16.sh # 先看计划
|
||||||
|
bash run_adaptive_concurrency_add16.sh # 正式跑（放 tmux）
|
||||||
|
```
|
||||||
|
|
||||||
|
## 9. 故障排查速查
|
||||||
|
|
||||||
|
| 现象 | 原因 | 处理 |
|
||||||
|
|---|---|---|
|
||||||
|
| 首次请求 TTFT 100+ 秒 | Triton/FlashKDA 内核首次 JIT 编译 | 正常现象，预热一次即可 |
|
||||||
|
| `Not enough GPU memory for hybrid mamba state cache` | 某节点显存被其他作业占用 | 检查 4 节点 `nvidia-smi`，确认 8 卡全空闲后重启部署 |
|
||||||
|
| `ibv_create_cq failed: Cannot allocate memory` | 容器 memlock 限制 | 部署配置已带 `--ulimit memlock=-1`，检查 profile 未被改动 |
|
||||||
|
| `NCCL error: invalid usage`（graph 捕获时） | 误设了 `NCCL_ALGO=TREE` | 确认环境变量里没有 NCCL_ALGO |
|
||||||
|
| 启动后 `/health` 一直不通 | 分布式初始化失败/节点未就绪 | 看 master 日志；确认 4 节点 GPU 全空闲后 `stop` + 重新 `start` |
|
||||||
|
| 推理结果带 `<\|open\|>think` 标签 | K3 思考型模型正常输出格式 | 非故障；如需精简可调 chat template（联系模型团队） |
|
||||||
|
| 并发压测时部分请求失败 | flashkda 后端已知问题（未修复） | 当前默认 triton 后端无此问题；如误用 flashkda 见附录 |
|
||||||
|
|
||||||
|
## 附录 A：flashkda 可选后端（默认不启用）
|
||||||
|
|
||||||
|
FlashKDA（MoonshotAI CUTLASS KDA 内核，支持 sm_120）可作为 KDA prefill 的备选后端。
|
||||||
|
实测与默认 triton prefill **性能等价**（16K prefill 2693 vs 2695 tok/s），仅作备份用途。
|
||||||
|
|
||||||
|
启用方式（需模型团队协助）：
|
||||||
|
1. 构建 wheel（脚本见 `/data/flashkda_deploy/01_build_flashkda.sh`，产物
|
||||||
|
`flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl` 分发到 4 节点 `/tmp/`）
|
||||||
|
2. profile 增加 wheel 挂载与安装步骤，并在 LAUNCH_ARGS 加
|
||||||
|
`--linear-attn-prefill-backend flashkda`
|
||||||
|
|
||||||
|
已知问题：flashkda 后端在 4/8 并发压测下服务异常（纯 triton 无此问题），修复前不建议生产使用。
|
||||||
@ -0,0 +1,56 @@
|
|||||||
|
# Adaptive concurrency search settings.
|
||||||
|
#
|
||||||
|
# For each fixed (TP, DP, ISL, OSL), probe:
|
||||||
|
# C = start, start * multiplier, ... up to max
|
||||||
|
# and stop after Total TPS has less than TPS_MIN_GAIN_PCT meaningful growth for
|
||||||
|
# PLATEAU_PATIENCE consecutive points.
|
||||||
|
|
||||||
|
SEARCH_START_CONCURRENCY="${SEARCH_START_CONCURRENCY:-1}"
|
||||||
|
SEARCH_MAX_CONCURRENCY="${SEARCH_MAX_CONCURRENCY:-64}"
|
||||||
|
|
||||||
|
# At the add16 initial probe, restart and retry C=8 then C=1 after an OOM.
|
||||||
|
ENABLE_INITIAL_OOM_BACKOFF="${ENABLE_INITIAL_OOM_BACKOFF:-1}"
|
||||||
|
SEARCH_MULTIPLIER="${SEARCH_MULTIPLIER:-2}"
|
||||||
|
NUM_PROMPTS_MULTIPLIER="${NUM_PROMPTS_MULTIPLIER:-5}"
|
||||||
|
|
||||||
|
# A gain below 2% is treated as throughput saturation. Two consecutive
|
||||||
|
# low-gain points prevent one noisy measurement from stopping the search.
|
||||||
|
TPS_MIN_GAIN_PCT="${TPS_MIN_GAIN_PCT:-2.0}"
|
||||||
|
PLATEAU_PATIENCE="${PLATEAU_PATIENCE:-2}"
|
||||||
|
|
||||||
|
# Stop a shape when p95 TTFT exceeds the SLO; keep group skipping disabled.
|
||||||
|
TTFT_SLO_MS="${TTFT_SLO_MS:-4000}"
|
||||||
|
ENABLE_TTFT_SLO_STOP="${ENABLE_TTFT_SLO_STOP:-1}"
|
||||||
|
|
||||||
|
# Keep the same random workload semantics as the fixed matrix baseline.
|
||||||
|
# DATASET_PATH must contain at least SEARCH_MAX_CONCURRENCY times
|
||||||
|
# NUM_PROMPTS_MULTIPLIER valid two-turn conversations. Set this explicitly to
|
||||||
|
# random-ids to use generated token IDs without a ShareGPT seed dataset.
|
||||||
|
BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}"
|
||||||
|
# SGLang interprets 0.0 as Uniform[1, requested_len]. Use 1.0 for fixed
|
||||||
|
# ISL/OSL points; lower values intentionally benchmark a length distribution.
|
||||||
|
RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1.0}"
|
||||||
|
# Before each measured point, warm up with the same concurrency so lazy kernel
|
||||||
|
# compilation and CUDA graph capture are excluded from TTFT/TPS. 0 means no
|
||||||
|
# cap; set a positive cap only when very high-concurrency warmup is impractical.
|
||||||
|
BENCH_WARMUP_MAX_REQUESTS="${BENCH_WARMUP_MAX_REQUESTS:-0}"
|
||||||
|
|
||||||
|
# Reject a point if the completed request count or actual token lengths do not
|
||||||
|
# match the requested workload.
|
||||||
|
INPUT_LENGTH_TOLERANCE_PCT="${INPUT_LENGTH_TOLERANCE_PCT:-5.0}"
|
||||||
|
OUTPUT_LENGTH_TOLERANCE_PCT="${OUTPUT_LENGTH_TOLERANCE_PCT:-10.0}"
|
||||||
|
|
||||||
|
MAX_POINT_RETRIES="${MAX_POINT_RETRIES:-1}"
|
||||||
|
SERVER_RESTART_COOLDOWN_S="${SERVER_RESTART_COOLDOWN_S:-10}"
|
||||||
|
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||||
|
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||||
|
|
||||||
|
# Optional space-separated filters, useful for smoke tests:
|
||||||
|
# TP_LIST="8" ISL_LIST="1024" OSL_LIST="128"
|
||||||
|
TP_LIST="${TP_LIST:-}"
|
||||||
|
ISL_LIST="${ISL_LIST:-}"
|
||||||
|
OSL_LIST="${OSL_LIST:-}"
|
||||||
|
|
||||||
|
DRY_RUN="${DRY_RUN:-0}"
|
||||||
|
# Counts ISL/OSL shapes per TP/DP config, not individual concurrency probes.
|
||||||
|
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||||
71
experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/config.env
Normal file
71
experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/config.env
Normal file
@ -0,0 +1,71 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Kimi-K3 TP=32 EP=32 benchmark experiment on 4x RTX 6000D (174.1.60.5~8).
|
||||||
|
# 固定单配置（TP32×EP32，跨 4 节点）；服务器生命周期走 deploy profile
|
||||||
|
# `pro6000/kimi3_pro6000_sglang_tp32ep32`（部署参数以 profile 为准，勿在此重复）。
|
||||||
|
|
||||||
|
EXPERIMENT="kimi3_pro6000_sglang_tp32ep32"
|
||||||
|
MODEL_NAME="Kimi-K3"
|
||||||
|
MODEL_PATH="/data/hf_models/Kimi-K3"
|
||||||
|
SERVED_MODEL_NAME="kimi-k3"
|
||||||
|
|
||||||
|
SGLANG_PORT="${SGLANG_PORT:-30000}"
|
||||||
|
|
||||||
|
# Python interpreter for orchestration scripts (parse_backend.py, compare.py, etc.)
|
||||||
|
# and the benchmark client. Defaults to the system python3 if the sglang venv
|
||||||
|
# does not exist on the host.
|
||||||
|
VENV_CLIENT="${VENV_CLIENT:-/root/.miniconda3/envs/sglang}"
|
||||||
|
|
||||||
|
# Run the benchmark client natively (0) or inside Docker (1).
|
||||||
|
USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-1}"
|
||||||
|
|
||||||
|
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
|
||||||
|
|
||||||
|
# Runtime working directory for logs, pid files, and tmp.
|
||||||
|
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||||
|
|
||||||
|
# 单配置：TP=32 DP=1（EP=32 与 RoCE 等参数固定在 deploy profile 内）
|
||||||
|
declare -a PARALLEL_CONFIGS=(
|
||||||
|
"32 1"
|
||||||
|
)
|
||||||
|
|
||||||
|
# K3 服务器参数（与 deploy profile 保持一致，供本地/诊断脚本引用）。
|
||||||
|
MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.88}"
|
||||||
|
MOE_RUNNER_BACKEND="${MOE_RUNNER_BACKEND:-marlin}"
|
||||||
|
EP_SIZE="${EP_SIZE:-32}"
|
||||||
|
MAX_RUNNING_REQUESTS="${MAX_RUNNING_REQUESTS:-64}"
|
||||||
|
|
||||||
|
# Deployment switch. 1 = Docker（走 deploy profile），0 = 本地 venv（仅单节点调试用）。
|
||||||
|
USE_DOCKER="${USE_DOCKER:-1}"
|
||||||
|
DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:kimi-k3}"
|
||||||
|
|
||||||
|
# Deploy profile used by start/stop scripts and the adaptive search loop.
|
||||||
|
DEPLOY_PROFILE="${DEPLOY_PROFILE:-pro6000/kimi3_pro6000_sglang_tp32ep32}"
|
||||||
|
|
||||||
|
# To use ShareGPT, set BENCH_DATASET_NAME=random and DATASET_PATH explicitly.
|
||||||
|
BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}"
|
||||||
|
DATASET_PATH="${DATASET_PATH:-${ROOT_DIR}/dataset/ShareGPT_V3_unfiltered_cleaned_split.json}"
|
||||||
|
SGLANG_BENCH_MODULE="${SGLANG_BENCH_MODULE:-sglang.benchmark.serving}"
|
||||||
|
|
||||||
|
# Matrix and concurrency rules are defined in matrix.json by default.
|
||||||
|
MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR:-.}/matrix.json}"
|
||||||
|
MATRIX_MODE="${MATRIX_MODE:-Y}"
|
||||||
|
|
||||||
|
# Sampling density for concurrency.
|
||||||
|
export CONCURRENCY_SAMPLES="${CONCURRENCY_SAMPLES:-2}"
|
||||||
|
|
||||||
|
# Per-scenario timeout to avoid hangs (seconds). K3 首请求含 JIT 编译（~110s），
|
||||||
|
# 预热由 adaptive 框架处理；超时给足。
|
||||||
|
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-3600}"
|
||||||
|
|
||||||
|
# GPU memory sampling interval (seconds).
|
||||||
|
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||||
|
|
||||||
|
# Dry-run mode: if 1, only log the server args and scenario plan without starting
|
||||||
|
# any server or sending requests.
|
||||||
|
DRY_RUN="${DRY_RUN:-0}"
|
||||||
|
|
||||||
|
# Per-config scenario limit for quick smoke tests. 0 = run all generated scenarios.
|
||||||
|
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||||
|
|
||||||
|
# PyTorch CUDA allocator setting for the SGLang server.
|
||||||
|
PYTORCH_CUDA_ALLOC_CONF="${PYTORCH_CUDA_ALLOC_CONF:-expandable_segments:True}"
|
||||||
@ -0,0 +1,38 @@
|
|||||||
|
{
|
||||||
|
"comment": "ISL/OSL matrix for kimi3_pro6000_sglang_tp32ep32 (multi-node TP32 EP32). Y=must test, P=optional, N=skip. K3 最大上下文 ~364K tokens（max_total_num_tokens），ISL 上限 16384 保守可测。",
|
||||||
|
"mode": "Y",
|
||||||
|
"matrix": {
|
||||||
|
"1024": {
|
||||||
|
"128": "Y",
|
||||||
|
"256": "Y",
|
||||||
|
"512": "Y",
|
||||||
|
"1024": "Y",
|
||||||
|
"2048": "Y",
|
||||||
|
"4096": "Y"
|
||||||
|
},
|
||||||
|
"4096": {
|
||||||
|
"128": "Y",
|
||||||
|
"256": "Y",
|
||||||
|
"512": "Y",
|
||||||
|
"1024": "Y",
|
||||||
|
"2048": "Y",
|
||||||
|
"4096": "Y"
|
||||||
|
},
|
||||||
|
"8192": {
|
||||||
|
"128": "Y",
|
||||||
|
"256": "Y",
|
||||||
|
"512": "Y",
|
||||||
|
"1024": "Y",
|
||||||
|
"2048": "Y",
|
||||||
|
"4096": "Y"
|
||||||
|
},
|
||||||
|
"16384": {
|
||||||
|
"128": "Y",
|
||||||
|
"256": "Y",
|
||||||
|
"512": "Y",
|
||||||
|
"1024": "Y",
|
||||||
|
"2048": "Y",
|
||||||
|
"4096": "Y"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -0,0 +1,163 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Find the Total-TPS saturation concurrency for each SGLang TP/DP/ISL/OSL shape.
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/adaptive_config.env"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||||
|
|
||||||
|
DEPLOY_PROFILE="${DEPLOY_PROFILE:-pro6000/kimi3_pro6000_sglang_tp32ep32}"
|
||||||
|
|
||||||
|
ENGINE="sglang"
|
||||||
|
ENGINE_PORT="$SGLANG_PORT"
|
||||||
|
RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
|
||||||
|
ACTIVE_ENGINE_SERVER_LOG=""
|
||||||
|
|
||||||
|
if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
|
||||||
|
PYTHON="${VENV_CLIENT}/bin/python"
|
||||||
|
else
|
||||||
|
PYTHON="$(command -v python3)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45}"
|
||||||
|
|
||||||
|
engine_is_healthy() {
|
||||||
|
curl --fail --silent --show-error --max-time 5 \
|
||||||
|
"http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
engine_stop_server() {
|
||||||
|
local tp="$1"
|
||||||
|
local dp="$2"
|
||||||
|
|
||||||
|
log "stopping sglang server tp=${tp} dp=${dp} via deploy profile"
|
||||||
|
bash "${SCRIPT_DIR}/stop_sglang_docker.sh" "$tp" "$dp" >> "${ADAPTIVE_LOG_DIR}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1 || true
|
||||||
|
ACTIVE_ENGINE_SERVER_LOG=""
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
|
||||||
|
engine_build_server_args() {
|
||||||
|
local tp="$1"
|
||||||
|
local dp="$2"
|
||||||
|
deploy_render_args "$DEPLOY_PROFILE" "$tp" "$dp" "$ENGINE_PORT" "$MODEL_PATH"
|
||||||
|
}
|
||||||
|
|
||||||
|
engine_start_server() {
|
||||||
|
local tp="$1"
|
||||||
|
local dp="$2"
|
||||||
|
local outer_log="${ADAPTIVE_LOG_DIR}/sglang_tp${tp}_dp${dp}.server.outer.log"
|
||||||
|
log "starting sglang server tp=${tp} dp=${dp}"
|
||||||
|
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||||
|
bash "${SCRIPT_DIR}/run_sglang_in_container.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||||
|
else
|
||||||
|
bash "${SCRIPT_DIR}/start_sglang_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||||
|
fi
|
||||||
|
if ! engine_is_healthy; then
|
||||||
|
log "ERROR: sglang health check failed tp=${tp} dp=${dp}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
if [[ -z "${CONTAINER_NAME:-}" ]]; then
|
||||||
|
ACTIVE_ENGINE_SERVER_LOG="$(
|
||||||
|
find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
|
||||||
|
-name "${EXPERIMENT}_sglang*tp${tp}_dp${dp}_*.log" \
|
||||||
|
-printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
|
||||||
|
)"
|
||||||
|
fi
|
||||||
|
log "sglang server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-container:/tmp/sglang_server.log}"
|
||||||
|
}
|
||||||
|
|
||||||
|
engine_detect_oom() {
|
||||||
|
local detail_log="$1"
|
||||||
|
local tp="$2"
|
||||||
|
local dp="$3"
|
||||||
|
local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
|
||||||
|
local outer_log="${ADAPTIVE_LOG_DIR}/sglang_tp${tp}_dp${dp}.server.outer.log"
|
||||||
|
local -a logs=("$detail_log" "$outer_log")
|
||||||
|
if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
|
||||||
|
logs+=("$ACTIVE_ENGINE_SERVER_LOG")
|
||||||
|
fi
|
||||||
|
if grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||||
|
docker exec "$CONTAINER_NAME" grep -Eiq "$pattern" /tmp/sglang_server.log 2>/dev/null
|
||||||
|
return $?
|
||||||
|
fi
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
engine_run_bench() {
|
||||||
|
local isl="$1"
|
||||||
|
local osl="$2"
|
||||||
|
local concurrency="$3"
|
||||||
|
local num_prompts="$4"
|
||||||
|
local output_file="$5"
|
||||||
|
local warmup_requests
|
||||||
|
warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
|
||||||
|
local -a bench_args=(
|
||||||
|
--backend sglang
|
||||||
|
--host 127.0.0.1
|
||||||
|
--port "$ENGINE_PORT"
|
||||||
|
--dataset-name "$BENCH_DATASET_NAME"
|
||||||
|
--random-input-len "$isl"
|
||||||
|
--random-output-len "$osl"
|
||||||
|
--random-range-ratio "$RANDOM_RANGE_RATIO"
|
||||||
|
--num-prompts "$num_prompts"
|
||||||
|
--max-concurrency "$concurrency"
|
||||||
|
--request-rate 10000
|
||||||
|
--warmup-requests "$warmup_requests"
|
||||||
|
--output-file "$output_file"
|
||||||
|
--output-details
|
||||||
|
--disable-tqdm
|
||||||
|
)
|
||||||
|
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||||
|
bench_args+=(--dataset-path "$DATASET_PATH")
|
||||||
|
elif [[ "$BENCH_DATASET_NAME" == "random-ids" ]]; then
|
||||||
|
: # random-ids does not need --tokenize-prompt
|
||||||
|
else
|
||||||
|
bench_args+=(--tokenize-prompt)
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then
|
||||||
|
local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}")
|
||||||
|
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||||
|
volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro")
|
||||||
|
fi
|
||||||
|
docker run --rm \
|
||||||
|
--network host \
|
||||||
|
"${volume_args[@]}" \
|
||||||
|
-e PYTHONUNBUFFERED=1 \
|
||||||
|
--entrypoint python3 \
|
||||||
|
"$DOCKER_IMAGE" \
|
||||||
|
-m "$SGLANG_BENCH_MODULE" "${bench_args[@]}"
|
||||||
|
else
|
||||||
|
"$PYTHON" -m "$SGLANG_BENCH_MODULE" "${bench_args[@]}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
export -f engine_run_bench
|
||||||
|
export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_IMAGE USE_DOCKER_CLIENT
|
||||||
|
export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON SGLANG_BENCH_MODULE
|
||||||
|
|
||||||
|
adaptive_main "$@"
|
||||||
|
|
||||||
|
export SEARCH_START_CONCURRENCY=16
|
||||||
|
export SEARCH_ADDEND=16
|
||||||
|
# If the initial concurrency violates the TTFT SLO, search downward. Stop at
|
||||||
|
# the first acceptable value (16 -> 8; only try 1 when 8 still violates it).
|
||||||
|
export SEARCH_INITIAL_BACKOFF_CONCURRENCIES="8 1"
|
||||||
|
# When concurrency 1 still has a severely excessive TTFT, stop the remaining
|
||||||
|
# shapes in this TP/DP group. Zero disables this rule.
|
||||||
|
export TTFT_GROUP_SKIP_MS="${TTFT_GROUP_SKIP_MS:-8000}"
|
||||||
528
experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/run_bench.sh
Executable file
528
experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/run_bench.sh
Executable file
@ -0,0 +1,528 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Kimi-K3 TP32×EP32 benchmark (multi-node SGLang).
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||||
|
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
RESULT_BASE="${SCRIPT_DIR}/results"
|
||||||
|
MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR}/matrix.json}"
|
||||||
|
MATRIX_MODE="${MATRIX_MODE:-Y}"
|
||||||
|
SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
|
||||||
|
GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
|
||||||
|
DRY_RUN="${DRY_RUN:-0}"
|
||||||
|
GRID_LIMIT="${GRID_LIMIT:-0}"
|
||||||
|
|
||||||
|
# Export variables used inside functions that are called via bash -c subshells.
|
||||||
|
export BENCH_DATASET_NAME DATASET_PATH MODEL_PATH RESULT_BASE DOCKER_IMAGE USE_DOCKER_CLIENT SGLANG_BENCH_MODULE
|
||||||
|
|
||||||
|
if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
|
||||||
|
PYTHON="${VENV_CLIENT}/bin/python"
|
||||||
|
else
|
||||||
|
PYTHON="$(command -v python3)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45}"
|
||||||
|
|
||||||
|
log_dir_global="${RESULT_BASE}/${RUN_ID}/logs"
|
||||||
|
mkdir -p "$log_dir_global"
|
||||||
|
log_init "${log_dir_global}/orchestrator.log"
|
||||||
|
|
||||||
|
log "experiment=${EXPERIMENT_NAME} run_id=${RUN_ID} platform=${PLATFORM} hardware=${HARDWARE}"
|
||||||
|
log "matrix_mode=${MATRIX_MODE} matrix_file=${MATRIX_FILE} dry_run=${DRY_RUN} grid_limit=${GRID_LIMIT}"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
is_server_healthy() {
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${SGLANG_PORT}/health" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_server() {
|
||||||
|
local tp="$1"
|
||||||
|
local dp="$2"
|
||||||
|
|
||||||
|
log "stopping sglang server tp=${tp} dp=${dp} via deploy profile"
|
||||||
|
bash "${SCRIPT_DIR}/stop_sglang_docker.sh" "$tp" "$dp" >> "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1 || true
|
||||||
|
|
||||||
|
# Fallback: kill any SGLang launch processes for this model.
|
||||||
|
pkill -9 -f "sglang.launch_server.*${MODEL_NAME}" 2>/dev/null || true
|
||||||
|
pkill -9 -f "sglang.launch_server.*${MODEL_PATH}" 2>/dev/null || true
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
|
||||||
|
build_server_args() {
|
||||||
|
local tp="$1"
|
||||||
|
local dp="$2"
|
||||||
|
|
||||||
|
deploy_render_args \
|
||||||
|
"${DEPLOY_PROFILE:-pro6000/kimi3_pro6000_sglang_tp32ep32}" \
|
||||||
|
"$tp" "$dp" \
|
||||||
|
"${SGLANG_PORT:-30000}" \
|
||||||
|
"$MODEL_PATH"
|
||||||
|
}
|
||||||
|
|
||||||
|
start_server() {
|
||||||
|
local tp="$1"
|
||||||
|
local dp="$2"
|
||||||
|
|
||||||
|
log "starting sglang server tp=${tp} dp=${dp}"
|
||||||
|
if [[ -n "${CONTAINER_NAME:-}" ]]; then
|
||||||
|
bash "${SCRIPT_DIR}/run_sglang_in_container.sh" "$tp" "$dp" \
|
||||||
|
>> "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1
|
||||||
|
else
|
||||||
|
bash "${SCRIPT_DIR}/start_sglang_dp.sh" "$tp" "$dp" \
|
||||||
|
>> "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log" 2>&1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! is_server_healthy; then
|
||||||
|
log "error: sglang server tp=${tp} dp=${dp} failed health check on port ${SGLANG_PORT}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
log "sglang server tp=${tp} dp=${dp} is healthy on port ${SGLANG_PORT}"
|
||||||
|
}
|
||||||
|
|
||||||
|
restart_server() {
|
||||||
|
local tp="$1"
|
||||||
|
local dp="$2"
|
||||||
|
log "restarting sglang server tp=${tp} dp=${dp} after non-OOM failure"
|
||||||
|
stop_server "$tp" "$dp"
|
||||||
|
sleep 10
|
||||||
|
start_server "$tp" "$dp"
|
||||||
|
}
|
||||||
|
|
||||||
|
run_bench_serving() {
|
||||||
|
# Inject the offline workload choice consistently for warmup and main runs.
|
||||||
|
local -a dataset_args=(--dataset-name "$BENCH_DATASET_NAME")
|
||||||
|
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||||
|
dataset_args+=(--dataset-path "$DATASET_PATH")
|
||||||
|
elif [[ "$BENCH_DATASET_NAME" == "random-ids" ]]; then
|
||||||
|
: # random-ids does not need --tokenize-prompt
|
||||||
|
else
|
||||||
|
dataset_args+=(--tokenize-prompt)
|
||||||
|
fi
|
||||||
|
if [[ "${USE_DOCKER_CLIENT:-1}" == "1" ]]; then
|
||||||
|
local vol_args=()
|
||||||
|
vol_args+=("-v" "${MODEL_PATH}:${MODEL_PATH}:ro")
|
||||||
|
if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
|
||||||
|
vol_args+=("-v" "${DATASET_PATH}:${DATASET_PATH}:ro")
|
||||||
|
fi
|
||||||
|
vol_args+=("-v" "${RESULT_BASE}:${RESULT_BASE}")
|
||||||
|
docker run --rm \
|
||||||
|
--network host \
|
||||||
|
"${vol_args[@]}" \
|
||||||
|
-e PYTHONUNBUFFERED=1 \
|
||||||
|
--entrypoint python3 \
|
||||||
|
"${DOCKER_IMAGE}" \
|
||||||
|
-m "$SGLANG_BENCH_MODULE" "${dataset_args[@]}" "$@"
|
||||||
|
else
|
||||||
|
"$PYTHON" -m "$SGLANG_BENCH_MODULE" "${dataset_args[@]}" "$@"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
export -f run_bench_serving
|
||||||
|
|
||||||
|
run_warmup() {
|
||||||
|
local input_len="$1"
|
||||||
|
local output_len="$2"
|
||||||
|
|
||||||
|
log "warming up (input=${input_len}, output=${output_len}, num=1)"
|
||||||
|
bash -c '
|
||||||
|
run_bench_serving \
|
||||||
|
--backend sglang \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "'"$SGLANG_PORT"'" \
|
||||||
|
--random-input-len "'"$input_len"'" \
|
||||||
|
--random-output-len "'"$output_len"'" \
|
||||||
|
--num-prompts 1 \
|
||||||
|
--max-concurrency 1 \
|
||||||
|
--request-rate 10000 \
|
||||||
|
--output-file /dev/null \
|
||||||
|
--output-details \
|
||||||
|
>> "'"${log_dir_global}/warmup.log"'" 2>&1
|
||||||
|
'
|
||||||
|
log "warmup completed"
|
||||||
|
}
|
||||||
|
|
||||||
|
scenario_already_completed() {
|
||||||
|
local output_file="$1"
|
||||||
|
local expected="$2"
|
||||||
|
[[ -s "$output_file" ]] || return 1
|
||||||
|
local completed
|
||||||
|
completed="$("$PYTHON" -c "
|
||||||
|
import json, sys
|
||||||
|
path = sys.argv[1]
|
||||||
|
try:
|
||||||
|
with open(path, 'r', encoding='utf-8') as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if line:
|
||||||
|
data = json.loads(line)
|
||||||
|
print(data.get('completed', 0))
|
||||||
|
break
|
||||||
|
except Exception:
|
||||||
|
print(0)
|
||||||
|
" "$output_file")"
|
||||||
|
[[ "${completed:-0}" -ge "$expected" ]]
|
||||||
|
}
|
||||||
|
|
||||||
|
scenario_already_processed() {
|
||||||
|
local result_root="$1"
|
||||||
|
local scenario_name="$2"
|
||||||
|
local json_path="${result_root}/results.json"
|
||||||
|
[[ -f "$json_path" ]] || return 1
|
||||||
|
"$PYTHON" -c "
|
||||||
|
import json, sys
|
||||||
|
path, name = sys.argv[1], sys.argv[2]
|
||||||
|
try:
|
||||||
|
with open(path, 'r', encoding='utf-8') as f:
|
||||||
|
data = json.load(f)
|
||||||
|
for s in data.get('scenarios', []):
|
||||||
|
if s.get('name') == name:
|
||||||
|
if s.get('status') or s.get('metrics', {}).get('success', 0) > 0:
|
||||||
|
sys.exit(0)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
sys.exit(1)
|
||||||
|
" "$json_path" "$scenario_name"
|
||||||
|
}
|
||||||
|
|
||||||
|
detect_oom() {
|
||||||
|
local detail_log="$1"
|
||||||
|
local server_outer_log="$2"
|
||||||
|
local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
|
||||||
|
if grep -Eiq "$pattern" "$detail_log" "$server_outer_log" 2>/dev/null; then
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
start_gpu_monitor() {
|
||||||
|
local csv_path="$1"
|
||||||
|
mkdir -p "$(dirname "$csv_path")"
|
||||||
|
nvidia-smi \
|
||||||
|
--query-gpu=timestamp,index,memory.used,memory.total,utilization.gpu \
|
||||||
|
--format=csv \
|
||||||
|
-l "$GPU_MEM_SAMPLE_INTERVAL_S" \
|
||||||
|
> "$csv_path" 2>/dev/null &
|
||||||
|
echo $!
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_gpu_monitor() {
|
||||||
|
local pid="$1"
|
||||||
|
if kill -0 "$pid" 2>/dev/null; then
|
||||||
|
kill "$pid" 2>/dev/null || true
|
||||||
|
sleep 1
|
||||||
|
kill -9 "$pid" 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
append_scenario_record() {
|
||||||
|
local result_root="$1"
|
||||||
|
local json_path="$result_root/results.json"
|
||||||
|
shift
|
||||||
|
local scenario_json
|
||||||
|
scenario_json="$("$PYTHON" -c "
|
||||||
|
import json, sys
|
||||||
|
pairs = [a.split('=', 1) for a in sys.argv[1:]]
|
||||||
|
d = {}
|
||||||
|
for k, v in pairs:
|
||||||
|
try:
|
||||||
|
d[k] = json.loads(v)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
d[k] = v
|
||||||
|
print(json.dumps(d, ensure_ascii=False))
|
||||||
|
" "$@")"
|
||||||
|
PYTHON="$PYTHON" append_scenario_to_json "$json_path" "$scenario_json"
|
||||||
|
}
|
||||||
|
|
||||||
|
record_skipped_csv() {
|
||||||
|
local csv_path="$1"
|
||||||
|
shift
|
||||||
|
# Args: key=value
|
||||||
|
local row
|
||||||
|
row="$("$PYTHON" -c "
|
||||||
|
import csv, json, sys, io
|
||||||
|
pairs = [a.split('=', 1) for a in sys.argv[1:]]
|
||||||
|
d = {}
|
||||||
|
for k, v in pairs:
|
||||||
|
try:
|
||||||
|
d[k] = json.loads(v)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
d[k] = v
|
||||||
|
buf = io.StringIO()
|
||||||
|
writer = csv.DictWriter(buf, fieldnames=['engine','tp','dp','mark','isl','dsl','concurrency','status','reason','detail_log'], extrasaction='ignore')
|
||||||
|
writer.writerow(d)
|
||||||
|
print(buf.getvalue().strip())
|
||||||
|
" "$@")"
|
||||||
|
echo "$row" >> "$csv_path"
|
||||||
|
}
|
||||||
|
|
||||||
|
skip_remaining_scenarios() {
|
||||||
|
local result_root="$1"
|
||||||
|
local scenario_tsv="$2"
|
||||||
|
local start_index="$3"
|
||||||
|
local status="$4"
|
||||||
|
local reason="$5"
|
||||||
|
local tp="$6"
|
||||||
|
local dp="$7"
|
||||||
|
local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
|
||||||
|
|
||||||
|
local i=0
|
||||||
|
tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl dsl conc num; do
|
||||||
|
if (( i < start_index )); then
|
||||||
|
i=$((i + 1))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
i=$((i + 1))
|
||||||
|
local sname="c${conc}_i${isl}_o${dsl}"
|
||||||
|
if scenario_already_processed "$result_root" "$sname"; then
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
append_scenario_record "$result_root" \
|
||||||
|
"name=${sname}" \
|
||||||
|
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset "$BENCH_DATASET_NAME" --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||||
|
"status=\"${status}\"" \
|
||||||
|
"note=\"${reason}\""
|
||||||
|
record_skipped_csv "$skipped_csv" \
|
||||||
|
"engine=sglang" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=${status}" "reason=${reason}"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Per-configuration runner
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
run_parallel_config() {
|
||||||
|
local tp="$1"
|
||||||
|
local dp="$2"
|
||||||
|
local config_label="tp${tp}_dp${dp}"
|
||||||
|
local result_root="${RESULT_BASE}/${RUN_ID}/${config_label}"
|
||||||
|
local raw_dir="${result_root}/raw_outputs"
|
||||||
|
local gpu_log_dir="${result_root}/gpu_logs"
|
||||||
|
local phase_log_dir="${result_root}/logs"
|
||||||
|
mkdir -p "$raw_dir" "$gpu_log_dir" "$phase_log_dir"
|
||||||
|
|
||||||
|
log "===== ${config_label} START ====="
|
||||||
|
|
||||||
|
# Generate scenario list for this config.
|
||||||
|
local scenario_tsv="${result_root}/scenarios.tsv"
|
||||||
|
"$PYTHON" "${SCRIPT_DIR}/generate_scenarios.py" \
|
||||||
|
--matrix "$MATRIX_FILE" \
|
||||||
|
--mode "$MATRIX_MODE" \
|
||||||
|
> "$scenario_tsv"
|
||||||
|
local total_scenarios
|
||||||
|
total_scenarios="$(tail -n +2 "$scenario_tsv" | wc -l)"
|
||||||
|
log "generated ${total_scenarios} scenarios for ${config_label}"
|
||||||
|
|
||||||
|
# Write metadata.
|
||||||
|
ensure_result_root "$result_root"
|
||||||
|
write_metadata_json \
|
||||||
|
"${result_root}/results.json" \
|
||||||
|
"${EXPERIMENT_NAME}_${config_label}" \
|
||||||
|
"$RUN_ID" \
|
||||||
|
"$MODEL_PATH" \
|
||||||
|
"sglang" \
|
||||||
|
"sglang" \
|
||||||
|
"$HARDWARE" \
|
||||||
|
"$ACCELERATOR" \
|
||||||
|
"$CHIP" \
|
||||||
|
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||||
|
"$DOCKER_IMAGE" \
|
||||||
|
"Kimi-K3 TP32×EP32 matrix on 4x RTX 6000D"
|
||||||
|
|
||||||
|
local server_args_str
|
||||||
|
server_args_str="$(build_server_args "$tp" "$dp")"
|
||||||
|
jq --arg tp "$tp" --arg dp "$dp" --arg cuda "$CUDA_VISIBLE_DEVICES" --arg args "$server_args_str" \
|
||||||
|
'.config = {
|
||||||
|
"tp": ($tp | tonumber),
|
||||||
|
"dp": ($dp | tonumber),
|
||||||
|
"cuda_visible_devices": $cuda,
|
||||||
|
"backend": "sglang",
|
||||||
|
"server_start_script": "experiments/'${EXPERIMENT_NAME}'/start_sglang_dp.sh",
|
||||||
|
"server_args": $args
|
||||||
|
}' "${result_root}/results.json" > "${result_root}/results.json.tmp" && \
|
||||||
|
mv "${result_root}/results.json.tmp" "${result_root}/results.json"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == "1" ]]; then
|
||||||
|
log "DRY_RUN: would start server with args: ${server_args_str}"
|
||||||
|
local line
|
||||||
|
tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl dsl conc num; do
|
||||||
|
log "DRY_RUN: ${config_label} scenario mark=${mark} c=${conc} i=${isl} o=${dsl} n=${num}"
|
||||||
|
done
|
||||||
|
log "===== ${config_label} DONE (dry run) ====="
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Initialize skipped_after_oom.csv for this run.
|
||||||
|
local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
|
||||||
|
if [[ ! -f "$skipped_csv" ]]; then
|
||||||
|
echo "engine,tp,dp,mark,isl,dsl,concurrency,status,reason,detail_log" > "$skipped_csv"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Start server once for this TP×DP config.
|
||||||
|
if ! start_server "$tp" "$dp"; then
|
||||||
|
log "ERROR: ${config_label} failed to start; skipping all scenarios"
|
||||||
|
skip_remaining_scenarios "$result_root" "$scenario_tsv" 0 "SKIPPED_SERVICE_START_FAILED" "service failed to start" "$tp" "$dp"
|
||||||
|
log "===== ${config_label} DONE ====="
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Warmup with a small prompt before the first scenario.
|
||||||
|
run_warmup 1024 128 || true
|
||||||
|
|
||||||
|
# Read scenarios into an array so we can skip remaining entries on failure.
|
||||||
|
local -a scenarios=()
|
||||||
|
while IFS= read -r line; do
|
||||||
|
scenarios+=("$line")
|
||||||
|
done < <(tail -n +2 "$scenario_tsv")
|
||||||
|
|
||||||
|
local i mark isl dsl conc num
|
||||||
|
local output_file detail_log gpu_csv sname bench_rc
|
||||||
|
for (( i = 0; i < ${#scenarios[@]}; i++ )); do
|
||||||
|
IFS=$'\t' read -r mark isl dsl conc num <<< "${scenarios[$i]}"
|
||||||
|
|
||||||
|
if [[ "$GRID_LIMIT" -gt 0 && "$i" -ge "$GRID_LIMIT" ]]; then
|
||||||
|
log "GRID_LIMIT=${GRID_LIMIT} reached; skipping remaining scenarios"
|
||||||
|
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$i" "SKIPPED_GRID_LIMIT" "GRID_LIMIT reached" "$tp" "$dp"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
|
||||||
|
sname="c${conc}_i${isl}_o${dsl}"
|
||||||
|
output_file="${raw_dir}/sglang_main_${conc}_${isl}_${dsl}.jsonl"
|
||||||
|
detail_log="${phase_log_dir}/sglang_${config_label}_${sname}.log"
|
||||||
|
gpu_csv="${gpu_log_dir}/gpu_mem_${conc}_${isl}_${dsl}.csv"
|
||||||
|
|
||||||
|
if scenario_already_completed "$output_file" "$num" || scenario_already_processed "$result_root" "$sname"; then
|
||||||
|
log "skipping already-processed ${config_label} scenario: ${sname}"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "running ${config_label} scenario: mark=${mark} c=${conc} i=${isl} o=${dsl} n=${num}"
|
||||||
|
|
||||||
|
local gpu_pid
|
||||||
|
gpu_pid="$(start_gpu_monitor "$gpu_csv")"
|
||||||
|
|
||||||
|
bench_rc=0
|
||||||
|
timeout "$SCENARIO_TIMEOUT_S" bash -c '
|
||||||
|
run_bench_serving \
|
||||||
|
--backend sglang \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "'"$SGLANG_PORT"'" \
|
||||||
|
--random-input-len "'"$isl"'" \
|
||||||
|
--random-output-len "'"$dsl"'" \
|
||||||
|
--num-prompts "'"$num"'" \
|
||||||
|
--max-concurrency "'"$conc"'" \
|
||||||
|
--request-rate 10000 \
|
||||||
|
--output-file "'"$output_file"'" \
|
||||||
|
--output-details \
|
||||||
|
> "'"$detail_log"'" 2>&1
|
||||||
|
' || bench_rc=$?
|
||||||
|
|
||||||
|
stop_gpu_monitor "$gpu_pid"
|
||||||
|
|
||||||
|
if [[ "$bench_rc" -eq 0 ]]; then
|
||||||
|
log "finished ${config_label} scenario: output=${output_file}"
|
||||||
|
append_scenario_record "$result_root" \
|
||||||
|
"name=${sname}" \
|
||||||
|
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset "$BENCH_DATASET_NAME" --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||||
|
"status=\"completed\"" \
|
||||||
|
"note=\"benchmark finished successfully\""
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Failure handling.
|
||||||
|
if detect_oom "$detail_log" "${log_dir_global}/sglang_tp${tp}_dp${dp}.server.outer.log"; then
|
||||||
|
log "ERROR: ${config_label} scenario ${sname} triggered OOM; stopping config"
|
||||||
|
append_scenario_record "$result_root" \
|
||||||
|
"name=${sname}" \
|
||||||
|
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset "$BENCH_DATASET_NAME" --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||||
|
"status=\"OOM\"" \
|
||||||
|
"note=\"detected CUDA out-of-memory\""
|
||||||
|
record_skipped_csv "$skipped_csv" \
|
||||||
|
"engine=sglang" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=OOM" "reason=detected CUDA out-of-memory" "detail_log=${detail_log}"
|
||||||
|
stop_server "$tp" "$dp"
|
||||||
|
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_AFTER_OOM" "previous case OOM" "$tp" "$dp"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "ERROR: ${config_label} scenario ${sname} failed (rc=${bench_rc}); see ${detail_log}"
|
||||||
|
if [[ "$mark" == "P" ]]; then
|
||||||
|
log "optional (P) scenario failed; recording as skipped and continuing"
|
||||||
|
append_scenario_record "$result_root" \
|
||||||
|
"name=${sname}" \
|
||||||
|
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset "$BENCH_DATASET_NAME" --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||||
|
"status=\"skipped_optional\"" \
|
||||||
|
"note=\"optional scenario failed (rc=${bench_rc})\""
|
||||||
|
record_skipped_csv "$skipped_csv" \
|
||||||
|
"engine=sglang" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=skipped_optional" "reason=optional scenario failed (rc=${bench_rc})" "detail_log=${detail_log}"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Mandatory scenario failed but not OOM: try to restart the server.
|
||||||
|
if restart_server "$tp" "$dp"; then
|
||||||
|
run_warmup 1024 128 || true
|
||||||
|
log "resuming ${config_label} after server restart"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "ERROR: ${config_label} server restart failed; skipping remaining scenarios"
|
||||||
|
append_scenario_record "$result_root" \
|
||||||
|
"name=${sname}" \
|
||||||
|
"config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$dsl" --arg dataset "$BENCH_DATASET_NAME" --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
|
||||||
|
"status=\"FAILED\"" \
|
||||||
|
"note=\"scenario failed and server restart failed (rc=${bench_rc})\""
|
||||||
|
record_skipped_csv "$skipped_csv" \
|
||||||
|
"engine=sglang" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "dsl=${dsl}" "concurrency=${conc}" "status=FAILED" "reason=scenario failed and server restart failed" "detail_log=${detail_log}"
|
||||||
|
skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_RESTART_FAILED" "server restart failed" "$tp" "$dp"
|
||||||
|
break
|
||||||
|
done
|
||||||
|
|
||||||
|
stop_server "$tp" "$dp"
|
||||||
|
|
||||||
|
# Parse results.
|
||||||
|
log "parsing ${config_label} results"
|
||||||
|
"$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/parse_backend.py" "$result_root" --backend sglang \
|
||||||
|
>> "${phase_log_dir}/parse.log" 2>&1 || {
|
||||||
|
log "WARNING: parser failed for ${config_label}; see ${phase_log_dir}/parse.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
log "===== ${config_label} DONE ====="
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Cleanup any leftovers.
|
||||||
|
for cfg in "${PARALLEL_CONFIGS[@]}"; do
|
||||||
|
read -r tp dp <<< "$cfg"
|
||||||
|
stop_server "$tp" "$dp"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Run each parallel configuration.
|
||||||
|
for cfg in "${PARALLEL_CONFIGS[@]}"; do
|
||||||
|
read -r tp dp <<< "$cfg"
|
||||||
|
run_parallel_config "$tp" "$dp"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Generate cross-configuration comparison.
|
||||||
|
log "generating comparison report"
|
||||||
|
"$PYTHON" "${SCRIPT_DIR}/compare.py" \
|
||||||
|
--run-root "${RESULT_BASE}/${RUN_ID}" \
|
||||||
|
--output "${RESULT_BASE}/${RUN_ID}/comparison.md" \
|
||||||
|
>> "${log_dir_global}/compare.log" 2>&1 || {
|
||||||
|
log "WARNING: comparison script failed; see ${log_dir_global}/compare.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
log "all results saved to ${RESULT_BASE}/${RUN_ID}"
|
||||||
33
experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/start_sglang_docker.sh
Executable file
33
experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/start_sglang_docker.sh
Executable file
@ -0,0 +1,33 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Start the SGLang TPxDP server through the shared deployment layer.
|
||||||
|
# Usage: start_sglang_docker.sh <TP> <DP>
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
TP="${1:-}"
|
||||||
|
DP="${2:-}"
|
||||||
|
if [[ -z "$TP" || -z "$DP" ]]; then
|
||||||
|
echo "Usage: $0 <TP> <DP>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||||
|
|
||||||
|
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||||
|
mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||||
|
|
||||||
|
log "starting SGLang server tp=${TP} dp=${DP} via deploy profile"
|
||||||
|
deploy_start \
|
||||||
|
"${DEPLOY_PROFILE:-pro6000/kimi3_pro6000_sglang_tp32ep32}" \
|
||||||
|
"$TP" "$DP" \
|
||||||
|
"${RUNTIME_BASE}/logs" \
|
||||||
|
"${SGLANG_PORT:-30000}" \
|
||||||
|
"$MODEL_PATH" \
|
||||||
|
"${EXPERIMENT}_sglang_tp${TP}_dp${DP}"
|
||||||
87
experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/start_sglang_dp.sh
Executable file
87
experiments/pro6000/kimi3_pro6000_sglang_tp32ep32/start_sglang_dp.sh
Executable file
@ -0,0 +1,87 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Start SGLang server for a given TP×DP configuration.
|
||||||
|
# Usage: start_sglang_dp.sh <TP> <DP>
|
||||||
|
#
|
||||||
|
# By default this delegates to the Docker start script because the experiment
|
||||||
|
# is intended to run SGLang inside a container. Set USE_DOCKER=0 to use the
|
||||||
|
# local VENV_CLIENT environment instead.
|
||||||
|
set -e
|
||||||
|
|
||||||
|
TP="${1}"
|
||||||
|
DP="${2}"
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
|
||||||
|
if [[ "${USE_DOCKER:-1}" == "1" ]]; then
|
||||||
|
exec "${SCRIPT_DIR}/start_sglang_docker.sh" "$@"
|
||||||
|
fi
|
||||||
|
|
||||||
|
RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
|
||||||
|
mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||||
|
|
||||||
|
VENV="${VENV_CLIENT}"
|
||||||
|
export PATH="$VENV/bin:$PATH"
|
||||||
|
export PYTHONUNBUFFERED=1
|
||||||
|
export PYTORCH_CUDA_ALLOC_CONF="${PYTORCH_CUDA_ALLOC_CONF:-expandable_segments:True}"
|
||||||
|
export TMPDIR="${RUNTIME_BASE}/tmp"
|
||||||
|
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES}"
|
||||||
|
|
||||||
|
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_sglang_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_sglang_tp${TP}_dp${DP}.pid"
|
||||||
|
|
||||||
|
rm -f "$PID_FILE"
|
||||||
|
|
||||||
|
SERVER_ARGS=(
|
||||||
|
python3 -m sglang.launch_server
|
||||||
|
--model-path "$MODEL_PATH"
|
||||||
|
--trust-remote-code
|
||||||
|
--tp-size "$TP"
|
||||||
|
--moe-runner-backend "$MOE_RUNNER_BACKEND"
|
||||||
|
--mem-fraction-static "$MEM_FRACTION_STATIC"
|
||||||
|
--context-length "$CONTEXT_LENGTH"
|
||||||
|
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||||
|
--host 0.0.0.0
|
||||||
|
--port "$SGLANG_PORT"
|
||||||
|
)
|
||||||
|
|
||||||
|
if [[ "$DP" -gt 1 ]]; then
|
||||||
|
SERVER_ARGS+=(
|
||||||
|
--dp-size "$DP"
|
||||||
|
)
|
||||||
|
fi
|
||||||
|
|
||||||
|
SERVER_ARGS_STR="${SERVER_ARGS[*]}"
|
||||||
|
|
||||||
|
echo "=== Starting SGLang server (TP=${TP}, DP=${DP}) ==="
|
||||||
|
echo "Model: $MODEL_PATH"
|
||||||
|
echo "Port: $SGLANG_PORT"
|
||||||
|
echo "Command: $SERVER_ARGS_STR"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
|
||||||
|
nohup "${SERVER_ARGS[@]}" > "$LOG" 2>&1 &
|
||||||
|
|
||||||
|
PID=$!
|
||||||
|
echo $PID > "$PID_FILE"
|
||||||
|
echo "PID: $PID"
|
||||||
|
echo "Waiting for health on port ${SGLANG_PORT}..."
|
||||||
|
|
||||||
|
for i in $(seq 1 240); do
|
||||||
|
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${SGLANG_PORT}/health" >/dev/null 2>&1; then
|
||||||
|
echo "SGLang server is ready at http://127.0.0.1:${SGLANG_PORT}"
|
||||||
|
echo "Log: $LOG"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
if ! kill -0 $PID 2>/dev/null; then
|
||||||
|
echo "ERROR: SGLang server exited early"
|
||||||
|
tail -200 "$LOG"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "Waiting... ($i/240)"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "ERROR: SGLang server not healthy after 240 retries"
|
||||||
|
tail -200 "$LOG"
|
||||||
|
exit 1
|
||||||
@ -0,0 +1,29 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Stop the SGLang TPxDP server through the shared deployment layer.
|
||||||
|
# Usage: stop_sglang_docker.sh <TP> <DP>
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
TP="${1:-}"
|
||||||
|
DP="${2:-}"
|
||||||
|
if [[ -z "$TP" || -z "$DP" ]]; then
|
||||||
|
echo "Usage: $0 <TP> <DP>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../../scripts/common/deploy_cli.sh"
|
||||||
|
|
||||||
|
log "stopping SGLang server tp=${TP} dp=${DP} via deploy profile"
|
||||||
|
deploy_stop \
|
||||||
|
"${DEPLOY_PROFILE:-pro6000/kimi3_pro6000_sglang_tp32ep32}" \
|
||||||
|
"$TP" "$DP" \
|
||||||
|
"${SGLANG_PORT:-30000}" \
|
||||||
|
"$MODEL_PATH" \
|
||||||
|
"${EXPERIMENT}_sglang_tp${TP}_dp${DP}"
|
||||||
8
platforms/patches/pro6000/kimi_k3/patch_k3_sm120.py
Normal file
8
platforms/patches/pro6000/kimi_k3/patch_k3_sm120.py
Normal file
@ -0,0 +1,8 @@
|
|||||||
|
"""Patch sglang kimi-k3 image: fall back from tcgen05 fused-TMA attn_res on sm_120."""
|
||||||
|
p = "/sgl-workspace/sglang/python/sglang/srt/layers/attn_residual.py"
|
||||||
|
s = open(p).read()
|
||||||
|
old = " major, _ = torch.cuda.get_device_capability()\n _FAST_SUPPORTED = major >= 10"
|
||||||
|
new = " major, _ = torch.cuda.get_device_capability()\n # RTX 6000D is sm_120 (major 12): tcgen05 exists only on GB100/GB200/GB300\n # (majors 10/11), so sm_120 must take the Triton score/combine pipeline.\n _FAST_SUPPORTED = major in (10, 11)"
|
||||||
|
assert old in s, "pattern not found in attn_residual.py"
|
||||||
|
open(p, "w").write(s.replace(old, new))
|
||||||
|
print("OK patched attn_residual._use_fast -> majors (10,11)")
|
||||||
Loading…
x
Reference in New Issue
Block a user