ddf807d458
- sskj.bench: 新增 --flush-cache 参数，透传给 sglang.bench_serving （warmup 后、main run 前 flush 服务器 KV/prefix cache，保证 TTFT/TPOT 测量纯净） - P/D PD profiles: LAUNCH_ARGS 加 --disable-radix-cache（关闭 RadixAttention 前缀缓存）
61 lines
3.6 KiB
Bash
61 lines
3.6 KiB
Bash
# Kimi-K3 PD 分离 - P 组 (prefill) deployment profile (4x RTX 6000D).
|
||
# Nodes: 174.1.60.1~4 (rank 0~3), 32x NVIDIA RTX 6000D (85GB, sm_120).
|
||
#
|
||
# 这是 PD (Prefill/Decode) 分离部署的 P 组 (prefill) 端 profile。
|
||
# 配套 D 组 profile: kimi3_pro6000_pd_decode.env
|
||
# 配套编排脚本: experiments/pro6000/kimi3_pro6000_pd_rdma/deploy_pd.sh
|
||
#
|
||
# 关键点（实测踩坑，勿随意改）:
|
||
# - 传输后端 mooncake + MOONCAKE_PROTOCOL=rdma（计算网 mlx5_0~3，4 链路）
|
||
# - 必须关掉 PYTORCH_CUDA_ALLOC_CONF=expandable_segments
|
||
# （实测: expandable_segments 分配的 GPU 段 mooncake RDMA 注册失败
|
||
# Bad address [14]，见飞书文档 PD 章节 & GitHub kvcache-ai/Mooncake#2511）
|
||
# - --disaggregation-ib-device mlx5_0~3 让 mooncake 走计算网 RDMA
|
||
# - NCCL/GLOO 走 bond1（计算网），NCCL_IB_HCA=mlx5_0..3
|
||
# - P 组先启动，D 组后启动（见 deploy_pd.sh）
|
||
# - mooncake master 需先在 174.1.60.1 运行（MOONCAKE_MASTER=174.1.60.1:50051）
|
||
# - 容器挂载 /mc_wheels 并在 BOOTSTRAP 里 pip install mooncake 0.3.12.post1
|
||
# （镜像自带 0.3.11.post1 无 dmabuf 修复，必须升级）
|
||
#
|
||
# Model-team only. Ops only run benchmark against the served router URL.
|
||
|
||
PLATFORM=pro6000
|
||
EXPERIMENT=kimi3_pro6000_pd_prefill
|
||
MODEL_NAME=Kimi-K3
|
||
ENGINE=sglang
|
||
RUNTIME=docker
|
||
DOCKER_IMAGE=lmsysorg/sglang:kimi-k3
|
||
CONTAINER_NAME=${EXPERIMENT}_node${NODE_RANK}
|
||
MODEL_PATH=/data/hf_models/Kimi-K3
|
||
SERVED_MODEL_NAME=kimi-k3
|
||
PORT=30000
|
||
HEALTH_PATH=/health
|
||
HEALTH_HOST=174.1.60.1
|
||
HEALTH_WAIT_S=2400
|
||
CONTAINER_PYTHON=python3
|
||
|
||
# ---- 并行度（固定，勿改）----
|
||
TP=32
|
||
DP=1
|
||
EP_SIZE=32
|
||
|
||
# ---- Multi-node topology (P 组 = prefill 4 节点) ----
|
||
NNODES=4
|
||
NODE_HOSTS="174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4"
|
||
NODE_SSH_USER=root
|
||
LOCAL_NODE_RANK=0
|
||
MASTER_IP=174.1.60.1
|
||
DIST_PORT=20000
|
||
|
||
DEVICE_VARS="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
|
||
ENGINE_ENV="MOONCAKE_MASTER=174.1.60.1:50051 MOONCAKE_PROTOCOL=rdma NCCL_SOCKET_IFNAME=bond1 GLOO_SOCKET_IFNAME=bond1 NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 NCCL_IB_GID_INDEX=3 NCCL_IB_TIMEOUT=22 NCCL_IB_RETRY_CNT=7 NCCL_CUMEM_ENABLE=1 SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 SGLANG_MOE_FUSED_GATE_RADIX=1"
|
||
|
||
DOCKER_FLAGS="--gpus all --network host --ipc=host --ulimit memlock=-1 --device /dev/infiniband --shm-size 32g --entrypoint ''"
|
||
VOLUMES="${MODEL_PATH}:${MODEL_PATH}:ro"
|
||
PATCH_MOUNTS="/tmp/patch_k3_sm120.py:/tmp/patch_k3_sm120.py:ro /tmp/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:/flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl:ro /data/flashkda_deploy/wheels:/mc_wheels:ro"
|
||
|
||
# BOOTSTRAP: 打补丁 → 装 flashkda + mooncake wheel → 按 rank 设 SGLANG_HOST_IP → 启动 prefill。
|
||
BOOTSTRAP="python3 /tmp/patch_k3_sm120.py && pip install /flash_kda-0.0.1-cp312-cp312-linux_x86_64.whl --no-deps -q && pip install /mc_wheels/mooncake_transfer_engine_cuda13-0.3.12.post1-cp312-cp312-manylinux_2_28_x86_64.whl --no-deps -q && export SGLANG_HOST_IP=\"174.1.60.$((1 + ${NODE_RANK}))\" && exec python3 -m sglang.launch_server ${LAUNCH_ARGS}"
|
||
|
||
LAUNCH_ARGS="--model-path ${MODEL_PATH} --served-model-name ${SERVED_MODEL_NAME} --tp-size ${TP} --ep-size 32 --nnodes ${NNODES} --node-rank ${NODE_RANK} --dist-init-addr ${MASTER_IP}:${DIST_PORT} --trust-remote-code --moe-runner-backend marlin --mem-fraction-static 0.88 --cuda-graph-max-bs-decode 16 --mamba-radix-cache-strategy extra_buffer --disable-radix-cache --dist-timeout 3600 --mamba-full-memory-ratio 0.36 --disaggregation-transfer-backend mooncake --disaggregation-bootstrap-port 28800 --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 --disaggregation-mode prefill --linear-attn-prefill-backend flashkda --host 0.0.0.0 --port ${PORT}"
|