benchmark Kimi-K3 DP2 TP32 and PP8 without speculation
This commit is contained in:
Zhiyi Hong
parent
c9305fb830
commit
411802e4b1
85
docs/Kimi-K3_DP2_TP32_PP8_16K512_C8对比.md
Normal file
85
docs/Kimi-K3_DP2_TP32_PP8_16K512_C8对比.md
Normal file
@ -0,0 +1,85 @@
|
|||||||
|
# Kimi-K3 DP2 无投机：TP32 与 PP8 对比
|
||||||
|
|
||||||
|
## 测试目的
|
||||||
|
|
||||||
|
在 8 台 RTX 6000D 上部署两个独立四节点 SGLang 副本，通过 SGLang Router `round_robin` 组成 DP2，比较两种单副本并行策略：
|
||||||
|
|
||||||
|
- TP32：`TP32/PP1/EP4`
|
||||||
|
- PP8：`TP4/PP8/EP4`
|
||||||
|
|
||||||
|
本轮不启用投机解码，重点观察长输入场景下 TTFT 与长 Decode 的取舍。
|
||||||
|
|
||||||
|
## 固定口径
|
||||||
|
|
||||||
|
| 项目 | 配置 |
|
||||||
|
|---|---|
|
||||||
|
| 数据集 | SPEED-Bench `/data/speedbench16k_complete.jsonl` |
|
||||||
|
| 请求 | 40 条，16K 输入，512 输出 |
|
||||||
|
| 客户端并发 | C=8 |
|
||||||
|
| Chunked Prefill | 8K |
|
||||||
|
| MoE | EP4，FlashInfer MXFP4，A2A=none |
|
||||||
|
| KV Cache | BF16 |
|
||||||
|
| Prefix Cache | 关闭 |
|
||||||
|
| Router | 两副本 `round_robin` |
|
||||||
|
| GPU | 601-608，共 64 卡 |
|
||||||
|
|
||||||
|
两种配置均在正式计时前分别预热两个副本，并由 benchmark 再执行一条标准 warmup。`Peak concurrent requests=12` 是 SGLang 报告的服务端峰值，可因排队高于客户端 `max_concurrency=8`，不表示客户端发出了 12 路并发。
|
||||||
|
|
||||||
|
## 结果
|
||||||
|
|
||||||
|
| 指标 | DP2 TP32/PP1/EP4 | DP2 TP4/PP8/EP4 | PP8 相对 TP32 |
|
||||||
|
|---|---:|---:|---:|
|
||||||
|
| 成功请求 | 40/40 | 40/40 | 相同 |
|
||||||
|
| Benchmark duration | 238.18s | 370.30s | +55.47% |
|
||||||
|
| Input TPS | 2797.64 | 1799.46 | -35.68% |
|
||||||
|
| Output TPS | 85.99 | 55.31 | -35.68% |
|
||||||
|
| Total TPS | 2883.63 | 1854.76 | -35.68% |
|
||||||
|
| TTFT mean | 8.35s | 7.84s | -6.15% |
|
||||||
|
| TTFT median | 5.30s | 7.03s | +32.62% |
|
||||||
|
| TTFT p95 | 20.79s | 11.68s | -43.82% |
|
||||||
|
| TPOT mean | 75.93ms | 126.86ms | +67.07% |
|
||||||
|
| TPOT median | 79.95ms | 136.09ms | +70.22% |
|
||||||
|
| TPOT p95 | 88.06ms | 145.13ms | +64.80% |
|
||||||
|
| E2E mean | 47.15s | 72.66s | +54.10% |
|
||||||
|
| E2E p95 | 59.24s | 82.83s | +39.82% |
|
||||||
|
|
||||||
|
## 初步结论
|
||||||
|
|
||||||
|
PP8 继续体现了 Prefill 优势：平均 TTFT 小幅下降 6.15%，TTFT p95 降低 43.82%，长尾更稳定。这来自 TP 通信域从 32 缩到 4，并由流水线分摊模型层。
|
||||||
|
|
||||||
|
但 512-token Decode 暴露了 PP8 的代价。每个 token 都要依次通过 8 个 Pipeline Stage，TPOT 增加 67.07%，最终使 Output TPS 降低 35.68%、E2E 平均时延增加 54.10%。因此：
|
||||||
|
|
||||||
|
- 以 P 节点 TTFT 为目标时，PP8 有价值，尤其能改善 TTFT 长尾。
|
||||||
|
- 作为不做 PD 分离的完整 16K→512 服务，TP32 综合表现更好。
|
||||||
|
- PP8 更适合作为 PD 分离中的 P 侧配置，由独立 D 侧承担长 Decode；不能用本轮 PP8 的 E2E/TPOT 直接否定其 P 侧价值。
|
||||||
|
|
||||||
|
TP32 的 TTFT median 更低但 p95 明显更高，说明该配置在 DP2 排队与长 Prefill 交错时波动更大。后续如继续优化单实例，应分别报告 median 与 p95，不能只看 mean。
|
||||||
|
|
||||||
|
## 数据质量与日志说明
|
||||||
|
|
||||||
|
两轮 Router 初始命令没有挂载模型目录，因此 worker tokenizer 注册出现错误。Router 使用固定 `round_robin`，不依赖 tokenizer 做 cache-aware 路由；40/40 请求成功，两副本 Prefill 数量接近均分，未发现请求级失败或重试，所以不影响本轮 A/B 结论。脚本已补上 Router 模型只读挂载，后续运行不会再产生该注册错误。
|
||||||
|
|
||||||
|
满负载期间 Router 偶有 `/health` 超时，这是模型服务忙于长 Prefill/Decode 时健康响应延迟；两轮均未出现 OOM、NCCL collective failure、Traceback 或 EngineDeadError。
|
||||||
|
|
||||||
|
## 原始结果
|
||||||
|
|
||||||
|
TP32：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-tp32-final-20260902-123051/
|
||||||
|
```
|
||||||
|
|
||||||
|
PP8：
|
||||||
|
|
||||||
|
```text
|
||||||
|
/data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-pp8-final-20260902-130711/
|
||||||
|
```
|
||||||
|
|
||||||
|
核心 benchmark 日志分别位于：
|
||||||
|
|
||||||
|
```text
|
||||||
|
bench/tp32_speedbench16k_c8_o512.log
|
||||||
|
bench/pp8_speedbench16k_c8_o512.log
|
||||||
|
```
|
||||||
|
|
||||||
|
每个结果目录同时保存完整启动命令、环境 manifest、输入校验值、Router 日志和八节点服务日志。
|
||||||
36
experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/README.md
Normal file
36
experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/README.md
Normal file
@ -0,0 +1,36 @@
|
|||||||
|
# Kimi-K3 DP2 无投机 A/B
|
||||||
|
|
||||||
|
两个独立四节点 SGLang 副本通过 SGLang Router 做 round-robin 负载均衡。固定 `EP4`，比较：
|
||||||
|
|
||||||
|
- `tp32`：每副本 `TP32/PP1/EP4`
|
||||||
|
- `pp8`：每副本 `TP4/PP8/EP4`
|
||||||
|
|
||||||
|
测试口径固定为 SPEED-Bench、`16K -> 512`、`C=8`、40 请求、chunk 8K、BF16 KV、FlashInfer MXFP4，不启用投机解码和 Prefix Cache。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab
|
||||||
|
|
||||||
|
# 只展开命令
|
||||||
|
bash run_dp2_nospec_ab.sh dry-run tp32
|
||||||
|
bash run_dp2_nospec_ab.sh dry-run pp8
|
||||||
|
|
||||||
|
# 完整实验
|
||||||
|
RUN_ID=dp2-tp32-$(date +%Y%m%d-%H%M%S) bash run_dp2_nospec_ab.sh all tp32
|
||||||
|
RUN_ID=dp2-pp8-$(date +%Y%m%d-%H%M%S) bash run_dp2_nospec_ab.sh all pp8
|
||||||
|
```
|
||||||
|
|
||||||
|
每轮结果保存在 `results/<RUN_ID>/`。正式计时前，脚本分别给两个副本发送一个 `16K -> 1` 请求，避免 Router 只预热到一个副本。
|
||||||
|
|
||||||
|
## 2026-09-02 正式结果
|
||||||
|
|
||||||
|
| 模式 | Input TPS | Output TPS | TTFT mean / p95 | TPOT mean / p95 | E2E mean |
|
||||||
|
|---|---:|---:|---:|---:|---:|
|
||||||
|
| `2 x TP32/PP1/EP4` | 2797.64 | 85.99 | 8.35s / 20.79s | 75.93ms / 88.06ms | 47.15s |
|
||||||
|
| `2 x TP4/PP8/EP4` | 1799.46 | 55.31 | 7.84s / 11.68s | 126.86ms / 145.13ms | 72.66s |
|
||||||
|
|
||||||
|
证据目录：
|
||||||
|
|
||||||
|
- `results/dp2-tp32-final-20260902-123051/`
|
||||||
|
- `results/dp2-pp8-final-20260902-130711/`
|
||||||
|
|
||||||
|
两轮均为 40/40 请求成功。Router 使用固定 `round_robin`，两副本负载接近均分。正式运行后补充了 Router 的模型目录挂载，避免后续 tokenizer 注册产生与本轮 round-robin 转发无关的错误日志。
|
||||||
27
experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/config.env
Normal file
27
experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/config.env
Normal file
@ -0,0 +1,27 @@
|
|||||||
|
EXPERIMENT_BASE=kimi3_pro6000_dp2_nospec_ab
|
||||||
|
IMAGE=local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1
|
||||||
|
MODEL_PATH=/data/hf_models/Kimi-K3
|
||||||
|
|
||||||
|
REPLICA_A_NODES=(174.1.60.1 174.1.60.2 174.1.60.3 174.1.60.4)
|
||||||
|
REPLICA_B_NODES=(174.1.60.5 174.1.60.6 174.1.60.7 174.1.60.8)
|
||||||
|
SSH_USER=user
|
||||||
|
SSH_OPTIONS=(-o BatchMode=yes -o ConnectTimeout=10)
|
||||||
|
|
||||||
|
SERVER_PORT=30000
|
||||||
|
DIST_PORT=20000
|
||||||
|
ROUTER_PORT=32000
|
||||||
|
EP_SIZE=4
|
||||||
|
CHUNK_SIZE=8192
|
||||||
|
PAGE_SIZE=64
|
||||||
|
KV_DTYPE=bfloat16
|
||||||
|
MAX_RUNNING_REQUESTS=8
|
||||||
|
CUDA_GRAPH_MAX_BS=8
|
||||||
|
IB_DEVICES=mlx5_0,mlx5_1,mlx5_2,mlx5_3
|
||||||
|
NCCL_DEBUG_LEVEL=${NCCL_DEBUG_LEVEL:-WARN}
|
||||||
|
JIT_CACHE=/data/hzy/cache/kimi-dflash-fi0618-situ4460
|
||||||
|
HEALTH_WAIT_S=2400
|
||||||
|
|
||||||
|
SPEED_BENCH_DATASET=/data/speedbench16k_complete.jsonl
|
||||||
|
NUM_PROMPTS=40
|
||||||
|
OUTPUT_LEN=512
|
||||||
|
CONCURRENCY=8
|
||||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_a_0 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.1 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 4 --pp-size 8 --ep-size 4 --nnodes 4 --node-rank 0 --dist-init-addr 174.1.60.1:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.84 --mamba-full-memory-ratio 0.36 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_a_1 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.2 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 4 --pp-size 8 --ep-size 4 --nnodes 4 --node-rank 1 --dist-init-addr 174.1.60.1:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.84 --mamba-full-memory-ratio 0.36 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_a_2 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.3 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 4 --pp-size 8 --ep-size 4 --nnodes 4 --node-rank 2 --dist-init-addr 174.1.60.1:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.84 --mamba-full-memory-ratio 0.36 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_a_3 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.4 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 4 --pp-size 8 --ep-size 4 --nnodes 4 --node-rank 3 --dist-init-addr 174.1.60.1:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.84 --mamba-full-memory-ratio 0.36 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_b_0 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.5 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 4 --pp-size 8 --ep-size 4 --nnodes 4 --node-rank 0 --dist-init-addr 174.1.60.5:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.84 --mamba-full-memory-ratio 0.36 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_b_1 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.6 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 4 --pp-size 8 --ep-size 4 --nnodes 4 --node-rank 1 --dist-init-addr 174.1.60.5:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.84 --mamba-full-memory-ratio 0.36 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_b_2 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 4 --pp-size 8 --ep-size 4 --nnodes 4 --node-rank 2 --dist-init-addr 174.1.60.5:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.84 --mamba-full-memory-ratio 0.36 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_b_3 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.8 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 4 --pp-size 8 --ep-size 4 --nnodes 4 --node-rank 3 --dist-init-addr 174.1.60.5:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.84 --mamba-full-memory-ratio 0.36 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run --rm --network host -v /data:/data -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.benchmark.serving --backend sglang --host 174.1.60.1 --port 32000 --model /data/hf_models/Kimi-K3 --tokenizer /data/hf_models/Kimi-K3 --dataset-name speed-bench --dataset-path /data/speedbench16k_complete.jsonl --num-prompts 40 --max-concurrency 8 --request-rate 10000 --speed-bench-output-len 512 --temperature 0.0 --top-p 1.0 --seed 42 --warmup-requests 1 --output-details --output-file /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-pp8-final-20260902-130711/bench/pp8_speedbench16k_c8_o512.jsonl
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_pp8_router_0 --network host --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang_router.launch_router --policy round_robin --worker-urls http://174.1.60.1:30000 http://174.1.60.5:30000 --host 0.0.0.0 --port 32000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run --rm --network host -v /data:/data -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.benchmark.serving --backend sglang --host 174.1.60.1 --port 30000 --model /data/hf_models/Kimi-K3 --tokenizer /data/hf_models/Kimi-K3 --dataset-name speed-bench --dataset-path /data/speedbench16k_complete.jsonl --num-prompts 1 --max-concurrency 8 --request-rate 10000 --speed-bench-output-len 1 --temperature 0.0 --top-p 1.0 --seed 42 --warmup-requests 0 --output-details --output-file /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-pp8-final-20260902-130711/bench/warm_a.jsonl
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run --rm --network host -v /data:/data -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.benchmark.serving --backend sglang --host 174.1.60.5 --port 30000 --model /data/hf_models/Kimi-K3 --tokenizer /data/hf_models/Kimi-K3 --dataset-name speed-bench --dataset-path /data/speedbench16k_complete.jsonl --num-prompts 1 --max-concurrency 8 --request-rate 10000 --speed-bench-output-len 1 --temperature 0.0 --top-p 1.0 --seed 42 --warmup-requests 0 --output-details --output-file /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-pp8-final-20260902-130711/bench/warm_b.jsonl
|
||||||
@ -0,0 +1,3 @@
|
|||||||
|
2af605e738d3a9b3f17310d54a81e81f1342b76eeeb12da7cd3f65abf6e0764f /data/speedbench16k_complete.jsonl
|
||||||
|
976aca142cb6183e0ed8f0fdd9395643233ce6ab9837e152f086b97c9bcfdf40 /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/run_dp2_nospec_ab.sh
|
||||||
|
28f994273a0f8b9d7ad0a3393400547c7f25cd369babc391d4a1f9553e7bcb9b /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/config.env
|
||||||
@ -0,0 +1,11 @@
|
|||||||
|
RUN_ID=dp2-pp8-final-20260902-130711
|
||||||
|
MODE=pp8
|
||||||
|
TP_SIZE=4
|
||||||
|
PP_SIZE=8
|
||||||
|
EP_SIZE=4
|
||||||
|
IMAGE=local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1
|
||||||
|
CHUNK_SIZE=8192
|
||||||
|
KV_DTYPE=bfloat16
|
||||||
|
NUM_PROMPTS=40
|
||||||
|
OUTPUT_LEN=512
|
||||||
|
CONCURRENCY=8
|
||||||
@ -0,0 +1,8 @@
|
|||||||
|
174.1.60.1 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.2 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.3 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.4 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.5 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.6 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.7 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.8 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_a_0 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.1 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 32 --pp-size 1 --ep-size 4 --nnodes 4 --node-rank 0 --dist-init-addr 174.1.60.1:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.86 --mamba-full-memory-ratio 0.21 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_a_1 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.2 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 32 --pp-size 1 --ep-size 4 --nnodes 4 --node-rank 1 --dist-init-addr 174.1.60.1:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.86 --mamba-full-memory-ratio 0.21 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_a_2 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.3 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 32 --pp-size 1 --ep-size 4 --nnodes 4 --node-rank 2 --dist-init-addr 174.1.60.1:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.86 --mamba-full-memory-ratio 0.21 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_a_3 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.4 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 32 --pp-size 1 --ep-size 4 --nnodes 4 --node-rank 3 --dist-init-addr 174.1.60.1:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.86 --mamba-full-memory-ratio 0.21 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_b_0 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.5 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 32 --pp-size 1 --ep-size 4 --nnodes 4 --node-rank 0 --dist-init-addr 174.1.60.5:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.86 --mamba-full-memory-ratio 0.21 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_b_1 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.6 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 32 --pp-size 1 --ep-size 4 --nnodes 4 --node-rank 1 --dist-init-addr 174.1.60.5:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.86 --mamba-full-memory-ratio 0.21 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_b_2 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 32 --pp-size 1 --ep-size 4 --nnodes 4 --node-rank 2 --dist-init-addr 174.1.60.5:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.86 --mamba-full-memory-ratio 0.21 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_b_3 --gpus all --network host --ipc=host --shm-size 32g --ulimit memlock=-1 --device /dev/infiniband -v /data/hf_models/Kimi-K3:/data/hf_models/Kimi-K3:ro -v /data/hzy/cache/kimi-dflash-fi0618-situ4460:/cache -e SGLANG_HOST_IP=174.1.60.8 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1 -e NCCL_IB_HCA=mlx5_0\,mlx5_1\,mlx5_2\,mlx5_3 -e NCCL_IB_GID_INDEX=3 -e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1 -e NCCL_DEBUG=WARN -e NCCL_DEBUG_SUBSYS=INIT\,ENV -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 -e SGLANG_MOE_FUSED_GATE_RADIX=1 -e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f -e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache -e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions -e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.launch_server --model-path /data/hf_models/Kimi-K3 --served-model-name kimi-k3 --trust-remote-code --tp-size 32 --pp-size 1 --ep-size 4 --nnodes 4 --node-rank 3 --dist-init-addr 174.1.60.5:20000 --moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none --kv-cache-dtype bfloat16 --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.86 --mamba-full-memory-ratio 0.21 --mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache --max-running-requests 8 --cuda-graph-max-bs-decode 8 --dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging --host 0.0.0.0 --port 30000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run --rm --network host -v /data:/data -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.benchmark.serving --backend sglang --host 174.1.60.1 --port 32000 --model /data/hf_models/Kimi-K3 --tokenizer /data/hf_models/Kimi-K3 --dataset-name speed-bench --dataset-path /data/speedbench16k_complete.jsonl --num-prompts 40 --max-concurrency 8 --request-rate 10000 --speed-bench-output-len 512 --temperature 0.0 --top-p 1.0 --seed 42 --warmup-requests 1 --output-details --output-file /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-tp32-final-20260902-123051/bench/tp32_speedbench16k_c8_o512.jsonl
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run -d --name kimi3_pro6000_dp2_nospec_ab_tp32_router_0 --network host --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang_router.launch_router --policy round_robin --worker-urls http://174.1.60.1:30000 http://174.1.60.5:30000 --host 0.0.0.0 --port 32000
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run --rm --network host -v /data:/data -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.benchmark.serving --backend sglang --host 174.1.60.1 --port 30000 --model /data/hf_models/Kimi-K3 --tokenizer /data/hf_models/Kimi-K3 --dataset-name speed-bench --dataset-path /data/speedbench16k_complete.jsonl --num-prompts 1 --max-concurrency 8 --request-rate 10000 --speed-bench-output-len 1 --temperature 0.0 --top-p 1.0 --seed 42 --warmup-requests 0 --output-details --output-file /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-tp32-final-20260902-123051/bench/warm_a.jsonl
|
||||||
@ -0,0 +1 @@
|
|||||||
|
docker run --rm --network host -v /data:/data -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 --entrypoint python3 local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1 -m sglang.benchmark.serving --backend sglang --host 174.1.60.5 --port 30000 --model /data/hf_models/Kimi-K3 --tokenizer /data/hf_models/Kimi-K3 --dataset-name speed-bench --dataset-path /data/speedbench16k_complete.jsonl --num-prompts 1 --max-concurrency 8 --request-rate 10000 --speed-bench-output-len 1 --temperature 0.0 --top-p 1.0 --seed 42 --warmup-requests 0 --output-details --output-file /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/results/dp2-tp32-final-20260902-123051/bench/warm_b.jsonl
|
||||||
@ -0,0 +1,3 @@
|
|||||||
|
2af605e738d3a9b3f17310d54a81e81f1342b76eeeb12da7cd3f65abf6e0764f /data/speedbench16k_complete.jsonl
|
||||||
|
976aca142cb6183e0ed8f0fdd9395643233ce6ab9837e152f086b97c9bcfdf40 /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/run_dp2_nospec_ab.sh
|
||||||
|
28f994273a0f8b9d7ad0a3393400547c7f25cd369babc391d4a1f9553e7bcb9b /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/config.env
|
||||||
@ -0,0 +1,11 @@
|
|||||||
|
RUN_ID=dp2-tp32-final-20260902-123051
|
||||||
|
MODE=tp32
|
||||||
|
TP_SIZE=32
|
||||||
|
PP_SIZE=1
|
||||||
|
EP_SIZE=4
|
||||||
|
IMAGE=local/sglang:kimi-k3-pp-dflash-33863-fi0618-situ4460-kvbounds1
|
||||||
|
CHUNK_SIZE=8192
|
||||||
|
KV_DTYPE=bfloat16
|
||||||
|
NUM_PROMPTS=40
|
||||||
|
OUTPUT_LEN=512
|
||||||
|
CONCURRENCY=8
|
||||||
@ -0,0 +1,8 @@
|
|||||||
|
174.1.60.1 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.2 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.3 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.4 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.5 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.6 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.7 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
|
174.1.60.8 sha256:7aa2be2d7af6cb11fea6d0c00d9bcd323ec986095dc9c4d3c997cff15ccb3b3f
|
||||||
317
experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/run_dp2_nospec_ab.sh
Executable file
317
experiments/pro6000/kimi3_pro6000_dp2_nospec_ab/run_dp2_nospec_ab.sh
Executable file
@ -0,0 +1,317 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
|
||||||
|
ACTION="${1:-status}"
|
||||||
|
MODE="${2:-tp32}"
|
||||||
|
DRY_RUN="${DRY_RUN:-0}"
|
||||||
|
[[ "$MODE" == tp32 || "$MODE" == pp8 ]] || {
|
||||||
|
echo "MODE must be tp32 or pp8" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
if [[ "$MODE" == tp32 ]]; then
|
||||||
|
TP_SIZE=32
|
||||||
|
PP_SIZE=1
|
||||||
|
MEM_FRACTION=0.86
|
||||||
|
MAMBA_RATIO=0.21
|
||||||
|
else
|
||||||
|
TP_SIZE=4
|
||||||
|
PP_SIZE=8
|
||||||
|
MEM_FRACTION=0.84
|
||||||
|
MAMBA_RATIO=0.36
|
||||||
|
fi
|
||||||
|
|
||||||
|
RUN_ID="${RUN_ID:-${EXPERIMENT_BASE}-${MODE}-$(date +%Y%m%d-%H%M%S)}"
|
||||||
|
[[ "$RUN_ID" =~ ^[a-zA-Z0-9_.-]+$ ]] || exit 2
|
||||||
|
RESULT_ROOT="${SCRIPT_DIR}/results/${RUN_ID}"
|
||||||
|
SUDO_PASSWORD_FILE="${SUDO_PASSWORD_FILE:-/data/hzy/.sudo_password}"
|
||||||
|
ALL_NODES=("${REPLICA_A_NODES[@]}" "${REPLICA_B_NODES[@]}")
|
||||||
|
mkdir -p "${RESULT_ROOT}"/{bench,commands,logs,metadata}
|
||||||
|
|
||||||
|
remote() {
|
||||||
|
local host="$1" command
|
||||||
|
shift
|
||||||
|
if [[ "$host" == "${REPLICA_A_NODES[0]}" ]]; then
|
||||||
|
"$@"
|
||||||
|
else
|
||||||
|
printf -v command '%q ' "$@"
|
||||||
|
ssh "${SSH_OPTIONS[@]}" "${SSH_USER}@${host}" "$command"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
root() {
|
||||||
|
local host="$1" password command
|
||||||
|
shift
|
||||||
|
IFS= read -r password <"$SUDO_PASSWORD_FILE"
|
||||||
|
if [[ "$host" == "${REPLICA_A_NODES[0]}" ]]; then
|
||||||
|
printf '%s\n' "$password" | sudo -S -p '' -- "$@"
|
||||||
|
else
|
||||||
|
printf -v command '%q ' "$@"
|
||||||
|
printf '%s\n' "$password" | ssh "${SSH_OPTIONS[@]}" \
|
||||||
|
"${SSH_USER}@${host}" "sudo -S -p '' -- ${command}"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
container_name() {
|
||||||
|
printf '%s_%s_%s_%s' "$EXPERIMENT_BASE" "$MODE" "$1" "$2"
|
||||||
|
}
|
||||||
|
|
||||||
|
record_command() {
|
||||||
|
local key="$1"
|
||||||
|
shift
|
||||||
|
{
|
||||||
|
printf '%q' "$1"
|
||||||
|
shift
|
||||||
|
printf ' %q' "$@"
|
||||||
|
printf '\n'
|
||||||
|
} >"${RESULT_ROOT}/commands/${key}.cmd.txt"
|
||||||
|
}
|
||||||
|
|
||||||
|
emit() {
|
||||||
|
local key="$1" host="$2"
|
||||||
|
shift 2
|
||||||
|
record_command "$key" "$@"
|
||||||
|
printf '%s %s: ' "$key" "$host"
|
||||||
|
cat "${RESULT_ROOT}/commands/${key}.cmd.txt"
|
||||||
|
[[ "$DRY_RUN" == 1 ]] || root "$host" "$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
build_server_command() {
|
||||||
|
local replica="$1" rank="$2" host head
|
||||||
|
if [[ "$replica" == a ]]; then
|
||||||
|
host="${REPLICA_A_NODES[$rank]}"
|
||||||
|
head="${REPLICA_A_NODES[0]}"
|
||||||
|
else
|
||||||
|
host="${REPLICA_B_NODES[$rank]}"
|
||||||
|
head="${REPLICA_B_NODES[0]}"
|
||||||
|
fi
|
||||||
|
HOST="$host"
|
||||||
|
CMD=(docker run -d --name "$(container_name "$replica" "$rank")"
|
||||||
|
--gpus all --network host --ipc=host --shm-size 32g
|
||||||
|
--ulimit memlock=-1 --device /dev/infiniband
|
||||||
|
-v "${MODEL_PATH}:${MODEL_PATH}:ro"
|
||||||
|
-v "${JIT_CACHE}:/cache"
|
||||||
|
-e "SGLANG_HOST_IP=${host}" -e PYTHONUNBUFFERED=1
|
||||||
|
-e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1
|
||||||
|
-e GLOO_SOCKET_IFNAME=bond0 -e NCCL_SOCKET_IFNAME=bond1
|
||||||
|
-e "NCCL_IB_HCA=${IB_DEVICES}" -e NCCL_IB_GID_INDEX=3
|
||||||
|
-e NCCL_IB_TIMEOUT=22 -e NCCL_IB_RETRY_CNT=7 -e NCCL_CUMEM_ENABLE=1
|
||||||
|
-e "NCCL_DEBUG=${NCCL_DEBUG_LEVEL}" -e NCCL_DEBUG_SUBSYS=INIT,ENV
|
||||||
|
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
||||||
|
-e SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
|
||||||
|
-e SGLANG_MOE_FUSED_GATE_RADIX=1
|
||||||
|
-e FLASHINFER_WORKSPACE_BASE=/cache -e FLASHINFER_CUDA_ARCH_LIST=12.0f
|
||||||
|
-e SGLANG_CACHE_DIR=/cache -e XDG_CACHE_HOME=/cache
|
||||||
|
-e TRITON_CACHE_DIR=/cache/triton -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions
|
||||||
|
-e CUDA_CACHE_PATH=/cache/cuda -e TMPDIR=/cache/tmp -e MAX_JOBS=4
|
||||||
|
--entrypoint python3 "$IMAGE" -m sglang.launch_server
|
||||||
|
--model-path "$MODEL_PATH" --served-model-name kimi-k3 --trust-remote-code
|
||||||
|
--tp-size "$TP_SIZE" --pp-size "$PP_SIZE" --ep-size "$EP_SIZE"
|
||||||
|
--nnodes 4 --node-rank "$rank" --dist-init-addr "${head}:${DIST_PORT}"
|
||||||
|
--moe-runner-backend flashinfer_mxfp4 --moe-a2a-backend none
|
||||||
|
--kv-cache-dtype "$KV_DTYPE"
|
||||||
|
--chunked-prefill-size "$CHUNK_SIZE" --page-size "$PAGE_SIZE"
|
||||||
|
--mem-fraction-static "$MEM_FRACTION"
|
||||||
|
--mamba-full-memory-ratio "$MAMBA_RATIO"
|
||||||
|
--mamba-radix-cache-strategy extra_buffer_lazy --disable-radix-cache
|
||||||
|
--max-running-requests "$MAX_RUNNING_REQUESTS"
|
||||||
|
--cuda-graph-max-bs-decode "$CUDA_GRAPH_MAX_BS"
|
||||||
|
--dist-timeout 3600 --enable-metrics --enable-request-time-stats-logging
|
||||||
|
--host 0.0.0.0 --port "$SERVER_PORT")
|
||||||
|
}
|
||||||
|
|
||||||
|
preflight() {
|
||||||
|
local host image reference="" gpu ports
|
||||||
|
for host in "${ALL_NODES[@]}"; do
|
||||||
|
image="$(root "$host" docker image inspect --format '{{.Id}}' "$IMAGE")"
|
||||||
|
[[ -z "$reference" ]] && reference="$image"
|
||||||
|
[[ "$image" == "$reference" ]] || {
|
||||||
|
echo "image mismatch: $host" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
remote "$host" test -r "${MODEL_PATH}/config.json"
|
||||||
|
remote "$host" test -e /dev/infiniband/uverbs0
|
||||||
|
root "$host" mkdir -p "${JIT_CACHE}/tmp"
|
||||||
|
gpu="$(remote "$host" nvidia-smi --query-compute-apps=pid --format=csv,noheader)"
|
||||||
|
[[ -z "$gpu" ]] || {
|
||||||
|
echo "GPU occupied: $host $gpu" >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
ports=("$SERVER_PORT" "$DIST_PORT")
|
||||||
|
[[ "$host" == "${REPLICA_A_NODES[0]}" ]] && ports+=("$ROUTER_PORT")
|
||||||
|
remote "$host" python3 -c 'import socket,sys
|
||||||
|
for p in map(int,sys.argv[1:]):
|
||||||
|
with socket.socket() as s:
|
||||||
|
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
|
||||||
|
try:
|
||||||
|
s.bind(("0.0.0.0",p)); s.listen(1)
|
||||||
|
except OSError as exc:
|
||||||
|
raise SystemExit(f"TCP port {p} unavailable: {exc}")' "${ports[@]}"
|
||||||
|
printf '%s %s\n' "$host" "$image"
|
||||||
|
done >"${RESULT_ROOT}/metadata/preflight.txt"
|
||||||
|
test -r "$SPEED_BENCH_DATASET"
|
||||||
|
sha256sum "$SPEED_BENCH_DATASET" "${SCRIPT_DIR}"/*.sh \
|
||||||
|
"${SCRIPT_DIR}/config.env" >"${RESULT_ROOT}/metadata/input_sha256.txt"
|
||||||
|
{
|
||||||
|
printf 'RUN_ID=%s\nMODE=%s\n' "$RUN_ID" "$MODE"
|
||||||
|
printf 'TP_SIZE=%s\nPP_SIZE=%s\nEP_SIZE=%s\n' "$TP_SIZE" "$PP_SIZE" "$EP_SIZE"
|
||||||
|
printf 'IMAGE=%s\nCHUNK_SIZE=%s\nKV_DTYPE=%s\n' "$IMAGE" "$CHUNK_SIZE" "$KV_DTYPE"
|
||||||
|
printf 'NUM_PROMPTS=%s\nOUTPUT_LEN=%s\nCONCURRENCY=%s\n' \
|
||||||
|
"$NUM_PROMPTS" "$OUTPUT_LEN" "$CONCURRENCY"
|
||||||
|
} >"${RESULT_ROOT}/metadata/manifest.env"
|
||||||
|
}
|
||||||
|
|
||||||
|
wait_health() {
|
||||||
|
local host="$1" deadline=$((SECONDS + HEALTH_WAIT_S)) replica rank running
|
||||||
|
[[ "$DRY_RUN" == 1 ]] && return 0
|
||||||
|
while (( SECONDS < deadline )); do
|
||||||
|
curl -fsS --max-time 3 "http://${host}:${SERVER_PORT}/health" >/dev/null 2>&1 && return 0
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
echo "health timeout: $host:$SERVER_PORT" >&2
|
||||||
|
for replica in a b; do
|
||||||
|
for rank in 0 1 2 3; do
|
||||||
|
build_server_command "$replica" "$rank"
|
||||||
|
running="$(root "$HOST" docker inspect --format '{{.State.Running}}' \
|
||||||
|
"$(container_name "$replica" "$rank")" 2>/dev/null || true)"
|
||||||
|
printf '%s %s running=%s\n' "$replica" "$rank" "$running" >&2
|
||||||
|
done
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
start() {
|
||||||
|
local replica rank
|
||||||
|
[[ "$DRY_RUN" == 1 ]] || preflight
|
||||||
|
for replica in a b; do
|
||||||
|
for rank in 1 2 3 0; do
|
||||||
|
build_server_command "$replica" "$rank"
|
||||||
|
emit "${replica}_${rank}" "$HOST" "${CMD[@]}"
|
||||||
|
done
|
||||||
|
if [[ "$replica" == a ]]; then
|
||||||
|
wait_health "${REPLICA_A_NODES[0]}"
|
||||||
|
else
|
||||||
|
wait_health "${REPLICA_B_NODES[0]}"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
emit router "${REPLICA_A_NODES[0]}" docker run -d \
|
||||||
|
--name "$(container_name router 0)" --network host \
|
||||||
|
-v "${MODEL_PATH}:${MODEL_PATH}:ro" \
|
||||||
|
-e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
|
||||||
|
--entrypoint python3 "$IMAGE" \
|
||||||
|
-m sglang_router.launch_router --policy round_robin \
|
||||||
|
--worker-urls "http://${REPLICA_A_NODES[0]}:${SERVER_PORT}" \
|
||||||
|
"http://${REPLICA_B_NODES[0]}:${SERVER_PORT}" \
|
||||||
|
--host 0.0.0.0 --port "$ROUTER_PORT"
|
||||||
|
[[ "$DRY_RUN" == 1 ]] || {
|
||||||
|
local deadline=$((SECONDS + 120))
|
||||||
|
until curl -fsS --max-time 3 "http://${REPLICA_A_NODES[0]}:${ROUTER_PORT}/health" >/dev/null 2>&1; do
|
||||||
|
(( SECONDS < deadline )) || return 1
|
||||||
|
sleep 2
|
||||||
|
done
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
bench_command() {
|
||||||
|
local host="$1" port="$2" prompts="$3" output_len="$4" warmups="$5" output="$6"
|
||||||
|
BENCH_CMD=(docker run --rm --network host -v /data:/data
|
||||||
|
-e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1
|
||||||
|
--entrypoint python3 "$IMAGE" -m sglang.benchmark.serving
|
||||||
|
--backend sglang --host "$host" --port "$port"
|
||||||
|
--model "$MODEL_PATH" --tokenizer "$MODEL_PATH"
|
||||||
|
--dataset-name speed-bench --dataset-path "$SPEED_BENCH_DATASET"
|
||||||
|
--num-prompts "$prompts" --max-concurrency "$CONCURRENCY" --request-rate 10000
|
||||||
|
--speed-bench-output-len "$output_len"
|
||||||
|
--temperature 0.0 --top-p 1.0 --seed 42 --warmup-requests "$warmups"
|
||||||
|
--output-details --output-file "$output")
|
||||||
|
}
|
||||||
|
|
||||||
|
warm_each_replica() {
|
||||||
|
local replica host output log
|
||||||
|
for replica in a b; do
|
||||||
|
[[ "$replica" == a ]] && host="${REPLICA_A_NODES[0]}" || host="${REPLICA_B_NODES[0]}"
|
||||||
|
output="${RESULT_ROOT}/bench/warm_${replica}.jsonl"
|
||||||
|
log="${RESULT_ROOT}/bench/warm_${replica}.log"
|
||||||
|
bench_command "$host" "$SERVER_PORT" 1 1 0 "$output"
|
||||||
|
record_command "warm_${replica}" "${BENCH_CMD[@]}"
|
||||||
|
root "${REPLICA_A_NODES[0]}" "${BENCH_CMD[@]}" >"$log" 2>&1
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
benchmark() {
|
||||||
|
local output="${RESULT_ROOT}/bench/${MODE}_speedbench16k_c${CONCURRENCY}_o${OUTPUT_LEN}.jsonl"
|
||||||
|
local log="${RESULT_ROOT}/bench/${MODE}_speedbench16k_c${CONCURRENCY}_o${OUTPUT_LEN}.log"
|
||||||
|
warm_each_replica
|
||||||
|
bench_command "${REPLICA_A_NODES[0]}" "$ROUTER_PORT" \
|
||||||
|
"$NUM_PROMPTS" "$OUTPUT_LEN" 1 "$output"
|
||||||
|
record_command benchmark "${BENCH_CMD[@]}"
|
||||||
|
root "${REPLICA_A_NODES[0]}" "${BENCH_CMD[@]}" >"$log" 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
logs() {
|
||||||
|
local replica rank
|
||||||
|
for replica in a b; do
|
||||||
|
for rank in 0 1 2 3; do
|
||||||
|
build_server_command "$replica" "$rank"
|
||||||
|
root "$HOST" docker logs "$(container_name "$replica" "$rank")" \
|
||||||
|
>"${RESULT_ROOT}/logs/${replica}_${rank}.log" 2>&1 || true
|
||||||
|
done
|
||||||
|
done
|
||||||
|
root "${REPLICA_A_NODES[0]}" docker logs "$(container_name router 0)" \
|
||||||
|
>"${RESULT_ROOT}/logs/router.log" 2>&1 || true
|
||||||
|
}
|
||||||
|
|
||||||
|
stop() {
|
||||||
|
local replica rank
|
||||||
|
root "${REPLICA_A_NODES[0]}" docker rm -f "$(container_name router 0)" >/dev/null 2>&1 || true
|
||||||
|
for replica in b a; do
|
||||||
|
for rank in 0 1 2 3; do
|
||||||
|
build_server_command "$replica" "$rank"
|
||||||
|
root "$HOST" docker rm -f "$(container_name "$replica" "$rank")" >/dev/null 2>&1 || true
|
||||||
|
done
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
on_error() {
|
||||||
|
local rc=$?
|
||||||
|
trap - ERR
|
||||||
|
logs || true
|
||||||
|
echo "FAILED rc=$rc; containers retained; evidence: $RESULT_ROOT" >&2
|
||||||
|
exit "$rc"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" != 1 ]]; then
|
||||||
|
ip -o -4 addr show | grep -q " ${REPLICA_A_NODES[0]}/" || {
|
||||||
|
echo 'Run this entry on 601 only' >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
trap on_error ERR
|
||||||
|
fi
|
||||||
|
|
||||||
|
case "$ACTION" in
|
||||||
|
dry-run) DRY_RUN=1; start ;;
|
||||||
|
preflight) preflight ;;
|
||||||
|
start) start ;;
|
||||||
|
bench) benchmark; logs ;;
|
||||||
|
logs) logs ;;
|
||||||
|
stop) stop ;;
|
||||||
|
status)
|
||||||
|
for host in "${ALL_NODES[@]}"; do
|
||||||
|
printf '%s\n' "$host"
|
||||||
|
root "$host" docker ps --filter "name=${EXPERIMENT_BASE}" --format '{{.Names}} {{.Status}}'
|
||||||
|
done
|
||||||
|
;;
|
||||||
|
all)
|
||||||
|
start
|
||||||
|
benchmark
|
||||||
|
logs
|
||||||
|
stop
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo 'Usage: run_dp2_nospec_ab.sh {dry-run|preflight|start|bench|logs|stop|status|all} {tp32|pp8}' >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
Loading…
x
Reference in New Issue
Block a user