6000D 双机 DeepSeek-V4-Pro 推理优化计划
-适用环境：
+174.1.51.5 + 174.1.51.7，每台 8 张 RTX PRO 6000 Blackwell Server Edition
当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例
当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离
计划版本：2026-07-30 15:52 CST适用环境：
174.1.51.5 + 174.1.51.7，每台 8 张 RTX PRO 6000 Blackwell Server Edition
当前部署：DeepSeek-V4-Pro，16 张 GPU 组成一个完整实例
当前约束：模型暂时只能使用全部 16 张 GPU，无法额外复制一套模型进行 PD 分离
计划版本：2026-07-30 17:54 CST
当前执行状态与阶段档案
| 6000D 双机通信与 NCCL 基础 | +已建立；覆盖计算网、RDMA、Bootstrap、NCCL 参数和日志判读 | +打开通信术语入门 | +|
| DeepSeek-V4-Pro / 双机 Pro6000D / SGLang TP16 快速性能地图 | -提前结束；3 个冷 Prefill 点完成，输入吞吐稳定约 65 token/s；旧脚本缓存口径审计已完成 | +网络错误已定位；RDMA fail-closed 代码与 dry-run 已完成，待真机 NET/IB 验证后重跑 | 打开实施记录 |
| DeepSeek-V4-Pro / 双机 Pro6000D / SGLang Prefill 硬件指标归因 | -设计已固化；代码尚未开始 | +设计已固化、代码尚未开始；等待修正后的 Phase 1 基线 | 打开 Phase 2 档案 |
0. 先看懂双机通信
++在分析 TP16 性能前，先区分设备、传输后端与 NCCL 参数。 +完整解释和本次网络误配置复盘见 +《6000D 双机通信与 NCCL 术语入门》。 +
+| 术语 | 一句话解释 | 本项目实例 |
|---|---|---|
| NCCL | NVIDIA 多 GPU 通信库，执行 collective 和点对点通信 | SGLang TP16 的跨 GPU 通信层 |
| RDMA | 网卡绕过常规 TCP/内核拷贝直接访问远端内存 | 双机高速数据面的目标路径 |
| IB / RoCE | IB 是高速网络/Verbs 体系；RoCE 在以太网上承载 RDMA | NCCL 日志统一显示为 NET/IB |
eth0/eth3 | 400G 物理端口的 Linux netdev/IP 入口 | 仅这两个接口用于节点间部署通信 |
mlx5_0/mlx5_3 | 映射到上述物理端口的 RDMA Verbs/HCA 入口 | 与 eth0/eth3 有映射关系，但不是同一个软件设备 |
| NCCL bootstrap | rank 启动时交换身份、地址、拓扑和连接信息的阶段 | 先建连，再选择真正的数据后端 |
NCCL_SOCKET_IFNAME | 选择 NCCL 可用的 IP 接口 | RDMA 失败时也决定 Socket 数据走哪张网卡 |
NCCL_IB_HCA | 选择 NCCL 可用的 RDMA HCA | mlx5_0,mlx5_3 |
NCCL_CROSS_NIC | 控制同一 ring/tree 能否在节点间跨不同 HCA | 只有真正使用多 HCA NET/IB 时才有意义 |
+400G 是每条物理 Ethernet 链路的标称线速，不是“TCP 速度”或“RDMA
+速度”。同一条链路可以承载 TCP，也可以承载 RoCE/RDMA；
+400 Gbit/s ≈ 50 GB/s 只是单向理论上限，NCCL 的
+algbw/busbw 与端到端模型吞吐都不能直接等同于该数字。
+
++2026-07-30 已确认：当时容器内没有
+/dev/infiniband，NCCL 日志显示 +NET/IB : No device found并回退NET/Socket。quick-map 又误选 +低速非计算网，因此 1K/32K 冷 Prefill 比原脚本的eth0Socket 路径慢约 +10.9×/13.25×。这不是NCCL_CROSS_NIC=1导致的。 +
1. 目标与原则
1.1 最终目标
在不做 PD 分离的前提下，定位 DeepSeek-V4-Pro 在双机 6000D 上的端到端瓶颈，并提高：
@@ -469,7 +510,7 @@6.3 网络
++++2026-07-30 控制组已确认：宿主机具备
+mlx5_0/eth0与 +mlx5_3/eth3两条 400G Rail，但原服务容器没有 +/dev/infiniband，NCCL 实际使用NET/Socket。 +当前第一优先级是校正容器数据面并重做 Phase 1，不再把约 65 token/s 当作模型或算子瓶颈。 +
+唯一启动入口现已在两端预检并透传
+rdma_cm/uverbs0/uverbs3，且服务健康后强制从两端 NCCL INFO
+日志确认 NET/IB 同时识别 mlx5_0/mlx5_3。代码、
+单测与 dry-run 已通过；真机服务尚未启动，因此这里仍不宣称 RDMA 已验证成功。
+
当前拓扑中需要分别观察两条 Compute Rail，确认：
- 两条 Rail 是否同时有流量。 @@ -744,8 +799,10 @@ NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING
- 分别执行相同消息范围的
all_reduce_perf，每个值至少重复 3 次，检查错误、algbw 和 busbw。
diff --git a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/README.md b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/README.md
index faeaf25..2417aac 100644
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/README.md
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/README.md
@@ -4,6 +4,20 @@ This directory contains the short, repeatable performance-map suite for
DeepSeek-V4-Pro on two RTX PRO 6000 Blackwell nodes. It does not modify or call the existing
`dsv4_pro6000_sglang_tp16/run_batch.sh`.
+## Network precondition
+
+The 2026-07-30 audit found that the current service container does not expose
+`/dev/infiniband`, so NCCL falls back to `NET/Socket`. The quick-map previously
+selected a low-speed non-compute interface and produced invalid 1K/32K cold
+Prefill baselines. The Socket default is now `eth0`, one of the two deployment
+compute interfaces (`eth0/eth3`).
+
+Declaring `NCCL_IB_HCA=mlx5_0,mlx5_3` does not by itself enable RDMA. The
+launcher now exposes only the three device nodes needed by the two deployment
+rails (`rdma_cm`, `uverbs0`, `uverbs3`) and fails before benchmarking unless
+both nodes' NCCL INFO logs prove `NET/IB` is using `mlx5_0` and `mlx5_3`.
+`NCCL_CROSS_NIC` has no effect while the active transport is `NET/Socket`.
+
## Scope
The fixed suite covers nine points:
diff --git a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/config.env b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/config.env
index e252d4c..a7a568d 100644
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/config.env
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/config.env
@@ -21,11 +21,18 @@ DOCKER_IMAGE="${DOCKER_IMAGE:-lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45
DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-$DOCKER_IMAGE}"
SGLANG_CACHE_DIR="${SGLANG_CACHE_DIR:-/data/hzy/sglang_cache/dsv4_pro_tp16}"
-# eth1 is the TCP bootstrap interface. mlx5_0/mlx5_3 are the two RoCE rails.
-NCCL_SOCKET_IFNAME="${NCCL_SOCKET_IFNAME:-eth1}"
-NCCL_IB_HCA="${NCCL_IB_HCA:-mlx5_0,mlx5_3}"
+# Only eth0/eth3 are deployment compute interfaces. eth0 is the bootstrap and
+# Socket-fallback interface; mlx5_0/mlx5_3 are the two RoCE HCA rails.
+NCCL_SOCKET_IFNAME="${NCCL_SOCKET_IFNAME:-eth0}"
+NCCL_IB_HCA="${NCCL_IB_HCA:-=mlx5_0:1,mlx5_3:1}"
NCCL_CROSS_NIC="${NCCL_CROSS_NIC:-1}"
-NCCL_DEBUG="${NCCL_DEBUG:-WARN}"
+NCCL_DEBUG="${NCCL_DEBUG:-INFO}"
+
+# Fail closed: a run labeled as RDMA must expose the exact HCA device nodes on
+# both nodes and show NET/IB in NCCL startup logs.
+ENABLE_RDMA="${ENABLE_RDMA:-1}"
+REQUIRE_NCCL_IB="${REQUIRE_NCCL_IB:-1}"
+RDMA_DEVICE_PATHS="${RDMA_DEVICE_PATHS:-/dev/infiniband/rdma_cm,/dev/infiniband/uverbs0,/dev/infiniband/uverbs3}"
# Keep the known working TP16 baseline. The quick map changes workload, not serving knobs.
MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.9}"
diff --git a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_results.py b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_results.py
index ee08ac7..27bb7d8 100755
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_results.py
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/quick_map_results.py
@@ -363,6 +363,9 @@ def write_manifest(args: argparse.Namespace) -> None:
"nccl_socket_ifname": args.nccl_socket_ifname,
"nccl_ib_hca": args.nccl_ib_hca,
"nccl_cross_nic": args.nccl_cross_nic,
+ "enable_rdma": bool(args.enable_rdma),
+ "require_nccl_ib": bool(args.require_nccl_ib),
+ "rdma_device_paths": args.rdma_device_paths,
"git_commit": args.git_commit,
"git_dirty": bool(args.git_dirty),
"scenario_file": args.scenario_file,
@@ -640,6 +643,9 @@ def add_manifest_arguments(parser: argparse.ArgumentParser) -> None:
parser.add_argument("--nccl-socket-ifname", required=True)
parser.add_argument("--nccl-ib-hca", required=True)
parser.add_argument("--nccl-cross-nic", required=True)
+ parser.add_argument("--enable-rdma", type=int, required=True)
+ parser.add_argument("--require-nccl-ib", type=int, required=True)
+ parser.add_argument("--rdma-device-paths", required=True)
parser.add_argument("--git-commit", required=True)
parser.add_argument("--git-dirty", type=int, required=True)
parser.add_argument("--scenario-file", required=True)
diff --git a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh
index 947f488..5da247a 100755
--- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh
+++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh
@@ -55,6 +55,78 @@ run_on_node() {
fi
}
+validate_network_config() {
+ case "${ENABLE_RDMA}" in
+ 0|1) ;;
+ *)
+ log "ERROR: ENABLE_RDMA must be 0 or 1, got: ${ENABLE_RDMA}"
+ return 1
+ ;;
+ esac
+ case "${REQUIRE_NCCL_IB}" in
+ 0|1) ;;
+ *)
+ log "ERROR: REQUIRE_NCCL_IB must be 0 or 1, got: ${REQUIRE_NCCL_IB}"
+ return 1
+ ;;
+ esac
+ if [[ "${REQUIRE_NCCL_IB}" == "1" && "${ENABLE_RDMA}" != "1" ]]; then
+ log "ERROR: REQUIRE_NCCL_IB=1 requires ENABLE_RDMA=1"
+ return 1
+ fi
+
+ local interface_spec="${NCCL_SOCKET_IFNAME#=}"
+ local -a interfaces=()
+ local interface
+ IFS=',' read -r -a interfaces <<< "${interface_spec}"
+ for interface in "${interfaces[@]}"; do
+ case "${interface#=}" in
+ eth0|eth3) ;;
+ *)
+ log "ERROR: only deployment compute interfaces eth0/eth3 are allowed; got: ${interface}"
+ return 1
+ ;;
+ esac
+ done
+
+ local hca_spec="${NCCL_IB_HCA#=}"
+ local -a hca_entries=()
+ local hca_entry hca saw_mlx5_0=0 saw_mlx5_3=0
+ IFS=',' read -r -a hca_entries <<< "${hca_spec}"
+ for hca_entry in "${hca_entries[@]}"; do
+ hca="${hca_entry%%:*}"
+ case "${hca}" in
+ mlx5_0) saw_mlx5_0=1 ;;
+ mlx5_3) saw_mlx5_3=1 ;;
+ *)
+ log "ERROR: only deployment RDMA HCAs mlx5_0/mlx5_3 are allowed; got: ${hca}"
+ return 1
+ ;;
+ esac
+ done
+ if [[ "${ENABLE_RDMA}" == "1" ]] \
+ && (( saw_mlx5_0 == 0 || saw_mlx5_3 == 0 )); then
+ log "ERROR: RDMA mode requires both mlx5_0 and mlx5_3"
+ return 1
+ fi
+}
+
+preflight_rdma_devices_on_node() {
+ local node="$1"
+ [[ "${ENABLE_RDMA}" == "1" ]] || return 0
+
+ local -a device_paths=()
+ local device_path quoted_path
+ IFS=',' read -r -a device_paths <<< "${RDMA_DEVICE_PATHS}"
+ for device_path in "${device_paths[@]}"; do
+ printf -v quoted_path '%q' "${device_path}"
+ if ! run_on_node "${node}" "test -c ${quoted_path}"; then
+ log "ERROR: ${node} is missing RDMA character device: ${device_path}"
+ return 1
+ fi
+ done
+}
+
service_is_healthy() {
curl --fail --silent --show-error --max-time 5 \
"http://${HEAD_IP}:${SGLANG_PORT}/health" >/dev/null 2>&1
@@ -78,8 +150,12 @@ remote_has_gpu_processes() {
preflight_service_node() {
local node="$1"
- run_on_node "${node}" \
- "test -d '${MODEL_PATH}' && command -v docker >/dev/null && command -v nvidia-smi >/dev/null && docker image inspect '${DOCKER_IMAGE}' >/dev/null"
+ if ! run_on_node "${node}" \
+ "test -d '${MODEL_PATH}' && command -v docker >/dev/null && command -v nvidia-smi >/dev/null && docker image inspect '${DOCKER_IMAGE}' >/dev/null"; then
+ log "ERROR: ${node} failed model, Docker, GPU, or image preflight"
+ return 1
+ fi
+ preflight_rdma_devices_on_node "${node}" || return 1
if [[ "${ALLOW_BUSY_GPU}" != "1" ]] && remote_has_gpu_processes "${node}"; then
log "ERROR: ${node} has active GPU compute processes"
log "This experiment will not evict another workload."
@@ -130,6 +206,16 @@ build_server_command() {
-e "NCCL_CROSS_NIC=${NCCL_CROSS_NIC}"
-e "NCCL_DEBUG=${NCCL_DEBUG}"
-e SGLANG_SHARED_EXPERT_TP1=1
+ )
+ if [[ "${ENABLE_RDMA}" == "1" ]]; then
+ local -a device_paths=()
+ local device_path
+ IFS=',' read -r -a device_paths <<< "${RDMA_DEVICE_PATHS}"
+ for device_path in "${device_paths[@]}"; do
+ DOCKER_CMD+=(--device "${device_path}")
+ done
+ fi
+ DOCKER_CMD+=(
--entrypoint python3
"${DOCKER_IMAGE}"
-m sglang.launch_server
@@ -162,16 +248,64 @@ start_service_node() {
build_server_command "${node_rank}" "${container_name}"
print_command "${DOCKER_CMD[@]}" > "${SERVER_ARTIFACT_DIR}/${role}_server_cmd.txt"
log "Starting ${role} node=${node} rank=${node_rank} container=${container_name}"
- run_on_node "${node}" "mkdir -p '${SGLANG_CACHE_DIR}'"
+ if ! run_on_node "${node}" "mkdir -p '${SGLANG_CACHE_DIR}'"; then
+ log "ERROR: failed to create SGLang cache directory on ${node}"
+ return 1
+ fi
run_on_node "${node}" "docker rm -f '${container_name}' >/dev/null 2>&1 || true"
local command
command="$(print_command "${DOCKER_CMD[@]}")"
- run_on_node "${node}" "${command}" \
- > "${SERVER_ARTIFACT_DIR}/${role}_container_id.txt"
+ if ! run_on_node "${node}" "${command}" \
+ > "${SERVER_ARTIFACT_DIR}/${role}_container_id.txt"; then
+ log "ERROR: failed to launch ${role} container on ${node}"
+ return 1
+ fi
+}
+
+verify_nccl_transport_node() {
+ local node="$1"
+ local container_name="$2"
+ local role="$3"
+ local transport_log="${SERVER_ARTIFACT_DIR}/${role}_nccl_transport.log"
+
+ run_on_node "${node}" "docker logs '${container_name}' 2>&1" \
+ > "${transport_log}" 2>&1
+
+ if grep -Fq "NET/IB : No device found" "${transport_log}"; then
+ log "ERROR: ${role} NCCL could not find an RDMA device"
+ return 1
+ fi
+ if ! grep -Eq 'NET/IB.*Using|Using network IB|via NET/IB' "${transport_log}"; then
+ log "ERROR: ${role} NCCL log does not prove NET/IB is active"
+ return 1
+ fi
+
+ local hca_spec="${NCCL_IB_HCA#=}"
+ local -a hca_entries=()
+ local hca_entry hca
+ IFS=',' read -r -a hca_entries <<< "${hca_spec}"
+ for hca_entry in "${hca_entries[@]}"; do
+ hca="${hca_entry%%:*}"
+ if ! grep -E "NET/IB.*${hca}|${hca}.*NET/IB" "${transport_log}" >/dev/null; then
+ log "ERROR: ${role} NCCL log does not show configured HCA ${hca}"
+ return 1
+ fi
+ done
+
+ log "Verified ${role} NCCL transport: NET/IB with mlx5_0 and mlx5_3"
+}
+
+verify_nccl_transport() {
+ [[ "${REQUIRE_NCCL_IB}" == "1" ]] || return 0
+ verify_nccl_transport_node \
+ "${WORKER_NODE}" "${WORKER_CONTAINER}" "worker" || return 1
+ verify_nccl_transport_node \
+ "${HEAD_NODE}" "${HEAD_CONTAINER}" "head" || return 1
}
start_service() {
+ validate_network_config || return 1
build_server_command 1 "${WORKER_CONTAINER}"
if [[ "${DRY_RUN}" == "1" ]]; then
printf '[DRY] worker (%s): ' "${WORKER_NODE}"
@@ -183,18 +317,21 @@ start_service() {
fi
mkdir -p "${SERVER_ARTIFACT_DIR}"
- preflight_service_node "${HEAD_NODE}"
- preflight_service_node "${WORKER_NODE}"
+ preflight_service_node "${HEAD_NODE}" || return 1
+ preflight_service_node "${WORKER_NODE}" || return 1
- start_service_node "${WORKER_NODE}" 1 "${WORKER_CONTAINER}" "worker"
+ start_service_node \
+ "${WORKER_NODE}" 1 "${WORKER_CONTAINER}" "worker" || return 1
sleep 5
- start_service_node "${HEAD_NODE}" 0 "${HEAD_CONTAINER}" "head"
+ start_service_node \
+ "${HEAD_NODE}" 0 "${HEAD_CONTAINER}" "head" || return 1
log "Waiting for SGLang health at ${HEAD_IP}:${SGLANG_PORT}"
local attempt
for (( attempt=1; attempt<=HEALTH_CHECK_RETRIES; attempt++ )); do
if service_is_healthy; then
log "SGLang is healthy after ${attempt} checks"
+ verify_nccl_transport || return 1
return 0
fi
if ! run_on_node "${HEAD_NODE}" \
@@ -502,6 +639,9 @@ write_run_manifest() {
--nccl-socket-ifname "${NCCL_SOCKET_IFNAME}" \
--nccl-ib-hca "${NCCL_IB_HCA}" \
--nccl-cross-nic "${NCCL_CROSS_NIC}" \
+ --enable-rdma "${ENABLE_RDMA}" \
+ --require-nccl-ib "${REQUIRE_NCCL_IB}" \
+ --rdma-device-paths "${RDMA_DEVICE_PATHS}" \
--git-commit "${git_commit}" \
--git-dirty "${git_dirty}" \
--scenario-file "${SCENARIO_FILE}"
该日志开销较高，不应在正式性能结果中长期启用。
6.4 NCCL_CROSS_NIC 快速 A/B
-Phase 1 固定使用 NCCL_CROSS_NIC=1。完成首轮 Prefill 硬件归因后，
-固定其余环境，快速比较 0/1/2，不能仅凭双 Rail 拓扑判断最优值。
+旧 Phase 1 虽然设置了 NCCL_CROSS_NIC=1，但由于 NCCL 回退
+NET/Socket，该参数没有参与实际路径选择。先让容器真正使用
+mlx5_0/mlx5_3 的 NET/IB，再固定其余环境比较
+0/1/2，不能仅凭双 Rail 拓扑判断最优值。