diff --git a/README.md b/README.md index d5baa83..b0d8824 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,13 @@ # sskj — 多平台大模型推理性能基准测试项目 +> **更新（2026-07-31 17:22:20 CST）** +> +> 完成 DeepSeek-V4-Pro 双机 Pro6000D SGLang Phase 2 正式收口。最终 Run `dsv4pro-phase2-20260731-163620` 在 28 分 44 秒内完成 8/8 benchmark、8/8 精确测量窗口和 18/18 采集器启停；混合 Prefill 令 Decode Output TPS 下降 23.96%、TPOT P95 增加 66.75%。GPU/CPU/NUMA、双 Rail RDMA、PCIe P2P、8/16-GPU AllReduce 与 `NCCL_CROSS_NIC=0/1/2` 数据均已归档；证据排除原始 Rail 带宽饱和、整机 CPU 饱和、GPU 降频和 CROSS_NIC 选择作为首要原因。实验结束后两节点容器、端口和 16 张 GPU 均已清理。 +> +> **更新（2026-07-31 17:08:30 CST）** +> +> 修正阶段档案门禁的适用范围：规则禁止在 Phase 尚无阶段结果时提前创建 HTML，但不追溯撤下已经完成正式 Run 和结果汇总的档案。Phase 2 已有正式 Run `dsv4pro-phase2-20260731-130125` 的 8/8 结果，因此恢复 `phase2_exp.html`、`phase2_code.html` 及主计划、Phase 1/2 页面之间的双向导航。 +> > **更新（2026-07-31 17:01:56 CST）** > > 固定阶段档案生成门禁：某个 Phase 在实验结束、结果汇总并完成汇报确认前，不创建或维护 `phaseN_exp.html` 与 `phaseN_code.html`；进行中只维护代码、原始结果和主计划状态。阶段确认完成后再一次性生成两份最终 HTML。Phase 2 尚待最终正式复跑，因此暂时撤下其两份 HTML 及导航；已完成的 Phase 1 档案继续保留。 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/phase1_exp.html b/docs/dsv4pro_pro6000d_2node_sglang/phase1_exp.html index 9bfd9a5..bdd63b6 100644 --- a/docs/dsv4pro_pro6000d_2node_sglang/phase1_exp.html +++ b/docs/dsv4pro_pro6000d_2node_sglang/phase1_exp.html @@ -843,7 +843,10 @@ curl -fsS http://10.101.0.11:30002/health || true docker ps --filter name=dsv4pro_pro6000d_2node_sglang_tp16_quick_map nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv -

下一阶段：Phase 2 正在进行，完成最终复跑与汇报后生成实验档案。

+

+ 下一阶段： + 打开 Phase 2 实验档案 +

返回推理优化主计划

diff --git a/docs/dsv4pro_pro6000d_2node_sglang/phase2_code.html b/docs/dsv4pro_pro6000d_2node_sglang/phase2_code.html new file mode 100644 index 0000000..5767ca3 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/phase2_code.html @@ -0,0 +1,435 @@ + + + + + + + Phase 2 Code：DSV4-Pro 双机 Pro6000D SGLang 硬件竞争归因 + + + +
+
+

Standalone Code Walkthrough / Phase 2

+

DSV4-Pro 双机 Pro6000D SGLang 硬件竞争归因：代码详解

+
+ 行号基线：39fc2ba565a3　 + 完成时间：2026-07-31 17:22:20 CST　 + 唯一入口：run_hardware_contention_attribution.sh all +
+
+
+ +
+

+ 返回推理优化主计划 · + 打开 Phase 2 实验档案 +

+
+ 文档边界：本文只解释提交 39fc2ba565a3 的 Phase 2 + 代码和文件调用关系。Phase 1 负责模型服务与请求；Phase 2 负责通信基线、 + 两节点监控、精确时间切片和逐指标报告。最终正式 Run + dsv4pro-phase2-20260731-163620 使用提交 + 5f24b7d22f98108f6cc234edba6768d55ea0a962，代码树包含本页所述修复。 +
+ +

1. 阅读导航

+ + +

2. 总体控制流

+ 
main "$@" → run_all
+  ├─ validate_config
+  ├─ preflight_node_tools
+  │    └─ 两节点 dcgmi discovery -l 必须成功
+  ├─ preflight_clock_sync + preflight_gpus_idle
+  ├─ run_communication_baseline
+  │    ├─ 两节点 CUDA P2P 全矩阵
+  │    ├─ 两节点各自 8-rank AllReduce
+  │    └─ 16-rank AllReduce，CROSS_NIC=0/1/2
+  ├─ start_service → Phase 1 start
+  ├─ capture_static_snapshots before
+  ├─ start_collectors → Head/Worker 同时采集
+  ├─ idle → fixed cases → mixed A/B → cooldown
+  ├─ check_collectors + stop_collectors
+  ├─ capture_static_snapshots after
+  ├─ stop_service → Phase 1 stop
+  ├─ summarize_results
+  │    └─ 按正式 benchmark 窗口生成第 5 节逐项数据表
+  └─ finish_manifest
+

+ all 是唯一正式入口。communication、summarize + 和 stop 是排错/恢复 action，不需要在正常执行前手工调用。 +

+ +

3. 文件职责与调用关系

+ + + + + + + + + +
文件行数职责
config.env61节点、Case、分层采样周期、通信尺寸、NCCL 选择和 fail-closed 策略。
run_hardware_contention_attribution.sh1036唯一 Shell 编排器：预检、通信文件分发、通信基线、Phase 1 委托、采集器、Case 和清理。
communication_baseline.py227CUDA P2P 全矩阵及 PyTorch/NCCL AllReduce 正确性、延迟和带宽测试。
hardware_contention_attribution.py1480解析所有原始采集器，按 Case 切片，聚合通信并生成 CSV/JSON/report.md。
tests/test_hardware_contention_attribution.py3229 项纯 Python 单元测试，覆盖 worker 无仓库依赖、精确窗口、解析器、RDMA 单位和通信聚合。
+ + 
用户
+  └─ Phase2/run_hardware_contention_attribution.sh all
+       ├─ source Phase2/config.env
+       ├─ docker/torchrun → Phase2/communication_baseline.py
+       ├─ env ... bash Phase1/run_quick_map.sh start/fixed/mixed/stop
+       │    └─ Phase1/quick_map_results.py 写 benchmark meta
+       ├─ Shell 采集 Head/Worker 原始时间序列
+       └─ Phase2/hardware_contention_attribution.py summarize
+            ├─ 读取 Phase1 bench/cases/*/meta.json
+            ├─ 读取 Head/Worker 原始监控
+            ├─ 读取 communication/COMM_RESULT
+            └─ 输出逐 Case、逐节点、逐指标表和 report.md
+ +

3.1 Phase 1 与 Phase 2 的边界

+ + + + + + + + +
问题由哪个文件负责证据
模型路径、镜像、TP16、EP、显存比例Phase 1 config.env + run_quick_map.shservice/head_server_cmd.txt、worker_server_cmd.txt
ISL/OSL/C、random 请求和 mixed A/BPhase 1 场景表与 benchmark 函数bench/*/bench_cmd.txt、bench.json
通信基线、监控周期、Case 选择Phase 2 config.envPhase 2 manifest.json
硬件归因和数值报告Phase 2 Python 汇总器case_*_summary.csv、report.md
+ +

4. 配置来源

+ + + + + + + + + +
行号配置组关键变量
config.env:L3-L16入口与节点PHASE1_ENTRY、Head/Worker、端口和容器名。
L18-L21诊断 Case五个 fixed Case、mixed A/B 开关。
L23-L39采样与严格性GPU/DCGM/RDMA 1 秒；CPU/进程/网络/NUMA/perf 5 秒；精确窗口和采集器 fail-closed。
L40-L55通信基线镜像、消息尺寸、迭代次数、P2P 大小、CROSS_NIC 列表、Socket/HCA。
L57-L61路径与模式RESULT_BASE、Runtime、Dry-run、是否允许部分采集器。
+

+ MEM_FRACTION_STATIC 不在 Phase 2 重复定义。它仍来自 Phase 1， + 最终展开为 SGLang 的 --mem-fraction-static。判断某次 Run 的真实值， + 应读取 service/head_server_cmd.txt，不能只看默认配置。 +

+ +

5. 通信微基准

+

5.1 Shell 如何编排

+

+ run_hardware_contention_attribution.sh:L281-L513 负责源码暂存、 + Docker 命令、两节点同步和清理。所有命令先写入 commands/*.txt： +

+ +

+ Docker 使用和 SGLang 一致的 CUDA 13 nightly 镜像，并显式透传 + rdma_cm、uverbs0、uverbs3。 + NCCL_DEBUG=INFO 只在微基准中打开，用于证明 NET/IB/GDRDMA 路径。 + Worker 不要求存在 Git 仓库；容器只读挂载自动分发的 + /tmp/.../<RUN_ID>/communication_baseline.py。结果目录同时保存 + 当次源码副本和 SHA256，避免两个节点 checkout 不一致造成版本漂移。 +

+ +

5.2 P2P 代码

+

+ communication_baseline.py:L45-L106 遍历所有源 GPU 和目标 GPU， + 先调用 torch.cuda.can_device_access_peer，再对 256 MiB FP16 Tensor + 做预热和 CUDA Event 计时。输出包括方向、P50/P95 latency 和 GB/s。 + 汇总器按拓扑拆成同 PCIe Switch 的 PIX 与跨 NUMA 的 SYS。 +

+ +

5.3 AllReduce 代码

+

+ communication_baseline.py:L107-L198 初始化 NCCL process group， + 对 1 MiB、64 MiB、1 GiB 分别预热和重复测量。每轮先把各 rank latency + gather 到 rank 0，使用最慢 rank 作为 collective 完成时间，并检查归约结果： +

+ 
algbw = message_bytes / latency
+busbw = algbw × 2 × (world_size - 1) / world_size
+wrong_values = count(output != expected_sum)
+

+ 这样不会用某个提前返回 rank 的时间美化结果；wrong_values=0 + 才算正确完成。 +

+ +

6. 两节点采集器

+

6.1 启动前门禁

+

+ Shell L67-L199 完成配置、工具、时钟和 GPU 空闲检查。 + preflight_node_tools 不只检查 dcgmi 文件存在， + 还实际运行 dcgmi discovery -l；两节点任一 Host Engine 不可用即退出。 +

+ +

6.2 采集器包装

+

+ start_stream_collector 位于 Shell L517-L551。 + 它保存完整命令、PID、唯一进程 tag 和日志；check_collectors 在 + L722-L740 检查采集器是否提前退出，默认不允许部分成功。 +

+ + + + + + + + + + + + + +
采集器Shell 位置周期输出
nvidia-smiL552-L5651 秒gpu_samples.csv
RDMA HCA countersL566-L5921 秒rdma.csv
DCGML645-L6551 秒dcgm_dmon.log
mpstatL656-L6635 秒mpstat.log
pidstat -durwL664-L6715 秒，进程级pidstat.log
sar -n DEV,EDEVL672-L6785 秒sar_net.log
perf statL680-L6895 秒perf_stat.log
numastatL618-L6445 秒numa_samples.csv
+

+ CPU、进程、perf 和 sar 的每行均由 Shell 增加 + wall_time_ns TAB node TAB payload。NUMA 直接转成结构化 CSV， + 避免旧版线程级 1 秒日志过大，也让所有指标能按 Case 切片。 +

+ +

7. 精确测量窗口

+

7.1 Phase 1 如何标记主测量

+

+ Phase 1 run_quick_map.sh:L547-L564 每 100 ms 观察 bench 日志； + 发现 Starting main benchmark run 后调用 + quick_map_results.py mark-measurement-start。 + quick_map_results.py:L340-L385 用这个起点和 + bench.json.duration 生成： +

+ 
measurement_started_at
+measurement_ended_at
+measurement_duration_s
+measurement_window_source = bench_main_marker_plus_duration
+ +

7.2 Phase 2 如何使用

+

+ hardware_contention_attribution.py:L509-L560 优先读取上述字段。 + 只有兼容旧结果时才可能使用进程级窗口；正式配置 + REQUIRE_PRECISE_WINDOWS=1 会拒绝任何 fallback。 + L561-L841 对 GPU、DCGM、CPU、进程、perf、NUMA、netdev 和 RDMA + 使用同一个 started_ns ≤ sample ≤ ended_ns 条件。 +

+ +

8. 逐指标报告

+

+ Python summarize 位于 + hardware_contention_attribution.py:L1036-L1378。 + 它不只生成一个抽象结论，而是按 Phase 2 第 5 节依次写出： +

+ + + + + + + + + + + + + +
指标解析函数Case 汇总文件
GPUsummarize_gpu_rows L377-L413case_gpu_summary.csv、case_gpu_node_summary.csv
DCGMparse_dcgm L167-L192case_dcgm_summary.csv
CPUparse_mpstat L193-L222case_cpu_summary.csv
进程parse_pidstat L223-L289case_process_summary.csv
perfparse_perf L290-L310case_perf_summary.csv
NUMA结构化 CSV + summarize_case_metricscase_numa_summary.csv
Linux netdevparse_sar_net L311-L358case_netdev_summary.csv
RDMAsummarize_rdma_rows L424-L484case_rdma_summary.csv
P2P/NCCLload_communication_rows + aggregate_communication_rows L842-L928communication_summary.csv、communication_aggregate.csv
+

+ report.md 对每组都打印有效样本数、Mean/P95/Max、Head/Worker + 或 Case 间比较和源文件。解析不到的值保留为 -，不会被写成 0。 +

+ +

9. 结果目录

+ 
results/<RUN_ID>/
+  manifest.json
+  commands/
+  communication/
+  service/
+  bench/<phase1-sub-run>/
+  head/
+    gpu_samples.csv
+    dcgm_dmon.log
+    mpstat.log
+    pidstat.log
+    perf_stat.log
+    sar_net.log
+    numa_samples.csv
+    rdma.csv
+    collector_commands/
+  worker/
+    ...同上...
+  case_windows.csv
+  bench_summary.csv
+  case_gpu_summary.csv
+  case_gpu_node_summary.csv
+  case_dcgm_summary.csv
+  case_cpu_summary.csv
+  case_process_summary.csv
+  case_perf_summary.csv
+  case_numa_summary.csv
+  case_netdev_summary.csv
+  case_rdma_summary.csv
+  communication_summary.csv
+  communication_aggregate.csv
+  summary.json
+  report.md
+ +

10. 函数行号索引

+

10.1 Shell 编排器

+ + + + + + + + + + + + + + + +
行号函数组职责
L26-L66日志、远端执行、命令证据基础设施。
L67-L199配置、工具、时钟、GPU 空闲门禁正式运行前 fail-fast。
L200-L268Manifest、marker、Phase 1 委托运行身份与复用边界。
L281-L513通信基线按 Run 分发源码、P2P、8/16-rank AllReduce、CROSS_NIC A/B 与清理。
L448-L516服务和静态快照启停 Phase 1 双机服务并保存环境。
L517-L710采集命令与启动两节点分层采样。
L711-L772采集器检查和停止fail-closed 与残留清理。
L782-L835fixed/mixed Case代表负载编排。
L836-L858汇总、Manifest、trap结果收口。
L859-L928run_all完整状态机。
L929-L968辅助 action 与 maincommunication/all/summarize/stop 分发。
+ +

10.2 Python 文件

+ + + + + + + + + + + + + + +
文件/行号职责
communication_baseline.py:L20-L44尺寸解析、分位数和 JSON 结果协议。
L45-L106CUDA P2P 全矩阵。
L107-L198NCCL AllReduce 与正确性。
hardware_contention_attribution.py:L76-L166时间、CSV、数字统计基础函数。
L167-L358DCGM、mpstat、pidstat、perf、sar 解析器。
L359-L508通信、GPU、RDMA、bench 读取与汇总。
L509-L841精确窗口和全部 Case 指标切片。
L842-L1035通信聚合、CSV、Marker、Manifest。
L1036-L1378全部输出表和逐指标 report.md。
L1379-L1480CLI 子命令。
+ +

11. 最终 Run 证据

+

+ 正式 Run 完成 8/8 benchmark、8/8 精确测量窗口和 18/18 采集器启停。 + 代码产生的各类输出与实验结论一一对应： +

+ + + +
+ + diff --git a/docs/dsv4pro_pro6000d_2node_sglang/phase2_exp.html b/docs/dsv4pro_pro6000d_2node_sglang/phase2_exp.html new file mode 100644 index 0000000..774d0e8 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/phase2_exp.html @@ -0,0 +1,785 @@ + + + + + + Phase 2：DeepSeek-V4-Pro 双机 Pro6000D SGLang 硬件与资源竞争归因 + + + +
+
+

Design, Implementation & Result Record

+

Phase 2：DeepSeek-V4-Pro 双机 Pro6000D SGLang 硬件与资源竞争归因

+
+ 节点：174.1.51.5 + 174.1.51.7 + 拓扑：SGLang TP16 / EP2 + 更新：2026-07-31 17:22:20 CST +
+
+
+ +
+ 返回推理优化主计划 + 打开 Phase 2 代码详解 + +

+ 当前状态：Phase 2 已完成。 + 最终 Run dsv4pro-phase2-20260731-163620 在 28 分 44 秒内完成 + 8/8 个 benchmark，正式测量窗口 8/8 精确，18/18 个采集器正常启停。 + Head/Worker 的 DCGM、CPU、NUMA、双 Rail RDMA 和通信微基准证据均有效； + 实验结束后两节点容器、服务端口和 16 张 GPU 已清理。 +

+ +

1. Phase 1 交接结果

+ + + + + + + + + + + + +
代表负载关键结果Phase 2 用途
128K → 1，C=1Input TPS 2,710.16；TTFT P95 48.344 s纯长 Prefill 的计算、显存与通信归因
32K → 1，C=16Input TPS 3,112.77；TTFT P95 162.087 s并发 Prefill 的排队、Chunk 调度与节点均衡
1K → 1K，C=32Output TPS 461.68；TPOT P95 63.31 ms普通 Decode 的 GPU、CPU 与通信基线
1K → 4K，C=16Output TPS 310.02；TPOT P95 50.33 ms持续 Decode、KV 增长和稳态资源占用
128K → 1K，C=1TTFT P95 49.326 s；TPOT P95 32.24 ms分离长 Prefill 与长上下文 Decode 成本
1K → 1K，C=32 + 128K 注入Output TPS -24.08%；TPOT P95 +66.55%Prefill 干扰 Decode 时的硬件资源竞争
+

+ 最终基线已由 Head 与 Worker 日志证明使用 + mlx5_0/mlx5_3 双 Rail NET/IB + GDRDMA， + 正式测量请求为冷 Prefix。正式矩阵 12/12、长 Decode 补测 2/2 均成功。 + Phase 2 保持相同服务配置和请求口径。 +

+ +

2. 本阶段的边界

+ + +

3. 待验证假设

+ + + + + + + + + + + + +
假设预期硬件表现后续方向
DSV4/NSA Prefill Kernel 计算受限GPU 持续忙、高功耗和稳定频率；双 Rail 流量不高Phase 3 捕获 Kernel 与 Attention/Indexer 时间线
权重或激活显存带宽受限GPU Memory Utilization 高，SM 指标未必饱和；功耗可能低于纯计算补 DCGM/Profiler 的 DRAM Active，再看 Kernel
TP16 跨机通信受限RoCE 吞吐高或两条 Rail 明显失衡，GPU 出现等待NCCL_CROSS_NIC 0/1/2 快速 A/B，随后看 NCCL Timeline
CPU Scheduler 或 Kernel Launch 受限GPU 利用率锯齿或有空洞，单 CPU 核持续满载定位 Scheduler/Tokenizer 线程与 launch gap
频率、功耗或温度限制P-state、SM Clock 或 Power 持续异常，可能出现节流原因修正电源、散热或 Clock Policy 后复测
节点或 Rank 不均衡两节点或不同 GPU 的利用率、功耗、网络流量存在固定偏差检查 NUMA、GPU-NIC 亲和与慢 Rank
+ +

4. 诊断 Run

+
    +
  1. 确认 16 张 GPU 空闲，先跑两节点 PCIe P2P、单机 8 rank AllReduce 和双机 16 rank AllReduce；双机分别测试 NCCL_CROSS_NIC=0/1/2。
    2. +
  2. 保存两节点静态快照：GPU/NIC/NUMA 拓扑、驱动、CUDA、镜像与服务命令。
    3. +
  3. 复用 Phase 1 已验证的 run_quick_map.sh start 启动同配置双机服务。
    4. +
  4. 在 Head 和 Worker 同时启动 GPU、CPU、网卡与 RDMA 采样，先记录 15 秒空闲基线。
    5. +
  5. 依次重放 128K → 1, C=1、32K → 1, C=16 与 1K → 1K, C=32。
    6. +
  6. 重放 1K → 4K, C=16 和 128K → 1K, C=1，观察持续与长上下文 Decode。
    7. +
  7. 重放 1K → 1K, C=32 Control 与 128K Prefill 注入 Treatment，保留相同注入时序。
    8. +
  8. 请求结束后继续采样 15 秒，再停止采集器和服务。
    9. +
  9. 按时间戳将请求、GPU、CPU 和双 Rail 指标对齐，生成摘要与判定。
    10. +
+ 
idle 15s
+  │ 128K→1 C1 │ 32K→1 C16 │ 1K→1K C32
+  │ 1K→4K C16 │ 128K→1K C1
+  │ Decode Control │ Decode + Prefill
+cooldown 15s
+
+Head 与 Worker 的所有采集器覆盖完整诊断窗口。
+

+ Phase 1 中服务加载约 5 分 30 秒；通信基线、五个固定负载、混合 A/B、 + 静态快照、采样和清理组成一次完整 Phase 2 Run。 +

+ +

4.1 你只需要运行的入口

+

+ 操作规则：先在 Worker .7 做一次 DCGM 准备，再只在 + Head .5 执行 Phase 2 的 all。 + 不要手工执行 Phase 1 的 start 或 stop。 + Phase 2 会在内部复用它们，并负责异常退出时的采集器、Head、Worker 清理。 +

+ 
# [仅在 Worker 174.1.51.7 执行一次]
+# 不需要 source、conda activate，也不要在 .7 运行 Phase 2 的 all
+systemctl start nvidia-dcgm
+systemctl is-active nvidia-dcgm
+dcgmi discovery -l
+
+# 预期：第二条输出 active，第三条列出本机 8 张 GPU
+
+# [以下仅在 Head 174.1.51.5 执行]
+cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
+
+# 第一次先展开全部命令，不启动服务、不占用 GPU、不发送请求
+DRY_RUN=1 RUN_ID=dsv4pro-phase2-dryrun-$(date +%Y%m%d-%H%M%S) \
+  bash run_hardware_contention_attribution.sh all
+
+# 正式实验：仍然只有同一个 all 入口，tmux 只负责断线后继续运行
+RUN_ID=dsv4pro-phase2-$(date +%Y%m%d-%H%M%S)
+tmux new-session -d -s dsv4pro-phase2 \
+  "RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
+   2>&1 | tee /data/hzy/${RUN_ID}.log"
+
+tmux attach -t dsv4pro-phase2
+

+ .7 的三条命令只负责让 Worker DCGM Host Engine 可用； + SGLang Worker、其余采集器和结果回收仍由 .5 的唯一入口通过 SSH 管理。 + 若需要机器重启后自动启动 DCGM，应由运维另行决定是否执行 + systemctl enable nvidia-dcgm。 +

+

all 内部执行顺序：

+ 
配置、工具、DCGM 与 GPU 空闲门禁
+  → 通信基线：两节点 P2P、两组单机 8-rank AllReduce、
+                三组双机 16-rank NCCL_CROSS_NIC A/B
+  → Phase 1 start：启动同配置 TP16 服务
+  → 两节点静态快照
+  → 启动两节点采集器并记录 15 秒 idle
+  → 五个固定 Case
+  → 混合 Prefill/Decode A/B
+  → 15 秒 cooldown
+  → 停止采集器并保存后快照
+  → Phase 1 stop：停止 Head/Worker
+  → 生成按第 5 节逐项对应的 CSV、JSON 与 report.md
+

+ Phase 1 的作用是提供已经验证过的双机 Docker 服务和 Benchmark 实现， + 不是第二个用户入口。实际展开的服务、Benchmark 和采集命令都会写入 + results/<RUN_ID>/service/、commands/ 和 + head|worker/collector_commands/，不依赖跨文档猜测。 +

+ +

5. 采集指标

+

+ 本节记录正式实现使用的命令，而不是建议性伪代码。命令由 + run_hardware_contention_attribution.sh 在 Head 和 Worker 同时启动； + 每条展开后的命令会另外保存在 + results/<RUN_ID>/head|worker/collector_commands/。 +

+ + + + + + + + + + + +
层级连续采样静态或前后快照
GPU利用率、Memory Utilization、显存、功耗、SM/Memory Clock、温度、P-statenvidia-smi topo -m、Compute Process
CPU每核利用率、上下文切换、服务进程 CPU/内存NUMA 拓扑、容器 PID 与 CPU Affinity
Networketh0/eth3 RX/TXethtool -S 错误计数前后差
RDMAmlx5_0/mlx5_3 port_xmit/recv_data 差分Port State、GID 与错误计数
DCGMSM Active、DRAM Active、Tensor Active、PCIe工具版本与可用 Field
+ +

5.1 时间对齐与 Case Marker

+ 
# 每条 GPU/RDMA 样本写入相同格式的宿主机墙钟时间
+date +%s%N
+
+# 实验前检查两节点秒级时钟差
+date +%s
+
+# Case 开始、结束和服务状态由 Python 写入 markers.csv
+python3 hardware_contention_attribution.py marker \
+  --path markers.csv \
+  --node head \
+  --event case_start \
+  --case-id long_prefill_latency_128k_c1
+ + + + + + + +
数据含义为什么需要
wall_time_nsUnix Epoch 纳秒时间把 GPU、CPU、RDMA 与 Benchmark 放到同一时间轴
case_start/case_end一个 Case 的编排边界从整段连续采样中切出对应负载
CLOCK_SKEW_TOLERANCE_S=2两节点允许的最大秒级时钟差避免 Head/Worker 的同一时刻被错位比较
+

+ 最终实现由 Phase 1 监听 bench.log 中的 + Starting main benchmark run，立刻写入 + measurement_start.json；再使用 SGLang bench.json + 的正式 benchmark duration 计算结束时间。Phase 2 优先读取 + measurement_started_at/measurement_ended_at，不会把数据生成和 + Warm-up 混入硬件均值。REQUIRE_PRECISE_WINDOWS=1 时，任何 Case + 缺少精确窗口都会让汇总失败，而不是悄悄回退。 +

+ +

5.2 GPU 基础状态：nvidia-smi

+ 
nvidia-smi \
+  --query-gpu=index,timestamp,utilization.gpu,utilization.memory,\
+memory.used,memory.total,power.draw,temperature.gpu,\
+clocks.sm,clocks.mem,pstate \
+  --format=csv,noheader,nounits
+

脚本每秒运行一次，并在每行前加入 wall_time_ns 和节点角色。

+ + + + + + + + + + +
字段代表什么
utilization.gpu采样周期内至少有一个 Kernel 在执行的时间比例
utilization.memory采样周期内显存控制器处于忙碌状态的时间比例
memory.used/total当前总显存分配量与设备显存容量
power.drawGPU 当前功耗，用于比较不同负载的能耗状态
clocks.sm/clocks.memSM 与显存当前工作频率
pstateGPU 性能状态，P0 通常是最高性能态
+

+ 原始输出为 head|worker/gpu_samples.csv； + gpu_summary.csv 汇总整段运行， + case_gpu_summary.csv 按节点、Case 和 GPU 汇总平均值与峰值。 +

+ +

5.3 GPU Profiling Counter：DCGM

+ 
DCGM_FIELD_IDS=1001,1002,1003,1004,1005,1009,1010
+
+dcgmi dmon \
+  -e 1001,1002,1003,1004,1005,1009,1010 \
+  -d 1000
+ + + + + + + + + + + +
Field IDField Tag含义
1001gr_engine_activeGraphics/Compute Engine 活跃比例，接近整体 GPU 执行忙碌度
1002sm_activeSM 至少有一个 Warp 活跃的比例
1003sm_occupancy活跃 Warp 相对硬件可容纳 Warp 的比例
1004tensor_activeTensor Core 指令活跃比例
1005dram_active设备显存接口活跃比例；Pro6000D 为 GDDR7，用于判断设备显存带宽压力
1009pcie_tx_bytesGPU 经 PCIe 发出的字节速率
1010pcie_rx_bytesGPU 经 PCIe 接收的字节速率
+

+ sm_active 高而 sm_occupancy 低，表示 SM 经常有工作， + 但同时驻留的 Warp 不多；后续通过 Kernel Timeline 区分小 Kernel、 + 寄存器/共享内存约束和同步。DCGM 是 NVIDIA Data Center GPU Manager： + nvidia-dcgm/nv-hostengine 是后台 Host Engine， + dcgmi 是客户端，Field ID 是指标编号。最终代码在两节点预检 + dcgmi discovery -l，任一 Host Engine 不可用即 fail-fast； + 正式结果必须同时包含 Head 和 Worker 的 case_dcgm_summary.csv。 +

+ +

5.4 CPU、进程与 Kernel Launch 侧证据

+ 
# 全部逻辑 CPU，每 5 秒输出一次
+mpstat -P ALL 5
+
+# 找到容器内进程对应的宿主 PID
+docker top <container> -eo pid,ppid,psr,pcpu,pmem,stat,comm,args
+
+# 最终命令：进程级 CPU、I/O、缺页、上下文切换，不展开全部线程
+pidstat -durw -p "<comma-separated-host-pids>" 5
+
+# 每 5 秒输出一次硬件/软件计数器增量
+perf stat -p "<comma-separated-host-pids>" -I 5000 \
+  -e cycles,instructions,cache-misses,context-switches,\
+cpu-migrations,page-faults
+
+# mpstat/pidstat/perf 每行都由包装器增加：
+# wall_time_ns TAB node TAB 原始输出
+ + + + + + + + + + + + +
命令/字段回答的问题
mpstat -P ALL整机是否 CPU 饱和，是否只有少量核心接近 100%，是否存在 I/O Wait
docker top把容器进程映射为宿主 PID、CPU 核 PSR 和进程状态
pidstat -u服务进程的用户态、内核态 CPU 时间
pidstat -d进程块设备 I/O
pidstat -r内存和 Page Fault 行为
pidstat -w主动/被动上下文切换，辅助发现线程阻塞或调度抖动
perf cycles/instructionsCPU 周期与指令执行量，可计算近似 IPC
cache-misses/migrationsCPU Cache 压力和线程跨核迁移
+

+ 最终实现使用进程级 5 秒采样，避免首轮线程级 1 秒采样产生数百 MB 日志。 + case_cpu_summary.csv、case_process_summary.csv 和 + case_perf_summary.csv 都按正式测量窗口切片；只有先发现异常进程， + 才在后续短窗口单独开启线程级采样。 +

+ +

5.5 NUMA 与 CPU/内存亲和

+ 
# 静态 NUMA 节点、CPU 和内存布局
+numactl --hardware
+numastat -m
+
+# 每 5 秒按容器宿主 PID 查看本地/远端 NUMA 内存
+numastat -p <host-pid>
+# 解析为：
+# wall_time_ns,node,node0_mib,node1_mib,total_mib,process_count
+
+# 同时保存 GPU、CPU、NIC 的拓扑关系
+nvidia-smi topo -m
+

+ NUMA 是多路 CPU 机器的“本地内存”结构。进程长期从远端 NUMA Node 取内存， + 或 GPU/NIC 对应的 CPU 线程被调度到另一侧，可能增加 Host 侧延迟。 + 最终采集器把 numastat -p 解析为 + numa_samples.csv，再按正式测量窗口生成 + case_numa_summary.csv。这样可以直接比较 Node0/Node1 MiB， + 而不是依靠人工阅读不断刷新的文本。 +

+ +

5.6 普通网卡统计与 RDMA 数据面

+ 
# Linux netdev 层，每 5 秒采样吞吐与错误
+sar -n DEV,EDEV 5
+
+# Case 前后保存物理端口状态和驱动计数器
+ethtool eth0
+ethtool eth3
+ethtool -S eth0
+ethtool -S eth3
+
+# RDMA 设备与端口状态
+ibdev2netdev
+ibstat
+rdma link show
+

+ sar 记录 Linux 普通网络栈中的 eth0/eth3 流量； + GDRDMA 数据量由 mlx5_0/mlx5_3 HCA 的 sysfs Counter 记录： +

+ 
for hca in mlx5_0 mlx5_3; do
+  base="/sys/class/infiniband/${hca}/ports/1"
+  cat "${base}/counters/port_xmit_data"
+  cat "${base}/counters/port_rcv_data"
+  cat "${base}/counters/port_xmit_wait"
+  cat "${base}/counters/port_xmit_discards"
+  cat "${base}/counters/port_rcv_errors"
+  cat "${base}/hw_counters/req_transport_retries_exceeded"
+  cat "${base}/hw_counters/req_rnr_retries_exceeded"
+done
+ + + + + + + + + + + +
Counter含义
port_xmit_data/port_rcv_dataHCA 发送/接收数据累计量；IB Counter 单位是 4 Octets，脚本用 delta × 4 × 8 / seconds 换算 Gbit/s
port_xmit_wait端口因缺少发送 Credit 等原因等待的时间，持续增长可能指向拥塞
port_xmit_discards/port_rcv_errors发送丢弃和接收错误增量
req_transport_retries_exceededRDMA Transport 重试耗尽
req_rnr_retries_exceededReceiver Not Ready 重试耗尽
roce_adp_retrans*RoCE 自适应重传及超时相关计数
np_ecn_marked* / *cnp*ECN 标记和拥塞通知包，用于辅助判断 RoCE 拥塞
+

+ 原始数据为 head|worker/rdma.csv； + case_rdma_summary.csv 按 Case、节点和 HCA 计算吞吐及错误增量。 + 它说明双 Rail 的实际流量、均衡性和错误增量； + case_netdev_summary.csv 同时保留 Linux netdev 层的 + eth0/eth3 RX/TX 与错误。Phase 3 再把 NCCL Collective + 放到请求 Timeline 中分析持续时间和计算重叠。 +

+ +

5.7 机内 PCIe 与 NCCL 通信基线

+ 
# 由 all 入口自动执行；不需要用户手工运行 torchrun
+# 每个节点：所有 GPU 源/目标对，FP16 256 MiB CUDA P2P copy
+python3 communication_baseline.py p2p \
+  --size 256M --warmup 3 --iterations 10
+
+# 每个节点：8 rank NCCL AllReduce
+torchrun --standalone --nproc-per-node=8 \
+  communication_baseline.py all-reduce \
+  --sizes 1M,64M,1G --repetitions 3 --warmup 5 --iterations 10
+
+# 双节点：16 rank；分别设置 NCCL_CROSS_NIC=0、1、2
+torchrun --nnodes=2 --nproc-per-node=8 \
+  --master-addr 10.101.0.11 --node-rank <0-or-1> \
+  communication_baseline.py all-reduce \
+  --sizes 1M,64M,1G --repetitions 3 --warmup 5 --iterations 10
+

+ P2P 结果按 same_pcie_switch（PIX）和 + cross_numa_sys（SYS）分别汇总，不用一个平均值掩盖跨 CPU 路径。 + AllReduce 同时报告 P50/P95 latency、algbw、 + busbw、正确性错误数和实际 NCCL 路径。1 MiB、64 MiB、1 GiB + 分别覆盖小消息延迟、中等消息和大消息带宽；双机 A/B 直接给出 + NCCL_CROSS_NIC=0/1/2 的数值比较。 +

+ +

5.8 静态快照与结果关系

+ 
nvidia-smi
+nvidia-smi topo -m
+lscpu
+numactl --hardware
+ip -details link show eth0
+ip -details link show eth3
+docker inspect <container>
+docker top <container> -eo pid,ppid,psr,pcpu,pmem,stat,comm,args
+ + + + + + + + + + + + + + + + + + +
结果文件内容主要用途
static_before.log / static_after.logGPU、CPU、NUMA、NIC、RDMA、容器前后快照证明运行环境，并比较错误计数和清理状态
collector_status.csv每个采集器的启动、停止或提前退出状态防止把缺失采集器当作 0 值
case_windows.csv每个 Benchmark Case 的起止时间从连续硬件日志中切片
bench_summary.csvTPS、TTFT、TPOT、ITL、E2E把硬件现象与用户侧性能对应
case_gpu_summary.csv每 Case、节点、GPU 的利用率、显存、功耗、频率比较负载与节点/GPU 不均衡
case_dcgm_summary.csv每 Case、节点、GPU 的 SM/Tensor/显存接口/PCIe 指标区分计算、设备显存和 PCIe 活跃度
case_cpu_summary.csv整机与逐核 CPU 利用率、I/O Wait识别整机饱和和少数热点核
case_process_summary.csv服务进程 CPU、I/O、缺页、内存与上下文切换定位 Host 进程开销与阻塞
case_perf_summary.csvcycles、instructions、cache miss、迁移与缺页计算 IPC 并判断 Cache/调度压力
case_numa_summary.csvNode0/Node1 进程内存分布识别跨 NUMA 放置
case_netdev_summary.csveth0/eth3 吞吐与错误与 RDMA HCA Counter 做分层核对
case_rdma_summary.csv每 Case、节点、Rail 的吞吐和错误增量判断双 Rail 使用、均衡和数据面错误
communication_summary.csv每次 P2P/AllReduce 原始测量保留每条 GPU 对、消息尺寸、CROSS_NIC 和重复实验
communication_aggregate.csvPIX/SYS P2P 与单/双机 AllReduce 聚合提供 P50/P95、algbw、busbw 和正确性比较
+ +

5.9 最终结果如何逐项汇报

+

+ 最终 report.md 的章节顺序与本节一一对应。每一项必须同时给出 + 原始文件、有效样本数、Head/Worker 数值、Case 间变化和解释； + 不能只写“GPU 较忙”“网络未饱和”这类抽象结论。 +

+ + + + + + + + + + + +
第 5 节指标报告中的数值最小分析动作
5.1 时间窗窗口来源、开始/结束、duration、采样数确认全部为 bench_main_marker_plus_duration
5.2 GPU利用率/显存/功耗/频率的 Mean、P95、Max比较两节点、8 卡离散度和不同 Case
5.3 DCGMSM Active/Occupancy、Tensor/DRAM Active、PCIe TX/RX比较计算、设备显存和 PCIe 哪一侧随负载上升
5.4 CPU/进程/perf整机/热点核、进程 CPU/I/O/缺页/切换、IPC/Cache miss区分整机容量、单线程热点和 Host 调度开销
5.5 NUMANode0/Node1 MiB 与比例比较服务内存是否偏离 GPU/NIC 所在 NUMA
5.6 Network/RDMAeth0/eth3、mlx5_0/mlx5_3 Gbit/s 与错误增量计算双 Rail 均衡比例并核对丢弃/重试
5.7 CommunicationPIX/SYS P2P、8/16 rank AllReduce P50/P95、algbw/busbw比较跨 NUMA 损失与 CROSS_NIC 0/1/2
+

+ 某个采集器无数据时报告显示 - 并附失败状态，不会把缺失值写成 + 0。正式 Run 默认 ALLOW_PARTIAL_COLLECTORS=0， + 因此必需采集器提前退出会让 Run 失败。 +

+ +

6. 精简代码设计

+

已新增目录：

+ 
/data/hzy/sskj/experiments/pro6000/
+dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/
+ + + + + + + + + + + + +
文件职责当前状态
run_hardware_contention_attribution.sh唯一 Shell 入口；按 Run 分发通信代码、通信基线、服务启停、双节点采集器、Case 编排、门禁和 Trap 清理已实现
config.envPhase 1 相对路径、节点、代表 Case、分层采样周期、通信基线与 fail-closed 策略已实现
communication_baseline.pyCUDA P2P 全矩阵与 PyTorch/NCCL 8/16-rank AllReduce 微基准已实现
hardware_contention_attribution.py精确窗口、全部采集器解析、逐 Case 汇总、通信聚合和逐指标报告已实现
tests/test_hardware_contention_attribution.pyWorker 无仓库依赖、GPU/RDMA、精确窗口、DCGM/CPU 解析、通信聚合和结果生成测试9/9 通过
README.md唯一入口、范围和结果目录说明已实现
+

+ Phase 2 不复制双机 Docker 启停实现。唯一入口在内部调用 Phase 1 的 + run_quick_map.sh start/fixed/mixed/stop，只新增通信基线、硬件采集、时间对齐和代表负载编排。 + 顶层仍只保留一个 Shell 文件，不创建额外 tmux、launch、start 或 stop 脚本。 +

+ +

7. 结果结构

+ 
results/<RUN_ID>/
+  manifest.json
+  run.log
+  bench/
+    bench_cmd.txt
+    bench.log
+    bench.jsonl
+  service/
+    head_server_cmd.txt
+    worker_server_cmd.txt
+    head_server.log
+    worker_server.log
+  communication/
+    communication_baseline.py
+    communication_baseline.sha256
+    p2p_head.log
+    p2p_worker.log
+    allreduce_head_8gpu.log
+    allreduce_worker_8gpu.log
+    allreduce_two_node_x0.log
+    allreduce_two_node_x1.log
+    allreduce_two_node_x2.log
+  head/
+    gpu_samples.csv
+    dcgm_dmon.log
+    mpstat.log
+    pidstat.log
+    sar_net.log
+    perf_stat.log
+    docker_top.log
+    numa_samples.csv
+    rdma.csv
+    static_before.log
+    static_after.log
+    collector_commands/
+  worker/
+    ...
+  collector_status.csv
+  markers.csv
+  bench_summary.csv
+  gpu_summary.csv
+  rdma_summary.csv
+  case_windows.csv
+  communication_summary.csv
+  communication_aggregate.csv
+  case_gpu_summary.csv
+  case_gpu_node_summary.csv
+  case_dcgm_summary.csv
+  case_cpu_summary.csv
+  case_process_summary.csv
+  case_perf_summary.csv
+  case_numa_summary.csv
+  case_netdev_summary.csv
+  case_rdma_summary.csv
+  summary.json
+  report.md
+ +

8. 最终验收

+ + +

9. 正式运行

+

+ Run：dsv4pro-phase2-20260731-163620。 + 运行时间为 16:36:20 至 17:05:04 CST，总用时 28 分 44 秒。 + Manifest 记录代码提交 5f24b7d22f98108f6cc234edba6768d55ea0a962， + git_dirty=false。 +

+ 
# 仅在 Head 174.1.51.5 执行
+cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution
+
+RUN_ID=dsv4pro-phase2-20260731-163620
+tmux new-session -d -s dsv4pro-phase2 \
+  "RUN_ID=${RUN_ID} bash run_hardware_contention_attribution.sh all \
+   2>&1 | tee /data/hzy/${RUN_ID}.log"
+ +

10. 端到端结果

+

10.1 五个代表负载

+ + + + + + + + + + + +
CaseInput TPSOutput TPSTTFT P95TPOT P95
128K → 1, C=12,641.380.0249.610 s—
32K → 1, C=163,115.450.10161.938 s—
1K → 1K, C=32448.95448.9510.167 s65.36 ms
1K → 4K, C=1679.39317.561.724 s49.98 ms
128K → 1K, C=11,614.9912.6248.279 s32.11 ms
+ +

10.2 混合 Prefill/Decode A/B

+ + + + + + + + + + +
Decode 指标Control注入 128K Prefill变化
Output TPS453.55344.89-23.96%
TTFT P959.443 s9.892 s+4.76%
TPOT P9566.24 ms110.45 ms+66.75%
E2E P9572.270 s118.076 s+63.38%
+

+ 这是 Phase 2 最关键的现象：长 Prefill 与 Decode 共存时，首 token 延迟只增加 + 4.76%，但 Decode 单 token 成本增加 66.75%，最终令 Output TPS 下降 23.96%。 + 问题主要发生在持续 Decode 阶段，而不是只表现为 Prefill 请求排队。 +

+ +

11. 第 5 节指标逐项结果

+

11.1 GPU 基础状态与 DCGM

+ + +

11.2 CPU、进程与 NUMA

+ + +

11.3 双 Rail RDMA

+ + +

11.4 PCIe 与 NCCL 通信基线

+ + + + + + + + + + +
测试结果解释
Head PCIe P2P 256 MiB同 Switch 53.61 GB/s；跨 NUMA 52.40 GB/s跨 NUMA 损失约 2.3%
Worker PCIe P2P 256 MiB同 Switch 53.50 GB/s；跨 NUMA 52.32 GB/s两节点表现对称
Head / Worker 8-GPU AllReduce 1 GiBbusbw 39.76 / 39.75 GB/s节点内基线一致
16-GPU AllReduce，CROSS_NIC=0busbw 39.345 GB/s；51.169 ms正确性 0 错误
16-GPU AllReduce，CROSS_NIC=1busbw 39.685 GB/s；50.732 ms本轮数值最好
16-GPU AllReduce，CROSS_NIC=2busbw 39.530 GB/s；50.931 ms正确性 0 错误
+

+ NCCL_CROSS_NIC=1 比 0 仅高 0.86%，比 2 仅高 0.39%。 + 差异小于 1%，不足以把它当成主要调优旋钮；保留当前值即可，Phase 3 不再重复测试。 + NCCL 日志明确证明跨机路径使用 mlx5_0,mlx5_3 和 + NET/IB/.../GDRDMA。 +

+ +

12. 结论与 Phase 3 入口

+

+ Phase 2 已把范围明显缩小：混合 Prefill/Decode 退化真实且稳定， + 但不是由整机 CPU 饱和、GPU 降频、双 Rail 原始带宽饱和、Rail 失衡、 + PCIe 跨 NUMA 带宽崩塌或 NCCL_CROSS_NIC 选择造成。 + Phase 3 应只捕获 Control 与 Treatment 的短时间线，定位 Attention/Indexer、 + MoE、NCCL Collective、Scheduler gap 和慢 Rank 同步之间的串行与重叠关系。 +

+ + +

13. 证据与清理说明

+ +

+ Worker 日志在所有 benchmark 完成后的编排关闭阶段出现 Gloo + Connection closed by peer；时间与 Head 主动退出进程组一致， + 未影响 8/8 结果。NCCL 日志中的可选 mlx5 symbol 探测提示同样未影响 + Collective，全部正确性检查为 0 错误。 +

+ +

返回 Phase 1 实验档案

+

返回推理优化主计划

+
+ + diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/analysis.md b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/analysis.md new file mode 100644 index 0000000..cf549bf --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/analysis.md @@ -0,0 +1,51 @@ +# Phase 2 Hardware Attribution Summary + +## Run + +- Run ID: `dsv4pro-phase2-20260731-163620` +- Status: `COMPLETED` +- Duration: 2026-07-31 16:36:20 to 17:05:04 CST, 28m44s +- Benchmarks: 8/8 successful +- Precise benchmark windows: 8/8 +- Collectors: 18 started, 18 stopped +- Run commit: `5f24b7d22f98108f6cc234edba6768d55ea0a962` +- Git dirty: `false` + +## Main Result + +Injecting one 128K prefill request into the `1K -> 1K, C=32` decode workload caused: + +| Metric | Control | Treatment | Delta | +|---|---:|---:|---:| +| Output TPS | 453.55 | 344.89 | -23.96% | +| TTFT P95 | 9.443 s | 9.892 s | +4.76% | +| TPOT P95 | 66.24 ms | 110.45 ms | +66.75% | +| E2E P95 | 72.270 s | 118.076 s | +63.38% | + +The dominant impact is sustained decode slowdown, not only first-token queueing. + +## Attribution + +- GPU utilization is generally 94-99%, with stable 2.39-2.42 GHz clocks. There is no evidence of frequency collapse. +- Whole-host CPU active is about 9.6-10.4%; CPU capacity is not globally saturated. +- Head and Worker show similar GPU, CPU and communication behavior. There is no obvious slow-node imbalance. +- The highest measured RDMA traffic is about 83.5 Gbit/s per 400G rail. Both rails are balanced, with zero wait/error/discard/retry deltas. +- PCIe P2P bandwidth is about 53.5 GB/s within a switch and 52.3-52.4 GB/s across NUMA, only about a 2.3% difference. +- 16-GPU 1 GiB AllReduce bus bandwidth is 39.345/39.685/39.530 GB/s for `NCCL_CROSS_NIC=0/1/2`. The spread is below 1%. + +The evidence excludes raw rail saturation, whole-host CPU saturation, GPU clock throttling, severe PCIe cross-NUMA collapse, and `NCCL_CROSS_NIC` selection as the primary cause. + +## Phase 3 Target + +Capture short Control/Treatment timelines and identify: + +1. Attention/indexer and MoE kernel serialization. +2. NCCL collective placement and compute/communication overlap. +3. Scheduler or CPU launch gaps. +4. Slow-rank synchronization around injected prefill. + +Do not repeat Phase 2's long-running DCGM, CPU, RDMA or communication microbenchmarks. + +## Cleanup + +After the run, both nodes had no related Docker containers, no GPU compute processes, and no listeners on the Phase 2 service ports. Worker Gloo `Connection closed by peer` tracebacks occurred during orchestrated shutdown after all benchmarks completed and did not affect the results. diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/bench_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/bench_summary.csv new file mode 100644 index 0000000..2224527 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/bench_summary.csv @@ -0,0 +1,9 @@ +run_id,suite,case_id,role,stage,repetition,isl,osl,concurrency,num_prompts,warmup_requests,status,error_type,exit_code,started_at,ended_at,elapsed_s,measurement_started_at,measurement_ended_at,measurement_duration_s,measurement_window_source,completed,failed,duration_s,actual_concurrency,peak_concurrent_requests,total_input_tokens,total_output_tokens,request_throughput,input_token_throughput,output_token_throughput,total_token_throughput,peak_output_token_throughput,e2e_mean_ms,e2e_p50_ms,e2e_p95_ms,e2e_p99_ms,ttft_mean_ms,ttft_p50_ms,ttft_p95_ms,ttft_p99_ms,tpot_mean_ms,tpot_p50_ms,tpot_p95_ms,tpot_p99_ms,itl_mean_ms,itl_p50_ms,itl_p95_ms,itl_p99_ms,bench_file,bench_log,phase2_bench_run +fixed_decode_throughput_1k_to_1k_c32,fixed,decode_throughput_1k_to_1k_c32,,decode_throughput,1,1024,1024,32,32,1,COMPLETED,,0,2026-07-31T16:49:35+0800,2026-07-31T16:51:24+0800,109.0,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.98790900100721,bench_main_marker_plus_duration,32,0,72.98790900100721,31.973027693348868,,32768,32768,0.43842878139663394,448.95107215015315,448.95107215015315,897.9021443003063,,72926.38861777596,72926.03143348242,72939.17302195623,72943.4486198437,8317.972617309351,8423.306765995221,10166.788510262268,10167.903182043228,63.155831867513776,63.05236280694347,65.35758210354297,67.36198566667542,63.155803331197355,61.26605049939826,62.53751848998945,65.70577920065261,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/bench.log,fixed_decode_throughput_1k_to_1k_c32 +fixed_long_context_decode_128k_to_1k_c1,fixed,long_context_decode_128k_to_1k_c1,,long_context_decode,1,131072,1024,1,1,0,COMPLETED,,0,2026-07-31T16:55:45+0800,2026-07-31T16:57:39+0800,114.0,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.1598442849936,bench_main_marker_plus_duration,1,0,81.1598442849936,0.9995611458686556,,131072,1024,0.012321364201839645,1614.985848663526,12.617076942683797,1627.6029256062097,,81124.22695202986,81124.22695202986,81124.22695202986,81124.22695202986,48279.420554987155,48279.420554987155,48279.420554987155,48279.420554987155,32.106360114411245,32.106360114411245,32.106360114411245,32.106360114411245,32.10634595800085,32.09696803241968,32.28448228328489,32.53332086256705,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/bench.log,fixed_long_context_decode_128k_to_1k_c1 +fixed_long_output_decode_1k_to_4k_c16,fixed,long_output_decode_1k_to_4k_c16,,long_output_decode,1,1024,4096,16,16,1,COMPLETED,,0,2026-07-31T16:51:33+0800,2026-07-31T16:55:36+0800,243.0,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074682698585,bench_main_marker_plus_duration,16,0,206.37074682698585,15.99550577130734,,16384,65536,0.07753036826199912,79.3910971002871,317.5643884011484,396.95548550143553,,206312.77949375362,206311.27990750247,206327.66740498482,206328.1800113793,1697.1331631830253,1714.9575955118053,1723.5681610036409,1724.2807810165687,49.96719080111614,49.96277789426234,49.98305734157135,50.02649938860528,49.96718302440562,49.99568348284811,50.960358188604005,53.57307427912019,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/bench.log,fixed_long_output_decode_1k_to_4k_c16 +fixed_long_prefill_latency_128k_c1,fixed,long_prefill_latency_128k_c1,,prefill_latency,1,131072,1,1,1,0,COMPLETED,,0,2026-07-31T16:44:30+0800,2026-07-31T16:45:54+0800,84.0,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567456041,bench_main_marker_plus_duration,1,0,49.622567456041,0.9997500965858968,,131072,1,0.02015212132838284,2641.3788467537956,0.02015212132838284,2641.398998875124,,49610.16660701716,49610.16660701716,49610.16660701716,49610.16660701716,49610.14223104576,49610.14223104576,49610.14223104576,49610.14223104576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/bench.log,fixed_long_prefill_latency_128k_c1 +fixed_mid_prefill_throughput_32k_c16,fixed,mid_prefill_throughput_32k_c16,,prefill_throughput,1,32768,1,16,16,0,COMPLETED,,0,2026-07-31T16:46:03+0800,2026-07-31T16:49:26+0800,203.0,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.2866359280306,bench_main_marker_plus_duration,16,0,168.2866359280306,8.764467555247327,,524288,16,0.09507587998159613,3115.446435236942,0.09507587998159613,3115.5415111169236,,92183.92253580896,92587.60345654446,161938.1903584872,166972.64813809306,92183.87291699764,92587.55126551841,161938.13879448862,166972.5966172962,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/bench.log,fixed_mid_prefill_throughput_32k_c16 +mixed_prefill_decode_interference,mixed,decode_control_1k_to_1k_c32,control,mixed_interference,1,1024,1024,32,64,1,COMPLETED,,0,2026-07-31T16:57:48+0800,2026-07-31T17:00:49+0800,181.0,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.4964013720164,bench_main_marker_plus_duration,64,0,144.4964013720164,31.983334150914263,,65536,65536,0.4429176048144437,453.54762732999035,453.54762732999035,907.0952546599807,,72210.57326071605,72215.36969349836,72269.92528844276,72271.96539751603,7325.20390184618,6846.627883001929,9442.68154159945,9444.044730461319,63.426558513069274,63.83972124386403,66.23843437823616,68.16205457722045,63.42653002822952,61.320325010456145,62.66533228044864,65.59292251302395,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/bench.log,mixed_prefill_decode_interference +mixed_prefill_decode_interference,mixed,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,1024,1024,32,64,0,COMPLETED,,0,2026-07-31T17:00:49+0800,2026-07-31T17:04:32+0800,223.0,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.02009954402456,bench_main_marker_plus_duration,64,0,190.02009954402456,31.986201956249488,,65536,65536,0.3368064754916742,344.8898309034744,344.8898309034744,689.7796618069488,,94969.08249627813,94968.0481730029,118076.02923363738,118079.88882219535,7606.184888919415,7407.497464475455,9891.837346865213,9893.77336708887,85.3987268889137,86.89664765445467,110.451060616212,111.28696239483178,85.39869778190757,61.08463852433488,61.97826337593142,65.3158738068305,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/bench.log,mixed_prefill_decode_interference +mixed_prefill_decode_interference,mixed,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,131072,1,1,1,0,COMPLETED,,0,2026-07-31T17:01:29+0800,2026-07-31T17:02:47+0800,78.0,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.06182585703209,bench_main_marker_plus_duration,1,0,46.06182585703209,0.9982293388832256,,131072,1,0.02170995138368649,2845.5667477625557,0.02170995138368649,2845.5884577139395,,45980.265973019414,45980.265973019414,45980.265973019414,45980.265973019414,45980.19455798203,45980.19455798203,45980.19455798203,45980.19455798203,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.jsonl,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/bench.log,mixed_prefill_decode_interference diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_cpu_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_cpu_summary.csv new file mode 100644 index 0000000..9124ae0 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_cpu_summary.csv @@ -0,0 +1,17 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,cpu_active_pct_mean,cpu_active_pct_p95,cpu_active_pct_max,iowait_pct_mean,iowait_pct_p95,iowait_pct_max,hot_cores_ge80_mean,hot_cores_ge80_p95,hot_cores_ge80_max +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,15,10.130666666666666,10.418000000000003,10.459999999999994,0.0,0.0,0.0,10.466666666666667,12.0,12.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,15,9.843333333333332,10.085999999999997,10.099999999999994,0.0,0.0,0.0,10.4,12.0,12.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,17,9.899411764705883,10.292000000000003,10.299999999999997,0.0,0.0,0.0,10.588235294117647,12.0,12.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,17,9.651176470588235,10.089999999999996,10.129999999999995,0.0,0.0,0.0,10.352941176470589,12.0,12.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,41,10.361707317073172,10.519999999999996,10.620000000000005,0.0,0.0,0.0,11.024390243902438,12.0,12.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,41,9.994390243902439,10.090000000000003,10.159999999999997,0.0,0.0,0.0,11.414634146341463,12.0,12.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,10,10.003,10.272999999999996,10.299999999999997,0.0,0.0,0.0,10.1,12.0,12.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,10,9.688,10.180500000000004,10.230000000000004,0.0,0.0,0.0,10.2,12.0,12.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,34,9.985588235294117,10.267000000000003,10.319999999999993,0.0,0.0,0.0,11.058823529411764,12.0,12.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,34,9.84235294117647,10.1135,10.14,0.0,0.0,0.0,11.205882352941176,12.0,12.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,29,10.262758620689656,10.436,10.489999999999995,0.0,0.0,0.0,10.620689655172415,12.0,12.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,29,9.976551724137929,10.108000000000004,10.120000000000005,0.0,0.0,0.0,11.241379310344827,12.0,12.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,38,10.341052631578947,11.3725,12.36,0.0,0.0,0.0,10.868421052631579,12.149999999999999,13.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,38,9.868684210526316,10.091500000000002,10.14,0.0,0.0,0.0,10.868421052631579,12.0,12.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,10,10.212,10.630499999999998,10.950000000000003,0.0,0.0,0.0,11.6,12.549999999999999,13.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,9,10.035555555555556,10.120000000000001,10.14,0.0,0.0,0.0,11.666666666666666,12.0,12.0 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_dcgm_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_dcgm_summary.csv new file mode 100644 index 0000000..f6be447 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_dcgm_summary.csv @@ -0,0 +1,17 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,gpus,samples,gr_engine_active_mean,gr_engine_active_p95,gr_engine_active_max,sm_active_mean,sm_active_p95,sm_active_max,sm_occupancy_mean,sm_occupancy_p95,sm_occupancy_max,tensor_active_mean,tensor_active_p95,tensor_active_max,dram_active_mean,dram_active_p95,dram_active_max,pcie_tx_bytes_per_s_mean,pcie_tx_bytes_per_s_p95,pcie_tx_bytes_per_s_max,pcie_rx_bytes_per_s_mean,pcie_rx_bytes_per_s_p95,pcie_rx_bytes_per_s_max +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,8,584,0.9710171232876712,0.999,1.0,0.5252157534246575,0.7,0.707,0.14078253424657533,0.291,0.294,0.05895205479452054,0.11499999999999999,0.12,0.41691609589041095,0.499,0.508,5205379813.392123,14778814130.95,15202540154.0,5200970877.481164,14816565450.150002,15254205047.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,8,576,0.9785503472222222,0.999,0.999,0.5226267361111111,0.6872499999999999,0.698,0.14059895833333333,0.28824999999999995,0.293,0.05846874999999999,0.113,0.117,0.4153368055555555,0.495,0.548,5236209935.171875,14647937823.25,16403294625.0,5231738841.786458,14714002991.25,16357346290.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,8,648,0.9613811728395062,0.9990000000000001,0.999,0.5782654320987655,0.76,0.768,0.21059722222222224,0.282,0.284,0.09468981481481482,0.162,0.168,0.3985987654320987,0.44500000000000006,0.479,6763589500.305555,13241413192.75,14068752272.0,6821390469.66821,13261233168.150002,14425674313.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,8,648,0.965466049382716,0.9990000000000001,1.0,0.5789398148148148,0.76,0.768,0.21097993827160494,0.281,0.285,0.09508796296296296,0.162,0.166,0.4005216049382716,0.44600000000000006,0.47,6780236898.856482,13234577683.95,13889246455.0,6837638258.785494,13289110201.45,13879127175.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,8,1648,0.9890279126213591,0.996,0.997,0.4903646844660194,0.5,0.54,0.13298240291262137,0.135,0.202,0.04484769417475728,0.048,0.102,0.41422026699029124,0.427,0.469,2620560854.958738,2632307969.6499996,11954964997.0,2634422729.365898,2682732685.6,10279631228.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,8,1640,0.9897713414634146,0.996,1.0,0.4890847560975609,0.498,0.683,0.13288231707317075,0.135,0.268,0.044690243902439024,0.047,0.129,0.4142451219512195,0.427,0.468,2618598082.628049,2632407781.75,14221424532.0,2633369965.667683,2681936017.75,13880734362.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,8,400,0.9295425,0.999,0.999,0.6828425,0.761,0.769,0.25086,0.282,0.284,0.1358075,0.163,0.168,0.37432499999999996,0.44904999999999995,0.505,10536527950.9525,13519030283.25,15163087193.0,10581948383.175,13517471298.849998,15273700217.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,8,392,0.9359923469387755,0.9989999999999999,1.0,0.6856326530612245,0.76,0.765,0.250795918367347,0.282,0.285,0.13560204081632654,0.162,0.165,0.37558673469387754,0.45324999999999993,0.485,10617939301.517857,13498405363.0,13917911496.0,10632410279.880102,13503375869.85,14629076948.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,8,1344,0.9804598214285715,0.999,1.0,0.7134583333333333,0.739,0.744,0.2776354166666667,0.287,0.288,0.1272514880952381,0.138,0.141,0.4404389880952381,0.467,0.509,12664097012.861607,14053446776.599998,15685065820.0,12719741431.203125,14071304248.15,15724306505.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,8,1344,0.9839747023809524,0.999,1.0,0.7149203869047619,0.739,0.743,0.27822619047619046,0.287,0.289,0.12745982142857143,0.137,0.14,0.44101190476190477,0.467,0.514,12698564254.736607,14063118317.15,15508567216.0,12754602448.509672,14079354109.05,15319006836.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,8,1152,0.9889574652777777,0.999,1.0,0.538584201388889,0.699,0.705,0.1456640625,0.292,0.295,0.060828125,0.11499999999999999,0.121,0.4266675347222222,0.5,0.558,5451444350.41493,14941958481.6,16367702095.0,5456132661.967882,14993495237.7,16348072773.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,8,1152,0.9896744791666667,0.999,1.0,0.5361883680555555,0.697,0.704,0.1450859375,0.291,0.295,0.06014930555555556,0.114,0.119,0.4251788194444444,0.5,0.55,5464990127.207465,14978142467.55,16515378482.0,5460547526.944445,14998798408.1,15300436571.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,8,1520,0.9768513157894737,0.999,1.0,0.5805618421052632,0.748,0.763,0.17383618421052632,0.291,0.295,0.08014671052631578,0.15404999999999996,0.168,0.4163407894736842,0.499,0.51,6865359213.189474,14886672661.75,15342998825.0,6868476165.281579,14922047967.4,15374888508.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,8,1520,0.978421052631579,0.999,1.0,0.578371052631579,0.748,0.763,0.17389407894736844,0.291,0.295,0.0797078947368421,0.154,0.175,0.4157486842105263,0.498,0.511,6864820605.633553,14934618195.95,15291598078.0,6872189917.845395,14950386000.25,15356103677.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,8,368,0.9974673913043478,0.9990000000000001,1.0,0.7229048913043478,0.756,0.763,0.2642554347826087,0.285,0.288,0.13833152173913044,0.16,0.165,0.40672282608695653,0.4545999999999999,0.48,11151168359.230978,13665801629.75,14505291121.0,11197667600.644022,13730586781.449997,14645337945.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,8,368,0.9973179347826087,0.9990000000000001,0.999,0.7197690217391305,0.757,0.763,0.2623777173913044,0.285,0.288,0.13687228260869566,0.15964999999999996,0.166,0.40697826086956523,0.45654999999999984,0.486,11068919985.380434,13568556767.3,14853005782.0,11109834217.782608,13585937423.0,14369923387.0 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_gpu_node_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_gpu_node_summary.csv new file mode 100644 index 0000000..997ddaf --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_gpu_node_summary.csv @@ -0,0 +1,17 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,gpu_util_pct_mean,gpu_util_pct_p95,gpu_util_pct_max,memory_util_pct_mean,memory_util_pct_p95,memory_util_pct_max,memory_used_mib_mean,memory_used_mib_p95,memory_used_mib_max,power_w_mean,power_w_p95,power_w_max,temperature_c_mean,temperature_c_p95,temperature_c_max,sm_clock_mhz_mean,sm_clock_mhz_p95,sm_clock_mhz_max,memory_clock_mhz_mean,memory_clock_mhz_p95,memory_clock_mhz_max,gpus +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,408,97.09313725490196,100.0,100.0,50.85539215686274,61.0,63.0,83300.0,83409.0,83409.0,230.21735294117644,299.2755,312.84,43.713235294117645,45.0,46.0,2402.950980392157,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,432,97.72222222222223,100.0,100.0,51.21527777777778,60.44999999999999,64.0,83300.0,83409.0,83409.0,221.8317592592593,289.236,307.91,45.844907407407405,47.0,49.0,2418.5486111111113,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,456,97.70394736842105,100.0,100.0,50.24122807017544,57.0,59.0,83219.06140350878,83453.0,83453.0,257.11598684210526,325.23,342.99,46.23026315789474,52.0,54.0,2412.627192982456,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,472,97.43008474576271,100.0,100.0,49.84110169491525,57.0,58.0,83221.32203389831,83453.0,83453.0,257.90165254237286,327.982,372.66,48.74576271186441,54.0,57.0,2417.531779661017,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,1184,98.82516891891892,100.0,100.0,50.710304054054056,53.0,59.0,83300.0,83409.0,83409.0,220.82934966216217,242.39100000000002,275.9,43.5304054054054,45.0,45.0,2402.7592905405404,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,1216,99.2467105263158,100.0,100.0,50.49753289473684,53.0,54.0,83300.0,83409.0,83409.0,215.73606907894737,236.6025,270.17,45.59950657894737,47.0,48.0,2412.5715460526317,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,288,94.44444444444444,100.0,100.0,46.74305555555556,56.0,58.0,83028.21527777778,83408.2,83443.0,293.3987847222222,330.7185,368.5,44.83680555555556,51.0,53.0,2392.215277777778,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,288,94.85069444444444,100.0,100.0,46.420138888888886,55.0,57.0,83057.76388888889,83408.2,83443.0,297.9095486111111,335.86899999999997,363.48,46.53819444444444,52.0,54.0,2398.0243055555557,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,952,98.20588235294117,100.0,100.0,53.52205882352941,58.0,59.0,83231.06722689075,83405.0,83409.0,299.4595693277311,316.97,332.42,50.438025210084035,55.0,56.0,2417.686974789916,2422.0,2422.0,12481.0,12481.0,12481.0,8 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,968,99.07438016528926,100.0,100.0,53.974173553719005,58.0,59.0,83231.94008264462,83405.0,83409.0,302.563285123967,322.293,339.05,53.75929752066116,59.0,61.0,2419.3326446280994,2422.0,2422.0,12481.0,12481.0,12481.0,8 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,824,98.52427184466019,100.0,100.0,51.5376213592233,61.0,65.0,83364.0,83473.0,83473.0,231.9156432038835,297.50299999999993,324.46,44.297330097087375,46.0,50.0,2405.38713592233,2422.0,2422.0,12481.0,12481.0,12481.0,8 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,856,98.60280373831776,100.0,100.0,51.441588785046726,61.0,64.0,83364.0,83473.0,83473.0,227.7883528037383,293.6475,322.45,46.43691588785047,49.0,52.0,2418.589953271028,2422.0,2422.0,12481.0,12481.0,12481.0,8 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,1072,99.60914179104478,100.0,100.0,51.59794776119403,60.0,65.0,83259.71082089552,83433.0,83433.0,248.31210820895524,315.4865,376.49,46.649253731343286,52.0,55.0,2411.4869402985073,2422.0,2422.0,12481.0,12481.0,12481.0,8 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,1112,98.92086330935251,100.0,100.0,51.085431654676256,60.0,64.0,83256.90647482015,83433.0,83433.0,247.74912769784174,321.8905,373.73,49.64118705035971,55.0,59.0,2420.7724820143885,2422.0,2422.0,12481.0,12481.0,12481.0,8 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,264,99.95075757575758,100.0,100.0,49.45075757575758,57.0,59.0,83281.49242424243,83413.0,83433.0,301.19901515151514,332.613,376.49,49.92424242424242,54.0,55.0,2421.1136363636365,2422.0,2422.0,12481.0,12481.0,12481.0,8 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,272,99.88602941176471,100.0,100.0,49.39705882352941,57.0,59.0,83282.94117647059,83413.0,83433.0,307.1604411764706,340.2845,373.73,52.86764705882353,57.0,59.0,2421.4595588235293,2422.0,2422.0,12481.0,12481.0,12481.0,8 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_netdev_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_netdev_summary.csv new file mode 100644 index 0000000..9223bfb --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_netdev_summary.csv @@ -0,0 +1,33 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,iface,samples,rx_gbps_mean,rx_gbps_p95,rx_gbps_max,tx_gbps_mean,tx_gbps_p95,tx_gbps_max,ifutil_pct_mean,ifutil_pct_p95,ifutil_pct_max,rx_errors_s_mean,rx_errors_s_p95,rx_errors_s_max,tx_errors_s_mean,tx_errors_s_p95,tx_errors_s_max,rx_drops_s_mean,rx_drops_s_p95,rx_drops_s_max,tx_drops_s_mean,tx_drops_s_p95,tx_drops_s_max +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,eth0,30,0.00015682133333333332,0.00016183999999999998,0.00016464,5.1274666666666665e-05,5.4591999999999995e-05,5.504e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,eth3,30,3.0933333333333335e-07,8.8e-07,8.8e-07,4.2666666666666665e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,eth0,30,5.1695999999999994e-05,5.4959999999999996e-05,5.608e-05,0.000156736,0.000161872,0.00016287999999999999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,eth3,30,3.0933333333333335e-07,8.8e-07,8.8e-07,4.2666666666666665e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,eth0,34,0.00015644235294117647,0.00015915199999999997,0.0001596,5.2367058823529417e-05,5.536e-05,5.568e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,eth3,34,3.435294117647059e-07,8.8e-07,8.8e-07,5.6470588235294123e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,eth0,34,5.264470588235294e-05,5.5552000000000006e-05,5.568e-05,0.0001562070588235294,0.00016,0.00016,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,eth3,34,3.435294117647059e-07,8.8e-07,8.8e-07,5.6470588235294123e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,eth0,82,0.0001566419512195122,0.00015984000000000001,0.00016272,4.933853658536585e-05,5.2560000000000005e-05,5.44e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,eth3,82,3.3365853658536586e-07,8.8e-07,8.8e-07,5.463414634146342e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,eth0,82,4.958634146341463e-05,5.4e-05,5.456e-05,0.00015623219512195122,0.00016128,0.00016287999999999999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,eth3,82,3.35609756097561e-07,8.8e-07,9.6e-07,5.6585365853658546e-08,3.2e-07,4.0000000000000003e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,eth0,20,0.00021578399999999998,0.0004841319999999994,0.00074848,5.9391999999999996e-05,8.969599999999994e-05,0.0001172,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,eth3,20,3.1200000000000004e-07,6.279999999999994e-07,8.8e-07,4e-08,2.1199999999999975e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,eth0,20,5.9592e-05,8.761999999999995e-05,0.00011336,0.000215264,0.00047957599999999935,0.00074,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,eth3,20,2.96e-07,6.279999999999994e-07,8.8e-07,3.2e-08,1.7599999999999967e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,eth0,68,0.0001568870588235294,0.000160908,0.00016112,5.124470588235294e-05,5.5707999999999996e-05,5.592e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,eth3,68,3.223529411764706e-07,8.8e-07,8.8e-07,4.705882352941176e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,eth0,68,5.149411764705883e-05,5.6268000000000004e-05,5.7520000000000005e-05,0.00015664235294117647,0.00016011200000000001,0.00016063999999999998,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,eth3,68,3.223529411764706e-07,8.8e-07,8.8e-07,4.705882352941176e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,eth0,58,0.00015664000000000002,0.000159888,0.00016,5.07751724137931e-05,5.4496e-05,5.456e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,eth3,58,3.3931034482758625e-07,8.8e-07,8.8e-07,5.517241379310345e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,eth0,58,5.0982068965517244e-05,5.4528000000000004e-05,5.568e-05,0.00015630896551724138,0.000160848,0.00016168,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,eth3,58,3.3931034482758625e-07,8.8e-07,8.8e-07,5.517241379310345e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,eth0,76,0.00015644631578947367,0.000160012,0.00016056,5.221263157894737e-05,5.579599999999999e-05,5.616e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,eth3,76,3.452631578947368e-07,8.8e-07,9.6e-07,6.105263157894737e-08,3.2e-07,4.0000000000000003e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,eth0,76,5.24042105263158e-05,5.641199999999999e-05,5.664e-05,0.0001559515789473684,0.00016033200000000004,0.00016424,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,eth3,76,3.431578947368421e-07,8.8e-07,8.8e-07,5.8947368421052637e-08,3.2e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,eth0,20,0.000156048,0.000158632,0.00015928,5.1064000000000004e-05,5.358e-05,5.376e-05,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,eth3,20,2.96e-07,6.279999999999994e-07,8.8e-07,3.2e-08,1.7599999999999967e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,eth0,18,5.1271111111111107e-05,5.4688e-05,5.52e-05,0.00015478222222222221,0.00016222399999999998,0.00016424,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,eth3,18,2.9333333333333337e-07,6.559999999999998e-07,8.8e-07,3.5555555555555554e-08,1.919999999999999e-07,3.2e-07,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_numa_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_numa_summary.csv new file mode 100644 index 0000000..93351fa --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_numa_summary.csv @@ -0,0 +1,17 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,node0_mib_mean,node0_mib_p95,node0_mib_max,node1_mib_mean,node1_mib_p95,node1_mib_max,total_mib_mean,total_mib_p95,total_mib_max,processes_mean,processes_p95,processes_max,numa_imbalance_pct_mean,numa_imbalance_pct_p95,numa_imbalance_pct_max +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,13,17941.21923076923,17941.472,17941.61,27565.280769230772,27565.394,27565.4,45506.51384615385,45506.69,45506.69,19.0,19.0,19.0,21.148755918904936,21.149743646540415,21.150655911887284 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,13,19029.48769230769,19029.536,19029.56,24814.318461538463,24814.381999999998,24814.4,43843.8,43843.854,43843.86,18.0,18.0,18.0,13.194182003167738,13.194329421344158,13.194370476132915 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,13,17946.50153846154,17948.994,17949.0,27569.42769230769,27571.648,27571.66,45515.943076923075,45520.61,45520.61,19.0,19.0,19.0,21.141880372077402,21.145764039410054,21.145782042670753 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,14,19037.215,19039.1035,19039.11,24816.07357142857,24818.423499999997,24818.43,43853.295,43857.54,43857.54,18.0,18.0,18.0,13.177706662062775,13.17809858730606,13.178146962903641 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,35,17941.668285714288,17942.12,17942.13,27565.19857142857,27566.617,27570.09,45506.88085714286,45506.89,45506.91,19.0,19.0,19.0,21.147417939082427,21.15369613423204,21.16893559148322 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,37,19029.40054054054,19029.452,19029.6,24814.58216216216,24814.65,24814.66,43843.96270270271,43843.97,43843.97,18.0,18.0,18.0,13.19493327010614,13.195223881414025,13.195246689567572 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,8,17869.37125,17877.1655,17878.8,27486.31625,27493.9225,27494.43,45355.7125,45371.1115,45373.25,19.0,19.0,19.0,21.203382762054954,21.207110348436384,21.207680920856813 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,8,18970.3375,18981.5545,18984.05,24731.71375,24743.9825,24745.68,43702.05125,43712.965,43715.31,18.0,18.0,18.0,13.183305003875798,13.21223924964188,13.21285380339291 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,28,17940.274285714284,17940.69,17940.72,27563.883928571428,27564.22,27564.23,45504.17285714286,45504.46,45504.46,19.0,19.0,19.0,21.148850835716523,21.149789612547433,21.149814841959135 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,30,19032.02333333333,19037.573,19037.75,24805.59,24810.55,24811.49,43837.58133333333,43841.73,43841.73,18.0,18.0,18.0,13.17036295064116,13.184651928653366,13.219751626067305 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,24,17946.96125,17951.6865,17951.9,27573.83541666667,27579.2765,27588.36,45520.80666666667,45520.8485,45520.85,19.0,19.0,19.0,21.148294318886816,21.17226656215764,21.212158491133064 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,26,19039.328461538462,19039.4575,19039.46,24818.277692307693,24818.565,24818.74,43857.62038461539,43857.65,43857.65,18.0,18.0,18.0,13.176613733571411,13.177963869896237,13.178761906998588 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,32,17955.7271875,17958.81,17958.83,27573.1746875,27577.874499999998,27578.04,45528.9121875,45535.71,45535.71,19.0,19.0,19.0,21.123824304701028,21.129656288079975,21.129778209910725 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,34,19042.789411764705,19046.17,19046.17,24823.40088235294,24827.86,24827.86,43866.178529411765,43873.26,43873.26,18.0,18.0,18.0,13.177832389566726,13.183982855860464,13.185098115743662 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,8,17951.77,17953.336,17953.98,27569.775,27571.0035,27571.7,45521.55625,45524.051,45525.71,19.0,19.0,19.0,21.128462660935995,21.129739766090218,21.129778209910725 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,8,19038.4325,19039.3115,19039.56,24819.52625,24820.626,24820.99,43857.942500000005,43858.878000000004,43859.48,18.0,18.0,18.0,13.181406637484468,13.184049863000734,13.184128037997716 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_perf_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_perf_summary.csv new file mode 100644 index 0000000..1f43764 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_perf_summary.csv @@ -0,0 +1,17 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,ipc,cache_misses_mean,cache_misses_p95,cache_misses_max,context_switches_mean,context_switches_p95,context_switches_max,cpu_migrations_mean,cpu_migrations_p95,cpu_migrations_max,page_faults_mean,page_faults_p95,page_faults_max +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,84,3.0487843541376685,6727288.428571428,15567285.149999999,18088446.0,41240.92857142857,47221.399999999994,58274.0,934.9285714285714,1975.849999999999,4063.0,46455.857142857145,161971.65,174542.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,84,3.0681062936795103,4913412.214285715,13142249.399999999,14049236.0,39682.0,45243.25,55120.0,188.57142857142858,247.7,249.0,28090.428571428572,113948.55,114568.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,96,2.8870888885724733,6295623.4375,12277537.0,21573640.0,44877.375,72712.0,151042.0,1255.1875,3534.25,9802.0,51319.875,105487.75,183145.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,96,2.9020975501844197,4051617.5625,7033576.5,10195935.0,43686.6875,70482.5,149831.0,177.5,304.0,364.0,12704.625,37867.0,65872.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,246,3.0979852154775953,5103378.804878049,5773629.0,14670415.0,43749.9756097561,44169.0,78897.0,998.9268292682926,1151.0,6568.0,18694.80487804878,28750.0,152065.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,246,3.1168786636888095,2962364.2682926827,2913232.0,13101053.0,40973.46341463415,41420.0,67313.0,165.17073170731706,209.0,213.0,11960.536585365853,14773.0,135313.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,60,2.727435494137309,9447284.2,31640896.59999995,53106505.0,50995.8,134631.5499999998,217846.0,2372.3,12092.349999999977,21775.0,108507.0,295694.8499999996,444909.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,60,2.7489508087655112,7473178.3,22657424.999999966,37078962.0,48210.5,120052.39999999983,191525.0,296.6,871.5499999999988,1394.0,107749.9,291126.8499999996,447550.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,204,2.8023469789284796,4623449.029411765,16572071.899999999,19670078.0,39421.294117647056,35324.499999999985,263082.0,861.4117647058823,365.9499999999998,20998.0,39086.14705882353,123447.0999999998,215308.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,204,2.8098589749897895,4072417.588235294,9610375.949999992,16711381.0,39129.382352941175,45578.64999999992,229122.0,164.7058823529412,246.5999999999997,423.0,36854.5,114042.04999999983,216180.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,174,3.0424513549285557,5137320.0,7469347.399999995,13520315.0,42398.8275862069,43356.2,91980.0,970.7586206896551,1053.6,5258.0,29120.41379310345,88587.9999999999,167311.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,174,3.0719083836401526,3223721.2413793104,5225881.199999996,9479153.0,40484.58620689655,39252.8,101396.0,167.3448275862069,197.2,236.0,16289.206896551725,65153.199999999895,95763.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,228,2.980275800109699,5456416.5,10599088.849999998,17028720.0,41223.05263157895,44802.899999999994,127813.0,1211.421052631579,1458.199999999999,15346.0,36016.86842105263,119273.24999999994,173922.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,228,3.0050223544930565,3784084.789473684,6404245.849999995,13894635.0,39834.92105263158,46921.44999999995,97372.0,167.5,262.15000000000003,299.0,22792.86842105263,79725.24999999994,157546.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,54,2.808104894473792,5020429.555555556,9669376.799999999,10387880.0,33399.666666666664,37769.399999999994,41129.0,338.8888888888889,811.1999999999997,1178.0,43405.88888888889,91275.39999999998,112743.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,60,2.8298366086279145,3857952.9,5373284.599999999,5873843.0,33911.7,40936.75,42415.0,130.4,172.39999999999998,185.0,24117.8,52089.24999999995,73750.0 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_process_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_process_summary.csv new file mode 100644 index 0000000..dc96686 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_process_summary.csv @@ -0,0 +1,17 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,samples,process_cpu_pct_mean,process_cpu_pct_p95,process_cpu_pct_max,process_wait_pct_mean,process_wait_pct_p95,process_wait_pct_max,process_read_kib_s_mean,process_read_kib_s_p95,process_read_kib_s_max,process_write_kib_s_mean,process_write_kib_s_p95,process_write_kib_s_max,process_major_faults_s_mean,process_major_faults_s_p95,process_major_faults_s_max,process_rss_kib_mean,process_rss_kib_p95,process_rss_kib_max,process_voluntary_switches_s_mean,process_voluntary_switches_s_p95,process_voluntary_switches_s_max,process_involuntary_switches_s_mean,process_involuntary_switches_s_p95,process_involuntary_switches_s_max +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,head,1140,1193.5866666666666,1223.7400000000002,1225.0000000000002,0.013333333333333334,0.05999999999999979,0.2,0.0,0.0,0.0,174.50666666666666,1036.1599999999996,1273.6000000000001,0.0,0.0,0.0,46588476.53333333,46588620.0,46588620.0,277.09333333333336,711.1799999999982,1938.0000000000002,13.32,31.619999999999997,34.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,72.987908864,bench_main_marker_plus_duration,worker,1080,1183.24,1210.06,1210.2,0.013333333333333334,0.05999999999999979,0.2,0.0,0.0,0.0,174.50666666666666,1036.1599999999996,1273.6000000000001,0.0,0.0,0.0,44887948.266666666,44888007.2,44888024.0,230.50666666666666,625.8599999999982,1744.6,10.986666666666666,31.319999999999997,31.599999999999998 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,head,1292,1173.3882352941175,1222.08,1222.4,0.011764705882352941,0.03999999999999986,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46598011.52941176,46602992.8,46602996.0,581.7294117647059,3130.9999999999986,5014.2,12.835294117647058,15.639999999999997,16.599999999999998 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.159844096,bench_main_marker_plus_duration,worker,1224,1160.7176470588236,1214.16,1215.6000000000001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44897810.35294118,44902216.0,44902216.0,545.2352941176471,3440.1999999999994,4371.400000000001,8.105882352941176,11.359999999999996,12.799999999999999 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,head,3116,1220.1268292682926,1225.6,1225.8,0.024390243902439025,0.2,0.2,0.0,0.0,0.0,9.990243902439024,0.0,409.59999999999997,0.0,0.0,0.0,46588905.26829268,46588920.0,46588920.0,225.06829268292682,227.6,647.8000000000001,7.86829268292683,10.200000000000001,28.2 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,206.37074688,bench_main_marker_plus_duration,worker,2952,1206.5365853658536,1211.0,1211.4,0.01951219512195122,0.2,0.2,0.0,0.0,0.0,9.990243902439024,0.0,409.59999999999997,0.0,0.0,0.0,44888251.90243903,44888308.0,44888316.0,176.34634146341463,176.2,602.6,5.590243902439025,7.400000000000001,18.200000000000003 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,head,1129,1147.0800000000002,1211.02,1211.1999999999998,0.02,0.10999999999999979,0.2,0.0,0.0,0.0,297.30909090909097,1635.2000000000003,3270.4000000000005,0.0,0.0,0.0,46439570.4,46454466.6,46456380.0,1940.6666666666672,11299.580000000002,13680.2,16.5,35.83999999999999,41.99999999999999 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,49.622567168,bench_main_marker_plus_duration,worker,720,1141.52,1210.62,1210.8,0.06000000000000001,0.2,0.2,0.0,0.0,0.0,327.0400000000001,1798.7199999999968,3270.4000000000005,0.0,0.0,0.0,44743050.0,44757778.6,44759728.0,872.5400000000006,4761.229999999995,8583.800000000007,14.919999999999998,27.009999999999973,37.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,head,2584,1184.6000000000001,1210.47,1211.0,0.03529411764705883,0.2,0.2,0.0,0.0,0.0,34.4470588235294,100.79999999999939,883.1999999999999,0.0,0.0,0.0,46582295.52941176,46586356.0,46586356.0,377.5411764705882,1431.4499999999923,8289.0,10.841176470588234,15.229999999999995,22.4 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,168.286636032,bench_main_marker_plus_duration,worker,2448,1182.3235294117646,1210.0,1210.2,0.011764705882352941,0.06999999999999958,0.2,0.0,0.0,0.0,34.44705882352941,271.3599999999999,620.8000000000001,0.0,0.0,0.0,44881817.88235294,44885672.0,44885672.0,348.7588235294118,1094.469999999994,8625.2,11.129411764705884,18.909999999999997,20.999999999999996 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,head,2204,1214.6758620689654,1220.12,1220.4,0.013793103448275864,0.11999999999999958,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46603182.06896552,46603218.4,46603220.0,214.6758620689655,201.43999999999997,1574.0,9.682758620689656,14.839999999999996,15.4 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.496400896,bench_main_marker_plus_duration,worker,2088,1204.3999999999999,1210.1200000000001,1210.4,0.020689655172413796,0.2,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44902315.862068966,44902356.0,44902356.0,170.98620689655175,146.44,1531.9999999999998,6.931034482758621,11.84,12.6 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,head,2888,1200.7473684210527,1223.2599999999998,1223.6,0.021052631578947368,0.2,0.2,0.0,0.0,0.0,11.452631578947367,0.0,435.19999999999993,0.0,0.0,0.0,46611446.84210526,46618484.0,46618484.0,281.61,426.2999999999979,4255.0,8.715789473684211,17.47999999999999,27.599999999999998 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,190.020100096,bench_main_marker_plus_duration,worker,2736,1189.6052631578948,1210.43,1211.4,0.021052631578947368,0.2,0.2,0.0,0.0,0.0,11.452631578947367,0.0,435.19999999999993,0.0,0.0,0.0,44911026.94736842,44918376.0,44918376.0,249.65263157894734,289.9599999999987,4796.0,6.526315789473684,11.919999999999998,18.8 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,head,749,1210.66,1217.6599999999999,1223.6,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,46604025.6,46607275.2,46610400.0,34.617999999999995,109.58999999999983,175.2,8.1,10.79,11.6 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,46.061826048,bench_main_marker_plus_duration,worker,648,1207.4,1210.92,1211.4,0.022222222222222223,0.11999999999999994,0.2,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,44902633.333333336,44903531.2,44904240.0,22.133333333333336,88.47999999999996,137.60000000000002,9.244444444444445,12.280000000000001,12.600000000000001 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_rdma_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_rdma_summary.csv new file mode 100644 index 0000000..012ea27 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_rdma_summary.csv @@ -0,0 +1,33 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,duration_s,window_source,node,hca,samples,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.344697344,bench_main_marker_plus_duration,head,mlx5_0,66,327229799692.0,327241990572.0,36.69282364341205,36.69419062713517,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.344697344,bench_main_marker_plus_duration,head,mlx5_3,66,327393677184.0,327405760832.0,36.71119950013029,36.712554459750265,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.149780736,bench_main_marker_plus_duration,worker,mlx5_0,68,327312761288.0,327298289132.0,36.80267266064959,36.801045427975055,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,71.149780736,bench_main_marker_plus_duration,worker,mlx5_3,68,327195798972.0,327191949064.0,36.78952155156224,36.789088672308345,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.07872896,bench_main_marker_plus_duration,head,mlx5_0,75,444257636548.0,444258274080.0,43.83469176160109,43.834754666583265,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,81.07872896,bench_main_marker_plus_duration,head,mlx5_3,75,445471520584.0,445472014020.0,43.95446512771776,43.95451381481548,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,80.561727488,bench_main_marker_plus_duration,worker,mlx5_0,77,450403115384.0,450403670936.0,44.72626190406251,44.7263170718964,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,80.561727488,bench_main_marker_plus_duration,worker,mlx5_3,77,444176313384.0,444176585108.0,44.10792342556576,44.10795040850254,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,205.0068928,bench_main_marker_plus_duration,head,mlx5_0,188,484528783604.0,484522438956.0,18.907804590812276,18.907557003117507,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,205.0068928,bench_main_marker_plus_duration,head,mlx5_3,188,484292016064.0,484262756928.0,18.898565192594344,18.897423411043476,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,204.622719744,bench_main_marker_plus_duration,worker,mlx5_0,194,483378692900.0,483384132588.0,18.898339089803788,18.89855176171067,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,204.622719744,bench_main_marker_plus_duration,worker,mlx5_3,194,483484778080.0,483489710268.0,18.902486632369254,18.902679462882155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.080653568,bench_main_marker_plus_duration,head,mlx5_0,45,428992803940.0,428992820060.0,71.37886398874002,71.3788666708999,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.080653568,bench_main_marker_plus_duration,head,mlx5_3,45,429439114384.0,429439126660.0,71.45312428445231,71.45312632702023,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.829253632,bench_main_marker_plus_duration,worker,mlx5_0,47,430547106324.0,430552482856.0,70.53920742984172,70.54008830048383,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,48.829253632,bench_main_marker_plus_duration,worker,mlx5_3,47,430553276516.0,430553408016.0,70.54021833073266,70.54023987519466,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,167.623034624,bench_main_marker_plus_duration,head,mlx5_0,154,1738843459984.0,1738965693932.0,82.98828207636016,82.99411583057059,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,167.623034624,bench_main_marker_plus_duration,head,mlx5_3,154,1739263067700.0,1739364620460.0,83.00830833191347,83.01315505290157,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,166.729046016,bench_main_marker_plus_duration,worker,mlx5_0,158,1739033672624.0,1739033174332.0,83.44238579555551,83.4423618864881,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,166.729046016,bench_main_marker_plus_duration,worker,mlx5_3,158,1739243327096.0,1739243022428.0,83.45244544512514,83.45243082653253,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,142.715803392,bench_main_marker_plus_duration,head,mlx5_0,131,678101307872.0,678090133532.0,38.011280699416155,38.010654316647916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,142.715803392,bench_main_marker_plus_duration,head,mlx5_3,131,677949641072.0,677941962640.0,38.00277894718437,38.00234852914697,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.266830336,bench_main_marker_plus_duration,worker,mlx5_0,137,680623741960.0,680629335892.0,37.74249370419051,37.742803903394964,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,144.266830336,bench_main_marker_plus_duration,worker,mlx5_3,137,680750304048.0,680766879432.0,37.74951192661656,37.75043107810614,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,188.527344896,bench_main_marker_plus_duration,head,mlx5_0,173,1102872843476.0,1102880032984.0,46.79948552118605,46.79979060193723,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,188.527344896,bench_main_marker_plus_duration,head,mlx5_3,173,1103015616220.0,1103029812428.0,46.80554396301384,46.806146367211824,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,189.088602368,bench_main_marker_plus_duration,worker,mlx5_0,179,1102994734488.0,1102998788292.0,46.66573111969494,46.665902628879486,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,189.088602368,bench_main_marker_plus_duration,worker,mlx5_3,179,1103130615912.0,1103144127560.0,46.67148001930277,46.67205167292254,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.866369024,bench_main_marker_plus_duration,head,mlx5_0,42,415875487184.0,415864521032.0,74.15362486080193,74.15166951612152,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.866369024,bench_main_marker_plus_duration,head,mlx5_3,42,416276974024.0,416281015316.0,74.22521288519235,74.22593347695637,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.584492544,bench_main_marker_plus_duration,worker,mlx5_0,43,415451144696.0,415441323744.0,74.5463045090838,74.54454229061909,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,44.584492544,bench_main_marker_plus_duration,worker,mlx5_3,43,415600251668.0,415656009484.0,74.57305945700259,74.58306433768075,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_windows.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_windows.csv new file mode 100644 index 0000000..09304da --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/case_windows.csv @@ -0,0 +1,9 @@ +phase2_bench_run,case_id,role,stage,repetition,status,started_at,ended_at,started_ns,ended_ns,duration_s,window_source,meta_path +fixed_decode_throughput_1k_to_1k_c32,decode_throughput_1k_to_1k_c32,,decode_throughput,1,COMPLETED,2026-07-31T16:50:08.318962+08:00,2026-07-31T16:51:21.306871+08:00,1785487808318962176,1785487881306871040,72.987908864,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_decode_throughput_1k_to_1k_c32/cases/decode_throughput_1k_to_1k_c32/rep1/meta.json +fixed_long_context_decode_128k_to_1k_c1,long_context_decode_128k_to_1k_c1,,long_context_decode,1,COMPLETED,2026-07-31T16:56:15.173982+08:00,2026-07-31T16:57:36.333826+08:00,1785488175173981952,1785488256333826048,81.159844096,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_context_decode_128k_to_1k_c1/cases/long_context_decode_128k_to_1k_c1/rep1/meta.json +fixed_long_output_decode_1k_to_4k_c16,long_output_decode_1k_to_4k_c16,,long_output_decode,1,COMPLETED,2026-07-31T16:52:06.490086+08:00,2026-07-31T16:55:32.860833+08:00,1785487926490086144,1785488132860833024,206.37074688,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_output_decode_1k_to_4k_c16/cases/long_output_decode_1k_to_4k_c16/rep1/meta.json +fixed_long_prefill_latency_128k_c1,long_prefill_latency_128k_c1,,prefill_latency,1,COMPLETED,2026-07-31T16:45:02.091364+08:00,2026-07-31T16:45:51.713931+08:00,1785487502091363840,1785487551713931008,49.622567168,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_long_prefill_latency_128k_c1/cases/long_prefill_latency_128k_c1/rep1/meta.json +fixed_mid_prefill_throughput_32k_c16,mid_prefill_throughput_32k_c16,,prefill_throughput,1,COMPLETED,2026-07-31T16:46:35.012884+08:00,2026-07-31T16:49:23.299520+08:00,1785487595012883968,1785487763299520000,168.286636032,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/fixed_mid_prefill_throughput_32k_c16/cases/mid_prefill_throughput_32k_c16/rep1/meta.json +mixed_prefill_decode_interference,decode_control_1k_to_1k_c32,control,mixed_interference,1,COMPLETED,2026-07-31T16:58:21.482129+08:00,2026-07-31T17:00:45.978530+08:00,1785488301482129152,1785488445978530048,144.496400896,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_control_1k_to_1k_c32/rep1/meta.json +mixed_prefill_decode_interference,decode_with_128k_prefill_1k_to_1k_c32,decode_background,mixed_interference,1,COMPLETED,2026-07-31T17:01:19.288049+08:00,2026-07-31T17:04:29.308149+08:00,1785488479288048896,1785488669308148992,190.020100096,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/decode_with_128k_prefill_1k_to_1k_c32/rep1/meta.json +mixed_prefill_decode_interference,long_prefill_injection_128k_to_1_c1,prefill_injection,mixed_interference,1,COMPLETED,2026-07-31T17:01:59.428991+08:00,2026-07-31T17:02:45.490817+08:00,1785488519428990976,1785488565490817024,46.061826048,bench_main_marker_plus_duration,/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution/results/dsv4pro-phase2-20260731-163620/bench/mixed_prefill_decode_interference/cases/long_prefill_injection_128k_to_1_c1/rep1/meta.json diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/collector_status.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/collector_status.csv new file mode 100644 index 0000000..d00c2f0 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/collector_status.csv @@ -0,0 +1,37 @@ +wall_time_ns,node,collector,status,pid +1785487450093977528,head,gpu_samples.csv,STARTED,1795123 +1785487450108679448,head,rdma.csv,STARTED,1795134 +1785487450121779856,head,dcgm_dmon.log,STARTED,1795147 +1785487450134856077,head,mpstat.log,STARTED,1795163 +1785487450147239051,head,pidstat.log,STARTED,1795181 +1785487450160535144,head,sar_net.log,STARTED,1795207 +1785487450175160784,head,perf_stat.log,STARTED,1795238 +1785487450193437502,head,docker_top.log,STARTED,1795276 +1785487450214351210,head,numa_samples.csv,STARTED,1795348 +1785487450236070096,worker,gpu_samples.csv,STARTED,1795394 +1785487450254195207,worker,rdma.csv,STARTED,1795451 +1785487450270545415,worker,dcgm_dmon.log,STARTED,1795504 +1785487450285967202,worker,mpstat.log,STARTED,1795552 +1785487450301984460,worker,pidstat.log,STARTED,1795567 +1785487450317255696,worker,sar_net.log,STARTED,1795577 +1785487450331874012,worker,perf_stat.log,STARTED,1795584 +1785487450348717530,worker,docker_top.log,STARTED,1795593 +1785487450364742164,worker,numa_samples.csv,STARTED,1795604 +1785488689789962654,head,gpu_samples.csv,STOPPED,1795123 +1785488689794788834,head,rdma.csv,STOPPED,1795134 +1785488689801075236,head,dcgm_dmon.log,STOPPED,1795147 +1785488689805422095,head,mpstat.log,STOPPED,1795163 +1785488689810734986,head,pidstat.log,STOPPED,1795181 +1785488689814680689,head,sar_net.log,STOPPED,1795207 +1785488689951029262,head,perf_stat.log,STOPPED,1795238 +1785488689955936272,head,docker_top.log,STOPPED,1795276 +1785488689960978121,head,numa_samples.csv,STOPPED,1795348 +1785488689965181539,worker,gpu_samples.csv,STOPPED,1795394 +1785488689969473724,worker,rdma.csv,STOPPED,1795451 +1785488689973568170,worker,dcgm_dmon.log,STOPPED,1795504 +1785488689977772276,worker,mpstat.log,STOPPED,1795552 +1785488689982041873,worker,pidstat.log,STOPPED,1795567 +1785488689986056700,worker,sar_net.log,STOPPED,1795577 +1785488689990039780,worker,perf_stat.log,STOPPED,1795584 +1785488689994084643,worker,docker_top.log,STOPPED,1795593 +1785488689998332256,worker,numa_samples.csv,STOPPED,1795604 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/communication_aggregate.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/communication_aggregate.csv new file mode 100644 index 0000000..e3bfc63 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/communication_aggregate.csv @@ -0,0 +1,20 @@ +test,scope,nccl_cross_nic,size_bytes,repetitions,mean_ms,algbw_GBps_mean,busbw_GBps_mean,busbw_GBps_min,wrong_values,path_class,samples,bandwidth_GBps_mean,bandwidth_GBps_p05,bandwidth_GBps_min +all_reduce,head_8gpu,2,1048576,3,1.1792767981688181,0.9351587648306614,1.6365278384536577,1.1619881842280209,0,,,,, +all_reduce,head_8gpu,2,1073741824,3,47.2596331278483,22.72016498475108,39.76028872331439,39.67252737812032,0,,,,, +all_reduce,head_8gpu,2,67108864,3,3.107903989156087,21.596044891582753,37.79307856026982,37.253524087802575,0,,,,, +all_reduce,two_node_16gpu,0,1048576,3,1.174121594429016,0.9225129006791467,1.7297116887734,1.3203079074091486,0,,,,, +all_reduce,two_node_16gpu,0,1073741824,3,51.1693105061849,20.984136487918786,39.34525591484772,39.296690886871644,0,,,,, +all_reduce,two_node_16gpu,0,67108864,3,3.4025983730951945,19.72330661126499,36.98119989612186,36.724996463125755,0,,,,, +all_reduce,two_node_16gpu,1,1048576,3,1.2971573332945505,0.877370746063573,1.6450701488691994,1.0568849765163506,0,,,,, +all_reduce,two_node_16gpu,1,1073741824,3,50.73157666524251,21.165255754284047,39.68485453928258,39.59303458131569,0,,,,, +all_reduce,two_node_16gpu,1,67108864,3,3.3778613169988,19.86842574374279,37.25329826951773,36.858805194578224,0,,,,, +all_reduce,two_node_16gpu,2,1048576,3,1.1436874707539877,0.936877857865813,1.7566459834983992,1.413443823860872,0,,,,, +all_reduce,two_node_16gpu,2,1073741824,3,50.931247075398765,21.082601839476187,39.52987844901785,39.33647232214203,0,,,,, +all_reduce,two_node_16gpu,2,67108864,3,3.4341418504714967,19.544645327030477,36.64620998818214,36.14018010573151,0,,,,, +all_reduce,worker_8gpu,2,1048576,3,0.9693984071413676,1.0921968526479142,1.9113444921338498,1.664030821168365,0,,,,, +all_reduce,worker_8gpu,2,1073741824,3,47.27112070719401,22.714924401498006,39.75111770262151,39.55926639799348,0,,,,, +all_reduce,worker_8gpu,2,67108864,3,3.0737973292668657,21.837161295289466,38.215032266756566,37.43531449904809,0,,,,, +p2p_copy,head,,268435456,,,,,,,cross_numa_sys,32,52.397332123279426,52.19168649638452,52.18129785806688 +p2p_copy,head,,268435456,,,,,,,same_pcie_switch,24,53.6126013471359,53.36177601211484,53.35073474580275 +p2p_copy,worker,,268435456,,,,,,,cross_numa_sys,32,52.31572860916867,52.14858683841258,52.07724663835188 +p2p_copy,worker,,268435456,,,,,,,same_pcie_switch,24,53.50030938041238,53.28731762907832,53.221028060678236 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/gpu_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/gpu_summary.csv new file mode 100644 index 0000000..cfd9d3d --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/gpu_summary.csv @@ -0,0 +1,17 @@ +node,gpu,samples,gpu_util_pct_mean,gpu_util_pct_p95,gpu_util_pct_max,memory_util_pct_mean,memory_util_pct_p95,memory_util_pct_max,memory_used_mib_mean,memory_used_mib_p95,memory_used_mib_max,power_w_mean,power_w_p95,power_w_max,temperature_c_mean,temperature_c_p95,temperature_c_max,sm_clock_mhz_mean,sm_clock_mhz_p95,sm_clock_mhz_max,memory_clock_mhz_mean,memory_clock_mhz_p95,memory_clock_mhz_max +head,0,885,73.19774011299435,100.0,100.0,38.57514124293785,58.0,64.0,83114.45536723164,83361.0,83361.0,205.47479096045197,307.69199999999995,333.9,43.47570621468927,52.0,55.0,2393.25197740113,2422.0,2430.0,12481.0,12481.0,12481.0 +head,1,885,73.50056497175142,100.0,100.0,37.15819209039548,56.0,62.0,83080.49830508475,83327.0,83327.0,206.62670056497177,313.188,350.0,43.737853107344634,53.0,56.0,2378.2225988700566,2422.0,2422.0,12481.0,12481.0,12481.0 +head,2,885,73.52542372881356,100.0,100.0,38.364971751412426,57.0,63.0,83048.5209039548,83295.0,83295.0,210.91588700564972,317.204,371.02,42.911864406779664,52.0,55.0,2391.7344632768363,2422.0,2422.0,12481.0,12481.0,12481.0 +head,3,885,73.10960451977401,100.0,100.0,38.39322033898305,58.0,65.0,83146.58870056497,83393.0,83393.0,204.69710734463277,310.6,340.24,43.44858757062147,52.0,55.0,2388.697175141243,2422.0,2422.0,12481.0,12481.0,12481.0 +head,4,885,73.17062146892656,100.0,100.0,39.108474576271185,58.0,65.0,83194.63389830508,83441.0,83441.0,207.94813559322034,316.352,339.4,44.054237288135596,53.0,56.0,2402.890395480226,2422.0,2422.0,12481.0,12481.0,12481.0 +head,5,885,73.44519774011299,100.0,100.0,38.371751412429376,57.0,63.0,83080.63389830508,83327.0,83327.0,209.06979661016948,317.16999999999996,376.49,43.85084745762712,53.0,55.0,2399.3717514124296,2422.0,2422.0,12481.0,12481.0,12481.0 +head,6,885,73.67570621468927,100.0,100.0,39.31412429378531,58.0,65.0,83048.65423728814,83295.0,83295.0,205.60972881355931,310.78,342.99,43.70395480225989,52.0,55.0,2405.515254237288,2422.0,2422.0,12481.0,12481.0,12481.0 +head,7,885,73.06553672316385,100.0,100.0,35.74124293785311,54.0,60.0,83227.83163841808,83473.0,83473.0,206.52320903954802,314.84799999999996,336.9,43.929943502824855,53.0,56.0,2388.154802259887,2422.0,2422.0,12481.0,12481.0,12481.0 +worker,0,913,73.23986856516977,100.0,100.0,37.39539978094195,56.0,62.0,83195.78422782038,83441.0,83441.0,210.65017524644028,326.81399999999996,347.67,46.10624315443592,56.0,60.0,2398.8400876232204,2422.0,2430.0,12481.0,12481.0,12481.0 +worker,1,913,73.35158817086528,100.0,100.0,38.92880613362541,58.0,64.0,83081.95947426069,83327.0,83327.0,202.77134720700985,310.3,335.13,45.61226725082147,55.0,58.0,2398.3099671412924,2422.0,2422.0,12481.0,12481.0,12481.0 +worker,2,913,73.34720700985761,100.0,100.0,38.708652792990144,58.0,64.0,83050.44140197152,83295.0,83295.0,203.5297152245345,312.52,373.73,46.55859802847755,57.0,60.0,2400.135815991238,2422.0,2422.0,12481.0,12481.0,12481.0 +worker,3,913,72.96276013143483,100.0,100.0,37.24096385542169,57.0,63.0,83229.27382256297,83473.0,83473.0,207.60696604600219,318.69399999999996,359.03,45.467688937568454,55.0,59.0,2409.73713033954,2422.0,2422.0,12481.0,12481.0,12481.0 +worker,4,913,72.84008762322016,100.0,100.0,36.52464403066813,55.0,63.0,83117.25410733845,83361.0,83361.0,204.54877327491786,314.376,347.28,45.67798466593647,56.0,59.0,2392.5739320920043,2422.0,2422.0,12481.0,12481.0,12481.0 +worker,5,913,73.26944140197152,100.0,100.0,38.52245345016429,58.0,64.0,83083.19058050384,83327.0,83327.0,204.3163417305586,315.148,356.12,46.33077765607886,56.0,60.0,2385.7513691128147,2422.0,2422.0,12481.0,12481.0,12481.0 +worker,6,913,73.3274917853231,100.0,100.0,38.77984665936473,58.0,64.0,83052.13253012048,83295.0,83295.0,206.93734939759034,319.012,355.56,46.1829134720701,56.0,59.0,2401.7447973713033,2422.0,2422.0,12481.0,12481.0,12481.0 +worker,7,913,73.27929901423877,100.0,100.0,37.28806133625411,57.0,64.0,83150.0230010953,83393.0,83393.0,206.63869660460023,320.18,353.19,47.00766703176342,57.0,61.0,2406.594742606791,2422.0,2422.0,12481.0,12481.0,12481.0 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/manifest.json b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/manifest.json new file mode 100644 index 0000000..cad1aa4 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/manifest.json @@ -0,0 +1,26 @@ +{ + "run_id": "dsv4pro-phase2-20260731-163620", + "status": "COMPLETED", + "created_at": "2026-07-31T16:36:20+08:00", + "completed_at": "2026-07-31T17:05:04+08:00", + "experiment": "dsv4pro_pro6000d_2node_sglang_hardware_contention_attribution", + "git_commit": "5f24b7d22f98108f6cc234edba6768d55ea0a962", + "git_dirty": false, + "phase1_entry": "/data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map/run_quick_map.sh", + "head_node": "10.101.0.11", + "worker_node": "10.101.0.13", + "fixed_case_ids": "long_prefill_latency_128k_c1,mid_prefill_throughput_32k_c16,decode_throughput_1k_to_1k_c32,long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1", + "run_mixed_case": true, + "sample_interval_s": 1, + "cpu_sample_interval_s": 5, + "process_sample_interval_s": 5, + "net_sample_interval_s": 5, + "perf_interval_ms": 5000, + "numastat_interval_s": 5, + "clock_skew_tolerance_s": 2, + "idle_baseline_s": 15, + "post_run_cooldown_s": 15, + "require_precise_windows": true, + "run_communication_baseline": true, + "dry_run": false +} diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/rdma_summary.csv b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/rdma_summary.csv new file mode 100644 index 0000000..eed93f4 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/rdma_summary.csv @@ -0,0 +1,5 @@ +node,hca,samples,duration_s,xmit_bytes,rcv_bytes,xmit_gbps,rcv_gbps,port_xmit_wait_delta,port_xmit_discards_delta,port_rcv_errors_delta,req_transport_retries_exceeded_delta,req_rnr_retries_exceeded_delta,roce_adp_retrans_delta,roce_adp_retrans_to_delta,np_ecn_marked_roce_packets_delta,np_cnp_sent_delta,rp_cnp_handled_delta +head,mlx5_0,1131,1239.614615552,5264991004888.0,5264992389656.0,33.97824413384156,33.978253070606144,0.0,0.0,0.0,0.0,,,,,, +head,mlx5_3,1130,1238.507734784,5266283609308.0,5266274185868.0,34.01696064644414,34.0168997768041,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +worker,mlx5_0,1168,1238.553399296,5272091713408.0,5272078266720.0,34.05322187257931,34.05313501842828,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +worker,mlx5_3,1168,1238.553399296,5264840174196.0,5264861659552.0,34.00638310589474,34.00652188299398,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/report.md b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/report.md new file mode 100644 index 0000000..69a170e --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/report.md @@ -0,0 +1,253 @@ +# Phase 2 Hardware Attribution + +- Generated: `2026-07-31T17:05:04+08:00` +- Bench rows: `8` +- Failed bench rows: `0` +- GPU summary rows: `16` +- RDMA summary rows: `4` +- Case windows: `8` +- Precise main-run windows: `8/8` +- Collector status counts: `{"STARTED": 18, "STOPPED": 18}` + +## 1. Benchmark results + +| Run | Case | Status | Input TPS | Output TPS | TTFT P95 (ms) | TPOT P95 (ms) | +|---|---|---|---:|---:|---:|---:| +| fixed_decode_throughput_1k_to_1k_c32 | decode_throughput_1k_to_1k_c32 | COMPLETED | 448.95107215015315 | 448.95107215015315 | 10166.788510262268 | 65.35758210354297 | +| fixed_long_context_decode_128k_to_1k_c1 | long_context_decode_128k_to_1k_c1 | COMPLETED | 1614.985848663526 | 12.617076942683797 | 48279.420554987155 | 32.106360114411245 | +| fixed_long_output_decode_1k_to_4k_c16 | long_output_decode_1k_to_4k_c16 | COMPLETED | 79.3910971002871 | 317.5643884011484 | 1723.5681610036409 | 49.98305734157135 | +| fixed_long_prefill_latency_128k_c1 | long_prefill_latency_128k_c1 | COMPLETED | 2641.3788467537956 | 0.02015212132838284 | 49610.14223104576 | 0.0 | +| fixed_mid_prefill_throughput_32k_c16 | mid_prefill_throughput_32k_c16 | COMPLETED | 3115.446435236942 | 0.09507587998159613 | 161938.13879448862 | 0.0 | +| mixed_prefill_decode_interference | decode_control_1k_to_1k_c32 | COMPLETED | 453.54762732999035 | 453.54762732999035 | 9442.68154159945 | 66.23843437823616 | +| mixed_prefill_decode_interference | decode_with_128k_prefill_1k_to_1k_c32 | COMPLETED | 344.8898309034744 | 344.8898309034744 | 9891.837346865213 | 110.451060616212 | +| mixed_prefill_decode_interference | long_prefill_injection_128k_to_1_c1 | COMPLETED | 2845.5667477625557 | 0.02170995138368649 | 45980.19455798203 | 0.0 | + +## 2. Measurement-window validity + +| Case | Role | Duration (s) | Window source | +|---|---|---:|---| +| decode_throughput_1k_to_1k_c32 | - | 72.99 | bench_main_marker_plus_duration | +| long_context_decode_128k_to_1k_c1 | - | 81.16 | bench_main_marker_plus_duration | +| long_output_decode_1k_to_4k_c16 | - | 206.37 | bench_main_marker_plus_duration | +| long_prefill_latency_128k_c1 | - | 49.62 | bench_main_marker_plus_duration | +| mid_prefill_throughput_32k_c16 | - | 168.29 | bench_main_marker_plus_duration | +| decode_control_1k_to_1k_c32 | control | 144.50 | bench_main_marker_plus_duration | +| decode_with_128k_prefill_1k_to_1k_c32 | decode_background | 190.02 | bench_main_marker_plus_duration | +| long_prefill_injection_128k_to_1_c1 | prefill_injection | 46.06 | bench_main_marker_plus_duration | + +## 3. GPU basic state (`nvidia-smi`) + +Data: `case_gpu_node_summary.csv`; raw: `head|worker/gpu_samples.csv`. + +| Case | Node | Samples | GPU util mean/p95 (%) | Memory used mean (MiB) | Power mean (W) | SM clock mean (MHz) | +|---|---|---:|---:|---:|---:|---:| +| decode_throughput_1k_to_1k_c32 | head | 408 | 97.09/100.00 | 83300.00 | 230.22 | 2402.95 | +| decode_throughput_1k_to_1k_c32 | worker | 432 | 97.72/100.00 | 83300.00 | 221.83 | 2418.55 | +| long_context_decode_128k_to_1k_c1 | head | 456 | 97.70/100.00 | 83219.06 | 257.12 | 2412.63 | +| long_context_decode_128k_to_1k_c1 | worker | 472 | 97.43/100.00 | 83221.32 | 257.90 | 2417.53 | +| long_output_decode_1k_to_4k_c16 | head | 1184 | 98.83/100.00 | 83300.00 | 220.83 | 2402.76 | +| long_output_decode_1k_to_4k_c16 | worker | 1216 | 99.25/100.00 | 83300.00 | 215.74 | 2412.57 | +| long_prefill_latency_128k_c1 | head | 288 | 94.44/100.00 | 83028.22 | 293.40 | 2392.22 | +| long_prefill_latency_128k_c1 | worker | 288 | 94.85/100.00 | 83057.76 | 297.91 | 2398.02 | +| mid_prefill_throughput_32k_c16 | head | 952 | 98.21/100.00 | 83231.07 | 299.46 | 2417.69 | +| mid_prefill_throughput_32k_c16 | worker | 968 | 99.07/100.00 | 83231.94 | 302.56 | 2419.33 | +| decode_control_1k_to_1k_c32 | head | 824 | 98.52/100.00 | 83364.00 | 231.92 | 2405.39 | +| decode_control_1k_to_1k_c32 | worker | 856 | 98.60/100.00 | 83364.00 | 227.79 | 2418.59 | +| decode_with_128k_prefill_1k_to_1k_c32 | head | 1072 | 99.61/100.00 | 83259.71 | 248.31 | 2411.49 | +| decode_with_128k_prefill_1k_to_1k_c32 | worker | 1112 | 98.92/100.00 | 83256.91 | 247.75 | 2420.77 | +| long_prefill_injection_128k_to_1_c1 | head | 264 | 99.95/100.00 | 83281.49 | 301.20 | 2421.11 | +| long_prefill_injection_128k_to_1_c1 | worker | 272 | 99.89/100.00 | 83282.94 | 307.16 | 2421.46 | + +## 4. GPU profiling counters (DCGM) + +Data: `case_dcgm_summary.csv`; raw: `head|worker/dcgm_dmon.log`. + +| Case | Node | Samples | GR active | SM active | SM occupancy | Tensor active | DRAM active | PCIe TX/RX mean (GB/s) | +|---|---|---:|---:|---:|---:|---:|---:|---:| +| decode_throughput_1k_to_1k_c32 | head | 584 | 0.971 | 0.525 | 0.141 | 0.059 | 0.417 | 5.205/5.201 | +| decode_throughput_1k_to_1k_c32 | worker | 576 | 0.979 | 0.523 | 0.141 | 0.058 | 0.415 | 5.236/5.232 | +| long_context_decode_128k_to_1k_c1 | head | 648 | 0.961 | 0.578 | 0.211 | 0.095 | 0.399 | 6.764/6.821 | +| long_context_decode_128k_to_1k_c1 | worker | 648 | 0.965 | 0.579 | 0.211 | 0.095 | 0.401 | 6.780/6.838 | +| long_output_decode_1k_to_4k_c16 | head | 1648 | 0.989 | 0.490 | 0.133 | 0.045 | 0.414 | 2.621/2.634 | +| long_output_decode_1k_to_4k_c16 | worker | 1640 | 0.990 | 0.489 | 0.133 | 0.045 | 0.414 | 2.619/2.633 | +| long_prefill_latency_128k_c1 | head | 400 | 0.930 | 0.683 | 0.251 | 0.136 | 0.374 | 10.537/10.582 | +| long_prefill_latency_128k_c1 | worker | 392 | 0.936 | 0.686 | 0.251 | 0.136 | 0.376 | 10.618/10.632 | +| mid_prefill_throughput_32k_c16 | head | 1344 | 0.980 | 0.713 | 0.278 | 0.127 | 0.440 | 12.664/12.720 | +| mid_prefill_throughput_32k_c16 | worker | 1344 | 0.984 | 0.715 | 0.278 | 0.127 | 0.441 | 12.699/12.755 | +| decode_control_1k_to_1k_c32 | head | 1152 | 0.989 | 0.539 | 0.146 | 0.061 | 0.427 | 5.451/5.456 | +| decode_control_1k_to_1k_c32 | worker | 1152 | 0.990 | 0.536 | 0.145 | 0.060 | 0.425 | 5.465/5.461 | +| decode_with_128k_prefill_1k_to_1k_c32 | head | 1520 | 0.977 | 0.581 | 0.174 | 0.080 | 0.416 | 6.865/6.868 | +| decode_with_128k_prefill_1k_to_1k_c32 | worker | 1520 | 0.978 | 0.578 | 0.174 | 0.080 | 0.416 | 6.865/6.872 | +| long_prefill_injection_128k_to_1_c1 | head | 368 | 0.997 | 0.723 | 0.264 | 0.138 | 0.407 | 11.151/11.198 | +| long_prefill_injection_128k_to_1_c1 | worker | 368 | 0.997 | 0.720 | 0.262 | 0.137 | 0.407 | 11.069/11.110 | + +## 5. CPU, process, and `perf` + +Data: `case_cpu_summary.csv`, `case_process_summary.csv`, `case_perf_summary.csv`; raw: `mpstat.log`, `pidstat.log`, `perf_stat.log`. + +| Case | Node | Samples CPU/process/perf | CPU active mean/p95 (%) | Hot cores max | Process CPU max (%) | Process wait max (%) | IPC | Context switches mean/interval | +|---|---|---|---:|---:|---:|---:|---:|---:| +| decode_throughput_1k_to_1k_c32 | head | 15/1140/84 | 10.13/10.42 | 12 | 1225.00 | 0.20 | 3.049 | 41240.93 | +| decode_throughput_1k_to_1k_c32 | worker | 15/1080/84 | 9.84/10.09 | 12 | 1210.20 | 0.20 | 3.068 | 39682.00 | +| long_context_decode_128k_to_1k_c1 | head | 17/1292/96 | 9.90/10.29 | 12 | 1222.40 | 0.20 | 2.887 | 44877.38 | +| long_context_decode_128k_to_1k_c1 | worker | 17/1224/96 | 9.65/10.09 | 12 | 1215.60 | 0.00 | 2.902 | 43686.69 | +| long_output_decode_1k_to_4k_c16 | head | 41/3116/246 | 10.36/10.52 | 12 | 1225.80 | 0.20 | 3.098 | 43749.98 | +| long_output_decode_1k_to_4k_c16 | worker | 41/2952/246 | 9.99/10.09 | 12 | 1211.40 | 0.20 | 3.117 | 40973.46 | +| long_prefill_latency_128k_c1 | head | 10/1129/60 | 10.00/10.27 | 12 | 1211.20 | 0.20 | 2.727 | 50995.80 | +| long_prefill_latency_128k_c1 | worker | 10/720/60 | 9.69/10.18 | 12 | 1210.80 | 0.20 | 2.749 | 48210.50 | +| mid_prefill_throughput_32k_c16 | head | 34/2584/204 | 9.99/10.27 | 12 | 1211.00 | 0.20 | 2.802 | 39421.29 | +| mid_prefill_throughput_32k_c16 | worker | 34/2448/204 | 9.84/10.11 | 12 | 1210.20 | 0.20 | 2.810 | 39129.38 | +| decode_control_1k_to_1k_c32 | head | 29/2204/174 | 10.26/10.44 | 12 | 1220.40 | 0.20 | 3.042 | 42398.83 | +| decode_control_1k_to_1k_c32 | worker | 29/2088/174 | 9.98/10.11 | 12 | 1210.40 | 0.20 | 3.072 | 40484.59 | +| decode_with_128k_prefill_1k_to_1k_c32 | head | 38/2888/228 | 10.34/11.37 | 13 | 1223.60 | 0.20 | 2.980 | 41223.05 | +| decode_with_128k_prefill_1k_to_1k_c32 | worker | 38/2736/228 | 9.87/10.09 | 12 | 1211.40 | 0.20 | 3.005 | 39834.92 | +| long_prefill_injection_128k_to_1_c1 | head | 10/749/54 | 10.21/10.63 | 13 | 1223.60 | 0.00 | 2.808 | 33399.67 | +| long_prefill_injection_128k_to_1_c1 | worker | 9/648/60 | 10.04/10.12 | 12 | 1211.40 | 0.20 | 2.830 | 33911.70 | + +## 6. NUMA memory placement + +Data/raw: `case_numa_summary.csv`, `head|worker/numa_samples.csv`. + +| Case | Node | Samples | Node 0 mean (MiB) | Node 1 mean (MiB) | Total mean (MiB) | Imbalance mean/max (%) | +|---|---|---:|---:|---:|---:|---:| +| decode_throughput_1k_to_1k_c32 | head | 13 | 17941.22 | 27565.28 | 45506.51 | 21.15/21.15 | +| decode_throughput_1k_to_1k_c32 | worker | 13 | 19029.49 | 24814.32 | 43843.80 | 13.19/13.19 | +| long_context_decode_128k_to_1k_c1 | head | 13 | 17946.50 | 27569.43 | 45515.94 | 21.14/21.15 | +| long_context_decode_128k_to_1k_c1 | worker | 14 | 19037.22 | 24816.07 | 43853.29 | 13.18/13.18 | +| long_output_decode_1k_to_4k_c16 | head | 35 | 17941.67 | 27565.20 | 45506.88 | 21.15/21.17 | +| long_output_decode_1k_to_4k_c16 | worker | 37 | 19029.40 | 24814.58 | 43843.96 | 13.19/13.20 | +| long_prefill_latency_128k_c1 | head | 8 | 17869.37 | 27486.32 | 45355.71 | 21.20/21.21 | +| long_prefill_latency_128k_c1 | worker | 8 | 18970.34 | 24731.71 | 43702.05 | 13.18/13.21 | +| mid_prefill_throughput_32k_c16 | head | 28 | 17940.27 | 27563.88 | 45504.17 | 21.15/21.15 | +| mid_prefill_throughput_32k_c16 | worker | 30 | 19032.02 | 24805.59 | 43837.58 | 13.17/13.22 | +| decode_control_1k_to_1k_c32 | head | 24 | 17946.96 | 27573.84 | 45520.81 | 21.15/21.21 | +| decode_control_1k_to_1k_c32 | worker | 26 | 19039.33 | 24818.28 | 43857.62 | 13.18/13.18 | +| decode_with_128k_prefill_1k_to_1k_c32 | head | 32 | 17955.73 | 27573.17 | 45528.91 | 21.12/21.13 | +| decode_with_128k_prefill_1k_to_1k_c32 | worker | 34 | 19042.79 | 24823.40 | 43866.18 | 13.18/13.19 | +| long_prefill_injection_128k_to_1_c1 | head | 8 | 17951.77 | 27569.78 | 45521.56 | 21.13/21.13 | +| long_prefill_injection_128k_to_1_c1 | worker | 8 | 19038.43 | 24819.53 | 43857.94 | 13.18/13.18 | + +## 7. Linux netdev and RDMA data path + +Netdev data: `case_netdev_summary.csv`; RDMA data: `case_rdma_summary.csv`; raw: `sar_net.log`, `rdma.csv`. + +### Linux interfaces + +| Case | Node | Interface | Samples | RX mean/max (Gbit/s) | TX mean/max (Gbit/s) | Util max (%) | RX/TX error max (/s) | +|---|---|---|---:|---:|---:|---:|---:| +| decode_throughput_1k_to_1k_c32 | head | eth0 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_throughput_1k_to_1k_c32 | head | eth3 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_throughput_1k_to_1k_c32 | worker | eth0 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_throughput_1k_to_1k_c32 | worker | eth3 | 30 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_context_decode_128k_to_1k_c1 | head | eth0 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_context_decode_128k_to_1k_c1 | head | eth3 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_context_decode_128k_to_1k_c1 | worker | eth0 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_context_decode_128k_to_1k_c1 | worker | eth3 | 34 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_output_decode_1k_to_4k_c16 | head | eth0 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_output_decode_1k_to_4k_c16 | head | eth3 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_output_decode_1k_to_4k_c16 | worker | eth0 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_output_decode_1k_to_4k_c16 | worker | eth3 | 82 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_prefill_latency_128k_c1 | head | eth0 | 20 | 0.000/0.001 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_prefill_latency_128k_c1 | head | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_prefill_latency_128k_c1 | worker | eth0 | 20 | 0.000/0.000 | 0.000/0.001 | 0.000 | 0.00/0.00 | +| long_prefill_latency_128k_c1 | worker | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| mid_prefill_throughput_32k_c16 | head | eth0 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| mid_prefill_throughput_32k_c16 | head | eth3 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| mid_prefill_throughput_32k_c16 | worker | eth0 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| mid_prefill_throughput_32k_c16 | worker | eth3 | 68 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_control_1k_to_1k_c32 | head | eth0 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_control_1k_to_1k_c32 | head | eth3 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_control_1k_to_1k_c32 | worker | eth0 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_control_1k_to_1k_c32 | worker | eth3 | 58 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_with_128k_prefill_1k_to_1k_c32 | head | eth0 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_with_128k_prefill_1k_to_1k_c32 | head | eth3 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_with_128k_prefill_1k_to_1k_c32 | worker | eth0 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| decode_with_128k_prefill_1k_to_1k_c32 | worker | eth3 | 76 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_prefill_injection_128k_to_1_c1 | head | eth0 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_prefill_injection_128k_to_1_c1 | head | eth3 | 20 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_prefill_injection_128k_to_1_c1 | worker | eth0 | 18 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | +| long_prefill_injection_128k_to_1_c1 | worker | eth3 | 18 | 0.000/0.000 | 0.000/0.000 | 0.000 | 0.00/0.00 | + +### RDMA HCAs + +| Case | Node | HCA | Samples | TX/RX (Gbit/s) | Wait delta | Discard/error delta | Retry exceeded delta | +|---|---|---|---:|---:|---:|---:|---:| +| decode_throughput_1k_to_1k_c32 | head | mlx5_0 | 66 | 36.69/36.69 | 0 | 0/0 | 0 | +| decode_throughput_1k_to_1k_c32 | head | mlx5_3 | 66 | 36.71/36.71 | 0 | 0/0 | 0 | +| decode_throughput_1k_to_1k_c32 | worker | mlx5_0 | 68 | 36.80/36.80 | 0 | 0/0 | 0 | +| decode_throughput_1k_to_1k_c32 | worker | mlx5_3 | 68 | 36.79/36.79 | 0 | 0/0 | 0 | +| long_context_decode_128k_to_1k_c1 | head | mlx5_0 | 75 | 43.83/43.83 | 0 | 0/0 | 0 | +| long_context_decode_128k_to_1k_c1 | head | mlx5_3 | 75 | 43.95/43.95 | 0 | 0/0 | 0 | +| long_context_decode_128k_to_1k_c1 | worker | mlx5_0 | 77 | 44.73/44.73 | 0 | 0/0 | 0 | +| long_context_decode_128k_to_1k_c1 | worker | mlx5_3 | 77 | 44.11/44.11 | 0 | 0/0 | 0 | +| long_output_decode_1k_to_4k_c16 | head | mlx5_0 | 188 | 18.91/18.91 | 0 | 0/0 | 0 | +| long_output_decode_1k_to_4k_c16 | head | mlx5_3 | 188 | 18.90/18.90 | 0 | 0/0 | 0 | +| long_output_decode_1k_to_4k_c16 | worker | mlx5_0 | 194 | 18.90/18.90 | 0 | 0/0 | 0 | +| long_output_decode_1k_to_4k_c16 | worker | mlx5_3 | 194 | 18.90/18.90 | 0 | 0/0 | 0 | +| long_prefill_latency_128k_c1 | head | mlx5_0 | 45 | 71.38/71.38 | 0 | 0/0 | 0 | +| long_prefill_latency_128k_c1 | head | mlx5_3 | 45 | 71.45/71.45 | 0 | 0/0 | 0 | +| long_prefill_latency_128k_c1 | worker | mlx5_0 | 47 | 70.54/70.54 | 0 | 0/0 | 0 | +| long_prefill_latency_128k_c1 | worker | mlx5_3 | 47 | 70.54/70.54 | 0 | 0/0 | 0 | +| mid_prefill_throughput_32k_c16 | head | mlx5_0 | 154 | 82.99/82.99 | 0 | 0/0 | 0 | +| mid_prefill_throughput_32k_c16 | head | mlx5_3 | 154 | 83.01/83.01 | 0 | 0/0 | 0 | +| mid_prefill_throughput_32k_c16 | worker | mlx5_0 | 158 | 83.44/83.44 | 0 | 0/0 | 0 | +| mid_prefill_throughput_32k_c16 | worker | mlx5_3 | 158 | 83.45/83.45 | 0 | 0/0 | 0 | +| decode_control_1k_to_1k_c32 | head | mlx5_0 | 131 | 38.01/38.01 | 0 | 0/0 | 0 | +| decode_control_1k_to_1k_c32 | head | mlx5_3 | 131 | 38.00/38.00 | 0 | 0/0 | 0 | +| decode_control_1k_to_1k_c32 | worker | mlx5_0 | 137 | 37.74/37.74 | 0 | 0/0 | 0 | +| decode_control_1k_to_1k_c32 | worker | mlx5_3 | 137 | 37.75/37.75 | 0 | 0/0 | 0 | +| decode_with_128k_prefill_1k_to_1k_c32 | head | mlx5_0 | 173 | 46.80/46.80 | 0 | 0/0 | 0 | +| decode_with_128k_prefill_1k_to_1k_c32 | head | mlx5_3 | 173 | 46.81/46.81 | 0 | 0/0 | 0 | +| decode_with_128k_prefill_1k_to_1k_c32 | worker | mlx5_0 | 179 | 46.67/46.67 | 0 | 0/0 | 0 | +| decode_with_128k_prefill_1k_to_1k_c32 | worker | mlx5_3 | 179 | 46.67/46.67 | 0 | 0/0 | 0 | +| long_prefill_injection_128k_to_1_c1 | head | mlx5_0 | 42 | 74.15/74.15 | 0 | 0/0 | 0 | +| long_prefill_injection_128k_to_1_c1 | head | mlx5_3 | 42 | 74.23/74.23 | 0 | 0/0 | 0 | +| long_prefill_injection_128k_to_1_c1 | worker | mlx5_0 | 43 | 74.55/74.54 | 0 | 0/0 | 0 | +| long_prefill_injection_128k_to_1_c1 | worker | mlx5_3 | 43 | 74.57/74.58 | 0 | 0/0 | 0 | + +## 8. PCIe P2P and NCCL communication baseline + +Data: `communication_aggregate.csv`; raw: `communication/*.log`. + +| Test | Scope | Path/CROSS_NIC | Samples/repetitions | Size (MiB) | Mean latency (ms) | Bandwidth / busbw (GB/s) | Minimum | Wrong values | +|---|---|---|---:|---:|---:|---:|---:|---:| +| all_reduce | head_8gpu | 2 | 3 | 1 | 1.18 | 1.64 | 1.16 | 0 | +| all_reduce | head_8gpu | 2 | 3 | 1024 | 47.26 | 39.76 | 39.67 | 0 | +| all_reduce | head_8gpu | 2 | 3 | 64 | 3.11 | 37.79 | 37.25 | 0 | +| all_reduce | two_node_16gpu | 0 | 3 | 1 | 1.17 | 1.73 | 1.32 | 0 | +| all_reduce | two_node_16gpu | 0 | 3 | 1024 | 51.17 | 39.35 | 39.30 | 0 | +| all_reduce | two_node_16gpu | 0 | 3 | 64 | 3.40 | 36.98 | 36.72 | 0 | +| all_reduce | two_node_16gpu | 1 | 3 | 1 | 1.30 | 1.65 | 1.06 | 0 | +| all_reduce | two_node_16gpu | 1 | 3 | 1024 | 50.73 | 39.68 | 39.59 | 0 | +| all_reduce | two_node_16gpu | 1 | 3 | 64 | 3.38 | 37.25 | 36.86 | 0 | +| all_reduce | two_node_16gpu | 2 | 3 | 1 | 1.14 | 1.76 | 1.41 | 0 | +| all_reduce | two_node_16gpu | 2 | 3 | 1024 | 50.93 | 39.53 | 39.34 | 0 | +| all_reduce | two_node_16gpu | 2 | 3 | 64 | 3.43 | 36.65 | 36.14 | 0 | +| all_reduce | worker_8gpu | 2 | 3 | 1 | 0.97 | 1.91 | 1.66 | 0 | +| all_reduce | worker_8gpu | 2 | 3 | 1024 | 47.27 | 39.75 | 39.56 | 0 | +| all_reduce | worker_8gpu | 2 | 3 | 64 | 3.07 | 38.22 | 37.44 | 0 | +| p2p_copy | head | cross_numa_sys | 32 | 256 | - | 52.40 | 52.18 | - | +| p2p_copy | head | same_pcie_switch | 24 | 256 | - | 53.61 | 53.35 | - | +| p2p_copy | worker | cross_numa_sys | 32 | 256 | - | 52.32 | 52.08 | - | +| p2p_copy | worker | same_pcie_switch | 24 | 256 | - | 53.50 | 53.22 | - | + +## 9. Machine-readable summaries + +- `gpu_summary.csv` +- `rdma_summary.csv` +- `bench_summary.csv` +- `case_windows.csv` +- `case_gpu_summary.csv` +- `case_gpu_node_summary.csv` +- `case_dcgm_summary.csv` +- `case_cpu_summary.csv` +- `case_process_summary.csv` +- `case_perf_summary.csv` +- `case_numa_summary.csv` +- `case_netdev_summary.csv` +- `case_rdma_summary.csv` +- `communication_summary.csv` +- `communication_aggregate.csv` +- `summary.json` + +Each conclusion must cite the corresponding table above and its raw file; missing samples are reported as `-`, never interpreted as zero. diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/head_nccl_transport.log b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/head_nccl_transport.log new file mode 100644 index 0000000..7cc4351 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/head_nccl_transport.log @@ -0,0 +1,1012 @@ +/usr/local/lib/python3.12/dist-packages/torchao/quantization/quant_api.py:1731: SyntaxWarning: invalid escape sequence '\.' + """Configuration class for applying different quantization configs to modules or parameters based on their fully qualified names (FQNs). +/sgl-workspace/sglang/python/sglang/launch_server.py:56: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint. + Example: sglang serve --model-path [options] + warnings.warn( +[2026-07-31 08:39:07] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:07] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:07] Breakable CUDA graph is incompatible with DeepSeek-V4 (heavy capture-pool memory pressure); disabling prefill CUDA graph. +[2026-07-31 08:39:08] Use dsv4 attention backend for DeepseekV4ForCausalLM, setting page_size to 256. +[2026-07-31 08:39:08] Setting swa_full_tokens_ratio to 0.1 for DeepseekV4ForCausalLM. +[2026-07-31 08:39:08] Setting KV cache dtype to fp8_e4m3 for DeepseekV4ForCausalLM. +[2026-07-31 08:39:08] Use flashinfer_mxfp4 as MoE runner backend on SM120 for DeepseekV4 +[2026-07-31 08:39:11] server_args=ServerArgs(model_path='/data/hf_models/DeepSeek-V4-Pro', tokenizer_path='/data/hf_models/DeepSeek-V4-Pro', tokenizer_mode='auto', tokenizer_backend='huggingface', tokenizer_worker_num=1, detokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', model_config_parser='auto', json_model_override_args='{}', dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='fp8_e4m3', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, enable_tf32_matmul=False, mem_fraction_static=0.9, max_running_requests=256, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=8192, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, disable_priority_preemption=False, default_priority_value=None, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, retraction_policy='length', schedule_conservativeness=1.0, page_size=256, swa_full_tokens_ratio=0.1, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', prefill_only_disable_kv_cache=False, disable_radix_cache=False, enable_page_major_kv_layout=False, enable_unified_memory=False, disable_chunked_prefix_cache=False, disable_overlap_schedule=False, num_continuous_decode_steps=1, scheduler_recv_interval=1, enable_mixed_chunk=False, nccl_port=None, dist_timeout=None, dist_init_addr='10.101.0.11:20002', nnodes=2, node_rank=0, tp_size=16, dcp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, enable_prefill_cp=False, cp_strategy=None, enable_dsa_cache_layer_split=False, enable_dsa_prefill_context_parallel=False, dsa_prefill_cp_mode='round-robin-split', enable_prefill_context_parallel=False, prefill_cp_mode='in-seq-split', enable_dp_attention=False, enable_dp_attention_local_control_broadcast=False, enable_dp_lm_head=False, enable_attn_tp_input_scattered=False, disable_attn_tp_gather=False, enable_p2p_check=False, device='cuda', base_gpu_id=0, gpu_id_step=1, random_seed=811545264, watchdog_timeout=300, soft_watchdog_timeout=None, sleep_on_idle=False, use_ray=False, custom_sigquit_handler=None, numa_node=None, gc_threshold=None, host='0.0.0.0', port=30002, fastapi_root_path='', smg_grpc_mode=False, grpc_mode=False, grpc_port=None, skip_server_warmup=False, warmups=None, enable_http2=False, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_keyfile_password=None, enable_ssl_refresh=False, api_key=None, admin_api_key=None, served_model_name='/data/hf_models/DeepSeek-V4-Pro', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, default_chat_template_kwargs=None, strip_thinking_cache=False, enable_strict_thinking=False, tool_call_parser=None, tool_server=None, sampling_defaults='model', asr_max_buffer_seconds=60, asr_max_concurrent_sessions=32, preferred_sampling_params=None, allow_auto_truncate=False, stream_interval=1, batch_notify_size=16, stream_response_default_include_usage=False, incremental_streaming_output=False, enable_streaming_session=False, enable_session_radix_cache=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, smg_http_sidecar_port=None, enable_mfu_metrics=False, enable_metrics_for_all_schedulers=False, load_snapshot_publish_interval=15, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_forward_pass_metrics=False, forward_pass_metrics_worker_id='', forward_pass_metrics_ipc_name=None, enable_trace=False, trace_modules='request', otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, stat_loggers=None, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, attention_backend='dsv4', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', radix_cache_backend=None, mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='auto', bf16_gemm_backend='auto', dsa_prefill_backend=None, dsa_decode_backend=None, dsa_paged_mqa_logits_backend='auto', dsa_topk_backend='sgl-kernel', disable_flashinfer_autotune=False, mamba_backend='triton', cuda_graph_config=CudaGraphConfig(decode=PhaseConfig(backend='full', max_bs=64, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], tc_compiler='eager', full_prefill_max_req=None), prefill=PhaseConfig(backend='disabled', max_bs=8192, bs=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192], tc_compiler='eager', full_prefill_max_req=None)), cuda_graph_backend_decode=None, cuda_graph_backend_prefill=None, cuda_graph_max_bs_decode=64, cuda_graph_max_bs_prefill=None, cuda_graph_bs_decode=None, cuda_graph_bs_prefill=None, cuda_graph_tc_compiler=None, disable_prefill_cuda_graph=False, disable_decode_cuda_graph=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, debug_cuda_graph=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, flashinfer_mla_disable_ragged=False, enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_fused_moe_sum_all_reduce=False, enable_deepseek_v4_fp4_indexer=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, pre_warm_nccl=False, enable_quant_communications=False, enable_flashinfer_allreduce_fusion=False, enforce_disable_flashinfer_allreduce_fusion=False, flashinfer_allreduce_fusion_backend=None, enable_aiter_allreduce_fusion=False, enable_torch_compile=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, torchao_config='', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_dflash_block_size=None, speculative_dspark_block_size=None, speculative_dspark_sps_table_path=None, speculative_dspark_confidence_sts_path=None, speculative_dspark_align_verify_tokens_to_graph_tier=False, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_use_rejection_sampling=False, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_draft_window_size=None, speculative_moe_runner_backend='flashinfer_mxfp4', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_skip_dp_mlp_sync=False, enable_multi_layer_eagle=False, speculative_adaptive=False, speculative_adaptive_config=None, decoupled_spec_bind_endpoint=None, decoupled_spec_connect_endpoints=None, decoupled_spec_rank=None, decoupled_spec_role='null', spec_trace_dir=None, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_max_trie_depth=18, speculative_ngram_capacity=10000000, speculative_ngram_external_corpus_path=None, speculative_ngram_external_sam_budget=0, speculative_ngram_external_corpus_max_tokens=10000000, ep_size=2, moe_a2a_backend='none', moe_runner_backend='flashinfer_mxfp4', flashinfer_mxfp4_moe_precision='default', deepep_mode='auto', fuseep_mode=2, deepep_dispatcher_output_dtype='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, enable_elastic_expert_backup=False, mooncake_ib_device=None, enable_waterfill=False, ep_join_mode=None, ep_join_rank_offset=0, elastic_ep_initial_size=None, max_ep_size=None, elastic_ep_scale_timeout=600, elastic_ep_rejoin=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, disable_shared_experts_fusion=False, enforce_shared_experts_fusion=False, max_mamba_cache_size=None, mamba_ssm_dtype=None, enable_mamba_cache_stochastic_rounding=False, mamba_cache_philox_rounds=0, mamba_full_memory_ratio=0.9, mamba_radix_cache_strategy='auto', uses_mamba_radix_cache=False, mamba_track_interval=256, enable_int8_mamba_checkpoint=False, int8_mamba_ckpt_size=None, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, enable_linear_replayssm=False, linear_replayssm_cache_len=16, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='page_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='timeout', hicache_storage_backend_extra_config=None, enable_hisparse=False, hisparse_config=None, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, enable_mm_global_cache=False, disable_fast_image_processor=False, mm_feature_transport='cpu', keep_mm_feature_on_device=False, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, experts_shared_outer_loras=None, lora_use_virtual_experts=False, lora_strict_loading=False, lora_drain_wait_threshold=0.0, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', enable_lmcache=False, lmcache_config_file=None, enable_flexkv=False, flexkv_config_file=None, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, dllm_fdfo=True, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_ib_device=None, disaggregation_decode_enable_radix_cache=False, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_extra_slots=None, disaggregation_decode_polling_interval=1, optimistic_prefill_attempts=0, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], encoder_bootstrap_port=8997, encoder_register_urls=[], enable_adaptive_dispatch_to_encoder=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, custom_weight_loader=[], weight_loader_disable_mmap=False, weight_loader_prefetch_checkpoints=False, weight_loader_prefetch_num_threads=4, weight_loader_drop_cache_after_load=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, engine_info_bootstrap_port=6789, modelexpress_config=None, download_dir=None, model_checksum=None, delete_ckpt_after_loading=False, decrypted_config_file=None, decrypted_draft_config_file=None, checkpoint_engine_wait_weights_before_ready=False, enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, prefill_delayer_queue_min_ratio=None, prefill_delayer_max_delay_ms=None, min_free_slots_delay=None, enable_deterministic_inference=False, rl_on_policy_target=None, kv_canary='none', kv_canary_real_data='none', kv_canary_sweep_interval=0, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, enable_custom_logit_processor=False, enable_return_hidden_states=False, enable_return_routed_experts=False, enable_return_indexer_topk=False, disable_outlines_disk_cache=False, enable_mis=False, forward_hooks=None, msprobe_dump_config=None) +[2026-07-31 08:39:11] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:11] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:11] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:11] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:11] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:11] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:11] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:11] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:11] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:11] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:13] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:13] No HuggingFace chat template found +[2026-07-31 08:39:13] No chat template found, defaulting to 'string' content format +[2026-07-31 08:39:25 TP3 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:25 TP3 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:25 TP5 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:25 TP5 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:25 TP7 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:25 TP7 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:25 TP2 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:25 TP2 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:25 TP1 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:25 TP1 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:26 TP6 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:26 TP6 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:26 TP0 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:26 TP0 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:26 TP4 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:26 TP4 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:27 TP3 EP0] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:27 TP3 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:27 TP3 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:27 TP5 EP0] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:27 TP7 EP0] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:27] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:27 TP3 EP0] Init torch distributed begin. +[2026-07-31 08:39:28 TP5 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:28 TP5 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:28 TP5 EP0] Init torch distributed begin. +[2026-07-31 08:39:28 TP7 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:28 TP7 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:28 TP2 EP0] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:28 TP1 EP0] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:28 TP6 EP0] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:28 TP7 EP0] Init torch distributed begin. +[2026-07-31 08:39:28 TP0 EP0] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:28 TP1 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:28 TP4 EP0] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:28 TP1 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:28 TP2 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:28 TP2 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:28 TP6 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:28 TP6 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:28 TP1 EP0] Init torch distributed begin. +[2026-07-31 08:39:28 TP0 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:28 TP0 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:28 TP2 EP0] Init torch distributed begin. +[2026-07-31 08:39:28 TP6 EP0] Init torch distributed begin. +[2026-07-31 08:39:28 TP0 EP0] Init torch distributed begin. +[2026-07-31 08:39:28 TP4 EP0] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:28 TP4 EP0] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:28 TP4 EP0] Init torch distributed begin. +[2026-07-31 08:39:36 TP0 EP0] sglang is using nccl==2.28.9 +gpu1:237:237 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu1:237:237 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:237:237 [0] NCCL INFO Bootstrap: Using eth0:10.101.0.11<0> +gpu1:237:237 [0] NCCL INFO cudaDriverVersion 13000 +gpu1:241:241 [4] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu1:241:241 [4] NCCL INFO cudaDriverVersion 13000 +gpu1:237:237 [0] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu1:241:241 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:241:241 [4] NCCL INFO Bootstrap: Using eth0:10.101.0.11<0> +gpu1:241:241 [4] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu1:238:238 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu1:238:238 [1] NCCL INFO cudaDriverVersion 13000 +gpu1:243:243 [6] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu1:243:243 [6] NCCL INFO cudaDriverVersion 13000 +gpu1:239:239 [2] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu1:242:242 [5] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu1:239:239 [2] NCCL INFO cudaDriverVersion 13000 +gpu1:242:242 [5] NCCL INFO cudaDriverVersion 13000 +gpu1:244:244 [7] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu1:244:244 [7] NCCL INFO cudaDriverVersion 13000 +gpu1:240:240 [3] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu1:240:240 [3] NCCL INFO cudaDriverVersion 13000 +gpu1:238:238 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:243:243 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:242:242 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:240:240 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:239:239 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:244:244 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:238:238 [1] NCCL INFO Bootstrap: Using eth0:10.101.0.11<0> +gpu1:243:243 [6] NCCL INFO Bootstrap: Using eth0:10.101.0.11<0> +gpu1:238:238 [1] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu1:243:243 [6] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu1:240:240 [3] NCCL INFO Bootstrap: Using eth0:10.101.0.11<0> +gpu1:242:242 [5] NCCL INFO Bootstrap: Using eth0:10.101.0.11<0> +gpu1:240:240 [3] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu1:242:242 [5] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu1:244:244 [7] NCCL INFO Bootstrap: Using eth0:10.101.0.11<0> +gpu1:239:239 [2] NCCL INFO Bootstrap: Using eth0:10.101.0.11<0> +gpu1:244:244 [7] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu1:239:239 [2] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu1:241:241 [4] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu1:241:241 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:241:241 [4] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu1:243:243 [6] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu1:243:243 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:243:243 [6] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu1:238:238 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu1:238:238 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:238:238 [1] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu1:241:241 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.11<0> +gpu1:241:241 [4] NCCL INFO Initialized NET plugin IB +gpu1:241:241 [4] NCCL INFO Assigned NET plugin IB to comm +gpu1:241:241 [4] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu1:241:241 [4] NCCL INFO Using network IB +gpu1:243:243 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.11<0> +gpu1:243:243 [6] NCCL INFO Initialized NET plugin IB +gpu1:243:243 [6] NCCL INFO Assigned NET plugin IB to comm +gpu1:243:243 [6] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu1:243:243 [6] NCCL INFO Using network IB +gpu1:238:238 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.11<0> +gpu1:238:238 [1] NCCL INFO Initialized NET plugin IB +gpu1:238:238 [1] NCCL INFO Assigned NET plugin IB to comm +gpu1:238:238 [1] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu1:238:238 [1] NCCL INFO Using network IB +gpu1:243:243 [6] NCCL INFO DMA-BUF is available on GPU device 6 +gpu1:243:243 [6] NCCL INFO ncclCommInitRank comm 0x555ca4d0 rank 6 nranks 16 cudaDev 6 nvmlDev 6 busId ae000 commId 0x98560d124efa1ab - Init START +gpu1:241:241 [4] NCCL INFO DMA-BUF is available on GPU device 4 +gpu1:241:241 [4] NCCL INFO ncclCommInitRank comm 0x31ec1b40 rank 4 nranks 16 cudaDev 4 nvmlDev 4 busId aa000 commId 0x98560d124efa1ab - Init START +gpu1:238:238 [1] NCCL INFO DMA-BUF is available on GPU device 1 +gpu1:238:238 [1] NCCL INFO ncclCommInitRank comm 0x218e7450 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 1a000 commId 0x98560d124efa1ab - Init START +gpu1:244:244 [7] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu1:244:244 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:244:244 [7] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu1:239:239 [2] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu1:239:239 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:239:239 [2] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu1:244:244 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.11<0> +gpu1:244:244 [7] NCCL INFO Initialized NET plugin IB +gpu1:244:244 [7] NCCL INFO Assigned NET plugin IB to comm +gpu1:244:244 [7] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu1:244:244 [7] NCCL INFO Using network IB +gpu1:244:244 [7] NCCL INFO DMA-BUF is available on GPU device 7 +gpu1:244:244 [7] NCCL INFO ncclCommInitRank comm 0x1aa961b0 rank 7 nranks 16 cudaDev 7 nvmlDev 7 busId af000 commId 0x98560d124efa1ab - Init START +gpu1:239:239 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.11<0> +gpu1:239:239 [2] NCCL INFO Initialized NET plugin IB +gpu1:244:244 [7] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu1:239:239 [2] NCCL INFO Assigned NET plugin IB to comm +gpu1:239:239 [2] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu1:239:239 [2] NCCL INFO Using network IB +gpu1:239:239 [2] NCCL INFO DMA-BUF is available on GPU device 2 +gpu1:239:239 [2] NCCL INFO ncclCommInitRank comm 0x45c147b0 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 1b000 commId 0x98560d124efa1ab - Init START +gpu1:237:237 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu1:237:237 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:237:237 [0] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu1:237:237 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.11<0> +gpu1:237:237 [0] NCCL INFO Initialized NET plugin IB +gpu1:237:237 [0] NCCL INFO Assigned NET plugin IB to comm +gpu1:237:237 [0] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu1:237:237 [0] NCCL INFO Using network IB +gpu1:237:237 [0] NCCL INFO DMA-BUF is available on GPU device 0 +gpu1:237:237 [0] NCCL INFO ncclCommInitRank comm 0x435cc030 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 19000 commId 0x98560d124efa1ab - Init START +gpu1:237:237 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu1:238:238 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu1:240:240 [3] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu1:240:240 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:240:240 [3] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu1:240:240 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.11<0> +gpu1:240:240 [3] NCCL INFO Initialized NET plugin IB +gpu1:240:240 [3] NCCL INFO Assigned NET plugin IB to comm +gpu1:240:240 [3] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu1:240:240 [3] NCCL INFO Using network IB +gpu1:242:242 [5] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu1:242:242 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu1:242:242 [5] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu1:240:240 [3] NCCL INFO DMA-BUF is available on GPU device 3 +gpu1:240:240 [3] NCCL INFO ncclCommInitRank comm 0x4adc8330 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId 1d000 commId 0x98560d124efa1ab - Init START +gpu1:239:239 [2] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu1:240:240 [3] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu1:242:242 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.11<0> +gpu1:242:242 [5] NCCL INFO Initialized NET plugin IB +gpu1:242:242 [5] NCCL INFO Assigned NET plugin IB to comm +gpu1:242:242 [5] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu1:242:242 [5] NCCL INFO Using network IB +gpu1:242:242 [5] NCCL INFO DMA-BUF is available on GPU device 5 +gpu1:242:242 [5] NCCL INFO ncclCommInitRank comm 0x1c3e90a0 rank 5 nranks 16 cudaDev 5 nvmlDev 5 busId ab000 commId 0x98560d124efa1ab - Init START +gpu1:241:241 [4] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu1:243:243 [6] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu1:242:242 [5] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu1:237:237 [0] NCCL INFO Bootstrap timings total 0.059976 (create 0.000028, send 0.000112, recv 0.000258, ring 0.059225, delay 0.000001) +gpu1:239:239 [2] NCCL INFO Bootstrap timings total 0.091019 (create 0.000029, send 0.000106, recv 0.055053, ring 0.035545, delay 0.000001) +gpu1:237:237 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu1:239:239 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu1:244:244 [7] NCCL INFO Bootstrap timings total 0.095930 (create 0.000028, send 0.000107, recv 0.000303, ring 0.095072, delay 0.000001) +gpu1:240:240 [3] NCCL INFO Bootstrap timings total 0.036125 (create 0.000027, send 0.000088, recv 0.000240, ring 0.035501, delay 0.000001) +gpu1:243:243 [6] NCCL INFO Bootstrap timings total 0.128327 (create 0.000042, send 0.000176, recv 0.032483, ring 0.002108, delay 0.000002) +gpu1:241:241 [4] NCCL INFO Bootstrap timings total 0.125690 (create 0.000105, send 0.000239, recv 0.122766, ring 0.002209, delay 0.000003) +gpu1:244:244 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu1:243:243 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu1:240:240 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu1:242:242 [5] NCCL INFO Bootstrap timings total 0.002941 (create 0.000044, send 0.000202, recv 0.000201, ring 0.002052, delay 0.000002) +gpu1:241:241 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu1:238:238 [1] NCCL INFO Bootstrap timings total 0.124940 (create 0.000034, send 0.000113, recv 0.034027, ring 0.059048, delay 0.000001) +gpu1:242:242 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu1:238:238 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu1:237:237 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu1:237:237 [0] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu1:237:237 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu1:239:239 [2] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 2 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu1:239:239 [2] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu1:239:239 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu1:240:240 [3] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 3 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu1:240:240 [3] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu1:240:240 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu1:244:244 [7] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 7 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu1:244:244 [7] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu1:244:244 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu1:243:243 [6] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 6 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu1:243:243 [6] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu1:243:243 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu1:238:238 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu1:242:242 [5] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 5 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu1:242:242 [5] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu1:242:242 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu1:238:238 [1] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu1:238:238 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu1:241:241 [4] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 4 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu1:241:241 [4] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu1:241:241 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu1:240:240 [3] NCCL INFO comm 0x4adc8330 rank 3 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 +gpu1:242:242 [5] NCCL INFO comm 0x1c3e90a0 rank 5 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 +gpu1:239:239 [2] NCCL INFO comm 0x45c147b0 rank 2 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 +gpu1:241:241 [4] NCCL INFO comm 0x31ec1b40 rank 4 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 +gpu1:237:237 [0] NCCL INFO comm 0x435cc030 rank 0 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 +gpu1:238:238 [1] NCCL INFO comm 0x218e7450 rank 1 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 +gpu1:242:242 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->7 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->7 +gpu1:242:242 [5] NCCL INFO P2P Chunksize set to 131072 +gpu1:240:240 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] 1/11/-1->3->-1 [2] -1/-1/-1->3->2 [3] 1/-1/-1->3->7 +gpu1:237:237 [0] NCCL INFO Channel 00/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +gpu1:237:237 [0] NCCL INFO Channel 01/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +gpu1:243:243 [6] NCCL INFO comm 0x555ca4d0 rank 6 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 +gpu1:237:237 [0] NCCL INFO Channel 02/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +gpu1:244:244 [7] NCCL INFO comm 0x1aa961b0 rank 7 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 +gpu1:237:237 [0] NCCL INFO Channel 03/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +gpu1:240:240 [3] NCCL INFO P2P Chunksize set to 131072 +gpu1:237:237 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] -1/-1/-1->0->2 [2] 1/-1/-1->0->4 [3] -1/-1/-1->0->2 +gpu1:241:241 [4] NCCL INFO Trees [0] 5/-1/-1->4->8 [1] -1/-1/-1->4->6 [2] 5/8/0->4->12 [3] -1/-1/-1->4->6 +gpu1:237:237 [0] NCCL INFO P2P Chunksize set to 131072 +gpu1:243:243 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 4/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 4/-1/-1->6->5 +gpu1:239:239 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 0/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 0/-1/-1->2->1 +gpu1:244:244 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] 5/-1/-1->7->11 [2] -1/-1/-1->7->6 [3] 5/11/3->7->15 +gpu1:238:238 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->3 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->3 +gpu1:243:243 [6] NCCL INFO P2P Chunksize set to 131072 +gpu1:241:241 [4] NCCL INFO P2P Chunksize set to 131072 +gpu1:244:244 [7] NCCL INFO P2P Chunksize set to 131072 +gpu1:239:239 [2] NCCL INFO P2P Chunksize set to 131072 +gpu1:238:238 [1] NCCL INFO P2P Chunksize set to 131072 +gpu1:243:243 [6] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu1:244:244 [7] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu1:243:243 [6] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:244:244 [7] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:242:242 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu1:242:242 [5] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:237:237 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu1:237:237 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:240:240 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu1:240:240 [3] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:241:241 [4] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu1:241:241 [4] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:239:239 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu1:239:239 [2] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:244:1586 [0] NCCL INFO [Proxy Service] Device 7 CPU core 97 +gpu1:237:1590 [0] NCCL INFO [Proxy Service] Device 0 CPU core 5 +gpu1:238:238 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu1:244:1588 [0] NCCL INFO [Proxy Service UDS] Device 7 CPU core 99 +gpu1:238:238 [1] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:243:1585 [0] NCCL INFO [Proxy Service] Device 6 CPU core 62 +gpu1:237:1592 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 6 +gpu1:243:1589 [0] NCCL INFO [Proxy Service UDS] Device 6 CPU core 118 +gpu1:242:1587 [0] NCCL INFO [Proxy Service] Device 5 CPU core 100 +gpu1:242:1591 [0] NCCL INFO [Proxy Service UDS] Device 5 CPU core 63 +gpu1:241:1593 [0] NCCL INFO [Proxy Service] Device 4 CPU core 112 +gpu1:240:1594 [0] NCCL INFO [Proxy Service] Device 3 CPU core 21 +gpu1:241:1597 [0] NCCL INFO [Proxy Service UDS] Device 4 CPU core 107 +gpu1:239:1595 [0] NCCL INFO [Proxy Service] Device 2 CPU core 85 +gpu1:240:1598 [0] NCCL INFO [Proxy Service UDS] Device 3 CPU core 22 +gpu1:238:1596 [0] NCCL INFO [Proxy Service] Device 1 CPU core 86 +gpu1:239:1599 [0] NCCL INFO [Proxy Service UDS] Device 2 CPU core 87 +gpu1:238:1600 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 24 +gpu1:243:243 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 00/0 : 7[7] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:1601 [0] NCCL INFO [Proxy Progress] Device 7 CPU core 58 +gpu1:244:244 [7] NCCL INFO Channel 01/0 : 7[7] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 02/0 : 7[7] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:241:1602 [0] NCCL INFO [Proxy Progress] Device 4 CPU core 101 +gpu1:241:241 [4] NCCL INFO Channel 00/0 : 3[3] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 01/0 : 3[3] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 3[3] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 03/0 : 3[3] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC +gpu1:237:1603 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 71 +gpu1:237:237 [0] NCCL INFO Channel 00/0 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 01/0 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 02/0 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 03/0 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC +gpu1:240:1604 [0] NCCL INFO [Proxy Progress] Device 3 CPU core 90 +gpu1:240:240 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:242:242 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:241:241 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:240:240 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:244:244 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:243:243 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:241:241 [4] NCCL INFO Channel 01/0 : 4[4] -> 6[6] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:242:242 [5] NCCL INFO Channel 01/0 : 5[5] -> 7[7] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:237:237 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:237:237 [0] NCCL INFO Channel 01/0 : 0[0] -> 2[2] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 01/0 : 1[1] -> 3[3] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 03/0 : 4[4] -> 6[6] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 03/0 : 5[5] -> 7[7] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 01/0 : 6[6] -> 4[4] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 01/0 : 2[2] -> 0[0] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Channel 02/0 : 0[0] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 3[3] -> 7[7] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 0[0] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 01/0 : 7[7] -> 11[3] [send] via NET/IB/1/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 03/0 : 3[3] -> 7[7] [send] via NET/IB/0/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 00/0 : 4[4] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 11[3] [send] via NET/IB/1/GDRDMA +gpu1:243:243 [6] NCCL INFO Channel 03/0 : 6[6] -> 4[4] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 4[4] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 01/0 : 11[3] -> 3[3] [receive] via NET/IB/0/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 15[7] -> 7[7] [receive] via NET/IB/1/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 15[7] [send] via NET/IB/1/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 01/0 : 3[3] -> 11[3] [send] via NET/IB/0/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 12[4] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 01/0 : 11[3] -> 7[7] [receive] via NET/IB/1/GDRDMA +gpu1:239:239 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 11[3] -> 7[7] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 4[4] -> 12[4] [send] via NET/IB/1/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 03/0 : 7[7] -> 3[3] [receive] via NET/IB/0/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 3[3] [send] via NET/IB/1/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 02/0 : 4[4] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 01/0 : 3[3] -> 1[1] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 00/0 : 8[0] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 8[0] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 01/0 : 7[7] -> 5[5] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 4[4] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 5[5] via P2P/IPC +gpu1:240:240 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC +gpu1:240:240 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Connected all trees +gpu1:238:238 [1] NCCL INFO Connected all trees +gpu1:240:240 [3] NCCL INFO Connected all trees +gpu1:239:239 [2] NCCL INFO Connected all trees +gpu1:242:242 [5] NCCL INFO Connected all trees +gpu1:241:241 [4] NCCL INFO Connected all trees +gpu1:244:244 [7] NCCL INFO Connected all trees +gpu1:243:243 [6] NCCL INFO Connected all trees +gpu1:237:237 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu1:237:237 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:237:237 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:237:237 [0] NCCL INFO CC Off, workFifoBytes 1048576 +gpu1:244:244 [7] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu1:244:244 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:244:244 [7] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:241:241 [4] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu1:241:241 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:241:241 [4] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:240:240 [3] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu1:240:240 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:240:240 [3] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:243:1605 [0] NCCL INFO [Proxy Progress] Device 6 CPU core 39 +gpu1:242:1606 [0] NCCL INFO [Proxy Progress] Device 5 CPU core 40 +gpu1:243:243 [6] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu1:243:243 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:243:243 [6] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:242:242 [5] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu1:242:242 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:242:242 [5] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:243:243 [6] NCCL INFO ncclCommInitRank comm 0x555ca4d0 rank 6 nranks 16 cudaDev 6 nvmlDev 6 busId ae000 commId 0x98560d124efa1ab - Init COMPLETE +gpu1:243:243 [6] NCCL INFO Init timings - ncclCommInitRank: rank 6 nranks 16 total 0.71 (kernels 0.30, alloc 0.03, bootstrap 0.13, allgathers 0.01, topo 0.04, graphs 0.00, connections 0.21, rest 0.00) +gpu1:242:242 [5] NCCL INFO ncclCommInitRank comm 0x1c3e90a0 rank 5 nranks 16 cudaDev 5 nvmlDev 5 busId ab000 commId 0x98560d124efa1ab - Init COMPLETE +gpu1:242:242 [5] NCCL INFO Init timings - ncclCommInitRank: rank 5 nranks 16 total 0.71 (kernels 0.41, alloc 0.04, bootstrap 0.00, allgathers 0.00, topo 0.04, graphs 0.00, connections 0.21, rest 0.00) +gpu1:244:244 [7] NCCL INFO ncclCommInitRank comm 0x1aa961b0 rank 7 nranks 16 cudaDev 7 nvmlDev 7 busId af000 commId 0x98560d124efa1ab - Init COMPLETE +gpu1:244:244 [7] NCCL INFO Init timings - ncclCommInitRank: rank 7 nranks 16 total 0.71 (kernels 0.34, alloc 0.01, bootstrap 0.10, allgathers 0.01, topo 0.04, graphs 0.00, connections 0.16, rest 0.05) +gpu1:238:1607 [0] NCCL INFO [Proxy Progress] Device 1 CPU core 92 +gpu1:239:1608 [0] NCCL INFO [Proxy Progress] Device 2 CPU core 93 +gpu1:241:241 [4] NCCL INFO ncclCommInitRank comm 0x31ec1b40 rank 4 nranks 16 cudaDev 4 nvmlDev 4 busId aa000 commId 0x98560d124efa1ab - Init COMPLETE +gpu1:241:241 [4] NCCL INFO Init timings - ncclCommInitRank: rank 4 nranks 16 total 0.75 (kernels 0.32, alloc 0.05, bootstrap 0.13, allgathers 0.00, topo 0.04, graphs 0.01, connections 0.16, rest 0.05) +gpu1:238:238 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu1:238:238 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:238:238 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:239:239 [2] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu1:239:239 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:239:239 [2] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:238:238 [1] NCCL INFO ncclCommInitRank comm 0x218e7450 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 1a000 commId 0x98560d124efa1ab - Init COMPLETE +gpu1:240:240 [3] NCCL INFO ncclCommInitRank comm 0x4adc8330 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId 1d000 commId 0x98560d124efa1ab - Init COMPLETE +gpu1:238:238 [1] NCCL INFO Init timings - ncclCommInitRank: rank 1 nranks 16 total 0.72 (kernels 0.31, alloc 0.02, bootstrap 0.12, allgathers 0.00, topo 0.04, graphs 0.01, connections 0.21, rest 0.00) +gpu1:237:237 [0] NCCL INFO ncclCommInitRank comm 0x435cc030 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 19000 commId 0x98560d124efa1ab - Init COMPLETE +gpu1:240:240 [3] NCCL INFO Init timings - ncclCommInitRank: rank 3 nranks 16 total 0.71 (kernels 0.40, alloc 0.01, bootstrap 0.04, allgathers 0.01, topo 0.04, graphs 0.01, connections 0.16, rest 0.05) +gpu1:239:239 [2] NCCL INFO ncclCommInitRank comm 0x45c147b0 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 1b000 commId 0x98560d124efa1ab - Init COMPLETE +gpu1:237:237 [0] NCCL INFO Init timings - ncclCommInitRank: rank 0 nranks 16 total 0.75 (kernels 0.42, alloc 0.01, bootstrap 0.06, allgathers 0.01, topo 0.03, graphs 0.00, connections 0.16, rest 0.05) +gpu1:239:239 [2] NCCL INFO Init timings - ncclCommInitRank: rank 2 nranks 16 total 0.71 (kernels 0.35, alloc 0.01, bootstrap 0.09, allgathers 0.01, topo 0.03, graphs 0.01, connections 0.21, rest 0.00) +[2026-07-31 08:39:37 TP0 EP0] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP1 EP0] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP5 EP0] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP2 EP0] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP4 EP0] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP6 EP0] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP3 EP0] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP7 EP0] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP0 EP0] Init torch distributed ends. elapsed=8.85 s, mem usage=0.42 GB +[2026-07-31 08:39:37 TP7 EP0] Init torch distributed ends. elapsed=9.31 s, mem usage=0.48 GB +[2026-07-31 08:39:37 TP6 EP0] Init torch distributed ends. elapsed=8.93 s, mem usage=0.39 GB +[2026-07-31 08:39:37 TP5 EP0] Init torch distributed ends. elapsed=9.43 s, mem usage=0.41 GB +[2026-07-31 08:39:37 TP4 EP0] Init torch distributed ends. elapsed=8.69 s, mem usage=0.46 GB +[2026-07-31 08:39:37 TP2 EP0] Init torch distributed ends. elapsed=8.95 s, mem usage=0.39 GB +[2026-07-31 08:39:37 TP1 EP0] Init torch distributed ends. elapsed=8.99 s, mem usage=0.41 GB +[2026-07-31 08:39:37 TP3 EP0] Init torch distributed ends. elapsed=9.74 s, mem usage=0.44 GB +[2026-07-31 08:39:40 TP6 EP0] Load weight begin. avail mem=82.11 GB +[2026-07-31 08:39:40 TP0 EP0] Load weight begin. avail mem=82.08 GB +[2026-07-31 08:39:40 TP0 EP0] Detected fp8 checkpoint. +[2026-07-31 08:39:40 TP0 EP0] Config does not support fused shared expert(s). Shared experts fusion optimization is disabled. +[2026-07-31 08:39:40 TP4 EP0] Load weight begin. avail mem=82.04 GB +[2026-07-31 08:39:40 TP2 EP0] Load weight begin. avail mem=82.11 GB +[2026-07-31 08:39:40 TP1 EP0] Load weight begin. avail mem=82.10 GB +[2026-07-31 08:39:40 TP5 EP0] Load weight begin. avail mem=82.10 GB +[2026-07-31 08:39:40 TP7 EP0] Load weight begin. avail mem=82.03 GB +[2026-07-31 08:39:40 TP3 EP0] Load weight begin. avail mem=82.07 GB +[2026-07-31 08:39:40 TP6 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP0 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP4 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP2 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP1 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP5 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP7 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP3 EP0] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:46 TP0 EP0] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP2 EP0] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP1 EP0] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP3 EP0] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP4 EP0] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP5 EP0] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP6 EP0] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP7 EP0] multimem all-gather disabled because the TP group spans across nodes. + Multi-thread loading shards: 0% Completed | 0/64 [00:003->2 [1] 1/11/-1->3->-1 [2] -1/-1/-1->3->2 [3] 1/-1/-1->3->7 +gpu1:240:240 [3] NCCL INFO P2P Chunksize set to 131072 +gpu1:243:243 [6] NCCL INFO comm 0x7ca785a0 rank 6 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 +gpu1:240:240 [3] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:241:241 [4] NCCL INFO comm 0x6816cee0 rank 4 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 +gpu1:238:238 [1] NCCL INFO comm 0x45c5a0e0 rank 1 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 +gpu1:239:239 [2] NCCL INFO comm 0x6be91a90 rank 2 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 +gpu1:242:242 [5] NCCL INFO comm 0x41815780 rank 5 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 +gpu1:243:243 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 4/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 4/-1/-1->6->5 +gpu1:243:243 [6] NCCL INFO P2P Chunksize set to 131072 +gpu1:237:237 [0] NCCL INFO comm 0x73224340 rank 0 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 +gpu1:243:243 [6] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:238:238 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->3 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->3 +gpu1:239:239 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 0/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 0/-1/-1->2->1 +gpu1:242:242 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->7 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->7 +gpu1:244:244 [7] NCCL INFO comm 0x390462b0 rank 7 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 +gpu1:238:238 [1] NCCL INFO P2P Chunksize set to 131072 +gpu1:242:242 [5] NCCL INFO P2P Chunksize set to 131072 +gpu1:239:239 [2] NCCL INFO P2P Chunksize set to 131072 +gpu1:241:241 [4] NCCL INFO Trees [0] 5/-1/-1->4->8 [1] -1/-1/-1->4->6 [2] 5/8/0->4->12 [3] -1/-1/-1->4->6 +gpu1:237:237 [0] NCCL INFO Channel 00/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +gpu1:238:238 [1] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:241:241 [4] NCCL INFO P2P Chunksize set to 131072 +gpu1:237:237 [0] NCCL INFO Channel 01/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +gpu1:242:242 [5] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:239:239 [2] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:237:237 [0] NCCL INFO Channel 02/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +gpu1:241:241 [4] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:237:237 [0] NCCL INFO Channel 03/04 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 +gpu1:244:244 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] 5/-1/-1->7->11 [2] -1/-1/-1->7->6 [3] 5/11/3->7->15 +gpu1:244:244 [7] NCCL INFO P2P Chunksize set to 131072 +gpu1:237:237 [0] NCCL INFO Trees [0] 1/8/-1->0->-1 [1] -1/-1/-1->0->2 [2] 1/-1/-1->0->4 [3] -1/-1/-1->0->2 +gpu1:237:237 [0] NCCL INFO P2P Chunksize set to 131072 +gpu1:244:244 [7] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:237:237 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu1:243:2378 [0] NCCL INFO [Proxy Service] Device 6 CPU core 109 +gpu1:243:2379 [0] NCCL INFO [Proxy Service UDS] Device 6 CPU core 46 +gpu1:238:2380 [0] NCCL INFO [Proxy Service] Device 1 CPU core 66 +gpu1:242:2384 [0] NCCL INFO [Proxy Service] Device 5 CPU core 35 +gpu1:237:2381 [0] NCCL INFO [Proxy Service] Device 0 CPU core 25 +gpu1:240:2386 [0] NCCL INFO [Proxy Service] Device 3 CPU core 20 +gpu1:241:2382 [0] NCCL INFO [Proxy Service] Device 4 CPU core 124 +gpu1:239:2383 [0] NCCL INFO [Proxy Service] Device 2 CPU core 3 +gpu1:238:2385 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 4 +gpu1:237:2388 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 26 +gpu1:240:2389 [0] NCCL INFO [Proxy Service UDS] Device 3 CPU core 91 +gpu1:239:2387 [0] NCCL INFO [Proxy Service UDS] Device 2 CPU core 5 +gpu1:241:2390 [0] NCCL INFO [Proxy Service UDS] Device 4 CPU core 61 +gpu1:242:2391 [0] NCCL INFO [Proxy Service UDS] Device 5 CPU core 36 +gpu1:244:2392 [0] NCCL INFO [Proxy Service] Device 7 CPU core 101 +gpu1:244:2393 [0] NCCL INFO [Proxy Service UDS] Device 7 CPU core 102 +gpu1:238:238 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 00/0 : 6[6] -> 7[7] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 00/0 : 5[5] -> 6[6] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 01/0 : 6[6] -> 7[7] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 01/0 : 5[5] -> 6[6] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 02/0 : 6[6] -> 7[7] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 02/0 : 5[5] -> 6[6] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 03/0 : 6[6] -> 7[7] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 03/0 : 5[5] -> 6[6] via P2P/IPC +gpu1:237:2394 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 28 +gpu1:237:237 [0] NCCL INFO Channel 00/0 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 01/0 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 02/0 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 03/0 : 15[7] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC +gpu1:241:2395 [0] NCCL INFO [Proxy Progress] Device 4 CPU core 62 +gpu1:241:241 [4] NCCL INFO Channel 00/0 : 3[3] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 01/0 : 3[3] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 3[3] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 03/0 : 3[3] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 00/0 : 4[4] -> 5[5] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 01/0 : 4[4] -> 5[5] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 4[4] -> 5[5] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 03/0 : 4[4] -> 5[5] via P2P/IPC +gpu1:240:2396 [0] NCCL INFO [Proxy Progress] Device 3 CPU core 85 +gpu1:240:240 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:244:2397 [0] NCCL INFO [Proxy Progress] Device 7 CPU core 39 +gpu1:244:244 [7] NCCL INFO Channel 00/0 : 7[7] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 01/0 : 7[7] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 02/0 : 7[7] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:240:240 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:238:238 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:239:239 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:237:237 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:237:237 [0] NCCL INFO Channel 01/0 : 0[0] -> 2[2] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 01/0 : 1[1] -> 3[3] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:242:242 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:244:244 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:243:243 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu1:241:241 [4] NCCL INFO Channel 01/0 : 4[4] -> 6[6] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 01/0 : 5[5] -> 7[7] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Channel 03/0 : 0[0] -> 2[2] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 03/0 : 4[4] -> 6[6] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 03/0 : 5[5] -> 7[7] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 03/0 : 1[1] -> 3[3] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 01/0 : 6[6] -> 4[4] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 01/0 : 2[2] -> 0[0] via P2P/IPC +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 0[0] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 02/0 : 0[0] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 03/0 : 3[3] -> 7[7] [send] via NET/IB/0/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 3[3] -> 7[7] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 00/0 : 4[4] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 01/0 : 7[7] -> 11[3] [send] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 4[4] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 11[3] [send] via NET/IB/1/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [send] via NET/IB/0/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 12[4] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 01/0 : 11[3] -> 3[3] [receive] via NET/IB/0/GDRDMA +gpu1:243:243 [6] NCCL INFO Channel 03/0 : 6[6] -> 4[4] via P2P/IPC +gpu1:240:240 [3] NCCL INFO Channel 01/0 : 3[3] -> 11[3] [send] via NET/IB/0/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 4[4] -> 12[4] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 15[7] -> 7[7] [receive] via NET/IB/1/GDRDMA +gpu1:239:239 [2] NCCL INFO Channel 03/0 : 2[2] -> 0[0] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 15[7] [send] via NET/IB/1/GDRDMA +gpu1:237:237 [0] NCCL INFO Channel 02/0 : 4[4] -> 0[0] [receive] via NET/IB/0/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 03/0 : 7[7] -> 3[3] [receive] via NET/IB/0/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 00/0 : 8[0] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 01/0 : 11[3] -> 7[7] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 8[0] -> 4[4] [receive] via NET/IB/1/GDRDMA +gpu1:241:241 [4] NCCL INFO Channel 02/0 : 4[4] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 11[3] -> 7[7] [receive] via NET/IB/1/GDRDMA +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 3[3] [send] via NET/IB/1/GDRDMA +gpu1:240:240 [3] NCCL INFO Channel 01/0 : 3[3] -> 1[1] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 01/0 : 7[7] -> 5[5] via P2P/IPC +gpu1:240:240 [3] NCCL INFO Channel 03/0 : 3[3] -> 1[1] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 03/0 : 7[7] -> 5[5] via P2P/IPC +gpu1:240:240 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 00/0 : 7[7] -> 6[6] via P2P/IPC +gpu1:240:240 [3] NCCL INFO Channel 02/0 : 3[3] -> 2[2] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC +gpu1:244:244 [7] NCCL INFO Channel 02/0 : 7[7] -> 6[6] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 00/0 : 6[6] -> 5[5] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 01/0 : 6[6] -> 5[5] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 02/0 : 2[2] -> 1[1] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 02/0 : 6[6] -> 5[5] via P2P/IPC +gpu1:239:239 [2] NCCL INFO Channel 03/0 : 2[2] -> 1[1] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC +gpu1:243:243 [6] NCCL INFO Channel 03/0 : 6[6] -> 5[5] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 00/0 : 5[5] -> 4[4] via P2P/IPC +gpu1:238:238 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/IPC +gpu1:242:242 [5] NCCL INFO Channel 02/0 : 5[5] -> 4[4] via P2P/IPC +gpu1:237:237 [0] NCCL INFO Connected all trees +gpu1:238:238 [1] NCCL INFO Connected all trees +gpu1:240:240 [3] NCCL INFO Connected all trees +gpu1:239:239 [2] NCCL INFO Connected all trees +gpu1:242:242 [5] NCCL INFO Connected all trees +gpu1:241:241 [4] NCCL INFO Connected all trees +gpu1:244:244 [7] NCCL INFO Connected all trees +gpu1:243:243 [6] NCCL INFO Connected all trees +gpu1:237:237 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:237:237 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:240:240 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:240:240 [3] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:237:237 [0] NCCL INFO CC Off, workFifoBytes 1048576 +gpu1:244:244 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:244:244 [7] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:241:241 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:241:241 [4] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:238:2398 [0] NCCL INFO [Proxy Progress] Device 1 CPU core 3 +gpu1:239:2399 [0] NCCL INFO [Proxy Progress] Device 2 CPU core 6 +gpu1:239:239 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:239:239 [2] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:238:238 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:238:238 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:237:237 [0] NCCL INFO ncclCommInitRankConfig comm 0x73224340 rank 0 nranks 16 cudaDev 0 nvmlDev 0 busId 19000 commId 0x1a548291cb460838 - Init COMPLETE +gpu1:239:239 [2] NCCL INFO ncclCommInitRankConfig comm 0x6be91a90 rank 2 nranks 16 cudaDev 2 nvmlDev 2 busId 1b000 commId 0x1a548291cb460838 - Init COMPLETE +gpu1:237:237 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.03, topo 0.07, graphs 0.01, connections 0.16, rest 0.05) +gpu1:240:240 [3] NCCL INFO ncclCommInitRankConfig comm 0x84d8a930 rank 3 nranks 16 cudaDev 3 nvmlDev 3 busId 1d000 commId 0x1a548291cb460838 - Init COMPLETE +gpu1:239:239 [2] NCCL INFO Init timings - ncclCommInitRankConfig: rank 2 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.03, topo 0.07, graphs 0.01, connections 0.21, rest 0.00) +gpu1:240:240 [3] NCCL INFO Init timings - ncclCommInitRankConfig: rank 3 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.10, graphs 0.00, connections 0.16, rest 0.05) +gpu1:238:238 [1] NCCL INFO ncclCommInitRankConfig comm 0x45c5a0e0 rank 1 nranks 16 cudaDev 1 nvmlDev 1 busId 1a000 commId 0x1a548291cb460838 - Init COMPLETE +gpu1:238:238 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.04, topo 0.07, graphs 0.00, connections 0.21, rest 0.00) +gpu1:243:2400 [0] NCCL INFO [Proxy Progress] Device 6 CPU core 113 +gpu1:242:2401 [0] NCCL INFO [Proxy Progress] Device 5 CPU core 104 +gpu1:243:243 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:243:243 [6] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:242:242 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu1:242:242 [5] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu1:242:242 [5] NCCL INFO ncclCommInitRankConfig comm 0x41815780 rank 5 nranks 16 cudaDev 5 nvmlDev 5 busId ab000 commId 0x1a548291cb460838 - Init COMPLETE +gpu1:243:243 [6] NCCL INFO ncclCommInitRankConfig comm 0x7ca785a0 rank 6 nranks 16 cudaDev 6 nvmlDev 6 busId ae000 commId 0x1a548291cb460838 - Init COMPLETE +gpu1:242:242 [5] NCCL INFO Init timings - ncclCommInitRankConfig: rank 5 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.03, topo 0.07, graphs 0.01, connections 0.21, rest 0.00) +gpu1:243:243 [6] NCCL INFO Init timings - ncclCommInitRankConfig: rank 6 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.02, topo 0.09, graphs 0.00, connections 0.21, rest 0.00) +gpu1:244:244 [7] NCCL INFO ncclCommInitRankConfig comm 0x390462b0 rank 7 nranks 16 cudaDev 7 nvmlDev 7 busId af000 commId 0x1a548291cb460838 - Init COMPLETE +gpu1:244:244 [7] NCCL INFO Init timings - ncclCommInitRankConfig: rank 7 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.04, topo 0.06, graphs 0.01, connections 0.16, rest 0.05) +gpu1:241:241 [4] NCCL INFO ncclCommInitRankConfig comm 0x6816cee0 rank 4 nranks 16 cudaDev 4 nvmlDev 4 busId aa000 commId 0x1a548291cb460838 - Init COMPLETE +gpu1:241:241 [4] NCCL INFO Init timings - ncclCommInitRankConfig: rank 4 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.02, topo 0.08, graphs 0.01, connections 0.16, rest 0.05) +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[2026-07-31 08:42:26 TP6 EP0] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP6 EP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP2 EP0] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP1 EP0] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP2 EP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP1 EP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP5 EP0] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP5 EP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP7 EP0] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP7 EP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP0 EP0] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP0 EP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP6 EP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.86 GB +[2026-07-31 08:42:26 TP1 EP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.83 GB +[2026-07-31 08:42:26 TP3 EP0] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP2 EP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.86 GB +[2026-07-31 08:42:26 TP3 EP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP4 EP0] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP5 EP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.83 GB +[2026-07-31 08:42:26 TP4 EP0] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP7 EP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.69 GB +[2026-07-31 08:42:26 TP0 EP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.80 GB +[2026-07-31 08:42:26 TP3 EP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.77 GB +[2026-07-31 08:42:26 TP4 EP0] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.72 GB + 0%| | 0/12 [00:00, ) +[2026-07-31 08:43:55 TP2 EP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP0 EP0] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP0 EP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP6 EP0] Capture target decode CUDA graph end. elapsed=88.83 s, mem usage=1.37 GB, avail mem=5.50 GB. +[2026-07-31 08:43:55 TP5 EP0] Capture target decode CUDA graph end. elapsed=88.82 s, mem usage=1.37 GB, avail mem=5.46 GB. +[2026-07-31 08:43:55 TP6 EP0] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP6 EP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP5 EP0] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP5 EP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP1 EP0] Capture target decode CUDA graph end. elapsed=88.83 s, mem usage=1.37 GB, avail mem=5.46 GB. +[2026-07-31 08:43:55 TP1 EP0] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP1 EP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP7 EP0] Capture target decode CUDA graph end. elapsed=88.83 s, mem usage=1.37 GB, avail mem=5.32 GB. +[2026-07-31 08:43:55 TP7 EP0] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP7 EP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP4 EP0] Capture target decode CUDA graph end. elapsed=88.83 s, mem usage=1.37 GB, avail mem=5.35 GB. +[2026-07-31 08:43:55 TP4 EP0] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP4 EP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP3 EP0] Capture target decode CUDA graph end. elapsed=88.85 s, mem usage=1.37 GB, avail mem=5.40 GB. +[2026-07-31 08:43:55 TP3 EP0] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP3 EP0] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:56] INFO: Started server process [1] +[2026-07-31 08:43:56] INFO: Waiting for application startup. +[2026-07-31 08:43:56] Using default chat sampling params from model generation config: {'temperature': 1.0, 'top_p': 1.0} +[2026-07-31 08:43:56] INFO: Application startup complete. +[2026-07-31 08:43:56] INFO: Uvicorn running on http://0.0.0.0:30002 (Press CTRL+C to quit) +[2026-07-31 08:43:57] INFO: 127.0.0.1:33282 - "GET /model_info HTTP/1.1" 200 OK +[2026-07-31 08:43:57] INFO: 10.101.0.11:46532 - "GET /health HTTP/1.1" 503 Service Unavailable +[2026-07-31 08:44:01 TP0 EP0] Prefill batch, #new-seq: 1, #new-token: 256, #cached-token: 0, full token usage: 0.00, swa token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 44.57 +[2026-07-31 08:44:01] INFO: 127.0.0.1:33294 - "POST /generate HTTP/1.1" 200 OK +[2026-07-31 08:44:01] The server is fired up and ready to roll! +[2026-07-31 08:44:04 TP0 EP0] Prefill batch, #new-seq: 1, #new-token: 256, #cached-token: 0, full token usage: 0.00, swa token usage: 0.00, #running-req: 0, #queue-req: 0, #pending-token: 0, cuda graph: False, input throughput (token/s): 93.57 +[2026-07-31 08:44:04] INFO: 10.101.0.11:46538 - "GET /health HTTP/1.1" 200 OK diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/head_server_cmd.txt b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/head_server_cmd.txt new file mode 100644 index 0000000..bd8d931 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/head_server_cmd.txt @@ -0,0 +1 @@ +docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_head --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 0 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/worker_nccl_transport.log b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/worker_nccl_transport.log new file mode 100644 index 0000000..144cb47 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/worker_nccl_transport.log @@ -0,0 +1,918 @@ +/usr/local/lib/python3.12/dist-packages/torchao/quantization/quant_api.py:1731: SyntaxWarning: invalid escape sequence '\.' + """Configuration class for applying different quantization configs to modules or parameters based on their fully qualified names (FQNs). +/sgl-workspace/sglang/python/sglang/launch_server.py:56: UserWarning: 'python -m sglang.launch_server' is still supported, but 'sglang serve' is the recommended entrypoint. + Example: sglang serve --model-path [options] + warnings.warn( +[2026-07-31 08:39:01] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:01] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:01] Breakable CUDA graph is incompatible with DeepSeek-V4 (heavy capture-pool memory pressure); disabling prefill CUDA graph. +[2026-07-31 08:39:01] Use dsv4 attention backend for DeepseekV4ForCausalLM, setting page_size to 256. +[2026-07-31 08:39:01] Setting swa_full_tokens_ratio to 0.1 for DeepseekV4ForCausalLM. +[2026-07-31 08:39:01] Setting KV cache dtype to fp8_e4m3 for DeepseekV4ForCausalLM. +[2026-07-31 08:39:01] Use flashinfer_mxfp4 as MoE runner backend on SM120 for DeepseekV4 +[2026-07-31 08:39:04] server_args=ServerArgs(model_path='/data/hf_models/DeepSeek-V4-Pro', tokenizer_path='/data/hf_models/DeepSeek-V4-Pro', tokenizer_mode='auto', tokenizer_backend='huggingface', tokenizer_worker_num=1, detokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', model_config_parser='auto', json_model_override_args='{}', dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='fp8_e4m3', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, enable_tf32_matmul=False, mem_fraction_static=0.9, max_running_requests=256, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=8192, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, disable_priority_preemption=False, default_priority_value=None, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, retraction_policy='length', schedule_conservativeness=1.0, page_size=256, swa_full_tokens_ratio=0.1, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', prefill_only_disable_kv_cache=False, disable_radix_cache=False, enable_page_major_kv_layout=False, enable_unified_memory=False, disable_chunked_prefix_cache=False, disable_overlap_schedule=False, num_continuous_decode_steps=1, scheduler_recv_interval=1, enable_mixed_chunk=False, nccl_port=None, dist_timeout=None, dist_init_addr='10.101.0.11:20002', nnodes=2, node_rank=1, tp_size=16, dcp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, enable_prefill_cp=False, cp_strategy=None, enable_dsa_cache_layer_split=False, enable_dsa_prefill_context_parallel=False, dsa_prefill_cp_mode='round-robin-split', enable_prefill_context_parallel=False, prefill_cp_mode='in-seq-split', enable_dp_attention=False, enable_dp_attention_local_control_broadcast=False, enable_dp_lm_head=False, enable_attn_tp_input_scattered=False, disable_attn_tp_gather=False, enable_p2p_check=False, device='cuda', base_gpu_id=0, gpu_id_step=1, random_seed=854656869, watchdog_timeout=300, soft_watchdog_timeout=None, sleep_on_idle=False, use_ray=False, custom_sigquit_handler=None, numa_node=None, gc_threshold=None, host='0.0.0.0', port=30002, fastapi_root_path='', smg_grpc_mode=False, grpc_mode=False, grpc_port=None, skip_server_warmup=False, warmups=None, enable_http2=False, ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_keyfile_password=None, enable_ssl_refresh=False, api_key=None, admin_api_key=None, served_model_name='/data/hf_models/DeepSeek-V4-Pro', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, default_chat_template_kwargs=None, strip_thinking_cache=False, enable_strict_thinking=False, tool_call_parser=None, tool_server=None, sampling_defaults='model', asr_max_buffer_seconds=60, asr_max_concurrent_sessions=32, preferred_sampling_params=None, allow_auto_truncate=False, stream_interval=1, batch_notify_size=16, stream_response_default_include_usage=False, incremental_streaming_output=False, enable_streaming_session=False, enable_session_radix_cache=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, smg_http_sidecar_port=None, enable_mfu_metrics=False, enable_metrics_for_all_schedulers=False, load_snapshot_publish_interval=15, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_forward_pass_metrics=False, forward_pass_metrics_worker_id='', forward_pass_metrics_ipc_name=None, enable_trace=False, trace_modules='request', otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, stat_loggers=None, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, attention_backend='dsv4', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', radix_cache_backend=None, mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='auto', bf16_gemm_backend='auto', dsa_prefill_backend=None, dsa_decode_backend=None, dsa_paged_mqa_logits_backend='auto', dsa_topk_backend='sgl-kernel', disable_flashinfer_autotune=False, mamba_backend='triton', cuda_graph_config=CudaGraphConfig(decode=PhaseConfig(backend='full', max_bs=64, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], tc_compiler='eager', full_prefill_max_req=None), prefill=PhaseConfig(backend='disabled', max_bs=8192, bs=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192], tc_compiler='eager', full_prefill_max_req=None)), cuda_graph_backend_decode=None, cuda_graph_backend_prefill=None, cuda_graph_max_bs_decode=64, cuda_graph_max_bs_prefill=None, cuda_graph_bs_decode=None, cuda_graph_bs_prefill=None, cuda_graph_tc_compiler=None, disable_prefill_cuda_graph=False, disable_decode_cuda_graph=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, debug_cuda_graph=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, flashinfer_mla_disable_ragged=False, enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_fused_moe_sum_all_reduce=False, enable_deepseek_v4_fp4_indexer=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, pre_warm_nccl=False, enable_quant_communications=False, enable_flashinfer_allreduce_fusion=False, enforce_disable_flashinfer_allreduce_fusion=False, flashinfer_allreduce_fusion_backend=None, enable_aiter_allreduce_fusion=False, enable_torch_compile=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, torchao_config='', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_dflash_block_size=None, speculative_dspark_block_size=None, speculative_dspark_sps_table_path=None, speculative_dspark_confidence_sts_path=None, speculative_dspark_align_verify_tokens_to_graph_tier=False, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_use_rejection_sampling=False, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_draft_window_size=None, speculative_moe_runner_backend='flashinfer_mxfp4', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_skip_dp_mlp_sync=False, enable_multi_layer_eagle=False, speculative_adaptive=False, speculative_adaptive_config=None, decoupled_spec_bind_endpoint=None, decoupled_spec_connect_endpoints=None, decoupled_spec_rank=None, decoupled_spec_role='null', spec_trace_dir=None, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_max_trie_depth=18, speculative_ngram_capacity=10000000, speculative_ngram_external_corpus_path=None, speculative_ngram_external_sam_budget=0, speculative_ngram_external_corpus_max_tokens=10000000, ep_size=2, moe_a2a_backend='none', moe_runner_backend='flashinfer_mxfp4', flashinfer_mxfp4_moe_precision='default', deepep_mode='auto', fuseep_mode=2, deepep_dispatcher_output_dtype='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, enable_elastic_expert_backup=False, mooncake_ib_device=None, enable_waterfill=False, ep_join_mode=None, ep_join_rank_offset=0, elastic_ep_initial_size=None, max_ep_size=None, elastic_ep_scale_timeout=600, elastic_ep_rejoin=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, disable_shared_experts_fusion=False, enforce_shared_experts_fusion=False, max_mamba_cache_size=None, mamba_ssm_dtype=None, enable_mamba_cache_stochastic_rounding=False, mamba_cache_philox_rounds=0, mamba_full_memory_ratio=0.9, mamba_radix_cache_strategy='auto', uses_mamba_radix_cache=False, mamba_track_interval=256, enable_int8_mamba_checkpoint=False, int8_mamba_ckpt_size=None, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, enable_linear_replayssm=False, linear_replayssm_cache_len=16, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='page_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='timeout', hicache_storage_backend_extra_config=None, enable_hisparse=False, hisparse_config=None, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, enable_mm_global_cache=False, disable_fast_image_processor=False, mm_feature_transport='cpu', keep_mm_feature_on_device=False, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, experts_shared_outer_loras=None, lora_use_virtual_experts=False, lora_strict_loading=False, lora_drain_wait_threshold=0.0, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', enable_lmcache=False, lmcache_config_file=None, enable_flexkv=False, flexkv_config_file=None, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, dllm_fdfo=True, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_ib_device=None, disaggregation_decode_enable_radix_cache=False, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_extra_slots=None, disaggregation_decode_polling_interval=1, optimistic_prefill_attempts=0, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], encoder_bootstrap_port=8997, encoder_register_urls=[], enable_adaptive_dispatch_to_encoder=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, custom_weight_loader=[], weight_loader_disable_mmap=False, weight_loader_prefetch_checkpoints=False, weight_loader_prefetch_num_threads=4, weight_loader_drop_cache_after_load=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, engine_info_bootstrap_port=6789, modelexpress_config=None, download_dir=None, model_checksum=None, delete_ckpt_after_loading=False, decrypted_config_file=None, decrypted_draft_config_file=None, checkpoint_engine_wait_weights_before_ready=False, enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, prefill_delayer_queue_min_ratio=None, prefill_delayer_max_delay_ms=None, min_free_slots_delay=None, enable_deterministic_inference=False, rl_on_policy_target=None, kv_canary='none', kv_canary_real_data='none', kv_canary_sweep_interval=0, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, enable_custom_logit_processor=False, enable_return_hidden_states=False, enable_return_routed_experts=False, enable_return_indexer_topk=False, disable_outlines_disk_cache=False, enable_mis=False, forward_hooks=None, msprobe_dump_config=None) +[2026-07-31 08:39:04] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:04] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:04] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:04] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:04] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:04] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:04] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:04] User lacks permission to set NUMA affinity, skipping NUMA node configuration for GPU. If using docker, try adding --cap-add SYS_NICE to your docker run command. +[2026-07-31 08:39:18 TP9 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:18 TP9 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:18 TP14 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:18 TP14 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:18 TP10 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:18 TP10 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:18 TP15 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:18 TP15 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:19 TP12 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:19 TP12 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:19 TP13 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:19 TP13 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:19 TP8 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:19 TP8 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:19 TP11 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:19 TP11 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:20 TP9 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:20 TP9 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:20 TP9 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:21 TP14 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:21 TP15 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:21 TP9 EP1] Init torch distributed begin. +[2026-07-31 08:39:21 TP10 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:21 TP12 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:21 TP14 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:21 TP14 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:21 TP15 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:21 TP15 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:21 TP8 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:21 TP10 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:21 TP10 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:21 TP14 EP1] Init torch distributed begin. +[2026-07-31 08:39:21 TP15 EP1] Init torch distributed begin. +[2026-07-31 08:39:21 TP10 EP1] Init torch distributed begin. +[2026-07-31 08:39:21 TP12 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:21 TP12 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:21 TP13 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:21 TP12 EP1] Init torch distributed begin. +[2026-07-31 08:39:21 TP8 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:21 TP8 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:21 TP8 EP1] Init torch distributed begin. +[2026-07-31 08:39:21 TP11 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:39:22 TP13 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:22 TP13 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:22 TP13 EP1] Init torch distributed begin. +[2026-07-31 08:39:22 TP11 EP1] Auto-detected DSV4 routed-expert layout: is_fp4_experts=True +[2026-07-31 08:39:22 TP11 EP1] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] +[2026-07-31 08:39:22 TP11 EP1] Init torch distributed begin. +gpu3:240:240 [4] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu3:240:240 [4] NCCL INFO cudaDriverVersion 13000 +gpu3:240:240 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:240:240 [4] NCCL INFO Bootstrap: Using eth0:10.101.0.13<0> +gpu3:238:238 [2] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu3:240:240 [4] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu3:238:238 [2] NCCL INFO cudaDriverVersion 13000 +gpu3:239:239 [3] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu3:239:239 [3] NCCL INFO cudaDriverVersion 13000 +gpu3:237:237 [1] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu3:237:237 [1] NCCL INFO cudaDriverVersion 13000 +gpu3:236:236 [0] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu3:236:236 [0] NCCL INFO cudaDriverVersion 13000 +gpu3:237:237 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:238:238 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:239:239 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:236:236 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:238:238 [2] NCCL INFO Bootstrap: Using eth0:10.101.0.13<0> +gpu3:238:238 [2] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu3:237:237 [1] NCCL INFO Bootstrap: Using eth0:10.101.0.13<0> +gpu3:239:239 [3] NCCL INFO Bootstrap: Using eth0:10.101.0.13<0> +gpu3:239:239 [3] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu3:237:237 [1] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu3:236:236 [0] NCCL INFO Bootstrap: Using eth0:10.101.0.13<0> +gpu3:236:236 [0] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu3:243:243 [7] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu3:243:243 [7] NCCL INFO cudaDriverVersion 13000 +gpu3:241:241 [5] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu3:241:241 [5] NCCL INFO cudaDriverVersion 13000 +gpu3:242:242 [6] NCCL INFO ENV/Plugin: Could not find: libnccl-env.so +gpu3:242:242 [6] NCCL INFO cudaDriverVersion 13000 +gpu3:243:243 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:241:241 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:242:242 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:243:243 [7] NCCL INFO Bootstrap: Using eth0:10.101.0.13<0> +gpu3:243:243 [7] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu3:241:241 [5] NCCL INFO Bootstrap: Using eth0:10.101.0.13<0> +gpu3:241:241 [5] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu3:242:242 [6] NCCL INFO Bootstrap: Using eth0:10.101.0.13<0> +gpu3:242:242 [6] NCCL INFO NCCL version 2.28.9+cuda13.0 +gpu3:240:240 [4] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu3:240:240 [4] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:240:240 [4] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu3:240:240 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.13<0> +gpu3:240:240 [4] NCCL INFO Initialized NET plugin IB +gpu3:240:240 [4] NCCL INFO Assigned NET plugin IB to comm +gpu3:240:240 [4] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:240:240 [4] NCCL INFO Using network IB +gpu3:243:243 [7] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu3:243:243 [7] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:243:243 [7] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu3:241:241 [5] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu3:241:241 [5] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:241:241 [5] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu3:240:240 [4] NCCL INFO DMA-BUF is available on GPU device 4 +gpu3:240:240 [4] NCCL INFO ncclCommInitRank comm 0x289f09f0 rank 12 nranks 16 cudaDev 4 nvmlDev 4 busId aa000 commId 0x98560d124efa1ab - Init START +gpu3:239:239 [3] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu3:239:239 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:239:239 [3] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu3:243:243 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.13<0> +gpu3:243:243 [7] NCCL INFO Initialized NET plugin IB +gpu3:241:241 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.13<0> +gpu3:241:241 [5] NCCL INFO Initialized NET plugin IB +gpu3:243:243 [7] NCCL INFO Assigned NET plugin IB to comm +gpu3:243:243 [7] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:243:243 [7] NCCL INFO Using network IB +gpu3:241:241 [5] NCCL INFO Assigned NET plugin IB to comm +gpu3:241:241 [5] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:241:241 [5] NCCL INFO Using network IB +gpu3:238:238 [2] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu3:238:238 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:238:238 [2] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu3:241:241 [5] NCCL INFO DMA-BUF is available on GPU device 5 +gpu3:241:241 [5] NCCL INFO ncclCommInitRank comm 0x266b3bb0 rank 13 nranks 16 cudaDev 5 nvmlDev 5 busId ab000 commId 0x98560d124efa1ab - Init START +gpu3:243:243 [7] NCCL INFO DMA-BUF is available on GPU device 7 +gpu3:243:243 [7] NCCL INFO ncclCommInitRank comm 0x1bdd1c30 rank 15 nranks 16 cudaDev 7 nvmlDev 7 busId af000 commId 0x98560d124efa1ab - Init START +gpu3:239:239 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.13<0> +gpu3:239:239 [3] NCCL INFO Initialized NET plugin IB +gpu3:239:239 [3] NCCL INFO Assigned NET plugin IB to comm +gpu3:239:239 [3] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:239:239 [3] NCCL INFO Using network IB +gpu3:242:242 [6] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu3:242:242 [6] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:242:242 [6] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu3:239:239 [3] NCCL INFO DMA-BUF is available on GPU device 3 +gpu3:239:239 [3] NCCL INFO ncclCommInitRank comm 0x43bffe40 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId 1d000 commId 0x98560d124efa1ab - Init START +gpu3:240:240 [4] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu3:238:238 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.13<0> +gpu3:238:238 [2] NCCL INFO Initialized NET plugin IB +gpu3:238:238 [2] NCCL INFO Assigned NET plugin IB to comm +gpu3:238:238 [2] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:238:238 [2] NCCL INFO Using network IB +gpu3:238:238 [2] NCCL INFO DMA-BUF is available on GPU device 2 +gpu3:238:238 [2] NCCL INFO ncclCommInitRank comm 0x41cc8e00 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 1b000 commId 0x98560d124efa1ab - Init START +gpu3:239:239 [3] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu3:242:242 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.13<0> +gpu3:242:242 [6] NCCL INFO Initialized NET plugin IB +gpu3:242:242 [6] NCCL INFO Assigned NET plugin IB to comm +gpu3:242:242 [6] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:242:242 [6] NCCL INFO Using network IB +gpu3:242:242 [6] NCCL INFO DMA-BUF is available on GPU device 6 +gpu3:242:242 [6] NCCL INFO ncclCommInitRank comm 0x527dcd40 rank 14 nranks 16 cudaDev 6 nvmlDev 6 busId ae000 commId 0x98560d124efa1ab - Init START +gpu3:241:241 [5] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu3:242:242 [6] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu3:237:237 [1] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu3:237:237 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:236:236 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so +gpu3:237:237 [1] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu3:236:236 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 +gpu3:236:236 [0] NCCL INFO NCCL_IB_HCA set to =mlx5_0:1,mlx5_3:1 +gpu3:237:237 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.13<0> +gpu3:237:237 [1] NCCL INFO Initialized NET plugin IB +gpu3:237:237 [1] NCCL INFO Assigned NET plugin IB to comm +gpu3:237:237 [1] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:237:237 [1] NCCL INFO Using network IB +gpu3:236:236 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/RoCE [1]mlx5_3:1/RoCE [RO]; OOB eth0:10.101.0.13<0> +gpu3:236:236 [0] NCCL INFO Initialized NET plugin IB +gpu3:236:236 [0] NCCL INFO Assigned NET plugin IB to comm +gpu3:236:236 [0] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:236:236 [0] NCCL INFO Using network IB +gpu3:237:237 [1] NCCL INFO DMA-BUF is available on GPU device 1 +gpu3:237:237 [1] NCCL INFO ncclCommInitRank comm 0x272bddb0 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 1a000 commId 0x98560d124efa1ab - Init START +gpu3:238:238 [2] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu3:236:236 [0] NCCL INFO DMA-BUF is available on GPU device 0 +gpu3:236:236 [0] NCCL INFO ncclCommInitRank comm 0x1d586c60 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 19000 commId 0x98560d124efa1ab - Init START +gpu3:237:237 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu3:236:236 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu3:243:243 [7] NCCL INFO RAS client listening socket at 127.0.0.1<28028> +gpu3:243:243 [7] NCCL INFO Bootstrap timings total 0.213386 (create 0.000023, send 0.000121, recv 0.155036, ring 0.057890, delay 0.000001) +gpu3:243:243 [7] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu3:242:242 [6] NCCL INFO Bootstrap timings total 0.183582 (create 0.000023, send 0.000150, recv 0.000368, ring 0.182767, delay 0.000001) +gpu3:242:242 [6] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu3:241:241 [5] NCCL INFO Bootstrap timings total 0.217698 (create 0.000025, send 0.000157, recv 0.034300, ring 0.182924, delay 0.000001) +gpu3:241:241 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu3:240:240 [4] NCCL INFO Bootstrap timings total 0.227838 (create 0.000029, send 0.000188, recv 0.010434, ring 0.192591, delay 0.000001) +gpu3:240:240 [4] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu3:239:239 [3] NCCL INFO Bootstrap timings total 0.193435 (create 0.000025, send 0.000152, recv 0.000298, ring 0.187819, delay 0.000001) +gpu3:239:239 [3] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu3:238:238 [2] NCCL INFO Bootstrap timings total 0.189104 (create 0.000023, send 0.000193, recv 0.000254, ring 0.148962, delay 0.000001) +gpu3:238:238 [2] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu3:236:236 [0] NCCL INFO Bootstrap timings total 0.146983 (create 0.000036, send 0.000161, recv 0.000278, ring 0.094612, delay 0.000001) +gpu3:237:237 [1] NCCL INFO Bootstrap timings total 0.150958 (create 0.000031, send 0.000150, recv 0.000257, ring 0.146145, delay 0.000001) +gpu3:236:236 [0] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu3:237:237 [1] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. +gpu3:241:241 [5] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 5 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu3:241:241 [5] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu3:241:241 [5] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu3:240:240 [4] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 4 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu3:240:240 [4] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu3:240:240 [4] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu3:242:242 [6] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 6 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu3:242:242 [6] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu3:242:242 [6] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu3:239:239 [3] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 3 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu3:239:239 [3] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu3:239:239 [3] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu3:238:238 [2] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 2 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu3:238:238 [2] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu3:238:238 [2] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu3:243:243 [7] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 7 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu3:243:243 [7] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu3:243:243 [7] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu3:237:237 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu3:236:236 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu3:237:237 [1] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu3:236:236 [0] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. +gpu3:236:236 [0] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu3:237:237 [1] NCCL INFO NCCL_CROSS_NIC set by environment to 1. +gpu3:241:241 [5] NCCL INFO comm 0x266b3bb0 rank 13 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 +gpu3:241:241 [5] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->15 [2] 14/-1/-1->13->12 [3] 14/-1/-1->13->15 +gpu3:241:241 [5] NCCL INFO P2P Chunksize set to 131072 +gpu3:242:242 [6] NCCL INFO comm 0x527dcd40 rank 14 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 +gpu3:243:243 [7] NCCL INFO comm 0x1bdd1c30 rank 15 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 +gpu3:240:240 [4] NCCL INFO comm 0x289f09f0 rank 12 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 +gpu3:242:242 [6] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 12/-1/-1->14->13 [2] 15/-1/-1->14->13 [3] 12/-1/-1->14->13 +gpu3:242:242 [6] NCCL INFO P2P Chunksize set to 131072 +gpu3:243:243 [7] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] 13/-1/-1->15->11 [2] -1/-1/-1->15->14 [3] 13/7/-1->15->-1 +gpu3:240:240 [4] NCCL INFO Trees [0] 13/-1/-1->12->8 [1] -1/-1/-1->12->14 [2] 13/4/-1->12->-1 [3] -1/-1/-1->12->14 +gpu3:243:243 [7] NCCL INFO P2P Chunksize set to 131072 +gpu3:240:240 [4] NCCL INFO P2P Chunksize set to 131072 +gpu3:239:239 [3] NCCL INFO comm 0x43bffe40 rank 11 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 +gpu3:238:238 [2] NCCL INFO comm 0x41cc8e00 rank 10 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 +gpu3:239:239 [3] NCCL INFO Trees [0] -1/-1/-1->11->10 [1] 9/7/15->11->3 [2] -1/-1/-1->11->10 [3] 9/-1/-1->11->7 +gpu3:239:239 [3] NCCL INFO P2P Chunksize set to 131072 +gpu3:238:238 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 8/-1/-1->10->9 [2] 11/-1/-1->10->9 [3] 8/-1/-1->10->9 +gpu3:238:238 [2] NCCL INFO P2P Chunksize set to 131072 +gpu3:236:236 [0] NCCL INFO comm 0x1d586c60 rank 8 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 +gpu3:237:237 [1] NCCL INFO comm 0x272bddb0 rank 9 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 +gpu3:241:241 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu3:242:242 [6] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu3:241:241 [5] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:240:240 [4] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu3:243:243 [7] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu3:242:242 [6] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:240:240 [4] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:243:243 [7] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:236:236 [0] NCCL INFO Trees [0] 9/4/12->8->0 [1] -1/-1/-1->8->10 [2] 9/-1/-1->8->4 [3] -1/-1/-1->8->10 +gpu3:236:236 [0] NCCL INFO P2P Chunksize set to 131072 +gpu3:237:237 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 10/-1/-1->9->11 [2] 10/-1/-1->9->8 [3] 10/-1/-1->9->11 +gpu3:237:237 [1] NCCL INFO P2P Chunksize set to 131072 +gpu3:243:1442 [0] NCCL INFO [Proxy Service] Device 7 CPU core 41 +gpu3:239:239 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu3:240:1439 [0] NCCL INFO [Proxy Service] Device 4 CPU core 124 +gpu3:239:239 [3] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:240:1443 [0] NCCL INFO [Proxy Service UDS] Device 4 CPU core 127 +gpu3:243:1444 [0] NCCL INFO [Proxy Service UDS] Device 7 CPU core 33 +gpu3:238:238 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu3:238:238 [2] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:242:1441 [0] NCCL INFO [Proxy Service] Device 6 CPU core 43 +gpu3:236:236 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu3:242:1446 [0] NCCL INFO [Proxy Service UDS] Device 6 CPU core 33 +gpu3:236:236 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:241:1440 [0] NCCL INFO [Proxy Service] Device 5 CPU core 38 +gpu3:237:237 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so +gpu3:237:237 [1] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:241:1445 [0] NCCL INFO [Proxy Service UDS] Device 5 CPU core 33 +gpu3:239:1447 [0] NCCL INFO [Proxy Service] Device 3 CPU core 85 +gpu3:238:1448 [0] NCCL INFO [Proxy Service] Device 2 CPU core 70 +gpu3:239:1449 [0] NCCL INFO [Proxy Service UDS] Device 3 CPU core 86 +gpu3:236:1451 [0] NCCL INFO [Proxy Service] Device 0 CPU core 8 +gpu3:238:1450 [0] NCCL INFO [Proxy Service UDS] Device 2 CPU core 71 +gpu3:236:1452 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 73 +gpu3:237:1453 [0] NCCL INFO [Proxy Service] Device 1 CPU core 10 +gpu3:237:1454 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 11 +gpu3:238:238 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 00/0 : 14[6] -> 15[7] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 00/0 : 13[5] -> 14[6] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 01/0 : 14[6] -> 15[7] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 01/0 : 13[5] -> 14[6] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 02/0 : 14[6] -> 15[7] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 02/0 : 10[2] -> 11[3] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 02/0 : 9[1] -> 10[2] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 02/0 : 13[5] -> 14[6] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 03/0 : 10[2] -> 11[3] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 03/0 : 9[1] -> 10[2] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 03/0 : 14[6] -> 15[7] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 03/0 : 13[5] -> 14[6] via P2P/IPC +gpu3:240:1455 [0] NCCL INFO [Proxy Progress] Device 4 CPU core 98 +gpu3:240:240 [4] NCCL INFO Channel 00/0 : 11[3] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 01/0 : 11[3] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 02/0 : 11[3] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 03/0 : 11[3] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 00/0 : 12[4] -> 13[5] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 01/0 : 12[4] -> 13[5] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 02/0 : 12[4] -> 13[5] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 03/0 : 12[4] -> 13[5] via P2P/IPC +gpu3:243:1456 [0] NCCL INFO [Proxy Progress] Device 7 CPU core 48 +gpu3:243:243 [7] NCCL INFO Channel 00/0 : 15[7] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 01/0 : 15[7] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 02/0 : 15[7] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 03/0 : 15[7] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu3:236:1457 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 12 +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 01/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 02/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 03/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 01/0 : 8[0] -> 9[1] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 02/0 : 8[0] -> 9[1] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 03/0 : 8[0] -> 9[1] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:239:1458 [0] NCCL INFO [Proxy Progress] Device 3 CPU core 87 +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 02/0 : 11[3] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 03/0 : 11[3] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:240:240 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:241:241 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:239:239 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:240:240 [4] NCCL INFO Channel 01/0 : 12[4] -> 14[6] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 01/0 : 13[5] -> 15[7] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:243:243 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:238:238 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:237:237 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:236:236 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:236:236 [0] NCCL INFO Channel 01/0 : 8[0] -> 10[2] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 01/0 : 9[1] -> 11[3] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 03/0 : 8[0] -> 10[2] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 03/0 : 13[5] -> 15[7] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 03/0 : 12[4] -> 14[6] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 03/0 : 9[1] -> 11[3] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 01/0 : 10[2] -> 8[0] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 4[4] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:242:242 [6] NCCL INFO Channel 01/0 : 14[6] -> 12[4] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 02/0 : 4[4] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 01/0 : 11[3] -> 15[7] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 00/0 : 8[0] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:238:238 [2] NCCL INFO Channel 03/0 : 10[2] -> 8[0] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 7[7] -> 11[3] [receive] via NET/IB/0/GDRDMA +gpu3:242:242 [6] NCCL INFO Channel 03/0 : 14[6] -> 12[4] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 03/0 : 7[7] -> 11[3] [receive] via NET/IB/0/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 02/0 : 4[4] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 15[7] [send] via NET/IB/0/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 02/0 : 12[4] -> 4[4] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 03/0 : 7[7] -> 15[7] [receive] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 03/0 : 15[7] -> 7[7] [send] via NET/IB/1/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 3[3] -> 11[3] [receive] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 3[3] [send] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 01/0 : 15[7] -> 11[3] [send] via NET/IB/1/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 12[4] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 15[7] -> 11[3] [receive] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 7[7] [send] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[7] [send] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 02/0 : 8[0] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 00/0 : 12[4] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 01/0 : 15[7] -> 13[5] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 9[1] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 03/0 : 11[3] -> 9[1] via P2P/IPC +gpu3:243:243 [7] NCCL INFO Channel 03/0 : 15[7] -> 13[5] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 00/0 : 11[3] -> 10[2] via P2P/IPC +gpu3:243:243 [7] NCCL INFO Channel 00/0 : 15[7] -> 14[6] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 02/0 : 11[3] -> 10[2] via P2P/IPC +gpu3:243:243 [7] NCCL INFO Channel 02/0 : 15[7] -> 14[6] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 00/0 : 9[1] -> 8[0] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 00/0 : 10[2] -> 9[1] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 00/0 : 14[6] -> 13[5] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 01/0 : 14[6] -> 13[5] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 01/0 : 10[2] -> 9[1] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 02/0 : 9[1] -> 8[0] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 00/0 : 13[5] -> 12[4] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 02/0 : 10[2] -> 9[1] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 02/0 : 14[6] -> 13[5] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 02/0 : 13[5] -> 12[4] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 03/0 : 10[2] -> 9[1] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 03/0 : 14[6] -> 13[5] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Connected all trees +gpu3:236:236 [0] NCCL INFO Connected all trees +gpu3:243:243 [7] NCCL INFO Connected all trees +gpu3:242:242 [6] NCCL INFO Connected all trees +gpu3:236:236 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu3:236:236 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:236:236 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:241:241 [5] NCCL INFO Connected all trees +gpu3:240:240 [4] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu3:240:240 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:237:237 [1] NCCL INFO Connected all trees +gpu3:240:240 [4] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:243:243 [7] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu3:243:243 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:243:243 [7] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:239:239 [3] NCCL INFO Connected all trees +gpu3:238:238 [2] NCCL INFO Connected all trees +gpu3:239:239 [3] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu3:239:239 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:239:239 [3] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:242:1459 [0] NCCL INFO [Proxy Progress] Device 6 CPU core 44 +gpu3:242:242 [6] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu3:242:242 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:242:242 [6] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:241:1460 [0] NCCL INFO [Proxy Progress] Device 5 CPU core 45 +gpu3:241:241 [5] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu3:241:241 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:241:241 [5] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:238:1461 [0] NCCL INFO [Proxy Progress] Device 2 CPU core 13 +gpu3:237:1462 [0] NCCL INFO [Proxy Progress] Device 1 CPU core 14 +gpu3:241:241 [5] NCCL INFO ncclCommInitRank comm 0x266b3bb0 rank 13 nranks 16 cudaDev 5 nvmlDev 5 busId ab000 commId 0x98560d124efa1ab - Init COMPLETE +gpu3:241:241 [5] NCCL INFO Init timings - ncclCommInitRank: rank 13 nranks 16 total 0.72 (kernels 0.23, alloc 0.01, bootstrap 0.22, allgathers 0.02, topo 0.02, graphs 0.00, connections 0.21, rest 0.00) +gpu3:238:238 [2] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu3:238:238 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:238:238 [2] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:237:237 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so +gpu3:237:237 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:237:237 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:243:243 [7] NCCL INFO ncclCommInitRank comm 0x1bdd1c30 rank 15 nranks 16 cudaDev 7 nvmlDev 7 busId af000 commId 0x98560d124efa1ab - Init COMPLETE +gpu3:243:243 [7] NCCL INFO Init timings - ncclCommInitRank: rank 15 nranks 16 total 0.72 (kernels 0.23, alloc 0.02, bootstrap 0.21, allgathers 0.02, topo 0.02, graphs 0.00, connections 0.16, rest 0.05) +gpu3:242:242 [6] NCCL INFO ncclCommInitRank comm 0x527dcd40 rank 14 nranks 16 cudaDev 6 nvmlDev 6 busId ae000 commId 0x98560d124efa1ab - Init COMPLETE +gpu3:240:240 [4] NCCL INFO ncclCommInitRank comm 0x289f09f0 rank 12 nranks 16 cudaDev 4 nvmlDev 4 busId aa000 commId 0x98560d124efa1ab - Init COMPLETE +gpu3:242:242 [6] NCCL INFO Init timings - ncclCommInitRank: rank 14 nranks 16 total 0.72 (kernels 0.27, alloc 0.01, bootstrap 0.18, allgathers 0.02, topo 0.02, graphs 0.00, connections 0.19, rest 0.02) +gpu3:240:240 [4] NCCL INFO Init timings - ncclCommInitRank: rank 12 nranks 16 total 0.75 (kernels 0.23, alloc 0.03, bootstrap 0.23, allgathers 0.02, topo 0.02, graphs 0.00, connections 0.16, rest 0.05) +gpu3:239:239 [3] NCCL INFO ncclCommInitRank comm 0x43bffe40 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId 1d000 commId 0x98560d124efa1ab - Init COMPLETE +gpu3:236:236 [0] NCCL INFO ncclCommInitRank comm 0x1d586c60 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 19000 commId 0x98560d124efa1ab - Init COMPLETE +gpu3:238:238 [2] NCCL INFO ncclCommInitRank comm 0x41cc8e00 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 1b000 commId 0x98560d124efa1ab - Init COMPLETE +gpu3:237:237 [1] NCCL INFO ncclCommInitRank comm 0x272bddb0 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 1a000 commId 0x98560d124efa1ab - Init COMPLETE +gpu3:239:239 [3] NCCL INFO Init timings - ncclCommInitRank: rank 11 nranks 16 total 0.75 (kernels 0.26, alloc 0.03, bootstrap 0.19, allgathers 0.02, topo 0.02, graphs 0.00, connections 0.16, rest 0.05) +gpu3:236:236 [0] NCCL INFO Init timings - ncclCommInitRank: rank 8 nranks 16 total 0.75 (kernels 0.31, alloc 0.04, bootstrap 0.15, allgathers 0.02, topo 0.02, graphs 0.00, connections 0.16, rest 0.05) +gpu3:238:238 [2] NCCL INFO Init timings - ncclCommInitRank: rank 10 nranks 16 total 0.75 (kernels 0.27, alloc 0.03, bootstrap 0.19, allgathers 0.02, topo 0.02, graphs 0.00, connections 0.21, rest 0.00) +gpu3:237:237 [1] NCCL INFO Init timings - ncclCommInitRank: rank 9 nranks 16 total 0.75 (kernels 0.30, alloc 0.03, bootstrap 0.15, allgathers 0.02, topo 0.02, graphs 0.00, connections 0.21, rest 0.00) +[2026-07-31 08:39:37 TP15 EP1] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP14 EP1] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP13 EP1] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP12 EP1] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP11 EP1] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP10 EP1] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP8 EP1] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP9 EP1] CustomAllreduce is disabled because this process group spans across nodes. +[2026-07-31 08:39:37 TP15 EP1] Init torch distributed ends. elapsed=15.99 s, mem usage=0.44 GB +[2026-07-31 08:39:37 TP12 EP1] Init torch distributed ends. elapsed=15.80 s, mem usage=0.42 GB +[2026-07-31 08:39:37 TP14 EP1] Init torch distributed ends. elapsed=16.02 s, mem usage=0.39 GB +[2026-07-31 08:39:37 TP11 EP1] Init torch distributed ends. elapsed=15.22 s, mem usage=0.48 GB +[2026-07-31 08:39:37 TP13 EP1] Init torch distributed ends. elapsed=15.39 s, mem usage=0.41 GB +[2026-07-31 08:39:37 TP10 EP1] Init torch distributed ends. elapsed=15.92 s, mem usage=0.39 GB +[2026-07-31 08:39:37 TP8 EP1] Init torch distributed ends. elapsed=15.68 s, mem usage=0.46 GB +[2026-07-31 08:39:37 TP9 EP1] Init torch distributed ends. elapsed=16.42 s, mem usage=0.41 GB +[2026-07-31 08:39:40 TP14 EP1] Load weight begin. avail mem=82.11 GB +[2026-07-31 08:39:40 TP13 EP1] Load weight begin. avail mem=82.10 GB +[2026-07-31 08:39:40 TP8 EP1] Load weight begin. avail mem=82.04 GB +[2026-07-31 08:39:40 TP10 EP1] Load weight begin. avail mem=82.11 GB +[2026-07-31 08:39:40 TP11 EP1] Load weight begin. avail mem=82.03 GB +[2026-07-31 08:39:40 TP15 EP1] Load weight begin. avail mem=82.07 GB +[2026-07-31 08:39:40 TP9 EP1] Load weight begin. avail mem=82.10 GB +[2026-07-31 08:39:40 TP12 EP1] Load weight begin. avail mem=82.08 GB +[2026-07-31 08:39:40 TP14 EP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP13 EP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP8 EP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP15 EP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP11 EP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP10 EP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP9 EP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:40 TP12 EP1] FlashInfer TRTLLM MoE deferred finalize is disabled (moe_runner_backend=flashinfer_mxfp4, quant_method=Mxfp4FlashinferCutlassMoEMethod). +[2026-07-31 08:39:46 TP15 EP1] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP14 EP1] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP12 EP1] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP11 EP1] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP13 EP1] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP9 EP1] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP8 EP1] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:39:46 TP10 EP1] multimem all-gather disabled because the TP group spans across nodes. +[2026-07-31 08:40:02 TP8 EP1] Execute dequant fp8 wo_a +[2026-07-31 08:40:02 TP9 EP1] Execute dequant fp8 wo_a +[2026-07-31 08:40:03 TP13 EP1] Execute dequant fp8 wo_a +[2026-07-31 08:40:03 TP11 EP1] Execute dequant fp8 wo_a +[2026-07-31 08:40:03 TP14 EP1] Execute dequant fp8 wo_a +[2026-07-31 08:40:03 TP15 EP1] Execute dequant fp8 wo_a +[2026-07-31 08:40:03 TP12 EP1] Execute dequant fp8 wo_a +[2026-07-31 08:40:04 TP10 EP1] Execute dequant fp8 wo_a +[2026-07-31 08:41:20 TP8 EP1] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results! +[2026-07-31 08:41:20 TP8 EP1] Load weight end. elapsed=99.84 s, type=DeepseekV4ForCausalLM, quant=fp8, fmt=e4m3, avail mem=25.17 GB, mem usage=56.87 GB. +[2026-07-31 08:41:23 TP15 EP1] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results! +[2026-07-31 08:41:23 TP15 EP1] Load weight end. elapsed=102.60 s, type=DeepseekV4ForCausalLM, quant=fp8, fmt=e4m3, avail mem=25.19 GB, mem usage=56.87 GB. +[2026-07-31 08:41:23 TP14 EP1] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results! +[2026-07-31 08:41:23 TP14 EP1] Load weight end. elapsed=102.78 s, type=DeepseekV4ForCausalLM, quant=fp8, fmt=e4m3, avail mem=25.24 GB, mem usage=56.87 GB. +[2026-07-31 08:41:23 TP12 EP1] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results! +[2026-07-31 08:41:23 TP12 EP1] Load weight end. elapsed=103.32 s, type=DeepseekV4ForCausalLM, quant=fp8, fmt=e4m3, avail mem=25.21 GB, mem usage=56.87 GB. +[2026-07-31 08:41:24 TP13 EP1] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results! +[2026-07-31 08:41:24 TP13 EP1] Load weight end. elapsed=104.14 s, type=DeepseekV4ForCausalLM, quant=fp8, fmt=e4m3, avail mem=25.23 GB, mem usage=56.87 GB. +[2026-07-31 08:41:24 TP10 EP1] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results! +[2026-07-31 08:41:24 TP10 EP1] Load weight end. elapsed=104.26 s, type=DeepseekV4ForCausalLM, quant=fp8, fmt=e4m3, avail mem=25.24 GB, mem usage=56.87 GB. +[2026-07-31 08:41:24 TP11 EP1] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results! +[2026-07-31 08:41:24 TP11 EP1] Load weight end. elapsed=104.27 s, type=DeepseekV4ForCausalLM, quant=fp8, fmt=e4m3, avail mem=25.16 GB, mem usage=56.87 GB. +[2026-07-31 08:41:25 TP9 EP1] Using FP8 KV cache but no scaling factors provided. Defaulting to scaling factors of 1.0. This may lead to less accurate results! +[2026-07-31 08:41:25 TP9 EP1] Load weight end. elapsed=105.31 s, type=DeepseekV4ForCausalLM, quant=fp8, fmt=e4m3, avail mem=25.23 GB, mem usage=56.87 GB. +[2026-07-31 08:41:56 TP12 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:41:56 TP8 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:41:56 TP9 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:41:56 TP10 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:41:56 TP15 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:41:56 TP13 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:41:56 TP11 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:41:56 TP14 EP1] Tokenizer for /data/hf_models/DeepSeek-V4-Pro is still TokenizersBackend after retries with --trust-remote-code. Model-specific tokenizer attributes may be missing. +[2026-07-31 08:41:56 TP14 EP1] DSV4 memory calculation: bytes_per_full_token=10993.84, available_bytes=16.93 GB, c128_state_fixed=3.92 GB, full_token=1270016 +[2026-07-31 08:41:56 TP15 EP1] DSV4 memory calculation: bytes_per_full_token=10993.84, available_bytes=16.93 GB, c128_state_fixed=3.92 GB, full_token=1270016 +[2026-07-31 08:41:56 TP13 EP1] DSV4 memory calculation: bytes_per_full_token=10993.84, available_bytes=16.93 GB, c128_state_fixed=3.92 GB, full_token=1270016 +[2026-07-31 08:41:56 TP14 EP1] DSV4 pool sizes: full=1270016, swa=126976, c4=317504, c128=9922, c4_state=7936, c128_state=0 +[2026-07-31 08:41:56 TP13 EP1] DSV4 pool sizes: full=1270016, swa=126976, c4=317504, c128=9922, c4_state=7936, c128_state=0 +[2026-07-31 08:41:56 TP15 EP1] DSV4 pool sizes: full=1270016, swa=126976, c4=317504, c128=9922, c4_state=7936, c128_state=0 +[2026-07-31 08:41:56 TP11 EP1] DSV4 memory calculation: bytes_per_full_token=10993.84, available_bytes=16.93 GB, c128_state_fixed=3.92 GB, full_token=1270016 +[2026-07-31 08:41:56 TP10 EP1] DSV4 memory calculation: bytes_per_full_token=10993.84, available_bytes=16.93 GB, c128_state_fixed=3.92 GB, full_token=1270016 +[2026-07-31 08:41:56 TP11 EP1] DSV4 pool sizes: full=1270016, swa=126976, c4=317504, c128=9922, c4_state=7936, c128_state=0 +[2026-07-31 08:41:56 TP10 EP1] DSV4 pool sizes: full=1270016, swa=126976, c4=317504, c128=9922, c4_state=7936, c128_state=0 +[2026-07-31 08:41:56 TP12 EP1] DSV4 memory calculation: bytes_per_full_token=10993.84, available_bytes=16.93 GB, c128_state_fixed=3.92 GB, full_token=1270016 +[2026-07-31 08:41:56 TP8 EP1] DSV4 memory calculation: bytes_per_full_token=10993.84, available_bytes=16.93 GB, c128_state_fixed=3.92 GB, full_token=1270016 +[2026-07-31 08:41:56 TP9 EP1] DSV4 memory calculation: bytes_per_full_token=10993.84, available_bytes=16.93 GB, c128_state_fixed=3.92 GB, full_token=1270016 +[2026-07-31 08:41:56 TP12 EP1] DSV4 pool sizes: full=1270016, swa=126976, c4=317504, c128=9922, c4_state=7936, c128_state=0 +[2026-07-31 08:41:56 TP9 EP1] DSV4 pool sizes: full=1270016, swa=126976, c4=317504, c128=9922, c4_state=7936, c128_state=0 +[2026-07-31 08:41:56 TP8 EP1] DSV4 pool sizes: full=1270016, swa=126976, c4=317504, c128=9922, c4_state=7936, c128_state=0 +[2026-07-31 08:41:56 TP13 EP1] Initialize DeepSeekV4TokenToKVPool with max_num_reqs=256 swa_size=126976 c4_size=317504 c4_logical_size=317504 c128_size=9922 c4_state_pool_size=7936 c128_state_pool_size=32896 +[2026-07-31 08:41:56 TP9 EP1] Initialize DeepSeekV4TokenToKVPool with max_num_reqs=256 swa_size=126976 c4_size=317504 c4_logical_size=317504 c128_size=9922 c4_state_pool_size=7936 c128_state_pool_size=32896 +[2026-07-31 08:41:56 TP8 EP1] Initialize DeepSeekV4TokenToKVPool with max_num_reqs=256 swa_size=126976 c4_size=317504 c4_logical_size=317504 c128_size=9922 c4_state_pool_size=7936 c128_state_pool_size=32896 +[2026-07-31 08:41:56 TP10 EP1] Initialize DeepSeekV4TokenToKVPool with max_num_reqs=256 swa_size=126976 c4_size=317504 c4_logical_size=317504 c128_size=9922 c4_state_pool_size=7936 c128_state_pool_size=32896 +[2026-07-31 08:41:56 TP11 EP1] Initialize DeepSeekV4TokenToKVPool with max_num_reqs=256 swa_size=126976 c4_size=317504 c4_logical_size=317504 c128_size=9922 c4_state_pool_size=7936 c128_state_pool_size=32896 +[2026-07-31 08:41:56 TP12 EP1] Initialize DeepSeekV4TokenToKVPool with max_num_reqs=256 swa_size=126976 c4_size=317504 c4_logical_size=317504 c128_size=9922 c4_state_pool_size=7936 c128_state_pool_size=32896 +[2026-07-31 08:41:56 TP14 EP1] Initialize DeepSeekV4TokenToKVPool with max_num_reqs=256 swa_size=126976 c4_size=317504 c4_logical_size=317504 c128_size=9922 c4_state_pool_size=7936 c128_state_pool_size=32896 +[2026-07-31 08:41:56 TP15 EP1] Initialize DeepSeekV4TokenToKVPool with max_num_reqs=256 swa_size=126976 c4_size=317504 c4_logical_size=317504 c128_size=9922 c4_state_pool_size=7936 c128_state_pool_size=32896 +[2026-07-31 08:41:58 TP12 EP1] Memory pool end. avail mem=7.14 GB +[2026-07-31 08:41:58 TP10 EP1] Memory pool end. avail mem=7.18 GB +[2026-07-31 08:41:58 TP8 EP1] Memory pool end. avail mem=7.10 GB +[2026-07-31 08:41:58 TP9 EP1] Memory pool end. avail mem=7.16 GB +[2026-07-31 08:41:58 TP13 EP1] Memory pool end. avail mem=7.16 GB +[2026-07-31 08:41:58 TP11 EP1] Memory pool end. avail mem=7.09 GB +[2026-07-31 08:41:58 TP15 EP1] Memory pool end. avail mem=7.13 GB +[2026-07-31 08:41:58 TP14 EP1] Memory pool end. avail mem=7.18 GB +[2026-07-31 08:41:58 TP8 EP1] Using DeepseekV4AttnBackend for dsv4 attention backend (CUDA). +[2026-07-31 08:41:58 TP12 EP1] Using DeepseekV4AttnBackend for dsv4 attention backend (CUDA). +[2026-07-31 08:41:58 TP13 EP1] Using DeepseekV4AttnBackend for dsv4 attention backend (CUDA). +[2026-07-31 08:41:58 TP10 EP1] Using DeepseekV4AttnBackend for dsv4 attention backend (CUDA). +[2026-07-31 08:41:58 TP15 EP1] Using DeepseekV4AttnBackend for dsv4 attention backend (CUDA). +[2026-07-31 08:41:58 TP9 EP1] Using DeepseekV4AttnBackend for dsv4 attention backend (CUDA). +[2026-07-31 08:41:58 TP8 EP1] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.14/sm120/4f24e70399961c55/rank_tp8_pp0_dp0.json +[2026-07-31 08:41:58 TP12 EP1] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.14/sm120/4f24e70399961c55/rank_tp12_pp0_dp0.json +[2026-07-31 08:41:58 TP13 EP1] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.14/sm120/4f24e70399961c55/rank_tp13_pp0_dp0.json +[2026-07-31 08:41:58 TP10 EP1] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.14/sm120/4f24e70399961c55/rank_tp10_pp0_dp0.json +[2026-07-31 08:41:58 TP15 EP1] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.14/sm120/4f24e70399961c55/rank_tp15_pp0_dp0.json +[2026-07-31 08:41:58 TP9 EP1] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.14/sm120/4f24e70399961c55/rank_tp9_pp0_dp0.json +[2026-07-31 08:41:58 TP11 EP1] Using DeepseekV4AttnBackend for dsv4 attention backend (CUDA). +[2026-07-31 08:41:58 TP14 EP1] Using DeepseekV4AttnBackend for dsv4 attention backend (CUDA). +[2026-07-31 08:41:58 TP11 EP1] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.14/sm120/4f24e70399961c55/rank_tp11_pp0_dp0.json +[2026-07-31 08:41:58 TP14 EP1] Running FlashInfer autotune with cache: /root/.cache/sglang/flashinfer/autotune/0.6.14/sm120/4f24e70399961c55/rank_tp14_pp0_dp0.json +gpu3:236:236 [0] NCCL INFO Comm config Blocking set to 1 +gpu3:242:242 [6] NCCL INFO Comm config Blocking set to 1 +gpu3:237:237 [1] NCCL INFO Comm config Blocking set to 1 +gpu3:241:241 [5] NCCL INFO Comm config Blocking set to 1 +gpu3:240:240 [4] NCCL INFO Comm config Blocking set to 1 +gpu3:243:243 [7] NCCL INFO Comm config Blocking set to 1 +gpu3:238:238 [2] NCCL INFO Comm config Blocking set to 1 +gpu3:236:236 [0] NCCL INFO Initialized NET plugin IB +gpu3:236:236 [0] NCCL INFO Assigned NET plugin IB to comm +gpu3:236:236 [0] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:236:236 [0] NCCL INFO Using network IB +gpu3:239:239 [3] NCCL INFO Comm config Blocking set to 1 +gpu3:236:236 [0] NCCL INFO DMA-BUF is available on GPU device 0 +gpu3:236:236 [0] NCCL INFO ncclCommInitRankConfig comm 0x3cb9f1f0 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 19000 commId 0x1a548291cb460838 - Init START +gpu3:242:242 [6] NCCL INFO Initialized NET plugin IB +gpu3:242:242 [6] NCCL INFO Assigned NET plugin IB to comm +gpu3:242:242 [6] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:242:242 [6] NCCL INFO Using network IB +gpu3:243:243 [7] NCCL INFO Initialized NET plugin IB +gpu3:242:242 [6] NCCL INFO DMA-BUF is available on GPU device 6 +gpu3:243:243 [7] NCCL INFO Assigned NET plugin IB to comm +gpu3:243:243 [7] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:243:243 [7] NCCL INFO Using network IB +gpu3:241:241 [5] NCCL INFO Initialized NET plugin IB +gpu3:241:241 [5] NCCL INFO Assigned NET plugin IB to comm +gpu3:241:241 [5] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:241:241 [5] NCCL INFO Using network IB +gpu3:243:243 [7] NCCL INFO DMA-BUF is available on GPU device 7 +gpu3:241:241 [5] NCCL INFO DMA-BUF is available on GPU device 5 +gpu3:238:238 [2] NCCL INFO Initialized NET plugin IB +gpu3:238:238 [2] NCCL INFO Assigned NET plugin IB to comm +gpu3:238:238 [2] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:238:238 [2] NCCL INFO Using network IB +gpu3:243:243 [7] NCCL INFO ncclCommInitRankConfig comm 0x47e41d60 rank 15 nranks 16 cudaDev 7 nvmlDev 7 busId af000 commId 0x1a548291cb460838 - Init START +gpu3:242:242 [6] NCCL INFO ncclCommInitRankConfig comm 0x8056cf50 rank 14 nranks 16 cudaDev 6 nvmlDev 6 busId ae000 commId 0x1a548291cb460838 - Init START +gpu3:238:238 [2] NCCL INFO DMA-BUF is available on GPU device 2 +gpu3:241:241 [5] NCCL INFO ncclCommInitRankConfig comm 0x47e6a3b0 rank 13 nranks 16 cudaDev 5 nvmlDev 5 busId ab000 commId 0x1a548291cb460838 - Init START +gpu3:239:239 [3] NCCL INFO Initialized NET plugin IB +gpu3:239:239 [3] NCCL INFO Assigned NET plugin IB to comm +gpu3:239:239 [3] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:239:239 [3] NCCL INFO Using network IB +gpu3:239:239 [3] NCCL INFO DMA-BUF is available on GPU device 3 +gpu3:238:238 [2] NCCL INFO ncclCommInitRankConfig comm 0x6968cfd0 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 1b000 commId 0x1a548291cb460838 - Init START +gpu3:239:239 [3] NCCL INFO ncclCommInitRankConfig comm 0x6c6a8260 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId 1d000 commId 0x1a548291cb460838 - Init START +gpu3:237:237 [1] NCCL INFO Initialized NET plugin IB +gpu3:237:237 [1] NCCL INFO Assigned NET plugin IB to comm +gpu3:237:237 [1] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:237:237 [1] NCCL INFO Using network IB +gpu3:237:237 [1] NCCL INFO DMA-BUF is available on GPU device 1 +gpu3:240:240 [4] NCCL INFO Initialized NET plugin IB +gpu3:240:240 [4] NCCL INFO Assigned NET plugin IB to comm +gpu3:240:240 [4] NCCL INFO Assigned GIN plugin GIN_IB_GDAKI to comm +gpu3:240:240 [4] NCCL INFO Using network IB +gpu3:237:237 [1] NCCL INFO ncclCommInitRankConfig comm 0x47b20f80 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 1a000 commId 0x1a548291cb460838 - Init START +gpu3:240:240 [4] NCCL INFO DMA-BUF is available on GPU device 4 +gpu3:240:240 [4] NCCL INFO ncclCommInitRankConfig comm 0x4600e310 rank 12 nranks 16 cudaDev 4 nvmlDev 4 busId aa000 commId 0x1a548291cb460838 - Init START +gpu3:241:241 [5] NCCL INFO Bootstrap timings total 0.004126 (create 0.000055, send 0.000268, recv 0.000824, ring 0.000884, delay 0.000000) +gpu3:240:240 [4] NCCL INFO Bootstrap timings total 0.002172 (create 0.000058, send 0.000273, recv 0.000651, ring 0.000845, delay 0.000000) +gpu3:238:238 [2] NCCL INFO Bootstrap timings total 0.003807 (create 0.000051, send 0.000256, recv 0.001579, ring 0.001224, delay 0.000000) +gpu3:242:242 [6] NCCL INFO Bootstrap timings total 0.004323 (create 0.000059, send 0.000289, recv 0.000753, ring 0.002660, delay 0.000000) +gpu3:239:239 [3] NCCL INFO Bootstrap timings total 0.003450 (create 0.000055, send 0.000226, recv 0.001817, ring 0.001018, delay 0.000000) +gpu3:243:243 [7] NCCL INFO Bootstrap timings total 0.004351 (create 0.000051, send 0.000277, recv 0.001162, ring 0.002505, delay 0.000000) +gpu3:237:237 [1] NCCL INFO Bootstrap timings total 0.002902 (create 0.000057, send 0.000215, recv 0.001074, ring 0.001208, delay 0.000000) +gpu3:236:236 [0] NCCL INFO Bootstrap timings total 0.005208 (create 0.000054, send 0.000292, recv 0.003038, ring 0.001523, delay 0.000000) +gpu3:236:236 [0] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 0 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu3:237:237 [1] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 1 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu3:239:239 [3] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 3 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu3:242:242 [6] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 6 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu3:243:243 [7] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 7 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu3:238:238 [2] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 2 is 0-31,64-95. (GPU affinity = 0-31,64-95 ; CPU affinity = 0-127). +gpu3:241:241 [5] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 5 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu3:240:240 [4] NCCL INFO ncclTopoGetCpuAffinity: Affinity for GPU 4 is 32-63,96-127. (GPU affinity = 32-63,96-127 ; CPU affinity = 0-127). +gpu3:241:241 [5] NCCL INFO comm 0x47e6a3b0 rank 13 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 +gpu3:240:240 [4] NCCL INFO comm 0x4600e310 rank 12 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 +gpu3:241:241 [5] NCCL INFO Trees [0] 14/-1/-1->13->12 [1] 14/-1/-1->13->15 [2] 14/-1/-1->13->12 [3] 14/-1/-1->13->15 +gpu3:241:241 [5] NCCL INFO P2P Chunksize set to 131072 +gpu3:239:239 [3] NCCL INFO comm 0x6c6a8260 rank 11 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 +gpu3:242:242 [6] NCCL INFO comm 0x8056cf50 rank 14 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 +gpu3:241:241 [5] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:240:240 [4] NCCL INFO Trees [0] 13/-1/-1->12->8 [1] -1/-1/-1->12->14 [2] 13/4/-1->12->-1 [3] -1/-1/-1->12->14 +gpu3:243:243 [7] NCCL INFO comm 0x47e41d60 rank 15 nRanks 16 nNodes 4 localRanks 4 localRank 3 MNNVL 0 +gpu3:240:240 [4] NCCL INFO P2P Chunksize set to 131072 +gpu3:240:240 [4] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:242:242 [6] NCCL INFO Trees [0] 15/-1/-1->14->13 [1] 12/-1/-1->14->13 [2] 15/-1/-1->14->13 [3] 12/-1/-1->14->13 +gpu3:242:242 [6] NCCL INFO P2P Chunksize set to 131072 +gpu3:238:238 [2] NCCL INFO comm 0x6968cfd0 rank 10 nRanks 16 nNodes 4 localRanks 4 localRank 2 MNNVL 0 +gpu3:239:239 [3] NCCL INFO Trees [0] -1/-1/-1->11->10 [1] 9/7/15->11->3 [2] -1/-1/-1->11->10 [3] 9/-1/-1->11->7 +gpu3:243:243 [7] NCCL INFO Trees [0] -1/-1/-1->15->14 [1] 13/-1/-1->15->11 [2] -1/-1/-1->15->14 [3] 13/7/-1->15->-1 +gpu3:239:239 [3] NCCL INFO P2P Chunksize set to 131072 +gpu3:242:242 [6] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:243:243 [7] NCCL INFO P2P Chunksize set to 131072 +gpu3:243:243 [7] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:239:239 [3] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:238:238 [2] NCCL INFO Trees [0] 11/-1/-1->10->9 [1] 8/-1/-1->10->9 [2] 11/-1/-1->10->9 [3] 8/-1/-1->10->9 +gpu3:238:238 [2] NCCL INFO P2P Chunksize set to 131072 +gpu3:238:238 [2] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:236:236 [0] NCCL INFO comm 0x3cb9f1f0 rank 8 nRanks 16 nNodes 4 localRanks 4 localRank 0 MNNVL 0 +gpu3:237:237 [1] NCCL INFO comm 0x47b20f80 rank 9 nRanks 16 nNodes 4 localRanks 4 localRank 1 MNNVL 0 +gpu3:236:236 [0] NCCL INFO Trees [0] 9/4/12->8->0 [1] -1/-1/-1->8->10 [2] 9/-1/-1->8->4 [3] -1/-1/-1->8->10 +gpu3:236:236 [0] NCCL INFO P2P Chunksize set to 131072 +gpu3:236:236 [0] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:237:237 [1] NCCL INFO Trees [0] 10/-1/-1->9->8 [1] 10/-1/-1->9->11 [2] 10/-1/-1->9->8 [3] 10/-1/-1->9->11 +gpu3:237:237 [1] NCCL INFO P2P Chunksize set to 131072 +gpu3:237:237 [1] NCCL INFO Check P2P Type isAllDirectP2p 1 directMode 0 isAllCudaP2p 1 +gpu3:242:2233 [0] NCCL INFO [Proxy Service] Device 6 CPU core 39 +gpu3:243:2234 [0] NCCL INFO [Proxy Service] Device 7 CPU core 58 +gpu3:239:2239 [0] NCCL INFO [Proxy Service UDS] Device 3 CPU core 66 +gpu3:241:2231 [0] NCCL INFO [Proxy Service] Device 5 CPU core 99 +gpu3:238:2236 [0] NCCL INFO [Proxy Service] Device 2 CPU core 69 +gpu3:242:2238 [0] NCCL INFO [Proxy Service UDS] Device 6 CPU core 104 +gpu3:237:2243 [0] NCCL INFO [Proxy Service] Device 1 CPU core 72 +gpu3:239:2235 [0] NCCL INFO [Proxy Service] Device 3 CPU core 65 +gpu3:243:2241 [0] NCCL INFO [Proxy Service UDS] Device 7 CPU core 123 +gpu3:236:2242 [0] NCCL INFO [Proxy Service] Device 0 CPU core 7 +gpu3:241:2240 [0] NCCL INFO [Proxy Service UDS] Device 5 CPU core 41 +gpu3:240:2237 [0] NCCL INFO [Proxy Service UDS] Device 4 CPU core 100 +gpu3:236:2245 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 10 +gpu3:237:2246 [0] NCCL INFO [Proxy Service UDS] Device 1 CPU core 11 +gpu3:240:2232 [0] NCCL INFO [Proxy Service] Device 4 CPU core 49 +gpu3:238:2244 [0] NCCL INFO [Proxy Service UDS] Device 2 CPU core 73 +gpu3:242:242 [6] NCCL INFO Channel 00/0 : 14[6] -> 15[7] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 00/0 : 10[2] -> 11[3] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 00/0 : 9[1] -> 10[2] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 00/0 : 13[5] -> 14[6] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 01/0 : 10[2] -> 11[3] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 01/0 : 9[1] -> 10[2] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 01/0 : 14[6] -> 15[7] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 01/0 : 13[5] -> 14[6] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 02/0 : 10[2] -> 11[3] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 02/0 : 9[1] -> 10[2] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 02/0 : 14[6] -> 15[7] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 02/0 : 13[5] -> 14[6] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 03/0 : 10[2] -> 11[3] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 03/0 : 9[1] -> 10[2] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 03/0 : 13[5] -> 14[6] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 03/0 : 14[6] -> 15[7] via P2P/IPC +gpu3:236:2247 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 72 +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 01/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 02/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 03/0 : 7[7] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 8[0] -> 9[1] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 01/0 : 8[0] -> 9[1] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 02/0 : 8[0] -> 9[1] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 03/0 : 8[0] -> 9[1] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 00/0 : 11[3] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:239:2248 [0] NCCL INFO [Proxy Progress] Device 3 CPU core 13 +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 02/0 : 11[3] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 03/0 : 11[3] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:243:2249 [0] NCCL INFO [Proxy Progress] Device 7 CPU core 125 +gpu3:243:243 [7] NCCL INFO Channel 00/0 : 15[7] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 01/0 : 15[7] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 02/0 : 15[7] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 03/0 : 15[7] -> 0[0] [send] via NET/IB/1/GDRDMA +gpu3:240:2250 [0] NCCL INFO [Proxy Progress] Device 4 CPU core 115 +gpu3:240:240 [4] NCCL INFO Channel 00/0 : 11[3] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 01/0 : 11[3] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 02/0 : 11[3] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 03/0 : 11[3] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 00/0 : 12[4] -> 13[5] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 01/0 : 12[4] -> 13[5] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 02/0 : 12[4] -> 13[5] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 03/0 : 12[4] -> 13[5] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:243:243 [7] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:241:241 [5] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:240:240 [4] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:241:241 [5] NCCL INFO Channel 01/0 : 13[5] -> 15[7] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:240:240 [4] NCCL INFO Channel 01/0 : 12[4] -> 14[6] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:237:237 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:236:236 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 +gpu3:237:237 [1] NCCL INFO Channel 01/0 : 9[1] -> 11[3] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 01/0 : 8[0] -> 10[2] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 03/0 : 12[4] -> 14[6] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 03/0 : 13[5] -> 15[7] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 03/0 : 8[0] -> 10[2] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 01/0 : 14[6] -> 12[4] via P2P/IPC +gpu3:243:243 [7] NCCL INFO Channel 01/0 : 11[3] -> 15[7] [receive] via NET/IB/1/GDRDMA +gpu3:238:238 [2] NCCL INFO Channel 01/0 : 10[2] -> 8[0] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 00/0 : 8[0] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 4[4] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 02/0 : 4[4] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:238:238 [2] NCCL INFO Channel 03/0 : 10[2] -> 8[0] via P2P/IPC +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 8[0] -> 12[4] [send] via NET/IB/0/GDRDMA +gpu3:242:242 [6] NCCL INFO Channel 03/0 : 14[6] -> 12[4] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 02/0 : 4[4] -> 12[4] [receive] via NET/IB/1/GDRDMA +gpu3:237:237 [1] NCCL INFO Channel 03/0 : 9[1] -> 11[3] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Channel 02/0 : 12[4] -> 4[4] [send] via NET/IB/1/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 7[7] -> 11[3] [receive] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 03/0 : 7[7] -> 11[3] [receive] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 15[7] [send] via NET/IB/0/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 03/0 : 7[7] -> 15[7] [receive] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 03/0 : 15[7] -> 7[7] [send] via NET/IB/1/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 0[0] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 8[0] -> 0[0] [send] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 3[3] -> 11[3] [receive] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 3[3] [send] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 12[4] -> 8[0] [receive] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 15[7] -> 11[3] [receive] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 00/0 : 8[0] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 7[7] [send] via NET/IB/0/GDRDMA +gpu3:236:236 [0] NCCL INFO Channel 02/0 : 8[0] -> 4[4] [send] via NET/IB/0/GDRDMA +gpu3:239:239 [3] NCCL INFO Channel 03/0 : 11[3] -> 7[7] [send] via NET/IB/0/GDRDMA +gpu3:240:240 [4] NCCL INFO Channel 00/0 : 12[4] -> 8[0] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 01/0 : 15[7] -> 11[3] [send] via NET/IB/1/GDRDMA +gpu3:243:243 [7] NCCL INFO Channel 01/0 : 15[7] -> 13[5] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 01/0 : 11[3] -> 9[1] via P2P/IPC +gpu3:243:243 [7] NCCL INFO Channel 03/0 : 15[7] -> 13[5] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 03/0 : 11[3] -> 9[1] via P2P/IPC +gpu3:243:243 [7] NCCL INFO Channel 00/0 : 15[7] -> 14[6] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 00/0 : 11[3] -> 10[2] via P2P/IPC +gpu3:239:239 [3] NCCL INFO Channel 02/0 : 11[3] -> 10[2] via P2P/IPC +gpu3:243:243 [7] NCCL INFO Channel 02/0 : 15[7] -> 14[6] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 00/0 : 10[2] -> 9[1] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 00/0 : 14[6] -> 13[5] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 01/0 : 10[2] -> 9[1] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 01/0 : 14[6] -> 13[5] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 02/0 : 10[2] -> 9[1] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 02/0 : 14[6] -> 13[5] via P2P/IPC +gpu3:238:238 [2] NCCL INFO Channel 03/0 : 10[2] -> 9[1] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 00/0 : 9[1] -> 8[0] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 00/0 : 13[5] -> 12[4] via P2P/IPC +gpu3:242:242 [6] NCCL INFO Channel 03/0 : 14[6] -> 13[5] via P2P/IPC +gpu3:241:241 [5] NCCL INFO Channel 02/0 : 13[5] -> 12[4] via P2P/IPC +gpu3:237:237 [1] NCCL INFO Channel 02/0 : 9[1] -> 8[0] via P2P/IPC +gpu3:240:240 [4] NCCL INFO Connected all trees +gpu3:241:241 [5] NCCL INFO Connected all trees +gpu3:243:243 [7] NCCL INFO Connected all trees +gpu3:242:242 [6] NCCL INFO Connected all trees +gpu3:236:236 [0] NCCL INFO Connected all trees +gpu3:237:237 [1] NCCL INFO Connected all trees +gpu3:239:239 [3] NCCL INFO Connected all trees +gpu3:238:238 [2] NCCL INFO Connected all trees +gpu3:236:236 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:236:236 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:240:240 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:240:240 [4] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:239:239 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:239:239 [3] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:243:243 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:243:243 [7] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:242:2251 [0] NCCL INFO [Proxy Progress] Device 6 CPU core 42 +gpu3:242:242 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:242:242 [6] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:238:2252 [0] NCCL INFO [Proxy Progress] Device 2 CPU core 14 +gpu3:237:2253 [0] NCCL INFO [Proxy Progress] Device 1 CPU core 79 +gpu3:241:2254 [0] NCCL INFO [Proxy Progress] Device 5 CPU core 107 +gpu3:241:241 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:241:241 [5] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:238:238 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:238:238 [2] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:237:237 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512 +gpu3:237:237 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 1 p2p channels per peer +gpu3:242:242 [6] NCCL INFO ncclCommInitRankConfig comm 0x8056cf50 rank 14 nranks 16 cudaDev 6 nvmlDev 6 busId ae000 commId 0x1a548291cb460838 - Init COMPLETE +gpu3:240:240 [4] NCCL INFO ncclCommInitRankConfig comm 0x4600e310 rank 12 nranks 16 cudaDev 4 nvmlDev 4 busId aa000 commId 0x1a548291cb460838 - Init COMPLETE +gpu3:243:243 [7] NCCL INFO ncclCommInitRankConfig comm 0x47e41d60 rank 15 nranks 16 cudaDev 7 nvmlDev 7 busId af000 commId 0x1a548291cb460838 - Init COMPLETE +gpu3:241:241 [5] NCCL INFO ncclCommInitRankConfig comm 0x47e6a3b0 rank 13 nranks 16 cudaDev 5 nvmlDev 5 busId ab000 commId 0x1a548291cb460838 - Init COMPLETE +gpu3:242:242 [6] NCCL INFO Init timings - ncclCommInitRankConfig: rank 14 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.05, topo 0.05, graphs 0.01, connections 0.20, rest 0.01) +gpu3:240:240 [4] NCCL INFO Init timings - ncclCommInitRankConfig: rank 12 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.05, topo 0.05, graphs 0.01, connections 0.16, rest 0.05) +gpu3:243:243 [7] NCCL INFO Init timings - ncclCommInitRankConfig: rank 15 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.05, topo 0.05, graphs 0.01, connections 0.16, rest 0.05) +gpu3:241:241 [5] NCCL INFO Init timings - ncclCommInitRankConfig: rank 13 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.05, topo 0.05, graphs 0.01, connections 0.21, rest 0.00) +gpu3:238:238 [2] NCCL INFO ncclCommInitRankConfig comm 0x6968cfd0 rank 10 nranks 16 cudaDev 2 nvmlDev 2 busId 1b000 commId 0x1a548291cb460838 - Init COMPLETE +gpu3:236:236 [0] NCCL INFO ncclCommInitRankConfig comm 0x3cb9f1f0 rank 8 nranks 16 cudaDev 0 nvmlDev 0 busId 19000 commId 0x1a548291cb460838 - Init COMPLETE +gpu3:238:238 [2] NCCL INFO Init timings - ncclCommInitRankConfig: rank 10 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.05, topo 0.05, graphs 0.01, connections 0.21, rest 0.00) +gpu3:239:239 [3] NCCL INFO ncclCommInitRankConfig comm 0x6c6a8260 rank 11 nranks 16 cudaDev 3 nvmlDev 3 busId 1d000 commId 0x1a548291cb460838 - Init COMPLETE +gpu3:236:236 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 8 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.01, allgathers 0.06, topo 0.05, graphs 0.01, connections 0.16, rest 0.05) +gpu3:237:237 [1] NCCL INFO ncclCommInitRankConfig comm 0x47b20f80 rank 9 nranks 16 cudaDev 1 nvmlDev 1 busId 1a000 commId 0x1a548291cb460838 - Init COMPLETE +gpu3:239:239 [3] NCCL INFO Init timings - ncclCommInitRankConfig: rank 11 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.05, topo 0.05, graphs 0.01, connections 0.16, rest 0.05) +gpu3:237:237 [1] NCCL INFO Init timings - ncclCommInitRankConfig: rank 9 nranks 16 total 0.32 (kernels 0.00, alloc 0.00, bootstrap 0.00, allgathers 0.06, topo 0.05, graphs 0.01, connections 0.21, rest 0.00) +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[TensorRT-LLM][INFO] Set logger level to INFO +[2026-07-31 08:42:26 TP15 EP1] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP15 EP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP14 EP1] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP14 EP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP9 EP1] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP9 EP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP15 EP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.77 GB +[2026-07-31 08:42:26 TP14 EP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.86 GB +[2026-07-31 08:42:26 TP9 EP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.83 GB +[2026-07-31 08:42:26 TP8 EP1] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP8 EP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP12 EP1] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP12 EP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP13 EP1] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP13 EP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP10 EP1] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP10 EP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP8 EP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.72 GB +[2026-07-31 08:42:26 TP11 EP1] FlashInfer autotune completed. +[2026-07-31 08:42:26 TP11 EP1] Disable prefill CUDA graph because cuda_graph_config resolved prefill.backend='disabled' (e.g. via --cuda-graph-backend-prefill=disabled or auto-disable rules). +[2026-07-31 08:42:26 TP12 EP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.80 GB +[2026-07-31 08:42:26 TP13 EP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.83 GB +[2026-07-31 08:42:26 TP10 EP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.86 GB +[2026-07-31 08:42:26 TP11 EP1] Capture target decode CUDA graph begin. backend=full, num_tokens_per_req=1, bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64], avail mem=6.69 GB +/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. + warnings.warn( +/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. + warnings.warn( +/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. + warnings.warn( +/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. + warnings.warn( +/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. + warnings.warn( +/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. + warnings.warn( +/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. + warnings.warn( +/usr/local/lib/python3.12/dist-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. + warnings.warn( +[2026-07-31 08:43:55 TP15 EP1] Capture target decode CUDA graph end. elapsed=88.99 s, mem usage=1.37 GB, avail mem=5.40 GB. +[2026-07-31 08:43:55 TP15 EP1] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP15 EP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP14 EP1] Capture target decode CUDA graph end. elapsed=88.99 s, mem usage=1.37 GB, avail mem=5.50 GB. +[2026-07-31 08:43:55 TP14 EP1] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP14 EP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP13 EP1] Capture target decode CUDA graph end. elapsed=88.99 s, mem usage=1.37 GB, avail mem=5.46 GB. +[2026-07-31 08:43:55 TP8 EP1] Capture target decode CUDA graph end. elapsed=88.99 s, mem usage=1.37 GB, avail mem=5.35 GB. +[2026-07-31 08:43:55 TP13 EP1] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP8 EP1] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP13 EP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP8 EP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP12 EP1] Capture target decode CUDA graph end. elapsed=89.00 s, mem usage=1.37 GB, avail mem=5.43 GB. +[2026-07-31 08:43:55 TP12 EP1] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP12 EP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP10 EP1] Capture target decode CUDA graph end. elapsed=89.00 s, mem usage=1.37 GB, avail mem=5.50 GB. +[2026-07-31 08:43:55 TP10 EP1] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP10 EP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP9 EP1] Capture target decode CUDA graph end. elapsed=89.02 s, mem usage=1.37 GB, avail mem=5.46 GB. +[2026-07-31 08:43:55 TP9 EP1] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP9 EP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:55 TP11 EP1] Capture target decode CUDA graph end. elapsed=89.04 s, mem usage=1.37 GB, avail mem=5.32 GB. +[2026-07-31 08:43:55 TP11 EP1] Init Unified RadixTree with components (, ) +[2026-07-31 08:43:55 TP11 EP1] Tree cache initialized: source=default impl=UnifiedRadixCache hybrid_swa=True hybrid_ssm=False hierarchical=False streaming_wrapped=False +[2026-07-31 08:43:56] Dummy health check server started in background thread at 0.0.0.0:30002 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/worker_server_cmd.txt b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/worker_server_cmd.txt new file mode 100644 index 0000000..3b18997 --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/service/worker_server_cmd.txt @@ -0,0 +1 @@ +docker run -d --name dsv4pro_pro6000d_2node_sglang_tp16_quick_map_worker --gpus all --network host --ipc host --shm-size 20g --ulimit memlock=-1 --ulimit stack=67108864 -v /data/hf_models/DeepSeek-V4-Pro:/data/hf_models/DeepSeek-V4-Pro:ro -v /data/hzy/sglang_cache/dsv4_pro_tp16:/root/.cache -e CUDA_VISIBLE_DEVICES=0\,1\,2\,3\,4\,5\,6\,7 -e PYTHONUNBUFFERED=1 -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e NCCL_SOCKET_IFNAME=eth0 -e NCCL_IB_HCA==mlx5_0:1\,mlx5_3:1 -e NCCL_CROSS_NIC=1 -e NCCL_DEBUG=INFO -e SGLANG_SHARED_EXPERT_TP1=1 --device /dev/infiniband/rdma_cm --device /dev/infiniband/uverbs0 --device /dev/infiniband/uverbs3 --entrypoint python3 lmsysorg/sglang:nightly-dev-cu13-20260720-b3570a45 -m sglang.launch_server --model-path /data/hf_models/DeepSeek-V4-Pro --tp-size 16 --ep-size 2 --nnodes 2 --node-rank 1 --dist-init-addr 10.101.0.11:20002 --trust-remote-code --host 0.0.0.0 --port 30002 --mem-fraction-static 0.9 --cuda-graph-max-bs-decode 64 --max-running-requests 256 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/summary.json b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/summary.json new file mode 100644 index 0000000..a86fd7c --- /dev/null +++ b/docs/dsv4pro_pro6000d_2node_sglang/results/dsv4pro-phase2-20260731-163620/summary.json @@ -0,0 +1,55 @@ +{ + "generated_at": "2026-07-31T17:05:04+08:00", + "bench_rows": 8, + "failed_bench_rows": 0, + "gpu_summary_rows": 16, + "rdma_summary_rows": 4, + "case_windows": 8, + "case_gpu_summary_rows": 128, + "case_rdma_summary_rows": 32, + "case_metric_rows": { + "gpu_node": 16, + "dcgm": 16, + "cpu": 16, + "process": 16, + "perf": 16, + "numa": 16, + "netdev": 32 + }, + "communication_aggregate_rows": 19, + "precise_windows": 8, + "collector_status_counts": { + "STARTED": 18, + "STOPPED": 18 + }, + "collector_files": { + "head": { + "dcgm_dmon.log": 1642103, + "docker_top.log": 2636664, + "gpu_samples.csv": 727301, + "markers.csv": 1860, + "mpstat.log": 3923940, + "numa_samples.csv": 11356, + "perf_stat.log": 188193, + "pidstat.log": 3070380, + "rdma.csv": 230893, + "sar_net.log": 534606, + "static_after.log": 1195708, + "static_before.log": 1306541 + }, + "worker": { + "dcgm_dmon.log": 1613150, + "docker_top.log": 2592658, + "gpu_samples.csv": 764832, + "markers.csv": 1902, + "mpstat.log": 3988645, + "numa_samples.csv": 12614, + "perf_stat.log": 191154, + "pidstat.log": 3024655, + "rdma.csv": 243227, + "sar_net.log": 609685, + "static_after.log": 1195276, + "static_before.log": 1306123 + } + } +} diff --git a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html index 79e1a71..93af13d 100644 --- a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html +++ b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.html @@ -442,14 +442,18 @@ DeepSeek-V4-Pro / 双机 Pro6000D / SGLang 硬件与资源竞争归因 -最终采集代码已完成；首轮 8/8 成功，Worker 分发修复与双节点通信 smoke test 已通过，等待最终正式复跑 -Phase 2 进行中；完成最终复跑与汇报后生成实验档案和代码详解 +Phase 2 已完成；最终 Run 8/8、精确窗口 8/8、采集器 18/18，双节点资源已清理 + +打开 Phase 2 实验档案
+打开 Phase 2 代码详解 +

-阶段档案生成门禁：Phase 在实验结束、结果汇总并完成汇报确认前， +阶段档案生成门禁：Phase 尚未产出正式阶段结果、完成汇总和汇报确认前， 不创建或维护 phaseN_exp.html 与 phaseN_code.html； -进行中只维护代码、原始结果和本页状态。阶段确认完成后再一次性生成两份最终 HTML， +此时只维护代码、原始结果和本页状态。已有正式阶段结果的 Phase 保留档案， +并在补测后更新；首次达到完成门槛时再一次性生成两份最终 HTML， 正文只保留成功实验、有效结果和结论，失败尝试压缩到末尾的经验教训。

0. 先看懂双机通信

@@ -486,7 +490,7 @@ GPU0–3 与 GPU4–7 各自在本地 PCIe Switch 内通信，两组之间还要 和 CPU/NUMA 互联。NCCL 机内日志中的 P2P/IPC 是 CUDA P2P over PCIe； 两机之间则使用 mlx5_0/mlx5_3 双 Rail NET/IB + GDRDMA。Phase 2 最终代码已加入机内 PCIe P2P 全矩阵、 -单机 8-rank 和双机 16-rank NCCL 微基准；正式数值在最终复跑后写入阶段档案。 +单机 8-rank 和双机 16-rank NCCL 微基准；正式数值已写入 Phase 2 实验档案。

1. 目标与原则

1.1 最终目标

@@ -736,21 +740,23 @@ TPOT P95 增加 66.55%。Phase 2 将围绕这两个现象采集硬件时间序 不伪装成当前已有能力；它们分别由后续硬件指标与 Profiling 阶段补齐。

6. Phase 2：同步采集轻量硬件指标

-本阶段尚在进行中，按照阶段档案生成门禁，暂不生成实验档案和代码详解 HTML。 -当前代码与原始结果保留在仓库实验目录；本阶段重放长 Prefill、并发 Prefill、 +本阶段已有正式 Run 结果，实验与代码说明见 +Phase 2 实验档案 和 +Phase 2 代码详解。本阶段重放长 Prefill、并发 Prefill、 普通 Decode、长输出 Decode、长上下文 Decode，以及 -1K → 1K, C=32 的混合 A/B。首轮 Run -dsv4pro-phase2-20260731-130125 在 26 分 26 秒内完成 8/8 个结果； -最终采集代码在提交 39fc2ba565a3 修复 Worker 分发，并由 -Run dsv4pro-phase2-stage-smoke-20260731-162326 完成双节点通信功能验证； -完成最终正式复跑和汇报后才进入 Phase 3。 +1K → 1K, C=32 的混合 A/B。最终 Run +dsv4pro-phase2-20260731-163620 在 28 分 44 秒内完成 8/8 个结果， +正式测量窗口 8/8 精确，18/18 个采集器正常启停；Head/Worker 的 DCGM、 +CPU、NUMA、RDMA 和通信微基准数据均有效。实验后两节点容器、端口和 +16 张 GPU 已清理，下一步进入 Phase 3。

-

6.1 首轮归因结果

+

6.1 最终归因结果

    -
  • 混合负载再次稳定复现：Decode Output TPS 下降 24.03%，TPOT P95 增加 66.79%，E2E P95 增加 63.30%。
    • -
  • 128K Prefill 注入窗口两节点 GPU 平均利用率约 99%，功耗约 274 W，频率稳定；显存每卡约 83.2–83.4 GiB，只剩约 2.3 GiB 余量。
    • -
  • 整机 CPU 平均约 8%–11%，没有全机 CPU 或 I/O Wait 饱和，但少量 CPU 核持续高负载，Scheduler/Affinity/NUMA 热点仍需关注。
    • -
  • 双 Rail 流量对称且错误增量为 0；最高平均总发送约 140 Gbit/s，即每条 400G Rail 约 70 Gbit/s，原始 RoCE 带宽未饱和。
    • +
  • 混合负载稳定复现：Decode Output TPS 下降 23.96%，TPOT P95 增加 66.75%，E2E P95 增加 63.38%。
    • +
  • GPU Util 多数为 94%–99%，频率稳定在 2.39–2.42 GHz；整机 CPU Active 约 9.6%–10.4%，没有全机 CPU 饱和或 GPU 降频。
    • +
  • 双 Rail 流量对称且错误增量为 0；最高约 83.5 Gbit/s/rail，仅约占单条 400G Rail 的 20.9%。
    • +
  • PCIe 跨 NUMA P2P 损失约 2.3%；16-GPU AllReduce busbw 约 39.3–39.7 GB/s。
    • +
  • NCCL_CROSS_NIC=0/1/2 差异小于 1%，不再作为主要调优方向。

最终代码已经把首轮的采集限制变成强制门禁：两节点 diff --git a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md index 46a842b..7ee5f4d 100644 --- a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md +++ b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md @@ -212,10 +212,21 @@ C = 16, 32, 48, 64, ... ## 6. Phase 2：同步采集硬件指标 -Phase 2 尚在进行中。按照阶段档案生成门禁，在最终正式复跑、结果汇总和汇报确认前， -不生成 `phase2_exp.html` 与 `phase2_code.html`。 -首轮 8/8 benchmark 已完成；最终采集代码、Worker 分发修复和双节点通信 -smoke test 已通过，等待最终正式复跑。 +Phase 2 已有正式阶段结果，详见 [Phase 2 实验档案](./phase2_exp.html) 和 +[Phase 2 代码详解](./phase2_code.html)。 +最终 Run `dsv4pro-phase2-20260731-163620` 在 28 分 44 秒内完成 8/8 +benchmark，正式测量窗口 8/8 精确，18/18 个采集器正常启停。两节点 +DCGM、CPU、NUMA、双 Rail RDMA 和通信微基准数据均有效，实验后容器、 +端口与 16 张 GPU 已清理。 + +最终归因： + +- 混合 Prefill/Decode 令 Output TPS 下降 23.96%、TPOT P95 增加 66.75%。 +- GPU 未降频、整机 CPU 未饱和，双 Rail 最高约 83.5 Gbit/s/rail 且零错误。 +- PCIe 跨 NUMA P2P 损失约 2.3%，16-GPU AllReduce busbw 约 39.3-39.7 GB/s。 +- `NCCL_CROSS_NIC=0/1/2` 差异小于 1%，不再作为主要调优方向。 +- Phase 3 只需捕获混合 Control/Treatment 的短 Timeline，定位 Kernel、 + Collective、Scheduler gap 与慢 Rank 同步。 Phase 2 只提供一个用户入口：