nsys 版本、子进程跟踪方式及动态 Capture 机制，再固化命令；
不直接对整轮 Benchmark 做长时间全量捕获。
-
+代码已集中在
+experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling，
+仍只有 run_timeline_profiling.sh 一个入口。双节点 PyTorch Profiler
+和 Nsight smoke 已通过；正式 Run 尚未执行，因此暂不生成 Phase 3 的
+exp/code HTML 档案。
+
| Case | 负载 | 动态窗口 |
|---|---|---|
| Decode Control | 1K → 1K，C=32 | 确认 Decode 活跃后跳过 2 Step，捕获 16 Step |
| Mixed Treatment | 1K → 1K，C=32 背景 + 128K → 1 注入 | 先保留 2 个纯 Decode Step，再注入 Prefill；总计捕获 32 Step |
| Long Prefill | 128K → 1，C=1 | 从首个 Chunk 开始捕获 16 Step |
+三段正式范围复用一次模型加载，因此 Nsight 使用
+--capture-range-end=repeat:3:defer。它与官方单段示例中的
+stop 目的相同，但允许同一个服务依次产出三段独立报告。
+
# 只在 Head 174.1.51.5 执行
+cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling
+RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S)
+tmux new-session -d -s dsv4pro-phase3 \
+ "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \
+ 2>&1 | tee /data/hzy/${RUN_ID}.log"
+tmux attach -t dsv4pro-phase3
+nsys stats <REPORT>.nsys-rep
重点查看：
diff --git a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md index 7ee5f4d..bba02f9 100644 --- a/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md +++ b/docs/dsv4pro_pro6000d_2node_sglang/推理优化计划.md @@ -320,7 +320,7 @@ NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH,TUNING ### 7.1 捕获策略 -- 只捕获预热后的 5 到 10 个 Engine Step。 +- 只捕获预热后的 10 到 32 个 Engine Step；混合场景窗口略长，用于覆盖 Prefill 注入前后。 - Prefill、Decode 和混合干扰分别生成报告。 - 两台机器分别保存原始报告。 - 优先保留所有 Rank；文件过大时至少保留代表 Rank 和跨机通信相关 Rank。 @@ -379,6 +379,31 @@ curl -X POST http://127.0.0.1:30000/start_profile \ 多机 Trace 自动合并要求两台机器能访问同一个共享输出目录。没有共享目录时分别保存，再在本地汇总。 +当前实现位于： + +```text +experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling +``` + +正式场景为： + +- Decode Control：`1K -> 1K, C=32`，确认 Decode 活跃后跳过 2 Step，捕获 16 Step。 +- Mixed Treatment：`1K -> 1K, C=32` 背景先捕获 2 个纯 Decode Step，再注入 `128K -> 1`，总计捕获 32 Step。 +- Long Prefill：`128K -> 1, C=1`，从首个 Chunk 开始捕获 16 Step。 + +双节点 PyTorch/Nsight smoke 已通过。正式 Run 尚未执行，因此暂不生成 Phase 3 的 `exp/code` HTML。正式命令只在 Head 执行： + +```bash +cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling +RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S) +tmux new-session -d -s dsv4pro-phase3 \ + "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \ + 2>&1 | tee /data/hzy/${RUN_ID}.log" +tmux attach -t dsv4pro-phase3 +``` + +三段范围复用一次模型加载，因此 Nsight 使用 `--capture-range-end=repeat:3:defer`，而不是单段示例中的 `stop`。 + ### 7.4 时间线必须回答的问题 1. Prefill 和 Decode 各自的 Top Kernel 是什么？ diff --git a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/README.md b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/README.md index 08732ce..84c6936 100644 --- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/README.md +++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/README.md @@ -3,29 +3,12 @@ Phase 3 captures short distributed timelines for the completed TP16 baseline. It does not rerun Phase 2 hardware sampling or communication microbenchmarks. -## One entry point +## Formal entry point Run only on Head `174.1.51.5`: ```bash cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling - -bash run_timeline_profiling.sh audit - -DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun-$(date +%Y%m%d-%H%M%S) \ - bash run_timeline_profiling.sh all - -RUN_ID=dsv4pro-phase3-torch-smoke-$(date +%Y%m%d-%H%M%S) \ - bash run_timeline_profiling.sh torch-smoke - -RUN_ID=dsv4pro-phase3-nsys-smoke-$(date +%Y%m%d-%H%M%S) \ - bash run_timeline_profiling.sh nsys-smoke -``` - -After both smoke tests pass, run the three formal capture ranges in one model -service lifetime: - -```bash RUN_ID=dsv4pro-phase3-$(date +%Y%m%d-%H%M%S) tmux new-session -d -s dsv4pro-phase3 \ "RUN_ID=${RUN_ID} bash run_timeline_profiling.sh all \ @@ -34,16 +17,28 @@ tmux new-session -d -s dsv4pro-phase3 \ tmux attach -t dsv4pro-phase3 ``` +PyTorch Profiler and Nsight smoke tests have already passed on this image and +topology. To inspect the exact commands without starting containers or sending +requests, run `DRY_RUN=1 RUN_ID=dsv4pro-phase3-dryrun bash +run_timeline_profiling.sh all` first. + The script starts and stops both nodes, triggers `/start_profile`, retrieves Worker artifacts, exports basic `nsys stats`, and cleans containers and Worker staging paths. No `source`, Conda activation, or command on Worker is required. ## Formal ranges -1. `decode_control`: `1K -> 256`, `C=32`. -2. `mixed_decode_with_128k_prefill`: active `1K -> 512`, `C=32` decode plus one `128K -> 1` injection. +1. `decode_control`: `1K -> 1K`, `C=32`; trigger after Decode becomes active, + skip 2 engine steps, then capture 16 steps. +2. `mixed_decode_with_128k_prefill`: active `1K -> 1K`, `C=32` decode; + capture 2 pure Decode steps before injecting one `128K -> 1` request, then + retain a 32-step range spanning both sides of the injection. 3. `long_prefill`: isolated `128K -> 1`, `C=1`. +The service remains under one Nsight process for all three ranges, so it uses +`--capture-range-end=repeat:3:defer` rather than the single-range `stop` form. +Every `/start_profile` request records both `start_step` and `num_steps`. + The primary run keeps CUDA Graph enabled. A later layer-wise NVTX run with disabled graphs is allowed only if these production-faithful traces cannot map the dominant kernels to model components. diff --git a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/config.env b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/config.env index 6a92f78..3278c53 100644 --- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/config.env +++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/config.env @@ -22,16 +22,22 @@ NSYS_MIXED_STEPS="${NSYS_MIXED_STEPS:-32}" NSYS_PREFILL_STEPS="${NSYS_PREFILL_STEPS:-16}" NSYS_SMOKE_STEPS="${NSYS_SMOKE_STEPS:-5}" TORCH_SMOKE_STEPS="${TORCH_SMOKE_STEPS:-3}" +TORCH_SMOKE_START_STEP="${TORCH_SMOKE_START_STEP:-1}" +NSYS_SMOKE_START_STEP="${NSYS_SMOKE_START_STEP:-1}" +NSYS_CONTROL_START_STEP="${NSYS_CONTROL_START_STEP:-2}" +NSYS_MIXED_START_STEP="${NSYS_MIXED_START_STEP:-0}" +NSYS_PREFILL_START_STEP="${NSYS_PREFILL_START_STEP:-0}" +MIXED_PRE_INJECTION_DECODE_STEPS="${MIXED_PRE_INJECTION_DECODE_STEPS:-2}" # Short profiling workloads. They reproduce the Phase 2 scheduler shapes but # avoid its full result matrix and long hardware sampling. CONTROL_ISL="${CONTROL_ISL:-1024}" -CONTROL_OSL="${CONTROL_OSL:-256}" +CONTROL_OSL="${CONTROL_OSL:-1024}" CONTROL_CONCURRENCY="${CONTROL_CONCURRENCY:-32}" CONTROL_PROMPTS="${CONTROL_PROMPTS:-32}" MIXED_ISL="${MIXED_ISL:-1024}" -MIXED_OSL="${MIXED_OSL:-512}" +MIXED_OSL="${MIXED_OSL:-1024}" MIXED_CONCURRENCY="${MIXED_CONCURRENCY:-32}" MIXED_PROMPTS="${MIXED_PROMPTS:-64}" MIXED_INJECTION_ISL="${MIXED_INJECTION_ISL:-131072}" diff --git a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh index 9eb7b62..46e9dd0 100644 --- a/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh +++ b/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_timeline_profiling/run_timeline_profiling.sh @@ -367,15 +367,16 @@ profile_done_count() { start_profile() { local case_id="$1" local activities_json="$2" - local steps="$3" + local start_step="$3" + local steps="$4" local output_dir="${PROFILE_ROOT_IN_CONTAINER}/torch/${case_id}" local response_file="${CASE_DIR}/${case_id}/start_profile_response.json" local payload - payload="{\"output_dir\":\"${output_dir}\",\"num_steps\":${steps},\"activities\":${activities_json},\"profile_id\":\"${RUN_ID}-${case_id}\",\"profile_prefix\":\"${case_id}\",\"merge_profiles\":false}" + payload="{\"output_dir\":\"${output_dir}\",\"start_step\":${start_step},\"num_steps\":${steps},\"activities\":${activities_json},\"profile_id\":\"${RUN_ID}-${case_id}\",\"profile_prefix\":\"${case_id}\",\"merge_profiles\":false}" mkdir -p "${CASE_DIR}/${case_id}" printf '%s\n' "${payload}" > "${CASE_DIR}/${case_id}/start_profile_request.json" if [[ "${DRY_RUN}" == "1" ]]; then - log "[DRY] POST /start_profile case=${case_id} steps=${steps} activities=${activities_json}" + log "[DRY] POST /start_profile case=${case_id} start_step=${start_step} steps=${steps} activities=${activities_json}" return 0 fi curl --fail --silent --show-error -X POST \ @@ -407,9 +408,16 @@ server_decode_count() { wait_for_decode_activity() { local before="$1" local bench_pid="$2" + wait_for_decode_steps "${before}" 1 "${bench_pid}" +} + +wait_for_decode_steps() { + local before="$1" + local required_steps="$2" + local bench_pid="$3" local waited=0 while (( waited < 180 )); do - if (( $(server_decode_count) > before )); then + if (( $(server_decode_count) >= before + required_steps )); then return 0 fi kill -0 "${bench_pid}" 2>/dev/null || return 1 @@ -482,28 +490,30 @@ run_bench() { run_profiled_case() { local case_id="$1" local activities_json="$2" - local steps="$3" - local isl="$4" - local osl="$5" - local concurrency="$6" - local prompts="$7" - local seed="$8" + local start_step="$3" + local steps="$4" + local isl="$5" + local osl="$6" + local concurrency="$7" + local prompts="$8" + local seed="$9" local before=0 flush_cache [[ "${DRY_RUN}" == "1" ]] || before="$(profile_done_count)" - start_profile "${case_id}" "${activities_json}" "${steps}" + start_profile "${case_id}" "${activities_json}" "${start_step}" "${steps}" run_bench "${case_id}" "${isl}" "${osl}" "${concurrency}" "${prompts}" "${seed}" [[ "${DRY_RUN}" == "1" ]] || wait_for_profile_stop "${before}" } run_profiled_decode_case() { local case_id="$1" - local steps="$2" - local isl="$3" - local osl="$4" - local concurrency="$5" - local prompts="$6" - local seed="$7" + local start_step="$2" + local steps="$3" + local isl="$4" + local osl="$5" + local concurrency="$6" + local prompts="$7" + local seed="$8" local case_path="${CASE_DIR}/${case_id}" local decode_before=0 local profile_before=0 @@ -517,7 +527,7 @@ run_profiled_decode_case() { log "[DRY] start decode workload, wait for Decode batch, then start Nsight range" printf '[DRY] ' cat "${case_path}/bench_cmd.txt" - start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}" + start_profile "${case_id}" '["CUDA_PROFILER"]' "${start_step}" "${steps}" return 0 fi decode_before="$(server_decode_count)" @@ -532,7 +542,7 @@ run_profiled_decode_case() { return 1 } profile_before="$(profile_done_count)" - start_profile "${case_id}" '["CUDA_PROFILER"]' "${steps}" + start_profile "${case_id}" '["CUDA_PROFILER"]' "${start_step}" "${steps}" wait_for_profile_stop "${profile_before}" wait "${bench_pid}" } @@ -557,6 +567,7 @@ run_mixed_profile() { local background_path="${CASE_DIR}/${background_id}" local before=0 local decode_before=0 + local mixed_baseline_before=0 flush_cache mkdir -p "${background_path}" prepare_bench_command "${background_path}/bench.jsonl" \ @@ -565,7 +576,9 @@ run_mixed_profile() { "${PROFILE_TIMEOUT_S}s" "${BENCH_CMD[@]}" > "${background_path}/bench_cmd.txt" if [[ "${DRY_RUN}" == "1" ]]; then log "[DRY] start decode background, wait for Decode batch, start profile, inject 128K prefill" - start_profile "${case_id}" '["CUDA_PROFILER"]' "${NSYS_MIXED_STEPS}" + start_profile "${case_id}" '["CUDA_PROFILER"]' \ + "${NSYS_MIXED_START_STEP}" "${NSYS_MIXED_STEPS}" + log "[DRY] wait ${MIXED_PRE_INJECTION_DECODE_STEPS} Decode steps before injection" run_bench "${injection_id}" "${MIXED_INJECTION_ISL}" \ "${MIXED_INJECTION_OSL}" 1 1 5202 return 0 @@ -582,7 +595,16 @@ run_mixed_profile() { return 1 } before="$(profile_done_count)" - start_profile "${case_id}" '["CUDA_PROFILER"]' "${NSYS_MIXED_STEPS}" + start_profile "${case_id}" '["CUDA_PROFILER"]' \ + "${NSYS_MIXED_START_STEP}" "${NSYS_MIXED_STEPS}" + mixed_baseline_before="$(server_decode_count)" + wait_for_decode_steps "${mixed_baseline_before}" \ + "${MIXED_PRE_INJECTION_DECODE_STEPS}" "${background_pid}" || { + kill "${background_pid}" 2>/dev/null || true + wait "${background_pid}" 2>/dev/null || true + log "ERROR: mixed profile did not capture the pre-injection Decode baseline" + return 1 + } run_bench "${injection_id}" "${MIXED_INJECTION_ISL}" \ "${MIXED_INJECTION_OSL}" 1 1 5202 wait_for_profile_stop "${before}" @@ -610,7 +632,12 @@ write_manifest() { "docker_image": "${DOCKER_IMAGE}", "cuda_graph_enabled": true, "tp_size": ${TP_SIZE}, - "ep_size": ${EP_SIZE} + "ep_size": ${EP_SIZE}, + "profile_windows": { + "decode_control": {"start_step": ${NSYS_CONTROL_START_STEP}, "num_steps": ${NSYS_CONTROL_STEPS}}, + "mixed": {"start_step": ${NSYS_MIXED_START_STEP}, "num_steps": ${NSYS_MIXED_STEPS}, "pre_injection_decode_steps": ${MIXED_PRE_INJECTION_DECODE_STEPS}}, + "long_prefill": {"start_step": ${NSYS_PREFILL_START_STEP}, "num_steps": ${NSYS_PREFILL_STEPS}} + } } EOF } @@ -648,7 +675,8 @@ run_torch_smoke() { start_service write_manifest RUNNING run_profiled_case torch_decode_control '["CPU","GPU"]' \ - "${TORCH_SMOKE_STEPS}" "${CONTROL_ISL}" 128 8 8 5101 + "${TORCH_SMOKE_START_STEP}" "${TORCH_SMOKE_STEPS}" \ + "${CONTROL_ISL}" 128 8 8 5101 stop_service CLEANUP_ON_EXIT=0 validate_torch_traces @@ -662,7 +690,8 @@ run_nsys_smoke() { start_service write_manifest RUNNING run_profiled_case nsys_decode_control '["CUDA_PROFILER"]' \ - "${NSYS_SMOKE_STEPS}" "${CONTROL_ISL}" "${CONTROL_OSL}" \ + "${NSYS_SMOKE_START_STEP}" "${NSYS_SMOKE_STEPS}" \ + "${CONTROL_ISL}" "${CONTROL_OSL}" \ "${CONTROL_CONCURRENCY}" "${CONTROL_PROMPTS}" 5102 stop_service CLEANUP_ON_EXIT=0 @@ -678,11 +707,13 @@ run_all() { start_service write_manifest RUNNING run_profiled_decode_case decode_control \ - "${NSYS_CONTROL_STEPS}" "${CONTROL_ISL}" "${CONTROL_OSL}" \ + "${NSYS_CONTROL_START_STEP}" "${NSYS_CONTROL_STEPS}" \ + "${CONTROL_ISL}" "${CONTROL_OSL}" \ "${CONTROL_CONCURRENCY}" "${CONTROL_PROMPTS}" 5301 run_mixed_profile run_profiled_case long_prefill '["CUDA_PROFILER"]' \ - "${NSYS_PREFILL_STEPS}" "${PREFILL_ISL}" "${PREFILL_OSL}" 1 1 5303 + "${NSYS_PREFILL_START_STEP}" "${NSYS_PREFILL_STEPS}" \ + "${PREFILL_ISL}" "${PREFILL_OSL}" 1 1 5303 stop_service CLEANUP_ON_EXIT=0 validate_nsys_reports