This directory contains the short, repeatable performance-map suite for DeepSeek-V4-Pro on two RTX PRO 6000 Blackwell nodes. It does not modify or call the existing dsv4_pro6000_sglang_tp16/run_batch.sh.

Network precondition

The launcher exposes only the three device nodes needed by the two deployment rails (rdma_cm, uverbs0, uverbs3) and fails before benchmarking unless both nodes' NCCL INFO logs prove NET/IB is using mlx5_0 and mlx5_3. The 2026-07-30 Phase 1 run verified cross-node channels over NET/IB/*/GDRDMA. eth0 remains the NCCL bootstrap and Socket-fallback interface.

Scope

The fixed suite covers eleven points:

Group	ISL	OSL	Concurrency
Prefill latency	1K / 32K / 128K	1	1
Prefill throughput	32K	1	16
Decode latency	1K	1K	1
Decode throughput	1K	1K	16 / 32 / 64
Long-output decode	1K	4K	16
Long-context decode	128K	1K	1
Balanced pressure	32K	1K	8

The same main entry also runs a mixed-interference A/B:

Run a finite 64-request 1K -> 1K, C=32 decode control.
Run the same decode workload again.
Wait for the background benchmark to enter its measured run, then inject one 128K -> 1, C=1 prefill request.
Report the change in output TPS and P95 TPOT/TTFT.

This aggregate mixed test is sufficient for the first performance-map pass. Per-request time-series attribution belongs to the later scheduler/interference phase.

Verified Phase 1 results

The no-profiler baseline completed in two runs over the same TP16 service configuration and the same dual-rail NET/IB + GDRDMA gate:

Run	Result	Scope
`dsv4pro-phase1-full-20260730-220916`	12/12	9 fixed points and 3 mixed A/B outputs
`dsv4pro-phase1-long-decode-20260730-234236`	2/2	Long-output and long-context Decode

The long-Decode supplement produced every configured output token:

Case	Output tokens	Output TPS	TTFT P95	TPOT P95	E2E P95
`1K -> 4K, C=16`	65,536	310.02	6.241 s	50.33 ms	211.364 s
`128K -> 1K, C=1`	1,024	12.43	49.326 s	32.24 ms	82.312 s

Compared with 1K -> 1K, C=16, the long-output case improved aggregate Output TPS by 4.99% while TPOT P95 changed by only +0.62%. The longer measured Decode interval amortized fixed startup and Prefill costs; no sustained-Decode throughput collapse was observed.

For 128K -> 1K, C=1, TTFT was only 2.03% above the 128K -> 1 Prefill-only case, while TPOT P95 was only 2.47% above 1K -> 1K, C=1. In this run, the large added latency came mainly from Prefill. Its 12.43 Output TPS includes the 49-second Prefill in benchmark duration and is not a pure Decode-rate metric.

Phase 1 therefore closes at 11/11 fixed points plus 3/3 mixed results. The next diagnostic stage should collect aligned GPU, CPU, SGLang scheduler and dual-rail RDMA time series for Prefill, ordinary Decode, sustained Decode, long-context Decode and the mixed A/B.

Files

File	Purpose
`run_quick_map.sh`	The only Shell entry: service lifecycle, fixed map, mixed A/B, cleanup
`config.env`	Machine, model, SGLang and benchmark configuration
`quick_map_scenarios.tsv`	Eleven fixed workload points
`quick_map_results.py`	Validate output and generate CSV/JSONL/Markdown summaries
`tests/test_quick_map_results.py`	Result parser regression tests

Before using GPUs

The scripts default to ports 20002 and 30002, and use unique container names. They never stop containers belonging to another experiment.

cd /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map

bash -n run_quick_map.sh
python3 -m unittest discover -s tests -v

DRY_RUN=1 bash run_quick_map.sh all

The launcher refuses to start while either node has an active GPU compute process. ALLOW_BUSY_GPU=1 exists only for deliberate operator override.

Before the complete run, use the same entry for a four-point Sanity pass:

CASE_IDS="short_prefill_latency_1k_c1,mid_prefill_latency_32k_c1,decode_latency_1k_to_1k_c1,decode_throughput_1k_to_1k_c32" \
  DRY_RUN=1 bash run_quick_map.sh fixed

Remove DRY_RUN=1 only after the TP16 service has passed the NET/IB startup gate. An unknown CASE_IDS value fails before any benchmark request is sent. Leave CASE_IDS empty to run all eleven fixed points.

To run only the two long-decode supplements:

export CASE_IDS="long_output_decode_1k_to_4k_c16,long_context_decode_128k_to_1k_c1"
export RUN_ID="dsv4pro-phase1-long-decode-$(date +%Y%m%d-%H%M%S)"

trap 'bash run_quick_map.sh stop' EXIT INT TERM
bash run_quick_map.sh start
bash run_quick_map.sh fixed

This uses the existing single entry for startup, the selected fixed cases, and trap-based cleanup. It does not run the mixed A/B suite.

GPU run

Run the complete quick map in tmux after both nodes are free:

tmux new-session -d -s dsv4pro-pro6000d-2node-sglang-quick-map \
  -c /data/hzy/sskj/experiments/pro6000/dsv4pro_pro6000d_2node_sglang_tp16_quick_map
tmux send-keys -t dsv4pro-pro6000d-2node-sglang-quick-map \
  'bash run_quick_map.sh all' Enter

The default all action starts the two-node service, runs the fixed and mixed suites, writes summaries, and then removes both service containers. The same entry supports targeted operations when debugging:

bash run_quick_map.sh start
bash run_quick_map.sh fixed
bash run_quick_map.sh mixed
bash run_quick_map.sh stop

Outputs

results/<RUN_ID>/
  run_manifest.json
  run.log
  summary.csv
  summary.jsonl
  aggregate.csv
  report.md
  cases/<case_id>/rep<N>/
    bench_cmd.txt
    bench.jsonl
    bench.log
    meta.json
  server/

summary.csv is one row per repetition. aggregate.csv groups repetitions and reports mean throughput plus coefficient of variation. Raw request arrays stay inside ignored result directories and are not committed.

Runtime policy

Quick default: one repetition and one measured request wave (num_prompts=C, including one request when C=1).
Milestone: three repetitions.
Short Prefill, ordinary Decode and 1K -> 4K long-output Decode use one warm-up request. Expensive 32K/128K Prefill and 128K -> 1K long-context Decode do not add a same-shape warm-up.
Every isolated point calls SGLang benchmark with --flush-cache; the benchmark performs its warm-up first and then clears the server Prefix Cache before measured requests.
No SLO-based early stop in the fixed map.
A case failure is recorded; an unhealthy service aborts the run to avoid producing a cascade of meaningless failures.
Benchmark and profiler results must not be mixed. This directory does not enable a profiler.

Hardware validation is recorded in the implementation report linked from the repository's optimization plan.