- Clean 128k/90% baseline filled (TTFT p50 14.70s, TPOT 51.7ms, hit 0.8999) - prior D-scenario bench was profiler-polluted
- chunk 16384 refuted: flashinfer cutlass MoE workspace scales with chunk -> CUDA OOM (3.08GB needed vs 2.11GB free at mem-frac 0.88);
physics kills the upside (AR is bandwidth-bound so call-count halving saves ~nothing; indexer q*k work is chunk-invariant)
- AR overlap/quant switch scan: no usable path on TP4+PP2+SM120+PCIe
(quant-communications is NPU-only via hard ValueError; flashinfer AR fusion auto-enable gated SM90/SM100; symm-mem/NVLS NVLink-oriented)
- indexer direction closed: 28% at 128k is DSA's inherent cost (per-request suffix queries are unique -> no cross-request reuse;
SM120 has only the deepgemm backend; model already shares topk across layers via index_topk_freq=4; ~219 TFLOPS/rank is reasonable for paged-gather)
- Conclusion: current config (chunk 8192 / mem-frac 0.88 / default NCCL) is config-optimal on this stack;
remaining gains are dev work: PP+MTP upstreaming (decode) > AR-chunk-overlap/quantized-AR kernel dev (prefill)
- Replaced archived deploy_par_605.sh with the actual /root/deploy_par.sh from 60.5 (md5 cf405176...) - includes sglang_patch mounts
MTP (multi-token prediction) speculative decoding was enabled, giving the
910C an unfair decode throughput advantage over the H20 baseline (which
has no MTP). Disable it so the benchmark measures pure model throughput.
- config.env: add DSV4_ENABLE_MTP=0 (default off)
- start_vllm_docker.sh: only add --speculative-config when DSV4_ENABLE_MTP=1
The previous params caused all 4 DP replicas to bind to the same 4 dies
(die 0-3), leaving 12 dies idle. Root cause was a combination of missing
official params + extra non-official params that interfered with DP
worker device placement.
Verified: with aligned params, TP=4 DP=4 correctly distributes 16 workers
across all 16 dies (8 cards x 2 dies), each at ~57 GB HBM (91% util).
Changes (align to docs.vllm.ai A3 tutorial):
- Add --max-num-batched-tokens 10240 (was missing; affects DP scheduling)
- Add --api-server-count 1 (was missing; without it vllm spawns N API
servers for N DP ranks, disturbing device assignment)
- Remove --kv-cache-dtype fp8 (official uses default bfloat16)
- Remove --trust-remote-code (official doesn't use it for DSV4)
- Remove enable_dsa_cp from additional-config (official doesn't have it)
- max-model-len: per-TP caps (32768/65536/131072) -> 1048576 for all TPs
(official uses full 1M; the caps were over-cautious)
- max-num-seqs: per-TP (128/256/256) -> 64 for all (official value)
The A3 910C has 16 dies (8 cards x 2 dies/card), and vllm-ascend's
tensor-parallel-size / data-parallel-size address DIES, not cards. The
old configs (2/4, 4/2, 8/1) all used only 8 dies = 4 cards, leaving half
the node idle. Switch to configs that use all 16 dies, and add per-TP
parameter overrides since each TP has a very different per-die memory
budget.
Why the old configs were wrong:
- A3 TP=4 (4 dies) is the per-CARD equivalent of H20 TP=4 (4 cards),
not a fair comparison. An A3 node has 2x the compute of an 8-card H20,
so benchmarking at 8 dies understates the A3 and never exercises the
cross-card / full-NVLink topology.
- TP*DP must equal 16 to use all dies on the A3 node.
New PARALLEL_CONFIGS (all use 16 dies):
- TP=4 DP=4: 4 dies/replica x 4 replicas, ~39 GiB weights/die
- TP=8 DP=2: 8 dies/replica x 2 replicas, ~35 GiB weights/die
- TP=16 DP=1: 16 dies/replica x 1 replica, ~17.5 GiB weights/die
Per-TP overrides (mirrors the glm52 6c81183 pattern):
- TP4: max_model_len=32768 max_num_seqs=128 (tight KV cache)
- TP8: max_model_len=65536 max_num_seqs=256 (balanced)
- TP16: max_model_len=131072 max_num_seqs=256 (max KV cache)
Changes:
- config.env: PARALLEL_CONFIGS -> "4 4"/"8 2"/"16 1"; add TP4_/TP8_/TP16_
env vars for per-TP gpu_mem_util/max_model_len/max_num_seqs.
- start_vllm_docker.sh: case "$TP" overrides the three params after
sourcing config.env, so the actual launch args match the per-TP budget.
- run_adaptive_concurrency_add16.sh: engine_build_server_args gets the
same case "$TP" so the recorded server_cmd.txt stays consistent with
the real launch.
