- Clean 128k/90% baseline filled (TTFT p50 14.70s, TPOT 51.7ms, hit 0.8999) - prior D-scenario bench was profiler-polluted
- chunk 16384 refuted: flashinfer cutlass MoE workspace scales with chunk -> CUDA OOM (3.08GB needed vs 2.11GB free at mem-frac 0.88);
physics kills the upside (AR is bandwidth-bound so call-count halving saves ~nothing; indexer q*k work is chunk-invariant)
- AR overlap/quant switch scan: no usable path on TP4+PP2+SM120+PCIe
(quant-communications is NPU-only via hard ValueError; flashinfer AR fusion auto-enable gated SM90/SM100; symm-mem/NVLS NVLink-oriented)
- indexer direction closed: 28% at 128k is DSA's inherent cost (per-request suffix queries are unique -> no cross-request reuse;
SM120 has only the deepgemm backend; model already shares topk across layers via index_topk_freq=4; ~219 TFLOPS/rank is reasonable for paged-gather)
- Conclusion: current config (chunk 8192 / mem-frac 0.88 / default NCCL) is config-optimal on this stack;
remaining gains are dev work: PP+MTP upstreaming (decode) > AR-chunk-overlap/quantized-AR kernel dev (prefill)
- Replaced archived deploy_par_605.sh with the actual /root/deploy_par.sh from 60.5 (md5 cf405176...) - includes sglang_patch mounts
MTP (multi-token prediction) speculative decoding was enabled, giving the
910C an unfair decode throughput advantage over the H20 baseline (which
has no MTP). Disable it so the benchmark measures pure model throughput.
- config.env: add DSV4_ENABLE_MTP=0 (default off)
- start_vllm_docker.sh: only add --speculative-config when DSV4_ENABLE_MTP=1
The previous params caused all 4 DP replicas to bind to the same 4 dies
(die 0-3), leaving 12 dies idle. Root cause was a combination of missing
official params + extra non-official params that interfered with DP
worker device placement.
Verified: with aligned params, TP=4 DP=4 correctly distributes 16 workers
across all 16 dies (8 cards x 2 dies), each at ~57 GB HBM (91% util).
Changes (align to docs.vllm.ai A3 tutorial):
- Add --max-num-batched-tokens 10240 (was missing; affects DP scheduling)
- Add --api-server-count 1 (was missing; without it vllm spawns N API
servers for N DP ranks, disturbing device assignment)
- Remove --kv-cache-dtype fp8 (official uses default bfloat16)
- Remove --trust-remote-code (official doesn't use it for DSV4)
- Remove enable_dsa_cp from additional-config (official doesn't have it)
- max-model-len: per-TP caps (32768/65536/131072) -> 1048576 for all TPs
(official uses full 1M; the caps were over-cautious)
- max-num-seqs: per-TP (128/256/256) -> 64 for all (official value)
The A3 910C has 16 dies (8 cards x 2 dies/card), and vllm-ascend's
tensor-parallel-size / data-parallel-size address DIES, not cards. The
old configs (2/4, 4/2, 8/1) all used only 8 dies = 4 cards, leaving half
the node idle. Switch to configs that use all 16 dies, and add per-TP
parameter overrides since each TP has a very different per-die memory
budget.
Why the old configs were wrong:
- A3 TP=4 (4 dies) is the per-CARD equivalent of H20 TP=4 (4 cards),
not a fair comparison. An A3 node has 2x the compute of an 8-card H20,
so benchmarking at 8 dies understates the A3 and never exercises the
cross-card / full-NVLink topology.
- TP*DP must equal 16 to use all dies on the A3 node.
New PARALLEL_CONFIGS (all use 16 dies):
- TP=4 DP=4: 4 dies/replica x 4 replicas, ~39 GiB weights/die
- TP=8 DP=2: 8 dies/replica x 2 replicas, ~35 GiB weights/die
- TP=16 DP=1: 16 dies/replica x 1 replica, ~17.5 GiB weights/die
Per-TP overrides (mirrors the glm52 6c81183 pattern):
- TP4: max_model_len=32768 max_num_seqs=128 (tight KV cache)
- TP8: max_model_len=65536 max_num_seqs=256 (balanced)
- TP16: max_model_len=131072 max_num_seqs=256 (max KV cache)
Changes:
- config.env: PARALLEL_CONFIGS -> "4 4"/"8 2"/"16 1"; add TP4_/TP8_/TP16_
env vars for per-TP gpu_mem_util/max_model_len/max_num_seqs.
- start_vllm_docker.sh: case "$TP" overrides the three params after
sourcing config.env, so the actual launch args match the per-TP budget.
- run_adaptive_concurrency_add16.sh: engine_build_server_args gets the
same case "$TP" so the recorded server_cmd.txt stays consistent with
the real launch.
Project-level documentation was scattered and duplicated across README.md,
BENCHMARK_WORKFLOW.md, and docs/EXPERIMENT_GUIDE.md (directory layout +
scripts/common component table repeated 3x). Reorganize into a clear
single-source-of-truth structure.
Changes:
- README.md: drop the 6 stale changelog entries at the top (latest was
07-21; history lives in git log). Replace the duplicated directory-
layout + scripts/common sections with a one-line link to
docs/EXPERIMENT_GUIDE.md. (151 -> 99 lines)
- BENCHMARK_WORKFLOW.md -> docs/BENCHMARK_WORKFLOW.md: relocate into docs/.
Replace its duplicated Directory Layout and Quick Start/Adding sections
with links to EXPERIMENT_GUIDE / README / NEW_PLATFORM_GUIDE; keep the
unique parts (Rules, Naming Conventions, Final JSON Schema, Checklist).
(394 -> 224 lines)
- docs/EXPERIMENT_GUIDE.md: now the single authority for directory layout
+ component table + experiment conventions. Add a cross-link from the
results.json field list to BENCHMARK_WORKFLOW's full JSON Schema and
Naming Conventions.
- docs/H200_QUICKSTART.md: deleted (outdated, repeatedly references
removed legacy scripts; H200 usage is covered by ADAPTIVE_CONCURRENCY_USAGE
and experiment READMEs).
- docs/DSV4_INFERENCE_COMPARISON_REPORT.md -> experiments/h200/
dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/: this is an
experiment report, not a project doc; relocate next to its sibling
report.md.
- envs/ASCEND_910C_ENV_SETUP.md §8: expand the vague "pip install sglang"
note into a full sglang client image build guide -- pin sglang 0.5.2
(not latest; >=0.5.16 deprecates bench_serving and breaks the parser),
--no-deps minimal install loop, docker commit to a local image, with
the exact commands used to build local/vllm-ascend:0.23-a3-dsv4-sglang.
- experiments/h200/dsv4_h200_vllm_tp2_custom_bench/README.md: fix the
now-broken link to BENCHMARK_WORKFLOW.md (../../ -> ../../../docs/).
- .gitignore: ignore *.bak.glm52orig scratch backups.
Also includes the add16 adaptive_results produced by the dsv4 TP=4/DP=2
runs on 910c.1.
