The A3 910C has 16 dies (8 cards x 2 dies/card), and vllm-ascend's
tensor-parallel-size / data-parallel-size address DIES, not cards. The
old configs (2/4, 4/2, 8/1) all used only 8 dies = 4 cards, leaving half
the node idle. Switch to configs that use all 16 dies, and add per-TP
parameter overrides since each TP has a very different per-die memory
budget.
Why the old configs were wrong:
- A3 TP=4 (4 dies) is the per-CARD equivalent of H20 TP=4 (4 cards),
not a fair comparison. An A3 node has 2x the compute of an 8-card H20,
so benchmarking at 8 dies understates the A3 and never exercises the
cross-card / full-NVLink topology.
- TP*DP must equal 16 to use all dies on the A3 node.
New PARALLEL_CONFIGS (all use 16 dies):
- TP=4 DP=4: 4 dies/replica x 4 replicas, ~39 GiB weights/die
- TP=8 DP=2: 8 dies/replica x 2 replicas, ~35 GiB weights/die
- TP=16 DP=1: 16 dies/replica x 1 replica, ~17.5 GiB weights/die
Per-TP overrides (mirrors the glm52 6c81183 pattern):
- TP4: max_model_len=32768 max_num_seqs=128 (tight KV cache)
- TP8: max_model_len=65536 max_num_seqs=256 (balanced)
- TP16: max_model_len=131072 max_num_seqs=256 (max KV cache)
Changes:
- config.env: PARALLEL_CONFIGS -> "4 4"/"8 2"/"16 1"; add TP4_/TP8_/TP16_
env vars for per-TP gpu_mem_util/max_model_len/max_num_seqs.
- start_vllm_docker.sh: case "$TP" overrides the three params after
sourcing config.env, so the actual launch args match the per-TP budget.
- run_adaptive_concurrency_add16.sh: engine_build_server_args gets the
same case "$TP" so the recorded server_cmd.txt stays consistent with
the real launch.
Project-level documentation was scattered and duplicated across README.md,
BENCHMARK_WORKFLOW.md, and docs/EXPERIMENT_GUIDE.md (directory layout +
scripts/common component table repeated 3x). Reorganize into a clear
single-source-of-truth structure.
Changes:
- README.md: drop the 6 stale changelog entries at the top (latest was
07-21; history lives in git log). Replace the duplicated directory-
layout + scripts/common sections with a one-line link to
docs/EXPERIMENT_GUIDE.md. (151 -> 99 lines)
- BENCHMARK_WORKFLOW.md -> docs/BENCHMARK_WORKFLOW.md: relocate into docs/.
Replace its duplicated Directory Layout and Quick Start/Adding sections
with links to EXPERIMENT_GUIDE / README / NEW_PLATFORM_GUIDE; keep the
unique parts (Rules, Naming Conventions, Final JSON Schema, Checklist).
(394 -> 224 lines)
- docs/EXPERIMENT_GUIDE.md: now the single authority for directory layout
+ component table + experiment conventions. Add a cross-link from the
results.json field list to BENCHMARK_WORKFLOW's full JSON Schema and
Naming Conventions.
- docs/H200_QUICKSTART.md: deleted (outdated, repeatedly references
removed legacy scripts; H200 usage is covered by ADAPTIVE_CONCURRENCY_USAGE
and experiment READMEs).
- docs/DSV4_INFERENCE_COMPARISON_REPORT.md -> experiments/h200/
dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/: this is an
experiment report, not a project doc; relocate next to its sibling
report.md.
- envs/ASCEND_910C_ENV_SETUP.md §8: expand the vague "pip install sglang"
note into a full sglang client image build guide -- pin sglang 0.5.2
(not latest; >=0.5.16 deprecates bench_serving and breaks the parser),
--no-deps minimal install loop, docker commit to a local image, with
the exact commands used to build local/vllm-ascend:0.23-a3-dsv4-sglang.
- experiments/h200/dsv4_h200_vllm_tp2_custom_bench/README.md: fix the
now-broken link to BENCHMARK_WORKFLOW.md (../../ -> ../../../docs/).
- .gitignore: ignore *.bak.glm52orig scratch backups.
Also includes the add16 adaptive_results produced by the dsv4 TP=4/DP=2
runs on 910c.1.
Sync the glm52 add16 fixes (98cdb67) into the DSV4 experiment so the
adaptive concurrency search can actually run end-to-end.
sglang client (the main blocker):
- Built local/vllm-ascend:0.23-a3-dsv4-sglang image: sglang 0.5.2 (not
0.5.16 -- 0.5.16 deprecates bench_serving and the glm52 parser fix
targets the 0.5.2 output format) + minimal deps (ipython/traitlets/
stack_data/executing/asttokens/pure_eval/prompt_toolkit/wcwidth) via
--no-deps, so the vllm env is untouched.
- Verified: python -m sglang.bench_serving --help works in the image.
- config.env DOCKER_IMAGE -> local/vllm-ascend:0.23-a3-dsv4-sglang.
config.env (sync glm52 98cdb67):
- CONTAINER_NAME: drop ${...:-} override -> fixed value (avoids the
double-suffix bug where CONTAINER_NAME already carries _tpX_dpY).
- CONTAINER_PYTHON: /usr/local/bin/python (does not exist) ->
/usr/local/python3.12.13/bin/python3 (matches glm52 fix).
run_adaptive_concurrency_add16.sh (sync glm52 98cdb67):
- --model $SERVED_MODEL_NAME -> --tokenizer $MODEL_PATH (bench_serving
0.5.2 wants the tokenizer path).
- docker exec env: add TORCH_DEVICE_BACKEND_AUTOLOAD=0 so the client
does not try to autoload torch_npu.
- export ENGINE_TP/ENGINE_DP in engine_start_server + export line;
container_name uses ${ENGINE_TP:-${tp}} (the bench runs in a subshell
where tp/dp are not in scope).
start_vllm_docker.sh (sync glm52 98cdb67):
- Health timeout configurable via HEALTH_MAX_RETRIES /
HEALTH_RETRY_INTERVAL_S (default 480x5s=40min; TP=16 compiles 16
graphs ~60min, old hardcoded 360x10s was too rigid).
- Container name: drop the double-suffix (CONTAINER_NAME no longer
re-overridden before appending _tpX_dpY).
- Mount /mnt (bench client reads dataset from there).
The dsv4_910c_vllm_tp_dp_matrix experiment was a placeholder (wrong
MODEL_PATH, glm52 image default, no DSV4 serve flags, no driver mounts).
Bring it to a working state, validated end-to-end on 910c.1.
config.env:
- MODEL_PATH: /mnt/models/DeepSeek-V4-Flash -> .../DeepSeek-V4-Flash-w8a8-mtp
(weights downloaded from ModelScope Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp,
~280 GiB, 70 shards, verified present and chowned to shishi)
- SERVED_MODEL_NAME: deepseek-v4-flash -> dsv4 (matches vllm-ascend tutorial)
- DOCKER_IMAGE: vllm-ascend:v0.23.0rc1-a3-openeuler ->
quay.io/ascend/vllm-ascend:v0.23.0rc1-a3-openeuler (full tag, present locally;
the glm5.2-a3 image carries GLM-specific patches and is NOT DSV4-compatible)
- Add DSV4_* serve-flag vars: tokenizer-mode/tool-call-parser/reasoning-parser
deepseek_v4, quantization ascend, expert-parallel, async-scheduling,
MTP speculative-config, compilation-config, additional-config, multithread
model-loader (128 threads for the 280GiB load)
- Rewrite the OOM-boundary comment to reflect actual w8a8 weight size
start_vllm_docker.sh (the main pitfalls vs the glm52 launcher):
- Inject the DSV4_* serve flags (GLM-5.2 needs none of them); without them
vllm rejects the model / lacks MTP
- Mount host driver libs (driver/lib64, dcmi, hccn_tool, npu-smi,
version.info, ascend_install.info, hccn.conf) -- otherwise the container
torch_npu fails with libascend_hal.so not found
- Mount all 16 dies via --device /dev/davinci0..15 + davinci_manager/
devmm_svm/hisi_hdc instead of relying on Ascend Docker Runtime injection
(ASCEND_VISIBLE_DEVICES-only), which was unreliable on this host
- --privileged --shm-size 512g for the 280GiB weight load
- LD_PRELOAD the openEuler jemalloc path /usr/lib64/libjemalloc.so.2
(the glm52 ubuntu path /usr/lib/aarch64-linux-gnu/... does not exist here)
- Raise health-wait budget 240x5s -> 360x10s (DSV4 load+compile ~8min)
- DRY_RUN mode for command preview without launching
Verified: start_vllm_docker.sh 4 2 brings the server up on port 30052 in
~8 min (130s weight load per die, 29s compile, 187s engine init); chat
completion returns correctly, system_fingerprint vllm-0.23.0-tp4-dp2-ep.
