MTP (multi-token prediction) speculative decoding was enabled, giving the
910C an unfair decode throughput advantage over the H20 baseline (which
has no MTP). Disable it so the benchmark measures pure model throughput.
- config.env: add DSV4_ENABLE_MTP=0 (default off)
- start_vllm_docker.sh: only add --speculative-config when DSV4_ENABLE_MTP=1
The previous params caused all 4 DP replicas to bind to the same 4 dies
(die 0-3), leaving 12 dies idle. Root cause was a combination of missing
official params + extra non-official params that interfered with DP
worker device placement.
Verified: with aligned params, TP=4 DP=4 correctly distributes 16 workers
across all 16 dies (8 cards x 2 dies), each at ~57 GB HBM (91% util).
Changes (align to docs.vllm.ai A3 tutorial):
- Add --max-num-batched-tokens 10240 (was missing; affects DP scheduling)
- Add --api-server-count 1 (was missing; without it vllm spawns N API
servers for N DP ranks, disturbing device assignment)
- Remove --kv-cache-dtype fp8 (official uses default bfloat16)
- Remove --trust-remote-code (official doesn't use it for DSV4)
- Remove enable_dsa_cp from additional-config (official doesn't have it)
- max-model-len: per-TP caps (32768/65536/131072) -> 1048576 for all TPs
(official uses full 1M; the caps were over-cautious)
- max-num-seqs: per-TP (128/256/256) -> 64 for all (official value)
The A3 910C has 16 dies (8 cards x 2 dies/card), and vllm-ascend's
tensor-parallel-size / data-parallel-size address DIES, not cards. The
old configs (2/4, 4/2, 8/1) all used only 8 dies = 4 cards, leaving half
the node idle. Switch to configs that use all 16 dies, and add per-TP
parameter overrides since each TP has a very different per-die memory
budget.
Why the old configs were wrong:
- A3 TP=4 (4 dies) is the per-CARD equivalent of H20 TP=4 (4 cards),
not a fair comparison. An A3 node has 2x the compute of an 8-card H20,
so benchmarking at 8 dies understates the A3 and never exercises the
cross-card / full-NVLink topology.
- TP*DP must equal 16 to use all dies on the A3 node.
New PARALLEL_CONFIGS (all use 16 dies):
- TP=4 DP=4: 4 dies/replica x 4 replicas, ~39 GiB weights/die
- TP=8 DP=2: 8 dies/replica x 2 replicas, ~35 GiB weights/die
- TP=16 DP=1: 16 dies/replica x 1 replica, ~17.5 GiB weights/die
Per-TP overrides (mirrors the glm52 6c81183 pattern):
- TP4: max_model_len=32768 max_num_seqs=128 (tight KV cache)
- TP8: max_model_len=65536 max_num_seqs=256 (balanced)
- TP16: max_model_len=131072 max_num_seqs=256 (max KV cache)
Changes:
- config.env: PARALLEL_CONFIGS -> "4 4"/"8 2"/"16 1"; add TP4_/TP8_/TP16_
env vars for per-TP gpu_mem_util/max_model_len/max_num_seqs.
- start_vllm_docker.sh: case "$TP" overrides the three params after
sourcing config.env, so the actual launch args match the per-TP budget.
- run_adaptive_concurrency_add16.sh: engine_build_server_args gets the
same case "$TP" so the recorded server_cmd.txt stays consistent with
the real launch.
Project-level documentation was scattered and duplicated across README.md,
BENCHMARK_WORKFLOW.md, and docs/EXPERIMENT_GUIDE.md (directory layout +
scripts/common component table repeated 3x). Reorganize into a clear
single-source-of-truth structure.
Changes:
- README.md: drop the 6 stale changelog entries at the top (latest was
07-21; history lives in git log). Replace the duplicated directory-
layout + scripts/common sections with a one-line link to
docs/EXPERIMENT_GUIDE.md. (151 -> 99 lines)
- BENCHMARK_WORKFLOW.md -> docs/BENCHMARK_WORKFLOW.md: relocate into docs/.
Replace its duplicated Directory Layout and Quick Start/Adding sections
with links to EXPERIMENT_GUIDE / README / NEW_PLATFORM_GUIDE; keep the
unique parts (Rules, Naming Conventions, Final JSON Schema, Checklist).
(394 -> 224 lines)
- docs/EXPERIMENT_GUIDE.md: now the single authority for directory layout
+ component table + experiment conventions. Add a cross-link from the
results.json field list to BENCHMARK_WORKFLOW's full JSON Schema and
Naming Conventions.
- docs/H200_QUICKSTART.md: deleted (outdated, repeatedly references
removed legacy scripts; H200 usage is covered by ADAPTIVE_CONCURRENCY_USAGE
and experiment READMEs).
- docs/DSV4_INFERENCE_COMPARISON_REPORT.md -> experiments/h200/
dsv4_h200_vllm_mtp_vs_default/results/20260708-160349/: this is an
experiment report, not a project doc; relocate next to its sibling
report.md.
- envs/ASCEND_910C_ENV_SETUP.md §8: expand the vague "pip install sglang"
note into a full sglang client image build guide -- pin sglang 0.5.2
(not latest; >=0.5.16 deprecates bench_serving and breaks the parser),
--no-deps minimal install loop, docker commit to a local image, with
the exact commands used to build local/vllm-ascend:0.23-a3-dsv4-sglang.
- experiments/h200/dsv4_h200_vllm_tp2_custom_bench/README.md: fix the
now-broken link to BENCHMARK_WORKFLOW.md (../../ -> ../../../docs/).
- .gitignore: ignore *.bak.glm52orig scratch backups.
Also includes the add16 adaptive_results produced by the dsv4 TP=4/DP=2
runs on 910c.1.
This directory held only a single one-off status file
(sglang_tp8_dp1_ctx262144_seq128.status.txt, content: "state=exited exit=0")
left over from a past capacity validation run. It has no ongoing use; remove
it to keep the repo tidy.
The datasets/ directory is gitignored (large, re-downloadable), but later
agents need to know how to obtain the dataset when it is missing. Add a
tracked README that documents the download steps.
- datasets/README.md: documents the ShareGPT_V3_unfiltered_cleaned_split.json
source (HF anon8231489123/ShareGPT_Vicuna_unfiltered), the hf-mirror.com
download command for CN environments, the JSON validity check (94145
entries), and the container path consistency note.
- .gitignore: change "datasets/" to "datasets/*" so the "!datasets/README.md"
negation actually works (git cannot re-include a file under a fully-ignored
directory). The dataset .json stays ignored; only README.md is tracked.
Sync the glm52 add16 fixes (98cdb67) into the DSV4 experiment so the
adaptive concurrency search can actually run end-to-end.
sglang client (the main blocker):
- Built local/vllm-ascend:0.23-a3-dsv4-sglang image: sglang 0.5.2 (not
0.5.16 -- 0.5.16 deprecates bench_serving and the glm52 parser fix
targets the 0.5.2 output format) + minimal deps (ipython/traitlets/
stack_data/executing/asttokens/pure_eval/prompt_toolkit/wcwidth) via
--no-deps, so the vllm env is untouched.
- Verified: python -m sglang.bench_serving --help works in the image.
- config.env DOCKER_IMAGE -> local/vllm-ascend:0.23-a3-dsv4-sglang.
config.env (sync glm52 98cdb67):
- CONTAINER_NAME: drop ${...:-} override -> fixed value (avoids the
double-suffix bug where CONTAINER_NAME already carries _tpX_dpY).
- CONTAINER_PYTHON: /usr/local/bin/python (does not exist) ->
/usr/local/python3.12.13/bin/python3 (matches glm52 fix).
run_adaptive_concurrency_add16.sh (sync glm52 98cdb67):
- --model $SERVED_MODEL_NAME -> --tokenizer $MODEL_PATH (bench_serving
0.5.2 wants the tokenizer path).
- docker exec env: add TORCH_DEVICE_BACKEND_AUTOLOAD=0 so the client
does not try to autoload torch_npu.
- export ENGINE_TP/ENGINE_DP in engine_start_server + export line;
container_name uses ${ENGINE_TP:-${tp}} (the bench runs in a subshell
where tp/dp are not in scope).
start_vllm_docker.sh (sync glm52 98cdb67):
- Health timeout configurable via HEALTH_MAX_RETRIES /
HEALTH_RETRY_INTERVAL_S (default 480x5s=40min; TP=16 compiles 16
graphs ~60min, old hardcoded 360x10s was too rigid).
- Container name: drop the double-suffix (CONTAINER_NAME no longer
re-overridden before appending _tpX_dpY).
- Mount /mnt (bench client reads dataset from there).
The dsv4_910c_vllm_tp_dp_matrix experiment was a placeholder (wrong
MODEL_PATH, glm52 image default, no DSV4 serve flags, no driver mounts).
Bring it to a working state, validated end-to-end on 910c.1.
config.env:
- MODEL_PATH: /mnt/models/DeepSeek-V4-Flash -> .../DeepSeek-V4-Flash-w8a8-mtp
(weights downloaded from ModelScope Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp,
~280 GiB, 70 shards, verified present and chowned to shishi)
- SERVED_MODEL_NAME: deepseek-v4-flash -> dsv4 (matches vllm-ascend tutorial)
- DOCKER_IMAGE: vllm-ascend:v0.23.0rc1-a3-openeuler ->
quay.io/ascend/vllm-ascend:v0.23.0rc1-a3-openeuler (full tag, present locally;
the glm5.2-a3 image carries GLM-specific patches and is NOT DSV4-compatible)
- Add DSV4_* serve-flag vars: tokenizer-mode/tool-call-parser/reasoning-parser
deepseek_v4, quantization ascend, expert-parallel, async-scheduling,
MTP speculative-config, compilation-config, additional-config, multithread
model-loader (128 threads for the 280GiB load)
- Rewrite the OOM-boundary comment to reflect actual w8a8 weight size
start_vllm_docker.sh (the main pitfalls vs the glm52 launcher):
- Inject the DSV4_* serve flags (GLM-5.2 needs none of them); without them
vllm rejects the model / lacks MTP
- Mount host driver libs (driver/lib64, dcmi, hccn_tool, npu-smi,
version.info, ascend_install.info, hccn.conf) -- otherwise the container
torch_npu fails with libascend_hal.so not found
- Mount all 16 dies via --device /dev/davinci0..15 + davinci_manager/
devmm_svm/hisi_hdc instead of relying on Ascend Docker Runtime injection
(ASCEND_VISIBLE_DEVICES-only), which was unreliable on this host
- --privileged --shm-size 512g for the 280GiB weight load
- LD_PRELOAD the openEuler jemalloc path /usr/lib64/libjemalloc.so.2
(the glm52 ubuntu path /usr/lib/aarch64-linux-gnu/... does not exist here)
- Raise health-wait budget 240x5s -> 360x10s (DSV4 load+compile ~8min)
- DRY_RUN mode for command preview without launching
Verified: start_vllm_docker.sh 4 2 brings the server up on port 30052 in
~8 min (130s weight load per die, 29s compile, 187s engine init); chat
completion returns correctly, system_fingerprint vllm-0.23.0-tp4-dp2-ep.
- Introduced `run_adaptive_concurrency_add16.sh` for adaptive concurrency testing with vLLM.
- Added `run_bench.sh` to execute TP×DP matrix benchmarks for Hy3-FP8 on vLLM.
- Created `start_vllm_docker.sh` to launch the vLLM server in a Docker container for specified TP and DP configurations.
- Implemented `start_vllm_dp.sh` to manage starting the vLLM server, defaulting to Docker unless specified otherwise.
- Enhanced logging and health check mechanisms for server readiness.
- Included GPU monitoring and OOM detection in benchmarking scripts.
Deploy Qwen3-235B-A22B on 8x Kunlun P800 XPU (TP=8) via sglang, adapted
from the proven qwen3-8b launch (attention-backend kunlun, dtype float16,
mem-fraction-static 0.9, XSGL_* XPU env vars). TP bumped to 8 with all 8
/dev/xpu* devices; context-length 8192 (the 8b used 4096 which truncated
2 outputs at 2k+2k).
Patch qwen3_moe.py in the sglang image: load_weights assigned to the
read-only @property routed_experts_weights_of_layer, raising
AttributeError for any qwen3 MoE model (Qwen3-8B is dense, so unaffected).
Fix: assign to the private _routed_experts_weights_of_layer, applied
idempotently in start_server.sh on every start.
bench_serving: isl=2048 osl=2048 concurrency=16 num_prompts=160.
160/160 success in 771s. Summary in results/qwen3_235b_tp8_run1/report.md.
Server on port 30010 (30000 held by the lingering qwen3_8b_bench_tp1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
