The datasets/ directory is gitignored (large, re-downloadable), but later
agents need to know how to obtain the dataset when it is missing. Add a
tracked README that documents the download steps.
- datasets/README.md: documents the ShareGPT_V3_unfiltered_cleaned_split.json
source (HF anon8231489123/ShareGPT_Vicuna_unfiltered), the hf-mirror.com
download command for CN environments, the JSON validity check (94145
entries), and the container path consistency note.
- .gitignore: change "datasets/" to "datasets/*" so the "!datasets/README.md"
negation actually works (git cannot re-include a file under a fully-ignored
directory). The dataset .json stays ignored; only README.md is tracked.
Sync the glm52 add16 fixes (98cdb67) into the DSV4 experiment so the
adaptive concurrency search can actually run end-to-end.
sglang client (the main blocker):
- Built local/vllm-ascend:0.23-a3-dsv4-sglang image: sglang 0.5.2 (not
0.5.16 -- 0.5.16 deprecates bench_serving and the glm52 parser fix
targets the 0.5.2 output format) + minimal deps (ipython/traitlets/
stack_data/executing/asttokens/pure_eval/prompt_toolkit/wcwidth) via
--no-deps, so the vllm env is untouched.
- Verified: python -m sglang.bench_serving --help works in the image.
- config.env DOCKER_IMAGE -> local/vllm-ascend:0.23-a3-dsv4-sglang.
config.env (sync glm52 98cdb67):
- CONTAINER_NAME: drop ${...:-} override -> fixed value (avoids the
double-suffix bug where CONTAINER_NAME already carries _tpX_dpY).
- CONTAINER_PYTHON: /usr/local/bin/python (does not exist) ->
/usr/local/python3.12.13/bin/python3 (matches glm52 fix).
run_adaptive_concurrency_add16.sh (sync glm52 98cdb67):
- --model $SERVED_MODEL_NAME -> --tokenizer $MODEL_PATH (bench_serving
0.5.2 wants the tokenizer path).
- docker exec env: add TORCH_DEVICE_BACKEND_AUTOLOAD=0 so the client
does not try to autoload torch_npu.
- export ENGINE_TP/ENGINE_DP in engine_start_server + export line;
container_name uses ${ENGINE_TP:-${tp}} (the bench runs in a subshell
where tp/dp are not in scope).
start_vllm_docker.sh (sync glm52 98cdb67):
- Health timeout configurable via HEALTH_MAX_RETRIES /
HEALTH_RETRY_INTERVAL_S (default 480x5s=40min; TP=16 compiles 16
graphs ~60min, old hardcoded 360x10s was too rigid).
- Container name: drop the double-suffix (CONTAINER_NAME no longer
re-overridden before appending _tpX_dpY).
- Mount /mnt (bench client reads dataset from there).
The dsv4_910c_vllm_tp_dp_matrix experiment was a placeholder (wrong
MODEL_PATH, glm52 image default, no DSV4 serve flags, no driver mounts).
Bring it to a working state, validated end-to-end on 910c.1.
config.env:
- MODEL_PATH: /mnt/models/DeepSeek-V4-Flash -> .../DeepSeek-V4-Flash-w8a8-mtp
(weights downloaded from ModelScope Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp,
~280 GiB, 70 shards, verified present and chowned to shishi)
- SERVED_MODEL_NAME: deepseek-v4-flash -> dsv4 (matches vllm-ascend tutorial)
- DOCKER_IMAGE: vllm-ascend:v0.23.0rc1-a3-openeuler ->
quay.io/ascend/vllm-ascend:v0.23.0rc1-a3-openeuler (full tag, present locally;
the glm5.2-a3 image carries GLM-specific patches and is NOT DSV4-compatible)
- Add DSV4_* serve-flag vars: tokenizer-mode/tool-call-parser/reasoning-parser
deepseek_v4, quantization ascend, expert-parallel, async-scheduling,
MTP speculative-config, compilation-config, additional-config, multithread
model-loader (128 threads for the 280GiB load)
- Rewrite the OOM-boundary comment to reflect actual w8a8 weight size
start_vllm_docker.sh (the main pitfalls vs the glm52 launcher):
- Inject the DSV4_* serve flags (GLM-5.2 needs none of them); without them
vllm rejects the model / lacks MTP
- Mount host driver libs (driver/lib64, dcmi, hccn_tool, npu-smi,
version.info, ascend_install.info, hccn.conf) -- otherwise the container
torch_npu fails with libascend_hal.so not found
- Mount all 16 dies via --device /dev/davinci0..15 + davinci_manager/
devmm_svm/hisi_hdc instead of relying on Ascend Docker Runtime injection
(ASCEND_VISIBLE_DEVICES-only), which was unreliable on this host
- --privileged --shm-size 512g for the 280GiB weight load
- LD_PRELOAD the openEuler jemalloc path /usr/lib64/libjemalloc.so.2
(the glm52 ubuntu path /usr/lib/aarch64-linux-gnu/... does not exist here)
- Raise health-wait budget 240x5s -> 360x10s (DSV4 load+compile ~8min)
- DRY_RUN mode for command preview without launching
Verified: start_vllm_docker.sh 4 2 brings the server up on port 30052 in
~8 min (130s weight load per die, 29s compile, 187s engine init); chat
completion returns correctly, system_fingerprint vllm-0.23.0-tp4-dp2-ep.
- Introduced `run_adaptive_concurrency_add16.sh` for adaptive concurrency testing with vLLM.
- Added `run_bench.sh` to execute TP×DP matrix benchmarks for Hy3-FP8 on vLLM.
- Created `start_vllm_docker.sh` to launch the vLLM server in a Docker container for specified TP and DP configurations.
- Implemented `start_vllm_dp.sh` to manage starting the vLLM server, defaulting to Docker unless specified otherwise.
- Enhanced logging and health check mechanisms for server readiness.
- Included GPU monitoring and OOM detection in benchmarking scripts.
Deploy Qwen3-235B-A22B on 8x Kunlun P800 XPU (TP=8) via sglang, adapted
from the proven qwen3-8b launch (attention-backend kunlun, dtype float16,
mem-fraction-static 0.9, XSGL_* XPU env vars). TP bumped to 8 with all 8
/dev/xpu* devices; context-length 8192 (the 8b used 4096 which truncated
2 outputs at 2k+2k).
Patch qwen3_moe.py in the sglang image: load_weights assigned to the
read-only @property routed_experts_weights_of_layer, raising
AttributeError for any qwen3 MoE model (Qwen3-8B is dense, so unaffected).
Fix: assign to the private _routed_experts_weights_of_layer, applied
idempotently in start_server.sh on every start.
bench_serving: isl=2048 osl=2048 concurrency=16 num_prompts=160.
160/160 success in 771s. Summary in results/qwen3_235b_tp8_run1/report.md.
Server on port 30010 (30000 held by the lingering qwen3_8b_bench_tp1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire the timing module into the dsv4_p800_sglang_profile
experiment's server startup so benchmarks produce per-operator
Prefill/Decode breakdowns alongside the existing aggregate metrics.
Changes to start_sglang_docker.sh:
- Mount p800_timing.py into the container site-packages (read-only)
via PATCH_ROOT so the path resolves correctly on any host.
- Set SGLANG_TIMING_ENABLED=1 in the container env to activate
the monkey-patches loaded by sitecustomize.
- Prepend import p800_timing to the launch_server python -c
entrypoint so the polling thread starts before model loading.
Combined with the previous two commits, a single run of
run_profile.sh now emits /tmp/p800_timing_results_{pid}.json
per TP worker with layer/attention/moe/hc/rmsnorm timing split
by prefill and decode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sitecustomize.py is imported automatically by Python at startup
in every process, including the SGLang scheduler/TP worker
processes spawned for tensor parallelism. Adding the p800_timing
import here ensures the monkey-patches are applied to all 8 TP
workers, not just the launch_server parent process (which never
runs the model forward and would record empty timing data).
The timing module self-disables unless SGLANG_TIMING_ENABLED=1,
so this import is a no-op when timing is not in use.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PyTorch Profiler crashes on P800 XPU due to a CUPTI bug in
cuptiActivityDisable() that SIGKILLs the server process. This
module provides an alternative by monkey-patching SGLang's
DeepSeek-V4 decoder layer and its sub-operators to record
per-op wall-clock timing.
What it does:
- Patches DeepseekV4DecoderLayer.forward and sub-operators
(hc_pre/hc_post, RMSNorm, MQALayer attention, QKV projection,
DeepseekV2MoE and its gate/shared/routed experts) with timed
wrappers that call torch.cuda.synchronize() before and after.
- Distinguishes prefill vs decode via forward_batch flags
(is_prefill_only / is_extend_in_batch / extend_num_tokens)
and propagates the phase to sub-operators via threading.local.
- Writes per-PID JSON summaries (count/avg/p50/p95/p99) every
5000 records plus on atexit/SIGTERM, so data survives when
the server is stopped.
Usage: set SGLANG_TIMING_ENABLED=1 in the container environment.
Output: /tmp/p800_timing_results_{pid}.json
Known limitations:
- torch.cuda.synchronize() adds overhead, inflating small-op times.
- moe_routed_ms only covers forward_normal (65% of calls); the
dual_stream path taken during CUDA graph capture is not patched.
- All-reduce patch fails (Communicator import path mismatch).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
