Deploy Qwen3-235B-A22B on 8x Kunlun P800 XPU (TP=8) via sglang, adapted
from the proven qwen3-8b launch (attention-backend kunlun, dtype float16,
mem-fraction-static 0.9, XSGL_* XPU env vars). TP bumped to 8 with all 8
/dev/xpu* devices; context-length 8192 (the 8b used 4096 which truncated
2 outputs at 2k+2k).
Patch qwen3_moe.py in the sglang image: load_weights assigned to the
read-only @property routed_experts_weights_of_layer, raising
AttributeError for any qwen3 MoE model (Qwen3-8B is dense, so unaffected).
Fix: assign to the private _routed_experts_weights_of_layer, applied
idempotently in start_server.sh on every start.
bench_serving: isl=2048 osl=2048 concurrency=16 num_prompts=160.
160/160 success in 771s. Summary in results/qwen3_235b_tp8_run1/report.md.
Server on port 30010 (30000 held by the lingering qwen3_8b_bench_tp1).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire the timing module into the dsv4_p800_sglang_profile
experiment's server startup so benchmarks produce per-operator
Prefill/Decode breakdowns alongside the existing aggregate metrics.
Changes to start_sglang_docker.sh:
- Mount p800_timing.py into the container site-packages (read-only)
via PATCH_ROOT so the path resolves correctly on any host.
- Set SGLANG_TIMING_ENABLED=1 in the container env to activate
the monkey-patches loaded by sitecustomize.
- Prepend import p800_timing to the launch_server python -c
entrypoint so the polling thread starts before model loading.
Combined with the previous two commits, a single run of
run_profile.sh now emits /tmp/p800_timing_results_{pid}.json
per TP worker with layer/attention/moe/hc/rmsnorm timing split
by prefill and decode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sitecustomize.py is imported automatically by Python at startup
in every process, including the SGLang scheduler/TP worker
processes spawned for tensor parallelism. Adding the p800_timing
import here ensures the monkey-patches are applied to all 8 TP
workers, not just the launch_server parent process (which never
runs the model forward and would record empty timing data).
The timing module self-disables unless SGLANG_TIMING_ENABLED=1,
so this import is a no-op when timing is not in use.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PyTorch Profiler crashes on P800 XPU due to a CUPTI bug in
cuptiActivityDisable() that SIGKILLs the server process. This
module provides an alternative by monkey-patching SGLang's
DeepSeek-V4 decoder layer and its sub-operators to record
per-op wall-clock timing.
What it does:
- Patches DeepseekV4DecoderLayer.forward and sub-operators
(hc_pre/hc_post, RMSNorm, MQALayer attention, QKV projection,
DeepseekV2MoE and its gate/shared/routed experts) with timed
wrappers that call torch.cuda.synchronize() before and after.
- Distinguishes prefill vs decode via forward_batch flags
(is_prefill_only / is_extend_in_batch / extend_num_tokens)
and propagates the phase to sub-operators via threading.local.
- Writes per-PID JSON summaries (count/avg/p50/p95/p99) every
5000 records plus on atexit/SIGTERM, so data survives when
the server is stopped.
Usage: set SGLANG_TIMING_ENABLED=1 in the container environment.
Output: /tmp/p800_timing_results_{pid}.json
Known limitations:
- torch.cuda.synchronize() adds overhead, inflating small-op times.
- moe_routed_ms only covers forward_normal (65% of calls); the
dual_stream path taken during CUDA graph capture is not patched.
- All-reduce patch fails (Communicator import path mismatch).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New experiment: dsv4_p800_sglang_profile for PyTorch profiler testing
- Documents XPU cuptiActivityDisable bug (error 17) when saving traces
- Includes benchmark results (ISL=4k, OSL=1k, C=16, TP8/DP1)
- PROFILE_REPORT.md with full analysis of P800 vs H20 performance gap
- Add dsv4_p800_sglang_tp_dp_official experiment config
- New experiment: dsv4_p800_sglang_profile for PyTorch profiler testing
- Documents XPU cuptiActivityDisable bug (error 17) when saving traces
- Includes benchmark results (ISL=4k, OSL=1k, C=16, TP8/DP1)
- PROFILE_REPORT.md with full analysis of P800 vs H20 performance gap
- Add dsv4_p800_sglang_tp_dp_official experiment config
- Created adaptive_summary.md to summarize adaptive concurrency search results, including metrics such as TP, DP, and TPS.
- Added run_manifest.json to document experiment configuration details, including hardware specifications and search parameters.
- Introduced shapes.tsv to define input and output lengths for various configurations in the adaptive experiment.
- Introduced a new CUDA kernel for transposing and packing FP32 into UE8.
- Added binary file for the new kernel.
- Created multiple autotune configuration files for version 0.6.14 of FlashInfer, supporting various input shapes and configurations.
- Added model information JSON for DeepseekV4ForCausalLM, detailing its architecture and capabilities.
- New experiments/p800/dsv4_p800_sglang_tp_dp_matrix: TP8/DP1, TP4/DP2,
TP2/DP4 matrix with smoke results; TP2/DP4 documents the weight-loading
OOM root cause (274 GiB INT8 weights sharded only across TP group).
- Launch args drop --ep-size/--chunked-prefill-size/--max-prefill-tokens/
--max-running-requests; experts fall back to TP sharding.
- Move dsv4_p800_256k_4k_probe under experiments/p800/.
- scripts/common: jq-free parsing fixes in adaptive_bench_lib.sh and
parse_backend.py.
- .gitignore: cover raw_outputs under nested platform experiment layout.
