3f28edd1a6
Deploy Qwen3-235B-A22B on 8x Kunlun P800 XPU (TP=8) via sglang, adapted from the proven qwen3-8b launch (attention-backend kunlun, dtype float16, mem-fraction-static 0.9, XSGL_* XPU env vars). TP bumped to 8 with all 8 /dev/xpu* devices; context-length 8192 (the 8b used 4096 which truncated 2 outputs at 2k+2k). Patch qwen3_moe.py in the sglang image: load_weights assigned to the read-only @property routed_experts_weights_of_layer, raising AttributeError for any qwen3 MoE model (Qwen3-8B is dense, so unaffected). Fix: assign to the private _routed_experts_weights_of_layer, applied idempotently in start_server.sh on every start. bench_serving: isl=2048 osl=2048 concurrency=16 num_prompts=160. 160/160 success in 771s. Summary in results/qwen3_235b_tp8_run1/report.md. Server on port 30010 (30000 held by the lingering qwen3_8b_bench_tp1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
52 lines
2.1 KiB
Bash
52 lines
2.1 KiB
Bash
# Experiment: Qwen3-235B-A22B on Kunlun P800 (8x XPU), SGLang, TP=8.
|
|
# Benchmark: input 2048 / output 2048, concurrency 16, num_prompts 160.
|
|
#
|
|
# Adapted from the proven qwen3-8b TP=1 launch on this same image
|
|
# (attention-backend kunlun, XSGL_* XPU env vars, --dtype float16).
|
|
# Only differences vs the 8b run: --tp-size 8, 8 XPU devices, model path,
|
|
# and context-length 8192 (8b used 4096 which truncated 2 outputs at 2k+2k).
|
|
|
|
EXPERIMENT="${EXPERIMENT:-qwen3_235b_p800_sglang_tp8}"
|
|
MODEL_NAME="${MODEL_NAME:-Qwen3-235B-A22B}"
|
|
MODEL_PATH="${MODEL_PATH:-/data1/models/Qwen3-235B-A22B}"
|
|
SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-/data1/models/Qwen3-235B-A22B}"
|
|
BACKEND="${BACKEND:-sglang}"
|
|
ENGINE="${ENGINE:-sglang-xpu}"
|
|
DATASET="${DATASET:-random}"
|
|
DATASET_PATH="${DATASET_PATH:-/data1/yy/sharegpt_data/ShareGPT_V3_unfiltered_cleaned_split.json}"
|
|
|
|
# Serving port + container.
|
|
# 30000 is taken by the lingering qwen3_8b_bench_tp1 container (docker-proxy),
|
|
# so default to 30010.
|
|
PORT="${PORT:-30010}"
|
|
CONTAINER_NAME="${CONTAINER_NAME:-qwen3_235b_bench_tp8}"
|
|
|
|
# Docker image: Kunlun P800 SGLang image (same one the qwen3-8b run used).
|
|
DOCKER_IMAGE="${DOCKER_IMAGE:-iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202}"
|
|
|
|
# Hardware: 8x Kunlun P800 XPU, TP=8.
|
|
TP="${TP:-8}"
|
|
XPU_VISIBLE_DEVICES="${XPU_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
|
|
CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
|
|
|
|
# SGLang launch parameters (mirrors the working qwen3-8b config, TP bumped to 8).
|
|
CONTEXT_LENGTH="${CONTEXT_LENGTH:-8192}"
|
|
MAX_RUNNING_REQUESTS="${MAX_RUNNING_REQUESTS:-16}"
|
|
ATTENTION_BACKEND="${ATTENTION_BACKEND:-kunlun}"
|
|
MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.9}"
|
|
DTYPE="${DTYPE:-float16}"
|
|
PAGE_SIZE="${PAGE_SIZE:-64}"
|
|
MAX_PREFILL_TOKENS="${MAX_PREFILL_TOKENS:-32768}"
|
|
CUDA_GRAPH_MAX_BS="${CUDA_GRAPH_MAX_BS:-16}"
|
|
CUDA_GRAPH_BS="${CUDA_GRAPH_BS:-1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16}"
|
|
|
|
# Bench scenario: "concurrency input_len output_len num_prompts".
|
|
SCENARIOS=(
|
|
"16 2048 2048 160"
|
|
)
|
|
|
|
WARMUP="${WARMUP:-16}"
|
|
REQUEST_RATE="${REQUEST_RATE:-10000}"
|
|
SEED="${SEED:-1}"
|
|
RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1}"
|