0fdd3749c7
- New experiments/p800/dsv4_p800_sglang_tp_dp_matrix: TP8/DP1, TP4/DP2, TP2/DP4 matrix with smoke results; TP2/DP4 documents the weight-loading OOM root cause (274 GiB INT8 weights sharded only across TP group). - Launch args drop --ep-size/--chunked-prefill-size/--max-prefill-tokens/ --max-running-requests; experts fall back to TP sharding. - Move dsv4_p800_256k_4k_probe under experiments/p800/. - scripts/common: jq-free parsing fixes in adaptive_bench_lib.sh and parse_backend.py. - .gitignore: cover raw_outputs under nested platform experiment layout.
35 lines
1.5 KiB
Bash
35 lines
1.5 KiB
Bash
# OOM probe for 256k input + 4k output on DeepSeek-V4-Flash-INT8 (Kunlun P800).
|
|
# Tests whether the P800 SGLang deployment can serve this context length
|
|
# without OOM at concurrency=2 before adding it to the full long-context matrix.
|
|
|
|
EXPERIMENT="${EXPERIMENT:-dsv4_p800_256k_4k_probe}"
|
|
MODEL_NAME="${MODEL_NAME:-DeepSeek-V4-Flash-INT8}"
|
|
MODEL_PATH="${MODEL_PATH:-/data1/models/DeepSeek-V4-Flash-INT8}"
|
|
SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-deepseek-v4-flash-int8}"
|
|
PORT="${PORT:-30000}"
|
|
BACKEND="${BACKEND:-sglang}"
|
|
ENGINE="${ENGINE:-sglang-xpu}"
|
|
DATASET="${DATASET:-random}"
|
|
DATASET_PATH="${DATASET_PATH:-/workspace/dummy_sharegpt.json}"
|
|
|
|
# Server launch mode for P800 SGLang.
|
|
# - "w8a8_int8" : default tuned config with EAGLE speculative decoding
|
|
# - "w8a8_int8_baseline": baseline config without speculative decoding,
|
|
# matching the long-context P800 tuning.
|
|
SERVER_MODE="${SERVER_MODE:-w8a8_int8_baseline}"
|
|
|
|
# Extra SGLang launch arguments for the 256k-context probe.
|
|
# 262144 input + 4096 output + padding -> use 270000.
|
|
# max-running-requests=2 matches the probe concurrency; mem-fraction-static
|
|
# limits the KV cache reservation to avoid OOM on P800.
|
|
SGLANG_EXTRA_LAUNCH_ARGS="${SGLANG_EXTRA_LAUNCH_ARGS:---context-length 270000 --max-running-requests 2 --mem-fraction-static 0.85}"
|
|
|
|
# Scenario list: "concurrency input_len output_len" (one per line).
|
|
SCENARIOS=(
|
|
"2 262144 4096"
|
|
)
|
|
|
|
# Number of prompts for the probe scenario.
|
|
NUM_PROMPTS="${NUM_PROMPTS:-4}"
|
|
WARMUP="${WARMUP:-0}"
|