feat(p800): add max context length exploration experiment
Add experiments/dsv4_p800_max_context_length/ to find the longest input context that P800 SGLang INT8 can serve for DeepSeek-V4-Flash-INT8. - Docker-based server start with dynamic --context-length. - Single-request, single-output-token probes for each candidate length. - Records success/failure, server args, and metrics per length. - Generates results.json and report.md summarizing the max supported length. - Smoke-tested at 4096 tokens successfully.
This commit is contained in:
Quantong Qiu
parent
8324d944f2
commit
3694614dd9
59
experiments/dsv4_p800_max_context_length/README.md
Normal file
59
experiments/dsv4_p800_max_context_length/README.md
Normal file
@ -0,0 +1,59 @@
|
|||||||
|
# DSV4 P800 最大上下文长度探索实验
|
||||||
|
|
||||||
|
在 8x Kunlun P800 XPU 上，逐步增大输入长度，测试 SGLang serving `DeepSeek-V4-Flash-INT8` 时能稳定处理的最长上下文。
|
||||||
|
|
||||||
|
## 目的
|
||||||
|
|
||||||
|
- 找到 P800 + SGLang INT8 在 TP=8 配置下，单请求最多能支持多长的输入 token。
|
||||||
|
- 每个长度只跑 1 个请求、输出 1 个 token，尽量减少 decode 阶段显存干扰。
|
||||||
|
- 每次测试单独启动 server，并把 `context-length` 刚好设为 `target_len + CONTEXT_PAD`，避免预分配过大。
|
||||||
|
|
||||||
|
## 默认候选长度
|
||||||
|
|
||||||
|
```text
|
||||||
|
4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288
|
||||||
|
```
|
||||||
|
|
||||||
|
可以通过环境变量覆盖：
|
||||||
|
|
||||||
|
```bash
|
||||||
|
CANDIDATE_LENS=(4096 8192 16384) bash run_bench.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## 控制变量
|
||||||
|
|
||||||
|
| 维度 | 配置 |
|
||||||
|
|---|---|
|
||||||
|
| 硬件 | 8x Kunlun P800 XPU，TP=8 |
|
||||||
|
| 模型 | `DeepSeek-V4-Flash-INT8`（`w8a8_int8`） |
|
||||||
|
| 输入长度 | `--random-input-len $target_len --random-range-ratio 1.0` |
|
||||||
|
| 输出长度 | 1 |
|
||||||
|
| 并发 / 请求数 | 1 / 1 |
|
||||||
|
| 压测客户端 | 容器内 `sglang.bench_serving --backend sglang` |
|
||||||
|
|
||||||
|
> `--random-range-ratio 1.0` 确保输入长度固定为 `target_len`，而不是默认的 `[1, target_len]` 均匀分布。
|
||||||
|
|
||||||
|
## 快速运行
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash experiments/dsv4_p800_max_context_length/run_bench.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
结果保存在 `experiments/dsv4_p800_max_context_length/results/<RUN_ID>/`：
|
||||||
|
|
||||||
|
```
|
||||||
|
results/<RUN_ID>/
|
||||||
|
├── results.json
|
||||||
|
├── report.md
|
||||||
|
└── raw_outputs/ # gitignored
|
||||||
|
```
|
||||||
|
|
||||||
|
## 产物说明
|
||||||
|
|
||||||
|
- `results.json`：每个候选长度的尝试记录，包括 `success`、TTFT/TPOT/E2E 指标、`server_args`。
|
||||||
|
- `report.md`：汇总表格，以及 SGLang INT8 最终支持的最大输入长度。
|
||||||
|
|
||||||
|
## 环境
|
||||||
|
|
||||||
|
- Docker image: `iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202`
|
||||||
|
- Container Python: `/root/miniconda/envs/python310_torch25_cuda/bin/python`
|
||||||
43
experiments/dsv4_p800_max_context_length/config.env
Normal file
43
experiments/dsv4_p800_max_context_length/config.env
Normal file
@ -0,0 +1,43 @@
|
|||||||
|
# Common configuration for the max context length exploration on P800.
|
||||||
|
# All values can be overridden via environment variables.
|
||||||
|
|
||||||
|
EXPERIMENT="dsv4_p800_max_context_length"
|
||||||
|
MODEL_NAME="DeepSeek-V4-Flash-INT8"
|
||||||
|
MODEL_PATH="/data1/models/DeepSeek-V4-Flash-INT8"
|
||||||
|
SERVED_MODEL_NAME="deepseek-v4-flash-int8"
|
||||||
|
|
||||||
|
# Port must differ from other running experiments.
|
||||||
|
PORT="${PORT:-30013}"
|
||||||
|
|
||||||
|
# We want exact input lengths, so fix the random range ratio to 1.0.
|
||||||
|
OUTPUT_LEN="${OUTPUT_LEN:-1}"
|
||||||
|
NUM_PROMPTS="${NUM_PROMPTS:-1}"
|
||||||
|
MAX_CONCURRENCY="${MAX_CONCURRENCY:-1}"
|
||||||
|
REQUEST_RATE="${REQUEST_RATE:-10000}"
|
||||||
|
|
||||||
|
# Padding added to target length when setting server context length.
|
||||||
|
CONTEXT_PAD="${CONTEXT_PAD:-1024}"
|
||||||
|
|
||||||
|
# Candidate input lengths (tokens). Override via CANDIDATE_LENS env var.
|
||||||
|
# Examples:
|
||||||
|
# CANDIDATE_LENS="4096 8192 16384" bash run_bench.sh
|
||||||
|
# CANDIDATE_LENS=(4096 8192 16384) bash run_bench.sh
|
||||||
|
if [[ -z "${CANDIDATE_LENS:-}" ]]; then
|
||||||
|
declare -a CANDIDATE_LENS=(
|
||||||
|
4096
|
||||||
|
8192
|
||||||
|
16384
|
||||||
|
32768
|
||||||
|
65536
|
||||||
|
131072
|
||||||
|
262144
|
||||||
|
524288
|
||||||
|
)
|
||||||
|
elif ! declare -p CANDIDATE_LENS 2>/dev/null | grep -q '^declare -a'; then
|
||||||
|
# Normalize string-form arrays: remove surrounding parentheses and commas.
|
||||||
|
_CAND_STR="${CANDIDATE_LENS#(}"
|
||||||
|
_CAND_STR="${_CAND_STR%)}"
|
||||||
|
_CAND_STR="${_CAND_STR//,/}"
|
||||||
|
read -ra CANDIDATE_LENS <<< "$_CAND_STR"
|
||||||
|
unset _CAND_STR
|
||||||
|
fi
|
||||||
75
experiments/dsv4_p800_max_context_length/extract_metrics.py
Executable file
75
experiments/dsv4_p800_max_context_length/extract_metrics.py
Executable file
@ -0,0 +1,75 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Extract a compact metrics dict from a sglang.bench_serving JSONL output.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 extract_metrics.py <raw_jsonl_path>
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def parse_jsonl(path: Path) -> dict | None:
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
return json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def compute(data: dict) -> dict:
|
||||||
|
return {
|
||||||
|
"success_count": data.get("completed", 0),
|
||||||
|
"total_count": len(data.get("input_lens", [])),
|
||||||
|
"duration_s": data.get("duration", 0.0),
|
||||||
|
"request_throughput": data.get("request_throughput", 0.0),
|
||||||
|
"input_token_throughput": data.get("input_throughput", 0.0),
|
||||||
|
"output_token_throughput": data.get("output_throughput", 0.0),
|
||||||
|
"total_input_tokens": data.get("total_input_tokens", 0),
|
||||||
|
"total_output_tokens": data.get("total_output_tokens", 0),
|
||||||
|
"e2e_ms": {
|
||||||
|
"mean": data.get("mean_e2e_latency_ms", 0.0),
|
||||||
|
"p50": data.get("median_e2e_latency_ms", 0.0),
|
||||||
|
"p90": data.get("p90_e2e_latency_ms", 0.0),
|
||||||
|
"p95": data.get("p95_e2e_latency_ms", 0.0),
|
||||||
|
"p99": data.get("p99_e2e_latency_ms", 0.0),
|
||||||
|
},
|
||||||
|
"ttft_ms": {
|
||||||
|
"mean": data.get("mean_ttft_ms", 0.0),
|
||||||
|
"p50": data.get("median_ttft_ms", 0.0),
|
||||||
|
"p90": data.get("p90_ttft_ms", 0.0),
|
||||||
|
"p95": data.get("p95_ttft_ms", 0.0),
|
||||||
|
"p99": data.get("p99_ttft_ms", 0.0),
|
||||||
|
},
|
||||||
|
"tpot_ms": {
|
||||||
|
"mean": data.get("mean_tpot_ms", 0.0),
|
||||||
|
"p50": data.get("median_tpot_ms", 0.0),
|
||||||
|
"p90": data.get("p90_tpot_ms", 0.0),
|
||||||
|
"p95": data.get("p95_tpot_ms", 0.0),
|
||||||
|
"p99": data.get("p99_tpot_ms", 0.0),
|
||||||
|
},
|
||||||
|
"itl_ms": {
|
||||||
|
"mean": data.get("mean_itl_ms", 0.0),
|
||||||
|
"p50": data.get("median_itl_ms", 0.0),
|
||||||
|
"p90": data.get("p90_itl_ms", 0.0),
|
||||||
|
"p95": data.get("p95_itl_ms", 0.0),
|
||||||
|
"p99": data.get("p99_itl_ms", 0.0),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
path = Path(sys.argv[1])
|
||||||
|
data = parse_jsonl(path)
|
||||||
|
if data is None:
|
||||||
|
raise SystemExit("No valid JSON found")
|
||||||
|
print(json.dumps(compute(data), indent=2, ensure_ascii=False))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
58
experiments/dsv4_p800_max_context_length/parse_results.py
Executable file
58
experiments/dsv4_p800_max_context_length/parse_results.py
Executable file
@ -0,0 +1,58 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Generate report.md from results.json for the max context length experiment.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 parse_results.py <result_root>
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("results")
|
||||||
|
results_json = result_root / "results.json"
|
||||||
|
report_path = result_root / "report.md"
|
||||||
|
|
||||||
|
with open(results_json, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
|
||||||
|
meta = data["metadata"]
|
||||||
|
cfg = data["config"]
|
||||||
|
|
||||||
|
with open(report_path, "w", encoding="utf-8") as f:
|
||||||
|
f.write("# P800 Max Context Length Exploration Report\n\n")
|
||||||
|
f.write(f"- Result root: `{result_root}`\n")
|
||||||
|
f.write(f"- Model: `{meta['model']}`\n")
|
||||||
|
f.write(f"- Hardware: {meta['hardware']}\n")
|
||||||
|
f.write(f"- TP: {cfg['tp']}, EP: {cfg['ep']}\n")
|
||||||
|
f.write(f"- Output len: {cfg['output_len']}, num prompts: {cfg['num_prompts']}, concurrency: {cfg['max_concurrency']}\n\n")
|
||||||
|
|
||||||
|
f.write("## Per-attempt results\n\n")
|
||||||
|
f.write("| Target len | Max context len | Status | TTFT mean(ms) | TTFT P95(ms) | E2E mean(ms) | E2E P99(ms) | Error |\n")
|
||||||
|
f.write("|---:|---:|---:|---:|---:|---:|---:|---:|\n")
|
||||||
|
|
||||||
|
max_supported = 0
|
||||||
|
for s in data.get("scenarios", []):
|
||||||
|
target = s["target_len"]
|
||||||
|
status = "✅ pass" if s["success"] else "❌ fail"
|
||||||
|
metrics = s.get("metrics") or {}
|
||||||
|
ttft = metrics.get("ttft_ms", {})
|
||||||
|
e2e = metrics.get("e2e_ms", {})
|
||||||
|
f.write(
|
||||||
|
f"| {target} | {s['max_context_len']} | {status} | "
|
||||||
|
f"{ttft.get('mean', 0):.2f} | {ttft.get('p95', 0):.2f} | "
|
||||||
|
f"{e2e.get('mean', 0):.2f} | {e2e.get('p99', 0):.2f} | "
|
||||||
|
f"{s.get('error', '') or ''} |\n"
|
||||||
|
)
|
||||||
|
if s["success"] and target > max_supported:
|
||||||
|
max_supported = target
|
||||||
|
|
||||||
|
f.write("\n## Maximum supported input length\n\n")
|
||||||
|
f.write(f"- **sglang-xpu**: {max_supported} tokens\n\n")
|
||||||
|
|
||||||
|
print(f"Wrote report to {report_path}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@ -0,0 +1,18 @@
|
|||||||
|
# P800 Max Context Length Exploration Report
|
||||||
|
|
||||||
|
- Result root: `/data1/yy/sskj/experiments/dsv4_p800_max_context_length/results/20260709-032422`
|
||||||
|
- Model: `/data1/models/DeepSeek-V4-Flash-INT8`
|
||||||
|
- Hardware: 8x Kunlun P800 XPU
|
||||||
|
- TP: 8, EP: 8
|
||||||
|
- Output len: 1, num prompts: 1, concurrency: 1
|
||||||
|
|
||||||
|
## Per-attempt results
|
||||||
|
|
||||||
|
| Target len | Max context len | Status | TTFT mean(ms) | TTFT P95(ms) | E2E mean(ms) | E2E P99(ms) | Error |
|
||||||
|
|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| 4096 | 5120 | ✅ pass | 231.33 | 0.00 | 231.34 | 231.34 | |
|
||||||
|
|
||||||
|
## Maximum supported input length
|
||||||
|
|
||||||
|
- **sglang-xpu**: 4096 tokens
|
||||||
|
|
||||||
@ -0,0 +1,76 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_max_context_length",
|
||||||
|
"run_id": "20260709-032422",
|
||||||
|
"timestamp": "2026-07-09T03:24:22+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_max_context_length/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "1d1c79d",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "P800 max context length exploration for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {
|
||||||
|
"tp": 8,
|
||||||
|
"ep": 8,
|
||||||
|
"xpu_visible_devices": "0,1,2,3,4,5,6,7",
|
||||||
|
"output_len": 1,
|
||||||
|
"num_prompts": 1,
|
||||||
|
"max_concurrency": 1,
|
||||||
|
"request_rate": 10000,
|
||||||
|
"context_pad": 1024
|
||||||
|
},
|
||||||
|
"scenarios": [
|
||||||
|
{
|
||||||
|
"backend": "sglang",
|
||||||
|
"target_len": 4096,
|
||||||
|
"max_context_len": 5120,
|
||||||
|
"success": true,
|
||||||
|
"server_args": "python -m sglang.launch_server --host 0.0.0.0 --port 30013 --model-path /models --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.8 --max-prefill-tokens 5120 --max-running-requests 64 --tensor-parallel-size 8 --ep-size 8 --disable-shared-experts-fusion --quantization w8a8_int8 --kv-cache-dtype float16 --disable-piecewise-cuda-graph --cuda-graph-max-bs 32 --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --constrained-json-disable-any-whitespace --enable-metrics --enable-request-time-stats-logging --context-length 5120",
|
||||||
|
"metrics": {
|
||||||
|
"success_count": 1,
|
||||||
|
"total_count": 1,
|
||||||
|
"duration_s": 3.3442696970305406,
|
||||||
|
"request_throughput": 0.2990189460162034,
|
||||||
|
"input_token_throughput": 1224.781602882369,
|
||||||
|
"output_token_throughput": 0.2990189460162034,
|
||||||
|
"total_input_tokens": 4096,
|
||||||
|
"total_output_tokens": 1,
|
||||||
|
"e2e_ms": {
|
||||||
|
"mean": 231.34461202425882,
|
||||||
|
"p50": 231.34461202425882,
|
||||||
|
"p90": 231.34461202425882,
|
||||||
|
"p95": 0.0,
|
||||||
|
"p99": 231.34461202425882
|
||||||
|
},
|
||||||
|
"ttft_ms": {
|
||||||
|
"mean": 231.33470199536532,
|
||||||
|
"p50": 231.33470199536532,
|
||||||
|
"p90": 0.0,
|
||||||
|
"p95": 0.0,
|
||||||
|
"p99": 231.33470199536532
|
||||||
|
},
|
||||||
|
"tpot_ms": {
|
||||||
|
"mean": 0.0,
|
||||||
|
"p50": 0.0,
|
||||||
|
"p90": 0.0,
|
||||||
|
"p95": 0.0,
|
||||||
|
"p99": 0.0
|
||||||
|
},
|
||||||
|
"itl_ms": {
|
||||||
|
"mean": 0.0,
|
||||||
|
"p50": 0.0,
|
||||||
|
"p90": 0.0,
|
||||||
|
"p95": 0.0,
|
||||||
|
"p99": 0.0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"error": null
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
245
experiments/dsv4_p800_max_context_length/run_bench.sh
Executable file
245
experiments/dsv4_p800_max_context_length/run_bench.sh
Executable file
@ -0,0 +1,245 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Max context length exploration for P800 SGLang INT8.
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
|
||||||
|
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
|
||||||
|
RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-${SCRIPT_DIR}/results/${RUN_ID}}"
|
||||||
|
LOG_DIR="${RESULT_ROOT}/logs"
|
||||||
|
RAW_DIR="${RESULT_ROOT}/raw_outputs"
|
||||||
|
CONTAINER_NAME="${CONTAINER_NAME:-sglang-dsv4-flash}"
|
||||||
|
CONTAINER_PYTHON="${CONTAINER_PYTHON:-/root/miniconda/envs/python310_torch25_cuda/bin/python}"
|
||||||
|
|
||||||
|
ensure_result_root "$RESULT_ROOT"
|
||||||
|
log_init "${LOG_DIR}/orchestrator.log"
|
||||||
|
|
||||||
|
log "experiment=${EXPERIMENT_NAME}"
|
||||||
|
log "run_id=${RUN_ID}"
|
||||||
|
log "result_root=${RESULT_ROOT}"
|
||||||
|
log "platform=${PLATFORM}"
|
||||||
|
log "hardware=${HARDWARE}"
|
||||||
|
log "model=${MODEL_PATH}"
|
||||||
|
log "candidate_lengths=${CANDIDATE_LENS[*]}"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
is_server_healthy() {
|
||||||
|
curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
stop_server() {
|
||||||
|
log "stopping container ${CONTAINER_NAME}"
|
||||||
|
docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
|
||||||
|
pkill -9 -f 'sglang.launch_server' 2>/dev/null || true
|
||||||
|
sleep 2
|
||||||
|
}
|
||||||
|
|
||||||
|
start_server() {
|
||||||
|
local target_len="$1"
|
||||||
|
log "starting server (target_len=${target_len})"
|
||||||
|
bash "${SCRIPT_DIR}/start_server.sh" "$target_len" >> "${LOG_DIR}/start_server_${target_len}.log" 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
build_server_args() {
|
||||||
|
local target_len="$1"
|
||||||
|
local max_context_len=$(( target_len + CONTEXT_PAD ))
|
||||||
|
echo "python -m sglang.launch_server --host 0.0.0.0 --port ${PORT} --model-path /models --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.8 --max-prefill-tokens ${max_context_len} --max-running-requests 64 --tensor-parallel-size 8 --ep-size 8 --disable-shared-experts-fusion --quantization w8a8_int8 --kv-cache-dtype float16 --disable-piecewise-cuda-graph --cuda-graph-max-bs 32 --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --constrained-json-disable-any-whitespace --enable-metrics --enable-request-time-stats-logging --context-length ${max_context_len}"
|
||||||
|
}
|
||||||
|
|
||||||
|
append_scenario() {
|
||||||
|
local scenario_json="$1"
|
||||||
|
python3 - "$RESULT_JSON" "$scenario_json" <<'PY'
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
results_path, scenario_json = sys.argv[1], sys.argv[2]
|
||||||
|
with open(results_path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
|
||||||
|
data["scenarios"].append(json.loads(scenario_json))
|
||||||
|
with open(results_path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
# Build a scenario JSON blob safely with Python.
|
||||||
|
build_scenario_json() {
|
||||||
|
local target_len="$1"
|
||||||
|
local max_context_len="$2"
|
||||||
|
local success="$3"
|
||||||
|
local server_args="$4"
|
||||||
|
local error_msg="$5"
|
||||||
|
local metrics_json="${6:-null}"
|
||||||
|
|
||||||
|
python3 - "$target_len" "$max_context_len" "$success" "$server_args" "$error_msg" "$metrics_json" <<'PY'
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
target_len, max_context_len, success, server_args, error_msg, metrics_json = sys.argv[1:7]
|
||||||
|
metrics = json.loads(metrics_json) if metrics_json not in ("null", "") else None
|
||||||
|
print(json.dumps({
|
||||||
|
"backend": "sglang",
|
||||||
|
"target_len": int(target_len),
|
||||||
|
"max_context_len": int(max_context_len),
|
||||||
|
"success": success == "true",
|
||||||
|
"server_args": server_args,
|
||||||
|
"metrics": metrics,
|
||||||
|
"error": error_msg or None,
|
||||||
|
}, ensure_ascii=False))
|
||||||
|
PY
|
||||||
|
}
|
||||||
|
|
||||||
|
run_length() {
|
||||||
|
local target_len="$1"
|
||||||
|
local max_context_len=$(( target_len + CONTEXT_PAD ))
|
||||||
|
local output_file="${RAW_DIR}/sglang_ctx_${target_len}.jsonl"
|
||||||
|
local container_output="/tmp/bench_outputs/sglang_ctx_${target_len}.jsonl"
|
||||||
|
local detail_log="${LOG_DIR}/sglang_ctx_${target_len}.log"
|
||||||
|
|
||||||
|
log "trying sglang target_len=${target_len} (context-length=${max_context_len})"
|
||||||
|
|
||||||
|
stop_server
|
||||||
|
local server_args
|
||||||
|
server_args="$(build_server_args "$target_len")"
|
||||||
|
if ! start_server "$target_len"; then
|
||||||
|
local err="server failed to start"
|
||||||
|
log "ERROR: ${err}"
|
||||||
|
append_scenario "$(build_scenario_json "$target_len" "$max_context_len" "false" "$server_args" "$err")"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
docker exec "$CONTAINER_NAME" mkdir -p "$(dirname "$container_output")"
|
||||||
|
|
||||||
|
local bench_rc=0
|
||||||
|
docker exec "$CONTAINER_NAME" \
|
||||||
|
env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
|
||||||
|
"${CONTAINER_PYTHON}" -m sglang.bench_serving \
|
||||||
|
--backend sglang \
|
||||||
|
--host 127.0.0.1 \
|
||||||
|
--port "$PORT" \
|
||||||
|
--model "$MODEL_PATH" \
|
||||||
|
--dataset-name random \
|
||||||
|
--dataset-path /workspace/dummy_sharegpt.json \
|
||||||
|
--random-input-len "$target_len" \
|
||||||
|
--random-output-len "$OUTPUT_LEN" \
|
||||||
|
--random-range-ratio 1.0 \
|
||||||
|
--num-prompts "$NUM_PROMPTS" \
|
||||||
|
--max-concurrency "$MAX_CONCURRENCY" \
|
||||||
|
--request-rate "$REQUEST_RATE" \
|
||||||
|
--output-file "$container_output" \
|
||||||
|
--output-details \
|
||||||
|
> "$detail_log" 2>&1 || bench_rc=$?
|
||||||
|
|
||||||
|
if [[ "$bench_rc" -ne 0 ]]; then
|
||||||
|
local err="bench_serving exited with code ${bench_rc}; see ${detail_log}"
|
||||||
|
log "ERROR: sglang target_len=${target_len} failed; ${err}"
|
||||||
|
append_scenario "$(build_scenario_json "$target_len" "$max_context_len" "false" "$server_args" "$err")"
|
||||||
|
stop_server
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
docker cp "${CONTAINER_NAME}:${container_output}" "$output_file" || {
|
||||||
|
local err="failed to copy output from container"
|
||||||
|
log "ERROR: ${err}"
|
||||||
|
append_scenario "$(build_scenario_json "$target_len" "$max_context_len" "false" "$server_args" "$err")"
|
||||||
|
stop_server
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Verify the request actually completed.
|
||||||
|
local completed
|
||||||
|
completed="$(python3 -c "
|
||||||
|
import json
|
||||||
|
with open('${output_file}') as f:
|
||||||
|
for line in f:
|
||||||
|
data = json.loads(line)
|
||||||
|
print(data.get('completed', 0))
|
||||||
|
break
|
||||||
|
")"
|
||||||
|
if [[ "${completed:-0}" -lt "$NUM_PROMPTS" ]]; then
|
||||||
|
local err="only ${completed}/${NUM_PROMPTS} requests completed"
|
||||||
|
log "ERROR: sglang target_len=${target_len} ${err}"
|
||||||
|
append_scenario "$(build_scenario_json "$target_len" "$max_context_len" "false" "$server_args" "$err")"
|
||||||
|
stop_server
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
local metrics_json
|
||||||
|
metrics_json="$(python3 "${SCRIPT_DIR}/extract_metrics.py" "$output_file")"
|
||||||
|
|
||||||
|
append_scenario "$(build_scenario_json "$target_len" "$max_context_len" "true" "$server_args" "" "$metrics_json")"
|
||||||
|
|
||||||
|
log "sglang target_len=${target_len} succeeded"
|
||||||
|
stop_server
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
mkdir -p "$RAW_DIR" "$LOG_DIR"
|
||||||
|
|
||||||
|
RESULT_JSON="${RESULT_ROOT}/results.json"
|
||||||
|
write_metadata_json \
|
||||||
|
"$RESULT_JSON" \
|
||||||
|
"$EXPERIMENT_NAME" \
|
||||||
|
"$RUN_ID" \
|
||||||
|
"$MODEL_PATH" \
|
||||||
|
"sglang" \
|
||||||
|
"sglang-xpu" \
|
||||||
|
"$HARDWARE" \
|
||||||
|
"$ACCELERATOR" \
|
||||||
|
"$CHIP" \
|
||||||
|
"experiments/${EXPERIMENT_NAME}/run_bench.sh" \
|
||||||
|
"" \
|
||||||
|
"P800 max context length exploration for DeepSeek-V4-Flash-INT8"
|
||||||
|
|
||||||
|
# Embed config.
|
||||||
|
python3 - "$RESULT_JSON" <<'PY'
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
path = sys.argv[1]
|
||||||
|
with open(path, "r", encoding="utf-8") as f:
|
||||||
|
data = json.load(f)
|
||||||
|
|
||||||
|
data["config"] = {
|
||||||
|
"tp": 8,
|
||||||
|
"ep": 8,
|
||||||
|
"xpu_visible_devices": "0,1,2,3,4,5,6,7",
|
||||||
|
"output_len": 1,
|
||||||
|
"num_prompts": 1,
|
||||||
|
"max_concurrency": 1,
|
||||||
|
"request_rate": 10000,
|
||||||
|
"context_pad": 1024,
|
||||||
|
}
|
||||||
|
with open(path, "w", encoding="utf-8") as f:
|
||||||
|
json.dump(data, f, indent=2, ensure_ascii=False)
|
||||||
|
PY
|
||||||
|
|
||||||
|
stop_server
|
||||||
|
|
||||||
|
for target_len in "${CANDIDATE_LENS[@]}"; do
|
||||||
|
if ! run_length "$target_len"; then
|
||||||
|
log "stopping exploration after first failure at target_len=${target_len}"
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
log "parsing results"
|
||||||
|
python3 "${SCRIPT_DIR}/parse_results.py" "$RESULT_ROOT" >> "${LOG_DIR}/parse.log" 2>&1 || {
|
||||||
|
log "WARNING: parse_results.py failed; see ${LOG_DIR}/parse.log"
|
||||||
|
}
|
||||||
|
|
||||||
|
log "all results saved to ${RESULT_ROOT}"
|
||||||
133
experiments/dsv4_p800_max_context_length/start_server.sh
Executable file
133
experiments/dsv4_p800_max_context_length/start_server.sh
Executable file
@ -0,0 +1,133 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Start P800 SGLang INT8 server for a specific target context length.
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
TARGET_LEN="${1:-}"
|
||||||
|
if [[ -z "$TARGET_LEN" ]]; then
|
||||||
|
echo "Usage: $0 <target_input_length_in_tokens>"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${SCRIPT_DIR}/config.env"
|
||||||
|
|
||||||
|
MAX_CONTEXT_LEN=$(( TARGET_LEN + CONTEXT_PAD ))
|
||||||
|
CONTAINER_NAME="${CONTAINER_NAME:-sglang-dsv4-flash}"
|
||||||
|
DOCKER_IMAGE="${DOCKER_IMAGE:-iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202}"
|
||||||
|
PATCH_ROOT="${PATCH_ROOT:-${ROOT_DIR}/platforms/patches/kunlun_p800}"
|
||||||
|
RESULT_ROOT="${RESULT_ROOT:-/tmp/${EXPERIMENT}}"
|
||||||
|
SERVER_LOG="${RESULT_ROOT}/logs/server_${TARGET_LEN}.outer.log"
|
||||||
|
|
||||||
|
mkdir -p "$(dirname "$SERVER_LOG")"
|
||||||
|
|
||||||
|
log "starting P800 SGLang INT8 server (target_len=${TARGET_LEN}, context-length=${MAX_CONTEXT_LEN})"
|
||||||
|
log "model: ${MODEL_PATH}"
|
||||||
|
log "port: ${PORT}"
|
||||||
|
log "server log: ${SERVER_LOG}"
|
||||||
|
|
||||||
|
# Stop any existing container with the same name.
|
||||||
|
docker rm -f "$CONTAINER_NAME" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Build device args.
|
||||||
|
device_args=""
|
||||||
|
for i in 0 1 2 3 4 5 6 7; do
|
||||||
|
device_args="${device_args} --device /dev/xpu${i}:/dev/xpu${i}"
|
||||||
|
done
|
||||||
|
device_args="${device_args} --device /dev/xpuctrl:/dev/xpuctrl"
|
||||||
|
|
||||||
|
# Environment variables required by the P800 SGLang INT8 image.
|
||||||
|
env_args=(
|
||||||
|
-e XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
-e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
|
||||||
|
-e CUDA_DEVICE_ORDER=OAM_ID
|
||||||
|
-e SGLANG_USE_TRANSFORMERS_V5_TOKENIZER=1
|
||||||
|
-e XMLIR_FORCE_USE_XPU_GRAPH=1
|
||||||
|
-e SGLANG_DSV4_MODE=2604
|
||||||
|
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
||||||
|
-e SGLANG_NSA_DUAL_STREAM=true
|
||||||
|
-e SGLANG_NSA_QUANT_WQ_B_WK=false
|
||||||
|
-e SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1
|
||||||
|
-e SGLANG_OPT_DEEPGEMM_HC_PRENORM=false
|
||||||
|
-e SGLANG_OPT_USE_TILELANG_MHC_PRE=1
|
||||||
|
-e SGLANG_OPT_USE_TILELANG_MHC_POST=1
|
||||||
|
-e SGLANG_CLEAN_REQUEST_WHEN_RETRACT=1
|
||||||
|
-e SGLANG_SET_CPU_AFFINITY=1
|
||||||
|
-e SGLANG_OPT_USE_KLX_TOPK_KERNEL=1
|
||||||
|
-e XSGL_INTERTYPE_BFP16=1
|
||||||
|
-e ENABLE_FAST_BFP16_ATTN=1
|
||||||
|
-e XSGL_USE_DEEP_GEMM_BMM=1
|
||||||
|
-e XSGL_XDNN_QUANT=1
|
||||||
|
-e XSGL_FUSE_RMS_NORM_QUANT=1
|
||||||
|
-e XSGL_TRANSPOSE_MATMUL_WEIGHT=1
|
||||||
|
-e XINFER_QUANT_SDNN=1
|
||||||
|
-e XSGL_USE_MOE_SIGMOID_GROUP_TOPK_NORM=1
|
||||||
|
-e XSGL_EARLY_FIRST_TOKEN=1
|
||||||
|
-e XSGL_ENABLE_TGEMM_FP16=1
|
||||||
|
-e SGLANG_ENABLE_SPEC_V2=True
|
||||||
|
-e SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||||
|
-e PYTHONDONTWRITEBYTECODE=1
|
||||||
|
-e XTORCH_OPS_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/xtorch_ops
|
||||||
|
-e XPU_RUNTIME_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/xcudart/lib
|
||||||
|
-e BKCL_TREE_THRESHOLD=1048576
|
||||||
|
-e CUDA_ENABLE_P2P_NO_UVA=1
|
||||||
|
-e NCCL_IB_GID_INDEX=3
|
||||||
|
-e IS_DSV4=1
|
||||||
|
-e MC_CUSTOM_TOPO_JSON=/workspace/nic_priority_matrix_test.json
|
||||||
|
# INT8 specific
|
||||||
|
-e SGLANG_DSV4_FP4_EXPERTS=false
|
||||||
|
-e SGLANG_APPLY_CONFIG_BACKUP=auto
|
||||||
|
-e BKCL_ENABLE_XDR=1
|
||||||
|
-e BKCL_RDMA_NICS=eth1,eth1,eth3,eth3,eth5,eth5,eth7,eth7
|
||||||
|
-e BKCL_RDMA_VERBS=1
|
||||||
|
-e XSGL_INT8_LM_HEAD=1
|
||||||
|
-e SGLANG_P800_ALL_GATHER_FALLBACK=0
|
||||||
|
)
|
||||||
|
|
||||||
|
# Launch args. Based on the proven P800 INT8 command from server_docker.sh,
|
||||||
|
# but with dynamic --context-length and --max-prefill-tokens for the max-context
|
||||||
|
# test. Keep --max-running-requests at 64 because setting it to 1 causes
|
||||||
|
# req_to_token_pool allocation failures in this image.
|
||||||
|
launch_args="--host 0.0.0.0 --port ${PORT} --model-path /models --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.8 --max-prefill-tokens ${MAX_CONTEXT_LEN} --max-running-requests 64 --tensor-parallel-size 8 --ep-size 8 --disable-shared-experts-fusion --quantization w8a8_int8 --kv-cache-dtype float16 --disable-piecewise-cuda-graph --cuda-graph-max-bs 32 --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --constrained-json-disable-any-whitespace --enable-metrics --enable-request-time-stats-logging --context-length ${MAX_CONTEXT_LEN}"
|
||||||
|
|
||||||
|
# Base64-encode the bootstrap command to avoid host-shell quoting issues.
|
||||||
|
server_cmd=$(cat <<EOF
|
||||||
|
cd /workspace
|
||||||
|
find /root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
|
||||||
|
/root/miniconda/envs/python310_torch25_cuda/bin/pip install --upgrade safetensors -q
|
||||||
|
/root/miniconda/envs/python310_torch25_cuda/bin/pip install https://files.pythonhosted.org/packages/14/8b/2a1333a6455c6fad401c2285dee6f58016c55b1cb44cae3a31f8a9cc7d83/apache_tvm_ffi-0.1.0b2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl -q
|
||||||
|
/root/miniconda/envs/python310_torch25_cuda/bin/python -c "import torch; torch.float8_e8m0fnu = torch.uint8; import runpy, sys; sys.argv[0] = 'sglang.launch_server'; runpy.run_module('sglang.launch_server', run_name='__main__')" ${launch_args}
|
||||||
|
EOF
|
||||||
|
)
|
||||||
|
server_cmd_b64=$(printf '%s' "$server_cmd" | base64 -w0)
|
||||||
|
|
||||||
|
patch_mounts=(
|
||||||
|
-v "${PATCH_ROOT}/nic_priority_matrix_test.json:/workspace/nic_priority_matrix_test.json:ro"
|
||||||
|
-v "${PATCH_ROOT}/dummy_sharegpt.json:/workspace/dummy_sharegpt.json:ro"
|
||||||
|
)
|
||||||
|
|
||||||
|
docker run -d \
|
||||||
|
--name "${CONTAINER_NAME}" \
|
||||||
|
--privileged \
|
||||||
|
--network host \
|
||||||
|
--ipc host \
|
||||||
|
${device_args} \
|
||||||
|
-v "${MODEL_PATH}:/models:ro" \
|
||||||
|
-v "${MODEL_PATH}:${MODEL_PATH}:ro" \
|
||||||
|
"${patch_mounts[@]}" \
|
||||||
|
"${env_args[@]}" \
|
||||||
|
"${DOCKER_IMAGE}" \
|
||||||
|
bash -c "echo '${server_cmd_b64}' | base64 -d | bash" \
|
||||||
|
>> "${SERVER_LOG}" 2>&1
|
||||||
|
|
||||||
|
log "container ${CONTAINER_NAME} started, waiting for health"
|
||||||
|
if health_check 127.0.0.1 "$PORT" 600; then
|
||||||
|
log "container ${CONTAINER_NAME} is healthy"
|
||||||
|
else
|
||||||
|
log "ERROR: container ${CONTAINER_NAME} failed health check"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-043404",
|
||||||
|
"timestamp": "2026-07-08T04:34:04+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-043830",
|
||||||
|
"timestamp": "2026-07-08T04:38:30+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-044535",
|
||||||
|
"timestamp": "2026-07-08T04:45:35+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-045327",
|
||||||
|
"timestamp": "2026-07-08T04:53:27+00:00",
|
||||||
|
"model": "/data1/models/deepseek-ai/DeepSeek-V4-Flash",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-045554",
|
||||||
|
"timestamp": "2026-07-08T04:55:54+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-045616",
|
||||||
|
"timestamp": "2026-07-08T04:56:16+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-050025",
|
||||||
|
"timestamp": "2026-07-08T05:00:25+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,16 @@
|
|||||||
|
# Benchmark Report: dsv4_p800_sglang
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
- **Run ID**: 20260708-050713
|
||||||
|
- **Timestamp**: 2026-07-08T05:07:13+00:00
|
||||||
|
- **Chip/Accelerator**: Kunlun P800 XPU / kunlun_p800
|
||||||
|
- **Engine/Backend**: sglang-xpu / sglang
|
||||||
|
- **Hardware**: 8x Kunlun P800 XPU
|
||||||
|
- **Model**: /data1/models/DeepSeek-V4-Flash-INT8
|
||||||
|
- **Git Commit**: 2c332ff
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Scenario | Conc | In/Out | Success | Failed | Req/s | OutTok/s | TTFT p50 | TTFT p99 | TPOT p50 | TPOT p99 | E2E p99 |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-050713",
|
||||||
|
"timestamp": "2026-07-08T05:07:13+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,16 @@
|
|||||||
|
# Benchmark Report: dsv4_p800_sglang
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
- **Run ID**: 20260708-050722
|
||||||
|
- **Timestamp**: 2026-07-08T05:07:22+00:00
|
||||||
|
- **Chip/Accelerator**: Kunlun P800 XPU / kunlun_p800
|
||||||
|
- **Engine/Backend**: sglang-xpu / sglang
|
||||||
|
- **Hardware**: 8x Kunlun P800 XPU
|
||||||
|
- **Model**: /data1/models/DeepSeek-V4-Flash-INT8
|
||||||
|
- **Git Commit**: 2c332ff
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Scenario | Conc | In/Out | Success | Failed | Req/s | OutTok/s | TTFT p50 | TTFT p99 | TPOT p50 | TPOT p99 | E2E p99 |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-050722",
|
||||||
|
"timestamp": "2026-07-08T05:07:22+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,16 @@
|
|||||||
|
# Benchmark Report: dsv4_p800_sglang
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
- **Run ID**: 20260708-050858
|
||||||
|
- **Timestamp**: 2026-07-08T05:08:58+00:00
|
||||||
|
- **Chip/Accelerator**: Kunlun P800 XPU / kunlun_p800
|
||||||
|
- **Engine/Backend**: sglang-xpu / sglang
|
||||||
|
- **Hardware**: 8x Kunlun P800 XPU
|
||||||
|
- **Model**: /data1/models/DeepSeek-V4-Flash-INT8
|
||||||
|
- **Git Commit**: 2c332ff
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Scenario | Conc | In/Out | Success | Failed | Req/s | OutTok/s | TTFT p50 | TTFT p99 | TPOT p50 | TPOT p99 | E2E p99 |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-050858",
|
||||||
|
"timestamp": "2026-07-08T05:08:58+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,16 @@
|
|||||||
|
# Benchmark Report: dsv4_p800_sglang
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
- **Run ID**: 20260708-051023
|
||||||
|
- **Timestamp**: 2026-07-08T05:10:23+00:00
|
||||||
|
- **Chip/Accelerator**: Kunlun P800 XPU / kunlun_p800
|
||||||
|
- **Engine/Backend**: sglang-xpu / sglang
|
||||||
|
- **Hardware**: 8x Kunlun P800 XPU
|
||||||
|
- **Model**: /data1/models/DeepSeek-V4-Flash-INT8
|
||||||
|
- **Git Commit**: 2c332ff
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Scenario | Conc | In/Out | Success | Failed | Req/s | OutTok/s | TTFT p50 | TTFT p99 | TPOT p50 | TPOT p99 | E2E p99 |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-051023",
|
||||||
|
"timestamp": "2026-07-08T05:10:23+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-051200",
|
||||||
|
"timestamp": "2026-07-08T05:12:00+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "2c332ff",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,17 @@
|
|||||||
|
# Benchmark Report: dsv4_p800_sglang
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
- **Run ID**: 20260708-051316
|
||||||
|
- **Timestamp**: 2026-07-08T05:13:16+00:00
|
||||||
|
- **Chip/Accelerator**: Kunlun P800 XPU / kunlun_p800
|
||||||
|
- **Engine/Backend**: sglang-xpu / sglang
|
||||||
|
- **Hardware**: 8x Kunlun P800 XPU
|
||||||
|
- **Model**: /data1/models/DeepSeek-V4-Flash-INT8
|
||||||
|
- **Git Commit**: 2c332ff
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Scenario | Conc | In/Out | Success | Failed | Req/s | OutTok/s | TTFT p50 | TTFT p99 | TPOT p50 | TPOT p99 | E2E p99 |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
|
| c32_i512_o256 | 32 | 512/256 | 10 | 0 | 1.87 | 262.65 | 506.4399079710711 | 507.2712521097855 | 21.948128755653826 | 29.154062995221466 | 5314.590020017349 |
|
||||||
2482
experiments/dsv4_p800_sglang/results/20260708-051316/results.json
Normal file
2482
experiments/dsv4_p800_sglang/results/20260708-051316/results.json
Normal file
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,16 @@
|
|||||||
|
# Benchmark Report: dsv4_p800_sglang
|
||||||
|
|
||||||
|
## Metadata
|
||||||
|
|
||||||
|
- **Run ID**: 20260708-052659
|
||||||
|
- **Timestamp**: 2026-07-08T05:26:59+00:00
|
||||||
|
- **Chip/Accelerator**: Kunlun P800 XPU / kunlun_p800
|
||||||
|
- **Engine/Backend**: sglang-xpu / sglang
|
||||||
|
- **Hardware**: 8x Kunlun P800 XPU
|
||||||
|
- **Model**: /data1/models/DeepSeek-V4-Flash-INT8
|
||||||
|
- **Git Commit**: 4ef6543
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
| Scenario | Conc | In/Out | Success | Failed | Req/s | OutTok/s | TTFT p50 | TTFT p99 | TPOT p50 | TPOT p99 | E2E p99 |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-052659",
|
||||||
|
"timestamp": "2026-07-08T05:26:59+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "4ef6543",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
@ -0,0 +1,20 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"experiment": "dsv4_p800_sglang",
|
||||||
|
"run_id": "20260708-054332",
|
||||||
|
"timestamp": "2026-07-08T05:43:32+00:00",
|
||||||
|
"model": "/data1/models/DeepSeek-V4-Flash-INT8",
|
||||||
|
"backend": "sglang",
|
||||||
|
"engine": "sglang-xpu",
|
||||||
|
"hardware": "8x Kunlun P800 XPU",
|
||||||
|
"accelerator": "Kunlun P800 XPU",
|
||||||
|
"chip": "kunlun_p800",
|
||||||
|
"script": "experiments/dsv4_p800_sglang/run_bench.sh",
|
||||||
|
"env": "",
|
||||||
|
"git_commit": "4ef6543",
|
||||||
|
"git_dirty": "dirty",
|
||||||
|
"description": "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
|
||||||
|
},
|
||||||
|
"config": {},
|
||||||
|
"scenarios": []
|
||||||
|
}
|
||||||
Loading…
x
Reference in New Issue
Block a user