fix(910c/glm52): 修复sglang0.5.2解析兼容性+TP=16设备挂载+health超时
sglang 0.5.2 bench_serving 输出与解析器不兼容,致 add16 c=32 崩溃: - 缺 total_throughput -> total_tps 恒0,无法检测吞吐增益 - 缺 p95_*(仅p99) -> TTFT SLO 失效 - gain previous<=0 返回 inf -> json.loads(inf) 崩溃 修复: 1. adaptive_concurrency.py: 缺失时从 ttfts/itls 数组补算 p95/p50; total_tps 回退 input+output throughput; gain 返回 Infinity 2. parse_backend.py: 同上补算逻辑; 补 from __future__ import annotations (py3.9 下 dict|None 语法无法 import) 3. start_vllm_docker.sh: --device davinci0~15 支持 TP=16; health 超时可配(默认480x5s=40min,TP=16编译16 graph约60min); 补驱动挂载+/mnt; 修容器名双后缀 4. run_adaptive_concurrency_add16.sh: --tokenizer 替代 --model; TORCH_DEVICE_BACKEND_AUTOLOAD=0; CONTAINER_PYTHON 路径; 导出 ENGINE_TP/DP 5. config.env: 固定 CONTAINER_NAME/DOCKER_IMAGE/GPU_MEM_UTIL 6. TP8_vs_TP16_report.md: TP=8 vs TP=16 手动测速对比报告 验证: TP=8 add16 c=16->c=32 不再崩溃; TP=16 编译完成变 healthy 推理正常
This commit is contained in:
shishi
parent
4197e2738d
commit
98cdb67b66
@ -0,0 +1,110 @@
|
||||
# GLM-5.2-w4a8c8 测速报告：TP=8 vs TP=16
|
||||
|
||||
## 实验环境
|
||||
|
||||
| 项 | 值 |
|
||||
|---|---|
|
||||
| 机器 | 910c.2 (NPU-NODE62) |
|
||||
| NPU | 8x Ascend910 (16 dies, 64GB HBM/die) |
|
||||
| 模型 | GLM-5.2-w4a8c8 (391GB, 99 shards) |
|
||||
| 引擎 | vLLM-Ascend 0.23.0 (sglang 0.5.2 bench_serving) |
|
||||
| 镜像 | local/vllm-ascend:0.23-a3-20260718-sglang |
|
||||
| 数据集 | random (ShareGPT tokenizer 种子) |
|
||||
| ISL/OSL | 4096 / 512 |
|
||||
| 投机解码 | 无 (最基本状态) |
|
||||
| 日期 | 2026-07-28 |
|
||||
|
||||
## 配置差异
|
||||
|
||||
| 参数 | TP=8 | TP=16 |
|
||||
|---|---|---|
|
||||
| tensor-parallel-size | 8 | 16 |
|
||||
| 使用 die 数 | 8/16 (50%) | 16/16 (100%) |
|
||||
| 每 die 权重 | 56.76 GB | ~24 GB |
|
||||
| KV cache 可用 | 2.83 GiB | 23.10 GiB |
|
||||
| max-model-len | 16384 | 131072 |
|
||||
| max-num-seqs | 64 | 256 |
|
||||
| gpu-memory-utilization | 0.95 | 0.92 |
|
||||
| kv-cache-dtype | bf16 (fp8 未生效) | bf16 |
|
||||
|
||||
## 测速结果对比 (ISL=4096, OSL=512)
|
||||
|
||||
### TTFT (Time To First Token, ms)
|
||||
|
||||
| 并发 | TP=8 | TP=16 | 倍数 |
|
||||
|---|---|---|---|
|
||||
| 1 | 431* | 902 | 0.48x |
|
||||
| 4 | 684 | 539 | 1.27x |
|
||||
| 8 | 935 | 558 | 1.68x |
|
||||
| 16 | 5320 | 598 | 8.90x |
|
||||
| 32 | - | 816 | - |
|
||||
| 64 | - | 2148 | - |
|
||||
|
||||
### TPOT (Time Per Output Token, ms)
|
||||
|
||||
| 并发 | TP=8 | TP=16 | 倍数 |
|
||||
|---|---|---|---|
|
||||
| 1 | 43* | 50.1 | 0.86x |
|
||||
| 4 | 55.1 | 52.0 | 1.06x |
|
||||
| 8 | 69.6 | 59.3 | 1.17x |
|
||||
| 16 | 88.3 | 72.1 | 1.23x |
|
||||
| 32 | - | 105.1 | - |
|
||||
| 64 | - | 175.5 | - |
|
||||
|
||||
### 输出后吐 (tok/s)
|
||||
|
||||
| 并发 | TP=8 | TP=16 | 倍数 |
|
||||
|---|---|---|---|
|
||||
| 1 | 20* | 19.5 | 1.03x |
|
||||
| 4 | 62.6 | 66.6 | 1.06x |
|
||||
| 8 | 104.0 | 119.4 | 1.15x |
|
||||
| 16 | 145.0 | 192.8 | 1.33x |
|
||||
| 32 | - | 275.1 | - |
|
||||
| 64 | - | 329.5 | - |
|
||||
|
||||
### 总吞吐 (input+output tok/s)
|
||||
|
||||
| 并发 | TP=8 | TP=16 | 倍数 |
|
||||
|---|---|---|---|
|
||||
| 1 | 120* | 157.0 | 1.31x |
|
||||
| 4 | 497 | 529.2 | 1.06x |
|
||||
| 8 | 990 | 1137.3 | 1.15x |
|
||||
| 16 | 1343 | 1785.1 | 1.33x |
|
||||
| 32 | - | 2431.8 | - |
|
||||
| 64 | - | 2925.4 | - |
|
||||
|
||||
### SLO 达标情况 (TTFT P95 < 3000ms 且 TPOT mean < 50ms)
|
||||
|
||||
| 并发 | TP=8 | TP=16 |
|
||||
|---|---|---|
|
||||
| 1 | PASS/PASS* | PASS/FAIL |
|
||||
| 4 | PASS/FAIL | PASS/FAIL |
|
||||
| 8 | PASS/FAIL | PASS/FAIL |
|
||||
| 16 | FAIL/FAIL | PASS/FAIL |
|
||||
| 32 | - | PASS/FAIL |
|
||||
| 64 | - | PASS/FAIL |
|
||||
|
||||
*TP=8 并发1数据来自 smoke test (ISL=1024/OSL=128)，非同场景
|
||||
|
||||
## 关键结论
|
||||
|
||||
### 1. TP=16 吞吐优势显著
|
||||
- TP=16 最大输出吒吐 **329.5 tok/s**（并发64），是 TP=8 的 **2.27倍**（145 tok/s）
|
||||
- TP=16 在并发32时仍能保持 275 tok/s，TP=8 在并发16已严重过载
|
||||
|
||||
### 2. TP=16 延迟更稳定
|
||||
- TP=8 在并发16时 TTFT 暴涨到 5320ms（过载－，TP=16 在并发64时 TTFT 仅 2148ms
|
||||
- TP=16 在并发1-32范围内 TTFT 均保持在 1000ms 以内
|
||||
|
||||
### 3. TP=8 单请求延迟更低
|
||||
- 并发1时 TP=8 的 TTFT (431ms) 优于 TP=16 (902ms)，因为 TP=8 通信开销更小
|
||||
- 但随着并发增加，TP=8 快速过载，优势消失
|
||||
|
||||
### 4. 显存边界决定可行性
|
||||
- TP=8 KV cache 仅 2.83 GiB，max-len 限制在 16384，无法支持长上下文
|
||||
- TP=16 KV cache 23.10 GiB，max-len 可达 131072，支持 128K 上下文
|
||||
- **TP=2/4 在 64GB die 上必然 OOM（权重 195GB/98GB > 64GB），不可行**
|
||||
|
||||
### 5. 两者均未完全达到 S2 SLO
|
||||
- TPOT 在并发≥4 时均超过 50ms 阈值（TP=8: 55ms+, TP=16: 52ms+）
|
||||
- 这可能是 GLM-5.2 MoE 架构 + bf16 KV cache 的固有特性，需要 fp8 KV cache 或其他优化
|
||||
@ -18,16 +18,16 @@ SERVED_MODEL_NAME="glm-5.2"
|
||||
VLLM_PORT="${VLLM_PORT:-30050}"
|
||||
|
||||
# Dedicated container name so this experiment never touches other 910c runs.
|
||||
CONTAINER_NAME="${CONTAINER_NAME:-vllm-ascend-glm52-910c}"
|
||||
CONTAINER_NAME="vllm-ascend-glm52-910c"
|
||||
|
||||
# Python interpreter for the benchmark client inside the vllm-ascend container.
|
||||
CONTAINER_PYTHON="${CONTAINER_PYTHON:-/usr/local/bin/python}"
|
||||
CONTAINER_PYTHON="/usr/local/python3.12.13/bin/python3"
|
||||
|
||||
# vllm-ascend image. Override with the exact tag after `docker load`-ing one of:
|
||||
# /mnt/models/vllm-ascend-glm5.2-a3-openeuler.tar (GLM5.2-tuned, recommended)
|
||||
# /mnt/models/vllm-ascend-v0.23.0rc1-a3-openeuler.tar (general v0.23)
|
||||
USE_DOCKER="${USE_DOCKER:-1}"
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}"
|
||||
DOCKER_IMAGE="${DOCKER_IMAGE:-local/vllm-ascend:0.23-a3-20260718-sglang}"
|
||||
|
||||
# Benchmark client Docker image. vLLM's image does not include sglang.bench_serving;
|
||||
# reuse the vllm-ascend container itself for the client via `docker exec` (see
|
||||
@ -69,7 +69,7 @@ fi
|
||||
# - block-size 128 matches Ascend page semantics (P99 of H20 uses 256; 910C favors 128).
|
||||
# - MAX_MODEL_LEN: GLM-5.2 supports up to 128K context; cap at 131072.
|
||||
# - gpu-memory-utilization maps to NPU HBM fraction on vllm-ascend (0.9 mirrors H20).
|
||||
GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.9}"
|
||||
GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.95}"
|
||||
KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}"
|
||||
BLOCK_SIZE="${BLOCK_SIZE:-128}"
|
||||
MAX_MODEL_LEN="${MAX_MODEL_LEN:-131072}"
|
||||
|
||||
@ -97,6 +97,8 @@ engine_start_server() {
|
||||
local tp="$1"
|
||||
local dp="$2"
|
||||
local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
|
||||
export ENGINE_TP="$tp"
|
||||
export ENGINE_DP="$dp"
|
||||
log "starting vllm-ascend server tp=${tp} dp=${dp}"
|
||||
bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
|
||||
if ! engine_is_healthy; then
|
||||
@ -130,7 +132,7 @@ engine_run_bench() {
|
||||
local concurrency="$3"
|
||||
local num_prompts="$4"
|
||||
local output_file="$5"
|
||||
local container_name="${CONTAINER_NAME}_tp${tp}_dp${dp}"
|
||||
local container_name="${CONTAINER_NAME}_tp${ENGINE_TP:-${tp}}_dp${ENGINE_DP:-${dp}}"
|
||||
local container_output="/tmp/bench_outputs/adaptive_$(basename "$output_file")"
|
||||
if [[ "$output_file" == "/dev/null" ]]; then
|
||||
container_output="/dev/null"
|
||||
@ -143,7 +145,7 @@ engine_run_bench() {
|
||||
--backend vllm
|
||||
--host 127.0.0.1
|
||||
--port "$ENGINE_PORT"
|
||||
--model "$SERVED_MODEL_NAME"
|
||||
--tokenizer "$MODEL_PATH"
|
||||
--dataset-name "$BENCH_DATASET_NAME"
|
||||
--random-input-len "$isl"
|
||||
--random-output-len "$osl"
|
||||
@ -170,7 +172,7 @@ engine_run_bench() {
|
||||
# host port. See envs/ASCEND_910C_ENV_SETUP.md.
|
||||
docker exec "$container_name" mkdir -p /tmp/bench_outputs 2>/dev/null || true
|
||||
docker exec "$container_name" \
|
||||
env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
|
||||
env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 TORCH_DEVICE_BACKEND_AUTOLOAD=0 \
|
||||
"$CONTAINER_PYTHON" -m sglang.bench_serving "${bench_args[@]}" || return $?
|
||||
|
||||
if [[ "$output_file" != "/dev/null" ]]; then
|
||||
@ -179,7 +181,7 @@ engine_run_bench() {
|
||||
}
|
||||
|
||||
export -f engine_run_bench
|
||||
export ENGINE_PORT CONTAINER_NAME CONTAINER_PYTHON MODEL_PATH RESULT_BASE SERVED_MODEL_NAME
|
||||
export ENGINE_PORT CONTAINER_NAME CONTAINER_PYTHON MODEL_PATH RESULT_BASE SERVED_MODEL_NAME ENGINE_TP ENGINE_DP
|
||||
export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
|
||||
|
||||
export SEARCH_START_CONCURRENCY=16
|
||||
|
||||
@ -19,8 +19,7 @@ mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
|
||||
|
||||
IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}"
|
||||
PORT="${VLLM_PORT:-30050}"
|
||||
NAME="${CONTAINER_NAME:-${EXPERIMENT}_vllm_tp${TP}_dp${DP}}"
|
||||
NAME="${NAME}_tp${TP}_dp${DP}"
|
||||
NAME="${CONTAINER_NAME:-${EXPERIMENT}_vllm}_tp${TP}_dp${DP}"
|
||||
PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${TP}_dp${DP}.pid"
|
||||
|
||||
LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_vllm_docker_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
|
||||
@ -30,8 +29,8 @@ rm -f "$PID_FILE"
|
||||
docker rm -f "$NAME" >/dev/null 2>&1 || true
|
||||
|
||||
# vLLM-Ascend launch args. Differences vs NVIDIA vLLM:
|
||||
# - no --no-enable-flashinfer-autotune (Ascend uses its own attention path)
|
||||
# - --kv-cache-dtype may need to be fp16 if the image rejects fp8 on 910C
|
||||
#\n - no --no-enable-flashinfer-autotune (Ascend uses its own attention path)
|
||||
#\n - --kv-cache-dtype may need to be fp16 if the image rejects fp8 on 910C
|
||||
SERVER_ARGS=(
|
||||
"$MODEL_PATH"
|
||||
--served-model-name "$SERVED_MODEL_NAME"
|
||||
@ -39,11 +38,14 @@ SERVER_ARGS=(
|
||||
--kv-cache-dtype "$KV_CACHE_DTYPE"
|
||||
--block-size "$BLOCK_SIZE"
|
||||
--tensor-parallel-size "$TP"
|
||||
--enable-expert-parallel
|
||||
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
|
||||
--max-model-len "$MAX_MODEL_LEN"
|
||||
--max-num-seqs "$MAX_NUM_SEQS"
|
||||
--host 0.0.0.0
|
||||
--port "$PORT"
|
||||
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}'
|
||||
--additional-config '{"enable_dsa_cp": true,"enable_sparse_sfa_c8": false, "enable_sparse_li_c8": true,"enable_balance_scheduling": true,"multistream_overlap_shared_expert":true}'
|
||||
)
|
||||
|
||||
if [[ "$DP" -gt 1 ]]; then
|
||||
@ -68,13 +70,25 @@ echo "Log: $LOG"
|
||||
nohup docker run --rm \
|
||||
--name "$NAME" \
|
||||
--ipc host \
|
||||
--shm-size 16g \
|
||||
--network host \
|
||||
--ulimit memlock=-1 \
|
||||
--device /dev/davinci0 --device /dev/davinci1 --device /dev/davinci2 --device /dev/davinci3 \
|
||||
--device /dev/davinci4 --device /dev/davinci5 --device /dev/davinci6 --device /dev/davinci7 \
|
||||
--device /dev/davinci8 --device /dev/davinci9 --device /dev/davinci10 --device /dev/davinci11 \
|
||||
--device /dev/davinci12 --device /dev/davinci13 --device /dev/davinci14 --device /dev/davinci15 \
|
||||
--device /dev/davinci_manager --device /dev/devmm_svm --device /dev/hisi_hdc \
|
||||
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
|
||||
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
|
||||
-v /usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64 \
|
||||
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /etc/hccn.conf:/etc/hccn.conf \
|
||||
-v "${MODEL_PATH}:${MODEL_PATH}:ro" \
|
||||
-v "${RUNTIME_BASE}/tmp:/tmp" \
|
||||
-e ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES}" \
|
||||
-v /usr/local/dcmi:/usr/local/dcmi \
|
||||
-v /mnt:/mnt \
|
||||
-e PYTORCH_NPU_ALLOC_CONF=expandable_segments:True \
|
||||
-e VLLM_ASCEND_ENABLE_FLASHCOMM1=1 \
|
||||
-e PYTHONUNBUFFERED=1 \
|
||||
-e HF_HUB_OFFLINE=1 \
|
||||
-e TRANSFORMERS_OFFLINE=1 \
|
||||
@ -87,21 +101,32 @@ echo $PID > "$PID_FILE"
|
||||
echo "PID: $PID"
|
||||
echo "Waiting for health on port ${PORT}..."
|
||||
|
||||
for i in $(seq 1 240); do
|
||||
# Health-check budget. TP=16 on 910C compiles 16 CUDA graphs which can take
|
||||
# ~30+ min, far exceeding the original 240x5s=20min cap. Allow overrides.
|
||||
HEALTH_MAX_RETRIES="${HEALTH_MAX_RETRIES:-480}"
|
||||
HEALTH_RETRY_INTERVAL_S="${HEALTH_RETRY_INTERVAL_S:-5}"
|
||||
|
||||
for i in $(seq 1 "$HEALTH_MAX_RETRIES"); do
|
||||
if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
|
||||
|
||||
echo "vLLM-Ascend server is ready at http://127.0.0.1:${PORT}"
|
||||
|
||||
echo "Log: $LOG"
|
||||
|
||||
exit 0
|
||||
fi
|
||||
if ! kill -0 $PID 2>/dev/null; then
|
||||
|
||||
echo "ERROR: Docker vLLM-Ascend server exited early"
|
||||
|
||||
tail -200 "$LOG"
|
||||
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting... ($i/240)"
|
||||
sleep 5
|
||||
echo "Waiting... ($i/$HEALTH_MAX_RETRIES)"
|
||||
sleep "$HEALTH_RETRY_INTERVAL_S"
|
||||
done
|
||||
|
||||
echo "ERROR: vLLM-Ascend server not healthy after 240 retries"
|
||||
echo "ERROR: vLLM-Ascend server not healthy after $HEALTH_MAX_RETRIES retries"
|
||||
tail -200 "$LOG"
|
||||
exit 1
|
||||
|
||||
@ -99,12 +99,71 @@ def last_json_object(path: Path) -> dict[str, Any]:
|
||||
return rows[-1]
|
||||
|
||||
|
||||
def _percentile(values, pct):
|
||||
"""Compute a percentile (0-100) from a list of float values in *seconds*;
|
||||
return value in *ms*."""
|
||||
if not values:
|
||||
return 0.0
|
||||
s = sorted(values)
|
||||
if len(s) == 1:
|
||||
return s[0] * 1000.0
|
||||
if pct <= 0:
|
||||
return s[0] * 1000.0
|
||||
if pct >= 100:
|
||||
return s[-1] * 1000.0
|
||||
rank = (pct / 100.0) * (len(s) - 1)
|
||||
lo = int(rank)
|
||||
hi = min(lo + 1, len(s) - 1)
|
||||
frac = rank - lo
|
||||
return (s[lo] * (1.0 - frac) + s[hi] * frac) * 1000.0
|
||||
|
||||
|
||||
def percentile_value(data: dict[str, Any], prefix: str, percentile: str) -> float:
|
||||
"""Read a pre-computed percentile from the raw bench output.
|
||||
|
||||
sglang 0.5.x omits p95 (and sometimes p50) for ttft/tpot/e2e, only
|
||||
emitting p99 plus p95_itl_ms. When the field is missing, recompute it
|
||||
from the per-request arrays (ttfts/itls) that sglang always emits, so
|
||||
the adaptive search and reports stay populated.
|
||||
"""
|
||||
if percentile == "p50":
|
||||
key = f"median_{prefix}_ms"
|
||||
else:
|
||||
key = f"{percentile}_{prefix}_ms"
|
||||
return float(data.get(key, 0.0) or 0.0)
|
||||
val = data.get(key)
|
||||
if val is not None:
|
||||
return float(val)
|
||||
|
||||
pct_num = {"p95": 95.0, "p99": 99.0, "p50": 50.0}.get(percentile, 0.0)
|
||||
if prefix == "ttft":
|
||||
arr = [float(v) for v in data.get("ttfts", []) if v is not None]
|
||||
return _percentile(arr, pct_num)
|
||||
if prefix == "itl":
|
||||
flat = []
|
||||
for sub in data.get("itls", []):
|
||||
if isinstance(sub, list):
|
||||
flat.extend(float(v) for v in sub if v is not None)
|
||||
elif sub is not None:
|
||||
flat.append(float(sub))
|
||||
return _percentile(flat, pct_num)
|
||||
if prefix == "tpot":
|
||||
per_req = []
|
||||
for sub in data.get("itls", []):
|
||||
if isinstance(sub, list) and sub:
|
||||
per_req.append(sum(float(v) for v in sub if v is not None) / len(sub))
|
||||
return _percentile(per_req, pct_num)
|
||||
if prefix == "e2e_latency":
|
||||
ttfts = [float(v) for v in data.get("ttfts", []) if v is not None]
|
||||
per_req = []
|
||||
itls = data.get("itls", [])
|
||||
for i, ttft in enumerate(ttfts):
|
||||
tail = itls[i] if i < len(itls) else []
|
||||
if isinstance(tail, list):
|
||||
per_req.append(ttft + sum(float(v) for v in tail if v is not None))
|
||||
else:
|
||||
per_req.append(ttft)
|
||||
return _percentile(per_req, pct_num)
|
||||
return 0.0
|
||||
|
||||
|
||||
def command_shapes(args: argparse.Namespace) -> int:
|
||||
@ -191,7 +250,12 @@ def command_parse_result(args: argparse.Namespace) -> int:
|
||||
"request_tps": float(data.get("request_throughput", 0.0) or 0.0),
|
||||
"input_tps": float(data.get("input_throughput", 0.0) or 0.0),
|
||||
"output_tps": float(data.get("output_throughput", 0.0) or 0.0),
|
||||
"total_tps": float(data.get("total_throughput", 0.0) or 0.0),
|
||||
"total_tps": float(
|
||||
data.get("total_throughput")
|
||||
or (float(data.get("input_throughput", 0.0) or 0.0)
|
||||
+ float(data.get("output_throughput", 0.0) or 0.0))
|
||||
or 0.0
|
||||
),
|
||||
"mean_input_tokens": mean_input,
|
||||
"mean_output_tokens": mean_output,
|
||||
"ttft_p50_ms": percentile_value(data, "ttft", "p50"),
|
||||
@ -223,7 +287,9 @@ def command_gain(args: argparse.Namespace) -> int:
|
||||
else:
|
||||
gain_pct = (args.current - args.previous) / args.previous * 100.0
|
||||
meaningful = gain_pct >= args.threshold_pct
|
||||
gain_text = "inf" if gain_pct == float("inf") else f"{gain_pct:.6f}"
|
||||
# Emit Infinity (JSON-valid) rather than inf so downstream json.loads()
|
||||
# in adaptive_append_point_from_metrics does not crash.
|
||||
gain_text = "Infinity" if gain_pct == float("inf") else f"{gain_pct:.6f}"
|
||||
print(f"{gain_text}\t{1 if meaningful else 0}")
|
||||
return 0
|
||||
|
||||
|
||||
@ -7,6 +7,8 @@ and updates results.json + report.md in the given result root.
|
||||
Usage:
|
||||
python3 parse_backend.py <result_root> [--backend sglang|vllm]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
@ -25,6 +27,72 @@ def parse_jsonl(path: Path) -> dict | None:
|
||||
return None
|
||||
|
||||
|
||||
def _percentile(values, pct):
|
||||
"""Compute a percentile (0-100) from a list of float values in *seconds*;
|
||||
return value in *ms*."""
|
||||
if not values:
|
||||
return 0.0
|
||||
s = sorted(values)
|
||||
if len(s) == 1:
|
||||
return s[0] * 1000.0
|
||||
if pct <= 0:
|
||||
return s[0] * 1000.0
|
||||
if pct >= 100:
|
||||
return s[-1] * 1000.0
|
||||
rank = (pct / 100.0) * (len(s) - 1)
|
||||
lo = int(rank)
|
||||
hi = min(lo + 1, len(s) - 1)
|
||||
frac = rank - lo
|
||||
return (s[lo] * (1.0 - frac) + s[hi] * frac) * 1000.0
|
||||
|
||||
|
||||
def _pct_fallback(data, prefix, percentile):
|
||||
"""Recompute a percentile from per-request arrays when the engine
|
||||
(sglang 0.5.x) omits the pre-computed p95/p50 field."""
|
||||
pct_num = {"p95": 95.0, "p99": 99.0, "p50": 50.0}.get(percentile, 0.0)
|
||||
if prefix == "ttft":
|
||||
arr = [float(v) for v in data.get("ttfts", []) if v is not None]
|
||||
return _percentile(arr, pct_num)
|
||||
if prefix == "itl":
|
||||
flat = []
|
||||
for sub in data.get("itls", []):
|
||||
if isinstance(sub, list):
|
||||
flat.extend(float(v) for v in sub if v is not None)
|
||||
elif sub is not None:
|
||||
flat.append(float(sub))
|
||||
return _percentile(flat, pct_num)
|
||||
if prefix == "tpot":
|
||||
per_req = []
|
||||
for sub in data.get("itls", []):
|
||||
if isinstance(sub, list) and sub:
|
||||
per_req.append(sum(float(v) for v in sub if v is not None) / len(sub))
|
||||
return _percentile(per_req, pct_num)
|
||||
if prefix == "e2e_latency":
|
||||
ttfts = [float(v) for v in data.get("ttfts", []) if v is not None]
|
||||
per_req = []
|
||||
itls = data.get("itls", [])
|
||||
for i, ttft in enumerate(ttfts):
|
||||
tail = itls[i] if i < len(itls) else []
|
||||
if isinstance(tail, list):
|
||||
per_req.append(ttft + sum(float(v) for v in tail if v is not None))
|
||||
else:
|
||||
per_req.append(ttft)
|
||||
return _percentile(per_req, pct_num)
|
||||
return 0.0
|
||||
|
||||
|
||||
def _pct(data, prefix, percentile):
|
||||
"""Read pre-computed percentile, fall back to per-request arrays."""
|
||||
if percentile == "p50":
|
||||
key = f"median_{prefix}_ms"
|
||||
else:
|
||||
key = f"{percentile}_{prefix}_ms"
|
||||
val = data.get(key)
|
||||
if val is not None:
|
||||
return float(val)
|
||||
return _pct_fallback(data, prefix, percentile)
|
||||
|
||||
|
||||
def compute_metrics(data: dict) -> dict:
|
||||
completed = data.get("completed", 0)
|
||||
total = len(data.get("input_lens", []))
|
||||
@ -38,36 +106,41 @@ def compute_metrics(data: dict) -> dict:
|
||||
"request_throughput": data.get("request_throughput", 0.0),
|
||||
"input_token_throughput": data.get("input_throughput", 0.0),
|
||||
"output_token_throughput": data.get("output_throughput", 0.0),
|
||||
"total_token_throughput": data.get("total_throughput", 0.0),
|
||||
"total_token_throughput": (
|
||||
data.get("total_throughput")
|
||||
if data.get("total_throughput")
|
||||
else (float(data.get("input_throughput", 0.0) or 0.0)
|
||||
+ float(data.get("output_throughput", 0.0) or 0.0))
|
||||
),
|
||||
"total_input_tokens": data.get("total_input_tokens", 0),
|
||||
"total_output_tokens": data.get("total_output_tokens", 0),
|
||||
"e2e_ms": {
|
||||
"mean": data.get("mean_e2e_latency_ms", 0.0),
|
||||
"p50": data.get("median_e2e_latency_ms", 0.0),
|
||||
"p90": data.get("p90_e2e_latency_ms", 0.0),
|
||||
"p95": data.get("p95_e2e_latency_ms", 0.0),
|
||||
"p99": data.get("p99_e2e_latency_ms", 0.0),
|
||||
"p95": _pct(data, "e2e_latency", "p95"),
|
||||
"p99": _pct(data, "e2e_latency", "p99"),
|
||||
},
|
||||
"ttft_ms": {
|
||||
"mean": data.get("mean_ttft_ms", 0.0),
|
||||
"p50": data.get("median_ttft_ms", 0.0),
|
||||
"p90": data.get("p90_ttft_ms", 0.0),
|
||||
"p95": data.get("p95_ttft_ms", 0.0),
|
||||
"p99": data.get("p99_ttft_ms", 0.0),
|
||||
"p95": _pct(data, "ttft", "p95"),
|
||||
"p99": _pct(data, "ttft", "p99"),
|
||||
},
|
||||
"tpot_ms": {
|
||||
"mean": data.get("mean_tpot_ms", 0.0),
|
||||
"p50": data.get("median_tpot_ms", 0.0),
|
||||
"p90": data.get("p90_tpot_ms", 0.0),
|
||||
"p95": data.get("p95_tpot_ms", 0.0),
|
||||
"p99": data.get("p99_tpot_ms", 0.0),
|
||||
"p95": _pct(data, "tpot", "p95"),
|
||||
"p99": _pct(data, "tpot", "p99"),
|
||||
},
|
||||
"itl_ms": {
|
||||
"mean": data.get("mean_itl_ms", 0.0),
|
||||
"p50": data.get("median_itl_ms", 0.0),
|
||||
"p90": data.get("p90_itl_ms", 0.0),
|
||||
"p95": data.get("p95_itl_ms", 0.0),
|
||||
"p99": data.get("p99_itl_ms", 0.0),
|
||||
"p95": _pct(data, "itl", "p95"),
|
||||
"p99": _pct(data, "itl", "p99"),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user