shishi
|
ea8302561e
|
fix(910c): bench client 可用化 - 本地 vllm-ascend-sglang 镜像 + torch_npu 自动加载禁用
- runner.py docker client 注入 TORCH_DEVICE_BACKEND_AUTOLOAD=0（bench 纯 HTTP
client 不需要 NPU backend，跳过 torch_npu 加载失败）
- dsv4/glm52 config.env: DOCKER_CLIENT_IMAGE 指向本地定制镜像
(local/vllm-ascend:0.23-a3-dsv4-sglang，内置 sglang 0.5.2 bench_serving)，
USE_DOCKER_CLIENT=1，sskj.bench/run_bench.sh 的 docker client 分支可用
- run_bench.sh docker client 分支同步注入 AUTOLOAD=0
- glm52 profile 修正 DOCKER_IMAGE 为本机存在的 glm5.2-a3-openeuler
|
2026-08-03 15:43:45 +08:00 |
|
shishi
|
e885fd0dc2
|
feat(adaptive): support tiered per-ISL TTFT SLO via get_ttft_slo_ms()
- adaptive_bench_lib.sh: add default get_ttft_slo_ms() fallback (flat TTFT_SLO_MS),
use it instead of hardcoded TTFT_SLO_MS in SLO comparison and logs,
add ttft_slo_tiers_desc to run_manifest.json
- glm52_910c config.env: define tiered SLO for GLM-5.2:
≤2048:5000ms, ≤8192:8000ms, ≤32768:12000ms, ≤131072:20000ms, >131072:30000ms
(~70-80% of DSv4-Pro values, since GLM-5.2 has simpler architecture)
- glm52_910c adaptive_config.env: update TTFT_SLO_MS comment noting tiered override
Backward compatible: experiments without get_ttft_slo_ms() keep flat 4000ms behavior.
|
2026-07-30 10:47:56 +08:00 |
|
shishi
|
455a78161b
|
fix(910c/glm52): 对齐官方A3教程参数(修DP die分配问题,同dsv4 99a22f0)
dsv4实验发现DP副本绑定到同一组die的问题(99a22f0),glm52存在相同问题:
缺--max-num-batched-tokens和--api-server-count导致DP worker设备分配异常
(不加api-server-count时vllm为N个DP rank启动N个API server)。
对齐docs.vllm.ai GLM5.2 A3官方教程:
- 加 --max-num-batched-tokens 8192 (官方值,影响DP调度)
- 加 --api-server-count 1 (官方值,避免多API server干扰设备分配)
- 去掉 --kv-cache-dtype fp8 (官方不指定,用默认bfloat16;且此镜像fp8本就未生效)
- 保留 --trust-remote-code / --enable-expert-parallel / enable_dsa_cp (官方有)
|
2026-07-29 15:28:54 +08:00 |
|
shishi
|
c222ed98b2
|
feat(910c/glm52): 并行配置改为 4 4 / 8 2 / 16 1 对标H20的 2 4 / 4 2 / 8 1
A3 910C 有16 dies(8卡x2die),H20有8卡。为公平对比,TP按卡数等效:
A3 TP=4 DP=4 (4die/副本x4) == H20 TP=2 DP=4 (2卡/副本x4)
A3 TP=8 DP=2 (8die/副本x2) == H20 TP=4 DP=2 (4卡/副本x2)
A3 TP=16 DP=1 (16die/副本x1) == H20 TP=8 DP=1 (8卡/副本x1)
与dsv4实验(a65849b)保持一致的配置思路。
新增TP=4 per-TP参数覆盖(expert-parallel下专家分4份+dense复制,
KV cache极紧): gpu_mem=0.97, max_model_len=4096, max_num_seqs=32
TP=4可能OOM,若发生会自动记录并跳过。
config.env: PARALLEL_CONFIGS 8 1/16 1 -> 4 4/8 2/16 1; 加TP4_变量
start_vllm_docker.sh: case $TP 加 TP=4 分支
run_adaptive_concurrency_add16.sh: case $tp 加 TP=4 分支
|
2026-07-29 13:44:40 +08:00 |
|
shishi
|
6c81183fd7
|
feat(910c/glm52): 并行配置改为只测 TP=8 和 TP=16,并按TP区分服务参数
config.env: PARALLEL_CONFIGS 默认值从 "2 4"/"4 2"/"8 1" 改为 "8 1"/"16 1"
run_adaptive_concurrency_add16.sh: engine_build_server_args 按 TP 覆盖参数
- TP=8: gpu_mem_util=0.95, max_model_len=16384, max_num_seqs=64 (64GB/die KV cache 紧张)
- TP=16: gpu_mem_util=0.92, max_model_len=131072, max_num_seqs=256 (16 die 全用)
可通过 TP8_*/TP16_* 环境变量进一步覆盖
|
2026-07-28 16:51:14 +08:00 |
|
shishi
|
98cdb67b66
|
fix(910c/glm52): 修复sglang0.5.2解析兼容性+TP=16设备挂载+health超时
sglang 0.5.2 bench_serving 输出与解析器不兼容,致 add16 c=32 崩溃:
- 缺 total_throughput -> total_tps 恒0,无法检测吞吐增益
- 缺 p95_*(仅p99) -> TTFT SLO 失效
- gain previous<=0 返回 inf -> json.loads(inf) 崩溃
修复:
1. adaptive_concurrency.py: 缺失时从 ttfts/itls 数组补算 p95/p50;
total_tps 回退 input+output throughput; gain 返回 Infinity
2. parse_backend.py: 同上补算逻辑; 补 from __future__ import annotations
(py3.9 下 dict|None 语法无法 import)
3. start_vllm_docker.sh: --device davinci0~15 支持 TP=16;
health 超时可配(默认480x5s=40min,TP=16编译16 graph约60min);
补驱动挂载+/mnt; 修容器名双后缀
4. run_adaptive_concurrency_add16.sh: --tokenizer 替代 --model;
TORCH_DEVICE_BACKEND_AUTOLOAD=0; CONTAINER_PYTHON 路径; 导出 ENGINE_TP/DP
5. config.env: 固定 CONTAINER_NAME/DOCKER_IMAGE/GPU_MEM_UTIL
6. TP8_vs_TP16_report.md: TP=8 vs TP=16 手动测速对比报告
验证: TP=8 add16 c=16->c=32 不再崩溃; TP=16 编译完成变 healthy 推理正常
|
2026-07-28 16:43:05 +08:00 |
|
shishi
|
46e79d63e7
|
feat(platform): add Ascend 910C NPU platform support
- platforms/ascend_910c.env: 8-card 910C config (16 dies, 64GB HBM/die),
Ascend Docker Runtime, ASCEND_VISIBLE_DEVICES device selection
- scripts/common/platform.sh: auto-detect 910C via npu-smi + Huawei PCI IDs
- scripts/common/npu_smi_sampler.py: standalone npu-smi -> nvidia-smi CSV
sampler so parse_backend.py needs no changes
- experiments/910c/glm52_910c_vllm_tp_dp_matrix/: GLM-5.2 (w4a8c8) experiment,
model present on host, ready for smoke after image load
- experiments/910c/dsv4_910c_vllm_tp_dp_matrix/: DSV4-Flash experiment
(placeholder MODEL_PATH, weights not yet downloaded)
- envs/ASCEND_910C_ENV_SETUP.md: full onboarding guide (permissions, image
load, Ascend Docker Runtime, NPU monitor, known pitfalls)
- Both experiments: TP2/DP4 + TP4/DP2 + TP8/DP1, matrix.json capped at 128K
context per 64GB HBM/die
|
2026-07-27 22:00:05 +08:00 |
|