diff --git a/.gitignore b/.gitignore index 4602a52..154ee63 100644 --- a/.gitignore +++ b/.gitignore @@ -83,6 +83,7 @@ experiments/**/raw_outputs/ envs/* !envs/README.md !envs/SM120_DSV4_DEPLOYMENT_GUIDE.md +!envs/ASCEND_910C_ENV_SETUP.md !envs/UV_ENV_SETUP.md # Tooling artifacts diff --git a/envs/ASCEND_910C_ENV_SETUP.md b/envs/ASCEND_910C_ENV_SETUP.md new file mode 100644 index 0000000..fc556bb --- /dev/null +++ b/envs/ASCEND_910C_ENV_SETUP.md @@ -0,0 +1,158 @@ +# Ascend 910C 环境搭建与部署指南 + +本文档说明如何在 Ascend 910C NPU 节点上搭建 vLLM-Ascend 推理环境并跑起 sskj 基准测试。 + +## 1. 环境信息（参考机型：910c.1 / NPU-NODE61） + +| 项目 | 值 | +|---|---| +| OS | openEuler 22.03 LTS SP4 (aarch64) | +| 内核 | 5.10.0-216.0.0.115.oe2203sp4.aarch64 | +| NPU | 8 × Ascend910（每卡 2 die，共 16 die） | +| HBM | 64 GB/die，合计 ~1 TB | +| 驱动 | 25.5.2（Innerversion V100R001C23SPC007B221） | +| CANN | 9.0.0 + ascend-toolkit | +| Docker | 26.1.3，默认 runtime = ascend（见 `/etc/docker/daemon.json`） | +| Python (host) | 3.9.9（仅用于编排脚本，推理在容器内） | + +## 2. 权限准备 🧑 + +新用户默认无法访问 NPU 设备节点和 Docker，需要管理员加入两个组： + +```bash +# 加入 HwHiAiUser 组才能访问 /dev/davinci* 设备节点 +sudo usermod -aG HwHiAiUser +# 加入 docker 组才能调用 docker（或每次 sudo docker） +sudo usermod -aG docker +# 重新登录生效 +exit # 然后重新 ssh +``` + +验证： + +```bash +id # 应看到 HwHiAiUser 和 docker 组 +npu-smi info # 应输出 8 卡 16 die 的状态表 +docker ps # 不应报 permission denied +``` + +## 3. 加载 vLLM-Ascend 镜像 + +910C 节点离线，vLLM-Ascend 镜像以 tarball 形式存放在 `/mnt/models/`： + +| tarball | 用途 | +|---|---| +| `vllm-ascend-v0.23.0rc1-a3-openeuler.tar` | 通用 v0.23，适合 DSV4-Flash | +| `vllm-ascend-glm5.2-a3-openeuler.tar` | GLM5.2 调优版（推荐跑 GLM5.2） | +| `vllm-ascend-v0.22.1rc1-a3.tar` | 旧版 v0.22 | +| `local-vllm-ascend-0.23-a3.tar` | 本地构建的 0.23 | + +加载（任选需要的）： + +```bash +docker load -i /mnt/models/vllm-ascend-glm5.2-a3-openeuler.tar +docker load -i /mnt/models/vllm-ascend-v0.23.0rc1-a3-openeuler.tar +docker images | grep vllm-ascend # 记下确切的 REPOSITORY:TAG +``` + +加载后把镜像 tag 写入对应实验的 `config.env`： + +```bash +# experiments/910c/glm52_910c_vllm_tp_dp_matrix/config.env +DOCKER_IMAGE="<加载后看到的 repository:tag>" +``` + +## 4. 模型权重 + +当前 `/mnt/models/` 下已有： + +- `GLM-5.2-w4a8c8/`（95 shards，默认用这个） +- `GLM-5.2-w8a8/`（181 shards，需切换时改 `MODEL_PATH`） + +**DeepSeek-V4-Flash 尚未下载**。需要时下载到 `/mnt/models/DeepSeek-V4-Flash`（FP8）或 `/mnt/models/DeepSeek-V4-Flash-INT8`，再改 `experiments/910c/dsv4_910c_vllm_tp_dp_matrix/config.env` 的 `MODEL_PATH`。 + +## 5. 数据集 + +`sglang.bench_serving --dataset-name random` 需要 ShareGPT 种子文件： + +```bash +mkdir -p /mnt/yy/sskj/datasets +# 放入 ShareGPT_V3_unfiltered_cleaned_split.json +# （从 https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered 下载） +``` + +`DATASET_PATH` 默认指向 `${ROOT_DIR}/datasets/ShareGPT_V3_unfiltered_cleaned_split.json`，无需改 config。 + +## 6. Ascend Docker Runtime 说明 + +本机的 `/etc/docker/daemon.json` 已配置： + +```json +{ + "default-runtime": "ascend", + "runtimes": { + "ascend": { + "path": "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime", + "runtimeArgs": [] + } + } +} +``` + +因此 `docker run` **无需** `--runtime ascend` 或 `--gpus`，只需通过环境变量 `ASCEND_VISIBLE_DEVICES=0,1,2,3,4,5,6,7` 指定要映射的 NPU 卡号，runtime 会自动把对应 die 的 `/dev/davinci*` 注入容器。 + +## 7. 冒烟测试 🤖 + +```bash +cd /mnt/yy/sskj/experiments/910c/glm52_910c_vllm_tp_dp_matrix + +# 1. dry-run：只打印搜索计划，不起服务 +DRY_RUN=1 bash run_adaptive_concurrency_add16.sh + +# 2. 单 shape 小并发冒烟（TP=8，1K/128，并发上限 8） +RUN_ID=smoke-$(date +%Y%m%d-%H%M%S) \ +TP_LIST="8" ISL_LIST="1024" OSL_LIST="128" GRID_LIMIT=1 SEARCH_MAX_CONCURRENCY=8 \ +bash run_adaptive_concurrency_add16.sh +``` + +冒烟常见失败原因： + +| 现象 | 根因 | 解决 | +|---|---|---| +| `dcmi module initialize failed` | 用户不在 HwHiAiUser 组 | 见 §2 | +| `docker: permission denied` | 用户不在 docker 组 | 见 §2 | +| 容器内 `ModuleNotFoundError: torch_npu` | 镜像未正确加载 / tag 写错 | `docker images` 核对 | +| `sglang.bench_serving` ModuleNotFoundError | vllm-ascend 镜像不含 sglang | 见 §8 | +| 启动即 OOM | TP 过小，专家权重放不下 | 见 config.env 显存预算注释 | + +## 8. Benchmark Client 说明 ⚠️ + +sskj 的压测客户端是 `sglang.bench_serving`，但 **vllm-ascend 官方镜像不含 sglang**。910C 上有两个选择： + +1. **(推荐) 容器内装 sglang**：进容器 `pip install sglang`（或 `sglang[all]`），之后 `docker exec` 跑客户端。需要把 sglang 装进镜像或每次启动后手动装。 +2. **外部 sglang 镜像**：设 `USE_DOCKER_CLIENT=1`，提供 `DOCKER_CLIENT_IMAGE=lmsysorg/sglang:xxx`，用独立容器通过 host 网络打 vLLM 的 OpenAI API。但 sglang 官方镜像多为 x86 + CUDA，aarch64 NPU 节点上可能拉不到对应架构镜像。 + +建议冒烟前先确认客户端方案，否则 adaptive 搜索会在 `engine_run_bench` 阶段失败。验证命令： + +```bash +# 进容器看是否有 sglang +docker run --rm python -c "import sglang; print(sglang.__version__)" +``` + +## 9. NPU 监控 + +公共库 `adaptive_bench_lib.sh` 的 GPU 监控写死 `nvidia-smi`，910C 实验脚本已用 `npu-smi info` 重写 `adaptive_start_gpu_monitor` / `start_gpu_monitor`，输出与 nvidia-smi 相同的 CSV 列（timestamp, index, memory.used, memory.total, utilization.gpu），下游 `parse_backend.py` 无需改动。 + +手动查看 NPU 状态： + +```bash +npu-smi info # 总览 +npu-smi info -t usages -i 0 # 单卡详细利用率 +``` + +## 10. 已知坑 + +1. **TP 与 die 的关系**：910C 每卡 2 die，vllm-ascend 按 die 分配 TP。`ASCEND_VISIBLE_DEVICES=0..7` 暴露 8 卡 = 16 die，因此 TP 最大 16（本实验限 TP≤8）。 +2. **KV cache dtype**：910C 支持 fp8 KV cache，但部分 vllm-ascend 版本在 NPU 上对 fp8 KV 支持不完整。若启动报 `kv-cache-dtype fp8 not supported`，改 `KV_CACHE_DTYPE=fp16`。 +3. **block-size**：NPU 推荐 128（NVIDIA H20 用 256）。若性能异常可尝试 64/128/256 对比。 +4. **DSV4-Flash FP8 显存**：路由专家 ~264 GiB，TP=2/4 在 64GB die 上几乎必 OOM（见 dsv4 config.env 注释）。用 INT8 权重或限 TP≥8。 diff --git a/envs/README.md b/envs/README.md index cd2c426..26846a0 100644 --- a/envs/README.md +++ b/envs/README.md @@ -1,3 +1,4 @@ + # envs 目录说明 本目录用于存放**环境搭建相关的文档和指南**，不包括具体的虚拟环境或容器镜像文件。 @@ -30,6 +31,7 @@ envs/ |---|---| | `UV_ENV_SETUP.md` | uv 虚拟环境搭建规范，含 vLLM 和 SGLang 的标准安装命令 | | `SM120_DSV4_DEPLOYMENT_GUIDE.md` | RTX 6000D (SM120) 上部署 DeepSeek-V4-Flash 的完整指南，含 vLLM 和 SGLang 的适配步骤 | +| `ASCEND_910C_ENV_SETUP.md` | 昇腾 910C NPU 上搭建 vLLM-Ascend 环境与部署指南，含权限、镜像加载、NPU 监控适配 | ## 排除的环境目录（已加入 .gitignore） diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_config.env b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_config.env new file mode 100644 index 0000000..3b05c7c --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_config.env @@ -0,0 +1,58 @@ +# Adaptive concurrency search settings. +# +# For each fixed (TP, DP, ISL, OSL), probe: +# C = start, start * multiplier, ... up to max +# and stop after Total TPS has less than TPS_MIN_GAIN_PCT meaningful growth for +# PLATEAU_PATIENCE consecutive points. + +SEARCH_START_CONCURRENCY="${SEARCH_START_CONCURRENCY:-1}" +SEARCH_MAX_CONCURRENCY="${SEARCH_MAX_CONCURRENCY:-256}" + +# At the add16 initial probe, restart and retry C=8 then C=1 after an OOM. +ENABLE_INITIAL_OOM_BACKOFF="${ENABLE_INITIAL_OOM_BACKOFF:-1}" +SEARCH_MULTIPLIER="${SEARCH_MULTIPLIER:-2}" +NUM_PROMPTS_MULTIPLIER="${NUM_PROMPTS_MULTIPLIER:-5}" + +# A gain below 2% is treated as throughput saturation. Two consecutive +# low-gain points prevent one noisy measurement from stopping the search. +TPS_MIN_GAIN_PCT="${TPS_MIN_GAIN_PCT:-2.0}" +PLATEAU_PATIENCE="${PLATEAU_PATIENCE:-2}" + +# TTFT SLO early-stop settings. +# When ttft_p95_ms exceeds TTFT_SLO_MS, stop searching the current (ISL, OSL) +# shape and move on to the next scenario. +TTFT_SLO_MS="${TTFT_SLO_MS:-4000}" +ENABLE_TTFT_SLO_STOP="${ENABLE_TTFT_SLO_STOP:-1}" + +# Keep the same random workload semantics as the fixed matrix baseline. +# DATASET_PATH must contain at least SEARCH_MAX_CONCURRENCY times +# NUM_PROMPTS_MULTIPLIER valid two-turn conversations. Set this explicitly to +# random-ids to use generated token IDs without a ShareGPT seed dataset. +BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}" +# SGLang interprets 0.0 as Uniform[1, requested_len]. Use 1.0 for fixed +# ISL/OSL points; lower values intentionally benchmark a length distribution. +RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1.0}" +# Before each measured point, warm up with the same concurrency so lazy kernel +# compilation and CUDA graph capture are excluded from TTFT/TPS. 0 means no +# cap; set a positive cap only when very high-concurrency warmup is impractical. +BENCH_WARMUP_MAX_REQUESTS="${BENCH_WARMUP_MAX_REQUESTS:-0}" + +# Reject a point if the completed request count or actual token lengths do not +# match the requested workload. +INPUT_LENGTH_TOLERANCE_PCT="${INPUT_LENGTH_TOLERANCE_PCT:-5.0}" +OUTPUT_LENGTH_TOLERANCE_PCT="${OUTPUT_LENGTH_TOLERANCE_PCT:-10.0}" + +MAX_POINT_RETRIES="${MAX_POINT_RETRIES:-1}" +SERVER_RESTART_COOLDOWN_S="${SERVER_RESTART_COOLDOWN_S:-10}" +SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}" +GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}" + +# Optional space-separated filters, useful for smoke tests: +# TP_LIST="8" ISL_LIST="1024" OSL_LIST="128" +TP_LIST="${TP_LIST:-}" +ISL_LIST="${ISL_LIST:-}" +OSL_LIST="${OSL_LIST:-}" + +DRY_RUN="${DRY_RUN:-0}" +# Counts ISL/OSL shapes per TP/DP config, not individual concurrency probes. +GRID_LIMIT="${GRID_LIMIT:-0}" diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/compare.py b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/compare.py new file mode 100644 index 0000000..002fa0c --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/compare.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Cross TP×DP configuration comparison for dsv4_h20_vllm_tp_dp_matrix. + +Usage: + python3 compare.py --run-root results/ [--output comparison.md] +""" +import argparse +import json +import re +from collections import defaultdict +from pathlib import Path + + +def load_result(result_root: Path) -> dict: + path = result_root / "results.json" + with open(path, "r", encoding="utf-8") as f: + return json.load(f) + + +def slo_status(ttft_p95_ms: float, tpot_mean_ms: float, + ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str: + ttft_ok = ttft_p95_ms < ttft_limit_ms + tpot_ok = tpot_mean_ms < tpot_limit_ms + if ttft_ok and tpot_ok: + return "PASS" + if ttft_ok or tpot_ok: + return "PARTIAL" + return "FAIL" + + +def gpu_memory_str(gpu: dict | None) -> str: + if not gpu: + return "-" + peak = gpu.get("peak_used_mb", 0) + total = gpu.get("memory_total_mb", 0) + if total: + return f"{peak:.0f}/{total:.0f} ({100*peak/total:.1f}%)" + return f"{peak:.0f}" + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--run-root", type=Path, required=True) + parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md")) + parser.add_argument("--ttft-limit", type=float, default=3000.0) + parser.add_argument("--tpot-limit", type=float, default=50.0) + args = parser.parse_args() + + # Discover configurations: tp*_dp* directories. + configs = [] + for subdir in sorted(args.run_root.iterdir()): + if not subdir.is_dir(): + continue + name = subdir.name + if not (name.startswith("tp") and "_dp" in name): + continue + results_json = subdir / "results.json" + if not results_json.exists(): + continue + configs.append((name, load_result(subdir))) + + if not configs: + print(f"No tp*_dp* results found under {args.run_root}") + return + + model = configs[0][1].get("metadata", {}).get("model", "unknown") + hardware = configs[0][1].get("metadata", {}).get("hardware", "unknown") + + # Group by scenario name. + by_scenario: dict[str, dict[str, dict]] = defaultdict(dict) + skipped: dict[str, dict[str, str]] = defaultdict(dict) + for label, data in configs: + for s in data.get("scenarios", []): + key = s["name"] + if s.get("status") == "skipped_oom": + skipped[key][label] = s.get("note", "skipped") + else: + by_scenario[key][label] = s + + with open(args.output, "w", encoding="utf-8") as f: + f.write(f"# vLLM TP×DP matrix comparison ({hardware})\n\n") + f.write("## Summary\n\n") + f.write(f"- Model: `{model}`\n") + f.write(f"- Hardware: {hardware}\n") + f.write("- Backend: vLLM (Docker)\n") + f.write("- Benchmark client: `sglang.bench_serving`\n") + f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n") + + # Configuration overview. + f.write("### Configurations\n\n") + f.write("| Config | TP | DP | GPUs/replica | Notes |\n") + f.write("|---|---:|---:|---:|---|\n") + for label, data in configs: + cfg = data.get("config", {}) + tp = cfg.get("tp", "?") + dp = cfg.get("dp", "?") + f.write(f"| {label} | {tp} | {dp} | {tp} | server args recorded per ISL in results.json |\n") + f.write("\n") + + # Side-by-side table. + f.write("## Side-by-side results\n\n") + headers = [ + "Scenario", "ISL", "OSL", "Config", "Conc", "Req/s", "OutTok/s", + "TTFT P95(ms)", "TTFT P99(ms)", "TPOT Mean(ms)", "TPOT P95(ms)", + "TPOT P99(ms)", "E2E P99(ms)", "Peak GPU mem", "SLO" + ] + f.write("| " + " | ".join(headers) + " |\n") + f.write("|" + "|".join(["---"] * len(headers)) + "|\n") + + for scenario_name in sorted(by_scenario.keys(), key=lambda x: tuple(map(int, re.findall(r"\d+", x)))): + _, isl, osl = re.findall(r"\d+", scenario_name) + # cfg_part not used; just for readability. + for label, data in configs: + s = by_scenario[scenario_name].get(label) + if s is None: + if scenario_name in skipped and label in skipped[scenario_name]: + note = skipped[scenario_name][label] + f.write(f"| {scenario_name} | {isl} | {osl} | {label} | - | - | - | - | - | - | - | - | - | - | {note} |\n") + continue + cfg = s["config"] + m = s["metrics"] + status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit) + gpu = m.get("gpu_memory") + f.write( + f"| {scenario_name} | {isl} | {osl} | {label} | {cfg['concurrency']} | " + f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | " + f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | " + f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | " + f"{m['e2e_ms']['p99']:.2f} | {gpu_memory_str(gpu)} | {status} |\n" + ) + + # Best throughput per ISL/OSL. + f.write("\n## Best throughput per (ISL, OSL)\n\n") + f.write("| ISL | OSL | Best Config | Concurrency | OutTok/s | TTFT P95(ms) | TPOT Mean(ms) | SLO |\n") + f.write("|---:|---:|---|---:|---:|---:|---:|---:|\n") + best_by_shape: dict[tuple[int, int], tuple[float, str, dict]] = {} + for scenario_name, backends in by_scenario.items(): + _, isl, osl = re.findall(r"\d+", scenario_name) + isl_i, osl_i = int(isl), int(osl) + for label, s in backends.items(): + m = s["metrics"] + out_tok = m["output_token_throughput"] + if (isl_i, osl_i) not in best_by_shape or out_tok > best_by_shape[(isl_i, osl_i)][0]: + best_by_shape[(isl_i, osl_i)] = (out_tok, label, s) + for (isl_i, osl_i), (out_tok, label, s) in sorted(best_by_shape.items()): + m = s["metrics"] + status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit) + f.write( + f"| {isl_i} | {osl_i} | {label} | {s['config']['concurrency']} | " + f"{out_tok:.2f} | {m['ttft_ms']['p95']:.2f} | {m['tpot_ms']['mean']:.2f} | {status} |\n" + ) + + f.write("\n## Notes\n\n") + f.write("- SLO check uses TTFT P95 and TPOT mean.\n") + f.write("- A PARTIAL indicates one of the two metrics is out of target; FAIL indicates both are out.\n") + f.write("- `Peak GPU mem` shows peak used / total MB and utilization percentage.\n") + f.write("- Optional (P) combinations that failed are marked as skipped/OOM and do not break the run.\n") + + print(f"Wrote comparison to {args.output}") + + +if __name__ == "__main__": + main() diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/config.env b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/config.env new file mode 100644 index 0000000..2b586c4 --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/config.env @@ -0,0 +1,88 @@ +# TP×DP matrix experiment for DeepSeek-V4-Flash on Ascend 910C (8 NPUs / 16 dies) +# using vLLM-Ascend. +# Tests vLLM with three parallel configurations: +# TP=2, DP=4 -> 2 dies per replica, 4 replicas +# TP=4, DP=2 -> 4 dies per replica, 2 replicas +# TP=8, DP=1 -> 8 dies, no data parallelism +# +# Platform: ascend_910c (see platforms/ascend_910c.env). +# Host: 910c.1 / NPU-NODE61, openEuler 22.03 SP4 aarch64, driver 25.5.2, CANN 9.0.0. +# +# ⚠️ PLACEHOLDER: DeepSeek-V4-Flash weights are NOT yet present on this host +# (only GLM-5.2-{w4a8c8,w8a8} under /mnt/models). Download DSV4-Flash first, +# then fix MODEL_PATH below. Smoke run will fail until the model exists. +# Recommended host path: /mnt/models/DeepSeek-V4-Flash (FP8) +# /mnt/models/DeepSeek-V4-Flash-INT8 (INT8) + +EXPERIMENT="dsv4_910c_vllm_tp_dp_matrix" +MODEL_NAME="DeepSeek-V4-Flash" +# TODO: point this at the real DSV4-Flash directory once downloaded. +MODEL_PATH="${MODEL_PATH:-/mnt/models/DeepSeek-V4-Flash}" +SERVED_MODEL_NAME="deepseek-v4-flash" + +VLLM_PORT="${VLLM_PORT:-30052}" + +# Dedicated container name so this experiment never touches other 910c runs. +CONTAINER_NAME="${CONTAINER_NAME:-vllm-ascend-dsv4-910c}" + +# Python interpreter for the benchmark client inside the vllm-ascend container. +CONTAINER_PYTHON="${CONTAINER_PYTHON:-/usr/local/bin/python}" + +# vllm-ascend image. Use the general v0.23 image for DSV4 (the GLM5.2-tuned +# variant may carry GLM-specific patches). Load from: +# /mnt/models/vllm-ascend-v0.23.0rc1-a3-openeuler.tar +USE_DOCKER="${USE_DOCKER:-1}" +DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:v0.23.0rc1-a3-openeuler}" + +DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}" +USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-0}" + +export ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}" +export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}" + +RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}" + +# Parallel configurations to test. Format: "TP DP" +# DSV4-Flash FP8 routed-expert weights ~264 GiB total. Per-die load = 264/TP GiB. +# TP=2 -> 132 GiB/die > 64 GiB HBM ❌ OOM expected (kept but will fail) +# TP=4 -> 66 GiB/die > 64 GiB HBM ❌ borderline OOM (KV cache leaves no room) +# TP=8 -> 33 GiB/die < 64 GiB HBM ✅ fits with room for KV cache +# TP=2/DP=4 and TP=4/DP=2 are likely infeasible for FP8 on 64GB dies; they are +# kept here so the smoke run records the OOM boundary explicitly. Switch to +# INT8 weights (MODEL_PATH=...-INT8, ~132 GiB total) to make TP=4 viable. +if [[ -n "${PARALLEL_CONFIGS_STR:-}" ]]; then + declare -a PARALLEL_CONFIGS=() + for pair in $PARALLEL_CONFIGS_STR; do + PARALLEL_CONFIGS+=("${pair//,/ }") + done +else + declare -a PARALLEL_CONFIGS=( + "2 4" + "4 2" + "8 1" + ) +fi + +# vLLM-Ascend server settings for DSV4-Flash. +GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.9}" +KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" +BLOCK_SIZE="${BLOCK_SIZE:-128}" +# DSV4-Flash supports up to 1M context, but 64GB HBM/die caps realistic ISL. +# Start at 128K; extend via matrix.json once TP=8 is verified. +MAX_MODEL_LEN="${MAX_MODEL_LEN:-131072}" +MAX_NUM_SEQS="${MAX_NUM_SEQS:-256}" + +VLLM_ASCEND_ATTENTION_BACKEND="${VLLM_ASCEND_ATTENTION_BACKEND:-atb}" + +DATASET_PATH="${DATASET_PATH:-${ROOT_DIR}/datasets/ShareGPT_V3_unfiltered_cleaned_split.json}" + +MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR:-.}/matrix.json}" +MATRIX_MODE="${MATRIX_MODE:-Y}" + +export CONCURRENCY_SAMPLES="${CONCURRENCY_SAMPLES:-2}" + +SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}" +GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}" + +DRY_RUN="${DRY_RUN:-0}" +GRID_LIMIT="${GRID_LIMIT:-0}" diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/generate_scenarios.py b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/generate_scenarios.py new file mode 100644 index 0000000..2bfb152 --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/generate_scenarios.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Generate the scenario list for the TP×DP matrix experiment. + +Reads matrix.json and prints TSV lines: + + mark input_len output_len concurrency num_prompts + +mark is one of Y/P/N. The caller (run_bench.sh) decides how to treat each. +""" +import argparse +import json +import math +import os +from pathlib import Path + + +def sample_concurrency(low: int, high: int, target: int) -> list[int]: + """Return only the low and high concurrency values in [low, high]. + + For the TP×DP matrix we only need the two endpoints of the concurrency + range (e.g. 1 and 128 for ISL=1024). The `target` argument is kept for + API compatibility but is ignored. + """ + assert 1 <= low <= high, f"invalid concurrency range: {low}-{high}" + if low == high: + return [low] + return [low, high] + + +def generate_scenarios(matrix_path: Path, mode: str, target_samples: int) -> list[dict]: + with open(matrix_path, "r", encoding="utf-8") as f: + data = json.load(f) + + matrix = data["matrix"] + concurrency_cfg = data["concurrency"] + + scenarios = [] + for isl_str in sorted(matrix.keys(), key=int): + osl_map = matrix[isl_str] + low = concurrency_cfg[isl_str]["low"] + high = concurrency_cfg[isl_str]["high"] + concurrencies = sample_concurrency(low, high, target_samples) + + for osl_str in sorted(osl_map.keys(), key=int): + mark = osl_map[osl_str] + if mode == "Y" and mark != "Y": + continue + if mode == "Y+P" and mark not in ("Y", "P"): + continue + # mode == "all" keeps everything, including N. + + for conc in concurrencies: + scenarios.append( + { + "mark": mark, + "input_len": int(isl_str), + "output_len": int(osl_str), + "concurrency": conc, + "num_prompts": conc * 5, + } + ) + + return scenarios + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--matrix", type=Path, default=Path("matrix.json")) + parser.add_argument("--mode", choices=["Y", "Y+P", "all"], default=None, + help="Scenario selection mode. Defaults to matrix.mode.") + parser.add_argument("--target-samples", type=int, default=0, + help="Target number of concurrency samples. 0 = heuristic (6-8).") + args = parser.parse_args() + + with open(args.matrix, "r", encoding="utf-8") as f: + data = json.load(f) + + mode = args.mode if args.mode else data.get("mode", "Y+P") + + target_samples = args.target_samples + if target_samples <= 0: + env_samples = os.getenv("CONCURRENCY_SAMPLES", "0") + try: + target_samples = int(env_samples) + except ValueError: + target_samples = 0 + if target_samples <= 0: + target_samples = 7 + + scenarios = generate_scenarios(args.matrix, mode, target_samples) + + print("mark\tinput_len\toutput_len\tconcurrency\tnum_prompts") + for s in scenarios: + print(f"{s['mark']}\t{s['input_len']}\t{s['output_len']}\t{s['concurrency']}\t{s['num_prompts']}") + + +if __name__ == "__main__": + main() diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json new file mode 100644 index 0000000..682be08 --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json @@ -0,0 +1,92 @@ +{ + "comment": "910C / DeepSeek-V4-Flash: 128K context cap (extend to 1M once TP=8 verified). ISL beyond 131072 excluded. DSV4-Flash FP8 routed-expert ~264GiB: TP=2/4 likely OOM at high ISL (see config.env). 'P' = probe-only, 'N' = skip (OOM expected).", + "mode": "Y", + "matrix": { + "1024": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "Y" + }, + "4096": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "Y" + }, + "8192": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "Y" + }, + "16384": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "P" + }, + "32768": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "P", + "4096": "N" + }, + "65536": { + "128": "Y", + "256": "Y", + "512": "P", + "1024": "P", + "2048": "N", + "4096": "N" + }, + "131072": { + "128": "Y", + "256": "P", + "512": "N", + "1024": "N", + "2048": "N", + "4096": "N" + } + }, + "concurrency": { + "1024": { + "low": 1, + "high": 128 + }, + "4096": { + "low": 1, + "high": 64 + }, + "8192": { + "low": 1, + "high": 64 + }, + "16384": { + "low": 1, + "high": 32 + }, + "32768": { + "low": 1, + "high": 16 + }, + "65536": { + "low": 1, + "high": 8 + }, + "131072": { + "low": 1, + "high": 4 + } + } +} diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_adaptive_concurrency.sh b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_adaptive_concurrency.sh new file mode 100755 index 0000000..f76a93d --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_adaptive_concurrency.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +# Find the Total-TPS saturation concurrency for each vLLM TP/DP/ISL/OSL shape. +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")" + +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/lib.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/platform.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/adaptive_config.env" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh" + +ENGINE="vllm" +ENGINE_PORT="$VLLM_PORT" +RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}" +ACTIVE_ENGINE_SERVER_LOG="" + +if [[ -x "${VENV_CLIENT}/bin/python" ]]; then + PYTHON="${VENV_CLIENT}/bin/python" +else + PYTHON="$(command -v python3)" +fi + +DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}" +DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}" + +engine_is_healthy() { + curl --fail --silent --show-error --max-time 5 \ + "http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1 +} + +engine_stop_server() { + local tp="$1" + local dp="$2" + local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid" + + if [[ -f "$pid_file" ]]; then + local pid + pid="$(cat "$pid_file")" + if kill -0 "$pid" 2>/dev/null; then + log "stopping vllm server pid=${pid} tp=${tp} dp=${dp}" + kill "$pid" 2>/dev/null || true + sleep 5 + kill -9 "$pid" 2>/dev/null || true + fi + rm -f "$pid_file" + fi + docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true + ACTIVE_ENGINE_SERVER_LOG="" + sleep 2 +} + +engine_build_server_args() { + local tp="$1" + local dp="$2" + local -a args=( + vllm serve "$MODEL_PATH" + --trust-remote-code + --kv-cache-dtype "$KV_CACHE_DTYPE" + --block-size "$BLOCK_SIZE" + --tensor-parallel-size "$tp" + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" + --max-model-len "$MAX_MODEL_LEN" + --max-num-seqs "$MAX_NUM_SEQS" + --no-enable-flashinfer-autotune + --host 0.0.0.0 + --port "$ENGINE_PORT" + ) + if (( dp > 1 )); then + args+=(--data-parallel-size "$dp") + fi + printf '%q ' "${args[@]}" +} + +engine_start_server() { + local tp="$1" + local dp="$2" + local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" + log "starting vllm server tp=${tp} dp=${dp}" + bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1 + if ! engine_is_healthy; then + log "ERROR: vllm health check failed tp=${tp} dp=${dp}" + return 1 + fi + ACTIVE_ENGINE_SERVER_LOG="$( + find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \ + -name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \ + -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2- + )" + log "vllm server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}" +} + +engine_detect_oom() { + local detail_log="$1" + local tp="$2" + local dp="$3" + local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory' + local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" + local -a logs=("$detail_log" "$outer_log") + if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then + logs+=("$ACTIVE_ENGINE_SERVER_LOG") + fi + grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null +} + +engine_run_bench() { + local isl="$1" + local osl="$2" + local concurrency="$3" + local num_prompts="$4" + local output_file="$5" + local warmup_requests + warmup_requests="$(adaptive_warmup_request_count "$concurrency")" + local -a bench_args=( + --backend vllm + --host 127.0.0.1 + --port "$ENGINE_PORT" + --dataset-name "$BENCH_DATASET_NAME" + --random-input-len "$isl" + --random-output-len "$osl" + --random-range-ratio "$RANDOM_RANGE_RATIO" + --num-prompts "$num_prompts" + --max-concurrency "$concurrency" + --request-rate 10000 + --warmup-requests "$warmup_requests" + --output-file "$output_file" + --output-details + --disable-tqdm + ) + if [[ "$BENCH_DATASET_NAME" == "random" ]]; then + bench_args+=(--dataset-path "$DATASET_PATH") + else + bench_args+=(--tokenize-prompt) + fi + + if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then + local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}") + if [[ "$BENCH_DATASET_NAME" == "random" ]]; then + volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro") + fi + docker run --rm \ + --network host \ + "${volume_args[@]}" \ + -e PYTHONUNBUFFERED=1 \ + -e HF_HUB_OFFLINE=1 \ + -e TRANSFORMERS_OFFLINE=1 \ + "$DOCKER_CLIENT_IMAGE" \ + python -m sglang.bench_serving "${bench_args[@]}" + else + "$PYTHON" -m sglang.bench_serving "${bench_args[@]}" + fi +} + +export -f engine_run_bench +export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT +export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON + +adaptive_main "$@" diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh new file mode 100755 index 0000000..e5d87eb --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh @@ -0,0 +1,190 @@ + +#!/usr/bin/env bash +# Find the Total-TPS saturation concurrency for each vLLM-Ascend TP/DP/ISL/OSL +# shape on Ascend 910C (GLM-5.2). +# +# Differences vs the H20 vLLM variant: +# - GPU monitor overridden to use npu-smi (shared library hardcodes nvidia-smi). +# - engine_run_bench runs the client inside the vllm-ascend container via +# `docker exec`. The official vllm-ascend image does NOT ship +# sglang.bench_serving; see envs/ASCEND_910C_ENV_SETUP.md for the client story +# (install bench_serving into the container or use an external sglang image). +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")" + +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/lib.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/platform.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/adaptive_config.env" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh" + +ENGINE="vllm" +ENGINE_PORT="$VLLM_PORT" +RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}" +ACTIVE_ENGINE_SERVER_LOG="" + +PYTHON="${PYTHON:-$(command -v python3)}" +DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}" + +# Override the shared library's nvidia-smi GPU monitor with an npu-smi sampler +# that emits the same CSV shape (timestamp, index, mem.used, mem.total, util). +adaptive_start_gpu_monitor() { + local csv_path="$1" + mkdir -p "$(dirname "$csv_path")" + # 910C: delegate to the shared npu-smi sampler so parse_backend.py sees the + # same CSV shape as nvidia-smi (see scripts/common/npu_smi_sampler.py). + "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/npu_smi_sampler.py" "$GPU_MEM_SAMPLE_INTERVAL_S" > "$csv_path" 2>/dev/null & + echo $! +} + +engine_is_healthy() { + curl --fail --silent --show-error --max-time 5 \ + "http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1 +} + +engine_stop_server() { + local tp="$1" + local dp="$2" + local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid" + if [[ -f "$pid_file" ]]; then + local pid + pid="$(cat "$pid_file")" + if kill -0 "$pid" 2>/dev/null; then + log "stopping vllm-ascend server pid=${pid} tp=${tp} dp=${dp}" + kill "$pid" 2>/dev/null || true + sleep 5 + kill -9 "$pid" 2>/dev/null || true + fi + rm -f "$pid_file" + fi + # Also remove the container by name in case the host process is gone but the + # container lingers (e.g. orphaned by a kill -9). + docker rm -f "${CONTAINER_NAME}_tp${tp}_dp${dp}" >/dev/null 2>&1 || true + ACTIVE_ENGINE_SERVER_LOG="" + sleep 2 +} + +engine_build_server_args() { + local tp="$1" + local dp="$2" + local -a args=( + vllm serve "$MODEL_PATH" + --served-model-name "$SERVED_MODEL_NAME" + --trust-remote-code + --kv-cache-dtype "$KV_CACHE_DTYPE" + --block-size "$BLOCK_SIZE" + --tensor-parallel-size "$tp" + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" + --max-model-len "$MAX_MODEL_LEN" + --max-num-seqs "$MAX_NUM_SEQS" + --host 0.0.0.0 + --port "$ENGINE_PORT" + ) + if (( dp > 1 )); then + args+=(--data-parallel-size "$dp") + fi + printf '%q ' "${args[@]}" +} + +engine_start_server() { + local tp="$1" + local dp="$2" + local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" + log "starting vllm-ascend server tp=${tp} dp=${dp}" + bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1 + if ! engine_is_healthy; then + log "ERROR: vllm-ascend health check failed tp=${tp} dp=${dp}" + return 1 + fi + ACTIVE_ENGINE_SERVER_LOG="$( + find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \ + -name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \ + -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2- + )" + log "vllm-ascend server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}" +} + +engine_detect_oom() { + local detail_log="$1" + local tp="$2" + local dp="$3" + local pattern='out of memory|OutOfMemory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory|NPU out of memory|acl.*memory|HBM' + local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" + local -a logs=("$detail_log" "$outer_log") + if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then + logs+=("$ACTIVE_ENGINE_SERVER_LOG") + fi + grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null +} + +engine_run_bench() { + local isl="$1" + local osl="$2" + local concurrency="$3" + local num_prompts="$4" + local output_file="$5" + local container_name="${CONTAINER_NAME}_tp${tp}_dp${dp}" + local container_output="/tmp/bench_outputs/adaptive_$(basename "$output_file")" + if [[ "$output_file" == "/dev/null" ]]; then + container_output="/dev/null" + fi + + local warmup_requests + warmup_requests="$(adaptive_warmup_request_count "$concurrency")" + + local -a bench_args=( + --backend vllm + --host 127.0.0.1 + --port "$ENGINE_PORT" + --model "$SERVED_MODEL_NAME" + --dataset-name "$BENCH_DATASET_NAME" + --random-input-len "$isl" + --random-output-len "$osl" + --random-range-ratio "$RANDOM_RANGE_RATIO" + --num-prompts "$num_prompts" + --max-concurrency "$concurrency" + --request-rate 10000 + --warmup-requests "$warmup_requests" + --output-file "$container_output" + --output-details + --disable-tqdm + ) + if [[ "$BENCH_DATASET_NAME" == "random" ]]; then + bench_args+=(--dataset-path "$DATASET_PATH") + else + bench_args+=(--tokenize-prompt) + fi + + # Run the benchmark client inside the running vllm-ascend container via + # `docker exec`. NOTE: the official vllm-ascend image does NOT include + # sglang.bench_serving. If `docker exec ... sglang.bench_serving` fails with + # ModuleNotFoundError, either (a) pip install sglang into the container, or + # (b) set USE_DOCKER_CLIENT=1 and run an external sglang image against the + # host port. See envs/ASCEND_910C_ENV_SETUP.md. + docker exec "$container_name" mkdir -p /tmp/bench_outputs 2>/dev/null || true + docker exec "$container_name" \ + env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \ + "$CONTAINER_PYTHON" -m sglang.bench_serving "${bench_args[@]}" || return $? + + if [[ "$output_file" != "/dev/null" ]]; then + docker cp "$container_name:${container_output}" "$output_file" || return 1 + fi +} + +export -f engine_run_bench +export ENGINE_PORT CONTAINER_NAME CONTAINER_PYTHON MODEL_PATH RESULT_BASE SERVED_MODEL_NAME +export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON + +export SEARCH_START_CONCURRENCY=16 +export SEARCH_ADDEND=16 +export SEARCH_INITIAL_BACKOFF_CONCURRENCIES="8 1" +export TTFT_GROUP_SKIP_MS="${TTFT_GROUP_SKIP_MS:-8000}" + +adaptive_main "$@" diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_bench.sh b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_bench.sh new file mode 100755 index 0000000..a4030b5 --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_bench.sh @@ -0,0 +1,546 @@ +#!/usr/bin/env bash +# TP×DP matrix benchmark for DeepSeek-V4-Flash on vLLM (Docker). +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")" + +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/lib.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/platform.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +# Export variables used inside functions that are called via bash -c subshells. +export DATASET_PATH MODEL_PATH RESULT_BASE \ + DOCKER_IMAGE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT + +RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" +RESULT_BASE="${SCRIPT_DIR}/results" +MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR}/matrix.json}" +MATRIX_MODE="${MATRIX_MODE:-Y}" +SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}" +GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}" +DRY_RUN="${DRY_RUN:-0}" +GRID_LIMIT="${GRID_LIMIT:-0}" + +if [[ -x "${VENV_CLIENT}/bin/python" ]]; then + PYTHON="${VENV_CLIENT}/bin/python" +else + PYTHON="$(command -v python3)" +fi + +DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}" +DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}" + +log_dir_global="${RESULT_BASE}/${RUN_ID}/logs" +mkdir -p "$log_dir_global" +log_init "${log_dir_global}/orchestrator.log" + +log "experiment=${EXPERIMENT_NAME} run_id=${RUN_ID} platform=${PLATFORM} hardware=${HARDWARE}" +log "matrix_mode=${MATRIX_MODE} matrix_file=${MATRIX_FILE} dry_run=${DRY_RUN} grid_limit=${GRID_LIMIT}" + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +is_server_healthy() { + curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${VLLM_PORT}/health" >/dev/null 2>&1 +} + +stop_server() { + local tp="$1" + local dp="$2" + + local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid" + if [[ -f "$pid_file" ]]; then + local pid + pid="$(cat "$pid_file")" + if kill -0 "$pid" 2>/dev/null; then + log "stopping vllm server pid=${pid} (tp=${tp}, dp=${dp})" + kill "$pid" 2>/dev/null || true + sleep 5 + kill -9 "$pid" 2>/dev/null || true + fi + rm -f "$pid_file" + fi + + # Fallback: remove any Docker container started by this experiment. + docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true + + # Fallback: kill any vllm serve processes for this model. + pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true + pkill -9 -f "vllm serve.*${MODEL_PATH}" 2>/dev/null || true + sleep 2 +} + +build_server_args() { + local tp="$1" + local dp="$2" + + local args=( + "vllm serve" "$MODEL_PATH" + --trust-remote-code + --kv-cache-dtype "$KV_CACHE_DTYPE" + --block-size "$BLOCK_SIZE" + --tensor-parallel-size "$tp" + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" + --max-model-len "$MAX_MODEL_LEN" + --max-num-seqs "$MAX_NUM_SEQS" + --no-enable-flashinfer-autotune + --host 0.0.0.0 + --port "$VLLM_PORT" + ) + if [[ "$dp" -gt 1 ]]; then + args+=( + --data-parallel-size "$dp" + ) + fi + printf '%s ' "${args[@]}" +} + +start_server() { + local tp="$1" + local dp="$2" + + log "starting vllm server tp=${tp} dp=${dp}" + bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" \ + >> "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1 + + if ! is_server_healthy; then + log "error: vllm server tp=${tp} dp=${dp} failed health check on port ${VLLM_PORT}" + return 1 + fi + log "vllm server tp=${tp} dp=${dp} is healthy on port ${VLLM_PORT}" +} + +restart_server() { + local tp="$1" + local dp="$2" + log "restarting vllm server tp=${tp} dp=${dp} after non-OOM failure" + stop_server "$tp" "$dp" + sleep 10 + start_server "$tp" "$dp" +} + +run_bench_serving() { + # Run sglang.bench_serving either natively or inside the SGLang Docker image. + if [[ "${USE_DOCKER_CLIENT:-1}" == "1" ]]; then + local vol_args=() + vol_args+=("-v" "${MODEL_PATH}:${MODEL_PATH}:ro") + if [[ -n "${DATASET_PATH:-}" && -f "${DATASET_PATH}" ]]; then + vol_args+=("-v" "${DATASET_PATH}:${DATASET_PATH}:ro") + fi + vol_args+=("-v" "${RESULT_BASE}:${RESULT_BASE}") + docker run --rm \ + --network host \ + "${vol_args[@]}" \ + -e PYTHONUNBUFFERED=1 \ + -e HF_HUB_OFFLINE=1 \ + -e TRANSFORMERS_OFFLINE=1 \ + -e HF_DATASETS_OFFLINE=1 \ + "${DOCKER_CLIENT_IMAGE}" \ + python -m sglang.bench_serving "$@" + else + "$PYTHON" -m sglang.bench_serving "$@" + fi +} +export -f run_bench_serving + +run_warmup() { + local input_len="$1" + local output_len="$2" + + log "warming up (input=${input_len}, output=${output_len}, num=1)" + bash -c ' + run_bench_serving \ + --backend vllm \ + --host 127.0.0.1 \ + --port "'"$VLLM_PORT"'" \ + --dataset-name random \ + --dataset-path "'"$DATASET_PATH"'" \ + --random-input-len "'"$input_len"'" \ + --random-output-len "'"$output_len"'" \ + --num-prompts 1 \ + --max-concurrency 1 \ + --request-rate 10000 \ + --output-file /dev/null \ + --output-details \ + >> "'"${log_dir_global}/warmup.log"'" 2>&1 + ' + log "warmup completed" +} + +scenario_already_completed() { + local output_file="$1" + local expected="$2" + [[ -s "$output_file" ]] || return 1 + local completed + completed="$("$PYTHON" -c " +import json, sys +path = sys.argv[1] +try: + with open(path, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if line: + data = json.loads(line) + print(data.get('completed', 0)) + break +except Exception: + print(0) +" "$output_file")" + [[ "${completed:-0}" -ge "$expected" ]] +} + +scenario_already_processed() { + local result_root="$1" + local scenario_name="$2" + local json_path="${result_root}/results.json" + [[ -f "$json_path" ]] || return 1 + "$PYTHON" -c " +import json, sys +path, name = sys.argv[1], sys.argv[2] +try: + with open(path, 'r', encoding='utf-8') as f: + data = json.load(f) + for s in data.get('scenarios', []): + if s.get('name') == name: + if s.get('status') or s.get('metrics', {}).get('success', 0) > 0: + sys.exit(0) +except Exception: + pass +sys.exit(1) +" "$json_path" "$scenario_name" +} + +detect_oom() { + local detail_log="$1" + local server_outer_log="$2" + local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory|NPU out of memory|acl.*memory|HBM' + if grep -Eiq "$pattern" "$detail_log" "$server_outer_log" 2>/dev/null; then + return 0 + fi + return 1 +} + +start_gpu_monitor() { + local csv_path="$1" + mkdir -p "$(dirname "$csv_path")" + # 910C: use npu-smi via a standalone python sampler that emits the same CSV + # shape as nvidia-smi, so parse_backend.py needs no changes. + "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/npu_smi_sampler.py" "$GPU_MEM_SAMPLE_INTERVAL_S" > "$csv_path" 2>/dev/null & + echo $! +} + +stop_gpu_monitor() { + local pid="$1" + if kill -0 "$pid" 2>/dev/null; then + kill "$pid" 2>/dev/null || true + sleep 1 + kill -9 "$pid" 2>/dev/null || true + fi +} + +append_scenario_record() { + local result_root="$1" + local json_path="$result_root/results.json" + shift + local scenario_json + scenario_json="$("$PYTHON" -c " +import json, sys +pairs = [a.split('=', 1) for a in sys.argv[1:]] +d = {} +for k, v in pairs: + try: + d[k] = json.loads(v) + except json.JSONDecodeError: + d[k] = v +print(json.dumps(d, ensure_ascii=False)) +" "$@")" + PYTHON="$PYTHON" append_scenario_to_json "$json_path" "$scenario_json" +} + +record_skipped_csv() { + local csv_path="$1" + shift + # Args: key=value + local row + row="$("$PYTHON" -c " +import csv, json, sys, io +pairs = [a.split('=', 1) for a in sys.argv[1:]] +d = {} +for k, v in pairs: + try: + d[k] = json.loads(v) + except json.JSONDecodeError: + d[k] = v +buf = io.StringIO() +writer = csv.DictWriter(buf, fieldnames=['engine','tp','dp','mark','isl','osl','concurrency','status','reason','detail_log'], extrasaction='ignore') +writer.writerow(d) +print(buf.getvalue().strip()) +" "$@")" + echo "$row" >> "$csv_path" +} + +skip_remaining_scenarios() { + local result_root="$1" + local scenario_tsv="$2" + local start_index="$3" + local status="$4" + local reason="$5" + local tp="$6" + local dp="$7" + local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv" + + local i=0 + tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl osl conc num; do + if (( i < start_index )); then + i=$((i + 1)) + continue + fi + i=$((i + 1)) + local sname="c${conc}_i${isl}_o${osl}" + if scenario_already_processed "$result_root" "$sname"; then + continue + fi + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"${status}\"" \ + "note=\"${reason}\"" + record_skipped_csv "$skipped_csv" \ + "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=${status}" "reason=${reason}" + done +} + +# --------------------------------------------------------------------------- +# Per-configuration runner +# --------------------------------------------------------------------------- + +run_parallel_config() { + local tp="$1" + local dp="$2" + local config_label="tp${tp}_dp${dp}" + local result_root="${RESULT_BASE}/${RUN_ID}/${config_label}" + local raw_dir="${result_root}/raw_outputs" + local gpu_log_dir="${result_root}/gpu_logs" + local phase_log_dir="${result_root}/logs" + mkdir -p "$raw_dir" "$gpu_log_dir" "$phase_log_dir" + + log "===== ${config_label} START =====" + + # Generate scenario list for this config. + local scenario_tsv="${result_root}/scenarios.tsv" + "$PYTHON" "${SCRIPT_DIR}/generate_scenarios.py" \ + --matrix "$MATRIX_FILE" \ + --mode "$MATRIX_MODE" \ + > "$scenario_tsv" + local total_scenarios + total_scenarios="$(tail -n +2 "$scenario_tsv" | wc -l)" + log "generated ${total_scenarios} scenarios for ${config_label}" + + # Write metadata. + ensure_result_root "$result_root" + write_metadata_json \ + "${result_root}/results.json" \ + "${EXPERIMENT_NAME}_${config_label}" \ + "$RUN_ID" \ + "$MODEL_PATH" \ + "vllm" \ + "vllm" \ + "$HARDWARE" \ + "$ACCELERATOR" \ + "$CHIP" \ + "experiments/${EXPERIMENT_NAME}/run_bench.sh" \ + "$DOCKER_IMAGE" \ + "H20 vLLM TP×DP matrix for DeepSeek-V4-Flash" + + local server_args_str + server_args_str="$(build_server_args "$tp" "$dp")" + jq --arg tp "$tp" --arg dp "$dp" --arg cuda "$CUDA_VISIBLE_DEVICES" --arg args "$server_args_str" \ + '.config = { + "tp": ($tp | tonumber), + "dp": ($dp | tonumber), + "cuda_visible_devices": $cuda, + "backend": "vllm", + "server_start_script": "experiments/'${EXPERIMENT_NAME}'/start_vllm_dp.sh", + "server_args": $args + }' "${result_root}/results.json" > "${result_root}/results.json.tmp" && \ + mv "${result_root}/results.json.tmp" "${result_root}/results.json" + + if [[ "$DRY_RUN" == "1" ]]; then + log "DRY_RUN: would start server with args: ${server_args_str}" + local line + tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl osl conc num; do + log "DRY_RUN: ${config_label} scenario mark=${mark} c=${conc} i=${isl} o=${osl} n=${num}" + done + log "===== ${config_label} DONE (dry run) =====" + return 0 + fi + + # Initialize skipped_after_oom.csv for this run. + local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv" + if [[ ! -f "$skipped_csv" ]]; then + echo "engine,tp,dp,mark,isl,osl,concurrency,status,reason,detail_log" > "$skipped_csv" + fi + + # Start server once for this TP×DP config. + if ! start_server "$tp" "$dp"; then + log "ERROR: ${config_label} failed to start; skipping all scenarios" + skip_remaining_scenarios "$result_root" "$scenario_tsv" 0 "SKIPPED_SERVICE_START_FAILED" "service failed to start" "$tp" "$dp" + log "===== ${config_label} DONE =====" + return 0 + fi + + # Warmup with a small prompt before the first scenario. + run_warmup 1024 128 || true + + # Read scenarios into an array so we can skip remaining entries on failure. + local -a scenarios=() + while IFS= read -r line; do + scenarios+=("$line") + done < <(tail -n +2 "$scenario_tsv") + + local i mark isl osl conc num + local output_file detail_log gpu_csv sname bench_rc + for (( i = 0; i < ${#scenarios[@]}; i++ )); do + IFS=$'\t' read -r mark isl osl conc num <<< "${scenarios[$i]}" + + if [[ "$GRID_LIMIT" -gt 0 && "$i" -ge "$GRID_LIMIT" ]]; then + log "GRID_LIMIT=${GRID_LIMIT} reached; skipping remaining scenarios" + skip_remaining_scenarios "$result_root" "$scenario_tsv" "$i" "SKIPPED_GRID_LIMIT" "GRID_LIMIT reached" "$tp" "$dp" + break + fi + + sname="c${conc}_i${isl}_o${osl}" + output_file="${raw_dir}/vllm_main_${conc}_${isl}_${osl}.jsonl" + detail_log="${phase_log_dir}/vllm_${config_label}_${sname}.log" + gpu_csv="${gpu_log_dir}/gpu_mem_${conc}_${isl}_${osl}.csv" + + if scenario_already_completed "$output_file" "$num" || scenario_already_processed "$result_root" "$sname"; then + log "skipping already-processed ${config_label} scenario: ${sname}" + continue + fi + + log "running ${config_label} scenario: mark=${mark} c=${conc} i=${isl} o=${osl} n=${num}" + + local gpu_pid + gpu_pid="$(start_gpu_monitor "$gpu_csv")" + + bench_rc=0 + timeout "$SCENARIO_TIMEOUT_S" bash -c ' + run_bench_serving \ + --backend vllm \ + --host 127.0.0.1 \ + --port "'"$VLLM_PORT"'" \ + --dataset-name random \ + --dataset-path "'"$DATASET_PATH"'" \ + --random-input-len "'"$isl"'" \ + --random-output-len "'"$osl"'" \ + --num-prompts "'"$num"'" \ + --max-concurrency "'"$conc"'" \ + --request-rate 10000 \ + --output-file "'"$output_file"'" \ + --output-details \ + > "'"$detail_log"'" 2>&1 + ' || bench_rc=$? + + stop_gpu_monitor "$gpu_pid" + + if [[ "$bench_rc" -eq 0 ]]; then + log "finished ${config_label} scenario: output=${output_file}" + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"completed\"" \ + "note=\"benchmark finished successfully\"" + continue + fi + + # Failure handling. + if detect_oom "$detail_log" "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log"; then + log "ERROR: ${config_label} scenario ${sname} triggered OOM; stopping config" + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"OOM\"" \ + "note=\"detected CUDA out-of-memory\"" + record_skipped_csv "$skipped_csv" \ + "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=OOM" "reason=detected CUDA out-of-memory" "detail_log=${detail_log}" + stop_server "$tp" "$dp" + skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_AFTER_OOM" "previous case OOM" "$tp" "$dp" + break + fi + + log "ERROR: ${config_label} scenario ${sname} failed (rc=${bench_rc}); see ${detail_log}" + if [[ "$mark" == "P" ]]; then + log "optional (P) scenario failed; recording as skipped and continuing" + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"skipped_optional\"" \ + "note=\"optional scenario failed (rc=${bench_rc})\"" + record_skipped_csv "$skipped_csv" \ + "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=skipped_optional" "reason=optional scenario failed (rc=${bench_rc})" "detail_log=${detail_log}" + continue + fi + + # Mandatory scenario failed but not OOM: try to restart the server. + if restart_server "$tp" "$dp"; then + run_warmup 1024 128 || true + log "resuming ${config_label} after server restart" + continue + fi + + log "ERROR: ${config_label} server restart failed; skipping remaining scenarios" + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"FAILED\"" \ + "note=\"scenario failed and server restart failed (rc=${bench_rc})\"" + record_skipped_csv "$skipped_csv" \ + "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=FAILED" "reason=scenario failed and server restart failed" "detail_log=${detail_log}" + skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_RESTART_FAILED" "server restart failed" "$tp" "$dp" + break + done + + stop_server "$tp" "$dp" + + # Parse results. + log "parsing ${config_label} results" + "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/parse_backend.py" "$result_root" --backend vllm \ + >> "${phase_log_dir}/parse.log" 2>&1 || { + log "WARNING: parser failed for ${config_label}; see ${phase_log_dir}/parse.log" + } + + log "===== ${config_label} DONE =====" +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +# Cleanup any leftovers. +for cfg in "${PARALLEL_CONFIGS[@]}"; do + read -r tp dp <<< "$cfg" + stop_server "$tp" "$dp" +done + +# Run each parallel configuration. +for cfg in "${PARALLEL_CONFIGS[@]}"; do + read -r tp dp <<< "$cfg" + run_parallel_config "$tp" "$dp" +done + +# Generate cross-configuration comparison. +log "generating comparison report" +"$PYTHON" "${SCRIPT_DIR}/compare.py" \ + --run-root "${RESULT_BASE}/${RUN_ID}" \ + --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \ + >> "${log_dir_global}/compare.log" 2>&1 || { + log "WARNING: comparison script failed; see ${log_dir_global}/compare.log" + } + +log "all results saved to ${RESULT_BASE}/${RUN_ID}" diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/start_vllm_docker.sh b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/start_vllm_docker.sh new file mode 100755 index 0000000..2553617 --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/start_vllm_docker.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# Start vLLM-Ascend server in Docker for a given TP×DP configuration on 910C. +# Usage: start_vllm_docker.sh +# +# Relies on the Ascend Docker Runtime being the default docker runtime on this +# host (see /etc/docker/daemon.json). NPU dies are injected via the +# ASCEND_VISIBLE_DEVICES env var; no --gpus / --device flags are needed. +set -e + +TP="${1}" +DP="${2}" + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}" +mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp" + +IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}" +PORT="${VLLM_PORT:-30050}" +NAME="${CONTAINER_NAME:-${EXPERIMENT}_vllm_tp${TP}_dp${DP}}" +NAME="${NAME}_tp${TP}_dp${DP}" +PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${TP}_dp${DP}.pid" + +LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_vllm_docker_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log" +rm -f "$PID_FILE" + +# Clean up any stale container with the same name. +docker rm -f "$NAME" >/dev/null 2>&1 || true + +# vLLM-Ascend launch args. Differences vs NVIDIA vLLM: +# - no --no-enable-flashinfer-autotune (Ascend uses its own attention path) +# - --kv-cache-dtype may need to be fp16 if the image rejects fp8 on 910C +SERVER_ARGS=( + "$MODEL_PATH" + --served-model-name "$SERVED_MODEL_NAME" + --trust-remote-code + --kv-cache-dtype "$KV_CACHE_DTYPE" + --block-size "$BLOCK_SIZE" + --tensor-parallel-size "$TP" + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" + --max-model-len "$MAX_MODEL_LEN" + --max-num-seqs "$MAX_NUM_SEQS" + --host 0.0.0.0 + --port "$PORT" +) + +if [[ "$DP" -gt 1 ]]; then + SERVER_ARGS+=(--data-parallel-size "$DP") +fi + +SERVER_ARGS_STR="vllm serve ${SERVER_ARGS[*]}" + +echo "=== Starting vLLM-Ascend server in Docker (TP=${TP}, DP=${DP}) ===" +echo "Image: $IMAGE" +echo "Model: $MODEL_PATH" +echo "Container name: $NAME" +echo "Host port: $PORT" +echo "ASCEND_VISIBLE_DEVICES: ${ASCEND_VISIBLE_DEVICES}" +echo "Command: $SERVER_ARGS_STR" +echo "Log: $LOG" + +# Run docker in the foreground; nohup backgrounds it and the host PID lets the +# adaptive search stop the container by killing the process (the container has +# --rm so it self-cleans). The Ascend Docker Runtime is the default runtime, so +# no --runtime flag is required. +nohup docker run --rm \ + --name "$NAME" \ + --ipc host \ + --shm-size 16g \ + --network host \ + --ulimit memlock=-1 \ + -v "${MODEL_PATH}:${MODEL_PATH}:ro" \ + -v "${RUNTIME_BASE}/tmp:/tmp" \ + -e ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES}" \ + -e PYTORCH_NPU_ALLOC_CONF=expandable_segments:True \ + -e PYTHONUNBUFFERED=1 \ + -e HF_HUB_OFFLINE=1 \ + -e TRANSFORMERS_OFFLINE=1 \ + "$IMAGE" \ + vllm serve "${SERVER_ARGS[@]}" \ + > "$LOG" 2>&1 & + +PID=$! +echo $PID > "$PID_FILE" +echo "PID: $PID" +echo "Waiting for health on port ${PORT}..." + +for i in $(seq 1 240); do + if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then + echo "vLLM-Ascend server is ready at http://127.0.0.1:${PORT}" + echo "Log: $LOG" + exit 0 + fi + if ! kill -0 $PID 2>/dev/null; then + echo "ERROR: Docker vLLM-Ascend server exited early" + tail -200 "$LOG" + exit 1 + fi + echo "Waiting... ($i/240)" + sleep 5 +done + +echo "ERROR: vLLM-Ascend server not healthy after 240 retries" +tail -200 "$LOG" +exit 1 diff --git a/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/start_vllm_dp.sh b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/start_vllm_dp.sh new file mode 100755 index 0000000..1f33f8f --- /dev/null +++ b/experiments/910c/dsv4_910c_vllm_tp_dp_matrix/start_vllm_dp.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Start vLLM-Ascend server for a given TP×DP configuration on 910C. +# Usage: start_vllm_dp.sh +# +# 910C only supports the Docker path (vllm-ascend runs inside a container with +# the Ascend Docker Runtime). USE_DOCKER=0 is not supported on this platform; +# set USE_DOCKER=1 (the default) or call start_vllm_docker.sh directly. +set -e + +TP="${1}" +DP="${2}" + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +if [[ "${USE_DOCKER:-1}" == "1" ]]; then + exec "${SCRIPT_DIR}/start_vllm_docker.sh" "$@" +fi + +echo "ERROR: USE_DOCKER=0 (native vllm-ascend) is not supported on 910C." >&2 +echo " vllm-ascend requires the Ascend Docker Runtime; set USE_DOCKER=1." >&2 +exit 1 diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/adaptive_config.env b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/adaptive_config.env new file mode 100644 index 0000000..3b05c7c --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/adaptive_config.env @@ -0,0 +1,58 @@ +# Adaptive concurrency search settings. +# +# For each fixed (TP, DP, ISL, OSL), probe: +# C = start, start * multiplier, ... up to max +# and stop after Total TPS has less than TPS_MIN_GAIN_PCT meaningful growth for +# PLATEAU_PATIENCE consecutive points. + +SEARCH_START_CONCURRENCY="${SEARCH_START_CONCURRENCY:-1}" +SEARCH_MAX_CONCURRENCY="${SEARCH_MAX_CONCURRENCY:-256}" + +# At the add16 initial probe, restart and retry C=8 then C=1 after an OOM. +ENABLE_INITIAL_OOM_BACKOFF="${ENABLE_INITIAL_OOM_BACKOFF:-1}" +SEARCH_MULTIPLIER="${SEARCH_MULTIPLIER:-2}" +NUM_PROMPTS_MULTIPLIER="${NUM_PROMPTS_MULTIPLIER:-5}" + +# A gain below 2% is treated as throughput saturation. Two consecutive +# low-gain points prevent one noisy measurement from stopping the search. +TPS_MIN_GAIN_PCT="${TPS_MIN_GAIN_PCT:-2.0}" +PLATEAU_PATIENCE="${PLATEAU_PATIENCE:-2}" + +# TTFT SLO early-stop settings. +# When ttft_p95_ms exceeds TTFT_SLO_MS, stop searching the current (ISL, OSL) +# shape and move on to the next scenario. +TTFT_SLO_MS="${TTFT_SLO_MS:-4000}" +ENABLE_TTFT_SLO_STOP="${ENABLE_TTFT_SLO_STOP:-1}" + +# Keep the same random workload semantics as the fixed matrix baseline. +# DATASET_PATH must contain at least SEARCH_MAX_CONCURRENCY times +# NUM_PROMPTS_MULTIPLIER valid two-turn conversations. Set this explicitly to +# random-ids to use generated token IDs without a ShareGPT seed dataset. +BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}" +# SGLang interprets 0.0 as Uniform[1, requested_len]. Use 1.0 for fixed +# ISL/OSL points; lower values intentionally benchmark a length distribution. +RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1.0}" +# Before each measured point, warm up with the same concurrency so lazy kernel +# compilation and CUDA graph capture are excluded from TTFT/TPS. 0 means no +# cap; set a positive cap only when very high-concurrency warmup is impractical. +BENCH_WARMUP_MAX_REQUESTS="${BENCH_WARMUP_MAX_REQUESTS:-0}" + +# Reject a point if the completed request count or actual token lengths do not +# match the requested workload. +INPUT_LENGTH_TOLERANCE_PCT="${INPUT_LENGTH_TOLERANCE_PCT:-5.0}" +OUTPUT_LENGTH_TOLERANCE_PCT="${OUTPUT_LENGTH_TOLERANCE_PCT:-10.0}" + +MAX_POINT_RETRIES="${MAX_POINT_RETRIES:-1}" +SERVER_RESTART_COOLDOWN_S="${SERVER_RESTART_COOLDOWN_S:-10}" +SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}" +GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}" + +# Optional space-separated filters, useful for smoke tests: +# TP_LIST="8" ISL_LIST="1024" OSL_LIST="128" +TP_LIST="${TP_LIST:-}" +ISL_LIST="${ISL_LIST:-}" +OSL_LIST="${OSL_LIST:-}" + +DRY_RUN="${DRY_RUN:-0}" +# Counts ISL/OSL shapes per TP/DP config, not individual concurrency probes. +GRID_LIMIT="${GRID_LIMIT:-0}" diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/compare.py b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/compare.py new file mode 100644 index 0000000..002fa0c --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/compare.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Cross TP×DP configuration comparison for dsv4_h20_vllm_tp_dp_matrix. + +Usage: + python3 compare.py --run-root results/ [--output comparison.md] +""" +import argparse +import json +import re +from collections import defaultdict +from pathlib import Path + + +def load_result(result_root: Path) -> dict: + path = result_root / "results.json" + with open(path, "r", encoding="utf-8") as f: + return json.load(f) + + +def slo_status(ttft_p95_ms: float, tpot_mean_ms: float, + ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str: + ttft_ok = ttft_p95_ms < ttft_limit_ms + tpot_ok = tpot_mean_ms < tpot_limit_ms + if ttft_ok and tpot_ok: + return "PASS" + if ttft_ok or tpot_ok: + return "PARTIAL" + return "FAIL" + + +def gpu_memory_str(gpu: dict | None) -> str: + if not gpu: + return "-" + peak = gpu.get("peak_used_mb", 0) + total = gpu.get("memory_total_mb", 0) + if total: + return f"{peak:.0f}/{total:.0f} ({100*peak/total:.1f}%)" + return f"{peak:.0f}" + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--run-root", type=Path, required=True) + parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md")) + parser.add_argument("--ttft-limit", type=float, default=3000.0) + parser.add_argument("--tpot-limit", type=float, default=50.0) + args = parser.parse_args() + + # Discover configurations: tp*_dp* directories. + configs = [] + for subdir in sorted(args.run_root.iterdir()): + if not subdir.is_dir(): + continue + name = subdir.name + if not (name.startswith("tp") and "_dp" in name): + continue + results_json = subdir / "results.json" + if not results_json.exists(): + continue + configs.append((name, load_result(subdir))) + + if not configs: + print(f"No tp*_dp* results found under {args.run_root}") + return + + model = configs[0][1].get("metadata", {}).get("model", "unknown") + hardware = configs[0][1].get("metadata", {}).get("hardware", "unknown") + + # Group by scenario name. + by_scenario: dict[str, dict[str, dict]] = defaultdict(dict) + skipped: dict[str, dict[str, str]] = defaultdict(dict) + for label, data in configs: + for s in data.get("scenarios", []): + key = s["name"] + if s.get("status") == "skipped_oom": + skipped[key][label] = s.get("note", "skipped") + else: + by_scenario[key][label] = s + + with open(args.output, "w", encoding="utf-8") as f: + f.write(f"# vLLM TP×DP matrix comparison ({hardware})\n\n") + f.write("## Summary\n\n") + f.write(f"- Model: `{model}`\n") + f.write(f"- Hardware: {hardware}\n") + f.write("- Backend: vLLM (Docker)\n") + f.write("- Benchmark client: `sglang.bench_serving`\n") + f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n") + + # Configuration overview. + f.write("### Configurations\n\n") + f.write("| Config | TP | DP | GPUs/replica | Notes |\n") + f.write("|---|---:|---:|---:|---|\n") + for label, data in configs: + cfg = data.get("config", {}) + tp = cfg.get("tp", "?") + dp = cfg.get("dp", "?") + f.write(f"| {label} | {tp} | {dp} | {tp} | server args recorded per ISL in results.json |\n") + f.write("\n") + + # Side-by-side table. + f.write("## Side-by-side results\n\n") + headers = [ + "Scenario", "ISL", "OSL", "Config", "Conc", "Req/s", "OutTok/s", + "TTFT P95(ms)", "TTFT P99(ms)", "TPOT Mean(ms)", "TPOT P95(ms)", + "TPOT P99(ms)", "E2E P99(ms)", "Peak GPU mem", "SLO" + ] + f.write("| " + " | ".join(headers) + " |\n") + f.write("|" + "|".join(["---"] * len(headers)) + "|\n") + + for scenario_name in sorted(by_scenario.keys(), key=lambda x: tuple(map(int, re.findall(r"\d+", x)))): + _, isl, osl = re.findall(r"\d+", scenario_name) + # cfg_part not used; just for readability. + for label, data in configs: + s = by_scenario[scenario_name].get(label) + if s is None: + if scenario_name in skipped and label in skipped[scenario_name]: + note = skipped[scenario_name][label] + f.write(f"| {scenario_name} | {isl} | {osl} | {label} | - | - | - | - | - | - | - | - | - | - | {note} |\n") + continue + cfg = s["config"] + m = s["metrics"] + status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit) + gpu = m.get("gpu_memory") + f.write( + f"| {scenario_name} | {isl} | {osl} | {label} | {cfg['concurrency']} | " + f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | " + f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | " + f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | " + f"{m['e2e_ms']['p99']:.2f} | {gpu_memory_str(gpu)} | {status} |\n" + ) + + # Best throughput per ISL/OSL. + f.write("\n## Best throughput per (ISL, OSL)\n\n") + f.write("| ISL | OSL | Best Config | Concurrency | OutTok/s | TTFT P95(ms) | TPOT Mean(ms) | SLO |\n") + f.write("|---:|---:|---|---:|---:|---:|---:|---:|\n") + best_by_shape: dict[tuple[int, int], tuple[float, str, dict]] = {} + for scenario_name, backends in by_scenario.items(): + _, isl, osl = re.findall(r"\d+", scenario_name) + isl_i, osl_i = int(isl), int(osl) + for label, s in backends.items(): + m = s["metrics"] + out_tok = m["output_token_throughput"] + if (isl_i, osl_i) not in best_by_shape or out_tok > best_by_shape[(isl_i, osl_i)][0]: + best_by_shape[(isl_i, osl_i)] = (out_tok, label, s) + for (isl_i, osl_i), (out_tok, label, s) in sorted(best_by_shape.items()): + m = s["metrics"] + status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit) + f.write( + f"| {isl_i} | {osl_i} | {label} | {s['config']['concurrency']} | " + f"{out_tok:.2f} | {m['ttft_ms']['p95']:.2f} | {m['tpot_ms']['mean']:.2f} | {status} |\n" + ) + + f.write("\n## Notes\n\n") + f.write("- SLO check uses TTFT P95 and TPOT mean.\n") + f.write("- A PARTIAL indicates one of the two metrics is out of target; FAIL indicates both are out.\n") + f.write("- `Peak GPU mem` shows peak used / total MB and utilization percentage.\n") + f.write("- Optional (P) combinations that failed are marked as skipped/OOM and do not break the run.\n") + + print(f"Wrote comparison to {args.output}") + + +if __name__ == "__main__": + main() diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/config.env b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/config.env new file mode 100644 index 0000000..d8b3a32 --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/config.env @@ -0,0 +1,95 @@ +# TP×DP matrix experiment for GLM-5.2 on Ascend 910C (8 NPUs / 16 dies) using vLLM-Ascend. +# Tests vLLM with three parallel configurations: +# TP=2, DP=4 -> 2 dies per replica, 4 replicas +# TP=4, DP=2 -> 4 dies per replica, 2 replicas +# TP=8, DP=1 -> 8 dies, no data parallelism +# +# Platform: ascend_910c (see platforms/ascend_910c.env). +# Host: 910c.1 / NPU-NODE61, openEuler 22.03 SP4 aarch64, driver 25.5.2, CANN 9.0.0. +# Image: vllm-ascend; load the tarball from /mnt/models first (see envs/ASCEND_910C_ENV_SETUP.md). + +EXPERIMENT="glm52_910c_vllm_tp_dp_matrix" +MODEL_NAME="GLM-5.2" +# GLM-5.2 ships two quantized variants on this host; w4a8c8 is the default. +# Switch to /mnt/models/GLM-5.2-w8a8 by overriding MODEL_PATH if needed. +MODEL_PATH="${MODEL_PATH:-/mnt/models/GLM-5.2-w4a8c8}" +SERVED_MODEL_NAME="glm-5.2" + +VLLM_PORT="${VLLM_PORT:-30050}" + +# Dedicated container name so this experiment never touches other 910c runs. +CONTAINER_NAME="${CONTAINER_NAME:-vllm-ascend-glm52-910c}" + +# Python interpreter for the benchmark client inside the vllm-ascend container. +CONTAINER_PYTHON="${CONTAINER_PYTHON:-/usr/local/bin/python}" + +# vllm-ascend image. Override with the exact tag after `docker load`-ing one of: +# /mnt/models/vllm-ascend-glm5.2-a3-openeuler.tar (GLM5.2-tuned, recommended) +# /mnt/models/vllm-ascend-v0.23.0rc1-a3-openeuler.tar (general v0.23) +USE_DOCKER="${USE_DOCKER:-1}" +DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}" + +# Benchmark client Docker image. vLLM's image does not include sglang.bench_serving; +# reuse the vllm-ascend container itself for the client via `docker exec` (see +# run_adaptive_concurrency_add16.sh), so this is only used if USE_DOCKER_CLIENT=1 +# with an external sglang image. Default off on 910c. +DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}" +USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-0}" + +# Device selection. ASCEND_VISIBLE_DEVICES selects NPU cards 0..7; the Ascend +# Docker Runtime (default runtime on this host) injects the matching dies. +export ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}" +# Keep CUDA_VISIBLE_DEVICES for parity with the shared library; vllm-ascend +# ignores it on NPU but some helper code reads it. +export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}" + +# Runtime working directory for logs, pid files, and tmp. +RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}" + +# Parallel configurations to test. Format: "TP DP" +# Each Ascend910 card has 2 dies; TP addresses dies, so TP=8 uses 8 dies across +# 4 cards and leaves room for DP. TP=2/DP=4 and TP=4/DP=2 and TP=8/DP=1 all fit +# within 8 cards (16 dies). Override via PARALLEL_CONFIGS_STR="8,1". +if [[ -n "${PARALLEL_CONFIGS_STR:-}" ]]; then + declare -a PARALLEL_CONFIGS=() + for pair in $PARALLEL_CONFIGS_STR; do + PARALLEL_CONFIGS+=("${pair//,/ }") + done +else + declare -a PARALLEL_CONFIGS=( + "2 4" + "4 2" + "8 1" + ) +fi + +# vLLM-Ascend server settings for GLM-5.2 (w4a8c8). +# Notes: +# - KV cache dtype fp8 is supported on 910C; fall back to fp16 if the image rejects it. +# - block-size 128 matches Ascend page semantics (P99 of H20 uses 256; 910C favors 128). +# - MAX_MODEL_LEN: GLM-5.2 supports up to 128K context; cap at 131072. +# - gpu-memory-utilization maps to NPU HBM fraction on vllm-ascend (0.9 mirrors H20). +GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.9}" +KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}" +BLOCK_SIZE="${BLOCK_SIZE:-128}" +MAX_MODEL_LEN="${MAX_MODEL_LEN:-131072}" +MAX_NUM_SEQS="${MAX_NUM_SEQS:-256}" + +# vLLM-Ascend-specific launch flags injected by start_vllm_docker.sh. +# attention backend for 910C: use the fused/atb attention path. Adjust per image. +VLLM_ASCEND_ATTENTION_BACKEND="${VLLM_ASCEND_ATTENTION_BACKEND:-atb}" + +# Dataset used by sglang.bench_serving --dataset-name random. +DATASET_PATH="${DATASET_PATH:-${ROOT_DIR}/datasets/ShareGPT_V3_unfiltered_cleaned_split.json}" + +# Matrix and concurrency rules are defined in matrix.json by default. +MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR:-.}/matrix.json}" +MATRIX_MODE="${MATRIX_MODE:-Y}" + +export CONCURRENCY_SAMPLES="${CONCURRENCY_SAMPLES:-2}" + +SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}" +GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}" + +DRY_RUN="${DRY_RUN:-0}" +GRID_LIMIT="${GRID_LIMIT:-0}" diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/generate_scenarios.py b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/generate_scenarios.py new file mode 100644 index 0000000..2bfb152 --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/generate_scenarios.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Generate the scenario list for the TP×DP matrix experiment. + +Reads matrix.json and prints TSV lines: + + mark input_len output_len concurrency num_prompts + +mark is one of Y/P/N. The caller (run_bench.sh) decides how to treat each. +""" +import argparse +import json +import math +import os +from pathlib import Path + + +def sample_concurrency(low: int, high: int, target: int) -> list[int]: + """Return only the low and high concurrency values in [low, high]. + + For the TP×DP matrix we only need the two endpoints of the concurrency + range (e.g. 1 and 128 for ISL=1024). The `target` argument is kept for + API compatibility but is ignored. + """ + assert 1 <= low <= high, f"invalid concurrency range: {low}-{high}" + if low == high: + return [low] + return [low, high] + + +def generate_scenarios(matrix_path: Path, mode: str, target_samples: int) -> list[dict]: + with open(matrix_path, "r", encoding="utf-8") as f: + data = json.load(f) + + matrix = data["matrix"] + concurrency_cfg = data["concurrency"] + + scenarios = [] + for isl_str in sorted(matrix.keys(), key=int): + osl_map = matrix[isl_str] + low = concurrency_cfg[isl_str]["low"] + high = concurrency_cfg[isl_str]["high"] + concurrencies = sample_concurrency(low, high, target_samples) + + for osl_str in sorted(osl_map.keys(), key=int): + mark = osl_map[osl_str] + if mode == "Y" and mark != "Y": + continue + if mode == "Y+P" and mark not in ("Y", "P"): + continue + # mode == "all" keeps everything, including N. + + for conc in concurrencies: + scenarios.append( + { + "mark": mark, + "input_len": int(isl_str), + "output_len": int(osl_str), + "concurrency": conc, + "num_prompts": conc * 5, + } + ) + + return scenarios + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--matrix", type=Path, default=Path("matrix.json")) + parser.add_argument("--mode", choices=["Y", "Y+P", "all"], default=None, + help="Scenario selection mode. Defaults to matrix.mode.") + parser.add_argument("--target-samples", type=int, default=0, + help="Target number of concurrency samples. 0 = heuristic (6-8).") + args = parser.parse_args() + + with open(args.matrix, "r", encoding="utf-8") as f: + data = json.load(f) + + mode = args.mode if args.mode else data.get("mode", "Y+P") + + target_samples = args.target_samples + if target_samples <= 0: + env_samples = os.getenv("CONCURRENCY_SAMPLES", "0") + try: + target_samples = int(env_samples) + except ValueError: + target_samples = 0 + if target_samples <= 0: + target_samples = 7 + + scenarios = generate_scenarios(args.matrix, mode, target_samples) + + print("mark\tinput_len\toutput_len\tconcurrency\tnum_prompts") + for s in scenarios: + print(f"{s['mark']}\t{s['input_len']}\t{s['output_len']}\t{s['concurrency']}\t{s['num_prompts']}") + + +if __name__ == "__main__": + main() diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/matrix.json b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/matrix.json new file mode 100644 index 0000000..5108ea1 --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/matrix.json @@ -0,0 +1,92 @@ +{ + "comment": "910C / GLM-5.2: 128K context cap. ISL beyond 131072 excluded. Concurrency high-end reduced for 64GB HBM/die (TP-sharded weights + KV cache). 'P' = probe-only (run if time permits), 'N' = skip (OOM expected).", + "mode": "Y", + "matrix": { + "1024": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "Y" + }, + "4096": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "Y" + }, + "8192": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "Y" + }, + "16384": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "P" + }, + "32768": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "Y", + "2048": "Y", + "4096": "P" + }, + "65536": { + "128": "Y", + "256": "Y", + "512": "Y", + "1024": "P", + "2048": "P", + "4096": "N" + }, + "131072": { + "128": "Y", + "256": "P", + "512": "P", + "1024": "N", + "2048": "N", + "4096": "N" + } + }, + "concurrency": { + "1024": { + "low": 1, + "high": 128 + }, + "4096": { + "low": 1, + "high": 64 + }, + "8192": { + "low": 1, + "high": 64 + }, + "16384": { + "low": 1, + "high": 32 + }, + "32768": { + "low": 1, + "high": 16 + }, + "65536": { + "low": 1, + "high": 8 + }, + "131072": { + "low": 1, + "high": 4 + } + } +} diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_adaptive_concurrency.sh b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_adaptive_concurrency.sh new file mode 100755 index 0000000..f76a93d --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_adaptive_concurrency.sh @@ -0,0 +1,164 @@ +#!/usr/bin/env bash +# Find the Total-TPS saturation concurrency for each vLLM TP/DP/ISL/OSL shape. +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")" + +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/lib.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/platform.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/adaptive_config.env" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh" + +ENGINE="vllm" +ENGINE_PORT="$VLLM_PORT" +RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}" +ACTIVE_ENGINE_SERVER_LOG="" + +if [[ -x "${VENV_CLIENT}/bin/python" ]]; then + PYTHON="${VENV_CLIENT}/bin/python" +else + PYTHON="$(command -v python3)" +fi + +DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}" +DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}" + +engine_is_healthy() { + curl --fail --silent --show-error --max-time 5 \ + "http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1 +} + +engine_stop_server() { + local tp="$1" + local dp="$2" + local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid" + + if [[ -f "$pid_file" ]]; then + local pid + pid="$(cat "$pid_file")" + if kill -0 "$pid" 2>/dev/null; then + log "stopping vllm server pid=${pid} tp=${tp} dp=${dp}" + kill "$pid" 2>/dev/null || true + sleep 5 + kill -9 "$pid" 2>/dev/null || true + fi + rm -f "$pid_file" + fi + docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true + ACTIVE_ENGINE_SERVER_LOG="" + sleep 2 +} + +engine_build_server_args() { + local tp="$1" + local dp="$2" + local -a args=( + vllm serve "$MODEL_PATH" + --trust-remote-code + --kv-cache-dtype "$KV_CACHE_DTYPE" + --block-size "$BLOCK_SIZE" + --tensor-parallel-size "$tp" + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" + --max-model-len "$MAX_MODEL_LEN" + --max-num-seqs "$MAX_NUM_SEQS" + --no-enable-flashinfer-autotune + --host 0.0.0.0 + --port "$ENGINE_PORT" + ) + if (( dp > 1 )); then + args+=(--data-parallel-size "$dp") + fi + printf '%q ' "${args[@]}" +} + +engine_start_server() { + local tp="$1" + local dp="$2" + local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" + log "starting vllm server tp=${tp} dp=${dp}" + bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1 + if ! engine_is_healthy; then + log "ERROR: vllm health check failed tp=${tp} dp=${dp}" + return 1 + fi + ACTIVE_ENGINE_SERVER_LOG="$( + find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \ + -name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \ + -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2- + )" + log "vllm server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}" +} + +engine_detect_oom() { + local detail_log="$1" + local tp="$2" + local dp="$3" + local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory' + local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" + local -a logs=("$detail_log" "$outer_log") + if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then + logs+=("$ACTIVE_ENGINE_SERVER_LOG") + fi + grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null +} + +engine_run_bench() { + local isl="$1" + local osl="$2" + local concurrency="$3" + local num_prompts="$4" + local output_file="$5" + local warmup_requests + warmup_requests="$(adaptive_warmup_request_count "$concurrency")" + local -a bench_args=( + --backend vllm + --host 127.0.0.1 + --port "$ENGINE_PORT" + --dataset-name "$BENCH_DATASET_NAME" + --random-input-len "$isl" + --random-output-len "$osl" + --random-range-ratio "$RANDOM_RANGE_RATIO" + --num-prompts "$num_prompts" + --max-concurrency "$concurrency" + --request-rate 10000 + --warmup-requests "$warmup_requests" + --output-file "$output_file" + --output-details + --disable-tqdm + ) + if [[ "$BENCH_DATASET_NAME" == "random" ]]; then + bench_args+=(--dataset-path "$DATASET_PATH") + else + bench_args+=(--tokenize-prompt) + fi + + if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then + local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}") + if [[ "$BENCH_DATASET_NAME" == "random" ]]; then + volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro") + fi + docker run --rm \ + --network host \ + "${volume_args[@]}" \ + -e PYTHONUNBUFFERED=1 \ + -e HF_HUB_OFFLINE=1 \ + -e TRANSFORMERS_OFFLINE=1 \ + "$DOCKER_CLIENT_IMAGE" \ + python -m sglang.bench_serving "${bench_args[@]}" + else + "$PYTHON" -m sglang.bench_serving "${bench_args[@]}" + fi +} + +export -f engine_run_bench +export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT +export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON + +adaptive_main "$@" diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh new file mode 100755 index 0000000..e5d87eb --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh @@ -0,0 +1,190 @@ + +#!/usr/bin/env bash +# Find the Total-TPS saturation concurrency for each vLLM-Ascend TP/DP/ISL/OSL +# shape on Ascend 910C (GLM-5.2). +# +# Differences vs the H20 vLLM variant: +# - GPU monitor overridden to use npu-smi (shared library hardcodes nvidia-smi). +# - engine_run_bench runs the client inside the vllm-ascend container via +# `docker exec`. The official vllm-ascend image does NOT ship +# sglang.bench_serving; see envs/ASCEND_910C_ENV_SETUP.md for the client story +# (install bench_serving into the container or use an external sglang image). +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")" + +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/lib.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/platform.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/adaptive_config.env" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh" + +ENGINE="vllm" +ENGINE_PORT="$VLLM_PORT" +RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}" +ACTIVE_ENGINE_SERVER_LOG="" + +PYTHON="${PYTHON:-$(command -v python3)}" +DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}" + +# Override the shared library's nvidia-smi GPU monitor with an npu-smi sampler +# that emits the same CSV shape (timestamp, index, mem.used, mem.total, util). +adaptive_start_gpu_monitor() { + local csv_path="$1" + mkdir -p "$(dirname "$csv_path")" + # 910C: delegate to the shared npu-smi sampler so parse_backend.py sees the + # same CSV shape as nvidia-smi (see scripts/common/npu_smi_sampler.py). + "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/npu_smi_sampler.py" "$GPU_MEM_SAMPLE_INTERVAL_S" > "$csv_path" 2>/dev/null & + echo $! +} + +engine_is_healthy() { + curl --fail --silent --show-error --max-time 5 \ + "http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1 +} + +engine_stop_server() { + local tp="$1" + local dp="$2" + local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid" + if [[ -f "$pid_file" ]]; then + local pid + pid="$(cat "$pid_file")" + if kill -0 "$pid" 2>/dev/null; then + log "stopping vllm-ascend server pid=${pid} tp=${tp} dp=${dp}" + kill "$pid" 2>/dev/null || true + sleep 5 + kill -9 "$pid" 2>/dev/null || true + fi + rm -f "$pid_file" + fi + # Also remove the container by name in case the host process is gone but the + # container lingers (e.g. orphaned by a kill -9). + docker rm -f "${CONTAINER_NAME}_tp${tp}_dp${dp}" >/dev/null 2>&1 || true + ACTIVE_ENGINE_SERVER_LOG="" + sleep 2 +} + +engine_build_server_args() { + local tp="$1" + local dp="$2" + local -a args=( + vllm serve "$MODEL_PATH" + --served-model-name "$SERVED_MODEL_NAME" + --trust-remote-code + --kv-cache-dtype "$KV_CACHE_DTYPE" + --block-size "$BLOCK_SIZE" + --tensor-parallel-size "$tp" + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" + --max-model-len "$MAX_MODEL_LEN" + --max-num-seqs "$MAX_NUM_SEQS" + --host 0.0.0.0 + --port "$ENGINE_PORT" + ) + if (( dp > 1 )); then + args+=(--data-parallel-size "$dp") + fi + printf '%q ' "${args[@]}" +} + +engine_start_server() { + local tp="$1" + local dp="$2" + local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" + log "starting vllm-ascend server tp=${tp} dp=${dp}" + bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1 + if ! engine_is_healthy; then + log "ERROR: vllm-ascend health check failed tp=${tp} dp=${dp}" + return 1 + fi + ACTIVE_ENGINE_SERVER_LOG="$( + find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \ + -name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \ + -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2- + )" + log "vllm-ascend server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}" +} + +engine_detect_oom() { + local detail_log="$1" + local tp="$2" + local dp="$3" + local pattern='out of memory|OutOfMemory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory|NPU out of memory|acl.*memory|HBM' + local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log" + local -a logs=("$detail_log" "$outer_log") + if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then + logs+=("$ACTIVE_ENGINE_SERVER_LOG") + fi + grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null +} + +engine_run_bench() { + local isl="$1" + local osl="$2" + local concurrency="$3" + local num_prompts="$4" + local output_file="$5" + local container_name="${CONTAINER_NAME}_tp${tp}_dp${dp}" + local container_output="/tmp/bench_outputs/adaptive_$(basename "$output_file")" + if [[ "$output_file" == "/dev/null" ]]; then + container_output="/dev/null" + fi + + local warmup_requests + warmup_requests="$(adaptive_warmup_request_count "$concurrency")" + + local -a bench_args=( + --backend vllm + --host 127.0.0.1 + --port "$ENGINE_PORT" + --model "$SERVED_MODEL_NAME" + --dataset-name "$BENCH_DATASET_NAME" + --random-input-len "$isl" + --random-output-len "$osl" + --random-range-ratio "$RANDOM_RANGE_RATIO" + --num-prompts "$num_prompts" + --max-concurrency "$concurrency" + --request-rate 10000 + --warmup-requests "$warmup_requests" + --output-file "$container_output" + --output-details + --disable-tqdm + ) + if [[ "$BENCH_DATASET_NAME" == "random" ]]; then + bench_args+=(--dataset-path "$DATASET_PATH") + else + bench_args+=(--tokenize-prompt) + fi + + # Run the benchmark client inside the running vllm-ascend container via + # `docker exec`. NOTE: the official vllm-ascend image does NOT include + # sglang.bench_serving. If `docker exec ... sglang.bench_serving` fails with + # ModuleNotFoundError, either (a) pip install sglang into the container, or + # (b) set USE_DOCKER_CLIENT=1 and run an external sglang image against the + # host port. See envs/ASCEND_910C_ENV_SETUP.md. + docker exec "$container_name" mkdir -p /tmp/bench_outputs 2>/dev/null || true + docker exec "$container_name" \ + env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \ + "$CONTAINER_PYTHON" -m sglang.bench_serving "${bench_args[@]}" || return $? + + if [[ "$output_file" != "/dev/null" ]]; then + docker cp "$container_name:${container_output}" "$output_file" || return 1 + fi +} + +export -f engine_run_bench +export ENGINE_PORT CONTAINER_NAME CONTAINER_PYTHON MODEL_PATH RESULT_BASE SERVED_MODEL_NAME +export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON + +export SEARCH_START_CONCURRENCY=16 +export SEARCH_ADDEND=16 +export SEARCH_INITIAL_BACKOFF_CONCURRENCIES="8 1" +export TTFT_GROUP_SKIP_MS="${TTFT_GROUP_SKIP_MS:-8000}" + +adaptive_main "$@" diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_bench.sh b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_bench.sh new file mode 100755 index 0000000..a4030b5 --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_bench.sh @@ -0,0 +1,546 @@ +#!/usr/bin/env bash +# TP×DP matrix benchmark for DeepSeek-V4-Flash on vLLM (Docker). +set -Eeuo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")" + +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/lib.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/../../../scripts/common/platform.sh" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +# Export variables used inside functions that are called via bash -c subshells. +export DATASET_PATH MODEL_PATH RESULT_BASE \ + DOCKER_IMAGE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT + +RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}" +RESULT_BASE="${SCRIPT_DIR}/results" +MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR}/matrix.json}" +MATRIX_MODE="${MATRIX_MODE:-Y}" +SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}" +GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}" +DRY_RUN="${DRY_RUN:-0}" +GRID_LIMIT="${GRID_LIMIT:-0}" + +if [[ -x "${VENV_CLIENT}/bin/python" ]]; then + PYTHON="${VENV_CLIENT}/bin/python" +else + PYTHON="$(command -v python3)" +fi + +DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}" +DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}" + +log_dir_global="${RESULT_BASE}/${RUN_ID}/logs" +mkdir -p "$log_dir_global" +log_init "${log_dir_global}/orchestrator.log" + +log "experiment=${EXPERIMENT_NAME} run_id=${RUN_ID} platform=${PLATFORM} hardware=${HARDWARE}" +log "matrix_mode=${MATRIX_MODE} matrix_file=${MATRIX_FILE} dry_run=${DRY_RUN} grid_limit=${GRID_LIMIT}" + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +is_server_healthy() { + curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${VLLM_PORT}/health" >/dev/null 2>&1 +} + +stop_server() { + local tp="$1" + local dp="$2" + + local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid" + if [[ -f "$pid_file" ]]; then + local pid + pid="$(cat "$pid_file")" + if kill -0 "$pid" 2>/dev/null; then + log "stopping vllm server pid=${pid} (tp=${tp}, dp=${dp})" + kill "$pid" 2>/dev/null || true + sleep 5 + kill -9 "$pid" 2>/dev/null || true + fi + rm -f "$pid_file" + fi + + # Fallback: remove any Docker container started by this experiment. + docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true + + # Fallback: kill any vllm serve processes for this model. + pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true + pkill -9 -f "vllm serve.*${MODEL_PATH}" 2>/dev/null || true + sleep 2 +} + +build_server_args() { + local tp="$1" + local dp="$2" + + local args=( + "vllm serve" "$MODEL_PATH" + --trust-remote-code + --kv-cache-dtype "$KV_CACHE_DTYPE" + --block-size "$BLOCK_SIZE" + --tensor-parallel-size "$tp" + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" + --max-model-len "$MAX_MODEL_LEN" + --max-num-seqs "$MAX_NUM_SEQS" + --no-enable-flashinfer-autotune + --host 0.0.0.0 + --port "$VLLM_PORT" + ) + if [[ "$dp" -gt 1 ]]; then + args+=( + --data-parallel-size "$dp" + ) + fi + printf '%s ' "${args[@]}" +} + +start_server() { + local tp="$1" + local dp="$2" + + log "starting vllm server tp=${tp} dp=${dp}" + bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" \ + >> "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1 + + if ! is_server_healthy; then + log "error: vllm server tp=${tp} dp=${dp} failed health check on port ${VLLM_PORT}" + return 1 + fi + log "vllm server tp=${tp} dp=${dp} is healthy on port ${VLLM_PORT}" +} + +restart_server() { + local tp="$1" + local dp="$2" + log "restarting vllm server tp=${tp} dp=${dp} after non-OOM failure" + stop_server "$tp" "$dp" + sleep 10 + start_server "$tp" "$dp" +} + +run_bench_serving() { + # Run sglang.bench_serving either natively or inside the SGLang Docker image. + if [[ "${USE_DOCKER_CLIENT:-1}" == "1" ]]; then + local vol_args=() + vol_args+=("-v" "${MODEL_PATH}:${MODEL_PATH}:ro") + if [[ -n "${DATASET_PATH:-}" && -f "${DATASET_PATH}" ]]; then + vol_args+=("-v" "${DATASET_PATH}:${DATASET_PATH}:ro") + fi + vol_args+=("-v" "${RESULT_BASE}:${RESULT_BASE}") + docker run --rm \ + --network host \ + "${vol_args[@]}" \ + -e PYTHONUNBUFFERED=1 \ + -e HF_HUB_OFFLINE=1 \ + -e TRANSFORMERS_OFFLINE=1 \ + -e HF_DATASETS_OFFLINE=1 \ + "${DOCKER_CLIENT_IMAGE}" \ + python -m sglang.bench_serving "$@" + else + "$PYTHON" -m sglang.bench_serving "$@" + fi +} +export -f run_bench_serving + +run_warmup() { + local input_len="$1" + local output_len="$2" + + log "warming up (input=${input_len}, output=${output_len}, num=1)" + bash -c ' + run_bench_serving \ + --backend vllm \ + --host 127.0.0.1 \ + --port "'"$VLLM_PORT"'" \ + --dataset-name random \ + --dataset-path "'"$DATASET_PATH"'" \ + --random-input-len "'"$input_len"'" \ + --random-output-len "'"$output_len"'" \ + --num-prompts 1 \ + --max-concurrency 1 \ + --request-rate 10000 \ + --output-file /dev/null \ + --output-details \ + >> "'"${log_dir_global}/warmup.log"'" 2>&1 + ' + log "warmup completed" +} + +scenario_already_completed() { + local output_file="$1" + local expected="$2" + [[ -s "$output_file" ]] || return 1 + local completed + completed="$("$PYTHON" -c " +import json, sys +path = sys.argv[1] +try: + with open(path, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if line: + data = json.loads(line) + print(data.get('completed', 0)) + break +except Exception: + print(0) +" "$output_file")" + [[ "${completed:-0}" -ge "$expected" ]] +} + +scenario_already_processed() { + local result_root="$1" + local scenario_name="$2" + local json_path="${result_root}/results.json" + [[ -f "$json_path" ]] || return 1 + "$PYTHON" -c " +import json, sys +path, name = sys.argv[1], sys.argv[2] +try: + with open(path, 'r', encoding='utf-8') as f: + data = json.load(f) + for s in data.get('scenarios', []): + if s.get('name') == name: + if s.get('status') or s.get('metrics', {}).get('success', 0) > 0: + sys.exit(0) +except Exception: + pass +sys.exit(1) +" "$json_path" "$scenario_name" +} + +detect_oom() { + local detail_log="$1" + local server_outer_log="$2" + local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory|NPU out of memory|acl.*memory|HBM' + if grep -Eiq "$pattern" "$detail_log" "$server_outer_log" 2>/dev/null; then + return 0 + fi + return 1 +} + +start_gpu_monitor() { + local csv_path="$1" + mkdir -p "$(dirname "$csv_path")" + # 910C: use npu-smi via a standalone python sampler that emits the same CSV + # shape as nvidia-smi, so parse_backend.py needs no changes. + "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/npu_smi_sampler.py" "$GPU_MEM_SAMPLE_INTERVAL_S" > "$csv_path" 2>/dev/null & + echo $! +} + +stop_gpu_monitor() { + local pid="$1" + if kill -0 "$pid" 2>/dev/null; then + kill "$pid" 2>/dev/null || true + sleep 1 + kill -9 "$pid" 2>/dev/null || true + fi +} + +append_scenario_record() { + local result_root="$1" + local json_path="$result_root/results.json" + shift + local scenario_json + scenario_json="$("$PYTHON" -c " +import json, sys +pairs = [a.split('=', 1) for a in sys.argv[1:]] +d = {} +for k, v in pairs: + try: + d[k] = json.loads(v) + except json.JSONDecodeError: + d[k] = v +print(json.dumps(d, ensure_ascii=False)) +" "$@")" + PYTHON="$PYTHON" append_scenario_to_json "$json_path" "$scenario_json" +} + +record_skipped_csv() { + local csv_path="$1" + shift + # Args: key=value + local row + row="$("$PYTHON" -c " +import csv, json, sys, io +pairs = [a.split('=', 1) for a in sys.argv[1:]] +d = {} +for k, v in pairs: + try: + d[k] = json.loads(v) + except json.JSONDecodeError: + d[k] = v +buf = io.StringIO() +writer = csv.DictWriter(buf, fieldnames=['engine','tp','dp','mark','isl','osl','concurrency','status','reason','detail_log'], extrasaction='ignore') +writer.writerow(d) +print(buf.getvalue().strip()) +" "$@")" + echo "$row" >> "$csv_path" +} + +skip_remaining_scenarios() { + local result_root="$1" + local scenario_tsv="$2" + local start_index="$3" + local status="$4" + local reason="$5" + local tp="$6" + local dp="$7" + local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv" + + local i=0 + tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl osl conc num; do + if (( i < start_index )); then + i=$((i + 1)) + continue + fi + i=$((i + 1)) + local sname="c${conc}_i${isl}_o${osl}" + if scenario_already_processed "$result_root" "$sname"; then + continue + fi + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"${status}\"" \ + "note=\"${reason}\"" + record_skipped_csv "$skipped_csv" \ + "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=${status}" "reason=${reason}" + done +} + +# --------------------------------------------------------------------------- +# Per-configuration runner +# --------------------------------------------------------------------------- + +run_parallel_config() { + local tp="$1" + local dp="$2" + local config_label="tp${tp}_dp${dp}" + local result_root="${RESULT_BASE}/${RUN_ID}/${config_label}" + local raw_dir="${result_root}/raw_outputs" + local gpu_log_dir="${result_root}/gpu_logs" + local phase_log_dir="${result_root}/logs" + mkdir -p "$raw_dir" "$gpu_log_dir" "$phase_log_dir" + + log "===== ${config_label} START =====" + + # Generate scenario list for this config. + local scenario_tsv="${result_root}/scenarios.tsv" + "$PYTHON" "${SCRIPT_DIR}/generate_scenarios.py" \ + --matrix "$MATRIX_FILE" \ + --mode "$MATRIX_MODE" \ + > "$scenario_tsv" + local total_scenarios + total_scenarios="$(tail -n +2 "$scenario_tsv" | wc -l)" + log "generated ${total_scenarios} scenarios for ${config_label}" + + # Write metadata. + ensure_result_root "$result_root" + write_metadata_json \ + "${result_root}/results.json" \ + "${EXPERIMENT_NAME}_${config_label}" \ + "$RUN_ID" \ + "$MODEL_PATH" \ + "vllm" \ + "vllm" \ + "$HARDWARE" \ + "$ACCELERATOR" \ + "$CHIP" \ + "experiments/${EXPERIMENT_NAME}/run_bench.sh" \ + "$DOCKER_IMAGE" \ + "H20 vLLM TP×DP matrix for DeepSeek-V4-Flash" + + local server_args_str + server_args_str="$(build_server_args "$tp" "$dp")" + jq --arg tp "$tp" --arg dp "$dp" --arg cuda "$CUDA_VISIBLE_DEVICES" --arg args "$server_args_str" \ + '.config = { + "tp": ($tp | tonumber), + "dp": ($dp | tonumber), + "cuda_visible_devices": $cuda, + "backend": "vllm", + "server_start_script": "experiments/'${EXPERIMENT_NAME}'/start_vllm_dp.sh", + "server_args": $args + }' "${result_root}/results.json" > "${result_root}/results.json.tmp" && \ + mv "${result_root}/results.json.tmp" "${result_root}/results.json" + + if [[ "$DRY_RUN" == "1" ]]; then + log "DRY_RUN: would start server with args: ${server_args_str}" + local line + tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl osl conc num; do + log "DRY_RUN: ${config_label} scenario mark=${mark} c=${conc} i=${isl} o=${osl} n=${num}" + done + log "===== ${config_label} DONE (dry run) =====" + return 0 + fi + + # Initialize skipped_after_oom.csv for this run. + local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv" + if [[ ! -f "$skipped_csv" ]]; then + echo "engine,tp,dp,mark,isl,osl,concurrency,status,reason,detail_log" > "$skipped_csv" + fi + + # Start server once for this TP×DP config. + if ! start_server "$tp" "$dp"; then + log "ERROR: ${config_label} failed to start; skipping all scenarios" + skip_remaining_scenarios "$result_root" "$scenario_tsv" 0 "SKIPPED_SERVICE_START_FAILED" "service failed to start" "$tp" "$dp" + log "===== ${config_label} DONE =====" + return 0 + fi + + # Warmup with a small prompt before the first scenario. + run_warmup 1024 128 || true + + # Read scenarios into an array so we can skip remaining entries on failure. + local -a scenarios=() + while IFS= read -r line; do + scenarios+=("$line") + done < <(tail -n +2 "$scenario_tsv") + + local i mark isl osl conc num + local output_file detail_log gpu_csv sname bench_rc + for (( i = 0; i < ${#scenarios[@]}; i++ )); do + IFS=$'\t' read -r mark isl osl conc num <<< "${scenarios[$i]}" + + if [[ "$GRID_LIMIT" -gt 0 && "$i" -ge "$GRID_LIMIT" ]]; then + log "GRID_LIMIT=${GRID_LIMIT} reached; skipping remaining scenarios" + skip_remaining_scenarios "$result_root" "$scenario_tsv" "$i" "SKIPPED_GRID_LIMIT" "GRID_LIMIT reached" "$tp" "$dp" + break + fi + + sname="c${conc}_i${isl}_o${osl}" + output_file="${raw_dir}/vllm_main_${conc}_${isl}_${osl}.jsonl" + detail_log="${phase_log_dir}/vllm_${config_label}_${sname}.log" + gpu_csv="${gpu_log_dir}/gpu_mem_${conc}_${isl}_${osl}.csv" + + if scenario_already_completed "$output_file" "$num" || scenario_already_processed "$result_root" "$sname"; then + log "skipping already-processed ${config_label} scenario: ${sname}" + continue + fi + + log "running ${config_label} scenario: mark=${mark} c=${conc} i=${isl} o=${osl} n=${num}" + + local gpu_pid + gpu_pid="$(start_gpu_monitor "$gpu_csv")" + + bench_rc=0 + timeout "$SCENARIO_TIMEOUT_S" bash -c ' + run_bench_serving \ + --backend vllm \ + --host 127.0.0.1 \ + --port "'"$VLLM_PORT"'" \ + --dataset-name random \ + --dataset-path "'"$DATASET_PATH"'" \ + --random-input-len "'"$isl"'" \ + --random-output-len "'"$osl"'" \ + --num-prompts "'"$num"'" \ + --max-concurrency "'"$conc"'" \ + --request-rate 10000 \ + --output-file "'"$output_file"'" \ + --output-details \ + > "'"$detail_log"'" 2>&1 + ' || bench_rc=$? + + stop_gpu_monitor "$gpu_pid" + + if [[ "$bench_rc" -eq 0 ]]; then + log "finished ${config_label} scenario: output=${output_file}" + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"completed\"" \ + "note=\"benchmark finished successfully\"" + continue + fi + + # Failure handling. + if detect_oom "$detail_log" "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log"; then + log "ERROR: ${config_label} scenario ${sname} triggered OOM; stopping config" + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"OOM\"" \ + "note=\"detected CUDA out-of-memory\"" + record_skipped_csv "$skipped_csv" \ + "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=OOM" "reason=detected CUDA out-of-memory" "detail_log=${detail_log}" + stop_server "$tp" "$dp" + skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_AFTER_OOM" "previous case OOM" "$tp" "$dp" + break + fi + + log "ERROR: ${config_label} scenario ${sname} failed (rc=${bench_rc}); see ${detail_log}" + if [[ "$mark" == "P" ]]; then + log "optional (P) scenario failed; recording as skipped and continuing" + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"skipped_optional\"" \ + "note=\"optional scenario failed (rc=${bench_rc})\"" + record_skipped_csv "$skipped_csv" \ + "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=skipped_optional" "reason=optional scenario failed (rc=${bench_rc})" "detail_log=${detail_log}" + continue + fi + + # Mandatory scenario failed but not OOM: try to restart the server. + if restart_server "$tp" "$dp"; then + run_warmup 1024 128 || true + log "resuming ${config_label} after server restart" + continue + fi + + log "ERROR: ${config_label} server restart failed; skipping remaining scenarios" + append_scenario_record "$result_root" \ + "name=${sname}" \ + "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \ + "status=\"FAILED\"" \ + "note=\"scenario failed and server restart failed (rc=${bench_rc})\"" + record_skipped_csv "$skipped_csv" \ + "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=FAILED" "reason=scenario failed and server restart failed" "detail_log=${detail_log}" + skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_RESTART_FAILED" "server restart failed" "$tp" "$dp" + break + done + + stop_server "$tp" "$dp" + + # Parse results. + log "parsing ${config_label} results" + "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/parse_backend.py" "$result_root" --backend vllm \ + >> "${phase_log_dir}/parse.log" 2>&1 || { + log "WARNING: parser failed for ${config_label}; see ${phase_log_dir}/parse.log" + } + + log "===== ${config_label} DONE =====" +} + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +# Cleanup any leftovers. +for cfg in "${PARALLEL_CONFIGS[@]}"; do + read -r tp dp <<< "$cfg" + stop_server "$tp" "$dp" +done + +# Run each parallel configuration. +for cfg in "${PARALLEL_CONFIGS[@]}"; do + read -r tp dp <<< "$cfg" + run_parallel_config "$tp" "$dp" +done + +# Generate cross-configuration comparison. +log "generating comparison report" +"$PYTHON" "${SCRIPT_DIR}/compare.py" \ + --run-root "${RESULT_BASE}/${RUN_ID}" \ + --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \ + >> "${log_dir_global}/compare.log" 2>&1 || { + log "WARNING: comparison script failed; see ${log_dir_global}/compare.log" + } + +log "all results saved to ${RESULT_BASE}/${RUN_ID}" diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/start_vllm_docker.sh b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/start_vllm_docker.sh new file mode 100755 index 0000000..2553617 --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/start_vllm_docker.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# Start vLLM-Ascend server in Docker for a given TP×DP configuration on 910C. +# Usage: start_vllm_docker.sh +# +# Relies on the Ascend Docker Runtime being the default docker runtime on this +# host (see /etc/docker/daemon.json). NPU dies are injected via the +# ASCEND_VISIBLE_DEVICES env var; no --gpus / --device flags are needed. +set -e + +TP="${1}" +DP="${2}" + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}" +mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp" + +IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}" +PORT="${VLLM_PORT:-30050}" +NAME="${CONTAINER_NAME:-${EXPERIMENT}_vllm_tp${TP}_dp${DP}}" +NAME="${NAME}_tp${TP}_dp${DP}" +PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${TP}_dp${DP}.pid" + +LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_vllm_docker_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log" +rm -f "$PID_FILE" + +# Clean up any stale container with the same name. +docker rm -f "$NAME" >/dev/null 2>&1 || true + +# vLLM-Ascend launch args. Differences vs NVIDIA vLLM: +# - no --no-enable-flashinfer-autotune (Ascend uses its own attention path) +# - --kv-cache-dtype may need to be fp16 if the image rejects fp8 on 910C +SERVER_ARGS=( + "$MODEL_PATH" + --served-model-name "$SERVED_MODEL_NAME" + --trust-remote-code + --kv-cache-dtype "$KV_CACHE_DTYPE" + --block-size "$BLOCK_SIZE" + --tensor-parallel-size "$TP" + --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" + --max-model-len "$MAX_MODEL_LEN" + --max-num-seqs "$MAX_NUM_SEQS" + --host 0.0.0.0 + --port "$PORT" +) + +if [[ "$DP" -gt 1 ]]; then + SERVER_ARGS+=(--data-parallel-size "$DP") +fi + +SERVER_ARGS_STR="vllm serve ${SERVER_ARGS[*]}" + +echo "=== Starting vLLM-Ascend server in Docker (TP=${TP}, DP=${DP}) ===" +echo "Image: $IMAGE" +echo "Model: $MODEL_PATH" +echo "Container name: $NAME" +echo "Host port: $PORT" +echo "ASCEND_VISIBLE_DEVICES: ${ASCEND_VISIBLE_DEVICES}" +echo "Command: $SERVER_ARGS_STR" +echo "Log: $LOG" + +# Run docker in the foreground; nohup backgrounds it and the host PID lets the +# adaptive search stop the container by killing the process (the container has +# --rm so it self-cleans). The Ascend Docker Runtime is the default runtime, so +# no --runtime flag is required. +nohup docker run --rm \ + --name "$NAME" \ + --ipc host \ + --shm-size 16g \ + --network host \ + --ulimit memlock=-1 \ + -v "${MODEL_PATH}:${MODEL_PATH}:ro" \ + -v "${RUNTIME_BASE}/tmp:/tmp" \ + -e ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES}" \ + -e PYTORCH_NPU_ALLOC_CONF=expandable_segments:True \ + -e PYTHONUNBUFFERED=1 \ + -e HF_HUB_OFFLINE=1 \ + -e TRANSFORMERS_OFFLINE=1 \ + "$IMAGE" \ + vllm serve "${SERVER_ARGS[@]}" \ + > "$LOG" 2>&1 & + +PID=$! +echo $PID > "$PID_FILE" +echo "PID: $PID" +echo "Waiting for health on port ${PORT}..." + +for i in $(seq 1 240); do + if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then + echo "vLLM-Ascend server is ready at http://127.0.0.1:${PORT}" + echo "Log: $LOG" + exit 0 + fi + if ! kill -0 $PID 2>/dev/null; then + echo "ERROR: Docker vLLM-Ascend server exited early" + tail -200 "$LOG" + exit 1 + fi + echo "Waiting... ($i/240)" + sleep 5 +done + +echo "ERROR: vLLM-Ascend server not healthy after 240 retries" +tail -200 "$LOG" +exit 1 diff --git a/experiments/910c/glm52_910c_vllm_tp_dp_matrix/start_vllm_dp.sh b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/start_vllm_dp.sh new file mode 100755 index 0000000..1f33f8f --- /dev/null +++ b/experiments/910c/glm52_910c_vllm_tp_dp_matrix/start_vllm_dp.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# Start vLLM-Ascend server for a given TP×DP configuration on 910C. +# Usage: start_vllm_dp.sh +# +# 910C only supports the Docker path (vllm-ascend runs inside a container with +# the Ascend Docker Runtime). USE_DOCKER=0 is not supported on this platform; +# set USE_DOCKER=1 (the default) or call start_vllm_docker.sh directly. +set -e + +TP="${1}" +DP="${2}" + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=/dev/null +source "${SCRIPT_DIR}/config.env" + +if [[ "${USE_DOCKER:-1}" == "1" ]]; then + exec "${SCRIPT_DIR}/start_vllm_docker.sh" "$@" +fi + +echo "ERROR: USE_DOCKER=0 (native vllm-ascend) is not supported on 910C." >&2 +echo " vllm-ascend requires the Ascend Docker Runtime; set USE_DOCKER=1." >&2 +exit 1 diff --git a/platforms/README.md b/platforms/README.md index f38ec98..a57aa85 100644 --- a/platforms/README.md +++ b/platforms/README.md @@ -1,3 +1,4 @@ + # Platform Configurations Each `.env` file in this directory describes one accelerator platform. @@ -18,6 +19,7 @@ PLATFORM=kunlun_p800 bash experiments/dsv4_p800_sglang/run_bench.sh | File | Chip/Accelerator | Engine | Notes | |---|---|---|---| +| `ascend_910c.env` | Huawei Ascend 910C | `vllm-ascend` | Docker-based，8 卡 16 die / 64GB HBM；Ascend Docker Runtime 为默认 runtime，见 `envs/ASCEND_910C_ENV_SETUP.md` | | `kunlun_p800.env` | Kunlun P800 XPU | `sglang-xpu` | Docker-based SGLang serving image | | `nvidia_h200.env` | NVIDIA H200 | `vllm-dspark` | Native host virtual environments | | `nvidia_h20.env` | NVIDIA H20 | `vllm` / `sglang` | Docker-based（vllm-openai / sglang 官方镜像） | diff --git a/platforms/ascend_910c.env b/platforms/ascend_910c.env new file mode 100644 index 0000000..246b1e2 --- /dev/null +++ b/platforms/ascend_910c.env @@ -0,0 +1,42 @@ +# Platform configuration for Huawei Ascend 910C NPU +# Source this file via scripts/common/platform.sh +# +# Host reference (910c.1 / NPU-NODE61): +# - openEuler 22.03 SP4, aarch64, 8x Ascend910 (16 dies, 64GB HBM/die) +# - CANN 9.0.0 + ascend-toolkit, driver 25.5.2 +# - Ascend Docker Runtime is the default docker runtime (see /etc/docker/daemon.json) +# - User must be in HwHiAiUser + docker groups to access NPU devices and docker. + +CHIP="ascend_910c" +ACCELERATOR="Huawei Ascend 910C" +HARDWARE="8x Ascend910 (16 dies, 64GB HBM/die)" +ENGINE="vllm-ascend" + +# Container runtime. vllm-ascend images ship as tarballs on this host and must +# be `docker load`-ed first (see envs/ASCEND_910C_ENV_SETUP.md). The exact tag +# is overridden per-experiment in config.env; this is the default for new ones. +# Known-good tarballs on /mnt/models: +# vllm-ascend-v0.23.0rc1-a3-openeuler.tar -> general vllm-ascend +# vllm-ascend-glm5.2-a3-openeuler.tar -> GLM5.2-tuned variant +DOCKER_IMAGE="${DOCKER_IMAGE:-quay.io/ascend/vllm-ascend:latest}" +CONTAINER_NAME="${CONTAINER_NAME:-vllm-ascend-910c}" + +# Device selection. Ascend Docker Runtime maps NPU dies via ASCEND_VISIBLE_DEVICES +# (0-based NPU card id). 8 cards => 0..7. Each card exposes 2 dies; vllm-ascend +# addresses dies individually so TP can go up to 16. +DEVICE_SELECT_ENV="ASCEND_VISIBLE_DEVICES=0,1,2,3,4,5,6,7" +ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}" + +# Default serving port and model root on the host. +# /mnt/models holds GLM-5.2-{w4a8c8,w8a8} and vllm-ascend image tarballs. +DEFAULT_PORT="30050" +MODEL_ROOT="/mnt/models" + +# Python interpreter inside the vllm-ascend container (used to run the +# benchmark client via `docker exec`). Standard path in the official image. +CONTAINER_PYTHON="${CONTAINER_PYTHON:-/usr/local/bin/python}" + +# CANN / toolkit paths on the host (informational; the container bundles its +# own CANN). Referenced by env setup docs, not by run scripts directly. +HOST_CANN_ROOT="/usr/local/Ascend/ascend-toolkit/latest" +HOST_DRIVER_VERSION="25.5.2" diff --git a/scripts/common/npu_smi_sampler.py b/scripts/common/npu_smi_sampler.py new file mode 100755 index 0000000..045b463 --- /dev/null +++ b/scripts/common/npu_smi_sampler.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +"""Sample npu-smi info and emit nvidia-smi-compatible CSV rows. + +Usage: npu_smi_sampler.py +Emits a header row first, then one row per chip every interval seconds. +Output columns match nvidia-smi --query-gpu=timestamp,index,memory.used,memory.total,utilization.gpu --format=csv +""" +import sys, re, datetime, time + +def sample(): + import subprocess + try: + out = subprocess.run(["npu-smi", "info"], capture_output=True, text=True, timeout=10).stdout + except Exception: + return [] + ts = datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S") + rows = [] + idx = None + for line in out.splitlines(): + m = re.match(r"\|\s*(\d+)\s+Ascend", line) + if m: + idx = m.group(1) + continue + if idx is None: + continue + hbm = re.search(r"(\d+)\s*/\s*(\d+)", line) + util = re.search(r"(\d+)\s*%", line) + if hbm: + u, t = hbm.group(1), hbm.group(2) + r = util.group(1) if util else "0" + rows.append(f"{ts}, {idx}, {u} MiB, {t} MiB, {r} %") + idx = None + return rows + +def main(): + interval = float(sys.argv[1]) if len(sys.argv) > 1 else 1.0 + print("timestamp, index, memory.used [MiB], memory.total [MiB], utilization.gpu [%]") + sys.stdout.flush() + while True: + for row in sample(): + print(row) + sys.stdout.flush() + time.sleep(interval) + +if __name__ == "__main__": + main() diff --git a/scripts/common/platform.sh b/scripts/common/platform.sh index 77c4f4f..23e4f31 100755 --- a/scripts/common/platform.sh +++ b/scripts/common/platform.sh @@ -1,3 +1,4 @@ + #!/usr/bin/env bash # Platform loader. # Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/platform.sh" @@ -19,6 +20,18 @@ if [[ -z "${PLATFORM:-}" ]]; then PLATFORM="kunlun_p800" elif lspci 2>/dev/null | grep -qiE "XPU|Kunlun"; then PLATFORM="kunlun_p800" + elif command -v npu-smi >/dev/null 2>&1; then + # Huawei Ascend NPU. npu-smi info lists cards like "Ascend910" / "910C". + _NPU_SMI_OUT=$(npu-smi info 2>/dev/null || true) + if [[ -n "$_NPU_SMI_OUT" ]] && grep -qiE "Ascend910|Ascend 910|910C|910c" <<< "$_NPU_SMI_OUT"; then + PLATFORM="ascend_910c" + else + PLATFORM="ascend_910c" + fi + unset _NPU_SMI_OUT + elif lspci 2>/dev/null | grep -qiE "Huawei.*a12d|Huawei.*a12e|d100|d80"; then + # Ascend 910 series exposes Huawei System-peripheral PCI IDs (a12d/a12e). + PLATFORM="ascend_910c" elif command -v nvidia-smi >/dev/null 2>&1; then _GPU_NAME=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -n 1 || true) if [[ -n "$_GPU_NAME" ]] && grep -qiE "RTX 6000D|PRO 6000D" <<< "$_GPU_NAME"; then