feat(platform): add Ascend 910C NPU platform support

- platforms/ascend_910c.env: 8-card 910C config (16 dies, 64GB HBM/die),
  Ascend Docker Runtime, ASCEND_VISIBLE_DEVICES device selection
- scripts/common/platform.sh: auto-detect 910C via npu-smi + Huawei PCI IDs
- scripts/common/npu_smi_sampler.py: standalone npu-smi -> nvidia-smi CSV
  sampler so parse_backend.py needs no changes
- experiments/910c/glm52_910c_vllm_tp_dp_matrix/: GLM-5.2 (w4a8c8) experiment,
  model present on host, ready for smoke after image load
- experiments/910c/dsv4_910c_vllm_tp_dp_matrix/: DSV4-Flash experiment
  (placeholder MODEL_PATH, weights not yet downloaded)
- envs/ASCEND_910C_ENV_SETUP.md: full onboarding guide (permissions, image
  load, Ascend Docker Runtime, NPU monitor, known pitfalls)
- Both experiments: TP2/DP4 + TP4/DP2 + TP8/DP1, matrix.json capped at 128K
  context per 64GB HBM/die

.gitignore

@@ -83,6 +83,7 @@ experiments/**/raw_outputs/
 envs/*
 !envs/README.md
 !envs/SM120_DSV4_DEPLOYMENT_GUIDE.md
+!envs/ASCEND_910C_ENV_SETUP.md
 !envs/UV_ENV_SETUP.md
 
 # Tooling artifacts

envs/ASCEND_910C_ENV_SETUP.md

@@ -0,0 +1,158 @@
+# Ascend 910C 环境搭建与部署指南
+
+本文档说明如何在 Ascend 910C NPU 节点上搭建 vLLM-Ascend 推理环境并跑起 sskj 基准测试。
+
+## 1. 环境信息（参考机型：910c.1 / NPU-NODE61）
+
+| 项目 | 值 |
+|---|---|
+| OS | openEuler 22.03 LTS SP4 (aarch64) |
+| 内核 | 5.10.0-216.0.0.115.oe2203sp4.aarch64 |
+| NPU | 8 × Ascend910（每卡 2 die，共 16 die） |
+| HBM | 64 GB/die，合计 ~1 TB |
+| 驱动 | 25.5.2（Innerversion V100R001C23SPC007B221） |
+| CANN | 9.0.0 + ascend-toolkit |
+| Docker | 26.1.3，默认 runtime = ascend（见 `/etc/docker/daemon.json`） |
+| Python (host) | 3.9.9（仅用于编排脚本，推理在容器内） |
+
+## 2. 权限准备 🧑
+
+新用户默认无法访问 NPU 设备节点和 Docker，需要管理员加入两个组：
+
+```bash
+# 加入 HwHiAiUser 组才能访问 /dev/davinci* 设备节点
+sudo usermod -aG HwHiAiUser <user>
+# 加入 docker 组才能调用 docker（或每次 sudo docker）
+sudo usermod -aG docker <user>
+# 重新登录生效
+exit  # 然后重新 ssh
+```
+
+验证：
+
+```bash
+id                        # 应看到 HwHiAiUser 和 docker 组
+npu-smi info              # 应输出 8 卡 16 die 的状态表
+docker ps                 # 不应报 permission denied
+```
+
+## 3. 加载 vLLM-Ascend 镜像
+
+910C 节点离线，vLLM-Ascend 镜像以 tarball 形式存放在 `/mnt/models/`：
+
+| tarball | 用途 |
+|---|---|
+| `vllm-ascend-v0.23.0rc1-a3-openeuler.tar` | 通用 v0.23，适合 DSV4-Flash |
+| `vllm-ascend-glm5.2-a3-openeuler.tar` | GLM5.2 调优版（推荐跑 GLM5.2） |
+| `vllm-ascend-v0.22.1rc1-a3.tar` | 旧版 v0.22 |
+| `local-vllm-ascend-0.23-a3.tar` | 本地构建的 0.23 |
+
+加载（任选需要的）：
+
+```bash
+docker load -i /mnt/models/vllm-ascend-glm5.2-a3-openeuler.tar
+docker load -i /mnt/models/vllm-ascend-v0.23.0rc1-a3-openeuler.tar
+docker images | grep vllm-ascend   # 记下确切的 REPOSITORY:TAG
+```
+
+加载后把镜像 tag 写入对应实验的 `config.env`：
+
+```bash
+# experiments/910c/glm52_910c_vllm_tp_dp_matrix/config.env
+DOCKER_IMAGE="<加载后看到的 repository:tag>"
+```
+
+## 4. 模型权重
+
+当前 `/mnt/models/` 下已有：
+
+- `GLM-5.2-w4a8c8/`（95 shards，默认用这个）
+- `GLM-5.2-w8a8/`（181 shards，需切换时改 `MODEL_PATH`）
+
+**DeepSeek-V4-Flash 尚未下载**。需要时下载到 `/mnt/models/DeepSeek-V4-Flash`（FP8）或 `/mnt/models/DeepSeek-V4-Flash-INT8`，再改 `experiments/910c/dsv4_910c_vllm_tp_dp_matrix/config.env` 的 `MODEL_PATH`。
+
+## 5. 数据集
+
+`sglang.bench_serving --dataset-name random` 需要 ShareGPT 种子文件：
+
+```bash
+mkdir -p /mnt/yy/sskj/datasets
+# 放入 ShareGPT_V3_unfiltered_cleaned_split.json
+# （从 https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered 下载）
+```
+
+`DATASET_PATH` 默认指向 `${ROOT_DIR}/datasets/ShareGPT_V3_unfiltered_cleaned_split.json`，无需改 config。
+
+## 6. Ascend Docker Runtime 说明
+
+本机的 `/etc/docker/daemon.json` 已配置：
+
+```json
+{
+  "default-runtime": "ascend",
+  "runtimes": {
+    "ascend": {
+      "path": "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime",
+      "runtimeArgs": []
+    }
+  }
+}
+```
+
+因此 `docker run` **无需** `--runtime ascend` 或 `--gpus`，只需通过环境变量 `ASCEND_VISIBLE_DEVICES=0,1,2,3,4,5,6,7` 指定要映射的 NPU 卡号，runtime 会自动把对应 die 的 `/dev/davinci*` 注入容器。
+
+## 7. 冒烟测试 🤖
+
+```bash
+cd /mnt/yy/sskj/experiments/910c/glm52_910c_vllm_tp_dp_matrix
+
+# 1. dry-run：只打印搜索计划，不起服务
+DRY_RUN=1 bash run_adaptive_concurrency_add16.sh
+
+# 2. 单 shape 小并发冒烟（TP=8，1K/128，并发上限 8）
+RUN_ID=smoke-$(date +%Y%m%d-%H%M%S) \
+TP_LIST="8" ISL_LIST="1024" OSL_LIST="128" GRID_LIMIT=1 SEARCH_MAX_CONCURRENCY=8 \
+bash run_adaptive_concurrency_add16.sh
+```
+
+冒烟常见失败原因：
+
+| 现象 | 根因 | 解决 |
+|---|---|---|
+| `dcmi module initialize failed` | 用户不在 HwHiAiUser 组 | 见 §2 |
+| `docker: permission denied` | 用户不在 docker 组 | 见 §2 |
+| 容器内 `ModuleNotFoundError: torch_npu` | 镜像未正确加载 / tag 写错 | `docker images` 核对 |
+| `sglang.bench_serving` ModuleNotFoundError | vllm-ascend 镜像不含 sglang | 见 §8 |
+| 启动即 OOM | TP 过小，专家权重放不下 | 见 config.env 显存预算注释 |
+
+## 8. Benchmark Client 说明 ⚠️
+
+sskj 的压测客户端是 `sglang.bench_serving`，但 **vllm-ascend 官方镜像不含 sglang**。910C 上有两个选择：
+
+1. **(推荐) 容器内装 sglang**：进容器 `pip install sglang`（或 `sglang[all]`），之后 `docker exec` 跑客户端。需要把 sglang 装进镜像或每次启动后手动装。
+2. **外部 sglang 镜像**：设 `USE_DOCKER_CLIENT=1`，提供 `DOCKER_CLIENT_IMAGE=lmsysorg/sglang:xxx`，用独立容器通过 host 网络打 vLLM 的 OpenAI API。但 sglang 官方镜像多为 x86 + CUDA，aarch64 NPU 节点上可能拉不到对应架构镜像。
+
+建议冒烟前先确认客户端方案，否则 adaptive 搜索会在 `engine_run_bench` 阶段失败。验证命令：
+
+```bash
+# 进容器看是否有 sglang
+docker run --rm <vllm-ascend-image> python -c "import sglang; print(sglang.__version__)"
+```
+
+## 9. NPU 监控
+
+公共库 `adaptive_bench_lib.sh` 的 GPU 监控写死 `nvidia-smi`，910C 实验脚本已用 `npu-smi info` 重写 `adaptive_start_gpu_monitor` / `start_gpu_monitor`，输出与 nvidia-smi 相同的 CSV 列（timestamp, index, memory.used, memory.total, utilization.gpu），下游 `parse_backend.py` 无需改动。
+
+手动查看 NPU 状态：
+
+```bash
+npu-smi info                # 总览
+npu-smi info -t usages -i 0 # 单卡详细利用率
+```
+
+## 10. 已知坑
+
+1. **TP 与 die 的关系**：910C 每卡 2 die，vllm-ascend 按 die 分配 TP。`ASCEND_VISIBLE_DEVICES=0..7` 暴露 8 卡 = 16 die，因此 TP 最大 16（本实验限 TP≤8）。
+2. **KV cache dtype**：910C 支持 fp8 KV cache，但部分 vllm-ascend 版本在 NPU 上对 fp8 KV 支持不完整。若启动报 `kv-cache-dtype fp8 not supported`，改 `KV_CACHE_DTYPE=fp16`。
+3. **block-size**：NPU 推荐 128（NVIDIA H20 用 256）。若性能异常可尝试 64/128/256 对比。
+4. **DSV4-Flash FP8 显存**：路由专家 ~264 GiB，TP=2/4 在 64GB die 上几乎必 OOM（见 dsv4 config.env 注释）。用 INT8 权重或限 TP≥8。

envs/README.md

@@ -1,3 +1,4 @@
+
 # envs 目录说明
 
 本目录用于存放**环境搭建相关的文档和指南**，不包括具体的虚拟环境或容器镜像文件。
@@ -30,6 +31,7 @@ envs/
 |---|---|
 | `UV_ENV_SETUP.md` | uv 虚拟环境搭建规范，含 vLLM 和 SGLang 的标准安装命令 |
 | `SM120_DSV4_DEPLOYMENT_GUIDE.md` | RTX 6000D (SM120) 上部署 DeepSeek-V4-Flash 的完整指南，含 vLLM 和 SGLang 的适配步骤 |
+| `ASCEND_910C_ENV_SETUP.md` | 昇腾 910C NPU 上搭建 vLLM-Ascend 环境与部署指南，含权限、镜像加载、NPU 监控适配 |
 
 ## 排除的环境目录（已加入 .gitignore）
 

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/adaptive_config.env

@@ -0,0 +1,58 @@
+# Adaptive concurrency search settings.
+#
+# For each fixed (TP, DP, ISL, OSL), probe:
+#   C = start, start * multiplier, ... up to max
+# and stop after Total TPS has less than TPS_MIN_GAIN_PCT meaningful growth for
+# PLATEAU_PATIENCE consecutive points.
+
+SEARCH_START_CONCURRENCY="${SEARCH_START_CONCURRENCY:-1}"
+SEARCH_MAX_CONCURRENCY="${SEARCH_MAX_CONCURRENCY:-256}"
+
+# At the add16 initial probe, restart and retry C=8 then C=1 after an OOM.
+ENABLE_INITIAL_OOM_BACKOFF="${ENABLE_INITIAL_OOM_BACKOFF:-1}"
+SEARCH_MULTIPLIER="${SEARCH_MULTIPLIER:-2}"
+NUM_PROMPTS_MULTIPLIER="${NUM_PROMPTS_MULTIPLIER:-5}"
+
+# A gain below 2% is treated as throughput saturation. Two consecutive
+# low-gain points prevent one noisy measurement from stopping the search.
+TPS_MIN_GAIN_PCT="${TPS_MIN_GAIN_PCT:-2.0}"
+PLATEAU_PATIENCE="${PLATEAU_PATIENCE:-2}"
+
+# TTFT SLO early-stop settings.
+# When ttft_p95_ms exceeds TTFT_SLO_MS, stop searching the current (ISL, OSL)
+# shape and move on to the next scenario.
+TTFT_SLO_MS="${TTFT_SLO_MS:-4000}"
+ENABLE_TTFT_SLO_STOP="${ENABLE_TTFT_SLO_STOP:-1}"
+
+# Keep the same random workload semantics as the fixed matrix baseline.
+# DATASET_PATH must contain at least SEARCH_MAX_CONCURRENCY times
+# NUM_PROMPTS_MULTIPLIER valid two-turn conversations. Set this explicitly to
+# random-ids to use generated token IDs without a ShareGPT seed dataset.
+BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}"
+# SGLang interprets 0.0 as Uniform[1, requested_len]. Use 1.0 for fixed
+# ISL/OSL points; lower values intentionally benchmark a length distribution.
+RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1.0}"
+# Before each measured point, warm up with the same concurrency so lazy kernel
+# compilation and CUDA graph capture are excluded from TTFT/TPS. 0 means no
+# cap; set a positive cap only when very high-concurrency warmup is impractical.
+BENCH_WARMUP_MAX_REQUESTS="${BENCH_WARMUP_MAX_REQUESTS:-0}"
+
+# Reject a point if the completed request count or actual token lengths do not
+# match the requested workload.
+INPUT_LENGTH_TOLERANCE_PCT="${INPUT_LENGTH_TOLERANCE_PCT:-5.0}"
+OUTPUT_LENGTH_TOLERANCE_PCT="${OUTPUT_LENGTH_TOLERANCE_PCT:-10.0}"
+
+MAX_POINT_RETRIES="${MAX_POINT_RETRIES:-1}"
+SERVER_RESTART_COOLDOWN_S="${SERVER_RESTART_COOLDOWN_S:-10}"
+SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
+GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
+
+# Optional space-separated filters, useful for smoke tests:
+#   TP_LIST="8" ISL_LIST="1024" OSL_LIST="128"
+TP_LIST="${TP_LIST:-}"
+ISL_LIST="${ISL_LIST:-}"
+OSL_LIST="${OSL_LIST:-}"
+
+DRY_RUN="${DRY_RUN:-0}"
+# Counts ISL/OSL shapes per TP/DP config, not individual concurrency probes.
+GRID_LIMIT="${GRID_LIMIT:-0}"

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/compare.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""Cross TP×DP configuration comparison for dsv4_h20_vllm_tp_dp_matrix.
+
+Usage:
+    python3 compare.py --run-root results/<run_id> [--output comparison.md]
+"""
+import argparse
+import json
+import re
+from collections import defaultdict
+from pathlib import Path
+
+
+def load_result(result_root: Path) -> dict:
+    path = result_root / "results.json"
+    with open(path, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def slo_status(ttft_p95_ms: float, tpot_mean_ms: float,
+               ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str:
+    ttft_ok = ttft_p95_ms < ttft_limit_ms
+    tpot_ok = tpot_mean_ms < tpot_limit_ms
+    if ttft_ok and tpot_ok:
+        return "PASS"
+    if ttft_ok or tpot_ok:
+        return "PARTIAL"
+    return "FAIL"
+
+
+def gpu_memory_str(gpu: dict | None) -> str:
+    if not gpu:
+        return "-"
+    peak = gpu.get("peak_used_mb", 0)
+    total = gpu.get("memory_total_mb", 0)
+    if total:
+        return f"{peak:.0f}/{total:.0f} ({100*peak/total:.1f}%)"
+    return f"{peak:.0f}"
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md"))
+    parser.add_argument("--ttft-limit", type=float, default=3000.0)
+    parser.add_argument("--tpot-limit", type=float, default=50.0)
+    args = parser.parse_args()
+
+    # Discover configurations: tp*_dp* directories.
+    configs = []
+    for subdir in sorted(args.run_root.iterdir()):
+        if not subdir.is_dir():
+            continue
+        name = subdir.name
+        if not (name.startswith("tp") and "_dp" in name):
+            continue
+        results_json = subdir / "results.json"
+        if not results_json.exists():
+            continue
+        configs.append((name, load_result(subdir)))
+
+    if not configs:
+        print(f"No tp*_dp* results found under {args.run_root}")
+        return
+
+    model = configs[0][1].get("metadata", {}).get("model", "unknown")
+    hardware = configs[0][1].get("metadata", {}).get("hardware", "unknown")
+
+    # Group by scenario name.
+    by_scenario: dict[str, dict[str, dict]] = defaultdict(dict)
+    skipped: dict[str, dict[str, str]] = defaultdict(dict)
+    for label, data in configs:
+        for s in data.get("scenarios", []):
+            key = s["name"]
+            if s.get("status") == "skipped_oom":
+                skipped[key][label] = s.get("note", "skipped")
+            else:
+                by_scenario[key][label] = s
+
+    with open(args.output, "w", encoding="utf-8") as f:
+        f.write(f"# vLLM TP×DP matrix comparison ({hardware})\n\n")
+        f.write("## Summary\n\n")
+        f.write(f"- Model: `{model}`\n")
+        f.write(f"- Hardware: {hardware}\n")
+        f.write("- Backend: vLLM (Docker)\n")
+        f.write("- Benchmark client: `sglang.bench_serving`\n")
+        f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n")
+
+        # Configuration overview.
+        f.write("### Configurations\n\n")
+        f.write("| Config | TP | DP | GPUs/replica | Notes |\n")
+        f.write("|---|---:|---:|---:|---|\n")
+        for label, data in configs:
+            cfg = data.get("config", {})
+            tp = cfg.get("tp", "?")
+            dp = cfg.get("dp", "?")
+            f.write(f"| {label} | {tp} | {dp} | {tp} | server args recorded per ISL in results.json |\n")
+        f.write("\n")
+
+        # Side-by-side table.
+        f.write("## Side-by-side results\n\n")
+        headers = [
+            "Scenario", "ISL", "OSL", "Config", "Conc", "Req/s", "OutTok/s",
+            "TTFT P95(ms)", "TTFT P99(ms)", "TPOT Mean(ms)", "TPOT P95(ms)",
+            "TPOT P99(ms)", "E2E P99(ms)", "Peak GPU mem", "SLO"
+        ]
+        f.write("| " + " | ".join(headers) + " |\n")
+        f.write("|" + "|".join(["---"] * len(headers)) + "|\n")
+
+        for scenario_name in sorted(by_scenario.keys(), key=lambda x: tuple(map(int, re.findall(r"\d+", x)))):
+            _, isl, osl = re.findall(r"\d+", scenario_name)
+            # cfg_part not used; just for readability.
+            for label, data in configs:
+                s = by_scenario[scenario_name].get(label)
+                if s is None:
+                    if scenario_name in skipped and label in skipped[scenario_name]:
+                        note = skipped[scenario_name][label]
+                        f.write(f"| {scenario_name} | {isl} | {osl} | {label} | - | - | - | - | - | - | - | - | - | - | {note} |\n")
+                    continue
+                cfg = s["config"]
+                m = s["metrics"]
+                status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
+                gpu = m.get("gpu_memory")
+                f.write(
+                    f"| {scenario_name} | {isl} | {osl} | {label} | {cfg['concurrency']} | "
+                    f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
+                    f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
+                    f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
+                    f"{m['e2e_ms']['p99']:.2f} | {gpu_memory_str(gpu)} | {status} |\n"
+                )
+
+        # Best throughput per ISL/OSL.
+        f.write("\n## Best throughput per (ISL, OSL)\n\n")
+        f.write("| ISL | OSL | Best Config | Concurrency | OutTok/s | TTFT P95(ms) | TPOT Mean(ms) | SLO |\n")
+        f.write("|---:|---:|---|---:|---:|---:|---:|---:|\n")
+        best_by_shape: dict[tuple[int, int], tuple[float, str, dict]] = {}
+        for scenario_name, backends in by_scenario.items():
+            _, isl, osl = re.findall(r"\d+", scenario_name)
+            isl_i, osl_i = int(isl), int(osl)
+            for label, s in backends.items():
+                m = s["metrics"]
+                out_tok = m["output_token_throughput"]
+                if (isl_i, osl_i) not in best_by_shape or out_tok > best_by_shape[(isl_i, osl_i)][0]:
+                    best_by_shape[(isl_i, osl_i)] = (out_tok, label, s)
+        for (isl_i, osl_i), (out_tok, label, s) in sorted(best_by_shape.items()):
+            m = s["metrics"]
+            status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
+            f.write(
+                f"| {isl_i} | {osl_i} | {label} | {s['config']['concurrency']} | "
+                f"{out_tok:.2f} | {m['ttft_ms']['p95']:.2f} | {m['tpot_ms']['mean']:.2f} | {status} |\n"
+            )
+
+        f.write("\n## Notes\n\n")
+        f.write("- SLO check uses TTFT P95 and TPOT mean.\n")
+        f.write("- A PARTIAL indicates one of the two metrics is out of target; FAIL indicates both are out.\n")
+        f.write("- `Peak GPU mem` shows peak used / total MB and utilization percentage.\n")
+        f.write("- Optional (P) combinations that failed are marked as skipped/OOM and do not break the run.\n")
+
+    print(f"Wrote comparison to {args.output}")
+
+
+if __name__ == "__main__":
+    main()

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/config.env

@@ -0,0 +1,88 @@
+# TP×DP matrix experiment for DeepSeek-V4-Flash on Ascend 910C (8 NPUs / 16 dies)
+# using vLLM-Ascend.
+# Tests vLLM with three parallel configurations:
+#   TP=2, DP=4  -> 2 dies per replica, 4 replicas
+#   TP=4, DP=2  -> 4 dies per replica, 2 replicas
+#   TP=8, DP=1  -> 8 dies, no data parallelism
+#
+# Platform: ascend_910c (see platforms/ascend_910c.env).
+# Host: 910c.1 / NPU-NODE61, openEuler 22.03 SP4 aarch64, driver 25.5.2, CANN 9.0.0.
+#
+# ⚠️ PLACEHOLDER: DeepSeek-V4-Flash weights are NOT yet present on this host
+#   (only GLM-5.2-{w4a8c8,w8a8} under /mnt/models). Download DSV4-Flash first,
+#   then fix MODEL_PATH below. Smoke run will fail until the model exists.
+#   Recommended host path: /mnt/models/DeepSeek-V4-Flash  (FP8)
+#                          /mnt/models/DeepSeek-V4-Flash-INT8 (INT8)
+
+EXPERIMENT="dsv4_910c_vllm_tp_dp_matrix"
+MODEL_NAME="DeepSeek-V4-Flash"
+# TODO: point this at the real DSV4-Flash directory once downloaded.
+MODEL_PATH="${MODEL_PATH:-/mnt/models/DeepSeek-V4-Flash}"
+SERVED_MODEL_NAME="deepseek-v4-flash"
+
+VLLM_PORT="${VLLM_PORT:-30052}"
+
+# Dedicated container name so this experiment never touches other 910c runs.
+CONTAINER_NAME="${CONTAINER_NAME:-vllm-ascend-dsv4-910c}"
+
+# Python interpreter for the benchmark client inside the vllm-ascend container.
+CONTAINER_PYTHON="${CONTAINER_PYTHON:-/usr/local/bin/python}"
+
+# vllm-ascend image. Use the general v0.23 image for DSV4 (the GLM5.2-tuned
+# variant may carry GLM-specific patches). Load from:
+#   /mnt/models/vllm-ascend-v0.23.0rc1-a3-openeuler.tar
+USE_DOCKER="${USE_DOCKER:-1}"
+DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:v0.23.0rc1-a3-openeuler}"
+
+DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
+USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-0}"
+
+export ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
+export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
+
+RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
+
+# Parallel configurations to test. Format: "TP DP"
+# DSV4-Flash FP8 routed-expert weights ~264 GiB total. Per-die load = 264/TP GiB.
+#   TP=2 -> 132 GiB/die > 64 GiB HBM  ❌ OOM expected (kept but will fail)
+#   TP=4 ->  66 GiB/die > 64 GiB HBM  ❌ borderline OOM (KV cache leaves no room)
+#   TP=8 ->  33 GiB/die < 64 GiB HBM  ✅ fits with room for KV cache
+# TP=2/DP=4 and TP=4/DP=2 are likely infeasible for FP8 on 64GB dies; they are
+# kept here so the smoke run records the OOM boundary explicitly. Switch to
+# INT8 weights (MODEL_PATH=...-INT8, ~132 GiB total) to make TP=4 viable.
+if [[ -n "${PARALLEL_CONFIGS_STR:-}" ]]; then
+  declare -a PARALLEL_CONFIGS=()
+  for pair in $PARALLEL_CONFIGS_STR; do
+    PARALLEL_CONFIGS+=("${pair//,/ }")
+  done
+else
+  declare -a PARALLEL_CONFIGS=(
+    "2 4"
+    "4 2"
+    "8 1"
+  )
+fi
+
+# vLLM-Ascend server settings for DSV4-Flash.
+GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.9}"
+KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}"
+BLOCK_SIZE="${BLOCK_SIZE:-128}"
+# DSV4-Flash supports up to 1M context, but 64GB HBM/die caps realistic ISL.
+# Start at 128K; extend via matrix.json once TP=8 is verified.
+MAX_MODEL_LEN="${MAX_MODEL_LEN:-131072}"
+MAX_NUM_SEQS="${MAX_NUM_SEQS:-256}"
+
+VLLM_ASCEND_ATTENTION_BACKEND="${VLLM_ASCEND_ATTENTION_BACKEND:-atb}"
+
+DATASET_PATH="${DATASET_PATH:-${ROOT_DIR}/datasets/ShareGPT_V3_unfiltered_cleaned_split.json}"
+
+MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR:-.}/matrix.json}"
+MATRIX_MODE="${MATRIX_MODE:-Y}"
+
+export CONCURRENCY_SAMPLES="${CONCURRENCY_SAMPLES:-2}"
+
+SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
+GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
+
+DRY_RUN="${DRY_RUN:-0}"
+GRID_LIMIT="${GRID_LIMIT:-0}"

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/generate_scenarios.py

@@ -0,0 +1,98 @@
+#!/usr/bin/env python3
+"""Generate the scenario list for the TP×DP matrix experiment.
+
+Reads matrix.json and prints TSV lines:
+
+    mark  input_len  output_len  concurrency  num_prompts
+
+mark is one of Y/P/N.  The caller (run_bench.sh) decides how to treat each.
+"""
+import argparse
+import json
+import math
+import os
+from pathlib import Path
+
+
+def sample_concurrency(low: int, high: int, target: int) -> list[int]:
+    """Return only the low and high concurrency values in [low, high].
+
+    For the TP×DP matrix we only need the two endpoints of the concurrency
+    range (e.g. 1 and 128 for ISL=1024). The `target` argument is kept for
+    API compatibility but is ignored.
+    """
+    assert 1 <= low <= high, f"invalid concurrency range: {low}-{high}"
+    if low == high:
+        return [low]
+    return [low, high]
+
+
+def generate_scenarios(matrix_path: Path, mode: str, target_samples: int) -> list[dict]:
+    with open(matrix_path, "r", encoding="utf-8") as f:
+        data = json.load(f)
+
+    matrix = data["matrix"]
+    concurrency_cfg = data["concurrency"]
+
+    scenarios = []
+    for isl_str in sorted(matrix.keys(), key=int):
+        osl_map = matrix[isl_str]
+        low = concurrency_cfg[isl_str]["low"]
+        high = concurrency_cfg[isl_str]["high"]
+        concurrencies = sample_concurrency(low, high, target_samples)
+
+        for osl_str in sorted(osl_map.keys(), key=int):
+            mark = osl_map[osl_str]
+            if mode == "Y" and mark != "Y":
+                continue
+            if mode == "Y+P" and mark not in ("Y", "P"):
+                continue
+            # mode == "all" keeps everything, including N.
+
+            for conc in concurrencies:
+                scenarios.append(
+                    {
+                        "mark": mark,
+                        "input_len": int(isl_str),
+                        "output_len": int(osl_str),
+                        "concurrency": conc,
+                        "num_prompts": conc * 5,
+                    }
+                )
+
+    return scenarios
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--matrix", type=Path, default=Path("matrix.json"))
+    parser.add_argument("--mode", choices=["Y", "Y+P", "all"], default=None,
+                        help="Scenario selection mode. Defaults to matrix.mode.")
+    parser.add_argument("--target-samples", type=int, default=0,
+                        help="Target number of concurrency samples. 0 = heuristic (6-8).")
+    args = parser.parse_args()
+
+    with open(args.matrix, "r", encoding="utf-8") as f:
+        data = json.load(f)
+
+    mode = args.mode if args.mode else data.get("mode", "Y+P")
+
+    target_samples = args.target_samples
+    if target_samples <= 0:
+        env_samples = os.getenv("CONCURRENCY_SAMPLES", "0")
+        try:
+            target_samples = int(env_samples)
+        except ValueError:
+            target_samples = 0
+    if target_samples <= 0:
+        target_samples = 7
+
+    scenarios = generate_scenarios(args.matrix, mode, target_samples)
+
+    print("mark\tinput_len\toutput_len\tconcurrency\tnum_prompts")
+    for s in scenarios:
+        print(f"{s['mark']}\t{s['input_len']}\t{s['output_len']}\t{s['concurrency']}\t{s['num_prompts']}")
+
+
+if __name__ == "__main__":
+    main()

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/matrix.json

@@ -0,0 +1,92 @@
+{
+  "comment": "910C / DeepSeek-V4-Flash: 128K context cap (extend to 1M once TP=8 verified). ISL beyond 131072 excluded. DSV4-Flash FP8 routed-expert ~264GiB: TP=2/4 likely OOM at high ISL (see config.env). 'P' = probe-only, 'N' = skip (OOM expected).",
+  "mode": "Y",
+  "matrix": {
+    "1024": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "Y"
+    },
+    "4096": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "Y"
+    },
+    "8192": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "Y"
+    },
+    "16384": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "P"
+    },
+    "32768": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "P",
+      "4096": "N"
+    },
+    "65536": {
+      "128": "Y",
+      "256": "Y",
+      "512": "P",
+      "1024": "P",
+      "2048": "N",
+      "4096": "N"
+    },
+    "131072": {
+      "128": "Y",
+      "256": "P",
+      "512": "N",
+      "1024": "N",
+      "2048": "N",
+      "4096": "N"
+    }
+  },
+  "concurrency": {
+    "1024": {
+      "low": 1,
+      "high": 128
+    },
+    "4096": {
+      "low": 1,
+      "high": 64
+    },
+    "8192": {
+      "low": 1,
+      "high": 64
+    },
+    "16384": {
+      "low": 1,
+      "high": 32
+    },
+    "32768": {
+      "low": 1,
+      "high": 16
+    },
+    "65536": {
+      "low": 1,
+      "high": 8
+    },
+    "131072": {
+      "low": 1,
+      "high": 4
+    }
+  }
+}

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_adaptive_concurrency.sh

@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+# Find the Total-TPS saturation concurrency for each vLLM TP/DP/ISL/OSL shape.
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
+
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/adaptive_config.env"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
+
+ENGINE="vllm"
+ENGINE_PORT="$VLLM_PORT"
+RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
+ACTIVE_ENGINE_SERVER_LOG=""
+
+if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
+  PYTHON="${VENV_CLIENT}/bin/python"
+else
+  PYTHON="$(command -v python3)"
+fi
+
+DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
+DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
+
+engine_is_healthy() {
+  curl --fail --silent --show-error --max-time 5 \
+    "http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
+}
+
+engine_stop_server() {
+  local tp="$1"
+  local dp="$2"
+  local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
+
+  if [[ -f "$pid_file" ]]; then
+    local pid
+    pid="$(cat "$pid_file")"
+    if kill -0 "$pid" 2>/dev/null; then
+      log "stopping vllm server pid=${pid} tp=${tp} dp=${dp}"
+      kill "$pid" 2>/dev/null || true
+      sleep 5
+      kill -9 "$pid" 2>/dev/null || true
+    fi
+    rm -f "$pid_file"
+  fi
+  docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
+  ACTIVE_ENGINE_SERVER_LOG=""
+  sleep 2
+}
+
+engine_build_server_args() {
+  local tp="$1"
+  local dp="$2"
+  local -a args=(
+    vllm serve "$MODEL_PATH"
+    --trust-remote-code
+    --kv-cache-dtype "$KV_CACHE_DTYPE"
+    --block-size "$BLOCK_SIZE"
+    --tensor-parallel-size "$tp"
+    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+    --max-model-len "$MAX_MODEL_LEN"
+    --max-num-seqs "$MAX_NUM_SEQS"
+    --no-enable-flashinfer-autotune
+    --host 0.0.0.0
+    --port "$ENGINE_PORT"
+  )
+  if (( dp > 1 )); then
+    args+=(--data-parallel-size "$dp")
+  fi
+  printf '%q ' "${args[@]}"
+}
+
+engine_start_server() {
+  local tp="$1"
+  local dp="$2"
+  local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
+  log "starting vllm server tp=${tp} dp=${dp}"
+  bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
+  if ! engine_is_healthy; then
+    log "ERROR: vllm health check failed tp=${tp} dp=${dp}"
+    return 1
+  fi
+  ACTIVE_ENGINE_SERVER_LOG="$(
+    find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
+      -name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \
+      -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
+  )"
+  log "vllm server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}"
+}
+
+engine_detect_oom() {
+  local detail_log="$1"
+  local tp="$2"
+  local dp="$3"
+  local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
+  local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
+  local -a logs=("$detail_log" "$outer_log")
+  if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
+    logs+=("$ACTIVE_ENGINE_SERVER_LOG")
+  fi
+  grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null
+}
+
+engine_run_bench() {
+  local isl="$1"
+  local osl="$2"
+  local concurrency="$3"
+  local num_prompts="$4"
+  local output_file="$5"
+  local warmup_requests
+  warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
+  local -a bench_args=(
+    --backend vllm
+    --host 127.0.0.1
+    --port "$ENGINE_PORT"
+    --dataset-name "$BENCH_DATASET_NAME"
+    --random-input-len "$isl"
+    --random-output-len "$osl"
+    --random-range-ratio "$RANDOM_RANGE_RATIO"
+    --num-prompts "$num_prompts"
+    --max-concurrency "$concurrency"
+    --request-rate 10000
+    --warmup-requests "$warmup_requests"
+    --output-file "$output_file"
+    --output-details
+    --disable-tqdm
+  )
+  if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
+    bench_args+=(--dataset-path "$DATASET_PATH")
+  else
+    bench_args+=(--tokenize-prompt)
+  fi
+
+  if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then
+    local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}")
+    if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
+      volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro")
+    fi
+    docker run --rm \
+      --network host \
+      "${volume_args[@]}" \
+      -e PYTHONUNBUFFERED=1 \
+      -e HF_HUB_OFFLINE=1 \
+      -e TRANSFORMERS_OFFLINE=1 \
+      "$DOCKER_CLIENT_IMAGE" \
+      python -m sglang.bench_serving "${bench_args[@]}"
+  else
+    "$PYTHON" -m sglang.bench_serving "${bench_args[@]}"
+  fi
+}
+
+export -f engine_run_bench
+export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT
+export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
+
+adaptive_main "$@"

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh

@@ -0,0 +1,190 @@
+
+#!/usr/bin/env bash
+# Find the Total-TPS saturation concurrency for each vLLM-Ascend TP/DP/ISL/OSL
+# shape on Ascend 910C (GLM-5.2).
+#
+# Differences vs the H20 vLLM variant:
+#   - GPU monitor overridden to use npu-smi (shared library hardcodes nvidia-smi).
+#   - engine_run_bench runs the client inside the vllm-ascend container via
+#     `docker exec`. The official vllm-ascend image does NOT ship
+#     sglang.bench_serving; see envs/ASCEND_910C_ENV_SETUP.md for the client story
+#     (install bench_serving into the container or use an external sglang image).
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
+
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/adaptive_config.env"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
+
+ENGINE="vllm"
+ENGINE_PORT="$VLLM_PORT"
+RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
+ACTIVE_ENGINE_SERVER_LOG=""
+
+PYTHON="${PYTHON:-$(command -v python3)}"
+DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}"
+
+# Override the shared library's nvidia-smi GPU monitor with an npu-smi sampler
+# that emits the same CSV shape (timestamp, index, mem.used, mem.total, util).
+adaptive_start_gpu_monitor() {
+  local csv_path="$1"
+  mkdir -p "$(dirname "$csv_path")"
+  # 910C: delegate to the shared npu-smi sampler so parse_backend.py sees the
+  # same CSV shape as nvidia-smi (see scripts/common/npu_smi_sampler.py).
+  "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/npu_smi_sampler.py" "$GPU_MEM_SAMPLE_INTERVAL_S" > "$csv_path" 2>/dev/null &
+  echo $!
+}
+
+engine_is_healthy() {
+  curl --fail --silent --show-error --max-time 5 \
+    "http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
+}
+
+engine_stop_server() {
+  local tp="$1"
+  local dp="$2"
+  local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
+  if [[ -f "$pid_file" ]]; then
+    local pid
+    pid="$(cat "$pid_file")"
+    if kill -0 "$pid" 2>/dev/null; then
+      log "stopping vllm-ascend server pid=${pid} tp=${tp} dp=${dp}"
+      kill "$pid" 2>/dev/null || true
+      sleep 5
+      kill -9 "$pid" 2>/dev/null || true
+    fi
+    rm -f "$pid_file"
+  fi
+  # Also remove the container by name in case the host process is gone but the
+  # container lingers (e.g. orphaned by a kill -9).
+  docker rm -f "${CONTAINER_NAME}_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
+  ACTIVE_ENGINE_SERVER_LOG=""
+  sleep 2
+}
+
+engine_build_server_args() {
+  local tp="$1"
+  local dp="$2"
+  local -a args=(
+    vllm serve "$MODEL_PATH"
+    --served-model-name "$SERVED_MODEL_NAME"
+    --trust-remote-code
+    --kv-cache-dtype "$KV_CACHE_DTYPE"
+    --block-size "$BLOCK_SIZE"
+    --tensor-parallel-size "$tp"
+    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+    --max-model-len "$MAX_MODEL_LEN"
+    --max-num-seqs "$MAX_NUM_SEQS"
+    --host 0.0.0.0
+    --port "$ENGINE_PORT"
+  )
+  if (( dp > 1 )); then
+    args+=(--data-parallel-size "$dp")
+  fi
+  printf '%q ' "${args[@]}"
+}
+
+engine_start_server() {
+  local tp="$1"
+  local dp="$2"
+  local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
+  log "starting vllm-ascend server tp=${tp} dp=${dp}"
+  bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
+  if ! engine_is_healthy; then
+    log "ERROR: vllm-ascend health check failed tp=${tp} dp=${dp}"
+    return 1
+  fi
+  ACTIVE_ENGINE_SERVER_LOG="$(
+    find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
+      -name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \
+      -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
+  )"
+  log "vllm-ascend server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}"
+}
+
+engine_detect_oom() {
+  local detail_log="$1"
+  local tp="$2"
+  local dp="$3"
+  local pattern='out of memory|OutOfMemory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory|NPU out of memory|acl.*memory|HBM'
+  local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
+  local -a logs=("$detail_log" "$outer_log")
+  if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
+    logs+=("$ACTIVE_ENGINE_SERVER_LOG")
+  fi
+  grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null
+}
+
+engine_run_bench() {
+  local isl="$1"
+  local osl="$2"
+  local concurrency="$3"
+  local num_prompts="$4"
+  local output_file="$5"
+  local container_name="${CONTAINER_NAME}_tp${tp}_dp${dp}"
+  local container_output="/tmp/bench_outputs/adaptive_$(basename "$output_file")"
+  if [[ "$output_file" == "/dev/null" ]]; then
+    container_output="/dev/null"
+  fi
+
+  local warmup_requests
+  warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
+
+  local -a bench_args=(
+    --backend vllm
+    --host 127.0.0.1
+    --port "$ENGINE_PORT"
+    --model "$SERVED_MODEL_NAME"
+    --dataset-name "$BENCH_DATASET_NAME"
+    --random-input-len "$isl"
+    --random-output-len "$osl"
+    --random-range-ratio "$RANDOM_RANGE_RATIO"
+    --num-prompts "$num_prompts"
+    --max-concurrency "$concurrency"
+    --request-rate 10000
+    --warmup-requests "$warmup_requests"
+    --output-file "$container_output"
+    --output-details
+    --disable-tqdm
+  )
+  if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
+    bench_args+=(--dataset-path "$DATASET_PATH")
+  else
+    bench_args+=(--tokenize-prompt)
+  fi
+
+  # Run the benchmark client inside the running vllm-ascend container via
+  # `docker exec`. NOTE: the official vllm-ascend image does NOT include
+  # sglang.bench_serving. If `docker exec ... sglang.bench_serving` fails with
+  # ModuleNotFoundError, either (a) pip install sglang into the container, or
+  # (b) set USE_DOCKER_CLIENT=1 and run an external sglang image against the
+  # host port. See envs/ASCEND_910C_ENV_SETUP.md.
+  docker exec "$container_name" mkdir -p /tmp/bench_outputs 2>/dev/null || true
+  docker exec "$container_name" \
+    env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
+    "$CONTAINER_PYTHON" -m sglang.bench_serving "${bench_args[@]}" || return $?
+
+  if [[ "$output_file" != "/dev/null" ]]; then
+    docker cp "$container_name:${container_output}" "$output_file" || return 1
+  fi
+}
+
+export -f engine_run_bench
+export ENGINE_PORT CONTAINER_NAME CONTAINER_PYTHON MODEL_PATH RESULT_BASE SERVED_MODEL_NAME
+export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
+
+export SEARCH_START_CONCURRENCY=16
+export SEARCH_ADDEND=16
+export SEARCH_INITIAL_BACKOFF_CONCURRENCIES="8 1"
+export TTFT_GROUP_SKIP_MS="${TTFT_GROUP_SKIP_MS:-8000}"
+
+adaptive_main "$@"

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/run_bench.sh

@@ -0,0 +1,546 @@
+#!/usr/bin/env bash
+# TP×DP matrix benchmark for DeepSeek-V4-Flash on vLLM (Docker).
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
+
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+# Export variables used inside functions that are called via bash -c subshells.
+export DATASET_PATH MODEL_PATH RESULT_BASE \
+  DOCKER_IMAGE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT
+
+RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
+RESULT_BASE="${SCRIPT_DIR}/results"
+MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR}/matrix.json}"
+MATRIX_MODE="${MATRIX_MODE:-Y}"
+SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
+GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
+DRY_RUN="${DRY_RUN:-0}"
+GRID_LIMIT="${GRID_LIMIT:-0}"
+
+if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
+  PYTHON="${VENV_CLIENT}/bin/python"
+else
+  PYTHON="$(command -v python3)"
+fi
+
+DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
+DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
+
+log_dir_global="${RESULT_BASE}/${RUN_ID}/logs"
+mkdir -p "$log_dir_global"
+log_init "${log_dir_global}/orchestrator.log"
+
+log "experiment=${EXPERIMENT_NAME} run_id=${RUN_ID} platform=${PLATFORM} hardware=${HARDWARE}"
+log "matrix_mode=${MATRIX_MODE} matrix_file=${MATRIX_FILE} dry_run=${DRY_RUN} grid_limit=${GRID_LIMIT}"
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+is_server_healthy() {
+  curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${VLLM_PORT}/health" >/dev/null 2>&1
+}
+
+stop_server() {
+  local tp="$1"
+  local dp="$2"
+
+  local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
+  if [[ -f "$pid_file" ]]; then
+    local pid
+    pid="$(cat "$pid_file")"
+    if kill -0 "$pid" 2>/dev/null; then
+      log "stopping vllm server pid=${pid} (tp=${tp}, dp=${dp})"
+      kill "$pid" 2>/dev/null || true
+      sleep 5
+      kill -9 "$pid" 2>/dev/null || true
+    fi
+    rm -f "$pid_file"
+  fi
+
+  # Fallback: remove any Docker container started by this experiment.
+  docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
+
+  # Fallback: kill any vllm serve processes for this model.
+  pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
+  pkill -9 -f "vllm serve.*${MODEL_PATH}" 2>/dev/null || true
+  sleep 2
+}
+
+build_server_args() {
+  local tp="$1"
+  local dp="$2"
+
+  local args=(
+    "vllm serve" "$MODEL_PATH"
+    --trust-remote-code
+    --kv-cache-dtype "$KV_CACHE_DTYPE"
+    --block-size "$BLOCK_SIZE"
+    --tensor-parallel-size "$tp"
+    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+    --max-model-len "$MAX_MODEL_LEN"
+    --max-num-seqs "$MAX_NUM_SEQS"
+    --no-enable-flashinfer-autotune
+    --host 0.0.0.0
+    --port "$VLLM_PORT"
+  )
+  if [[ "$dp" -gt 1 ]]; then
+    args+=(
+      --data-parallel-size "$dp"
+    )
+  fi
+  printf '%s ' "${args[@]}"
+}
+
+start_server() {
+  local tp="$1"
+  local dp="$2"
+
+  log "starting vllm server tp=${tp} dp=${dp}"
+  bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" \
+    >> "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1
+
+  if ! is_server_healthy; then
+    log "error: vllm server tp=${tp} dp=${dp} failed health check on port ${VLLM_PORT}"
+    return 1
+  fi
+  log "vllm server tp=${tp} dp=${dp} is healthy on port ${VLLM_PORT}"
+}
+
+restart_server() {
+  local tp="$1"
+  local dp="$2"
+  log "restarting vllm server tp=${tp} dp=${dp} after non-OOM failure"
+  stop_server "$tp" "$dp"
+  sleep 10
+  start_server "$tp" "$dp"
+}
+
+run_bench_serving() {
+  # Run sglang.bench_serving either natively or inside the SGLang Docker image.
+  if [[ "${USE_DOCKER_CLIENT:-1}" == "1" ]]; then
+    local vol_args=()
+    vol_args+=("-v" "${MODEL_PATH}:${MODEL_PATH}:ro")
+    if [[ -n "${DATASET_PATH:-}" && -f "${DATASET_PATH}" ]]; then
+      vol_args+=("-v" "${DATASET_PATH}:${DATASET_PATH}:ro")
+    fi
+    vol_args+=("-v" "${RESULT_BASE}:${RESULT_BASE}")
+    docker run --rm \
+      --network host \
+      "${vol_args[@]}" \
+      -e PYTHONUNBUFFERED=1 \
+      -e HF_HUB_OFFLINE=1 \
+      -e TRANSFORMERS_OFFLINE=1 \
+      -e HF_DATASETS_OFFLINE=1 \
+      "${DOCKER_CLIENT_IMAGE}" \
+      python -m sglang.bench_serving "$@"
+  else
+    "$PYTHON" -m sglang.bench_serving "$@"
+  fi
+}
+export -f run_bench_serving
+
+run_warmup() {
+  local input_len="$1"
+  local output_len="$2"
+
+  log "warming up (input=${input_len}, output=${output_len}, num=1)"
+  bash -c '
+    run_bench_serving \
+      --backend vllm \
+      --host 127.0.0.1 \
+      --port "'"$VLLM_PORT"'" \
+      --dataset-name random \
+      --dataset-path "'"$DATASET_PATH"'" \
+      --random-input-len "'"$input_len"'" \
+      --random-output-len "'"$output_len"'" \
+      --num-prompts 1 \
+      --max-concurrency 1 \
+      --request-rate 10000 \
+      --output-file /dev/null \
+      --output-details \
+      >> "'"${log_dir_global}/warmup.log"'" 2>&1
+  '
+  log "warmup completed"
+}
+
+scenario_already_completed() {
+  local output_file="$1"
+  local expected="$2"
+  [[ -s "$output_file" ]] || return 1
+  local completed
+  completed="$("$PYTHON" -c "
+import json, sys
+path = sys.argv[1]
+try:
+    with open(path, 'r', encoding='utf-8') as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                data = json.loads(line)
+                print(data.get('completed', 0))
+                break
+except Exception:
+    print(0)
+" "$output_file")"
+  [[ "${completed:-0}" -ge "$expected" ]]
+}
+
+scenario_already_processed() {
+  local result_root="$1"
+  local scenario_name="$2"
+  local json_path="${result_root}/results.json"
+  [[ -f "$json_path" ]] || return 1
+  "$PYTHON" -c "
+import json, sys
+path, name = sys.argv[1], sys.argv[2]
+try:
+    with open(path, 'r', encoding='utf-8') as f:
+        data = json.load(f)
+    for s in data.get('scenarios', []):
+        if s.get('name') == name:
+            if s.get('status') or s.get('metrics', {}).get('success', 0) > 0:
+                sys.exit(0)
+except Exception:
+    pass
+sys.exit(1)
+" "$json_path" "$scenario_name"
+}
+
+detect_oom() {
+  local detail_log="$1"
+  local server_outer_log="$2"
+  local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory|NPU out of memory|acl.*memory|HBM'
+  if grep -Eiq "$pattern" "$detail_log" "$server_outer_log" 2>/dev/null; then
+    return 0
+  fi
+  return 1
+}
+
+start_gpu_monitor() {
+  local csv_path="$1"
+  mkdir -p "$(dirname "$csv_path")"
+  # 910C: use npu-smi via a standalone python sampler that emits the same CSV
+  # shape as nvidia-smi, so parse_backend.py needs no changes.
+  "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/npu_smi_sampler.py" "$GPU_MEM_SAMPLE_INTERVAL_S" > "$csv_path" 2>/dev/null &
+  echo $!
+}
+
+stop_gpu_monitor() {
+  local pid="$1"
+  if kill -0 "$pid" 2>/dev/null; then
+    kill "$pid" 2>/dev/null || true
+    sleep 1
+    kill -9 "$pid" 2>/dev/null || true
+  fi
+}
+
+append_scenario_record() {
+  local result_root="$1"
+  local json_path="$result_root/results.json"
+  shift
+  local scenario_json
+  scenario_json="$("$PYTHON" -c "
+import json, sys
+pairs = [a.split('=', 1) for a in sys.argv[1:]]
+d = {}
+for k, v in pairs:
+    try:
+        d[k] = json.loads(v)
+    except json.JSONDecodeError:
+        d[k] = v
+print(json.dumps(d, ensure_ascii=False))
+" "$@")"
+  PYTHON="$PYTHON" append_scenario_to_json "$json_path" "$scenario_json"
+}
+
+record_skipped_csv() {
+  local csv_path="$1"
+  shift
+  # Args: key=value
+  local row
+  row="$("$PYTHON" -c "
+import csv, json, sys, io
+pairs = [a.split('=', 1) for a in sys.argv[1:]]
+d = {}
+for k, v in pairs:
+    try:
+        d[k] = json.loads(v)
+    except json.JSONDecodeError:
+        d[k] = v
+buf = io.StringIO()
+writer = csv.DictWriter(buf, fieldnames=['engine','tp','dp','mark','isl','osl','concurrency','status','reason','detail_log'], extrasaction='ignore')
+writer.writerow(d)
+print(buf.getvalue().strip())
+" "$@")"
+  echo "$row" >> "$csv_path"
+}
+
+skip_remaining_scenarios() {
+  local result_root="$1"
+  local scenario_tsv="$2"
+  local start_index="$3"
+  local status="$4"
+  local reason="$5"
+  local tp="$6"
+  local dp="$7"
+  local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
+
+  local i=0
+  tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl osl conc num; do
+    if (( i < start_index )); then
+      i=$((i + 1))
+      continue
+    fi
+    i=$((i + 1))
+    local sname="c${conc}_i${isl}_o${osl}"
+    if scenario_already_processed "$result_root" "$sname"; then
+      continue
+    fi
+    append_scenario_record "$result_root" \
+      "name=${sname}" \
+      "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+      "status=\"${status}\"" \
+      "note=\"${reason}\""
+    record_skipped_csv "$skipped_csv" \
+      "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=${status}" "reason=${reason}"
+  done
+}
+
+# ---------------------------------------------------------------------------
+# Per-configuration runner
+# ---------------------------------------------------------------------------
+
+run_parallel_config() {
+  local tp="$1"
+  local dp="$2"
+  local config_label="tp${tp}_dp${dp}"
+  local result_root="${RESULT_BASE}/${RUN_ID}/${config_label}"
+  local raw_dir="${result_root}/raw_outputs"
+  local gpu_log_dir="${result_root}/gpu_logs"
+  local phase_log_dir="${result_root}/logs"
+  mkdir -p "$raw_dir" "$gpu_log_dir" "$phase_log_dir"
+
+  log "===== ${config_label} START ====="
+
+  # Generate scenario list for this config.
+  local scenario_tsv="${result_root}/scenarios.tsv"
+  "$PYTHON" "${SCRIPT_DIR}/generate_scenarios.py" \
+    --matrix "$MATRIX_FILE" \
+    --mode "$MATRIX_MODE" \
+    > "$scenario_tsv"
+  local total_scenarios
+  total_scenarios="$(tail -n +2 "$scenario_tsv" | wc -l)"
+  log "generated ${total_scenarios} scenarios for ${config_label}"
+
+  # Write metadata.
+  ensure_result_root "$result_root"
+  write_metadata_json \
+    "${result_root}/results.json" \
+    "${EXPERIMENT_NAME}_${config_label}" \
+    "$RUN_ID" \
+    "$MODEL_PATH" \
+    "vllm" \
+    "vllm" \
+    "$HARDWARE" \
+    "$ACCELERATOR" \
+    "$CHIP" \
+    "experiments/${EXPERIMENT_NAME}/run_bench.sh" \
+    "$DOCKER_IMAGE" \
+    "H20 vLLM TP×DP matrix for DeepSeek-V4-Flash"
+
+  local server_args_str
+  server_args_str="$(build_server_args "$tp" "$dp")"
+  jq --arg tp "$tp" --arg dp "$dp" --arg cuda "$CUDA_VISIBLE_DEVICES" --arg args "$server_args_str" \
+    '.config = {
+       "tp": ($tp | tonumber),
+       "dp": ($dp | tonumber),
+       "cuda_visible_devices": $cuda,
+       "backend": "vllm",
+       "server_start_script": "experiments/'${EXPERIMENT_NAME}'/start_vllm_dp.sh",
+       "server_args": $args
+     }' "${result_root}/results.json" > "${result_root}/results.json.tmp" && \
+    mv "${result_root}/results.json.tmp" "${result_root}/results.json"
+
+  if [[ "$DRY_RUN" == "1" ]]; then
+    log "DRY_RUN: would start server with args: ${server_args_str}"
+    local line
+    tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl osl conc num; do
+      log "DRY_RUN: ${config_label} scenario mark=${mark} c=${conc} i=${isl} o=${osl} n=${num}"
+    done
+    log "===== ${config_label} DONE (dry run) ====="
+    return 0
+  fi
+
+  # Initialize skipped_after_oom.csv for this run.
+  local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
+  if [[ ! -f "$skipped_csv" ]]; then
+    echo "engine,tp,dp,mark,isl,osl,concurrency,status,reason,detail_log" > "$skipped_csv"
+  fi
+
+  # Start server once for this TP×DP config.
+  if ! start_server "$tp" "$dp"; then
+    log "ERROR: ${config_label} failed to start; skipping all scenarios"
+    skip_remaining_scenarios "$result_root" "$scenario_tsv" 0 "SKIPPED_SERVICE_START_FAILED" "service failed to start" "$tp" "$dp"
+    log "===== ${config_label} DONE ====="
+    return 0
+  fi
+
+  # Warmup with a small prompt before the first scenario.
+  run_warmup 1024 128 || true
+
+  # Read scenarios into an array so we can skip remaining entries on failure.
+  local -a scenarios=()
+  while IFS= read -r line; do
+    scenarios+=("$line")
+  done < <(tail -n +2 "$scenario_tsv")
+
+  local i mark isl osl conc num
+  local output_file detail_log gpu_csv sname bench_rc
+  for (( i = 0; i < ${#scenarios[@]}; i++ )); do
+    IFS=$'\t' read -r mark isl osl conc num <<< "${scenarios[$i]}"
+
+    if [[ "$GRID_LIMIT" -gt 0 && "$i" -ge "$GRID_LIMIT" ]]; then
+      log "GRID_LIMIT=${GRID_LIMIT} reached; skipping remaining scenarios"
+      skip_remaining_scenarios "$result_root" "$scenario_tsv" "$i" "SKIPPED_GRID_LIMIT" "GRID_LIMIT reached" "$tp" "$dp"
+      break
+    fi
+
+    sname="c${conc}_i${isl}_o${osl}"
+    output_file="${raw_dir}/vllm_main_${conc}_${isl}_${osl}.jsonl"
+    detail_log="${phase_log_dir}/vllm_${config_label}_${sname}.log"
+    gpu_csv="${gpu_log_dir}/gpu_mem_${conc}_${isl}_${osl}.csv"
+
+    if scenario_already_completed "$output_file" "$num" || scenario_already_processed "$result_root" "$sname"; then
+      log "skipping already-processed ${config_label} scenario: ${sname}"
+      continue
+    fi
+
+    log "running ${config_label} scenario: mark=${mark} c=${conc} i=${isl} o=${osl} n=${num}"
+
+    local gpu_pid
+    gpu_pid="$(start_gpu_monitor "$gpu_csv")"
+
+    bench_rc=0
+    timeout "$SCENARIO_TIMEOUT_S" bash -c '
+      run_bench_serving \
+        --backend vllm \
+        --host 127.0.0.1 \
+        --port "'"$VLLM_PORT"'" \
+        --dataset-name random \
+        --dataset-path "'"$DATASET_PATH"'" \
+        --random-input-len "'"$isl"'" \
+        --random-output-len "'"$osl"'" \
+        --num-prompts "'"$num"'" \
+        --max-concurrency "'"$conc"'" \
+        --request-rate 10000 \
+        --output-file "'"$output_file"'" \
+        --output-details \
+        > "'"$detail_log"'" 2>&1
+    ' || bench_rc=$?
+
+    stop_gpu_monitor "$gpu_pid"
+
+    if [[ "$bench_rc" -eq 0 ]]; then
+      log "finished ${config_label} scenario: output=${output_file}"
+      append_scenario_record "$result_root" \
+        "name=${sname}" \
+        "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+        "status=\"completed\"" \
+        "note=\"benchmark finished successfully\""
+      continue
+    fi
+
+    # Failure handling.
+    if detect_oom "$detail_log" "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log"; then
+      log "ERROR: ${config_label} scenario ${sname} triggered OOM; stopping config"
+      append_scenario_record "$result_root" \
+        "name=${sname}" \
+        "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+        "status=\"OOM\"" \
+        "note=\"detected CUDA out-of-memory\""
+      record_skipped_csv "$skipped_csv" \
+        "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=OOM" "reason=detected CUDA out-of-memory" "detail_log=${detail_log}"
+      stop_server "$tp" "$dp"
+      skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_AFTER_OOM" "previous case OOM" "$tp" "$dp"
+      break
+    fi
+
+    log "ERROR: ${config_label} scenario ${sname} failed (rc=${bench_rc}); see ${detail_log}"
+    if [[ "$mark" == "P" ]]; then
+      log "optional (P) scenario failed; recording as skipped and continuing"
+      append_scenario_record "$result_root" \
+        "name=${sname}" \
+        "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+        "status=\"skipped_optional\"" \
+        "note=\"optional scenario failed (rc=${bench_rc})\""
+      record_skipped_csv "$skipped_csv" \
+        "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=skipped_optional" "reason=optional scenario failed (rc=${bench_rc})" "detail_log=${detail_log}"
+      continue
+    fi
+
+    # Mandatory scenario failed but not OOM: try to restart the server.
+    if restart_server "$tp" "$dp"; then
+      run_warmup 1024 128 || true
+      log "resuming ${config_label} after server restart"
+      continue
+    fi
+
+    log "ERROR: ${config_label} server restart failed; skipping remaining scenarios"
+    append_scenario_record "$result_root" \
+      "name=${sname}" \
+      "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+      "status=\"FAILED\"" \
+      "note=\"scenario failed and server restart failed (rc=${bench_rc})\""
+    record_skipped_csv "$skipped_csv" \
+      "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=FAILED" "reason=scenario failed and server restart failed" "detail_log=${detail_log}"
+    skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_RESTART_FAILED" "server restart failed" "$tp" "$dp"
+    break
+  done
+
+  stop_server "$tp" "$dp"
+
+  # Parse results.
+  log "parsing ${config_label} results"
+  "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/parse_backend.py" "$result_root" --backend vllm \
+    >> "${phase_log_dir}/parse.log" 2>&1 || {
+      log "WARNING: parser failed for ${config_label}; see ${phase_log_dir}/parse.log"
+    }
+
+  log "===== ${config_label} DONE ====="
+}
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+# Cleanup any leftovers.
+for cfg in "${PARALLEL_CONFIGS[@]}"; do
+  read -r tp dp <<< "$cfg"
+  stop_server "$tp" "$dp"
+done
+
+# Run each parallel configuration.
+for cfg in "${PARALLEL_CONFIGS[@]}"; do
+  read -r tp dp <<< "$cfg"
+  run_parallel_config "$tp" "$dp"
+done
+
+# Generate cross-configuration comparison.
+log "generating comparison report"
+"$PYTHON" "${SCRIPT_DIR}/compare.py" \
+  --run-root "${RESULT_BASE}/${RUN_ID}" \
+  --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \
+  >> "${log_dir_global}/compare.log" 2>&1 || {
+    log "WARNING: comparison script failed; see ${log_dir_global}/compare.log"
+  }
+
+log "all results saved to ${RESULT_BASE}/${RUN_ID}"

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/start_vllm_docker.sh

@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+# Start vLLM-Ascend server in Docker for a given TP×DP configuration on 910C.
+# Usage: start_vllm_docker.sh <TP> <DP>
+#
+# Relies on the Ascend Docker Runtime being the default docker runtime on this
+# host (see /etc/docker/daemon.json). NPU dies are injected via the
+# ASCEND_VISIBLE_DEVICES env var; no --gpus / --device flags are needed.
+set -e
+
+TP="${1}"
+DP="${2}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
+mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
+
+IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}"
+PORT="${VLLM_PORT:-30050}"
+NAME="${CONTAINER_NAME:-${EXPERIMENT}_vllm_tp${TP}_dp${DP}}"
+NAME="${NAME}_tp${TP}_dp${DP}"
+PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${TP}_dp${DP}.pid"
+
+LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_vllm_docker_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
+rm -f "$PID_FILE"
+
+# Clean up any stale container with the same name.
+docker rm -f "$NAME" >/dev/null 2>&1 || true
+
+# vLLM-Ascend launch args. Differences vs NVIDIA vLLM:
+#   - no --no-enable-flashinfer-autotune (Ascend uses its own attention path)
+#   - --kv-cache-dtype may need to be fp16 if the image rejects fp8 on 910C
+SERVER_ARGS=(
+  "$MODEL_PATH"
+  --served-model-name "$SERVED_MODEL_NAME"
+  --trust-remote-code
+  --kv-cache-dtype "$KV_CACHE_DTYPE"
+  --block-size "$BLOCK_SIZE"
+  --tensor-parallel-size "$TP"
+  --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+  --max-model-len "$MAX_MODEL_LEN"
+  --max-num-seqs "$MAX_NUM_SEQS"
+  --host 0.0.0.0
+  --port "$PORT"
+)
+
+if [[ "$DP" -gt 1 ]]; then
+  SERVER_ARGS+=(--data-parallel-size "$DP")
+fi
+
+SERVER_ARGS_STR="vllm serve ${SERVER_ARGS[*]}"
+
+echo "=== Starting vLLM-Ascend server in Docker (TP=${TP}, DP=${DP}) ==="
+echo "Image: $IMAGE"
+echo "Model: $MODEL_PATH"
+echo "Container name: $NAME"
+echo "Host port: $PORT"
+echo "ASCEND_VISIBLE_DEVICES: ${ASCEND_VISIBLE_DEVICES}"
+echo "Command: $SERVER_ARGS_STR"
+echo "Log: $LOG"
+
+# Run docker in the foreground; nohup backgrounds it and the host PID lets the
+# adaptive search stop the container by killing the process (the container has
+# --rm so it self-cleans). The Ascend Docker Runtime is the default runtime, so
+# no --runtime flag is required.
+nohup docker run --rm \
+  --name "$NAME" \
+  --ipc host \
+  --shm-size 16g \
+  --network host \
+  --ulimit memlock=-1 \
+  -v "${MODEL_PATH}:${MODEL_PATH}:ro" \
+  -v "${RUNTIME_BASE}/tmp:/tmp" \
+  -e ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES}" \
+  -e PYTORCH_NPU_ALLOC_CONF=expandable_segments:True \
+  -e PYTHONUNBUFFERED=1 \
+  -e HF_HUB_OFFLINE=1 \
+  -e TRANSFORMERS_OFFLINE=1 \
+  "$IMAGE" \
+  vllm serve "${SERVER_ARGS[@]}" \
+  > "$LOG" 2>&1 &
+
+PID=$!
+echo $PID > "$PID_FILE"
+echo "PID: $PID"
+echo "Waiting for health on port ${PORT}..."
+
+for i in $(seq 1 240); do
+  if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
+    echo "vLLM-Ascend server is ready at http://127.0.0.1:${PORT}"
+    echo "Log: $LOG"
+    exit 0
+  fi
+  if ! kill -0 $PID 2>/dev/null; then
+    echo "ERROR: Docker vLLM-Ascend server exited early"
+    tail -200 "$LOG"
+    exit 1
+  fi
+  echo "Waiting... ($i/240)"
+  sleep 5
+done
+
+echo "ERROR: vLLM-Ascend server not healthy after 240 retries"
+tail -200 "$LOG"
+exit 1

experiments/910c/dsv4_910c_vllm_tp_dp_matrix/start_vllm_dp.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Start vLLM-Ascend server for a given TP×DP configuration on 910C.
+# Usage: start_vllm_dp.sh <TP> <DP>
+#
+# 910C only supports the Docker path (vllm-ascend runs inside a container with
+# the Ascend Docker Runtime). USE_DOCKER=0 is not supported on this platform;
+# set USE_DOCKER=1 (the default) or call start_vllm_docker.sh directly.
+set -e
+
+TP="${1}"
+DP="${2}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+if [[ "${USE_DOCKER:-1}" == "1" ]]; then
+  exec "${SCRIPT_DIR}/start_vllm_docker.sh" "$@"
+fi
+
+echo "ERROR: USE_DOCKER=0 (native vllm-ascend) is not supported on 910C." >&2
+echo "       vllm-ascend requires the Ascend Docker Runtime; set USE_DOCKER=1." >&2
+exit 1

experiments/910c/glm52_910c_vllm_tp_dp_matrix/adaptive_config.env

@@ -0,0 +1,58 @@
+# Adaptive concurrency search settings.
+#
+# For each fixed (TP, DP, ISL, OSL), probe:
+#   C = start, start * multiplier, ... up to max
+# and stop after Total TPS has less than TPS_MIN_GAIN_PCT meaningful growth for
+# PLATEAU_PATIENCE consecutive points.
+
+SEARCH_START_CONCURRENCY="${SEARCH_START_CONCURRENCY:-1}"
+SEARCH_MAX_CONCURRENCY="${SEARCH_MAX_CONCURRENCY:-256}"
+
+# At the add16 initial probe, restart and retry C=8 then C=1 after an OOM.
+ENABLE_INITIAL_OOM_BACKOFF="${ENABLE_INITIAL_OOM_BACKOFF:-1}"
+SEARCH_MULTIPLIER="${SEARCH_MULTIPLIER:-2}"
+NUM_PROMPTS_MULTIPLIER="${NUM_PROMPTS_MULTIPLIER:-5}"
+
+# A gain below 2% is treated as throughput saturation. Two consecutive
+# low-gain points prevent one noisy measurement from stopping the search.
+TPS_MIN_GAIN_PCT="${TPS_MIN_GAIN_PCT:-2.0}"
+PLATEAU_PATIENCE="${PLATEAU_PATIENCE:-2}"
+
+# TTFT SLO early-stop settings.
+# When ttft_p95_ms exceeds TTFT_SLO_MS, stop searching the current (ISL, OSL)
+# shape and move on to the next scenario.
+TTFT_SLO_MS="${TTFT_SLO_MS:-4000}"
+ENABLE_TTFT_SLO_STOP="${ENABLE_TTFT_SLO_STOP:-1}"
+
+# Keep the same random workload semantics as the fixed matrix baseline.
+# DATASET_PATH must contain at least SEARCH_MAX_CONCURRENCY times
+# NUM_PROMPTS_MULTIPLIER valid two-turn conversations. Set this explicitly to
+# random-ids to use generated token IDs without a ShareGPT seed dataset.
+BENCH_DATASET_NAME="${BENCH_DATASET_NAME:-random}"
+# SGLang interprets 0.0 as Uniform[1, requested_len]. Use 1.0 for fixed
+# ISL/OSL points; lower values intentionally benchmark a length distribution.
+RANDOM_RANGE_RATIO="${RANDOM_RANGE_RATIO:-1.0}"
+# Before each measured point, warm up with the same concurrency so lazy kernel
+# compilation and CUDA graph capture are excluded from TTFT/TPS. 0 means no
+# cap; set a positive cap only when very high-concurrency warmup is impractical.
+BENCH_WARMUP_MAX_REQUESTS="${BENCH_WARMUP_MAX_REQUESTS:-0}"
+
+# Reject a point if the completed request count or actual token lengths do not
+# match the requested workload.
+INPUT_LENGTH_TOLERANCE_PCT="${INPUT_LENGTH_TOLERANCE_PCT:-5.0}"
+OUTPUT_LENGTH_TOLERANCE_PCT="${OUTPUT_LENGTH_TOLERANCE_PCT:-10.0}"
+
+MAX_POINT_RETRIES="${MAX_POINT_RETRIES:-1}"
+SERVER_RESTART_COOLDOWN_S="${SERVER_RESTART_COOLDOWN_S:-10}"
+SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
+GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
+
+# Optional space-separated filters, useful for smoke tests:
+#   TP_LIST="8" ISL_LIST="1024" OSL_LIST="128"
+TP_LIST="${TP_LIST:-}"
+ISL_LIST="${ISL_LIST:-}"
+OSL_LIST="${OSL_LIST:-}"
+
+DRY_RUN="${DRY_RUN:-0}"
+# Counts ISL/OSL shapes per TP/DP config, not individual concurrency probes.
+GRID_LIMIT="${GRID_LIMIT:-0}"

experiments/910c/glm52_910c_vllm_tp_dp_matrix/compare.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""Cross TP×DP configuration comparison for dsv4_h20_vllm_tp_dp_matrix.
+
+Usage:
+    python3 compare.py --run-root results/<run_id> [--output comparison.md]
+"""
+import argparse
+import json
+import re
+from collections import defaultdict
+from pathlib import Path
+
+
+def load_result(result_root: Path) -> dict:
+    path = result_root / "results.json"
+    with open(path, "r", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def slo_status(ttft_p95_ms: float, tpot_mean_ms: float,
+               ttft_limit_ms: float = 3000.0, tpot_limit_ms: float = 50.0) -> str:
+    ttft_ok = ttft_p95_ms < ttft_limit_ms
+    tpot_ok = tpot_mean_ms < tpot_limit_ms
+    if ttft_ok and tpot_ok:
+        return "PASS"
+    if ttft_ok or tpot_ok:
+        return "PARTIAL"
+    return "FAIL"
+
+
+def gpu_memory_str(gpu: dict | None) -> str:
+    if not gpu:
+        return "-"
+    peak = gpu.get("peak_used_mb", 0)
+    total = gpu.get("memory_total_mb", 0)
+    if total:
+        return f"{peak:.0f}/{total:.0f} ({100*peak/total:.1f}%)"
+    return f"{peak:.0f}"
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--run-root", type=Path, required=True)
+    parser.add_argument("-o", "--output", type=Path, default=Path("comparison.md"))
+    parser.add_argument("--ttft-limit", type=float, default=3000.0)
+    parser.add_argument("--tpot-limit", type=float, default=50.0)
+    args = parser.parse_args()
+
+    # Discover configurations: tp*_dp* directories.
+    configs = []
+    for subdir in sorted(args.run_root.iterdir()):
+        if not subdir.is_dir():
+            continue
+        name = subdir.name
+        if not (name.startswith("tp") and "_dp" in name):
+            continue
+        results_json = subdir / "results.json"
+        if not results_json.exists():
+            continue
+        configs.append((name, load_result(subdir)))
+
+    if not configs:
+        print(f"No tp*_dp* results found under {args.run_root}")
+        return
+
+    model = configs[0][1].get("metadata", {}).get("model", "unknown")
+    hardware = configs[0][1].get("metadata", {}).get("hardware", "unknown")
+
+    # Group by scenario name.
+    by_scenario: dict[str, dict[str, dict]] = defaultdict(dict)
+    skipped: dict[str, dict[str, str]] = defaultdict(dict)
+    for label, data in configs:
+        for s in data.get("scenarios", []):
+            key = s["name"]
+            if s.get("status") == "skipped_oom":
+                skipped[key][label] = s.get("note", "skipped")
+            else:
+                by_scenario[key][label] = s
+
+    with open(args.output, "w", encoding="utf-8") as f:
+        f.write(f"# vLLM TP×DP matrix comparison ({hardware})\n\n")
+        f.write("## Summary\n\n")
+        f.write(f"- Model: `{model}`\n")
+        f.write(f"- Hardware: {hardware}\n")
+        f.write("- Backend: vLLM (Docker)\n")
+        f.write("- Benchmark client: `sglang.bench_serving`\n")
+        f.write(f"- SLO reference: TTFT P95 < {args.ttft_limit}ms, TPOT mean < {args.tpot_limit}ms\n\n")
+
+        # Configuration overview.
+        f.write("### Configurations\n\n")
+        f.write("| Config | TP | DP | GPUs/replica | Notes |\n")
+        f.write("|---|---:|---:|---:|---|\n")
+        for label, data in configs:
+            cfg = data.get("config", {})
+            tp = cfg.get("tp", "?")
+            dp = cfg.get("dp", "?")
+            f.write(f"| {label} | {tp} | {dp} | {tp} | server args recorded per ISL in results.json |\n")
+        f.write("\n")
+
+        # Side-by-side table.
+        f.write("## Side-by-side results\n\n")
+        headers = [
+            "Scenario", "ISL", "OSL", "Config", "Conc", "Req/s", "OutTok/s",
+            "TTFT P95(ms)", "TTFT P99(ms)", "TPOT Mean(ms)", "TPOT P95(ms)",
+            "TPOT P99(ms)", "E2E P99(ms)", "Peak GPU mem", "SLO"
+        ]
+        f.write("| " + " | ".join(headers) + " |\n")
+        f.write("|" + "|".join(["---"] * len(headers)) + "|\n")
+
+        for scenario_name in sorted(by_scenario.keys(), key=lambda x: tuple(map(int, re.findall(r"\d+", x)))):
+            _, isl, osl = re.findall(r"\d+", scenario_name)
+            # cfg_part not used; just for readability.
+            for label, data in configs:
+                s = by_scenario[scenario_name].get(label)
+                if s is None:
+                    if scenario_name in skipped and label in skipped[scenario_name]:
+                        note = skipped[scenario_name][label]
+                        f.write(f"| {scenario_name} | {isl} | {osl} | {label} | - | - | - | - | - | - | - | - | - | - | {note} |\n")
+                    continue
+                cfg = s["config"]
+                m = s["metrics"]
+                status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
+                gpu = m.get("gpu_memory")
+                f.write(
+                    f"| {scenario_name} | {isl} | {osl} | {label} | {cfg['concurrency']} | "
+                    f"{m['request_throughput']:.2f} | {m['output_token_throughput']:.2f} | "
+                    f"{m['ttft_ms']['p95']:.2f} | {m['ttft_ms']['p99']:.2f} | "
+                    f"{m['tpot_ms']['mean']:.2f} | {m['tpot_ms']['p95']:.2f} | {m['tpot_ms']['p99']:.2f} | "
+                    f"{m['e2e_ms']['p99']:.2f} | {gpu_memory_str(gpu)} | {status} |\n"
+                )
+
+        # Best throughput per ISL/OSL.
+        f.write("\n## Best throughput per (ISL, OSL)\n\n")
+        f.write("| ISL | OSL | Best Config | Concurrency | OutTok/s | TTFT P95(ms) | TPOT Mean(ms) | SLO |\n")
+        f.write("|---:|---:|---|---:|---:|---:|---:|---:|\n")
+        best_by_shape: dict[tuple[int, int], tuple[float, str, dict]] = {}
+        for scenario_name, backends in by_scenario.items():
+            _, isl, osl = re.findall(r"\d+", scenario_name)
+            isl_i, osl_i = int(isl), int(osl)
+            for label, s in backends.items():
+                m = s["metrics"]
+                out_tok = m["output_token_throughput"]
+                if (isl_i, osl_i) not in best_by_shape or out_tok > best_by_shape[(isl_i, osl_i)][0]:
+                    best_by_shape[(isl_i, osl_i)] = (out_tok, label, s)
+        for (isl_i, osl_i), (out_tok, label, s) in sorted(best_by_shape.items()):
+            m = s["metrics"]
+            status = slo_status(m["ttft_ms"]["p95"], m["tpot_ms"]["mean"], args.ttft_limit, args.tpot_limit)
+            f.write(
+                f"| {isl_i} | {osl_i} | {label} | {s['config']['concurrency']} | "
+                f"{out_tok:.2f} | {m['ttft_ms']['p95']:.2f} | {m['tpot_ms']['mean']:.2f} | {status} |\n"
+            )
+
+        f.write("\n## Notes\n\n")
+        f.write("- SLO check uses TTFT P95 and TPOT mean.\n")
+        f.write("- A PARTIAL indicates one of the two metrics is out of target; FAIL indicates both are out.\n")
+        f.write("- `Peak GPU mem` shows peak used / total MB and utilization percentage.\n")
+        f.write("- Optional (P) combinations that failed are marked as skipped/OOM and do not break the run.\n")
+
+    print(f"Wrote comparison to {args.output}")
+
+
+if __name__ == "__main__":
+    main()

experiments/910c/glm52_910c_vllm_tp_dp_matrix/config.env

@@ -0,0 +1,95 @@
+# TP×DP matrix experiment for GLM-5.2 on Ascend 910C (8 NPUs / 16 dies) using vLLM-Ascend.
+# Tests vLLM with three parallel configurations:
+#   TP=2, DP=4  -> 2 dies per replica, 4 replicas
+#   TP=4, DP=2  -> 4 dies per replica, 2 replicas
+#   TP=8, DP=1  -> 8 dies, no data parallelism
+#
+# Platform: ascend_910c (see platforms/ascend_910c.env).
+# Host: 910c.1 / NPU-NODE61, openEuler 22.03 SP4 aarch64, driver 25.5.2, CANN 9.0.0.
+# Image: vllm-ascend; load the tarball from /mnt/models first (see envs/ASCEND_910C_ENV_SETUP.md).
+
+EXPERIMENT="glm52_910c_vllm_tp_dp_matrix"
+MODEL_NAME="GLM-5.2"
+# GLM-5.2 ships two quantized variants on this host; w4a8c8 is the default.
+# Switch to /mnt/models/GLM-5.2-w8a8 by overriding MODEL_PATH if needed.
+MODEL_PATH="${MODEL_PATH:-/mnt/models/GLM-5.2-w4a8c8}"
+SERVED_MODEL_NAME="glm-5.2"
+
+VLLM_PORT="${VLLM_PORT:-30050}"
+
+# Dedicated container name so this experiment never touches other 910c runs.
+CONTAINER_NAME="${CONTAINER_NAME:-vllm-ascend-glm52-910c}"
+
+# Python interpreter for the benchmark client inside the vllm-ascend container.
+CONTAINER_PYTHON="${CONTAINER_PYTHON:-/usr/local/bin/python}"
+
+# vllm-ascend image. Override with the exact tag after `docker load`-ing one of:
+#   /mnt/models/vllm-ascend-glm5.2-a3-openeuler.tar          (GLM5.2-tuned, recommended)
+#   /mnt/models/vllm-ascend-v0.23.0rc1-a3-openeuler.tar      (general v0.23)
+USE_DOCKER="${USE_DOCKER:-1}"
+DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}"
+
+# Benchmark client Docker image. vLLM's image does not include sglang.bench_serving;
+# reuse the vllm-ascend container itself for the client via `docker exec` (see
+# run_adaptive_concurrency_add16.sh), so this is only used if USE_DOCKER_CLIENT=1
+# with an external sglang image. Default off on 910c.
+DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
+USE_DOCKER_CLIENT="${USE_DOCKER_CLIENT:-0}"
+
+# Device selection. ASCEND_VISIBLE_DEVICES selects NPU cards 0..7; the Ascend
+# Docker Runtime (default runtime on this host) injects the matching dies.
+export ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
+# Keep CUDA_VISIBLE_DEVICES for parity with the shared library; vllm-ascend
+# ignores it on NPU but some helper code reads it.
+export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
+
+# Runtime working directory for logs, pid files, and tmp.
+RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
+
+# Parallel configurations to test. Format: "TP DP"
+# Each Ascend910 card has 2 dies; TP addresses dies, so TP=8 uses 8 dies across
+# 4 cards and leaves room for DP. TP=2/DP=4 and TP=4/DP=2 and TP=8/DP=1 all fit
+# within 8 cards (16 dies). Override via PARALLEL_CONFIGS_STR="8,1".
+if [[ -n "${PARALLEL_CONFIGS_STR:-}" ]]; then
+  declare -a PARALLEL_CONFIGS=()
+  for pair in $PARALLEL_CONFIGS_STR; do
+    PARALLEL_CONFIGS+=("${pair//,/ }")
+  done
+else
+  declare -a PARALLEL_CONFIGS=(
+    "2 4"
+    "4 2"
+    "8 1"
+  )
+fi
+
+# vLLM-Ascend server settings for GLM-5.2 (w4a8c8).
+# Notes:
+#   - KV cache dtype fp8 is supported on 910C; fall back to fp16 if the image rejects it.
+#   - block-size 128 matches Ascend page semantics (P99 of H20 uses 256; 910C favors 128).
+#   - MAX_MODEL_LEN: GLM-5.2 supports up to 128K context; cap at 131072.
+#   - gpu-memory-utilization maps to NPU HBM fraction on vllm-ascend (0.9 mirrors H20).
+GPU_MEMORY_UTILIZATION="${GPU_MEMORY_UTILIZATION:-0.9}"
+KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-fp8}"
+BLOCK_SIZE="${BLOCK_SIZE:-128}"
+MAX_MODEL_LEN="${MAX_MODEL_LEN:-131072}"
+MAX_NUM_SEQS="${MAX_NUM_SEQS:-256}"
+
+# vLLM-Ascend-specific launch flags injected by start_vllm_docker.sh.
+# attention backend for 910C: use the fused/atb attention path. Adjust per image.
+VLLM_ASCEND_ATTENTION_BACKEND="${VLLM_ASCEND_ATTENTION_BACKEND:-atb}"
+
+# Dataset used by sglang.bench_serving --dataset-name random.
+DATASET_PATH="${DATASET_PATH:-${ROOT_DIR}/datasets/ShareGPT_V3_unfiltered_cleaned_split.json}"
+
+# Matrix and concurrency rules are defined in matrix.json by default.
+MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR:-.}/matrix.json}"
+MATRIX_MODE="${MATRIX_MODE:-Y}"
+
+export CONCURRENCY_SAMPLES="${CONCURRENCY_SAMPLES:-2}"
+
+SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
+GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
+
+DRY_RUN="${DRY_RUN:-0}"
+GRID_LIMIT="${GRID_LIMIT:-0}"

experiments/910c/glm52_910c_vllm_tp_dp_matrix/generate_scenarios.py

@@ -0,0 +1,98 @@
+#!/usr/bin/env python3
+"""Generate the scenario list for the TP×DP matrix experiment.
+
+Reads matrix.json and prints TSV lines:
+
+    mark  input_len  output_len  concurrency  num_prompts
+
+mark is one of Y/P/N.  The caller (run_bench.sh) decides how to treat each.
+"""
+import argparse
+import json
+import math
+import os
+from pathlib import Path
+
+
+def sample_concurrency(low: int, high: int, target: int) -> list[int]:
+    """Return only the low and high concurrency values in [low, high].
+
+    For the TP×DP matrix we only need the two endpoints of the concurrency
+    range (e.g. 1 and 128 for ISL=1024). The `target` argument is kept for
+    API compatibility but is ignored.
+    """
+    assert 1 <= low <= high, f"invalid concurrency range: {low}-{high}"
+    if low == high:
+        return [low]
+    return [low, high]
+
+
+def generate_scenarios(matrix_path: Path, mode: str, target_samples: int) -> list[dict]:
+    with open(matrix_path, "r", encoding="utf-8") as f:
+        data = json.load(f)
+
+    matrix = data["matrix"]
+    concurrency_cfg = data["concurrency"]
+
+    scenarios = []
+    for isl_str in sorted(matrix.keys(), key=int):
+        osl_map = matrix[isl_str]
+        low = concurrency_cfg[isl_str]["low"]
+        high = concurrency_cfg[isl_str]["high"]
+        concurrencies = sample_concurrency(low, high, target_samples)
+
+        for osl_str in sorted(osl_map.keys(), key=int):
+            mark = osl_map[osl_str]
+            if mode == "Y" and mark != "Y":
+                continue
+            if mode == "Y+P" and mark not in ("Y", "P"):
+                continue
+            # mode == "all" keeps everything, including N.
+
+            for conc in concurrencies:
+                scenarios.append(
+                    {
+                        "mark": mark,
+                        "input_len": int(isl_str),
+                        "output_len": int(osl_str),
+                        "concurrency": conc,
+                        "num_prompts": conc * 5,
+                    }
+                )
+
+    return scenarios
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--matrix", type=Path, default=Path("matrix.json"))
+    parser.add_argument("--mode", choices=["Y", "Y+P", "all"], default=None,
+                        help="Scenario selection mode. Defaults to matrix.mode.")
+    parser.add_argument("--target-samples", type=int, default=0,
+                        help="Target number of concurrency samples. 0 = heuristic (6-8).")
+    args = parser.parse_args()
+
+    with open(args.matrix, "r", encoding="utf-8") as f:
+        data = json.load(f)
+
+    mode = args.mode if args.mode else data.get("mode", "Y+P")
+
+    target_samples = args.target_samples
+    if target_samples <= 0:
+        env_samples = os.getenv("CONCURRENCY_SAMPLES", "0")
+        try:
+            target_samples = int(env_samples)
+        except ValueError:
+            target_samples = 0
+    if target_samples <= 0:
+        target_samples = 7
+
+    scenarios = generate_scenarios(args.matrix, mode, target_samples)
+
+    print("mark\tinput_len\toutput_len\tconcurrency\tnum_prompts")
+    for s in scenarios:
+        print(f"{s['mark']}\t{s['input_len']}\t{s['output_len']}\t{s['concurrency']}\t{s['num_prompts']}")
+
+
+if __name__ == "__main__":
+    main()

experiments/910c/glm52_910c_vllm_tp_dp_matrix/matrix.json

@@ -0,0 +1,92 @@
+{
+  "comment": "910C / GLM-5.2: 128K context cap. ISL beyond 131072 excluded. Concurrency high-end reduced for 64GB HBM/die (TP-sharded weights + KV cache). 'P' = probe-only (run if time permits), 'N' = skip (OOM expected).",
+  "mode": "Y",
+  "matrix": {
+    "1024": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "Y"
+    },
+    "4096": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "Y"
+    },
+    "8192": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "Y"
+    },
+    "16384": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "P"
+    },
+    "32768": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "Y",
+      "2048": "Y",
+      "4096": "P"
+    },
+    "65536": {
+      "128": "Y",
+      "256": "Y",
+      "512": "Y",
+      "1024": "P",
+      "2048": "P",
+      "4096": "N"
+    },
+    "131072": {
+      "128": "Y",
+      "256": "P",
+      "512": "P",
+      "1024": "N",
+      "2048": "N",
+      "4096": "N"
+    }
+  },
+  "concurrency": {
+    "1024": {
+      "low": 1,
+      "high": 128
+    },
+    "4096": {
+      "low": 1,
+      "high": 64
+    },
+    "8192": {
+      "low": 1,
+      "high": 64
+    },
+    "16384": {
+      "low": 1,
+      "high": 32
+    },
+    "32768": {
+      "low": 1,
+      "high": 16
+    },
+    "65536": {
+      "low": 1,
+      "high": 8
+    },
+    "131072": {
+      "low": 1,
+      "high": 4
+    }
+  }
+}

experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_adaptive_concurrency.sh

@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+# Find the Total-TPS saturation concurrency for each vLLM TP/DP/ISL/OSL shape.
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
+
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/adaptive_config.env"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
+
+ENGINE="vllm"
+ENGINE_PORT="$VLLM_PORT"
+RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
+ACTIVE_ENGINE_SERVER_LOG=""
+
+if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
+  PYTHON="${VENV_CLIENT}/bin/python"
+else
+  PYTHON="$(command -v python3)"
+fi
+
+DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
+DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
+
+engine_is_healthy() {
+  curl --fail --silent --show-error --max-time 5 \
+    "http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
+}
+
+engine_stop_server() {
+  local tp="$1"
+  local dp="$2"
+  local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
+
+  if [[ -f "$pid_file" ]]; then
+    local pid
+    pid="$(cat "$pid_file")"
+    if kill -0 "$pid" 2>/dev/null; then
+      log "stopping vllm server pid=${pid} tp=${tp} dp=${dp}"
+      kill "$pid" 2>/dev/null || true
+      sleep 5
+      kill -9 "$pid" 2>/dev/null || true
+    fi
+    rm -f "$pid_file"
+  fi
+  docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
+  ACTIVE_ENGINE_SERVER_LOG=""
+  sleep 2
+}
+
+engine_build_server_args() {
+  local tp="$1"
+  local dp="$2"
+  local -a args=(
+    vllm serve "$MODEL_PATH"
+    --trust-remote-code
+    --kv-cache-dtype "$KV_CACHE_DTYPE"
+    --block-size "$BLOCK_SIZE"
+    --tensor-parallel-size "$tp"
+    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+    --max-model-len "$MAX_MODEL_LEN"
+    --max-num-seqs "$MAX_NUM_SEQS"
+    --no-enable-flashinfer-autotune
+    --host 0.0.0.0
+    --port "$ENGINE_PORT"
+  )
+  if (( dp > 1 )); then
+    args+=(--data-parallel-size "$dp")
+  fi
+  printf '%q ' "${args[@]}"
+}
+
+engine_start_server() {
+  local tp="$1"
+  local dp="$2"
+  local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
+  log "starting vllm server tp=${tp} dp=${dp}"
+  bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
+  if ! engine_is_healthy; then
+    log "ERROR: vllm health check failed tp=${tp} dp=${dp}"
+    return 1
+  fi
+  ACTIVE_ENGINE_SERVER_LOG="$(
+    find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
+      -name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \
+      -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
+  )"
+  log "vllm server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}"
+}
+
+engine_detect_oom() {
+  local detail_log="$1"
+  local tp="$2"
+  local dp="$3"
+  local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory'
+  local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
+  local -a logs=("$detail_log" "$outer_log")
+  if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
+    logs+=("$ACTIVE_ENGINE_SERVER_LOG")
+  fi
+  grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null
+}
+
+engine_run_bench() {
+  local isl="$1"
+  local osl="$2"
+  local concurrency="$3"
+  local num_prompts="$4"
+  local output_file="$5"
+  local warmup_requests
+  warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
+  local -a bench_args=(
+    --backend vllm
+    --host 127.0.0.1
+    --port "$ENGINE_PORT"
+    --dataset-name "$BENCH_DATASET_NAME"
+    --random-input-len "$isl"
+    --random-output-len "$osl"
+    --random-range-ratio "$RANDOM_RANGE_RATIO"
+    --num-prompts "$num_prompts"
+    --max-concurrency "$concurrency"
+    --request-rate 10000
+    --warmup-requests "$warmup_requests"
+    --output-file "$output_file"
+    --output-details
+    --disable-tqdm
+  )
+  if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
+    bench_args+=(--dataset-path "$DATASET_PATH")
+  else
+    bench_args+=(--tokenize-prompt)
+  fi
+
+  if [[ "$USE_DOCKER_CLIENT" == "1" ]]; then
+    local -a volume_args=(-v "${MODEL_PATH}:${MODEL_PATH}:ro" -v "${RESULT_BASE}:${RESULT_BASE}")
+    if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
+      volume_args+=(-v "${DATASET_PATH}:${DATASET_PATH}:ro")
+    fi
+    docker run --rm \
+      --network host \
+      "${volume_args[@]}" \
+      -e PYTHONUNBUFFERED=1 \
+      -e HF_HUB_OFFLINE=1 \
+      -e TRANSFORMERS_OFFLINE=1 \
+      "$DOCKER_CLIENT_IMAGE" \
+      python -m sglang.bench_serving "${bench_args[@]}"
+  else
+    "$PYTHON" -m sglang.bench_serving "${bench_args[@]}"
+  fi
+}
+
+export -f engine_run_bench
+export ENGINE_PORT MODEL_PATH RESULT_BASE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT
+export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
+
+adaptive_main "$@"

experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_adaptive_concurrency_add16.sh

@@ -0,0 +1,190 @@
+
+#!/usr/bin/env bash
+# Find the Total-TPS saturation concurrency for each vLLM-Ascend TP/DP/ISL/OSL
+# shape on Ascend 910C (GLM-5.2).
+#
+# Differences vs the H20 vLLM variant:
+#   - GPU monitor overridden to use npu-smi (shared library hardcodes nvidia-smi).
+#   - engine_run_bench runs the client inside the vllm-ascend container via
+#     `docker exec`. The official vllm-ascend image does NOT ship
+#     sglang.bench_serving; see envs/ASCEND_910C_ENV_SETUP.md for the client story
+#     (install bench_serving into the container or use an external sglang image).
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
+
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/adaptive_config.env"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/adaptive_bench_lib.sh"
+
+ENGINE="vllm"
+ENGINE_PORT="$VLLM_PORT"
+RESULT_BASE="${RESULT_BASE:-${SCRIPT_DIR}/adaptive_results}"
+ACTIVE_ENGINE_SERVER_LOG=""
+
+PYTHON="${PYTHON:-$(command -v python3)}"
+DOCKER_IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}"
+
+# Override the shared library's nvidia-smi GPU monitor with an npu-smi sampler
+# that emits the same CSV shape (timestamp, index, mem.used, mem.total, util).
+adaptive_start_gpu_monitor() {
+  local csv_path="$1"
+  mkdir -p "$(dirname "$csv_path")"
+  # 910C: delegate to the shared npu-smi sampler so parse_backend.py sees the
+  # same CSV shape as nvidia-smi (see scripts/common/npu_smi_sampler.py).
+  "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/npu_smi_sampler.py" "$GPU_MEM_SAMPLE_INTERVAL_S" > "$csv_path" 2>/dev/null &
+  echo $!
+}
+
+engine_is_healthy() {
+  curl --fail --silent --show-error --max-time 5 \
+    "http://127.0.0.1:${ENGINE_PORT}/health" >/dev/null 2>&1
+}
+
+engine_stop_server() {
+  local tp="$1"
+  local dp="$2"
+  local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
+  if [[ -f "$pid_file" ]]; then
+    local pid
+    pid="$(cat "$pid_file")"
+    if kill -0 "$pid" 2>/dev/null; then
+      log "stopping vllm-ascend server pid=${pid} tp=${tp} dp=${dp}"
+      kill "$pid" 2>/dev/null || true
+      sleep 5
+      kill -9 "$pid" 2>/dev/null || true
+    fi
+    rm -f "$pid_file"
+  fi
+  # Also remove the container by name in case the host process is gone but the
+  # container lingers (e.g. orphaned by a kill -9).
+  docker rm -f "${CONTAINER_NAME}_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
+  ACTIVE_ENGINE_SERVER_LOG=""
+  sleep 2
+}
+
+engine_build_server_args() {
+  local tp="$1"
+  local dp="$2"
+  local -a args=(
+    vllm serve "$MODEL_PATH"
+    --served-model-name "$SERVED_MODEL_NAME"
+    --trust-remote-code
+    --kv-cache-dtype "$KV_CACHE_DTYPE"
+    --block-size "$BLOCK_SIZE"
+    --tensor-parallel-size "$tp"
+    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+    --max-model-len "$MAX_MODEL_LEN"
+    --max-num-seqs "$MAX_NUM_SEQS"
+    --host 0.0.0.0
+    --port "$ENGINE_PORT"
+  )
+  if (( dp > 1 )); then
+    args+=(--data-parallel-size "$dp")
+  fi
+  printf '%q ' "${args[@]}"
+}
+
+engine_start_server() {
+  local tp="$1"
+  local dp="$2"
+  local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
+  log "starting vllm-ascend server tp=${tp} dp=${dp}"
+  bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" >> "$outer_log" 2>&1
+  if ! engine_is_healthy; then
+    log "ERROR: vllm-ascend health check failed tp=${tp} dp=${dp}"
+    return 1
+  fi
+  ACTIVE_ENGINE_SERVER_LOG="$(
+    find "${RUNTIME_BASE}/logs" -maxdepth 1 -type f \
+      -name "${EXPERIMENT}_vllm*tp${tp}_dp${dp}_*.log" \
+      -printf '%T@ %p\n' 2>/dev/null | sort -nr | head -n 1 | cut -d' ' -f2-
+  )"
+  log "vllm-ascend server healthy tp=${tp} dp=${dp} log=${ACTIVE_ENGINE_SERVER_LOG:-unknown}"
+}
+
+engine_detect_oom() {
+  local detail_log="$1"
+  local tp="$2"
+  local dp="$3"
+  local pattern='out of memory|OutOfMemory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory|NPU out of memory|acl.*memory|HBM'
+  local outer_log="${ADAPTIVE_LOG_DIR}/vllm_tp${tp}_dp${dp}.server.outer.log"
+  local -a logs=("$detail_log" "$outer_log")
+  if [[ -n "$ACTIVE_ENGINE_SERVER_LOG" ]]; then
+    logs+=("$ACTIVE_ENGINE_SERVER_LOG")
+  fi
+  grep -Eiq "$pattern" "${logs[@]}" 2>/dev/null
+}
+
+engine_run_bench() {
+  local isl="$1"
+  local osl="$2"
+  local concurrency="$3"
+  local num_prompts="$4"
+  local output_file="$5"
+  local container_name="${CONTAINER_NAME}_tp${tp}_dp${dp}"
+  local container_output="/tmp/bench_outputs/adaptive_$(basename "$output_file")"
+  if [[ "$output_file" == "/dev/null" ]]; then
+    container_output="/dev/null"
+  fi
+
+  local warmup_requests
+  warmup_requests="$(adaptive_warmup_request_count "$concurrency")"
+
+  local -a bench_args=(
+    --backend vllm
+    --host 127.0.0.1
+    --port "$ENGINE_PORT"
+    --model "$SERVED_MODEL_NAME"
+    --dataset-name "$BENCH_DATASET_NAME"
+    --random-input-len "$isl"
+    --random-output-len "$osl"
+    --random-range-ratio "$RANDOM_RANGE_RATIO"
+    --num-prompts "$num_prompts"
+    --max-concurrency "$concurrency"
+    --request-rate 10000
+    --warmup-requests "$warmup_requests"
+    --output-file "$container_output"
+    --output-details
+    --disable-tqdm
+  )
+  if [[ "$BENCH_DATASET_NAME" == "random" ]]; then
+    bench_args+=(--dataset-path "$DATASET_PATH")
+  else
+    bench_args+=(--tokenize-prompt)
+  fi
+
+  # Run the benchmark client inside the running vllm-ascend container via
+  # `docker exec`. NOTE: the official vllm-ascend image does NOT include
+  # sglang.bench_serving. If `docker exec ... sglang.bench_serving` fails with
+  # ModuleNotFoundError, either (a) pip install sglang into the container, or
+  # (b) set USE_DOCKER_CLIENT=1 and run an external sglang image against the
+  # host port. See envs/ASCEND_910C_ENV_SETUP.md.
+  docker exec "$container_name" mkdir -p /tmp/bench_outputs 2>/dev/null || true
+  docker exec "$container_name" \
+    env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
+    "$CONTAINER_PYTHON" -m sglang.bench_serving "${bench_args[@]}" || return $?
+
+  if [[ "$output_file" != "/dev/null" ]]; then
+    docker cp "$container_name:${container_output}" "$output_file" || return 1
+  fi
+}
+
+export -f engine_run_bench
+export ENGINE_PORT CONTAINER_NAME CONTAINER_PYTHON MODEL_PATH RESULT_BASE SERVED_MODEL_NAME
+export BENCH_DATASET_NAME DATASET_PATH RANDOM_RANGE_RATIO BENCH_WARMUP_MAX_REQUESTS PYTHON
+
+export SEARCH_START_CONCURRENCY=16
+export SEARCH_ADDEND=16
+export SEARCH_INITIAL_BACKOFF_CONCURRENCIES="8 1"
+export TTFT_GROUP_SKIP_MS="${TTFT_GROUP_SKIP_MS:-8000}"
+
+adaptive_main "$@"

experiments/910c/glm52_910c_vllm_tp_dp_matrix/run_bench.sh

@@ -0,0 +1,546 @@
+#!/usr/bin/env bash
+# TP×DP matrix benchmark for DeepSeek-V4-Flash on vLLM (Docker).
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
+
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/lib.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../../scripts/common/platform.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+# Export variables used inside functions that are called via bash -c subshells.
+export DATASET_PATH MODEL_PATH RESULT_BASE \
+  DOCKER_IMAGE DOCKER_CLIENT_IMAGE USE_DOCKER_CLIENT
+
+RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
+RESULT_BASE="${SCRIPT_DIR}/results"
+MATRIX_FILE="${MATRIX_FILE:-${SCRIPT_DIR}/matrix.json}"
+MATRIX_MODE="${MATRIX_MODE:-Y}"
+SCENARIO_TIMEOUT_S="${SCENARIO_TIMEOUT_S:-1800}"
+GPU_MEM_SAMPLE_INTERVAL_S="${GPU_MEM_SAMPLE_INTERVAL_S:-1}"
+DRY_RUN="${DRY_RUN:-0}"
+GRID_LIMIT="${GRID_LIMIT:-0}"
+
+if [[ -x "${VENV_CLIENT}/bin/python" ]]; then
+  PYTHON="${VENV_CLIENT}/bin/python"
+else
+  PYTHON="$(command -v python3)"
+fi
+
+DOCKER_IMAGE="${DOCKER_IMAGE:-vllm/vllm-openai:latest}"
+DOCKER_CLIENT_IMAGE="${DOCKER_CLIENT_IMAGE:-lmsysorg/sglang:latest}"
+
+log_dir_global="${RESULT_BASE}/${RUN_ID}/logs"
+mkdir -p "$log_dir_global"
+log_init "${log_dir_global}/orchestrator.log"
+
+log "experiment=${EXPERIMENT_NAME} run_id=${RUN_ID} platform=${PLATFORM} hardware=${HARDWARE}"
+log "matrix_mode=${MATRIX_MODE} matrix_file=${MATRIX_FILE} dry_run=${DRY_RUN} grid_limit=${GRID_LIMIT}"
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+is_server_healthy() {
+  curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${VLLM_PORT}/health" >/dev/null 2>&1
+}
+
+stop_server() {
+  local tp="$1"
+  local dp="$2"
+
+  local pid_file="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${tp}_dp${dp}.pid"
+  if [[ -f "$pid_file" ]]; then
+    local pid
+    pid="$(cat "$pid_file")"
+    if kill -0 "$pid" 2>/dev/null; then
+      log "stopping vllm server pid=${pid} (tp=${tp}, dp=${dp})"
+      kill "$pid" 2>/dev/null || true
+      sleep 5
+      kill -9 "$pid" 2>/dev/null || true
+    fi
+    rm -f "$pid_file"
+  fi
+
+  # Fallback: remove any Docker container started by this experiment.
+  docker rm -f "${EXPERIMENT}_vllm_tp${tp}_dp${dp}" >/dev/null 2>&1 || true
+
+  # Fallback: kill any vllm serve processes for this model.
+  pkill -9 -f "vllm serve.*${MODEL_NAME}" 2>/dev/null || true
+  pkill -9 -f "vllm serve.*${MODEL_PATH}" 2>/dev/null || true
+  sleep 2
+}
+
+build_server_args() {
+  local tp="$1"
+  local dp="$2"
+
+  local args=(
+    "vllm serve" "$MODEL_PATH"
+    --trust-remote-code
+    --kv-cache-dtype "$KV_CACHE_DTYPE"
+    --block-size "$BLOCK_SIZE"
+    --tensor-parallel-size "$tp"
+    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+    --max-model-len "$MAX_MODEL_LEN"
+    --max-num-seqs "$MAX_NUM_SEQS"
+    --no-enable-flashinfer-autotune
+    --host 0.0.0.0
+    --port "$VLLM_PORT"
+  )
+  if [[ "$dp" -gt 1 ]]; then
+    args+=(
+      --data-parallel-size "$dp"
+    )
+  fi
+  printf '%s ' "${args[@]}"
+}
+
+start_server() {
+  local tp="$1"
+  local dp="$2"
+
+  log "starting vllm server tp=${tp} dp=${dp}"
+  bash "${SCRIPT_DIR}/start_vllm_dp.sh" "$tp" "$dp" \
+    >> "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log" 2>&1
+
+  if ! is_server_healthy; then
+    log "error: vllm server tp=${tp} dp=${dp} failed health check on port ${VLLM_PORT}"
+    return 1
+  fi
+  log "vllm server tp=${tp} dp=${dp} is healthy on port ${VLLM_PORT}"
+}
+
+restart_server() {
+  local tp="$1"
+  local dp="$2"
+  log "restarting vllm server tp=${tp} dp=${dp} after non-OOM failure"
+  stop_server "$tp" "$dp"
+  sleep 10
+  start_server "$tp" "$dp"
+}
+
+run_bench_serving() {
+  # Run sglang.bench_serving either natively or inside the SGLang Docker image.
+  if [[ "${USE_DOCKER_CLIENT:-1}" == "1" ]]; then
+    local vol_args=()
+    vol_args+=("-v" "${MODEL_PATH}:${MODEL_PATH}:ro")
+    if [[ -n "${DATASET_PATH:-}" && -f "${DATASET_PATH}" ]]; then
+      vol_args+=("-v" "${DATASET_PATH}:${DATASET_PATH}:ro")
+    fi
+    vol_args+=("-v" "${RESULT_BASE}:${RESULT_BASE}")
+    docker run --rm \
+      --network host \
+      "${vol_args[@]}" \
+      -e PYTHONUNBUFFERED=1 \
+      -e HF_HUB_OFFLINE=1 \
+      -e TRANSFORMERS_OFFLINE=1 \
+      -e HF_DATASETS_OFFLINE=1 \
+      "${DOCKER_CLIENT_IMAGE}" \
+      python -m sglang.bench_serving "$@"
+  else
+    "$PYTHON" -m sglang.bench_serving "$@"
+  fi
+}
+export -f run_bench_serving
+
+run_warmup() {
+  local input_len="$1"
+  local output_len="$2"
+
+  log "warming up (input=${input_len}, output=${output_len}, num=1)"
+  bash -c '
+    run_bench_serving \
+      --backend vllm \
+      --host 127.0.0.1 \
+      --port "'"$VLLM_PORT"'" \
+      --dataset-name random \
+      --dataset-path "'"$DATASET_PATH"'" \
+      --random-input-len "'"$input_len"'" \
+      --random-output-len "'"$output_len"'" \
+      --num-prompts 1 \
+      --max-concurrency 1 \
+      --request-rate 10000 \
+      --output-file /dev/null \
+      --output-details \
+      >> "'"${log_dir_global}/warmup.log"'" 2>&1
+  '
+  log "warmup completed"
+}
+
+scenario_already_completed() {
+  local output_file="$1"
+  local expected="$2"
+  [[ -s "$output_file" ]] || return 1
+  local completed
+  completed="$("$PYTHON" -c "
+import json, sys
+path = sys.argv[1]
+try:
+    with open(path, 'r', encoding='utf-8') as f:
+        for line in f:
+            line = line.strip()
+            if line:
+                data = json.loads(line)
+                print(data.get('completed', 0))
+                break
+except Exception:
+    print(0)
+" "$output_file")"
+  [[ "${completed:-0}" -ge "$expected" ]]
+}
+
+scenario_already_processed() {
+  local result_root="$1"
+  local scenario_name="$2"
+  local json_path="${result_root}/results.json"
+  [[ -f "$json_path" ]] || return 1
+  "$PYTHON" -c "
+import json, sys
+path, name = sys.argv[1], sys.argv[2]
+try:
+    with open(path, 'r', encoding='utf-8') as f:
+        data = json.load(f)
+    for s in data.get('scenarios', []):
+        if s.get('name') == name:
+            if s.get('status') or s.get('metrics', {}).get('success', 0) > 0:
+                sys.exit(0)
+except Exception:
+    pass
+sys.exit(1)
+" "$json_path" "$scenario_name"
+}
+
+detect_oom() {
+  local detail_log="$1"
+  local server_outer_log="$2"
+  local pattern='CUDA out of memory|torch\.OutOfMemoryError|OutOfMemory|out of memory|OOM|RESOURCE_EXHAUSTED|Failed to allocate memory|NPU out of memory|acl.*memory|HBM'
+  if grep -Eiq "$pattern" "$detail_log" "$server_outer_log" 2>/dev/null; then
+    return 0
+  fi
+  return 1
+}
+
+start_gpu_monitor() {
+  local csv_path="$1"
+  mkdir -p "$(dirname "$csv_path")"
+  # 910C: use npu-smi via a standalone python sampler that emits the same CSV
+  # shape as nvidia-smi, so parse_backend.py needs no changes.
+  "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/npu_smi_sampler.py" "$GPU_MEM_SAMPLE_INTERVAL_S" > "$csv_path" 2>/dev/null &
+  echo $!
+}
+
+stop_gpu_monitor() {
+  local pid="$1"
+  if kill -0 "$pid" 2>/dev/null; then
+    kill "$pid" 2>/dev/null || true
+    sleep 1
+    kill -9 "$pid" 2>/dev/null || true
+  fi
+}
+
+append_scenario_record() {
+  local result_root="$1"
+  local json_path="$result_root/results.json"
+  shift
+  local scenario_json
+  scenario_json="$("$PYTHON" -c "
+import json, sys
+pairs = [a.split('=', 1) for a in sys.argv[1:]]
+d = {}
+for k, v in pairs:
+    try:
+        d[k] = json.loads(v)
+    except json.JSONDecodeError:
+        d[k] = v
+print(json.dumps(d, ensure_ascii=False))
+" "$@")"
+  PYTHON="$PYTHON" append_scenario_to_json "$json_path" "$scenario_json"
+}
+
+record_skipped_csv() {
+  local csv_path="$1"
+  shift
+  # Args: key=value
+  local row
+  row="$("$PYTHON" -c "
+import csv, json, sys, io
+pairs = [a.split('=', 1) for a in sys.argv[1:]]
+d = {}
+for k, v in pairs:
+    try:
+        d[k] = json.loads(v)
+    except json.JSONDecodeError:
+        d[k] = v
+buf = io.StringIO()
+writer = csv.DictWriter(buf, fieldnames=['engine','tp','dp','mark','isl','osl','concurrency','status','reason','detail_log'], extrasaction='ignore')
+writer.writerow(d)
+print(buf.getvalue().strip())
+" "$@")"
+  echo "$row" >> "$csv_path"
+}
+
+skip_remaining_scenarios() {
+  local result_root="$1"
+  local scenario_tsv="$2"
+  local start_index="$3"
+  local status="$4"
+  local reason="$5"
+  local tp="$6"
+  local dp="$7"
+  local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
+
+  local i=0
+  tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl osl conc num; do
+    if (( i < start_index )); then
+      i=$((i + 1))
+      continue
+    fi
+    i=$((i + 1))
+    local sname="c${conc}_i${isl}_o${osl}"
+    if scenario_already_processed "$result_root" "$sname"; then
+      continue
+    fi
+    append_scenario_record "$result_root" \
+      "name=${sname}" \
+      "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+      "status=\"${status}\"" \
+      "note=\"${reason}\""
+    record_skipped_csv "$skipped_csv" \
+      "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=${status}" "reason=${reason}"
+  done
+}
+
+# ---------------------------------------------------------------------------
+# Per-configuration runner
+# ---------------------------------------------------------------------------
+
+run_parallel_config() {
+  local tp="$1"
+  local dp="$2"
+  local config_label="tp${tp}_dp${dp}"
+  local result_root="${RESULT_BASE}/${RUN_ID}/${config_label}"
+  local raw_dir="${result_root}/raw_outputs"
+  local gpu_log_dir="${result_root}/gpu_logs"
+  local phase_log_dir="${result_root}/logs"
+  mkdir -p "$raw_dir" "$gpu_log_dir" "$phase_log_dir"
+
+  log "===== ${config_label} START ====="
+
+  # Generate scenario list for this config.
+  local scenario_tsv="${result_root}/scenarios.tsv"
+  "$PYTHON" "${SCRIPT_DIR}/generate_scenarios.py" \
+    --matrix "$MATRIX_FILE" \
+    --mode "$MATRIX_MODE" \
+    > "$scenario_tsv"
+  local total_scenarios
+  total_scenarios="$(tail -n +2 "$scenario_tsv" | wc -l)"
+  log "generated ${total_scenarios} scenarios for ${config_label}"
+
+  # Write metadata.
+  ensure_result_root "$result_root"
+  write_metadata_json \
+    "${result_root}/results.json" \
+    "${EXPERIMENT_NAME}_${config_label}" \
+    "$RUN_ID" \
+    "$MODEL_PATH" \
+    "vllm" \
+    "vllm" \
+    "$HARDWARE" \
+    "$ACCELERATOR" \
+    "$CHIP" \
+    "experiments/${EXPERIMENT_NAME}/run_bench.sh" \
+    "$DOCKER_IMAGE" \
+    "H20 vLLM TP×DP matrix for DeepSeek-V4-Flash"
+
+  local server_args_str
+  server_args_str="$(build_server_args "$tp" "$dp")"
+  jq --arg tp "$tp" --arg dp "$dp" --arg cuda "$CUDA_VISIBLE_DEVICES" --arg args "$server_args_str" \
+    '.config = {
+       "tp": ($tp | tonumber),
+       "dp": ($dp | tonumber),
+       "cuda_visible_devices": $cuda,
+       "backend": "vllm",
+       "server_start_script": "experiments/'${EXPERIMENT_NAME}'/start_vllm_dp.sh",
+       "server_args": $args
+     }' "${result_root}/results.json" > "${result_root}/results.json.tmp" && \
+    mv "${result_root}/results.json.tmp" "${result_root}/results.json"
+
+  if [[ "$DRY_RUN" == "1" ]]; then
+    log "DRY_RUN: would start server with args: ${server_args_str}"
+    local line
+    tail -n +2 "$scenario_tsv" | while IFS=$'\t' read -r mark isl osl conc num; do
+      log "DRY_RUN: ${config_label} scenario mark=${mark} c=${conc} i=${isl} o=${osl} n=${num}"
+    done
+    log "===== ${config_label} DONE (dry run) ====="
+    return 0
+  fi
+
+  # Initialize skipped_after_oom.csv for this run.
+  local skipped_csv="${RESULT_BASE}/${RUN_ID}/skipped_after_oom.csv"
+  if [[ ! -f "$skipped_csv" ]]; then
+    echo "engine,tp,dp,mark,isl,osl,concurrency,status,reason,detail_log" > "$skipped_csv"
+  fi
+
+  # Start server once for this TP×DP config.
+  if ! start_server "$tp" "$dp"; then
+    log "ERROR: ${config_label} failed to start; skipping all scenarios"
+    skip_remaining_scenarios "$result_root" "$scenario_tsv" 0 "SKIPPED_SERVICE_START_FAILED" "service failed to start" "$tp" "$dp"
+    log "===== ${config_label} DONE ====="
+    return 0
+  fi
+
+  # Warmup with a small prompt before the first scenario.
+  run_warmup 1024 128 || true
+
+  # Read scenarios into an array so we can skip remaining entries on failure.
+  local -a scenarios=()
+  while IFS= read -r line; do
+    scenarios+=("$line")
+  done < <(tail -n +2 "$scenario_tsv")
+
+  local i mark isl osl conc num
+  local output_file detail_log gpu_csv sname bench_rc
+  for (( i = 0; i < ${#scenarios[@]}; i++ )); do
+    IFS=$'\t' read -r mark isl osl conc num <<< "${scenarios[$i]}"
+
+    if [[ "$GRID_LIMIT" -gt 0 && "$i" -ge "$GRID_LIMIT" ]]; then
+      log "GRID_LIMIT=${GRID_LIMIT} reached; skipping remaining scenarios"
+      skip_remaining_scenarios "$result_root" "$scenario_tsv" "$i" "SKIPPED_GRID_LIMIT" "GRID_LIMIT reached" "$tp" "$dp"
+      break
+    fi
+
+    sname="c${conc}_i${isl}_o${osl}"
+    output_file="${raw_dir}/vllm_main_${conc}_${isl}_${osl}.jsonl"
+    detail_log="${phase_log_dir}/vllm_${config_label}_${sname}.log"
+    gpu_csv="${gpu_log_dir}/gpu_mem_${conc}_${isl}_${osl}.csv"
+
+    if scenario_already_completed "$output_file" "$num" || scenario_already_processed "$result_root" "$sname"; then
+      log "skipping already-processed ${config_label} scenario: ${sname}"
+      continue
+    fi
+
+    log "running ${config_label} scenario: mark=${mark} c=${conc} i=${isl} o=${osl} n=${num}"
+
+    local gpu_pid
+    gpu_pid="$(start_gpu_monitor "$gpu_csv")"
+
+    bench_rc=0
+    timeout "$SCENARIO_TIMEOUT_S" bash -c '
+      run_bench_serving \
+        --backend vllm \
+        --host 127.0.0.1 \
+        --port "'"$VLLM_PORT"'" \
+        --dataset-name random \
+        --dataset-path "'"$DATASET_PATH"'" \
+        --random-input-len "'"$isl"'" \
+        --random-output-len "'"$osl"'" \
+        --num-prompts "'"$num"'" \
+        --max-concurrency "'"$conc"'" \
+        --request-rate 10000 \
+        --output-file "'"$output_file"'" \
+        --output-details \
+        > "'"$detail_log"'" 2>&1
+    ' || bench_rc=$?
+
+    stop_gpu_monitor "$gpu_pid"
+
+    if [[ "$bench_rc" -eq 0 ]]; then
+      log "finished ${config_label} scenario: output=${output_file}"
+      append_scenario_record "$result_root" \
+        "name=${sname}" \
+        "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+        "status=\"completed\"" \
+        "note=\"benchmark finished successfully\""
+      continue
+    fi
+
+    # Failure handling.
+    if detect_oom "$detail_log" "${log_dir_global}/vllm_tp${tp}_dp${dp}.server.outer.log"; then
+      log "ERROR: ${config_label} scenario ${sname} triggered OOM; stopping config"
+      append_scenario_record "$result_root" \
+        "name=${sname}" \
+        "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+        "status=\"OOM\"" \
+        "note=\"detected CUDA out-of-memory\""
+      record_skipped_csv "$skipped_csv" \
+        "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=OOM" "reason=detected CUDA out-of-memory" "detail_log=${detail_log}"
+      stop_server "$tp" "$dp"
+      skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_AFTER_OOM" "previous case OOM" "$tp" "$dp"
+      break
+    fi
+
+    log "ERROR: ${config_label} scenario ${sname} failed (rc=${bench_rc}); see ${detail_log}"
+    if [[ "$mark" == "P" ]]; then
+      log "optional (P) scenario failed; recording as skipped and continuing"
+      append_scenario_record "$result_root" \
+        "name=${sname}" \
+        "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+        "status=\"skipped_optional\"" \
+        "note=\"optional scenario failed (rc=${bench_rc})\""
+      record_skipped_csv "$skipped_csv" \
+        "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=skipped_optional" "reason=optional scenario failed (rc=${bench_rc})" "detail_log=${detail_log}"
+      continue
+    fi
+
+    # Mandatory scenario failed but not OOM: try to restart the server.
+    if restart_server "$tp" "$dp"; then
+      run_warmup 1024 128 || true
+      log "resuming ${config_label} after server restart"
+      continue
+    fi
+
+    log "ERROR: ${config_label} server restart failed; skipping remaining scenarios"
+    append_scenario_record "$result_root" \
+      "name=${sname}" \
+      "config=$(jq -n --arg phase main --argjson c "$conc" --argjson i "$isl" --argjson o "$osl" --arg dataset random --argjson n "$num" '{phase: $phase, concurrency: $c, input_len: $i, output_len: $o, dataset: $dataset, num_prompts: $n}')" \
+      "status=\"FAILED\"" \
+      "note=\"scenario failed and server restart failed (rc=${bench_rc})\""
+    record_skipped_csv "$skipped_csv" \
+      "engine=vllm" "tp=${tp}" "dp=${dp}" "mark=${mark}" "isl=${isl}" "osl=${osl}" "concurrency=${conc}" "status=FAILED" "reason=scenario failed and server restart failed" "detail_log=${detail_log}"
+    skip_remaining_scenarios "$result_root" "$scenario_tsv" "$((i + 1))" "SKIPPED_RESTART_FAILED" "server restart failed" "$tp" "$dp"
+    break
+  done
+
+  stop_server "$tp" "$dp"
+
+  # Parse results.
+  log "parsing ${config_label} results"
+  "$PYTHON" "${SCRIPT_DIR}/../../../scripts/common/parse_backend.py" "$result_root" --backend vllm \
+    >> "${phase_log_dir}/parse.log" 2>&1 || {
+      log "WARNING: parser failed for ${config_label}; see ${phase_log_dir}/parse.log"
+    }
+
+  log "===== ${config_label} DONE ====="
+}
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+# Cleanup any leftovers.
+for cfg in "${PARALLEL_CONFIGS[@]}"; do
+  read -r tp dp <<< "$cfg"
+  stop_server "$tp" "$dp"
+done
+
+# Run each parallel configuration.
+for cfg in "${PARALLEL_CONFIGS[@]}"; do
+  read -r tp dp <<< "$cfg"
+  run_parallel_config "$tp" "$dp"
+done
+
+# Generate cross-configuration comparison.
+log "generating comparison report"
+"$PYTHON" "${SCRIPT_DIR}/compare.py" \
+  --run-root "${RESULT_BASE}/${RUN_ID}" \
+  --output "${RESULT_BASE}/${RUN_ID}/comparison.md" \
+  >> "${log_dir_global}/compare.log" 2>&1 || {
+    log "WARNING: comparison script failed; see ${log_dir_global}/compare.log"
+  }
+
+log "all results saved to ${RESULT_BASE}/${RUN_ID}"

experiments/910c/glm52_910c_vllm_tp_dp_matrix/start_vllm_docker.sh

@@ -0,0 +1,107 @@
+#!/usr/bin/env bash
+# Start vLLM-Ascend server in Docker for a given TP×DP configuration on 910C.
+# Usage: start_vllm_docker.sh <TP> <DP>
+#
+# Relies on the Ascend Docker Runtime being the default docker runtime on this
+# host (see /etc/docker/daemon.json). NPU dies are injected via the
+# ASCEND_VISIBLE_DEVICES env var; no --gpus / --device flags are needed.
+set -e
+
+TP="${1}"
+DP="${2}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+RUNTIME_BASE="${RUNTIME_BASE:-${SCRIPT_DIR}/runtime}"
+mkdir -p "${RUNTIME_BASE}/logs" "${RUNTIME_BASE}/tmp"
+
+IMAGE="${DOCKER_IMAGE:-vllm-ascend:glm5.2-a3-openeuler}"
+PORT="${VLLM_PORT:-30050}"
+NAME="${CONTAINER_NAME:-${EXPERIMENT}_vllm_tp${TP}_dp${DP}}"
+NAME="${NAME}_tp${TP}_dp${DP}"
+PID_FILE="${RUNTIME_BASE}/${EXPERIMENT}_vllm_tp${TP}_dp${DP}.pid"
+
+LOG="${RUNTIME_BASE}/logs/${EXPERIMENT}_vllm_docker_tp${TP}_dp${DP}_$(date +%Y%m%d_%H%M%S).log"
+rm -f "$PID_FILE"
+
+# Clean up any stale container with the same name.
+docker rm -f "$NAME" >/dev/null 2>&1 || true
+
+# vLLM-Ascend launch args. Differences vs NVIDIA vLLM:
+#   - no --no-enable-flashinfer-autotune (Ascend uses its own attention path)
+#   - --kv-cache-dtype may need to be fp16 if the image rejects fp8 on 910C
+SERVER_ARGS=(
+  "$MODEL_PATH"
+  --served-model-name "$SERVED_MODEL_NAME"
+  --trust-remote-code
+  --kv-cache-dtype "$KV_CACHE_DTYPE"
+  --block-size "$BLOCK_SIZE"
+  --tensor-parallel-size "$TP"
+  --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+  --max-model-len "$MAX_MODEL_LEN"
+  --max-num-seqs "$MAX_NUM_SEQS"
+  --host 0.0.0.0
+  --port "$PORT"
+)
+
+if [[ "$DP" -gt 1 ]]; then
+  SERVER_ARGS+=(--data-parallel-size "$DP")
+fi
+
+SERVER_ARGS_STR="vllm serve ${SERVER_ARGS[*]}"
+
+echo "=== Starting vLLM-Ascend server in Docker (TP=${TP}, DP=${DP}) ==="
+echo "Image: $IMAGE"
+echo "Model: $MODEL_PATH"
+echo "Container name: $NAME"
+echo "Host port: $PORT"
+echo "ASCEND_VISIBLE_DEVICES: ${ASCEND_VISIBLE_DEVICES}"
+echo "Command: $SERVER_ARGS_STR"
+echo "Log: $LOG"
+
+# Run docker in the foreground; nohup backgrounds it and the host PID lets the
+# adaptive search stop the container by killing the process (the container has
+# --rm so it self-cleans). The Ascend Docker Runtime is the default runtime, so
+# no --runtime flag is required.
+nohup docker run --rm \
+  --name "$NAME" \
+  --ipc host \
+  --shm-size 16g \
+  --network host \
+  --ulimit memlock=-1 \
+  -v "${MODEL_PATH}:${MODEL_PATH}:ro" \
+  -v "${RUNTIME_BASE}/tmp:/tmp" \
+  -e ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES}" \
+  -e PYTORCH_NPU_ALLOC_CONF=expandable_segments:True \
+  -e PYTHONUNBUFFERED=1 \
+  -e HF_HUB_OFFLINE=1 \
+  -e TRANSFORMERS_OFFLINE=1 \
+  "$IMAGE" \
+  vllm serve "${SERVER_ARGS[@]}" \
+  > "$LOG" 2>&1 &
+
+PID=$!
+echo $PID > "$PID_FILE"
+echo "PID: $PID"
+echo "Waiting for health on port ${PORT}..."
+
+for i in $(seq 1 240); do
+  if curl --fail --silent --show-error --max-time 5 "http://127.0.0.1:${PORT}/health" >/dev/null 2>&1; then
+    echo "vLLM-Ascend server is ready at http://127.0.0.1:${PORT}"
+    echo "Log: $LOG"
+    exit 0
+  fi
+  if ! kill -0 $PID 2>/dev/null; then
+    echo "ERROR: Docker vLLM-Ascend server exited early"
+    tail -200 "$LOG"
+    exit 1
+  fi
+  echo "Waiting... ($i/240)"
+  sleep 5
+done
+
+echo "ERROR: vLLM-Ascend server not healthy after 240 retries"
+tail -200 "$LOG"
+exit 1

experiments/910c/glm52_910c_vllm_tp_dp_matrix/start_vllm_dp.sh

@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+# Start vLLM-Ascend server for a given TP×DP configuration on 910C.
+# Usage: start_vllm_dp.sh <TP> <DP>
+#
+# 910C only supports the Docker path (vllm-ascend runs inside a container with
+# the Ascend Docker Runtime). USE_DOCKER=0 is not supported on this platform;
+# set USE_DOCKER=1 (the default) or call start_vllm_docker.sh directly.
+set -e
+
+TP="${1}"
+DP="${2}"
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+if [[ "${USE_DOCKER:-1}" == "1" ]]; then
+  exec "${SCRIPT_DIR}/start_vllm_docker.sh" "$@"
+fi
+
+echo "ERROR: USE_DOCKER=0 (native vllm-ascend) is not supported on 910C." >&2
+echo "       vllm-ascend requires the Ascend Docker Runtime; set USE_DOCKER=1." >&2
+exit 1

platforms/README.md

@@ -1,3 +1,4 @@
+
 # Platform Configurations
 
 Each `.env` file in this directory describes one accelerator platform.
@@ -18,6 +19,7 @@ PLATFORM=kunlun_p800 bash experiments/dsv4_p800_sglang/run_bench.sh
 
 | File | Chip/Accelerator | Engine | Notes |
 |---|---|---|---|
+| `ascend_910c.env` | Huawei Ascend 910C | `vllm-ascend` | Docker-based，8 卡 16 die / 64GB HBM；Ascend Docker Runtime 为默认 runtime，见 `envs/ASCEND_910C_ENV_SETUP.md` |
 | `kunlun_p800.env` | Kunlun P800 XPU | `sglang-xpu` | Docker-based SGLang serving image |
 | `nvidia_h200.env` | NVIDIA H200 | `vllm-dspark` | Native host virtual environments |
 | `nvidia_h20.env` | NVIDIA H20 | `vllm` / `sglang` | Docker-based（vllm-openai / sglang 官方镜像） |

platforms/ascend_910c.env

@@ -0,0 +1,42 @@
+# Platform configuration for Huawei Ascend 910C NPU
+# Source this file via scripts/common/platform.sh
+#
+# Host reference (910c.1 / NPU-NODE61):
+#   - openEuler 22.03 SP4, aarch64, 8x Ascend910 (16 dies, 64GB HBM/die)
+#   - CANN 9.0.0 + ascend-toolkit, driver 25.5.2
+#   - Ascend Docker Runtime is the default docker runtime (see /etc/docker/daemon.json)
+#   - User must be in HwHiAiUser + docker groups to access NPU devices and docker.
+
+CHIP="ascend_910c"
+ACCELERATOR="Huawei Ascend 910C"
+HARDWARE="8x Ascend910 (16 dies, 64GB HBM/die)"
+ENGINE="vllm-ascend"
+
+# Container runtime. vllm-ascend images ship as tarballs on this host and must
+# be `docker load`-ed first (see envs/ASCEND_910C_ENV_SETUP.md). The exact tag
+# is overridden per-experiment in config.env; this is the default for new ones.
+# Known-good tarballs on /mnt/models:
+#   vllm-ascend-v0.23.0rc1-a3-openeuler.tar      -> general vllm-ascend
+#   vllm-ascend-glm5.2-a3-openeuler.tar          -> GLM5.2-tuned variant
+DOCKER_IMAGE="${DOCKER_IMAGE:-quay.io/ascend/vllm-ascend:latest}"
+CONTAINER_NAME="${CONTAINER_NAME:-vllm-ascend-910c}"
+
+# Device selection. Ascend Docker Runtime maps NPU dies via ASCEND_VISIBLE_DEVICES
+# (0-based NPU card id). 8 cards => 0..7. Each card exposes 2 dies; vllm-ascend
+# addresses dies individually so TP can go up to 16.
+DEVICE_SELECT_ENV="ASCEND_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
+ASCEND_VISIBLE_DEVICES="${ASCEND_VISIBLE_DEVICES:-0,1,2,3,4,5,6,7}"
+
+# Default serving port and model root on the host.
+# /mnt/models holds GLM-5.2-{w4a8c8,w8a8} and vllm-ascend image tarballs.
+DEFAULT_PORT="30050"
+MODEL_ROOT="/mnt/models"
+
+# Python interpreter inside the vllm-ascend container (used to run the
+# benchmark client via `docker exec`). Standard path in the official image.
+CONTAINER_PYTHON="${CONTAINER_PYTHON:-/usr/local/bin/python}"
+
+# CANN / toolkit paths on the host (informational; the container bundles its
+# own CANN). Referenced by env setup docs, not by run scripts directly.
+HOST_CANN_ROOT="/usr/local/Ascend/ascend-toolkit/latest"
+HOST_DRIVER_VERSION="25.5.2"

scripts/common/npu_smi_sampler.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+"""Sample npu-smi info and emit nvidia-smi-compatible CSV rows.
+
+Usage: npu_smi_sampler.py <interval_seconds>
+Emits a header row first, then one row per chip every interval seconds.
+Output columns match nvidia-smi --query-gpu=timestamp,index,memory.used,memory.total,utilization.gpu --format=csv
+"""
+import sys, re, datetime, time
+
+def sample():
+    import subprocess
+    try:
+        out = subprocess.run(["npu-smi", "info"], capture_output=True, text=True, timeout=10).stdout
+    except Exception:
+        return []
+    ts = datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S")
+    rows = []
+    idx = None
+    for line in out.splitlines():
+        m = re.match(r"\|\s*(\d+)\s+Ascend", line)
+        if m:
+            idx = m.group(1)
+            continue
+        if idx is None:
+            continue
+        hbm = re.search(r"(\d+)\s*/\s*(\d+)", line)
+        util = re.search(r"(\d+)\s*%", line)
+        if hbm:
+            u, t = hbm.group(1), hbm.group(2)
+            r = util.group(1) if util else "0"
+            rows.append(f"{ts}, {idx}, {u} MiB, {t} MiB, {r} %")
+            idx = None
+    return rows
+
+def main():
+    interval = float(sys.argv[1]) if len(sys.argv) > 1 else 1.0
+    print("timestamp, index, memory.used [MiB], memory.total [MiB], utilization.gpu [%]")
+    sys.stdout.flush()
+    while True:
+        for row in sample():
+            print(row)
+        sys.stdout.flush()
+        time.sleep(interval)
+
+if __name__ == "__main__":
+    main()

scripts/common/platform.sh

@@ -1,3 +1,4 @@
+
 #!/usr/bin/env bash
 # Platform loader.
 # Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/platform.sh"
@@ -19,6 +20,18 @@ if [[ -z "${PLATFORM:-}" ]]; then
     PLATFORM="kunlun_p800"
   elif lspci 2>/dev/null | grep -qiE "XPU|Kunlun"; then
     PLATFORM="kunlun_p800"
+  elif command -v npu-smi >/dev/null 2>&1; then
+    # Huawei Ascend NPU. npu-smi info lists cards like "Ascend910" / "910C".
+    _NPU_SMI_OUT=$(npu-smi info 2>/dev/null || true)
+    if [[ -n "$_NPU_SMI_OUT" ]] && grep -qiE "Ascend910|Ascend 910|910C|910c" <<< "$_NPU_SMI_OUT"; then
+      PLATFORM="ascend_910c"
+    else
+      PLATFORM="ascend_910c"
+    fi
+    unset _NPU_SMI_OUT
+  elif lspci 2>/dev/null | grep -qiE "Huawei.*a12d|Huawei.*a12e|d100|d80"; then
+    # Ascend 910 series exposes Huawei System-peripheral PCI IDs (a12d/a12e).
+    PLATFORM="ascend_910c"
   elif command -v nvidia-smi >/dev/null 2>&1; then
     _GPU_NAME=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -n 1 || true)
     if [[ -n "$_GPU_NAME" ]] && grep -qiE "RTX 6000D|PRO 6000D" <<< "$_GPU_NAME"; then