refactor: experiments/ + platforms/ layout and P800 SGLang support

- Restructure repo around experiments/<name>/ and platforms/<chip>.env. - Add shared scripts under scripts/common/ for platform/server/bench-client logic. - Add Kunlun P800 platform config and runtime patches. - Add dsv4_p800_sglang experiment with INT8 smoke-test support. - Update BENCHMARK_WORKFLOW.md and README.md with chip/engine recording rules. - Add scripts/analysis/compare_experiments.py for cross-experiment comparison. - Ignore experiments/*/results/ raw output directories by default.
2026-07-08 05:15:59 +00:00 · 2026-07-08 05:15:59 +00:00 · 4ef6543c97
commit 4ef6543c97
parent 2c332ff712
25 changed files with 7732 additions and 50 deletions
--- a/.gitignore
+++ b/.gitignore
@ -23,5 +23,8 @@ datasets/
 # 原始请求级输出（方案 A：极简版，不存原始 jsonl）
 bench_results/**/raw_outputs/

+# 实验级原始结果目录（report.md / results.json 可单独保留，默认忽略整个目录）
+experiments/*/results/
+
 # 无关项目
 loomeval_yy/
--- a/BENCHMARK_WORKFLOW.md
+++ b/BENCHMARK_WORKFLOW.md
@ -4,20 +4,29 @@

 ```
 /data/user1/yy/
-├── scripts/                    # all benchmark/orchestrator/utility scripts
-│   ├── benchmark_dspark_0707/  # DSpark benchmark suite
-│   ├── benchmark_dsv4_backend_comparison.sh
-│   ├── start_dsv4_dspark_8card.sh
-│   ├── start_sglang_dsv4_8card.sh
+├── platforms/                  # chip/accelerator platform configs
+│   ├── kunlun_p800.env
+│   ├── nvidia_h200.env
+│   └── patches/kunlun_p800/    # runtime patches required by some images
+├── scripts/                    # shared benchmark/orchestrator/utility scripts
+│   ├── common/                 # reusable components (lib.sh, platform.sh, ...)
+│   ├── analysis/               # cross-experiment comparison tools
+│   ├── benchmark_dspark_0707/  # legacy DSpark benchmark suite
 │   └── ...
-├── bench_results/              # all benchmark outputs and reports
-│   ├── dsv4_backend_comparison_20260707/
-│   │   ├── raw_outputs/        # JSONL raw outputs
-│   │   ├── logs/               # per-run logs
-│   │   └── README.md           # output manifest + provenance
+├── experiments/                # experiment-centric directories (preferred)
+│   └── dsv4_p800_sglang/
+│       ├── README.md
+│       ├── config.env          # experiment-level configuration
+│       ├── start_server.sh
+│       ├── run_bench.sh
+│       ├── parse_results.py
+│       └── results/
+│           └── 20260708-XXXXXX/
+│               ├── report.md
+│               ├── results.json
+│               └── logs/
+├── bench_results/              # legacy benchmark outputs (read-only history)
 │   ├── dspark_grid_20260707-132641/
-│   ├── dspark_st_comparison_20260707-150649/
-│   ├── eagle_grid/
 │   └── ...
 ├── logs/                       # server logs (stdout/stderr from start scripts)
 ├── datasets/                   # benchmark datasets
@ -26,16 +35,16 @@

 ## Rules

-1. **Scripts live in `scripts/` only.**
-   - Group related scripts into subdirectories, e.g. `scripts/benchmark_dspark_0707/`.
-   - Each script group should have its own `README.md` listing scripts, purpose, and outputs.
+1. **Experiments are the primary organization unit.**
+   - Each experiment lives under `experiments/<experiment_name>/` and contains its scripts, configuration, and results.
+   - Shared orchestration code lives in `scripts/common/`; do not copy server start / health check logic into every experiment.
+   - Legacy experiments may remain under `scripts/<group>/` with outputs in `bench_results/`, but new work should use `experiments/`.

-2. **Benchmark outputs live in `bench_results/` only.**
-   - Never leave `.jsonl`, `.json`, `.log`, or `.md` reports in the project root.
-   - Each benchmark run gets its own directory: `bench_results/<experiment>_<timestamp>/`.
-   - Raw outputs go in `raw_outputs/`.
-   - Logs go in `logs/`.
-   - Reports (e.g. `report.md`, `comparison_report.md`) go in the run root.
+2. **Benchmark outputs live with their experiment.**
+   - For `experiments/<name>/`, results go in `experiments/<name>/results/<RUN_ID>/`.
+   - Each run directory must contain `report.md` (human-readable) and `results.json` (structured data).
+   - Raw outputs go in `raw_outputs/`; logs go in `logs/`.
+   - Legacy `bench_results/<experiment>_<timestamp>/` directories remain valid for archived runs.

 3. **Each `bench_results/<run>/` directory must contain two final artifacts.**
   - A Markdown report for human reading (e.g. `report.md`, `comparison_report.md`).
@ -50,7 +59,9 @@
   - Keep the schema stable so downstream Python scripts can parse all experiments uniformly.
   - See [Final JSON Schema](#final-json-schema) below for the recommended structure.

-5. **Scripts should default `RESULT_ROOT` to `bench_results/<experiment>_${RUN_ID}`.**
+5. **Scripts should default `RESULT_ROOT` to the experiment's results directory.**
+   - For `experiments/<name>/run_bench.sh`, default to `experiments/<name>/results/${RUN_ID}/`.
+   - For legacy scripts, default to `bench_results/<experiment>_${RUN_ID}/`.
   - Allow override via `RESULT_ROOT` env var.
   - Use `RUN_ID=$(date '+%Y%m%d-%H%M%S')` unless specified.

@ -63,9 +74,33 @@
   - Final reports and JSON outputs must include both the accelerator/chip family and the inference engine/backend used for the run.
   - Do not rely on the experiment name alone to identify the platform or engine.

+8. **Use platform configuration files for chip-specific constants.**
+   - Put per-platform settings in `platforms/<chip>.env` (e.g. `platforms/kunlun_p800.env`, `platforms/nvidia_h200.env`).
+   - Scripts load the platform file via `scripts/common/platform.sh`; the active platform is selected by the `PLATFORM` env var or auto-detected.
+   - Keep experiment scripts free of hardcoded device IDs, image names, or model root paths.
+
 ## Naming Conventions

-### Result directories
+### Experiment result directories
+
+For experiment-centric layout:
+
+```
+experiments/<experiment>/results/<YYYYMMDD-HHMMSS>/
+```
+
+Examples:
+
+- `experiments/dsv4_p800_sglang/results/20260708-120000/`
+- `experiments/dspark_grid/results/20260707-132641/`
+
+The `results.json` metadata already records `chip`/`accelerator` and `engine`, so the directory path does not need to encode them. If a single experiment must distinguish across platforms in its directory tree, use:
+
+```
+experiments/<experiment>/results/<chip>_<engine>_<YYYYMMDD-HHMMSS>/
+```
+
+### Legacy result directories

 ```
 bench_results/<experiment>_<YYYYMMDD-HHMMSS>/
@ -76,13 +111,6 @@ Examples:
 - `bench_results/dspark_grid_20260707-132641/`
 - `bench_results/dsv4_backend_comparison_20260707/`

-When the same experiment is repeated across chips or engines, include them in the directory name or organize by subdirectories so results are not confused:
-
-```
-bench_results/<experiment>_<chip>_<engine>_<YYYYMMDD-HHMMSS>/
-bench_results/<experiment>/<chip>/<engine>/<YYYYMMDD-HHMMSS>/
-```
-
 ### Raw output files

 Include the accelerator and inference engine in raw output filenames so files from different platforms cannot overwrite each other.
@ -190,7 +218,13 @@ The JSON file inside each `bench_results/<run>/` directory should follow a stabl

 ## Quick Start

-### Run DSpark grid benchmark
+### Run P800 SGLang benchmark
+
+```bash
+bash experiments/dsv4_p800_sglang/run_bench.sh
+```
+
+### Run legacy DSpark grid benchmark

 ```bash
 bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
@ -202,28 +236,33 @@ bash scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh
 bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
 ```

-### Run SGLang vs vLLM backend comparison
-
-```bash
-# Start SGLang on port 30000 and vLLM on port 8000, then:
-bash scripts/benchmark_dsv4_backend_comparison.sh all
-```
-
 ### Parse results

 ```bash
+# Experiment-centric layout
+python3 experiments/dsv4_p800_sglang/parse_results.py \
+  experiments/dsv4_p800_sglang/results/<run_id>
+
+# Legacy layout
 /data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
  /data/user1/yy/bench_results/dspark_grid_<run_id>
 ```

+### Cross-experiment comparison
+
+```bash
+python3 scripts/analysis/compare_experiments.py
+```
+
 ## Checklist Before Committing / Archiving

- [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in `/data/user1/yy/` root.
- [ ] All outputs moved to `bench_results/<experiment>_<timestamp>/`.
- [ ] `bench_results/<run>/report.md` (or equivalent human-readable `.md`) exists.
- [ ] `bench_results/<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
- [ ] `bench_results/<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
+- [ ] No `.jsonl`, `.json`, `.log`, or `.md` files left in the project root.
+- [ ] For `experiments/<name>/`, outputs live in `experiments/<name>/results/<timestamp>/`.
+- [ ] For legacy runs, outputs live in `bench_results/<experiment>_<timestamp>/`.
+- [ ] `<run>/report.md` (or equivalent human-readable `.md`) exists.
+- [ ] `<run>/results.json` exists and follows the [Final JSON Schema](#final-json-schema).
+- [ ] `<run>/results.json` metadata records the `chip`/`accelerator` and `engine`/`backend` used.
 - [ ] Raw output filenames include the chip/accelerator and engine when cross-platform runs may collide.
- [ ] `bench_results/<run>/README.md` exists and documents provenance (or the report itself covers provenance).
- [ ] Scripts moved to `scripts/` (or `scripts/<group>/`).
+- [ ] `<run>/README.md` exists and documents provenance (or the report itself covers provenance).
+- [ ] Scripts either live under `experiments/<name>/` or in `scripts/` (or `scripts/<group>/`).
 - [ ] Script path references updated after moving.
--- a/README.md
+++ b/README.md
@ -6,8 +6,12 @@

 | 目录/文件 | 说明 |
 |---|---|
-| `scripts/` | 所有 benchmark 脚本、服务启动脚本、结果解析脚本 |
-| `bench_results/` | 各次实验的最终产物：人可读 `.md` 报告 + 结构化 `results.json`（含原始数据与分位统计） |
+| `platforms/` | 芯片/加速器平台配置（`*.env`），如 P800、H200 |
+| `scripts/common/` | 跨实验复用的 orchestration 组件（server 启停、health check、元数据生成） |
+| `scripts/analysis/` | 跨实验对比分析工具 |
+| `scripts/` | 旧版/legacy benchmark 脚本 |
+| `experiments/` | 以实验为单位的自包含目录（脚本 + 配置 + 结果） |
+| `bench_results/` | 历史归档的实验产物（只读） |
 | `BENCHMARK_WORKFLOW.md` | benchmark 目录与命名规范 |
 | `scripts/SLO_STANDARDS.md` | 推理服务 SLO 标准（TTFT/TPOT） |

@ -15,6 +19,14 @@

 ## 实验索引

+### 新结构（experiments/）
+
+| 实验 | 脚本 | 说明 |
+|---|---|---|
+| DSV4 P800 SGLang | `experiments/dsv4_p800_sglang/run_bench.sh` | Kunlun P800 + SGLang + DeepSeek-V4-Flash-INT8 |
+
+### 旧结构（scripts/ + bench_results/）
+
 | 实验 | 脚本 | 最终报告 | 说明 |
 |---|---|---|---|
 | DSpark grid benchmark | `scripts/benchmark_dspark_0707/run_dspark_benchmark_grid.sh` | `bench_results/dspark_grid_20260707-132641/report.md` | P1/P2/P3 全量网格 |
@ -28,6 +40,12 @@

 ## 快速复现

+### P800 SGLang
+
+```bash
+bash experiments/dsv4_p800_sglang/run_bench.sh
+```
+
 ### DSpark grid

 ```bash
@ -43,6 +61,11 @@ bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh
 ### 解析已有结果

 ```bash
+# 新结构
+python3 experiments/dsv4_p800_sglang/parse_results.py \
+  experiments/dsv4_p800_sglang/results/<run_id>
+
+# 旧结构
 /data/user1/yy/envs/sglang/bin/python scripts/benchmark_dspark_0707/parse_results.py \
  /data/user1/yy/bench_results/dspark_grid_20260707-132641
 ```
@ -58,6 +81,7 @@ bash scripts/benchmark_dspark_0707/run_dspark_st_comparison.sh

 ## 环境要求

- Python env：`/data/user1/yy/envs/vllm-dspark`（服务端）、`/data/user1/yy/envs/sglang`（压测客户端）
- 模型：`/data/models/DeepSeek-V4-Flash`、`/data/models/DeepSeek-V4-Flash-DSpark`
- 硬件：8× H200（当前配置）
+- **NVIDIA H200**：Python env `/data/user1/yy/envs/vllm-dspark`（服务端）、`/data/user1/yy/envs/sglang`（压测客户端）；模型 `/data/models/DeepSeek-V4-Flash`、`/data/models/DeepSeek-V4-Flash-DSpark`。
+- **Kunlun P800**：Docker 镜像 `iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202`；模型 `/data1/models/DeepSeek-V4-Flash-INT8`；压测客户端在容器内运行。
+
+平台相关常量见 `platforms/*.env`。
--- a/experiments/dsv4_p800_sglang/README.md
+++ b/experiments/dsv4_p800_sglang/README.md
@ -0,0 +1,28 @@
+# DSV4 P800 SGLang Benchmark
+
+Kunlun P800 XPU + SGLang + `DeepSeek-V4-Flash-INT8` benchmark experiment.
+
+## Quick Start
+
+```bash
+# Run all configured scenarios (starts server, runs benchmark, stops server)
+bash experiments/dsv4_p800_sglang/run_bench.sh
+
+# Reuse an already-running server
+SKIP_MANAGE_SERVER=1 bash experiments/dsv4_p800_sglang/run_bench.sh
+```
+
+Results land in `experiments/dsv4_p800_sglang/results/<RUN_ID>/`.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `config.env` | Experiment-level configuration (model, port, scenarios) |
+| `start_server.sh` | Start the P800 SGLang Docker container |
+| `run_bench.sh` | Orchestrator: server → benchmark → stop server |
+| `parse_results.py` | Parse logs and generate `results.json` + `report.md` |
+
+## Platform
+
+This experiment targets `platforms/kunlun_p800.env`.
--- a/experiments/dsv4_p800_sglang/config.env
+++ b/experiments/dsv4_p800_sglang/config.env
@ -0,0 +1,35 @@
+# Experiment-level configuration for dsv4_p800_sglang
+# All values can be overridden via environment variables.
+
+EXPERIMENT="${EXPERIMENT:-dsv4_p800_sglang}"
+MODEL_NAME="${MODEL_NAME:-DeepSeek-V4-Flash-INT8}"
+MODEL_PATH="${MODEL_PATH:-/data1/models/DeepSeek-V4-Flash-INT8}"
+SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-deepseek-v4-flash-int8}"
+PORT="${PORT:-30000}"
+BACKEND="${BACKEND:-sglang}"
+ENGINE="${ENGINE:-sglang-xpu}"
+DATASET="${DATASET:-random}"
+# Path to a local ShareGPT-style dataset inside the container.
+# Required when HF Hub is offline. The repo ships a dummy dataset under
+# platforms/patches/kunlun_p800/dummy_sharegpt.json; for managed mode it is
+# mounted to /workspace/dummy_sharegpt.json.
+DATASET_PATH="${DATASET_PATH:-/workspace/dummy_sharegpt.json}"
+WARMUP="${WARMUP:-100}"
+NUM_PROMPTS="${NUM_PROMPTS:-512}"
+
+# Server launch mode for P800 SGLang.
+#   - "fp8"      : non-INT8 DeepSeek-V4-Flash checkpoint
+#   - "w8a8_int8": DeepSeek-V4-Flash-INT8 checkpoint
+SERVER_MODE="${SERVER_MODE:-w8a8_int8}"
+
+# Scenario list: "concurrency input_len output_len" (one per line)
+# Override example: SCENARIOS=("32 512 256")
+if [[ -z "${SCENARIOS:-}" ]]; then
+  SCENARIOS=(
+    "32 512 256"
+    "128 512 256"
+    "256 512 256"
+    "512 512 256"
+  )
+fi
+
--- a/experiments/dsv4_p800_sglang/parse_results.py
+++ b/experiments/dsv4_p800_sglang/parse_results.py
@ -0,0 +1,313 @@
+#!/usr/bin/env python3
+"""Parse P800 SGLang benchmark outputs and produce results.json + report.md.
+
+Usage:
+    python3 parse_results.py <result_root>
+"""
+
+import json
+import os
+import re
+import sys
+from pathlib import Path
+from collections import OrderedDict
+
+METRIC_PATTERNS = OrderedDict(
+    [
+        ("successful_requests", [r"Successful requests:\s+(\d+)"]),
+        ("benchmark_duration_s", [r"Benchmark duration \(s\):\s+([\d.]+)"]),
+        ("request_throughput", [r"Request throughput \(req/s\):\s+([\d.]+)"]),
+        ("input_token_throughput", [r"Input token throughput \(tok/s\):\s+([\d.]+)"]),
+        ("output_token_throughput", [r"Output token throughput \(tok/s\):\s+([\d.]+)"]),
+        ("total_token_throughput", [r"Total token throughput \(tok/s\):\s+([\d.]+)"]),
+        ("ttft_mean", [r"Mean TTFT \(ms\):\s+([\d.]+)"]),
+        ("ttft_p50", [r"Median TTFT \(ms\):\s+([\d.]+)", r"P50 TTFT \(ms\):\s+([\d.]+)"]),
+        ("ttft_p90", [r"P90 TTFT \(ms\):\s+([\d.]+)"]),
+        ("ttft_p99", [r"P99 TTFT \(ms\):\s+([\d.]+)"]),
+        ("tpot_mean", [r"Mean TPOT \(ms\):\s+([\d.]+)"]),
+        ("tpot_p50", [r"Median TPOT \(ms\):\s+([\d.]+)", r"P50 TPOT \(ms\):\s+([\d.]+)"]),
+        ("tpot_p90", [r"P90 TPOT \(ms\):\s+([\d.]+)"]),
+        ("tpot_p99", [r"P99 TPOT \(ms\):\s+([\d.]+)"]),
+        ("e2e_mean", [r"Mean E2E \(ms\):\s+([\d.]+)"]),
+        ("e2e_p50", [r"Median E2E \(ms\):\s+([\d.]+)", r"P50 E2E \(ms\):\s+([\d.]+)"]),
+        ("e2e_p90", [r"P90 E2E \(ms\):\s+([\d.]+)"]),
+        ("e2e_p99", [r"P99 E2E \(ms\):\s+([\d.]+)"]),
+    ]
+)
+
+
+def find_float(text: str, patterns: list[str]) -> float | None:
+    for pat in patterns:
+        m = re.search(pat, text)
+        if m:
+            try:
+                return float(m.group(1))
+            except ValueError:
+                return None
+    return None
+
+
+def parse_summary_log(log_text: str) -> dict:
+    result = {}
+    for key, patterns in METRIC_PATTERNS.items():
+        result[key] = find_float(log_text, patterns)
+    return result
+
+
+def percentile(values: list[float], p: float) -> float:
+    if not values:
+        return 0.0
+    sorted_values = sorted(values)
+    k = (len(sorted_values) - 1) * p / 100.0
+    f = int(k)
+    c = min(f + 1, len(sorted_values) - 1)
+    if f == c:
+        return sorted_values[f]
+    return sorted_values[f] * (c - k) + sorted_values[c] * (k - f)
+
+
+def parse_jsonl(jsonl_path: Path) -> tuple[list[dict], dict]:
+    """Parse sglang.bench_serving --output-file output.
+
+    Newer sglang writes one aggregate JSON object. Older versions write one
+    JSON object per request. Returns (raw_requests, aggregates).
+    """
+    text = jsonl_path.read_text(encoding="utf-8", errors="replace").strip()
+    if not text:
+        return [], {}
+
+    # Try single aggregate JSON object first.
+    try:
+        data = json.loads(text)
+        if isinstance(data, dict) and "mean_e2e_latency_ms" in data:
+            aggregates = {
+                "success": data.get("total_output_tokens", 0) > 0 and 1 or 0,
+                "failed": 0,
+                "input_tokens": data.get("total_input_tokens", 0),
+                "output_tokens": data.get("total_output_tokens", 0),
+                "latencies": {
+                    "e2e_ms": {
+                        "mean": data.get("mean_e2e_latency_ms"),
+                        "p50": data.get("median_e2e_latency_ms"),
+                        "p90": data.get("p90_e2e_latency_ms"),
+                        "p95": None,
+                        "p99": data.get("p99_e2e_latency_ms"),
+                    },
+                    "ttft_ms": {
+                        "mean": data.get("mean_ttft_ms"),
+                        "p50": data.get("median_ttft_ms"),
+                        "p90": None,
+                        "p95": None,
+                        "p99": data.get("p99_ttft_ms"),
+                    },
+                    "tpot_ms": {
+                        "mean": data.get("mean_tpot_ms"),
+                        "p50": data.get("median_tpot_ms"),
+                        "p90": None,
+                        "p95": None,
+                        "p99": data.get("p99_tpot_ms"),
+                    },
+                    "itl_ms": {
+                        "mean": data.get("mean_itl_ms"),
+                        "p50": data.get("median_itl_ms"),
+                        "p90": None,
+                        "p95": data.get("p95_itl_ms"),
+                        "p99": data.get("p99_itl_ms"),
+                    },
+                },
+            }
+            return [data], aggregates
+    except json.JSONDecodeError:
+        pass
+
+    # Fall back to JSONL per-request parsing.
+    raw_requests = []
+    ttfts = []
+    tpots = []
+    itls = []
+    e2es = []
+    input_tokens = []
+    output_tokens = []
+    success = 0
+    failed = 0
+
+    for line in text.splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            req = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+
+        raw_requests.append(req)
+
+        ttft = req.get("ttft") or req.get("ttft_ms") or 0
+        tpot = req.get("tpot") or req.get("tpot_ms") or 0
+        itl = req.get("inter_token_latency") or req.get("itl") or req.get("itl_ms") or 0
+        e2e = req.get("e2e_latency") or req.get("e2e") or req.get("e2e_ms") or 0
+        in_tok = req.get("input_tokens") or req.get("prompt_tokens") or 0
+        out_tok = req.get("output_tokens") or req.get("completion_tokens") or 0
+
+        if ttft:
+            ttfts.append(float(ttft))
+        if tpot:
+            tpots.append(float(tpot))
+        if itl:
+            itls.append(float(itl))
+        if e2e:
+            e2es.append(float(e2e))
+        if in_tok:
+            input_tokens.append(int(in_tok))
+        if out_tok:
+            output_tokens.append(int(out_tok))
+
+        if req.get("success", True):
+            success += 1
+        else:
+            failed += 1
+
+    def latency_bucket(values: list[float]) -> dict:
+        if not values:
+            return {"mean": None, "p50": None, "p90": None, "p95": None, "p99": None}
+        return {
+            "mean": round(sum(values) / len(values), 2),
+            "p50": round(percentile(values, 50), 2),
+            "p90": round(percentile(values, 90), 2),
+            "p95": round(percentile(values, 95), 2),
+            "p99": round(percentile(values, 99), 2),
+        }
+
+    aggregates = {
+        "success": success,
+        "failed": failed,
+        "input_tokens": sum(input_tokens),
+        "output_tokens": sum(output_tokens),
+        "latencies": {
+            "e2e_ms": latency_bucket(e2es),
+            "ttft_ms": latency_bucket(ttfts),
+            "tpot_ms": latency_bucket(tpots),
+            "itl_ms": latency_bucket(itls),
+        },
+    }
+
+    return raw_requests, aggregates
+
+
+def parse_scenario(result_root: Path, raw_file: Path) -> dict | None:
+    """Parse one raw output file into a scenario dict."""
+    # Filename: {chip}_{engine}_{MMDD}_{concurrency}_{input_len}_{output_len}.jsonl
+    parts = raw_file.stem.split("_")
+    if len(parts) < 6:
+        return None
+    try:
+        concurrency = int(parts[-3])
+        input_len = int(parts[-2])
+        output_len = int(parts[-1])
+    except ValueError:
+        return None
+
+    log_file = result_root / "logs" / f"sglang_c{concurrency}_i{input_len}_o{output_len}.log"
+    summary = {}
+    if log_file.exists():
+        summary = parse_summary_log(log_file.read_text(encoding="utf-8", errors="replace"))
+
+    raw_requests, aggregates = parse_jsonl(raw_file)
+
+    # For newer sglang aggregate JSON, success/failed are not present in the
+    # raw output file. Override them from the human-readable summary log when
+    # it is available.
+    if summary.get("successful_requests") is not None:
+        aggregates["success"] = int(summary["successful_requests"])
+        # The summary log only reports successes; assume failures are zero
+        # unless the aggregate JSON already provided a non-zero failed count.
+        if not aggregates.get("failed"):
+            aggregates["failed"] = 0
+
+    scenario = {
+        "name": f"c{concurrency}_i{input_len}_o{output_len}",
+        "concurrency": concurrency,
+        "input_len": input_len,
+        "output_len": output_len,
+        "success": aggregates["success"],
+        "failed": aggregates["failed"],
+        "duration_s": summary.get("benchmark_duration_s"),
+        "request_throughput": summary.get("request_throughput"),
+        "input_token_throughput": summary.get("input_token_throughput"),
+        "output_token_throughput": summary.get("output_token_throughput"),
+        "total_token_throughput": summary.get("total_token_throughput"),
+        "accept_length": None,
+        "latencies": aggregates["latencies"],
+        "raw_requests": raw_requests[:100] if len(raw_requests) <= 100 else None,
+    }
+
+    return scenario
+
+
+def main() -> None:
+    result_root = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("results")
+    raw_dir = result_root / "raw_outputs"
+    json_path = result_root / "results.json"
+    report_path = result_root / "report.md"
+
+    if not json_path.exists():
+        raise SystemExit(f"metadata results.json not found: {json_path}")
+
+    with open(json_path, "r", encoding="utf-8") as f:
+        data = json.load(f)
+
+    scenarios = []
+    if raw_dir.exists():
+        for raw_file in sorted(raw_dir.glob("*.jsonl")):
+            scenario = parse_scenario(result_root, raw_file)
+            if scenario:
+                scenarios.append(scenario)
+
+    data["scenarios"] = scenarios
+
+    with open(json_path, "w", encoding="utf-8") as f:
+        json.dump(data, f, indent=2, ensure_ascii=False)
+
+    # Generate report.md
+    with open(report_path, "w", encoding="utf-8") as f:
+        meta = data["metadata"]
+        f.write(f"# Benchmark Report: {meta['experiment']}\n\n")
+        f.write("## Metadata\n\n")
+        f.write(f"- **Run ID**: {meta['run_id']}\n")
+        f.write(f"- **Timestamp**: {meta['timestamp']}\n")
+        f.write(f"- **Chip/Accelerator**: {meta.get('accelerator', '')} / {meta.get('chip', '')}\n")
+        f.write(f"- **Engine/Backend**: {meta.get('engine', '')} / {meta.get('backend', '')}\n")
+        f.write(f"- **Hardware**: {meta.get('hardware', '')}\n")
+        f.write(f"- **Model**: {meta['model']}\n")
+        f.write(f"- **Git Commit**: {meta.get('git_commit', 'unknown')}\n")
+        f.write("\n## Results\n\n")
+        f.write(
+            "| Scenario | Conc | In/Out | Success | Failed | Req/s | OutTok/s | "
+            "TTFT p50 | TTFT p99 | TPOT p50 | TPOT p99 | E2E p99 |\n"
+        )
+        f.write(
+            "|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n"
+        )
+        for s in scenarios:
+            lat = s["latencies"]
+            f.write(
+                f"| {s['name']} "
+                f"| {s['concurrency']} "
+                f"| {s['input_len']}/{s['output_len']} "
+                f"| {s['success']} "
+                f"| {s['failed']} "
+                f"| {s.get('request_throughput') or ''} "
+                f"| {s.get('output_token_throughput') or ''} "
+                f"| {lat['ttft_ms']['p50'] or ''} "
+                f"| {lat['ttft_ms']['p99'] or ''} "
+                f"| {lat['tpot_ms']['p50'] or ''} "
+                f"| {lat['tpot_ms']['p99'] or ''} "
+                f"| {lat['e2e_ms']['p99'] or ''} |\n"
+            )
+
+    print(f"Updated {json_path}")
+    print(f"Wrote {report_path}")
+
+
+if __name__ == "__main__":
+    main()
--- a/experiments/dsv4_p800_sglang/run_bench.sh
+++ b/experiments/dsv4_p800_sglang/run_bench.sh
@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
+
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../scripts/common/lib.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../scripts/common/platform.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../scripts/common/server_docker.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../scripts/common/bench_client_docker.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+RUN_ID="${RUN_ID:-$(date '+%Y%m%d-%H%M%S')}"
+RESULT_ROOT="${RESULT_ROOT:-${SCRIPT_DIR}/results/${RUN_ID}}"
+RAW_DIR="${RESULT_ROOT}/raw_outputs"
+LOG_DIR="${RESULT_ROOT}/logs"
+
+ensure_result_root "$RESULT_ROOT"
+log_init "${LOG_DIR}/orchestrator.log"
+
+log "experiment=${EXPERIMENT_NAME}"
+log "run_id=${RUN_ID}"
+log "result_root=${RESULT_ROOT}"
+log "platform=${PLATFORM}"
+log "chip=${CHIP}"
+log "accelerator=${ACCELERATOR}"
+log "engine=${ENGINE}"
+log "hardware=${HARDWARE}"
+
+# Write initial metadata for this run.
+METADATA_JSON="${RESULT_ROOT}/results.json"
+write_metadata_json \
+  "$METADATA_JSON" \
+  "$EXPERIMENT_NAME" \
+  "$RUN_ID" \
+  "$MODEL_PATH" \
+  "$BACKEND" \
+  "$ENGINE" \
+  "$HARDWARE" \
+  "$ACCELERATOR" \
+  "$CHIP" \
+  "experiments/${EXPERIMENT_NAME}/run_bench.sh" \
+  "" \
+  "Kunlun P800 SGLang benchmark for DeepSeek-V4-Flash-INT8"
+
+# Start server unless SKIP_MANAGE_SERVER is set.
+if [[ -n "${SKIP_MANAGE_SERVER:-}" ]]; then
+  log "SKIP_MANAGE_SERVER is set, assuming server is already running on port ${PORT}"
+  if ! health_check 127.0.0.1 "${PORT}" 30; then
+    log "ERROR: no healthy server found at port ${PORT}"
+    exit 1
+  fi
+else
+  docker_server_start "${MODEL_PATH}" "${PORT}" "${LOG_DIR}/server.outer.log" "${SERVER_MODE}"
+fi
+
+on_exit() {
+  local code=$?
+  if [[ -z "${SKIP_MANAGE_SERVER:-}" ]]; then
+    docker_server_stop
+  fi
+  log "orchestrator exiting with code=${code}"
+  exit "$code"
+}
+trap on_exit EXIT
+
+log "===== BENCHMARK START ====="
+
+for scenario in "${SCENARIOS[@]}"; do
+  # Allow SCENARIOS to be passed as a string like "(32 512 256)" from env.
+  scenario="${scenario//[()]/}"
+  read -r concurrency input_len output_len <<< "$scenario"
+
+  tmp_output_file="/tmp/bench_outputs/${CHIP}_${ENGINE}_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
+  host_output_file="${RAW_DIR}/${CHIP}_${ENGINE}_$(date '+%m%d')_${concurrency}_${input_len}_${output_len}.jsonl"
+  detail_log="${LOG_DIR}/${BACKEND}_c${concurrency}_i${input_len}_o${output_len}.log"
+
+  log "running scenario: concurrency=${concurrency} input=${input_len} output=${output_len}"
+
+  run_random_case \
+    "$BACKEND" \
+    "$PORT" \
+    "$MODEL_PATH" \
+    "$tmp_output_file" \
+    "$concurrency" \
+    "$input_len" \
+    "$output_len" \
+    "$NUM_PROMPTS" \
+    "${DATASET_PATH:-}" \
+    > "$detail_log" 2>&1 || {
+      log "ERROR: scenario c=${concurrency} i=${input_len} o=${output_len} failed"
+      continue
+    }
+
+  # Copy the JSONL from the container to the host result directory.
+  docker cp "${CONTAINER_NAME}:${tmp_output_file}" "$host_output_file" || {
+    log "ERROR: failed to copy ${tmp_output_file} from container"
+    continue
+  }
+  log "finished scenario: output=${host_output_file}"
+done
+
+log "===== BENCHMARK DONE ====="
+log "parsing results..."
+"${SCRIPT_DIR}/parse_results.py" "$RESULT_ROOT"
+
+log "all results saved to ${RESULT_ROOT}"
--- a/experiments/dsv4_p800_sglang/start_server.sh
+++ b/experiments/dsv4_p800_sglang/start_server.sh
@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+EXPERIMENT_NAME="$(basename "$SCRIPT_DIR")"
+
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/../../scripts/common/server_docker.sh"
+# shellcheck source=/dev/null
+source "${SCRIPT_DIR}/config.env"
+
+log_init "${RESULT_ROOT:-/tmp/${EXPERIMENT_NAME}}/logs/start_server.log"
+log "starting server for ${EXPERIMENT_NAME}"
+log "model: ${MODEL_PATH}"
+log "port: ${PORT}"
+
+docker_server_start "${MODEL_PATH}" "${PORT}" "${LOG_DIR}/server.outer.log" "${SERVER_MODE}"
--- a/platforms/README.md
+++ b/platforms/README.md
@ -0,0 +1,36 @@
+# Platform Configurations
+
+Each `.env` file in this directory describes one accelerator platform.
+They are meant to be sourced by benchmark scripts through
+`scripts/common/platform.sh`, not executed directly.
+
+## Usage
+
+```bash
+# Default platform for the current machine
+bash experiments/dsv4_p800_sglang/run_bench.sh
+
+# Explicitly select a platform
+PLATFORM=kunlun_p800 bash experiments/dsv4_p800_sglang/run_bench.sh
+```
+
+## Current platforms
+
+| File | Chip/Accelerator | Engine | Notes |
+|---|---|---|---|
+| `kunlun_p800.env` | Kunlun P800 XPU | `sglang-xpu` | Docker-based SGLang serving image |
+| `nvidia_h200.env` | NVIDIA H200 | native vllm/sglang | Host virtual environments |
+
+## What belongs here
+
+- Chip/accelerator identity (`CHIP`, `ACCELERATOR`, `HARDWARE`).
+- Device selection environment variables.
+- Platform-wide paths that rarely change (model root, default port).
+- Container image / interpreter paths for Docker-based platforms.
+
+## What does NOT belong here
+
+- Specific model names or experiment scenarios — those go in
+  `experiments/<name>/config.env`.
+- Engine-specific launch flags — those go in the experiment's
+  `start_server.sh` or `run_bench.sh`.
--- a/platforms/kunlun_p800.env
+++ b/platforms/kunlun_p800.env
@ -0,0 +1,27 @@
+# Platform configuration for Kunlun P800 XPU
+# Source this file via scripts/common/platform.sh
+
+CHIP="kunlun_p800"
+ACCELERATOR="Kunlun P800 XPU"
+HARDWARE="8x Kunlun P800 XPU"
+ENGINE="sglang-xpu"
+
+# Container runtime
+DOCKER_IMAGE="iregistry.baidu-int.com/xpu/sglang-p800-pd-disagg-0510:20260511_4202"
+CONTAINER_NAME="${CONTAINER_NAME:-sglang-dsv4-flash}"
+
+# Device selection
+DEVICE_SELECT_ENV="XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
+CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
+CUDA_DEVICE_ORDER="OAM_ID"
+
+# Default serving port and model root on the host
+DEFAULT_PORT="30000"
+MODEL_ROOT="/data1/models"
+
+# Python interpreter inside the container
+CONTAINER_PYTHON="/root/miniconda/envs/python310_torch25_cuda/bin/python"
+
+# Host mount points for patches required by the P800 SGLang image
+# These are baked into the container start command in server_docker.sh.
+PATCH_ROOT="${PATCH_ROOT:-/data1/yy/sskj/platforms/patches/kunlun_p800}"
--- a/platforms/nvidia_h200.env
+++ b/platforms/nvidia_h200.env
@ -0,0 +1,18 @@
+# Platform configuration for NVIDIA H200
+# Source this file via scripts/common/platform.sh
+
+CHIP="nvidia_h200"
+ACCELERATOR="NVIDIA H200"
+HARDWARE="8x NVIDIA H200 143GB"
+
+# Device selection
+DEVICE_SELECT_ENV="CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7"
+CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
+
+# Default serving port and model root on the host
+DEFAULT_PORT="30004"
+MODEL_ROOT="/data/models"
+
+# Virtual environments on the host (used by native H200 scripts)
+VENV_VLLM_DSPARK="/data/user1/yy/envs/vllm-dspark"
+VENV_SGLANG="/data/user1/yy/envs/sglang"
--- a/platforms/patches/kunlun_p800/config_backup_small_fp8.json
+++ b/platforms/patches/kunlun_p800/config_backup_small_fp8.json
@ -0,0 +1,114 @@
+{
+  "architectures": [
+    "DeepseekXYZForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "hc_eps": 1e-06,
+  "hc_mult": 4,
+  "hc_sinkhorn_iters": 20,
+  "head_dim": 512,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "index_head_dim": 128,
+  "index_n_heads": 64,
+  "index_topk": 512,
+  "initializer_range": 0.02,
+  "max_position_embeddings": 1048576,
+  "model_type": "deepseek_ref",
+  "moe_intermediate_size": 2048,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 43,
+  "num_hash_layers": 3,
+  "num_key_value_heads": 1,
+  "num_nextn_predict_layers": 1,
+  "o_groups": 8,
+  "o_lora_rank": 1024,
+  "q_lora_rank": 1024,
+  "qk_rope_head_dim": 64,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 16.0,
+    "original_max_position_embeddings": 65536,
+    "type": "yarn"
+  },
+  "rope_theta": 10000,
+  "routed_scaling_factor": 1.5,
+  "scoring_func": "sqrtsoftplus",
+  "sliding_window": 128,
+  "swiglu_limit": 10.0,
+  "tie_word_embeddings": false,
+  "n_group": 8,
+  "topk_group": 8,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 129280,
+  "compress_rope_theta": 160000,
+  "compress_ratios": [
+    0,
+    0,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    128,
+    4,
+    0
+  ],
+  "quantization_config": {
+    "activation_scheme": "dynamic",
+    "fmt": "e4m3",
+    "quant_method": "fp8",
+    "scale_fmt": "ue8m0",
+    "weight_block_size": [
+      128,
+      128
+    ]
+  },
+  "expert_dtype": "fp4"
+}
--- a/platforms/patches/kunlun_p800/config_backup_small_w8a8_int8.json
+++ b/platforms/patches/kunlun_p800/config_backup_small_w8a8_int8.json
@ -0,0 +1,58 @@
+{
+  "architectures": [
+    "DeepseekXYZForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "hc_eps": 1e-06,
+  "hc_mult": 4,
+  "hc_sinkhorn_iters": 20,
+  "head_dim": 512,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "index_head_dim": 128,
+  "index_n_heads": 64,
+  "index_topk": 512,
+  "initializer_range": 0.02,
+  "max_position_embeddings": 1048576,
+  "model_type": "deepseek_ref",
+  "moe_intermediate_size": 2048,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 43,
+  "num_hash_layers": 3,
+  "num_key_value_heads": 1,
+  "num_nextn_predict_layers": 1,
+  "o_groups": 8,
+  "o_lora_rank": 1024,
+  "q_lora_rank": 1024,
+  "qk_rope_head_dim": 64,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "beta_fast": 32,
+    "beta_slow": 1,
+    "factor": 16,
+    "original_max_position_embeddings": 65536,
+    "type": "yarn"
+  },
+  "rope_theta": 10000,
+  "routed_scaling_factor": 1.5,
+  "scoring_func": "sqrtsoftplus",
+  "sliding_window": 128,
+  "swiglu_limit": 10.0,
+  "tie_word_embeddings": false,
+  "n_group": 8,
+  "topk_group": 8,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.57.1",
+  "use_cache": true,
+  "vocab_size": 129280,
+  "compress_rope_theta": 160000,
+  "compress_ratios": [0, 0, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 128, 4, 0]
+}
--- a/platforms/patches/kunlun_p800/dummy_sharegpt.json
+++ b/platforms/patches/kunlun_p800/dummy_sharegpt.json
--- a/platforms/patches/kunlun_p800/fp8_utils.py.patched
+++ b/platforms/patches/kunlun_p800/fp8_utils.py.patched
--- a/platforms/patches/kunlun_p800/hf_transformers_utils.py.patched
+++ b/platforms/patches/kunlun_p800/hf_transformers_utils.py.patched
--- a/platforms/patches/kunlun_p800/nic_priority_matrix_test.json
+++ b/platforms/patches/kunlun_p800/nic_priority_matrix_test.json
@ -0,0 +1,98 @@
+{
+  "cpu:0": [
+    [
+      "mlx5_0"
+    ],
+    []
+  ],
+  "cpu:1": [
+    [
+      "mlx5_0"
+    ],
+    []
+  ],
+  "cpu:2": [
+    [
+      "mlx5_0"
+    ],
+    []
+  ],
+  "cpu:3": [
+    [
+      "mlx5_0"
+    ],
+    []
+  ],
+  "cpu:4": [
+    [
+      "mlx5_0"
+    ],
+    []
+  ],
+  "cpu:5": [
+    [
+      "mlx5_0"
+    ],
+    []
+  ],
+  "cpu:6": [
+    [
+      "mlx5_0"
+    ],
+    []
+  ],
+  "cpu:7": [
+    [
+      "mlx5_0"
+    ],
+    []
+  ],
+  "cuda:0": [
+    [
+      "mlx5_1"
+    ],
+    []
+  ],
+  "cuda:1": [
+    [
+      "mlx5_1"
+    ],
+    []
+  ],
+  "cuda:2": [
+    [
+      "mlx5_2"
+    ],
+    []
+  ],
+  "cuda:3": [
+    [
+      "mlx5_2"
+    ],
+    []
+  ],
+  "cuda:4": [
+    [
+      "mlx5_3"
+    ],
+    []
+  ],
+  "cuda:5": [
+    [
+      "mlx5_3"
+    ],
+    []
+  ],
+  "cuda:6": [
+    [
+      "mlx5_4"
+    ],
+    []
+  ],
+  "cuda:7": [
+    [
+      "mlx5_4"
+    ],
+    []
+  ]
+}
--- a/platforms/patches/kunlun_p800/parallel_state.py
+++ b/platforms/patches/kunlun_p800/parallel_state.py
--- a/platforms/patches/kunlun_p800/sitecustomize_xpu.py
+++ b/platforms/patches/kunlun_p800/sitecustomize_xpu.py
@ -0,0 +1,7 @@
+import torch
+# DeepSeek-V4-Flash safetensors contain F8_E8M0FN dtype metadata, but
+# PyTorch 2.5.1 does not define this dtype. Alias it to uint8 so that
+# safetensors deserialization can load the scales; sglang reinterprets
+# them on P800 via its own FP8 kernels.
+if not hasattr(torch, "float8_e8m0fnu"):
+    torch.float8_e8m0fnu = torch.uint8
--- a/platforms/patches/kunlun_p800/vocab_parallel_embedding.py
+++ b/platforms/patches/kunlun_p800/vocab_parallel_embedding.py
@ -0,0 +1,680 @@
+# Adapted from https://github.com/vllm-project/vllm/blob/v0.6.3.post1/vllm/model_executor/layers/vocab_parallel_embedding.py
+
+import logging
+from dataclasses import dataclass
+from typing import List, Optional, Sequence, Tuple
+
+import torch
+import torch.nn.functional as F
+from torch.nn.parameter import Parameter, UninitializedParameter
+
+from sglang.srt.distributed import (
+    divide,
+    get_tensor_model_parallel_rank,
+    get_tensor_model_parallel_world_size,
+    get_tp_group,
+    tensor_model_parallel_all_gather,
+    tensor_model_parallel_all_reduce,
+)
+from sglang.srt.distributed.device_communicators.pynccl_allocator import (
+    use_symmetric_memory,
+)
+from sglang.srt.layers.amx_utils import PackWeightMethod
+from sglang.srt.layers.communicator import get_attn_tp_context
+from sglang.srt.layers.dp_attention import (
+    attn_tp_all_reduce,
+    get_attention_tp_rank,
+    get_attention_tp_size,
+    is_allocation_symmetric,
+)
+from sglang.srt.layers.parameter import BasevLLMParameter
+from sglang.srt.layers.quantization.base_config import (
+    QuantizationConfig,
+    QuantizeMethodBase,
+    method_has_implemented_embedding,
+)
+from sglang.srt.layers.quantization.unquant import UnquantizedEmbeddingMethod
+from sglang.srt.utils import (
+    cpu_has_amx_support,
+    is_cpu,
+    is_npu,
+    set_weight_attrs,
+)
+
+DEFAULT_VOCAB_PADDING_SIZE = 64
+
+_is_cpu_amx_available = cpu_has_amx_support()
+_is_cpu = is_cpu()
+_is_npu = is_npu()
+
+logger = logging.getLogger(__name__)
+
+
+def pad_vocab_size(vocab_size: int, pad_to: int = DEFAULT_VOCAB_PADDING_SIZE) -> int:
+    """Pad the vocab size to the given value."""
+    return ((vocab_size + pad_to - 1) // pad_to) * pad_to
+
+
+def vocab_range_from_per_partition_vocab_size(
+    per_partition_vocab_size: int, rank: int, offset: int = 0
+) -> Sequence[int]:
+    index_f = rank * per_partition_vocab_size
+    index_l = index_f + per_partition_vocab_size
+    return index_f + offset, index_l + offset
+
+
+def vocab_range_from_global_vocab_size(
+    global_vocab_size: int, rank: int, world_size: int, offset: int = 0
+) -> Sequence[int]:
+    per_partition_vocab_size = divide(global_vocab_size, world_size)
+    return vocab_range_from_per_partition_vocab_size(
+        per_partition_vocab_size, rank, offset=offset
+    )
+
+
+@dataclass
+class VocabParallelEmbeddingShardIndices:
+    """Indices for a shard of a vocab parallel embedding."""
+
+    padded_org_vocab_start_index: int
+    padded_org_vocab_end_index: int
+    padded_added_vocab_start_index: int
+    padded_added_vocab_end_index: int
+
+    org_vocab_start_index: int
+    org_vocab_end_index: int
+    added_vocab_start_index: int
+    added_vocab_end_index: int
+
+    @property
+    def num_org_elements(self) -> int:
+        return self.org_vocab_end_index - self.org_vocab_start_index
+
+    @property
+    def num_added_elements(self) -> int:
+        return self.added_vocab_end_index - self.added_vocab_start_index
+
+    @property
+    def num_org_elements_padded(self) -> int:
+        return self.padded_org_vocab_end_index - self.padded_org_vocab_start_index
+
+    @property
+    def num_added_elements_padded(self) -> int:
+        return self.padded_added_vocab_end_index - self.padded_added_vocab_start_index
+
+    @property
+    def num_org_vocab_padding(self) -> int:
+        return self.num_org_elements_padded - self.num_org_elements
+
+    @property
+    def num_added_vocab_padding(self) -> int:
+        return self.num_added_elements_padded - self.num_added_elements
+
+    @property
+    def num_elements_padded(self) -> int:
+        return self.num_org_elements_padded + self.num_added_elements_padded
+
+    def __post_init__(self):
+        # sanity checks
+        assert self.padded_org_vocab_start_index <= self.padded_org_vocab_end_index
+        assert self.padded_added_vocab_start_index <= self.padded_added_vocab_end_index
+
+        assert self.org_vocab_start_index <= self.org_vocab_end_index
+        assert self.added_vocab_start_index <= self.added_vocab_end_index
+
+        assert self.org_vocab_start_index <= self.padded_org_vocab_start_index
+        assert self.added_vocab_start_index <= self.padded_added_vocab_start_index
+        assert self.org_vocab_end_index <= self.padded_org_vocab_end_index
+        assert self.added_vocab_end_index <= self.padded_added_vocab_end_index
+
+        assert self.num_org_elements <= self.num_org_elements_padded
+        assert self.num_added_elements <= self.num_added_elements_padded
+
+
+def get_masked_input_and_mask(
+    input_: torch.Tensor,
+    org_vocab_start_index: int,
+    org_vocab_end_index: int,
+    num_org_vocab_padding: int,
+    added_vocab_start_index: int,
+    added_vocab_end_index: int,
+) -> Tuple[torch.Tensor, torch.Tensor]:
+    # torch.compile will fuse all of the pointwise ops below
+    # into a single kernel, making it very fast
+    org_vocab_mask = (input_ >= org_vocab_start_index) & (input_ < org_vocab_end_index)
+    added_vocab_mask = (input_ >= added_vocab_start_index) & (
+        input_ < added_vocab_end_index
+    )
+    added_offset = (
+        added_vocab_start_index
+        - (org_vocab_end_index - org_vocab_start_index)
+        - num_org_vocab_padding
+    )
+    valid_offset = (org_vocab_start_index * org_vocab_mask) + (
+        added_offset * added_vocab_mask
+    )
+    vocab_mask = org_vocab_mask | added_vocab_mask
+    input_ = vocab_mask * (input_ - valid_offset)
+    return input_, ~vocab_mask
+
+
+class VocabParallelEmbedding(torch.nn.Module):
+    """Embedding parallelized in the vocabulary dimension.
+
+    Adapted from torch.nn.Embedding, note that we pad the vocabulary size to
+    make sure it is divisible by the number of model parallel GPUs.
+
+    In order to support various loading methods, we ensure that LoRA-added
+    embeddings are always at the end of TP-sharded tensors. In other words,
+    we shard base embeddings and LoRA embeddings separately (both padded),
+    and place them in the same tensor.
+    In this example, we will have the original vocab size = 1010,
+    added vocab size = 16 and padding to 64. Therefore, the total
+    vocab size with padding will be 1088 (because we first pad 1010 to
+    1024, add 16, and then pad to 1088).
+    Therefore, the tensor format looks like the following:
+    TP1, rank 0 (no sharding):
+                            |< --------BASE-------- >|< -BASE PADDING-- >|< -----LORA------ >|< -LORA PADDING-- >|
+    corresponding token_id: |  0  |  1  | ... | 1009 |  -1  | ... |  -1  | 1010 | ... | 1015 |  -1  | ... |  -1  |
+                     index: |  0  |  1  | ... | 1009 | 1010 | ... | 1023 | 1024 | ... | 1039 | 1040 | ... | 1087 |
+
+    TP2, rank 0:
+                            |< --------------------BASE--------------------- >|< -----LORA------ >|< -LORA PADDING- >|
+    corresponding token_id: |  0  |  1  |  2  | ... | 497  | 498 | ...  | 511 | 1000 | ... | 1015 |  -1  | ... |  -1 |
+                     index: |  0  |  1  |  2  | ... | 497  | 498 | ...  | 511 | 512  | ... | 527  |  520 | ... | 543 |
+    TP2, rank 1:
+                            |< -----------BASE----------- >|< -BASE PADDING- >|< -----------LORA PADDING----------- >|
+    corresponding token_id: | 512 | 513 | 514 | ... | 1009 | -1  | ...  | -1  |  -1  | ... |  -1  | -1  | ... |   -1 |
+                     index: |  0  |  1  |  2  | ... | 497  | 498 | ...  | 511 | 512  | ... | 519  | 520 | ... |  543 |
+
+    Args:
+        num_embeddings: vocabulary size.
+        embedding_dim: size of hidden state.
+        params_dtype: type of the parameters.
+        org_num_embeddings: original vocabulary size (without LoRA).
+        padding_size: padding size for the vocabulary.
+        quant_config: quant config for the layer
+        prefix: full name of the layer in the state dict
+    """  # noqa: E501
+
+    def __init__(
+        self,
+        num_embeddings: int,
+        embedding_dim: int,
+        *,
+        params_dtype: Optional[torch.dtype] = None,
+        org_num_embeddings: Optional[int] = None,
+        padding_size: int = DEFAULT_VOCAB_PADDING_SIZE,
+        quant_config: Optional[QuantizationConfig] = None,
+        prefix: str = "",
+        enable_tp: bool = True,
+        use_attn_tp_group: bool = False,
+        use_presharded_weights: bool = False,
+        enable_over_encoding: bool = False,
+    ):
+        super().__init__()
+        self.quant_config = quant_config
+        self.enable_over_encoding = enable_over_encoding
+
+        self.enable_tp = enable_tp
+        self.use_attn_tp_group = use_attn_tp_group
+        if self.enable_tp:
+            if use_attn_tp_group:
+                tp_rank = get_attention_tp_rank()
+                self.tp_size = get_attention_tp_size()
+            else:
+                tp_rank = get_tensor_model_parallel_rank()
+                self.tp_size = get_tensor_model_parallel_world_size()
+        else:
+            assert use_attn_tp_group is False
+            tp_rank = 0
+            self.tp_size = 1
+
+        self.num_embeddings = num_embeddings
+        self.org_vocab_size = org_num_embeddings or num_embeddings
+
+        # Support the case where the vocab size is not divisible by the TP size.
+        if (
+            _is_cpu
+            and pad_vocab_size(self.org_vocab_size, padding_size) % self.tp_size != 0
+        ):
+            padding_size *= self.tp_size
+        self.padding_size = padding_size
+
+        num_added_embeddings = num_embeddings - self.org_vocab_size
+        self.use_presharded_weights = use_presharded_weights
+        if use_presharded_weights:
+            assert (
+                num_added_embeddings == 0
+            ), "Lora is not supported with presharded weights."
+
+        self.org_vocab_size_padded = pad_vocab_size(
+            self.org_vocab_size, self.padding_size
+        )
+        self.num_embeddings_padded = pad_vocab_size(
+            self.org_vocab_size_padded + num_added_embeddings, self.padding_size
+        )
+        assert self.org_vocab_size_padded <= self.num_embeddings_padded
+
+        self.shard_indices = self._get_indices(
+            self.num_embeddings_padded,
+            self.org_vocab_size_padded,
+            self.num_embeddings,
+            self.org_vocab_size,
+            tp_rank,
+            self.tp_size,
+        )
+        self.embedding_dim = embedding_dim
+
+        quant_method = None
+        if quant_config is not None:
+            quant_method = quant_config.get_quant_method(self, prefix=prefix)
+        if quant_method is None:
+            quant_method = UnquantizedEmbeddingMethod()
+
+        # If we are making an embedding layer, then our quantization linear
+        # method must implement the embedding operation. If we are another
+        # layer type like ParallelLMHead, this is not important.
+        is_embedding_layer = type(self.__class__) is VocabParallelEmbedding
+        quant_method_implements_embedding = method_has_implemented_embedding(
+            type(quant_method)
+        )
+        if is_embedding_layer and not quant_method_implements_embedding:
+            raise NotImplementedError(
+                f"The class {type(quant_method).__name__} must implement "
+                "the 'embedding' method, see UnquantizedEmbeddingMethod."
+            )
+
+        self.quant_method: QuantizeMethodBase = quant_method
+
+        if params_dtype is None:
+            params_dtype = torch.get_default_dtype()
+        # Divide the weight matrix along the vocaburaly dimension.
+        self.num_added_embeddings = self.num_embeddings - self.org_vocab_size
+        self.num_embeddings_per_partition = divide(
+            self.num_embeddings_padded, self.tp_size
+        )
+        assert (
+            self.shard_indices.num_elements_padded == self.num_embeddings_per_partition
+        )
+        self.num_org_embeddings_per_partition = (
+            self.shard_indices.org_vocab_end_index
+            - self.shard_indices.org_vocab_start_index
+        )
+        self.num_added_embeddings_per_partition = (
+            self.shard_indices.added_vocab_end_index
+            - self.shard_indices.added_vocab_start_index
+        )
+
+        self.quant_method.create_weights(
+            self,
+            self.embedding_dim,
+            [self.num_embeddings_per_partition],
+            self.embedding_dim,
+            self.num_embeddings_padded,
+            params_dtype=params_dtype,
+            weight_loader=self.weight_loader,
+            host_tensor=(
+                True
+                if (
+                    self.__class__ is VocabParallelEmbedding
+                    and self.enable_over_encoding
+                )
+                else False
+            ),
+        )
+
+    @classmethod
+    def _get_indices(
+        cls,
+        vocab_size_padded: int,
+        org_vocab_size_padded: int,
+        vocab_size: int,
+        org_vocab_size: int,
+        tp_rank: int,
+        tp_size: int,
+    ) -> VocabParallelEmbeddingShardIndices:
+        """Get start and end indices for vocab parallel embedding, following the
+        layout outlined in the class docstring, based on the given tp_rank and
+        tp_size."""
+        num_added_embeddings_padded = vocab_size_padded - org_vocab_size_padded
+        padded_org_vocab_start_index, padded_org_vocab_end_index = (
+            vocab_range_from_global_vocab_size(org_vocab_size_padded, tp_rank, tp_size)
+        )
+        padded_added_vocab_start_index, padded_added_vocab_end_index = (
+            vocab_range_from_global_vocab_size(
+                num_added_embeddings_padded, tp_rank, tp_size, offset=org_vocab_size
+            )
+        )
+        # remove padding
+        org_vocab_start_index = min(padded_org_vocab_start_index, org_vocab_size)
+        org_vocab_end_index = min(padded_org_vocab_end_index, org_vocab_size)
+        added_vocab_start_index = min(padded_added_vocab_start_index, vocab_size)
+        added_vocab_end_index = min(padded_added_vocab_end_index, vocab_size)
+        return VocabParallelEmbeddingShardIndices(
+            padded_org_vocab_start_index,
+            padded_org_vocab_end_index,
+            padded_added_vocab_start_index,
+            padded_added_vocab_end_index,
+            org_vocab_start_index,
+            org_vocab_end_index,
+            added_vocab_start_index,
+            added_vocab_end_index,
+        )
+
+    def get_sharded_to_full_mapping(self) -> Optional[List[int]]:
+        """Get a mapping that can be used to reindex the gathered
+        logits for sampling.
+
+        During sampling, we gather logits from all ranks. The relationship
+        of index->token_id will follow the same format as outlined in the class
+        docstring. However, after the gather, we want to reindex the final
+        logits tensor to map index->token_id one-to-one (the index is always
+        equal the token_id it corresponds to). The indices returned by this
+        method allow us to do that.
+        """
+        if self.tp_size < 2:
+            return None
+
+        base_embeddings: List[int] = []
+        added_embeddings: List[int] = []
+        padding: List[int] = []
+        for tp_rank in range(self.tp_size):
+            shard_indices = self._get_indices(
+                self.num_embeddings_padded,
+                self.org_vocab_size_padded,
+                self.num_embeddings,
+                self.org_vocab_size,
+                tp_rank,
+                self.tp_size,
+            )
+            range_start = self.num_embeddings_per_partition * tp_rank
+            range_end = self.num_embeddings_per_partition * (tp_rank + 1)
+            base_embeddings.extend(
+                range(range_start, range_start + shard_indices.num_org_elements)
+            )
+            padding.extend(
+                range(
+                    range_start + shard_indices.num_org_elements,
+                    range_start + shard_indices.num_org_elements_padded,
+                )
+            )
+            added_embeddings.extend(
+                range(
+                    range_start + shard_indices.num_org_elements_padded,
+                    range_start
+                    + shard_indices.num_org_elements_padded
+                    + shard_indices.num_added_elements,
+                )
+            )
+            padding.extend(
+                range(
+                    range_start
+                    + shard_indices.num_org_elements_padded
+                    + shard_indices.num_added_elements,
+                    range_start
+                    + shard_indices.num_org_elements_padded
+                    + shard_indices.num_added_elements_padded,
+                )
+            )
+            assert (
+                range_start
+                + shard_indices.num_org_elements_padded
+                + shard_indices.num_added_elements_padded
+                == range_end
+            )
+        ret = base_embeddings + added_embeddings + padding
+        assert len(ret) == self.num_embeddings_padded
+        return ret
+
+    def weight_loader(self, param: Parameter, loaded_weight: torch.Tensor):
+        output_dim = getattr(param, "output_dim", None)
+        packed_dim = getattr(param, "packed_dim", None)
+
+        # If the parameter is a gguf weight, then load it directly.
+        if getattr(param, "is_gguf_weight_type", None):
+            param.data.copy_(loaded_weight)
+            param.weight_type = loaded_weight.item()
+            return
+        elif isinstance(param, UninitializedParameter):
+            shape = list(loaded_weight.shape)
+            if output_dim is not None:
+                shape[output_dim] = shape[output_dim] // self.tp_size
+            param.materialize(tuple(shape), dtype=loaded_weight.dtype)
+
+        # If parameter does not have output dim, then it should
+        # be copied onto all gpus (e.g. g_idx for act_order gptq).
+        if output_dim is None:
+            assert param.data.shape == loaded_weight.shape
+            param.data.copy_(loaded_weight)
+            return
+
+        # Shard indexes for loading the weight
+        start_idx = self.shard_indices.org_vocab_start_index
+        shard_size = self.shard_indices.org_vocab_end_index - start_idx
+
+        # If param packed on the same dim we are sharding on, then
+        # need to adjust offsets of loaded weight by pack_factor.
+        if packed_dim is not None and packed_dim == output_dim:
+            packed_factor = (
+                param.packed_factor
+                if isinstance(param, BasevLLMParameter)
+                else param.packed_factor
+            )
+            assert loaded_weight.shape[output_dim] == (
+                self.org_vocab_size // param.packed_factor
+            )
+            start_idx = start_idx // packed_factor
+            shard_size = shard_size // packed_factor
+        else:
+            assert loaded_weight.shape[output_dim] == (
+                self.org_vocab_size
+                // (self.tp_size if self.use_presharded_weights else 1)
+            ), f"{self.org_vocab_size=} {self.use_presharded_weights=} {loaded_weight.shape[output_dim]=}"
+
+        # Copy the data.
+        if not self.use_presharded_weights:
+            loaded_weight = loaded_weight.narrow(output_dim, start_idx, shard_size)
+        param[: loaded_weight.shape[0]].data.copy_(loaded_weight)
+        param[loaded_weight.shape[0] :].data.fill_(0)
+
+    def forward(self, input_):
+        if self.tp_size > 1:
+            # Build the mask.
+            masked_input, input_mask = get_masked_input_and_mask(
+                input_,
+                self.shard_indices.org_vocab_start_index,
+                self.shard_indices.org_vocab_end_index,
+                self.shard_indices.num_org_vocab_padding,
+                self.shard_indices.added_vocab_start_index,
+                self.shard_indices.added_vocab_end_index,
+            )
+        else:
+            masked_input = input_
+
+        # Get the embeddings.
+        with use_symmetric_memory(
+            get_tp_group(), disabled=not is_allocation_symmetric()
+        ):
+            output_parallel = self.quant_method.embedding(self, masked_input.long())
+
+        if self.tp_size > 1:
+            # Mask the output embedding.
+            output_parallel.masked_fill_(input_mask.unsqueeze(-1), 0)
+            if not get_attn_tp_context().input_scattered:
+                if self.use_attn_tp_group:
+                    output_parallel = attn_tp_all_reduce(output_parallel)
+                else:
+                    # Reduce across all the model parallel GPUs.
+                    output_parallel = tensor_model_parallel_all_reduce(output_parallel)
+        return output_parallel
+
+    def extra_repr(self) -> str:
+        s = f"num_embeddings={self.num_embeddings_per_partition}"
+        s += f", embedding_dim={self.embedding_dim}"
+        s += f", org_vocab_size={self.org_vocab_size}"
+        s += f", num_embeddings_padded={self.num_embeddings_padded}"
+        if self.enable_tp:
+            s += f", tp_size={self.tp_size}"
+        return s
+
+
+class KLXVocabParallelEmbedding(torch.nn.Module):
+
+    def __init__(
+        self,
+        num_embeddings: int,
+        embedding_dim: int,
+        params_dtype: Optional[torch.dtype] = None,
+        org_num_embeddings: Optional[int] = None,
+        padding_size: int = DEFAULT_VOCAB_PADDING_SIZE,
+        quant_config: Optional[QuantizationConfig] = None,
+        prefix: str = "",
+        enable_tp: bool = True,
+        use_attn_tp_group: bool = False,
+        use_presharded_weights: bool = False,
+    ):
+        super().__init__()
+        self.quant_config = quant_config
+
+        self.enable_tp = enable_tp
+        self.use_attn_tp_group = use_attn_tp_group
+        if self.enable_tp:
+            if use_attn_tp_group:
+                self.tp_rank = get_attention_tp_rank()
+                self.tp_size = get_attention_tp_size()
+            else:
+                self.tp_rank = get_tensor_model_parallel_rank()
+                self.tp_size = get_tensor_model_parallel_world_size()
+        else:
+            assert use_attn_tp_group is False
+            self.tp_rank = 0
+            self.tp_size = 1
+        # vocal_size, embedding_dim -> vocal_size, embedding_dim // tp_size
+        # P800 workaround: keep the embedding weight replicated on every TP rank
+        # to avoid the unstable all_gather across the BKCL/XCCL stack.
+        self.local_dim = embedding_dim
+
+        self.num_embeddings = num_embeddings
+        self.padding_size = padding_size
+        self.org_vocab_size = org_num_embeddings or num_embeddings
+        num_added_embeddings = num_embeddings - self.org_vocab_size
+        self.use_presharded_weights = use_presharded_weights
+        if use_presharded_weights:
+            assert (
+                num_added_embeddings == 0
+            ), "Lora is not supported with presharded weights."
+
+        self.embedding_dim = embedding_dim
+
+        """Create weights for embedding layer."""
+        weight = Parameter(
+            torch.empty(
+                (self.num_embeddings, self.local_dim),
+                dtype=params_dtype,
+            ),
+            requires_grad=False,
+        )
+        set_weight_attrs(
+            weight,
+            {"input_dim": 1, "output_dim": 0, "weight_loader": self.weight_loader},
+        )
+        self.register_parameter("weight", weight)
+
+    def weight_loader(self, param: Parameter, loaded_weight: torch.Tensor):
+        # With the replicated-weight workaround, every rank loads the full weight.
+        param.copy_(loaded_weight)
+
+    def forward(self, input_):
+        hidden_states = F.embedding(input_, self.weight)
+        # No all_gather needed because the full embedding weight is replicated.
+        return hidden_states
+
+    def extra_repr(self) -> str:
+        s = f"num_embeddings={self.num_embeddings}"
+        s += f", embedding_dim={self.embedding_dim}"
+        s += f", org_vocab_size={self.org_vocab_size}"
+        if self.enable_tp:
+            s += f", tp_size={self.tp_size}"
+        return s
+
+
+class ParallelLMHead(VocabParallelEmbedding):
+    """Parallelized LM head.
+
+    Output logits weight matrices used in the Sampler. The weight and bias
+    tensors are padded to make sure they are divisible by the number of
+    model parallel GPUs.
+
+    Args:
+        num_embeddings: vocabulary size.
+        embedding_dim: size of hidden state.
+        bias: whether to use bias.
+        params_dtype: type of the parameters.
+        org_num_embeddings: original vocabulary size (without LoRA).
+        padding_size: padding size for the vocabulary.
+    """
+
+    def __init__(
+        self,
+        num_embeddings: int,
+        embedding_dim: int,
+        *,
+        bias: bool = False,
+        params_dtype: Optional[torch.dtype] = None,
+        org_num_embeddings: Optional[int] = None,
+        padding_size: int = DEFAULT_VOCAB_PADDING_SIZE,
+        quant_config: Optional[QuantizationConfig] = None,
+        prefix: str = "",
+        use_attn_tp_group: bool = False,
+        use_presharded_weights: bool = False,
+        enable_over_encoding: bool = False,
+    ):
+        super().__init__(
+            num_embeddings,
+            embedding_dim,
+            params_dtype=params_dtype,
+            org_num_embeddings=org_num_embeddings,
+            padding_size=padding_size,
+            quant_config=quant_config,
+            prefix=prefix,
+            use_attn_tp_group=use_attn_tp_group,
+            use_presharded_weights=use_presharded_weights,
+            enable_over_encoding=enable_over_encoding,
+        )
+        self.quant_config = quant_config
+
+        # We only support pack LMHead if it's not quantized.
+        if _is_cpu and _is_cpu_amx_available:
+            if hasattr(self, "weight") and self.weight.dtype in [
+                torch.bfloat16,
+                torch.float16,
+            ]:
+                self.quant_method = PackWeightMethod(weight_names=["weight"])
+
+        if bias:
+            self.bias = Parameter(
+                torch.empty(self.num_embeddings_per_partition, dtype=params_dtype)
+            )
+            set_weight_attrs(
+                self.bias,
+                {
+                    "output_dim": 0,
+                    "weight_loader": self.weight_loader,
+                },
+            )
+        else:
+            self.register_parameter("bias", None)
+
+    def tie_weights(self, embed_tokens: VocabParallelEmbedding):
+        """Tie the weights with word embeddings."""
+        # GGUF quantized embed_tokens.
+        if self.quant_config and self.quant_config.get_name() == "gguf":
+            return embed_tokens
+        else:
+            self.weight = embed_tokens.weight
+            return self
+
+    def forward(self, input_):
+        del input_
+        raise RuntimeError("LMHead's weights should be used in the sampler.")
--- a/scripts/analysis/compare_experiments.py
+++ b/scripts/analysis/compare_experiments.py
@ -0,0 +1,77 @@
+#!/usr/bin/env python3
+"""Load results.json from multiple experiments and produce a comparison report.
+
+Usage:
+    python3 scripts/analysis/compare_experiments.py
+    python3 scripts/analysis/compare_experiments.py --output comparison.md
+"""
+
+import argparse
+import json
+from collections import defaultdict
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+
+
+def load_results():
+    """Glob experiments/*/results/*/results.json and load them."""
+    results = []
+    for path in sorted(REPO_ROOT.glob("experiments/*/results/*/results.json")):
+        try:
+            with open(path, "r", encoding="utf-8") as f:
+                data = json.load(f)
+            data["_source"] = str(path.relative_to(REPO_ROOT))
+            results.append(data)
+        except (json.JSONDecodeError, OSError) as e:
+            print(f"WARN: failed to load {path}: {e}")
+    return results
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Compare benchmark experiments.")
+    parser.add_argument("--output", "-o", default="comparison_report.md", help="Output markdown file")
+    args = parser.parse_args()
+
+    results = load_results()
+    if not results:
+        print("No experiments found under experiments/*/results/*/results.json")
+        return
+
+    # Group by scenario name.
+    by_scenario = defaultdict(list)
+    for data in results:
+        meta = data.get("metadata", {})
+        for scenario in data.get("scenarios", []):
+            key = scenario.get("name", "unknown")
+            by_scenario[key].append((meta, scenario))
+
+    output_path = Path(args.output)
+    with open(output_path, "w", encoding="utf-8") as f:
+        f.write("# Cross-Experiment Comparison\n\n")
+        f.write("| Experiment | Chip | Engine | Scenario | Conc | In/Out | Req/s | OutTok/s | TTFT p99 | TPOT p99 | E2E p99 |\n")
+        f.write("|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|\n")
+
+        for scenario_name in sorted(by_scenario.keys()):
+            for meta, scenario in by_scenario[scenario_name]:
+                lat = scenario.get("latencies", {})
+                f.write(
+                    f"| {meta.get('experiment', '')} "
+                    f"| {meta.get('chip', '')} "
+                    f"| {meta.get('engine', '')} "
+                    f"| {scenario_name} "
+                    f"| {scenario.get('concurrency', '')} "
+                    f"| {scenario.get('input_len', '')}/{scenario.get('output_len', '')} "
+                    f"| {scenario.get('request_throughput') or ''} "
+                    f"| {scenario.get('output_token_throughput') or ''} "
+                    f"| {lat.get('ttft_ms', {}).get('p99') or ''} "
+                    f"| {lat.get('tpot_ms', {}).get('p99') or ''} "
+                    f"| {lat.get('e2e_ms', {}).get('p99') or ''} |\n"
+                )
+
+    print(f"Loaded {len(results)} experiment runs.")
+    print(f"Wrote {output_path}")
+
+
+if __name__ == "__main__":
+    main()
--- a/scripts/common/bench_client_docker.sh
+++ b/scripts/common/bench_client_docker.sh
@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Helpers to run sglang.bench_serving inside the P800 Docker container.
+# Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/bench_client_docker.sh"
+
+set -Eeuo pipefail
+
+_BENCH_CLIENT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=/dev/null
+source "${_BENCH_CLIENT_DIR}/lib.sh"
+# shellcheck source=/dev/null
+source "${_BENCH_CLIENT_DIR}/platform.sh"
+
+# Required platform variables: CONTAINER_NAME, CONTAINER_PYTHON
+
+# Run sglang.bench_serving inside the running container.
+# All arguments are forwarded to bench_serving.
+run_bench_in_container() {
+  local container="${CONTAINER_NAME}"
+
+  if ! docker inspect "$container" >/dev/null 2>&1; then
+    log "ERROR: container ${container} is not running"
+    return 1
+  fi
+
+  log "running bench_serving in container ${container}"
+  docker exec "$container" \
+    env HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1 \
+    "${CONTAINER_PYTHON}" -m sglang.bench_serving "$@"
+}
+
+# Convenience wrapper for a single random-dataset case.
+# Args:
+#   $1: backend (e.g. sglang)
+#   $2: port
+#   $3: model path or served model name
+#   $4: output jsonl path (inside the container)
+#   $5: concurrency
+#   $6: input length
+#   $7: output length
+#   $8: num prompts (optional, default 512)
+#   $9: dataset path inside container (optional)
+run_random_case() {
+  local backend="$1"
+  local port="$2"
+  local model="$3"
+  local output_file="$4"
+  local concurrency="$5"
+  local input_len="$6"
+  local output_len="$7"
+  local num_prompts="${8:-512}"
+  local dataset_path="${9:-}"
+  local warmup="${WARMUP:-100}"
+
+  # Ensure the output directory exists inside the container.
+  docker exec "${CONTAINER_NAME}" mkdir -p "$(dirname "$output_file")"
+
+  local args=(
+    --backend "$backend"
+    --host 127.0.0.1
+    --port "$port"
+    --model "$model"
+    --dataset-name random
+    --random-input-len "$input_len"
+    --random-output-len "$output_len"
+    --num-prompts "$num_prompts"
+    --max-concurrency "$concurrency"
+    --warmup-requests "$warmup"
+    --output-file "$output_file"
+    --output-details
+  )
+  if [[ -n "$dataset_path" ]]; then
+    args+=(--dataset-path "$dataset_path")
+  fi
+
+  run_bench_in_container "${args[@]}"
+}
--- a/scripts/common/lib.sh
+++ b/scripts/common/lib.sh
@ -0,0 +1,153 @@
+#!/usr/bin/env bash
+# Common helpers for benchmark orchestrators.
+# Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/lib.sh"
+
+set -Eeuo pipefail
+
+# Resolve repository root relative to this file.
+_COMMON_LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT_DIR="$(cd "${_COMMON_LIB_DIR}/../.." && pwd)"
+
+# -----------------------------------------------------------------------------
+# Logging
+# -----------------------------------------------------------------------------
+
+LOG_FILE=""
+
+log_init() {
+  local log_path="$1"
+  LOG_FILE="$log_path"
+  mkdir -p "$(dirname "$LOG_FILE")"
+  : > "$LOG_FILE"
+}
+
+log() {
+  local msg="[$(date --iso-8601=seconds)] $*"
+  echo "$msg"
+  if [[ -n "${LOG_FILE:-}" ]]; then
+    echo "$msg" >> "$LOG_FILE"
+  fi
+}
+
+# -----------------------------------------------------------------------------
+# Result directories
+# -----------------------------------------------------------------------------
+
+ensure_result_root() {
+  local result_root="$1"
+  mkdir -p "${result_root}/logs"
+  mkdir -p "${result_root}/raw_outputs"
+  echo "$result_root"
+}
+
+# -----------------------------------------------------------------------------
+# Server health check
+# -----------------------------------------------------------------------------
+
+health_check() {
+  local host="${1:-127.0.0.1}"
+  local port="${2:-30000}"
+  local max_wait="${3:-120}"
+
+  for ((i = 1; i <= max_wait; i++)); do
+    if curl --fail --silent --show-error --max-time 5 "http://${host}:${port}/health" >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep 1
+  done
+  return 1
+}
+
+# -----------------------------------------------------------------------------
+# Git metadata
+# -----------------------------------------------------------------------------
+
+git_commit() {
+  cd "$ROOT_DIR" || return 1
+  git rev-parse --short HEAD 2>/dev/null || echo "unknown"
+}
+
+git_dirty() {
+  cd "$ROOT_DIR" || return 1
+  if [[ -n "$(git status --porcelain 2>/dev/null)" ]]; then
+    echo "dirty"
+  else
+    echo "clean"
+  fi
+}
+
+# -----------------------------------------------------------------------------
+# results.json metadata
+# -----------------------------------------------------------------------------
+
+write_metadata_json() {
+  local output_path="$1"
+  local experiment="$2"
+  local run_id="$3"
+  local model="$4"
+  local backend="$5"
+  local engine="$6"
+  local hardware="$7"
+  local accelerator="$8"
+  local chip="$9"
+  local script="${10}"
+  local env_path="${11:-}"
+  local description="${12:-}"
+
+  mkdir -p "$(dirname "$output_path")"
+
+  cat > "$output_path" <<EOF
+{
+  "metadata": {
+    "experiment": "${experiment}",
+    "run_id": "${run_id}",
+    "timestamp": "$(date --iso-8601=seconds)",
+    "model": "${model}",
+    "backend": "${backend}",
+    "engine": "${engine}",
+    "hardware": "${hardware}",
+    "accelerator": "${accelerator}",
+    "chip": "${chip}",
+    "script": "${script}",
+    "env": "${env_path}",
+    "git_commit": "$(git_commit)",
+    "git_dirty": "$(git_dirty)",
+    "description": "${description}"
+  },
+  "config": {},
+  "scenarios": []
+}
+EOF
+}
+
+# -----------------------------------------------------------------------------
+# JSON append helper (naive but sufficient for small result JSONs)
+# -----------------------------------------------------------------------------
+
+append_scenario_to_json() {
+  local json_path="$1"
+  local scenario_json="$2"
+
+  if [[ ! -f "$json_path" ]]; then
+    echo "ERROR: $json_path not found" >&2
+    return 1
+  fi
+
+  # Use Python for safe JSON manipulation.
+  "${PYTHON:-python3}" - "$json_path" "$scenario_json" <<'PY'
+import json
+import sys
+
+json_path = sys.argv[1]
+scenario_json = sys.argv[2]
+
+with open(json_path, "r", encoding="utf-8") as f:
+    data = json.load(f)
+
+scenario = json.loads(scenario_json)
+data["scenarios"].append(scenario)
+
+with open(json_path, "w", encoding="utf-8") as f:
+    json.dump(data, f, indent=2, ensure_ascii=False)
+PY
+}
--- a/scripts/common/platform.sh
+++ b/scripts/common/platform.sh
@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+# Platform loader.
+# Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/platform.sh"
+
+set -Eeuo pipefail
+
+_PLATFORM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT_DIR="$(cd "${_PLATFORM_DIR}/../.." && pwd)"
+
+# Auto-detect platform if not set.
+if [[ -z "${PLATFORM:-}" ]]; then
+  if lspci 2>/dev/null | grep -qiE "XPU|Kunlun"; then
+    PLATFORM="kunlun_p800"
+  elif command -v nvidia-smi >/dev/null 2>&1; then
+    PLATFORM="nvidia_h200"
+  else
+    echo "ERROR: Could not auto-detect PLATFORM. Set PLATFORM env var explicitly." >&2
+    exit 1
+  fi
+  echo "Auto-detected platform: ${PLATFORM}"
+fi
+
+PLATFORM_FILE="${ROOT_DIR}/platforms/${PLATFORM}.env"
+if [[ ! -f "$PLATFORM_FILE" ]]; then
+  echo "ERROR: Platform config not found: ${PLATFORM_FILE}" >&2
+  exit 1
+fi
+
+# shellcheck source=/dev/null
+source "$PLATFORM_FILE"
+
+# Export key variables if not already set.
+export CHIP="${CHIP:-$PLATFORM}"
+export ACCELERATOR="${ACCELERATOR:-$PLATFORM}"
+export HARDWARE="${HARDWARE:-$PLATFORM}"
+export DEFAULT_PORT="${DEFAULT_PORT:-30000}"
+export MODEL_ROOT="${MODEL_ROOT:-/data1/models}"
--- a/scripts/common/server_docker.sh
+++ b/scripts/common/server_docker.sh
@ -0,0 +1,205 @@
+#!/usr/bin/env bash
+# Docker-based SGLang server lifecycle helpers for Kunlun P800.
+# Usage: source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/../scripts/common/server_docker.sh"
+
+set -Eeuo pipefail
+
+_DOCKER_COMMON_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=/dev/null
+source "${_DOCKER_COMMON_DIR}/lib.sh"
+# shellcheck source=/dev/null
+source "${_DOCKER_COMMON_DIR}/platform.sh"
+
+# Required platform variables: DOCKER_IMAGE, CONTAINER_NAME, DEFAULT_PORT, PATCH_ROOT
+
+docker_server_status() {
+  docker inspect -f '{{.State.Status}}' "$CONTAINER_NAME" 2>/dev/null || echo "missing"
+}
+
+docker_server_stop() {
+  local container="${1:-$CONTAINER_NAME}"
+  log "stopping container ${container}"
+  docker rm -f "$container" 2>/dev/null || true
+
+  # Clean up leftover shared memory on the host.
+  ipcs -m 2>/dev/null | awk '$4 == 666 {print $2}' | while read -r shmid; do
+    ipcrm -m "$shmid" 2>/dev/null || true
+  done
+
+  # Kill lingering sglang processes just in case.
+  pkill -9 -f 'sglang.launch_server' 2>/dev/null || true
+  pkill -9 -f 'multiprocessing.spawn' 2>/dev/null || true
+  pkill -9 -f 'multiprocessing.resource_tracker' 2>/dev/null || true
+  sleep 2
+}
+
+# Build device args for docker run (8 XPU devices).
+_docker_device_args() {
+  local args=""
+  for i in 0 1 2 3 4 5 6 7; do
+    args="${args} --device /dev/xpu${i}:/dev/xpu${i}"
+  done
+  args="${args} --device /dev/xpuctrl:/dev/xpuctrl"
+  echo "$args"
+}
+
+# Start the SGLang server container for DeepSeek-V4 on P800.
+# Args:
+#   $1: model path on host (e.g. /data1/models/DeepSeek-V4-Flash-INT8)
+#   $2: port (optional, defaults to DEFAULT_PORT)
+#   $3: server log path (optional)
+#   $4: server mode (optional): "fp8" or "w8a8_int8" (default "fp8")
+docker_server_start() {
+  local model_path="$1"
+  local port="${2:-$DEFAULT_PORT}"
+  local server_log="${3:-${RESULT_ROOT:-/tmp}/logs/server.outer.log}"
+  local mode="${4:-fp8}"
+  local container="${CONTAINER_NAME}"
+  local image="${DOCKER_IMAGE}"
+
+  mkdir -p "$(dirname "$server_log")"
+
+  log "starting container ${container} from ${image}"
+  log "model path: ${model_path} -> /models"
+  log "port: ${port}"
+  log "mode: ${mode}"
+  log "server log: ${server_log}"
+
+  docker_server_stop "$container"
+
+  local device_args
+  device_args="$(_docker_device_args)"
+
+  # Ensure patch files exist under PATCH_ROOT. Fall back to /tmp if the repo
+  # copy is not present (legacy path).
+  local patch_root="${PATCH_ROOT:-/tmp}"
+
+  # Common environment variables shared by all modes.
+  local env_args=(
+    -e XPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+    -e CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
+    -e CUDA_DEVICE_ORDER=OAM_ID
+    -e SGLANG_USE_TRANSFORMERS_V5_TOKENIZER=1
+    -e XMLIR_FORCE_USE_XPU_GRAPH=1
+    -e SGLANG_DSV4_MODE=2604
+    -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+    -e SGLANG_NSA_DUAL_STREAM=true
+    -e SGLANG_NSA_QUANT_WQ_B_WK=false
+    -e SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1
+    -e SGLANG_OPT_DEEPGEMM_HC_PRENORM=false
+    -e SGLANG_OPT_USE_TILELANG_MHC_PRE=1
+    -e SGLANG_OPT_USE_TILELANG_MHC_POST=1
+    -e SGLANG_CLEAN_REQUEST_WHEN_RETRACT=1
+    -e SGLANG_SET_CPU_AFFINITY=1
+    -e SGLANG_OPT_USE_KLX_TOPK_KERNEL=1
+    -e XSGL_INTERTYPE_BFP16=1
+    -e ENABLE_FAST_BFP16_ATTN=1
+    -e XSGL_USE_DEEP_GEMM_BMM=1
+    -e XSGL_XDNN_QUANT=1
+    -e XSGL_FUSE_RMS_NORM_QUANT=1
+    -e XSGL_TRANSPOSE_MATMUL_WEIGHT=1
+    -e XINFER_QUANT_SDNN=1
+    -e XSGL_USE_MOE_SIGMOID_GROUP_TOPK_NORM=1
+    -e XSGL_EARLY_FIRST_TOKEN=1
+    -e USE_FAST_BFP16_MOE=1
+    -e XSGL_ENABLE_TGEMM_FP16=1
+    -e SGLANG_ENABLE_SPEC_V2=True
+    -e SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+    -e PYTHONDONTWRITEBYTECODE=1
+    -e XTORCH_OPS_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/xtorch_ops
+    -e XPU_RUNTIME_LIB_DIR=/root/miniconda/envs/python310_torch25_cuda/xcudart/lib
+    -e BKCL_TREE_THRESHOLD=1048576
+    -e CUDA_ENABLE_P2P_NO_UVA=1
+    -e NCCL_IB_GID_INDEX=3
+    -e IS_DSV4=1
+    -e MC_CUSTOM_TOPO_JSON=/workspace/nic_priority_matrix_test.json
+  )
+
+  # Mode-specific environment variables.
+  if [[ "$mode" == "w8a8_int8" ]]; then
+    env_args+=(
+      -e SGLANG_DSV4_FP4_EXPERTS=false
+      -e SGLANG_APPLY_CONFIG_BACKUP=auto
+      -e BKCL_ENABLE_XDR=1
+      -e BKCL_RDMA_NICS=eth1,eth1,eth3,eth3,eth5,eth5,eth7,eth7
+      -e BKCL_RDMA_VERBS=1
+      -e XSGL_INT8_LM_HEAD=1
+      -e SGLANG_P800_ALL_GATHER_FALLBACK=0
+    )
+  else
+    env_args+=(
+      -e SGLANG_DSV4_FP4_EXPERTS=true
+      -e SGLANG_OPT_FUSE_WQA_WKV=false
+      -e SGLANG_DSV4_2604_SUBMODE=2604B
+      -e SGLANG_APPLY_CONFIG_BACKUP=small
+      -e BKCL_ENABLE_XDR=0
+      -e BKCL_FORCE_PCIE_P2P=1
+      -e BKCL_ENABLE_PCIE_P2P=1
+      -e BKCL_SOCKET_IFNAME=bond0
+      -e BKCL_NET_ENABLE_INTRA=1
+      -e XSHMEM_MODE=0
+      -e SGLANG_P800_ALL_GATHER_FALLBACK=1
+    )
+  fi
+
+  # Mode-specific launch arguments.
+  local launch_args
+  if [[ "$mode" == "w8a8_int8" ]]; then
+    launch_args="--host 0.0.0.0 --port ${port} --model-path /models --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --disable-custom-all-reduce --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.8 --max-prefill-tokens 16384 --max-running-requests 64 --tensor-parallel-size 8 --ep-size 8 --disable-shared-experts-fusion --quantization w8a8_int8 --kv-cache-dtype float16 --disable-piecewise-cuda-graph --cuda-graph-max-bs 32 --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --constrained-json-disable-any-whitespace --enable-metrics --enable-request-time-stats-logging"
+  else
+    launch_args="--host 0.0.0.0 --port ${port} --model-path /models --quantization fp8 --attention-backend nsa --nsa-prefill klxdsa --nsa-decode klxdsa --trust-remote-code --chunked-prefill-size 8192 --page-size 64 --mem-fraction-static 0.85 --max-prefill-tokens 16384 --context-length 65536 --max-running-requests 8 --max-total-tokens 524288 --tensor-parallel-size 8 --ep-size 8 --moe-runner-backend deep_gemm --disable-shared-experts-fusion --kv-cache-dtype float16 --disable-piecewise-cuda-graph --disable-cuda-graph --watchdog-timeout 3000000 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --enable-metrics --enable-request-time-stats-logging"
+  fi
+
+  # Build the full server bootstrap command and base64-encode it to avoid
+  # host-shell quoting hell.
+  local server_cmd
+  server_cmd=$(cat <<EOF
+cd /workspace
+find /root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
+/root/miniconda/envs/python310_torch25_cuda/bin/pip install --upgrade safetensors -q
+/root/miniconda/envs/python310_torch25_cuda/bin/pip install https://files.pythonhosted.org/packages/14/8b/2a1333a6455c6fad401c2285dee6f58016c55b1cb44cae3a31f8a9cc7d83/apache_tvm_ffi-0.1.0b2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl -q
+/root/miniconda/envs/python310_torch25_cuda/bin/python -c "import torch; torch.float8_e8m0fnu = torch.uint8; import runpy, sys; sys.argv[0] = 'sglang.launch_server'; runpy.run_module('sglang.launch_server', run_name='__main__')" ${launch_args}
+EOF
+)
+  local server_cmd_b64
+  server_cmd_b64=$(printf '%s' "$server_cmd" | base64 -w0)
+
+  # Patch mounts. For w8a8_int8 the working container relies on patches baked
+  # into the image, so we only mount the runtime config, nic matrix, and a
+  # local dummy dataset for offline benchmarking.
+  local patch_mounts=(
+    -v "${patch_root}/nic_priority_matrix_test.json:/workspace/nic_priority_matrix_test.json:ro"
+    -v "${patch_root}/dummy_sharegpt.json:/workspace/dummy_sharegpt.json:ro"
+  )
+  if [[ "$mode" == "fp8" ]]; then
+    patch_mounts+=(
+      -v "${patch_root}/sitecustomize_xpu.py:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sitecustomize.py:ro"
+      -v "${patch_root}/hf_transformers_utils.py.patched:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/utils/hf_transformers_utils.py:ro"
+      -v "${patch_root}/fp8_utils.py.patched:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/layers/quantization/fp8_utils.py:ro"
+      -v "${patch_root}/config_backup_small_fp8.json:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/configs/config_backup_small.json:ro"
+      -v "${patch_root}/parallel_state.py:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/distributed/parallel_state.py:ro"
+      -v "${patch_root}/vocab_parallel_embedding.py:/root/miniconda/envs/python310_torch25_cuda/lib/python3.10/site-packages/sglang/srt/layers/vocab_parallel_embedding.py:ro"
+    )
+  fi
+
+  docker run -d \
+    --name "${container}" \
+    --privileged \
+    --network host \
+    --ipc host \
+    ${device_args} \
+    -v "${model_path}:/models:ro" \
+    "${patch_mounts[@]}" \
+    "${env_args[@]}" \
+    "${image}" \
+    bash -c "echo '${server_cmd_b64}' | base64 -d | bash" \
+    >> "${server_log}" 2>&1
+
+  log "container ${container} started, waiting for health"
+  if health_check 127.0.0.1 "$port" 600; then
+    log "container ${container} is healthy"
+  else
+    log "ERROR: container ${container} failed health check"
+    return 1
+  fi
+}