This experiment is the real-serving acceptance gate for Kimi-K3 FlashInfer MXFP4 MoE on NVIDIA SM120. It does not modify the existing baseline experiment.

Fixed matrix

Dimension	Values
Nodes	601-604, 8 GPUs per node
Parallelism	TP32, EP4, DP1
MoE runner	`marlin`, `flashinfer_mxfp4`
Input / output	16K / 1 token
Concurrency	8, 16
Chunked prefill	8K, 16K
Requests	40 per repeat
Repeats	3

The eight configurations are identical except for MoE runner, chunk size, and benchmark concurrency. The directory retains its historical tp32ep32 name so existing evidence paths remain stable; the production comparison and the default EP_SIZE are EP4. The client uses SGLang random-ids, which produces the exact synthetic token length without depending on a ShareGPT file or text truncation. Radix cache is disabled. OSL=1 makes TTFT, E2E latency and input TPS the primary metrics; TPOT is not meaningful for this matrix.

The defaults above can be narrowed for a capacity-first run with EP_SIZE, BACKENDS_CSV, CHUNKED_PREFILL_SIZES_CSV, and CONCURRENCIES_CSV. For example, the TP32/EP4 maximum-pressure probe uses Marlin, 16K chunk, and C=16 before spending time on the complete backend comparison.

Accepted EP4 result

Run ID: kimi3-ep4-moe-full-20260818-151349.

All 24 repeat files completed 40/40 requests with empty error arrays. No OOM, Traceback, NCCL error, or EngineDeadError was found. Values below are medians across three repeats.

Chunk	C	Backend	Input TPS	TTFT P50	TTFT P95
8K	8	Marlin	2935.02	43.51 s	46.22 s
8K	8	FlashInfer MXFP4	3257.96	39.19 s	41.62 s
8K	16	Marlin	2934.90	87.09 s	89.86 s
8K	16	FlashInfer MXFP4	3260.14	78.39 s	80.88 s
16K	8	Marlin	2812.91	45.47 s	45.86 s
16K	8	FlashInfer MXFP4	3027.50	42.27 s	42.63 s
16K	16	Marlin	2812.33	90.97 s	91.36 s
16K	16	FlashInfer MXFP4	3027.34	84.52 s	84.85 s

FlashInfer improves Input TPS by 11.00%-11.08% with an 8K chunk and by 7.63%-7.65% with a 16K chunk. It reduces median TTFT by 9.92%-9.99% and 7.04%-7.09%, respectively. The accepted Prefill setting is therefore flashinfer_mxfp4, EP4, and an 8K chunk for this 16K/1 workload.

The only matching existing EP32 evidence is Marlin/8K/C8 with two repeats. Its median Input TPS is 2531.43 and TTFT P50/P95 are 50.52/53.64 seconds. Against that evidence, EP4 raises Input TPS by 15.94% and lowers TTFT P50 by 13.88%. EP32 was not rerun.

Unique entrypoint

Run on 601 only. The sudo password is supplied at runtime and is never written to Git or result files.

cd /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill

read -rsp 'sudo password: ' SUDO_PASSWORD; echo
export SUDO_PASSWORD
export RUN_ID=kimi3-moe-prefill-$(date +%Y%m%d-%H%M%S)

bash run_moe_backend_prefill.sh build-image
bash run_moe_backend_prefill.sh run

For an unattended run, use the same entrypoint under tmux:

umask 077
printf '%s\n' "$SUDO_PASSWORD" > /tmp/kimi3-phase5-sudo
unset SUDO_PASSWORD

tmux new-session -d -s kimi3-moe-prefill \
  "cd /data/hzy/sskj/experiments/pro6000/kimi3_pro6000_sglang_tp32ep32_moe_backend_prefill && \
   SUDO_PASSWORD_FILE=/tmp/kimi3-phase5-sudo RUN_ID=\"${RUN_ID}\" \
   bash run_moe_backend_prefill.sh all 2>&1 | tee /data/hzy/${RUN_ID}.log"

Delete /tmp/kimi3-phase5-sudo after the run. The file is mode 0600 and is never copied to result directories.

Outputs

results/<RUN_ID>/
  build/            per-node image build and smoke evidence
  service/          exact docker commands and all four server logs
  raw/              one benchmark JSONL per repeat
  bench/            human-readable benchmark logs
  gpu/              per-node GPU snapshots
  results.csv       flat repeat-level metrics
  summary.json      median metrics and FlashInfer-vs-Marlin deltas
  orchestrator.log  lifecycle and progress

The run aborts on a failed service start, any failed request, a missing result, or an image capability smoke failure. Containers are removed on exit.

The candidate image preserves the Kimi base image's dependency set. It installs the patched FlashInfer wheel/AOT libraries and applies a minimal diff against the image's exact two SGLang source files; it does not replace the complete SGLang Python tree or require a newer sglang-kernel package.

Audited software identity and upstream boundary

The accepted run used the following immutable candidate identity:

image: local/sglang:kimi-k3-sm120-flashinfer-mxfp4-phase5
image ID: sha256:0b8e8ddceccecc0daf10b59ddd4177036affc81facc8bf87ea9e2456ab1e96b9
SGLang: 0.5.16
flashinfer-python: 0.6.18 (candidate source wheel)
flashinfer-cubin: 0.6.15.post1
flashinfer-jit-cache: 0.6.15.post1+cu130
Kimi SiTU: beta=4.0, linear_beta=25.0

These results remain valid for that candidate. They do not mean stock FlashInfer 0.6.17 supports the complete Kimi SM120 combination. The CUTLASS SiTU kernel is being upstreamed in FlashInfer PR #4460. The final SGLang change depends on that API and retains only Kimi layout/parameter integration, noncontiguous input handling, the SM120 attention-residual guard, and tests. See ../kimi3_pro6000_sglang_sm120_flashinfer_mxfp4/UPSTREAM_DUPLICATION_AUDIT.md.