# MiniMax-H3 / SGLang automated profile

The suite is scoped to one 8-GPU RTX 6000D host and never kills unrelated GPU processes.
It aborts if a GPU process or required port already exists. Every phase has a `DONE` marker;
reuse the same `RUN_ID` to resume after interruption.

Smoke validation:

```bash
MODE=smoke RUN_ID=h3-profile-smoke bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
```

Lightweight single-sample stage profile (recommended first pass):

```bash
MODE=light LIGHT_SCENARIO=RVA LIGHT_RESIDENT_INSTANCES=4 LIGHT_SAMPLES=5 \
  RUN_ID=h3-light-rva-$(date +%Y%m%d-%H%M%S) \
  bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
```

To distribute five total samples across all four TP2 services (`2/1/1/1`) and start the
four clients concurrently, add `LIGHT_ACTIVE_INSTANCES=4`.

This produces five measured outputs by default (`LIGHT_SAMPLES=5`). It starts four TP2
model instances but sends the requests only to instance 0. It enables synchronized SGLang stage timing and system
telemetry, but does not enable Torch/Nsight by default. For a second, targeted kernel pass:

```bash
MODE=light LIGHT_SCENARIO=RVA LIGHT_RESIDENT_INSTANCES=4 LIGHT_TORCH_PROFILE=1 \
  RUN_ID=h3-light-rva-torch-$(date +%Y%m%d-%H%M%S) \
  bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
```

Use `LIGHT_SCENARIO=F3` for the first+last-keyframe FL2VA path. Use
`LIGHT_RESIDENT_INSTANCES=1` to remove the three idle resident models and obtain the
clean TP2 baseline.

When GPUs 0-3 are occupied but GPUs 4-7 are free, use two resident TP2 instances on
the high GPU half (requests still go only to the first instance):

```bash
MODE=light LIGHT_SCENARIO=RVA LIGHT_SAMPLES=5 \
  LIGHT_RESIDENT_INSTANCES=2 LIGHT_GPU_OFFSET=4 \
  RUN_ID=h3-light-rva-5-gpu4-7-$(date +%Y%m%d-%H%M%S) \
  bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
```

Full run in tmux:

```bash
tmux new -s h3-profile
MODE=full REPEATS=1 RUN_ID=h3-profile-$(date +%Y%m%d-%H%M%S) \
  bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
```

Fast two-batch input matrix (one sample per scenario):

```bash
MODE=quick_matrix RUN_ID=h3-quick-input-matrix-$(date +%Y%m%d-%H%M%S) \
  bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
```

The FL2VA batch maps F0/F1/F2/F3 to replicas 0/1/2/3. The Ref2VA batch maps
R1/R5/RV5/RVA_EMBEDDED to replicas 0/1/2/3. All four requests in a batch start
concurrently. R5 uses five distinct frames from `REFERENCE_IMAGES_DIR` when present.

Targeted evidence pass after the quick matrix:

```bash
MODE=targeted_profile RUN_ID=h3-targeted-profile-$(date +%Y%m%d-%H%M%S) \
  bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
```

This captures one all-stage Torch trace and one bounded Nsight Systems trace for F3 and
RVA_EMBEDDED. It is deliberately separate from clean stage timing because profilers perturb latency.

Result layout:

- `metadata/`: exact hardware, topology, software and copied scripts
- `phases/*/client_*/`: request JSONL and request-level perf dumps
- `phases/*/server_*/`: server logs, outputs, Torch traces and Nsight reports
- `phases/*/system/`: GPU/DCGM, CPU, disk and VM telemetry
- `summary.csv`, `summary.json`: latency/QPS/serving-gap aggregation
- `bottleneck_comparisons.json`: P2/P3/P4/P5 versus P1 contention ratios
- `profile_inventory.json`: trace inventory and missing-artifact detection

The `e2e_*` phases disable synchronized stage timing and provide clean latency controls.
The remaining P1-P6 phases enable synchronized stage timing for attribution. Nsight starts
after a 70-second delay to omit most model-loading CUDA activity and has a bounded 30-minute
capture window; override `NSYS_DELAY_SECONDS`/`NSYS_DURATION_SECONDS` when needed.

`ENABLE_NCU=0` is the safe default because an unfiltered Nsight Compute pass can take
many hours. After Torch/Nsight identifies the top kernel, rerun with the same `RUN_ID`:

```bash
ENABLE_NCU=1 NCU_KERNEL_REGEX='your_demangled_kernel_regex' \
  RUN_ID=<existing-run-id> bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
```

P1-P6 isolate a single TP2 baseline, four resident models with one active, four active
FL2VA, four active Ref2VA, a 2+2 production mix, and TP8. The scenario matrix includes
T2VA, first/last/both keyframes, 1/5/6/9 images, 1/5/10-second videos, image+audio,
video+audio, embedded video/audio, and a mixed reference workload.