104 lines
4.3 KiB
Markdown
104 lines
4.3 KiB
Markdown
# MiniMax-H3 / SGLang automated profile
|
|
|
|
The suite is scoped to one 8-GPU RTX 6000D host and never kills unrelated GPU processes.
|
|
It aborts if a GPU process or required port already exists. Every phase has a `DONE` marker;
|
|
reuse the same `RUN_ID` to resume after interruption.
|
|
|
|
Smoke validation:
|
|
|
|
```bash
|
|
MODE=smoke RUN_ID=h3-profile-smoke bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
|
|
```
|
|
|
|
Lightweight single-sample stage profile (recommended first pass):
|
|
|
|
```bash
|
|
MODE=light LIGHT_SCENARIO=RVA LIGHT_RESIDENT_INSTANCES=4 LIGHT_SAMPLES=5 \
|
|
RUN_ID=h3-light-rva-$(date +%Y%m%d-%H%M%S) \
|
|
bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
|
|
```
|
|
|
|
To distribute five total samples across all four TP2 services (`2/1/1/1`) and start the
|
|
four clients concurrently, add `LIGHT_ACTIVE_INSTANCES=4`.
|
|
|
|
This produces five measured outputs by default (`LIGHT_SAMPLES=5`). It starts four TP2
|
|
model instances but sends the requests only to instance 0. It enables synchronized SGLang stage timing and system
|
|
telemetry, but does not enable Torch/Nsight by default. For a second, targeted kernel pass:
|
|
|
|
```bash
|
|
MODE=light LIGHT_SCENARIO=RVA LIGHT_RESIDENT_INSTANCES=4 LIGHT_TORCH_PROFILE=1 \
|
|
RUN_ID=h3-light-rva-torch-$(date +%Y%m%d-%H%M%S) \
|
|
bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
|
|
```
|
|
|
|
Use `LIGHT_SCENARIO=F3` for the first+last-keyframe FL2VA path. Use
|
|
`LIGHT_RESIDENT_INSTANCES=1` to remove the three idle resident models and obtain the
|
|
clean TP2 baseline.
|
|
|
|
When GPUs 0-3 are occupied but GPUs 4-7 are free, use two resident TP2 instances on
|
|
the high GPU half (requests still go only to the first instance):
|
|
|
|
```bash
|
|
MODE=light LIGHT_SCENARIO=RVA LIGHT_SAMPLES=5 \
|
|
LIGHT_RESIDENT_INSTANCES=2 LIGHT_GPU_OFFSET=4 \
|
|
RUN_ID=h3-light-rva-5-gpu4-7-$(date +%Y%m%d-%H%M%S) \
|
|
bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
|
|
```
|
|
|
|
Full run in tmux:
|
|
|
|
```bash
|
|
tmux new -s h3-profile
|
|
MODE=full REPEATS=1 RUN_ID=h3-profile-$(date +%Y%m%d-%H%M%S) \
|
|
bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
|
|
```
|
|
|
|
Fast two-batch input matrix (one sample per scenario):
|
|
|
|
```bash
|
|
MODE=quick_matrix RUN_ID=h3-quick-input-matrix-$(date +%Y%m%d-%H%M%S) \
|
|
bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
|
|
```
|
|
|
|
The FL2VA batch maps F0/F1/F2/F3 to replicas 0/1/2/3. The Ref2VA batch maps
|
|
R1/R5/RV5/RVA_EMBEDDED to replicas 0/1/2/3. All four requests in a batch start
|
|
concurrently. R5 uses five distinct frames from `REFERENCE_IMAGES_DIR` when present.
|
|
|
|
Targeted evidence pass after the quick matrix:
|
|
|
|
```bash
|
|
MODE=targeted_profile RUN_ID=h3-targeted-profile-$(date +%Y%m%d-%H%M%S) \
|
|
bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
|
|
```
|
|
|
|
This captures one all-stage Torch trace and one bounded Nsight Systems trace for F3 and
|
|
RVA_EMBEDDED. It is deliberately separate from clean stage timing because profilers perturb latency.
|
|
|
|
Result layout:
|
|
|
|
- `metadata/`: exact hardware, topology, software and copied scripts
|
|
- `phases/*/client_*/`: request JSONL and request-level perf dumps
|
|
- `phases/*/server_*/`: server logs, outputs, Torch traces and Nsight reports
|
|
- `phases/*/system/`: GPU/DCGM, CPU, disk and VM telemetry
|
|
- `summary.csv`, `summary.json`: latency/QPS/serving-gap aggregation
|
|
- `bottleneck_comparisons.json`: P2/P3/P4/P5 versus P1 contention ratios
|
|
- `profile_inventory.json`: trace inventory and missing-artifact detection
|
|
|
|
The `e2e_*` phases disable synchronized stage timing and provide clean latency controls.
|
|
The remaining P1-P6 phases enable synchronized stage timing for attribution. Nsight starts
|
|
after a 70-second delay to omit most model-loading CUDA activity and has a bounded 30-minute
|
|
capture window; override `NSYS_DELAY_SECONDS`/`NSYS_DURATION_SECONDS` when needed.
|
|
|
|
`ENABLE_NCU=0` is the safe default because an unfiltered Nsight Compute pass can take
|
|
many hours. After Torch/Nsight identifies the top kernel, rerun with the same `RUN_ID`:
|
|
|
|
```bash
|
|
ENABLE_NCU=1 NCU_KERNEL_REGEX='your_demangled_kernel_regex' \
|
|
RUN_ID=<existing-run-id> bash /data/wxy/h3_profile/run_h3_sglang_full_profile.sh
|
|
```
|
|
|
|
P1-P6 isolate a single TP2 baseline, four resident models with one active, four active
|
|
FL2VA, four active Ref2VA, a 2+2 production mix, and TP8. The scenario matrix includes
|
|
T2VA, first/last/both keyframes, 1/5/6/9 images, 1/5/10-second videos, image+audio,
|
|
video+audio, embedded video/audio, and a mixed reference workload.
|