Failed to get device capability: SM 12.x requires CUDA >= 12.9. Failed to get device capability: SM 12.x requires CUDA >= 12.9. [08-31 15:10:18] Applying performance_mode=speed [08-31 15:10:18] server_args: {"model_path": "/data/hf_models/MiniMax-H3", "model_subfolder": null, "model_variant": "Ref2VA", "model_id": null, "backend": "sglang", "attention_backend": null, "attention_backend_config": {}, "component_attention_backends": {}, "cache_dit_config": null, "nccl_port": null, "trust_remote_code": false, "revision": null, "num_gpus": 2, "performance_mode": "speed", "base_gpu_id": 0, "gpu_ids": null, "tp_size": 2, "sp_degree": 1, "ulysses_degree": 1, "ring_degree": 1, "dp_size": 1, "dp_degree": 1, "enable_cfg_parallel": false, "cfg_parallel_degree": 1, "encoder_parallel": "auto", "hsdp_replicate_dim": 1, "hsdp_shard_dim": 2, "dist_timeout": 3600, "pipeline_class_name": null, "lora_path": null, "lora_nickname": "default", "lora_scale": 1.0, "lora_merge_mode": "auto", "lora_weight_name": null, "component_paths": {}, "transformer_weights_path": null, "component_transformer_weights_paths": {}, "quantization": null, "quantization_ignored_layers": null, "lora_target_modules": null, "dit_cpu_offload": false, "dit_layerwise_offload": false, "layerwise_offload_components": null, "dit_offload_prefetch_size": 0.0, "dit_layerwise_resident_layers": 0.0, "offload_during_compile": true, "text_encoder_cpu_offload": false, "image_encoder_cpu_offload": false, "vae_cpu_offload": false, "use_fsdp_inference": false, "pin_cpu_memory": true, "ltx2_two_stage_device_mode": null, "comfyui_mode": false, "enable_torch_compile": false, "regional_compile": false, "enable_breakable_cuda_graph": false, "bcg_text_buckets": null, "enable_layerwise_nvtx_marker": false, "warmup_mode": "server", "warmup": true, "server_warmup": true, "warmup_resolutions": null, "warmup_steps": 1, "disable_autocast": false, "master_port": 35000, "host": "0.0.0.0", "port": 34010, "webui": false, "webui_port": 12312, "scheduler_port": 36000, "batching_mode": "dynamic", "batching_max_size": 1, "batching_delay_ms": 0.0, "batching_config": null, "enable_batching_metrics": false, "strict_ports": false, "output_path": "/data/wxy/sskj-h3/throughput/sglang-base/results/ref2va-feishu-base-tp2x4-768p-15s-20steps-20260831-150959/server_0_port34010/outputs", "input_save_path": "inputs/uploads", "prompt_file_path": null, "model_paths": {}, "model_loaded": {"transformer": true, "vae": true, "video_vae": true, "audio_vae": true, "video_dit": true, "audio_dit": true, "dual_tower_bridge": true}, "boundary_ratio": null, "disagg_role": "monolithic", "disagg_timeout": 3600, "disagg_downstream_wait_timeout": 1800, "disagg_dispatch_policy": "round_robin", "disagg_mode": false, "disagg_instance_id": 0, "disagg_max_slots_per_instance": 8, "disagg_transfer_redundancy": 1.25, "disagg_role_device": "auto", "disagg_transfer_backend": "auto", "disagg_transfer_pool_size": 268435456, "disagg_transfer_pin_memory": "auto", "disagg_p2p_hostname": "127.0.0.1", "disagg_ib_device": null, "disagg_server_addr": null, "encoder_urls": null, "denoiser_urls": null, "decoder_urls": null, "encoder_tp": null, "denoiser_tp": null, "denoiser_sp": null, "denoiser_ulysses": null, "denoiser_ring": null, "decoder_sp": null, "decoder_tp": null, "pool_work_endpoint": null, "pool_result_endpoint": null, "pool_control_endpoint": null, "pool_control_advertised_endpoint": null, "log_level": "info", "log_requests": false, "log_requests_level": 2, "log_requests_format": "text", "log_requests_target": null, "uvicorn_access_log_exclude_prefixes": [], "enable_trace": false, "otlp_traces_endpoint": "localhost:4317", "srt_encoder_url": null, "srt_encoder_connect_timeout": 3.05, "srt_encoder_timeout": 100, "pe_server_url": null} [08-31 15:10:18] Starting server... Failed to get device capability: SM 12.x requires CUDA >= 12.9. Failed to get device capability: SM 12.x requires CUDA >= 12.9. Failed to get device capability: SM 12.x requires CUDA >= 12.9. Failed to get device capability: SM 12.x requires CUDA >= 12.9. [08-31 15:10:32] Scheduler bind at endpoint: tcp://0.0.0.0:36000 [08-31 15:10:32] torch.compile cache: TORCHINDUCTOR_CACHE_DIR=/root/.cache/sgl_diffusion/torch_compile_cache/inductor TRITON_CACHE_DIR=/root/.cache/sgl_diffusion/torch_compile_cache/triton [08-31 15:10:32] Initializing distributed environment with world_size=2, device=cuda:0, timeout=3600 [08-31 15:10:32] Setting distributed timeout to 3600 seconds [08-31 15:10:34] Found nccl from library libnccl.so.2 [08-31 15:10:34] sglang-diffusion is using nccl==2.28.9 [08-31 15:10:37] reading GPU P2P access cache from /root/.cache/sglang/gpu_p2p_access_cache_for_0,1.json [08-31 15:10:37] reading GPU P2P access cache from /root/.cache/sglang/gpu_p2p_access_cache_for_0,1.json [08-31 15:10:37] Found nccl from library libnccl.so.2 [08-31 15:10:37] sglang-diffusion is using nccl==2.28.9 [08-31 15:10:37] No pipeline_class_name specified, using model_index.json [08-31 15:10:37] Using pipeline from model_index.json: MiniMaxH3Pipeline [08-31 15:10:37] Loading pipeline modules... [08-31 15:10:37] Model path: /data/hf_models/MiniMax-H3/Ref2VA [08-31 15:10:37] Diffusers version: 0.32.2 [08-31 15:10:37] Loading pipeline modules from config: {'_class_name': 'MiniMaxH3Pipeline', '_diffusers_version': '0.32.2', 'text_encoder': ['transformers', 'MiniMaxH3Qwen3VLHFEncoder'], 'tokenizer': ['transformers', 'Qwen2TokenizerFast'], 'video_vae': ['diffusers', 'MiniMaxH3VideoVAE'], 'audio_vae': ['diffusers', 'MiniMaxH3AudioVAE'], 'scheduler': None, 'transformer': ['diffusers', 'MiniMaxH3DiTModel'], 'processor': ['transformers', 'Qwen3VLProcessor'], '_minimax_h3': {'schema_version': 1, 'partition': 'ref2va', 'tasks': ['ref2va'], 'task_aliases': {}, 'sigma_shift_scales': {'video': 12.0, 'audio': 3.0}}} [08-31 15:10:37] Loading required components: ['processor', 'text_encoder', 'tokenizer', 'video_vae', 'audio_vae', 'transformer'] [08-31 15:10:37] Memory-aware component load order: ['text_encoder', 'transformer', 'audio_vae', 'video_vae', 'processor', 'tokenizer'] Loading required modules: 0%| | 0/6 [00:00 neg_prompt: None seed: 1101 infer_steps: 5 num_outputs_per_prompt: 1 guidance_scale: 1.0 embedded_guidance_scale: 6.0 n_tokens: None flow_shift: 12.0 image_path: None save_output: True output_file_path: /data/wxy/sskj-h3/throughput/sglang-base/results/ref2va-feishu-base-tp2x4-768p-15s-20steps-20260831-150959/server_0_port34010/outputs/54126532-fff6-4b43-92f0-9355970c2bc9.mp4 [08-31 15:11:42] Running pipeline stages: ['InputValidationStage', 'MiniMaxH3PartitionAdmissionStage', 'MiniMaxH3TextEncodingStage', 'MiniMaxH3VisualEncodingStage', 'MiniMaxH3AudioEncodingStage', 'MiniMaxH3LatentPreparationStage', 'MiniMaxH3TimestepPreparationStage', 'MiniMaxH3DenoisingStage', 'MiniMaxH3DecodingStage'] [08-31 15:11:42] [InputValidationStage] started... [08-31 15:11:42] [InputValidationStage] finished in 0.0001 seconds [08-31 15:11:42] [MiniMaxH3PartitionAdmissionStage] started... [08-31 15:11:42] [MiniMaxH3PartitionAdmissionStage] finished in 0.0001 seconds [08-31 15:11:42] [MiniMaxH3TextEncodingStage] started... [2026-08-31 15:11:43] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:44] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:45] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:46] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:47] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:48] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [08-31 15:11:49] [MiniMaxH3TextEncodingStage] finished in 6.9642 seconds [2026-08-31 15:11:49] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:50] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [08-31 15:11:50] [MiniMaxH3VisualEncodingStage] started... [2026-08-31 15:11:51] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:52] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:53] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [2026-08-31 15:11:54] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK [08-31 15:11:54] [MiniMaxH3VisualEncodingStage] finished in 3.4498 seconds [08-31 15:11:54] [MiniMaxH3AudioEncodingStage] started... [08-31 15:11:54] [MiniMaxH3AudioEncodingStage] finished in 0.4868 seconds [08-31 15:11:54] [MiniMaxH3LatentPreparationStage] started... [08-31 15:11:54] [MiniMaxH3LatentPreparationStage] finished in 0.1038 seconds [08-31 15:11:54] [MiniMaxH3TimestepPreparationStage] started... [08-31 15:11:54] [MiniMaxH3TimestepPreparationStage] finished in 0.0003 seconds [08-31 15:11:54] [MiniMaxH3DenoisingStage] started... [2026-08-31 15:11:55] INFO: 127.0.0.1:60222 - "GET /v1/videos/54126532-fff6-4b43-92f0-9355970c2bc9 HTTP/1.1" 200 OK minimax_h3 denoise: 0%| | 0/4 [00:00 neg_prompt: None seed: 1101 infer_steps: 20 num_outputs_per_prompt: 1 guidance_scale: 1.0 embedded_guidance_scale: 6.0 n_tokens: None flow_shift: 12.0 image_path: None save_output: True output_file_path: /data/wxy/sskj-h3/throughput/sglang-base/results/ref2va-feishu-base-tp2x4-768p-15s-20steps-20260831-150959/server_0_port34010/outputs/7513b404-346a-4a36-b2ec-50dbbaf743fd.mp4 [08-31 15:21:12] Running pipeline stages: ['InputValidationStage', 'MiniMaxH3PartitionAdmissionStage', 'MiniMaxH3TextEncodingStage', 'MiniMaxH3VisualEncodingStage', 'MiniMaxH3AudioEncodingStage', 'MiniMaxH3LatentPreparationStage', 'MiniMaxH3TimestepPreparationStage', 'MiniMaxH3DenoisingStage', 'MiniMaxH3DecodingStage'] [08-31 15:21:12] [InputValidationStage] started... [08-31 15:21:12] [InputValidationStage] finished in 0.0001 seconds [08-31 15:21:12] [MiniMaxH3PartitionAdmissionStage] started... [08-31 15:21:12] [MiniMaxH3PartitionAdmissionStage] finished in 0.0001 seconds [08-31 15:21:12] [MiniMaxH3TextEncodingStage] started... [2026-08-31 15:21:13] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:14] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:15] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:16] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:17] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:18] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [08-31 15:21:19] [MiniMaxH3TextEncodingStage] finished in 6.8931 seconds [2026-08-31 15:21:19] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:20] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [08-31 15:21:21] [MiniMaxH3VisualEncodingStage] started... [2026-08-31 15:21:21] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:22] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:23] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [2026-08-31 15:21:24] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK [08-31 15:21:24] [MiniMaxH3VisualEncodingStage] finished in 3.4687 seconds [08-31 15:21:24] [MiniMaxH3AudioEncodingStage] started... [08-31 15:21:25] [MiniMaxH3AudioEncodingStage] finished in 0.2002 seconds [08-31 15:21:25] [MiniMaxH3LatentPreparationStage] started... [08-31 15:21:25] [MiniMaxH3LatentPreparationStage] finished in 0.1032 seconds [08-31 15:21:25] [MiniMaxH3TimestepPreparationStage] started... [08-31 15:21:25] [MiniMaxH3TimestepPreparationStage] finished in 0.0004 seconds [08-31 15:21:25] [MiniMaxH3DenoisingStage] started... [2026-08-31 15:21:25] INFO: 127.0.0.1:60222 - "GET /v1/videos/7513b404-346a-4a36-b2ec-50dbbaf743fd HTTP/1.1" 200 OK minimax_h3 denoise: 0%| | 0/19 [00:00. at 0x7fdab4241940> handle: .> Traceback (most recent call last): File "uvloop/cbhandles.pyx", line 61, in uvloop.loop.Handle._run File "/root/.miniconda3/envs/sglang/lib/python3.12/site-packages/zmq/asyncio.py", line 151, in self._fd, lambda: self._handle_events(0, 0) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/.miniconda3/envs/sglang/lib/python3.12/site-packages/zmq/_future.py", line 675, in _handle_events zmq_events = self._shadow_sock.get(EVENTS) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "zmq/backend/cython/_zmq.py", line 964, in zmq.backend.cython._zmq.Socket.get _getsockopt( ^^^^^^^^^^^ File "zmq/backend/cython/_zmq.py", line 1491, in zmq.backend.cython._zmq._getsockopt _check_rc(rc) ^^^^^^^^^^^ File "zmq/backend/cython/_zmq.py", line 188, in zmq.backend.cython._zmq._check_rc raise ContextTerminated(errno) ^^^^^^^^^^^ zmq.error.ContextTerminated: Context was terminated [2026-08-31 15:30:10] INFO: Application shutdown complete. [2026-08-31 15:30:10] INFO: Finished server process [2257385] /root/.miniconda3/envs/sglang/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '