qqtang/sskj
1
3
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

r37 addendum: nomtp sweep raw logs (primary evidence, force-added past *.log ignore)

Browse Source
This commit is contained in:
yy-fighting 2026-09-09 09:24:43 +08:00
parent 4d6dac010c
commit 0c89fd4fbe
6 changed files with 316 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
experiments/pro6000/glm53_ppmtp_r37_verify_graph/results/nomtp_sweep_all.log Normal file
View File

@ -0,0 +1,36 @@
=== cc=8 nreq=16 seed=6508 start 08:47:29 ===
=== cc=8 done rc=0 08:49:27 ===
Successful requests: 16
Benchmark duration (s): 78.92
Output token throughput (tok/s): 103.81
Mean TTFT (ms): 18179.50
Mean TPOT (ms): 41.59
=== cc=16 nreq=32 seed=6516 start 08:49:27 ===
=== cc=16 done rc=0 08:52:00 ===
Successful requests: 32
Benchmark duration (s): 128.71
Output token throughput (tok/s): 127.29
Mean TTFT (ms): 30898.35
Mean TPOT (ms): 48.53
=== cc=32 nreq=64 seed=6532 start 08:52:00 ===
=== cc=32 done rc=0 08:56:29 ===
Successful requests: 64
Benchmark duration (s): 244.71
Output token throughput (tok/s): 133.91
Mean TTFT (ms): 74930.30
Mean TPOT (ms): 48.54
=== cc=40 nreq=80 seed=6540 start 08:56:29 ===
=== cc=40 done rc=0 09:02:00 ===
Successful requests: 80
Benchmark duration (s): 304.60
Output token throughput (tok/s): 134.47
Mean TTFT (ms): 93857.54
Mean TPOT (ms): 53.23
=== cc=64 nreq=128 seed=6564 start 09:02:00 ===
=== cc=64 done rc=0 09:10:18 ===
Successful requests: 128
Benchmark duration (s): 473.55
Output token throughput (tok/s): 138.39
Mean TTFT (ms): 161310.10
Mean TPOT (ms): 50.23
SWEEP-DONE

56
experiments/pro6000/glm53_ppmtp_r37_verify_graph/results/nomtpsweep_cc16.log Normal file
View File

@ -0,0 +1,56 @@
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
warnings.warn(
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=32, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=16, output_file='/data/hf_models/bs_results/nomtpsweep_cc16.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6516, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
Server ready in 0.0s.
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=32, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=16, output_file='/data/hf_models/bs_results/nomtpsweep_cc16.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6516, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
#Input tokens: 524288
#Output tokens: 16384
Starting warmup with 1 sequences...
Warmup completed with 1 sequences. Starting main benchmark run...
0%| | 0/32 [00:00<?, ?it/s] 3%|▎ | 1/32 [00:54<28:03, 54.29s/it] 50%|█████ | 16/32 [01:48<01:34, 5.90s/it] 75%|███████▌ | 24/32 [01:49<00:27, 3.42s/it] 94%|█████████▍| 30/32 [02:00<00:06, 3.42s/it] 97%|█████████▋| 31/32 [02:08<00:03, 3.21s/it] 100%|██████████| 32/32 [02:08<00:00, 4.02s/it]
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 32
Benchmark duration (s): 128.71
Total input tokens: 524288
Total input text tokens: 524288
Total generated tokens: 16384
Total generated tokens (retokenized): 16382
Request throughput (req/s): 0.25
Input token throughput (tok/s): 4073.29
Output token throughput (tok/s): 127.29
Peak output token throughput (tok/s): 368.00
Peak concurrent requests: 31
Total token throughput (tok/s): 4200.58
Concurrency: 13.85
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 55694.74
Median E2E Latency (ms): 54445.67
P90 E2E Latency (ms): 54686.04
P95 E2E Latency (ms): 63545.86
P99 E2E Latency (ms): 98164.60
---------------Time to First Token----------------
Mean TTFT (ms): 30898.35
Median TTFT (ms): 32450.28
P90 TTFT (ms): 32916.51
P95 TTFT (ms): 44236.01
P99 TTFT (ms): 59379.03
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 48.53
Median TPOT (ms): 43.40
P90 TPOT (ms): 77.67
P95 TPOT (ms): 84.84
P99 TPOT (ms): 95.77
---------------Inter-Token Latency----------------
Mean ITL (ms): 48.59
Median ITL (ms): 42.63
P90 ITL (ms): 43.95
P95 ITL (ms): 44.32
P99 ITL (ms): 44.93
Max ITL (ms): 28619.03
==================================================

56
experiments/pro6000/glm53_ppmtp_r37_verify_graph/results/nomtpsweep_cc32.log Normal file
View File

@ -0,0 +1,56 @@
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
warnings.warn(
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=64, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=32, output_file='/data/hf_models/bs_results/nomtpsweep_cc32.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6532, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
Server ready in 0.0s.
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=64, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=32, output_file='/data/hf_models/bs_results/nomtpsweep_cc32.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6532, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
#Input tokens: 1048576
#Output tokens: 32768
Starting warmup with 1 sequences...
Warmup completed with 1 sequences. Starting main benchmark run...
0%| | 0/64 [00:00<?, ?it/s] 2%|▏ | 1/64 [00:54<57:21, 54.63s/it] 25%|██▌ | 16/64 [01:49<04:44, 5.92s/it] 38%|███▊ | 24/64 [01:49<02:17, 3.43s/it] 47%|████▋ | 30/64 [02:00<01:56, 3.43s/it] 48%|████▊ | 31/64 [02:43<02:42, 4.93s/it] 61%|██████ | 39/64 [02:44<01:19, 3.16s/it] 70%|███████ | 45/64 [03:00<01:00, 3.16s/it] 72%|███████▏ | 46/64 [03:38<01:22, 4.58s/it] 95%|█████████▌| 61/64 [04:04<00:09, 3.21s/it] 100%|██████████| 64/64 [04:04<00:00, 3.82s/it]
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 32
Successful requests: 64
Benchmark duration (s): 244.71
Total input tokens: 1048576
Total input text tokens: 1048576
Total generated tokens: 32768
Total generated tokens (retokenized): 32754
Request throughput (req/s): 0.26
Input token throughput (tok/s): 4285.04
Output token throughput (tok/s): 133.91
Peak output token throughput (tok/s): 360.00
Peak concurrent requests: 47
Total token throughput (tok/s): 4418.95
Concurrency: 26.08
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 99736.29
Median E2E Latency (ms): 109235.40
P90 E2E Latency (ms): 109445.39
P95 E2E Latency (ms): 159345.69
P99 E2E Latency (ms): 163876.35
---------------Time to First Token----------------
Mean TTFT (ms): 74930.30
Median TTFT (ms): 87071.13
P90 TTFT (ms): 87201.04
P95 TTFT (ms): 118429.44
P99 TTFT (ms): 141758.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 48.54
Median TPOT (ms): 43.29
P90 TPOT (ms): 65.78
P95 TPOT (ms): 85.98
P99 TPOT (ms): 94.45
---------------Inter-Token Latency----------------
Mean ITL (ms): 48.61
Median ITL (ms): 43.07
P90 ITL (ms): 44.03
P95 ITL (ms): 44.30
P99 ITL (ms): 45.35
Max ITL (ms): 28652.24
==================================================

56
experiments/pro6000/glm53_ppmtp_r37_verify_graph/results/nomtpsweep_cc40.log Normal file
View File

@ -0,0 +1,56 @@
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
warnings.warn(
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=80, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=40, output_file='/data/hf_models/bs_results/nomtpsweep_cc40.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6540, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
Server ready in 0.0s.
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=80, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=40, output_file='/data/hf_models/bs_results/nomtpsweep_cc40.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6540, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
#Input tokens: 1310720
#Output tokens: 40960
Starting warmup with 1 sequences...
Warmup completed with 1 sequences. Starting main benchmark run...
0%| | 0/80 [00:00<?, ?it/s] 1%|▏ | 1/80 [00:54<1:12:15, 54.87s/it] 20%|██ | 16/80 [01:48<06:13, 5.84s/it] 39%|███▉ | 31/80 [02:43<03:45, 4.60s/it] 49%|████▉ | 39/80 [02:43<02:11, 3.21s/it] 56%|█████▋ | 45/80 [03:00<01:52, 3.21s/it] 57%|█████▊ | 46/80 [03:37<02:30, 4.44s/it] 76%|███████▋ | 61/80 [04:32<01:17, 4.07s/it] 95%|█████████▌| 76/80 [05:04<00:13, 3.32s/it] 100%|██████████| 80/80 [05:04<00:00, 3.81s/it]
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 40
Successful requests: 80
Benchmark duration (s): 304.60
Total input tokens: 1310720
Total input text tokens: 1310720
Total generated tokens: 40960
Total generated tokens (retokenized): 40950
Request throughput (req/s): 0.26
Input token throughput (tok/s): 4303.11
Output token throughput (tok/s): 134.47
Peak output token throughput (tok/s): 375.00
Peak concurrent requests: 55
Total token throughput (tok/s): 4437.58
Concurrency: 31.79
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 121055.83
Median E2E Latency (ms): 109193.09
P90 E2E Latency (ms): 163874.40
P95 E2E Latency (ms): 163882.77
P99 E2E Latency (ms): 163934.61
---------------Time to First Token----------------
Mean TTFT (ms): 93857.54
Median TTFT (ms): 87217.63
P90 TTFT (ms): 141069.97
P95 TTFT (ms): 141620.33
P99 TTFT (ms): 141666.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 53.23
Median TPOT (ms): 43.46
P90 TPOT (ms): 91.75
P95 TPOT (ms): 95.66
P99 TPOT (ms): 99.92
---------------Inter-Token Latency----------------
Mean ITL (ms): 53.30
Median ITL (ms): 42.89
P90 ITL (ms): 44.34
P95 ITL (ms): 44.58
P99 ITL (ms): 45.17
Max ITL (ms): 29114.99
==================================================

56
experiments/pro6000/glm53_ppmtp_r37_verify_graph/results/nomtpsweep_cc64.log Normal file
View File

@ -0,0 +1,56 @@
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
warnings.warn(
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=128, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=64, output_file='/data/hf_models/bs_results/nomtpsweep_cc64.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6564, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
Server ready in 0.0s.
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=128, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=64, output_file='/data/hf_models/bs_results/nomtpsweep_cc64.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6564, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
#Input tokens: 2097152
#Output tokens: 65536
Starting warmup with 1 sequences...
Warmup completed with 1 sequences. Starting main benchmark run...
0%| | 0/128 [00:00<?, ?it/s] 1%| | 1/128 [00:54<1:56:06, 54.86s/it] 7%|▋ | 9/128 [00:55<08:49, 4.45s/it] 12%|█▏ | 15/128 [01:10<08:23, 4.45s/it] 12%|█▎ | 16/128 [01:48<11:27, 6.14s/it] 19%|█▉ | 24/128 [01:48<05:43, 3.30s/it] 23%|██▎ | 30/128 [02:00<05:23, 3.30s/it] 24%|██▍ | 31/128 [02:43<07:58, 4.94s/it] 30%|███ | 39/128 [02:43<04:35, 3.10s/it] 35%|███▌ | 45/128 [03:00<04:17, 3.10s/it] 36%|███▌ | 46/128 [03:38<06:19, 4.63s/it] 42%|████▏ | 54/128 [03:38<03:46, 3.06s/it] 47%|████▋ | 60/128 [03:50<03:27, 3.06s/it] 48%|████▊ | 61/128 [04:32<04:57, 4.44s/it] 54%|█████▍ | 69/128 [04:32<02:56, 2.99s/it] 59%|█████▊ | 75/128 [04:50<02:38, 2.99s/it] 59%|█████▉ | 76/128 [05:26<03:46, 4.36s/it] 71%|███████ | 91/128 [06:21<02:28, 4.01s/it] 83%|████████▎ | 106/128 [07:15<01:24, 3.85s/it] 95%|█████████▍| 121/128 [07:53<00:23, 3.38s/it] 100%|██████████| 128/128 [07:53<00:00, 3.70s/it]
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 64
Successful requests: 128
Benchmark duration (s): 473.55
Total input tokens: 2097152
Total input text tokens: 2097152
Total generated tokens: 65536
Total generated tokens (retokenized): 65509
Request throughput (req/s): 0.27
Input token throughput (tok/s): 4428.55
Output token throughput (tok/s): 138.39
Peak output token throughput (tok/s): 368.00
Peak concurrent requests: 79
Total token throughput (tok/s): 4566.94
Concurrency: 50.54
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 186979.29
Median E2E Latency (ms): 217737.21
P90 E2E Latency (ms): 271600.25
P95 E2E Latency (ms): 272187.63
P99 E2E Latency (ms): 272559.88
---------------Time to First Token----------------
Mean TTFT (ms): 161310.10
Median TTFT (ms): 189947.02
P90 TTFT (ms): 225814.00
P95 TTFT (ms): 249709.49
P99 TTFT (ms): 250822.91
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 50.23
Median TPOT (ms): 42.91
P90 TPOT (ms): 75.43
P95 TPOT (ms): 91.04
P99 TPOT (ms): 98.53
---------------Inter-Token Latency----------------
Mean ITL (ms): 50.30
Median ITL (ms): 42.86
P90 ITL (ms): 44.64
P95 ITL (ms): 45.12
P99 ITL (ms): 46.02
Max ITL (ms): 28771.25
==================================================

56
experiments/pro6000/glm53_ppmtp_r37_verify_graph/results/nomtpsweep_cc8.log Normal file
View File

@ -0,0 +1,56 @@
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
warnings.warn(
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=16, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=8, output_file='/data/hf_models/bs_results/nomtpsweep_cc8.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6508, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
Server ready in 0.0s.
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=16, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=8, output_file='/data/hf_models/bs_results/nomtpsweep_cc8.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6508, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
#Input tokens: 262144
#Output tokens: 8192
Starting warmup with 1 sequences...
Warmup completed with 1 sequences. Starting main benchmark run...
0%| | 0/16 [00:00<?, ?it/s] 6%|▋ | 1/16 [00:39<09:59, 39.94s/it] 56%|█████▋ | 9/16 [01:18<00:53, 7.69s/it] 100%|██████████| 16/16 [01:18<00:00, 4.93s/it]
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 8
Successful requests: 16
Benchmark duration (s): 78.92
Total input tokens: 262144
Total input text tokens: 262144
Total generated tokens: 8192
Total generated tokens (retokenized): 8191
Request throughput (req/s): 0.20
Input token throughput (tok/s): 3321.83
Output token throughput (tok/s): 103.81
Peak output token throughput (tok/s): 200.00
Peak concurrent requests: 16
Total token throughput (tok/s): 3425.63
Concurrency: 7.99
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 39431.98
Median E2E Latency (ms): 39432.80
P90 E2E Latency (ms): 39938.59
P95 E2E Latency (ms): 39940.22
P99 E2E Latency (ms): 39941.13
---------------Time to First Token----------------
Mean TTFT (ms): 18179.50
Median TTFT (ms): 18006.12
P90 TTFT (ms): 19473.09
P95 TTFT (ms): 19474.85
P99 TTFT (ms): 19476.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 41.59
Median TPOT (ms): 40.96
P90 TPOT (ms): 44.77
P95 TPOT (ms): 45.94
P99 TPOT (ms): 48.28
---------------Inter-Token Latency----------------
Mean ITL (ms): 41.65
Median ITL (ms): 40.70
P90 ITL (ms): 41.54
P95 ITL (ms): 41.70
P99 ITL (ms): 42.21
Max ITL (ms): 4036.70
==================================================