Compare commits
No commits in common. "0c89fd4fbea0db9db2071df2f78403c64c3bc466" and "911a9a2fb0aed3d120b34b5647ffecb810d46deb" have entirely different histories.
0c89fd4fbe
...
911a9a2fb0
@ -64,10 +64,5 @@ draft 图（DRAFTEAGER=0）实测无增益（噪声带内互有胜负），保
|
|||||||
- 语料池 61,772 token 剩余（< 一次 i8k 262,144），i8k 永久无重跑余量，killer random-ids 为标准口径。
|
- 语料池 61,772 token 剩余（< 一次 i8k 262,144），i8k 永久无重跑余量，killer random-ids 为标准口径。
|
||||||
- 09-09 补测 cc 扫频（i16k/o512，cc 8/16/32/40/64，nreq=2cc）：吞吐 cc16 起饱和 130-135 tok/s，
|
- 09-09 补测 cc 扫频（i16k/o512，cc 8/16/32/40/64，nreq=2cc）：吞吐 cc16 起饱和 130-135 tok/s，
|
||||||
accept 全档 3.46-3.68，详见 results/bench_results.md；扫频脚本 scripts/r37_cc_sweep.sh、日志 results/r37sweep_cc*.log。
|
accept 全档 3.46-3.68，详见 results/bench_results.md；扫频脚本 scripts/r37_cc_sweep.sh、日志 results/r37sweep_cc*.log。
|
||||||
- **09-09 B'(nomtp) 同口径对拍**（同栈同脚本 nomtp 模式、同 seed/nreq/flush-cache，唯一变量=开不开 MTP）：
|
|
||||||
cc8 打平（拐点实证：每 token 41.0 vs 41.6ms）、**cc16 mtp +6% 且 TTFT -32%**、cc≥32 B' 反超 0.9-5.4%
|
|
||||||
且单请求 TPOT 好 2×（verify 大步×chunk 交错放大排队相位停顿）。图修复把 r36 时代 1.81× 惨败收敛到 ±6%
|
|
||||||
水平线。脚本 scripts/nomtp_cc_sweep.sh、日志 results/nomtpsweep_cc*.log + nomtp_sweep_all.log，
|
|
||||||
判读全表见 results/bench_results.md "B'(nomtp) vs PP+MTP(r37)" 节。
|
|
||||||
- 勘误：历次 bench 的 rc=1 与 JSON 缺失真因 = 容器内 /data/hf_models/bs_results/ 目录从未存在（写 output-file
|
- 勘误：历次 bench 的 rc=1 与 JSON 缺失真因 = 容器内 /data/hf_models/bs_results/ 目录从未存在（写 output-file
|
||||||
必失败，指标在 stdout 日志无损）；09-09 已补建目录，此后 JSON 可正常落盘。
|
必失败，指标在 stdout 日志无损）；09-09 已补建目录，此后 JSON 可正常落盘。
|
||||||
|
|||||||
@ -62,45 +62,8 @@ bench_serving 的 rc=1 真因 = 跑完打印全部指标后写 `--output-file`
|
|||||||
从未写成功过**（本 README 早先版本"JSON 被 A16 复测覆盖"的说法有误，目录 09-09 已补建，此后 JSON 可正常落盘）。
|
从未写成功过**（本 README 早先版本"JSON 被 A16 复测覆盖"的说法有误，目录 09-09 已补建，此后 JSON 可正常落盘）。
|
||||||
指标以 stdout 日志为准（results/ 下 5 份扫频日志 + 会话记录中的 killer/A16 数字均完整）。
|
指标以 stdout 日志为准（results/ 下 5 份扫频日志 + 会话记录中的 killer/A16 数字均完整）。
|
||||||
|
|
||||||
## B'(nomtp) vs PP+MTP(r37) 输入/输出吞吐对拍（09-09）
|
|
||||||
|
|
||||||
用户问题："pp不开投机 vs pp+mtp 的输入吞吐和输出吞吐对比"。此前 B' 仅有语料 i8k 口径数据
|
|
||||||
（r36 判决：B' e2e 61.5s ≡ B 基线 62s；当年 PP+MTP eager 输它 1.81×）——语料池已耗尽无法复测 mtp 侧，
|
|
||||||
故 09-09 用 **deploy_ppmtp_r37.sh nomtp 模式**（= B' 同栈同脚本同 9 挂载，无 spec 无图捕获路径差异）
|
|
||||||
在 60.8 临时部署，跑 **与 r37 扫频完全相同**的 random-ids i16k/o512 负载（同 seed 6508-6564、同 nreq=2×cc、
|
|
||||||
同 --flush-cache、同 mrr16/memfrac/chunk），唯一变量 = 是否开 MTP。跑完 r37 mtp 生产已恢复。
|
|
||||||
|
|
||||||
| cc (nreq) | B' 输入 (tok/s) | r37 输入 (tok/s) | B' 输出 (tok/s) | r37 输出 (tok/s) | 输出 Δ | B' TPOT (ms) | r37 TPOT (ms) | B' TTFT (s) | r37 TTFT (s) |
|
|
||||||
|---|---|---|---|---|---|---|---|---|---|
|
|
||||||
| 8 (16) | 3321.8 | 3379.4 | 103.8 | 105.6 | **mtp +1.7%** | 41.6 | 41.0 | 18.2 | **14.9** |
|
|
||||||
| 16 (32) | 4073.3 | 4318.6 | 127.3 | 135.0 | **mtp +6.0%** | 48.5 | 69.3 | 30.9 | **21.0** |
|
|
||||||
| 32 (64) | 4285.0 | 4247.4 | 133.9 | 132.7 | B' +0.9% | 48.5 | 93.1 | 74.9 | **62.0** |
|
|
||||||
| 40 (80) | 4303.1 | 4171.5 | 134.5 | 130.4 | B' +3.1% | 53.2 | 100.7 | 93.9 | **86.1** |
|
|
||||||
| 64 (128) | 4428.6 | 4187.9 | 138.4 | 130.9 | B' +5.4% | 50.2 | 102.1 | 161.3 | **155.6** |
|
|
||||||
|
|
||||||
（r37 accept 全档 3.46-3.68；输入吞吐与输出吞吐同比率变化——同 token 总量÷同时长，天然一致。）
|
|
||||||
|
|
||||||
**判读：**
|
|
||||||
- **cc8（单波纯 decode 相位）= 拐点数学的直接实证**：mtp 每步 41.0ms×3.46=142ms 产 3.46 tok vs
|
|
||||||
B' 每步 41.6ms 产 1 tok → 每输出 token 41.0 vs 41.6ms，**打平**。图修复后 MTP 恰好落在净收益线上
|
|
||||||
（r36 eager 时代每步 272-275ms，是它输 B' 1.81× 的原因；图化把步成本砍近半，把 1.81× 差距收敛到 ±6%）。
|
|
||||||
- **cc16（甜点）mtp +6%**：唯一显著胜点，且 TTFT 30.9→21.0s（-32%）——decode 总步数少 3.46× → decode
|
|
||||||
与 prefill 的争用窗口更少 → 准入队列消化更快。输入吞吐同 +6%。
|
|
||||||
- **cc≥32（过饱和排队档）B' 反超 0.9-5.4%**，且单请求 TPOT 差到 2×（B' 48-53ms 恒定 vs mtp 93-102ms）：
|
|
||||||
verify 大步（16 req × 4 draft token = 64 token/步）与 chunk prefill 相撞时，单步停顿被放大 3.4×；
|
|
||||||
mrr16 准入上限下两臂吞吐都在 130-138 tok/s 饱和，差异全部来自排队相位的长短。
|
|
||||||
- **TTFT mtp 全档更低**（decode 步数少 → prefill 窗口多），与吞吐胜负无关的独立改善。
|
|
||||||
- **口径警示**：历史 B' 语料 i8k 数据（61.5s，out ≈265 tok/s 口径）与本表 random-ids i16k 不可直接比——
|
|
||||||
语料含前缀缓存/不同输入长度/不同输出长度，仅可作"图修复前后相对关系"的参照。
|
|
||||||
|
|
||||||
**生产含义**：cc≤16 负载形态（生产甜点）留 r37 mtp——+6% 吞吐 + TTFT -32%；持续过饱和大队列
|
|
||||||
（cc>>16 长驻排队）业务则 B' nomtp 更优（吞吐 +1-5%，单请求 decode 延迟好 2×）。
|
|
||||||
r37 修复的真正价值 = 把 PP+MTP 从 1.81× 惨败拉回与 B' 同一水平线并在甜点小胜，而非大幅放大吞吐。
|
|
||||||
|
|
||||||
## 生产部署（终态）
|
## 生产部署（终态）
|
||||||
- 容器 glm53-nvfp4 @ 60.8:30000，`--restart unless-stopped`，health 200。
|
- 容器 glm53-nvfp4 @ 60.8:30000，`--restart unless-stopped`，health 200。
|
||||||
- 启动命令：`bash /root/deploy_ppmtp_r37.sh '--tp 4 --pp-size 2 --disable-overlap-schedule --max-prefill-tokens 16384' mtp 8192 0.88 1 1 0 0`
|
- 启动命令：`bash /root/deploy_ppmtp_r37.sh '--tp 4 --pp-size 2 --disable-overlap-schedule --max-prefill-tokens 16384' mtp 8192 0.88 1 1 0 0`
|
||||||
（degloo=1, drafteager=1, verifyeager=0, specdebug=0）
|
（degloo=1, drafteager=1, verifyeager=0, specdebug=0）
|
||||||
- 与 A16 对比后胜出留役；A16 恢复脚本 deploy_glm53_605.sh 留盘可随时切回。
|
- 与 A16 对比后胜出留役；A16 恢复脚本 deploy_glm53_605.sh 留盘可随时切回。
|
||||||
- 09-09 曾为 B' 对拍临时换 nomtp 部署约 1.5h（扫频后已按上述命令恢复 mtp 并核验 health 200 +
|
|
||||||
生成抽检正常 + restart=unless-stopped 已设回）。
|
|
||||||
|
|||||||
@ -1,36 +0,0 @@
|
|||||||
=== cc=8 nreq=16 seed=6508 start 08:47:29 ===
|
|
||||||
=== cc=8 done rc=0 08:49:27 ===
|
|
||||||
Successful requests: 16
|
|
||||||
Benchmark duration (s): 78.92
|
|
||||||
Output token throughput (tok/s): 103.81
|
|
||||||
Mean TTFT (ms): 18179.50
|
|
||||||
Mean TPOT (ms): 41.59
|
|
||||||
=== cc=16 nreq=32 seed=6516 start 08:49:27 ===
|
|
||||||
=== cc=16 done rc=0 08:52:00 ===
|
|
||||||
Successful requests: 32
|
|
||||||
Benchmark duration (s): 128.71
|
|
||||||
Output token throughput (tok/s): 127.29
|
|
||||||
Mean TTFT (ms): 30898.35
|
|
||||||
Mean TPOT (ms): 48.53
|
|
||||||
=== cc=32 nreq=64 seed=6532 start 08:52:00 ===
|
|
||||||
=== cc=32 done rc=0 08:56:29 ===
|
|
||||||
Successful requests: 64
|
|
||||||
Benchmark duration (s): 244.71
|
|
||||||
Output token throughput (tok/s): 133.91
|
|
||||||
Mean TTFT (ms): 74930.30
|
|
||||||
Mean TPOT (ms): 48.54
|
|
||||||
=== cc=40 nreq=80 seed=6540 start 08:56:29 ===
|
|
||||||
=== cc=40 done rc=0 09:02:00 ===
|
|
||||||
Successful requests: 80
|
|
||||||
Benchmark duration (s): 304.60
|
|
||||||
Output token throughput (tok/s): 134.47
|
|
||||||
Mean TTFT (ms): 93857.54
|
|
||||||
Mean TPOT (ms): 53.23
|
|
||||||
=== cc=64 nreq=128 seed=6564 start 09:02:00 ===
|
|
||||||
=== cc=64 done rc=0 09:10:18 ===
|
|
||||||
Successful requests: 128
|
|
||||||
Benchmark duration (s): 473.55
|
|
||||||
Output token throughput (tok/s): 138.39
|
|
||||||
Mean TTFT (ms): 161310.10
|
|
||||||
Mean TPOT (ms): 50.23
|
|
||||||
SWEEP-DONE
|
|
||||||
@ -1,56 +0,0 @@
|
|||||||
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
|
|
||||||
warnings.warn(
|
|
||||||
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=32, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=16, output_file='/data/hf_models/bs_results/nomtpsweep_cc16.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6516, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
|
|
||||||
Server ready in 0.0s.
|
|
||||||
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=32, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=16, output_file='/data/hf_models/bs_results/nomtpsweep_cc16.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6516, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
|
|
||||||
#Input tokens: 524288
|
|
||||||
#Output tokens: 16384
|
|
||||||
Starting warmup with 1 sequences...
|
|
||||||
Warmup completed with 1 sequences. Starting main benchmark run...
|
|
||||||
0%| | 0/32 [00:00<?, ?it/s]
3%|▎ | 1/32 [00:54<28:03, 54.29s/it]
50%|█████ | 16/32 [01:48<01:34, 5.90s/it]
75%|███████▌ | 24/32 [01:49<00:27, 3.42s/it]
94%|█████████▍| 30/32 [02:00<00:06, 3.42s/it]
97%|█████████▋| 31/32 [02:08<00:03, 3.21s/it]
100%|██████████| 32/32 [02:08<00:00, 4.02s/it]
|
|
||||||
|
|
||||||
============ Serving Benchmark Result ============
|
|
||||||
Backend: sglang
|
|
||||||
Traffic request rate: inf
|
|
||||||
Max request concurrency: 16
|
|
||||||
Successful requests: 32
|
|
||||||
Benchmark duration (s): 128.71
|
|
||||||
Total input tokens: 524288
|
|
||||||
Total input text tokens: 524288
|
|
||||||
Total generated tokens: 16384
|
|
||||||
Total generated tokens (retokenized): 16382
|
|
||||||
Request throughput (req/s): 0.25
|
|
||||||
Input token throughput (tok/s): 4073.29
|
|
||||||
Output token throughput (tok/s): 127.29
|
|
||||||
Peak output token throughput (tok/s): 368.00
|
|
||||||
Peak concurrent requests: 31
|
|
||||||
Total token throughput (tok/s): 4200.58
|
|
||||||
Concurrency: 13.85
|
|
||||||
----------------End-to-End Latency----------------
|
|
||||||
Mean E2E Latency (ms): 55694.74
|
|
||||||
Median E2E Latency (ms): 54445.67
|
|
||||||
P90 E2E Latency (ms): 54686.04
|
|
||||||
P95 E2E Latency (ms): 63545.86
|
|
||||||
P99 E2E Latency (ms): 98164.60
|
|
||||||
---------------Time to First Token----------------
|
|
||||||
Mean TTFT (ms): 30898.35
|
|
||||||
Median TTFT (ms): 32450.28
|
|
||||||
P90 TTFT (ms): 32916.51
|
|
||||||
P95 TTFT (ms): 44236.01
|
|
||||||
P99 TTFT (ms): 59379.03
|
|
||||||
-----Time per Output Token (excl. 1st token)------
|
|
||||||
Mean TPOT (ms): 48.53
|
|
||||||
Median TPOT (ms): 43.40
|
|
||||||
P90 TPOT (ms): 77.67
|
|
||||||
P95 TPOT (ms): 84.84
|
|
||||||
P99 TPOT (ms): 95.77
|
|
||||||
---------------Inter-Token Latency----------------
|
|
||||||
Mean ITL (ms): 48.59
|
|
||||||
Median ITL (ms): 42.63
|
|
||||||
P90 ITL (ms): 43.95
|
|
||||||
P95 ITL (ms): 44.32
|
|
||||||
P99 ITL (ms): 44.93
|
|
||||||
Max ITL (ms): 28619.03
|
|
||||||
==================================================
|
|
||||||
@ -1,56 +0,0 @@
|
|||||||
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
|
|
||||||
warnings.warn(
|
|
||||||
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=64, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=32, output_file='/data/hf_models/bs_results/nomtpsweep_cc32.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6532, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
|
|
||||||
Server ready in 0.0s.
|
|
||||||
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=64, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=32, output_file='/data/hf_models/bs_results/nomtpsweep_cc32.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6532, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
|
|
||||||
#Input tokens: 1048576
|
|
||||||
#Output tokens: 32768
|
|
||||||
Starting warmup with 1 sequences...
|
|
||||||
Warmup completed with 1 sequences. Starting main benchmark run...
|
|
||||||
0%| | 0/64 [00:00<?, ?it/s]
2%|▏ | 1/64 [00:54<57:21, 54.63s/it]
25%|██▌ | 16/64 [01:49<04:44, 5.92s/it]
38%|███▊ | 24/64 [01:49<02:17, 3.43s/it]
47%|████▋ | 30/64 [02:00<01:56, 3.43s/it]
48%|████▊ | 31/64 [02:43<02:42, 4.93s/it]
61%|██████ | 39/64 [02:44<01:19, 3.16s/it]
70%|███████ | 45/64 [03:00<01:00, 3.16s/it]
72%|███████▏ | 46/64 [03:38<01:22, 4.58s/it]
95%|█████████▌| 61/64 [04:04<00:09, 3.21s/it]
100%|██████████| 64/64 [04:04<00:00, 3.82s/it]
|
|
||||||
|
|
||||||
============ Serving Benchmark Result ============
|
|
||||||
Backend: sglang
|
|
||||||
Traffic request rate: inf
|
|
||||||
Max request concurrency: 32
|
|
||||||
Successful requests: 64
|
|
||||||
Benchmark duration (s): 244.71
|
|
||||||
Total input tokens: 1048576
|
|
||||||
Total input text tokens: 1048576
|
|
||||||
Total generated tokens: 32768
|
|
||||||
Total generated tokens (retokenized): 32754
|
|
||||||
Request throughput (req/s): 0.26
|
|
||||||
Input token throughput (tok/s): 4285.04
|
|
||||||
Output token throughput (tok/s): 133.91
|
|
||||||
Peak output token throughput (tok/s): 360.00
|
|
||||||
Peak concurrent requests: 47
|
|
||||||
Total token throughput (tok/s): 4418.95
|
|
||||||
Concurrency: 26.08
|
|
||||||
----------------End-to-End Latency----------------
|
|
||||||
Mean E2E Latency (ms): 99736.29
|
|
||||||
Median E2E Latency (ms): 109235.40
|
|
||||||
P90 E2E Latency (ms): 109445.39
|
|
||||||
P95 E2E Latency (ms): 159345.69
|
|
||||||
P99 E2E Latency (ms): 163876.35
|
|
||||||
---------------Time to First Token----------------
|
|
||||||
Mean TTFT (ms): 74930.30
|
|
||||||
Median TTFT (ms): 87071.13
|
|
||||||
P90 TTFT (ms): 87201.04
|
|
||||||
P95 TTFT (ms): 118429.44
|
|
||||||
P99 TTFT (ms): 141758.20
|
|
||||||
-----Time per Output Token (excl. 1st token)------
|
|
||||||
Mean TPOT (ms): 48.54
|
|
||||||
Median TPOT (ms): 43.29
|
|
||||||
P90 TPOT (ms): 65.78
|
|
||||||
P95 TPOT (ms): 85.98
|
|
||||||
P99 TPOT (ms): 94.45
|
|
||||||
---------------Inter-Token Latency----------------
|
|
||||||
Mean ITL (ms): 48.61
|
|
||||||
Median ITL (ms): 43.07
|
|
||||||
P90 ITL (ms): 44.03
|
|
||||||
P95 ITL (ms): 44.30
|
|
||||||
P99 ITL (ms): 45.35
|
|
||||||
Max ITL (ms): 28652.24
|
|
||||||
==================================================
|
|
||||||
@ -1,56 +0,0 @@
|
|||||||
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
|
|
||||||
warnings.warn(
|
|
||||||
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=80, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=40, output_file='/data/hf_models/bs_results/nomtpsweep_cc40.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6540, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
|
|
||||||
Server ready in 0.0s.
|
|
||||||
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=80, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=40, output_file='/data/hf_models/bs_results/nomtpsweep_cc40.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6540, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
|
|
||||||
#Input tokens: 1310720
|
|
||||||
#Output tokens: 40960
|
|
||||||
Starting warmup with 1 sequences...
|
|
||||||
Warmup completed with 1 sequences. Starting main benchmark run...
|
|
||||||
0%| | 0/80 [00:00<?, ?it/s]
1%|▏ | 1/80 [00:54<1:12:15, 54.87s/it]
20%|██ | 16/80 [01:48<06:13, 5.84s/it]
39%|███▉ | 31/80 [02:43<03:45, 4.60s/it]
49%|████▉ | 39/80 [02:43<02:11, 3.21s/it]
56%|█████▋ | 45/80 [03:00<01:52, 3.21s/it]
57%|█████▊ | 46/80 [03:37<02:30, 4.44s/it]
76%|███████▋ | 61/80 [04:32<01:17, 4.07s/it]
95%|█████████▌| 76/80 [05:04<00:13, 3.32s/it]
100%|██████████| 80/80 [05:04<00:00, 3.81s/it]
|
|
||||||
|
|
||||||
============ Serving Benchmark Result ============
|
|
||||||
Backend: sglang
|
|
||||||
Traffic request rate: inf
|
|
||||||
Max request concurrency: 40
|
|
||||||
Successful requests: 80
|
|
||||||
Benchmark duration (s): 304.60
|
|
||||||
Total input tokens: 1310720
|
|
||||||
Total input text tokens: 1310720
|
|
||||||
Total generated tokens: 40960
|
|
||||||
Total generated tokens (retokenized): 40950
|
|
||||||
Request throughput (req/s): 0.26
|
|
||||||
Input token throughput (tok/s): 4303.11
|
|
||||||
Output token throughput (tok/s): 134.47
|
|
||||||
Peak output token throughput (tok/s): 375.00
|
|
||||||
Peak concurrent requests: 55
|
|
||||||
Total token throughput (tok/s): 4437.58
|
|
||||||
Concurrency: 31.79
|
|
||||||
----------------End-to-End Latency----------------
|
|
||||||
Mean E2E Latency (ms): 121055.83
|
|
||||||
Median E2E Latency (ms): 109193.09
|
|
||||||
P90 E2E Latency (ms): 163874.40
|
|
||||||
P95 E2E Latency (ms): 163882.77
|
|
||||||
P99 E2E Latency (ms): 163934.61
|
|
||||||
---------------Time to First Token----------------
|
|
||||||
Mean TTFT (ms): 93857.54
|
|
||||||
Median TTFT (ms): 87217.63
|
|
||||||
P90 TTFT (ms): 141069.97
|
|
||||||
P95 TTFT (ms): 141620.33
|
|
||||||
P99 TTFT (ms): 141666.45
|
|
||||||
-----Time per Output Token (excl. 1st token)------
|
|
||||||
Mean TPOT (ms): 53.23
|
|
||||||
Median TPOT (ms): 43.46
|
|
||||||
P90 TPOT (ms): 91.75
|
|
||||||
P95 TPOT (ms): 95.66
|
|
||||||
P99 TPOT (ms): 99.92
|
|
||||||
---------------Inter-Token Latency----------------
|
|
||||||
Mean ITL (ms): 53.30
|
|
||||||
Median ITL (ms): 42.89
|
|
||||||
P90 ITL (ms): 44.34
|
|
||||||
P95 ITL (ms): 44.58
|
|
||||||
P99 ITL (ms): 45.17
|
|
||||||
Max ITL (ms): 29114.99
|
|
||||||
==================================================
|
|
||||||
@ -1,56 +0,0 @@
|
|||||||
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
|
|
||||||
warnings.warn(
|
|
||||||
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=128, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=64, output_file='/data/hf_models/bs_results/nomtpsweep_cc64.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6564, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
|
|
||||||
Server ready in 0.0s.
|
|
||||||
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=128, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=64, output_file='/data/hf_models/bs_results/nomtpsweep_cc64.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6564, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
|
|
||||||
#Input tokens: 2097152
|
|
||||||
#Output tokens: 65536
|
|
||||||
Starting warmup with 1 sequences...
|
|
||||||
Warmup completed with 1 sequences. Starting main benchmark run...
|
|
||||||
0%| | 0/128 [00:00<?, ?it/s]
1%| | 1/128 [00:54<1:56:06, 54.86s/it]
7%|▋ | 9/128 [00:55<08:49, 4.45s/it]
12%|█▏ | 15/128 [01:10<08:23, 4.45s/it]
12%|█▎ | 16/128 [01:48<11:27, 6.14s/it]
19%|█▉ | 24/128 [01:48<05:43, 3.30s/it]
23%|██▎ | 30/128 [02:00<05:23, 3.30s/it]
24%|██▍ | 31/128 [02:43<07:58, 4.94s/it]
30%|███ | 39/128 [02:43<04:35, 3.10s/it]
35%|███▌ | 45/128 [03:00<04:17, 3.10s/it]
36%|███▌ | 46/128 [03:38<06:19, 4.63s/it]
42%|████▏ | 54/128 [03:38<03:46, 3.06s/it]
47%|████▋ | 60/128 [03:50<03:27, 3.06s/it]
48%|████▊ | 61/128 [04:32<04:57, 4.44s/it]
54%|█████▍ | 69/128 [04:32<02:56, 2.99s/it]
59%|█████▊ | 75/128 [04:50<02:38, 2.99s/it]
59%|█████▉ | 76/128 [05:26<03:46, 4.36s/it]
71%|███████ | 91/128 [06:21<02:28, 4.01s/it]
83%|████████▎ | 106/128 [07:15<01:24, 3.85s/it]
95%|█████████▍| 121/128 [07:53<00:23, 3.38s/it]
100%|██████████| 128/128 [07:53<00:00, 3.70s/it]
|
|
||||||
|
|
||||||
============ Serving Benchmark Result ============
|
|
||||||
Backend: sglang
|
|
||||||
Traffic request rate: inf
|
|
||||||
Max request concurrency: 64
|
|
||||||
Successful requests: 128
|
|
||||||
Benchmark duration (s): 473.55
|
|
||||||
Total input tokens: 2097152
|
|
||||||
Total input text tokens: 2097152
|
|
||||||
Total generated tokens: 65536
|
|
||||||
Total generated tokens (retokenized): 65509
|
|
||||||
Request throughput (req/s): 0.27
|
|
||||||
Input token throughput (tok/s): 4428.55
|
|
||||||
Output token throughput (tok/s): 138.39
|
|
||||||
Peak output token throughput (tok/s): 368.00
|
|
||||||
Peak concurrent requests: 79
|
|
||||||
Total token throughput (tok/s): 4566.94
|
|
||||||
Concurrency: 50.54
|
|
||||||
----------------End-to-End Latency----------------
|
|
||||||
Mean E2E Latency (ms): 186979.29
|
|
||||||
Median E2E Latency (ms): 217737.21
|
|
||||||
P90 E2E Latency (ms): 271600.25
|
|
||||||
P95 E2E Latency (ms): 272187.63
|
|
||||||
P99 E2E Latency (ms): 272559.88
|
|
||||||
---------------Time to First Token----------------
|
|
||||||
Mean TTFT (ms): 161310.10
|
|
||||||
Median TTFT (ms): 189947.02
|
|
||||||
P90 TTFT (ms): 225814.00
|
|
||||||
P95 TTFT (ms): 249709.49
|
|
||||||
P99 TTFT (ms): 250822.91
|
|
||||||
-----Time per Output Token (excl. 1st token)------
|
|
||||||
Mean TPOT (ms): 50.23
|
|
||||||
Median TPOT (ms): 42.91
|
|
||||||
P90 TPOT (ms): 75.43
|
|
||||||
P95 TPOT (ms): 91.04
|
|
||||||
P99 TPOT (ms): 98.53
|
|
||||||
---------------Inter-Token Latency----------------
|
|
||||||
Mean ITL (ms): 50.30
|
|
||||||
Median ITL (ms): 42.86
|
|
||||||
P90 ITL (ms): 44.64
|
|
||||||
P95 ITL (ms): 45.12
|
|
||||||
P99 ITL (ms): 46.02
|
|
||||||
Max ITL (ms): 28771.25
|
|
||||||
==================================================
|
|
||||||
@ -1,56 +0,0 @@
|
|||||||
/sgl-workspace/sglang/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`).
|
|
||||||
warnings.warn(
|
|
||||||
benchmark_args=Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=16, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=8, output_file='/data/hf_models/bs_results/nomtpsweep_cc8.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6508, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
Waiting up to 60s for http://127.0.0.1:30000/v1/models to become ready...
|
|
||||||
Server ready in 0.0s.
|
|
||||||
Namespace(backend='sglang', base_url=None, host='127.0.0.1', port=30000, ready_check_timeout_sec=60, dataset_name='random-ids', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='/data/hf_models/GLM-5.3-NVFP4', served_model_name=None, tokenizer='/data/hf_models/GLM-5.3-NVFP4', num_prompts=16, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=16384, random_output_len=512, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=8, output_file='/data/hf_models/bs_results/nomtpsweep_cc8.json', output_details=False, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=6508, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=True, flush_cache_timeout=60.0, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None)
|
|
||||||
|
|
||||||
#Input tokens: 262144
|
|
||||||
#Output tokens: 8192
|
|
||||||
Starting warmup with 1 sequences...
|
|
||||||
Warmup completed with 1 sequences. Starting main benchmark run...
|
|
||||||
0%| | 0/16 [00:00<?, ?it/s]
6%|▋ | 1/16 [00:39<09:59, 39.94s/it]
56%|█████▋ | 9/16 [01:18<00:53, 7.69s/it]
100%|██████████| 16/16 [01:18<00:00, 4.93s/it]
|
|
||||||
|
|
||||||
============ Serving Benchmark Result ============
|
|
||||||
Backend: sglang
|
|
||||||
Traffic request rate: inf
|
|
||||||
Max request concurrency: 8
|
|
||||||
Successful requests: 16
|
|
||||||
Benchmark duration (s): 78.92
|
|
||||||
Total input tokens: 262144
|
|
||||||
Total input text tokens: 262144
|
|
||||||
Total generated tokens: 8192
|
|
||||||
Total generated tokens (retokenized): 8191
|
|
||||||
Request throughput (req/s): 0.20
|
|
||||||
Input token throughput (tok/s): 3321.83
|
|
||||||
Output token throughput (tok/s): 103.81
|
|
||||||
Peak output token throughput (tok/s): 200.00
|
|
||||||
Peak concurrent requests: 16
|
|
||||||
Total token throughput (tok/s): 3425.63
|
|
||||||
Concurrency: 7.99
|
|
||||||
----------------End-to-End Latency----------------
|
|
||||||
Mean E2E Latency (ms): 39431.98
|
|
||||||
Median E2E Latency (ms): 39432.80
|
|
||||||
P90 E2E Latency (ms): 39938.59
|
|
||||||
P95 E2E Latency (ms): 39940.22
|
|
||||||
P99 E2E Latency (ms): 39941.13
|
|
||||||
---------------Time to First Token----------------
|
|
||||||
Mean TTFT (ms): 18179.50
|
|
||||||
Median TTFT (ms): 18006.12
|
|
||||||
P90 TTFT (ms): 19473.09
|
|
||||||
P95 TTFT (ms): 19474.85
|
|
||||||
P99 TTFT (ms): 19476.29
|
|
||||||
-----Time per Output Token (excl. 1st token)------
|
|
||||||
Mean TPOT (ms): 41.59
|
|
||||||
Median TPOT (ms): 40.96
|
|
||||||
P90 TPOT (ms): 44.77
|
|
||||||
P95 TPOT (ms): 45.94
|
|
||||||
P99 TPOT (ms): 48.28
|
|
||||||
---------------Inter-Token Latency----------------
|
|
||||||
Mean ITL (ms): 41.65
|
|
||||||
Median ITL (ms): 40.70
|
|
||||||
P90 ITL (ms): 41.54
|
|
||||||
P95 ITL (ms): 41.70
|
|
||||||
P99 ITL (ms): 42.21
|
|
||||||
Max ITL (ms): 4036.70
|
|
||||||
==================================================
|
|
||||||
@ -1,26 +0,0 @@
|
|||||||
#!/bin/bash
|
|
||||||
# r37 PP+MTP verify-graph cc 扫频：i16k/o512 random-ids，cc 8/16/32/40/64
|
|
||||||
# 口径：nreq = 2*cc（每档至少两波，测稳态吞吐；与 D 方案 cc40/64 的 nreq=2cc 约定一致），
|
|
||||||
# seed 逐档更换，--flush-cache，输出文件逐档独立（防覆盖）。
|
|
||||||
# 用法: nohup bash r37_cc_sweep.sh > /root/r37_sweep.log 2>&1 &
|
|
||||||
for cc in 8 16 32 40 64; do
|
|
||||||
nreq=$((cc*2))
|
|
||||||
seed=$((6500+cc))
|
|
||||||
echo "=== cc=$cc nreq=$nreq seed=$seed start $(date +%T) ==="
|
|
||||||
docker exec glm53-nvfp4 python3 -m sglang.bench_serving \
|
|
||||||
--backend sglang --host 127.0.0.1 --port 30000 \
|
|
||||||
--dataset-name random-ids --tokenizer /data/hf_models/GLM-5.3-NVFP4 \
|
|
||||||
--num-prompts $nreq --random-input-len 16384 --random-output-len 512 \
|
|
||||||
--random-range-ratio 1.0 --max-concurrency $cc \
|
|
||||||
--temperature 0.0 --flush-cache --warmup-requests 1 --seed $seed \
|
|
||||||
--output-file /data/hf_models/bs_results/nomtpsweep_cc${cc}.json \
|
|
||||||
> /root/nomtpsweep_cc${cc}.log 2>&1
|
|
||||||
rc=$?
|
|
||||||
echo "=== cc=$cc done rc=$rc $(date +%T) ==="
|
|
||||||
grep -E 'Successful requests|Benchmark duration|Output token throughput|Mean TPOT|Accept length|Mean TTFT' /root/nomtpsweep_cc${cc}.log | head -8
|
|
||||||
if [ $rc -ne 0 ]; then echo "BENCH-FAILED cc=$cc"; fi
|
|
||||||
if ! docker ps --format '{{.Names}}' | grep -q '^glm53-nvfp4$'; then
|
|
||||||
echo "CONTAINER-DIED at cc=$cc $(date +%T)"; docker logs --tail 40 glm53-nvfp4 2>&1 | grep -iE 'error|assert' | tail -8; break
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
echo SWEEP-DONE
|
|
||||||
Loading…
x
Reference in New Issue
Block a user