115m39s next to eta 2h51m01s read inconsistently; all durations now
render as 45s / 7m15s / 1h55m39s / 2h51m01s / 3d02h00m.
Co-Authored-By: Claude <noreply@anthropic.com>
95 waiters all tokenizing 2M-token docs through the 32-thread default
executor saturated the GIL: the rich render thread and the event loop
starved, so the bar froze and jumped (and the gate probe went blind).
Truncation now runs on a dedicated 8-thread pool; the remaining
workers queue and the loop/renderer stay responsive.
eta formats as 45s / 7m15s / 2h35m40s / 6d03h12m as it grows.
Co-Authored-By: Claude <noreply@anthropic.com>
The slash form read as a fraction and kept inviting 'why is the
denominator growing' -- it is the pipeline depth (tokenizing +
gate-queued + generating), bounded by the global semaphore, while the
gate-admitted count is the real server load.
Co-Authored-By: Claude <noreply@anthropic.com>
The counter was taken just past the GLOBAL semaphore (lifted to 96 in
auto mode so the gate is the sole limiter) but BEFORE the pool gate --
so 94 gate-queued workers counted as in-flight. The gate now pushes
its actually-admitted count and the bar shows 'admitted/held'
(e.g. in-flight 2/96 = 2 really hitting the server, 94 queued on the
gate). Verified: admitted never exceeds the gate limit.
Co-Authored-By: Claude <noreply@anthropic.com>
- new flag wraps even a single endpoint as a one-member pool so the
per-endpoint AdaptiveGate takes over; --concurrency becomes the gate's
STARTING point (global semaphore lifted to the gate ceiling of 96)
- demand-driven AIMD fallback for endpoints without /metrics (404,
gateway-stripped, non-sglang): ramp +1 while callers wait on acquire
and the interval is failure-free; the old code early-returned on
fetch errors and never adapted at all. Real GLM endpoint verified:
/metrics is 404, so this fallback is the live path there
- probe fetch moved to a thread: a blocked urlopen parked the whole
event loop for its 4s timeout
- current limit surfaces on the progress bar as a magenta 'gate N'
field (pushed on every change)
Co-Authored-By: Claude <noreply@anthropic.com>
The bar kept its stale 100% generation state during scoring with only
the description counter moving. Now set_scoring() refills the bar with
judged samples (0->100%, fresh clock/rate/eta for the phase), clears
the generation '+N new' marker, and the next bench retargets back to
generating cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
- Completed X/Y now carries (+N new): samples generated by THIS run,
excluding the checkpoint-restored head start (previously 98/164 told
you nothing about how much work this invocation actually did)
- rate/eta computed over fresh samples only (restored ones counted
toward 141/6s = 23/s when 1 sample had been generated)
- fix KeyError 'success': advance() still read the success/failed task
fields after they were dropped from the column set -- first advance
on a resumed bench killed the whole benchmark
- set_overall: give the overall task the fields the shared columns
actually read (cur/retries/elapsed/eta); the old field set was from
a previous column layout
Co-Authored-By: Claude <noreply@anthropic.com>
