User-directed changes:
- steady no longer pins the converged level: the endpoint is shared,
other tenants move its capacity mid-run, so steady keeps judging
forever (+1 when rate beats reference by 5%, -1 when 15% below,
reference drifts by EWMA). Large drops are still handled by the
failure channel's multiplicative x0.7; the +-1 path tracks drift.
- dwell fallback scales with the OBSERVED completion cadence:
max(120s, 3x inter-completion gap EMA). A 25s timer judged 60s-per-
request benches on one lone sample.
- MIN_OK floor raised to 5 (evidence = max(5, 2x level)).
Simulated capacity drift 8->3->8: gate follows down then recovers.
Co-Authored-By: Claude <noreply@anthropic.com>
Demanding a 10% gain to keep doubling settled [1,2]->1 on the first
noisy plateau (lbv2: 4k..2M-token docs, completion-rate noise dwarfs
10%). Now: keep climbing while not clearly worse (>=0.9x); bisect only
on clear degradation; samples per level doubled (max(3, 2*level)) to
shrink noise; steady re-probes +1 after ~60s so a noise-induced settle
cannot pin the gate forever. Overshoot past the true knee is trimmed
by the failure channel (timeouts -> x0.7), which is the real ceiling
finder on a prefill-bound endpoint.
Noise-swept at +-25%: capacities 4/8/16 settle at 11/31/16 without a
failure model; production failures pull the overshoot back down.
Co-Authored-By: Claude <noreply@anthropic.com>
The 5s probe fetched /metrics via asyncio.to_thread, which shares the
DEFAULT executor with second-long truncation tokenizations -- 96 of
those queue-jumped the probe and the state machine never ticked (gate
frozen at 1 while results flowed, ETA 6h). Probes now run on a
dedicated single-thread executor, and after 3 consecutive fetch
failures the gate stops asking for /metrics entirely (this endpoint
404s; pure demand mode from then on).
Verified under a choked default executor: gate ticks 1->2 on schedule
and metrics_dead engages after 3 failures.
Co-Authored-By: Claude <noreply@anthropic.com>
The counter was taken just past the GLOBAL semaphore (lifted to 96 in
auto mode so the gate is the sole limiter) but BEFORE the pool gate --
so 94 gate-queued workers counted as in-flight. The gate now pushes
its actually-admitted count and the bar shows 'admitted/held'
(e.g. in-flight 2/96 = 2 really hitting the server, 94 queued on the
gate). Verified: admitted never exceeds the gate limit.
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces the +1/5s linear ramp: start at 1, double while measured
completions/s keeps improving (>10% over the previous level); the
first plateau opens a bisect [last_good, bad] that narrows to the
knee, then holds steady. Failures still cut x0.7 instantly and
restart probing from the shrunken level; zero completions = hold.
Judging a level needs max(MIN_OK, level) completions -- a
2-completion rate estimate at level 8 is quantization noise (caught
by simulation converging to 1 on a capacity-8 endpoint).
Simulated against throughput curves min(level, capacity):
capacity 8 -> 1,2,4,8,16 | bisect 12,10,9 -> steady 8
capacity 16 -> 1,2,4,8,16,32 | bisect ... -> steady 16
capacity 4 -> 1,2,4,8 | bisect 6,5 -> steady 4
Co-Authored-By: Claude <noreply@anthropic.com>
Ramping on demand alone was dangerous with slow endpoints: if the 2
in-flight longbench_v2 requests hang, 'no failures + waiters queued'
kept adding +1 every 5s all the way to 96 -- piling prefills onto a
server that had not answered anything. Both ramp paths (demand-driven
and /metrics) now require at least one SUCCESSFUL completion in the
probe interval; hangs hold the gate until read-timeouts fire and the
x0.7 backoff takes over.
Unit-verified: hang 4 probe intervals with 20 waiters -> limit stays 2;
one success -> +1; one failure -> x0.7.
Co-Authored-By: Claude <noreply@anthropic.com>
- new flag wraps even a single endpoint as a one-member pool so the
per-endpoint AdaptiveGate takes over; --concurrency becomes the gate's
STARTING point (global semaphore lifted to the gate ceiling of 96)
- demand-driven AIMD fallback for endpoints without /metrics (404,
gateway-stripped, non-sglang): ramp +1 while callers wait on acquire
and the interval is failure-free; the old code early-returned on
fetch errors and never adapted at all. Real GLM endpoint verified:
/metrics is 404, so this fallback is the live path there
- probe fetch moved to a thread: a blocked urlopen parked the whole
event loop for its 4s timeout
- current limit surfaces on the progress bar as a magenta 'gate N'
field (pushed on every change)
Co-Authored-By: Claude <noreply@anthropic.com>
Scoring progress: docker-exec benches (humaneval etc.) score for
minutes with zero feedback -- the bar sat at 'generating 100%' and
looked hung. evaluate() now takes on_scored(i, n) (atomic counter,
fires from worker threads), run_eval passes it through, and the CLI
shows 'scoring 42/164' on the bar + milestone log lines every 10%
(also fixes the phase match: 'scoring' never matched the capitalized
'Scoring predictions...' status message, so the bar never even
switched its label).
PooledAdapter:
- one release per acquire: the exception path released True (inner
finally) AND False (except handler), double-decrementing _inflight
(over-admission) and applying the x0.7 backoff twice
- AdaptiveGate: rebuild the Condition + probe task when the event loop
changes -- pools are cached across benchmarks and the CLI runs
asyncio.run() per bench/repeat; a loop-bound Condition from a closed
loop raises 'bound to a different event loop' under contention
Co-Authored-By: Claude <noreply@anthropic.com>
Explicit key flows only into request headers (adapter attribute / pool
member), so it cannot leak into the spec string, EvalReport, or logs --
verified by scanning a report produced with a sentinel key. Two-key
setups run twice with different --api-key, or use per-host env vars.
Co-Authored-By: Claude <noreply@anthropic.com>
