assemble() runs the max_input_tokens truncation tokenizer inline in the
coroutine; longbench_v2's 2M-token docs take seconds of CPU per encode,
and each one BLOCKED the whole loop -- heartbeat frozen, gate probes
dead, zero HTTP while the process sat at 100% single-core. encode now
runs in a worker thread (loop stays live, encodes parallelize).
Co-Authored-By: Claude <noreply@anthropic.com>
The auto-stream trigger only looked at max_tokens (output budget);
longbench_v2 sends ~128k-token INPUTS with a small 8k output budget and
hit the gateway's whole-request buffering: 20-minute read timeouts
through all 6 retries. es survives the same bench because it streams
by default. Payloads over ~300k chars (~90k+ tokens) now take the
stream-aggregate path; verified live: a 350k-char prompt returns in
20s instead of hanging.
Co-Authored-By: Claude <noreply@anthropic.com>
Full traceback finally caught it: adapter._parse ALWAYS ran the
text-protocol tool-call fallback, even for requests with NO tools. On
humaneval, model code like regex-matched as a
'call', ast.literal_eval turned the literal into an Ellipsis
(no exception -- it's a legal literal), and json.dumps(args) died
mid-generation, killing the benchmark.
Two layers:
- the fallback now only runs when the request actually carried tools
(also stops polluting plain predictions with phantom calls, and the
SyntaxWarning spam from ast.parse-ing model code disappears)
- json.dumps(args, default=str) as belt-and-braces for the
text-tools path where an Ellipsis arg now stringifies
Reproduced the exact crash input as a unit case: no-tools code reply
yields 0 tool_calls; in text mode serializes
{'key': 'Ellipsis'} without raising; normal fc calls unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
User feedback: the out-dir report-reuse layer was one concept too many.
Now each checkpoint line carries {key, ts, pred, score}:
- --resume restores predictions AND their scores; when every sample's
cached score matches the scoring-setup fingerprint (recipe/extract/
scorers/judge), the report is replayed with NO scorer, extractor or
docker container touching anything
- fingerprint mismatch (recipe or judge changed) -> automatic re-eval
and backfill of the fresh scores
- no --resume -> nothing read, nothing written (full fresh run)
- --rescore = ignore cached scores, re-evaluate, refresh the cache
- aggregation always recomputed from cached per-sample scores (cheap,
survives aggregator changes without invalidating)
- legacy checkpoints without a score field backfill on first evaluation
Removed: the out-dir report-reuse block (superseded; also the source of
the UnboundLocalError path).
Co-Authored-By: Claude <noreply@anthropic.com>
--auto-concurrency wraps a single endpoint as a 1-member pool; the
spec parser rejected it with 'needs a {start..end} port range' and
killed every benchmark. Range-less comma segments are now plain
members; only a fully empty endpoint list errors.
Co-Authored-By: Claude <noreply@anthropic.com>
- new flag wraps even a single endpoint as a one-member pool so the
per-endpoint AdaptiveGate takes over; --concurrency becomes the gate's
STARTING point (global semaphore lifted to the gate ceiling of 96)
- demand-driven AIMD fallback for endpoints without /metrics (404,
gateway-stripped, non-sglang): ramp +1 while callers wait on acquire
and the interval is failure-free; the old code early-returned on
fetch errors and never adapted at all. Real GLM endpoint verified:
/metrics is 404, so this fallback is the live path there
- probe fetch moved to a thread: a blocked urlopen parked the whole
event loop for its 4s timeout
- current limit surfaces on the progress bar as a magenta 'gate N'
field (pushed on every change)
Co-Authored-By: Claude <noreply@anthropic.com>
Scoring progress: docker-exec benches (humaneval etc.) score for
minutes with zero feedback -- the bar sat at 'generating 100%' and
looked hung. evaluate() now takes on_scored(i, n) (atomic counter,
fires from worker threads), run_eval passes it through, and the CLI
shows 'scoring 42/164' on the bar + milestone log lines every 10%
(also fixes the phase match: 'scoring' never matched the capitalized
'Scoring predictions...' status message, so the bar never even
switched its label).
PooledAdapter:
- one release per acquire: the exception path released True (inner
finally) AND False (except handler), double-decrementing _inflight
(over-admission) and applying the x0.7 backoff twice
- AdaptiveGate: rebuild the Condition + probe task when the event loop
changes -- pools are cached across benchmarks and the CLI runs
asyncio.run() per bench/repeat; a loop-bound Condition from a closed
loop raises 'bound to a different event loop' under contention
Co-Authored-By: Claude <noreply@anthropic.com>
- perf_stats aggregator lives in eval/, not model/: the import failed
silently and EVERY perf column was empty (not just ttft). Now warns
on stderr instead of swallowing.
- repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2
previously restored repeat 1's predictions and finished instantly with
identical scores. rep1 keeps the legacy key (existing checkpoints still
resume).
- repeats summary: report the MEAN score and aggregate time/tokens over
ALL runs (was: last run only).
- README: six-benchmark command as the primary example.
Co-Authored-By: Claude <noreply@anthropic.com>
Was: raw load_raw_records() hit the hub on EVERY run -- offline machines
stalled in 5x HF retries then silently degraded to 0-shot (changing the
benchmark's default contract, e.g. gsm8k 4-shot). Now the few-shot split
is a proper Dataset entry: first use downloads+cache, every later run is
a pure cache hit (verified: second run with HF_HUB_OFFLINE=1 loads
4-shot from cache, zero network lines, acc unchanged).
Co-Authored-By: Claude <noreply@anthropic.com>
Explicit key flows only into request headers (adapter attribute / pool
member), so it cannot leak into the spec string, EvalReport, or logs --
verified by scanning a report produced with a sentinel key. Two-key
setups run twice with different --api-key, or use per-host env vars.
Co-Authored-By: Claude <noreply@anthropic.com>
- --judge now accepts a bare model name paired with --judge-api-url,
mirroring --model/--api-url; full legacy specs keep working
(_compose_judge_spec, verified: name+url -> spec, spec passthrough,
empty -> None; end-to-end on simple_qa with a live judge endpoint)
- mock adapter spellings: mock-boxed / mock-oracle / mock-fc preferred,
colon forms still accepted; bare 'mock' stays echo
- fix mock adapter singleton mode pollution: resolve_adapter memoizes
one instance, so mock-boxed then mock in one process leaked the
boxed mode into the echo run -- each mock spec now builds a fresh
instance
- README: mock-boxed in examples, judge flags row updated
Co-Authored-By: Claude <noreply@anthropic.com>
