Tokens are cumulative (they include restored predictions' usage) while
time was this-run wall -- fully replayed benches showed 0s in the same
column, reading as broken. run_info now carries gen_fresh; a bench with
zero fresh generations renders its time cell as 'cached' (tokens keep
showing the true cumulative investment).
Co-Authored-By: Claude <noreply@anthropic.com>
One sample whose stream never got a first byte (prefill queue at high
gate levels) exhausted 6 adapter retries and killed the WHOLE gather --
500 samples died with it. Terminal failures are now contained: empty
prediction (scores wrong, es-parity for timeouts), NOT checkpointed so
a rerun retries them, prominently counted; only a 100% wipeout fails
the bench. Stream-aggregate read timeout 60s -> 240s: a 128k prompt
queued behind other prefills legitimately takes minutes to start
answering; real hangs are now the gate's job (x0.7) and contained
failures rather than bench death.
Verified: 1-in-3 terminal failures -> bench completes 6 ok / 3 empty,
checkpoint holds only the 6.
Co-Authored-By: Claude <noreply@anthropic.com>
95 waiters all tokenizing 2M-token docs through the 32-thread default
executor saturated the GIL: the rich render thread and the event loop
starved, so the bar froze and jumped (and the gate probe went blind).
Truncation now runs on a dedicated 8-thread pool; the remaining
workers queue and the loop/renderer stay responsive.
eta formats as 45s / 7m15s / 2h35m40s / 6d03h12m as it grows.
Co-Authored-By: Claude <noreply@anthropic.com>
assemble() runs the max_input_tokens truncation tokenizer inline in the
coroutine; longbench_v2's 2M-token docs take seconds of CPU per encode,
and each one BLOCKED the whole loop -- heartbeat frozen, gate probes
dead, zero HTTP while the process sat at 100% single-core. encode now
runs in a worker thread (loop stays live, encodes parallelize).
Co-Authored-By: Claude <noreply@anthropic.com>
User feedback: the out-dir report-reuse layer was one concept too many.
Now each checkpoint line carries {key, ts, pred, score}:
- --resume restores predictions AND their scores; when every sample's
cached score matches the scoring-setup fingerprint (recipe/extract/
scorers/judge), the report is replayed with NO scorer, extractor or
docker container touching anything
- fingerprint mismatch (recipe or judge changed) -> automatic re-eval
and backfill of the fresh scores
- no --resume -> nothing read, nothing written (full fresh run)
- --rescore = ignore cached scores, re-evaluate, refresh the cache
- aggregation always recomputed from cached per-sample scores (cheap,
survives aggregator changes without invalidating)
- legacy checkpoints without a score field backfill on first evaluation
Removed: the out-dir report-reuse block (superseded; also the source of
the UnboundLocalError path).
Co-Authored-By: Claude <noreply@anthropic.com>
--auto-concurrency wraps a single endpoint as a 1-member pool; the
spec parser rejected it with 'needs a {start..end} port range' and
killed every benchmark. Range-less comma segments are now plain
members; only a fully empty endpoint list errors.
Co-Authored-By: Claude <noreply@anthropic.com>
Scoring progress: docker-exec benches (humaneval etc.) score for
minutes with zero feedback -- the bar sat at 'generating 100%' and
looked hung. evaluate() now takes on_scored(i, n) (atomic counter,
fires from worker threads), run_eval passes it through, and the CLI
shows 'scoring 42/164' on the bar + milestone log lines every 10%
(also fixes the phase match: 'scoring' never matched the capitalized
'Scoring predictions...' status message, so the bar never even
switched its label).
PooledAdapter:
- one release per acquire: the exception path released True (inner
finally) AND False (except handler), double-decrementing _inflight
(over-admission) and applying the x0.7 backoff twice
- AdaptiveGate: rebuild the Condition + probe task when the event loop
changes -- pools are cached across benchmarks and the CLI runs
asyncio.run() per bench/repeat; a loop-bound Condition from a closed
loop raises 'bound to a different event loop' under contention
Co-Authored-By: Claude <noreply@anthropic.com>
- perf_stats aggregator lives in eval/, not model/: the import failed
silently and EVERY perf column was empty (not just ttft). Now warns
on stderr instead of swallowing.
- repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2
previously restored repeat 1's predictions and finished instantly with
identical scores. rep1 keeps the legacy key (existing checkpoints still
resume).
- repeats summary: report the MEAN score and aggregate time/tokens over
ALL runs (was: last run only).
- README: six-benchmark command as the primary example.
Co-Authored-By: Claude <noreply@anthropic.com>
Was: raw load_raw_records() hit the hub on EVERY run -- offline machines
stalled in 5x HF retries then silently degraded to 0-shot (changing the
benchmark's default contract, e.g. gsm8k 4-shot). Now the few-shot split
is a proper Dataset entry: first use downloads+cache, every later run is
a pure cache hit (verified: second run with HF_HUB_OFFLINE=1 loads
4-shot from cache, zero network lines, acc unchanged).
Co-Authored-By: Claude <noreply@anthropic.com>
Explicit key flows only into request headers (adapter attribute / pool
member), so it cannot leak into the spec string, EvalReport, or logs --
verified by scanning a report produced with a sentinel key. Two-key
setups run twice with different --api-key, or use per-host env vars.
Co-Authored-By: Claude <noreply@anthropic.com>
- --judge now accepts a bare model name paired with --judge-api-url,
mirroring --model/--api-url; full legacy specs keep working
(_compose_judge_spec, verified: name+url -> spec, spec passthrough,
empty -> None; end-to-end on simple_qa with a live judge endpoint)
- mock adapter spellings: mock-boxed / mock-oracle / mock-fc preferred,
colon forms still accepted; bare 'mock' stays echo
- fix mock adapter singleton mode pollution: resolve_adapter memoizes
one instance, so mock-boxed then mock in one process leaked the
boxed mode into the echo run -- each mock spec now builds a fresh
instance
- README: mock-boxed in examples, judge flags row updated
Co-Authored-By: Claude <noreply@anthropic.com>
