The preflight did a bare 'docker pull' -- docker.io is slow/unreachable
from CN without luck; it now walks the same fallback chain as the SWE
prefetch (daemon mirrors -> daocloud -> 1ms.run -> baidubce -> sjtu ->
rat.dev), retagging the hit to the canonical name.
Co-Authored-By: Claude <noreply@anthropic.com>
1140 real predictions scored 0.0% because the recipe referenced
'bigcodebench-sandbox:latest' -- a name nothing builds and docker.io
does not have; every sample then burned 3 pull-retries (~200s each).
- image -> bigcodebench/bigcodebench-evaluate:latest (the official hub
image, same one evalscope uses)
- ensure_image() preflight in evaluate(): recipe-level AND sample-level
images are verified/pulled ONCE before any container runs; missing ->
seconds-fast bench failure with a fix hint instead of a silent 0.0%
- docker exec: 'Unable to find image'/'manifest unknown' class errors
are permanent -- no 3x retry amplification
Verified with a bogus image: preflight raises in one pull-attempt with
the fix hint.
Co-Authored-By: Claude <noreply@anthropic.com>
Five categories (Code & Engineering / Reasoning & Math / Knowledge &
Language / Long Context / Agents & Tools) now annotate the console
summary table (new column), summary.csv (new field), and each report's
run_info.category.
Co-Authored-By: Claude <noreply@anthropic.com>
Tokens are cumulative (they include restored predictions' usage) while
time was this-run wall -- fully replayed benches showed 0s in the same
column, reading as broken. run_info now carries gen_fresh; a bench with
zero fresh generations renders its time cell as 'cached' (tokens keep
showing the true cumulative investment).
Co-Authored-By: Claude <noreply@anthropic.com>
115m39s next to eta 2h51m01s read inconsistently; all durations now
render as 45s / 7m15s / 1h55m39s / 2h51m01s / 3d02h00m.
Co-Authored-By: Claude <noreply@anthropic.com>
One sample whose stream never got a first byte (prefill queue at high
gate levels) exhausted 6 adapter retries and killed the WHOLE gather --
500 samples died with it. Terminal failures are now contained: empty
prediction (scores wrong, es-parity for timeouts), NOT checkpointed so
a rerun retries them, prominently counted; only a 100% wipeout fails
the bench. Stream-aggregate read timeout 60s -> 240s: a 128k prompt
queued behind other prefills legitimately takes minutes to start
answering; real hangs are now the gate's job (x0.7) and contained
failures rather than bench death.
Verified: 1-in-3 terminal failures -> bench completes 6 ok / 3 empty,
checkpoint holds only the 6.
Co-Authored-By: Claude <noreply@anthropic.com>
User-directed changes:
- steady no longer pins the converged level: the endpoint is shared,
other tenants move its capacity mid-run, so steady keeps judging
forever (+1 when rate beats reference by 5%, -1 when 15% below,
reference drifts by EWMA). Large drops are still handled by the
failure channel's multiplicative x0.7; the +-1 path tracks drift.
- dwell fallback scales with the OBSERVED completion cadence:
max(120s, 3x inter-completion gap EMA). A 25s timer judged 60s-per-
request benches on one lone sample.
- MIN_OK floor raised to 5 (evidence = max(5, 2x level)).
Simulated capacity drift 8->3->8: gate follows down then recovers.
Co-Authored-By: Claude <noreply@anthropic.com>
Demanding a 10% gain to keep doubling settled [1,2]->1 on the first
noisy plateau (lbv2: 4k..2M-token docs, completion-rate noise dwarfs
10%). Now: keep climbing while not clearly worse (>=0.9x); bisect only
on clear degradation; samples per level doubled (max(3, 2*level)) to
shrink noise; steady re-probes +1 after ~60s so a noise-induced settle
cannot pin the gate forever. Overshoot past the true knee is trimmed
by the failure channel (timeouts -> x0.7), which is the real ceiling
finder on a prefill-bound endpoint.
Noise-swept at +-25%: capacities 4/8/16 settle at 11/31/16 without a
failure model; production failures pull the overshoot back down.
Co-Authored-By: Claude <noreply@anthropic.com>
95 waiters all tokenizing 2M-token docs through the 32-thread default
executor saturated the GIL: the rich render thread and the event loop
starved, so the bar froze and jumped (and the gate probe went blind).
Truncation now runs on a dedicated 8-thread pool; the remaining
workers queue and the loop/renderer stay responsive.
eta formats as 45s / 7m15s / 2h35m40s / 6d03h12m as it grows.
Co-Authored-By: Claude <noreply@anthropic.com>
The 5s probe fetched /metrics via asyncio.to_thread, which shares the
DEFAULT executor with second-long truncation tokenizations -- 96 of
those queue-jumped the probe and the state machine never ticked (gate
frozen at 1 while results flowed, ETA 6h). Probes now run on a
dedicated single-thread executor, and after 3 consecutive fetch
failures the gate stops asking for /metrics entirely (this endpoint
404s; pure demand mode from then on).
Verified under a choked default executor: gate ticks 1->2 on schedule
and metrics_dead engages after 3 failures.
Co-Authored-By: Claude <noreply@anthropic.com>
The slash form read as a fraction and kept inviting 'why is the
denominator growing' -- it is the pipeline depth (tokenizing +
gate-queued + generating), bounded by the global semaphore, while the
gate-admitted count is the real server load.
Co-Authored-By: Claude <noreply@anthropic.com>
96 worker threads racing transformers 5.x lazy imports on the FIRST
_get_tokenizer call raised ImportError and degraded that whole first
batch to the char approximation (the old single-threaded path never
raced). First load now holds a threading.Lock; the transformers
'>model_max_length' logging is silenced inside truncation (counting a
2M-token doc before trimming it is the point), and the per-sample
degradation print becomes a one-shot warning.
Co-Authored-By: Claude <noreply@anthropic.com>
The counter was taken just past the GLOBAL semaphore (lifted to 96 in
auto mode so the gate is the sole limiter) but BEFORE the pool gate --
so 94 gate-queued workers counted as in-flight. The gate now pushes
its actually-admitted count and the bar shows 'admitted/held'
(e.g. in-flight 2/96 = 2 really hitting the server, 94 queued on the
gate). Verified: admitted never exceeds the gate limit.
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces the +1/5s linear ramp: start at 1, double while measured
completions/s keeps improving (>10% over the previous level); the
first plateau opens a bisect [last_good, bad] that narrows to the
knee, then holds steady. Failures still cut x0.7 instantly and
restart probing from the shrunken level; zero completions = hold.
Judging a level needs max(MIN_OK, level) completions -- a
2-completion rate estimate at level 8 is quantization noise (caught
by simulation converging to 1 on a capacity-8 endpoint).
Simulated against throughput curves min(level, capacity):
capacity 8 -> 1,2,4,8,16 | bisect 12,10,9 -> steady 8
capacity 16 -> 1,2,4,8,16,32 | bisect ... -> steady 16
capacity 4 -> 1,2,4,8 | bisect 6,5 -> steady 4
Co-Authored-By: Claude <noreply@anthropic.com>
Ramping on demand alone was dangerous with slow endpoints: if the 2
in-flight longbench_v2 requests hang, 'no failures + waiters queued'
kept adding +1 every 5s all the way to 96 -- piling prefills onto a
server that had not answered anything. Both ramp paths (demand-driven
and /metrics) now require at least one SUCCESSFUL completion in the
probe interval; hangs hold the gate until read-timeouts fire and the
x0.7 backoff takes over.
Unit-verified: hang 4 probe intervals with 20 waiters -> limit stays 2;
one success -> +1; one failure -> x0.7.
Co-Authored-By: Claude <noreply@anthropic.com>
assemble() runs the max_input_tokens truncation tokenizer inline in the
coroutine; longbench_v2's 2M-token docs take seconds of CPU per encode,
and each one BLOCKED the whole loop -- heartbeat frozen, gate probes
dead, zero HTTP while the process sat at 100% single-core. encode now
runs in a worker thread (loop stays live, encodes parallelize).
Co-Authored-By: Claude <noreply@anthropic.com>
--auto-concurrency removed as promised; 'auto' now starts the adaptive
gate at 2 (it ramps on its own demand/health signals). Plan displays
'auto (start 2, gate decides)'.
Co-Authored-By: Claude <noreply@anthropic.com>
The auto-stream trigger only looked at max_tokens (output budget);
longbench_v2 sends ~128k-token INPUTS with a small 8k output budget and
hit the gateway's whole-request buffering: 20-minute read timeouts
through all 6 retries. es survives the same bench because it streams
by default. Payloads over ~300k chars (~90k+ tokens) now take the
stream-aggregate path; verified live: a 350k-char prompt returns in
20s instead of hanging.
Co-Authored-By: Claude <noreply@anthropic.com>
Full traceback finally caught it: adapter._parse ALWAYS ran the
text-protocol tool-call fallback, even for requests with NO tools. On
humaneval, model code like regex-matched as a
'call', ast.literal_eval turned the literal into an Ellipsis
(no exception -- it's a legal literal), and json.dumps(args) died
mid-generation, killing the benchmark.
Two layers:
- the fallback now only runs when the request actually carried tools
(also stops polluting plain predictions with phantom calls, and the
SyntaxWarning spam from ast.parse-ing model code disappears)
- json.dumps(args, default=str) as belt-and-braces for the
text-tools path where an Ellipsis arg now stringifies
Reproduced the exact crash input as a unit case: no-tools code reply
yields 0 tool_calls; in text mode serializes
{'key': 'Ellipsis'} without raising; normal fc calls unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
Adding config/sample_counts.yaml made TWO yamls in config/, and the
auto-load rule was 'exactly one yaml' -- so every bench silently lost
its repeats/temperature/max_tokens (humaneval ran once instead of 3).
Manifest renamed to .json, and the rule hardened: a lone yaml still
wins, otherwise default.yaml wins explicitly. Verified: aime 12x +
humaneval 3x repeats active again.
Co-Authored-By: Claude <noreply@anthropic.com>
- sample-counts manifest (config/sample_counts.yaml, harvested from
real runs): uncached benches still show exact numbers in the plan
instead of 'counts when datasets load' -- 'cache+est.' marks the mix
- '--concurrency auto' is now an alias for --auto-concurrency
- Concurrency row shows 'auto (start 8, gate decides)' when the gate
drives, instead of a bare misleading 8
Also verified end-to-end: thinking-mode humaneval rep1/rep2 both
pass 98.8%, matching the es reference runs (98.17/98.78/98.78) on the
same model -- framework alignment holds on the thinking path too.
Co-Authored-By: Claude <noreply@anthropic.com>
syy's es runs: humaneval x3, gpqa x2, aime25/26 x12 (already aligned),
mmlu_pro/longbench_v2 x1 (temp=0 deterministic -- repeats are noise,
and 3x 12k samples is pure cost). temp=1 benches get 3.
Co-Authored-By: Claude <noreply@anthropic.com>
A finished 12-minute humaneval bench died at the save line: some scorer
detail carried a non-JSON object (Ellipsis) and report.save's json.dumps
had no default. Every dumps on the save path now stringifies exotic
objects instead of killing the run; verified by replaying the exact
crashed checkpoint end-to-end (pass 82.3%, 15s, no crash).
Co-Authored-By: Claude <noreply@anthropic.com>
User feedback: the out-dir report-reuse layer was one concept too many.
Now each checkpoint line carries {key, ts, pred, score}:
- --resume restores predictions AND their scores; when every sample's
cached score matches the scoring-setup fingerprint (recipe/extract/
scorers/judge), the report is replayed with NO scorer, extractor or
docker container touching anything
- fingerprint mismatch (recipe or judge changed) -> automatic re-eval
and backfill of the fresh scores
- no --resume -> nothing read, nothing written (full fresh run)
- --rescore = ignore cached scores, re-evaluate, refresh the cache
- aggregation always recomputed from cached per-sample scores (cheap,
survives aggregator changes without invalidating)
- legacy checkpoints without a score field backfill on first evaluation
Removed: the out-dir report-reuse block (superseded; also the source of
the UnboundLocalError path).
Co-Authored-By: Claude <noreply@anthropic.com>
The progress-reporter setup lived inside the generation branch; reuse
skipped it but the success path still advanced the overall bar ->
'cannot access local variable progress_reporter'. Setup now runs for
every path (reuse included).
Co-Authored-By: Claude <noreply@anthropic.com>
time_s/time_h summed per-prediction latency_s, which includes RESTORED
predictions' original generation time -- days old and from a slower
setup, it once reported 15.9h for a one-hour aime25 run. All rows now
report the bench's actual wall clock; token totals stay as the true
cost of the predictions used.
n for repeats>1 is num_samples x repeats (12 runs over 30 problems is
360 generations, not 30).
Co-Authored-By: Claude <noreply@anthropic.com>
The checkpoint stores predictions, not scores -- reruns re-executed the
whole docker scoring pass even when nothing could change. When the
saved report matches (same model, same sample count) and every
prediction is checkpointed, reuse it; --rescore forces evaluation
(recipe/judge changed).
Also fixes EvalReport.load: metric_groups declared Dict[str, float]
rejected the None perf values (and the repeats scores list) the file
format actually contains -- every load raised ValidationError, which
is what silently killed reuse in testing.
Repeats>1 benches never reuse: their whole point is fresh sampling.
Co-Authored-By: Claude <noreply@anthropic.com>
Previously both were written only after ALL benches finished -- a
multi-hour mmlu_pro left the summary stale for hours, and a crashed
run left the PREVIOUS run's files in place. Now each completed bench
(including failed ones) rewrites the summaries with everything so far.
Co-Authored-By: Claude <noreply@anthropic.com>
Each file carries that run's own score and samples; the top-level
report.jsonl stays the mean-summary view (mean + repeats group).
Co-Authored-By: Claude <noreply@anthropic.com>
12 repeats each scored on the terminal but the artifacts kept only the
final repeat's report; the run list scrolled away. Now the last report
carries metric_groups['repeats'] = {n_runs, scores[12], mean, min, max,
std} -- lands in report.jsonl's header line, the xlsx categories sheet,
and summary.csv's categories column (numeric entries); the console
summary line also prints the full runs=[...] list.
Co-Authored-By: Claude <noreply@anthropic.com>
--auto-concurrency wraps a single endpoint as a 1-member pool; the
spec parser rejected it with 'needs a {start..end} port range' and
killed every benchmark. Range-less comma segments are now plain
members; only a fully empty endpoint list errors.
Co-Authored-By: Claude <noreply@anthropic.com>
- new flag wraps even a single endpoint as a one-member pool so the
per-endpoint AdaptiveGate takes over; --concurrency becomes the gate's
STARTING point (global semaphore lifted to the gate ceiling of 96)
- demand-driven AIMD fallback for endpoints without /metrics (404,
gateway-stripped, non-sglang): ramp +1 while callers wait on acquire
and the interval is failure-free; the old code early-returned on
fetch errors and never adapted at all. Real GLM endpoint verified:
/metrics is 404, so this fallback is the live path there
- probe fetch moved to a thread: a blocked urlopen parked the whole
event loop for its 4s timeout
- current limit surfaces on the progress bar as a magenta 'gate N'
field (pushed on every change)
Co-Authored-By: Claude <noreply@anthropic.com>
The bar kept its stale 100% generation state during scoring with only
the description counter moving. Now set_scoring() refills the bar with
judged samples (0->100%, fresh clock/rate/eta for the phase), clears
the generation '+N new' marker, and the next bench retargets back to
generating cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
The bar's 'scoring N/164' counter and 21 milestone log lines were both
active on terminals -- the lines are noise there. Emit them only for
pipes/redirects where no bar exists (every ~10% now).
Co-Authored-By: Claude <noreply@anthropic.com>
An unbounded docker rm against a bloated daemon hangs for minutes and
silently eats the worker pool: 7 of 8 scoring workers were observed
stuck in cleanup while only 1 execution ran.
Co-Authored-By: Claude <noreply@anthropic.com>
- docker exec: named containers; a timed-out/killed 'docker run' only
kills the CLI client while the container lives on (--rm fires on
EXIT) -- rm -f the name on timeout/interrupt so runs stop leaking
- exit 125 = daemon-side failure, not model failure: retry up to 2x
(a bloated daemon was turning healthy samples into pass=0)
- scoring milestones: first completion logs immediately, then every
~5% (10% was too sparse when docker is slow: minutes of silence
right after the 'scoring' phase starts, looks hung)
Co-Authored-By: Claude <noreply@anthropic.com>
Scoring progress: docker-exec benches (humaneval etc.) score for
minutes with zero feedback -- the bar sat at 'generating 100%' and
looked hung. evaluate() now takes on_scored(i, n) (atomic counter,
fires from worker threads), run_eval passes it through, and the CLI
shows 'scoring 42/164' on the bar + milestone log lines every 10%
(also fixes the phase match: 'scoring' never matched the capitalized
'Scoring predictions...' status message, so the bar never even
switched its label).
PooledAdapter:
- one release per acquire: the exception path released True (inner
finally) AND False (except handler), double-decrementing _inflight
(over-admission) and applying the x0.7 backoff twice
- AdaptiveGate: rebuild the Condition + probe task when the event loop
changes -- pools are cached across benchmarks and the CLI runs
asyncio.run() per bench/repeat; a loop-bound Condition from a closed
loop raises 'bound to a different event loop' under contention
Co-Authored-By: Claude <noreply@anthropic.com>
- Completed X/Y now carries (+N new): samples generated by THIS run,
excluding the checkpoint-restored head start (previously 98/164 told
you nothing about how much work this invocation actually did)
- rate/eta computed over fresh samples only (restored ones counted
toward 141/6s = 23/s when 1 sample had been generated)
- fix KeyError 'success': advance() still read the success/failed task
fields after they were dropped from the column set -- first advance
on a resumed bench killed the whole benchmark
- set_overall: give the overall task the fields the shared columns
actually read (cur/retries/elapsed/eta); the old field set was from
a previous column layout
Co-Authored-By: Claude <noreply@anthropic.com>
Samples row said 'full dataset (counted when each loads)' -- now it
counts cached samples.jsonl entries (never touches the network, plan
stays instant on cold machines) and multiplies by YAML repeats:
12,957 samples (cached) → 13,617 generations (repeats)
--limit caps each bench before summing; uncached benches are flagged.
YAML config loading extracted to _load_bench_cfg, shared by the run
loop and the plan so repeats can't disagree between the two.
Co-Authored-By: Claude <noreply@anthropic.com>
- accurate output structure (summary.xlsx/csv + per-bench xlsx; drop
stale detail.md/summary.md mentions)
- dedicated YAML-config section documenting the keys that actually work
(generation params + repeats + max_input_tokens) and precedence
- perf-stats table: what is always collected vs --perf streaming-only
- FAQ: endpoint probe, context-overflow shrink, thinking-mode notes
(non-stream chat_template_kwargs vs cloud-API param), auto-stream
- config/README.md rewritten to match the flat-key reality (old file
documented a directory scheme + judge/env keys that are not consumed)
Co-Authored-By: Claude <noreply@anthropic.com>
- perf_stats aggregator lives in eval/, not model/: the import failed
silently and EVERY perf column was empty (not just ttft). Now warns
on stderr instead of swallowing.
- repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2
previously restored repeat 1's predictions and finished instantly with
identical scores. rep1 keeps the legacy key (existing checkpoints still
resume).
- repeats summary: report the MEAN score and aggregate time/tokens over
ALL runs (was: last run only).
- README: six-benchmark command as the primary example.
Co-Authored-By: Claude <noreply@anthropic.com>
