- watchdog: 10s was the user's experiment; back to 5min default
(EVALHARNESS_PULL_IDLE_S overrides). Zero-byte detection itself is
sound: select(fd, timeout) + any-chunk reset, unit-verified.
- a FAILED bench left its 'scoring 0/2' bar on screen through the
NEXT bench's entire dataset download; begin_bench() relabels to
'<name> · loading' and zeroes counters at each bench start.
Co-Authored-By: Claude <noreply@anthropic.com>
The containment check ('{img}' in template) matched ALL entries, so
every announce said 'via docker.io(daemon mirrors)' while the chain
silently rotated real mirrors. The daemon-default entry is exactly
'{img}' -- compare equality.
Co-Authored-By: Claude <noreply@anthropic.com>
16 -> 6 (daemon default, daocloud, 1ms.run, 1panel, rat.dev, ustc):
the long tail never delivered anyway. All-6 failure now prints an
explicit self-load recipe (save/load via an egress machine, or pull
when the network recovers) instead of a terse RuntimeError.
Co-Authored-By: Claude <noreply@anthropic.com>
16 total. The intl entries need real egress (usually blocked on this
host) but cost only 10s each with the idle watchdog on networks that
have them. Also noted: a 10s watchdog can kill sources mid slow
negotiation (rat.dev emitted 'Pulling fs layer' then went quiet) --
EVALHARNESS_PULL_IDLE_S=30 softens that.
Co-Authored-By: Claude <noreply@anthropic.com>
Was 6 explicit sources (the daemon's first hop hides 12 more of its
own); added 1panel.live, dockerproxy, 163, baidubce, ustc, sjtug, nju,
tuna. With the 10s idle watchdog the full sweep costs ~2min worst
case. EVALHARNESS_DOCKER_MIRRORS replaces the whole list.
Co-Authored-By: Claude <noreply@anthropic.com>
Any docker output resets the timer, so this only fires on totally
silent (dead) mirrors; EVALHARNESS_PULL_IDLE_S overrides.
Co-Authored-By: Claude <noreply@anthropic.com>
A dead mirror emitted zero bytes and the fixed 3600s attempt timeout
let it stall the whole chain for an hour (third occurrence tonight:
15m49s frozen at 0 bytes). select() with a 300s idle timeout now kills
the attempt and moves to the next mirror; a trickling source (1 line/s
Downloading updates) resets the timer and stays alive.
Co-Authored-By: Claude <noreply@anthropic.com>
docker refreshes Downloading lines with \r, not \n -- readline()
parked them in the buffer, so the byte snapshots never surfaced and
the overall-progress dict stayed empty (the ⏳ summary never printed).
Chunked reads now split on both terminators; verified with a synthetic
\r-stream that the parser yields the refresh lines and computes the
52.2/65.1MB-style summary.
Co-Authored-By: Claude <noreply@anthropic.com>
Per-layer events gave no sense of how much is LEFT. Downloading lines
are parsed per-layer and summed; every 3s the snapshot becomes
'⏳ 3.14/9.27 GB (34%) · 最大层 3f4a2b1c9d0e: 0.51/1.2 GB' instead of
a bare layer line. Key events (Pulling from / Status / errors) still
print immediately.
Co-ached-By: Claude <noreply@anthropic.com>
With --entrypoint python3, our 'python /work/main.py' runner became
interpreter args: python3 tried to open a FILE named 'python'
("can't open file '/app/python'"). When an entrypoint is set it IS
the interpreter -- pass just /work/<entry>.
Co-Authored-By: Claude <noreply@anthropic.com>
Cold pulls now tick a 'Downloading xMB/yGB' snapshot at most every 3s
(was 10s) -- keyword hits (Pull complete / Status / errors) still
print immediately.
Co-Authored-By: Claude <noreply@anthropic.com>
The official image's ENTRYPOINT is 'python3 -m bigcodebench.evaluate'
-- our 'python /work/main.py' runner was swallowed as CLI args and the
official evaluator died on 'No samples provided'. execution scorer now
accepts entrypoint= and docker exec passes --entrypoint; the BCB recipe
sets entrypoint python3. Also swept 7 leaked eh-exec containers (the
--rm path never fires when our timeout kills the CLI first).
Co-Authored-By: Claude <noreply@anthropic.com>
The scoring phase only reset the bar on the first judged sample; until
then the stale GENERATION state (100%, +N new) stayed on screen --
through multi-minute image-pull preflights it read as 'already done'
while nothing was scored yet.
Co-Authored-By: Claude <noreply@anthropic.com>
Two pull UX/correctness fixes: (1) a multi-GB pull with captured
output is minutes of silence reading as a hang -- layer progress now
streams (throttled) to stderr; (2) the mirror-tag -> canonical retag
could fail silently and the following rmi then deleted the ONLY tag,
losing a 342s pull and forcing a full re-download -- retag is now
verified and the mirror tag kept on failure.
Co-Authored-By: Claude <noreply@anthropic.com>
A multi-GB docker pull ran with fully captured output -- minutes of
silence that read as a hang (user interrupted a healthy run over it).
Each mirror attempt now prints which source it is trying and how long
a hit took.
Co-Authored-By: Claude <noreply@anthropic.com>
One flag for the whole ladder: off == --disable-thinking,
low..max map to reasoning_effort, full = plain default. Takes
precedence over the two older flags. Run Plan shows the active mode
('disabled' / 'enabled · effort=low' / 'enabled').
Co-Authored-By: Claude <noreply@anthropic.com>
The adapter dropped reasoning_effort/thinking from the payload, so the
middle rung of the ladder (es reference: full 98.3 / effort_low 94 /
no-think 82.3 on humaneval) was unreachable. Both keys now pass
through; --reasoning-effort {minimal,low,medium,high,max} overrides
the YAML, and config/effort_low.yaml mirrors default.yaml with
reasoning_effort: low for one-command low-thinking runs.
Probe on the endpoint: same question, default = 319 chars reasoning /
262 tok, low = 47 chars / 117 tok -- the server honors it.
Co-Authored-By: Claude <noreply@anthropic.com>
The preflight did a bare 'docker pull' -- docker.io is slow/unreachable
from CN without luck; it now walks the same fallback chain as the SWE
prefetch (daemon mirrors -> daocloud -> 1ms.run -> baidubce -> sjtu ->
rat.dev), retagging the hit to the canonical name.
Co-Authored-By: Claude <noreply@anthropic.com>
1140 real predictions scored 0.0% because the recipe referenced
'bigcodebench-sandbox:latest' -- a name nothing builds and docker.io
does not have; every sample then burned 3 pull-retries (~200s each).
- image -> bigcodebench/bigcodebench-evaluate:latest (the official hub
image, same one evalscope uses)
- ensure_image() preflight in evaluate(): recipe-level AND sample-level
images are verified/pulled ONCE before any container runs; missing ->
seconds-fast bench failure with a fix hint instead of a silent 0.0%
- docker exec: 'Unable to find image'/'manifest unknown' class errors
are permanent -- no 3x retry amplification
Verified with a bogus image: preflight raises in one pull-attempt with
the fix hint.
Co-Authored-By: Claude <noreply@anthropic.com>
Five categories (Code & Engineering / Reasoning & Math / Knowledge &
Language / Long Context / Agents & Tools) now annotate the console
summary table (new column), summary.csv (new field), and each report's
run_info.category.
Co-Authored-By: Claude <noreply@anthropic.com>
Tokens are cumulative (they include restored predictions' usage) while
time was this-run wall -- fully replayed benches showed 0s in the same
column, reading as broken. run_info now carries gen_fresh; a bench with
zero fresh generations renders its time cell as 'cached' (tokens keep
showing the true cumulative investment).
Co-Authored-By: Claude <noreply@anthropic.com>
115m39s next to eta 2h51m01s read inconsistently; all durations now
render as 45s / 7m15s / 1h55m39s / 2h51m01s / 3d02h00m.
Co-Authored-By: Claude <noreply@anthropic.com>
One sample whose stream never got a first byte (prefill queue at high
gate levels) exhausted 6 adapter retries and killed the WHOLE gather --
500 samples died with it. Terminal failures are now contained: empty
prediction (scores wrong, es-parity for timeouts), NOT checkpointed so
a rerun retries them, prominently counted; only a 100% wipeout fails
the bench. Stream-aggregate read timeout 60s -> 240s: a 128k prompt
queued behind other prefills legitimately takes minutes to start
answering; real hangs are now the gate's job (x0.7) and contained
failures rather than bench death.
Verified: 1-in-3 terminal failures -> bench completes 6 ok / 3 empty,
checkpoint holds only the 6.
Co-Authored-By: Claude <noreply@anthropic.com>
User-directed changes:
- steady no longer pins the converged level: the endpoint is shared,
other tenants move its capacity mid-run, so steady keeps judging
forever (+1 when rate beats reference by 5%, -1 when 15% below,
reference drifts by EWMA). Large drops are still handled by the
failure channel's multiplicative x0.7; the +-1 path tracks drift.
- dwell fallback scales with the OBSERVED completion cadence:
max(120s, 3x inter-completion gap EMA). A 25s timer judged 60s-per-
request benches on one lone sample.
- MIN_OK floor raised to 5 (evidence = max(5, 2x level)).
Simulated capacity drift 8->3->8: gate follows down then recovers.
Co-Authored-By: Claude <noreply@anthropic.com>
Demanding a 10% gain to keep doubling settled [1,2]->1 on the first
noisy plateau (lbv2: 4k..2M-token docs, completion-rate noise dwarfs
10%). Now: keep climbing while not clearly worse (>=0.9x); bisect only
on clear degradation; samples per level doubled (max(3, 2*level)) to
shrink noise; steady re-probes +1 after ~60s so a noise-induced settle
cannot pin the gate forever. Overshoot past the true knee is trimmed
by the failure channel (timeouts -> x0.7), which is the real ceiling
finder on a prefill-bound endpoint.
Noise-swept at +-25%: capacities 4/8/16 settle at 11/31/16 without a
failure model; production failures pull the overshoot back down.
Co-Authored-By: Claude <noreply@anthropic.com>
95 waiters all tokenizing 2M-token docs through the 32-thread default
executor saturated the GIL: the rich render thread and the event loop
starved, so the bar froze and jumped (and the gate probe went blind).
Truncation now runs on a dedicated 8-thread pool; the remaining
workers queue and the loop/renderer stay responsive.
eta formats as 45s / 7m15s / 2h35m40s / 6d03h12m as it grows.
Co-Authored-By: Claude <noreply@anthropic.com>
The 5s probe fetched /metrics via asyncio.to_thread, which shares the
DEFAULT executor with second-long truncation tokenizations -- 96 of
those queue-jumped the probe and the state machine never ticked (gate
frozen at 1 while results flowed, ETA 6h). Probes now run on a
dedicated single-thread executor, and after 3 consecutive fetch
failures the gate stops asking for /metrics entirely (this endpoint
404s; pure demand mode from then on).
Verified under a choked default executor: gate ticks 1->2 on schedule
and metrics_dead engages after 3 failures.
Co-Authored-By: Claude <noreply@anthropic.com>
The slash form read as a fraction and kept inviting 'why is the
denominator growing' -- it is the pipeline depth (tokenizing +
gate-queued + generating), bounded by the global semaphore, while the
gate-admitted count is the real server load.
Co-Authored-By: Claude <noreply@anthropic.com>
96 worker threads racing transformers 5.x lazy imports on the FIRST
_get_tokenizer call raised ImportError and degraded that whole first
batch to the char approximation (the old single-threaded path never
raced). First load now holds a threading.Lock; the transformers
'>model_max_length' logging is silenced inside truncation (counting a
2M-token doc before trimming it is the point), and the per-sample
degradation print becomes a one-shot warning.
Co-Authored-By: Claude <noreply@anthropic.com>
The counter was taken just past the GLOBAL semaphore (lifted to 96 in
auto mode so the gate is the sole limiter) but BEFORE the pool gate --
so 94 gate-queued workers counted as in-flight. The gate now pushes
its actually-admitted count and the bar shows 'admitted/held'
(e.g. in-flight 2/96 = 2 really hitting the server, 94 queued on the
gate). Verified: admitted never exceeds the gate limit.
Co-Authored-By: Claude <noreply@anthropic.com>
Replaces the +1/5s linear ramp: start at 1, double while measured
completions/s keeps improving (>10% over the previous level); the
first plateau opens a bisect [last_good, bad] that narrows to the
knee, then holds steady. Failures still cut x0.7 instantly and
restart probing from the shrunken level; zero completions = hold.
Judging a level needs max(MIN_OK, level) completions -- a
2-completion rate estimate at level 8 is quantization noise (caught
by simulation converging to 1 on a capacity-8 endpoint).
Simulated against throughput curves min(level, capacity):
capacity 8 -> 1,2,4,8,16 | bisect 12,10,9 -> steady 8
capacity 16 -> 1,2,4,8,16,32 | bisect ... -> steady 16
capacity 4 -> 1,2,4,8 | bisect 6,5 -> steady 4
Co-Authored-By: Claude <noreply@anthropic.com>
Ramping on demand alone was dangerous with slow endpoints: if the 2
in-flight longbench_v2 requests hang, 'no failures + waiters queued'
kept adding +1 every 5s all the way to 96 -- piling prefills onto a
server that had not answered anything. Both ramp paths (demand-driven
and /metrics) now require at least one SUCCESSFUL completion in the
probe interval; hangs hold the gate until read-timeouts fire and the
x0.7 backoff takes over.
Unit-verified: hang 4 probe intervals with 20 waiters -> limit stays 2;
one success -> +1; one failure -> x0.7.
Co-Authored-By: Claude <noreply@anthropic.com>
assemble() runs the max_input_tokens truncation tokenizer inline in the
coroutine; longbench_v2's 2M-token docs take seconds of CPU per encode,
and each one BLOCKED the whole loop -- heartbeat frozen, gate probes
dead, zero HTTP while the process sat at 100% single-core. encode now
runs in a worker thread (loop stays live, encodes parallelize).
Co-Authored-By: Claude <noreply@anthropic.com>
--auto-concurrency removed as promised; 'auto' now starts the adaptive
gate at 2 (it ramps on its own demand/health signals). Plan displays
'auto (start 2, gate decides)'.
Co-Authored-By: Claude <noreply@anthropic.com>
The auto-stream trigger only looked at max_tokens (output budget);
longbench_v2 sends ~128k-token INPUTS with a small 8k output budget and
hit the gateway's whole-request buffering: 20-minute read timeouts
through all 6 retries. es survives the same bench because it streams
by default. Payloads over ~300k chars (~90k+ tokens) now take the
stream-aggregate path; verified live: a 350k-char prompt returns in
20s instead of hanging.
Co-Authored-By: Claude <noreply@anthropic.com>
Full traceback finally caught it: adapter._parse ALWAYS ran the
text-protocol tool-call fallback, even for requests with NO tools. On
humaneval, model code like regex-matched as a
'call', ast.literal_eval turned the literal into an Ellipsis
(no exception -- it's a legal literal), and json.dumps(args) died
mid-generation, killing the benchmark.
Two layers:
- the fallback now only runs when the request actually carried tools
(also stops polluting plain predictions with phantom calls, and the
SyntaxWarning spam from ast.parse-ing model code disappears)
- json.dumps(args, default=str) as belt-and-braces for the
text-tools path where an Ellipsis arg now stringifies
Reproduced the exact crash input as a unit case: no-tools code reply
yields 0 tool_calls; in text mode serializes
{'key': 'Ellipsis'} without raising; normal fc calls unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
Adding config/sample_counts.yaml made TWO yamls in config/, and the
auto-load rule was 'exactly one yaml' -- so every bench silently lost
its repeats/temperature/max_tokens (humaneval ran once instead of 3).
Manifest renamed to .json, and the rule hardened: a lone yaml still
wins, otherwise default.yaml wins explicitly. Verified: aime 12x +
humaneval 3x repeats active again.
Co-Authored-By: Claude <noreply@anthropic.com>
- sample-counts manifest (config/sample_counts.yaml, harvested from
real runs): uncached benches still show exact numbers in the plan
instead of 'counts when datasets load' -- 'cache+est.' marks the mix
- '--concurrency auto' is now an alias for --auto-concurrency
- Concurrency row shows 'auto (start 8, gate decides)' when the gate
drives, instead of a bare misleading 8
Also verified end-to-end: thinking-mode humaneval rep1/rep2 both
pass 98.8%, matching the es reference runs (98.17/98.78/98.78) on the
same model -- framework alignment holds on the thinking path too.
Co-Authored-By: Claude <noreply@anthropic.com>
syy's es runs: humaneval x3, gpqa x2, aime25/26 x12 (already aligned),
mmlu_pro/longbench_v2 x1 (temp=0 deterministic -- repeats are noise,
and 3x 12k samples is pure cost). temp=1 benches get 3.
Co-Authored-By: Claude <noreply@anthropic.com>
A finished 12-minute humaneval bench died at the save line: some scorer
detail carried a non-JSON object (Ellipsis) and report.save's json.dumps
had no default. Every dumps on the save path now stringifies exotic
objects instead of killing the run; verified by replaying the exact
crashed checkpoint end-to-end (pass 82.3%, 15s, no crash).
Co-Authored-By: Claude <noreply@anthropic.com>
User feedback: the out-dir report-reuse layer was one concept too many.
Now each checkpoint line carries {key, ts, pred, score}:
- --resume restores predictions AND their scores; when every sample's
cached score matches the scoring-setup fingerprint (recipe/extract/
scorers/judge), the report is replayed with NO scorer, extractor or
docker container touching anything
- fingerprint mismatch (recipe or judge changed) -> automatic re-eval
and backfill of the fresh scores
- no --resume -> nothing read, nothing written (full fresh run)
- --rescore = ignore cached scores, re-evaluate, refresh the cache
- aggregation always recomputed from cached per-sample scores (cheap,
survives aggregator changes without invalidating)
- legacy checkpoints without a score field backfill on first evaluation
Removed: the out-dir report-reuse block (superseded; also the source of
the UnboundLocalError path).
Co-Authored-By: Claude <noreply@anthropic.com>
The progress-reporter setup lived inside the generation branch; reuse
skipped it but the success path still advanced the overall bar ->
'cannot access local variable progress_reporter'. Setup now runs for
every path (reuse included).
Co-Authored-By: Claude <noreply@anthropic.com>
time_s/time_h summed per-prediction latency_s, which includes RESTORED
predictions' original generation time -- days old and from a slower
setup, it once reported 15.9h for a one-hour aime25 run. All rows now
report the bench's actual wall clock; token totals stay as the true
cost of the predictions used.
n for repeats>1 is num_samples x repeats (12 runs over 30 problems is
360 generations, not 30).
Co-Authored-By: Claude <noreply@anthropic.com>
The checkpoint stores predictions, not scores -- reruns re-executed the
whole docker scoring pass even when nothing could change. When the
saved report matches (same model, same sample count) and every
prediction is checkpointed, reuse it; --rescore forces evaluation
(recipe/judge changed).
Also fixes EvalReport.load: metric_groups declared Dict[str, float]
rejected the None perf values (and the repeats scores list) the file
format actually contains -- every load raised ValidationError, which
is what silently killed reuse in testing.
Repeats>1 benches never reuse: their whole point is fresh sampling.
Co-Authored-By: Claude <noreply@anthropic.com>
Previously both were written only after ALL benches finished -- a
multi-hour mmlu_pro left the summary stale for hours, and a crashed
run left the PREVIOUS run's files in place. Now each completed bench
(including failed ones) rewrites the summaries with everything so far.
Co-Authored-By: Claude <noreply@anthropic.com>
Each file carries that run's own score and samples; the top-level
report.jsonl stays the mean-summary view (mean + repeats group).
Co-Authored-By: Claude <noreply@anthropic.com>
12 repeats each scored on the terminal but the artifacts kept only the
final repeat's report; the run list scrolled away. Now the last report
carries metric_groups['repeats'] = {n_runs, scores[12], mean, min, max,
std} -- lands in report.jsonl's header line, the xlsx categories sheet,
and summary.csv's categories column (numeric entries); the console
summary line also prints the full runs=[...] list.
Co-Authored-By: Claude <noreply@anthropic.com>
--auto-concurrency wraps a single endpoint as a 1-member pool; the
spec parser rejected it with 'needs a {start..end} port range' and
killed every benchmark. Range-less comma segments are now plain
members; only a fully empty endpoint list errors.
Co-Authored-By: Claude <noreply@anthropic.com>
