The entrypoint override added only DockerSandbox the parameter; the
execution scorer now always passes it, so every local-sandbox bench
(live_code_bench) died with 'got an unexpected keyword argument' and
scored 0. Accepted (and ignored) in local/base for parity.
Co-Authored-By: Claude <noreply@anthropic.com>
Even a real 9GB/74s pull printed only 'Pulling from' + 'Status' -- the
CLI suppresses per-layer Downloading lines for non-TTY stdout since
docker 25, so the CR-splitting/summary pipeline had nothing to parse.
--progress=plain forces them out.
Co-Authored-By: Claude <noreply@anthropic.com>
- watchdog: 10s was the user's experiment; back to 5min default
(EVALHARNESS_PULL_IDLE_S overrides). Zero-byte detection itself is
sound: select(fd, timeout) + any-chunk reset, unit-verified.
- a FAILED bench left its 'scoring 0/2' bar on screen through the
NEXT bench's entire dataset download; begin_bench() relabels to
'<name> · loading' and zeroes counters at each bench start.
Co-Authored-By: Claude <noreply@anthropic.com>
The containment check ('{img}' in template) matched ALL entries, so
every announce said 'via docker.io(daemon mirrors)' while the chain
silently rotated real mirrors. The daemon-default entry is exactly
'{img}' -- compare equality.
Co-Authored-By: Claude <noreply@anthropic.com>
16 -> 6 (daemon default, daocloud, 1ms.run, 1panel, rat.dev, ustc):
the long tail never delivered anyway. All-6 failure now prints an
explicit self-load recipe (save/load via an egress machine, or pull
when the network recovers) instead of a terse RuntimeError.
Co-Authored-By: Claude <noreply@anthropic.com>
16 total. The intl entries need real egress (usually blocked on this
host) but cost only 10s each with the idle watchdog on networks that
have them. Also noted: a 10s watchdog can kill sources mid slow
negotiation (rat.dev emitted 'Pulling fs layer' then went quiet) --
EVALHARNESS_PULL_IDLE_S=30 softens that.
Co-Authored-By: Claude <noreply@anthropic.com>
Was 6 explicit sources (the daemon's first hop hides 12 more of its
own); added 1panel.live, dockerproxy, 163, baidubce, ustc, sjtug, nju,
tuna. With the 10s idle watchdog the full sweep costs ~2min worst
case. EVALHARNESS_DOCKER_MIRRORS replaces the whole list.
Co-Authored-By: Claude <noreply@anthropic.com>
Any docker output resets the timer, so this only fires on totally
silent (dead) mirrors; EVALHARNESS_PULL_IDLE_S overrides.
Co-Authored-By: Claude <noreply@anthropic.com>
A dead mirror emitted zero bytes and the fixed 3600s attempt timeout
let it stall the whole chain for an hour (third occurrence tonight:
15m49s frozen at 0 bytes). select() with a 300s idle timeout now kills
the attempt and moves to the next mirror; a trickling source (1 line/s
Downloading updates) resets the timer and stays alive.
Co-Authored-By: Claude <noreply@anthropic.com>
docker refreshes Downloading lines with \r, not \n -- readline()
parked them in the buffer, so the byte snapshots never surfaced and
the overall-progress dict stayed empty (the ⏳ summary never printed).
Chunked reads now split on both terminators; verified with a synthetic
\r-stream that the parser yields the refresh lines and computes the
52.2/65.1MB-style summary.
Co-Authored-By: Claude <noreply@anthropic.com>
Per-layer events gave no sense of how much is LEFT. Downloading lines
are parsed per-layer and summed; every 3s the snapshot becomes
'⏳ 3.14/9.27 GB (34%) · 最大层 3f4a2b1c9d0e: 0.51/1.2 GB' instead of
a bare layer line. Key events (Pulling from / Status / errors) still
print immediately.
Co-ached-By: Claude <noreply@anthropic.com>
With --entrypoint python3, our 'python /work/main.py' runner became
interpreter args: python3 tried to open a FILE named 'python'
("can't open file '/app/python'"). When an entrypoint is set it IS
the interpreter -- pass just /work/<entry>.
Co-Authored-By: Claude <noreply@anthropic.com>
Cold pulls now tick a 'Downloading xMB/yGB' snapshot at most every 3s
(was 10s) -- keyword hits (Pull complete / Status / errors) still
print immediately.
Co-Authored-By: Claude <noreply@anthropic.com>
The official image's ENTRYPOINT is 'python3 -m bigcodebench.evaluate'
-- our 'python /work/main.py' runner was swallowed as CLI args and the
official evaluator died on 'No samples provided'. execution scorer now
accepts entrypoint= and docker exec passes --entrypoint; the BCB recipe
sets entrypoint python3. Also swept 7 leaked eh-exec containers (the
--rm path never fires when our timeout kills the CLI first).
Co-Authored-By: Claude <noreply@anthropic.com>
Two pull UX/correctness fixes: (1) a multi-GB pull with captured
output is minutes of silence reading as a hang -- layer progress now
streams (throttled) to stderr; (2) the mirror-tag -> canonical retag
could fail silently and the following rmi then deleted the ONLY tag,
losing a 342s pull and forcing a full re-download -- retag is now
verified and the mirror tag kept on failure.
Co-Authored-By: Claude <noreply@anthropic.com>
A multi-GB docker pull ran with fully captured output -- minutes of
silence that read as a hang (user interrupted a healthy run over it).
Each mirror attempt now prints which source it is trying and how long
a hit took.
Co-Authored-By: Claude <noreply@anthropic.com>
The preflight did a bare 'docker pull' -- docker.io is slow/unreachable
from CN without luck; it now walks the same fallback chain as the SWE
prefetch (daemon mirrors -> daocloud -> 1ms.run -> baidubce -> sjtu ->
rat.dev), retagging the hit to the canonical name.
Co-Authored-By: Claude <noreply@anthropic.com>
1140 real predictions scored 0.0% because the recipe referenced
'bigcodebench-sandbox:latest' -- a name nothing builds and docker.io
does not have; every sample then burned 3 pull-retries (~200s each).
- image -> bigcodebench/bigcodebench-evaluate:latest (the official hub
image, same one evalscope uses)
- ensure_image() preflight in evaluate(): recipe-level AND sample-level
images are verified/pulled ONCE before any container runs; missing ->
seconds-fast bench failure with a fix hint instead of a silent 0.0%
- docker exec: 'Unable to find image'/'manifest unknown' class errors
are permanent -- no 3x retry amplification
Verified with a bogus image: preflight raises in one pull-attempt with
the fix hint.
Co-Authored-By: Claude <noreply@anthropic.com>
An unbounded docker rm against a bloated daemon hangs for minutes and
silently eats the worker pool: 7 of 8 scoring workers were observed
stuck in cleanup while only 1 execution ran.
Co-Authored-By: Claude <noreply@anthropic.com>
- docker exec: named containers; a timed-out/killed 'docker run' only
kills the CLI client while the container lives on (--rm fires on
EXIT) -- rm -f the name on timeout/interrupt so runs stop leaking
- exit 125 = daemon-side failure, not model failure: retry up to 2x
(a bloated daemon was turning healthy samples into pass=0)
- scoring milestones: first completion logs immediately, then every
~5% (10% was too sparse when docker is slow: minutes of silence
right after the 'scoring' phase starts, looks hung)
Co-Authored-By: Claude <noreply@anthropic.com>
