With --entrypoint python3, our 'python /work/main.py' runner became
interpreter args: python3 tried to open a FILE named 'python'
("can't open file '/app/python'"). When an entrypoint is set it IS
the interpreter -- pass just /work/<entry>.
Co-Authored-By: Claude <noreply@anthropic.com>
The official image's ENTRYPOINT is 'python3 -m bigcodebench.evaluate'
-- our 'python /work/main.py' runner was swallowed as CLI args and the
official evaluator died on 'No samples provided'. execution scorer now
accepts entrypoint= and docker exec passes --entrypoint; the BCB recipe
sets entrypoint python3. Also swept 7 leaked eh-exec containers (the
--rm path never fires when our timeout kills the CLI first).
Co-Authored-By: Claude <noreply@anthropic.com>
The preflight did a bare 'docker pull' -- docker.io is slow/unreachable
from CN without luck; it now walks the same fallback chain as the SWE
prefetch (daemon mirrors -> daocloud -> 1ms.run -> baidubce -> sjtu ->
rat.dev), retagging the hit to the canonical name.
Co-Authored-By: Claude <noreply@anthropic.com>
1140 real predictions scored 0.0% because the recipe referenced
'bigcodebench-sandbox:latest' -- a name nothing builds and docker.io
does not have; every sample then burned 3 pull-retries (~200s each).
- image -> bigcodebench/bigcodebench-evaluate:latest (the official hub
image, same one evalscope uses)
- ensure_image() preflight in evaluate(): recipe-level AND sample-level
images are verified/pulled ONCE before any container runs; missing ->
seconds-fast bench failure with a fix hint instead of a silent 0.0%
- docker exec: 'Unable to find image'/'manifest unknown' class errors
are permanent -- no 3x retry amplification
Verified with a bogus image: preflight raises in one pull-attempt with
the fix hint.
Co-Authored-By: Claude <noreply@anthropic.com>
An unbounded docker rm against a bloated daemon hangs for minutes and
silently eats the worker pool: 7 of 8 scoring workers were observed
stuck in cleanup while only 1 execution ran.
Co-Authored-By: Claude <noreply@anthropic.com>
- docker exec: named containers; a timed-out/killed 'docker run' only
kills the CLI client while the container lives on (--rm fires on
EXIT) -- rm -f the name on timeout/interrupt so runs stop leaking
- exit 125 = daemon-side failure, not model failure: retry up to 2x
(a bloated daemon was turning healthy samples into pass=0)
- scoring milestones: first completion logs immediately, then every
~5% (10% was too sparse when docker is slow: minutes of silence
right after the 'scoring' phase starts, looks hung)
Co-Authored-By: Claude <noreply@anthropic.com>
