Single-image benches keep fail-fast; >8 distinct images (swe's 500
per-instance) now pull concurrently with resume state inside the run
itself -- exactly like the old auto-pull behavior, just resilient:
scoring proceeds with whatever images landed, missing ones score 0
and only a total wipeout fails the bench. Ctrl+C-safe (state file),
network recovery resumes automatically on the next run.
Co-Authored-By: Claude <noreply@anthropic.com>
- naming corrected to the Docker Hub truth es verified 500/500:
swebench/sweb.eval.x86_64.{instance_id.lower(), __->_1776_}:latest
(my repo-base rewrite was wrong; per-instance images ARE published)
- 'evalharness sandbox pull swe_bench_verified': concurrent pulls,
resume state in <cache-dir>/swe_pull_state.json, --dry-run N,
--retry-failed; self-contained (own dataset registry, no evalscope
import)
- save/load intentionally omitted per user call (pull-only for now)
Co-Authored-By: Claude <noreply@anthropic.com>
tau2's current reward_info carries 'reward' (composite) + db_check /
action_checks; the scorer read the old environment_reward /
communication_reward split that no longer exists -- simulations scored
1.0 came out 0.0. Fallback to the old split kept for older engines.
Co-Authored-By: Claude <noreply@anthropic.com>
Function-level silencers kept losing the race to whichever 'import
tau2.*' fired first in the process. The filter + TAU2_DATA_DIR now
install when the env registry loads the plugin, strictly before any
tau2 import can happen anywhere.
Co-Authored-By: Claude <noreply@anthropic.com>
The startup noise (.env found note, data directory note) is itself
WARNING-level, so the WARNING filter passed it through. Level is now
ERROR and the data-dir warnings die at the source with the correct
TAU2_DATA_DIR.
Co-Authored-By: Claude <noreply@anthropic.com>
YAML default section and per-bench (tau2_bench: 200) both set it; the
runner's generic 8-starved every agent dialog (tau2 reward-0 root
cause). CLI wiring pops the key into args.max_turns before the
generation keys are stripped.
Co-Authored-By: Claude <noreply@anthropic.com>
Both reward-0 runs died with 'Termination reason: max_steps' after 3-4
exchanges -- the runner passes its generic default (8) down, far too
few for greet->verify->find->policy->act->confirm. Official tau2 runs
use 40+; floor, don't cap.
Co-Authored-By: Claude <noreply@anthropic.com>
loguru restricted to WARNING+ (every orchestrator step dumped full
messages -- thousands of lines per bench). AssistantMessage.is_final_
chunk now False on tool-call turns (official adapter semantics: True
means the agent is FINISHED talking); always-True handed the turn
back to the user prematurely mid-action-sequence.
Co-Authored-By: Claude <noreply@anthropic.com>
The user simulator (GLM via our adapter) inlines its scenario reasoning
in content as '...instructions...</think>reply' -- passed through
unstripped, the AGENT receives the scenario's secret instructions
(task goal, disclosure strategy), inflating rewards. Both channels now
trimmed at the last </think>.
Co-Authored-By: Claude <noreply@anthropic.com>
Same UX as the docker image chain: try local sources first, then the
GitHub URL, and on total failure print the exact manual commands --
with a warning that PyPI's 'tau2' is an unrelated physics package.
Co-Authored-By: Claude <noreply@anthropic.com>
mrcr's summary row displayed 'extraction_failure_rate 0.0%' as its
score because dict insertion order put the diagnostic first. Both
primary-metric picks (row build and repeats mean) now skip it.
Co-Authored-By: Claude <noreply@anthropic.com>
- config env entries: bfcl_v3->bfcl_mock, tau2_bench->tau2_official;
the eval loop previously POPPED the env key and discarded it -- now
it feeds args.env (CLI --env still works as default/fallback)
- swe_bench_verified images: per-instance names are never published;
the official docker.io layout is repo-level BASE images
(swebench/sweb.eval.x86_64.{repo}) with per-instance images built
on top -- naming corrected (note: the swebench/* namespace is
currently blocked on every reachable CN mirror all the same)
Co-Authored-By: Claude <noreply@anthropic.com>
'/data/hf_models/GLM-5.3-NVFP4' contains slashes, so the bare-name
heuristic ('/' not in judge) misclassified it as a full spec and passed
it through un-prefixed -- resolve_adapter then blew up. With
--judge-api-url given, combination is now unconditional (mirrors the
main model flags); bare names without a url fail fast; full specs
without a url pass through.
Co-Authored-By: Claude <noreply@anthropic.com>
A bare --judge-model without --judge-api-url produced a malformed spec
that exploded deep inside run_eval -- and it did so for longbench_v2,
which does not even use a judge. Now the CLI rejects the combination
up front, and run_eval constructs the judge adapter only when the
recipe's scorers actually include llm_judge.
Co-Authored-By: Claude <noreply@anthropic.com>
The mirror's CDN assigns 6KB/s or 4.8MB/s to the SAME file depending
on which edge a connection lands on -- a single-connection download
is one dice roll that can stall for the whole file. aria2c (-x8 -s8)
splits the file so each segment rolls independently, and
--lowest-speed-limit=50K re-opens stalled segments. Falls back to the
urllib path when aria2c is absent. Live test: the mrcr file that sat
at 2.1MB/190MB for 6 minutes came down in 43s (4.4MB/s).
Co-Authored-By: Claude <noreply@anthropic.com>
Hybrid-thinking gateways inline the reasoning channel into content
wrapped in <think>..</think>; extractors then produced answers like
'3</think>Let me analyze...' (drop em 0.0 with f1 0.5 on samples the
model actually answered). Blocks and stray closers removed pre-extract.
Co-Authored-By: Claude <noreply@anthropic.com>
The entrypoint override added only DockerSandbox the parameter; the
execution scorer now always passes it, so every local-sandbox bench
(live_code_bench) died with 'got an unexpected keyword argument' and
scored 0. Accepted (and ignored) in local/base for parity.
Co-Authored-By: Claude <noreply@anthropic.com>
The tree API intermittently returns truncated listings (9 parquets
listed as [.gitattributes, README.md] at night); previously that failed
the bench even though 5 of the files were already in the shared blob
store. Empty selection now uses matching cached blobs with a notice;
only a truly cold cache still raises.
Co-Authored-By: Claude <noreply@anthropic.com>
Even a real 9GB/74s pull printed only 'Pulling from' + 'Status' -- the
CLI suppresses per-layer Downloading lines for non-TTY stdout since
docker 25, so the CR-splitting/summary pipeline had nothing to parse.
--progress=plain forces them out.
Co-Authored-By: Claude <noreply@anthropic.com>
- watchdog: 10s was the user's experiment; back to 5min default
(EVALHARNESS_PULL_IDLE_S overrides). Zero-byte detection itself is
sound: select(fd, timeout) + any-chunk reset, unit-verified.
- a FAILED bench left its 'scoring 0/2' bar on screen through the
NEXT bench's entire dataset download; begin_bench() relabels to
'<name> · loading' and zeroes counters at each bench start.
Co-Authored-By: Claude <noreply@anthropic.com>
The containment check ('{img}' in template) matched ALL entries, so
every announce said 'via docker.io(daemon mirrors)' while the chain
silently rotated real mirrors. The daemon-default entry is exactly
'{img}' -- compare equality.
Co-Authored-By: Claude <noreply@anthropic.com>
16 -> 6 (daemon default, daocloud, 1ms.run, 1panel, rat.dev, ustc):
the long tail never delivered anyway. All-6 failure now prints an
explicit self-load recipe (save/load via an egress machine, or pull
when the network recovers) instead of a terse RuntimeError.
Co-Authored-By: Claude <noreply@anthropic.com>
16 total. The intl entries need real egress (usually blocked on this
host) but cost only 10s each with the idle watchdog on networks that
have them. Also noted: a 10s watchdog can kill sources mid slow
negotiation (rat.dev emitted 'Pulling fs layer' then went quiet) --
EVALHARNESS_PULL_IDLE_S=30 softens that.
Co-Authored-By: Claude <noreply@anthropic.com>
Was 6 explicit sources (the daemon's first hop hides 12 more of its
own); added 1panel.live, dockerproxy, 163, baidubce, ustc, sjtug, nju,
tuna. With the 10s idle watchdog the full sweep costs ~2min worst
case. EVALHARNESS_DOCKER_MIRRORS replaces the whole list.
Co-Authored-By: Claude <noreply@anthropic.com>
Any docker output resets the timer, so this only fires on totally
silent (dead) mirrors; EVALHARNESS_PULL_IDLE_S overrides.
Co-Authored-By: Claude <noreply@anthropic.com>
A dead mirror emitted zero bytes and the fixed 3600s attempt timeout
let it stall the whole chain for an hour (third occurrence tonight:
15m49s frozen at 0 bytes). select() with a 300s idle timeout now kills
the attempt and moves to the next mirror; a trickling source (1 line/s
Downloading updates) resets the timer and stays alive.
Co-Authored-By: Claude <noreply@anthropic.com>
docker refreshes Downloading lines with \r, not \n -- readline()
parked them in the buffer, so the byte snapshots never surfaced and
the overall-progress dict stayed empty (the ⏳ summary never printed).
Chunked reads now split on both terminators; verified with a synthetic
\r-stream that the parser yields the refresh lines and computes the
52.2/65.1MB-style summary.
Co-Authored-By: Claude <noreply@anthropic.com>
Per-layer events gave no sense of how much is LEFT. Downloading lines
are parsed per-layer and summed; every 3s the snapshot becomes
'⏳ 3.14/9.27 GB (34%) · 最大层 3f4a2b1c9d0e: 0.51/1.2 GB' instead of
a bare layer line. Key events (Pulling from / Status / errors) still
print immediately.
Co-ached-By: Claude <noreply@anthropic.com>
With --entrypoint python3, our 'python /work/main.py' runner became
interpreter args: python3 tried to open a FILE named 'python'
("can't open file '/app/python'"). When an entrypoint is set it IS
the interpreter -- pass just /work/<entry>.
Co-Authored-By: Claude <noreply@anthropic.com>
Cold pulls now tick a 'Downloading xMB/yGB' snapshot at most every 3s
(was 10s) -- keyword hits (Pull complete / Status / errors) still
print immediately.
Co-Authored-By: Claude <noreply@anthropic.com>
The official image's ENTRYPOINT is 'python3 -m bigcodebench.evaluate'
-- our 'python /work/main.py' runner was swallowed as CLI args and the
official evaluator died on 'No samples provided'. execution scorer now
accepts entrypoint= and docker exec passes --entrypoint; the BCB recipe
sets entrypoint python3. Also swept 7 leaked eh-exec containers (the
--rm path never fires when our timeout kills the CLI first).
Co-Authored-By: Claude <noreply@anthropic.com>
The scoring phase only reset the bar on the first judged sample; until
then the stale GENERATION state (100%, +N new) stayed on screen --
through multi-minute image-pull preflights it read as 'already done'
while nothing was scored yet.
Co-Authored-By: Claude <noreply@anthropic.com>
Two pull UX/correctness fixes: (1) a multi-GB pull with captured
output is minutes of silence reading as a hang -- layer progress now
streams (throttled) to stderr; (2) the mirror-tag -> canonical retag
could fail silently and the following rmi then deleted the ONLY tag,
losing a 342s pull and forcing a full re-download -- retag is now
verified and the mirror tag kept on failure.
Co-Authored-By: Claude <noreply@anthropic.com>
A multi-GB docker pull ran with fully captured output -- minutes of
silence that read as a hang (user interrupted a healthy run over it).
Each mirror attempt now prints which source it is trying and how long
a hit took.
Co-Authored-By: Claude <noreply@anthropic.com>
One flag for the whole ladder: off == --disable-thinking,
low..max map to reasoning_effort, full = plain default. Takes
precedence over the two older flags. Run Plan shows the active mode
('disabled' / 'enabled · effort=low' / 'enabled').
Co-Authored-By: Claude <noreply@anthropic.com>
The adapter dropped reasoning_effort/thinking from the payload, so the
middle rung of the ladder (es reference: full 98.3 / effort_low 94 /
no-think 82.3 on humaneval) was unreachable. Both keys now pass
through; --reasoning-effort {minimal,low,medium,high,max} overrides
the YAML, and config/effort_low.yaml mirrors default.yaml with
reasoning_effort: low for one-command low-thinking runs.
Probe on the endpoint: same question, default = 319 chars reasoning /
262 tok, low = 47 chars / 117 tok -- the server honors it.
Co-Authored-By: Claude <noreply@anthropic.com>
The preflight did a bare 'docker pull' -- docker.io is slow/unreachable
from CN without luck; it now walks the same fallback chain as the SWE
prefetch (daemon mirrors -> daocloud -> 1ms.run -> baidubce -> sjtu ->
rat.dev), retagging the hit to the canonical name.
Co-Authored-By: Claude <noreply@anthropic.com>
1140 real predictions scored 0.0% because the recipe referenced
'bigcodebench-sandbox:latest' -- a name nothing builds and docker.io
does not have; every sample then burned 3 pull-retries (~200s each).
- image -> bigcodebench/bigcodebench-evaluate:latest (the official hub
image, same one evalscope uses)
- ensure_image() preflight in evaluate(): recipe-level AND sample-level
images are verified/pulled ONCE before any container runs; missing ->
seconds-fast bench failure with a fix hint instead of a silent 0.0%
- docker exec: 'Unable to find image'/'manifest unknown' class errors
are permanent -- no 3x retry amplification
Verified with a bogus image: preflight raises in one pull-attempt with
the fix hint.
Co-Authored-By: Claude <noreply@anthropic.com>
Five categories (Code & Engineering / Reasoning & Math / Knowledge &
Language / Long Context / Agents & Tools) now annotate the console
summary table (new column), summary.csv (new field), and each report's
run_info.category.
Co-Authored-By: Claude <noreply@anthropic.com>
Tokens are cumulative (they include restored predictions' usage) while
time was this-run wall -- fully replayed benches showed 0s in the same
column, reading as broken. run_info now carries gen_fresh; a bench with
zero fresh generations renders its time cell as 'cached' (tokens keep
showing the true cumulative investment).
Co-Authored-By: Claude <noreply@anthropic.com>
115m39s next to eta 2h51m01s read inconsistently; all durations now
render as 45s / 7m15s / 1h55m39s / 2h51m01s / 3d02h00m.
Co-Authored-By: Claude <noreply@anthropic.com>
One sample whose stream never got a first byte (prefill queue at high
gate levels) exhausted 6 adapter retries and killed the WHOLE gather --
500 samples died with it. Terminal failures are now contained: empty
prediction (scores wrong, es-parity for timeouts), NOT checkpointed so
a rerun retries them, prominently counted; only a 100% wipeout fails
the bench. Stream-aggregate read timeout 60s -> 240s: a 128k prompt
queued behind other prefills legitimately takes minutes to start
answering; real hangs are now the gate's job (x0.7) and contained
failures rather than bench death.
Verified: 1-in-3 terminal failures -> bench completes 6 ok / 3 empty,
checkpoint holds only the 6.
Co-Authored-By: Claude <noreply@anthropic.com>
