The scorer returned {'acc': ...} while the recipe registers the metric
as 'resolved' -- the aggregator looked for 'resolved' in scores, found
nothing, and reported 0.0 despite per-sample resolved=1.0 in the
checkpoint. Key names now match the recipe registration.
Co-Authored-By: Claude <noreply@anthropic.com>
get_eval_report returns {instance_id: {resolved: bool, ...}}, not a flat
dict. report.get('resolved') was always None -> every sample scored 0
despite the grader judging resolved=true (confirmed by direct grader
call on the real test output: astropy-13453 resolved=true).
Co-Authored-By: Claude <noreply@anthropic.com>
sample.metadata vs env_state mismatch confirmed: sample[0] said
astropy-12907 while the paired prediction held 13453's test_patch --
make_test_spec built the eval_script for the WRONG bug, scoring real
patches against wrong tests (0/10 despite manual verification that
the patches are correct). The env_state IS the actual instance the
agent ran; it takes priority, sample.metadata fills gaps.
Co-Authored-By: Claude <noreply@anthropic.com>
The official eval_script runs pip install -e '.[test]' -- same pypi.org
unreachability as agent containers would burn the 1800s timeout on
network retries. Same tuna mirror injected.
Co-Authored-By: Claude <noreply@anthropic.com>
Both swe variants now score through make_test_spec -> container ->
apply-patch -> official eval_script -> get_eval_report -- the exact
pipeline es's eval_instance drives (swebench==4.1.0 installed). Our
in-container protocol stays as fallback when the package is absent.
env_reward gains a recipe-level 'backend' delegation hook (tau2's
reward_info reader and the swe official scorer both ride it); the
generic bfcl call-sequence comparison remains the default.
Co-Authored-By: Claude <noreply@anthropic.com>
Single-image benches keep fail-fast; >8 distinct images (swe's 500
per-instance) now pull concurrently with resume state inside the run
itself -- exactly like the old auto-pull behavior, just resilient:
scoring proceeds with whatever images landed, missing ones score 0
and only a total wipeout fails the bench. Ctrl+C-safe (state file),
network recovery resumes automatically on the next run.
Co-Authored-By: Claude <noreply@anthropic.com>
tau2's current reward_info carries 'reward' (composite) + db_check /
action_checks; the scorer read the old environment_reward /
communication_reward split that no longer exists -- simulations scored
1.0 came out 0.0. Fallback to the old split kept for older engines.
Co-Authored-By: Claude <noreply@anthropic.com>
Hybrid-thinking gateways inline the reasoning channel into content
wrapped in <think>..</think>; extractors then produced answers like
'3</think>Let me analyze...' (drop em 0.0 with f1 0.5 on samples the
model actually answered). Blocks and stray closers removed pre-extract.
Co-Authored-By: Claude <noreply@anthropic.com>
The official image's ENTRYPOINT is 'python3 -m bigcodebench.evaluate'
-- our 'python /work/main.py' runner was swallowed as CLI args and the
official evaluator died on 'No samples provided'. execution scorer now
accepts entrypoint= and docker exec passes --entrypoint; the BCB recipe
sets entrypoint python3. Also swept 7 leaked eh-exec containers (the
--rm path never fires when our timeout kills the CLI first).
Co-Authored-By: Claude <noreply@anthropic.com>
1140 real predictions scored 0.0% because the recipe referenced
'bigcodebench-sandbox:latest' -- a name nothing builds and docker.io
does not have; every sample then burned 3 pull-retries (~200s each).
- image -> bigcodebench/bigcodebench-evaluate:latest (the official hub
image, same one evalscope uses)
- ensure_image() preflight in evaluate(): recipe-level AND sample-level
images are verified/pulled ONCE before any container runs; missing ->
seconds-fast bench failure with a fix hint instead of a silent 0.0%
- docker exec: 'Unable to find image'/'manifest unknown' class errors
are permanent -- no 3x retry amplification
Verified with a bogus image: preflight raises in one pull-attempt with
the fix hint.
Co-Authored-By: Claude <noreply@anthropic.com>
A finished 12-minute humaneval bench died at the save line: some scorer
detail carried a non-JSON object (Ellipsis) and report.save's json.dumps
had no default. Every dumps on the save path now stringifies exotic
objects instead of killing the run; verified by replaying the exact
crashed checkpoint end-to-end (pass 82.3%, 15s, no crash).
Co-Authored-By: Claude <noreply@anthropic.com>
User feedback: the out-dir report-reuse layer was one concept too many.
Now each checkpoint line carries {key, ts, pred, score}:
- --resume restores predictions AND their scores; when every sample's
cached score matches the scoring-setup fingerprint (recipe/extract/
scorers/judge), the report is replayed with NO scorer, extractor or
docker container touching anything
- fingerprint mismatch (recipe or judge changed) -> automatic re-eval
and backfill of the fresh scores
- no --resume -> nothing read, nothing written (full fresh run)
- --rescore = ignore cached scores, re-evaluate, refresh the cache
- aggregation always recomputed from cached per-sample scores (cheap,
survives aggregator changes without invalidating)
- legacy checkpoints without a score field backfill on first evaluation
Removed: the out-dir report-reuse block (superseded; also the source of
the UnboundLocalError path).
Co-Authored-By: Claude <noreply@anthropic.com>
The checkpoint stores predictions, not scores -- reruns re-executed the
whole docker scoring pass even when nothing could change. When the
saved report matches (same model, same sample count) and every
prediction is checkpointed, reuse it; --rescore forces evaluation
(recipe/judge changed).
Also fixes EvalReport.load: metric_groups declared Dict[str, float]
rejected the None perf values (and the repeats scores list) the file
format actually contains -- every load raised ValidationError, which
is what silently killed reuse in testing.
Repeats>1 benches never reuse: their whole point is fresh sampling.
Co-Authored-By: Claude <noreply@anthropic.com>
Scoring progress: docker-exec benches (humaneval etc.) score for
minutes with zero feedback -- the bar sat at 'generating 100%' and
looked hung. evaluate() now takes on_scored(i, n) (atomic counter,
fires from worker threads), run_eval passes it through, and the CLI
shows 'scoring 42/164' on the bar + milestone log lines every 10%
(also fixes the phase match: 'scoring' never matched the capitalized
'Scoring predictions...' status message, so the bar never even
switched its label).
PooledAdapter:
- one release per acquire: the exception path released True (inner
finally) AND False (except handler), double-decrementing _inflight
(over-admission) and applying the x0.7 backoff twice
- AdaptiveGate: rebuild the Condition + probe task when the event loop
changes -- pools are cached across benchmarks and the CLI runs
asyncio.run() per bench/repeat; a loop-bound Condition from a closed
loop raises 'bound to a different event loop' under contention
Co-Authored-By: Claude <noreply@anthropic.com>
