- AdaptiveGate rewritten (Netflix Gradient2): window-vs-window per-stream
speed gradient, count-driven windows with admission stamps, no thresholds
or mode state machine; failures x0.7 + 30s drain pause
- session-level admission for multi-turn agents (_SessionGate): in-progress
sessions hold slots until done, newcomers queue at the door; capacity
follows the model gate's discovered limit (CONCUR-style continuity)
- image service: memory-first register (zero docker calls for known
images), TTL-cached docker images listing, optimistic ready when the
daemon is unreachable (docker save contention no longer kills runs);
es tar loading removed in favor of ModelScope shipping (ms_images.py
per-image tar upload/pull with round-trip verification)
- runner: circuit breaker (12 consecutive failures abort the bench),
first-failure error printed immediately
- swe_agentic: image wait / docker run / rm off the event loop; exec
timeout becomes an observation the agent can react to; container gets
curlrc + git low-speed aborts (stalled github downloads fail fast)
- eval run excludes its own endpoints from http_proxy (a sick personal
proxy read as 'endpoint dead' and killed whole runs)
- progress bar shows failed count; swe agentic exec_workers 2 -> 4
Co-Authored-By: Claude <noreply@anthropic.com>
MAP_REPO_VERSION_TO_SPECS[repo][version] KeyError'd on empty version --
env_state didn't store the version field, the scorer's fallback to
sample.metadata hit the positional misalign again. The four official
metadata fields now stored alongside the rest.
Co-Authored-By: Claude <noreply@anthropic.com>
pypi.org is unreachable from this host's containers; agent pip installs
hit 600s _exec timeouts and burned 4/10 samples. The env var makes pip
use tuna (~7s for the same package, verified in-container).
Co-Authored-By: Claude <noreply@anthropic.com>
The model DISCUSSES the sentinel command in prose; splitting on the
first occurrence captured conversational English as the 'patch'
(145B of chatter instead of the diff). On sentinel detection we now
read /testbed/patch.txt directly from the container -- the file the
agent actually created. Empty/missing file falls through to git diff.
Co-Authored-By: Claude <noreply@anthropic.com>
One ImageService per process. When tasks arrive (runner) or a sample
starts (env), its images are REGISTERED; a background worker pool
delivers them -- local tar shipments first (es's swebench_v 500-image
batch set, disk-cached index), network mirror chain second. Sample
execution waits on a readiness barrier instead of the old failing
timings (docker-run implicit pull killed at 120s; score-phase batch
pull ran after generation had already failed).
- runner registers every pending sample's image up front (pull-ahead
overlaps generation)
- env blocks on wait_ready(1800s) before docker run -- a slow pull
delays that sample, never fails it
- tar index cached at /tmp/evalharness_tar_index.json (full scan costs
minutes; only the first process pays)
- images the service loaded are released at exit (atexit; opt out with
EVALHARNESS_KEEP_SWE_IMAGES); pre-existing local images never touched
- EVALHARNESS_IMAGE_WORKERS (default 2) tunes the pool
Verified E2E: register -> background load from swebench_batch_001.tar.gz
-> image present locally (matplotlib-14623); wait barrier semantics
confirmed (blocks until load completes).
Co-Authored-By: Claude <noreply@anthropic.com>
Per-instance images are 1-4GB each; a --limit 10 run pulls up to 10 of
them and previously stranded the whole footprint on Ctrl+C/exit. Every
image THIS process pulls is now registered and released by an atexit
hook (containers first, then rmi) -- images that already existed
locally are never touched. EVALHARNESS_KEEP_SWE_IMAGES=1 opts out for
prefetch-style runs that want to keep them.
Co-Authored-By: Claude <noreply@anthropic.com>
docker run's implicit pull prints nothing and the 120s start timeout
kills runs on slow mirrors; ensure_image brings the progress display,
mirror chain and watchdog to per-instance pulls (each sample = its own
image, 10 samples = 10 pulls).
Co-Authored-By: Claude <noreply@anthropic.com>
The captured diff was stripped, and git apply rejects diffs whose last
line lacks a trailing newline ('corrupt patch at line N') -- the first
agentic score was resolved=0 with patch_apply_failed despite a correct
fix. Verified: same payload + newline applies clean in the official
container.
Co-Authored-By: Claude <noreply@anthropic.com>
Without it get_env('swe_agentic') raised TypeError at env creation --
and the swallowed exception silently degraded the run to SINGLE-TURN
prose generation (421s of model monologue, no container, empty
trajectory). This was the agentic 'run' that looked like it worked.
Co-Authored-By: Claude <noreply@anthropic.com>
