14 Commits

Author SHA1 Message Date
sora
869d2c41fb Mirror chain trimmed to 6 mainstream sources; manual-load guidance
16 -> 6 (daemon default, daocloud, 1ms.run, 1panel, rat.dev, ustc):
the long tail never delivered anyway. All-6 failure now prints an
explicit self-load recipe (save/load via an egress machine, or pull
when the network recovers) instead of a terse RuntimeError.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 05:53:09 +00:00
sora
adb46d4927 Mirror chain: +xuanyuan/dockerhub.icu (CN) + mirror.gcr.io/public.ecr (intl)
16 total. The intl entries need real egress (usually blocked on this
host) but cost only 10s each with the idle watchdog on networks that
have them. Also noted: a 10s watchdog can kill sources mid slow
negotiation (rat.dev emitted 'Pulling fs layer' then went quiet) --
EVALHARNESS_PULL_IDLE_S=30 softens that.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 05:51:16 +00:00
sora
731ab6e6c0 Pull chain: 12 CN mirrors + EVALHARNESS_DOCKER_MIRRORS override
Was 6 explicit sources (the daemon's first hop hides 12 more of its
own); added 1panel.live, dockerproxy, 163, baidubce, ustc, sjtug, nju,
tuna. With the 10s idle watchdog the full sweep costs ~2min worst
case. EVALHARNESS_DOCKER_MIRRORS replaces the whole list.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 05:50:21 +00:00
sora
dacffb02f2 Pull idle watchdog: 10s default (was 5min), env-tunable
Any docker output resets the timer, so this only fires on totally
silent (dead) mirrors; EVALHARNESS_PULL_IDLE_S overrides.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 05:49:08 +00:00
sora
d58fdb4198 Pull chain: 5-minute idle watchdog per mirror
A dead mirror emitted zero bytes and the fixed 3600s attempt timeout
let it stall the whole chain for an hour (third occurrence tonight:
15m49s frozen at 0 bytes). select() with a 300s idle timeout now kills
the attempt and moves to the next mirror; a trickling source (1 line/s
Downloading updates) resets the timer and stays alive.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 05:39:58 +00:00
sora
3ae1578325 Pull progress: split the stream on CR too
docker refreshes Downloading lines with \r, not \n -- readline()
parked them in the buffer, so the byte snapshots never surfaced and
the overall-progress dict stayed empty (the  summary never printed).
Chunked reads now split on both terminators; verified with a synthetic
\r-stream that the parser yields the refresh lines and computes the
52.2/65.1MB-style summary.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 03:51:02 +00:00
sora
e41cd48272 Pull progress: overall byte summary per tick
Per-layer events gave no sense of how much is LEFT. Downloading lines
are parsed per-layer and summed; every 3s the snapshot becomes
' 3.14/9.27 GB (34%) · 最大层 3f4a2b1c9d0e: 0.51/1.2 GB' instead of
a bare layer line. Key events (Pulling from / Status / errors) still
print immediately.

Co-ached-By: Claude <noreply@anthropic.com>
2026-09-17 03:42:44 +00:00
sora
774d2b9910 Pull progress: 3s snapshot throttle
Cold pulls now tick a 'Downloading xMB/yGB' snapshot at most every 3s
(was 10s) -- keyword hits (Pull complete / Status / errors) still
print immediately.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 03:10:16 +00:00
sora
4f6e64e65c Stream docker pull progress; never rmi after a failed retag
Two pull UX/correctness fixes: (1) a multi-GB pull with captured
output is minutes of silence reading as a hang -- layer progress now
streams (throttled) to stderr; (2) the mirror-tag -> canonical retag
could fail silently and the following rmi then deleted the ONLY tag,
losing a 342s pull and forcing a full re-download -- retag is now
verified and the mirror tag kept on failure.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 02:49:57 +00:00
sora
f4f10649df Announce sandbox image pulls: source + duration
A multi-GB docker pull ran with fully captured output -- minutes of
silence that read as a hang (user interrupted a healthy run over it).
Each mirror attempt now prints which source it is trying and how long
a hit took.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 02:36:25 +00:00
sora
5e98f6b701 ensure_image pulls through the CN mirror chain
The preflight did a bare 'docker pull' -- docker.io is slow/unreachable
from CN without luck; it now walks the same fallback chain as the SWE
prefetch (daemon mirrors -> daocloud -> 1ms.run -> baidubce -> sjtu ->
rat.dev), retagging the hit to the canonical name.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-16 02:32:11 +00:00
7114564301 Multi-endpoint model pool (openai-pool/{lo..hi} round-robin with failover), nothink/textools spec flags (chat_template_kwargs enable_thinking=false; text-protocol tool calls for backends without --enable-auto-tool-choice), global cross-bench queue in ladder runs 2026-08-26 06:26:52 +00:00
a32d902e94 Comparison-driven fixes: MCQ choices in prompt + letter contract, Answer: suffix for QA, markdown answer cleaning, trivia_qa answer_phrase priority, DROP gold-as-alternatives (OR) official semantics, retry on 5xx, numpy/scipy compat 2026-08-24 15:48:42 +00:00
3d16ab9103 Add SWE-bench single-turn harness (patch apply + FAIL_TO_PASS in official sweb image, sh entries in docker sandbox), bigcodebench official all-libs image, docker default for humaneval/LCB, input truncation (max_input_chars) + context assembly (passage/context), API retry with backoff, judge thread-safe bridge, parquet blob integrity check, summary.csv in out-dir, sandbox prefetch CLI 2026-08-24 08:43:49 +00:00