12 Commits

Author SHA1 Message Date
sora
731ab6e6c0 Pull chain: 12 CN mirrors + EVALHARNESS_DOCKER_MIRRORS override
Was 6 explicit sources (the daemon's first hop hides 12 more of its
own); added 1panel.live, dockerproxy, 163, baidubce, ustc, sjtug, nju,
tuna. With the 10s idle watchdog the full sweep costs ~2min worst
case. EVALHARNESS_DOCKER_MIRRORS replaces the whole list.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 05:50:21 +00:00
sora
dacffb02f2 Pull idle watchdog: 10s default (was 5min), env-tunable
Any docker output resets the timer, so this only fires on totally
silent (dead) mirrors; EVALHARNESS_PULL_IDLE_S overrides.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 05:49:08 +00:00
sora
d58fdb4198 Pull chain: 5-minute idle watchdog per mirror
A dead mirror emitted zero bytes and the fixed 3600s attempt timeout
let it stall the whole chain for an hour (third occurrence tonight:
15m49s frozen at 0 bytes). select() with a 300s idle timeout now kills
the attempt and moves to the next mirror; a trickling source (1 line/s
Downloading updates) resets the timer and stays alive.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 05:39:58 +00:00
sora
3ae1578325 Pull progress: split the stream on CR too
docker refreshes Downloading lines with \r, not \n -- readline()
parked them in the buffer, so the byte snapshots never surfaced and
the overall-progress dict stayed empty (the  summary never printed).
Chunked reads now split on both terminators; verified with a synthetic
\r-stream that the parser yields the refresh lines and computes the
52.2/65.1MB-style summary.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 03:51:02 +00:00
sora
e41cd48272 Pull progress: overall byte summary per tick
Per-layer events gave no sense of how much is LEFT. Downloading lines
are parsed per-layer and summed; every 3s the snapshot becomes
' 3.14/9.27 GB (34%) · 最大层 3f4a2b1c9d0e: 0.51/1.2 GB' instead of
a bare layer line. Key events (Pulling from / Status / errors) still
print immediately.

Co-ached-By: Claude <noreply@anthropic.com>
2026-09-17 03:42:44 +00:00
sora
774d2b9910 Pull progress: 3s snapshot throttle
Cold pulls now tick a 'Downloading xMB/yGB' snapshot at most every 3s
(was 10s) -- keyword hits (Pull complete / Status / errors) still
print immediately.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 03:10:16 +00:00
sora
4f6e64e65c Stream docker pull progress; never rmi after a failed retag
Two pull UX/correctness fixes: (1) a multi-GB pull with captured
output is minutes of silence reading as a hang -- layer progress now
streams (throttled) to stderr; (2) the mirror-tag -> canonical retag
could fail silently and the following rmi then deleted the ONLY tag,
losing a 342s pull and forcing a full re-download -- retag is now
verified and the mirror tag kept on failure.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 02:49:57 +00:00
sora
f4f10649df Announce sandbox image pulls: source + duration
A multi-GB docker pull ran with fully captured output -- minutes of
silence that read as a hang (user interrupted a healthy run over it).
Each mirror attempt now prints which source it is trying and how long
a hit took.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-17 02:36:25 +00:00
sora
5e98f6b701 ensure_image pulls through the CN mirror chain
The preflight did a bare 'docker pull' -- docker.io is slow/unreachable
from CN without luck; it now walks the same fallback chain as the SWE
prefetch (daemon mirrors -> daocloud -> 1ms.run -> baidubce -> sjtu ->
rat.dev), retagging the hit to the canonical name.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-16 02:32:11 +00:00
7114564301 Multi-endpoint model pool (openai-pool/{lo..hi} round-robin with failover), nothink/textools spec flags (chat_template_kwargs enable_thinking=false; text-protocol tool calls for backends without --enable-auto-tool-choice), global cross-bench queue in ladder runs 2026-08-26 06:26:52 +00:00
a32d902e94 Comparison-driven fixes: MCQ choices in prompt + letter contract, Answer: suffix for QA, markdown answer cleaning, trivia_qa answer_phrase priority, DROP gold-as-alternatives (OR) official semantics, retry on 5xx, numpy/scipy compat 2026-08-24 15:48:42 +00:00
3d16ab9103 Add SWE-bench single-turn harness (patch apply + FAIL_TO_PASS in official sweb image, sh entries in docker sandbox), bigcodebench official all-libs image, docker default for humaneval/LCB, input truncation (max_input_chars) + context assembly (passage/context), API retry with backoff, judge thread-safe bridge, parquet blob integrity check, summary.csv in out-dir, sandbox prefetch CLI 2026-08-24 08:43:49 +00:00