Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • 8c3d32bf2e swe scorer: return 'resolved' key (recipe metric name, not 'acc') main sora 2026-09-18 17:15:37 +00:00
  • 9889fb6c0e swe scorer: unwrap grader's per-instance return shape sora 2026-09-18 16:14:00 +00:00
  • 7085624da0 swe_agentic: env_state carries version/hints/env_commit (make_test_spec need) sora 2026-09-18 16:03:21 +00:00
  • 014e8bcd9b swe scorer: metadata from env_state (positional pair misalign fix) sora 2026-09-18 15:28:05 +00:00
  • 4618c990b1 official swe scorer: PIP_INDEX_URL tuna in eval containers sora 2026-09-18 10:45:51 +00:00
  • b3c837d3f3 swe_agentic: PIP_INDEX_URL env → tuna mirror in agent containers sora 2026-09-18 10:42:29 +00:00
  • c7811fa774 swe_agentic: sentinel patch from container file, not text split sora 2026-09-18 10:35:39 +00:00
  • fc8ade3c03 Background image service: pull-ahead + wait-barrier, unified lifecycle sora 2026-09-18 09:45:39 +00:00
  • e9e3230e46 swe_agentic: auto-release pulled images at process exit sora 2026-09-18 09:03:17 +00:00
  • f09fe184fe swe_agentic: ensure per-instance image before docker run sora 2026-09-18 08:54:30 +00:00
  • a93996094d swe_agentic: sentinel patch must end with a newline sora 2026-09-18 08:22:59 +00:00
  • 565ba11790 swe datasets: carry version/patch/hints/created_at in metadata sora 2026-09-18 08:11:18 +00:00
  • ace38eaf76 thinking.yaml: swe_bench_verified_agentic entry (env + max_turns) sora 2026-09-18 04:57:38 +00:00
  • 2a2335f842 swe_agentic: accept adapter kwarg in __init__ (get_env uniform signature) sora 2026-09-18 04:01:48 +00:00
  • de847206e5 swe scoring: official swebench harness (es-identical path), backend hook sora 2026-09-18 02:40:58 +00:00
  • f662006517 swe_bench_verified_agentic: multi-turn SWE agent (mini-swe-agent protocol) sora 2026-09-18 02:29:35 +00:00
  • f4e2c2d1b4 config: swe_bench_verified entry (patch generation params) sora 2026-09-17 10:10:05 +00:00
  • c23665fed7 Scoring preflight: auto batch-pull for multi-image benches sora 2026-09-17 09:38:16 +00:00
  • 587274d8b6 sandbox pull: SWE-bench per-instance pre-pull with resume (es port) sora 2026-09-17 09:36:20 +00:00
  • f844775f1e tau2 scorer: read the composite 'reward' field sora 2026-09-17 09:24:54 +00:00
  • 15b1574857 tau2: module-level silencing (order-independent) sora 2026-09-17 09:21:16 +00:00
  • a69cea2b76 tau2: ERROR+ log level; TAU2_DATA_DIR kills the data-dir banners sora 2026-09-17 09:19:01 +00:00
  • 30af23212e tau2: silence loguru BEFORE the first tau2 import sora 2026-09-17 09:14:57 +00:00
  • e015b5035f max_turns: config-driven (default 200), runner default 8 -> 200 sora 2026-09-17 09:12:17 +00:00
  • b62088997d tau2: floor max_turns at 40 (runner's generic 8 starved every sim) sora 2026-09-17 09:10:32 +00:00
  • fa60065099 tau2: silence per-message DEBUG floods; is_final_chunk only when done sora 2026-09-17 09:07:53 +00:00
  • d244d1a435 tau2 bridge: strip </think> leakage from simulator output sora 2026-09-17 09:03:52 +00:00
  • bcb2128b02 tau2 env: auto-install the engine (local checkout -> GitHub), manual fallback sora 2026-09-17 08:56:48 +00:00
  • 97a458d9b8 README: agent-env extra dependencies (tau2-bench engine, swebench) sora 2026-09-17 08:55:05 +00:00
  • 8519e9ae18 Primary metric skips extraction_failure_rate (diagnostics) sora 2026-09-17 08:38:27 +00:00
  • 081249d47c swe: repo-base image naming (correct indentation this time) sora 2026-09-17 08:33:45 +00:00
  • 5c14d11852 env per-bench from YAML config; swe image naming to official repo-base sora 2026-09-17 08:33:08 +00:00
  • b3c10d23f2 judge spec: url+model always combines (path-shaped model ids) sora 2026-09-17 08:08:13 +00:00
  • fe902db330 run_eval: construct the judge only when the recipe uses llm_judge sora 2026-09-17 08:07:49 +00:00
  • dc8d4605ac Judge: fail fast on missing --judge-api-url; build only when needed sora 2026-09-17 08:07:27 +00:00
  • 3f2ca8bfd5 aria2c multi-connection dataset downloads (CDN edge roulette fix) sora 2026-09-17 08:03:34 +00:00
  • 9f765fc5ce Strip <think> blocks from raw predictions before extraction sora 2026-09-17 07:41:33 +00:00
  • 6da45e0218 LocalSandbox.exec: accept entrypoint kwarg (signature parity) sora 2026-09-17 07:24:48 +00:00
  • 3cc0158236 hf_raw: fall back to cached blobs when the mirror listing degrades sora 2026-09-17 06:37:55 +00:00
  • 180b7b5007 docker pull --progress=plain: docker 25+ hides layer progress when piped sora 2026-09-17 06:26:04 +00:00
  • ca8e1d6038 Idle watchdog back to 300s default; begin_bench clears the stale bar sora 2026-09-17 06:08:33 +00:00
  • cfc1f2904e Fix mirror label: every template contains {img} sora 2026-09-17 05:54:12 +00:00
  • 869d2c41fb Mirror chain trimmed to 6 mainstream sources; manual-load guidance sora 2026-09-17 05:53:09 +00:00
  • adb46d4927 Mirror chain: +xuanyuan/dockerhub.icu (CN) + mirror.gcr.io/public.ecr (intl) sora 2026-09-17 05:51:16 +00:00
  • 731ab6e6c0 Pull chain: 12 CN mirrors + EVALHARNESS_DOCKER_MIRRORS override sora 2026-09-17 05:50:21 +00:00
  • dacffb02f2 Pull idle watchdog: 10s default (was 5min), env-tunable sora 2026-09-17 05:49:08 +00:00
  • d58fdb4198 Pull chain: 5-minute idle watchdog per mirror sora 2026-09-17 05:39:58 +00:00
  • 3ae1578325 Pull progress: split the stream on CR too sora 2026-09-17 03:51:02 +00:00
  • e41cd48272 Pull progress: overall byte summary per tick sora 2026-09-17 03:42:44 +00:00
  • cf9a3f6d3f entrypoint override passes only the script path sora 2026-09-17 03:31:24 +00:00
  • 774d2b9910 Pull progress: 3s snapshot throttle sora 2026-09-17 03:10:16 +00:00
  • fd7e1af3a4 bigcodebench: override the image ENTRYPOINT so our runner executes sora 2026-09-17 03:05:13 +00:00
  • 420c18c574 Retarget the bar to 0/N the moment scoring starts sora 2026-09-17 02:53:56 +00:00
  • 4f6e64e65c Stream docker pull progress; never rmi after a failed retag sora 2026-09-17 02:49:57 +00:00
  • f4f10649df Announce sandbox image pulls: source + duration sora 2026-09-17 02:36:25 +00:00
  • e9b79a2a41 thinking.yaml: 64k budgets for the full-thinking ladder (es parity) sora 2026-09-17 02:10:50 +00:00
  • b1c88bef25 --thinking: unified switch (off/low/medium/high/max/full) sora 2026-09-16 04:02:34 +00:00
  • 80194e845e --reasoning-effort: control thinking intensity (verified honored) sora 2026-09-16 03:36:03 +00:00
  • 5e98f6b701 ensure_image pulls through the CN mirror chain sora 2026-09-16 02:32:11 +00:00
  • 2949e10371 bigcodebench: official image + fail-fast preflight + no-retry on missing image sora 2026-09-16 02:13:14 +00:00
  • 1e72a553ec Benchmark categories in results (English) sora 2026-09-15 10:06:51 +00:00
  • 1fd0dcbdbb Summary table: 'cached' instead of a misleading ~0s time sora 2026-09-15 10:01:09 +00:00
  • f4ab4b4416 Unified duration format: elapsed/cur use the same tiered format as eta sora 2026-09-15 06:37:45 +00:00
  • 7b40ecc733 Contain terminal sample failures; 240s stream first-byte timeout sora 2026-09-15 04:39:51 +00:00
  • e021c94f44 Gate evidence: 2x level exactly (drop the 5-sample floor) sora 2026-09-15 03:51:48 +00:00
  • 8fc7df5b26 Gate: continuous scaling on shared endpoints; adaptive dwell; MIN_OK 5 sora 2026-09-15 03:46:14 +00:00
  • b81bff662d Gate knee criteria: not-worse (0.9x) instead of must-improve (1.1x) sora 2026-09-15 03:32:08 +00:00
  • ecb29309ef Bounded truncation pool (8 threads); human eta up to days sora 2026-09-15 03:05:04 +00:00
  • 6b336a9e5d Gate probe: dedicated executor + /metrics negative cache sora 2026-09-15 02:56:04 +00:00
  • 8cc5f6a9c6 in-flight label: '2 gen · 94 wait' instead of '2/96' sora 2026-09-15 02:52:13 +00:00
  • d83c2cc1df Serialize tokenizer first load; one-shot degradation warning sora 2026-09-15 02:49:34 +00:00
  • dafd171d4d in-flight shows admitted/held: bare 96 with gate 2 read as broken sora 2026-09-15 02:49:10 +00:00
  • 52547de9a3 Gate: exponential probe + binary search for capacity discovery sora 2026-09-15 02:42:53 +00:00
  • 4eb853c8fc Gate ramp requires completions: zero throughput = hold, not +1 sora 2026-09-15 02:33:21 +00:00
  • 3be48addfc Thread the truncation tokenize: inline encode froze the event loop sora 2026-09-15 02:26:15 +00:00
  • f8ff19d4a5 --concurrency auto is the only form; gate starts at 2 sora 2026-09-15 02:14:07 +00:00
  • 820a1dceac Auto-stream on huge INPUT too (longbench_v2 ReadTimeout fix) sora 2026-09-15 02:07:18 +00:00
  • a07431b324 Fix the ellipsis crash: text-tool-call parsing ran on plain code replies sora 2026-09-14 09:43:24 +00:00
  • f6ee6c7a8b Fix config auto-load silently disabled by the sample-counts manifest sora 2026-09-14 09:23:41 +00:00
  • 7921688149 Run plan: concrete sample counts + 'auto' concurrency display/alias sora 2026-09-14 09:11:07 +00:00
  • acb94e3e20 humaneval/gpqa_diamond: repeats 3 (align with the es reference runs) sora 2026-09-14 07:39:55 +00:00
  • 77d2c4569a JSON-tolerant report/checkpoint saves (default=str) sora 2026-09-14 07:15:50 +00:00
  • 652c13db36 Scores bound to predictions in the checkpoint; --resume controls both sora 2026-09-14 06:38:22 +00:00
  • 6f19719df2 Fix UnboundLocalError on the report-reuse path sora 2026-09-14 06:24:10 +00:00
  • 7eb7b26786 Summary time = this run's wall clock; n counts repeat generations sora 2026-09-14 06:15:51 +00:00
  • f272146b54 Report reuse: skip re-scoring when all predictions are checkpointed sora 2026-09-14 06:12:30 +00:00
  • 97c4eef8fb Incremental summary: flush summary.csv/xlsx after every benchmark sora 2026-09-14 05:56:42 +00:00
  • 0d925fc46f repeats: per-run reports under <out-dir>/<bench>/reps/repNN.report.jsonl sora 2026-09-14 05:51:29 +00:00
  • 9a64896c97 repeats: persist every per-run score, not just the last report sora 2026-09-14 05:43:50 +00:00
  • 9049bd683d openai-pool: allow plain-URL members (no port range required) sora 2026-09-14 04:05:18 +00:00
  • 6ad68bdfc0 --auto-concurrency: let the adaptive gate drive request concurrency sora 2026-09-14 04:01:45 +00:00
  • e1a96f18c0 Scoring phase retargets the same progress bar sora 2026-09-14 03:51:25 +00:00
  • 25e51c5b30 Scoring milestones: only when no live bar sora 2026-09-14 03:48:01 +00:00
  • 841b7b1ffb Bound the sandbox rm -f cleanup calls (60s) sora 2026-09-14 03:34:13 +00:00
  • 92b36c5e6c Sandbox: reap timed-out containers, retry daemon-side failures; scoring milestones sora 2026-09-14 03:26:35 +00:00
  • a4d4592864 Scoring-phase progress; pool: fix double-release + cross-loop reuse sora 2026-09-14 03:14:08 +00:00
  • f5c2e4c5af Progress bar: fresh-this-run counter; fix KeyError 'success' sora 2026-09-14 03:05:18 +00:00
  • 20d2d5537a Run plan: show real sample/generation counts from local cache sora 2026-09-14 02:45:20 +00:00
  • e9d7e7f4eb README overhaul: features overview, config section, perf stats, FAQ sora 2026-09-14 02:30:38 +00:00
  • a17dd88611 Untrack pip build/ artifacts, restore gen_profiles.yaml sora 2026-09-11 13:38:43 +00:00