Previously both were written only after ALL benches finished -- a
multi-hour mmlu_pro left the summary stale for hours, and a crashed
run left the PREVIOUS run's files in place. Now each completed bench
(including failed ones) rewrites the summaries with everything so far.
Co-Authored-By: Claude <noreply@anthropic.com>
Each file carries that run's own score and samples; the top-level
report.jsonl stays the mean-summary view (mean + repeats group).
Co-Authored-By: Claude <noreply@anthropic.com>
12 repeats each scored on the terminal but the artifacts kept only the
final repeat's report; the run list scrolled away. Now the last report
carries metric_groups['repeats'] = {n_runs, scores[12], mean, min, max,
std} -- lands in report.jsonl's header line, the xlsx categories sheet,
and summary.csv's categories column (numeric entries); the console
summary line also prints the full runs=[...] list.
Co-Authored-By: Claude <noreply@anthropic.com>
--auto-concurrency wraps a single endpoint as a 1-member pool; the
spec parser rejected it with 'needs a {start..end} port range' and
killed every benchmark. Range-less comma segments are now plain
members; only a fully empty endpoint list errors.
Co-Authored-By: Claude <noreply@anthropic.com>
- new flag wraps even a single endpoint as a one-member pool so the
per-endpoint AdaptiveGate takes over; --concurrency becomes the gate's
STARTING point (global semaphore lifted to the gate ceiling of 96)
- demand-driven AIMD fallback for endpoints without /metrics (404,
gateway-stripped, non-sglang): ramp +1 while callers wait on acquire
and the interval is failure-free; the old code early-returned on
fetch errors and never adapted at all. Real GLM endpoint verified:
/metrics is 404, so this fallback is the live path there
- probe fetch moved to a thread: a blocked urlopen parked the whole
event loop for its 4s timeout
- current limit surfaces on the progress bar as a magenta 'gate N'
field (pushed on every change)
Co-Authored-By: Claude <noreply@anthropic.com>
The bar kept its stale 100% generation state during scoring with only
the description counter moving. Now set_scoring() refills the bar with
judged samples (0->100%, fresh clock/rate/eta for the phase), clears
the generation '+N new' marker, and the next bench retargets back to
generating cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
The bar's 'scoring N/164' counter and 21 milestone log lines were both
active on terminals -- the lines are noise there. Emit them only for
pipes/redirects where no bar exists (every ~10% now).
Co-Authored-By: Claude <noreply@anthropic.com>
An unbounded docker rm against a bloated daemon hangs for minutes and
silently eats the worker pool: 7 of 8 scoring workers were observed
stuck in cleanup while only 1 execution ran.
Co-Authored-By: Claude <noreply@anthropic.com>
- docker exec: named containers; a timed-out/killed 'docker run' only
kills the CLI client while the container lives on (--rm fires on
EXIT) -- rm -f the name on timeout/interrupt so runs stop leaking
- exit 125 = daemon-side failure, not model failure: retry up to 2x
(a bloated daemon was turning healthy samples into pass=0)
- scoring milestones: first completion logs immediately, then every
~5% (10% was too sparse when docker is slow: minutes of silence
right after the 'scoring' phase starts, looks hung)
Co-Authored-By: Claude <noreply@anthropic.com>
Scoring progress: docker-exec benches (humaneval etc.) score for
minutes with zero feedback -- the bar sat at 'generating 100%' and
looked hung. evaluate() now takes on_scored(i, n) (atomic counter,
fires from worker threads), run_eval passes it through, and the CLI
shows 'scoring 42/164' on the bar + milestone log lines every 10%
(also fixes the phase match: 'scoring' never matched the capitalized
'Scoring predictions...' status message, so the bar never even
switched its label).
PooledAdapter:
- one release per acquire: the exception path released True (inner
finally) AND False (except handler), double-decrementing _inflight
(over-admission) and applying the x0.7 backoff twice
- AdaptiveGate: rebuild the Condition + probe task when the event loop
changes -- pools are cached across benchmarks and the CLI runs
asyncio.run() per bench/repeat; a loop-bound Condition from a closed
loop raises 'bound to a different event loop' under contention
Co-Authored-By: Claude <noreply@anthropic.com>
- Completed X/Y now carries (+N new): samples generated by THIS run,
excluding the checkpoint-restored head start (previously 98/164 told
you nothing about how much work this invocation actually did)
- rate/eta computed over fresh samples only (restored ones counted
toward 141/6s = 23/s when 1 sample had been generated)
- fix KeyError 'success': advance() still read the success/failed task
fields after they were dropped from the column set -- first advance
on a resumed bench killed the whole benchmark
- set_overall: give the overall task the fields the shared columns
actually read (cur/retries/elapsed/eta); the old field set was from
a previous column layout
Co-Authored-By: Claude <noreply@anthropic.com>
Samples row said 'full dataset (counted when each loads)' -- now it
counts cached samples.jsonl entries (never touches the network, plan
stays instant on cold machines) and multiplies by YAML repeats:
12,957 samples (cached) → 13,617 generations (repeats)
--limit caps each bench before summing; uncached benches are flagged.
YAML config loading extracted to _load_bench_cfg, shared by the run
loop and the plan so repeats can't disagree between the two.
Co-Authored-By: Claude <noreply@anthropic.com>
- accurate output structure (summary.xlsx/csv + per-bench xlsx; drop
stale detail.md/summary.md mentions)
- dedicated YAML-config section documenting the keys that actually work
(generation params + repeats + max_input_tokens) and precedence
- perf-stats table: what is always collected vs --perf streaming-only
- FAQ: endpoint probe, context-overflow shrink, thinking-mode notes
(non-stream chat_template_kwargs vs cloud-API param), auto-stream
- config/README.md rewritten to match the flat-key reality (old file
documented a directory scheme + judge/env keys that are not consumed)
Co-Authored-By: Claude <noreply@anthropic.com>
- perf_stats aggregator lives in eval/, not model/: the import failed
silently and EVERY perf column was empty (not just ttft). Now warns
on stderr instead of swallowing.
- repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2
previously restored repeat 1's predictions and finished instantly with
identical scores. rep1 keeps the legacy key (existing checkpoints still
resume).
- repeats summary: report the MEAN score and aggregate time/tokens over
ALL runs (was: last run only).
- README: six-benchmark command as the primary example.
Co-Authored-By: Claude <noreply@anthropic.com>
