- perf_stats aggregator lives in eval/, not model/: the import failed
silently and EVERY perf column was empty (not just ttft). Now warns
on stderr instead of swallowing.
- repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2
previously restored repeat 1's predictions and finished instantly with
identical scores. rep1 keeps the legacy key (existing checkpoints still
resume).
- repeats summary: report the MEAN score and aggregate time/tokens over
ALL runs (was: last run only).
- README: six-benchmark command as the primary example.
Co-Authored-By: Claude <noreply@anthropic.com>
Explicit key flows only into request headers (adapter attribute / pool
member), so it cannot leak into the spec string, EvalReport, or logs --
verified by scanning a report produced with a sentinel key. Two-key
setups run twice with different --api-key, or use per-host env vars.
Co-Authored-By: Claude <noreply@anthropic.com>
- --judge now accepts a bare model name paired with --judge-api-url,
mirroring --model/--api-url; full legacy specs keep working
(_compose_judge_spec, verified: name+url -> spec, spec passthrough,
empty -> None; end-to-end on simple_qa with a live judge endpoint)
- mock adapter spellings: mock-boxed / mock-oracle / mock-fc preferred,
colon forms still accepted; bare 'mock' stays echo
- fix mock adapter singleton mode pollution: resolve_adapter memoizes
one instance, so mock-boxed then mock in one process leaked the
boxed mode into the echo run -- each mock spec now builds a fresh
instance
- README: mock-boxed in examples, judge flags row updated
Co-Authored-By: Claude <noreply@anthropic.com>
- cli: rich Run Summary table for multi-benchmark runs (green/red rows,
fallback to aligned plain text); unified _fmt_score (fractions render
as percentages everywhere -- was 1.0 in summary vs 100.0% in detail);
fix the stray "summary csv -> None/viz/..." print without --out-dir;
summary.md upgraded to a proper table with model/timestamp/ok-count
header -- one table for a whole N-benchmark run
- text renderer: single-bench headline deduped (dataset==recipe) and
compacted to one facts line; adaptive metric-name column (long names
no longer break alignment)
- md_compare: auto-switches to one-row-per-benchmark when comparing
different benchmarks with different metrics; same-bench model
comparison gains baseline delta markers (+/- percentage points)
- README: conda create/activate in the install block
Co-Authored-By: Claude <noreply@anthropic.com>
