--auto-concurrency removed as promised; 'auto' now starts the adaptive
gate at 2 (it ramps on its own demand/health signals). Plan displays
'auto (start 2, gate decides)'.
Co-Authored-By: Claude <noreply@anthropic.com>
- new flag wraps even a single endpoint as a one-member pool so the
per-endpoint AdaptiveGate takes over; --concurrency becomes the gate's
STARTING point (global semaphore lifted to the gate ceiling of 96)
- demand-driven AIMD fallback for endpoints without /metrics (404,
gateway-stripped, non-sglang): ramp +1 while callers wait on acquire
and the interval is failure-free; the old code early-returned on
fetch errors and never adapted at all. Real GLM endpoint verified:
/metrics is 404, so this fallback is the live path there
- probe fetch moved to a thread: a blocked urlopen parked the whole
event loop for its 4s timeout
- current limit surfaces on the progress bar as a magenta 'gate N'
field (pushed on every change)
Co-Authored-By: Claude <noreply@anthropic.com>
- accurate output structure (summary.xlsx/csv + per-bench xlsx; drop
stale detail.md/summary.md mentions)
- dedicated YAML-config section documenting the keys that actually work
(generation params + repeats + max_input_tokens) and precedence
- perf-stats table: what is always collected vs --perf streaming-only
- FAQ: endpoint probe, context-overflow shrink, thinking-mode notes
(non-stream chat_template_kwargs vs cloud-API param), auto-stream
- config/README.md rewritten to match the flat-key reality (old file
documented a directory scheme + judge/env keys that are not consumed)
Co-Authored-By: Claude <noreply@anthropic.com>
- perf_stats aggregator lives in eval/, not model/: the import failed
silently and EVERY perf column was empty (not just ttft). Now warns
on stderr instead of swallowing.
- repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2
previously restored repeat 1's predictions and finished instantly with
identical scores. rep1 keeps the legacy key (existing checkpoints still
resume).
- repeats summary: report the MEAN score and aggregate time/tokens over
ALL runs (was: last run only).
- README: six-benchmark command as the primary example.
Co-Authored-By: Claude <noreply@anthropic.com>
Explicit key flows only into request headers (adapter attribute / pool
member), so it cannot leak into the spec string, EvalReport, or logs --
verified by scanning a report produced with a sentinel key. Two-key
setups run twice with different --api-key, or use per-host env vars.
Co-Authored-By: Claude <noreply@anthropic.com>
- --judge now accepts a bare model name paired with --judge-api-url,
mirroring --model/--api-url; full legacy specs keep working
(_compose_judge_spec, verified: name+url -> spec, spec passthrough,
empty -> None; end-to-end on simple_qa with a live judge endpoint)
- mock adapter spellings: mock-boxed / mock-oracle / mock-fc preferred,
colon forms still accepted; bare 'mock' stays echo
- fix mock adapter singleton mode pollution: resolve_adapter memoizes
one instance, so mock-boxed then mock in one process leaked the
boxed mode into the echo run -- each mock spec now builds a fresh
instance
- README: mock-boxed in examples, judge flags row updated
Co-Authored-By: Claude <noreply@anthropic.com>
- cli: rich Run Summary table for multi-benchmark runs (green/red rows,
fallback to aligned plain text); unified _fmt_score (fractions render
as percentages everywhere -- was 1.0 in summary vs 100.0% in detail);
fix the stray "summary csv -> None/viz/..." print without --out-dir;
summary.md upgraded to a proper table with model/timestamp/ok-count
header -- one table for a whole N-benchmark run
- text renderer: single-bench headline deduped (dataset==recipe) and
compacted to one facts line; adaptive metric-name column (long names
no longer break alignment)
- md_compare: auto-switches to one-row-per-benchmark when comparing
different benchmarks with different metrics; same-bench model
comparison gains baseline delta markers (+/- percentage points)
- README: conda create/activate in the install block
Co-Authored-By: Claude <noreply@anthropic.com>
