6 Commits

Author SHA1 Message Date
sora
a4d4592864 Scoring-phase progress; pool: fix double-release + cross-loop reuse
Scoring progress: docker-exec benches (humaneval etc.) score for
minutes with zero feedback -- the bar sat at 'generating 100%' and
looked hung. evaluate() now takes on_scored(i, n) (atomic counter,
fires from worker threads), run_eval passes it through, and the CLI
shows 'scoring 42/164' on the bar + milestone log lines every 10%
(also fixes the phase match: 'scoring' never matched the capitalized
'Scoring predictions...' status message, so the bar never even
switched its label).

PooledAdapter:
- one release per acquire: the exception path released True (inner
  finally) AND False (except handler), double-decrementing _inflight
  (over-admission) and applying the x0.7 backoff twice
- AdaptiveGate: rebuild the Condition + probe task when the event loop
  changes -- pools are cached across benchmarks and the CLI runs
  asyncio.run() per bench/repeat; a loop-bound Condition from a closed
  loop raises 'bound to a different event loop' under contention

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-14 03:14:08 +00:00
sora
c78b0d6f0f Add --api-key/--judge-api-key: explicit keys override env resolution, never serialized
Explicit key flows only into request headers (adapter attribute / pool
member), so it cannot leak into the spec string, EvalReport, or logs --
verified by scanning a report produced with a sentinel key. Two-key
setups run twice with different --api-key, or use per-host env vars.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 06:50:18 +00:00
sora
46bef7d3dd dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-08 05:55:04 +00:00
e32da10ce9 longbench_v2 chain of fixes: reasoning_content output fallback (Qwen3 thinking channel), nothink soft switch /no_think (template-400 immune), truncation input+output budget, adaptive context shrink on server 400 (cross-tokenizer margin, body-aware detection), pool fails over on 4xx 2026-08-27 06:27:59 +00:00
f2a914d0ff Performance profile: TTFT/ITL via SSE streaming (collect_perf flag), retries/http_status per request, pool success/retry/failure rates, perf_stats aggregator (P50/P95/P99 latency+ttft, tokens, retry_rate) in metric_groups['perf']; verified on fake SSE server 2026-08-27 02:45:51 +00:00
7114564301 Multi-endpoint model pool (openai-pool/{lo..hi} round-robin with failover), nothink/textools spec flags (chat_template_kwargs enable_thinking=false; text-protocol tool calls for backends without --enable-auto-tool-choice), global cross-bench queue in ladder runs 2026-08-26 06:26:52 +00:00