EvalHarness

History

sora 08605d9ab2 CLI flag symmetry: --judge-api-url split; mock-boxed hyphen names

- --judge now accepts a bare model name paired with --judge-api-url,
  mirroring --model/--api-url; full legacy specs keep working
  (_compose_judge_spec, verified: name+url -> spec, spec passthrough,
  empty -> None; end-to-end on simple_qa with a live judge endpoint)
- mock adapter spellings: mock-boxed / mock-oracle / mock-fc preferred,
  colon forms still accepted; bare 'mock' stays echo
- fix mock adapter singleton mode pollution: resolve_adapter memoizes
  one instance, so mock-boxed then mock in one process leaked the
  boxed mode into the echo run -- each mock spec now builds a fresh
  instance
- README: mock-boxed in examples, judge flags row updated

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-10 06:43:27 +00:00

agent

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

data

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

eval

dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes

2026-09-08 05:55:04 +00:00

model

CLI flag symmetry: --judge-api-url split; mock-boxed hyphen names