EvalHarness

History

sora 652c13db36 Scores bound to predictions in the checkpoint; --resume controls both

User feedback: the out-dir report-reuse layer was one concept too many.
Now each checkpoint line carries {key, ts, pred, score}:

- --resume restores predictions AND their scores; when every sample's
  cached score matches the scoring-setup fingerprint (recipe/extract/
  scorers/judge), the report is replayed with NO scorer, extractor or
  docker container touching anything
- fingerprint mismatch (recipe or judge changed) -> automatic re-eval
  and backfill of the fresh scores
- no --resume -> nothing read, nothing written (full fresh run)
- --rescore = ignore cached scores, re-evaluate, refresh the cache
- aggregation always recomputed from cached per-sample scores (cheap,
  survives aggregator changes without invalidating)
- legacy checkpoints without a score field backfill on first evaluation

Removed: the out-dir report-reuse block (superseded; also the source of
the UnboundLocalError path).

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-14 06:38:22 +00:00

agent

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

config

README overhaul: features overview, config section, perf stats, FAQ

2026-09-14 02:30:38 +00:00

data

Fix NameError _fmt (module-level helper restored); aime/hmmt max_tokens 32768->8192 (non-stream path, gateway honors nothink)

2026-09-11 07:20:02 +00:00

eval

Scores bound to predictions in the checkpoint; --resume controls both