Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

repeats: persist every per-run score, not just the last report

Browse Source 
12 repeats each scored on the terminal but the artifacts kept only the
final repeat's report; the run list scrolled away. Now the last report
carries metric_groups['repeats'] = {n_runs, scores[12], mean, min, max,
std} -- lands in report.jsonl's header line, the xlsx categories sheet,
and summary.csv's categories column (numeric entries); the console
summary line also prints the full runs=[...] list.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-14 05:43:50 +00:00
parent 9049bd683d
commit 9a64896c97
1 changed files with 15 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
evalharness/cli.py
View File

@ -784,9 +784,22 @@ def _cmd_eval_run(args) -> int:
if _repeats > 1 and _scores: if _repeats > 1 and _scores:
_mean = sum(_scores) / len(_scores) _mean = sum(_scores) / len(_scores)
_spread = f'{min(_scores):.3f}–{max(_scores):.3f}' if len(_scores) > 1 else f'{_scores[0]:.3f}' _spread = f'{min(_scores):.3f}–{max(_scores):.3f}' if len(_scores) > 1 else f'{_scores[0]:.3f}'
print(f'\n{name}: {_repeats} runs | mean={_mean:.4f} | range={_spread}', flush=True) _runs_txt = ' '.join(f'{s:.2f}' for s in _scores)
print(f'\n{name}: {_repeats} runs | mean={_mean:.4f} | range={_spread} '
f'| runs=[{_runs_txt}]', flush=True)
# summary/xlsx report the MEAN over repeats (es parity); # summary/xlsx report the MEAN over repeats (es parity);
# per-run scores stay in report.jsonl / the per-run metrics # EVERY per-run score lands in metric_groups['repeats'] so
# the 12 runs are queryable in report.jsonl / the categories
# columns instead of only scrolling past on the terminal
_var = sum((s - _mean) ** 2 for s in _scores) / len(_scores)
report.metric_groups['repeats'] = {
'n_runs': len(_scores),
'scores': [round(s, 4) for s in _scores],
'mean': round(_mean, 4),
'min': round(min(_scores), 4),
'max': round(max(_scores), 4),
'std': round(_var ** 0.5, 4),
}
_primary = next(iter(report.metrics), '') _primary = next(iter(report.metrics), '')
if _primary: if _primary:
report.metrics[f'{_primary}_last_run'] = report.metrics[_primary] report.metrics[f'{_primary}_last_run'] = report.metrics[_primary]