Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

repeats: per-run reports under <out-dir>/<bench>/reps/repNN.report.jsonl

Browse Source 
Each file carries that run's own score and samples; the top-level
report.jsonl stays the mean-summary view (mean + repeats group).

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-14 05:51:29 +00:00
parent 9a64896c97
commit 0d925fc46f
1 changed files with 9 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
evalharness/cli.py
View File

@ -776,6 +776,15 @@ def _cmd_eval_run(args) -> int:
if k != 'extraction_failure_rate'), None)
if _m is not None:
_scores.append(_m)
if _repeats > 1 and out_dir:
# per-run report: <out-dir>/<bench>/reps/repNN.report.jsonl
# (each carries THAT run's own score + samples; the
# top-level report.jsonl stays the mean-summary view)
import pathlib as _pl
_rdir = _pl.Path(out_dir) / name / 'reps'
_rdir.mkdir(parents=True, exist_ok=True)
report.save(str(_rdir / f'rep{_rep + 1:02d}.report.jsonl'))
_rep_info = report.metric_groups.get('run_info', {}) or {}
_rep_secs += sum(float((s.usage or {}).get('latency_s', 0) or 0)
for s in report.samples)