Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Retarget the bar to 0/N the moment scoring starts

Browse Source 
The scoring phase only reset the bar on the first judged sample; until
then the stale GENERATION state (100%, +N new) stayed on screen --
through multi-minute image-pull preflights it read as 'already done'
while nothing was scored yet.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-17 02:53:56 +00:00
parent 4f6e64e65c
commit 420c18c574
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
evalharness/model/runner.py
View File

@ -829,6 +829,13 @@ async def run_eval(
if status_callback: if status_callback:
status_callback('Scoring predictions against the benchmark recipe') status_callback('Scoring predictions against the benchmark recipe')
# retarget the bar to scoring IMMEDIATELY (0/N): waiting for the first
# on_scored left the stale GENERATION counters (100%, +N new) on screen
# through minute-long preflights (image pulls) -- reading as 'done'
if progress_reporter is not None and _records is None:
_ss0 = getattr(progress_reporter, 'set_scoring', None)
if _ss0 is not None:
_ss0(0, len(samples))
_meta = {'gen_input_tokens': usage.input_tokens, _meta = {'gen_input_tokens': usage.input_tokens,
'gen_output_tokens': usage.output_tokens, 'gen_output_tokens': usage.output_tokens,
'gen_total_tokens': usage.total_tokens, 'gen_total_tokens': usage.total_tokens,