370953729b
- perf_stats aggregator lives in eval/, not model/: the import failed silently and EVERY perf column was empty (not just ttft). Now warns on stderr instead of swallowing. - repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2 previously restored repeat 1's predictions and finished instantly with identical scores. rep1 keeps the legacy key (existing checkpoints still resume). - repeats summary: report the MEAN score and aggregate time/tokens over ALL runs (was: last run only). - README: six-benchmark command as the primary example. Co-Authored-By: Claude <noreply@anthropic.com>
7 lines
238 B
Python
7 lines
238 B
Python
"""Built-in dataset plugins.
|
|
|
|
Each subdirectory is a self-contained plugin: a ``plugin.py`` (registration)
|
|
plus its data files. Subpackages are imported lazily by
|
|
``evalharness.data._discover_builtin_datasets`` via their __init__.py.
|
|
"""
|