370953729b
- perf_stats aggregator lives in eval/, not model/: the import failed silently and EVERY perf column was empty (not just ttft). Now warns on stderr instead of swallowing. - repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2 previously restored repeat 1's predictions and finished instantly with identical scores. rep1 keeps the legacy key (existing checkpoints still resume). - repeats summary: report the MEAN score and aggregate time/tokens over ALL runs (was: last run only). - README: six-benchmark command as the primary example. Co-Authored-By: Claude <noreply@anthropic.com>
15 lines
592 B
Python
15 lines
592 B
Python
#!/usr/bin/env python3
|
|
"""Show reference summary and incomplete cells."""
|
|
import json
|
|
import sys
|
|
|
|
ref = json.load(open(sys.argv[1]))
|
|
print("protocol:", ref["protocol"], "| model:", ref["model"],
|
|
"| cells:", len(ref["cells"]), "| sppc:", ref["samplesPerCell"])
|
|
tv = sum(c["validCount"] for c in ref["cells"].values())
|
|
tt = sum(c["totalCount"] for c in ref["cells"].values())
|
|
print(f"valid {tv}/{tt}")
|
|
bad = [(cid, c["validCount"], c["totalCount"])
|
|
for cid, c in sorted(ref["cells"].items())
|
|
if c["validCount"] < c["totalCount"]]
|
|
print("incomplete:", bad if bad else "NONE") |