370953729b
- perf_stats aggregator lives in eval/, not model/: the import failed silently and EVERY perf column was empty (not just ttft). Now warns on stderr instead of swallowing. - repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2 previously restored repeat 1's predictions and finished instantly with identical scores. rep1 keeps the legacy key (existing checkpoints still resume). - repeats summary: report the MEAN score and aggregate time/tokens over ALL runs (was: last run only). - README: six-benchmark command as the primary example. Co-Authored-By: Claude <noreply@anthropic.com>
68 lines
2.7 KiB
Python
68 lines
2.7 KiB
Python
"""fp_fusion：模型指纹基准（API 端点身份核验）。
|
||
|
||
回答一个问题：API 背后跑的，到底是不是它声称的那个模型？
|
||
向 OpenAI 兼容端点发送探针电池（回答分布 / 自我身份 / 元知识 / 能力边界 /
|
||
文风五维），与参考指纹库比对，输出五档裁决 + 0~1 融合分 + 证据链。
|
||
一次 `--mode full` 运行产出五个视图：verify / attribution / variant /
|
||
adversarial / robustness。
|
||
|
||
CLI（推荐）::
|
||
|
||
evalharness fingerprint run --api-url http://localhost:8000/v1 \\
|
||
--model Qwen3-8B --mode full --cells core16 --text-skip pruned7 \\
|
||
--reference glm53 --report-path reports/fp_glm.json
|
||
|
||
evalharness fingerprint list # 列出内置参考指纹库
|
||
|
||
亦可 `python -m evalharness.fingerprint.run_fp_fusion ...` 或直接执行
|
||
`run_fp_fusion.py`，参数完全一致。
|
||
|
||
详细方法论文档见包内 `fp_fusion_介绍.md`；离线分析/参考采集脚本见包内
|
||
`*_snr.py` / `validate_*.py` / `collect_*.py`（均可在任意目录直接运行）。
|
||
"""
|
||
|
||
from pathlib import Path
|
||
|
||
REFERENCES_DIR = Path(__file__).resolve().parent / 'references'
|
||
|
||
__all__ = ['REFERENCES_DIR', 'main', 'list_references']
|
||
|
||
|
||
def list_references():
|
||
"""打印包内 references/ 的参考指纹清单（含报告口径后缀说明）。"""
|
||
fusion = sorted(REFERENCES_DIR.glob('*_fusion_reference.json'))
|
||
legacy = sorted(p for p in REFERENCES_DIR.glob('*_reference.json')
|
||
if not p.name.endswith('_fusion_reference.json'))
|
||
if not fusion and not legacy:
|
||
print(f'no bundled references found under {REFERENCES_DIR}')
|
||
return 0
|
||
print(f'bundled fingerprint references ({REFERENCES_DIR}):')
|
||
if fusion:
|
||
print(' fp_fusion 口径（--reference 短名直接可用）:')
|
||
for p in fusion:
|
||
print(f' {p.stem[:-len("_fusion_reference")]:24s} -> {p.name}')
|
||
if legacy:
|
||
print(' detector 旧口径（兼容保留）:')
|
||
for p in legacy:
|
||
print(f' {p.stem[:-len("_reference")]:24s} -> {p.name}')
|
||
print('\n用法: --reference <短名> （如 --reference glm53）或完整路径')
|
||
return 0
|
||
|
||
|
||
def main(argv=None):
|
||
"""`evalharness fingerprint` 子命令入口。
|
||
|
||
`fingerprint run <flags>` 与 `fingerprint <flags>` 等价（run 可省略）；
|
||
`fingerprint list` 列出内置参考库；其余全部透传给 run_fp_fusion。
|
||
"""
|
||
import sys
|
||
|
||
argv = list(sys.argv[1:] if argv is None else argv)
|
||
if argv and argv[0] == 'run':
|
||
argv = argv[1:]
|
||
if argv and argv[0] in ('list', 'references'):
|
||
return list_references()
|
||
from .run_fp_fusion import main as run_main
|
||
|
||
return run_main(argv)
|