09b2add673
New evalharness/fingerprint/ package (from evalstone fp_fusion v1.1, 2026-09-07 pruning final): probe battery -> concurrent collection -> five scoring views (verify/attribution/variant/adversarial/robustness), bundled family aliases + 27 reference fingerprints (12 fp_fusion schema). - CLI: 'evalharness fingerprint run ...' (REMAINDER passthrough, single source of arg definitions) + 'fingerprint list' for bundled references - imports rewritten package-relative; direct 'python3 run_fp_fusion.py' execution kept working via package bootstrap - offline analysis/collection scripts made path-independent (previously pinned to a /opt/evalscope path absent on this host) - shell scripts: hardcoded API key -> FP_API_KEY/OPENAI_API_KEY env vars - --reference accepts short names resolved against bundled references/ - pyproject: +httpx dependency, package-data references/*.json - tests/test_fingerprint.py: 10 offline tests (battery definitions, assembly counts, normalization, signals, verdict ladder, CLI wiring) - README: fingerprint section + architecture entry Verified on H20-1: tests 10/10, installed CLI OK, full-protocol run vs vectron GLM-5.3 reproduces baseline (score 0.9451, s_idn 0.846).
68 lines
2.7 KiB
Python
68 lines
2.7 KiB
Python
"""fp_fusion：模型指纹基准（API 端点身份核验）。
|
||
|
||
回答一个问题：API 背后跑的，到底是不是它声称的那个模型？
|
||
向 OpenAI 兼容端点发送探针电池（回答分布 / 自我身份 / 元知识 / 能力边界 /
|
||
文风五维），与参考指纹库比对，输出五档裁决 + 0~1 融合分 + 证据链。
|
||
一次 `--mode full` 运行产出五个视图：verify / attribution / variant /
|
||
adversarial / robustness。
|
||
|
||
CLI（推荐）::
|
||
|
||
evalharness fingerprint run --api-url http://localhost:8000/v1 \\
|
||
--model Qwen3-8B --mode full --cells core16 --text-skip pruned7 \\
|
||
--reference glm53 --report-path reports/fp_glm.json
|
||
|
||
evalharness fingerprint list # 列出内置参考指纹库
|
||
|
||
亦可 `python -m evalharness.fingerprint.run_fp_fusion ...` 或直接执行
|
||
`run_fp_fusion.py`，参数完全一致。
|
||
|
||
详细方法论文档见包内 `fp_fusion_介绍.md`；离线分析/参考采集脚本见包内
|
||
`*_snr.py` / `validate_*.py` / `collect_*.py`（均可在任意目录直接运行）。
|
||
"""
|
||
|
||
from pathlib import Path
|
||
|
||
REFERENCES_DIR = Path(__file__).resolve().parent / 'references'
|
||
|
||
__all__ = ['REFERENCES_DIR', 'main', 'list_references']
|
||
|
||
|
||
def list_references():
|
||
"""打印包内 references/ 的参考指纹清单（含报告口径后缀说明）。"""
|
||
fusion = sorted(REFERENCES_DIR.glob('*_fusion_reference.json'))
|
||
legacy = sorted(p for p in REFERENCES_DIR.glob('*_reference.json')
|
||
if not p.name.endswith('_fusion_reference.json'))
|
||
if not fusion and not legacy:
|
||
print(f'no bundled references found under {REFERENCES_DIR}')
|
||
return 0
|
||
print(f'bundled fingerprint references ({REFERENCES_DIR}):')
|
||
if fusion:
|
||
print(' fp_fusion 口径（--reference 短名直接可用）:')
|
||
for p in fusion:
|
||
print(f' {p.stem[:-len("_fusion_reference")]:24s} -> {p.name}')
|
||
if legacy:
|
||
print(' detector 旧口径（兼容保留）:')
|
||
for p in legacy:
|
||
print(f' {p.stem[:-len("_reference")]:24s} -> {p.name}')
|
||
print('\n用法: --reference <短名> （如 --reference glm53）或完整路径')
|
||
return 0
|
||
|
||
|
||
def main(argv=None):
|
||
"""`evalharness fingerprint` 子命令入口。
|
||
|
||
`fingerprint run <flags>` 与 `fingerprint <flags>` 等价（run 可省略）；
|
||
`fingerprint list` 列出内置参考库；其余全部透传给 run_fp_fusion。
|
||
"""
|
||
import sys
|
||
|
||
argv = list(sys.argv[1:] if argv is None else argv)
|
||
if argv and argv[0] == 'run':
|
||
argv = argv[1:]
|
||
if argv and argv[0] in ('list', 'references'):
|
||
return list_references()
|
||
from .run_fp_fusion import main as run_main
|
||
|
||
return run_main(argv)
|