09b2add673
New evalharness/fingerprint/ package (from evalstone fp_fusion v1.1, 2026-09-07 pruning final): probe battery -> concurrent collection -> five scoring views (verify/attribution/variant/adversarial/robustness), bundled family aliases + 27 reference fingerprints (12 fp_fusion schema). - CLI: 'evalharness fingerprint run ...' (REMAINDER passthrough, single source of arg definitions) + 'fingerprint list' for bundled references - imports rewritten package-relative; direct 'python3 run_fp_fusion.py' execution kept working via package bootstrap - offline analysis/collection scripts made path-independent (previously pinned to a /opt/evalscope path absent on this host) - shell scripts: hardcoded API key -> FP_API_KEY/OPENAI_API_KEY env vars - --reference accepts short names resolved against bundled references/ - pyproject: +httpx dependency, package-data references/*.json - tests/test_fingerprint.py: 10 offline tests (battery definitions, assembly counts, normalization, signals, verdict ladder, CLI wiring) - README: fingerprint section + architecture entry Verified on H20-1: tests 10/10, installed CLI OK, full-protocol run vs vectron GLM-5.3 reproduces baseline (score 0.9451, s_idn 0.846).
36 lines
1.5 KiB
Bash
36 lines
1.5 KiB
Bash
#!/bin/bash
|
||
# 任务1：采集缺失模型参考（detector 16-cell + fp_fusion 26-cell）
|
||
# 用法: bash collect_ref.sh <model_id> <shortname>
|
||
# 依赖包外的 llm-fingerprint-detector（Node 工具，不在本仓库内），需显式指定：
|
||
# export FP_DET_TOOL=/path/to/llm-fingerprint-detector
|
||
# export FP_API_KEY=sk-xxxx # 或复用 OPENAI_API_KEY
|
||
set -e
|
||
MODEL="$1"
|
||
NAME="$2"
|
||
API="${FP_API_URL:-https://api.vectron.meta-stone.com/v1}"
|
||
KEY="${FP_API_KEY:-${OPENAI_API_KEY:-}}"
|
||
FP="$(cd "$(dirname "$0")" && pwd)"
|
||
DET="${FP_DET_TOOL:?需设置 FP_DET_TOOL 指向 llm-fingerprint-detector 检出目录}"
|
||
if [ -z "$KEY" ]; then echo "ERROR: 未设置 FP_API_KEY/OPENAI_API_KEY" >&2; exit 1; fi
|
||
|
||
echo "=== [$NAME] 16-cell detector 参考 ==="
|
||
cd "$DET"
|
||
LLM_FINGERPRINT_API_KEY="$KEY" node dist/cli.js fingerprint \
|
||
--base-url "$API" --model "$MODEL" --preset strict \
|
||
--concurrency 2 --timeout 90000 \
|
||
--out "$FP/references/${NAME}_reference.json" --quiet 2>&1 | tail -3
|
||
|
||
echo "=== [$NAME] 10 新增 cell（fp_fusion extra）==="
|
||
cd "$FP"
|
||
python3 collect_fp_extra_reference.py \
|
||
--api-url "$API" --model "$MODEL" --api-key "$KEY" \
|
||
--out /tmp/bfd/${NAME}_extra_cells.json 2>&1 | tail -2
|
||
|
||
echo "=== [$NAME] 合并 26-cell fusion ==="
|
||
python3 merge_fusion_reference.py "$NAME" "${NAME}_reference.json" \
|
||
/tmp/bfd/${NAME}_extra_cells.json 2>&1 | tail -3
|
||
|
||
echo "=== [$NAME] 校验 ==="
|
||
python3 check_ref.py "$FP/references/${NAME}_reference.json"
|
||
python3 check_ref.py "$FP/references/${NAME}_fusion_reference.json"
|