ruoxi_sun
58657935fc
Vendor LLMmap / llm-verify / llm-fingerprint-detector under bash/fingerprint/tools so the three fingerprint benchmarks run with only /data1/eval mounted (no /data1/xii dependency): - run.py DEFAULT_TOOLS_ROOT prefers builtin tools/, falls back to /data1/xii - exclude .git / node_modules / template backups - detector dist/ (pre-built) retained; node_modules not needed at runtime
256 lines
15 KiB
Markdown
256 lines
15 KiB
Markdown
# llm-fingerprint-detector
|
||
|
||
[](LICENSE)
|
||
[](package.json)
|
||
[](package.json)
|
||
[](https://arxiv.org/abs/2607.10252)
|
||
|
||
**Is that API really serving the model it claims?** Fingerprint and verify any OpenAI-compatible LLM endpoint from single-token output distributions — no logits, no weights, no privileged access. Just ~100–400 cheap one-word completions. Catches **model substitution**, silent quantization and server-side prompt injection by API resellers, gateways and aggregators.
|
||
|
||
An independent, open-source TypeScript implementation of:
|
||
|
||
> Tomáš Bruckner, **"One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions"**, [arXiv:2607.10252](https://arxiv.org/abs/2607.10252).
|
||
> Dataset: [DOI 10.5281/zenodo.21278557](https://doi.org/10.5281/zenodo.21278557) (CC-BY-4.0) · Paper code: [DOI 10.5281/zenodo.21278793](https://doi.org/10.5281/zenodo.21278793) (MIT)
|
||
|
||
This package is **not affiliated with the paper's author** — it is a from-scratch implementation of the published method, engineered as a reusable library + CLI. If you use the method in research, cite the paper.
|
||
|
||
- **Zero runtime dependencies** — Node ≥ 18 built-in `fetch`, nothing else
|
||
- **Library + CLI** — embed it, or run `llm-fingerprint verify` in CI
|
||
- **Reasoning-model aware** — auto-detects how to disable hidden "thinking" (OpenRouter / Zhipu / OpenAI-style), with a graceful fallback
|
||
- **Filter-resistant probes** — every probe is a plain semantic question drawn from a paraphrase pool; there is no magic string a gateway can special-case
|
||
- **Bundled sample references** for 11 popular models, derived from the paper's public dataset
|
||
|
||
> Prefer a no-install web version? Try the free online checker: **[tosea.ai/free-tools/llm-api-fingerprint-checker](https://tosea.ai/free-tools/llm-api-fingerprint-checker)** — same method, runs entirely in your browser.
|
||
|
||
---
|
||
|
||
## How it works
|
||
|
||
LLMs answer "*Name a random number between 1 and 100*" with model-specific, surprisingly stable biases (GPT-family models love 42 and 73; other families prefer 57, 37, 7…). The paper's key result: the **empirical distribution of one-word answers** across a small battery of such tasks is a reliable *behavioral fingerprint* — stable across time, load and providers for the same model, and clearly different between models.
|
||
|
||
```
|
||
probe battery (task × language cells) collect at temperature 1.0
|
||
┌──────────────────────────────┐ ┌─────────────────────────┐
|
||
│ random number 1-100 (en/zh) │ │ "42" ×19 "73" ×4 ... │
|
||
│ random color / letter / city │ ──25×──▶ │ per-cell answer │
|
||
│ coin flip / animal / fav-num │ │ distributions │
|
||
└──────────────────────────────┘ └───────────┬─────────────┘
|
||
▼
|
||
reference fingerprint ──── mean per-cell Jensen-Shannon
|
||
(trusted endpoint) divergence (base 2) ──▶ verdict
|
||
```
|
||
|
||
1. **Probe** — ask one-word questions (random numbers, colors, letters, coin flips…) in English and Chinese, `temperature=1.0`, `max_tokens=16`, a fixed minimal system prompt, hidden reasoning disabled. Requests are shuffled and paraphrased per call.
|
||
2. **Normalize** — NFC, punctuation/emoji stripping, case folding, first word, digit unification (`seven`/`七`/`٧`/`７` → `7`), color and coin-word canonicalization; answers are classified valid / invalid / refusal / empty.
|
||
3. **Compare** — per-cell Jensen-Shannon divergence (base 2, range 0–1 bit), averaged over cells where both sides have ≥10 valid samples.
|
||
4. **Verdict** — three bands calibrated against the paper's baselines (see [Interpreting results](#interpreting-results)).
|
||
|
||
## Install
|
||
|
||
```bash
|
||
npm install llm-fingerprint-detector # library + `llm-fingerprint` CLI
|
||
# or run the CLI ad hoc:
|
||
npx llm-fingerprint-detector --help
|
||
```
|
||
|
||
Requires Node ≥ 18.17 (built-in `fetch`). The core library is runtime-agnostic and also works in browsers/edge runtimes; only the CLI and the bundled-reference loader touch the filesystem.
|
||
|
||
## Quick start — CLI
|
||
|
||
```bash
|
||
# 1. Fingerprint the endpoint you trust (key read from OPENAI_API_KEY)
|
||
export OPENAI_API_KEY=sk-...
|
||
llm-fingerprint fingerprint \
|
||
--base-url https://api.openai.com/v1 \
|
||
--model gpt-4o-mini \
|
||
--out reference.gpt-4o-mini.json
|
||
|
||
# 2. Verify the endpoint you don't
|
||
export LLM_FINGERPRINT_API_KEY=sk-... # key for the endpoint under test
|
||
llm-fingerprint verify \
|
||
--base-url https://cheap-llm-reseller.example.com/v1 \
|
||
--model gpt-4o-mini \
|
||
--reference reference.gpt-4o-mini.json
|
||
```
|
||
|
||
```
|
||
Verdict: MISMATCH — behavior differs from the reference
|
||
Mean JSD: 0.481 over 8 comparable cell(s)
|
||
|
||
Interpretation scale (paper baselines, arXiv:2607.10252):
|
||
same model ≈ 0.14 · same model, other provider ≈ 0.227 · different model ≈ 0.463
|
||
thresholds: match ≤ 0.25 < uncertain ≤ 0.35 < mismatch
|
||
|
||
Per-cell JSD (most divergent first):
|
||
random-number-1-100:en 0.712 (24 vs 25 valid)
|
||
...
|
||
```
|
||
|
||
Exit codes are CI-friendly: `0` match · `2` mismatch · `3` uncertain · `4` insufficient · `1` error. API keys are only ever read from environment variables (`--api-key-env NAME`, defaulting to `LLM_FINGERPRINT_API_KEY` then `OPENAI_API_KEY`) and are never logged.
|
||
|
||
More: `llm-fingerprint --help`, [`examples/cli-examples.sh`](examples/cli-examples.sh).
|
||
|
||
## Quick start — library
|
||
|
||
```ts
|
||
import { fingerprint, compare, verify } from 'llm-fingerprint-detector'
|
||
|
||
// Collect a fingerprint
|
||
const run = await fingerprint(
|
||
{ baseUrl: 'https://api.openai.com/v1', model: 'gpt-4o-mini', apiKey: process.env.OPENAI_API_KEY },
|
||
{ cells: 8, samplesPerCell: 25, onProgress: (e) => console.log(e.done, '/', e.total) },
|
||
)
|
||
console.log(run.fingerprint) // JSON-serializable artifact
|
||
console.log(run.splitHalfJsd) // self-consistency (≈0.14 is normal)
|
||
|
||
// Verify another endpoint against it
|
||
const result = await verify(
|
||
{ baseUrl: 'https://suspect.example.com/v1', model: 'gpt-4o-mini', apiKey: process.env.SUSPECT_KEY },
|
||
run.fingerprint,
|
||
)
|
||
console.log(result.verdict, result.meanJsd) // 'match' | 'uncertain' | 'mismatch' | 'insufficient'
|
||
|
||
// Or compare two saved fingerprints offline
|
||
const distance = compare(fingerprintA, fingerprintB)
|
||
```
|
||
|
||
Bundled sample references (Node only):
|
||
|
||
```ts
|
||
import { listBundledReferences, loadBundledReference } from 'llm-fingerprint-detector/references'
|
||
|
||
const reference = loadBundledReference('openai/gpt-4o-mini')
|
||
const result = await verify(suspectEndpoint, reference)
|
||
```
|
||
|
||
Runnable examples: [`examples/01-fingerprint-endpoint.mjs`](examples/01-fingerprint-endpoint.mjs), [`examples/02-verify-endpoint.mjs`](examples/02-verify-endpoint.mjs).
|
||
|
||
## Tutorial: "Is my cheap API really GPT-4o / Claude / DeepSeek?"
|
||
|
||
You bought API access from a reseller/aggregator at half price. Are you getting the real model, a cheaper substitute, or a quantized clone? Ten minutes:
|
||
|
||
1. **Collect a trusted reference.** Fingerprint the *official* API (or any endpoint you fully trust) for the model in question:
|
||
|
||
```bash
|
||
llm-fingerprint fingerprint --base-url https://api.openai.com/v1 \
|
||
--model gpt-4o-mini --api-key-env OFFICIAL_KEY --out ref.json
|
||
```
|
||
|
||
No official access? Start with a bundled sample (`llm-fingerprint references`) — good enough for a first signal, with the caveats below.
|
||
|
||
2. **Verify the suspect endpoint** with the *same* model id:
|
||
|
||
```bash
|
||
llm-fingerprint verify --base-url https://reseller.example.com/v1 \
|
||
--model gpt-4o-mini --api-key-env RESELLER_KEY --reference ref.json
|
||
```
|
||
|
||
3. **Read the verdict.**
|
||
- `match` — the endpoint's one-token behavior is statistically consistent with your reference. That is strong (not absolute) evidence it's the same model.
|
||
- `mismatch` — the behavior is as far from the reference as *different* models typically are. Common causes, in practice: a substituted cheaper model, a heavily quantized deployment, or an injected system prompt.
|
||
- `uncertain` — the gray zone. Raise `--samples` (e.g. 40), use `--preset strict` (all 16 cells), or refresh your reference — models drift when providers ship updates.
|
||
- Also watch the warnings: a high **split-half JSD** means the endpoint disagrees *with itself* between the first and second half of your own run — a classic sign of an aggregator rotating several backends.
|
||
|
||
4. **Re-run before you conclude anything.** Two independent mismatch runs on different days are a much stronger signal than one.
|
||
|
||
## Interpreting results
|
||
|
||
Distances are mean Jensen-Shannon divergence (base 2), so 0 = identical behavior, 1 = disjoint answer sets. Paper baselines:
|
||
|
||
| meanJsd | reading |
|
||
|---|---|
|
||
| ≈ 0.14 | same model, same endpoint (sampling noise floor) |
|
||
| ≈ 0.227 | same model, different provider (median) |
|
||
| **≤ 0.25** | → verdict **match** |
|
||
| 0.25 – 0.35 | → verdict **uncertain** |
|
||
| **> 0.35** | → verdict **mismatch** |
|
||
| ≈ 0.463 | different models (median) |
|
||
|
||
**Error rates (paper):** distinguishing same-model vs different-model pairs achieves an equal error rate of ≈ **10.6% with 8 cells** and ≈ **7.3% with 40 cells**. A single run is *evidence*, not proof — treat verdicts accordingly.
|
||
|
||
### Limitations you must know
|
||
|
||
- **Fingerprints drift.** Providers silently update models; a 3-month-old reference can legitimately mismatch today's deployment. Always check `collectedAt`, refresh references regularly.
|
||
- **You need a trusted reference.** The method compares two endpoints; it cannot conjure ground truth. If your reference is wrong, your verdict is wrong.
|
||
- **The system prompt must be identical on both sides.** Swapping only the system prompt shifts fingerprints by JSD ≈ 0.44–0.46 — the magnitude of a model swap. This tool pins the same minimal system prompt on both sides automatically; if an endpoint *injects* its own server-side prompt, that will (correctly) surface as divergence.
|
||
- **Reasoning fallback lowers confidence.** When hidden reasoning can't be disabled, the run is flagged `postReasoning` — distributions shift measurably in that channel.
|
||
- **Quantization/serving stack changes** the same weights can move distances into the uncertain band.
|
||
- **A mismatch is a statistical observation, not an accusation.** Do not treat any single verdict as proof of fraud by a provider; investigate, re-run, and compare notes before drawing conclusions.
|
||
|
||
## Bundled sample references
|
||
|
||
`data/reference-fingerprints.sample.json` ships fingerprints of 11 popular models (GPT-4o, GPT-4o-mini, GPT-4.1-mini, Claude Sonnet 4.5, Gemini 2.5 Flash, DeepSeek-chat, Llama-3.1-8B, Qwen3-30B-A3B, Mistral Small 3.2, GLM-4.5, Kimi K2), derived from the paper's public dataset:
|
||
|
||
> Bruckner, T. (2026). *Single-token output distributions as behavioral fingerprints of large language models* [Data set]. Zenodo. [https://doi.org/10.5281/zenodo.21278557](https://doi.org/10.5281/zenodo.21278557) — CC-BY-4.0. Counts reconstructed from the published per-cell distributions and re-normalized with this package's normalizer (see [`scripts/build-sample-references.mjs`](scripts/build-sample-references.mjs)).
|
||
|
||
Those samples were collected by the paper's harness (via OpenRouter) under the paper's prompt protocol — close to, but not identical to, this package's battery. `compare()` flags such pairs with `protocolMismatch: true` and the verdict should be read as *indicative*. For anything that matters, collect your own reference:
|
||
|
||
```bash
|
||
llm-fingerprint fingerprint --base-url <trusted-url> --model <id> --out my-reference.json
|
||
```
|
||
|
||
To rebuild or extend the bundled samples from the Zenodo dataset, download the dataset, then:
|
||
|
||
```bash
|
||
npm run build
|
||
node scripts/build-sample-references.mjs path/to/distributions.json --models openai/gpt-4o,another/model
|
||
```
|
||
|
||
## API surface (TypeScript, fully typed)
|
||
|
||
| export | what it does |
|
||
|---|---|
|
||
| `fingerprint(endpoint, options?)` | probe an endpoint → `FingerprintRun` (fingerprint, adapter, split-half, warnings) |
|
||
| `compare(a, b)` | two fingerprints → `ComparisonResult` (meanJsd, per-cell JSD, verdict, baselines) |
|
||
| `verify(endpoint, reference, options?)` | fingerprint + compare in one call → `VerifyResult` |
|
||
| `normalizeAnswer(raw, domain)` | the full normalization pipeline (pure, unit-tested) |
|
||
| `jensenShannonDivergence(p, q)` | base-2 JSD over count maps |
|
||
| `splitHalfJsd(samplesByCell)` | endpoint self-consistency check |
|
||
| `detectReasoningAdapter(endpoint)` | which reasoning-disable field the endpoint accepts |
|
||
| `PROBE_TASKS`, `CELL_PRIORITY_ORDER`, `SYSTEM_PROMPTS` | the battery itself |
|
||
| `llm-fingerprint-detector/references` | bundled sample loader (Node only) |
|
||
|
||
All options (`cells`, `samplesPerCell`, `concurrency`, `timeoutMs`, `maxRetries`, `signal`, `onProgress`, …) are documented in [`src/types.ts`](src/types.ts).
|
||
|
||
## Development
|
||
|
||
```bash
|
||
git clone https://github.com/ToseaAI/llm-fingerprint-detector.git
|
||
cd llm-fingerprint-detector
|
||
npm install
|
||
npm run build # tsc → dist/
|
||
npm test # builds, then runs node --test against the built output
|
||
```
|
||
|
||
No test framework, no bundler — TypeScript and the Node built-in test runner only.
|
||
|
||
## Contributing
|
||
|
||
Issues and PRs are welcome. Especially valuable:
|
||
|
||
- **More languages** in the probe battery (the paper also used Arabic and Russian) — requires matching normalizer support, see `src/normalizer.ts`
|
||
- **Reference fingerprints** for more models/providers, collected with this tool's protocol (`one-token/v1`) and a documented date/channel
|
||
- **Threshold calibration data**: pairs of same-model / different-model runs to sharpen the match/mismatch cut points
|
||
|
||
Please keep changes dependency-free and covered by `node --test` tests.
|
||
|
||
## Citation & license
|
||
|
||
Method: cite the paper —
|
||
|
||
```bibtex
|
||
@article{bruckner2026onetoken,
|
||
title = {One Token Is Enough: Fingerprinting and Verifying Large Language Models
|
||
from Single-Token Output Distributions},
|
||
author = {Bruckner, Tom{\'a}{\v s}},
|
||
journal = {arXiv preprint arXiv:2607.10252},
|
||
year = {2026}
|
||
}
|
||
```
|
||
|
||
This package: [MIT](LICENSE). Bundled sample data: CC-BY-4.0, © Tomáš Bruckner (see above).
|
||
|
||
---
|
||
|
||
*Built and maintained by [Tosea.ai](https://tosea.ai). Want the zero-setup version? → [LLM API Fingerprint Checker](https://tosea.ai/free-tools/llm-api-fingerprint-checker) (runs in your browser, your key never leaves it).*
|