ruoxi_sun
58657935fc
Vendor LLMmap / llm-verify / llm-fingerprint-detector under bash/fingerprint/tools so the three fingerprint benchmarks run with only /data1/eval mounted (no /data1/xii dependency): - run.py DEFAULT_TOOLS_ROOT prefers builtin tools/, falls back to /data1/xii - exclude .git / node_modules / template backups - detector dist/ (pre-built) retained; node_modules not needed at runtime
154 lines
7.1 KiB
Markdown
154 lines
7.1 KiB
Markdown
# 📦 Extended Context — LLM Verify
|
|
|
|
> This file holds deeper context for complex features, domain-specific knowledge,
|
|
> architecture diagrams, and session-specific notes. Copilot reads this alongside
|
|
> `copilot-instructions.md` for richer understanding.
|
|
|
|
---
|
|
|
|
## 🏛 Architecture Overview
|
|
|
|
```
|
|
┌─────────────┐ ┌──────────────┐ ┌──────────────────┐
|
|
│ CLI / UI │────▶│ Handlers │────▶│ Services │
|
|
│ (FastAPI) │ │ (thin layer)│ │ (business logic) │
|
|
└─────────────┘ └──────────────┘ └────────┬─────────┘
|
|
│
|
|
┌──────────────┼──────────────┐
|
|
▼ ▼ ▼
|
|
┌───────────┐ ┌───────────┐ ┌───────────┐
|
|
│ Repos │ │ Adapters │ │ Prompts │
|
|
│ (DB CRUD) │ │ (AI APIs) │ │ (suites) │
|
|
└─────┬─────┘ └───────────┘ └───────────┘
|
|
▼
|
|
┌──────────────┐
|
|
│ SQLite DB │
|
|
│ (aiosqlite) │
|
|
└──────────────┘
|
|
```
|
|
|
|
### Request Flow
|
|
|
|
1. **CLI/API** receives request (run benchmark, view results)
|
|
2. **Handler** validates input via Pydantic schemas, delegates to service
|
|
3. **Service** orchestrates: loads prompt suite → calls adapters → stores results
|
|
4. **Adapter** wraps a specific AI provider API (OpenAI, Anthropic, generic OpenAI-compatible)
|
|
5. **Repository** persists benchmark runs & individual results to SQLite
|
|
6. **Fingerprint service** compares results across models to detect identity
|
|
|
|
---
|
|
|
|
## 🔍 Domain-Specific Knowledge
|
|
|
|
### Model Fingerprinting Strategy
|
|
|
|
AI models have behavioral fingerprints that are hard to fake:
|
|
|
|
1. **Identity probes** — Ask "Who made you?" in various indirect ways
|
|
2. **Capability tests** — Tasks where models differ (code gen, math, languages)
|
|
3. **Style analysis** — Measure response length, vocabulary, formatting patterns
|
|
4. **Edge cases** — Known model-specific behaviors (refusal patterns, hallucination tendencies)
|
|
5. **Latency profiling** — Response time patterns can indicate underlying infrastructure
|
|
6. **Token usage patterns** — Different models tokenize differently
|
|
|
|
### What Makes This Hard
|
|
|
|
- Resellers can add system prompts that say "You are Claude" to any model
|
|
- Simple identity questions are easy to fake with system prompts
|
|
- Need **behavioral** tests that can't be overridden by system prompts
|
|
- Models update over time, so fingerprints need periodic recalibration
|
|
|
|
### Suspect API Testing
|
|
|
|
A "suspect API" is an endpoint that claims to serve Model X but might actually be Model Y.
|
|
The system compares the suspect's responses against known baselines from verified APIs.
|
|
|
|
### Suspect API Protocol Detection
|
|
|
|
The suspect provider in the adapter factory now defaults to **Anthropic Messages protocol** (not OpenAI).
|
|
This is configured via:
|
|
|
|
- `_DEFAULT_PROTOCOL` map in `src/adapters/factory.py` — `suspect` → `anthropic`
|
|
- Can be overridden per-request via `protocol` field on `ModelConfig` schema
|
|
- Auth uses `x-api-key` header (Anthropic style), NOT `Authorization: Bearer` (OpenAI style)
|
|
|
|
### Known Suspect: opuscode.pro
|
|
|
|
| Field | Value |
|
|
| --------------------- | ------------------------------------------------------------------------ |
|
|
| Base URL | `https://opuscode.pro/api` |
|
|
| Protocol | Anthropic Messages API |
|
|
| Endpoint | `/v1/messages` |
|
|
| Auth | `x-api-key` header |
|
|
| Available Models | `Opus 4.6`, `Sonnet 4.5`, `Haiku 4.5` (their naming) |
|
|
| Default Model | `Opus 4.6` |
|
|
| Actual Model (tested) | `claude-3-5-sonnet-20241022` (Claude 3.5 Sonnet) |
|
|
| Red Flags | Inconsistent knowledge cutoffs, mentions "proxy server", 14s avg latency |
|
|
|
|
---
|
|
|
|
## 🧩 Multi-File Feature Notes
|
|
|
|
### Feature: Benchmark Runner Pipeline
|
|
|
|
**Files involved:**
|
|
|
|
- `src/services/benchmark_runner.py` — orchestrates a full benchmark run
|
|
- `src/adapters/base.py` — defines `ModelAdapter` interface
|
|
- `src/adapters/generic_adapter.py` — OpenAI-compatible adapter for suspect APIs
|
|
- `src/prompts/identity.py` — identity probe prompt suite
|
|
- `src/schemas/benchmark.py` — request/response models
|
|
- `src/repositories/result_repo.py` — stores results
|
|
|
|
**Flow:**
|
|
|
|
```
|
|
benchmark_runner.run(config) →
|
|
for each prompt_suite:
|
|
for each model_adapter:
|
|
adapter.complete(prompt) → response
|
|
store result in DB
|
|
return BenchmarkRunResult
|
|
```
|
|
|
|
### Feature: Model Comparator
|
|
|
|
**Files involved:**
|
|
|
|
- `src/services/model_comparator.py` — compares two sets of benchmark results
|
|
- `src/services/fingerprint.py` — statistical fingerprinting algorithms
|
|
- `src/repositories/result_repo.py` — fetches stored results
|
|
|
|
**Comparison dimensions:**
|
|
|
|
- Response similarity (cosine similarity on embeddings or n-gram overlap)
|
|
- Latency distribution (mean, p50, p95, p99)
|
|
- Token usage patterns
|
|
- Refusal patterns (what does each model refuse to answer?)
|
|
- Formatting habits (markdown usage, list styles, code block formatting)
|
|
|
|
---
|
|
|
|
## 📅 Session Context
|
|
|
|
> _Temporary notes for the current development session. Clear after each major milestone._
|
|
|
|
- **Session date:** 2026-02-17
|
|
- **Focus:** Live suspect API testing & fraud analysis
|
|
- **Notes:**
|
|
- Factory updated: `suspect` → Anthropic protocol by default
|
|
- `ModelConfig` now has `protocol` field for OpenAI/Anthropic override
|
|
- First benchmark run against opuscode.pro confirmed fraud: Claude 3.5 Sonnet served as Sonnet 4
|
|
- README updated with no-API-key usage guide and red flags documentation
|
|
- Server runs on port 8001 (via `python -m uvicorn src.main:app --host 127.0.0.1 --port 8001`)
|
|
|
|
---
|
|
|
|
## 🗺 Future Architecture Considerations
|
|
|
|
- **Plugin system** for custom prompt suites (load from YAML/JSON files)
|
|
- **Webhook support** to trigger benchmarks from CI/CD
|
|
- **Result export** to JSON/CSV for external analysis
|
|
- **Embedding-based comparison** using a local model for deeper similarity analysis
|
|
- **Historical tracking** to detect when a suspect API switches underlying models
|