evalstone/bash/fingerprint/tools/llm-verify/.copilot/context.md

📦 Extended Context — LLM Verify

This file holds deeper context for complex features, domain-specific knowledge, architecture diagrams, and session-specific notes. Copilot reads this alongside copilot-instructions.md for richer understanding.

---

🏛 Architecture Overview

┌─────────────┐     ┌──────────────┐     ┌──────────────────┐
│   CLI / UI  │────▶│   Handlers   │────▶│    Services       │
│  (FastAPI)  │     │  (thin layer)│     │  (business logic) │
└─────────────┘     └──────────────┘     └────────┬─────────┘
                                                   │
                                    ┌──────────────┼──────────────┐
                                    ▼              ▼              ▼
                             ┌───────────┐  ┌───────────┐  ┌───────────┐
                             │   Repos   │  │  Adapters  │  │  Prompts  │
                             │ (DB CRUD) │  │ (AI APIs)  │  │ (suites)  │
                             └─────┬─────┘  └───────────┘  └───────────┘
                                   ▼
                            ┌──────────────┐
                            │   SQLite DB  │
                            │ (aiosqlite)  │
                            └──────────────┘

Request Flow

1. CLI/API receives request (run benchmark, view results)
2. Handler validates input via Pydantic schemas, delegates to service
3. Service orchestrates: loads prompt suite → calls adapters → stores results
4. Adapter wraps a specific AI provider API (OpenAI, Anthropic, generic OpenAI-compatible)
5. Repository persists benchmark runs & individual results to SQLite
6. Fingerprint service compares results across models to detect identity

---

🔍 Domain-Specific Knowledge

Model Fingerprinting Strategy

AI models have behavioral fingerprints that are hard to fake:

1. Identity probes — Ask "Who made you?" in various indirect ways
2. Capability tests — Tasks where models differ (code gen, math, languages)
3. Style analysis — Measure response length, vocabulary, formatting patterns
4. Edge cases — Known model-specific behaviors (refusal patterns, hallucination tendencies)
5. Latency profiling — Response time patterns can indicate underlying infrastructure
6. Token usage patterns — Different models tokenize differently

What Makes This Hard

• Resellers can add system prompts that say "You are Claude" to any model
• Simple identity questions are easy to fake with system prompts
• Need behavioral tests that can't be overridden by system prompts
• Models update over time, so fingerprints need periodic recalibration

Suspect API Testing

A "suspect API" is an endpoint that claims to serve Model X but might actually be Model Y. The system compares the suspect's responses against known baselines from verified APIs.

Suspect API Protocol Detection

The suspect provider in the adapter factory now defaults to Anthropic Messages protocol (not OpenAI). This is configured via:

• _DEFAULT_PROTOCOL map in src/adapters/factory.py — suspect → anthropic
• Can be overridden per-request via protocol field on ModelConfig schema
• Auth uses x-api-key header (Anthropic style), NOT Authorization: Bearer (OpenAI style)

Known Suspect: opuscode.pro

Field	Value
Base URL	https://opuscode.pro/api
Protocol	Anthropic Messages API
Endpoint	/v1/messages
Auth	x-api-key header
Available Models	Opus 4.6, Sonnet 4.5, Haiku 4.5 (their naming)
Default Model	Opus 4.6
Actual Model (tested)	claude-3-5-sonnet-20241022 (Claude 3.5 Sonnet)
Red Flags	Inconsistent knowledge cutoffs, mentions "proxy server", 14s avg latency

---

🧩 Multi-File Feature Notes

Feature: Benchmark Runner Pipeline

Files involved:

• src/services/benchmark_runner.py — orchestrates a full benchmark run
• src/adapters/base.py — defines ModelAdapter interface
• src/adapters/generic_adapter.py — OpenAI-compatible adapter for suspect APIs
• src/prompts/identity.py — identity probe prompt suite
• src/schemas/benchmark.py — request/response models
• src/repositories/result_repo.py — stores results

Flow:

benchmark_runner.run(config) →
  for each prompt_suite:
    for each model_adapter:
      adapter.complete(prompt) → response
      store result in DB
  return BenchmarkRunResult

Feature: Model Comparator

Files involved:

• src/services/model_comparator.py — compares two sets of benchmark results
• src/services/fingerprint.py — statistical fingerprinting algorithms
• src/repositories/result_repo.py — fetches stored results

Comparison dimensions:

• Response similarity (cosine similarity on embeddings or n-gram overlap)
• Latency distribution (mean, p50, p95, p99)
• Token usage patterns
• Refusal patterns (what does each model refuse to answer?)
• Formatting habits (markdown usage, list styles, code block formatting)

---

📅 Session Context

Temporary notes for the current development session. Clear after each major milestone.

• Session date: 2026-02-17
• Focus: Live suspect API testing & fraud analysis
• Notes:
  • Factory updated: suspect → Anthropic protocol by default
  • ModelConfig now has protocol field for OpenAI/Anthropic override
  • First benchmark run against opuscode.pro confirmed fraud: Claude 3.5 Sonnet served as Sonnet 4
  • README updated with no-API-key usage guide and red flags documentation
  • Server runs on port 8001 (via python -m uvicorn src.main:app --host 127.0.0.1 --port 8001)

---

🗺 Future Architecture Considerations

• Plugin system for custom prompt suites (load from YAML/JSON files)
• Webhook support to trigger benchmarks from CI/CD
• Result export to JSON/CSV for external analysis
• Embedding-based comparison using a local model for deeper similarity analysis
• Historical tracking to detect when a suspect API switches underlying models