EvalHarness/README.md

EvalHarness

A plugin-based LLM/agent evaluation harness. Currently: data + evaluation + model + sandbox + agent-driver layers (datasets/eval-recipes/model-adapters/ deployers/sandboxes/environments as plugins, lazy materialization cache, official-aligned scorers, async generation, sandboxed code execution, agent message pump, report artifacts & console visualization). Tool/skill layers land next.

Features

• Unified Sample schema (pydantic): input / choices / target / task_type / tools / sandbox / files / setup / metadata. Raw dataset formats are unconstrained; each dataset plugin converts its records into Sample.
• Dataset registration: @register_dataset(DatasetSpec(...)) decorator, import-time registration; get_dataset(name) returns a lazy handle — listing and registration never download anything.
• Lazy materialization + content-addressed cache: the first real use (iteration / len / indexing) triggers download -> convert -> cache write. Cache dir layout: datasets/<benchmark>/<subset>_<split>[-<version>]-<hash6>/ — readable benchmark folders, one readable subdir per subset/split/version. A file lock prevents duplicate concurrent downloads; tmp+rename atomic writes prevent torn caches.
• Two plugin styles: pure FieldSpec declarative mapping when records are well-shaped (zero conversion code), or a custom record_to_sample function.
• CLI: list / fetch (concurrent) / unload / stats / show.
• 28 built-in datasets registered against their official sources (see table below).

Install

pip install .                  # core + all light deps (datasets/pyarrow/sympy/numpy/scipy):
                               # every benchmark scores officially, nothing conflicts
pip install '.[bfcl]'          # + heavy official bfcl ast_checker (5 cloud SDKs + qwen-agent;
                               # only for official-parity BFCL scoring -- native default works)

Heavy execution environments never enter the venv: code benchmarks run in docker sandboxes (python:3.11-slim, the official bigcodebench image, sweb.eval.* per-instance images via sandbox prefetch).

Quick start

evalharness data list                 # list registered datasets (no network, no download)
evalharness data fetch gsm8k          # materialize: first run from source, then cache
evalharness data fetch gsm8k mmlu arc --workers 8   # concurrent prefetch
evalharness data unload gsm8k         # drop the cache entry (raw/ + samples + meta)
evalharness data stats cmmlu          # materialize + stats (count/lengths/answers/cache path)
evalharness data show gsm8k -n 2      # print the first N samples

# spec overrides: offline demo / local data / picking a subset
evalharness data fetch gsm8k --source examples/data/gsm8k_main_test.jsonl  # bundled tiny set
evalharness data fetch mmlu --subset anatomy        # one of MMLU's 57 subjects
evalharness data fetch bbh --subset word_sorting    # one of BBH's 27 subtasks
# note: with multiple names, --source/--split/--subset apply to ALL of them;
# run separately when you need per-benchmark overrides

Python API:

from evalharness import get_dataset, list_datasets

ds = get_dataset('gsm8k')      # lazy handle: zero network/disk cost here
len(ds)                        # first use -> materialize (download->convert->cache)
for s in ds:
    print(s.input, s.target)

ds_sub = get_dataset('mmlu', subset='anatomy')   # spec override, separate cache
hard = ds.view([s for s in ds if len(s.input_text) > 100],
               lineage={'tool': 'length_filter'})   # derived data: same class, with lineage

Cache root: ~/.cache/evalharness/ (override with the EVALHARNESS_CACHE environment variable). Layout example:

datasets/
├── gsm8k/
│   ├── main_test-e06f82/     <- official openai/gsm8k
│   │   ├── raw/              # NATIVE source data, byte-exact as downloaded
│   │   ├── samples.jsonl     # converted unified Sample view
│   │   └── meta.json         # spec + provenance + raw file list
│   └── main_test-daafcc/     <- local demo via --source
├── mmlu/
│   ├── all_test-1a3b4c/
│   └── anatomy_test-9e666f/
└── bbh/
    └── boolean_expressions_test-…    (one dir per subtask)

.raw/<repo-hash>/              shared download blobs (ModelScope sources);
                               cache entries hardlink from here, so multi-
                               subset mirrors download only once

Every cache entry is self-contained and preserves native data: raw/ holds the original file(s) exactly as downloaded (never converted); samples.jsonl is the derived unified view. HF-hub sources keep an exact pre-conversion record dump in raw/records.jsonl. Rebuild any entry with evalharness data fetch <name> --force.

Why the 6-char hash suffix: two variants with the same subset/split but different sources (--source) or params would otherwise collide and serve stale data. The short hash keeps them apart while staying readable.

Evaluation layer

evalharness eval list                          # 28 recipes, per benchmark
evalharness eval run gsm8k preds.jsonl --model mymodel --out gsm8k.report.json
evalharness viz show gsm8k.report.json         # console table
evalharness viz show r1.json r2.json --style md_compare

from evalharness import get_dataset
from evalharness.eval import evaluate, get_eval
from evalharness.viz import render

ds = get_dataset('gsm8k')
report = evaluate(ds, predictions, model='mymodel')   # recipe auto-resolved
report.save('gsm8k.report.json')
print(render(report, style='text'))

Pipeline: extract -> score -> aggregate, each stage a registered plugin:

• Extractors (eval/extractor.py): math_boxed / mcq_letter / answer_phrase / answer_spans / gsm8k_hash / last_number / code_block / quoted_list / identity + cascades (first stage that succeeds wins).
• Scorers (eval/scorer.py): text compare (exact / math_equal / em_f1 / alias_match — official implementations), llm_judge (labels-> scores contract, wired to a ModelAdapter via evaluate(judge=...)), execution / env_reward (raise LayerNotReady until sandbox/agent land).
• Aggregators (eval/aggregator.py): mean / pass_at_k / grouped_avg / weighted_group_avg / binned_avg / simpleqa_official (is_given_attempted + accuracy_given_attempted).
• Recipes (eval/recipes/): one per benchmark, 5-20 lines each — pure bindings of the primitives above, @register_eval('gsm8k').

Per-sample results keep raw_prediction + extraction note + score details; extraction_failure_rate is reported as a health metric. Changing a recipe and re-running evaluate() re-scores the same predictions — the model is never re-queried.

Model layer (calling + deploying, separate plugins on purpose)

# generate + score in one command (async, concurrent)
evalharness eval run gsm8k --model mock:boxed --limit 100          # offline pipeline check
evalharness eval run gsm8k --model openai/http://gpu03:8000/v1?qwen3-8b
evalharness eval run hle   --model openai/...?qwen3-8b --judge openai/...?gpt-4o
evalharness eval run bfcl_v3 --model mock:fc --env bfcl_mock     # agent pump
# future: --model deploy:vllm/qwen3-8b  (Deployer pulls a pinned docker env)

Model spec grammar (plain strings):

spec	meaning
mock / mock:boxed / mock:tool	offline adapter (echo / oracle-boxed / tool-call)
openai/<api_base>?<model_id>	any OpenAI-protocol endpoint: vllm, sglang, lmdeploy, ollama, cloud APIs
deploy:<engine>/<model>	Deployer resolves the endpoint first (vllm/sglang: pinned docker image; external: models.yaml)

Design:

• ModelAdapter = how to call (protocol). All adapters are async and return structured ModelOutput(text, tool_calls, usage) — the hinge the future agent loops hang on; single-turn recipes just read .text.
• Deployer = how to run (environment, separate lifecycle). Docker images are pinned per model in models.yaml, so vllm:v0.9.2 and vllm:v0.6.6 stacks coexist on one machine; external connects to existing endpoints.
• Async boundary = waiting on the model: run_eval() fans out calls with a semaphore (default 32), collects raws + per-sample usage, then hands them to the synchronous evaluate(). Data/scoring stay sync (fast, CPU/disk).

Sandbox layer (environments for BOTH eval execution and model serving)

One docker implementation, two faces:

• exec() runs untrusted model-generated code hard-isolated: --network none, cpu/mem/pids caps, read-only rootfs, tmpfs /tmp. Host file sharing via bind mounts (mounts={'/out': host_dir}) — artifacts land on the host directly, no docker cp.
• serve() trusted engine containers (vllm/sglang) with network + GPU passthrough; consumed by the model Deployer through the same layer.
• Lifecycle: refcounted acquire()/release(); containers stop+rm at refcount 0 or process exit (atexit); images are never auto-deleted — re-acquire re-runs the local image instantly.

from evalharness.sandbox import get_sandbox
r = get_sandbox('docker').exec({'main.py': 'print(42)'})   # or 'local' for dev

Agent evaluation driver (message pump, not a thinking framework)

We EVALUATE agents: the model under test thinks; we only execute its tool_calls against Environment plugins and feed observations back.

from evalharness.model import run_eval
report = await run_eval(ds, 'openai/http://gpu03:8000/v1?qwen3-8b',
                        env='bfcl_mock')        # agent pump per sample
# CLI: evalharness eval run bfcl_v3 --model mock:fc --env bfcl_mock

• agent/loop.py::drive() — pump until no more calls / env done / max_turns; records a full Trajectory (messages, per-turn usage, env final state) into SampleResult.trajectory / env_state.
• agent/envs/bfcl_mock.py — BFCL official-style: record call sequence, compare against ground_truth (env_reward scorer), incl. irrelevance categories (correct = call nothing). tau2 / swe envs land later on the same Environment contract.
• Single-turn is the degenerate case: no env -> one generate, done.

Built-in datasets (28, official sources)

Family	Datasets (source)
Math	gsm8k (openai/gsm8k), competition_math (EleutherAI/hendrycks_math), aime24 (HuggingFaceH4), aime25 (yentinglin), aime26*, hmmt26*, imo_answerbench*
Knowledge / MCQ	mmlu (cais/mmlu), mmlu_pro (TIGER-Lab), cmmlu (haonan-li), gpqa_diamond (Idavidrein/gpqa), arc (allenai/ai2_arc), hellaswag, winogrande
QA	trivia_qa (mandarjoshi), drop (ucinlp), simple_qa (mirror of OpenAI's CSV), hle (cais/hle), bbh (lukaemon/bbh)
Long context	longbench_v2 (THUDM), openai_mrcr (openai-mirror)
Coding	humaneval (openai), bigcodebench (bigcode), live_code_bench (livecodebench)
Agent / tools	swe_bench_verified (princeton-nlp), tau2_bench (official GitHub), bfcl_v3 (official GitHub), general_fc (evalscope-native)

* aime26 / hmmt26 / imo_answerbench have no official standalone release and use community curations (evalscope); simple_qa's official artifact is the CSV in openai/simple-evals (HF is a mirror); tau2_bench / bfcl_v3 are officially released on GitHub — clone and point --source at the local files.

Adding a dataset

Drop a single-file plugin into evalharness/data/datasets/ — auto-discovered, no central file to edit.

Well-shaped records (column names map directly) — pure declaration:

# evalharness/data/datasets/cmmlu.py
@register_dataset(DatasetSpec(name='cmmlu', source='haonan-li/cmmlu', split='test', task_type='mcq'))
def cmmlu():
    return FieldSpec(input='question', choices='choices', target='answer', metadata=['category'])

Custom conversion — return a record -> Sample function:

@register_dataset(DatasetSpec(name='gsm8k', source='openai/gsm8k', subset='main',
                              split='test', task_type='math'))
def gsm8k():
    def to_sample(record):
        parts = record['answer'].split('####')
        return Sample(input=record['question'], target=parts.pop().strip())
    return to_sample

Sources supported: local .jsonl/.json/.csv/.tsv files; local directories (probed as {subset}_{split}.jsonl etc.); HF hub dataset ids (requires the hub extra, imported lazily); official GitHub releases (clone, then --source the local path).

Conventions: choices holds option contents; target is a letter for MCQ (e.g. 'B') and text otherwise (use List[str] for multiple gold answers); long contexts/passages go to metadata, not input; id is assigned sequentially at materialize time when absent.

Design decisions

The data layer mirrors conclusions from a close reading of seven evaluation frameworks (evalscope, lm-evaluation-harness, inspect_ai, deepeval, VLMEvalKit, harbor, deepseek-harness); see ../README.md for the full analysis:

• Data layer vs eval layer: strict separation (unlike evalscope's DataAdapter, which welds record conversion + extraction + scoring + prompt into one class). The data layer answers what is the question/target; the eval layer answers how to judge a response. Consequences: predictions can be re-scored under a new recipe without re-running the model; extraction rules are shared primitives (math_boxed, mcq_letter, ...) instead of per-benchmark copies; official grader fixes (e.g. DROP's Hungarian alignment) land in one scorer and apply to every recipe.
• Sample base + one record_to_sample per plugin (evalscope/inspect_ai): raw formats vary wildly; the unified format only exists after conversion, and everything downstream sees only Sample.
• Datasets as first-class citizens (inspect_ai): a Dataset from get_dataset() can be filtered/synthesized/exported freely — it is not welded into an evaluation recipe.
• Registration is cheap, materialization pays: the registry holds only metadata + conversion recipes; list never touches the network.
• Config-sensitive cache keys (evalscope): a cache hit is always correct; no invalidation logic exists.
• Raw predictions are immutable artifacts: every SampleResult keeps raw_prediction; extraction/score details are re-derivable. Failed extractions are surfaced (extraction_failure_rate), never silently zeroed.
• Official grader parity: scorers replicate official logic where it exists (PRM800K math equivalence; DROP's answer_to_bags + linear_sum_assignment; SimpleQA's A/B/C judge with NOT_ATTEMPTED fallback and accuracy_given_attempted). Official test cases anchor tests/.
• Sandbox/tool fields on Sample are declarations only (sandbox/files/setup/tools): the data layer never executes; a future sandbox layer will materialize them.
• Data load/unload vs environment load/unload are different layers: fetch/unload move bytes only (raw files + converted samples). Heavy execution environments (e.g. the ~1GB-per-instance sweb.eval.* images declared by swe_bench_verified) are pulled lazily at eval time by the sandbox layer — never at data-fetch time — and their removal is refcounted there because docker layers are shared across instances and benchmarks. DatasetSpec.requires (e.g. ['docker']) is the declaration the sandbox/deploy layer reads.

Layout

EvalHarness/
├── pyproject.toml
├── evalharness/
│   ├── cli.py                     # CLI: data list/fetch/unload/stats/show
│   │                              #      eval list/run, viz show
│   ├── data/                      # ---- data layer ----
│   │   ├── sample.py              # Sample / ChatMessage / SandboxSpec / ToolInfo
│   │   ├── spec.py                # DatasetSpec (metadata) / FieldSpec (field mapping)
│   │   ├── registry.py            # Registry + @register_dataset + get_dataset
│   │   ├── loader.py              # raw loading (local/HF/ModelScope native)
│   │   ├── dataset.py             # Dataset: lazy materialize + cache + derived views
│   │   └── datasets/              # 28 built-in single-file dataset plugins
│   ├── model/                     # ---- model layer ----
│   │   ├── output.py              # ModelOutput/ToolCall/Usage (agent hinge)
│   │   ├── adapter.py             # @register_adapter: openai_compatible / mock
│   │   ├── deployer.py            # @register_deployer: vllm / sglang / external
│   │   └── runner.py              # async run_eval(): generate -> evaluate
│   ├── sandbox/                   # ---- environment layer ----
│   │   ├── base.py                # Sandbox iface + refcounted acquire/release + atexit
│   │   ├── docker.py              # exec (isolated) + serve (engines) one impl
│   │   └── local.py               # dev-only, no isolation
│   ├── agent/                     # ---- agent evaluation driver ----
│   │   ├── loop.py                # drive(): message pump + Trajectory
│   │   └── envs/bfcl_mock.py      # BFCL official-style env (tau2/swe later)
│   ├── eval/                      # ---- evaluation layer ----
│   │   ├── record.py              # SampleResult / EvalReport artifacts
│   │   ├── extractor.py           # answer-extraction primitives (+cascades)
│   │   ├── scorer.py              # scoring primitives (official implementations)
│   │   ├── math_grader.py         # PRM800K sympy equivalence (optional [math])
│   │   ├── aggregator.py          # mean/pass@k/grouped/binned/simpleqa_official
│   │   ├── recipe.py              # EvalRecipe bindings + @register_eval
│   │   ├── runner.py              # evaluate(dataset, predictions) -> EvalReport
│   │   └── recipes/               # 28 per-benchmark recipes
│   └── viz/                       # ---- visualization (report consumer) ----
│       └── renderers/text.py      # text / md / md_compare / radar / errors
├── examples/
│   └── data/                      # offline demo subsets (gsm8k/cmmlu, 5 rows each)
└── tests/
    └── test_eval.py               # official-anchor regression tests

Roadmap (not built yet, one layer at a time)

• Data layer (28 dataset plugins, lazy cache, native HF/ModelScope loaders)
• Evaluation layer (extract/score/aggregate plugins, official scorers, recipes)
• Visualization (console/markdown renderers over report artifacts)
• Model layer (async ModelAdapter openai_compatible+mock, ModelOutput with tool_calls, Deployer registry vllm/sglang/external + models.yaml env pinning, run_eval generate->score)
• Sandbox layer (docker exec hard-isolation + serve environments, refcounted acquire/release, atexit teardown, images kept, bind-mount host sharing; Deployer now consumes it)
• Agent evaluation driver (message pump + Trajectory + bfcl_mock env with official call-sequence scoring; tau2/swe envs pending)
• Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
• Skill layer (full evaluation pipelines as composable skills)
• tau2 / swe-bench environments (user simulator; per-instance sweb.* images)
• Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
• Web/API interface
• Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
• Skill layer (full evaluation pipelines as composable skills)
• Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
• Web/API interface