EvalHarness

A plugin-based LLM/agent evaluation harness. Currently: the data layer only (dataset registration + lazy materialization cache + CLI). Model/eval/sandbox/ tool/skill layers are intentionally not built yet — they land one at a time.

Features

Unified Sample schema (pydantic): input / choices / target / task_type / tools / sandbox / files / setup / metadata. Raw dataset formats are unconstrained; each dataset plugin converts its records into Sample.
Dataset registration: @register_dataset(DatasetSpec(...)) decorator, import-time registration; get_dataset(name) returns a lazy handle — listing and registration never download anything.
Lazy materialization + content-addressed cache: the first real use (iteration / len / indexing) triggers download -> convert -> cache write. Cache dir layout: datasets/<benchmark>/<subset>_<split>[-<version>]-<hash6>/ — readable benchmark folders, one readable subdir per subset/split/version. A file lock prevents duplicate concurrent downloads; tmp+rename atomic writes prevent torn caches.
Two plugin styles: pure FieldSpec declarative mapping when records are well-shaped (zero conversion code), or a custom record_to_sample function.
CLI: list / fetch (concurrent) / unload / stats / show.
28 built-in datasets registered against their official sources (see table below).

Install

pip install .               # from this repo; core only (pydantic)
pip install '.[hub]'        # + HuggingFace `datasets`, needed for hub sources

For development: pip install -e '.[hub]'.

To build and install the wheel yourself:

pip install build
python -m build             # produces dist/evalharness-0.1.0-py3-none-any.whl + .tar.gz
pip install dist/evalharness-0.1.0-py3-none-any.whl

Publishing to PyPI (twine upload dist/*) makes pip install evalharness work for everyone — note the name may be taken, so pick a unique distribution name (e.g. evalharness-x) in pyproject.toml before upload.

Quick start

evalharness data list                 # list registered datasets (no network, no download)
evalharness data fetch gsm8k          # materialize: first run from source, then cache
evalharness data fetch gsm8k mmlu arc --workers 8   # concurrent prefetch
evalharness data unload gsm8k         # drop the cache entry (raw/ + samples + meta)
evalharness data stats cmmlu          # materialize + stats (count/lengths/answers/cache path)
evalharness data show gsm8k -n 2      # print the first N samples

# spec overrides: offline demo / local data / picking a subset
evalharness data fetch gsm8k --source examples/data/gsm8k_main_test.jsonl  # bundled tiny set
evalharness data fetch mmlu --subset anatomy        # one of MMLU's 57 subjects
evalharness data fetch bbh --subset word_sorting    # one of BBH's 27 subtasks
# note: with multiple names, --source/--split/--subset apply to ALL of them;
# run separately when you need per-benchmark overrides

Python API:

from evalharness import get_dataset, list_datasets

ds = get_dataset('gsm8k')      # lazy handle: zero network/disk cost here
len(ds)                        # first use -> materialize (download->convert->cache)
for s in ds:
    print(s.input, s.target)

ds_sub = get_dataset('mmlu', subset='anatomy')   # spec override, separate cache
hard = ds.view([s for s in ds if len(s.input_text) > 100],
               lineage={'tool': 'length_filter'})   # derived data: same class, with lineage

Cache root: ~/.cache/evalharness/ (override with the EVALHARNESS_CACHE environment variable). Layout example:

datasets/
├── gsm8k/
│   ├── main_test-e06f82/     <- official openai/gsm8k
│   │   ├── raw/              # NATIVE source data, byte-exact as downloaded
│   │   ├── samples.jsonl     # converted unified Sample view
│   │   └── meta.json         # spec + provenance + raw file list
│   └── main_test-daafcc/     <- local demo via --source
├── mmlu/
│   ├── all_test-1a3b4c/
│   └── anatomy_test-9e666f/
└── bbh/
    └── boolean_expressions_test-…    (one dir per subtask)

.raw/<repo-hash>/              shared download blobs (ModelScope sources);
                               cache entries hardlink from here, so multi-
                               subset mirrors download only once

Every cache entry is self-contained and preserves native data: raw/ holds the original file(s) exactly as downloaded (never converted); samples.jsonl is the derived unified view. HF-hub sources keep an exact pre-conversion record dump in raw/records.jsonl. Rebuild any entry with evalharness data fetch <name> --force.

Why the 6-char hash suffix: two variants with the same subset/split but different sources (--source) or params would otherwise collide and serve stale data. The short hash keeps them apart while staying readable.

Built-in datasets (28, official sources)

Family	Datasets (source)
Math	gsm8k (openai/gsm8k), competition_math (EleutherAI/hendrycks_math), aime24 (HuggingFaceH4), aime25 (yentinglin), aime26, hmmt26, imo_answerbench*
Knowledge / MCQ	mmlu (cais/mmlu), mmlu_pro (TIGER-Lab), cmmlu (haonan-li), gpqa_diamond (Idavidrein/gpqa), arc (allenai/ai2_arc), hellaswag, winogrande
QA	trivia_qa (mandarjoshi), drop (ucinlp), simple_qa (mirror of OpenAI's CSV), hle (cais/hle), bbh (lukaemon/bbh)
Long context	longbench_v2 (THUDM), openai_mrcr (openai-mirror)
Coding	humaneval (openai), bigcodebench (bigcode), live_code_bench (livecodebench)
Agent / tools	swe_bench_verified (princeton-nlp), tau2_bench (official GitHub), bfcl_v3 (official GitHub), general_fc (evalscope-native)

* aime26 / hmmt26 / imo_answerbench have no official standalone release and use community curations (evalscope); simple_qa's official artifact is the CSV in openai/simple-evals (HF is a mirror); tau2_bench / bfcl_v3 are officially released on GitHub — clone and point --source at the local files.

Adding a dataset

Drop a single-file plugin into evalharness/data/datasets/ — auto-discovered, no central file to edit.

Well-shaped records (column names map directly) — pure declaration:

# evalharness/data/datasets/cmmlu.py
@register_dataset(DatasetSpec(name='cmmlu', source='haonan-li/cmmlu', split='test', task_type='mcq'))
def cmmlu():
    return FieldSpec(input='question', choices='choices', target='answer', metadata=['category'])

Custom conversion — return a record -> Sample function:

@register_dataset(DatasetSpec(name='gsm8k', source='openai/gsm8k', subset='main',
                              split='test', task_type='math'))
def gsm8k():
    def to_sample(record):
        parts = record['answer'].split('####')
        return Sample(input=record['question'], target=parts.pop().strip())
    return to_sample

Sources supported: local .jsonl/.json/.csv/.tsv files; local directories (probed as {subset}_{split}.jsonl etc.); HF hub dataset ids (requires the hub extra, imported lazily); official GitHub releases (clone, then --source the local path).

Conventions: choices holds option contents; target is a letter for MCQ (e.g. 'B') and text otherwise (use List[str] for multiple gold answers); long contexts/passages go to metadata, not input; id is assigned sequentially at materialize time when absent.

Design decisions

The data layer mirrors conclusions from a close reading of seven evaluation frameworks (evalscope, lm-evaluation-harness, inspect_ai, deepeval, VLMEvalKit, harbor, deepseek-harness); see ../README.md for the full analysis:

Sample base + one record_to_sample per plugin (evalscope/inspect_ai): raw formats vary wildly; the unified format only exists after conversion, and everything downstream sees only Sample.
Datasets as first-class citizens (inspect_ai): a Dataset from get_dataset() can be filtered/synthesized/exported freely — it is not welded into an evaluation recipe.
Registration is cheap, materialization pays: the registry holds only metadata + conversion recipes; list never touches the network.
Config-sensitive cache keys (evalscope): a cache hit is always correct; no invalidation logic exists.
Sandbox/tool fields on Sample are declarations only (sandbox/files/setup/tools): the data layer never executes; a future sandbox layer will materialize them.
Data load/unload vs environment load/unload are different layers: fetch/unload move bytes only (raw files + converted samples). Heavy execution environments (e.g. the ~1GB-per-instance sweb.eval.* images declared by swe_bench_verified) are pulled lazily at eval time by the sandbox layer — never at data-fetch time — and their removal is refcounted there because docker layers are shared across instances and benchmarks. DatasetSpec.requires (e.g. ['docker']) is the declaration the sandbox/deploy layer reads.

Layout

EvalHarness/
├── pyproject.toml
├── evalharness/
│   ├── cli.py                     # argparse CLI: data list/fetch(concurrent)/stats/show
│   └── data/
│       ├── sample.py              # Sample / ChatMessage / SandboxSpec / ToolInfo
│       ├── spec.py                # DatasetSpec (metadata) / FieldSpec (field mapping)
│       ├── registry.py            # Registry + @register_dataset + get_dataset
│       ├── loader.py              # raw record loading (local file/dir/hub)
│       ├── dataset.py             # Dataset: lazy materialize + cache + derived views
│       └── datasets/              # 28 built-in single-file dataset plugins
├── examples/
│   └── data/                      # offline demo subsets (gsm8k/cmmlu, 5 rows each)

Roadmap (not built yet, one layer at a time)

Model layer (ModelAdapter: unified URL-based invocation)
Evaluation engine (evaluator + scorer/metric)
Sandbox layer (materialize Sample.sandbox: lazy per-instance image pull, refcounted image unload, container lifecycle; requires gating)
Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
Skill layer (full evaluation pipelines as composable skills)
Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
Web/API interface