# EvalHarness

A plugin-based LLM/agent evaluation harness. **Currently: the data layer only**
(dataset registration + lazy materialization cache + CLI). Model/eval/sandbox/
tool/skill layers are intentionally not built yet — they land one at a time.

## Features

- **Unified `Sample` schema** (pydantic): `input / choices / target / task_type /
  tools / sandbox / files / setup / metadata`. Raw dataset formats are
  unconstrained; each dataset plugin converts its records into `Sample`.
- **Dataset registration**: `@register_dataset(DatasetSpec(...))` decorator,
  import-time registration; `get_dataset(name)` returns a **lazy handle** —
  listing and registration never download anything.
- **Lazy materialization + content-addressed cache**: the first real use
  (iteration / `len` / indexing) triggers `download -> convert -> cache write`.
  Cache dir layout: `datasets/<benchmark>/<subset>_<split>[-<version>]-<hash6>/`
  — readable benchmark folders, one readable subdir per subset/split/version.
  A file lock prevents duplicate concurrent downloads; `tmp+rename` atomic
  writes prevent torn caches.
- **Two plugin styles**: pure `FieldSpec` declarative mapping when records are
  well-shaped (zero conversion code), or a custom `record_to_sample` function.
- **CLI**: `list / fetch (concurrent) / unload / stats / show`.
- **28 built-in datasets** registered against their official sources
  (see table below).

## Install

```bash
pip install .               # from this repo; core only (pydantic)
pip install '.[hub]'        # + HuggingFace `datasets`, needed for hub sources
```

For development: `pip install -e '.[hub]'`.

To build and install the wheel yourself:

```bash
pip install build
python -m build             # produces dist/evalharness-0.1.0-py3-none-any.whl + .tar.gz
pip install dist/evalharness-0.1.0-py3-none-any.whl
```

> Publishing to PyPI (`twine upload dist/*`) makes `pip install evalharness`
> work for everyone — note the name may be taken, so pick a unique
> distribution name (e.g. `evalharness-x`) in `pyproject.toml` before upload.

## Quick start

```bash
evalharness data list                 # list registered datasets (no network, no download)
evalharness data fetch gsm8k          # materialize: first run from source, then cache
evalharness data fetch gsm8k mmlu arc --workers 8   # concurrent prefetch
evalharness data unload gsm8k         # drop the cache entry (raw/ + samples + meta)
evalharness data stats cmmlu          # materialize + stats (count/lengths/answers/cache path)
evalharness data show gsm8k -n 2      # print the first N samples

# spec overrides: offline demo / local data / picking a subset
evalharness data fetch gsm8k --source examples/data/gsm8k_main_test.jsonl  # bundled tiny set
evalharness data fetch mmlu --subset anatomy        # one of MMLU's 57 subjects
evalharness data fetch bbh --subset word_sorting    # one of BBH's 27 subtasks
# note: with multiple names, --source/--split/--subset apply to ALL of them;
# run separately when you need per-benchmark overrides
```

Python API:

```python
from evalharness import get_dataset, list_datasets

ds = get_dataset('gsm8k')      # lazy handle: zero network/disk cost here
len(ds)                        # first use -> materialize (download->convert->cache)
for s in ds:
    print(s.input, s.target)

ds_sub = get_dataset('mmlu', subset='anatomy')   # spec override, separate cache
hard = ds.view([s for s in ds if len(s.input_text) > 100],
               lineage={'tool': 'length_filter'})   # derived data: same class, with lineage
```

Cache root: `~/.cache/evalharness/` (override with the `EVALHARNESS_CACHE`
environment variable). Layout example:

```
datasets/
├── gsm8k/
│   ├── main_test-e06f82/     <- official openai/gsm8k
│   │   ├── raw/              # NATIVE source data, byte-exact as downloaded
│   │   ├── samples.jsonl     # converted unified Sample view
│   │   └── meta.json         # spec + provenance + raw file list
│   └── main_test-daafcc/     <- local demo via --source
├── mmlu/
│   ├── all_test-1a3b4c/
│   └── anatomy_test-9e666f/
└── bbh/
    └── boolean_expressions_test-…    (one dir per subtask)

.raw/<repo-hash>/              shared download blobs (ModelScope sources);
                               cache entries hardlink from here, so multi-
                               subset mirrors download only once
```

> Every cache entry is **self-contained and preserves native data**: `raw/`
> holds the original file(s) exactly as downloaded (never converted);
> `samples.jsonl` is the derived unified view. HF-hub sources keep an exact
> pre-conversion record dump in `raw/records.jsonl`. Rebuild any entry with
> `evalharness data fetch <name> --force`.

> Why the 6-char hash suffix: two variants with the same subset/split but
> different sources (`--source`) or params would otherwise collide and serve
> stale data. The short hash keeps them apart while staying readable.

## Built-in datasets (28, official sources)

| Family | Datasets (source) |
|---|---|
| Math | gsm8k (openai/gsm8k), competition_math (EleutherAI/hendrycks_math), aime24 (HuggingFaceH4), aime25 (yentinglin), aime26*, hmmt26*, imo_answerbench* |
| Knowledge / MCQ | mmlu (cais/mmlu), mmlu_pro (TIGER-Lab), cmmlu (haonan-li), gpqa_diamond (Idavidrein/gpqa), arc (allenai/ai2_arc), hellaswag, winogrande |
| QA | trivia_qa (mandarjoshi), drop (ucinlp), simple_qa (mirror of OpenAI's CSV), hle (cais/hle), bbh (lukaemon/bbh) |
| Long context | longbench_v2 (THUDM), openai_mrcr (openai-mirror) |
| Coding | humaneval (openai), bigcodebench (bigcode), live_code_bench (livecodebench) |
| Agent / tools | swe_bench_verified (princeton-nlp), tau2_bench (official GitHub), bfcl_v3 (official GitHub), general_fc (evalscope-native) |

\* aime26 / hmmt26 / imo_answerbench have no official standalone release and
use community curations (evalscope); simple_qa's official artifact is the CSV
in `openai/simple-evals` (HF is a mirror); tau2_bench / bfcl_v3 are officially
released on GitHub — clone and point `--source` at the local files.

## Adding a dataset

Drop a single-file plugin into `evalharness/data/datasets/` — auto-discovered,
no central file to edit.

Well-shaped records (column names map directly) — pure declaration:

```python
# evalharness/data/datasets/cmmlu.py
@register_dataset(DatasetSpec(name='cmmlu', source='haonan-li/cmmlu', split='test', task_type='mcq'))
def cmmlu():
    return FieldSpec(input='question', choices='choices', target='answer', metadata=['category'])
```

Custom conversion — return a `record -> Sample` function:

```python
@register_dataset(DatasetSpec(name='gsm8k', source='openai/gsm8k', subset='main',
                              split='test', task_type='math'))
def gsm8k():
    def to_sample(record):
        parts = record['answer'].split('####')
        return Sample(input=record['question'], target=parts.pop().strip())
    return to_sample
```

Sources supported: local `.jsonl/.json/.csv/.tsv` files; local directories
(probed as `{subset}_{split}.jsonl` etc.); HF hub dataset ids (requires the
`hub` extra, imported lazily); official GitHub releases (clone, then
`--source` the local path).

Conventions: `choices` holds option *contents*; `target` is a **letter** for
MCQ (e.g. `'B'`) and text otherwise (use `List[str]` for multiple gold
answers); long contexts/passages go to `metadata`, not `input`; `id` is
assigned sequentially at materialize time when absent.

## Design decisions

The data layer mirrors conclusions from a close reading of seven evaluation
frameworks (evalscope, lm-evaluation-harness, inspect_ai, deepeval, VLMEvalKit,
harbor, deepseek-harness); see `../README.md` for the full analysis:

- **`Sample` base + one `record_to_sample` per plugin** (evalscope/inspect_ai):
  raw formats vary wildly; the unified format only exists after conversion,
  and everything downstream sees only `Sample`.
- **Datasets as first-class citizens** (inspect_ai): a `Dataset` from
  `get_dataset()` can be filtered/synthesized/exported freely — it is not
  welded into an evaluation recipe.
- **Registration is cheap, materialization pays**: the registry holds only
  metadata + conversion recipes; `list` never touches the network.
- **Config-sensitive cache keys** (evalscope): a cache hit is always correct;
  no invalidation logic exists.
- **Sandbox/tool fields on `Sample` are declarations only**
  (`sandbox/files/setup/tools`): the data layer never executes; a future
  sandbox layer will materialize them.
- **Data load/unload vs environment load/unload are different layers**:
  `fetch`/`unload` move bytes only (raw files + converted samples). Heavy
  execution environments (e.g. the ~1GB-per-instance `sweb.eval.*` images
  declared by swe_bench_verified) are pulled lazily *at eval time* by the
  sandbox layer — never at data-fetch time — and their removal is
  refcounted there because docker layers are shared across instances and
  benchmarks. `DatasetSpec.requires` (e.g. `['docker']`) is the declaration
  the sandbox/deploy layer reads.

## Layout

```
EvalHarness/
├── pyproject.toml
├── evalharness/
│   ├── cli.py                     # argparse CLI: data list/fetch(concurrent)/stats/show
│   └── data/
│       ├── sample.py              # Sample / ChatMessage / SandboxSpec / ToolInfo
│       ├── spec.py                # DatasetSpec (metadata) / FieldSpec (field mapping)
│       ├── registry.py            # Registry + @register_dataset + get_dataset
│       ├── loader.py              # raw record loading (local file/dir/hub)
│       ├── dataset.py             # Dataset: lazy materialize + cache + derived views
│       └── datasets/              # 28 built-in single-file dataset plugins
├── examples/
│   └── data/                      # offline demo subsets (gsm8k/cmmlu, 5 rows each)
```

## Roadmap (not built yet, one layer at a time)

- [ ] Model layer (ModelAdapter: unified URL-based invocation)
- [ ] Evaluation engine (evaluator + scorer/metric)
- [ ] Sandbox layer (materialize `Sample.sandbox`: lazy per-instance image
      pull, refcounted image unload, container lifecycle; `requires` gating)
- [ ] Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
- [ ] Skill layer (full evaluation pipelines as composable skills)
- [ ] Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
- [ ] Web/API interface