221 lines
10 KiB
Markdown
221 lines
10 KiB
Markdown
# EvalHarness
|
|
|
|
A plugin-based LLM/agent evaluation harness. **Currently: the data layer only**
|
|
(dataset registration + lazy materialization cache + CLI). Model/eval/sandbox/
|
|
tool/skill layers are intentionally not built yet — they land one at a time.
|
|
|
|
## Features
|
|
|
|
- **Unified `Sample` schema** (pydantic): `input / choices / target / task_type /
|
|
tools / sandbox / files / setup / metadata`. Raw dataset formats are
|
|
unconstrained; each dataset plugin converts its records into `Sample`.
|
|
- **Dataset registration**: `@register_dataset(DatasetSpec(...))` decorator,
|
|
import-time registration; `get_dataset(name)` returns a **lazy handle** —
|
|
listing and registration never download anything.
|
|
- **Lazy materialization + content-addressed cache**: the first real use
|
|
(iteration / `len` / indexing) triggers `download -> convert -> cache write`.
|
|
Cache dir layout: `datasets/<benchmark>/<subset>_<split>[-<version>]-<hash6>/`
|
|
— readable benchmark folders, one readable subdir per subset/split/version.
|
|
A file lock prevents duplicate concurrent downloads; `tmp+rename` atomic
|
|
writes prevent torn caches.
|
|
- **Two plugin styles**: pure `FieldSpec` declarative mapping when records are
|
|
well-shaped (zero conversion code), or a custom `record_to_sample` function.
|
|
- **CLI**: `list / fetch (concurrent) / unload / stats / show`.
|
|
- **28 built-in datasets** registered against their official sources
|
|
(see table below).
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install . # from this repo; core only (pydantic)
|
|
pip install '.[hub]' # + HuggingFace `datasets`, needed for hub sources
|
|
```
|
|
|
|
For development: `pip install -e '.[hub]'`.
|
|
|
|
To build and install the wheel yourself:
|
|
|
|
```bash
|
|
pip install build
|
|
python -m build # produces dist/evalharness-0.1.0-py3-none-any.whl + .tar.gz
|
|
pip install dist/evalharness-0.1.0-py3-none-any.whl
|
|
```
|
|
|
|
> Publishing to PyPI (`twine upload dist/*`) makes `pip install evalharness`
|
|
> work for everyone — note the name may be taken, so pick a unique
|
|
> distribution name (e.g. `evalharness-x`) in `pyproject.toml` before upload.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
evalharness data list # list registered datasets (no network, no download)
|
|
evalharness data fetch gsm8k # materialize: first run from source, then cache
|
|
evalharness data fetch gsm8k mmlu arc --workers 8 # concurrent prefetch
|
|
evalharness data unload gsm8k # drop the cache entry (raw/ + samples + meta)
|
|
evalharness data stats cmmlu # materialize + stats (count/lengths/answers/cache path)
|
|
evalharness data show gsm8k -n 2 # print the first N samples
|
|
|
|
# spec overrides: offline demo / local data / picking a subset
|
|
evalharness data fetch gsm8k --source examples/data/gsm8k_main_test.jsonl # bundled tiny set
|
|
evalharness data fetch mmlu --subset anatomy # one of MMLU's 57 subjects
|
|
evalharness data fetch bbh --subset word_sorting # one of BBH's 27 subtasks
|
|
# note: with multiple names, --source/--split/--subset apply to ALL of them;
|
|
# run separately when you need per-benchmark overrides
|
|
```
|
|
|
|
Python API:
|
|
|
|
```python
|
|
from evalharness import get_dataset, list_datasets
|
|
|
|
ds = get_dataset('gsm8k') # lazy handle: zero network/disk cost here
|
|
len(ds) # first use -> materialize (download->convert->cache)
|
|
for s in ds:
|
|
print(s.input, s.target)
|
|
|
|
ds_sub = get_dataset('mmlu', subset='anatomy') # spec override, separate cache
|
|
hard = ds.view([s for s in ds if len(s.input_text) > 100],
|
|
lineage={'tool': 'length_filter'}) # derived data: same class, with lineage
|
|
```
|
|
|
|
Cache root: `~/.cache/evalharness/` (override with the `EVALHARNESS_CACHE`
|
|
environment variable). Layout example:
|
|
|
|
```
|
|
datasets/
|
|
├── gsm8k/
|
|
│ ├── main_test-e06f82/ <- official openai/gsm8k
|
|
│ │ ├── raw/ # NATIVE source data, byte-exact as downloaded
|
|
│ │ ├── samples.jsonl # converted unified Sample view
|
|
│ │ └── meta.json # spec + provenance + raw file list
|
|
│ └── main_test-daafcc/ <- local demo via --source
|
|
├── mmlu/
|
|
│ ├── all_test-1a3b4c/
|
|
│ └── anatomy_test-9e666f/
|
|
└── bbh/
|
|
└── boolean_expressions_test-… (one dir per subtask)
|
|
|
|
.raw/<repo-hash>/ shared download blobs (ModelScope sources);
|
|
cache entries hardlink from here, so multi-
|
|
subset mirrors download only once
|
|
```
|
|
|
|
> Every cache entry is **self-contained and preserves native data**: `raw/`
|
|
> holds the original file(s) exactly as downloaded (never converted);
|
|
> `samples.jsonl` is the derived unified view. HF-hub sources keep an exact
|
|
> pre-conversion record dump in `raw/records.jsonl`. Rebuild any entry with
|
|
> `evalharness data fetch <name> --force`.
|
|
|
|
> Why the 6-char hash suffix: two variants with the same subset/split but
|
|
> different sources (`--source`) or params would otherwise collide and serve
|
|
> stale data. The short hash keeps them apart while staying readable.
|
|
|
|
## Built-in datasets (28, official sources)
|
|
|
|
| Family | Datasets (source) |
|
|
|---|---|
|
|
| Math | gsm8k (openai/gsm8k), competition_math (EleutherAI/hendrycks_math), aime24 (HuggingFaceH4), aime25 (yentinglin), aime26*, hmmt26*, imo_answerbench* |
|
|
| Knowledge / MCQ | mmlu (cais/mmlu), mmlu_pro (TIGER-Lab), cmmlu (haonan-li), gpqa_diamond (Idavidrein/gpqa), arc (allenai/ai2_arc), hellaswag, winogrande |
|
|
| QA | trivia_qa (mandarjoshi), drop (ucinlp), simple_qa (mirror of OpenAI's CSV), hle (cais/hle), bbh (lukaemon/bbh) |
|
|
| Long context | longbench_v2 (THUDM), openai_mrcr (openai-mirror) |
|
|
| Coding | humaneval (openai), bigcodebench (bigcode), live_code_bench (livecodebench) |
|
|
| Agent / tools | swe_bench_verified (princeton-nlp), tau2_bench (official GitHub), bfcl_v3 (official GitHub), general_fc (evalscope-native) |
|
|
|
|
\* aime26 / hmmt26 / imo_answerbench have no official standalone release and
|
|
use community curations (evalscope); simple_qa's official artifact is the CSV
|
|
in `openai/simple-evals` (HF is a mirror); tau2_bench / bfcl_v3 are officially
|
|
released on GitHub — clone and point `--source` at the local files.
|
|
|
|
## Adding a dataset
|
|
|
|
Drop a single-file plugin into `evalharness/data/datasets/` — auto-discovered,
|
|
no central file to edit.
|
|
|
|
Well-shaped records (column names map directly) — pure declaration:
|
|
|
|
```python
|
|
# evalharness/data/datasets/cmmlu.py
|
|
@register_dataset(DatasetSpec(name='cmmlu', source='haonan-li/cmmlu', split='test', task_type='mcq'))
|
|
def cmmlu():
|
|
return FieldSpec(input='question', choices='choices', target='answer', metadata=['category'])
|
|
```
|
|
|
|
Custom conversion — return a `record -> Sample` function:
|
|
|
|
```python
|
|
@register_dataset(DatasetSpec(name='gsm8k', source='openai/gsm8k', subset='main',
|
|
split='test', task_type='math'))
|
|
def gsm8k():
|
|
def to_sample(record):
|
|
parts = record['answer'].split('####')
|
|
return Sample(input=record['question'], target=parts.pop().strip())
|
|
return to_sample
|
|
```
|
|
|
|
Sources supported: local `.jsonl/.json/.csv/.tsv` files; local directories
|
|
(probed as `{subset}_{split}.jsonl` etc.); HF hub dataset ids (requires the
|
|
`hub` extra, imported lazily); official GitHub releases (clone, then
|
|
`--source` the local path).
|
|
|
|
Conventions: `choices` holds option *contents*; `target` is a **letter** for
|
|
MCQ (e.g. `'B'`) and text otherwise (use `List[str]` for multiple gold
|
|
answers); long contexts/passages go to `metadata`, not `input`; `id` is
|
|
assigned sequentially at materialize time when absent.
|
|
|
|
## Design decisions
|
|
|
|
The data layer mirrors conclusions from a close reading of seven evaluation
|
|
frameworks (evalscope, lm-evaluation-harness, inspect_ai, deepeval, VLMEvalKit,
|
|
harbor, deepseek-harness); see `../README.md` for the full analysis:
|
|
|
|
- **`Sample` base + one `record_to_sample` per plugin** (evalscope/inspect_ai):
|
|
raw formats vary wildly; the unified format only exists after conversion,
|
|
and everything downstream sees only `Sample`.
|
|
- **Datasets as first-class citizens** (inspect_ai): a `Dataset` from
|
|
`get_dataset()` can be filtered/synthesized/exported freely — it is not
|
|
welded into an evaluation recipe.
|
|
- **Registration is cheap, materialization pays**: the registry holds only
|
|
metadata + conversion recipes; `list` never touches the network.
|
|
- **Config-sensitive cache keys** (evalscope): a cache hit is always correct;
|
|
no invalidation logic exists.
|
|
- **Sandbox/tool fields on `Sample` are declarations only**
|
|
(`sandbox/files/setup/tools`): the data layer never executes; a future
|
|
sandbox layer will materialize them.
|
|
- **Data load/unload vs environment load/unload are different layers**:
|
|
`fetch`/`unload` move bytes only (raw files + converted samples). Heavy
|
|
execution environments (e.g. the ~1GB-per-instance `sweb.eval.*` images
|
|
declared by swe_bench_verified) are pulled lazily *at eval time* by the
|
|
sandbox layer — never at data-fetch time — and their removal is
|
|
refcounted there because docker layers are shared across instances and
|
|
benchmarks. `DatasetSpec.requires` (e.g. `['docker']`) is the declaration
|
|
the sandbox/deploy layer reads.
|
|
|
|
## Layout
|
|
|
|
```
|
|
EvalHarness/
|
|
├── pyproject.toml
|
|
├── evalharness/
|
|
│ ├── cli.py # argparse CLI: data list/fetch(concurrent)/stats/show
|
|
│ └── data/
|
|
│ ├── sample.py # Sample / ChatMessage / SandboxSpec / ToolInfo
|
|
│ ├── spec.py # DatasetSpec (metadata) / FieldSpec (field mapping)
|
|
│ ├── registry.py # Registry + @register_dataset + get_dataset
|
|
│ ├── loader.py # raw record loading (local file/dir/hub)
|
|
│ ├── dataset.py # Dataset: lazy materialize + cache + derived views
|
|
│ └── datasets/ # 28 built-in single-file dataset plugins
|
|
├── examples/
|
|
│ └── data/ # offline demo subsets (gsm8k/cmmlu, 5 rows each)
|
|
```
|
|
|
|
## Roadmap (not built yet, one layer at a time)
|
|
|
|
- [ ] Model layer (ModelAdapter: unified URL-based invocation)
|
|
- [ ] Evaluation engine (evaluator + scorer/metric)
|
|
- [ ] Sandbox layer (materialize `Sample.sandbox`: lazy per-instance image
|
|
pull, refcounted image unload, container lifecycle; `requires` gating)
|
|
- [ ] Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
|
|
- [ ] Skill layer (full evaluation pipelines as composable skills)
|
|
- [ ] Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
|
|
- [ ] Web/API interface
|