384 lines
20 KiB
Markdown
384 lines
20 KiB
Markdown
# EvalHarness
|
|
|
|
A plugin-based LLM/agent evaluation harness. **Currently: data + evaluation +
|
|
model + sandbox + agent-driver layers** (datasets/eval-recipes/model-adapters/
|
|
deployers/sandboxes/environments as plugins, lazy materialization cache,
|
|
official-aligned scorers, async generation, sandboxed code execution, agent
|
|
message pump, report artifacts & console visualization). Tool/skill layers
|
|
land next.
|
|
|
|
## Features
|
|
|
|
- **Unified `Sample` schema** (pydantic): `input / choices / target / task_type /
|
|
tools / sandbox / files / setup / metadata`. Raw dataset formats are
|
|
unconstrained; each dataset plugin converts its records into `Sample`.
|
|
- **Dataset registration**: `@register_dataset(DatasetSpec(...))` decorator,
|
|
import-time registration; `get_dataset(name)` returns a **lazy handle** —
|
|
listing and registration never download anything.
|
|
- **Lazy materialization + content-addressed cache**: the first real use
|
|
(iteration / `len` / indexing) triggers `download -> convert -> cache write`.
|
|
Cache dir layout: `datasets/<benchmark>/<subset>_<split>[-<version>]-<hash6>/`
|
|
— readable benchmark folders, one readable subdir per subset/split/version.
|
|
A file lock prevents duplicate concurrent downloads; `tmp+rename` atomic
|
|
writes prevent torn caches.
|
|
- **Two plugin styles**: pure `FieldSpec` declarative mapping when records are
|
|
well-shaped (zero conversion code), or a custom `record_to_sample` function.
|
|
- **CLI**: `list / fetch (concurrent) / unload / stats / show`.
|
|
- **28 built-in datasets** registered against their official sources
|
|
(see table below).
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install . # core only (pydantic)
|
|
pip install '.[hub]' # + HuggingFace datasets
|
|
pip install '.[math]' # + sympy/pylatexenc (aime/math official grader)
|
|
pip install '.[exec]' # + numpy/scipy (official DROP aligner)
|
|
pip install '.[hub,parquet,math,exec]' # everything
|
|
```
|
|
|
|
For development: `pip install -e '.[hub,parquet,math,exec]'`.
|
|
|
|
Dependency tiers (deliberate — no monolithic image): text-compare recipes
|
|
run on core; symbolic math needs `[math]`; DROP needs `[exec]`; code-execution
|
|
benchmarks (humaneval/bigcodebench/LCB/bfcl) additionally require the sandbox
|
|
layer (roadmap) since they run model-generated code in isolation.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
evalharness data list # list registered datasets (no network, no download)
|
|
evalharness data fetch gsm8k # materialize: first run from source, then cache
|
|
evalharness data fetch gsm8k mmlu arc --workers 8 # concurrent prefetch
|
|
evalharness data unload gsm8k # drop the cache entry (raw/ + samples + meta)
|
|
evalharness data stats cmmlu # materialize + stats (count/lengths/answers/cache path)
|
|
evalharness data show gsm8k -n 2 # print the first N samples
|
|
|
|
# spec overrides: offline demo / local data / picking a subset
|
|
evalharness data fetch gsm8k --source examples/data/gsm8k_main_test.jsonl # bundled tiny set
|
|
evalharness data fetch mmlu --subset anatomy # one of MMLU's 57 subjects
|
|
evalharness data fetch bbh --subset word_sorting # one of BBH's 27 subtasks
|
|
# note: with multiple names, --source/--split/--subset apply to ALL of them;
|
|
# run separately when you need per-benchmark overrides
|
|
```
|
|
|
|
Python API:
|
|
|
|
```python
|
|
from evalharness import get_dataset, list_datasets
|
|
|
|
ds = get_dataset('gsm8k') # lazy handle: zero network/disk cost here
|
|
len(ds) # first use -> materialize (download->convert->cache)
|
|
for s in ds:
|
|
print(s.input, s.target)
|
|
|
|
ds_sub = get_dataset('mmlu', subset='anatomy') # spec override, separate cache
|
|
hard = ds.view([s for s in ds if len(s.input_text) > 100],
|
|
lineage={'tool': 'length_filter'}) # derived data: same class, with lineage
|
|
```
|
|
|
|
Cache root: `~/.cache/evalharness/` (override with the `EVALHARNESS_CACHE`
|
|
environment variable). Layout example:
|
|
|
|
```
|
|
datasets/
|
|
├── gsm8k/
|
|
│ ├── main_test-e06f82/ <- official openai/gsm8k
|
|
│ │ ├── raw/ # NATIVE source data, byte-exact as downloaded
|
|
│ │ ├── samples.jsonl # converted unified Sample view
|
|
│ │ └── meta.json # spec + provenance + raw file list
|
|
│ └── main_test-daafcc/ <- local demo via --source
|
|
├── mmlu/
|
|
│ ├── all_test-1a3b4c/
|
|
│ └── anatomy_test-9e666f/
|
|
└── bbh/
|
|
└── boolean_expressions_test-… (one dir per subtask)
|
|
|
|
.raw/<repo-hash>/ shared download blobs (ModelScope sources);
|
|
cache entries hardlink from here, so multi-
|
|
subset mirrors download only once
|
|
```
|
|
|
|
> Every cache entry is **self-contained and preserves native data**: `raw/`
|
|
> holds the original file(s) exactly as downloaded (never converted);
|
|
> `samples.jsonl` is the derived unified view. HF-hub sources keep an exact
|
|
> pre-conversion record dump in `raw/records.jsonl`. Rebuild any entry with
|
|
> `evalharness data fetch <name> --force`.
|
|
|
|
> Why the 6-char hash suffix: two variants with the same subset/split but
|
|
> different sources (`--source`) or params would otherwise collide and serve
|
|
> stale data. The short hash keeps them apart while staying readable.
|
|
|
|
## Evaluation layer
|
|
|
|
```bash
|
|
evalharness eval list # 28 recipes, per benchmark
|
|
evalharness eval run gsm8k preds.jsonl --model mymodel --out gsm8k.report.json
|
|
evalharness viz show gsm8k.report.json # console table
|
|
evalharness viz show r1.json r2.json --style md_compare
|
|
```
|
|
|
|
```python
|
|
from evalharness import get_dataset
|
|
from evalharness.eval import evaluate, get_eval
|
|
from evalharness.viz import render
|
|
|
|
ds = get_dataset('gsm8k')
|
|
report = evaluate(ds, predictions, model='mymodel') # recipe auto-resolved
|
|
report.save('gsm8k.report.json')
|
|
print(render(report, style='text'))
|
|
```
|
|
|
|
Pipeline: **extract -> score -> aggregate**, each stage a registered plugin:
|
|
|
|
- **Extractors** (`eval/extractor.py`): `math_boxed / mcq_letter /
|
|
answer_phrase / answer_spans / gsm8k_hash / last_number / code_block /
|
|
quoted_list / identity` + cascades (first stage that succeeds wins).
|
|
- **Scorers** (`eval/scorer.py`): text compare (`exact / math_equal /
|
|
em_f1 / alias_match` — official implementations), `llm_judge` (labels->
|
|
scores contract, wired to a ModelAdapter via `evaluate(judge=...)`),
|
|
`execution` / `env_reward` (raise LayerNotReady until sandbox/agent land).
|
|
- **Aggregators** (`eval/aggregator.py`): `mean / pass_at_k /
|
|
grouped_avg / weighted_group_avg / binned_avg / simpleqa_official`
|
|
(is_given_attempted + accuracy_given_attempted).
|
|
- **Recipes** (`eval/recipes/`): one per benchmark, 5-20 lines each — pure
|
|
bindings of the primitives above, `@register_eval('gsm8k')`.
|
|
|
|
Per-sample results keep `raw_prediction` + extraction note + score details;
|
|
`extraction_failure_rate` is reported as a health metric. Changing a recipe
|
|
and re-running `evaluate()` re-scores the same predictions — the model is
|
|
never re-queried.
|
|
|
|
## Model layer (calling + deploying, separate plugins on purpose)
|
|
|
|
```bash
|
|
# generate + score in one command (async, concurrent)
|
|
evalharness eval run gsm8k --model mock:boxed --limit 100 # offline pipeline check
|
|
evalharness eval run gsm8k --model openai/http://gpu03:8000/v1?qwen3-8b
|
|
evalharness eval run hle --model openai/...?qwen3-8b --judge openai/...?gpt-4o
|
|
evalharness eval run bfcl_v3 --model mock:fc --env bfcl_mock # agent pump
|
|
# future: --model deploy:vllm/qwen3-8b (Deployer pulls a pinned docker env)
|
|
```
|
|
|
|
Model spec grammar (plain strings):
|
|
|
|
| spec | meaning |
|
|
|---|---|
|
|
| `mock` / `mock:boxed` / `mock:tool` | offline adapter (echo / oracle-boxed / tool-call) |
|
|
| `openai/<api_base>?<model_id>` | any OpenAI-protocol endpoint: vllm, sglang, lmdeploy, ollama, cloud APIs |
|
|
| `deploy:<engine>/<model>` | Deployer resolves the endpoint first (vllm/sglang: pinned docker image; external: models.yaml) |
|
|
|
|
Design:
|
|
|
|
- **ModelAdapter = how to call** (protocol). All adapters are `async` and
|
|
return structured `ModelOutput(text, tool_calls, usage)` — the hinge the
|
|
future agent loops hang on; single-turn recipes just read `.text`.
|
|
- **Deployer = how to run** (environment, separate lifecycle). Docker images
|
|
are pinned per model in `models.yaml`, so `vllm:v0.9.2` and `vllm:v0.6.6`
|
|
stacks coexist on one machine; `external` connects to existing endpoints.
|
|
- **Async boundary = waiting on the model**: `run_eval()` fans out calls with
|
|
a semaphore (default 32), collects raws + per-sample usage, then hands them
|
|
to the synchronous `evaluate()`. Data/scoring stay sync (fast, CPU/disk).
|
|
|
|
## Sandbox layer (environments for BOTH eval execution and model serving)
|
|
|
|
One docker implementation, two faces:
|
|
|
|
- **exec()** runs untrusted model-generated code hard-isolated:
|
|
`--network none`, cpu/mem/pids caps, read-only rootfs, tmpfs /tmp.
|
|
Host file sharing via **bind mounts** (`mounts={'/out': host_dir}`) —
|
|
artifacts land on the host directly, no `docker cp`.
|
|
- **serve()** trusted engine containers (vllm/sglang) with network + GPU
|
|
passthrough; consumed by the model Deployer through the same layer.
|
|
- **Lifecycle**: refcounted `acquire()/release()`; containers stop+rm at
|
|
refcount 0 or process exit (atexit); **images are never auto-deleted** —
|
|
re-acquire re-runs the local image instantly.
|
|
|
|
```python
|
|
from evalharness.sandbox import get_sandbox
|
|
r = get_sandbox('docker').exec({'main.py': 'print(42)'}) # or 'local' for dev
|
|
```
|
|
|
|
## Agent evaluation driver (message pump, not a thinking framework)
|
|
|
|
We EVALUATE agents: the model under test thinks; we only execute its
|
|
tool_calls against Environment plugins and feed observations back.
|
|
|
|
```python
|
|
from evalharness.model import run_eval
|
|
report = await run_eval(ds, 'openai/http://gpu03:8000/v1?qwen3-8b',
|
|
env='bfcl_mock') # agent pump per sample
|
|
# CLI: evalharness eval run bfcl_v3 --model mock:fc --env bfcl_mock
|
|
```
|
|
|
|
- `agent/loop.py::drive()` — pump until no more calls / env done / max_turns;
|
|
records a full `Trajectory` (messages, per-turn usage, env final state)
|
|
into `SampleResult.trajectory / env_state`.
|
|
- `agent/envs/bfcl_mock.py` — BFCL official-style: record call sequence,
|
|
compare against ground_truth (`env_reward` scorer), incl. irrelevance
|
|
categories (correct = call nothing). tau2 / swe envs land later on the
|
|
same `Environment` contract.
|
|
- Single-turn is the degenerate case: no env -> one generate, done.
|
|
|
|
## Built-in datasets (28, official sources)
|
|
|
|
| Family | Datasets (source) |
|
|
|---|---|
|
|
| Math | gsm8k (openai/gsm8k), competition_math (EleutherAI/hendrycks_math), aime24 (HuggingFaceH4), aime25 (yentinglin), aime26*, hmmt26*, imo_answerbench* |
|
|
| Knowledge / MCQ | mmlu (cais/mmlu), mmlu_pro (TIGER-Lab), cmmlu (haonan-li), gpqa_diamond (Idavidrein/gpqa), arc (allenai/ai2_arc), hellaswag, winogrande |
|
|
| QA | trivia_qa (mandarjoshi), drop (ucinlp), simple_qa (mirror of OpenAI's CSV), hle (cais/hle), bbh (lukaemon/bbh) |
|
|
| Long context | longbench_v2 (THUDM), openai_mrcr (openai-mirror) |
|
|
| Coding | humaneval (openai), bigcodebench (bigcode), live_code_bench (livecodebench) |
|
|
| Agent / tools | swe_bench_verified (princeton-nlp), tau2_bench (official GitHub), bfcl_v3 (official GitHub), general_fc (evalscope-native) |
|
|
|
|
\* aime26 / hmmt26 / imo_answerbench have no official standalone release and
|
|
use community curations (evalscope); simple_qa's official artifact is the CSV
|
|
in `openai/simple-evals` (HF is a mirror); tau2_bench / bfcl_v3 are officially
|
|
released on GitHub — clone and point `--source` at the local files.
|
|
|
|
## Adding a dataset
|
|
|
|
Drop a single-file plugin into `evalharness/data/datasets/` — auto-discovered,
|
|
no central file to edit.
|
|
|
|
Well-shaped records (column names map directly) — pure declaration:
|
|
|
|
```python
|
|
# evalharness/data/datasets/cmmlu.py
|
|
@register_dataset(DatasetSpec(name='cmmlu', source='haonan-li/cmmlu', split='test', task_type='mcq'))
|
|
def cmmlu():
|
|
return FieldSpec(input='question', choices='choices', target='answer', metadata=['category'])
|
|
```
|
|
|
|
Custom conversion — return a `record -> Sample` function:
|
|
|
|
```python
|
|
@register_dataset(DatasetSpec(name='gsm8k', source='openai/gsm8k', subset='main',
|
|
split='test', task_type='math'))
|
|
def gsm8k():
|
|
def to_sample(record):
|
|
parts = record['answer'].split('####')
|
|
return Sample(input=record['question'], target=parts.pop().strip())
|
|
return to_sample
|
|
```
|
|
|
|
Sources supported: local `.jsonl/.json/.csv/.tsv` files; local directories
|
|
(probed as `{subset}_{split}.jsonl` etc.); HF hub dataset ids (requires the
|
|
`hub` extra, imported lazily); official GitHub releases (clone, then
|
|
`--source` the local path).
|
|
|
|
Conventions: `choices` holds option *contents*; `target` is a **letter** for
|
|
MCQ (e.g. `'B'`) and text otherwise (use `List[str]` for multiple gold
|
|
answers); long contexts/passages go to `metadata`, not `input`; `id` is
|
|
assigned sequentially at materialize time when absent.
|
|
|
|
## Design decisions
|
|
|
|
The data layer mirrors conclusions from a close reading of seven evaluation
|
|
frameworks (evalscope, lm-evaluation-harness, inspect_ai, deepeval, VLMEvalKit,
|
|
harbor, deepseek-harness); see `../README.md` for the full analysis:
|
|
|
|
- **Data layer vs eval layer: strict separation** (unlike evalscope's
|
|
DataAdapter, which welds record conversion + extraction + scoring + prompt
|
|
into one class). The data layer answers *what is the question/target*; the
|
|
eval layer answers *how to judge a response*. Consequences: predictions can
|
|
be re-scored under a new recipe without re-running the model; extraction
|
|
rules are shared primitives (`math_boxed`, `mcq_letter`, ...) instead of
|
|
per-benchmark copies; official grader fixes (e.g. DROP's Hungarian
|
|
alignment) land in one scorer and apply to every recipe.
|
|
- **`Sample` base + one `record_to_sample` per plugin** (evalscope/inspect_ai):
|
|
raw formats vary wildly; the unified format only exists after conversion,
|
|
and everything downstream sees only `Sample`.
|
|
- **Datasets as first-class citizens** (inspect_ai): a `Dataset` from
|
|
`get_dataset()` can be filtered/synthesized/exported freely — it is not
|
|
welded into an evaluation recipe.
|
|
- **Registration is cheap, materialization pays**: the registry holds only
|
|
metadata + conversion recipes; `list` never touches the network.
|
|
- **Config-sensitive cache keys** (evalscope): a cache hit is always correct;
|
|
no invalidation logic exists.
|
|
- **Raw predictions are immutable artifacts**: every SampleResult keeps
|
|
raw_prediction; extraction/score details are re-derivable. Failed
|
|
extractions are surfaced (extraction_failure_rate), never silently zeroed.
|
|
- **Official grader parity**: scorers replicate official logic where it
|
|
exists (PRM800K math equivalence; DROP's answer_to_bags +
|
|
linear_sum_assignment; SimpleQA's A/B/C judge with NOT_ATTEMPTED fallback
|
|
and accuracy_given_attempted). Official test cases anchor `tests/`.
|
|
- **Sandbox/tool fields on `Sample` are declarations only**
|
|
(`sandbox/files/setup/tools`): the data layer never executes; a future
|
|
sandbox layer will materialize them.
|
|
- **Data load/unload vs environment load/unload are different layers**:
|
|
`fetch`/`unload` move bytes only (raw files + converted samples). Heavy
|
|
execution environments (e.g. the ~1GB-per-instance `sweb.eval.*` images
|
|
declared by swe_bench_verified) are pulled lazily *at eval time* by the
|
|
sandbox layer — never at data-fetch time — and their removal is
|
|
refcounted there because docker layers are shared across instances and
|
|
benchmarks. `DatasetSpec.requires` (e.g. `['docker']`) is the declaration
|
|
the sandbox/deploy layer reads.
|
|
|
|
## Layout
|
|
|
|
```
|
|
EvalHarness/
|
|
├── pyproject.toml
|
|
├── evalharness/
|
|
│ ├── cli.py # CLI: data list/fetch/unload/stats/show
|
|
│ │ # eval list/run, viz show
|
|
│ ├── data/ # ---- data layer ----
|
|
│ │ ├── sample.py # Sample / ChatMessage / SandboxSpec / ToolInfo
|
|
│ │ ├── spec.py # DatasetSpec (metadata) / FieldSpec (field mapping)
|
|
│ │ ├── registry.py # Registry + @register_dataset + get_dataset
|
|
│ │ ├── loader.py # raw loading (local/HF/ModelScope native)
|
|
│ │ ├── dataset.py # Dataset: lazy materialize + cache + derived views
|
|
│ │ └── datasets/ # 28 built-in single-file dataset plugins
|
|
│ ├── model/ # ---- model layer ----
|
|
│ │ ├── output.py # ModelOutput/ToolCall/Usage (agent hinge)
|
|
│ │ ├── adapter.py # @register_adapter: openai_compatible / mock
|
|
│ │ ├── deployer.py # @register_deployer: vllm / sglang / external
|
|
│ │ └── runner.py # async run_eval(): generate -> evaluate
|
|
│ ├── sandbox/ # ---- environment layer ----
|
|
│ │ ├── base.py # Sandbox iface + refcounted acquire/release + atexit
|
|
│ │ ├── docker.py # exec (isolated) + serve (engines) one impl
|
|
│ │ └── local.py # dev-only, no isolation
|
|
│ ├── agent/ # ---- agent evaluation driver ----
|
|
│ │ ├── loop.py # drive(): message pump + Trajectory
|
|
│ │ └── envs/bfcl_mock.py # BFCL official-style env (tau2/swe later)
|
|
│ ├── eval/ # ---- evaluation layer ----
|
|
│ │ ├── record.py # SampleResult / EvalReport artifacts
|
|
│ │ ├── extractor.py # answer-extraction primitives (+cascades)
|
|
│ │ ├── scorer.py # scoring primitives (official implementations)
|
|
│ │ ├── math_grader.py # PRM800K sympy equivalence (optional [math])
|
|
│ │ ├── aggregator.py # mean/pass@k/grouped/binned/simpleqa_official
|
|
│ │ ├── recipe.py # EvalRecipe bindings + @register_eval
|
|
│ │ ├── runner.py # evaluate(dataset, predictions) -> EvalReport
|
|
│ │ └── recipes/ # 28 per-benchmark recipes
|
|
│ └── viz/ # ---- visualization (report consumer) ----
|
|
│ └── renderers/text.py # text / md / md_compare / radar / errors
|
|
├── examples/
|
|
│ └── data/ # offline demo subsets (gsm8k/cmmlu, 5 rows each)
|
|
└── tests/
|
|
└── test_eval.py # official-anchor regression tests
|
|
```
|
|
|
|
## Roadmap (not built yet, one layer at a time)
|
|
|
|
- [x] Data layer (28 dataset plugins, lazy cache, native HF/ModelScope loaders)
|
|
- [x] Evaluation layer (extract/score/aggregate plugins, official scorers, recipes)
|
|
- [x] Visualization (console/markdown renderers over report artifacts)
|
|
- [x] Model layer (async ModelAdapter openai_compatible+mock, ModelOutput
|
|
with tool_calls, Deployer registry vllm/sglang/external + models.yaml
|
|
env pinning, run_eval generate->score)
|
|
- [x] Sandbox layer (docker exec hard-isolation + serve environments,
|
|
refcounted acquire/release, atexit teardown, images kept, bind-mount
|
|
host sharing; Deployer now consumes it)
|
|
- [x] Agent evaluation driver (message pump + Trajectory + bfcl_mock env
|
|
with official call-sequence scoring; tau2/swe envs pending)
|
|
- [ ] Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
|
|
- [ ] Skill layer (full evaluation pipelines as composable skills)
|
|
- [ ] tau2 / swe-bench environments (user simulator; per-instance sweb.* images)
|
|
- [ ] Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
|
|
- [ ] Web/API interface
|
|
- [ ] Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
|
|
- [ ] Skill layer (full evaluation pipelines as composable skills)
|
|
- [ ] Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
|
|
- [ ] Web/API interface
|