EvalHarness/README.md

# EvalHarness

A plugin-based LLM/agent evaluation harness. **Currently: data + evaluation +
model + sandbox + agent-driver layers** (datasets/eval-recipes/model-adapters/
deployers/sandboxes/environments as plugins, lazy materialization cache,
official-aligned scorers, async generation, sandboxed code execution, agent
message pump, report artifacts & console visualization). Tool/skill layers
land next.

## Features

- **Unified `Sample` schema** (pydantic): `input / choices / target / task_type /
  tools / sandbox / files / setup / metadata`. Raw dataset formats are
  unconstrained; each dataset plugin converts its records into `Sample`.
- **Dataset registration**: `@register_dataset(DatasetSpec(...))` decorator,
  import-time registration; `get_dataset(name)` returns a **lazy handle** —
  listing and registration never download anything.
- **Lazy materialization + content-addressed cache**: the first real use
  (iteration / `len` / indexing) triggers `download -> convert -> cache write`.
  Cache dir layout: `datasets/<benchmark>/<subset>_<split>[-<version>]-<hash6>/`
  — readable benchmark folders, one readable subdir per subset/split/version.
  A file lock prevents duplicate concurrent downloads; `tmp+rename` atomic
  writes prevent torn caches.
- **Two plugin styles**: pure `FieldSpec` declarative mapping when records are
  well-shaped (zero conversion code), or a custom `record_to_sample` function.
- **CLI**: `list / fetch (concurrent) / unload / stats / show`.
- **28 built-in datasets** registered against their official sources
  (see table below).

## Install

```bash
pip install .                  # core + all light deps (datasets/pyarrow/sympy/numpy/scipy):
                               # every benchmark scores officially, nothing conflicts
pip install '.[bfcl]'          # + heavy official bfcl ast_checker (5 cloud SDKs + qwen-agent;
                               # only for official-parity BFCL scoring -- native default works)
```

Heavy execution environments never enter the venv: code benchmarks run in
docker sandboxes (`python:3.11-slim`, the official bigcodebench image,
`sweb.eval.*` per-instance images via `sandbox prefetch`).

## Quick start

```bash
evalharness data list                 # list registered datasets (no network, no download)
evalharness data fetch gsm8k          # materialize: first run from source, then cache
evalharness data fetch gsm8k mmlu arc --workers 8   # concurrent prefetch
evalharness data unload gsm8k         # drop the cache entry (raw/ + samples + meta)
evalharness data stats cmmlu          # materialize + stats (count/lengths/answers/cache path)
evalharness data show gsm8k -n 2      # print the first N samples

# spec overrides: offline demo / local data / picking a subset
evalharness data fetch gsm8k --source examples/data/gsm8k_main_test.jsonl  # bundled tiny set
evalharness data fetch mmlu --subset anatomy        # one of MMLU's 57 subjects
evalharness data fetch bbh --subset word_sorting    # one of BBH's 27 subtasks
# note: with multiple names, --source/--split/--subset apply to ALL of them;
# run separately when you need per-benchmark overrides
```

Python API:

```python
from evalharness import get_dataset, list_datasets

ds = get_dataset('gsm8k')      # lazy handle: zero network/disk cost here
len(ds)                        # first use -> materialize (download->convert->cache)
for s in ds:
    print(s.input, s.target)

ds_sub = get_dataset('mmlu', subset='anatomy')   # spec override, separate cache
hard = ds.view([s for s in ds if len(s.input_text) > 100],
               lineage={'tool': 'length_filter'})   # derived data: same class, with lineage
```

Cache root: `~/.cache/evalharness/` (override with the `EVALHARNESS_CACHE`
environment variable). Layout example:

```
datasets/
├── gsm8k/
│   ├── main_test-e06f82/     <- official openai/gsm8k
│   │   ├── raw/              # NATIVE source data, byte-exact as downloaded
│   │   ├── samples.jsonl     # converted unified Sample view
│   │   └── meta.json         # spec + provenance + raw file list
│   └── main_test-daafcc/     <- local demo via --source
├── mmlu/
│   ├── all_test-1a3b4c/
│   └── anatomy_test-9e666f/
└── bbh/
    └── boolean_expressions_test-…    (one dir per subtask)

.raw/<repo-hash>/              shared download blobs (ModelScope sources);
                               cache entries hardlink from here, so multi-
                               subset mirrors download only once
```

> Every cache entry is **self-contained and preserves native data**: `raw/`
> holds the original file(s) exactly as downloaded (never converted);
> `samples.jsonl` is the derived unified view. HF-hub sources keep an exact
> pre-conversion record dump in `raw/records.jsonl`. Rebuild any entry with
> `evalharness data fetch <name> --force`.

> Why the 6-char hash suffix: two variants with the same subset/split but
> different sources (`--source`) or params would otherwise collide and serve
> stale data. The short hash keeps them apart while staying readable.

## Evaluation layer

```bash
evalharness eval list                          # 28 recipes, per benchmark
evalharness eval run gsm8k preds.jsonl --model mymodel --out gsm8k.report.json
evalharness viz show gsm8k.report.json         # console table
evalharness viz show r1.json r2.json --style md_compare
```

```python
from evalharness import get_dataset
from evalharness.eval import evaluate, get_eval
from evalharness.viz import render

ds = get_dataset('gsm8k')
report = evaluate(ds, predictions, model='mymodel')   # recipe auto-resolved
report.save('gsm8k.report.json')
print(render(report, style='text'))
```

Pipeline: **extract -> score -> aggregate**, each stage a registered plugin:

- **Extractors** (`eval/extractor.py`): `math_boxed / mcq_letter /
  answer_phrase / answer_spans / gsm8k_hash / last_number / code_block /
  quoted_list / identity` + cascades (first stage that succeeds wins).
- **Scorers** (`eval/scorer.py`): text compare (`exact / math_equal /
  em_f1 / alias_match` — official implementations), `llm_judge` (labels->
  scores contract, wired to a ModelAdapter via `evaluate(judge=...)`),
  `execution` / `env_reward` (raise LayerNotReady until sandbox/agent land).
- **Aggregators** (`eval/aggregator.py`): `mean / pass_at_k /
  grouped_avg / weighted_group_avg / binned_avg / simpleqa_official`
  (is_given_attempted + accuracy_given_attempted).
- **Recipes** (`eval/recipes/`): one per benchmark, 5-20 lines each — pure
  bindings of the primitives above, `@register_eval('gsm8k')`.

Per-sample results keep `raw_prediction` + extraction note + score details;
`extraction_failure_rate` is reported as a health metric. Changing a recipe
and re-running `evaluate()` re-scores the same predictions — the model is
never re-queried.

## Model layer (calling + deploying, separate plugins on purpose)

```bash
# generate + score in one command (async, concurrent)
evalharness eval run gsm8k --model mock:boxed --limit 100          # offline pipeline check
evalharness eval run gsm8k --model openai/http://gpu03:8000/v1?qwen3-8b
evalharness eval run hle   --model openai/...?qwen3-8b --judge openai/...?gpt-4o
evalharness eval run bfcl_v3 --model mock:fc --env bfcl_mock     # agent pump
# future: --model deploy:vllm/qwen3-8b  (Deployer pulls a pinned docker env)
```

Model spec grammar (plain strings):

| spec | meaning |
|---|---|
| `mock` / `mock:boxed` / `mock:tool` | offline adapter (echo / oracle-boxed / tool-call) |
| `openai/<api_base>?<model_id>` | any OpenAI-protocol endpoint: vllm, sglang, lmdeploy, ollama, cloud APIs |
| `deploy:<engine>/<model>` | Deployer resolves the endpoint first (vllm/sglang: pinned docker image; external: models.yaml) |

Design:

- **ModelAdapter = how to call** (protocol). All adapters are `async` and
  return structured `ModelOutput(text, tool_calls, usage)` — the hinge the
  future agent loops hang on; single-turn recipes just read `.text`.
- **Deployer = how to run** (environment, separate lifecycle). Docker images
  are pinned per model in `models.yaml`, so `vllm:v0.9.2` and `vllm:v0.6.6`
  stacks coexist on one machine; `external` connects to existing endpoints.
- **Async boundary = waiting on the model**: `run_eval()` fans out calls with
  a semaphore (default 32), collects raws + per-sample usage, then hands them
  to the synchronous `evaluate()`. Data/scoring stay sync (fast, CPU/disk).

## Sandbox layer (environments for BOTH eval execution and model serving)

One docker implementation, two faces:

- **exec()** runs untrusted model-generated code hard-isolated:
  `--network none`, cpu/mem/pids caps, read-only rootfs, tmpfs /tmp.
  Host file sharing via **bind mounts** (`mounts={'/out': host_dir}`) —
  artifacts land on the host directly, no `docker cp`.
- **serve()** trusted engine containers (vllm/sglang) with network + GPU
  passthrough; consumed by the model Deployer through the same layer.
- **Lifecycle**: refcounted `acquire()/release()`; containers stop+rm at
  refcount 0 or process exit (atexit); **images are never auto-deleted** —
  re-acquire re-runs the local image instantly.

```python
from evalharness.sandbox import get_sandbox
r = get_sandbox('docker').exec({'main.py': 'print(42)'})   # or 'local' for dev
```

## Agent evaluation driver (message pump, not a thinking framework)

We EVALUATE agents: the model under test thinks; we only execute its
tool_calls against Environment plugins and feed observations back.

```python
from evalharness.model import run_eval
report = await run_eval(ds, 'openai/http://gpu03:8000/v1?qwen3-8b',
                        env='bfcl_mock')        # agent pump per sample
# CLI: evalharness eval run bfcl_v3 --model mock:fc --env bfcl_mock
```

- `agent/loop.py::drive()` — pump until no more calls / env done / max_turns;
  records a full `Trajectory` (messages, per-turn usage, env final state)
  into `SampleResult.trajectory / env_state`.
- `agent/envs/bfcl_mock.py` — BFCL official-style: record call sequence,
  compare against ground_truth (`env_reward` scorer), incl. irrelevance
  categories (correct = call nothing). tau2 / swe envs land later on the
  same `Environment` contract.
- Single-turn is the degenerate case: no env -> one generate, done.

## Built-in datasets (28, official sources)

| Family | Datasets (source) |
|---|---|
| Math | gsm8k (openai/gsm8k), competition_math (EleutherAI/hendrycks_math), aime24 (HuggingFaceH4), aime25 (yentinglin), aime26*, hmmt26*, imo_answerbench* |
| Knowledge / MCQ | mmlu (cais/mmlu), mmlu_pro (TIGER-Lab), cmmlu (haonan-li), gpqa_diamond (Idavidrein/gpqa), arc (allenai/ai2_arc), hellaswag, winogrande |
| QA | trivia_qa (mandarjoshi), drop (ucinlp), simple_qa (mirror of OpenAI's CSV), hle (cais/hle), bbh (lukaemon/bbh) |
| Long context | longbench_v2 (THUDM), openai_mrcr (openai-mirror) |
| Coding | humaneval (openai), bigcodebench (bigcode), live_code_bench (livecodebench) |
| Agent / tools | swe_bench_verified (princeton-nlp), tau2_bench (official GitHub), bfcl_v3 (official GitHub), general_fc (evalscope-native) |

\* aime26 / hmmt26 / imo_answerbench have no official standalone release and
use community curations (evalscope); simple_qa's official artifact is the CSV
in `openai/simple-evals` (HF is a mirror); tau2_bench / bfcl_v3 are officially
released on GitHub — clone and point `--source` at the local files.

## Adding a dataset

Drop a single-file plugin into `evalharness/data/datasets/` — auto-discovered,
no central file to edit.

Well-shaped records (column names map directly) — pure declaration:

```python
# evalharness/data/datasets/cmmlu.py
@register_dataset(DatasetSpec(name='cmmlu', source='haonan-li/cmmlu', split='test', task_type='mcq'))
def cmmlu():
    return FieldSpec(input='question', choices='choices', target='answer', metadata=['category'])
```

Custom conversion — return a `record -> Sample` function:

```python
@register_dataset(DatasetSpec(name='gsm8k', source='openai/gsm8k', subset='main',
                              split='test', task_type='math'))
def gsm8k():
    def to_sample(record):
        parts = record['answer'].split('####')
        return Sample(input=record['question'], target=parts.pop().strip())
    return to_sample
```

Sources supported: local `.jsonl/.json/.csv/.tsv` files; local directories
(probed as `{subset}_{split}.jsonl` etc.); HF hub dataset ids (requires the
`hub` extra, imported lazily); official GitHub releases (clone, then
`--source` the local path).

Conventions: `choices` holds option *contents*; `target` is a **letter** for
MCQ (e.g. `'B'`) and text otherwise (use `List[str]` for multiple gold
answers); long contexts/passages go to `metadata`, not `input`; `id` is
assigned sequentially at materialize time when absent.

## Design decisions

The data layer mirrors conclusions from a close reading of seven evaluation
frameworks (evalscope, lm-evaluation-harness, inspect_ai, deepeval, VLMEvalKit,
harbor, deepseek-harness); see `../README.md` for the full analysis:

- **Data layer vs eval layer: strict separation** (unlike evalscope's
  DataAdapter, which welds record conversion + extraction + scoring + prompt
  into one class). The data layer answers *what is the question/target*; the
  eval layer answers *how to judge a response*. Consequences: predictions can
  be re-scored under a new recipe without re-running the model; extraction
  rules are shared primitives (`math_boxed`, `mcq_letter`, ...) instead of
  per-benchmark copies; official grader fixes (e.g. DROP's Hungarian
  alignment) land in one scorer and apply to every recipe.
- **`Sample` base + one `record_to_sample` per plugin** (evalscope/inspect_ai):
  raw formats vary wildly; the unified format only exists after conversion,
  and everything downstream sees only `Sample`.
- **Datasets as first-class citizens** (inspect_ai): a `Dataset` from
  `get_dataset()` can be filtered/synthesized/exported freely — it is not
  welded into an evaluation recipe.
- **Registration is cheap, materialization pays**: the registry holds only
  metadata + conversion recipes; `list` never touches the network.
- **Config-sensitive cache keys** (evalscope): a cache hit is always correct;
  no invalidation logic exists.
- **Raw predictions are immutable artifacts**: every SampleResult keeps
  raw_prediction; extraction/score details are re-derivable. Failed
  extractions are surfaced (extraction_failure_rate), never silently zeroed.
- **Official grader parity**: scorers replicate official logic where it
  exists (PRM800K math equivalence; DROP's answer_to_bags +
  linear_sum_assignment; SimpleQA's A/B/C judge with NOT_ATTEMPTED fallback
  and accuracy_given_attempted). Official test cases anchor `tests/`.
- **Sandbox/tool fields on `Sample` are declarations only**
  (`sandbox/files/setup/tools`): the data layer never executes; a future
  sandbox layer will materialize them.
- **Data load/unload vs environment load/unload are different layers**:
  `fetch`/`unload` move bytes only (raw files + converted samples). Heavy
  execution environments (e.g. the ~1GB-per-instance `sweb.eval.*` images
  declared by swe_bench_verified) are pulled lazily *at eval time* by the
  sandbox layer — never at data-fetch time — and their removal is
  refcounted there because docker layers are shared across instances and
  benchmarks. `DatasetSpec.requires` (e.g. `['docker']`) is the declaration
  the sandbox/deploy layer reads.

## Layout

```
EvalHarness/
├── pyproject.toml
├── evalharness/
│   ├── cli.py                     # CLI: data list/fetch/unload/stats/show
│   │                              #      eval list/run, viz show
│   ├── data/                      # ---- data layer ----
│   │   ├── sample.py              # Sample / ChatMessage / SandboxSpec / ToolInfo
│   │   ├── spec.py                # DatasetSpec (metadata) / FieldSpec (field mapping)
│   │   ├── registry.py            # Registry + @register_dataset + get_dataset
│   │   ├── loader.py              # raw loading (local/HF/ModelScope native)
│   │   ├── dataset.py             # Dataset: lazy materialize + cache + derived views
│   │   └── datasets/              # 28 built-in single-file dataset plugins
│   ├── model/                     # ---- model layer ----
│   │   ├── output.py              # ModelOutput/ToolCall/Usage (agent hinge)
│   │   ├── adapter.py             # @register_adapter: openai_compatible / mock
│   │   ├── deployer.py            # @register_deployer: vllm / sglang / external
│   │   └── runner.py              # async run_eval(): generate -> evaluate
│   ├── sandbox/                   # ---- environment layer ----
│   │   ├── base.py                # Sandbox iface + refcounted acquire/release + atexit
│   │   ├── docker.py              # exec (isolated) + serve (engines) one impl
│   │   └── local.py               # dev-only, no isolation
│   ├── agent/                     # ---- agent evaluation driver ----
│   │   ├── loop.py                # drive(): message pump + Trajectory
│   │   └── envs/bfcl_mock.py      # BFCL official-style env (tau2/swe later)
│   ├── eval/                      # ---- evaluation layer ----
│   │   ├── record.py              # SampleResult / EvalReport artifacts
│   │   ├── extractor.py           # answer-extraction primitives (+cascades)
│   │   ├── scorer.py              # scoring primitives (official implementations)
│   │   ├── math_grader.py         # PRM800K sympy equivalence (optional [math])
│   │   ├── aggregator.py          # mean/pass@k/grouped/binned/simpleqa_official
│   │   ├── recipe.py              # EvalRecipe bindings + @register_eval
│   │   ├── runner.py              # evaluate(dataset, predictions) -> EvalReport
│   │   └── recipes/               # 28 per-benchmark recipes
│   └── viz/                       # ---- visualization (report consumer) ----
│       └── renderers/text.py      # text / md / md_compare / radar / errors
├── examples/
│   └── data/                      # offline demo subsets (gsm8k/cmmlu, 5 rows each)
└── tests/
    └── test_eval.py               # official-anchor regression tests
```

## Roadmap (not built yet, one layer at a time)

- [x] Data layer (28 dataset plugins, lazy cache, native HF/ModelScope loaders)
- [x] Evaluation layer (extract/score/aggregate plugins, official scorers, recipes)
- [x] Visualization (console/markdown renderers over report artifacts)
- [x] Model layer (async ModelAdapter openai_compatible+mock, ModelOutput
      with tool_calls, Deployer registry vllm/sglang/external + models.yaml
      env pinning, run_eval generate->score)
- [x] Sandbox layer (docker exec hard-isolation + serve environments,
      refcounted acquire/release, atexit teardown, images kept, bind-mount
      host sharing; Deployer now consumes it)
- [x] Agent evaluation driver (message pump + Trajectory + bfcl_mock env
      with official call-sequence scoring; tau2/swe envs pending)
- [ ] Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
- [ ] Skill layer (full evaluation pipelines as composable skills)
- [ ] tau2 / swe-bench environments (user simulator; per-instance sweb.* images)
- [ ] Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
- [ ] Web/API interface
- [ ] Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
- [ ] Skill layer (full evaluation pipelines as composable skills)
- [ ] Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
- [ ] Web/API interface