evalstone/evalscope/AGENTS.md

AGENTS.md

EvalScope — LLM evaluation framework with a registry-based plugin architecture. This file is the contract for AI coding agents working in this repo.

Setup

pip install -e .       # basic install
make dev               # dev + perf + docs extras + pre-commit

Python ≥ 3.10 (3.10 / 3.11 / 3.12). Dependencies: requirements/framework.txt + pyproject.toml [project.optional-dependencies] (extras: opencompass, vlmeval, rag, perf, app, aigc, sandbox, service, dev, docs, all, plus per-benchmark extras).

Build, lint, test

make lint                                                                       # apply Ruff fixes/formatting and run all pre-commit checks
pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings        # CI smoke test
pytest tests/perf/test_perf_basic.py::TestPerfBasic::test_multi_parallel_sweep -v -s    # perf

Commits failing make lint are rejected on main.

Docs generation

Benchmark detail pages (docs/{zh,en}/benchmarks/<name>.md) and meta cache (evalscope/benchmarks/_meta/<name>.json) are auto-generated from each adapter's BenchmarkMeta.description + dataset statistics. Do not hand-edit those files.

Every BenchmarkMeta.description must be English Markdown with these sections in this order:

1. ## Overview: benchmark purpose and scope.
2. ## Task Description: bullet fields for Task Type, Input, Output, and Domain (use a more precise fourth field such as Modalities or Grading only when Domain does not apply).
3. ## Key Features: dataset scale/source, evaluated capabilities, and version-specific behavior.
4. ## Evaluation Notes: metrics, scoring procedure, runtime/dependency requirements, and compatibility limits.

Do not replace these required headings with benchmark-specific headings. Add extra sections only when the four required sections are insufficient.

When you add a benchmark or change its BenchmarkMeta.description, run:

make docs-pipeline BENCHMARK="<name1> <name2>" FORCE=1   # update _meta JSON + translate descriptions to zh
make docs-generate                                        # render .md files from _meta

Targets: docs-update (meta only), docs-update-stats (+ dataset statistics), docs-translate (zh), docs-pipeline (stats + translate), docs-generate (.md), docs (full Sphinx HTML build).

Conventions:

• BENCHMARK="a b c" selects benchmarks; omit for --all.
• FORCE=1 appends --force to recompute even if data is cached.
• WORKERS=N parallelism (default 4).
• --translate calls an LLM; needs DASHSCOPE_API_KEY (or equivalent) in env.

Quick eval

evalscope eval --model Qwen/Qwen2.5-0.5B-Instruct --datasets gsm8k --limit 5

from evalscope import run_task, TaskConfig
run_task(TaskConfig(model='Qwen/Qwen2.5-0.5B-Instruct', datasets=['gsm8k'], limit=5))

Code style (enforced)

• Line width 120, 4-space indent, LF endings, trailing newline at EOF.
• Quotes: single quotes, enforced by the Ruff formatter.
• Linting: Ruff's E, F, and W rules for maintained source files.
• Imports: Ruff's I rules, with evalscope detected as first-party and standard import sections.
• f-strings for formatting (no % or .format() unless necessary).
• Type hints required on every function signature.
• English only for comments and docstrings.
• Public APIs need docstrings; internal helpers only when intent is non-obvious.
• # TODO: prefix for pending work.

Element	Style
Class	PascalCase
Function / variable	snake_case
Constant	UPPER_SNAKE_CASE
Private	_leading_underscore
Handler function	handle_ prefix
Benchmark adapter file	<name>_adapter.py

Ruff ignore list (pyproject.toml): E501, F401. Do not expand — new ignores must be justified in the PR.

Design rules

• Early returns over nested conditionals.
• Minimal changes: only touch code related to the current task; no drive-by cleanup.
• Pydantic-first: cross-module data contracts use Pydantic models. Use TaskConfig / Arguments for configuration — never raw dicts at module boundaries.
• Web API responses: successful JSON responses consumed by the dashboard use models from evalscope/service/api_models/ and json_response(). Regenerate frontend contracts with cd evalscope/web && npm run contracts:generate; never hand-edit generated artifacts or add parallel response schemas.
• Reuse existing patterns: new benchmarks / models / metrics go through existing registries and adapter base classes — no parallel mechanisms.
• DRY but don't over-abstract just to remove minor duplication.

Tests

• Live under tests/; files *test*.py, classes Test*, functions test_*.
• New benchmark / model / metric must ship a minimal runnable test (pattern: tests/cli/test_all.py::TestRun::test_ci_lite).
• Mock external services — no reliance on real network / paid APIs.

Architecture pointers

Don't try to learn the architecture from this file — read these and grep:

Topic	Source of truth
Main flow	evalscope/run.py → evalscope/evaluator/evaluator.py
Config schema	evalscope/config.py (TaskConfig)
Registries	evalscope/api/registry.py
Benchmark contract	evalscope/api/benchmark/benchmark.py (DataAdapter, BenchmarkMeta)
Model layer	evalscope/api/model/model.py, evalscope/models/model_apis.py
CLI dispatch	evalscope/cli/
Cache schema	evalscope/api/evaluator/cache.py
Web API response contracts	evalscope/service/api_models/, evalscope/service/responses.py, evalscope/web/src/api/generated/

Registry decorators: @register_benchmark, @register_model_api, @register_metric, @register_aggregation, @register_filter, @register_evaluator.

Adapter base classes (extend, don't reinvent): DefaultDataAdapter, MultiChoiceAdapter, VisionLanguageAdapter, Text2ImageAdapter, ImageEditAdapter, NERAdapter, AgentAdapter. Optional capabilities via mixins: LLMJudgeMixin, CodeExecutionSandboxMixin.

Non-native backends live under evalscope/backend/ (OpenCompass, VLMEvalKit, RAGEval) and are dispatched from run.py with their own BackendManager.

Adding a benchmark

1. Create evalscope/benchmarks/<name>/<name>_adapter.py.
2. Extend DefaultDataAdapter, override record_to_sample() (and optionally sample_to_fewshot(), extract_answer()).
3. Reuse the standard dataset flow (load_subset() and existing DataLoader implementations) for shuffle, limit, repeats, filtering, conversion, and indexing. Override the full load() flow only when the standard loaders cannot represent the source format, and keep custom loading limited to benchmark-specific parsing or validation.
4. Use download_dataset_file() or download_dataset_snapshot() for benchmark media and raw files; do not duplicate hub resolution, cache, path-safety, or download state inside an adapter.
5. Decorate with @register_benchmark(BenchmarkMeta(name=..., ...)).
6. Auto-discovered by globbing evalscope/benchmarks/*/**/*_adapter.py.
7. Add a smoke test.

Evaluation versioning

BenchmarkMeta.evaluation_version is the published version of a benchmark's evaluation semantics. New benchmarks must declare their initial version explicitly. Raise the minor version when data, sample conversion, default prompt, choice/target mapping, default scoring/judge, or aggregation semantics change; raise the major version for a rename or task-definition replacement. Documentation, tests, and pure refactors do not change it.

Adding a judge-scored benchmark

An adapter must never call self.llm_judge.judge() or parse a judge reply itself — that debt is fenced off by tests/api/judge/test_gates.py, which scans every file under evalscope/benchmarks/ (helpers included, so moving a parser into utils.py does not evade it). Score through the JSON output contract in evalscope/api/judge/ instead:

1. Pick a scoring_policy (JUDGE_ONLY or JUDGE_DEFAULT). Judge scoring always goes through the contract; there is no opt-in flag and no legacy path.
2. Single verdict per sample: implement judge_definition(context) and return JudgeDefinition.labels(...) for a label mapping or JudgeDefinition.numeric(...) for a 0-1 rating. A generic prompt_template must state grading criteria only: OutputContract.instruction() appends the reply format. An adapter that preserves an official fixed output template may keep that format instruction instead, provided its OutputContract schema matches the official template.
3. Custom shape (multiple cases, ratings, rubrics): judge_definition(context) declares a Pydantic schema_model, wraps it in OutputContract, and returns JudgeDefinition.workflow(...):
   • cases contains one JudgeCase(case_id, output_contract, metadata) per thing to judge.
   • request(case, placement, completed, context) renders messages and appends case.output_contract.instruction() so the prompt and parser cannot drift. An official fixed output template may be used instead when its required fields and constraints match the case's OutputContract schema.
   • reduce(verdicts, context) folds parsed verdicts into {metric: value}. Read a verdict's context from CaseVerdict.metadata, never by parsing case_id.
   • Optional expand, fallback, and finalize callbacks handle staged cases, rule fallbacks, and score finalization. They may be nested functions or private adapter helpers, but are passed only through the returned definition.
4. Rule short-circuit: if deterministic scoring settles the sample before judge I/O, return JudgeDefinition.skip(score, reason='...'). The non-empty reason is persisted in Score.metadata as judge_skipped=True and judge_skip_reason; the web review panel displays it as rule-based scoring.
5. The executor owns request execution, position swap, repeats, multi-judge aggregation and fail-closed exclusion. Transport retries belong to the model implementation; a reply that fails the contract is not automatically retried and excludes the sample from the metric — never scored 0 or full credit — so a metric's num can be below the sample count.
6. Add a scripted-judge test in tests/api/judge/test_migrated_adapters.py covering: a valid verdict, a parse failure (prose / malformed), and a transport [ERROR] — each must exclude, not silently score. A judge double must carry the surface the definition reads (score_type, score_mapping, build_prompt), and be injected through the llm_judge setter rather than a private attribute.

Conventions & gotchas

• eval_type: openai_api, llm_ckpt, mock_llm, text2image, image_editing. Deprecated aliases: server → openai_api, checkpoint → llm_ckpt.
• limit: int = count, float = fraction.
• repeats: duplicates items for k-metrics. generation_config.n is deprecated and mapped.
• Use generation_config for runtime params. TaskConfig.timeout / stream are deprecated — forwarded with a warning.
• dataset_args merges into BenchmarkMeta._update() (supports local_path, filters OrderedDict prepended).
• Models are memoized by (name, eval_type, config, base_url, api_key, args).
• Use @thread_safe for model creation, run_in_threads_with_progress for concurrent eval.
• Outputs land in outputs/<timestamp>/{logs,predictions,reviews,reports,configs}/ (see OutputsStructure). use_cache resumes runs; rerun_review recomputes scores only.
• evalscope app CLI command is deprecated (see evalscope/cli/start_app.py) — use evalscope service for the Web dashboard.

Submission

make dev      # once
make lint     # apply fixes and run all checks before every commit
pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings