sora/evalstone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

readme：如下

Browse Source
This commit is contained in:
sora 2026-07-08 08:56:47 +00:00
commit d7547f642a
2171 changed files with 331424 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
.gitignore vendored Normal file
View File

@ -0,0 +1,34 @@
# 本地数据集缓存
/bash/dataset/*
# Python 缓存
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
# 环境
.env
.venv
venv/
env/
# 评测输出结果
/temp/
/outputs/
/output/
# 日志
*.log
# 密钥/配置文件
*.secret
*.key
api_keys.json
config_private.py
# 前端构建产物
node_modules/
*.zip
*.tar.gz

177
README.md Normal file
View File

@ -0,0 +1,177 @@
# Sora Benchmarks
## 1. 环境安装
```bash
conda create -n evalscope python==3.12 -y
conda activate evalscope
cd evalscope
pip install -e .
# 安装特定 benchmark 依赖
pip install 'evalscope[terminal_bench,swe_bench,openai_mrcr]' \
'git+https://github.com/sierra-research/tau2-bench@v0.2.0'
pip install 'evalscope[sandbox]'
pip install 'swebench==4.1.0'
```
## 2. 常用 Benchmarks
```python
benchmark_categories = {
"代码与工程": [
"terminal_bench_v2",
"live_code_bench",
"swe_bench_multilingual_agentic",
"swe_bench_pro",
"swe_bench_verified",
"bigcodebench",
"humaneval",
],
"推理与数学": [
"gpqa_diamond",
"hle",
"aime24",
"aime25",
"aime26",
"hmmt26",
"imo_answerbench",
"super_gpqa",
"gsm8k",
"competition_math",
],
"智能体与工具": [
"browsecomp",
"mcp_atlas",
"tau2_bench",
],
"知识与语言理解": [
"mmlu_pro",
"simple_qa",
"arc",
"bbh",
"cmmlu",
"drop",
"hellaswag",
"mmlu",
"trivia_qa",
"winogrande",
],
"长上下文": [
"longbench_v2",
"openai_mrcr",
],
}
```
## 3. 分组运行
```python
group_1 = [
'terminal_bench_v2',
'gpqa_diamond',
'hle',
'aime24',
'aime25',
'mmlu_pro',
'simple_qa',
'arc',
'bbh',
'browsecomp',
]
group_2 = [
'live_code_bench',
'swe_bench_pro',
'aime26',
'hmmt26',
'imo_answerbench',
'super_gpqa',
'drop',
'hellaswag',
'mmlu',
'openai_mrcr',
'mcp_atlas',
]
group_3 = [
'swe_bench_multilingual_agentic',
'swe_bench_verified',
'bigcodebench',
'humaneval',
'gsm8k',
'competition_math',
'cmmlu',
'trivia_qa',
'winogrande',
'longbench_v2',
'tau2_bench',
]
```
## 4. 运行示例
```python
from evalscope import run_task, TaskConfig
datasets = [
'live_code_bench',
'swe_bench_pro',
'aime26',
'hmmt26',
'imo_answerbench',
'super_gpqa',
'drop',
'hellaswag',
'mmlu',
'openai_mrcr',
'mcp_atlas',
]
task_cfg = TaskConfig(
collect_perf=True,
work_dir='/data1/sora/benchmarks/temp/output',
use_cache='/data1/sora/benchmarks/temp/output',
no_timestamp=True,
model='',
api_url='http://localhost:30000/v1',
eval_type='openai_api',
datasets=datasets,
dataset_dir='/data1/sora/benchmarks/bash/datasets',
generation_config={
'temperature': 0.0,
'stream': True,
},
eval_batch_size=1,
judge_model_args={
"model_id": "deepseek-v4-flash",
"api_url": "https://api.deepseek.com/v1",
"api_key": "<YOUR_DEEPSEEK_API_KEY>",
"eval_type": "openai_api",
"generation_config": {
"temperature": 0.0,
"max_tokens": 1024 * 10,
},
}
)
run_task(task_cfg)
```
## 5. 评测结果
P800 模型能力评测结果保存在 `temp/output_*` 目录下。
## 6. UI 界面
```bash
pip install flask sse_starlette
cd evalscope/web
npm install
npm run build
cd /data1/sora/benchmarks
evalscope service
```
访问 `http://127.0.0.1:9000/dashboard`。

52
bash/test.py Normal file
View File

@ -0,0 +1,52 @@
from evalscope import run_task, TaskConfig
datasets = [
# 'live_code_bench',
'aime26',
# 'hmmt26',
# 'imo_answerbench',
# 'super_gpqa',
# 'drop',
# 'hellaswag',
# 'mmlu',
# 'openai_mrcr',
# # 'swe_bench_pro',
# 'mcp_atlas',
]
batch_size_list = [1, 2, 4, 8, 16, 32, 64,128]
for batch_size in batch_size_list:
work_dir = f'/data1/sora/benchmarks/temp/output_{batch_size}'
task_cfg = TaskConfig(
seed=42,
collect_perf= True,
work_dir=work_dir,
use_cache=work_dir,
no_timestamp=True,
model='DeepSeek-V4-Flash-Int8',
api_url='http://localhost:30000/v1',
# api_key='',
eval_type='openai_api',
datasets=datasets,
dataset_dir='/data1/sora/benchmarks/bash/datasets',
generation_config = {
'temperature': 0.0,
'stream': True,
"max_tokens": 1024 * 32,
},
dataset_args={
'live_code_bench': {
'subset_list': ['release_v6']
}
},
eval_batch_size = batch_size,
judge_model_args = {
"model_id": "deepseek-v4-pro",
"api_url": "https://api.deepseek.com/v1",
"api_key": "sk-9ed86ef546ca47e3afa7c3b014dea268",
"eval_type": "openai_api",
"generation_config": {
"temperature": 0.0,
"max_tokens": 1024 * 10,
},
}
)
run_task(task_cfg)

47
evalscope/.github/ISSUE_TEMPLATE/1-issue-模版-中文版-.md vendored Normal file
View File

@ -0,0 +1,47 @@
---
name: 1-Issue 模版（中文版）
about: 模型评测遇到的问题
title: ''
labels: ''
assignees: ''
---
## 自查清单
在提交 issue 之前，请确保您已完成以下步骤:
- [ ] 我已仔细阅读了[相关使用说明文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)
- [ ] 我已查看了[常见问题解答](https://evalscope.readthedocs.io/zh-cn/latest/get_started/faq.html)
- [ ] 我已搜索并查看了现有的 issues，确认这不是一个重复的问题
## 问题描述
请简要描述您遇到的问题。
## EvalScope 版本（必填）
v0.xx.x
## 使用的工具
- [ ] Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena / 竞技场模式
## 执行的代码或指令
请提供您执行的主要代码或指令。
## 错误日志
请粘贴完整的错误日志或控制台输出。
## 运行环境
- 操作系统：
- Python版本：
## 其他信息
如果有其他相关信息，请在此处提供。

47
evalscope/.github/ISSUE_TEMPLATE/2-issue-template--english-version-.md vendored Normal file
View File

@ -0,0 +1,47 @@
---
name: 2-Issue template (English Version)
about: Issues encountered during model evaluation
title: ''
labels: ''
assignees: ''
---
## Self-Check List
Before submitting an issue, please ensure you have completed the following steps:
- [ ] I have carefully read the [relevant user documentation](https://evalscope.readthedocs.io/en/latest/get_started/parameters.html)
- [ ] I have reviewed the [Frequently Asked Questions](https://evalscope.readthedocs.io/en/latest/get_started/faq.html)
- [ ] I have searched and reviewed existing issues to confirm this is not a duplicate problem
## Problem Description
Please briefly describe the problem you encountered.
## EvalScope Version (Required)
v0.xx.x
## Tools Used
- [ ] Native / Native framework
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / Model inference stress testing tool
- [ ] Arena / Arena mode
## Executed Code or Instructions
Please provide the main code or instructions you executed.
## Error Log
Please paste the complete error log or console output.
## Running Environment
- Operating System:
- Python Version:
## Additional Information
If there is any other relevant information, please provide it here.

24
evalscope/.github/ISSUE_TEMPLATE/3-feature-request.md vendored Normal file
View File

@ -0,0 +1,24 @@
---
name: 3-Feature request
about: Suggest an idea for this project
title: ''
labels: enhancement
assignees: ''
---
## 功能描述 / Feature Description
请简要描述您希望添加的功能。 / Please briefly describe the feature you would like to request.
## 需求背景 / Background
为什么需要这个功能？ / Why is this feature needed?
## 预期行为 / Expected Behavior
这个功能应该如何工作？ / How should this feature work?
## 其他信息 / Additional Information
还有其他相关信息吗？ / Any other relevant information?

12
evalscope/.github/SECURITY.md vendored Normal file
View File

@ -0,0 +1,12 @@
# Security Policy
The EvalScope team and community take all security bugs in EvalScope seriously. We appreciate your efforts and responsible disclosure and will make every effort to acknowledge your contributions.
## Reporting a Vulnerability
To report a security issue, please use the **[GitHub Security Advisory "Report a Vulnerability" tab](https://github.com/modelscope/evalscope/security/advisories/new)**.
**Please do not report security vulnerabilities through public GitHub issues.**
You should receive a response within 48 hours. If for some reason you do not, please follow up to ensure we received your original message.
When reporting a vulnerability, please include the following information to help us better understand the issue:
- A description of the vulnerability
- Steps to reproduce the issue
- The potential impact of the vulnerability
- Any known mitigations or workarounds

1
evalscope/.github/copilot-instructions.md vendored Symbolic link
View File

@ -0,0 +1 @@
../AGENTS.md

71
evalscope/.github/workflows/ci_test.yaml vendored Normal file
View File

@ -0,0 +1,71 @@
name: CI Tests Lite
on:
push:
branches:
- main
pull_request:
branches:
- main
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
unit-tests:
runs-on: ubuntu-latest
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e '.[dev]'
- name: Create .env file
run: |
echo "DASHSCOPE_API_KEY=${{ secrets.DASHSCOPE_API_KEY }}" > .env
- name: Run integration tests
run: |
python -m pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings
perf-tests:
runs-on: ubuntu-latest
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e '.[dev,perf]'
- name: Create .env file
run: |
echo "DASHSCOPE_API_KEY=${{ secrets.DASHSCOPE_API_KEY }}" > .env
- name: Run perf test
run: |
python -m pytest tests/perf/test_perf_basic.py::TestPerfBasic::test_multi_parallel_sweep -v -s -p no:warnings

73
evalscope/.github/workflows/ci_test_full.yaml vendored Normal file
View File

@ -0,0 +1,73 @@
name: CI Tests Full
on:
push:
branches:
- "release/**"
pull_request:
branches:
- "release/**"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
unit-tests:
runs-on: ubuntu-latest
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e '.[dev,ifeval,ifbench,multi_if,needle_haystack,arena_hard]'
pip install git+https://github.com/sierra-research/tau-bench
pip install bfcl-eval==2025.10.27.1
- name: Create .env file
run: |
echo "DASHSCOPE_API_KEY=${{ secrets.DASHSCOPE_API_KEY }}" > .env
- name: Run integration tests
run: |
python -m pytest tests/cli/test_all.py::TestRun::test_benchmarks -v -s -p no:warnings
perf-tests:
runs-on: ubuntu-latest
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e '.[dev,perf]'
- name: Create .env file
run: |
echo "DASHSCOPE_API_KEY=${{ secrets.DASHSCOPE_API_KEY }}" > .env
- name: Run perf test
run: |
python -m pytest tests/perf/test_perf_basic.py::TestPerfBasic::test_multi_parallel_sweep -v -s -p no:warnings

32
evalscope/.github/workflows/lint.yml vendored Normal file
View File

@ -0,0 +1,32 @@
name: Lint Test
on:
push:
branches:
- main
pull_request:
branches:
- main
jobs:
run-pre-commit:
runs-on: ubuntu-latest
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install pre-commit
run: |
python -m pip install --upgrade pip
pip install pre-commit
- name: Run pre-commit
run: pre-commit run --all-files

37
evalscope/.github/workflows/publish.yaml vendored Normal file
View File

@ -0,0 +1,37 @@
name: release
on:
push:
tags:
- 'v**'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-publish
cancel-in-progress: true
jobs:
build-n-publish:
runs-on: ubuntu-22.04
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
#if: startsWith(github.event.ref, 'refs/tags')
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '22'
- name: Build frontend
run: make web-build
- name: Install wheel
run: pip install wheel && pip install -r requirements/framework.txt
- name: Build EvalScope
run: python setup.py sdist bdist_wheel
- name: Publish package to PyPI
run: |
pip install twine
twine upload dist/* --skip-existing -u __token__ -p ${{ secrets.PYPI_TOKEN }}

184
evalscope/.gitignore vendored Normal file
View File

@ -0,0 +1,184 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
!evalscope/web/src/lib/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
/package
/temp
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.vscode
.idea
# custom
*.pkl
*.pkl.json
*.log.json
*.whl
*.tar.gz
*.swp
*.log
*.tar.gz
source.sh
tensorboard.sh
.DS_Store
replace.sh
result.png
result.jpg
result.mp4
*.itag
# Pytorch
*.pth
*.pt
# personal info
private/
# others
*.tokenization
# outputs
outputs/
evalscope/outputs/
evalscope/*_temp.py
evalscope/data/
*/output_res/*
/tmp
/data*
*.zip
output/
/test*.*
*.ttf
*.zip
_build/
swift.test*
/cache
evalscope/backend/rag_eval/ragas/prompts/chinese
logs
.gradio
*.h5
CLAUDE.md
Tool/
docs/zh/learn/
examples/api_test/
.qoder
.claude
# Frontend (evalscope/web)
node_modules/
evalscope/web/dist/
evalscope/web/.vite/
ragas_mini_testset_score.json
# User benchmark outputs and secrets
benchmark_outputs/
test_scripts/
*.secret
*.key
api_keys.json
# Docker build artifacts (pre-cloned agent sources)
evalscope/agent/external/dockerfiles/hermes-agent-src/
evalscope/agent/external/dockerfiles/hermes-install.sh

52
evalscope/.pre-commit-config.yaml Normal file
View File

@ -0,0 +1,52 @@
repos:
- repo: https://github.com/pycqa/flake8.git
rev: 7.3.0
hooks:
- id: flake8
exclude: |
(?x)^(
examples/|
docs/|
tests/|
evalscope/utils/utils.py|
evalscope/third_party/|
evalscope/backend/rag_eval/clip_benchmark/tasks|
evalscope/backend/rag_eval/cmteb/tasks|
evalscope/metrics/vision/t2v_metrics
)
- repo: https://github.com/PyCQA/isort.git
rev: 7.0.0
hooks:
- id: isort
- repo: https://github.com/google/yapf
rev: v0.43.0
hooks:
- id: yapf
exclude: |
(?x)^(
examples/|
docs/|
tests/|
evalscope/utils/utils.py|
evalscope/third_party/|
evalscope/backend/rag_eval/clip_benchmark/tasks|
evalscope/backend/rag_eval/cmteb/tasks
)
- repo: https://github.com/pre-commit/pre-commit-hooks.git
rev: v6.0.0
hooks:
- id: trailing-whitespace
exclude: thirdparty/|docs/|examples
- id: check-yaml
exclude: thirdparty/|docs/|examples
- id: end-of-file-fixer
exclude: thirdparty/|docs/|examples|.*\.json
- id: requirements-txt-fixer
exclude: thirdparty/|docs/|examples
- id: double-quote-string-fixer
exclude: thirdparty/|docs/|examples|.*\.json|cl_bench_adapter.py
- id: check-merge-conflict
exclude: thirdparty/|docs/|examples
- id: mixed-line-ending
exclude: thirdparty/|docs/|examples
args: ["--fix=lf"]

136
evalscope/AGENTS.md Normal file
View File

@ -0,0 +1,136 @@
# AGENTS.md
EvalScope — LLM evaluation framework with a registry-based plugin architecture. This file is the contract for AI coding agents working in this repo.
## Setup
```bash
pip install -e . # basic install
make dev # dev + perf + docs extras + pre-commit
```
Python ≥ 3.10 (3.10 / 3.11 / 3.12). Dependencies: `requirements/framework.txt` + `pyproject.toml [project.optional-dependencies]` (extras: `opencompass`, `vlmeval`, `rag`, `perf`, `app`, `aigc`, `sandbox`, `service`, `dev`, `docs`, `all`, plus per-benchmark extras).
## Build, lint, test
```bash
make lint # required before commit (yapf + isort + flake8 + basic pre-commit hooks)
pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings # CI smoke test
pytest tests/perf/test_perf_basic.py::TestPerfBasic::test_multi_parallel_sweep -v -s # perf
```
Commits failing `make lint` are rejected on `main`.
## Docs generation
Benchmark detail pages (`docs/{zh,en}/benchmarks/<name>.md`) and meta cache (`evalscope/benchmarks/_meta/<name>.json`) are **auto-generated** from each adapter's `BenchmarkMeta.description` + dataset statistics. Do not hand-edit those files.
When you add a benchmark or change its `BenchmarkMeta.description`, run:
```bash
make docs-pipeline BENCHMARK="<name1> <name2>" FORCE=1 # update _meta JSON + translate descriptions to zh
make docs-generate # render .md files from _meta
```
Targets: `docs-update` (meta only), `docs-update-stats` (+ dataset statistics), `docs-translate` (zh), `docs-pipeline` (stats + translate), `docs-generate` (.md), `docs` (full Sphinx HTML build).
Conventions:
- `BENCHMARK="a b c"` selects benchmarks; omit for `--all`.
- `FORCE=1` appends `--force` to recompute even if data is cached.
- `WORKERS=N` parallelism (default 4).
- `--translate` calls an LLM; needs `DASHSCOPE_API_KEY` (or equivalent) in env.
## Quick eval
```bash
evalscope eval --model Qwen/Qwen2.5-0.5B-Instruct --datasets gsm8k --limit 5
```
```python
from evalscope import run_task, TaskConfig
run_task(TaskConfig(model='Qwen/Qwen2.5-0.5B-Instruct', datasets=['gsm8k'], limit=5))
```
## Code style (enforced)
- **Line width 120**, 4-space indent, LF endings, trailing newline at EOF.
- **Quotes** governed by `double-quote-string-fixer` hook — follow existing file style; do not mix.
- **f-strings** for formatting (no `%` or `.format()` unless necessary).
- **Imports**: isort with `first_party = evalscope`, groups `STDLIB / THIRDPARTY / LOCALFOLDER`, `multi_line_output=3`.
- **Type hints required** on every function signature.
- **English only** for comments and docstrings.
- **Public APIs need docstrings**; internal helpers only when intent is non-obvious.
- `# TODO:` prefix for pending work.
| Element | Style |
| --- | --- |
| Class | `PascalCase` |
| Function / variable | `snake_case` |
| Constant | `UPPER_SNAKE_CASE` |
| Private | `_leading_underscore` |
| Handler function | `handle_` prefix |
| Benchmark adapter file | `<name>_adapter.py` |
**flake8 ignore list** (`setup.cfg`): `F401, F403, F405, F821, W503, E251, W504, F824, F541, E501, E226, E121-E129, E131, E741`. Do not expand — new ignores must be justified in the PR.
## Design rules
- **Early returns** over nested conditionals.
- **Minimal changes**: only touch code related to the current task; no drive-by cleanup.
- **Pydantic-first**: cross-module data contracts use Pydantic models. Use `TaskConfig` / `Arguments` for configuration — never raw dicts at module boundaries.
- **Reuse existing patterns**: new benchmarks / models / metrics go through existing registries and adapter base classes — no parallel mechanisms.
- **DRY** but don't over-abstract just to remove minor duplication.
## Tests
- Live under `tests/`; files `*test*.py`, classes `Test*`, functions `test_*`.
- New benchmark / model / metric **must** ship a minimal runnable test (pattern: `tests/cli/test_all.py::TestRun::test_ci_lite`).
- Mock external services — no reliance on real network / paid APIs.
## Architecture pointers
Don't try to learn the architecture from this file — read these and grep:
| Topic | Source of truth |
| --- | --- |
| Main flow | `evalscope/run.py` → `evalscope/evaluator/evaluator.py` |
| Config schema | `evalscope/config.py` (`TaskConfig`) |
| Registries | `evalscope/api/registry.py` |
| Benchmark contract | `evalscope/api/benchmark/benchmark.py` (`DataAdapter`, `BenchmarkMeta`) |
| Model layer | `evalscope/api/model/model.py`, `evalscope/models/model_apis.py` |
| CLI dispatch | `evalscope/cli/` |
| Cache schema | `evalscope/api/evaluator/cache.py` |
**Registry decorators**: `@register_benchmark`, `@register_model_api`, `@register_metric`, `@register_aggregation`, `@register_filter`, `@register_evaluator`.
**Adapter base classes** (extend, don't reinvent): `DefaultDataAdapter`, `MultiChoiceAdapter`, `VisionLanguageAdapter`, `Text2ImageAdapter`, `ImageEditAdapter`, `NERAdapter`, `AgentAdapter`. Optional capabilities via mixins: `LLMJudgeMixin`, `SandboxMixin`.
**Non-native backends** live under `evalscope/backend/` (OpenCompass, VLMEvalKit, RAGEval) and are dispatched from `run.py` with their own BackendManager.
## Adding a benchmark
1. Create `evalscope/benchmarks/<name>/<name>_adapter.py`.
2. Extend `DefaultDataAdapter`, override `record_to_sample()` (and optionally `sample_to_fewshot()`, `extract_answer()`).
3. Decorate with `@register_benchmark(BenchmarkMeta(name=..., ...))`.
4. Auto-discovered by globbing `evalscope/benchmarks/*/**/*_adapter.py`.
5. Add a smoke test.
## Conventions & gotchas
- `eval_type`: `openai_api`, `llm_ckpt`, `mock_llm`, `text2image`, `image_editing`. Deprecated aliases: `server` → `openai_api`, `checkpoint` → `llm_ckpt`.
- `limit`: `int` = count, `float` = fraction.
- `repeats`: duplicates items for k-metrics. `generation_config.n` is deprecated and mapped.
- Use `generation_config` for runtime params. `TaskConfig.timeout` / `stream` are deprecated — forwarded with a warning.
- `dataset_args` merges into `BenchmarkMeta._update()` (supports `local_path`, `filters` OrderedDict prepended).
- Models are memoized by `(name, config, base_url, api_key, args)`.
- Use `@thread_safe` for model creation, `run_in_threads_with_progress` for concurrent eval.
- Outputs land in `outputs/<timestamp>/{logs,predictions,reviews,reports,configs}/` (see `OutputsStructure`). `use_cache` resumes runs; `rerun_review` recomputes scores only.
- `evalscope app` CLI command is **deprecated** (see `evalscope/cli/start_app.py`) — use `evalscope service` for the Web dashboard.
## Submission
```bash
make dev # once
make lint # before every commit
pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings
```

355
evalscope/CONTRIBUTING.md Normal file
View File

@ -0,0 +1,355 @@
# Contributing to EvalScope
Thank you for considering contributing to EvalScope! This guide covers everything you need to get started.
## Table of Contents
- [Contributing to EvalScope](#contributing-to-evalscope)
- [Table of Contents](#table-of-contents)
- [Quick Start](#quick-start)
- [Development Setup](#development-setup)
- [Backend (Python)](#backend-python)
- [Frontend (React + Vite)](#frontend-react--vite)
- [Full-Stack Development](#full-stack-development)
- [Project Structure](#project-structure)
- [Adding a New Benchmark](#adding-a-new-benchmark)
- [Step 1: Create the adapter directory](#step-1-create-the-adapter-directory)
- [Step 2: Write the adapter](#step-2-write-the-adapter)
- [Key methods you can override](#key-methods-you-can-override)
- [BenchmarkMeta key fields](#benchmarkmeta-key-fields)
- [Step 3: Add extra dependencies (if any)](#step-3-add-extra-dependencies-if-any)
- [Step 4: Update documentation (optional)](#step-4-update-documentation-optional)
- [Verify your benchmark](#verify-your-benchmark)
- [Code Quality](#code-quality)
- [Linting](#linting)
- [Testing](#testing)
- [Git Workflow](#git-workflow)
---
## Quick Start
```bash
# 1. Fork & clone
git clone https://github.com/<your-username>/evalscope.git
cd evalscope
# 2. Install in editable mode with dev dependencies
make dev
# 3. Install pre-commit hooks
pre-commit install
```
---
## Development Setup
### Backend (Python)
EvalScope requires **Python >= 3.10**.
```bash
# Base install (editable)
pip install -e .
# With all dev tools
pip install -e '.[dev,perf,docs]'
# With the web service
pip install -e '.[service]'
```
**Run the backend service:**
```bash
evalscope service --host 0.0.0.0 --port 9000
```
Optional dependency groups (install via `pip install -e '.[<group>]'`):
| Group | Purpose | Key Packages |
|-------|---------|-------------|
| `dev` | Testing & linting | pytest, pytest-cov |
| `service` | Web dashboard & REST API | flask, plotly |
| `perf` | Performance benchmarking | — |
| `docs` | Documentation build | sphinx |
| `rag` | RAG evaluation | — |
| `aigc` | AIGC evaluation | — |
| `sandbox` | Sandboxed code execution | ms-enclave |
Some benchmarks have their own extra dependencies (e.g. `pip install -e '.[bfcl]'`).
### Frontend (React + Vite)
The dashboard is a React SPA located at `evalscope/web/`.
```bash
# Install dependencies
make web-install
# Start dev server (hot reload, proxies API to localhost:9000)
make web-dev
# Production build
make web-build
```
The dev server runs at `http://localhost:5173` and automatically proxies `/api/v1/*` and `/health` to the backend at `http://127.0.0.1:9000`.
**Tech stack:** React 19 · TypeScript · Vite · Tailwind CSS 4 · React Router · Plotly.js
### Full-Stack Development
For the best development experience, run both servers simultaneously:
```bash
# Terminal 1: Backend
evalscope service --debug
# Terminal 2: Frontend (hot reload)
make web-dev
```
Open `http://localhost:5173` in your browser — changes to frontend code are reflected instantly.
---
## Project Structure
```
evalscope/
├── api/ # Core API: registry, benchmark base classes, dataset, metric, model
│ ├── benchmark/ # DataAdapter, BenchmarkMeta, adapter subclasses
│ ├── dataset/ # Dataset loading, Sample dataclass
│ ├── evaluator/ # TaskState, evaluation loop
│ ├── messages/ # Chat message types
│ ├── metric/ # Score, AggScore, metric registry
│ ├── model/ # Model abstraction (OpenAI-compatible)
│ └── registry.py # register_benchmark(), BENCHMARK_REGISTRY
│
├── benchmarks/ # All benchmark adapters (auto-discovered)
│ └── <name>/
│ ├── __init__.py
│ └── <name>_adapter.py
│
├── cli/ # CLI entry points (evalscope eval/perf/service/app)
├── constants.py # Global constants & tags
├── perf/ # Performance benchmarking subsystem
├── report/ # Report generation & visualization
├── service/ # Flask REST API + SPA serving
│ ├── app.py # Flask app factory, run_service()
│ └── blueprints/ # API route handlers (eval, perf, reports)
├── utils/ # Shared utilities (logging, IO, etc.)
└── web/ # React SPA (dashboard UI)
├── src/
│ ├── api/ # API client & type definitions
│ ├── components/ # UI components
│ ├── pages/ # Route pages
│ └── i18n/ # Internationalization
└── vite.config.ts
```
---
## Adding a New Benchmark
EvalScope uses a **decorator-based registry** pattern. Adding a benchmark requires only two files.
### Step 1: Create the adapter directory
```
evalscope/benchmarks/my_benchmark/
├── __init__.py # (empty)
└── my_benchmark_adapter.py
```
Adapters are **auto-discovered**: any `*_adapter.py` under `evalscope/benchmarks/` is automatically imported at startup, which triggers the `@register_benchmark` decorator.
### Step 2: Write the adapter
Choose a base class depending on your benchmark type:
| Base Class | Use When |
|------------|----------|
| `DefaultDataAdapter` | General text QA, math, coding |
| `MultiChoiceAdapter` | Multiple-choice questions |
| `AgentAdapter` | Function calling, tool use |
| `VisionLanguageAdapter` | Image + text (VQA, etc.) |
| `MultiTurnAdapter` | Multi-turn conversations |
| `Text2ImageAdapter` | Text-to-image generation |
| `NERAdapter` | Named entity recognition |
| `ImageEditAdapter` | Image editing |
**Minimal example** (text QA benchmark):
```python
from typing import Any, Dict
from evalscope.api.benchmark import BenchmarkMeta, DefaultDataAdapter
from evalscope.api.dataset import Sample
from evalscope.api.registry import register_benchmark
from evalscope.constants import Tags
DESCRIPTION = """
## Overview
Brief description of what this benchmark evaluates.
## Task Description
- **Task Type**: ...
- **Input**: ...
- **Output**: ...
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
"""
@register_benchmark(
BenchmarkMeta(
name='my_benchmark', # unique identifier (snake_case)
pretty_name='MyBenchmark', # display name
dataset_id='org/dataset-name', # ModelScope / HuggingFace dataset ID
tags=[Tags.REASONING], # category tags
description=DESCRIPTION,
subset_list=['default'], # dataset subsets
metric_list=['acc'], # evaluation metrics
eval_split='test', # split to evaluate on
few_shot_num=0, # number of few-shot examples
prompt_template='{question}', # prompt template with placeholders
)
)
class MyBenchmarkAdapter(DefaultDataAdapter):
def record_to_sample(self, record: Dict[str, Any]) -> Sample:
"""Convert a dataset row to a Sample object."""
return Sample(
input=record['question'],
target=record['answer'],
)
```
### Key methods you can override
| Method | Purpose | When to Override |
|--------|---------|------------------|
| `record_to_sample()` | Map dataset row → `Sample` | Always |
| `extract_answer()` | Extract structured answer from model output | When default extraction is insufficient |
| `match_score()` | Custom scoring logic | When `acc` / built-in metrics don't fit |
| `sample_to_fewshot()` | Format a sample as few-shot example | When using few-shot with custom format |
| `_on_inference()` | Custom model interaction | For agent/tool-use benchmarks |
### BenchmarkMeta key fields
```python
BenchmarkMeta(
name='...', # Required: unique snake_case ID
dataset_id='...', # Required: remote dataset ID or local path
pretty_name='...', # Display name
tags=[...], # From evalscope.constants.Tags
description='...', # Markdown description for docs
subset_list=['default'], # Dataset subsets
metric_list=['acc'], # Metric names or dicts: [{'acc': {'numeric': True}}]
aggregation='mean', # 'mean', 'pass@k', 'f1', etc.
eval_split='test', # Evaluation split name
train_split='train', # Training split (for few-shot)
few_shot_num=0, # Few-shot count
prompt_template='...', # Prompt template with {placeholders}
filters=OrderedDict(), # Output filters
extra_params={}, # Additional configurable parameters
sandbox_config={}, # Sandboxed execution config (for code benchmarks)
review_timeout=None, # Per-sample timeout in seconds
)
```
### Step 3: Add extra dependencies (if any)
If your benchmark needs additional packages, create a `requirements.txt` in the benchmark directory:
```
evalscope/benchmarks/my_benchmark/requirements.txt
```
Then register it in `pyproject.toml`:
```toml
[tool.setuptools.dynamic.optional-dependencies]
my_benchmark = {file = ["evalscope/benchmarks/my_benchmark/requirements.txt"]}
```
Users can install via `pip install 'evalscope[my_benchmark]'`.
### Step 4: Update documentation (optional)
Run the doc pipeline to auto-generate benchmark documentation:
```bash
make docs-update BENCHMARK=my_benchmark
make docs-translate BENCHMARK=my_benchmark
make docs-generate
```
### Verify your benchmark
```bash
# Check it's registered
evalscope eval --benchmarks my_benchmark --model dummy --limit 5
# Run via service
evalscope service
# Then use the Web dashboard or API: POST /api/v1/eval/invoke
```
---
## Code Quality
### Linting
This project uses **pre-commit** with the following hooks:
- **flake8** — Python style checker
- **isort** — Import sorting
- **yapf** — Code formatting
- Trailing whitespace, YAML checks, line ending fixes
```bash
# Run all checks
make lint
# or
pre-commit run --all-files
```
### Testing
```bash
# Run all tests
pytest tests/
# Run a specific test
pytest tests/benchmark/test_xxx.py
```
---
## Git Workflow
1. **Create a branch** with a descriptive name:
```bash
git checkout -b feature/my-benchmark
```
2. **Make your changes** and commit with clear messages:
```bash
git commit -m "feat: add MyBenchmark adapter"
```
3. **Run quality checks** before pushing:
```bash
pre-commit run --all-files
pytest tests/
```
4. **Push and open a Pull Request** against the `main` branch. Provide a clear description of your changes.
---
Thank you for your contribution!

682
evalscope/DESIGN.md Normal file
View File

@ -0,0 +1,682 @@
---
name: EvalScope Console
colors:
# Brand & Accent — IDENTICAL across both themes. Violet is the brand constant.
accent: "#816DF8"
accent-dark: "#5B3FD6"
accent-dim: "rgba(129,109,248,0.12)"
purple: "#a78bfa"
# Surface ladder (sunken → elevated) — DARK
bg: "#0c0c1a"
bg-deep: "#09091a"
bg-card: "#12122b"
bg-card2: "#16163a"
surface-glass: "rgba(18,18,43,0.7)"
# Text (3-step ladder) — DARK
text: "#e2e8f0"
text-muted: "#8896aa"
text-dim: "#7a8195"
on-filled: "#ffffff"
# Hairline borders — DARK (translucent violet, the near-black bg lets even 10% read)
border: "rgba(129,109,248,0.10)"
border-md: "rgba(129,109,248,0.18)"
border-strong: "rgba(129,109,248,0.28)"
# Semantic states
success: "#10b981"
warning: "#f59e0b"
danger: "#ef4444"
info: "#60a5fa"
# Deep saturated pass/fail (boolean badges)
pass: "rgb(45,104,62)"
fail: "rgb(151,31,44)"
# ────────────────────────────────────────────────────────────────
# Light theme — warm-cream Console (used when [data-theme="light"])
# Distinct palette philosophy: warm-neutral surfaces, solid warm-grey
# hairlines (NOT translucent violet), warm ink text. Same violet accent.
# ────────────────────────────────────────────────────────────────
# Surface ladder — warm-cream, sunken → elevated
bg-light: "#faf9f5" # warm cream canvas — was cool #f5f6fa
bg-deep-light: "#f0ebe1" # input wells, one step below canvas — was cool #e8eaf2
bg-card-light: "#ffffff" # pure white — strongest possible contrast against cream canvas
bg-card2-light: "#f5f0e7" # hover / elevated — warm cream-soft, was cool #eef0f7
surface-glass-light: "rgba(250,249,245,0.80)" # warm-tinted glass — was pure white
# Accent (unchanged from dark — violet is the brand constant)
accent-light: "#6c57e8"
accent-dim-light: "rgba(108,87,232,0.14)" # slightly stronger on white card
# Text — warm-ink ladder
text-light: "#141413" # warm near-black — was cool #1a1f2e
text-muted-light: "#6c6a64" # warm grey — was cool #5a6378
text-dim-light: "#8e8b82" # warm grey — was cool #7c8497
# Hairlines — SOLID warm hex, not translucent violet. Three concrete tones.
# Critical: translucent violet at 0.20 alpha composites to near-invisible
# on white cards. Solid warm-grey gives every card a definite boundary.
border-light: "#e6dfd8" # standard hairline — was rgba(violet,0.20)
border-md-light: "#d6cdbe" # emphasized — was rgba(violet,0.30)
border-strong-light: "#c1b6a3" # hover / focus boundary — was rgba(violet,0.40)
# Compare slot accents (per-model tagging in compare view)
compare-0: "#818cf8"
compare-1: "#34d399"
compare-2: "#fbbf24"
# Chat-bubble role accents
bubble-user: "#818cf8"
bubble-bot: "#34d399"
bubble-tool: "#fbbf24"
bubble-reasoning: "#34d399"
bubble-system: "rgba(148,163,184,1)"
typography:
display-xl:
fontFamily: System Sans
fontSize: 24px
fontWeight: 700
letterSpacing: -0.02em
lineHeight: 1.2
title-md:
fontFamily: System Sans
fontSize: 16px
fontWeight: 700
letterSpacing: normal
lineHeight: 1.25
body-sm:
fontFamily: System Sans
fontSize: 14px
fontWeight: 400
letterSpacing: normal
lineHeight: 1.5
body-sm-strong:
fontFamily: System Sans
fontSize: 14px
fontWeight: 500
letterSpacing: normal
lineHeight: 1.5
body-xs:
fontFamily: System Sans
fontSize: 12px
fontWeight: 400
letterSpacing: normal
lineHeight: 1.5
label-xs:
fontFamily: System Sans
fontSize: 12px
fontWeight: 600
letterSpacing: 0.05em
textTransform: uppercase
lineHeight: 1.4
table-xs:
fontFamily: System Sans
fontSize: 10px
fontWeight: 600
letterSpacing: 0.05em
textTransform: uppercase
lineHeight: 1.4
caption-mono:
fontFamily: System Mono
fontSize: 12px
fontWeight: 400
letterSpacing: normal
lineHeight: 1.4
code:
fontFamily: System Mono
fontSize: 13px
fontWeight: 400
letterSpacing: normal
lineHeight: 1.5
button-sm:
fontFamily: System Sans
fontSize: 12px
fontWeight: 500
letterSpacing: normal
lineHeight: 1.4
button-md:
fontFamily: System Sans
fontSize: 14px
fontWeight: 500
letterSpacing: normal
lineHeight: 1.4
button-lg:
fontFamily: System Sans
fontSize: 16px
fontWeight: 500
letterSpacing: normal
lineHeight: 1.4
fontFamily:
sans: 'system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif'
mono: 'ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", "DejaVu Sans Mono", "Courier New", monospace'
rounded:
none: 0px
xs: 4px
sm: 8px
md: 12px
lg: 16px
xl: 20px
full: 9999px
spacing:
xs: 4px
sm: 8px
md: 12px
lg: 16px
xl: 20px
2xl: 24px
3xl: 32px
4xl: 48px
5xl: 64px
shadows:
# Dark theme — single deep drop (works on near-black surfaces).
sm: "0 2px 8px rgba(0,0,0,0.4)"
md: "0 4px 20px rgba(0,0,0,0.55)"
lg: "0 8px 40px rgba(0,0,0,0.6)"
glow: "0 0 20px rgba(129,109,248,0.25)"
glow-soft: "0 0 12px rgba(129,109,248,0.2)"
# Light theme — two-stop stacks tinted with warm-ink (matches text colour),
# not slate. Slate-tinted drops on cream read as a cool-grey smudge and break
# the warm canvas. Warm-ink stays consistent with the rest of the palette.
sm-light: "0 1px 2px rgba(20,20,19,0.04), 0 4px 12px rgba(20,20,19,0.06)"
md-light: "0 4px 16px rgba(20,20,19,0.07), 0 12px 32px rgba(20,20,19,0.05)"
lg-light: "0 12px 24px rgba(20,20,19,0.09), 0 24px 48px rgba(20,20,19,0.07)"
glow-light: "0 0 20px rgba(108,87,232,0.22)"
glow-soft-light: "0 0 12px rgba(108,87,232,0.18)"
gradients:
brand: "linear-gradient(135deg, #816DF8 0%, #a78bfa 100%)"
accent: "linear-gradient(135deg, #0F9C7E 0%, #06b6d4 100%)"
surface: "linear-gradient(135deg, rgba(129,109,248,0.08) 0%, rgba(167,139,250,0.05) 100%)"
kpi-0: "linear-gradient(135deg, #6366f1, #8b5cf6)"
kpi-1: "linear-gradient(135deg, #10b981, #06b6d4)"
kpi-2: "linear-gradient(135deg, #f59e0b, #f97316)"
kpi-3: "linear-gradient(135deg, #ec4899, #8b5cf6)"
nav-hairline: "linear-gradient(90deg, transparent 0%, #816DF8 50%, transparent 100%)"
transition:
fast: "150ms cubic-bezier(0.4, 0, 0.2, 1)"
base: "180ms ease"
slow: "400ms cubic-bezier(0.4, 0, 0.2, 1)"
breakpoints:
sm: 640px
md: 768px
lg: 1024px
xl: 1280px
container:
max-width: 1600px
page-padding-x: 16px
page-padding-y: 20px
score-formula:
foreground: "hsl(score * 120, var(--score-fg-s), var(--score-fg-l)) # dark: 70%/45%, light: 85%/32%"
background: "RGB-interpolated translucent companion, alpha * var(--score-bg-a-mul) # dark: ×1, light: ×1.6"
description: "0 → red, 0.5 → yellow, 1 → green — used for all dynamic score chips, badges, and SVG rings. Saturation, lightness, and bg alpha multiplier are theme-scoped CSS vars so yellow mid-tones stay legible on warm-cream without re-coding the brand HSL. Light theme bumps saturation +15pt to compensate for the lightness compression at hue ≈ 60 (olive)."
---
# Design System: EvalScope Console
## Overview
EvalScope's web dashboard is a developer-platform brand for **LLM evaluation and benchmarking** — the page is an instrument panel for engineers running evals, written for people who already know the syntax. It earns that posture through **two equally weighted themes** rather than one canonical mode with a translated companion. Both themes share the same vocabulary — same type, same spacing, same radii, same components — but each carries its own surface philosophy. They are two voices of one brand, not one design re-tinted.
**Dark Console** (default) is a stark dark-indigo system: near-black `{colors.bg}` canvas, ice-cool `{colors.text}` body, a 3-step text ladder. Hairlines are translucent violet at 10-28 % alpha because the near-black bg lets even a faint violet tint read as a definite edge. Shadows are single deep drops at `rgba(0,0,0,0.55)` — near-black eats soft shadows. The mood is *late-night terminal session*: low-light, low-distraction, the violet glow on a primary CTA is the only saturated thing on screen.
**Warm Console** (light theme) is a warm-cream system: cream `{colors.bg-light}` canvas (`#faf9f5`), pure white `{colors.bg-card-light}` cards, warm-ink `{colors.text-light}` body (`#141413`). Hairlines are **solid warm-grey** (`{colors.border-light}` `#e6dfd8`, `#d6cdbe`, `#c1b6a3`) — *not* translucent violet, because violet at 20 % alpha disappears against a white card and leaves every card boundary undefined. Shadows are two-stop stacks tinted with the same warm-ink as the text, so cards lift off the cream without printing a cool slate smudge. The mood is *morning code review*: high-contrast, restful on the eyes, the violet CTA is the only cool note in an otherwise warm palette.
The brand constant across both themes is the single violet `{colors.accent}` (`#816DF8` dark / `#6c57e8` light) used for primary CTAs, active nav states, focus rings, and the wordmark accent — plus the dynamic HSL score gradient (`hsl(score × 120, 70%, 45%)`) that maps a 0-1 metric to red → yellow → green. Both signals work over either canvas. Everything else — surface ladder, hairline material, shadow tint, on-canvas text colour — is theme-specific by design, because dark and light surfaces need *different* materials to produce the same hierarchy.
Type is the second decisive voice and is **theme-agnostic**. The brand uses cross-platform system font stacks (no web font is loaded) — `system-ui, -apple-system, "Segoe UI", Roboto, ...` for narrative and `ui-monospace, SFMono-Regular, Menlo, Consolas, "Liberation Mono", ...` for technical labels. Each OS resolves to its own native UI face. Headlines are sentence-case with `tracking-tight` on display numbers; **all-caps + `tracking-wider`** is reserved for tiny section eyebrows (12 px / 10 px), never headlines. Weight ceiling is **700**; the working set is 400 / 500 / 600 / 700.
**Key Characteristics:**
- **Dual-theme parity, not dual-theme translation.** Dark uses translucent violet hairlines on near-black; light uses solid warm-grey hairlines on cream. They produce the same hierarchy through opposite material choices. Theme is persisted to `localStorage` and applied via `data-theme` on `<html>` before first paint to avoid a flash.
- A single violet primary CTA `{colors.accent}` carries every conversion target on both themes, paired with a transparent **ghost** secondary. The brand uses a `{rounded.sm}` 8-px button shape for primary/secondary in the *console* (no marketing pills — this is an in-product surface).
- The primary CTA **glows on hover** with violet at 20-25 % alpha on both themes. That violet glow is the brand's signature interaction — identical animation, identical colour, identical timing across themes.
- Every card section title, form label, and table header sets in `{typography.label-xs}` — 12 px (10 px for tables), `font-semibold`, **UPPERCASE**, `tracking-wider`, muted color. Body and titles stay sentence-case. The contrast between these two voices does most of the hierarchy work.
- A dynamic HSL **score chip** (`hsl(score × 120, 70%, 45%)`) is the second-most-recognizable component after the brand violet — it is how the product communicates pass/fail. Identical formula on both themes; the chip's saturation works over cream and over near-black.
- **Light theme uses solid hex hairlines, not translucent violet.** This is the most important light-theme rule, and the one most often broken on first attempt: a violet alpha overlay disappears into a near-white page, so light surfaces require concrete warm-grey edges (`#e6dfd8` / `#d6cdbe` / `#c1b6a3`) to keep their boundaries.
- A complete domain token set exists for **chat bubbles** (5 semantic roles: user / bot / tool / reasoning / system), **compare slots** (3 per-model accent colors), and **KPI gradients** (4 named gradient pairs) — these are first-class brand tokens, not ad-hoc colors.
- A glassmorphic sticky top-nav (52 px, 12 px backdrop-blur, 1-px violet-to-transparent gradient hairline along the top edge) is the only "marketing-y" flourish the product allows itself. Dark uses translucent indigo glass; light uses translucent cream glass.
## Colors
> **Note on dual theme.** Every color token below has a *dark* (default) and *light* value. Hex pairs are listed as `dark / light`. Components reference tokens by name — never by raw hex — so theme switching is free.
>
> **Light values use SOLID hex for hairlines, not translucent violet.** This is the structural difference from earlier light-theme generations and the single most-broken light-theme rule. See `{colors.border-light}` below and *Elevation & Depth* for the reason.
### Brand & Accent
The accent family is **identical in spirit on both themes** — a single violet handles every conversion target. Slightly different hex values per theme (the dark violet is brighter to read on near-black; the light violet is a half-step deeper to hold weight against white cards) but the same brand voltage.
- **Violet** (`{colors.accent}` — `#816DF8` / `#6c57e8`): The single brand color. Primary CTA fill, active nav fill, focus ring, brand-wordmark accent, table-header active-sort color. Used sparingly — should occupy under ~10 % of any screen.
- **Violet Deep** (`{colors.accent-dark}` — `#5B3FD6` / `#4f3ec8`): Hover state for primary CTAs.
- **Lavender** (`{colors.purple}` — `#a78bfa` / `#7c6be0`): The gradient companion stop; the second half of `{gradients.brand}`.
- **Violet Mist** (`{colors.accent-dim}` — `rgba(129,109,248,0.12)` / `rgba(108,87,232,0.14)`): The low-alpha violet used as pill background and focus-ring fill. Light theme runs slightly stronger (0.14 vs 0.12) because white cards need a touch more saturation to read the mist.
- **Violet Glow** (`{shadows.glow}` — `0 0 20px rgba(129,109,248,0.25)` / `0 0 20px rgba(108,87,232,0.22)`): The signature hover halo on primary buttons and active nav. Same effect, same magnitude, on both themes.
### Surface
Each theme operates with a 4-step surface ladder, sunken-to-elevated. The **ladder structure is shared**; the **material is different** — dark walks an indigo ladder, light walks a warm-cream ladder. The semantic of each step is the same: `bg-deep` is *below* the page, `bg-card` is the working surface, `bg-card2` is the elevated state.
- **Page** (`{colors.bg}` — `#0c0c1a` / `#faf9f5`): The page body. Dark = near-black instrument-panel mood. Light = warm cream canvas (Claude-style), deliberately not cool grey-white — the cool variant (`#f5f6fa`) reads as "any other SaaS dashboard" and washes out white cards.
- **Sunken** (`{colors.bg-deep}` — `#09091a` / `#f0ebe1`): One step *below* the page — used for input wells, the deep-well that holds pill-style tab containers, and the icon tile in empty-states. Dark goes deeper-black; light goes warmer-cream. On both themes, inputs read as "wells" because they're one step below the surface that contains them.
- **Card** (`{colors.bg-card}` — `#12122b` / `#ffffff`): The default card / dialog / table surface. Dark = indigo card on near-black page. Light = pure white card on cream canvas — the cream-to-white contrast (ΔL ≈ 7) is what gives a light-theme card its lift, NOT a heavy shadow.
- **Card Elevated** (`{colors.bg-card2}` — `#16163a` / `#f5f0e7`): Hover state for clickable cards and rows; also the inactive-tab fill in pill-tab containers. Light theme's elevated state is warm-cream-soft — the elevated state is darker on dark theme but lighter-than-card-but-warmer on light theme (the white card with a soft-cream hover reads as "depressed into the cream canvas").
- **Glass** (`{colors.surface-glass}` — `rgba(18,18,43,0.7)` / `rgba(250,249,245,0.80)`): Translucent surface for the sticky top-nav, used with a 12-px backdrop-blur. Light theme uses tinted cream glass (matches the canvas), NOT pure white — white glass on cream reads as a foreign sheet floating in space.
### Text
- **Ink** (`{colors.text}` — `#e2e8f0` / `#141413`): All headings, body, table cell values, button labels on non-filled surfaces. Light theme uses warm-near-black (`#141413`, ≈ the same value Claude.com uses) rather than a cool slate (`#1a1f2e`), so the text temperature matches the canvas temperature.
- **Muted** (`{colors.text-muted}` — `#8896aa` / `#6c6a64`): Secondary labels, nav-link inactive text, card-header micro-labels, button "ghost" idle text. *This is also the color section-eyebrow uppercase labels are set in.* Light theme uses warm-grey (`#6c6a64`) rather than cool-slate (`#5a6378`) to stay coherent with the warm canvas.
- **Dim** (`{colors.text-dim}` — `#7a8195` / `#8e8b82`): Lowest-priority text — placeholder text, timestamps in compact rows, table empty-state. **Contrast tuned to ~3.6 : 1** against `{colors.bg-card}` on both themes — sits just above the WCAG AA Large floor (3 : 1), still **below AA Normal (4.5 : 1)**. ⚠️ Reserve for ≥ 14 px non-essential metadata. Light theme uses a warm-grey at the same luminance step as the dark theme's cool-grey — the perceived hierarchy stays identical.
- **On Filled** (`{colors.on-filled}` — `#ffffff` / `#ffffff`): Text on `{colors.accent}` and other saturated fills. Identical on both themes — the violet CTA is dark enough on both that white text holds.
### Hairlines (the structural difference between themes)
- **Border** (`{colors.border}` — `rgba(129,109,248,0.10)` / `#e6dfd8`): The default 1-px card / input / divider boundary. **Dark uses translucent violet at 10 % alpha** because the near-black bg-to-card luminance step already does most of the boundary work — the violet hairline just tints it. **Light uses a SOLID warm-grey hex** (`#e6dfd8`, Claude-style cream-hairline) because the white-card-on-cream luminance step is gentle enough that a translucent violet overlay disappears into the page. Borders on light theme are concrete materials, not tints.
- **Border Emphasized** (`{colors.border-md}` — `rgba(129,109,248,0.18)` / `#d6cdbe`): One step stronger — used on form inputs after focus, on the active-state of hover cards, on the boundary between a card and a nested section.
- **Border Strong** (`{colors.border-strong}` — `rgba(129,109,248,0.28)` / `#c1b6a3`): The strongest boundary — used by `{components.card-hover}` on hover lift, and by elevated cards in modal contexts. On both themes this is the "this thing is grabbing attention" hairline.
### Semantic
- **Success** (`{colors.success}` — `#10b981` / `#059669`) / **Success Bg** (`rgba(16,185,129,0.08)`) / **Success Border** (`rgba(16,185,129,0.20)`): Confirmed / passed states; success toasts; chat-bot bubble border.
- **Warning** (`{colors.warning}` — `#f59e0b` / `#d97706`) / **Warning Bg** (`rgba(245,158,11,0.08)`) / **Warning Border** (`rgba(245,158,11,0.20)`): Pending / caution; tool-call chat bubbles.
- **Danger** (`{colors.danger}` — `#ef4444` / `#dc2626`) / **Danger Bg** (`rgba(239,68,68,0.08)`) / **Danger Border** (`rgba(239,68,68,0.20)`): Errors, fail badges, validation, destructive actions.
- **Info** (`{colors.info}` — `#60a5fa` / `#3b82f6`): Latency chart series, informational toasts.
- **Pass** (`{colors.pass}` — `rgb(45,104,62)` / `rgb(16,108,55)`) / **Fail** (`{colors.fail}` — `rgb(151,31,44)` / `rgb(180,30,42)`): Deep saturated greens / crimsons for boolean pass/fail badges where the tone needs more weight than the soft semantic family.
### Score Gradient (Signature)
The product's emotional core. A 0-1 score maps to **`hsl(score × 120, 70%, 45%)`**: 0 → red, 0.5 → yellow, 1 → green. Used as both foreground and translucent background on score chips, dataset chips, and group-header best-score callouts. This is computed inline (`scoreColor` / `scoreBg` helpers), never stored as a static palette. **Treat the formula as a brand asset** — do not reskin to a 5-step bucket, do not introduce a 4th hue.
Foreground uses HSL for predictable hue progression; background uses an RGB-interpolated stop pair (rich mid-tones, more saturated yellows) for visual continuity with the existing chip/badge appearance. Do not rewrite the bg to HSL alpha — the mid-tone shift will break existing visual reads.
**Per-theme legibility knobs**: `hsl(h 70%, 45%)` reads cleanly on the near-black dark canvas but the yellow mid (hue ≈ 60) collapses into warm-cream `#faf9f5` on the light theme. To keep the brand formula intact while fixing legibility, three CSS vars scope per-theme:
- **`--score-fg-s`** — saturation for the HSL foreground. Dark: `70%`. Light: `85%` (the +15pt bump compensates for the lightness compression at hue ≈ 60; without it, mid scores render as washed-out olive on cream).
- **`--score-fg-l`** — lightness for the HSL foreground. Dark: `45%`. Light: `32%` (darker olive/forest/maroon read cleanly on cream).
- **`--score-bg-a-mul`** — alpha multiplier for the RGB-interpolated bg. Dark: `1`. Light: `1.6` (boosts washed-out yellows so the pill bg is actually visible).
`scoreColor` / `scoreBg` emit CSS expressions (`hsl(h var(--score-fg-s) var(--score-fg-l))`, `rgb(r g b / calc(α * var(--score-bg-a-mul)))`) — the browser picks the right value per active theme. **Do not** hardcode these in JS or fork separate light/dark helpers.
**Score Ring** (`{components.score-ring}` — SVG circular progress used in `<ReportSummaryStats>` and the "Overall Score" callout in `<DetailsTab>`): the active arc is `stroke={scoreColor(score)}`. Stroke width must be **≥6 px** for the 48 × 48 mini ring and **8 px** for the 72 × 80 summary ring — anything thinner reduces the colored area to the point where the mid-hue olive stops carrying. Background arc uses `var(--border)` for a neutral track.
### Compare Slots
Three per-model accent colors used to tag side-by-side model comparisons. Each slot has a `dot`, `border`, `bg`, and `bg-header` tint at ~10-30 % alpha:
- **Slot 0** (`{compare.0.dot}` — `#818cf8` / `#6c57e8`): Indigo.
- **Slot 1** (`{compare.1.dot}` — `#34d399` / `#0a8a6e`): Mint.
- **Slot 2** (`{compare.2.dot}` — `#fbbf24` / `#d97706`): Amber.
If a comparison view exceeds 3 models, *do not invent a 4th brand color* — collapse into a numbered legend instead.
### Chat Bubble Roles
Five semantic roles, each with a complete 7-token set (`bg`, `bg-hl`, `border`, `border-hl`, `icon-bg`, `icon-border`, `color`):
- **User** — Indigo family (`#818cf8` color).
- **Bot** — Emerald family (`#34d399` color).
- **Tool** — Amber family (`#fbbf24` color).
- **Reasoning** — Dim emerald (lower-saturation companion to Bot).
- **System** — Slate gray (`rgba(148,163,184,*)`).
Bubble containers are `{rounded.md}` with the role's tint background and border; hover/highlight states use the `*-hl` variants.
### KPI Gradients
Four named linear gradients for the four hero KPI tiles on the dashboard:
- **Indigo→Violet** (`{gradients.kpi-0}` — `linear-gradient(135deg, #6366f1, #8b5cf6)`)
- **Emerald→Teal** (`{gradients.kpi-1}` — `linear-gradient(135deg, #10b981, #06b6d4)`)
- **Amber→Orange** (`{gradients.kpi-2}` — `linear-gradient(135deg, #f59e0b, #f97316)`)
- **Pink→Violet** (`{gradients.kpi-3}` — `linear-gradient(135deg, #ec4899, #8b5cf6)`)
Always applied to the 40 × 40 `{rounded.md}` icon tile inside a `{components.kpi-card}`. **Same gradient values on both themes** — they're saturated enough to work over either canvas.
### Chart Palette (Perf Metrics)
Four hue tokens used to mark perf-metric series (latency / TTFT / TPOT / token-usage) across the KPI strip, the chart series legends, and the percentile-table accent headers in `<PerfMetricsPanel>`. **The two themes use different RGB values for the same hue** — unlike the KPI gradients, these aren't shared across themes:
- **Latency** (`{chart.latency}` — `#60a5fa` dark / `#2563eb` light)
- **TTFT** (`{chart.ttft}` — `#34d399` dark / `#047857` light)
- **TPOT** (`{chart.tpot}` — `#a78bfa` dark / `#5b48e0` light)
- **Token** (`{chart.token}` — `#94a3b8` dark / `#5a6378` light)
**Why the fork**: warm-cream bg (`{colors.bg-card-light}` / `{colors.bg-deep-light}`) compresses cool hues via simultaneous contrast. Mid-tone slate blue / violet that pop on near-black dark canvases read as washed-out lavender on cream. The light palette pulls each hue darker **and** more saturated to restore the data-bearing punch of the same hue family. Same hue identity, different RGB — not a re-tint.
**KPI strip surface**: the strip in `<PerfMetricsPanel>` uses `{colors.bg-card}` (matching the outer card), not `{colors.bg-deep}`. On light theme, `{colors.bg-deep}` is even warmer than the cards and pushes the chart hues into the warm-on-warm range — losing the contrast that makes the colored numbers carry. Visual separation comes from the border + dividers, not bg differentiation.
### Brand Gradients (Decorative)
- **Brand** (`{gradients.brand}` — `linear-gradient(135deg, #816DF8 → #a78bfa)`): For `gradient-text` and large brand moments.
- **Accent** (`{gradients.accent}` — `linear-gradient(135deg, #0F9C7E → #06b6d4)`): For the optional emerald-to-cyan accent text.
- **Surface** (`{gradients.surface}` — `linear-gradient(135deg, rgba(129,109,248,0.08) → rgba(167,139,250,0.05))`): The subtle violet wash layered behind KPI cards via `::before`.
## Typography
### Font Family
Two cross-platform **system font stacks** carry the entire system — each OS resolves to its own native UI face. No `@font-face` is loaded; this is deliberate. Each stack starts with the modern CSS `system-ui` / `ui-monospace` generic family and falls back to named faces for older browsers and per-OS targets:
1. **System sans** (`{typography.font-sans}` — `system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif`): Every display, body, button, link, and label. Working weights: 400 / 500 / 600 / 700. Resolves to **SF Pro** on macOS / iOS, **Segoe UI** on Windows, **Roboto** on Android / ChromeOS, and the desktop default (Cantarell / Noto / DejaVu) on Linux.
2. **System mono** (`{typography.font-mono}` — `ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", "DejaVu Sans Mono", "Courier New", monospace`): Timestamps, scores, model IDs, and any tabular-numeric data. Weight 400 only. Resolves to **SF Mono** on macOS, **Consolas** on Windows, **Liberation Mono / DejaVu Sans Mono** on Linux. The full chain matters — without the OS-specific named fallbacks, non-mac users land on `Courier New`, breaking the tabular-numeric look.
Antialiasing is forced (`-webkit-font-smoothing: antialiased`). No web font is loaded — the brand reads as "native developer tool, not marketing site" precisely because of this. The trade-off is per-OS rendering variance; if pixel-identical screenshots across platforms are required, see *Note on Font Substitutes* below.
### Hierarchy
| Token | Size | Weight | Tracking | Use |
|---|---|---|---|---|
| `{typography.display-xl}` | 24px | 700 | tight | KPI value, hero number (dashboard `{components.kpi-card}`). |
| `{typography.title-md}` | 16px | 700 | normal | Card-title model name, group-header titles, brand wordmark. |
| `{typography.body-sm}` | 14px | 400 / 500 | normal | Default body text, button-md label, table-cell text, paragraph copy. |
| `{typography.label-xs}` | 12px | 600 | wider | **UPPERCASE** card-section header, form label, badge text — the brand's signature eyebrow. |
| `{typography.body-xs}` | 12px | 400 / 500 | normal | Empty-state hint, pill / badge body, mobile-nav link. |
| `{typography.table-xs}` | 10px | 600 | wider | **UPPERCASE** table-header micro-text — whispers, doesn't shout. |
| `{typography.caption-mono}` | 12px | 400 (mono) | normal | Timestamps, score values, dataset names in chips. |
| `{typography.code}` | 13–14px | 400 (mono) | normal | Log viewer, JSON viewer, terminal-style output. |
| `{typography.button-sm}` | 12px | 500 | normal | `{components.button}` `size="sm"`. |
| `{typography.button-md}` | 14px | 500 | normal | `{components.button}` `size="md"` (default). |
| `{typography.button-lg}` | 16px | 500 | normal | `{components.button}` `size="lg"` — for hero callouts only. |
### Principles
- **UPPERCASE + `tracking-wider` is the eyebrow voice — never the headline voice.** It marks "this introduces a region." Card titles, model names, and KPI labels stay sentence-case (or lower-case for the brand wordmark's lowercase "v").
- **`tracking-tight` is reserved for the display tier** — KPI numbers and the brand wordmark only. It tells the reader "this is set big on purpose."
- **`tabular-nums` + `font-mono` everywhere numbers must align** — KPI values, score chips, timestamps, percentage cells. Treat numbers as data, not prose.
- **Weight 700 is the display ceiling.** No `font-extrabold` / `font-black`. The brand reads as a calmer system because of this.
- **Line-heights inherit Tailwind defaults** (1.5 for body, 1.25 for headings). Don't override globally; rely on padding for vertical rhythm in tight stacks (cards, chips).
- **No web font.** The system stack is the system. Loading Inter or Geist on top would break the "native console" feel.
### Note on Font Substitutes
EvalScope uses the OS-native stack, so there are no proprietary faces to substitute. If a future skin needs to enforce a *single* face across all OSes for screenshot consistency:
- **Sans substitute** — *Inter* (400 / 500 / 600 / 700) is the closest stylistic match to the SF-on-macOS rendering; preserves the geometric / neutral character.
- **Mono substitute** — *JetBrains Mono* (400) at 12–14 px matches the technical voice well; *IBM Plex Mono* is a close second.
## Layout
### Spacing System
- **Base unit**: 4 px (Tailwind's default scale).
- **Tokens** (Tailwind-aligned):
- `{spacing.xs}` 4 px · `{spacing.sm}` 8 px · `{spacing.md}` 12 px · `{spacing.lg}` 16 px · `{spacing.xl}` 20 px · `{spacing.2xl}` 24 px · `{spacing.3xl}` 32 px · `{spacing.4xl}` 48 px · `{spacing.5xl}` 64 px.
- **Page padding**: 16 px horizontal, 20 px vertical (`px-4 py-5`) at all breakpoints — *no responsive expansion*. The 1600 px container does the breathing for us.
- **Card interior padding**: 20 px body (`{spacing.xl}`), 12 px header strips (`{spacing.md} {spacing.xl}`).
- **Inline gap**: `{spacing.md}` (12 px) for component rows inside a card, `{spacing.xl}` (20 px) for inter-section gaps between major page blocks.
- **Pill / chip gap**: `{spacing.sm}` (6-8 px) — tight, scan-friendly. Pills are meant to wrap.
### Grid & Container
- **Max width**: `1600px` (`max-w-[1600px]`). Centered (`mx-auto`). Wide enough for 4-up KPI strip + side-by-side comparison; narrow enough that line-length never sprawls.
- **Column patterns**:
- **KPI strip**: 4-up at `lg+`, 2-up below (`grid-cols-2 lg:grid-cols-4`).
- **Compare view**: 2-3 columns at desktop, vertical stack on mobile.
- **Eval timeline**: single column of full-width rows, no grid.
- **Form pairs**: `grid-cols-1 md:grid-cols-2` for label-value pairs.
- **Gutters**: 16 px horizontal at all sizes.
### Whitespace Philosophy
Whitespace separates the *bands* — not the components inside a band. Section spacing is generous (`flex flex-col gap-5` → 20 px between major blocks); card interiors are tight (`gap-2` / `gap-3` between rows inside a card). The page reads as engineered — *large gaps + tight interior, never the other way around*. The dark page lets cards float visually without needing margin to assert themselves; the hairline border does the bordering work.
The brand's voice is **information-rich but uncluttered** — a typical dashboard packs a 4-tile KPI strip, a 30-row sortable eval list, search + sort + view-toggle controls, and the page still feels scannable because:
1. Tokens enforce a 3-step text hierarchy on every row.
2. UPPERCASE eyebrows visually section the layout without horizontal rules.
3. Score chips compress a percentage + benchmark name + color signal into 6 characters of mono.
### Responsive Strategy
#### Breakpoints (Tailwind defaults)
| Name | Width | Key Changes |
|---|---|---|
| Mobile | < 640px | Nav collapses to hamburger drawer; KPI grid drops to 2-up; all controls stack. |
| Small | 640–767px | GitHub icon and locale toggle appear in nav; KPI still 2-up. |
| Tablet | 768–1023px | Nav switches to **icon-only mode** with tooltips; main content full-width. |
| Desktop | 1024–1279px | Full pill-style nav with icon + label; KPI strip goes 4-up. |
| Wide | ≥ 1280px | Container holds at `max-w-[1600px]`; bands stretch but content centers. |
#### Touch Targets
The top-nav icon-only buttons (tablet) are 32 × 32 — *under the 44 × 44 WCAG floor*. This is a known compromise for the developer-tool density; on actual touch devices, hit areas are extended via padding. Primary buttons reach ~36 px tall in `md` size and ~44 px in `lg` — meet the floor at `lg`.
#### Collapsing Strategy
- **Nav**: Desktop = pill-row with icon + label; tablet = icon-only pills; mobile = logo + hamburger toggling a stacked drop-down (`max-height` animated, 300 ms).
- **KPI strip**: 4-up → 2-up below `lg`. Each tile keeps its `{rounded.md}` 12-px shape and 20-px padding.
- **Eval timeline**: Single column at all sizes — already a vertical list.
- **Forms**: Two-column label/value at `md+`, single column below.
- **Table**: Horizontal scroll wrapper preserves all columns rather than dropping them. The card-shell border keeps the scroll area framed.
#### Image / Icon Behavior
- **Iconography**: `lucide-react`, almost always 14–18 px, color inherits `currentColor`. The only non-Lucide mark is the brand SVG in the top-nav (a hand-drawn triangle with an amber check, rendered with `currentColor` so it follows theme).
- **No marketing imagery**: This is a console — no hero photo, no customer-logo strip, no illustrated empty-states. Empty states use a single Lucide icon in a 64 × 64 `{rounded.lg}` deep-well tile.
- **Charts**: Plotly-based, theme-aware — chart series colors are `{colors.chart-*}` tokens (latency / TTFT / TPOT / token).
## Elevation & Depth
| Level | Treatment | Use |
|---|---|---|
| **L0 — Flat** | No border, no shadow. | Page body, large empty regions, full-bleed dark sections. |
| **L1 — Hairline** | 1-px `{colors.border}` only. Dark: **translucent violet at 10 % alpha** (the near-black page lets even a faint hairline read; the slight violet tint stays on-brand). Light: **solid warm-grey hex** `#e6dfd8` (translucent violet at any plausible alpha composites to invisible against a white card on cream — solid hex is the only thing that reads). | Default content cards (`{components.card}`), table chrome, form inputs. |
| **L2 — Lifted** | `{shadows.sm}` + L1 hairline. Dark: single deep drop `0 2px 8px rgba(0,0,0,0.4)`. Light: two-stop stack `0 1px 2px rgba(20,20,19,0.04), 0 4px 12px rgba(20,20,19,0.06)` — warm-ink tinted (matches `{colors.text-light}`), not slate. Slate-tinted drops on cream read as a cool-grey smudge against the warm canvas. | KPI cards, path-bar at the top of the dashboard, anything that needs to read as "above the page." |
| **L3 — Floating** | `{shadows.md}` + L1 hairline. Dark `0 4px 20px rgba(0,0,0,0.55)`. Light `0 4px 16px rgba(20,20,19,0.07), 0 12px 32px rgba(20,20,19,0.05)`. | Default for the primary card variants when the page already has heavy chrome. |
| **L4 — Elevated** | `{shadows.lg}` + `{colors.border-strong}`. Dark `0 8px 40px rgba(0,0,0,0.6)`. Light `0 12px 24px rgba(20,20,19,0.09), 0 24px 48px rgba(20,20,19,0.07)`. | Hover state for clickable cards, modal / dialog surfaces, dropdown menus. |
| **L5 — Glow** | `{shadows.glow}` on top of L1-L3. Dark `0 0 20px rgba(129,109,248,0.25)`. Light `0 0 20px rgba(108,87,232,0.22)`. | The signature violet halo — primary button hover, active nav pill, sometimes active tab. Identical magnitude across themes — the glow is the brand's interaction constant. |
**Brand rule (depth)**: dark and light themes are *not* the same shadow scaled down. Dark surfaces use a **single deep drop** (`rgba(0,0,0,0.4-0.6)`) — the near-black canvas eats soft drops. Light surfaces use a **two-stop stack** tinted with warm-ink `rgba(20,20,19,*)` (the same hex as `{colors.text-light}`), NOT slate `rgba(15,23,42,*)` — the warm-ink tint stays coherent with the cream canvas and the warm-ink body text. Light cards lift through the *combination* of cream-to-white surface contrast + a solid warm-grey hairline + the two-stop warm-ink shadow stack — no single layer carries the weight.
**Brand rule (hairlines)**: light theme borders are **SOLID HEX**, not translucent violet at any alpha. The earlier light-theme generation used `rgba(108,87,232,0.20-0.40)` and the violet alpha composited to within a few luminance steps of the white card — borders effectively dissolved, especially on outline buttons (`Go to Index`, `Find msg id`) and on input rings (`Score Threshold` field). The current system uses three concrete warm-grey hex values (`#e6dfd8` / `#d6cdbe` / `#c1b6a3`) — each step is a definite material, not a tint. The dark theme keeps translucent violet at 10-28 % because the indigo-bg-to-card luminance delta is already doing most of the boundary work; light theme has no such luminance assist and must rely on the hairline alone.
### Decorative Depth
- **Backdrop-blur**: 12 px on `{colors.surface-glass}` for the sticky top-nav. This is the only blur effect in the system.
- **Hairline gradient line**: The top of the nav draws a 1-px `transparent → {colors.accent} → transparent` line at 40 % opacity. The closest the design comes to "decoration."
- **Card lift**: Hover lifts L1/L2 cards `-2px` (`{components.card-hover}`) or `-3px` (`{components.kpi-card}`), simultaneously upgrading their shadow ladder one step.
- **Gradient text**: `.gradient-text` and `.gradient-text-accent` utilities apply `{gradients.brand}` / `{gradients.accent}` to text via `background-clip: text`. Use sparingly — reserved for hero brand moments, never for body or table cells.
## Shapes
### Border Radius Scale
| Token | Value | Use |
|---|---|---|
| `{rounded.none}` | 0px | Full-bleed sections, table cell interiors. |
| `{rounded.xs}` | 4px | `{tokens.radius-xs}` — tightest inline pill (rarely used directly). |
| `{rounded.sm}` | 8px | `{tokens.radius-sm}` — buttons (sm/md), inputs, tabs, view-toggle buttons. Most in-product chrome lives here. |
| `{rounded.md}` | 12px | `{tokens.radius}` — **default card radius**. Cards, KPI tiles, group-header containers, large buttons. |
| `{rounded.lg}` | 16px | `{tokens.radius-lg}` — empty-state icon tile, KPI icon tile. |
| `{rounded.xl}` | 20px | `{tokens.radius-xl}` — reserved (used by occasional rounded-2xl Tailwind class on welcome / empty states). |
| `{rounded.full}` | 9999px | Badges, chips, score pills, mobile-nav icon buttons. |
**No pill (100-px) shape.** Unlike Vercel's marketing pill, EvalScope is an in-product surface; all CTAs use `{rounded.sm}` 8 px. Pills (`{rounded.full}`) are exclusively for *data* — badges, chips, score indicators.
### "Photography" — Iconography Geometry
- **Brand mark**: Hand-drawn SVG triangle with an amber check; rendered inline at 28 × 25 px with `currentColor` so it follows theme.
- **Lucide icons**: 14 px in dense lists, 16 px in nav, 18 px in form controls, 28 px in empty-state hero tiles. Stroke width 1.5–2.
- **KPI icon tile**: 40 × 40, `{rounded.md}`, filled with one of `{gradients.kpi-0..3}`, white icon centered.
- **Chart**: Plotly canvas, no rounded corners; sits inside a `{rounded.md}` card frame.
## Components
### Buttons
The brand operates with three button variants — *all in-product scale*, no marketing pill:
**`{components.button.primary}`** — the canonical violet CTA.
- Background `{colors.accent}`, text `{colors.on-filled}`, shape `{rounded.sm}` 8 px (md/sm) or `{rounded.md}` 12 px (lg). Hover adds `{shadows.glow}` violet halo + `bg → {colors.accent-dark}`. Press scales to 0.98. **The glow IS the brand interaction.**
**`{components.button.ghost}`** — the transparent secondary.
- Transparent background → `{colors.bg-card}` on hover → `{colors.bg-card2}` on active. Text `{colors.text}`. Used inside top-nav, toolbars, and inline icon-buttons.
**`{components.button.outline}`** — the hairline tertiary.
- Transparent background with 1-px `{colors.border-md}` border. Hover swaps both border and text to `{colors.accent}` (no fill). For "alternative action" CTAs.
**Sizes**:
- `sm` — 12-px text (`{typography.button-sm}`), 6/12 padding, `{rounded.sm}`, ~28 px tall.
- `md` — 14-px text (`{typography.button-md}`), 8/16 padding, `{rounded.sm}`, ~36 px tall *(default)*.
- `lg` — 16-px text (`{typography.button-lg}`), 10/24 padding, `{rounded.md}`, ~44 px tall.
Disabled = `opacity: 0.5` + `cursor: not-allowed`. Transitions use `{tokens.transition}` (0.18 s ease) consistently — never custom per button.
### Cards & Containers
**`{components.card}`** — the canonical card.
- Background `{colors.bg-card}`, text `{colors.text}`, padding 20 px (`{spacing.xl}`), shape `{rounded.md}` 12 px, border 1-px `{colors.border}`, shadow `{shadows.sm}` (L2). Optional **uppercase section header** in a 12 px / 20 px header strip with a bottom border. Collapsible variant rotates a chevron 90 °.
**`{components.card-hover}`** — utility applied to clickable cards.
- Adds `-2 px translateY` on hover, upgrades shadow to `{shadows.lg}` (L4), strengthens border to `{colors.border-strong}`. All transitions in `{tokens.transition}`.
**`{components.kpi-card}`** — the dashboard hero metric tile.
- Same chrome as `{components.card}` but layered with `{gradients.surface}` (subtle violet wash, 5-8 % alpha) via `::before`. Hosts a 40 × 40 `{rounded.md}` icon tile filled with `{gradients.kpi-0..3}`. Hover lifts `-3 px` and ramps to L4 shadow. Stagger-animated on first paint (60 ms steps).
**`{components.card-glass}`** — the glassmorphic surface used by the top-nav.
- `{colors.surface-glass}` background + 12-px backdrop-blur. Always combined with a 1-px hairline border. Reserve for sticky-positioned surfaces — diffuse blur is performance-sensitive.
**`{components.row-card}`** — borderless button styled as a list row (the eval-run rows on the dashboard).
- Background `{colors.bg-card}`, padding 16-20 px, `{rounded.md}`, 1-px `{colors.border}` border. Hover only changes border tint to `{colors.border-md}` — *no transform* — kept calm for dense lists.
### Inputs & Forms
**`{components.input}`** — the canonical text input.
- Background `{colors.bg-deep}` (one step *below* its container — inputs read as "wells"), 1-px `{colors.border}` border, text `{colors.text}` in `{typography.body-sm}`. Focus: border → `{colors.accent}` + 1-px ring in `{colors.accent-dim}`. **Soft halo, never harsh outline.** Padding 8/12, `{rounded.sm}` 8 px.
- Error state: border + ring swap to the danger family; 12 px `{colors.danger}` helper line below.
**`{components.label}`** — input label.
- 12 px, `font-medium`, **UPPERCASE**, `tracking-wider`, `{colors.text-muted}`, placed above the input with 6-px gap.
**`{components.select}`** / **`{components.search-input}`** — same chrome as `{components.input}`. The select uses the native `<select>` styled with the same class.
### Tabs
**`{components.tabs}`** — pill-container.
- An outer container with `{colors.bg-deep}` background and 1-px border, holding inline buttons. Active tab fills with `{colors.accent}` + white text + soft glow `0 0 12 px rgba(129,109,248,0.2)`; inactive uses `{colors.bg-card}` → `{colors.bg-card2}` on hover. Reads as a segmented control, not a tab strip.
### Navigation
**`{components.top-nav}`** — sticky top navigation.
- 52 px tall, `position: sticky`, `z-50`, `{components.card-glass}` chrome with 12-px blur, 1-px violet-to-transparent gradient line on the top edge. Container caps at 1600 px. Three modes:
- **Desktop (lg+)** — pill links with icon + label; active = solid `{colors.accent}` + glow.
- **Tablet (md–lg)** — 32 × 32 icon-only buttons with `title` tooltips; same active state.
- **Mobile (< md)** — logo + hamburger toggling a stacked drop-down with animated `max-height` (300 ms ease-in-out).
**`{components.nav-link}`** — the pill-style nav button.
- Padding 6/12, `{rounded.sm}`, text `{colors.text-muted}` → `{colors.text}` on hover, active = `{colors.accent}` background + white text + violet glow shadow.
**`{components.icon-button}`** — circular ghost icon container in the nav.
- 32 × 32, `{rounded.sm}`, transparent background → `{colors.bg-card2}` on hover, icon inherits `{colors.text-muted}` → `{colors.text}`.
### Tables
**`{components.table}`** — sortable data table.
- Wrapped in `{components.card}` chrome (`{rounded.md}` border + bg) with `overflow-x-auto`.
- **Header cells**: `{typography.table-xs}` — 10 px, semibold, UPPERCASE, `tracking-wider`, `{colors.text-dim}`. They *whisper*. Sortable headers show a triple-state chevron (`ChevronsUpDown` idle, `ChevronUp/Down` active) and lift to `{colors.text}` on hover; active sort column turns `{colors.accent}`.
- **Row dividers**: 1-px `{colors.border}`. Clickable rows hover-fill `{colors.bg-card2}`.
- **Empty state**: Centered, dimmed "No data" cell — no illustration.
### Badges, Chips & Pills
**`{components.badge}`** — fully rounded inline pill.
- `{rounded.full}`, 8/2 padding, `{typography.body-xs}`. Four variants pair an 8-10 % alpha background with a saturated foreground: `default → accent`, `success → green`, `warning → yellow`, `danger → red`.
**`{components.filter-chip}`** — dismissable filter chip.
- Same pill shape as `{components.badge}`, with an optional 3.5 × 3.5 circular dismiss button (`X` icon).
**`{components.score-chip}`** — the signature dynamic-score pill.
- `{rounded.full}`, 8/2 padding, `{typography.caption-mono}`, **outline treatment**: transparent bg + 1-px `scoreColor` border + `scoreColor` text. Format: `"benchmark-name 87.3"` — the chip *is* the data point. Used in the eval timeline, dashboard tiles, and the leaderboard rows.
- Outline (not filled) on purpose: a filled pill at hue ≈ 60 paints high-luminance yellow (`rgb(255, 255, 0)`) which dominates whatever surface it sits on, regardless of theme. Outline keeps hue legible while letting the chip recede into the row.
### Signature Components
**`{components.kpi-card}`** — see *Cards & Containers* above.
**`{components.eval-run-card}`** — a full-width borderless button-styled card row.
- Two rows of content: bold model name + colored score pill on the right; second row has a mono timestamp + a wrap of `{components.score-chip}` entries (one per benchmark). Hover only changes border tint (no lift).
**`{components.chat-bubble}`** — five-role chat surface.
- `{rounded.md}` container with role-specific bg / border / icon-bg / icon-border / color from the `{colors.bubble-*}` token family (user / bot / tool / reasoning / system). Hover strengthens border via `*-hl` variants.
**Two variants**:
- `bar` (default in chat lists) — `rounded-sm` container, 3-px left vertical accent in the role's color, role's tint as bg. Used by the streaming chat log where many bubbles stack and visual weight must stay low.
- `card` — `rounded-md` container with role's full bg + 1-px border. Used in standalone bubbles outside a scrolling list (eval result preview, single-message dialogs).
**`{components.empty-state}`** — first-contact / no-data state.
- Vertical-center stack: 64 × 64 `{rounded.lg}` deep-well tile holding a 28-px Lucide icon (`{colors.accent}` for welcome states, `{colors.text-dim}` for empty/no-results), followed by a 2-line message — title in `{typography.body-sm}` `{colors.text}` (welcome) or `{colors.text-muted}` (empty), hint in `{typography.body-xs}` `{colors.text-dim}`.
**`{components.path-bar}`** — the dashboard's "scan this directory" input row.
- `{components.card}` chrome at L2, hosting an icon, an input, and a primary button — flex-row with 12-px gap.
**`{components.score-badge}`** — the bold percentage pill at the top of an eval row.
- `{rounded.full}`, 10/2 padding, `{typography.body-sm}` bold + `tabular-nums`, HSL-computed fg/bg. Distinct from `{components.score-chip}` by size and weight.
### Examples (illustrative)
> These `ex-*` surfaces mirror the brand-native primitives for downstream consumers (kits, mockups, Stitch generation). Each references existing components so a re-skin re-skins all surfaces consistently.
**`ex-metric-tile`** — Dashboard KPI tile. Re-uses `{components.kpi-card}` with gradient icon tile + tabular value + uppercase label.
- Properties: `backgroundColor`, `iconGradient`, `rounded`, `padding`, `valueTypography`, `labelTypography`.
**`ex-eval-run-row`** — A row in the eval timeline. Re-uses `{components.eval-run-card}` chrome.
- Properties: `backgroundColor`, `borderColor`, `rounded`, `padding`, `scoreColor`, `chipColor`.
**`ex-score-chip`** — Dynamic HSL score pill. Re-uses `{components.score-chip}`.
- Properties: `score (0-1)`, `label`, computed `bg` and `fg` via `scoreColor`.
**`ex-model-group-header`** — Collapsible header for the grouped-by-model dashboard view. Re-uses `{components.row-card}` chrome with a chevron, model name, run-count, and best-score callout.
- Properties: `backgroundColor`, `borderColor`, `rounded`, `expanded`.
**`ex-leaderboard-row`** — A row in a benchmark leaderboard. Re-uses `{components.table}` row chrome.
- Properties: `rowBackground`, `borderColor`, `cellTypography`, `scoreChip`.
**`ex-chat-bubble-user`** / **`ex-chat-bubble-bot`** / **`ex-chat-bubble-tool`** — Three of the five chat-bubble roles. Re-use `{components.chat-bubble}` with the role's token family.
- Properties: `backgroundColor`, `borderColor`, `iconBackground`, `color`, `rounded`, `padding`.
**`ex-empty-state-card`** — No-data state. Re-uses `{components.empty-state}` inside a `{components.card}` shell.
- Properties: `backgroundColor`, `iconColor`, `messageTypography`.
**`ex-skeleton-row`** — Loading state for a table or list row.
- Properties: `backgroundColor`, `pulseAnimation` (`skeletonPulse`, 1.5 s).
**`ex-form-field`** — Label + input + optional error helper. Re-uses `{components.label}` + `{components.input}`.
- Properties: `labelTypography`, `inputBackground`, `borderColor`, `focusRingColor`, `errorState`.
**`ex-compare-column`** — A model column in the compare view, using one of the three `{compare.0..2}` slot colors.
- Properties: `dotColor`, `borderColor`, `headerBackground`.
## Do's and Don'ts
### Do
- Reserve `{colors.accent}` (`#816DF8`) for primary CTAs, active states, focus rings, and the wordmark accent. Brand violet IS the conversion target — keep it under ~10 % of any screen.
- Use `{rounded.sm}` 8 px for buttons / inputs / tabs, `{rounded.md}` 12 px for cards, `{rounded.full}` only for badges / chips / score pills. Each shape signals its category.
- Set every section eyebrow, form label, and table header in `{typography.label-xs}` — UPPERCASE + `tracking-wider`. This is the brand's hierarchy signal; without it, the design flattens.
- Use `{typography.caption-mono}` + `tabular-nums` for any numeric column — scores, timestamps, percentages. Treat numbers as data, not prose.
- Compute score colors with `hsl(score × 120, 70%, 45%)` and use the same formula for foreground and a translucent background. The dynamic chip is the product's emotional signal.
- Layer stacked shadows (a deep multi-stop shadow + a 1-px hairline) rather than single heavy drops. Cards sit on the page, not above it. On dark the hairline is translucent violet; on light the hairline is solid warm-grey hex — see `{colors.border}`.
- Cycle page chrome through the surface ladder `{colors.bg}` → `{colors.bg-deep}` → `{colors.bg-card}` → `{colors.bg-card2}`. The ladder semantics are theme-agnostic: inputs sit *deeper* than cards, hover sits *higher* — even though dark walks an indigo ladder and light walks a warm-cream ladder.
- **Theme parity:** when adding a new colour token, define BOTH a dark and a light value at the same time, in the same commit. The light value is not a translation of the dark — pick it for the warm-cream context. Token names without a light pair will eventually fall back to a default that breaks one of the themes.
- Animate page transitions with `fadeInUp` (12 px translate + opacity, 400 ms ease-out) and stagger children at 60 ms. The motion is subtle — don't lengthen it.
- Persist the `data-theme` to `localStorage` and apply it pre-paint in `index.html` to avoid FOUC. Tokens are theme-agnostic by name; values flip.
### Don't
- Don't introduce a 6th brand hue or a 4th compare-slot color. The palette is closed at violet + emerald + amber + red + slate (plus the dynamic HSL score). New accents flatten the voice.
- Don't render headlines in all-caps. UPPERCASE is the eyebrow voice (12 px / 10 px micro-labels) — never the title voice. Card titles and model names stay sentence-case.
- Don't promote the sans to `font-extrabold` / `font-black`. The display weight ceiling is **700**.
- Don't use `{colors.text-dim}` for essential UI text on either theme — its ~3.6 : 1 contrast against `{colors.bg-card}` clears AA Large (3 : 1) but is below WCAG AA Normal (4.5 : 1). Reserve for ≥ 14 px non-essential metadata (timestamps, "empty" labels, scrollbar thumb). Every code-side use must carry the inline note `// text-dim allowed: non-essential ≥14px metadata (DESIGN.md §Text)` so reviewers can audit it.
- Don't reuse a dark-theme shadow value verbatim on light. The light palette stacks two **warm-ink-tinted** drops (`rgba(20,20,19,0.04)` + `rgba(20,20,19,0.06)`) — a single `rgba(0,0,0,0.07)` drop on cream reads as a page smudge, not as a lifted card. Do not slate-tint the light shadows either (`rgba(15,23,42,*)`) — slate on cream reads as a cool-grey smear that fights the warm canvas. See *Elevation & Depth*.
- **Don't use translucent violet for light-theme hairlines.** This is the most-broken light-theme rule. The light `{colors.border-light}` is a SOLID warm-grey hex (`#e6dfd8`) — translucent violet at *any* plausible alpha (0.10 / 0.20 / 0.30 / 0.40) composites to near-invisible against a white card on cream and leaves every card boundary undefined. Outline buttons and input rings will vanish. The dark theme uses translucent violet because the near-black bg-to-card luminance delta carries the boundary; the light theme has no such delta and must use a concrete material.
- Don't introduce a cool-grey or pure-white surface to the light theme. The light palette is warm-cream by design (`#faf9f5` canvas, `#f0ebe1` deep, `#f5f0e7` elevated, `#ffffff` cards). A cool-grey `#f5f6fa` or `#eef0f7` band breaks the warm-coherent rhythm and reverts the system to "any other AI dashboard."
- Don't drop a single heavy 8-px-blur drop-shadow on a card. The dark theme requires *deeper* multi-stop shadows (`rgba(0,0,0,0.55)` at 20-40 px) — soft drops disappear on near-black.
- Don't apply `{gradients.brand}` to body text or table cells. Gradient-text is for hero / wordmark moments only.
- Don't bypass `{components.button}` to write custom `bg-[var(--accent)]` buttons inline. The button variants encode the glow, the scale-press, and the disabled state — re-deriving them by hand drifts the brand.
- Don't use inline `style={{ background: 'var(--xxx)' }}` when a Tailwind class or the `formStyles` helper would do. Inline styles bypass the token abstraction and break theme switching for the hover state.
- Don't pair `{rounded.full}` pill shapes with `{rounded.md}` cards as siblings inside the same control group — pills are for *data*, sm/md radii are for *interactive containers*. Mixing them on the same row breaks the shape grammar.
- Don't ignore `prefers-color-scheme` on first visit. If the user has never toggled, fall back to the OS preference before defaulting to dark.
- Don't widen `{spacing.xl}` (20 px) section gaps past `{spacing.2xl}` (24 px). The product is information-dense by design; extra whitespace makes the dashboard feel half-empty rather than airy.

203
evalscope/LICENSE Normal file
View File

@ -0,0 +1,203 @@
Copyright 2022-2023 Alibaba ModelScope. All rights reserved.
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2022-2023 Alibaba ModelScope.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

19
evalscope/MANIFEST.in Normal file
View File

@ -0,0 +1,19 @@
include README.md
# Include all resources (code + other files) inside the package
recursive-include evalscope *
# Exclude cache/compiled artifacts
global-exclude *.py[cod] __pycache__ *.so *.dylib
# Exclude large data files
global-exclude *.h5 *.hdf5 *.parquet *.bin *.safetensors *.gguf *.pth *.pt
# Exclude frontend dev files (only dist/ is needed at runtime)
prune evalscope/web/node_modules
prune evalscope/web/src
exclude evalscope/web/package.json
exclude evalscope/web/package-lock.json
exclude evalscope/web/tsconfig*.json
exclude evalscope/web/vite.config.ts
exclude evalscope/web/eslint.config.js

130
evalscope/Makefile Normal file
View File

@ -0,0 +1,130 @@
# default rule
default: install
# ============================================================================
# Documentation Generation
# ============================================================================
#
# WORKFLOW (full pipeline):
# docs-update → docs-translate → docs-generate → docs-en / docs-zh
#
# STEP-BY-STEP:
# Step 1 docs-update[/stats] Read adapter metadata → write _meta/<name>.json
# Step 2 docs-translate Translate readme.en → readme.zh via LLM API
# Step 3 docs-generate Read all _meta/*.json → write docs/*/benchmarks/*.md
# Step 4 docs-en / docs-zh Sphinx build → docs/*/build/html/
#
# WHAT IS AFFECTED:
# docs-update Writes evalscope/benchmarks/_meta/<name>.json (metadata only)
# docs-update-stats Same as above + downloads dataset to compute sample statistics
# docs-translate Updates readme.zh field inside each _meta/<name>.json
# docs-generate Overwrites docs/en/benchmarks/*.md + docs/zh/benchmarks/*.md
#
# PARAMETERS:
# BENCHMARK Specific benchmark name (e.g. gsm8k, mmlu).
# Omit to process ALL registered benchmarks.
# FORCE=1 Force recompute/re-translate even when data already exists.
# Applies to docs-update, docs-update-stats, and docs-translate.
# WORKERS Parallel worker count for update / translate (default: 4).
#
# COMMON USAGE:
# make docs # Full pipeline: translate → generate → build HTML
# make docs-update # Update metadata for ALL benchmarks
# make docs-update BENCHMARK=gsm8k # Update metadata for ONE benchmark
# make docs-update BENCHMARK="gsm8k mmlu" # Update metadata for MULTIPLE benchmarks
# make docs-update-stats # Update metadata + stats for ALL benchmarks
# make docs-update-stats BENCHMARK=gsm8k # Update metadata + stats for ONE benchmark
# make docs-update-stats BENCHMARK="gsm8k mmlu" # Update metadata + stats for MULTIPLE benchmarks
# make docs-translate # Translate only untranslated benchmarks (ALL)
# make docs-translate BENCHMARK=gsm8k # Translate ONE benchmark (skip if done)
# make docs-translate BENCHMARK="gsm8k mmlu" # Translate MULTIPLE benchmarks
# make docs-translate FORCE=1 # Force re-translate ALL benchmarks
# make docs-translate BENCHMARK=gsm8k FORCE=1 # Force re-translate ONE benchmark
# make docs-pipeline BENCHMARK=gsm8k # update-stats + translate + generate for ONE benchmark
# make docs-pipeline BENCHMARK="gsm8k mmlu" # update-stats + translate + generate for MULTIPLE
# make docs-pipeline BENCHMARK=gsm8k FORCE=1 # Force update-stats + translate + generate
# make docs-generate # Regenerate .md files from persisted JSON data
# make docs-en # Build English HTML docs only
# make docs-zh # Build Chinese HTML docs only
#
# ============================================================================
# Parameters
# BENCHMARK: one or more benchmark names, space-separated (e.g. BENCHMARK="gsm8k mmlu")
BENCHMARK ?=
FORCE ?=
WORKERS ?= 4
# Internal helpers
# When BENCHMARK is set: pass name(s) as positional args; otherwise use --all flag
_BENCH_ARGS = $(if $(BENCHMARK),$(BENCHMARK),--all)
# When FORCE is non-empty (e.g. FORCE=1): append --force flag
_FORCE_FLAG = $(if $(FORCE),--force,)
.PHONY: docs
docs: docs-translate docs-generate
$(MAKE) docs-en
$(MAKE) docs-zh
.PHONY: docs-update
docs-update:
python -m evalscope.cli.cli benchmark-info $(_BENCH_ARGS) --update $(_FORCE_FLAG) --workers $(WORKERS)
.PHONY: docs-update-stats
docs-update-stats:
python -m evalscope.cli.cli benchmark-info $(_BENCH_ARGS) --update --compute-stats $(_FORCE_FLAG) --workers $(WORKERS)
.PHONY: docs-translate
docs-translate:
python -m evalscope.cli.cli benchmark-info $(_BENCH_ARGS) --translate $(_FORCE_FLAG) --workers $(WORKERS)
.PHONY: docs-pipeline
docs-pipeline:
python -m evalscope.cli.cli benchmark-info $(_BENCH_ARGS) --update --compute-stats $(_FORCE_FLAG) --workers $(WORKERS)
python -m evalscope.cli.cli benchmark-info $(_BENCH_ARGS) --translate $(_FORCE_FLAG) --workers $(WORKERS)
python -m evalscope.cli.cli benchmark-info --generate-docs
.PHONY: docs-generate
docs-generate:
python -m evalscope.cli.cli benchmark-info --generate-docs
.PHONY: docs-en
docs-en:
cd docs/en && make clean && make html
.PHONY: docs-zh
docs-zh:
cd docs/zh && make clean && make html
# ============================================================================
# Frontend (evalscope/web)
# ============================================================================
.PHONY: web-install
web-install:
cd evalscope/web && npm install
.PHONY: web-build
web-build:
cd evalscope/web && npm install && npm run build
.PHONY: web-dev
web-dev:
cd evalscope/web && npm install && npm run dev
# ============================================================================
# Development
# ============================================================================
.PHONY: lint
lint:
pre-commit run --all-files
.PHONY: dev
dev:
pip install -e '.[dev,perf,docs]'
pip install pre-commit
.PHONY: install
install:
pip install -e .

176
evalscope/README.md Normal file
View File

@ -0,0 +1,176 @@
# EvalScope 评测仓库
## 0. Benchmarks
### 常用 benchmark 分类
```python
benchmark_categories = {
"代码与工程": [
"terminal_bench_v2",
"live_code_bench",
"swe_bench_multilingual_agentic",
"swe_bench_pro",
"swe_bench_verified",
"bigcodebench",
"humaneval",
],
"推理与数学": [
"gpqa_diamond",
"hle",
"aime24",
"aime25",
"aime26",
"hmmt26",
"imo_answerbench",
"super_gpqa",
"gsm8k",
"competition_math",
],
"智能体与工具": [
"browsecomp",
"mcp_atlas",
"tau2_bench",
],
"知识与语言理解": [
"mmlu_pro",
"simple_qa",
"arc",
"bbh",
"cmmlu",
"drop",
"hellaswag",
"mmlu",
"trivia_qa",
"winogrande",
],
"长上下文": [
"longbench_v2",
"openai_mrcr",
],
}
```
### 分组运行
```python
group_1 = [
'terminal_bench_v2',
'gpqa_diamond',
'hle',
'aime24',
'aime25',
'mmlu_pro',
'simple_qa',
'arc',
'bbh',
'browsecomp',
]
group_2 = [
'live_code_bench',
'swe_bench_pro',
'aime26',
'hmmt26',
'imo_answerbench',
'super_gpqa',
'drop',
'hellaswag',
'mmlu',
'openai_mrcr',
'mcp_atlas',
]
group_3 = [
'swe_bench_multilingual_agentic',
'swe_bench_verified',
'bigcodebench',
'humaneval',
'gsm8k',
'competition_math',
'cmmlu',
'trivia_qa',
'winogrande',
'longbench_v2',
'tau2_bench',
]
```
## 1. 安装
```bash
conda create -n evalscope python==3.12 -y
git clone https://github.com/modelscope/evalscope
pip install -e .
pip install 'evalscope[terminal_bench,swe_bench,openai_mrcr]' \
'git+https://github.com/sierra-research/tau2-bench@v0.2.0'
pip install 'evalscope[sandbox]'
pip install 'swebench==4.1.0'
```
## 2. 运行命令
```python
from evalscope import run_task, TaskConfig
datasets = [
'live_code_bench',
'swe_bench_pro',
'aime26',
'hmmt26',
'imo_answerbench',
'super_gpqa',
'drop',
'hellaswag',
'mmlu',
'openai_mrcr',
'mcp_atlas',
]
task_cfg = TaskConfig(
collect_perf=True,
work_dir='/data1/sora/benchmarks/temp/output',
use_cache='/data1/sora/benchmarks/temp/output',
no_timestamp=True,
limit=1,
model='',
api_url='http://localhost:30000/v1',
eval_type='openai_api',
datasets=datasets,
dataset_dir='/data1/sora/benchmarks/bash/datasets',
generation_config={
'temperature': 0.0,
'stream': True,
'max_tokens': 4096,
},
eval_batch_size=1,
judge_model_args={
"model_id": "deepseek-v4-flash",
"api_url": "https://api.deepseek.com/v1",
"api_key": "<YOUR_DEEPSEEK_API_KEY>",
"eval_type": "openai_api",
"generation_config": {
"temperature": 0.0,
"max_tokens": 1024 * 10,
},
}
)
run_task(task_cfg)
```
## 3. 评测结果
P800 模型能力评测结果。
## 4. UI 界面
```bash
pip install flask sse_starlette
cd ./evalscope/web
npm install
npm run build
evalscope service
```
然后访问 `http://127.0.0.1:9000/dashboard`。

392
evalscope/README_zh.md Normal file
View File

@ -0,0 +1,392 @@
<p align="center">
<br>
<img src="docs/en/_static/images/evalscope_logo.png"/>
<br>
<p>
<p align="center">
中文 &nbsp ｜ &nbsp <a href="README.md">English</a> &nbsp
</p>
<p align="center">
<img src="https://img.shields.io/badge/python-%E2%89%A53.10-5be.svg">
<a href="https://badge.fury.io/py/evalscope"><img src="https://badge.fury.io/py/evalscope.svg" alt="PyPI version" height="18"></a>
<a href="https://pypi.org/project/evalscope"><img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/evalscope"></a>
<a href="https://github.com/modelscope/evalscope/pulls"><img src="https://img.shields.io/badge/PR-welcome-55EB99.svg"></a>
<a href='https://evalscope.readthedocs.io/zh-cn/latest/?badge=latest'><img src='https://readthedocs.org/projects/evalscope/badge/?version=latest' alt='Documentation Status' /></a>
<p>
<p align="center">
<a href="https://evalscope.readthedocs.io/zh-cn/latest/"> 📖 中文文档</a> &nbsp ｜ &nbsp <a href="https://evalscope.readthedocs.io/en/latest/"> 📖 English Documents</a>
<p>
> ⭐ 如果你喜欢这个项目，请点击右上角的 "Star" 按钮支持我们。你的支持是我们前进的动力！
## 📝 简介
EvalScope 是由[魔搭社区](https://modelscope.cn/)打造的一站式大模型评测框架。一行命令即可开始评测，支持模型能力评估、推理性能压测和结果可视化。
```bash
pip install evalscope
evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5
```
## ✨ 主要特性
- **📚 全面的评测基准**: 内置 MMLU, C-Eval, GSM8K 等多个业界公认的评测基准。
- **🧩 多模态与多领域支持**: 支持大语言模型 (LLM)、多模态 (VLM)、Embedding、Reranker、AIGC 等多种模型的评测。
- **🚀 多后端集成**: 无缝集成 OpenCompass, VLMEvalKit, RAGEval 等多种评测后端，满足不同评测需求。
- **🤖 Agent 评测模式**: 在受控的多轮 AgentLoop 中驱动 GSM8K、AIME、SWE-bench Agentic 等基准；支持可插拔的策略、工具与 Docker 沙箱，每条样本完整记录 Agent Trace 并可在仪表盘中按步骤回放。
- **⚡ 推理性能测试**: 提供强大的模型服务压力测试工具，支持 TTFT, TPOT 等多项性能指标。
- **📊 交互式报告**: 提供 WebUI 可视化界面，支持多维度模型对比、报告概览和详情查阅。
- **⚔️ 竞技场模式**: 支持多模型对战 (Pairwise Battle)，直观地对模型进行排名和评估。
- **🔧 高度可扩展**: 开发者可以轻松添加自定义数据集、模型和评测指标。
## 📊 可视化效果展示
EvalScope 提供交互式 Web Dashboard，支持多维度模型对比和深入分析。
<table>
<tr>
<td style="text-align: center;">
<img src="https://sail-moe.oss-cn-hangzhou.aliyuncs.com/yunlin/images/evalscope/dashboard/dashboard_overview.png" alt="Dashboard" style="width: 100%;" />
<p>仪表盘概览</p>
</td>
<td style="text-align: center;">
<img src="https://sail-moe.oss-cn-hangzhou.aliyuncs.com/yunlin/images/evalscope/dashboard/compare_score_tab.png" alt="Model Compare" style="width: 90%;" />
<p>多模型对比</p>
</td>
</tr>
<tr>
<td style="text-align: center;">
<img src="https://sail-moe.oss-cn-hangzhou.aliyuncs.com/yunlin/images/evalscope/dashboard/report_overview_tab.png" alt="Report Overview" style="width: 100%;" />
<p>报告概览</p>
</td>
<td style="text-align: center;">
<img src="https://sail-moe.oss-cn-hangzhou.aliyuncs.com/yunlin/images/evalscope/dashboard/report_predictions_tab.png" alt="Report Predictions" style="width: 80%;" />
<p>预测标签</p>
</td>
</tr>
</table>
详情请参考 [📖 可视化评测结果](https://evalscope.readthedocs.io/zh-cn/latest/get_started/visualization.html)。
## 🎉 内容更新
- 🔥 **[2026.07.03]** 新增 **CharXiv** & **BabyVision**（图表理解、视觉认知）及 **ERQA** & **WorldVQA**（实体识别 QA，支持 LLM judge + CoT）多模态基准。
- 🔥 **[2026.06.23]** Agent 与代码评测大幅扩展：新增 **BigCodeBench**、**SWE-bench Multilingual**、**BrowseComp**、**MCP-Atlas**、**GDPval** 基准；新增 **OpenCode** / **OpenHands** runner；适配器架构重构，引入 `AudioLanguageAdapter`、统一 `FunctionCallAdapter`、公开 `run_agent_loop` API。
- 🔥 **[2026.06.16]** 新增全参考**图像质量指标**（SSIM、PSNR 等）、长上下文基准（**LoCoMo QA**、**LongMemEval**）、**Caption** & **Maritime-OCR-Bench** 基准；Perf 模块支持统一 `--data-source` 参数及并行化请求生成。
- 🔥 **[2026.06.02]** **RAG 评测**模块重构：升级 MTEB 2.x 与 RAGAS 0.4.x，配置统一为 Pydantic 模型。参考[RAGEval 使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/index.html)。
- 🔥 **[2026.05.27]** 新增 **Trie agentic 轨迹回放**压测：三个数据集插件（`trie_agentic_coding` / `trie_code_qa` / `trie_office_work`）可回放真实多轮 Agent 轨迹，支持逐轮 token 上限和工具调用延迟模拟；同时新增 `--duration` 墙钟时间预算（适用于所有压测模式）和 `Turn` 数据类。
- 🔥 **[2026.05.27]** 新增 **Vendor Verifier 基准**（`k2_verifier`、`kimi_verifier`、`minimax_verifier`），用于验证第三方 API 部署是否忠实复现官方模型行为，共享 `FunctionCallAdapter` 基类。
- 🔥 **[2026.05.26]** 新增 [GAIA](https://evalscope.readthedocs.io/zh-cn/latest/third_party/gaia.html) agent 基准（Docker sandbox 内多轮 ReAct + `bash`，复用官方规则评分器）和通用 [MCP 服务器](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/agent/native.html#mcp-工具接入)接入 —— 任何基于 `NativeAgentConfig` 的 benchmark 都可直接挂载 stdio / HTTP / SSE 的 MCP server（`fetch`、网页搜索、GitHub 等），无需 benchmark 端改动。
- 🔥 **[2026.05.22]** 新增 **外部 Agent Bridge** 模式：可直接评测 Anthropic [Claude Code](https://github.com/anthropics/claude-code)、OpenAI [Codex](https://github.com/openai/codex) 等成品 Agent CLI。Bridge 透明转发 CLI 的 LLM 请求（Anthropic Messages / OpenAI Chat / OpenAI Responses，含 SSE 流式响应）到评测模型，同时把完整交互轨迹录制为 `agent_trace`；通过 `@register_runner` 可接入任意第三方 CLI。详见[外部 Agent Bridge 指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/agent/bridge.html)。
- 🔥 **[2026.05.19]** 新增对 [SWE-bench_Pro](https://evalscope.readthedocs.io/zh-cn/latest/third_party/swe_bench_pro.html) 与 [τ³-bench](https://evalscope.readthedocs.io/zh-cn/latest/third_party/tau3_bench.html) 的支持：SWE-bench_Pro 是 Scale AI 推出的更具挑战性的多语言、长周期软件工程基准，相比原始 SWE-bench 数据污染更少、覆盖语言更广，**推荐替代原始 SWE-bench 使用**，每个实例的 Docker 镜像直接从 DockerHub 拉取，无需本地构建；τ³-bench 是 tau-bench 系列的 v1.0.0 版本，在 τ²-bench 基础上新增 `banking_knowledge` 知识检索领域（RAG）、修复 75+ 项任务，并提供可插拔的检索流水线（BM25 / 稠密嵌入 / 重排序器 / 沙箱 shell）。
- 🔥 **[2026.05.15]** 新增 **Agent 评测模式**：所有基于 `DefaultDataAdapter` 的基准（GSM8K、AIME、IFEval 等）现在均可通过多轮 AgentLoop 驱动，支持可插拔策略（`function_calling` / `react` / `swe_bench_*`）、工具（`bash` / `python_exec` / `submit`）以及 `local` / `docker` 运行环境，每条样本的 `agent_trace` 会随评测结果落盘，并在仪表盘的预测视图中按步骤回放。详见[Agent 评测指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/agent/native.html)。
- 🔥 **[2026.05.08]** 与 [LightSeek](https://lightseek.org/) 联合推出 [TokenSpeed](https://lightseek.org/blog/lightseek-tokenspeed.html)——面向 Agentic 工作负载的极速 LLM 推理引擎。EvalScope 提供 SWE-smith 压测流水线，基于真实 Coding Agent 轨迹衡量单 GPU 吞吐（TPM）与单用户延迟（TPS），作为 TokenSpeed 性能评测的官方基准工具。参考 [SWE-smith 使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/multi_turn.html#swe-smith) 快速上手。
- 🔥 **[2026.05.07]** 全新 Web 界面升级：使用 React + Vite 重构可视化平台，替换原有 Gradio 界面，提供更流畅的交互体验。
- 🔥 **[2026.04.23]** 支持在评测任务中记录性能（perf）指标，可在单次评测运行中同时追踪模型准确率与 TTFT、TPOT、吞吐量等推理效率指标。
- 🔥 **[2026.04.17]** 支持多轮对话性能压测，可对具备多轮上下文的对话模型服务进行负载测试，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/examples.html)。
<details><summary>更多历史更新</summary>
- 🔥 **[2026.04.10]** 新增支持 [TIR-Bench](https://arxiv.org/abs/2511.01833)（Thinking-with-Images Reasoning Benchmark），一个面向视觉语言模型的多模态推理基准。
- 🔥 **[2026.03.24]** 支持 Agent Skill，任何支持 Skill/Tool 调用的 Agent 模型均可通过自然语言直接驱动 EvalScope 完成模型评测、性能压测和结果可视化。
- 🔥 **[2026.03.09]** 支持评测进度追踪和自动生成HTML格式可视化报告。
- 🔥 **[2026.03.02]** 支持Anthropic Claude API评测，通过`--eval-type anthropic_api`指定使用Anthropic API服务进行评测。
- 🔥 **[2026.02.03]** 全面更新数据集说明文档，添加数据信息统计、数据样例、使用方法等部分，参考[支持的数据集](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/llm.html)
- 🔥 **[2026.01.13]** 支持Embedding和Rerank模型服务压测，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/examples.html#embedding)
- 🔥 **[2025.12.26]** 支持Terminal-Bench-2.0，用于评估 AI Agent在 89 个真实世界的多步骤终端任务上的表现，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/third_party/terminal_bench.html)
- 🔥 **[2025.12.18]** 支持SLA自动调优模型API服务，自动测试模型服务在特定时延、TTFT、吞吐量下的最高并发，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/sla_auto_tune.html)
- 🔥 **[2025.12.16]** 支持Fleurs、LibriSpeech等音频评测基准；支持MultiplE、MBPP等多语言代码评测基准。
- 🔥 **[2025.12.02]** 支持自定义多模态VQA评测，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/vlm.html) ；支持模型服务压测在 ClearML 上可视化，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/examples.html#clearml)。
- 🔥 **[2025.11.26]** 新增支持 OpenAI-MRCR、GSM8K-V、MGSM、MicroVQA、IFBench、SciCode 评测基准。
- 🔥 **[2025.11.18]** 支持自定义 Function-Call（工具调用）数据集，来测试模型能否适时并正确调用工具，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/llm.html#fc)
- 🔥 **[2025.11.14]** 新增支持SWE-bench_Verified, SWE-bench_Lite, SWE-bench_Verified_mini 代码评测基准，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/third_party/swe_bench.html)。
- 🔥 **[2025.11.12]** 新增`pass@k`、`vote@k`、`pass^k`等指标聚合方法；新增支持A_OKVQA, CMMU, ScienceQ, V*Bench等多模态评测基准。
- 🔥 **[2025.11.07]** 新增支持τ²-bench，是 τ-bench 的扩展与增强版本，包含一系列代码修复，并新增了电信（telecom）领域的故障排查场景，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/third_party/tau2_bench.html)。
- 🔥 **[2025.10.30]** 新增支持BFCL-v4，支持agent的网络搜索和长期记忆能力的评测，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/third_party/bfcl_v4.html)。
- 🔥 **[2025.10.27]** 新增支持LogiQA, HaluEval, MathQA, MRI-QA, PIQA, QASC, CommonsenseQA等评测基准。感谢 @[penguinwang96825](https://github.com/penguinwang96825) 提供代码实现。
- 🔥 **[2025.10.26]** 新增支持Conll-2003, CrossNER, Copious, GeniaNER, HarveyNER, MIT-Movie-Trivia, MIT-Restaurant, OntoNotes5, WNUT2017 等命名实体识别评测基准。感谢 @[penguinwang96825](https://github.com/penguinwang96825) 提供代码实现。
- 🔥 **[2025.10.21]** 优化代码评测中的沙箱环境使用，支持在本地和远程两种模式下运行，具体参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/sandbox.html)。
- 🔥 **[2025.10.20]** 新增支持PolyMath, SimpleVQA, MathVerse, MathVision, AA-LCR 等评测基准；优化evalscope perf表现，对齐vLLM Bench，具体参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/vs_vllm_bench.html)。
- 🔥 **[2025.10.14]** 新增支持OCRBench, OCRBench-v2, DocVQA, InfoVQA, ChartQA, BLINK 等图文多模态评测基准。
- 🔥 **[2025.09.22]** 代码评测基准(HumanEval, LiveCodeBench)支持在沙箱环境中运行，要使用该功能需先安装[ms-enclave](https://github.com/modelscope/ms-enclave)。
- 🔥 **[2025.09.19]** 新增支持RealWorldQA、AI2D、MMStar、MMBench、OmniBench等图文多模态评测基准，和Multi-IF、HealthBench、AMC等纯文本评测基准。
- 🔥 **[2025.09.05]** 支持视觉-语言多模态大模型的评测任务，例如：MathVista、MMMU，更多支持数据集请[参考](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/vlm.html)。
- 🔥 **[2025.09.04]** 支持图像编辑任务评测，支持[GEdit-Bench](https://modelscope.cn/datasets/stepfun-ai/GEdit-Bench) 评测基准，使用方法[参考](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/aigc/image_edit.html)。
- 🔥 **[2025.08.22]** Version 1.0 重构，不兼容的更新请[参考](https://evalscope.readthedocs.io/zh-cn/latest/get_started/basic_usage.html#v1-0)。
- 🔥 **[2025.07.18]** 模型压测支持随机生成图文数据，用于多模态模型压测，使用方法[参考](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/examples.html#id4)。
- 🔥 **[2025.07.16]** 支持[τ-bench](https://github.com/sierra-research/tau-bench)，用于评估 AI Agent在动态用户和工具交互的实际环境中的性能和可靠性，使用方法[参考](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/llm.html#bench)。
- 🔥 **[2025.07.14]** 支持"人类最后的考试"([Humanity's-Last-Exam](https://modelscope.cn/datasets/cais/hle))，这一高难度评测基准，使用方法[参考](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/llm.html#humanity-s-last-exam)。
- 🔥 **[2025.07.03]** 重构了竞技场模式，支持自定义模型对战，输出模型排行榜，以及对战结果可视化，使用[参考](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/arena.html)。
- 🔥 **[2025.06.28]** 优化自定义数据集评测，支持无参考答案评测；优化LLM裁判使用，预置"无参考答案直接打分" 和 "判断答案是否与参考答案一致"两种模式，使用[参考](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/llm.html#qa)
- 🔥 **[2025.06.19]** 新增支持[BFCL-v3](https://modelscope.cn/datasets/AI-ModelScope/bfcl_v3)评测基准，用于评测模型在多种场景下的函数调用能力，使用[参考](https://evalscope.readthedocs.io/zh-cn/latest/third_party/bfcl_v3.html)。
- 🔥 **[2025.06.02]** 新增支持大海捞针测试（Needle-in-a-Haystack），指定`needle_haystack`即可进行测试，并在`outputs/reports`文件夹下生成对应的heatmap，直观展现模型性能，使用[参考](https://evalscope.readthedocs.io/zh-cn/latest/third_party/needle_haystack.html)。
- 🔥 **[2025.05.29]** 新增支持[DocMath](https://modelscope.cn/datasets/yale-nlp/DocMath-Eval/summary)和[FRAMES](https://modelscope.cn/datasets/iic/frames/summary)两个长文档评测基准，使用注意事项请查看[文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/index.html)
- 🔥 **[2025.05.16]** 模型服务性能压测支持设置多种并发，并输出性能压测报告，[参考示例](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/quick_start.html#id3)。
- 🔥 **[2025.05.13]** 新增支持[ToolBench-Static](https://modelscope.cn/datasets/AI-ModelScope/ToolBench-Static)数据集，评测模型的工具调用能力，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/third_party/toolbench.html)；支持[DROP](https://modelscope.cn/datasets/AI-ModelScope/DROP/dataPeview)和[Winogrande](https://modelscope.cn/datasets/AI-ModelScope/winogrande_val)评测基准，评测模型的推理能力。
- 🔥 **[2025.04.29]** 新增Qwen3评测最佳实践，[欢迎阅读📖](https://evalscope.readthedocs.io/zh-cn/latest/best_practice/qwen3.html)
- 🔥 **[2025.04.27]** 支持文生图评测：支持MPS、HPSv2.1Score等8个指标，支持EvalMuse、GenAI-Bench等评测基准，参考[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/aigc/t2i.html)
- 🔥 **[2025.04.10]** 模型服务压测工具支持`/v1/completions`端点（也是vLLM基准测试的默认端点）
- 🔥 **[2025.04.08]** 支持OpenAI API兼容的Embedding模型服务评测，查看[使用文档](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/mteb.html#configure-evaluation-parameters)
- 🔥 **[2025.03.27]** 新增支持[AlpacaEval](https://www.modelscope.cn/datasets/AI-ModelScope/alpaca_eval/dataPeview)和[ArenaHard](https://modelscope.cn/datasets/AI-ModelScope/arena-hard-auto-v0.1/summary)评测基准，使用注意事项请查看[文档](https://evalscope.readthedocs.io/zh-cn/latest/get_started/supported_dataset/index.html)
- 🔥 **[2025.03.20]** 模型推理服务压测支持random生成指定范围长度的prompt，参考[使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/examples.html#random)
- 🔥 **[2025.03.13]** 新增支持[LiveCodeBench](https://www.modelscope.cn/datasets/evalscope/livecodebench_code_generation_lite_parquet/summary)代码评测基准，指定`live_code_bench`即可使用；支持QwQ-32B 在LiveCodeBench上评测，参考[最佳实践](https://evalscope.readthedocs.io/zh-cn/latest/best_practice/eval_qwq.html)。
- 🔥 **[2025.03.11]** 新增支持[SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/SimpleQA/summary)和[Chinese SimpleQA](https://modelscope.cn/datasets/AI-ModelScope/Chinese-SimpleQA/summary)评测基准，用与评测模型的事实正确性，指定`simple_qa`和`chinese_simpleqa`使用。同时支持指定裁判模型，参考[相关参数说明](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)。
- 🔥 **[2025.03.07]** 新增QwQ-32B模型评测最佳实践，评测了模型的推理能力以及推理效率，参考[📖QwQ-32B模型评测最佳实践](https://evalscope.readthedocs.io/zh-cn/latest/best_practice/eval_qwq.html)。
- 🔥 **[2025.03.04]** 新增支持[SuperGPQA](https://modelscope.cn/datasets/m-a-p/SuperGPQA/summary)数据集，其覆盖 13 个门类、72 个一级学科和 285 个二级学科，共 26,529 个问题，指定`super_gpqa`即可使用。
- 🔥 **[2025.03.03]** 新增支持评测模型的智商和情商，参考[📖智商和情商评测最佳实践](https://evalscope.readthedocs.io/zh-cn/latest/best_practice/iquiz.html)，来测测你家的AI有多聪明？
- 🔥 **[2025.02.27]** 新增支持评测推理模型的思考效率，参考[📖思考效率评测最佳实践](https://evalscope.readthedocs.io/zh-cn/latest/best_practice/think_eval.html)，该实现参考了[Overthinking](https://doi.org/10.48550/arXiv.2412.21187) 和 [Underthinking](https://doi.org/10.48550/arXiv.2501.18585)两篇工作。
- 🔥 **[2025.02.25]** 新增支持[MuSR](https://modelscope.cn/datasets/AI-ModelScope/MuSR)和[ProcessBench](https://www.modelscope.cn/datasets/Qwen/ProcessBench/summary)两个模型推理相关评测基准，datasets分别指定`musr`和`process_bench`即可使用。
- 🔥 **[2025.02.18]** 支持AIME25数据集，包含15道题目（Grok3 在该数据集上得分为93分）
- 🔥 **[2025.02.13]** 支持DeepSeek蒸馏模型评测，包括AIME24, MATH-500, GPQA-Diamond数据集，参考[最佳实践](https://evalscope.readthedocs.io/zh-cn/latest/best_practice/deepseek_r1_distill.html)；支持指定`eval_batch_size`参数，加速模型评测
- 🔥 **[2025.01.20]** 支持可视化评测结果，包括单模型评测结果和多模型评测结果对比，参考[📖可视化评测结果](https://evalscope.readthedocs.io/zh-cn/latest/get_started/visualization.html)
- 🔥 **[2025.01.07]** Native backend: 支持模型API评测，参考[📖模型API评测指南](https://evalscope.readthedocs.io/zh-cn/latest/get_started/basic_usage.html#api)；新增支持`ifeval`评测基准。
- 🔥🔥 **[2024.12.31]** 支持基准评测添加，参考[📖基准评测添加指南](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/add_benchmark.html)
- 🔥 **[2024.12.13]** 模型评测优化，不再需要传递`--template-type`参数；支持`evalscope eval --args`启动评测。
- 🔥 **[2024.11.26]** 模型推理压测工具重构完成：支持本地启动推理服务、支持Speed Benchmark。
- 🔥 **[2024.10.31]** 多模态RAG评测最佳实践发布。
- 🔥 **[2024.10.23]** 支持多模态RAG评测。
- 🔥 **[2024.10.8]** 支持RAG评测。
- 🔥 **[2024.09.18]** 文档添加博客模块。
- 🔥 **[2024.09.12]** 支持 LongWriter 评测。
- 🔥 **[2024.08.30]** 支持自定义数据集评测。
- 🔥 **[2024.08.20]** 更新了官方文档。
- 🔥 **[2024.08.09]** 简化安装方式，优化多模态模型评测体验。
- 🔥 **[2024.07.31]** 重要修改：`llmuses`包名修改为`evalscope`，请同步修改您的代码。
- 🔥 **[2024.07.26]** 支持**VLMEvalKit**作为第三方评测框架。
- 🔥 **[2024.06.29]** 支持**OpenCompass**作为第三方评测框架。
- 🔥 **[2024.06.13]** EvalScope与SWIFT微调框架集成；接入Agent评测集ToolBench。
</details>
## 🚀 快速开始
### 安装
```shell
pip install evalscope
```
> 详细安装说明（源码安装、额外依赖等）请参考 [📖 安装指南](https://evalscope.readthedocs.io/zh-cn/latest/get_started/installation.html)。
### 方式1. 评测在线模型 API（推荐入门方式，无需 GPU）
支持任意 OpenAI API 兼容的模型服务，只需配置 `$OPENAI_API_BASE_URL` 与 `$OPENAI_API_KEY` 即可开始评测：
```bash
evalscope eval \
--model your-model-name \
--api-url $OPENAI_API_BASE_URL \
--api-key $OPENAI_API_KEY \
--eval-type openai_api \
--datasets gsm8k arc \
--limit 5
```
### 方式2. 评测本地模型
使用本地模型（自动从 ModelScope 下载）：
```bash
evalscope eval \
--model Qwen/Qwen2.5-0.5B-Instruct \
--datasets gsm8k arc \
--limit 5
```
### 方式3. 使用 Python 代码
```python
from evalscope import run_task, TaskConfig
task_cfg = TaskConfig(
model='your-model-name',
api_url='https://your-openai-compatible-endpoint/v1',
api_key='your_api_key',
eval_type='openai_api',
datasets=['gsm8k', 'arc'],
limit=5
)
run_task(task_cfg)
```
<details><summary><b>💡 提示：</b> <code>run_task</code> 还支持字典、YAML 或 JSON 文件作为配置。</summary>
**使用 Python 字典**
```python
from evalscope.run import run_task
task_cfg = {
'model': 'Qwen/Qwen2.5-0.5B-Instruct',
'datasets': ['gsm8k', 'arc'],
'limit': 5
}
run_task(task_cfg=task_cfg)
```
**使用 YAML 文件** (`config.yaml`)
```yaml
model: Qwen/Qwen2.5-0.5B-Instruct
datasets:
- gsm8k
- arc
limit: 5
```
```python
from evalscope.run import run_task
run_task(task_cfg="config.yaml")
```
</details>
### 输出结果
评测完成后，您将在终端看到如下格式的报告：
```text
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Model Name | Dataset Name | Metric Name | Category Name | Subset Name | Num | Score |
+=======================+================+=================+=================+===============+=======+=========+
| Qwen2.5-0.5B-Instruct | gsm8k | AverageAccuracy | default | main | 5 | 0.4 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Qwen2.5-0.5B-Instruct | ai2_arc | AverageAccuracy | default | ARC-Easy | 5 | 0.8 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
| Qwen2.5-0.5B-Instruct | ai2_arc | AverageAccuracy | default | ARC-Challenge | 5 | 0.4 |
+-----------------------+----------------+-----------------+-----------------+---------------+-------+---------+
```
**启动可视化面板**：
```bash
pip install 'evalscope[service]'
evalscope service
```
访问 `http://127.0.0.1:9000` 即可打开可视化界面。
## 📈 进阶用法
### 自定义评测参数
您可以通过命令行参数精细化控制模型加载、推理和数据集配置。
```shell
evalscope eval \
--model Qwen/Qwen3-0.6B \
--model-args '{"revision": "master", "precision": "torch.float16", "device_map": "auto"}' \
--generation-config '{"do_sample":true,"temperature":0.6,"max_tokens":512}' \
--dataset-args '{"gsm8k": {"few_shot_num": 0, "few_shot_random": false}}' \
--datasets gsm8k \
--limit 10
```
- `--model-args`: 模型加载参数，如 `revision`, `precision` 等。
- `--generation-config`: 模型生成参数，如 `temperature`, `max_tokens` 等。
- `--dataset-args`: 数据集配置参数，如 `few_shot_num` 等。
详情请参考 [📖 全部参数说明](https://evalscope.readthedocs.io/zh-cn/latest/get_started/parameters.html)。
### ⚔️ 竞技场模式 (Arena)
竞技场模式通过模型间的两两对战（Pairwise Battle）来评估模型性能，并给出胜率和排名，非常适合多模型横向对比。
```text
# 评测结果示例
Model WinRate (%) CI (%)
------------ ------------- ---------------
qwen2.5-72b 69.3 (-13.3 / +12.2)
qwen2.5-7b 50 (+0.0 / +0.0)
qwen2.5-0.5b 4.7 (-2.5 / +4.4)
```
详情请参考 [📖 竞技场模式使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/arena.html)。
### 🖊️ 自定义数据集评测
EvalScope 允许您轻松添加和评测自己的数据集。详情请参考 [📖 自定义数据集评测指南](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/index.html)。
## ⚡ 推理性能评测工具
EvalScope 提供了一个强大的压力测试工具，用于评估大语言模型服务的性能。
- **关键指标**: 支持吞吐量 (Tokens/s)、首字延迟 (TTFT)、Token 生成延迟 (TPOT) 等。
- **结果记录**: 支持将结果记录到 `wandb` 和 `swanlab`。
- **速度基准**: 可生成类似官方报告的速度基准测试结果。
详情请参考 [📖 性能测试使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/index.html)。
<p align="center">
<img src="docs/zh/user_guides/stress_test/images/multi_perf.png" style="width: 80%;">
</p>
## 🧪 其他评测后端
EvalScope 支持通过第三方评测框架（我们称之为"后端"）发起评测任务，以满足多样化的评测需求。
- **Native**: EvalScope 的默认评测框架，功能全面。
- **OpenCompass**: 专注于纯文本评测。 [📖 使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/opencompass_backend.html)
- **VLMEvalKit**: 专注于多模态评测。 [📖 使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/vlmevalkit_backend.html)
- **RAGEval**: 专注于 RAG 评测，支持 Embedding 和 Reranker 模型。 [📖 使用指南](https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/index.html)
- **第三方评测工具**: 支持 [ToolBench](https://evalscope.readthedocs.io/zh-cn/latest/third_party/toolbench.html) 等评测任务。
<details><summary>🏛️ 整体架构</summary>
<p align="center">
<img src="https://sail-moe.oss-cn-hangzhou.aliyuncs.com/yunlin/images/evalscope/doc/EvalScope%E6%9E%B6%E6%9E%84%E5%9B%BE.png" style="width: 70%;">
<br>EvalScope 整体架构图.
</p>
1. **输入层**
- **模型来源**: API模型（OpenAI API）、本地模型（ModelScope）
- **数据集**: 标准评测基准（MMLU/GSM8k等）、自定义数据（MCQ/QA）
2. **核心功能**
- **多后端评估**: 原生后端、OpenCompass、MTEB、VLMEvalKit、RAGAS
- **性能监控**: 支持多种模型服务 API 和数据格式，追踪 TTFT/TPOP 等指标
- **工具扩展**: 集成 Tool-Bench, Needle-in-a-Haystack 等
3. **输出层**
- **结构化报告**: 支持 JSON, Table, Logs
- **可视化平台**: 支持 Web Dashboard, Wandb, SwanLab
</details>
## ❤️ 社区与支持
欢迎加入我们的社区，与其他开发者交流并获取帮助。
[Discord Group](https://discord.gg/xc66bMxc4h) | 微信群 | 钉钉群
:-------------------------:|:-------------------------:|:-------------------------:
<img src="docs/asset/discord_qr.png" width="160" height="160"> | <img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/asset/wechat.png" width="160" height="160"> | <img src="docs/asset/dingding.png" width="160" height="160">
## 👷‍♂️ 贡献
我们欢迎来自社区的任何贡献！如果您希望添加新的评测基准、模型或功能，请参考我们的 [贡献指南](https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/add_benchmark.html)。
感谢所有为 EvalScope 做出贡献的开发者！
<a href="https://github.com/modelscope/evalscope/graphs/contributors" target="_blank">
<table>
<tr>
<th colspan="2">
<br><img src="https://contrib.rocks/image?repo=modelscope/evalscope"><br><br>
</th>
</tr>
</table>
</a>
## 📚 引用
如果您在研究中使用了 EvalScope，请引用我们的工作：
```bibtex
@misc{evalscope_2024,
title={{EvalScope}: Evaluation Framework for Large Models},
author={ModelScope Team},
year={2024},
url={https://github.com/modelscope/evalscope}
}
```
## ⭐ Star History
[![Star History Chart](https://api.star-history.com/svg?repos=modelscope/evalscope&type=Date)](https://star-history.com/#modelscope/evalscope&Date)

BIN
evalscope/custom_eval/multimodal/images/AMNH.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

BIN
evalscope/custom_eval/multimodal/images/dog.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 112 KiB

BIN
evalscope/custom_eval/multimodal/images/running.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.3 KiB

BIN
evalscope/custom_eval/multimodal/images/tesla.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 34 KiB

BIN
evalscope/custom_eval/multimodal/images/tokyo.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 75 KiB

3
evalscope/custom_eval/multimodal/mcq/example.jsonl Normal file
View File

@ -0,0 +1,3 @@
{"question": "Which image shows a dog?", "options": ["<image 1>", "<image 2>", "<image 3>", "<image 4>"], "image_1": "custom_eval/multimodal/images/dog.jpg", "image_2": "custom_eval/multimodal/images/AMNH.jpg", "image_3": "custom_eval/multimodal/images/tesla.jpg", "image_4": "custom_eval/multimodal/images/tokyo.jpg", "answer": "A"}
{"question": "<image 1> What building is this?", "options": ["School", "Hospital", "Park", "Museum"], "image_1": "custom_eval/multimodal/images/AMNH.jpg", "answer": "D"}
{"question": "<image 1> Which city's skyline is this?", "options": ["New York", "Tokyo", "Shanghai", "Paris"], "image_1": "custom_eval/multimodal/images/tokyo.jpg", "answer": "B"}

4
evalscope/custom_eval/multimodal/mcq/example.tsv Normal file
View File

@ -0,0 +1,4 @@
question options answer image_1 image_2 image_3 image_4
Which image shows a dog? ["<image 1>", "<image 2>", "<image 3>", "<image 4>"] A custom_eval/multimodal/images/dog.jpg custom_eval/multimodal/images/AMNH.jpg custom_eval/multimodal/images/tesla.jpg custom_eval/multimodal/images/tokyo.jpg
<image 1> What building is this? ["School", "Hospital", "Park", "Museum"] D custom_eval/multimodal/images/AMNH.jpg
<image 1> Which city's skyline is this? ["New York", "Tokyo", "Shanghai", "Paris"] B custom_eval/multimodal/images/tokyo.jpg
Can't render this file because it contains an unexpected character in line 2 and column 27.

1
evalscope/custom_eval/multimodal/mcq/example_video.jsonl Normal file
View File

@ -0,0 +1 @@
{"question": "<video 1> What type of media is provided in this sample?", "options": ["Image", "Audio", "Video", "Text"], "video_1": "custom_eval/multimodal/videos/sample.mp4", "answer": "C"}

10
evalscope/custom_eval/multimodal/t2i/example.jsonl Normal file
View File

@ -0,0 +1,10 @@
{"image_path": "custom_eval/multimodal/images/dog.jpg", "prompt": "dog"}
{"image_path": "custom_eval/multimodal/images/dog.jpg", "prompt": "cat"}
{"image_path": "custom_eval/multimodal/images/AMNH.jpg", "prompt": "building"}
{"image_path": "custom_eval/multimodal/images/AMNH.jpg", "prompt": "Grand historic building with columns, glass doors, digital screen, and busy open area. Classical yet modern."}
{"image_path": "custom_eval/multimodal/images/tokyo.jpg", "prompt": "city tokyo"}
{"image_path": "custom_eval/multimodal/images/tokyo.jpg", "prompt": "city newyork"}
{"image_path": "custom_eval/multimodal/images/tesla.jpg", "prompt": "car tesla"}
{"image_path": "custom_eval/multimodal/images/tesla.jpg", "prompt": "car toyota"}
{"image_path": "custom_eval/multimodal/images/running.jpg", "prompt": "man running"}
{"image_path": "custom_eval/multimodal/images/running.jpg", "prompt": "man eating"}

5
evalscope/custom_eval/multimodal/text-image-retrieval/image_queries.jsonl Normal file
View File

@ -0,0 +1,5 @@
{"image_path": "custom_eval/multimodal/images/dog.jpg", "query": ["dog"]}
{"image_path": "custom_eval/multimodal/images/AMNH.jpg", "query": ["building"]}
{"image_path": "custom_eval/multimodal/images/tokyo.jpg", "query": ["city", "tokyo"]}
{"image_path": "custom_eval/multimodal/images/tesla.jpg", "query": ["car", "tesla"]}
{"image_path": "custom_eval/multimodal/images/running.jpg", "query": ["man", "running"]}

BIN
evalscope/custom_eval/multimodal/videos/sample.mp4 Normal file
View File

Binary file not shown.

5
evalscope/custom_eval/multimodal/vqa/example_openai.jsonl Normal file
View File

@ -0,0 +1,5 @@
{"messages": [{"role": "user", "content": [{"type": "text", "text": "What animal is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/dog.jpg"}}]}], "answer": "Dog"}
{"messages": [{"role": "user", "content": [{"type": "text", "text": "What building is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/AMNH.jpg"}}]}], "answer": "Museum"}
{"messages": [{"role": "user", "content": [{"type": "text", "text": "Which city's skyline is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/tokyo.jpg"}}]}], "answer": "Tokyo"}
{"messages": [{"role": "user", "content": [{"type": "text", "text": "What is the brand of this car?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/tesla.jpg"}}]}], "answer": "Tesla"}
{"messages": [{"role": "user", "content": [{"type": "text", "text": "What is the person in the picture doing?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/running.jpg"}}]}], "answer": "Running"}

6
evalscope/custom_eval/multimodal/vqa/example_openai.tsv Normal file
View File

@ -0,0 +1,6 @@
messages answer
[{"role": "user", "content": [{"type": "text", "text": "What animal is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/dog.jpg"}}]}] Dog
[{"role": "user", "content": [{"type": "text", "text": "What building is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/AMNH.jpg"}}]}] Museum
[{"role": "user", "content": [{"type": "text", "text": "Which city's skyline is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/tokyo.jpg"}}]}] Tokyo
[{"role": "user", "content": [{"type": "text", "text": "What is the brand of this car?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/tesla.jpg"}}]}] Tesla
[{"role": "user", "content": [{"type": "text", "text": "What is the person in the picture doing?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/running.jpg"}}]}] Running
Can't render this file because it contains an unexpected character in line 2 and column 3.

1
evalscope/custom_eval/multimodal/vqa/example_video.jsonl Normal file
View File

@ -0,0 +1 @@
{"messages": [{"role": "user", "content": [{"type": "text", "text": "What type of media is provided in this sample?"}, {"type": "video_url", "video_url": {"url": "custom_eval/multimodal/videos/sample.mp4"}}]}], "answer": "Video"}

3
evalscope/custom_eval/text/fc/example.jsonl Normal file
View File

@ -0,0 +1,3 @@
{"messages":[{"role":"system","content":"你是助手"},{"role":"user","content":"请把 2 和 3 相加"}],"tools":[{"type":"function","function":{"name":"add","description":"将两个数字相加","parameters":{"type":"object","properties":{"a":{"type":"number","description":"第一个数字"},"b":{"type":"number","description":"第二个数字"}},"required":["a","b"],"additionalProperties":false}}}],"should_call_tool":true}
{"messages":[{"role":"system","content":"你是助手"},{"role":"user","content":"今天天气不错，我们聊聊天"}],"tools":[{"type":"function","function":{"name":"add","description":"将两个数字相加","parameters":{"type":"object","properties":{"a":{"type":"number","description":"第一个数字"},"b":{"type":"number","description":"第二个数字"}},"required":["a","b"],"additionalProperties":false}}}],"should_call_tool":false}
{"messages":[{"role":"system","content":"你是助手"},{"role":"user","content":"把 37 摄氏度转换为华氏度"}],"tools":[{"type":"function","function":{"name":"convert_temperature","description":"将摄氏度转换为华氏度","parameters":{"type":"object","properties":{"celsius":{"type":"number","description":"摄氏温度值"}},"required":["celsius"],"additionalProperties":false}}}],"should_call_tool":true}

3
evalscope/custom_eval/text/mcq/example_dev.csv Normal file
View File

@ -0,0 +1,3 @@
id,question,A,B,C,D,answer
1,通常来说，组成动物蛋白质的氨基酸有____,4种,22种,20种,19种,C
2,血液内存在的下列物质中，不属于代谢终产物的是____。,尿素,尿酸,丙酮酸,二氧化碳,C
1 id question A B C D answer
2 1 通常来说，组成动物蛋白质的氨基酸有____ 4种 22种 20种 19种 C
3 2 血液内存在的下列物质中，不属于代谢终产物的是____。 尿素 尿酸 丙酮酸 二氧化碳 C

2
evalscope/custom_eval/text/mcq/example_dev.jsonl Normal file
View File

@ -0,0 +1,2 @@
{"id": "1", "question": "通常来说，组成动物蛋白质的氨基酸有____", "A": "4种", "B": "22种", "C": "20种", "D": "19种", "answer": "C"}
{"id": "2", "question": "血液内存在的下列物质中，不属于代谢终产物的是____。", "A": "尿素", "B": "尿酸", "C": "丙酮酸", "D": "二氧化碳", "answer": "C"}

2
evalscope/custom_eval/text/mcq/example_multi_dev.jsonl Normal file
View File

@ -0,0 +1,2 @@
{"id": "d1", "category": "示例", "question": "下列哪些是水果？", "A": "苹果", "B": "萝卜", "C": "香蕉", "D": "胡萝卜", "answer": ["A", "C"]}
{"id": "d2", "category": "示例", "question": "下列哪些是偶数？", "A": "2", "B": "3", "C": "4", "D": "5", "answer": ["A", "C"]}

10
evalscope/custom_eval/text/mcq/example_multi_val.jsonl Normal file
View File

@ -0,0 +1,10 @@
{"id": "1", "category": "生物学", "question": "下列属于哺乳动物的是？", "A": "鲸", "B": "蛇", "C": "蝙蝠", "D": "鲨鱼", "answer": ["A", "C"]}
{"id": "2", "category": "地理", "question": "下列哪些国家是欧盟成员国？", "A": "法国", "B": "瑞士", "C": "德国", "D": "挪威", "answer": ["A", "C"]}
{"id": "3", "category": "数学", "question": "下列哪些数是质数？", "A": "2", "B": "4", "C": "7", "D": "11", "answer": ["A", "C", "D"]}
{"id": "4", "category": "物理学", "question": "下列哪些是矢量？", "A": "速度", "B": "质量", "C": "加速度", "D": "温度", "answer": ["A", "C"]}
{"id": "5", "category": "化学", "question": "下列哪些是碱金属元素？", "A": "锂", "B": "镁", "C": "钠", "D": "钾", "answer": ["A", "C", "D"]}
{"id": "6", "category": "文学", "question": "下列作品出自鲁迅之手的有？", "A": "《狂人日记》", "B": "《骆驼祥子》", "C": "《阿Q正传》", "D": "《家》", "answer": ["A", "C"]}
{"id": "7", "category": "计算机科学", "question": "下列哪些属于面向对象的特性？", "A": "封装", "B": "递归", "C": "继承", "D": "多态", "answer": ["A", "C", "D"]}
{"id": "8", "category": "计算机科学", "question": "下列哪些是 Python 的内置可变类型？", "A": "list", "B": "tuple", "C": "dict", "D": "str", "answer": ["A", "C"]}
{"id": "9", "category": "历史", "question": "下列哪些属于中国四大发明？", "A": "造纸术", "B": "算盘", "C": "印刷术", "D": "火药", "answer": ["A", "C", "D"]}
{"id": "10", "category": "艺术", "question": "下列哪些是文艺复兴时期的画家？", "A": "达·芬奇", "B": "梵高", "C": "米开朗基罗", "D": "毕加索", "answer": ["A", "C"]}

13
evalscope/custom_eval/text/mcq/example_val.csv Normal file
View File

@ -0,0 +1,13 @@
id,category,question,A,B,C,D,answer
1,数学,以下哪个公式是二次函数的标准形式？,y = mx + b,y = ax^2 + bx + c,y = x^2 + 2x + 1,y = a + bx,B
2,物理学,空气中的声速是多少？,343 m/s,792 km/s,225 km/h,60 m/s,A
3,化学,水的化学式是什么？,H2O,CO2,NaCl,O2,A
4,生物学,遗传物质的载体是？,DNA,RNA,蛋白质,脂质,A
5,地理,世界上最大的沙漠是？,撒哈拉沙漠,戈壁滩,南极大陆,阿塔卡马沙漠,C
6,历史,谁是中国历史上建立秦朝的皇帝？,秦始皇,汉高祖,唐太宗,宋太祖,A
7,文学,《红楼梦》的作者是谁？,曹雪芹,鲁迅,巴金,老舍,A
8,艺术,《蒙娜丽莎》的画家是谁？,达·芬奇,米开朗基罗,拉斐尔,梵高,A
9,计算机科学,以下哪个是C语言的关键词？,for,print,echo,def,A
10,经济学,需求法则描述的是？,价格和需求量的反向关系,价格和供给量的正向关系,供求平衡,市场垄断,A
11,生物学,通常来说，组成动物蛋白质的氨基酸有____,4种,22种,20种,19种,C
12,生物学,血液内存在的下列物质中，不属于代谢终产物的是____。,尿素,尿酸,丙酮酸,二氧化碳,C
1 id category question A B C D answer
2 1 数学 以下哪个公式是二次函数的标准形式？ y = mx + b y = ax^2 + bx + c y = x^2 + 2x + 1 y = a + bx B
3 2 物理学 空气中的声速是多少？ 343 m/s 792 km/s 225 km/h 60 m/s A
4 3 化学 水的化学式是什么？ H2O CO2 NaCl O2 A
5 4 生物学 遗传物质的载体是？ DNA RNA 蛋白质 脂质 A
6 5 地理 世界上最大的沙漠是？ 撒哈拉沙漠 戈壁滩 南极大陆 阿塔卡马沙漠 C
7 6 历史 谁是中国历史上建立秦朝的皇帝？ 秦始皇 汉高祖 唐太宗 宋太祖 A
8 7 文学 《红楼梦》的作者是谁？ 曹雪芹 鲁迅 巴金 老舍 A
9 8 艺术 《蒙娜丽莎》的画家是谁？ 达·芬奇 米开朗基罗 拉斐尔 梵高 A
10 9 计算机科学 以下哪个是C语言的关键词？ for print echo def A
11 10 经济学 需求法则描述的是？ 价格和需求量的反向关系 价格和供给量的正向关系 供求平衡 市场垄断 A
12 11 生物学 通常来说，组成动物蛋白质的氨基酸有____ 4种 22种 20种 19种 C
13 12 生物学 血液内存在的下列物质中，不属于代谢终产物的是____。 尿素 尿酸 丙酮酸 二氧化碳 C

12
evalscope/custom_eval/text/mcq/example_val.jsonl Normal file
View File

@ -0,0 +1,12 @@
{"id": "1", "category": "数学", "question": "以下哪个公式是二次函数的标准形式？", "A": "y = mx + b", "B": "y = ax^2 + bx + c", "C": "y = x^2 + 2x + 1", "D": "y = a + bx", "answer": "B"}
{"id": "2", "category": "物理学", "question": "空气中的声速是多少？", "A": "343 m/s", "B": "792 km/s", "C": "225 km/h", "D": "60 m/s", "answer": "A"}
{"id": "3", "category": "化学", "question": "水的化学式是什么？", "A": "H2O", "B": "CO2", "C": "NaCl", "D": "O2", "answer": "A"}
{"id": "4", "category": "生物学", "question": "遗传物质的载体是？", "A": "DNA", "B": "RNA", "C": "蛋白质", "D": "脂质", "answer": "A"}
{"id": "5", "category": "地理", "question": "世界上最大的沙漠是？", "A": "撒哈拉沙漠", "B": "戈壁滩", "C": "南极大陆", "D": "阿塔卡马沙漠", "answer": "C"}
{"id": "6", "category": "历史", "question": "谁是中国历史上建立秦朝的皇帝？", "A": "秦始皇", "B": "汉高祖", "C": "唐太宗", "D": "宋太祖", "answer": "A"}
{"id": "7", "category": "文学", "question": "《红楼梦》的作者是谁？", "A": "曹雪芹", "B": "鲁迅", "C": "巴金", "D": "老舍", "answer": "A"}
{"id": "8", "category": "艺术", "question": "《蒙娜丽莎》的画家是谁？", "A": "达·芬奇", "B": "米开朗基罗", "C": "拉斐尔", "D": "梵高", "answer": "A"}
{"id": "9", "category": "计算机科学", "question": "以下哪个是C语言的关键词？", "A": "for", "B": "print", "C": "echo", "D": "def", "answer": "A"}
{"id": "10", "category": "经济学", "question": "需求法则描述的是？", "A": "价格和需求量的反向关系", "B": "价格和供给量的正向关系", "C": "供求平衡", "D": "市场垄断", "answer": "A"}
{"id": "11", "category": "生物学", "question": "通常来说，组成动物蛋白质的氨基酸有____", "A": "4种", "B": "22种", "C": "20种", "D": "19种", "answer": "C"}
{"id": "12", "category": "生物学", "question": "血液内存在的下列物质中，不属于代谢终产物的是____。", "A": "尿素", "B": "尿酸", "C": "丙酮酸", "D": "二氧化碳", "answer": "C"}

10
evalscope/custom_eval/text/qa/arena.jsonl Normal file
View File

@ -0,0 +1,10 @@
{"query": "How can I improve my time management skills?", "category": "generic"}
{"query": "What are the most effective ways to deal with stress?", "category": "generic"}
{"query": "What are the main differences between Python and JavaScript programming languages?", "category": "generic"}
{"query": "How can I increase my productivity while working from home?", "category": "generic"}
{"query": "Can you explain the basics of quantum computing?", "category": "generic"}
{"query": "What are the differences between plant-based and animal-based protein sources?", "category": "generic"}
{"query": "How can I develop my critical thinking skills?", "category": "generic"}
{"query": "What are the major challenges faced by the education sector today?", "category": "generic"}
{"query": "What are the primary factors that influence consumer behavior?", "category": "generic"}
{"query": "What are the most effective strategies for conflict resolution in the workplace?", "category": "generic"}

13
evalscope/custom_eval/text/qa/example.jsonl Normal file
View File

@ -0,0 +1,13 @@
{"system": "你是一位天文学家", "query": "太阳系中最大的行星是哪颗？", "response": "是木星"}
{"query": "什么动物被称为沙漠之舟？", "response": "是骆驼"}
{"messages": [{"role": "system", "content": "你是一位地理学家"}, {"role": "user", "content": "世界上最大的沙漠是哪个？"}], "response": "是撒哈拉沙漠"}
{"query": "世界上最长的河流是哪条？", "response": "是尼罗河"}
{"query": "地球上最大的洲是哪个？", "response": "是亚洲"}
{"query": "什么是世界上最繁忙的航空港？", "response": "亚特兰大机场"}
{"query": "世界上最古老的七大奇迹是哪个？", "response": "金字塔"}
{"query": "什么国家是世界上最大的生产者？", "response": "中国"}
{"query": "世界上最大的淡水湖是哪个？", "response": "是苏必利尔湖"}
{"query": "太阳系中离太阳最近的行星是哪颗？", "response": "是水星"}
{"query": "中国的首都是哪里？", "response": "中国的首都是北京"}
{"query": "世界上最高的山是哪座山？", "response": "是珠穆朗玛峰"}
{"query": "为什么北极见不到企鹅？", "response": "因为企鹅大多生活在南极"}

6
evalscope/custom_eval/text/qa/test.jsonl Normal file
View File

@ -0,0 +1,6 @@
{"query": "世界上最高的山是哪座山？", "response": "是珠穆朗玛峰"}
{"query": "为什么北极见不到企鹅？", "response": "因为企鹅大多生活在南极"}
{"query": "太阳系中最大的行星是哪颗？", "response": "是木星"}
{"query": "地球上最长的河流是哪条？", "response": "是尼罗河"}
{"query": "谁发明了相对论？", "response": "是爱因斯坦"}
{"query": "谁是中国的第一位皇帝？", "response": "是秦始皇"}

10
evalscope/custom_eval/text/rerank/example.jsonl Normal file
View File

@ -0,0 +1,10 @@
{"query": "What is machine learning?", "documents": ["Machine learning is a subset of artificial intelligence that enables systems to learn from data.", "ML algorithms can identify patterns and make predictions without explicit programming."]}
{"query": "How do neural networks work?", "documents": ["Neural networks are computing systems inspired by biological neural networks in animal brains.", "They consist of interconnected nodes that process information through weighted connections and activation functions."]}
{"query": "What is the difference between supervised and unsupervised learning?", "documents": ["Supervised learning uses labeled data to train models, while unsupervised learning finds patterns in unlabeled data.", "Common supervised tasks include classification and regression, while clustering is a typical unsupervised task."]}
{"query": "What is deep learning?", "documents": ["Deep learning is a subset of machine learning that uses neural networks with multiple layers.", "Deep learning has revolutionized fields like computer vision, natural language processing, and speech recognition."]}
{"query": "How does natural language processing work?", "documents": ["NLP enables computers to understand, interpret, and generate human language.", "Modern NLP systems use transformer models and attention mechanisms to process text data."]}
{"query": "What is a convolutional neural network?", "documents": ["CNNs are specialized neural networks designed for processing grid-like data such as images.", "They use convolutional layers to automatically learn spatial hierarchies of features."]}
{"query": "What is reinforcement learning?", "documents": ["Reinforcement learning is a type of ML where agents learn to make decisions by receiving rewards or penalties.", "It's commonly used in robotics, game playing, and autonomous systems."]}
{"query": "What is overfitting in machine learning?", "documents": ["Overfitting occurs when a model learns the training data too well, including noise and outliers.", "Techniques like regularization, cross-validation, and dropout help prevent overfitting."]}
{"query": "What is a transformer model?", "documents": ["Transformers are neural network architectures based on self-attention mechanisms.", "They have become the foundation for modern language models like GPT and BERT."]}
{"query": "How does gradient descent work?", "documents": ["Gradient descent is an optimization algorithm used to minimize loss functions in machine learning.", "It iteratively adjusts model parameters in the direction of steepest descent of the loss function."]}

10
evalscope/custom_eval/text/retrieval/corpus.jsonl Normal file
View File

@ -0,0 +1,10 @@
{"_id": "doc1", "text": "气候变化正在导致更极端的天气模式。"}
{"_id": "doc2", "text": "今天股市大幅上涨，科技股领涨。"}
{"_id": "doc3", "text": "人工智能正在通过自动化任务和提供见解来改变各种行业。"}
{"_id": "doc4", "text": "随着技术的进步，风能和太阳能等可再生能源变得越来越普及。"}
{"_id": "doc5", "text": "最新研究表明，均衡饮食和定期锻炼可以显著改善心理健康。"}
{"_id": "doc6", "text": "虚拟现实正在教育、娱乐和培训方面创造新的机会。"}
{"_id": "doc7", "text": "由于环保优势和电池技术的进步，电动汽车越来越受欢迎。"}
{"_id": "doc8", "text": "太空探索任务正在揭示关于我们的太阳系及其以外的新信息。"}
{"_id": "doc9", "text": "区块链技术在加密货币之外还有潜在的应用，包括供应链管理和安全投票系统。"}
{"_id": "doc10", "text": "远程工作的好处包括更大的灵活性和减少通勤时间。"}

10
evalscope/custom_eval/text/retrieval/qrels.jsonl Normal file
View File

@ -0,0 +1,10 @@
{"query-id": "query1", "corpus-id": "doc1", "score": 1}
{"query-id": "query2", "corpus-id": "doc2", "score": 1}
{"query-id": "query3", "corpus-id": "doc3", "score": 1}
{"query-id": "query4", "corpus-id": "doc4", "score": 1}
{"query-id": "query5", "corpus-id": "doc5", "score": 1}
{"query-id": "query6", "corpus-id": "doc6", "score": 1}
{"query-id": "query7", "corpus-id": "doc7", "score": 1}
{"query-id": "query8", "corpus-id": "doc8", "score": 1}
{"query-id": "query9", "corpus-id": "doc9", "score": 1}
{"query-id": "query10", "corpus-id": "doc10", "score": 1}

10
evalscope/custom_eval/text/retrieval/queries.jsonl Normal file
View File

@ -0,0 +1,10 @@
{"_id": "query1", "text": "气候变化的影响是什么？"}
{"_id": "query2", "text": "今天股市上涨的原因是什么？"}
{"_id": "query3", "text": "人工智能如何改变行业？"}
{"_id": "query4", "text": "可再生能源有哪些进展？"}
{"_id": "query5", "text": "均衡饮食如何改善心理健康？"}
{"_id": "query6", "text": "虚拟现实创造了哪些新机会？"}
{"_id": "query7", "text": "为什么电动汽车越来越受欢迎？"}
{"_id": "query8", "text": "太空探索任务揭示了哪些新信息？"}
{"_id": "query9", "text": "区块链技术在加密货币之外有哪些应用？"}
{"_id": "query10", "text": "远程工作的好处是什么？"}

BIN
evalscope/docs/asset/dingding.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 458 KiB

BIN
evalscope/docs/asset/discord_qr.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 168 KiB

28
evalscope/docs/en/.readthedocs.yaml Normal file
View File

@ -0,0 +1,28 @@
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the OS, Python version and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.12"
# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/en/conf.py
# Optionally build your docs in additional formats such as PDF and ePub
# formats:
# - pdf
# - epub
# Optional but recommended, declare the Python requirements required
# to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: requirements/docs.txt

20
evalscope/docs/en/Makefile Normal file
View File

@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

BIN
evalscope/docs/en/_static/images/evalscope_icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

1
evalscope/docs/en/_static/images/evalscope_icon.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 6.6 KiB

BIN
evalscope/docs/en/_static/images/evalscope_icon_dark.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 23 KiB

BIN
evalscope/docs/en/_static/images/evalscope_logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 280 KiB

532
evalscope/docs/en/advanced_guides/add_benchmark.md Normal file
View File

@ -0,0 +1,532 @@
# 👍 Contribute Benchmark
EvalScope, as the official evaluation tool of [ModelScope](https://modelscope.cn), is continuously optimizing its benchmark evaluation features! We invite you to refer to this tutorial to easily add your own benchmark and share your contributions with the community. Let's work together to enhance EvalScope and make our tool even better!
Below, we will introduce how to add two types of benchmark evaluations: **General Text Reasoning** and **Multiple Choice**, which mainly include three steps: uploading the dataset, registering the dataset, and writing the evaluation task.
## Basic Concepts
```{tip}
You can skip this section and start directly from [Preparing Benchmark Evaluation Dataset](#1-preparing-benchmark-evaluation-dataset). Refer back to the specific implementation when encountering code you don't understand.
```
The evaluation process of EvalScope mainly includes the following steps:
1. **Data Preparation**: Load and preprocess the dataset through `DataAdapter`.
2. **Task Definition**: Define the configuration of the evaluation task through `TaskConfig`, including models, datasets, evaluation metrics, etc.
3. **Evaluation Execution**: Execute the evaluation task through the `run_task` function and output the evaluation results.
Among them, `DataAdapter` is the core component of benchmark evaluation that we need to focus on.
### DataAdapter Architecture and Call Flow
DataAdapter adopts a Pipeline architecture, supporting custom behavior through hook methods. Taking `DefaultDataAdapter` as an example, the complete evaluation process is as follows:
```
1. Data Loading Phase
load_dataset()
├── load()
│ ├── load_from_remote() / load_from_disk()
│ │ ├── load_subsets()
│ │ │ └── load_subset() / load_fewshot_subset()
│ │ │ └── record_to_sample() [User Implementation]
│ │ └── _post_process_samples()
│ │ └── process_sample_input()
│ │ ├── sample_to_fewshot() [User Implementation]
│ │ ├── format_fewshot_template() [Optional User Implementation]
│ │ └── format_prompt_template() [Optional User Implementation]
│ └── Returns DatasetDict
2. Model Inference Phase (Per Sample)
run_inference()
├── _on_inference_start() [Hook Method]
├── _on_inference() [Hook Method]
└── _on_inference_end() [Hook Method]
└── Returns TaskState
3. Metric Calculation Phase (Per Sample)
calculate_metrics()
├── filter_prediction()
│ └── extract_answer() [Optional User Implementation]
├── match_score() / llm_match_score()
└── Returns SampleScore
4. Result Aggregation Phase
aggregate_scores()
└── Returns List[AggScore]
5. Report Generation Phase
generate_report()
├── _on_generate_report() [Hook Method]
└── _on_generate_report_end() [Hook Method]
└── Returns Report
```
### Core Data Structures
#### 1. Sample Object
Represents a single evaluation sample, including input, target answer, and metadata:
```python
@dataclass
class Sample:
input: Any # Input content (question text or list of chat messages)
target: str # Target answer (correct answer)
choices: Optional[List[str]] = None # Choices (used for multiple choice questions)
subset_key: Optional[str] = None # Subset division key (used for grouping by category)
metadata: Optional[Dict] = None # Metadata (reasoning process, ID, etc.)
tools: Optional[List] = None # Tool call information
```
#### 2. TaskState Object
Represents the complete state of a single inference task:
```python
@dataclass
class TaskState:
model: str # Model name
sample: Sample # Input sample
messages: List[ChatMessage] # Chat message history
output: ModelOutput # Model raw output
completed: bool # Whether the task is completed
sample_id: Optional[str] = None # Sample ID
group_id: Optional[str] = None # Group ID
metadata: Optional[Dict] = None # Task metadata
```
#### 3. ModelOutput Object
Represents the raw output of the model:
```python
@dataclass
class ModelOutput:
completion: str # Text generated by the model
message: ChatMessage # Formatted chat message
# Other model-specific fields...
```
#### 4. Score Object
Represents the scoring result of a single sample:
```python
@dataclass
class Score:
value: Dict[str, float] # Scores for each metric {"acc": 1.0, "f1": 0.8}
extracted_prediction: str # Extracted prediction answer
prediction: str # Raw prediction text
metadata: Dict = None # Scoring metadata
```
#### 5. SampleScore Object
Encapsulates the complete scoring information of a single sample:
```python
@dataclass
class SampleScore:
score: Score # Scoring object
sample_id: Optional[str] # Unique identifier for the sample
group_id: Optional[str] # Group identifier
sample_metadata: Optional[Dict] = None # Sample metadata
```
#### 6. AggScore Object
Represents aggregated scoring statistics:
```python
@dataclass
class AggScore:
metric: str # Metric name
value: float # Aggregated value (e.g., average score)
subset: str # Subset name
num_samples: int # Number of samples
agg_method: str # Aggregation method (mean, median, etc.)
metadata: Dict = None # Aggregation metadata
```
#### 7. DatasetDict Object
Manages multiple dataset subsets:
```python
class DatasetDict(dict):
"""Dataset dictionary, keys are subset names, values are Dataset objects"""
@classmethod
def from_dataset(cls, dataset, subset_list=None, limit=None, repeats=1):
"""Create a multi-subset dataset dictionary from a single dataset"""
pass
```
### Core Methods of DataAdapter
Based on the above call flow, here are the key methods that need to be implemented by the user or can be optionally overridden:
#### Methods That Must Be Implemented
1. **`record_to_sample(record: Dict[str, Any]) -> Sample`**
- **Purpose**: Convert raw data records into standard Sample objects
- **Input**: Raw record dictionary from the dataset
- **Output**: Standardized Sample object
- **Example**:
```python
def record_to_sample(self, record: Dict[str, Any]) -> Sample:
return Sample(
input=record['question'],
target=record['answer'],
metadata={'reasoning': record.get('explanation', '')}
)
```
#### Methods That Can Be Optionally Implemented
2. **`sample_to_fewshot(sample: Sample) -> str`**
- **Purpose**: Convert sample into a few-shot example string
- **Input**: Sample object
- **Output**: Formatted few-shot example text
- **Call Timing**: When constructing few-shot prompts
3. **`extract_answer(prediction: str, task_state: TaskState) -> str`**
- **Purpose**: Extract the final answer from the model's raw output
- **Input**: Model prediction text and task state
- **Output**: Extracted answer string
- **Call Timing**: Before calculating metrics for answer cleaning
4. **`format_prompt_template(sample: Sample) -> str`**
- **Purpose**: Format the basic prompt template
- **Input**: Sample object
- **Output**: Formatted prompt text
- **Default Implementation**: Uses `prompt_template.format(question=sample.input)`
5. **`format_fewshot_template(fewshot: str, sample: Sample) -> str`**
- **Purpose**: Format the prompt template containing few-shot examples
- **Input**: Few-shot example string and Sample object
- **Output**: Complete few-shot prompt
- **Default Implementation**: Uses `few_shot_prompt_template.format()`
6. **`sample_filter(sample: Sample) -> bool`**
- **Purpose**: Filter dataset samples
- **Input**: Sample object
- **Output**: Whether to retain the sample
- **Default Implementation**: Returns True (retains all samples)
### Hook Method System
DataAdapter provides a hook method system, supporting custom logic insertion at key points:
#### Inference Phase Hooks
- **`_on_inference_start(model, sample)`**: Before inference starts
- **`_on_inference(model, sample)`**: Execute inference
- **`_on_inference_end(model, sample, model_output, output_dir)`**: After inference ends
#### Report Generation Hooks
- **`_on_generate_report(scores, model_name)`**: Generate report
- **`_on_generate_report_end(report, output_dir)`**: After report generation
### Adapter Types
EvalScope provides two main adapter base classes:
1. **`DefaultDataAdapter`**: Basic adapter for general text reasoning tasks
- Suitable for open-ended question answering, mathematical reasoning, code generation, etc.
- Requires custom answer extraction logic
2. **`MultiChoiceAdapter`**: Specialized adapter for multiple choice tasks
- Inherits from `DefaultDataAdapter`
- Built-in choice formatting and answer extraction logic
- Supports single-choice and multiple-choice modes
Principles for choosing adapter types:
- If the task involves selecting answers from fixed options → Use `MultiChoiceAdapter`
- If the task requires generating open-ended answers → Use `DefaultDataAdapter`
## 1. Preparing Benchmark Evaluation Dataset
You have two ways to prepare the benchmark evaluation dataset:
1. **Upload to ModelScope (Recommended)**: Upload the dataset to the ModelScope platform, so other users can easily load your dataset, making it more convenient to use and benefiting more users from your contribution. If you need to upload to ModelScope, refer to the [Dataset Upload Tutorial](https://www.modelscope.cn/docs/datasets/create).
2. **Local Use**: You can also directly use the local dataset for evaluation, suitable for datasets that are still in development or contain sensitive information.
Regardless of the method chosen, ensure that the data format is correct and can be loaded. If using a local dataset, you can test with the following code:
```python
from modelscope import MsDataset
dataset = MsDataset.load("/path/to/your/dataset") # Replace with your dataset
```
## 2. Creating File Structure
First, [Fork EvalScope](https://github.com/modelscope/evalscope/fork) repository, i.e., create your own EvalScope repository copy, and clone it locally.
```bash
git clone https://github.com/your_username/evalscope.git
cd evalscope
```
Then, add benchmark evaluation in the `evalscope/benchmarks/` directory, with the structure as follows:
```text
evalscope/benchmarks/
├── benchmark_name
│ ├── __init__.py
│ ├── benchmark_name_adapter.py
│ └── ...
```
Specifically for `GSM8K` and `MMLU-Pro`, the structure is as follows:
```text
evalscope/benchmarks/
├── gsm8k
│ ├── __init__.py
│ ├── gsm8k_adapter.py
├── mmlu_pro
│ ├── __init__.py
│ ├── mmlu_pro_adapter.py
│ └── ...
```
## 3. Writing Evaluation Logic
Below, we will take **GSM8K** and **MMLU-Pro** as examples to introduce two types of evaluation tasks: **General Text Reasoning** and **Multiple Choice**.
### General Text Reasoning
General text reasoning tasks usually require the model to analyze and reason about the given problem and then generate an answer. Taking GSM8K (mathematical reasoning) as an example:
We need to register `Benchmark` and implement the `GSM8KAdapter` class in `gsm8k_adapter.py`:
```python
from typing import Any, Dict
from evalscope.api.benchmark import BenchmarkMeta, DefaultDataAdapter
from evalscope.api.dataset import Sample
from evalscope.api.evaluator import TaskState
from evalscope.api.registry import register_benchmark
from evalscope.constants import Tags
# Define prompt template
PROMPT_TEMPLATE = """
Solve the following math problem step by step. The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem.
{question}
Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \\boxed command.
Reasoning:
""".lstrip()
# Register benchmark evaluation
@register_benchmark(
BenchmarkMeta(
name='gsm8k', # Benchmark test name
pretty_name='GSM8K', # Readable name
dataset_id='AI-ModelScope/gsm8k', # Dataset ID or local path
tags=[Tags.MATH, Tags.REASONING], # Tags
description='GSM8K (Grade School Math 8K) is a dataset of grade school math problems, designed to evaluate the mathematical reasoning abilities of AI models.',
subset_list=['main'], # Subset list
few_shot_num=4, # Few-shot example number
train_split='train', # Training set split name
eval_split='test', # Evaluation set split name
metric_list=['acc'], # Evaluation metrics
prompt_template=PROMPT_TEMPLATE, # Prompt template
)
)
class GSM8KAdapter(DefaultDataAdapter):
def record_to_sample(self, record: Dict[str, Any]) -> Sample:
"""Convert raw data records into Sample objects"""
DELIM = '####'
question = record['question']
answer = record['answer'].split(DELIM)
target = answer.pop().strip() # Extract final answer
reasoning = DELIM.join(answer) # Extract reasoning process
return Sample(
input=question,
target=target,
metadata={'reasoning': reasoning.strip()}
)
def sample_to_fewshot(self, sample: Sample) -> str:
"""Convert sample into few-shot example"""
if sample.metadata:
return (
f'{sample.input}\n\nReasoning:\n' +
f"{sample.metadata['reasoning']}\n\n" +
f'ANSWER: {sample.target}'
)
else:
return ''
def extract_answer(self, prediction: str, task_state: TaskState):
"""Extract answer from model prediction"""
from evalscope.filters.extraction import RegexFilter
# Use regular expression to extract numeric answer
regex = RegexFilter(regex_pattern=r'(-?[0-9.,]{2,})|(-?[0-9]+)', group_select=-1)
res = regex(prediction)
return res.replace(',', '').replace('+', '').strip().strip('.')
```
### Multiple Choice
Multiple choice tasks require the model to select the correct answer from given options. Taking MMLU-Pro as an example, we need to inherit `MultiChoiceAdapter`:
```python
from typing import Any, Dict
from evalscope.api.benchmark import BenchmarkMeta, MultiChoiceAdapter
from evalscope.api.dataset import Sample
from evalscope.api.registry import register_benchmark
from evalscope.constants import Tags
# Define prompt template
USER_PROMPT_TEMPLATE = """Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Think step by step before answering.
Question:
{question}
Options:
{choices}
""".lstrip()
SUBSET_LIST = [
'computer science', 'math', 'chemistry', 'engineering', 'law', 'biology',
'health', 'physics', 'business', 'philosophy', 'economics', 'other',
'psychology', 'history'
]
@register_benchmark(
BenchmarkMeta(
name='mmlu_pro',
pretty_name='MMLU-Pro',
tags=[Tags.MULTIPLE_CHOICE, Tags.KNOWLEDGE],
description='MMLU-Pro is a benchmark for evaluating language models on multiple-choice questions across various subjects.',
dataset_id='modelscope/MMLU-Pro',
subset_list=SUBSET_LIST,
metric_list=['acc'],
few_shot_num=5,
train_split='validation',
eval_split='test',
prompt_template=USER_PROMPT_TEMPLATE,
)
)
class MMLUProAdapter(MultiChoiceAdapter):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.reformat_subset = True # Enable subset division
def record_to_sample(self, record: Dict[str, Any]) -> Sample:
"""Convert raw data records into Sample objects"""
return Sample(
input=record['question'],
choices=record['options'], # Choice list
target=record['answer'], # Correct answer (e.g., 'A')
subset_key=record['category'].lower(), # Key for subset division
metadata={
'cot_content': record['cot_content'],
'subject': record['category'].lower(),
'question_id': record['question_id'],
},
)
def sample_to_fewshot(self, sample: Sample) -> str:
"""Convert sample into few-shot example"""
q_str = f"""Question:\n{str(sample.input)}"""
options = sample.choices if sample.choices is not None else []
# Format choices
opt_str_list = []
for i, opt in enumerate(options):
opt_str_list.append(f"""{chr(65 + i)} {opt}""")
opt_str = f"""Options:\n{'\n'.join(opt_str_list)}"""
# Handle answer and reasoning process
ans_str = sample.metadata['cot_content'] if sample.metadata is not None else ''
ans_str = ans_str.replace('The answer is', 'ANSWER:')
ans_opt = ans_str.split('ANSWER:')[-1].split('.')[0].strip().strip('(').strip(')')
ans_str = ans_str.replace(f'ANSWER: ({ans_opt})', f'ANSWER: {ans_opt}')
final_str = '\n'.join([q_str, opt_str, ans_str])
return final_str
```
### Key Differences Explanation
**General Text Reasoning** vs **Multiple Choice**:
1. **Inherited Base Class**:
- General Text Reasoning: Inherits `DefaultDataAdapter`
- Multiple Choice: Inherits `MultiChoiceAdapter`
2. **Sample Object Structure**:
- General Text Reasoning: Mainly includes `input` and `target`
- Multiple Choice: Additionally includes `choices` (choice list)
3. **Answer Extraction Method**:
- General Text Reasoning: Requires custom `extract_answer()` method
- Multiple Choice: `MultiChoiceAdapter` provides standard answer extraction logic
4. **Prompt Template**:
- General Text Reasoning: Focuses more on guiding the reasoning process
- Multiple Choice: Focuses on displaying choices and answer format
## 4. Running Evaluation
Debug the code to see if it can run normally.
**GSM8K Example**:
```python
from evalscope import run_task, TaskConfig
task_cfg = TaskConfig(
model='Qwen/Qwen2.5-0.5B-Instruct',
datasets=['gsm8k'],
limit=10,
debug=True
)
run_task(task_cfg=task_cfg)
```
**MMLU-Pro Example**:
```python
from evalscope import run_task, TaskConfig
task_cfg = TaskConfig(
model='Qwen/Qwen2.5-0.5B-Instruct',
datasets=['mmlu_pro'],
limit=10,
dataset_args={'mmlu_pro': {'subset_list': ['computer science', 'math']}},
debug=True
)
run_task(task_cfg=task_cfg)
```
Output Example:
```text
+-----------------------+-----------+-----------------+------------------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=======================+===========+=================+==================+=======+=========+=========+
| Qwen2.5-0.5B-Instruct | gsm8k | mean_acc | main | 10 | 0.3 | default |
+-----------------------+-----------+-----------------+------------------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | mmlu_pro | mean_acc | computer science | 10 | 0.1 | default |
+-----------------------+-----------+-----------------+------------------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | mmlu_pro | mean_acc | math | 10 | 0.1 | default |
+-----------------------+-----------+-----------------+------------------+-------+---------+---------+
```
## 5. Benchmark Evaluation Document Generation
After completing the benchmark evaluation implementation, you can use the tools provided by EvalScope to generate standard documents. This will ensure that your benchmark evaluation has a consistent document format and can be easily understood and used by other users.
To generate Chinese and English documents, run the following command, which will generate documents based on registration information:
```bash
pip install -e '.[docs]'
make docs
```
## 6. Submitting PR
After completing the implementation of these methods and document generation, your benchmark evaluation is ready! You can submit a [PR](https://github.com/modelscope/evalscope/pulls). Before submitting, please run the following command, which will automatically format the code:
```bash
make lint
```
Ensure there are no formatting issues, and we will merge your contribution as soon as possible, allowing more users to use the benchmark evaluation you contributed. If you don't know how to submit a PR, you can check our [Guide](https://github.com/modelscope/evalscope/blob/main/CONTRIBUTING.md). Give it a try 🚀

114
evalscope/docs/en/advanced_guides/collection/evaluate.md Normal file
View File

@ -0,0 +1,114 @@
# Unified Evaluation with Your Index
In a nutshell: Use the mixed JSONL (generated from the sampling phase) as a single "dataset" to run evaluation, obtaining multi-level capability scores and weighted index total score in one go.
## Evaluation Configuration
::::{tab-set}
:::{tab-item} Using `TaskConfig` Object
```python
from evalscope import TaskConfig, run_task
import os
task_cfg = TaskConfig(
model='qwen2.5-7b-instruct',
api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
api_key=os.getenv('DASHSCOPE_API_KEY'),
eval_type='openai_api',
datasets=['data_collection'],
dataset_args={
'data_collection': {
'local_path': 'outputs/stratified_mixed_data.jsonl',
'shuffle': True
}
},
eval_batch_size=5,
generation_config={
'temperature': 0.0
}
)
run_task(task_cfg=task_cfg)
```
:::
:::{tab-item} Using CLI Command
```bash
evalscope eval \
--model qwen2.5-7b-instruct \
--eval-type openai_api \
--api-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
--api-key "$DASHSCOPE_API_KEY" \
--datasets data_collection \
--dataset-args '{"data_collection":{"local_path":"outputs/stratified_mixed_data.jsonl","shuffle":true}}' \
--eval-batch-size 5 \
--generation-config '{"temperature":0.0}'
```
:::
::::
## Usage Conventions
- `datasets` uses `data_collection` as the fixed mixed entry point.
- `dataset_args.local_path` points to the mixed JSONL; optional `shuffle` controls sample traversal order.
- Output structure: `outputs/<timestamp>/{logs,predictions,reviews,reports,configs}`.
- Inference concurrency: `eval_batch_size`.
## Metrics & Output
The system outputs reports from five "observation perspectives":
1. **subset_level**: Subsets within original datasets (e.g., ARC-Challenge / ARC-Easy).
2. **dataset_level**: Aggregation of all subsets from the same dataset (e.g., arc).
3. **task_level**: Same task type (e.g., reasoning).
4. **tag_level**: Sample tags dimension (language, mode, etc.; Schema names can be merged into tags).
5. **category_level**: Custom grouping (e.g., reasoning_index).
**Metric Descriptions**:
- **micro_avg.**: Sum across samples then calculate overall accuracy (total correct / total samples). Subsets with more samples have greater influence.
- **macro_avg.**: Simple average of scores for "members" (subsets) at this level, each member weighted equally regardless of sample count.
- **weighted_avg.**: Weighted by defined leaf weights $w_i$ (weights are defined before sampling; sampling process doesn't change weight meaning).
**Example Output**:
```text
2025-11-24 18:44:50 - evalscope - data_collection_adapter.py - generate_report - 171 - INFO: subset_level Report:
+-----------+--------------+---------------+------------+-------+
| task_type | dataset_name | subset_name | micro_avg. | count |
+-----------+--------------+---------------+------------+-------+
| reasoning | arc | ARC-Challenge | 0.8 | 5 |
| reasoning | arc | ARC-Easy | 1.0 | 4 |
| reasoning | ceval | logic | 0.0 | 1 |
+-----------+--------------+---------------+------------+-------+
2025-11-24 18:44:50 - evalscope - data_collection_adapter.py - generate_report - 171 - INFO: dataset_level Report:
+-----------+--------------+------------+------------+---------------+-------+
| task_type | dataset_name | micro_avg. | macro_avg. | weighted_avg. | count |
+-----------+--------------+------------+------------+---------------+-------+
| reasoning | arc | 0.8889 | 0.9 | 0.8889 | 9 |
| reasoning | ceval | 0.0 | 0.0 | 0.0 | 1 |
+-----------+--------------+------------+------------+---------------+-------+
2025-11-24 18:44:50 - evalscope - data_collection_adapter.py - generate_report - 171 - INFO: task_level Report:
+-----------+------------+------------+---------------+-------+
| task_type | micro_avg. | macro_avg. | weighted_avg. | count |
+-----------+------------+------------+---------------+-------+
| reasoning | 0.8 | 0.6 | 0.3556 | 10 |
+-----------+------------+------------+---------------+-------+
2025-11-24 18:44:50 - evalscope - data_collection_adapter.py - generate_report - 171 - INFO: tag_level Report:
+------+------------+------------+---------------+-------+
| tags | micro_avg. | macro_avg. | weighted_avg. | count |
+------+------------+------------+---------------+-------+
| en | 0.8889 | 0.9 | 0.8889 | 9 |
| zh | 0.0 | 0.0 | 0.0 | 1 |
+------+------------+------------+---------------+-------+
2025-11-24 18:44:50 - evalscope - data_collection_adapter.py - generate_report - 171 - INFO: category_level Report:
+-----------------+------------+------------+---------------+-------+
| category0 | micro_avg. | macro_avg. | weighted_avg. | count |
+-----------------+------------+------------+---------------+-------+
| reasoning_index | 0.8 | 0.6 | 0.3556 | 10 |
+-----------------+------------+------------+---------------+-------+
```
## Common Issues
- **Index score doesn't match expectations**: Expand Schema to verify normalized weights; confirm you're not misunderstanding weight effects with uniform sampling.
- **Changes only require re-running**: If JSONL content unchanged, enable `--use-cache` to save inference time.
- **Slow evaluation with large datasets**: Increase `eval_batch_size`; ensure model/API supports concurrent requests.
## Next Steps
- **Visualization**: `evalscope service` opens an interactive interface to filter dimensions and models.
- **Versioning**: Generate multiple JSONLs for different weight configurations, named `weighted_v1.jsonl` / `weighted_v2.jsonl` and evaluate separately.

117
evalscope/docs/en/advanced_guides/collection/index.md Normal file
View File

@ -0,0 +1,117 @@
# Building an Evaluation Index
Through **Collection**, you can define your own **Evaluation Index** in EvalScope:
Combine multiple datasets according to business value proportions, run evaluation once, and get a comprehensive score that aligns with your real-world scenarios.
---
## When Do You Need a Custom Index?
Many teams only look at single benchmarks (e.g., MMLU scores) when selecting models, but often encounter several typical issues:
- **A single dataset doesn't equal real-world scenarios**
Your actual scenarios might be:
- RAG Q&A (knowledge base retrieval + answering)
- Vertical domain analysis like finance/legal
- Code completion and code review
These are often difficult to fully cover with any single public dataset.
- **Public leaderboards don't equal your best model**
Leaderboards focus on "general capabilities," while you care more about "business value":
- Which model is better suited for your RAG workflow?
- Which model has the lowest error rate in domains you care about?
- Which model offers the best balance between cost and performance?
- **You need an index that "reflects your own values"**
Just like:
- [Artificial Analysis Intelligence Index](https://artificialanalysis.ai/methodology/intelligence-benchmarking#intelligence-index-evaluation-suite-summary) focuses on general intelligence capabilities
- [Vals Index](https://www.vals.ai/benchmarks/vals_index) focuses on business value
You can also define an **Index that serves only your business decisions**.
EvalScope's **Collection** is designed for this purpose:
You can "package" multiple datasets, set weights based on business importance, and create an evaluation index that truly belongs to you.
---
## Core Mental Model: Collection = Weighted Multi-Dataset Evaluation
You can think of Collection as:
> A **configuration file** describing "what I care about and their respective weights"
> → Sample a mixed dataset
> → Run evaluation once with EvalScope
> → Get a "comprehensive model score sheet" weighted by your values.
From a user's perspective, you only need to focus on three things:
1. Which **datasets** do I want to evaluate?
2. How much **weight** does each dataset have in my decision-making?
3. How do I want to **sample** from these datasets (how many samples, what distribution)?
---
## From 0 to 1: How to Build an Index Using Collection?
Below is a minimum viable workflow to help you get started quickly.
### 1. Define Schema: Declare "What I Care About"
In the **Schema**, you need to do two things:
- Select the **list of datasets** to include in the Index
For example:
- `gsm8k`: Basic math reasoning
- An internally constructed RAG Q&A dataset
- A code completion dataset
- Set **weights** for each dataset
Weights can reflect your emphasis on different capability dimensions:
- RAG scenarios: 50%
- Financial Q&A: 30%
- General reasoning: 20%
**For more details, see:** [Defining Your Schema](schema.md).
---
### 2. Sample Data: Extract Representative Samples from Multiple Datasets
Based on the Schema, EvalScope can generate a **mixed dataset JSONL** for you.
You can control the sampling method through several common strategies:
- **Weight-based sampling**: Data volume roughly allocated by weight proportion;
- **Stratified sampling**: Different difficulty levels/subtasks are all covered;
- **Uniform sampling**: Fixed number of samples from each dataset for easy comparison.
The final result is a "pre-mixed" JSONL file, and subsequent evaluation only needs to run once on this JSONL.
**Sampling methods and command examples:** See [How to Sample](sample.md).
---
### 3. Unified Evaluation: Get Your Custom Index Score in One Run
After obtaining the mixed JSONL, you can:
- Call `evalscope eval` just like evaluating a regular dataset
- Get all at once:
- Scores for each sub-dataset
- **Total Index score** aggregated by weights
- Corresponding detailed logs, predictions, reviews, and reports
After evaluation, you can use:
- Report summaries to compare different models' performance under the same Index;
- EvalScope's **visualization app (`evalscope service`)** to visually compare model strengths and weaknesses.
**Specific evaluation commands and sample outputs:** See [How to Evaluate](evaluate.md).
:::{toctree}
:maxdepth: 1
schema.md
sample.md
evaluate.md
:::

120
evalscope/docs/en/advanced_guides/collection/sample.md Normal file
View File

@ -0,0 +1,120 @@
# Sampling Your Index Data
In a nutshell: Based on the defined Collection Schema, sample multiple datasets using specified strategies to generate a mixed JSONL for unified one-time evaluation.
## Output
- A single JSONL file (mixed samples) for subsequent `evalscope eval`.
- Each line represents a standardized sample.
## Field Description
| Field | Description |
| ---- | ---- |
| index | Sample sequence number (renumbered in the mixed set) |
| prompt | Original input structure (may include question / context / code, etc.) |
| tags | Merged tags (dataset's own + upper hierarchy) |
| task_type | Task type or capability classification |
| weight | Leaf node normalized weight (for interpretation, not equal to actual sample repetition weight) |
| dataset_name | Original dataset name |
| subset_name | Subset or difficulty label (can be empty) |
Example:
```json
{
"index": 0,
"prompt": {"question": "What is the capital of France?"},
"tags": ["en", "reasoning"],
"task_type": "question_answering",
"weight": 1.0,
"dataset_name": "arc",
"subset_name": "ARC-Easy"
}
```
## Three Sampling Strategies (Formulas & Use Cases)
Given:
- Total sample size: $N$
- Number of datasets: $K$
- Schema weights: $w_i$
- Original dataset sample size: $m_i$
### 1. Weighted Sampling
Formula (expected sample count):
$$
n_i \approx N \cdot \frac{w_i}{\sum_{j=1}^{K} w_j}
$$
**Use case**: You've expressed business priorities through weights and want "high-weight capabilities" to occupy more resources.
**Characteristics**: Reinforces values; if a dataset has much higher weight, it will dominate proportionally.
### 2. Stratified Sampling
Formula:
$$
n_i \approx N \cdot \frac{m_i}{\sum_{j=1}^{K} m_j}
$$
**Use case**: Preserve original distribution (e.g., let datasets with larger corpus sizes be more representative).
**Characteristics**: Doesn't reflect business bias; avoids over-amplifying small datasets. Guarantees at least ≥1 sample per dataset.
### 3. Uniform Sampling
Formula:
$$
n_i \approx \frac{N}{K}
$$
**Use case**: Want each capability to have "equal voice," facilitating model comparison across capabilities.
**Characteristics**: Ignores original size and weights; makes results easier for comparing individual capability weaknesses.
**Strategy Selection Recommendation**:
- For "business decision index": Weighted
- For "objective coverage / preserving corpus structure": Stratified
- For "capability alignment / diagnosis": Uniform
## Prerequisite Schema Example
```python
from evalscope.collections import CollectionSchema, DatasetInfo
schema = CollectionSchema(
name='reasoning_index',
datasets=[
DatasetInfo(name='arc', weight=2.0, task_type='reasoning', tags=['en']),
DatasetInfo(name='ceval', weight=3.0, task_type='reasoning', tags=['zh'], args={'subset_list': ['logic']}),
],
)
```
### Weighted Sampling (Emphasizing Business Importance)
```python
from evalscope.collections import WeightedSampler
from evalscope.utils.io_utils import dump_jsonl_data
sampler = WeightedSampler(schema)
mixed_data = sampler.sample(10) # N=10
dump_jsonl_data(mixed_data, 'outputs/weighted_mixed_data.jsonl')
```
Expected: arc : ceval ≈ 2 : 3 → 4 : 6
### Stratified Sampling (Preserving Source Data Size Distribution)
```python
from evalscope.collections import StratifiedSampler
from evalscope.utils.io_utils import dump_jsonl_data
sampler = StratifiedSampler(schema)
mixed_data = sampler.sample(10)
dump_jsonl_data(mixed_data, 'outputs/stratified_mixed_data.jsonl')
```
Expected: Approximately allocated by original sample sizes (example: if arc=2000, ceval=10 → roughly 9 : 1)
### Uniform Sampling (Aligning Capability Comparison)
```python
from evalscope.collections import UniformSampler
from evalscope.utils.io_utils import dump_jsonl_data
sampler = UniformSampler(schema)
mixed_data = sampler.sample(10)
dump_jsonl_data(mixed_data, 'outputs/uniform_mixed_data.jsonl')
```
Expected: arc : ceval = 5 : 5
## Common Issues
- Weights don't affect Uniform / Stratified allocation (only Weighted uses them).
- Dataset with too few samples: Stratified sampling will force at least 1 sample; increase N or switch to Weighted.
- The `weight` field in JSONL is the leaf normalized weight, not equal to the sample's appearance probability (especially for Stratified and Uniform strategies).

134
evalscope/docs/en/advanced_guides/collection/schema.md Normal file
View File

@ -0,0 +1,134 @@
# Defining Your Schema
In a nutshell: Schema = your capability checklist + relative value (weight) for each capability + optional hierarchical grouping.
## Core Concepts & Objects
- **CollectionSchema**: A nestable grouping node that forms hierarchical structures and reusable modules.
- **DatasetInfo**: The final evaluable dataset entry (leaf node), containing name, weight, and adapter parameters.
- **flatten()**: Expands the entire tree to get a normalized list of leaf datasets (with hierarchical path `hierarchy`).
## Field Overview
**CollectionSchema**:
- `name`: str (required) - Group/index name
- `weight`: float (optional, default 1.0) - Relative weight for the entire subgroup (combined with child nodes, then normalized)
- `datasets`: list[DatasetInfo | CollectionSchema]
**DatasetInfo**:
- `name`: str (required) - Dataset registration name
- `weight`: float (required, > 0)
- `task_type`: str (optional) - Custom capability label
- `tags`: list[str] (optional) - Custom tags; parent CollectionSchema names are automatically appended
- `args`: dict (optional) - Pass-through to dataset adapter (e.g., subset_list, local_path, review_timeout, etc.)
## Why Set Weights?
**Sampling Phase**:
- Under weighted sampling strategy, sample proportion ∝ normalized weight.
- Normalization for flat (non-nested) structure:
$$
\mathbf{p_i} = \frac{\mathbf{w_i}}{\sum_{j=1}^{n} \mathbf{w_j}}
$$
- For multi-level nesting (product of weights along the leaf path, then overall normalization):
$$
\mathbf{p_i} = \frac{\prod_{k \in \text{path}(i)} \mathbf{w_k}}{\sum_{j=1}^{n} \prod_{k \in \text{path}(j)} \mathbf{w_k}}
$$
**Scoring Phase**:
- Aggregation uses linear weighting:
$$
\mathbf{S} = \sum_{i=1}^{n} \boldsymbol{\alpha_i} \, \mathbf{s_i}
$$
where:
$$
\boldsymbol{\alpha_i} = \mathbf{p_i}, \quad \sum_{i=1}^{n} \boldsymbol{\alpha_i} = 1
$$
## Minimal Python Example
```python
from evalscope.collections import CollectionSchema, DatasetInfo
simple_schema = CollectionSchema(
name='reasoning_index',
datasets=[
DatasetInfo(name='arc', weight=2.0, task_type='reasoning', tags=['en']),
DatasetInfo(name='ceval', weight=3.0, task_type='reasoning', tags=['zh'], args={'subset_list': ['logic']}),
],
)
```
## Nested & Multi-Capability Example
```python
from evalscope.collections import CollectionSchema, DatasetInfo
complex_schema = CollectionSchema(name='math_index', datasets=[
CollectionSchema(name='math', weight=3.0, datasets=[
DatasetInfo(name='gsm8k', weight=1.0, task_type='math', tags=['en']),
DatasetInfo(name='aime25', weight=1.0, task_type='math', tags=['en']),
]),
CollectionSchema(name='reasoning', weight=1.0, datasets=[
DatasetInfo(name='arc', weight=1.0, task_type='reasoning', tags=['en']),
DatasetInfo(name='ceval', weight=1.0, task_type='reasoning', tags=['zh'], args={'subset_list': ['logic']}),
]),
])
```
**Hierarchical Effects**:
- Parent names are appended to each leaf's tags/hierarchy, supporting multi-level aggregated analysis.
- Parent weight scales the entire group, then jointly normalizes with child nodes.
## Flatten / Serialize / Restore
```python
print(simple_schema.flatten()) # Flatten + normalize
print(complex_schema.flatten()) # View normalized weights after nesting
complex_schema.dump_json('outputs/complex_schema.json')
loaded = CollectionSchema.from_json('outputs/complex_schema.json')
```
**Output Example (automatically normalized weights)**:
```python
>>> [DatasetInfo(name='arc', weight=0.4, task_type='reasoning', tags=['en'], args={}, hierarchy=['reasoning_index']),
DatasetInfo(name='ceval', weight=0.6, task_type='reasoning', tags=['zh'], args={'subset_list': ['logic']}, hierarchy=['reasoning_index'])]
>>> [DatasetInfo(name='gsm8k', weight=0.375, task_type='math', tags=['en'], args={}, hierarchy=['math_index', 'math']),
DatasetInfo(name='aime25', weight=0.375, task_type='math', tags=['en'], args={}, hierarchy=['math_index', 'math']),
DatasetInfo(name='arc', weight=0.125, task_type='reasoning', tags=['en'], args={}, hierarchy=['math_index', 'reasoning']),
DatasetInfo(name='ceval', weight=0.125, task_type='reasoning', tags=['zh'], args={'subset_list': ['logic']}, hierarchy=['math_index', 'reasoning'])]
```
## Common Issues
- Weights must be > 0; directly remove entries with weight = 0.
- Deep nesting can easily cause some leaf weights to be "diluted"; verify manually by flattening first.
- Too many tags affect downstream visualization dimensions; keep them concise (e.g., en / zh / math / rag).
- `args` must match corresponding dataset adapter fields; incorrect fields will be ignored or trigger warnings.
- Cramming too many capabilities into one Schema hinders iteration; split into multiple versions for comparison.
## Reference Cases
### Case A: Enterprise RAG Assistant Index
- **Pain Point**: High scores on general leaderboards, but inaccurate internal knowledge retrieval.
- **Approach**: Knowledge tasks + long-context retrieval + instruction following.
- **Configuration Example**:
```python
from evalscope.collections import CollectionSchema, DatasetInfo
schema = CollectionSchema(name='rag_index', datasets=[
DatasetInfo(name='chinese_simpleqa', weight=0.3, task_type='knowledge', tags=['rag']),
DatasetInfo(name='aa_lcr', weight=0.3, task_type='long_context', tags=['rag']),
DatasetInfo(name='ifeval', weight=0.4, task_type='instruction_following', tags=['rag']),
])
```
### Case B: Code Completion & Analysis Index
- **Pain Point**: General leaderboards cannot comprehensively reflect code understanding and generation capabilities.
- **Approach**: Code generation + real issues.
- **Configuration Example**:
```python
from evalscope.collections import CollectionSchema, DatasetInfo
schema = CollectionSchema(name='code_index', datasets=[
DatasetInfo(name='humaneval', weight=2.0, task_type='code', tags=['python']),
DatasetInfo(name='live_code_bench', weight=1.0, task_type='code', tags=['zh', 'finance'], args={'subset_list': ['v5'], 'review_timeout': 6, 'extra_params': {'start_date': '2024-08-01','end_date': '2025-02-28'},}),
])
```

195
evalscope/docs/en/advanced_guides/custom_dataset/clip.md Normal file
View File

@ -0,0 +1,195 @@
# CLIP Model
## Custom Image-Text Retrieval Dataset
### 1. Prepare the Dataset
Prepare the `image_queries.jsonl` dataset for image-text retrieval in the following format (file name must be fixed):
```{code-block} json
:caption: custom_eval/multimodal/text-image-retrieval/image_queries.jsonl
{"image_path": "custom_eval/multimodal/images/dog.jpg", "query": ["dog"]}
{"image_path": "custom_eval/multimodal/images/AMNH.jpg", "query": ["building"]}
{"image_path": "custom_eval/multimodal/images/tokyo.jpg", "query": ["city", "tokyo"]}
{"image_path": "custom_eval/multimodal/images/tesla.jpg", "query": ["car", "tesla"]}
{"image_path": "custom_eval/multimodal/images/running.jpg", "query": ["man", "running"]}
```
Where:
- `image_path`: Path to the image, supporting local paths.
- `query`: Text descriptions for image-text retrieval, supporting multiple descriptions, such as `["dog", "cat"]`.
### 2. Configure Evaluation Parameters
```python
task_cfg = {
"eval_backend": "RAGEval",
"eval_config": {
"tool": "clip_benchmark",
"eval": {
"models": [
{
"model_name": "AI-ModelScope/chinese-clip-vit-large-patch14-336px",
}
],
"dataset_name": ["custom"],
"data_dir": "custom_eval/multimodal/text-image-retrieval",
"split": "test",
"batch_size": 128,
"num_workers": 1,
"verbose": True,
"skip_existing": False,
"limit": 1000,
},
},
}
```
```{seealso}
[Full Parameter Explanation](../../user_guides/backend/rageval_backend/clip_benchmark.md#configure-evaluation-parameters)
```
Where:
- `dataset_name`: Dataset name, must be specified as `custom`.
- `data_dir`: Dataset directory, containing the `image_queries.jsonl` file.
### 3. Run Evaluation Task
```python
from evalscope.run import run_task
run_task(task_cfg=task_cfg)
```
The evaluation output is as follows:
```json
{"dataset": "custom", "model": "AI-ModelScope/chinese-clip-vit-large-patch14-336px", "task": "zeroshot_retrieval", "metrics": {"image_retrieval_recall@5": 1.0, "text_retrieval_recall@5": 1.0}}
```
## Convert Image-Text Retrieval Data to Text Retrieval Data
To facilitate the evaluation of different multimodal retrieval methods, this framework supports converting image-text retrieval problems into text retrieval problems using a multimodal large model, followed by text retrieval evaluation.
### 1. Prepare the Dataset
Supported input datasets include [image-text retrieval datasets](../../user_guides/backend/rageval_backend/clip_benchmark.md#supported-datasets) and the custom image-text retrieval dataset mentioned above.
### 2. Configure Evaluation Parameters
```python
task_cfg = {
"eval_backend": "RAGEval",
"eval_config": {
"tool": "clip_benchmark",
"eval": {
"models": [
{
"model_name": "internvl2-8b",
"api_base": "http://localhost:8008/v1",
"api_key": "xxx",
"prompt": "用中文描述这张图片",
}
],
"dataset_name": ["muge"],
"split": "test",
"task": "image_caption",
"batch_size": 2,
"num_workers": 1,
"verbose": True,
"skip_existing": False,
"limit": 10,
},
},
}
```
Parameter Explanation:
- The `models` list must include a multimodal large model configuration:
- `model_name`: Name of the multimodal large model, e.g., `internvl2-8b`.
- `api_base`: API address of the multimodal large model, e.g., `http://localhost:8008/v1`.
- `api_key`: API key for the multimodal large model, e.g., `xxx`.
- `prompt`: Prompt for the multimodal large model input, e.g., `"用中文描述这张图片"`.
- `task`: Evaluation task, must be specified as `image_caption`.
### 3. Run the Conversion Task
Run the following code to start the conversion:
```python
from evalscope.run import run_task
run_task(task_cfg=task_cfg)
```
The output is as follows:
```
2024-10-22 19:56:09,832 - evalscope - INFO - Write files to outputs/internvl2-8b/muge/retrieval_data
2024-10-22 19:56:10,543 - evalscope - INFO - Evaluation results: {'dataset': 'muge', 'model': 'internvl2-8b', 'task': 'image_caption', 'metrics': {'convertion_successful': True, 'save_path': 'outputs/internvl2-8b/muge/retrieval_data'}}
2024-10-22 19:56:10,544 - evalscope - INFO - Dump results to: outputs/internvl2-8b/muge_image_caption.json
```
The output file directory structure is as follows:
```
muge
├── retrieval_data
│ ├── corpus.jsonl
│ ├── queries.jsonl
│ └── qrels
│ └── test.tsv
└── muge_image_caption.json
```
The specific contents of the files are as follows:
```{code-block} json
:caption: outputs/internvl2-8b/muge/retrieval_data/corpus.jsonl
{"_id":0,"text":"这是一张展示澳亚奇品牌的产品广告图片，图片中包含了六罐澳亚奇品牌的饮料，饮料罐上印有品牌的名称和图案。饮料罐排列在纸箱上，纸箱上也有品牌名称和图案。整个包装以红色和黄色为主基调，给人以醒目和吸引人的感觉。"}
{"_id":1,"text":"这是一副时尚的眼镜，镜框是金属材质的，颜色为玫瑰金色，镜腿部分是黑色的。镜腿的内侧有品牌标志，看起来像是“The Row”。这款眼镜的设计比较现代，适合日常佩戴。"}
{"_id":2,"text":"这张图片展示了一位女性，她正在用手机拍摄自己的侧脸自拍。她有长长的棕色头发，并佩戴着一对精美的耳环。耳环的设计有点像是字母“A”。背景是室内环境，可以看到淡蓝色墙壁和浅色的柜子。"}
{"_id":3,"text":"这是一张黑色塑料瓶的图片，瓶身上贴有红色标签，标签上有白色和黄色的文字。标签上内容包括产品名称、品牌和一些图案。瓶口是红色和灰色的盖子。"}
{"_id":4,"text":"这是一张客厅的照片，里面有一把单人沙发椅。沙发的靠背和坐垫上有黑白相间的斑马纹图案，椅子的框架是黑色的木制结构，带有卷曲的扶手。沙发的腿部是黑色的，造型优雅。沙发布置在一个铺有地毯的地板上，背景中可以看到部分沙发和装饰画，整个房间的装饰风格显得温馨且现代。"}
{"_id":5,"text":"这是一张一次性纸杯的图片。纸杯呈圆柱形，杯壁较为光滑，没有明显的装饰或花纹。杯口部分略微向外扩展，便于抓握。杯子整体呈浅灰色或乳白色，质地看起来较为轻薄。这种纸杯常用于盛装饮料或冷食，适合一次性使用。"}
{"_id":6,"text":"这张图片展示的是四个卡通人物，背景有五彩斑斓的光芒。从左到右，这四个角色分别是：\n\n1. 一个穿着蓝色服装、戴着紫色头巾和发饰的角色。\n2. 一个穿着蓝绿色服装、戴着蓝色发饰和翅膀的角色。\n3. 一个穿着粉红色服装、带着红色头饰和翅膀的角色。\n4. 一个穿着红色和白色服装、戴着红色头饰的角色。\n\n背景中，有“新格林童话”和“NEW GREEN”的字样。"}
{"_id":7,"text":"这是一张展示手中握着蓝色葡萄的照片。手的主人穿着绿色的毛衣，手指修长。葡萄颜色深蓝，表面光滑，每颗葡萄看起来都十分饱满多汁。旁边有一些绿色叶子和干燥的枝条做装饰。背景是一张木质的桌子，整体画面给人一种自然清新的感觉。"}
{"_id":8,"text":"这张图片展示了一个可爱的小马克杯，杯身是浅绿色，配有圆弧形的手柄。杯子上绘有可爱的卡通图案，包括一只戴着耳机的小兔子，并配有“热爱学习”字样，旁边还有两只小耳朵和几颗星星。整个马克杯的设计简洁可爱，适合用作日常饮品盛器。"}
{"_id":9,"text":"这是一张展示塑料包装中大量线状物体的图片。这些线状物体堆叠在一起，看起来像是一些纤维或麻线，可能是用于编织或加工的。"}
```
```{code-block} json
:caption: outputs/internvl2-8b/muge/retrieval_data/queries.jsonl
{"_id":0,"text":"酸角汁饮料 整箱 云南"}
{"_id":1,"text":"达芬奇眼镜"}
{"_id":2,"text":"水钻蝴蝶结耳钉"}
{"_id":3,"text":"邓州黄酒"}
{"_id":4,"text":"斑马纹老虎椅"}
{"_id":5,"text":"布丁杯模具"}
{"_id":6,"text":"光之美少女盒蛋"}
{"_id":7,"text":"蓝莓模型"}
{"_id":8,"text":"少女心喝水杯"}
{"_id":9,"text":"炸面"}
```
```{code-block}
:caption: outputs/internvl2-8b/muge/retrieval_data/qrels/test.tsv
query-id corpus-id score
0 0 1
1 1 1
2 2 1
3 3 1
4 4 1
5 5 1
6 6 1
7 7 1
8 8 1
9 9 1
```
### 4. Execute Text Retrieval Task
Once the dataset is ready, you can perform text retrieval tasks as per the CMTEB tutorial.
```{seealso}
Refer to [Custom Text Retrieval Evaluation](./embedding.md)
```

182
evalscope/docs/en/advanced_guides/custom_dataset/embedding.md Normal file
View File

@ -0,0 +1,182 @@
# Custom Text Retrieval Evaluation Dataset
## Overview
When you need to evaluate the retrieval capabilities of Embedding models on domain-specific data (e.g., legal, medical, financial) or private datasets, you can build a custom dataset for evaluation. EvalScope, built on the MTEB framework, supports defining corpus, queries, and relevance judgments through simple JSONL files for end-to-end retrieval evaluation.
## Step 1: Prepare Data
### Directory Structure
Create a directory containing the following three JSONL files:
```
my_retrieval_data/
├── corpus.jsonl
├── queries.jsonl
└── qrels.jsonl
```
### corpus.jsonl
The corpus file, with one JSON object per line representing a document to be retrieved.
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `_id` | str | Yes | Unique document identifier |
| `text` | str | Yes | Document body content |
| `title` | str | No | Document title, used to assist retrieval |
```json
{"_id": "doc1", "text": "Climate change is leading to more extreme weather patterns.", "title": "Climate Change"}
{"_id": "doc2", "text": "The stock market surged today, led by tech stocks."}
{"_id": "doc3", "text": "Artificial intelligence is transforming industries by automating tasks and providing insights."}
```
### queries.jsonl
The queries file, with one JSON object per line representing a retrieval query.
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `_id` | str | Yes | Unique query identifier |
| `text` | str | Yes | Query text |
```json
{"_id": "query1", "text": "What are the impacts of climate change?"}
{"_id": "query2", "text": "What caused the stock market to rise today?"}
{"_id": "query3", "text": "How is artificial intelligence changing industries?"}
```
### qrels.jsonl
The relevance judgments file, with one JSON object per line defining the relevance between a query and a document.
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `query-id` | str | Yes | Corresponds to `_id` in queries.jsonl |
| `corpus-id` | str | Yes | Corresponds to `_id` in corpus.jsonl |
| `score` | int/float | Yes | Relevance score |
```json
{"query-id": "query1", "corpus-id": "doc1", "score": 1}
{"query-id": "query2", "corpus-id": "doc2", "score": 1}
{"query-id": "query3", "corpus-id": "doc3", "score": 1}
```
**Score semantics:**
- **Binary (0/1)**: `1` means relevant, `0` means not relevant. Suitable for most retrieval scenarios.
- **Graded relevance**: Use multi-level scores (e.g., 0, 1, 2, 3), where higher values indicate greater relevance. Suitable for fine-grained evaluation that distinguishes "partially relevant" from "highly relevant".
### Data Quality Recommendations
```{tip}
- The corpus should contain at least **100** documents and at least **50** queries for statistically meaningful evaluation results.
- Ensure each query has at least **1** relevant document annotation.
- When using binary labels, consider also annotating negative samples (score=0) to more accurately assess the model's discrimination ability.
- Include enough irrelevant "distractor" documents in the corpus to simulate real retrieval scenarios.
```
## Step 2: Configure and Run Evaluation
### Complete Configuration Example
```python
from evalscope.run import run_task
task_cfg = {
"work_dir": "outputs",
"eval_backend": "RAGEval",
"eval_config": {
"tool": "MTEB",
"models": [
{
"model_name_or_path": "AI-ModelScope/m3e-base",
"pooling_mode": None,
"max_seq_length": 512,
"prompt": "",
"model_kwargs": {"torch_dtype": "auto"},
"encode_kwargs": {
"batch_size": 128,
},
}
],
"eval": {
"custom_tasks": [
{
"name": "CustomRetrieval",
"data_path": "my_retrieval_data",
}
],
"overwrite_results": True,
"limits": 500,
},
},
}
run_task(task_cfg=task_cfg)
```
### Run Command
Save the above code as `run_eval.py`, then execute:
```bash
python run_eval.py
```
### Interpreting Results
After evaluation completes, results are saved in the `outputs/` directory. Key metrics include:
- **NDCG@k**: Normalized Discounted Cumulative Gain, measuring ranking quality (k=1,3,5,10)
- **MAP@k**: Mean Average Precision
- **Recall@k**: Recall rate
- **Precision@k**: Precision rate
Scores range from 0 to 1, with higher values indicating better retrieval performance.
## Parameter Reference
Each task in the `custom_tasks` list corresponds to a `CustomTaskConfig` with the following fields:
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `name` | str | `"CustomRetrieval"` | Task name, used to identify evaluation results |
| `data_path` | str | (required) | Dataset directory path, must contain `corpus.jsonl`, `queries.jsonl`, `qrels.jsonl` |
| `eval_splits` | List[str] | `["test"]` | Evaluation splits list |
```{note}
Other evaluation parameters (such as `models`, `limits`, `overwrite_results`, etc.) are consistent with the default configuration. See [MTEB Evaluation Parameter Reference](../../user_guides/backend/rageval_backend/mteb.md#parameter-explanation) for details.
```
## FAQ
### Field Format Errors
- Data files must be in **JSONL** format (one independent JSON object per line), not standard JSON array format.
- The `_id` field value must be a **string** type. Even numeric IDs should be written as `"123"` rather than `123`.
- Field names in `qrels.jsonl` use hyphens: `query-id` and `corpus-id`, not underscores.
### Impact of Insufficient Data
- When the corpus has too few documents (e.g., fewer than 10), metrics like Recall@10 may always be 1.0, making it impossible to effectively differentiate between models.
- Too few queries lead to high variance in evaluation results, lacking statistical significance.
- It is recommended to prepare at least hundreds of samples based on your actual business scenario.
### Converting from Other Formats
If you already have data in TSV or CSV format, you can convert it to JSONL as follows:
```python
import csv
import json
# Example: TSV/CSV to corpus.jsonl
with open("corpus.tsv", "r") as fin, open("corpus.jsonl", "w") as fout:
reader = csv.DictReader(fin, delimiter="\t") # For CSV, use delimiter=","
for row in reader:
obj = {"_id": row["id"], "text": row["text"]}
fout.write(json.dumps(obj, ensure_ascii=False) + "\n")
```

12
evalscope/docs/en/advanced_guides/custom_dataset/index.md Normal file
View File

@ -0,0 +1,12 @@
# Custom Datasets
The following section introduces how to use EvalScope to customize evaluation datasets, including large model evaluation datasets, multimodal evaluation datasets, embedding models, and CLIP model evaluations.
:::{toctree}
:maxdepth: 2
llm.md
vlm.md
embedding.md
clip.md
:::

498
evalscope/docs/en/advanced_guides/custom_dataset/llm.md Normal file
View File

@ -0,0 +1,498 @@
# Large Language Model
This framework supports multiple-choice questions, question-answering questions, and function calling, with three predefined dataset formats. The usage process is as follows:
## Multiple-Choice Question Format (MCQ)
Suitable for scenarios where users need multiple-choice questions. The evaluation metric is accuracy.
### Data Preparation
Prepare files in multiple-choice question format, supporting both CSV and JSONL formats. The directory structure is as follows:
**CSV Format**
```text
mcq/
├── example_dev.csv # (Optional) File name composed of `{subset_name}_dev.csv`, used for few-shot evaluation
└── example_val.csv # File name composed of `{subset_name}_val.csv`, used for actual evaluation data
```
CSV files should be in the following format:
```text
id,question,A,B,C,D,answer
1,Generally speaking, the amino acids that make up animal proteins are ____,4 types,22 types,20 types,19 types,C
2,Among the substances present in the blood, which one is not a metabolic end product?____,Urea,Uric acid,Pyruvic acid,Carbon dioxide,C
```
**JSONL Format**
```text
mcq/
├── example_dev.jsonl # (Optional) File name composed of `{subset_name}_dev.jsonl`, used for few-shot evaluation
└── example_val.jsonl # File name composed of `{subset_name}_val.jsonl`, used for actual evaluation data
```
JSONL files should be in the following format:
```json
{"id": "1", "question": "Generally speaking, the amino acids that make up animal proteins are ____", "A": "4 types", "B": "22 types", "C": "20 types", "D": "19 types", "answer": "C"}
{"id": "2", "question": "Among the substances present in the blood, which one is not a metabolic end product?____", "A": "Urea", "B": "Uric acid", "C": "Pyruvic acid", "D": "Carbon dioxide", "answer": "C"}
```
Where:
- `id` is the serial number (optional field)
- `question` is the query
- `A`, `B`, `C`, `D`, etc., are the options, supporting up to 10 choices
- `answer` is the correct option
### Configuration Task
````{note}
The default `prompt_template` supports four options, as shown below, where `{query}` is the placeholder for the prompt. If you need fewer or more options, you can customize the `prompt_template`.
```text
Please answer the question and select the correct answer from the options. The last line of your answer should be in the following format: "答案是：LETTER" (without quotes), where LETTER is one of A, B, C, or D.
{query}
```
````
Run the following code to start the evaluation:
```python
from evalscope import TaskConfig, run_task
task_cfg = TaskConfig(
model='Qwen/Qwen2-0.5B-Instruct',
datasets=['general_mcq'], # Data format: for multiple-choice questions, use 'general_mcq'
dataset_args={
'general_mcq': {
# 'prompt_template': 'xxx', # You can customize the prompt template if needed
"local_path": "custom_eval/text/mcq", # Path to your custom dataset
"subset_list": [
"example" # Name of the evaluation dataset (the subset_name above)
]
}
},
)
run_task(task_cfg=task_cfg)
```
Results:
```text
+---------------------+-------------+-----------------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+=====================+=============+=================+==========+=======+=========+=========+
| Qwen2-0.5B-Instruct | general_mcq | AverageAccuracy | example | 12 | 0.5833 | default |
+---------------------+-------------+-----------------+----------+-------+---------+---------+
```
### Multiple Correct Answers
When a question has more than one correct option, enable multiple-answer mode via `extra_params`. In this mode the `answer` field **must be a list of letters** (JSONL format only).
**Data format**
```text
mcq/
├── example_multi_dev.jsonl # (Optional) Few-shot samples, answer is also a list
└── example_multi_val.jsonl # Evaluation data
```
Example record:
```json
{"id": "1", "category": "Biology", "question": "Which of the following are mammals?", "A": "Whale", "B": "Snake", "C": "Bat", "D": "Shark", "answer": ["A", "C"]}
```
**extra_params**
| Param | Type | Default | Description |
|---|---|---|---|
| `multiple_correct` | bool | `False` | When `True`, switches to `CHINESE_MULTIPLE_ANSWER_TEMPLATE` and requires `answer` to be a list of letters |
| `use_cot` | bool | `False` | When `True`, uses the CoT variant (`*_COT`). Can be combined with `multiple_correct` |
**Configuration example**
```python
from evalscope import TaskConfig, run_task
task_cfg = TaskConfig(
model='Qwen/Qwen2-0.5B-Instruct',
datasets=['general_mcq'],
dataset_args={
'general_mcq': {
'local_path': 'custom_eval/text/mcq',
'subset_list': ['example_multi'],
'extra_params': {
'multiple_correct': True,
# 'use_cot': True, # Optional: enable CoT template
}
}
},
)
run_task(task_cfg=task_cfg)
```
```{note}
When `multiple_correct=True`, a `ValueError` is raised at data-loading time if the `answer` field is not a list (e.g., a string like `"AC"`), to make the format mismatch easy to diagnose.
```
## Question-Answering Format (QA)
This framework accommodates two formats for question-and-answer tasks: those with reference answers and those without.
1. **Reference Answer** Q&A: Suitable for questions with clear correct answers, with default evaluation metrics being `ROUGE` and `BLEU`. It can also be configured with an LLM judge for semantic correctness assessment.
2. **Reference-free Answer** Q&A: Suitable for questions without definitive correct answers, such as open-ended questions. By default, no evaluation metrics are provided, but an LLM judge can be configured to score the generated answers.
Here's how to use it:
### Data Preparation
Prepare a JSONL file in the Q&A format, for example, a directory containing a file:
```text
qa/
└── example.jsonl
```
This JSONL file contains one sample per line and supports the following three structures (please choose one and keep it consistent within the same file):
```json
{"system": "You are a geographer", "query": "What is the capital of China?", "response": "The capital of China is Beijing"}
{"query": "What is the highest mountain in the world?", "response": "It is Mount Everest"}
{"messages": [{"role": "system", "content": "You are a geographer"}, {"role": "user", "content": "What is the largest desert in the world?"}], "response": "It is the Sahara Desert"}
```
Field descriptions and required fields:
- Format 1 (system + query [+ response])
- query: Required
- response: Required when evaluating with reference answers; can be omitted or left empty when evaluating without reference answers
- system: Optional
- Format 2 (query [+ response])
- query: Required
- response: Required when evaluating with reference answers; can be omitted or left empty when evaluating without reference answers
- Format 3 (messages [+ response])
- messages: Required, array elements are {"role": "system"|"user"|"assistant", "content": "<text>"}; the last entry is recommended to be a user question
- response: Required when evaluating with reference answers; can be omitted or left empty when evaluating without reference answers
### Reference Answer Q&A
Below is how to configure the evaluation of reference answer Q&A tasks using the `Qwen2.5` model on `example.jsonl`.
**Method 1: Evaluation based on `ROUGE` and `BLEU`**
Simply run the following code:
```python
from evalscope import TaskConfig, run_task
task_cfg = TaskConfig(
model='Qwen/Qwen2.5-0.5B-Instruct',
datasets=['general_qa'], # Data format, fixed as 'general_qa' for Q&A tasks
dataset_args={
'general_qa': {
"local_path": "custom_eval/text/qa", # Custom dataset path
"subset_list": [
# Evaluation dataset name, the * in *.jsonl above, multiple subsets can be configured
"example"
]
}
},
)
run_task(task_cfg=task_cfg)
```
<details><summary>Click to view evaluation results</summary>
```text
+----------------+------------+-----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+================+============+===========+==========+=======+=========+=========+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-1-R | example | 12 | 0.694 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-1-P | example | 12 | 0.176 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-1-F | example | 12 | 0.2276 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-2-R | example | 12 | 0.4667 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-2-P | example | 12 | 0.0939 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-2-F | example | 12 | 0.1226 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-L-R | example | 12 | 0.6528 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-L-P | example | 12 | 0.1628 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | Rouge-L-F | example | 12 | 0.2063 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | bleu-1 | example | 12 | 0.164 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | bleu-2 | example | 12 | 0.0935 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | bleu-3 | example | 12 | 0.065 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
| Qwen2.5-0.5B-Instruct | general_qa | bleu-4 | example | 12 | 0.0556 | default |
+----------------+------------+-----------+----------+-------+---------+---------+
```
</details>
**Method 2: Evaluation based on LLM**
LLM-based evaluation can conveniently assess the correctness of model outputs (or other dimensions of metrics, requiring custom prompt settings). Below is an example configuring `judge_model_args` parameters, using the preset `pattern` mode to determine the correctness of model outputs.
For a complete explanation of judge parameters, please refer to [documentation](../../get_started/parameters.md#judge-parameters).
```python
import os
from evalscope import TaskConfig, run_task
from evalscope.constants import JudgeStrategy
task_cfg = TaskConfig(
model='Qwen/Qwen2.5-0.5B-Instruct',
datasets=[
'general_qa',
],
dataset_args={
'general_qa': {
'dataset_id': 'custom_eval/text/qa',
'subset_list': [
'example'
],
}
},
# judge related parameters
judge_model_args={
'model_id': 'qwen2.5-72b-instruct',
'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
'api_key': os.getenv('DASHSCOPE_API_KEY'),
'generation_config': {
'temperature': 0.0,
'max_tokens': 4096
},
# Determine if the model output is correct based on reference answers and model output
'score_type': 'pattern',
},
# eval concurrency number
eval_batch_size=5,
# Use LLM for evaluation
judge_strategy=JudgeStrategy.LLM,
)
run_task(task_cfg=task_cfg)
```
<details><summary>Click to view evaluation results</summary>
```text
+----------------+------------+----------------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+================+============+================+==========+=======+=========+=========+
| Qwen2.5-0.5B-Instruct | general_qa | AverageAccuracy | example | 12 | 0.583 | default |
+----------------+------------+----------------+----------+-------+---------+---------+
```
</details>
### Reference-free Answer Q&A
If the dataset lacks reference answers, an LLM judge can be used to evaluate the model's output answers. Without configuring an LLM, no scoring results will be available.
Below is an example configuring `judge_model_args` parameters, using the preset `numeric` mode to automatically assess model output scores from dimensions such as accuracy, relevance, and usefulness. Higher scores indicate better model output.
For a complete explanation of judge parameters, please refer to [documentation](../../get_started/parameters.md#judge-parameters).
```python
import os
from evalscope import TaskConfig, run_task
from evalscope.constants import JudgeStrategy
task_cfg = TaskConfig(
model='Qwen/Qwen2.5-0.5B-Instruct',
datasets=[
'general_qa',
],
dataset_args={
'general_qa': {
'dataset_id': 'custom_eval/text/qa',
'subset_list': [
'example'
],
}
},
# judge related parameters
judge_model_args={
'model_id': 'qwen2.5-72b-instruct',
'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
'api_key': os.getenv('DASHSCOPE_API_KEY'),
'generation_config': {
'temperature': 0.0,
'max_tokens': 4096
},
# Direct scoring
'score_type': 'numeric',
},
# eval concurrency number
eval_batch_size=5,
# Use LLM for evaluation
judge_strategy=JudgeStrategy.LLM,
)
run_task(task_cfg=task_cfg)
```
<details><summary>Click to view evaluation results</summary>
```text
+----------------+------------+----------------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+================+============+================+==========+=======+=========+=========+
| Qwen2.5-0.5B-Instruct | general_qa | AverageAccuracy | example | 12 | 0.6375 | default |
+----------------+------------+----------------+----------+-------+---------+---------+
```
</details>
## Function Calling Format (FC)
Applicable to evaluation scenarios that require function call decision-making and parameter structure validation. Supports:
- Preset dataset: evalscope/GeneralFunctionCall-Test
- Custom dataset: Pass in JSONL files via local path
The data is synthesized from official MoonshotAI [public samples](https://github.com/MoonshotAI/K2-Vendor-Verifier), including labels for "whether tool should be triggered" (K2-Thinking model as label source), facilitating experiment reproduction and automated regression testing.
### Data Format
Data is provided in JSONL format, with one sample per line, containing conversation context, tool definitions, and labels indicating whether tools should be triggered.
Example structure (one JSON object per line):
```json
{
"messages": [
{
"role": "system",
"content": "You are an assistant"
},
{
"role": "user",
"content": "Please add 2 and 3"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "add",
"description": "Add two numbers together",
"parameters": {
"type": "object",
"properties": {
"a": {
"type": "number",
"description": "The first number"
},
"b": {
"type": "number",
"description": "The second number"
}
},
"required": [
"a",
"b"
],
"additionalProperties": false
}
}
}
],
"should_call_tool": true
}
```
Field descriptions:
- messages: Evaluation input context (OpenAI-style role/content)
- tools: Available function list (OpenAI tool definition, `type=function, function={name, parameters(JSON Schema)}`)
- should_call_tool: Label indicating whether tool should be triggered (derived from official K2-thinking model results where `finish_reason == "tool_calls"`)
```{note}
- During evaluation, if the model returns `finish_reason == "tool_calls"`, it is considered as "model predicts tool call".
- Parameter validity is validated using JSON Schema (name must match, parameters must be parseable as JSON and satisfy the schema).
```
### Usage
```{note}
**Using model API services** is recommended for function calling evaluation, and ensure the model supports function calling. Local model inference does not currently support function calling.
```
Method 1: Using preset dataset ([evalscope/GeneralFunctionCall-Test](https://modelscope.cn/datasets/evalscope/GeneralFunctionCall-Test/dataPeview))
```python
from evalscope import TaskConfig, run_task
task_cfg = TaskConfig(
model='qwen-plus',
api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
api_key=env.get('DASHSCOPE_API_KEY'),
datasets=['general_fc'], # Function calling format is fixed as 'general_fc'
# To explicitly specify, add dataset_args={"general_fc": {"dataset_id": "evalscope/GeneralFunctionCall-Test"}}
)
run_task(task_cfg=task_cfg)
```
Method 2: Using custom local dataset
Example directory structure:
```text
fc/
└── example.jsonl # One sample per line, structure as described above
```
Simple example (example.jsonl, 3 lines):
```json
{"messages":[{"role":"system","content":"You are an assistant"},{"role":"user","content":"Please add 2 and 3"}],"tools":[{"type":"function","function":{"name":"add","description":"Add two numbers together","parameters":{"type":"object","properties":{"a":{"type":"number","description":"The first number"},"b":{"type":"number","description":"The second number"}},"required":["a","b"],"additionalProperties":false}}}],"should_call_tool":true}
{"messages":[{"role":"system","content":"You are an assistant"},{"role":"user","content":"The weather is nice today, let's chat"}],"tools":[{"type":"function","function":{"name":"add","description":"Add two numbers together","parameters":{"type":"object","properties":{"a":{"type":"number","description":"The first number"},"b":{"type":"number","description":"The second number"}},"required":["a","b"],"additionalProperties":false}}}],"should_call_tool":false}
{"messages":[{"role":"system","content":"You are an assistant"},{"role":"user","content":"Convert 37 degrees Celsius to Fahrenheit"}],"tools":[{"type":"function","function":{"name":"convert_temperature","description":"Convert Celsius to Fahrenheit","parameters":{"type":"object","properties":{"celsius":{"type":"number","description":"Temperature value in Celsius"}},"required":["celsius"],"additionalProperties":false}}}],"should_call_tool":true}
```
Execution example:
```python
from evalscope import TaskConfig, run_task
task_cfg = TaskConfig(
model='qwen-plus',
api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
api_key=env.get('DASHSCOPE_API_KEY'),
datasets=['general_fc'],
dataset_args={
'general_fc': {
"local_path": "custom_eval/text/fc", # Custom dataset root directory
"subset_list": ["example"] # Corresponds to example.jsonl
}
},
)
run_task(task_cfg=task_cfg)
```
### Evaluation Metrics
general_fc outputs the following metrics:
- `count_finish_reason_tool_call`: Number of samples where model predicts tool call (`finish_reason == "tool_calls"`)
- `count_successful_tool_call`: Among samples attempting tool calls, number of samples with matching tool name and parameters passing JSON Schema validation
- `schema_accuracy`: Among samples attempting tool calls, proportion of parameters passing validation
- `tool_call_f1`: Binary classification F1 based on label `should_call_tool` and whether model calls tool
Example output:
```text
+-----------+------------+-------------------------------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+===========+============+===============================+==========+=======+=========+=========+
| qwen-plus | general_fc | count_finish_reason_tool_call | default | 10 | 3 | default |
+-----------+------------+-------------------------------+----------+-------+---------+---------+
| qwen-plus | general_fc | count_successful_tool_call | default | 10 | 2 | default |
+-----------+------------+-------------------------------+----------+-------+---------+---------+
| qwen-plus | general_fc | schema_accuracy | default | 10 | 0.6667 | default |
+-----------+------------+-------------------------------+----------+-------+---------+---------+
| qwen-plus | general_fc | tool_call_f1 | default | 10 | 0.5 | default |
+-----------+------------+-------------------------------+----------+-------+---------+---------+
```

600
evalscope/docs/en/advanced_guides/custom_dataset/vlm.md Normal file
View File

@ -0,0 +1,600 @@
# Multimodal Large Models
This framework supports two custom multimodal evaluation methods:
- **General-VQA Format**: Based on OpenAI message format, supports multi-image/audio input, system prompts, and base64 encoding, suitable for Q&A-based multimodal evaluation tasks.
- **General-VMCQ Format**: Similar to MMMU format, question text can contain image placeholders `<image x>`, suitable for multiple-choice multimodal evaluation tasks.
## General-VQA Format
### 1. Data Preparation
Prepare data files conforming to OpenAI message format, supporting **JSONL** or **TSV** formats:
**JSONL Format Example** (`example_openai.jsonl`):
```json
{"messages": [{"role": "user", "content": [{"type": "text", "text": "What animal is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/dog.jpg"}}]}], "answer": "Dog"}
{"messages": [{"role": "user", "content": [{"type": "text", "text": "What building is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/AMNH.jpg"}}]}], "answer": "Museum"}
```
**TSV Format Example** (`example_openai.tsv`):
```text
messages answer
[{"role": "user", "content": [{"type": "text", "text": "What animal is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/dog.jpg"}}]}] Dog
[{"role": "user", "content": [{"type": "text", "text": "What building is this?"}, {"type": "image_url", "image_url": {"url": "custom_eval/multimodal/images/AMNH.jpg"}}]}] Museum
```
**Field Descriptions**:
- `messages`: OpenAI format message array, supporting:
- Text content: `{"type": "text", "text": "question text"}`
- Image URL: `{"type": "image_url", "image_url": {"url": "path or base64"}}`
- Audio input: `{"type": "input_audio", "input_audio": {"data": "path or base64", "format": "wav"}}`
- Video URL: `{"type": "video_url", "video_url": {"url": "path or base64"}}`
- System message: `{"role": "system", "content": "system prompt"}`
- `answer`: Reference answer (optional, used to calculate BLEU and Rouge scores)
**Supported Image Formats**:
- Local path: `"url": "custom_eval/multimodal/images/dog.jpg"`
- HTTP URL: `"url": "https://example.com/image.jpg"` (requires model service support)
- Base64 encoding: `"url": "data:image/jpeg;base64,/9j/4AAQSkZJRg..."`
**Supported Audio Formats**:
- Local path: `"data": "custom_eval/multimodal/audio/sample.wav"`
- Base64 encoding: `"data": "data:audio/wav;base64,UklGRiQ..."`
- Audio format (`format` field): supports `"wav"` and `"mp3"`
**Supported Video Formats**:
- Local path: `"url": "custom_eval/multimodal/videos/sample.mp4"`
- HTTP URL: `"url": "https://example.com/video.mp4"` (requires model service support)
- Base64 encoding: `"url": "data:video/mp4;base64,AAAAIGZ0eX..."`
- Video format is inferred from the path, URL, or data URI; supported formats are `"mp4"`, `"mpeg"`, and `"mov"`.
**Multi-image Input**
Supports using multiple images in one question:
```json
{
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Compare these two images:"},
{"type": "image_url", "image_url": {"url": "image1.jpg"}},
{"type": "text", "text": "and"},
{"type": "image_url", "image_url": {"url": "image2.jpg"}},
{"type": "text", "text": "What are the differences?"}
]
}
],
"answer": "The main differences are..."
}
```
**System Prompt**
You can add system messages to set the evaluation context:
```json
{
"messages": [
{"role": "system", "content": "You are a medical AI assistant."},
{
"role": "user",
"content": [
{"type": "text", "text": "Analyze this X-ray:"},
{"type": "image_url", "image_url": {"url": "xray.jpg", "detail": "high"}}
]
}
],
"answer": "The X-ray shows..."
}
```
**Base64 Images**
Supports directly using base64 encoded images:
```json
{
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "What's in this image?"},
{
"type": "image_url",
"image_url": {
"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD..."
}
}
]
}
],
"answer": "A beautiful landscape"
}
```
**Audio Input**
Supports audio content input using OpenAI `input_audio` format. The `data` field accepts either a local file path or base64-encoded data. The `format` field supports `"wav"` and `"mp3"`:
```json
{
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the content of this audio clip."},
{
"type": "input_audio",
"input_audio": {
"data": "custom_eval/multimodal/audio/sample.wav",
"format": "wav"
}
}
]
}
],
"answer": "A piano music performance."
}
```
You can also use base64-encoded audio data:
```json
{
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "What is being said in this audio?"},
{
"type": "input_audio",
"input_audio": {
"data": "UklGRiQAAABXQVZFZm10IBAAAA...",
"format": "wav"
}
}
]
}
],
"answer": "Hello, world."
}
```
**Video Input**
Supports video content input using OpenAI-compatible `video_url` format. The `url` field accepts either a local file path, HTTP URL, or base64-encoded data URL:
```json
{
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the content of this video clip."},
{
"type": "video_url",
"video_url": {
"url": "custom_eval/multimodal/videos/sample.mp4"
}
}
]
}
],
"answer": "A short video clip."
}
```
### 2. Configure Evaluation Task
Evaluate using Python API or CLI:
**Python API**:
```python
from evalscope.run import run_task
from evalscope.config import TaskConfig
from os import environ as env
task_cfg = TaskConfig(
model='qwen-vl-plus',
api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
api_key=env.get('DASHSCOPE_API_KEY'),
eval_type='openai_api',
datasets=['general_vqa'],
dataset_args={
'general_vqa': {
'local_path': 'custom_eval/multimodal/vqa', # Dataset directory
'subset_list': ['example_openai'], # Filename (without extension)
}
},
limit=5, # Optional: limit number of evaluation samples
)
result = run_task(task_cfg=task_cfg)
```
**CLI**:
```bash
evalscope eval \
--model qwen-vl-plus \
--api-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
--api-key "$DASHSCOPE_API_KEY" \
--eval-type openai_api \
--datasets general_vqa \
--dataset-args '{"general_vqa": {"local_path": "custom_eval/multimodal/vqa", "subset_list": ["example_openai"]}}' \
--limit 5
```
Evaluation will output BLEU and Rouge metrics:
```text
+--------------+-------------+----------------+----------------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+==============+=============+================+================+=======+=========+=========+
| qwen-vl-plus | general_vqa | mean_bleu-1 | example_openai | 5 | 0.0067 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_bleu-2 | example_openai | 5 | 0 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_bleu-3 | example_openai | 5 | 0 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_bleu-4 | example_openai | 5 | 0 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-1-R | example_openai | 5 | 0.4 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-1-P | example_openai | 5 | 0.0062 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-1-F | example_openai | 5 | 0.0121 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-2-R | example_openai | 5 | 0 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-2-P | example_openai | 5 | 0 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-2-F | example_openai | 5 | 0 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-L-R | example_openai | 5 | 0.4 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-L-P | example_openai | 5 | 0.0047 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
| qwen-vl-plus | general_vqa | mean_Rouge-L-F | example_openai | 5 | 0.0093 | default |
+--------------+-------------+----------------+----------------+-------+---------+---------+
```
### 3. Configure Judge Model
You can specify a judge model through the `judge_model` parameter to generate reference answers for evaluation, which will obtain accuracy metrics:
```python
from evalscope.run import run_task
from evalscope.constants import EvalType, JudgeStrategy
from os import environ as env
task_cfg = TaskConfig(
model='qwen-vl-plus',
api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
api_key=env.get('DASHSCOPE_API_KEY'),
eval_type='openai_api',
datasets=['general_vqa'],
dataset_args={
'general_vqa': {
'local_path': 'custom_eval/multimodal/vqa',
'subset_list': ['example_openai'],
}
},
limit=5,
judge_model_args={
'model_id': 'qwen-plus', # Does not need to be a multimodal model
'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
'api_key': env.get('DASHSCOPE_API_KEY'),
'generation_config': {
'temperature': 0.0,
'max_tokens': 4096
},
},
eval_batch_size=5,
judge_strategy=JudgeStrategy.LLM,
)
result = run_task(task_cfg=task_cfg)
```
**CLI** (equivalent):
```bash
evalscope eval \
--model qwen-vl-plus \
--api-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
--api-key "$DASHSCOPE_API_KEY" \
--eval-type openai_api \
--datasets general_vqa \
--dataset-args '{"general_vqa": {"local_path": "custom_eval/multimodal/vqa", "subset_list": ["example_openai"]}}' \
--limit 5 \
--judge-model-args '{"model_id": "qwen-plus", "api_url": "https://dashscope.aliyuncs.com/compatible-mode/v1", "api_key": "$DASHSCOPE_API_KEY", "generation_config": {"temperature": 0.0, "max_tokens": 4096}}' \
--judge-worker-num 5 \
--judge-strategy llm
```
Evaluation will output accuracy metrics:
```text
+--------------+-------------+----------+----------------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+==============+=============+==========+================+=======+=========+=========+
| qwen-vl-plus | general_vqa | mean_acc | example_openai | 5 | 1 | default |
+--------------+-------------+----------+----------------+-------+---------+---------+
```
## General-VMCQ Format
### 1. Data Preparation
General-VMCQ adopts a structure similar to MMMU: question text can contain image placeholders `<image x>` and video placeholders `<video x>`; `options` is a Python list string, options can be text or media placeholders.
Media support the following forms (all strings):
- Image local or remote path/URL: `"custom_eval/multimodal/images/dog.jpg"` or `"https://.../dog.jpg"`
- Image Base64 Data URL: `"data:image/jpeg;base64,/9j/4AAQSk..."`
- Video local or remote path/URL: `"custom_eval/multimodal/videos/sample.mp4"` or `"https://.../sample.mp4"`
- Video Base64 Data URL: `"data:video/mp4;base64,AAAAIGZ0eX..."`
Supports up to 100 images (`image_1` to `image_100`) and 100 videos (`video_1` to `video_100`). Missing media placeholders are ignored.
**JSONL Example** (`example.jsonl`):
```json
{"question": "Which image shows a dog?", "options": ["<image 1>", "<image 2>", "<image 3>", "<image 4>"], "image_1": "custom_eval/multimodal/images/dog.jpg", "image_2": "custom_eval/multimodal/images/AMNH.jpg", "image_3": "custom_eval/multimodal/images/tesla.jpg", "image_4": "custom_eval/multimodal/images/tokyo.jpg", "answer": "A"}
{"question": "<image 1> What building is this?", "options": ["School", "Hospital", "Park", "Museum"], "image_1": "custom_eval/multimodal/images/AMNH.jpg", "answer": "D"}
{"question": "<video 1> What type of media is provided in this sample?", "options": ["Image", "Audio", "Video", "Text"], "video_1": "custom_eval/multimodal/videos/sample.mp4", "answer": "C"}
```
**TSV Example** (`example.tsv`):
```text
question options answer image_1 image_2 image_3 image_4
Which image shows a dog? ["<image 1>", "<image 2>", "<image 3>", "<image 4>"] A custom_eval/multimodal/images/dog.jpg custom_eval/multimodal/images/AMNH.jpg custom_eval/multimodal/images/tesla.jpg custom_eval/multimodal/images/tokyo.jpg
<image 1> What building is this? ["School", "Hospital", "Park", "Museum"] D custom_eval/multimodal/images/AMNH.jpg
```
**Field Descriptions**:
- `question`: Question text, can contain `<image x>` or `<video x>` placeholders
- `options`: List (JSON array), elements can be text (e.g., `"School"`) or media placeholders (e.g., `"<image 1>"`, `"<video 1>"`), no need to add prefixes like `A.`, `B.`
- `answer`: Correct answer letter (e.g., `"A"`, `"B"`)
- `image_k`: Image string (local/remote path or base64 Data URL), k ∈ [1, 100]
- `video_k`: Video string (local/remote path or base64 Data URL), k ∈ [1, 100]
- `video_k_format`: Optional video format hint; supports `"mp4"`, `"mpeg"`, and `"mov"`
### 2. Configure Evaluation Task
**Python API**:
```python
from evalscope.run import run_task
from evalscope.config import TaskConfig
from os import environ as env
task_cfg = TaskConfig(
model='qwen-vl-plus',
api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
api_key=env.get('DASHSCOPE_API_KEY'),
eval_type='openai_api',
datasets=['general_vmcq'],
dataset_args={
'general_vmcq': {
'local_path': 'custom_eval/multimodal/mcq',
'subset_list': ['example'],
}
},
limit=10,
)
result = run_task(task_cfg=task_cfg)
print(result)
```
**CLI**:
```bash
evalscope eval \
--model qwen-vl-plus \
--api-url https://dashscope.aliyuncs.com/compatible-mode/v1 \
--api-key "$DASHSCOPE_API_KEY" \
--eval-type openai_api \
--datasets general_vmcq \
--dataset-args '{"general_vmcq": {"local_path": "custom_eval/multimodal/mcq", "subset_list": ["example"]}}' \
--limit 10
```
### 3. Evaluation Results
Evaluation will output accuracy metrics:
```text
+--------------+--------------+----------+----------+-------+---------+---------+
| Model | Dataset | Metric | Subset | Num | Score | Cat.0 |
+==============+==============+==========+==========+=======+=========+=========+
| qwen-vl-plus | general_vmcq | mean_acc | example | 3 | 1 | default |
+--------------+--------------+----------+----------+-------+---------+---------+
```
---
## Based on VLMEvalKit (Deprecated)
````{warning}
The following format is the Legacy version. It is recommended to use the **General Multimodal Format** described above.
Legacy format requires additional VLMEvalKit dependencies:
```bash
pip install evalscope[vlmeval]
```
Reference: [Evaluating with VLMEvalKit Backend](../../user_guides/backend/vlmevalkit_backend.md)
````
### Multiple Choice Format (MCQ)
#### 1. Data Preparation
The evaluation metric is accuracy, and you need to define a tsv file in the following format (separated by `\t`):
```text
index category answer question A B C D image_path
1 Animals A What animal is this? Dog Cat Tiger Elephant /root/LMUData/images/custom_mcq/dog.jpg
2 Buildings D What building is this? School Hospital Park Museum /root/LMUData/images/custom_mcq/AMNH.jpg
3 Cities B Which city's skyline is this? New York Tokyo Shanghai Paris /root/LMUData/images/custom_mcq/tokyo.jpg
4 Vehicles C What is the brand of this car? BMW Audi Tesla Mercedes /root/LMUData/images/custom_mcq/tesla.jpg
5 Activities A What is the person in the picture doing? Running Swimming Reading Singing /root/LMUData/images/custom_mcq/running.jpg
```
Where:
- `index` is the question number
- `question` is the question
- `answer` is the answer
- `A`, `B`, `C`, `D` are options, must have at least two options
- `answer` is the answer option
- `image_path` is the image path (absolute path recommended); can also be replaced with `image` field, which should be base64 encoded image
- `category` is the category (optional field)
Place this file in the `~/LMUData` path, and you can use the filename for evaluation. For example, if the filename is `custom_mcq.tsv`, use `custom_mcq` for evaluation.
#### 2. Configuration File
The configuration file can be in `python dict`, `yaml`, or `json` format. For example, the following `config.yaml` file:
```yaml
eval_backend: VLMEvalKit
eval_config:
model:
- type: qwen-vl-chat # Deployed model name
name: CustomAPIModel # Fixed value
api_base: http://localhost:8000/v1/chat/completions
key: EMPTY
temperature: 0.0
img_size: -1
data:
- custom_mcq # Custom dataset name, placed in `~/LMUData` path
mode: all
limit: 10
reuse: false
work_dir: outputs
nproc: 1
```
#### 3. Run Evaluation
Run the following code to start evaluation:
```python
from evalscope.run import run_task
run_task(task_cfg='config.yaml')
```
Evaluation results:
```text
---------- ----
split none
Overall 1.0
Activities 1.0
Animals 1.0
Buildings 1.0
Cities 1.0
Vehicles 1.0
---------- ----
```
### Custom Question-Answer Format (VQA)
#### 1. Data Preparation
Prepare a tsv file in question-answer format as follows:
```text
index answer question image_path
1 Dog What animal is this? /root/LMUData/images/custom_mcq/dog.jpg
2 Museum What building is this? /root/LMUData/images/custom_mcq/AMNH.jpg
3 Tokyo Which city's skyline is this? /root/LMUData/images/custom_mcq/tokyo.jpg
4 Tesla What is the brand of this car? /root/LMUData/images/custom_mcq/tesla.jpg
5 Running What is the person in the picture doing? /root/LMUData/images/custom_mcq/running.jpg
```
This file is the same format as the multiple-choice format, where:
- `index` is the question number
- `question` is the question
- `answer` is the answer
- `image_path` is the image path (absolute path recommended); can also be replaced with `image` field, which should be base64 encoded image
Place this file in the `~/LMUData` path, and you can use the filename for evaluation. For example, if the filename is `custom_vqa.tsv`, use `custom_vqa` for evaluation.
#### 2. Custom Evaluation Script
The following is an example of a custom dataset. This example implements a custom evaluation script for question-answer format, which automatically loads the dataset, uses default prompts for Q&A, and finally calculates accuracy as the evaluation metric.
```python
import os
import numpy as np
from vlmeval.dataset.image_base import ImageBaseDataset
from vlmeval.dataset.image_vqa import CustomVQADataset
from vlmeval.smp import load, dump, d2df
class CustomDataset:
def load_data(self, dataset):
# Custom dataset loading
data_path = os.path.join(os.path.expanduser("~/LMUData"), f'{dataset}.tsv')
return load(data_path)
def build_prompt(self, line):
msgs = ImageBaseDataset.build_prompt(self, line)
# Add prompts or custom instructions here
msgs[-1]['value'] += '\nAnswer the question with one word or phrase.'
return msgs
def evaluate(self, eval_file, **judge_kwargs):
data = load(eval_file)
assert 'answer' in data and 'prediction' in data
data['prediction'] = [str(x) for x in data['prediction']]
data['answer'] = [str(x) for x in data['answer']]
print(data)
# ========Calculate evaluation metrics as needed=========
# Exact match
result = np.mean(data['answer'] == data['prediction'])
ret = {'Overall': result}
ret = d2df(ret).round(2)
# Save results
suffix = eval_file.split('.')[-1]
result_file = eval_file.replace(f'.{suffix}', '_acc.csv')
dump(ret, result_file)
return ret
# ====================================
# Need to keep the following code to override default dataset class
CustomVQADataset.load_data = CustomDataset.load_data
CustomVQADataset.build_prompt = CustomDataset.build_prompt
CustomVQADataset.evaluate = CustomDataset.evaluate
```
#### 3. Configuration File
The configuration file can be in `python dict`, `yaml`, or `json` format. For example, the following `config.yaml` file:
```{code-block} yaml
:caption: config.yaml
eval_backend: VLMEvalKit
eval_config:
model:
- type: qwen-vl-chat
name: CustomAPIModel
api_base: http://localhost:8000/v1/chat/completions
key: EMPTY
temperature: 0.0
img_size: -1
data:
- custom_vqa # Custom dataset name, placed in `~/LMUData` path
mode: all
limit: 10
reuse: false
work_dir: outputs
nproc: 1
```
#### 4. Run Evaluation
The complete evaluation script is as follows:
```{code-block} python
:emphasize-lines: 1
from custom_dataset import CustomDataset # Import custom dataset
from evalscope.run import run_task
run_task(task_cfg='config.yaml')
```
Evaluation results:
```text
{'qwen-vl-chat_custom_vqa_acc': {'Overall': '1.0'}}
```

211
evalscope/docs/en/advanced_guides/custom_model.md Normal file
View File

@ -0,0 +1,211 @@
# Custom Model Evaluation
EvalScope supports the evaluation of various models through the `ModelAPI` abstract interface. This document will guide you on how to implement a custom model adapter by referencing `MockLLM`.
## When Do You Need a Custom Model Adapter?
You might need to create a custom model adapter in the following scenarios:
- Your model does not support the standard OpenAI API format.
- You need to perform special processing on model inputs and outputs.
- You require specific inference parameters or configurations.
## ModelAPI Interface Definition
All models must implement the `ModelAPI` abstract base class:
```python
from abc import ABC, abstractmethod
from typing import List, Optional
from evalscope.api.messages import ChatMessage
from evalscope.api.model import GenerateConfig, ModelOutput
from evalscope.api.tool import ToolChoice, ToolInfo
class ModelAPI(ABC):
"""Base interface for model API providers"""
def __init__(
self,
model_name: str,
base_url: Optional[str] = None,
api_key: Optional[str] = None,
config: GenerateConfig = GenerateConfig(),
**kwargs
) -> None:
self.model_name = model_name
self.base_url = base_url
self.api_key = api_key
self.config = config
@abstractmethod
def generate(
self,
input: List[ChatMessage],
tools: List[ToolInfo],
tool_choice: ToolChoice,
config: GenerateConfig,
) -> ModelOutput:
"""Generate model output (must be implemented)"""
pass
```
## MockLLM Reference Implementation
`MockLLM` is a simple test model implementation that demonstrates the basic structure of `ModelAPI`:
```python
from typing import Any, Dict, List, Optional
from evalscope.api.messages import ChatMessage
from evalscope.api.model import GenerateConfig, ModelAPI, ModelOutput
from evalscope.api.tool import ToolChoice, ToolInfo
class MockLLM(ModelAPI):
"""Mock model implementation for testing purposes"""
default_output = 'Default output from mockllm/model'
def __init__(
self,
model_name: str,
base_url: Optional[str] = None,
api_key: Optional[str] = None,
config: GenerateConfig = GenerateConfig(),
custom_output: Optional[str] = None,
**model_args: Dict[str, Any],
) -> None:
super().__init__(model_name, base_url, api_key, config)
self.model_args = model_args
self.custom_output = custom_output
def generate(
self,
input: List[ChatMessage],
tools: List[ToolInfo],
tool_choice: ToolChoice,
config: GenerateConfig,
) -> ModelOutput:
# Use custom output if provided, otherwise use default output
output_text = self.custom_output if self.custom_output is not None else self.default_output
return ModelOutput.from_content(
model=self.model_name,
content=output_text
)
```
## Custom Model Implementation
Create your model by referencing the structure of `MockLLM`:
```python
from typing import List, Optional, Dict, Any
from evalscope.api.model import ModelAPI, GenerateConfig, ModelOutput
from evalscope.api.messages import ChatMessage
from evalscope.api.tool import ToolChoice, ToolInfo
from evalscope.api.registry import register_model_api
# 1. Register the model using register_model_api
@register_model_api(name='my_custom_model')
class MyCustomModel(ModelAPI):
"""Custom model implementation"""
def __init__(
self,
model_name: str,
base_url: Optional[str] = None,
api_key: Optional[str] = None,
config: GenerateConfig = GenerateConfig(),
**model_args: Dict[str, Any],
) -> None:
super().__init__(model_name, base_url, api_key, config)
self.model_args = model_args
print(self.model_args)
# 2. Initialize your model here
# For example: load model files, establish connections, etc.
def generate(
self,
input: List[ChatMessage],
tools: List[ToolInfo],
tool_choice: ToolChoice,
config: GenerateConfig,
) -> ModelOutput:
# 3. Implement model inference logic
# 3.1 Process input messages
input_text = self._process_messages(input)
# 3.2 Call your model
response = self._call_model(input_text, config)
# 3.3 Return standardized output
return ModelOutput.from_content(
model=self.model_name,
content=response
)
def _process_messages(self, messages: List[ChatMessage]) -> str:
"""Convert chat messages to text"""
text_parts = []
for message in messages:
role = getattr(message, 'role', 'user')
content = getattr(message, 'content', str(message))
text_parts.append(f"{role}: {content}")
return '\n'.join(text_parts)
def _call_model(self, input_text: str, config: GenerateConfig) -> str:
"""Invoke your model for inference"""
# Implement your model invocation logic here
# For example: API calls, local model inference, etc.
return f"Response to: {input_text}"
```
## Using the Custom Model
### Direct Usage
```python
from evalscope import run_task, TaskConfig
# Create a model instance
custom_model = MyCustomModel(
model_name='my-model',
model_args={'test': 'test'}
)
# Configure the evaluation task
task_config = TaskConfig(
model=custom_model,
datasets=['gsm8k'],
limit=5
)
# Run the evaluation
results = run_task(task_cfg=task_config)
```
### Usage Through Registration
```python
from evalscope import run_task, TaskConfig
from xxx import MyCustomModel # Import the model beforehand for automatic registration
# Use the registered model
task_config = TaskConfig(
model='my-model',
eval_type='my_custom_model', # Name used in register_model_api
datasets=['gsm8k'],
model_args={'test': 'test'},
limit=5
)
results = run_task(task_cfg=task_config)
```
## Key Points
1. **Inherit from `ModelAPI`** and implement the `generate` method.
2. **Return a `ModelOutput`** object, which can be created using `ModelOutput.from_content()`.
3. **Handle exceptions** to ensure the stability of model calls.
4. **Register the model** for use in configurations.

BIN
evalscope/docs/en/advanced_guides/images/mmlu_pro_preview.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1022 KiB

112
evalscope/docs/en/advanced_guides/sandbox.md Normal file
View File

@ -0,0 +1,112 @@
# Sandbox Execution
EvalScope uses [ms_enclave](https://github.com/modelscope/ms-enclave) to run
model-generated code inside isolated sandboxes. Two execution paths share
the same service layer:
* **Pooled** — a warm pool of long-lived sandboxes used by code benchmarks
(e.g. HumanEval, MBPP) through the `SandboxMixin`.
* **Per-sample** — one sandbox per sample created by agent environments
(e.g. `EnclaveAgentEnvironment`) for SWE-bench-style benchmarks.
Both paths are backed by the process-wide `SandboxService` defined in
`evalscope.api.sandbox`, which caches `SandboxManager` instances keyed on
`(engine, manager_config)` and cleans them up at exit.
## Installation
```bash
pip install 'evalscope[sandbox]'
```
Requires a reachable Docker daemon or valid Volcengine credentials
(depending on the chosen engine).
## Configuration
Prefer the nested `sandbox` object on `TaskConfig`. The legacy fields
`use_sandbox` / `sandbox_type` / `sandbox_manager_config` remain as
deprecated aliases and are kept in sync automatically.
### Python
```python
from evalscope import TaskConfig
from evalscope.config import SandboxTaskConfig
task = TaskConfig(
datasets=['humaneval'],
sandbox=SandboxTaskConfig(
enabled=True,
engine='docker', # 'docker' | 'volcengine'
default_config={
'image': 'python:3.11-slim',
'working_dir': '/workspace',
},
manager_config={
# e.g. 'base_url': 'unix:///var/run/docker.sock'
},
pool_size=8, # optional, defaults to eval_batch_size
),
)
```
### YAML / JSON
```yaml
sandbox:
enabled: true
engine: volcengine
manager_config:
api_key: ${VOLC_API_KEY}
region: cn-beijing
default_config:
image: my-registry/my-image:latest
```
## Supported engines
| `engine` | Aliases | ms_enclave backend |
|--------------|----------------------|--------------------------------|
| `docker` | (default) | `SandboxManagerFactory` |
| `volcengine` | `volcano`, `volc` | `VolcengineSandboxManager` |
## Per-sample overrides (agent benchmarks)
Agent benchmarks that need a different sandbox per sample (e.g. SWE-bench
Pro, which pins an image per instance) can override `build_sandbox_config`
on their adapter:
```python
class MyBenchmarkAdapter(DefaultDataAdapter):
def build_sandbox_config(self, sample):
return {'image': sample.metadata['docker_image']}
```
The returned dict is merged on top of `sandbox.default_config` and passed
to the environment constructor. `agent_config.environment_extra` still
has the final word if the user wants to override per-run.
## Pool vs per-sample
| Aspect | Pool (SandboxMixin) | Per-sample (Agent env) |
|------------------|------------------------------------|-------------------------------|
| Lifetime | Reused across samples | Fresh container per sample |
| Warmup | `manager.initialize_pool` | None |
| Execution API | `PoolHandle.execute_tool` | `SandboxHandle.execute_tool` |
| Typical use | Code benchmarks | SWE-bench, Agent tool use |
Managers are shared across both paths when `(engine, manager_config)`
match, so you only pay for one connection even if a benchmark uses both.
## API reference
The public surface lives in `evalscope.api.sandbox`:
- `SandboxEngine`, `resolve_engine(value)`
- `SandboxService`, `get_sandbox_service()`
- `PoolHandle`, `SandboxHandle`
- `build_sandbox_config(engine, cfg_dict)`,
`merge_sandbox_config_dicts(*dicts)`
See `tests/agent/test_sandbox_service.py` for runnable examples.

144
evalscope/docs/en/benchmarks/a_okvqa.md Normal file
View File

@ -0,0 +1,144 @@
# A-OKVQA
## Overview
A-OKVQA (Augmented OK-VQA) is a benchmark designed to evaluate commonsense reasoning and external world knowledge in visual question answering. It extends beyond basic VQA tasks that rely solely on image content, requiring models to leverage a broad spectrum of commonsense and factual knowledge about the world.
## Task Description
- **Task Type**: Visual Question Answering with Knowledge Reasoning
- **Input**: Image + natural language question requiring external knowledge
- **Output**: Answer (multiple-choice or open-ended)
- **Domains**: Commonsense reasoning, factual knowledge, visual understanding
## Key Features
- Requires commonsense reasoning beyond direct visual observation
- Combines visual understanding with external world knowledge
- Includes both multiple-choice and open-ended question formats
- Questions annotated with rationales explaining the reasoning process
- More challenging than standard VQA benchmarks
## Evaluation Notes
- Default evaluation uses the **validation** split
- Primary metric: **Accuracy** for multiple-choice questions
- Uses Chain-of-Thought (CoT) prompting for better reasoning
- Questions require reasoning beyond what is directly visible in images
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `a_okvqa` |
| **Dataset ID** | [HuggingFaceM4/A-OKVQA](https://modelscope.cn/datasets/HuggingFaceM4/A-OKVQA/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `MCQ`, `MultiModal` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `validation` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 1,145 |
| Prompt Length (Mean) | 310.84 chars |
| Prompt Length (Min/Max) | 276 / 405 chars |
**Image Statistics:**
| Metric | Value |
|--------|-------|
| Total Images | 1,145 |
| Images per Sample | min: 1, max: 1, mean: 1 |
| Resolution Range | 305x229 - 640x640 |
| Formats | jpeg |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "3d2b0351",
"content": [
{
"text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\n\nWhat is in the motorcyclist's mouth?\n\nA) toothpick\nB) food\nC) popsicle stick\nD) cigarette"
},
{
"image": "[BASE64_IMAGE: jpeg, ~53.4KB]"
}
]
}
],
"choices": [
"toothpick",
"food",
"popsicle stick",
"cigarette"
],
"target": "D",
"id": 0,
"group_id": 0,
"metadata": {
"question_id": "22jbM6gDxdaMaunuzgrsBB",
"direct_answers": "['cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette']",
"difficult_direct_answer": false,
"rationales": [
"He's smoking while riding.",
"The motorcyclist has a lit cigarette in his mouth while he rides on the street.",
"The man is smoking."
]
}
}
```
## Prompt Template
**Prompt Template:**
```text
Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets a_okvqa \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['a_okvqa'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

140
evalscope/docs/en/benchmarks/aa_lcr.md Normal file
View File

@ -0,0 +1,140 @@
# AA-LCR
## Overview
AA-LCR (Artificial Analysis Long Context Retrieval) is a benchmark for evaluating long-context retrieval and reasoning capabilities of language models. It requires models to find and synthesize information across multiple documents.
## Task Description
- **Task Type**: Long-Context Question Answering
- **Input**: Multiple documents + question requiring cross-document reasoning
- **Output**: Answer synthesized from document information
- **Context**: Very long context (multiple documents concatenated)
## Key Features
- Tests long-context retrieval abilities
- Multiple document understanding
- Cross-document reasoning required
- LLM-based judging for answer correctness
- Auto-download of document corpus
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Primary metric: **Accuracy** (via LLM judge)
- Evaluates on **test** split
- Documents auto-downloaded if `text_dir` not specified
- Judge prompt compares candidate answer against reference
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `aa_lcr` |
| **Dataset ID** | [evalscope/AA-LCR](https://modelscope.cn/datasets/evalscope/AA-LCR/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `LongContext`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 100 |
| Prompt Length (Mean) | 414674.06 chars |
| Prompt Length (Min/Max) | 240709 / 548771 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "0b26b81d",
"content": "\nBEGIN INPUT DOCUMENTS\n\nBEGIN DOCUMENT 1:\n[Contents lists available at ScienceDirect](https://www.elsevier.com/locate/techfore)\n\n# Technological Forecasting & Social Change\n\n[journal homepage: www.elsevier.com/locate/techfore](http://www.else ... [TRUNCATED] ... and undertakings issued by the ACCC. Identify and rank the industries explicitly mentioned in the paragraphs, according to the number of infringements over the past three decades. Exclude Broadcasting Industry from the answer.\n\nEND QUESTION\n"
}
],
"target": "1. Airline Industry (12)\\n2. Accommodation Industry (4)",
"id": 0,
"group_id": 0,
"metadata": {
"question": "Based on the provided documents, there appears to be a correlation between industry concentration and the frequency of consumer-related infringements and undertakings issued by the ACCC. Identify and rank the industries explicitly mentioned in the paragraphs, according to the number of infringements over the past three decades. Exclude Broadcasting Industry from the answer.",
"data_source_urls": "https://competition-policy.ec.europa.eu/system/files/2024-06/A_taxonomy_of_industry_competition_launch.pdf;https://www.industry.gov.au/sites/default/files/2023-11/barriers-to-collaboration-and-commercialisation-iisa.pdf;https://e61.in/wp-content/uploads/2023/08/The-State-of-Competition.pdf;https://uu.diva-portal.org/smash/get/diva2:1798138/FULLTEXT01.pdf;https://one.oecd.org/document/DAF/COMP(2023)14/en/pdf",
"input_tokens": 94494
}
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
BEGIN INPUT DOCUMENTS
{documents_text}
END INPUT DOCUMENTS
Answer the following question using the input documents provided above.
START QUESTION
{question}
END QUESTION
```
## Extra Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `text_dir` | `str | null` | `None` | Local directory containing extracted AA-LCR text files; if null will auto-download & extract. |
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets aa_lcr \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['aa_lcr'],
dataset_args={
'aa_lcr': {
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

452
evalscope/docs/en/benchmarks/acebench.md Normal file
View File

@ -0,0 +1,452 @@
# ACEBench
## Overview
ACEBench is a tool-use benchmark for evaluating whether large language models can select APIs, fill
arguments, handle abnormal requests, and complete realistic agent tasks.
## Task Description
- **Task Type**: Function calling and agentic tool use
- **Input**: Conversation history, API specifications, optional time/profile context, and agent task context
- **Output**: Function calls or diagnostic text for special cases
- **Subsets**: normal, special, and agent
## Evaluation Notes
- The adapter passes ACEBench API specifications as EvalScope tools and also includes concise text
instructions for text-only models.
- Normal samples are scored by matching function names and arguments.
- Special samples are scored against ACEBench's diagnostic text contract.
- Agent samples report `process_acc` against ACEBench milestones. If a model returns a final-state JSON object,
`end_state_acc` is also reported and used as `acc`; otherwise `acc` follows `process_acc`.
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `acebench` |
| **Dataset ID** | [evalscope/acebench](https://modelscope.cn/datasets/evalscope/acebench/summary) |
| **Paper** | N/A |
| **Tags** | `Agent`, `FunctionCalling`, `MultiTurn` |
| **Metrics** | `acc`, `process_acc`, `end_state_acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `normal` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 1,023 |
| Prompt Length (Mean) | 4676.74 chars |
| Prompt Length (Min/Max) | 1007 / 10555 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `normal` | 823 | 4859.68 | 1070 | 10555 |
| `special` | 150 | 3449.69 | 1007 | 8692 |
| `agent` | 50 | 5346.72 | 4454 | 5656 |
## Sample Example
**Subset**: `normal`
```json
{
"input": [
{
"id": "134dd42a",
"content": "You are evaluating ACEBench tool-use tasks.\n\nUse the available tool schemas when native function calling is supported.\n\nFor text-only output, return API calls as [ApiName(key1='value1', key2=2)]. Return only the call list and no extra explana ... [TRUNCATED 2553 chars] ... \"}, \"effects\": {\"description\": \"List of audio effects to apply.\", \"type\": \"array\", \"items\": {\"type\": \"string\", \"enum\": [\"reverb\", \"echo\", \"distortion\"]}}}, \"required\": [\"frequency\", \"gain\"]}}}, \"required\": [\"microphone\", \"performanceTime\"]}}]"
},
{
"id": "da9681d9",
"content": "I have been fascinated recently with total solar eclipses. I am planning my next travel and would like to know when the next total solar eclipse will be visible in Greece, specifically in Athens, over the next five years.",
"role": "user"
}
],
"target": "{\"ground_truth\": {\"NightSkyAnalysis_performEclipseAnalysis\": {\"dateRange\": {\"startDate\": \"2023-01-01\", \"endDate\": \"2028-01-01\"}, \"location\": {\"latitude\": 37.9838, \"longitude\": 23.7275}, \"eclipseType\": \"total\"}}, \"mile_stone\": []}",
"id": 0,
"group_id": 0,
"tools": [
{
"name": "NightSkyAnalysis_performEclipseAnalysis",
"description": "Analyzes the occurrence of solar eclipses, categorizes them into types, and predicts future occurrences based on historical data and celestial mechanics.",
"parameters": {
"type": "object",
"properties": {
"dateRange": {
"type": "object",
"description": "The range of dates for which to analyze solar eclipses.",
"properties": {
"startDate": {
"type": "string",
"description": "The starting date for the analysis in YYYY-MM-DD format."
},
"endDate": {
"type": "string",
"description": "The ending date for the analysis in YYYY-MM-DD format."
}
},
"required": [
"startDate",
"endDate"
]
},
"location": {
"type": "object",
"description": "Geographical coordinates to focus the eclipse analysis.",
"properties": {
"latitude": {
"type": "number",
"description": "Latitude of the location."
},
"longitude": {
"type": "number",
"description": "Longitude of the location."
}
},
"required": [
"latitude",
"longitude"
]
},
"eclipseType": {
"type": "string",
"description": "The type of solar eclipse to specifically analyze.",
"enum": [
"total",
"annular",
"partial"
]
}
},
"required": [
"dateRange",
"location"
]
}
},
{
"name": "AudioPerformanceOptimizer_optimizeMicrophoneSettings",
"description": "Optimizes microphone settings for live performances, focusing on dynamic microphones to enhance sound quality and reduce feedback.",
"parameters": {
"type": "object",
"properties": {
"microphone": {
"type": "object",
"description": "Details of the microphone used.",
"properties": {
"type": {
"type": "string",
"description": "Type of the microphone.",
"enum": [
"dynamic",
"condenser",
"ribbon"
]
},
"model": {
"type": "string",
"description": "Model of the microphone."
}
},
"required": [
"type",
"model"
]
},
"performanceTime": {
"type": "string",
"description": "Scheduled time for the performance.",
"enum": [
"morning",
"afternoon",
"evening",
"night"
]
},
"environment": {
"type": "object",
"description": "Environmental conditions of the performance area.",
"properties": {
"humidity": {
"type": "integer",
"description": "Humidity level as a percentage."
},
"temperature": {
"type": "integer",
"description": "Temperature in Celsius."
}
}
},
"soundSettings": {
"type": "array",
"description": "Specific sound settings to apply.",
"items": {
"type": "object",
"properties": {
"frequency": {
"type": "integer",
"description": "Frequency adjustments in Hz."
},
"gain": {
"type": "integer",
"description": "Gain adjustments in dB."
},
"effects": {
"type": "array",
"description": "List of audio effects to apply.",
"items": {
"type": "string",
"enum": [
"reverb",
"echo",
"distortion"
]
}
}
},
"required": [
"frequency",
"gain"
]
}
}
},
"required": [
"microphone",
"performanceTime"
]
}
}
],
"metadata": {
"id": "normal_single_turn_single_function_0",
"sub_category": "data_normal_single_turn_single_function",
"question": "user: I have been fascinated recently with total solar eclipses. I am planning my next travel and would like to know when the next total solar eclipse will be visible in Greece, specifically in Athens, over the next five years.\n",
"time": "",
"profile": "",
"functions": [
{
"name": "NightSkyAnalysis_performEclipseAnalysis",
"description": "Analyzes the occurrence of solar eclipses, categorizes them into types, and predicts future occurrences based on historical data and celestial mechanics.",
"parameters": {
"type": "object",
"properties": {
"dateRange": {
"description": "The range of dates for which to analyze solar eclipses.",
"type": "object",
"properties": {
"startDate": {
"description": "The starting date for the analysis in YYYY-MM-DD format.",
"type": "string"
},
"endDate": {
"description": "The ending date for the analysis in YYYY-MM-DD format.",
"type": "string"
}
},
"required": [
"startDate",
"endDate"
]
},
"location": {
"description": "Geographical coordinates to focus the eclipse analysis.",
"type": "object",
"properties": {
"latitude": {
"description": "Latitude of the location.",
"type": "number",
"minimum": -90,
"maximum": 90
},
"longitude": {
"description": "Longitude of the location.",
"type": "number",
"minimum": -180,
"maximum": 180
}
},
"required": [
"latitude",
"longitude"
]
},
"eclipseType": {
"description": "The type of solar eclipse to specifically analyze.",
"type": "string",
"enum": [
"total",
"annular",
"partial"
]
}
},
"required": [
"dateRange",
"location"
]
}
},
{
"name": "AudioPerformanceOptimizer_optimizeMicrophoneSettings",
"description": "Optimizes microphone settings for live performances, focusing on dynamic microphones to enhance sound quality and reduce feedback.",
"parameters": {
"type": "object",
"properties": {
"microphone": {
"description": "Details of the microphone used.",
"type": "object",
"properties": {
"type": {
"description": "Type of the microphone.",
"type": "string",
"enum": [
"dynamic",
"condenser",
"ribbon"
]
},
"model": {
"description": "Model of the microphone.",
"type": "string"
}
},
"required": [
"type",
"model"
]
},
"performanceTime": {
"description": "Scheduled time for the performance.",
"type": "string",
"enum": [
"morning",
"afternoon",
"evening",
"night"
]
},
"environment": {
"description": "Environmental conditions of the performance area.",
"type": "object",
"properties": {
"humidity": {
"description": "Humidity level as a percentage.",
"type": "integer",
"minimum": 0,
"maximum": 100
},
"temperature": {
"description": "Temperature in Celsius.",
"type": "integer"
}
}
},
"soundSettings": {
"description": "Specific sound settings to apply.",
"type": "array",
"items": {
"type": "object",
"properties": {
"frequency": {
"description": "Frequency adjustments in Hz.",
"type": "integer"
},
"gain": {
"description": "Gain adjustments in dB.",
"type": "integer"
},
"effects": {
"description": "List of audio effects to apply.",
"type": "array",
"items": {
"type": "string",
"enum": [
"reverb",
"echo",
"distortion"
]
}
}
},
"required": [
"frequency",
"gain"
]
}
}
},
"required": [
"microphone",
"performanceTime"
]
}
}
],
"ground_truth": {
"NightSkyAnalysis_performEclipseAnalysis": {
"dateRange": {
"startDate": "2023-01-01",
"endDate": "2028-01-01"
},
"location": {
"latitude": 37.9838,
"longitude": 23.7275
},
"eclipseType": "total"
}
},
"mile_stone": [],
"initial_config": {},
"involved_classes": []
}
}
```
## Prompt Template
*No prompt template defined.*
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets acebench \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['acebench'],
dataset_args={
'acebench': {
# subset_list: ['normal', 'special', 'agent'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

134
evalscope/docs/en/benchmarks/ai2d.md Normal file
View File

@ -0,0 +1,134 @@
# AI2D
## Overview
AI2D (AI2 Diagrams) is a benchmark dataset for evaluating AI systems' ability to understand and reason about scientific diagrams. It contains over 5,000 diverse diagrams from science textbooks covering topics like the water cycle, food webs, and biological processes.
## Task Description
- **Task Type**: Diagram Understanding and Visual Reasoning
- **Input**: Scientific diagram image + multiple-choice question
- **Output**: Correct answer choice
- **Domains**: Science education, visual reasoning, diagram comprehension
## Key Features
- Diagrams sourced from real science textbooks
- Requires joint understanding of visual layouts, symbols, and text labels
- Tests interpretation of relationships between diagram elements
- Multiple-choice format with challenging distractors
- Covers diverse scientific domains (biology, physics, earth science)
## Evaluation Notes
- Default evaluation uses the **test** split
- Primary metric: **Accuracy** on multiple-choice questions
- Uses Chain-of-Thought (CoT) prompting for reasoning
- Requires understanding both textual labels and visual elements
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `ai2d` |
| **Dataset ID** | [lmms-lab/ai2d](https://modelscope.cn/datasets/lmms-lab/ai2d/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `MultiModal`, `QA` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 3,088 |
| Prompt Length (Mean) | 324.64 chars |
| Prompt Length (Min/Max) | 256 / 1024 chars |
**Image Statistics:**
| Metric | Value |
|--------|-------|
| Total Images | 3,088 |
| Images per Sample | min: 1, max: 1, mean: 1 |
| Resolution Range | 177x131 - 1500x1500 |
| Formats | png |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "789e28fa",
"content": [
{
"text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\n\nwhich of these define dairy item\n\nA) c\nB) D\nC) b\nD) a"
},
{
"image": "[BASE64_IMAGE: png, ~226.2KB]"
}
]
}
],
"choices": [
"c",
"D",
"b",
"a"
],
"target": "B",
"id": 0,
"group_id": 0
}
```
## Prompt Template
**Prompt Template:**
```text
Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}. Think step by step before answering.
{question}
{choices}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets ai2d \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['ai2d'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

112
evalscope/docs/en/benchmarks/aime24.md Normal file
View File

@ -0,0 +1,112 @@
# AIME-2024
## Overview
AIME 2024 (American Invitational Mathematics Examination 2024) is a benchmark based on problems from the prestigious AIME competition. These problems represent some of the most challenging high school mathematics problems, requiring creative problem-solving and advanced mathematical reasoning.
## Task Description
- **Task Type**: Competition Mathematics Problem Solving
- **Input**: AIME-level mathematical problem
- **Output**: Integer answer (0-999) with step-by-step reasoning
- **Difficulty**: Advanced high school / early undergraduate level
## Key Features
- Problems from the 2024 AIME I and AIME II competitions
- Answers are always integers between 0 and 999
- Requires creative mathematical reasoning and problem-solving
- Topics: algebra, geometry, number theory, combinatorics, probability
- Represents top-tier high school mathematics competition difficulty
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Answers should be formatted within `\boxed{}` for proper extraction
- Only integer answers are accepted (matching AIME format)
- Problems are significantly harder than GSM8K or standard MATH benchmark
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `aime24` |
| **Dataset ID** | [evalscope/aime24](https://modelscope.cn/datasets/evalscope/aime24/summary) |
| **Paper** | N/A |
| **Tags** | `Math`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 30 |
| Prompt Length (Mean) | 462.33 chars |
| Prompt Length (Min/Max) | 242 / 1066 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "32187d6f",
"content": "\nSolve the following math problem step by step. Put your answer inside \\boxed{}.\n\nEvery morning Aya goes for a $9$-kilometer-long walk and stops at a coffee shop afterwards. When she walks at a constant speed of $s$ kilometers per hour, the w ... [TRUNCATED 164 chars] ... g $t$ minutes spent in the coffee shop. Suppose Aya walks at $s+\\frac{1}{2}$ kilometers per hour. Find the number of minutes the walk takes her, including the $t$ minutes spent in the coffee shop.\n\nRemember to put your answer inside \\boxed{}."
}
],
"target": "\\boxed{204}",
"id": 0,
"group_id": 0
}
```
## Prompt Template
**Prompt Template:**
```text
Solve the following math problem step by step. Put your answer inside \boxed{{}}.
{question}
Remember to put your answer inside \boxed{{}}.
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets aime24 \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['aime24'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

111
evalscope/docs/en/benchmarks/aime25.md Normal file
View File

@ -0,0 +1,111 @@
# AIME-2025
## Overview
AIME 2025 (American Invitational Mathematics Examination 2025) is a benchmark based on problems from the prestigious AIME competition, one of the most challenging high school mathematics contests in the United States. It tests advanced mathematical reasoning and problem-solving skills.
## Task Description
- **Task Type**: Competition Mathematics Problem Solving
- **Input**: AIME-level mathematical problem
- **Output**: Integer answer (0-999) with step-by-step reasoning
- **Difficulty**: Advanced high school / early undergraduate level
## Key Features
- Problems from AIME I and AIME II 2025 competitions
- Answers are always integers between 0 and 999
- Requires creative mathematical reasoning and problem-solving
- Topics: algebra, geometry, number theory, combinatorics, probability
- Represents top-tier high school mathematics competition difficulty
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Answers should be formatted within `\boxed{}` for proper extraction
- Uses LLM-as-judge for mathematical equivalence checking
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `aime25` |
| **Dataset ID** | [evalscope/aime25](https://modelscope.cn/datasets/evalscope/aime25/summary) |
| **Paper** | N/A |
| **Tags** | `Math`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 30 |
| Prompt Length (Mean) | 604.93 chars |
| Prompt Length (Min/Max) | 208 / 1862 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "bff66863",
"content": "\nSolve the following math problem step by step. Put your answer inside \\boxed{}.\n\nFind the sum of all integer bases $b>9$ for which $17_b$ is a divisor of $97_b.$\n\nRemember to put your answer inside \\boxed{}."
}
],
"target": "70",
"id": 0,
"group_id": 0
}
```
## Prompt Template
**Prompt Template:**
```text
Solve the following math problem step by step. Put your answer inside \boxed{{}}.
{question}
Remember to put your answer inside \boxed{{}}.
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets aime25 \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['aime25'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

111
evalscope/docs/en/benchmarks/aime26.md Normal file
View File

@ -0,0 +1,111 @@
# AIME-2026
## Overview
AIME 2026 (American Invitational Mathematics Examination 2026) is a benchmark based on problems from the prestigious AIME competition, one of the most challenging high school mathematics contests in the United States. It tests advanced mathematical reasoning and problem-solving skills.
## Task Description
- **Task Type**: Competition Mathematics Problem Solving
- **Input**: AIME-level mathematical problem
- **Output**: Integer answer (0-999) with step-by-step reasoning
- **Difficulty**: Advanced high school / early undergraduate level
## Key Features
- Problems from AIME I and AIME II 2026 competitions
- Answers are always integers between 0 and 999
- Requires creative mathematical reasoning and problem-solving
- Topics: algebra, geometry, number theory, combinatorics, probability
- Represents top-tier high school mathematics competition difficulty
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Answers should be formatted within `\boxed{}` for proper extraction
- Uses LLM-as-judge for mathematical equivalence checking
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `aime26` |
| **Dataset ID** | [evalscope/aime26](https://modelscope.cn/datasets/evalscope/aime26/summary) |
| **Paper** | N/A |
| **Tags** | `Math`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 30 |
| Prompt Length (Mean) | 517.7 chars |
| Prompt Length (Min/Max) | 263 / 1026 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "a6d469bc",
"content": "\nSolve the following math problem step by step. Put your answer inside \\boxed{}.\n\nPatrick started walking at a constant rate along a straight road from school to the park. One hour after Patrick left, Tanya started running along the same road ... [TRUNCATED 270 chars] ... ya ran, and all three arrived at the park at the same time. The distance from the school to the park is $\\frac{m}{n}$ miles, where $m$ and $n$ are relatively prime positive integers. Find $m + n$.\n\nRemember to put your answer inside \\boxed{}."
}
],
"target": "277",
"id": 0,
"group_id": 0
}
```
## Prompt Template
**Prompt Template:**
```text
Solve the following math problem step by step. Put your answer inside \boxed{{}}.
{question}
Remember to put your answer inside \boxed{{}}.
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets aime26 \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['aime26'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

175
evalscope/docs/en/benchmarks/air_bench_chat.md Normal file
View File

@ -0,0 +1,175 @@
# AIR-Bench-Chat
## Overview
AIR-Bench Chat is the generative half of [AIR-Bench](https://arxiv.org/abs/2402.07729) (Audio InstRuction Benchmark, ACL 2024 main conference) — the first instruction-following benchmark for large audio-language models (LALMs), covering **human speech, natural sounds and music**. It contains roughly 2k open-ended audio QA pairs covering speech, sound, music and mixed-audio scenes; responses are graded by a GPT-4 judge against a reference answer.
## Task Description
- **Task Type**: Open-ended audio question answering.
- **Input**: An audio clip plus a free-form question.
- **Output**: A textual answer evaluated against the reference response.
## Categories (8 tasks → 5 reported categories)
The 8 Chat tasks are aggregated by the official `cal_score.py` into five categories:
- `speech`: `speech_QA`, `speech_dialogue_QA`
- `sound`: `sound_QA`, `sound_generation_QA`
- `music`: `music_QA`, `music_generation_analysis_QA`
- `speech_and_sound`: `speech_and_sound_QA`
- `speech_and_music`: `speech_and_music_QA`
The paper's **Mixed-audio = mean(speech_and_sound, speech_and_music)**.
## Dataset Access
- The dataset is hosted on ModelScope: [`evalscope/AIR-Bench-Dataset`](https://modelscope.cn/datasets/evalscope/AIR-Bench-Dataset). It uses an *audiofolder + JSON metadata* layout. evalscope downloads it lazily via `modelscope.dataset_snapshot_download` on first run; the full release is ~49 GB, so it is recommended to limit which tasks are pulled via `extra_params`.
- If the dataset is already on disk, pass `dataset_args={'air_bench_chat': {'local_path': '/path/to/AIR-Bench-Dataset'}}`; the local root should contain `Chat/`.
## Evaluation Protocol
- The judge LLM (default: GPT-4) receives the question, the textual audio description (`meta_info` from the dataset), the reference answer (`answer_gt`), and the model's response. It outputs a single line with two integer scores in `[1, 10]`.
- To remove position bias, every sample is judged twice with the order of reference and prediction swapped, then averaged. This mirrors `cal_score.py` in the official repository — disable it via `extra_params={'do_swap': False}` to halve judge cost.
- Reported metric `gpt_score` is the model's mean judge score; `win_rate` records how often the model strictly beats the reference.
```{warning}
The official leaderboard uses `gpt-4-0125-preview` as the judge model. If that exact snapshot is unavailable, use an available GPT-4-class judge; absolute scores can drift versus the published numbers because the judge model changed.
```
## Implementation Notes
- The judge model is selected via `--judge-model-args`; ensure the model id supports long contexts (`meta_info` may exceed 4k tokens for dialogue tasks).
- Set `extra_params={'tasks': [...]}` to evaluate only specific Chat task names — useful for partial runs.
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `air_bench_chat` |
| **Dataset ID** | [evalscope/AIR-Bench](https://modelscope.cn/datasets/evalscope/AIR-Bench/summary) |
| **Paper** | [Paper](https://aclanthology.org/2024.acl-long.109/) |
| **Tags** | `Audio`, `InstructionFollowing`, `QA` |
| **Metrics** | `gpt_score`, `win_rate` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 2,200 |
| Prompt Length (Mean) | 83.89 chars |
| Prompt Length (Min/Max) | 17 / 423 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `speech_QA` | 400 | 64.33 | 23 | 148 |
| `speech_dialogue_QA` | 400 | 77.03 | 29 | 206 |
| `sound_QA` | 400 | 73.29 | 17 | 166 |
| `sound_generation_QA` | 100 | 222.52 | 130 | 423 |
| `music_QA` | 400 | 57.54 | 24 | 202 |
| `music_generation_analysis_QA` | 100 | 267.52 | 148 | 395 |
| `speech_and_sound_QA` | 200 | 63.98 | 25 | 127 |
| `speech_and_music_QA` | 200 | 69.37 | 32 | 127 |
**Audio Statistics:**
| Metric | Value |
|--------|-------|
| Total Audio Files | 2,200 |
| Audio per Sample | min: 1, max: 1, mean: 1 |
| Formats | mp3, wav |
## Sample Example
**Subset**: `speech_QA`
```json
{
"input": [
{
"id": "5781ee73",
"content": [
{
"audio": "/root/.cache/modelscope/hub/datasets/evalscope/AIR-Bench-Dataset/Chat/speech_QA_iemocap/Ses01F_script01_1_M025.wav",
"format": "wav"
},
{
"text": "Who is the speaker addressing at the end of the speech?"
}
]
}
],
"target": "The speaker is addressing Mom at the end of the speech.",
"id": 0,
"group_id": 0,
"subset_key": "speech_QA",
"metadata": {
"uniq_id": 400,
"task_name": "speech_QA",
"dataset_name": "iemocap",
"category": "speech",
"meta_info": "{'emotion': 'neutral', 'gender': 'male', 'transcription': \"And then we'll thrash it out with father. Okay Mom? Don't avoid me.\"}",
"question": "Who is the speaker addressing at the end of the speech?"
}
}
```
## Prompt Template
**Prompt Template:**
```text
{question}
```
## Extra Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `tasks` | `list` | `None` | Optional list of Chat task names to evaluate (subset of ['music_QA', 'music_generation_analysis_QA', 'sound_QA', 'sound_generation_QA', 'speech_QA', 'speech_and_music_QA', 'speech_and_sound_QA', 'speech_dialogue_QA']). Defaults to all tasks. |
| `do_swap` | `bool` | `True` | When True (default), each sample is judged twice with the order of reference vs. prediction swapped, then scores are averaged. Disable to halve judge cost at the price of position bias. |
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets air_bench_chat \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['air_bench_chat'],
dataset_args={
'air_bench_chat': {
# subset_list: ['speech_QA', 'speech_dialogue_QA', 'sound_QA'] # optional, evaluate specific subsets
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

193
evalscope/docs/en/benchmarks/air_bench_foundation.md Normal file
View File

@ -0,0 +1,193 @@
# AIR-Bench-Foundation
## Overview
AIR-Bench Foundation is the discriminative half of [AIR-Bench](https://arxiv.org/abs/2402.07729) (Audio InstRuction Benchmark, ACL 2024 main conference) — the first instruction-following benchmark for large audio-language models (LALMs), covering **human speech, natural sounds and music**. The Foundation track contains roughly 25k single-choice questions spanning 19 logical tasks across three audio categories.
## Task Description
- **Task Type**: Single-choice question answering grounded on an audio clip.
- **Input**: One audio clip + a question with up to four candidate answers (A/B/C/D).
- **Output**: A single letter chosen from the provided options.
## Categories (19 tasks / 25 source-dataset subsets)
- **Speech** (11 dirs / 9 tasks): speech grounding, language ID, gender, emotion (IEMOCAP+MELD), age, speech entity recognition, intent classification, speaker counting, synthesized-voice detection.
- **Sound** (6 dirs / 4 tasks): audio grounding, vocal sound classification, acoustic scene classification (CochlScene+TUT2017), sound QA (avqa+clothoaqa).
- **Music** (8 dirs / 6 tasks): instruments, genre, MIDI pitch, MIDI velocity, music QA, music emotion.
## Prompt Template (matches official `Inference_Foundation.py`)
```text
Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.
{question}
A. {choice_a}
B. {choice_b}
C. {choice_c}
D. {choice_d}
```
## Dataset Access
- The dataset is hosted on ModelScope: [`evalscope/AIR-Bench-Dataset`](https://modelscope.cn/datasets/evalscope/AIR-Bench-Dataset). It uses an *audiofolder + JSON metadata* layout. evalscope downloads it lazily via `modelscope.dataset_snapshot_download` on first run; the full release is ~49 GB, so it is recommended to limit which subsets are pulled via `extra_params`.
- If the dataset is already on disk, pass `dataset_args={'air_bench_foundation': {'local_path': '/path/to/AIR-Bench-Dataset'}}`; the local root should contain `Foundation/`.
- Some Foundation samples are FLAC. For OpenAI-compatible audio input evalscope converts them to cached WAV files, which requires either `soundfile` (`pip install "evalscope[air_bench]"`) or a working `ffmpeg` binary.
## Evaluation Notes
- Metric: **accuracy** (per source-dataset subset, plus per-category aggregation).
- Default prompt follows the official `Inference_Foundation.py` formatting so existing AIR-Bench leaderboard numbers can be reproduced.
- Set `extra_params={'subsets': [...]}` to limit to a subset of the 25 source directories — useful for partial downloads.
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `air_bench_foundation` |
| **Dataset ID** | [evalscope/AIR-Bench](https://modelscope.cn/datasets/evalscope/AIR-Bench/summary) |
| **Paper** | [Paper](https://aclanthology.org/2024.acl-long.109/) |
| **Tags** | `Audio`, `Knowledge`, `MCQ` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 21,426 |
| Prompt Length (Mean) | 236.68 chars |
| Prompt Length (Min/Max) | 179 / 321 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `Speaker_Age_Prediction_common_voice_13.0_en` | 1,000 | 291.98 | 278 | 305 |
| `Speaker_Emotion_Recontion_iemocap` | 1,000 | 227.13 | 218 | 238 |
| `Speaker_Emotion_Recontion_meld` | 1,000 | 233.21 | 218 | 250 |
| `Speaker_Gender_Recognition_common_voice_13_en` | 780 | 226.19 | 213 | 241 |
| `Speaker_Gender_Recognition_meld` | 1,000 | 226.42 | 213 | 241 |
| `Speaker_Intent_Classification_slurp` | 662 | 268.52 | 232 | 295 |
| `Speaker_Number_Verification_voxceleb1` | 314 | 208.24 | 194 | 221 |
| `Speech_Entity_Reconition_slurp` | 1,000 | 253.14 | 226 | 316 |
| `Speech_Grounding_librispeech` | 981 | 253.92 | 230 | 282 |
| `Spoken_Language_Identification_covost2` | 495 | 207.12 | 191 | 232 |
| `Synthesized_Voice_Detection_fake_or_real` | 1,000 | 224.79 | 203 | 242 |
| `Acoustic_Scene_Classification_CochlScene` | 1,000 | 240.97 | 213 | 278 |
| `Acoustic_Scene_Classification_TUT2017` | 1,000 | 241.72 | 207 | 284 |
| `Audio_Grounding_AudioGrounding` | 896 | 273.74 | 249 | 321 |
| `Sound_AQA_avqa` | 1,000 | 227.21 | 193 | 298 |
| `Sound_AQA_clothoaqa` | 1,000 | 199.89 | 179 | 262 |
| `vocal_sound_classification_VocalSound` | 985 | 232.62 | 210 | 253 |
| `Music_AQA_music_avqa` | 814 | 208.7 | 185 | 238 |
| `Music_Genre_Recognition_MTJ-Jamendo` | 1,000 | 223.84 | 200 | 248 |
| `Music_Genre_Recognition_fma` | 1,000 | 224.59 | 201 | 250 |
| `Music_Instruments_Classfication_MTJ-Jamendo` | 1,000 | 236.52 | 218 | 262 |
| `Music_Instruments_Classfication_nsynth` | 996 | 228.12 | 216 | 247 |
| `Music_Midi_Pitch_Analysis_nsynth` | 493 | 253.88 | 243 | 264 |
| `Music_Midi_Velocity_Analysis_nsynth` | 484 | 270.6 | 259 | 279 |
| `Music_Mood_Recognition_MTJ-Jamendo` | 526 | 229.67 | 210 | 248 |
**Audio Statistics:**
| Metric | Value |
|--------|-------|
| Total Audio Files | 21,426 |
| Audio per Sample | min: 1, max: 1, mean: 1 |
| Formats | mp3, wav |
## Sample Example
**Subset**: `Speaker_Age_Prediction_common_voice_13.0_en`
```json
{
"input": [
{
"id": "544443c0",
"content": [
{
"audio": "[BASE64_AUDIO: mp3, ~25.9KB]",
"format": "mp3"
},
{
"text": "Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\nWhich age range do you believe best matches the speaker's voice?\nA. teens to twenties\nB. thirties to fourties\nC. fifties to sixties\nD. seventies to eighties"
}
]
}
],
"target": "B",
"id": 0,
"group_id": 0,
"subset_key": "Speaker_Age_Prediction_common_voice_13.0_en",
"metadata": {
"uniq_id": 5973,
"task_name": "Speaker_Age_Prediction",
"dataset_name": "common_voice_13.0_en",
"category": "speech",
"answer_gt_text": "thirties to fourties",
"choices": {
"A": "teens to twenties",
"B": "thirties to fourties",
"C": "fifties to sixties",
"D": "seventies to eighties"
}
}
}
```
## Prompt Template
**Prompt Template:**
```text
Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.
```
## Extra Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `subsets` | `list` | `None` | Optional list of Foundation source-dataset directories to evaluate. Defaults to all 25 directories. Useful when only a subset has been downloaded locally. |
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets air_bench_foundation \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['air_bench_foundation'],
dataset_args={
'air_bench_foundation': {
# subset_list: ['Speaker_Age_Prediction_common_voice_13.0_en', 'Speaker_Emotion_Recontion_iemocap', 'Speaker_Emotion_Recontion_meld'] # optional, evaluate specific subsets
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

114
evalscope/docs/en/benchmarks/alpaca_eval.md Normal file
View File

@ -0,0 +1,114 @@
# AlpacaEval2.0
## Overview
AlpacaEval 2.0 is an evaluation framework for instruction-following language models that uses an LLM judge to compare model outputs against a strong baseline. It provides win-rate metrics reflecting human preferences.
## Task Description
- **Task Type**: Instruction-Following Evaluation (Pairwise Comparison)
- **Input**: User instruction/question
- **Output**: Model response compared against GPT-4 Turbo baseline
- **Metric**: Win rate against baseline model
## Key Features
- Auto-annotator for scalable evaluation
- Compares against GPT-4 Turbo baseline outputs
- High correlation with human preferences
- Cost-effective evaluation method
- Tests general instruction-following capabilities
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Uses LLM judge (default: gpt-4-1106-preview)
- Baseline model: gpt-4-turbo outputs
- Reports win rate metric
- Note: Length-controlled win rate not currently supported
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `alpaca_eval` |
| **Dataset ID** | [AI-ModelScope/alpaca_eval](https://modelscope.cn/datasets/AI-ModelScope/alpaca_eval/summary) |
| **Paper** | N/A |
| **Tags** | `Arena`, `InstructionFollowing` |
| **Metrics** | `winrate` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `eval` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 805 |
| Prompt Length (Mean) | 164.92 chars |
| Prompt Length (Min/Max) | 12 / 1917 chars |
## Sample Example
**Subset**: `alpaca_eval_gpt4_baseline`
```json
{
"input": [
{
"id": "95236545",
"content": "What are the names of some famous actors that started their careers on Broadway?"
}
],
"target": "Several famous actors started their careers on Broadway before making it big in film and television. Here are a few notable examples:\n\n1. Sarah Jessica Parker - Before she was Carrie Bradshaw on \"Sex and the City,\" Sarah Jessica Parker was a ... [TRUNCATED] ... f the many performers who have transitioned from the Broadway stage to broader fame in the entertainment industry. Broadway often serves as a proving ground for talent, and many actors continue to return to the stage throughout their careers.",
"id": 0,
"group_id": 0,
"metadata": {
"generator": "gpt4_1106_preview",
"dataset": "helpful_base"
}
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
{question}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets alpaca_eval \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['alpaca_eval'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

127
evalscope/docs/en/benchmarks/amc.md Normal file
View File

@ -0,0 +1,127 @@
# AMC
## Overview
AMC (American Mathematics Competitions) is a benchmark based on problems from the AMC 10/12 competitions from 2022-2024. These multiple-choice problems test mathematical problem-solving skills at the high school level and serve as qualifiers for the AIME competition.
## Task Description
- **Task Type**: Competition Mathematics (Multiple Choice)
- **Input**: AMC-level mathematical problem
- **Output**: Correct answer with step-by-step reasoning
- **Years Covered**: 2022, 2023, 2024
## Key Features
- Problems from AMC 10 and AMC 12 competitions (2022-2024)
- Multiple-choice format with 5 answer options
- Topics: algebra, geometry, number theory, combinatorics
- Difficulty ranges from accessible to challenging
- Official competition problems with verified solutions
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Answers should be formatted within `\boxed{}` for proper extraction
- Three subsets available: `amc22`, `amc23`, `amc24`
- Problems include original URLs for reference
- Solutions available in metadata for verification
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `amc` |
| **Dataset ID** | [evalscope/amc_22-24](https://modelscope.cn/datasets/evalscope/amc_22-24/summary) |
| **Paper** | N/A |
| **Tags** | `Math`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `N/A` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 134 |
| Prompt Length (Mean) | 324.58 chars |
| Prompt Length (Min/Max) | 98 / 1218 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `amc22` | 43 | 337.42 | 129 | 934 |
| `amc23` | 46 | 337.98 | 143 | 1218 |
| `amc24` | 45 | 298.62 | 98 | 882 |
## Sample Example
**Subset**: `amc22`
```json
{
"input": [
{
"id": "54851a8f",
"content": "What is the value of\\[3+\\frac{1}{3+\\frac{1}{3+\\frac13}}?\\]\nPlease reason step by step, and put your final answer within \\boxed{}."
}
],
"target": "\\frac{109}{33}",
"id": 0,
"group_id": 0,
"metadata": {
"year": 2022,
"url": "https://artofproblemsolving.com/wiki/index.php/2022_AMC_12A_Problems/Problem_1",
"solution": "We have\\begin{align*} 3+\\frac{1}{3+\\frac{1}{3+\\frac13}} &= 3+\\frac{1}{3+\\frac{1}{\\left(\\frac{10}{3}\\right)}} \\\\ &= 3+\\frac{1}{3+\\frac{3}{10}} \\\\ &= 3+\\frac{1}{\\left(\\frac{33}{10}\\right)} \\\\ &= 3+\\frac{10}{33} \\\\ &= \\boxed{\\textbf{(D)}\\ \\frac{109}{33}}. \\end{align*}"
}
}
```
## Prompt Template
**Prompt Template:**
```text
{question}
Please reason step by step, and put your final answer within \boxed{{}}.
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets amc \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['amc'],
dataset_args={
'amc': {
# subset_list: ['amc22', 'amc23', 'amc24'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

190
evalscope/docs/en/benchmarks/anat_em.md Normal file
View File

@ -0,0 +1,190 @@
# AnatEM
## Overview
The AnatEM corpus is an extensive resource for anatomical entity recognition, created by extending and combining previous corpora. It includes over 13,000 annotations across 1,212 biomedical documents, focusing on identifying anatomical structures from subcellular components to organ systems.
## Task Description
- **Task Type**: Biomedical Named Entity Recognition (NER)
- **Input**: Biomedical text from PubMed abstracts
- **Output**: Identified anatomical entity spans with types
- **Domain**: Anatomy, biomedical literature
## Key Features
- Over 13,000 anatomical entity annotations
- 1,212 biomedical documents from PubMed
- Comprehensive anatomical coverage (cells to organs)
- Manual expert annotation
- Useful for biomedical text mining
## Evaluation Notes
- Default configuration uses **5-shot** evaluation
- Metrics: Precision, Recall, F1-Score, Accuracy
- Entity types: ANATOMY (subcellular structures, cells, tissues, organs)
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `anat_em` |
| **Dataset ID** | [extraordinarylab/anat-em](https://modelscope.cn/datasets/extraordinarylab/anat-em/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `NER` |
| **Metrics** | `precision`, `recall`, `f1_score`, `accuracy` |
| **Default Shots** | 5-shot |
| **Evaluation Split** | `test` |
| **Train Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 3,830 |
| Prompt Length (Mean) | 3007.08 chars |
| Prompt Length (Min/Max) | 2861 / 3652 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "64b20a23",
"content": "Here are some examples of named entity recognition:\n\nInput:\nImmunostaining and confocal analysis\n\nOutput:\n<response>Immunostaining and confocal analysis</response>\n\nInput:\nDNA labelling and staining with 5 - bromo - 2 ' - deoxyuridine ( BrdU ... [TRUNCATED] ... Do not include explanations, just the tagged text.\n6. If entity spans overlap, choose the most specific entity type.\n7. Ensure every opening tag has a matching closing tag.\n\nText to process:\n( a ) Schematic drawing of the magnetic tweezers .\n"
}
],
"target": "<response>( a ) Schematic drawing of the magnetic tweezers .</response>",
"id": 0,
"group_id": 0,
"metadata": {
"tokens": [
"(",
"a",
")",
"Schematic",
"drawing",
"of",
"the",
"magnetic",
"tweezers",
"."
],
"ner_tags": [
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O"
]
}
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
<details>
<summary>Few-shot Template</summary>
```text
Here are some examples of named entity recognition:
{fewshot}
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
</details>
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets anat_em \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['anat_em'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

133
evalscope/docs/en/benchmarks/arc.md Normal file
View File

@ -0,0 +1,133 @@
# ARC
## Overview
ARC (AI2 Reasoning Challenge) is a benchmark designed to evaluate science question answering capabilities of AI models. It consists of multiple-choice science questions from grade 3 to grade 9, divided into an Easy set and a Challenge set based on difficulty.
## Task Description
- **Task Type**: Multiple-Choice Science Question Answering
- **Input**: Science question with 3-5 answer choices
- **Output**: Correct answer letter (A, B, C, D, or E)
- **Difficulty Levels**: ARC-Easy and ARC-Challenge
## Key Features
- 7,787 science questions from standardized tests (grades 3-9)
- ARC-Easy: Questions answerable by retrieval or word co-occurrence
- ARC-Challenge: Questions requiring deeper reasoning
- Questions cover physics, chemistry, biology, and earth science
- Designed to test both factual knowledge and reasoning
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Two subsets available: `ARC-Easy` and `ARC-Challenge`
- Challenge set is commonly used for leaderboard comparisons
- Supports few-shot evaluation with train split examples
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `arc` |
| **Dataset ID** | [allenai/ai2_arc](https://modelscope.cn/datasets/allenai/ai2_arc/summary) |
| **Paper** | N/A |
| **Tags** | `MCQ`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
| **Train Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 3,548 |
| Prompt Length (Mean) | 424.43 chars |
| Prompt Length (Min/Max) | 253 / 1157 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `ARC-Easy` | 2,376 | 409.9 | 253 | 1157 |
| `ARC-Challenge` | 1,172 | 453.91 | 253 | 1111 |
## Sample Example
**Subset**: `ARC-Easy`
```json
{
"input": [
{
"id": "76edd7a5",
"content": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nWhich statement best explains why photosynthesis is the foundation of most food webs?\n\nA) Sunlight is the source of energy for nearly all ecosystems.\nB) Most ecosystems are found on land instead of in water.\nC) Carbon dioxide is more available than other gases.\nD) The producers in all ecosystems are plants."
}
],
"choices": [
"Sunlight is the source of energy for nearly all ecosystems.",
"Most ecosystems are found on land instead of in water.",
"Carbon dioxide is more available than other gases.",
"The producers in all ecosystems are plants."
],
"target": "A",
"id": 0,
"group_id": 0,
"metadata": {
"id": "Mercury_417466"
}
}
```
## Prompt Template
**Prompt Template:**
```text
Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.
{question}
{choices}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets arc \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['arc'],
dataset_args={
'arc': {
# subset_list: ['ARC-Easy', 'ARC-Challenge'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

112
evalscope/docs/en/benchmarks/arena_hard.md Normal file
View File

@ -0,0 +1,112 @@
# ArenaHard
## Overview
ArenaHard is a challenging benchmark that evaluates language models through competitive pairwise comparison. Models are judged against a GPT-4 baseline on difficult tasks requiring reasoning, understanding, and generation capabilities.
## Task Description
- **Task Type**: Competitive Model Evaluation (Arena-style)
- **Input**: Challenging instruction/question
- **Output**: Model response compared against GPT-4-0314 baseline
- **Scoring**: Elo-based rating from pairwise battles
## Key Features
- 500 challenging user prompts
- Two-game battle system (A vs B and B vs A)
- Elo rating calculation for model ranking
- Tests reasoning, instruction-following, and generation
- High correlation with Chatbot Arena rankings
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Uses LLM judge (default: gpt-4-1106-preview)
- Baseline model: gpt-4-0314 outputs
- Reports win rate and Elo-based scores
- Note: Style-controlled win rate not currently supported
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `arena_hard` |
| **Dataset ID** | [AI-ModelScope/arena-hard-auto-v0.1](https://modelscope.cn/datasets/AI-ModelScope/arena-hard-auto-v0.1/summary) |
| **Paper** | N/A |
| **Tags** | `Arena`, `InstructionFollowing` |
| **Metrics** | `winrate` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
| **Aggregation** | `elo` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 500 |
| Prompt Length (Mean) | 406.36 chars |
| Prompt Length (Min/Max) | 29 / 9140 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "088243c7",
"content": "Use ABC notation to write a melody in the style of a folk tune."
}
],
"target": "X:1\nT:Untitled Folk Tune\nM:4/4\nL:1/8\nK:G\n|:G2A2|B2A2|G2E2|D4|E2F2|G2F2|E2C2|B,4|\nA2B2|c2B2|A2F2|E4|D2E2|F2E2|D2B,2|C4:|",
"id": 0,
"group_id": 0,
"metadata": {
"capability": "ABC Sequence Puzzles & Groups"
}
}
```
## Prompt Template
**Prompt Template:**
```text
{question}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets arena_hard \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['arena_hard'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

131
evalscope/docs/en/benchmarks/arxivmath.md Normal file
View File

@ -0,0 +1,131 @@
# ArXiv-Math
## Overview
ArXiv-Math is a benchmark of 103 research-level mathematics problems extracted from arXiv preprints. These problems represent cutting-edge mathematical research and test the ability of language models to reason about advanced mathematical concepts at the frontier of knowledge.
## Task Description
- **Task Type**: Research-Level Mathematics Problem Solving
- **Input**: Advanced mathematical problem from arXiv papers
- **Output**: Step-by-step solution with final answer
- **Difficulty**: Research / graduate level
## Key Features
- 103 problems sourced from arXiv preprints (December 2024 - March 2025)
- Four monthly subsets: december, february, january, march
- Covers diverse areas: algebra, combinatorics, analysis, geometry, number theory
- Problems require deep mathematical reasoning and domain expertise
- Represents the frontier of mathematical research difficulty
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Answers should be formatted within `\boxed{}` for proper extraction
- Numeric accuracy metric with symbolic equivalence checking
- Results can be broken down by monthly competition subset
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `arxivmath` |
| **Dataset ID** | [evalscope/arxivmath](https://modelscope.cn/datasets/evalscope/arxivmath/summary) |
| **Paper** | N/A |
| **Tags** | `Math`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 103 |
| Prompt Length (Mean) | 622.88 chars |
| Prompt Length (Min/Max) | 224 / 1392 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `arxiv/december` | 17 | 720.88 | 256 | 1392 |
| `arxiv/february` | 32 | 573.78 | 269 | 1147 |
| `arxiv/january` | 23 | 711.17 | 325 | 1270 |
| `arxiv/march` | 31 | 554.32 | 224 | 1213 |
## Sample Example
**Subset**: `arxiv/december`
```json
{
"input": [
{
"id": "c7cbf85d",
"content": "Problem:\nLet $k$ be a field, let $V$ be a $k$-vector space of dimension $d$, and let $G\\subseteq GL(V)$ be a finite group. Set $r:=\\dim_k (V^*)^G$ and assume $r\\ge 1$. Let $R:=k[V]^G$ be the invariant ring, and write its Hilbert quasi-polynom ... [TRUNCATED 71 chars] ... {d-2}+\\cdots+a_1(n)n+a_0(n),\n\\]\nwhere each $a_i(n)$ is a periodic function of $n$. Compute the sum of the indices $i\\in\\{0,1,\\dots,d-1\\}$ for which $a_i(n)$ is constant.\n\nPlease reason step by step, and put your final answer within \\boxed{}.\n"
}
],
"target": "\\frac{r(2d-r-1)}{2}",
"id": 0,
"group_id": 0,
"subset_key": "arxiv/december",
"metadata": {
"problem_idx": 1,
"problem_type": [
""
],
"source": 2512.00811
}
}
```
## Prompt Template
**Prompt Template:**
```text
Problem:
{question}
Please reason step by step, and put your final answer within \boxed{{}}.
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets arxivmath \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['arxivmath'],
dataset_args={
'arxivmath': {
# subset_list: ['arxiv/december', 'arxiv/february', 'arxiv/january'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

203
evalscope/docs/en/benchmarks/arxivrollbench.md Normal file
View File

@ -0,0 +1,203 @@
# ArxivRollBench
## Overview
ArxivRollBench is a rolling benchmark built from recent arXiv papers. It evaluates whether large language models can reason over fresh scientific text through three task formats: sequencing, cloze, and next-fragment prediction.
## Task Description
- **Task Type**: Multiple-choice scientific text reasoning
- **Input**: Recent arXiv text fragments with four answer choices
- **Output**: Single correct answer letter (A, B, C, or D)
- **Domains**: Computer Science, Quantitative Finance, Mathematics, Physics, Statistics, Quantitative Biology, Economics, and Electrical Engineering/System Science
- **Releases**: 2024b, 2025a, and 2026a rolling snapshots
## Key Features
- Time-aware benchmark snapshots reduce contamination-related overestimation
- Covers multiple arXiv domains and scientific writing styles
- Includes sequencing, cloze, and prediction formats under the SCP framework
- Compact `-50` split is suitable for cost-controlled API evaluation
- Full split is available as `arxivrollbench_full`
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- The default `arxivrollbench` benchmark uses compact `-50` datasets
- Use `arxivrollbench_full` for the complete public splits
- Each subset is loaded from the public ModelScope mirror under the `liangzid` namespace
- Answers are normalized to A-D and evaluated with accuracy
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `arxivrollbench` |
| **Dataset ID** | [liangzid/arxivrollbench](https://modelscope.cn/datasets/liangzid/arxivrollbench/summary) |
| **Paper** | [Paper](https://ojs.aaai.org/index.php/AAAI/article/view/41098) |
| **Tags** | `Knowledge`, `MCQ`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 3,254 |
| Prompt Length (Mean) | 1514.19 chars |
| Prompt Length (Min/Max) | 307 / 14112 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `2024b_cs_s` | 42 | 949.6 | 590 | 1805 |
| `2024b_cs_c` | 31 | 307 | 307 | 307 |
| `2024b_cs_p` | 50 | 2617.32 | 922 | 7512 |
| `2024b_q_fin_s` | 49 | 1042.31 | 586 | 2329 |
| `2024b_q_fin_c` | 44 | 307 | 307 | 307 |
| `2024b_q_fin_p` | 50 | 3430.52 | 872 | 9106 |
| `2024b_math_s` | 34 | 829.85 | 593 | 2115 |
| `2024b_math_c` | 15 | 307 | 307 | 307 |
| `2024b_math_p` | 51 | 1957.24 | 869 | 6260 |
| `2024b_physics_s` | 45 | 957.11 | 576 | 4402 |
| `2024b_physics_c` | 28 | 307 | 307 | 307 |
| `2024b_physics_p` | 51 | 2948.1 | 885 | 13643 |
| `2024b_stat_s` | 45 | 936.4 | 582 | 1678 |
| `2024b_stat_c` | 33 | 307 | 307 | 307 |
| `2024b_stat_p` | 50 | 2946.44 | 861 | 7026 |
| `2024b_q_bio_s` | 43 | 975 | 583 | 2555 |
| `2024b_q_bio_c` | 34 | 307 | 307 | 307 |
| `2024b_q_bio_p` | 49 | 3354.53 | 883 | 8867 |
| `2024b_econ_s` | 48 | 1021.58 | 586 | 2070 |
| `2024b_econ_c` | 43 | 307 | 307 | 307 |
| `2024b_econ_p` | 50 | 3257.76 | 846 | 8967 |
| `2024b_eess_s` | 48 | 1034.56 | 574 | 2922 |
| `2024b_eess_c` | 42 | 307 | 307 | 307 |
| `2024b_eess_p` | 51 | 2612.69 | 882 | 8609 |
| `2025a_cs_s` | 50 | 921.2 | 592 | 1632 |
| `2025a_cs_c` | 44 | 307 | 307 | 307 |
| `2025a_cs_p` | 51 | 2895.02 | 942 | 6540 |
| `2025a_q_fin_s` | 50 | 931.08 | 589 | 2202 |
| `2025a_q_fin_c` | 43 | 307 | 307 | 307 |
| `2025a_q_fin_p` | 51 | 2837.86 | 793 | 7577 |
| `2025a_math_s` | 42 | 852.52 | 580 | 1595 |
| `2025a_math_c` | 28 | 307 | 307 | 307 |
| `2025a_math_p` | 51 | 2449.49 | 889 | 6893 |
| `2025a_physics_s` | 44 | 939.32 | 587 | 1874 |
| `2025a_physics_c` | 34 | 307 | 307 | 307 |
| `2025a_physics_p` | 49 | 3568.29 | 1001 | 9325 |
| `2025a_stat_s` | 48 | 932.81 | 600 | 2063 |
| `2025a_stat_c` | 42 | 307 | 307 | 307 |
| `2025a_stat_p` | 50 | 3115.36 | 822 | 7349 |
| `2025a_q_bio_s` | 49 | 1074.12 | 591 | 1810 |
| `2025a_q_bio_c` | 49 | 307 | 307 | 307 |
| `2025a_q_bio_p` | 50 | 3639.26 | 1038 | 8890 |
| `2025a_econ_s` | 48 | 982.19 | 591 | 2322 |
| `2025a_econ_c` | 45 | 307 | 307 | 307 |
| `2025a_econ_p` | 51 | 2860.9 | 884 | 6494 |
| `2025a_eess_s` | 46 | 1017.35 | 588 | 1807 |
| `2025a_eess_c` | 42 | 307 | 307 | 307 |
| `2025a_eess_p` | 50 | 3541.1 | 943 | 14112 |
| `2026a_cs_s` | 51 | 944.12 | 584 | 1795 |
| `2026a_cs_c` | 38 | 307 | 307 | 307 |
| `2026a_cs_p` | 51 | 2629.06 | 919 | 5234 |
| `2026a_q_fin_s` | 48 | 1025.44 | 608 | 2320 |
| `2026a_q_fin_c` | 45 | 307 | 307 | 307 |
| `2026a_q_fin_p` | 51 | 3094.78 | 872 | 6644 |
| `2026a_math_s` | 44 | 844.05 | 575 | 1381 |
| `2026a_math_c` | 30 | 307 | 307 | 307 |
| `2026a_math_p` | 51 | 2160.27 | 860 | 12385 |
| `2026a_physics_s` | 47 | 1082.04 | 599 | 2522 |
| `2026a_physics_c` | 41 | 307 | 307 | 307 |
| `2026a_physics_p` | 50 | 3420.58 | 894 | 8788 |
| `2026a_stat_s` | 49 | 1013.47 | 575 | 2482 |
| `2026a_stat_c` | 46 | 307 | 307 | 307 |
| `2026a_stat_p` | 51 | 2564.47 | 955 | 6387 |
| `2026a_q_bio_s` | 47 | 1019.7 | 584 | 1707 |
| `2026a_q_bio_c` | 40 | 307 | 307 | 307 |
| `2026a_q_bio_p` | 48 | 3030.71 | 954 | 6468 |
| `2026a_econ_s` | 48 | 989.67 | 580 | 2320 |
| `2026a_econ_c` | 47 | 307 | 307 | 307 |
| `2026a_econ_p` | 51 | 2920.76 | 885 | 7061 |
| `2026a_eess_s` | 51 | 988.14 | 579 | 2231 |
| `2026a_eess_c` | 45 | 307 | 307 | 307 |
| `2026a_eess_p` | 51 | 2812.61 | 922 | 5589 |
## Sample Example
**Subset**: `2024b_cs_s`
```json
{
"input": [
{
"id": "7b220fd6",
"content": "Answer the following ArxivRollBench multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nSelect the option that correctly compl ... [TRUNCATED 381 chars] ... m a diagonal matrix into the identity, allows us to write the input matrix as a product of transvections. **C**: Note that row and column operations are effected by left- and right multiplications by transvections\n\nA) BAC\nB) ABC\nC) ACB\nD) BCA"
}
],
"choices": [
"BAC",
"ABC",
"ACB",
"BCA"
],
"target": "C",
"id": 0,
"group_id": 0,
"metadata": {
"original_label": "Selection 3",
"task_type": "s/c"
}
}
```
## Prompt Template
**Prompt Template:**
```text
Answer the following ArxivRollBench multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.
{question}
{choices}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets arxivrollbench \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['arxivrollbench'],
dataset_args={
'arxivrollbench': {
# subset_list: ['2024b_cs_s', '2024b_cs_c', '2024b_cs_p'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

203
evalscope/docs/en/benchmarks/arxivrollbench_full.md Normal file
View File

@ -0,0 +1,203 @@
# ArxivRollBench-Full
## Overview
ArxivRollBench is a rolling benchmark built from recent arXiv papers. It evaluates whether large language models can reason over fresh scientific text through three task formats: sequencing, cloze, and next-fragment prediction.
## Task Description
- **Task Type**: Multiple-choice scientific text reasoning
- **Input**: Recent arXiv text fragments with four answer choices
- **Output**: Single correct answer letter (A, B, C, or D)
- **Domains**: Computer Science, Quantitative Finance, Mathematics, Physics, Statistics, Quantitative Biology, Economics, and Electrical Engineering/System Science
- **Releases**: 2024b, 2025a, and 2026a rolling snapshots
## Key Features
- Time-aware benchmark snapshots reduce contamination-related overestimation
- Covers multiple arXiv domains and scientific writing styles
- Includes sequencing, cloze, and prediction formats under the SCP framework
- Compact `-50` split is suitable for cost-controlled API evaluation
- Full split is available as `arxivrollbench_full`
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- The default `arxivrollbench` benchmark uses compact `-50` datasets
- Use `arxivrollbench_full` for the complete public splits
- Each subset is loaded from the public ModelScope mirror under the `liangzid` namespace
- Answers are normalized to A-D and evaluated with accuracy
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `arxivrollbench_full` |
| **Dataset ID** | [liangzid/arxivrollbench-full](https://modelscope.cn/datasets/liangzid/arxivrollbench-full/summary) |
| **Paper** | [Paper](https://ojs.aaai.org/index.php/AAAI/article/view/41098) |
| **Tags** | `Knowledge`, `MCQ`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 245,433 |
| Prompt Length (Mean) | 1499.93 chars |
| Prompt Length (Min/Max) | 307 / 28864 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `2024b_cs_s` | 2,931 | 962.16 | 574 | 4774 |
| `2024b_cs_c` | 2,377 | 307 | 307 | 307 |
| `2024b_cs_p` | 3,166 | 2663.27 | 793 | 10327 |
| `2024b_q_fin_s` | 852 | 1026.01 | 574 | 3549 |
| `2024b_q_fin_c` | 747 | 307 | 307 | 307 |
| `2024b_q_fin_p` | 881 | 3207.96 | 793 | 16189 |
| `2024b_math_s` | 2,107 | 886.2 | 574 | 3466 |
| `2024b_math_c` | 1,238 | 307 | 307 | 307 |
| `2024b_math_p` | 2,532 | 2295.3 | 793 | 11911 |
| `2024b_physics_s` | 1,966 | 984.28 | 575 | 4225 |
| `2024b_physics_c` | 1,482 | 307 | 307 | 307 |
| `2024b_physics_p` | 2,141 | 3166.87 | 793 | 28864 |
| `2024b_stat_s` | 3,482 | 985.03 | 574 | 6098 |
| `2024b_stat_c` | 2,800 | 307 | 307 | 307 |
| `2024b_stat_p` | 3,704 | 3000.94 | 793 | 15321 |
| `2024b_q_bio_s` | 1,485 | 1039.14 | 574 | 3895 |
| `2024b_q_bio_c` | 1,318 | 307 | 307 | 307 |
| `2024b_q_bio_p` | 1,550 | 3332.41 | 804 | 16126 |
| `2024b_econ_s` | 879 | 1023.84 | 576 | 3421 |
| `2024b_econ_c` | 764 | 307 | 307 | 307 |
| `2024b_econ_p` | 919 | 3176.67 | 851 | 15040 |
| `2024b_eess_s` | 3,771 | 1014.36 | 574 | 4356 |
| `2024b_eess_c` | 3,278 | 307 | 307 | 307 |
| `2024b_eess_p` | 3,976 | 3048.85 | 793 | 17290 |
| `2025a_cs_s` | 12,806 | 981.57 | 574 | 5696 |
| `2025a_cs_c` | 11,244 | 307 | 307 | 307 |
| `2025a_cs_p` | 13,331 | 2823.48 | 793 | 20389 |
| `2025a_q_fin_s` | 851 | 1013.21 | 576 | 2609 |
| `2025a_q_fin_c` | 758 | 307 | 307 | 307 |
| `2025a_q_fin_p` | 884 | 3128.37 | 793 | 13025 |
| `2025a_math_s` | 10,362 | 908.79 | 574 | 6001 |
| `2025a_math_c` | 6,344 | 307 | 307 | 307 |
| `2025a_math_p` | 12,145 | 2444.85 | 793 | 12037 |
| `2025a_physics_s` | 10,696 | 1002.06 | 574 | 4761 |
| `2025a_physics_c` | 8,358 | 307 | 307 | 307 |
| `2025a_physics_p` | 11,595 | 3369.68 | 793 | 25245 |
| `2025a_stat_s` | 5,288 | 985.58 | 574 | 8627 |
| `2025a_stat_c` | 4,285 | 307 | 307 | 307 |
| `2025a_stat_p` | 5,589 | 2935.37 | 793 | 15676 |
| `2025a_q_bio_s` | 1,598 | 1043.55 | 574 | 3115 |
| `2025a_q_bio_c` | 1,443 | 307 | 307 | 307 |
| `2025a_q_bio_p` | 1,669 | 3370.82 | 796 | 18074 |
| `2025a_econ_s` | 951 | 998.31 | 574 | 2900 |
| `2025a_econ_c` | 827 | 307 | 307 | 307 |
| `2025a_econ_p` | 982 | 3176.93 | 793 | 11038 |
| `2025a_eess_s` | 8,171 | 1011.86 | 574 | 3844 |
| `2025a_eess_c` | 7,155 | 307 | 307 | 307 |
| `2025a_eess_p` | 8,577 | 3042.87 | 793 | 18934 |
| `2026a_cs_s` | 1,857 | 981.82 | 574 | 3532 |
| `2026a_cs_c` | 1,648 | 307 | 307 | 307 |
| `2026a_cs_p` | 1,933 | 2724.96 | 814 | 11328 |
| `2026a_q_fin_s` | 986 | 985.79 | 574 | 2961 |
| `2026a_q_fin_c` | 886 | 307 | 307 | 307 |
| `2026a_q_fin_p` | 1,046 | 2727.72 | 802 | 10072 |
| `2026a_math_s` | 2,435 | 869.86 | 574 | 3795 |
| `2026a_math_c` | 1,600 | 307 | 307 | 307 |
| `2026a_math_p` | 2,777 | 1953.57 | 808 | 12053 |
| `2026a_physics_s` | 1,863 | 1007.76 | 574 | 3813 |
| `2026a_physics_c` | 1,575 | 307 | 307 | 307 |
| `2026a_physics_p` | 2,019 | 3072.96 | 798 | 13540 |
| `2026a_stat_s` | 3,126 | 964.56 | 574 | 3136 |
| `2026a_stat_c` | 2,627 | 307 | 307 | 307 |
| `2026a_stat_p` | 3,322 | 2549.38 | 814 | 10028 |
| `2026a_q_bio_s` | 1,502 | 1020.61 | 574 | 3281 |
| `2026a_q_bio_c` | 1,373 | 307 | 307 | 307 |
| `2026a_q_bio_p` | 1,569 | 3074.52 | 806 | 11848 |
| `2026a_econ_s` | 914 | 995.97 | 574 | 3043 |
| `2026a_econ_c` | 828 | 307 | 307 | 307 |
| `2026a_econ_p` | 973 | 2858.55 | 818 | 11577 |
| `2026a_eess_s` | 4,200 | 1006.27 | 574 | 3698 |
| `2026a_eess_c` | 3,710 | 307 | 307 | 307 |
| `2026a_eess_p` | 4,409 | 2790.21 | 817 | 13794 |
## Sample Example
**Subset**: `2024b_cs_s`
```json
{
"input": [
{
"id": "509c2daa",
"content": "Answer the following ArxivRollBench multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nSelect the option that correctly compl ... [TRUNCATED 283 chars] ... rators can be used directly to verify representations of classical groups [12].\n**C**: In practice it is the generating set produced by the constructive recognition algorithms from [10, 11] as implemented in MAGMA\n\nA) CAB\nB) ACB\nC) BAC\nD) CAB"
}
],
"choices": [
"CAB",
"ACB",
"BAC",
"CAB"
],
"target": "B",
"id": 0,
"group_id": 0,
"metadata": {
"original_label": "Selection 2",
"task_type": "s/c"
}
}
```
## Prompt Template
**Prompt Template:**
```text
Answer the following ArxivRollBench multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.
{question}
{choices}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets arxivrollbench_full \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['arxivrollbench_full'],
dataset_args={
'arxivrollbench_full': {
# subset_list: ['2024b_cs_s', '2024b_cs_c', '2024b_cs_p'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

148
evalscope/docs/en/benchmarks/baby_vision.md Normal file
View File

@ -0,0 +1,148 @@
# BabyVision
## Overview
BabyVision is a visual perception benchmark that evaluates the fundamental visual abilities of
multimodal large language models through tasks inspired by infant and early childhood visual
development. It focuses on fine-grained discrimination, spatial perception, visual pattern
recognition, and visual tracking.
## Task Description
- **Task Type**: Visual Perception (Choice + Fill-in-the-blank)
- **Input**: Image + question
- **Output**: Choice letter or free-form short answer
- **Domains**: Fine-grained discrimination, spatial perception, visual pattern recognition, visual tracking
## Key Features
- 388 test samples across 4 major visual ability categories and 22 subtypes
- Two answer types: choice (135 samples) and blank (253 samples)
- Subtypes include: Find the different, Find the same, Count clusters, Maze,
3D cube unfold, Pattern completion, Paper folding, Rotation patterns, etc.
- Tests low-level visual perception rather than high-level reasoning or knowledge
- Includes Chain-of-Thought (CoT) reference for analysis
## Evaluation Notes
- Default evaluation uses the **train** split (388 samples, single split dataset)
- Primary metric: **Accuracy** via LLM-as-judge
- Subsets organized by `type` field (4 categories)
- LLM judge evaluates both choice and blank answer types uniformly
- Requires `judge_model_args` configuration for LLM judge
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `baby_vision` |
| **Dataset ID** | [evalscope/BabyVision](https://modelscope.cn/datasets/evalscope/BabyVision/summary) |
| **Paper** | N/A |
| **Tags** | `MultiModal`, `QA`, `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 388 |
| Prompt Length (Mean) | 167.37 chars |
| Prompt Length (Min/Max) | 33 / 450 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `Fine-grained Discrimination` | 163 | 152.09 | 33 | 450 |
| `Spatial Perception` | 91 | 157.18 | 73 | 370 |
| `Visual Pattern Recognition` | 51 | 178.92 | 94 | 319 |
| `Visual Tracking` | 83 | 201.45 | 97 | 389 |
**Image Statistics:**
| Metric | Value |
|--------|-------|
| Total Images | 388 |
| Images per Sample | min: 1, max: 1, mean: 1 |
| Resolution Range | 174x144 - 2378x1448 |
| Formats | jpeg, png, webp |
## Sample Example
**Subset**: `Fine-grained Discrimination`
```json
{
"input": [
{
"id": "8b83904b",
"content": [
{
"image": "[BASE64_IMAGE: jpeg, ~77.0KB]"
},
{
"text": "The image shows a total of 49 tiger patterns arranged in 7 rows and 7 columns. One of them is different from the others. Which row and column is it in? The answer format is (x,y). (For example, the answer for the 2nd row and 3rd column is (2,3))."
}
]
}
],
"target": "(4,7)",
"id": 0,
"group_id": 0,
"subset_key": "Fine-grained Discrimination",
"metadata": {
"taskId": 445,
"type": "Fine-grained Discrimination",
"subtype": "Find the different",
"ansType": "blank",
"coT": "The image shows 49 tiger patterns arranged in 7 rows and 7 columns.\nNow, we need to find the coordinates of the one tiger pattern that is different from the other 48.\nIt can be observed that the tiger in the fourth row and seventh column has no ears (the ears are located in the upper right corner of each tiger pattern), while the other 48 tigers have ears.\nTherefore, the correct answer is (4,7)."
}
}
```
## Prompt Template
*No prompt template defined.*
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets baby_vision \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['baby_vision'],
dataset_args={
'baby_vision': {
# subset_list: ['Fine-grained Discrimination', 'Spatial Perception', 'Visual Pattern Recognition'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

166
evalscope/docs/en/benchmarks/bbh.md Normal file
View File

@ -0,0 +1,166 @@
# BBH
## Overview
BBH (BIG-Bench Hard) is a subset of 23 challenging tasks from the BIG-Bench benchmark that are specifically selected because language models initially struggled with them. These tasks require complex reasoning abilities that benefit from Chain-of-Thought (CoT) prompting.
## Task Description
- **Task Type**: Mixed (Multiple-Choice and Free-Form)
- **Input**: Task-specific questions requiring reasoning
- **Output**: Answers in specified format
- **Subsets**: 27 reasoning tasks divided into multiple-choice (17) and free-form (10)
## Key Features
- 27 challenging reasoning tasks from BIG-Bench
- Multiple-choice tasks: temporal sequences, disambiguation, logical deduction, etc.
- Free-form tasks: arithmetic, navigation, boolean expressions, etc.
- Each task comes with curated Chain-of-Thought examples
- Designed to test advanced reasoning capabilities
## Evaluation Notes
- Default configuration uses **3-shot** with CoT prompting (recommended)
- CoT prompts are pre-defined for each subset in `cot_prompts/` directory
- Answers should follow the format: "So the answer is [ANSWER]"
- Setting `few_shot_num=0` disables few-shot examples
- Multiple-choice answers are normalized to single letters (A, B, C, etc.)
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `bbh` |
| **Dataset ID** | [evalscope/bbh](https://modelscope.cn/datasets/evalscope/bbh/summary) |
| **Paper** | N/A |
| **Tags** | `Reasoning` |
| **Metrics** | `acc` |
| **Default Shots** | 3-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 6,511 |
| Prompt Length (Mean) | 3307.29 chars |
| Prompt Length (Min/Max) | 1060 / 7885 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `temporal_sequences` | 250 | 3746.18 | 3646 | 3876 |
| `disambiguation_qa` | 250 | 4047.48 | 3993 | 4099 |
| `date_understanding` | 250 | 1550.66 | 1491 | 1641 |
| `tracking_shuffled_objects_three_objects` | 250 | 3257.42 | 3195 | 3316 |
| `penguins_in_a_table` | 146 | 3030.88 | 2922 | 3201 |
| `geometric_shapes` | 250 | 5270.24 | 5201 | 5384 |
| `snarks` | 178 | 3493.68 | 3339 | 3693 |
| `ruin_names` | 250 | 3832.01 | 3781 | 3948 |
| `tracking_shuffled_objects_seven_objects` | 250 | 3598.1 | 3506 | 3682 |
| `tracking_shuffled_objects_five_objects` | 250 | 3419.36 | 3338 | 3489 |
| `logical_deduction_three_objects` | 250 | 3093.32 | 3014 | 3165 |
| `hyperbaton` | 250 | 3433.3 | 3386 | 3486 |
| `logical_deduction_five_objects` | 250 | 3264.38 | 3118 | 3379 |
| `logical_deduction_seven_objects` | 250 | 3434.09 | 3217 | 3633 |
| `movie_recommendation` | 250 | 2489.85 | 2436 | 2613 |
| `salient_translation_error_detection` | 250 | 7401.64 | 7223 | 7885 |
| `reasoning_about_colored_objects` | 250 | 2818.32 | 2572 | 3102 |
| `multistep_arithmetic_two` | 250 | 2596.98 | 2594 | 2600 |
| `navigate` | 250 | 2508.7 | 2452 | 2626 |
| `dyck_languages` | 250 | 2723.8 | 2680 | 2874 |
| `word_sorting` | 250 | 2481.34 | 2397 | 2569 |
| `sports_understanding` | 250 | 1077.42 | 1060 | 1122 |
| `boolean_expressions` | 250 | 1991.7 | 1980 | 1998 |
| `object_counting` | 250 | 1706.66 | 1647 | 1787 |
| `formal_fallacies` | 250 | 5185.5 | 4918 | 5514 |
| `causal_judgement` | 187 | 4877.42 | 4194 | 6311 |
| `web_of_lies` | 250 | 3300.84 | 3267 | 3340 |
## Sample Example
**Subset**: `temporal_sequences`
```json
{
"input": [
{
"id": "7d1767c8",
"content": "Task description: Answer questions about which times certain events could have occurred.\n\nQ: Today, Emily went to the museum. Between what times could they have gone?\nWe know that:\nEmily woke up at 1pm.\nElizabeth saw Emily reading at the libr ... [TRUNCATED] ... \nOptions:\n(A) 6pm to 9pm\n(B) 7am to 11am\n(C) 1pm to 2pm\n(D) 2pm to 6pm\nA: Let's think step by step. Put your final answer in the format of \"So the answer is [ANSWER]\" (without quotes and markdown) where [ANSWER] is the answer to the problem.\n"
}
],
"target": "A",
"id": 0,
"group_id": 0,
"subset_key": "temporal_sequences",
"metadata": {
"task_type": "multiple_choice"
}
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
Q: {question}
A: Let's think step by step. Put your final answer in the format of "So the answer is [ANSWER]" (without quotes and markdown) where [ANSWER] is the answer to the problem.
```
<details>
<summary>Few-shot Template</summary>
```text
{fewshot}
Q: {question}
A: Let's think step by step. Put your final answer in the format of "So the answer is [ANSWER]" (without quotes and markdown) where [ANSWER] is the answer to the problem.
```
</details>
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets bbh \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['bbh'],
dataset_args={
'bbh': {
# subset_list: ['temporal_sequences', 'disambiguation_qa', 'date_understanding'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

214
evalscope/docs/en/benchmarks/bc2gm.md Normal file
View File

@ -0,0 +1,214 @@
# BC2GM
## Overview
The BC2GM (BioCreative II Gene Mention) dataset is a widely used corpus for gene mention recognition, consisting of 20,000 sentences from MEDLINE abstracts where gene and protein names have been manually annotated by domain experts.
## Task Description
- **Task Type**: Biomedical Named Entity Recognition (NER)
- **Input**: Biomedical text from MEDLINE abstracts
- **Output**: Identified gene and protein name spans
- **Domain**: Molecular biology, genetics
## Key Features
- 20,000 sentences from MEDLINE abstracts
- Expert-annotated gene and protein mentions
- Benchmark from BioCreative II challenge
- Widely used for biomedical NER evaluation
- High-quality manual annotations
## Evaluation Notes
- Default configuration uses **5-shot** evaluation
- Metrics: Precision, Recall, F1-Score, Accuracy
- Entity types: GENE (gene and protein names)
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `bc2gm` |
| **Dataset ID** | [extraordinarylab/bc2gm](https://modelscope.cn/datasets/extraordinarylab/bc2gm/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `NER` |
| **Metrics** | `precision`, `recall`, `f1_score`, `accuracy` |
| **Default Shots** | 5-shot |
| **Evaluation Split** | `test` |
| **Train Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 5,000 |
| Prompt Length (Mean) | 2228.52 chars |
| Prompt Length (Min/Max) | 2073 / 3129 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "06f75061",
"content": "Here are some examples of named entity recognition:\n\nInput:\nComparison with alkaline phosphatases and 5 - nucleotidase\n\nOutput:\n<response>Comparison with <gene>alkaline phosphatases</gene> and 5 - nucleotidase</response>\n\nInput:\nPharmacologic ... [TRUNCATED] ... most specific entity type.\n7. Ensure every opening tag has a matching closing tag.\n\nText to process:\nPhenotypic analysis demonstrates that trio and Abl cooperate in regulating axon outgrowth in the embryonic central nervous system ( CNS ) .\n"
}
],
"target": "<response>Phenotypic analysis demonstrates that <gene>trio</gene> and <gene>Abl</gene> cooperate in regulating axon outgrowth in the embryonic central nervous system ( CNS ) .</response>",
"id": 0,
"group_id": 0,
"metadata": {
"tokens": [
"Phenotypic",
"analysis",
"demonstrates",
"that",
"trio",
"and",
"Abl",
"cooperate",
"in",
"regulating",
"axon",
"outgrowth",
"in",
"the",
"embryonic",
"central",
"nervous",
"system",
"(",
"CNS",
")",
"."
],
"ner_tags": [
"O",
"O",
"O",
"O",
"B-GENE",
"O",
"B-GENE",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O"
]
}
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
<details>
<summary>Few-shot Template</summary>
```text
Here are some examples of named entity recognition:
{fewshot}
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
</details>
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets bc2gm \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['bc2gm'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

226
evalscope/docs/en/benchmarks/bc4chemd.md Normal file
View File

@ -0,0 +1,226 @@
# BC4CHEMD
## Overview
The BC4CHEMD (BioCreative IV CHEMDNER) dataset is a corpus of 10,000 PubMed abstracts with 84,355 chemical entity mentions manually annotated by experts for chemical named entity recognition.
## Task Description
- **Task Type**: Chemical Named Entity Recognition (NER)
- **Input**: Scientific text from PubMed abstracts
- **Output**: Identified chemical compound name spans
- **Domain**: Chemistry, pharmacology, drug discovery
## Key Features
- 10,000 PubMed abstracts
- 84,355 chemical entity mentions
- Expert manual annotations
- Benchmark from BioCreative IV challenge
- Comprehensive chemical compound coverage
## Evaluation Notes
- Default configuration uses **5-shot** evaluation
- Metrics: Precision, Recall, F1-Score, Accuracy
- Entity types: CHEMICAL (chemical and drug names)
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `bc4chemd` |
| **Dataset ID** | [extraordinarylab/bc4chemd](https://modelscope.cn/datasets/extraordinarylab/bc4chemd/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `NER` |
| **Metrics** | `precision`, `recall`, `f1_score`, `accuracy` |
| **Default Shots** | 5-shot |
| **Evaluation Split** | `test` |
| **Train Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 26,364 |
| Prompt Length (Mean) | 3042.58 chars |
| Prompt Length (Min/Max) | 2879 / 3709 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "5067e19c",
"content": "Here are some examples of named entity recognition:\n\nInput:\nDPP6 as a candidate gene for neuroleptic - induced tardive dyskinesia .\n\nOutput:\n<response>DPP6 as a candidate gene for neuroleptic - induced tardive dyskinesia .</response>\n\nInput:\n ... [TRUNCATED] ... ry opening tag has a matching closing tag.\n\nText to process:\nEffects of docosahexaenoic acid and methylmercury on child ' s brain development due to consumption of fish by Finnish mother during pregnancy : a probabilistic modeling approach .\n"
}
],
"target": "<response>Effects of <chemical>docosahexaenoic acid</chemical> and <chemical>methylmercury</chemical> on child ' s brain development due to consumption of fish by Finnish mother during pregnancy : a probabilistic modeling approach .</response>",
"id": 0,
"group_id": 0,
"metadata": {
"tokens": [
"Effects",
"of",
"docosahexaenoic",
"acid",
"and",
"methylmercury",
"on",
"child",
"'",
"s",
"brain",
"development",
"due",
"to",
"consumption",
"of",
"fish",
"by",
"Finnish",
"mother",
"during",
"pregnancy",
":",
"a",
"probabilistic",
"modeling",
"approach",
"."
],
"ner_tags": [
"O",
"O",
"B-CHEMICAL",
"I-CHEMICAL",
"O",
"B-CHEMICAL",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O"
]
}
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
<details>
<summary>Few-shot Template</summary>
```text
Here are some examples of named entity recognition:
{fewshot}
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
</details>
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets bc4chemd \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['bc4chemd'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

214
evalscope/docs/en/benchmarks/bc5cdr.md Normal file
View File

@ -0,0 +1,214 @@
# BC5CDR
## Overview
The BC5CDR corpus is a manually annotated resource of 1,500 PubMed articles developed for the BioCreative V challenge, containing over 4,400 chemical mentions, 5,800 disease mentions, and 3,100 chemical-disease interactions.
## Task Description
- **Task Type**: Biomedical Named Entity Recognition (NER)
- **Input**: PubMed article text
- **Output**: Identified chemical and disease entity spans
- **Domain**: Pharmacology, medical informatics, toxicology
## Key Features
- 1,500 PubMed articles with expert annotations
- 4,400+ chemical mentions
- 5,800+ disease mentions
- 3,100+ chemical-disease interactions
- Benchmark from BioCreative V challenge
## Evaluation Notes
- Default configuration uses **5-shot** evaluation
- Metrics: Precision, Recall, F1-Score, Accuracy
- Entity types: CHEMICAL, DISEASE
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `bc5cdr` |
| **Dataset ID** | [extraordinarylab/bc5cdr](https://modelscope.cn/datasets/extraordinarylab/bc5cdr/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `NER` |
| **Metrics** | `precision`, `recall`, `f1_score`, `accuracy` |
| **Default Shots** | 5-shot |
| **Evaluation Split** | `test` |
| **Train Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 4,797 |
| Prompt Length (Mean) | 3598.18 chars |
| Prompt Length (Min/Max) | 3455 / 4087 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "154a621d",
"content": "Here are some examples of named entity recognition:\n\nInput:\nSelegiline - induced postural hypotension in Parkinson ' s disease : a longitudinal study on the effects of drug withdrawal .\n\nOutput:\n<response><chemical>Selegiline</chemical> - ind ... [TRUNCATED] ... e.\n7. Ensure every opening tag has a matching closing tag.\n\nText to process:\nTorsade de pointes ventricular tachycardia during low dose intermittent dobutamine treatment in a patient with dilated cardiomyopathy and congestive heart failure .\n"
}
],
"target": "<response><disease>Torsade de pointes ventricular tachycardia</disease> during low dose intermittent <chemical>dobutamine</chemical> treatment in a patient with <disease>dilated cardiomyopathy</disease> and <disease>congestive heart failure</disease> .</response>",
"id": 0,
"group_id": 0,
"metadata": {
"tokens": [
"Torsade",
"de",
"pointes",
"ventricular",
"tachycardia",
"during",
"low",
"dose",
"intermittent",
"dobutamine",
"treatment",
"in",
"a",
"patient",
"with",
"dilated",
"cardiomyopathy",
"and",
"congestive",
"heart",
"failure",
"."
],
"ner_tags": [
"B-DISEASE",
"I-DISEASE",
"I-DISEASE",
"I-DISEASE",
"I-DISEASE",
"O",
"O",
"O",
"O",
"B-CHEMICAL",
"O",
"O",
"O",
"O",
"O",
"B-DISEASE",
"I-DISEASE",
"O",
"B-DISEASE",
"I-DISEASE",
"I-DISEASE",
"O"
]
}
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
<details>
<summary>Few-shot Template</summary>
```text
Here are some examples of named entity recognition:
{fewshot}
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
</details>
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets bc5cdr \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['bc5cdr'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

236
evalscope/docs/en/benchmarks/bfcl_v3.md Normal file
View File

@ -0,0 +1,236 @@
# BFCL-v3
## Overview
BFCL (Berkeley Function Calling Leaderboard) v3 is the first comprehensive and executable function call evaluation benchmark for assessing LLMs' ability to invoke functions. It evaluates various forms of function calls, diverse scenarios, and executability.
## Task Description
- **Task Type**: Function Calling / Tool Use Evaluation
- **Input**: User query with available function definitions
- **Output**: Correct function call with parameters
- **Categories**: AST_NON_LIVE, AST_LIVE, RELEVANCE, MULTI_TURN
## Key Features
- Comprehensive function call evaluation
- Tests simple, multiple, and parallel function calls
- Multi-language support (Python, Java, JavaScript)
- Relevance detection (irrelevance handling)
- Multi-turn conversation function calling
- Live API endpoints for execution testing
## Evaluation Notes
- Requires `pip install bfcl-eval==2025.10.27.1` before evaluation
- Set `is_fc_model=True` for models with native function calling support
- `underscore_to_dot` parameter controls function name formatting
- See the [usage documentation](https://evalscope.readthedocs.io/en/latest/third_party/bfcl_v3.html) for detailed setup
- Results broken down by category (AST, Relevance, Multi-turn)
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `bfcl_v3` |
| **Dataset ID** | [AI-ModelScope/bfcl_v3](https://modelscope.cn/datasets/AI-ModelScope/bfcl_v3/summary) |
| **Paper** | N/A |
| **Tags** | `Agent`, `FunctionCalling` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 4,441 |
| Prompt Length (Mean) | 291.25 chars |
| Prompt Length (Min/Max) | 36 / 10805 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `simple` | 400 | 119.66 | 57 | 303 |
| `multiple` | 200 | 120.75 | 65 | 229 |
| `parallel` | 200 | 298.92 | 69 | 781 |
| `parallel_multiple` | 200 | 432.17 | 88 | 1249 |
| `java` | 100 | 228.93 | 138 | 531 |
| `javascript` | 50 | 222.34 | 118 | 637 |
| `live_simple` | 258 | 161.74 | 47 | 1247 |
| `live_multiple` | 1,053 | 138.76 | 42 | 4828 |
| `live_parallel` | 16 | 150.44 | 88 | 365 |
| `live_parallel_multiple` | 24 | 217.67 | 72 | 650 |
| `irrelevance` | 240 | 85.96 | 55 | 203 |
| `live_relevance` | 18 | 263.83 | 71 | 1214 |
| `live_irrelevance` | 882 | 188.26 | 36 | 10805 |
| `multi_turn_base` | 200 | 803.33 | 106 | 1756 |
| `multi_turn_miss_func` | 200 | 806.75 | 110 | 1760 |
| `multi_turn_miss_param` | 200 | 858.2 | 167 | 1791 |
| `multi_turn_long_context` | 200 | 803.29 | 106 | 1756 |
## Sample Example
**Subset**: `simple`
```json
{
"input": [
{
"id": "15e84989",
"content": "[[{\"role\": \"user\", \"content\": \"Find the area of a triangle with a base of 10 units and height of 5 units.\"}]]"
}
],
"target": "[{\"calculate_triangle_area\": {\"base\": [10], \"height\": [5], \"unit\": [\"units\", \"\"]}}]",
"id": 0,
"group_id": 0,
"subset_key": "simple",
"metadata": {
"id": "simple_0",
"multi_turn": false,
"functions": [
{
"name": "calculate_triangle_area",
"description": "Calculate the area of a triangle given its base and height.",
"parameters": {
"type": "dict",
"properties": {
"base": {
"type": "integer",
"description": "The base of the triangle."
},
"height": {
"type": "integer",
"description": "The height of the triangle."
},
"unit": {
"type": "string",
"description": "The unit of measure (defaults to 'units' if not specified)"
}
},
"required": [
"base",
"height"
]
}
}
],
"tools": [
{
"type": "function",
"function": {
"name": "calculate_triangle_area",
"description": "Calculate the area of a triangle given its base and height. Note that the provided function is in Python 3 syntax.",
"parameters": {
"type": "object",
"properties": {
"base": {
"type": "integer",
"description": "The base of the triangle."
},
"height": {
"type": "integer",
"description": "The height of the triangle."
},
"unit": {
"type": "string",
"description": "The unit of measure (defaults to 'units' if not specified)"
}
},
"required": [
"base",
"height"
]
}
}
}
],
"missed_functions": "{}",
"initial_config": {},
"involved_classes": [],
"turns": [
[
{
"role": "user",
"content": "Find the area of a triangle with a base of 10 units and height of 5 units."
}
]
],
"language": "Python",
"test_category": "simple",
"subset": "simple",
"ground_truth": [
{
"calculate_triangle_area": {
"base": [
10
],
"height": [
5
],
"unit": [
"units",
""
]
}
}
],
"should_execute_tool_calls": false,
"missing_functions": {},
"is_fc_model": true
}
}
```
## Prompt Template
*No prompt template defined.*
## Extra Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `underscore_to_dot` | `bool` | `True` | Convert underscores to dots in function names for evaluation. |
| `is_fc_model` | `bool` | `True` | Indicates the evaluated model natively supports function calling. |
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets bfcl_v3 \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['bfcl_v3'],
dataset_args={
'bfcl_v3': {
# subset_list: ['simple', 'multiple', 'parallel'] # optional, evaluate specific subsets
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

182
evalscope/docs/en/benchmarks/bfcl_v4.md Normal file
View File

@ -0,0 +1,182 @@
# BFCL-v4
## Overview
BFCL-v4 (Berkeley Function-Calling Leaderboard V4) is a comprehensive benchmark for evaluating agentic function-calling capabilities of LLMs. It tests web search, memory operations, and format sensitivity as building blocks for agentic applications.
## Task Description
- **Task Type**: Agentic Function Calling Evaluation
- **Input**: Scenarios requiring function calls for web search, memory, etc.
- **Output**: Correct function calls with proper arguments
- **Capabilities**: Web search, memory read/write, format handling
## Key Features
- Holistic agentic evaluation framework
- Tests building blocks for LLM agents (search, memory, formatting)
- Multiple scoring categories for detailed analysis
- Supports both FC models and non-FC models
- Optional web search with SerpAPI integration
## Evaluation Notes
- **Installation Required**: `pip install bfcl-eval==2025.10.27.1`
- Primary metric: **Accuracy** per category
- Configure `is_fc_model` based on model capabilities
- Optional: Set `SERPAPI_API_KEY` for web search tasks
- [Usage Example](https://evalscope.readthedocs.io/en/latest/third_party/bfcl_v4.html)
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `bfcl_v4` |
| **Dataset ID** | [berkeley-function-call-leaderboard](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) |
| **Paper** | N/A |
| **Tags** | `Agent`, `FunctionCalling` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 5,106 |
| Prompt Length (Mean) | 350.4 chars |
| Prompt Length (Min/Max) | 36 / 10805 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `irrelevance` | 240 | 85.98 | 55 | 203 |
| `live_irrelevance` | 884 | 198.72 | 36 | 10805 |
| `live_multiple` | 1,053 | 149.16 | 42 | 4828 |
| `live_parallel` | 16 | 163.88 | 88 | 365 |
| `live_parallel_multiple` | 24 | 217.67 | 72 | 650 |
| `live_relevance` | 16 | 385.94 | 71 | 1830 |
| `live_simple` | 258 | 171.91 | 47 | 1247 |
| `memory_kv` | 155 | 646.46 | 585 | 826 |
| `memory_rec_sum` | 155 | 646.46 | 585 | 826 |
| `memory_vector` | 155 | 646.46 | 585 | 826 |
| `multi_turn_base` | 200 | 808.43 | 106 | 1756 |
| `multi_turn_long_context` | 200 | 808.41 | 106 | 1756 |
| `multi_turn_miss_func` | 200 | 811.88 | 110 | 1760 |
| `multi_turn_miss_param` | 200 | 863.88 | 167 | 1791 |
| `multiple` | 200 | 120.75 | 65 | 229 |
| `parallel` | 200 | 297.93 | 69 | 781 |
| `parallel_multiple` | 200 | 432.17 | 88 | 1249 |
| `simple_java` | 100 | 228.93 | 138 | 531 |
| `simple_javascript` | 50 | 222.34 | 118 | 637 |
| `simple_python` | 400 | 119.66 | 57 | 303 |
| `web_search_base` | 100 | 831.01 | 748 | 963 |
| `web_search_no_snippet` | 100 | 831.01 | 748 | 963 |
## Sample Example
**Subset**: `irrelevance`
```json
{
"input": [
{
"id": "4e05e910",
"content": "[[{\"role\": \"user\", \"content\": \"Calculate the area of a triangle given the base is 10 meters and height is 5 meters.\"}]]"
}
],
"target": "{}",
"id": 0,
"group_id": 0,
"metadata": {
"id": "irrelevance_0",
"question": [
[
{
"role": "user",
"content": "Calculate the area of a triangle given the base is 10 meters and height is 5 meters."
}
]
],
"function": [
{
"name": "determine_body_mass_index",
"description": "Calculate body mass index given weight and height.",
"parameters": {
"type": "dict",
"properties": {
"weight": {
"type": "float",
"description": "Weight of the individual in kilograms."
},
"height": {
"type": "float",
"description": "Height of the individual in meters."
}
},
"required": [
"weight",
"height"
]
}
}
],
"category": "irrelevance",
"ground_truth": {}
}
}
```
## Prompt Template
*No prompt template defined.*
## Extra Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `underscore_to_dot` | `bool` | `True` | Convert underscores to dots in function names for evaluation. |
| `is_fc_model` | `bool` | `True` | Indicates the evaluated model natively supports function calling. |
| `SERPAPI_API_KEY` | `str | null` | `None` | SerpAPI key enabling web-search capability in BFCL V4. Null disables web search. |
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets bfcl_v4 \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['bfcl_v4'],
dataset_args={
'bfcl_v4': {
# subset_list: ['irrelevance', 'live_irrelevance', 'live_multiple'] # optional, evaluate specific subsets
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

142
evalscope/docs/en/benchmarks/bigcodebench.md Normal file
View File

@ -0,0 +1,142 @@
# BigCodeBench
## Overview
BigCodeBench is an easy-to-use benchmark for solving practical and challenging tasks via code. It evaluates the true programming capabilities of large language models (LLMs) in a more realistic setting with diverse function calls from 139 popular libraries covering 723 API calls.
## Task Description
- **Task Type**: Code Generation (Python)
- **Input**: Programming task description (docstring or natural language instruction)
- **Output**: Complete Python function implementation
- **Libraries**: 139 popular Python libraries (numpy, pandas, sklearn, etc.)
## Key Features
- 1,140 rich-context programming tasks in Python
- Two evaluation modes: Complete (docstring) and Instruct (natural language)
- Covers diverse function calls from 139 popular libraries
- Uses unittest.TestCase for thorough correctness verification
- Supports pass@k metric calculation
## Evaluation Notes
- **Sandbox Required**: Requires sandbox environment with 70+ Python libraries pre-installed
- Two modes available via `split` parameter: `complete` (docstring completion) or `instruct` (NL instruction)
- Default timeout is 240 seconds per problem
- `calibrate` option prepends code_prompt to align function signatures
- See [sandbox documentation](https://evalscope.readthedocs.io/en/latest/user_guides/sandbox.html) for setup
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `bigcodebench` |
| **Dataset ID** | [evalscope/bigcodebench](https://modelscope.cn/datasets/evalscope/bigcodebench/summary) |
| **Paper** | N/A |
| **Tags** | `Coding` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `v0.1.4` |
| **Aggregation** | `mean_and_pass_at_k` |
## Data Statistics
*Statistics not available.*
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "657d2473",
"content": "Calculates the average of the sums of absolute differences between each pair of consecutive numbers for all permutations of a given list. Each permutation is shuffled before calculating the differences. Args: - numbers (list): A list of numbe ... [TRUNCATED 75 chars] ... loat: The average of the sums of absolute differences for each shuffled permutation of the list.\nYou should write self-contained code starting with:\n```\nimport itertools\nfrom random import shuffle\ndef task_func(numbers=list(range(1, 3))):\n```"
}
],
"target": " permutations = list(itertools.permutations(numbers))\n sum_diffs = 0\n\n for perm in permutations:\n perm = list(perm)\n shuffle(perm)\n diffs = [abs(perm[i] - perm[i+1]) for i in range(len(perm)-1)]\n sum_diffs += sum(diffs)\n\n avg_sum_diffs = sum_diffs / len(permutations)\n \n return avg_sum_diffs",
"id": 0,
"group_id": 0,
"metadata": {
"task_id": "BigCodeBench/0",
"entry_point": "task_func",
"complete_prompt": "import itertools\nfrom random import shuffle\n\ndef task_func(numbers=list(range(1, 3))):\n \"\"\"\n Calculates the average of the sums of absolute differences between each pair of consecutive numbers \n for all permutations of a given list. ... [TRUNCATED 187 chars] ... age of the sums of absolute differences for each shuffled permutation of the list.\n\n Requirements:\n - itertools\n - random.shuffle\n\n Example:\n >>> result = task_func([1, 2, 3])\n >>> isinstance(result, float)\n True\n \"\"\"\n",
"code_prompt": "import itertools\nfrom random import shuffle\ndef task_func(numbers=list(range(1, 3))):\n",
"test": "import unittest\nfrom unittest.mock import patch\nfrom random import seed, shuffle\nimport itertools\nclass TestCases(unittest.TestCase):\n def test_default_numbers(self):\n # Test with default number range (1 to 10) to check that the res ... [TRUNCATED 2578 chars] ... x: seed(1) or shuffle(x)):\n result1 = task_func([1, 2, 3])\n with patch('random.shuffle', side_effect=lambda x: seed(1) or shuffle(x)):\n result2 = task_func([1, 2, 4])\n self.assertNotEqual(result1, result2)"
}
}
```
## Prompt Template
**Prompt Template:**
```text
{prompt}
```
## Extra Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `split` | `str` | `instruct` | Evaluation mode: "complete" (docstring completion) or "instruct" (NL instruction). Choices: ['complete', 'instruct'] |
| `version` | `str` | `default` | Dataset version. Use "default" for the latest available version. |
| `calibrate` | `bool` | `True` | Whether to prepend code_prompt to the solution for function signature alignment. |
## Sandbox Configuration
This benchmark requires a sandbox environment for code execution.
```json
{
"image": "bigcodebench/bigcodebench-evaluate:latest",
"tools_config": {
"shell_executor": {},
"python_executor": {}
},
"memory_limit": "4g"
}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets bigcodebench \
--sandbox '{"enabled": true}' \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['bigcodebench'],
sandbox={'enabled': True},
dataset_args={
'bigcodebench': {
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

146
evalscope/docs/en/benchmarks/bigcodebench_hard.md Normal file
View File

@ -0,0 +1,146 @@
# BigCodeBench-Hard
## Overview
BigCodeBench-Hard is a curated subset of BigCodeBench containing 148 tasks that are more aligned with real-world programming tasks. These tasks require more complex reasoning and multi-step problem solving.
## Task Description
- **Task Type**: Code Generation (Python)
- **Input**: Programming task description (docstring or natural language instruction)
- **Output**: Complete Python function implementation
- **Difficulty**: Higher than BigCodeBench-Full, closer to real-world complexity
## Key Features
- 148 challenging tasks selected from BigCodeBench
- Requires complex reasoning and multi-tool usage
- Two evaluation modes: Complete (docstring) and Instruct (natural language)
- Uses unittest.TestCase for thorough correctness verification
- Supports pass@k metric calculation
## Evaluation Notes
- **Sandbox Required**: Requires sandbox environment with 70+ Python libraries pre-installed
- Two modes available via `split` parameter: `complete` (docstring completion) or `instruct` (NL instruction)
- Default timeout is 240 seconds per problem
- `calibrate` option prepends code_prompt to align function signatures
- See [sandbox documentation](https://evalscope.readthedocs.io/en/latest/user_guides/sandbox.html) for setup
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `bigcodebench_hard` |
| **Dataset ID** | [evalscope/bigcodebench-hard](https://modelscope.cn/datasets/evalscope/bigcodebench-hard/summary) |
| **Paper** | N/A |
| **Tags** | `Coding` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `v0.1.4` |
| **Aggregation** | `mean_and_pass_at_k` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 148 |
| Prompt Length (Mean) | 777.94 chars |
| Prompt Length (Min/Max) | 274 / 1816 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "207adc46",
"content": "Download all files from a specific directory on an FTP server using wget in a subprocess. Args: ftp_server (str): The FTP server address. Default is 'ftp.dlptest.com'. ftp_user (str): The FTP server username. Default is 'dlpuser'. ftp_passwor ... [TRUNCATED 798 chars] ... FTP server.\nYou should write self-contained code starting with:\n```\nimport subprocess\nimport ftplib\nimport os\ndef task_func(ftp_server='ftp.dlptest.com', ftp_user='dlpuser', ftp_password='rNrKYTX9g7z3RgJRmxWuGHbeu', ftp_dir='/ftp/test'):\n```"
}
],
"target": " # Attempt to connect to the FTP server\n try:\n ftp_obj = ftplib.FTP(ftp_server)\n except Exception as e:\n raise Exception(f'Failed to connect to FTP server {ftp_server}: {str(e)}')\n\n # Attempt to login to the FTP serv ... [TRUNCATED 625 chars] ... command = f'wget ftp://{ftp_user}:{ftp_password}@{ftp_server}{ftp_dir}/{filename} -P {download_dir}'\n subprocess.call(command, shell=True)\n downloaded_files.append(filename)\n\n ftp_obj.quit()\n return downloaded_files",
"id": 0,
"group_id": 0,
"metadata": {
"task_id": "BigCodeBench/13",
"entry_point": "task_func",
"complete_prompt": "import subprocess\nimport ftplib\nimport os\n\ndef task_func(ftp_server='ftp.dlptest.com', ftp_user='dlpuser', ftp_password='rNrKYTX9g7z3RgJRmxWuGHbeu', ftp_dir='/ftp/test'):\n \"\"\"\n Download all files from a specific directory on an FTP serv ... [TRUNCATED 909 chars] ... ctory. Outputs the message \"Failed to change to directory {ftp_dir} on server {ftp_server}: {str(e)}\"\n \n Requirements:\n - subprocess\n - ftplib\n - os\n\n Example:\n >>> task_func()\n ['file1.txt', 'file2.jpg', ...]\n \"\"\"\n",
"code_prompt": "import subprocess\nimport ftplib\nimport os\ndef task_func(ftp_server='ftp.dlptest.com', ftp_user='dlpuser', ftp_password='rNrKYTX9g7z3RgJRmxWuGHbeu', ftp_dir='/ftp/test'):\n",
"test": "import unittest\nfrom unittest.mock import patch\nimport os\nclass TestCases(unittest.TestCase):\n def setUp(self):\n \"\"\"Setup a clean test environment before each test.\"\"\"\n if not os.path.exists(\"downloaded_files\"):\n o ... [TRUNCATED 2565 chars] ... with self.assertRaises(Exception) as context:\n task_func(ftp_dir=\"/invalid_directory\")\n self.assertEqual(str(context.exception), f'Failed to change to directory /invalid_directory on server ftp.dlptest.com: {error_message}')"
}
}
```
## Prompt Template
**Prompt Template:**
```text
{prompt}
```
## Extra Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `split` | `str` | `instruct` | Evaluation mode: "complete" (docstring completion) or "instruct" (NL instruction). Choices: ['complete', 'instruct'] |
| `version` | `str` | `default` | Dataset version. Use "default" for the latest available version. |
| `calibrate` | `bool` | `True` | Whether to prepend code_prompt to the solution for function signature alignment. |
## Sandbox Configuration
This benchmark requires a sandbox environment for code execution.
```json
{
"image": "bigcodebench/bigcodebench-evaluate:latest",
"tools_config": {
"shell_executor": {},
"python_executor": {}
},
"memory_limit": "4g"
}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets bigcodebench_hard \
--sandbox '{"enabled": true}' \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['bigcodebench_hard'],
sandbox={'enabled': True},
dataset_args={
'bigcodebench_hard': {
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

117
evalscope/docs/en/benchmarks/biomix_qa.md Normal file
View File

@ -0,0 +1,117 @@
# BioMixQA
## Overview
BiomixQA is a curated biomedical question-answering dataset designed to evaluate AI models on biomedical knowledge and reasoning. It has been utilized to validate the Knowledge Graph based Retrieval-Augmented Generation (KG-RAG) framework across different LLMs.
## Task Description
- **Task Type**: Biomedical Multiple-Choice Question Answering
- **Input**: Biomedical question with multiple answer choices
- **Output**: Correct answer letter
- **Domain**: Biomedical sciences, healthcare, life sciences
## Key Features
- Curated biomedical questions from diverse sources
- Tests medical and biological knowledge comprehension
- Validates RAG framework effectiveness for biomedical domain
- Multiple-choice format for standardized evaluation
- Useful for evaluating healthcare AI systems
## Evaluation Notes
- Default configuration uses **0-shot** evaluation
- Simple accuracy metric for performance measurement
- Evaluates on test split
- No few-shot examples provided
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `biomix_qa` |
| **Dataset ID** | [extraordinarylab/biomix-qa](https://modelscope.cn/datasets/extraordinarylab/biomix-qa/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `MCQ`, `Medical` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `test` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 306 |
| Prompt Length (Mean) | 344.93 chars |
| Prompt Length (Min/Max) | 316 / 393 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "5e830918",
"content": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D,E.\n\nOut of the given list, which Gene is associated with head and neck cancer and uveal melanoma.\n\nA) ABO\nB) CACNA2D1\nC) PSCA\nD) TERT\nE) SULT1B1"
}
],
"choices": [
"ABO",
"CACNA2D1",
"PSCA",
"TERT",
"SULT1B1"
],
"target": "B",
"id": 0,
"group_id": 0,
"metadata": {}
}
```
## Prompt Template
**Prompt Template:**
```text
Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.
{question}
{choices}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets biomix_qa \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['biomix_qa'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

163
evalscope/docs/en/benchmarks/blink.md Normal file
View File

@ -0,0 +1,163 @@
# BLINK
## Overview
BLINK is a benchmark designed to evaluate the core visual perception abilities of Multimodal Large Language Models (MLLMs). It transforms 14 classic computer vision tasks into 3,807 multiple-choice questions with single or multiple images and visual prompts.
## Task Description
- **Task Type**: Visual Perception Multiple-Choice QA
- **Input**: One or more images + multiple-choice question
- **Output**: Single answer letter
- **Domains**: Visual perception, correspondence, reasoning, detection
## Key Features
- Covers 14 diverse visual perception tasks
- Supports single and multi-image inputs (up to 4 images)
- Tests fundamental visual understanding capabilities
- Categories include: Art Style, Counting, Forensic Detection, IQ Test, Jigsaw, Multi-view Reasoning, Object Localization, and more
- Questions derived from classic computer vision benchmarks
## Evaluation Notes
- Default evaluation uses the **val** split
- Primary metric: **Accuracy** on multiple-choice questions
- Uses "ANSWER: [LETTER]" format for responses
- Results can be analyzed across 14 different perception categories
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `blink` |
| **Dataset ID** | [evalscope/BLINK](https://modelscope.cn/datasets/evalscope/BLINK/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `MCQ`, `MultiModal` |
| **Metrics** | `acc` |
| **Default Shots** | 0-shot |
| **Evaluation Split** | `val` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 1,901 |
| Prompt Length (Mean) | 577.53 chars |
| Prompt Length (Min/Max) | 252 / 1125 chars |
**Per-Subset Statistics:**
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|--------|---------|-------------|------------|------------|
| `Art_Style` | 117 | 553 | 553 | 553 |
| `Counting` | 120 | 285.21 | 270 | 317 |
| `Forensic_Detection` | 132 | 480 | 480 | 480 |
| `Functional_Correspondence` | 130 | 1118.34 | 1113 | 1125 |
| `IQ_Test` | 150 | 884.6 | 548 | 922 |
| `Jigsaw` | 150 | 543 | 543 | 543 |
| `Multi-view_Reasoning` | 133 | 549 | 549 | 549 |
| `Object_Localization` | 122 | 531.86 | 527 | 548 |
| `Relative_Depth` | 124 | 359 | 359 | 359 |
| `Relative_Reflectance` | 134 | 498 | 498 | 498 |
| `Semantic_Correspondence` | 139 | 952 | 952 | 952 |
| `Spatial_Relation` | 143 | 263.97 | 252 | 282 |
| `Visual_Correspondence` | 172 | 587 | 587 | 587 |
| `Visual_Similarity` | 135 | 414 | 414 | 414 |
**Image Statistics:**
| Metric | Value |
|--------|-------|
| Total Images | 3,675 |
| Images per Sample | min: 1, max: 4, mean: 1.93 |
| Resolution Range | 200x83 - 3072x4096 |
| Formats | jpeg |
## Sample Example
**Subset**: `Art_Style`
```json
{
"input": [
{
"id": "a522940e",
"content": [
{
"text": "Answer the following multiple choice question. The last line of your response should be of the following format:\n'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B.\n\nSome most common art painting styles include Realism, Impressi ... [TRUNCATED] ... of art paintings, use the first image as the reference image, and determine which one of the second or the third image shares the same style as the reference image?\nSelect from the following choices.\n(A) the second image\n(B) the third image\n"
},
{
"image": "[BASE64_IMAGE: jpeg, ~477.8KB]"
},
{
"image": "[BASE64_IMAGE: jpeg, ~876.1KB]"
},
{
"image": "[BASE64_IMAGE: jpeg, ~329.2KB]"
}
]
}
],
"choices": [
"the second image",
"the third image"
],
"target": "A",
"id": 0,
"group_id": 0
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
Answer the following multiple choice question. The last line of your response should be of the following format:
'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.
{question}
```
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets blink \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['blink'],
dataset_args={
'blink': {
# subset_list: ['Art_Style', 'Counting', 'Forensic_Detection'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

200
evalscope/docs/en/benchmarks/broad_twitter_corpus.md Normal file
View File

@ -0,0 +1,200 @@
# BroadTwitterCorpus
## Overview
BroadTwitterCorpus is a dataset of tweets collected over stratified times, places, and social uses. The goal is to represent a broad range of activities, giving a dataset more representative of the language used in this hardest of social media formats to process.
## Task Description
- **Task Type**: Social Media Named Entity Recognition (NER)
- **Input**: Diverse Twitter text (tweets)
- **Output**: Identified entity spans with types
- **Domain**: Social media, diverse contexts
## Key Features
- Stratified sampling across times, places, and uses
- Representative of diverse Twitter language
- Addresses challenges in social media NER
- Three standard NER entity types (PER, ORG, LOC)
- Useful for robust social media NLP evaluation
## Evaluation Notes
- Default configuration uses **5-shot** evaluation
- Metrics: Precision, Recall, F1-Score, Accuracy
- Entity types: PER, ORG, LOC
## Properties
| Property | Value |
|----------|-------|
| **Benchmark Name** | `broad_twitter_corpus` |
| **Dataset ID** | [extraordinarylab/broad-twitter-corpus](https://modelscope.cn/datasets/extraordinarylab/broad-twitter-corpus/summary) |
| **Paper** | N/A |
| **Tags** | `Knowledge`, `NER` |
| **Metrics** | `precision`, `recall`, `f1_score`, `accuracy` |
| **Default Shots** | 5-shot |
| **Evaluation Split** | `test` |
| **Train Split** | `train` |
## Data Statistics
| Metric | Value |
|--------|-------|
| Total Samples | 2,000 |
| Prompt Length (Mean) | 2341.71 chars |
| Prompt Length (Min/Max) | 2246 / 2398 chars |
## Sample Example
**Subset**: `default`
```json
{
"input": [
{
"id": "62394bfa",
"content": "Here are some examples of named entity recognition:\n\nInput:\nI hate the words chunder , vomit and puke . BUUH .\n\nOutput:\n<response>I hate the words chunder , vomit and puke . BUUH .</response>\n\nInput:\n♥ . . ) ) ( ♫ . ( ړײ ) ♫ . ♥ . « ▓ » ♥ . ♫ ... [TRUNCATED] ... planations, just the tagged text.\n6. If entity spans overlap, choose the most specific entity type.\n7. Ensure every opening tag has a matching closing tag.\n\nText to process:\n@ colgo hey , congrats to you and the team ! Always worth a read :)\n"
}
],
"target": "<response><person>@</person> <person>colgo</person> hey , congrats to you and the team ! Always worth a read :)</response>",
"id": 0,
"group_id": 0,
"metadata": {
"tokens": [
"@",
"colgo",
"hey",
",",
"congrats",
"to",
"you",
"and",
"the",
"team",
"!",
"Always",
"worth",
"a",
"read",
":)"
],
"ner_tags": [
"B-PER",
"B-PER",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O",
"O"
]
}
}
```
*Note: Some content was truncated for display.*
## Prompt Template
**Prompt Template:**
```text
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
<details>
<summary>Few-shot Template</summary>
```text
Here are some examples of named entity recognition:
{fewshot}
You are a named entity recognition system that identifies the following entity types:
{entities}
Process the provided text and mark all named entities with XML-style tags.
For example:
<person>John Smith</person> works at <organization>Google</organization> in <location>Mountain View</location>.
Available entity tags: {entity_list}
INSTRUCTIONS:
1. Wrap your entire response in <response>...</response> tags.
2. Inside these tags, include the original text with entity tags inserted.
3. Do not change the original text in any way (preserve spacing, punctuation, case, etc.).
4. Tag ALL entities you can identify using the exact tag names provided.
5. Do not include explanations, just the tagged text.
6. If entity spans overlap, choose the most specific entity type.
7. Ensure every opening tag has a matching closing tag.
Text to process:
{text}
```
</details>
## Usage
### Using CLI
```bash
evalscope eval \
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets broad_twitter_corpus \
--limit 10 # Remove this line for formal evaluation
```
### Using Python
```python
from evalscope import run_task
from evalscope.config import TaskConfig
task_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['broad_twitter_corpus'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)
```

Some files were not shown because too many files have changed in this diff Show More