13274243a0
Keep K3 suite selection and report-schema scoring in bash, merge K3/vision dataset_args into dpv4 yamls, and pin EvalScope at 735d920ee911 with local patches. Co-authored-by: Cursor <cursoragent@cursor.com>
104 lines
3.9 KiB
Markdown
104 lines
3.9 KiB
Markdown
# SkillsBench
|
|
|
|
SkillsBench evaluates whether agents can use task-bundled Agent Skills. Each task contains `task.md`, `environment/Dockerfile`, `environment/skills/`, `oracle/`, and `verifier/`. EvalScope builds a local Docker image from the task environment, runs the agent or oracle in a container, then executes the task's `verifier/test.sh` and reads `/logs/verifier/reward.txt` as the score.
|
|
|
|
## Prerequisites
|
|
|
|
- Docker is installed and running.
|
|
- The SkillsBench repository is cloned locally, and you pass the `tasks/` directory path.
|
|
- Real agent runs require `agent_config`, for example the Codex external runner.
|
|
- External runners execute inside the task container. Codex/OpenCode/Gemini CLI runners probe for the CLI first and can install it during runner setup when `auto_install=True`.
|
|
- Verifier scripts may install dependencies through apt/pip/uv, so network access is usually required.
|
|
|
|
## Skill Mode
|
|
|
|
The default `skill_mode` is `no-skill`.
|
|
|
|
- `no-skill`: EvalScope removes `environment/skills` from the temporary build context and strips skill copy/path residues from the Dockerfile.
|
|
- `with-skill`: EvalScope injects task-bundled skills into `/skills`; the runner installs them into its own skill discovery path before launch.
|
|
|
|
`self-gen` is not supported in this version. EvalScope does not automatically compute lift. Run `no-skill` and `with-skill` separately, then compare the two runs with EvalScope's run comparison tools.
|
|
|
|
## Oracle Smoke
|
|
|
|
Use oracle first to validate the image, paths, and verifier contract:
|
|
|
|
```python
|
|
from evalscope import TaskConfig, run_task
|
|
|
|
run_task(TaskConfig(
|
|
model='dummy',
|
|
datasets=['skillsbench'],
|
|
limit=1,
|
|
dataset_args={
|
|
'skillsbench': {
|
|
'extra_params': {
|
|
'tasks_dir': '/path/to/skillsbench/tasks',
|
|
'task_ids': ['offer-letter-generator'],
|
|
'runner': 'oracle',
|
|
'skill_mode': 'no-skill',
|
|
}
|
|
}
|
|
},
|
|
))
|
|
```
|
|
|
|
## Real Agent
|
|
|
|
`no-skill`:
|
|
|
|
```python
|
|
from evalscope import TaskConfig, run_task
|
|
|
|
run_task(TaskConfig(
|
|
model='your-model',
|
|
datasets=['skillsbench'],
|
|
limit=1,
|
|
agent_config={
|
|
'mode': 'external',
|
|
'framework': 'codex',
|
|
'environment': 'docker',
|
|
'timeout': 900,
|
|
'kwargs': {
|
|
'auto_install': True,
|
|
},
|
|
},
|
|
dataset_args={
|
|
'skillsbench': {
|
|
'extra_params': {
|
|
'tasks_dir': '/path/to/skillsbench/tasks',
|
|
'task_ids': ['offer-letter-generator'],
|
|
'skill_mode': 'no-skill',
|
|
}
|
|
}
|
|
},
|
|
))
|
|
```
|
|
|
|
For `with-skill`, only change `skill_mode`:
|
|
|
|
```python
|
|
dataset_args={
|
|
'skillsbench': {
|
|
'extra_params': {
|
|
'tasks_dir': '/path/to/skillsbench/tasks',
|
|
'task_ids': ['offer-letter-generator'],
|
|
'skill_mode': 'with-skill',
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Image Cache
|
|
|
|
EvalScope creates a temporary build context for each task and skill mode, then tags the local Docker image from the context hash. `no-skill` and `with-skill` use different cache keys. The first run builds the image; later runs with the same context reuse it. Set `force_rebuild=true` to rebuild.
|
|
|
|
The task image remains the source of truth for task dependencies and verifier inputs. If that image already contains the external CLI, set the runner's `auto_install` to `False`; otherwise leave `auto_install=True` so the runner installs the CLI inside the task container during setup.
|
|
|
|
## Current Limits
|
|
|
|
- The adapter targets the official SkillsBench `tasks/` set by default and does not include `tasks-extra/` automatically.
|
|
- The supported verifier contract is `verifier/test.sh` writing `/logs/verifier/reward.txt`.
|
|
- Full BenchFlow verifier features such as `reward-kit`, `llm-judge`, `agent-judge`, `ors-episode`, and multi-service verifiers are not supported.
|
|
- EvalScope saves key agent/verifier logs and metadata by default, not the full container filesystem.
|