9.3 KiB
SWE-bench_Pro
Introduction
SWE-bench_Pro is a more challenging benchmark built by Scale AI on top of SWE-bench, designed to evaluate LLMs / agents on long-horizon, multilingual, real-world software engineering tasks. Given a codebase and an issue description, the model drives a multi-turn agent loop inside a per-instance Docker container, autonomously exploring the repository, editing source files and submitting a patch that must pass the hidden unit tests.
**SWE-bench_Pro is strongly recommended over the original SWE-bench.** Compared to `swe_bench_verified` / `swe_bench_lite`:
- **Harder, less contaminated**: samples are drawn from newer, commercial, or non-mainstream repos rather than popular Python projects (Django/Flask/...) that are likely to leak into training data.
- **Multilingual**: covers Python, JavaScript/TypeScript, Go, and more (`repo_language` field), giving a fuller picture of real engineering ability.
- **Longer task horizon**: each instance touches more files and longer modification paths on average, better separating frontier models.
- **Officially-maintained Docker images**: pre-built per-instance images are pulled directly from DockerHub (`jefzda/sweap-images:{tag}`) — **no local image build required**, saving the multi-hour build that the original SWE-bench would require.
The original `swe_bench_*` series is best reserved for historical comparisons or quick smoke tests.
Task Description
- Task type: Automated software engineering / bug fixing (agentic)
- Input: GitHub issue description (
problem_statement) - Output: Code patch in unified-diff format, collected after autonomous editing
- Languages: Multiple (
repo_languagefield; e.g. JavaScript/TypeScript, Python, Go)
Key Features
- Multi-turn agent loop with a per-instance DockerHub image (
jefzda/sweap-images:{tag}) - Sentinel-based patch submission protocol (
COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT) - Container-side evaluation:
git applypatch → run the instance'srun_script.sh→ parse withparser.py→ check(fail_to_pass | pass_to_pass) ⊆ PASSED - Supports both
toolcall(function-calling) andbackticks(text-based fallback for models without function-calling support) action protocols - Inference and evaluation share a single sandbox configuration:
memory_limit/cpu_limit/platformare configured once
Install Dependencies
SWE-bench_Pro uses Docker to ensure reproducibility.
- Install Docker: see the Docker Installation Guide.
- Linux users: follow the post-installation steps so you don't need
sudoevery time. - Install Python dependencies:
pip install 'evalscope[sandbox]'
evalscope[sandbox] installs ms-enclave and the docker SDK.
On the first run, EvalScope will pull each instance's image from DockerHub on demand (~1–4 GB per image) and auto-clone `scaleapi/SWE-bench_Pro-os` into `~/.cache/evalscope/swe_bench_pro/SWE-bench_Pro-os` (pinned to commit `ca10a60`) for per-instance `run_script.sh`, `parser.py`, and Dockerfiles.
Make sure you have enough disk space and a stable network.
Run Example
The example below mirrors test_swe_bench_pro in tests/benchmark/test_agent.py:
import os
from evalscope import TaskConfig, run_task
from evalscope.config import SandboxTaskConfig
task_cfg = TaskConfig(
model='qwen3-max',
api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
api_key=os.getenv('DASHSCOPE_API_KEY'),
eval_type='openai_api',
datasets=['swe_bench_pro'],
dataset_args={
'swe_bench_pro': {
'extra_params': {
'action_protocol': 'toolcall', # 'toolcall' or 'backticks'
'max_steps': 250, # max agent loop steps
'command_timeout': 60.0, # per-bash-command timeout (seconds)
'eval_timeout': 1800, # per-instance eval timeout (seconds)
}
}
},
# Inference and evaluation share this single sandbox config
sandbox=SandboxTaskConfig(
default_config={
'platform': 'linux/amd64', # default; sweap-images are amd64-only
'memory_limit': '12g', # container memory cap; avoids OOM-Killed (e.g. NodeBB)
'cpu_limit': 4.0, # container CPU quota (number of CPUs)
},
),
eval_batch_size=4, # number of concurrent containers
limit=5, # quick smoke test; remove for formal evaluation
generation_config={
'temperature': 0.7,
'parallel_tool_calls': True,
'stream': True,
}
)
run_task(task_cfg=task_cfg)
Intermediate artifacts are saved under outputs/<timestamp>/swe_bench_pro_log/<instance_id>/:
workspace/patch.diff: the patch the model submittedworkspace/stdout.log/workspace/stderr.log: test-run logsworkspace/output.json: structured test results parsed byparser.pycontainer.log: container's full execution log
Parameters
extra_params (dataset-level)
| Parameter | Type | Default | Description |
|---|---|---|---|
action_protocol |
str | toolcall |
Bash interaction protocol: toolcall or backticks |
max_steps |
int | 250 |
Max agent loop steps per sample |
command_timeout |
float | 60.0 |
Per-bash-command timeout (seconds) |
eval_timeout |
int | 3600 |
Per-instance evaluation timeout (seconds) |
swe_bench_pro_repo_path |
str | '' |
Local path to an existing SWE-bench_Pro-os clone; empty = auto-clone and pin to ca10a60 |
dockerhub_username |
str | jefzda |
DockerHub user/org hosting the sweap-images repository |
sandbox.default_config (task-level, shared by inference and evaluation)
TaskConfig.sandbox.default_config is passed straight to ms-enclave's DockerSandboxConfig and translated into docker's mem_limit / cpu_quota etc. when the container is created. Common fields:
| Field | Type | Default | Description |
|---|---|---|---|
platform |
str | linux/amd64 |
Container platform. sweap-images only ship for amd64; Apple Silicon must keep this default (runs under QEMU). |
memory_limit |
str | 4g (ms-enclave default) |
Container memory cap, e.g. '8g', '12g'. Strongly recommended to set explicitly — some instances (e.g. NodeBB) easily OOM-Kill their test runs. |
cpu_limit |
float | 2.0 (ms-enclave default) |
Container CPU quota (number of CPUs). On multi-core hosts, raising to 4+ speeds up tests. |
network_enabled |
bool | true |
Whether the container has network access. |
**Why configure memory/CPU under `sandbox.default_config` rather than `extra_params`?**
For SWE-bench_Pro, the inference stage (agent loop) and evaluation stage (running `run_script.sh`) share the **same** sandbox config. Setting it once under `TaskConfig.sandbox.default_config` applies to both, with no duplication.
Troubleshooting
1. OOM-Killed / container killed mid-test
Some instances (e.g. nodebb__nodebb-*) have memory-hungry test suites. Raise memory_limit:
sandbox=SandboxTaskConfig(default_config={'memory_limit': '16g', 'cpu_limit': 8.0})
2. Very slow on Apple Silicon
sweap-images are amd64-only; on Apple Silicon they run under QEMU emulation, so a single-instance evaluation may take 10–30 minutes. This is expected. Mitigations:
- Run on an amd64 Linux box; or
- Raise
eval_timeout(the 3600 s default may be insufficient); - Lower
eval_batch_sizeto avoid host overload from too many concurrent containers.
3. git clone https://github.com/scaleapi/SWE-bench_Pro-os.git fails
When the network is restricted, the auto-clone will fail. Clone manually and pass the path:
git clone https://github.com/scaleapi/SWE-bench_Pro-os.git /path/to/SWE-bench_Pro-os
git -C /path/to/SWE-bench_Pro-os checkout ca10a60
'extra_params': {'swe_bench_pro_repo_path': '/path/to/SWE-bench_Pro-os', ...}
4. DockerHub rate limiting / pull failures
Anonymous pulls are limited to 100 per 6h. Recommended:
docker login
Or pre-warm common images with docker pull jefzda/sweap-images:<tag>.
5. docker_image missing for instance ...
Usually means _post_process_samples did not run, or the sample is missing repo / instance_id. Confirm the dataset is ScaleAI/SWE-bench_Pro and not overridden by a custom dataset args.
6. Missing run_script for instance_xxx
The SWE-bench_Pro-os clone is not at commit ca10a60, or some subdirectory is missing. Delete ~/.cache/evalscope/swe_bench_pro/SWE-bench_Pro-os and let it re-clone, or pass swe_bench_pro_repo_path explicitly.
7. pip install evalscope[sandbox] doesn't seem to install the docker SDK
Verify both docker SDK and ms-enclave are present:
python -c "import docker, ms_enclave; print(docker.__version__, ms_enclave.__version__)"