{
  "meta": {
    "pretty_name": "SWE-bench_Lite",
    "dataset_id": "princeton-nlp/SWE-bench_Lite",
    "paper_url": null,
    "tags": [
      "Coding"
    ],
    "metrics": [
      "acc"
    ],
    "few_shot_num": 0,
    "eval_split": "test",
    "train_split": "",
    "subset_list": [
      "default"
    ],
    "description": "\n## Overview\n\nSWE-bench Lite is a focused subset of SWE-bench containing 300 Issue-Pull Request pairs from 11 popular Python repositories. It provides a more accessible entry point for evaluating automated software engineering capabilities.\n\n## Task Description\n\n- **Task Type**: Automated Software Engineering / Bug Fixing\n- **Input**: GitHub issue description with repository context\n- **Output**: Code patch (diff format) that resolves the issue\n- **Size**: 300 carefully selected test instances\n\n## Key Features\n\n- 300 test Issue-Pull Request pairs\n- 11 popular Python repositories covered\n- Real-world bugs with verified solutions\n- Evaluation via unit test verification\n- More manageable than full SWE-bench while still challenging\n\n## Evaluation Notes\n\n- Requires `pip install swebench==4.1.0` before evaluation\n- Docker images are built/pulled automatically for each repository\n- See the [usage documentation](https://evalscope.readthedocs.io/en/latest/third_party/swe_bench.html) for detailed setup instructions\n- Popular benchmark variant for initial model comparison\n",
    "prompt_template": "{question}",
    "system_prompt": "",
    "few_shot_prompt_template": "",
    "aggregation": "mean",
    "extra_params": {
      "build_docker_images": {
        "type": "bool",
        "description": "Build Docker images locally for each sample.",
        "value": true
      },
      "pull_remote_images_if_available": {
        "type": "bool",
        "description": "Attempt to pull existing remote Docker images before building.",
        "value": true
      },
      "inference_dataset_id": {
        "type": "str",
        "description": "Oracle dataset ID used to fetch inference context.",
        "value": "princeton-nlp/SWE-bench_oracle"
      },
      "force_arch": {
        "type": "str",
        "description": "Optionally force the docker images to be pulled/built for a specific architecture.",
        "value": "",
        "choices": [
          "",
          "arm64",
          "x86_64"
        ]
      },
      "dockerhub_username": {
        "type": "str",
        "description": "DockerHub user/org namespace for remote SWE-bench images.",
        "value": "swebench"
      }
    },
    "sandbox_config": {},
    "category": "llm"
  },
  "statistics": {
    "total_samples": 0,
    "subset_stats": [],
    "prompt_length": {
      "mean": 0,
      "min": 0,
      "max": 0,
      "std": null
    },
    "target_length_mean": null,
    "computed_at": "2026-01-28T14:17:46.975287"
  },
  "sample_example": {},
  "readme": {
    "en": "# SWE-bench_Lite\n\n\n## Overview\n\nSWE-bench Lite is a focused subset of SWE-bench containing 300 Issue-Pull Request pairs from 11 popular Python repositories. It provides a more accessible entry point for evaluating automated software engineering capabilities.\n\n## Task Description\n\n- **Task Type**: Automated Software Engineering / Bug Fixing\n- **Input**: GitHub issue description with repository context\n- **Output**: Code patch (diff format) that resolves the issue\n- **Size**: 300 carefully selected test instances\n\n## Key Features\n\n- 300 test Issue-Pull Request pairs\n- 11 popular Python repositories covered\n- Real-world bugs with verified solutions\n- Evaluation via unit test verification\n- More manageable than full SWE-bench while still challenging\n\n## Evaluation Notes\n\n- Requires `pip install swebench==4.1.0` before evaluation\n- Docker images are built/pulled automatically for each repository\n- See the [usage documentation](https://evalscope.readthedocs.io/en/latest/third_party/swe_bench.html) for detailed setup instructions\n- Popular benchmark variant for initial model comparison\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `swe_bench_lite` |\n| **Dataset ID** | [princeton-nlp/SWE-bench_Lite](https://modelscope.cn/datasets/princeton-nlp/SWE-bench_Lite/summary) |\n| **Paper** | N/A |\n| **Tags** | `Coding` |\n| **Metrics** | `acc` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n\n\n## Data Statistics\n\n*Statistics not available.*\n\n## Sample Example\n\n*Sample example not available.*\n\n## Prompt Template\n\n**Prompt Template:**\n```text\n{question}\n```\n\n## Extra Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `build_docker_images` | `bool` | `True` | Build Docker images locally for each sample. |\n| `pull_remote_images_if_available` | `bool` | `True` | Attempt to pull existing remote Docker images before building. |\n| `inference_dataset_id` | `str` | `princeton-nlp/SWE-bench_oracle` | Oracle dataset ID used to fetch inference context. |\n| `force_arch` | `str` | `` | Optionally force the docker images to be pulled/built for a specific architecture. Choices: ['', 'arm64', 'x86_64'] |\n| `dockerhub_username` | `str` | `swebench` | DockerHub user/org namespace for remote SWE-bench images. |\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets swe_bench_lite \\\n    --limit 10  # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['swe_bench_lite'],\n    dataset_args={\n        'swe_bench_lite': {\n            # extra_params: {}  # uses default extra parameters\n        }\n    },\n    limit=10,  # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n",
    "zh": "# SWE-bench_Lite\n\n\n## 概述\n\nSWE-bench Lite 是 SWE-bench 的一个聚焦子集，包含来自 11 个热门 Python 仓库的 300 个 Issue-Pull Request 对。它为评估自动化软件工程能力提供了一个更易上手的入口。\n\n## 任务描述\n\n- **任务类型**：自动化软件工程 / 缺陷修复\n- **输入**：带有仓库上下文的 GitHub issue 描述\n- **输出**：解决该 issue 的代码补丁（diff 格式）\n- **规模**：300 个精心挑选的测试实例\n\n## 主要特性\n\n- 包含 300 个测试用的 Issue-Pull Request 对\n- 覆盖 11 个热门 Python 仓库\n- 真实世界中的缺陷及其已验证的解决方案\n- 通过单元测试进行评估验证\n- 相比完整版 SWE-bench 更易于管理，同时仍具挑战性\n\n## 评估说明\n\n- 评估前需先安装 `pip install swebench==4.1.0`\n- 会为每个仓库自动构建或拉取 Docker 镜像\n- 详细设置说明请参阅 [使用文档](https://evalscope.readthedocs.io/zh-cn/latest/third_party/swe_bench.html)\n- 是用于初步模型对比的常用基准测试变体\n\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `swe_bench_lite` |\n| **数据集ID** | [princeton-nlp/SWE-bench_Lite](https://modelscope.cn/datasets/princeton-nlp/SWE-bench_Lite/summary) |\n| **论文** | N/A |\n| **标签** | `Coding` |\n| **指标** | `acc` |\n| **默认示例数** | 0-shot |\n| **评估划分** | `test` |\n\n\n## 数据统计\n\n*统计数据暂不可用。*\n\n## 样例示例\n\n*样例示例暂不可用。*\n\n## 提示模板\n\n**提示模板：**\n```text\n{question}\n```\n\n## 额外参数\n\n| 参数 | 类型 | 默认值 | 描述 |\n|-----------|------|---------|-------------|\n| `build_docker_images` | `bool` | `True` | 为每个样本在本地构建 Docker 镜像。 |\n| `pull_remote_images_if_available` | `bool` | `True` | 在构建前尝试拉取已存在的远程 Docker 镜像。 |\n| `inference_dataset_id` | `str` | `princeton-nlp/SWE-bench_oracle` | 用于获取推理上下文的 Oracle 数据集 ID。 |\n| `force_arch` | `str` | `` | 可选地强制为特定架构拉取/构建 docker 镜像。选项：['', 'arm64', 'x86_64'] |\n| `dockerhub_username` | `str` | `swebench` | 远程 SWE-bench 镜像在 DockerHub 上的用户/组织命名空间。 |\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n    --model YOUR_MODEL \\\n    --api-url OPENAI_API_COMPAT_URL \\\n    --api-key EMPTY_TOKEN \\\n    --datasets swe_bench_lite \\\n    --limit 10  # 正式评估时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n    model='YOUR_MODEL',\n    api_url='OPENAI_API_COMPAT_URL',\n    api_key='EMPTY_TOKEN',\n    datasets=['swe_bench_lite'],\n    dataset_args={\n        'swe_bench_lite': {\n            # extra_params: {}  # 使用默认额外参数\n        }\n    },\n    limit=10,  # 正式评估时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```",
    "content_hash": "daa9d22131c158e695f6c72c3170bc57",
    "needs_translation": false
  },
  "updated_at": "2026-06-16T10:56:20.660351",
  "translation_updated_at": "2026-06-16T10:56:45"
}