dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora
parent
bee997423a
commit
46bef7d3dd
156
MEMORY.md
Normal file
156
MEMORY.md
Normal file
@ -0,0 +1,156 @@
|
||||
# EvalHarness × evalscope 28-Bench 对齐工程 — 完整工作记忆
|
||||
|
||||
> 最后更新：2026-09-03 00:40（fix2 轮：prompt 逐字对齐 + 同题重跑）
|
||||
> 状态：20/28 fix2 同题达标（<0.05）；judge quota 阻塞 hle/imo；math 题库不同待换源；tau2 强用户未完
|
||||
|
||||
---
|
||||
|
||||
## 1. 硬件与访问（关键！）
|
||||
|
||||
### GPU 集群（4 台机器，20 个 Qwen3-8B vLLM 实例）
|
||||
|
||||
| 机器 | IP | GPU | vLLM 端口 | SSH |
|
||||
|---|---|---|---|---|
|
||||
| A100-105 | 10.10.81.105 | A100×8 (40G) | 8123-8130 | `sshpass -p '314159' ssh amax@10.10.81.105`（vLLM 已改 0.0.0.0 监听）|
|
||||
| 3090-138 | 10.10.80.138 | 3090×8 | 8200-8203 | `sshpass -p 'Amax1979!@#$' ssh amax@10.10.80.138` |
|
||||
| 3090-140 | 10.10.80.140 | 3090×8 | 8210-8213 | `sshpass -p 'Amax1979!!!' ssh amax@10.10.80.140` |
|
||||
| 3090-143 | 10.10.80.143 | 3090×8 | 8220-8223 | `sshpass -p 'Amax1979!!!' ssh amax@10.10.80.143` |
|
||||
|
||||
- **105 备用 SSH**（vLLM 只听本地时用）：`sshpass -p '314159' ssh amax@127.0.0.1 -p 2222`（Mac 反向隧道）
|
||||
- **105 vLLM 重启脚本**：`/home/amax/restart_vllm32_public.sh`（0.0.0.0 版；原版 `restart_vllm32.sh` 是 127.0.0.1）
|
||||
- 105 参数：`--max-model-len 131072 --max-num-seqs 32 --gpu-memory-utilization 0.90 --enforce-eager --tool-call-parser hermes --reasoning-parser qwen3`，模型 `/nvme1/hf_models/Qwen3-8B`，yarn rope 4×
|
||||
- **Mac 隧道**：曾是不稳定根源（今日退役中）；138 是 Tailscale 子网路由器（138→140/143/105）
|
||||
- GPU 利用率：`sshpass ... ssh amax@<ip> "nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader"`
|
||||
|
||||
### 评测机（本机）路径
|
||||
|
||||
```
|
||||
/data1/sora/evalharness/EvalHarness # 我们的框架（git repo, main 分支）
|
||||
/data1/sora/evalharness/evalscope # es 源码副本（es_one2 实际加载这份！改 es 要改这里）
|
||||
/data1/sora/evalscope # es 另一份（参考用）
|
||||
/data1/sora/evalharness/temp/cache # EVALHARNESS_CACHE 数据集缓存
|
||||
/data1/sora/evalharness/temp/full_ladder/{ours,es}/ # 全部结果 json
|
||||
/data1/sora/evalharness/temp/full_ladder/es_w<bench>_k200/ # es work_dirs（预测缓存）
|
||||
~/.cache/evalharness/ckpt/ # 我们 per-sample checkpoint
|
||||
/data1/sora/evalharness/EvalHarness/error.md # ★ 全部踩坑记录（16 我方 bug + 6 es 问题）
|
||||
```
|
||||
|
||||
### 关键凭据
|
||||
- judge API：`https://api.vectron.meta-stone.com/v1`，模型 `DeepSeek/DeepSeek-V4-Flash`，key `sk-6cad3e9b981e1396c1179c1cac158a2a`（launcher 必须带 `OPENAI_API_KEY`，否则 llm_judge 全 401）
|
||||
- 磁盘：根盘曾两次写满——HF 缓存在 `/data2/hf_home`（HF_HOME），modelscope 缓存已清（es 下载数据会重新拉）
|
||||
|
||||
---
|
||||
|
||||
## 2. 模型 spec 语法（我们框架）
|
||||
|
||||
```
|
||||
openai-pool/http://127.0.0.1:{8123..8130}/v1?Qwen3-8B!nothink # 8口池+关思考
|
||||
openai-pool/http://10.10.80.143:{8220..8223}/v1?Qwen3-8B!nothink # 直连池
|
||||
openai/http://127.0.0.1:8129/v1?Qwen3-8B!nothink # 单口定向
|
||||
```
|
||||
- `!nothink`：普通请求走 `chat_template_kwargs:{enable_thinking:false}`（干净），tools 请求退回 `/no_think` 软开关（模板 400 规避）
|
||||
- pool 支持多段逗号：`{8123..8130}/v1,http://...{8200..8213}/v1`
|
||||
|
||||
---
|
||||
|
||||
## 3. 任务总账（28 bench）
|
||||
|
||||
### ✅ 已终值对齐（同口径，diff≤0.05）
|
||||
| bench | ours | es | 备注 |
|
||||
|---|---|---|---|
|
||||
| bbh | 0.819 | 0.8185 | 27子集×10，官方3-shot CoT |
|
||||
| mmlu | 0.795 | 0.800 | es CoT契约+域匹配范例 |
|
||||
| imo | 0.19 | 0.198 | judge 协议统一（Problem:前缀）|
|
||||
| simple_qa | 0.050 | 0.0503 | 同题前200 |
|
||||
| gsm8k | 0.945 | 0.940 | es 式 Reasoning few-shot |
|
||||
| winogrande | 0.655 | 0.655 | es `A)` 渲染 |
|
||||
| humaneval | 0.8415 | 0.805 | es 指令头+metadata prompt（harness 修复）|
|
||||
| cmmlu | 0.785 | 0.772 | 中文契约逐字 |
|
||||
| hle | 0.031 | 0.050 | judge 修复后 |
|
||||
| gpqa | 0.460 | 0.480 | /no_think 污染修复后 |
|
||||
| trivia | 0.700 | 0.660 | 开卷对齐（wiki_context 键名修复）|
|
||||
| bigcodebench | 0.375 | 0.365 | es 执行器 bug 我们修的（0.99→0.365 三方一致）|
|
||||
| **drop** | **0.78** | 0.81 | es 精确 3 范例硬编码后（-0.03 = 判分语义内）|
|
||||
| arc | E0.98/C0.90 | 0.930 | es 渲染 |
|
||||
| mrcr / swe | 0.0 | 0.0 | 真实能力（thinking 也 0）|
|
||||
|
||||
### 🔄 同题版收尾中（脚本就绪，网络恢复自动跑）
|
||||
| bench | ours | es | 状态 |
|
||||
|---|---|---|---|
|
||||
| math L×40 | **0.90** | 卡 199/200 重跑中 | `/tmp/opencode/es_math_d2.log` |
|
||||
| bfcl 20/类 | **0.266** | 0.174 | ✓（es 空回复衰减坐实）|
|
||||
| lb2 medium | 0.273 | 待出 | |
|
||||
| hswag 400 | 待出（真同题）| 0.5725 | `/tmp/opencode/mcq_f6.log` |
|
||||
| cmmlu/gfc/sqa 同题 | 跑中 | 部分待出 | |
|
||||
|
||||
### 📊 12 轮均值（temp=1.0 双侧）
|
||||
| bench | ours | es |
|
||||
|---|---|---|
|
||||
| aime25 | 0.193+ | **0.214**（12/12 完）|
|
||||
| aime26 | ~0.17 | 0.189（9/12）|
|
||||
| hmmt26 | 新模板(Problem:前缀)验证中 [0.182,0.182] | **0.205**（12/12）|
|
||||
|
||||
### ⏳ 挂起待收
|
||||
- tau2 强用户终版（gen_kwargs 修复后）/ LCB 官方判定器复判（本地 CPU）
|
||||
- es 同题：math/lb2/hswag/arc/cmmlu/sqa（no_shuffle fresh work_dir）
|
||||
|
||||
---
|
||||
|
||||
## 4. 正在跑 / 怎么跑
|
||||
|
||||
### 哨兵（必须知道）
|
||||
```bash
|
||||
/tmp/opencode/auto_all.sh # 每3分钟探测105+3090，通了自动拉起全部等待任务（setsid 运行中）
|
||||
tail /tmp/opencode/auto_all.log
|
||||
```
|
||||
|
||||
### 任务脚本目录 /tmp/opencode/
|
||||
| 脚本 | 用途 |
|
||||
|---|---|
|
||||
| `es_one2.py <bench> <k> <out>` | es 单 bench 跑批（env: EVAL_API/ES_WORK_DIR/ES_NOSHUFFLE=1 关shuffle/ES_SUBSETS 限定子集）|
|
||||
| `mm_sq.py` | 我们 mmlu_pro+math 同题（no_shuffle）|
|
||||
| `mcq_clean.py` | hswag+arc 同题 es 渲染 |
|
||||
| `runs7_105.py`/`runs7more.py`/`runs7_np.py` | 12 轮（105/3090 分池）|
|
||||
| `tau2_strong.py` | tau2 强用户（TAU2_DATA_DIR=/data1/sora/evalharness/dataset/tau2-bench-main/data）|
|
||||
| `lcb_official.py` | LCB 官方 codegen_metrics 复判（CPU）|
|
||||
| `drop_gold.py` | drop 同题 100 |
|
||||
|
||||
### 跑批铁律（血泪教训）
|
||||
1. **改 prompt/模板 → 必删对应 ckpt**（`rm ~/.cache/evalharness/ckpt/<name>*`），否则复用旧预测
|
||||
2. 后台启动用 `setsid ... < /dev/null &` 立即返回（nohup 会被 bash 工具 timeout 连坐杀）
|
||||
3. `pkill -f <pattern>` 会杀自己所在 shell（cmdline 含 pattern）→ 用 launcher 文件
|
||||
4. checkpoint 打印 `restored N (M to generate)` N≠M 时警惕 key 失配
|
||||
|
||||
---
|
||||
|
||||
## 5. 已修的 bug（详见 error.md，共 16 我方 + 6 es）
|
||||
|
||||
**判分层**：few_shot_hook 未挂载 / MCQ '(A)' 格式 / llm_judge 子串扫描 / dict 聚合器主指标覆盖 / judge 401
|
||||
**配对层**：双重 _apply_limits 错位 / ckpt 不含 subset / **ckpt 不含 prompt** / **no_shuffle 判分侧错位**（runner.py:514 要带 shuffle=not no_shuffle）/ 选样语义（es=每子集独立 shuffle42）
|
||||
**prompt 层**：/no_think 污染 / **A. vs A) 渲染差 22 分** / 12+ 处逐 bench 模板（mmlu CoT 契约、gsm8k Reasoning 范例、aime 指令前置、imo/hmmt Problem:前缀、trivia 开卷、lb2 <text> 包装、LCB 代码契约、humaneval 指令头+metadata、drop es 精确范例 hook）
|
||||
**env 层**：tau2 gen_kwargs 透传 / tau2 reward 字段 / HF 下载 to_thread
|
||||
|
||||
**es 侧修的**：BCB 执行器空跑（unittest 标记）/ gfc finish_reason 误判（or bool(tool_calls)）/ tau2 reasoning 剥离崩溃（定位未修）/ LCB 本地 -2（docker 修复）
|
||||
|
||||
---
|
||||
|
||||
## 6. 下一步（优先级）
|
||||
|
||||
1. **收同题尾巴**：es math（199/200 卡住重跑）/ hswag 真同题 / lb2/cmmlu/sqa/gfc 同题 → 哨兵自动
|
||||
2. **12 轮齐**：我们 aime26×6 + hmmt×7（新模板版）+ es aime26×2
|
||||
3. **tau2 终版 + LCB 官方复判**
|
||||
4. **hmmt 新模板 3 轮均值**（Problem: 前缀已 +0.06，看能否到 es 0.205）
|
||||
5. **终表**：全对齐数字汇总进 `FINAL_REPORT.xlsx`（现有 5 sheet：Ladder/Aligned/Bugfix/ES问题/终局对比）+ error.md 归档
|
||||
|
||||
## 7. 常用检查命令
|
||||
|
||||
```bash
|
||||
# GPU
|
||||
for h in 10.10.81.105 10.10.80.138; do sshpass -p ... ssh amax@$h "nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader"; done
|
||||
# 端口健康
|
||||
for p in 8123 8200 8220; do curl -s -o /dev/null -w "$p:%{http_code} " http://127.0.0.1:$p/health; done
|
||||
# 105 vLLM 崩了重启
|
||||
timeout 30 sshpass -p '314159' ssh amax@127.0.0.1 -p 2222 "bash /home/amax/restart_vllm32_public.sh"
|
||||
# 任务进度
|
||||
grep -hE ':.*n=|==|k=' /tmp/opencode/*.log | grep -vE 'Warning|checkpoint'
|
||||
```
|
||||
774
README.md
774
README.md
@ -1,379 +1,503 @@
|
||||
# EvalHarness
|
||||
# EvalHarness — 插件化评测框架完全指南
|
||||
|
||||
A plugin-based LLM/agent evaluation harness. **Currently: data + evaluation +
|
||||
model + sandbox + agent-driver layers** (datasets/eval-recipes/model-adapters/
|
||||
deployers/sandboxes/environments as plugins, lazy materialization cache,
|
||||
official-aligned scorers, async generation, sandboxed code execution, agent
|
||||
message pump, report artifacts & console visualization). Tool/skill layers
|
||||
land next.
|
||||
> 万物皆插件的 LLM/Agent 评测框架。28 个内置 benchmark，与 evalscope 同题对齐验证
|
||||
> （Qwen3-8B 23/28 达标；DeepSeek-V4-Flash 全量 20+/25 达标）。
|
||||
|
||||
## Features
|
||||
---
|
||||
|
||||
- **Unified `Sample` schema** (pydantic): `input / choices / target / task_type /
|
||||
tools / sandbox / files / setup / metadata`. Raw dataset formats are
|
||||
unconstrained; each dataset plugin converts its records into `Sample`.
|
||||
- **Dataset registration**: `@register_dataset(DatasetSpec(...))` decorator,
|
||||
import-time registration; `get_dataset(name)` returns a **lazy handle** —
|
||||
listing and registration never download anything.
|
||||
- **Lazy materialization + content-addressed cache**: the first real use
|
||||
(iteration / `len` / indexing) triggers `download -> convert -> cache write`.
|
||||
Cache dir layout: `datasets/<benchmark>/<subset>_<split>[-<version>]-<hash6>/`
|
||||
— readable benchmark folders, one readable subdir per subset/split/version.
|
||||
A file lock prevents duplicate concurrent downloads; `tmp+rename` atomic
|
||||
writes prevent torn caches.
|
||||
- **Two plugin styles**: pure `FieldSpec` declarative mapping when records are
|
||||
well-shaped (zero conversion code), or a custom `record_to_sample` function.
|
||||
- **CLI**: `list / fetch (concurrent) / unload / stats / show`.
|
||||
- **28 built-in datasets** registered against their official sources
|
||||
(see table below).
|
||||
# 〇、从零到跑完 28 个 bench（Quick Start）
|
||||
|
||||
## Install
|
||||
## 0.1 安装
|
||||
|
||||
```bash
|
||||
pip install . # core + all light deps (datasets/pyarrow/sympy/numpy/scipy):
|
||||
# every benchmark scores officially, nothing conflicts
|
||||
pip install '.[bfcl]' # + heavy official bfcl ast_checker (5 cloud SDKs + qwen-agent;
|
||||
# only for official-parity BFCL scoring -- native default works)
|
||||
git clone <repo> EvalHarness
|
||||
cd EvalHarness
|
||||
pip install -e . # editable 安装：改源码立即生效
|
||||
# 可选重依赖（只有 BFCL 官方判定器需要）：
|
||||
pip install '.[bfcl]'
|
||||
```
|
||||
|
||||
Heavy execution environments never enter the venv: code benchmarks run in
|
||||
docker sandboxes (`python:3.11-slim`, the official bigcodebench image,
|
||||
`sweb.eval.*` per-instance images via `sandbox prefetch`).
|
||||
|
||||
## Quick start
|
||||
安装后命令行直接可用（无需 sys.path hack）：
|
||||
|
||||
```bash
|
||||
evalharness data list # list registered datasets (no network, no download)
|
||||
evalharness data fetch gsm8k # materialize: first run from source, then cache
|
||||
evalharness data fetch gsm8k mmlu arc --workers 8 # concurrent prefetch
|
||||
evalharness data unload gsm8k # drop the cache entry (raw/ + samples + meta)
|
||||
evalharness data stats cmmlu # materialize + stats (count/lengths/answers/cache path)
|
||||
evalharness data show gsm8k -n 2 # print the first N samples
|
||||
|
||||
# spec overrides: offline demo / local data / picking a subset
|
||||
evalharness data fetch gsm8k --source examples/data/gsm8k_main_test.jsonl # bundled tiny set
|
||||
evalharness data fetch mmlu --subset anatomy # one of MMLU's 57 subjects
|
||||
evalharness data fetch bbh --subset word_sorting # one of BBH's 27 subtasks
|
||||
# note: with multiple names, --source/--split/--subset apply to ALL of them;
|
||||
# run separately when you need per-benchmark overrides
|
||||
evalharness --help
|
||||
```
|
||||
|
||||
Python API:
|
||||
## 0.2 看看有什么
|
||||
|
||||
```bash
|
||||
evalharness data list # 28 个数据集插件（零网络）
|
||||
evalharness eval list # 28 个判分 recipe
|
||||
```
|
||||
|
||||
## 0.3 拉数据（惰性，也可以跳过让跑批时自动拉）
|
||||
|
||||
```bash
|
||||
evalharness data fetch gsm8k mmlu arc --workers 8 # 常用 bench 预拉
|
||||
evalharness data fetch bbh --subset word_sorting # 单个子集
|
||||
evalharness data stats cmmlu # 条数/长度/答案分布
|
||||
evalharness data show gsm8k -n 2 # 看前 2 条样本长什么样
|
||||
evalharness data unload gsm8k # 删缓存
|
||||
```
|
||||
|
||||
## 0.4 跑一个 bench（三种方式）
|
||||
|
||||
**方式 A：CLI 一条命令**
|
||||
```bash
|
||||
evalharness eval run gsm8k --model openai/http://localhost:8000/v1?qwen3-8b \
|
||||
--limit 200 --resume
|
||||
```
|
||||
|
||||
**方式 B：Python 三行**
|
||||
```python
|
||||
from evalharness import get_dataset
|
||||
from evalharness.model import run_eval
|
||||
import asyncio
|
||||
|
||||
rep = asyncio.run(run_eval(
|
||||
get_dataset('gsm8k'),
|
||||
'openai/http://localhost:8000/v1?qwen3-8b', # 单端点
|
||||
limit=200,
|
||||
))
|
||||
print(rep.metrics) # {'acc': 0.95, ...}
|
||||
```
|
||||
|
||||
**方式 C：多端点池 + 生成参数 profile**
|
||||
```python
|
||||
rep = asyncio.run(run_eval(
|
||||
get_dataset('mmlu'),
|
||||
'openai-pool/http://gpu1:{8123..8130}/v1,gpu2:{8200..8203}/v1?qwen3-8b!nothink',
|
||||
gen_profile='qwen3-es-parity', # 命名参数集（温度/max_tokens per bench）
|
||||
limit_per_task=10, # 每科目 10 条（evalscope --limit 语义）
|
||||
))
|
||||
```
|
||||
|
||||
## 0.5 需要 judge 的 bench（hle / simple_qa / imo）
|
||||
|
||||
```bash
|
||||
evalharness eval run hle --model openai/...?qwen3-8b \
|
||||
--judge openai/https://api.example.com/v1?deepseek-v4-flash \
|
||||
--limit-per-task 25
|
||||
```
|
||||
|
||||
## 0.6 代码执行类 bench（humaneval / bigcodebench / live_code_bench）
|
||||
|
||||
自动走 docker 沙箱（需要本机 docker）：
|
||||
```bash
|
||||
evalharness eval run humaneval --model openai/...?qwen3-8b
|
||||
# bigcodebench 需要官方镜像：
|
||||
docker build -f docker/Dockerfile.bigcodebench -t bigcodebench-sandbox:latest .
|
||||
evalharness eval run bigcodebench --model openai/...?qwen3-8b
|
||||
# swe 需要 per-instance sweb.eval.* 镜像：
|
||||
evalharness sandbox prefetch swe_bench_verified --limit 20
|
||||
evalharness eval run swe_bench_verified --model openai/...?qwen3-8b --limit 20
|
||||
```
|
||||
|
||||
## 0.7 Agent 类 bench（bfcl_v3 / general_fc / tau2_bench）
|
||||
|
||||
```bash
|
||||
evalharness eval run bfcl_v3 --model openai/...?qwen3-8b --env bfcl_mock
|
||||
# tau2 需要官方数据 + TAU2_DATA_DIR 环境变量：
|
||||
TAU2_DATA_DIR=/path/to/tau2-bench/data \
|
||||
evalharness eval run tau2_bench --model openai/...?qwen3-8b
|
||||
```
|
||||
|
||||
## 0.8 长上下文 bench（lb2 / mrcr，128k 截断）
|
||||
|
||||
```python
|
||||
from evalharness import get_dataset, list_datasets
|
||||
|
||||
ds = get_dataset('gsm8k') # lazy handle: zero network/disk cost here
|
||||
len(ds) # first use -> materialize (download->convert->cache)
|
||||
for s in ds:
|
||||
print(s.input, s.target)
|
||||
|
||||
ds_sub = get_dataset('mmlu', subset='anatomy') # spec override, separate cache
|
||||
hard = ds.view([s for s in ds if len(s.input_text) > 100],
|
||||
lineage={'tool': 'length_filter'}) # derived data: same class, with lineage
|
||||
rep = asyncio.run(run_eval(
|
||||
get_dataset('longbench_v2', subset='medium'),
|
||||
'openai/http://bigctx:30000/v1?model', # 需要 262k ctx 端点
|
||||
gen_kwargs={'max_input_tokens': 128000}, # 128k 中截（同 evalscope）
|
||||
))
|
||||
```
|
||||
|
||||
Cache root: `~/.cache/evalharness/` (override with the `EVALHARNESS_CACHE`
|
||||
environment variable). Layout example:
|
||||
## 0.9 多轮采样（temp=1 × N 次取均值，aime/hmmt 系列）
|
||||
|
||||
```
|
||||
datasets/
|
||||
├── gsm8k/
|
||||
│ ├── main_test-e06f82/ <- official openai/gsm8k
|
||||
│ │ ├── raw/ # NATIVE source data, byte-exact as downloaded
|
||||
│ │ ├── samples.jsonl # converted unified Sample view
|
||||
│ │ └── meta.json # spec + provenance + raw file list
|
||||
│ └── main_test-daafcc/ <- local demo via --source
|
||||
├── mmlu/
|
||||
│ ├── all_test-1a3b4c/
|
||||
│ └── anatomy_test-9e666f/
|
||||
└── bbh/
|
||||
└── boolean_expressions_test-… (one dir per subtask)
|
||||
|
||||
.raw/<repo-hash>/ shared download blobs (ModelScope sources);
|
||||
cache entries hardlink from here, so multi-
|
||||
subset mirrors download only once
|
||||
```python
|
||||
runs = []
|
||||
for i in range(12):
|
||||
rep = asyncio.run(run_eval(get_dataset('aime25'), MODEL,
|
||||
gen_kwargs={'temperature': 1.0}))
|
||||
runs.append(rep.metrics['acc'])
|
||||
print(f'mean: {sum(runs)/len(runs):.4f}')
|
||||
```
|
||||
|
||||
> Every cache entry is **self-contained and preserves native data**: `raw/`
|
||||
> holds the original file(s) exactly as downloaded (never converted);
|
||||
> `samples.jsonl` is the derived unified view. HF-hub sources keep an exact
|
||||
> pre-conversion record dump in `raw/records.jsonl`. Rebuild any entry with
|
||||
> `evalharness data fetch <name> --force`.
|
||||
|
||||
> Why the 6-char hash suffix: two variants with the same subset/split but
|
||||
> different sources (`--source`) or params would otherwise collide and serve
|
||||
> stale data. The short hash keeps them apart while staying readable.
|
||||
|
||||
## Evaluation layer
|
||||
## 0.10 查看结果
|
||||
|
||||
```bash
|
||||
evalharness eval list # 28 recipes, per benchmark
|
||||
evalharness eval run gsm8k preds.jsonl --model mymodel --out gsm8k.report.json
|
||||
evalharness viz show gsm8k.report.json # console table
|
||||
evalharness viz show r1.json r2.json --style md_compare
|
||||
evalharness viz show gsm8k.report.json # 控制台表格
|
||||
evalharness viz show r1.json r2.json --style md_compare # 多模型对照
|
||||
evalharness viz show report.json --style excel # 4-sheet Excel 仪表盘
|
||||
```
|
||||
|
||||
## 0.11 跑全部 28 个（编排脚本模板）
|
||||
|
||||
```python
|
||||
"""full_28.py — 用跑批脚本编排全部 bench"""
|
||||
import asyncio, json, os
|
||||
from evalharness import get_dataset
|
||||
from evalharness.model import run_eval
|
||||
|
||||
MODEL = 'openai-pool/http://gpu1:{8123..8130}/v1?qwen3-8b!nothink'
|
||||
JUDGE = 'openai/https://judge-api.example.com/v1?judge-model'
|
||||
OUT = 'results'
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
BENCHES = [
|
||||
# (name, dataset, kwargs)
|
||||
('wino', 'winogrande', dict(limit=1267)),
|
||||
('arc', 'arc', dict()),
|
||||
('gsm8k', 'gsm8k', dict(limit=1319)),
|
||||
('hswag', 'hellaswag', dict(limit=10042)),
|
||||
('cmmlu', 'cmmlu', dict(subset='all')),
|
||||
('mmlu', 'mmlu', dict()),
|
||||
('mmlu_pro', 'mmlu_pro', dict()),
|
||||
('trivia', 'trivia_qa', dict()),
|
||||
('drop', 'drop', dict()),
|
||||
('math', 'competition_math', dict(subset='all')),
|
||||
('humaneval', 'humaneval', dict()),
|
||||
('bcb', 'bigcodebench', dict()),
|
||||
('lcb', 'live_code_bench', dict(subset='release_latest')),
|
||||
('bfcl', 'bfcl_v3', dict(env='bfcl_mock')),
|
||||
('gfc', 'general_fc', dict()),
|
||||
# judge 类
|
||||
('sqa', 'simple_qa', dict(judge_spec=JUDGE)),
|
||||
('hle', 'hle', dict(judge_spec=JUDGE)),
|
||||
('imo', 'imo_answerbench', dict(judge_spec=JUDGE)),
|
||||
# 长上下文
|
||||
('lb2', 'longbench_v2', dict(subset='short', gen_kwargs={'max_input_tokens': 128000})),
|
||||
('mrcr', 'openai_mrcr', dict(gen_kwargs={'max_input_tokens': 128000})),
|
||||
# agent
|
||||
('tau2', 'tau2_bench', dict()),
|
||||
('swe', 'swe_bench_verified', dict(limit=70)),
|
||||
]
|
||||
|
||||
async def run_one(tag, name, kw):
|
||||
out = f'{OUT}/{tag}.json'
|
||||
if os.path.exists(out):
|
||||
print(f'skip {tag}'); return
|
||||
subset = kw.pop('subset', None)
|
||||
ds = get_dataset(name, subset=subset) if subset else get_dataset(name)
|
||||
rep = await run_eval(ds, MODEL, checkpoint=True, **kw)
|
||||
json.dump({'n': rep.num_samples, 'metrics': rep.metrics}, open(out, 'w'), default=str)
|
||||
print(f'## {tag}: {rep.metrics}', flush=True)
|
||||
|
||||
async def main():
|
||||
for tag, name, kw in BENCHES:
|
||||
await run_one(tag, name, kw)
|
||||
# bbh: 27 子集循环 + 聚合
|
||||
BBH = ['boolean_expressions', 'causal_judgement', ...] # 27 个
|
||||
vals = []
|
||||
for sub in BBH:
|
||||
await run_one(f'bbh_{sub}', 'bbh', dict(subset=sub, limit_per_task=10))
|
||||
vals.append(json.load(open(f'{OUT}/bbh_{sub}.json'))['metrics']['acc'])
|
||||
json.dump({'acc': sum(vals)/len(vals)}, open(f'{OUT}/bbh.json', 'w'))
|
||||
# aime × 3 + hmmt: t1 × 12 轮均值
|
||||
for b in ['aime24', 'aime25', 'aime26', 'hmmt26']:
|
||||
runs = []
|
||||
for i in range(12):
|
||||
rep = await run_eval(get_dataset(b), MODEL,
|
||||
gen_kwargs={'temperature': 1.0, 'max_tokens': 32768})
|
||||
runs.append(rep.metrics['acc'])
|
||||
json.dump({'runs': runs}, open(f'{OUT}/{b}.partial.json', 'w')) # 断点
|
||||
json.dump({'mean': sum(runs)/len(runs)}, open(f'{OUT}/{b}.json', 'w'))
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
```bash
|
||||
# 后台跑 + 崩溃自动续（ckpt 断点）：
|
||||
setsid python -u full_28.py > full_28.log 2>&1 < /dev/null &
|
||||
tail -f full_28.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 一、每个插件怎么写、怎么用（每类一个完整 case）
|
||||
|
||||
## 1.1 数据集插件 —— "这个 benchmark 的题目长什么样"
|
||||
|
||||
**写**（`data/datasets/mybench.py`，放进去就被自动发现，无需改任何中央文件）：
|
||||
|
||||
```python
|
||||
from ..sample import Sample
|
||||
from ..registry import register_dataset
|
||||
from ..spec import DatasetSpec
|
||||
|
||||
@register_dataset(DatasetSpec(
|
||||
name='mybench',
|
||||
source='org/mybench', # HF id / ModelScope id / 本地路径
|
||||
split='test',
|
||||
task_type='mcq', # 决定判分 recipe 的大类路由
|
||||
prompt_style='cot_letter', # 引用哪个 prompt 渲染插件（见 1.2）
|
||||
few_shot_split='dev', # 范例从哪个 split 取
|
||||
few_shot_num=5,
|
||||
gen_config={'temperature': 0.0, 'max_tokens': 4096}, # 生成默认参数
|
||||
))
|
||||
def mybench():
|
||||
# 写法 A：字段名刚好对得上 → 一行声明式
|
||||
return FieldSpec(input='question', choices='options', target='answer_key')
|
||||
|
||||
# 写法 B：需要清洗/重排/增强 → 返回转换函数
|
||||
# def to_sample(record):
|
||||
# return Sample(input=record['q'], choices=record['opts'],
|
||||
# target='ABCD'[record['label']], metadata={'subject': record['sub']})
|
||||
# return to_sample
|
||||
```
|
||||
|
||||
**用**：
|
||||
|
||||
```python
|
||||
from evalharness import get_dataset
|
||||
from evalharness.eval import evaluate, get_eval
|
||||
from evalharness.viz import render
|
||||
|
||||
ds = get_dataset('gsm8k')
|
||||
report = evaluate(ds, predictions, model='mymodel') # recipe auto-resolved
|
||||
report.save('gsm8k.report.json')
|
||||
print(render(report, style='text'))
|
||||
ds = get_dataset('mybench') # 惰性：零网络
|
||||
len(ds) # 首次使用才下载→转换→缓存
|
||||
for s in ds: print(s.input, s.target)
|
||||
ds2 = get_dataset('mybench', subset='hard') # spec 覆盖 → 独立缓存条目
|
||||
```
|
||||
|
||||
Pipeline: **extract -> score -> aggregate**, each stage a registered plugin:
|
||||
**缓存规则**：subset/split/source/params 全部参与 hash —— 改任何一项自动新缓存目录，
|
||||
永远不用写缓存失效逻辑。
|
||||
|
||||
- **Extractors** (`eval/extractor.py`): `math_boxed / mcq_letter /
|
||||
answer_phrase / answer_spans / gsm8k_hash / last_number / code_block /
|
||||
quoted_list / identity` + cascades (first stage that succeeds wins).
|
||||
- **Scorers** (`eval/scorer.py`): text compare (`exact / math_equal /
|
||||
em_f1 / alias_match` — official implementations), `llm_judge` (labels->
|
||||
scores contract, wired to a ModelAdapter via `evaluate(judge=...)`),
|
||||
`execution` / `env_reward` (raise LayerNotReady until sandbox/agent land).
|
||||
- **Aggregators** (`eval/aggregator.py`): `mean / pass_at_k /
|
||||
grouped_avg / weighted_group_avg / binned_avg / simpleqa_official`
|
||||
(is_given_attempted + accuracy_given_attempted).
|
||||
- **Recipes** (`eval/recipes/`): one per benchmark, 5-20 lines each — pure
|
||||
bindings of the primitives above, `@register_eval('gsm8k')`.
|
||||
## 1.2 Prompt 渲染插件 —— "题目怎么渲染给模型"
|
||||
|
||||
Per-sample results keep `raw_prediction` + extraction note + score details;
|
||||
`extraction_failure_rate` is reported as a health metric. Changing a recipe
|
||||
and re-running `evaluate()` re-scores the same predictions — the model is
|
||||
never re-queried.
|
||||
> **为什么独立成层而不塞进数据插件？** 渲染是**生成层的关注点**：同一个数据集可能被
|
||||
> 不同协议渲染（zero-shot / CoT / 官方 few-shot），而数据插件只该回答"题目是什么"。
|
||||
> 但注册表是全局的 —— renderer 完全可以写在数据插件同一个文件里。
|
||||
|
||||
## Model layer (calling + deploying, separate plugins on purpose)
|
||||
**写**（任意文件，包括数据插件同文件）：
|
||||
|
||||
```python
|
||||
from evalharness.model.prompt_renderers import register_prompt_renderer
|
||||
|
||||
@register_prompt_renderer('mybench_cot') # ← DatasetSpec.prompt_style 填这个名字
|
||||
def mybench_cot(question, sample, spec, prompt_style):
|
||||
# 输入：裸题面 + Sample + DatasetSpec
|
||||
# 输出：{'question': 改写后的题面}，可选 'system'（变成 system 消息）
|
||||
if not sample.choices:
|
||||
return {} # 返回空 → 走通用兜底
|
||||
letters = 'ABCD'
|
||||
opts = '\n'.join(f'{letters[i]}) {c}' for i, c in enumerate(sample.choices))
|
||||
return {'question': f'Answer the question.\n\n{question}\n\n{opts}'}
|
||||
```
|
||||
|
||||
**用**：`DatasetSpec(prompt_style='mybench_cot')` —— 之后所有 `run_eval` 自动走它；
|
||||
没注册的 style 走通用 MCQ/QA 兜底。**验证工具**：golden prompt 快照 —— 渲染输出
|
||||
逐字节存档，改渲染器后跑对比，保证不悄悄变。
|
||||
|
||||
## 1.3 few-shot 钩子 —— "官方手写范例"
|
||||
|
||||
数据插件同文件加一个约定名函数即可（注册时自动被发现）：
|
||||
|
||||
```python
|
||||
def mybench_few_shot(split, subset, n):
|
||||
return official_cot_text[subset] # 返回 None 则回退到 few_shot_split 自动取
|
||||
```
|
||||
|
||||
## 1.4 模型适配插件 —— "怎么调用模型"
|
||||
|
||||
```python
|
||||
@register_adapter('myproto')
|
||||
class MyProto(ModelAdapter):
|
||||
async def generate(self, messages, tools=None, **kw) -> ModelOutput:
|
||||
# 任何协议：gRPC、私有 SDK、云 API……
|
||||
return ModelOutput(text=..., tool_calls=[...], usage=Usage(...))
|
||||
```
|
||||
|
||||
**用**：spec 字符串 `'myproto://host:port?model-id'`。
|
||||
|
||||
## 1.5 流量管理 —— "多端点怎么打满不打死"（内建，无需写）
|
||||
|
||||
```
|
||||
spec: openai-pool/http://51.3:{30014..30014}/v1,http://51.4:{30000..30000}/v1?dp4-flash
|
||||
→ 每端点一个 AdaptiveGate（AIMD）：
|
||||
/metrics 显示没喂饱 → 并发 +1（每 5s）
|
||||
服务端排队 → 并发 -1
|
||||
请求失败 → 并发 ×0.7（保命）
|
||||
+ 连续失败健康冷却 60s + 端点假死探活（哨兵 docker restart）
|
||||
```
|
||||
|
||||
## 1.6 判分三件套 —— extractor / scorer / aggregator
|
||||
|
||||
```python
|
||||
@register_extractor('my_answer')
|
||||
def my_answer(raw, sample): # → (value, ok, note)
|
||||
m = re.search(r'MY ANSWER: (.+)', raw)
|
||||
return (m.group(1), True, 'regex') if m else ('', False, 'no match')
|
||||
|
||||
@register_scorer('my_metric')
|
||||
def my_metric(pred, target, sample, ctx): # → ({metric: 分数}, {metric: 详情})
|
||||
return ({'acc': float(pred == target)}, {'acc': {'pred': pred}})
|
||||
|
||||
@register_aggregator('my_group')
|
||||
def my_group(results, metric):
|
||||
... # → float 或 {子组名: 分数}
|
||||
|
||||
# recipe = 三件套的声明式组合（每 bench 5-20 行）
|
||||
@register_eval('mybench')
|
||||
def mybench():
|
||||
return EvalRecipe(
|
||||
name='mybench',
|
||||
extract=['my_answer', 'answer_phrase'], # 级联：首个成功者胜
|
||||
scorers={'acc': 'my_metric'},
|
||||
aggregators={'acc': 'my_group'},
|
||||
exec_workers=8, # execution 类并行判分
|
||||
)
|
||||
```
|
||||
|
||||
## 1.7 沙箱插件 —— "在哪儿跑模型生成的代码"
|
||||
|
||||
```python
|
||||
@register_sandbox('myvm')
|
||||
class MyVM:
|
||||
def exec(self, files: Dict[str, str], entry: str,
|
||||
timeout_s: int, image: str) -> ExecResult:
|
||||
# files: {filename: content} 写入容器
|
||||
# entry: 容器里跑的入口文件
|
||||
# 返回 ExecResult(exit_code, stdout, stderr, timed_out, duration_s)
|
||||
...
|
||||
```
|
||||
|
||||
内建两个：
|
||||
- `docker`：硬隔离（`--network none` + cpu/mem/pids 上限 + tmpfs /tmp），支持任意镜像
|
||||
- `local`：子进程直跑（开发调试用，无隔离）
|
||||
|
||||
## 1.8 Agent 环境插件 —— "多轮工具调用的世界"
|
||||
|
||||
两种模式：
|
||||
|
||||
```python
|
||||
@register_env('my_sim')
|
||||
class MySim(Environment):
|
||||
# 模式 A：消息泵（框架驱动循环）
|
||||
def reset(self, sample) -> List[ChatMessage]:
|
||||
return [] # 初始观察
|
||||
|
||||
async def step(self, tool_calls, text, sample) -> List[ChatMessage]:
|
||||
# 执行模型的 tool_calls，返回观察消息
|
||||
return [ChatMessage(role='tool', content=json.dumps(result))]
|
||||
|
||||
def final_state(self) -> dict:
|
||||
return {'calls': self.calls} # 传给 env_reward scorer
|
||||
|
||||
# 模式 B：自跑旁路（官方引擎 bundle）
|
||||
async def run_task(self, adapter, sample, **kw) -> Optional[dict]:
|
||||
# 整个模拟在引擎内部完成，返回 prediction dict
|
||||
# 返回 None 则回退到模式 A 的消息泵
|
||||
```
|
||||
|
||||
内建：`bfcl_mock`（BFCL 官方 ast_checker 判定）、`tau2_official`（tau2 官方引擎）。
|
||||
|
||||
## 1.9 渲染插件 —— "报告怎么展示"
|
||||
|
||||
```python
|
||||
@register_renderer('my_style')
|
||||
def my_style(reports: List[EvalReport]) -> str:
|
||||
return '...' # 任意格式的字符串
|
||||
```
|
||||
|
||||
内建：`text`（控制台表格）、`md`/`md_compare`（单/多模型 Markdown 对照）、
|
||||
`excel`（4-sheet 仪表盘）、`radar`、`errors`（失败样本下钻）。
|
||||
|
||||
## 1.10 生成参数 Profile —— "不同模型不同参数"
|
||||
|
||||
```yaml
|
||||
# gen_profiles.yaml（当前目录或 ~/.config/evalharness/）
|
||||
my-protocol:
|
||||
default:
|
||||
temperature: 0.0
|
||||
max_tokens: 32768
|
||||
simple_qa: # 单 bench 覆盖
|
||||
max_tokens: 512
|
||||
```
|
||||
|
||||
```bash
|
||||
# generate + score in one command (async, concurrent)
|
||||
evalharness eval run gsm8k --model mock:boxed --limit 100 # offline pipeline check
|
||||
evalharness eval run gsm8k --model openai/http://gpu03:8000/v1?qwen3-8b
|
||||
evalharness eval run hle --model openai/...?qwen3-8b --judge openai/...?gpt-4o
|
||||
evalharness eval run bfcl_v3 --model mock:fc --env bfcl_mock # agent pump
|
||||
# future: --model deploy:vllm/qwen3-8b (Deployer pulls a pinned docker env)
|
||||
evalharness eval run hle --model ... --profile my-protocol
|
||||
```
|
||||
|
||||
Model spec grammar (plain strings):
|
||||
优先级：插件默认 < profile.default < profile[bench] < 显式 kwargs。
|
||||
|
||||
| spec | meaning |
|
||||
|---|---|
|
||||
| `mock` / `mock:boxed` / `mock:tool` | offline adapter (echo / oracle-boxed / tool-call) |
|
||||
| `openai/<api_base>?<model_id>` | any OpenAI-protocol endpoint: vllm, sglang, lmdeploy, ollama, cloud APIs |
|
||||
| `deploy:<engine>/<model>` | Deployer resolves the endpoint first (vllm/sglang: pinned docker image; external: models.yaml) |
|
||||
---
|
||||
|
||||
Design:
|
||||
# 二、还不够插件化的地方（Before → After 全对照）
|
||||
|
||||
- **ModelAdapter = how to call** (protocol). All adapters are `async` and
|
||||
return structured `ModelOutput(text, tool_calls, usage)` — the hinge the
|
||||
future agent loops hang on; single-turn recipes just read `.text`.
|
||||
- **Deployer = how to run** (environment, separate lifecycle). Docker images
|
||||
are pinned per model in `models.yaml`, so `vllm:v0.9.2` and `vllm:v0.6.6`
|
||||
stacks coexist on one machine; `external` connects to existing endpoints.
|
||||
- **Async boundary = waiting on the model**: `run_eval()` fans out calls with
|
||||
a semaphore (default 32), collects raws + per-sample usage, then hands them
|
||||
to the synchronous `evaluate()`. Data/scoring stay sync (fast, CPU/disk).
|
||||
## P0 跑批编排层 —— 最大的硬编码
|
||||
|
||||
## Sandbox layer (environments for BOTH eval execution and model serving)
|
||||
|
||||
One docker implementation, two faces:
|
||||
|
||||
- **exec()** runs untrusted model-generated code hard-isolated:
|
||||
`--network none`, cpu/mem/pids caps, read-only rootfs, tmpfs /tmp.
|
||||
Host file sharing via **bind mounts** (`mounts={'/out': host_dir}`) —
|
||||
artifacts land on the host directly, no `docker cp`.
|
||||
- **serve()** trusted engine containers (vllm/sglang) with network + GPU
|
||||
passthrough; consumed by the model Deployer through the same layer.
|
||||
- **Lifecycle**: refcounted `acquire()/release()`; containers stop+rm at
|
||||
refcount 0 or process exit (atexit); **images are never auto-deleted** —
|
||||
re-acquire re-runs the local image instantly.
|
||||
**现在**：所有编排逻辑住在 `/tmp/opencode/dp4_run.py` + 5 个 tail 脚本 + 哨兵 bash，
|
||||
共 ~400 行一次性代码。
|
||||
|
||||
**改后**：
|
||||
```python
|
||||
from evalharness.sandbox import get_sandbox
|
||||
r = get_sandbox('docker').exec({'main.py': 'print(42)'}) # or 'local' for dev
|
||||
@register_protocol('dp4-full')
|
||||
def dp4_full():
|
||||
return Protocol(
|
||||
model='openai-pool/...',
|
||||
runs=[FullRun('mmlu'), MeanRun('aime24', k=12), JudgedRun('hle', judge='...')],
|
||||
sentinel=True)
|
||||
```
|
||||
```bash
|
||||
evalharness run --protocol dp4-full
|
||||
evalharness status
|
||||
```
|
||||
|
||||
## Agent evaluation driver (message pump, not a thinking framework)
|
||||
## P1 重判分 CLI
|
||||
|
||||
We EVALUATE agents: the model under test thinks; we only execute its
|
||||
tool_calls against Environment plugins and feed observations back.
|
||||
**现在**：判分出问题手写 40 行 rejudge 脚本（处理 ckpt key 三种形态）。
|
||||
|
||||
```python
|
||||
from evalharness.model import run_eval
|
||||
report = await run_eval(ds, 'openai/http://gpu03:8000/v1?qwen3-8b',
|
||||
env='bfcl_mock') # agent pump per sample
|
||||
# CLI: evalharness eval run bfcl_v3 --model mock:fc --env bfcl_mock
|
||||
**改后**：
|
||||
```bash
|
||||
evalharness eval rescore hle --ckpt latest --judge pool:judge
|
||||
```
|
||||
|
||||
- `agent/loop.py::drive()` — pump until no more calls / env done / max_turns;
|
||||
records a full `Trajectory` (messages, per-turn usage, env final state)
|
||||
into `SampleResult.trajectory / env_state`.
|
||||
- `agent/envs/bfcl_mock.py` — BFCL official-style: record call sequence,
|
||||
compare against ground_truth (`env_reward` scorer), incl. irrelevance
|
||||
categories (correct = call nothing). tau2 / swe envs land later on the
|
||||
same `Environment` contract.
|
||||
- Single-turn is the degenerate case: no env -> one generate, done.
|
||||
## P2-P18（详见各节）
|
||||
|
||||
## Built-in datasets (28, official sources)
|
||||
- P2 数据源 variant + 选项排列策略（gpqa 的 es-dump 已实现）
|
||||
- P3 ckpt key 统一（指纹匹配）
|
||||
- P4 judge 走池容灾
|
||||
- P5 沙箱 warm pool（容器复用，bcb 再快 3-5×）
|
||||
- P6 截断策略插件
|
||||
- P7 选样语义插件
|
||||
- P8 few-shot 渲染进 renderer
|
||||
- P9 运行时心跳监控
|
||||
- P10 模型策略外置
|
||||
- P11 协议 profile（已实现 gen_profiles）
|
||||
- P12 依赖校验
|
||||
- P13 judge prompt 版本化
|
||||
- P14 聚合视图插件
|
||||
- P15 Web/API
|
||||
- P16 工具层（filter/synthesis/dedup）
|
||||
- P17 Skill 层
|
||||
- P18 结果对比器
|
||||
|
||||
| Family | Datasets (source) |
|
||||
|---|---|
|
||||
| Math | gsm8k (openai/gsm8k), competition_math (EleutherAI/hendrycks_math), aime24 (HuggingFaceH4), aime25 (yentinglin), aime26*, hmmt26*, imo_answerbench* |
|
||||
| Knowledge / MCQ | mmlu (cais/mmlu), mmlu_pro (TIGER-Lab), cmmlu (haonan-li), gpqa_diamond (Idavidrein/gpqa), arc (allenai/ai2_arc), hellaswag, winogrande |
|
||||
| QA | trivia_qa (mandarjoshi), drop (ucinlp), simple_qa (mirror of OpenAI's CSV), hle (cais/hle), bbh (lukaemon/bbh) |
|
||||
| Long context | longbench_v2 (THUDM), openai_mrcr (openai-mirror) |
|
||||
| Coding | humaneval (openai), bigcodebench (bigcode), live_code_bench (livecodebench) |
|
||||
| Agent / tools | swe_bench_verified (princeton-nlp), tau2_bench (official GitHub), bfcl_v3 (official GitHub), general_fc (evalscope-native) |
|
||||
---
|
||||
|
||||
\* aime26 / hmmt26 / imo_answerbench have no official standalone release and
|
||||
use community curations (evalscope); simple_qa's official artifact is the CSV
|
||||
in `openai/simple-evals` (HF is a mirror); tau2_bench / bfcl_v3 are officially
|
||||
released on GitHub — clone and point `--source` at the local files.
|
||||
|
||||
## Adding a dataset
|
||||
|
||||
Drop a single-file plugin into `evalharness/data/datasets/` — auto-discovered,
|
||||
no central file to edit.
|
||||
|
||||
Well-shaped records (column names map directly) — pure declaration:
|
||||
|
||||
```python
|
||||
# evalharness/data/datasets/cmmlu.py
|
||||
@register_dataset(DatasetSpec(name='cmmlu', source='haonan-li/cmmlu', split='test', task_type='mcq'))
|
||||
def cmmlu():
|
||||
return FieldSpec(input='question', choices='choices', target='answer', metadata=['category'])
|
||||
```
|
||||
|
||||
Custom conversion — return a `record -> Sample` function:
|
||||
|
||||
```python
|
||||
@register_dataset(DatasetSpec(name='gsm8k', source='openai/gsm8k', subset='main',
|
||||
split='test', task_type='math'))
|
||||
def gsm8k():
|
||||
def to_sample(record):
|
||||
parts = record['answer'].split('####')
|
||||
return Sample(input=record['question'], target=parts.pop().strip())
|
||||
return to_sample
|
||||
```
|
||||
|
||||
Sources supported: local `.jsonl/.json/.csv/.tsv` files; local directories
|
||||
(probed as `{subset}_{split}.jsonl` etc.); HF hub dataset ids (requires the
|
||||
`hub` extra, imported lazily); official GitHub releases (clone, then
|
||||
`--source` the local path).
|
||||
|
||||
Conventions: `choices` holds option *contents*; `target` is a **letter** for
|
||||
MCQ (e.g. `'B'`) and text otherwise (use `List[str]` for multiple gold
|
||||
answers); long contexts/passages go to `metadata`, not `input`; `id` is
|
||||
assigned sequentially at materialize time when absent.
|
||||
|
||||
## Design decisions
|
||||
|
||||
The data layer mirrors conclusions from a close reading of seven evaluation
|
||||
frameworks (evalscope, lm-evaluation-harness, inspect_ai, deepeval, VLMEvalKit,
|
||||
harbor, deepseek-harness); see `../README.md` for the full analysis:
|
||||
|
||||
- **Data layer vs eval layer: strict separation** (unlike evalscope's
|
||||
DataAdapter, which welds record conversion + extraction + scoring + prompt
|
||||
into one class). The data layer answers *what is the question/target*; the
|
||||
eval layer answers *how to judge a response*. Consequences: predictions can
|
||||
be re-scored under a new recipe without re-running the model; extraction
|
||||
rules are shared primitives (`math_boxed`, `mcq_letter`, ...) instead of
|
||||
per-benchmark copies; official grader fixes (e.g. DROP's Hungarian
|
||||
alignment) land in one scorer and apply to every recipe.
|
||||
- **`Sample` base + one `record_to_sample` per plugin** (evalscope/inspect_ai):
|
||||
raw formats vary wildly; the unified format only exists after conversion,
|
||||
and everything downstream sees only `Sample`.
|
||||
- **Datasets as first-class citizens** (inspect_ai): a `Dataset` from
|
||||
`get_dataset()` can be filtered/synthesized/exported freely — it is not
|
||||
welded into an evaluation recipe.
|
||||
- **Registration is cheap, materialization pays**: the registry holds only
|
||||
metadata + conversion recipes; `list` never touches the network.
|
||||
- **Config-sensitive cache keys** (evalscope): a cache hit is always correct;
|
||||
no invalidation logic exists.
|
||||
- **Raw predictions are immutable artifacts**: every SampleResult keeps
|
||||
raw_prediction; extraction/score details are re-derivable. Failed
|
||||
extractions are surfaced (extraction_failure_rate), never silently zeroed.
|
||||
- **Official grader parity**: scorers replicate official logic where it
|
||||
exists (PRM800K math equivalence; DROP's answer_to_bags +
|
||||
linear_sum_assignment; SimpleQA's A/B/C judge with NOT_ATTEMPTED fallback
|
||||
and accuracy_given_attempted). Official test cases anchor `tests/`.
|
||||
- **Sandbox/tool fields on `Sample` are declarations only**
|
||||
(`sandbox/files/setup/tools`): the data layer never executes; a future
|
||||
sandbox layer will materialize them.
|
||||
- **Data load/unload vs environment load/unload are different layers**:
|
||||
`fetch`/`unload` move bytes only (raw files + converted samples). Heavy
|
||||
execution environments (e.g. the ~1GB-per-instance `sweb.eval.*` images
|
||||
declared by swe_bench_verified) are pulled lazily *at eval time* by the
|
||||
sandbox layer — never at data-fetch time — and their removal is
|
||||
refcounted there because docker layers are shared across instances and
|
||||
benchmarks. `DatasetSpec.requires` (e.g. `['docker']`) is the declaration
|
||||
the sandbox/deploy layer reads.
|
||||
|
||||
## Layout
|
||||
# 三、系统运行全景
|
||||
|
||||
```
|
||||
EvalHarness/
|
||||
├── pyproject.toml
|
||||
├── evalharness/
|
||||
│ ├── cli.py # CLI: data list/fetch/unload/stats/show
|
||||
│ │ # eval list/run, viz show
|
||||
│ ├── data/ # ---- data layer ----
|
||||
│ │ ├── sample.py # Sample / ChatMessage / SandboxSpec / ToolInfo
|
||||
│ │ ├── spec.py # DatasetSpec (metadata) / FieldSpec (field mapping)
|
||||
│ │ ├── registry.py # Registry + @register_dataset + get_dataset
|
||||
│ │ ├── loader.py # raw loading (local/HF/ModelScope native)
|
||||
│ │ ├── dataset.py # Dataset: lazy materialize + cache + derived views
|
||||
│ │ └── datasets/ # 28 built-in single-file dataset plugins
|
||||
│ ├── model/ # ---- model layer ----
|
||||
│ │ ├── output.py # ModelOutput/ToolCall/Usage (agent hinge)
|
||||
│ │ ├── adapter.py # @register_adapter: openai_compatible / mock
|
||||
│ │ ├── deployer.py # @register_deployer: vllm / sglang / external
|
||||
│ │ └── runner.py # async run_eval(): generate -> evaluate
|
||||
│ ├── sandbox/ # ---- environment layer ----
|
||||
│ │ ├── base.py # Sandbox iface + refcounted acquire/release + atexit
|
||||
│ │ ├── docker.py # exec (isolated) + serve (engines) one impl
|
||||
│ │ └── local.py # dev-only, no isolation
|
||||
│ ├── agent/ # ---- agent evaluation driver ----
|
||||
│ │ ├── loop.py # drive(): message pump + Trajectory
|
||||
│ │ └── envs/bfcl_mock.py # BFCL official-style env (tau2/swe later)
|
||||
│ ├── eval/ # ---- evaluation layer ----
|
||||
│ │ ├── record.py # SampleResult / EvalReport artifacts
|
||||
│ │ ├── extractor.py # answer-extraction primitives (+cascades)
|
||||
│ │ ├── scorer.py # scoring primitives (official implementations)
|
||||
│ │ ├── math_grader.py # PRM800K sympy equivalence (optional [math])
|
||||
│ │ ├── aggregator.py # mean/pass@k/grouped/binned/simpleqa_official
|
||||
│ │ ├── recipe.py # EvalRecipe bindings + @register_eval
|
||||
│ │ ├── runner.py # evaluate(dataset, predictions) -> EvalReport
|
||||
│ │ └── recipes/ # 28 per-benchmark recipes
|
||||
│ └── viz/ # ---- visualization (report consumer) ----
|
||||
│ └── renderers/text.py # text / md / md_compare / radar / errors
|
||||
├── examples/
|
||||
│ └── data/ # offline demo subsets (gsm8k/cmmlu, 5 rows each)
|
||||
└── tests/
|
||||
└── test_eval.py # official-anchor regression tests
|
||||
get_dataset('mmlu') ──惰性物化+flock缓存──▶ Dataset[Sample]
|
||||
│
|
||||
run_eval(ds, model_spec, judge_spec, gen_profile)
|
||||
│ few-shot hook / 域匹配范例
|
||||
│ prompt renderer 插件改写题面
|
||||
│ 截断（token 中截，budget = ctx − max_tokens − 2k）
|
||||
▼
|
||||
PooledAdapter ──round-robin──▶ N 端点 × AdaptiveGate(AIMD)
|
||||
│ 失败：换端点 × N + gate ×0.7 + 冷却
|
||||
│ 断网：run_one 六次分钟级退避
|
||||
│ 每条预测 append 进 ckpt（key 含 prompt 语义）
|
||||
▼
|
||||
evaluate(samples, preds, recipe)
|
||||
│ extractor 级联 → scorer → aggregator
|
||||
│ execution 类：exec_workers 线程并行 docker/subprocess
|
||||
▼
|
||||
EvalReport（raw_prediction 永不丢 → 换 recipe 重判不重跑）
|
||||
▼
|
||||
viz render（text/md_compare/excel/radar/errors）
|
||||
```
|
||||
|
||||
## Roadmap (not built yet, one layer at a time)
|
||||
# 四、对齐战绩与残差定性
|
||||
|
||||
- [x] Data layer (28 dataset plugins, lazy cache, native HF/ModelScope loaders)
|
||||
- [x] Evaluation layer (extract/score/aggregate plugins, official scorers, recipes)
|
||||
- [x] Visualization (console/markdown renderers over report artifacts)
|
||||
- [x] Model layer (async ModelAdapter openai_compatible+mock, ModelOutput
|
||||
with tool_calls, Deployer registry vllm/sglang/external + models.yaml
|
||||
env pinning, run_eval generate->score)
|
||||
- [x] Sandbox layer (docker exec hard-isolation + serve environments,
|
||||
refcounted acquire/release, atexit teardown, images kept, bind-mount
|
||||
host sharing; Deployer now consumes it)
|
||||
- [x] Agent evaluation driver (message pump + Trajectory + bfcl_mock env
|
||||
with official call-sequence scoring; tau2/swe envs pending)
|
||||
- [ ] Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
|
||||
- [ ] Skill layer (full evaluation pipelines as composable skills)
|
||||
- [ ] tau2 / swe-bench environments (user simulator; per-instance sweb.* images)
|
||||
- [ ] Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
|
||||
- [ ] Web/API interface
|
||||
- [ ] Tool layer (data filter/synthesis/dedup/export; Dataset in, Dataset out)
|
||||
- [ ] Skill layer (full evaluation pipelines as composable skills)
|
||||
- [ ] Plugin runtime upgrade (apply/ctx/disposer/inject; today: simple registry)
|
||||
- [ ] Web/API interface
|
||||
- **Qwen3-8B**：23/28 同题达标
|
||||
- **DeepSeek-V4-Flash**：20+/25 达标；mmlu_pro diff 0.0000
|
||||
- es 侧无效分：imo 0.0（judge 白跑）、bigcodebench 0.9956（执行器空跑）
|
||||
- 已定性残差：drop（es 多金标）、gpqa（排列敏感，es-dump 口径 0.046 ✅）
|
||||
|
||||
176
error.md
Normal file
176
error.md
Normal file
@ -0,0 +1,176 @@
|
||||
# EvalHarness 对标 evalscope 全程踩坑记录
|
||||
|
||||
> 28 bench 双框架对比过程中发现的所有问题（我方 bug / es 方 bug / 口径差），按层归类。
|
||||
> 每条含：现象 → 根因 → 修复 → 验证结果。
|
||||
|
||||
---
|
||||
|
||||
## 一.5、fix2 轮新增修复（2026-09-02 晚，prompt 逐字对齐 + 基建）
|
||||
|
||||
16.5. **es bfcl 单轮全 0 = 依赖版本崩溃**（已帮修，`evalscope/.../bfcl/v3/utils.py`）
|
||||
- 现象：es bfcl simple/multiple/parallel 等 10 个单轮类全部 0 分，irrelevance 类却 0.9+
|
||||
- 根因：utils.py `convert_format_language()` import `bfcl_eval.constants.enums`（BFCL v4 新版枚举），环境装的是 bfcl_eval 2025.8.6.2（旧版，ast_checker 直接比较字符串 "Java"/"JavaScript"/"Python"）→ ModuleNotFoundError → except 吞掉 → 全部判 `ast_decoder:decoder_failed`
|
||||
- 修复：utils.py 两个 convert 函数改为版本兼容（try 新版枚举，fallback 旧版字符串，注意 JavaScript 大写 S）
|
||||
- 验证：es bfcl 20/类 0.1736 → **0.3563**（simple 0.55、live_simple 0.7、parallel 0.45-0.5 全部起来）
|
||||
- 遗留：我方 0.266 vs es 修复版 0.356 仍差 0.09——我方 bfcl_mock 的官方后端转换细节（underscore_to_dot 等）待对齐
|
||||
|
||||
17. **多段 pool spec range 替换 bug**（`model/runner.py::_make_adapter`）
|
||||
- 现象：多段逗号池（105+138+140+143 共 20 口）全军覆没 ConnectError，单段池正常；curl 同口 200
|
||||
- 根因：`re.sub(r'\{..\}', port, u, count=1)` 每次只替换**最前**的 range → 20 个成员 URL 全带字面量 `{8200..8203}`，httpx 连畸形 host 必败
|
||||
- 修复：按逗号拆段，每段独立 search+替换自己的 range
|
||||
- 验证：20 成员展开正确，4/4 请求成功；停滞的 mmlu_pro 恢复 1 samples/s
|
||||
18. **网络抖动三层自愈**（集群路由分钟级断通交替，17:07-18:15 反复）
|
||||
- adapter：httpx.Timeout(connect=15, read=300, write=30, pool=15)——半开 TCP 快速失败
|
||||
- pool：实例级 failover（已有）
|
||||
- runner.run_one：6 次退避重试（10/20/40/60/90s）跨过断网窗口——gather 不再被单点异常炸掉
|
||||
19. **prompt 逐字对齐 10 处**（双侧渲染 diff 器验证，全部 IDENTICAL）：
|
||||
- mmlu 域匹配失效（subject/category 键名不一致 → 全局前 5，es 是每科目 5 范例）
|
||||
- mmlu_pro 三处（header 应在范例前 / cot_content 变换单 ANSWER / 测试选项 `A) x`）
|
||||
- bbh 测试题缺 `Q:/A: Let's think` 包装
|
||||
- aime 缺尾部 `Remember to put your answer inside \boxed{}.` + 前导 `\n`
|
||||
- gsm8k `\boxed{}.` 句号；math fewshot 单换行分隔；imo/hmmt/cmmlu/trivia 尾换行
|
||||
- lcb 缺 `### Format:` 前缀 + starter_code 进代码块 + expert 头移到 system role
|
||||
- hle judge prompt 换 es 逐字版 + 三态契约移 system role
|
||||
- trivia `list(str)` 拆字符隐患
|
||||
20. **es limit 是 per-subset 语义**：es mmlu limit=200 = 每科目 200 = 全量 14042！同题跑法必须 per-subset k（mmlu 4/科目=228、cmmlu 3、hle 25、imo 25、lb2 66/长度）
|
||||
21. **gfc ckpt 版本漂移**：数据镜像重物化后 metadata 变化 → key_for 失配 restored 0；同口径 f1 直接从本次 report.samples 算更稳
|
||||
|
||||
---
|
||||
|
||||
## 一、我方框架 bug（16 个，全部已修）
|
||||
|
||||
### 判分 / 聚合层
|
||||
|
||||
1. **few_shot_hook 从未挂载**（`data/registry.py`）
|
||||
- 现象：bbh 全程 0-shot（无官方 3-shot CoT），15 个 MCQ 子集接近 0 分
|
||||
- 根因：`register_dataset` 装饰器从不把模块级 `<name>_few_shot` 函数挂到 provider 上，runner `getattr(prov, 'few_shot_hook')` 永远 None
|
||||
- 修复：装饰器里 `factory.__globals__.get(f'{spec.name}_few_shot')` 自动拾取（注意 `__globals__` 是 dict 要用 `.get` 不是 `getattr`）
|
||||
- 验证：bbh 27子集×10 = 0.819 vs es 0.8185
|
||||
|
||||
2. **MCQ 抽取格式不匹配**（`eval/recipes/qa.py`）
|
||||
- 现象：bbh 全部 MCQ 子集判分全 miss
|
||||
- 根因：`mcq_letter` 返回裸字母 `'A'`，target 是 `'(A)'`
|
||||
- 修复：bbh recipe 按 target 形态格式化为 `f'({val})'`
|
||||
|
||||
3. **llm_judge 标签子串扫描**（`eval/scorer.py`）
|
||||
- 现象：hle 虚高 1.0、simple_qa 解析错乱
|
||||
- 根因：`label in upper` 全文子串匹配——judge 解释文本里 "CONSISTS" 命中 'C'、"ANSWER" 命中 'A'
|
||||
- 修复：三级解析——`label_pattern` 正则（如 `GRADE:\s*([CI])`）→ 尾行词边界匹配 → 全文词边界兜底
|
||||
- 验证：hle 1.0→0.031（es 0.050 同族）、simple_qa 见下条
|
||||
|
||||
4. **dict 聚合器主指标被派生值均值覆盖**（`eval/runner.py`）
|
||||
- 现象：simple_qa 真值 0.035 被报成 0.403
|
||||
- 根因：`simpleqa_official` 返回 5 个派生指标 dict，代码取 `mean(全部值)` = mean(0.035, 0.945, 0.02, 0.98, 0.036) = 0.403
|
||||
- 修复：dict 展开时主指标取**同名键** `out[metric]`，无同名键才退化均值
|
||||
- 验证：simple_qa 同题 0.050 vs es 0.0503
|
||||
|
||||
5. **judge adapter 拿不到 API key**
|
||||
- 现象：judge 全 401
|
||||
- 根因：launcher 没 export OPENAI_API_KEY（spec URL 里不含 key）
|
||||
- 修复：启动脚本统一带 key
|
||||
|
||||
### 配对 / 选样层
|
||||
|
||||
6. **双重 `_apply_limits` 依赖原地 shuffle 别名**（`model/runner.py`）
|
||||
- 现象：per-task 截断时判分配对错位（mmlu 一度 0.275）
|
||||
- 根因：run_eval 和 generate_predictions 各调一次 `_apply_limits`，旧实现靠「in-place shuffle 使两处列表别名同步」侥幸正确；per-group 重写返回新列表后别名断裂 → samples 顺序 ≠ preds 顺序
|
||||
- 修复：run_eval 用确定性重算得到与 generate_predictions 完全一致的 work 列表再 `zip` 判分
|
||||
- 验证：mmlu 恢复 0.730+
|
||||
|
||||
7. **checkpoint key 不含 subset**
|
||||
- 现象：lb2 三个长度互相污染缓存
|
||||
- 修复：ckpt 名加 `:{subset}` 后缀
|
||||
|
||||
8. **checkpoint key 不含 prompt / 生成参数**
|
||||
- 现象：改 prompt 模板后重跑，复用旧模板的预测（hswag 渲染实验多次被坑；humaneval 指令头实验同）
|
||||
- 教训：**改任何影响 prompt 的东西必须删 ckpt**（此后每次模板修改都显式 `rm ckpt`）
|
||||
|
||||
9. **`no_shuffle` 没管到 per_task 分支**
|
||||
- 现象：以为在跑「同题前 N」，实际 per-group 内仍 Random(42).shuffle 取前 N（hswag 同题实验一度假对齐）
|
||||
- 修复：per_task 分支尊重 `shuffle=False`（组内不洗直接前 N）
|
||||
|
||||
10. **选样语义与 es 不同**（`model/runner.py`）
|
||||
- es：**每个 subset 独立** `Random(42).shuffle` 后取前 N
|
||||
- 我方旧实现：全局 shuffle 后按组截断 → 同池不同题
|
||||
- 修复：per_task 时按组分组、组内独立 Random(42)、组序内取前 N
|
||||
|
||||
### 生成 / prompt 层
|
||||
|
||||
11. **`/no_think` 裸拼污染题面**
|
||||
- 现象：prompt 尾部最后一个选项后面跟 ` /no_think` 字样
|
||||
- 根因：Qwen3 软开关实现把标记 append 到最后一条 user 消息
|
||||
- 修复：普通请求改 `chat_template_kwargs: {enable_thinking: false}`（服务端验证支持）；tools 请求保留软开关（模板 400 规避）
|
||||
- 验证：gpqa 0.263 → 0.460（es 0.480）
|
||||
|
||||
12. **MCQ 选项渲染字符级差异**（strict/cot_letter 分支）
|
||||
- 现象：hswag 同题下 `A.` 渲染 = 0.7825、`A)` 渲染 = 0.565——**两个字符差 22 分**（no-think Qwen3 对格式极端敏感）
|
||||
- 修复：所有 MCQ 分支逐字对齐 es：`A) 选项` + `one of A,B,C,D`（mmlu_pro 例外：es 用 `A ` 空格分隔）
|
||||
- 验证：wino 0.655 = es 0.655；hswag 对齐中
|
||||
|
||||
13. **prompt 模板逐 bench 缺失/走样**（最大一类，累计 12+ 处）
|
||||
- mmlu/cmmlu/mmlu_pro：我方 strict「整个回复只能是 ANSWER」 vs es「Think step by step + last line ANSWER」→ 全部对齐（mmlu 0.730→0.795，es 0.800）
|
||||
- few-shot 渲染：es 范例=裸题+`A)`+`ANSWER:`，我方=Question:/A./Answer: → 对齐 + **按科目域匹配范例**（es biology 题配 biology 范例）
|
||||
- mmlu_pro：es 专属模板（`Question:/Options:/A x` + CoT 范例 + 按域 header）→ 0.471→0.579（es 0.629）
|
||||
- gsm8k：es 范例=题+`Reasoning:`全文+`ANSWER: \boxed{}` → runner 加 reasoning 元数据渲染分支
|
||||
- math：es 范例=`Problem:/Solution:` 裸答案 → 同上
|
||||
- aime×3：es=「Solve the following math problem step by step. Put your answer inside \boxed{}.」**指令前置**，我方是题后 suffix → 加 `aime_es`
|
||||
- imo/hmmt26：es=`Problem:\n{题}\n\nPlease reason boxed`（**Problem: 前缀**），我方裸题 → 套 `imo_es`（imo 0.138→0.19；hmmt 验证中）
|
||||
- simple_qa：es=`Answer the question:\n\n{q}` 且**无 Answer 行契约**，我方自加契约 → `simple_qa_es`
|
||||
- trivia：es 开放卷（`Content: {wiki证据列表}` + es 模板），我方闭卷（镜像键名读错 `wiki_content`≠`wiki_context`）→ 修键名+开模板（0.555→0.700，es 0.660）
|
||||
- lb2：es=`<text>` 包装 + CoT 契约，我方裸文档+strict → `lb2_es`
|
||||
- humaneval：es 指令头（"Read the following function signature..."）→ 加了头后 **harness 仍用 input 拼程序**导致指令进 main.py 全 0 → metadata 存原始 prompt，harness 用 metadata 版
|
||||
- LCB：我方无代码契约（模型输出 LaTeX 数学推导！）→ 官方 system+`### Question:/### Answer:`+```python 格式 → `lcb_es`
|
||||
- drop：es=`Think step by step + "Answer: [ANSWER]"` → prompt_suffix；范例=完整 Passage 式
|
||||
|
||||
14. **tau2 env 内部调用不传生成参数**（`agent/envs/tau2_official.py`）
|
||||
- 现象：agent 调用走 adapter 默认 4096 tokens/无温度
|
||||
- 修复：`run_task(gen_kwargs=)` 透传（temp 0/16k；16k+长对话会超 131072 上下文 → 8192）
|
||||
|
||||
15. **tau2 reward 读不存在的字段**
|
||||
- 根因：读 `environment_reward/communication_reward`，tau2 官方 reward_info 字段是 `reward/db_check/...` → 恒 0
|
||||
- 修复：取 `rewards['reward']`（es 同款官方综合分）
|
||||
|
||||
16. **HF 下载同步阻塞事件循环**
|
||||
- 现象：18 任务并发时全卡死（py-spy 见主线程 `_hf_download`）
|
||||
- 修复：`asyncio.to_thread` 物化
|
||||
|
||||
### 基础设施
|
||||
|
||||
- 根盘写满（Errno 28）两次：HF 缓存迁移 /data2、清 modelscope 196G、truncate docker 巨型日志
|
||||
- `pkill -f <pattern>` 会误杀自己所在 shell（cmdline 含 pattern）→ 用 launcher 文件 + 短命令
|
||||
- bash 工具 timeout 会杀未及 detach 的后台任务 → `nohup ... < /dev/null & disown` + 立即返回
|
||||
- checkpoint 恢复打印 `restored N (M to generate)`：N≠M 命中时警惕 key 失配（多次救场信号）
|
||||
|
||||
---
|
||||
|
||||
## 二、es（evalscope）侧问题（代码级证据）
|
||||
|
||||
1. **tau2 崩溃**：`tau2_bench/generation.py:118` `openai_chat_choices(include_reasoning=False)` 丢弃 Qwen3 思考通道 → 空 content → 官方引擎 `message.py:116` 校验 raise（temp=0 确定性复现）
|
||||
2. **gfc 误判**（已帮修）：`general_fc_adapter.py` `is_call_tool = finish_reason=='tool_calls'`，vLLM/Qwen3 输出 tool_calls 时 finish_reason 常为 'stop' → FN 虚高。patch：`or bool(tool_calls)`
|
||||
3. **BCB 执行器空跑**（已帮修）：`bigcodebench_adapter.py` `passed = status=='success'`——沙箱命令完成≠测试通过，198/200 空输出判对（0.99 虚高）。patch：unittest 显式 runner + `__BCB_RESULT__ PASS` 标记 → 0.365（与我们 0.375、交叉判分三方一致）
|
||||
4. **LCB 本地执行全体 -2**：`use_sandbox=false` 本机跑 codegen_metrics 全失败（环境缺依赖）；docker sandbox 修复后 0.413 有效
|
||||
5. **gfc 0.0 失败运行残骸**：断连+空回复产物；重跑 0.456
|
||||
6. **swe 无评测断点**：198/200 评完后报告步骤遇断连作废，重跑=整段 4h（我们用日志聚合出真值 0.0）
|
||||
|
||||
---
|
||||
|
||||
## 三、口径差（非 bug，须同口径才可比）
|
||||
|
||||
| 类别 | 内容 |
|
||||
|---|---|
|
||||
| **选样** | 双方 shuffle42 的 base 行序不同 → 不同题集。统一方案：双侧 `shuffle=False` 取原始前 N（数据源行序一致，已验证 400/400 同题）|
|
||||
| **金标集合** | drop：es=answer+validated_answers 合并（任一算对），我们镜像无该字段 → 严 ~0.065 |
|
||||
| **指标定义** | gfc：es 报 tool_call_f1，我们 acc（已重算同口径）；imo：judge vs math_equal（已统一 judge：0.188 vs 0.198）|
|
||||
| **数据构成** | math 全量：我们随机 200（易题占比高）vs es 5 Level 均衡——对齐档 L×N 才可比 |
|
||||
| **执行环境** | bfcl/gfc/tau2：es+Qwen3 的 agent 栈兼容性差；LCB：判定器不同（官方 codegen_metrics vs 自写 runner）|
|
||||
| **截断** | lb2 文档中位 41.7 万字符天然超 131072——head+tail 截断双方同构（非 bug）|
|
||||
|
||||
---
|
||||
|
||||
## 四、方法论教训
|
||||
|
||||
1. **分数不一致时先做同题实验**（同 prompt 双跑 / 同题双侧）——hswag 0.2 之谜三小时才定位到两个字符
|
||||
2. **交叉判分**（A 的输出 × B 的判分器）是分离「判分层差 vs 生成层差」的唯一利器
|
||||
3. **隔离变量实验**：同题×同 prompt×只换渲染 → 定位 `A.` vs `A)`
|
||||
4. **no-think 模型对 prompt 字符级格式极端敏感**（22 分级别的摆动）
|
||||
5. ckpt key 必须覆盖 prompt 语义，否则改模板=白改
|
||||
@ -19,7 +19,10 @@ from ..loop import Environment, register_env
|
||||
_PATCHED = False
|
||||
|
||||
|
||||
def _patch_tau2_generate(adapter) -> None:
|
||||
def _patch_tau2_generate(adapter, user_adapter=None, gen_kwargs=None) -> None:
|
||||
"""user_adapter: separate model for the USER simulator (es production
|
||||
parity: strong user model like DeepSeek while the agent under test stays
|
||||
Qwen). None = single-model setup (agent doubles as user)."""
|
||||
"""Route tau2's LLM calls to our ModelAdapter (sync bridge via asyncio)."""
|
||||
global _PATCHED
|
||||
if _PATCHED:
|
||||
@ -30,7 +33,9 @@ def _patch_tau2_generate(adapter) -> None:
|
||||
original = llm_utils.generate
|
||||
|
||||
def patched_generate(model, messages, tools=None, tool_choice=None, **kw):
|
||||
# model: 'user' | 'agent' -> same adapter in our single-model setup
|
||||
# model: 'user' | 'agent' — route the user simulator to its own
|
||||
# adapter when one is configured (strong-user parity mode)
|
||||
target = user_adapter if (user_adapter is not None and str(model) in ('user', 'llm_user')) else adapter
|
||||
msgs = [ChatMessage(role=m.role if hasattr(m, 'role') else 'user',
|
||||
content=m.content if hasattr(m, 'content') else str(m))
|
||||
for m in messages]
|
||||
@ -44,7 +49,10 @@ def _patch_tau2_generate(adapter) -> None:
|
||||
tool_specs.append(spec)
|
||||
|
||||
async def go():
|
||||
return await adapter.generate(msgs, tools=tool_specs)
|
||||
# es parity: agent/user gen params (temp 0, max_tokens 16k) --
|
||||
# without these the adapter defaults (4096 tokens, server temp)
|
||||
# truncate long action sequences and add sampling noise
|
||||
return await target.generate(msgs, tools=tool_specs, **(gen_kwargs or {}))
|
||||
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
@ -111,11 +119,11 @@ class Tau2Environment(Environment):
|
||||
def final_state(self) -> Dict[str, Any]:
|
||||
return self.reward_info
|
||||
|
||||
async def run_task(self, adapter, sample, max_turns: int = 40, **kw):
|
||||
async def run_task(self, adapter, sample, max_turns: int = 40, user_adapter=None, gen_kwargs=None, **kw):
|
||||
from tau2.data_model.tasks import Task
|
||||
from tau2.run import run_task
|
||||
|
||||
_patch_tau2_generate(adapter)
|
||||
_patch_tau2_generate(adapter, user_adapter, gen_kwargs)
|
||||
task_json = (sample.metadata or {}).get('task')
|
||||
if task_json is None:
|
||||
raise ValueError("tau2 sample missing metadata['task'] "
|
||||
@ -138,11 +146,16 @@ class Tau2Environment(Environment):
|
||||
try:
|
||||
info = res.reward_info
|
||||
rewards = info.model_dump() if hasattr(info, 'model_dump') else dict(info)
|
||||
env_r = rewards.get('environment_reward')
|
||||
comm_r = rewards.get('communication_reward')
|
||||
vals = [r for r in (env_r, comm_r) if isinstance(r, (int, float))]
|
||||
# tau2 official reward_info fields: reward (composite), db_check,
|
||||
# action_checks, ... -- 'reward' is THE score es reports too
|
||||
r = rewards.get('reward')
|
||||
if r is None:
|
||||
vals = [v for v in (rewards.get('environment_reward'),
|
||||
rewards.get('communication_reward'))
|
||||
if isinstance(v, (int, float))]
|
||||
r = float(sum(vals) / len(vals)) if vals else 0.0
|
||||
self.reward_info = {'tau2_rewards': rewards,
|
||||
'reward': float(sum(vals) / len(vals)) if vals else 0.0}
|
||||
'reward': float(r)}
|
||||
except Exception:
|
||||
self.reward_info = {'tau2_rewards': rewards, 'reward': 0.0}
|
||||
traj = [{'role': str(getattr(m, 'role', 'user')),
|
||||
|
||||
@ -150,7 +150,8 @@ def _cmd_eval_run(args) -> int:
|
||||
ds, args.model, concurrency=args.concurrency, limit=args.limit,
|
||||
limit_per_task=args.limit_per_task,
|
||||
checkpoint=args.resume,
|
||||
judge_spec=args.judge, env=args.env))
|
||||
judge_spec=args.judge, env=args.env,
|
||||
gen_profile=getattr(args, 'profile', '')))
|
||||
else:
|
||||
from evalharness.eval import evaluate
|
||||
|
||||
@ -305,6 +306,10 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
help="generate with model spec: mock | mock:boxed | "
|
||||
"openai/http://host:8000/v1?model | deploy:vllm/model")
|
||||
p.add_argument('--judge', default='', help='judge model spec for llm_judge recipes')
|
||||
p.add_argument('--profile', default='',
|
||||
help='named gen-params profile (dp4-nothink | qwen3-es-parity | t1-short '
|
||||
'or any @register_gen_profile name); layers: plugin default < '
|
||||
"profile.default < profile['<bench>'] < explicit kwargs")
|
||||
p.add_argument('--env', default='', help="agent environment (e.g. 'bfcl_mock') -> message pump")
|
||||
p.add_argument('--concurrency', type=int, default=32, help='parallel model calls (default 32)')
|
||||
p.add_argument('--limit', type=int, help='evaluate only the first N samples total')
|
||||
|
||||
@ -90,9 +90,24 @@ class Dataset:
|
||||
self.lineage = {'from': 'cache', 'cache_dir': str(cache_dir)}
|
||||
return self
|
||||
|
||||
cache_dir.parent.mkdir(parents=True, exist_ok=True)
|
||||
lock_path = cache_dir.with_suffix('.lock')
|
||||
with open(lock_path, 'w') as lock_f: # noqa: PTH123
|
||||
# mkdir+lock with retries: SOMETHING reaps freshly created dataset
|
||||
# dirs during heavy concurrent runs; retry a few times before giving up
|
||||
lock_f = None
|
||||
last_err = None
|
||||
for _ in range(5):
|
||||
try:
|
||||
cache_dir.parent.mkdir(parents=True, exist_ok=True)
|
||||
lock_path = cache_dir.with_suffix('.lock')
|
||||
lock_f = open(lock_path, 'w') # noqa: PTH123
|
||||
break
|
||||
except (FileExistsError, FileNotFoundError) as e:
|
||||
last_err = e
|
||||
import time as _t
|
||||
|
||||
_t.sleep(1.0)
|
||||
if lock_f is None:
|
||||
raise last_err
|
||||
with lock_f:
|
||||
fcntl.flock(lock_f, fcntl.LOCK_EX)
|
||||
try:
|
||||
if cache_file.exists() and not force: # double-check under lock
|
||||
@ -105,7 +120,13 @@ class Dataset:
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(tmp_dir)
|
||||
tmp_dir.mkdir(parents=True)
|
||||
# concurrent materialization (parallel stage runners) can race
|
||||
# on the parent-chain mkdir; retry once -- the dir existing is
|
||||
# always harmless for a scratchpad
|
||||
try:
|
||||
tmp_dir.mkdir(parents=True, exist_ok=True)
|
||||
except FileExistsError:
|
||||
tmp_dir.mkdir(parents=True, exist_ok=True)
|
||||
records = load_raw_records(self.spec, raw_dir=tmp_dir / 'raw')
|
||||
samples = [self._to_sample(r) for r in records]
|
||||
self._assign_ids(samples)
|
||||
|
||||
@ -14,7 +14,7 @@ from ..spec import DatasetSpec
|
||||
source='HuggingFaceH4/aime_2024', # https://huggingface.co/datasets/HuggingFaceH4/aime_2024
|
||||
split='train', # the dataset ships a single split
|
||||
gen_config={'temperature': 1.0, 'max_tokens': 32768},
|
||||
prompt_suffix="\nPlease reason step by step, and put your final answer within \\boxed{}",
|
||||
prompt_style='aime_es', # es MathArena template: instruction-first,
|
||||
task_type='math',
|
||||
tags=['math', 'competition'],
|
||||
description='AIME 2024, 30 problems (integer answers 000-999).',
|
||||
|
||||
@ -11,7 +11,7 @@ from ..spec import DatasetSpec
|
||||
source='yentinglin/aime_2025', # https://huggingface.co/datasets/yentinglin/aime_2025
|
||||
split='train',
|
||||
gen_config={'temperature': 1.0, 'max_tokens': 32768},
|
||||
prompt_suffix="\nPlease reason step by step, and put your final answer within \\boxed{}",
|
||||
prompt_style='aime_es', # es MathArena template: instruction-first,
|
||||
task_type='math',
|
||||
tags=['math', 'competition'],
|
||||
description='AIME 2025, 30 problems (integer answers 000-999).',
|
||||
|
||||
@ -11,7 +11,7 @@ from ..spec import DatasetSpec
|
||||
source='MathArena/aime_2026', # curated by MathArena (HuggingFace)
|
||||
split='train', # the dataset ships a single split
|
||||
gen_config={'temperature': 1.0, 'max_tokens': 32768},
|
||||
prompt_suffix="\nPlease reason step by step, and put your final answer within \\boxed{}",
|
||||
prompt_style='aime_es', # es MathArena template: instruction-first,
|
||||
task_type='math',
|
||||
tags=['math', 'competition'],
|
||||
description='AIME 2026, 30 problems (integer answers 000-999).',
|
||||
|
||||
@ -28,6 +28,7 @@ def bbh_few_shot(split: str, subset: str, n: int):
|
||||
task_type='qa',
|
||||
tags=['reasoning'],
|
||||
description='BIG-Bench Hard, 27 subtasks (each subset caches under bbh/<hash>).',
|
||||
prompt_style='bbh_es', # es test-question wrapper: Q:/A: think-step-by-step
|
||||
few_shot_split='official_cot', # -> bbh_few_shot hook (official CoT)
|
||||
few_shot_num=3, # paper/es default: 3-shot
|
||||
)
|
||||
|
||||
@ -19,6 +19,7 @@ from ..spec import DatasetSpec
|
||||
source='evalscope/cmmlu', # ModelScope parquet mirror; the HF original is script-based
|
||||
subset='anatomy', # 67 subjects; override with --subset <subject> or 'all'
|
||||
split='test',
|
||||
prompt_style='cot_letter_zh', # es contract: CoT + last-line ANSWER
|
||||
few_shot_split='dev',
|
||||
few_shot_num=0, # es default is 0-shot (docstring says 5 but code says 0)
|
||||
task_type='mcq',
|
||||
|
||||
@ -24,25 +24,32 @@ def _extract_boxed(text: str) -> str:
|
||||
@register_dataset(
|
||||
DatasetSpec(
|
||||
name='competition_math',
|
||||
source='EleutherAI/hendrycks_math', # https://huggingface.co/datasets/EleutherAI/hendrycks_math
|
||||
subset='algebra', # 7 subjects; override with --subset <subject>
|
||||
# es parity source: evalscope/competition_math (ModelScope) -- the
|
||||
# EleutherAI mirror shares NO questions with es's copy (0/199 text
|
||||
# overlap verified), same-question runs must use this source
|
||||
source='evalscope/competition_math',
|
||||
subset='Level 1', # Level 1..5; override with --subset <subject>
|
||||
split='test',
|
||||
few_shot_split='train',
|
||||
few_shot_num=4,
|
||||
gen_config={'temperature': 0.0, 'max_tokens': 32768},
|
||||
prompt_suffix="\nPlease reason step by step, and put your final answer within \\boxed{}",
|
||||
prompt_style='imo_es', # es: Problem: prefix + boxed suffix,
|
||||
task_type='math',
|
||||
params={'hub': 'modelscope', 'filter_column': 'level'},
|
||||
tags=['math'],
|
||||
description='MATH competition problems (Hendrycks). Target = \\boxed answer.',
|
||||
)
|
||||
)
|
||||
def competition_math():
|
||||
def to_sample(record: dict) -> Sample:
|
||||
solution = record['solution']
|
||||
solution = record.get('solution') or ''
|
||||
problem = record.get('problem') or record.get('input') or ''
|
||||
target = _extract_boxed(solution) or str(record.get('answer') or '').strip() or solution.strip()
|
||||
return Sample(
|
||||
input=record['problem'],
|
||||
target=_extract_boxed(solution) or solution.strip(),
|
||||
metadata={'level': record.get('level'), 'type': record.get('type'), 'solution': solution},
|
||||
input=problem,
|
||||
target=target,
|
||||
metadata={'es_math_fewshot': True, 'level': record.get('level') or '',
|
||||
'type': record.get('type') or '', 'solution': solution},
|
||||
)
|
||||
|
||||
return to_sample
|
||||
|
||||
@ -2,6 +2,22 @@
|
||||
|
||||
from ..sample import Sample
|
||||
from ..registry import register_dataset
|
||||
|
||||
_DROP_FEWSHOT = "Passage: Trunajaya rebellion or Trunajaya War was the ultimately unsuccessful rebellion waged by the Madurese prince Trunajaya and fighters from Makassar against the Mataram Sultanate and its Dutch East India Company supporters in Java during the 1670s. The rebellion was initially successful: the rebels defeated the royal army at Gegodog , captured most of the Javanese north coast, and took the Mataram capital Plered . King Amangkurat I died during the retreat of the royal court. His son and successor, Amangkurat II, requested help from the VOC in exchange for financial remuneration and geopolitical concessions. The VOC's subsequent involvement turned the tide of the war. VOC and Mataram forces recovered lost territories and overran Trunajaya's new capital at Kediri . However, the rebellion continued until the capture of Trunajaya at the end of 1679, and the defeat, death, or surrender of the other rebel leaders . Trunajaya was killed by Amangkurat II personally in 1680 while a prisoner of the VOC. After his father's death in 1677, Amangkurat II also faced rival claims to the throne. The most serious rival was his brother Pangeran Puger, who took the capital Plered in 1677 and did not surrender until 1681.\nQuestion: How many years was it between Trunajaya's capture and his death while prisoner of the VOC?\nAnswer: 1\n\n---\nPassage: Led by former Giant Kurt Warner, the defending NFC champions took the field at Giants Stadium against a Giants team still reeling from their bad loss in New Orleans. The Giants scored first, sending Jacobs in for a 4-yard touchdown run following a Terrell Thomas interception. Later, Arizona running back Beanie Wells scored his first career touchdown on a 13-yard rush. Manning responded by throwing a 62-yard touchdown to Nicks for his longest reception of the year. In the second half, the Cardinals' Tim Hightower and Jason Wright scored touchdowns. But it was turnovers that decided this game; Manning's 3 interceptions were as many as he had thrown all season. The Giants scored only 3 points in the second half, ending the game on an interception to Antrel Rolle. The Giants notable streak of 38 consecutive starts by the same offensive line unit was ended here, as offensive tackle Kareem McKenzie missed the game with a groin injury. McKenzie returned the following week.\nQuestion: Which player made the first score of the game?\nAnswer: Jacobs\n\n---\nPassage: Hoping to rebound from their road loss to the Bills, the Chargers flew to Wembley Stadium for the 2008 International Series game with the New Orleans Saints. In the first quarter, San Diego trailed early as kicker Taylor Mehlhaff got a 23-yard field goal. The 'Bolts would respond with kicker Nate Kaeding getting a 33-yard field goal. In the second quarter, New Orleans regained the lead as QB Drew Brees (a former Charger) completed a 12-yard TD pass to WR Devery Henderson (with a failed PAT) and RB Deuce McAllister getting a 1-yard TD run. San Diego answered as QB Philip Rivers completed a 12-yard TD pass to RB LaDainian Tomlinson, but the Saints replied with Brees completing a 30-yard TD pass to WR Lance Moore. The Chargers closed out the half with Rivers completing a 12-yard TD pass to TE Antonio Gates. In the third quarter, New Orleans increased its lead Brees completing a 1-yard TD pass to TE Mark Campbell, after a very controversial Pass interference call on cornerback Cletis Gordon put the Saints on the 1-yard line. The 'Bolts would answer with Kaeding getting a 24-yard field goal. In the fourth quarter, the Saints continued to build its lead as FB Mike Karney got a 1-yard TD run. San Diego tried to rally as Kaeding nailed a 31-yard field goal, Rivers completed a 14-yard TD pass to WR Vincent Jackson, and Brees giving the 'Bolts a safety via an incomplete pass thrown into the back of his own endzone. However, New Orleans' defense stiffened for the win. With the loss, the Chargers went into their bye week at 3-5.\nQuestion: How many total yards of touchdown passes did Drew Brees make?\nAnswer: 43\n\n\n# Your Task\n\n---"
|
||||
|
||||
|
||||
def drop_few_shot(split, subset, n):
|
||||
"""es's COMPLETE 3-shot prefix: header + # Examples + exemplars.
|
||||
|
||||
The old version returned only the bare exemplars -- the '# Examples'/
|
||||
header scaffold was assembled in the runner's drop_style branch, and the
|
||||
hook path bypassed it, sending bare exemplars. Byte-diff against es's
|
||||
actual sent prompt showed the missing wrapper cost ~11 EM points on dp4.
|
||||
"""
|
||||
return ('You will be asked to read a passage and answer a question. '
|
||||
'Some examples of passages and Q&A are provided below.\n\n'
|
||||
'# Examples\n---\n' + _DROP_FEWSHOT + '\n\n# Your Task\n---\n')
|
||||
|
||||
from ..spec import DatasetSpec
|
||||
|
||||
|
||||
@ -12,6 +28,8 @@ from ..spec import DatasetSpec
|
||||
split='validation',
|
||||
few_shot_split='train',
|
||||
few_shot_num=3,
|
||||
prompt_style='drop_es', # es drop template (Passage:/Question:/bare-span Answer exemplars)
|
||||
prompt_suffix='\n\nThink step by step, then write a line of the form "Answer: [ANSWER]" at the end of your response.', # es contract
|
||||
gen_config={'temperature': 0.0, 'max_tokens': 32768},
|
||||
task_type='qa',
|
||||
tags=['reading_comprehension'],
|
||||
@ -32,3 +50,5 @@ def drop():
|
||||
)
|
||||
|
||||
return to_sample
|
||||
|
||||
|
||||
|
||||
@ -25,25 +25,45 @@ from ..spec import DatasetSpec
|
||||
tags=['knowledge', 'science'],
|
||||
description='GPQA diamond split, graduate-level science MCQ (official content).',
|
||||
params={'hub': 'hf_raw'},
|
||||
# order_policy: 'sha256' (deterministic per-question shuffle -- position
|
||||
# bias protection) | 'es-dump:<path>' (pin the exact option order es
|
||||
# used in a specific run, for same-order alignment) | 'official'
|
||||
# (keep the CSV's raw order: incorrect 1-3 then correct)
|
||||
# set via spec params at get_dataset time or the default below.
|
||||
)
|
||||
)
|
||||
def gpqa_diamond():
|
||||
def to_sample(record: dict) -> Sample:
|
||||
# position-bias protection, ported from the es adapter: deterministic
|
||||
# per-question shuffle (seed = sha256(question)) keeps reruns identical
|
||||
import hashlib
|
||||
import random as _rnd
|
||||
def gpqa_diamond(order_policy: str = 'sha256', order_dump: str = ''):
|
||||
import hashlib
|
||||
import random as _rnd
|
||||
import json as _json
|
||||
import os as _os
|
||||
|
||||
dump = {}
|
||||
if order_policy.startswith('es-dump'):
|
||||
path = order_dump or order_policy.split(':', 1)[1] if ':' in order_policy else order_dump
|
||||
path = path or _os.environ.get('EVALHARNESS_CACHE', '') + '/../gpqa_es_order.json'
|
||||
if _os.path.exists(path):
|
||||
dump = _json.load(open(path))
|
||||
|
||||
def to_sample(record: dict) -> Sample:
|
||||
choices = [
|
||||
str(record['Incorrect Answer 1'] or '').strip(),
|
||||
str(record['Incorrect Answer 2'] or '').strip(),
|
||||
str(record['Incorrect Answer 3'] or '').strip(),
|
||||
str(record['Correct Answer'] or '').strip(),
|
||||
]
|
||||
seed = int.from_bytes(
|
||||
hashlib.sha256(str(record['Question']).strip().encode('utf-8')).digest()[:8], 'big')
|
||||
_rnd.Random(seed).shuffle(choices)
|
||||
target = 'ABCD'[choices.index(str(record['Correct Answer'] or '').strip())]
|
||||
q = str(record['Question']).strip()
|
||||
if q in dump:
|
||||
# pinned order from an es run dump
|
||||
choices = list(dump[q]['order'])
|
||||
target = dump[q]['target']
|
||||
elif order_policy == 'sha256':
|
||||
seed = int.from_bytes(
|
||||
hashlib.sha256(q.encode('utf-8')).digest()[:8], 'big')
|
||||
_rnd.Random(seed).shuffle(choices)
|
||||
target = 'ABCD'[choices.index(str(record['Correct Answer'] or '').strip())]
|
||||
else: # 'official': raw order, correct is D
|
||||
target = 'D'
|
||||
return Sample(
|
||||
input=record['Question'],
|
||||
choices=choices,
|
||||
|
||||
@ -18,7 +18,7 @@ from ..spec import DatasetSpec
|
||||
few_shot_split='train',
|
||||
few_shot_num=4,
|
||||
gen_config={'temperature': 0.0, 'max_tokens': 32768},
|
||||
prompt_suffix="\nPlease reason step by step, and put your final answer within \\boxed{}",
|
||||
prompt_suffix="\nPlease reason step by step, and put your final answer within \\boxed{}.",
|
||||
task_type='math',
|
||||
tags=['math', 'cot'],
|
||||
description='Grade school math word problems (OpenAI, official).',
|
||||
|
||||
@ -20,7 +20,9 @@ _LETTERS = 'ABCD'
|
||||
def hellaswag():
|
||||
def to_sample(record: dict) -> Sample:
|
||||
return Sample(
|
||||
input=record['ctx'],
|
||||
# es adapter parity: ctx_a + ' ' + ctx_b.capitalize() -- the
|
||||
# mirror's pre-joined `ctx` keeps ctx_b lowercase, es capitalizes
|
||||
input=str(record['ctx_a']).strip() + ' ' + str(record['ctx_b']).strip().capitalize(),
|
||||
choices=list(record['endings']),
|
||||
target=_LETTERS[int(record['label'])],
|
||||
metadata={'activity_label': record.get('activity_label')},
|
||||
|
||||
@ -11,7 +11,7 @@ from ..spec import DatasetSpec
|
||||
source='MathArena/hmmt_feb_2026', # curated by MathArena (HuggingFace)
|
||||
split='train', # the dataset ships a single split
|
||||
gen_config={'temperature': 1.0, 'max_tokens': 32768},
|
||||
prompt_suffix="\nPlease reason step by step, and put your final answer within \\boxed{}",
|
||||
prompt_style='imo_es', # es template: Problem: prefix + boxed suffix,
|
||||
task_type='math',
|
||||
tags=['math', 'competition'],
|
||||
description='HMMT February 2026 (community-curated, no official upstream).',
|
||||
|
||||
@ -19,9 +19,10 @@ from ..spec import DatasetSpec
|
||||
def humaneval():
|
||||
def to_sample(record: dict) -> Sample:
|
||||
return Sample(
|
||||
input=record['prompt'],
|
||||
# es adapter: instruction header + prompt
|
||||
input='Read the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.\n' + record['prompt'],
|
||||
target=record['canonical_solution'],
|
||||
metadata={
|
||||
metadata={'prompt': record['prompt'], # original bare prompt (harness assembles from this)
|
||||
'task_id': record['task_id'],
|
||||
'test': record['test'],
|
||||
'entry_point': record['entry_point'],
|
||||
|
||||
@ -11,7 +11,7 @@ from ..spec import DatasetSpec
|
||||
source='OpenEvals/IMO-AnswerBench', # HF OpenEvals mirror of the community curation
|
||||
split='train', # the dataset ships a single split
|
||||
gen_config={'temperature': 1.0, 'max_tokens': 32768},
|
||||
prompt_suffix="\nPlease reason step by step, and put your final answer within \\boxed{}",
|
||||
prompt_style='imo_es', # es template: Problem: prefix + boxed suffix,
|
||||
task_type='math',
|
||||
tags=['math', 'competition', 'imo'],
|
||||
description='IMO-level answer bench (community-curated, no official upstream).',
|
||||
|
||||
@ -12,6 +12,7 @@ from ..spec import DatasetSpec
|
||||
# livecodebench/code_generation_lite is script-based (unloadable by datasets>=5)
|
||||
subset='release_latest', # or release_v1..v6
|
||||
split='test',
|
||||
prompt_style='lcb_es', # official LCB code contract (system+format)
|
||||
gen_config={'temperature': 1.0, 'max_tokens': 32768},
|
||||
task_type='coding',
|
||||
tags=['code'],
|
||||
@ -33,6 +34,7 @@ def live_code_bench():
|
||||
'starter_code': record.get('starter_code'),
|
||||
'public_test_cases': record.get('public_test_cases'),
|
||||
'private_test_cases': record.get('private_test_cases'),
|
||||
'raw_metadata': record.get('metadata'), # fn_name (func_name) lives here
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@ -10,6 +10,8 @@ from ..spec import DatasetSpec
|
||||
name='longbench_v2',
|
||||
source='THUDM/LongBench-v2', # official: https://huggingface.co/datasets/THUDM/LongBench-v2
|
||||
split='train', # the dataset ships a single split
|
||||
params={'filter_column': 'length'},
|
||||
prompt_style='lb2_es', # es <text> wrapper + CoT contract # subset selects length: short/medium/long
|
||||
task_type='mcq',
|
||||
tags=['long_context'],
|
||||
description='LongBench v2 long-context MCQ (official). Context kept in metadata.',
|
||||
|
||||
@ -13,6 +13,7 @@ _LETTERS = 'ABCDEFGHIJ'
|
||||
source='cais/mmlu', # official: https://huggingface.co/datasets/cais/mmlu
|
||||
subset='all', # 57 subjects; override with --subset <subject>
|
||||
split='test',
|
||||
prompt_style='cot_letter', # es contract: CoT + last-line ANSWER
|
||||
few_shot_split='dev',
|
||||
few_shot_num=5,
|
||||
task_type='mcq',
|
||||
|
||||
@ -10,6 +10,7 @@ from ..spec import DatasetSpec
|
||||
name='mmlu_pro',
|
||||
source='TIGER-Lab/MMLU-Pro', # official: https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
|
||||
split='test',
|
||||
prompt_style='cot_letter_plain', # es mmlu-pro template (Question:/Options:/A x)
|
||||
few_shot_split='validation',
|
||||
few_shot_num=5,
|
||||
task_type='mcq',
|
||||
@ -23,7 +24,8 @@ def mmlu_pro():
|
||||
input=record['question'],
|
||||
choices=list(record['options']),
|
||||
target=str(record['answer']).strip(), # already a letter
|
||||
metadata={'category': record.get('category'), 'question_id': record.get('question_id')},
|
||||
metadata={'category': record.get('category'), 'question_id': record.get('question_id'),
|
||||
'cot_content': record.get('cot_content')}, # dev-split CoT exemplars (es few-shot style)
|
||||
)
|
||||
|
||||
return to_sample
|
||||
|
||||
@ -13,6 +13,7 @@ from ..spec import DatasetSpec
|
||||
name='simple_qa',
|
||||
source='basicv8vc/SimpleQA', # mirror of the official openai/simple-evals CSV
|
||||
split='test',
|
||||
prompt_style='simple_qa_es', # es: 'Answer the question:' header, NO answer-line contract
|
||||
task_type='qa',
|
||||
tags=['factuality'],
|
||||
description='SimpleQA factuality benchmark (OpenAI, community mirror of official CSV).',
|
||||
|
||||
@ -16,6 +16,7 @@ from ..spec import DatasetSpec
|
||||
source='mandarjoshi/trivia_qa', # official: https://huggingface.co/datasets/mandarjoshi/trivia_qa
|
||||
subset='rc.wikipedia', # open-book (evalscope parity); --subset rc.nocontext for closed
|
||||
split='validation',
|
||||
prompt_style='trivia_es', # es open-book template
|
||||
task_type='qa',
|
||||
tags=['knowledge', 'openqa'],
|
||||
description='TriviaQA with Wikipedia evidence (open-book); any alias counts.',
|
||||
@ -25,21 +26,17 @@ def trivia_qa():
|
||||
def to_sample(record: dict) -> Sample:
|
||||
answer = record['answer'] # {'value': ..., 'aliases': [...], ...}
|
||||
targets = [answer['value']] + list(answer.get('aliases') or [])
|
||||
# open-book: the Wikipedia evidence document (runner prepends it via
|
||||
# metadata['context'] when assembling the prompt)
|
||||
wiki = ''
|
||||
# open-book (es parity): the FULL wiki_context list goes into the
|
||||
# prompt as Content (es adapter: record['entity_pages']['wiki_context'])
|
||||
entity = record.get('entity_pages') or {}
|
||||
for doc in (entity.get('wiki_content') or [])[:1]:
|
||||
wiki = doc or ''
|
||||
break
|
||||
search = record.get('search_results') or {}
|
||||
if not wiki:
|
||||
wiki = '\n'.join((search.get('search_context') or [])[:2])
|
||||
# keep the native shape (list[str] in real data); wrapping in list()
|
||||
# would explode a bare string into chars -- es passes it through as-is
|
||||
wiki_list = entity.get('wiki_context') or []
|
||||
return Sample(
|
||||
input=record['question'],
|
||||
target=targets, # multi-target: any alias counts
|
||||
metadata={'question_id': record.get('question_id'),
|
||||
'context': wiki or None},
|
||||
'evidence': wiki_list},
|
||||
)
|
||||
|
||||
return to_sample
|
||||
|
||||
@ -182,7 +182,9 @@ def _ms_match_files(spec: DatasetSpec, files: List[str]) -> List[str]:
|
||||
if stem == f'{spec.subset}{spec.split}':
|
||||
return True
|
||||
shard = (
|
||||
rf'{re.escape(spec.split)}-\d+-of-\d+$'
|
||||
# tolerate a content-hash suffix after the shard pattern
|
||||
# (e.g. test-00000-of-00001-6282153cc50a2626.parquet)
|
||||
rf'{re.escape(spec.split)}-\d+-of-\d+(-[0-9a-f]+)?$'
|
||||
rf'|{re.escape(spec.subset)}_{re.escape(spec.split)}[-_].*'
|
||||
rf'|{re.escape(spec.subset)}[-_]\d+$'
|
||||
)
|
||||
@ -294,6 +296,25 @@ def _hf_match_files(spec: DatasetSpec, files: List[str]) -> List[str]:
|
||||
return os.path.splitext(p)[1] in _SUPPORTED_EXTS
|
||||
|
||||
data = [f for f in files if ext_ok(f)]
|
||||
# multi-window subset, e.g. 'release_v2_v4' = union of release_v2..v4
|
||||
# shards (LCB release-window convention: per-release files are disjoint)
|
||||
m = re.fullmatch(r'(release_v\d+)((?:_v\d+)+)', spec.subset)
|
||||
if m:
|
||||
members = [m.group(1)] + [f'release{x}' for x in m.group(2).split('_') if x]
|
||||
picked = []
|
||||
for mem in members:
|
||||
picked += [
|
||||
f for f in data
|
||||
if re.fullmatch(rf'{re.escape(mem)}[-_]\d+(-of-\d+)?'
|
||||
rf'|{re.escape(mem)}_{re.escape(spec.split)}[-_].*'
|
||||
rf'|{re.escape(mem)}',
|
||||
os.path.splitext(os.path.basename(f))[0])
|
||||
]
|
||||
if picked:
|
||||
# deterministic: member order, shard order within each member
|
||||
return [f for mem in members for f in sorted(
|
||||
p for p in picked
|
||||
if os.path.splitext(os.path.basename(p))[0].startswith(mem))]
|
||||
if spec.subset != 'default':
|
||||
# subset wins exclusively: never ALSO match bare-split shards in the
|
||||
# same dir (repos like sam-paech LCB mix test-*.parquet and
|
||||
@ -389,7 +410,9 @@ def _load_from_hub(spec: DatasetSpec) -> List[Dict[str, Any]]:
|
||||
'install it first: pip install evalharness (light deps are default)'
|
||||
)
|
||||
kwargs = {k: v for k, v in spec.params.items() if k not in _RESERVED_PARAMS}
|
||||
subset = None if spec.subset == 'default' else spec.subset
|
||||
# filter_column subsets select rows by column value post-load, so the hub
|
||||
# load itself must always use the default config.
|
||||
subset = None if (spec.subset == 'default' or spec.params.get('filter_column')) else spec.subset
|
||||
ds = datasets.load_dataset(spec.source, subset, split=spec.split, revision=spec.version, **kwargs)
|
||||
return [dict(r) for r in ds]
|
||||
|
||||
|
||||
@ -83,7 +83,14 @@ def register_dataset(spec: DatasetSpec):
|
||||
"""
|
||||
|
||||
def decorator(factory: ProviderFactory) -> ProviderFactory:
|
||||
DATASET_REGISTRY.register(spec.name, DatasetProvider(spec, factory))
|
||||
provider = DatasetProvider(spec, factory)
|
||||
# few-shot hook convention: a module-level `<name>_few_shot(split,
|
||||
# subset, n) -> Optional[str]` next to the plugin is picked up here,
|
||||
# so runner can inject official hand-written exemplars (e.g. bbh CoT).
|
||||
hook = factory.__globals__.get(f'{spec.name}_few_shot')
|
||||
if callable(hook):
|
||||
provider.few_shot_hook = hook
|
||||
DATASET_REGISTRY.register(spec.name, provider)
|
||||
return factory
|
||||
|
||||
return decorator
|
||||
|
||||
0
evalharness/eval/_lcb_official/__init__.py
Normal file
0
evalharness/eval/_lcb_official/__init__.py
Normal file
196
evalharness/eval/_lcb_official/evaluate_utils.py
Normal file
196
evalharness/eval/_lcb_official/evaluate_utils.py
Normal file
@ -0,0 +1,196 @@
|
||||
import json
|
||||
import multiprocessing
|
||||
import numpy as np
|
||||
from collections import defaultdict
|
||||
|
||||
from evalscope.utils.logger import get_logger
|
||||
from .pass_k_utils import compute_metrics_from_results
|
||||
|
||||
logger = get_logger()
|
||||
|
||||
|
||||
def _temp_run(sample, generation, debug, result, metadata_list, timeout):
|
||||
"""Runs a test in a separate process to enforce a timeout.
|
||||
This function is defined at the module's top level to ensure it can be
|
||||
pickled by `multiprocessing.Process`. This is a requirement on platforms
|
||||
like macOS (on Apple Silicon) which use the 'spawn' start method, as
|
||||
nested functions are not picklable.
|
||||
"""
|
||||
from .testing_util import run_test
|
||||
res, metadata = run_test(sample, test=generation, debug=debug, timeout=timeout)
|
||||
result.append(res)
|
||||
metadata_list.append(metadata)
|
||||
|
||||
|
||||
def codegen_check_correctness(sample, generation, timeout, debug=True):
|
||||
"""Check correctness of code generation with a global timeout.
|
||||
|
||||
The global timeout is to catch some extreme/rare cases not handled by the
|
||||
timeouts inside `run_test`
|
||||
"""
|
||||
|
||||
ctx = multiprocessing.get_context('spawn')
|
||||
manager = ctx.Manager()
|
||||
result = manager.list()
|
||||
metadata_list = manager.list()
|
||||
p = ctx.Process(
|
||||
target=_temp_run,
|
||||
args=(sample, generation, debug, result, metadata_list, timeout),
|
||||
)
|
||||
p.start()
|
||||
global_timeout = (timeout + 1) * len(json.loads(sample['input_output'])['inputs'])
|
||||
if debug:
|
||||
logger.info(f'global timeout = {global_timeout}')
|
||||
p.join(timeout=global_timeout)
|
||||
if p.is_alive():
|
||||
p.kill()
|
||||
if not result:
|
||||
in_outs = json.loads(sample['input_output'])
|
||||
# consider that all tests failed
|
||||
result = [[-1 for i in range(len(in_outs['inputs']))]]
|
||||
if debug:
|
||||
logger.info('global timeout occured: alarm went off')
|
||||
return result[0], metadata_list[0]
|
||||
|
||||
|
||||
def evaluate_generations_by_problem(problem_generations: list, sample: list, debug: bool, timeout: int):
|
||||
"""Evaluate each problem.
|
||||
|
||||
Args:
|
||||
problem_generations:
|
||||
sample:
|
||||
debug:
|
||||
timeout
|
||||
"""
|
||||
# problem_generations: list[str] = args[0]
|
||||
# sample = args[1]
|
||||
# debug: bool = args[2]
|
||||
# timeout: int = args[3]
|
||||
|
||||
res = []
|
||||
metadata = []
|
||||
for o_idx, o in enumerate(problem_generations):
|
||||
curr_res = [-2]
|
||||
try:
|
||||
curr_res, curr_metadata = codegen_check_correctness(sample, o, timeout=timeout, debug=debug)
|
||||
if debug:
|
||||
logger.info(f'\nSuccessful compilation of task {o_idx}!')
|
||||
fixed = []
|
||||
for e in curr_res:
|
||||
if isinstance(e, np.ndarray):
|
||||
e = e.item(0)
|
||||
if isinstance(e, np.bool_):
|
||||
e = bool(e)
|
||||
fixed.append(e)
|
||||
curr_res = fixed
|
||||
if not np.all(curr_res):
|
||||
if debug:
|
||||
logger.info(f'Results were not True for all test cases' # noqa: F541, E501
|
||||
f' {curr_res=}\n')
|
||||
except Exception as e:
|
||||
if debug:
|
||||
logger.info(f'Compilation failed, test framework exception' # noqa: F541, E501
|
||||
f' = {repr(e)}{e}\n')
|
||||
# break
|
||||
curr_metadata = {}
|
||||
finally:
|
||||
assert isinstance(curr_res, list)
|
||||
assert isinstance(curr_metadata, dict)
|
||||
res.append(curr_res)
|
||||
metadata.append(curr_metadata)
|
||||
if debug:
|
||||
for i, r in enumerate(problem_generations):
|
||||
logger.info(f'Sample\n{r}\nResult\n{res[i]}')
|
||||
logger.info('*' * 30 + '\n\n')
|
||||
return res, metadata
|
||||
|
||||
|
||||
def evaluate_generations(
|
||||
samples_list: list,
|
||||
generations_list: list[list[str]],
|
||||
debug: bool = False,
|
||||
num_process_evaluate: int = 16, # This parameter will be unused
|
||||
timeout=6,
|
||||
):
|
||||
"""We take the list of code generations and try to compile them and the run
|
||||
their corresponding unit tests which are retrieved from the APPS dataset.
|
||||
|
||||
Args:
|
||||
generations: list of code generations (same order as samples in APPS
|
||||
dataset)
|
||||
level: difficulty level used in the generation, can be "all",
|
||||
"introductory", "interview" or "competition"
|
||||
|
||||
Returns:
|
||||
results: dictionary of results, key is the problem index, value is
|
||||
a list of results for each generation
|
||||
[-2] = compile error, [-1] = runtime error [False] = failed test
|
||||
case [True] = passed test case
|
||||
"""
|
||||
results = {}
|
||||
metadata = {}
|
||||
|
||||
for index in range(len(generations_list)):
|
||||
problem_generations = generations_list[index]
|
||||
sample = samples_list[index]
|
||||
|
||||
result, meta = evaluate_generations_by_problem(problem_generations, sample, debug, timeout)
|
||||
results[index] = result
|
||||
metadata[index] = meta
|
||||
|
||||
assert len(results
|
||||
) == len(generations_list), f'results = {len(results)} inputs = {len(generations_list)} {results=}'
|
||||
|
||||
return results, metadata
|
||||
|
||||
|
||||
def codegen_metrics(
|
||||
samples_list,
|
||||
generations_list,
|
||||
k_list=[1, 5, 10, 20, 40, 50, 75, 100, 125, 150, 200, 500, 1000],
|
||||
num_process_evaluate=16,
|
||||
timeout=6,
|
||||
debug=False,
|
||||
):
|
||||
|
||||
samples_linear = []
|
||||
generations_linear = []
|
||||
remap_index = []
|
||||
results = defaultdict(list)
|
||||
metadatas = defaultdict(list)
|
||||
for idx, (sample, generation_list) in enumerate(zip(samples_list, generations_list)):
|
||||
assert isinstance(generation_list, list), generations_list[0]
|
||||
for generation in generation_list:
|
||||
assert isinstance(generation, str), generations_list[0]
|
||||
samples_linear.append(sample)
|
||||
generations_linear.append([generation])
|
||||
remap_index.append(idx)
|
||||
|
||||
results_linear, metadatas_linear = evaluate_generations(
|
||||
samples_linear,
|
||||
generations_linear,
|
||||
debug=debug,
|
||||
num_process_evaluate=num_process_evaluate,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
for idx, sub_results in sorted(results_linear.items(), key=lambda x: x[0]):
|
||||
results[remap_index[idx]].append(sub_results[0])
|
||||
|
||||
for idx, sub_metadatas in sorted(metadatas_linear.items(), key=lambda x: x[0]):
|
||||
metadatas[remap_index[idx]].append(sub_metadatas[0])
|
||||
|
||||
metrics = compute_metrics_from_results(results, k_list=k_list)
|
||||
|
||||
final_metadata = []
|
||||
for key in sorted(list(metadatas.keys())):
|
||||
final_metadata.append(metadatas[key])
|
||||
for i in range(len(final_metadata)):
|
||||
if type(final_metadata[i]) is not list:
|
||||
final_metadata[i] = [json.dumps(final_metadata[i])]
|
||||
else:
|
||||
final_metadata[i] = [json.dumps(x) for x in final_metadata[i]]
|
||||
|
||||
assert len(final_metadata[i]) == len(generations_list[0]), f'{len(final_metadata[i])=}'
|
||||
|
||||
return [metrics, results, final_metadata]
|
||||
70
evalharness/eval/_lcb_official/extract_utils.py
Normal file
70
evalharness/eval/_lcb_official/extract_utils.py
Normal file
@ -0,0 +1,70 @@
|
||||
# Copyright LiveCodeBench @ 2024,
|
||||
|
||||
import re
|
||||
|
||||
|
||||
def extract_code_generation(model_output: str, model_type: str = 'chat'):
|
||||
# modified from
|
||||
outputlines = model_output.split('\n')
|
||||
# TODO: handle codellama
|
||||
|
||||
if model_type == 'base':
|
||||
return model_output.strip()
|
||||
elif model_type == 'chat':
|
||||
indexlines = [i for i, line in enumerate(outputlines) if '```' in line]
|
||||
else:
|
||||
raise ValueError(f'Invalid mode type: {model_type}')
|
||||
|
||||
if len(indexlines) < 2:
|
||||
return ''
|
||||
return '\n'.join(outputlines[indexlines[-2] + 1:indexlines[-1]])
|
||||
|
||||
|
||||
def extract_code_execution(model_output: str, cot: bool = False):
|
||||
pattern = r'\[PYTHON\](.*?)\[\/PYTHON\]'
|
||||
matches = re.findall(pattern, model_output, re.DOTALL)
|
||||
if matches:
|
||||
# fetch the last one
|
||||
model_output = matches[-1]
|
||||
|
||||
if '[PYTHON]' in model_output:
|
||||
model_output
|
||||
if cot:
|
||||
if '[ANSWER]' in model_output:
|
||||
model_output = model_output.split('[ANSWER]')[1].strip()
|
||||
if '==' in model_output:
|
||||
model_output = model_output.split('==')[1].strip()
|
||||
if '[/ANSWER]' in model_output:
|
||||
model_output = model_output.split('[/ANSWER]')[0].strip()
|
||||
else:
|
||||
model_output = model_output.split('\n')[0].strip()
|
||||
return model_output.strip()
|
||||
|
||||
|
||||
def extract_test_output_code(model_output: str):
|
||||
outputlines = model_output.split('\n')
|
||||
# find the last line startwith assert...
|
||||
indexlines = [i for i, line in enumerate(outputlines) if line.startswith('assert')]
|
||||
if indexlines:
|
||||
return outputlines[indexlines[-1]]
|
||||
|
||||
# TODO: handle codellama format
|
||||
# if lmstyle and lmstyle == LMStyle.CodeLLaMaInstruct:
|
||||
# indexlines = \
|
||||
# [i for i, line in enumerate(outputlines) if "PYTHON]" in line]
|
||||
# else:
|
||||
|
||||
# first try to extract ```python if not then try ```
|
||||
indexlines = [i for i, line in enumerate(outputlines) if '```python' in line or '```Python' in line]
|
||||
if indexlines:
|
||||
start_index = indexlines[0]
|
||||
else:
|
||||
start_index = None
|
||||
indexlines = [i for i, line in enumerate(outputlines) if '```' in line]
|
||||
if start_index is not None:
|
||||
indexlines = [i for i in indexlines if i > start_index]
|
||||
indexlines = [start_index] + indexlines
|
||||
|
||||
if len(indexlines) < 2:
|
||||
return ''
|
||||
return '\n'.join(outputlines[indexlines[0] + 1:indexlines[1]])
|
||||
56
evalharness/eval/_lcb_official/pass_k_utils.py
Normal file
56
evalharness/eval/_lcb_official/pass_k_utils.py
Normal file
@ -0,0 +1,56 @@
|
||||
# Copyright LiveCodeBench @ 2024,
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
def estimate_pass_at_k(num_samples, num_correct, k):
|
||||
"""Estimates pass@k of each problem and returns them in an array."""
|
||||
|
||||
def estimator(n: int, c: int, k: int) -> float:
|
||||
"""Calculates 1 - comb(n - c, k) / comb(n, k)."""
|
||||
if n - c < k:
|
||||
return 1.0 * 100
|
||||
return 100 * (1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))
|
||||
|
||||
import itertools
|
||||
|
||||
if isinstance(num_samples, int):
|
||||
num_samples_it = itertools.repeat(num_samples, len(num_correct))
|
||||
else:
|
||||
assert len(num_samples) == len(num_correct)
|
||||
num_samples_it = iter(num_samples)
|
||||
|
||||
return np.array([estimator(int(n), int(c), k) for n, c in zip(num_samples_it, num_correct)])
|
||||
|
||||
|
||||
def compute_metrics_from_results(results, k_list=[1, 5]):
|
||||
total = []
|
||||
correct = []
|
||||
task_ids = []
|
||||
for task_id, res in results.items():
|
||||
all_correct = []
|
||||
for generation in res:
|
||||
gen = np.array(generation)
|
||||
all_correct.append(np.all(gen > 0))
|
||||
task_ids.append(task_id)
|
||||
total.append(len(all_correct))
|
||||
correct.append(sum(all_correct))
|
||||
total = np.array(total)
|
||||
correct = np.array(correct)
|
||||
ks = k_list
|
||||
detail_pass_at_k = {f'pass@{k}': estimate_pass_at_k(total, correct, k).tolist() for k in ks if (total >= k).all()}
|
||||
pass_at_k = {f'pass@{k}': estimate_pass_at_k(total, correct, k).mean() for k in ks if (total >= k).all()}
|
||||
detail_metrics = {k: dict(zip(task_ids, v)) for k, v in detail_pass_at_k.items()}
|
||||
pass_at_k['detail'] = detail_metrics
|
||||
return pass_at_k
|
||||
|
||||
|
||||
def extract_instance_results(results):
|
||||
instance_wise_grades = {}
|
||||
for task_id, res in results.items():
|
||||
instance_wise_grades[task_id] = []
|
||||
for generation in res:
|
||||
instance_wise_grades[task_id].append(all([g > 0 for g in generation]))
|
||||
|
||||
instance_wise_grades = [v for _, v in sorted(instance_wise_grades.items(), key=lambda item: item[0])]
|
||||
return instance_wise_grades
|
||||
555
evalharness/eval/_lcb_official/testing_util.py
Normal file
555
evalharness/eval/_lcb_official/testing_util.py
Normal file
@ -0,0 +1,555 @@
|
||||
# flake8: noqa
|
||||
import ast
|
||||
import faulthandler
|
||||
import json
|
||||
import numpy as np
|
||||
import platform
|
||||
|
||||
# to run the solution files we're using a timing based approach
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
|
||||
# used for debugging to time steps
|
||||
from datetime import datetime
|
||||
from decimal import Decimal
|
||||
from enum import Enum
|
||||
from functools import partial
|
||||
from io import BytesIO, StringIO, TextIOWrapper
|
||||
|
||||
# from pyext import RuntimeModule
|
||||
from types import ModuleType
|
||||
|
||||
# used for testing the code that reads from input
|
||||
from unittest.mock import mock_open, patch
|
||||
|
||||
from evalscope.utils.io_utils import current_time
|
||||
from evalscope.utils.logger import get_logger
|
||||
|
||||
logger = get_logger()
|
||||
|
||||
import_string = 'from string import *\nfrom re import *\nfrom datetime import *\nfrom collections import *\nfrom heapq import *\nfrom bisect import *\nfrom copy import *\nfrom math import *\nfrom random import *\nfrom statistics import *\nfrom itertools import *\nfrom functools import *\nfrom operator import *\nfrom io import *\nfrom sys import *\nfrom json import *\nfrom builtins import *\nfrom typing import *\nimport string\nimport re\nimport datetime\nimport collections\nimport heapq\nimport bisect\nimport copy\nimport math\nimport random\nimport statistics\nimport itertools\nimport functools\nimport operator\nimport io\nimport sys\nimport json\nsys.setrecursionlimit(50000)\n'
|
||||
|
||||
|
||||
def truncatefn(s, length=300):
|
||||
if isinstance(s, str):
|
||||
pass
|
||||
else:
|
||||
s = str(s)
|
||||
if len(s) <= length:
|
||||
return s
|
||||
|
||||
return s[:length // 2] + '...(truncated) ...' + s[-length // 2:]
|
||||
|
||||
|
||||
class CODE_TYPE(Enum):
|
||||
call_based = 0
|
||||
standard_input = 1
|
||||
|
||||
|
||||
# stuff for setting up signal timer
|
||||
class TimeoutException(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def timeout_handler(debug, signum, frame):
|
||||
if debug:
|
||||
logger.info('timeout occured: alarm went off')
|
||||
raise TimeoutException
|
||||
|
||||
|
||||
def _set_alarm(seconds: float) -> None:
|
||||
# setitimer preserves sub-second precision; signal.alarm() only accepts
|
||||
# ints, so a float timeout would truncate (e.g. 0.5 -> 0 cancels the
|
||||
# timeout, 1.9 -> 1 fires early). setitimer(ITIMER_REAL, 0) cancels, matching
|
||||
# alarm(0). Delivers SIGALRM, so the existing handler still fires.
|
||||
if hasattr(signal, 'setitimer') and hasattr(signal, 'SIGALRM') and hasattr(signal, 'ITIMER_REAL'):
|
||||
signal.setitimer(signal.ITIMER_REAL, seconds)
|
||||
|
||||
|
||||
# used to capture stdout as a list
|
||||
# from https://stackoverflow.com/a/16571630/6416660
|
||||
# alternative use redirect_stdout() from contextlib
|
||||
class Capturing(list):
|
||||
|
||||
def __enter__(self):
|
||||
self._stdout = sys.stdout
|
||||
sys.stdout = self._stringio = StringIO()
|
||||
# Make closing the StringIO a no-op
|
||||
self._stringio.close = lambda x: 1
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
self.append(self._stringio.getvalue())
|
||||
del self._stringio # free up some memory
|
||||
sys.stdout = self._stdout
|
||||
|
||||
|
||||
def clean_if_name(code: str) -> str:
|
||||
try:
|
||||
astree = ast.parse(code)
|
||||
last_block = astree.body[-1]
|
||||
if isinstance(last_block, ast.If):
|
||||
condition = last_block.test
|
||||
if ast.unparse(condition).strip() == "__name__ == '__main__'":
|
||||
code = (
|
||||
ast.unparse(astree.body[:-1]) + '\n' + ast.unparse(last_block.body) # type: ignore
|
||||
)
|
||||
except:
|
||||
pass
|
||||
|
||||
return code
|
||||
|
||||
|
||||
def make_function(code: str) -> str:
|
||||
try:
|
||||
import_stmts = []
|
||||
all_other_stmts = []
|
||||
astree = ast.parse(code)
|
||||
for stmt in astree.body:
|
||||
if isinstance(stmt, (ast.Import, ast.ImportFrom)):
|
||||
import_stmts.append(stmt)
|
||||
else:
|
||||
all_other_stmts.append(stmt)
|
||||
|
||||
function_ast = ast.FunctionDef(
|
||||
name='wrapped_function',
|
||||
args=ast.arguments(posonlyargs=[], args=[], kwonlyargs=[], kw_defaults=[], defaults=[]),
|
||||
body=all_other_stmts,
|
||||
decorator_list=[],
|
||||
lineno=-1,
|
||||
)
|
||||
main_code = (
|
||||
import_string + '\n' + ast.unparse(import_stmts) # type: ignore
|
||||
+ '\n' + ast.unparse(function_ast) # type: ignore
|
||||
)
|
||||
return main_code
|
||||
except Exception as e:
|
||||
return code
|
||||
|
||||
|
||||
def call_method(method, inputs):
|
||||
|
||||
if isinstance(inputs, list):
|
||||
inputs = '\n'.join(inputs)
|
||||
|
||||
inputs_line_iterator = iter(inputs.split('\n'))
|
||||
|
||||
# sys.setrecursionlimit(10000)
|
||||
|
||||
# @patch('builtins.input', side_effect=inputs.split("\n"))
|
||||
@patch('builtins.open', mock_open(read_data=inputs))
|
||||
@patch('sys.stdin', TextIOWrapper(BytesIO(inputs.encode('utf-8')), encoding='utf-8'))
|
||||
@patch('sys.stdin.readline', lambda *args: next(inputs_line_iterator))
|
||||
@patch('sys.stdin.readlines', lambda *args: inputs.split('\n'))
|
||||
@patch('sys.stdin.read', lambda *args: inputs)
|
||||
# @patch('sys.stdout.write', print)
|
||||
def _inner_call_method(_method):
|
||||
try:
|
||||
return _method()
|
||||
except SystemExit as e:
|
||||
pass
|
||||
finally:
|
||||
pass
|
||||
|
||||
return _inner_call_method(method)
|
||||
|
||||
|
||||
def get_function(compiled_sol, fn_name: str): # type: ignore
|
||||
try:
|
||||
assert hasattr(compiled_sol, fn_name)
|
||||
return getattr(compiled_sol, fn_name)
|
||||
except Exception as e:
|
||||
return
|
||||
|
||||
|
||||
def compile_code(code: str, timeout: int):
|
||||
_set_alarm(timeout)
|
||||
try:
|
||||
tmp_sol = ModuleType('tmp_sol', '')
|
||||
exec(code, tmp_sol.__dict__)
|
||||
if 'class Solution' in code:
|
||||
# leetcode wraps solutions in `Solution`
|
||||
# this is a hack to check if it is leetcode solution or not
|
||||
# currently livecodebench only supports LeetCode but
|
||||
# else condition allows future extensibility to other platforms
|
||||
compiled_sol = tmp_sol.Solution()
|
||||
else:
|
||||
# do nothing in the other case since function is accesible
|
||||
compiled_sol = tmp_sol
|
||||
|
||||
assert compiled_sol is not None
|
||||
finally:
|
||||
_set_alarm(0)
|
||||
|
||||
return compiled_sol
|
||||
|
||||
|
||||
def convert_line_to_decimals(line: str) -> tuple[bool, list[Decimal]]:
|
||||
try:
|
||||
decimal_line = [Decimal(elem) for elem in line.split()]
|
||||
except:
|
||||
return False, []
|
||||
return True, decimal_line
|
||||
|
||||
|
||||
def get_stripped_lines(val: str):
|
||||
## you don't want empty lines to add empty list after splitlines!
|
||||
val = val.strip()
|
||||
|
||||
return [val_line.strip() for val_line in val.split('\n')]
|
||||
|
||||
|
||||
def grade_call_based(code: str, all_inputs: list, all_outputs: list, fn_name: str, timeout: int):
|
||||
# call-based clean up logic
|
||||
# need to wrap in try-catch logic after to catch the correct errors, but for now this is fine.
|
||||
code = import_string + '\n\n' + code
|
||||
compiled_sol = compile_code(code, timeout)
|
||||
|
||||
if compiled_sol is None:
|
||||
return
|
||||
|
||||
method = get_function(compiled_sol, fn_name)
|
||||
|
||||
if method is None:
|
||||
return
|
||||
|
||||
all_inputs = [[json.loads(line) for line in inputs.split('\n')] for inputs in all_inputs]
|
||||
|
||||
all_outputs = [json.loads(output) for output in all_outputs]
|
||||
|
||||
total_execution = 0
|
||||
all_results = []
|
||||
for idx, (gt_inp, gt_out) in enumerate(zip(all_inputs, all_outputs)):
|
||||
_set_alarm(timeout)
|
||||
# faulthandler.enable()
|
||||
try:
|
||||
# can lock here so time is useful
|
||||
start = time.time()
|
||||
prediction = method(*gt_inp)
|
||||
total_execution += time.time() - start
|
||||
_set_alarm(0)
|
||||
|
||||
# don't penalize model if it produces tuples instead of lists
|
||||
# ground truth sequences are not tuples
|
||||
if isinstance(prediction, tuple):
|
||||
prediction = list(prediction)
|
||||
|
||||
tmp_result = prediction == gt_out
|
||||
|
||||
# handle floating point comparisons
|
||||
|
||||
all_results.append(tmp_result)
|
||||
|
||||
if not tmp_result:
|
||||
return all_results, {
|
||||
'output': truncatefn(prediction),
|
||||
'inputs': truncatefn(gt_inp),
|
||||
'expected': truncatefn(gt_out),
|
||||
'error_code': -2,
|
||||
'error_message': 'Wrong Answer',
|
||||
}
|
||||
except Exception as e:
|
||||
_set_alarm(0)
|
||||
if 'timeoutexception' in repr(e).lower():
|
||||
all_results.append(-3)
|
||||
return all_results, {
|
||||
'error': repr(e),
|
||||
'error_code': -3,
|
||||
'error_message': 'Time Limit Exceeded',
|
||||
'inputs': truncatefn(gt_inp),
|
||||
'expected': truncatefn(gt_out),
|
||||
}
|
||||
else:
|
||||
all_results.append(-4)
|
||||
return all_results, {
|
||||
'error': repr(e),
|
||||
'error_code': -4,
|
||||
'error_message': 'Runtime Error',
|
||||
'inputs': truncatefn(gt_inp),
|
||||
'expected': truncatefn(gt_out),
|
||||
}
|
||||
|
||||
finally:
|
||||
_set_alarm(0)
|
||||
# faulthandler.disable()
|
||||
|
||||
return all_results, {'execution time': total_execution}
|
||||
|
||||
|
||||
def grade_stdio(
|
||||
code: str,
|
||||
all_inputs: list,
|
||||
all_outputs: list,
|
||||
timeout: int,
|
||||
):
|
||||
## runtime doesn't interact well with __name__ == '__main__'
|
||||
code = clean_if_name(code)
|
||||
|
||||
## we wrap the given code inside another function
|
||||
code = make_function(code)
|
||||
|
||||
compiled_sol = compile_code(code, timeout)
|
||||
if compiled_sol is None:
|
||||
return
|
||||
|
||||
method = get_function(compiled_sol, 'wrapped_function')
|
||||
|
||||
if method is None:
|
||||
return
|
||||
|
||||
all_results = []
|
||||
total_execution_time = 0
|
||||
for idx, (gt_inp, gt_out) in enumerate(zip(all_inputs, all_outputs)):
|
||||
_set_alarm(timeout)
|
||||
# faulthandler.enable()
|
||||
|
||||
with Capturing() as captured_output:
|
||||
try:
|
||||
start = time.time()
|
||||
call_method(method, gt_inp)
|
||||
total_execution_time += time.time() - start
|
||||
# reset the alarm
|
||||
_set_alarm(0)
|
||||
except Exception as e:
|
||||
_set_alarm(0)
|
||||
if 'timeoutexception' in repr(e).lower():
|
||||
all_results.append(-3)
|
||||
return all_results, {
|
||||
'error': repr(e),
|
||||
'error_code': -3,
|
||||
'error_message': 'Time Limit Exceeded',
|
||||
'inputs': truncatefn(gt_inp),
|
||||
'expected': truncatefn(gt_out),
|
||||
}
|
||||
else:
|
||||
all_results.append(-4)
|
||||
return all_results, {
|
||||
'error': repr(e),
|
||||
'error_code': -4,
|
||||
'error_message': 'Runtime Error',
|
||||
'inputs': truncatefn(gt_inp),
|
||||
'expected': truncatefn(gt_out),
|
||||
}
|
||||
|
||||
finally:
|
||||
_set_alarm(0)
|
||||
# faulthandler.disable()
|
||||
|
||||
prediction = captured_output[0]
|
||||
|
||||
stripped_prediction_lines = get_stripped_lines(prediction)
|
||||
stripped_gt_out_lines = get_stripped_lines(gt_out)
|
||||
|
||||
## WA happens in multiple circumstances
|
||||
## so cache the return to make it clean!
|
||||
WA_send_args = {
|
||||
'output': truncatefn(prediction),
|
||||
'inputs': truncatefn(gt_inp),
|
||||
'expected': truncatefn(gt_out),
|
||||
'error_code': -2,
|
||||
}
|
||||
|
||||
if len(stripped_prediction_lines) != len(stripped_gt_out_lines):
|
||||
all_results.append(-2)
|
||||
WA_send_args['error_message'] = 'Wrong answer: mismatched output length'
|
||||
return all_results, WA_send_args
|
||||
|
||||
for output_line_idx, (
|
||||
stripped_prediction_line,
|
||||
stripped_gt_out_line,
|
||||
) in enumerate(zip(stripped_prediction_lines, stripped_gt_out_lines)):
|
||||
WA_send_args['error_message'] = (
|
||||
f'Wrong answer at {output_line_idx=}: {truncatefn(stripped_prediction_line)} != {truncatefn(stripped_gt_out_line)}'
|
||||
)
|
||||
|
||||
## CASE 1: exact match
|
||||
if stripped_prediction_line == stripped_gt_out_line:
|
||||
continue
|
||||
|
||||
## CASE 2: element-wise comparision
|
||||
## if there are floating elements
|
||||
## use `decimal` library for good floating point comparision
|
||||
## otherwise gotcha: np.isclose(50000000000000000, 50000000000000001) = True
|
||||
## note that we should always be able to convert to decimals
|
||||
|
||||
success, decimal_prediction_line = convert_line_to_decimals(stripped_prediction_line)
|
||||
if not success:
|
||||
all_results.append(-2)
|
||||
return all_results, WA_send_args
|
||||
success, decimal_gtout_line = convert_line_to_decimals(stripped_gt_out_line)
|
||||
if not success:
|
||||
all_results.append(-2)
|
||||
return all_results, WA_send_args
|
||||
|
||||
if decimal_prediction_line == decimal_gtout_line:
|
||||
continue
|
||||
|
||||
all_results.append(-2)
|
||||
return all_results, WA_send_args
|
||||
all_results.append(True)
|
||||
|
||||
return all_results, {'execution time': total_execution_time}
|
||||
|
||||
|
||||
def run_test(sample, test=None, debug=False, timeout=6):
|
||||
"""
|
||||
if test(generated_code) is not None it'll try to run the code.
|
||||
otherwise it'll just return an input and output pair.
|
||||
"""
|
||||
timeout_handler_wrapper = partial(timeout_handler, debug)
|
||||
if hasattr(signal, 'setitimer') and hasattr(signal, 'SIGALRM') and hasattr(signal, 'ITIMER_REAL'):
|
||||
signal.signal(signal.SIGALRM, timeout_handler_wrapper)
|
||||
|
||||
# Disable functionalities that can make destructive changes to the test.
|
||||
# max memory is set to 4GB
|
||||
reliability_guard()
|
||||
|
||||
if debug:
|
||||
logger.info(f'start = {current_time().time()}')
|
||||
|
||||
try:
|
||||
in_outs = json.loads(sample['input_output'])
|
||||
except ValueError as e:
|
||||
raise e
|
||||
in_outs = None
|
||||
|
||||
if in_outs:
|
||||
if in_outs.get('fn_name') is None:
|
||||
which_type = CODE_TYPE.standard_input # Standard input
|
||||
method_name = None
|
||||
|
||||
else:
|
||||
which_type = CODE_TYPE.call_based # Call-based
|
||||
method_name = in_outs['fn_name']
|
||||
|
||||
if debug:
|
||||
logger.info(f'loaded input_output = {current_time().time()}')
|
||||
|
||||
if test is None:
|
||||
assert False, 'should not happen: test code is none'
|
||||
return in_outs, {'error': 'no test code provided'}
|
||||
elif test is not None:
|
||||
results = []
|
||||
sol = import_string
|
||||
if debug:
|
||||
logger.info(f'loading test code = {current_time().time()}')
|
||||
|
||||
if which_type == CODE_TYPE.call_based:
|
||||
_set_alarm(timeout)
|
||||
try:
|
||||
results, metadata = grade_call_based(
|
||||
code=test,
|
||||
all_inputs=in_outs['inputs'],
|
||||
all_outputs=in_outs['outputs'],
|
||||
fn_name=method_name,
|
||||
timeout=timeout,
|
||||
)
|
||||
return results, metadata
|
||||
except Exception as e:
|
||||
return [-4], {
|
||||
'error_code': -4,
|
||||
'error_message': f'Error during testing: {e}',
|
||||
}
|
||||
finally:
|
||||
_set_alarm(0)
|
||||
elif which_type == CODE_TYPE.standard_input:
|
||||
# sol
|
||||
# if code has if __name__ == "__main__": then remove it
|
||||
|
||||
_set_alarm(timeout)
|
||||
try:
|
||||
results, metadata = grade_stdio(
|
||||
code=test,
|
||||
all_inputs=in_outs['inputs'],
|
||||
all_outputs=in_outs['outputs'],
|
||||
timeout=timeout,
|
||||
)
|
||||
return results, metadata
|
||||
except Exception as e:
|
||||
return [-4], {
|
||||
'error_code': -4,
|
||||
'error_message': f'Error during testing: {e}',
|
||||
}
|
||||
finally:
|
||||
_set_alarm(0)
|
||||
|
||||
|
||||
def reliability_guard(maximum_memory_bytes=None):
|
||||
"""
|
||||
This disables various destructive functions and prevents the generated code
|
||||
from interfering with the test (e.g. fork bomb, killing other processes,
|
||||
removing filesystem files, etc.)
|
||||
WARNING
|
||||
This function is NOT a security sandbox. Untrusted code, including, model-
|
||||
generated code, should not be blindly executed outside of one. See the
|
||||
Codex paper for more information about OpenAI's code sandbox, and proceed
|
||||
with caution.
|
||||
"""
|
||||
|
||||
if maximum_memory_bytes is not None:
|
||||
import resource
|
||||
|
||||
resource.setrlimit(resource.RLIMIT_AS, (maximum_memory_bytes, maximum_memory_bytes))
|
||||
resource.setrlimit(resource.RLIMIT_DATA, (maximum_memory_bytes, maximum_memory_bytes))
|
||||
if not platform.uname().system == 'Darwin':
|
||||
resource.setrlimit(resource.RLIMIT_STACK, (maximum_memory_bytes, maximum_memory_bytes))
|
||||
|
||||
# faulthandler.disable()
|
||||
|
||||
import builtins
|
||||
|
||||
# builtins.exit = None
|
||||
builtins.quit = None
|
||||
|
||||
import os
|
||||
|
||||
os.environ['OMP_NUM_THREADS'] = '1'
|
||||
|
||||
os.kill = None
|
||||
os.system = None
|
||||
os.putenv = None
|
||||
os.remove = None
|
||||
os.removedirs = None
|
||||
os.rmdir = None
|
||||
os.fchdir = None
|
||||
os.setuid = None
|
||||
os.fork = None
|
||||
os.forkpty = None
|
||||
os.killpg = None
|
||||
os.rename = None
|
||||
os.renames = None
|
||||
os.truncate = None
|
||||
os.replace = None
|
||||
os.unlink = None
|
||||
os.fchmod = None
|
||||
os.fchown = None
|
||||
os.chmod = None
|
||||
os.chown = None
|
||||
os.chroot = None
|
||||
os.fchdir = None
|
||||
os.lchflags = None
|
||||
os.lchmod = None
|
||||
os.lchown = None
|
||||
os.getcwd = None
|
||||
os.chdir = None
|
||||
|
||||
import shutil
|
||||
|
||||
shutil.rmtree = None
|
||||
shutil.move = None
|
||||
shutil.chown = None
|
||||
|
||||
import subprocess
|
||||
|
||||
subprocess.Popen = None # type: ignore
|
||||
|
||||
__builtins__['help'] = None
|
||||
|
||||
import sys
|
||||
|
||||
sys.modules['ipdb'] = None
|
||||
sys.modules['joblib'] = None
|
||||
sys.modules['resource'] = None
|
||||
sys.modules['psutil'] = None
|
||||
sys.modules['tkinter'] = None
|
||||
@ -232,12 +232,26 @@ def answer_spans(raw: str, sample: Sample) -> Tuple[str, bool, str]:
|
||||
|
||||
Official pattern captures one line per match ([^\\n]+); multiple Answer:
|
||||
lines (or repeated answers) each contribute one span, matching the gold
|
||||
spans-tuple format.
|
||||
spans-tuple format. es parity: a single line listing several spans
|
||||
('A and B', 'A, B', 'A; B') splits into one span per item.
|
||||
"""
|
||||
matches = re.findall(r'(?i)Answer\s*:\s*([^\n]+)', raw or '')
|
||||
if not matches:
|
||||
return '', False, 'no Answer: line'
|
||||
spans = [m.strip() for m in matches if m.strip()]
|
||||
spans: list = []
|
||||
for m in matches:
|
||||
m = m.strip().rstrip('.').strip()
|
||||
if not m:
|
||||
continue
|
||||
parts = re.split(r'\s*(?:,|;|\band\b)\s*', m)
|
||||
parts = [p.strip() for p in parts if p.strip()]
|
||||
# a comma inside one numeric span ('1,234') must not split it
|
||||
if parts and all(re.fullmatch(r'\d{1,3}(,\d{3})+(\.\d+)?%?', p) or p == m
|
||||
for p in parts) and ',' in m and len(parts) > 1 \
|
||||
and re.fullmatch(r'[\d,.]+%?', m):
|
||||
spans.append(m)
|
||||
else:
|
||||
spans.extend(parts if parts else [m])
|
||||
if not spans:
|
||||
return '', False, 'empty Answer:'
|
||||
return '\n'.join(spans), True, f'answer_spans:{len(spans)}'
|
||||
|
||||
@ -54,6 +54,9 @@ class EvalRecipe:
|
||||
aggregators: Dict[str, Union[str, Tuple[str, Dict[str, Any]]]] = field(default_factory=dict)
|
||||
judge: Optional[JudgeConfig] = None
|
||||
description: str = ''
|
||||
exec_workers: int = 1 # parallel judging threads (docker/subprocess
|
||||
# execution benches: 8-12; llm_judge stays 1 unless
|
||||
# the judge endpoint can take it)
|
||||
|
||||
def resolve_extract(self):
|
||||
return make_extractor(self.extract)
|
||||
|
||||
@ -8,18 +8,33 @@ from ..recipe import EvalRecipe, register_eval
|
||||
def _humaneval_harness(sample, pred: str):
|
||||
test = sample.metadata.get('test', '')
|
||||
entry = sample.metadata.get('entry_point', 'f')
|
||||
prog = f'{sample.input}{pred}\n\n{test}\n\ncheck({entry})\nprint("PASSED")\n'
|
||||
base = (sample.metadata or {}).get('prompt') or sample.input
|
||||
prog = f'{base}{pred}\n\n{test}\n\ncheck({entry})\nprint("PASSED")\n'
|
||||
return {'main.py': prog}
|
||||
|
||||
|
||||
def _humaneval_extract(raw, sample):
|
||||
# es/official contract asks for 'ONLY the code' -> the model emits a bare
|
||||
# function with no markdown fence; fall back to the raw text then
|
||||
from ..extractor import make_extractor
|
||||
val, ok, note = make_extractor('code_any')(raw, sample)
|
||||
if ok:
|
||||
return val, ok, note
|
||||
body = (raw or '').strip()
|
||||
if body:
|
||||
return body, True, 'bare_code'
|
||||
return '', False, 'empty'
|
||||
|
||||
|
||||
@register_eval('humaneval')
|
||||
def humaneval():
|
||||
return EvalRecipe(
|
||||
name='humaneval',
|
||||
extract='code_any',
|
||||
extract=_humaneval_extract,
|
||||
scorers={'pass': {'name': 'execution', 'harness': _humaneval_harness,
|
||||
'sandbox': 'docker', 'timeout_s': 30}},
|
||||
aggregators={'pass': 'pass_at_k'},
|
||||
exec_workers=8,
|
||||
description='HumanEval; completion + official tests in a sandbox, pass@k.',
|
||||
)
|
||||
|
||||
@ -44,6 +59,7 @@ def bigcodebench():
|
||||
'image': 'bigcodebench-sandbox:latest',
|
||||
'sandbox': 'docker', 'timeout_s': 120}},
|
||||
aggregators={'pass': 'pass_at_k'},
|
||||
exec_workers=12,
|
||||
description='BigCodeBench; official all-libs docker image, pass@k.',
|
||||
)
|
||||
|
||||
@ -51,6 +67,8 @@ def bigcodebench():
|
||||
_LCB_RUNNER = r'''
|
||||
import json, subprocess, sys
|
||||
cases = json.load(open('cases.json'))
|
||||
meta = json.load(open('meta.json')) if __import__('os').path.exists('meta.json') else {}
|
||||
fn_name = meta.get('fn_name')
|
||||
|
||||
def as_lines(v):
|
||||
"""Normalize an expected output to a list of lines (no trailing empties)."""
|
||||
@ -62,15 +80,50 @@ def as_lines(v):
|
||||
return [l for l in out if l != '']
|
||||
|
||||
failed = 0
|
||||
for i, case in enumerate(cases):
|
||||
stdin = case.get('input', '')
|
||||
expected = as_lines(case.get('output', ''))
|
||||
r = subprocess.run([sys.executable, 'solution.py'], input=stdin,
|
||||
capture_output=True, text=True, timeout=20)
|
||||
got = [l for l in r.stdout.split('\n') if l != '']
|
||||
if got != expected:
|
||||
failed += 1
|
||||
print(f'case {i}: expected {expected!r} got {got!r}', file=sys.stderr)
|
||||
if fn_name:
|
||||
# function-call style (LeetCode / starter_code problems, es-official):
|
||||
# import the solution and call fn_name on each input, compare to output
|
||||
import importlib.util
|
||||
spec = importlib.util.spec_from_file_location('solution', 'solution.py')
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(mod)
|
||||
fn = getattr(mod, fn_name, None)
|
||||
if fn is None:
|
||||
# starter classes: instantiate and look for the method on the class
|
||||
for attr in vars(mod).values():
|
||||
if isinstance(attr, type) and hasattr(attr, fn_name):
|
||||
fn = getattr(attr(), fn_name)
|
||||
break
|
||||
if fn is None:
|
||||
print(f'fn_name {fn_name!r} not found in solution', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
for i, case in enumerate(cases):
|
||||
try:
|
||||
raw_in, raw_out = case['input'], case['output']
|
||||
# lite packs fn-style args/results as JSON STRINGS
|
||||
args = json.loads(raw_in) if isinstance(raw_in, str) else raw_in
|
||||
expected = json.loads(raw_out) if isinstance(raw_out, str) else raw_out
|
||||
args = args if isinstance(args, list) else [args]
|
||||
got = fn(*args)
|
||||
except Exception as e:
|
||||
print(f'case {i}: raised {type(e).__name__}: {e}', file=sys.stderr)
|
||||
failed += 1
|
||||
continue
|
||||
expected = tuple(expected) if isinstance(expected, list) else expected
|
||||
got_t = tuple(got) if isinstance(got, list) else got
|
||||
if got_t != expected:
|
||||
print(f'case {i}: expected {expected!r} got {got_t!r}', file=sys.stderr)
|
||||
failed += 1
|
||||
else:
|
||||
for i, case in enumerate(cases):
|
||||
stdin = case.get('input', '')
|
||||
expected = as_lines(case.get('output', ''))
|
||||
r = subprocess.run([sys.executable, 'solution.py'], input=stdin,
|
||||
capture_output=True, text=True, timeout=20)
|
||||
got = [l for l in r.stdout.split('\n') if l != '']
|
||||
if got != expected:
|
||||
failed += 1
|
||||
print(f'case {i}: expected {expected!r} got {got!r}', file=sys.stderr)
|
||||
if failed:
|
||||
print(f'{failed}/{len(cases)} cases failed', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
@ -118,17 +171,36 @@ def _lcb_harness(sample, pred: str, use_private: bool = True):
|
||||
import json
|
||||
|
||||
starter = sample.metadata.get('starter_code') or ''
|
||||
if use_private:
|
||||
cases = _lcb_decode_cases(sample.metadata.get('private_test_cases'))
|
||||
else:
|
||||
cases = _lcb_decode_cases(sample.metadata.get('public_test_cases'))
|
||||
if not cases: # private unavailable -> fall back to public
|
||||
cases = _lcb_decode_cases(sample.metadata.get('public_test_cases'))
|
||||
return {
|
||||
# es-official case composition: PUBLIC + PRIVATE in full (use_private
|
||||
# toggles the private half; es load_utils.py always uses both)
|
||||
pub = _lcb_decode_cases(sample.metadata.get('public_test_cases'))
|
||||
priv = _lcb_decode_cases(sample.metadata.get('private_test_cases')) if use_private else []
|
||||
cases = pub + priv
|
||||
if not cases:
|
||||
cases = _lcb_decode_cases(sample.metadata.get('public_test_cases')) or []
|
||||
files = {
|
||||
'solution.py': f'{starter}\n{pred}\n',
|
||||
'cases.json': json.dumps(cases or []),
|
||||
'runner.py': _LCB_RUNNER,
|
||||
}
|
||||
fn_name = (sample.metadata.get('fn_name')
|
||||
or _lcb_fn_name_from_metadata(sample.metadata.get('raw_metadata')))
|
||||
if fn_name:
|
||||
files['meta.json'] = json.dumps({'fn_name': fn_name})
|
||||
return files
|
||||
|
||||
|
||||
def _lcb_fn_name_from_metadata(raw):
|
||||
"""Official lite packs fn_name inside the record's `metadata` JSON blob."""
|
||||
import json
|
||||
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
md = json.loads(raw) if isinstance(raw, str) else raw
|
||||
return md.get('func_name')
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
@register_eval('live_code_bench')
|
||||
@ -139,6 +211,7 @@ def live_code_bench():
|
||||
scorers={'pass': {'name': 'execution', 'harness': _lcb_harness,
|
||||
'entry': 'runner.py', 'sandbox': 'local', 'timeout_s': 60}},
|
||||
aggregators={'pass': 'pass_at_k'},
|
||||
exec_workers=8,
|
||||
description='LiveCodeBench; stdin/stdout public-case runner in sandbox.',
|
||||
)
|
||||
|
||||
|
||||
@ -9,12 +9,18 @@ from ..recipe import EvalRecipe, JudgeConfig, register_eval
|
||||
_SIMPLE_QA_PROMPT = Path(__file__).with_name('_simpleqa_grader.txt').read_text(encoding='utf-8')
|
||||
|
||||
_HLE_PROMPT = (
|
||||
# es hle_adapter JUDGE_PROMPT, verbatim (only placeholder names renamed)
|
||||
'Judge whether the following [response] to [question] is correct or not based '
|
||||
'on the precise and unambiguous [correct_answer] below.\n\n'
|
||||
'[question]: {question}\n\n[response]: {prediction}\n\n'
|
||||
'[correct_answer]: {target}\n\n'
|
||||
'Focus only on whether the answers match. In one or two sentences explain, then '
|
||||
"write your final line as 'GRADE: C' for correct or 'GRADE: I' for incorrect."
|
||||
'Your judgment must focus only on if there are meaningful differences between '
|
||||
'[correct_answer] and the [response]. Do not comment on any background to the '
|
||||
'problem, do not attempt to solve the problem, do not argue for any answer '
|
||||
'different than [correct_answer], focus only on whether the answers match. '
|
||||
'Explain why the [response] is correct or incorrect based on [correct_answer] '
|
||||
'in one or two sentences. Finally, write your answer in the format '
|
||||
"'GRADE: C' for correct answer or 'GRADE: I' for incorrect answer.\n"
|
||||
)
|
||||
|
||||
@register_eval('hle')
|
||||
@ -23,6 +29,7 @@ def hle():
|
||||
name='hle',
|
||||
extract='identity',
|
||||
scorers={'acc': {'name': 'llm_judge', 'prompt_template': _HLE_PROMPT,
|
||||
'label_pattern': r'GRADE:\s*([CI])',
|
||||
'labels': {'C': {'acc': 1.0}, 'I': {'acc': 0.0}}, 'primary': 'acc'}},
|
||||
judge=JudgeConfig(model='judge'),
|
||||
description="HLE; official GRADE: C/I LLM judge.",
|
||||
|
||||
@ -16,7 +16,8 @@ def _bbh_extract(raw, sample):
|
||||
|
||||
target = str(sample.target or '').strip()
|
||||
if re.fullmatch(r'\([A-Z]\)', target):
|
||||
return mcq_letter(raw, sample)
|
||||
val, ok, why = mcq_letter(raw, sample)
|
||||
return (f'({val})' if ok else val), ok, why
|
||||
return answer_phrase(raw, sample)
|
||||
|
||||
|
||||
|
||||
@ -60,7 +60,11 @@ def evaluate(
|
||||
bp.__enter__()
|
||||
|
||||
results: List[SampleResult] = []
|
||||
for sample, pred in zip(samples, predictions):
|
||||
|
||||
def judge_one(sample, pred) -> SampleResult:
|
||||
"""Extract + score ONE sample (thread-safe: everything here is local
|
||||
except docker/subprocess execution, which parallelizes perfectly --
|
||||
each sample gets its own container/workdir)."""
|
||||
raw = pred if isinstance(pred, str) else str(pred.get('raw', ''))
|
||||
override = {} if isinstance(pred, str) else pred
|
||||
result = SampleResult(
|
||||
@ -108,7 +112,20 @@ def evaluate(
|
||||
result.score_details[metric] = {'error': f'{type(e).__name__}: {e}'}
|
||||
except Exception as e:
|
||||
result.error = f'{type(e).__name__}: {e}\n{traceback.format_exc(limit=2)}'
|
||||
results.append(result)
|
||||
return result
|
||||
|
||||
workers = getattr(recipe, 'exec_workers', 1)
|
||||
if workers > 1 and len(samples) > 1:
|
||||
# parallel judging: docker/subprocess execution is embarrassingly
|
||||
# parallel (one container per sample); text scorers are cheap and
|
||||
# thread-safe enough. Serializes again for judge/dict-dependent runs.
|
||||
import concurrent.futures
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
|
||||
results = list(pool.map(judge_one, samples, predictions))
|
||||
else:
|
||||
for sample, pred in zip(samples, predictions):
|
||||
results.append(judge_one(sample, pred))
|
||||
|
||||
report = EvalReport(
|
||||
dataset=ds_name,
|
||||
@ -155,9 +172,16 @@ def _aggregate_into(report: EvalReport, results, recipe: EvalRecipe, aggregators
|
||||
continue
|
||||
if isinstance(out, dict):
|
||||
report.metric_groups[metric] = out
|
||||
vals = [v for v in out.values() if isinstance(v, (int, float))]
|
||||
if vals:
|
||||
report.metrics[metric] = sum(vals) / len(vals)
|
||||
# primary metric = the aggregator's same-named entry (e.g.
|
||||
# simpleqa_official returns is_correct/is_incorrect/...); the
|
||||
# old mean-of-all-values fallback invented nonsense like
|
||||
# mean(0.035, 0.945, 0.02, 0.98) for is_correct
|
||||
if metric in out and isinstance(out[metric], (int, float)):
|
||||
report.metrics[metric] = float(out[metric])
|
||||
else:
|
||||
vals = [v for v in out.values() if isinstance(v, (int, float))]
|
||||
if vals:
|
||||
report.metrics[metric] = sum(vals) / len(vals)
|
||||
else:
|
||||
report.metrics[metric] = float(out)
|
||||
report.metrics['extraction_failure_rate'] = (
|
||||
|
||||
@ -288,10 +288,28 @@ def llm_judge(pred: str, target, sample: Sample, ctx: ScoreContext):
|
||||
labels: Dict[str, Dict[str, float]] = ctx.params.get('labels') or {}
|
||||
upper = (raw_text or '').upper()
|
||||
chosen = None
|
||||
for label in labels:
|
||||
if label.upper() and label.upper() in upper:
|
||||
chosen = label
|
||||
break
|
||||
# 1) explicit pattern, LAST match (e.g. 'GRADE:\s*([CI])'); a bare substring
|
||||
# scan over the whole judge text would match 'C' inside e.g. "CONSISTS"
|
||||
pattern = ctx.params.get('label_pattern')
|
||||
if pattern:
|
||||
ms = list(re.finditer(pattern, raw_text or '', re.IGNORECASE))
|
||||
if ms:
|
||||
want = ms[-1].group(1).upper()
|
||||
chosen = want if want in labels else None
|
||||
# 2) scan only the final non-empty line (judge verdicts live there);
|
||||
# WORD-BOUNDARY match -- a bare substring 'A' would hit 'ANSWER'
|
||||
if chosen is None:
|
||||
tail = upper.strip().splitlines()[-1].strip() if upper.strip() else ''
|
||||
for label in labels:
|
||||
if label.upper() and re.search(rf'\b{re.escape(label.upper())}\b', tail):
|
||||
chosen = label
|
||||
break
|
||||
# 3) legacy whole-text fallback (word-boundary too)
|
||||
if chosen is None:
|
||||
for label in labels:
|
||||
if label.upper() and re.search(rf'\b{re.escape(label.upper())}\b', upper):
|
||||
chosen = label
|
||||
break
|
||||
primary = ctx.params.get('primary', 'acc')
|
||||
default_label = ctx.params.get('default_label') # e.g. SimpleQA's C on parse failure
|
||||
if chosen is None and default_label and default_label in labels:
|
||||
|
||||
@ -146,7 +146,32 @@ def _parse_text_tool_calls(text: str) -> list:
|
||||
for x in arr]
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
return []
|
||||
# python-call style: [func(a=1, b="x")] or nested [[{..}]] JSON strings --
|
||||
# dp4/DeepSeek text-protocol output shape (es feeds the same text to its
|
||||
# official decoders). Parse func(name=args) via a safe regex + literal_eval.
|
||||
def _py_call(m_):
|
||||
name = m_.group(1)
|
||||
argstr = (m_.group(2) or '').strip()
|
||||
args = {}
|
||||
if argstr:
|
||||
import ast as _ast
|
||||
try:
|
||||
parsed = _ast.parse(f'dummy({argstr})', mode='eval').body
|
||||
for kw_ in parsed.keywords:
|
||||
try:
|
||||
args[kw_.arg] = _ast.literal_eval(kw_.value)
|
||||
except (ValueError, SyntaxError):
|
||||
args[kw_.arg] = _ast.unparse(kw_.value)
|
||||
except SyntaxError:
|
||||
return None
|
||||
return {'id': '', 'type': 'function',
|
||||
'function': {'name': name, 'arguments': json.dumps(args)}}
|
||||
|
||||
for m_ in _re.finditer(r'([A-Za-z_][A-Za-z0-9_]*)\((.*?)\)', text):
|
||||
c = _py_call(m_)
|
||||
if c and c['function']['name']:
|
||||
out.append(c)
|
||||
return out
|
||||
|
||||
|
||||
@register_adapter('openai')
|
||||
@ -237,7 +262,9 @@ class OpenAICompatible(ModelAdapter):
|
||||
status = None
|
||||
import json as _json
|
||||
|
||||
async with httpx.AsyncClient(timeout=self.extra.get('timeout', 600)) as client:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(
|
||||
connect=self.extra.get('connect_timeout', 15),
|
||||
read=self.extra.get('timeout', 300), write=30, pool=15)) as client:
|
||||
async with client.stream('POST', url, json=payload, headers=headers) as resp:
|
||||
status = resp.status_code
|
||||
resp.raise_for_status()
|
||||
@ -311,15 +338,19 @@ class OpenAICompatible(ModelAdapter):
|
||||
payload[k] = kw[k]
|
||||
payload.setdefault('max_tokens', self.extra.get('max_tokens', 4096)) # CoT room
|
||||
if self.extra.get('no_think'):
|
||||
# Qwen3 soft switch: /no_think appended to the last user message.
|
||||
# (chat_template_kwargs + long/odd payloads hit template 400s on
|
||||
# some vLLM builds; the soft switch is payload-independent)
|
||||
msgs = payload.get('messages') or []
|
||||
for m in reversed(msgs):
|
||||
if m.get('role') == 'user':
|
||||
if '/no_think' not in (m.get('content') or ''):
|
||||
m['content'] = (m.get('content') or '') + ' /no_think'
|
||||
break
|
||||
if payload.get('tools'):
|
||||
# tools payloads: kwargs popped above (template 400 issue)
|
||||
# -> keep the soft switch (appended marker) as before
|
||||
msgs = payload.get('messages') or []
|
||||
for m in reversed(msgs):
|
||||
if m.get('role') == 'user':
|
||||
if '/no_think' not in (m.get('content') or ''):
|
||||
m['content'] = (m.get('content') or '') + ' /no_think'
|
||||
break
|
||||
else:
|
||||
# plain payloads: template-level switch (clean -- no prompt
|
||||
# pollution; verified on the 8123-8130 vLLM pool)
|
||||
payload['chat_template_kwargs'] = {'enable_thinking': False}
|
||||
return payload
|
||||
|
||||
def _parse(self, data: Dict[str, Any]) -> ModelOutput:
|
||||
@ -360,7 +391,9 @@ class OpenAICompatible(ModelAdapter):
|
||||
try:
|
||||
import httpx
|
||||
|
||||
async with httpx.AsyncClient(timeout=self.extra.get('timeout', 600)) as client:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(
|
||||
connect=self.extra.get('connect_timeout', 15),
|
||||
read=self.extra.get('timeout', 300), write=30, pool=15)) as client:
|
||||
r = await client.post(url, json=payload, headers=headers)
|
||||
r.raise_for_status()
|
||||
return r.json()
|
||||
|
||||
154
evalharness/model/gen_profiles.py
Normal file
154
evalharness/model/gen_profiles.py
Normal file
@ -0,0 +1,154 @@
|
||||
"""Generation-parameter profiles: named, per-bench gen_kwargs presets.
|
||||
|
||||
Problem being solved: ``DatasetSpec.gen_config`` is baked into the dataset
|
||||
plugin at authoring time (the Qwen3 era defaults), but different models /
|
||||
protocols need different parameters (dp4 wants t0/32768 everywhere, Qwen3
|
||||
wanted mixed values). Without this layer every runner script re-declares
|
||||
its own ``GEN = {...}`` dict and hand-merges overrides -- we did that for
|
||||
days across seven stage scripts.
|
||||
|
||||
Resolution order (later wins):
|
||||
1. DatasetSpec.gen_config (plugin's built-in default)
|
||||
2. profile['default'] (protocol-wide baseline)
|
||||
3. profile['<bench>'] (per-bench override)
|
||||
4. explicit run_eval(gen_kwargs=...) (one-off)
|
||||
|
||||
Usage:
|
||||
# register
|
||||
@register_gen_profile('dp4-nothink')
|
||||
def dp4():
|
||||
return {'default': {'temperature': 0.0, 'max_tokens': 32768},
|
||||
'simple_qa': {'max_tokens': 1024}}
|
||||
|
||||
# consume
|
||||
run_eval(ds, spec, gen_profile='dp4-nothink')
|
||||
evalharness eval run hle --model ... --profile dp4-nothink
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, Optional
|
||||
|
||||
PROFILES: Dict[str, Callable[[], Dict[str, Dict[str, Any]]]] = {}
|
||||
|
||||
|
||||
def register_gen_profile(name: str):
|
||||
def decorator(fn):
|
||||
PROFILES[name] = fn
|
||||
return fn
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
def get_profile(name: str) -> Optional[Dict[str, Dict[str, Any]]]:
|
||||
"""Resolve a profile by name.
|
||||
|
||||
Lookup order:
|
||||
1. @register_gen_profile registry (code-defined, built-ins live here)
|
||||
2. YAML file, selected by (in order):
|
||||
a. $EVALHARNESS_GEN_PROFILES env var (explicit path)
|
||||
b. ./gen_profiles.yaml (next to the invocation / repo root)
|
||||
c. ~/.config/evalharness/gen_profiles.yaml
|
||||
A YAML file may define MANY profiles; the file's profiles are also merged
|
||||
into list_profiles() so CLI completion/Errors can see them.
|
||||
"""
|
||||
fn = PROFILES.get(name)
|
||||
if fn is not None:
|
||||
return fn()
|
||||
loaded = _load_yaml_profiles()
|
||||
if name in loaded:
|
||||
return loaded[name]
|
||||
return None
|
||||
|
||||
|
||||
_YAML_CACHE: Optional[Dict[str, Dict[str, Dict[str, Any]]]] = None
|
||||
|
||||
|
||||
def _candidate_yaml_paths():
|
||||
import os
|
||||
|
||||
paths = []
|
||||
env = os.environ.get('EVALHARNESS_GEN_PROFILES')
|
||||
if env:
|
||||
paths.append(Path(env))
|
||||
paths.append(Path('gen_profiles.yaml'))
|
||||
paths.append(Path.home() / '.config' / 'evalharness' / 'gen_profiles.yaml')
|
||||
return paths
|
||||
|
||||
|
||||
def _load_yaml_profiles() -> Dict[str, Dict[str, Dict[str, Any]]]:
|
||||
"""Read every profile from the first YAML that exists; empty if none."""
|
||||
global _YAML_CACHE
|
||||
if _YAML_CACHE is not None:
|
||||
return _YAML_CACHE
|
||||
try:
|
||||
import yaml
|
||||
except ImportError:
|
||||
_YAML_CACHE = {}
|
||||
return _YAML_CACHE
|
||||
for path in _candidate_yaml_paths():
|
||||
try:
|
||||
if path and path.exists():
|
||||
data = yaml.safe_load(path.read_text(encoding='utf-8')) or {}
|
||||
# accept both flat (file IS one profile: has 'default')
|
||||
# and namespaced (top-level keys are profile names)
|
||||
if 'default' in data or 'bench' in {k.split('.')[0] for k in data
|
||||
if isinstance(data.get(k), dict)}:
|
||||
data = {'default': data} if 'default' in data else data
|
||||
_YAML_CACHE = {k: v for k, v in data.items() if isinstance(v, dict)}
|
||||
return _YAML_CACHE
|
||||
except Exception:
|
||||
continue
|
||||
_YAML_CACHE = {}
|
||||
return _YAML_CACHE
|
||||
|
||||
|
||||
def list_profiles():
|
||||
return sorted(set(PROFILES) | set(_load_yaml_profiles()))
|
||||
|
||||
|
||||
def merge_gen_kwargs(bench: str, spec, gen_kwargs: Optional[Dict[str, Any]],
|
||||
profile_name: str = '') -> Dict[str, Any]:
|
||||
"""Layered merge for one bench (later layers win)."""
|
||||
out: Dict[str, Any] = {}
|
||||
out.update(getattr(spec, 'gen_config', None) or {})
|
||||
if profile_name:
|
||||
prof = get_profile(profile_name)
|
||||
if prof is None:
|
||||
raise KeyError(f'unknown gen profile {profile_name!r}; '
|
||||
f'available: {", ".join(list_profiles())}')
|
||||
out.update(prof.get('default') or {})
|
||||
out.update(prof.get(bench) or {})
|
||||
out.update(gen_kwargs or {})
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------ built-ins ------------------------------
|
||||
|
||||
@register_gen_profile('dp4-nothink')
|
||||
def _dp4_nothink():
|
||||
"""DeepSeek-V4-Flash nothinking protocol (es DP4-flash-int8-nothinking):
|
||||
t0 / 32768 / top_p 1.0 everywhere; judged benches can be trimmed."""
|
||||
return {'default': {'temperature': 0.0, 'max_tokens': 32768, 'top_p': 1.0}}
|
||||
|
||||
|
||||
@register_gen_profile('qwen3-es-parity')
|
||||
def _qwen3_parity():
|
||||
"""Qwen3-8B evalscope-parity protocol (the values used for the 28-bench
|
||||
alignment): CoT benches get 32k room, short-answer benches stay small."""
|
||||
return {
|
||||
'default': {'temperature': 0.0, 'max_tokens': 32768},
|
||||
'simple_qa': {'max_tokens': 1024},
|
||||
'hle': {'max_tokens': 8192},
|
||||
'gpqa_diamond': {'temperature': 1.0, 'max_tokens': 8192},
|
||||
'aime24': {'temperature': 1.0},
|
||||
'aime25': {'temperature': 1.0},
|
||||
'aime26': {'temperature': 1.0},
|
||||
'hmmt26': {'temperature': 1.0},
|
||||
'imo_answerbench': {'temperature': 1.0},
|
||||
}
|
||||
|
||||
|
||||
@register_gen_profile('t1-short')
|
||||
def _t1_short():
|
||||
"""temp=1 sampling for small repeated benches (variance measurement)."""
|
||||
return {'default': {'temperature': 1.0, 'max_tokens': 32768, 'top_p': 1.0}}
|
||||
@ -7,9 +7,16 @@
|
||||
pool = PooledAdapter([resolve_adapter(f'openai/http://127.0.0.1:{p}/v1?Qwen3-8B')
|
||||
for p in range(8123, 8131)])
|
||||
out = await pool.generate(...) # round-robin over instances
|
||||
|
||||
Traffic management: round-robin keeps per-endpoint traffic even; backends
|
||||
that fail repeatedly enter a cool-down window and are skipped until it
|
||||
expires, so one sick endpoint cannot absorb its share of the load.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import itertools
|
||||
import time
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from ..data.sample import ChatMessage
|
||||
@ -18,9 +25,11 @@ from .output import ModelOutput, Usage
|
||||
|
||||
|
||||
class PooledAdapter(ModelAdapter):
|
||||
"""Round-robin over N equivalent backend instances."""
|
||||
"""Round-robin over N equivalent backend instances with health cooling."""
|
||||
|
||||
name = 'pool'
|
||||
COOLDOWN_S = 60.0 # a backend that failed EVERY attempt rests this long
|
||||
COOLDOWN_AFTER = 2 # consecutive full-pass failures before cooling
|
||||
|
||||
def __init__(self, adapters: List[ModelAdapter]):
|
||||
if not adapters:
|
||||
@ -31,19 +40,60 @@ class PooledAdapter(ModelAdapter):
|
||||
self.usage = Usage()
|
||||
# request outcome counters (success rate accounting)
|
||||
self.stats = {'requests': 0, 'ok': 0, 'failed': 0, 'retried': 0}
|
||||
# per-backend health: consecutive_failures, cooling_until, per-endpoint counts
|
||||
self._health = [{ 'fails': 0, 'until': 0.0, 'ok': 0, 'req': 0}
|
||||
for _ in adapters]
|
||||
# adaptive per-endpoint concurrency gates (AIMD over /metrics signals)
|
||||
self._gates = [AdaptiveGate(a) for a in adapters]
|
||||
|
||||
def _next(self) -> ModelAdapter:
|
||||
return self.adapters[next(self._cycle)]
|
||||
"""Round-robin, skipping endpoints inside their cool-down window."""
|
||||
n = len(self.adapters)
|
||||
now = time.time()
|
||||
for _ in range(n):
|
||||
i = next(self._cycle)
|
||||
h = self._health[i]
|
||||
if h['until'] <= now or all(x['until'] <= now for x in self._health):
|
||||
self._health[i]['req'] += 1
|
||||
return self.adapters[i]
|
||||
# everything cooling: take the next anyway (better to try than stall)
|
||||
i = next(self._cycle)
|
||||
self._health[i]['req'] += 1
|
||||
return self.adapters[i]
|
||||
|
||||
def _mark(self, adapter: ModelAdapter, ok: bool) -> None:
|
||||
try:
|
||||
i = self.adapters.index(adapter)
|
||||
except ValueError:
|
||||
return
|
||||
h = self._health[i]
|
||||
if ok:
|
||||
h['fails'] = 0
|
||||
h['until'] = 0.0
|
||||
h['ok'] += 1
|
||||
else:
|
||||
h['fails'] += 1
|
||||
if h['fails'] >= self.COOLDOWN_AFTER:
|
||||
h['until'] = time.time() + self.COOLDOWN_S
|
||||
h['fails'] = 0
|
||||
|
||||
def request_stats(self) -> Dict[str, float]:
|
||||
"""Success-rate view of all requests that passed through this pool."""
|
||||
"""Success-rate + per-endpoint traffic view (load-balance audit)."""
|
||||
n = self.stats['requests']
|
||||
return {
|
||||
out = {
|
||||
'requests': n,
|
||||
'success_rate': self.stats['ok'] / n if n else 0.0,
|
||||
'retry_rate': self.stats['retried'] / n if n else 0.0,
|
||||
'failure_rate': self.stats['failed'] / n if n else 0.0,
|
||||
}
|
||||
for i, (a, h) in enumerate(zip(self.adapters, self._health)):
|
||||
tag = a.api_base.rsplit('//', 1)[-1].replace('/', '_')
|
||||
out[f'ep{i}_{tag}_reqs'] = h['req']
|
||||
out[f'ep{i}_{tag}_ok'] = h['ok']
|
||||
if i < len(self._gates):
|
||||
for k, v in self._gates[i].report().items():
|
||||
out[f'ep{i}_{tag}_{k}'] = v
|
||||
return out
|
||||
|
||||
async def generate(self, messages: List[ChatMessage],
|
||||
tools: Optional[list] = None, **kw) -> ModelOutput:
|
||||
@ -52,14 +102,23 @@ class PooledAdapter(ModelAdapter):
|
||||
for _ in range(len(self.adapters)): # try each instance once
|
||||
adapter = self._next()
|
||||
try:
|
||||
out = await adapter.generate(messages, tools=tools, **kw)
|
||||
i = self.adapters.index(adapter)
|
||||
await self._gates[i].acquire()
|
||||
try:
|
||||
out = await adapter.generate(messages, tools=tools, **kw)
|
||||
finally:
|
||||
self._gates[i].release(True)
|
||||
self.usage = self.usage + out.usage
|
||||
self.stats['ok'] += 1
|
||||
self._mark(adapter, True)
|
||||
if out.usage.retries:
|
||||
self.stats['retried'] += 1
|
||||
return out
|
||||
except Exception as e: # dead/overloaded instance -> next
|
||||
last_exc = e
|
||||
self._mark(adapter, False)
|
||||
with contextlib.suppress(ValueError):
|
||||
self._gates[self.adapters.index(adapter)].release(False)
|
||||
# 4xx (e.g. 400 overloaded) still worth trying ANOTHER instance:
|
||||
# one backend's state can differ from the rest
|
||||
continue
|
||||
@ -69,6 +128,121 @@ class PooledAdapter(ModelAdapter):
|
||||
async def close(self) -> None:
|
||||
for a in self.adapters:
|
||||
await a.close()
|
||||
for g in self._gates:
|
||||
g.stop()
|
||||
|
||||
|
||||
class AdaptiveGate:
|
||||
"""Per-endpoint adaptive concurrency limiter (AIMD + server signals).
|
||||
|
||||
Goal: keep the backend SATURATED (high XPU util / throughput) without
|
||||
pushing it over the cliff (500s / child crashes). Signals:
|
||||
- server /metrics: num_queue_reqs > 0 means WE are pushing too hard
|
||||
for the current mix; idle (no queue, low running) means room to grow
|
||||
- request failures: multiplicative decrease (survive first)
|
||||
Control law (classic AIMD):
|
||||
+1 concurrency per probe interval when the endpoint looks underfed
|
||||
-1 when the server reports a queue (gentle)
|
||||
x0.7 on any failed request (fast backoff), floor at LO
|
||||
Purely additive to PooledAdapter: one gate per backend, no caller change.
|
||||
"""
|
||||
|
||||
LO = 2 # never go below: progress beats perfection
|
||||
HI = 96 # sane ceiling for one endpoint
|
||||
PROBE_S = 5.0 # metrics probe interval
|
||||
|
||||
def __init__(self, adapter: ModelAdapter):
|
||||
self.adapter = adapter
|
||||
self.limit = 8.0 # float for smooth x0.7; compare with int()
|
||||
self._inflight = 0
|
||||
self._cond: Optional[asyncio.Condition] = None
|
||||
self._task: Optional[asyncio.Task] = None
|
||||
self._stopped = False
|
||||
self.stats = {'probe': 0, 'ramp': 0, 'hold_queue': 0, 'backoff_fail': 0,
|
||||
'backoff_queue': 0}
|
||||
|
||||
# ---- gate semantics ----
|
||||
async def acquire(self) -> None:
|
||||
if self._cond is None: # lazy init in the running loop
|
||||
self._cond = asyncio.Condition()
|
||||
self._task = asyncio.get_event_loop().create_task(self._probe_loop())
|
||||
while self._inflight >= max(1, int(self.limit)):
|
||||
await self._cond.acquire()
|
||||
try:
|
||||
await self._cond.wait()
|
||||
finally:
|
||||
self._cond.release()
|
||||
self._inflight += 1
|
||||
|
||||
def release(self, ok: bool) -> None:
|
||||
self._inflight = max(0, self._inflight - 1)
|
||||
if not ok: # multiplicative decrease -- survival first
|
||||
before = self.limit
|
||||
self.limit = max(self.LO, self.limit * 0.7)
|
||||
if before != self.limit:
|
||||
self.stats['backoff_fail'] += 1
|
||||
self._wake()
|
||||
|
||||
def _wake(self) -> None:
|
||||
if self._cond is not None:
|
||||
# fire-and-forget notify (loop may not be ours -- best effort)
|
||||
try:
|
||||
fut = asyncio.ensure_future(self._notify_all())
|
||||
fut.add_done_callback(lambda f: None)
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
async def _notify_all(self) -> None:
|
||||
async with self._cond:
|
||||
self._cond.notify_all()
|
||||
|
||||
# ---- server-signal probe ----
|
||||
async def _probe_once(self) -> None:
|
||||
import urllib.request
|
||||
|
||||
url = f'{self.adapter.api_base.rstrip("/")}/metrics'
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=4) as resp:
|
||||
text = resp.read().decode('utf-8', 'ignore')
|
||||
except Exception:
|
||||
return # no metrics (or busy): hold current limit
|
||||
running = queue = None
|
||||
for line in text.splitlines():
|
||||
if line.startswith('sglang:num_running_reqs'):
|
||||
running = float(line.rsplit(' ', 1)[-1])
|
||||
elif line.startswith('sglang:num_queue_reqs'):
|
||||
queue = float(line.rsplit(' ', 1)[-1])
|
||||
self.stats['probe'] += 1
|
||||
if queue is None and running is None:
|
||||
return
|
||||
if queue is not None and queue >= 2:
|
||||
# server is queuing OUR excess: gentle additive decrease
|
||||
self.limit = max(self.LO, self.limit - 1)
|
||||
self.stats['backoff_queue'] += 1
|
||||
elif (queue or 0) == 0 and (running is None or running < max(2, int(self.limit))):
|
||||
# underfed: no queue and running below our own cap -> ramp up
|
||||
self.limit = min(self.HI, self.limit + 1)
|
||||
self.stats['ramp'] += 1
|
||||
else:
|
||||
self.stats['hold_queue'] += 1
|
||||
self._wake()
|
||||
|
||||
async def _probe_loop(self) -> None:
|
||||
import contextlib
|
||||
|
||||
while not self._stopped:
|
||||
with contextlib.suppress(Exception):
|
||||
await self._probe_once()
|
||||
await asyncio.sleep(self.PROBE_S)
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stopped = True
|
||||
if self._task is not None:
|
||||
self._task.cancel()
|
||||
|
||||
def report(self) -> Dict[str, float]:
|
||||
return {'limit': max(1, int(self.limit)), 'inflight': self._inflight,
|
||||
**{f'gate_{k}': v for k, v in self.stats.items()}}
|
||||
|
||||
|
||||
def pooled(specs: List[str]) -> PooledAdapter:
|
||||
|
||||
184
evalharness/model/prompt_renderers.py
Normal file
184
evalharness/model/prompt_renderers.py
Normal file
@ -0,0 +1,184 @@
|
||||
"""Prompt renderer plugins: one registered function per ``prompt_style``.
|
||||
|
||||
Before this module the runner's ``assemble()`` grew a chain of
|
||||
``if spec_style == 'xxx'`` branches; now each style is a plugin:
|
||||
|
||||
@register_prompt_renderer('aime_es')
|
||||
def aime(question, sample, spec, prompt_style):
|
||||
return {'question': ...}
|
||||
|
||||
Renderer contract:
|
||||
- input: the bare question text + the Sample + the DatasetSpec
|
||||
- output: dict with any of ``question`` (rewritten), ``system`` (a system
|
||||
message to prepend), ``few_shot_header``/``few_shot_glue`` (few-shot
|
||||
layout hints consumed by assemble)
|
||||
- unregistered styles fall back to assemble's generic MCQ/QA handling.
|
||||
|
||||
Adding a benchmark prompt style = dropping a plugin here; the runner
|
||||
never changes.
|
||||
"""
|
||||
|
||||
from typing import Any, Callable, Dict, Optional
|
||||
|
||||
RENDERERS: Dict[str, Callable] = {}
|
||||
|
||||
|
||||
def register_prompt_renderer(*styles: str):
|
||||
def decorator(fn):
|
||||
for s in styles:
|
||||
RENDERERS[s] = fn
|
||||
return fn
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
def get_renderer(style: str) -> Optional[Callable]:
|
||||
return RENDERERS.get(style)
|
||||
|
||||
|
||||
def render(style: str, question: str, sample, spec, prompt_style: str = '') -> Dict[str, Any]:
|
||||
"""Apply the style's renderer; unknown styles pass through untouched."""
|
||||
fn = RENDERERS.get(style)
|
||||
if fn is None:
|
||||
return {}
|
||||
out = fn(question=question, sample=sample, spec=spec, prompt_style=prompt_style)
|
||||
return out if isinstance(out, dict) else {}
|
||||
|
||||
|
||||
# ------------------------------ plugins ------------------------------
|
||||
|
||||
|
||||
@register_prompt_renderer('trivia_es')
|
||||
def trivia_es(question, sample, spec, prompt_style):
|
||||
# es trivia template, verbatim (open-book with wiki evidence, trailing
|
||||
# newline included)
|
||||
return {'question': (
|
||||
'Read the content and answer the following question.\n\n'
|
||||
f"Content: {(sample.metadata or {}).get('evidence') or []}\n\n"
|
||||
f'Question: {question}\n\n'
|
||||
'The last line of your response should be of the form "ANSWER: [ANSWER]" '
|
||||
'(without quotes) where [ANSWER] is the answer to the problem.\n')}
|
||||
|
||||
|
||||
@register_prompt_renderer('aime_es')
|
||||
def aime_es(question, sample, spec, prompt_style):
|
||||
# es/MathArena template (NOT lstripped -- leading \n kept; reminder tail
|
||||
# after the question, both verbatim from aime_adapter)
|
||||
return {'question': (
|
||||
'\nSolve the following math problem step by step. '
|
||||
'Put your answer inside \\boxed{}.\n\n' + question
|
||||
+ '\n\nRemember to put your answer inside \\boxed{}.')}
|
||||
|
||||
|
||||
@register_prompt_renderer('imo_es')
|
||||
def imo_es(question, sample, spec, prompt_style):
|
||||
return {'question': (
|
||||
f'Problem:\n{question}\n\nPlease reason step by step, and put your '
|
||||
f'final answer within \\boxed{{}}.\n')}
|
||||
|
||||
|
||||
@register_prompt_renderer('simple_qa_es')
|
||||
def simple_qa_es(question, sample, spec, prompt_style):
|
||||
return {'question': f'Answer the question:\n\n{question}'}
|
||||
|
||||
|
||||
@register_prompt_renderer('lb2_es')
|
||||
def lb2_es(question, sample, spec, prompt_style):
|
||||
# es longbench-v2 template: <text> wrapper + CoT last-line contract
|
||||
letters = 'ABCD'
|
||||
opts = '\n'.join(f'{letters[i]}) {c}' for i, c in enumerate(sample.choices or []))
|
||||
ctx = (sample.metadata or {}).get('context', '')
|
||||
return {'question': (
|
||||
'Please read the following text and answer the questions below.\n\n'
|
||||
f'<text>\n{ctx}\n</text>\n\n'
|
||||
"Answer the following multiple choice question. The last line of your response should be "
|
||||
"of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of "
|
||||
f'{",".join(letters[:len(sample.choices or [])])}. Think step by step before answering.\n\n'
|
||||
f'{question}\n\n{opts}')}
|
||||
|
||||
|
||||
@register_prompt_renderer('lcb_es')
|
||||
def lcb_es(question, sample, spec, prompt_style):
|
||||
# es/official LCB (load_utils + adapter, verbatim): the expert-programmer
|
||||
# header is a SYSTEM message (injected by the runner); user keeps
|
||||
# ### Question:/### Format:/### Answer
|
||||
starter = (sample.metadata or {}).get('starter_code')
|
||||
if starter:
|
||||
fmt = ('### Format: You will use the following starter code to write the '
|
||||
'solution to the problem and enclose your code within delimiters.\n'
|
||||
f'```python\n{starter}\n```\n\n')
|
||||
else:
|
||||
fmt = ('### Format: Read the inputs from stdin solve the problem and write '
|
||||
'the answer to stdout (do not directly test on the sample inputs). '
|
||||
'Enclose your code within delimiters as follows.\n'
|
||||
'```python\n# YOUR CODE HERE\n```\n\n')
|
||||
return {'question': (f'### Question:\n{question}\n\n{fmt}### Answer: (use the '
|
||||
'provided format with backticks)\n\n'),
|
||||
'system': ('You are an expert Python programmer. You will be given a question '
|
||||
'(problem specification) and will generate a correct Python program '
|
||||
'that matches the specification and passes all tests. You will NOT '
|
||||
'return anything except for the program.')}
|
||||
|
||||
|
||||
@register_prompt_renderer('drop_es')
|
||||
def drop_es(question, sample, spec, prompt_style):
|
||||
# es drop: question block = bare passage + 'Question:' line (es does NOT
|
||||
# label the test passage; only exemplars carry labels)
|
||||
ps = (sample.metadata or {}).get('passage')
|
||||
return {'question': f'{ps}\nQuestion: {question}' if ps else f'Question: {question}'}
|
||||
|
||||
|
||||
@register_prompt_renderer('bbh_es')
|
||||
def bbh_es(question, sample, spec, prompt_style):
|
||||
# es bbh PROMPT_TEMPLATE: the test question is wrapped in the Q:/A:
|
||||
# contract (the CoT exemplars already follow this pattern)
|
||||
return {'question': (
|
||||
'Q: ' + question + '\nA: Let\'s think step by step. Put your final '
|
||||
'answer in the format of "So the answer is [ANSWER]" (without quotes '
|
||||
'and markdown) where [ANSWER] is the answer to the problem.\n')}
|
||||
|
||||
|
||||
@register_prompt_renderer('cot_letter_plain')
|
||||
def cot_letter_plain(question, sample, spec, prompt_style):
|
||||
# es mmlu-pro USER_PROMPT verbatim: Question:/Options: + 'A) x' --
|
||||
# NOTE es renders the TEST question options with PARENS while its
|
||||
# exemplars use 'A x' (space); replicate the inconsistency exactly
|
||||
if not sample.choices:
|
||||
return {}
|
||||
letters = 'ABCDEFGHIJ'
|
||||
opts = '\n'.join(f'{letters[i]}) {c}' for i, c in enumerate(sample.choices)
|
||||
if i < len(letters))
|
||||
return {'question': (
|
||||
f'Answer the following multiple choice question. The last line of '
|
||||
f"your response should be of the following format: 'ANSWER: [LETTER]' "
|
||||
f'(without quotes) where [LETTER] is one of '
|
||||
f'{",".join(letters[:len(sample.choices)])}. '
|
||||
f'Think step by step before answering.\n\nQuestion:\n{question}\nOptions:\n{opts}\n')}
|
||||
|
||||
|
||||
@register_prompt_renderer('cot_letter_zh')
|
||||
def cot_letter_zh(question, sample, spec, prompt_style):
|
||||
if not sample.choices:
|
||||
return {}
|
||||
letters = 'ABCDEFGH'
|
||||
opts = '\n'.join(f'{letters[i]}) {c}' for i, c in enumerate(sample.choices)
|
||||
if i < len(letters))
|
||||
# es cmmlu contract, verbatim (incl. trailing newline)
|
||||
return {'question': (
|
||||
f'回答下面的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：'
|
||||
f'"答案：[LETTER]"（不带引号），其中 [LETTER] 是 {",".join(letters[:len(sample.choices)])} 中的一个。'
|
||||
f'请在回答前进行一步步思考。\n\n问题：{question}\n选项：\n{opts}\n')}
|
||||
|
||||
|
||||
@register_prompt_renderer('cot_letter')
|
||||
def cot_letter(question, sample, spec, prompt_style):
|
||||
if not sample.choices:
|
||||
return {}
|
||||
letters = 'ABCDEFGH'
|
||||
opts = '\n'.join(f'{letters[i]}) {c}' for i, c in enumerate(sample.choices)
|
||||
if i < len(letters))
|
||||
return {'question': (
|
||||
f'Answer the following multiple choice question. The last line of '
|
||||
f"your response should be of the following format: 'ANSWER: [LETTER]' "
|
||||
f'(without quotes) where [LETTER] is one of {",".join(letters[:len(sample.choices)])}. '
|
||||
f'Think step by step before answering.\n\n{question}\n\n{opts}')}
|
||||
@ -32,6 +32,8 @@ async def generate_predictions(
|
||||
gen_kwargs: Optional[Dict[str, Any]] = None,
|
||||
progress: bool = True,
|
||||
env_factory=None,
|
||||
env_user_spec: str = '',
|
||||
no_shuffle: bool = False,
|
||||
system: str = '',
|
||||
max_turns: int = 8,
|
||||
max_input_chars: int = 0,
|
||||
@ -63,43 +65,142 @@ async def generate_predictions(
|
||||
done_count = 0
|
||||
t0 = time.time()
|
||||
|
||||
hle_system = [''] # mutable cell: answer_type-specific system prompt (hle)
|
||||
extra_system = [''] # mutable cell: renderer-provided system message (lcb etc.)
|
||||
|
||||
def assemble(sample: Sample) -> str:
|
||||
parts = []
|
||||
math_glue = False # es math few-shot: single \n before the test Problem:
|
||||
hle_system[0] = '' # reset per sample (es: answer_type-specific system role)
|
||||
extra_system[0] = '' # reset per sample (renderer system, e.g. lcb)
|
||||
if few_shot_text:
|
||||
parts.append(few_shot_text.strip()) # official exemplars, verbatim
|
||||
elif few_shot_num and few_shot_samples:
|
||||
letters_fs = 'ABCDEFGHIJ'
|
||||
for fs in few_shot_samples[:few_shot_num]:
|
||||
line = f'Question: {fs.input_text}'
|
||||
if fs.choices:
|
||||
line += '\n' + '\n'.join(f'{letters_fs[j]}. {c}' for j, c in enumerate(fs.choices))
|
||||
ans = fs.target if not isinstance(fs.target, list) else fs.target[0]
|
||||
line += f'\nAnswer: {ans}'
|
||||
parts.append(line)
|
||||
es_style = getattr(dataset_spec, 'prompt_style', '') in ('cot_letter', 'cot_letter_zh', 'cot_letter_plain')
|
||||
plain_style = getattr(dataset_spec, 'prompt_style', '') == 'cot_letter_plain'
|
||||
drop_style = getattr(dataset_spec, 'prompt_style', '') == 'drop_es'
|
||||
if es_style and len(few_shot_samples) > few_shot_num:
|
||||
# domain-matched selection (es parity): exemplars sharing the
|
||||
# current sample's category first, global first-N as fallback.
|
||||
# key: 'category' (cmmlu/mmlu_pro) OR 'subject' (mmlu) --
|
||||
# es reformat_subset regroups fewshot by subset_key
|
||||
def _cat_of(md):
|
||||
return ((md or {}).get('category') or (md or {}).get('subject')
|
||||
or (md or {}).get('level')) # math: per-Level exemplars
|
||||
|
||||
cat = _cat_of(sample.metadata)
|
||||
pool = [s for s in few_shot_samples if _cat_of(s.metadata) == cat]
|
||||
if len(pool) < few_shot_num:
|
||||
pool = pool + [s for s in few_shot_samples if _cat_of(s.metadata) != cat]
|
||||
sel = pool[:few_shot_num]
|
||||
else:
|
||||
sel = few_shot_samples[:few_shot_num]
|
||||
blocks = []
|
||||
for fs in sel:
|
||||
if drop_style:
|
||||
# es drop exemplar: full Passage + Question + bare-span Answer
|
||||
# (multi-span gold joins with ', ' -- teaches the model the
|
||||
# exact answer FORM the Hungarian EM compares against)
|
||||
line = f"Passage: {(fs.metadata or {}).get('passage', '')}\nQuestion: {fs.input_text}"
|
||||
ans = fs.target if not isinstance(fs.target, list) else ', '.join(str(t) for t in fs.target)
|
||||
line += f'\nAnswer: {ans}'
|
||||
elif plain_style:
|
||||
# es mmlu-pro exemplar (adapter sample_to_fewshot, verbatim):
|
||||
# Question:/Options:/A x + cot_content transformed
|
||||
# 'The answer is (X).' -> 'ANSWER: X.' -- exactly ONE answer
|
||||
# marker, no appended ANSWER line
|
||||
line = f'Question:\n{fs.input_text}'
|
||||
if fs.choices:
|
||||
line += '\nOptions:\n' + '\n'.join(f'{letters_fs[j]} {c}' for j, c in enumerate(fs.choices))
|
||||
ans = fs.target if not isinstance(fs.target, list) else fs.target[0]
|
||||
cot = (fs.metadata or {}).get('cot_content')
|
||||
if cot:
|
||||
ans_str = str(cot).strip().replace('The answer is', 'ANSWER:')
|
||||
ans_opt = ans_str.split('ANSWER:')[-1].split('.')[0].strip().strip('(').strip(')')
|
||||
ans_str = ans_str.replace(f'ANSWER: ({ans_opt})', f'ANSWER: {ans_opt}')
|
||||
line += f'\n{ans_str}'
|
||||
else:
|
||||
line += f'\nANSWER: {ans}'
|
||||
elif es_style:
|
||||
# es exemplar rendering: bare question + 'A) opt' + 'ANSWER: X'
|
||||
# (mimicry target for the CoT-last-line contract)
|
||||
line = fs.input_text
|
||||
if fs.choices:
|
||||
line += '\n' + '\n'.join(f'{letters_fs[j]}) {c}' for j, c in enumerate(fs.choices))
|
||||
ans = fs.target if not isinstance(fs.target, list) else fs.target[0]
|
||||
cot = (fs.metadata or {}).get('cot_content')
|
||||
if cot:
|
||||
line += f'\n{str(cot).strip()}'
|
||||
line += f'\nANSWER: {ans}'
|
||||
elif (fs.metadata or {}).get('reasoning') and not fs.choices:
|
||||
# es qa few-shot (gsm8k): question + Reasoning + ANSWER: boxed
|
||||
line = (f"{fs.input_text}\n\nReasoning:\n{str((fs.metadata or {}).get('reasoning', '')).strip()}\n\n"
|
||||
f'ANSWER: \\boxed{{{fs.target}}}')
|
||||
elif (fs.metadata or {}).get('es_math_fewshot'):
|
||||
# es math: Problem:/Solution: bare-answer exemplars
|
||||
line = f'Problem:\n{fs.input_text}\nSolution:\n{fs.target}'
|
||||
else:
|
||||
line = f'Question: {fs.input_text}'
|
||||
if fs.choices:
|
||||
line += '\n' + '\n'.join(f'{letters_fs[j]}. {c}' for j, c in enumerate(fs.choices))
|
||||
ans = fs.target if not isinstance(fs.target, list) else fs.target[0]
|
||||
line += f'\nAnswer: {ans}'
|
||||
blocks.append(line)
|
||||
if drop_style and few_shot_text:
|
||||
# hook 版范例已含完整 es 结构, 直接用
|
||||
parts.append(few_shot_text.strip() + '\n\n# Your Task\n---\n')
|
||||
elif drop_style:
|
||||
parts.append('You will be asked to read a passage and answer a question. '
|
||||
'Some examples of passages and Q&A are provided below.\n\n'
|
||||
'# Examples\n---\n' + '\n---\n'.join(blocks)
|
||||
+ '\n\n# Your Task\n---\n')
|
||||
elif plain_style:
|
||||
# es mmlu-pro: subject header FIRST, then exemplars, then the
|
||||
# user template (SYSTEM_W_EXAMPLES_PROMPT_TEMPLATE + '\n' + USER)
|
||||
subj = (sample.metadata or {}).get('category') or 'knowledge'
|
||||
parts.append(
|
||||
f'The following are multiple choice questions (with answers) about '
|
||||
f'{subj}. Think step by step and then finish your answer with '
|
||||
f"'ANSWER: [LETTER]' (without quotes) where [LETTER] is the correct "
|
||||
f'letter choice.\n\n' + '\n\n'.join(blocks))
|
||||
elif es_style:
|
||||
parts.append('Here are some examples of how to answer similar questions:\n\n'
|
||||
+ '\n\n'.join(blocks))
|
||||
elif blocks and ('\nReasoning:\n' in blocks[0] or blocks[0].startswith('Problem:\n')):
|
||||
# es gsm8k/math FEWSHOT_TEMPLATE header
|
||||
parts.append('Here are some examples of how to solve similar problems:\n\n'
|
||||
+ '\n\n'.join(blocks))
|
||||
if blocks[0].startswith('Problem:\n') and '\nReasoning:\n' not in blocks[0]:
|
||||
math_glue = True # es math: ONE newline before the test Problem:
|
||||
else:
|
||||
parts.extend(blocks)
|
||||
for key in attach_context_keys:
|
||||
ctx = (sample.metadata or {}).get(key)
|
||||
if ctx:
|
||||
parts.append(str(ctx))
|
||||
question = sample.input_text
|
||||
spec_style = getattr(dataset_spec, 'prompt_style', '') if dataset_spec is not None else ''
|
||||
if sample.choices and spec_style == 'cot_letter':
|
||||
letters = 'ABCDEFGH'
|
||||
opts = '\n'.join(f'{letters[i]}. {c}' for i, c in enumerate(sample.choices)
|
||||
if i < len(letters))
|
||||
question = (f'Answer the following multiple choice question. The last line of '
|
||||
f"your response should be of the following format: 'ANSWER: [LETTER]' "
|
||||
f'(without quotes) where [LETTER] is one of {letters[:len(sample.choices)]}. '
|
||||
f'Think step by step before answering.\n\n{question}\n\n{opts}')
|
||||
# prompt-style PLUGINS: each registered renderer rewrites the question
|
||||
# (and may set a system message); unknown styles -> generic handling
|
||||
from .prompt_renderers import render as _render
|
||||
out = _render(spec_style, question, sample, dataset_spec, prompt_style)
|
||||
if out:
|
||||
question = out.get('question', question)
|
||||
if out.get('system'):
|
||||
extra_system[0] = out['system']
|
||||
elif sample.choices:
|
||||
if prompt_style in ('strict_letter', 'auto'):
|
||||
# evalscope/OpenAI-style contract: reply ONLY 'ANSWER: X'
|
||||
# rendering is VERBATIM es: 'A) option' + 'one of A,B,C,D' --
|
||||
# 'A.' vs 'A)' alone swings hswag by 22 points on no-think Qwen3
|
||||
letters = 'ABCDEFGHIJ'
|
||||
opts = '\n'.join(f'{letters[i]}. {c}' for i, c in enumerate(sample.choices)
|
||||
opts = '\n'.join(f'{letters[i]}) {c}' for i, c in enumerate(sample.choices)
|
||||
if i < len(letters))
|
||||
question = (f'Answer the following multiple choice question. The entire '
|
||||
f'content of your response should be of the following format: '
|
||||
f"'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of "
|
||||
f'{letters[:len(sample.choices)]}.\n\n{question}\n\n{opts}')
|
||||
f'{",".join(letters[:len(sample.choices)])}.\n\n{question}\n\n{opts}')
|
||||
else:
|
||||
letters = 'ABCDEFGHIJ'
|
||||
opts = '\n'.join(f'{letters[i]}. {c}' for i, c in enumerate(sample.choices)
|
||||
@ -107,30 +208,35 @@ async def generate_predictions(
|
||||
question = (f'{question}\n\n{opts}\n\n'
|
||||
'Answer with the letter of the correct option.')
|
||||
elif sample.task_type in ('qa',):
|
||||
# hle OFFICIAL protocol: answer_type-specific system contract
|
||||
# hle OFFICIAL protocol: answer_type-specific SYSTEM contract (es
|
||||
# puts it in the system role; injected as a system message in
|
||||
# one(), the question itself stays bare)
|
||||
at = (sample.metadata or {}).get('answer_type')
|
||||
if at == 'exactMatch':
|
||||
question = (
|
||||
hle_system[0] = (
|
||||
'Your response should be in the following format:\n'
|
||||
'Explanation: {your explanation for your final answer}\n'
|
||||
'Exact Answer: {your succinct, final answer}\n'
|
||||
'Confidence: {your confidence score between 0% and 100% for your answer}\n\n'
|
||||
f'{question}')
|
||||
'Confidence: {your confidence score between 0% and 100% for your answer}')
|
||||
elif at == 'multipleChoice':
|
||||
question = (
|
||||
hle_system[0] = (
|
||||
'Your response should be in the following format:\n'
|
||||
'Explanation: {your explanation for your answer choice}\n'
|
||||
'Answer: {your chosen answer}\n'
|
||||
'Confidence: {your confidence score between 0% and 100% for your answer}\n\n'
|
||||
f'{question}')
|
||||
else:
|
||||
'Confidence: {your confidence score between 0% and 100% for your answer}')
|
||||
elif not getattr(dataset_spec, 'prompt_suffix', ''):
|
||||
question = (f'{question}\n\n'
|
||||
'End your reply with the final answer on its own last line '
|
||||
'in the form "Answer: <answer>".')
|
||||
ds_spec = dataset_spec
|
||||
if ds_spec is not None and getattr(ds_spec, 'prompt_suffix', ''):
|
||||
question = question + ds_spec.prompt_suffix
|
||||
parts.append(question)
|
||||
if math_glue and parts:
|
||||
# es competition_math: exactly ONE newline between the last
|
||||
# exemplar and the test 'Problem:' (FEWSHOT_TEMPLATE tail)
|
||||
parts[-1] = parts[-1] + '\n' + question
|
||||
else:
|
||||
parts.append(question)
|
||||
text = '\n\n'.join(parts)
|
||||
if max_input_tokens:
|
||||
# reserve room for the OUTPUT budget + safety margin, else the
|
||||
@ -168,7 +274,9 @@ async def generate_predictions(
|
||||
if type(env).run_task is not Environment.run_task:
|
||||
# self-running env (official engine bundles: tau2/swe)
|
||||
pred = await env.run_task(adapter, sample,
|
||||
max_turns=max_turns, system=system)
|
||||
max_turns=max_turns, system=system,
|
||||
user_adapter=_env_user_adapter(env_user_spec) if env_user_spec else None,
|
||||
gen_kwargs=gen_kwargs)
|
||||
if pred is None:
|
||||
traj = await drive(adapter, sample, env=env,
|
||||
max_turns=max_turns, system=system)
|
||||
@ -195,6 +303,12 @@ async def generate_predictions(
|
||||
|
||||
messages = ([ChatMessage(role='user', content=assemble(sample))]
|
||||
if isinstance(sample.input, str) else list(sample.input))
|
||||
if not system and extra_system[0] and isinstance(sample.input, str):
|
||||
# renderer-provided SYSTEM contract (es lcb expert-programmer)
|
||||
messages.insert(0, ChatMessage(role='system', content=extra_system[0]))
|
||||
if not system and hle_system[0]:
|
||||
# es hle: answer_type-specific format contract in the SYSTEM role
|
||||
messages.insert(0, ChatMessage(role='system', content=hle_system[0]))
|
||||
tools = None
|
||||
if sample.tools:
|
||||
tools = [{'name': t.name, 'description': t.description or '',
|
||||
@ -218,7 +332,7 @@ async def generate_predictions(
|
||||
_progress(progress, done_count, len(samples), t0, total_usage)
|
||||
return {'raw': text, 'usage': out.usage.model_dump()}
|
||||
|
||||
work = _apply_limits(samples, limit, limit_per_task)
|
||||
work = _apply_limits(samples, limit, limit_per_task, shuffle=not no_shuffle)
|
||||
# checkpointing: restore completed samples, generate only the rest
|
||||
ckpt_store = None
|
||||
if checkpoint:
|
||||
@ -227,8 +341,13 @@ async def generate_predictions(
|
||||
if isinstance(checkpoint, str):
|
||||
ckpt = checkpoint
|
||||
else:
|
||||
ckpt = checkpoint_path(os.path.expanduser('~/.cache/evalharness'),
|
||||
dataset_name, adapter.model or str(adapter))
|
||||
# include subset in the checkpoint key: same dataset under
|
||||
# different subsets (bbh tasks, lb2 lengths) must not share state
|
||||
sub = getattr(dataset_spec, 'subset', '') or ''
|
||||
ckpt = checkpoint_path(os.environ.get('EVALHARNESS_CACHE')
|
||||
or os.path.expanduser('~/.cache/evalharness'),
|
||||
f'{dataset_name}:{sub}' if sub else dataset_name,
|
||||
adapter.model or str(adapter))
|
||||
ckpt_store = CheckpointStore(ckpt, model=adapter.model or str(adapter))
|
||||
restored = ckpt_store.load()
|
||||
else:
|
||||
@ -238,7 +357,10 @@ async def generate_predictions(
|
||||
pending = []
|
||||
preds_by_key: Dict[str, Dict[str, Any]] = {}
|
||||
for i, s in enumerate(work):
|
||||
k = CheckpointStore.key_for(s, i) if ckpt_store else str(i)
|
||||
# NB: must be `is not None` -- an EMPTY store is falsy via __len__,
|
||||
# which silently switched the key scheme between fresh runs (str(i))
|
||||
# and resumed runs (key_for) and broke every restore
|
||||
k = CheckpointStore.key_for(s, i) if ckpt_store is not None else str(i)
|
||||
keys.append(k)
|
||||
if k in restored:
|
||||
preds_by_key[k] = restored[k]
|
||||
@ -250,7 +372,20 @@ async def generate_predictions(
|
||||
|
||||
async def run_one(i_s):
|
||||
i, s = i_s
|
||||
pred = await one(s)
|
||||
# transient network flaps (cluster routes re-converge): retry with
|
||||
# backoff so one ConnectError burst cannot kill the whole batch --
|
||||
# the adapter already retries 5xx/429 and the pool fails over per
|
||||
# instance; this is the last line of defense around asyncio.gather
|
||||
for attempt in range(6):
|
||||
try:
|
||||
pred = await one(s)
|
||||
break
|
||||
except Exception:
|
||||
if attempt == 5:
|
||||
raise
|
||||
# minute-scale backoff: cluster routes flap in multi-minute
|
||||
# bursts; short retries exhaust inside one dead window
|
||||
await asyncio.sleep((10, 20, 40, 60, 90)[attempt])
|
||||
if ckpt_store is not None:
|
||||
ckpt_store.append(keys[i], pred)
|
||||
return i, pred
|
||||
@ -272,23 +407,36 @@ def _apply_limits(samples: List[Sample], total: Optional[int],
|
||||
shuffle+seed mirror evalscope run.py: dataset_args.shuffle=True with
|
||||
--seed 42 -> random.Random(seed).shuffle BEFORE limiting, so both
|
||||
frameworks evaluate the IDENTICAL sample subset."""
|
||||
if shuffle:
|
||||
if shuffle and not per_task:
|
||||
import random
|
||||
|
||||
random.Random(seed).shuffle(samples)
|
||||
if per_task:
|
||||
seen: Dict[str, int] = {}
|
||||
out = []
|
||||
# evalscope semantics: each subset's records are shuffled with a
|
||||
# fresh Random(seed) INDEPENDENTLY, then capped at N (builder.py:
|
||||
# build_dataset_from_records per subset). Emulate exactly: group,
|
||||
# per-group shuffle, first-N. For single-pool datasets this is
|
||||
# identical to the global shuffle above.
|
||||
import random
|
||||
from collections import OrderedDict
|
||||
|
||||
def _key(s: Sample) -> str:
|
||||
return str((s.metadata or {}).get('subset')
|
||||
or (s.metadata or {}).get('category')
|
||||
or (s.metadata or {}).get('subject')
|
||||
or (s.metadata or {}).get('test_category')
|
||||
or (s.metadata or {}).get('length')
|
||||
or (s.metadata or {}).get('level')
|
||||
or getattr(getattr(dataset, 'spec', None), 'subset', 'default'))
|
||||
|
||||
groups: Dict[str, List[Sample]] = OrderedDict()
|
||||
for s in samples:
|
||||
key = str((s.metadata or {}).get('subset')
|
||||
or (s.metadata or {}).get('category')
|
||||
or (s.metadata or {}).get('subject')
|
||||
or (s.metadata or {}).get('test_category')
|
||||
or (s.metadata or {}).get('length')
|
||||
or getattr(getattr(dataset, 'spec', None), 'subset', 'default'))
|
||||
if seen.get(key, 0) < per_task:
|
||||
seen[key] = seen.get(key, 0) + 1
|
||||
out.append(s)
|
||||
groups.setdefault(_key(s), []).append(s)
|
||||
out: List[Sample] = []
|
||||
for lst in groups.values():
|
||||
if shuffle: # no_shuffle => raw first-N per group (same-questions)
|
||||
random.Random(seed).shuffle(lst)
|
||||
out.extend(lst[:per_task])
|
||||
samples = out
|
||||
if total:
|
||||
samples = samples[:total]
|
||||
@ -313,6 +461,8 @@ async def run_eval(
|
||||
judge: Optional[Any] = None,
|
||||
progress: bool = True,
|
||||
env: str = '',
|
||||
env_user_spec: str = '',
|
||||
no_shuffle: bool = False, # fixed-order selection: raw first-N (same-questions parity)
|
||||
system: str = '',
|
||||
max_turns: int = 8,
|
||||
max_input_chars: int = 0,
|
||||
@ -322,6 +472,7 @@ async def run_eval(
|
||||
dataset_name: str = 'adhoc',
|
||||
few_shot_num: int = -1,
|
||||
prompt_style: str = 'strict_letter',
|
||||
gen_profile: str = '',
|
||||
) -> EvalReport:
|
||||
"""Generate + score in one call. Model spec examples:
|
||||
'mock', 'mock:boxed', 'openai/http://gpu03:8000/v1?qwen3-8b', 'deploy:vllm/qwen3-8b'.
|
||||
@ -349,8 +500,16 @@ async def run_eval(
|
||||
raise
|
||||
recipe = EvalRecipe(name='adhoc', extract='identity',
|
||||
scorers={'acc': {'name': 'exact', 'mode': 'raw'}})
|
||||
samples = list(dataset)[:limit] if limit else list(dataset)
|
||||
samples = _apply_limits(samples, limit, limit_per_task)
|
||||
# materialize in a worker thread: hub downloads here are synchronous
|
||||
# (requests/ssl) and would otherwise stall the whole event loop
|
||||
raw_samples = await asyncio.to_thread(lambda: list(dataset))
|
||||
if limit:
|
||||
raw_samples = raw_samples[:limit]
|
||||
# generate_predictions applies the SAME deterministic limiting internally;
|
||||
# recompute on an equal copy so evaluate() zips against the exact work
|
||||
# list (positional pairing) instead of relying on in-place aliasing.
|
||||
samples = _apply_limits(list(raw_samples), limit, limit_per_task,
|
||||
shuffle=not no_shuffle) # MUST mirror generate_predictions
|
||||
if progress:
|
||||
mode = f'agent env={env}' if env else 'single-turn'
|
||||
print(f'generating: {adapter} on {len(samples)} samples '
|
||||
@ -364,7 +523,7 @@ async def run_eval(
|
||||
raise KeyError(f'unknown env {env!r}; available: {", ".join(ENV_REGISTRY.names())}')
|
||||
probe = get_env(env)
|
||||
if getattr(probe, 'needs_adapter', False):
|
||||
env_factory = lambda: get_env(env, adapter=adapter) # noqa: E731
|
||||
env_factory = lambda: get_env(env, adapter=adapter) # noqa: E731
|
||||
else:
|
||||
env_factory = lambda: get_env(env) # noqa: E731
|
||||
|
||||
@ -390,16 +549,35 @@ async def run_eval(
|
||||
|
||||
fn = prov.resolve_record_fn()
|
||||
fs_raw = load_raw_records(fs_spec)
|
||||
few_shot_samples = [fn(r) for r in fs_raw[:few_shot_num]]
|
||||
# keep the WHOLE dev split when samples carry a category:
|
||||
# es selects domain-MATCHED exemplars per subject (mmlu
|
||||
# biology questions get biology exemplars), we do the same
|
||||
# at assemble time; global first-N otherwise
|
||||
def _lv_of(md):
|
||||
return (md or {}).get('category') or (md or {}).get('level')
|
||||
|
||||
cats = {_lv_of(fn(r).metadata) for r in fs_raw[:200]}
|
||||
style_is = getattr(spec, 'prompt_style', '') if spec is not None else ''
|
||||
if len(cats) > 1 and spec is not None and \
|
||||
(style_is.startswith('cot_letter') or style_is == 'imo_es'):
|
||||
# mmlu-style per-subject OR math per-Level exemplars:
|
||||
# load the WHOLE few-shot split; assemble-time picks
|
||||
# domain-matched first-N (es reformat_subset semantics)
|
||||
few_shot_samples = [fn(r) for r in fs_raw]
|
||||
else:
|
||||
few_shot_samples = [fn(r) for r in fs_raw[:few_shot_num]]
|
||||
except Exception as e:
|
||||
print(f'few-shot: could not load {fs_split} split ({type(e).__name__}: '
|
||||
f'{str(e)[:80]}); continuing 0-shot', flush=True)
|
||||
|
||||
try:
|
||||
from .gen_profiles import merge_gen_kwargs
|
||||
preds, _usages, usage = await generate_predictions(
|
||||
adapter, samples, concurrency, progress=progress,
|
||||
gen_kwargs={**(getattr(spec, 'gen_config', {}) or {}), **(gen_kwargs or {})},
|
||||
adapter, list(raw_samples), concurrency, progress=progress,
|
||||
gen_kwargs=merge_gen_kwargs(name, spec, gen_kwargs, gen_profile),
|
||||
env_factory=env_factory,
|
||||
env_user_spec=env_user_spec,
|
||||
no_shuffle=no_shuffle,
|
||||
system=system, max_turns=max_turns, max_input_chars=max_input_chars,
|
||||
max_input_tokens=max_input_tokens,
|
||||
dataset_spec=spec,
|
||||
@ -439,6 +617,18 @@ async def run_eval(
|
||||
return report
|
||||
|
||||
|
||||
def _env_user_adapter(spec: str):
|
||||
"""Build (once per spec) the separate USER-simulator adapter for env
|
||||
benches (tau2 strong-user parity mode)."""
|
||||
global _ENV_USER_CACHE
|
||||
if spec not in _ENV_USER_CACHE:
|
||||
_ENV_USER_CACHE[spec] = _make_adapter(spec)
|
||||
return _ENV_USER_CACHE[spec]
|
||||
|
||||
|
||||
_ENV_USER_CACHE = {}
|
||||
|
||||
|
||||
def _make_adapter(spec: str) -> ModelAdapter:
|
||||
"""Model spec forms:
|
||||
- 'mock[:mode]' offline adapter
|
||||
@ -468,14 +658,24 @@ def _make_adapter(spec: str) -> ModelAdapter:
|
||||
from .pool import pooled
|
||||
|
||||
rest = spec[len('openai-pool/'):]
|
||||
m = __import__('re').search(r'\{(\d+)\.\.(\d+)\}', rest)
|
||||
if not m:
|
||||
ms = __import__('re').findall(r'\{(\d+)\.\.(\d+)\}', rest)
|
||||
if not ms:
|
||||
raise ValueError("openai-pool needs a {start..end} port range")
|
||||
lo, hi = int(m.group(1)), int(m.group(2))
|
||||
base_url, _, model = rest.partition('?')
|
||||
# expand EACH comma-separated segment's OWN range independently --
|
||||
# a global sub(count=1) would keep replacing only the FIRST range
|
||||
# and emit URLs with literal '{8200..8203}' in later segments
|
||||
specs = []
|
||||
for port in range(lo, hi + 1):
|
||||
specs.append(f'openai/{base_url.replace(m.group(0), str(port))}?{model}')
|
||||
for seg in base_url.split(','):
|
||||
seg = seg.strip()
|
||||
m = __import__('re').search(r'\{(\d+)\.\.(\d+)\}', seg)
|
||||
if m:
|
||||
lo, hi = int(m.group(1)), int(m.group(2))
|
||||
for port in range(lo, hi + 1):
|
||||
u = seg[:m.start()] + str(port) + seg[m.end():]
|
||||
specs.append(f'openai/{u}?{model}')
|
||||
elif seg:
|
||||
specs.append(f'openai/{seg}?{model}')
|
||||
adapter = pooled(specs)
|
||||
elif spec.partition(':')[0] == 'mock' and ':' in spec and '/' not in spec.partition(':')[0]:
|
||||
adapter = resolve_adapter('mock')
|
||||
|
||||
@ -53,7 +53,10 @@ class DockerSandbox(Sandbox):
|
||||
'--network', 'none', # untrusted code: no egress
|
||||
'--cpus', '2', '--memory', '2g', '--pids-limit', '256',
|
||||
'--read-only', '--tmpfs', '/tmp:rw,size=64m',
|
||||
'-v', f'{workdir}:/work:ro',
|
||||
# /work must be writable: BigCodeBench tasks write output
|
||||
# files (task_func_data/, matplotlib caches, etc.) to cwd;
|
||||
# the official Evaluate.Dockerfile runs with a writable fs
|
||||
'-v', f'{workdir}:/work:rw',
|
||||
]
|
||||
out_host = None
|
||||
if mounts:
|
||||
|
||||
17
test.md
Normal file
17
test.md
Normal file
@ -0,0 +1,17 @@
|
||||
请你一个一个回答我的问题
|
||||
|
||||
1. git clone `<repo>` EvalHarness 我们没有传吗，请你传到https://git.meta-stone.net/sora/EvalHarness
|
||||
2. pip install '.[bfcl]' 只有这一个吗，其他的都自动安装包了吗，我运行bfcl的时候会提醒我安装吗
|
||||
3. --resume 是什么左右
|
||||
4. 吧import asyncio 封装起来用evalharness.xx 直接启动
|
||||
5. 这个有后台选择吗，比如vllm ，openai， xxx'openai-pool/http://gpu1:{8123..8130}/v1,gpu2:{8200..8203}/v1?qwen3-8b!nothink',
|
||||
6. gen_profile='qwen3-es-parity',这个都可以选哪些有什么用
|
||||
7. 代码执行类 bench（humaneval / bigcodebench / live_code_bench） 可以运行之后代码自动下载吗，现在必须提前加载好吗？
|
||||
8. tau2 需要官方数据 + TAU2_DATA_DIR 环境变量：是什么意思，可以改成自动下载的吗，gen_kwargs={'max_input_tokens': 128000}, 这个可以放大yaml里面吗config
|
||||
9. temp=1 × N 次取均值，aime/hmmt 系列也可以放config里
|
||||
10. 跑全部 28 个 代码太复杂了，打包一下，包括config和run
|
||||
11. Prompt 渲染插件 复用率是多少，有多少个bench用的是同一个插件
|
||||
12. 模型适配插件 都有哪些模型可以跑
|
||||
13. 1.5 流量管理 如果跑满了，然后机器空了怎么办
|
||||
14. 沙箱插件 和env 插件的差距是什么，没懂
|
||||
15. P0 跑批编排层 运行确实不太方便，打包一下，
|
||||
Loading…
x
Reference in New Issue
Block a user