370953729b
- perf_stats aggregator lives in eval/, not model/: the import failed silently and EVERY perf column was empty (not just ttft). Now warns on stderr instead of swallowing. - repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2 previously restored repeat 1's predictions and finished instantly with identical scores. rep1 keeps the legacy key (existing checkpoints still resume). - repeats summary: report the MEAN score and aggregate time/tokens over ALL runs (was: last run only). - README: six-benchmark command as the primary example. Co-Authored-By: Claude <noreply@anthropic.com>
64 lines
1.4 KiB
Markdown
64 lines
1.4 KiB
Markdown
# 配置目录
|
||
|
||
每个模型/协议一个文件夹，内含逐 bench 的 YAML 配置。
|
||
|
||
```
|
||
config/
|
||
├── README.md ← 本文件
|
||
└── dp4-nothink/ ← 模型/协议名
|
||
├── default.yaml ← 协议级默认参数（所有 bench 继承）
|
||
├── aime24.yaml ← 逐 bench 覆盖
|
||
├── aime25.yaml
|
||
├── ...
|
||
└── swe_bench_verified.yaml
|
||
```
|
||
|
||
## 每个 YAML 的结构
|
||
|
||
```yaml
|
||
# generation: 传给模型的参数
|
||
generation:
|
||
temperature: 1.0
|
||
max_tokens: 8192
|
||
top_p: 1.0
|
||
|
||
# run: 运行方式
|
||
run:
|
||
repeats: 12 # 跑 12 遍取均值（temp=1 方差测量用）
|
||
limit: null # 全量 / limit_per_task: 10 每子集 10 条
|
||
checkpoint: true
|
||
resume: true
|
||
|
||
# judge: LLM-judge 类 bench 需要
|
||
judge:
|
||
model: dp4-flash
|
||
api_url: http://174.1.51.4:30000/v1
|
||
```
|
||
|
||
## 优先级
|
||
|
||
```
|
||
DatasetSpec.gen_config < default.yaml < <bench>.yaml < 命令行显式参数
|
||
```
|
||
|
||
## 使用
|
||
|
||
```bash
|
||
# 单 bench
|
||
evalharness eval run aime25 --config dp4-nothink --api-url ... --model ...
|
||
|
||
# 多 bench（自动读各自的 yaml）
|
||
evalharness eval run aime24 aime25 aime26 hmmt26 --config dp4-nothink ...
|
||
|
||
# 查看某 bench 的生效配置
|
||
evalharness config show dp4-nothink aime25
|
||
```
|
||
|
||
## 新增模型配置
|
||
|
||
```bash
|
||
mkdir config/qwen3-es-parity
|
||
cp config/dp4-nothink/default.yaml config/qwen3-es-parity/
|
||
# 编辑 default.yaml，然后按需添加逐 bench 覆盖
|
||
```
|