sora 370953729b Fix perf stats (wrong import path), per-repeat checkpoints, README

- perf_stats aggregator lives in eval/, not model/: the import failed
  silently and EVERY perf column was empty (not just ttft). Now warns
  on stderr instead of swallowing.
- repeats > 1 get their own checkpoint key (:rep2, :rep3, ...): repeat 2
  previously restored repeat 1's predictions and finished instantly with
  identical scores. rep1 keeps the legacy key (existing checkpoints still
  resume).
- repeats summary: report the MEAN score and aggregate time/tokens over
  ALL runs (was: last run only).
- README: six-benchmark command as the primary example.

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-11 13:38:04 +00:00

1.4 KiB

Raw Blame History

配置目录

每个模型/协议一个文件夹，内含逐 bench 的 YAML 配置。

config/
├── README.md              ← 本文件
└── dp4-nothink/           ← 模型/协议名
    ├── default.yaml       ← 协议级默认参数（所有 bench 继承）
    ├── aime24.yaml        ← 逐 bench 覆盖
    ├── aime25.yaml
    ├── ...
    └── swe_bench_verified.yaml

每个 YAML 的结构

# generation: 传给模型的参数
generation:
  temperature: 1.0
  max_tokens: 8192
  top_p: 1.0

# run: 运行方式
run:
  repeats: 12          # 跑 12 遍取均值（temp=1 方差测量用）
  limit: null          # 全量 / limit_per_task: 10 每子集 10 条
  checkpoint: true
  resume: true

# judge: LLM-judge 类 bench 需要
judge:
  model: dp4-flash
  api_url: http://174.1.51.4:30000/v1

优先级

DatasetSpec.gen_config  <  default.yaml  <  <bench>.yaml  <  命令行显式参数

使用

# 单 bench
evalharness eval run aime25 --config dp4-nothink --api-url ... --model ...

# 多 bench（自动读各自的 yaml）
evalharness eval run aime24 aime25 aime26 hmmt26 --config dp4-nothink ...

# 查看某 bench 的生效配置
evalharness config show dp4-nothink aime25

新增模型配置

mkdir config/qwen3-es-parity
cp config/dp4-nothink/default.yaml config/qwen3-es-parity/
# 编辑 default.yaml，然后按需添加逐 bench 覆盖

1.4 KiB Raw Blame History Unescape Escape

配置目录

每个 YAML 的结构

优先级

使用

新增模型配置

1.4 KiB

Raw Blame History