2c3672f2cb
- AdaptiveGate rewritten (Netflix Gradient2): window-vs-window per-stream speed gradient, count-driven windows with admission stamps, no thresholds or mode state machine; failures x0.7 + 30s drain pause - session-level admission for multi-turn agents (_SessionGate): in-progress sessions hold slots until done, newcomers queue at the door; capacity follows the model gate's discovered limit (CONCUR-style continuity) - image service: memory-first register (zero docker calls for known images), TTL-cached docker images listing, optimistic ready when the daemon is unreachable (docker save contention no longer kills runs); es tar loading removed in favor of ModelScope shipping (ms_images.py per-image tar upload/pull with round-trip verification) - runner: circuit breaker (12 consecutive failures abort the bench), first-failure error printed immediately - swe_agentic: image wait / docker run / rm off the event loop; exec timeout becomes an observation the agent can react to; container gets curlrc + git low-speed aborts (stalled github downloads fail fast) - eval run excludes its own endpoints from http_proxy (a sick personal proxy read as 'endpoint dead' and killed whole runs) - progress bar shows failed count; swe agentic exec_workers 2 -> 4 Co-Authored-By: Claude <noreply@anthropic.com>
配置目录
逐 bench 生成参数的 YAML，eval run 时自动应用。
config/
├── README.md ← 本文件
└── default.yaml ← 当前唯一配置（自动加载）
YAML 结构（单文件，扁平键）
default: # 协议级默认，所有 bench 继承
temperature: 0.0
top_p: 1.0
max_tokens: 32768
aime25: # 逐 bench 覆盖（键与 default 同级合并）
temperature: 1.0 # temp=1 方差测量
max_tokens: 8192
repeats: 12 # 跑 12 遍报均值；断点按轮隔离（:rep2 :rep3 ...）
longbench_v2:
max_tokens: 8192
max_input_tokens: 128000 # 超长输入的 tokenizer 中段截断预算
支持的键
| 键 | 作用 |
|---|---|
temperature top_p max_tokens |
生成参数，直传模型 |
max_input_tokens |
输入截断预算（tokenized 中段截断，头部尾部保留） |
repeats |
该 bench 重复轮数；summary 报均值，时间/token 报总和 |
优先级
DatasetSpec.gen_config < default 段 < bench 段 < 命令行显式参数
使用
# 目录里只有一个 yaml 时自动加载；--config 显式指定（省略 .yaml 后缀）
evalharness eval run aime25 --config default --api-url ... --model ...
# 多 bench：各自读自己的段
evalharness eval run aime24 aime25 aime26 hmmt26 --config default ...
新增一套配置
cp default.yaml glm53-nothink.yaml # 编辑后
evalharness eval run aime25 --config glm53-nothink ...
注意：同时存在多个 yaml 时不自动加载，必须 --config 指明。