Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

--concurrency auto is the only form; gate starts at 2

Browse Source 
--auto-concurrency removed as promised; 'auto' now starts the adaptive
gate at 2 (it ramps on its own demand/health signals). Plan displays
'auto (start 2, gate decides)'.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-15 02:14:07 +00:00
parent 820a1dceac
commit f8ff19d4a5
2 changed files with 6 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -161,7 +161,7 @@ rep.save('gsm8k.report.json')
| `--limit N` / `--limit-per-task N` | 全局前 N / 每子集前 N（多科目 bench 用后者；可组合取交集） |
| `--subset` `--split` `--source` | 覆盖子集 / split / 数据源（可指本地路径离线跑） |
| `--concurrency N` | 并发（默认 32；长输出 bench 建议 8-16） |
| `--auto-concurrency` | 自适应并发门：按端点健康状况自动决定并发（健康且供不应求时 +1 爬坡，请求失败 ×0.7 退避，服务端 `/metrics` 可用时按排队信号调节）；当前值显示在进度条 `gate N`。此时 `--concurrency` 是起点不是上限 |
| `--concurrency auto` | 自适应并发门：从 2 起步，健康且供不应求时 +1 爬坡，请求失败 ×0.7 退避（服务端 `/metrics` 可用时按排队信号调节）；当前值显示在进度条 `gate N` |
| `--resume [PATH]` | 断点续跑；默认 `<cache-dir>/ckpt/<bench>.jsonl` |
| `--env NAME` | agent 环境（`bfcl_mock` 等） |
| `--perf` | 采集流式 TTFT / ITL / 重试率入报告 |

15
evalharness/cli.py
View File

@ -1152,12 +1152,6 @@ def build_parser() -> argparse.ArgumentParser:
p.add_argument('--provider', default='openai-chat',
choices=('openai-chat', 'openai-pool'),
help='API protocol/provider (default: openai-chat)')
p.add_argument('--auto-concurrency', action='store_true',
help='let the per-endpoint adaptive gate decide concurrency '
'(ramps while healthy, backs off x0.7 on failures, '
'server /metrics when available); current limit shows '
'on the progress bar as "gate N". --concurrency '
'becomes the starting point, not a cap')
p.add_argument('--judge-model', '--judge', dest='judge', default='',
help='judge model name with --judge-api-url, or full spec')
p.add_argument('--judge-api-url', default='',
@ -1185,9 +1179,10 @@ def build_parser() -> argparse.ArgumentParser:
help='send tools as text instead of native tool calls')
p.add_argument('--env', default='', help="agent environment (e.g. 'bfcl_mock') -> message pump")
p.add_argument('--concurrency', default='32',
help="parallel model calls (default 32); 'auto' = same as "
"--auto-concurrency (the adaptive gate decides, this "
"value becomes its starting point)")
help="parallel model calls (default 32); 'auto' = the "
"adaptive gate decides (starts at 2, ramps +1 while "
"healthy, backs off x0.7 on failures -- see 'gate N' "
"on the progress bar)")
p.add_argument('--progress', action='store_true', default=True,
help='show per-sample progress (default: on)')
p.add_argument('--no-progress', dest='progress', action='store_false',
@ -1257,7 +1252,7 @@ def main(argv=None) -> int:
# an int + the flag
if str(getattr(args, 'concurrency', '32')).strip().lower() == 'auto':
args.auto_concurrency = True
args.concurrency = 8
args.concurrency = 2 # gate start; it ramps on its own signals
else:
args.concurrency = int(args.concurrency)
return args.func(args)