sora/evalstone
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Trim official suite to Tencent comparison table union (~11 benchmarks)

Browse Source
This commit is contained in:
sora 2026-07-23 03:44:53 +00:00
parent 0c72d89d1c
commit 9755c2a5a0
2 changed files with 8 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
bash/run.py
View File

@ -148,22 +148,17 @@ SUITES = {
'single': ['openai_mrcr', 'longbench_v2', 'bfcl_v3', 'mmlu', 'cmmlu', 'bbh', 'simple_qa'], 'single': ['openai_mrcr', 'longbench_v2', 'bfcl_v3', 'mmlu', 'cmmlu', 'bbh', 'simple_qa'],
'agent': ['tau2_bench', 'general_fc'], 'agent': ['tau2_bench', 'general_fc'],
}, },
# 腾讯/官方入库标准对比套件：覆盖各厂商公开对比表里的全部 benchmark # 腾讯/官方入库标准对比套件：取各厂商公开对比表的并集，约 11 个 benchmark
# 注：swe_bench_*/terminal_bench_v2/browsecomp/mcp_atlas 需要额外镜像/工具 # 注：swe_bench_verified/terminal_bench_v2/browsecomp/tau2_bench 需要额外镜像/工具
'official': { 'official': {
'multi': [ 'multi': [],
'aime24', 'aime25', 'aime26', 'hmmt26',
'imo_answerbench', 'live_code_bench', 'humaneval', 'gpqa_diamond',
],
'single': [ 'single': [
'bigcodebench', 'bfcl_v3', 'competition_math', 'gsm8k', 'hle', 'super_gpqa', 'aime25', 'aime26',
'arc', 'bbh', 'cmmlu', 'drop', 'hellaswag', 'mmlu', 'mmlu_pro', 'hle', 'mmlu_pro', 'gpqa_diamond',
'simple_qa', 'trivia_qa', 'winogrande', 'openai_mrcr', 'longbench_v2', 'longbench_v2', 'live_code_bench', 'swe_bench_verified',
'swe_bench_verified', 'swe_bench_pro', 'swe_bench_multilingual_agentic',
'terminal_bench_v2',
], ],
'agent': [ 'agent': [
'tau2_bench', 'tau2_bench_retail', 'browsecomp', 'mcp_atlas', 'general_fc', 'tau2_bench', 'terminal_bench_v2', 'browsecomp',
], ],
}, },
} }

2
myread.md
View File

@ -409,7 +409,7 @@ python bash/run.py --suite full --thinking --folder-name my_exp_v1 --limit none
| `group1` | 代码 + 数学/推理 | 约 **22h** | 多机组 1 | | `group1` | 代码 + 数学/推理 | 约 **22h** | 多机组 1 |
| `group2` | 重知识 | 约 **26h** | 多机组 2 | | `group2` | 重知识 | 约 **26h** | 多机组 2 |
| `group3` | 长文本 + Agent + 知识 | 约 **27h** | 多机组 3 | | `group3` | 长文本 + Agent + 知识 | 约 **27h** | 多机组 3 |
| `official` | 官方对比表全部 benchmark | 视配置而定 | 覆盖 Kimi/GLM/DS 等公开对比表里的所有任务 | | `official` | `aime25`, `aime26`, `hle`, `mmlu_pro`, `gpqa_diamond`, `longbench_v2`, `live_code_bench`, `swe_bench_verified`, `tau2_bench`, `terminal_bench_v2`, `browsecomp` | 视配置而定 | 腾讯入库标准对比表的并集，约 11 个 benchmark；其中 swe/terminal/browsecomp/tau2 需额外镜像/工具 |
> **分组时间说明** > **分组时间说明**
> >