Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
EvalHarness/evalharness/data/datasets
History
sora 78459c974e tau2 via OFFICIAL engine: self-running env plugin (run_task hook + needs_adapter generic dispatch, no name hardcoding), deep generate() patch, official ToolCall shape, data plugin keeps verbatim Task json, official reward scoring; checkpoint plugin (per-sample resume) + summary csv with time/categories
2026-08-25 11:06:08 +00:00
..
__init__.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
_bbh_cot_prompts.py
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
aime24.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
aime25.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
aime26.py
Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests
2026-08-24 06:09:38 +00:00
arc.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
bbh.py
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
bfcl_v3.py
Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests
2026-08-24 06:09:38 +00:00
bigcodebench.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
cmmlu.py
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
competition_math.py
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
drop.py
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
general_fc.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
gpqa_diamond.py
Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests
2026-08-24 06:09:38 +00:00
gsm8k.py
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
hellaswag.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
hle.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
hmmt26.py
Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests
2026-08-24 06:09:38 +00:00
humaneval.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
imo_answerbench.py
Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests
2026-08-24 06:09:38 +00:00
live_code_bench.py
Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests
2026-08-24 06:09:38 +00:00
longbench_v2.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
mmlu_pro.py
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
mmlu.py
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
openai_mrcr.py
Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests
2026-08-24 06:09:38 +00:00
simple_qa.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
swe_bench_verified.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
tau2_bench.py
tau2 via OFFICIAL engine: self-running env plugin (run_task hook + needs_adapter generic dispatch, no name hardcoding), deep generate() patch, official ToolCall shape, data plugin keeps verbatim Task json, official reward scoring; checkpoint plugin (per-sample resume) + summary csv with time/categories
2026-08-25 11:06:08 +00:00
trivia_qa.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00
winogrande.py
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
2026-08-24 03:35:11 +00:00