|
|
bee997423a
|
GPQA 0.15->0.55: (1) per-bench prompt_style cot_letter (es GPQA CoT template, matches its 'Think step by step' contract), (2) mcq_letter extractor case-insensitive answer-tail (ANSWER: vs answer), (3) deterministic choice shuffle
|
2026-08-28 18:01:16 +00:00 |
|
|
|
da989ea665
|
Align with local evalscope run.py (full read-through): (1) shuffle+seed42 sample selection identical to es _shuffle_in_place -- same exact sample subset both frameworks; (2) official MATH_PROMPT_TEMPLATE CoT boxed suffix on 7 math benches via DatasetSpec.prompt_suffix; (3) per-bench gen_config (temp/max_tokens from dpv4 yaml) on DatasetSpec, default-merged into gen_kwargs
|
2026-08-27 11:25:17 +00:00 |
|
|
|
a8d3400ed5
|
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
|
2026-08-25 02:21:59 +00:00 |
|
|
|
f8cd15fea1
|
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
|
2026-08-24 03:35:11 +00:00 |
|
