sora bee997423a GPQA 0.15->0.55: (1) per-bench prompt_style cot_letter (es GPQA CoT template, matches its 'Think step by step' contract), (2) mcq_letter extractor case-insensitive answer-tail (ANSWER: vs answer), (3) deterministic choice shuffle

2026-08-28 18:01:16 +00:00

..

recipes

Fix numeric normalization in math_equal: strip $, commas, markdown (**), trailing units ('540 meters') — gsm8k 0.45->0.80 on ladder20 cross-check; winogrande cross-scoring 20/20 agreement (delta = generation variance)

2026-08-26 17:15:13 +00:00

__init__.py

Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests

2026-08-24 06:09:38 +00:00

aggregator.py

!perf spec flag; perf_stats full dashboard columns (success_rate, latency/ttft/tpot mean+P90+P99, tokens mean/total, output TPS, request QPS); summary.csv perf columns aligned

2026-08-27 02:49:55 +00:00

checkpoint.py

tau2 via OFFICIAL engine: self-running env plugin (run_task hook + needs_adapter generic dispatch, no name hardcoding), deep generate() patch, official ToolCall shape, data plugin keeps verbatim Task json, official reward scoring; checkpoint plugin (per-sample resume) + summary csv with time/categories

2026-08-25 11:06:08 +00:00

extractor.py

GPQA 0.15->0.55: (1) per-bench prompt_style cot_letter (es GPQA CoT template, matches its 'Think step by step' contract), (2) mcq_letter extractor case-insensitive answer-tail (ANSWER: vs answer), (3) deterministic choice shuffle

2026-08-28 18:01:16 +00:00

math_grader.py

Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests

2026-08-24 06:09:38 +00:00

recipe.py

Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests

2026-08-24 06:09:38 +00:00

record.py

Add model layer: async ModelAdapter (openai_compatible + mock) returning structured ModelOutput(text,tool_calls,usage), Deployer registry (vllm/sglang docker-pinned via models.yaml, external), async run_eval generate->score, CLI --model, agent-ready SampleResult.trajectory/env_state, tests

2026-08-24 06:28:00 +00:00

registry.py

Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests

2026-08-24 06:09:38 +00:00

runner.py

Drop logprob scoring path entirely; MCQ = strict-letter generation (evalscope parity 'ANSWER: X'), few-shot defaults stay; fix output.py duplication

2026-08-25 02:26:26 +00:00

scorer.py

Fix numeric normalization in math_equal: strip $, commas, markdown (**), trailing units ('540 meters') — gsm8k 0.45->0.80 on ladder20 cross-check; winogrande cross-scoring 20/20 agreement (delta = generation variance)

2026-08-26 17:15:13 +00:00