EvalHarness

History

sora de847206e5 swe scoring: official swebench harness (es-identical path), backend hook

Both swe variants now score through make_test_spec -> container ->
apply-patch -> official eval_script -> get_eval_report -- the exact
pipeline es's eval_instance drives (swebench==4.1.0 installed). Our
in-container protocol stays as fallback when the package is absent.

env_reward gains a recipe-level 'backend' delegation hook (tau2's
reward_info reader and the swe official scorer both ride it); the
generic bfcl call-sequence comparison remains the default.

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-18 02:40:58 +00:00

__init__.py

Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests

2026-08-24 06:09:38 +00:00

_simpleqa_grader.txt

Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt

2026-08-25 02:21:59 +00:00

agent.py

swe scoring: official swebench harness (es-identical path), backend hook

2026-09-18 02:40:58 +00:00

judged.py

dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes