sora 9f765fc5ce Strip <think> blocks from raw predictions before extraction ...

Hybrid-thinking gateways inline the reasoning channel into content
wrapped in <think>..</think>; extractors then produced answers like
'3</think>Let me analyze...' (drop em 0.0 with f1 0.5 on samples the
model actually answered). Blocks and stray closers removed pre-extract.

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-17 07:41:33 +00:00

..

_lcb_official

dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes

2026-09-08 05:55:04 +00:00

recipes

bigcodebench: override the image ENTRYPOINT so our runner executes

2026-09-17 03:05:13 +00:00

__init__.py

Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests

2026-08-24 06:09:38 +00:00

aggregator.py

!perf spec flag; perf_stats full dashboard columns (success_rate, latency/ttft/tpot mean+P90+P99, tokens mean/total, output TPS, request QPS); summary.csv perf columns aligned

2026-08-27 02:49:55 +00:00

checkpoint.py

JSON-tolerant report/checkpoint saves (default=str)

2026-09-14 07:15:50 +00:00

extractor.py

dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes

2026-09-08 05:55:04 +00:00

math_grader.py

Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests

2026-08-24 06:09:38 +00:00

recipe.py

dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes

2026-09-08 05:55:04 +00:00

record.py

JSON-tolerant report/checkpoint saves (default=str)

2026-09-14 07:15:50 +00:00

registry.py

Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests

2026-08-24 06:09:38 +00:00

runner.py

Strip <think> blocks from raw predictions before extraction

2026-09-17 07:41:33 +00:00

scorer.py

bigcodebench: override the image ENTRYPOINT so our runner executes

2026-09-17 03:05:13 +00:00