Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Strip <think> blocks from raw predictions before extraction

Browse Source 
Hybrid-thinking gateways inline the reasoning channel into content
wrapped in <think>..</think>; extractors then produced answers like
'3</think>Let me analyze...' (drop em 0.0 with f1 0.5 on samples the
model actually answered). Blocks and stray closers removed pre-extract.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-17 07:41:33 +00:00
parent 6da45e0218
commit 9f765fc5ce
1 changed files with 9 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
evalharness/eval/runner.py
View File

@ -120,6 +120,15 @@ def evaluate(
each sample gets its own container/workdir).""" each sample gets its own container/workdir)."""
result = _shell(sample, pred) result = _shell(sample, pred)
raw = result.raw_prediction raw = result.raw_prediction
# hybrid-thinking backends sometimes inline the reasoning channel
# into content wrapped in <think>...</think> (or leave a stray
# closer): extractors then fish answers out of reasoning text
# ('3</think>Let me analyze...'). Strip the blocks before extract.
if '<think>' in raw or '</think>' in raw:
import re as _re0
raw = _re0.sub(r'<think>.*?</think>', '', raw, flags=_re0.S)
raw = raw.replace('</think>', '')
try: try:
if bp is not None and sample.sandbox and sample.sandbox.image: if bp is not None and sample.sandbox and sample.sandbox.image:
bp.ensure(sample.sandbox.image) # wait only if this one still pulling bp.ensure(sample.sandbox.image) # wait only if this one still pulling