Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tau2 bridge: strip </think> leakage from simulator output

Browse Source 
The user simulator (GLM via our adapter) inlines its scenario reasoning
in content as '...instructions...</think>reply' -- passed through
unstripped, the AGENT receives the scenario's secret instructions
(task goal, disclosure strategy), inflating rewards. Both channels now
trimmed at the last </think>.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-17 09:03:52 +00:00
parent bcb2128b02
commit d244d1a435
1 changed files with 9 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
evalharness/agent/envs/tau2_official.py
View File

@ -119,6 +119,14 @@ def _patch_tau2_generate(adapter, user_adapter=None, gen_kwargs=None) -> None:
import json as _json import json as _json
# strip thinking leakage: hybrid backends inline reasoning in
# content as '...scenario reasoning...</think>visible reply' -- the
# USER simulator's leak hands the agent the scenario's secret
# instructions (task goal, what to disclose), inflating rewards
text = out.text or ''
if '</think>' in text:
text = text.rsplit('</think>', 1)[-1].strip()
calls = [] calls = []
for c in out.tool_calls: for c in out.tool_calls:
try: try:
@ -127,7 +135,7 @@ def _patch_tau2_generate(adapter, user_adapter=None, gen_kwargs=None) -> None:
args = {'raw': c.arguments} args = {'raw': c.arguments}
calls.append(TauToolCall(id=c.id or f'call_{c.name}', name=c.name, calls.append(TauToolCall(id=c.id or f'call_{c.name}', name=c.name,
arguments=args or {})) arguments=args or {}))
return AssistantMessage(role='assistant', content=out.text or '', return AssistantMessage(role='assistant', content=text,
tool_calls=calls or None, cost=None, tool_calls=calls or None, cost=None,
usage=None, raw_data=None) usage=None, raw_data=None)