Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tau2: floor max_turns at 40 (runner's generic 8 starved every sim)

Browse Source 
Both reward-0 runs died with 'Termination reason: max_steps' after 3-4
exchanges -- the runner passes its generic default (8) down, far too
few for greet->verify->find->policy->act->confirm. Official tau2 runs
use 40+; floor, don't cap.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-17 09:10:32 +00:00
parent fa60065099
commit b62088997d
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
evalharness/agent/envs/tau2_official.py
View File

@ -193,6 +193,10 @@ class Tau2Environment(Environment):
return self.reward_info return self.reward_info
async def run_task(self, adapter, sample, max_turns: int = 40, user_adapter=None, gen_kwargs=None, **kw): async def run_task(self, adapter, sample, max_turns: int = 40, user_adapter=None, gen_kwargs=None, **kw):
# the runner's generic default (8 turns) is for single-tool benches;
# a tau2 phone call needs greet -> verify -> find -> policy -> act ->
# confirm (official runs use 40+). Floor it, never cap below it.
max_turns = max(int(max_turns or 0), 40)
from tau2.data_model.tasks import Task from tau2.data_model.tasks import Task
from tau2.run import run_task from tau2.run import run_task