Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tau2 scorer: read the composite 'reward' field

Browse Source 
tau2's current reward_info carries 'reward' (composite) + db_check /
action_checks; the scorer read the old environment_reward /
communication_reward split that no longer exists -- simulations scored
1.0 came out 0.0. Fallback to the old split kept for older engines.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-17 09:24:54 +00:00
parent 15b1574857
commit f844775f1e
1 changed files with 15 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
evalharness/eval/recipes/agent.py
View File

@ -262,16 +262,23 @@ def swe_bench_verified():
def _tau2_reward(pred, target, sample, ctx):
"""Score from the official engine's reward_info (env_state)."""
"""Score from the official engine's reward_info (env_state).
Current tau2 reward_info carries the COMPOSITE 'reward' plus detail
fields (db_check / action_checks / communicate_checks); the old
environment_reward/communication_reward split no longer exists."""
env_state = ctx.params.get('env_state') or {}
rewards = env_state.get('tau2_rewards') or {}
env_r = rewards.get('environment_reward')
comm_r = rewards.get('communication_reward')
vals = [r for r in (env_r, comm_r) if isinstance(r, (int, float))]
score = float(sum(vals) / len(vals)) if vals else 0.0
return ({'acc': score}, {'acc': {'mode': 'official_tau2',
'env_reward': env_r,
'comm_reward': comm_r}})
r = rewards.get('reward')
if not isinstance(r, (int, float)):
vals = [v for v in (rewards.get('environment_reward'),
rewards.get('communication_reward'))
if isinstance(v, (int, float))]
r = sum(vals) / len(vals) if vals else 0.0
return ({'acc': float(r)}, {'acc': {'mode': 'official_tau2',
'reward': r,
'db_check': rewards.get('db_check'),
'note': str((rewards.get('info') or {}).get('note', ''))[:120]}})
@register_eval('tau2_bench')