Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
194 Commits 1 Branch 0 Tags
Commit Graph

10 Commits

Author SHA1 Message Date
sora
b62088997d tau2: floor max_turns at 40 (runner's generic 8 starved every sim) 
Both reward-0 runs died with 'Termination reason: max_steps' after 3-4
exchanges -- the runner passes its generic default (8) down, far too
few for greet->verify->find->policy->act->confirm. Official tau2 runs
use 40+; floor, don't cap.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-17 09:10:32 +00:00
sora
fa60065099 tau2: silence per-message DEBUG floods; is_final_chunk only when done 
loguru restricted to WARNING+ (every orchestrator step dumped full
messages -- thousands of lines per bench). AssistantMessage.is_final_
chunk now False on tool-call turns (official adapter semantics: True
means the agent is FINISHED talking); always-True handed the turn
back to the user prematurely mid-action-sequence.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-17 09:07:53 +00:00
sora
d244d1a435 tau2 bridge: strip </think> leakage from simulator output 
The user simulator (GLM via our adapter) inlines its scenario reasoning
in content as '...instructions...</think>reply' -- passed through
unstripped, the AGENT receives the scenario's secret instructions
(task goal, disclosure strategy), inflating rewards. Both channels now
trimmed at the last </think>.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-17 09:03:52 +00:00
sora
bcb2128b02 tau2 env: auto-install the engine (local checkout -> GitHub), manual fallback 
Same UX as the docker image chain: try local sources first, then the
GitHub URL, and on total failure print the exact manual commands --
with a warning that PyPI's 'tau2' is an unrelated physics package.

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-17 08:56:48 +00:00
sora
27cf8b3c7e Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker 
- progress/: Rich per-sample terminal progress plugin (Run Plan panel,
  in-flight/rate/ETA bar); shared console + log-through-live to avoid
  interleaved writes, rollback() pairs begin_sample on the retry path,
  begin moved inside the semaphore (in-flight = actually generating),
  graceful degradation when rich is absent
- cli.py: --provider/--api-url/--model composition (openai-chat |
  openai-pool), --disable-thinking/--perf/--textools as first-class
  flags, per-bench phase lines and done/failed result lines
- __init__: top-level run()/arun() entries (event-loop safe for notebooks)
- third_party/bfcl: vendored official BFCL ast_checker + type mappings
  (Apache-2.0, provenance in __init__.py); imports rerouted locally,
  underscore_to_dot parameterized; verified bit-identical with the
  bfcl-eval package on 100 real rows -- removes the heavy extra
  (pinned numpy + cloud SDK wall) from the install path
- runner: progress/status hooks through generate+evaluate, checkpoint
  key scheme fix (empty-store falsy bug), tiered retry backoff,
  multi-segment pool {range} expansion fix, adapter-instance passthrough
- pyproject: tree_sitter family joins core deps; [bfcl] extra retired
- README: rewritten (zh) -- install/quickstart/flags reference/bench
  table/reliability/extension/architecture/validation

Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-10 05:46:45 +00:00
sora
46bef7d3dd dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes 
Co-Authored-By: Claude <noreply@anthropic.com>
 2026-09-08 05:55:04 +00:00
sora
111336cbee ladder20 complete 28/28 (Qwen3-8B pool): fixes en route — tau2 off-loop executor, double-encoded tool args, shared adapter cache; swe real-model 0.0 (oracle path verified 1.0) 2026-08-26 10:37:54 +00:00
sora
78459c974e tau2 via OFFICIAL engine: self-running env plugin (run_task hook + needs_adapter generic dispatch, no name hardcoding), deep generate() patch, official ToolCall shape, data plugin keeps verbatim Task json, official reward scoring; checkpoint plugin (per-sample resume) + summary csv with time/categories 2026-08-25 11:06:08 +00:00
sora
a32d902e94 Comparison-driven fixes: MCQ choices in prompt + letter contract, Answer: suffix for QA, markdown answer cleaning, trivia_qa answer_phrase priority, DROP gold-as-alternatives (OR) official semantics, retry on 5xx, numpy/scipy compat 2026-08-24 15:48:42 +00:00
sora
b2e7133b20 Add sandbox layer (docker exec hard-isolation + serve envs, refcounted acquire/release, atexit teardown, bind-mount sharing) and agent evaluation driver (message pump drive(), Trajectory, bfcl_mock env with official call-sequence scoring, mock:fc oracle); Deployer delegates to sandbox.serve_env; code_any extractor fixes indentation-stripping; CLI --env 2026-08-24 07:11:03 +00:00