sora 8c3d32bf2e swe scorer: return 'resolved' key (recipe metric name, not 'acc') ...

The scorer returned {'acc': ...} while the recipe registers the metric
as 'resolved' -- the aggregator looked for 'resolved' in scores, found
nothing, and reported 0.0 despite per-sample resolved=1.0 in the
checkpoint. Key names now match the recipe registration.

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-18 17:15:37 +00:00

..

agent

swe_agentic: env_state carries version/hints/env_commit (make_test_spec need)

2026-09-18 16:03:21 +00:00

config

thinking.yaml: swe_bench_verified_agentic entry (env + max_turns)

2026-09-18 04:57:38 +00:00

data

swe datasets: carry version/patch/hints/created_at in metadata

2026-09-18 08:11:18 +00:00

eval

swe scorer: return 'resolved' key (recipe metric name, not 'acc')

2026-09-18 17:15:37 +00:00

fingerprint

fingerprint: integrate fp_fusion model fingerprint benchmark

2026-09-11 03:52:21 +00:00

model

Background image service: pull-ahead + wait-barrier, unified lifecycle

2026-09-18 09:45:39 +00:00

progress

Idle watchdog back to 300s default; begin_bench clears the stale bar

2026-09-17 06:08:33 +00:00

sandbox

Background image service: pull-ahead + wait-barrier, unified lifecycle

2026-09-18 09:45:39 +00:00

themes

Fixes: theme markup closing tags ([bold green]...[/bold green]); evaluate() runs off the event loop (sympy equivalence on hard math froze the progress clock for the whole bench)

2026-09-11 04:36:51 +00:00

third_party/bfcl

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

viz

Unified, nicer result tables + conda env setup in README

2026-09-10 06:30:22 +00:00

__init__.py

Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker

2026-09-10 05:46:45 +00:00

__main__.py

EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show

2026-08-24 03:35:11 +00:00

cli.py

sandbox pull: SWE-bench per-instance pre-pull with resume (es port)

2026-09-17 09:36:20 +00:00

hooks.py

Pluginize the run shell: @register_progress (rich/plain), @register_theme, @register_hook (on_benchmark_failed/done), @register_prober; CLI gains --progress-plugin/--theme

2026-09-11 03:26:19 +00:00