sora 565ba11790 swe datasets: carry version/patch/hints/created_at in metadata ...

Official make_test_spec indexes MAP_REPO_VERSION_TO_SPECS[repo][version]
-- a None version KeyError'd the official scorer (first agentic run
scored 0 with 'KeyError: None' hidden in details). The base converter
never copied these fields; both variants now carry them, verified with
a live make_test_spec call (spec + 2KB eval script + correct image key).

Co-Authored-By: Claude <noreply@anthropic.com>

2026-09-18 08:11:18 +00:00

..

datasets

swe datasets: carry version/patch/hints/created_at in metadata

2026-09-18 08:11:18 +00:00

__init__.py

EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show

2026-08-24 03:35:11 +00:00

dataset.py

dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes

2026-09-08 05:55:04 +00:00

loader.py

aria2c multi-connection dataset downloads (CDN edge roulette fix)

2026-09-17 08:03:34 +00:00

registry.py

dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes

2026-09-08 05:55:04 +00:00

sample.py

EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show

2026-08-24 03:35:11 +00:00

spec.py

GPQA 0.15->0.55: (1) per-bench prompt_style cot_letter (es GPQA CoT template, matches its 'Think step by step' contract), (2) mcq_letter extractor case-insensitive answer-tail (ANSWER: vs answer), (3) deterministic choice shuffle

2026-08-28 18:01:16 +00:00