Identity fingerprints ignore limit so later larger runs reuse cached predictions. Summary tables upsert by benchmark and take agent TTFT/latency from the report's per-request stats.
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep K3 suite selection and report-schema scoring in bash, merge K3/vision dataset_args into dpv4 yamls, and pin EvalScope at 735d920ee911 with local patches.
Co-authored-by: Cursor <cursoragent@cursor.com>
- make_test_spec() for all 500 samples was the real bottleneck that made
swe_bench_verified appear stuck at 'Processing records: 0%' even when all
instance images were already loaded locally.
- Now image names are computed directly from instance IDs, matching the
swebench naming convention. Local images are listed once and missing ones
are reported immediately. make_test_spec is only called for images that
actually need to be built.
- With 500 pre-loaded images, build_images() now completes in ~45s instead of
hanging indefinitely.
Also add bash/diagnose_swe_images.py for quickly checking which images are
missing locally.
- Add --folder-name CLI arg; defaults to safe model name, or
{model}_THINKING when --thinking is enabled.
- Move benchmark outputs under output/{folder_name}/{benchmark}/...
- Remove redundant {model_name} subfolder from predictions/reports/reviews.
- Update collect_results.py to write both CSV and Excel to
results/{folder_name}.csv/xlsx.
- Update perf backup/restore paths to match the new layout.
- Patch evalscope CacheManager to drop model_name from cache paths.
