- Add --folder-name and --thinking-max-tokens-scale to parameter table.
- Add thinking mode examples.
- Add official suite to suite details.
- Update output directory structure to reflect new folder-based layout.
- Add --folder-name CLI arg; defaults to safe model name, or
{model}_THINKING when --thinking is enabled.
- Move benchmark outputs under output/{folder_name}/{benchmark}/...
- Remove redundant {model_name} subfolder from predictions/reports/reviews.
- Update collect_results.py to write both CSV and Excel to
results/{folder_name}.csv/xlsx.
- Update perf backup/restore paths to match the new layout.
- Patch evalscope CacheManager to drop model_name from cache paths.
- Add --thinking-max-tokens-scale to multiply max_tokens when
--thinking is enabled.
- Add 'official' suite covering all benchmarks in the public
comparison tables (Kimi/GLM/DS etc).
- Print scale factor in run summary when thinking is on.
If a benchmark is not present in the YAML config, use default
parameters instead of skipping:
- temperature: 0.0
- top_p: 1.0
- stream: true
- max_tokens: 32768
- Allow --dataset swe_bench_verified,swe_bench_lite to pre-pull
multiple SWE-bench variants in one run.
- Validate dataset keys before loading.
- Update myread.md with multi-dataset example.
