- scripts/build_and_upload_docker.sh: sync context, build, save and upload
evalscope-complete-py312 image to ModelScope.
- scripts/deploy_remote_machine.sh: SSH to a fresh machine and run install.sh.
- scripts/run_multi_machine.sh: distribute official-suite benchmarks across
3 machines (gpu048, gpu049, gpu051).
- Check whether the local tokenizer path actually exists before trying to load.
- Log clear warnings when local path is missing or fails.
- Respect USE_MODELSCOPE_HUB when falling back to remote model hub.
- Raise a descriptive RuntimeError instead of silently retrying HuggingFace
and hanging on network-unreachable errors.
- Install torch==2.1.2 (CPU wheel) so AutoTokenizer can load models like
DeepSeek-V4-Flash without runtime import errors.
- Keep numpy==1.26.4 pinned; torch 2.1.x is compatible with it.
- Verify torch and transformers AutoTokenizer import in the image build.
Note: tools/docker/* is gitignored, so this file is force-added.
- make_test_spec() for all 500 samples was the real bottleneck that made
swe_bench_verified appear stuck at 'Processing records: 0%' even when all
instance images were already loaded locally.
- Now image names are computed directly from instance IDs, matching the
swebench naming convention. Local images are listed once and missing ones
are reported immediately. make_test_spec is only called for images that
actually need to be built.
- With 500 pre-loaded images, build_images() now completes in ~45s instead of
hanging indefinitely.
Also add bash/diagnose_swe_images.py for quickly checking which images are
missing locally.
- No need to keep compatibility with the old evalstone/ subdirectory layout.
- Now simply removes everything except datasets/ and docker/, then clones
the code directly into root.
- Previously the script cloned code into /evalstone/, so after
mounting the host directory to /opt/evalscope the container could not find
bash/run.py and the evalscope package.
- Now the script clones directly into root, preserving existing
datasets/ and docker/ directories.
- Also handles migration from old evalstone/ subdirectory layouts.
Allow adding a fixed number of tokens to every benchmark's max_tokens
on top of the configured value. Applied after --thinking-max-tokens-scale.
Update myread.md with examples.
- Add --folder-name and --thinking-max-tokens-scale to parameter table.
- Add thinking mode examples.
- Add official suite to suite details.
- Update output directory structure to reflect new folder-based layout.
- Add --folder-name CLI arg; defaults to safe model name, or
{model}_THINKING when --thinking is enabled.
- Move benchmark outputs under output/{folder_name}/{benchmark}/...
- Remove redundant {model_name} subfolder from predictions/reports/reviews.
- Update collect_results.py to write both CSV and Excel to
results/{folder_name}.csv/xlsx.
- Update perf backup/restore paths to match the new layout.
- Patch evalscope CacheManager to drop model_name from cache paths.
- Add --thinking-max-tokens-scale to multiply max_tokens when
--thinking is enabled.
- Add 'official' suite covering all benchmarks in the public
comparison tables (Kimi/GLM/DS etc).
- Print scale factor in run summary when thinking is on.
If a benchmark is not present in the YAML config, use default
parameters instead of skipping:
- temperature: 0.0
- top_p: 1.0
- stream: true
- max_tokens: 32768
- Allow --dataset swe_bench_verified,swe_bench_lite to pre-pull
multiple SWE-bench variants in one run.
- Validate dataset keys before loading.
- Update myread.md with multi-dataset example.
