Keep generation max_tokens from being used as terminus-2 context, rewrite Debian/Ubuntu .sources to the Tsinghua mirror, and skip injecting truncation_tokens into DeepSWE extra_params.
Co-authored-by: Cursor <cursoragent@cursor.com>
Vendor LLMmap / llm-verify / llm-fingerprint-detector under
bash/fingerprint/tools so the three fingerprint benchmarks run with only
/data1/eval mounted (no /data1/xii dependency):
- run.py DEFAULT_TOOLS_ROOT prefers builtin tools/, falls back to /data1/xii
- exclude .git / node_modules / template backups
- detector dist/ (pre-built) retained; node_modules not needed at runtime
Agent CSV rows now use jsonl per-call stats or the report request summary only, so TTFT/latency stay on the same request口径.
Co-authored-by: Cursor <cursoragent@cursor.com>
Identity fingerprints ignore limit so later larger runs reuse cached predictions. Summary tables upsert by benchmark and take agent TTFT/latency from the report's per-request stats.
Co-authored-by: Cursor <cursoragent@cursor.com>
Completes the evalstone-built-in llmmap model library (conf.json +
model.pt + templates.json) so clone-only environments can run llmmap
without depending on /data1/xii mounts. 12MB inference weight.
Keep K3 suite selection and report-schema scoring in bash, merge K3/vision dataset_args into dpv4 yamls, and pin EvalScope at 735d920ee911 with local patches.
Co-authored-by: Cursor <cursoragent@cursor.com>
Move required path fields up front, persist custom benchmark suites, enlarge readable typography, show full suite benchmark names, add capability-domain trend charts with local Chart.js, and polish ranking name display.
Add a FastAPI panel under webui/ to select suites or benchmarks, configure model endpoint and thinking mode, stream evaluation logs, and visualize multi-model scores from output/.
- scripts/build_and_upload_docker.sh: sync context, build, save and upload
evalscope-complete-py312 image to ModelScope.
- scripts/deploy_remote_machine.sh: SSH to a fresh machine and run install.sh.
- scripts/run_multi_machine.sh: distribute official-suite benchmarks across
3 machines (gpu048, gpu049, gpu051).
- Check whether the local tokenizer path actually exists before trying to load.
- Log clear warnings when local path is missing or fails.
- Respect USE_MODELSCOPE_HUB when falling back to remote model hub.
- Raise a descriptive RuntimeError instead of silently retrying HuggingFace
and hanging on network-unreachable errors.
- Install torch==2.1.2 (CPU wheel) so AutoTokenizer can load models like
DeepSeek-V4-Flash without runtime import errors.
- Keep numpy==1.26.4 pinned; torch 2.1.x is compatible with it.
- Verify torch and transformers AutoTokenizer import in the image build.
Note: tools/docker/* is gitignored, so this file is force-added.
- make_test_spec() for all 500 samples was the real bottleneck that made
swe_bench_verified appear stuck at 'Processing records: 0%' even when all
instance images were already loaded locally.
- Now image names are computed directly from instance IDs, matching the
swebench naming convention. Local images are listed once and missing ones
are reported immediately. make_test_spec is only called for images that
actually need to be built.
- With 500 pre-loaded images, build_images() now completes in ~45s instead of
hanging indefinitely.
Also add bash/diagnose_swe_images.py for quickly checking which images are
missing locally.
