Vendor LLMmap / llm-verify / llm-fingerprint-detector under
bash/fingerprint/tools so the three fingerprint benchmarks run with only
/data1/eval mounted (no /data1/xii dependency):
- run.py DEFAULT_TOOLS_ROOT prefers builtin tools/, falls back to /data1/xii
- exclude .git / node_modules / template backups
- detector dist/ (pre-built) retained; node_modules not needed at runtime
Identity fingerprints ignore limit so later larger runs reuse cached predictions. Summary tables upsert by benchmark and take agent TTFT/latency from the report's per-request stats.
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep K3 suite selection and report-schema scoring in bash, merge K3/vision dataset_args into dpv4 yamls, and pin EvalScope at 735d920ee911 with local patches.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Check whether the local tokenizer path actually exists before trying to load.
- Log clear warnings when local path is missing or fails.
- Respect USE_MODELSCOPE_HUB when falling back to remote model hub.
- Raise a descriptive RuntimeError instead of silently retrying HuggingFace
and hanging on network-unreachable errors.
Allow adding a fixed number of tokens to every benchmark's max_tokens
on top of the configured value. Applied after --thinking-max-tokens-scale.
Update myread.md with examples.
- Add --folder-name CLI arg; defaults to safe model name, or
{model}_THINKING when --thinking is enabled.
- Move benchmark outputs under output/{folder_name}/{benchmark}/...
- Remove redundant {model_name} subfolder from predictions/reports/reviews.
- Update collect_results.py to write both CSV and Excel to
results/{folder_name}.csv/xlsx.
- Update perf backup/restore paths to match the new layout.
- Patch evalscope CacheManager to drop model_name from cache paths.
- Add --thinking-max-tokens-scale to multiply max_tokens when
--thinking is enabled.
- Add 'official' suite covering all benchmarks in the public
comparison tables (Kimi/GLM/DS etc).
- Print scale factor in run summary when thinking is on.
If a benchmark is not present in the YAML config, use default
parameters instead of skipping:
- temperature: 0.0
- top_p: 1.0
- stream: true
- max_tokens: 32768
