sora/evalstone
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
71 Commits 2 Branches 0 Tags
Commit Graph

10 Commits

Author SHA1 Message Date
sora
8b0a413cf1 Drop Harbor trial wall-clock from summary perf metrics. 
Agent CSV rows now use jsonl per-call stats or the report request summary only, so TTFT/latency stay on the same request口径.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-09-03 04:11:23 +00:00
sora
44303d8953 Allow resume when only sample limit changes, and keep CSV rows across runs. 
Identity fingerprints ignore limit so later larger runs reuse cached predictions. Summary tables upsert by benchmark and take agent TTFT/latency from the report's per-request stats.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-09-03 03:54:26 +00:00
ruoxi_sun
4f274c2c32 add fingerprint model library & fp_fusion integration 
- fingerprint benchmark: add fp_fusion(26-cell fusion) to collect_results/run.py
- run_llmmap.py: default model path to evalstone built-in model_library
- add model libraries (llmdetector 11 refs / fp_fusion 8 fusion refs /
  llmmap templates 60 models incl 8 new: GLM-5.2/5.3, DeepSeek-Flash/Pro/
  Flash-0731, Kimi-K3, MiniMax-M2.7, TianGong-Taie)
- add fp_fusion engine (battery/engine/scorer) + docs
- gitignore: exclude binary model weights and temp backups
 2026-09-03 02:38:35 +00:00
sora
13274243a0 Bump vendored EvalScope and add K3-ready DPV4 configs. 
Keep K3 suite selection and report-schema scoring in bash, merge K3/vision dataset_args into dpv4 yamls, and pin EvalScope at 735d920ee911 with local patches.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-09-02 07:30:48 +00:00
aaxiyao
8b7dff96e7 add fingerprint benchmark 2026-08-25 02:06:23 +00:00
sora
ba2b5c1772 fix: terminal_bench_v2_1 result collection, apt mirror, and restore run.py env var 2026-08-05 03:56:44 +00:00
sora
b1eb4e116a chore: update evalscope runner, thinking config and webui 2026-07-31 02:36:51 +00:00
sora
991a4a5f7f Add sandbox images download from ModelScope; fix Excel sample count total 2026-07-27 05:34:16 +00:00
sora
e26fa0fe44 feat: restructure output layout and add --folder-name 
- Add --folder-name CLI arg; defaults to safe model name, or
  {model}_THINKING when --thinking is enabled.
- Move benchmark outputs under output/{folder_name}/{benchmark}/...
- Remove redundant {model_name} subfolder from predictions/reports/reviews.
- Update collect_results.py to write both CSV and Excel to
  results/{folder_name}.csv/xlsx.
- Update perf backup/restore paths to match the new layout.
- Patch evalscope CacheManager to drop model_name from cache paths.
 2026-07-23 03:06:52 +00:00
sora
2ee0af0728 docs: add resource prep section and sync docker/bash helpers 
- Rewrite myread.md with proper markdown and four download sources:
  code, data, evalscope-complete-py312 image, execution images.
- Add bash/collect_results.py, perf_backup.py, pull_swe_bench_images.py
  for result aggregation, breakpoint perf recovery, and SWE-bench
  image pre-pulling.
- Update bash/run.py with multi-run suites, perf backup/restore, and
  whitelist-based summary.
- Update config/dpv4-int8_nothinking.yaml benchmark parameters.
- Ignore /docker_images and /results in .gitignore.
 2026-07-23 02:18:15 +00:00