sora/evalstone
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • ad61a2c44e Fix DeepSWE Pier routing and Terminal-Bench context/apt setup. main sora 2026-09-08 07:32:07 +00:00
  • 0c2698d0d7 fix Exeception typo in vendored LLMmap (breaks live inference) ruoxi_sun 2026-09-03 06:33:33 +00:00
  • 58657935fc bundle fingerprint tool repos into evalstone for self-containment ruoxi_sun 2026-09-03 06:14:36 +00:00
  • 8b0a413cf1 Drop Harbor trial wall-clock from summary perf metrics. sora 2026-09-03 04:11:23 +00:00
  • 44303d8953 Allow resume when only sample limit changes, and keep CSV rows across runs. sora 2026-09-03 03:53:36 +00:00
  • e9f2c9d3c6 gitignore: exclude fingerprint markdown docs ruoxi_sun 2026-09-03 02:36:54 +00:00
  • 2f4bbfe871 stop tracking fingerprint markdown docs (keep files on disk) ruoxi_sun 2026-09-03 02:36:45 +00:00
  • b28828f3e7 add llmmap model.pt weights to built-in model library ruoxi_sun 2026-09-03 02:33:55 +00:00
  • 4f274c2c32 add fingerprint model library & fp_fusion integration ruoxi_sun 2026-09-03 02:31:35 +00:00
  • a37a1165b2 Preload Pier uv and mini-swe-agent wheels so DeepSWE sandbox builds skip GitHub. sora 2026-09-02 09:44:17 +00:00
  • ee54cd6d5d Rename SciCode preload script away from the K3 sandbox name. sora 2026-09-02 08:41:09 +00:00
  • 70fe0c1d95 Build SciCode sandbox image from local wheels to avoid PyPI timeouts. sora 2026-09-02 08:08:46 +00:00
  • 13274243a0 Bump vendored EvalScope and add K3-ready DPV4 configs. sora 2026-09-02 07:30:48 +00:00
  • d85e9986c8 Add DeepSWE task image preload script. sora 2026-09-01 03:36:30 +00:00
  • 0f7c460f1b Add SciCode sandbox preload and extra bench configs for dpv4. sora 2026-08-31 07:24:51 +00:00
  • 8b7dff96e7 add fingerprint benchmark aaxiyao 2026-08-25 02:06:23 +00:00
  • 897f8fb1b9 document benchmark and environment matrix sora 2026-08-18 05:48:47 +00:00
  • ff946f221a add beginner docker evaluation SOP sora 2026-08-18 03:32:51 +00:00
  • 4d54a8c7ea update benchmark suites and evaluation docs sora 2026-08-18 03:19:01 +00:00
  • 922ee3af43 chore: add pyproject.toml and requirements-docker.txt for non-Docker env reproduction sora 2026-08-06 06:47:59 +00:00
  • 9fb2a0d2c8 config: increase terminal_bench_v2_1 timeout_multiplier and max_turns sora 2026-08-06 02:47:24 +00:00
  • 0d7608c867 feat: add docker image cleanup script for benchmark images sora 2026-08-05 05:38:20 +00:00
  • ba2b5c1772 fix: terminal_bench_v2_1 result collection, apt mirror, and restore run.py env var sora 2026-08-05 03:56:44 +00:00
  • 8ad6d26533 feat: add script to pre-download terminal_bench_v2_1 dataset and docker images sora 2026-08-04 02:23:47 +00:00
  • 610d6bb2be fix: set LITELLM_LOCAL_MODEL_COST_MAP=True to avoid GitHub fetch timeout sora 2026-08-03 07:14:05 +00:00
  • eb0309baad fix: default API key to EMPTY for local endpoints sora 2026-08-03 07:11:41 +00:00
  • 5fdf75e349 chore: remove accidentally committed experimental api_case.sh sora 2026-08-03 05:54:18 +00:00
  • 863af5dd12 refactor: move k3_report_test.py to bash/case and make repo-root detection robust sora 2026-08-03 05:53:59 +00:00
  • cdfe59cbee feat: add k3_report_test.py to reproduce available Kimi K3 report benchmarks sora 2026-08-03 05:40:44 +00:00
  • 4f33521567 chore: upgrade vendored evalscope to upstream v1.9.1 and reapply local patches sora 2026-08-03 05:28:50 +00:00
  • e388a7561d feat: support --thinking-budget-tokens to control reasoning budget via API sora 2026-07-31 06:03:06 +00:00
  • 6ccc38bcb5 docs: restructure README workflow to price first, then select datasets, then test sora 2026-07-31 03:30:19 +00:00
  • bd1e2d7e46 feat: add multi-model pricing, cost prediction and budget filter sora 2026-07-31 03:04:28 +00:00
  • 025763be2e feat: refactor GLM52_API_TEST1.sh to generic API runner and add API test README sora 2026-07-31 02:50:23 +00:00
  • 11d921071b feat: add benchmark cost calculator and generic API test runner sora 2026-07-31 02:44:14 +00:00
  • b1eb4e116a chore: update evalscope runner, thinking config and webui sora 2026-07-31 02:36:51 +00:00
  • bde21f6624 Replace meaningless trend charts with overall rank and win matrix in results page sora 2026-07-29 08:19:04 +00:00
  • 28f22c11e3 Redesign webui: compact layout, search/filter, command preview, progress bar, presets sora 2026-07-29 08:08:06 +00:00
  • 541b0b477d Improve EvalStone web UI paths, custom suites, and results charts. sora 2026-07-29 07:38:05 +00:00
  • 7c449d3126 Add local EvalStone web UI for launch and results comparison. webui-only sora 2026-07-29 06:49:39 +00:00
  • 8a4c6b279e Remove accidental MARKER_TEST file sora 2026-07-29 06:37:43 +00:00
  • b585c1e814 Update thinking config auto-selection, deploy script, README, and SWE-bench utils sora 2026-07-29 06:37:36 +00:00
  • b307e01cbf Add sync_to_machine.sh for rsync code/data to remote machines sora 2026-07-28 03:09:25 +00:00
  • f758f94828 Update Dockerfile: install torch 2.4.0 (CPU) for tokenizer sora 2026-07-28 02:46:39 +00:00
  • ee27130d4d Update multi-machine distribution: move hle to gpu051 sora 2026-07-28 02:40:37 +00:00
  • 5ab3f249c6 Add deployment and multi-machine benchmark scripts sora 2026-07-28 02:34:46 +00:00
  • 5f0b63bf0f Improve tokenizer loading fallback in bash/run.py sora 2026-07-28 02:28:31 +00:00
  • a194724bcb Add torch (CPU) to evalscope-complete-py312 Docker image for tokenizer support sora 2026-07-28 02:20:26 +00:00
  • c3b7f340e8 Add install.sh wrapper pointing to scripts/deploy_evalscope.sh sora 2026-07-28 02:16:19 +00:00
  • ae5a1a192e Optimize SWE-bench build_images: skip make_test_spec when images are pre-loaded sora 2026-07-27 08:46:15 +00:00
  • 0a3b880da3 Simplify deploy_evalscope.sh: remove old evalstone/ migration sora 2026-07-27 07:59:57 +00:00
  • 279a83051d Fix deploy_evalscope.sh: clone code into EVALSCOPE_DIR root instead of evalstone/ subdirectory sora 2026-07-27 07:49:16 +00:00
  • 5532c9403e Add checkpoint skip for loaded images and --download-swe flag sora 2026-07-27 06:22:39 +00:00
  • 8d6c5d8c1f Add logging for sandbox image preparation step sora 2026-07-27 06:16:21 +00:00
  • 6cb82bbc61 Remove docker pull fallback for sandbox images; use ModelScope only sora 2026-07-27 05:43:39 +00:00
  • 991a4a5f7f Add sandbox images download from ModelScope; fix Excel sample count total sora 2026-07-27 05:34:16 +00:00
  • 10eecf787d Add SWE image save/load scripts and update Docker config docs sora 2026-07-24 06:27:49 +00:00
  • 92f9c970e3 Restore hle in official suite sora 2026-07-24 03:18:41 +00:00
  • b15d961153 Expand README.md with full CLI parameters, suite benchmark lists, timing, and output statistics sora 2026-07-23 07:00:08 +00:00
  • 6aacfc14bf Rewrite README.md to focus on code and scripts, not model-specific customization sora 2026-07-23 06:55:56 +00:00
  • e9f7940c20 Add project README.md sora 2026-07-23 06:54:36 +00:00
  • 69b240ad6d Update official suite: aime25/aime26/live_code_bench multi-run + core single/agent benchmarks sora 2026-07-23 03:58:09 +00:00
  • 9755c2a5a0 Trim official suite to Tencent comparison table union (~11 benchmarks) sora 2026-07-23 03:44:53 +00:00
  • 0c72d89d1c feat(run.py): add --max-tokens-add parameter sora 2026-07-23 03:18:30 +00:00
  • a8c6fbdc75 docs(myread.md): document --folder-name, thinking scale, official suite sora 2026-07-23 03:08:33 +00:00
  • e26fa0fe44 feat: restructure output layout and add --folder-name sora 2026-07-23 03:06:52 +00:00
  • 1308ddd251 feat(run.py): add --thinking-max-tokens-scale and official suite sora 2026-07-23 02:57:53 +00:00
  • a3251f056f feat(run.py): default generation config for unknown benchmarks sora 2026-07-23 02:40:32 +00:00
  • 3afd57b5a4 feat(pull_swe_bench_images): support comma-separated --dataset sora 2026-07-23 02:34:49 +00:00
  • 2ee0af0728 docs: add resource prep section and sync docker/bash helpers sora 2026-07-23 02:18:15 +00:00
  • 03b49a39d0 all sora 2026-07-22 03:26:58 +00:00
  • cfa58f869f all sora 2026-07-21 09:32:49 +00:00
  • 85e1a9796d xii aaxiyao 2026-07-10 04:02:28 +00:00
  • d7547f642a readme：如下 sora 2026-07-08 08:56:47 +00:00