sora
|
fe1852302a
|
--config flag loads evalharness/config/<name>.yaml (per-bench generation params); yaml added to package-data; verified end-to-end
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-11 09:00:28 +00:00 |
|
ruoxi_sun
|
09b2add673
|
fingerprint: integrate fp_fusion model fingerprint benchmark
New evalharness/fingerprint/ package (from evalstone fp_fusion v1.1,
2026-09-07 pruning final): probe battery -> concurrent collection ->
five scoring views (verify/attribution/variant/adversarial/robustness),
bundled family aliases + 27 reference fingerprints (12 fp_fusion schema).
- CLI: 'evalharness fingerprint run ...' (REMAINDER passthrough, single
source of arg definitions) + 'fingerprint list' for bundled references
- imports rewritten package-relative; direct 'python3 run_fp_fusion.py'
execution kept working via package bootstrap
- offline analysis/collection scripts made path-independent (previously
pinned to a /opt/evalscope path absent on this host)
- shell scripts: hardcoded API key -> FP_API_KEY/OPENAI_API_KEY env vars
- --reference accepts short names resolved against bundled references/
- pyproject: +httpx dependency, package-data references/*.json
- tests/test_fingerprint.py: 10 offline tests (battery definitions,
assembly counts, normalization, signals, verdict ladder, CLI wiring)
- README: fingerprint section + architecture entry
Verified on H20-1: tests 10/10, installed CLI OK, full-protocol run vs
vectron GLM-5.3 reproduces baseline (score 0.9451, s_idn 0.846).
|
2026-09-11 03:52:21 +00:00 |
|
sora
|
bd35b5b724
|
Auto-export consolidated excel workbook (results.xlsx: Summary/Perf/Categories/Samples) on every --out-dir run; per-bench artifact .md instead of .txt; xlsxwriter joins core deps
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 10:29:13 +00:00 |
|
sora
|
07bc56f423
|
Package *.txt/*.md data files (simpleqa grader prompt crashed import on fresh installs)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 06:54:21 +00:00 |
|
sora
|
27cf8b3c7e
|
Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker
- progress/: Rich per-sample terminal progress plugin (Run Plan panel,
in-flight/rate/ETA bar); shared console + log-through-live to avoid
interleaved writes, rollback() pairs begin_sample on the retry path,
begin moved inside the semaphore (in-flight = actually generating),
graceful degradation when rich is absent
- cli.py: --provider/--api-url/--model composition (openai-chat |
openai-pool), --disable-thinking/--perf/--textools as first-class
flags, per-bench phase lines and done/failed result lines
- __init__: top-level run()/arun() entries (event-loop safe for notebooks)
- third_party/bfcl: vendored official BFCL ast_checker + type mappings
(Apache-2.0, provenance in __init__.py); imports rerouted locally,
underscore_to_dot parameterized; verified bit-identical with the
bfcl-eval package on 100 real rows -- removes the heavy extra
(pinned numpy + cloud SDK wall) from the install path
- runner: progress/status hooks through generate+evaluate, checkpoint
key scheme fix (empty-store falsy bug), tiered retry backoff,
multi-segment pool {range} expansion fix, adapter-instance passthrough
- pyproject: tree_sitter family joins core deps; [bfcl] extra retired
- README: rewritten (zh) -- install/quickstart/flags reference/bench
table/reliability/extension/architecture/validation
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 05:46:45 +00:00 |
|
|
|
89e721414f
|
Simplify install: light deps (datasets/pyarrow/sympy/pylatexenc/numpy/scipy) are DEFAULT; single [bfcl] extra for the heavy official checker
|
2026-08-25 02:51:55 +00:00 |
|
|
|
32b7276142
|
Extras: add [all] (conflict-free bundle) and [bfcl] (official checker, heavy deps isolated)
|
2026-08-25 02:47:27 +00:00 |
|
|
|
4a15f80897
|
Add evaluation layer + visualization: extract/score/aggregate plugins, 28 recipes, official-aligned scorers (PRM800K math, DROP Hungarian EM/F1, SimpleQA A/B/C judge), report artifacts, console renderers, CLI eval/viz, regression tests
|
2026-08-24 06:09:38 +00:00 |
|
|
|
f8cd15fea1
|
EvalHarness data layer: 28 dataset plugins, lazy materialize cache (raw/ + samples.jsonl + meta.json), ModelScope native loader, CLI list/fetch/unload/stats/show
|
2026-08-24 03:35:11 +00:00 |
|
