Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • 370953729b Fix perf stats (wrong import path), per-repeat checkpoints, README sora 2026-09-11 13:38:04 +00:00
  • 52872bcbc7 Per-benchmark Excel: each <bench>/<bench>.xlsx (4 sheets) alongside report.jsonl sora 2026-09-11 09:58:57 +00:00
  • f118f4092b Remove detail.md from per-benchmark output (user preference: report.jsonl + summary.xlsx only) sora 2026-09-11 09:51:29 +00:00
  • ed675e790f Auto-stream threshold 8192→100000 (streaming strips all thinking params on GLM gateway; non-streaming works for 32k); EVALHARNESS_NO_AUTOSTREAM=1 to disable entirely sora 2026-09-11 09:50:03 +00:00
  • f5aaddb870 Progress bar: colored fields (cur=yellow, elapsed=green, eta=cyan, retries=red, rate=dim) sora 2026-09-11 09:45:01 +00:00
  • 624b2fd37e aime/hmmt max_tokens 32768→8192: stays on non-stream path where GLM gateway honors enable_thinking:false (thinking off, 15s/题) sora 2026-09-11 09:43:36 +00:00
  • 3f555878b8 Fix: max_input_tokens extracted from YAML config and passed as the separate run_eval param (was going into gen_kwargs where truncation never saw it -> raw 128k text sent to gateway -> 400) sora 2026-09-11 09:28:13 +00:00
  • ff5d40aad5 Read timeout 300→600s base (GLM gateway 30+s startup on 100k+ token inputs) sora 2026-09-11 09:24:16 +00:00
  • 7ada0113ad max_input_tokens back to 128000 (user confirmed same behavior as 120k) sora 2026-09-11 09:23:43 +00:00
  • b75fbc1a12 max_input_tokens 128000→120000 (GLM gateway unstable on very long inputs, 30s timeout at 128k) sora 2026-09-11 09:22:55 +00:00
  • ebc51de274 Rename dp4-nothink.yaml → default.yaml (auto-loaded) sora 2026-09-11 09:12:41 +00:00
  • ed345c5ac1 Config auto-loads: single yaml in config/ becomes default generation params sora 2026-09-11 09:04:23 +00:00
  • fe1852302a --config flag loads evalharness/config/<name>.yaml (per-bench generation params); yaml added to package-data; verified end-to-end sora 2026-09-11 09:00:28 +00:00
  • ebf7a0e7e5 Config: generation params + repeats + max_input_tokens only; judge/env/limit controlled elsewhere sora 2026-09-11 08:47:25 +00:00
  • 60bdef3936 Complete config: all fields included (env, limit, limit_per_task, judge, judge_url, repeats, max_input_tokens, max_turns, concurrency) sora 2026-09-11 08:45:35 +00:00
  • eb6028e8c9 Config simplified: default block + per-bench overrides only (removed env/limit/judge/concurrency; max_input_tokens 128000) sora 2026-09-11 08:42:21 +00:00
  • 0815f6e4f8 Config: evalscope generation_config format + our extras (repeats, env) at bench level; parallel_tool_calls noted as unsupported in adapter sora 2026-09-11 08:37:18 +00:00
  • 0ada7d8a0e Config: exact copy of evalscope dpv4-int8_nothinking.yaml (generation_config + dataset_args + agent_config, zero custom fields) sora 2026-09-11 08:35:05 +00:00
  • 3a4be917e6 Restored checkpoint predictions' usage counted in totals (was 0 for fully-restored benches); token stats now correct on resume sora 2026-09-11 08:27:55 +00:00
  • b8ed18771a Config exactly matches evalscope dpv4-int8_nothinking.yaml (verified ✅ all match); concurrency removed (CLI controls it) sora 2026-09-11 08:25:15 +00:00
  • 787a8a28e9 Standard YAML indentation, no inline braces sora 2026-09-11 08:09:09 +00:00
  • 4d7567adf8 Flatten config: direct key-value per bench, no nested groups sora 2026-09-11 08:05:36 +00:00
  • d16009d054 Config: single dp4-nothink.yaml with all 28 bench configs (evalscope-style) sora 2026-09-11 08:02:43 +00:00
  • c27a22555d Config directory: dp4-nothink/ with 28 per-bench YAMLs (aime repeats=12, judged, agent, long-context) + README sora 2026-09-11 07:59:42 +00:00
  • 77b67f8bbc gen_profiles.yaml moved to evalharness/config/ (search path added); /data2/hf_home deleted (8.6G freed) sora 2026-09-11 07:48:41 +00:00
  • 471e0ca1a9 gen_profiles.yaml: YAML profiles (dp4-nothink, qwen3-es-parity, glm53-nothink, t1-short) — edit file to tune parameters, no code changes needed sora 2026-09-11 07:45:51 +00:00
  • 63ffd1b613 Fix NameError _fmt (module-level helper restored); aime/hmmt max_tokens 32768->8192 (non-stream path, gateway honors nothink) sora 2026-09-11 07:20:02 +00:00
  • d64b236540 run_one: exactly 1 attempt (adapter handles all retries internally) sora 2026-09-11 07:07:22 +00:00
  • 55b2beab54 Retry chain sanity: adapter 3 retries for long budgets / 6 for normal (was 6 flat); run_one 3 retries (was 6) with shorter backoff; worst case now 12 attempts ~5min (was 42+ attempts ~20min); progress bar shows live retry count; heartbeat recreated on every reset_samples sora 2026-09-11 07:06:11 +00:00
  • d00f330430 Fix frozen progress clock: heartbeat recreated on every reset_samples (previous one died during pause/resume cycles); auto-stream path appends /no_think when gateway drops chat_template_kwargs on streaming sora 2026-09-11 06:39:55 +00:00
  • 8bac3483b6 Fixes: theme markup closing tags ([bold green]...[/bold green]); evaluate() runs off the event loop (sympy equivalence on hard math froze the progress clock for the whole bench) sora 2026-09-11 04:35:27 +00:00
  • 09b2add673 fingerprint: integrate fp_fusion model fingerprint benchmark ruoxi_sun 2026-09-11 03:08:37 +00:00
  • 7473170784 Auto-stream long generations (max_tokens>8192) and aggregate to the non-stream response shape: gateways hang on large buffered NON-streaming requests (aime25's 32k budget stalled forever); streaming starts emitting immediately so a stuck endpoint surfaces in ~60s instead of the full adaptive read timeout; lazy httpx import fix sora 2026-09-11 03:41:24 +00:00
  • 5af8e2a7fd Pluginize the run shell: @register_progress (rich/plain), @register_theme, @register_hook (on_benchmark_failed/done), @register_prober; CLI gains --progress-plugin/--theme sora 2026-09-11 03:26:19 +00:00
  • 98007cccd7 Color scheme: green for facts/phrases/scores, blue for paths sora 2026-09-11 03:18:52 +00:00
  • 0ca8aedefe Scoring line: colon restored, full sentence (score over N samples), score keeps solid green with yellow phrase tail; path coloring covers 'to /abs' endings sora 2026-09-11 03:16:08 +00:00
  • d7f06395a0 Highlight whole noun phrases (1319 samples from cache / 4/4 predictions / 0 samples left to run), not bare numbers; score rule runs first so scores keep solid green sora 2026-09-11 03:09:15 +00:00
  • a8c04311c0 Semantic numbers in narration: bold yellow (bold alone was hard to see) sora 2026-09-11 03:02:03 +00:00
  • a1fa75bf89 All narration lines as full sentences (Checkpoint: 4/4 predictions already generated, 0 samples left to run / Generation skipped: ... / Scoring complete: acc 100.0% / Writing results to ...) sora 2026-09-11 02:59:22 +00:00
  • 20e9b87248 Probe success line as a full sentence: model name, endpoint, response time, pool size, and what happens next sora 2026-09-11 02:49:06 +00:00
  • 470496b7ef Probe line carries model/endpoint/instances/probe-latency (ANSI-colored on tty); semantic numbers (counts, checkpoint fractions) bold in narration lines sora 2026-09-11 02:43:10 +00:00
  • b8ff031be7 Scores in narration lines get bold green (the key fact, previously lost in all-white lines); paths stay cyan sora 2026-09-11 02:35:49 +00:00
  • 9eb1f7e58b Fix KeyError 'cur' regression (new-task branch missed the field); live bars hard-disable unless the real stream is a tty (FORCE_COLOR env made rich claim terminal-ness on pipes -> refresh thread stalled the whole run) sora 2026-09-10 12:02:27 +00:00
  • c6046dd02e Progress bar keeps its own elapsed/ETA clock (rich's columns freeze on cross-benchmark task reuse); in-flight shows current sample's elapsed; adaptive read timeout scales with the generation budget (32k-token CoTs were timing out at the fixed 300s and retrying forever -- the 'hang') sora 2026-09-10 11:36:19 +00:00
  • e7792a559c Narration lines get stage icons (⬇ download / 📦 ready / ✳ few-shot / ◷ checkpoint / 🤖 generating / ⏭ skipped / ★ scoring / 📝 writing / 🔗 endpoint) and cyan-highlighted paths sora 2026-09-10 11:29:07 +00:00
  • 3a045e0569 Narration density: preparing+checkpoint merged into one line (N samples · checkpoint k/N restored, m to generate); zero-pending runs say 'generation skipped' instead of the misleading generating-0/complete pair; scoring complete carries the primary metric sora 2026-09-10 11:26:23 +00:00
  • 3911019315 writing-results line shows the destination directory sora 2026-09-10 11:22:04 +00:00
  • 5b595f3a10 Narration lines all plain white (color scheme toggle kept in _phase_color for quick restore) sora 2026-09-10 11:19:58 +00:00
  • 5734f7fbe6 Uniform [i/N] prefix on every narration line (status_callback restores the tag; checkpoint-restored detail routes through status_callback; probe line remains pre-callback) sora 2026-09-10 11:15:04 +00:00
  • 392cc8daca Persistent benchmark counter [i/N] on the single progress bar; phase lines all route through the live channel for strict ordering sora 2026-09-10 11:13:34 +00:00
  • 9818515247 Narration colors by stage semantics: start/in-progress events yellow, completion events green (dataset ready / generation complete / scoring complete / checkpoint restored) sora 2026-09-10 11:09:38 +00:00
  • 89f950d8a0 Narration lines uniformly uncolored (phase prints drop cyan, rich number auto-highlight off); color reserved for results/errors only sora 2026-09-10 11:06:41 +00:00
  • 8632b2735b Drop the overall benches bar from the CLI (per-benchmark result lines already track progress; reporter keeps the capability for API users) sora 2026-09-10 11:04:00 +00:00
  • b7be2a8df3 Progress bars self-describing: overall bar carries the current benchmark (benches · 1/6 humaneval), sample bar carries the stage tag (humaneval · generating/scoring/writing) sora 2026-09-10 11:02:41 +00:00
  • dd7e1933ff HF endpoint auto-fallback: when huggingface.co is unreachable, probe mirrors (hf-mirror.com) and switch automatically with a one-line notice; error only if all fail sora 2026-09-10 10:57:16 +00:00
  • 8ea83fea4f Pause the rich live display while datasets materialize so hub tqdm progress (download/generating splits) is visible instead of being erased; reporter gains pause()/resume() sora 2026-09-10 10:54:04 +00:00
  • c728db5f35 HF hub downloads: reachability probe fails fast with the mirror fix hint (was minutes of silent retries looking like a hang); status lines around hub downloads (start + records + elapsed) sora 2026-09-10 10:51:09 +00:00
  • afe2fb8d28 Progress coverage for minute-scale work: overall bench bar (which benchmark of N, covers loading/scoring), byte-level download bars for dataset fetches (ModelScope/HF raw, tty-only), one shared reporter across benchmarks (caller-owned lifecycle) sora 2026-09-10 10:47:21 +00:00
  • c54207a180 Error panel centered; comma-joined benchmark names get an immediate fix hint sora 2026-09-10 10:43:41 +00:00
  • a3269252fd Closing line simplified to one line (status + result path); tree removed sora 2026-09-10 10:42:04 +00:00
  • 7f3c42d39c Drop summary.md from outputs; reports switch to report.jsonl (header line + one sample per line, round-trip verified) sora 2026-09-10 10:40:00 +00:00
  • 4aa3385345 README: new output directory layout sora 2026-09-10 10:31:58 +00:00
  • f9cb34fc17 Output dir restructure (evalscope/inspect-style): top-level summary.{xlsx,md,csv} + one directory per benchmark (report.json + detail.md); drops the opaque viz/ and reports/ layers sora 2026-09-10 10:31:38 +00:00
  • bd35b5b724 Auto-export consolidated excel workbook (results.xlsx: Summary/Perf/Categories/Samples) on every --out-dir run; per-bench artifact .md instead of .txt; xlsxwriter joins core deps sora 2026-09-10 10:29:13 +00:00
  • 8cd45e4d16 Single-bench runs use the unified Run Summary table (panel behind --verbose); short model names in panel titles; plain-text paths in closing block (OSC8 links unreliable across terminals) sora 2026-09-10 10:25:56 +00:00
  • 1ef230f4c1 Run Plan Samples row: show the actual sampling config instead of the confusing placeholder sora 2026-09-10 10:23:49 +00:00
  • e9c0b5be77 Fail-fast endpoint probe (English, full curl hint, red panel on failure); slim run narration (drop duplicate generating/materializing lines); probe success line sora 2026-09-10 10:21:27 +00:00
  • 044523f97e README: cache-dir default + directory layout tree in cache section sora 2026-09-10 10:09:41 +00:00
  • 3a1bfb9fde Simplify closing block to one clean status+directory tree; drop lat/trunc columns from summary sora 2026-09-10 10:08:21 +00:00
  • ed36fda365 Auto-save results by default (evalharness-results/<stamp>-<model>/); final notice: run finished + saved location with clickable links sora 2026-09-10 10:05:29 +00:00
  • c1896e0a63 Center run plan, result panel, and Run Summary table; add lat p50/p90 + truncation columns sora 2026-09-10 10:03:19 +00:00
  • b4d39c560e Run Summary: tok in/out columns with per-direction throughput (in/s, out/s), totals; clickable file:// links in artifacts notice; drop note column sora 2026-09-10 09:59:02 +00:00
  • 82d9d03b88 Run Summary: tokens/throughput/note columns + totals row; final artifacts notice (where results were saved, or how to save) sora 2026-09-10 09:49:36 +00:00
  • 075e8728cb Score bars: rich-style half-cell glyphs (━━━╸╌╌) replacing block chars sora 2026-09-10 09:30:25 +00:00
  • 5c3d622f41 Rich result panel: metrics+bar with score colors, run stats (wall/model-time/throughput/tokens in-out/tok-s), top+bottom group highlights, perf row sora 2026-09-10 09:20:44 +00:00
  • b6c473ac26 Output polish: 1-based phase indices everywhere, drop [1/1] prefix on single-bench runs, slim progress bar (drop Waiting/Last columns) sora 2026-09-10 09:16:33 +00:00
  • 2dfb60cdf2 Few-shot split loading goes through cached materialization sora 2026-09-10 07:47:48 +00:00
  • 9991c15816 README rewritten in evalscope style (numbered flow, code-first, 289->190 lines); config entry points: --hf-endpoint flag, ckpt follows cache root, EVALHARNESS_DOCKER_MIRRORS override sora 2026-09-10 07:27:19 +00:00
  • 07bc56f423 Package *.txt/*.md data files (simpleqa grader prompt crashed import on fresh installs) sora 2026-09-10 06:54:21 +00:00
  • 5bb4b75b07 Rename --judge to --judge-model (alias kept) for symmetry with --model sora 2026-09-10 06:52:31 +00:00
  • c78b0d6f0f Add --api-key/--judge-api-key: explicit keys override env resolution, never serialized sora 2026-09-10 06:50:18 +00:00
  • ad1cd18f04 Add --judge-provider flag; document key resolution rules (env per host, judge included) sora 2026-09-10 06:45:26 +00:00
  • 08605d9ab2 CLI flag symmetry: --judge-api-url split; mock-boxed hyphen names sora 2026-09-10 06:43:27 +00:00
  • ea93602dfa Unified, nicer result tables + conda env setup in README sora 2026-09-10 06:30:22 +00:00
  • 27cf8b3c7e Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker sora 2026-09-10 05:46:45 +00:00
  • 46bef7d3dd dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes sora 2026-09-08 05:55:04 +00:00
  • bee997423a GPQA 0.15->0.55: (1) per-bench prompt_style cot_letter (es GPQA CoT template, matches its 'Think step by step' contract), (2) mcq_letter extractor case-insensitive answer-tail (ANSWER: vs answer), (3) deterministic choice shuffle sora 2026-08-28 18:01:16 +00:00
  • 60b2bfc0dc GPQA deterministic per-question choice shuffle (es-adapter parity: sha256(question) seed, position-bias protection) — was fixed-A; lb2 subset grouping aligned to es (length field, 60 samples) sora 2026-08-28 16:24:11 +00:00
  • f7c7536400 Fix images_for_samples export (swe prefetch import error) sora 2026-08-27 15:27:30 +00:00
  • da989ea665 Align with local evalscope run.py (full read-through): (1) shuffle+seed42 sample selection identical to es _shuffle_in_place -- same exact sample subset both frameworks; (2) official MATH_PROMPT_TEMPLATE CoT boxed suffix on 7 math benches via DatasetSpec.prompt_suffix; (3) per-bench gen_config (temp/max_tokens from dpv4 yaml) on DatasetSpec, default-merged into gen_kwargs sora 2026-08-27 11:25:17 +00:00
  • 7ce04b8ac6 cmmlu few_shot_num 5->0 (es code default is 0-shot despite docstring claiming 5); 28-bench parity check all green sora 2026-08-27 06:43:09 +00:00
  • e11cae038b Parity pre-flight check (truncation/few-shot/judge/no-think endpoints vs es run.py); bbh few_shot_num=3 fixed (was 0, es/paper default 3-shot CoT) sora 2026-08-27 06:37:57 +00:00
  • e32da10ce9 longbench_v2 chain of fixes: reasoning_content output fallback (Qwen3 thinking channel), nothink soft switch /no_think (template-400 immune), truncation input+output budget, adaptive context shrink on server 400 (cross-tokenizer margin, body-aware detection), pool fails over on 4xx sora 2026-08-27 06:27:59 +00:00
  • c4473a1e1c Token-level middle truncation (ported from evalside run.py: head+tail 128k, mrcr message-window); max_input_tokens param; perf fields (ttft/itl/status/retries) verified persisted in checkpoints sora 2026-08-27 03:23:33 +00:00
  • 3c63338ac1 Excel renderer (xlsxwriter, 4 sheets: Summary dashboard with 21 columns, Perf detail, Categories, Samples drill-down); !perf spec flag; viz --style excel sora 2026-08-27 02:56:07 +00:00
  • e77ea4ce5c !perf spec flag; perf_stats full dashboard columns (success_rate, latency/ttft/tpot mean+P90+P99, tokens mean/total, output TPS, request QPS); summary.csv perf columns aligned sora 2026-08-27 02:49:55 +00:00
  • f2a914d0ff Performance profile: TTFT/ITL via SSE streaming (collect_perf flag), retries/http_status per request, pool success/retry/failure rates, perf_stats aggregator (P50/P95/P99 latency+ttft, tokens, retry_rate) in metric_groups['perf']; verified on fake SSE server sora 2026-08-27 02:45:51 +00:00
  • 456d304a69 Fix numeric normalization in math_equal: strip $, commas, markdown (**), trailing units ('540 meters') — gsm8k 0.45->0.80 on ladder20 cross-check; winogrande cross-scoring 20/20 agreement (delta = generation variance) sora 2026-08-26 17:15:13 +00:00
  • 111336cbee ladder20 complete 28/28 (Qwen3-8B pool): fixes en route — tau2 off-loop executor, double-encoded tool args, shared adapter cache; swe real-model 0.0 (oracle path verified 1.0) sora 2026-08-26 10:37:54 +00:00
  • 0e1e31ba10 swe_bench_verified end-to-end verified (oracle patch -> official sweb image -> resolved); image naming mapped to local swebench/ set; ladder2 complete 28/28 sora 2026-08-26 06:40:55 +00:00