Block a user
0ada7d8a0e
Config: exact copy of evalscope dpv4-int8_nothinking.yaml (generation_config + dataset_args + agent_config, zero custom fields)
3a4be917e6
Restored checkpoint predictions' usage counted in totals (was 0 for fully-restored benches); token stats now correct on resume
b8ed18771a
Config exactly matches evalscope dpv4-int8_nothinking.yaml (verified ✅ all match); concurrency removed (CLI controls it)
c27a22555d
Config directory: dp4-nothink/ with 28 per-bench YAMLs (aime repeats=12, judged, agent, long-context) + README
77b67f8bbc
gen_profiles.yaml moved to evalharness/config/ (search path added); /data2/hf_home deleted (8.6G freed)
471e0ca1a9
gen_profiles.yaml: YAML profiles (dp4-nothink, qwen3-es-parity, glm53-nothink, t1-short) — edit file to tune parameters, no code changes needed
63ffd1b613
Fix NameError _fmt (module-level helper restored); aime/hmmt max_tokens 32768->8192 (non-stream path, gateway honors nothink)
55b2beab54
Retry chain sanity: adapter 3 retries for long budgets / 6 for normal (was 6 flat); run_one 3 retries (was 6) with shorter backoff; worst case now 12 attempts ~5min (was 42+ attempts ~20min); progress bar shows live retry count; heartbeat recreated on every reset_samples
d00f330430
Fix frozen progress clock: heartbeat recreated on every reset_samples (previous one died during pause/resume cycles); auto-stream path appends /no_think when gateway drops chat_template_kwargs on streaming
8bac3483b6
Fixes: theme markup closing tags ([bold green]...[/bold green]); evaluate() runs off the event loop (sympy equivalence on hard math froze the progress clock for the whole bench)
sora
transferred repository 2026-09-11 03:46:40 +00:00
meta-eval/EvalHarness to Meta-Eval/EvalHarness
7473170784
Auto-stream long generations (max_tokens>8192) and aggregate to the non-stream response shape: gateways hang on large buffered NON-streaming requests (aime25's 32k budget stalled forever); streaming starts emitting immediately so a stuck endpoint surfaces in ~60s instead of the full adaptive read timeout; lazy httpx import fix
5af8e2a7fd
Pluginize the run shell: @register_progress (rich/plain), @register_theme, @register_hook (on_benchmark_failed/done), @register_prober; CLI gains --progress-plugin/--theme
0ca8aedefe
Scoring line: colon restored, full sentence (score over N samples), score keeps solid green with yellow phrase tail; path coloring covers 'to /abs' endings