sora
|
f5aaddb870
|
Progress bar: colored fields (cur=yellow, elapsed=green, eta=cyan, retries=red, rate=dim)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-11 09:45:01 +00:00 |
|
sora
|
63ffd1b613
|
Fix NameError _fmt (module-level helper restored); aime/hmmt max_tokens 32768->8192 (non-stream path, gateway honors nothink)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-11 07:20:02 +00:00 |
|
sora
|
55b2beab54
|
Retry chain sanity: adapter 3 retries for long budgets / 6 for normal (was 6 flat); run_one 3 retries (was 6) with shorter backoff; worst case now 12 attempts ~5min (was 42+ attempts ~20min); progress bar shows live retry count; heartbeat recreated on every reset_samples
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-11 07:06:11 +00:00 |
|
sora
|
d00f330430
|
Fix frozen progress clock: heartbeat recreated on every reset_samples (previous one died during pause/resume cycles); auto-stream path appends /no_think when gateway drops chat_template_kwargs on streaming
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-11 06:39:55 +00:00 |
|
sora
|
5af8e2a7fd
|
Pluginize the run shell: @register_progress (rich/plain), @register_theme, @register_hook (on_benchmark_failed/done), @register_prober; CLI gains --progress-plugin/--theme
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-11 03:26:19 +00:00 |
|
sora
|
9eb1f7e58b
|
Fix KeyError 'cur' regression (new-task branch missed the field); live bars hard-disable unless the real stream is a tty (FORCE_COLOR env made rich claim terminal-ness on pipes -> refresh thread stalled the whole run)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 12:02:27 +00:00 |
|
sora
|
c6046dd02e
|
Progress bar keeps its own elapsed/ETA clock (rich's columns freeze on cross-benchmark task reuse); in-flight shows current sample's elapsed; adaptive read timeout scales with the generation budget (32k-token CoTs were timing out at the fixed 300s and retrying forever -- the 'hang')
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 11:36:19 +00:00 |
|
sora
|
392cc8daca
|
Persistent benchmark counter [i/N] on the single progress bar; phase lines all route through the live channel for strict ordering
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 11:13:34 +00:00 |
|
sora
|
89f950d8a0
|
Narration lines uniformly uncolored (phase prints drop cyan, rich number auto-highlight off); color reserved for results/errors only
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 11:06:41 +00:00 |
|
sora
|
b7be2a8df3
|
Progress bars self-describing: overall bar carries the current benchmark (benches · 1/6 humaneval), sample bar carries the stage tag (humaneval · generating/scoring/writing)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 11:02:41 +00:00 |
|
sora
|
8ea83fea4f
|
Pause the rich live display while datasets materialize so hub tqdm progress (download/generating splits) is visible instead of being erased; reporter gains pause()/resume()
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 10:54:04 +00:00 |
|
sora
|
afe2fb8d28
|
Progress coverage for minute-scale work: overall bench bar (which benchmark of N, covers loading/scoring), byte-level download bars for dataset fetches (ModelScope/HF raw, tty-only), one shared reporter across benchmarks (caller-owned lifecycle)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 10:47:21 +00:00 |
|
sora
|
b6c473ac26
|
Output polish: 1-based phase indices everywhere, drop [1/1] prefix on single-bench runs, slim progress bar (drop Waiting/Last columns)
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 09:16:33 +00:00 |
|
sora
|
27cf8b3c7e
|
Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker
- progress/: Rich per-sample terminal progress plugin (Run Plan panel,
in-flight/rate/ETA bar); shared console + log-through-live to avoid
interleaved writes, rollback() pairs begin_sample on the retry path,
begin moved inside the semaphore (in-flight = actually generating),
graceful degradation when rich is absent
- cli.py: --provider/--api-url/--model composition (openai-chat |
openai-pool), --disable-thinking/--perf/--textools as first-class
flags, per-bench phase lines and done/failed result lines
- __init__: top-level run()/arun() entries (event-loop safe for notebooks)
- third_party/bfcl: vendored official BFCL ast_checker + type mappings
(Apache-2.0, provenance in __init__.py); imports rerouted locally,
underscore_to_dot parameterized; verified bit-identical with the
bfcl-eval package on 100 real rows -- removes the heavy extra
(pinned numpy + cloud SDK wall) from the install path
- runner: progress/status hooks through generate+evaluate, checkpoint
key scheme fix (empty-store falsy bug), tiered retry backoff,
multi-segment pool {range} expansion fix, adapter-instance passthrough
- pyproject: tree_sitter family joins core deps; [bfcl] extra retired
- README: rewritten (zh) -- install/quickstart/flags reference/bench
table/reliability/extension/architecture/validation
Co-Authored-By: Claude <noreply@anthropic.com>
|
2026-09-10 05:46:45 +00:00 |
|