sora
e1a96f18c0
Scoring phase retargets the same progress bar
...
The bar kept its stale 100% generation state during scoring with only
the description counter moving. Now set_scoring() refills the bar with
judged samples (0->100%, fresh clock/rate/eta for the phase), clears
the generation '+N new' marker, and the next bench retargets back to
generating cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-14 03:51:25 +00:00
sora
f5c2e4c5af
Progress bar: fresh-this-run counter; fix KeyError 'success'
...
- Completed X/Y now carries (+N new): samples generated by THIS run,
excluding the checkpoint-restored head start (previously 98/164 told
you nothing about how much work this invocation actually did)
- rate/eta computed over fresh samples only (restored ones counted
toward 141/6s = 23/s when 1 sample had been generated)
- fix KeyError 'success': advance() still read the success/failed task
fields after they were dropped from the column set -- first advance
on a resumed bench killed the whole benchmark
- set_overall: give the overall task the fields the shared columns
actually read (cur/retries/elapsed/eta); the old field set was from
a previous column layout
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-14 03:05:18 +00:00
sora
f5aaddb870
Progress bar: colored fields (cur=yellow, elapsed=green, eta=cyan, retries=red, rate=dim)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 09:45:01 +00:00
sora
63ffd1b613
Fix NameError _fmt (module-level helper restored); aime/hmmt max_tokens 32768->8192 (non-stream path, gateway honors nothink)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 07:20:02 +00:00
sora
55b2beab54
Retry chain sanity: adapter 3 retries for long budgets / 6 for normal (was 6 flat); run_one 3 retries (was 6) with shorter backoff; worst case now 12 attempts ~5min (was 42+ attempts ~20min); progress bar shows live retry count; heartbeat recreated on every reset_samples
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 07:06:11 +00:00
sora
d00f330430
Fix frozen progress clock: heartbeat recreated on every reset_samples (previous one died during pause/resume cycles); auto-stream path appends /no_think when gateway drops chat_template_kwargs on streaming
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 06:39:55 +00:00
sora
9eb1f7e58b
Fix KeyError 'cur' regression (new-task branch missed the field); live bars hard-disable unless the real stream is a tty (FORCE_COLOR env made rich claim terminal-ness on pipes -> refresh thread stalled the whole run)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 12:02:27 +00:00
sora
c6046dd02e
Progress bar keeps its own elapsed/ETA clock (rich's columns freeze on cross-benchmark task reuse); in-flight shows current sample's elapsed; adaptive read timeout scales with the generation budget (32k-token CoTs were timing out at the fixed 300s and retrying forever -- the 'hang')
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 11:36:19 +00:00
sora
392cc8daca
Persistent benchmark counter [i/N] on the single progress bar; phase lines all route through the live channel for strict ordering
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 11:13:34 +00:00
sora
89f950d8a0
Narration lines uniformly uncolored (phase prints drop cyan, rich number auto-highlight off); color reserved for results/errors only
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 11:06:41 +00:00
sora
b7be2a8df3
Progress bars self-describing: overall bar carries the current benchmark (benches · 1/6 humaneval), sample bar carries the stage tag (humaneval · generating/scoring/writing)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 11:02:41 +00:00
sora
8ea83fea4f
Pause the rich live display while datasets materialize so hub tqdm progress (download/generating splits) is visible instead of being erased; reporter gains pause()/resume()
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 10:54:04 +00:00
sora
afe2fb8d28
Progress coverage for minute-scale work: overall bench bar (which benchmark of N, covers loading/scoring), byte-level download bars for dataset fetches (ModelScope/HF raw, tty-only), one shared reporter across benchmarks (caller-owned lifecycle)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 10:47:21 +00:00
sora
b6c473ac26
Output polish: 1-based phase indices everywhere, drop [1/1] prefix on single-bench runs, slim progress bar (drop Waiting/Last columns)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 09:16:33 +00:00
sora
27cf8b3c7e
Usability round: progress plugin, CLI provider flags, top-level run(), vendored BFCL checker
...
- progress/: Rich per-sample terminal progress plugin (Run Plan panel,
in-flight/rate/ETA bar); shared console + log-through-live to avoid
interleaved writes, rollback() pairs begin_sample on the retry path,
begin moved inside the semaphore (in-flight = actually generating),
graceful degradation when rich is absent
- cli.py: --provider/--api-url/--model composition (openai-chat |
openai-pool), --disable-thinking/--perf/--textools as first-class
flags, per-bench phase lines and done/failed result lines
- __init__: top-level run()/arun() entries (event-loop safe for notebooks)
- third_party/bfcl: vendored official BFCL ast_checker + type mappings
(Apache-2.0, provenance in __init__.py); imports rerouted locally,
underscore_to_dot parameterized; verified bit-identical with the
bfcl-eval package on 100 real rows -- removes the heavy extra
(pinned numpy + cloud SDK wall) from the install path
- runner: progress/status hooks through generate+evaluate, checkpoint
key scheme fix (empty-store falsy bug), tiered retry backoff,
multi-segment pool {range} expansion fix, adapter-instance passthrough
- pyproject: tree_sitter family joins core deps; [bfcl] extra retired
- README: rewritten (zh) -- install/quickstart/flags reference/bench
table/reliability/extension/architecture/validation
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 05:46:45 +00:00
