Meta-Eval/EvalHarness
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Thread the truncation tokenize: inline encode froze the event loop

Browse Source 
assemble() runs the max_input_tokens truncation tokenizer inline in the
coroutine; longbench_v2's 2M-token docs take seconds of CPU per encode,
and each one BLOCKED the whole loop -- heartbeat frozen, gate probes
dead, zero HTTP while the process sat at 100% single-core. encode now
runs in a worker thread (loop stays live, encodes parallelize).

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
sora 2026-09-15 02:26:15 +00:00
parent f8ff19d4a5
commit 3be48addfc
1 changed files with 9 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
evalharness/model/runner.py
View File

@ -317,7 +317,15 @@ async def generate_predictions(
_progress(progress, done_count, len(samples), t0, total_usage) _progress(progress, done_count, len(samples), t0, total_usage)
return pred return pred
messages = ([ChatMessage(role='user', content=assemble(sample))] # assemble() tokenizes for the max_input_tokens truncation -- on
# long-context benches that is SECONDS of CPU per sample (2M-token
# docs), and running it inline FROZE the whole event loop: heartbeat,
# gate probes and every other request serialized behind one encode.
# Thread it: the loop stays live and encodes parallelize (the Rust
# fast tokenizer releases the GIL).
text = await asyncio.to_thread(assemble, sample) \
if isinstance(sample.input, str) else None
messages = ([ChatMessage(role='user', content=text)]
if isinstance(sample.input, str) else list(sample.input)) if isinstance(sample.input, str) else list(sample.input))
if not system and extra_system[0] and isinstance(sample.input, str): if not system and extra_system[0] and isinstance(sample.input, str):
# renderer-provided SYSTEM contract (es lcb expert-programmer) # renderer-provided SYSTEM contract (es lcb expert-programmer)