One sample whose stream never got a first byte (prefill queue at high
gate levels) exhausted 6 adapter retries and killed the WHOLE gather --
500 samples died with it. Terminal failures are now contained: empty
prediction (scores wrong, es-parity for timeouts), NOT checkpointed so
a rerun retries them, prominently counted; only a 100% wipeout fails
the bench. Stream-aggregate read timeout 60s -> 240s: a 128k prompt
queued behind other prefills legitimately takes minutes to start
answering; real hangs are now the gate's job (x0.7) and contained
failures rather than bench death.
Verified: 1-in-3 terminal failures -> bench completes 6 ok / 3 empty,
checkpoint holds only the 6.
Co-Authored-By: Claude <noreply@anthropic.com>
The auto-stream trigger only looked at max_tokens (output budget);
longbench_v2 sends ~128k-token INPUTS with a small 8k output budget and
hit the gateway's whole-request buffering: 20-minute read timeouts
through all 6 retries. es survives the same bench because it streams
by default. Payloads over ~300k chars (~90k+ tokens) now take the
stream-aggregate path; verified live: a 350k-char prompt returns in
20s instead of hanging.
Co-Authored-By: Claude <noreply@anthropic.com>
Full traceback finally caught it: adapter._parse ALWAYS ran the
text-protocol tool-call fallback, even for requests with NO tools. On
humaneval, model code like regex-matched as a
'call', ast.literal_eval turned the literal into an Ellipsis
(no exception -- it's a legal literal), and json.dumps(args) died
mid-generation, killing the benchmark.
Two layers:
- the fallback now only runs when the request actually carried tools
(also stops polluting plain predictions with phantom calls, and the
SyntaxWarning spam from ast.parse-ing model code disappears)
- json.dumps(args, default=str) as belt-and-braces for the
text-tools path where an Ellipsis arg now stringifies
Reproduced the exact crash input as a unit case: no-tools code reply
yields 0 tool_calls; in text mode serializes
{'key': 'Ellipsis'} without raising; normal fc calls unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
