sora
a07431b324
Fix the ellipsis crash: text-tool-call parsing ran on plain code replies
...
Full traceback finally caught it: adapter._parse ALWAYS ran the
text-protocol tool-call fallback, even for requests with NO tools. On
humaneval, model code like regex-matched as a
'call', ast.literal_eval turned the literal into an Ellipsis
(no exception -- it's a legal literal), and json.dumps(args) died
mid-generation, killing the benchmark.
Two layers:
- the fallback now only runs when the request actually carried tools
(also stops polluting plain predictions with phantom calls, and the
SyntaxWarning spam from ast.parse-ing model code disappears)
- json.dumps(args, default=str) as belt-and-braces for the
text-tools path where an Ellipsis arg now stringifies
Reproduced the exact crash input as a unit case: no-tools code reply
yields 0 tool_calls; in text mode serializes
{'key': 'Ellipsis'} without raising; normal fc calls unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-14 09:43:24 +00:00
sora
ed675e790f
Auto-stream threshold 8192→100000 (streaming strips all thinking params on GLM gateway; non-streaming works for 32k); EVALHARNESS_NO_AUTOSTREAM=1 to disable entirely
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 09:50:03 +00:00
sora
ff5d40aad5
Read timeout 300→600s base (GLM gateway 30+s startup on 100k+ token inputs)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 09:24:16 +00:00
sora
55b2beab54
Retry chain sanity: adapter 3 retries for long budgets / 6 for normal (was 6 flat); run_one 3 retries (was 6) with shorter backoff; worst case now 12 attempts ~5min (was 42+ attempts ~20min); progress bar shows live retry count; heartbeat recreated on every reset_samples
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 07:06:11 +00:00
sora
d00f330430
Fix frozen progress clock: heartbeat recreated on every reset_samples (previous one died during pause/resume cycles); auto-stream path appends /no_think when gateway drops chat_template_kwargs on streaming
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 06:39:55 +00:00
sora
7473170784
Auto-stream long generations (max_tokens>8192) and aggregate to the non-stream response shape: gateways hang on large buffered NON-streaming requests (aime25's 32k budget stalled forever); streaming starts emitting immediately so a stuck endpoint surfaces in ~60s instead of the full adaptive read timeout; lazy httpx import fix
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 03:41:24 +00:00
sora
c6046dd02e
Progress bar keeps its own elapsed/ETA clock (rich's columns freeze on cross-benchmark task reuse); in-flight shows current sample's elapsed; adaptive read timeout scales with the generation budget (32k-token CoTs were timing out at the fixed 300s and retrying forever -- the 'hang')
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-10 11:36:19 +00:00
sora
46bef7d3dd
dp4-flash 28-bench alignment: renderer plugin layer, exec_workers, gen_profiles, AIMD pool, BCB/LCB/bfcl/gfc judge fixes
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-08 05:55:04 +00:00
e32da10ce9
longbench_v2 chain of fixes: reasoning_content output fallback (Qwen3 thinking channel), nothink soft switch /no_think (template-400 immune), truncation input+output budget, adaptive context shrink on server 400 (cross-tokenizer margin, body-aware detection), pool fails over on 4xx
2026-08-27 06:27:59 +00:00
f2a914d0ff
Performance profile: TTFT/ITL via SSE streaming (collect_perf flag), retries/http_status per request, pool success/retry/failure rates, perf_stats aggregator (P50/P95/P99 latency+ttft, tokens, retry_rate) in metric_groups['perf']; verified on fake SSE server
2026-08-27 02:45:51 +00:00
111336cbee
ladder20 complete 28/28 (Qwen3-8B pool): fixes en route — tau2 off-loop executor, double-encoded tool args, shared adapter cache; swe real-model 0.0 (oracle path verified 1.0)
2026-08-26 10:37:54 +00:00
3d16e9647a
Tool-call parsing: Qwen3 native <tool_call> XML fallback in _parse (parser-agnostic, zero overhead when server converts properly)
2026-08-26 06:29:49 +00:00
7114564301
Multi-endpoint model pool (openai-pool/{lo..hi} round-robin with failover), nothink/textools spec flags (chat_template_kwargs enable_thinking=false; text-protocol tool calls for backends without --enable-auto-tool-choice), global cross-bench queue in ladder runs
2026-08-26 06:26:52 +00:00
78459c974e
tau2 via OFFICIAL engine: self-running env plugin (run_task hook + needs_adapter generic dispatch, no name hardcoding), deep generate() patch, official ToolCall shape, data plugin keeps verbatim Task json, official reward scoring; checkpoint plugin (per-sample resume) + summary csv with time/categories
2026-08-25 11:06:08 +00:00
d830e08923
Drop logprob scoring path entirely; MCQ = strict-letter generation (evalscope parity 'ANSWER: X'), few-shot defaults stay; fix output.py duplication
2026-08-25 02:26:26 +00:00
a8d3400ed5
Paper-faithful eval: P1 logprob scoring (completions_echo + chat_first_token fallback, sglang top_logprobs parser), few_shot_num dataset defaults (mmlu5/cmmlu5/mmlu_pro5/gsm8k4/math4/drop3), official BBH 3-shot CoT vendored, strict_letter default prompt (evalscope parity), official simpleqa grader prompt
2026-08-25 02:21:59 +00:00
a32d902e94
Comparison-driven fixes: MCQ choices in prompt + letter contract, Answer: suffix for QA, markdown answer cleaning, trivia_qa answer_phrase priority, DROP gold-as-alternatives (OR) official semantics, retry on 5xx, numpy/scipy compat
2026-08-24 15:48:42 +00:00
3d16ab9103
Add SWE-bench single-turn harness (patch apply + FAIL_TO_PASS in official sweb image, sh entries in docker sandbox), bigcodebench official all-libs image, docker default for humaneval/LCB, input truncation (max_input_chars) + context assembly (passage/context), API retry with backoff, judge thread-safe bridge, parquet blob integrity check, summary.csv in out-dir, sandbox prefetch CLI
2026-08-24 08:43:49 +00:00
b2e7133b20
Add sandbox layer (docker exec hard-isolation + serve envs, refcounted acquire/release, atexit teardown, bind-mount sharing) and agent evaluation driver (message pump drive(), Trajectory, bfcl_mock env with official call-sequence scoring, mock:fc oracle); Deployer delegates to sandbox.serve_env; code_any extractor fixes indentation-stripping; CLI --env
2026-08-24 07:11:03 +00:00
6b3bb330c7
Add model layer: async ModelAdapter (openai_compatible + mock) returning structured ModelOutput(text,tool_calls,usage), Deployer registry (vllm/sglang docker-pinned via models.yaml, external), async run_eval generate->score, CLI --model, agent-ready SampleResult.trajectory/env_state, tests
2026-08-24 06:28:00 +00:00
