Block a user
bee997423a
GPQA 0.15->0.55: (1) per-bench prompt_style cot_letter (es GPQA CoT template, matches its 'Think step by step' contract), (2) mcq_letter extractor case-insensitive answer-tail (ANSWER: vs answer), (3) deterministic choice shuffle
60b2bfc0dc
GPQA deterministic per-question choice shuffle (es-adapter parity: sha256(question) seed, position-bias protection) — was fixed-A; lb2 subset grouping aligned to es (length field, 60 samples)
da989ea665
Align with local evalscope run.py (full read-through): (1) shuffle+seed42 sample selection identical to es _shuffle_in_place -- same exact sample subset both frameworks; (2) official MATH_PROMPT_TEMPLATE CoT boxed suffix on 7 math benches via DatasetSpec.prompt_suffix; (3) per-bench gen_config (temp/max_tokens from dpv4 yaml) on DatasetSpec, default-merged into gen_kwargs
7ce04b8ac6
cmmlu few_shot_num 5->0 (es code default is 0-shot despite docstring claiming 5); 28-bench parity check all green
e11cae038b
Parity pre-flight check (truncation/few-shot/judge/no-think endpoints vs es run.py); bbh few_shot_num=3 fixed (was 0, es/paper default 3-shot CoT)
e32da10ce9
longbench_v2 chain of fixes: reasoning_content output fallback (Qwen3 thinking channel), nothink soft switch /no_think (template-400 immune), truncation input+output budget, adaptive context shrink on server 400 (cross-tokenizer margin, body-aware detection), pool fails over on 4xx
c4473a1e1c
Token-level middle truncation (ported from evalside run.py: head+tail 128k, mrcr message-window); max_input_tokens param; perf fields (ttft/itl/status/retries) verified persisted in checkpoints
3c63338ac1
Excel renderer (xlsxwriter, 4 sheets: Summary dashboard with 21 columns, Perf detail, Categories, Samples drill-down); !perf spec flag; viz --style excel
e77ea4ce5c
!perf spec flag; perf_stats full dashboard columns (success_rate, latency/ttft/tpot mean+P90+P99, tokens mean/total, output TPS, request QPS); summary.csv perf columns aligned
f2a914d0ff
Performance profile: TTFT/ITL via SSE streaming (collect_perf flag), retries/http_status per request, pool success/retry/failure rates, perf_stats aggregator (P50/P95/P99 latency+ttft, tokens, retry_rate) in metric_groups['perf']; verified on fake SSE server
456d304a69
Fix numeric normalization in math_equal: strip $, commas, markdown (**), trailing units ('540 meters') — gsm8k 0.45->0.80 on ladder20 cross-check; winogrande cross-scoring 20/20 agreement (delta = generation variance)
111336cbee
ladder20 complete 28/28 (Qwen3-8B pool): fixes en route — tau2 off-loop executor, double-encoded tool args, shared adapter cache; swe real-model 0.0 (oracle path verified 1.0)
0e1e31ba10
swe_bench_verified end-to-end verified (oracle patch -> official sweb image -> resolved); image naming mapped to local swebench/ set; ladder2 complete 28/28
3d16e9647a
Tool-call parsing: Qwen3 native <tool_call> XML fallback in _parse (parser-agnostic, zero overhead when server converts properly)
7114564301
Multi-endpoint model pool (openai-pool/{lo..hi} round-robin with failover), nothink/textools spec flags (chat_template_kwargs enable_thinking=false; text-protocol tool calls for backends without --enable-auto-tool-choice), global cross-bench queue in ladder runs
f3514d7c08
Fix checkpoint cross-contamination (scope by dataset name from run_eval, not inferred); ladder20 comparison artifacts