sora
acb94e3e20
humaneval/gpqa_diamond: repeats 3 (align with the es reference runs)
...
syy's es runs: humaneval x3, gpqa x2, aime25/26 x12 (already aligned),
mmlu_pro/longbench_v2 x1 (temp=0 deterministic -- repeats are noise,
and 3x 12k samples is pure cost). temp=1 benches get 3.
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-14 07:39:55 +00:00
sora
e9d7e7f4eb
README overhaul: features overview, config section, perf stats, FAQ
...
- accurate output structure (summary.xlsx/csv + per-bench xlsx; drop
stale detail.md/summary.md mentions)
- dedicated YAML-config section documenting the keys that actually work
(generation params + repeats + max_input_tokens) and precedence
- perf-stats table: what is always collected vs --perf streaming-only
- FAQ: endpoint probe, context-overflow shrink, thinking-mode notes
(non-stream chat_template_kwargs vs cloud-API param), auto-stream
- config/README.md rewritten to match the flat-key reality (old file
documented a directory scheme + judge/env keys that are not consumed)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-14 02:30:38 +00:00
sora
624b2fd37e
aime/hmmt max_tokens 32768→8192: stays on non-stream path where GLM gateway honors enable_thinking:false (thinking off, 15s/题)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 09:43:36 +00:00
sora
7ada0113ad
max_input_tokens back to 128000 (user confirmed same behavior as 120k)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 09:23:43 +00:00
sora
b75fbc1a12
max_input_tokens 128000→120000 (GLM gateway unstable on very long inputs, 30s timeout at 128k)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 09:22:55 +00:00
sora
ebc51de274
Rename dp4-nothink.yaml → default.yaml (auto-loaded)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 09:12:41 +00:00
sora
ebf7a0e7e5
Config: generation params + repeats + max_input_tokens only; judge/env/limit controlled elsewhere
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:47:25 +00:00
sora
60bdef3936
Complete config: all fields included (env, limit, limit_per_task, judge, judge_url, repeats, max_input_tokens, max_turns, concurrency)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:45:35 +00:00
sora
eb6028e8c9
Config simplified: default block + per-bench overrides only (removed env/limit/judge/concurrency; max_input_tokens 128000)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:42:21 +00:00
sora
0815f6e4f8
Config: evalscope generation_config format + our extras (repeats, env) at bench level; parallel_tool_calls noted as unsupported in adapter
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:37:18 +00:00
sora
0ada7d8a0e
Config: exact copy of evalscope dpv4-int8_nothinking.yaml (generation_config + dataset_args + agent_config, zero custom fields)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:35:05 +00:00
sora
b8ed18771a
Config exactly matches evalscope dpv4-int8_nothinking.yaml (verified ✅ all match); concurrency removed (CLI controls it)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:25:15 +00:00
sora
787a8a28e9
Standard YAML indentation, no inline braces
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:09:09 +00:00
sora
4d7567adf8
Flatten config: direct key-value per bench, no nested groups
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:05:36 +00:00
sora
d16009d054
Config: single dp4-nothink.yaml with all 28 bench configs (evalscope-style)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 08:02:43 +00:00
sora
c27a22555d
Config directory: dp4-nothink/ with 28 per-bench YAMLs (aime repeats=12, judged, agent, long-context) + README
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 07:59:42 +00:00
sora
77b67f8bbc
gen_profiles.yaml moved to evalharness/config/ (search path added); /data2/hf_home deleted (8.6G freed)
...
Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-11 07:48:41 +00:00
