docs(myread.md): document --folder-name, thinking scale, official suite
- Add --folder-name and --thinking-max-tokens-scale to parameter table. - Add thinking mode examples. - Add official suite to suite details. - Update output directory structure to reflect new folder-based layout.
This commit is contained in:
sora
parent
e26fa0fe44
commit
a8c6fbdc75
50
myread.md
50
myread.md
@ -364,6 +364,7 @@ python bash/run.py --model MyModel --api-url http://10.0.0.5:30000/v1
|
||||
| `--api-url` | `http://localhost:30000/v1` | 模型服务地址 |
|
||||
| `--dataset-dir` | `/data1/sora/evalscope` | 数据集父目录，evalscope 会自动找其下的 `datasets/` |
|
||||
| `--output-dir` | `/data1/sora/evalscope/output` | 评测输出根目录 |
|
||||
| `--folder-name` | `--model`（thinking 时为 `{model}_THINKING`） | 顶层输出文件夹名，汇总表也用它命名 |
|
||||
| `--config` | `config/dpv4-int8_nothinking.yaml` | 各 benchmark 的 `max_tokens` / `temperature` 等配置 |
|
||||
| `--tokenizer-path` | `/data1/models/DeepSeek-V4-Flash-INT8` | 本地 tokenizer 路径，用于长文本 middle-truncation |
|
||||
| `--suite` | `full` | 预置套件：`full` / `lite` / `mid` / `group1` / `group2` / `group3` |
|
||||
@ -374,6 +375,7 @@ python bash/run.py --model MyModel --api-url http://10.0.0.5:30000/v1
|
||||
| `--batch-size` | `4` | 评测并发 |
|
||||
| `--thinking` | `False` | 开启 sglang thinking 模式 |
|
||||
| `--no-thinking` | `False` | 关闭 thinking 模式（默认） |
|
||||
| `--thinking-max-tokens-scale` | `1.0` | `--thinking` 开启时，max_tokens 乘以该系数 |
|
||||
| `--judge-model` | `DeepSeek/DeepSeek-V4-Pro` | 裁判模型名（LLM-as-judge） |
|
||||
| `--judge-api-url` | Vectron 地址 | 裁判模型 API 地址 |
|
||||
| `--judge-api-key` | 内置 key | 裁判模型 API key |
|
||||
@ -381,7 +383,17 @@ python bash/run.py --model MyModel --api-url http://10.0.0.5:30000/v1
|
||||
| `--truncation-tokens` | `131072` | 长文本 middle-truncation token 上限 |
|
||||
| `--no-summary` | `False` | 禁止每次 benchmark 后自动写汇总表 |
|
||||
|
||||
> 汇总表文件名由模型名自动派生，例如 `--model DeepSeek-V4-Flash-Int8` → `<safe_model_name>.csv/.xlsx`。
|
||||
> 汇总表文件名由 `--folder-name` 决定，默认 `results/{folder_name}.csv/.xlsx`；thinking 模式下默认 `{model_name}_THINKING`。
|
||||
|
||||
**thinking 模式示例：**
|
||||
|
||||
```bash
|
||||
# 开启 thinking，max_tokens 自动翻倍
|
||||
python bash/run.py --suite full --thinking --thinking-max-tokens-scale 2.0 --limit none
|
||||
|
||||
# 自定义输出文件夹名
|
||||
python bash/run.py --suite full --thinking --folder-name my_exp_v1 --limit none
|
||||
```
|
||||
|
||||
### 4.3 套件详情
|
||||
|
||||
@ -393,6 +405,7 @@ python bash/run.py --model MyModel --api-url http://10.0.0.5:30000/v1
|
||||
| `group1` | 代码 + 数学/推理 | 约 **22h** | 多机组 1 |
|
||||
| `group2` | 重知识 | 约 **26h** | 多机组 2 |
|
||||
| `group3` | 长文本 + Agent + 知识 | 约 **27h** | 多机组 3 |
|
||||
| `official` | 官方对比表全部 benchmark | 视配置而定 | 覆盖 Kimi/GLM/DS 等公开对比表里的所有任务 |
|
||||
|
||||
> **分组时间说明**
|
||||
>
|
||||
@ -543,31 +556,36 @@ python bash/collect_results.py \
|
||||
|
||||
```text
|
||||
<output-dir>/
|
||||
├── <benchmark_name>/
|
||||
│ ├── seed_<seed>/
|
||||
│ │ ├── reports/<model_name>/<benchmark_name>.json
|
||||
│ │ ├── predictions/<model_name>/<benchmark_name>.jsonl
|
||||
│ │ └── logs/eval_log.log
|
||||
│ └── seed_<seed>_run_<idx>/ # multi-run 额外目录
|
||||
├── perf_stats_backup/ # 累计 summary 持久化（断点恢复）
|
||||
│ └── <benchmark>__<model>.json
|
||||
├── predictions_archive/ # 原始每条样本归档（断点恢复）
|
||||
│ └── <benchmark>__<model>.jsonl
|
||||
└── <safe_model_name>/ # 汇总表（按模型名）
|
||||
├── <safe_model_name>.xlsx
|
||||
└── <safe_model_name>.csv
|
||||
└── <folder_name>/ # 默认 --model；thinking 时默认 {model}_THINKING
|
||||
├── <benchmark_name>/
|
||||
│ ├── seed_<seed>/
|
||||
│ │ ├── reports/<benchmark_name>.json
|
||||
│ │ ├── predictions/<benchmark_name>_<subset>.jsonl
|
||||
│ │ ├── reviews/<benchmark_name>_<subset>.jsonl
|
||||
│ │ └── logs/eval_log.log
|
||||
│ └── seed_<seed>_run_<idx>/ # multi-run 额外目录
|
||||
├── perf_stats_backup/ # 累计 summary 持久化（断点恢复）
|
||||
│ └── <benchmark>__<model>.json
|
||||
├── predictions_archive/ # 原始每条样本归档（断点恢复）
|
||||
│ └── <benchmark>__<model>.jsonl
|
||||
└── active_time/ # 各 benchmark 实际运行时间累计
|
||||
└── <benchmark>__<model>.json
|
||||
|
||||
results/ # 项目级汇总表目录
|
||||
└── <folder_name>.xlsx
|
||||
└── <folder_name>.csv
|
||||
```
|
||||
|
||||
### 查看分数
|
||||
|
||||
```bash
|
||||
# 单个 benchmark
|
||||
cat output/aime24/seed_42/reports/DeepSeek-V4-Flash-Int8/aime24.json
|
||||
cat output/DeepSeek-V4-Flash-Int8/aime24/seed_42/reports/aime24.json
|
||||
|
||||
# 汇总所有分数
|
||||
python3 -c "
|
||||
import json, glob
|
||||
for f in sorted(glob.glob('output/*/seed_*/reports/*/*.json')):
|
||||
for f in sorted(glob.glob('output/*/*/seed_*/reports/*.json')):
|
||||
data = json.load(open(f))
|
||||
score = data.get('score', data.get('mean_acc', 'N/A'))
|
||||
print(f'{f}: {score}')
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user