docs(datasets): add README with ShareGPT download instructions
The datasets/ directory is gitignored (large, re-downloadable), but later agents need to know how to obtain the dataset when it is missing. Add a tracked README that documents the download steps. - datasets/README.md: documents the ShareGPT_V3_unfiltered_cleaned_split.json source (HF anon8231489123/ShareGPT_Vicuna_unfiltered), the hf-mirror.com download command for CN environments, the JSON validity check (94145 entries), and the container path consistency note. - .gitignore: change "datasets/" to "datasets/*" so the "!datasets/README.md" negation actually works (git cannot re-include a file under a fully-ignored directory). The dataset .json stays ignored; only README.md is tracked.
This commit is contained in:
parent
d6e00d61dc
commit
cd7b2d4a62
6
.gitignore
vendored
6
.gitignore
vendored
@ -27,8 +27,10 @@ __pycache__/
|
|||||||
logs/
|
logs/
|
||||||
*.log
|
*.log
|
||||||
*.csv
|
*.csv
|
||||||
# 数据集(可重新下载)
|
# 数据集(可重新下载,但保留下载说明文档入库)
|
||||||
datasets/
|
# 用 datasets/* 而非 datasets/,否则 !datasets/README.md 例外无法生效
|
||||||
|
datasets/*
|
||||||
|
!datasets/README.md
|
||||||
|
|
||||||
# 原始请求级输出与中间产物不入库:只提交 adaptive_results 下的汇总级文件
|
# 原始请求级输出与中间产物不入库:只提交 adaptive_results 下的汇总级文件
|
||||||
# (adaptive_points.jsonl / adaptive_shapes.jsonl / adaptive_summary.* / run_manifest.json / shapes.tsv),
|
# (adaptive_points.jsonl / adaptive_shapes.jsonl / adaptive_summary.* / run_manifest.json / shapes.tsv),
|
||||||
|
|||||||
65
datasets/README.md
Normal file
65
datasets/README.md
Normal file
@ -0,0 +1,65 @@
|
|||||||
|
# datasets/
|
||||||
|
|
||||||
|
本目录存放基准测试用的数据集文件。数据集本身**不入库**(体积大、可重新下载),
|
||||||
|
本说明文档是仓库中此目录唯一的跟踪文件。
|
||||||
|
|
||||||
|
如果运行实验时遇到 `FileNotFoundError` 指向本目录下的某个数据集,按下方说明下载。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ShareGPT_V3_unfiltered_cleaned_split.json
|
||||||
|
|
||||||
|
`sglang.bench_serving --dataset-name random` 在 `random`(非 `random-ids`)模式下,
|
||||||
|
会用这个 ShareGPT 文件作为词表采样的种子语料。文件缺失时 bench_serving 会报错。
|
||||||
|
|
||||||
|
### 来源
|
||||||
|
|
||||||
|
HuggingFace 数据集 `anon8231489123/ShareGPT_Vicuna_unfiltered` 的清洗切分版:
|
||||||
|
|
||||||
|
```
|
||||||
|
https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
|
||||||
|
```
|
||||||
|
|
||||||
|
文件约 **642 MB**,94145 条多轮对话,JSON 数组格式。
|
||||||
|
|
||||||
|
### 下载方法(国内环境用 hf-mirror 镜像)
|
||||||
|
|
||||||
|
HuggingFace 直连在国内通常不通,改用 `hf-mirror.com` 镜像:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 在仓库根目录下执行
|
||||||
|
mkdir -p datasets
|
||||||
|
curl -L -o datasets/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||||
|
"https://hf-mirror.com/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
校验:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -c "import json; d=json.load(open('datasets/ShareGPT_V3_unfiltered_cleaned_split.json')); print('OK, 条目数:', len(d))"
|
||||||
|
# 期望输出: OK, 条目数: 94145
|
||||||
|
```
|
||||||
|
|
||||||
|
### 下载方法(能直连 HuggingFace 时)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -L -o datasets/ShareGPT_V3_unfiltered_cleaned_split.json \
|
||||||
|
"https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 容器内路径一致性
|
||||||
|
|
||||||
|
实验脚本(`start_vllm_docker.sh`)已挂载 `/mnt:/mnt`,而本仓库位于
|
||||||
|
`/mnt/yy/sskj`,因此容器内数据集路径与宿主机一致,均为:
|
||||||
|
```
|
||||||
|
/mnt/yy/sskj/datasets/ShareGPT_V3_unfiltered_cleaned_split.json
|
||||||
|
```
|
||||||
|
`config.env` 里的 `DATASET_PATH="${ROOT_DIR}/datasets/ShareGPT_V3_unfiltered_cleaned_split.json"`
|
||||||
|
会正确解析到这个路径,无需额外配置。
|
||||||
|
|
||||||
|
### 注意
|
||||||
|
|
||||||
|
- 数据集文件被 `.gitignore` 忽略(`datasets/` 规则),本 README 通过 `!datasets/README.md`
|
||||||
|
例外规则入库。新增其它数据集说明文档时,同样需要在该例外规则中列出。
|
||||||
|
- 若实验只用 `--dataset-name random-ids`(基于生成的 token id,无需种子语料),
|
||||||
|
则不需要此文件;当前 910C 实验默认用 `random`,需要下载。
|
||||||
Loading…
x
Reference in New Issue
Block a user