148 lines
3.7 KiB
Markdown
148 lines
3.7 KiB
Markdown
# FLEURS
|
|
|
|
|
|
## Overview
|
|
|
|
FLEURS (Few-shot Learning Evaluation of Universal Representations of Speech) is a massively multilingual benchmark covering 102 languages for evaluating automatic speech recognition (ASR), spoken language understanding, and speech translation.
|
|
|
|
## Task Description
|
|
|
|
- **Task Type**: Automatic Speech Recognition (ASR)
|
|
- **Input**: Audio recordings with speech in various languages
|
|
- **Output**: Transcribed text in the corresponding language
|
|
- **Languages**: 102 languages including Mandarin Chinese, Cantonese, English, and many more
|
|
|
|
## Key Features
|
|
|
|
- Massive multilingual coverage (102 languages)
|
|
- Derived from FLoRes-101 machine translation benchmark
|
|
- Includes diverse language families and scripts
|
|
- High-quality human recordings and transcriptions
|
|
- Metadata includes gender, language group, and speaker information
|
|
|
|
## Evaluation Notes
|
|
|
|
- Default configuration uses **test** split
|
|
- Primary metric: **Word Error Rate (WER)**
|
|
- Default subsets: `cmn_hans_cn` (Mandarin), `en_us` (English), `yue_hant_hk` (Cantonese)
|
|
- Language-specific text normalization applied during evaluation
|
|
- Prompt: "Please recognize the speech and only output the recognized content"
|
|
|
|
|
|
## Properties
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **Benchmark Name** | `fleurs` |
|
|
| **Dataset ID** | [lmms-lab/fleurs](https://modelscope.cn/datasets/lmms-lab/fleurs/summary) |
|
|
| **Paper** | N/A |
|
|
| **Tags** | `Audio`, `MultiLingual`, `SpeechRecognition` |
|
|
| **Metrics** | `wer` |
|
|
| **Default Shots** | 0-shot |
|
|
| **Evaluation Split** | `test` |
|
|
|
|
|
|
## Data Statistics
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Total Samples | 2,411 |
|
|
| Prompt Length (Mean) | 67 chars |
|
|
| Prompt Length (Min/Max) | 67 / 67 chars |
|
|
|
|
**Per-Subset Statistics:**
|
|
|
|
| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
|
|
|--------|---------|-------------|------------|------------|
|
|
| `cmn_hans_cn` | 945 | 67 | 67 | 67 |
|
|
| `en_us` | 647 | 67 | 67 | 67 |
|
|
| `yue_hant_hk` | 819 | 67 | 67 | 67 |
|
|
|
|
**Audio Statistics:**
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Total Audio Files | 2,411 |
|
|
| Audio per Sample | min: 1, max: 1, mean: 1 |
|
|
| Formats | wav |
|
|
|
|
|
|
## Sample Example
|
|
|
|
**Subset**: `cmn_hans_cn`
|
|
|
|
```json
|
|
{
|
|
"input": [
|
|
{
|
|
"id": "daf508c3",
|
|
"content": [
|
|
{
|
|
"text": "Please recognize the speech and only output the recognized content:"
|
|
},
|
|
{
|
|
"audio": "[BASE64_AUDIO: wav, ~648.8KB]",
|
|
"format": "wav"
|
|
}
|
|
]
|
|
}
|
|
],
|
|
"target": "这 并 不 是 告 别 这 是 一 个 篇 章 的 结 束 也 是 新 篇 章 的 开 始",
|
|
"id": 0,
|
|
"group_id": 0,
|
|
"metadata": {
|
|
"id": 1906,
|
|
"num_samples": 166080,
|
|
"raw_transcription": "“这并不是告别。这是一个篇章的结束，也是新篇章的开始。”",
|
|
"language": "Mandarin Chinese",
|
|
"gender": 0,
|
|
"lang_id": "cmn_hans",
|
|
"lang_group_id": 6
|
|
}
|
|
}
|
|
```
|
|
|
|
## Prompt Template
|
|
|
|
**Prompt Template:**
|
|
```text
|
|
Please recognize the speech and only output the recognized content:
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Using CLI
|
|
|
|
```bash
|
|
evalscope eval \
|
|
--model YOUR_MODEL \
|
|
--api-url OPENAI_API_COMPAT_URL \
|
|
--api-key EMPTY_TOKEN \
|
|
--datasets fleurs \
|
|
--limit 10 # Remove this line for formal evaluation
|
|
```
|
|
|
|
### Using Python
|
|
|
|
```python
|
|
from evalscope import run_task
|
|
from evalscope.config import TaskConfig
|
|
|
|
task_cfg = TaskConfig(
|
|
model='YOUR_MODEL',
|
|
api_url='OPENAI_API_COMPAT_URL',
|
|
api_key='EMPTY_TOKEN',
|
|
datasets=['fleurs'],
|
|
dataset_args={
|
|
'fleurs': {
|
|
# subset_list: ['cmn_hans_cn', 'en_us', 'yue_hant_hk'] # optional, evaluate specific subsets
|
|
}
|
|
},
|
|
limit=10, # Remove this line for formal evaluation
|
|
)
|
|
|
|
run_task(task_cfg=task_cfg)
|
|
```
|
|
|
|
|