13274243a0
Keep K3 suite selection and report-schema scoring in bash, merge K3/vision dataset_args into dpv4 yamls, and pin EvalScope at 735d920ee911 with local patches. Co-authored-by: Cursor <cursoragent@cursor.com>
120 lines
4.0 KiB
Markdown
120 lines
4.0 KiB
Markdown
# HellaSwag-Hindi
|
|
|
|
|
|
## Overview
|
|
|
|
HellaSwag-Hindi is a Hindi translation of the HellaSwag commonsense sentence-completion benchmark's
|
|
full validation set. The context stem stays in English; the 4 candidate continuations are translated
|
|
into Hindi, so the model must connect an English scenario to its most plausible Hindi-phrased ending.
|
|
Sourced from `ai4bharat/hellaswag-translated` (the canonical name; the older `ai4bharat/hellaswag-hi`
|
|
ID redirects here), the same dataset used by lighteval's `community_hellaswag_hin` tasks.
|
|
|
|
## Task Description
|
|
|
|
- **Task Type**: Commonsense Sentence Completion (mixed-language)
|
|
- **Input**: An English context sentence with 4 Hindi-language candidate continuations
|
|
- **Output**: Correct answer letter
|
|
- **Coverage**: Full HellaSwag validation set (10,042 examples)
|
|
|
|
## Key Features
|
|
|
|
- Full HellaSwag validation set: 10,042 examples with gold labels
|
|
- English context stem paired with Hindi-translated candidate endings (mixed-language setup)
|
|
- Same dataset used by lighteval's `community_hellaswag_hin` task suite
|
|
|
|
## Evaluation Notes
|
|
|
|
- Default configuration uses **0-shot** evaluation (validation split, the only labeled split
|
|
available — HellaSwag's `test` split ships without gold labels)
|
|
- Loads from ModelScope by default (mirrored as `ai4bharat/hellaswag-translated`), no token required
|
|
|
|
|
|
## Properties
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| **Benchmark Name** | `hellaswag_hi` |
|
|
| **Dataset ID** | [ai4bharat/hellaswag-translated](https://modelscope.cn/datasets/ai4bharat/hellaswag-translated/summary) |
|
|
| **Paper** | N/A |
|
|
| **Tags** | `MCQ`, `Reasoning` |
|
|
| **Metrics** | `accuracy` |
|
|
| **Default Shots** | 0-shot |
|
|
| **Evaluation Split** | `validation` |
|
|
|
|
|
|
## Data Statistics
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| Total Samples | 10,042 |
|
|
| Prompt Length (Mean) | 1021.77 chars |
|
|
| Prompt Length (Min/Max) | 367 / 1977 chars |
|
|
|
|
## Sample Example
|
|
|
|
**Subset**: `hi`
|
|
|
|
```json
|
|
{
|
|
"input": [
|
|
{
|
|
"id": "f96132ec",
|
|
"content": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nA man is sitting on a roof. he\n\nA) वह स्की की एक जोड़ी को लपेटने के लिए रैप का उपयोग कर रहा है।\nB) यह स्तर की टाइलों को चीर रहा है।\nC) वह एक रूबिक क्यूब पकड़े हुए है।\nD) एक छत पर छत खींचना शुरू करता है।"
|
|
}
|
|
],
|
|
"choices": [
|
|
"वह स्की की एक जोड़ी को लपेटने के लिए रैप का उपयोग कर रहा है।",
|
|
"यह स्तर की टाइलों को चीर रहा है।",
|
|
"वह एक रूबिक क्यूब पकड़े हुए है।",
|
|
"एक छत पर छत खींचना शुरू करता है।"
|
|
],
|
|
"target": "D",
|
|
"id": 0,
|
|
"group_id": 0,
|
|
"metadata": {
|
|
"activity_label": "Roof shingle removal"
|
|
}
|
|
}
|
|
```
|
|
|
|
## Prompt Template
|
|
|
|
**Prompt Template:**
|
|
```text
|
|
Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.
|
|
|
|
{question}
|
|
|
|
{choices}
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Using CLI
|
|
|
|
```bash
|
|
evalscope eval \
|
|
--model YOUR_MODEL \
|
|
--api-url OPENAI_API_COMPAT_URL \
|
|
--api-key EMPTY_TOKEN \
|
|
--datasets hellaswag_hi \
|
|
--limit 10 # Remove this line for formal evaluation
|
|
```
|
|
|
|
### Using Python
|
|
|
|
```python
|
|
from evalscope import run_task
|
|
from evalscope.config import TaskConfig
|
|
|
|
task_cfg = TaskConfig(
|
|
model='YOUR_MODEL',
|
|
api_url='OPENAI_API_COMPAT_URL',
|
|
api_key='EMPTY_TOKEN',
|
|
datasets=['hellaswag_hi'],
|
|
limit=10, # Remove this line for formal evaluation
|
|
)
|
|
|
|
run_task(task_cfg=task_cfg)
|
|
```
|