evalstone/evalscope/docs/en/benchmarks/indic_param.md

IndicParam

Overview

IndicParam is a graduate-level benchmark evaluating LLM understanding of low- and extremely low-resource Indic languages. All 13,207 multiple-choice questions are sourced from official UGC-NET language question papers and answer keys, presented in each language's native script (or code-mixed form for Sanskrit-English).

Task Description

• Task Type: Graduate-Level Multiple-Choice Question Answering
• Input: A UGC-NET exam question with 4 answer choices, in a low-resource Indic language
• Output: Correct answer letter
• Languages: Bodo, Dogri, Gujarati (Surya script), Konkani, Maithili, Marathi, Nepali, Oriya, Rajasthani, Sanskrit, Sanskrit-English code-mixed, Santali

Key Features

• 13,207 multiple-choice questions sourced from official UGC-NET language question papers
• 12 low-resource Indic languages/scripts, including extremely low-resource ones like Bodo and Santali
• Questions are presented in each language's native script (or code-mixed form for Sanskrit-English)
• All languages ship in a single dataset config, differentiated by the subject field

Evaluation Notes

• Default configuration uses 0-shot evaluation (test split, the only split available)
• Use subset_list to evaluate specific languages
• All languages ship in a single dataset config, differentiated by the subject field; this adapter reformats by that field

Properties

Property	Value
Benchmark Name	indic_param
Dataset ID	bharatgenai/IndicParam
Paper	N/A
Tags	Knowledge, MCQ, MultiLingual
Metrics	accuracy
Default Shots	0-shot
Evaluation Split	test

Data Statistics

Metric	Value
Total Samples	13,207
Prompt Length (Mean)	376.02 chars
Prompt Length (Min/Max)	218 / 1413 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
Bodo	1,313	461.37	256	738
Dogri	1,027	487.72	245	853
Gujarati_surya	1,044	395.79	255	611
Konkani	1,328	396.77	245	1413
Maithili	1,286	284.67	218	451
Marathi	1,245	382.66	242	957
Nepali	1,038	406.12	260	857
Oriya	577	365.04	239	924
Rajasthani	1,190	321.32	237	1136
Sanskrit	1,315	304.51	229	833
Sanskrit Mix	971	352.41	253	693
Santali	873	366.16	233	809

Sample Example

Subset: Bodo

{
  "input": [
    {
      "id": "0616580a",
      "content": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nआथिखालाव सुबुं थुनलाइफोरखौ बुथुमनो थाखाय बबे आदबखौ रासिनै बाहायनाय जायो\n\nA) फट' दैखांनाय\nB) रेकरडिं खालामनाय\nC) सल बुंहोनाय\nD) सल खोनासंनाय"
    }
  ],
  "choices": [
    "फट' दैखांनाय",
    "रेकरडिं खालामनाय",
    "सल बुंहोनाय",
    "सल खोनासंनाय"
  ],
  "target": "B",
  "id": 0,
  "group_id": 0,
  "subset_key": "Bodo",
  "metadata": {
    "subject": "Bodo",
    "exam_name": "Question Papers of NET Dec. 2012 Bodo Paper III hindi"
  }
}

Prompt Template

Prompt Template:

Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.

{question}

{choices}

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets indic_param \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['indic_param'],
    dataset_args={
        'indic_param': {
            # subset_list: ['Bodo', 'Dogri', 'Gujarati_surya']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)