evalstone/evalscope/docs/en/benchmarks/arc_indic.md

ARC-Challenge-Indic

Overview

ARC-Challenge-Indic is a translation of the AI2 Reasoning Challenge (ARC-Challenge) science question-answering benchmark into 10 Indic languages, plus the original English set, for evaluating multilingual scientific reasoning.

Task Description

• Task Type: Multilingual Multiple-Choice Science Question Answering
• Input: Science question with answer choices in one of 11 languages
• Output: Correct answer letter
• Languages: Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu

Evaluation Notes

• Default configuration uses 0-shot evaluation (test split)
• Use subset_list to evaluate specific languages (e.g., ['hi', 'ta'])
• Same underlying science-exam questions as arc (Challenge split), machine/human translated per language

Properties

Property	Value
Benchmark Name	arc_indic
Dataset ID	sarvamai/arc-challenge-indic
Paper	N/A
Tags	MCQ, MultiLingual, Reasoning
Metrics	accuracy
Default Shots	0-shot
Evaluation Split	test
Train Split	validation

Data Statistics

Metric	Value
Total Samples	12,647
Prompt Length (Mean)	448.01 chars
Prompt Length (Min/Max)	236 / 2053 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
bn	1,150	432.51	242	1137
en	1,147	454.88	253	1111
gu	1,150	426.57	243	1098
hi	1,150	443.47	236	1162
kn	1,150	456.08	245	1199
ml	1,150	473.31	239	2053
mr	1,150	434.22	242	1133
or	1,150	440.04	243	1374
pa	1,150	443.35	236	1132
ta	1,150	479.12	243	1295
te	1,150	444.53	244	1172

Sample Example

Subset: bn

{
  "input": [
    {
      "id": "f750462b",
      "content": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nএকজন খগোলবিদ পর্যবেক্ষণ করেন যে একটি উল্কা পতনের পরে একটি গ্রহের ঘূর্ণন গতি বেড়ে যায়। ঘূর্ণন বৃদ্ধির ফলে কোন প্রভাবটি সবচেয়ে বেশি সম্ভাব্য?\n\nA) গ্রহের ঘনত্ব কমে যাবে।\nB) গ্রহীয় বছরগুলি আরও দীর্ঘ হবে।\nC) গ্রহের দিনগুলি ছোট হয়ে যাবে।\nD) গ্রহের মাধ্যাকর্ষণ শক্তি আরও বৃদ্ধি পাবে।"
    }
  ],
  "choices": [
    "গ্রহের ঘনত্ব কমে যাবে।",
    "গ্রহীয় বছরগুলি আরও দীর্ঘ হবে।",
    "গ্রহের দিনগুলি ছোট হয়ে যাবে।",
    "গ্রহের মাধ্যাকর্ষণ শক্তি আরও বৃদ্ধি পাবে।"
  ],
  "target": "C",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "id": "Mercury_7175875",
    "language": "Bengali"
  }
}

Prompt Template

Prompt Template:

Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.

{question}

{choices}

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets arc_indic \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['arc_indic'],
    dataset_args={
        'arc_indic': {
            # subset_list: ['bn', 'en', 'gu']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)