evalstone/evalscope/docs/en/benchmarks/sanskriti.md

Sanskriti

Overview

Sanskriti is a multiple-choice trivia benchmark testing knowledge of Indian states' culture, history, and geography, sourced from state-specific attributes (art, cuisine, festivals, etc.) with Wikipedia-backed answers. From the SANSKRITI paper (arXiv:2506.15355); this adapter loads the dataset mirrored to ModelScope as evalscope/Sanskriti.

Task Description

• Task Type: Multiple-Choice Trivia Question Answering
• Input: A question about a specific Indian state's culture/geography/history, with 4 answer choices
• Output: Correct answer letter
• Subsets: association (state-attribute association trivia), country (country-level trivia), gk (general knowledge), states (state-identification trivia)

Evaluation Notes

• Default configuration uses 0-shot evaluation (the dataset's only split, named train upstream despite being evaluation data)
• Questions and choices are in English
• The paper acknowledges some questions involve ambiguous cultural elements; a small number of rows (~0.6%) whose answer doesn't match any of the 4 listed options are skipped at load time

Properties

Property	Value
Benchmark Name	sanskriti
Dataset ID	evalscope/Sanskriti
Paper	N/A
Tags	Knowledge, MCQ
Metrics	accuracy
Default Shots	0-shot
Evaluation Split	train

Data Statistics

Metric	Value
Total Samples	21,726
Prompt Length (Mean)	322.93 chars
Prompt Length (Min/Max)	256 / 636 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
association	5,453	343.41	273	523
country	5,563	284.48	256	417
gk	5,328	346.94	263	547
states	5,382	318.17	260	636

Sample Example

Subset: association

{
  "input": [
    {
      "id": "0629b222",
      "content": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nWhich of the given regions is home to the Jarawa body painting?\n\nA) Surguja district\nB) South Andaman and Middle Andaman Islands\nC) Buddha Marg, Patna\nD) Telangana"
    }
  ],
  "choices": [
    "Surguja district",
    "South Andaman and Middle Andaman Islands",
    "Buddha Marg, Patna",
    "Telangana"
  ],
  "target": "B",
  "id": 0,
  "group_id": 0,
  "subset_key": "association",
  "metadata": {
    "state": "Andaman_and_Nicobar",
    "attribute": "Art"
  }
}

Prompt Template

Prompt Template:

Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.

{question}

{choices}

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets sanskriti \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['sanskriti'],
    dataset_args={
        'sanskriti': {
            # subset_list: ['association', 'country', 'gk']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)