MILU#

Overview#

MILU (Multi-task Indic Language Understanding Benchmark) is a comprehensive evaluation dataset for assessing LLM performance across 11 Indic languages. It spans 8 domains and 41 subjects, combining translated general-knowledge questions with culturally specific Indian content.

Task Description#

Task Type: Multilingual Multiple-Choice Question Answering
Input: Question with four answer choices in one of 11 languages
Output: Single correct answer letter
Languages: English, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu

Key Features#

8 domains / 41 subjects, including India-specific culture, history, and current affairs
Native-language questions rather than machine-translated MMLU
Each language is a separate dataset config, loaded independently

Evaluation Notes#

Default configuration uses 0-shot evaluation (test split)
Use subset_list to evaluate specific languages (e.g., ['Hindi', 'Tamil']), or limit to cap sample count — evaluating all 11 languages’ full test splits is a large run
Set few_shot_num > 0 to enable few-shot prompting; examples are drawn from the validation split
Loads from ModelScope by default (evalscope’s default dataset_hub), where this dataset is public and needs no token. If you explicitly set dataset_hub to huggingface, note that ai4bharat/MILU is gated there — accept the dataset terms on huggingface.co and set HF_TOKEN (or run huggingface-cli login) first

Properties#

Property	Value
Benchmark Name	`milu`
Dataset ID	ai4bharat/MILU
Paper	N/A
Tags	`Knowledge`, `MCQ`, `MultiLingual`
Metrics	`accuracy`
Default Shots	0-shot
Evaluation Split	`test`
Train Split	`validation`

Data Statistics#

Metric	Value
Total Samples	79,608
Prompt Length (Mean)	377.16 chars
Prompt Length (Min/Max)	223 / 2110 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
`English`	13,535	397.01	227	1930
`Bengali`	6,637	359.93	232	1828
`Gujarati`	4,826	359.36	230	1785
`Hindi`	14,831	367.43	229	1907
`Kannada`	6,234	364.45	229	1753
`Malayalam`	4,321	388.2	239	2110
`Marathi`	6,924	394.85	223	1888
`Odia`	4,525	366.63	238	1825
`Punjabi`	4,099	364.93	234	1874
`Tamil`	6,372	382.22	230	1934
`Telugu`	7,304	384.05	233	1806

Sample Example#

Subset: English

{
  "input": [
    {
      "id": "84726982",
      "content": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nBakelite is what type of polymer?\n\nA) Thermosetting polymer\nB) Thermoplastic polymer\nC) Fibre\nD) Elastomer"
    }
  ],
  "choices": [
    "Thermosetting polymer",
    "Thermoplastic polymer",
    "Fibre",
    "Elastomer"
  ],
  "target": "A",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "language": "English"
  }
}

Prompt Template#

Prompt Template:

Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.

{question}

{choices}

Usage#

Using CLI#

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets milu \
    --limit 10  # Remove this line for formal evaluation

Using Python#

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['milu'],
    dataset_args={
        'milu': {
            # subset_list: ['English', 'Bengali', 'Gujarati']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)