HellaSwag-Hindi

Overview

HellaSwag-Hindi is a Hindi translation of the HellaSwag commonsense sentence-completion benchmark’s full validation set. The context stem stays in English; the 4 candidate continuations are translated into Hindi, so the model must connect an English scenario to its most plausible Hindi-phrased ending. Sourced from ai4bharat/hellaswag-translated (the canonical name; the older ai4bharat/hellaswag-hi ID redirects here), the same dataset used by lighteval’s community_hellaswag_hin tasks.

Task Description

• Task Type: Commonsense Sentence Completion (mixed-language)

• Input: An English context sentence with 4 Hindi-language candidate continuations

• Output: Correct answer letter

• Coverage: Full HellaSwag validation set (10,042 examples)

Key Features

• Full HellaSwag validation set: 10,042 examples with gold labels

• English context stem paired with Hindi-translated candidate endings (mixed-language setup)

• Same dataset used by lighteval’s community_hellaswag_hin task suite

Evaluation Notes

• Default configuration uses 0-shot evaluation (validation split, the only labeled split available — HellaSwag’s test split ships without gold labels)

• Loads from ModelScope by default (mirrored as ai4bharat/hellaswag-translated), no token required

Properties

Property	Value
Benchmark Name	hellaswag_hi
Dataset ID	ai4bharat/hellaswag-translated
Paper	N/A
Tags	MCQ, Reasoning
Metrics	accuracy
Default Shots	0-shot
Evaluation Split	validation

Data Statistics

Metric	Value
Total Samples	10,042
Prompt Length (Mean)	1021.77 chars
Prompt Length (Min/Max)	367 / 1977 chars

Sample Example

Subset: hi

{
  "input": [
    {
      "id": "f96132ec",
      "content": "Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nA man is sitting on a roof. he\n\nA) वह स्की की एक जोड़ी को लपेटने के लिए रैप का उपयोग कर रहा है।\nB) यह स्तर की टाइलों को चीर रहा है।\nC) वह एक रूबिक क्यूब पकड़े हुए है।\nD) एक छत पर छत खींचना शुरू करता है।"
    }
  ],
  "choices": [
    "वह स्की की एक जोड़ी को लपेटने के लिए रैप का उपयोग कर रहा है।",
    "यह स्तर की टाइलों को चीर रहा है।",
    "वह एक रूबिक क्यूब पकड़े हुए है।",
    "एक छत पर छत खींचना शुरू करता है।"
  ],
  "target": "D",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "activity_label": "Roof shingle removal"
  }
}

Prompt Template

Prompt Template:

Answer the following multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of {letters}.

{question}

{choices}

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets hellaswag_hi \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['hellaswag_hi'],
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)