GSM8K-Indic#

Overview#

GSM8K-Indic translates the GSM8K grade-school math word problems into 10 Indic languages, each available in native script and a romanized (Latin transliteration) variant, plus the original English.

Task Description#

  • Task Type: Multilingual Mathematical Word Problem Solving

  • Input: Grade-school math word problem in one of 21 language/script variants

  • Output: Numerical answer derived through step-by-step reasoning

  • Languages: Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu — each Indic language in both native script and a _roman transliterated variant

Evaluation Notes#

  • Default configuration uses 0-shot evaluation (test split, the only split available)

  • Use subset_list to evaluate specific languages/scripts (e.g., ['hi', 'hi_roman'])

  • Gold answers are the original English reasoning chain’s final numeric value; only the question is translated

Properties#

Property

Value

Benchmark Name

gsm8k_indic

Dataset ID

sarvamai/gsm8k-indic

Paper

N/A

Tags

Math, MultiLingual, Reasoning

Metrics

accuracy

Default Shots

0-shot

Evaluation Split

test

Data Statistics#

Metric

Value

Total Samples

27,670

Prompt Length (Mean)

335.91 chars

Prompt Length (Min/Max)

135 / 1045 chars

Per-Subset Statistics:

Subset

Samples

Prompt Mean

Prompt Min

Prompt Max

en

1,319

310.87

144

919

bn

1,319

306.06

136

790

gu

1,319

303.06

135

770

hi

1,319

311.1

147

761

kn

1,319

338.11

145

831

ml

1,319

345.21

156

907

mr

1,319

315.01

146

785

or

1,319

313.28

141

815

pa

1,319

312.77

152

800

ta

1,319

367.12

161

972

te

1,319

330.78

142

877

bn_roman

1,319

330.69

153

816

gu_roman

1,318

335.73

148

885

hi_roman

1,319

340.77

154

869

kn_roman

1,316

368.49

149

959

ml_roman

1,319

363.39

159

937

mr_roman

1,310

339.56

156

848

or_roman

1,319

339.92

158

930

pa_roman

1,319

334.74

155

847

ta_roman

1,303

387.6

163

1045

te_roman

1,319

360.46

145

1025

Sample Example#

Subset: en

{
  "input": [
    {
      "id": "92205129",
      "content": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?\nPlease reason step by step, and put your final answer within \\boxed{}."
    }
  ],
  "target": "18",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "language": "English"
  }
}

Prompt Template#

Prompt Template:

{question}
Please reason step by step, and put your final answer within \boxed{{}}.

Usage#

Using CLI#

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets gsm8k_indic \
    --limit 10  # Remove this line for formal evaluation

Using Python#

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['gsm8k_indic'],
    dataset_args={
        'gsm8k_indic': {
            # subset_list: ['en', 'bn', 'gu']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)