OCRBench-v2

Overview

OCRBench v2 is a large-scale bilingual text-centric benchmark with the most comprehensive set of OCR tasks (4x more than OCRBench v1), covering 31 diverse scenarios including street scenes, receipts, formulas, diagrams, and more.

Task Description

• Task Type: Optical Character Recognition and Document Understanding

• Input: Image + OCR/document question

• Output: Text recognition, extraction, or analysis result

• Languages: English and Chinese (bilingual)

Key Features

• 10,000 human-verified question-answering pairs

• 31 diverse scenarios (street scene, receipt, formula, diagram, etc.)

• High proportion of difficult samples

• Comprehensive OCR task coverage

• Bilingual (English and Chinese) evaluation

Evaluation Notes

• Default configuration uses 0-shot evaluation

• Evaluates on test split

• Requires: apted, distance, Levenshtein, lxml, Polygon3, zss packages

• Simple accuracy metric

Properties

Property	Value
Benchmark Name	ocr_bench_v2
Dataset ID	evalscope/OCRBench_v2
Paper	N/A
Tags	Knowledge, MultiModal, QA
Metrics	acc
Default Shots	0-shot
Evaluation Split	test

Data Statistics

Metric	Value
Total Samples	10,000
Prompt Length (Mean)	155.62 chars
Prompt Length (Min/Max)	6 / 1863 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
APP agent en	300	36.9	16	84
ASCII art classification en	200	174.62	162	193
key information extraction cn	400	32.45	9	162
key information extraction en	400	506.23	327	1261
key information mapping en	300	664.02	429	1647
VQA with position en	300	557.69	539	612
chart parsing en	400	67	67	67
cognition VQA cn	200	15.91	6	48
cognition VQA en	800	44.73	14	179
diagram QA en	300	118.23	54	356
document classification en	200	314	314	314
document parsing cn	300	43	43	43
document parsing en	400	51	51	51
formula recognition cn	200	19	19	19
formula recognition en	400	58.39	53	60
handwritten answer extraction cn	200	39.75	22	60
math QA en	300	119	119	119
full-page OCR cn	200	31	31	31
full-page OCR en	200	91	91	91
reasoning VQA en	600	80.45	26	256
reasoning VQA cn	400	163.96	62	633
fine-grained text recognition en	200	155.63	152	156
science QA en	300	387.54	159	1863
table parsing cn	300	139.4	79	211
table parsing en	400	65.39	57	134
text counting en	200	113.87	101	127
text grounding en	200	361.5	357	379
text recognition en	800	37.26	29	104
text spotting en	200	446	446	446
text translation cn	400	230.88	96	291

Image Statistics:

Metric	Value
Total Images	10,000
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	19x10 - 3912x21253
Formats	jpeg

Sample Example

Subset: APP agent en

{
  "input": [
    {
      "id": "51292e32",
      "content": [
        {
          "text": "What is the wrong answer 2?"
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~121.7KB]"
        }
      ]
    }
  ],
  "target": "[\"enabled\", \"on\"]",
  "id": 0,
  "group_id": 0,
  "subset_key": "APP agent en",
  "metadata": {
    "question": "What is the wrong answer 2?",
    "answers": [
      "enabled",
      "on"
    ],
    "eval": "None",
    "dataset_name": "rico",
    "type": "APP agent en",
    "bbox": null,
    "bbox_list": null,
    "content": null
  }
}

Prompt Template

Prompt Template:

{question}

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets ocr_bench_v2 \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['ocr_bench_v2'],
    dataset_args={
        'ocr_bench_v2': {
            # subset_list: ['APP agent en', 'ASCII art classification en', 'key information extraction cn']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)