CC-OCR-V2#

Overview#

CC-OCR V2 is a challenging OCR benchmark tailored to real-world enterprise document processing. It deliberately over-samples the hard and corner cases that prior OCR benchmarks under-represent, such as photographed and scanned tables, handwritten formulas, multi-page receipts, and low-quality multilingual scene text.

Task Description#

Task Type: Text recognition, document parsing, document grounding, key information extraction, and document VQA
Input: One or more document images plus the task instruction shipped with each sample
Output: Free-form text, LaTeX, HTML tables, SMILES strings, JSON objects, or bounding boxes, depending on the track
Modalities: Image + text, bilingual (Chinese / English) with 32 additional languages in the recognition track

Key Features#

7,093 official samples over 5 tracks and 16 sub-tasks, evaluated as one benchmark; 7,091 are loaded because the dataset repository ships no image for two of them
recognition: multilingual (32 languages) and natural-scene text reading
parsing: complex tables, general documents, handwritten formulas, molecular structures, and information boards
grounding: text grounding (single box) and object grounding (multi-box detection with labels)
extraction: schema-driven key information extraction over business, public-service, and regulated records
qa: question answering over blueprints, dashboards, and financial documents
Prompts come from the official dataset, so results stay comparable to the published leaderboard

Evaluation Notes#

Every sample yields one score in [0, 1]; each track uses its official metric: recognition = token-level F1, parsing = edit similarity / TEDS, grounding = IoU, extraction = field-level F1, qa = substring match with ANLS fallback
Subset scores are sample means; the per-track category also reports a macro average over its sub-tasks
The grounding prompts ask for boxes on a 0-1000 grid; predictions in absolute pixels are rescaled as if normalized and therefore score close to zero, matching the official leaderboard behavior
Full-page parsing targets are long, so allow a generous max_tokens (4096 or more)
Requires: apted, distance, lxml, python-Levenshtein, scipy, zss (pip install 'evalscope[cc_ocr_v2]')
The dataset is a file tree of images and answers (about 5 GB). Only the tracks listed in subset_list are downloaded, so restricting subsets keeps the download small

Properties#

Property	Value
Benchmark Name	`cc_ocr_v2`
Dataset ID	evalscope/CC-OCR-V2
Paper	Paper
Tags	`Grounding`, `MultiLingual`, `MultiModal`, `QA`
Metrics	`normalized_score`
Default Shots	0-shot
Evaluation Split	`test`

Data Statistics#

Metric	Value
Total Samples	7,091
Prompt Length (Mean)	272.16 chars
Prompt Length (Min/Max)	10 / 1330 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
`multi_lingual_recognition`	639	101	101	101
`natural_scene_recognition`	1,150	101.87	101	103
`complex_table_parsing`	300	327	327	327
`formula_parsing`	100	119	119	119
`general_documents_parsing`	299	258	258	258
`info_board_parsing`	26	701	701	701
`molecular_parsing`	100	232	232	232
`object_grounding`	734	491.83	306	1330
`text_grounding`	734	369.88	358	468
`business_transactions`	340	793.91	702	1105
`public_services`	369	735.45	687	902
`regulated_records`	300	798.84	722	898
`blueprint_qa`	100	25.4	10	69
`dashboards_fact_qa`	400	66.59	19	159
`dashboards_numeric_qa`	500	70.49	19	148
`financial_documents_qa`	1,000	41.77	14	115

Image Statistics:

Metric	Value
Total Images	7,116
Images per Sample	min: 1, max: 3, mean: 1.0
Resolution Range	70x71 - 5313x7219
Formats	jpeg, png

Sample Example#

Subset: multi_lingual_recognition

{
  "input": [
    {
      "id": "253834d2",
      "content": [
        {
          "image": "~/.cache/modelscope/hub/datasets/evalscope/CC-OCR-V2/recognition/multi_lingual_recognition/images/multi_lan_ocr_Arabic_Arabic_20/0c780237abcb.jpg"
        },
        {
          "text": "Please output only the text content from the image without any additional descriptions or formatting."
        }
      ]
    }
  ],
  "target": "الآن بحق السماء يا دجونا، أكل طعامك المتحجر غير المطابقة ....Demonstrandum\n.للمواصفات واتركني بسلام .”قال دجونا بحزن وهو ينظر إلى الحقيبة الفارغة: “لقد ذهب كل شيء أنا هنا!” صرخ بصوت غالي مرح، وكتم إليري أنينًا آخر عندما رأى السيد دوفال يقفز“\n ... [TRUNCATED 1733 chars] ... حنة الحشد“\nكان هناك ضحكة خفيفة. كان الشخص الضعيف القلب الذي خاطبه المرافق شابًا زنجيًا\nقويًا، يرتدي ملابس بنية سيمفونية أنيقة، وقبعته القشية مبهرة على الكربون السخام\n!الموجود في جلده. ضحكت فتاة جميلة ملونة على ذراعه. “هيا يا عزيزتي، سوف نريهم",
  "id": 0,
  "group_id": 0,
  "subset_key": "multi_lingual_recognition",
  "metadata": {
    "id": "0c780237abcb",
    "task": "recognition",
    "sub_task": "multi_lingual_recognition",
    "scenario": "multi_lan_ocr_Arabic_Arabic_20",
    "image_paths": [
      "~/.cache/modelscope/hub/datasets/evalscope/CC-OCR-V2/recognition/multi_lingual_recognition/images/multi_lan_ocr_Arabic_Arabic_20/0c780237abcb.jpg"
    ]
  }
}

Note: Some content was truncated for display.

Prompt Template#

No prompt template defined.

Usage#

Using CLI#

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets cc_ocr_v2 \
    --limit 10  # Remove this line for formal evaluation

Using Python#

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['cc_ocr_v2'],
    dataset_args={
        'cc_ocr_v2': {
            # subset_list: ['multi_lingual_recognition', 'natural_scene_recognition', 'complex_table_parsing']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)