OmniDocBench#

Overview#

This adapter preserves EvalScope’s original 981-page OmniDocBench TSV integration for compatibility with existing evaluations.

Task Description#

Task Type: Document Parsing and Understanding
Input: PDF page image
Output: Parsed document structure in Markdown format
Domain: Document understanding, OCR, layout analysis

Key Features#

Uses the legacy evalscope/OmniDocBench_tsv dataset with 981 PDF pages
Covers text blocks, formulas, tables, and reading order
Keeps the existing local Python scoring implementation and metric names unchanged
Remains available for reproducing existing EvalScope results

Evaluation Notes#

This legacy TSV dataset is not labeled as a specific upstream OmniDocBench release.
For new evaluations, use the recommended omni_doc_bench_v1_6 benchmark.
Implements the existing end2end and quick_match scoring paths.
Metrics: Edit_dist, BLEU, METEOR (text), TEDS (tables)
Install the evalscope[omnidoc_bench] extra for legacy scoring dependencies.
Output format: Markdown with LaTeX formulas and HTML tables
Scores from this legacy integration are not directly comparable with v1.6 scores.

Properties#

Property	Value
Benchmark Name	`omni_doc_bench`
Dataset ID	evalscope/OmniDocBench_tsv
Paper	N/A
Tags	`Knowledge`, `MultiModal`, `QA`
Metrics	`text_block`, `display_formula`, `table`, `reading_order`, `normalized_score`
Default Shots	0-shot
Evaluation Split	`train`

Data Statistics#

Metric	Value
Total Samples	981
Prompt Length (Mean)	1408 chars
Prompt Length (Min/Max)	1408 / 1408 chars

Image Statistics:

Metric	Value
Total Images	981
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	516x729 - 10142x14342
Formats	jpeg

Sample Example#

Subset: default

{
  "input": [
    {
      "id": "7c6fda98",
      "content": [
        {
          "image": "[BASE64_IMAGE: png, ~321.8KB]"
        },
        {
          "text": " You are an AI assistant specialized in converting PDF images to Markdown format. Please follow these instructions for the conversion:\n\n    1. Text Processing:\n    - Accurately recognize all text content in the PDF image without guessing or i ... [TRUNCATED 924 chars] ... sible.\n\n    Please strictly follow these guidelines to ensure accuracy and consistency in the conversion. Your task is to accurately convert the content of the PDF image into Markdown format without adding any extra explanations or comments.\n"
        }
      ]
    }
  ],
  "target": "{\"layout_dets\": [{\"category_type\": \"title\", \"poly\": [102.5999912116609, 120.87255879760278, 719.3118659856144, 120.87255879760278, 719.3118659856144, 194.14083813380114, 102.5999912116609, 194.14083813380114], \"ignore\": false, \"order\": 1, \"an ... [TRUNCATED 9876 chars] ... nguage\": \"simplified_chinese\", \"layout\": \"1andmore_column\", \"special_issue\": [\"watermark\"]}, \"page_no\": 11, \"height\": 1500, \"width\": 2667, \"image_path\": \"eastmoney_59cde7e939acc3124df9d3f2c85b5a0ec41b9da1157d5be38e098672022b47cb.pdf_11.jpg\"}}",
  "id": 0,
  "group_id": 0
}

Note: Some content was truncated for display.

Prompt Template#

Prompt Template:

 You are an AI assistant specialized in converting PDF images to Markdown format. Please follow these instructions for the conversion:

    1. Text Processing:
    - Accurately recognize all text content in the PDF image without guessing or inferring.
    - Convert the recognized text into Markdown format.
    - Maintain the original document structure, including headings, paragraphs, lists, etc.

    2. Mathematical Formula Processing:
    - Convert all mathematical formulas to LaTeX format.
    - Enclose inline formulas with \( \). For example: This is an inline formula \( E = mc^2 \)
    - Enclose block formulas with \\[ \\]. For example: \[ \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} \]

    3. Table Processing:
    - Convert tables to HTML format.
    - Wrap the entire table with <table> and </table>.

    4. Figure Handling:
    - Ignore figures content in the PDF image. Do not attempt to describe or convert images.

    5. Output Format:
    - Ensure the output Markdown document has a clear structure with appropriate line breaks between elements.
    - For complex layouts, try to maintain the original document's structure and format as closely as possible.

    Please strictly follow these guidelines to ensure accuracy and consistency in the conversion. Your task is to accurately convert the content of the PDF image into Markdown format without adding any extra explanations or comments.

Extra Parameters#

Parameter	Type	Default	Description
`match_method`	`str`	`quick_match`	Scoring match method used for evaluation. Choices: [‘quick_match’, ‘simple_match’, ‘no_split’]

Usage#

Using CLI#

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets omni_doc_bench \
    --limit 10  # Remove this line for formal evaluation

Using Python#

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['omni_doc_bench'],
    dataset_args={
        'omni_doc_bench': {
            # extra_params: {}  # uses default extra parameters
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)