evalstone/evalscope/docs/en/benchmarks/omni_doc_bench.md

OmniDocBench

Overview

This adapter preserves EvalScope's original 981-page OmniDocBench TSV integration for compatibility with existing evaluations.

Task Description

• Task Type: Document Parsing and Understanding
• Input: PDF page image
• Output: Parsed document structure in Markdown format
• Domain: Document understanding, OCR, layout analysis

Key Features

• Uses the legacy evalscope/OmniDocBench_tsv dataset with 981 PDF pages
• Covers text blocks, formulas, tables, and reading order
• Keeps the existing local Python scoring implementation and metric names unchanged
• Remains available for reproducing existing EvalScope results

Evaluation Notes

• This legacy TSV dataset is not labeled as a specific upstream OmniDocBench release.
• For new evaluations, use the recommended omni_doc_bench_v1_6 benchmark.
• Implements the existing end2end and quick_match scoring paths.
• Metrics: Edit_dist, BLEU, METEOR (text), TEDS (tables)
• Install the evalscope[omnidoc_bench] extra for legacy scoring dependencies.
• Output format: Markdown with LaTeX formulas and HTML tables
• Scores from this legacy integration are not directly comparable with v1.6 scores.

Properties

Property	Value
Benchmark Name	omni_doc_bench
Dataset ID	evalscope/OmniDocBench_tsv
Paper	N/A
Tags	Knowledge, MultiModal, QA
Metrics	text_block, display_formula, table, reading_order, normalized_score
Default Shots	0-shot
Evaluation Split	train

Data Statistics

Metric	Value
Total Samples	981
Prompt Length (Mean)	1408 chars
Prompt Length (Min/Max)	1408 / 1408 chars

Image Statistics:

Metric	Value
Total Images	981
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	516x729 - 10142x14342
Formats	jpeg

Sample Example

Subset: default

{
  "input": [
    {
      "id": "7c6fda98",
      "content": [
        {
          "image": "[BASE64_IMAGE: png, ~321.8KB]"
        },
        {
          "text": " You are an AI assistant specialized in converting PDF images to Markdown format. Please follow these instructions for the conversion:\n\n    1. Text Processing:\n    - Accurately recognize all text content in the PDF image without guessing or i ... [TRUNCATED 924 chars] ... sible.\n\n    Please strictly follow these guidelines to ensure accuracy and consistency in the conversion. Your task is to accurately convert the content of the PDF image into Markdown format without adding any extra explanations or comments.\n"
        }
      ]
    }
  ],
  "target": "{\"layout_dets\": [{\"category_type\": \"title\", \"poly\": [102.5999912116609, 120.87255879760278, 719.3118659856144, 120.87255879760278, 719.3118659856144, 194.14083813380114, 102.5999912116609, 194.14083813380114], \"ignore\": false, \"order\": 1, \"an ... [TRUNCATED 9876 chars] ... nguage\": \"simplified_chinese\", \"layout\": \"1andmore_column\", \"special_issue\": [\"watermark\"]}, \"page_no\": 11, \"height\": 1500, \"width\": 2667, \"image_path\": \"eastmoney_59cde7e939acc3124df9d3f2c85b5a0ec41b9da1157d5be38e098672022b47cb.pdf_11.jpg\"}}",
  "id": 0,
  "group_id": 0
}

Note: Some content was truncated for display.

Prompt Template

Prompt Template:

 You are an AI assistant specialized in converting PDF images to Markdown format. Please follow these instructions for the conversion:

    1. Text Processing:
    - Accurately recognize all text content in the PDF image without guessing or inferring.
    - Convert the recognized text into Markdown format.
    - Maintain the original document structure, including headings, paragraphs, lists, etc.

    2. Mathematical Formula Processing:
    - Convert all mathematical formulas to LaTeX format.
    - Enclose inline formulas with \( \). For example: This is an inline formula \( E = mc^2 \)
    - Enclose block formulas with \\[ \\]. For example: \[ \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} \]

    3. Table Processing:
    - Convert tables to HTML format.
    - Wrap the entire table with <table> and </table>.

    4. Figure Handling:
    - Ignore figures content in the PDF image. Do not attempt to describe or convert images.

    5. Output Format:
    - Ensure the output Markdown document has a clear structure with appropriate line breaks between elements.
    - For complex layouts, try to maintain the original document's structure and format as closely as possible.

    Please strictly follow these guidelines to ensure accuracy and consistency in the conversion. Your task is to accurately convert the content of the PDF image into Markdown format without adding any extra explanations or comments.

Extra Parameters

Parameter	Type	Default	Description
match_method	str	quick_match	Scoring match method used for evaluation. Choices: ['quick_match', 'simple_match', 'no_split']

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets omni_doc_bench \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['omni_doc_bench'],
    dataset_args={
        'omni_doc_bench': {
            # extra_params: {}  # uses default extra parameters
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)