evalstone/evalscope/docs/en/benchmarks/olmocr_bench.md

olmOCR-Bench

Overview

olmOCR-Bench evaluates end-to-end document transcription: a model reads one rendered PDF page and returns the full Markdown transcription of that page, which is then checked against human-written unit tests instead of a single reference answer.

Task Description

• Task Type: PDF page to Markdown transcription
• Input: A rendered PDF page image
• Output: The complete Markdown transcription of the page
• Domain: Academic papers, scanned books, historical scans, and internal documents

Key Features

• 1,403 PDF pages and 7,019 unit tests in the released bench data; each test states one property a correct transcription must satisfy (text present, text absent, reading order, table structure, or a baseline sanity check)
• Scoring rules are ported 1:1 from the official olmocr bench implementation, so per-subset scores are directly comparable with the official report
• This adapter covers the five non-math sources (headers_footers, long_tiny_text, multi_column, old_scans, table_tests), 845 pages and 3,634 unit tests; the two math-only sources (arxiv_math, old_scans_math) require KaTeX-rendered equation comparison and are not included
• Pages are pre-rendered once to PNG images (longest side 2048 px, matching the official renderer render_pdf_to_base64png with target_longest_image_dim=2048) and packaged, together with the unit tests, into a single parquet on ModelScope; the adapter loads it through the standard remote flow (one download, native parsing) with no PDF rasterization at eval time

Evaluation Notes

• Each sample is one PDF page; its unit tests are evaluated against the model transcription and the subset score is the fraction of unit tests that pass, matching the official per-source metric
• The primary metric is pass_rate. Each subset score equals the official per-source pass rate (the fraction of that source's unit tests that pass), so per-subset scores match the official report exactly
• The official total is the unweighted mean of the per-source pass rates. It is recorded as the report's macro_score (per category), while the headline overall score is EvalScope's native sample-weighted (per-page) micro average; the two differ unless the subsets have equal page counts, so compare per-subset scores or macro_score for parity with the official report
• num reports the number of PDF pages (samples) per subset, so the overall sample count matches the prediction records rather than the unit-test count
• Fuzzy matching thresholds (max_diffs), positional constraints (first_n/last_n), case sensitivity, and table relationship checks follow the official implementation exactly
• A bare null reply is treated as an empty transcription, mirroring how the official harness stores natural_text=null as an empty file
• Requires pip install evalscope[olmocr_bench] (rapidfuzz, fuzzysearch, beautifulsoup4); the benchmark is a single parquet on ModelScope (evalscope/olmOCR-Bench, image bytes + unit tests), a mirror of the Hugging Face release, loaded natively in one download
• Paper | Code | Dataset

Properties

Property	Value
Benchmark Name	olmocr_bench
Dataset ID	evalscope/olmOCR-Bench
Paper	Paper
Tags	MultiModal, QA
Metrics	pass_rate
Default Shots	0-shot
Evaluation Split	test

Data Statistics

Metric	Value
Total Samples	845
Prompt Length (Mean)	599 chars
Prompt Length (Min/Max)	599 / 599 chars

Per-Subset Statistics:

Subset	Samples	Prompt Mean	Prompt Min	Prompt Max
headers_footers	266	599	599	599
long_tiny_text	62	599	599	599
multi_column	231	599	599	599
old_scans	98	599	599	599
table_tests	188	599	599	599

Image Statistics:

Metric	Value
Total Images	845
Images per Sample	min: 1, max: 1, mean: 1
Resolution Range	2048x773 - 2048x1957
Formats	png

Sample Example

Subset: headers_footers

{
  "input": [
    {
      "id": "0c5d3fad",
      "content": [
        {
          "text": "Below is the image of one page of a PDF document. Just return the plain text representation of this document as if you were reading it naturally.\nTurn equations into a LaTeX representation, and tables into markdown format. Remove the headers  ... [TRUNCATED 115 chars] ... l in the document, so be sure to preserve any sentences that come from the previous page, or continue onto the next page, exactly as they are.\nIf there is no text at all that you think you should read, you can output null.\nDo not hallucinate."
        },
        {
          "image": "[BASE64_IMAGE: png, ~297.1KB]"
        }
      ]
    }
  ],
  "target": "headers_footers/0058e04004009cc0df75aab998d3e107dc646b46_page_1_processed.pdf#page=1",
  "id": 0,
  "group_id": 0,
  "subset_key": "headers_footers",
  "metadata": {
    "pdf": "headers_footers/0058e04004009cc0df75aab998d3e107dc646b46_page_1_processed.pdf",
    "page": 1,
    "tests": [
      {
        "pdf": "headers_footers/0058e04004009cc0df75aab998d3e107dc646b46_page_1_processed.pdf",
        "page": 1,
        "id": "0058e04004009cc0df75aab998d3e107dc646b46_page_1_processed_pg1_header_01",
        "type": "absent",
        "max_diffs": 2,
        "checked": "verified",
        "url": "https://webges.uv.es/uvTaeWeb/DescargarCertificadoPublicacion.do?codigo=ANUNCIO-C9-2022-1285",
        "text": "Certificado de publicación disponible en http://fandango.accv.es:8070/fa",
        "case_sensitive": false,
        "first_n": null,
        "last_n": null
      },
      {
        "pdf": "headers_footers/0058e04004009cc0df75aab998d3e107dc646b46_page_1_processed.pdf",
        "page": 1,
        "id": "0058e04004009cc0df75aab998d3e107dc646b46_page_1_processed_pg1_header_02",
        "type": "absent",
        "max_diffs": 4,
        "checked": "verified",
        "url": "https://webges.uv.es/uvTaeWeb/DescargarCertificadoPublicacion.do?codigo=ANUNCIO-C9-2022-1285",
        "text": "Este documento será custodiado por la Agencia de Tecnología y Certificación Electrónica - ISTEC Pista de Ademuz S/N. 46100 Burjassot (Valencia). Tel. 902 482 481 Correo-e: accv@accv.es",
        "case_sensitive": false,
        "first_n": null,
        "last_n": null
      }
    ]
  }
}

Note: Some content was truncated for display.

Prompt Template

Prompt Template:

Below is the image of one page of a PDF document. Just return the plain text representation of this document as if you were reading it naturally.
Turn equations into a LaTeX representation, and tables into markdown format. Remove the headers and footers, but keep references and footnotes.
Read any natural handwriting.
This is likely one page out of several in the document, so be sure to preserve any sentences that come from the previous page, or continue onto the next page, exactly as they are.
If there is no text at all that you think you should read, you can output null.
Do not hallucinate.

Usage

Using CLI

evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets olmocr_bench \
    --limit 10  # Remove this line for formal evaluation

Using Python

from evalscope import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='YOUR_MODEL',
    api_url='OPENAI_API_COMPAT_URL',
    api_key='EMPTY_TOKEN',
    datasets=['olmocr_bench'],
    dataset_args={
        'olmocr_bench': {
            # subset_list: ['headers_footers', 'long_tiny_text', 'multi_column']  # optional, evaluate specific subsets
        }
    },
    limit=10,  # Remove this line for formal evaluation
)

run_task(task_cfg=task_cfg)