# Scanned PDF to EPUB Benchmark: Methodology and PDF Craft Test Results

Our 6-dimension evaluation framework for scanned PDF to reflowable EPUB conversion across 121 sampled pages, plus PDF Craft's blind-reviewed scores (78.6–80.0), speed, and known limitations.

There is no off-the-shelf benchmark designed specifically for converting a full scanned PDF into a readable, reflowable EPUB. Combining four established public standards from document parsing and digital publishing, we built a 6-dimension evaluation rubric across 121 sampled pages. On November 21, 2026, two independent reviewers (Gemini and Deepseek) blind-scored PDF Craft's production outputs under this rubric: **its weighted overall score landed between 78.6 and 80.0 out of 100**.

## 1. Evaluation Methodology

This benchmark tests a single, clearly bounded task: **converting image-based scanned PDFs into original-language, reflowable EPUBs (without translation)**. Conversion speed and pricing are measured separately and never blended into the quality score.

### Public Standards Referenced

| Public Reference                                                                         | What We Adopted                                                      | What It Does Not Cover                                                          |
| ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| [OCR-D Evaluation Specification](https://ocr-d.de/en/spec/ocrd_eval.html)                | Separating text recognition accuracy from layout structure           | Focuses on recognition metrics rather than reflowed reading quality             |
| [OmniDocBench](https://github.com/opendatalab/OmniDocBench) (CVPR 2025)                  | Scoring body text, tables, formulas, and reading order independently | Built for document parsing rather than EPUB navigation and cross-page footnotes |
| [olmOCR-Bench](https://github.com/allenai/olmocr/blob/main/olmocr/bench/README.md) (Ai2) | Breaking page quality into concrete, verifiable checkpoints          | Centered on single-page PDFs rather than book-length continuity                 |
| [EPUBCheck](https://www.w3.org/publishing/epubcheck/) (W3C / DAISY)                      | Automated validation of EPUB packaging, XML, assets, and links       | Checks technical validity, not OCR or content fidelity                          |

> Note: These references document the origin of our evaluation design. Dimension weights and scoring tiers were set for full-book reading and do not imply endorsement by those organizations.

### Test Set and Sampling Rules (9 Books, 121 Sampled Pages)

The main test set locks 8 scanned PDFs across 5 categories (552 pages total), verified by SHA-256 hashes. One additional 88-page native digital art catalog is included purely as a diagnostic control to separate OCR errors from layout-parsing errors, and is excluded from the overall score. Sampling is fixed: **books up to 30 pages are reviewed in full; longer books sample 4 pages each from the beginning, middle, and end (12 pages)**, totaling 121 sampled pages:

| Category (Weight)               | Test Book                                                   | Total Pages | Sampled Pages | Key Challenges                                                                              |
| ------------------------------- | ----------------------------------------------------------- | ----------: | ------------: | ------------------------------------------------------------------------------------------- |
| Literature (25%)                | *Border Town* (real Chinese scan)                           |         187 |            12 | Long-form fiction, dialogue, chapter TOC, illustrations, vertical calligraphed cover text   |
| Literature (25%)                | *Alice in Wonderland Coloring Book*                         |          20 |      20 (all) | Illustrated English reader with full-page line drawings                                     |
| Humanities (25%)                | *Fundamentals of Psychoanalytic Technique*, Ch. 1 (Chinese) |          24 |      24 (all) | Academic prose, small footnotes, translator notes, mixed English/French terms               |
| Humanities (25%)                | *Baopuzi Inner Chapters*, Vol. 1 (Classical Chinese)        |          10 |      10 (all) | Traditional vertical classical text with dense commentary                                   |
| STEM (25%)                      | *Calculus Made Easy*                                        |         278 |            12 | Aged scan with bleed-through, heavy inline/block math, pre-decimal British currency, tables |
| STEM (25%)                      | *MapVision* (CVPR 2024 technical report)                    |           5 |       5 (all) | Two-column English paper, multi-column tables, inline math, references                      |
| Technical Manual (15%)          | *HP LaserJet Pro M404dn* Paper Trays chapter                |          16 |      16 (all) | Numbered procedures, orientation tables, diagrams, caution/note blocks                      |
| Art Catalog (10%)               | *Roman Mosaics* (Italy scanned excerpt)                     |          12 |      12 (all) | Full-page plates, artwork metadata, captions, grouped endnotes                              |
| Art Catalog (control, excluded) | *Roman Mosaics* (native digital PDF)                        |          88 |            12 | Diagnostic baseline separating scan OCR issues from layout parsing                          |

### Six Dimensions, Four-Tier Scale, and Hard Rules

The first five dimensions break down into 16 concrete checkpoints inspected page by page against the source images and the unpacked EPUB XHTML/MathML markup. The sixth dimension is computed automatically by validation scripts.

| Dimension                  | Weight | What Is Checked                                                                                                 |
| -------------------------- | -----: | --------------------------------------------------------------------------------------------------------------- |
| **Text Accuracy**          |    30% | Word-for-word body accuracy, proper nouns and numbers, punctuation and spacing                                  |
| **Completeness**           |    20% | Missing paragraphs, cross-page and cross-column sentence continuity, retained figures and notes                 |
| **Structure & Navigation** |    20% | Reading order, heading detection and hierarchy, usable table of contents, two-way footnote links                |
| **Formulas & Tables**      |    15% | HTML table structure and headers, math structure (such as MathML), plain-text fallback readability              |
| **Layout & Readability**   |    10% | Reflow integrity (no clipping, overlap, collapsed indexes, or stray gaps), visual hierarchy and caption pairing |
| **File Validity**          |     5% | Valid EPUB container, well-formed XML, intact spine and internal asset references (automated)                   |

Hard scoring rules:

* **Four fixed tiers**: Correct = `1`, Minor issue = `0.9` (readable without changing meaning), Major issue = `0.5` (alters meaning or breaks usability), Critical or missing = `0`. No intermediate values are allowed; the worst issue within a checkpoint determines its tier; each defect is penalized only once in its most specific dimension.
* **Automatic zero for critical failures**: A corrupted EPUB that fails to open, or an image-wrapped EPUB with no reflowable text, scores **0** for that book.
* **Strict N/A handling**: A checkpoint is marked N/A (with its weight redistributed) only when the sampled source pages genuinely lack that element. Omissions by the converter never qualify as N/A.
* **Verifiable and blind**: Every score requires page numbers plus source and output quotes. Totals are calculated strictly by script, and all vendor metadata is stripped from the anonymous evaluation package.

## 2. PDF Craft Test Results (Blind Dual-Review)

All 9 source files were converted once on `pdfcraft.ai` in November 2026 (EPUB output, footnote refinement enabled, translation off, no retries or manual edits). On November 21, 2026, Gemini and Deepseek independently blind-scored the anonymized outputs against the same 121 sampled pages.

### Overall Scores, Speed, and Cost

| Metric                       | Gemini Blind Review | Deepseek Blind Review |
| ---------------------------- | ------------------: | --------------------: |
| **Weighted Overall Score**   |            **78.6** |              **80.0** |
| Text Accuracy (30%)          |                72.3 |                  76.3 |
| Completeness (20%)           |                79.3 |                  82.7 |
| Structure & Navigation (20%) |                74.7 |                  72.2 |
| Formulas & Tables (15%)      |                71.1 |                  71.1 |
| Layout & Readability (10%)   |                84.4 |                  81.7 |
| File Validity (5%)           |               100.0 |                 100.0 |

* **Conversion speed (main weakness)**: The 8 main scanned books (552 pages) took **132.1 minutes** (640 pages including the native control took **163.1 minutes**), averaging **24.0–25.5 minutes per 100 pages**. Long books take time; the 278-page *Calculus Made Easy* took 77.2 minutes.
* **Conversion cost**: PDF Craft uses pay-as-you-go page pricing with no subscription. Standard conversion is **$0.30 per 100 pages** (about **$1.92** for 640 pages), and refined mode is **$1.10 per 100 pages** (about **$7.04** for 640 pages). See the [Recharge page](https://pdfcraft.ai/recharge/) for current rates.

### Per-Book Scores and Conversion Time

| Book                                       | Category    | Pages | Gemini Blind Review | Deepseek Blind Review |     Time |
| ------------------------------------------ | ----------- | ----: | ------------------: | --------------------: | -------: |
| *Border Town*                              | Literature  |   187 |                67.6 |                  74.7 | 31.0 min |
| *Alice* Coloring Book                      | Literature  |    20 |                94.9 |                  95.7 |  3.1 min |
| *Psychoanalytic Technique*, Ch. 1          | Humanities  |    24 |                88.0 |                  87.6 |  6.0 min |
| *Baopuzi* (Vol. 1)                         | Humanities  |    10 |                61.6 |                  60.6 |  2.4 min |
| *Calculus Made Easy*                       | STEM        |   278 |                77.8 |                  80.5 | 77.2 min |
| *MapVision* (CVPR paper)                   | STEM        |     5 |                74.5 |                  71.2 |  2.4 min |
| *HP Paper Trays* Manual                    | Manual      |    16 |                92.5 |                  96.3 |  5.0 min |
| *Roman Mosaics* (scanned)                  | Art Catalog |    12 |                67.0 |                  67.6 |  5.0 min |
| *Roman Mosaics* (native control, excluded) | Art Catalog |    88 |                68.8 |                  69.4 | 31.0 min |

Single-book scores ranged from **60.6 to 96.3**. Both blind reviews agreed closely on ranking: illustrated English books (*Alice*), technical manuals (*HP*), and modern academic chapters (*Psychoanalytic Technique*) scored highest (87.6–96.3), while vertical classical Chinese (*Baopuzi*) and endnote-heavy art catalogs scored lowest (60.6–69.4).

### Strengths and Verified Weaknesses

**What worked well:**

1. **Clean EPUB packaging and real HTML tables**: Every EPUB passed automated packaging and link validation (100/100). Data tables across *Calculus Made Easy*, *MapVision*, and the *HP* manual stayed structured as HTML `<table>` elements instead of flattened text.
2. **MathML formulas and modern footnotes**: Fractions, roots, and superscripts in *Calculus Made Easy* were preserved in standard MathML; 21 of 22 footnotes in *Psychoanalytic Technique* had working two-way jump links.
3. **High fidelity on illustrated readers and manuals**: *Alice* (94.9–95.7) and the *HP* manual (92.5–96.3) preserved body text, steps, proper nouns, and figures accurately.

**Key areas for improvement (grouped by issue type):**

1. **Partially missing anchors on complex notes**: Two-way note links can still be missing in vertical classical commentary, endnote-heavy art catalogs, and specially marked translator notes.
2. **Inconsistent chapter TOC and rare proper nouns in long Chinese fiction**: Some Chinese numeral chapter headings are not grouped cleanly into the table of contents, and uncommon place names or proper nouns may still need manual review.
3. **Two-column breaks and table-header subscripts**: Two-column papers can occasionally drop text across column boundaries, and subscripts in dense table headers may degrade or collide.
4. **Plain-text math fallback and archaic symbols**: While MathML preserves equation structure in supported readers, plain-text fallback and search can lose readability on fractions and archaic currency or unit notation.
5. **Layout ordering and dense indexes in classical books and art catalogs**: Vertical commentary, full-page plate placement, and multi-column indexes can still experience reordering or merged paragraphs after reflow.

For concrete examples of strengths, limitations, and side-by-side output screenshots, see our comparison report: [PDF Craft vs Convertio: Scanned PDF to EPUB Quality, Speed, and Pricing](https://pdfcraft.ai/blog/pdf-craft-vs-convertio-pdf-to-epub/).

## 3. Limitations and Head-to-Head Comparison

* **Scope and limitations**: Scores reflect 121 sampled pages (12 pages per long book, full coverage for books up to 30 pages). Clean sampled pages do not guarantee an error-free full book, and these scores are not character-level accuracy percentages. Scores were produced by Gemini and Deepseek blind-reviewing an anonymous evaluation package and do not represent third-party certification.
* **Head-to-head comparison**: For side-by-side quality, speed, and pricing results against Convertio on the exact same 121 pages, see [PDF Craft vs Convertio: Scanned PDF to EPUB Quality, Speed, and Pricing](https://pdfcraft.ai/blog/pdf-craft-vs-convertio-pdf-to-epub/).

