Skip to main content

Scanned PDF to EPUB Benchmark: Methodology and PDF Craft Test Results

Our 6-dimension evaluation framework for scanned PDF to reflowable EPUB conversion across 121 sampled pages, plus PDF Craft's blind-reviewed scores (78.6–80.0), speed, and known limitations.

New user gift

Test EPUB conversion on your own scanned PDF

Upload a few representative pages with footnotes, columns, or formulas and inspect the resulting EPUB.

Free starter credit

$1

About 330 PDF pages or 90K EPUB words

No card required, credited right after sign-up

Convert PDF to EPUB

There is no off-the-shelf benchmark designed specifically for converting a full scanned PDF into a readable, reflowable EPUB. Combining four established public standards from document parsing and digital publishing, we built a 6-dimension evaluation rubric across 121 sampled pages. On November 21, 2026, two independent reviewers (Gemini and Deepseek) blind-scored PDF Craft's production outputs under this rubric: its weighted overall score landed between 78.6 and 80.0 out of 100.

1. Evaluation Methodology

This benchmark tests a single, clearly bounded task: converting image-based scanned PDFs into original-language, reflowable EPUBs (without translation). Conversion speed and pricing are measured separately and never blended into the quality score.

Public Standards Referenced

Public ReferenceWhat We AdoptedWhat It Does Not Cover
OCR-D Evaluation SpecificationSeparating text recognition accuracy from layout structureFocuses on recognition metrics rather than reflowed reading quality
OmniDocBench (CVPR 2025)Scoring body text, tables, formulas, and reading order independentlyBuilt for document parsing rather than EPUB navigation and cross-page footnotes
olmOCR-Bench (Ai2)Breaking page quality into concrete, verifiable checkpointsCentered on single-page PDFs rather than book-length continuity
EPUBCheck (W3C / DAISY)Automated validation of EPUB packaging, XML, assets, and linksChecks technical validity, not OCR or content fidelity

Note: These references document the origin of our evaluation design. Dimension weights and scoring tiers were set for full-book reading and do not imply endorsement by those organizations.

Test Set and Sampling Rules (9 Books, 121 Sampled Pages)

The main test set locks 8 scanned PDFs across 5 categories (552 pages total), verified by SHA-256 hashes. One additional 88-page native digital art catalog is included purely as a diagnostic control to separate OCR errors from layout-parsing errors, and is excluded from the overall score. Sampling is fixed: books up to 30 pages are reviewed in full; longer books sample 4 pages each from the beginning, middle, and end (12 pages), totaling 121 sampled pages:

Category (Weight)Test BookTotal PagesSampled PagesKey Challenges
Literature (25%)Border Town (real Chinese scan)18712Long-form fiction, dialogue, chapter TOC, illustrations, vertical calligraphed cover text
Literature (25%)Alice in Wonderland Coloring Book2020 (all)Illustrated English reader with full-page line drawings
Humanities (25%)Fundamentals of Psychoanalytic Technique, Ch. 1 (Chinese)2424 (all)Academic prose, small footnotes, translator notes, mixed English/French terms
Humanities (25%)Baopuzi Inner Chapters, Vol. 1 (Classical Chinese)1010 (all)Traditional vertical classical text with dense commentary
STEM (25%)Calculus Made Easy27812Aged scan with bleed-through, heavy inline/block math, pre-decimal British currency, tables
STEM (25%)MapVision (CVPR 2024 technical report)55 (all)Two-column English paper, multi-column tables, inline math, references
Technical Manual (15%)HP LaserJet Pro M404dn Paper Trays chapter1616 (all)Numbered procedures, orientation tables, diagrams, caution/note blocks
Art Catalog (10%)Roman Mosaics (Italy scanned excerpt)1212 (all)Full-page plates, artwork metadata, captions, grouped endnotes
Art Catalog (control, excluded)Roman Mosaics (native digital PDF)8812Diagnostic baseline separating scan OCR issues from layout parsing

Six Dimensions, Four-Tier Scale, and Hard Rules

The first five dimensions break down into 16 concrete checkpoints inspected page by page against the source images and the unpacked EPUB XHTML/MathML markup. The sixth dimension is computed automatically by validation scripts.

DimensionWeightWhat Is Checked
Text Accuracy30%Word-for-word body accuracy, proper nouns and numbers, punctuation and spacing
Completeness20%Missing paragraphs, cross-page and cross-column sentence continuity, retained figures and notes
Structure & Navigation20%Reading order, heading detection and hierarchy, usable table of contents, two-way footnote links
Formulas & Tables15%HTML table structure and headers, math structure (such as MathML), plain-text fallback readability
Layout & Readability10%Reflow integrity (no clipping, overlap, collapsed indexes, or stray gaps), visual hierarchy and caption pairing
File Validity5%Valid EPUB container, well-formed XML, intact spine and internal asset references (automated)

Hard scoring rules:

  • Four fixed tiers: Correct = 1, Minor issue = 0.9 (readable without changing meaning), Major issue = 0.5 (alters meaning or breaks usability), Critical or missing = 0. No intermediate values are allowed; the worst issue within a checkpoint determines its tier; each defect is penalized only once in its most specific dimension.
  • Automatic zero for critical failures: A corrupted EPUB that fails to open, or an image-wrapped EPUB with no reflowable text, scores 0 for that book.
  • Strict N/A handling: A checkpoint is marked N/A (with its weight redistributed) only when the sampled source pages genuinely lack that element. Omissions by the converter never qualify as N/A.
  • Verifiable and blind: Every score requires page numbers plus source and output quotes. Totals are calculated strictly by script, and all vendor metadata is stripped from the anonymous evaluation package.

2. PDF Craft Test Results (Blind Dual-Review)

All 9 source files were converted once on pdfcraft.ai in November 2026 (EPUB output, footnote refinement enabled, translation off, no retries or manual edits). On November 21, 2026, Gemini and Deepseek independently blind-scored the anonymized outputs against the same 121 sampled pages.

Overall Scores, Speed, and Cost

MetricGemini Blind ReviewDeepseek Blind Review
Weighted Overall Score78.680.0
Text Accuracy (30%)72.376.3
Completeness (20%)79.382.7
Structure & Navigation (20%)74.772.2
Formulas & Tables (15%)71.171.1
Layout & Readability (10%)84.481.7
File Validity (5%)100.0100.0
  • Conversion speed (main weakness): The 8 main scanned books (552 pages) took 132.1 minutes (640 pages including the native control took 163.1 minutes), averaging 24.0–25.5 minutes per 100 pages. Long books take time; the 278-page Calculus Made Easy took 77.2 minutes.
  • Conversion cost: PDF Craft uses pay-as-you-go page pricing with no subscription. Standard conversion is $0.30 per 100 pages (about $1.92 for 640 pages), and refined mode is $1.10 per 100 pages (about $7.04 for 640 pages). See the Recharge page for current rates.

Per-Book Scores and Conversion Time

BookCategoryPagesGemini Blind ReviewDeepseek Blind ReviewTime
Border TownLiterature18767.674.731.0 min
Alice Coloring BookLiterature2094.995.73.1 min
Psychoanalytic Technique, Ch. 1Humanities2488.087.66.0 min
Baopuzi (Vol. 1)Humanities1061.660.62.4 min
Calculus Made EasySTEM27877.880.577.2 min
MapVision (CVPR paper)STEM574.571.22.4 min
HP Paper Trays ManualManual1692.596.35.0 min
Roman Mosaics (scanned)Art Catalog1267.067.65.0 min
Roman Mosaics (native control, excluded)Art Catalog8868.869.431.0 min

Single-book scores ranged from 60.6 to 96.3. Both blind reviews agreed closely on ranking: illustrated English books (Alice), technical manuals (HP), and modern academic chapters (Psychoanalytic Technique) scored highest (87.6–96.3), while vertical classical Chinese (Baopuzi) and endnote-heavy art catalogs scored lowest (60.6–69.4).

Strengths and Verified Weaknesses

What worked well:

  1. Clean EPUB packaging and real HTML tables: Every EPUB passed automated packaging and link validation (100/100). Data tables across Calculus Made Easy, MapVision, and the HP manual stayed structured as HTML <table> elements instead of flattened text.
  2. MathML formulas and modern footnotes: Fractions, roots, and superscripts in Calculus Made Easy were preserved in standard MathML; 21 of 22 footnotes in Psychoanalytic Technique had working two-way jump links.
  3. High fidelity on illustrated readers and manuals: Alice (94.9–95.7) and the HP manual (92.5–96.3) preserved body text, steps, proper nouns, and figures accurately.

Key areas for improvement (grouped by issue type):

  1. Partially missing anchors on complex notes: Two-way note links can still be missing in vertical classical commentary, endnote-heavy art catalogs, and specially marked translator notes.
  2. Inconsistent chapter TOC and rare proper nouns in long Chinese fiction: Some Chinese numeral chapter headings are not grouped cleanly into the table of contents, and uncommon place names or proper nouns may still need manual review.
  3. Two-column breaks and table-header subscripts: Two-column papers can occasionally drop text across column boundaries, and subscripts in dense table headers may degrade or collide.
  4. Plain-text math fallback and archaic symbols: While MathML preserves equation structure in supported readers, plain-text fallback and search can lose readability on fractions and archaic currency or unit notation.
  5. Layout ordering and dense indexes in classical books and art catalogs: Vertical commentary, full-page plate placement, and multi-column indexes can still experience reordering or merged paragraphs after reflow.

For concrete examples of strengths, limitations, and side-by-side output screenshots, see our comparison report: PDF Craft vs Convertio: Scanned PDF to EPUB Quality, Speed, and Pricing.

3. Limitations and Head-to-Head Comparison

  • Scope and limitations: Scores reflect 121 sampled pages (12 pages per long book, full coverage for books up to 30 pages). Clean sampled pages do not guarantee an error-free full book, and these scores are not character-level accuracy percentages. Scores were produced by Gemini and Deepseek blind-reviewing an anonymous evaluation package and do not represent third-party certification.
  • Head-to-head comparison: For side-by-side quality, speed, and pricing results against Convertio on the exact same 121 pages, see PDF Craft vs Convertio: Scanned PDF to EPUB Quality, Speed, and Pricing.