There is no off-the-shelf benchmark designed specifically for converting a full scanned PDF into a readable, reflowable EPUB. Combining four established public standards from document parsing and digital publishing, we built a 6-dimension evaluation rubric across 121 sampled pages. On November 21, 2026, two independent reviewers (Gemini and Deepseek) blind-scored PDF Craft's production outputs under this rubric: its weighted overall score landed between 78.6 and 80.0 out of 100.
1. Evaluation Methodology
This benchmark tests a single, clearly bounded task: converting image-based scanned PDFs into original-language, reflowable EPUBs (without translation). Conversion speed and pricing are measured separately and never blended into the quality score.
Public Standards Referenced
| Public Reference | What We Adopted | What It Does Not Cover |
|---|---|---|
| OCR-D Evaluation Specification | Separating text recognition accuracy from layout structure | Focuses on recognition metrics rather than reflowed reading quality |
| OmniDocBench (CVPR 2025) | Scoring body text, tables, formulas, and reading order independently | Built for document parsing rather than EPUB navigation and cross-page footnotes |
| olmOCR-Bench (Ai2) | Breaking page quality into concrete, verifiable checkpoints | Centered on single-page PDFs rather than book-length continuity |
| EPUBCheck (W3C / DAISY) | Automated validation of EPUB packaging, XML, assets, and links | Checks technical validity, not OCR or content fidelity |
Note: These references document the origin of our evaluation design. Dimension weights and scoring tiers were set for full-book reading and do not imply endorsement by those organizations.
Test Set and Sampling Rules (9 Books, 121 Sampled Pages)
The main test set locks 8 scanned PDFs across 5 categories (552 pages total), verified by SHA-256 hashes. One additional 88-page native digital art catalog is included purely as a diagnostic control to separate OCR errors from layout-parsing errors, and is excluded from the overall score. Sampling is fixed: books up to 30 pages are reviewed in full; longer books sample 4 pages each from the beginning, middle, and end (12 pages), totaling 121 sampled pages:
| Category (Weight) | Test Book | Total Pages | Sampled Pages | Key Challenges |
|---|---|---|---|---|
| Literature (25%) | Border Town (real Chinese scan) | 187 | 12 | Long-form fiction, dialogue, chapter TOC, illustrations, vertical calligraphed cover text |
| Literature (25%) | Alice in Wonderland Coloring Book | 20 | 20 (all) | Illustrated English reader with full-page line drawings |
| Humanities (25%) | Fundamentals of Psychoanalytic Technique, Ch. 1 (Chinese) | 24 | 24 (all) | Academic prose, small footnotes, translator notes, mixed English/French terms |
| Humanities (25%) | Baopuzi Inner Chapters, Vol. 1 (Classical Chinese) | 10 | 10 (all) | Traditional vertical classical text with dense commentary |
| STEM (25%) | Calculus Made Easy | 278 | 12 | Aged scan with bleed-through, heavy inline/block math, pre-decimal British currency, tables |
| STEM (25%) | MapVision (CVPR 2024 technical report) | 5 | 5 (all) | Two-column English paper, multi-column tables, inline math, references |
| Technical Manual (15%) | HP LaserJet Pro M404dn Paper Trays chapter | 16 | 16 (all) | Numbered procedures, orientation tables, diagrams, caution/note blocks |
| Art Catalog (10%) | Roman Mosaics (Italy scanned excerpt) | 12 | 12 (all) | Full-page plates, artwork metadata, captions, grouped endnotes |
| Art Catalog (control, excluded) | Roman Mosaics (native digital PDF) | 88 | 12 | Diagnostic baseline separating scan OCR issues from layout parsing |
Six Dimensions, Four-Tier Scale, and Hard Rules
The first five dimensions break down into 16 concrete checkpoints inspected page by page against the source images and the unpacked EPUB XHTML/MathML markup. The sixth dimension is computed automatically by validation scripts.
| Dimension | Weight | What Is Checked |
|---|---|---|
| Text Accuracy | 30% | Word-for-word body accuracy, proper nouns and numbers, punctuation and spacing |
| Completeness | 20% | Missing paragraphs, cross-page and cross-column sentence continuity, retained figures and notes |
| Structure & Navigation | 20% | Reading order, heading detection and hierarchy, usable table of contents, two-way footnote links |
| Formulas & Tables | 15% | HTML table structure and headers, math structure (such as MathML), plain-text fallback readability |
| Layout & Readability | 10% | Reflow integrity (no clipping, overlap, collapsed indexes, or stray gaps), visual hierarchy and caption pairing |
| File Validity | 5% | Valid EPUB container, well-formed XML, intact spine and internal asset references (automated) |
Hard scoring rules:
- Four fixed tiers: Correct =
1, Minor issue =0.9(readable without changing meaning), Major issue =0.5(alters meaning or breaks usability), Critical or missing =0. No intermediate values are allowed; the worst issue within a checkpoint determines its tier; each defect is penalized only once in its most specific dimension. - Automatic zero for critical failures: A corrupted EPUB that fails to open, or an image-wrapped EPUB with no reflowable text, scores 0 for that book.
- Strict N/A handling: A checkpoint is marked N/A (with its weight redistributed) only when the sampled source pages genuinely lack that element. Omissions by the converter never qualify as N/A.
- Verifiable and blind: Every score requires page numbers plus source and output quotes. Totals are calculated strictly by script, and all vendor metadata is stripped from the anonymous evaluation package.
2. PDF Craft Test Results (Blind Dual-Review)
All 9 source files were converted once on pdfcraft.ai in November 2026 (EPUB output, footnote refinement enabled, translation off, no retries or manual edits). On November 21, 2026, Gemini and Deepseek independently blind-scored the anonymized outputs against the same 121 sampled pages.
Overall Scores, Speed, and Cost
| Metric | Gemini Blind Review | Deepseek Blind Review |
|---|---|---|
| Weighted Overall Score | 78.6 | 80.0 |
| Text Accuracy (30%) | 72.3 | 76.3 |
| Completeness (20%) | 79.3 | 82.7 |
| Structure & Navigation (20%) | 74.7 | 72.2 |
| Formulas & Tables (15%) | 71.1 | 71.1 |
| Layout & Readability (10%) | 84.4 | 81.7 |
| File Validity (5%) | 100.0 | 100.0 |
- Conversion speed (main weakness): The 8 main scanned books (552 pages) took 132.1 minutes (640 pages including the native control took 163.1 minutes), averaging 24.0–25.5 minutes per 100 pages. Long books take time; the 278-page Calculus Made Easy took 77.2 minutes.
- Conversion cost: PDF Craft uses pay-as-you-go page pricing with no subscription. Standard conversion is $0.30 per 100 pages (about $1.92 for 640 pages), and refined mode is $1.10 per 100 pages (about $7.04 for 640 pages). See the Recharge page for current rates.
Per-Book Scores and Conversion Time
| Book | Category | Pages | Gemini Blind Review | Deepseek Blind Review | Time |
|---|---|---|---|---|---|
| Border Town | Literature | 187 | 67.6 | 74.7 | 31.0 min |
| Alice Coloring Book | Literature | 20 | 94.9 | 95.7 | 3.1 min |
| Psychoanalytic Technique, Ch. 1 | Humanities | 24 | 88.0 | 87.6 | 6.0 min |
| Baopuzi (Vol. 1) | Humanities | 10 | 61.6 | 60.6 | 2.4 min |
| Calculus Made Easy | STEM | 278 | 77.8 | 80.5 | 77.2 min |
| MapVision (CVPR paper) | STEM | 5 | 74.5 | 71.2 | 2.4 min |
| HP Paper Trays Manual | Manual | 16 | 92.5 | 96.3 | 5.0 min |
| Roman Mosaics (scanned) | Art Catalog | 12 | 67.0 | 67.6 | 5.0 min |
| Roman Mosaics (native control, excluded) | Art Catalog | 88 | 68.8 | 69.4 | 31.0 min |
Single-book scores ranged from 60.6 to 96.3. Both blind reviews agreed closely on ranking: illustrated English books (Alice), technical manuals (HP), and modern academic chapters (Psychoanalytic Technique) scored highest (87.6–96.3), while vertical classical Chinese (Baopuzi) and endnote-heavy art catalogs scored lowest (60.6–69.4).
Strengths and Verified Weaknesses
What worked well:
- Clean EPUB packaging and real HTML tables: Every EPUB passed automated packaging and link validation (100/100). Data tables across Calculus Made Easy, MapVision, and the HP manual stayed structured as HTML
<table>elements instead of flattened text. - MathML formulas and modern footnotes: Fractions, roots, and superscripts in Calculus Made Easy were preserved in standard MathML; 21 of 22 footnotes in Psychoanalytic Technique had working two-way jump links.
- High fidelity on illustrated readers and manuals: Alice (94.9–95.7) and the HP manual (92.5–96.3) preserved body text, steps, proper nouns, and figures accurately.
Key areas for improvement (grouped by issue type):
- Partially missing anchors on complex notes: Two-way note links can still be missing in vertical classical commentary, endnote-heavy art catalogs, and specially marked translator notes.
- Inconsistent chapter TOC and rare proper nouns in long Chinese fiction: Some Chinese numeral chapter headings are not grouped cleanly into the table of contents, and uncommon place names or proper nouns may still need manual review.
- Two-column breaks and table-header subscripts: Two-column papers can occasionally drop text across column boundaries, and subscripts in dense table headers may degrade or collide.
- Plain-text math fallback and archaic symbols: While MathML preserves equation structure in supported readers, plain-text fallback and search can lose readability on fractions and archaic currency or unit notation.
- Layout ordering and dense indexes in classical books and art catalogs: Vertical commentary, full-page plate placement, and multi-column indexes can still experience reordering or merged paragraphs after reflow.
For concrete examples of strengths, limitations, and side-by-side output screenshots, see our comparison report: PDF Craft vs Convertio: Scanned PDF to EPUB Quality, Speed, and Pricing.
3. Limitations and Head-to-Head Comparison
- Scope and limitations: Scores reflect 121 sampled pages (12 pages per long book, full coverage for books up to 30 pages). Clean sampled pages do not guarantee an error-free full book, and these scores are not character-level accuracy percentages. Scores were produced by Gemini and Deepseek blind-reviewing an anonymous evaluation package and do not represent third-party certification.
- Head-to-head comparison: For side-by-side quality, speed, and pricing results against Convertio on the exact same 121 pages, see PDF Craft vs Convertio: Scanned PDF to EPUB Quality, Speed, and Pricing.

