jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pdftract/tests/fixtures/scanned/GEN_MANIFEST.md
jedarden 3d795a2d11 feat(bf-2he4t): assemble scanned fixtures corpus with ground-truth transcripts 
Created tests/fixtures/scanned/ directory structure for WER gate testing:

- README.md: Corpus overview and WER targets (<3% on clean 300-DPI scans)
- GEN_MANIFEST.md: Fixture specifications and generation checklist
- receipt/receipt-300dpi.txt: Ground truth for AS-02 test scenario (37 lines)
- documents/invoice-300dpi.txt: Business invoice ground truth (55 lines)
- documents/form-300dpi.txt: Employment application form (78 lines)
- multi-page/doc-10page-300dpi.txt: Performance fixture (255 lines, 10 pages)

Generation tools:
- generate_scanned_fixtures.py: Python script for PDF generation
- generate_scanned_fixtures.rs: Rust alternative for fixture metadata
- calculate_wer.py: WER/CER calculation utility for OCR validation

Test stub:
- wer_gate_stub.rs: Placeholder for WER gate tests (marked #[ignore])

Total ground-truth content: 425 lines across 4 fixtures

Next steps:
1. Generate PDFs from ground truth using generation script
2. Verify WER < 3% on generated fixtures
3. Enable WER gate tests

Closes bf-2he4t
2026-06-01 09:25:53 -04:00

3.7 KiB
Raw Blame History

Scanned Fixtures Generation Manifest

This document tracks the generation status and specifications for all scanned fixtures.

Fixture Specifications

receipt-300dpi

  • Purpose: AS-02 test scenario, basic receipt OCR
  • Ground Truth: receipt/receipt-300dpi.txt
  • Target PDF: receipt/receipt-300dpi.pdf
  • Specifications:
    • Font: Helvetica 10pt
    • Page size: Letter (8.5" x 11")
    • Margins: 0.5" all sides
    • Line spacing: 14pt
    • Content: Supermarket receipt with items, prices, totals
  • WER Target: < 3%
  • Status: Ground truth created, PDF generation pending

invoice-300dpi

  • Purpose: Business document OCR testing
  • Ground Truth: documents/invoice-300dpi.txt
  • Target PDF: documents/invoice-300dpi.pdf
  • Specifications:
    • Font: Helvetica 11pt
    • Page size: Letter (8.5" x 11")
    • Margins: 0.75" all sides
    • Line spacing: 16pt
    • Content: Service invoice with line items, totals, payment terms
  • WER Target: < 3%
  • Status: Ground truth created, PDF generation pending

form-300dpi

  • Purpose: Form structure OCR testing
  • Ground Truth: documents/form-300dpi.txt
  • Target PDF: documents/form-300dpi.pdf
  • Specifications:
    • Font: Helvetica 11pt
    • Page size: Letter (8.5" x 11")
    • Margins: 0.75" all sides
    • Line spacing: 18pt
    • Content: Employment application form with fields and checkboxes
  • WER Target: < 3%
  • Status: Ground truth created, PDF generation pending

doc-10page-300dpi

  • Purpose: Multi-page performance testing
  • Ground Truth: multi-page/doc-10page-300dpi.txt
  • Target PDF: multi-page/doc-10page-300dpi.pdf
  • Specifications:
    • Font: Times-Roman 12pt
    • Page size: Letter (8.5" x 11")
    • Margins: 1" left/right, 0.75" top/bottom
    • Line spacing: 18pt
    • Content: 10 pages with diverse content types
    • Page markers: "Page N:" format for explicit page breaks
  • WER Target: < 3% average, no page > 5%
  • Performance Target: < 30 seconds on 4-core CI
  • Status: Ground truth created, PDF generation pending

Generation Checklist

For each fixture, complete these steps:

  1. Verify ground truth .txt file exists and is complete
  2. Run generation script: python3 generate_scanned_fixtures.py <fixture-name>
  3. Verify generated PDF is readable and displays correctly
  4. Test OCR extraction: pdftract extract <pdf> --ocr --text
  5. Compute WER against ground truth
  6. Update this manifest with WER result
  7. If WER < 3%, mark as PASS; otherwise, investigate

WER Results

To be populated after PDF generation and testing:

Fixture WER Pass/Fail Notes
receipt-300dpi TBD TBD -
invoice-300dpi TBD TBD -
form-300dpi TBD TBD -
doc-10page-300dpi TBD TBD Per-page breakdown needed

Dependencies

Required for PDF Generation

  • Python 3.8+
  • reportlab: pip3 install reportlab
  • (Optional) Pillow: pip3 install Pillow
  • (Optional) img2pdf: pip3 install img2pdf

Required for Scan Simulation

  • poppler-utils: apt-get install poppler-utils (provides pdftoppm)

Required for WER Calculation

  • jiwer: pip3 install jiwer
  • Or: Python implementation for basic WER

Manual Generation Alternative

If the generation script fails, manual generation steps:

  1. Create a new document in LibreOffice/Word
  2. Copy ground truth text from .txt file
  3. Set font to Helvetica/Arial at specified size
  4. Set page size to Letter
  5. Set margins as specified
  6. Export to PDF
  7. (Optional) Use a scanner or PDF printer to simulate scan at 300 DPI

Related Beads

  • bf-2he4t: Initial corpus assembly (this bead)
  • (Future) WER gate implementation
  • (Future) AS-02 test scenario implementation