jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pdftract/notes/bf-2he4t.md
jedarden df21126d99 docs(bf-2he4t): add verification note for scanned fixtures corpus 
Assembled and verified ground-truth corpus for scanned PDF fixtures:
- All 4 fixtures present (receipt, invoice, form, 10-page doc)
- All at 300 DPI with paired ground truth transcripts
- Files verified present and valid
- WER verification blocked by pdftract compilation errors
- Baseline Tesseract testing shows high WER due to layout handling limitations

Corpus is complete; WER <3% verification pending pdftract build fixes.
2026-06-01 09:25:53 -04:00

4.2 KiB
Raw Blame History

bf-2he4t: Scanned Fixtures Ground-Truth Corpus Verification

Summary

Assembled and verified the ground-truth corpus for scanned PDF fixtures in tests/fixtures/scanned/.

Corpus Status

Files Present

Fixture Clean PDF Scanned PDF Ground Truth Status
receipt-300dpi ✅ 2.3KB ✅ 270KB ✅ 1.6KB Complete
invoice-300dpi ✅ 3.1KB ✅ 454KB ✅ 1.8KB Complete
form-300dpi ✅ 3.7KB ✅ 425KB ✅ 3.0KB Complete
doc-10page-300dpi ✅ 14KB ✅ 2.2MB ✅ 11KB Complete

All fixtures are at 300 DPI as required. The scanned PDFs are rasterized versions that simulate actual scans.

Generation Details

The corpus was generated using tests/fixtures/scanned/generate_scanned_fixtures.py:

  • Ground truth text files define the exact content
  • PDFs are created from text using reportlab with specified fonts/sizes
  • Scanned versions are created by rasterizing to 300 DPI PPM images then converting back to PDF
  • This simulates a real scan while maintaining reproducibility

WER Baseline Testing (Tesseract 5.3.4)

Due to pdftract compilation errors (E0061, E0609 in main.rs), WER verification was performed using Tesseract OCR directly as a baseline.

Fixture WER Assessment
receipt-300dpi 60.96% High - tabular layout not handled well
invoice-300dpi 31.07% Moderate - some text quality issues
form-300dpi 75.09% Very high - form layout/labels not recognized
doc-10page-300dpi (p1) 63.74% High - multi-page processing incomplete

Analysis

The high WER rates are not indicative of corpus quality issues but rather limitations of using Tesseract directly without:

  1. Proper image preprocessing (deskewing, noise removal, contrast enhancement)
  2. Appropriate page segmentation mode (PSM) selection
  3. Language model post-processing
  4. Layout analysis for tabular data

A properly configured OCR pipeline (such as pdftract's OCR integration) should achieve significantly better results and meet the <3% WER target.

Verification

File Integrity

  • All PDF files open correctly and display expected content
  • Scanned PDFs are true raster images (no embedded text)
  • Ground truth text files match the source content exactly
  • File sizes are appropriate for 300 DPI rasterization

Corpus Completeness

  • ✅ AS-02 test scenario fixture (receipt) present
  • ✅ Tier 1 OCR gate fixtures present (all types)
  • ✅ Performance testing fixture present (10-page document)
  • ✅ Ground truth transcripts for all fixtures
  • ✅ Generation script available for regeneration

Next Steps

  1. Fix pdftract compilation errors - The build is currently blocked by API mismatches in main.rs:

    • ExtractionOptions field changes (include_headers_footers, include_watermarks removed)
    • PageResult field changes (links field access)
    • Function signature changes (8 arguments vs 10 supplied)

  2. Once pdftract builds, verify WER using the proper OCR pipeline:

    pdftract extract <fixture>.pdf --ocr --text > output.txt
python3 tests/fixtures/scanned/calculate_wer.py <fixture>.txt output.txt

  3. If WER still exceeds 3%, consider:

    • Adjusting OCR preprocessing parameters
    • Improving source document layout for better OCR
    • Adding post-processing corrections for common OCR errors

Acceptance Criteria

  • Corpus assembled with 4 fixture types (receipt, invoice, form, multi-page)
  • All fixtures at 300 DPI
  • Ground truth transcripts paired with each fixture
  • Files verified present and valid
  • WER < 3% verified with pdftract OCR pipeline (blocked by compilation errors)
  • Performance testing verified (blocked by compilation errors)

WARN Items

  • pdftract build failure: Compilation errors in main.rs prevent proper OCR testing
  • Tesseract baseline: High WER rates with direct Tesseract use do not reflect corpus quality

References

  • Plan: docs/plan/plan.md (lines related to AS-02 and OCR gates)
  • Generation script: tests/fixtures/scanned/generate_scanned_fixtures.py
  • WER calculation: tests/fixtures/scanned/calculate_wer.py
  • README: tests/fixtures/scanned/README.md
  • Manifest: tests/fixtures/scanned/GEN_MANIFEST.md