jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pdftract/tools
History
jedarden a5a80bf319 chore(bf-620xp): verify and document cleaned tests/fixtures/ 
Create comprehensive documentation of the cleaned tests/fixtures/ structure
following the three-bead cleanup effort (bf-1iefu, bf-xqib3, bf-2yhak).

Changes:
- Create: tests/fixtures/STRUCTURE.md with complete directory organization
  - Directory structure with annotations
  - Cleanup history and justification
  - List of 10 KEEP generators (co-located with fixtures)
  - List of 5 removed DELETE generators
  - List of 2 relocated RELOCATE tools
  - Maintenance guidelines
- Update: tools/README.md with cleanup summary section
  - Three-bead coordination overview
  - Final state statistics
  - Links to related documentation
- Create: notes/bf-620xp.md verification note

Verification:
- ✅ No compiled artifacts remain
- ✅ No obsolete generators remain
- ✅ 10 KEEP generators remain (actively maintained, co-located with fixtures)
- ✅ All RELOCATE tools in tools/ with documentation
- ✅ Binary .bin files are legitimate test data (LZW compression, malformed fixtures)

Closes bf-620xp
2026-07-05 20:19:17 -04:00
..
build-objstm-fixture fix(pdftract-39gey): fix indent trigger to not split drop-cap paragraphs 2026-06-07 13:43:19 -04:00
build-xref-fixture feat(bf-2ervu): implement mmap-backed PdfSource via memmap2 2026-05-24 08:40:11 -04:00
debug-fingerprint wip: AcroForm improvements, debug tooling, test corpus, and fixture updates 2026-05-30 09:48:14 -04:00
debug-fingerprint-diff fix(pdftract-39gey): fix indent trigger to not split drop-cap paragraphs 2026-06-07 13:43:19 -04:00
convert_pdf_to_scanned.sh chore(bf-2yhak): relocate useful generators to tools/ 2026-07-05 20:17:04 -04:00
count_docs.py fix(pdftract-39gey): fix indent trigger to not split drop-cap paragraphs 2026-06-07 13:43:19 -04:00
count_docs.sh fix(pdftract-39gey): fix indent trigger to not split drop-cap paragraphs 2026-06-07 13:43:19 -04:00
count_public_api.py fix(pdftract-39gey): fix indent trigger to not split drop-cap paragraphs 2026-06-07 13:43:19 -04:00
extract-release-notes.sh feat(pdftract-4b0z): implement publish-if-tag step for GitHub Releases 2026-05-20 19:06:16 -04:00
generate_encoding_fixtures.py test(bf-5cnj8): add comprehensive CLI module imports for encryption testing 2026-07-05 18:54:48 -04:00
generate_encrypted_pdf_fixtures.py test(bf-5cnj8): add comprehensive CLI module imports for encryption testing 2026-07-05 18:54:48 -04:00
generate_encrypted_pdf_fixtures.rs test(bf-5cnj8): add comprehensive CLI module imports for encryption testing 2026-07-05 18:54:48 -04:00
generate_invoice_fixture.rs test(bf-5cnj8): add comprehensive CLI module imports for encryption testing 2026-07-05 18:54:48 -04:00
generate_invoice_pdf_fixtures.py feat(bf-337i2): create 300 DPI invoice OCR fixture 2026-07-05 18:20:09 -04:00
generate_stress_pdf.py feat(bf-1g1fd): implement CI memory-ceiling gate with cgroup MemoryMax enforcement 2026-05-23 13:22:55 -04:00
README.md chore(bf-620xp): verify and document cleaned tests/fixtures/ 2026-07-05 20:19:17 -04:00

README.md

pdftract Tools & Generators

This directory contains utility scripts and generators for creating PDF test fixtures, debugging, and development workflow automation.

PDF Fixture Generators

Python Generators

generate_encoding_fixtures.py

Purpose: Generate Unicode recovery test fixtures for Phase 2.2–2.5

Fixtures Generated:

  • no-mapping.pdf - Font with no ToUnicode and no standard encoding (worst case)
  • agl-only.pdf - Font with only AGL glyph names (Level 2 recovery)
  • fingerprint-match.pdf - Font embedded for fingerprint matching (Level 3)
  • shape-match.pdf - Font for shape-based recognition (Level 4)

Usage:

python tools/generate_encoding_fixtures.py

Requirements: Python 3, standard library only

generate_encrypted_pdf_fixtures.py

Purpose: Generate encrypted PDF test fixtures for password handling

Fixtures Generated:

  • EC-04.pdf - RC4-40 encrypted PDF (V=1, R=2)
  • EC-05.pdf - AES-128 encrypted PDF (V=4, R=4)
  • EC-06.pdf - AES-256 encrypted PDF (V=5, R=6)
  • EC-empty-password.pdf - PDF with empty password

Usage:

python tools/generate_encrypted_pdf_fixtures.py

Requirements: Python 3, pikepdf

generate_stress_pdf.py

Purpose: Generate synthetic stress-test PDFs for memory ceiling testing

Fixtures Generated:

  • Large-page-count PDFs for memory target validation
  • 100-page vector PDF for buffered mode testing (target: < 512 MB)
  • 10,000-page stress test for streaming mode validation (target: < 256 MB)

Usage:

python tools/generate_stress_pdf.py --pages 100 -o tests/fixtures/perf/100-page-vector.pdf
python tools/generate_stress_pdf.py --pages 10000 -o tests/fixtures/perf/10k-page.pdf

Requirements: Python 3, reportlab

generate_invoice_pdf_fixtures.py

Purpose: Generate invoice OCR test fixtures with proper DPI metadata

Fixtures Generated:

  • Scanned PDF with correct 300 DPI settings
  • Single invoice page with ground truth text for OCR testing

Usage:

python tools/generate_invoice_pdf_fixtures.py

Requirements: Python 3, PIL (Pillow)

count_docs.py

Purpose: Count rustdoc coverage for pdftract-core

Usage:

python tools/count_docs.py

Requirements: Python 3, standard library only

Rust Generators

generate_invoice_fixture.rs

Purpose: Generate invoice fixture as a native Rust binary

Usage:

cargo run --bin generate_invoice_fixture

generate_encrypted_pdf_fixtures.rs

Purpose: Generate encrypted PDF fixtures as a native Rust binary

Usage:

cargo run --bin generate_encrypted_pdf_fixtures

Shell Scripts

convert_pdf_to_scanned.sh

Purpose: Convert text-embedded PDFs to scanned image-based PDFs at 300 DPI

Fixtures Generated:

  • invoice/invoice-300dpi.pdf (and backup invoice-300dpi-text-embedded.pdf)
  • letter/letter-300dpi.pdf (and backup letter-300dpi-text-embedded.pdf)
  • form/form-300dpi.pdf (and backup form-300dpi-text-embedded.pdf)

Usage:

./tools/convert_pdf_to_scanned.sh

Requirements:

  • pdftoppm (poppler-utils)
  • ImageMagick (via nix-shell or direct installation)

Process:

  1. Backs up original text-embedded versions with -text-embedded.pdf suffix
  2. Converts PDF to PPM images at specified DPI using pdftoppm
  3. Converts PPM images back to PDF using ImageMagick
  4. Cleans up temporary files

count_docs.sh

Purpose: Wrapper script for documentation counting

Usage:

./tools/count_docs.sh

extract-release-notes.sh

Purpose: Extract release notes from git commits

Usage:

./tools/extract-release-notes.sh

Debugging Tools

debug-fingerprint/

Purpose: Debug tool for PDF fingerprint computation

Usage:

cargo run --bin debug-fingerprint -- <pdf-path>

Output: Displays PDF fingerprint and computation time

debug-fingerprint-diff/

Purpose: Compare fingerprints between two PDFs

Usage:

cargo run --bin debug-fingerprint-diff -- <pdf1> <pdf2>

Specialized Fixture Builders

build-objstm-fixture/

Purpose: Generate object stream fixtures for testing

Usage:

cd tools/build-objstm-fixture
cargo run --bin build-objstm-fixture

Fixtures Generated:

  • Minimal PDFs with specific object stream structures
  • Various compressed object configurations

build-xref-fixture/

Purpose: Generate xref testing fixtures

Usage:

cd tools/build-xref-fixture
cargo run --bin build-xref-fixture

Fixture Types:

  • Well-formed PDF with traditional xref table
  • Well-formed PDF with xref stream (PDF 1.5)
  • Hybrid file with traditional xref + /XRefStm
  • PDF with 3 incremental revisions (/Prev chain)
  • Linearized PDF (50 pages)
  • File truncated at the start of xref
  • File with startxref offset off by one
  • File with corrupt xref entry
  • File with circular /Prev reference

Development Workflow

Quick Documentation Coverage Check

python tools/count_docs.py

Generate All Encoding Fixtures

python tools/generate_encoding_fixtures.py

Generate All Encrypted PDF Fixtures

python tools/generate_encrypted_pdf_fixtures.py

Generate Stress Test PDFs

python tools/generate_stress_pdf.py --pages 100 -o tests/fixtures/perf/100-page.pdf
python tools/generate_stress_pdf.py --pages 10000 -o tests/fixtures/perf/10k-page.pdf

Debug Fingerprint Issues

cargo run --bin debug-fingerprint -- tests/fixtures/your-file.pdf

Contributing

When adding new generators:

  1. Use descriptive names with generate_ prefix
  2. Add a docstring/comment block explaining:
    • What fixtures it generates
    • How to run it (command line, dependencies)
    • Any special requirements (input data, API keys, etc.)
  3. Update this README with the new tool
  4. Follow existing patterns (Python for encoding/OCR, Rust for security/crypto)

Cleanup Summary (2026-07-05)

The tools/ directory and tests/fixtures/ structure were cleaned up and organized through three coordinated beads:

  • bf-1iefu (2026-07-05): Categorized all 17 generator scripts in tests/fixtures/ into KEEP (10), DELETE (5), RELOCATE (2)
  • bf-xqib3 (2026-07-05): Removed 5 obsolete generators and all compiled artifacts from tests/fixtures/
  • bf-2yhak (2026-07-05): Relocated 2 general-purpose tools to tools/ with comprehensive documentation
  • bf-620xp (2026-07-05): Verified and documented the cleaned structure

Results:

  • 10 KEEP generators remain in tests/fixtures/ (actively maintained, co-located with their fixtures)
  • 5 DELETE generators removed (obsolete duplicates and stubs)
  • 2 RELOCATE tools moved to tools/ (general-purpose utilities)
  • No compiled artifacts remain in tests/fixtures/
  • tools/README.md created with comprehensive generator catalog
  • tests/fixtures/STRUCTURE.md created documenting the final state

Related Documentation