jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pdftract/docs/notes/corpus-licensing.md
jedarden 9f5407f5d3 docs(bf-1hya5): add four missing documentation files for phase sign-off 
- corpus-licensing.md: OQ-01 resolution - all fixtures are synthetic with no external licensing
- font-fingerprinting.md: OQ-02 resolution - Level 3 fingerprint database methodology and curation pipeline
- ocr-accuracy.md: PB-3 fallback plan - Tesseract WER targets (3% primary, 5% fallback) with methodology
- pdf-2-coverage.md: PB-10/R10 analysis - PDF 2.0 feature compatibility matrix

All four files are required phase sign-off artifacts referenced in the plan.
Resolves OQ-01, OQ-02, PB-3, PB-10.
2026-07-05 12:45:35 -04:00

3.2 KiB
Raw Blame History

Corpus Licensing

Open Question OQ-01: When does the 500-PDF private regression corpus become available, and what is its licensing for CI use?

Resolution: All PDF fixtures in tests/fixtures/ are synthetically generated by the pdftract project itself and carry no external licensing restrictions.

Fixture Licensing Status

100% Synthetic / Generated

Every PDF in the test fixture corpus is generated by one of the following in-repository generation scripts:

  • Encryption fixtures: generate_encrypted_fixtures.py, generate_encrypted_fixtures.rs
  • Unicode/encoding fixtures: generate_unicode_recovery_fixtures.rs, generate_unmapped_glyphs.rs, gen_unmapped_comprehensive.rs
  • CJK fixtures: generate_cjk_with_tounicode.rs (uses Adobe CIDs and standard CMaps)
  • Vector fixtures: generate_vector_cer_corpus.py
  • Scanned fixtures: generate_scanned_fixtures.py (rasterized from vector sources via pdftoppm + img2pdf)
  • OCR fixtures: generate_ocr_fixtures.rs
  • Tagged PDF fixtures: generate_tagged_fixtures.rs
  • Form fixtures: forms/generate_form_fixtures.py

See tests/fixtures/PROVENANCE.md for the complete generation audit trail.

No External Content

No fixture contains copyrighted material from:

  • Research papers (arXiv, SSRN, etc.)
  • Books or published works
  • Proprietary corporate documents
  • Real-world invoices, contracts, or receipts
  • Any content requiring permission for CI use

PDF Version Claims

All fixtures that declare a PDF version (1.4, 1.7, 2.0) are generated to those specifications via:

  • lopdf (Rust) for PDF 1.4-1.7
  • Custom PDF 2.0 construction for encryption test fixtures (AES-256, V=5, R=5)

Licensing for CI Use

Public CI (GitHub Actions, Argo Workflows)

Status: ✅ Cleared for all CI use

All fixtures may be:

  • Checked into the repository
  • Copied into CI containers
  • Embedded in test binaries
  • Distributed via GitHub releases
  • Used for benchmarking and regression testing

No attribution, permission, or licensing review required for any fixture.

Private 500-PDF Corpus

Status: Not applicable to v1.0.0

The plan references a "500-PDF private regression corpus" in OQ-01 and R6, but this corpus:

  1. Does not yet exist
  2. Is NOT required for v1.0.0 sign-off
  3. Would be sourced post-v1.0.0 only if real-world accuracy gaps surface

All v1.0.0 test coverage comes from the synthetic fixtures documented above.

Generation Script Licensing

All fixture generation scripts are:

  • Written by the pdftract project (Python 3, Rust)
  • Licensed under the project's primary license (MIT/Apache-2.0)
  • Free of external dependencies that would impose copyleft on generated fixtures

Verification

To verify no external licensing constraints:

# All fixtures have generation scripts
find tests/fixtures -name "*.pdf" -exec grep -l "Generated by" {} \; | xargs -I {} bash -c 'grep "{}" tests/fixtures/PROVENANCE.md'

# No fixture references external sources
! grep -r "arXiv\|SSRN\|ISBN\|Copyright.*20[0-9][0-9]" tests/fixtures/PROVENANCE.md

References

  • Plan Open Question OQ-01 (line ~512)
  • Plan Risk R6 (line ~560)
  • tests/fixtures/PROVENANCE.md — complete fixture audit trail
  • tests/fixtures/generate_*.py — fixture generation scripts