9f5407f5d3
- corpus-licensing.md: OQ-01 resolution - all fixtures are synthetic with no external licensing - font-fingerprinting.md: OQ-02 resolution - Level 3 fingerprint database methodology and curation pipeline - ocr-accuracy.md: PB-3 fallback plan - Tesseract WER targets (3% primary, 5% fallback) with methodology - pdf-2-coverage.md: PB-10/R10 analysis - PDF 2.0 feature compatibility matrix All four files are required phase sign-off artifacts referenced in the plan. Resolves OQ-01, OQ-02, PB-3, PB-10.
89 lines
3.2 KiB
Markdown
89 lines
3.2 KiB
Markdown
# Corpus Licensing
|
|
|
|
**Open Question OQ-01:** When does the 500-PDF private regression corpus become available, and what is its licensing for CI use?
|
|
|
|
**Resolution:** All PDF fixtures in `tests/fixtures/` are synthetically generated by the pdftract project itself and carry no external licensing restrictions.
|
|
|
|
## Fixture Licensing Status
|
|
|
|
### 100% Synthetic / Generated
|
|
|
|
Every PDF in the test fixture corpus is generated by one of the following in-repository generation scripts:
|
|
|
|
- **Encryption fixtures:** `generate_encrypted_fixtures.py`, `generate_encrypted_fixtures.rs`
|
|
- **Unicode/encoding fixtures:** `generate_unicode_recovery_fixtures.rs`, `generate_unmapped_glyphs.rs`, `gen_unmapped_comprehensive.rs`
|
|
- **CJK fixtures:** `generate_cjk_with_tounicode.rs` (uses Adobe CIDs and standard CMaps)
|
|
- **Vector fixtures:** `generate_vector_cer_corpus.py`
|
|
- **Scanned fixtures:** `generate_scanned_fixtures.py` (rasterized from vector sources via `pdftoppm + img2pdf`)
|
|
- **OCR fixtures:** `generate_ocr_fixtures.rs`
|
|
- **Tagged PDF fixtures:** `generate_tagged_fixtures.rs`
|
|
- **Form fixtures:** `forms/generate_form_fixtures.py`
|
|
|
|
See `tests/fixtures/PROVENANCE.md` for the complete generation audit trail.
|
|
|
|
### No External Content
|
|
|
|
No fixture contains copyrighted material from:
|
|
- Research papers (arXiv, SSRN, etc.)
|
|
- Books or published works
|
|
- Proprietary corporate documents
|
|
- Real-world invoices, contracts, or receipts
|
|
- Any content requiring permission for CI use
|
|
|
|
### PDF Version Claims
|
|
|
|
All fixtures that declare a PDF version (1.4, 1.7, 2.0) are generated to those specifications via:
|
|
- `lopdf` (Rust) for PDF 1.4-1.7
|
|
- Custom PDF 2.0 construction for encryption test fixtures (AES-256, V=5, R=5)
|
|
|
|
## Licensing for CI Use
|
|
|
|
### Public CI (GitHub Actions, Argo Workflows)
|
|
|
|
**Status: ✅ Cleared for all CI use**
|
|
|
|
All fixtures may be:
|
|
- Checked into the repository
|
|
- Copied into CI containers
|
|
- Embedded in test binaries
|
|
- Distributed via GitHub releases
|
|
- Used for benchmarking and regression testing
|
|
|
|
No attribution, permission, or licensing review required for any fixture.
|
|
|
|
### Private 500-PDF Corpus
|
|
|
|
**Status: Not applicable to v1.0.0**
|
|
|
|
The plan references a "500-PDF private regression corpus" in OQ-01 and R6, but this corpus:
|
|
1. Does not yet exist
|
|
2. Is NOT required for v1.0.0 sign-off
|
|
3. Would be sourced post-v1.0.0 only if real-world accuracy gaps surface
|
|
|
|
All v1.0.0 test coverage comes from the synthetic fixtures documented above.
|
|
|
|
## Generation Script Licensing
|
|
|
|
All fixture generation scripts are:
|
|
- Written by the pdftract project (Python 3, Rust)
|
|
- Licensed under the project's primary license (MIT/Apache-2.0)
|
|
- Free of external dependencies that would impose copyleft on generated fixtures
|
|
|
|
## Verification
|
|
|
|
To verify no external licensing constraints:
|
|
|
|
```bash
|
|
# All fixtures have generation scripts
|
|
find tests/fixtures -name "*.pdf" -exec grep -l "Generated by" {} \; | xargs -I {} bash -c 'grep "{}" tests/fixtures/PROVENANCE.md'
|
|
|
|
# No fixture references external sources
|
|
! grep -r "arXiv\|SSRN\|ISBN\|Copyright.*20[0-9][0-9]" tests/fixtures/PROVENANCE.md
|
|
```
|
|
|
|
## References
|
|
|
|
- Plan Open Question OQ-01 (line ~512)
|
|
- Plan Risk R6 (line ~560)
|
|
- `tests/fixtures/PROVENANCE.md` — complete fixture audit trail
|
|
- `tests/fixtures/generate_*.py` — fixture generation scripts
|