Commit graph

7 commits

Author SHA1 Message Date
jedarden
dfc94f33d5 chore(bf-2yhak): relocate useful generators to tools/
Relocate convert_to_scanned.sh from tests/fixtures/scanned/ to tools/
with comprehensive documentation header. Delete incomplete
regenerate.sh stub from tests/fixtures/grep-corpus/. Create
tools/README.md cataloging all 18 generators with usage examples.

Changes:
- Move: tests/fixtures/scanned/convert_to_scanned.sh → tools/convert_pdf_to_scanned.sh
- Delete: tests/fixtures/grep-corpus/regenerate.sh (incomplete stub)
- Create: tools/README.md (comprehensive generator catalog)
- Enhance: convert_pdf_to_scanned.sh with detailed documentation

Closes bf-2yhak
2026-07-05 20:17:04 -04:00
jedarden
bbbb7640d4 chore(bf-xqib3): remove obsolete generators and compiled artifacts from tests/fixtures/
Remove 5 DELETE-category generator files identified in bf-1iefu:
- tests/fixtures/scanned/generate_scanned_fixtures.rs (Rust stub, use Python version)
- tests/fixtures/security/generate_sensitive_fixture.py (duplicate, Rust version preferred)
- tests/fixtures/encoding/generate_unmapped_glyphs.rs (superseded by Python version)
- tests/fixtures/malformed/gen-bomb-10k-2g.sh (superseded by Python version)
- tests/fixtures/forms/generate_form_fixtures.d (orphaned Makefile dependency)

Verification: No compiled object files (.o, .so, .a, .dylib, .dll) or executable binaries found. The .bin files in fixtures root are test data files, not compiled artifacts.
2026-07-05 20:14:39 -04:00
jedarden
a68d01c726 feat(bf-337i2): create 300 DPI invoice OCR fixture
Add invoice fixture with ground truth text for OCR testing.

Files:
- tests/fixtures/scanned/invoice/invoice-300dpi.pdf: 300 DPI scanned invoice
- tests/fixtures/scanned/invoice/invoice-300dpi-ground-truth.txt: complete text content
- tools/generate_invoice_pdf_fixtures.py: fixture generator script
- tests/fixtures/PROVENANCE.md: added provenance entries for new fixtures
- notes/bf-337i2.md: verification note

Verification:
- PDF is image-based (type=image, not text-embedded)
- Resolution: 300 DPI (x-ppi=300, y-ppi=300)
- Dimensions: 2550 x 3300 pixels (letter size at 300 DPI)
- Ground truth contains complete invoice text with headers, addresses, line items, totals

Closes bf-337i2
2026-07-05 18:20:09 -04:00
jedarden
8473941ebb docs(bf-e4uvb-child-1): document CMAP and ToUnicode entry creation points
- Add notes/bf-e4uvb-child-1.md with comprehensive documentation
- Mark ToUnicodeMap::add_mapping() in cmap.rs (line 86)
- Mark DifferencesOverlay entry creation in encoding.rs (line 195)
- Mark CodespaceRange creation in codespace.rs (line 256)

These are the critical insertion points for future unmapped glyph skip logic.
Closes bf-e4uvb-child-1.
2026-07-05 14:48:53 -04:00
jedarden
a0c3ebb237 feat(bf-5for4): design manifest schema for grep-corpus
- Add manifest.csv with header row: filename,source_url,page_count,file_size,checksum,license
- Update README.md with comprehensive schema documentation and field descriptions
- Schema supports provenance tracking, integrity verification, and license compliance

Closes bf-5for4
2026-07-05 12:21:25 -04:00
jedarden
8fe61a1ba5 docs(pdftract-25k4x): add verification note for figure/caption detection 2026-06-01 09:35:02 -04:00
jedarden
3d795a2d11 feat(bf-2he4t): assemble scanned fixtures corpus with ground-truth transcripts
Created tests/fixtures/scanned/ directory structure for WER gate testing:

- README.md: Corpus overview and WER targets (<3% on clean 300-DPI scans)
- GEN_MANIFEST.md: Fixture specifications and generation checklist
- receipt/receipt-300dpi.txt: Ground truth for AS-02 test scenario (37 lines)
- documents/invoice-300dpi.txt: Business invoice ground truth (55 lines)
- documents/form-300dpi.txt: Employment application form (78 lines)
- multi-page/doc-10page-300dpi.txt: Performance fixture (255 lines, 10 pages)

Generation tools:
- generate_scanned_fixtures.py: Python script for PDF generation
- generate_scanned_fixtures.rs: Rust alternative for fixture metadata
- calculate_wer.py: WER/CER calculation utility for OCR validation

Test stub:
- wer_gate_stub.rs: Placeholder for WER gate tests (marked #[ignore])

Total ground-truth content: 425 lines across 4 fixtures

Next steps:
1. Generate PDFs from ground truth using generation script
2. Verify WER < 3% on generated fixtures
3. Enable WER gate tests

Closes bf-2he4t
2026-06-01 09:25:53 -04:00