Add comprehensive documentation of all CMAP and ToUnicode entry creation
locations in the pdftract codebase. Identifies key insertion points for
future skip logic for unmapped glyphs.
Key locations documented:
- CMAP codespace range creation (cmap/codespace.rs:60, 356)
- ToUnicode mapping creation (font/cmap.rs:86)
- Type1 font encoding differences (font/encoding.rs:197)
- ToUnicode resolution success (font/resolver.rs:398)
All locations already have MARKER comments in code referencing this note.
Closes bf-e4uvb
- Create fuzz/fuzz_targets/profile_yaml.rs to test YAML parsing
- Register profile_yaml target in fuzz/Cargo.toml
- Verify cargo-fuzz 0.13.1 is installed and working
- All 7 fuzz targets now operational per plan requirement (line 3236)
Closes bf-1b7od. Verification: notes/bf-1b7od.md
- Add test_forms_extraction() function to test pdftract extract on fixtures
- Uses walkdir for recursive PDF discovery in tests/fixtures/forms/
- Calls extract_pdf() and result_to_json() on each discovered fixture
- Handles missing fixtures gracefully without panic
- All tests pass (6/6)
Closes bf-184rf. Verification: notes/bf-184rf.md
- cargo-fuzz 0.13.1 installed ✓
- fuzz/Cargo.toml exists with libfuzzer-sys dependency ✓
- profile_yaml fuzz target is MISSING ✗
Verified 6 existing fuzz targets but profile_yaml.rs is not present
in fuzz_targets/ directory or registered in fuzz/Cargo.toml.
Verify all acceptance criteria for grep-corpus manifest and validation:
- manifest.csv exists with proper schema (1000 entries)
- validate-corpus.sh is executable and functional
- make validate-corpus runs successfully (1000/1000 files valid)
- Corpus contains 10,590 pages across 6.5MB of synthetic PDFs
Closes bf-4zuss.
Add 6 release workflow templates that were missing from pdftract/.ci/argo-workflows/:
- pdftract-build-binaries.yaml: Cross-compile 5 arch binaries
- pdftract-crates-publish.yaml: Publish to crates.io with ordering
- pdftract-docker-build.yaml: Multi-arch Docker images
- pdftract-docs-build.yaml: mdbook to Cloudflare Pages
- pdftract-github-release.yaml: Populate GitHub Release page
- pdftract-node-publish.yaml: npm publish workflow
These templates already existed in declarative-config and are synced via ArgoCD.
The in-tree copies are now complete for all 10 release pipeline templates.
Verification: notes/bf-5o8cg.md
Closes: bf-5o8cg
- Fuzz harness compiles successfully with proper instrumentation
- Runtime failure due to missing libstdc++.so.6 (NixOS environment issue)
- No crashes or panics during compilation/startup
- Verification note: notes/bf-vrpz1.md
- Built content fuzz harness successfully with cargo fuzz build content
- Verified binary was created at fuzz/target/x86_64-unknown-linux-gnu/release/content
- No compilation errors or warnings
- Binary size: 54M
Closes bf-57ysd
Located and verified js_in_openaction.pdf fixture:
- Path: tests/document_model/fixtures/js_in_openaction.pdf
- Size: 632 bytes, PDF version 1.4
- Currently malformed (missing /Root reference)
- Documented fixture properties and expected behavior
Closes bf-4axmo
Verified fuzz harness starts up and executes without crashes.
- Content stream interpreter handles all inputs without panics
- No crash artifacts generated
- libFuzzer initializes and runs successfully
- Minor: time limit flag not respected (libFuzzer quirk)
Acceptance: PASS (no crashes/panics), WARN (time limit)
See notes/bf-qibsy.md for details.
Create comprehensive documentation of the cleaned tests/fixtures/ structure
following the three-bead cleanup effort (bf-1iefu, bf-xqib3, bf-2yhak).
Changes:
- Create: tests/fixtures/STRUCTURE.md with complete directory organization
- Directory structure with annotations
- Cleanup history and justification
- List of 10 KEEP generators (co-located with fixtures)
- List of 5 removed DELETE generators
- List of 2 relocated RELOCATE tools
- Maintenance guidelines
- Update: tools/README.md with cleanup summary section
- Three-bead coordination overview
- Final state statistics
- Links to related documentation
- Create: notes/bf-620xp.md verification note
Verification:
- ✅ No compiled artifacts remain
- ✅ No obsolete generators remain
- ✅ 10 KEEP generators remain (actively maintained, co-located with fixtures)
- ✅ All RELOCATE tools in tools/ with documentation
- ✅ Binary .bin files are legitimate test data (LZW compression, malformed fixtures)
Closes bf-620xp
Verified that the repository already contains PDF test fixtures with JavaScript actions:
- tests/fixtures/security/embedded-js.pdf (working, contains app.alert actions)
- tests/document_model/fixtures/js_in_openaction.pdf (parsing error)
Documented extraction results and fixture locations for reference.
No new fixture creation needed - existing fixtures serve the purpose.
- Analyze current test assertions in error_recovery_integration.rs
- Analyze current test assertions in stream_decoder_fixtures.rs
- Identify 5 major coverage gaps in truncated-flate recovery testing
- Document missing diagnostic emission in FlateDecoder
- Document lack of actual extraction in integration test
- Map out recommendations for new assertions at high/medium/low priority
Closes bf-303t6.
The file test_truncated_flate_recovery.rs does not exist.
Documented related test files and fixture patterns.
Identified that test_truncated_mid_stream() in error_recovery_integration.rs
is the closest existing test for truncated stream handling.
Acceptance criteria:
- Test file located: NOT FOUND (file does not exist)
- Related test structures: DOCUMENTED
- Existing test functions: LISTED
- Verify all encryption test attributes are properly configured
- Confirm #[test], #[cfg(feature = "decrypt")], and #[ignore] usage
- Validate test framework recognition and compilation success
- Document 18 passing tests with 1 ignored performance test
- Closes bf-53l6m
Add comprehensive test utility and error handling imports to encryption test files:
- Added std::error::Error and std::io for error handling
- Expanded pdftract_core::diagnostics imports to include Diagnostic, DiagnosticsCollector, and ObjRef
- Ensured both test_encryption_errors.rs and test_encryption_unsupported.rs have consistent imports
All imports verified against actual crate structure and compile successfully.
Closes bf-1h5og. Verification: notes/bf-1h5og.md.
Add inline code comments at ToUnicode entry creation points and update
comprehensive documentation notes.
Changes:
- Add MARKER comment at resolver.rs:398-402 for ToUnicode resolution entry creation
- Update notes/bf-2nob5-child-1.md to document all CMAP and ToUnicode creation points
- Verify existing MARKER comments in cmap.rs, codespace.rs, and encoding.rs
Acceptance criteria:
- Code comments added at ToUnicode creation points
- Comprehensive notes file updated with all findings
- Notes file includes file paths, function names, line numbers, and data flow summaries
Closes bf-65hvp
- Added docstring MARKER at CodespaceRange::new() (codespace.rs:56-60) noting
CMAP entry creation point for multi-byte CJK encodings
- Added inline comment MARKER at codespace range creation (codespace.rs:354)
where ranges are created from begincodespacerange blocks
- Created comprehensive notes file at notes/bf-2nob5-child-1.md with:
* All CMAP and ToUnicode entry creation points
* File paths, function names, and line numbers
* Complete data flow summaries and diagrams
* Related files and test locations
* Special cases and edge cases documentation
Verified existing MARKER comments at:
- ToUnicodeMap::add_mapping() (cmap.rs:86-88)
- DifferencesOverlay::parse() (encoding.rs:196-199)
Acceptance criteria met:
✅ Code comments added at CMAP creation points
✅ Code comments added at ToUnicode creation points
✅ Comprehensive notes file created with all findings
✅ Notes include file paths, function names, line numbers, data flows
Closes bf-65hvp
Created comprehensive documentation for ToUnicode CMap processing:
- Clarified that ToUnicode processes byte sequences, not glyph names
- Documented complete parsing flow (beginbfchar and beginbfrange)
- Identified entry creation point at cmap.rs:82-88
- Explained UTF-16BE decoding and special cases (ligatures, surrogates)
- Compared ToUnicode (Level 1) with AGL glyph name processing (Level 2)
Acceptance criteria:
- Data flow documented: which glyph names are processed ✓
- Entry addition mechanism documented ✓
- Key data structures identified ✓
- Documentation saved to notes/bf-e4uvb-child-4-tounicode-flow.md ✓
See notes/bf-e4uvb-child-4-tounicode-flow.md for complete documentation.
- Identified all CMAP entry creation points in codebase
- ToUnicodeMap::add_mapping() for character code → Unicode mappings
- CodespaceRange::new() for byte-width boundaries in CJK encodings
- DifferencesOverlay::parse() for Type1 font encoding differences
- Created comprehensive reference with file paths, functions, and line numbers
- Documented call sites and data structures
Closes bf-4hwap
- Verified invoice, letter, and form OCR fixtures using WER measurement script
- Confirmed all PDFs are ~300 DPI using pdfimages and pdfinfo
- Validated all PDFs contain scanned content (no embedded text)
- Tested WER calculation: 0.00% with perfect input, 96.30% with errors
- All acceptance criteria PASSED
Closes bf-hdn7k.
Confirm form-300dpi.pdf and form-300dpi-ground-truth.txt exist
and meet all acceptance criteria:
- PDF is true scanned content (no text layer)
- Exactly 300 DPI (2550x3300 pixels for 8.5x11" page)
- Ground truth contains complete 79-line employment application form
Closes bf-7mvji
Add invoice fixture with ground truth text for OCR testing.
Files:
- tests/fixtures/scanned/invoice/invoice-300dpi.pdf: 300 DPI scanned invoice
- tests/fixtures/scanned/invoice/invoice-300dpi-ground-truth.txt: complete text content
- tools/generate_invoice_pdf_fixtures.py: fixture generator script
- tests/fixtures/PROVENANCE.md: added provenance entries for new fixtures
- notes/bf-337i2.md: verification note
Verification:
- PDF is image-based (type=image, not text-embedded)
- Resolution: 300 DPI (x-ppi=300, y-ppi=300)
- Dimensions: 2550 x 3300 pixels (letter size at 300 DPI)
- Ground truth contains complete invoice text with headers, addresses, line items, totals
Closes bf-337i2
