This commit cleans up the remaining working tree modifications as part of bead bf-56090:
1. test_truncated_fate_recovery.rs: Migrate test fixture path from
tests/fixtures/malformed/truncated-flate.pdf to
tests/error_recovery/fixtures/truncated_mid_stream.pdf and improve
test output formatting.
2. generate_form_fixtures.rs: Update lopdf API calls to use new StringFormat
enum instead of deprecated boolean parameter, and migrate from
save_to_bytes() to save_to() with Cursor.
Both changes are necessary for compilation with current lopdf version.
Fixtures verified to exist at both old and new paths.
Related to: bf-56090
Added typed config struct for unmapped glyph names configuration:
- unmapped_glyph_names: Vec<String> - list of glyph names to skip
- description: Option<String> - optional documentation
- version: Option<String> - format version identifier
The struct represents the JSON structure from build/unmapped-glyph-names.json
and is ready for parsing implementation in follow-up beads.
Closes bf-3vo80. Verification: notes/bf-3vo80.md
- Identified build/unmapped-glyph-names.json as the primary config file
- Documented DifferencesOverlay struct in encoding.rs as the Rust config structure
- Described the build-time generation process via build.rs
- Listed all files involved in the config pipeline
Closes bf-10vsm
- Add std::process::Stdio to imports in both test files
- Update code to use imported Stdio instead of fully qualified paths
- Verify compilation succeeds with cargo check
Closes bf-3x4rp. Verification: notes/bf-3x4rp.md
- Updated degraded-200dpi-ground-truth.txt with accurate transcription
- Matches first 2500 chars of source document (as created by PDF generation script)
- Contains Abraham Lincoln public domain text with metadata
- Verified with diff against source content
- Includes verification note at notes/bf-cc7ru.md
Acceptance criteria:
✓ Ground truth file exists in tests/fixtures/scanned/low-quality/
✓ Accurately reflects source document content (verified)
✓ Properly formatted and readable
✓ Special characters handled appropriately
Closes bf-cc7ru
- Created degraded-200dpi.pdf (588KB) with intentional degradation
- Applied 200 DPI, blur, noise, contrast reduction, compression
- Used public domain Abraham Lincoln text as source
- Added supporting infrastructure (creation script, ground truth)
- PDF verified renderable and suitable for OCR testing
- Updated PROVENANCE.md with fixture metadata
Closes bf-4ozna.
Fuzz harness executed successfully:
- No crashes or panics on startup
- Process ran for full 0.5-second duration
- Coverage expanded from 124 to 518+ edges
- Generated 203+ test cases from empty corpus
- Clean exit via timeout (expected)
Verification: notes/bf-1bpzw.md
Acceptance criteria: PASS (all 4 criteria met)
All acceptance criteria verified:
- Glyph names checked against unmapped_glyph_names config
- Unmapped glyphs skipped during CMAP generation
- Code compiles without errors
- Tests demonstrate skip behavior
The implementation was already complete in encoding.rs:200.
See notes/bf-1puzp.md for detailed verification.
- Create fuzz/fuzz_targets/profile_yaml.rs to test YAML parsing
- Register profile_yaml target in fuzz/Cargo.toml
- Verify cargo-fuzz 0.13.1 is installed and working
- All 7 fuzz targets now operational per plan requirement (line 3236)
Closes bf-1b7od. Verification: notes/bf-1b7od.md
- Add test_forms_extraction() function to test pdftract extract on fixtures
- Uses walkdir for recursive PDF discovery in tests/fixtures/forms/
- Calls extract_pdf() and result_to_json() on each discovered fixture
- Handles missing fixtures gracefully without panic
- All tests pass (6/6)
Closes bf-184rf. Verification: notes/bf-184rf.md
Create comprehensive documentation of the cleaned tests/fixtures/ structure
following the three-bead cleanup effort (bf-1iefu, bf-xqib3, bf-2yhak).
Changes:
- Create: tests/fixtures/STRUCTURE.md with complete directory organization
- Directory structure with annotations
- Cleanup history and justification
- List of 10 KEEP generators (co-located with fixtures)
- List of 5 removed DELETE generators
- List of 2 relocated RELOCATE tools
- Maintenance guidelines
- Update: tools/README.md with cleanup summary section
- Three-bead coordination overview
- Final state statistics
- Links to related documentation
- Create: notes/bf-620xp.md verification note
Verification:
- ✅ No compiled artifacts remain
- ✅ No obsolete generators remain
- ✅ 10 KEEP generators remain (actively maintained, co-located with fixtures)
- ✅ All RELOCATE tools in tools/ with documentation
- ✅ Binary .bin files are legitimate test data (LZW compression, malformed fixtures)
Closes bf-620xp
Remove 5 DELETE-category generator files identified in bf-1iefu:
- tests/fixtures/scanned/generate_scanned_fixtures.rs (Rust stub, use Python version)
- tests/fixtures/security/generate_sensitive_fixture.py (duplicate, Rust version preferred)
- tests/fixtures/encoding/generate_unmapped_glyphs.rs (superseded by Python version)
- tests/fixtures/malformed/gen-bomb-10k-2g.sh (superseded by Python version)
- tests/fixtures/forms/generate_form_fixtures.d (orphaned Makefile dependency)
Verification: No compiled object files (.o, .so, .a, .dylib, .dll) or executable binaries found. The .bin files in fixtures root are test data files, not compiled artifacts.
Add invoice fixture with ground truth text for OCR testing.
Files:
- tests/fixtures/scanned/invoice/invoice-300dpi.pdf: 300 DPI scanned invoice
- tests/fixtures/scanned/invoice/invoice-300dpi-ground-truth.txt: complete text content
- tools/generate_invoice_pdf_fixtures.py: fixture generator script
- tests/fixtures/PROVENANCE.md: added provenance entries for new fixtures
- notes/bf-337i2.md: verification note
Verification:
- PDF is image-based (type=image, not text-embedded)
- Resolution: 300 DPI (x-ppi=300, y-ppi=300)
- Dimensions: 2550 x 3300 pixels (letter size at 300 DPI)
- Ground truth contains complete invoice text with headers, addresses, line items, totals
Closes bf-337i2
All necessary imports and test infrastructure are already in place:
- Standard library imports (fs, path, process)
- pdftract CLI imports (exit codes)
- pdftract Core imports (diagnostics, decryption errors)
- Test helpers (pdftract_bin, encrypted_fixture, assertion functions)
- Test modules (unsupported_handlers, exit_codes, password_handling, consistency)
File compiles successfully and all imports match actual crate structure.
Infrastructure was implemented in previous commit as part of bf-5for4.
Closes bf-2nl4x
Add discover_pdf_fixtures function to recursively find all PDF files in
tests/fixtures/forms/ directory using walkdir. Includes test that prints
discovered fixture names to stdout.
Acceptance criteria:
- walkdir is available in crates/pdftract-cli/Cargo.toml
- discover_pdf_fixtures function returns Vec<PathBuf> of discovered PDFs
- test_discover_pdf_fixtures runs and prints fixture names
- No compilation errors
Closes bf-1pxdm.
- Created tests/forms_integration.rs with basic test module structure
- Created tests/mod.rs to aggregate test modules
- Added module documentation for forms integration tests
- File compiles successfully with cargo test --test forms_integration
Acceptance criteria:
- tests/forms_integration.rs exists ✓
- tests/mod.rs includes the module declaration ✓
- cargo test --test forms_integration compiles ✓
- File has basic test module skeleton ✓
Closes bf-2y7uh
- Add notes/bf-e4uvb-child-1.md with comprehensive documentation
- Mark ToUnicodeMap::add_mapping() in cmap.rs (line 86)
- Mark DifferencesOverlay entry creation in encoding.rs (line 195)
- Mark CodespaceRange creation in codespace.rs (line 256)
These are the critical insertion points for future unmapped glyph skip logic.
Closes bf-e4uvb-child-1.
- download-grep-corpus.sh now generates manifest.csv entries during download
- Implements SHA256 checksum computation via sha256sum
- Extracts page counts using pdfinfo
- Captures file sizes in bytes via stat
- Records source URLs for each downloaded file
- Appends entries to manifest.csv with all 6 required fields
- Handles incremental downloads (skips existing files, appends to manifest)
- Added grep-corpus-generate-manifest.sh helper for bootstrapping existing corpus
- Generated manifest entries for all 1,260 existing PDFs
- Verified checksum accuracy against actual files
Acceptance criteria:
- ✅ download-grep-corpus.sh writes manifest.csv entries for each downloaded PDF
- ✅ All 6 fields populated: filename, source_url, page_count, file_size, checksum, license
- ✅ Script handles incremental downloads (appends to existing manifest)
Closes bf-20fot
Verification: notes/bf-20fot.md
- Fix /Differences array construction in generate_unmapped_glyphs.py
- Changed from '[code glyph code glyph ...]' to '[starting_code glyph1 glyph2 ...]' format
- All 7 unmapped glyphs from design spec now properly encoded without ToUnicode CMap
- Fixture validation passes: no ToUnicode, correct encoding format, all glyphs present
Closes bf-84xr8. Verification: notes/bf-84xr8.md
- Implement parameterized generator for unmapped glyph PDF fixtures
- Support custom glyph names via --glyphs JSON parameter
- Support custom font titles via --title parameter
- Include built-in test set matching design specification (notes/bf-68f9i-design.md)
- Generate valid PDF 1.4 with Type1 custom encoding
- Passes all acceptance criteria:
* Script exists in tests/fixtures/encoding/
* Accepts parameters for glyph names and encoding
* Generates valid PDF with specified structure
* Tested with dry runs (default and custom glyphs)
* Committed to git
Closes bf-3cwge
- Added PDF Structure section based on bf-4ozvw findings
- Updated SHA256 in profiles/PROVENANCE.md for encoding fixtures
- Section includes file characteristics, font analysis, and technical significance
Successfully executed generate_unicode_recovery_fixtures_bin to create
no-mapping.pdf fixture (650 bytes). Validated PDF structure includes
custom /CustomNoMap font with /Differences encoding, non-AGL glyph
names, and no ToUnicode entry. Ground truth contains U+FFFD × 3.
Closes bf-ttbb5.
The no-mapping.pdf fixture uses custom glyph names (/g001, /g002, /g003)
that are not in the Adobe Glyph List and cannot be recovered through any
Unicode recovery mechanism (no ToUnicode CMap, no standard encoding,
no AGL match).
The ground truth has been corrected from 'ABC' to three U+FFFD replacement
characters (���), which is the expected output when all encoding recovery
methods fail per the Failure Mode Taxonomy.
Also updated PROVENANCE.md with new SHA256 hashes for encoding fixtures
that were regenerated on 2026-07-02.
This fixture correctly exercises the ENCODING_NO_MAPPING failure mode as
specified in the plan (line 733).
Closes bf-f0xqd.
CJK fixtures and tests already exist from previous work:
- tests/fixtures/cjk/ contains all 4 required PDFs
- Ground truth files for each encoding (GB18030, Shift-JIS, EUC-KR, Big5)
- Tests in crates/pdftract-core/tests/cjk_encoding.rs and tests/test_encoding.rs
- Tests fail due to unimplemented CJK encoding (expected for Phase 2.3)
- Updated PROVENANCE.md with CJK fixture entries
Fixtures are ready for CJK encoding implementation.
Closes bf-3ourh
- no-mapping.txt: fix garbled unicode to correct 'ABC' output
- shape-match.txt: fix from 'Shape' to 'S' (actual PDF content)
- Add PROVENANCE.md entries for all 4 encoding fixtures
- PDFs remain unchanged (already valid)
Fixes ground truth for Level 2-4 Unicode recovery fixtures:
- no-mapping.pdf: PDF with no ToUnicode, no standard encoding
- agl-only.pdf: PDF with AGL glyph names only
- fingerprint-match.pdf: PDF with embedded font for fingerprint matching
- shape-match.pdf: PDF with subset font for shape recognition
Closes bf-512z1
The indent trigger was using .abs() which fired on both increased indent
(non-indented → indented) AND decreased indent (indented → non-indented).
This caused drop-cap style paragraphs (indented first line, flush-left
continuation) to incorrectly split into two blocks.
Per plan Phase 4.4 heuristic #2, indent change should only trigger when the
current line is MORE indented (to the right, larger x0) than the block
average - i.e., a new paragraph starting after non-indented text. It should
NOT trigger for decreased indent (first line indented, rest flush-left).
Fix: Remove .abs() and only check if line_x0 - block_avg_x0 > threshold.
Tests:
- test_indented_first_line_new_block: PASS (non-indented → indented splits)
- test_indented_first_line_of_paragraph_not_split: PASS (drop cap stays together)
- All 179 line module tests: PASS
