diff --git a/tests/fixtures/encoding/no-mapping.md b/tests/fixtures/encoding/no-mapping.md new file mode 100644 index 0000000..2441c42 --- /dev/null +++ b/tests/fixtures/encoding/no-mapping.md @@ -0,0 +1,206 @@ +# no-mapping.pdf Fixture + +## Purpose + +Level 4 Unicode recovery test fixture - worst case scenario where glyph names cannot be recovered via any standard method. This PDF uses custom glyph names (`/g001`, `/g002`, `/g003`) that are not in the Adobe Glyph List (AGL), has no ToUnicode CMap, and uses a custom encoding with no standard fallback. + +Expected output: `���` (three U+FFFD replacement characters) since the glyph names have no mapping. + +## Structure + +### PDF Properties +- **PDF Version:** 1.4 +- **Pages:** 1 +- **Page Size:** 612 x 792 pts (Letter) +- **File Size:** 660 bytes +- **Encrypted:** No +- **Tagged:** No + +### Font Details +``` +Name: CustomNoMap +Type: Type 1 +Encoding: Custom (Differences array) +Embedded: No +ToUnicode CMap: No +``` + +**Font Object (4 0 obj):** +``` +<< +/Type /Font +/Subtype /Type1 +/BaseFont /CustomNoMap +/Encoding << +/Type /Encoding +/Differences [0 /g001 /g002 /g003] +>> +>> +``` + +### Content Stream + +**Location:** Object 5 0 obj + +**Raw Content:** +``` +BT +/F1 12 Tf +50 700 Td + Tj +ET +``` + +**Explanation:** +- Sets font F1 (CustomNoMap) at 12pt +- Moves to position (50, 700) +- Shows three glyphs by name: g001, g002, g003 +- These glyph names are NOT in the AGL and cannot be mapped to Unicode + +### Glyph Names + +The content uses three custom glyph names: +- `/g001` - No standard mapping (not in AGL) +- `/g002` - No standard mapping (not in AGL) +- `/g003` - No standard mapping (not in AGL) + +Since these glyphs have no ToUnicode mapping and their names are not in any standard encoding or the AGL, they should render as U+FFFD (replacement character) during extraction. + +## Inspection Commands + +### Basic PDF info: +```bash +pdfinfo tests/fixtures/encoding/no-mapping.pdf +``` + +**Expected Output:** +``` +Pages: 1 +Page size: 612 x 792 pts (letter) +File size: 660 bytes +PDF version: 1.4 +``` + +### Font details: +```bash +pdffonts tests/fixtures/encoding/no-mapping.pdf +``` + +**Expected Output:** +``` +name type encoding emb sub uni object ID +------------------------------------ ----------------- ---------------- --- --- --- --------- +CustomNoMap Type 1 Custom no no no 5 0 +``` + +### Content stream inspection: +```bash +# Extract raw content stream +pdftk tests/fixtures/encoding/no-mapping.pdf unpack_files output /tmp/no-mapping-unpack/ +cat /tmp/no-mapping-unpack/doc_5_0.txt + +# Or examine the hex dump +hexdump -C tests/fixtures/encoding/no-mapping.pdf | grep -A 5 "g001" +``` + +**Expected Content:** +``` + Tj +``` + +### Test extraction: +```bash +# Build pdftract first if needed +cargo build --release + +# Extract to see the U+FFFD replacement characters +./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \ + --format markdown -o /tmp/no-mapping-output +cat /tmp/no-mapping-output.md +``` + +## Regeneration Instructions + +### Method 1: Using the Rust generator (Recommended) + +The fixture is generated by `tests/fixtures/generate_encoding_fixtures.rs`. + +**Steps:** +```bash +# Build the generator binary +cargo build --bin generate_encoding_fixtures + +# Run the generator (creates all 4 encoding fixtures) +./target/debug/generate_encoding_fixtures + +# Or run directly via cargo +cargo run --bin generate_encoding_fixtures +``` + +**Output location:** `tests/fixtures/encoding/no-mapping.pdf` + +**Ground truth:** `tests/fixtures/encoding/no-mapping.txt` (should contain `���`) + +### Method 2: Manual reconstruction (Not recommended) + +To manually reconstruct this PDF: + +1. Create the 5 PDF objects as specified in `generate_encoding_fixtures.rs` +2. Build the xref table with correct byte offsets +3. Add the trailer with `/Size 6` and `/Root 1 0 R` +4. Write the complete PDF to `tests/fixtures/encoding/no-mapping.pdf` + +**Use the Rust generator instead** - it computes all offsets automatically and ensures the xref table is correct. + +## Verification + +After regeneration, verify: + +```bash +# Check SHA256 matches expected value +sha256sum tests/fixtures/encoding/no-mapping.pdf +# Expected: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0 + +# Verify extraction produces U+FFFD characters +./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \ + --format text -o /tmp/no-mapping-output.txt +cat /tmp/no-mapping-output.txt | od -c +# Should see: 0000000 f f f d f f f d f f f d +``` + +## Test References + +This fixture is used in the following tests: + +- `tests/debug_encoding_pdf.rs` - Encoding recovery debug tests +- `tests/debug_encoding_fixtures.rs` - Fixture validation tests +- `tests/encoding_recovery.rs` - Unicode recovery integration tests + +## Related Fixtures + +- `agl-only.pdf` - Uses standard Type1 font with AGL glyph names (Level 2) +- `fingerprint-match.pdf` - Uses embedded Type1 font subset (Level 3) +- `shape-match.pdf` - Uses custom glyph names with shape recognition (Level 4) + +## History + +- **Generated:** 2026-06-09 +- **Regenerated:** 2026-07-02 (bf-512z1) +- **Regenerated:** 2026-07-03 (bf-f0xqd) - Corrected ground truth from "ABC" to U+FFFD +- **Regenerated:** 2026-07-03 (bf-1m30m) - Via `generate_encoding_fixtures.rs` +- **SHA256:** `b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0` + +## Notes + +The key insight of this fixture is that it represents the **worst-case scenario** for Unicode recovery: +1. No ToUnicode CMap +2. Custom encoding (no StandardEncoding/WinAnsiEncoding/MacRomanEncoding fallback) +3. Custom glyph names not in AGL (`/g001`, `/g002`, `/g003`) +4. No embedded font program for fingerprinting + +This forces the extractor to either: +- Emit U+FFFD replacement characters (correct behavior) +- Attempt glyph shape recognition (if implemented) +- Fail gracefully + +The fixture validates that pdftract handles unmapped glyphs correctly rather than producing garbage text or crashing.