pdftract/tests/fixtures/encoding/no-mapping.md

no-mapping.pdf Fixture

Purpose

Level 4 Unicode recovery test fixture - worst case scenario where glyph names cannot be recovered via any standard method. This PDF uses custom glyph names (/g001, /g002, /g003) that are not in the Adobe Glyph List (AGL), has no ToUnicode CMap, and uses a custom encoding with no standard fallback.

Expected output: <EFBFBD><EFBFBD><EFBFBD> (three U+FFFD replacement characters) since the glyph names have no mapping.

Structure

PDF Properties

• PDF Version: 1.4
• Pages: 1
• Page Size: 612 x 792 pts (Letter)
• File Size: 660 bytes
• Encrypted: No
• Tagged: No

Font Details

Name: CustomNoMap
Type: Type 1
Encoding: Custom (Differences array)
Embedded: No
ToUnicode CMap: No

Font Object (4 0 obj):

<<
/Type /Font
/Subtype /Type1
/BaseFont /CustomNoMap
/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>
>>

Content Stream

Location: Object 5 0 obj

Raw Content:

BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET

Explanation:

• Sets font F1 (CustomNoMap) at 12pt
• Moves to position (50, 700)
• Shows three glyphs by name: g001, g002, g003
• These glyph names are NOT in the AGL and cannot be mapped to Unicode

Glyph Names

The content uses three custom glyph names:

• /g001 - No standard mapping (not in AGL)
• /g002 - No standard mapping (not in AGL)
• /g003 - No standard mapping (not in AGL)

Since these glyphs have no ToUnicode mapping and their names are not in any standard encoding or the AGL, they should render as U+FFFD (replacement character) during extraction.

Inspection Commands

Basic PDF info:

pdfinfo tests/fixtures/encoding/no-mapping.pdf

Expected Output:

Pages:           1
Page size:       612 x 792 pts (letter)
File size:       660 bytes
PDF version:     1.4

Font details:

pdffonts tests/fixtures/encoding/no-mapping.pdf

Expected Output:

name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
CustomNoMap                          Type 1            Custom           no  no  no       5  0

Content stream inspection:

# Extract raw content stream
pdftk tests/fixtures/encoding/no-mapping.pdf unpack_files output /tmp/no-mapping-unpack/
cat /tmp/no-mapping-unpack/doc_5_0.txt

# Or examine the hex dump
hexdump -C tests/fixtures/encoding/no-mapping.pdf | grep -A 5 "g001"

Expected Content:

<g001><g002><g003> Tj

Test extraction:

# Build pdftract first if needed
cargo build --release

# Extract to see the U+FFFD replacement characters
./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \
  --format markdown -o /tmp/no-mapping-output
cat /tmp/no-mapping-output.md

Regeneration Instructions

Method 1: Using the Rust generator (Recommended)

The fixture is generated by tests/fixtures/generate_encoding_fixtures.rs.

Steps:

# Build the generator binary
cargo build --bin generate_encoding_fixtures

# Run the generator (creates all 4 encoding fixtures)
./target/debug/generate_encoding_fixtures

# Or run directly via cargo
cargo run --bin generate_encoding_fixtures

Output location: tests/fixtures/encoding/no-mapping.pdf

Ground truth: tests/fixtures/encoding/no-mapping.txt (should contain <EFBFBD><EFBFBD><EFBFBD>)

Method 2: Manual reconstruction (Not recommended)

To manually reconstruct this PDF:

1. Create the 5 PDF objects as specified in generate_encoding_fixtures.rs
2. Build the xref table with correct byte offsets
3. Add the trailer with /Size 6 and /Root 1 0 R
4. Write the complete PDF to tests/fixtures/encoding/no-mapping.pdf

Use the Rust generator instead - it computes all offsets automatically and ensures the xref table is correct.

Verification

After regeneration, verify:

# Check SHA256 matches expected value
sha256sum tests/fixtures/encoding/no-mapping.pdf
# Expected: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0

# Verify extraction produces U+FFFD characters
./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \
  --format text -o /tmp/no-mapping-output.txt
cat /tmp/no-mapping-output.txt | od -c
# Should see: 0000000   f f f d   f f f d   f f f d

Test References

This fixture is used in the following tests:

• tests/debug_encoding_pdf.rs - Encoding recovery debug tests
• tests/debug_encoding_fixtures.rs - Fixture validation tests
• tests/encoding_recovery.rs - Unicode recovery integration tests

Related Fixtures

• agl-only.pdf - Uses standard Type1 font with AGL glyph names (Level 2)
• fingerprint-match.pdf - Uses embedded Type1 font subset (Level 3)
• shape-match.pdf - Uses custom glyph names with shape recognition (Level 4)

History

• Generated: 2026-06-09
• Regenerated: 2026-07-02 (bf-512z1)
• Regenerated: 2026-07-03 (bf-f0xqd) - Corrected ground truth from "ABC" to U+FFFD
• Regenerated: 2026-07-03 (bf-1m30m) - Via generate_encoding_fixtures.rs
• SHA256: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0

Notes

The key insight of this fixture is that it represents the worst-case scenario for Unicode recovery:

1. No ToUnicode CMap
2. Custom encoding (no StandardEncoding/WinAnsiEncoding/MacRomanEncoding fallback)
3. Custom glyph names not in AGL (/g001, /g002, /g003)
4. No embedded font program for fingerprinting

This forces the extractor to either:

• Emit U+FFFD replacement characters (correct behavior)
• Attempt glyph shape recognition (if implemented)
• Fail gracefully

The fixture validates that pdftract handles unmapped glyphs correctly rather than producing garbage text or crashing.