pdftract/tests/fixtures/no-mapping.md

no-mapping.pdf Fixture Documentation

Overview

The no-mapping.pdf fixture is a Level 4 Unicode recovery test case designed to exercise the worst-case scenario for text extraction: a PDF with custom glyph names that cannot be mapped to Unicode characters through any standard method.

Location: tests/fixtures/encoding/no-mapping.pdf
Generated: 2026-06-09
Last Regenerated: 2026-07-03
SHA256: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0

Fixture Structure

PDF Properties

$ pdfinfo tests/fixtures/encoding/no-mapping.pdf
Pages:           1
Page size:       612 x 792 pts (letter)
Page rot:        0
File size:       660 bytes
PDF version:     1.4
Encrypted:       no

Font Information

$ pdffonts tests/fixtures/encoding/no-mapping.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
CustomNoMap                          Type 1            Custom           no  no  no       5  0

Key characteristics:

• Font Name: /CustomNoMap — non-standard, custom font
• Subtype: Type1
• Encoding: Custom encoding with differences array
• Embedded: No — font program not embedded
• Subset: No
• ToUnicode: No CMap present

Content Stream Analysis

The PDF contains a single content stream that displays three glyphs by name:

BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET

The content references glyphs /g001, /g002, and /g003 through a custom encoding dictionary.

Custom Encoding Dictionary

/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>

This mapping assigns:

• Position 0 → /g001
• Position 1 → /g002
• Position 2 → /g003

Critical: The glyph names /g001, /g002, /g003 are not in the Adobe Glyph List (AGL), so they cannot be mapped to standard Unicode characters.

Raw PDF Structure

The fixture is a minimal hand-written PDF with 5 indirect objects:

%PDF-1.4
1 0 obj           # Catalog
<<
/Type /Catalog
/Pages 2 0 R
>>
endobj

2 0 obj           # Pages node
<<
/Type /Pages
/Kids [3 0 R]
/Count 1
>>
endobj

3 0 obj           # Page
<<
/Type /Page
/Parent 2 0 R
/MediaBox [0 0 612 792]
/Resources <<
/Font <<
/F1 5 0 R
>>
>>
/Contents 4 0 R
>>
endobj

4 0 obj           # Content stream
<<
/Length 44
>>
stream
BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET
endstream
endobj

5 0 obj           # Font dictionary
<<
/Type /Font
/Subtype /Type1
/BaseFont /CustomNoMap
/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>
>>
endobj

Expected Extraction Behavior

Ground Truth

The tests/fixtures/encoding/no-mapping.txt ground truth file contains:

<EFBFBD><EFBFBD><EFBFBD>

This is three U+FFFD (REPLACEMENT CHARACTER) codepoints, representing the three unmapped glyphs.

Why U+FFFD?

The glyph names /g001, /g002, /g003 cannot be recovered through any of the standard Unicode recovery levels:

1. Level 1 (ToUnicode CMap): ❌ No ToUnicode CMap present
2. Level 2 (Adobe Glyph List): ❌ Glyph names not in AGL
3. Level 3 (Font Fingerprinting): ❌ Font not embedded, no fingerprint available
4. Level 4 (Glyph Shape Recognition): ❌ No embedded glyph outlines

Since all recovery methods fail, pdftract should emit U+FFFD replacement characters to indicate unmappable content.

Extraction Verification

$ cargo run --release -- pdftract mcp tests/fixtures/encoding/no-mapping.pdf
# Should output: <20><><EFBFBD> (three U+FFFD characters)

Or with the CLI:

$ cargo run --release --bin pdftract-cli -- extract \
    tests/fixtures/encoding/no-mapping.pdf
# Expected: Three replacement characters in output

Regeneration Instructions

Prerequisites

• Rust toolchain (1.70+)
• Working cargo build environment

Regeneration Command

The fixture is generated by the Rust binary at tests/fixtures/generate_encoding_fixtures.rs.

To regenerate only the no-mapping.pdf fixture:

# From the pdftract repository root
cargo run --bin generate_encoding_fixtures

This will regenerate all four encoding fixtures in tests/fixtures/encoding/:

• no-mapping.pdf — Custom glyph names (this fixture)
• agl-only.pdf — AGL glyph names
• fingerprint-match.pdf — Embedded Type1 subset
• shape-match.pdf — Custom glyph with shape database

Verification After Regeneration

After regeneration, verify the fixture with:

# 1. Check PDF structure
pdfinfo tests/fixtures/encoding/no-mapping.pdf
pdffonts tests/fixtures/encoding/no-mapping.pdf

# 2. Verify SHA256 checksum
sha256sum tests/fixtures/encoding/no-mapping.pdf
# Expected: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0

# 3. Verify ground truth extraction
cargo nextest run encoding_recovery

# 4. Update PROVENANCE.md if SHA256 changed
# Add new entry at tests/fixtures/PROVENANCE.md line 208

Manual Regeneration (Alternative)

If the binary fails to build, the fixture can be manually reconstructed from the raw PDF structure shown above. Create a file no-mapping.pdf with the exact content:

cat > tests/fixtures/encoding/no-mapping.pdf << 'EOF'
%PDF-1.4
1 0 obj
<<
/Type /Catalog
/Pages 2 0 R
>>
endobj
2 0 obj
<<
/Type /Pages
/Kids [3 0 R]
/Count 1
>>
endobj
3 0 obj
<<
/Type /Page
/Parent 2 0 R
/MediaBox [0 0 612 792]
/Resources <<
/Font <<
/F1 5 0 R
>>
>>
/Contents 4 0 R
>>
endobj
4 0 obj
<<
/Length 44
>>
stream
BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET
endstream
endobj
5 0 obj
<<
/Type /Font
/Subtype /Type1
/BaseFont /CustomNoMap
/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>
>>
endobj
xref
0 6
0000000000 65535 f 
0000000009 00000 n 
0000000058 00000 n 
0000000115 00000 n 
0000000241 00000 n 
0000000338 00000 n 
trailer
<<
/Size 6
/Root 1 0 R
>>
startxref
477
%%EOF
EOF

Inspection Commands

PDF Structure Inspection

# Basic PDF info
pdfinfo tests/fixtures/encoding/no-mapping.pdf

# Font details
pdffonts tests/fixtures/encoding/no-mapping.pdf

# View raw PDF structure (human-readable)
pdftk tests/fixtures/encoding/no-mapping.pdf dump_data output

# Extract and display content stream
pdftk tests/fixtures/encoding/no-mapping.pdf uncompress output /tmp/no-mapping-uncompressed.pdf
cat /tmp/no-mapping-uncompressed.pdf

Cross-Reference Table Verification

# Check xref table integrity
pdftk tests/fixtures/encoding/no-mapping.pdf dump_data verbose | grep -A 20 "xref"

Glyph Name Extraction

# Extract glyph names from encoding dictionary
grep -o "g[0-9]\{3\}" tests/fixtures/encoding/no-mapping.pdf
# Expected output: g001, g002, g003

Content Stream Inspection

# Show content stream (formatted)
strings tests/fixtures/encoding/no-mapping.pdf | grep -A 5 "BT"

Test Coverage

This fixture exercises the following test scenarios:

1. Custom Encoding Handling: Non-standard encoding with differences array
2. Missing ToUnicode: No CMap fallback for glyph-to-Unicode mapping
3. Non-AGL Glyph Names: Glyph names not in Adobe Glyph List
4. U+FFFD Emission: Proper replacement character generation
5. Error Resilience: Graceful handling of completely unmappable content

Related Tests

• tests/encoding_recovery.rs — Unicode recovery level tests
• tests/cjk_encoding.rs — CJK encoding with proper ToUnicode
• tests/encoding_recovery_integration.rs — Full extraction pipeline

Troubleshooting

Issue: Extraction produces wrong characters

Symptom: pdftract outputs "ABC" or other incorrect text instead of U+FFFD.

Diagnosis:

# Verify the fixture structure matches expected
pdffonts tests/fixtures/encoding/no-mapping.pdf | grep CustomNoMap
# Should show: CustomNoMap | Type 1 | Custom | no | no | no

Resolution:

• Ensure fixture was regenerated with correct version of generate_encoding_fixtures.rs
• Check that /g001, /g002, /g003 are not in AGL (grep -r g001 crates/pdftract-core/src/font/agl.rs)

Issue: SHA256 checksum mismatch

Symptom: Regenerated fixture has different hash.

Diagnosis:

# Compare file sizes
ls -l tests/fixtures/encoding/no-mapping.pdf
# Expected: 660 bytes

Resolution:

• Minimal differences (whitespace in PDF) are acceptable
• Update SHA256 in PROVENANCE.md if content is semantically identical
• If structure changed, investigate generate_encoding_fixtures.rs for unintended modifications

Issue: Ground test file mismatch

Symptom: Test expects "ABC" but gets "<22><><EFBFBD>" (or vice versa).

History:

• 2026-07-03 (bf-f0xqd): Ground truth corrected from "ABC" to U+FFFD
• The fixture was originally designed for glyph shape recovery (Level 4), but that feature is not yet implemented
• Current behavior correctly emits U+FFFD

Resolution:

• Current expected output is three U+FFFD characters
• Tests should verify replacement character emission, not glyph shape recovery

References

Plan Documentation

• Phase 2.3: Encoding recovery and Unicode mapping (lines 1420-1650)
• TH-03: Text extraction and encoding recovery tests
• INV-14: Unicode recovery levels and fallback strategies

Related Fixtures

• agl-only.pdf — Level 2 recovery (Adobe Glyph List)
• fingerprint-match.pdf — Level 3 recovery (font fingerprinting)
• shape-match.pdf — Level 4 recovery (glyph shape database)

Internal Documentation

• tests/fixtures/PROVENANCE.md — Fixture generation history
• tests/fixtures/generate_encoding_fixtures.rs — Generation source code
• crates/pdftract-core/src/font/ — Font resolution and encoding implementation