pdftract/tests/fixtures/no-mapping.md

no-mapping.pdf Fixture Documentation

Overview

The no-mapping.pdf fixture is a Level 4 Unicode recovery test case designed to exercise the worst-case scenario for text extraction: a PDF with custom glyph names that cannot be mapped to Unicode characters through any standard method.

Location: tests/fixtures/encoding/no-mapping.pdf
Generated: 2026-06-09
Last Regenerated: 2026-07-03
SHA256: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0

Fixture Structure

PDF Properties

$ pdfinfo tests/fixtures/encoding/no-mapping.pdf
Pages:           1
Page size:       612 x 792 pts (letter)
Page rot:        0
File size:       660 bytes
PDF version:     1.4
Encrypted:       no

Font Information

$ pdffonts tests/fixtures/encoding/no-mapping.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
CustomNoMap                          Type 1            Custom           no  no  no       5  0

Key characteristics:

• Font Name: /CustomNoMap — non-standard, custom font
• Subtype: Type1
• Encoding: Custom encoding with differences array
• Embedded: No — font program not embedded
• Subset: No
• ToUnicode: No CMap present

Content Stream Analysis

The PDF contains a single content stream that displays three glyphs by name:

BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET

The content references glyphs /g001, /g002, and /g003 through a custom encoding dictionary.

Custom Encoding Dictionary

/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>

This mapping assigns:

• Position 0 → /g001
• Position 1 → /g002
• Position 2 → /g003

Critical: The glyph names /g001, /g002, /g003 are not in the Adobe Glyph List (AGL), so they cannot be mapped to standard Unicode characters.

Raw PDF Structure

The fixture is a minimal hand-written PDF with 5 indirect objects:

%PDF-1.4
1 0 obj           # Catalog
<<
/Type /Catalog
/Pages 2 0 R
>>
endobj

2 0 obj           # Pages node
<<
/Type /Pages
/Kids [3 0 R]
/Count 1
>>
endobj

3 0 obj           # Page
<<
/Type /Page
/Parent 2 0 R
/MediaBox [0 0 612 792]
/Resources <<
/Font <<
/F1 5 0 R
>>
>>
/Contents 4 0 R
>>
endobj

4 0 obj           # Content stream
<<
/Length 44
>>
stream
BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET
endstream
endobj

5 0 obj           # Font dictionary
<<
/Type /Font
/Subtype /Type1
/BaseFont /CustomNoMap
/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>
>>
endobj

PDF Structure

Based on analysis from bead bf-4ozvw, the fixture has the following structural characteristics:

File Characteristics

• Size: Extremely small (660 bytes) — minimal fixture
• Pages: Single page document
• Format: PDF 1.4 (standard version)
• Page Size: US Letter (612 x 792 pts)
• Complexity: No JavaScript, forms, or metadata streams
• Optimization: Not optimized
• User Properties: None
• Tagged PDF: No
• Metadata Stream: No

Font Analysis

• Font Name: CustomNoMap — descriptive name indicating purpose
• Font Type: Type 1 (PostScript font)
• Encoding: Custom — this is the key characteristic
• Embedding: Not embedded (emb: no)
• Subsetting: Not subset (sub: no)
• Unicode: No Unicode mapping (uni: no)
• Object ID: 5 0 R

Technical Significance

This fixture is particularly valuable for testing:

1. Font encoding edge cases — Custom encodings that don't map to standard character sets
2. Glyph recognition without standard mappings — Glyphs with no direct Unicode equivalent
3. Error handling in the text extraction pipeline — Fallback strategies when encoding mappings are unavailable
4. No Font Embedding scenario — PDF readers must rely on encoding information provided

The fixture name "no-mapping" combined with the custom encoding indicates this is intentionally testing the case where pdftract must handle glyphs that have no direct mapping to readable text — a critical edge case for robust PDF text extraction.

Expected Extraction Behavior

Ground Truth

The tests/fixtures/encoding/no-mapping.txt ground truth file contains:

<EFBFBD><EFBFBD><EFBFBD>

This is three U+FFFD (REPLACEMENT CHARACTER) codepoints, representing the three unmapped glyphs.

Why U+FFFD?

The glyph names /g001, /g002, /g003 cannot be recovered through any of the standard Unicode recovery levels:

1. Level 1 (ToUnicode CMap): ❌ No ToUnicode CMap present
2. Level 2 (Adobe Glyph List): ❌ Glyph names not in AGL
3. Level 3 (Font Fingerprinting): ❌ Font not embedded, no fingerprint available
4. Level 4 (Glyph Shape Recognition): ❌ No embedded glyph outlines

Since all recovery methods fail, pdftract should emit U+FFFD replacement characters to indicate unmappable content.

Extraction Verification

$ cargo run --release -- pdftract mcp tests/fixtures/encoding/no-mapping.pdf
# Should output: <20><><EFBFBD> (three U+FFFD characters)

Or with the CLI:

$ cargo run --release --bin pdftract-cli -- extract \
    tests/fixtures/encoding/no-mapping.pdf
# Expected: Three replacement characters in output

Regeneration Instructions

Prerequisites

• Rust toolchain (1.70+)
• Working cargo build environment

Regeneration Command

The fixture is generated by the Python script at tools/generate_encoding_fixtures.py.

To regenerate all encoding fixtures (including no-mapping.pdf):

# From the pdftract repository root
python3 tools/generate_encoding_fixtures.py

Note: The Python generator creates functionally equivalent fixtures with slight structural differences (object IDs, font names). The regenerated fixtures may have different SHA256 hashes but should produce identical extraction results.

This will regenerate all four encoding fixtures in tests/fixtures/encoding/:

• no-mapping.pdf — Custom glyph names (this fixture)
• agl-only.pdf — AGL glyph names
• fingerprint-match.pdf — Embedded Type1 subset
• shape-match.pdf — Custom glyph with shape database

Verification After Regeneration

After regeneration, verify the fixture with:

# 1. Check PDF structure
pdfinfo tests/fixtures/encoding/no-mapping.pdf
pdffonts tests/fixtures/encoding/no-mapping.pdf

# 2. Verify SHA256 checksum
sha256sum tests/fixtures/encoding/no-mapping.pdf
# Expected: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0

# 3. Verify ground truth extraction
cargo nextest run encoding_recovery

# 4. Update PROVENANCE.md if SHA256 changed
# Add new entry at tests/fixtures/PROVENANCE.md line 208

Manual Regeneration (Alternative)

If the binary fails to build, the fixture can be manually reconstructed from the raw PDF structure shown above. Create a file no-mapping.pdf with the exact content:

cat > tests/fixtures/encoding/no-mapping.pdf << 'EOF'
%PDF-1.4
1 0 obj
<<
/Type /Catalog
/Pages 2 0 R
>>
endobj
2 0 obj
<<
/Type /Pages
/Kids [3 0 R]
/Count 1
>>
endobj
3 0 obj
<<
/Type /Page
/Parent 2 0 R
/MediaBox [0 0 612 792]
/Resources <<
/Font <<
/F1 5 0 R
>>
>>
/Contents 4 0 R
>>
endobj
4 0 obj
<<
/Length 44
>>
stream
BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET
endstream
endobj
5 0 obj
<<
/Type /Font
/Subtype /Type1
/BaseFont /CustomNoMap
/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>
>>
endobj
xref
0 6
0000000000 65535 f 
0000000009 00000 n 
0000000058 00000 n 
0000000115 00000 n 
0000000241 00000 n 
0000000338 00000 n 
trailer
<<
/Size 6
/Root 1 0 R
>>
startxref
477
%%EOF
EOF

Inspection Commands

PDF Structure Inspection

# Basic PDF info
pdfinfo tests/fixtures/encoding/no-mapping.pdf

# Font details
pdffonts tests/fixtures/encoding/no-mapping.pdf

# View raw PDF structure (human-readable)
pdftk tests/fixtures/encoding/no-mapping.pdf dump_data output

# Extract and display content stream
pdftk tests/fixtures/encoding/no-mapping.pdf uncompress output /tmp/no-mapping-uncompressed.pdf
cat /tmp/no-mapping-uncompressed.pdf

Cross-Reference Table Verification

# Check xref table integrity
pdftk tests/fixtures/encoding/no-mapping.pdf dump_data verbose | grep -A 20 "xref"

Glyph Name Extraction

# Extract glyph names from encoding dictionary
grep -o "g[0-9]\{3\}" tests/fixtures/encoding/no-mapping.pdf
# Expected output: g001, g002, g003

Content Stream Inspection

# Show content stream (formatted)
strings tests/fixtures/encoding/no-mapping.pdf | grep -A 5 "BT"

Test Coverage

This fixture exercises the following test scenarios:

1. Custom Encoding Handling: Non-standard encoding with differences array
2. Missing ToUnicode: No CMap fallback for glyph-to-Unicode mapping
3. Non-AGL Glyph Names: Glyph names not in Adobe Glyph List
4. U+FFFD Emission: Proper replacement character generation
5. Error Resilience: Graceful handling of completely unmappable content

Related Tests

• tests/encoding_recovery.rs — Unicode recovery level tests
• tests/cjk_encoding.rs — CJK encoding with proper ToUnicode
• tests/encoding_recovery_integration.rs — Full extraction pipeline

Troubleshooting

Issue: Extraction produces wrong characters

Symptom: pdftract outputs "ABC" or other incorrect text instead of U+FFFD.

Diagnosis:

# Verify the fixture structure matches expected
pdffonts tests/fixtures/encoding/no-mapping.pdf | grep CustomNoMap
# Should show: CustomNoMap | Type 1 | Custom | no | no | no

Resolution:

• Ensure fixture was regenerated with correct version of generate_encoding_fixtures.rs
• Check that /g001, /g002, /g003 are not in AGL (grep -r g001 crates/pdftract-core/src/font/agl.rs)

Issue: SHA256 checksum mismatch

Symptom: Regenerated fixture has different hash.

Diagnosis:

# Compare file sizes
ls -l tests/fixtures/encoding/no-mapping.pdf
# Expected: 660 bytes

Resolution:

• Minimal differences (whitespace in PDF) are acceptable
• Update SHA256 in PROVENANCE.md if content is semantically identical
• If structure changed, investigate generate_encoding_fixtures.rs for unintended modifications

Issue: Ground test file mismatch

Symptom: Test expects "ABC" but gets "<22><><EFBFBD>" (or vice versa).

History:

• 2026-07-03 (bf-f0xqd): Ground truth corrected from "ABC" to U+FFFD
• The fixture was originally designed for glyph shape recovery (Level 4), but that feature is not yet implemented
• Current behavior correctly emits U+FFFD

Resolution:

• Current expected output is three U+FFFD characters
• Tests should verify replacement character emission, not glyph shape recovery

References

Plan Documentation

• Phase 2.3: Encoding recovery and Unicode mapping (lines 1420-1650)
• TH-03: Text extraction and encoding recovery tests
• INV-14: Unicode recovery levels and fallback strategies

Related Fixtures

• agl-only.pdf — Level 2 recovery (Adobe Glyph List)
• fingerprint-match.pdf — Level 3 recovery (font fingerprinting)
• shape-match.pdf — Level 4 recovery (glyph shape database)

Internal Documentation

• tests/fixtures/PROVENANCE.md — Fixture generation history
• tests/fixtures/generate_encoding_fixtures.rs — Generation source code
• crates/pdftract-core/src/font/ — Font resolution and encoding implementation