pdftract/tests/fixtures/no-mapping.md

# no-mapping.pdf Fixture Documentation

## Overview

The `no-mapping.pdf` fixture is a **Level 4 Unicode recovery test case** designed to exercise the worst-case scenario for text extraction: a PDF with custom glyph names that cannot be mapped to Unicode characters through any standard method.

**Location:** `tests/fixtures/encoding/no-mapping.pdf`  
**Generated:** 2026-06-09  
**Last Regenerated:** 2026-07-03  
**SHA256:** `b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0`

## Fixture Structure

### PDF Properties

```bash
$ pdfinfo tests/fixtures/encoding/no-mapping.pdf
Pages:           1
Page size:       612 x 792 pts (letter)
Page rot:        0
File size:       660 bytes
PDF version:     1.4
Encrypted:       no
```

### Font Information

```bash
$ pdffonts tests/fixtures/encoding/no-mapping.pdf
name                                 type              encoding         emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
CustomNoMap                          Type 1            Custom           no  no  no       5  0
```

**Key characteristics:**
- **Font Name:** `/CustomNoMap` — non-standard, custom font
- **Subtype:** Type1
- **Encoding:** Custom encoding with differences array
- **Embedded:** No — font program not embedded
- **Subset:** No
- **ToUnicode:** No CMap present

### Content Stream Analysis

The PDF contains a single content stream that displays three glyphs by name:

```pdf
BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET
```

The content references glyphs `/g001`, `/g002`, and `/g003` through a custom encoding dictionary.

### Custom Encoding Dictionary

```pdf
/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>
```

This mapping assigns:
- Position 0 → `/g001`
- Position 1 → `/g002`  
- Position 2 → `/g003`

**Critical:** The glyph names `/g001`, `/g002`, `/g003` are **not in the Adobe Glyph List (AGL)**, so they cannot be mapped to standard Unicode characters.

### Raw PDF Structure

The fixture is a minimal hand-written PDF with 5 indirect objects:

```pdf
%PDF-1.4
1 0 obj           # Catalog
<<
/Type /Catalog
/Pages 2 0 R
>>
endobj

2 0 obj           # Pages node
<<
/Type /Pages
/Kids [3 0 R]
/Count 1
>>
endobj

3 0 obj           # Page
<<
/Type /Page
/Parent 2 0 R
/MediaBox [0 0 612 792]
/Resources <<
/Font <<
/F1 5 0 R
>>
>>
/Contents 4 0 R
>>
endobj

4 0 obj           # Content stream
<<
/Length 44
>>
stream
BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET
endstream
endobj

5 0 obj           # Font dictionary
<<
/Type /Font
/Subtype /Type1
/BaseFont /CustomNoMap
/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>
>>
endobj
```

## PDF Structure

Based on analysis from bead bf-4ozvw, the fixture has the following structural characteristics:

### File Characteristics

- **Size:** Extremely small (660 bytes) — minimal fixture
- **Pages:** Single page document  
- **Format:** PDF 1.4 (standard version)
- **Page Size:** US Letter (612 x 792 pts)
- **Complexity:** No JavaScript, forms, or metadata streams
- **Optimization:** Not optimized
- **User Properties:** None
- **Tagged PDF:** No
- **Metadata Stream:** No

### Font Analysis

- **Font Name:** `CustomNoMap` — descriptive name indicating purpose
- **Font Type:** Type 1 (PostScript font)
- **Encoding:** Custom — this is the key characteristic
- **Embedding:** Not embedded (`emb: no`)
- **Subsetting:** Not subset (`sub: no`)
- **Unicode:** No Unicode mapping (`uni: no`)
- **Object ID:** 5 0 R

### Technical Significance

This fixture is particularly valuable for testing:

1. **Font encoding edge cases** — Custom encodings that don't map to standard character sets
2. **Glyph recognition without standard mappings** — Glyphs with no direct Unicode equivalent
3. **Error handling in the text extraction pipeline** — Fallback strategies when encoding mappings are unavailable
4. **No Font Embedding scenario** — PDF readers must rely on encoding information provided

The fixture name "no-mapping" combined with the custom encoding indicates this is intentionally testing the case where pdftract must handle glyphs that have no direct mapping to readable text — a critical edge case for robust PDF text extraction.

## Expected Extraction Behavior

### Ground Truth

The `tests/fixtures/encoding/no-mapping.txt` ground truth file contains:

```
<EFBFBD><EFBFBD><EFBFBD>
```

This is **three U+FFFD (REPLACEMENT CHARACTER)** codepoints, representing the three unmapped glyphs.

### Why U+FFFD?

The glyph names `/g001`, `/g002`, `/g003` cannot be recovered through any of the standard Unicode recovery levels:

1. **Level 1 (ToUnicode CMap):** ❌ No ToUnicode CMap present
2. **Level 2 (Adobe Glyph List):** ❌ Glyph names not in AGL
3. **Level 3 (Font Fingerprinting):** ❌ Font not embedded, no fingerprint available
4. **Level 4 (Glyph Shape Recognition):** ❌ No embedded glyph outlines

Since all recovery methods fail, pdftract should emit U+FFFD replacement characters to indicate unmappable content.

### Extraction Verification

```bash
$ cargo run --release -- pdftract mcp tests/fixtures/encoding/no-mapping.pdf
# Should output: <20><><EFBFBD> (three U+FFFD characters)
```

Or with the CLI:

```bash
$ cargo run --release --bin pdftract-cli -- extract \
    tests/fixtures/encoding/no-mapping.pdf
# Expected: Three replacement characters in output
```

## Regeneration Instructions

### Prerequisites

- Rust toolchain (1.70+)
- Working cargo build environment

### Regeneration Command

The fixture is generated by the Python script at `tools/generate_encoding_fixtures.py`.

To regenerate **all** encoding fixtures (including no-mapping.pdf):

```bash
# From the pdftract repository root
python3 tools/generate_encoding_fixtures.py
```

**Note:** The Python generator creates functionally equivalent fixtures with slight structural differences (object IDs, font names). The regenerated fixtures may have different SHA256 hashes but should produce identical extraction results.

This will regenerate **all four encoding fixtures** in `tests/fixtures/encoding/`:
- `no-mapping.pdf` — Custom glyph names (this fixture)
- `agl-only.pdf` — AGL glyph names
- `fingerprint-match.pdf` — Embedded Type1 subset
- `shape-match.pdf` — Custom glyph with shape database

### Verification After Regeneration

After regeneration, verify the fixture with:

```bash
# 1. Check PDF structure
pdfinfo tests/fixtures/encoding/no-mapping.pdf
pdffonts tests/fixtures/encoding/no-mapping.pdf

# 2. Verify SHA256 checksum
sha256sum tests/fixtures/encoding/no-mapping.pdf
# Expected: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0

# 3. Verify ground truth extraction
cargo nextest run encoding_recovery

# 4. Update PROVENANCE.md if SHA256 changed
# Add new entry at tests/fixtures/PROVENANCE.md line 208
```

### Manual Regeneration (Alternative)

If the binary fails to build, the fixture can be manually reconstructed from the raw PDF structure shown above. Create a file `no-mapping.pdf` with the exact content:

```bash
cat > tests/fixtures/encoding/no-mapping.pdf << 'EOF'
%PDF-1.4
1 0 obj
<<
/Type /Catalog
/Pages 2 0 R
>>
endobj
2 0 obj
<<
/Type /Pages
/Kids [3 0 R]
/Count 1
>>
endobj
3 0 obj
<<
/Type /Page
/Parent 2 0 R
/MediaBox [0 0 612 792]
/Resources <<
/Font <<
/F1 5 0 R
>>
>>
/Contents 4 0 R
>>
endobj
4 0 obj
<<
/Length 44
>>
stream
BT
/F1 12 Tf
50 700 Td
<g001><g002><g003> Tj
ET
endstream
endobj
5 0 obj
<<
/Type /Font
/Subtype /Type1
/BaseFont /CustomNoMap
/Encoding <<
/Type /Encoding
/Differences [0 /g001 /g002 /g003]
>>
>>
endobj
xref
0 6
0000000000 65535 f 
0000000009 00000 n 
0000000058 00000 n 
0000000115 00000 n 
0000000241 00000 n 
0000000338 00000 n 
trailer
<<
/Size 6
/Root 1 0 R
>>
startxref
477
%%EOF
EOF
```

## Inspection Commands

### PDF Structure Inspection

```bash
# Basic PDF info
pdfinfo tests/fixtures/encoding/no-mapping.pdf

# Font details
pdffonts tests/fixtures/encoding/no-mapping.pdf

# View raw PDF structure (human-readable)
pdftk tests/fixtures/encoding/no-mapping.pdf dump_data output

# Extract and display content stream
pdftk tests/fixtures/encoding/no-mapping.pdf uncompress output /tmp/no-mapping-uncompressed.pdf
cat /tmp/no-mapping-uncompressed.pdf
```

### Cross-Reference Table Verification

```bash
# Check xref table integrity
pdftk tests/fixtures/encoding/no-mapping.pdf dump_data verbose | grep -A 20 "xref"
```

### Glyph Name Extraction

```bash
# Extract glyph names from encoding dictionary
grep -o "g[0-9]\{3\}" tests/fixtures/encoding/no-mapping.pdf
# Expected output: g001, g002, g003
```

### Content Stream Inspection

```bash
# Show content stream (formatted)
strings tests/fixtures/encoding/no-mapping.pdf | grep -A 5 "BT"
```

## Test Coverage

This fixture exercises the following test scenarios:

1. **Custom Encoding Handling:** Non-standard encoding with differences array
2. **Missing ToUnicode:** No CMap fallback for glyph-to-Unicode mapping
3. **Non-AGL Glyph Names:** Glyph names not in Adobe Glyph List
4. **U+FFFD Emission:** Proper replacement character generation
5. **Error Resilience:** Graceful handling of completely unmappable content

### Related Tests

- `tests/encoding_recovery.rs` — Unicode recovery level tests
- `tests/cjk_encoding.rs` — CJK encoding with proper ToUnicode
- `tests/encoding_recovery_integration.rs` — Full extraction pipeline

## Troubleshooting

### Issue: Extraction produces wrong characters

**Symptom:** pdftract outputs "ABC" or other incorrect text instead of U+FFFD.

**Diagnosis:**
```bash
# Verify the fixture structure matches expected
pdffonts tests/fixtures/encoding/no-mapping.pdf | grep CustomNoMap
# Should show: CustomNoMap | Type 1 | Custom | no | no | no
```

**Resolution:** 
- Ensure fixture was regenerated with correct version of `generate_encoding_fixtures.rs`
- Check that `/g001`, `/g002`, `/g003` are not in AGL (`grep -r g001 crates/pdftract-core/src/font/agl.rs`)

### Issue: SHA256 checksum mismatch

**Symptom:** Regenerated fixture has different hash.

**Diagnosis:**
```bash
# Compare file sizes
ls -l tests/fixtures/encoding/no-mapping.pdf
# Expected: 660 bytes
```

**Resolution:**
- Minimal differences (whitespace in PDF) are acceptable
- Update SHA256 in PROVENANCE.md if content is semantically identical
- If structure changed, investigate `generate_encoding_fixtures.rs` for unintended modifications

### Issue: Ground test file mismatch

**Symptom:** Test expects "ABC" but gets "<22><><EFBFBD>" (or vice versa).

**History:**
- 2026-07-03 (bf-f0xqd): Ground truth corrected from "ABC" to U+FFFD
- The fixture was originally designed for glyph shape recovery (Level 4), but that feature is not yet implemented
- Current behavior correctly emits U+FFFD

**Resolution:**
- Current expected output is three U+FFFD characters
- Tests should verify replacement character emission, not glyph shape recovery

## References

### Plan Documentation

- **Phase 2.3:** Encoding recovery and Unicode mapping (lines 1420-1650)
- **TH-03:** Text extraction and encoding recovery tests  
- **INV-14:** Unicode recovery levels and fallback strategies

### Related Fixtures

- `agl-only.pdf` — Level 2 recovery (Adobe Glyph List)
- `fingerprint-match.pdf` — Level 3 recovery (font fingerprinting)
- `shape-match.pdf` — Level 4 recovery (glyph shape database)

### Internal Documentation

- `tests/fixtures/PROVENANCE.md` — Fixture generation history
- `tests/fixtures/generate_encoding_fixtures.rs` — Generation source code
- `crates/pdftract-core/src/font/` — Font resolution and encoding implementation