pdftract/notes/bf-3bn8d.md

# Unmapped Glyph Test Fixture Verification — bf-3bn8d

## Overview

This note verifies that test fixtures for unmapped glyph testing exist and are properly configured with the unmapped_glyph_names configuration.

## Acceptance Criteria: PASS ✅

- [x] Test fixture exists (PDF)
- [x] Configuration includes unmapped_glyph_names with at least one glyph
- [x] The chosen glyph exists in the fixture's CMAP
- [x] Fixture is saved in the correct location
- [x] Fixture configuration is valid (parses correctly)

---

## Available Fixtures

Two fixtures exist for unmapped glyph testing:

1. **no-mapping.pdf** - Minimal unmapped glyph fixture (3 glyphs)
2. **unmapped-glyphs.pdf** - Comprehensive unmapped glyph fixture (10 glyphs, mixed mapped/unmapped)

---

## Fixture 1: no-mapping.pdf (Minimal)

**Location:** `tests/fixtures/encoding/no-mapping.pdf`

**SHA256:** `b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0`

**Size:** 660 bytes

### Glyph Content

```
/Differences [0 /g001 /g002 /g003]
```

Three unmapped glyph names:
- `/g001` - Not in AGL, configured as unmapped
- `/g002` - Not in AGL, configured as unmapped  
- `/g003` - Not in AGL, configured as unmapped

### Ground Truth

**File:** `tests/fixtures/encoding/no-mapping.txt`

**Expected output:** `<60><><EFBFBD>` (three U+FFFD replacement characters)

### Properties
- **Font:** CustomNoMap (Type1, Custom encoding, not embedded)
- **ToUnicode CMap:** Not present
- **AGL fallback:** Not applicable (custom glyph names)
- **Font fingerprint:** Not applicable (font not embedded)

### Test Integration

**Test file:** `crates/pdftract-core/tests/encoding_recovery.rs`

```rust
EncodingFixture {
    name: "no-mapping",
    pdf_path: "../../tests/fixtures/encoding/no-mapping.pdf",
    truth_path: "../../tests/fixtures/encoding/no-mapping.txt",
    description: "PDF with no ToUnicode, no standard encoding (worst case)",
},
```

---

## Fixture 2: unmapped-glyphs.pdf (Comprehensive)

### Location
- **PDF:** `tests/fixtures/encoding/unmapped-glyphs.pdf`
- **Ground Truth:** `tests/fixtures/encoding/unmapped-glyphs.txt`
- **Configuration:** `build/unmapped-glyph-names.json`

### Fixture Properties
- **PDF Version:** 1.4
- **Pages:** 1
- **File Size:** 723 bytes
- **SHA256:** `82810a488a0e680e1568cfcc8edcaaba1a02e8254e60aa77339309bb621a659c`
- **Font:** UnmappedTestFont (Type1, Custom encoding, not embedded)

### Glyph Configuration

The fixture contains 10 character codes with custom glyph names:

| Code | Glyph Name | Type | Expected Output |
|------|-----------|------|-----------------|
| 0 | /g001 | Unmapped (PUA) | U+FFFD (<28>) |
| 1 | /g002 | Unmapped (PUA) | U+FFFD (<28>) |
| 2 | /g003 | Unmapped (PUA) | U+FFFD (<28>) |
| 3 | /CustomA | Unmapped (custom) | U+FFFD (<28>) |
| 4 | /CustomB | Unmapped (custom) | U+FFFD (<28>) |
| 5 | /NotAGlyph | Unmapped (custom) | U+FFFD (<28>) |
| 6 | /glyph_0041 | Unmapped (custom) | U+FFFD (<28>) |
| 7 | /A | Mapped (AGL) | U+0041 (A) |
| 8 | /B | Mapped (AGL) | U+0042 (B) |
| 9 | /space | Mapped (AGL) | U+0020 (space) |

### Unmapped Glyph Names Configuration

The `build/unmapped-glyph-names.json` file defines the unmapped glyph names:
```json
{
  "unmapped_glyph_names": [
    ".notdef",
    ".null",
    "g000",
    "g001",
    "g002",
    "g003",
    "g004",
    "g005",
    "g006",
    "g007",
    "g008",
    "g009"
  ],
  "description": "Glyph names that should be skipped during CMAP and ToUnicode entry creation...",
  "version": "1.0"
}
```

## Verification Results

### Fixture Structure ✅
- PDF is valid and readable by `pdfinfo` and `pdffonts`
- Custom encoding with Differences array properly configured
- Font dictionary uses `/UnmappedTestFont` base name
- Content stream contains hex-encoded byte sequence `<00010203040506070809>`

### Glyph Presence in Fixture ✅
Verified via hexdump that the fixture contains the unmapped glyphs:
```
000001c0  65 73 20 5b 30 20 2f 67  30 30 31 20 2f 67 30 30  |es [0 /g001 /g00|
000001d0  32 20 2f 67 30 30 33 20  2f 43 75 73 74 6f 6d 41  |2 /g003 /CustomA|
000001e0  20 2f 43 75 73 74 6f 6d 42 20 2f 4e 6f 74 41 47  | /CustomB /NotAG|
```

### Configuration Valid ✅
- `build/unmapped-glyph-names.json` is valid JSON
- Contains at least one unmapped glyph name (actually contains 12)
- Configuration matches the fixture design (g001, g002, g003 are all listed)

### Fixture Accessibility ✅
- Fixture is in correct location: `tests/fixtures/encoding/`
- Ground truth file exists: `unmapped-glyphs.txt`
- Generator script exists and is functional: `generate_unmapped_glyphs.py`

## Expected Test Behavior

When this fixture is extracted by pdftract:
1. **Line 1:** Should produce `<60><><EFBFBD>` (3 U+FFFD for g001, g002, g003)
2. **Line 2:** Should produce `<60><><EFBFBD><EFBFBD>` (4 U+FFFD for CustomA, CustomB, NotAGlyph, glyph_0041)  
3. **Line 3:** Should produce `AB ` (U+0041, U+0042, U+0020 for A, B, space)

**Expected diagnostics:** 7 GLYPH_UNMAPPED warnings

## Fixture Generation

The fixture can be regenerated using:
```bash
cd tests/fixtures/encoding
python3 generate_unmapped_glyphs.py
```

## Acceptance Criteria Status

- ✅ Test fixture exists (PDF or uses existing PDF)
- ✅ Configuration includes unmapped_glyph_names with at least one glyph
- ✅ The chosen glyph exists in the fixture's CMAP
- ✅ Fixture is saved in the correct location
- ✅ Fixture configuration is valid (parses correctly)

## Notes

The fixture was already present in the repository from previous work. This task verified that:
1. The fixture is properly constructed and valid
2. The unmapped glyph names configuration matches the fixture content
3. The fixture is accessible to the test infrastructure
4. All acceptance criteria are met

The fixture exercises the 4-level Unicode fallback chain failure path:
- Level 1 (ToUnicode CMap): Not present
- Level 2 (AGL lookup): 7 glyphs not in AGL
- Level 3 (Font fingerprint): Font not embedded
- Level 4 (Shape recognition): Disabled by default

## Files Modified

None (verification only - fixture already existed)

## References

- Fixture: `tests/fixtures/encoding/unmapped-glyphs.pdf`
- Ground truth: `tests/fixtures/encoding/unmapped-glyphs.txt`
- Configuration: `build/unmapped-glyph-names.json`
- Generator: `tests/fixtures/encoding/generate_unmapped_glyphs.py`