206 lines
5.7 KiB
Markdown
206 lines
5.7 KiB
Markdown
# no-mapping.pdf Fixture
|
||
|
||
## Purpose
|
||
|
||
Level 4 Unicode recovery test fixture - worst case scenario where glyph names cannot be recovered via any standard method. This PDF uses custom glyph names (`/g001`, `/g002`, `/g003`) that are not in the Adobe Glyph List (AGL), has no ToUnicode CMap, and uses a custom encoding with no standard fallback.
|
||
|
||
Expected output: `<60><><EFBFBD>` (three U+FFFD replacement characters) since the glyph names have no mapping.
|
||
|
||
## Structure
|
||
|
||
### PDF Properties
|
||
- **PDF Version:** 1.4
|
||
- **Pages:** 1
|
||
- **Page Size:** 612 x 792 pts (Letter)
|
||
- **File Size:** 660 bytes
|
||
- **Encrypted:** No
|
||
- **Tagged:** No
|
||
|
||
### Font Details
|
||
```
|
||
Name: CustomNoMap
|
||
Type: Type 1
|
||
Encoding: Custom (Differences array)
|
||
Embedded: No
|
||
ToUnicode CMap: No
|
||
```
|
||
|
||
**Font Object (4 0 obj):**
|
||
```
|
||
<<
|
||
/Type /Font
|
||
/Subtype /Type1
|
||
/BaseFont /CustomNoMap
|
||
/Encoding <<
|
||
/Type /Encoding
|
||
/Differences [0 /g001 /g002 /g003]
|
||
>>
|
||
>>
|
||
```
|
||
|
||
### Content Stream
|
||
|
||
**Location:** Object 5 0 obj
|
||
|
||
**Raw Content:**
|
||
```
|
||
BT
|
||
/F1 12 Tf
|
||
50 700 Td
|
||
<g001><g002><g003> Tj
|
||
ET
|
||
```
|
||
|
||
**Explanation:**
|
||
- Sets font F1 (CustomNoMap) at 12pt
|
||
- Moves to position (50, 700)
|
||
- Shows three glyphs by name: g001, g002, g003
|
||
- These glyph names are NOT in the AGL and cannot be mapped to Unicode
|
||
|
||
### Glyph Names
|
||
|
||
The content uses three custom glyph names:
|
||
- `/g001` - No standard mapping (not in AGL)
|
||
- `/g002` - No standard mapping (not in AGL)
|
||
- `/g003` - No standard mapping (not in AGL)
|
||
|
||
Since these glyphs have no ToUnicode mapping and their names are not in any standard encoding or the AGL, they should render as U+FFFD (replacement character) during extraction.
|
||
|
||
## Inspection Commands
|
||
|
||
### Basic PDF info:
|
||
```bash
|
||
pdfinfo tests/fixtures/encoding/no-mapping.pdf
|
||
```
|
||
|
||
**Expected Output:**
|
||
```
|
||
Pages: 1
|
||
Page size: 612 x 792 pts (letter)
|
||
File size: 660 bytes
|
||
PDF version: 1.4
|
||
```
|
||
|
||
### Font details:
|
||
```bash
|
||
pdffonts tests/fixtures/encoding/no-mapping.pdf
|
||
```
|
||
|
||
**Expected Output:**
|
||
```
|
||
name type encoding emb sub uni object ID
|
||
------------------------------------ ----------------- ---------------- --- --- --- ---------
|
||
CustomNoMap Type 1 Custom no no no 5 0
|
||
```
|
||
|
||
### Content stream inspection:
|
||
```bash
|
||
# Extract raw content stream
|
||
pdftk tests/fixtures/encoding/no-mapping.pdf unpack_files output /tmp/no-mapping-unpack/
|
||
cat /tmp/no-mapping-unpack/doc_5_0.txt
|
||
|
||
# Or examine the hex dump
|
||
hexdump -C tests/fixtures/encoding/no-mapping.pdf | grep -A 5 "g001"
|
||
```
|
||
|
||
**Expected Content:**
|
||
```
|
||
<g001><g002><g003> Tj
|
||
```
|
||
|
||
### Test extraction:
|
||
```bash
|
||
# Build pdftract first if needed
|
||
cargo build --release
|
||
|
||
# Extract to see the U+FFFD replacement characters
|
||
./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \
|
||
--format markdown -o /tmp/no-mapping-output
|
||
cat /tmp/no-mapping-output.md
|
||
```
|
||
|
||
## Regeneration Instructions
|
||
|
||
### Method 1: Using the Rust generator (Recommended)
|
||
|
||
The fixture is generated by `tests/fixtures/generate_encoding_fixtures.rs`.
|
||
|
||
**Steps:**
|
||
```bash
|
||
# Build the generator binary
|
||
cargo build --bin generate_encoding_fixtures
|
||
|
||
# Run the generator (creates all 4 encoding fixtures)
|
||
./target/debug/generate_encoding_fixtures
|
||
|
||
# Or run directly via cargo
|
||
cargo run --bin generate_encoding_fixtures
|
||
```
|
||
|
||
**Output location:** `tests/fixtures/encoding/no-mapping.pdf`
|
||
|
||
**Ground truth:** `tests/fixtures/encoding/no-mapping.txt` (should contain `<60><><EFBFBD>`)
|
||
|
||
### Method 2: Manual reconstruction (Not recommended)
|
||
|
||
To manually reconstruct this PDF:
|
||
|
||
1. Create the 5 PDF objects as specified in `generate_encoding_fixtures.rs`
|
||
2. Build the xref table with correct byte offsets
|
||
3. Add the trailer with `/Size 6` and `/Root 1 0 R`
|
||
4. Write the complete PDF to `tests/fixtures/encoding/no-mapping.pdf`
|
||
|
||
**Use the Rust generator instead** - it computes all offsets automatically and ensures the xref table is correct.
|
||
|
||
## Verification
|
||
|
||
After regeneration, verify:
|
||
|
||
```bash
|
||
# Check SHA256 matches expected value
|
||
sha256sum tests/fixtures/encoding/no-mapping.pdf
|
||
# Expected: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0
|
||
|
||
# Verify extraction produces U+FFFD characters
|
||
./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \
|
||
--format text -o /tmp/no-mapping-output.txt
|
||
cat /tmp/no-mapping-output.txt | od -c
|
||
# Should see: 0000000 f f f d f f f d f f f d
|
||
```
|
||
|
||
## Test References
|
||
|
||
This fixture is used in the following tests:
|
||
|
||
- `tests/debug_encoding_pdf.rs` - Encoding recovery debug tests
|
||
- `tests/debug_encoding_fixtures.rs` - Fixture validation tests
|
||
- `tests/encoding_recovery.rs` - Unicode recovery integration tests
|
||
|
||
## Related Fixtures
|
||
|
||
- `agl-only.pdf` - Uses standard Type1 font with AGL glyph names (Level 2)
|
||
- `fingerprint-match.pdf` - Uses embedded Type1 font subset (Level 3)
|
||
- `shape-match.pdf` - Uses custom glyph names with shape recognition (Level 4)
|
||
|
||
## History
|
||
|
||
- **Generated:** 2026-06-09
|
||
- **Regenerated:** 2026-07-02 (bf-512z1)
|
||
- **Regenerated:** 2026-07-03 (bf-f0xqd) - Corrected ground truth from "ABC" to U+FFFD
|
||
- **Regenerated:** 2026-07-03 (bf-1m30m) - Via `generate_encoding_fixtures.rs`
|
||
- **SHA256:** `b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0`
|
||
|
||
## Notes
|
||
|
||
The key insight of this fixture is that it represents the **worst-case scenario** for Unicode recovery:
|
||
1. No ToUnicode CMap
|
||
2. Custom encoding (no StandardEncoding/WinAnsiEncoding/MacRomanEncoding fallback)
|
||
3. Custom glyph names not in AGL (`/g001`, `/g002`, `/g003`)
|
||
4. No embedded font program for fingerprinting
|
||
|
||
This forces the extractor to either:
|
||
- Emit U+FFFD replacement characters (correct behavior)
|
||
- Attempt glyph shape recognition (if implemented)
|
||
- Fail gracefully
|
||
|
||
The fixture validates that pdftract handles unmapped glyphs correctly rather than producing garbage text or crashing.
|