docs(bf-54x8x): document no-mapping.pdf fixture structure and regeneration
This commit is contained in:
jedarden
parent
8c2bf56dfd
commit
0ea2ae7b5c
1 changed files with 206 additions and 0 deletions
206
tests/fixtures/encoding/no-mapping.md
vendored
Normal file
206
tests/fixtures/encoding/no-mapping.md
vendored
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
# no-mapping.pdf Fixture
|
||||
|
||||
## Purpose
|
||||
|
||||
Level 4 Unicode recovery test fixture - worst case scenario where glyph names cannot be recovered via any standard method. This PDF uses custom glyph names (`/g001`, `/g002`, `/g003`) that are not in the Adobe Glyph List (AGL), has no ToUnicode CMap, and uses a custom encoding with no standard fallback.
|
||||
|
||||
Expected output: `<60><><EFBFBD>` (three U+FFFD replacement characters) since the glyph names have no mapping.
|
||||
|
||||
## Structure
|
||||
|
||||
### PDF Properties
|
||||
- **PDF Version:** 1.4
|
||||
- **Pages:** 1
|
||||
- **Page Size:** 612 x 792 pts (Letter)
|
||||
- **File Size:** 660 bytes
|
||||
- **Encrypted:** No
|
||||
- **Tagged:** No
|
||||
|
||||
### Font Details
|
||||
```
|
||||
Name: CustomNoMap
|
||||
Type: Type 1
|
||||
Encoding: Custom (Differences array)
|
||||
Embedded: No
|
||||
ToUnicode CMap: No
|
||||
```
|
||||
|
||||
**Font Object (4 0 obj):**
|
||||
```
|
||||
<<
|
||||
/Type /Font
|
||||
/Subtype /Type1
|
||||
/BaseFont /CustomNoMap
|
||||
/Encoding <<
|
||||
/Type /Encoding
|
||||
/Differences [0 /g001 /g002 /g003]
|
||||
>>
|
||||
>>
|
||||
```
|
||||
|
||||
### Content Stream
|
||||
|
||||
**Location:** Object 5 0 obj
|
||||
|
||||
**Raw Content:**
|
||||
```
|
||||
BT
|
||||
/F1 12 Tf
|
||||
50 700 Td
|
||||
<g001><g002><g003> Tj
|
||||
ET
|
||||
```
|
||||
|
||||
**Explanation:**
|
||||
- Sets font F1 (CustomNoMap) at 12pt
|
||||
- Moves to position (50, 700)
|
||||
- Shows three glyphs by name: g001, g002, g003
|
||||
- These glyph names are NOT in the AGL and cannot be mapped to Unicode
|
||||
|
||||
### Glyph Names
|
||||
|
||||
The content uses three custom glyph names:
|
||||
- `/g001` - No standard mapping (not in AGL)
|
||||
- `/g002` - No standard mapping (not in AGL)
|
||||
- `/g003` - No standard mapping (not in AGL)
|
||||
|
||||
Since these glyphs have no ToUnicode mapping and their names are not in any standard encoding or the AGL, they should render as U+FFFD (replacement character) during extraction.
|
||||
|
||||
## Inspection Commands
|
||||
|
||||
### Basic PDF info:
|
||||
```bash
|
||||
pdfinfo tests/fixtures/encoding/no-mapping.pdf
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
Pages: 1
|
||||
Page size: 612 x 792 pts (letter)
|
||||
File size: 660 bytes
|
||||
PDF version: 1.4
|
||||
```
|
||||
|
||||
### Font details:
|
||||
```bash
|
||||
pdffonts tests/fixtures/encoding/no-mapping.pdf
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
```
|
||||
name type encoding emb sub uni object ID
|
||||
------------------------------------ ----------------- ---------------- --- --- --- ---------
|
||||
CustomNoMap Type 1 Custom no no no 5 0
|
||||
```
|
||||
|
||||
### Content stream inspection:
|
||||
```bash
|
||||
# Extract raw content stream
|
||||
pdftk tests/fixtures/encoding/no-mapping.pdf unpack_files output /tmp/no-mapping-unpack/
|
||||
cat /tmp/no-mapping-unpack/doc_5_0.txt
|
||||
|
||||
# Or examine the hex dump
|
||||
hexdump -C tests/fixtures/encoding/no-mapping.pdf | grep -A 5 "g001"
|
||||
```
|
||||
|
||||
**Expected Content:**
|
||||
```
|
||||
<g001><g002><g003> Tj
|
||||
```
|
||||
|
||||
### Test extraction:
|
||||
```bash
|
||||
# Build pdftract first if needed
|
||||
cargo build --release
|
||||
|
||||
# Extract to see the U+FFFD replacement characters
|
||||
./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \
|
||||
--format markdown -o /tmp/no-mapping-output
|
||||
cat /tmp/no-mapping-output.md
|
||||
```
|
||||
|
||||
## Regeneration Instructions
|
||||
|
||||
### Method 1: Using the Rust generator (Recommended)
|
||||
|
||||
The fixture is generated by `tests/fixtures/generate_encoding_fixtures.rs`.
|
||||
|
||||
**Steps:**
|
||||
```bash
|
||||
# Build the generator binary
|
||||
cargo build --bin generate_encoding_fixtures
|
||||
|
||||
# Run the generator (creates all 4 encoding fixtures)
|
||||
./target/debug/generate_encoding_fixtures
|
||||
|
||||
# Or run directly via cargo
|
||||
cargo run --bin generate_encoding_fixtures
|
||||
```
|
||||
|
||||
**Output location:** `tests/fixtures/encoding/no-mapping.pdf`
|
||||
|
||||
**Ground truth:** `tests/fixtures/encoding/no-mapping.txt` (should contain `<60><><EFBFBD>`)
|
||||
|
||||
### Method 2: Manual reconstruction (Not recommended)
|
||||
|
||||
To manually reconstruct this PDF:
|
||||
|
||||
1. Create the 5 PDF objects as specified in `generate_encoding_fixtures.rs`
|
||||
2. Build the xref table with correct byte offsets
|
||||
3. Add the trailer with `/Size 6` and `/Root 1 0 R`
|
||||
4. Write the complete PDF to `tests/fixtures/encoding/no-mapping.pdf`
|
||||
|
||||
**Use the Rust generator instead** - it computes all offsets automatically and ensures the xref table is correct.
|
||||
|
||||
## Verification
|
||||
|
||||
After regeneration, verify:
|
||||
|
||||
```bash
|
||||
# Check SHA256 matches expected value
|
||||
sha256sum tests/fixtures/encoding/no-mapping.pdf
|
||||
# Expected: b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0
|
||||
|
||||
# Verify extraction produces U+FFFD characters
|
||||
./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \
|
||||
--format text -o /tmp/no-mapping-output.txt
|
||||
cat /tmp/no-mapping-output.txt | od -c
|
||||
# Should see: 0000000 f f f d f f f d f f f d
|
||||
```
|
||||
|
||||
## Test References
|
||||
|
||||
This fixture is used in the following tests:
|
||||
|
||||
- `tests/debug_encoding_pdf.rs` - Encoding recovery debug tests
|
||||
- `tests/debug_encoding_fixtures.rs` - Fixture validation tests
|
||||
- `tests/encoding_recovery.rs` - Unicode recovery integration tests
|
||||
|
||||
## Related Fixtures
|
||||
|
||||
- `agl-only.pdf` - Uses standard Type1 font with AGL glyph names (Level 2)
|
||||
- `fingerprint-match.pdf` - Uses embedded Type1 font subset (Level 3)
|
||||
- `shape-match.pdf` - Uses custom glyph names with shape recognition (Level 4)
|
||||
|
||||
## History
|
||||
|
||||
- **Generated:** 2026-06-09
|
||||
- **Regenerated:** 2026-07-02 (bf-512z1)
|
||||
- **Regenerated:** 2026-07-03 (bf-f0xqd) - Corrected ground truth from "ABC" to U+FFFD
|
||||
- **Regenerated:** 2026-07-03 (bf-1m30m) - Via `generate_encoding_fixtures.rs`
|
||||
- **SHA256:** `b24f88d3add958bfec1d6b134f2cd030cd41bb1932bedbe99405599bd01fa8f0`
|
||||
|
||||
## Notes
|
||||
|
||||
The key insight of this fixture is that it represents the **worst-case scenario** for Unicode recovery:
|
||||
1. No ToUnicode CMap
|
||||
2. Custom encoding (no StandardEncoding/WinAnsiEncoding/MacRomanEncoding fallback)
|
||||
3. Custom glyph names not in AGL (`/g001`, `/g002`, `/g003`)
|
||||
4. No embedded font program for fingerprinting
|
||||
|
||||
This forces the extractor to either:
|
||||
- Emit U+FFFD replacement characters (correct behavior)
|
||||
- Attempt glyph shape recognition (if implemented)
|
||||
- Fail gracefully
|
||||
|
||||
The fixture validates that pdftract handles unmapped glyphs correctly rather than producing garbage text or crashing.
|
||||
Loading…
Add table
Reference in a new issue