diff --git a/notes/bf-e4uvb-child-3-cmap-flow.md b/notes/bf-e4uvb-child-3-cmap-flow.md new file mode 100644 index 0000000..59bb424 --- /dev/null +++ b/notes/bf-e4uvb-child-3-cmap-flow.md @@ -0,0 +1,330 @@ +# CMAP Data Flow and Glyph Processing Documentation + +## Overview + +This document describes how CMAP (Character Map) processes glyph names and creates encoding entries in the pdftract codebase. The CMAP system is a critical component of the 4-level encoding resolution chain for text extraction from PDFs. + +## Data Flow Architecture + +### Level 1: ToUnicode CMap (Confidence 1.0) + +The primary and most reliable source for glyph-to-Unicode mapping is the `/ToUnicode` CMap stream embedded in PDF fonts. + +#### Entry Creation Flow + +1. **CMap Parsing** (`crates/pdftract-core/src/font/cmap.rs:86-88`) + - **Entry Point**: `ToUnicodeMap::add_mapping()` method + - **MARKER**: Line 85-88 explicitly marks this as the CMAP entry creation point + - Called by: + - `parse_beginbfchar()` (single character mappings) + - `parse_beginbfrange()` (range mappings) + +2. **Data Structure**: `ToUnicodeMap` (`cmap.rs:67-71`) + ```rust + pub struct ToUnicodeMap { + /// Mapping from source byte sequence to destination Unicode codepoints. + /// Uses Vec as key (source bytes) and Vec as value (destination chars). + mappings: HashMap, Vec>, + } + ``` + +3. **Mapping Types Processed**: + + a. **Single Character Mappings** (`beginbfchar...endbfchar` blocks) + - Format: `beginbfchar ... endbfchar` + - Example: `beginbfchar 1 <00> <0041> endbfchar` maps byte `0x00` to 'A' + - Supports ligature expansion: `<00660069>` → `['f', 'i']` (two codepoints) + + b. **Range Mappings** (`beginbfrange...endbfrange` blocks) + - **Contiguous form**: ` ` - maps range with incrementing destination + - Example: `beginbfrange 1 <0041> <005A> <0041> endbfrange` maps A-Z to U+0041..U+005A + - **Explicit array form**: ` [ ...]` - each destination explicitly specified + - Example: `beginbfrange 1 <0001> <0003> [ ] endbfrange` + - Maps codes 1,2,3 to ligatures fi, fl, ffi + +4. **Glyph Names Processed**: + - CMap does NOT process glyph names directly + - It processes **raw byte sequences** (character codes) as keys + - Source byte sequences can be 1-4 bytes (variable-length) + - Destination is always Unicode codepoints (UTF-16BE decoded) + +5. **UTF-16BE Decoding** (`cmap.rs:329-367`) + - Hex strings in CMap are UTF-16BE encoded + - Decoded to `Vec` supporting surrogate pairs and multi-codepoint results + - Unpaired surrogates produce replacement character (U+FFFD) + - Odd byte counts emit diagnostic but continue processing + +#### Lookup Flow + +1. **Resolution Entry** (`resolver.rs:302-377`) + - `resolve_unicode()` is the main entry point + - Checks per-font cache first (thread-safe LRU) + - Calls `resolve_level1()` for ToUnicode lookup + +2. **Level 1 Lookup** (`resolver.rs:383-399`) + ```rust + fn resolve_level1(char_code: &[u8], to_unicode: Option<&ToUnicodeMap>) -> ResolvedGlyph + ``` + - Returns failure if ToUnicode CMap is missing + - Returns failure if character code not in map + - Returns failure if result is empty or U+FFFD only + - Success: Returns `ResolvedGlyph` with chars, source=ToUnicode, confidence=1.0 + +### Level 2: Named Encoding + AGL (Confidence 0.9) + +When ToUnicode is absent or fails, the system falls back to named encoding + AGL lookup. + +#### Encoding Processing + +1. **Named Encodings** (`encoding.rs:22-60`) + - 6 standard encodings: WinAnsi, MacRoman, MacExpert, Standard, Symbol, ZapfDingbats + - Map character codes (0-255) to glyph names + - WinAnsiEncoding is most common (Windows-1252 superset) + +2. **Differences Overlay** (`encoding.rs:129-150`) + - Sparse overrides from `/Differences` array + - Format: `[n /Name1 /Name2 ... m /OtherName ...]` + - Each integer resets position, subsequent names assign to consecutive codes + - Example: `[39 /quotesingle 96 /grave]` assigns code 39→"quotesingle", 96→"grave" + +#### AGL Lookup (Glyph Name Processing) + +1. **Adobe Glyph List** (`agl.rs:41-60`) + - Compile-time phf::Map of ~4,400 glyph name → Unicode mappings + - Generated from Adobe's glyphlist.txt at build time + +2. **Algorithmic Patterns** (`agl.rs:96-100`) + - `uniXXXX` (4 hex digits) → Unicode codepoint + - `uXXXXXX` (up to 6 hex digits) → Unicode codepoint + - Example: `uni20AC` → U+20AC (Euro sign) + +3. **Variant Suffix Stripping** (`agl.rs:47-56`) + - Strips suffixes like `.sc`, `.alt`, `.oldstyle` before lookup + - Example: `A.sc` → `A` → U+0041 + +4. **Ligature Support** (`agl.rs:79-94`) + - `unicode_for_glyph_name_multi()` handles multi-codepoint glyphs + - Example: `fi` → `['f', 'i']` + +#### Level 2 Resolution + +- Character code → Encoding table → Glyph name → AGL lookup → Unicode +- Returns `ResolvedGlyph` with source=Agl, confidence=0.9 + +### Level 3: Font Fingerprint Cache (Confidence 0.85) + +When both ToUnicode and AGL fail, the system uses font fingerprinting. + +#### Fingerprint Processing (`fingerprint.rs`) + +1. **Hash Computation** + - SHA-256 hash of decoded font program bytes + - Computed over embedded font program (post-stream decode) + - Stable across different stream filters + +2. **Lookup Database** + - Compile-time phf::Map: `[u8; 32]` → `&'static [(u16, u32)]` + - Maps glyph ID (GID) to Unicode codepoint + - Generated from `build/font-fingerprints.json` + +3. **Fallback Trigger** +- Used when: + - `/ToUnicode` is missing or empty + - Embedded font subset has stripped glyph names + - Font binary matches a known fingerprint + +### Level 4: Shape Recognition (Confidence 0.7, cfg-gated) + +Final fallback: visual glyph shape matching. + +#### Shape Processing (`shape.rs`) + +1. **Rasterization** + - Type 3 glyphs: Execute content stream to 32×32 bitmap + - Embedded fonts: Extract glyph outline, render to bitmap + +2. **Perceptual Hashing** + - Compute pHash of bitmap + - Look up in shape database (compile-time phf::Map) + +3. **Feature-Gated** + - Only available when `shape-db` feature is enabled + - Requires `build/glyph-shapes.json` at compile time + +## Key Data Structures + +### 1. `ToUnicodeMap` (`cmap.rs:67-71`) +```rust +pub struct ToUnicodeMap { + mappings: HashMap, Vec>, +} +``` +- **Purpose**: Stores character code → Unicode mappings from `/ToUnicode` stream +- **Key**: Variable-length byte sequence (1-4 bytes) +- **Value**: Unicode codepoints (1-4 chars for ligatures) +- **Entry point**: `add_mapping()` at line 86 + +### 2. `ResolvedGlyph` (`resolver.rs:168-175`) +```rust +pub struct ResolvedGlyph { + pub chars: SmallVec<[char; 4]>, + pub source: UnicodeSource, + pub confidence: f32, +} +``` +- **Purpose**: Result of encoding resolution at any level +- **chars**: 1-4 Unicode characters (supports ligature expansion) +- **source**: Which level produced the mapping (ToUnicode, Agl, Fingerprint, ShapeMatch, Unknown) +- **confidence**: Always 1.0, 0.9, 0.85, or 0.7 per INV-30 + +### 3. `FontEncoding` (`encoding.rs`) +```rust +pub enum FontEncoding { + Named(NamedEncoding), + Custom(DifferencesOverlay), +} +``` +- **Purpose**: Level 2 character code → glyph name mapping +- **Named**: One of 6 standard encodings (WinAnsi, MacRoman, etc.) +- **Custom**: Base encoding + Differences array overlay + +### 4. `ResolverCache` (`resolver.rs:223-278`) +```rust +pub struct ResolverCache { + cache: DashMap, + emitted_misses: DashMap<(FontId, SmallVec<[u8; 4]>), ()>, +} +``` +- **Purpose**: Per-font LRU cache for resolved glyphs +- **Thread-safe**: Uses DashMap for concurrent access +- **Miss tracking**: Ensures GLYPH_UNMAPPED diagnostic emitted once per (font, code) + +### 5. `Font` (`resolver.rs:33-46`) +```rust +pub struct Font { + id: FontId, + to_unicode: Option, + encoding: Option, + fingerprint: Option, + has_embedded_program: bool, + cache: ResolverCache, +} +``` +- **Purpose**: Encapsulates all data for 4-level fallback chain +- **to_unicode**: Level 1 CMap (highest priority) +- **encoding**: Level 2 named encoding + AGL +- **fingerprint**: Level 3 font fingerprint cache +- **has_embedded_program**: Skip Level 3 if false (Type 3 fonts) + +## Entry Addition Mechanism Details + +### CMAP Entry Creation (`cmap.rs:82-88`) + +```rust +pub fn add_mapping(&mut self, src: Vec, dst: Vec) { + self.mappings.insert(src, dst); +} +``` + +**Call Sites**: + +1. **From `parse_beginbfchar()` (line 209)** + - Processes single-character mappings + - Each `beginbfchar` block produces N calls to `add_mapping()` + - Source: hex string from CMap + - Destination: UTF-16BE decoded to `Vec` + +2. **From `parse_beginbfrange()` (lines 279, 292)** + - Processes range mappings + - **Contiguous form**: Calls `add_mapping()` in loop, incrementing both source and destination + - **Explicit array form**: Calls `add_mapping()` once per array entry + - Range expansion can produce hundreds of entries + +### Encoding Entry Creation + +**Named Encodings** (`encoding.rs`): +- Compile-time generated from JSON files in `build/named-encodings.json` +- Each encoding is a 256-element array: `[Option<&str>; 256]` +- Loaded at build time via `build.rs` + +**Differences Overlay** (`encoding.rs:144-183`): +- Parsed at runtime from `/Differences` array +- Sparse vector of `(u8, Arc)` pairs +- Created during `FontEncoding::parse_from_font()` + +## Code Locations Reference + +### CMAP Processing +- **ToUnicode parsing**: `crates/pdftract-core/src/font/cmap.rs:114-505` +- **Entry creation**: `cmap.rs:82-88` (MARKER at line 85) +- **bfchar parsing**: `cmap.rs:190-216` +- **bfrange parsing**: `cmap.rs:223-309` +- **UTF-16BE decoding**: `cmap.rs:329-367` + +### Resolution Chain +- **Main resolver**: `crates/pdftract-core/src/font/resolver.rs:302-377` +- **Level 1 (ToUnicode)**: `resolver.rs:383-399` +- **Level 2 (AGL)**: `resolver.rs:405-432` +- **Level 3 (Fingerprint)**: `resolver.rs:438-451` +- **Level 4 (Shape)**: `resolver.rs:457-470` (cfg-gated) + +### Glyph Name Processing +- **AGL lookup**: `crates/pdftract-core/src/font/agl.rs:41-94` +- **Algorithmic patterns**: `agl.rs:96-141` +- **Named encodings**: `crates/pdftract-core/src/font/encoding.rs:22-110` +- **Differences overlay**: `encoding.rs:144-183` + +### Data Structures +- **ToUnicodeMap**: `cmap.rs:67-112` +- **ResolvedGlyph**: `resolver.rs:168-197` +- **Font**: `resolver.rs:33-104` +- **ResolverCache**: `resolver.rs:223-284` +- **FontEncoding**: `encoding.rs` (module) + +## Glyph Name Summary + +### Processed by CMAP +- **None directly** - CMAP processes byte sequences, not glyph names + +### Processed by AGL (Level 2) +- Standard glyph names: `A`, `B`, `quotesingle`, `quotedblleft`, etc. +- Algorithmic patterns: `uniXXXX`, `uXXXXXX` +- Ligatures: `fi`, `fl`, `ffi`, `ffl` +- Variant suffixes: `A.sc`, `A.alt`, `one.oldstyle` +- Symbol font: `alpha`, `beta`, `gamma`, etc. +- ZapfDingbats: `a1`, `a2`, ..., `a202` + +### Total Coverage +- AGL: ~4,400 entries (glyphlist.txt) +- AGL Multi: ~770 entries (aglfn.txt for ligatures) +- Algorithmic patterns: Unlimited (any valid Unicode codepoint) +- Named encodings: 6 × 256 = 1,536 potential mappings + +## Confidence Hierarchy + +Per INV-30, confidence is always one of {1.0, 0.9, 0.85, 0.7, 0.0}: + +1. **1.0** - ToUnicode CMap (authoritative source) +2. **0.9** - Named encoding + AGL (reliable but indirect) +3. **0.85** - Font fingerprint (content-based, assumes standard fonts) +4. **0.7** - Shape recognition (visual similarity, error-prone) +5. **0.0** - Unknown (U+FFFD replacement character) + +## Related Documentation + +- **CMAP entry creation**: `notes/bf-e4uvb-child-1.md` (if exists) +- **ToUnicode entry creation code locations**: Commit `155e00ab` - `docs(bf-4hwap)` +- **4-level fallback chain**: `docs/plan/plan.md` (INV-30, EC-NN references) + +## Testing References + +- **CMap parsing tests**: `crates/pdftract-core/src/font/cmap.rs:507-732` +- **Resolver tests**: `crates/pdftract-core/src/font/resolver.rs` (mod tests) +- **AGL tests**: `crates/pdftract-core/src/font/agl.rs` (if present) + +--- + +**Document Version**: 1.0 +**Last Updated**: 2026-07-05 +**Related Bead**: bf-3ztvp +**Plan References**: (See plan.md for related EC-NN and INV-30 tags)