docs(bf-3ztvp): document CMAP data flow and glyph processing

- Created comprehensive documentation at notes/bf-e4uvb-child-3-cmap-flow.md - Documented 4-level encoding resolution chain (ToUnicode → AGL → Fingerprint → Shape) - Identified key data structures: ToUnicodeMap, ResolvedGlyph, Font, ResolverCache - Documented CMAP entry creation mechanism at cmap.rs:82-88 (MARKER at line 85) - Explained glyph name processing: CMAP processes byte sequences, AGL processes glyph names - Documented UTF-16BE decoding and ligature expansion support - Listed all code locations with line numbers for easy reference - Covered confidence hierarchy: 1.0 (ToUnicode) → 0.9 (AGL) → 0.85 (Fingerprint) → 0.7 (Shape) Acceptance criteria met: ✓ Data flow documented: which glyph names are processed ✓ Entry addition mechanism documented ✓ Key data structures identified ✓ Documentation saved to notes/bf-e4uvb-child-3-cmap-flow.md Closes bf-3ztvp
2026-07-05 18:33:56 -04:00 · 2026-07-05 18:33:56 -04:00 · fbeac96b8c
commit fbeac96b8c
parent 39148311f4
1 changed files with 330 additions and 0 deletions
--- a/notes/bf-e4uvb-child-3-cmap-flow.md
+++ b/notes/bf-e4uvb-child-3-cmap-flow.md
@ -0,0 +1,330 @@
+# CMAP Data Flow and Glyph Processing Documentation
+
+## Overview
+
+This document describes how CMAP (Character Map) processes glyph names and creates encoding entries in the pdftract codebase. The CMAP system is a critical component of the 4-level encoding resolution chain for text extraction from PDFs.
+
+## Data Flow Architecture
+
+### Level 1: ToUnicode CMap (Confidence 1.0)
+
+The primary and most reliable source for glyph-to-Unicode mapping is the `/ToUnicode` CMap stream embedded in PDF fonts.
+
+#### Entry Creation Flow
+
+1. **CMap Parsing** (`crates/pdftract-core/src/font/cmap.rs:86-88`)
+   - **Entry Point**: `ToUnicodeMap::add_mapping()` method
+   - **MARKER**: Line 85-88 explicitly marks this as the CMAP entry creation point
+   - Called by:
+     - `parse_beginbfchar()` (single character mappings)
+     - `parse_beginbfrange()` (range mappings)
+
+2. **Data Structure**: `ToUnicodeMap` (`cmap.rs:67-71`)
+   ```rust
+   pub struct ToUnicodeMap {
+       /// Mapping from source byte sequence to destination Unicode codepoints.
+       /// Uses Vec<u8> as key (source bytes) and Vec<char> as value (destination chars).
+       mappings: HashMap<Vec<u8>, Vec<char>>,
+   }
+   ```
+
+3. **Mapping Types Processed**:
+   
+   a. **Single Character Mappings** (`beginbfchar...endbfchar` blocks)
+      - Format: `beginbfchar <count> <src1> <dst1> <src2> <dst2> ... endbfchar`
+      - Example: `beginbfchar 1 <00> <0041> endbfchar` maps byte `0x00` to 'A'
+      - Supports ligature expansion: `<00660069>` → `['f', 'i']` (two codepoints)
+   
+   b. **Range Mappings** (`beginbfrange...endbfrange` blocks)
+      - **Contiguous form**: `<lo> <hi> <dst>` - maps range with incrementing destination
+        - Example: `beginbfrange 1 <0041> <005A> <0041> endbfrange` maps A-Z to U+0041..U+005A
+      - **Explicit array form**: `<lo> <hi> [<d0> <d1> ...]` - each destination explicitly specified
+        - Example: `beginbfrange 1 <0001> <0003> [<FB01> <FB02> <FB03>] endbfrange`
+        - Maps codes 1,2,3 to ligatures fi, fl, ffi
+
+4. **Glyph Names Processed**:
+   - CMap does NOT process glyph names directly
+   - It processes **raw byte sequences** (character codes) as keys
+   - Source byte sequences can be 1-4 bytes (variable-length)
+   - Destination is always Unicode codepoints (UTF-16BE decoded)
+
+5. **UTF-16BE Decoding** (`cmap.rs:329-367`)
+   - Hex strings in CMap are UTF-16BE encoded
+   - Decoded to `Vec<char>` supporting surrogate pairs and multi-codepoint results
+   - Unpaired surrogates produce replacement character (U+FFFD)
+   - Odd byte counts emit diagnostic but continue processing
+
+#### Lookup Flow
+
+1. **Resolution Entry** (`resolver.rs:302-377`)
+   - `resolve_unicode()` is the main entry point
+   - Checks per-font cache first (thread-safe LRU)
+   - Calls `resolve_level1()` for ToUnicode lookup
+
+2. **Level 1 Lookup** (`resolver.rs:383-399`)
+   ```rust
+   fn resolve_level1(char_code: &[u8], to_unicode: Option<&ToUnicodeMap>) -> ResolvedGlyph
+   ```
+   - Returns failure if ToUnicode CMap is missing
+   - Returns failure if character code not in map
+   - Returns failure if result is empty or U+FFFD only
+   - Success: Returns `ResolvedGlyph` with chars, source=ToUnicode, confidence=1.0
+
+### Level 2: Named Encoding + AGL (Confidence 0.9)
+
+When ToUnicode is absent or fails, the system falls back to named encoding + AGL lookup.
+
+#### Encoding Processing
+
+1. **Named Encodings** (`encoding.rs:22-60`)
+   - 6 standard encodings: WinAnsi, MacRoman, MacExpert, Standard, Symbol, ZapfDingbats
+   - Map character codes (0-255) to glyph names
+   - WinAnsiEncoding is most common (Windows-1252 superset)
+
+2. **Differences Overlay** (`encoding.rs:129-150`)
+   - Sparse overrides from `/Differences` array
+   - Format: `[n /Name1 /Name2 ... m /OtherName ...]`
+   - Each integer resets position, subsequent names assign to consecutive codes
+   - Example: `[39 /quotesingle 96 /grave]` assigns code 39→"quotesingle", 96→"grave"
+
+#### AGL Lookup (Glyph Name Processing)
+
+1. **Adobe Glyph List** (`agl.rs:41-60`)
+   - Compile-time phf::Map of ~4,400 glyph name → Unicode mappings
+   - Generated from Adobe's glyphlist.txt at build time
+
+2. **Algorithmic Patterns** (`agl.rs:96-100`)
+   - `uniXXXX` (4 hex digits) → Unicode codepoint
+   - `uXXXXXX` (up to 6 hex digits) → Unicode codepoint
+   - Example: `uni20AC` → U+20AC (Euro sign)
+
+3. **Variant Suffix Stripping** (`agl.rs:47-56`)
+   - Strips suffixes like `.sc`, `.alt`, `.oldstyle` before lookup
+   - Example: `A.sc` → `A` → U+0041
+
+4. **Ligature Support** (`agl.rs:79-94`)
+   - `unicode_for_glyph_name_multi()` handles multi-codepoint glyphs
+   - Example: `fi` → `['f', 'i']`
+
+#### Level 2 Resolution
+
+- Character code → Encoding table → Glyph name → AGL lookup → Unicode
+- Returns `ResolvedGlyph` with source=Agl, confidence=0.9
+
+### Level 3: Font Fingerprint Cache (Confidence 0.85)
+
+When both ToUnicode and AGL fail, the system uses font fingerprinting.
+
+#### Fingerprint Processing (`fingerprint.rs`)
+
+1. **Hash Computation**
+   - SHA-256 hash of decoded font program bytes
+   - Computed over embedded font program (post-stream decode)
+   - Stable across different stream filters
+
+2. **Lookup Database**
+   - Compile-time phf::Map: `[u8; 32]` → `&'static [(u16, u32)]`
+   - Maps glyph ID (GID) to Unicode codepoint
+   - Generated from `build/font-fingerprints.json`
+
+3. **Fallback Trigger**
+- Used when:
+  - `/ToUnicode` is missing or empty
+  - Embedded font subset has stripped glyph names
+  - Font binary matches a known fingerprint
+
+### Level 4: Shape Recognition (Confidence 0.7, cfg-gated)
+
+Final fallback: visual glyph shape matching.
+
+#### Shape Processing (`shape.rs`)
+
+1. **Rasterization**
+   - Type 3 glyphs: Execute content stream to 32×32 bitmap
+   - Embedded fonts: Extract glyph outline, render to bitmap
+
+2. **Perceptual Hashing**
+   - Compute pHash of bitmap
+   - Look up in shape database (compile-time phf::Map)
+
+3. **Feature-Gated**
+   - Only available when `shape-db` feature is enabled
+   - Requires `build/glyph-shapes.json` at compile time
+
+## Key Data Structures
+
+### 1. `ToUnicodeMap` (`cmap.rs:67-71`)
+```rust
+pub struct ToUnicodeMap {
+    mappings: HashMap<Vec<u8>, Vec<char>>,
+}
+```
+- **Purpose**: Stores character code → Unicode mappings from `/ToUnicode` stream
+- **Key**: Variable-length byte sequence (1-4 bytes)
+- **Value**: Unicode codepoints (1-4 chars for ligatures)
+- **Entry point**: `add_mapping()` at line 86
+
+### 2. `ResolvedGlyph` (`resolver.rs:168-175`)
+```rust
+pub struct ResolvedGlyph {
+    pub chars: SmallVec<[char; 4]>,
+    pub source: UnicodeSource,
+    pub confidence: f32,
+}
+```
+- **Purpose**: Result of encoding resolution at any level
+- **chars**: 1-4 Unicode characters (supports ligature expansion)
+- **source**: Which level produced the mapping (ToUnicode, Agl, Fingerprint, ShapeMatch, Unknown)
+- **confidence**: Always 1.0, 0.9, 0.85, or 0.7 per INV-30
+
+### 3. `FontEncoding` (`encoding.rs`)
+```rust
+pub enum FontEncoding {
+    Named(NamedEncoding),
+    Custom(DifferencesOverlay),
+}
+```
+- **Purpose**: Level 2 character code → glyph name mapping
+- **Named**: One of 6 standard encodings (WinAnsi, MacRoman, etc.)
+- **Custom**: Base encoding + Differences array overlay
+
+### 4. `ResolverCache` (`resolver.rs:223-278`)
+```rust
+pub struct ResolverCache {
+    cache: DashMap<CacheKey, ResolvedGlyph>,
+    emitted_misses: DashMap<(FontId, SmallVec<[u8; 4]>), ()>,
+}
+```
+- **Purpose**: Per-font LRU cache for resolved glyphs
+- **Thread-safe**: Uses DashMap for concurrent access
+- **Miss tracking**: Ensures GLYPH_UNMAPPED diagnostic emitted once per (font, code)
+
+### 5. `Font` (`resolver.rs:33-46`)
+```rust
+pub struct Font {
+    id: FontId,
+    to_unicode: Option<ToUnicodeMap>,
+    encoding: Option<FontEncoding>,
+    fingerprint: Option<CachedFingerprint>,
+    has_embedded_program: bool,
+    cache: ResolverCache,
+}
+```
+- **Purpose**: Encapsulates all data for 4-level fallback chain
+- **to_unicode**: Level 1 CMap (highest priority)
+- **encoding**: Level 2 named encoding + AGL
+- **fingerprint**: Level 3 font fingerprint cache
+- **has_embedded_program**: Skip Level 3 if false (Type 3 fonts)
+
+## Entry Addition Mechanism Details
+
+### CMAP Entry Creation (`cmap.rs:82-88`)
+
+```rust
+pub fn add_mapping(&mut self, src: Vec<u8>, dst: Vec<char>) {
+    self.mappings.insert(src, dst);
+}
+```
+
+**Call Sites**:
+
+1. **From `parse_beginbfchar()` (line 209)**
+   - Processes single-character mappings
+   - Each `beginbfchar` block produces N calls to `add_mapping()`
+   - Source: hex string from CMap
+   - Destination: UTF-16BE decoded to `Vec<char>`
+
+2. **From `parse_beginbfrange()` (lines 279, 292)**
+   - Processes range mappings
+   - **Contiguous form**: Calls `add_mapping()` in loop, incrementing both source and destination
+   - **Explicit array form**: Calls `add_mapping()` once per array entry
+   - Range expansion can produce hundreds of entries
+
+### Encoding Entry Creation
+
+**Named Encodings** (`encoding.rs`):
+- Compile-time generated from JSON files in `build/named-encodings.json`
+- Each encoding is a 256-element array: `[Option<&str>; 256]`
+- Loaded at build time via `build.rs`
+
+**Differences Overlay** (`encoding.rs:144-183`):
+- Parsed at runtime from `/Differences` array
+- Sparse vector of `(u8, Arc<str>)` pairs
+- Created during `FontEncoding::parse_from_font()`
+
+## Code Locations Reference
+
+### CMAP Processing
+- **ToUnicode parsing**: `crates/pdftract-core/src/font/cmap.rs:114-505`
+- **Entry creation**: `cmap.rs:82-88` (MARKER at line 85)
+- **bfchar parsing**: `cmap.rs:190-216`
+- **bfrange parsing**: `cmap.rs:223-309`
+- **UTF-16BE decoding**: `cmap.rs:329-367`
+
+### Resolution Chain
+- **Main resolver**: `crates/pdftract-core/src/font/resolver.rs:302-377`
+- **Level 1 (ToUnicode)**: `resolver.rs:383-399`
+- **Level 2 (AGL)**: `resolver.rs:405-432`
+- **Level 3 (Fingerprint)**: `resolver.rs:438-451`
+- **Level 4 (Shape)**: `resolver.rs:457-470` (cfg-gated)
+
+### Glyph Name Processing
+- **AGL lookup**: `crates/pdftract-core/src/font/agl.rs:41-94`
+- **Algorithmic patterns**: `agl.rs:96-141`
+- **Named encodings**: `crates/pdftract-core/src/font/encoding.rs:22-110`
+- **Differences overlay**: `encoding.rs:144-183`
+
+### Data Structures
+- **ToUnicodeMap**: `cmap.rs:67-112`
+- **ResolvedGlyph**: `resolver.rs:168-197`
+- **Font**: `resolver.rs:33-104`
+- **ResolverCache**: `resolver.rs:223-284`
+- **FontEncoding**: `encoding.rs` (module)
+
+## Glyph Name Summary
+
+### Processed by CMAP
+- **None directly** - CMAP processes byte sequences, not glyph names
+
+### Processed by AGL (Level 2)
+- Standard glyph names: `A`, `B`, `quotesingle`, `quotedblleft`, etc.
+- Algorithmic patterns: `uniXXXX`, `uXXXXXX`
+- Ligatures: `fi`, `fl`, `ffi`, `ffl`
+- Variant suffixes: `A.sc`, `A.alt`, `one.oldstyle`
+- Symbol font: `alpha`, `beta`, `gamma`, etc.
+- ZapfDingbats: `a1`, `a2`, ..., `a202`
+
+### Total Coverage
+- AGL: ~4,400 entries (glyphlist.txt)
+- AGL Multi: ~770 entries (aglfn.txt for ligatures)
+- Algorithmic patterns: Unlimited (any valid Unicode codepoint)
+- Named encodings: 6 × 256 = 1,536 potential mappings
+
+## Confidence Hierarchy
+
+Per INV-30, confidence is always one of {1.0, 0.9, 0.85, 0.7, 0.0}:
+
+1. **1.0** - ToUnicode CMap (authoritative source)
+2. **0.9** - Named encoding + AGL (reliable but indirect)
+3. **0.85** - Font fingerprint (content-based, assumes standard fonts)
+4. **0.7** - Shape recognition (visual similarity, error-prone)
+5. **0.0** - Unknown (U+FFFD replacement character)
+
+## Related Documentation
+
+- **CMAP entry creation**: `notes/bf-e4uvb-child-1.md` (if exists)
+- **ToUnicode entry creation code locations**: Commit `155e00ab` - `docs(bf-4hwap)`
+- **4-level fallback chain**: `docs/plan/plan.md` (INV-30, EC-NN references)
+
+## Testing References
+
+- **CMap parsing tests**: `crates/pdftract-core/src/font/cmap.rs:507-732`
+- **Resolver tests**: `crates/pdftract-core/src/font/resolver.rs` (mod tests)
+- **AGL tests**: `crates/pdftract-core/src/font/agl.rs` (if present)
+
+---
+
+**Document Version**: 1.0  
+**Last Updated**: 2026-07-05  
+**Related Bead**: bf-3ztvp  
+**Plan References**: (See plan.md for related EC-NN and INV-30 tags)