pdftract/notes/bf-35gpt.md

GLYPH_UNMAPPED Message Format Specification

Overview

The GLYPH_UNMAPPED diagnostic (diagnostic code: FONT_GLYPH_UNMAPPED) is emitted when a character code cannot be resolved to a Unicode character through pdftract's 4-level fallback chain. This specification documents the complete message format for parsing and processing these diagnostics.

JSON Schema

All GLYPH_UNMAPPED diagnostics follow the DiagnosticJson structure:

pub struct DiagnosticJson {
    /// Stable string identifier (always "FONT_GLYPH_UNMAPPED")
    pub code: String,
    
    /// Human-readable description of the diagnostic
    pub message: String,
    
    /// Severity level (always "error" for GLYPH_UNMAPPED)
    pub severity: String,
    
    /// Page index where diagnostic occurred, or null for document-level
    pub page_index: Option<usize>,
    
    /// PDF object reference where issue originated, if applicable
    pub location: Option<ObjectLocationJson>,
    
    /// Optional hint for resolution
    pub hint: Option<String>,
}

Base JSON Structure

{
  "code": "FONT_GLYPH_UNMAPPED",
  "message": "...",
  "severity": "error",
  "page_index": 0,
  "location": null,
  "hint": null
}

Field Descriptions:

Field	Type	Value	Description
code	string	"FONT_GLYPH_UNMAPPED"	Stable identifier for the diagnostic type
message	string	(varies)	Human-readable description with context-specific details
severity	string	"error"	Severity level for GLYPH_UNMAPPED
page_index	number|null	integer or null	Zero-based page index where the glyph resolution failed
location	object|null	object or null	PDF object reference ({object_number, generation_number})
hint	string|null	string or null	Optional resolution hint (rarely used for GLYPH_UNMAPPED)

Message Format Patterns

Pattern 1: Standard Font Miss (Regular Fonts)

Format: Character code <HEX> could not be resolved to Unicode (font ID: <FONT_ID>)

Examples:

{
  "code": "FONT_GLYPH_UNMAPPED",
  "message": "Character code 41 could not be resolved to Unicode (font ID: FontId(123456))",
  "severity": "error",
  "page_index": 0,
  "location": null,
  "hint": null
}

Components:

• <HEX>: Uppercase hexadecimal string (1+ bytes, no 0x prefix)
  • Single-byte: 41, 01, FF
  • Multi-byte: 0041, 000101
• <FONT_ID>: FontId struct in debug format: FontId(<u32>)

Emission Conditions:

• All 4 resolution levels fail:
  1. ToUnicode CMap
  2. Adobe Glyph List (AGL)
  3. Fingerprint matching
  4. Shape recognition (if available)

Deduplication: Emitted exactly once per (font_id, char_code) pair

Applicable Font Types: Type1, TrueType, CIDFont, Type0

---

Pattern 2: Type3 Font - Generic Failure

Format: Type3 font: character code 0x<HEX> could not be resolved to Unicode

Examples:

{
  "code": "FONT_GLYPH_UNMAPPED",
  "message": "Type3 font: character code 0x41 could not be resolved to Unicode",
  "severity": "error",
  "page_index": 1,
  "location": null,
  "hint": null
}

Components:

• <HEX>: Single-byte hexadecimal (range 00-FF)
• Always prefixed with 0x
• Font ID is NOT included (implicit from context)

Emission Conditions:

• All Type3-specific resolution levels fail

Deduplication: Once per (font, char_code) pair

Applicable Font Types: Type3 only

---

Pattern 3: Type3 Font - Shape Recognition Disabled

Format: Type3 font: character code 0x<HEX> could not be resolved (shape recognition disabled)

Examples:

{
  "code": "FONT_GLYPH_UNMAPPED",
  "message": "Type3 font: character code 0x7F could not be resolved (shape recognition disabled)",
  "severity": "error",
  "page_index": 0,
  "location": null,
  "hint": null
}

Components:

• <HEX>: Single-byte hexadecimal (00-FF)

Emission Conditions:

• Shape recognition feature disabled at compile time (--no-default-features --features shape-db)
• Level 4 fallback unavailable

Conditional Compilation: Only emitted when shape-db feature is disabled

Resolution Hint: Recompile with --features shape-db to enable Level 4 fallback

---

Pattern 4: Type3 Font - No Glyph Name in Encoding

Format: Type3 font: character code 0x<HEX> has no glyph name in encoding

Examples:

{
  "code": "FONT_GLYPH_UNMAPPED",
  "message": "Type3 font: character code 0x20 has no glyph name in encoding",
  "severity": "error",
  "page_index": 0,
  "location": null,
  "hint": null
}

Components:

• <HEX>: Single-byte hexadecimal (00-FF)

Emission Conditions:

• Encoding dictionary has no entry for the character code
• Cannot proceed to shape recognition (no glyph name to look up)

Implications: Immediate resolution failure - no fallback possible

---

Pattern 5: Type3 Font - Glyph Not in CharProcs

Format: Type3 font: glyph '<NAME>' not found in /CharProcs for code 0x<HEX>

Examples:

{
  "code": "FONT_GLYPH_UNMAPPED",
  "message": "Type3 font: glyph 'A' not found in /CharProcs for code 0x41",
  "severity": "error",
  "page_index": 0,
  "location": null,
  "hint": null
}

Components:

• <NAME>: Glyph name from encoding (e.g., A, custom, g003)
• <HEX>: Single-byte hexadecimal (00-FF)

Emission Conditions:

• Glyph name present in encoding dictionary
• Glyph name missing from /CharProcs dictionary
• Indicates malformed Type3 font (orphan encoding entry)

Implications: Malformed PDF - encoding references non-existent glyph

---

Pattern 6: Type3 Font - Rasterization Failed

Format: Type3 font: failed to rasterize glyph '<NAME>' for code 0x<HEX>

Examples:

{
  "code": "FONT_GLYPH_UNMAPPED",
  "message": "Type3 font: failed to rasterize glyph 'complex' for code 0x02",
  "severity": "error",
  "page_index": 1,
  "location": null,
  "hint": null
}

Components:

• <NAME>: Glyph name from encoding
• <HEX>: Single-byte hexadecimal (00-FF)

Emission Conditions:

• Glyph exists in /CharProcs
• Content stream rendering failed during bitmap generation
• Indicates corrupt or malformed content stream

Implications: Malformed glyph content stream - cannot generate bitmap for shape matching

---

Pattern 7: Type3 Font - Shape Match Below Threshold

Format: Type3 font: shape match for '<NAME>' (code 0x<HEX>) found but distance <DIST> exceeds threshold

Examples:

{
  "code": "FONT_GLYPH_UNMAPPED",
  "message": "Type3 font: shape match for 'A' (code 0x41) found but distance 12 exceeds threshold",
  "severity": "error",
  "page_index": 0,
  "location": null,
  "hint": null
}

Components:

• <NAME>: Glyph name
• <HEX>: Single-byte hexadecimal (00-FF)
• <DIST>: Hamming distance (integer, typically 0-255)

Emission Conditions:

• Shape match was found in database
• Hamming distance exceeded acceptable threshold (default: 8)
• Indicates "close but not confident enough" match

Implications: Low-confidence match - rejected to avoid false positives

Conditional Compilation: Only emitted when shape-db feature is enabled

---

Pattern Selection Logic

Regular Fonts (Type1, TrueType, CIDFont, Type0)

• Pattern used: Pattern 1 only
• Character codes: Can be multi-byte (e.g., 0041 for 2-byte CID key)
• Font ID: Always included
• Failure reasons: Generic (all 4 levels failed)

Type3 Fonts

• Patterns used: Patterns 2-7
• Character codes: Always single-byte (00-FF)
• Font ID: NOT included (implicit from context)
• Failure reasons: Specific (encoding, CharProcs, rasterization, shape match)

Hex Formatting Rules

Pattern	Hex Format	Prefix	Example
Pattern 1	Concatenated uppercase hex	None	41, 0041, 000101
Patterns 2-7	Single-byte uppercase hex	0x	0x41, 0xFF, 0x01

Deduplication Behavior

All patterns use the same deduplication mechanism:

1. Cache key: (font_id, char_code) for Pattern 1; (font, char_code) for Type3 patterns
2. Storage: ResolverCache::emitted_misses (HashSet)
3. Behavior: First occurrence is emitted; subsequent occurrences are suppressed
4. Scope: Per-document (cache resets on new document)

Code Location Reference

Component	File	Lines
Diagnostic code enum	diagnostics.rs	576
Code string constant	diagnostics.rs	1291
JSON schema definition	schema/mod.rs	813-834
Pattern 1 emission	font/resolver.rs	681-708
Pattern 2 emission	font/resolver.rs	565-571
Pattern 3 emission	font/resolver.rs	575-583
Pattern 4 emission	font/resolver.rs	616-624
Pattern 5 emission	font/resolver.rs	631-638
Pattern 6 emission	font/resolver.rs	645-652
Pattern 7 emission	font/resolver.rs	667-674

Example Parsing Code

use serde_json::Value;

fn parse_glyph_unmapped(diagnostic: &Value) -> GlyphUnmappedInfo {
    let message = diagnostic["message"].as_str().unwrap();
    
    // Pattern 1: Regular font miss
    if let Some(caps) = regex!(r"Character code ([0-9A-F]+) could not be resolved to Unicode \(font ID: FontId\((\d+)\)\)")
        .captures(message)
    {
        return GlyphUnmappedInfo::RegularFont {
            char_code: caps.get(1).unwrap().as_str().to_string(),
            font_id: caps.get(2).unwrap().as_str().parse::<u32>().unwrap(),
        };
    }
    
    // Pattern 5: Glyph not in CharProcs
    if let Some(caps) = regex!(r"Type3 font: glyph '([^']+)' not found in /CharProcs for code 0x([0-9A-F]{2})")
        .captures(message)
    {
        return GlyphUnmappedInfo::Type3GlyphNotInCharProcs {
            glyph_name: caps.get(1).unwrap().as_str().to_string(),
            char_code: u8::from_str_radix(caps.get(2).unwrap().as_str(), 16).unwrap(),
        };
    }
    
    // ... other patterns
    
    GlyphUnmappedInfo::Unknown
}

Testing Reference

• Test fixture: tests/fixtures/encoding/no-mapping.pdf
• Test code: crates/pdftract-core/tests/font_integration.rs (GLYPH_UNMAPPED emission tests)
• Schema test: crates/pdftract-core/src/schema/mod.rs::test_diagnostic_json()

Related Documentation

• Analysis of message patterns: notes/bf-5n4dp.md
• Font resolver implementation: crates/pdftract-core/src/font/resolver.rs
• Diagnostic system overview: crates/pdftract-core/src/diagnostics.rs (module documentation)

Version History

• 2025-01-XX: Initial specification creation (bf-35gpt)
• Previous analysis: bf-5n4dp (pattern identification)

---

This specification is maintained as part of the pdftract project. For the latest version, refer to the source code and documentation in the /home/coding/pdftract repository.