From aaa3772e62138212c880d65e7da91ec62fa9113a Mon Sep 17 00:00:00 2001 From: jedarden Date: Mon, 6 Jul 2026 15:39:28 -0400 Subject: [PATCH] docs(bf-3tzsn): document CMAP test structure and fixture exploration - Located existing CMAP test files in crates/pdftract-core/tests/ - Identified test fixture patterns in tests/fixtures/encoding/ - Found unmapped_glyph_names configuration examples - Documented CMAP entry creation points and build-time processing - Documented test creation patterns for future reference Acceptance criteria: - Located existing CMAP test file(s) - Identified test fixture patterns - Found examples of unmapped_glyph_names config - Documented the test structure pattern in note - Ready to create test fixture based on this understanding Closes bf-3tzsn --- notes/bf-3tzsn.md | 378 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 378 insertions(+) create mode 100644 notes/bf-3tzsn.md diff --git a/notes/bf-3tzsn.md b/notes/bf-3tzsn.md new file mode 100644 index 0000000..c93977d --- /dev/null +++ b/notes/bf-3tzsn.md @@ -0,0 +1,378 @@ +# CMAP Test Structure and Fixture Exploration — bf-3tzsn + +## Overview + +This note documents the CMAP (character map) test infrastructure, including existing test patterns, fixture structures, and configuration mechanisms for unmapped glyph handling. + +## Acceptance Criteria: PASS ✅ + +- [x] Located existing CMAP test file(s) +- [x] Identified test fixture patterns +- [x] Found examples of unmapped_glyph_names config +- [x] Documented the test structure pattern in this note +- [x] Ready to create test fixture based on this understanding + +--- + +## Test Files Structure + +### 1. Unit Tests: `crates/pdftract-core/tests/unmapped_glyph_names_config.rs` + +Tests the `UnmappedGlyphNamesConfig` structure that reads from `build/unmapped-glyph-names.json`: + +```rust +struct UnmappedGlyphNamesConfig { + #[serde(default)] + unmapped_glyph_names: Vec, + + #[serde(default)] + description: Option, + + #[serde(default)] + version: Option, +} +``` + +**Test patterns observed:** +- `test_unmapped_glyph_names_defaults_to_empty()` - Config without field defaults to empty Vec +- `test_unmapped_glyph_names_specified()` - Config with field parses correctly +- `test_unmapped_glyph_names_empty_array()` - Explicit `[]` parses correctly +- `test_unmapped_glyph_names_minimal_config()` - Empty `{}` defaults all fields + +### 2. Integration Tests: `crates/pdftract-core/tests/encoding_recovery.rs` + +Tests Unicode recovery from PDFs without ToUnicode CMaps using the `EncodingFixture` pattern: + +```rust +struct EncodingFixture { + name: &'static str, + pdf_path: &'static str, + truth_path: &'static str, + description: &'static str, +} +``` + +**Test fixtures in corpus:** +- `no-mapping` - PDF with no ToUnicode, custom encoding (worst case) +- `agl-only` - AGL glyph names only (Level 2 recovery) +- `fingerprint-match` - Embedded font for fingerprinting (Level 3) +- `shape-match` - Subset font for shape recognition (Level 4) + +**Test patterns observed:** +- `test_no_mapping_fixture()` - Verifies CER ≥ 0.0 and recovery_rate ≤ 1.0 +- `test_agl_only_fixture()` - Perfect match assertions (CER = 0.0, rate = 1.0) +- `test_corpus_recovery_rate()` - Calculates weighted average across corpus + +--- + +## Fixture Structure + +### Directory Layout + +``` +tests/fixtures/ +├── encoding/ # Encoding/Unicode recovery fixtures +│ ├── no-mapping.pdf +│ ├── no-mapping.txt +│ ├── no-mapping.md +│ ├── agl-only.pdf +│ ├── agl-only.txt +│ ├── fingerprint-match.pdf +│ ├── fingerprint-match.txt +│ ├── shape-match.pdf +│ ├── shape-match.txt +│ ├── generate_unmapped_glyphs.py +│ └── create_unmapped_comprehensive.py +└── forms/ # Form fixtures (with generator pattern) + └── generate_form_fixtures.rs +``` + +### Fixture Documentation Pattern + +Each fixture has a corresponding `.md` file with this structure: + +**Example:** `tests/fixtures/encoding/no-mapping.md` + +```markdown +# no-mapping.pdf Fixture + +## Purpose +Level 4 Unicode recovery test fixture - worst case scenario... + +## Structure +### PDF Properties +- PDF Version: 1.4 +- Pages: 1 +- Page Size: 612 x 792 pts (Letter) + +### Font Details +Name: CustomNoMap +Type: Type 1 +Encoding: Custom (Differences array) +Embedded: No +ToUnicode CMap: No + +## Inspection Commands +### Basic PDF info: +pdfinfo tests/fixtures/encoding/no-mapping.pdf + +### Font details: +pdffonts tests/fixtures/encoding/no-mapping.pdf + +## Verification +# Check SHA256 matches expected value +sha256sum tests/fixtures/encoding/no-mapping.pdf + +# Verify extraction produces U+FFFD characters +./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \ + --format text -o /tmp/no-mapping-output.txt + +## Test References +- tests/debug_encoding_pdf.rs +- tests/debug_encoding_fixtures.rs +- tests/encoding_recovery.rs +``` + +--- + +## Unmapped Glyph Names Configuration + +### Configuration File: `build/unmapped-glyph-names.json` + +```json +{ + "unmapped_glyph_names": [ + ".notdef", + ".null", + "g000", + "g001", + "g002", + "g003", + "g004", + "g005", + "g006", + "g007", + "g008", + "g009" + ], + "description": "Glyph names that should be skipped during CMAP and ToUnicode entry creation...", + "version": "1.0" +} +``` + +### Build-Time Processing: `crates/pdftract-core/build.rs` + +The `generate_unmapped_glyph_names()` function: + +1. Reads `build/unmapped-glyph-names.json` +2. Emits build warning if not found (uses default `.notdef` only) +3. Generates `UNMAPPED_GLYPH_NAMES` LazyLock> in `$OUT_DIR` +4. Include path: `include!(concat!(env!("OUT_DIR"), "/unmapped_glyph_names.rs"));` + +### Runtime Usage: `crates/pdftract-core/src/font/encoding.rs` + +The `DifferencesOverlay` struct uses the unmapped set: + +```rust +pub struct DifferencesOverlay { + entries: Vec<(u8, Arc)>, + unmapped_glyph_names: std::collections::HashSet, +} + +impl DifferencesOverlay { + fn default_unmapped_glyph_names() -> std::collections::HashSet { + crate::font::unmapped::UNMAPPED_GLYPH_NAMES + .iter() + .map(|s| s.to_string()) + .collect() + } + + fn is_unmapped_glyph_name(&self, name: &str) -> bool { + self.unmapped_glyph_names.contains(name) + } +} +``` + +**CMAP entry creation marker (line 221-228 in encoding.rs):** +```rust +// MARKER: CMAP entry creation point - Type1 font encoding differences. +// See notes/bf-e4uvb-child-1.md for documentation. +if !overlay.is_unmapped_glyph_name(&name) { + overlay.entries.push((cursor as u8, Arc::clone(name))); +} +``` + +--- + +## Generator Patterns + +### Python Generator: `tests/fixtures/encoding/generate_unmapped_glyphs.py` + +**Purpose:** Generate PDF fixtures with custom glyph names and encodings + +**Usage:** +```bash +python3 generate_unmapped_glyphs.py [OPTIONS] + --output PATH Output PDF path (default: unmapped-glyphs.pdf) + --ground-truth PATH Ground truth .txt path (default: unmapped-glyphs.txt) + --title NAME Font base name (default: UnmappedTestFont) + --glyphs JSON Glyph mappings as JSON string +``` + +**Default glyph set (10 character codes):** +- Codes 0-2: `/g001`, `/g002`, `/g003` (PUA unmapped) +- Codes 3-6: `/CustomA`, `/CustomB`, `/NotAGlyph`, `/glyph_0041` (unmapped) +- Codes 7-9: `/A`, `/B`, `/space` (AGL mapped) + +### Rust Generator: `tests/fixtures/forms/generate_form_fixtures.rs` + +**Pattern for creating PDF fixtures programmatically:** + +```rust +use lopdf::dictionary; +use lopdf::{Dictionary, Document, Object, ObjectId, StringFormat}; + +fn create_simple_page(content: &[u8], doc: &mut Document) -> ObjectId { + let mut page_dict = Dictionary::new(); + page_dict.set("Type", "Page"); + page_dict.set("MediaBox", Object::Array(vec![ + Object::Real(0.0), Object::Real(0.0), + Object::Real(612.0), Object::Real(792.0) + ])); + // ... add resources and content stream + doc.add_object(page_dict) +} +``` + +--- + +## CMAP Source Code Structure + +### CMap Parser: `crates/pdftract-core/src/font/cmap.rs` + +**Purpose:** Parse `/ToUnicode` streams from PDF fonts as PostScript CMap programs + +**Supported CMap syntax:** +- `beginbfchar` / `endbfchar`: Single-character mappings +- `beginbfrange` / `endbfrange`: Range mappings (contiguous and explicit array) +- `usecmap`: Inheritance from named CMaps (stub - emits diagnostic) +- Comments: `%` to end of line (stripped by lexer) + +**CMAP entry creation marker (line 83-85 in cmap.rs):** +```rust +/// MARKER: CMAP entry creation point - this is where individual ToUnicode +/// mappings are stored. Called by parse_beginbfchar() and parse_beginbfrange(). +/// See notes/bf-e4uvb-child-1.md for documentation. +pub fn add_mapping(&mut self, src: Vec, dst: Vec) { + self.mappings.insert(src, dst); +} +``` + +--- + +## Predefined CMaps + +**Location:** `crates/pdftract-core/build/predefined-cmaps/` + +**Files:** +- `adobe-gb1.json` - Adobe-GB1-UCS2 (Simplified Chinese) +- `adobe-cns1.json` - Adobe-CNS1-UCS2 (Traditional Chinese) +- `adobe-japan1.json` - Adobe-Japan1-UCS2 (Japanese) +- `adobe-korea1.json` - Adobe-Korea1-UCS2 (Korean) + +**Example structure (adobe-japan1.json):** +```json +{ + "236": "あ", + "237": "い", + "238": "う" +} +``` + +--- + +## Test Creation Pattern Summary + +### To create a new CMAP test: + +1. **Create fixture files:** + - `.pdf` - Test PDF + - `.txt` - Ground truth expected output + - `.md` - Documentation following the pattern above + +2. **Add fixture to corpus:** + - Add `EncodingFixture` entry to `get_fixtures()` in `encoding_recovery.rs` + - Include paths relative to `crates/pdftract-core/tests/` + +3. **Write test function:** + ```rust + #[test] + fn test__fixture() { + let fixture = &get_fixtures()[index]; + let result = test_encoding_fixture(fixture).unwrap(); + // Assertions based on fixture expectations + } + ``` + +4. **Update unmapped_glyph_names if needed:** + - Edit `build/unmapped-glyph-names.json` + - Add new glyph names to `unmapped_glyph_names` array + - Rebuild to regenerate code + +5. **Regenerate CHECKSUMS.sha256:** + ```bash + cd crates/pdftract-core/build + sha256sum unmapped-glyph-names.json >> CHECKSUMS.sha256 + ``` + +--- + +## Build-Time Configuration + +**Build.rs watches these files:** +```rust +println!("cargo:rerun-if-changed=build/unmapped-glyph-names.json"); +println!("cargo:rerun-if-changed=build/std14-metrics.json"); +println!("cargo:rerun-if-changed=build/named-encodings.json"); +println!("cargo:rerun-if-changed=build/agl.json"); +println!("cargo:rerun-if-changed=build/font-fingerprints.json"); +println!("cargo:rerun-if-changed=build/predefined-cmaps/"); +println!("cargo:rerun-if-changed=build/glyph-shapes.json"); +``` + +**Checksum verification (TH-06 supply-chain gate):** +- All build data files are checksummed in `CHECKSUMS.sha256` +- Build fails if checksums don't match +- To regenerate: `sha256sum > CHECKSUMS.sha256` + +--- + +## Key Learnings + +1. **Configuration is build-time, not runtime** - `unmapped-glyph-names.json` is read at compile time and generates Rust code +2. **CMAP tests use two patterns** - Unit tests for config parsing, integration tests for extraction behavior +3. **Fixtures are documented in Markdown** - Each fixture has a corresponding `.md` file with structure, verification, and references +4. **Unmapped glyphs are filtered at entry creation** - The `is_unmapped_glyph_name()` check prevents unmapped glyphs from being added to CMAP entries +5. **Python generators for encoding fixtures** - `generate_unmapped_glyphs.py` creates PDFs with custom encodings +6. **Rust generators for form fixtures** - `generate_form_fixtures.rs` uses `lopdf` library for programmatic PDF creation + +--- + +## References + +- `crates/pdftract-core/tests/unmapped_glyph_names_config.rs` - Config parsing tests +- `crates/pdftract-core/tests/encoding_recovery.rs` - Integration tests +- `crates/pdftract-core/src/font/cmap.rs` - ToUnicode CMap parser +- `crates/pdftract-core/src/font/encoding.rs` - Named encodings and unmapped glyph handling +- `crates/pdftract-core/build.rs` - Build-time code generation +- `build/unmapped-glyph-names.json` - Unmapped glyph names configuration +- `tests/fixtures/encoding/no-mapping.md` - Example fixture documentation + +--- + +## Status + +✅ **Task Complete** - All acceptance criteria met. Ready to create test fixtures based on this understanding. + +**Date:** 2026-07-06 +**Bead:** bf-3tzsn