docs(bf-3tzsn): document CMAP test structure and fixture exploration
- Located existing CMAP test files in crates/pdftract-core/tests/ - Identified test fixture patterns in tests/fixtures/encoding/ - Found unmapped_glyph_names configuration examples - Documented CMAP entry creation points and build-time processing - Documented test creation patterns for future reference Acceptance criteria: - Located existing CMAP test file(s) - Identified test fixture patterns - Found examples of unmapped_glyph_names config - Documented the test structure pattern in note - Ready to create test fixture based on this understanding Closes bf-3tzsn
This commit is contained in:
jedarden
parent
d8e17018d3
commit
aaa3772e62
1 changed files with 378 additions and 0 deletions
378
notes/bf-3tzsn.md
Normal file
378
notes/bf-3tzsn.md
Normal file
|
|
@ -0,0 +1,378 @@
|
|||
# CMAP Test Structure and Fixture Exploration — bf-3tzsn
|
||||
|
||||
## Overview
|
||||
|
||||
This note documents the CMAP (character map) test infrastructure, including existing test patterns, fixture structures, and configuration mechanisms for unmapped glyph handling.
|
||||
|
||||
## Acceptance Criteria: PASS ✅
|
||||
|
||||
- [x] Located existing CMAP test file(s)
|
||||
- [x] Identified test fixture patterns
|
||||
- [x] Found examples of unmapped_glyph_names config
|
||||
- [x] Documented the test structure pattern in this note
|
||||
- [x] Ready to create test fixture based on this understanding
|
||||
|
||||
---
|
||||
|
||||
## Test Files Structure
|
||||
|
||||
### 1. Unit Tests: `crates/pdftract-core/tests/unmapped_glyph_names_config.rs`
|
||||
|
||||
Tests the `UnmappedGlyphNamesConfig` structure that reads from `build/unmapped-glyph-names.json`:
|
||||
|
||||
```rust
|
||||
struct UnmappedGlyphNamesConfig {
|
||||
#[serde(default)]
|
||||
unmapped_glyph_names: Vec<String>,
|
||||
|
||||
#[serde(default)]
|
||||
description: Option<String>,
|
||||
|
||||
#[serde(default)]
|
||||
version: Option<String>,
|
||||
}
|
||||
```
|
||||
|
||||
**Test patterns observed:**
|
||||
- `test_unmapped_glyph_names_defaults_to_empty()` - Config without field defaults to empty Vec
|
||||
- `test_unmapped_glyph_names_specified()` - Config with field parses correctly
|
||||
- `test_unmapped_glyph_names_empty_array()` - Explicit `[]` parses correctly
|
||||
- `test_unmapped_glyph_names_minimal_config()` - Empty `{}` defaults all fields
|
||||
|
||||
### 2. Integration Tests: `crates/pdftract-core/tests/encoding_recovery.rs`
|
||||
|
||||
Tests Unicode recovery from PDFs without ToUnicode CMaps using the `EncodingFixture` pattern:
|
||||
|
||||
```rust
|
||||
struct EncodingFixture {
|
||||
name: &'static str,
|
||||
pdf_path: &'static str,
|
||||
truth_path: &'static str,
|
||||
description: &'static str,
|
||||
}
|
||||
```
|
||||
|
||||
**Test fixtures in corpus:**
|
||||
- `no-mapping` - PDF with no ToUnicode, custom encoding (worst case)
|
||||
- `agl-only` - AGL glyph names only (Level 2 recovery)
|
||||
- `fingerprint-match` - Embedded font for fingerprinting (Level 3)
|
||||
- `shape-match` - Subset font for shape recognition (Level 4)
|
||||
|
||||
**Test patterns observed:**
|
||||
- `test_no_mapping_fixture()` - Verifies CER ≥ 0.0 and recovery_rate ≤ 1.0
|
||||
- `test_agl_only_fixture()` - Perfect match assertions (CER = 0.0, rate = 1.0)
|
||||
- `test_corpus_recovery_rate()` - Calculates weighted average across corpus
|
||||
|
||||
---
|
||||
|
||||
## Fixture Structure
|
||||
|
||||
### Directory Layout
|
||||
|
||||
```
|
||||
tests/fixtures/
|
||||
├── encoding/ # Encoding/Unicode recovery fixtures
|
||||
│ ├── no-mapping.pdf
|
||||
│ ├── no-mapping.txt
|
||||
│ ├── no-mapping.md
|
||||
│ ├── agl-only.pdf
|
||||
│ ├── agl-only.txt
|
||||
│ ├── fingerprint-match.pdf
|
||||
│ ├── fingerprint-match.txt
|
||||
│ ├── shape-match.pdf
|
||||
│ ├── shape-match.txt
|
||||
│ ├── generate_unmapped_glyphs.py
|
||||
│ └── create_unmapped_comprehensive.py
|
||||
└── forms/ # Form fixtures (with generator pattern)
|
||||
└── generate_form_fixtures.rs
|
||||
```
|
||||
|
||||
### Fixture Documentation Pattern
|
||||
|
||||
Each fixture has a corresponding `.md` file with this structure:
|
||||
|
||||
**Example:** `tests/fixtures/encoding/no-mapping.md`
|
||||
|
||||
```markdown
|
||||
# no-mapping.pdf Fixture
|
||||
|
||||
## Purpose
|
||||
Level 4 Unicode recovery test fixture - worst case scenario...
|
||||
|
||||
## Structure
|
||||
### PDF Properties
|
||||
- PDF Version: 1.4
|
||||
- Pages: 1
|
||||
- Page Size: 612 x 792 pts (Letter)
|
||||
|
||||
### Font Details
|
||||
Name: CustomNoMap
|
||||
Type: Type 1
|
||||
Encoding: Custom (Differences array)
|
||||
Embedded: No
|
||||
ToUnicode CMap: No
|
||||
|
||||
## Inspection Commands
|
||||
### Basic PDF info:
|
||||
pdfinfo tests/fixtures/encoding/no-mapping.pdf
|
||||
|
||||
### Font details:
|
||||
pdffonts tests/fixtures/encoding/no-mapping.pdf
|
||||
|
||||
## Verification
|
||||
# Check SHA256 matches expected value
|
||||
sha256sum tests/fixtures/encoding/no-mapping.pdf
|
||||
|
||||
# Verify extraction produces U+FFFD characters
|
||||
./target/release/pdftract extract tests/fixtures/encoding/no-mapping.pdf \
|
||||
--format text -o /tmp/no-mapping-output.txt
|
||||
|
||||
## Test References
|
||||
- tests/debug_encoding_pdf.rs
|
||||
- tests/debug_encoding_fixtures.rs
|
||||
- tests/encoding_recovery.rs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Unmapped Glyph Names Configuration
|
||||
|
||||
### Configuration File: `build/unmapped-glyph-names.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"unmapped_glyph_names": [
|
||||
".notdef",
|
||||
".null",
|
||||
"g000",
|
||||
"g001",
|
||||
"g002",
|
||||
"g003",
|
||||
"g004",
|
||||
"g005",
|
||||
"g006",
|
||||
"g007",
|
||||
"g008",
|
||||
"g009"
|
||||
],
|
||||
"description": "Glyph names that should be skipped during CMAP and ToUnicode entry creation...",
|
||||
"version": "1.0"
|
||||
}
|
||||
```
|
||||
|
||||
### Build-Time Processing: `crates/pdftract-core/build.rs`
|
||||
|
||||
The `generate_unmapped_glyph_names()` function:
|
||||
|
||||
1. Reads `build/unmapped-glyph-names.json`
|
||||
2. Emits build warning if not found (uses default `.notdef` only)
|
||||
3. Generates `UNMAPPED_GLYPH_NAMES` LazyLock<HashSet<&'static str>> in `$OUT_DIR`
|
||||
4. Include path: `include!(concat!(env!("OUT_DIR"), "/unmapped_glyph_names.rs"));`
|
||||
|
||||
### Runtime Usage: `crates/pdftract-core/src/font/encoding.rs`
|
||||
|
||||
The `DifferencesOverlay` struct uses the unmapped set:
|
||||
|
||||
```rust
|
||||
pub struct DifferencesOverlay {
|
||||
entries: Vec<(u8, Arc<str>)>,
|
||||
unmapped_glyph_names: std::collections::HashSet<String>,
|
||||
}
|
||||
|
||||
impl DifferencesOverlay {
|
||||
fn default_unmapped_glyph_names() -> std::collections::HashSet<String> {
|
||||
crate::font::unmapped::UNMAPPED_GLYPH_NAMES
|
||||
.iter()
|
||||
.map(|s| s.to_string())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn is_unmapped_glyph_name(&self, name: &str) -> bool {
|
||||
self.unmapped_glyph_names.contains(name)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**CMAP entry creation marker (line 221-228 in encoding.rs):**
|
||||
```rust
|
||||
// MARKER: CMAP entry creation point - Type1 font encoding differences.
|
||||
// See notes/bf-e4uvb-child-1.md for documentation.
|
||||
if !overlay.is_unmapped_glyph_name(&name) {
|
||||
overlay.entries.push((cursor as u8, Arc::clone(name)));
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Generator Patterns
|
||||
|
||||
### Python Generator: `tests/fixtures/encoding/generate_unmapped_glyphs.py`
|
||||
|
||||
**Purpose:** Generate PDF fixtures with custom glyph names and encodings
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python3 generate_unmapped_glyphs.py [OPTIONS]
|
||||
--output PATH Output PDF path (default: unmapped-glyphs.pdf)
|
||||
--ground-truth PATH Ground truth .txt path (default: unmapped-glyphs.txt)
|
||||
--title NAME Font base name (default: UnmappedTestFont)
|
||||
--glyphs JSON Glyph mappings as JSON string
|
||||
```
|
||||
|
||||
**Default glyph set (10 character codes):**
|
||||
- Codes 0-2: `/g001`, `/g002`, `/g003` (PUA unmapped)
|
||||
- Codes 3-6: `/CustomA`, `/CustomB`, `/NotAGlyph`, `/glyph_0041` (unmapped)
|
||||
- Codes 7-9: `/A`, `/B`, `/space` (AGL mapped)
|
||||
|
||||
### Rust Generator: `tests/fixtures/forms/generate_form_fixtures.rs`
|
||||
|
||||
**Pattern for creating PDF fixtures programmatically:**
|
||||
|
||||
```rust
|
||||
use lopdf::dictionary;
|
||||
use lopdf::{Dictionary, Document, Object, ObjectId, StringFormat};
|
||||
|
||||
fn create_simple_page(content: &[u8], doc: &mut Document) -> ObjectId {
|
||||
let mut page_dict = Dictionary::new();
|
||||
page_dict.set("Type", "Page");
|
||||
page_dict.set("MediaBox", Object::Array(vec![
|
||||
Object::Real(0.0), Object::Real(0.0),
|
||||
Object::Real(612.0), Object::Real(792.0)
|
||||
]));
|
||||
// ... add resources and content stream
|
||||
doc.add_object(page_dict)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CMAP Source Code Structure
|
||||
|
||||
### CMap Parser: `crates/pdftract-core/src/font/cmap.rs`
|
||||
|
||||
**Purpose:** Parse `/ToUnicode` streams from PDF fonts as PostScript CMap programs
|
||||
|
||||
**Supported CMap syntax:**
|
||||
- `beginbfchar` / `endbfchar`: Single-character mappings
|
||||
- `beginbfrange` / `endbfrange`: Range mappings (contiguous and explicit array)
|
||||
- `usecmap`: Inheritance from named CMaps (stub - emits diagnostic)
|
||||
- Comments: `%` to end of line (stripped by lexer)
|
||||
|
||||
**CMAP entry creation marker (line 83-85 in cmap.rs):**
|
||||
```rust
|
||||
/// MARKER: CMAP entry creation point - this is where individual ToUnicode
|
||||
/// mappings are stored. Called by parse_beginbfchar() and parse_beginbfrange().
|
||||
/// See notes/bf-e4uvb-child-1.md for documentation.
|
||||
pub fn add_mapping(&mut self, src: Vec<u8>, dst: Vec<char>) {
|
||||
self.mappings.insert(src, dst);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Predefined CMaps
|
||||
|
||||
**Location:** `crates/pdftract-core/build/predefined-cmaps/`
|
||||
|
||||
**Files:**
|
||||
- `adobe-gb1.json` - Adobe-GB1-UCS2 (Simplified Chinese)
|
||||
- `adobe-cns1.json` - Adobe-CNS1-UCS2 (Traditional Chinese)
|
||||
- `adobe-japan1.json` - Adobe-Japan1-UCS2 (Japanese)
|
||||
- `adobe-korea1.json` - Adobe-Korea1-UCS2 (Korean)
|
||||
|
||||
**Example structure (adobe-japan1.json):**
|
||||
```json
|
||||
{
|
||||
"236": "あ",
|
||||
"237": "い",
|
||||
"238": "う"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Creation Pattern Summary
|
||||
|
||||
### To create a new CMAP test:
|
||||
|
||||
1. **Create fixture files:**
|
||||
- `<name>.pdf` - Test PDF
|
||||
- `<name>.txt` - Ground truth expected output
|
||||
- `<name>.md` - Documentation following the pattern above
|
||||
|
||||
2. **Add fixture to corpus:**
|
||||
- Add `EncodingFixture` entry to `get_fixtures()` in `encoding_recovery.rs`
|
||||
- Include paths relative to `crates/pdftract-core/tests/`
|
||||
|
||||
3. **Write test function:**
|
||||
```rust
|
||||
#[test]
|
||||
fn test_<name>_fixture() {
|
||||
let fixture = &get_fixtures()[index];
|
||||
let result = test_encoding_fixture(fixture).unwrap();
|
||||
// Assertions based on fixture expectations
|
||||
}
|
||||
```
|
||||
|
||||
4. **Update unmapped_glyph_names if needed:**
|
||||
- Edit `build/unmapped-glyph-names.json`
|
||||
- Add new glyph names to `unmapped_glyph_names` array
|
||||
- Rebuild to regenerate code
|
||||
|
||||
5. **Regenerate CHECKSUMS.sha256:**
|
||||
```bash
|
||||
cd crates/pdftract-core/build
|
||||
sha256sum unmapped-glyph-names.json >> CHECKSUMS.sha256
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Build-Time Configuration
|
||||
|
||||
**Build.rs watches these files:**
|
||||
```rust
|
||||
println!("cargo:rerun-if-changed=build/unmapped-glyph-names.json");
|
||||
println!("cargo:rerun-if-changed=build/std14-metrics.json");
|
||||
println!("cargo:rerun-if-changed=build/named-encodings.json");
|
||||
println!("cargo:rerun-if-changed=build/agl.json");
|
||||
println!("cargo:rerun-if-changed=build/font-fingerprints.json");
|
||||
println!("cargo:rerun-if-changed=build/predefined-cmaps/");
|
||||
println!("cargo:rerun-if-changed=build/glyph-shapes.json");
|
||||
```
|
||||
|
||||
**Checksum verification (TH-06 supply-chain gate):**
|
||||
- All build data files are checksummed in `CHECKSUMS.sha256`
|
||||
- Build fails if checksums don't match
|
||||
- To regenerate: `sha256sum <files> > CHECKSUMS.sha256`
|
||||
|
||||
---
|
||||
|
||||
## Key Learnings
|
||||
|
||||
1. **Configuration is build-time, not runtime** - `unmapped-glyph-names.json` is read at compile time and generates Rust code
|
||||
2. **CMAP tests use two patterns** - Unit tests for config parsing, integration tests for extraction behavior
|
||||
3. **Fixtures are documented in Markdown** - Each fixture has a corresponding `.md` file with structure, verification, and references
|
||||
4. **Unmapped glyphs are filtered at entry creation** - The `is_unmapped_glyph_name()` check prevents unmapped glyphs from being added to CMAP entries
|
||||
5. **Python generators for encoding fixtures** - `generate_unmapped_glyphs.py` creates PDFs with custom encodings
|
||||
6. **Rust generators for form fixtures** - `generate_form_fixtures.rs` uses `lopdf` library for programmatic PDF creation
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- `crates/pdftract-core/tests/unmapped_glyph_names_config.rs` - Config parsing tests
|
||||
- `crates/pdftract-core/tests/encoding_recovery.rs` - Integration tests
|
||||
- `crates/pdftract-core/src/font/cmap.rs` - ToUnicode CMap parser
|
||||
- `crates/pdftract-core/src/font/encoding.rs` - Named encodings and unmapped glyph handling
|
||||
- `crates/pdftract-core/build.rs` - Build-time code generation
|
||||
- `build/unmapped-glyph-names.json` - Unmapped glyph names configuration
|
||||
- `tests/fixtures/encoding/no-mapping.md` - Example fixture documentation
|
||||
|
||||
---
|
||||
|
||||
## Status
|
||||
|
||||
✅ **Task Complete** - All acceptance criteria met. Ready to create test fixtures based on this understanding.
|
||||
|
||||
**Date:** 2026-07-06
|
||||
**Bead:** bf-3tzsn
|
||||
Loading…
Add table
Reference in a new issue