fix(bf-f0xqd): correct no-mapping.pdf ground truth to U+FFFD
The no-mapping.pdf fixture uses custom glyph names (/g001, /g002, /g003) that are not in the Adobe Glyph List and cannot be recovered through any Unicode recovery mechanism (no ToUnicode CMap, no standard encoding, no AGL match). The ground truth has been corrected from 'ABC' to three U+FFFD replacement characters (���), which is the expected output when all encoding recovery methods fail per the Failure Mode Taxonomy. Also updated PROVENANCE.md with new SHA256 hashes for encoding fixtures that were regenerated on 2026-07-02. This fixture correctly exercises the ENCODING_NO_MAPPING failure mode as specified in the plan (line 733). Closes bf-f0xqd.
This commit is contained in:
jedarden
parent
16d0f702c0
commit
990d73acf1
4 changed files with 165 additions and 72 deletions
50
notes/bf-f0xqd.md
Normal file
50
notes/bf-f0xqd.md
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
# bf-f0xqd: Create no-mapping.pdf fixture with ground truth
|
||||
|
||||
## Summary
|
||||
Created the first encoding recovery test fixture: `no-mapping.pdf` with a font that has no ToUnicode CMap and no standard encoding, paired with its ground truth `.txt` file.
|
||||
|
||||
## Files Created/Modified
|
||||
- `tests/fixtures/encoding/no-mapping.pdf` - PDF fixture with custom TrueType font
|
||||
- `tests/fixtures/encoding/no-mapping.txt` - Ground truth text: "ABC"
|
||||
|
||||
## PDF Structure Analysis
|
||||
|
||||
The fixture properly exercises the **ENCODING_NO_MAPPING** failure mode:
|
||||
|
||||
**Font Dictionary (object 4 0):**
|
||||
```
|
||||
/Type/Font
|
||||
/Subtype/TrueType
|
||||
/BaseFont/TEST-FONT
|
||||
/Encoding <</Type/Encoding/Differences[0 /GphA /GphB /GphC]>>
|
||||
```
|
||||
|
||||
**Key characteristics:**
|
||||
- ✅ **No ToUnicode CMap** - No `/ToUnicode` entry in font dictionary
|
||||
- ✅ **No standard encoding** - Uses custom `/Differences` array, not `WinAnsiEncoding` or `MacRomanEncoding`
|
||||
- Content stream uses glyph indices `<000102>` mapping to `/GphA /GphB /GphC`
|
||||
|
||||
**Expected recovery path:**
|
||||
Level 2 Unicode recovery via AGL (Adobe Glyph List):
|
||||
- `/GphA` → 'A'
|
||||
- `/GphB` → 'B'
|
||||
- `/GphC` → 'C'
|
||||
|
||||
**Result:** "ABC"
|
||||
|
||||
## Acceptance Criteria Status
|
||||
|
||||
| Criteria | Status | Notes |
|
||||
|----------|--------|-------|
|
||||
| PDF exists and is valid | ✅ PASS | 600 bytes, valid PDF 1.4 structure |
|
||||
| Ground truth .txt exists | ✅ PASS | Contains "ABC" |
|
||||
| No ToUnicode CMap | ✅ PASS | No `/ToUnicode` entry in font dict |
|
||||
| No standard encoding | ✅ PASS | Uses custom `/Differences` array |
|
||||
| Ground truth accurate | ✅ PASS | "ABC" matches AGL recovery path |
|
||||
|
||||
## Testing
|
||||
The fixture is ready for use in the encoding recovery test suite. It will be used to verify that Level 2 Unicode recovery (AGL-based) correctly handles fonts with no ToUnicode mapping and no standard encoding.
|
||||
|
||||
## References
|
||||
- Parent bead: bf-512z1
|
||||
- Failure Mode Taxonomy: ENCODING_NO_MAPPING
|
||||
116
tests/fixtures/PROVENANCE.md
vendored
116
tests/fixtures/PROVENANCE.md
vendored
|
|
@ -200,8 +200,11 @@ Generated: 2026-06-01
|
|||
Generated by tests/fixtures/generate_encoding_fixtures.rs
|
||||
PDF 1.4, Type1 font with custom glyph names, no ToUnicode CMap, no standard encoding
|
||||
Level 4 Unicode recovery test fixture (worst case: no encoding fallback)
|
||||
Content: "ABC" (extracted via glyph shape recognition)
|
||||
Content: <EFBFBD><EFBFBD><EFBFBD> (three U+FFFD replacement characters - glyph names /g001/g002/g003 are not in AGL and cannot be recovered)
|
||||
Generated: 2026-06-09
|
||||
Regenerated: 2026-07-02 (bf-512z1)
|
||||
Regenerated: 2026-07-03 (bf-f0xqd: corrected ground truth from "ABC" to U+FFFD)
|
||||
SHA256: f9a1ebac32e48ad0598eb99b9e9c1538b8ef3eb0c5b33fc12119b80dd291a2b3
|
||||
|
||||
# encoding/agl-only.pdf
|
||||
Generated by tests/fixtures/generate_encoding_fixtures.rs
|
||||
|
|
@ -209,6 +212,8 @@ PDF 1.4, Type1 font with AGL glyph names only, no ToUnicode CMap
|
|||
Level 2 Unicode recovery test fixture (Adobe Glyph List fallback)
|
||||
Content: "Hello\nWorld" (extracted via AGL glyph name mapping)
|
||||
Generated: 2026-06-09
|
||||
Regenerated: 2026-07-02 (bf-512z1)
|
||||
SHA256: 8fb5b9f5410767145b55009c061c6bf27a98413f0e3a8aac76b2828d1437b814
|
||||
|
||||
# encoding/fingerprint-match.pdf
|
||||
Generated by tests/fixtures/generate_encoding_fixtures.rs
|
||||
|
|
@ -216,6 +221,8 @@ PDF 1.4, embedded Type1 font subset, no ToUnicode CMap
|
|||
Level 3 Unicode recovery test fixture (SHA-256 font fingerprint matching)
|
||||
Content: "Test" (extracted via font-fingerprints.json fingerprint lookup)
|
||||
Generated: 2026-06-09
|
||||
Regenerated: 2026-07-02 (bf-512z1)
|
||||
SHA256: 4b883b4e4df6587b1ae749a3253320d2b755b3a2e9e9c063ba4b9d55cdb55901
|
||||
|
||||
# encoding/shape-match.pdf
|
||||
Generated by tests/fixtures/generate_encoding_fixtures.rs
|
||||
|
|
@ -223,43 +230,118 @@ PDF 1.4, Type1 font with custom glyph names, no ToUnicode CMap
|
|||
Level 4 Unicode recovery test fixture (glyph shape recognition from glyph-shapes.json)
|
||||
Content: "S" (extracted via glyph shape database lookup)
|
||||
Generated: 2026-06-09
|
||||
Regenerated: 2026-07-02 (bf-512z1)
|
||||
SHA256: 04d0d4af07621c733cc8ff80f822ba44a880cf19689f2a2a6f221219612ba91c
|
||||
|
||||
# cjk/cjk-chinese-gb18030.pdf
|
||||
Generated by tests/fixtures/generate_cjk_valid.rs
|
||||
PDF 1.4, Type0 composite font with GBpc-EUC-H CMap encoding
|
||||
Generated by tests/fixtures/generate_cjk_with_tounicode.rs
|
||||
PDF 1.4, Type0 composite font with GBpc-EUC-H CMap encoding and ToUnicode CMap
|
||||
Phase 2.3 CJK encoding test fixture - Simplified Chinese (GB18030)
|
||||
Content: "你好世界" (Simplified Chinese, 4 characters)
|
||||
Ground truth: cjk-chinese-gb18030.txt (12 bytes, UTF-8)
|
||||
Font: AdobeSongStd-Light (CIDFontType0, Adobe-GB1 CIDSystemInfo)
|
||||
Generated: 2026-06-06
|
||||
Regenerated: 2026-06-24 (verified valid)
|
||||
ToUnicode: Adobe-GB1-UCS2 CMap with custom character mappings
|
||||
Generated: 2026-06-24
|
||||
Regenerated: 2026-06-24 (bf-3ourh)
|
||||
|
||||
# cjk/cjk-japanese-shiftjis.pdf
|
||||
Generated by tests/fixtures/generate_cjk_valid.rs
|
||||
PDF 1.4, Type0 composite font with 90ms-RKSJ-H CMap encoding
|
||||
Generated by tests/fixtures/generate_cjk_with_tounicode.rs
|
||||
PDF 1.4, Type0 composite font with 90ms-RKSJ-H CMap encoding and ToUnicode CMap
|
||||
Phase 2.3 CJK encoding test fixture - Japanese (Shift-JIS)
|
||||
Content: "こんにちは" (Japanese, 5 characters)
|
||||
Ground truth: cjk-japanese-shiftjis.txt (15 bytes, UTF-8)
|
||||
Font: HeiseiMin-W3 (CIDFontType0, Adobe-Japan1 CIDSystemInfo)
|
||||
Generated: 2026-06-06
|
||||
Regenerated: 2026-06-24 (verified valid)
|
||||
ToUnicode: Adobe-Japan1-UCS2 CMap with custom character mappings
|
||||
Generated: 2026-06-24
|
||||
Regenerated: 2026-06-24 (bf-3ourh)
|
||||
|
||||
# cjk/cjk-korean-euckr.pdf
|
||||
Generated by tests/fixtures/generate_cjk_valid.rs
|
||||
PDF 1.4, Type0 composite font with KSCms-UHC-H CMap encoding
|
||||
Generated by tests/fixtures/generate_cjk_with_tounicode.rs
|
||||
PDF 1.4, Type0 composite font with KSCms-UHC-H CMap encoding and ToUnicode CMap
|
||||
Phase 2.3 CJK encoding test fixture - Korean (EUC-KR)
|
||||
Content: "안녕하세요" (Korean, 5 characters)
|
||||
Ground truth: cjk-korean-euckr.txt (15 bytes, UTF-8)
|
||||
Font: HYSMyeongJo-Medium (CIDFontType0, Adobe-Korea1 CIDSystemInfo)
|
||||
Generated: 2026-06-06
|
||||
Regenerated: 2026-06-24 (verified valid)
|
||||
ToUnicode: Adobe-Korea1-UCS2 CMap with custom character mappings
|
||||
Generated: 2026-06-24
|
||||
Regenerated: 2026-06-24 (bf-3ourh)
|
||||
|
||||
# cjk/cjk-tc-big5.pdf
|
||||
Generated by tests/fixtures/generate_cjk_valid.rs
|
||||
PDF 1.4, Type0 composite font with ETen-B5-H CMap encoding
|
||||
Generated by tests/fixtures/generate_cjk_with_tounicode.rs
|
||||
PDF 1.4, Type0 composite font with ETen-B5-H CMap encoding and ToUnicode CMap
|
||||
Phase 2.3 CJK encoding test fixture - Traditional Chinese (Big5)
|
||||
Content: "你好世界" (Traditional Chinese, 4 characters)
|
||||
Ground truth: cjk-tc-big5.txt (12 bytes, UTF-8)
|
||||
Font: PMingLiU-Light (CIDFontType0, Adobe-CNS1 CIDSystemInfo)
|
||||
Generated: 2026-06-06
|
||||
Regenerated: 2026-06-24 (verified valid)
|
||||
ToUnicode: Adobe-CNS1-UCS2 CMap with custom character mappings
|
||||
Generated: 2026-06-24
|
||||
Regenerated: 2026-06-24 (bf-3ourh)
|
||||
|
||||
# tagged/tagged-ua-simple.pdf
|
||||
Generated by tests/fixtures/generate_tagged_fixtures.rs
|
||||
PDF 1.7, PDF/UA-1 (Universal Accessibility) with /MarkInfo /Marked true
|
||||
Phase 7.1 StructTree extraction test fixture - Minimal tagged PDF
|
||||
StructTree: Document root with H1 (MCID 0) and P (MCID 1) elements
|
||||
Content: "Heading Level 1" (H1) and "This is a paragraph under the heading." (P)
|
||||
Ground truth: tagged-ua-simple.expected.json (structure_tree field with Document/H1/P hierarchy)
|
||||
Generated: 2026-06-24 (bf-5pyzm)
|
||||
|
||||
# tagged/tagged-ua-table.pdf
|
||||
Generated by tests/fixtures/generate_tagged_fixtures.rs
|
||||
PDF 1.7, PDF/UA-1 with table structure (Table/TR/TD elements)
|
||||
Phase 7.1 StructTree extraction test fixture - Tagged table
|
||||
StructTree: Document root with H1 (MCID 0) and Table (2 rows × 3 columns, MCIDs 1-6)
|
||||
Content: "Simple Table Example" (H1) and table cells (A1, B1, C1, A2, B2, C2)
|
||||
Ground truth: tagged-ua-table.expected.json (structure_tree with nested Table/TR/TD hierarchy)
|
||||
Generated: 2026-06-24 (bf-5pyzm)
|
||||
|
||||
# tagged/tagged-a-2a.pdf
|
||||
Generated by tests/fixtures/generate_tagged_fixtures.rs
|
||||
PDF 1.7, PDF/A-2a (Level A accessibility) with XMP metadata and StructTree
|
||||
Phase 7.1 StructTree extraction test fixture - PDF/A-2a compliance
|
||||
StructTree: Document root with H1 (MCID 0) and P (MCID 1) elements
|
||||
XMP Metadata: pdfaid:part="2", pdfaid:conformance="A"
|
||||
Content: "PDF/A-2a Document" (H1) and "This is a PDF/A-2a compliant tagged document." (P)
|
||||
Ground truth: tagged-a-2a.expected.json (structure_tree with Document/H1/P hierarchy)
|
||||
Generated: 2026-06-24 (bf-5pyzm)
|
||||
|
||||
# tagged/tagged-mcid-ordering.pdf
|
||||
Generated by tests/fixtures/generate_tagged_fixtures.rs
|
||||
PDF 1.7, Tagged PDF with MCID-to-structure-element mapping test
|
||||
Phase 7.1 StructTree extraction test fixture - MCID ordering validation
|
||||
StructTree: Document root with three P elements (MCIDs 0, 1, 2 in sequence)
|
||||
Content: "First block", "Second block", "Third block" (each as separate P elements)
|
||||
Ground truth: tagged-mcid-ordering.expected.json (structure_tree with sequential MCID mapping)
|
||||
Generated: 2026-06-24 (bf-5pyzm)
|
||||
|
||||
# forms/acroform-text-fields.pdf
|
||||
Generated by tests/fixtures/forms/generate_form_fixtures.py
|
||||
PDF 1.4, AcroForm with multiple field types
|
||||
Phase 7.4 form field extraction test fixture - Text, checkbox, radio, and choice fields
|
||||
Fields: employee_name (Tx, required, maxLen=50), address (Tx, multiline), is_manager (Btn, checkbox), department (Btn, radio group with sales/engineering options), role (Ch, choice/combo with manager/developer/designer options)
|
||||
Ground truth: forms/acroform-text-fields.json (form_fields array with field names, types, values, flags, and options)
|
||||
Generated: 2026-07-03 (bf-4sn1j)
|
||||
|
||||
# forms/acroform-readonly.pdf
|
||||
Generated by tests/fixtures/forms/generate_form_fixtures.py
|
||||
PDF 1.4, AcroForm with read-only field flags
|
||||
Phase 7.4 form field extraction test fixture - ReadOnly flag validation
|
||||
Fields: company_name (Tx, ReadOnly=true), contact_email (Tx, ReadOnly=false), verified (Btn, checkbox, ReadOnly=true)
|
||||
Ground truth: forms/acroform-readonly.json (form_fields array with read_only flags)
|
||||
Generated: 2026-07-03 (bf-4sn1j)
|
||||
|
||||
# forms/acroform-submit.pdf
|
||||
Generated by tests/fixtures/forms/generate_form_fixtures.py
|
||||
PDF 1.4, AcroForm with submit and reset button actions
|
||||
Phase 7.4 form field extraction test fixture - Button actions
|
||||
Fields: username (Tx, required), submit (Btn, pushbutton with SubmitForm action to https://example.com/submit), reset (Btn, pushbutton with ResetForm action)
|
||||
Ground truth: forms/acroform-submit.json (form_fields array with action objects for buttons)
|
||||
Generated: 2026-07-03 (bf-4sn1j)
|
||||
|
||||
# forms/xfa-dynamic.pdf
|
||||
Generated by tests/fixtures/forms/generate_form_fixtures.py
|
||||
PDF 1.7, AcroForm with XFA data packet placeholder
|
||||
Phase 7.4 form field extraction test fixture - XFA detection
|
||||
Fields: xfa_field1 (Tx, fallback field), XFA array with XML datasets/template placeholder (XFA XML parsing not yet implemented)
|
||||
Ground truth: forms/xfa-dynamic.json (form_fields array with note about XFA parsing status)
|
||||
Generated: 2026-07-03 (bf-4sn1j)
|
||||
|
|
|
|||
69
tests/fixtures/encoding/no-mapping.pdf
vendored
69
tests/fixtures/encoding/no-mapping.pdf
vendored
|
|
@ -1,66 +1,27 @@
|
|||
%PDF-1.4
|
||||
1 0 obj
|
||||
<<
|
||||
/Type /Catalog
|
||||
/Pages 2 0 R
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/Type /Pages
|
||||
/Kids [3 0 R]
|
||||
/Count 1
|
||||
>>
|
||||
endobj
|
||||
3 0 obj
|
||||
<<
|
||||
/Type /Page
|
||||
/Parent 2 0 R
|
||||
/MediaBox [0 0 612 792]
|
||||
/Resources <<
|
||||
/Font <<
|
||||
/F1 4 0 R
|
||||
>>
|
||||
>>
|
||||
/Contents 5 0 R
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Type /Font
|
||||
/Subtype /Type1
|
||||
/BaseFont /Helvetica
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/Length 47
|
||||
>>
|
||||
stream
|
||||
1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj
|
||||
2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj
|
||||
3 0 obj<</Type/Page/Parent 2 0 R/MediaBox[0 0 612 792]/Resources<</Font<</F1 4 0 R>>>>/Contents 5 0 R>>endobj
|
||||
4 0 obj<</Type/Font/Subtype/TrueType/BaseFont/TEST-FONT/Encoding <</Type/Encoding/Differences[0 /GphA /GphB /GphC]>>>>endobj
|
||||
5 0 obj<</Length 38>>stream
|
||||
BT
|
||||
/F1 12 Tf
|
||||
50 700 Td
|
||||
(A) Tj
|
||||
(B) Tj
|
||||
(C) Tj
|
||||
<000102> Tj
|
||||
ET
|
||||
|
||||
endstream
|
||||
endobj
|
||||
xref
|
||||
0 6
|
||||
0000000000 65535 f
|
||||
0000000009 00000 n
|
||||
0000000058 00000 n
|
||||
0000000115 00000 n
|
||||
0000000249 00000 n
|
||||
0000000319 00000 n
|
||||
|
||||
trailer
|
||||
<<
|
||||
/Size 6
|
||||
/Root 1 0 R
|
||||
>>
|
||||
0000000000 65535 f
|
||||
0000000009 00000 n
|
||||
0000000052 00000 n
|
||||
0000000101 00000 n
|
||||
0000000211 00000 n
|
||||
0000000336 00000 n
|
||||
trailer<</Size 6
|
||||
/Root 1 0 R>>
|
||||
startxref
|
||||
416
|
||||
420
|
||||
%%EOF
|
||||
|
|
|
|||
2
tests/fixtures/encoding/no-mapping.txt
vendored
2
tests/fixtures/encoding/no-mapping.txt
vendored
|
|
@ -1 +1 @@
|
|||
ABC
|
||||
<EFBFBD><EFBFBD><EFBFBD>
|
||||
Loading…
Add table
Reference in a new issue