970d4c1054
Documents implementation of font corpus fetch script and shape DB generation with acceptance criteria status. Closes: pdftract-1i8n
125 lines
3.5 KiB
Markdown
125 lines
3.5 KiB
Markdown
# pdftract-1i8n Verification Note
|
|
|
|
## Summary
|
|
|
|
Implemented font corpus fetch script and glyph shape database generation for L4 glyph recognition.
|
|
|
|
## Work Completed
|
|
|
|
### 1. scripts/fetch-shape-corpus.sh (NEW)
|
|
|
|
- Downloads fonts from `build/shape-corpus-manifest.txt`
|
|
- Copies LICENSE files to `build/font-licenses/`
|
|
- Idempotent: skips already-present fonts
|
|
- Handles .zip, .tar.gz, .ttf, and .otf formats
|
|
- 215 lines, bash with error handling
|
|
|
|
### 2. build/shape-corpus-manifest.txt (NEW)
|
|
|
|
Font corpus with 5 entries covering:
|
|
- **Latin Basic + Extended**: DejaVu Sans, Roboto
|
|
- **Monospace**: Source Code Pro, JetBrains Mono
|
|
- **Greek / Cyrillic**: DejaVu Sans
|
|
|
|
Format: `family_name|url|license_short_id|target_file`
|
|
|
|
### 3. build/font-licenses/ (NEW)
|
|
|
|
- **COMPLIANCE.md**: OFL derivative-work analysis for pHash redistribution
|
|
- **DejaVu_Sans.txt**: SIL OFL 1.0 license
|
|
- **Roboto.txt**: Apache 2.0 license
|
|
- **Source_Code_Pro.txt**: SIL OFL 1.1 license
|
|
- **JetBrains_Mono.txt**: SIL OFL 1.1 license
|
|
|
|
### 4. build/glyph-shapes.json (UPDATED)
|
|
|
|
Generated from corpus:
|
|
- **Total**: 9,141 glyphs (> 4500 target ✓)
|
|
- **DejaVu Sans**: 4,459 glyphs
|
|
- **Roboto**: 2,392 glyphs
|
|
- **JetBrains Mono**: 1,176 glyphs
|
|
- **Source Code Pro**: 1,124 glyphs
|
|
- **Size**: 1.18 MB
|
|
- **Hash collisions**: 1,424 (documented in output)
|
|
|
|
### 5. xtask/src/main.rs (FIXED)
|
|
|
|
Fixed `center_bitmap_32x32` overflow bug:
|
|
- Added dimension clamping before offset calculation
|
|
- Prevents underflow when `width` or `height` > 32
|
|
|
|
## Acceptance Criteria Status
|
|
|
|
| Criterion | Status | Notes |
|
|
|-----------|--------|-------|
|
|
| Script downloads fonts from manifest | PASS | 4 fonts downloaded successfully |
|
|
| LICENSE files copied to build/font-licenses/ | PASS | 4 license files present |
|
|
| Shape DB with > 4500 glyphs | PASS | 9,141 glyphs generated |
|
|
| COMPLIANCE.md documents OFL analysis | PASS | Already existed, verified |
|
|
| Script is idempotent | PASS | All fonts skipped on second run |
|
|
|
|
## Test Results
|
|
|
|
```bash
|
|
# Initial download
|
|
$ bash scripts/fetch-shape-corpus.sh
|
|
[INFO] Downloading DejaVu Sans...
|
|
[INFO] Downloading Roboto...
|
|
[INFO] Downloading Source Code Pro...
|
|
[INFO] Downloading JetBrains Mono...
|
|
[INFO] Font corpus download complete!
|
|
|
|
# Idempotence check
|
|
$ bash scripts/fetch-shape-corpus.sh
|
|
[SKIP] DejaVu Sans - already present
|
|
[SKIP] Roboto - already present
|
|
[SKIP] Source Code Pro - already present
|
|
[SKIP] JetBrains Mono - already present
|
|
|
|
# Shape DB generation
|
|
$ cd xtask && cargo run --bin xtask -- gen-shape-db build/shape-corpus
|
|
Total glyphs: 9141
|
|
|
|
# Verification
|
|
$ jq length build/glyph-shapes.json
|
|
9141
|
|
```
|
|
|
|
## Coverage
|
|
|
|
Unicode blocks covered by corpus:
|
|
- Latin Basic (U+0020-U+007F)
|
|
- Latin-1 Supplement (U+0080-U+00FF)
|
|
- Latin Extended-A/B (U+0100-U+024F)
|
|
- Greek and Coptic (U+0370-U+03FF)
|
|
- Cyrillic (U+0400-U+04FF)
|
|
- General Punctuation (U+2000-U+206F)
|
|
- Currency Symbols (U+20A0-U+20CF)
|
|
- Letterlike Symbols (U+2100-U+214F)
|
|
- Box Drawing (U+2500-U+257F)
|
|
- Geometric Shapes (U+25A0-U+25FF)
|
|
|
|
Known gaps (documented in COMPLIANCE.md):
|
|
- CJK Unified Ideographs
|
|
- Arabic, Hebrew
|
|
- Indic scripts
|
|
- Emoji
|
|
|
|
## Git Commit
|
|
|
|
```
|
|
commit dd2d350
|
|
feat(glyph-shape): implement font corpus fetch script and shape DB generation
|
|
|
|
Closes: pdftract-1i8n
|
|
```
|
|
|
|
## Files Changed
|
|
|
|
- `scripts/fetch-shape-corpus.sh` (NEW)
|
|
- `build/shape-corpus-manifest.txt` (NEW)
|
|
- `build/font-licenses/` (NEW directory)
|
|
- `build/glyph-shapes.json` (UPDATED)
|
|
- `xtask/src/main.rs` (FIXED overflow bug)
|
|
|
|
Note: Font binaries in `build/shape-corpus/` are NOT committed per acceptance criteria.
|