jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pdftract/build
History
jedarden dd2d3502c6 feat(glyph-shape): implement font corpus fetch script and shape DB generation 
Implemented scripts/fetch-shape-corpus.sh for downloading open-licensed
font corpus and generating glyph shape database for L4 recognition.

- Script downloads fonts from build/shape-corpus-manifest.txt
- Copies LICENSE files to build/font-licenses/ for compliance
- Idempotent: skips already-present fonts
- Fixed xtask center_bitmap_32x32 overflow bug (width/height > 32)

Generated build/glyph-shapes.json with 9,141 glyphs (> 4500 target):
  - DejaVu Sans: 4,459 glyphs (Latin Extended, Greek, Cyrillic)
  - Roboto: 2,392 glyphs (Latin Basic, extended)
  - JetBrains Mono: 1,176 glyphs (monospace)
  - Source Code Pro: 1,124 glyphs (monospace)

build/font-licenses/COMPLIANCE.md documents OFL derivative-work analysis
for pHash data redistribution.

Closes: pdftract-1i8n
2026-05-24 09:48:29 -04:00
..
font-licenses feat(glyph-shape): implement font corpus fetch script and shape DB generation 2026-05-24 09:48:29 -04:00
frequency.json feat(xtask): implement gen-shape-db subcommand for glyph pHash database 2026-05-24 05:40:44 -04:00
glyph-shapes.json feat(glyph-shape): implement font corpus fetch script and shape DB generation 2026-05-24 09:48:29 -04:00
README.md feat(xtask): implement gen-shape-db subcommand for glyph pHash database 2026-05-24 05:40:44 -04:00
shape-corpus-manifest.txt feat(glyph-shape): implement font corpus fetch script and shape DB generation 2026-05-24 09:48:29 -04:00

README.md

Glyph Shape Database Generation

Overview

The cargo xtask gen-shape-db command generates a perceptual hash (pHash) database from TrueType/OpenType font files. This database is used for glyph shape recognition in PDF text extraction.

Usage

# From workspace root
cargo xtask gen-shape-db <fonts-dir> [output-path]

# Example
cargo xtask gen-shape-db /path/to/fonts build/glyph-shapes.json

Arguments

  • fonts-dir: Path to directory containing .ttf or .otf font files (recursively searched)
  • output-path: Optional output path (default: build/glyph-shapes.json)

Font Requirements

Fonts MUST be open-licensed:

  • Google Fonts (Apache 2.0 / OFL)
  • SIL Open Font License fonts
  • Other permissive licenses compatible with PDF extraction

Output Format

The output is a JSON array of glyph entries:

[
  {
    "phash_hex": "0123456789abcdef",
    "char": "A",
    "source_font": "LiberationSans-Regular.ttf",
    "frequency_rank": 30
  },
  ...
]

Character Frequency

The command reads build/frequency.json for character frequency rankings. If not found, all characters are assigned rank 0.

Format: {"A": 30, "B": 47, ...} where higher values = more common.

Suggested Fonts

For comprehensive coverage, use these open-licensed fonts:

  • Liberation Sans
  • DejaVu Sans
  • Source Code Pro
  • Noto Sans (covers Latin, Greek, Cyrillic)
  • Roboto

Example Setup

# Download Google Fonts
git clone https://github.com/google/fonts.git /tmp/fonts

# Generate database
cargo xtask gen-shape-db /tmp/fonts/ofl/liberationsans build/glyph-shapes.json

# Expected: ~5000 glyphs covering Latin, Greek, Cyrillic, symbols

License Attribution

Font license texts should be stored in build/font-licenses/ with a README.md documenting the source and license terms for each font used.

Algorithm

  1. Load each font file using fontdue
  2. For each Unicode codepoint (0x0000-0xFFFF):
    • Check if font has a glyph for the character
    • Rasterize at 32x32 pixels
    • Center the bitmap on a 32x32 canvas
    • Compute pHash via 32x32 DCT → 8x8 low-freq coefficients → median threshold
  3. Deduplicate by (pHash, char) pairs
  4. Handle cross-character collisions by keeping higher-frequency character
  5. Sort by pHash ascending and output JSON

Determinism

The output is byte-identical when re-run on the same input fonts and frequency data.