jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pdftract/scripts
History
jedarden dd2d3502c6 feat(glyph-shape): implement font corpus fetch script and shape DB generation 
Implemented scripts/fetch-shape-corpus.sh for downloading open-licensed
font corpus and generating glyph shape database for L4 recognition.

- Script downloads fonts from build/shape-corpus-manifest.txt
- Copies LICENSE files to build/font-licenses/ for compliance
- Idempotent: skips already-present fonts
- Fixed xtask center_bitmap_32x32 overflow bug (width/height > 32)

Generated build/glyph-shapes.json with 9,141 glyphs (> 4500 target):
  - DejaVu Sans: 4,459 glyphs (Latin Extended, Greek, Cyrillic)
  - Roboto: 2,392 glyphs (Latin Basic, extended)
  - JetBrains Mono: 1,176 glyphs (monospace)
  - Source Code Pro: 1,124 glyphs (monospace)

build/font-licenses/COMPLIANCE.md documents OFL derivative-work analysis
for pHash data redistribution.

Closes: pdftract-1i8n
2026-05-24 09:48:29 -04:00
..
ci docs(pdftract-5l9m): add CI validation script and verification note 2026-05-18 01:05:33 -04:00
check-provenance.sh fix(pdftract-5z5d8): fix provenance validation script 2026-05-17 23:43:37 -04:00
check-secrets.sh feat(pdftract-59zz): implement MCP bearer token ingress channels and TH-03 enforcement 2026-05-18 02:47:54 -04:00
fetch-shape-corpus.sh feat(glyph-shape): implement font corpus fetch script and shape DB generation 2026-05-24 09:48:29 -04:00
generate-minimal-pdf.sh feat(bf-1g1fd): implement CI memory-ceiling gate with cgroup MemoryMax enforcement 2026-05-23 13:22:55 -04:00
generate_test_corpus.py test(classifier): add 200-document labeled corpus for Phase 5.6 2026-05-17 07:16:02 -04:00
README.md test(bf-5dnh1): add memory ceiling enforcement for proptests 2026-05-23 13:39:04 -04:00
run-fuzz-with-limits.sh feat(bf-1g1fd): implement CI memory-ceiling gate with cgroup MemoryMax enforcement 2026-05-23 13:22:55 -04:00
run-proptest-with-limits.sh test(bf-5dnh1): add memory ceiling enforcement for proptests 2026-05-23 13:39:04 -04:00

README.md

Scripts

This directory contains utility scripts for pdftract development and testing.

Memory Ceiling Enforcement

Fuzz Tests (run-fuzz-with-limits.sh)

Runs cargo-fuzz targets with memory limits to ensure pathological inputs fail fast:

scripts/run-fuzz-with-limits.sh [target]

Memory limits:

  • Cgroup MemoryMax: 1536 MB (hard ceiling)
  • Libfuzzer RSS limit: 1024 MB (per-execution)
  • Libfuzzer malloc limit: 1024 MB (total)

Environment:

  • FUZZ_TIME_SECONDS: Time per target (default: 60)
  • MEMORY_MAX_MB: Cgroup limit in MB (default: 1536)
  • RSS_LIMIT_MB: Libfuzzer RSS limit (default: 1024)

Implementation: Uses cgroup v2 MemoryMax (preferred) or cgroup v1 memory.limit_in_bytes with OOM killer disabled for clean failure mode.

Property Tests (run-proptest-with-limits.sh)

Runs proptest modules with memory limits:

scripts/run-proptest-with-limits.sh [test_name]

Memory limits:

  • Cgroup MemoryMax: 2048 MB (hard ceiling)

Environment:

  • PROPTEST_CASES: Test cases per module (default: 1000)
  • MEMORY_MAX_MB: Cgroup limit in MB (default: 2048)
  • PROPTEST_SEED: Proptest seed (default: random)

Proptest modules: lexer, object_parser, xref, stream, cmap_parser

Input size caps: All proptest strategies are bounded:

  • Lexer/object parser: up to 10 KB inputs
  • Xref/stream parsers: up to 100 KB inputs
  • Nested structures: depth-limited (e.g., 500 for parser depth checks)

These bounds ensure tests complete quickly while still exercising edge cases.

Why Memory Ceilings?

Per bf-1g1fd and the Quality Targets (plan.md Phase 0.4), adversarial inputs must not OOM the host. Memory ceilings enforce:

  1. Clean failure mode - Allocation errors instead of host OOM
  2. Fast failure - Pathological cases abort immediately at the limit
  3. Regressions as test failures - Memory growth is caught in CI

CI enforces these limits via cgroup MemoryMax in .ci/argo-workflows/pdftract-ci.yaml (proptests) and .ci/argo-workflows/pdftract-nightly-fuzz.yaml (fuzz).

Other Scripts

generate-minimal-pdf.sh

Generates minimal valid PDF documents for testing.

check-provenance.sh

Verifies binary provenance and SBOM signatures.

check-secrets.sh

Scans for accidental secrets in committed code.

generate_test_corpus.py

Generates synthetic PDF test corpus.