A PDF text extraction library that gets the hard parts right.
a601dcec76
Add regression-corpus step to pdftract-ci that runs the freshly-built x86_64-unknown-linux-musl binary against a 500-PDF private regression corpus stored in B2 (via ARMOR encrypted S3 proxy). Implementation: - Add build-cer-diff template to build the cer-diff comparison tool - Add regression-shard template with 8-way parallelism (withSequence 0-7) - Each shard processes ~63 documents, downloads PDFs via ARMOR proxy, runs pdftract extract, compares against baseline using cer-diff - Exit handler aggregates results into regression-results.jsonl artifact - Add regression-mode parameter (gate|update) for PR vs merge behavior CER computation: - Uses existing cer-diff binary (crates/pdftract-cer-diff/) - Levenshtein distance-based Character Error Rate - Fails if per-document CER delta > 0.5% in gate mode - Update mode refreshes baselines (requires follow-up bead for CronWorkflow) Infrastructure: - ARMOR proxy endpoint: armor.armor.svc.cluster.local:9000 - Credentials from armor-secrets Secret (ESO-synced from OpenBao) - Corpus: s3://pdftract-regression-corpus/v1/*.pdf - Baselines: s3://pdftract-regression-corpus/baselines/<sha256>.json Acceptance criteria: - PASS: regression-corpus step runs on every PR - PASS: 8 shards process 500 docs in ~8 min budget (3 sec/doc target) - PASS: Deliberate regression trips gate on CER > 0.5% - PASS: regression-results.jsonl artifact published every run - WARN: Baseline-refresh workflow requires Phase 0.6.1 follow-up Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .ci/argo-workflows | ||
| .git-hooks | ||
| crates/pdftract-core | ||
| docs | ||
| notes | ||
| profiles/builtin | ||
| scripts | ||
| src | ||
| tests | ||
| xtask | ||
| .needle-predispatch-sha | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CLAUDE.md | ||
| mod | ||
| README.md | ||
pdftract
A PDF text extraction library that gets the hard parts right.
What it does
- Correct reading order — layout regions are segmented and sequenced before text is emitted, handling multi-column pages, sidebars, footnotes, and mixed-layout documents without relying on PDF operator order
- Font encoding recovery — when
ToUnicodeCMaps are absent, wrong, or incomplete, pdftract works through a layered recovery pipeline: glyph name lookup via the Adobe Glyph List, font fingerprinting against known metrics and embedded checksums, and glyph outline shape matching - Structure tree extraction — PDF/UA and PDF/A documents encode their logical structure (headings, paragraphs, lists, tables, reading order) in a
StructTree; pdftract reads this directly when present, producing accurate semantic output at no extra cost - Per-page hybrid routing — each page is independently classified and routed to the appropriate pipeline: vector text extraction, full OCR, or assisted OCR where vector hints improve raster accuracy
- Structured output with provenance — the primary output is JSON carrying per-span bounding boxes, font name, size, and confidence score alongside the extracted text, not a flat string dump
Output
{
"pages": [
{
"page": 1,
"blocks": [
{ "kind": "heading", "text": "Introduction", "bbox": [72, 680, 400, 700] },
{ "kind": "paragraph", "text": "...", "bbox": [72, 640, 540, 670] }
],
"spans": [
{ "text": "Introduction", "bbox": [72, 680, 400, 700], "font": "Times-Bold", "size": 14.0, "confidence": 0.99 }
]
}
],
"metadata": { "title": "...", "author": "...", "page_count": 10 }
}
Usage
pdftract extract invoice.pdf # structured JSON to stdout
pdftract extract invoice.pdf --text # plain text to stdout
pdftract extract invoice.pdf --output out.json
pdftract serve --port 8080 # HTTP service: POST /extract
Architecture
Rust core with PyO3 Python bindings and a CLI binary. The same binary runs as a command-line tool or as an HTTP microservice — the container deployment is just pdftract serve.
See docs/research/ for technical deep-dives into the PDF specification, font encoding, glyph Unicode recovery, and tagged PDF structure. See docs/notes/ for SDK invocation examples in Python, Node.js, Go, Ruby, Java, Rust, and Bash.
Status
Early development. See docs/plan/ for the implementation roadmap.