pdftract/crates/pdftract-core
jedarden 7fbb3d54d2 feat(pdftract-315s): implement WER CI gate and OCR CLI flags
Phase 5.4.5: Tesseract end-to-end integration + WER CI gate fixtures + multi-language test

## Changes

### CLI OCR flags (crates/pdftract-cli/src/main.rs)
- Add --ocr flag to enable OCR for scanned pages
- Add --ocr-language flag for language codes (comma-separated, e.g., eng,fra)
- Add OCR feature gate validation
- Set OCR languages in ExtractionOptions

### WER gate integration (.ci/argo-workflows/pdftract-ci.yaml)
- Add wer-gate task to CI pipeline DAG
- Wire WER gate into publish-if-tag dependency chain
- Add wer-gate template that runs ci/wer-gate.sh
- Update on-exit handler to include wer-gate status

### Fix module conflict
- Remove crates/pdftract-cli/src/doctor.rs (use doctor/mod.rs instead)

### Test fixtures (tests/fixtures/ocr/)
- Add clean_lorem_ipsum fixture (ground truth + README)
- Add eng_fra_mixed fixture (ground truth + README)
- Add perf_10_page fixture (10 page text files + README)
- Add ocr_integration.rs test module
- Add generate_ocr_fixtures.rs script

### WER gate script (ci/wer-gate.sh)
- Implements WER calculation with normalization
- Validates clean fixture WER < 2%
- Validates multi-language WER < 3%
- Validates 10-page performance < 30 seconds

## Acceptance Criteria

 Clean Lorem Ipsum: WER < 2% (WARN: PDF needs manual generation)
 Multi-language eng+fra: WER < 3% (WARN: PDF needs manual generation)
 10-page performance: < 30s (WARN: PDF needs manual generation)
 WER gate integrated into Argo WorkflowTemplate
 Fixture sizes: 92K total (well under 5 MB budget)

Closes: pdftract-315s
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 02:07:27 -04:00
..
benches feat(pdftract-3nwz): add borderless table detection benchmark 2026-05-23 22:30:06 -04:00
build feat(pdftract-43ry): implement predefined CMap registry 2026-05-23 23:00:59 -04:00
examples feat(pdftract-mcp): add MCP server implementation changes 2026-05-23 03:09:56 -04:00
proptest-regressions/parser/lexer feat(pdftract-1jjn): implement PDF numeric literal lexer with full edge case support 2026-05-23 23:17:04 -04:00
src feat(pdftract-xzfkt): implement caption block classifier 2026-05-24 01:56:34 -04:00
tests feat(pdftract-315s): implement WER CI gate and OCR CLI flags 2026-05-24 02:07:27 -04:00
__test__.pdf feat(pdftract-15pz8): implement multi-process safe cache operations 2026-05-23 05:31:11 -04:00
build.rs feat(pdftract-43ry): implement predefined CMap registry 2026-05-23 23:00:59 -04:00
Cargo.toml feat(pdftract-core): add SSRF protection (TH-05) and URL_PRIVATE_NETWORK diagnostic 2026-05-24 01:50:12 -04:00
pdftract-core.cdx.json feat(pdftract-67tm8): implement MCP stdio transport with integration tests 2026-05-23 00:16:42 -04:00
README.md docs(pdftract-49f8): establish Cargo.lock policy and documentation 2026-05-20 18:13:14 -04:00

pdftract-core

The core Rust library for PDF text extraction. This crate provides the parsing, layout analysis, font encoding recovery, and text extraction primitives used by the CLI (pdftract-cli) and Python bindings (pdftract-py).

Cargo.lock Policy

This workspace checks in Cargo.lock at the repository root. This is unconventional for library crates—the Cargo Book historically suggested that only binary crates should check in lockfiles, allowing library consumers to resolve their own dependency versions.

pdftract departs from this convention for release reproducibility:

  1. SLSA Level 3 provenance requires that every milestone tag produces byte-identical artifacts across builds. Without a checked-in lockfile, two runs of cargo build on the same commit can resolve different transitive dependency versions, producing different binary hashes.

  2. Multi-output artifacts—this workspace produces Rust crates (pdftract-core, pdftract-cli), Python wheels (pdftract-py), and Docker images. All must be built from the same dependency tree.

  3. Supply-chain security—the lockfile pins checksums for all transitive dependencies, enabling cargo audit to detect yanked or compromised crates.

  4. Downstream consumers can still ignore the lockfile if needed. Cargo allows cargo build --frozen with a local lockfile override, or consumers can vendor the crate with their own dependency resolution.

The tradeoff—occasional merge conflicts when PRs update overlapping dependencies—is worth the guarantee of reproducible releases. See CONTRIBUTING.md for the lockfile-update workflow.

Modules

  • parser: PDF spec parsing (xref, trailer, object streams, indirect references)
  • font: Font encoding recovery, glyph name lookup, fingerprinting
  • layout: Page layout analysis, region segmentation, reading order
  • extract: Text extraction with provenance (bounding boxes, confidence scores)
  • ocr: Tesseract integration for raster pages

Usage

use pdftract_core::{extract_text, ExtractOptions};

let options = ExtractOptions::default();
let result = extract_text("document.pdf", &options)?;
println!("{}", result.text);