Implement the deskew preprocessing step using leptonica's
pixFindSkewAndDeskew (Hough line transform). The function:
- Detects dominant text angle on grayscale input
- Rotates by negative angle if >= 0.3 deg threshold
- Returns input unchanged for negligible skews (< 0.3 deg)
- Emits IMG_DESKEW_OUT_OF_RANGE diagnostic for angles > 15 deg
- Returns detected angle for quality tracking
Changes:
- Add leptonica-plumbing dependency (ocr feature)
- Create preprocess.rs module with deskew() function
- Add ImgDeskewOutOfRange diagnostic code
- Expose preprocess module in lib.rs
The implementation uses pixFindSkewAndDeskew which both detects
the skew angle and performs deskewing in one call, returning
the detected angle for debugging purposes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add build.rs that generates compile-time std14 metrics from JSON
- Add std14.rs module with Std14Metrics struct and get_std14_metrics()
- Add build/std14-metrics.json with AFM-derived widths for all 14 fonts
- Re-export Std14Metrics, NamedEncoding, get_std14_metrics in lib.rs
Acceptance criteria:
- All 14 Standard fonts (Courier, Helvetica, Times, Symbol, ZapfDingbats
and their variants) return valid metrics from the registry
- Subset-prefixed names (ABCDEF+Helvetica) resolve via strip_subset_prefix()
- Width tables match Adobe AFM data within rounding tolerance
- Binary footprint < 60 KB (generated source: 20 KB, actual data ~8 KB)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement FontKind enum and classify_font() function for Phase 2.1
font type detection. Includes strip_subset_prefix() for handling
font subset names (e.g., ABCDEF+Times-Roman).
FontKind variants:
- Type1, Type1Std14 (Standard 14)
- TrueType, OpenTypeCFF
- Type0, CIDFontType0, CIDFontType2
- Type3
Classifier reads /Subtype, /BaseFont, and for Type0 fonts, descendant
CIDFont subtype. OpenTypeCFF detected via /FontDescriptor /FontFile3
with /Subtype /OpenType.
All 27 font tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add decode_page_content_streams() function for per-page lazy decode
- Update extract_page_from_dict() to support lazy stream decoding
- Modify extract_pdf() and extract_pdf_ndjson() to enable lazy decoding
- Fix borrow checker issue in LazyPageIter::next()
This ensures content streams are decoded lazily per page and dropped
immediately after processing, keeping peak RSS flat across page count.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fix extract_page_inner typo: changed to extract_page (function was undefined)
- Add error_count field to ExtractionMetadata struct
- Add error field to PageResult struct (missing in constructor)
- Add semaphore module to lib.rs exports
The parallelism capping implementation was already in place but had bugs
preventing compilation. This fixes those bugs so the semaphore-based
bounding of in-flight pages works correctly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements Phase 6.9.1: the two-byte-prefix directory scheme that keeps
any single directory under 65K entries even at millions of cached entries.
Changes:
- Add zstd dependency to Cargo.toml
- Create cache module with layout.rs implementing path construction
- Add CacheIndex struct for index.json metadata (schema version, timestamps)
- Implement entry_path(), fingerprint_dir(), parse helpers
- Add load_index()/save_index() for cache metadata persistence
- Ensure mkdir -p semantics with ensure_fingerprint_dir()
- 18 tests covering all acceptance criteria
Acceptance criteria verified:
✓ entry_path produces correct two-level prefix layout
✓ Different opts_hashes for same fingerprint share fp_dir
✓ Different fingerprints with same prefix share first-level dir
✓ index.json round-trips with schema version check
✓ Future schema version rejects cache with clear error
✓ mkdir -p creates prefix dirs; idempotent on concurrent writes
✓ Unicode-correct path handling via std::path::PathBuf
✓ Path length stays under 4096 bytes
Co-Authored-By: Claude Code <noreply@anthropic.com>
Implement the --receipts CLI flag accepting "off" | "lite" | "svg" with default "off".
Thread the ExtractionOptions.receipts field through the extraction pipeline so that
receipts are generated for spans and blocks based on the selected mode.
Changes:
- CLI: Added --receipts flag with clap value_parser for runtime validation
- CLI: Added feature check for SVG mode (requires 'receipts' feature)
- MCP tools: Added receipts field to ExtractArgs, ExtractTextArgs, ExtractMarkdownArgs
- MCP tools: Added build_extraction_options() to parse receipts mode
- Core: Added extract.rs module with extract_pdf(), extract_page(), generate_receipt()
- Core: Added ExtractionOptions with ReceiptsMode enum (Off/Lite/SvgClip)
- Core: Added receipts feature flag to Cargo.toml
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add value_parser = ["off", "lite", "svg"] to --receipts CLI flag for clap validation
- Add receipts field to ExtractTextArgs and ExtractMarkdownArgs in MCP tools args
- Add ExtractionOptions and ReceiptsMode to pdftract-core (options.rs module)
- Expose options module in pdftract-core/lib.rs
The CLI now validates receipts mode at parse time with helpful error messages.
MCP tools accept receipts argument matching the schema defined in sibling 6.7.5.
ExtractionOptions struct provides the threading mechanism for the extraction pipeline.
Acceptance criteria:
- PASS: CLI validates --receipts values (off/lite/svg only)
- PASS: CLI shows proper help text with possible values
- PASS: ExtractionOptions serializes for HTTP/MCP transport
- PASS: MCP tools args have receipts field
- WARN: Full extraction implementation pending (deferred to extraction beads)
Closes pdftract-39g4j
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement the Receipt struct and lite-mode JSON serialization for
visual citation receipts. This provides cryptographic proof of
provenance for extracted text.
Changes:
- Add Receipt struct with 6 fields (pdf_fingerprint, page_index,
bbox, content_hash, extraction_version, svg_clip)
- Implement Receipt::lite() constructor with NFC normalization
- Integrate Receipt into SpanJson and BlockJson schemas
- Add unicode-normalization and serde_json dependencies
Acceptance criteria:
- Receipt::lite() produces valid receipts with svg_clip=None
- Lite mode JSON omits svg_clip key via skip_serializing_if
- Content hash uses NFC normalization for cross-platform stability
- Receipt wired into SpanJson and BlockJson types
Note: 100 receipts aggregate size is ~27 KB (not 15 KB as planned).
The 15 KB target is not achievable with required field sizes.
Refs: pdftract-5zm86, Phase 6.8 Visual Citation Receipts (lines 2351-2417)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Make diagnostics module visible to fingerprint module and fix
hash_page_geometry signature to match usage.
Changes:
- Add `pub mod diagnostics;` to lib.rs for module visibility
- Modify hash_page_geometry to create diagnostics internally
The canonicalize module already has complete implementation:
- canonicalize_f64: banker's rounding to 4dp for geometry
- normalize_content_stream: whitespace normalization via lexer
- serialize_dict_canonical: sorted-key dict serialization
- hash_resource_dict_canonical: order-independent resource hashing
Verification: notes/pdftract-154mz.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement Merkle SHA-256 fingerprint algorithm for PDF structural
fingerprinting as specified in Phase 1.7 of the plan.
Components:
- FingerprintInput struct with page data and catalog flags
- Per-page hashing: content streams (normalized), resources (sorted),
geometry (4dp banker's rounding)
- Structure tree hash for tagged PDFs
- Catalog feature flag byte (encryption, JS, XFA, OCG)
Acceptance criteria:
- INV-3: 100% reproducible fingerprints (test passes)
- INV-13: Output format ^pdftract-v1:[0-9a-f]{64}$ (test passes)
- Performance: 100-page PDF in < 1ms (test passes)
- KU-7: WARN - no linearized fixtures available
Closes pdftract-q15sh
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement the document catalog parser (/Root traversal) for PDF documents.
The catalog parser extracts all key entries from the document catalog
including Pages, Outlines, MarkInfo, StructTreeRoot, AcroForm, Names,
Metadata, PageLabels, OCProperties, OpenAction, AA, and Version.
Key structures:
- MarkInfo: parses /MarkInfo dictionary with is_tagged, user_properties, suspects
- PageLabelStyle: enum for all label styles (D, R, r, A, a)
- PageLabel: single page label with style, prefix, and start value
- PageLabelsTree: number tree parser for /PageLabels with /Nums and /Kids support
- OcProperties: stub for OCG implementation (delegated to dedicated bead)
- Catalog: main catalog struct with all required and optional fields
Number tree implementation:
- Parses /Nums arrays (leaf nodes with alternating key-value pairs)
- Supports /Kids arrays (internal nodes for recursive tree traversal)
- Provides get_label_with_start() and get_label() methods for lookup
- Correctly formats roman numerals (uppercase/lowercase) and letter sequences
All 27 tests pass including proptests for fuzzing robustness (INV-8).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
