Commit graph

9 commits

Author SHA1 Message Date
jedarden
d84f8da3a4 feat(pdftract-5qj50): implement mojibake detection and repair via encoding_rs
Implements Phase 4.7 Correction Pipeline step 3: mojibake detection
and repair for Latin-1 bytes misinterpreted as UTF-8.

Changes:
- Add layout::correction module with detect_and_repair_mojibake function
- Implement CorrectableText trait for mutable text access
- Add trait implementations for hybrid::Span and schema::SpanJson
- Make encoding_rs a non-optional dependency (was cjk-gated)
- Detection heuristic: 2+ occurrences of telltale sequences (é, è, ’, etc.)
- Re-decode via encoding_rs::WINDOWS_1252 when detected
- Accept repair only if readability score improves by >0.05 epsilon
- Fast-path pass-through for ASCII-only and clean UTF-8 text

Closes: pdftract-5qj50

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 17:01:53 -04:00
jedarden
84b4448648 feat(pdftract-5qca): implement form_fields JSON output + schema integration
Phase 7.4.5 implementation: Wire combined Vec<(String, FormFieldValue)> from
combiner into document-level /form_fields JSON output with tagged union schema.

- Add FormFieldJson, FormFieldTypeJson, FormFieldValueJson, ChoiceValueJson to schema
- Add form_fields: Vec<FormFieldJson> to ExtractionResult (always emitted, empty when none)
- Implement acro_field_to_value() converter for Phase 7.4.2 type-specific extraction
- Wire form field extraction in extract_pdf(): walk AcroForm, extract XFA, combine with XFA-wins
- Add convert_form_field_to_json() helper for FormFieldValue → FormFieldJson conversion
- Update docs/schema/v1.0/pdftract.schema.json with form_fields $defs and required field
- Add form_fields_to_markdown() to markdown module for Form Fields footer table

Schema shape: /form_fields is array of {name, type, value, default?, page_index?, rect?,
required, read_only, multiline?, max_length?, options?, multi_select?, selected?,
state_name?, pushbutton?, radio?}. Type field is tagged enum: "text", "button", "choice",
"signature". Value field varies by type (string|boolean|string|array|uint|null).

Closes: pdftract-5qca

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 14:36:03 -04:00
jedarden
a639794133 feat(pdftract-29gu): implement Phase 5.5.3 region-level confidence policy
- Add OcrFallback variant to SpanSource enum for fallback spans
- Add page_seg_mode field to TessOpts for PSM_SPARSE_TEXT support
- Add ASSISTED_OCR_KEEP_THRESH (0.7) and ASSISTED_OCR_FALLBACK_THRESH (0.3) constants
- Implement apply_region_level_confidence_policy() for region-level decision making
- Group words by baseline proximity (12pt tolerance) for region computation
- Add TODO for Phase 6.1 confidence_source enum to include "ocr-fallback"

Closes: pdftract-29gu
2026-05-24 05:15:46 -04:00
jedarden
67b3fde4d6 feat(pdftract-j6yd): implement signatures array output + validation_status enum + schema integration
Add document-level /signatures array output per Phase 7.3 of the plan.

Changes:
- Add SignatureJson struct to schema module with all signature metadata fields
- Update ExtractionResult to include signatures: Vec<SignatureJson>
- Integrate signature extraction into extract_pdf() pipeline
- Update result_to_json() to include signatures in JSON output
- Update JSON schema with signatures array and SignatureJson definition
- Add markdown sink signatures footer when signatures are present
- Add comprehensive tests for signature JSON serialization and validation

Acceptance criteria:
- Schema tests: 5/5 signature JSON tests pass
- Markdown sink emits Signatures footer when count > 0
- PyO3 binding automatically handles Vec<SignatureJson> via serde
- docs/schema/v1.0/pdftract.schema.json updated with signatures shape

Verification note: notes/pdftract-j6yd.md

Closes: pdftract-j6yd
2026-05-24 04:05:34 -04:00
jedarden
99709354f5 feat(pdftract-oh30a): implement per-page readability aggregation
Implement char-weighted median aggregation of per-span readability
scores into a page-level score stored in extraction_quality.readability.

Algorithm:
- Collect (score, char_count) pairs from spans
- Sort by score ascending
- Walk sorted list accumulating character counts
- Return score at half-total-char position

Acceptance criteria:
- Single span: returns its score
- Multiple spans: char-weighted median (longer spans count more)
- Empty page: returns 0.0
- All-perfect: returns 1.0

Closes: pdftract-oh30a

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 03:28:41 -04:00
jedarden
92e90af0b0 feat(pdftract-zy2jx): generate JSON Schema from Rust output types
- Add schemars dependency to pdftract-core (v1.2)
- Add JsonSchema derives to output types (ExtractionResult, PageResult, ExtractionMetadata, SpanJson, BlockJson, CellJson, RowJson, TableJson, ExtractionQuality, Receipt, ReceiptsMode)
- Create xtask/src/bin/gen_schema.rs for schema generation
- Add gen-schema command to xtask main.rs
- Generate docs/schema/v1.0/pdftract.schema.json using Draft 2020-12

Schema includes:
- $schema: "https://json-schema.org/draft/2020-12/schema"
- $defs with all output type definitions
- Proper type annotations for all fields

Closes: pdftract-zy2jx
2026-05-24 01:29:14 -04:00
jedarden
ba551b04d1 feat(pdftract-5mph): implement table block + table JSON output schema integration
- Fix table block bbox to use actual grid bbox instead of placeholder
- Add schema validation tests for tables array emission
- Verify two-page table detection integration

Files modified:
- crates/pdftract-core/src/extract.rs: Use grid bbox for table blocks
- crates/pdftract-core/src/schema/mod.rs: Add tests for tables array emission

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 00:49:01 -04:00
jedarden
e3a149fbf8 feat(pdftract-sg6): implement DPI selection logic for OCR rendering
Implement Phase 5.2.3 DPI selection that picks per-page DPI based on
image filter signals (JBIG2 detection) and font size signals from Phase 4.

- Add select_dpi() function implementing the DPI selection table:
  * JBIG2Decode filter present -> 200 DPI (already binary)
  * Median font_size < 7.0 pt -> 400 DPI (fine print)
  * Median font_size >= 7.0 pt -> 300 DPI (standard)
  * Default -> 300 DPI for scanned pages
- Add Pdf1Filter enum for PDF 1.x filter name parsing
- Add FontSizeSpan struct for Phase 4 font size data
- Add ocr_dpi_override option to ExtractionOptions
- Export ExtractionQuality from schema module for DPI tracking
- Add comprehensive unit tests (19 tests, all passing)

Acceptance criteria:
- Unit tests: each branch tested with synthetic inputs
- Integration: legal-document -> 400 DPI, textbook -> 300 DPI, JBIG2 -> 200 DPI
- DPI override option works correctly
- extraction_quality.dpi_used schema field ready

Co-Authored-By: Claude Code <claude-code@anthropic.com>
2026-05-23 17:37:40 -04:00
jedarden
9f18c6cb9c feat(pdftract-5zm86): implement Receipt struct + lite-mode serialization
Implement the Receipt struct and lite-mode JSON serialization for
visual citation receipts. This provides cryptographic proof of
provenance for extracted text.

Changes:
- Add Receipt struct with 6 fields (pdf_fingerprint, page_index,
  bbox, content_hash, extraction_version, svg_clip)
- Implement Receipt::lite() constructor with NFC normalization
- Integrate Receipt into SpanJson and BlockJson schemas
- Add unicode-normalization and serde_json dependencies

Acceptance criteria:
- Receipt::lite() produces valid receipts with svg_clip=None
- Lite mode JSON omits svg_clip key via skip_serializing_if
- Content hash uses NFC normalization for cross-platform stability
- Receipt wired into SpanJson and BlockJson types

Note: 100 receipts aggregate size is ~27 KB (not 15 KB as planned).
The 15 KB target is not achievable with required field sizes.

Refs: pdftract-5zm86, Phase 6.8 Visual Citation Receipts (lines 2351-2417)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 03:30:24 -04:00