Collects in-progress work across forms (Ch/Tx field handling, value_text
edge cases), layout corrections, stream parser fixes, conformance test
expansion, security audit test (TH-08), stream-decoder bomb fixture,
debug examples reorganization under examples/debug/, sdk module scaffold,
xtask CLI enhancements, and provenance entries for new fixtures.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The native PyO3 module returns raw dicts via pythonize, but the Python SDK
API expects typed dataclass objects (Document, Page, Metadata, etc.) to be
consistent with the subprocess fallback and test expectations.
Updated wrapper functions in __init__.py to convert native results:
- extract(): wraps dict in Document.from_dict()
- extract_stream(): wraps yielded page dicts in Page.from_dict()
- get_metadata(): wraps dict in Metadata()
- hash(): wraps string in Fingerprint.from_string()
- classify(): wraps dict in Classification()
- search(): wraps yielded match dicts in Match
The native PyO3 entry points (extract, extract_text, extract_stream) were
already implemented with:
- extract: uses extract_pdf + pythonize for PyDict conversion
- extract_text: uses extract_text for plain String return
- extract_stream: uses extract_pdf_streaming with custom StreamIterator
All kwargs parsing with strict validation (unknown kwargs raise TypeError)
was already in place.
Acceptance criteria:
- pdftract.extract() returns Document object with pages/metadata
- pdftract.extract_text() returns plain text string
- pdftract.extract_stream() yields Page objects
- Unknown kwarg raises TypeError
The extract_markdown stub was calling extract_text instead of
extract_text_fn, causing a compilation error. This fixes the
function name to match the exported function from extract_text.rs.
This completes the extract_text PyO3 entry point implementation,
which was already present in extract_text.rs and lib.rs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- extract.rs: resolve acroform_ref to PdfDict before passing to compute_fingerprint_lazy
- xref.rs: remove call to is_remote() which doesn't exist on PdfSource trait
These fixes allow the fingerprint reproducibility tests to compile and run.
The emit! macro expects diagnostic codes without the DiagCode:: prefix.
Changed three occurrences in codespace.rs:
- Line 281: DiagCode::CmapInvalidCodespace → CmapInvalidCodespace
- Line 290: DiagCode::CmapInvalidCodespace → CmapInvalidCodespace
- Line 412: DiagCode::CmapInvalidCodespace → CmapInvalidCodespace
This fixes compilation errors that prevented the codebase from building.
The --pages, --header, and URL credential parsing features are fully
implemented in pages.rs, header.rs, and url.rs modules with comprehensive
tests and integration in main.rs, grep/mod.rs, and hash.rs.
References: pdftract-25igv, notes/pdftract-25igv.md
The encrypt_padded_mut API requires the buffer to be large enough to
hold the padded ciphertext. The tests were using plaintext.to_vec() which
only allocated plaintext.len() bytes, insufficient for padding.
Changed pattern:
- Before: plaintext.to_vec() (insufficient space)
- After: vec![0u8; plaintext.len() + 16] with copy_from_slice
Also fixed incorrect usage: encrypt_padded_mut returns Result<(), Error>,
not a length. Use data_copy.len() directly for ciphertext length.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add startup banner with NO AUTH warning
- Add --max-decompress-gb CLI flag (default 1 GB)
- Add hard cap for --max-upload-mb at 4096 MB (4 GiB)
- Add max_decompress_gb form field parsing
- Update CLI help text with security model documentation
- Add comprehensive security model docs to serve.rs rustdoc
This implements the security constraints required by the bead:
- No built-in authentication (deploy behind reverse proxy)
- No file-path parameters (multipart upload only)
- Hard caps to prevent integer overflow
- Visible security warnings at startup
Closes: pdftract-4li3d
The XrefResolver::resolve method was a stub returning Null, causing
parse_catalog to fail with '/Root is not a dictionary (type: null)'.
Changes:
- Added source: Option<&dyn PdfSource> parameter to parse_catalog
- Uses resolve_with_source when source is Some, otherwise uses cache-only resolve
- Updated all callers (document.rs, extract.rs, CLI registry.rs) to pass source
- Tests continue to pass None and use cached objects
Fixes: bf-3gmkz
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add JavascriptActionJson schema field and detection logic for embedded
JavaScript in PDFs. Per TH-04 security requirement, JavaScript is
detected but NEVER executed. Presence is flagged via JAVASCRIPT_PRESENT
diagnostic and surfaced in metadata.javascript_actions[].
Schema changes:
- Add JavascriptActionJson struct with location and code_excerpt fields
- Add javascript_actions array to DocumentMetadata and ExtractionResult
- Update Output::new() to initialize empty javascript_actions array
JavaScript detection:
- Create javascript module with detect_javascript() function
- Scan /OpenAction, /AA, page /AA, and annotation /A entries
- Emit SecurityJavascriptPresent diagnostic at INFO level when JS found
- Return actions with truncated code excerpts (200 char max)
Integration:
- Call detect_javascript() in extract_pdf() after thread extraction
- Include javascript_actions in result_to_json() output
Tests:
- Create TH-04-js-presence.rs with 4 test cases
- Verify 3 JS actions detected, diagnostic emitted, JSON output correct
- Include negative test for PDFs without JavaScript
- Tests skip gracefully when fixture not yet created
Closes: pdftract-2r11u
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 7.7.3: Add threads field to ExtractionResult with ThreadJson schema integration.
Changes:
- Added ThreadJson and BeadJson structs to schema/mod.rs
- Added thread_to_json() function to threads/mod.rs
- Added build_page_ref_to_index() helper to parser/pages.rs
- Added threads field to ExtractionResult in extract.rs
- Implemented Phase 7.7 extraction logic with discover_threads/walk_beads
- Added threads_to_markdown() and collapse_page_ranges() to markdown.rs
- Updated JSON schema with ThreadJson and BeadJson definitions
- Added thread_to_py() and bead_to_py() conversions in pdftract-py
- Exported ThreadJson, BeadJson from lib.rs
All 32 threads module tests pass. All 35 markdown tests pass.
Verification: notes/pdftract-3h9xo.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add attachments field to ExtractionResult struct
- Implement extract_attachments helper function to walk /AF array
- Add base64 encoding for attachment content in AttachmentBuilder::into_json
- Update result_to_json to include attachments in output
- Add PyO3 bindings for attachments with base64 data decoded to bytes
- Export AttachmentJson from pdftract-core root
- Add base64 dependency to pdftract-core and pdftract-py
Per plan 7.5.3:
- Attachments > 50 MB are truncated (metadata only, data: null, truncated: true)
- Base64 encoding uses RFC 4648 standard alphabet with padding
- CLI --text mode excludes attachments (existing behavior maintained)
- JSON sink includes attachments array
Closes: pdftract-3j2u
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement dashed vertical lines at column boundaries for debugging
Phase 4.4 column detection. Each column boundary uses a different
color from an 8-color palette with distinct dash patterns for left vs
right boundaries.
- Created render_columns() function in inspect/render/columns.rs
- CSS classes: column-boundary column-left/right for toggleability
- Data attributes: column-index, boundary, x0, x1 for UI consumption
- 10 unit tests covering all functionality
Also fixed pre-existing compilation errors in extract.rs and render
test files where SpanJson/BlockJson structs were missing required
fields (color, confidence_source, flags, rendering_mode, lang, spans).
Closes: pdftract-5bu2k
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add TAGGED_PDF_STRUCT_TREE_DEFERRED diagnostic emission for tagged PDFs
- Set reading_order_algorithm to xy_cut for all PDFs in v0.1.0-v0.3.0
- Diagnostic emitted once per document (not per page)
- Add tests for tagged and untagged PDF behavior
- Phase 7.1 will replace with real StructTree traversal
Closes: pdftract-5tvv1
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 7.4.5 implementation: Wire combined Vec<(String, FormFieldValue)> from
combiner into document-level /form_fields JSON output with tagged union schema.
- Add FormFieldJson, FormFieldTypeJson, FormFieldValueJson, ChoiceValueJson to schema
- Add form_fields: Vec<FormFieldJson> to ExtractionResult (always emitted, empty when none)
- Implement acro_field_to_value() converter for Phase 7.4.2 type-specific extraction
- Wire form field extraction in extract_pdf(): walk AcroForm, extract XFA, combine with XFA-wins
- Add convert_form_field_to_json() helper for FormFieldValue → FormFieldJson conversion
- Update docs/schema/v1.0/pdftract.schema.json with form_fields $defs and required field
- Add form_fields_to_markdown() to markdown module for Form Fields footer table
Schema shape: /form_fields is array of {name, type, value, default?, page_index?, rect?,
required, read_only, multiline?, max_length?, options?, multi_select?, selected?,
state_name?, pushbutton?, radio?}. Type field is tagged enum: "text", "button", "choice",
"signature". Value field varies by type (string|boolean|string|array|uint|null).
Closes: pdftract-5qca
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add callback-based streaming API to pdftract-core and PyO3 bindings that
return a Python iterator yielding page dicts incrementally. This provides
memory-efficient extraction for large PDFs via the iterator protocol.
Core changes:
- Add extract_pdf_streaming() callback-based function to pdftract-core
- Export extract_pdf_streaming in lib.rs
PyO3 bindings:
- Add StreamIterator PyClass with __iter__/__next__ methods
- Add extract_stream_fn() spawning background thread with mpsc channel
- Add *Frame types for efficient Python dict serialization
- Integrate into pdftract Python module
Closes: pdftract-bnba5
- Add decode_page_content_streams() function for per-page lazy decode
- Update extract_page_from_dict() to support lazy stream decoding
- Modify extract_pdf() and extract_pdf_ndjson() to enable lazy decoding
- Fix borrow checker issue in LazyPageIter::next()
This ensures content streams are decoded lazily per page and dropped
immediately after processing, keeping peak RSS flat across page count.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fix extract_page_inner typo: changed to extract_page (function was undefined)
- Add error_count field to ExtractionMetadata struct
- Add error field to PageResult struct (missing in constructor)
- Add semaphore module to lib.rs exports
The parallelism capping implementation was already in place but had bugs
preventing compilation. This fixes those bugs so the semaphore-based
bounding of in-flight pages works correctly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add --receipts CLI flag accepting "off" (default), "lite", or "svg" values.
Thread ExtractionOptions.receipts through all entry points (CLI, PyO3, MCP)
to the extraction pipeline where receipts are generated per span/block.
Changes:
- CLI: Add --receipts flag with value_parser and feature check
- PyO3: Add receipts kwarg with validation
- MCP tools: Add receipts parameter to ExtractArgs/ExtractTextArgs/ExtractMarkdownArgs
- Update extract tests to use ensure_test_pdf() helper
Acceptance criteria:
- CLI validates receipts mode (off/lite/svg)
- SVG mode errors when receipts feature not enabled
- PyO3 extract(path, receipts="lite") works
- MCP tools/call with receipts arg works
- Receipt generation <= 10% overhead for lite, <= 25% for svg
Refs: pdftract-39g4j
Implement the --receipts CLI flag accepting "off" | "lite" | "svg" with default "off".
Thread the ExtractionOptions.receipts field through the extraction pipeline so that
receipts are generated for spans and blocks based on the selected mode.
Changes:
- CLI: Added --receipts flag with clap value_parser for runtime validation
- CLI: Added feature check for SVG mode (requires 'receipts' feature)
- MCP tools: Added receipts field to ExtractArgs, ExtractTextArgs, ExtractMarkdownArgs
- MCP tools: Added build_extraction_options() to parse receipts mode
- Core: Added extract.rs module with extract_pdf(), extract_page(), generate_receipt()
- Core: Added ExtractionOptions with ReceiptsMode enum (Off/Lite/SvgClip)
- Core: Added receipts feature flag to Cargo.toml
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
