Names the legacy .github/workflows/schema-gen.yml as inert/disabled,
lists the three Argo WorkflowTemplates, and adds a manual trigger snippet.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add call-site diagnostic emission for DCTDecode SOI/EOI marker validation.
Previously, DCTDecoder.validate_markers() created diagnostics but they were
dropped because StreamDecoder trait doesn't support returning them. Now
diagnostics are emitted in decode_stream_impl() like JBIG2/JPX/CCITT.
Also include source module refactoring:
- Add PdfSource adapter trait for source::PdfSource compatibility
- Feature-gate http_range module with `remote` feature
- Update document.rs to use new source traits
Acceptance criteria:
- DCTDecode emits STREAM_INVALID_JPEG for missing SOI/EOI markers
- JBIG2Decode emits OCR_JBIG2_UNSUPPORTED when full-render disabled
- JPXDecode emits OCR_JPX_UNSUPPORTED and validates JP2 magic
- CCITTFaxDecode emits OCR_CCITT_UNSUPPORTED when no libtiff
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bead-Id: pdftract-4xmp6
Bead-Id: pdftract-57np8
Bead-Id: pdftract-3954u
The --header CLI flag implementation was already complete in the codebase.
This note documents the implementation and verifies all acceptance criteria.
Acceptance criteria verified:
- Single header with URL: PASS
- Multiple headers: PASS
- Managed header rejection: PASS
- CRLF injection protection: PASS
- No colon error: PASS
- Local file silent ignore: PASS
No new code was required - the feature was already fully implemented
in main.rs, header.rs, source/mod.rs, and http_range.rs.
Documents the implementation, acceptance criteria status, and design
decisions for the CMap codespace range parser.
Co-Authored-By: Claude Code <noreply@anthropic.com>
- Remove unused jpx::JpxDecoder import from stream.rs (code uses fully qualified paths)
- Add notes/pdftract-36glh.md with acceptance criteria verification
The JPXDecode passthrough implementation was already complete in commit 4ba4687.
This change is minor cleanup only.
References: pdftract-36glh
This commit fixes a compilation error in the javascript tests that were
using PageDict::default(). The JBIG2 decoder module was already fully
implemented; this change only enables the tests to compile and run.
Changes:
- Add Default impl for PageDict in parser/pages.rs
- Verify all 11 JBIG2-related tests pass
The JBIG2Decode passthrough filter implementation is complete:
- Passthrough of raw JBIG2 bytes
- /JBIG2Globals reference recording for downstream consumers
- OCR_JBIG2_UNSUPPORTED diagnostic emission when full-render disabled
Co-Authored-By: Claude Code <noreply@anthropic.com>
- Made map_error_to_exit_code() function public in hash.rs so it can be
called from main.rs
- Added test file test_hash_exit_codes.rs to verify exit code behavior
- Updated verification note with current implementation status
The hash subcommand was already implemented but map_error_to_exit_code
was private, causing a compilation error. This fix resolves the issue.
Related: pdftract-3954u
Implement O'Gorman 1993 Docstrum algorithm for reading order detection
on irregular layouts (magazines with sidebars) where XY-cut produces
fragmented regions.
Implementation:
- k=5 nearest neighbors per block (Docstrum standard)
- Euclidean center-to-center distance in PDF user space
- Angle constraints: ±30° from horizontal (within-line) and vertical (between-line)
- Root detection: nodes with no incoming edges from blocks above
- Root sorting by (column ASC, y DESC)
- DFS traversal per component in y-then-x order
Acceptance criteria PASS:
- Magazine main+sidebar: 2 components; main first, sidebar second
- Pathological scattered: each a root, visited (column, y desc)
- All-one-line horizontal: 1 component, left-to-right
- All-one-column vertical: 1 component, top-to-bottom
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The detect_xfa function was already implemented in the codebase at the
time of bead assignment. This note documents the verification of the
existing implementation against the bead's acceptance criteria.
All 6 tests pass, covering all acceptance criteria:
- XFA stream presence → true
- XFA array packet form → true
- No XFA key → false
- XFA null → false
- No AcroForm → false
- XFA as indirect reference → true
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Verification confirms the CLI parsing and validation for multi-format
output flags is already fully implemented in crates/pdftract-cli/src/output.rs.
All acceptance criteria verified:
- Duplicate format rejection ✓
- NDJSON exclusivity ✓
- At most one stdout ✓
- Auto-naming with --format + -o ✓
No code changes required.
Update the verification note for pdftract-2qw5j to clarify that the
bead's "Critical considerations" enum values differ from the actual
implementation:
- confidence_source: bead lists ["vector", "ocr", ...] but plan/Rust
code uses ["native", "heuristic", "ocr"] (per plan line 363)
- severity: bead omits "fatal" but Rust code includes it for
extraction-aborting conditions
The schema generation system is complete and correct per the plan
specification. The bead requirements appear to be from an earlier
spec version and are superseded by the plan.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add explicit enum constraints to page_type, severity, and confidence_source
fields in the generated JSON Schema for better validation.
Changes:
- Modified xtask/src/bin/gen_schema.rs to add explicit enum constraints
during schema generation via add_enum_constraints() function
- page_type enum: ["text", "scanned", "mixed", "broken_vector", "blank", "figure_only"]
- severity enum: ["info", "warning", "error", "fatal"]
- confidence_source enum: ["native", "heuristic", "ocr"]
- Regenerated docs/schema/v1.0/pdftract.schema.json with enum constraints
- Added .github/workflows/schema-gen.yml CI workflow for schema validation
The CI workflow validates:
1. Generated schema matches committed file (fails on diff)
2. JSON syntax is valid
3. Schema structure is correct ($id, $schema, title, $defs)
4. Enum constraints are present and have correct values
This ensures schema changes are reviewable in PRs and forces
developers to commit the updated schema when type definitions change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- test_open_valid_file: byte string is 22 bytes, not 20
- test_seek_from_end: seeking -2 from end of "Hello" gives "lo", not "el"
The MmapSource implementation was already complete with all acceptance
criteria met:
- open() returns Ok/Err appropriately
- read_range() with bounds checking
- len() matches file size
- Read+Seek trait implementations
- Send + Sync for concurrent access
- MADV_SEQUENTIAL via advise_sequential()
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The schema now reflects the latest doc comments from the Rust types,
including updated descriptions for annotations and other fields.
Changes:
- AnnotationJson description updates (phase 7.6.4 reference)
- Format consistency updates (float vs double)
- Subtype-specific field documentation
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add schema-gen step to quality-matrix that regenerates
docs/schema/v1.0/pdftract.schema.json and compares to committed file.
Fails build on any diff with actionable error message.
Bead: pdftract-16h0a (Phase 6.1.3)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add std::sync::Arc import for thread sharing
- Fix lifetime issue in test_sync_multiple_threads using Arc
- Add mut to source in test_empty_file for Read trait
All FileSource tests pass (12/12).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The --pages RANGE CLI flag implementation was already complete in the
codebase. All required functionality was present including:
- Range parser in pages.rs with comprehensive tests
- CLI integration in main.rs
- HTTP serve support in serve.rs
- MCP tools integration
- PyO3 bindings in pdftract-py
All acceptance criteria verified PASS.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement FileSource as a PdfSource fallback for when memory-mapping
is not available or desired. Uses parking_lot::Mutex<File> for
thread-safe concurrent access across rayon workers.
Changes:
- Add parking_lot = "0.12" dependency to pdftract-core/Cargo.toml
- Rewrite FileSource to use Mutex<File> for Send + Sync support
- Implement PdfSource, Read, and Seek traits
- Add 12 comprehensive tests including concurrent read tests
All tests pass. Thread-safe concurrent access verified via
test_sync_multiple_threads and test_concurrent_read_range.
Co-Authored-By: Claude Code (claude-opus-4.7) <noreply@anthropic.com>
Bead-Id: pdftract-5ik66
- Implemented aes_128_decrypt with CBC mode + PKCS#7 padding
- Implemented derive_aes_128_object_key with 'sAlT' suffix
- Implemented is_identity_filter for crypt filter handling
- All 11 unit tests passing
- Integration work deferred to coordinator bead pdftract-1z0qt
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Define the PdfSource trait abstraction over PDF byte sources. This trait
provides a uniform API for reading PDF data from different sources:
local files (MmapSource, FileSource), and eventually remote HTTPS PDFs.
Trait features:
- Read + Seek + Send + Sync supertrait bounds for rayon page-parallelism
- len() returns total source length
- read_range() returns Bytes for zero-copy slicing
- prefetch() with no-op default (MmapSource overrides for MADV_SEQUENTIAL)
MmapSource:
- Memory-mapped file access via memmap2
- Applies MADV_SEQUENTIAL advice via prefetch()
- Zero-copy read_range() using Bytes::copy_from_slice()
- Fallback for platforms/filesystems where mmap fails
FileSource:
- Standard I/O implementation using std::fs::File
- Read+Seek delegation to underlying File
- read_range() uses try_clone() for thread-safe concurrent access
Re-exports from pdftract-core::source::PdfSource.
Verification note: notes/pdftract-1mmq9.md documents completion status.
Parser module migration to use new PdfSource is deferred to follow-up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixed test_aes_128_decrypt_roundtrip_with_valid_padding and two similar
tests to use the ciphertext slice returned by encrypt_padded_mut instead of
the entire buffer. The buffer is over-allocated to accommodate padding, but
only the returned slice contains valid ciphertext. Using the entire buffer
included trailing zeros that caused decryption to fail with invalid padding.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Blocker identified:
- Extraction pipeline (extract.rs) doesn't use Phase 4 layout pipeline
- Column detection functions never called in production
- SpanJson.column hardcoded to None (lines 1059, 1916)
- No end-to-end tests for acceptance criteria
Span struct HAS column field (line 179) but extraction doesn't use it.
Coordinator CANNOT CLOSE - sub-phase not end-to-end functional.
The encrypt_padded_mut API requires the buffer to be large enough to
hold the padded ciphertext. The tests were using plaintext.to_vec() which
only allocated plaintext.len() bytes, insufficient for padding.
Changed pattern:
- Before: plaintext.to_vec() (insufficient space)
- After: vec![0u8; plaintext.len() + 16] with copy_from_slice
Also fixed incorrect usage: encrypt_padded_mut returns Result<(), Error>,
not a length. Use data_copy.len() directly for ciphertext length.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This bead asked for implementation of BMC/BDC/EMC marked-content
operators and MarkedContentStack, but these were already fully
implemented in the codebase with comprehensive test coverage.
Verification note documents:
- MarkedContentStack in marked_content_stack.rs
- BMC/BDC/EMC parsers in marked_content_operators.rs
- Integration into execute_with_do in content_stream.rs
- All 6 acceptance criteria covered by passing tests
- 57 marked-content tests all passing
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixed compilation errors in Span constructors by adding missing `column: None` field.
Verified that the existing multi-output CLI parsing implementation meets all
acceptance criteria for bead pdftract-37qim.
Changes:
- crates/pdftract-core/src/span/mod.rs: Add column field to new() and empty() constructors
Verification:
- All 23 output::tests pass
- CLI parsing validated for duplicate format detection, ndjson exclusivity, stdout uniqueness
- Format auto-naming (--format with -o) works correctly
- Default behavior (no flags -> JSON to stdout) confirmed
See notes/pdftract-37qim.md for detailed verification results.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
All 4 children beads closed with verification:
- Line struct + baseline computation (pdftract-sdx9z)
- Baseline clustering algorithm (pdftract-6bwq4)
- Within-line span sorting (pdftract-1jkme)
- RTL direction detection (pdftract-1ofnz)
Acceptance criteria:
- ✅ All 4 children closed
- ✅ Two-column layout: columns NOT merged into one line (test_two_column_separate_blocks)
- ✅ Superscript span at higher y: clustered with baseline text
- ✅ Arabic text: bidi R characters detected, spans sorted right-to-left
- ✅ Mixed Latin+Arabic line: detected as "mixed" direction
44/44 tests pass in layout::line module.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add map_confidence_source to confidence module re-exports in lib.rs
- Remove duplicate map_confidence_source function from span/mod.rs
- Add Ocr case to map_unicode_source_to_confidence helper
- Add comprehensive tests for map_confidence_source in span module
The ConfidenceSource enum and map_confidence_source function were already
implemented in the confidence module from bead pdftract-2etcd. This change
completes the public API exposure and removes the duplicate implementation.
Acceptance criteria (all PASS):
- Single-glyph to_unicode span: confidence_source == Native
- Single-glyph shape_match span: confidence_source == Heuristic
- Mixed-glyph span (agl + shape_match): confidence_source == Heuristic
- 4.7 correction applied: Native -> Heuristic override
- OCR span: confidence_source == Ocr
- JSON serialization: lowercase strings
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The OutputOptions struct with block-kind filtering and CLI flags
was already implemented in the codebase. All 8 acceptance criteria
tests pass.
- Struct defined in pdftract-core/src/options.rs
- CLI flags wired in pdftract-cli/src/main.rs
- Tests: default values, block kind filtering, span filtering
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The map_confidence_source function was already implemented in
crates/pdftract-core/src/confidence.rs with comprehensive tests.
All acceptance criteria PASS:
- Unit tests for all 12 (UnicodeSource, corrected) combinations
- ToUnicode + corrected=true correctly downgrades to Heuristic
- Ocr is unaffected by correction flag
- Exhaustive match enforces compiler completeness
- INV-9 mapping table documented
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement the map_confidence_source(unicode_source: UnicodeSource,
corrected_in_4_7: bool) -> ConfidenceSource function that collapses the
6 internal UnicodeSource variants down to the 3 schema-exposed
ConfidenceSource variants.
- Mapping follows INV-9 stable taxonomy
- Phase 4.7 correction override: corrected Unicode downgrades
Native -> Heuristic
- OCR is never affected by corrections (corrections apply to vector
text, not raster OCR output)
- Exhaustive match on UnicodeSource ensures compiler-enforced
completeness
Acceptance criteria:
- Unit tests for all (UnicodeSource, corrected) combinations PASS
- ToUnicode + corrected=true → Heuristic (override applies)
- Ocr + corrected=true → Ocr (override does NOT apply)
- INV-9 mapping table documented in code comments
Also fixed pre-existing compilation errors in encryption module:
- detection.rs: syntax error in PdfObject::Array construction
- mod.rs: removed duplicate EncryptionInfo struct definition
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The invisible text filter in serialize_page_text() was always recomputing
block text from spans, but when block.spans is empty (no span data available),
this produced empty text for all blocks. Added fallback to use pre-computed
block.text when span data is missing, maintaining backward compatibility.
Also added special case for figure blocks to always emit empty text regardless
of span data.
All 111 text module tests pass, including all invisible text filtering tests
for Tr=0-7 and include_invisible=true/false combinations.
Acceptance criteria PASS:
- rendering_mode 3 excluded by default: ✓
- rendering_mode 3 included when flagged: ✓
- Mixed block emits visible: ✓
- All-invisible block produces empty (no spurious \n\n): ✓
- Tr=4 treated same as Tr=3: ✓
Closes pdftract-38p8h
Add two example reverse-proxy configuration files to help operators
deploy pdftract serve with TLS and authentication in front of the
no-auth pdftract server.
- docs/operations/serve-nginx-example.conf: nginx config with Basic Auth,
proxy_pass to localhost:8080, /extract and /health endpoints
- docs/operations/serve-traefik-example.yaml: Traefik dynamic config with
BasicAuth middleware, buffering limits, separate health router
Both configs include top comments explaining the deployment model:
pdftract serve binds to 127.0.0.1:8080 with no auth; the reverse
proxy provides TLS termination and authentication.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MarkedContentStack implementation was already complete.
All 45 tests pass (20 stack tests + 25 operator parser tests).
Acceptance criteria:
- push_bmc 64 times → all push; 65th emits MARKED_CONTENT_DEPTH_EXCEEDED ✅
- push_bmc N then pop_emc N → empty stack ✅
- pop_emc on empty stack → EmcUnderflow diagnostic ✅
- top_mcid returns Some(mcid) when top has MCID; None when empty ✅
- Unit tests cover push/pop balance, overflow, underflow ✅
- INV-8 (no panic) verified on all stack operations ✅
See notes/pdftract-1qoeb.md for details.
- Add detect_line_direction() function using unicode_bidi::bidi_class
- Count L (LTR) vs R/AL (RTL) characters, return dominant direction
- Default to Ltr for empty/neutral-only strings (per bead acceptance criteria)
- Return Mixed only when LTR and RTL counts are tied (both > 0)
- Add comprehensive tests for Latin, Arabic, Hebrew, Cyrillic, and edge cases
- Fix header_footer test: remove nonexistent reading_order_rank field
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
