jedarden
bcccc98fd7
docs(plan): fix 30 gaps from Round 1 gap review
...
CRITICAL fixes:
- Remove jpeg-decoder from Phase 1.5 crates (contradicted dep matrix)
- Specify word boundary adaptive threshold: text space, per-font-switch window, 20-glyph seed
- Add page_number (1-based) alongside page_index (0-based) to resolve SDK/schema mismatch
- Add mcid: Option<u32> to Glyph struct (was defined in 3.4 but missing from 3.2)
- Add aes + rc4 crates under new decrypt feature; document crypto dependency
HIGH fixes:
- Specify font fingerprint database format (phf::Map, SHA-256, ~500KB, JSON source)
- Fix Level 4 shape DB cross-ref (was "Phase 2.3", corrected to research doc); add Phase 2.5 definition
- Document header/footer cross-page pass as sequential post-rayon with Levenshtein matching
- Replace Tesseract box-file hint approach with PSM_SPARSE_TEXT + post-OCR validation
- Add HTTP serve security constraints: decompression bomb limit, auth guidance, no path params
- Add JavaScript detection spec to Phase 1.4 (all four JS action locations)
- Align CI benchmark gate to 10x pdfminer.six (was 5x, contradicted primary objectives)
- Add cargo bloat CI gate for phf word list size; bloomfilter fallback if >250KB
- Add pdftract-py-ci WorkflowTemplate note with manylinux/osxcross/cross approach
- Add ConfidenceSource enum → schema string mapping table in Phase 4.1
MEDIUM fixes:
- Define docs/schema/v1.0/pdftract.schema.json as Phase 6.1 deliverable
- Add unicode-bidi crate to dep matrix and Phase 4.2 for RTL detection
- Define Color enum with CSS hex conversion rules in Phase 3.1
- Remove bytes crate from Phase 1.2 (belongs in serve feature only; use Arc<[u8]>)
- Specify NDJSON buffer Condvar blocking behavior at window saturation
- Clarify pdftract:ocr vs pdftract:full Docker image tags and size budgets
- Add Docstrum parameters: k=5, Euclidean, ±30° constraints, root node definition
- Add code and formula block kind detection heuristics to Phase 4.4
- Add OCG visibility handling to Phase 1.4 (ON/OFF from /OCProperties /D /AS)
- Add linearized PDF detection and dual-xref merge to Phase 1.3
- Add HTTP 413 to error table with custom JSON rejection handler
- Add Phase 0: CI Infrastructure section (pdftract-ci WorkflowTemplate)
LOW fixes:
- Clarify Name length limit: 127 bytes pre-expansion, matching PDF spec 7.3.5
- Reorder preprocessing pipeline: contrast normalization before binarization (was after)
- Add CIDToGIDMap stream form: 2-byte big-endian GID array
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 17:45:04 -04:00
jedarden
d161d109b3
docs(plan): revise plan to center accuracy/speed/weight as hard targets
...
- Add Primary Objectives section with CI-gated measurable targets:
accuracy (CER <0.5%, WER <3%, readability >0.85), speed (100pp <3s,
10x vs pdfminer), weight (<4MB default binary, <20 default deps)
- Add feature-flag strategy: axum/tokio/pdfium/pyo3 are all optional;
default build is core extraction + CLI only
- Add Phase 4.7: text readability validation and correction pipeline
(ligature repair, hyphenation, mojibake detection, readability scoring)
- Make pdfium-render explicitly optional (full-render feature) vs. the
always-present direct image compositing path
- Add Tier 4 competitive benchmark suite (vs. pdfminer.six, pypdf, pdfplumber)
- Remove jpeg-decoder and whichlang from dependency matrix (unnecessary)
- Rename implementation-plan.md → plan.md (matches CLAUDE.md reference)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 17:07:48 -04:00
jedarden
12fad41596
Add research: span merging, Unicode normalization, implementation plan
...
Two new research documents covering the glyph-to-span-to-block assembly
pipeline (inter-operator merging, adaptive word gap threshold, column
detection, ligature bbox splitting, multi-granularity output) and
Unicode post-processing (NFC normalization, selective NFKC decomposition
for ligatures, PUA preservation, soft hyphen resolution, ZWJ/ZWNJ
handling, combining character reordering).
Also adds docs/plan/implementation-plan.md: the full 7-phase Rust
implementation roadmap covering core parser, font/encoding pipeline,
content stream processing, text assembly, OCR integration, API surface,
and advanced features — with crate selections, complexity ratings,
test strategy, and v0.1–v1.0 release milestones.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-16 16:15:14 -04:00
