pdftract/notes/pdftract-5qj50.md

pdftract-5qj50: Mojibake detection + Latin-1-as-UTF-8 re-decode via encoding_rs

Summary

Implemented detect_and_repair_mojibake function in Phase 4.7 Correction Pipeline. The function detects Latin-1 bytes misinterpreted as UTF-8 (e.g., "café" → "café") and attempts recovery via encoding_rs::WINDOWS_1252 re-decoding.

Implementation

Files Created

• crates/pdftract-core/src/layout/correction.rs - New module with:
  • CorrectableText trait for mutable text access
  • detect_and_repair_mojibake<T, F>(span, scorer) -> bool function
  • TestCorrectable for unit tests
  • contains_mojibake_indicators helper function

Files Modified

• crates/pdftract-core/Cargo.toml - Made encoding_rs non-optional (was cjk-gated)
• crates/pdftract-core/src/hybrid.rs - Added CorrectableText impl for Span
• crates/pdftract-core/src/schema/mod.rs - Added CorrectableText impl for SpanJson
• crates/pdftract-core/src/layout/mod.rs - Added correction module and export

Key Features

1. Detection Heuristic: Checks for ≥2 occurrences of telltale 2-char sequences:

   • Latin-1 vowels: é, è, ê, î, ô, û, â, ç, ñ, etc.
   • Windows-1252 smart quotes: ’, â€", “, â€
   • NBSP pattern: Â followed by non-ASCII

2. Correction Process:

   • Encode text as UTF-8 bytes
   • Decode bytes as windows-1252 via encoding_rs
   • Score both original and candidate via scorer callback
   • Accept if candidate_score > original_score + 0.05

3. Invariants:

   • Clean ASCII/pure UTF-8 pass-through unchanged (fast-path)
   • Re-decoding REVERTED if score doesn't improve (false-positive safety)
   • Uses windows-1252 (not pure Latin-1) for Microsoft-isms support

Acceptance Criteria Status

Criterion	Status	Notes
Span text "café" → no à detected → no change	PASS	Clean UTF-8 fast-path works
Span text "café" → detected, re-decoded to "café"	PASS	Test: test_mojibake_detected_and_repaired
Span text "cafésandbar" (one Ã) → threshold check	PASS	Below 2-indicator threshold, no change
Span text "éèî" (multiple) → re-decoded if score improves	PASS	Test: test_mojibake_multiple_indicators
Asian text → pass-through unaffected	PASS	Test: test_asian_text_unaffected
Epsilon threshold (0.05) prevents noise	PASS	Tests: test_epsilon_threshold_prevents_noise, test_exact_epsilon_boundary
Re-decoding rejected if score doesn't improve	PASS	Test: test_replacement_rejected_if_score_doesnt_improve

Test Results

All unit tests in layout::correction::tests pass:

• test_clean_utf8_no_change ✓
• test_ascii_only_no_change ✓
• test_empty_string_no_change ✓
• test_mojibake_detected_and_repaired ✓
• test_mojibake_multiple_indicators ✓
• test_mojibake_single_indicator_threshold ✓
• test_smart_quote_mojibake ✓
• test_em_dash_mojibake ✓
• test_replacement_rejected_if_score_doesnt_improve ✓
• test_epsilon_threshold_prevents_noise ✓
• test_asian_text_unaffected ✓
• test_windows1252_specific ✓
• test_mixed_ascii_and_mojibake ✓
• test_nbsp_indicator ✓
• test_multiple_mojibake_patterns ✓
• test_exact_epsilon_boundary ✓
• test_just_above_epsilon ✓

Git Commits

• d84f8da - feat(pdftract-5qj50): implement mojibake detection and repair via encoding_rs

References

• Plan section: Phase 4.7 Correction pipeline step 3 (line 1797)
• Critical test: "Latin-1 mojibake é corrected to é when re-decode raises readability score" (line 1792)