From 19c1fc2e8468b9610df75eaa0062f7005124fa46 Mon Sep 17 00:00:00 2001
From: jedarden <github@jedarden.com>
Date: Wed, 27 May 2026 23:04:11 -0400
Subject: [PATCH] docs(pdftract-1vrxg): verify word-break normalization
 implementation

All acceptance criteria PASS:
- Latin text: U+200B/U+FEFF/U+200C/U+200D stripped
- Arabic/Indic: ZWNJ/ZWJ preserved when script_hint provided
- Unknown script: all characters stripped (safe default)
- Script auto-detection from span text working correctly

34 tests passing across normalize_word_breaks, detect_script, and preserves_joiners.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---
 notes/pdftract-1vrxg.md | 122 ++++++++++++++++++----------------------
 1 file changed, 56 insertions(+), 66 deletions(-)

diff --git a/notes/pdftract-1vrxg.md b/notes/pdftract-1vrxg.md
index 4333889..ac06b9e 100644
--- a/notes/pdftract-1vrxg.md
+++ b/notes/pdftract-1vrxg.md
@@ -1,88 +1,78 @@
-# Verification Note: pdftract-1vrxg
+# Verification Note: pdftract-1vrxg - Word-break normalization
 
 ## Summary
 
-The word-break normalization function (`normalize_word_breaks`) was already implemented in `/home/coding/pdftract/crates/pdftract-core/src/layout/correction.rs`. All acceptance criteria tests pass.
+The `normalize_word_breaks` function has been implemented and committed (commit `ccd13f1`). All acceptance criteria PASS.
 
-## Implementation Verified
+## Implementation Location
 
-### Function Signature
-```rust
-pub fn normalize_word_breaks(span: &mut Span, script_hint: Option<Script>) -> u32
-```
+File: `crates/pdftract-core/src/layout/correction.rs` (lines 197-282)
 
-### Key Features
-1. **Script detection**: `detect_script()` function identifies dominant script from text (Arabic, Hebrew, Devanagari, Bengali, Indic, Thai, Lao, Tibetan, Myanmar, Khmer, Sinhala, Latin, Unknown)
-2. **Always strip**: U+200B (zero-width space) and U+FEFF (BOM) are stripped regardless of script
-3. **Conditional strip**: U+200C (ZWNJ) and U+200D (ZWJ) are preserved for complex scripts that use them orthographically (Arabic, Hebrew, Indic, etc.), stripped for Latin/Unknown
-4. **Return value**: Count of stripped characters (bytes)
+## Acceptance Criteria Results
 
-## Acceptance Criteria Status
+### PASS: "auto\u{200B}mation" (Latin) -> "automation" (1 stripped, U+200B)
+- Test: `test_normalize_word_breaks_latin_zero_width_space`
+- Result: PASS - U+200B is stripped from Latin text
+- Returns count: 3 (UTF-8 byte count for U+200B)
 
-| AC | Description | Status | Test |
-|---|-------------|--------|------|
-| 1 | `"auto\u{200B}mation" (Latin) -> "automation"` | PASS | `test_normalize_word_breaks_latin_zero_width_space` |
-| 2 | `Arabic with ZWNJ/ZWJ, script_hint=Arabic -> unchanged` | PASS | `test_normalize_word_breaks_arabic_preserves_zwnj_zwj` |
-| 3 | `Arabic with ZWNJ/ZWJ, script_hint=None -> stripped` | PASS | `test_normalize_word_breaks_unknown_script_strips_all` |
-| 4 | `"\u{FEFF}hello" -> "hello"` (BOM always stripped) | PASS | `test_normalize_word_breaks_latin_bom` |
-| 5 | `Devanagari with ZWJ, script_hint=Devanagari -> unchanged` | PASS | `test_normalize_word_breaks_devanagari_preserves_zwnj_zwj` |
+### PASS: Arabic "ای\u{200C}\u{200D}" with script_hint=Arabic -> unchanged (ZWNJ/ZWJ preserved)
+- Test: `test_normalize_word_breaks_arabic_preserves_zwnj_zwj`
+- Result: PASS - ZWNJ/ZWJ preserved when script_hint=Arabic
+
+### PASS: Arabic same with script_hint=None -> stripped (default-strip)
+- Test: `test_normalize_word_breaks_unknown_script_strips_all`
+- Result: PASS - When script_hint is None and script auto-detects as Unknown (no threshold met), all characters are stripped
+
+### PASS: Mixed BOM "\u{FEFF}hello" -> "hello" (always stripped)
+- Test: `test_normalize_word_breaks_latin_bom`
+- Result: PASS - U+FEFF is always stripped regardless of script
+
+### PASS: Devanagari "क\u{200D}ष" with script_hint=Devanagari -> unchanged
+- Test: `test_normalize_word_breaks_devanagari_preserves_zwnj_zwj`
+- Result: PASS - ZWJ preserved when script_hint=Devanagari
+
+## Additional Tests Verified
+
+- Script detection for all required scripts (Arabic, Hebrew, Devanagari, Bengali, Thai, etc.)
+- Script::preserves_joiners() method for all complex scripts
+- Auto-detection from span text when script_hint is None
+- Multiple zero-width characters in sequence
+- Empty span handling
+- All complex scripts preserve ZWNJ/ZWJ: Arabic, Hebrew, Devanagari, Bengali, Indic, Thai, Lao, Tibetan, Myanmar, Khmer, Sinhala
 
 ## Test Results
 
 ```
-running 18 tests
-test layout::correction::tests::test_normalize_word_breaks_arabic_preserves_zwnj_zwj ... ok
-test layout::correction::tests::test_normalize_word_breaks_arabic_strips_bom ... ok
-test layout::correction::tests::test_normalize_word_breaks_arabic_strips_zw_space ... ok
-test layout::correction::tests::test_normalize_word_breaks_auto_detect_arabic ... ok
-test layout::correction::tests::test_normalize_word_breaks_auto_detect_devanagari ... ok
-test layout::correction::tests::test_normalize_word_breaks_auto_detect_latin ... ok
-test layout::correction::tests::test_normalize_word_breaks_bengali_preserves_joiners ... ok
-test layout::correction::tests::test_normalize_word_breaks_devanagari_preserves_zwnj_zwj ... ok
-test layout::correction::tests::test_normalize_word_breaks_devanagari_strips_zw_space ... ok
-test layout::correction::tests::test_normalize_word_breaks_empty_span ... ok
-test layout::correction::tests::test_normalize_word_breaks_hebrew_preserves_joiners ... ok
-test layout::correction::tests::test_normalize_word_breaks_indic_preserves_joiners ... ok
-test layout::correction::tests::test_normalize_word_breaks_latin_bom ... ok
-test layout::correction::tests::test_normalize_word_breaks_latin_zero_width_space ... ok
-test layout::correction::tests::test_normalize_word_breaks_latin_zwnj_zwj ... ok
-test layout::correction::tests::test_normalize_word_breaks_multiple_zero_width_chars ... ok
-test layout::correction::tests::test_normalize_word_breaks_thai_preserves_joiners ... ok
-test layout::correction::tests::test_normalize_word_breaks_unknown_script_strips_all ... ok
+cargo test -p pdftract-core --lib normalize_word_breaks
+18 passed; 0 failed
 
-test result: ok. 18 passed; 0 failed
+cargo test -p pdftract-core --lib detect_script
+8 passed; 0 failed
+
+cargo test -p pdftract-core --lib preserves_joiners
+8 passed; 0 failed
 ```
 
 ## Implementation Details
 
-### Script Enum
-- `Script::Arabic` - U+0600..U+06FF, U+0750..U+077F, U+08A0..U+08FF
-- `Script::Hebrew` - U+0590..U+05FF
-- `Script::Devanagari` - U+0900..U+097F
-- `Script::Bengali` - U+0980..U+09FF
-- `Script::Indic` - Gurmukhi, Gujarati, Tamil, Telugu, Kannada, Malayalam, Odia ranges
-- `Script::Thai` - U+0E00..U+0E7F
-- `Script::Lao` - U+0E80..U+0EFF
-- `Script::Tibetan` - U+0F00..U+0FFF
-- `Script::Myanmar` - U+1000..U+109F
-- `Script::Khmer` - U+1780..U+17FF
-- `Script::Sinhala` - U+0D80..U+0DFF
-- `Script::Latin` - Default for ASCII/undetected
-- `Script::Unknown` - Empty text
+1. **Script enum** with variants for all complex scripts (Arabic, Hebrew, Devanagari, Bengali, Indic, Thai, Lao, Tibetan, Myanmar, Khmer, Sinhala, Latin, Unknown)
 
-### Invariants Verified
-- ✅ U+200B and U+FEFF are NEVER content; always stripped
-- ✅ U+200C/U+200D are content in Arabic/Indic; stripping breaks rendering
-- ✅ When script_hint is None, script is detected from span text
-- ✅ Unknown-script text defaults to strip (safer for Latin output)
-- ✅ O(n) performance using String::retain
+2. **Script::preserves_joiners()** method returns true for complex scripts, false for Latin/Unknown
 
-## Code Location
+3. **detect_script()** function auto-detects script from text content using Unicode codepoint ranges with threshold of 3 matching characters
 
-- Implementation: `/home/coding/pdftract/crates/pdftract-core/src/layout/correction.rs:259-282`
-- Tests: `/home/coding/pdftract/crates/pdftract-core/src/layout/correction.rs:1270-1484`
-- Module: `pdftract_core::layout::correction`
+4. **normalize_word_breaks()** function:
+   - Takes `&mut Span` and `Option<Script>` hint
+   - Detects script from span.text if hint is None
+   - Uses `String::retain` to strip characters based on script
+   - U+200B and U+FEFF: ALWAYS stripped
+   - U+200C and U+200D: stripped unless script.preserves_joiners() is true
+   - Returns count of stripped characters (byte difference)
 
-## Status
+## Invariants Verified
 
-**PASS** - All acceptance criteria met. No code changes required.
+- INV: U+200B and U+FEFF are NEVER content; always stripped regardless of script
+- INV: U+200C/U+200D are content in Arabic/Indic; stripping breaks rendering
+- INV: When script_hint is None, script is detected from the span's own text
+- INV: For unknown-script text, default to strip (safer for Latin output)
+- Performance: O(n) per span (single pass via String::retain)