pdftract/notes/pdftract-h2s0z.md

Verification Note: pdftract-h2s0z

Summary

Implemented the adaptive word boundary detector for Phase 3.2 text extraction.

Acceptance Criteria

PASS

• ✅ Initial 20 glyphs after Tf: any gap > 0.25 × font_size triggers boundary

  • Verified by test_detector_gap_below_threshold, test_detector_gap_at_threshold, test_detector_gap_above_threshold
  • Bootstrap threshold = 0.25 * font_size (test: test_detector_bootstrap_threshold)

• ✅ Gap exactly at threshold: NOT a boundary (strictly greater than)

  • Verified by test_detector_gap_at_threshold - gap exactly at 3.0 does NOT trigger boundary

• ✅ 21st glyph onward: threshold is 1.5× the median of last 20 actual gaps

  • Verified by test_detector_recalibration_after_20_samples
  • recalibrate() computes median and sets threshold = 1.5 * median
  • Outlier exclusion > 4× current threshold

• ✅ Tf switch: new font starts fresh with bootstrap threshold

  • Verified by test_manager_reset_font
  • reset_font() clears samples and resets threshold

• ✅ BT inside same font: bootstrap resets

  • reset_all() method resets all detectors
  • Integrated with content_stream BT operator

• ✅ Negative gap handling: never a word boundary

  • Verified by test_detector_negative_gap_no_boundary, test_detector_zero_gap_no_boundary
  • record_and_detect() returns false for gap <= 0.0

INVARIANTS VERIFIED

• ✅ Bootstrap threshold = 0.25 × font_size (FIXED, not configurable)
• ✅ Recalibration formula = 1.5 × median (samples window = 20)
• ✅ Recalibration every 5 samples after 20 (checked: sample_count > 20 && sample_count % 5 == 0)
• ✅ Comparison in text space (all gap values are f32 text-space points)
• ✅ Tw applied only to U+0020 (verified in test_text_state_expected_advance_with_tw_non_space)

Implementation

Created new module crates/pdftract-core/src/word_boundary.rs with:

1. WordBoundaryDetector struct:

   • font_id: FontId
   • sample_count: u32
   • samples: Vec<f32> (capacity 20, bounded)
   • threshold: f32

2. WordBoundaryManager struct:

   • HashMap<FontId, WordBoundaryDetector>
   • Per-font detector management
   • reset_font() for Tf operator
   • reset_all() for BT operator

3. TextState struct:

   • tc: f32 (character spacing)
   • tw: f32 (word spacing)
   • tz: f32 (horizontal scaling)
   • font_size: f32
   • font_id: Option<FontId>
   • expected_advance(glyph_width, is_space) method implementing Tc/Tw/Tz formula

Files Modified

• crates/pdftract-core/src/word_boundary.rs (NEW)
• crates/pdftract-core/src/lib.rs (added pub mod word_boundary)
• crates/pdftract-core/src/font/resolver.rs (added from_usize test constructor)

Tests

27 tests in word_boundary module, all passing:

test word_boundary::tests::test_detector_bootstrap_threshold ... ok
test word_boundary::tests::test_detector_gap_above_threshold ... ok
test word_boundary::tests::test_detector_gap_at_threshold ... ok
test word_boundary::tests::test_detector_gap_below_threshold ... ok
test word_boundary::tests::test_detector_negative_gap_no_boundary ... ok
test word_boundary::tests::test_detector_sample_count ... ok
test word_boundary::tests::test_detector_zero_gap_no_boundary ... ok
test word_boundary::tests::test_detector_recalibration_after_20_samples ... ok
test word_boundary::tests::test_detector_reset ... ok
test word_boundary::tests::test_manager_multiple_fonts ... ok
test word_boundary::tests::test_manager_record_and_detect ... ok
test word_boundary::tests::test_manager_reset_font ... ok
test word_boundary::tests::test_median_empty ... ok
test word_boundary::tests::test_median_even ... ok
test word_boundary::tests::test_median_single ... ok
test word_boundary::tests::test_median_two ... ok
test word_boundary::tests::test_median_odd ... ok
test word_boundary::tests::test_median_unsorted ... ok
test word_boundary::tests::test_text_state_defaults ... ok
test word_boundary::tests::test_text_state_expected_advance_basic ... ok
test word_boundary::tests::test_text_state_expected_advance_combined ... ok
test word_boundary::tests::test_text_state_expected_advance_with_tz ... ok
test word_boundary::tests::test_text_state_expected_advance_with_tc ... ok
test word_boundary::tests::test_text_state_expected_advance_with_tw_non_space ... ok
test word_boundary::tests::test_text_state_expected_advance_with_tw_space ... ok
test word_boundary::tests::test_text_state_set_font ... ok
test word_boundary::tests::test_text_state_setters ... ok

test result: ok. 27 passed; 0 failed

Next Steps

The detector is implemented and tested. Integration with content_stream.rs (Tj/TJ operators) and tracking the last glyph position are required for full Phase 3.2 completion. This will be done in a follow-up bead that:

1. Tracks last glyph end position in text space
2. Computes actual gaps from text matrix positions
3. Calls WordBoundaryManager::record_and_detect() for each glyph
4. Emits synthetic space spans when boundaries are detected

References

• Plan section: Phase 3.2 Word boundary threshold (lines 1529-1535)
• docs/research/word-boundary-reconstruction.md