feat(pdftract-19oy): codespace range parser + multi-byte tokenizer
Implemented codespace range parsing from begincodespacerange/endcodespacerange
blocks and multi-byte CJK tokenizer with widest-first matching per ISO 32000-1
9.10.3.1.
Changes:
- codespace.rs: Added pending_count handling for count-before-keyword syntax
- codespace.rs: Improved error recovery (skip invalid ranges, continue parsing)
- tokenize.rs: Added cfg guards for cjk feature diagnostic emission
- mod.rs: Added tokenize module exports
All acceptance criteria PASS:
- [<00>-<7F>, <8140>-<FEFE>] tokenizes to [0x41, 0x82A0, 0x42]
- [<00>-<7F>, <8000>-<FFFF>] tokenizes to [0x41, 0x82A0, 0x42]
- Widest-first matching for overlapping ranges
- Unrecognized bytes emit U+FFFD + diagnostic
- 1-byte-only codespace handles ASCII correctly
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
19c6328542