pdftract/tests/lexer/fixtures
jedarden 585d861efc test(pdftract-sy8x): implement lexer proptest harness and curated corpus
Add property-based testing infrastructure for the lexer module with 6+
property tests covering INV-8 (no panic), string/hex roundtrips, name
length bounds, and position monotonicity. Create 8 curated fixture files
with golden token outputs for critical edge cases including EC-01 empty
file test and whitespace-only inputs.

Changes:
- Add prop_string_roundtrip to tests/proptest/lexer.rs
- Create tests/lexer/fixtures/ with 8 fixtures + .tokens.txt golden files
- Add gen_lexer_golden.rs binary for regenerating golden outputs
- Fix missing ObjRef import in marked_content_operators.rs

Acceptance criteria:
- cargo test --features proptest -p pdftract-core: 105 lexer tests pass
- tests/lexer/fixtures/ contains 8 fixtures with .tokens.txt outputs
- EC-01 empty file test: 0-byte input -> Token::Eof, no panic
- Whitespace-only file test passes
- INV-8 verified by prop_lexer_never_panics

Closes: pdftract-sy8x
2026-05-24 02:36:37 -04:00
..
bom_utf16_string.pdf.in test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
bom_utf16_string.pdf.tokens.txt test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
empty.bin test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
empty.tokens.txt test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
every_token.pdf.in test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
every_token.pdf.tokens.txt test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
hex_string_edge_cases.pdf.in test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
hex_string_edge_cases.pdf.tokens.txt test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
name_edge_cases.pdf.in test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
name_edge_cases.pdf.tokens.txt test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
numeric_edge_cases.pdf.in test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
numeric_edge_cases.pdf.tokens.txt test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
string_escapes.pdf.in test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
string_escapes.pdf.tokens.txt test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
whitespace_only.bin test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00
whitespace_only.tokens.txt test(pdftract-sy8x): implement lexer proptest harness and curated corpus 2026-05-24 02:36:37 -04:00