test(pdftract-5upi): add acceptance criteria tests for structural token lexer
Add comprehensive tests for array/dict delimiters, keywords, indirect
references, stream header validation, and edge cases like case-mismatched
keywords.
All tests verify the existing lexer implementation handles:
- [1 2 3] -> ArrayStart, Integer(1), Integer(2), Integer(3), ArrayEnd
- << /A 1 >> -> DictStart, Name(b"A"), Integer(1), DictEnd
- <48> -> String(b"\x48") (NOT dict - < vs << distinction)
- <<<48>>> -> DictStart, String(b"\x48"), DictEnd
- true false null -> Bool(true), Bool(false), Null
- 12 0 obj null endobj -> Integer(12), Integer(0), Obj, Null, EndObj
- 5 0 R -> Integer(5), Integer(0), IndirectRef
- stream\n vs stream\r -> StructInvalidStreamHeader for lone CR
- True (case-mismatched) -> Token::Keyword(b"True")
- proptest: random bytes never panic, always terminate with Eof
Addresses pdftract-5upi acceptance criteria.
60ae7ea561