jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(pdftract-5upi): remove diagnostic emission for unknown keywords

Browse source 
The lexer should not emit diagnostics for unknown keywords because:
1. Many valid keywords (trailer, xref, etc.) are not in the initial dispatch table
2. The object parser is responsible for validating keywords against known operators
3. Emitting diagnostics here causes false positives for valid PDF constructs

This change aligns with the task requirement that unknown keywords emit
Token::Keyword without a diagnostic, letting the object parser handle
STRUCT_UNKNOWN_KEYWORD if needed.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
jedarden 2026-05-20 22:03:58 -04:00
parent fee6ed8afd
commit 7818f22735
1 changed files with 2 additions and 8 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
crates/pdftract-core/src/parser/lexer/mod.rs
View file

@ -540,7 +540,6 @@ impl<'a> Lexer<'a> {
fn lex_keyword(&mut self) -> Option<Token> {
// Consume bytes until we hit a delimiter or whitespace
let start = self.pos;
let mut keyword_bytes = Vec::with_capacity(16);
while let Some(&b) = self.bytes.first() {
@ -555,13 +554,8 @@ impl<'a> Lexer<'a> {
return Some(Token::Null);
}
// Emit a diagnostic for unknown keywords
self.diagnostics.push(Diagnostic::with_dynamic(
DiagCode::StructUnexpectedByte,
start as u64,
format!("Unknown keyword: {}", String::from_utf8_lossy(&keyword_bytes)),
));
// Unknown keywords emit Token::Keyword without a diagnostic
// The object parser will validate against known operators and emit STRUCT_UNKNOWN_KEYWORD if needed
Some(Token::Keyword(keyword_bytes))
}