pdftract/docs/errors-array-format.md

Errors Array Format and Test Integration Guide

This document explains the complete structure of the errors/diagnostics array in pdftract extraction results and how to integrate assertions in tests.

Overview

The pdftract extraction system uses a unified diagnostic system to report errors, warnings, and informational messages. All diagnostics are emitted during PDF parsing and extraction without panicking (per INV-8), allowing the parser to always attempt recovery and continue processing.

Extraction Result Structure

Main Result Structure

pub struct ExtractionResult {
    pub fingerprint: String,
    pub pages: Vec<PageResult>,
    pub metadata: ExtractionMetadata,
    pub signatures: Vec<SignatureJson>,
    pub form_fields: Vec<FormFieldJson>,
    pub links: Vec<LinkJson>,
    pub attachments: Vec<AttachmentJson>,
    pub threads: Vec<ThreadJson>,
    pub javascript_actions: Vec<JavascriptActionJson>,
}

Metadata Structure (Contains Errors/Diagnostics)

pub struct ExtractionMetadata {
    pub page_count: usize,
    pub receipts_mode: ReceiptsMode,
    pub span_count: usize,
    pub block_count: usize,
    pub cache_status: Option<String>,
    pub cache_age_seconds: Option<u64>,
    pub error_count: usize,              // Number of pages that failed to extract
    pub reading_order_algorithm: Option<String>,
    pub diagnostics: Vec<String>,        // THE ERRORS ARRAY
    pub profile_name: Option<String>,
    pub profile_version: Option<String>,
    pub profile_fields: Option<serde_json::Value>,
}

Errors Array Format

Location

• Field: ExtractionResult.metadata.diagnostics
• Type: Vec<String>
• Description: Array of diagnostic message strings emitted during extraction

String Format

Each diagnostic in the array follows this format:

{DIAGNOSTIC_CODE}: {Human-readable message} (byte offset {offset})?

Examples:

• STREAM_DECODE_ERROR: zlib stream truncated mid-inflation (byte offset 12345)
• STRUCT_INVALID_NAME: Name object exceeds 127-byte limit (byte offset 67890)
• FONT_GLYPH_UNMAPPED: Glyph could not be mapped to Unicode
• XREF_REPAIRED: Xref was reconstructed via forward scan

Components

1. Diagnostic Code (e.g., STREAM_DECODE_ERROR):

   • Identifies the type of error/warning/info
   • Follows naming convention: CATEGORY_SPECIFIC_ISSUE
   • Categories: STRUCT_*, STREAM_*, XREF_*, ENCRYPTION_*, PAGE_*, FONT_*, OCR_*, REMOTE_*, GSTATE_*, LAYOUT_*, MCP_*, CACHE_*, etc.

2. Message: Human-readable description of what occurred

3. Byte Offset (optional): Location in the PDF file where the error occurred

Accessing Error Information

From Extraction Result

use pdftract_core::extract::extract_pdf;

let result = extract_pdf("test.pdf", &Default::default())?;

// Access the errors array
let diagnostics = &result.metadata.diagnostics;

// Get error count
let error_count = result.metadata.error_count;

// Iterate through all diagnostics
for diagnostic in diagnostics {
    println!("{}", diagnostic);
}

Checking for Specific Errors

// Check if any stream decode errors occurred
let has_stream_errors = diagnostics.iter()
    .any(|d| d.contains("STREAM_DECODE_ERROR"));

// Check for encryption errors
let has_encryption_errors = diagnostics.iter()
    .any(|d| d.contains("ENCRYPTION"));

// Count specific error types
let glyph_unmapped_count = diagnostics.iter()
    .filter(|d| d.contains("FONT_GLYPH_UNMAPPED"))
    .count();

Diagnostic Categories and Codes

Structure Errors (STRUCT_*)

• STRUCT_INVALID_NAME - Invalid name character or malformed name object
• STRUCT_INVALID_HEX - Invalid hex character in hex string
• STRUCT_MISSING_KEY - Missing required dictionary key
• STRUCT_CIRCULAR_REF - Circular reference detected
• STRUCT_UNEXPECTED_BYTE - Unexpected byte during parsing
• STRUCT_UNEXPECTED_EOF - Unexpected end of file

Stream Errors (STREAM_*)

• STREAM_DECODE_ERROR - Stream decompression failed (corrupt data)
• STREAM_BOMB - Decompression bomb limit exceeded
• STREAM_UNKNOWN_FILTER - Unknown filter name
• STREAM_INVALID_PARAMS - Invalid filter parameters
• STREAM_TRUNCATED - Stream data truncated

XRef Errors (XREF_*)

• XREF_INVALID_HEADER - Invalid xref keyword or header
• XREF_REPAIRED - Xref was reconstructed via forward scan
• XREF_TRUNCATED - Truncated xref table

Encryption Errors (ENCRYPTION_*)

• ENCRYPTION_UNSUPPORTED - Unsupported encryption or no password supplied
• ENCRYPTION_WRONG_PASSWORD - Password incorrect
• ENCRYPTION_INVALID_DICT - Invalid encryption dictionary

Font Errors (FONT_*)

• FONT_GLYPH_UNMAPPED - Glyph could not be mapped to Unicode
• FONT_NOT_FOUND - Font not found or couldn't be parsed
• FONT_INVALID_CMAP - Invalid CMap format
• FONT_PARSE_FAILED - Font program parsing failed

Page Errors (PAGE_*)

• PAGE_OUT_OF_RANGE - Page number out of range
• PAGE_INVALID_COUNT - Invalid /Count key in /Pages tree
• PAGE_INVALID_ROTATE - Invalid /Rotate value (not multiple of 90)

Graphics State Errors (GSTATE_*)

• GSTATE_STACK_OVERFLOW - Graphics state stack overflow
• GSTATE_STACK_UNDERFLOW - Graphics state stack underflow
• CM_ARG_COUNT - Invalid argument count for cm operator

OCR Errors (OCR_*)

• OCR_JBIG2_UNSUPPORTED - JBIG2 decoder not available
• OCR_TESSERACT_FAILED - Tesseract OCR failed
• OCR_LANGUAGE_UNAVAILABLE - Requested language pack not available

Assertion Patterns for Tests

Pattern 1: Check for No Errors (Clean Extraction)

#[test]
fn test_clean_pdf_extracts_without_errors() {
    let result = extract_pdf("tests/fixtures/clean.pdf", &Default::default())
        .expect("Extraction should succeed");
    
    // Verify no diagnostics were emitted
    assert!(
        result.metadata.diagnostics.is_empty(),
        "Expected no diagnostics, but got: {:?}",
        result.metadata.diagnostics
    );
    
    // Verify error count is zero
    assert_eq!(result.metadata.error_count, 0);
}

Pattern 2: Check for Specific Error Code

#[test]
fn test_truncated_stream_emits_decode_error() {
    let result = extract_pdf("tests/fixtures/truncated.pdf", &Default::default())
        .expect("Extraction should succeed (with recovery)");
    
    // Check for STREAM_DECODE_ERROR diagnostic
    let has_decode_error = result.metadata.diagnostics.iter()
        .any(|d| d.contains("STREAM_DECODE_ERROR"));
    
    assert!(
        has_decode_error,
        "Expected STREAM_DECODE_ERROR diagnostic, got: {:?}",
        result.metadata.diagnostics
    );
}

Pattern 3: Check for Multiple Error Types

#[test]
fn test_malformed_pdf_emits_multiple_warnings() {
    let result = extract_pdf("tests/fixtures/malformed.pdf", &Default::default())
        .expect("Extraction should succeed (with recovery)");
    
    // Check for multiple expected diagnostics
    let diagnostics = &result.metadata.diagnostics;
    
    assert!(
        diagnostics.iter().any(|d| d.contains("STRUCT_INVALID_NAME")),
        "Expected STRUCT_INVALID_NAME"
    );
    
    assert!(
        diagnostics.iter().any(|d| d.contains("STREAM_DECODE_ERROR")),
        "Expected STREAM_DECODE_ERROR"
    );
    
    assert!(
        diagnostics.iter().any(|d| d.contains("XREF_REPAIRED")),
        "Expected XREF_REPAIRED"
    );
}

Pattern 4: Count Specific Errors

#[test]
fn test_pdf_with_unmapped_glyphs() {
    let result = extract_pdf("tests/fixtures/custom-font.pdf", &Default::default())
        .expect("Extraction should succeed");
    
    // Count unmapped glyph diagnostics
    let unmapped_count = result.metadata.diagnostics.iter()
        .filter(|d| d.contains("FONT_GLYPH_UNMAPPED"))
        .count();
    
    assert!(
        unmapped_count > 0,
        "Expected at least one FONT_GLYPH_UNMAPPED diagnostic"
    );
    
    println!("Found {} unmapped glyphs", unmapped_count);
}

Pattern 5: Verify Error Message Content

#[test]
fn test_encryption_error_message() {
    let result = extract_pdf_encrypted("tests/fixtures/encrypted.pdf", None)
        .expect("Should produce error result");
    
    // Find the encryption error diagnostic
    let encryption_diag = result.metadata.diagnostics.iter()
        .find(|d| d.contains("ENCRYPTION_UNSUPPORTED"))
        .expect("Expected ENCRYPTION_UNSUPPORTED diagnostic");
    
    // Verify message contains expected text
    assert!(
        encryption_diag.contains("no password supplied") || 
        encryption_diag.contains("Unsupported encryption"),
        "Encryption error message should mention password or unsupported algorithm"
    );
}

Pattern 6: Check Error Count Matches

#[test]
fn test_partial_extraction_error_count() {
    let result = extract_pdf("tests/fixtures/partial-corrupt.pdf", &Default::default())
        .expect("Extraction should succeed");
    
    // Verify error_count field matches actual diagnostics
    let actual_error_count = result.metadata.diagnostics.iter()
        .filter(|d| d.contains("ERROR") || d.contains("FATAL"))
        .count();
    
    assert_eq!(
        result.metadata.error_count as usize,
        actual_error_count,
        "error_count field should match number of ERROR/FATAL diagnostics"
    );
}

Pattern 7: Assert No Fatal Errors

#[test]
fn test_extraction_has_no_fatal_errors() {
    let result = extract_pdf("tests/fixtures/complex.pdf", &Default::default())
        .expect("Extraction should succeed");
    
    // Ensure no fatal-level diagnostics were emitted
    let has_fatal = result.metadata.diagnostics.iter()
        .any(|d| {
            // Fatal codes include ENCRYPTION_UNSUPPORTED, REMOTE_TLS_FAILED, etc.
            d.contains("ENCRYPTION_UNSUPPORTED") ||
            d.contains("REMOTE_TLS_FAILED") ||
            d.contains("REMOTE_DNS_FAILED")
        });
    
    assert!(
        !has_fatal,
        "Extraction should not have fatal errors: {:?}",
        result.metadata.diagnostics
    );
}

Pattern 8: Verify Expected Number of Diagnostics

#[test]
fn test_specific_diagnostic_count() {
    let result = extract_pdf("tests/fixtures/known-issues.pdf", &Default::default())
        .expect("Extraction should succeed");
    
    // If we know this PDF produces exactly 3 warnings
    assert_eq!(
        result.metadata.diagnostics.len(),
        3,
        "Expected exactly 3 diagnostics, got {}: {:?}",
        result.metadata.diagnostics.len(),
        result.metadata.diagnostics
    );
}

Test Integration Examples

Example 1: Truncated Flate Stream Test

#[test]
fn test_truncated_flate_stream_recovery() {
    let fixture_path = "tests/fixtures/malformed/truncated-flate.pdf";
    let result = extract_pdf(fixture_path, &Default::default())
        .expect("Extraction should succeed with recovery");
    
    // Should have emitted STREAM_DECODE_ERROR
    let has_stream_error = result.metadata.diagnostics.iter()
        .any(|d| d.contains("STREAM_DECODE_ERROR") || d.contains("STREAM_TRUNCATED"));
    
    assert!(has_stream_error, 
        "Expected stream decode error for truncated-flate.pdf");
    
    // Should still produce some output (recovery)
    assert!(!result.pages.is_empty(), 
        "Should produce at least one page despite stream error");
    
    // Verify the error message mentions truncation
    let error_msg = result.metadata.diagnostics.iter()
        .find(|d| d.contains("STREAM"))
        .expect("Should have stream-related diagnostic");
    
    assert!(error_msg.to_lowercase().contains("truncat") ||
            error_msg.to_lowercase().contains("incomplete"),
        "Stream error message should mention truncation or incomplete data");
}

Example 2: Encryption Error Test

#[test]
fn test_encrypted_pdf_without_password() {
    let fixture_path = "tests/fixtures/encrypted/livecycle.pdf";
    let result = extract_pdf_encrypted(fixture_path, None);
    
    // Should fail with encryption error
    assert!(result.is_err(), "Should fail when encrypted PDF lacks password");
    
    // Check the error contains ENCRYPTION_UNSUPPORTED
    if let Err(e) = result {
        let error_str = e.to_string();
        assert!(error_str.contains("ENCRYPTION_UNSUPPORTED") ||
                error_str.contains("encryption"),
                "Error should mention unsupported encryption: {}", error_str);
    }
}

Example 3: Font Glyph Unmapped Test

#[test]
fn test_custom_font_with_unmapped_glyphs() {
    let fixture_path = "tests/fixtures/fonts/custom-subset.pdf";
    let result = extract_pdf(fixture_path, &Default::default())
        .expect("Extraction should succeed");
    
    // Should have FONT_GLYPH_UNMAPPED diagnostics
    let unmapped_diagnostics: Vec<_> = result.metadata.diagnostics.iter()
        .filter(|d| d.contains("FONT_GLYPH_UNMAPPED"))
        .collect();
    
    assert!(!unmapped_diagnostics.is_empty(),
        "Custom font PDF should have unmapped glyph diagnostics");
    
    // Each unmapped glyph produces U+FFFD in output
    let text = result.pages.iter()
        .flat_map(|p| p.blocks.iter().flat_map(|b| b.spans.iter().map(|s| &s.text)))
        .collect::<String>();
    
    let fffd_count = text.matches('<27>').count();
    assert!(fffd_count > 0,
        "Output should contain replacement character for unmapped glyphs");
}

Best Practices for Test Assertions

1. Be Specific: Check for exact diagnostic codes when testing specific error conditions
2. Allow Recovery: Most tests should expect extraction to succeed despite errors (per INV-8 recovery principle)
3. Verify Output: When errors occur, verify that meaningful output was still produced
4. Count Errors: Use counts to verify expected number of diagnostics
5. Check Messages: Verify diagnostic messages contain expected information
6. Test Severity: Ensure fatal errors only occur for truly unrecoverable conditions
7. Use Helper Functions: Create reusable assertion helpers for common patterns

Helper Functions for Common Assertions

/// Assert that extraction produces a specific diagnostic code
pub fn assert_has_diagnostic(result: &ExtractionResult, code: &str) {
    assert!(
        result.metadata.diagnostics.iter().any(|d| d.contains(code)),
        "Expected diagnostic containing '{}', got: {:?}",
        code,
        result.metadata.diagnostics
    );
}

/// Assert that extraction has no diagnostics
pub fn assert_no_diagnostics(result: &ExtractionResult) {
    assert!(
        result.metadata.diagnostics.is_empty(),
        "Expected no diagnostics, got: {:?}",
        result.metadata.diagnostics
    );
}

/// Count diagnostics containing a specific code
pub fn count_diagnostics(result: &ExtractionResult, code: &str) -> usize {
    result.metadata.diagnostics.iter()
        .filter(|d| d.contains(code))
        .count()
}

/// Get all diagnostics containing a specific code
pub fn get_diagnostics(result: &ExtractionResult, code: &str) -> Vec<&String> {
    result.metadata.diagnostics.iter()
        .filter(|d| d.contains(code))
        .collect()
}

Summary

The errors array (ExtractionResult.metadata.diagnostics) provides comprehensive visibility into the PDF extraction process:

• Format: Vec<String> with each string formatted as CODE: message (offset)?
• Access: result.metadata.diagnostics
• Error Count: result.metadata.error_count
• Assertion Patterns: Check for specific codes, count errors, verify messages, ensure recovery
• Categories: STRUCT, STREAM, XREF, ENCRYPTION, PAGE, FONT, OCR, REMOTE, GSTATE, LAYOUT, MCP, CACHE, etc.

This unified diagnostic system allows tests to verify that errors are properly detected, reported, and recovered from during PDF extraction.