3d795a2d11
Created tests/fixtures/scanned/ directory structure for WER gate testing: - README.md: Corpus overview and WER targets (<3% on clean 300-DPI scans) - GEN_MANIFEST.md: Fixture specifications and generation checklist - receipt/receipt-300dpi.txt: Ground truth for AS-02 test scenario (37 lines) - documents/invoice-300dpi.txt: Business invoice ground truth (55 lines) - documents/form-300dpi.txt: Employment application form (78 lines) - multi-page/doc-10page-300dpi.txt: Performance fixture (255 lines, 10 pages) Generation tools: - generate_scanned_fixtures.py: Python script for PDF generation - generate_scanned_fixtures.rs: Rust alternative for fixture metadata - calculate_wer.py: WER/CER calculation utility for OCR validation Test stub: - wer_gate_stub.rs: Placeholder for WER gate tests (marked #[ignore]) Total ground-truth content: 425 lines across 4 fixtures Next steps: 1. Generate PDFs from ground truth using generation script 2. Verify WER < 3% on generated fixtures 3. Enable WER gate tests Closes bf-2he4t
70 lines
2.2 KiB
Rust
70 lines
2.2 KiB
Rust
//! Stub for WER (Word Error Rate) gate test.
|
|
//!
|
|
//! This test will be implemented when the scanned PDF fixtures are fully generated.
|
|
//! It serves as a placeholder for the <3% WER Tier 1 OCR gate.
|
|
|
|
#[cfg(test)]
|
|
mod wer_gate_tests {
|
|
// TODO: Implement WER calculation
|
|
// TODO: Test each fixture against ground truth
|
|
// TODO: Verify WER < 3% for clean 300-DPI scans
|
|
// TODO: Verify processing time < 30s for 10-page fixture
|
|
|
|
#[test]
|
|
#[ignore = "Waiting for scanned PDF generation (bf-2he4t)"]
|
|
fn test_receipt_300dpi_wer() {
|
|
// pdftract extract tests/fixtures/scanned/receipt/receipt-300dpi.pdf --ocr --text
|
|
// Compare output with receipt-300dpi.txt
|
|
// Assert WER < 3%
|
|
}
|
|
|
|
#[test]
|
|
#[ignore = "Waiting for scanned PDF generation (bf-2he4t)"]
|
|
fn test_invoice_300dpi_wer() {
|
|
// Similar to receipt test
|
|
}
|
|
|
|
#[test]
|
|
#[ignore = "Waiting for scanned PDF generation (bf-2he4t)"]
|
|
fn test_form_300dpi_wer() {
|
|
// Similar to receipt test
|
|
}
|
|
|
|
#[test]
|
|
#[ignore = "Waiting for scanned PDF generation (bf-2he4t)"]
|
|
fn test_doc_10page_300dpi_wer() {
|
|
// Multi-page test
|
|
// Verify average WER < 3%
|
|
// Verify no page exceeds 5% WER
|
|
}
|
|
|
|
#[test]
|
|
#[ignore = "Waiting for scanned PDF generation (bf-2he4t)"]
|
|
fn test_10page_performance() {
|
|
// Verify processing time < 30s on 4-core CI
|
|
}
|
|
|
|
#[test]
|
|
#[ignore = "Waiting for scanned PDF generation (bf-2he4t)"]
|
|
fn test_as_02_scenario() {
|
|
// AS-02: Extract a scanned receipt via OCR
|
|
// Setup: receipt-300dpi.pdf
|
|
// Action: pdftract extract receipt-300dpi.pdf --ocr --text
|
|
// Verify: WER < 3%, total line present, latency < 30s
|
|
}
|
|
}
|
|
|
|
// Helper functions to be implemented:
|
|
|
|
// fn calculate_wer(ground_truth: &str, hypothesis: &str) -> f64 {
|
|
// // Implement Levenshtein distance-based WER calculation
|
|
// // WER = (substitutions + insertions + deletions) / total_words
|
|
// }
|
|
|
|
// fn extract_text_from_pdf(pdf_path: &str) -> Result<String, Error> {
|
|
// // Use pdftract CLI or library API
|
|
// }
|
|
|
|
// fn load_ground_truth(fixture_name: &str) -> String {
|
|
// // Load from corresponding .txt file
|
|
// }
|