diff --git a/crates/pdftract-cli/tests/test_encryption_errors.rs b/crates/pdftract-cli/tests/test_encryption_errors.rs index 324a110..db3ac9b 100644 --- a/crates/pdftract-cli/tests/test_encryption_errors.rs +++ b/crates/pdftract-cli/tests/test_encryption_errors.rs @@ -19,9 +19,13 @@ //! - Plan line 1132: RC4 and AES-128/256 decryption implementation //! - Plan line 1149: Encrypted file with unknown handler error handling +use std::error::Error; +use std::env; use std::fs; +use std::io::{self, Write}; use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; +use std::time::Duration; // CLI module imports for encryption testing use pdftract_cli::password; @@ -31,7 +35,7 @@ use pdftract_core::diagnostics::{ /// Get the workspace root directory fn workspace_root() -> PathBuf { - let manifest_dir = std::env::var("CARGO_MANIFEST_DIR").unwrap(); + let manifest_dir = env::var("CARGO_MANIFEST_DIR").unwrap(); let path = PathBuf::from(manifest_dir); // We're in crates/pdftract-cli, so go up two levels to reach workspace root path.parent().unwrap().parent().unwrap().to_path_buf() diff --git a/notes/bf-4w3x9.md b/notes/bf-4w3x9.md index 14be742..83882b8 100644 --- a/notes/bf-4w3x9.md +++ b/notes/bf-4w3x9.md @@ -1,129 +1,88 @@ -# Verification Note: bf-4w3x9 - Verify Degraded Fixture with WER Measurement Script +# Verification Note for bf-4w3x9: Degraded Fixture WER Measurement -## Task Completed -Verified the degraded fixture integration with the WER measurement script and documented infrastructure requirements. - -## What Was Done - -### 1. Examined Fixture Structure -- **Degraded PDF**: `tests/fixtures/scanned/low-quality/degraded-200dpi.pdf` (588KB) - - Intentionally degraded at 200 DPI with blur, noise, and compression artifacts - - Created by `create_degraded_200dpi.py` script - -- **Ground Truth**: `tests/fixtures/scanned/low-quality/degraded-200dpi-ground-truth.txt` (1967 bytes) - - Contains the exact Abraham Lincoln public domain text - - To be used as reference for WER calculation - -### 2. Verified Measurement Script -- **Script**: `scripts/measure-wer.sh` -- **Dependencies**: Uses `tests/fixtures/scanned/calculate_wer.py` for WER calculation -- **Usage**: `./scripts/measure-wer.sh ` -- **Exit codes**: - - 0 if WER ≤ 3% (passes quality gate) - - 1 if WER > 3% (fails quality gate) - - 2 for errors (missing files, etc.) - -### 3. Script Functionality Verification -Tested the WER calculation logic with simulated OCR output: -```bash -# Created simulated OCR output with intentional degradation errors -cat > /tmp/degraded-ocr-output-errors.txt << 'EOF' -ABRAHAM LlNCOLN: THE PEOPLE'S LEADER IN THE STRUGGLE FOR NATIONAL EXlSTENCE -[... with character substitutions simulating OCR degradation ...] -EOF - -# Ran WER measurement script -bash scripts/measure-wer.sh /tmp/degraded-ocr-output-errors.txt \ - tests/fixtures/scanned/low-quality/degraded-200dpi-ground-truth.txt -``` - -**Result**: Script executed successfully and calculated WER correctly. - -### 4. Infrastructure Requirements Documented -To run the complete workflow (PDF → OCR → WER measurement), the following are required: - -#### Required System Dependencies: -```bash -# OCR build dependencies (missing on current system) -pkg-config # Package configuration tool -leptonica # Image processing library -tesseract-ocr # OCR engine -``` - -#### Required Rust Features: -```bash -# Build pdftract with OCR feature enabled -cargo build --release --features ocr -``` - -#### Complete Workflow: -```bash -# Step 1: Extract text with OCR -pdftract extract tests/fixtures/scanned/low-quality/degraded-200dpi.pdf \ - --ocr --text - > /tmp/ocr-output.txt - -# Step 2: Measure WER -bash scripts/measure-wer.sh /tmp/ocr-output.txt \ - tests/fixtures/scanned/low-quality/degraded-200dpi-ground-truth.txt - -# Expected: WER > 3% due to intentional degradation -``` +## Summary +Verified the degraded PDF fixture and WER measurement script infrastructure. The fixture files are in place and the measurement script works correctly, but OCR extraction is not available in the current environment. ## Acceptance Criteria Status -✅ **PASS** - `scripts/measure-wer.sh` script exists and is executable -✅ **PASS** - Script executes without errors when provided with valid input files -✅ **PASS** - WER calculation logic verified with simulated OCR data -✅ **PASS** - Script properly integrates with existing test infrastructure (calculate_wer.py) -✅ **WARN** - Full end-to-end test requires OCR feature with system dependencies (pkg-config, leptonica) +### ✅ PASS: Script infrastructure is ready +- `scripts/measure-wer.sh` is executable and functional +- `tests/fixtures/scanned/calculate_wer.py` implements correct WER calculation +- Script accepts OCR output and ground truth as inputs +- Returns exit code 0 for WER ≤ 3%, exit code 1 for WER > 3% -**WARN Details**: The current NixOS environment lacks pkg-config and leptonica build dependencies, preventing the complete OCR extraction workflow. The measurement script logic is verified and functional, but running OCR on the degraded PDF requires: -1. Installing system build dependencies (pkg-config, leptonica, tesseract) -2. Rebuilding pdftract with `--features ocr` -3. Running `pdftract extract --ocr` on the degraded PDF +### ✅ PASS: Fixture files are properly installed +- `tests/fixtures/scanned/low-quality/degraded-200dpi.pdf` (588KB) - verified exists +- `tests/fixtures/scanned/low-quality/degraded-200dpi-ground-truth.txt` (1,967 chars) - verified exists +- Ground truth contains 321 words from Abraham Lincoln biography +- Source: Public domain Project Gutenberg eBook #11728 -## Technical Findings +### ✅ PASS: WER calculation verified +Tested with identical files (expected 0% WER): +``` +WER: 0.0000 (0.00%) +Reference words: 321 +Hypothesis words: 321 +Exit code: 0 +``` -### Script Behavior -- **Input validation**: Properly checks for file existence and readability -- **Error handling**: Clear error messages for missing files or invalid arguments -- **WER threshold**: Set at 3% (configurable in calculate_wer.py) -- **Output formats**: Supports WER only, or WER + CER with `--verbose` +Tested with modified file (expected small WER): +``` +WER: 0.0125 (1.25%) +Reference words: 321 +Hypothesis words: 325 (4 extra words added) +Exit code: 0 (passes 3% threshold) +``` -### Fixture Integration -- **PDF fixture**: Valid 200 DPI degraded PDF (verified with pdfinfo) -- **Ground truth**: Accurate transcription of source content -- **Test pattern**: Follows established pattern from `wer_gate_stub.rs` tests +### ⚠️ WARN: OCR extraction not available in current environment +Attempted to build pdftract with `--features ocr` but failed due to missing system dependencies: +``` +pkg-config: Package lept was not found in the pkg-config search path +error: The system library 'lept' required by crate 'leptonica-sys' was not found +``` -### Expected WER Results -Based on the intentional degradation in the fixture: -- **Blur**: Gaussian blur radius 0.3 (simulating poor focus) -- **Noise**: Random noise amount 12 (simulating scan artifacts) -- **Contrast**: Reduced to 0.9 (simulating poor scan quality) -- **Sharpness**: Reduced to 0.85 -- **Compression**: JPEG quality 85 +**Impact**: Cannot generate OCR output from the degraded PDF in this environment. -These effects should produce WER significantly greater than 3%, which is **expected and acceptable** for this edge case fixture. The degraded fixture is specifically designed to test OCR quality boundaries, not to meet the 3% threshold required for clean 300-DPI scans. +## What Would Be Needed for Full Verification -## Files Examined -- `scripts/measure-wer.sh` - WER measurement script (verified functional) -- `tests/fixtures/scanned/calculate_wer.py` - WER calculation logic (verified) -- `tests/fixtures/scanned/low-quality/degraded-200dpi.pdf` - Degraded PDF fixture (valid) -- `tests/fixtures/scanned/low-quality/degraded-200dpi-ground-truth.txt` - Ground truth (verified) -- `tests/fixtures/scanned/low-quality/create_degraded_200dpi.py` - Fixture creation script +To complete the full WER measurement as intended by the bead: -## Recommendations +1. **Install system dependencies**: `leptonica` library development headers +2. **Rebuild pdftract with OCR**: `cargo build --bin pdftract --release --features ocr` +3. **Extract text from degraded PDF**: + ```bash + ./target/release/pdftract extract --text /tmp/degraded-ocr.txt \ + --ocr tests/fixtures/scanned/low-quality/degraded-200dpi.pdf + ``` +4. **Run WER measurement**: + ```bash + VERBOSE=1 scripts/measure-wer.sh \ + /tmp/degraded-ocr.txt \ + tests/fixtures/scanned/low-quality/degraded-200dpi-ground-truth.txt + ``` -### For Full Testing: -1. **Install OCR dependencies**: Add pkg-config, leptonica, tesseract to the environment -2. **Build with OCR**: Run `cargo build --release --features ocr` -3. **Run complete workflow**: Extract with OCR, then measure WER +## Expected WER Result -### For Current Environment: -The script is verified and ready to use. When OCR dependencies become available, the degraded fixture can be tested end-to-end to confirm WER > 3% as expected. +Based on the intentional degradation applied (200 DPI, blur, noise, contrast reduction, compression): +- **Expected WER**: > 3% (will fail the quality gate, which is acceptable for this edge case) +- The degradation was specifically designed to test OCR quality limits +- This fixture is for validating that OCR quality gates work correctly ---- +## Test Infrastructure Integration -**Generated**: 2026-07-06 -**Bead ID**: bf-4w3x9 -**Status**: COMPLETE (with documented infrastructure requirements) +The degraded fixture integrates properly with the test infrastructure: +- Located in standard fixture directory: `tests/fixtures/scanned/low-quality/` +- Follows naming convention: `{fixture-name}-ground-truth.txt` +- Documented in `tests/fixtures/PROVENANCE.md` +- Measurement script follows standard pattern for scanned fixtures + +## Conclusion + +The degraded fixture and WER measurement infrastructure are properly set up and verified. The only blocker is the missing OCR system library in the current environment, which prevents actual OCR extraction from the PDF. Once OCR dependencies are available, the full WER measurement can be performed as designed. + +## Files Verified +- ✅ `scripts/measure-wer.sh` - executable and functional +- ✅ `tests/fixtures/scanned/calculate_wer.py` - correct WER implementation +- ✅ `tests/fixtures/scanned/low-quality/degraded-200dpi.pdf` - 588KB fixture +- ✅ `tests/fixtures/scanned/low-quality/degraded-200dpi-ground-truth.txt` - 1,967 chars +- ✅ `tests/fixtures/PROVENANCE.md` - fixture documented