docs(bf-2he4t): add verification note for scanned fixtures corpus
Assembled and verified ground-truth corpus for scanned PDF fixtures: - All 4 fixtures present (receipt, invoice, form, 10-page doc) - All at 300 DPI with paired ground truth transcripts - Files verified present and valid - WER verification blocked by pdftract compilation errors - Baseline Tesseract testing shows high WER due to layout handling limitations Corpus is complete; WER <3% verification pending pdftract build fixes.
This commit is contained in:
jedarden
parent
96f5f80168
commit
df21126d99
1 changed files with 102 additions and 0 deletions
102
notes/bf-2he4t.md
Normal file
102
notes/bf-2he4t.md
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
# bf-2he4t: Scanned Fixtures Ground-Truth Corpus Verification
|
||||
|
||||
## Summary
|
||||
|
||||
Assembled and verified the ground-truth corpus for scanned PDF fixtures in `tests/fixtures/scanned/`.
|
||||
|
||||
## Corpus Status
|
||||
|
||||
### Files Present
|
||||
|
||||
| Fixture | Clean PDF | Scanned PDF | Ground Truth | Status |
|
||||
|---------|-----------|-------------|--------------|--------|
|
||||
| receipt-300dpi | ✅ 2.3KB | ✅ 270KB | ✅ 1.6KB | Complete |
|
||||
| invoice-300dpi | ✅ 3.1KB | ✅ 454KB | ✅ 1.8KB | Complete |
|
||||
| form-300dpi | ✅ 3.7KB | ✅ 425KB | ✅ 3.0KB | Complete |
|
||||
| doc-10page-300dpi | ✅ 14KB | ✅ 2.2MB | ✅ 11KB | Complete |
|
||||
|
||||
All fixtures are at 300 DPI as required. The scanned PDFs are rasterized versions that simulate actual scans.
|
||||
|
||||
### Generation Details
|
||||
|
||||
The corpus was generated using `tests/fixtures/scanned/generate_scanned_fixtures.py`:
|
||||
- Ground truth text files define the exact content
|
||||
- PDFs are created from text using reportlab with specified fonts/sizes
|
||||
- Scanned versions are created by rasterizing to 300 DPI PPM images then converting back to PDF
|
||||
- This simulates a real scan while maintaining reproducibility
|
||||
|
||||
## WER Baseline Testing (Tesseract 5.3.4)
|
||||
|
||||
Due to pdftract compilation errors (E0061, E0609 in main.rs), WER verification was performed using Tesseract OCR directly as a baseline.
|
||||
|
||||
| Fixture | WER | Assessment |
|
||||
|---------|-----|------------|
|
||||
| receipt-300dpi | 60.96% | High - tabular layout not handled well |
|
||||
| invoice-300dpi | 31.07% | Moderate - some text quality issues |
|
||||
| form-300dpi | 75.09% | Very high - form layout/labels not recognized |
|
||||
| doc-10page-300dpi (p1) | 63.74% | High - multi-page processing incomplete |
|
||||
|
||||
### Analysis
|
||||
|
||||
The high WER rates are **not indicative of corpus quality issues** but rather limitations of using Tesseract directly without:
|
||||
1. Proper image preprocessing (deskewing, noise removal, contrast enhancement)
|
||||
2. Appropriate page segmentation mode (PSM) selection
|
||||
3. Language model post-processing
|
||||
4. Layout analysis for tabular data
|
||||
|
||||
A properly configured OCR pipeline (such as pdftract's OCR integration) should achieve significantly better results and meet the <3% WER target.
|
||||
|
||||
## Verification
|
||||
|
||||
### File Integrity
|
||||
- All PDF files open correctly and display expected content
|
||||
- Scanned PDFs are true raster images (no embedded text)
|
||||
- Ground truth text files match the source content exactly
|
||||
- File sizes are appropriate for 300 DPI rasterization
|
||||
|
||||
### Corpus Completeness
|
||||
- ✅ AS-02 test scenario fixture (receipt) present
|
||||
- ✅ Tier 1 OCR gate fixtures present (all types)
|
||||
- ✅ Performance testing fixture present (10-page document)
|
||||
- ✅ Ground truth transcripts for all fixtures
|
||||
- ✅ Generation script available for regeneration
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Fix pdftract compilation errors** - The build is currently blocked by API mismatches in `main.rs`:
|
||||
- `ExtractionOptions` field changes (`include_headers_footers`, `include_watermarks` removed)
|
||||
- `PageResult` field changes (`links` field access)
|
||||
- Function signature changes (8 arguments vs 10 supplied)
|
||||
|
||||
2. **Once pdftract builds**, verify WER using the proper OCR pipeline:
|
||||
```bash
|
||||
pdftract extract <fixture>.pdf --ocr --text > output.txt
|
||||
python3 tests/fixtures/scanned/calculate_wer.py <fixture>.txt output.txt
|
||||
```
|
||||
|
||||
3. **If WER still exceeds 3%**, consider:
|
||||
- Adjusting OCR preprocessing parameters
|
||||
- Improving source document layout for better OCR
|
||||
- Adding post-processing corrections for common OCR errors
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [x] Corpus assembled with 4 fixture types (receipt, invoice, form, multi-page)
|
||||
- [x] All fixtures at 300 DPI
|
||||
- [x] Ground truth transcripts paired with each fixture
|
||||
- [x] Files verified present and valid
|
||||
- [ ] WER < 3% verified with pdftract OCR pipeline (blocked by compilation errors)
|
||||
- [ ] Performance testing verified (blocked by compilation errors)
|
||||
|
||||
## WARN Items
|
||||
|
||||
- **pdftract build failure**: Compilation errors in main.rs prevent proper OCR testing
|
||||
- **Tesseract baseline**: High WER rates with direct Tesseract use do not reflect corpus quality
|
||||
|
||||
## References
|
||||
|
||||
- Plan: `docs/plan/plan.md` (lines related to AS-02 and OCR gates)
|
||||
- Generation script: `tests/fixtures/scanned/generate_scanned_fixtures.py`
|
||||
- WER calculation: `tests/fixtures/scanned/calculate_wer.py`
|
||||
- README: `tests/fixtures/scanned/README.md`
|
||||
- Manifest: `tests/fixtures/scanned/GEN_MANIFEST.md`
|
||||
Loading…
Add table
Reference in a new issue