d5d7249ea6
Verify all acceptance criteria for grep-corpus manifest and validation: - manifest.csv exists with proper schema (1000 entries) - validate-corpus.sh is executable and functional - make validate-corpus runs successfully (1000/1000 files valid) - Corpus contains 10,590 pages across 6.5MB of synthetic PDFs Closes bf-4zuss.
85 lines
3.1 KiB
Markdown
85 lines
3.1 KiB
Markdown
# bf-4zuss: Corpus manifest and validation
|
||
|
||
## Summary
|
||
Verified that the grep-corpus manifest and validation infrastructure is fully operational.
|
||
|
||
## Acceptance Criteria Verification
|
||
|
||
### PASS: `tests/fixtures/grep-corpus/manifest.csv` exists with proper schema
|
||
- **Location**: `/home/coding/pdftract/tests/fixtures/grep-corpus/manifest.csv`
|
||
- **Schema**: filename,source_url,page_count,file_size,checksum,license
|
||
- **Sample entry**: `synthetic_1000.pdf,synthetic-generation,5,3751,<sha256>,public-domain`
|
||
|
||
### PASS: `scripts/validate-corpus.sh` exists and is executable
|
||
- **Location**: `/home/coding/pdftract/scripts/validate-corpus.sh`
|
||
- **Permissions**: `-rwxr-xr-x` (executable)
|
||
- **Lines**: 183 lines of bash validation logic
|
||
|
||
### PASS: `make validate-corpus` runs validation successfully
|
||
```bash
|
||
$ make validate-corpus
|
||
Validating grep-corpus...
|
||
[INFO] Validated 200 files...
|
||
[INFO] Validated 400 files...
|
||
[INFO] Validated 600 files...
|
||
[INFO] Validated 800 files...
|
||
[INFO] Validated 1000 files...
|
||
|
||
=== Corpus Validation Summary ===
|
||
Total files in manifest: 1000
|
||
Valid files: 1000
|
||
|
||
Corpus metrics (for valid files):
|
||
Total pages: 10590
|
||
Total size: 6870643 bytes
|
||
|
||
[✓] VALIDATION PASSED
|
||
```
|
||
|
||
### PASS: Manifest contains ≥1000 entries with all required fields
|
||
- **Entry count**: 1000 PDF files
|
||
- **All fields present**: filename, source_url, page_count, file_size, checksum, license
|
||
- **All entries validated**: 0 missing files, 0 size mismatches, 0 checksum mismatches
|
||
|
||
### PASS: Validation script confirms corpus meets size/count targets
|
||
- **Files validated**: 1000/1000
|
||
- **Total pages**: 10,590 pages
|
||
- **Total size**: 6,870,643 bytes (~6.5 MB)
|
||
- **Licenses**: All synthetic PDFs marked as `public-domain`
|
||
|
||
## Related Files
|
||
|
||
### Core Implementation
|
||
- `tests/fixtures/grep-corpus/manifest.csv` - Corpus manifest with 1000 entries
|
||
- `tests/fixtures/grep-corpus/corpus/` - Directory containing 1000 synthetic PDFs
|
||
- `scripts/validate-corpus.sh` - Validation script that verifies:
|
||
- File existence
|
||
- File size matching
|
||
- SHA256 checksum verification
|
||
- License presence
|
||
- Corpus quality metrics
|
||
|
||
### Supporting Scripts
|
||
- `scripts/download-grep-corpus.sh` - Downloads PDFs and generates manifest entries
|
||
- `scripts/grep-corpus-generate-manifest.sh` - Bootstraps manifest for existing corpus
|
||
|
||
### Makefile Integration
|
||
- `Makefile:22-24` - `validate-corpus` target that calls the validation script
|
||
|
||
## Corpus Quality Metrics
|
||
- **Corpus type**: Synthetic PDFs generated via ReportLab (public-domain)
|
||
- **Page count distribution**: 1-20 pages per PDF
|
||
- **Average file size**: ~6.9 KB per PDF
|
||
- **Total corpus size**: ~6.5 MB for 1000 PDFs
|
||
- **Validation status**: All 1000 files valid with correct checksums
|
||
|
||
## Implementation Notes
|
||
The corpus generation uses synthetic PDFs rather than downloaded content to ensure:
|
||
1. **Determinism**: Same script produces identical corpus
|
||
2. **Known license**: Public domain (no licensing concerns)
|
||
3. **Controlled variety**: Mix of page counts (1-20) and content
|
||
4. **Fast regeneration**: No network dependencies
|
||
|
||
## References
|
||
- Parent bead: bf-38sa3
|
||
- Phase 7.8 (plan lines 2796–2824)
|