jedarden a0c3ebb237 feat(bf-5for4): design manifest schema for grep-corpus

- Add manifest.csv with header row: filename,source_url,page_count,file_size,checksum,license
- Update README.md with comprehensive schema documentation and field descriptions
- Schema supports provenance tracking, integrity verification, and license compliance

Closes bf-5for4

2026-07-05 12:21:25 -04:00

2.2 KiB

Raw Blame History

pdftract grep-corpus

Benchmark corpus for pdftract-grep-1000 CI benchmark.

Purpose

This corpus contains 1000 PDFs (~100 MB total) used to benchmark and validate the grep feature's performance and correctness.

Structure

tests/fixtures/grep-corpus/
├── corpus/              # Actual PDF files
├── manifest.csv         # File metadata and expected match counts
├── regenerate.sh        # Script to rebuild the corpus
└── README.md            # This file

Usage

Running the benchmark

cargo bench --bench grep_1000

Regenerating the corpus

cd tests/fixtures/grep-corpus
./regenerate.sh

Corpus Requirements

The corpus must satisfy:

Size: 1000 PDF files, ~100 MB total
Content: Mix of vector and scanned PDFs
License: Public domain or permissive (CC BY-SA, MIT, etc.)
Determinism: Regenerable from source (no manual uploads)

CI Gates

The benchmark enforces these gates on every PR:

Throughput: ≥ 50 MB/s on 4-core CI machine
vs pdfgrep: ≥ 2× faster
vs pdftotext+ripgrep: ≥ 3× faster
Regression: ≤ 10% vs historical main

Status

TODO: Populate corpus (blocks on 7.8.1-7.8.9 grep implementation).

Sources (TODO)

Potential corpus sources:

arXiv API (public domain metadata)
Wikipedia article exports (CC BY-SA)
Synthetic PDFs via pdfjoin

Manifest Format

filename,source_url,page_count,file_size,checksum,license
doc001.pdf,https://example.com/doc001.pdf,10,102400,abc123...,public-domain
doc002.pdf,https://arxiv.org/pdf/1234.5678.pdf,15,98304,def456...,cc-by-4.0
...

Fields

filename: Relative path from corpus/ directory (e.g., doc001.pdf)
source_url: URL where the PDF was downloaded from
page_count: Number of pages in the PDF
file_size: File size in bytes
checksum: SHA256 hash of the file contents (for integrity verification)
license: License identifier:
- public-domain: Public domain
- cc-by-4.0: Creative Commons Attribution 4.0
- cc-by-sa-4.0: Creative Commons Attribution-ShareAlike 4.0
- mit: MIT License
- other: Other permissive license (document in source_url comments)

2.2 KiB Raw Blame History Unescape Escape