# pdftract Tools & Generators

This directory contains utility scripts and generators for creating PDF test fixtures, debugging, and development workflow automation.

## PDF Fixture Generators

### Python Generators

#### `generate_encoding_fixtures.py`
**Purpose:** Generate Unicode recovery test fixtures for Phase 2.2–2.5

**Fixtures Generated:**
- `no-mapping.pdf` - Font with no ToUnicode and no standard encoding (worst case)
- `agl-only.pdf` - Font with only AGL glyph names (Level 2 recovery)
- `fingerprint-match.pdf` - Font embedded for fingerprint matching (Level 3)
- `shape-match.pdf` - Font for shape-based recognition (Level 4)

**Usage:**
```bash
python tools/generate_encoding_fixtures.py
```

**Requirements:** Python 3, standard library only

---

#### `generate_encrypted_pdf_fixtures.py`
**Purpose:** Generate encrypted PDF test fixtures for password handling

**Fixtures Generated:**
- `EC-04.pdf` - RC4-40 encrypted PDF (V=1, R=2)
- `EC-05.pdf` - AES-128 encrypted PDF (V=4, R=4)
- `EC-06.pdf` - AES-256 encrypted PDF (V=5, R=6)
- `EC-empty-password.pdf` - PDF with empty password

**Usage:**
```bash
python tools/generate_encrypted_pdf_fixtures.py
```

**Requirements:** Python 3, `pikepdf`

---

#### `generate_stress_pdf.py`
**Purpose:** Generate synthetic stress-test PDFs for memory ceiling testing

**Fixtures Generated:**
- Large-page-count PDFs for memory target validation
- 100-page vector PDF for buffered mode testing (target: < 512 MB)
- 10,000-page stress test for streaming mode validation (target: < 256 MB)

**Usage:**
```bash
python tools/generate_stress_pdf.py --pages 100 -o tests/fixtures/perf/100-page-vector.pdf
python tools/generate_stress_pdf.py --pages 10000 -o tests/fixtures/perf/10k-page.pdf
```

**Requirements:** Python 3, `reportlab`

---

#### `generate_invoice_pdf_fixtures.py`
**Purpose:** Generate invoice OCR test fixtures with proper DPI metadata

**Fixtures Generated:**
- Scanned PDF with correct 300 DPI settings
- Single invoice page with ground truth text for OCR testing

**Usage:**
```bash
python tools/generate_invoice_pdf_fixtures.py
```

**Requirements:** Python 3, `PIL` (Pillow)

---

#### `count_docs.py`
**Purpose:** Count rustdoc coverage for pdftract-core

**Usage:**
```bash
python tools/count_docs.py
```

**Requirements:** Python 3, standard library only

---

### Rust Generators

#### `generate_invoice_fixture.rs`
**Purpose:** Generate invoice fixture as a native Rust binary

**Usage:**
```bash
cargo run --bin generate_invoice_fixture
```

---

#### `generate_encrypted_pdf_fixtures.rs`
**Purpose:** Generate encrypted PDF fixtures as a native Rust binary

**Usage:**
```bash
cargo run --bin generate_encrypted_pdf_fixtures
```

---

### Shell Scripts

#### `convert_pdf_to_scanned.sh`
**Purpose:** Convert text-embedded PDFs to scanned image-based PDFs at 300 DPI

**Fixtures Generated:**
- `invoice/invoice-300dpi.pdf` (and backup invoice-300dpi-text-embedded.pdf)
- `letter/letter-300dpi.pdf` (and backup letter-300dpi-text-embedded.pdf)
- `form/form-300dpi.pdf` (and backup form-300dpi-text-embedded.pdf)

**Usage:**
```bash
./tools/convert_pdf_to_scanned.sh
```

**Requirements:**
- `pdftoppm` (poppler-utils)
- ImageMagick (via nix-shell or direct installation)

**Process:**
1. Backs up original text-embedded versions with `-text-embedded.pdf` suffix
2. Converts PDF to PPM images at specified DPI using `pdftoppm`
3. Converts PPM images back to PDF using ImageMagick
4. Cleans up temporary files

---

#### `count_docs.sh`
**Purpose:** Wrapper script for documentation counting

**Usage:**
```bash
./tools/count_docs.sh
```

---

#### `extract-release-notes.sh`
**Purpose:** Extract release notes from git commits

**Usage:**
```bash
./tools/extract-release-notes.sh
```

---

## Debugging Tools

### `debug-fingerprint/`
**Purpose:** Debug tool for PDF fingerprint computation

**Usage:**
```bash
cargo run --bin debug-fingerprint -- <pdf-path>
```

**Output:** Displays PDF fingerprint and computation time

---

### `debug-fingerprint-diff/`
**Purpose:** Compare fingerprints between two PDFs

**Usage:**
```bash
cargo run --bin debug-fingerprint-diff -- <pdf1> <pdf2>
```

---

## Specialized Fixture Builders

### `build-objstm-fixture/`
**Purpose:** Generate object stream fixtures for testing

**Usage:**
```bash
cd tools/build-objstm-fixture
cargo run --bin build-objstm-fixture
```

**Fixtures Generated:**
- Minimal PDFs with specific object stream structures
- Various compressed object configurations

---

### `build-xref-fixture/`
**Purpose:** Generate xref testing fixtures

**Usage:**
```bash
cd tools/build-xref-fixture
cargo run --bin build-xref-fixture
```

**Fixture Types:**
- Well-formed PDF with traditional xref table
- Well-formed PDF with xref stream (PDF 1.5)
- Hybrid file with traditional xref + `/XRefStm`
- PDF with 3 incremental revisions (`/Prev` chain)
- Linearized PDF (50 pages)
- File truncated at the start of xref
- File with `startxref` offset off by one
- File with corrupt xref entry
- File with circular `/Prev` reference

---

## Development Workflow

### Quick Documentation Coverage Check
```bash
python tools/count_docs.py
```

### Generate All Encoding Fixtures
```bash
python tools/generate_encoding_fixtures.py
```

### Generate All Encrypted PDF Fixtures
```bash
python tools/generate_encrypted_pdf_fixtures.py
```

### Generate Stress Test PDFs
```bash
python tools/generate_stress_pdf.py --pages 100 -o tests/fixtures/perf/100-page.pdf
python tools/generate_stress_pdf.py --pages 10000 -o tests/fixtures/perf/10k-page.pdf
```

### Debug Fingerprint Issues
```bash
cargo run --bin debug-fingerprint -- tests/fixtures/your-file.pdf
```

## Contributing

When adding new generators:
1. Use descriptive names with `generate_` prefix
2. Add a docstring/comment block explaining:
   - What fixtures it generates
   - How to run it (command line, dependencies)
   - Any special requirements (input data, API keys, etc.)
3. Update this README with the new tool
4. Follow existing patterns (Python for encoding/OCR, Rust for security/crypto)

## Cleanup Summary (2026-07-05)

The tools/ directory and tests/fixtures/ structure were cleaned up and organized through three coordinated beads:

- **bf-1iefu** (2026-07-05): Categorized all 17 generator scripts in tests/fixtures/ into KEEP (10), DELETE (5), RELOCATE (2)
- **bf-xqib3** (2026-07-05): Removed 5 obsolete generators and all compiled artifacts from tests/fixtures/
- **bf-2yhak** (2026-07-05): Relocated 2 general-purpose tools to tools/ with comprehensive documentation
- **bf-620xp** (2026-07-05): Verified and documented the cleaned structure

**Results:**
- 10 KEEP generators remain in tests/fixtures/ (actively maintained, co-located with their fixtures)
- 5 DELETE generators removed (obsolete duplicates and stubs)
- 2 RELOCATE tools moved to tools/ (general-purpose utilities)
- No compiled artifacts remain in tests/fixtures/
- tools/README.md created with comprehensive generator catalog
- tests/fixtures/STRUCTURE.md created documenting the final state

## Related Documentation

- [Test Fixtures Structure](../tests/fixtures/STRUCTURE.md) - Complete fixtures directory organization
- [Fixture Provenance Log](../tests/fixtures/PROVENANCE.md) - Generation history for each fixture
- [Generator Categorization (bf-1iefu)](../notes/bf-1iefu.md) - Rationale for KEEP/DELETE/RELOCATE decisions
- [Obsolete Generator Removal (bf-xqib3)](../notes/bf-xqib3.md) - Cleanup verification