pdftract/notes/bf-4votp.md

# Fixtures Directory Structure Exploration

Bead: bf-4votp
Date: 2026-07-06
Task: Explore fixtures directory structure

## Overview

The `tests/fixtures/` directory contains **1,301 PDF files** organized into **16 top-level categories** with a maximum depth of 2 levels (very flat structure).

## Directory Structure

### Top-level Categories

```
tests/fixtures/
├── cjk/                    # 4 PDFs - Chinese, Japanese, Korean
├── classifier/             # 200 PDFs - Document type classification fixtures
│   ├── contract/          
│   ├── invoice/            # 50 numbered invoice PDFs (01-50.pdf)
│   ├── misc/
│   └── scientific_paper/
├── encoding/               # 6 PDFs - Font encoding edge cases
├── encrypted/              # 5 PDFs - RC4, AES-128, AES-256 encrypted PDFs
├── fonts/                  # 0 PDFs (empty)
├── forms/                  # 0 PDFs (has generate_form_fixtures.rs)
├── grep-corpus/            # 1,000 PDFs - Folder search benchmark corpus
│   └── corpus/
├── json_schema/            # 6 PDFs - Schema validation fixtures
├── malformed/              # 15 PDFs - Truncated, corrupt xref, circular refs
├── ocr/                    # 2 PDFs - OCR test fixtures
│   ├── brokenvector_aligned/
│   ├── brokenvector_misaligned/
│   ├── clean_lorem_ipsum/
│   ├── eng_fra_mixed/
│   └── perf_10_page/
├── page_class/             # 4 PDFs - Page classification test fixtures
│   ├── brokenvector_pdfa/
│   ├── hybrid_header_body/
│   ├── scanned_single/
│   └── vector_pure/
├── perf/                   # 2 PDFs - Performance benchmarking
│   └── 10k-page.pdf (2.4 MB)
├── portfolios/             # 0 PDFs (empty)
├── preprocess/             # 0 PDFs (empty)
│   ├── clean_digital/
│   ├── jbig2_scan/
│   ├── skewed_2deg/
│   ├── uneven_lighting/
├── profiles/               # 20 PDFs + 52 symlinks - Document profile fixtures
│   ├── invoice/            # Symlinks → ../../classifier/invoice/
│   ├── receipt/
│   ├── bank_statement/
│   ├── book_chapter/
│   ├── contract/
│   ├── form/
│   ├── invalid/            # 5 YAML files (malformed profiles)
│   ├── legal_filing/
│   ├── resolution/
│   ├── scientific_paper/
│   ├── slide_deck/
│   └── valid/              # 2 YAML files
├── scanned/                # 16 PDFs - Physical scans at various DPI
│   ├── documents/
│   ├── form/
│   ├── invoice/
│   ├── letter/
│   ├── low-quality/
│   ├── multi-page/         # Large files up to 2.2 MB
│   └── receipt/
├── security/               # 4 PDFs - Security test fixtures
│   └── xss-payload.pdf
├── vector/                 # 10 PDFs - Clean LaTeX/Word/InDesign outputs
│   ├── academic-paper/
│   ├── code-documentation/
│   ├── conference-proceedings/
│   ├── financial-report/
│   ├── legal-contract/
│   ├── medical-research/
│   ├── multi-page-academic/
│   ├── scientific-report/
│   ├── technical-documentation/
│   └── user-manual/
├── sample.pdf              # Root-level test PDF
├── remote_100page.pdf      # 5.9 MB - Remote source testing
├── test-minimal.pdf
├── tagged-suspects-*.pdf   # 3 files - Tagged PDF variants
└── valid-minimal.pdf
```

## Symlinks (Structural Oddity)

**53 symlinks** exist, primarily in the `profiles/` directory:

1. **Profile fixtures reference classifier fixtures:**
   - `tests/fixtures/profiles/invoice/*.pdf` → `../../classifier/invoice/*.pdf`
   - This allows profile validation tests to use the same underlying PDFs as classifier tests

2. **Self-referential directory symlink:**
   - `tests/fixtures/classifier/scientific_paper/scientific_paper` → `/home/coding/pdftract/tests/fixtures/classifier/scientific_paper`
   - Purpose unclear, appears to be a legacy artifact

## Non-PDF Files

### Binary Fixtures (LZW compression tests)
- `tests/fixtures/lzw_*.bin` (13 files) - Test fixtures for LZW stream decoder

### Markdown Files
- `STRUCTURE.md` - Fixtures documentation
- `PROVENANCE.md` - Source/origin tracking
- `no-mapping.md` - Encoding test notes

### YAML Profile Files
- 6 YAML files for document profile validation:
  - `profiles/valid/*.yaml` (2 files)
  - `profiles/invalid/*.yaml` (5 files)
  - `profiles/resolution/custom-invoice.yaml`

### Code Files
- `forms/generate_form_fixtures.rs` - Form fixture generator

### Expected Output Files
- `encrypted/EC-05-aes128-encrypted.expected.json` - Expected JSON output

## Size Distribution

| Category | Size Range | Largest File |
|----------|------------|--------------|
| Root-level | 534 B - 5.9 MB | `remote_100page.pdf` (5.9 MB) |
| scanned/ | 416K - 2.2 MB | `multi-page/report-300dpi.pdf` (2.2 MB) |
| perf/ | 2.4 MB | `10k-page.pdf` |
| Most categories | < 500 KB | N/A |

## Structural Characteristics

### Positive Attributes
1. **Flat hierarchy** - Max depth of 2 levels makes navigation simple
2. **No empty directories** - All categories contain files
3. **Clear naming** - Directory names self-document their purpose
4. **Logical organization** - Grouped by feature/test type

### Potential Issues
1. **Symlink complexity** - 53 symlinks could complicate:
   - File discovery algorithms
   - Test fixture copying/archiving
   - Docker volume mounts (symlink handling varies)
   
   The symlinks from `profiles/` to `classifier/` are intentional for test reuse but may break tools that don't follow symlinks.

2. **Large grep-corpus** - 1,000 PDFs in a single flat directory could:
   - Impact filesystem performance on some systems
   - Cause name collisions if more added
   - Benefit from sharding (e.g., `grep-corpus/batch-1/`, `grep-corpus/batch-2/`)

3. **Mixed content at root level** - 7 PDFs at the root (`sample.pdf`, `test-minimal.pdf`, etc.) lack clear categorization

## Recommendations for Discovery Implementation

1. **Follow symlinks** - Use `followSymlinks: true` or equivalent when walking the directory tree
2. **Handle grepcorpus specially** - Consider batching or streaming for the 1,000-PDF directory
3. **Canonicalize paths** - Resolve symlinks to their targets before deduplication
4. **Root-level special case** - Either categorize root PDFs or document them as "general purpose"

## Summary

- **Total PDFs:** 1,301
- **Total categories:** 16 top-level directories
- **Max depth:** 2 levels
- **Symlinks:** 53 (mostly in `profiles/` → `classifier/`)
- **Largest single file:** `remote_100page.pdf` (5.9 MB)
- **Most populated category:** `grep-corpus/` with 1,000 PDFs

The structure is well-organized and flat, with symlink-based fixture reuse being the primary complexity to handle in discovery implementation.