jedarden b970bc2d66 feat(bf-2f5ew): add Tier 4 competitive benchmark runner and results structure

- Create benches/results/.gitkeep and README.md
- Create benches/competitors/run_all.py Python orchestration script
- Script runs competitor benchmarks (pdftract, pdfminer.six, pypdf, pdfplumber)
- Emits benches/results/<commit-sha>.json with throughput metrics
- Add tier4-competitor-runner to pdftract-ci Argo WorkflowTemplate
- Runs only on main branch to track performance over time
- Acceptance criteria: ratio_pdfminer ≥ 10.0, ratio_pypdf ≥ 5.0

Closes bf-2f5ew

Files created:
- benches/results/.gitkeep
- benches/results/README.md
- benches/competitors/run_all.py
- notes/bf-2f5ew.md

Files modified:
- .ci/argo-workflows/pdftract-ci.yaml

2026-07-05 12:42:15 -04:00

1.3 KiB

Raw Blame History

Competitive Benchmark Results

This directory stores per-commit benchmark results tracking pdftract performance against competitor PDF extraction libraries.

Output Format

Each commit produces a JSON file named <commit-sha>.json with the following schema:

{
  "commit": "abc123...",
  "date": "2026-07-05T12:00:00Z",
  "pdftract_mbs": 125.5,
  "pdfminer_mbs": 12.3,
  "pypdf_mbs": 25.7,
  "pdfplumber_mbs": 11.8,
  "ratio_pdfminer": 10.2,
  "ratio_pypdf": 4.9
}

Metrics

*_mbs: Median throughput in megabytes per second (MB/s) for each tool
ratio_pdfminer: pdftract_mbs / pdfminer_mbs (target: ≥ 10.0)
ratio_pypdf: pdftract_mbs / pypdf_mbs (target: ≥ 5.0)

Acceptance Criteria

Per the plan (Tier 4 competitive benchmarks), pdftract must achieve:

≥ 10× faster than pdfminer.six on vector PDFs (ratio_pdfminer ≥ 10.0)
≥ 5× faster than pypdf on vector PDFs (ratio_pypdf ≥ 5.0)

Usage

# Run all competitive benchmarks
python3 benches/competitors/run_all.py

# Results are written to benches/results/<commit-sha>.json
cat benches/results/$(git rev-parse HEAD).json | jq '.'

CI Integration

The benchmark runs as a Tier 4 step in pdftract-ci Argo WorkflowTemplate, executing only on the main branch to track performance over time.

1.3 KiB Raw Blame History Unescape Escape