Commit graph

3 commits

Author SHA1 Message Date
jedarden
b970bc2d66 feat(bf-2f5ew): add Tier 4 competitive benchmark runner and results structure
- Create benches/results/.gitkeep and README.md
- Create benches/competitors/run_all.py Python orchestration script
- Script runs competitor benchmarks (pdftract, pdfminer.six, pypdf, pdfplumber)
- Emits benches/results/<commit-sha>.json with throughput metrics
- Add tier4-competitor-runner to pdftract-ci Argo WorkflowTemplate
- Runs only on main branch to track performance over time
- Acceptance criteria: ratio_pdfminer ≥ 10.0, ratio_pypdf ≥ 5.0

Closes bf-2f5ew

Files created:
- benches/results/.gitkeep
- benches/results/README.md
- benches/competitors/run_all.py
- notes/bf-2f5ew.md

Files modified:
- .ci/argo-workflows/pdftract-ci.yaml
2026-07-05 12:42:15 -04:00
jedarden
398ab747fc fix(pdftract-60h): fix bugs in benchmark runner script
- Add extraction of pdftract_geomean from tool_geomeans array for regression gate
- Fix vector geomean calculation to properly pass bash array values to Python

The benchmark infrastructure was complete but had two bugs:
1. $pdftract_geomean was used but never set (line 308)
2. Vector geomean calculation had broken Python code for array expansion

These fixes ensure the regression and 10x-faster gates will work correctly
once the pdftract binary with extract/grep subcommands is available.

Refs pdftract-60h
2026-05-18 01:29:41 -04:00
jedarden
857f928732 feat(pdftract-5omc): implement SDK conformance test runner pattern
Implement the conformance test runner pattern that every SDK will
implement to validate against the shared test suite.

- Rust reference implementation (crates/pdftract-core/tests/conformance.rs)
  * Full test suite loader and executor
  * Comparison engine with min/max, string constraints, tolerances
  * Skip logic for unsupported features and schema versions
  * Report generation in JSON format

- CLI compare subcommand (crates/pdftract-cli/src/main.rs)
  * pdftract compare - Compare actual vs expected with tolerances
  * Cross-language comparison tool to avoid reimplementations

- Documentation (docs/conformance/sdk-contract.md)
  * Complete pattern specification with pseudocode
  * Per-language runner locations
  * CI integration requirements

- Python reference stub (tests/python-conformance/test_conformance.py)
  * Full pytest-based implementation following the pattern

Closes: pdftract-5omc
2026-05-18 01:22:23 -04:00