- Add schema.json with complete baseline metrics definition
- Required fields: commit_sha, timestamp, pdftract_geomean, grep_1000_mean_ms
- Optional fields: throughput_mb_per_sec, files_per_sec, total_runtime_sec
- All fields include types, units, descriptions, and examples
- Follows JSON Schema draft-07 specification
- Add README.md with comprehensive documentation
- Purpose and use cases for baseline metrics
- Complete field reference with types and units
- Schema validation instructions
- CI/CD integration guidelines
- Performance targets from project plan
- Update procedures and regression detection
- Validate existing main.json against schema structure
- All required fields present and correctly formatted
Acceptance criteria:
✓ benches/baselines/ directory exists
✓ JSON schema documented with all required fields
✓ README explains baseline format and purpose
✓ Schema includes types and units for each metric
- Add extraction of pdftract_geomean from tool_geomeans array for regression gate
- Fix vector geomean calculation to properly pass bash array values to Python
The benchmark infrastructure was complete but had two bugs:
1. $pdftract_geomean was used but never set (line 308)
2. Vector geomean calculation had broken Python code for array expansion
These fixes ensure the regression and 10x-faster gates will work correctly
once the pdftract binary with extract/grep subcommands is available.
Refs pdftract-60h
Implement the conformance test runner pattern that every SDK will
implement to validate against the shared test suite.
- Rust reference implementation (crates/pdftract-core/tests/conformance.rs)
* Full test suite loader and executor
* Comparison engine with min/max, string constraints, tolerances
* Skip logic for unsupported features and schema versions
* Report generation in JSON format
- CLI compare subcommand (crates/pdftract-cli/src/main.rs)
* pdftract compare - Compare actual vs expected with tolerances
* Cross-language comparison tool to avoid reimplementations
- Documentation (docs/conformance/sdk-contract.md)
* Complete pattern specification with pseudocode
* Per-language runner locations
* CI integration requirements
- Python reference stub (tests/python-conformance/test_conformance.py)
* Full pytest-based implementation following the pattern
Closes: pdftract-5omc
