pdftract/notes/pdftract-1e5ud.md

pdftract-1e5ud: Rust SDK conformance test rig

Summary

The Rust SDK conformance test rig is fully implemented at crates/pdftract-core/tests/conformance.rs (1264 lines). The rig loads and executes shared SDK conformance cases from tests/sdk-conformance/cases.json and validates the 9-method SDK contract.

Implementation Details

Test Rig Structure

• File: crates/pdftract-core/tests/conformance.rs
• Test functions:
  1. test_sdk_public_api_contract - Compile-time API contract validation
  2. test_sdk_conformance_minimal - Fast smoke test with available fixtures
  3. test_sdk_conformance_quick - Subset of fast test cases
  4. test_sdk_conformance - Full conformance suite

Core Features

1. Dynamic case loading: Reads tests/sdk-conformance/cases.json at runtime
2. All 9 methods covered: extract, extract_text, extract_markdown, extract_stream, search, get_metadata, hash, classify, verify_receipt
3. Feature gating: is_feature_enabled() checks for ocr, decrypt, receipts, remote, xmp features
4. Tolerance support: Numeric comparisons with abs/rel tolerances via wildcard patterns
5. Fixture resolution: resolve_fixture_path() handles multiple fixture locations
6. Error reporting: Detailed diffs with case ID, field path, expected vs actual

Documentation

• ✅ CONTRIBUTING.md lines 107-120: Documents conformance suite with run commands
• ✅ crates/pdftract-core/README.md lines 33-50: Documents conformance test purpose and usage

Current Test Status (2026-06-02)

When running cargo test --test conformance:

• Total cases: 31 defined in cases.json
• Passed: 5 (extract-stream-cancellation, search-no-match, 2 minimal tests, api-contract)
• Failed: 26

Failure Categories

1. Stub fixture PDFs (majority): Most fixtures in tests/sdk-conformance/fixtures/ are minimal stub PDFs (<1KB each) without proper content streams
2. SDK implementation gaps: classify() has page index bounds checking issues
3. Expectation mismatches: Some test expectations may need adjustment

Example Failures

• extract-vector-scientific-paper: fixture has 0 pages (stub PDF)
• classify-*: "Page index 0 out of bounds" errors
• extract-text-*: Missing expected substrings (stub PDFs have no text)

Acceptance Criteria Status

Criterion	Status	Notes
cargo test passes on all cases	⚠️ PARTIAL	Rig works; fixtures need completion
New cases auto-run in CI	✅ PASS	Rig loads cases.json dynamically
Feature-gated skip messages	✅ PASS	is_feature_enabled() + clear skip reasons
Failed output shows ID + diff	✅ PASS	Prints case ID and detailed error messages
All 9 methods exercised	✅ PASS	cases.json covers all 9 methods
Documented in CONTRIBUTING.md	✅ PASS	Lines 107-120
Documented in README.md	✅ PASS	Lines 33-50

Key Files

File	Purpose
crates/pdftract-core/tests/conformance.rs	Test rig implementation (1264 lines)
tests/sdk-conformance/cases.json	Shared conformance test cases (31 cases)
tests/sdk-conformance/schema.json	Case format JSON schema
tests/sdk-conformance/fixtures/	Test fixture PDFs (currently stubs)

Verification

Run commands:

# Full conformance suite
cargo test -p pdftract-core --test conformance

# With all features
cargo test -p pdftract-core --test conformance --features ocr,profiles,remote,receipts

# Quick smoke test
cargo test -p pdftract-core --test conformance -- test_sdk_conformance_minimal

Conclusion

The conformance test rig is fully implemented and meets all functional requirements. The test failures are due to incomplete fixture PDFs and some SDK implementation gaps, not rig issues. The rig correctly:

• Loads and parses cases.json
• Executes all 9 SDK methods
• Applies tolerances correctly
• Skips feature-gated tests appropriately
• Reports detailed failure information

To achieve 100% pass rate, a follow-up task should complete the fixture PDFs and fix SDK implementation gaps (classify bounds checking, etc.).