jedarden
|
a601dcec76
|
feat(pdftract-2t9): implement regression corpus runner with CER gate
Add regression-corpus step to pdftract-ci that runs the freshly-built
x86_64-unknown-linux-musl binary against a 500-PDF private regression
corpus stored in B2 (via ARMOR encrypted S3 proxy).
Implementation:
- Add build-cer-diff template to build the cer-diff comparison tool
- Add regression-shard template with 8-way parallelism (withSequence 0-7)
- Each shard processes ~63 documents, downloads PDFs via ARMOR proxy,
runs pdftract extract, compares against baseline using cer-diff
- Exit handler aggregates results into regression-results.jsonl artifact
- Add regression-mode parameter (gate|update) for PR vs merge behavior
CER computation:
- Uses existing cer-diff binary (crates/pdftract-cer-diff/)
- Levenshtein distance-based Character Error Rate
- Fails if per-document CER delta > 0.5% in gate mode
- Update mode refreshes baselines (requires follow-up bead for CronWorkflow)
Infrastructure:
- ARMOR proxy endpoint: armor.armor.svc.cluster.local:9000
- Credentials from armor-secrets Secret (ESO-synced from OpenBao)
- Corpus: s3://pdftract-regression-corpus/v1/*.pdf
- Baselines: s3://pdftract-regression-corpus/baselines/<sha256>.json
Acceptance criteria:
- PASS: regression-corpus step runs on every PR
- PASS: 8 shards process 500 docs in ~8 min budget (3 sec/doc target)
- PASS: Deliberate regression trips gate on CER > 0.5%
- PASS: regression-results.jsonl artifact published every run
- WARN: Baseline-refresh workflow requires Phase 0.6.1 follow-up
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
2026-05-18 01:17:58 -04:00 |
|