diff --git a/docs/operations/manual-release.md b/docs/operations/manual-release.md new file mode 100644 index 0000000..d030f8c --- /dev/null +++ b/docs/operations/manual-release.md @@ -0,0 +1,682 @@ +# Manual Release Runbook (PB-13) + +> **Purpose:** This runbook is the fallback procedure for reproducing the milestone release locally when the Argo Workflows runner in `iad-ci` is degraded or unavailable for a prolonged window. Per PB-13 (plan line 567), this is the R13 mitigation strategy. + +**Trigger:** Argo Workflows in `iad-ci` cluster is unavailable for > 4 hours during a release window, or the release lead determines that automated release cannot proceed. + +**Executor:** Release lead or designated engineer with access to OpenBao secrets and cross-compilation toolchains. + +--- + +## Prerequisites + +### Hardware and OS + +- A clean Linux build environment (preferably the same Hetzner host that runs Argo; developer laptops are acceptable but NOT preferred due to build environment differences) +- At least 100 GB free disk space for Rust target directories and cross-compilation +- Stable internet connection for fetching crates and pushing artifacts + +### Installed Tools + +```bash +# Verify all required tools are present +cargo --version # >= 1.83 +rustup --version +maturin --version # >= 1.0 +mdbook --version # >= 0.12 +jq --version +gh --version # GitHub CLI +git --version +shasum --version # or sha256sum +gpg --version # for signing checksums +wrangler --version # Cloudflare Pages CLI +python3 --version # >= 3.11 +``` + +### Cross-Compilation Toolchains + +All five target triples MUST be available: + +| Target Triple | Toolchain Install Command | +|---------------|---------------------------| +| `x86_64-unknown-linux-musl` | `rustup target add x86_64-unknown-linux-musl` | +| `aarch64-unknown-linux-musl` | `rustup target add aarch64-unknown-linux-musl` | +| `x86_64-apple-darwin` | `rustup target add x86_64-apple-darwin` | +| `aarch64-apple-darwin` | `rustup target add aarch64-apple-darwin` | +| `x86_64-pc-windows-gnu` | `rustup target add x86_64-pc-windows-gnu` | + +Install `cross` for cross-compilation: + +```bash +cargo install cross --locked +``` + +### OpenBao Secrets Access + +The following secrets MUST be accessible via OpenBao (same ESO-style read access that Argo uses): + +| Secret | OpenBao Path | Purpose | +|--------|--------------|---------| +| PyPI token | `rs-manager/iad-ci/pypi/pdftract` | Upload Python wheels to PyPI | +| crates.io token | `rs-manager/iad-ci/crates-io/pdftract` | Publish Rust crates to crates.io | +| GitHub Release token | `rs-manager/iad-ci/github/pat/pdftract` | Create GitHub Release with artifacts | +| Cloudflare token | `rs-manager/iad-ci/cloudflare/api_token` | Deploy docs to Cloudflare Pages | +| GPG signing key | `rs-manager/iad-ci/gpg/pdftract_signing_key` | Sign checksums for release artifacts | + +**DO NOT copy secrets to disk on the developer machine.** Fetch them via environment variables or inject them directly into commands. + +--- + +## Step-by-Step Release Procedure + +### Step 1: Verify the Tag + +Ensure the tag matches the planned version and is annotated (not lightweight): + +```bash +git fetch --tags +git describe --exact-match +``` + +Expected output: `vX.Y.Z` where `X.Y.Z` matches the planned release version. + +If the command fails, you are not on a tagged commit. Checkout the correct tag: + +```bash +git checkout vX.Y.Z +``` + +Verify the tag is annotated: + +```bash +git tag -l vX.Y.Z -n9 +``` + +Annotated tags show a message; lightweight tags do not. Only annotated tags are acceptable for releases. + +### Step 2: Run Full CI Suite Locally + +Before building, verify the code passes all tests: + +```bash +cd /path/to/pdftract +cargo test --workspace --all-features +``` + +Expected: All tests pass (exit code 0). + +If tests fail, **DO NOT PROCEED**. Fix the failures and create a new tag. + +### Step 3: Cross-Compile Binaries for All Triples + +Build all 10 binary archives (5 triples × 2 feature variants: `default`, `full`): + +```bash +# Set environment variables for reproducible builds +export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD) +export CARGO_TARGET_DIR=./target + +# Build matrix +TARGETS=( + "x86_64-unknown-linux-musl" + "aarch64-unknown-linux-musl" + "x86_64-apple-darwin" + "aarch64-apple-darwin" + "x86_64-pc-windows-gnu" +) + +FEATURES=( + "default" + "full" +) + +for TARGET in "${TARGETS[@]}"; do + for FEATURES in "${FEATURES[@]}"; do + echo "Building ${TARGET} (features: ${FEATURES})..." + + FEATURE_FLAG="" + if [ "${FEATURES}" != "default" ]; then + FEATURE_FLAG="--features ${FEATURES}" + fi + + cross build --release --target ${TARGET} ${FEATURE_FLAG} --locked --frozen + + # Strip the binary + BINARY_PATH="target/${TARGET}/release/pdftract" + if [ "${TARGET}" = "x86_64-pc-windows-gnu" ]; then + BINARY_PATH="target/${TARGET}/release/pdftract.exe" + fi + + # Strip (use appropriate strip command for target) + if [[ "${TARGET}" == *-apple-darwin ]]; then + # macOS targets use x86_64-apple-darwin-strip or aarch64-apple-darwin-strip + ${TARGET%-unknown}-strip "${BINARY_PATH}" + else + strip "${BINARY_PATH}" + fi + done +done +``` + +Expected: All 10 builds complete without errors. + +### Step 4: Verify Each Binary Works + +Quick smoke test on the native Linux binary: + +```bash +./target/x86_64-unknown-linux-musl/release/pdftract --version +``` + +Expected: Version string matches the tag. + +For cross-compiled binaries, verify the file output exists and is non-empty: + +```bash +ls -lh target/*/release/pdftract* +``` + +Expected: All 10 binaries are present and non-zero size. + +### Step 5: Generate SHA-256 Checksums + +Create checksums for all binary archives: + +```bash +VERSION=$(git describe --exact-match | sed 's/^v//') +DIST_DIR="./dist" +mkdir -p ${DIST_DIR} + +# Generate archives and checksums in one pass +cd ${DIST_DIR} +for TARGET in "${TARGETS[@]}"; do + for FEATURES in "${FEATURES[@]}"; do + ARCHIVE_NAME="pdftract" + if [ "${FEATURES}" != "default" ]; then + ARCHIVE_NAME="pdftract-${FEATURES}" + fi + ARCHIVE_DIR="${ARCHIVE_NAME}-v${VERSION}-${TARGET}" + mkdir -p "${ARCHIVE_DIR}" + + # Copy binary + if [ "${TARGET}" = "x86_64-pc-windows-gnu" ]; then + cp "../target/${TARGET}/release/pdftract.exe" "${ARCHIVE_DIR}/" + else + cp "../target/${TARGET}/release/pdftract" "${ARCHIVE_DIR}/" + fi + + # Copy license files + cp ../LICENSE-MIT "${ARCHIVE_DIR}/" + cp ../LICENSE-APACHE "${ARCHIVE_DIR}/" + cp ../README.md "${ARCHIVE_DIR}/" + + # Extract CHANGELOG excerpt + python3 < CHECKSUMS.sha256 +``` + +Expected: `CHECKSUMS.sha256` contains SHA-256 hashes for all 10 archives. + +### Step 6: GPG Sign the Checksums File + +Sign the checksums file with the GPG key: + +```bash +# Fetch GPG key from OpenBao (example via environment variable) +export GPG_KEY=$(bao read -field=private_key rs-manager/iad-ci/gpg/pdftract_signing_key) + +# Import key temporarily (DO NOT persist to disk) +gpg --import <<< "${GPG_KEY}" + +# Sign the checksums +gpg --detach-sign --digest-algo SHA256 --armor --output CHECKSUMS.sha256.asc CHECKSUMS.sha256 + +# Verify the signature +gpg --verify CHECKSUMS.sha256.asc CHECKSUMS.sha256 +``` + +Expected: `gpg: Good signature` from the pdftract signing key. + +### Step 7: Build Python Wheels + +Build wheels for all three platforms (Linux, macOS, Windows): + +```bash +# Install maturin if not present +cargo install maturin --locked + +# Build wheels (this uses cross for non-native platforms) +maturin build --release --strip + +# Wheels are in target/wheels/ +ls -lh target/wheels/ +``` + +Expected: Wheels for `pdftract` and `pdftract-core` are present in `target/wheels/`. + +### Step 8: Upload Wheels to PyPI + +```bash +# Fetch PyPI token from OpenBao +export PYPI_TOKEN=$(bao read -field=token rs-manager/iad-ci/pypi/pdftract) + +# Upload using maturin (token is passed via environment variable) +echo "pypi-token = ${PYPI_TOKEN}" > ~/.pypirc +maturin upload --repository pypi target/wheels/*.whl +``` + +Expected: All wheels upload successfully (HTTP 200 responses). + +**Failure mode - PyPI upload fails midway:** + +- If upload fails mid-stream, DO NOT force overwrite (`--skip-existing` is acceptable for re-running) +- Check PyPI status page: https://status.pypi.org/ +- If PyPI is down, retry in 5 minutes +- If a specific wheel fails, check the wheel with `unzip -l` and verify it's not corrupted +- Document the failure and proceed to other channels (crates.io, GitHub Release can still succeed) + +### Step 9: Publish to crates.io + +Publish `pdftract-core` first, then `pdftract-cli` (dependency order matters): + +```bash +# Fetch crates.io token from OpenBao +export CARGO_REGISTRY_TOKEN=$(bao read -field=token rs-manager/iad-ci/crates-io/pdftract) + +# Publish pdftract-core first +cargo publish --package pdftract-core + +# Wait for index propagation (crates.io can take 1-5 minutes) +echo "Waiting for pdftract-core to appear in the index..." +sleep 60 # Wait at least 1 minute + +# Verify pdftract-core is available +cargo search pdftract-core --limit 1 + +# Publish pdftract-cli +cargo publish --package pdftract-cli +``` + +Expected: Both crates publish successfully. + +**Failure mode - crates.io publish fails:** + +- Check if the version already exists (this is a hard error from crates.io) +- If it's a network error, retry after 60 seconds +- If it's a validation error, fix `Cargo.toml` and create a new tag +- Document the failure and proceed to other channels + +### Step 10: Create GitHub Release + +```bash +VERSION=$(git describe --exact-match | sed 's/^v//') +REPO="jedarden/pdftract" + +# Fetch GitHub token from OpenBao +export GH_TOKEN=$(bao read -field=token rs-manager/iad-ci/github/pat/pdftract) + +# Create release with all artifacts +gh release create v${VERSION} \ + ./dist/pdftract*.tar.gz \ + ./dist/pdftract*.zip \ + ./dist/CHECKSUMS.sha256 \ + ./dist/CHECKSUMS.sha256.asc \ + --title "pdftract v${VERSION}" \ + --notes "Release notes: https://github.com/${REPO}/blob/v${VERSION}/CHANGELOG.md" +``` + +Expected: GitHub Release created at `https://github.com/jedarden/pdftract/releases/tag/v${VERSION}`. + +### Step 11: Build mdBook + +```bash +cd docs/user-docs +mdbook build +``` + +Expected: `docs/user-docs/book/` directory contains the built HTML site. + +### Step 12: Deploy Docs to Cloudflare Pages + +```bash +# Fetch Cloudflare token from OpenBao +export CLOUDFLARE_API_TOKEN=$(bao read -field=token rs-manager/iad-ci/cloudflare/api_token) + +# Deploy using wrangler +cd docs/user-docs +wrangler pages publish build --project-name=pdftract-docs +``` + +Expected: Docs deployed to `https://pdftract.com` (or the configured Pages domain). + +**Failure mode - wrangler publish fails:** + +- Check Cloudflare status: https://www.cloudflarestatus.com/ +- Verify the project name matches your Cloudflare Pages project +- If the build directory is missing, re-run `mdbook build` +- Document the failure and proceed (docs can be deployed later) + +### Step 13: Generate SLSA Level 2 Attestation + +Generate SLSA Provenance v1.0 predicate for the release artifacts: + +```bash +python3 <<'EOF' +import hashlib +import json +import subprocess +from datetime import datetime, timezone + +VERSION = "${VERSION}" +COMMIT_SHA = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip() +TAG = "v${VERSION}" +REPO = "jedarden/pdftract" + +# Compute SOURCE_DATE_EPOCH for reproducibility +SOURCE_DATE_EPOCH = subprocess.check_output( + ["git", "show", "-s", "--format=%ct", "HEAD"] +).decode().strip() + +# Collect all binary archives +subjects = [] +for f in ["pdftract*.tar.gz", "pdftract*.zip"]: + for path in glob.glob(f): + sha256 = hashlib.sha256() + with open(path, "rb") as fp: + for chunk in iter(lambda: fp.read(65536), b""): + sha256.update(chunk) + digest = sha256.hexdigest() + subjects.append({ + "name": os.path.basename(path), + "digest": {"sha256": digest} + }) + +# Build invocation ID +invocation_id = f"manual-release-{COMMIT_SHA}-{TAG}" + +# Build timestamp +started_on = datetime.fromtimestamp(int(SOURCE_DATE_EPOCH), tz=timezone.utc).isoformat() +finished_on = datetime.now(timezone.utc).isoformat() + +# Construct SLSA Provenance v1.0 predicate +provenance = { + "_type": "https://in-toto.io/Statement/v1", + "subject": subjects, + "predicateType": "https://slsa.dev/provenance/v1.0", + "predicate": { + "buildDefinition": { + "buildType": "https://github.com/jedarden/pdftract/releases/manual", + "externalParameters": { + "tag": TAG, + "version": VERSION, + "commit": COMMIT_SHA, + "source": f"https://github.com/{REPO}" + }, + "internalParameters": { + "workflow": "manual-release", + "targetTriples": [ + "x86_64-unknown-linux-musl", + "aarch64-unknown-linux-musl", + "x86_64-apple-darwin", + "aarch64-apple-darwin", + "x86_64-pc-windows-gnu" + ] + }, + "resolvedDependencies": [ + { + "uri": f"git+https://github.com/{REPO}@{COMMIT_SHA}", + "digest": {"gitCommit": COMMIT_SHA} + } + ] + }, + "runDetails": { + "builder": { + "id": "https://github.com/jedarden/pdftract/releases/manual", + "builderDependencies": [], + "version": { + "manual": "PB-13-fallback" + } + }, + "metadata": { + "invocationId": invocation_id, + "startedOn": started_on, + "finishedOn": finished_on + }, + "byproducts": [] + } + } +} + +# Write to multiple.intoto.jsonl +with open("multiple.intoto.jsonl", "w") as f: + f.write(json.dumps(provenance) + "\n") + +print(f"SLSA provenance written to multiple.intoto.jsonl") +print(f"Subjects: {len(subjects)}") +EOF +``` + +Upload the attestation to the GitHub Release: + +```bash +gh release upload v${VERSION} multiple.intoto.jsonl +``` + +Expected: `multiple.intoto.jsonl` appears in the GitHub Release assets. + +**Failure mode - SLSA attestation generation fails:** + +- The release can still ship without SLSA attestation, but document this in the CHANGELOG +- Add a note: `This release was produced via the PB-13 manual fallback; SLSA Level 2 attestation is not available.` +- This is a degraded outcome; restore Argo-driven releases for the next milestone + +--- + +## Post-Release Manual Release Record + +Append a manual release record to `CHANGELOG.md`: + +```markdown +## [Unreleased] + +### Release Operations + +- **vX.Y.Z (YYYY-MM-DD)**: Manual release (PB-13 fallback) + - Reason: Argo Workflows in `iad-ci` degraded for + - Release lead: + - All channels completed successfully: PyPI, crates.io, GitHub Release, Cloudflare Pages + - Next milestone: Resume Argo-driven releases +``` + +Commit this change: + +```bash +git add CHANGELOG.md +git commit -m "docs(changelog): record manual release v${VERSION} (PB-13 fallback)" +git push +``` + +--- + +## Failure Modes and Recovery + +### Failure Mode 1: One Triple Build Fails + +**Symptom:** Cross-compilation fails for a single target triple (e.g., `aarch64-apple-darwin` fails, but others succeed). + +**Can the release proceed?** **NO**. All five triples MUST ship. + +**Recovery:** + +1. Check the error message: + - Missing SDK? Install the required toolchain (e.g., osxcross for macOS targets) + - Compilation error? Fix the code and create a new tag + - Timeout? Increase resources and retry + +2. If the issue is environmental (e.g., macOS SDK unavailable), document the missing triple and proceed with a partial release **ONLY IF** the release lead approves. This is a degraded release. + +3. Open a follow-up issue to fix the missing triple for the next release. + +### Failure Mode 2: PyPI Upload Fails Midway + +**Symptom:** `maturin upload` fails partway through uploading wheels. + +**Can the release proceed?** **YES**, to other channels. PyPI can be retried. + +**Recovery:** + +1. Check PyPI status: https://status.pypi.org/ + +2. Retry the upload with `--skip-existing` flag: + ```bash + maturin upload --repository pypi --skip-existing target/wheels/*.whl + ``` + +3. If the failure persists, check the wheel integrity: + ```bash + unzip -l target/wheels/pdftract-*.whl + ``` + +4. Document the failure and proceed to crates.io and GitHub Release. PyPI can be fixed post-release. + +**DO NOT force overwrite** (`--force`) on PyPI. This can corrupt the index. + +### Failure Mode 3: SLSA Attestation Generation Fails + +**Symptom:** The SLSA provenance script errors out or produces invalid JSON. + +**Can the release proceed?** **YES**, but this is a degraded outcome. + +**Recovery:** + +1. Check the error: + - Missing `git` command? Install `git` + - Invalid JSON? Check the Python script syntax + - Missing artifacts? Re-run Step 5 (checksums) + +2. If the failure cannot be resolved, proceed without SLSA attestation. Add a note to the GitHub Release description: + > **Note:** This release was produced via the PB-13 manual fallback. SLSA Level 2 attestation is not available for this release. + +3. Open a follow-up issue to restore SLSA generation for the next release. + +4. Update the CHANGELOG with the degraded outcome: + ```markdown + - **vX.Y.Z (YYYY-MM-DD)**: Manual release (PB-13 fallback, DEGRADED) + - SLSA Level 2 attestation not available + ``` + +--- + +## Idempotency and Safe Re-Run Rules + +Each step in this procedure is designed to be safe to re-run after a transient failure: + +| Step | Safe to Re-Run? | Notes | +|------|-----------------|-------| +| Step 1: Verify Tag | YES | No state changes | +| Step 2: Run Tests | YES | No state changes | +| Step 3: Build Binaries | YES | `cargo build` is idempotent | +| Step 4: Verify Binaries | YES | No state changes | +| Step 5: Generate Checksums | YES | Re-running overwrites `CHECKSUMS.sha256` | +| Step 6: GPG Sign | YES | Re-running overwrites `CHECKSUMS.sha256.asc` | +| Step 7: Build Wheels | YES | `maturin build` is idempotent | +| Step 8: PyPI Upload | YES | Use `--skip-existing` on re-run | +| Step 9: crates.io Publish | NO | Version cannot be overwritten | +| Step 10: GitHub Release | NO | Release cannot be overwritten (must delete and recreate) | +| Step 11: Build mdBook | YES | `mdbook build` is idempotent | +| Step 12: Deploy Docs | YES | Wrangler overwrites the existing deployment | +| Step 13: SLSA Attestation | YES | Re-running overwrites `multiple.intoto.jsonl` | + +**Special Re-Run Cases:** + +- **Step 9 (crates.io):** If publish fails, you cannot retry the same version. You must fix the issue, create a new tag (`vX.Y.Z+1`), and restart the release. + +- **Step 10 (GitHub Release):** If release creation fails, delete the draft release and retry: + ```bash + gh release delete v${VERSION} --yes + gh release create v${VERSION} [...] + ``` + +--- + +## Completion Criteria + +The manual release is **complete** when: + +1. All 10 binary archives are built and available on the GitHub Release. +2. PyPI wheels are published and installable (`pip install pdftract` works). +3. crates.io packages are published and usable (`cargo install pdftract` works). +4. GitHub Release exists with all artifacts, checksums, and GPG signature. +5. Docs are deployed to Cloudflare Pages and accessible. +6. SLSA attestation is attached to the GitHub Release (or degraded outcome is documented). +7. CHANGELOG.md includes the manual release record. + +The manual release is **blocked** when: + +1. Any single triple build fails (all five MUST ship). +2. `cargo test` fails (no broken releases). +3. crates.io publish fails (version collision is a hard stop). +4. GitHub Release creation fails (release must exist for users to download artifacts). + +**On blockage:** Open an incident issue, document the failure, and notify the project lead. The milestone release is delayed until the blocker is resolved. + +--- + +## References + +- **Bead:** `pdftract-4sj0` (this runbook) +- **Plan:** R13 risk register (line 549), PB-13 fallback procedure (line 567) +- **Sibling workflows:** + - `pdftract-release-cascade` (Argo orchestration template) + - `pdftract-build-binaries` (Argo build template) + - `pdftract-crates-publish` (Argo crates.io publish) + - `pdftract-py-ci` (Argo PyPI upload) + - `pdftract-github-release` (Argo GitHub Release creation) + - `pdftract-docs-build` (Argo mdBook deployment) +- **External Secret paths:** + - `rs-manager/iad-ci/pypi/pdftract` + - `rs-manager/iad-ci/crates-io/pdftract` + - `rs-manager/iad-ci/github/pat/pdftract` + - `rs-manager/iad-ci/cloudflare/api_token` + - `rs-manager/iad-ci/gpg/pdftract_signing_key` + +--- + +## Continuity Plan + +This runbook is written for a stranger. If the release lead is unavailable, any engineer with: + +1. Access to the OpenBao secret paths listed above +2. A Linux build environment with the prerequisites installed +3. Git access to `jedarden/pdftract` + +...should be able to execute this procedure and ship the release. + +If you find this runbook unclear or missing steps, open an issue to improve it for the next PB-13 activation.