pdftract/docs/operations/manual-release.md

Manual Release Runbook (PB-13)

Purpose: This runbook is the fallback procedure for reproducing the milestone release locally when the Argo Workflows runner in iad-ci is degraded or unavailable for a prolonged window. Per PB-13 (plan line 567), this is the R13 mitigation strategy.

Trigger: Argo Workflows in iad-ci cluster is unavailable for > 4 hours during a release window, or the release lead determines that automated release cannot proceed.

Executor: Release lead or designated engineer with access to OpenBao secrets and cross-compilation toolchains.

---

Prerequisites

Hardware and OS

• A clean Linux build environment (preferably the same Hetzner host that runs Argo; developer laptops are acceptable but NOT preferred due to build environment differences)
• At least 100 GB free disk space for Rust target directories and cross-compilation
• Stable internet connection for fetching crates and pushing artifacts

Installed Tools

# Verify all required tools are present
cargo --version        # >= 1.83
rustup --version
maturin --version      # >= 1.0
mdbook --version       # >= 0.12
jq --version
gh --version           # GitHub CLI
git --version
shasum --version       # or sha256sum
gpg --version          # for signing checksums
wrangler --version     # Cloudflare Pages CLI
python3 --version      # >= 3.11

Cross-Compilation Toolchains

All five target triples MUST be available:

Target Triple	Toolchain Install Command
x86_64-unknown-linux-musl	rustup target add x86_64-unknown-linux-musl
aarch64-unknown-linux-musl	rustup target add aarch64-unknown-linux-musl
x86_64-apple-darwin	rustup target add x86_64-apple-darwin
aarch64-apple-darwin	rustup target add aarch64-apple-darwin
x86_64-pc-windows-gnu	rustup target add x86_64-pc-windows-gnu

Install cross for cross-compilation:

cargo install cross --locked

OpenBao Secrets Access

The following secrets MUST be accessible via OpenBao (same ESO-style read access that Argo uses):

Secret	OpenBao Path	Purpose
PyPI token	rs-manager/iad-ci/pypi/pdftract	Upload Python wheels to PyPI
crates.io token	rs-manager/iad-ci/crates-io/pdftract	Publish Rust crates to crates.io
GitHub Release token	rs-manager/iad-ci/github/pat/pdftract	Create GitHub Release with artifacts
Cloudflare token	rs-manager/iad-ci/cloudflare/api_token	Deploy docs to Cloudflare Pages
GPG signing key	rs-manager/iad-ci/gpg/pdftract_signing_key	Sign checksums for release artifacts

DO NOT copy secrets to disk on the developer machine. Fetch them via environment variables or inject them directly into commands.

---

Step-by-Step Release Procedure

Step 1: Verify the Tag

Ensure the tag matches the planned version and is annotated (not lightweight):

git fetch --tags
git describe --exact-match

Expected output: vX.Y.Z where X.Y.Z matches the planned release version.

If the command fails, you are not on a tagged commit. Checkout the correct tag:

git checkout vX.Y.Z

Verify the tag is annotated:

git tag -l vX.Y.Z -n9

Annotated tags show a message; lightweight tags do not. Only annotated tags are acceptable for releases.

Step 2: Run Full CI Suite Locally

Before building, verify the code passes all tests:

cd /path/to/pdftract
cargo test --workspace --all-features

Expected: All tests pass (exit code 0).

If tests fail, DO NOT PROCEED. Fix the failures and create a new tag.

Step 3: Cross-Compile Binaries for All Triples

Build all 10 binary archives (5 triples × 2 feature variants: default, full):

# Set environment variables for reproducible builds
export SOURCE_DATE_EPOCH=$(git show -s --format=%ct HEAD)
export CARGO_TARGET_DIR=./target

# Build matrix
TARGETS=(
  "x86_64-unknown-linux-musl"
  "aarch64-unknown-linux-musl"
  "x86_64-apple-darwin"
  "aarch64-apple-darwin"
  "x86_64-pc-windows-gnu"
)

FEATURES=(
  "default"
  "full"
)

for TARGET in "${TARGETS[@]}"; do
  for FEATURES in "${FEATURES[@]}"; do
    echo "Building ${TARGET} (features: ${FEATURES})..."

    FEATURE_FLAG=""
    if [ "${FEATURES}" != "default" ]; then
      FEATURE_FLAG="--features ${FEATURES}"
    fi

    cross build --release --target ${TARGET} ${FEATURE_FLAG} --locked --frozen

    # Strip the binary
    BINARY_PATH="target/${TARGET}/release/pdftract"
    if [ "${TARGET}" = "x86_64-pc-windows-gnu" ]; then
      BINARY_PATH="target/${TARGET}/release/pdftract.exe"
    fi

    # Strip (use appropriate strip command for target)
    if [[ "${TARGET}" == *-apple-darwin ]]; then
      # macOS targets use x86_64-apple-darwin-strip or aarch64-apple-darwin-strip
      ${TARGET%-unknown}-strip "${BINARY_PATH}"
    else
      strip "${BINARY_PATH}"
    fi
  done
done

Expected: All 10 builds complete without errors.

Step 4: Verify Each Binary Works

Quick smoke test on the native Linux binary:

./target/x86_64-unknown-linux-musl/release/pdftract --version

Expected: Version string matches the tag.

For cross-compiled binaries, verify the file output exists and is non-empty:

ls -lh target/*/release/pdftract*

Expected: All 10 binaries are present and non-zero size.

Step 5: Generate SHA-256 Checksums

Create checksums for all binary archives:

VERSION=$(git describe --exact-match | sed 's/^v//')
DIST_DIR="./dist"
mkdir -p ${DIST_DIR}

# Generate archives and checksums in one pass
cd ${DIST_DIR}
for TARGET in "${TARGETS[@]}"; do
  for FEATURES in "${FEATURES[@]}"; do
    ARCHIVE_NAME="pdftract"
    if [ "${FEATURES}" != "default" ]; then
      ARCHIVE_NAME="pdftract-${FEATURES}"
    fi
    ARCHIVE_DIR="${ARCHIVE_NAME}-v${VERSION}-${TARGET}"
    mkdir -p "${ARCHIVE_DIR}"

    # Copy binary
    if [ "${TARGET}" = "x86_64-pc-windows-gnu" ]; then
      cp "../target/${TARGET}/release/pdftract.exe" "${ARCHIVE_DIR}/"
    else
      cp "../target/${TARGET}/release/pdftract" "${ARCHIVE_DIR}/"
    fi

    # Copy license files
    cp ../LICENSE-MIT "${ARCHIVE_DIR}/"
    cp ../LICENSE-APACHE "${ARCHIVE_DIR}/"
    cp ../README.md "${ARCHIVE_DIR}/"

    # Extract CHANGELOG excerpt
    python3 <<EOF
import re
version = "${VERSION}".replace(".", r"\.")
with open("../CHANGELOG.md", "r") as f:
    content = f.read()
pattern = rf"^## \[{version}\](?:.*?)(?=^## |\Z)"
match = re.search(pattern, content, re.MULTILINE | re.DOTALL)
if match:
    with open("${ARCHIVE_DIR}/CHANGELOG.md", "w") as out:
        out.write(f"## [{version}]\n")
        out.write(match.group(0))
else:
    with open("${ARCHIVE_DIR}/CHANGELOG.md", "w") as out:
        out.write(f"## [{version}]\n\nSee https://github.com/jedarden/pdftract/releases/tag/v${VERSION}\n")
EOF

    # Create archive
    if [ "${TARGET}" = "x86_64-pc-windows-gnu" ]; then
      ARCHIVE_FILE="${ARCHIVE_NAME}-v${VERSION}-${TARGET}.zip"
      zip -r "${ARCHIVE_FILE}" "${ARCHIVE_DIR}"
    else
      ARCHIVE_FILE="${ARCHIVE_NAME}-v${VERSION}-${TARGET}.tar.gz"
      tar czf "${ARCHIVE_FILE}" "${ARCHIVE_DIR}"
    fi
  done
done

# Generate checksums
shasum -a 256 pdftract*.tar.gz pdftract*.zip > CHECKSUMS.sha256

Expected: CHECKSUMS.sha256 contains SHA-256 hashes for all 10 archives.

Step 6: GPG Sign the Checksums File

Sign the checksums file with the GPG key:

# Fetch GPG key from OpenBao (example via environment variable)
export GPG_KEY=$(bao read -field=private_key rs-manager/iad-ci/gpg/pdftract_signing_key)

# Import key temporarily (DO NOT persist to disk)
gpg --import <<< "${GPG_KEY}"

# Sign the checksums
gpg --detach-sign --digest-algo SHA256 --armor --output CHECKSUMS.sha256.asc CHECKSUMS.sha256

# Verify the signature
gpg --verify CHECKSUMS.sha256.asc CHECKSUMS.sha256

Expected: gpg: Good signature from the pdftract signing key.

Step 7: Build Python Wheels

Build wheels for all three platforms (Linux, macOS, Windows):

# Install maturin if not present
cargo install maturin --locked

# Build wheels (this uses cross for non-native platforms)
maturin build --release --strip

# Wheels are in target/wheels/
ls -lh target/wheels/

Expected: Wheels for pdftract and pdftract-core are present in target/wheels/.

Step 8: Upload Wheels to PyPI

# Fetch PyPI token from OpenBao
export PYPI_TOKEN=$(bao read -field=token rs-manager/iad-ci/pypi/pdftract)

# Upload using maturin (token is passed via environment variable)
echo "pypi-token = ${PYPI_TOKEN}" > ~/.pypirc
maturin upload --repository pypi target/wheels/*.whl

Expected: All wheels upload successfully (HTTP 200 responses).

Failure mode - PyPI upload fails midway:

• If upload fails mid-stream, DO NOT force overwrite (--skip-existing is acceptable for re-running)
• Check PyPI status page: https://status.pypi.org/
• If PyPI is down, retry in 5 minutes
• If a specific wheel fails, check the wheel with unzip -l and verify it's not corrupted
• Document the failure and proceed to other channels (crates.io, GitHub Release can still succeed)

Step 9: Publish to crates.io

Publish pdftract-core first, then pdftract-cli (dependency order matters):

# Fetch crates.io token from OpenBao
export CARGO_REGISTRY_TOKEN=$(bao read -field=token rs-manager/iad-ci/crates-io/pdftract)

# Publish pdftract-core first
cargo publish --package pdftract-core

# Wait for index propagation (crates.io can take 1-5 minutes)
echo "Waiting for pdftract-core to appear in the index..."
sleep 60  # Wait at least 1 minute

# Verify pdftract-core is available
cargo search pdftract-core --limit 1

# Publish pdftract-cli
cargo publish --package pdftract-cli

Expected: Both crates publish successfully.

Failure mode - crates.io publish fails:

• Check if the version already exists (this is a hard error from crates.io)
• If it's a network error, retry after 60 seconds
• If it's a validation error, fix Cargo.toml and create a new tag
• Document the failure and proceed to other channels

Step 10: Create GitHub Release

VERSION=$(git describe --exact-match | sed 's/^v//')
REPO="jedarden/pdftract"

# Fetch GitHub token from OpenBao
export GH_TOKEN=$(bao read -field=token rs-manager/iad-ci/github/pat/pdftract)

# Create release with all artifacts
gh release create v${VERSION} \
  ./dist/pdftract*.tar.gz \
  ./dist/pdftract*.zip \
  ./dist/CHECKSUMS.sha256 \
  ./dist/CHECKSUMS.sha256.asc \
  --title "pdftract v${VERSION}" \
  --notes "Release notes: https://github.com/${REPO}/blob/v${VERSION}/CHANGELOG.md"

Expected: GitHub Release created at https://github.com/jedarden/pdftract/releases/tag/v${VERSION}.

Step 11: Build mdBook

cd docs/user-docs
mdbook build

Expected: docs/user-docs/book/ directory contains the built HTML site.

Step 12: Deploy Docs to Cloudflare Pages

# Fetch Cloudflare token from OpenBao
export CLOUDFLARE_API_TOKEN=$(bao read -field=token rs-manager/iad-ci/cloudflare/api_token)

# Deploy using wrangler
cd docs/user-docs
wrangler pages publish build --project-name=pdftract-docs

Expected: Docs deployed to https://pdftract.com (or the configured Pages domain).

Failure mode - wrangler publish fails:

• Check Cloudflare status: https://www.cloudflarestatus.com/
• Verify the project name matches your Cloudflare Pages project
• If the build directory is missing, re-run mdbook build
• Document the failure and proceed (docs can be deployed later)

Step 13: Generate SLSA Level 2 Attestation

Generate SLSA Provenance v1.0 predicate for the release artifacts:

python3 <<'EOF'
import hashlib
import json
import subprocess
from datetime import datetime, timezone

VERSION = "${VERSION}"
COMMIT_SHA = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip()
TAG = "v${VERSION}"
REPO = "jedarden/pdftract"

# Compute SOURCE_DATE_EPOCH for reproducibility
SOURCE_DATE_EPOCH = subprocess.check_output(
    ["git", "show", "-s", "--format=%ct", "HEAD"]
).decode().strip()

# Collect all binary archives
subjects = []
for f in ["pdftract*.tar.gz", "pdftract*.zip"]:
    for path in glob.glob(f):
        sha256 = hashlib.sha256()
        with open(path, "rb") as fp:
            for chunk in iter(lambda: fp.read(65536), b""):
                sha256.update(chunk)
        digest = sha256.hexdigest()
        subjects.append({
            "name": os.path.basename(path),
            "digest": {"sha256": digest}
        })

# Build invocation ID
invocation_id = f"manual-release-{COMMIT_SHA}-{TAG}"

# Build timestamp
started_on = datetime.fromtimestamp(int(SOURCE_DATE_EPOCH), tz=timezone.utc).isoformat()
finished_on = datetime.now(timezone.utc).isoformat()

# Construct SLSA Provenance v1.0 predicate
provenance = {
    "_type": "https://in-toto.io/Statement/v1",
    "subject": subjects,
    "predicateType": "https://slsa.dev/provenance/v1.0",
    "predicate": {
        "buildDefinition": {
            "buildType": "https://github.com/jedarden/pdftract/releases/manual",
            "externalParameters": {
                "tag": TAG,
                "version": VERSION,
                "commit": COMMIT_SHA,
                "source": f"https://github.com/{REPO}"
            },
            "internalParameters": {
                "workflow": "manual-release",
                "targetTriples": [
                    "x86_64-unknown-linux-musl",
                    "aarch64-unknown-linux-musl",
                    "x86_64-apple-darwin",
                    "aarch64-apple-darwin",
                    "x86_64-pc-windows-gnu"
                ]
            },
            "resolvedDependencies": [
                {
                    "uri": f"git+https://github.com/{REPO}@{COMMIT_SHA}",
                    "digest": {"gitCommit": COMMIT_SHA}
                }
            ]
        },
        "runDetails": {
            "builder": {
                "id": "https://github.com/jedarden/pdftract/releases/manual",
                "builderDependencies": [],
                "version": {
                    "manual": "PB-13-fallback"
                }
            },
            "metadata": {
                "invocationId": invocation_id,
                "startedOn": started_on,
                "finishedOn": finished_on
            },
            "byproducts": []
        }
    }
}

# Write to multiple.intoto.jsonl
with open("multiple.intoto.jsonl", "w") as f:
    f.write(json.dumps(provenance) + "\n")

print(f"SLSA provenance written to multiple.intoto.jsonl")
print(f"Subjects: {len(subjects)}")
EOF

Upload the attestation to the GitHub Release:

gh release upload v${VERSION} multiple.intoto.jsonl

Expected: multiple.intoto.jsonl appears in the GitHub Release assets.

Failure mode - SLSA attestation generation fails:

• The release can still ship without SLSA attestation, but document this in the CHANGELOG
• Add a note: This release was produced via the PB-13 manual fallback; SLSA Level 2 attestation is not available.
• This is a degraded outcome; restore Argo-driven releases for the next milestone

---

Post-Release Manual Release Record

Append a manual release record to CHANGELOG.md:

## [Unreleased]

### Release Operations

- **vX.Y.Z (YYYY-MM-DD)**: Manual release (PB-13 fallback)
  - Reason: Argo Workflows in `iad-ci` degraded for <duration>
  - Release lead: <your initials>
  - All channels completed successfully: PyPI, crates.io, GitHub Release, Cloudflare Pages
  - Next milestone: Resume Argo-driven releases

Commit this change:

git add CHANGELOG.md
git commit -m "docs(changelog): record manual release v${VERSION} (PB-13 fallback)"
git push

---

Failure Modes and Recovery

Failure Mode 1: One Triple Build Fails

Symptom: Cross-compilation fails for a single target triple (e.g., aarch64-apple-darwin fails, but others succeed).

Can the release proceed? NO. All five triples MUST ship.

Recovery:

1. Check the error message:

   • Missing SDK? Install the required toolchain (e.g., osxcross for macOS targets)
   • Compilation error? Fix the code and create a new tag
   • Timeout? Increase resources and retry

2. If the issue is environmental (e.g., macOS SDK unavailable), document the missing triple and proceed with a partial release ONLY IF the release lead approves. This is a degraded release.

3. Open a follow-up issue to fix the missing triple for the next release.

Failure Mode 2: PyPI Upload Fails Midway

Symptom: maturin upload fails partway through uploading wheels.

Can the release proceed? YES, to other channels. PyPI can be retried.

Recovery:

1. Check PyPI status: https://status.pypi.org/

2. Retry the upload with --skip-existing flag:

   maturin upload --repository pypi --skip-existing target/wheels/*.whl
   

3. If the failure persists, check the wheel integrity:

   unzip -l target/wheels/pdftract-*.whl
   

4. Document the failure and proceed to crates.io and GitHub Release. PyPI can be fixed post-release.

DO NOT force overwrite (--force) on PyPI. This can corrupt the index.

Failure Mode 3: SLSA Attestation Generation Fails

Symptom: The SLSA provenance script errors out or produces invalid JSON.

Can the release proceed? YES, but this is a degraded outcome.

Recovery:

1. Check the error:

   • Missing git command? Install git
   • Invalid JSON? Check the Python script syntax
   • Missing artifacts? Re-run Step 5 (checksums)

2. If the failure cannot be resolved, proceed without SLSA attestation. Add a note to the GitHub Release description:

   Note: This release was produced via the PB-13 manual fallback. SLSA Level 2 attestation is not available for this release.

3. Open a follow-up issue to restore SLSA generation for the next release.

4. Update the CHANGELOG with the degraded outcome:

   - **vX.Y.Z (YYYY-MM-DD)**: Manual release (PB-13 fallback, DEGRADED)
     - SLSA Level 2 attestation not available
   

---

Idempotency and Safe Re-Run Rules

Each step in this procedure is designed to be safe to re-run after a transient failure:

Step	Safe to Re-Run?	Notes
Step 1: Verify Tag	YES	No state changes
Step 2: Run Tests	YES	No state changes
Step 3: Build Binaries	YES	cargo build is idempotent
Step 4: Verify Binaries	YES	No state changes
Step 5: Generate Checksums	YES	Re-running overwrites CHECKSUMS.sha256
Step 6: GPG Sign	YES	Re-running overwrites CHECKSUMS.sha256.asc
Step 7: Build Wheels	YES	maturin build is idempotent
Step 8: PyPI Upload	YES	Use --skip-existing on re-run
Step 9: crates.io Publish	NO	Version cannot be overwritten
Step 10: GitHub Release	NO	Release cannot be overwritten (must delete and recreate)
Step 11: Build mdBook	YES	mdbook build is idempotent
Step 12: Deploy Docs	YES	Wrangler overwrites the existing deployment
Step 13: SLSA Attestation	YES	Re-running overwrites multiple.intoto.jsonl

Special Re-Run Cases:

• Step 9 (crates.io): If publish fails, you cannot retry the same version. You must fix the issue, create a new tag (vX.Y.Z+1), and restart the release.

• Step 10 (GitHub Release): If release creation fails, delete the draft release and retry:

  gh release delete v${VERSION} --yes
  gh release create v${VERSION} [...]
  

---

Completion Criteria

The manual release is complete when:

1. All 10 binary archives are built and available on the GitHub Release.
2. PyPI wheels are published and installable (pip install pdftract works).
3. crates.io packages are published and usable (cargo install pdftract works).
4. GitHub Release exists with all artifacts, checksums, and GPG signature.
5. Docs are deployed to Cloudflare Pages and accessible.
6. SLSA attestation is attached to the GitHub Release (or degraded outcome is documented).
7. CHANGELOG.md includes the manual release record.

The manual release is blocked when:

1. Any single triple build fails (all five MUST ship).
2. cargo test fails (no broken releases).
3. crates.io publish fails (version collision is a hard stop).
4. GitHub Release creation fails (release must exist for users to download artifacts).

On blockage: Open an incident issue, document the failure, and notify the project lead. The milestone release is delayed until the blocker is resolved.

---

References

• Bead: pdftract-4sj0 (this runbook)
• Plan: R13 risk register (line 549), PB-13 fallback procedure (line 567)
• Sibling workflows:
  • pdftract-release-cascade (Argo orchestration template)
  • pdftract-build-binaries (Argo build template)
  • pdftract-crates-publish (Argo crates.io publish)
  • pdftract-py-ci (Argo PyPI upload)
  • pdftract-github-release (Argo GitHub Release creation)
  • pdftract-docs-build (Argo mdBook deployment)
• External Secret paths:
  • rs-manager/iad-ci/pypi/pdftract
  • rs-manager/iad-ci/crates-io/pdftract
  • rs-manager/iad-ci/github/pat/pdftract
  • rs-manager/iad-ci/cloudflare/api_token
  • rs-manager/iad-ci/gpg/pdftract_signing_key

---

Continuity Plan

This runbook is written for a stranger. If the release lead is unavailable, any engineer with:

1. Access to the OpenBao secret paths listed above
2. A Linux build environment with the prerequisites installed
3. Git access to jedarden/pdftract

...should be able to execute this procedure and ship the release.

If you find this runbook unclear or missing steps, open an issue to improve it for the next PB-13 activation.