pdftract/docs/notes/git-mirroring.md

Git Mirroring Setup and Troubleshooting

Date: 2026-07-05
Purpose: Comprehensive documentation of git mirroring setup, workspace conventions, and resolution processes for the pdftract repository.

Table of Contents

• Workspace Convention
• Forgejo Push Mirror Configuration
• Verifying Mirror Status
• Troubleshooting Common Issues
• Incident History: 84-Commit Divergence
• References and Related Documentation

---

Workspace Convention

Primary vs. Mirror Remotes

The pdftract repository follows a primary-mirror git workflow:

Remote	URL	Role	Usage
origin	https://git.ardenone.com/jedarden/pdftract.git	Primary (Forgejo)	All pushes go here first
github	https://github.com/jedarden/pdftract.git	Mirror (GitHub)	Automatic sync from Forgejo

Key Principles

1. Forgejo (origin) is the source of truth

   • All commits must be pushed to Forgejo first
   • CI/CD runs on Forgejo via Argo Workflows
   • GitHub is kept in sync automatically

2. GitHub is a read-only mirror

   • Used for GitHub Issues, PRs, and visibility
   • Never push directly to GitHub unless recovering from a sync issue
   • The Forgejo push mirror keeps GitHub up-to-date

3. Branch tracking must point to origin

   # Correct configuration
   git branch --set-upstream-to=origin/main main
   
   # Verify with:
   git branch -vv
   # Should show: main -> origin/main
   

Verification Commands

# Check remote configuration
git remote -v

# Check branch tracking
git branch -vv

# Verify remote fetch URLs
git config --get remote.origin.url
git config --get remote.github.url

---

Forgejo Push Mirror Configuration

Mirror Setup Details

The Forgejo-to-GitHub push mirror is configured in the Forgejo web UI or API:

Setting	Value	Purpose
Remote Address	https://github.com/jedarden/pdftract.git	GitHub mirror target
Sync Interval	10m0s	Sync every 10 minutes
Sync on Commit	true	Sync immediately after each push
Created	2026-05-16T19:51:17Z	Mirror creation timestamp

API Response Example

{
  "repo_name": "pdftract",
  "remote_name": "remote_mirror_nfF0JdlNzC",
  "remote_address": "https://github.com/jedarden/pdftract.git",
  "created": "2026-05-16T19:51:17Z",
  "last_update": "2026-07-05T21:59:42Z",
  "interval": "10m0s",
  "sync_on_commit": true,
  "last_error": "PushRejected Error: large files detected"
}

How to Check Mirror Status

Via Forgejo Web UI

1. Navigate to: https://git.ardenone.com/jedarden/pdftract/settings/mirrors
2. Verify the push mirror to GitHub exists
3. Check the "Last Sync" timestamp
4. Review "Last Error" field for sync failures

Via Forgejo API

# Requires Forgejo API token
curl -H "Authorization: token <FORGEJO_TOKEN>" \
  https://git.ardenone.com/api/v1/repos/jedarden/pdftract/push_mirrors

Expected response:

• Mirror entry for GitHub should exist
• sync_on_commit should be true
• interval should be 10m0s
• Check last_error for any sync failures

---

Verifying Mirror Status

Quick Health Check

# 1. Check if Forgejo and GitHub main branches match
FORGEJO_SHA=$(git ls-remote git.ardenone.com:jedarden/pdftract.git refs/heads/main | awk '{print $1}')
GITHUB_SHA=$(git ls-remote github.com:jedarden/pdftract.git refs/heads/main | awk '{print $1}')

if [ "$FORGEJO_SHA" = "$GITHUB_SHA" ]; then
    echo "✓ Mirror is in sync"
else
    echo "✗ Mirror is out of sync"
    echo "Forgejo: $FORGEJO_SHA"
    echo "GitHub: $GITHUB_SHA"
fi

Count Commits Behind

# Fetch latest from both remotes
git fetch origin
git fetch github

# Count commits behind GitHub
git log --oneline origin/main ^github/main | wc -l

Find Divergence Point

# Find merge-base (common ancestor)
git merge-base origin/main github/main

# Show when divergence started
git log --oneline --date-order origin/main ^github/main | head -1

---

Troubleshooting Common Issues

Issue 1: Branch Tracking Wrong Remote

Symptoms:

• git push goes to GitHub instead of Forgejo
• Commits appear on GitHub but not Forgejo
• Mirror shows old commits

Diagnosis:

git branch -vv
# If main tracks github instead of origin:
# main -> github/main

Fix:

git branch --set-upstream-to=origin/main main
git branch -vv  # Verify: main -> origin/main

Prevention:

• Always use git push origin main for explicit pushes
• Never use bare git push without verifying remote

Issue 2: Mirror Sync Failing Due to Large Files

Symptoms:

• Forgejo has newer commits than GitHub
• Mirror last_error shows "large files detected"
• Push to GitHub rejected with GH001: Large files detected

Diagnosis:

# Find large files in history
git rev-list --objects --all |
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
  awk '/^blob/ {print substr($0,6)}' |
  sort -n -k2 |
  tail -20

Fix Options:

Option 1: Git LFS (Recommended)

# Install git-lfs
sudo apt-get install git-lfs  # Debian/Ubuntu
# or
brew install git-lfs  # macOS

# Initialize LFS
git lfs install
git lfs track "*.ppm" "*.pdf" "*.bin" "*.psd"
git add .gitattributes
git commit -m "chore: enable git-lfs for large files"
git push origin main

Option 2: Remove Large Files from History

# Use git-filter-repo (preferred over filter-branch)
pip install git-filter-repo
git filter-repo --path --1.ppm --invert-paths
git push origin main --force-with-lease

⚠️ Warning: History rewrites change commit SHAs and require force-pushes.

Issue 3: Mirror Disabled or Missing

Symptoms:

• GitHub never updates despite Forgejo commits
• No mirror entry in Forgejo settings
• last_update timestamp is very old

Diagnosis:

# Check via API
curl -H "Authorization: token <TOKEN>" \
  https://git.ardenone.com/api/v1/repos/jedarden/pdftract/push_mirrors

Fix:

1. Navigate to Forgejo repository settings
2. Go to "Mirrors" section
3. Add new push mirror: https://github.com/jedarden/pdftract.git
4. Enable "Sync on commit"
5. Set interval to 10m

Issue 4: Authentication Issues

Symptoms:

• Mirror fails with authentication errors
• last_error shows "permission denied"

Fix:

1. Verify GitHub credentials in Forgejo mirror settings
2. Use GitHub personal access token with repo scope
3. Update mirror credentials in Forgejo web UI

---

Incident History: 84-Commit Divergence

Timeline

Date	Event	Details
2026-06-01	Divergence started	Commit 1c6f26ec was first to diverge
2026-07-05	Issue discovered	GitHub found to be 160 commits behind Forgejo
2026-07-05	Root cause identified	Local branch tracked github instead of origin
2026-07-05	Mirror config verified	Push mirror exists but blocked by large files
2026-07-05	Remotes fixed	Branch tracking corrected to origin/main
2026-07-05	Documentation created	This comprehensive guide

Root Cause Analysis

The divergence had two compounding issues:

1. Local branch tracking misconfiguration

   • Local main branch tracked github instead of origin
   • Regular git push went to GitHub (behind) instead of Forgejo (source of truth)
   • Manual git push origin main required for Forgejo updates

2. Mirror blocked by large files

   • Forgejo push mirror configured correctly
   • Sync failing due to --1.ppm (235.13 MB) and test_parse_simple (60.74 MB)
   • GitHub pre-receive hook rejected all sync attempts

Blocker Details

Large files in git history:

• --1.ppm: 235.13 MB (exceeds GitHub's 100 MB hard limit)
• test_parse_simple: 60.74 MB (exceeds GitHub's 50 MB recommended limit)

Both files were added and removed in the sync range:

• Added in: 1c6f26ec fix(bf-4mkhv): clean up unused imports in hash.rs
• Removed in: 007439e7 chore(bf-8031): remove tracked debug/scratch artifacts

GitHub's behavior: Even though the files were removed from the working tree, they still exist in git's object database. GitHub checks all objects being pushed, including historical ones.

Resolution Attempt

Attempted to sync GitHub main to Forgejo main:

git fetch github
git push github main

Result: Rejected by GitHub pre-receive hook:

remote: error: File --1.ppm is 235.13 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected.

Recommended Resolution Path

1. Set up Git LFS on both repositories
2. Migrate large files to LFS tracking
3. Push to Forgejo to propagate LFS pointers
4. Verify mirror sync succeeds on next push
5. Confirm GitHub matches Forgejo

Final Commit Range

• Forgejo/main: 02bfffef docs(bf-1o0la): document Forgejo push mirror configuration and status
• GitHub/main: 88b4f0da fix(pdftract-2rc4): fix CI schema gate script and add verification note
• Commits behind: 160 (updated from initial 84 estimate during investigation)
• Merge-base: 88b4f0da276c7257ade02d3cecfaeb09f7881acc

---

References and Related Documentation

Child Bead Documentation

This comprehensive document synthesizes findings from four child beads:

1. Diagnosis (bf-1t8i9)

   • Initial divergence discovery and state assessment
   • Identified local branch tracking misconfiguration
   • Documented 158-commit divergence (updated to 160 during investigation)

2. Mirror Configuration (bf-1o0la)

   • Forgejo push mirror API verification
   • Confirmed mirror exists and is configured correctly
   • Identified large file blocker preventing sync

3. Sync Verification (bf-682zv)

   • Attempted GitHub sync from Forgejo
   • Documented large file rejection by GitHub
   • Recommended Git LFS resolution path

4. Remotes Fixed (bf-67zia)

   • Corrected local branch tracking to origin/main
   • Verified remote configuration follows workspace convention
   • Confirmed both remotes are accessible

Parent Bead

• [Genesis: Git Mirror Setup and Resolution (bf-320gz)] - Parent coordination bead for this work

External References

• Forgejo API Documentation: https://forgejo.jedarden.com/api/v1/
• GitHub Large File Storage: https://git-lfs.github.com/
• GitHub File Size Limits: https://docs.github.com/en/repositories/working-with-files/using-files/large-files

---

Quick Reference Commands

Daily Workflow

# 1. Pull latest from Forgejo
git pull origin main

# 2. Do your work
git commit -m "feat: something"

# 3. Push to Forgejo (source of truth)
git push origin main

# 4. Verify mirror status (optional)
# Check GitHub web UI or use API:
curl -H "Authorization: token <TOKEN>" \
  https://git.ardenone.com/api/v1/repos/jedarden/pdftract/push_mirrors

Recovery Commands

# If GitHub gets out of sync:
git fetch github
git push github origin/main:main --force-with-lease

# If branch tracking gets messed up:
git branch --set-upstream-to=origin/main main

# If mirror is disabled or missing:
# Check Forgejo web UI: https://git.ardenone.com/jedarden/pdftract/settings/mirrors

---

Document Status: ✅ Complete
Last Updated: 2026-07-05
Maintainer: jedarden
Version: 1.0