jedarden/pdftract
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(bf-585u5): add comprehensive git mirroring setup and troubleshooting guide

Browse source 
This commit adds docs/notes/git-mirroring.md documenting:
- Workspace git remote conventions (origin = Forgejo, github = GitHub)
- Forgejo push mirror configuration and API usage
- Verification commands and health checks
- Troubleshooting guide for common mirror issues
- Complete incident history of the 160-commit divergence
- Quick reference commands for daily workflow and recovery

Synthesizes findings from child beads: bf-1t8i9 (diagnosis), bf-1o0la (mirror config), bf-682zv (sync blocker), bf-67zia (remotes fixed).

Acceptance criteria:
- PASS: docs/notes/git-mirroring.md exists and is complete
- PASS: Documents workspace convention for git remotes
- PASS: Documents Forgejo push mirror configuration and API usage
- PASS: Includes troubleshooting section
- PASS: References all child bead note files
- PASS: Committed to git repository

Closes bf-585u5
This commit is contained in:
jedarden 2026-07-05 18:12:51 -04:00
parent 3d02fbbe9b
commit 57988bc970
1 changed files with 411 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

411
docs/notes/git-mirroring.md Normal file
View file

@ -0,0 +1,411 @@
# Git Mirroring Setup and Troubleshooting
**Date:** 2026-07-05
**Purpose:** Comprehensive documentation of git mirroring setup, workspace conventions, and resolution processes for the pdftract repository.
## Table of Contents
- [Workspace Convention](#workspace-convention)
- [Forgejo Push Mirror Configuration](#forgejo-push-mirror-configuration)
- [Verifying Mirror Status](#verifying-mirror-status)
- [Troubleshooting Common Issues](#troubleshooting-common-issues)
- [Incident History: 84-Commit Divergence](#incident-history-84-commit-divergence)
- [References and Related Documentation](#references-and-related-documentation)
---
## Workspace Convention
### Primary vs. Mirror Remotes
The pdftract repository follows a **primary-mirror** git workflow:
| Remote | URL | Role | Usage |
|--------|-----|------|-------|
| `origin` | `https://git.ardenone.com/jedarden/pdftract.git` | **Primary** (Forgejo) | All pushes go here first |
| `github` | `https://github.com/jedarden/pdftract.git` | **Mirror** (GitHub) | Automatic sync from Forgejo |
### Key Principles
1. **Forgejo (`origin`) is the source of truth**
- All commits must be pushed to Forgejo first
- CI/CD runs on Forgejo via Argo Workflows
- GitHub is kept in sync automatically
2. **GitHub is a read-only mirror**
- Used for GitHub Issues, PRs, and visibility
- Never push directly to GitHub unless recovering from a sync issue
- The Forgejo push mirror keeps GitHub up-to-date
3. **Branch tracking must point to `origin`**
```bash
# Correct configuration
git branch --set-upstream-to=origin/main main
# Verify with:
git branch -vv
# Should show: main -> origin/main
```
### Verification Commands
```bash
# Check remote configuration
git remote -v
# Check branch tracking
git branch -vv
# Verify remote fetch URLs
git config --get remote.origin.url
git config --get remote.github.url
```
---
## Forgejo Push Mirror Configuration
### Mirror Setup Details
The Forgejo-to-GitHub push mirror is configured in the Forgejo web UI or API:
| Setting | Value | Purpose |
|---------|-------|---------|
| **Remote Address** | `https://github.com/jedarden/pdftract.git` | GitHub mirror target |
| **Sync Interval** | `10m0s` | Sync every 10 minutes |
| **Sync on Commit** | `true` | Sync immediately after each push |
| **Created** | `2026-05-16T19:51:17Z` | Mirror creation timestamp |
### API Response Example
```json
{
"repo_name": "pdftract",
"remote_name": "remote_mirror_nfF0JdlNzC",
"remote_address": "https://github.com/jedarden/pdftract.git",
"created": "2026-05-16T19:51:17Z",
"last_update": "2026-07-05T21:59:42Z",
"interval": "10m0s",
"sync_on_commit": true,
"last_error": "PushRejected Error: large files detected"
}
```
### How to Check Mirror Status
#### Via Forgejo Web UI
1. Navigate to: `https://git.ardenone.com/jedarden/pdftract/settings/mirrors`
2. Verify the push mirror to GitHub exists
3. Check the "Last Sync" timestamp
4. Review "Last Error" field for sync failures
#### Via Forgejo API
```bash
# Requires Forgejo API token
curl -H "Authorization: token <FORGEJO_TOKEN>" \
https://git.ardenone.com/api/v1/repos/jedarden/pdftract/push_mirrors
```
Expected response:
- Mirror entry for GitHub should exist
- `sync_on_commit` should be `true`
- `interval` should be `10m0s`
- Check `last_error` for any sync failures
---
## Verifying Mirror Status
### Quick Health Check
```bash
# 1. Check if Forgejo and GitHub main branches match
FORGEJO_SHA=$(git ls-remote git.ardenone.com:jedarden/pdftract.git refs/heads/main | awk '{print $1}')
GITHUB_SHA=$(git ls-remote github.com:jedarden/pdftract.git refs/heads/main | awk '{print $1}')
if [ "$FORGEJO_SHA" = "$GITHUB_SHA" ]; then
echo "✓ Mirror is in sync"
else
echo "✗ Mirror is out of sync"
echo "Forgejo: $FORGEJO_SHA"
echo "GitHub: $GITHUB_SHA"
fi
```
### Count Commits Behind
```bash
# Fetch latest from both remotes
git fetch origin
git fetch github
# Count commits behind GitHub
git log --oneline origin/main ^github/main | wc -l
```
### Find Divergence Point
```bash
# Find merge-base (common ancestor)
git merge-base origin/main github/main
# Show when divergence started
git log --oneline --date-order origin/main ^github/main | head -1
```
---
## Troubleshooting Common Issues
### Issue 1: Branch Tracking Wrong Remote
**Symptoms:**
- `git push` goes to GitHub instead of Forgejo
- Commits appear on GitHub but not Forgejo
- Mirror shows old commits
**Diagnosis:**
```bash
git branch -vv
# If main tracks github instead of origin:
# main -> github/main
```
**Fix:**
```bash
git branch --set-upstream-to=origin/main main
git branch -vv # Verify: main -> origin/main
```
**Prevention:**
- Always use `git push origin main` for explicit pushes
- Never use bare `git push` without verifying remote
### Issue 2: Mirror Sync Failing Due to Large Files
**Symptoms:**
- Forgejo has newer commits than GitHub
- Mirror `last_error` shows "large files detected"
- Push to GitHub rejected with `GH001: Large files detected`
**Diagnosis:**
```bash
# Find large files in history
git rev-list --objects --all |
git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
awk '/^blob/ {print substr($0,6)}' |
sort -n -k2 |
tail -20
```
**Fix Options:**
#### Option 1: Git LFS (Recommended)
```bash
# Install git-lfs
sudo apt-get install git-lfs # Debian/Ubuntu
# or
brew install git-lfs # macOS
# Initialize LFS
git lfs install
git lfs track "*.ppm" "*.pdf" "*.bin" "*.psd"
git add .gitattributes
git commit -m "chore: enable git-lfs for large files"
git push origin main
```
#### Option 2: Remove Large Files from History
```bash
# Use git-filter-repo (preferred over filter-branch)
pip install git-filter-repo
git filter-repo --path --1.ppm --invert-paths
git push origin main --force-with-lease
```
⚠️ **Warning:** History rewrites change commit SHAs and require force-pushes.
### Issue 3: Mirror Disabled or Missing
**Symptoms:**
- GitHub never updates despite Forgejo commits
- No mirror entry in Forgejo settings
- `last_update` timestamp is very old
**Diagnosis:**
```bash
# Check via API
curl -H "Authorization: token <TOKEN>" \
https://git.ardenone.com/api/v1/repos/jedarden/pdftract/push_mirrors
```
**Fix:**
1. Navigate to Forgejo repository settings
2. Go to "Mirrors" section
3. Add new push mirror: `https://github.com/jedarden/pdftract.git`
4. Enable "Sync on commit"
5. Set interval to `10m`
### Issue 4: Authentication Issues
**Symptoms:**
- Mirror fails with authentication errors
- `last_error` shows "permission denied"
**Fix:**
1. Verify GitHub credentials in Forgejo mirror settings
2. Use GitHub personal access token with `repo` scope
3. Update mirror credentials in Forgejo web UI
---
## Incident History: 84-Commit Divergence
### Timeline
| Date | Event | Details |
|------|-------|---------|
| 2026-06-01 | Divergence started | Commit `1c6f26ec` was first to diverge |
| 2026-07-05 | Issue discovered | GitHub found to be 160 commits behind Forgejo |
| 2026-07-05 | Root cause identified | Local branch tracked `github` instead of `origin` |
| 2026-07-05 | Mirror config verified | Push mirror exists but blocked by large files |
| 2026-07-05 | Remotes fixed | Branch tracking corrected to `origin/main` |
| 2026-07-05 | Documentation created | This comprehensive guide |
### Root Cause Analysis
The divergence had **two compounding issues**:
1. **Local branch tracking misconfiguration**
- Local `main` branch tracked `github` instead of `origin`
- Regular `git push` went to GitHub (behind) instead of Forgejo (source of truth)
- Manual `git push origin main` required for Forgejo updates
2. **Mirror blocked by large files**
- Forgejo push mirror configured correctly
- Sync failing due to `--1.ppm` (235.13 MB) and `test_parse_simple` (60.74 MB)
- GitHub pre-receive hook rejected all sync attempts
### Blocker Details
**Large files in git history:**
- `--1.ppm`: 235.13 MB (exceeds GitHub's 100 MB hard limit)
- `test_parse_simple`: 60.74 MB (exceeds GitHub's 50 MB recommended limit)
Both files were added and removed in the sync range:
- Added in: `1c6f26ec` fix(bf-4mkhv): clean up unused imports in hash.rs
- Removed in: `007439e7` chore(bf-8031): remove tracked debug/scratch artifacts
**GitHub's behavior:** Even though the files were removed from the working tree, they still exist in git's object database. GitHub checks all objects being pushed, including historical ones.
### Resolution Attempt
Attempted to sync GitHub main to Forgejo main:
```bash
git fetch github
git push github main
```
**Result:** Rejected by GitHub pre-receive hook:
```
remote: error: File --1.ppm is 235.13 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected.
```
### Recommended Resolution Path
1. **Set up Git LFS** on both repositories
2. **Migrate large files** to LFS tracking
3. **Push to Forgejo** to propagate LFS pointers
4. **Verify mirror sync** succeeds on next push
5. **Confirm GitHub** matches Forgejo
### Final Commit Range
- **Forgejo/main**: `02bfffef` docs(bf-1o0la): document Forgejo push mirror configuration and status
- **GitHub/main**: `88b4f0da` fix(pdftract-2rc4): fix CI schema gate script and add verification note
- **Commits behind**: 160 (updated from initial 84 estimate during investigation)
- **Merge-base**: `88b4f0da276c7257ade02d3cecfaeb09f7881acc`
---
## References and Related Documentation
### Child Bead Documentation
This comprehensive document synthesizes findings from four child beads:
1. **[Diagnosis (bf-1t8i9)](notes/bf-1t8i9-diagnosis.md)**
- Initial divergence discovery and state assessment
- Identified local branch tracking misconfiguration
- Documented 158-commit divergence (updated to 160 during investigation)
2. **[Mirror Configuration (bf-1o0la)](notes/bf-1o0la-mirror-config.md)**
- Forgejo push mirror API verification
- Confirmed mirror exists and is configured correctly
- Identified large file blocker preventing sync
3. **[Sync Verification (bf-682zv)](notes/bf-682zv.md)**
- Attempted GitHub sync from Forgejo
- Documented large file rejection by GitHub
- Recommended Git LFS resolution path
4. **[Remotes Fixed (bf-67zia)](notes/bf-67zia-remotes-fixed.md)**
- Corrected local branch tracking to `origin/main`
- Verified remote configuration follows workspace convention
- Confirmed both remotes are accessible
### Parent Bead
- **[Genesis: Git Mirror Setup and Resolution (bf-320gz)]** - Parent coordination bead for this work
### External References
- **Forgejo API Documentation**: https://forgejo.jedarden.com/api/v1/
- **GitHub Large File Storage**: https://git-lfs.github.com/
- **GitHub File Size Limits**: https://docs.github.com/en/repositories/working-with-files/using-files/large-files
---
## Quick Reference Commands
### Daily Workflow
```bash
# 1. Pull latest from Forgejo
git pull origin main
# 2. Do your work
git commit -m "feat: something"
# 3. Push to Forgejo (source of truth)
git push origin main
# 4. Verify mirror status (optional)
# Check GitHub web UI or use API:
curl -H "Authorization: token <TOKEN>" \
https://git.ardenone.com/api/v1/repos/jedarden/pdftract/push_mirrors
```
### Recovery Commands
```bash
# If GitHub gets out of sync:
git fetch github
git push github origin/main:main --force-with-lease
# If branch tracking gets messed up:
git branch --set-upstream-to=origin/main main
# If mirror is disabled or missing:
# Check Forgejo web UI: https://git.ardenone.com/jedarden/pdftract/settings/mirrors
```
---
**Document Status:** ✅ Complete
**Last Updated:** 2026-07-05
**Maintainer:** jedarden
**Version:** 1.0