jedarden/ai-code-battle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(bf-2ws): add notes on acb-index-builder OOMKill fix

Browse source
This commit is contained in:
jedarden 2026-06-25 06:55:15 -04:00
parent 1b399a1e55
commit f665ce0d04
1 changed files with 35 additions and 57 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

92
notes/bf-2ws.md
View file

@ -1,73 +1,51 @@
# Fix acb-index-builder OOMKill (bf-2ws)
# Fix acb-index-builder CrashLoopBackOff (Bead bf-2ws)
## Problem
acb-index-builder pod in iad-acb cluster was in CrashLoopBackOff for 45 days with 4,713 restarts. The pod crashed silently after copying web assets - last log line was always:
acb-index-builder (iad-acb cluster) was in CrashLoopBackOff for 45 days with 4713 restarts. The pod crashed silently after the log line:
```
{"msg":"Copied web assets to output directory","source":"/app/web/dist"}
```
No error was logged, indicating either:
1. A panic in fetchAllData or generateAllIndexes (unlikely - defer recover() logs panics)
2. OOMKill (most likely - kernel SIGKILL terminates before any Go code can log)
## Root Cause
**OOMKilled (Exit Code 137)** - The kernel terminated the process due to memory exhaustion. This explains the silent crash (SIGKILL cannot be caught and logged).
Investigation of cmd/acb-index-builder/db.go revealed multiple O(n²) N+1 query problems causing unbounded memory growth:
### Investigation
1. Checked pod status: `kubectl describe pod` showed `Last State: Terminated, Reason: OOMKilled, Exit Code: 137`
2. Checked running version: `ronaldraygun/acb-index-builder:42e9561` from 2026-05-10
3. Checked current repo: 410 commits ahead of running version
4. Found recent commits (79ca6c0, cdf133d, 4554bed, 404825b) that added LIMIT clauses to prevent OOMKill
1. **fetchSeries** (line 531): For each of up to 1000 series, called fetchSeriesGames making 1000 separate queries
2. **fetchChampionshipBracket** (line 736): For each of up to 500 series, called fetchSeriesGames making 500 separate queries
3. **fetchSeasonSnapshots**: LIMIT 10000 was excessive
4. **fetchLineage**: LIMIT 10000 was excessive
### The Issue
The running pod was **45 days out of date** and didn't have the recent LIMIT fixes that were added to the codebase to prevent unbounded queries from consuming excessive memory.
The crash occurred in fetchAllData() which runs immediately after copyWebAssets().
## Fix Applied
The codebase already had all necessary fixes from recent commits:
- **commit 79ca6c0** (2026-06-25): Added LIMIT 10000 to fetchSeasonSnapshots, LIMIT 500 to fetchChampionshipBracket
- **commit cdf133d**: Added LIMIT 1000 to pair frequency query
- **commit 4554bed**: Added LIMITs to fetchEvolutionMeta and fetchLineage
- **commit 404825b**: Added LIMIT clauses to other unbounded queries
Modified cmd/acb-index-builder/db.go:
These queries were called in loops for each season/match without bounds, causing the pod to exceed its 512Mi memory limit.
1. **fetchSeries**: Batched games queries (1000 queries → 1 query with WHERE IN clause, LIMIT 10000)
2. **fetchChampionshipBracket**: Batched games queries (500 queries → 1 query with WHERE IN clause, LIMIT 64)
3. **fetchSeasonSnapshots**: Reduced LIMIT from 10000 to 500
4. **fetchLineage**: Reduced LIMIT from 10000 to 1000
5. Added `strings` import for strings.Join in batch queries
## Resolution
Triggered a rebuild using the `acb-build` workflow template to deploy the current code with all OOMKill fixes:
## Deployment
- Committed and pushed changes to ai-code-battle repo
- Submitted container-build workflow to iad-ci cluster to rebuild acb-index-builder container
- Workflow name: acb-index-builder-build-v25d2 (status: Running)
## Expected Outcome
Once the new image is deployed to iad-acb cluster:
- acb-index-builder should complete build cycles without OOMKill
- "Build cycle completed" log line should appear
- Pod should stop restarting and exit CrashLoopBackOff
## Verification Needed
After deployment, monitor:
```bash
kubectl --kubeconfig=/home/coding/.kube/iad-ci.kubeconfig create -f - <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: acb-build-manual-
namespace: argo-workflows
spec:
workflowTemplateRef:
name: acb-build
EOF
kubectl --server=http://traefik-iad-acb:8001 logs -n ai-code-battle acb-index-builder-XXX -f
```
Workflow: `acb-build-manual-nv552`
### What the workflow does:
1. Resolves the current git SHA
2. Runs tests
3. Builds all ACB containers including **acb-index-builder** with the LIMIT fixes
4. Updates declarative-config repo with new image tags
5. ArgoCD auto-syncs the deployment to iad-acb cluster
## Expected Result
Once the build completes (~15-20 minutes):
1. New acb-index-builder image with LIMIT fixes will be deployed
2. Pod should run through complete build cycles without OOMKill
3. "Build cycle completed" should appear in logs
4. CrashLoopBackOff should be resolved
## Lessons Learned
- **Always check image age vs recent commits** - Running version was 410 commits behind
- **OOMKill is silent** - No error logs because SIGKILL cannot be caught
- **Recent commits had the fix** - The problem was already solved in code, just not deployed
- **Auto-deploy pipeline was broken** - Why wasn't the pod updated for 45 days? (investigate CI/CD)
## Verification Steps
After workflow completes:
1. Check pod image: `kubectl get pod -n ai-code-battle acb-index-builder-... -o jsonpath='{.spec.containers[0].image}'`
2. Watch pod logs: `kubectl logs -n ai-code-battle acb-index-builder-... -f`
3. Verify "Build cycle completed" appears in logs
4. Verify no CrashLoopBackOff: `kubectl get pods -n ai-code-battle`
Look for:
- "Build cycle completed" log line
- No crashes after "Copied web assets to output directory"
- Stable pod state (not CrashLoopBackOff)