diff --git a/notes/bf-2ws.md b/notes/bf-2ws.md index 72df023..0d33db2 100644 --- a/notes/bf-2ws.md +++ b/notes/bf-2ws.md @@ -1,73 +1,51 @@ -# Fix acb-index-builder OOMKill (bf-2ws) +# Fix acb-index-builder CrashLoopBackOff (Bead bf-2ws) ## Problem -acb-index-builder pod in iad-acb cluster was in CrashLoopBackOff for 45 days with 4,713 restarts. The pod crashed silently after copying web assets - last log line was always: +acb-index-builder (iad-acb cluster) was in CrashLoopBackOff for 45 days with 4713 restarts. The pod crashed silently after the log line: ``` {"msg":"Copied web assets to output directory","source":"/app/web/dist"} ``` +No error was logged, indicating either: +1. A panic in fetchAllData or generateAllIndexes (unlikely - defer recover() logs panics) +2. OOMKill (most likely - kernel SIGKILL terminates before any Go code can log) + ## Root Cause -**OOMKilled (Exit Code 137)** - The kernel terminated the process due to memory exhaustion. This explains the silent crash (SIGKILL cannot be caught and logged). +Investigation of cmd/acb-index-builder/db.go revealed multiple O(n²) N+1 query problems causing unbounded memory growth: -### Investigation -1. Checked pod status: `kubectl describe pod` showed `Last State: Terminated, Reason: OOMKilled, Exit Code: 137` -2. Checked running version: `ronaldraygun/acb-index-builder:42e9561` from 2026-05-10 -3. Checked current repo: 410 commits ahead of running version -4. Found recent commits (79ca6c0, cdf133d, 4554bed, 404825b) that added LIMIT clauses to prevent OOMKill +1. **fetchSeries** (line 531): For each of up to 1000 series, called fetchSeriesGames making 1000 separate queries +2. **fetchChampionshipBracket** (line 736): For each of up to 500 series, called fetchSeriesGames making 500 separate queries +3. **fetchSeasonSnapshots**: LIMIT 10000 was excessive +4. **fetchLineage**: LIMIT 10000 was excessive -### The Issue -The running pod was **45 days out of date** and didn't have the recent LIMIT fixes that were added to the codebase to prevent unbounded queries from consuming excessive memory. +The crash occurred in fetchAllData() which runs immediately after copyWebAssets(). ## Fix Applied -The codebase already had all necessary fixes from recent commits: -- **commit 79ca6c0** (2026-06-25): Added LIMIT 10000 to fetchSeasonSnapshots, LIMIT 500 to fetchChampionshipBracket -- **commit cdf133d**: Added LIMIT 1000 to pair frequency query -- **commit 4554bed**: Added LIMITs to fetchEvolutionMeta and fetchLineage -- **commit 404825b**: Added LIMIT clauses to other unbounded queries +Modified cmd/acb-index-builder/db.go: -These queries were called in loops for each season/match without bounds, causing the pod to exceed its 512Mi memory limit. +1. **fetchSeries**: Batched games queries (1000 queries → 1 query with WHERE IN clause, LIMIT 10000) +2. **fetchChampionshipBracket**: Batched games queries (500 queries → 1 query with WHERE IN clause, LIMIT 64) +3. **fetchSeasonSnapshots**: Reduced LIMIT from 10000 to 500 +4. **fetchLineage**: Reduced LIMIT from 10000 to 1000 +5. Added `strings` import for strings.Join in batch queries -## Resolution -Triggered a rebuild using the `acb-build` workflow template to deploy the current code with all OOMKill fixes: +## Deployment +- Committed and pushed changes to ai-code-battle repo +- Submitted container-build workflow to iad-ci cluster to rebuild acb-index-builder container +- Workflow name: acb-index-builder-build-v25d2 (status: Running) +## Expected Outcome +Once the new image is deployed to iad-acb cluster: +- acb-index-builder should complete build cycles without OOMKill +- "Build cycle completed" log line should appear +- Pod should stop restarting and exit CrashLoopBackOff + +## Verification Needed +After deployment, monitor: ```bash -kubectl --kubeconfig=/home/coding/.kube/iad-ci.kubeconfig create -f - <<'EOF' -apiVersion: argoproj.io/v1alpha1 -kind: Workflow -metadata: - generateName: acb-build-manual- - namespace: argo-workflows -spec: - workflowTemplateRef: - name: acb-build -EOF +kubectl --server=http://traefik-iad-acb:8001 logs -n ai-code-battle acb-index-builder-XXX -f ``` - -Workflow: `acb-build-manual-nv552` - -### What the workflow does: -1. Resolves the current git SHA -2. Runs tests -3. Builds all ACB containers including **acb-index-builder** with the LIMIT fixes -4. Updates declarative-config repo with new image tags -5. ArgoCD auto-syncs the deployment to iad-acb cluster - -## Expected Result -Once the build completes (~15-20 minutes): -1. New acb-index-builder image with LIMIT fixes will be deployed -2. Pod should run through complete build cycles without OOMKill -3. "Build cycle completed" should appear in logs -4. CrashLoopBackOff should be resolved - -## Lessons Learned -- **Always check image age vs recent commits** - Running version was 410 commits behind -- **OOMKill is silent** - No error logs because SIGKILL cannot be caught -- **Recent commits had the fix** - The problem was already solved in code, just not deployed -- **Auto-deploy pipeline was broken** - Why wasn't the pod updated for 45 days? (investigate CI/CD) - -## Verification Steps -After workflow completes: -1. Check pod image: `kubectl get pod -n ai-code-battle acb-index-builder-... -o jsonpath='{.spec.containers[0].image}'` -2. Watch pod logs: `kubectl logs -n ai-code-battle acb-index-builder-... -f` -3. Verify "Build cycle completed" appears in logs -4. Verify no CrashLoopBackOff: `kubectl get pods -n ai-code-battle` +Look for: +- "Build cycle completed" log line +- No crashes after "Copied web assets to output directory" +- Stable pod state (not CrashLoopBackOff)