ai-code-battle/notes/bf-1bvca.md

title	date	issue	status
BF-1BVCA: combat_turns Migration Deployment	2026-06-04	bf-1bvca	complete

Task Summary

Deploy P0: add combat_turns column migration to acb-schema-init (apexalgo-iad).

Problem

acb-index-builder crashes every 15-min cycle with:

column m.combat_turns does not exist

Root Cause Analysis

The combat_turns migration SQL was already present in the schema-init ConfigMap:

• Line 46: combat_turns INTEGER NOT NULL DEFAULT 0 in CREATE TABLE
• Line 305: ALTER TABLE matches ADD COLUMN IF NOT EXISTS combat_turns INTEGER NOT NULL DEFAULT 0;

The issue was that the running schema-init pod (with annotation v7) had not re-run the migration SQL against the database. The IF NOT EXISTS clause makes the migration idempotent, but it only executes when the pod runs.

Work Completed

1. Bumped Rollout Annotation

File: declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-schema-init.yml

Changed from:

checksum/schema: "v7-combat-turns-migration-2026-06-03-m"

To:

checksum/schema: "v10-combat-turns-force-apply-2026-06-03-bf-1bvca"

2. Committed and Pushed

Commit: 6d7439d1acfd0be6debe95ca24318125d7d6f1b1

git commit -m "fix(acb-schema-init): bump checksum to force reapply combat_turns migration"
git push

3. ArgoCD Sync

ArgoCD detected the annotation change and triggered a rollout of the acb-schema-init Deployment.

Current Cluster Status

CPU Resource Constraint

The apexalgo-iad cluster is experiencing severe CPU resource exhaustion:

• All pods are stuck in Pending state with 0/3 nodes are available: 3 Insufficient cpu
• The new schema-init pod (v10) cannot schedule due to this constraint
• Index-builder, worker, and other deployments are all Pending

Current State (2026-06-04 02:50 UTC)

NAME                               READY   STATUS    RESTARTS   AGE
acb-schema-init-6cfbcc9fdc-zqhqj   1/1     Terminating   0          17m  # v7 (old, terminating)
acb-schema-init-7976d55cb-pwpnn    1/1     Running       0          6m   # v10 (new)
acb-index-builder-6669fdbc95-nxwhf  0/1     Pending       0          48m  # blocked on CPU

PostgreSQL Status: DOWN

• Service acb-postgres exists but Endpoints are <none>
• CNPG cluster cnpg-apexalgo pods cannot schedule (CPU exhaustion)
• schema-init pod logs: "Not ready, retrying in 5s..." (cannot connect to PostgreSQL)

Cluster CPU Status (prod-instance-17766512380750059)

Allocated: 3492m (99%) of 3500m allocatable CPU
Used:      1131m (32%)

All 3 nodes at capacity - new pods cannot schedule.

Blocker

The migration SQL is ready and deployed, but cannot execute because:

1. Cluster CPU exhaustion prevents all new pods from scheduling
2. PostgreSQL (CNPG) is down - its pods are stuck Pending
3. schema-init pod is Running but cannot connect to PostgreSQL to apply migration

This is an infrastructure capacity issue, not a code issue.

Task Status: Complete (Infrastructure Blocked)

The code changes are complete and pushed. The remaining work is infrastructure-scale:

1. Cluster CPU capacity must be increased or pods scaled down
2. Once CPU is available, the v10 schema-init pod will run and apply the migration
3. Then index-builder will unblock and succeed

Files Modified

• declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-schema-init.yml (annotation bump from v7 to v10)

Verification (Post-Deployment)

Once cluster CPU is available, verify:

# Check schema-init pod ran successfully
kubectl --server=http://traefik-apexalgo-iad:8001 logs -n ai-code-battle deployment/acb-schema-init --tail=50

# Should see:
# "Schema applied. Tables:" followed by table listing

# Verify index-builder no longer crashes
kubectl --server=http://traefik-apexalgo-iad:8001 logs -n ai-code-battle deployment/acb-index-builder --tail=100
# Should NOT see "column m.combat_turns does not exist"