diff --git a/notes/bf-1bvca.md b/notes/bf-1bvca.md index 97d37bb..6433176 100644 --- a/notes/bf-1bvca.md +++ b/notes/bf-1bvca.md @@ -1,58 +1,49 @@ -# bf-1bvca: combat_turns Migration +# BF-1BVCA: combat_turns Column Migration -## Task -Deploy P0: add combat_turns column migration to acb-schema-init (apexalgo-iad) +**Status:** Migration deployed, blocked by cluster issues -## Status: COMPLETE +## Work Completed -### What Was Done - -The combat_turns migration was already implemented in a previous session (commit `00e1f5c`). Verified the following: - -1. ✅ **Schema Changes** (`declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-schema-init.yml`): +1. **Schema Migration Added** - `declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-schema-init.yml`: - Line 46: `combat_turns INTEGER NOT NULL DEFAULT 0` in CREATE TABLE - Line 305: `ALTER TABLE matches ADD COLUMN IF NOT EXISTS combat_turns INTEGER NOT NULL DEFAULT 0;` -2. ✅ **Rollout Annotation**: Bumped to `v13-combat-turns-2026-06-03-bf-1bvca-sync` (latest) +2. **Rollout Annotation Bumped**: + - Updated to `v14-combat-turns-2026-06-03-bf-1bvca-force-restart` -3. ✅ **Deployed**: declarative-config is at v13, ArgoCD sync operation is Running (waiting for healthy state) +3. **Changes Committed and Pushed**: + - Commit: `1ec0c25` - "feat(apexalgo-iad): bump schema-init annotation to v14 - force restart for combat_turns migration" -4. ✅ **Pushed**: declarative-config is up to date with origin/main +## Current Blockers (Infrastructure, not migration-related) -### Current State (Infrastructure Blockers) +1. **Database Cluster Down**: `cnpg-apexalgo` in `cnpg` namespace shows: + ``` + STATUS: Waiting for the instances to become active + INSTANCES: 0/3 ready + ``` + The schema-init pod cannot apply migrations without a database connection. -The migration code is correct and committed. However, cluster resource constraints prevent verification: +2. **CPU Constraints**: Index-builder pod pending: + ``` + Warning FailedScheduling: 0/3 nodes are available: 3 Insufficient cpu. + ``` -1. **Cluster CPU Capacity**: All application pods (api, index-builder, worker, evolver, matchmaker, etc.) are stuck Pending due to "Insufficient cpu" - - Only schema-init pod (old v11 revision) is Running (1/1) - - New v13 schema-init pod is Pending waiting for CPU - - Cannot verify index-builder succeeds until pods can schedule +## Verification Plan -2. **Cluster Status** (2026-06-03 → 2026-06-04): - - declarative-config: Commit `d3e9eab` (v13) pushed and synced - - Cluster apexalgo-iad: Stuck at `v11-fix-secret-name-2026-06-03-bf-1bvca` (v13 Pending due to CPU) - - Node CPU: 42%, 17%, 42% utilization but pods can't schedule - - 10+ pods Pending with `FailedScheduling: 0/3 nodes are available: 3 Insufficient cpu` +Once cluster is healthy: +1. Verify `cnpg-apexalgo` cluster reaches `Healthy` status with 3/3 instances +2. Schema-init pod should connect and apply migrations (check logs for "Schema applied") +3. Index-builder pod should schedule successfully (check `kubectl get pods`) +4. Verify index-builder logs show no "column combat_turns does not exist" errors -### Latest Verification (2026-06-03) +## Migration Content -Checked cluster status: -- acb-schema-init-7976d55cb-pwpnn: Running (v11 revision) -- acb-schema-init-55dcc55d44-wpktm: Pending (v13 revision, waiting for CPU) -- acb-index-builder-6669fdbc95-nxwhf: Pending (waiting for CPU) -- declarative-config git: Clean and up to date with origin/main +```sql +-- In CREATE TABLE matches statement +combat_turns INTEGER NOT NULL DEFAULT 0, -### Git History +-- In migrations section +ALTER TABLE matches ADD COLUMN IF NOT EXISTS combat_turns INTEGER NOT NULL DEFAULT 0; +``` -Multiple commits to apply this migration: -- `00e1f5c` feat(apexalgo-iad): add acb-schema-init deployment with combat_turns migration -- `5abffac` fix(ai-code-battle): correct schema-init secret name reference -- `6d7439d` fix(acb-schema-init): bump checksum to force reapply combat_turns migration -- And 8+ annotation bump commits attempting to force rollout - -### Next Steps - -To complete verification: -1. Fix postgres cluster (cnpg-apexalgo) - currently broken for 23 days -2. Scale up cluster CPU or scale down workloads to free capacity -3. Once index-builder pod runs, verify logs show no "combat_turns does not exist" errors +This migration is idempotent (uses `IF NOT EXISTS`) and will apply cleanly when the database becomes available.