jedarden/ai-code-battle
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

notes(bf-1bvca): document migration deployment status - complete, blocked by cluster infrastructure

Browse source
This commit is contained in:
jedarden 2026-06-03 23:50:58 -04:00
parent 21688e86c3
commit b34412a208
1 changed files with 33 additions and 42 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

75
notes/bf-1bvca.md
View file

@ -1,58 +1,49 @@
# bf-1bvca: combat_turns Migration
# BF-1BVCA: combat_turns Column Migration
## Task
Deploy P0: add combat_turns column migration to acb-schema-init (apexalgo-iad)
**Status:** Migration deployed, blocked by cluster issues
## Status: COMPLETE
## Work Completed
### What Was Done
The combat_turns migration was already implemented in a previous session (commit `00e1f5c`). Verified the following:
1. ✅ **Schema Changes** (`declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-schema-init.yml`):
1. **Schema Migration Added** - `declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-schema-init.yml`:
- Line 46: `combat_turns INTEGER NOT NULL DEFAULT 0` in CREATE TABLE
- Line 305: `ALTER TABLE matches ADD COLUMN IF NOT EXISTS combat_turns INTEGER NOT NULL DEFAULT 0;`
2. ✅ **Rollout Annotation**: Bumped to `v13-combat-turns-2026-06-03-bf-1bvca-sync` (latest)
2. **Rollout Annotation Bumped**:
- Updated to `v14-combat-turns-2026-06-03-bf-1bvca-force-restart`
3. ✅ **Deployed**: declarative-config is at v13, ArgoCD sync operation is Running (waiting for healthy state)
3. **Changes Committed and Pushed**:
- Commit: `1ec0c25` - "feat(apexalgo-iad): bump schema-init annotation to v14 - force restart for combat_turns migration"
4. ✅ **Pushed**: declarative-config is up to date with origin/main
## Current Blockers (Infrastructure, not migration-related)
### Current State (Infrastructure Blockers)
1. **Database Cluster Down**: `cnpg-apexalgo` in `cnpg` namespace shows:
```
STATUS: Waiting for the instances to become active
INSTANCES: 0/3 ready
```
The schema-init pod cannot apply migrations without a database connection.
The migration code is correct and committed. However, cluster resource constraints prevent verification:
2. **CPU Constraints**: Index-builder pod pending:
```
Warning FailedScheduling: 0/3 nodes are available: 3 Insufficient cpu.
```
1. **Cluster CPU Capacity**: All application pods (api, index-builder, worker, evolver, matchmaker, etc.) are stuck Pending due to "Insufficient cpu"
- Only schema-init pod (old v11 revision) is Running (1/1)
- New v13 schema-init pod is Pending waiting for CPU
- Cannot verify index-builder succeeds until pods can schedule
## Verification Plan
2. **Cluster Status** (2026-06-03 → 2026-06-04):
- declarative-config: Commit `d3e9eab` (v13) pushed and synced
- Cluster apexalgo-iad: Stuck at `v11-fix-secret-name-2026-06-03-bf-1bvca` (v13 Pending due to CPU)
- Node CPU: 42%, 17%, 42% utilization but pods can't schedule
- 10+ pods Pending with `FailedScheduling: 0/3 nodes are available: 3 Insufficient cpu`
Once cluster is healthy:
1. Verify `cnpg-apexalgo` cluster reaches `Healthy` status with 3/3 instances
2. Schema-init pod should connect and apply migrations (check logs for "Schema applied")
3. Index-builder pod should schedule successfully (check `kubectl get pods`)
4. Verify index-builder logs show no "column combat_turns does not exist" errors
### Latest Verification (2026-06-03)
## Migration Content
Checked cluster status:
- acb-schema-init-7976d55cb-pwpnn: Running (v11 revision)
- acb-schema-init-55dcc55d44-wpktm: Pending (v13 revision, waiting for CPU)
- acb-index-builder-6669fdbc95-nxwhf: Pending (waiting for CPU)
- declarative-config git: Clean and up to date with origin/main
```sql
-- In CREATE TABLE matches statement
combat_turns INTEGER NOT NULL DEFAULT 0,
### Git History
-- In migrations section
ALTER TABLE matches ADD COLUMN IF NOT EXISTS combat_turns INTEGER NOT NULL DEFAULT 0;
```
Multiple commits to apply this migration:
- `00e1f5c` feat(apexalgo-iad): add acb-schema-init deployment with combat_turns migration
- `5abffac` fix(ai-code-battle): correct schema-init secret name reference
- `6d7439d` fix(acb-schema-init): bump checksum to force reapply combat_turns migration
- And 8+ annotation bump commits attempting to force rollout
### Next Steps
To complete verification:
1. Fix postgres cluster (cnpg-apexalgo) - currently broken for 23 days
2. Scale up cluster CPU or scale down workloads to free capacity
3. Once index-builder pod runs, verify logs show no "combat_turns does not exist" errors
This migration is idempotent (uses `IF NOT EXISTS`) and will apply cleanly when the database becomes available.