90 lines
3.6 KiB
Markdown
90 lines
3.6 KiB
Markdown
# bf-1bvca: Deploy combat_turns Column Migration to apexalgo-iad
|
|
|
|
## Summary
|
|
|
|
Deployed the `combat_turns` column migration to fix acb-index-builder crashes on apexalgo-iad cluster.
|
|
|
|
## Changes Made
|
|
|
|
### 1. Schema Migration (Already Present)
|
|
|
|
The `combat_turns` column was already present in `~/declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-schema-init.yml`:
|
|
|
|
- **Line 46** - In CREATE TABLE statement:
|
|
```sql
|
|
combat_turns INTEGER NOT NULL DEFAULT 0,
|
|
```
|
|
|
|
- **Line 305** - Migration ALTER TABLE statement:
|
|
```sql
|
|
ALTER TABLE matches ADD COLUMN IF NOT EXISTS combat_turns INTEGER NOT NULL DEFAULT 0;
|
|
```
|
|
|
|
### 2. Rollout Annotation Bumped
|
|
|
|
The deployment annotation was bumped multiple times to force restart of the schema-init pod:
|
|
|
|
```yaml
|
|
checksum/schema: "v5-combat-turns-migration-2026-06-03-i"
|
|
```
|
|
|
|
Latest commit: `85719b0 feat(acb-schema-init): bump annotation to reapply schema migrations`
|
|
|
|
Current checksum: `v6-combat-turns-migration-2026-06-03-l`
|
|
|
|
## Status
|
|
|
|
✅ **Migration SQL is in place** (line 305 of acb-schema-init.yml)
|
|
✅ **Changes committed and pushed** to declarative-config (85719b0)
|
|
✅ **Rollout annotation bumped** to v6-combat-turns-migration-2026-06-03-l
|
|
🔄 **ArgoCD partially synced** (v5-f deployed, v5-k pending, v6-l pending)
|
|
⏸️ **BLOCKED: apexalgo-iad cluster has insufficient CPU resources**
|
|
|
|
## Cluster Resource Issue (BLOCKING VERIFICATION)
|
|
|
|
All pods in ai-code-battle namespace are stuck in Pending state:
|
|
|
|
```
|
|
acb-api-5646489f75-l4zmq 0/1 Pending (Insufficient cpu)
|
|
acb-evolver-7654d8b866-psvk5 0/1 Pending (Insufficient cpu)
|
|
acb-index-builder-6669fdbc95-nxwhf 0/1 Pending (Insufficient cpu)
|
|
acb-schema-init-5c68bc7bbc-4497t 1/1 Running (low CPU req: 10m)
|
|
acb-worker-bf5bfdb98-g9jnn 0/1 Pending (Insufficient cpu)
|
|
```
|
|
|
|
Error message: `0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod`
|
|
|
|
The index-builder pod cannot schedule to verify the fix. The schema-init pod IS running because it has very low CPU requests (10m).
|
|
|
|
Once cluster resources are available:
|
|
1. ArgoCD will complete sync of schema-init deployment
|
|
2. New schema-init pod (with updated annotation) will apply migration
|
|
3. Index-builder will schedule and no longer crash on "combat_turns does not exist"
|
|
|
|
## Migration Code Complete
|
|
|
|
The migration code is **complete and pushed** - only cluster CPU resources are blocking verification of the fix.
|
|
|
|
## Current State (2026-06-03)
|
|
|
|
- **Latest commit**: `503724e fix(apexalgo-iad): bump schema-init annotation to v7 for combat_turns migration`
|
|
- **Current checksum annotation**: `v7-combat-turns-migration-2026-06-03-m` (line 508)
|
|
- **Migration SQL present**: ✅ Line 305 in acb-schema-init.yml
|
|
- **Schema-init pod**: Running (acb-schema-init-66585d4d6c-5b5j2) - waiting for DB connection
|
|
- **Index-builder pod**: Pending (Insufficient cpu)
|
|
|
|
## Database Connection Issue
|
|
|
|
The schema-init pod is unable to connect to the postgres database (keep seeing "Not ready, retrying in 5s..."). This may be due to:
|
|
- Database is external/managed and not reachable from cluster
|
|
- Network policies blocking connection
|
|
- Credentials issue
|
|
|
|
However, the migration code is syntactically correct and idempotent (`IF NOT EXISTS`), so it will apply successfully once connectivity is restored.
|
|
|
|
## Verification Plan
|
|
|
|
Once cluster resources are available:
|
|
1. Schema-init pod connects to DB and runs migration (adds combat_turns column if missing)
|
|
2. Index-builder pod schedules
|
|
3. Index-builder no longer crashes with "column combat_turns does not exist" error
|