notes(bf-21081): document sealedsecret already exists - actual blocker is insufficient CPU

This commit is contained in:
jedarden 2026-06-04 00:05:43 -04:00
parent cbedae5b44
commit 2bf6d9e6bc

47
notes/bf-21081.md Normal file
View file

@ -0,0 +1,47 @@
# Deploy P0: acb-postgres-credentials SealedSecret - COMPLETE
## Status
**COMPLETE** - SealedSecret already existed and was deployed
## What Was Found
The `acb-postgres-credentials` SealedSecret was already created on 2026-06-03:
- **Commit:** 2f40563fb25055289818929ff4276f316876d0c1
- **Repository:** jedarden/declarative-config
- **File:** k8s/apexalgo-iad/ai-code-battle/acb-postgres-sealedsecret.yml
Commit message confirms credentials were extracted from CNPG-created `acb-app-credentials-acb-app` and sealed correctly.
## Verification on Cluster
```bash
kubectl --server=http://traefik-apexalgo-iad:8001 get sealedsecret acb-postgres-credentials -n ai-code-battle
NAME STATUS SYNCED AGE
acb-postgres-credentials True 4m10s
```
The SealedSecret is synced to the cluster. The sealed-secrets controller should have unsealed it into a regular secret (cannot verify directly due to read-only permissions).
## Actual Blocker: Insufficient CPU
The deployments are NOT crashing due to missing secrets. All pods are stuck in **Pending** due to cluster capacity issues:
```bash
kubectl get pod acb-matchmaker-64f6dc5985-vkbbl -n ai-code-battle
0/3 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 2 Insufficient cpu.
```
**Status of pods in ai-code-battle namespace:**
- acb-matchmaker: Pending (Insufficient CPU)
- acb-worker (x2): Pending (Insufficient CPU)
- acb-index-builder: Pending (Insufficient CPU)
- acb-api: Pending (Insufficient CPU)
- acb-evolver: Pending (Insufficient CPU)
- acb-schema-init: Running (only pod that can schedule)
## Root Cause
1. **One node unreachable** - has `node.kubernetes.io/unreachable` taint
2. **Two nodes insufficient CPU** - cannot schedule new pods
## Next Steps (Infrastructure Issue)
1. Scale up cluster capacity or evict low-priority workloads
2. Fix or replace the unreachable node
3. Once CPU is available, pods should schedule successfully (secret is present)