From 2bf6d9e6bcf27e229f97e87163a986d1f4489eac Mon Sep 17 00:00:00 2001 From: jedarden Date: Thu, 4 Jun 2026 00:05:43 -0400 Subject: [PATCH] notes(bf-21081): document sealedsecret already exists - actual blocker is insufficient CPU --- notes/bf-21081.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 notes/bf-21081.md diff --git a/notes/bf-21081.md b/notes/bf-21081.md new file mode 100644 index 0000000..905bb3d --- /dev/null +++ b/notes/bf-21081.md @@ -0,0 +1,47 @@ +# Deploy P0: acb-postgres-credentials SealedSecret - COMPLETE + +## Status +**COMPLETE** - SealedSecret already existed and was deployed + +## What Was Found +The `acb-postgres-credentials` SealedSecret was already created on 2026-06-03: + +- **Commit:** 2f40563fb25055289818929ff4276f316876d0c1 +- **Repository:** jedarden/declarative-config +- **File:** k8s/apexalgo-iad/ai-code-battle/acb-postgres-sealedsecret.yml + +Commit message confirms credentials were extracted from CNPG-created `acb-app-credentials-acb-app` and sealed correctly. + +## Verification on Cluster +```bash +kubectl --server=http://traefik-apexalgo-iad:8001 get sealedsecret acb-postgres-credentials -n ai-code-battle +NAME STATUS SYNCED AGE +acb-postgres-credentials True 4m10s +``` + +The SealedSecret is synced to the cluster. The sealed-secrets controller should have unsealed it into a regular secret (cannot verify directly due to read-only permissions). + +## Actual Blocker: Insufficient CPU +The deployments are NOT crashing due to missing secrets. All pods are stuck in **Pending** due to cluster capacity issues: + +```bash +kubectl get pod acb-matchmaker-64f6dc5985-vkbbl -n ai-code-battle +0/3 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 2 Insufficient cpu. +``` + +**Status of pods in ai-code-battle namespace:** +- acb-matchmaker: Pending (Insufficient CPU) +- acb-worker (x2): Pending (Insufficient CPU) +- acb-index-builder: Pending (Insufficient CPU) +- acb-api: Pending (Insufficient CPU) +- acb-evolver: Pending (Insufficient CPU) +- acb-schema-init: Running (only pod that can schedule) + +## Root Cause +1. **One node unreachable** - has `node.kubernetes.io/unreachable` taint +2. **Two nodes insufficient CPU** - cannot schedule new pods + +## Next Steps (Infrastructure Issue) +1. Scale up cluster capacity or evict low-priority workloads +2. Fix or replace the unreachable node +3. Once CPU is available, pods should schedule successfully (secret is present)