notes: document bf-22vc5 infrastructure investigation - all kubeconfigs missing

notes/bf-22vc5-investigation-2026-06-04-2.md

@@ -0,0 +1,61 @@
+# BF-22VC5: Investigation Summary (2026-06-04 11:04 UTC)
+
+## Task
+Deploy P0: build acb-enrichment Docker image and re-enable deployment (apexalgo-iad)
+
+## Investigation Results
+
+### Infrastructure Status
+- ❌ `/home/coding/.kube/iad-ci.kubeconfig` - Does not exist
+- ❌ `/home/coding/.kube/rs-manager.kubeconfig` - Does not exist  
+- ❌ `/home/coding/.kube/ardenone-manager.kubeconfig` - Does not exist
+- ❌ No kubeconfigs available in ~/.kube/ directory
+
+### Build Workflow Status
+- ✅ Webhook endpoint `https://webhooks-ci.ardenone.com/ai-code-battle` responds with "success"
+- ✅ Webhook triggered with commit af188b5
+- ❌ Docker Hub: `ronaldraygun/acb-enrichment` does not exist (404)
+- ❌ Forgejo registry: Returns 503 Service Unavailable
+
+### Workflow Templates Verified
+Two enrichment build workflows exist in declarative-config:
+1. `acb-enrichment-build` → Docker Hub (`ronaldraygun/acb-enrichment`)
+2. `acb-images-build` → Forgejo registry (`forgejo.ardenone.com/ai-code-battle/acb-enrichment`)
+
+### Dockerfile Verified
+`cmd/acb-enrichment/Dockerfile` is valid multi-stage Go build:
+- Stage 1: golang:1.25-alpine builder
+- Stage 2: alpine:3.19 runtime
+- Binary: `/app/acb-enrichment`
+
+### Deployment Manifest
+`declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-enrichment-deployment.yml`
+- Current state: Has placeholder SHA `sha256:placeholder`
+- File is NOT disabled (not `.disabled` suffix)
+- Ready to be updated once image is built
+
+## Root Cause
+The iad-ci kubeconfig is completely missing from this server. This prevents:
+1. Submitting workflows manually via kubectl
+2. Checking workflow status and pod logs
+3. Verifying secrets exist (`docker-hub-registry`, `forgejo-container-registry`)
+4. Debugging why workflows aren't producing images
+
+## Resolution Required
+This task CANNOT be completed without obtaining the iad-ci kubeconfig from:
+- Rackspace Spot console → iad-ci cluster → Download kubeconfig for `argocd-manager` SA
+- Save to `/home/coding/.kube/iad-ci.kubeconfig`
+
+## Current Status
+**BLOCKED** - Infrastructure access required
+
+## Next Steps (once kubeconfig is available)
+1. Submit workflow: `kubectl create -f workflow-manual-trigger.yml`
+2. Monitor: `kubectl get workflows -n argo-workflows`
+3. Get image SHA: `docker inspect ronaldraygun/acb-enrichment:sha-<commit>`
+4. Update deployment manifest
+5. Push to declarative-config
+
+---
+**Generated**: 2026-06-04 11:04 UTC
+**Commit**: af188b5