notes: document bf-22vc5 investigation findings

## Investigation Summary
- Verified Dockerfile is valid at cmd/acb-enrichment/Dockerfile
- Located deployment manifest with placeholder SHA
- Found two workflow templates: acb-enrichment-build (Docker Hub) and acb-build-images (Forgejo)
- Webhook trigger succeeds but no image appears (likely missing docker-hub-registry secret)

## Blockers Identified
1. iad-ci kubeconfig missing at ~/.kube/iad-ci.kubeconfig
2. No Docker daemon access (permission denied)
3. argo-ci.ardenone.com returns 502
4. Cannot verify workflows or secrets without cluster access

## Required to Complete
- Obtain iad-ci kubeconfig from Rackspace Spot UI
- Verify docker-hub-registry secret exists
- Check workflow status/logs
- Re-run build if needed
- Update deployment with real SHA

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

notes/bf-22vc5-findings-2026-06-04.md

@@ -0,0 +1,61 @@
+# BF-22VC5: Findings (2026-06-04)
+
+## Task
+Deploy P0: build acb-enrichment Docker image and re-enable deployment (apexalgo-iad)
+
+## Investigation Summary
+
+### 1. Dockerfile Verification
+- ✅ `cmd/acb-enrichment/Dockerfile` exists and is valid
+- ✅ Uses multi-stage build (golang:1.25-alpine → alpine:3.19)
+- ✅ All required packages included (ca-certificates, tzdata)
+
+### 2. Deployment Manifest Status
+- ✅ Located: `/home/coding/declarative-config/k8s/apexalgo-iad/ai-code-battle/acb-enrichment-deployment.yml`
+- ❌ Contains placeholder: `ronaldraygun/acb-enrichment@sha256:placeholder`
+- ✅ ArgoCD image updater annotations configured correctly
+
+### 3. Workflow Templates Found
+- `acb-enrichment-build` → pushes to Docker Hub (`ronaldraygun/acb-enrichment`)
+- `acb-build-images` → pushes to Forgejo registry (includes enrichment)
+
+### 4. Build Attempts
+- Commit `ce82cb3` pushed, webhook triggered manually
+- Webhook returns "success" but no image appears on Docker Hub
+- Repository now exists on Docker Hub (previously 404) but has 0 tags
+- This suggests the workflow triggers but fails to push (likely missing `docker-hub-registry` secret)
+
+### 5. Infrastructure Access Blockers
+
+| Access Point | Status | Impact |
+|--------------|--------|--------|
+| `~/.kube/iad-ci.kubeconfig` | ❌ Missing | Cannot check workflows or logs |
+| rs-manager kubectl-proxy | ❌ No argo-workflows namespace | Wrong cluster |
+| argo-ci.ardenone.com | ❌ 502 Bad Gateway | Cannot access UI |
+| Docker daemon | ❌ Permission denied | Cannot build locally |
+| Docker credentials | ❌ Empty config.json | Cannot push manually |
+
+### 6. Root Cause
+The `acb-enrichment-build` workflow requires the `docker-hub-registry` secret in iad-ci, but without access to the cluster, cannot verify if:
+1. The secret exists
+2. The workflow is actually running
+3. The workflow fails at the push step
+
+## Required Actions
+
+1. **Obtain iad-ci kubeconfig** from Rackspace Spot UI → `~/.kube/iad-ci.kubeconfig`
+2. **Verify secret exists**: `kubectl get secret docker-hub-registry -n argo-workflows`
+3. **Check recent workflows**: `kubectl get workflows -n argo-workflows | grep acb-enrichment`
+4. **Fix secret or workflow** if missing/broken
+5. **Re-run build** manually or via webhook
+6. **Update deployment** with real SHA once image exists
+7. **Push to declarative-config**
+
+## Alternative: Use Forgejo Registry
+If Docker Hub access cannot be restored, update deployment to use Forgejo registry:
+- Change image from `ronaldraygun/acb-enrichment@sha256:...`
+- To `forgejo.ardenone.com/ai-code-battle/acb-enrichment:sha-{commit}`
+- But Forgejo registry is also currently returning "no available server"
+
+## Time
+2026-06-04 06:55 UTC