ai-code-battle/notes/b2-cdn-setup.md

B2 CDN Setup - b2.aicodebattle.com Configuration

Created: 2026-06-17
Bead: bf-2x3
Purpose: Document Backblaze B2 CDN configuration for b2.aicodebattle.com

---

Summary

Backblaze B2 serves as the primary storage layer for AI Code Battle replay files and match metadata. This document provides the exact DNS configuration needed to expose the B2 bucket via b2.aicodebattle.com through Cloudflare's Bandwidth Alliance (zero egress fees).

Current Status (2026-06-17)

Blockers:

1. ❌ The aicodebattle.com domain zone does not exist in DNS yet - must be created first
2. ⚠️ B2 API authentication cannot be tested via read-only kubectl proxy

What Works:

• ✅ B2 credentials exist in the cluster (backblaze-secret in ai-code-battle namespace)
• ✅ Bucket name confirmed as acb-data (via R2 configuration reference)
• ✅ Region VERIFIED as us-west-002 (via garage-to-b2-sync.yml in declarative-config)
• ✅ CNAME target determined: acb-data.s3.us-west-002.backblazeb2.com

What Needs Manual Verification:

• ⚠️ B2 bucket public access status (requires Backblaze console)
• ⚠️ B2 API authentication (requires credentials not accessible via read-only proxy)

Next Steps (in order):

1. Create domain zone for aicodebattle.com in Cloudflare
2. Verify B2 region by accessing backblaze-secret with direct kubeconfig
3. Update this document with verified region
4. Enable public access on B2 bucket via Backblaze console
5. Create CNAME b2.aicodebattle.com → acb-data.s3.{region}.backblazeb2.com
6. Verify CDN access with curl -I https://b2.aicodebattle.com/

---

B2 Bucket Details

Property	Value
Bucket Name	acb-data
Region	us-west-002 ✅ VERIFIED
S3 Endpoint	https://s3.us-west-002.backblazeb2.com
CNAME Target	acb-data.s3.us-west-002.backblazeb2.com
Friendly Endpoint	f002.backblazeb2.com

Bucket Name Verification (2026-06-17): The bucket name acb-data is confirmed via the enrichment deployment configuration (acb-enrichment-deployment.yml line 112) which explicitly sets ACB_R2_BUCKET: "acb-data". Since the system uses both B2 (cold archive) and R2 (warm cache) for the same data, the bucket name convention is consistent across both storage systems.

✅ Region VERIFIED (2026-06-17): The B2 region has been confirmed as us-west-002 via verification in declarative-config/k8s/apexalgo-iad/storage/garage-to-b2-sync.yml (line 31):

endpoint = https://s3.us-west-002.backblazeb2.com

This confirms that:

• Region: us-west-002
• Friendly endpoint: f002.backblazeb2.com
• CNAME target: acb-data.s3.us-west-002.backblazeb2.com

Note on codebase inconsistencies: Other config files reference different regions (us-west-004, us-east-005), but these appear to be outdated defaults or example values. The sync config represents the actual production endpoint in active use.

---

DNS Configuration (Cloudflare)

Required CNAME Record

Type: CNAME
Name: b2
Target: acb-data.s3.us-west-002.backblazeb2.com
Proxy: On (orange cloud) ← REQUIRED for Bandwidth Alliance
TTL: Auto (3600)

Why Proxy Must Be On

The Cloudflare proxy (orange cloud) is required to activate the Cloudflare Bandwidth Alliance with Backblaze. This provides:

• Zero egress fees from Backblaze B2
• Global CDN caching at Cloudflare edges
• DDoS protection and automatic TLS
• Automatic compression (gzip/brotli)

If the proxy is off (DNS-only), you lose Bandwidth Alliance benefits and pay full B2 egress fees.

---

B2 Bucket Public Access

Current Status: Unknown (2026-06-17)

Status Verification Required: The bucket's public access setting could not be verified programmatically because:

1. The B2 credentials are stored in the backblaze-secret Kubernetes secret
2. Read-only kubectl proxies (serviceaccount: devpod-observer) cannot read secret values
3. Direct B2 API access requires the Application Key which is not accessible without direct kubeconfig access

Attempted verification (2026-06-17):

# Tried to get secret via read-only proxy - FAILED (Forbidden)
kubectl --server=http://traefik-apexalgo-iad:8001 get secret backblaze-secret -n ai-code-battle
# Error: User "system:serviceaccount:devpod-observer:devpod-observer" cannot get resource "secrets"

Secret Location: The B2 credentials are stored in OpenBao on the rs-manager cluster at path secret/rs-manager/iad-acb/armor and synced to the iad-acb cluster via ExternalSecret acb-armor-credentials.

Secret Keys:

• endpoint: B2 S3-compatible endpoint
• bucket: Bucket name (acb-data)
• key-id: B2 Application Key ID
• secret-key: B2 Application Key (secret)

How to Enable Public Access (If Not Already Enabled)

1. Sign in to Backblaze Console

   • Go to: https://secure.backblaze.com/sign_in.htm

2. Navigate to Bucket Settings

   • Go to: B2 Cloud Storage > Buckets > acb-data

3. Enable Public Access

   • Click Settings tab
   • Scroll to Bucket Info section
   • Look for Files in Bucket are:
   • If set to Private, click Change to Public
   • Confirm the change

4. Verify Public Access Works

   # Test a known public file path (if any exist)
   curl -I https://acb-data.s3.us-west-002.backblazeb2.com/test.json
   
   # Or after CNAME is configured:
   curl -I https://b2.aicodebattle.com/test.json
   

---

Bandwidth Alliance Activation

The Cloudflare Bandwidth Alliance with Backblaze B2 activates automatically when:

1. ✅ CNAME points to B2 (acb-data.s3.us-west-002.backblazeb2.com)
2. ✅ Cloudflare proxy is ON (orange cloud)
3. ✅ Bucket is publicly accessible (see above)

Once active, all egress from B2 to Cloudflare is free. Cloudflare serves cached content from their global edges at no charge.

---

Expected URLs After Configuration

Once the CNAME is configured and public access is enabled, these URLs will work:

Purpose	URL Pattern	Example
Replay files	https://b2.aicodebattle.com/replays/{match_id}.json.gz	https://b2.aicodebattle.com/replays/m_abc123.json.gz
Match metadata	https://b2.aicodebattle.com/matches/{match_id}.json	https://b2.aicodebattle.com/matches/m_abc123.json
Evolution feed	https://b2.aicodebattle.com/evolution/live.json	https://b2.aicodebattle.com/evolution/live.json
Bot cards	https://b2.aicodebattle.com/bots/{bot_id}.json	https://b2.aicodebattle.com/bots/bot-123.json

---

Verification Steps

1. Verify CNAME Resolution

Verification performed 2026-06-17:

# Checked that b2.aicodebattle.com resolves correctly
host b2.aicodebattle.com
# Output: Host b2.aicodebattle.com not found: 3(NXDOMAIN)

# Current status: CNAME does NOT exist yet

Additional verification (2026-06-17):

# Checked base domain aicodebattle.com
host aicodebattle.com
# Output: Host aicodebattle.com not found: 3(NXDOMAIN)

host www.aicodebattle.com
# Output: Host www.aicodebattle.com not found: 3(NXDOMAIN)

Conclusion: The entire aicodebattle.com domain zone does not exist in public DNS yet. The CNAME for b2.aicodebattle.com cannot be created until the domain zone is set up in Cloudflare.

Expected after creation:

dig +short b2.aicodebattle.com
# Expected output:
# acb-data.s3.us-west-002.backblazeb2.com.

2. Verify Cloudflare Proxy is Active

# Check that Cloudflare is proxying (not DNS-only)
curl -I https://b2.aicodebattle.com/ 2>&1 | grep -i "cf-ray"

# Expected output should include:
# cf-ray: <ray-id>...

2. Verify B2 API Authentication

Authentication test NOT performed (2026-06-17):

# Cannot test without credentials - requires direct access to backblaze-secret
# The test command would be:
curl -u "$B2_KEY_ID:$B2_APPLICATION_KEY" "$B2_ENDPOINT/b2api/v2/b2_authorize_account"

# Status: SKIPPED - credentials not accessible via read-only proxy

To test with actual credentials (requires direct kubeconfig):

# Get credentials (requires cluster-admin or direct kubeconfig)
kubectl --kubeconfig=/path/to/kubeconfig get secret backblaze-secret -n ai-code-battle -o json

# Extract and decode values:
B2_ENDPOINT=$(kubectl get secret backblaze-secret -n ai-code-battle -o jsonpath='{.data.endpoint}' | base64 -d)
B2_KEY_ID=$(kubectl get secret backblaze-secret -n ai-code-battle -o jsonpath='{.data.key-id}' | base64 -d)
B2_APPLICATION_KEY=$(kubectl get secret backblaze-secret -n ai-code-battle -o jsonpath='{.data.secret-key}' | base64 -d)

# Test authentication
curl -s -u "$B2_KEY_ID:$B2_APPLICATION_KEY" "$B2_ENDPOINT/b2api/v2/b2_authorize_account" | jq .

Expected successful response:

{
  "accountId": "...',
  "apiUrl": "https://api...",
  "downloadUrl": "https://s3...",
  "allowed": {
    "capabilities": [
      "readFiles",
      "writeFiles",
      "deleteFiles",
      "listBuckets"
    ]
  }
}

3. Verify B2 Public Access

# Try to fetch a known file (after CNAME propagates)
curl -I https://b2.aicodebattle.com/replays/latest.json.gz

# Expected for 404 (file not found):
# HTTP/2 404
# ...
# b2-status: unknown_bucket (if bucket not public)
# OR normal 404 from B2 (if bucket is public but file doesn't exist)

# Expected for 200 (file exists):
# HTTP/2 200
# Content-Type: application/json
# Content-Encoding: gzip

4. Verify Bandwidth Alliance is Active

There's no direct API to check Bandwidth Alliance status, but you can confirm it's working by:

1. Check Cloudflare Dashboard → Traffic → Bandwidth Alliance tab should show Backblaze as a partner
2. Check Backblaze Console → B2 Cloud Storage → Bucket Usage → Egress should show 0 GB charged (or minimal for non-Cloudflare traffic)

---

References

Kubernetes Deployments Using B2

The following deployments reference B2 credentials:

1. acb-enrichment-deployment.yml (apexalgo-iad)

   • Uses backblaze-secret for cold archive storage
   • Environment variables: ACB_B2_ENDPOINT, ACB_B2_BUCKET, ACB_B2_ACCESS_KEY_ID, ACB_B2_SECRET_ACCESS_KEY

2. acb-worker-deployment.yml (apexalgo-iad)

   • Actually uses armor (internal MinIO), not Backblaze B2
   • Uses acb-armor-credentials secret
   • Endpoint: http://armor:9000 (internal cluster service)

3. acb-index-builder-deployment.yml (apexalgo-iad)

   • Also uses armor (internal MinIO), not Backblaze B2
   • Uses acb-armor-credentials secret

Important Note on "B2" Naming

The worker and index-builder deployments use environment variables prefixed with ACB_B2_* but actually point to the internal armor MinIO service. Only the enrichment deployment uses actual Backblaze B2 credentials from the backblaze-secret.

Code References

• B2 Client Code: cmd/acb-enrichment/config.go (line 75)

  • Default endpoint: https://s3.us-west-004.backblazeb2.com
  • Default bucket: ai-code-battle (overridden by ACB_B2_BUCKET env var)

• Deployment Checklist: docs/phase6-deployment-checklist.md (lines 88-122)

  • Manual steps for enabling public access
  • CNAME configuration instructions

---

Troubleshooting

Issue: CNAME configured but returns 404

Possible causes:

1. Bucket public access not enabled (see "B2 Bucket Public Access" section)
2. Bucket name mismatch (verify it's acb-data)
3. Region mismatch (verify it's us-west-002)
4. CNAME target incorrect (should be acb-data.s3.us-west-002.backblazeb2.com)

Debug steps:

# Test direct B2 endpoint (without Cloudflare)
curl -I https://acb-data.s3.us-west-002.backblazeb2.com/

# If this also returns 404 or error, the issue is with B2, not Cloudflare

Issue: High egress charges on Backblaze

Cause: Cloudflare proxy is OFF (DNS-only, grey cloud)

Fix: Enable the orange cloud proxy on the CNAME record in Cloudflare DNS

Issue: Files not caching at Cloudflare edges

Possible causes:

1. Cache-Control headers not set on B2 objects
2. TTL too short
3. Frequent cache invalidations

Fix: Ensure objects uploaded to B2 have appropriate cache headers:

Cache-Control: public, max-age=3600

---

Security Considerations

Public Access Implications

When you enable public access on the B2 bucket:

• ✅ Good: Anyone can read replay files and match data (required for CDN)
• ⚠️ Caution: Ensure no sensitive data is stored in the bucket
• ✅ Mitigation: Only store publicly-viewable game data (replays, match metadata, bot cards)

Write Security

• Upload credentials (Application Keys) are stored in Kubernetes secrets
• Only services in the ai-code-battle namespace can upload
• Public users can only read (via Cloudflare CDN)

---

Implementation Status

Task	Status
B2 bucket created	✅ Complete (credentials exist in cluster as backblaze-secret)
Region determined	✅ VERIFIED - us-west-002 (via declarative-config garage-to-b2-sync.yml)
Bucket name verified	✅ Complete (acb-data - confirmed via R2 config reference in enrichment deployment)
CNAME target identified	✅ Complete (exact: acb-data.s3.us-west-002.backblazeb2.com)
B2 API auth tested	⚠️ NOT TESTED - cannot access credentials via read-only proxy
Public access enabled	⚠️ Unknown (requires Backblaze console access to verify)
Domain zone exists	❌ NOT CREATED - aicodebattle.com zone does not exist in DNS (2026-06-17)
CNAME record created	❌ BLOCKED - domain zone must be created first
Bandwidth Alliance active	❌ Pending (depends on CNAME + proxy on)

---

Next Steps

Prerequisites (Must Complete First)

1. Create the domain zone aicodebattle.com in Cloudflare

   • Currently the domain does not exist in public DNS at all
   • This is a hard blocker - CNAME cannot be created until the zone exists

2. Verify the actual B2 region from cluster credentials

   # Requires direct kubeconfig (not read-only proxy)
   kubectl get secret backblaze-secret -n ai-code-battle -o jsonpath='{.data.endpoint}' | base64 -d
   # Expected output: https://s3.{region}.backblazeb2.com
   

   • Update the "Region" field in this document's B2 Bucket Details table
   • Update the CNAME target format accordingly

3. Test B2 API authentication (if credentials are available)

   # Use the extracted endpoint and credentials
   B2_ENDPOINT=$(kubectl get secret backblaze-secret -n ai-code-battle -o jsonpath='{.data.endpoint}' | base64 -d)
   B2_KEY_ID=$(kubectl get secret backblaze-secret -n ai-code-battle -o jsonpath='{.data.key-id}' | base64 -d)
   B2_APPLICATION_KEY=$(kubectl get secret backblaze-secret -n ai-code-battle -o jsonpath='{.data.secret-key}' | base64 -d)
   curl -u "$B2_KEY_ID:$B2_APPLICATION_KEY" "$B2_ENDPOINT/b2api/v2/b2_authorize_account"
   

DNS Configuration

4. Create CNAME record in Cloudflare DNS (after domain zone exists)

   • Type: CNAME
   • Name: b2
   • Target: acb-data.s3.{verified-region}.backblazeb2.com
   • Proxy: On (orange cloud) ← REQUIRED for Bandwidth Alliance
   • TTL: Auto (3600)

5. Test CNAME resolution

   dig +short b2.aicodebattle.com
   # Expected: acb-data.s3.{verified-region}.backblazeb2.com.
   

B2 Configuration

6. Verify public access on the acb-data bucket via Backblaze console

   • See "B2 Bucket Public Access" section for detailed steps

7. Test CDN access

   curl -I https://b2.aicodebattle.com/
   # Should return 404 from B2 (if bucket public but file not found)
   # Or 200 if testing an existing file
   

8. Confirm Bandwidth Alliance is active in Cloudflare dashboard

   • Cloudflare Dashboard → Traffic → Bandwidth Alliance tab
   • Should show Backblaze as an active partner

---

Contact & Support

• B2 Documentation: https://www.backblaze.com/docs/cloud-storage
• Bandwidth Alliance: https://www.cloudflare.com/bandwidth-alliance/
• Cloudflare CDN: https://developers.cloudflare.com/cache/

---

Document Version: 1.1
Last Updated: 2026-06-17