docs: remove R2 warm cache tier from plan — B2-only storage

This commit is contained in:
jedarden 2026-05-31 11:59:29 -04:00
parent 31fec6c412
commit e06091fb64

View file

@ -23,11 +23,7 @@ All compute runs in the **apexalgo-iad** Kubernetes cluster (Rackspace Spot),
which acts as a **match factory**: it runs battles, generates replays, and
periodically publishes the updated site to Pages.
Replay files use **tiered storage**: Backblaze B2 is the permanent cold
archive (all replays, forever), and Cloudflare R2 is a warm cache for recent
replays (free tier, ≤10 GB). The browser fetches replays from R2; old replays
that have aged out of R2 fall back to B2 (free egress via Cloudflare Bandwidth
Alliance).
Replay files are stored in and served directly from **Backblaze B2** (via Cloudflare CDN / Bandwidth Alliance). B2 is the single storage layer — all replays, thumbnails, bot cards, match metadata, and evolution status files live in B2. Free egress via Cloudflare Bandwidth Alliance (zero egress fees).
### Cloudflare (Static Tier)
@ -36,19 +32,13 @@ Alliance).
indexes, series, seasons, evolution data, blog posts. Updated every ~90
minutes by the K8s index builder via `wrangler pages deploy`. Global CDN,
zero-config TLS, instant cache invalidation on deploy.
- **Cloudflare R2** (warm replay cache): Stores recent replay files and
per-match metadata — the subset of data too numerous for Pages' 20K file
limit and too hot for B2-only serving. Capped at ≤10 GB to stay within
the free tier. The index builder manages the R2 lifecycle: promotes recent
replays from B2, prunes old ones when approaching the cap.
### Backblaze B2 (Cold Archive)
### Backblaze B2 (Storage)
- **B2 bucket**: Permanent archive for **all** replay files and match
metadata. Match workers upload directly to B2 after each match. B2 is the
source of truth for replay data — R2 is just a CDN cache in front of it.
Free egress to browsers via Cloudflare Bandwidth Alliance (B2 + Cloudflare
= zero egress fees). S3-compatible API.
- **B2 bucket**: Permanent storage for **all** replay files, match metadata,
thumbnails, bot cards, and evolution status files. Match workers upload
directly to B2 after each match. Served to browsers via Cloudflare CDN
(Bandwidth Alliance = zero egress fees). S3-compatible API.
### apexalgo-iad (Compute Tier)
@ -72,9 +62,7 @@ All backend compute runs in the `ai-code-battle` namespace:
every 4h.
- **Index Builder Deployment**: Sleep-loop (15 min cycle). Reads PostgreSQL,
generates all JSON index files, deploys them to Cloudflare Pages via
`wrangler pages deploy`. Manages R2 warm cache (copies recent replays
from B2, prunes old replays to stay within free tier). Self-restarts
every 4h.
`wrangler pages deploy`. Self-restarts every 4h.
### Go API (deferred)
@ -128,20 +116,7 @@ Pages project (ai-code-battle.pages.dev):
└── {map_id}.json
```
**Cloudflare R2** (warm replay cache, free tier ≤10 GB):
```
R2 bucket:
├── replays/
│ └── {match_id}.json.gz (recent replay files, promoted from B2)
├── matches/
│ └── {match_id}.json (recent per-match metadata)
├── thumbnails/
│ └── {match_id}.png (auto-generated match thumbnails)
└── cards/
└── {bot_id}.png (bot profile card images)
```
**Backblaze B2** (cold archive, permanent):
**Backblaze B2** (storage — all replay data, served via Cloudflare CDN):
```
B2 bucket:
├── replays/
@ -150,8 +125,10 @@ B2 bucket:
│ └── {match_id}.json (ALL per-match metadata)
├── thumbnails/
│ └── {match_id}.png (ALL thumbnails)
└── cards/
└── {bot_id}.png (ALL bot card images)
├── cards/
│ └── {bot_id}.png (ALL bot card images)
└── evolution/
└── live.json (evolver status, updated each cycle)
```
**Data loading pattern in the SPA:**
@ -159,24 +136,18 @@ B2 bucket:
```js
// SPA shell + index data from Cloudflare Pages (same origin)
const PAGES = '' // relative — same origin as the SPA
// Replays from R2 warm cache (recent), B2 cold archive (old)
const R2 = 'https://r2.aicodebattle.com' // or R2 public URL
const B2 = 'https://b2.aicodebattle.com' // B2 via Cloudflare CDN
// Leaderboard, bot profiles, match indexes — all from Pages (same origin):
const lb = await fetch(`${PAGES}/data/leaderboard.json`).then(r => r.json())
// Replay viewer — try R2 warm cache first, fall back to B2 cold archive:
// Replay viewer — fetches directly from B2:
async function fetchReplay(matchId) {
const r2 = await fetch(`${R2}/replays/${matchId}.json.gz`)
if (r2.ok) return r2
return fetch(`${B2}/replays/${matchId}.json.gz`) // cold fallback
return fetch(`${B2}/replays/${matchId}.json.gz`)
}
// Match metadata — same R2-then-B2 pattern:
const meta = await fetch(`${R2}/matches/${matchId}.json`)
.then(r => r.ok ? r : fetch(`${B2}/matches/${matchId}.json`))
.then(r => r.json())
// Match metadata — fetches directly from B2:
const meta = await fetch(`${B2}/matches/${matchId}.json`).then(r => r.json())
```
**Cache behavior:**
@ -184,64 +155,54 @@ const meta = await fetch(`${R2}/matches/${matchId}.json`)
- **Pages assets**: Cloudflare Pages handles caching automatically. Deploys
via `wrangler pages deploy` invalidate the cache globally. Index data is
at most ~90 minutes stale (the index builder's cycle time).
- **R2 objects** (warm cache): Served with appropriate `Cache-Control` headers:
- **B2 objects**: Served via Cloudflare CDN with appropriate `Cache-Control` headers:
- `replays/*.json.gz`: `immutable, max-age=31536000` (content-addressed)
- `matches/*.json`: `immutable, max-age=31536000` (content-addressed)
- `thumbnails/`, `cards/`: `max-age=86400` (regenerated rarely)
- **B2 objects** (cold archive): Same cache headers. B2 egress through
Cloudflare Bandwidth Alliance = zero egress fees. Cloudflare CDN caches
B2 responses, so frequently accessed cold replays still perform well.
- `evolution/live.json`: `max-age=10` (updated each evolver cycle)
B2 egress through Cloudflare Bandwidth Alliance = zero egress fees. Cloudflare CDN
caches B2 responses so frequently accessed replays perform well globally.
**Tiered storage lifecycle:**
**Data flow:**
1. Match worker completes a match → uploads `replays/{match_id}.json.gz`
and `matches/{match_id}.json` to **B2** (cold archive, permanent)
and `matches/{match_id}.json` to **B2**
2. Worker writes match result to **PostgreSQL** (scores, ratings, metadata)
3. Index builder (every ~15 min) reads new results from PostgreSQL, rebuilds
all JSON index files, deploys to **Pages** via `wrangler pages deploy`
4. Index builder promotes recent replays from B2 to **R2** (warm cache)
5. Index builder prunes oldest replays from R2 when approaching 10 GB cap
6. Browser loads SPA + indexes from Pages, fetches replays from R2 (warm)
with B2 fallback (cold)
4. Browser loads SPA + indexes from Pages, fetches replays and match data
directly from **B2** (via Cloudflare CDN)
**Storage budget:**
- **R2 (warm cache)**: ≤10 GB free tier. At ~50 KB/replay (gzipped), holds
~200K replays. At 60 matches/hour, that's ~139 days of warm replays.
Class A writes: index builder promotes ~1,440 replays/day = ~43K/month
(well within 1M/month free tier).
- **B2 (cold archive)**: First 10 GB free, $0.006/GB/month after. At 60
matches/hour, ~2.2 GB/month. Year one: ~26 GB = ~$0.10/month.
- **B2**: First 10 GB free, $0.006/GB/month after. At 60 matches/hour,
~2.2 GB/month for replays. Year one: ~26 GB ≈ $0.10/month.
Free egress via Cloudflare Bandwidth Alliance.
- **Pages**: 20K file limit per deployment. Only SPA + JSON indexes — well
within limits (replays are on R2/B2, not Pages).
within limits (replays and match data are on B2, not Pages).
```
┌────────── Cloudflare ──────────────────────────────────┐
│ │
│ ┌─────────────────────┐ ┌─────────────────────────┐ │
│ │ Cloudflare Pages │ │ Cloudflare R2 │ │
│ │ (static site) │ │ (warm replay cache) │ │
│ │ │ │ │ │
│ │ SPA shell (HTML/ │ │ replays/*.json.gz │ │
│ │ JS/CSS) │ │ matches/*.json │ │
│ │ data/*.json indexes │ │ thumbnails/*.png │ │
│ │ maps/*.json │ │ cards/*.png │ │
│ │ docs/, img/ │ │ ≤10 GB free tier │ │
│ └─────────────────────┘ └─────────────────────────┘ │
│ ▲ wrangler deploy ▲ S3 PUT (promote) │
└────────┼─────────────────────────┼───────────────────────┘
│ │
┌────────┼─────────────────────────┼───────────────────────┐
│ │ apexalgo-iad cluster — ai-code-battle ns │
│ │ │ │
│ ┌─────┴──────────────────┐ │ │
│ │ Index Builder Dep. │ │ │
│ │ Reads PostgreSQL, │──────┘ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ Cloudflare Pages (static site) │ │
│ │ │ │
│ │ SPA shell (HTML/JS/CSS) │ │
│ │ data/*.json indexes │ │
│ │ maps/*.json │ │
│ │ docs/, img/ │ │
│ └─────────────────────────────────────────────────┘ │
│ ▲ wrangler deploy │
└────────┼──────────────────────────────────────────────────┘
┌────────┼──────────────────────────────────────────────────┐
│ │ apexalgo-iad cluster — ai-code-battle ns │
│ │ │
│ ┌─────┴──────────────────┐ │
│ │ Index Builder Dep. │ │
│ │ Reads PostgreSQL, │ │
│ │ generates JSON indexes,│ │
│ │ deploys to Pages, │ │
│ │ promotes replays to R2,│ │
│ │ prunes R2 warm cache │ │
│ │ deploys to Pages │ │
│ └─────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────┐ │
@ -299,15 +260,14 @@ const meta = await fetch(`${R2}/matches/${matchId}.json`)
| Component | Where | Role |
|-----------|-------|------|
| **Cloudflare Pages** | Cloudflare | Static site: SPA (HTML/JS/CSS) and pre-computed JSON index files. Updated every ~90 min by the index builder via `wrangler pages deploy`. Global CDN with automatic cache invalidation. |
| **Cloudflare R2** | Cloudflare | Warm replay cache: recent replays, match metadata, thumbnails, bot cards. Free tier ≤10 GB. Managed by the index builder (promote from B2, prune when approaching cap). |
| **Backblaze B2** | Backblaze | Cold archive: ALL replays and match data, permanently. Workers upload directly after each match. Free egress via Cloudflare Bandwidth Alliance. |
| **Backblaze B2** | Backblaze | Storage: ALL replays, match metadata, thumbnails, bot cards, and evolution status. Workers upload directly after each match. Served via Cloudflare CDN (Bandwidth Alliance = zero egress fees). |
| **Matchmaker** | Deployment (ai-code-battle ns) | Internal scheduler: computes pairings, enqueues jobs to Valkey, health checks bots, reaps stale jobs. No external exposure. |
| **PostgreSQL** | CNPG cluster (cnpg ns, `cnpg-apexalgo`) | Relational database — bot registry, match queue, ratings, results, series, seasons. Source of truth for structured data. |
| **Valkey** | Cluster service | Job queue (`acb:jobs:pending`), ephemeral caching. |
| **Match Workers** | Deployment (ai-code-battle ns) | Stateless match execution — BRPOP from Valkey, run simulation, upload replay to B2, write result to PostgreSQL. |
| **Bot Containers** | Deployments + Services (ai-code-battle ns) | Strategy bots (x6) + evolved bots (0-50) — HTTP servers called by workers during matches via cluster-internal Service DNS. |
| **Evolver** | Deployment (ai-code-battle ns) | Evolution pipeline — reads lineage/meta from PostgreSQL, generates candidates, writes evolution data to PostgreSQL. |
| **Index Builder** | Deployment (ai-code-battle ns) | Sleep-loop (15 min cycle). Reads PostgreSQL, generates JSON indexes, deploys to Pages. Promotes recent replays from B2 to R2 warm cache. Prunes R2 to stay within free tier. Self-restarts every 4h. |
| **Index Builder** | Deployment (ai-code-battle ns) | Sleep-loop (15 min cycle). Reads PostgreSQL, generates JSON indexes, deploys to Pages. Self-restarts every 4h. |
| **Go API** | Deferred | Social features (predictions, comments, voting) and third-party bot registration. Not required for v1. |
| **ArgoCD** | Cluster (argocd ns) | GitOps: syncs all K8s manifests from git. All deployments are declarative. |
| **Argo Workflows** | Cluster (argo ns) | CI pipelines: builds container images, pushes to Forgejo registry, builds static site. |
@ -1174,26 +1134,18 @@ encoding — only recording events that changed from the previous turn.
### 7.2 Storage
Replays use **tiered storage**: Backblaze B2 is the permanent cold archive
(all replays, forever), and Cloudflare R2 is a warm cache for recent replays
(free tier, ≤10 GB). Pre-computed JSON index files are deployed to
**Cloudflare Pages** by the index builder. No PersistentVolumes are used for
web-facing data.
All replay files, match metadata, thumbnails, bot cards, and evolution status
files are stored in **Backblaze B2** and served via Cloudflare CDN (Bandwidth
Alliance). Pre-computed JSON index files are deployed to **Cloudflare Pages**
by the index builder. No PersistentVolumes are used for web-facing data.
**B2 data layout** (cold archive — all data, permanently):
**B2 data layout** (all data — served via Cloudflare CDN):
```
replays/{match_id}.json.gz # ALL replay files
matches/{match_id}.json # ALL per-match metadata
thumbnails/{match_id}.png # ALL match thumbnails
cards/{bot_id}.png # ALL bot profile card images
```
**R2 data layout** (warm cache — recent subset, ≤10 GB):
```
replays/{match_id}.json.gz # recent replay files (promoted from B2)
matches/{match_id}.json # recent per-match metadata
thumbnails/{match_id}.png # recent match thumbnails
cards/{bot_id}.png # bot profile card images
evolution/live.json # evolver status (updated each cycle)
```
**Pages data layout** (static site):
@ -1215,36 +1167,23 @@ maps/{map_id}.json # map definitions
**How data flows:**
1. Match worker completes a match → uploads `replays/{match_id}.json.gz`
and `matches/{match_id}.json` to **B2** (cold archive, via S3-compatible
API)
and `matches/{match_id}.json` to **B2** (via S3-compatible API)
2. Worker writes match result to **PostgreSQL** (scores, ratings, metadata)
3. Index builder (every ~15 min) reads new results from PostgreSQL, rebuilds
all JSON index files, deploys to **Pages** via `wrangler pages deploy`
4. Index builder promotes recent replays from **B2 to R2** (warm cache)
5. Index builder prunes oldest replays from **R2** when approaching 10 GB
6. Browser loads SPA + indexes from Pages, fetches replays from R2 (warm)
with B2 fallback (cold)
4. Browser loads SPA + indexes from Pages, fetches replays and match data
directly from **B2** (via Cloudflare CDN)
**Tiered retention:**
- **B2 (cold)**: All replays retained permanently. No pruning. The canonical
archive.
- **R2 (warm)**: Recent replays, capped at ≤10 GB. The index builder prunes
by age when approaching the cap — oldest replays are removed from R2
(they remain in B2).
- **Exemptions from R2 pruning**: Replays referenced by playlists ("Closest
Finishes", "Biggest Upsets", etc.), rivalry pages, series, or season
archives are kept warm longer. The pruning job checks the exemption list
from PostgreSQL before removing from R2.
**Retention:**
- **B2**: All replays retained permanently. No pruning. The canonical
store.
- **PostgreSQL**: Match metadata retained indefinitely (rows are small).
- Index files are append-with-rotation: `index.json` holds the last 1000;
older pages at `index-{page}.json`.
**Storage costs:**
- **R2 (warm cache)**: ≤10 GB free tier. At ~50 KB/replay, holds ~200K warm
replays (~139 days at 60 matches/hour). Class A writes: ~43K/month
(index builder promoting replays). Well within free tier limits.
- **B2 (cold archive)**: First 10 GB free, $0.006/GB/month after. At 60
matches/hour, ~2.2 GB/month. Year one: ~26 GB ≈ $0.10/month. Free
- **B2**: First 10 GB free, $0.006/GB/month after. At 60 matches/hour,
~2.2 GB/month for replays. Year one: ~26 GB ≈ $0.10/month. Free
egress via Cloudflare Bandwidth Alliance.
- **Pages**: No per-file storage costs. 20K file limit per deployment — only
SPA + JSON indexes, well within limits.
@ -1255,14 +1194,14 @@ The replay viewer is a client-side TypeScript application rendered on
HTML5 Canvas.
**Rendering pipeline:**
1. Fetch `replay.json.gz` from R2 warm cache (falling back to B2 cold
archive); browser handles gzip decompression via `Accept-Encoding`
1. Fetch `replay.json.gz` from B2 (via Cloudflare CDN); browser handles
gzip decompression via `Accept-Encoding`
2. Parse and index: build per-turn game state by replaying events from turn 0
3. Render the current turn to canvas
4. User controls advance/rewind the turn index
No API invocations — the viewer is a static page (served from Pages) loading
a replay file from R2 (warm cache) or B2 (cold archive).
a replay file from B2 (via Cloudflare CDN).
**Visual design:**
@ -1299,11 +1238,11 @@ replay viewer is the landing page for any match. No login required to watch.
The web platform spans **Cloudflare** (static site and data files) and the
**apexalgo-iad** Kubernetes cluster (API, compute, databases). **Cloudflare
Pages** serves the SPA globally. **Cloudflare R2** stores replays and
per-match data. The **Go API Deployment** on K8s handles registration, job
coordination, and scheduling logic. **CNPG PostgreSQL** stores all relational
data. **Valkey** provides the job queue and caching. No PersistentVolumes
are used for web-facing data.
Pages** serves the SPA globally. **Backblaze B2** (via Cloudflare CDN) stores
replays and all per-match data. The **Go API Deployment** on K8s handles
registration, job coordination, and scheduling logic. **CNPG PostgreSQL** stores
all relational data. **Valkey** provides the job queue and caching. No
PersistentVolumes are used for web-facing data.
### 8.1 Cloudflare Pages (Static Site)
@ -1316,9 +1255,9 @@ single Pages project. The index builder on K8s updates the data files every
/ → Landing page, featured replays, leaderboard summary
/leaderboard → Full leaderboard (fetches data/leaderboard.json from Pages)
/matches → Match history (fetches data/matches/index.json from Pages)
/replay/{match_id} → Replay viewer (fetches replays/{match_id}.json.gz from R2)
/replay/{match_id} → Replay viewer (fetches replays/{match_id}.json.gz from B2)
/bot/{bot_id} → Bot profile (fetches data/bots/{bot_id}.json from Pages)
/evolution → Evolution dashboard (fetches data/evolution/*.json from Pages + evolution/live.json from R2)
/evolution → Evolution dashboard (fetches data/evolution/*.json from Pages + evolution/live.json from B2)
/register → Bot registration form (submits to Go API on K8s)
/docs → Protocol spec, starter kit links, getting started
```
@ -1333,8 +1272,8 @@ No build-time data fetching -- all data loaded at runtime.
```js
// SPA shell + index data from Pages (same origin)
const leaderboard = await fetch('/data/leaderboard.json').then(r => r.json())
// Replays from R2 (cross-origin, CORS enabled on R2 bucket)
const replay = await fetch(`https://r2.aicodebattle.com/replays/${matchId}.json.gz`)
// Replays from B2 (cross-origin, CORS enabled on B2 bucket)
const replay = await fetch(`https://b2.aicodebattle.com/replays/${matchId}.json.gz`)
// Dynamic operations from K8s API
const result = await fetch('https://api.aicodebattle.com/api/register', { method: 'POST', body: ... })
```
@ -1343,7 +1282,7 @@ Index JSON files are rebuilt and deployed to Pages every ~15 minutes by
the index builder (with actual Pages deploys batched every ~90 minutes to
stay well within Cloudflare's deploy limits). Visitors see index data that
is at most ~90 minutes old. Replays and per-match metadata are uploaded to
R2 in real time by match workers and available immediately.
B2 in real time by match workers and available immediately.
### 8.2 Go API Service
@ -1592,16 +1531,10 @@ start → build indexes → sleep 15m → build indexes → sleep 15m → ...
build) and deploys to Cloudflare Pages via `wrangler pages deploy`. Only
the `data/` directory changes between deploys; the SPA shell updates
only when a new site build is triggered.
4. **Prune on R2:** On one cycle per week (checked via day-of-week), runs
replay pruning — lists objects in R2's `replays/` prefix older than
90 days, queries PostgreSQL for exempt match IDs, deletes non-exempt
objects via S3 DeleteObjects API.
**Environment:** The Deployment Pod has PostgreSQL credentials and a
**Cloudflare API token** (for `wrangler pages deploy`) stored as
SealedSecrets. It also has R2 credentials (S3-compatible access key) for
the weekly pruning cycle. No PersistentVolumes needed — all output goes to
Cloudflare.
SealedSecrets. No PersistentVolumes needed — all output goes to Cloudflare.
### 8.5 Bot Registration
@ -1687,8 +1620,8 @@ time range, human-only vs all). Auto-refresh every 60 seconds. Public
- Each entry: match_id, participants, scores, date, link to replay
**Match detail** (`/replay/{match_id}`):
- Fetches `matches/{match_id}.json` from R2 for metadata
- Fetches `replays/{match_id}.json.gz` from R2 for the replay
- Fetches `matches/{match_id}.json` from B2 for metadata
- Fetches `replays/{match_id}.json.gz` from B2 for the replay
- Embedded replay viewer (auto-plays)
- Score breakdown, participants, match duration
@ -1700,21 +1633,21 @@ time range, human-only vs all). Auto-refresh every 60 seconds. Public
Compute runs in the **apexalgo-iad** Kubernetes cluster (Rackspace Spot) in
a dedicated `ai-code-battle` namespace. The public-facing product is a
**Cloudflare Pages** static site. Replay files use tiered storage:
**Cloudflare R2** (warm cache, free tier ≤10 GB) and **Backblaze B2** (cold
archive, permanent). The cluster is existing infrastructure shared with other
workloads — it already provides PostgreSQL (CNPG), Valkey, Traefik ingress,
cert-manager, ArgoCD, Argo Workflows, Argo Events, Forgejo (git + container
registry), SATA (Cinder CSI) storage, and Sealed Secrets.
**Cloudflare Pages** static site. All replay files and match data are stored
in **Backblaze B2** and served via Cloudflare CDN (Bandwidth Alliance).
The cluster is existing infrastructure shared with other workloads — it
already provides PostgreSQL (CNPG), Valkey, Traefik ingress, cert-manager,
ArgoCD, Argo Workflows, Argo Events, Forgejo (git + container registry),
SATA (Cinder CSI) storage, and Sealed Secrets.
Key principles:
- **Static-first architecture** — the public product is a static site on
Pages. All data visitors see is pre-computed JSON. K8s is the factory
that generates data and publishes it to Pages.
- **Tiered storage** — B2 is the permanent archive for all replays. R2 is
a warm CDN cache for recent replays, capped at the free tier. The index
builder manages the R2 lifecycle.
- **B2 as single storage layer** — B2 stores all replays, match metadata,
thumbnails, bot cards, and evolution status. Served directly via Cloudflare
CDN (Bandwidth Alliance = zero egress fees). No warm cache tier needed.
- **GitOps via ArgoCD** — all K8s manifests are committed to git and synced
by ArgoCD. Never apply manifests directly with `kubectl`.
- **Argo Workflows for CI** — container image builds and static site builds
@ -1744,8 +1677,6 @@ Cross-namespace dependencies:
- **Cloudflare Pages project**: `ai-code-battle` (`ai-code-battle.pages.dev`)
— hosts the static SPA and data indexes. Deployed by the index builder
via `wrangler pages deploy`.
- **Cloudflare R2 bucket**: Warm cache for recent replays, thumbnails, bot
cards. Free tier (≤10 GB). Managed by the index builder.
- **DNS** (when custom domain is desired): `aicodebattle.com` CNAME to Pages.
**Backblaze B2 infrastructure requirements:**
@ -1763,7 +1694,7 @@ declarative-config/k8s/apexalgo-iad/ai-code-battle/
├── acb-schema-init.yml (ConfigMap + Deployment for schema migration)
├── acb-matchmaker-deployment.yml (matchmaker: pairings, job enqueue, health, reaper)
├── acb-worker-deployment.yml (match workers: run matches, upload to B2)
├── acb-index-builder-deployment.yml (index builder: generate JSON, deploy to Pages, manage R2)
├── acb-index-builder-deployment.yml (index builder: generate JSON, deploy to Pages)
├── acb-evolver-deployment.yml (LLM evolution pipeline)
├── acb-strategy-random-deployment.yml (RandomBot — Python)
├── acb-strategy-random-service.yml
@ -1777,7 +1708,7 @@ declarative-config/k8s/apexalgo-iad/ai-code-battle/
├── acb-strategy-swarm-service.yml
├── acb-strategy-hunter-deployment.yml (HunterBot — Java)
├── acb-strategy-hunter-service.yml
├── acb-secret.yml.template (template: B2/R2/Cloudflare/bot secrets)
├── acb-secret.yml.template (template: B2/Cloudflare/bot secrets)
└── acb-sealedsecret.yml (sealed version of above)
```
@ -1796,7 +1727,7 @@ container registry (`forgejo.ardenone.com/ai-code-battle/<image>`).
| `acb-matchmaker` | Go binary on Alpine | Matchmaking, health checks, stale job reaping | Deployment (1 replica) |
| `acb-worker` | Go binary on Alpine | Match execution, B2 upload | Deployment (2-10 replicas) |
| `acb-evolver` | Go binary on Alpine | Evolution pipeline | Deployment (1 replica) |
| `acb-index-builder` | Go binary on Alpine (includes `wrangler` CLI) | Reads PostgreSQL, generates JSON indexes, deploys to Pages, manages R2 warm cache (promote from B2, prune) | Deployment (sleep-loop, 15 min cycle, Pages deploy every ~90 min, self-restarts every 4h) |
| `acb-index-builder` | Go binary on Alpine (includes `wrangler` CLI) | Reads PostgreSQL, generates JSON indexes, deploys to Pages | Deployment (sleep-loop, 15 min cycle, Pages deploy every ~90 min, self-restarts every 4h) |
| `acb-strategy-random` | Python 3.13 slim | RandomBot | Deployment (1 replica) |
| `acb-strategy-gatherer` | Go on Alpine | GathererBot | Deployment (1 replica) |
| `acb-strategy-rusher` | Rust on Alpine | RusherBot | Deployment (1 replica) |
@ -1826,8 +1757,7 @@ Match workers coordinate via **Valkey** (job queue) and **PostgreSQL**
7. Worker writes result directly to **PostgreSQL** (scores, winner, turn
count, condition) and updates ratings (Glicko-2)
8. Index builder (next ~15-min cycle) reads new results from PostgreSQL,
rebuilds all index JSON files, deploys to Pages, promotes recent
replays from B2 to R2 warm cache
rebuilds all index JSON files, deploys to Pages
**Stale job recovery:**
- Reaper ticker (in matchmaker) checks PostgreSQL every 5 minutes for
@ -1867,11 +1797,10 @@ Match workers coordinate via **Valkey** (job queue) and **PostgreSQL**
**External traffic:**
- `ai-code-battle.pages.dev` (or custom domain) → Cloudflare Pages
(static SPA + data indexes)
- R2 public URL → Cloudflare R2 (warm replay cache)
- B2 public URL (via Cloudflare CDN) → Backblaze B2 (cold replay archive)
- B2 public URL (via Cloudflare CDN) → Backblaze B2 (replay/match data storage)
- No K8s services are exposed externally in v1. The Go API IngressRoute
at `api.aicodebattle.com` is planned for when social features are added.
- TLS: Pages and R2 handle TLS automatically. B2 via Cloudflare gets TLS
- TLS: Pages handles TLS automatically. B2 via Cloudflare CDN gets TLS
from the CDN layer.
**Cluster-internal traffic:**
@ -1884,8 +1813,7 @@ Match workers coordinate via **Valkey** (job queue) and **PostgreSQL**
- Workers -> B2: HTTPS (S3-compatible API, for replay upload)
- Index Builder -> PostgreSQL: same
- Index Builder -> Cloudflare Pages: HTTPS (wrangler CLI)
- Index Builder -> R2: HTTPS (S3-compatible API, for warm cache management)
- Index Builder -> B2: HTTPS (S3-compatible API, for promoting replays to R2)
- Index Builder -> B2: HTTPS (S3-compatible API, read stats for index building)
- Evolver -> PostgreSQL: same
- All cluster-internal traffic is plaintext (trusted network)
@ -1896,9 +1824,9 @@ Match workers coordinate via **Valkey** (job queue) and **PostgreSQL**
- PostgreSQL credentials in SealedSecrets (encrypted in git, decrypted
in-cluster)
- Valkey access is cluster-internal only (no external exposure)
- Pages, R2, and B2 serve public-read data only — no secrets stored there
- Pages and B2 serve public-read data only — no secrets stored there
- B2 write credentials (SealedSecret) are scoped to the acb bucket
- Cloudflare API token (SealedSecret) is scoped to Pages deploy + R2 only
- Cloudflare API token (SealedSecret) is scoped to Pages deploy only
- NetworkPolicy can restrict egress from bot pods to prevent data
exfiltration (future hardening)
@ -1986,7 +1914,6 @@ appropriate Argo Workflow(s).
| Worker queue depth | Valkey LLEN on `acb:jobs:pending` | >50 pending for >30 min |
| Bot health failures | Matchmaker health checker ticker | >50% failing |
| Stale jobs | Matchmaker reaper ticker count | >10 stale in a cycle |
| R2 usage | Index builder tracks warm cache size | >8 GB (approaching 10 GB free tier cap) |
| B2 usage | B2 dashboard / API metrics | Informational only (no cap) |
Alerts via matchmaker -> webhook to Discord/Slack. Cluster-level monitoring
@ -2485,7 +2412,7 @@ ai-code-battle/
│ ├── acb-mapgen/ # CLI: generate symmetric maps
│ ├── acb-worker/ # Container: match execution worker
│ ├── acb-evolver/ # Container: LLM evolution pipeline
│ ├── acb-index-builder/ # Container: PostgreSQL -> JSON -> Pages + R2 pruning
│ ├── acb-index-builder/ # Container: PostgreSQL -> JSON -> Pages
│ # replay pruning is handled by acb-index-builder (weekly cycle)
├── cmd/acb-api/ # Go API service (replaces Cloudflare Worker)
@ -2549,7 +2476,7 @@ ai-code-battle/
│ │ │ ├── blog-post.ts # Markdown renderer for blog content
│ │ │ └── skeleton.ts # Per-page skeleton screens
│ │ ├── lib/
│ │ │ ├── data.ts # Data fetching from Pages (same origin) + R2 (cross-origin), caching
│ │ │ ├── data.ts # Data fetching from Pages (same origin) + B2 (cross-origin), caching
│ │ │ ├── preload.ts # Hover preload + route cache
│ │ │ ├── disclosure.ts # Progressive feature revelation (XP)
│ │ │ ├── accessibility.ts # Color palettes, keyboard shortcuts
@ -2638,7 +2565,7 @@ ai-code-battle/
| `acb-api` | `cmd/acb-api/` | Go on Alpine | API + scheduling | Deployment (1 replica) |
| `acb-worker` | `cmd/acb-worker/` | Go on Alpine | Match execution | Deployment (2-10 replicas) |
| `acb-evolver` | `cmd/acb-evolver/` | Go on Alpine | LLM evolution pipeline | Deployment (1 replica, self-restarts every 4h) |
| `acb-index-builder` | `cmd/acb-index-builder/` | Go on Alpine (includes wrangler) | PostgreSQL -> JSON -> Cloudflare Pages + weekly R2 replay pruning | Deployment (sleep-loop) |
| `acb-index-builder` | `cmd/acb-index-builder/` | Go on Alpine (includes wrangler) | PostgreSQL -> JSON -> Cloudflare Pages | Deployment (sleep-loop) |
| `acb-strategy-random` | `bots/random/` | Python 3.13 slim | RandomBot | Deployment (1 replica) |
| `acb-strategy-gatherer` | `bots/gatherer/` | Go on Alpine | GathererBot | Deployment (1 replica) |
| `acb-strategy-rusher` | `bots/rusher/` | Rust on Alpine | RusherBot | Deployment (1 replica) |
@ -2717,6 +2644,7 @@ Index builder Deployment (every ~15 min build, ~90 min deploy):
├── Query PostgreSQL directly
├── Generate JSON index files
└── Deploy to Cloudflare Pages via wrangler pages deploy
```
### 11.4 Shared Libraries & Code Reuse
@ -2797,7 +2725,7 @@ match with all visual elements rendering correctly.
`rating_history` tables in the `acb` database
- Index builder Deployment (`acb-index-builder`): reads PostgreSQL directly,
generates index JSON files every ~15 min, deploys to Cloudflare Pages
every ~90 min, manages R2 warm cache (promote from B2, prune old)
every ~90 min
- Match worker Deployment (`acb-worker`): BRPOPs jobs from Valkey, runs
matches, uploads replays to B2, writes results + Glicko-2 ratings to
PostgreSQL
@ -2813,13 +2741,12 @@ System recovers from worker pod failure via the stale job reaper.
**Deliverables:**
- Cloudflare Pages static SPA (`ai-code-battle.pages.dev`): leaderboard,
match history, bot profiles, replay viewer, docs/getting-started page
- SPA fetches replay files from R2 warm cache with B2 cold archive fallback
- SPA fetches replay files directly from B2 (via Cloudflare CDN)
- Index builder deploying leaderboard, bot profiles, playlists to Pages
- Match workers uploading replays and per-match metadata to B2
- Index builder promoting recent replays from B2 to R2 warm cache
**Exit criteria:** anyone can browse matches, view leaderboards, and watch
replays — SPA from Pages, recent replays from R2, old replays from B2.
replays — SPA from Pages, replay/match data from B2.
All data is static and pre-computed.
### Phase 6: Deployment & Production
@ -2829,20 +2756,19 @@ All data is static and pre-computed.
namespace, Deployments, Services, SealedSecrets (flat directory structure)
- Cloudflare Pages project (`ai-code-battle`, already exists at
`ai-code-battle.pages.dev`)
- Cloudflare R2 bucket (warm replay cache, free tier)
- Backblaze B2 bucket (cold replay archive)
- Backblaze B2 bucket (replay/match data storage)
- ArgoCD Application syncing the manifests directory
- Argo Events sensor: GitHub webhook triggers on push to `ai-code-battle` repo
- Argo Workflows: image build (Kaniko -> Forgejo registry), site build
(npm build -> artifact for Pages deploy)
- SealedSecrets for PostgreSQL, Valkey, B2, R2, Cloudflare API token
- SealedSecrets for PostgreSQL, Valkey, B2, Cloudflare API token
(most already provisioned in the namespace)
- Monitoring: matchmaker metrics endpoint + Discord/Slack alerting webhooks
**Exit criteria:** platform is publicly accessible — SPA from Pages, recent
replays from R2, old replays from B2. All K8s manifests are GitOps-managed
by ArgoCD, CI pipelines rebuild images and site on git push, matches run
autonomously, and the leaderboard updates every ~90 minutes.
**Exit criteria:** platform is publicly accessible — SPA from Pages, replay/match
data from B2. All K8s manifests are GitOps-managed by ArgoCD, CI pipelines
rebuild images and site on git push, matches run autonomously, and the
leaderboard updates every ~90 minutes.
### Phase 7: LLM-Driven Evolution
@ -2924,12 +2850,12 @@ event timelines, and shareable bot profile cards.
**Deliverables:**
- Weekly meta report: auto-generated blog post deployed to Pages,
rendered on `/blog` with LLM-enhanced narrative sections
- Public match data: documented static JSON file paths on Pages and R2,
- Public match data: documented static JSON file paths on Pages and B2,
OpenAPI-style documentation at `/docs/api`, versioned replay format spec
- Accessibility suite: Tol color-blind palette + shape-per-player, keyboard
shortcuts for replay viewer, high contrast mode, reduced motion, screen
reader transcript, focus indicators
- Live evolution observatory: evolver writes `live.json` to R2
- Live evolution observatory: evolver writes `live.json` to B2
every cycle, observatory page polls and renders live feed + lineage tree
+ meta shift chart
- Narrative engine: weekly story arc detection, LLM-generated 200-word
@ -4290,17 +4216,18 @@ auto-generated.
### 15.2 Public Match Data (Static JSON)
All platform data is pre-computed and stored as static JSON files, split
between **Cloudflare Pages** (indexes) and **Cloudflare R2** (replays and
per-match data). Index files are rebuilt every ~15 min by the index builder
and deployed to Pages every ~90 min. Replays and per-match data are uploaded
to R2 in real time by match workers. The "API" is simply **documented file
paths** -- no dynamic endpoints, no query parameters, no rate limiting needed.
between **Cloudflare Pages** (indexes) and **Backblaze B2** (replays and
per-match data, served via Cloudflare CDN). Index files are rebuilt every ~15 min
by the index builder and deployed to Pages every ~90 min. Replays and per-match
data are uploaded to B2 in real time by match workers. The "API" is simply
**documented file paths** -- no dynamic endpoints, no query parameters, no
rate limiting needed.
**Documented data paths:**
```
PAGES = https://aicodebattle.com (Cloudflare Pages)
R2 = https://r2.aicodebattle.com (Cloudflare R2)
B2 = https://b2.aicodebattle.com (Backblaze B2 via Cloudflare CDN)
API = https://api.aicodebattle.com (K8s Go API, dynamic only)
--- Index files on Pages (deployed every ~90 min by index builder) ---
@ -4348,22 +4275,22 @@ Maps:
GET {PAGES}/maps/index.json
GET {PAGES}/maps/{map_id}.json
--- Real-time data on R2 (written by workers/evolver) ---
--- Data on B2 (written by workers/evolver, served via Cloudflare CDN) ---
Individual match metadata:
GET {R2}/matches/{match_id}.json
GET {B2}/matches/{match_id}.json
Replays:
GET {R2}/replays/{match_id}.json.gz
GET {B2}/replays/{match_id}.json.gz
Evolution (live feed):
GET {R2}/evolution/live.json
GET {B2}/evolution/live.json
Thumbnails:
GET {R2}/thumbnails/{match_id}.png
GET {B2}/thumbnails/{match_id}.png
Bot cards:
GET {R2}/cards/{bot_id}.png
GET {B2}/cards/{bot_id}.png
```
**Replay format specification:**
@ -4387,7 +4314,7 @@ no rate limiting -- it's just static files served by Cloudflare.
**Why static JSON, not a dynamic API:**
All this data already exists as static files on Pages and R2. The index
All this data already exists as static files on Pages and B2. The index
builder Deployment already produces leaderboard.json, bot profiles, match
indexes, playlists, etc. Adding a dynamic API layer would add complexity
for data that's already pre-computed and publicly readable. Cloudflare
@ -4395,7 +4322,7 @@ serves these globally with automatic CDN caching.
Third-party tools just `fetch()` the URLs. If they need to poll for
updates, they check the `updated_at` field in each JSON file. Index files
on Pages refresh every ~90 minutes. R2 cache headers guide freshness for
on Pages refresh every ~90 minutes. B2 cache headers guide freshness for
replays (immutable) and the evolution live feed (10s).
### 15.3 Accessibility Suite
@ -4509,18 +4436,18 @@ rejected, and promoted.
**Data flow:**
The evolver Deployment writes a status file to R2 at each stage of every
The evolver Deployment writes a status file to B2 at each stage of every
evolution cycle:
```
Upload to R2: evolution/live.json
Served as: https://r2.aicodebattle.com/evolution/live.json
Upload to B2: evolution/live.json
Served as: https://b2.aicodebattle.com/evolution/live.json
```
Updated at every state transition: generation start, validation
complete, each evaluation match result, promotion decision. At ~15
minutes per cycle with ~5 state transitions, that's ~20 R2 writes
per hour. R2 serves the file with `Cache-Control: max-age=10`.
minutes per cycle with ~5 state transitions, that's ~20 B2 writes
per hour. B2 serves the file via Cloudflare CDN with `Cache-Control: max-age=10`.
**`live.json` schema:**
@ -4654,7 +4581,7 @@ Each entry shows the candidate ID, island, result, and reason.
Both visualizations are built from `data/evolution/lineage.json` and
`data/evolution/meta.json` (served from Pages, produced by the index builder
Deployment). The live feed overlay is the only component that polls
`evolution/live.json` (written by the evolver to R2).
`evolution/live.json` (written by the evolver to B2).
### 15.5 Narrative Engine (Chronicles)
@ -4990,7 +4917,7 @@ downloads replay viewer or WASM code.
**Data fetching:**
All data files are served by Cloudflare (Pages for indexes, R2 for replays) with appropriate cache behavior.
All data files are served by Cloudflare (Pages for indexes, B2 via Cloudflare CDN for replays and match data) with appropriate cache behavior.
Subsequent visits hit the edge cache. The SPA uses `stale-while-revalidate`
pattern: show cached data immediately, fetch fresh data in the background,
update the UI when it arrives. The leaderboard never shows a loading