acb-index-builder has been in CrashLoopBackOff for 45 days with silent crashes
after "Copied web assets to output directory". Investigation revealed O(n²) N+1
query loops causing unbounded memory growth and OOMKill.
Changes:
- fetchSeries: batch games query (1000 queries → 1 query) with LIMIT 10000
- fetchChampionshipBracket: batch games query (500 queries → 1 query) with LIMIT 64
- fetchSeasonSnapshots: reduce LIMIT from 10000 to 500
- fetchLineage: reduce LIMIT from 10000 to 1000
- Add strings import for strings.Join in batch queries
These changes prevent the pod from being OOMKilled during fetchAllData() which
runs after copyWebAssets() in the build cycle.
Co-Authored-By: Claude <noreply@anthropic.com>
- Reduce fetchBots LIMIT from 10000 to 2000
- Reduce fetchRatingHistory LIMIT from 10000 to 5000
- Reduce fetchFeedback LIMIT from 5000 to 1000
- Fix O(n²) participant name lookup in generateBotProfiles by using botNameMap
- Add panic recovery in runBuildCycle to log panics via slog before crashing
- Add R2/B2 client helper functions in s3.go
This fixes acb-index-builder CrashLoopBackOff caused by OOMKill after
web asset copy. The pod was silently crashing during fetchAllData()
due to unbounded query results consuming all memory.
Co-Authored-By: Claude <noreply@anthropic.com>
The bot match stats query was introduced in b35a2aa to fix an N+1 query
problem, but it was unbounded and could return an unlimited number of rows.
With many bots in the database, this query could consume excessive memory
and cause OOMKill, resulting in silent crashes after 'Copied web assets'.
Add LIMIT 20000 to prevent unbounded result sets while supporting large
bot populations (the main bots query already limits to 10000 bots).
This fix continues the pattern of adding LIMITs to prevent OOMKill crashes
in acb-index-builder.
Fixes bead bf-2ws: acb-index-builder CrashLoopBackOff investigation
The previous implementation called getBotMatchStats for each bot in a loop,
causing 10,000+ separate database queries when there are many bots. This N+1
query problem caused the pod to exceed memory limits and get OOMKilled,
resulting in CrashLoopBackOff.
Replaced with a single batch query that fetches match stats for all bots at
once, then maps the results to each bot. This reduces database round trips
from O(n) to O(1).
Fixes bead bf-2ws: acb-index-builder CrashLoopBackOff (silent crash after web asset copy)
The fetchSeriesGames function was querying all games for a series without a limit.
With up to 1000 series being fetched, and potentially many games per series,
this could return an unbounded number of rows and cause OOMKill.
A typical series has 3-7 games (best-of-5 or best-of-7), so LIMIT 100 is
more than sufficient to handle edge cases while preventing memory exhaustion.
Fixes acb-index-builder CrashLoopBackOff caused by OOMKill after web asset copy.
The query in fetchRecentMatchIDs was fetching all completed matches from
the last 24 hours without a LIMIT clause. In a high-traffic environment
with thousands of matches per day, this would cause the pod to run out
of memory and be OOMKilled.
This fix adds LIMIT 5000 to cap the number of recent matches fetched,
preventing unbounded memory growth while still providing sufficient
data for warm asset bundling.
Fixes acb-index-builder CrashLoopBackOff (4713 restarts over 45 days).
The generateBotProfiles function had two nested loops that caused O(n²) memory usage:
- Iterating through all rating history entries (10,000) for each bot (10,000) = 100M iterations
- Iterating through all matches (1,000) for each bot (10,000) = 10M iterations
This caused acb-index-builder to run out of memory and get OOMKilled during the build cycle.
Fixed by pre-building lookup maps (O(n) build + O(1) lookup):
- historyMap[botID] -> []RatingHistoryEntry
- matchMap[botID] -> []MatchSummary
Reduces complexity from O(bots × matches) to O(matches + bots) for lookups.
Resolves acb-index-builder CrashLoopBackOff after 45 days of failure.
- Add LIMIT 10000 to fetchSeasonSnapshots (season_snapshots per season)
- Add LIMIT 500 to fetchChampionshipBracket (series per season bracket)
These queries were called in a loop for each season without LIMITs,
causing acb-index-builder to be OOMKilled with 512Mi memory limit.
Fixes OOMKill after web asset copy in build cycle.
The fetchOpenPredictions function had an unbounded query building a pair
frequency map for rivalry detection. With thousands of bots and matches,
this could return tens of thousands of rows and cause OOMKill.
- Add ORDER BY COUNT(*) DESC to prioritize most common pairings
- Add LIMIT 1000 - sufficient to detect rivalries (pairs with >= 3 matches)
This fixes the 45-day CrashLoopBackOff with 4700+ restarts.
Co-Authored-By: Claude <noreply@anthropic.com>
- Add LIMIT 100 to island populations query (fetchEvolutionMeta)
- Add LIMIT 10000 to lineage programs query (fetchLineage)
These queries had no row limits, causing OOMKill when the programs table
grew large. The pod crashed silently after "Copied web assets" because
Go panics and OOMKills exit without logging to slog.
Fixes acb-index-builder CrashLoopBackOff (4700+ restarts, 45 days).
- Add LIMIT 1000 to fetchChampionshipBracket (was unbounded)
- Reduce fetchSeries from LIMIT 5000 to LIMIT 1000
- Reduce fetchLineage from LIMIT 50000 to LIMIT 10000
- Reduce fetchFeedback from LIMIT 5000 to LIMIT 1000
- Reduce fetchRatingHistory from LIMIT 10000 to LIMIT 5000
The acb-index-builder pod has been in CrashLoopBackOff with OOMKill
(exit code 137) for 45 days with 4713 restarts. These unbounded queries
were loading too much data into memory, causing the kernel to kill the
process before any logs could be written.
Co-Authored-By: Claude <noreply@anthropic.com>
Added LIMIT clauses to 4 unbounded queries that were causing
acb-index-builder to crash with OOMKill after copying web assets:
- fetchPredictorStats: LIMIT 100 (was loading all predictor stats)
- fetchMaps: LIMIT 500 (was loading all maps)
- fetchSeasonSnapshots: LIMIT 1000 (was loading all season snapshots)
- fetchSeasons: LIMIT 100 (was loading all seasons)
These queries had ORDER BY but no LIMIT, causing them to load
massive datasets into memory on each build cycle, leading to
container OOM after the web asset copy phase.
Fixes bead bf-2ws
- Add LIMIT 50000 to fetchLineage (evolution programs table)
- Add LIMIT 10000 to fetchBots
- Add LIMIT 5000 to fetchSeries
These queries had no bounds and could grow arbitrarily large,
causing acb-index-builder to OOM during build cycles.
The lineage table in particular grows unbounded with evolution.
Fixes CrashLoopBackOff that has persisted for 45 days.
- Add matches_today and active_bots fields to LiveData Totals (evolver)
- Query matches table for COUNT(*) WHERE completed_at >= today
- Query bots table for COUNT(*) WHERE status = 'active'
- Add fields to index builder EvolutionMeta struct
- Update homepage to render "X matches today · Y bots active · Gen #Z evolving"
- Add CSS styling for .home-live-stats section
Closes: bf-4m8mo
Index builder:
- Add slog import for structured logging
- Improve fetchEvolutionMeta to return empty meta instead of error when programs table is empty
- Add logging to show evolution system status (running vs not initialized)
- Add logging in generateEvolutionMeta to show when evolution data is written
Evolver:
- Add automatic schema initialization and population seeding in RunEvolutionLoop
- Programs table is now automatically seeded with 6 initial strategy bots on startup
- Log seeding status to indicate whether programs table was already initialized
These changes ensure the evolution system properly initializes when deployed
and provides better visibility into its status via structured logging.
Closes: bf-4zde
- Add island_populations: program count per island
- Add best_ratings: top 10 evolved bots with bot_id, name, rating, island, language
- Add total_promoted and promotion_rate: all-time promotion statistics
- Queries programs table and joins with bots table for current ratings
Closes: bf-1cxv
Adds MetaIndex struct and generateMetaIndex function to create
data/meta/index.json listing all available meta data files
(archetypes.json, rivalries.json) with descriptions.
Also adds the new file to the R2 warm cache upload list.
Closes: bf-66rk
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The plan previously specified ZoneShrinkStep=2, but the engine uses
ZoneShrinkStep=1 (per commit 0577fcd). The value of 1 was found to
improve combat density because it matches bot movement speed (1 tile/turn).
A value of 2 caused the zone to shrink faster than bots could move,
killing them before combat could occur.
Updated zone parameters table and rationale in §3.7.1.
Closes: bf-3mrj
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Modified bundleWarmAssetsForCycle to return the deduplicated match IDs
of warm-set replays, and added a call to promoteRecentReplays in the
build cycle. This implements plan §8.2.5: index builder promotes recent
replays from B2 cold archive to R2 warm cache.
Acceptance criteria for plan-gap bead bf-jfmo:
- /r2/replays/{id}.json.gz will return valid gzipped replay JSON once
the Cloudflare Pages R2 bucket binding (ACB_BUCKET) is configured
- R2 bucket exists with replay objects after each build cycle
- Index builder has R2 S3 client configured (credentials via env vars)
Closes: bf-jfmo
Per bead bf-3e60: instead of copying B2->R2 (warm cache), bundle warm-set
replays, thumbnails, cards, and evolution live.json directly into the Pages
deploy directory as static assets (dist/data/). This serves replays same-origin,
eliminating R2 dependency and 404 errors.
Changes:
- Add B2Client interface for testable B2 operations
- Add bundleWarmReplays(): copies replays/*.json.gz from B2 to dist/data/replays/
- Add bundleWarmThumbnails(): copies thumbnails/*.png from B2 to dist/data/thumbnails/
- Add bundleWarmCards(): copies cards/*.png from B2 to dist/data/cards/
- Add bundleEvolutionLive(): copies evolution/live.json from B2 to dist/data/evolution/
- Replace promoteRecentReplaysForCycle() with bundleWarmAssetsForCycle()
- Remove R2 pruning logic from main loop (no longer needed)
- Add unit tests for all bundling functions with mock B2 client
Replays are served gzipped (as-is from B2) to keep deploy size under Pages'
25MB file limit. Frontend will gunzip client-side (separate bead bf-5cwi).
All tests pass (go test ./...).
Closes: bf-3e60
Plan §15.4 Live Evolution Observatory requires data/evolution/meta.json
and data/evolution/lineage.json files, which the web platform expects
but the index-builder was not generating.
- Add EvolutionMeta type (generation, promoted_today, top_10_count, updated_at)
- Add LineageNode type (full program lineage with parent relationships)
- Add fetchEvolutionMeta() to query evolver database for stats
- Add fetchLineage() to query evolver programs table for lineage tree
- Add generateEvolutionMeta() to write data/evolution/meta.json
- Add generateLineage() to write data/evolution/lineage.json
- Wire generation into generateAllIndexes()
- Add files to R2 upload list
The implementation gracefully handles missing evolver database by
returning empty/placeholder data, allowing the index-builder to run
without evolution data while still producing valid JSON files.
Closes: bf-6cp0
The WinnerID field is a player-slot integer as string (e.g. "2"), not a bot_id.
The SQL query already computes the correct winner status in p.Won field.
Fixed in 3 functions:
- matchToSummary: Changed Won: p.BotID == m.WinnerID to Won: p.Won
- buildPlaylistMatch: Changed Won: p.BotID == m.WinnerID to Won: p.Won
- ratingUpsetMagnitude: Use p.Won to identify winner instead of comparing with m.WinnerID
- maxScoreDiff: Use p.Won to identify winner instead of comparing with m.WinnerID
- isEvolutionBreakthrough: Find winner using p.Won before checking if evolved
This fixes the issue where 984/1000 prod matches had winner_id set but all participants showed won: false.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix TestBuildNarrativePrompt_Comeback to check for current ELO
instead of old rating (comeback arc shows bottom 25%→top 25%)
- Fix TestDetectRivalryArcs to use 10+ matches (grudge match spec)
instead of only 5 matches
Story arc detection (per §3.7 chronicles):
✓ Comeback bots: recovered from bottom 25% to top 25%
✓ Grudge matches: same pair meets 10+ times
✓ Underdog victories: bottom-10 beats top-10
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Change filter from 'idea'/'mistake' to 'insight'/'idea' (mapping to 'hint'/'strategy' from plan §13.6)
- Increase upvote threshold from 3 to 10 for higher quality signals
- The evolver consumes community_hints.json for LLM prompt context
Add sitemap.xml generation as a final pass in the index builder. The
sitemap covers all public pages: home, leaderboard, bots list, bot
profiles, matches list, featured replays, seasons, rivalries,
predictions, and docs.
- Add SiteURL config field (ACB_SITE_URL env var, defaults to
https://aicodebattle.com)
- Add generateSitemap() function with proper XML encoding
- Add SitemapURL and Sitemap types for XML marshaling
- Call generateSitemap() at the end of generateAllIndexes()
- Write sitemap.xml to output directory alongside leaderboard.json
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds combat_turns metric (distinct turns where ≥1 bot died from enemy
focus-fire, excluding self-collisions). Worker computes it after each
match; index builder sorts matches/index.json and the new most-combat
playlist descending by it, and bumps interest score for combat-heavy
matches so they surface in highlights.
Also switches homepage featured replay default view from influence to
standard so the actual bot-on-bot combat is visible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_worker.js static file approach fails — Cloudflare rejects it when uploaded
as a static asset. Instead, copy web/functions/ into the image and set
wrangler CWD to /app/web/ so it discovers functions/ and uploads the Pages
Functions bundle correctly on every deploy cycle.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Every index-builder deploy was overwriting the production Pages deployment
without functions (wrangler ran from /tmp, no functions/ dir visible).
Compiling functions/ to dist/_worker.js during the Docker web-builder stage
means the worker is always included in every Pages deploy, regardless of CWD.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Verified /watch/replays shows real completed matches (not just demo)
- Match cards display bot names, turn count, winner badges, map ID
- 'Watch Replay' links point to real match IDs (m_test_*)
- Curated playlists render with real data (featured, comebacks, upsets, etc.)
- Pagination/infinite scroll works via IntersectionObserver
- Mobile testing on Pixel 6 via ADB: layout responsive, touch targets usable
- Created MATCH_LIST_TEST_RESULTS.md with full verification details
- Thumbnails not implemented (clean UI without broken images due to R2 issues)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The AWS SDK requires a valid AWS region name even when using custom
S3-compatible endpoints (ARMOR/B2). Using "auto" as the region causes
an error: "Invalid region: region was not a valid DNS name."
This fixes the replay upload pipeline which was failing with the
invalid region error. Replays should now upload successfully to B2
via the ARMOR proxy.
Related to ai-code-battle-o43: Replay viewer verification task.
Verification results:
1. ✅ /data/blog/index.json exists and has 1 post (meta-week-13-season-1)
2. ✅ Individual post pages load correctly at /blog/{slug}
3. ✅ Blog post JSON structure matches frontend expectations (content_md field)
4. ✅ Tags and filters implemented in UI (All, Meta Reports, Chronicles buttons)
5. ✅ Blog page builds successfully (blog-D4QMd11d.js included in build)
Current state: Blog infrastructure is fully implemented with:
- LLM-powered narrative generation (blog.go, narrative.go)
- Story arc detection (rise, fall, rivalry, upset, evolution milestones)
- Weekly meta report generation with ELO movers, strategy analysis
- Chronicles for story arcs (rivalry, upset, rise/fall, evolution)
- Tag-based filtering and search
Note: Current blog content is placeholder/template-based. Meaningful
match commentary will be generated when:
- ACB_LLM_BASE_URL and ACB_LLM_API_KEY are configured in index-builder
- Real match data exists in PostgreSQL database
- Story arcs are detected from rating history and match results
The main leaderboard SPA is now served at / (index.html) and the
standalone replay viewer lives at /replay.html. This removes the
_redirects workaround in index-builder that patched over the inverted
entry points.
- Rename web/app.html → web/index.html (main SPA)
- Rename web/index.html → web/replay.html (standalone viewer)
- Update vite.config.ts: main→index.html, replay→replay.html
- Remove _redirects injection from deploy.go verifyMergedOutput
- Update pages.json routes and README dev URL
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The replay viewer was baked into index.html (served at /) while the
leaderboard app was at /app.html. Add a _redirects file so visitors
landing on / get redirected to the main leaderboard app.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add formatSeasonChampionshipContext helper to inject season progress,
championship bracket positioning, and seed lines into LLM prompts.
Add §13.2 critical moment / turning point summary to the match-of-the-week
section of buildSpotlightPrompt. These complete the §15.1/§15.5 alignment
for structured contextual match data in sports-journalism-style prompts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Restore detailed system prompt constant framing the LLM as a sports
journalist covering an emergent bot league, with specific guidance on
ELO deltas, rivalry context, head-to-head records, and scouting-style
lineage framing. Enrich per-arc prompts with critical moment summaries
(§13.2), community tactical hints, ELO before/after deltas, and
head-to-head records. Fix rivalry arc to include ELO context for both
bots. Ensure fall arc shows both wins and losses in key match listings.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Generates contextual turning-point descriptions for matches used in blog
narratives and rivalry chronicles (§13.2). Summarizes close scores, ELO
upsets, non-standard end conditions, and marathon matches.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two functions referenced in generateLLMChronicle were undefined:
- getCurrentSeasonTheme: returns the active season's theme string
- buildHeadToHeadFromArc: computes W/L head-to-head records for a bot
against all opponents from match data, enriching LLM narrative prompts
Also improves the sports journalist system prompt with more detailed
coverage style guidance for better narrative quality.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add data/meta/rivalries.json to R2 upload list in uploadMetaJSONToR2
- Add attachCommunityHints() to narrative.go to enrich story arcs with
highest-upvote community tactical hints (upvotes >= 3, idea/mistake types)
- Fix detectRivalryArcs() key separator from "-" to "|" to avoid UUID
hyphen collisions when parsing bot ID pairs
- Fix partitionBots() call sites in bot_strategies_phase13.go to use
struct field access (.friendly, .enemy) matching updated return type
generator.go already contains generateArchetypes, generateCommunityHints,
and generateMatchFeedback (all called from generateAllIndexes). main.go
uploads all four outputs to R2 on every build cycle.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add transcript panel with turn-by-turn summaries generated from replay events
- Each turn shows: player moves, combat, deaths, captures, energy collection, spawns, win probability
- Add 'T' key shortcut to toggle transcript panel
- Panel supports three view modes: All Turns, ±10 Turns from Current, Recent 20 Turns
- Click on transcript entry to jump to that turn
- Current turn is highlighted in transcript with smooth scroll
- Panel content is selectable/copyable for screen reader users
- Transcript generation logic already existed in replay-viewer.ts; this adds the UI
- Transcript button slides in from right side of screen
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Remove unused encoding/json and net/http imports from cmd/acb-evolver/run.go
that caused build failure. Include other pre-dispatch changes from prior work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add ARIA live region announcement during auto-playback using detailed transcript text
- Transcript panel shows turn-by-turn summaries with current turn highlighting
- T key toggles transcript panel (collapsible UI)
- Panel content is selectable/copyable text for screen reader users
- Fix build errors in clip-maker.ts (remove unused lastExportBlob references)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add ReplayPlayer to type imports in replay-viewer.ts
- Add explicit type annotation for entry parameter in replay.ts transcript map
- Fixes TypeScript compilation errors for §15.3 screen reader transcript feature
