miroir/docs/notes/phase3-task-store-verification.md

Phase 3 — Task Registry + Persistence Verification

DoD Checklist

✅ 1. rusqlite-backed store initializing every table idempotently at startup

Location: crates/miroir-core/src/task_store/sqlite.rs

• SqliteTaskStore::new() creates/opens the SQLite database
• initialize() calls init_schema() which creates all 14 tables with CREATE TABLE IF NOT EXISTS
• Schema version is tracked in schema_version table
• WAL mode enabled for better concurrency

✅ 2. Redis-backed store mirrors the same API

Location: crates/miroir-core/src/task_store/redis.rs

• RedisTaskStore implements the same TaskStore trait
• All 14 tables mapped to Redis hashes with _index secondary sets
• Runtime backend selection via task_store.backend config

✅ 3. Migrations/versioning

Location: crates/miroir-core/src/task_store/schema.rs, sqlite.rs, redis.rs

• SCHEMA_VERSION constant (currently 1)
• Schema version stored in schema_version table (SQLite) or miroir:schema_version key (Redis)
• Version check on initialization - rejects mismatched versions loudly

✅ 4. Property tests

Location: crates/miroir-core/tests/task_store.rs

• task_insert_get_roundtrip() - Round-trip test for tasks
• alias_upsert_roundtrip() - Upsert semantics for aliases
• idempotency_cache_roundtrip() - Idempotency cache behavior
• leader_lease_acquire_renew() - Leader lease acquisition
• job_enqueue_dequeue() - Job queue operations
• canary_run_history() - Canary run history tracking
• prop_task_list_filter_by_status() - Proptest for task list filtering

✅ 5. Integration test: restart survival

Location: crates/miroir-core/tests/task_store.rs::restart_survival

• Creates a store, inserts data, closes connection
• Reopens store and verifies data survived
• Tests both task persistence and status updates

✅ 6. Redis-backend integration test

Location: crates/miroir-core/tests/task_store_redis.rs

• Uses testcontainers to spin up real Redis instance
• Tests all Redis-specific operations:
  • redis_task_insert_get_roundtrip()
  • redis_leader_lease_acquire_renew()
  • redis_idempotency_cache_ttl()
  • redis_ratelimit_increment()
  • redis_ratelimit_backoff()
  • redis_cdc_overflow()
  • redis_scoped_key_rotation()
  • And more...

✅ 7. miroir:tasks:_index-style iteration

Location: crates/miroir-core/src/task_store/redis.rs

• index_key() method generates miroir:{table}:_index keys
• task_list() uses smembers(&index_key) to get all IDs
• alias_list(), canary_list(), tenant_list(), etc. all use this pattern
• No SCAN - O(cardinality) list-wide queries

✅ 8. Helm schema enforcement

Location: charts/miroir/values.schema.json

Lines 142-160 enforce:

{
  "if": {
    "properties": {
      "replicas": {"minimum": 2}
    },
    "required": ["replicas"]
  },
  "then": {
    "properties": {
      "taskStore": {
        "properties": {
          "backend": {"const": "redis"}
        },
        "required": ["backend"]
      }
    }
  },
  "errorMessage": "taskStore.backend must be 'redis' when replicas > 1"
}

Also enforces HPA requirements (lines 162-186).

✅ 9. Redis memory accounting validation

Location: This document

Redis Memory Accounting (Plan §14.7)

Keyspace Structure

The task store uses the following Redis keyspace pattern:

miroir:{table}:{id}           # Hash: row data
miroir:{table}:_index         # Set: all IDs for table
miroir:schema_version         # String: schema version
miroir:jobs:enqueued          # List: job queue
miroir:ratelimit:{key}        # String with TTL: rate limit counters
miroir:ratelimit:backoff:{key} # String with TTL: rate limit backoffs
miroir:cdc:overflow:{sink}    # String: CDC overflow buffer
miroir:search_ui_scoped_key:{index}         # String with TTL: scoped keys
miroir:search_ui_scoped_key_observed:{pod}:{index}  # String: observation tracking
miroir:admin_session:revoked  # Pub/Sub: instant logout channel

Per-Table Memory Analysis

Table	Index Size (per entry)	Data Size (per entry)	Notes
tasks	~40 bytes (UUID string)	~200-500 bytes (JSON)	One entry per fan-out write
aliases	~20 bytes (name)	~150 bytes (JSON)	Static, admin-controlled
sessions	~40 bytes (UUID)	~100 bytes (JSON)	TTL-based expiration
idempotency_cache	~50 bytes (key hash)	~500 bytes (response)	TTL 1 hour
jobs	~40 bytes (job ID)	~300 bytes (JSON)	Short-lived
leader_lease	~40 bytes (lease ID)	~150 bytes (JSON)	Single entry
canaries	~20 bytes (name)	~200 bytes (JSON)	Static, admin-controlled
canary_runs	~40 bytes (run ID)	~150 bytes (JSON)	Per-run, pruned periodically
cdc_cursors	~50 bytes (sink:index)	~100 bytes (cursor)	One per (sink, index) pair
tenant_map	~30 bytes (API key)	~200 bytes (JSON)	Static, admin-controlled
rollover_policies	~20 bytes (name)	~150 bytes (JSON)	Static, admin-controlled
search_ui_config	~20 bytes (index)	~1-5 KB (config JSON)	Static, per-index
admin_sessions	~40 bytes (session ID)	~100 bytes (JSON)	TTL 24 hours
node_settings_version	~50 bytes (index:node)	~50 bytes (version + timestamp)	One per (index, node)

Rate Limiter Memory (§13.21)

The plan specifies: "~20 MB per 10k active IPs"

Calculation:

• Each IP bucket: ~2 KB (key + counter + timestamp)
• 10,000 IPs × 2 KB = ~20 MB
• With default TTL of 60 seconds, memory is bounded even under scan attacks

Representative Load Calculation

Scenario: 10 TB corpus, 20 kQPS (from §14.7 sizing matrix)

Assumptions:

• 12 orchestrator pods
• 100 active indexes
• 10,000 concurrent users
• 1,000 writes/second
• 5,000 searches/second

Memory breakdown:

Category	Calculation	Memory
tasks (1M writes, 10 min retention)	1M × (40 + 350) bytes	~390 MB
sessions (10k users, 24h TTL)	10k × (40 + 100) bytes	~1.4 MB
idempotency (50k requests, 1h TTL)	50k × (50 + 500) bytes	~27.5 MB
jobs (100 concurrent)	100 × (40 + 300) bytes	~34 KB
canary_runs (100 canaries × 100 runs)	10k × (40 + 150) bytes	~1.9 MB
cdc_cursors (10 sinks × 100 indexes)	1k × (50 + 100) bytes	~150 KB
rate_limit (10k active IPs)	10k × 2 KB	~20 MB
search_ui_config (100 indexes)	100 × (20 + 3 KB)	~300 KB
admin_sessions (100 admins)	100 × (40 + 100) bytes	~14 KB
Total		~440 MB

Redis Sizing Recommendations

Based on the analysis:

Corpus / QPS	Orchestrator Pods	Redis Memory	Recommendation
≤ 10 GB / ≤ 500	2	512 MB	Single Redis instance
≤ 50 GB / ≤ 2k	2-4	1 GB	Single Redis with persistence
≤ 200 GB / ≤ 5k	4-8	2 GB	Redis with AOF persistence
≤ 1 TB / ≤ 20k	8-12	4 GB	Redis Sentinel or clustered
≤ 5 TB / ≤ 100k	12-24	8+ GB	Redis Cluster

Memory Monitoring

Key Redis metrics to monitor:

1. used_memory - Total memory used
2. used_memory_peak - Peak memory usage
3. used_memory_perc - Percentage of maxmemory
4. keyspace counts - Track growth per table
5. Eviction rate - Should be zero (TTL-based cleanup)

Alert thresholds:

• Warning: > 70% of maxmemory
• Critical: > 85% of maxmemory

Verification

The memory accounting above validates that:

1. Memory usage scales linearly with workload
2. TTL-based expiration prevents unbounded growth
3. Rate limiter state (~20 MB per 10k IPs) fits within the §14.2 per-pod budget
4. For the representative 20 kQPS load, total Redis memory is < 500 MB

This confirms the plan §14.7 sizing matrix is conservative and provides headroom for bursts.