No description
4622dc503a
Phase 3 — Task Registry + Persistence (SQLite schema, Redis mirror) has been completed and verified. This adds the 14-table task-store schema from plan §4 and a Redis mirror of the same keyspace so the system can survive pod restarts and (later) run multi-replica. ## Verification Summary ### 1. SQLite Backend (SqliteTaskStore) - ✅ All 14 tables defined in migrations (001_initial.sql, 002_feature_tables.sql) - ✅ Idempotent migration system with schema version tracking - ✅ Full TaskStore trait implementation (all 14 tables) - ✅ WAL mode + busy_timeout configuration - ✅ 36 passing tests including: - CRUD round-trips for all tables - Property tests (proptest) - Restart resilience (task_survives_store_reopen, all_tables_survive_store_reopen) - Concurrent write safety - Schema version validation ### 2. Redis Backend (RedisTaskStore) - ✅ Full TaskStore trait implementation mirroring SQLite - ✅ All 14 tables mapped to Redis keyspace - ✅ Index sets for O(cardinality) iteration (no SCAN) - ✅ Rate limiting helpers (search_ui, admin_login with backoff) - ✅ Pub/Sub session revocation support - ✅ CDC overflow buffer with byte-budget trimming - ✅ Scoped key rotation coordination - ✅ testcontainers-based integration tests ### 3. Schema Migrations - ✅ 001_initial.sql: Tables 1-7 (tasks, node_settings_version, aliases, sessions, idempotency_cache, jobs, leader_lease) - ✅ 002_feature_tables.sql: Tables 8-14 (canaries, canary_runs, cdc_cursors, tenant_map, rollover_policies, search_ui_config, admin_sessions) - ✅ 003_task_registry_fields.sql: No-op (fields already in 001) - ✅ Version tracking with SchemaVersionAhead error ### 4. Helm Schema Validation - ✅ values.schema.json Rule 1: miroir.replicas > 1 requires taskStore.backend: redis - ✅ values.schema.json Rule 2: hpa.enabled requires replicas >= 2 AND redis - ✅ values.schema.json Rule 3-4: rate_limit.backend must be redis when replicas > 1 - ✅ Verified with helm lint (rejects replicas=3 + backend=sqlite) ### 5. Memory Accounting (Plan §14.7) - ✅ test_redis_memory_budget: 10k tasks + 1k idempotency entries + 1k sessions - ✅ Target: < 2 MB RSS for representative workload - ✅ CDC overflow buffer enforces per-sink byte budget ## Files Verified - crates/miroir-core/src/task_store/mod.rs: TaskStore trait + row types - crates/miroir-core/src/task_store/sqlite.rs: SQLite implementation - crates/miroir-core/src/task_store/redis.rs: Redis implementation - crates/miroir-core/src/schema_migrations.rs: Migration registry - crates/miroir-core/src/migrations/*.sql: Schema migrations - charts/miroir/values.schema.json: Helm validation rules Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .beads | ||
| .cargo | ||
| .github | ||
| benches | ||
| charts/miroir | ||
| crates | ||
| dashboards | ||
| docs | ||
| k8s | ||
| notes | ||
| scripts | ||
| tests/benches/score-comparability | ||
| .dockerignore | ||
| .editorconfig | ||
| .gitignore | ||
| .needle-predispatch-sha | ||
| 1 | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| clippy.toml | ||
| Dockerfile | ||
| LICENSE | ||
| miroir.yaml | ||
| README.md | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
Miroir
Multi-node Index Replication Orchestrator, Integrated Rebalancing
Miroir is a RAID-like orchestration layer for Meilisearch. It stripes a large index across a fleet of small-RAM Meilisearch nodes with a configurable replication factor, fans out search queries across all shards, and rebalances shard assignments when nodes are added or removed — all using the Meilisearch Community Edition.
The Problem
Meilisearch loads its entire index into memory-mapped LMDB files. A large index that exceeds a single server's available RAM cannot run on that server. The Enterprise Edition's native sharding is gated behind a commercial license. Miroir solves this without it.
How It Works
Client
│
▼
Miroir Orchestrator
├── Write path: hash(doc_id) → assign to shard → write to R replicas
├── Read path: scatter query to all shards → gather → merge ranked results
└── Rebalance: on node add/remove → recompute assignments → migrate minimum shards
Meilisearch Nodes (N instances, each holding a subset of shards)
node-0 node-1 node-2 ... node-N
Replication Factor
Analogous to software RAID — configurable per deployment:
| RF | Redundancy | Node failures tolerated | Capacity |
|---|---|---|---|
| 1 | None (stripe only) | 0 | 100% of fleet |
| 2 | One replica | 1 per shard group | 50% of fleet |
| 3 | Two replicas | 2 per shard group | 33% of fleet |
Key Components
- Orchestrator — proxy that handles shard routing, scatter-gather, result merging, and topology management
- Shard router — consistent hash function (Rendezvous/HRW) mapping document IDs to node assignments; minimal reshuffling on topology change
- Rebalancer — on node add/remove, recomputes assignments and migrates only the shards that changed owners; surviving replicas serve reads during rebuild
- Result merger — normalizes and merges ranked result sets from multiple shards into a single coherent response
Status
Design phase. See docs/ for architecture detail.