No description
ac80d1f765
Completes Phase 3 of the Miroir implementation: the 14-table task-store
schema from plan §4 with both SQLite and Redis backends.
## What Was Done
### 1. SQLite Backend (SqliteTaskStore)
- All 14 tables implemented with CRUD operations
- WAL mode for concurrent access
- Schema version tracking with migration system
- Idempotent migrations (safe to run on every startup)
- Schema version ahead detection (refuses to start if store > binary)
### 2. Redis Backend (RedisTaskStore)
- All 14 tables mapped to Redis keyspace
- Hash per row + index sets for O(cardinality) iteration
- testcontainers-based integration tests
- Leader lease with Redis SET NX/EX semantics
- Pub/Sub for session revocation
- Memory budget test (plan §14.7)
### 3. Schema Migrations
- Migration 1: Core tables (1-7)
- Migration 2: Feature tables (8-14)
- Migration 3: Task registry fields (no-op)
### 4. Tests
- SQLite: 36 tests pass (CRUD, property tests, restart resilience)
- Redis: Comprehensive integration tests (testcontainers)
- Helm validation: multi-replica requires Redis enforced
### 5. Helm Validation
- values.schema.json enforces redis + multi-replica constraint
- Test cases verify lint behavior (pass/fail as expected)
## Definition of Done — VERIFIED ✅
- rusqlite-backed store initializing every table idempotently
- Redis-backed store mirrors the same API (TaskStore trait)
- Migrations/versioning with schema version tracking
- Property tests on SQLite backend
- Integration test: restart resilience
- Redis-backend integration test (testcontainers)
- miroir:tasks:_index-style iteration for list endpoints
- taskStore.backend: redis + replicas > 1 enforced by Helm
- Plan §14.7 Redis memory accounting validated
## Files
- crates/miroir-core/src/task_store/mod.rs — TaskStore trait
- crates/miroir-core/src/task_store/sqlite.rs — SQLite impl
- crates/miroir-core/src/task_store/redis.rs — Redis impl
- crates/miroir-core/src/schema_migrations.rs — Migration registry
- crates/miroir-core/src/migrations/*.sql — Migration files
- charts/miroir/values.schema.json — Helm validation
- charts/miroir/tests/*.yaml — Test cases
- notes/miroir-r3j-phase3-completion.md — Completion notes
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|---|---|---|
| .beads | ||
| .cargo | ||
| .github | ||
| benches | ||
| charts/miroir | ||
| crates | ||
| dashboards | ||
| docs | ||
| k8s | ||
| notes | ||
| scripts | ||
| tests/benches/score-comparability | ||
| .dockerignore | ||
| .editorconfig | ||
| .gitignore | ||
| .needle-predispatch-sha | ||
| 1 | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| clippy.toml | ||
| Dockerfile | ||
| LICENSE | ||
| miroir.yaml | ||
| README.md | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
Miroir
Multi-node Index Replication Orchestrator, Integrated Rebalancing
Miroir is a RAID-like orchestration layer for Meilisearch. It stripes a large index across a fleet of small-RAM Meilisearch nodes with a configurable replication factor, fans out search queries across all shards, and rebalances shard assignments when nodes are added or removed — all using the Meilisearch Community Edition.
The Problem
Meilisearch loads its entire index into memory-mapped LMDB files. A large index that exceeds a single server's available RAM cannot run on that server. The Enterprise Edition's native sharding is gated behind a commercial license. Miroir solves this without it.
How It Works
Client
│
▼
Miroir Orchestrator
├── Write path: hash(doc_id) → assign to shard → write to R replicas
├── Read path: scatter query to all shards → gather → merge ranked results
└── Rebalance: on node add/remove → recompute assignments → migrate minimum shards
Meilisearch Nodes (N instances, each holding a subset of shards)
node-0 node-1 node-2 ... node-N
Replication Factor
Analogous to software RAID — configurable per deployment:
| RF | Redundancy | Node failures tolerated | Capacity |
|---|---|---|---|
| 1 | None (stripe only) | 0 | 100% of fleet |
| 2 | One replica | 1 per shard group | 50% of fleet |
| 3 | Two replicas | 2 per shard group | 33% of fleet |
Key Components
- Orchestrator — proxy that handles shard routing, scatter-gather, result merging, and topology management
- Shard router — consistent hash function (Rendezvous/HRW) mapping document IDs to node assignments; minimal reshuffling on topology change
- Rebalancer — on node add/remove, recomputes assignments and migrates only the shards that changed owners; surviving replicas serve reads during rebuild
- Result merger — normalizes and merges ranked result sets from multiple shards into a single coherent response
Status
Design phase. See docs/ for architecture detail.