No description
1124d97c14
Implements the complete Redis backend for the TaskStore trait, mirroring all 14 SQLite tables to Redis keyspace as specified in plan §4. Key features: - Tables 1-14: Full CRUD operations with Redis data structures - tasks → miroir:tasks:<id> hash + miroir:tasks:_index set - node_settings_version → miroir:node_settings_version:<index>:<node> hash - aliases → miroir:aliases:<name> hash + index - sessions → miroir:session:<id> hash with EXPIRE - idempotency_cache → miroir:idemp:<key> hash with EXPIRE - jobs → miroir:jobs:<id> hash + miroir:jobs:_queued set - leader_lease → miroir:lease:<scope> string via SET NX EX - canaries → miroir:canary:<id> hash + index - canary_runs → miroir:canary_runs:<canary_id> sorted set - cdc_cursors → miroir:cdc_cursor:<sink>:<index> string - tenant_map → miroir:tenant_map:<sha256> hash - rollover_policies → miroir:rollover:<name> hash + index - search_ui_config → miroir:search_ui_config:<index> hash - admin_sessions → miroir:admin_session:<id> hash with EXPIRE - Extras from plan §4 footnotes: - search_ui_scoped_key with observation tracking - Rate limiting for search_ui and admin_login - CDC overflow buffer with LPUSH/LTRIM - Pub/Sub for admin_session revocation - Integration tests (testcontainers): - test_redis_tasks_crud: Full task CRUD operations - test_redis_leader_lease: Lease acquisition and renewal - test_redis_lease_race: Concurrent lease acquisition (exactly one wins) - test_redis_memory_budget: 10k tasks + 1k sessions + 1k idempotency - test_redis_pubsub_session_invalidation: Pub/Sub revocation - Tests for all 14 tables covering CRUD operations - Secondary _index sets for efficient list-wide queries - MULTI/EXEC pipelines for atomic multi-key operations - TTL-based garbage collection for sessions/idempotency - Sync-to-async bridge using dedicated runtime (avoids nesting) Acceptance criteria met: ✓ testcontainers-based integration tests for trait-level behavior ✓ Lease race test: two pods SET NX EX → exactly one wins ✓ Memory budget test: verifies workload creation ✓ Pub/Sub test: subscribe to miroir:admin_session:revoked Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .beads | ||
| .cargo | ||
| charts/miroir | ||
| crates | ||
| docs | ||
| tests/benches/score-comparability | ||
| .editorconfig | ||
| .gitignore | ||
| .needle-predispatch-sha | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| clippy.toml | ||
| LICENSE | ||
| README.md | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
Miroir
Multi-node Index Replication Orchestrator, Integrated Rebalancing
Miroir is a RAID-like orchestration layer for Meilisearch. It stripes a large index across a fleet of small-RAM Meilisearch nodes with a configurable replication factor, fans out search queries across all shards, and rebalances shard assignments when nodes are added or removed — all using the Meilisearch Community Edition.
The Problem
Meilisearch loads its entire index into memory-mapped LMDB files. A large index that exceeds a single server's available RAM cannot run on that server. The Enterprise Edition's native sharding is gated behind a commercial license. Miroir solves this without it.
How It Works
Client
│
▼
Miroir Orchestrator
├── Write path: hash(doc_id) → assign to shard → write to R replicas
├── Read path: scatter query to all shards → gather → merge ranked results
└── Rebalance: on node add/remove → recompute assignments → migrate minimum shards
Meilisearch Nodes (N instances, each holding a subset of shards)
node-0 node-1 node-2 ... node-N
Replication Factor
Analogous to software RAID — configurable per deployment:
| RF | Redundancy | Node failures tolerated | Capacity |
|---|---|---|---|
| 1 | None (stripe only) | 0 | 100% of fleet |
| 2 | One replica | 1 per shard group | 50% of fleet |
| 3 | Two replicas | 2 per shard group | 33% of fleet |
Key Components
- Orchestrator — proxy that handles shard routing, scatter-gather, result merging, and topology management
- Shard router — consistent hash function (Rendezvous/HRW) mapping document IDs to node assignments; minimal reshuffling on topology change
- Rebalancer — on node add/remove, recomputes assignments and migrates only the shards that changed owners; surviving replicas serve reads during rebuild
- Result merger — normalizes and merges ranked result sets from multiple shards into a single coherent response
Status
Design phase. See docs/ for architecture detail.