No description
a046c3aff2
Implements deterministic, coordination-free routing primitives that everything else depends on. Any Miroir pod can independently compute identical write targets and covering sets given a fixed topology. Core routing (router.rs): - score(): Rendezvous hashing with XxHash64 seed 0 (matches Meilisearch Enterprise) - assign_shard_in_group(): HRW assignment with tie-breaking - write_targets(): Returns exactly RG × RF nodes, one from each group - query_group(): Round-robin query distribution across replica groups - covering_set(): One node per shard with intra-group replica rotation - shard_for_key(): Hash-based document-to-shard mapping Topology management (topology.rs): - NodeId, NodeStatus, Node, Group, Topology structs - Node health state machine (Healthy/Degraded/Draining/Failed/Joining/Active/Removed) - State transition validation - Write eligibility logic (Draining nodes conditionally eligible) - Healthy node filtering Scatter primitives (scatter.rs): - Scatter trait with StubScatter implementation - ScatterRequest, ScatterResponse, NodeResponse structs Result merger (merger.rs): - Global sort by _rankingScore descending - Offset/limit application after merge - Facet count aggregation across shards - Estimated total hits summation - Conditional _rankingScore stripping - Always strips _miroir_shard Task registry (task.rs): - TaskRegistry trait with StubTaskRegistry implementation - MiroirTask, TaskStatus, NodeTask, NodeTaskStatus - TaskFilter for listing Acceptance tests (all passing): - AT-1: Rendezvous determinism (1000 runs) - AT-2: Reshuffle bound on add (2 × 1/4 × 64) - AT-3: Reshuffle bound on remove (~RF × S / Ng) - AT-4: Uniformity (64 shards, 3 nodes, RF=1 → 18–26 per node) - AT-5: Top-RF placement stability - AT-6: shard_for_key fixture verification - AT-7: Tie-breaking on node_id - AT-8: Canonical concatenation order (shard_id, node_id) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .beads | ||
| .github/workflows | ||
| charts/miroir | ||
| crates | ||
| docs | ||
| notes | ||
| .editorconfig | ||
| .gitignore | ||
| .needle-predispatch-sha | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| clippy.toml | ||
| lcov.info | ||
| LICENSE | ||
| README.md | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
Miroir
Multi-node Index Replication Orchestrator, Integrated Rebalancing
Miroir is a RAID-like orchestration layer for Meilisearch. It stripes a large index across a fleet of small-RAM Meilisearch nodes with a configurable replication factor, fans out search queries across all shards, and rebalances shard assignments when nodes are added or removed — all using the Meilisearch Community Edition.
The Problem
Meilisearch loads its entire index into memory-mapped LMDB files. A large index that exceeds a single server's available RAM cannot run on that server. The Enterprise Edition's native sharding is gated behind a commercial license. Miroir solves this without it.
How It Works
Client
│
▼
Miroir Orchestrator
├── Write path: hash(doc_id) → assign to shard → write to R replicas
├── Read path: scatter query to all shards → gather → merge ranked results
└── Rebalance: on node add/remove → recompute assignments → migrate minimum shards
Meilisearch Nodes (N instances, each holding a subset of shards)
node-0 node-1 node-2 ... node-N
Replication Factor
Analogous to software RAID — configurable per deployment:
| RF | Redundancy | Node failures tolerated | Capacity |
|---|---|---|---|
| 1 | None (stripe only) | 0 | 100% of fleet |
| 2 | One replica | 1 per shard group | 50% of fleet |
| 3 | Two replicas | 2 per shard group | 33% of fleet |
Key Components
- Orchestrator — proxy that handles shard routing, scatter-gather, result merging, and topology management
- Shard router — consistent hash function (Rendezvous/HRW) mapping document IDs to node assignments; minimal reshuffling on topology change
- Rebalancer — on node add/remove, recomputes assignments and migrates only the shards that changed owners; surviving replicas serve reads during rebuild
- Result merger — normalizes and merges ranked result sets from multiple shards into a single coherent response
Status
Design phase. See docs/ for architecture detail.