409f952f59
- LICENSE: MIT (per plan §12) - CHANGELOG.md: Keep a Changelog 1.1.0 skeleton with [Unreleased] and [0.1.0] sections matching the awk extractor from plan §7 - .gitignore: Rust target/, editor junk; Cargo.lock kept in VCS Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
45 lines
2.1 KiB
Markdown
45 lines
2.1 KiB
Markdown
# Miroir
|
|
|
|
**Multi-node Index Replication Orchestrator, Integrated Rebalancing**
|
|
|
|
Miroir is a RAID-like orchestration layer for Meilisearch. It stripes a large index across a fleet of small-RAM Meilisearch nodes with a configurable replication factor, fans out search queries across all shards, and rebalances shard assignments when nodes are added or removed — all using the Meilisearch Community Edition.
|
|
|
|
## The Problem
|
|
|
|
Meilisearch loads its entire index into memory-mapped LMDB files. A large index that exceeds a single server's available RAM cannot run on that server. The Enterprise Edition's native sharding is gated behind a commercial license. Miroir solves this without it.
|
|
|
|
## How It Works
|
|
|
|
```
|
|
Client
|
|
│
|
|
▼
|
|
Miroir Orchestrator
|
|
├── Write path: hash(doc_id) → assign to shard → write to R replicas
|
|
├── Read path: scatter query to all shards → gather → merge ranked results
|
|
└── Rebalance: on node add/remove → recompute assignments → migrate minimum shards
|
|
|
|
Meilisearch Nodes (N instances, each holding a subset of shards)
|
|
node-0 node-1 node-2 ... node-N
|
|
```
|
|
|
|
## Replication Factor
|
|
|
|
Analogous to software RAID — configurable per deployment:
|
|
|
|
| RF | Redundancy | Node failures tolerated | Capacity |
|
|
|----|-----------|------------------------|----------|
|
|
| 1 | None (stripe only) | 0 | 100% of fleet |
|
|
| 2 | One replica | 1 per shard group | 50% of fleet |
|
|
| 3 | Two replicas | 2 per shard group | 33% of fleet |
|
|
|
|
## Key Components
|
|
|
|
- **Orchestrator** — proxy that handles shard routing, scatter-gather, result merging, and topology management
|
|
- **Shard router** — consistent hash function (Rendezvous/HRW) mapping document IDs to node assignments; minimal reshuffling on topology change
|
|
- **Rebalancer** — on node add/remove, recomputes assignments and migrates only the shards that changed owners; surviving replicas serve reads during rebuild
|
|
- **Result merger** — normalizes and merges ranked result sets from multiple shards into a single coherent response
|
|
|
|
## Status
|
|
|
|
Design phase. See [`docs/`](docs/) for architecture detail.
|