No description
e5902bb47f
Implements the 14-table task-store schema from plan §4 with both SQLite and Redis backends. Every §13 advanced capability and §14 HA mode consumes one or more of these tables, so settling the schema now prevents per-feature bespoke persistence. ## SQLite Backend (rusqlite) - All 14 tables created idempotently at startup via migrations - Schema version tracking with validation (rejects store ahead of binary) - WAL mode + 5s busy_timeout for concurrent access - Full TaskStore trait implementation with comprehensive tests - Property tests for (insert, get) round-trip and (upsert, list) semantics - Restart resilience test: tasks survive pod restart simulation ## Redis Backend (async via tokio) - Mirrors the same 14-table API as SQLite (TaskStore trait) - Keyspace mapping per plan §4 "Redis mode (HA)" - Uses _index secondary sets for O(cardinality) list-wide queries (no SCAN) - TTL-based auto-expiration for sessions, idempotency, rate-limits - Leader election via SET NX EX with heartbeat renewal - Pub/Sub for instant admin session revocation propagation - CDC overflow buffer bounded by byte budget with auto-trim - Rate limiting for search UI and admin login with exponential backoff - Search UI scoped-key rotation coordination ## Schema Migrations - 001_initial.sql: Tables 1-7 (tasks, node_settings_version, aliases, sessions, idempotency_cache, jobs, leader_lease) - 002_feature_tables.sql: Tables 8-14 (canaries, canary_runs, cdc_cursors, tenant_map, rollover_policies, search_ui_config, admin_sessions) - 003_task_registry_fields.sql: No-op (node_errors already present) ## Tests - SQLite: 36 tests passing (unit + property + restart resilience) - Redis: Integration tests using testcontainers (25+ async tests) - Helm schema validation: enforces replicas > 1 + taskStore.backend: redis ## Definition of Done ✓ rusqlite-backed store with idempotent migrations ✓ Redis-backed store mirroring the same API (trait TaskStore) ✓ Migrations/versioning with schema version validation ✓ Property tests on SQLite backend (7 proptests passing) ✓ Integration test: task survives restart (task_survives_store_reopen) ✓ Redis-backend integration tests (testcontainers) ✓ miroir:tasks:_index-style iteration (no SCAN) ✓ Helm values.schema.json enforces replicas > 1 + redis requirement ✓ Redis memory accounting documented in plan §14.7 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .beads | ||
| .cargo | ||
| .github | ||
| benches | ||
| charts/miroir | ||
| crates | ||
| dashboards | ||
| docs | ||
| k8s | ||
| notes | ||
| scripts | ||
| tests/benches/score-comparability | ||
| .dockerignore | ||
| .editorconfig | ||
| .gitignore | ||
| .needle-predispatch-sha | ||
| 1 | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| clippy.toml | ||
| Dockerfile | ||
| LICENSE | ||
| miroir.yaml | ||
| README.md | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
Miroir
Multi-node Index Replication Orchestrator, Integrated Rebalancing
Miroir is a RAID-like orchestration layer for Meilisearch. It stripes a large index across a fleet of small-RAM Meilisearch nodes with a configurable replication factor, fans out search queries across all shards, and rebalances shard assignments when nodes are added or removed — all using the Meilisearch Community Edition.
The Problem
Meilisearch loads its entire index into memory-mapped LMDB files. A large index that exceeds a single server's available RAM cannot run on that server. The Enterprise Edition's native sharding is gated behind a commercial license. Miroir solves this without it.
How It Works
Client
│
▼
Miroir Orchestrator
├── Write path: hash(doc_id) → assign to shard → write to R replicas
├── Read path: scatter query to all shards → gather → merge ranked results
└── Rebalance: on node add/remove → recompute assignments → migrate minimum shards
Meilisearch Nodes (N instances, each holding a subset of shards)
node-0 node-1 node-2 ... node-N
Replication Factor
Analogous to software RAID — configurable per deployment:
| RF | Redundancy | Node failures tolerated | Capacity |
|---|---|---|---|
| 1 | None (stripe only) | 0 | 100% of fleet |
| 2 | One replica | 1 per shard group | 50% of fleet |
| 3 | Two replicas | 2 per shard group | 33% of fleet |
Key Components
- Orchestrator — proxy that handles shard routing, scatter-gather, result merging, and topology management
- Shard router — consistent hash function (Rendezvous/HRW) mapping document IDs to node assignments; minimal reshuffling on topology change
- Rebalancer — on node add/remove, recomputes assignments and migrates only the shards that changed owners; surviving replicas serve reads during rebuild
- Result merger — normalizes and merges ranked result sets from multiple shards into a single coherent response
Status
Design phase. See docs/ for architecture detail.