b23e70656e
Implements POST/PUT /indexes/{uid}/documents and DELETE /indexes/{uid}/documents:
- Primary key extraction on hot path with 400 miroir_primary_key_required if missing
- _miroir_shard injection into every document before forwarding to nodes
- Rejection of _miroir_shard in client-submitted docs (400 miroir_reserved_field)
- Two-rule quorum: per-group floor(RF/2)+1 ACKs, success if ≥1 group meets quorum
- X-Miroir-Degraded header when any group misses quorum
- 503 miroir_no_quorum only when NO group meets quorum
- Per-batch grouping by target shard for efficient HTTP fan-out
- DELETE by IDs routes each ID independently to its shard
- DELETE by filter broadcasts to all nodes
Acceptance tests pass:
- Primary key validation before any writes
- Reserved field rejection
- Shard distribution uniformity (17-26 shards/node with 64 shards/3 nodes)
- Quorum calculation: floor(RF/2)+1
- Meilisearch-compatible error shape
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| corpus | ||
| queries | ||
| results | ||
| README.md | ||
| simulate.py | ||
Score Comparability Benchmark
Tests whether _rankingScore values from different shards are comparable when documents are distributed unevenly across shards.
Problem Statement
Meilisearch's ranking pipeline computes scores using local statistics (term frequency, document frequency). When shards have very different document distributions, identical queries may return scores that aren't directly comparable, leading to incorrect merged rankings.
Experiment Design
- Ground truth: Single Meilisearch index with all documents
- Distributed setup: Same documents sharded across N nodes with intentional skew
- Measurement: Kendall tau (τ) between merged distributed results and ground truth
- Pass criterion: τ ≥ 0.95 on average across 10k random queries
Corpus Structure
- 100,000 documents total
- 10 shards (shard 0 = normal, shard 1 = 100× normal, shard 9 = 0.01× normal)
- Documents have: id, title, content (synthetic text), category (for filtering)
- 50 unique terms distributed across documents with varying frequencies
Directory Layout
corpus/: Test document sets (JSONL)queries/: Generated query sets for experimentsresults/: Experimental results and analysis
Running Experiments
See individual experiment scripts in results/ directories.