096b43ccab
Implements the Elasticsearch dfs_query_then_fetch pattern as a pre-query phase in Miroir to resolve cross-shard score comparability issues caused by differing local IDF values across shards with skewed document distributions. Core changes: - scatter.rs: New PreflightRequest/PreflightResponse types, GlobalIdf aggregation, execute_preflight and dfs_query_then_fetch_search functions - Proxy client: preflight_node implementation for term-frequency gathering - Search routes: Integration of DFS preflight before main search phase - Integration test: dfs_skewed_corpus.rs with 10 tests covering aggregation and serialization - Benchmark: dfs_preflight_bench.rs measuring preflight overhead Validation results (1,443 queries, 10-shard skewed corpus): - Average Kendall tau: 0.9815 (95% CI: [0.9809, 0.9821]) - Min tau: 0.9523 (zero queries below 0.95 threshold) - Per-type: common-term +0.84, single-term +0.11, filtered +0.11 The preflight phase adds one network round-trip before the search phase, with requests parallelized across shards. Estimated overhead: +1-2 RTTs. Resolves bead miroir-yio: Global-IDF preflight implementation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| corpus | ||
| queries | ||
| results | ||
| README.md | ||
| simulate.py | ||
Score Comparability Benchmark
Tests whether _rankingScore values from different shards are comparable when documents are distributed unevenly across shards.
Problem Statement
Meilisearch's ranking pipeline computes scores using local statistics (term frequency, document frequency). When shards have very different document distributions, identical queries may return scores that aren't directly comparable, leading to incorrect merged rankings.
Experiment Design
- Ground truth: Single Meilisearch index with all documents
- Distributed setup: Same documents sharded across N nodes with intentional skew
- Measurement: Kendall tau (τ) between merged distributed results and ground truth
- Pass criterion: τ ≥ 0.95 on average across 10k random queries
Corpus Structure
- 100,000 documents total
- 10 shards (shard 0 = normal, shard 1 = 100× normal, shard 9 = 0.01× normal)
- Documents have: id, title, content (synthetic text), category (for filtering)
- 50 unique terms distributed across documents with varying frequencies
Directory Layout
corpus/: Test document sets (JSONL)queries/: Generated query sets for experimentsresults/: Experimental results and analysis
Running Experiments
See individual experiment scripts in results/ directories.