72f9a197b5
Completed Plan §15 Open Problem #4 research on cross-shard score comparability. ## Key Finding Average Kendall tau: 0.79 vs. 0.95 threshold — FAIL Cross-shard score comparability is a significant issue: - Common-term queries: τ = 0.15 (catastrophic) - Local IDF statistics cause score inflation on small shards - Documents from 10-doc shards outrank 93K-doc shard results ## Recommendation Implement Reciprocal Rank Fusion (RRF) for result merging. Follow-up bead: miroir-nsu ## Artifacts Added - Benchmark infrastructure: tests/benches/score-comparability/ - Corpus generator with extreme shard skew (100× variance) - Query generator (10K random queries across 5 types) - BM25-based simulation with global vs local IDF - Kendall tau comparison tool - Full experimental results (τ = 0.79 ± 0.01, 95% CI) - Research writeup: docs/research/score-normalization-at-scale.md Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
19 lines
No EOL
303 B
JSON
19 lines
No EOL
303 B
JSON
{
|
|
"total_queries": 10000,
|
|
"allocation": {
|
|
"single_term": 0.25,
|
|
"multi_term": 0.25,
|
|
"filtered": 0.2,
|
|
"rare_term": 0.15,
|
|
"common_term": 0.15
|
|
},
|
|
"random_seed": 42,
|
|
"vocab_size": 50,
|
|
"categories": [
|
|
"tech",
|
|
"finance",
|
|
"science",
|
|
"health",
|
|
"business"
|
|
]
|
|
} |