72f9a197b5
Completed Plan §15 Open Problem #4 research on cross-shard score comparability. ## Key Finding Average Kendall tau: 0.79 vs. 0.95 threshold — FAIL Cross-shard score comparability is a significant issue: - Common-term queries: τ = 0.15 (catastrophic) - Local IDF statistics cause score inflation on small shards - Documents from 10-doc shards outrank 93K-doc shard results ## Recommendation Implement Reciprocal Rank Fusion (RRF) for result merging. Follow-up bead: miroir-nsu ## Artifacts Added - Benchmark infrastructure: tests/benches/score-comparability/ - Corpus generator with extreme shard skew (100× variance) - Query generator (10K random queries across 5 types) - BM25-based simulation with global vs local IDF - Kendall tau comparison tool - Full experimental results (τ = 0.79 ± 0.01, 95% CI) - Research writeup: docs/research/score-normalization-at-scale.md Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
36 lines
No EOL
452 B
JSON
36 lines
No EOL
452 B
JSON
{
|
|
"total_documents": 100000,
|
|
"shard_count": 10,
|
|
"vocabulary_size": 50,
|
|
"categories": [
|
|
"tech",
|
|
"finance",
|
|
"science",
|
|
"health",
|
|
"business"
|
|
],
|
|
"skew_factors": [
|
|
1.0,
|
|
100.0,
|
|
1.0,
|
|
1.0,
|
|
1.0,
|
|
1.0,
|
|
1.0,
|
|
1.0,
|
|
0.5,
|
|
0.01
|
|
],
|
|
"shard_sizes": {
|
|
"0": 930,
|
|
"1": 93015,
|
|
"2": 930,
|
|
"3": 930,
|
|
"4": 930,
|
|
"5": 930,
|
|
"6": 930,
|
|
"7": 930,
|
|
"8": 465,
|
|
"9": 10
|
|
}
|
|
} |