jedarden/miroir
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
325 commits 1 branch 0 tags 58 MiB
Commit graph

5 commits

Author SHA1 Message Date
jedarden
c7be4ccbec P12.OP4.1: Validate dfs_query_then_fetch benchmark (τ=0.9817) and document latency 
Re-ran the 10K-query score-comparability benchmark with fresh results:
- DFS (global IDF preflight): avg τ = 0.9817, min τ = 0.9523, 0 queries below 0.95 → PASS
- Score merge (local IDF): avg τ = 0.7938, 62.9% queries below 0.95 → FAIL
- RRF merge: avg τ = 0.1361, 100% queries below 0.95 → CATASTROPHIC

Added Criterion latency benchmarks to the research doc:
- Global IDF aggregation: 285ns (3 shards) → 3.31µs (50 shards)
- Query term extraction: 69ns (1 word) → 726ns (9 words)
- IDF computation: ~113ps per term (trivial)
- Coordinator-side overhead is sub-microsecond; dominant cost is network round-trip

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-19 05:31:13 -04:00
jedarden
096b43ccab P12.OP4: Implement dfs_query_then_fetch for cross-shard comparability 
Implements the Elasticsearch dfs_query_then_fetch pattern as a pre-query
phase in Miroir to resolve cross-shard score comparability issues caused
by differing local IDF values across shards with skewed document distributions.

Core changes:
- scatter.rs: New PreflightRequest/PreflightResponse types, GlobalIdf
  aggregation, execute_preflight and dfs_query_then_fetch_search functions
- Proxy client: preflight_node implementation for term-frequency gathering
- Search routes: Integration of DFS preflight before main search phase
- Integration test: dfs_skewed_corpus.rs with 10 tests covering aggregation
  and serialization
- Benchmark: dfs_preflight_bench.rs measuring preflight overhead

Validation results (1,443 queries, 10-shard skewed corpus):
- Average Kendall tau: 0.9815 (95% CI: [0.9809, 0.9821])
- Min tau: 0.9523 (zero queries below 0.95 threshold)
- Per-type: common-term +0.84, single-term +0.11, filtered +0.11

The preflight phase adds one network round-trip before the search phase,
with requests parallelized across shards. Estimated overhead: +1-2 RTTs.

Resolves bead miroir-yio: Global-IDF preflight implementation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-19 03:43:10 -04:00
jedarden
b201f0ff58 P12.OP4: Finalize score normalization validation — RRF τ=0.14, score τ=0.79 
Research complete: both score-based and RRF merge fail 0.95 threshold.
Updated research doc with full RRF validation results and confidence intervals.
Added benchmark result reports and helper tests. Follow-up bead miroir-n6v
created for global-IDF preflight (dfs_query_then_fetch pattern).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-19 02:40:54 -04:00
jedarden
9ce1b36206 P12.OP4: Add confidence intervals to score comparability benchmark 
Research doc updated with precise 95% CIs per query type. compare.py
now computes and reports confidence intervals. Kendall τ = 0.79
(95% CI [0.7873, 0.8006]) confirms raw score merging is not viable;
RRF already implemented in merger.rs as mitigation. Follow-up bead
created (miroir-zfo) for RRF quality validation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-19 00:07:42 -04:00
jedarden
72f9a197b5 P12.OP4: Score normalization at scale — research & benchmark infrastructure 
Completed Plan §15 Open Problem #4 research on cross-shard score comparability.

## Key Finding
Average Kendall tau: 0.79 vs. 0.95 threshold — FAIL

Cross-shard score comparability is a significant issue:
- Common-term queries: τ = 0.15 (catastrophic)
- Local IDF statistics cause score inflation on small shards
- Documents from 10-doc shards outrank 93K-doc shard results

## Recommendation
Implement Reciprocal Rank Fusion (RRF) for result merging.
Follow-up bead: miroir-nsu

## Artifacts Added
- Benchmark infrastructure: tests/benches/score-comparability/
  - Corpus generator with extreme shard skew (100× variance)
  - Query generator (10K random queries across 5 types)
  - BM25-based simulation with global vs local IDF
  - Kendall tau comparison tool
  - Full experimental results (τ = 0.79 ± 0.01, 95% CI)
- Research writeup: docs/research/score-normalization-at-scale.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-18 23:58:08 -04:00