jedarden/miroir
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
325 commits 1 branch 0 tags 58 MiB
Commit graph

4 commits

Author SHA1 Message Date
jedarden
096b43ccab P12.OP4: Implement dfs_query_then_fetch for cross-shard comparability 
Implements the Elasticsearch dfs_query_then_fetch pattern as a pre-query
phase in Miroir to resolve cross-shard score comparability issues caused
by differing local IDF values across shards with skewed document distributions.

Core changes:
- scatter.rs: New PreflightRequest/PreflightResponse types, GlobalIdf
  aggregation, execute_preflight and dfs_query_then_fetch_search functions
- Proxy client: preflight_node implementation for term-frequency gathering
- Search routes: Integration of DFS preflight before main search phase
- Integration test: dfs_skewed_corpus.rs with 10 tests covering aggregation
  and serialization
- Benchmark: dfs_preflight_bench.rs measuring preflight overhead

Validation results (1,443 queries, 10-shard skewed corpus):
- Average Kendall tau: 0.9815 (95% CI: [0.9809, 0.9821])
- Min tau: 0.9523 (zero queries below 0.95 threshold)
- Per-type: common-term +0.84, single-term +0.11, filtered +0.11

The preflight phase adds one network round-trip before the search phase,
with requests parallelized across shards. Estimated overhead: +1-2 RTTs.

Resolves bead miroir-yio: Global-IDF preflight implementation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-19 03:43:10 -04:00
jedarden
612e7ce0ea P1.5: Implement scatter module with covering-set construction + dispatch trait 
- Add NodeClient trait for HTTP calls to Meilisearch nodes (seam between pure miroir-core and networked miroir-proxy)
- Add ScatterPlan struct containing chosen_group, target_shards, shard_to_node mapping, deadline_ms, hedging_eligible
- Implement plan_search_scatter() pure function that constructs the covering set without I/O
- Implement execute_scatter() async function that fans out to nodes with partial-failure handling
- Add MockNodeClient for testing with pre-programmed responses/errors
- Add unit tests for plan construction, query group rotation, shard-to-node mapping, hedging eligibility, and scatter execution

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-19 00:20:29 -04:00
jedarden
601988829d P0.1: Set up Cargo workspace + toolchain pin 
- Update workspace Cargo.toml: explicit members list, edition 2021, MIT license, rust-version 1.87
- Simplify workspace.dependencies to core shared deps
- Update member crates to use explicit dependency versions where workspace inheritance was removed

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-18 20:52:53 -04:00
jedarden
409f952f59 Add repo hygiene: LICENSE, CHANGELOG, .gitignore 
- LICENSE: MIT (per plan §12)
- CHANGELOG.md: Keep a Changelog 1.1.0 skeleton with [Unreleased]
  and [0.1.0] sections matching the awk extractor from plan §7
- .gitignore: Rust target/, editor junk; Cargo.lock kept in VCS

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-18 20:47:36 -04:00