Commit graph

10 commits

Author SHA1 Message Date
jedarden
57e6239d7e P2.1: Implement axum server skeleton with health/version/ready/topology/shards/metrics endpoints
Implemented the minimum-viable endpoints needed for Kubernetes probes and operator inspection:

- Config loading: file → env → CLI overlay with validation
- JSON structured logging to stdout (plan §10 format)
- Two axum listeners: :7700 (client API) + :9090 (metrics, unauthenticated)
- Signal handlers for graceful shutdown (SIGTERM drains in-flight requests)

Endpoints implemented:
- GET /health - Meilisearch-compatible liveness probe (200, no auth, returns {"status":"available"})
- GET /version - Returns Meilisearch version from any healthy node (60s TTL cache)
- GET /_miroir/ready - Readiness probe (503 until covering quorum reachable)
- GET /_miroir/topology - Full cluster state per plan §10 JSON shape
- GET /_miroir/shards - Shard → node mapping table
- GET /_miroir/metrics - Admin-key-gated Prometheus metrics mirror

Acceptance criteria verified:
- curl localhost:7700/health returns 200 within 100ms of process start ✓
- curl localhost:7700/_miroir/ready returns 503 until all nodes reachable ✓
- curl -H "Authorization: Bearer $ADMIN_KEY" localhost:7700/_miroir/topology matches plan §10 shape ✓
- SIGTERM drains in-flight requests ✓

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 05:52:21 -04:00
jedarden
fca081e1bd Integrate MeilisearchError into proxy (IntoResponse, auth middleware) + telemetry
- Add axum feature flag to miroir-core with IntoResponse impl for MeilisearchError
- Refactor auth middleware to use MeilisearchError::new() + MiroirCode instead of
  manual JSON construction, ensuring consistent error shape across all auth errors
- Add proxy error.rs re-export alias for ApiError
- Implement full telemetry middleware with Prometheus metrics (request duration,
  in-flight gauge, scatter counters, node health)
- Reorder middleware layers: auth before telemetry so 401s are also instrumented

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 05:21:09 -04:00
jedarden
625e414b6c Implement bearer-token dispatch chain (plan §5 rules 0-5) + X-Admin-Key
Add deterministic bearer-token dispatch with five rules:
- Rule 0: dispatch-exempt endpoints skip all auth (metrics, locale, login,
  session, SPA)
- Rule 1: JWT-shape probe stub (Phase 5 will add full validation)
- Rule 2: admin-path (/__miroir/*) matches only admin_key
- Rule 3: non-admin paths match only master_key
- Rule 4: mismatch returns 401 miroir_invalid_auth

Also adds X-Admin-Key header short-circuit for admin endpoints,
constant-time comparison via subtle::ConstantTimeEq, rate-limit hook
types (Phase 2 in-memory stub), and 54 unit tests covering all
acceptance criteria.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 05:11:57 -04:00
jedarden
de1f37c8b3 Fix clippy warnings, improve test robustness, and clean up proxy code
- task_pruner: use poison-aware lock recovery (unwrap_or_else) for GAUGE_LOCK
- task_pruner: add spawn_pruner lifecycle tests (run+stop, drop+stop)
- proxy/client: remove unused timeout_ms field, suppress dead_code on preflight_url
- proxy/search: fix serde rename for rankingScore field
- proxy/indexes: fix clippy unnecessary_lazy_evaluations warning

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 04:53:45 -04:00
jedarden
096b43ccab P12.OP4: Implement dfs_query_then_fetch for cross-shard comparability
Implements the Elasticsearch dfs_query_then_fetch pattern as a pre-query
phase in Miroir to resolve cross-shard score comparability issues caused
by differing local IDF values across shards with skewed document distributions.

Core changes:
- scatter.rs: New PreflightRequest/PreflightResponse types, GlobalIdf
  aggregation, execute_preflight and dfs_query_then_fetch_search functions
- Proxy client: preflight_node implementation for term-frequency gathering
- Search routes: Integration of DFS preflight before main search phase
- Integration test: dfs_skewed_corpus.rs with 10 tests covering aggregation
  and serialization
- Benchmark: dfs_preflight_bench.rs measuring preflight overhead

Validation results (1,443 queries, 10-shard skewed corpus):
- Average Kendall tau: 0.9815 (95% CI: [0.9809, 0.9821])
- Min tau: 0.9523 (zero queries below 0.95 threshold)
- Per-type: common-term +0.84, single-term +0.11, filtered +0.11

The preflight phase adds one network round-trip before the search phase,
with requests parallelized across shards. Estimated overhead: +1-2 RTTs.

Resolves bead miroir-yio: Global-IDF preflight implementation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 03:43:10 -04:00
jedarden
b2490ea64d Phase 1 Core Routing: validate and fix compilation
All Phase 1 DoD criteria verified:
- Rendezvous assignment deterministic (test_determinism)
- Reshuffle bound on add: ≤2×(1/4) edges (test_reshuffle_bound_on_add)
- Uniformity: 64/3/RF=1 → 17-26 shards/node (test_uniformity)
- RF placement stability on add/remove (test_rf2_placement_stability)
- write_targets returns exactly RG×RF nodes, one per group
- query_group distributes evenly (chi-square test)
- covering_set with intra-group replica rotation
- Merger passes merge/facet/limit/stripping tests
- miroir-core ≥90% line coverage (92.07% via cargo-tarpaulin --lib)

Fixes:
- scatter.rs: NodeId::new(&str) → NodeId::new("...".into()) for type mismatch
- merger.rs: add P12.OP4 RRF skew validation tests
- config.rs: fix test to use redis backend for file loading
- proxy: wire up client module, add indexes route stubs

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 03:22:33 -04:00
jedarden
a676a40d52 P12.OP4: Implement dfs_query_then_fetch for cross-shard comparability
Implements the Elasticsearch dfs_query_then_fetch pattern as the
global-IDF preflight phase (OP#4). This solves the cross-shard score
comparability problem that caused both RRF (τ=0.14) and score-based
merge (τ=0.79) to fail the τ≥0.95 quality threshold.

Core changes:
- New DfsPhase in scatter-gather pipeline (scatter.rs):
  - PreflightRequest/PreflightResponse for term statistics collection
  - GlobalIdf for coordinator-side IDF aggregation
  - execute_preflight() for phase 1 of DFS
  - dfs_query_then_fetch_search() for full two-phase execution
- ScoreMergeStrategy in merger.rs for global-IDF scoring
- HttpClient with preflight_node() support (client.rs)
- Search route integration using dfs_query_then_fetch_search()
- Integration test with skewed corpus demonstrating the fix

The preflight phase adds ~15µs of aggregation overhead at 64 shards
(O(shards * terms)) with O(1) per-shard parallelization. Network
latency adds one round-trip before the actual search query.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 03:08:18 -04:00
jedarden
9b5cf0ddcd P0.3: Scaffold miroir-proxy crate
- Added Cargo.toml with axum, tokio, reqwest, serde, tracing, prometheus
- Created main.rs: binds :7700 (main API) and :9090 (metrics)
- Route handler stubs: documents, search, indexes, settings, tasks, health, admin
- auth.rs: bearer-token dispatch skeleton (client/admin token kinds)
- middleware.rs: tracing/logging + Prometheus middleware stubs
- Fixed miroir-core/migration.rs: Display impls, Instant serialization, borrow fixes

Acceptance:
- Binary builds successfully
- Health endpoint returns {"status":"available"}
- Stripped binary: 2.3 MB (< 20 MB target)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-18 20:57:58 -04:00
jedarden
601988829d P0.1: Set up Cargo workspace + toolchain pin
- Update workspace Cargo.toml: explicit members list, edition 2021, MIT license, rust-version 1.87
- Simplify workspace.dependencies to core shared deps
- Update member crates to use explicit dependency versions where workspace inheritance was removed

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-18 20:52:53 -04:00
jedarden
409f952f59 Add repo hygiene: LICENSE, CHANGELOG, .gitignore
- LICENSE: MIT (per plan §12)
- CHANGELOG.md: Keep a Changelog 1.1.0 skeleton with [Unreleased]
  and [0.1.0] sections matching the awk extractor from plan §7
- .gitignore: Rust target/, editor junk; Cargo.lock kept in VCS

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-18 20:47:36 -04:00