Add OTel distributed tracing support with zero overhead when disabled.
Configuration (plan §10):
- tracing.enabled: false (default, zero overhead)
- tracing.endpoint: "http://tempo.monitoring.svc:4317"
- tracing.service_name: "miroir"
- tracing.sample_rate: 0.1 (head-based sampling)
Span hierarchy:
- Parent: inbound request (POST /indexes/:index/search)
- Child: scatter plan construction
- Parallel children: one per node in covering set
- Child: merge operation
Resource attributes: service.name, service.version, host.name
When disabled (tracing.enabled: false), no OTel library calls are made.
Shutdown handler flushes pending traces before exit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Uses FROM scratch for minimal image size (14.2 MB)
- Includes OCI labels: source, version, revision, licenses
- Exposes ports 7700 (main) and 9090 (metrics)
- Static musl binary for zero libc dependency
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- merger: deduplicate hits by primary key when multiple shards map to same node
- search: use shared AppState with live topology from health checker
- search: strip _miroir_shard always, _rankingScore only when not requested
- search: include facetDistribution only when facets were requested
- credentials: add mutex guards for env-var test isolation
- Add Phase 2 DoD integration tests: shard coverage, dedup, facets, paging,
degraded writes, error shape parity, topology shape, auth errors, reserved fields
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fix middleware module export from lib.rs so the crate compiles as a library.
Remove unused settings mock assertions from test_create_index_broadcasts_to_all_nodes
(the settings injection flow is already covered by test_miroir_shard_in_filterable_attributes).
All 11 acceptance tests pass:
- POST /indexes broadcasts to all nodes with rollback on failure
- _miroir_shard in filterableAttributes after creation
- GET /indexes/{uid}/stats logical doc count (divided by RG*RF)
- Settings broadcast sequential with rollback
- DELETE /indexes broadcasts to all nodes
- PATCH /indexes/{uid} snapshot and rollback
- /keys CRUD broadcasts with all-or-nothing semantics
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements POST/PUT /indexes/{uid}/documents and DELETE /indexes/{uid}/documents:
- Primary key extraction on hot path with 400 miroir_primary_key_required if missing
- _miroir_shard injection into every document before forwarding to nodes
- Rejection of _miroir_shard in client-submitted docs (400 miroir_reserved_field)
- Two-rule quorum: per-group floor(RF/2)+1 ACKs, success if ≥1 group meets quorum
- X-Miroir-Degraded header when any group misses quorum
- 503 miroir_no_quorum only when NO group meets quorum
- Per-batch grouping by target shard for efficient HTTP fan-out
- DELETE by IDs routes each ID independently to its shard
- DELETE by filter broadcasts to all nodes
Acceptance tests pass:
- Primary key validation before any writes
- Reserved field rejection
- Shard distribution uniformity (17-26 shards/node with 64 shards/3 nodes)
- Quorum calculation: floor(RF/2)+1
- Meilisearch-compatible error shape
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Remove unused ShardHitPage import from p23_search_read_path.rs
- All 10 acceptance tests pass:
- Unique-keyword search returns exactly 1 hit (RRF deduplication)
- Facet counts sum correctly across shards
- Paging with no dupes/gaps (5 pages of 10 = 50 unique results)
- Node down with RF=2: search still covers all shards
- Group down with fallback: uses other group, not degraded
- X-Miroir-Degraded header includes actual shard IDs
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement the search read path with scatter-gather + merge + group selection:
1. Group-unavailability fallback: When a shard has no available replica
in the primary group, the Fallback policy tries other replica groups
before failing. This provides full results (not degraded) when an
alternate group is healthy.
2. X-Miroir-Degraded header: Now includes actual shard IDs in the format
"X-Miroir-Degraded: shards=3,7,11" instead of just "partial".
3. Acceptance tests for P2.3:
- Unique-keyword search deduplicates correctly (RRF)
- Facet counts sum across shards
- Paging with no dupes/gaps
- Node down with RF=2 still covers all shards
- Group down falls back to other group (not degraded)
- Degraded header includes actual shard IDs
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Load Config (file + env + CLI args overlay) via MiroirConfig::load()
- Initialize tracing with JSON-to-stdout format (plan §10)
- Start two axum listeners: :7700 (client API) + :9090 (metrics, unauthenticated)
- Signal handlers for graceful shutdown (SIGTERM → drain → exit)
- GET /health returns {"status":"available"} immediately (Meilisearch-compatible)
- GET /version returns Meilisearch version from healthy node (60s TTL cache)
- GET /_miroir/ready returns 503 during startup, 200 once covering quorum reachable
- GET /_miroir/topology returns cluster state per plan §10 JSON shape
- GET /_miroir/shards returns shard → node mapping table
- GET /_miroir/metrics returns admin-key-gated Prometheus metrics
- Background health checker promotes nodes to Active when reachable
- UnifiedState bundles AuthState, Metrics, and admin_endpoints::AppState
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per plan §10, GET /_miroir/metrics is admin-key-gated so it can be
exposed outside the cluster. It was incorrectly marked as dispatch-exempt
with comment "admin-key-optional" - changed to require admin authentication.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
RRF merge (k=60) benchmarked against ground truth with 10K queries on
skewed 10-shard corpus (93% on shard 1). Result: Kendall τ = 0.1369
(95% CI [0.1339, 0.1399]), far below the 0.95 threshold. 9,998 of 10,000
queries fell below τ=0.95, confirming RRF alone is insufficient for
cross-shard ranking quality with skewed distributions.
DFS preflight (already implemented) achieves τ = 0.9818, passing the
threshold. Add full 10K-query DFS comparison report and fix paths in
experiment.json.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add axum feature flag to miroir-core with IntoResponse impl for MeilisearchError
- Refactor auth middleware to use MeilisearchError::new() + MiroirCode instead of
manual JSON construction, ensuring consistent error shape across all auth errors
- Add proxy error.rs re-export alias for ApiError
- Implement full telemetry middleware with Prometheus metrics (request duration,
in-flight gauge, scatter counters, node health)
- Reorder middleware layers: auth before telemetry so 401s are also instrumented
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement the API error response format from plan §5:
- ErrorType enum: invalid_request, auth, internal, system
- MiroirCode enum with all 10 miroir_* codes and their HTTP status mappings
- MeilisearchError struct with Meilisearch-compatible JSON shape
- Forwarding support for Meilisearch-native node errors (verbatim passthrough)
- Doc links pointing to docs/errors.md#<code>
- 21 unit tests covering every code's JSON shape, HTTP status, and forwarding
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the Elasticsearch dfs_query_then_fetch pattern as the
global-IDF preflight phase (OP#4). This solves the cross-shard score
comparability problem that caused both RRF (τ=0.14) and score-based
merge (τ=0.79) to fail the τ≥0.95 quality threshold.
Core changes:
- New DfsPhase in scatter-gather pipeline (scatter.rs):
- PreflightRequest/PreflightResponse for term statistics collection
- GlobalIdf for coordinator-side IDF aggregation
- execute_preflight() for phase 1 of DFS
- dfs_query_then_fetch_search() for full two-phase execution
- ScoreMergeStrategy in merger.rs for global-IDF scoring
- HttpClient with preflight_node() support (client.rs)
- Search route integration using dfs_query_then_fetch_search()
- Integration test with skewed corpus demonstrating the fix
The preflight phase adds ~15µs of aggregation overhead at 64 shards
(O(shards * terms)) with O(1) per-shard parallelization. Network
latency adds one round-trip before the actual search query.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
RRF τ = 0.14 (95% CI [0.134, 0.140]) — worse than score-based merge
(τ = 0.79) on the skewed corpus. Neither meets the 0.95 threshold.
Follow-up bead miroir-yio tracks the global-IDF preflight fix.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
RRF validation confirmed τ=0.14 against ground truth — RRF alone is
insufficient for cross-shard comparability. Follow-up bead miroir-yio
tracks the dfs_query_then_fetch implementation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Research complete: both score-based and RRF merge fail 0.95 threshold.
Updated research doc with full RRF validation results and confidence intervals.
Added benchmark result reports and helper tests. Follow-up bead miroir-n6v
created for global-IDF preflight (dfs_query_then_fetch pattern).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Research validated that both score-based (τ=0.79) and RRF (τ=0.14) merging
fail the 0.95 Kendall tau threshold with skewed shard distributions. Created
follow-up bead miroir-n6v for global-IDF preflight implementation.
Also: add __pycache__/ and tarpaulin-report.json to .gitignore, fix
task_pruner gauge test race.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Wire scatter (fan-out) directly into the RRF merger via scatter_gather_search(),
completing the full read path: plan → scatter → RRF merge. Add RRF simulation
mode to score-comparability benchmark for measuring rank correlation against
global BM25 ground truth.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add NodeClient trait for HTTP calls to Meilisearch nodes (seam between pure miroir-core and networked miroir-proxy)
- Add ScatterPlan struct containing chosen_group, target_shards, shard_to_node mapping, deadline_ms, hedging_eligible
- Implement plan_search_scatter() pure function that constructs the covering set without I/O
- Implement execute_scatter() async function that fans out to nodes with partial-failure handling
- Add MockNodeClient for testing with pre-programmed responses/errors
- Add unit tests for plan construction, query group rotation, shard-to-node mapping, hedging eligibility, and scatter execution
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Background pruner batch-deletes terminal tasks (succeeded/failed/canceled)
older than task_registry.ttl_seconds (default 7d). Runs every prune_interval_s
(default 300s) with batch_size=10000. Uses advisory lock via leader_lease table
to prevent concurrent pruning in single-pod deployments. Exposes
miroir_task_registry_size gauge updated after each cycle.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extends SqliteTaskStore with full CRUD operations for:
- Table 8: canaries (upsert, get, list, delete)
- Table 9: canary_runs (insert with auto-prune to run_history_limit)
- Table 10: cdc_cursors (upsert, get, list by sink)
- Table 11: tenant_map (insert, get by BLOB key, delete)
- Table 12: rollover_policies (upsert, get, list, delete)
- Table 13: search_ui_config (upsert, get, delete)
- Table 14: admin_sessions (insert, get, revoke, delete_expired)
Key implementation details:
- prune_tasks uses subquery for LIMIT support (SQLite doesn't support LIMIT in DELETE)
- canary_runs auto-prune keeps only N most recent runs per canary_id
- tenant_map.api_key_hash is a 32-byte BLOB (raw sha256)
- admin_sessions has expires_at index for lazy eviction
- All bool fields stored as INTEGER (0/1) with conversion on read/write
Adds 12 comprehensive unit tests covering:
- CRUD round-trip for each table
- Auto-prune logic for canary_runs
- Nullable fields (tenant_map.group_id, admin_sessions.user_agent/source_ip)
- Composite PK behavior (cdc_cursors, canary_runs)
- prune_tasks batch deletion with status filter
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement a first-class schema version system with the following components:
- schema_versions table (SQLite) tracking applied migrations
- Numbered migration files (001_initial.sql, 002_feature_tables.sql)
- MigrationRegistry for version validation and pending migration detection
- Startup: read current version → apply pending migrations → record latest
- Refuse to start if DB version > binary version (SchemaVersionAhead error)
Acceptance criteria met:
- First run creates schema at version 001
- Second run is a no-op (single SELECT for version check)
- Store version > binary version fails with SchemaVersionAhead error
- Migration metadata structure is backend-agnostic (shared by SQLite/Redis)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Research doc updated with precise 95% CIs per query type. compare.py
now computes and reports confidence intervals. Kendall τ = 0.79
(95% CI [0.7873, 0.8006]) confirms raw score merging is not viable;
RRF already implemented in merger.rs as mitigation. Follow-up bead
created (miroir-zfo) for RRF quality validation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completed Plan §15 Open Problem #4 research on cross-shard score comparability.
## Key Finding
Average Kendall tau: 0.79 vs. 0.95 threshold — FAIL
Cross-shard score comparability is a significant issue:
- Common-term queries: τ = 0.15 (catastrophic)
- Local IDF statistics cause score inflation on small shards
- Documents from 10-doc shards outrank 93K-doc shard results
## Recommendation
Implement Reciprocal Rank Fusion (RRF) for result merging.
Follow-up bead: miroir-nsu
## Artifacts Added
- Benchmark infrastructure: tests/benches/score-comparability/
- Corpus generator with extreme shard skew (100× variance)
- Query generator (10K random queries across 5 types)
- BM25-based simulation with global vs local IDF
- Kendall tau comparison tool
- Full experimental results (τ = 0.79 ± 0.01, 95% CI)
- Research writeup: docs/research/score-normalization-at-scale.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Define the TaskStore trait in miroir-core with SQLite backend for the
seven always-present tables from plan §4: tasks, node_settings_version,
aliases, sessions, idempotency_cache, jobs, leader_lease.
Key design choices:
- serde_json Value columns for tasks.node_tasks and aliases JSON fields
- BLOB (32 raw bytes) for idempotency_cache.body_sha256
- CAS operations for job claims and leader lease acquisition
- Idempotent migrations via schema_versions table with version gating
- WAL mode + busy_timeout=5000 for concurrent write safety
- Mutex<Connection> for thread-safe single-process access
15 tests covering CRUD round-trips, alias flipping with history
retention, session/idempotency expiry, job claim/renew, leader lease
acquire/renew/steal, migration idempotency, WAL mode, and concurrent
writes without deadlock.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Expand NodeStatus with Active/Degraded/Removed variants and implement
state-machine transitions covering the full plan §2 lifecycle (Joining→Active→
Draining→Removed, failure/recovery paths). Add is_write_eligible_for() for
shard-aware write eligibility during drain. Restructure Topology with
shards/replica_groups/rf fields, Vec<Node> storage, and custom serde that
auto-rebuilds the group index on deserialization. Add Group::healthy_nodes()
helper. Rename Node.url→address to match plan §4 YAML schema.
13 new tests: YAML round-trip, group iteration, all legal/illegal state
transitions, write-eligibility correctness table, healthy_nodes filtering.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add Default impls for TaskStateMachine and RaftTaskRegistry (clippy::new_without_default)
- Remove openraft dep that fails on stable Rust 1.87 (validit uses let_chains)
- Silence dead_code warnings in raft_proto benchmark module
- Add autobenches = false to miroir-core Cargo.toml
- Update Cargo.lock
All Phase 0 DoD criteria pass: build, test (73), clippy, fmt, musl release.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Verified CI smoke pipeline runs end-to-end in ~5:39 on iad-ci.
All three checks pass: fmt, clippy, test.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
