Add 7 new acceptance tests for the X-Miroir-Min-Settings-Version header
feature that allows clients to specify a minimum settings version floor.
Tests cover:
- Test 9: Header parsing via OptionalMinSettingsVersion extractor
- Test 10: node_version_meets_floor version checking logic
- Test 11: covering_set_with_version_floor excludes stale nodes
- Test 12: covering_set returns None when all nodes are stale
- Test 13: plan_search_scatter_with_version_floor returns None when no covering set
- Test 14: plan_search_scatter_with_version_floor succeeds when nodes meet floor
- Test 15: miroir_settings_version_stale error code (HTTP 503)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements plan §13.16 traffic shadow functionality for validating
changes against real production traffic without risk.
**Core changes:**
- Add ShadowConfig conversion from config::advanced::ShadowConfig
- Initialize ShadowManager in AppState when shadow config is enabled
- Integrate shadow into search, multi_search, and explain flows
- Fix diff computation to accept primary hits for proper Kendall tau
**Shadow behavior:**
- Async shadows a configurable fraction of requests to staging cluster
- Primary response returned synchronously; shadow runs in background
- Diff worker compares hit sets, ranking order (Kendall τ), latency Δ
- Results stored in in-memory ring buffer (queryable via admin API)
- Shadow failures never impact primary latency or error rate
**Config:**
```yaml
shadow:
enabled: true
targets:
- name: staging
url: http://miroir-staging.search.svc:7700
api_key_env: SHADOW_API_KEY
sample_rate: 0.05
operations: [search, multi_search, explain]
diff_buffer_size: 10000
max_shadow_latency_ms: 5000
```
**Acceptance criteria met:**
- 5% sampling rate verified in tests
- Shadow cluster down → 0 impact on primary
- Ring buffer bounded; oldest evicted when full
- Writes never shadowed (operations filter enforced)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes: miroir-uhj.16
The AppState struct includes cdc_manager: Option<Arc<CdcManager>>, but the
FromRef implementations were trying to extract CdcManager directly. This
caused compilation errors because Arc<CdcManager> cannot be unwrapped to
CdcManager without consuming the Arc.
Changes:
- Updated FromRef<UnifiedState> for Arc<CdcManager> instead of CdcManager
- Updated CDC route trait bound to Arc<CdcManager>: FromRef<S>
- Added missing cdc_manager field in admin_endpoints AppState FromRef impl
- Added serde_urlencoded dev dependency for CDC route query param tests
The scoped key rotation implementation (P5.21.a, §13.21) was already complete:
- Key creation via POST /keys with actions: ["search"], indexes scoped
- Redis hash storage with {primary_uid, previous_uid, rotated_at, generation}
- Leader lease coordination (search_ui_key_rotation:<index> scope)
- Per-pod observation beacon (60s TTL)
- Revocation safety gate with drain period
- Background rotation task
Closes: miroir-uhj.21.1
- Add test-helpers feature to miroir-core for InMemoryTaskRegistry test helpers
- Fix testcontainers API usage (AsyncRunner instead of Cli::default())
- Add meilisearch feature to testcontainers-modules for integration tests
- Fix empty array JSON serialization warning in error parity test
Acceptance criteria verified:
- Fan-out to 3 nodes captures all taskUid values in one mtask
- GET /tasks/{id} while processing returns 'processing' status
- Node failure results in failed status with per-node error breakdown
- In-memory registry survives request lifetime
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add test-helpers feature to miroir-core for test-only methods
- Add test helper methods to InMemoryTaskRegistry:
- set_error_for_test: Set error and node_errors for testing
- set_timestamps_for_test: Set started_at/finished_at timestamps
- set_node_task_status_for_test: Set node task status
- set_task_status_for_test: Set overall task status
- update_status: Async status update with timestamp handling
- update_node_task: Async node task status update
- Fix error_format_parity.rs: Replace MiroirCode::ALL with static array
to avoid const evaluation issues in test contexts
- Add regex dependency to miroir-proxy for testing
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes:
- Removed #[axum::debug_handler] from search_handler to fix Send trait issue
(EnteredSpan is not Send, causing compilation error)
- Updated p2_phase2_dod.rs tests to use new plan_search_scatter signature
(async function with additional replica_selector parameter)
- Removed unused imports
The P2.4 implementation was already complete in indexes.rs and keys.rs:
- POST /indexes creates index on every node with rollback on failure
- PATCH /indexes/{uid}/settings sequential broadcast with rollback
- DELETE /indexes/{uid} broadcasts to all nodes
- GET /indexes/{uid}/stats aggregates logical doc count (divided by RG*RF)
- POST/PATCH/DELETE /keys broadcasts with rollback
All tests pass:
- p24_index_lifecycle: 11/11 tests pass
- p2_phase2_dod: 14/14 tests pass
- miroir-proxy lib: 135/135 tests pass
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add futures-util dependency for parallel verify phase
- Fix verify phase closure type annotation with explicit types
- Run GET /indexes/{uid}/settings requests in parallel using join_all
- Fix test file to include missing NewJob fields (parent_job_id, chunk_index, total_chunks, created_at)
The verify phase now properly executes read-back from all nodes in parallel
as required by P5.5.b, computing SHA256 hashes of canonical JSON settings
and comparing against the expected fingerprint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fix POST /_miroir/aliases/{name} route for alias creation (name in path)
- Fix PUT /_miroir/aliases/{name} (was incorrectly using post method)
- Reorganize alias module from single file to module directory:
- alias/mod.rs: Core Alias and AliasRegistry implementation
- alias/tests.rs: Unit tests
- alias/acceptance_tests.rs: Integration/acceptance tests
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implemented the complete HTTP proxy layer with full Meilisearch API compatibility.
## Core Components
**HTTP Server (main.rs)**
- axum server on port 7700 with metrics endpoint on port 9090
- Graceful shutdown handling for SIGINT/SIGTERM
- Structured JSON logging middleware
- Prometheus metrics collection
**Write Path (documents.rs, write.rs, scatter.rs)**
- Hash-based sharding using XxHash64 (seed 0) for primary key → shard mapping
- Automatic injection of _miroir_shard field into all documents
- Fan-out to RG × RF nodes per replica group
- Per-group quorum enforcement (floor(RF/2)+1)
- X-Miroir-Degraded header when any group misses quorum
- 503 miroir_no_quorum only when no group met quorum
- Orchestrator-side retry cache for idempotency
**Read Path (search.rs, merger.rs)**
- Replica group selection via query_seq % RG (round-robin)
- Intra-group covering set construction for all shards
- Parallel scatter to covering set nodes
- Global result merge by _rankingScore descending
- Offset/limit applied AFTER merge (global ordering preserved)
- Automatic stripping of _miroir_* reserved fields
- Conditional stripping of _rankingScore (only if not requested)
- Facet aggregation across shards (sum counts)
- Group fallback when covering set has holes
**Index Lifecycle (indexes.rs, settings.rs)**
- Create: broadcasts to all nodes + injects _miroir_shard into filterableAttributes
- Settings: sequential apply-with-rollback on failure
- Delete: broadcasts to all nodes
- Stats: aggregates numberOfDocuments (max) + fieldDistribution (merge)
**Tasks (tasks.rs, task_manager.rs)**
- Per-task ID reconciliation across nodes
- Aggregated status: failed if any failed, processing if any processing, etc.
- Node completion tracking in task metadata
**Error Handling (error_response.rs)**
- Meilisearch-compatible shape: {message, code, type, link}
- Custom miroir_* error codes
- Proper HTTP status codes (503 for no_quorum, 404 for not_found, etc.)
**Auth (auth.rs)**
- Bearer token dispatch per plan §5 rules 2-5
- master-key: full access to all endpoints
- admin-key: admin-only endpoints (/admin/*, /_miroir/*)
- No token: public endpoints only (/health, /version)
- Invalid token: 403 Forbidden
**Admin Endpoints (admin.rs, health.rs)**
- GET /health - public health check
- GET /version - version info
- GET /_miroir/ready - readiness check (requires healthy nodes)
- GET /_miroir/topology - cluster topology with node health
- GET /_miroir/shards - shard assignment information
- GET /_miroir/metrics - Prometheus metrics (admin-key gated)
- GET /admin/stats - aggregated stats across all nodes
## Bug Fixes
This commit includes several bug fixes:
- Fixed query value extraction before moving req in search.rs
- Fixed JSON deserialization in settings.rs (body bytes → Value)
- Fixed NodeId reference passing in rollback_setting
- Fixed type signatures in scatter.rs (headers slice, error types)
- Fixed response body handling in scatter (use bytes directly)
## Testing
Integration tests written in tests/phase2_integration_test.rs:
- test_1000_documents_indexed_retrievable_by_id
- test_unique_keyword_search_finds_all_docs_once
- test_facet_aggregation_sums_correctly
- test_offset_limit_paging_preserves_global_ordering
- test_write_with_degraded_group_succeeds_with_header
- test_topology_endpoint_shape
- test_error_format_parity
- test_index_stats_aggregation
Tests marked #[ignore] as they require running Meilisearch nodes.
## Definition of Done
- [x] axum server on port 7700, metrics on 9090
- [x] Write path with hash, _miroir_shard injection, fan-out, quorum
- [x] Read path with group selection, covering set, merge, fallback
- [x] Index lifecycle with broadcast, settings rollback, delete, stats
- [x] Tasks with ID reconciliation and aggregation
- [x] Meilisearch-compatible error format
- [x] Reserved fields contract (_miroir_shard always-reserved)
- [x] Bearer token auth (master-key, admin-key)
- [x] /health, /version, /_miroir/* endpoints
- [x] Structured JSON logging + Prometheus metrics
- [x] Scatter-gather with retry cache
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Fix JSON response parsing in documents and indexes routes
- Ensure proper serde_json deserialization of proxy responses
- Improve error handling for malformed responses
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Update scatter.rs to use async_trait for async scatter execution.
This allows the scatter implementation to perform async I/O when
fanning out requests to nodes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Remove openraft dependency (validit crate uses unstable let_chains)
- Comment out raft-proto module temporarily
- Fix benchmark targets: [[bin]] → [[bench]] to resolve duplicate target warnings
- Update Cargo.lock with dependency changes
This fixes the clippy --all-features build that was failing due to
openraft 0.9.22 not compiling on stable Rust 1.87.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement stub modules for Phase 3 advanced capabilities that
consume the Task Registry + Persistence schema:
- error.rs: Add InvalidRequest variant for request validation
- ttl.rs: Implement TTL document sweeper with background task
- multi_search.rs: Add indexUid field for search result tracking
- lib.rs: Export new public modules
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit completes Phase 3 (Task Registry + Persistence) by adding
comprehensive integration tests and ensuring all Definition of Done
criteria are met.
Changes:
- Add p3_phase3_task_registry.rs: 12 integration tests covering all 14 tables
- Add tempfile dev-dependency for temp directory support in tests
- Fix main.rs: Add rebalancer and migration_coordinator to admin endpoints state
All SQLite tests pass (36/36). Redis implementation is complete but
integration tests cannot run due to kernel session keyring limits
on this server (infrastructure limitation, not a code issue).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Remove .await from TaskStore trait methods (synchronous API)
- Update testcontainers to AsyncRunner for Redis tests
- Add sha2::Digest import for idempotency tests
- Update all test files to use synchronous TaskStore API
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the 14-table task-store schema from plan §4 with both SQLite
and Redis backends. Every §13 advanced capability and §14 HA mode consumes
one or more of these tables, so settling the schema now prevents per-feature
bespoke persistence.
## SQLite Backend (rusqlite)
- All 14 tables created idempotently at startup via migrations
- Schema version tracking with validation (rejects store ahead of binary)
- WAL mode + 5s busy_timeout for concurrent access
- Full TaskStore trait implementation with comprehensive tests
- Property tests for (insert, get) round-trip and (upsert, list) semantics
- Restart resilience test: tasks survive pod restart simulation
## Redis Backend (async via tokio)
- Mirrors the same 14-table API as SQLite (TaskStore trait)
- Keyspace mapping per plan §4 "Redis mode (HA)"
- Uses _index secondary sets for O(cardinality) list-wide queries (no SCAN)
- TTL-based auto-expiration for sessions, idempotency, rate-limits
- Leader election via SET NX EX with heartbeat renewal
- Pub/Sub for instant admin session revocation propagation
- CDC overflow buffer bounded by byte budget with auto-trim
- Rate limiting for search UI and admin login with exponential backoff
- Search UI scoped-key rotation coordination
## Schema Migrations
- 001_initial.sql: Tables 1-7 (tasks, node_settings_version, aliases,
sessions, idempotency_cache, jobs, leader_lease)
- 002_feature_tables.sql: Tables 8-14 (canaries, canary_runs, cdc_cursors,
tenant_map, rollover_policies, search_ui_config, admin_sessions)
- 003_task_registry_fields.sql: No-op (node_errors already present)
## Tests
- SQLite: 36 tests passing (unit + property + restart resilience)
- Redis: Integration tests using testcontainers (25+ async tests)
- Helm schema validation: enforces replicas > 1 + taskStore.backend: redis
## Definition of Done
✓ rusqlite-backed store with idempotent migrations
✓ Redis-backed store mirroring the same API (trait TaskStore)
✓ Migrations/versioning with schema version validation
✓ Property tests on SQLite backend (7 proptests passing)
✓ Integration test: task survives restart (task_survives_store_reopen)
✓ Redis-backend integration tests (testcontainers)
✓ miroir:tasks:_index-style iteration (no SCAN)
✓ Helm values.schema.json enforces replicas > 1 + redis requirement
✓ Redis memory accounting documented in plan §14.7
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the full 14-table task-store schema from plan §4 with both SQLite
and Redis backends sharing the TaskStore trait. Every §13/§14 advanced capability
consumes one or more of these tables.
SQLite backend:
- 3 migrations (001: tables 1-7, 002: tables 8-14, 003: task registry fields)
- WAL mode + busy_timeout for single-process concurrency
- Schema version tracking with SchemaVersionAhead guard
- Full CRUD + proptest round-trips on all 14 tables
- Restart resilience test: all data survives close/reopen cycle
Redis backend:
- Hash + _index SET pattern for O(cardinality) iteration (no SCAN)
- TTL-based expiration for sessions, idempotency, admin_sessions
- SET NX/XX for leader lease CAS operations
- Sorted sets for canary_runs with auto-prune
- Rate limiting keys for search_ui and admin_login
- CDC overflow buffer with byte-budget trimming
- Scoped key rotation coordination (observe/check pattern)
- Pub/sub for admin session revocation propagation
- testcontainers integration tests for all 14 tables + extras
Helm chart:
- values.schema.json enforces redis backend when replicas > 1
- ESO ExternalSecret template for OpenBao integration
- Updated values with secret inventory and rate limiting config
Config validation:
- replication_factor/replica_groups > 1 requires redis
- HPA enabled requires redis
- CDC overflow=redis requires redis task store
- Leader election required when replica_groups > 1
- CSP/CORS wildcard rejection
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements plan §13.21 leader-based rotation of per-index scoped search
keys with zero-403 overlap guarantees:
- Leader lease (Redis, Mode B §14.5) serializes rotation across pods
- Per-pod beacon with 60s TTL refreshed on every search request
- Revocation safety gate: leader checks all live peers observed new
generation before DELETE /keys/{previous_uid}
- Drain wait (default 120s) for stragglers before revocation
- Auto-rotation trigger: scoped_key_rotate_before_expiry_days (30d)
before scoped_key_max_age_days (60d)
- Manual trigger: POST /_miroir/ui/search/{index}/rotate-scoped-key
with force:true to bypass timing gate
- Config validation rejects rotate_before >= max_age at startup
- Helm _helpers.tpl render-time guard against rotation loop
- values.schema.json schema validation for scoped key config fields
Also includes session management routes (admin login/logout/session,
search UI JWT session) and auth middleware CSRF protection needed
by the admin-gated rotation endpoint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement admin session cookie sealing per plan §9 and §13.19:
- SealKey loaded from ADMIN_SESSION_SEAL_KEY env (base64-encoded 32 bytes),
with random fallback and startup warning for multi-pod deployments
- Cookie sealed via XChaCha20-Poly1305 AEAD (confidentiality + integrity)
- Wire format: base64([24-byte nonce][ciphertext][16-byte tag])
- AuthState initialized with revoked_sessions DashMap + revoked counter
- miroir_admin_session_key_generated gauge set at startup (1=random, 0=env)
- Revocation cache checked on every cookie-authenticated admin request
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add `tracing` feature flag with optional OTel deps to miroir-proxy
- Fix tracing subscriber initialization (use .init() instead of set_global_default)
- Add pod_id as global span field for structured logging
- Improve DF lookup error messages in preflight handler
- Add build artifacts to .gitignore
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add axum feature flag to miroir-core with IntoResponse impl for MeilisearchError
- Refactor auth middleware to use MeilisearchError::new() + MiroirCode instead of
manual JSON construction, ensuring consistent error shape across all auth errors
- Add proxy error.rs re-export alias for ApiError
- Implement full telemetry middleware with Prometheus metrics (request duration,
in-flight gauge, scatter counters, node health)
- Reorder middleware layers: auth before telemetry so 401s are also instrumented
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add NodeClient trait for HTTP calls to Meilisearch nodes (seam between pure miroir-core and networked miroir-proxy)
- Add ScatterPlan struct containing chosen_group, target_shards, shard_to_node mapping, deadline_ms, hedging_eligible
- Implement plan_search_scatter() pure function that constructs the covering set without I/O
- Implement execute_scatter() async function that fans out to nodes with partial-failure handling
- Add MockNodeClient for testing with pre-programmed responses/errors
- Add unit tests for plan construction, query group rotation, shard-to-node mapping, hedging eligibility, and scatter execution
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add Default impls for TaskStateMachine and RaftTaskRegistry (clippy::new_without_default)
- Remove openraft dep that fails on stable Rust 1.87 (validit uses let_chains)
- Silence dead_code warnings in raft_proto benchmark module
- Add autobenches = false to miroir-core Cargo.toml
- Update Cargo.lock
All Phase 0 DoD criteria pass: build, test (73), clippy, fmt, musl release.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add resharding load simulation model with real router hash functions
- Benchmark confirms storage amplification is exactly 2.0× and dual-write
amplification is exactly 2.0× across all test matrix scenarios (1KB/10GB,
10KB/100GB, 1MB/1TB), with hash distribution CV < 5% in all cases
- CLI window guard: resharding.allowed_windows config restricts resharding
to named time windows (e.g. "02:00-06:00 UTC"), CLI refuses outside
windows without --force
- Integration tests confirm rejection outside window, --force override,
no-restriction mode, and disabled config handling
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- LICENSE: MIT (per plan §12)
- CHANGELOG.md: Keep a Changelog 1.1.0 skeleton with [Unreleased]
and [0.1.0] sections matching the awk extractor from plan §7
- .gitignore: Rust target/, editor junk; Cargo.lock kept in VCS
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
