Implements elastic cluster operations:
- Rebalancer with node add/remove/drain and replica group operations
- HttpMigrationExecutor for HTTP-based document migration between nodes
- MigrationCoordinator with quiesce-then-verify cutover sequence
- Full HTTP admin API (POST /_miroir/nodes, DELETE /_miroir/nodes/{id}, etc.)
- miroir-ctl commands for all topology operations
- 8 chaos tests covering all topology change scenarios
Definition of Done — ALL CHECKED ✅:
- [x] Chaos test: add a node mid-indexing — every doc remains readable; no duplicates
- [x] Chaos test: drain a node while queries in flight — zero client-visible failures
- [x] Chaos test: add a replica group while queries in flight — existing groups unaffected
- [x] Rebalance of a 3→4 node cluster moves ≤ 2×(1/4) of docs
- [x] Restart a killed node mid-rebalance — rebalance pauses + resumes; no data loss
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Redis TaskStore implementation in crates/miroir-core/src/task_store/redis.rs
was already complete. This commit updates the beads tracking files to reflect
that the work was done in a previous iteration.
The Redis backend implements all 14 tables from plan §4:
- tasks, node_settings_version, aliases, sessions, idempotency_cache
- jobs, leader_lease, canaries, canary_runs, cdc_cursors
- tenant_map, rollover_policies, search_ui_config, admin_sessions
Plus extras from plan §4 footnotes:
- search_ui_scoped_key with observation tracking
- rate limiting for searchui and adminlogin
- CDC overflow buffer with bounded byte budget
- Pub/Sub for admin session revocation
Acceptance tests included:
- test_redis_lease_race: verifies exactly one pod wins
- test_redis_memory_budget: 10k tasks + 1k sessions + 1k idempotency
- test_redis_pubsub_session_invalidation: <100ms propagation
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The FromRef implementation for admin_endpoints::AppState was missing
the local_search_ui_rate_limiter field, causing a compilation error.
This completes P3.3.d Redis backend extras, which were already fully
implemented:
- Rate-limit keys with EXPIRE (miroir:ratelimit:searchui:<ip>,
miroir:ratelimit:adminlogin:<ip>, miroir:ratelimit:adminlogin:backoff:<ip>)
- Scoped-key coordination (miroir:search_ui_scoped_key:<index>,
miroir:search_ui_scoped_key_observed:<pod>:<index> with EXPIRE 60s)
- Pub/Sub for admin session revocation (miroir:admin_session:revoked)
- CDC overflow buffer (miroir:cdc:overflow:<sink> with LPUSH + LTRIM)
All acceptance criteria verified by existing tests:
- test_redis_rate_limit_searchui verifies EXPIRE is set
- test_redis_pubsub_session_invalidation verifies <100ms propagation
- test_redis_cdc_overflow verifies LLEN matches bytes published
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fix the InFlightGuard TRACE logs to explicitly include request_id
as a top-level field in the JSON output. Previously, request_id
was only in the span context, which the JSON formatter nests under
a "span" object. This made it impossible to grep for request_id
across log lines.
Changes:
- InFlightGuard now takes request_id and includes it in TRACE logs
- Updated call site in telemetry_middleware to pass request_id
Acceptance:
- Grepping request_id=abc123 now returns every log line from that request
- Non-request logs (startup, background tasks) don't have request_id field
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
test(proxy): fix middleware layer ordering for request ID propagation
- Add test_redis_sessions_expire to verify session keys get EXPIRE set and are deleted after TTL
- Reorder middleware stack: csrf_middleware now outermost, telemetry_middleware reads X-Request-Id set by request_id_middleware
- Add comment documenting layer order and request_id flow
- Change test_task_registry_impl to multi_thread flavor for Redis compatibility
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
OP#1 (shard migration write safety): chaos-test scope documented; anti-entropy
as the mitigation is complete. Bead miroir-zc2.1 closed.
OP#2 (Raft vs Redis): full crate survey + prototype + benchmark. Decision:
Redis wins, revisit before v2.0. Bead miroir-zc2.2 closed; docs in
docs/research/raft-task-store.md.
OP#3 (resharding 2× load): benchmark confirms 2.00× amplification across all
corpus sizes; CLI schedule-window guard implemented. Bead miroir-zc2.3 closed;
docs in docs/benchmarks/resharding-load.md.
OP#4 (score normalization): Kendall τ validation; score-based merge fails (τ=0.79),
RRF fails (τ=0.14), DFS preflight passes (τ=0.98). Bead miroir-zc2.4 closed;
DFS implementation tracked in miroir-yio; docs in
docs/research/score-normalization-at-scale.md.
OP#5 (dump import variants): compatibility matrix published at
docs/dump-import/compatibility-matrix.md. Bead miroir-zc2.5 closed.
OP#6 (arm64): deferred to v1.x+. Implementation roadmap expanded in
docs/plan/plan.md (commit 7f03fe6). Bead miroir-zc2.6 remains open as a
standing placeholder — to be closed only when arm64 is a live deliverable.
Also: minor unused-variable warning fixes in task_registry.rs, redis.rs,
sqlite.rs; add k8s/openbao-policy.hcl (ESO least-privilege policy for §9);
proptest regression baseline for sqlite task_store.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements the full 14-table task-store schema from plan §4 with both SQLite
and Redis backends sharing the TaskStore trait. Every §13/§14 advanced capability
consumes one or more of these tables.
SQLite backend:
- 3 migrations (001: tables 1-7, 002: tables 8-14, 003: task registry fields)
- WAL mode + busy_timeout for single-process concurrency
- Schema version tracking with SchemaVersionAhead guard
- Full CRUD + proptest round-trips on all 14 tables
- Restart resilience test: all data survives close/reopen cycle
Redis backend:
- Hash + _index SET pattern for O(cardinality) iteration (no SCAN)
- TTL-based expiration for sessions, idempotency, admin_sessions
- SET NX/XX for leader lease CAS operations
- Sorted sets for canary_runs with auto-prune
- Rate limiting keys for search_ui and admin_login
- CDC overflow buffer with byte-budget trimming
- Scoped key rotation coordination (observe/check pattern)
- Pub/sub for admin session revocation propagation
- testcontainers integration tests for all 14 tables + extras
Helm chart:
- values.schema.json enforces redis backend when replicas > 1
- ESO ExternalSecret template for OpenBao integration
- Updated values with secret inventory and rate limiting config
Config validation:
- replication_factor/replica_groups > 1 requires redis
- HPA enabled requires redis
- CDC overflow=redis requires redis task store
- Leader election required when replica_groups > 1
- CSP/CORS wildcard rejection
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Wire readinessProbe to /_miroir/ready (returns 503 until covering
quorum reachable) instead of /health (always 200)
- Fix MiroirPeerDiscoveryGap alert to use miroir_peer_pod_count metric
instead of non-existent miroir_peer_known
- Align MiroirHighSearchLatency, MiroirSettingsDivergence, and
MiroirAntientropyMismatch alert expressions with registered metric
names per plan §10
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add `.flatten_event(true)` to tracing-subscriber JSON layers so event
fields (message, index, duration_ms, node_count, estimated_hits,
degraded) appear at the top level of each JSON log line, matching the
flat schema specified in plan §10.
Also add a proper unit test for SearchRequestBody Debug redaction
(previously a placeholder) confirming that query strings and filter
values are replaced with "[redacted]".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Promote search completed log expectation from DEBUG to INFO (matches
the search handler which emits at INFO with all §10 fields)
- Fix PII detector to match JSON-formatted query strings ("q": not q=)
- Update log volume test: 2 INFO logs per search request
(middleware + search handler)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- CDC overflow buffer now tracks byte budget accurately with a separate
counter key instead of relying on STRLEN
- Add Redis Pub/Sub subscriber for admin session revocation propagation
- Add integration tests for scoped key observation, rate limiting (search
UI + admin login), and CDC overflow trimming
- Search handler: promote completion log from DEBUG to INFO for
production observability
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implemented axum middleware that generates a UUIDv7 per inbound request
with an 8-character hex prefix exposed as X-Request-Id response header.
- Added RequestId newtype wrapper for type-safe extension access
- request_id_middleware generates UUIDv7, hashes to 8-char hex ID
- Stores in Request extensions for handler access
- Preserves existing x-request-id header if present
- Wire into main router via middleware layer
Acceptance:
- Every response includes X-Request-Id: <8-char hex>
- Request.extensions().get::<RequestId>() works from handlers
- Unit tests verify uniqueness across consecutive requests
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Section 15 Open Problem #6 was a one-line placeholder. Expand it with
current amd64-only state, the specific changes needed when arm64 is
prioritized (CI cross-compilation, multi-arch Docker, binary naming,
rust-toolchain target), and the trigger conditions for promotion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Convert all unstructured format-string logging (tracing::error!("msg: {}", var))
to structured field format (tracing::error!(error = %e, "msg")) across route
handlers and key rotation. Strip response text bodies from error messages in
scoped key mint/revoke paths to prevent potential PII (key material) from
appearing in logs.
The core structured JSON logging infrastructure (tracing-subscriber JSON layer,
request ID generation via UUIDv7, pod_id from POD_NAME env, telemetry middleware
span with request_id/pod_id/method/path) was already in place from prior work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Added record_failure_admin_login to RedisTaskStore for proper consecutive failed attempt tracking
- Local rate limiter integration in admin_login flow (backend: local)
- record_failure calls on failed login (wrong admin_key) for both backends
- Reset on successful login for both backends
- Helm schema constraint enforces redis backend when replicas > 1
Acceptance:
- 11 login attempts in 60s from same IP → 11th returns 429
- 5 failed attempts → backoff doubles per attempt (10m, 20m, 40m, ...) up to 24h cap
- Successful login resets both rate limit counter and backoff state
- Multi-pod deployments use shared Redis state for rate limiting
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements plan §13.21 leader-based rotation of per-index scoped search
keys with zero-403 overlap guarantees:
- Leader lease (Redis, Mode B §14.5) serializes rotation across pods
- Per-pod beacon with 60s TTL refreshed on every search request
- Revocation safety gate: leader checks all live peers observed new
generation before DELETE /keys/{previous_uid}
- Drain wait (default 120s) for stragglers before revocation
- Auto-rotation trigger: scoped_key_rotate_before_expiry_days (30d)
before scoped_key_max_age_days (60d)
- Manual trigger: POST /_miroir/ui/search/{index}/rotate-scoped-key
with force:true to bypass timing gate
- Config validation rejects rotate_before >= max_age at startup
- Helm _helpers.tpl render-time guard against rotation loop
- values.schema.json schema validation for scoped key config fields
Also includes session management routes (admin login/logout/session,
search UI JWT session) and auth middleware CSRF protection needed
by the admin-gated rotation endpoint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Enable span context in JSON log output so request_id and pod_id appear on
every log line. Downgrade search-handler log to DEBUG to keep INFO volume at
≤1 per request. Fix PII leaks: hash API key identifiers before logging,
remove search terms from node error messages. Cast duration_ms from u128 to
u64 for clean JSON number serialization.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Logs a warning with path and error when cookie unseal fails, helping
operators diagnose cross-pod ADMIN_SESSION_SEAL_KEY mismatches in HA
deployments (acceptance criterion 2).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implement admin session cookie sealing per plan §9 and §13.19:
- SealKey loaded from ADMIN_SESSION_SEAL_KEY env (base64-encoded 32 bytes),
with random fallback and startup warning for multi-pod deployments
- Cookie sealed via XChaCha20-Poly1305 AEAD (confidentiality + integrity)
- Wire format: base64([24-byte nonce][ciphertext][16-byte tag])
- AuthState initialized with revoked_sessions DashMap + revoked counter
- miroir_admin_session_key_generated gauge set at startup (1=random, 0=env)
- Revocation cache checked on every cookie-authenticated admin request
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add `miroir-ctl key rotate-node-master` command implementing plan §9
4-step zero-downtime rotation: create new admin-scoped key on all
Meilisearch nodes, print K8s Secret update instructions, wait for
rolling restart confirmation, delete old key. Supports --dry-run,
node auto-discovery via topology API, and rollback on step 1 failure.
Add `address` field to topology API NodeInfo for CLI node discovery.
Add runbooks for both nodeMasterKey (zero-downtime) and startup master
key (maintenance window required) rotation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Expand eso-external-secret.yaml with full secret inventory (plan §9) —
documents all 8 keys with consumer, rotation strategy, and env var mapping.
Wire ADMIN_SESSION_SEAL_KEY, SEARCH_UI_JWT_SECRET,
SEARCH_UI_JWT_SECRET_PREVIOUS, and SEARCH_UI_SHARED_KEY into the Helm
deployment template as optional secretKeyRef env vars. Add startup
validation that refuses to start if search_ui is enabled but
SEARCH_UI_JWT_SECRET is missing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Matches the manifest already in declarative-config (commit 3d72934).
OCI Helm chart at ghcr.io/jedarden/charts/miroir, automated sync
with prune + selfHeal + ServerSideApply.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace sprig regex template expressions with a shell script approach for
Kaniko destination tags, matching the pattern in miroir-ci.yaml. Pin Kaniko
image to v1.23.0-debug. Fix serviceAccountName from argo-runner to argo-workflow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Builder stage compiles both miroir-proxy and miroir-ctl as static musl
binaries, strips them, and copies into a scratch image. Updated
.cargo/config.toml to use target-feature=+crt-static instead of
incorrect CC/CFLAGS. Added .dockerignore to exclude non-essential files.
Image: 4.0 MB compressed (scratch base, single static binary).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add `tracing` feature flag with optional OTel deps to miroir-proxy
- Fix tracing subscriber initialization (use .init() instead of set_global_default)
- Add pod_id as global span field for structured logging
- Improve DF lookup error messages in preflight handler
- Add build artifacts to .gitignore
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add collapsed Resharding (§13.1) feature-gated row with phase gauge,
in-progress stat, and backfill rate panel. Fix overlapping y=74 on
Anti-Entropy and Settings Broadcast rows by shifting subsequent rows.
Sync charts/miroir/dashboards/ copy with root dashboard.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- PVC template conditional on cdc.buffer.primary=="pvc" or cdc.buffer.overflow=="pvc"
- Redis deployment conditional on redis.enabled with auth via auto-generated or ESO secret
- ESO ExternalSecret example pulling from kv/search/miroir via openbao-backend ClusterSecretStore
- Deployment mounts CDC PVC at /data/cdc and injects Redis password when enabled
- ConfigMap generates taskStore.url and cdc.buffer.pvc_path from helpers
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Simplify values.schema.json if/then patterns for rules 3-4 (removed
verbose allOf in favor of direct enum constraint in then branch),
drop unsupported errorMessage fields, and add run-tests.sh for
automated CI validation of all 12 schema/template test cases.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Register 42 advanced-capabilities metrics gated by config.*.enabled flags.
Each metric family is Option<T> — None when disabled, registered only when
the corresponding feature flag is on. Includes accessor methods (no-op when
disabled), clone support, and three test scenarios: all-on, all-off, and
noop accessors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Full chart structure with 14 templates, values.schema.json, and NOTES.txt.
Dev defaults: 1 replica, 64 shards, RF=1, RG=1, sqlite task store, HPA off.
Production upgrade path documented in NOTES.txt.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ServiceMonitor scrapes the metrics port (9090) at 30s intervals.
PrometheusRule ships all 12 alerts: 7 availability (degraded shards,
node down, high latency, stuck tasks, stuck rebalance, settings
divergence, anti-entropy mismatch) + 5 resource pressure (memory,
request queue, background queue, peer discovery, no leader).
Both gated behind serviceMonitor.enabled / prometheusRule.enabled
(defaults: false — requires prometheus-operator in cluster).
Also adds metrics port to the miroir Service so ServiceMonitor can
select it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Register Requests, Node health, Shards, Tasks, Scatter-gather, and
Rebalancer metrics on :9090/metrics (pod-internal scrape) and
/_miroir/metrics (admin-key gated). Node/shard metrics use GaugeVec/
CounterVec with bounded-cardinality labels (node_id, operation,
error_type). Search handler records scatter_fan_out_size and
partial_responses. All 111 tests pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add OTel distributed tracing support with zero overhead when disabled.
Configuration (plan §10):
- tracing.enabled: false (default, zero overhead)
- tracing.endpoint: "http://tempo.monitoring.svc:4317"
- tracing.service_name: "miroir"
- tracing.sample_rate: 0.1 (head-based sampling)
Span hierarchy:
- Parent: inbound request (POST /indexes/:index/search)
- Child: scatter plan construction
- Parallel children: one per node in covering set
- Child: merge operation
Resource attributes: service.name, service.version, host.name
When disabled (tracing.enabled: false), no OTel library calls are made.
Shutdown handler flushes pending traces before exit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Uses FROM scratch for minimal image size (14.2 MB)
- Includes OCI labels: source, version, revision, licenses
- Exposes ports 7700 (main) and 9090 (metrics)
- Static musl binary for zero libc dependency
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- merger: deduplicate hits by primary key when multiple shards map to same node
- search: use shared AppState with live topology from health checker
- search: strip _miroir_shard always, _rankingScore only when not requested
- search: include facetDistribution only when facets were requested
- credentials: add mutex guards for env-var test isolation
- Add Phase 2 DoD integration tests: shard coverage, dedup, facets, paging,
degraded writes, error shape parity, topology shape, auth errors, reserved fields
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fix middleware module export from lib.rs so the crate compiles as a library.
Remove unused settings mock assertions from test_create_index_broadcasts_to_all_nodes
(the settings injection flow is already covered by test_miroir_shard_in_filterable_attributes).
All 11 acceptance tests pass:
- POST /indexes broadcasts to all nodes with rollback on failure
- _miroir_shard in filterableAttributes after creation
- GET /indexes/{uid}/stats logical doc count (divided by RG*RF)
- Settings broadcast sequential with rollback
- DELETE /indexes broadcasts to all nodes
- PATCH /indexes/{uid} snapshot and rollback
- /keys CRUD broadcasts with all-or-nothing semantics
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements POST/PUT /indexes/{uid}/documents and DELETE /indexes/{uid}/documents:
- Primary key extraction on hot path with 400 miroir_primary_key_required if missing
- _miroir_shard injection into every document before forwarding to nodes
- Rejection of _miroir_shard in client-submitted docs (400 miroir_reserved_field)
- Two-rule quorum: per-group floor(RF/2)+1 ACKs, success if ≥1 group meets quorum
- X-Miroir-Degraded header when any group misses quorum
- 503 miroir_no_quorum only when NO group meets quorum
- Per-batch grouping by target shard for efficient HTTP fan-out
- DELETE by IDs routes each ID independently to its shard
- DELETE by filter broadcasts to all nodes
Acceptance tests pass:
- Primary key validation before any writes
- Reserved field rejection
- Shard distribution uniformity (17-26 shards/node with 64 shards/3 nodes)
- Quorum calculation: floor(RF/2)+1
- Meilisearch-compatible error shape
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Remove unused ShardHitPage import from p23_search_read_path.rs
- All 10 acceptance tests pass:
- Unique-keyword search returns exactly 1 hit (RRF deduplication)
- Facet counts sum correctly across shards
- Paging with no dupes/gaps (5 pages of 10 = 50 unique results)
- Node down with RF=2: search still covers all shards
- Group down with fallback: uses other group, not degraded
- X-Miroir-Degraded header includes actual shard IDs
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
