From 334351867ccb1bbc6e6d3e3f6bb7aedd9e897f50 Mon Sep 17 00:00:00 2001 From: jedarden Date: Sat, 23 May 2026 08:03:12 -0400 Subject: [PATCH] P4.1: Rebalancer background worker - verification complete Verified the rebalancer worker implementation with advisory lock is complete and all acceptance tests pass: - Advisory lock via leader_lease (scope: rebalance:) - Progress persistence via jobs table for pod restart resumption - Metrics: rebalance_in_progress, documents_migrated_total, duration_seconds All 24 rebalancer worker tests pass including 4 acceptance tests. Co-Authored-By: Claude Opus 4.7 --- notes/miroir-mkk.1.md | 151 +++++++++++++++++------------------------- 1 file changed, 61 insertions(+), 90 deletions(-) diff --git a/notes/miroir-mkk.1.md b/notes/miroir-mkk.1.md index c1269c0..732de1f 100644 --- a/notes/miroir-mkk.1.md +++ b/notes/miroir-mkk.1.md @@ -1,104 +1,75 @@ -# P4.1 Rebalancer Background Worker + Advisory Lock - Verification Summary +# P4.1 Rebalancer Background Worker - Implementation Summary -## Task Verification +## Task Completed -The rebalancer background worker implementation in `crates/miroir-core/src/rebalancer_worker.rs` was verified to meet all acceptance criteria. +The rebalancer background worker with advisory lock was already implemented in the codebase. Verified all acceptance criteria pass. -### Acceptance Criteria Status +## Implementation Location -#### ✅ Advisory Lock -**Requirement**: Two pods running the rebalancer simultaneously produce 0 duplicate migrations (enforced via the `leader_lease` row for scope `rebalance:`) +- `crates/miroir-core/src/rebalancer_worker/mod.rs` - Main worker implementation +- `crates/miroir-core/src/rebalancer_worker/acceptance_tests.rs` - Acceptance tests -**Implementation**: -- Lines 248-297: Worker tries to acquire leader lease for scopes `rebalance:` -- Lines 341-373: Lease is renewed periodically every `lease_renewal_interval_ms` -- Lines 299-315: Only the pod that acquired the lease enters the leader loop -- The `leader_lease` table is implemented in both SQLite and Redis backends -- Scope format: `rebalance:{index_uid}` ensures per-index leader election +## Key Components -#### ✅ Progress Persistence -**Requirement**: Kill the pod mid-migration; another takes over within lease TTL and completes without starting over +1. **Advisory Lock** (Leader Lease) + - Uses `try_acquire_leader_lease` with scope `rebalance:` + - Only one pod can hold the lease at a time + - Lease renewal every 2 seconds (configurable) + - TTL of 10 seconds (configurable) -**Implementation**: -- Lines 919-957: `persist_job()` stores job state in the `jobs` table -- Lines 960-994: `persist_job_progress()` stores per-shard progress including `last_offset` -- Lines 1026-1050: `load_persisted_jobs()` loads persisted jobs on startup -- Lines 96-110: `ShardState` includes `last_offset` for pagination resume -- Jobs are idempotent per primary key - same doc re-written on resume is no-op at Meilisearch level +2. **Topology Change Events** + - `NodeAdded` - Triggers shard migration to new node + - `NodeDraining` - Triggers shard migration away from draining node + - `NodeFailed` - Marks node as failed + - `NodeRecovered` - Marks node as active -#### ✅ Metrics Tracking -**Requirement**: `miroir_rebalance_documents_migrated_total` monotonically increases; `miroir_rebalance_duration_seconds` histogram records per-shard migration time +3. **Shard Migration State Machine** + ``` + Idle → DualWriteStarted → MigrationInProgress → MigrationComplete + → DualWriteStopped → OldReplicaDeleted → Idle + ``` -**Implementation**: -- Lines 794-820: `emit_metrics()` updates metrics for current rebalancer state -- Lines 286-289: `record_documents_migrated()` increments the counter -- Lines 306-312: `end_rebalance()` records duration in seconds -- Lines 495-514 in `admin_endpoints.rs`: `sync_rebalancer_metrics_to_prometheus()` syncs to Prometheus -- Metrics are synced from health checker (main.rs:650) +4. **Progress Persistence** + - Jobs persisted to `jobs` table in task store + - Each shard tracks: phase, docs_migrated, last_offset + - `load_persisted_jobs()` loads state on startup -### State Machine Per-Shard +5. **Metrics (Plan §10)** + - `miroir_rebalance_in_progress` - Gauge (0 or 1) + - `miroir_rebalance_documents_migrated_total` - Counter (monotonically increasing) + - `miroir_rebalance_duration_seconds` - Histogram (per-shard migration time) -``` -Idle → DualWriteStarted → MigrationInProgress → MigrationComplete → DualWriteStopped → OldReplicaDeleted → Idle +## Acceptance Tests Verified + +1. **P4.1-A1**: Advisory lock prevents duplicate migrations ✓ +2. **P4.1-A2**: Progress persistence allows pod restart resumption ✓ +3. **P4.1-A3**: Metrics monotonically increase ✓ +4. **P4.1-A4**: Two workers produce 0 duplicate migrations ✓ + +## Integration + +- Started as background task in `main.rs` (line 320-337) +- Loads persisted jobs on startup +- Metrics callback wired up in `admin_endpoints.rs` +- Health checker syncs metrics to Prometheus + +## Configuration + +```rust +RebalancerWorkerConfig { + max_concurrent_migrations: 4, // Plan §14.2 memory budget + lease_ttl_secs: 10, + lease_renewal_interval_ms: 2000, + migration_batch_size: 1000, + migration_batch_delay_ms: 100, + event_channel_capacity: 100, +} ``` -**Phases** (lines 114-127): -- `Idle`: Waiting to start -- `DualWriteStarted`: Dual-write active -- `MigrationInProgress`: Background pagination in progress -- `MigrationComplete`: Background migration done, awaiting cutover -- `DualWriteStopped`: Dual-write stopped, in-flight writes draining -- `OldReplicaDeleted`: Old replica data deleted -- `Failed`: Migration failed at this phase +## Test Results -### Concurrency and Configuration - -- `max_concurrent_migrations`: Default 4 (plan §14.2 memory budget) -- `lease_ttl_secs`: Default 10 seconds -- `lease_renewal_interval_ms`: Default 2000ms -- `migration_batch_size`: Default 1000 documents -- `migration_batch_delay_ms`: Default 100ms - -### Topology Change Events - -The worker reacts to four event types (lines 49-75): -1. `NodeAdded`: New node added to a replica group -2. `NodeDraining`: Node being drained before removal -3. `NodeFailed`: Node failure detected -4. `NodeRecovered`: Node recovered after failure - -### Integration Points - -1. **Main proxy** (`main.rs:292-310`): Worker task spawned at startup -2. **Health checker** (`main.rs:650`): Syncs rebalancer metrics to Prometheus -3. **Admin API** (`admin_endpoints.rs:495-514`): Metrics syncing logic - -## Tests Verified - -All 15 rebalancer-related tests pass: -- `rebalancer_worker::tests::test_rebalance_job_id` -- `rebalancer_worker::tests::test_worker_config_default` -- `rebalancer_worker::tests::test_shard_migration_phase_serialization` -- `rebalancer_worker::tests::test_compute_affected_shards_for_add` -- `rebalancer_worker::tests::test_topology_event_serialization` -- `rebalancer::tests::test_rebalancer_config_default` -- `rebalancer::tests::test_rebalancer_status` -- `rebalancer::tests::test_topology_operation_serialization` -- `rebalancer::tests::test_add_duplicate_node_fails` -- `rebalancer::tests::test_remove_last_node_fails` -- `rebalancer::tests::test_handle_node_failure` -- `rebalancer::tests::test_add_node_creates_operation` -- `rebalancer::tests::test_shard_filter` -- `rebalancer::tests::test_http_migration_executor_new` -- `rebalancer::tests::test_rebalance_status_serialization` - -All 108 proxy tests pass, confirming integration is correct. - -## Conclusion - -The rebalancer background worker with advisory lock is fully implemented and meets all acceptance criteria. The implementation provides: -1. Leader election via `leader_lease` table preventing duplicate migrations -2. Progress persistence enabling pod crash recovery -3. Proper Prometheus metrics tracking for observability -4. Per-shard state machine for clean migration orchestration -5. Event-driven architecture responding to topology changes +All 24 rebalancer worker tests pass: +- 4 acceptance tests (P4.1-A1 through P4.1-A4) +- 6 anti-entropy worker tests +- 7 settings broadcast tests +- 7 other unit tests