P5.5 §13.5: Complete two-phase settings broadcast + drift reconciler
Implements the propose/verify/commit flow for settings changes with drift
detection and repair. Replaces sequential settings apply with a safer
two-phase broadcast that prevents partial settings apply.
Key components:
- SettingsBroadcast coordinator (miroir-core/src/settings.rs):
* Phase 1 (Propose): PATCH all nodes in parallel, collect task UIDs
* Phase 2 (Verify): GET settings, verify SHA256 fingerprints
* Phase 3 (Commit): Increment settings_version, persist to task store
* Retry loop with exponential backoff for hash mismatches
* Per-(index, node) version tracking for client-pinned freshness
- DriftReconciler background worker (rebalancer_worker/drift_reconciler.rs):
* Mode B leader election for singleton execution
* Periodic settings hash comparison across all nodes
* Auto-repair drifted nodes with consensus settings
* Catches out-of-band changes (operator SSH'd to a node)
- Config (config/advanced.rs):
* settings_broadcast.strategy: two_phase or sequential (legacy)
* settings_broadcast.verify_timeout_s: 60s default
* settings_broadcast.max_repair_retries: 3 default
* settings_drift_check.interval_s: 300s (5 min) default
* settings_drift_check.auto_repair: true default
- Integration (main.rs, admin_endpoints.rs, indexes.rs):
* Drift reconciler started as background task
* Two-phase broadcast in PATCH /indexes/{uid}/settings
* X-Miroir-Settings-Version response header
* Legacy sequential mode for rollback compatibility
- Router (router.rs):
* covering_set_with_version_floor() filters stale nodes
* 503 when no floor-satisfying covering set exists
Acceptance criteria:
- ✅ Normal flow: add synonym; propose+verify succeed; version increments once
- ✅ Mid-broadcast node failure: verify fails, reissue succeeds after backoff
- ✅ Out-of-band drift: direct PATCH detected and repaired within interval_s
- ✅ X-Miroir-Min-Settings-Version floor excludes stale nodes; 503 when no floor-satisfying set
- ✅ Legacy sequential strategy still works
Tests: 15 total (7 acceptance + 8 integration), all passing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cfc0001ada