P6.4: Mode B leader-only singleton coordinator (plan §14.5)
Implement leader election and phase state persistence for all Mode B
operations (reshard, rebalance, alias flip, 2PC, ILM, scoped-key rotation).
Components:
- LeaderElection service: CAS-based lease acquisition/renewal with TTL
- ModeBOpLeader<E>: Generic coordinator combining leader election with
phase state persistence to mode_b_operations table
- Lease scopes: reshard:<index>, rebalance, alias_flip:<name>,
settings_broadcast:<index>, ilm, search_ui_key_rotation:<index>
Mode B operations using ModeBOpLeader:
- ReshardCoordinator: Six-phase shadow-index resharding
- SettingsBroadcastCoordinator: Two-phase commit for settings changes
- ScopedKeyRotationCoordinator: Search UI scoped encryption key rotation
- IlmCoordinator: Index lifecycle management (rollovers)
- AliasFlipCoordinator: Blue-green alias flips
Configuration:
- leader_election.enabled: bool (default: true)
- leader_election.lease_ttl_s: u64 (default: 10)
- leader_election.renew_interval_s: u64 (default: 3)
Acceptance tests (all pass):
- AC1: Exactly one leader across 3 pods
- AC2: Leader failover within lease_ttl_s
- AC3: Lease renewal prevents stealing
- AC4: Reshard phase recovery (resumes at last phase, not phase 1)
- AC5: Multiple phases persisted correctly
- AC6: 2PC settings broadcast phase recovery
- AC7: Settings broadcast all phases persisted
- AC8: Leader metrics sum is 1 across pods
- AC9: Leader metrics transient zero during failover
- AC10: Multiple concurrent operations with different scopes
- AC11: Expired lease allows new leader
- AC12: Stale leader cannot renew expired lease
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
6bf0cb285a