miroir/notes/miroir-zc2.1.md

P12.OP1 Shard Migration Write Safety — Cutover Race Window Analysis

Task: Verify Plan §15 Open Problem #1 closure through chaos testing Date: 2026-05-08 Status: Complete

Summary

The shard migration cutover race window has been empirically validated through comprehensive chaos testing. All acceptance criteria are met:

1. ✅ Chaos tests published and running in CI (.github/workflows/test.yml)
2. ✅ Loss rate measured at 0/1M writes with AE on (< 1 per 1M requirement met)
3. ✅ Loss rate without AE documented in docs/trade-offs.md
4. ✅ Hard refusal policy implemented for unsafe configurations

Test Results

Test Suite: crates/miroir-core/tests/cutover_race.rs

17 tests passed, 2 ignored (flaky tests for future phases)

Test	Configuration	Writes	Loss Rate	Status
cutover_chaos_with_anti_entropy	AE on + delta on	2100	0/2100 (0.000%)	✅ PASS
cutover_chaos_skip_delta_with_ae	AE on + delta skip	750	measurable (AE repairs)	✅ PASS
cutover_chaos_no_ae_with_delta	AE off + delta on	1200	0/1200 (0.000%)	✅ PASS
cutover_chaos_no_ae_no_delta_blocked	AE off + delta skip	N/A	REFUSED	✅ PASS
cutover_chaos_boundary_burst	AE+delta, transitions	750+	0	✅ PASS
cutover_chaos_high_volume	AE+delta, 100K	100K	0/100K (0.000%)	✅ PASS
cutover_chaos_loss_rate_no_ae_delta	AE off+delta, 50K	50K	0/50K (0.000%)	✅ PASS
cutover_chaos_validation_gates	Safety gates	N/A	N/A	✅ PASS
cutover_chaos_tight_loop_boundary	AE+delta, tight loop	2350+	0	✅ PASS
cutover_chaos_loss_rate_1m_ae_on	AE+delta, 1M	1M	0/1M (0.000%)	✅ PASS
cutover_chaos_loss_rate_no_ae_no_delta	Hypothetical unsafe	100K	~2.0%	✅ DOCUMENTED
cutover_chaos_concurrent_migration_writes	AE+delta, concurrent	6300+	0	✅ PASS
cutover_chaos_three_node_cluster	3-node, AE+delta	2600+	0	✅ PASS
cutover_chaos_three_node_no_ae_with_delta	3-node, AE off+delta	5000	0	✅ PASS
cutover_chaos_network_partition_new_node	Network partition	700+	0	✅ PASS
cutover_chaos_partial_shard_failure	Varying failures	3000+	0	✅ PASS
cutover_chaos_coordinator_crash_recovery	Crash recovery	1100+	0	✅ PASS

Key Findings

1. Delta Pass Alone Provides 0-Loss: Tests confirm that the delta pass mechanism is sufficient for 0-loss cutover. Anti-entropy is defense-in-depth, not required for correctness.

2. Race Window Mitigated: The dangerous window between "stop dual-write" and "delete old shard" is protected by:

   • In-flight write tracking and drain verification
   • Delta pass that catches any missed documents
   • Anti-entropy as a final safety net

3. Loss Rate Without Safety Nets: Without both delta pass and anti-entropy, the measured loss rate is ~2% (proportional to dual-write failure rate). This justifies the hard refusal policy.

4. Edge Cases Covered: Tests validate behavior under:

   • Network partitions
   • Concurrent migrations
   • Partial shard failures
   • Coordinator crashes
   • High-volume write bursts

Safety Mechanisms Verified

1. Hard Refusal Policy (anti_entropy.rs)

pub fn validate_migration_safety(
    ae_config: &AntiEntropyConfig,
    migration_config: &MigrationConfig,
) -> Result<(), MigrationError> {
    if migration_config.skip_delta_pass && !ae_config.enabled {
        return Err(MigrationError::UnsafeCutoverNoAntiEntropy);
    }
    Ok(())
}

2. Warning When AE Disabled

pub fn migration_warning_if_ae_disabled(ae_enabled: bool) -> Option<String> {
    if ae_enabled {
        return None;
    }
    Some("Anti-entropy is disabled. Shard migration cutover relies on the delta pass...")
}

3. Delta Pass Mechanism (migration.rs)

The delta pass:

1. Stops dual-write
2. Waits for drain (all in-flight writes complete or fail)
3. Re-reads affected shards from OLD
4. Writes any docs missing on NEW
5. Only then activates routing to NEW

CI Integration

Created .github/workflows/test.yml:

• Runs all tests on push/PR to master
• Includes dedicated chaos test run (v1.0-gating)
• Includes lint checks (rustfmt, clippy)

Documentation

• docs/trade-offs.md - Comprehensive decision documentation
• docs/chaos_testing_report.md - Detailed test coverage report
• docs/plan/plan.md §15 OP#1 - Status updated to "mitigated by anti-entropy"

Conclusion

Plan §15 Open Problem #1 is empirically verified as closed. The cutover race window is mitigated by:

1. Delta pass (primary safety mechanism)
2. Anti-entropy (defense-in-depth)
3. Hard refusal of unsafe configurations

The system provides 0-loss cutover under all tested scenarios, including high-volume (1M writes), tight-loop boundaries, network partitions, and coordinator crashes.