jedarden/miroir
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
miroir/notes/miroir-zc2.3.md
jedarden 4ec0444b64 miroir-zc2.3: Validate 2× transient load caveat for online resharding (P12.OP3) 
- Fixed duplicate ReshardingConfig: added allowed_windows to advanced.rs
- Ran benchmark confirming storage/dual-write amplification at exactly 2.0×
- Verified CLI window guard integration tests (4/4 passing)
- Updated benchmark doc with latest run date (2026-05-20)

Key findings:
- Storage amplification is exactly 2× across all scenarios
- Peak write amplification varies from 12× to 502× depending on throttle
- Operators should set throttle to keep peak writes ≤ 3× normal

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bead-Id: miroir-r3j.2
2026-05-20 07:24:22 -04:00

2.9 KiB
Raw Blame History

P12.OP3 Online resharding — validate 2× transient load caveat under real corpora

Summary

Bead miroir-zc2.3: Validated the 2× transient storage/write load caveat for online resharding (plan §15 OP#3).

What was done

1. Fixed duplicate ReshardingConfig definitions

The ReshardingConfig struct was defined in two places with different fields:

  • crates/miroir-core/src/reshard.rs - Had allowed_windows: Vec<String>
  • crates/miroir-core/src/config/advanced.rs - Missing allowed_windows

This meant operators couldn't configure schedule windows in their main YAML config. Fixed by adding allowed_windows to advanced::ReshardingConfig.

2. Ran benchmark to validate 2× load caveat

Executed cargo run --bin bench-reshard-load which simulates the full test matrix:

Doc size Corpus Write rate RG RF Storage Amp Peak Write Amp
1 KB 10 GB 100 dps 2 1 2.00× 102.00×
10 KB 100 GB 1000 dps 2 2 2.00× 12.00×
1 MB 1 TB 10 dps 2 1 2.00× 502.00×

All invariants PASSED:

  • Storage amplification == 2.0× (exact)
  • Dual-write amplification == 2.0× (exact)
  • Hash distribution CV < 5% (all < 1.04%)

3. Verified CLI window guard integration

All 4 integration tests in crates/miroir-ctl/tests/window_guard.rs pass:

  • rejected_outside_configured_window
  • force_overrides_window_guard
  • no_windows_allows_any_time
  • disabled_config_rejects_even_with_no_windows

4. Updated documentation

Updated docs/benchmarks/resharding-load.md with latest run date (2026-05-20).

Acceptance criteria status

  • Benchmark doc published with real numbers
  • CLI window guard implemented; integration test confirms rejection outside window
  • Benchmark run in Phase 9 performance suite as part of v1.0 validation

Note: There is no explicit "Phase 9 performance suite" infrastructure in the codebase beyond the standard Criterion benchmarks. The bench-reshard-load binary is a standalone benchmark that can be run manually as part of v1.0 validation.

Key findings

  1. Storage amplification is exactly 2× - The shadow index doubles storage during resharding, no more, no less.

  2. Peak write amplification varies wildly - Depends on backfill throttle relative to incoming write rate:

    • High-write corpora: 12× (backfill is small fraction of normal traffic)
    • Low-write corpora: 502× (backfill dominates)

  3. Operator guidance: Set throttle_docs_per_sec conservatively. Formula for peak write rate:

    peak_writes = (backfill_throttle_dps + write_rate) × 2 × RF × RG

    Aim for peak total writes ≤ 3× normal to avoid overwhelming the cluster.

Files changed

  • crates/miroir-core/src/config/advanced.rs - Added allowed_windows field
  • docs/benchmarks/resharding-load.md - Updated run date
  • notes/miroir-zc2.3.md - This file