diff --git a/notes/bf-2ynu5.md b/notes/bf-2ynu5.md new file mode 100644 index 0000000..1b9ce82 --- /dev/null +++ b/notes/bf-2ynu5.md @@ -0,0 +1,78 @@ +# bf-2ynu5 — Add per-node document-count query helper for reshard backfill (umbrella close-out) + +Split-child of bf-49a17 (reshard backfill progress never populated). This was +the **foundation/umbrella** piece: a reusable mechanism to compute a reshard +source index's real document count, with no change to the reshard state machine. + +The implementation landed as **four closed sibling/child beads**, each owning +one slice. This bead's own deliverable is the umbrella close-out: confirm the +children integrate, build, and pass, and that every acceptance criterion below +is met in the current tree. No code was authored under bf-2ynu5 directly. + +## Child beads (all closed) + +| Bead | Commit | What it owned | +|------|--------|---------------| +| bf-2r01e | f10a675 | Register `index_stats` module (`pub mod index_stats;` lib.rs:37) | +| bf-14wgq | 3dd2b75 | Add `ReshardExecutor::compute_source_document_count` (executor.rs:538) | +| bf-1s9od | 1523cae | Unit-test the executor helper against mockito nodes (executor.rs:970+) | +| bf-3r626 | 3b52aba | Migrate ILM off its duplicate stats code onto the shared module | + +## Acceptance criteria — verified against current tree (2026-07-10) + +1. **Helper queries each healthy source node's `GET /indexes/{uid}/stats` and + returns an aggregated count.** + `compute_source_document_count` (reshard/executor.rs:538) delegates to + `index_stats::aggregate_index_stats` (index_stats.rs:130), which iterates the + source `node_addresses`, GETs `/indexes/{uid}/stats`, and reduces + `numberOfDocuments` with `max` (each address hosts a full replica, so `sum` + would over-count by the replication factor — same reduction ILM relies on). + A node that fails to respond (network / non-2xx) is logged and skipped; a 404 + (absent replica) counts as zero. "Healthy" = "responds successfully", matching + the original `ilm::fetch_index_stats` precedent. + +2. **Reuses/extracts the existing ilm.rs aggregation rather than duplicating.** + The HTTP+reduce logic lives once in the shared `index_stats` module + (index_stats.rs). ILM's old `fetch_index_stats` is gone; ilm.rs now calls + `crate::index_stats::aggregate_index_stats` at :701 and :1067. Reshard and + ILM therefore compute an index's count the *same* way. + +3. **Unit-tested against a mock node client returning known per-node counts.** + - Pure reduction policy `reduce_document_counts` (index_stats.rs:170), 9 tests. + - Executor helper driven against mockito `GET /indexes/{uid}/stats` servers, + 4 tests (max across healthy; one failing node skipped; 404 tolerated; + all-fail → 0). See "Test run" below. + +4. **No change to the reshard state machine / backfill migration.** + `start_backfill` (executor.rs:513) still initializes `total_documents: 0` + (`// Will be updated`); `advance_backfill` (executor.rs:640) still accumulates + `total_docs_in_shard` incrementally. `compute_source_document_count` is a pure + additive helper that is *not* called from the state machine in this umbrella. + Wiring the up-front denominator into `start_backfill` is intentionally a + sibling bead's job. + +## Test run (this close-out) + +Disk pre-flight: 46 GB free on `/` (above the ~20 G build threshold) — no +`target/` cleanup needed. + +``` +$ cargo test -p miroir-core --lib index_stats +test result: ok. 9 passed; 0 failed + +$ cargo test -p miroir-core --lib compute_source_doc_count +test result: ok. 4 passed; 0 failed +``` + +Whole `miroir-core` lib compiled clean (712 tests in the suite, no warnings). +Both seams green: the pure reduction and the real-reqwest transport against +mockito nodes. + +## Conclusion + +All four acceptance criteria are met and verified. The reshard backfill now has +the missing denominator (source index real document count, computed before the +first page is fetched) available via `compute_source_document_count`, computed +identically to ILM. Umbrella bf-2ynu5 closes; the remaining work (populating the +persisted `ReshardOperation.total_documents` / `start_backfill` wiring) is +tracked under other children of bf-49a17.