From b3d29df1fee4b9409297d9eba5969acb2ca57710 Mon Sep 17 00:00:00 2001 From: jedarden Date: Fri, 10 Jul 2026 14:50:33 -0400 Subject: [PATCH] docs(bf-2ynu5): close out reshard source doc-count helper umbrella MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bf-2ynu5 was the umbrella/foundation piece for bf-49a17's reshard backfill denominator. The implementation landed as four closed child beads (bf-2r01e module registration, bf-14wgq compute_source_document_count, bf-1s9od mockito tests, bf-3r626 ILM dedup). This commit is the umbrella close-out: verifies all four acceptance criteria are met in the current tree and records the passing cargo test run (9 index_stats unit tests + 4 reshard executor helper tests), with no state-machine change — the helper remains pure additive. Co-Authored-By: Claude --- notes/bf-2ynu5.md | 78 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 notes/bf-2ynu5.md diff --git a/notes/bf-2ynu5.md b/notes/bf-2ynu5.md new file mode 100644 index 0000000..1b9ce82 --- /dev/null +++ b/notes/bf-2ynu5.md @@ -0,0 +1,78 @@ +# bf-2ynu5 — Add per-node document-count query helper for reshard backfill (umbrella close-out) + +Split-child of bf-49a17 (reshard backfill progress never populated). This was +the **foundation/umbrella** piece: a reusable mechanism to compute a reshard +source index's real document count, with no change to the reshard state machine. + +The implementation landed as **four closed sibling/child beads**, each owning +one slice. This bead's own deliverable is the umbrella close-out: confirm the +children integrate, build, and pass, and that every acceptance criterion below +is met in the current tree. No code was authored under bf-2ynu5 directly. + +## Child beads (all closed) + +| Bead | Commit | What it owned | +|------|--------|---------------| +| bf-2r01e | f10a675 | Register `index_stats` module (`pub mod index_stats;` lib.rs:37) | +| bf-14wgq | 3dd2b75 | Add `ReshardExecutor::compute_source_document_count` (executor.rs:538) | +| bf-1s9od | 1523cae | Unit-test the executor helper against mockito nodes (executor.rs:970+) | +| bf-3r626 | 3b52aba | Migrate ILM off its duplicate stats code onto the shared module | + +## Acceptance criteria — verified against current tree (2026-07-10) + +1. **Helper queries each healthy source node's `GET /indexes/{uid}/stats` and + returns an aggregated count.** + `compute_source_document_count` (reshard/executor.rs:538) delegates to + `index_stats::aggregate_index_stats` (index_stats.rs:130), which iterates the + source `node_addresses`, GETs `/indexes/{uid}/stats`, and reduces + `numberOfDocuments` with `max` (each address hosts a full replica, so `sum` + would over-count by the replication factor — same reduction ILM relies on). + A node that fails to respond (network / non-2xx) is logged and skipped; a 404 + (absent replica) counts as zero. "Healthy" = "responds successfully", matching + the original `ilm::fetch_index_stats` precedent. + +2. **Reuses/extracts the existing ilm.rs aggregation rather than duplicating.** + The HTTP+reduce logic lives once in the shared `index_stats` module + (index_stats.rs). ILM's old `fetch_index_stats` is gone; ilm.rs now calls + `crate::index_stats::aggregate_index_stats` at :701 and :1067. Reshard and + ILM therefore compute an index's count the *same* way. + +3. **Unit-tested against a mock node client returning known per-node counts.** + - Pure reduction policy `reduce_document_counts` (index_stats.rs:170), 9 tests. + - Executor helper driven against mockito `GET /indexes/{uid}/stats` servers, + 4 tests (max across healthy; one failing node skipped; 404 tolerated; + all-fail → 0). See "Test run" below. + +4. **No change to the reshard state machine / backfill migration.** + `start_backfill` (executor.rs:513) still initializes `total_documents: 0` + (`// Will be updated`); `advance_backfill` (executor.rs:640) still accumulates + `total_docs_in_shard` incrementally. `compute_source_document_count` is a pure + additive helper that is *not* called from the state machine in this umbrella. + Wiring the up-front denominator into `start_backfill` is intentionally a + sibling bead's job. + +## Test run (this close-out) + +Disk pre-flight: 46 GB free on `/` (above the ~20 G build threshold) — no +`target/` cleanup needed. + +``` +$ cargo test -p miroir-core --lib index_stats +test result: ok. 9 passed; 0 failed + +$ cargo test -p miroir-core --lib compute_source_doc_count +test result: ok. 4 passed; 0 failed +``` + +Whole `miroir-core` lib compiled clean (712 tests in the suite, no warnings). +Both seams green: the pure reduction and the real-reqwest transport against +mockito nodes. + +## Conclusion + +All four acceptance criteria are met and verified. The reshard backfill now has +the missing denominator (source index real document count, computed before the +first page is fetched) available via `compute_source_document_count`, computed +identically to ILM. Umbrella bf-2ynu5 closes; the remaining work (populating the +persisted `ReshardOperation.total_documents` / `start_backfill` wiring) is +tracked under other children of bf-49a17.