docs(bf-2ynu5): close out reshard source doc-count helper umbrella

bf-2ynu5 was the umbrella/foundation piece for bf-49a17's reshard
backfill denominator. The implementation landed as four closed child
beads (bf-2r01e module registration, bf-14wgq compute_source_document_count,
bf-1s9od mockito tests, bf-3r626 ILM dedup). This commit is the
umbrella close-out: verifies all four acceptance criteria are met in
the current tree and records the passing cargo test run (9 index_stats
unit tests + 4 reshard executor helper tests), with no state-machine
change — the helper remains pure additive.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
jedarden 2026-07-10 14:50:33 -04:00
parent be24ae3e5c
commit b3d29df1fe

78
notes/bf-2ynu5.md Normal file
View file

@ -0,0 +1,78 @@
# bf-2ynu5 — Add per-node document-count query helper for reshard backfill (umbrella close-out)
Split-child of bf-49a17 (reshard backfill progress never populated). This was
the **foundation/umbrella** piece: a reusable mechanism to compute a reshard
source index's real document count, with no change to the reshard state machine.
The implementation landed as **four closed sibling/child beads**, each owning
one slice. This bead's own deliverable is the umbrella close-out: confirm the
children integrate, build, and pass, and that every acceptance criterion below
is met in the current tree. No code was authored under bf-2ynu5 directly.
## Child beads (all closed)
| Bead | Commit | What it owned |
|------|--------|---------------|
| bf-2r01e | f10a675 | Register `index_stats` module (`pub mod index_stats;` lib.rs:37) |
| bf-14wgq | 3dd2b75 | Add `ReshardExecutor::compute_source_document_count` (executor.rs:538) |
| bf-1s9od | 1523cae | Unit-test the executor helper against mockito nodes (executor.rs:970+) |
| bf-3r626 | 3b52aba | Migrate ILM off its duplicate stats code onto the shared module |
## Acceptance criteria — verified against current tree (2026-07-10)
1. **Helper queries each healthy source node's `GET /indexes/{uid}/stats` and
returns an aggregated count.**
`compute_source_document_count` (reshard/executor.rs:538) delegates to
`index_stats::aggregate_index_stats` (index_stats.rs:130), which iterates the
source `node_addresses`, GETs `/indexes/{uid}/stats`, and reduces
`numberOfDocuments` with `max` (each address hosts a full replica, so `sum`
would over-count by the replication factor — same reduction ILM relies on).
A node that fails to respond (network / non-2xx) is logged and skipped; a 404
(absent replica) counts as zero. "Healthy" = "responds successfully", matching
the original `ilm::fetch_index_stats` precedent.
2. **Reuses/extracts the existing ilm.rs aggregation rather than duplicating.**
The HTTP+reduce logic lives once in the shared `index_stats` module
(index_stats.rs). ILM's old `fetch_index_stats` is gone; ilm.rs now calls
`crate::index_stats::aggregate_index_stats` at :701 and :1067. Reshard and
ILM therefore compute an index's count the *same* way.
3. **Unit-tested against a mock node client returning known per-node counts.**
- Pure reduction policy `reduce_document_counts` (index_stats.rs:170), 9 tests.
- Executor helper driven against mockito `GET /indexes/{uid}/stats` servers,
4 tests (max across healthy; one failing node skipped; 404 tolerated;
all-fail → 0). See "Test run" below.
4. **No change to the reshard state machine / backfill migration.**
`start_backfill` (executor.rs:513) still initializes `total_documents: 0`
(`// Will be updated`); `advance_backfill` (executor.rs:640) still accumulates
`total_docs_in_shard` incrementally. `compute_source_document_count` is a pure
additive helper that is *not* called from the state machine in this umbrella.
Wiring the up-front denominator into `start_backfill` is intentionally a
sibling bead's job.
## Test run (this close-out)
Disk pre-flight: 46 GB free on `/` (above the ~20 G build threshold) — no
`target/` cleanup needed.
```
$ cargo test -p miroir-core --lib index_stats
test result: ok. 9 passed; 0 failed
$ cargo test -p miroir-core --lib compute_source_doc_count
test result: ok. 4 passed; 0 failed
```
Whole `miroir-core` lib compiled clean (712 tests in the suite, no warnings).
Both seams green: the pure reduction and the real-reqwest transport against
mockito nodes.
## Conclusion
All four acceptance criteria are met and verified. The reshard backfill now has
the missing denominator (source index real document count, computed before the
first page is fetched) available via `compute_source_document_count`, computed
identically to ILM. Umbrella bf-2ynu5 closes; the remaining work (populating the
persisted `ReshardOperation.total_documents` / `start_backfill` wiring) is
tracked under other children of bf-49a17.