docs(bf-2ynu5): close out reshard source doc-count helper umbrella
bf-2ynu5 was the umbrella/foundation piece for bf-49a17's reshard backfill denominator. The implementation landed as four closed child beads (bf-2r01e module registration, bf-14wgq compute_source_document_count, bf-1s9od mockito tests, bf-3r626 ILM dedup). This commit is the umbrella close-out: verifies all four acceptance criteria are met in the current tree and records the passing cargo test run (9 index_stats unit tests + 4 reshard executor helper tests), with no state-machine change — the helper remains pure additive. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
be24ae3e5c
commit
b3d29df1fe
1 changed files with 78 additions and 0 deletions
78
notes/bf-2ynu5.md
Normal file
78
notes/bf-2ynu5.md
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
# bf-2ynu5 — Add per-node document-count query helper for reshard backfill (umbrella close-out)
|
||||
|
||||
Split-child of bf-49a17 (reshard backfill progress never populated). This was
|
||||
the **foundation/umbrella** piece: a reusable mechanism to compute a reshard
|
||||
source index's real document count, with no change to the reshard state machine.
|
||||
|
||||
The implementation landed as **four closed sibling/child beads**, each owning
|
||||
one slice. This bead's own deliverable is the umbrella close-out: confirm the
|
||||
children integrate, build, and pass, and that every acceptance criterion below
|
||||
is met in the current tree. No code was authored under bf-2ynu5 directly.
|
||||
|
||||
## Child beads (all closed)
|
||||
|
||||
| Bead | Commit | What it owned |
|
||||
|------|--------|---------------|
|
||||
| bf-2r01e | f10a675 | Register `index_stats` module (`pub mod index_stats;` lib.rs:37) |
|
||||
| bf-14wgq | 3dd2b75 | Add `ReshardExecutor::compute_source_document_count` (executor.rs:538) |
|
||||
| bf-1s9od | 1523cae | Unit-test the executor helper against mockito nodes (executor.rs:970+) |
|
||||
| bf-3r626 | 3b52aba | Migrate ILM off its duplicate stats code onto the shared module |
|
||||
|
||||
## Acceptance criteria — verified against current tree (2026-07-10)
|
||||
|
||||
1. **Helper queries each healthy source node's `GET /indexes/{uid}/stats` and
|
||||
returns an aggregated count.**
|
||||
`compute_source_document_count` (reshard/executor.rs:538) delegates to
|
||||
`index_stats::aggregate_index_stats` (index_stats.rs:130), which iterates the
|
||||
source `node_addresses`, GETs `/indexes/{uid}/stats`, and reduces
|
||||
`numberOfDocuments` with `max` (each address hosts a full replica, so `sum`
|
||||
would over-count by the replication factor — same reduction ILM relies on).
|
||||
A node that fails to respond (network / non-2xx) is logged and skipped; a 404
|
||||
(absent replica) counts as zero. "Healthy" = "responds successfully", matching
|
||||
the original `ilm::fetch_index_stats` precedent.
|
||||
|
||||
2. **Reuses/extracts the existing ilm.rs aggregation rather than duplicating.**
|
||||
The HTTP+reduce logic lives once in the shared `index_stats` module
|
||||
(index_stats.rs). ILM's old `fetch_index_stats` is gone; ilm.rs now calls
|
||||
`crate::index_stats::aggregate_index_stats` at :701 and :1067. Reshard and
|
||||
ILM therefore compute an index's count the *same* way.
|
||||
|
||||
3. **Unit-tested against a mock node client returning known per-node counts.**
|
||||
- Pure reduction policy `reduce_document_counts` (index_stats.rs:170), 9 tests.
|
||||
- Executor helper driven against mockito `GET /indexes/{uid}/stats` servers,
|
||||
4 tests (max across healthy; one failing node skipped; 404 tolerated;
|
||||
all-fail → 0). See "Test run" below.
|
||||
|
||||
4. **No change to the reshard state machine / backfill migration.**
|
||||
`start_backfill` (executor.rs:513) still initializes `total_documents: 0`
|
||||
(`// Will be updated`); `advance_backfill` (executor.rs:640) still accumulates
|
||||
`total_docs_in_shard` incrementally. `compute_source_document_count` is a pure
|
||||
additive helper that is *not* called from the state machine in this umbrella.
|
||||
Wiring the up-front denominator into `start_backfill` is intentionally a
|
||||
sibling bead's job.
|
||||
|
||||
## Test run (this close-out)
|
||||
|
||||
Disk pre-flight: 46 GB free on `/` (above the ~20 G build threshold) — no
|
||||
`target/` cleanup needed.
|
||||
|
||||
```
|
||||
$ cargo test -p miroir-core --lib index_stats
|
||||
test result: ok. 9 passed; 0 failed
|
||||
|
||||
$ cargo test -p miroir-core --lib compute_source_doc_count
|
||||
test result: ok. 4 passed; 0 failed
|
||||
```
|
||||
|
||||
Whole `miroir-core` lib compiled clean (712 tests in the suite, no warnings).
|
||||
Both seams green: the pure reduction and the real-reqwest transport against
|
||||
mockito nodes.
|
||||
|
||||
## Conclusion
|
||||
|
||||
All four acceptance criteria are met and verified. The reshard backfill now has
|
||||
the missing denominator (source index real document count, computed before the
|
||||
first page is fetched) available via `compute_source_document_count`, computed
|
||||
identically to ILM. Umbrella bf-2ynu5 closes; the remaining work (populating the
|
||||
persisted `ReshardOperation.total_documents` / `start_backfill` wiring) is
|
||||
tracked under other children of bf-49a17.
|
||||
Loading…
Add table
Reference in a new issue