jedarden/miroir
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

P2.1: Verify axum server skeleton implementation - all endpoints present

Browse source 
Verified all acceptance criteria for miroir-9dj.1:
- Config loading (file + env + CLI): MiroirConfig::load()
- Structured JSON logging: tracing_subscriber with JSON layer
- Two listeners: :7700 (main API) + :9090 (metrics)
- Signal handlers: shutdown_signal() with graceful drain
- GET /health: Returns {"status":"available"} immediately
- GET /version: Cached Meilisearch version (60s TTL)
- GET /_miroir/ready: 503 until covering quorum exists
- GET /_miroir/topology: Plan §10 JSON shape
- GET /_miroir/shards: Shard → node mapping
- GET /_miroir/metrics: Admin-key-gated Prometheus metrics

All 135 unit tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jedarden 2026-05-23 12:53:27 -04:00
parent ea3f3c2490
commit 4f2ff49270
4 changed files with 3939 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
.beads/issues.jsonl
View file

@ -29,14 +29,14 @@
{"id":"miroir-89x.5","title":"P9.5 Performance benches (criterion) + regression gate","description":"## What\n\nPlan §8 \"Performance benchmarks\" at `benches/` using criterion:\n\n| Benchmark | Target |\n|-----------|--------|\n| Rendezvous (64 shards, 3 nodes, 10K docs) | < 1 ms total |\n| Merger (1000 hits, 3 shards) | < 1 ms |\n| End-to-end search latency vs. single-node | < 2× single-node |\n| Ingest throughput (1000 docs through Miroir) | > 80% single-node |\n\nPlus a CI bot that comments on any PR increasing measured search latency by > 20% over the previous release.\n\n## Why\n\nPlan §8: \"A PR that increases measured search latency by > 20% over the previous release triggers a review comment.\" Without a regression gate, performance drifts. With it, drift is noticed at the PR level.\n\n## Details\n\n**criterion output artifact**: `target/criterion/` HTML reports; CI uploads as artifact.\n\n**Delta computation**: compare current PR's bench output vs. the most recent `main` run's stored bench output. `critcmp` is the typical tool.\n\n**Gating vs. commenting**: plan §8 says \"review comment,\" not \"block merge.\" Keep the tool advisory — operators trigger reruns for transient noise.\n\n**End-to-end search latency bench** needs a running docker-compose stack; run as part of integration benches, not unit benches.\n\n## Acceptance\n\n- [ ] `cargo bench -p miroir-core` runs in CI and records timings\n- [ ] Rendezvous bench passes `< 1 ms` target on iad-ci hardware\n- [ ] Merger bench passes `< 1 ms` target\n- [ ] End-to-end `< 2×` and ingest `> 80%` verified on a 3-node docker-compose\n- [ ] PR with intentional 30% slowdown triggers the comment bot","design":"","acceptance_criteria":"","notes":"","status":"open","priority":1,"issue_type":"task","created_at":"2026-04-18T21:45:18.407337766Z","created_by":"coding","updated_at":"2026-04-18T21:45:22.172471772Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-9"],"dependencies":[{"issue_id":"miroir-89x.5","depends_on_id":"miroir-89x.2","type":"blocks","created_at":"2026-04-18T21:45:22.172432130Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-89x.6","title":"P9.6 Property tests + fuzz for router + config + parser","description":"## What\n\nAdd proptest + cargo-fuzz coverage for the critical invariants:\n\n**Router** (`proptest`, in addition to P1.6):\n- Given random `(N, RG, RF, S)` and random doc IDs, `write_targets` + `covering_set` satisfy:\n - `|write_targets| == RG × RF` (counting duplicates)\n - Every group has exactly `RF` entries\n - `covering_set` unions to cover every shard in the chosen group\n - Reshuffle on topology change ≤ theoretical optimum\n\n**Config parser**: fuzz `Config::from_yaml` — every valid YAML in the plan parses; adversarial inputs don't crash.\n\n**Filter DSL parser** (§13.4): fuzz the filter grammar — every Meilisearch valid filter parses; malformed filters return `Err`, not panic.\n\n**Canonical-JSON** (for settings hashing §13.5): two equivalent JSONs must hash identically.\n\n## Why\n\nPlan §8 lists property tests in the \"Router correctness\" section. Adding fuzz to parsers closes the class-of-errors where a single crafted input OOMs or panics the orchestrator.\n\n## Details\n\n**Proptest configs**: 1024 cases per property by default; 8192 in the nightly CI run.\n\n**cargo-fuzz targets** (in `fuzz/fuzz_targets/`):\n- `config_parser.rs` — feeds random UTF-8 to `Config::from_yaml_str`\n- `filter_parser.rs` — feeds random strings to the §13.4 filter grammar\n- `canonical_json.rs` — roundtrips random JSON through the canonicalizer\n\n**Corpus seeding**: include every plan-referenced valid config, filter, and settings block as seeds so fuzz discovers edge cases rather than rediscovering syntax.\n\n## Acceptance\n\n- [ ] `cargo test` runs all property tests at 1024 cases; no rejects\n- [ ] `cargo +nightly fuzz run config_parser -- -max_total_time=60` finds no panics in 60s\n- [ ] Weekly CI fuzz run (scheduled via Argo Workflow) uploads artifacts showing 0 new crashes","design":"","acceptance_criteria":"","notes":"","status":"open","priority":1,"issue_type":"task","created_at":"2026-04-18T21:45:18.438638293Z","created_by":"coding","updated_at":"2026-04-18T21:45:18.438638293Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-9"]}
{"id":"miroir-9dj","title":"Phase 2 — Proxy + API Surface (HTTP routes, quorum, errors)","description":"## Phase 2 Epic — Proxy + API Surface\n\nWires the Phase 1 primitives into a live HTTP proxy. After this phase, a client pointing a Meilisearch SDK at `http://miroir:7700` can CRUD indexes, write documents, search, and poll tasks — with documents actually sharded across nodes.\n\n## Why This Sits Here\n\nPlan §1 principle 1 (**invisible federation**) and plan §5 (**API Surface and Compatibility**) are the product. Phase 1 gave us math; this phase turns the math into behavior a Meilisearch client sees as drop-in. Every downstream phase assumes these HTTP surfaces exist and return shapes that match the Meilisearch spec exactly, so §8 \"API compatibility tests\" can pin the contract from here on.\n\n## Scope (plan §3 Lifecycle + §5 API Surface)\n\n- `axum` server listening on `server.port` (default 7700) and metrics on 9090\n- **Write path** (plan §2 write path) — hash primary key, inject `_miroir_shard`, fan out to `RG × RF` nodes, per-group quorum (`floor(RF/2)+1`), `X-Miroir-Degraded` on any group missing quorum, 503 `miroir_no_quorum` only when no group met quorum for a shard\n- **Read path** (plan §2 read path) — pick group via `query_seq % RG`, build intra-group covering set, scatter, merge by `_rankingScore`, strip `_miroir_shard` always + `_rankingScore` if client didn't request, aggregate facets + estimatedTotalHits, report max processingTimeMs, group-fallback when a covering set has holes\n- **Index lifecycle** (plan §3) — create broadcasts + atomically injects `_miroir_shard` into `filterableAttributes`; settings sequential apply-with-rollback (§3 legacy; §13.5 replaces in Phase 5); delete broadcasts; stats aggregate `numberOfDocuments` + merge `fieldDistribution`\n- **Tasks** — per plan §3 task ID reconciliation; `GET /tasks`, `GET /tasks/{uid}`, `DELETE /tasks/{uid}`\n- **Error shape** — every error matches Meilisearch `{message,code,type,link}`; new `miroir_*` codes per plan §5\n- **Reserved fields contract** — `_miroir_shard` always-reserved; `_miroir_updated_at` / `_miroir_expires_at` reserved only when their feature flag is on (Phase 5)\n- **Auth** — master-key/admin-key bearer dispatch per §5 \"Bearer token dispatch\" rules 2–5; JWT path stubbed (Phase 5)\n- **/health + /version + /_miroir/ready + /_miroir/topology + /_miroir/shards** + **/_miroir/metrics** (admin-key gated mirror of port 9090 /metrics per plan §10)\n- **Middleware** — structured JSON log per plan §10; Prometheus metrics (`miroir_request_duration_seconds`, etc.)\n- **Scatter-gather dispatcher** — per-node retries with orchestrator-side retry cache keyed by `sha256(batch || target_node || idempotency_or_mtask)` (plan §4 note on `scatter.retry_on_timeout`)\n\n## Out of Scope (moved to later phases)\n\n- Two-phase settings broadcast (→ Phase 5 / §13.5)\n- Persistent task store (→ Phase 3)\n- Rebalancer (→ Phase 4)\n- Any §13 feature (→ Phase 5)\n- Multi-replica coordination / Redis / HPA (→ Phase 6)\n\n## Definition of Done\n\n- [ ] Integration test: 1000 documents indexed across 3 nodes, each retrievable by ID (plan §8)\n- [ ] Integration test: unique-keyword search finds every doc exactly once (plan §8)\n- [ ] Integration test: facet aggregation across 3 color values sums correctly (plan §8)\n- [ ] Integration test: offset/limit paging preserves global ordering (plan §8)\n- [ ] Integration test: write with one group completely down still succeeds on remaining group and stamps `X-Miroir-Degraded`\n- [ ] Error-format parity test: every `invalid_request`/`not_found`/`document_*` code matches Meilisearch output byte-for-byte on equivalent input\n- [ ] `GET /_miroir/topology` matches the shape in plan §10","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"epic","assignee":"","created_at":"2026-04-18T21:18:33.148045077Z","created_by":"coding","updated_at":"2026-05-09T19:47:20.348179739Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase","phase-2"],"dependencies":[{"issue_id":"miroir-9dj","depends_on_id":"miroir-cdo","type":"blocks","created_at":"2026-04-18T21:23:08.570130243Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-9dj.1","title":"P2.1 axum server skeleton + config loader + /health + /version + /_miroir/ready","description":"## What\n\nFlesh out `miroir-proxy::main`:\n- Load `Config` (file + env + CLI args overlay)\n- Initialize tracing (JSON-to-stdout per plan §10 log format)\n- Start two axum listeners: `:7700` (client API) + `:9090` (metrics, unauthenticated, pod-internal)\n- Signal handlers for graceful shutdown (SIGTERM → stop accepting new requests → drain in-flight → exit)\n- Implement: `GET /health`, `GET /version`, `GET /_miroir/ready`, `GET /_miroir/topology`, `GET /_miroir/shards`, `GET /_miroir/metrics`\n\n## Why\n\nThese are the minimum-viable endpoints Kubernetes needs to probe and operators need to inspect. `GET /health` is Meilisearch-compatible — the K8s liveness probe — and must return 200 immediately regardless of internal state (Meilisearch semantics). `GET /_miroir/ready` is the readiness probe and *blocks* 503 until a covering quorum is reachable on first startup (plan §10).\n\n## Details\n\n**`/health`** (plan §10) — returns `{\"status\":\"available\"}`. Never gate on internal state.\n\n**`/version`** — per plan §5 \"Orchestrator-local\": return the Meilisearch version from any healthy node. Cache at ~60s TTL.\n\n**`/_miroir/ready`** — 503 during startup; 200 once Miroir has loaded config + verified a covering quorum of nodes is reachable. This is specifically where the \"there's at least one full covering set somewhere in the topology\" check lives.\n\n**`/_miroir/topology`** — shape exactly per plan §10 JSON sample: `shards`, `replication_factor`, `nodes[]` with `id/status/shard_count/last_seen_ms[/error]`, `degraded_node_count`, `rebalance_in_progress`, `fully_covered`.\n\n**`/_miroir/shards`** — shard → node mapping table for the current topology (useful for runbooks and for §13.20 explain).\n\n**`/_miroir/metrics`** — admin-key-gated mirror of port 9090 `/metrics`. Same data; admin-authenticated so it can be exposed outside the cluster.\n\n## Acceptance\n\n- [ ] `curl localhost:7700/health` returns 200 within 100ms of process start\n- [ ] `curl localhost:7700/_miroir/ready` returns 503 until all configured nodes are reachable, then 200\n- [ ] `curl -H \"Authorization: Bearer $ADMIN_KEY\" localhost:7700/_miroir/topology | jq .` matches the plan §10 shape\n- [ ] SIGTERM drains in-flight requests (test by sending signal during a long-running search)","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"task","created_at":"2026-04-18T21:28:30.051416112Z","created_by":"coding","updated_at":"2026-04-18T21:28:35.581876770Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"],"dependencies":[{"issue_id":"miroir-9dj.1","depends_on_id":"miroir-9dj.8","type":"blocks","created_at":"2026-04-18T21:28:35.581837637Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-9dj.2","title":"P2.2 Document write path: primary key → hash → shard → fan-out → quorum","description":"## What\n\nImplement:\n- `POST /indexes/{uid}/documents`\n- `PUT /indexes/{uid}/documents`\n- `DELETE /indexes/{uid}/documents/{id}`\n- `DELETE /indexes/{uid}/documents` (by IDs array or filter)\n\n## Why\n\nPlan §2 \"Write path\" is the heart of the product. Four properties that MUST be right:\n\n1. **Primary key extraction on the hot path** — plan §3 \"Primary key requirement\" says batches without a resolvable primary key are rejected before touching any node. This is a cheap, up-front check and a big UX win.\n2. **`_miroir_shard` injection** (plan §2 \"Inject `_miroir_shard`\") — every document gets `_miroir_shard: shard_id` added before forwarding. Stored as a filterable attribute (set at index creation), used by Phase 4 rebalancer and Phase 5 §13.8 anti-entropy for targeted shard retrieval. Stripped from all API responses.\n3. **Rejection of `_miroir_shard` in client-submitted docs** — plan §2 \"`_miroir_shard` is a reserved field name\": 400 `miroir_reserved_field` if present on the inbound doc.\n4. **Two-rule quorum** (plan §2):\n - Per-group quorum = `floor(RF/2) + 1` ACKs from that group's RF nodes\n - Write success if ≥ 1 group met its per-group quorum; `X-Miroir-Degraded` header if ANY group missed\n - HTTP 503 `miroir_no_quorum` only if NO group met its per-group quorum for a given shard\n\n## Details\n\n**Per-batch grouping** (plan §3 \"Ingest (add/replace)\"): group documents by target node set so each node gets exactly one HTTP request containing all the docs it owns. This minimizes HTTP fan-out count (critical at scale).\n\n**Retry-on-timeout** (plan §4 \"Note on `scatter.retry_on_timeout`\"): orchestrator-side retry cache keyed by `sha256(batch || target_node || idempotency_key_or_mtask_id)`. When a timeout retries, check the cache first; if the prior dispatch has a cached terminal response, return it rather than creating a duplicate node-side task.\n\n**Delete-by-filter** (plan §5 \"Broadcast to all nodes\"): cannot be shard-routed; broadcast to every node.\n\n**Delete-by-IDs array**: route each ID to its shard independently (same routing as the write path).\n\n## Acceptance (plan §8)\n\n- [ ] 1000 docs indexed via POST — every doc fetch-by-id returns the same doc\n- [ ] Docs distribute across all configured nodes (no node holds < 20% under RF=1/3-node)\n- [ ] Batch with one missing primary key → 400 `miroir_primary_key_required`, no docs written anywhere\n- [ ] Doc containing `_miroir_shard` → 400 `miroir_reserved_field`\n- [ ] RG=2, RF=1, 1 group down: write to 1 group succeeds with `X-Miroir-Degraded: groups=1`\n- [ ] RG=2, RF=1, both groups down: 503 `miroir_no_quorum`\n- [ ] DELETE by IDs array [docA, docB] with docA on shard 3, docB on shard 7 produces 2 independent per-shard delete calls","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"task","created_at":"2026-04-18T21:28:30.071116940Z","created_by":"coding","updated_at":"2026-04-18T21:28:35.549186215Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"],"dependencies":[{"issue_id":"miroir-9dj.2","depends_on_id":"miroir-9dj.1","type":"blocks","created_at":"2026-04-18T21:28:35.455097028Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-9dj.2","depends_on_id":"miroir-9dj.6","type":"blocks","created_at":"2026-04-18T21:28:35.534066064Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-9dj.2","depends_on_id":"miroir-9dj.7","type":"blocks","created_at":"2026-04-18T21:28:35.549164039Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-9dj.1","title":"P2.1 axum server skeleton + config loader + /health + /version + /_miroir/ready","description":"## What\n\nFlesh out `miroir-proxy::main`:\n- Load `Config` (file + env + CLI args overlay)\n- Initialize tracing (JSON-to-stdout per plan §10 log format)\n- Start two axum listeners: `:7700` (client API) + `:9090` (metrics, unauthenticated, pod-internal)\n- Signal handlers for graceful shutdown (SIGTERM → stop accepting new requests → drain in-flight → exit)\n- Implement: `GET /health`, `GET /version`, `GET /_miroir/ready`, `GET /_miroir/topology`, `GET /_miroir/shards`, `GET /_miroir/metrics`\n\n## Why\n\nThese are the minimum-viable endpoints Kubernetes needs to probe and operators need to inspect. `GET /health` is Meilisearch-compatible — the K8s liveness probe — and must return 200 immediately regardless of internal state (Meilisearch semantics). `GET /_miroir/ready` is the readiness probe and *blocks* 503 until a covering quorum is reachable on first startup (plan §10).\n\n## Details\n\n**`/health`** (plan §10) — returns `{\"status\":\"available\"}`. Never gate on internal state.\n\n**`/version`** — per plan §5 \"Orchestrator-local\": return the Meilisearch version from any healthy node. Cache at ~60s TTL.\n\n**`/_miroir/ready`** — 503 during startup; 200 once Miroir has loaded config + verified a covering quorum of nodes is reachable. This is specifically where the \"there's at least one full covering set somewhere in the topology\" check lives.\n\n**`/_miroir/topology`** — shape exactly per plan §10 JSON sample: `shards`, `replication_factor`, `nodes[]` with `id/status/shard_count/last_seen_ms[/error]`, `degraded_node_count`, `rebalance_in_progress`, `fully_covered`.\n\n**`/_miroir/shards`** — shard → node mapping table for the current topology (useful for runbooks and for §13.20 explain).\n\n**`/_miroir/metrics`** — admin-key-gated mirror of port 9090 `/metrics`. Same data; admin-authenticated so it can be exposed outside the cluster.\n\n## Acceptance\n\n- [ ] `curl localhost:7700/health` returns 200 within 100ms of process start\n- [ ] `curl localhost:7700/_miroir/ready` returns 503 until all configured nodes are reachable, then 200\n- [ ] `curl -H \"Authorization: Bearer $ADMIN_KEY\" localhost:7700/_miroir/topology | jq .` matches the plan §10 shape\n- [ ] SIGTERM drains in-flight requests (test by sending signal during a long-running search)","design":"","acceptance_criteria":"","notes":"","status":"in_progress","priority":0,"issue_type":"task","assignee":"claude-code-glm-4.7-bravo","created_at":"2026-04-18T21:28:30.051416112Z","created_by":"coding","updated_at":"2026-05-23T16:51:35.100927487Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"],"dependencies":[{"issue_id":"miroir-9dj.1","depends_on_id":"miroir-9dj.8","type":"blocks","created_at":"2026-04-18T21:28:35.581837637Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-9dj.2","title":"P2.2 Document write path: primary key → hash → shard → fan-out → quorum","description":"## What\n\nImplement:\n- `POST /indexes/{uid}/documents`\n- `PUT /indexes/{uid}/documents`\n- `DELETE /indexes/{uid}/documents/{id}`\n- `DELETE /indexes/{uid}/documents` (by IDs array or filter)\n\n## Why\n\nPlan §2 \"Write path\" is the heart of the product. Four properties that MUST be right:\n\n1. **Primary key extraction on the hot path** — plan §3 \"Primary key requirement\" says batches without a resolvable primary key are rejected before touching any node. This is a cheap, up-front check and a big UX win.\n2. **`_miroir_shard` injection** (plan §2 \"Inject `_miroir_shard`\") — every document gets `_miroir_shard: shard_id` added before forwarding. Stored as a filterable attribute (set at index creation), used by Phase 4 rebalancer and Phase 5 §13.8 anti-entropy for targeted shard retrieval. Stripped from all API responses.\n3. **Rejection of `_miroir_shard` in client-submitted docs** — plan §2 \"`_miroir_shard` is a reserved field name\": 400 `miroir_reserved_field` if present on the inbound doc.\n4. **Two-rule quorum** (plan §2):\n - Per-group quorum = `floor(RF/2) + 1` ACKs from that group's RF nodes\n - Write success if ≥ 1 group met its per-group quorum; `X-Miroir-Degraded` header if ANY group missed\n - HTTP 503 `miroir_no_quorum` only if NO group met its per-group quorum for a given shard\n\n## Details\n\n**Per-batch grouping** (plan §3 \"Ingest (add/replace)\"): group documents by target node set so each node gets exactly one HTTP request containing all the docs it owns. This minimizes HTTP fan-out count (critical at scale).\n\n**Retry-on-timeout** (plan §4 \"Note on `scatter.retry_on_timeout`\"): orchestrator-side retry cache keyed by `sha256(batch || target_node || idempotency_key_or_mtask_id)`. When a timeout retries, check the cache first; if the prior dispatch has a cached terminal response, return it rather than creating a duplicate node-side task.\n\n**Delete-by-filter** (plan §5 \"Broadcast to all nodes\"): cannot be shard-routed; broadcast to every node.\n\n**Delete-by-IDs array**: route each ID to its shard independently (same routing as the write path).\n\n## Acceptance (plan §8)\n\n- [ ] 1000 docs indexed via POST — every doc fetch-by-id returns the same doc\n- [ ] Docs distribute across all configured nodes (no node holds < 20% under RF=1/3-node)\n- [ ] Batch with one missing primary key → 400 `miroir_primary_key_required`, no docs written anywhere\n- [ ] Doc containing `_miroir_shard` → 400 `miroir_reserved_field`\n- [ ] RG=2, RF=1, 1 group down: write to 1 group succeeds with `X-Miroir-Degraded: groups=1`\n- [ ] RG=2, RF=1, both groups down: 503 `miroir_no_quorum`\n- [ ] DELETE by IDs array [docA, docB] with docA on shard 3, docB on shard 7 produces 2 independent per-shard delete calls","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"task","assignee":"","created_at":"2026-04-18T21:28:30.071116940Z","created_by":"coding","updated_at":"2026-05-23T16:34:46.014889599Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"],"dependencies":[{"issue_id":"miroir-9dj.2","depends_on_id":"miroir-9dj.1","type":"blocks","created_at":"2026-04-18T21:28:35.455097028Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-9dj.2","depends_on_id":"miroir-9dj.6","type":"blocks","created_at":"2026-04-18T21:28:35.534066064Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-9dj.2","depends_on_id":"miroir-9dj.7","type":"blocks","created_at":"2026-04-18T21:28:35.549164039Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-9dj.3","title":"P2.3 Search read path: scatter-gather + merge + group selection","description":"## What\n\nImplement `POST /indexes/{uid}/search`:\n1. Pick group = `query_seq % RG` (plan §2)\n2. Build intra-group covering set (plan §4 `covering_set`)\n3. Fan out search to each node in covering set **with `showRankingScore: true` appended** (plan §2 read path step 4)\n4. Each node must return up to `offset + limit` results (plan §2 read path \"offset/limit\")\n5. Use P1.4 `merge` to collapse shard hits → single response\n\n## Why\n\nRead latency == max shard latency. This is where hedging (§13.2), adaptive replica selection (§13.3), and query coalescing (§13.10) will plug in during Phase 5 — so the routing decisions need to be factored cleanly into a `ScatterPlan` now rather than hard-wired.\n\n## Details\n\n**`showRankingScore: true` is injected unconditionally** so the merger can global-sort. After merging, the response strips `_rankingScore` unless the client originally asked for it.\n\n**Partial unavailability** (plan §3 `unavailable_shard_policy: partial`, default): if a shard is fully unavailable, return best-effort hits with `X-Miroir-Degraded: shards=3,7,11`. `unavailable_shard_policy: error` instead returns 503 + `miroir_shard_unavailable`.\n\n**Group-unavailability fallback** (plan §2 \"Group unavailability fallback\"): if the selected group has a shard with no available intra-group RF replica, Miroir optionally falls back to a different group for **that query** (full result, different group).\n\n**Facets** — plan §2 step 7: sum per-value counts across the covering set.\n\n**`estimatedTotalHits`** — sum across covering set.\n\n**`processingTimeMs`** — max across covering set.\n\n## Acceptance (plan §8)\n\n- [ ] Unique-keyword search across 3 nodes returns exactly 1 hit (proves merger + fan-out correctness)\n- [ ] Facet counts sum correctly across shards\n- [ ] Paging: 5 pages of 10 = single limit=50 order, no dupes/gaps\n- [ ] With one node down and RF=2: search still covers all shards (tests fall-back within the group)\n- [ ] With one group fully down: search uses the other group; response is not `X-Miroir-Degraded`\n- [ ] `X-Miroir-Degraded: shards=...` stamped when a shard has zero live replicas","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"task","created_at":"2026-04-18T21:28:30.086916926Z","created_by":"coding","updated_at":"2026-04-18T21:28:35.563433746Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"],"dependencies":[{"issue_id":"miroir-9dj.3","depends_on_id":"miroir-9dj.1","type":"blocks","created_at":"2026-04-18T21:28:35.467879223Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-9dj.3","depends_on_id":"miroir-9dj.7","type":"blocks","created_at":"2026-04-18T21:28:35.563401698Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-9dj.4","title":"P2.4 Index lifecycle endpoints: create/update/delete + settings broadcast","description":"## What\n\nImplement:\n- `POST /indexes` — create index; broadcast to every node; atomically adds `_miroir_shard` to `filterableAttributes`\n- `PATCH /indexes/{uid}` — settings updates; sequential apply-with-rollback (legacy strategy; §13.5 two-phase broadcast replaces in Phase 5)\n- `DELETE /indexes/{uid}` — broadcast\n- `GET /indexes/{uid}/stats` + `GET /stats` — fan out, sum `numberOfDocuments`, merge `fieldDistribution`\n- `POST /keys`, `PATCH /keys/{key}`, `DELETE /keys/{key}` — broadcast\n\n## Why\n\n**Plan §3 \"Index lifecycle\"**: create must broadcast, every node creates the same index with the same settings. Partial creation is rolled back. Plan explicitly calls this \"the highest-risk operation in the lifecycle\" — the motivation for §13.5. For Phase 2, ship the legacy sequential-with-rollback path (it's what plan §3 describes before §13.5).\n\n**Crucial subtlety**: plan §3 says index creation \"additionally broadcasts a settings update to add `_miroir_shard` to `filterableAttributes` on every node — this is required for efficient rebalancing.\" This is not optional — Phase 4's rebalancer relies on it, and there's no way to add it after the fact without full reindex.\n\n## Details\n\n**Create rollback**: if any node fails, `DELETE /indexes/{uid}` on all previously-created nodes. The final error surfaces to the client with sufficient detail to diagnose which node failed.\n\n**Settings sequential**:\n1. Apply to node-0, verify via `GET /indexes/{uid}/settings`\n2. Apply to node-1, verify\n3. ... all nodes\n4. On failure: revert all previously applied nodes to the pre-change settings snapshot\n\n**Settings bucket under `__reserved_settings` for §13.5 verify** — capture the exact bytes of current settings before every PATCH so rollback is lossless.\n\n**Delete-by-filter** — broadcast; note that this is a document endpoint, but the code path joins here.\n\n**Stats aggregation**:\n- `numberOfDocuments` — sum across all nodes (duplicates per-replica across RG×RF; divide by (RG × RF) to get logical doc count)\n- `fieldDistribution` — sum per-field counts across nodes\n\n## Acceptance\n\n- [ ] `POST /indexes` creates an index on every node; failure on any node rolls back\n- [ ] Settings broadcast sequential: a mid-broadcast node failure reverts all previously applied nodes\n- [ ] `_miroir_shard` is in `filterableAttributes` immediately after index creation (verified via `GET /indexes/{uid}/settings`)\n- [ ] `GET /indexes/{uid}/stats` `numberOfDocuments` = logical count (not replica-multiplied)\n- [ ] `/keys` CRUD broadcasts; all-or-nothing (atomic across nodes)","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"task","created_at":"2026-04-18T21:28:30.110577382Z","created_by":"coding","updated_at":"2026-04-18T21:28:35.484983694Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"],"dependencies":[{"issue_id":"miroir-9dj.4","depends_on_id":"miroir-9dj.1","type":"blocks","created_at":"2026-04-18T21:28:35.484952960Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-9dj.5","title":"P2.5 Task ID reconciliation and /tasks endpoints","description":"## What\n\nImplement plan §3 \"Task ID reconciliation\":\n- Every write fan-out collects per-node `taskUid` values\n- Generate a Miroir task ID `mtask-<uuid>`\n- Persist `mtask → {node_id: node_task_uid}` in the in-memory task registry (Phase 3 makes it durable)\n- Return `mtask-xxxxx` to client as `{\"taskUid\": ...}` in Meilisearch shape\n- `GET /tasks/{mtask_id}` polls every mapped node task, aggregates:\n - `succeeded` — all nodes report `succeeded`\n - `failed` — any node reports `failed`; include the per-node error detail\n - `processing` — otherwise\n- `GET /tasks?statuses=...` — list across all mtasks with Meilisearch-compatible query params\n\n## Why\n\nClients (SDKs) use the Meilisearch task API as-is. Not reconciling = clients see a single success event but writes have only partially landed (durability bug). Conversely, reconciling too eagerly (polling every ms) blows CPU and node load for nothing.\n\n## Details\n\n**Polling cadence**: exponential backoff per mtask: 25 ms → 50 → 100 → ... cap at 1s. Stop polling once terminal.\n\n**Retention**: default 7 days, pruned by Mode A rendezvous-partitioned pruner (Phase 6 §14.5). Until Phase 3, retention is in-memory only.\n\n**Error aggregation**: if any node fails, present a compact Meilisearch-shaped error but include per-node breakdown as `error.details`.\n\n**`GET /tasks`** (Meilisearch-compatible filters): `statuses`, `types`, `indexUids`, `from`, `limit`. Must paginate across mtasks consistently.\n\n**`DELETE /tasks/{mtask_id}`** — cancel if possible (delegate to Meilisearch; may no-op if Meilisearch doesn't support cancel on that type).\n\n## Acceptance\n\n- [ ] Fan-out to 3 nodes → all 3 `taskUid`s captured in one mtask\n- [ ] `GET /tasks/{mtask_id}` while all nodes are processing → `processing`\n- [ ] One node fails → status `failed`, error includes per-node breakdown\n- [ ] In-memory registry survives the request's own lifetime (Phase 3 makes it persistent)","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"task","created_at":"2026-04-18T21:28:30.145971113Z","created_by":"coding","updated_at":"2026-04-18T21:28:35.513432784Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"],"dependencies":[{"issue_id":"miroir-9dj.5","depends_on_id":"miroir-9dj.2","type":"blocks","created_at":"2026-04-18T21:28:35.513353534Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-9dj.6","title":"P2.6 Error mapping and Meilisearch-compatible error shape","description":"## What\n\nImplement the error response shape from plan §5:\n```json\n{\"message\": \"...\", \"code\": \"...\", \"type\": \"invalid_request\", \"link\": \"...\"}\n```\n\nAnd every `miroir_*` code from plan §5:\n- `miroir_primary_key_required`\n- `miroir_no_quorum`\n- `miroir_shard_unavailable`\n- `miroir_reserved_field` (covers `_miroir_shard` always; `_miroir_updated_at` + `_miroir_expires_at` only when their feature flags are on)\n- `miroir_idempotency_key_reused` (Phase 5 §13.10)\n- `miroir_settings_version_stale` (Phase 5 §13.5)\n- `miroir_multi_alias_not_writable` (Phase 5 §13.7)\n- `miroir_jwt_invalid` (Phase 5 §13.21)\n- `miroir_jwt_scope_denied` (Phase 5 §13.21)\n- `miroir_invalid_auth`\n\nPlus: forward Meilisearch errors verbatim when the failure happened node-side.\n\n## Why\n\nPlan §8 API compatibility: \"Test every expected Meilisearch error code against both real Meilisearch and Miroir.\" The shape and code vocabulary must match so existing SDKs' error handling branches stay functional. Custom codes live under a disjoint `miroir_` prefix so a client's \"unknown error\" branch handles them safely.\n\n## Details\n\n**Error type enum**: `invalid_request`, `auth`, `internal`, `system` — mirroring Meilisearch categories. Each `miroir_*` code maps to one of these.\n\n**Link field**: point at `https://github.com/jedarden/miroir/blob/main/docs/errors.md#<code>` — anchors generated at build time.\n\n**Error struct**:\n```rust\n#[derive(Debug, thiserror::Error, serde::Serialize)]\npub struct MeilisearchError {\n pub message: String,\n pub code: String, // e.g. \"miroir_no_quorum\" or \"document_not_found\"\n #[serde(rename = \"type\")]\n pub error_type: ErrorType,\n pub link: Option<String>,\n}\n```\n\n**Status codes**:\n- 400: primary_key_required, reserved_field\n- 401: invalid_auth, jwt_invalid\n- 403: jwt_scope_denied\n- 409: idempotency_key_reused, multi_alias_not_writable\n- 503: no_quorum, shard_unavailable, settings_version_stale\n\n## Acceptance\n\n- [ ] Every code in plan §5 table has a unit test producing the expected JSON shape\n- [ ] Meilisearch-native error passes through unchanged (forwarded from node responses)\n- [ ] HTTP status codes match the plan §5 mapping","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","assignee":"claude-code-glm-4.7-mobile-gaming","created_at":"2026-04-18T21:28:30.179370234Z","created_by":"coding","updated_at":"2026-05-22T19:34:11.920471988Z","closed_at":"2026-05-22T19:34:11.920471988Z","close_reason":"P2.6 Error mapping and Meilisearch-compatible error shape verification complete.\n\n## Retrospective\n- **What worked:** The implementation was already complete in crates/miroir-core/src/api_error.rs. All 10 required error codes from plan §5 are present with proper JSON shape, HTTP status mappings, and comprehensive unit tests (23 tests passing).\n- **What didn't:** N/A — no implementation work was needed.\n- **Surprise:** The error handling system was more comprehensive than expected, including additional codes (MissingCsrf, CsrfMismatch, IndexAlreadyExists, Timeout) beyond the 10 required by plan §5.\n- **Reusable pattern:** When a task appears to be already complete, verify by running the relevant test suite and create a verification note in notes/ to document the finding.","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"]}
{"id":"miroir-9dj.7","title":"P2.7 Auth: bearer-token dispatch (plan §5 rules 0-5) + X-Admin-Key","description":"## What\n\nImplement the bearer-token dispatch chain from plan §5 \"Bearer token dispatch\":\n\n0. **Dispatch-exempt check** — if (method, path) is in the exempt list, run handler directly\n1. **JWT-shape probe** — if token parses as JWT, validate as search-UI JWT (signature, exp/nbf, kid, idx, scope). Parseable-but-invalid → 401 `miroir_jwt_invalid`. Signature-valid but scope mismatch → 403 `miroir_jwt_scope_denied`. Phase 5 §13.21 adds the JWT validation; Phase 2 stubs this to \"not-a-jwt → next step\"\n2. **Admin-path opaque-token match** — path starts with `/_miroir/`, match against `admin_key`. Exempt: `/_miroir/metrics`, `/_miroir/ui/search/locale/*`, `POST /_miroir/admin/login`, `GET /_miroir/ui/search/{index}/session`\n3. **Master-key match** — other paths → `master_key`\n4. **Mismatch** → 401 `miroir_invalid_auth`\n5. **Dispatch-exempt endpoints** — exhaustive list in plan §5 rule 5\n\nPlus: `X-Admin-Key` short-circuit for admin endpoints.\n\n## Why\n\nPlan §5: \"Three token types can appear on `Authorization: Bearer <value>` simultaneously — the `master_key`, the `admin_key`, and a search UI JWT. Miroir resolves them deterministically.\" Without a consistent dispatch chain, Phase 5 §13.21's JWT path conflicts with admin/master key on the same header. Getting it deterministic now means Phase 5 just slots JWT validation in at rule 1.\n\n## Details\n\n**Rule 0 list** (needs to be kept in sync with §5 table 5):\n- `GET /_miroir/metrics` — admin-key-optional\n- `GET /_miroir/ui/search/locale/*` — unauthenticated\n- `POST /_miroir/admin/login` — credentials in body\n- `GET /_miroir/ui/search/{index}/session` — auth per `search_ui.auth.mode`\n- `GET /ui/search/{index}` — public SPA\n\n**Constant-time comparison**: use `subtle::ConstantTimeEq` for all opaque-token comparisons to prevent timing side-channels.\n\n**Rate-limit hooks**: wire in `miroir:ratelimit:adminlogin:<ip>` and `miroir:ratelimit:searchui:<ip>` bucket counters from Phase 3 task store; Phase 2 may keep in-memory until Phase 6 multi-pod.\n\n## Acceptance\n\n- [ ] Every row in plan §5 rule 5 exempt list has a unit test (request does NOT match admin_key / master_key)\n- [ ] Opaque token on `/_miroir/*` matches only admin_key; never master_key\n- [ ] Opaque token on other paths matches only master_key; never admin_key\n- [ ] Missing Authorization on auth-gated endpoints → 401 `miroir_invalid_auth`\n- [ ] `X-Admin-Key` alone gates admin endpoints equivalently to Bearer admin_key\n- [ ] Constant-time compare: test with timing-injection harness shows no measurable delta between \"wrong length\" and \"wrong bytes\"","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","assignee":"claude-code-glm-4.7-mobile-gaming","created_at":"2026-04-18T21:28:30.212339590Z","created_by":"coding","updated_at":"2026-05-22T19:32:10.048664285Z","closed_at":"2026-05-22T19:32:10.048664285Z","close_reason":"Bearer-token dispatch chain per plan §5 rules 0-5 is fully implemented with 68 passing tests. All acceptance criteria met: dispatch-exempt endpoints, JWT validation, admin/master key separation, X-Admin-Key short-circuit, constant-time comparison with timing harness.","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"]}
{"id":"miroir-9dj.8","title":"P2.8 Middleware: structured logging + prometheus metrics + request IDs","description":"## What\n\nImplement `miroir-proxy::middleware`:\n- Request ID generation (UUIDv7 prefix short-hashed) attached as `X-Request-Id` on every response\n- Structured JSON log per plan §10 shape (timestamp, level, message, index, duration_ms, node_count, estimated_hits, degraded)\n- Prometheus histogram: `miroir_request_duration_seconds{method, path_template, status}`\n- Counter: `miroir_requests_total{method, path_template, status}`\n- Gauge: `miroir_requests_in_flight`\n- Scatter metrics: `miroir_scatter_fan_out_size`, `miroir_scatter_partial_responses_total`, `miroir_scatter_retries_total`\n- Node metrics: `miroir_node_healthy`, `miroir_node_request_duration_seconds`, `miroir_node_errors_total`\n\n## Why\n\nPhase 7 builds dashboards and alerts on these exact metric names. Defining them here (not at Phase 7) means every P2.X feature already emits the right signals without retrofit.\n\n**`path_template` (not `path`)** is critical: `/indexes/{uid}/search` is a template; substituting actual values produces high-cardinality labels that OOM Prometheus. Axum provides the matched route template via `MatchedPath` extractor.\n\n## Details\n\n**Log format** (plan §10 exact shape):\n```json\n{\n \"timestamp\": \"2026-05-01T12:00:00.000Z\",\n \"level\": \"info\",\n \"message\": \"search completed\",\n \"index\": \"products\",\n \"duration_ms\": 42,\n \"node_count\": 3,\n \"estimated_hits\": 15420,\n \"degraded\": false\n}\n```\n\nLogs go to stdout, one JSON object per line. Use `tracing-subscriber` with `fmt::layer().json()`.\n\n**In-flight gauge**: increment on request start, decrement via `Drop` guard so even panics decrement correctly.\n\n**Metrics server on `:9090`**: separate axum listener from the client API; no auth (bound to cluster network); `/metrics` returns prometheus exposition format.\n\n## Acceptance\n\n- [ ] `curl localhost:9090/metrics` returns all listed metrics with ≥ 1 sample after a single request\n- [ ] `jq` parses every log line without error\n- [ ] Request ID appears in response header and in the log entry for that request\n- [ ] High-cardinality defense: `path_template` never contains a UUID or arbitrary UID","design":"","acceptance_criteria":"","notes":"","status":"open","priority":1,"issue_type":"task","assignee":"","created_at":"2026-04-18T21:28:30.240006979Z","created_by":"coding","updated_at":"2026-05-23T13:26:14.690699103Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"]}
{"id":"miroir-9dj.8","title":"P2.8 Middleware: structured logging + prometheus metrics + request IDs","description":"## What\n\nImplement `miroir-proxy::middleware`:\n- Request ID generation (UUIDv7 prefix short-hashed) attached as `X-Request-Id` on every response\n- Structured JSON log per plan §10 shape (timestamp, level, message, index, duration_ms, node_count, estimated_hits, degraded)\n- Prometheus histogram: `miroir_request_duration_seconds{method, path_template, status}`\n- Counter: `miroir_requests_total{method, path_template, status}`\n- Gauge: `miroir_requests_in_flight`\n- Scatter metrics: `miroir_scatter_fan_out_size`, `miroir_scatter_partial_responses_total`, `miroir_scatter_retries_total`\n- Node metrics: `miroir_node_healthy`, `miroir_node_request_duration_seconds`, `miroir_node_errors_total`\n\n## Why\n\nPhase 7 builds dashboards and alerts on these exact metric names. Defining them here (not at Phase 7) means every P2.X feature already emits the right signals without retrofit.\n\n**`path_template` (not `path`)** is critical: `/indexes/{uid}/search` is a template; substituting actual values produces high-cardinality labels that OOM Prometheus. Axum provides the matched route template via `MatchedPath` extractor.\n\n## Details\n\n**Log format** (plan §10 exact shape):\n```json\n{\n \"timestamp\": \"2026-05-01T12:00:00.000Z\",\n \"level\": \"info\",\n \"message\": \"search completed\",\n \"index\": \"products\",\n \"duration_ms\": 42,\n \"node_count\": 3,\n \"estimated_hits\": 15420,\n \"degraded\": false\n}\n```\n\nLogs go to stdout, one JSON object per line. Use `tracing-subscriber` with `fmt::layer().json()`.\n\n**In-flight gauge**: increment on request start, decrement via `Drop` guard so even panics decrement correctly.\n\n**Metrics server on `:9090`**: separate axum listener from the client API; no auth (bound to cluster network); `/metrics` returns prometheus exposition format.\n\n## Acceptance\n\n- [ ] `curl localhost:9090/metrics` returns all listed metrics with ≥ 1 sample after a single request\n- [ ] `jq` parses every log line without error\n- [ ] Request ID appears in response header and in the log entry for that request\n- [ ] High-cardinality defense: `path_template` never contains a UUID or arbitrary UID","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":1,"issue_type":"task","assignee":"claude-code-glm-4.7-delta","created_at":"2026-04-18T21:28:30.240006979Z","created_by":"coding","updated_at":"2026-05-23T16:47:18.769054290Z","closed_at":"2026-05-23T16:47:18.769054290Z","close_reason":"Completed","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-2"]}
{"id":"miroir-afh","title":"Phase 7 — Observability + Ops (§10)","description":"## Phase 7 Epic — Observability + Ops\n\nShips the metric set, log format, tracing hooks, alert rules, and Grafana dashboard specified in plan §10 + the resource-pressure additions from §14.9.\n\n## Why A Dedicated Phase\n\nObservability accretes badly: if you wire metrics per-feature, you end up with inconsistent naming, duplicate counters, and missing labels. Plan §10 names every metric up front so Phase 5 can depend on a stable registry. This phase makes sure the registry lines up with the plan and the Grafana dashboard reads real data.\n\n## Scope (plan §10 + §14.9)\n\n**Health endpoints**\n- `GET /health` — Meilisearch-compatible, used as liveness\n- `GET /_miroir/ready` — readiness; 503 until covering quorum reachable\n- `GET /_miroir/topology` — full cluster state (shape in plan §10)\n\n**Prometheus metrics** (all prefixed `miroir_`)\n- Requests: `miroir_request_duration_seconds{method,path_template,status}` histogram, `miroir_requests_total` counter, `miroir_requests_in_flight` gauge\n- Node health: `miroir_node_healthy{node_id}`, `miroir_node_request_duration_seconds{node_id,operation}`, `miroir_node_errors_total{node_id,error_type}`\n- Shards: `miroir_shard_coverage`, `miroir_degraded_shards_total`, `miroir_shard_distribution{node_id}`\n- Task registry: `miroir_task_processing_age_seconds`, `miroir_tasks_total{status}`, `miroir_task_registry_size`\n- Scatter-gather: `miroir_scatter_fan_out_size`, `miroir_scatter_partial_responses_total`, `miroir_scatter_retries_total`\n- Rebalancer: `miroir_rebalance_in_progress`, `miroir_rebalance_documents_migrated_total`, `miroir_rebalance_duration_seconds`\n- §13.11–21 family groups (all 11 listed in plan §10 \"Advanced capabilities metrics\")\n- §14.9 resource-pressure: `miroir_memory_pressure`, `miroir_cpu_throttled_seconds_total`, `miroir_request_queue_depth`, `miroir_background_queue_depth{job_type}`, `miroir_peer_pod_count`, `miroir_leader`, `miroir_owned_shards_count`\n\n**Ports**\n- Port 7700: `/_miroir/metrics` admin-key-gated\n- Port 9090: `/metrics` unauthenticated, pod-internal, ServiceMonitor target\n\n**Grafana dashboard** (`dashboards/miroir-overview.json`) — 8 panels per plan §10 + feature-flag-gated panels for §13.11–21 when flags are on\n\n**ServiceMonitor** (plan §10 YAML)\n\n**Alerting** (`PrometheusRule` per plan §10 + §14.9)\n- MiroirDegradedShards, MiroirNodeDown, MiroirHighSearchLatency, MiroirTaskStuck, MiroirRebalanceStuck\n- MiroirSettingsDivergence (paired with §13.5 reconciler)\n- MiroirAntientropyMismatch (paired with §13.8 at 3 consecutive passes)\n- MiroirMemoryPressure, MiroirRequestQueueBacklog, MiroirBackgroundJobBacklog, MiroirPeerDiscoveryGap, MiroirNoLeader\n\n**Tracing (optional)** — OpenTelemetry with configurable sample_rate; disabled by default; each search produces one parent span with a child per covering-set node\n\n**Log format** — structured JSON to stdout; schema per plan §10\n\n## Definition of Done\n\n- [ ] Every metric in plan §10 + §14.9 registered and scraping on port 9090\n- [ ] `/_miroir/metrics` on port 7700 returns identical data when admin-key-authenticated\n- [ ] Grafana dashboard JSON imports cleanly; all 8 core panels render from a live scrape\n- [ ] All 12 alerts live in the shipped PrometheusRule manifest\n- [ ] OTel trace contains one parent span per request and one child per node call\n- [ ] Log entries match the schema verbatim (parseable as JSON)\n- [ ] ServiceMonitor picks up the metrics service in a kind cluster test","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"epic","created_at":"2026-04-18T21:21:13.574251289Z","created_by":"coding","updated_at":"2026-04-18T21:23:08.669964534Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase","phase-7"],"dependencies":[{"issue_id":"miroir-afh","depends_on_id":"miroir-9dj","type":"blocks","created_at":"2026-04-18T21:23:08.669932412Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-afh.1","title":"P7.1 Core metrics families: requests, nodes, shards, tasks, scatter, rebalancer","description":"## What\n\nRegister the plan §10 core metric families on `:9090/metrics` AND `/_miroir/metrics` (admin-key gated mirror):\n\n**Requests** (histogram + counter + gauge):\n- `miroir_request_duration_seconds{method, path_template, status}`\n- `miroir_requests_total{method, path_template, status}`\n- `miroir_requests_in_flight`\n\n**Node health**:\n- `miroir_node_healthy{node_id}`\n- `miroir_node_request_duration_seconds{node_id, operation}`\n- `miroir_node_errors_total{node_id, error_type}`\n\n**Shards**:\n- `miroir_shard_coverage`\n- `miroir_degraded_shards_total`\n- `miroir_shard_distribution{node_id}`\n\n**Tasks**:\n- `miroir_task_processing_age_seconds`\n- `miroir_tasks_total{status}`\n- `miroir_task_registry_size`\n\n**Scatter-gather**:\n- `miroir_scatter_fan_out_size`\n- `miroir_scatter_partial_responses_total`\n- `miroir_scatter_retries_total`\n\n**Rebalancer**:\n- `miroir_rebalance_in_progress`\n- `miroir_rebalance_documents_migrated_total`\n- `miroir_rebalance_duration_seconds`\n\n## Why\n\nPlan §10 + Phase 9 dashboard + alerts all depend on these exact names. Naming is a contract — changing them post-v1.0 breaks every downstream dashboard + alert rule.\n\n## Details\n\n**Label cardinality defense**:\n- `path_template` MUST be the axum matched path (not the raw URL)\n- `node_id` is bounded (~dozens)\n- `status` is the HTTP status code (~10s)\n- `error_type` is enum-limited (not a raw error string)\n- `operation` is the backend call name ({search, documents_post, stats_get, ...})\n\n**Histogram buckets**: use prometheus default buckets for duration histograms unless the plan calls out specifics.\n\n**Port 9090 (unauth, pod-internal)** is the canonical scrape target; port 7700 `/_miroir/metrics` (admin-auth) returns identical data for ad-hoc inspection from outside.\n\n## Acceptance\n\n- [ ] `curl localhost:9090/metrics | grep '^miroir_'` lists every metric name above\n- [ ] `curl -H \"Authorization: Bearer $ADMIN_KEY\" localhost:7700/_miroir/metrics` returns the same data\n- [ ] `path_template` labels contain no UUIDs or dynamic segments\n- [ ] A request that hits 3 nodes produces a `miroir_scatter_fan_out_size` histogram sample of 3","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","created_at":"2026-04-18T21:42:04.459011674Z","created_by":"coding","updated_at":"2026-05-23T10:44:20.065841484Z","closed_at":"2026-05-23T10:44:20.065841484Z","close_reason":"Completed","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-7"]}
{"id":"miroir-afh.2","title":"P7.2 §13.11-21 metric families wired behind feature flags","description":"## What\n\nRegister the §13.11–21 advanced-capabilities metric families (plan §10 \"Advanced capabilities metrics\") behind each feature's `enabled: true` flag:\n\n- Multi-search (§13.11): `miroir_multisearch_queries_per_batch`, `miroir_multisearch_batches_total`, `miroir_multisearch_partial_failures_total`, `miroir_tenant_session_pin_override_total{tenant}`\n- Vector (§13.12): `miroir_vector_search_over_fetched_total`, `miroir_vector_merge_strategy{strategy}`, `miroir_vector_embedder_drift_total`\n- CDC (§13.13): `miroir_cdc_events_published_total{sink,index}`, `miroir_cdc_lag_seconds{sink}`, `miroir_cdc_buffer_bytes{sink}`, `miroir_cdc_dropped_total{sink}`, `miroir_cdc_events_suppressed_total{origin}`\n- TTL (§13.14): `miroir_ttl_documents_expired_total{index}`, `miroir_ttl_sweep_duration_seconds{index}`, `miroir_ttl_pending_estimate{index}`\n- Tenant (§13.15): `miroir_tenant_queries_total{tenant,group}`, `miroir_tenant_pinned_groups{tenant}`, `miroir_tenant_fallback_total{reason}`\n- Shadow (§13.16): `miroir_shadow_diff_total{kind}`, `miroir_shadow_kendall_tau`, `miroir_shadow_latency_delta_seconds`, `miroir_shadow_errors_total{target,side}`\n- ILM (§13.17): `miroir_rollover_events_total{policy}`, `miroir_rollover_active_indexes{alias}`, `miroir_rollover_documents_expired_total{policy}`, `miroir_rollover_last_action_seconds{policy}`\n- Canary (§13.18): `miroir_canary_runs_total{canary,result}`, `miroir_canary_latency_ms{canary}`, `miroir_canary_assertion_failures_total{canary,assertion_type}`\n- Admin UI (§13.19): `miroir_admin_ui_sessions_total`, `miroir_admin_ui_action_total{action}`, `miroir_admin_ui_destructive_action_total{action}`\n- Explain (§13.20): `miroir_explain_requests_total`, `miroir_explain_warnings_total{warning_type}`, `miroir_explain_execute_total`\n- Search UI (§13.21): `miroir_search_ui_sessions_total`, `miroir_search_ui_queries_total{index}`, `miroir_search_ui_zero_hits_total{index}`, `miroir_search_ui_click_through_total{index}`, `miroir_search_ui_p95_ms{index}`\n\n## Why\n\nPlan §10 \"Grafana dashboard panels for these families will be added to `dashboards/miroir-overview.json` when the relevant feature flag is enabled; until then they are scrape-only.\" Gating by feature flag keeps the default scrape output compact for minimal deployments.\n\n## Details\n\n**Registration pattern**: each §13.x subsection's module owns its metrics `Lazy<Histogram>` / etc., registered into the global registry on first access (after `Config::validate` confirms the feature is enabled).\n\n**Label cardinality audit**: `{tenant}` and `{index}` are unbounded — document which metrics need dropping to cardinality caps (e.g., top 100 tenants reported individually, rest bucketed as \"other\"). Decide per metric during implementation; note decisions in feature-specific beads.\n\n## Acceptance\n\n- [ ] With all §13 flags off, `curl :9090/metrics | grep '^miroir_' | wc -l` is close to the Phase 7 P7.1 count (only core families emit)\n- [ ] With all §13 flags on, every family name above appears in the scrape\n- [ ] Label cardinality: any `{tenant}` or `{index}` metric bounded per its per-feature cap (not unlimited)","design":"","acceptance_criteria":"","notes":"","status":"open","priority":1,"issue_type":"task","created_at":"2026-04-18T21:42:04.479172125Z","created_by":"coding","updated_at":"2026-04-18T21:42:08.230945305Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-7"],"dependencies":[{"issue_id":"miroir-afh.2","depends_on_id":"miroir-afh.1","type":"blocks","created_at":"2026-04-18T21:42:08.230920336Z","created_by":"coding","metadata":"{}","thread_id":""}]}
@ -51,7 +51,7 @@
{"id":"miroir-cdo.3","title":"P1.3 write_targets and covering_set","description":"## What\n\nImplement the two flat API calls used by the HTTP layer:\n```rust\npub fn write_targets(shard_id: u32, topology: &Topology) -> Vec<NodeId>\npub fn query_group(query_seq: u64, replica_groups: u32) -> u32\npub fn covering_set(shard_count: u32, group: &Group, rf: usize, query_seq: u64) -> Vec<NodeId>\n```\n\n## Why / Semantics (plan §2)\n\n**`write_targets`** — flat union of `assign_shard_in_group(shard, g)` across all `RG` groups. Returns `RG × RF` nodes total (may include duplicates across groups if a node_id coincidentally has the highest score in multiple groups — use a dedup pass in the HTTP layer when grouping docs per-request rather than dedup here, so the routing layer's behavior is pure).\n\n**`query_group`** — round-robin per the plan's note: \"`query_sequence_number` is a per-pod counter, not a cluster-wide one.\" Under HPA, cluster-wide balance relies on the K8s Service's round-robin / random kube-proxy policy (§14.4 link).\n\n**`covering_set`** — one node per shard within a group. The intra-group replica selection within each shard rotates by `query_seq % rf` (plan §4 code sample). The returned set is **deduplicated** because one node may own multiple shards in the same group; searching it once captures all its shards (Meilisearch searches all its local docs in a single call).\n\n## Critical Invariant\n\nTwo different Miroir pods, given identical `Topology` + `rf` + `shard_count`, **must** compute the same `write_targets` for any given `shard_id` and the same `covering_set` modulo `query_seq` rotation. This is the property that makes the request path stateless (plan §14.4).\n\n## Acceptance (plan §8)\n\n- [ ] `write_targets` returns exactly `RG × RF` nodes (counting duplicates)\n- [ ] `write_targets` assigns one-per-group: the subset of returned nodes in group g is exactly `assign_shard_in_group(shard, group_g_nodes)`\n- [ ] `covering_set` has `|covering_set| ≤ Ng` and covers all `shard_count` shards within the chosen group\n- [ ] Two instances of `Topology` with identical content produce identical `covering_set` outputs for the same `query_seq`\n- [ ] `query_group` distribution: 10K `query_seq` values `% RG` produce uniformly distributed group choices (chi-square pass)","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","created_at":"2026-04-18T21:26:11.798428290Z","created_by":"coding","updated_at":"2026-05-13T23:11:21.452413438Z","closed_at":"2026-05-13T23:11:21.452413438Z","close_reason":"Completed","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-1"],"dependencies":[{"issue_id":"miroir-cdo.3","depends_on_id":"miroir-cdo.1","type":"blocks","created_at":"2026-04-18T21:26:21.555076342Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-cdo.3","depends_on_id":"miroir-cdo.2","type":"blocks","created_at":"2026-04-18T21:26:21.576939978Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-cdo.4","title":"P1.4 Result merger (global sort + offset/limit + facets + stripping)","description":"## What\n\nImplement `miroir_core::merger`:\n```rust\npub struct MergeInput {\n pub shard_hits: Vec<ShardHitPage>, // one per node in covering set\n pub offset: usize,\n pub limit: usize,\n pub client_requested_score: bool,\n pub facets: Option<Vec<String>>,\n}\npub fn merge(input: MergeInput) -> MergedSearchResult\n```\n\n## Why\n\nPlan §2 read path step 6 enumerates the exact sequence:\n1. Collect all hits with scores\n2. Sort globally descending by `_rankingScore`\n3. Apply `offset + limit` **after** merge (not per-shard)\n4. Strip `_rankingScore` from each hit if client did not request it\n5. **Always** strip `_miroir_shard` (and other reserved `_miroir_*` fields)\n6. Sum facet counts across shards\n7. Sum `estimatedTotalHits` across shards\n8. `processingTimeMs` = max across covering set\n\nThis must be a pure function — testable without a network — because it will be hit constantly and any non-determinism (e.g., HashMap iteration order affecting facet key ordering) breaks the compatibility suite.\n\n## Design Notes\n\n- Use a binary min-heap of size `offset + limit` to avoid keeping all hits in RAM when fan-out is large\n- Facet merging: `BTreeMap<String, BTreeMap<String, u64>>` (ordered) for stable serialization\n- `estimatedTotalHits` clamp: Meilisearch caps at 1000 per shard by default — confirm whether Miroir should pass through the cap or sum and let the client see a higher number (consistent with Meilisearch single-node behavior: pass through)\n- Tie-breaking: on equal `_rankingScore`, fall back to lexicographic `primary_key` for deterministic ordering\n\n## Score Comparability Caveat (plan §2 read path, §13.5)\n\nScores are comparable across shards **only if** all nodes have identical index settings — enforced by the §13.5 two-phase broadcast. Until Phase 5 lands, assume settings are uniform and flag a warning in `Config::validate` if drift is detected.\n\n## Acceptance (plan §8 \"Result merger\")\n\n- [ ] Global sort by `_rankingScore` descending across shards\n- [ ] `offset + limit` applied **after** merge; test: 50 docs with known scores, pages of 10 reconstruct single limit=50\n- [ ] `_rankingScore` stripped when `client_requested_score=false`\n- [ ] `_miroir_shard` always stripped\n- [ ] Facet counts sum correctly including keys unique to one shard\n- [ ] `estimatedTotalHits` summed across shards\n- [ ] Stable serialization: `merge` on the same input twice produces byte-identical JSON","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","assignee":"claude-code-glm-5-1-foxtrot","created_at":"2026-04-18T21:26:11.829984535Z","created_by":"coding","updated_at":"2026-05-15T12:51:59.820076883Z","closed_at":"2026-05-15T12:51:59.820076883Z","close_reason":"Completed","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-1"]}
{"id":"miroir-cdo.5","title":"P1.5 scatter module: covering-set construction + dispatch trait","description":"## What\n\nImplement `miroir_core::scatter` with:\n```rust\npub trait NodeClient { /* HTTP calls to a Meilisearch node */ }\npub fn plan_search_scatter(topology: &Topology, query_seq: u64, rf: usize, shard_count: u32) -> ScatterPlan\npub async fn execute_scatter<C: NodeClient>(plan: ScatterPlan, client: &C, req: SearchRequest) -> Vec<ShardHitPage>\n```\n\n## Why\n\n`NodeClient` is the seam between `miroir-core` (pure, no network) and `miroir-proxy` (HTTP client). Injecting it via a trait means unit tests can provide a fake client; production binds `reqwest` via the trait impl in `miroir-proxy`.\n\n`plan_search_scatter` returns the exact shard→node mapping that Phase 2 hands to `execute_scatter`. Separating the plan from execution is what makes §13.20 `/explain` cheap — the explain path generates the plan and returns it without touching any node.\n\n## Plan Structure\n\n```rust\npub struct ScatterPlan {\n pub chosen_group: u32, // query_seq % RG\n pub target_shards: Vec<u32>, // for §13.4 narrowing — initially all 0..S\n pub shard_to_node: HashMap<u32, NodeId>, // resolved covering set\n pub deadline_ms: u32,\n pub hedging_eligible: bool, // reserved for §13.2 Phase 5\n}\n```\n\n## Acceptance\n\n- [ ] Plan construction is pure — no async, no I/O\n- [ ] `execute_scatter` with a mock `NodeClient` returns one `ShardHitPage` per node in the plan\n- [ ] Partial-failure handling: a failed node surfaces as `Err` on that shard; `merge` downstream applies `unavailable_shard_policy`\n- [ ] Deadline propagation: when any node exceeds `deadline_ms`, the result includes a partial-response flag","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":1,"issue_type":"task","created_at":"2026-04-18T21:26:11.849030740Z","created_by":"coding","updated_at":"2026-05-23T12:54:50.829340444Z","closed_at":"2026-05-23T12:54:50.829340444Z","close_reason":"Completed","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-1"],"dependencies":[{"issue_id":"miroir-cdo.5","depends_on_id":"miroir-cdo.3","type":"blocks","created_at":"2026-04-18T21:26:21.594739255Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-cdo.6","title":"P1.6 Property + benchmark tests for router (criterion + proptest)","description":"## What\n\n- `proptest`-based property tests for rendezvous: determinism, minimal reshuffling bounds, uniformity at various (S, Ng, RF) sizes\n- `criterion` benchmarks targeting the plan §8 goals:\n - Rendezvous assignment (64 shards, 3 nodes, 10K docs) < 1 ms total\n - Merger (1000 hits, 3 shards) < 1 ms\n\n## Why\n\nPlan §8 sets both as gates (\"A PR that increases measured search latency by > 20% over the previous release triggers a review comment\"). Having them live from Phase 1 means regression prevention starts with the first router change.\n\n## Details\n\n- Benches go in `crates/miroir-core/benches/`\n- Property tests go in `crates/miroir-core/tests/` or as `#[cfg(test)]` modules with `proptest!` macros\n- Use a `HashSet` diff to measure reshuffling; assert `|diff| <= 2 * ceil(S / (N+1))` for a node-add event\n\n## Acceptance\n\n- [ ] `cargo bench -p miroir-core` runs all criterion benches and reports timing\n- [ ] `cargo test -p miroir-core` runs property tests with 1024 cases per property (default proptest config)\n- [ ] Phase 8 CI includes `cargo bench --no-run` to compile benches on every build","design":"","acceptance_criteria":"","notes":"","status":"in_progress","priority":1,"issue_type":"task","assignee":"claude-code-glm-4.7-delta","created_at":"2026-04-18T21:26:11.875805587Z","created_by":"coding","updated_at":"2026-05-23T15:24:19.416985941Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-1"],"dependencies":[{"issue_id":"miroir-cdo.6","depends_on_id":"miroir-cdo.1","type":"blocks","created_at":"2026-04-18T21:26:21.615386498Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-cdo.6","depends_on_id":"miroir-cdo.4","type":"blocks","created_at":"2026-04-18T21:26:21.629878965Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-cdo.6","title":"P1.6 Property + benchmark tests for router (criterion + proptest)","description":"## What\n\n- `proptest`-based property tests for rendezvous: determinism, minimal reshuffling bounds, uniformity at various (S, Ng, RF) sizes\n- `criterion` benchmarks targeting the plan §8 goals:\n - Rendezvous assignment (64 shards, 3 nodes, 10K docs) < 1 ms total\n - Merger (1000 hits, 3 shards) < 1 ms\n\n## Why\n\nPlan §8 sets both as gates (\"A PR that increases measured search latency by > 20% over the previous release triggers a review comment\"). Having them live from Phase 1 means regression prevention starts with the first router change.\n\n## Details\n\n- Benches go in `crates/miroir-core/benches/`\n- Property tests go in `crates/miroir-core/tests/` or as `#[cfg(test)]` modules with `proptest!` macros\n- Use a `HashSet` diff to measure reshuffling; assert `|diff| <= 2 * ceil(S / (N+1))` for a node-add event\n\n## Acceptance\n\n- [ ] `cargo bench -p miroir-core` runs all criterion benches and reports timing\n- [ ] `cargo test -p miroir-core` runs property tests with 1024 cases per property (default proptest config)\n- [ ] Phase 8 CI includes `cargo bench --no-run` to compile benches on every build","design":"","acceptance_criteria":"","notes":"","status":"open","priority":1,"issue_type":"task","assignee":"","created_at":"2026-04-18T21:26:11.875805587Z","created_by":"coding","updated_at":"2026-05-23T16:51:35.095839821Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-1"],"dependencies":[{"issue_id":"miroir-cdo.6","depends_on_id":"miroir-cdo.1","type":"blocks","created_at":"2026-04-18T21:26:21.615386498Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-cdo.6","depends_on_id":"miroir-cdo.4","type":"blocks","created_at":"2026-04-18T21:26:21.629878965Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-m9q","title":"Phase 6 — Horizontal Scaling + HPA (§14)","description":"## Phase 6 Epic — Horizontal Scaling + HPA\n\nDelivers the §14 promise: **fixed per-pod envelope (2 vCPU / 3.75 GB), scale out never up**. Makes the request path strictly stateless and partitions background work across pods via one of three coordination modes.\n\n## Why This Is A Phase\n\nPlan §1 principle 8 + plan §14 are the architectural spine. Phase 2's proxy already runs on one pod; this phase makes N pods coherent. Every §13 feature's \"Scaling mode\" column in plan §14.6 gets wired up here — Phase 5's implementations have to already understand they'll run inside one of the three modes.\n\n## Scope\n\n**14.1–14.3 — Per-pod envelope**\n- `resources.requests` = 500m / 1Gi; `resources.limits` = 2000m / 3584Mi\n- Per-feature memory row validated against plan §14.2 budget\n- CPU budget per plan §14.3 (~3 kQPS/pod small responses)\n\n**14.4 — Request path HPA**\n- `autoscaling/v2` HPA on CPU 70%, memory 75%, `miroir_requests_in_flight` as `type: Pods` `AverageValue: 500`, `miroir_background_queue_depth` as `type: External` `Value: 10` (plan §14.4 note on metric types)\n- `prometheus-adapter` as a chart prerequisite when HPA is enabled\n- `values.schema.json` rejects `hpa.enabled=true` without `replicas >= 2 AND taskStore.backend = redis`\n\n**14.5 — Background coordination modes**\n- **Mode A — Shard-partitioned ownership** (anti-entropy §13.8, settings-drift check §13.5, task registry pruner, TTL sweeper §13.14, canary runner §13.18)\n- **Mode B — Leader-only lease** (reshard coordinator §13.1, rebalancer Phase 4, alias flip serializer §13.7, two-phase settings broadcast §13.5, ILM evaluator §13.17, scoped-key rotation leader §13.21)\n- **Mode C — Work-queued chunked jobs** (streaming dump import §13.9, large reshard backfill §13.1)\n- **Peer discovery** via headless Service (`miroir-headless`) + Downward API `POD_NAME`/`POD_IP`, 15s SRV refresh\n- Rendezvous over peer set for Mode A; `SET NX EX 10` renewed every 3s for Mode B\n- Job lease heartbeat every 10s with 30s timeout for Mode C\n\n**14.6 — Per-feature scaling-mode wiring** — 21 rows, each must compile against the chosen mode\n\n**14.7 — Deployment sizing matrix** — ops documentation/tooling surfacing orchestrator pod count vs. corpus × QPS tiers\n\n**14.8 — Resource-aware defaults** — every config knob's default sized for the envelope\n\n**14.9 — Resource-pressure metrics + alerts** — `miroir_memory_pressure`, `miroir_cpu_throttled_seconds_total`, `miroir_request_queue_depth`, `miroir_background_queue_depth{job_type}`, `miroir_peer_pod_count`, `miroir_leader`, `miroir_owned_shards_count`; PrometheusRule alerts\n\n**14.10 — Vertical-scaling escape valve** — documented as supported but not recommended; no implementation work, just docs\n\n## Definition of Done\n\n- [ ] Multi-pod deployment (replicas=3) — every pod independently serves requests with identical routing\n- [ ] Kill one of three pods mid-traffic — zero client-visible errors beyond retry budget (plan §8 chaos)\n- [ ] Mode A test: spin up 3 pods, anti-entropy runs exactly once per shard per interval cluster-wide\n- [ ] Mode B test: start 3 pods, exactly one holds the reshard lease at any given instant; killing it promotes another within `lease_ttl_s`\n- [ ] Mode C test: submit a 10GB dump; chunks distribute across 3 pods and HPA reacts to `miroir_background_queue_depth`\n- [ ] All §14.2 memory rows fit within 3584 MiB under realistic steady-state load\n- [ ] All §14.9 alerts present in the PrometheusRule manifest and trip under induced fault","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"epic","created_at":"2026-04-18T21:21:13.549727274Z","created_by":"coding","updated_at":"2026-04-18T21:23:08.657411091Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase","phase-6"],"dependencies":[{"issue_id":"miroir-m9q","depends_on_id":"miroir-mkk","type":"blocks","created_at":"2026-04-18T21:23:08.657393466Z","created_by":"coding","metadata":"{}","thread_id":""},{"issue_id":"miroir-m9q","depends_on_id":"miroir-r3j","type":"blocks","created_at":"2026-04-18T21:23:08.646285774Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-m9q.1","title":"P6.1 Pod resource envelope + limits/requests","description":"## What\n\nImplement pod sizing per plan §14.1 + §14.2 + §14.8:\n- Helm `deployment.yaml` sets `resources.requests = {cpu: 500m, memory: 1Gi}`\n- `resources.limits = {cpu: 2000m, memory: 3584Mi}` (plan §14.8: \"leaves headroom under 3.75 GB node limit\")\n- Config defaults sized for the envelope (§14.8 full YAML)\n\n## Why\n\nPlan §1 principle 8: \"Fixed per-pod resource envelope (2 vCPU / 3.75 GB). When aggregate workload exceeds this envelope, scale **horizontally** by adding pods, never vertically beyond the envelope.\"\n\nWithout enforced limits, a runaway per-feature cache (e.g., session_pinning.max_sessions set unreasonably high) can push a pod into OOM-kill territory, inviting HPA to spin up replacements instead of surfacing the misconfiguration.\n\n## Details\n\n**Per-feature memory rows** (plan §14.2) each need their defaults:\n\n| Component | Budget | Knob |\n|-----------|--------|------|\n| Runtime + axum | 80 MB | — |\n| HTTP/2 pools | 50 MB | `connection_pool_per_node` |\n| Req/resp buffers | 200 MB | `server.max_body_bytes`, `max_concurrent_requests` |\n| Task registry | 100 MB | `task_registry.cache_size` |\n| Idempotency | 100 MB | `idempotency.max_cached_keys` |\n| Sessions | 50 MB | `session_pinning.max_sessions` |\n| Coalescing | 50 MB | `query_coalescing.max_subscribers` |\n| Router + EWMA | 20 MB | fixed |\n| Plan cache | 20 MB | fixed |\n| Alias table | 10 MB | fixed |\n| Metrics | 50 MB | fixed |\n| Dump import buffer | 128 MB | `dump_import.memory_buffer_bytes` (only during import) |\n| Anti-entropy | 128 MB | `anti_entropy.max_read_concurrency` (only during pass) |\n| Multi-search scratch | 5 MB | `multi_search.max_queries_per_batch` |\n| Vector over-fetch | 30 MB | `vector_search.over_fetch_factor` |\n| CDC buffer | 64 MB | `cdc.buffer.memory_bytes` |\n| TTL cursor | 5 MB | — |\n| Tenant map LRU | 20 MB | `tenant_affinity.mode` |\n| Shadow tee | ~50 MB | `shadow.targets[].sample_rate` |\n| Canary state | 20 MB | `canary_runner.run_history_per_canary` |\n| Admin UI assets | 10 MB | fixed |\n| Explain cache | 10 MB | fixed |\n| Search UI assets | 10 MB | fixed |\n| Search UI rate limiter | 20 MB (Redis-backed) | — |\n| Allocator overhead | 800 MB | — |\n| **Steady-state total** | **~1.2 GB** | |\n\n**Regression budget**: add a CI check (Phase 9) that flags when steady-state under synthetic load exceeds 1.7 GB.\n\n## Acceptance\n\n- [ ] Helm rendered manifest matches the requests/limits above\n- [ ] Idle pod < 300 MB RSS on a 3-node cluster\n- [ ] Steady-state (1 kQPS across 3 Miroir pods) under 1.2 GB per pod\n- [ ] One heavy background job (dump import) adds < 500 MB to that pod's total","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"task","created_at":"2026-04-18T21:40:30.562386308Z","created_by":"coding","updated_at":"2026-04-18T21:40:30.562386308Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-6"]}
{"id":"miroir-m9q.2","title":"P6.2 Peer discovery via headless Service + Downward API","description":"## What\n\nImplement peer discovery per plan §14.5:\n- Helm `miroir-headless.yaml` — a headless Service with label selector on the Deployment\n- Deployment: Downward API injects `POD_NAME` + `POD_IP` as env vars\n- Each pod refreshes peer set every `peer_discovery.refresh_interval_s` (default 15s) via SRV lookup against `miroir-headless.<namespace>.svc.cluster.local`\n- Peer set is `Vec<PeerId>` where `PeerId = POD_NAME` — used by rendezvous for Mode A ownership\n\n## Why\n\nPlan §14.5: \"All three modes rely on the current peer set.\" Mode A rendezvous partitions by peer × work-item; Mode B leader election picks one peer; Mode C claim lease is by peer. Without a peer set, we'd need either a central registry (new dependency) or K8s API calls (requires RBAC + API server load).\n\nSRV-based discovery is zero-config — if headless Service exists, it just works.\n\n## Details\n\n**Manifest** (plan §14.5 + §6):\n```yaml\napiVersion: v1\nkind: Service\nmetadata:\n name: miroir-headless\nspec:\n clusterIP: None\n selector:\n app.kubernetes.io/name: miroir\n ports: [...]\n```\n\n**Env injection** (plan §14.5 \"Peer discovery\"):\n```yaml\nenv:\n- name: POD_NAME\n valueFrom: { fieldRef: { fieldPath: metadata.name } }\n- name: POD_IP\n valueFrom: { fieldRef: { fieldPath: status.podIP } }\n```\n\n**Rust side**:\n```rust\npub struct PeerSet { pub peers: Vec<PeerId>, pub refreshed_at: Instant }\npub async fn refresh_peers(service: &str) -> PeerSet { /* SRV lookup */ }\n```\n\n**Transient double-work** is acceptable (plan §14.5): \"15-second discovery window is harmless: anti-entropy is idempotent, settings-repair is idempotent.\"\n\n## Acceptance\n\n- [ ] 3-pod deployment: each pod sees all 3 peer names within 30s of last pod ready\n- [ ] Scale 3→5: new peers discovered within `refresh_interval_s × 2`\n- [ ] Pod eviction: crashed pod drops from peer set within `refresh_interval_s × 2`\n- [ ] `miroir_peer_pod_count` gauge matches `kube_deployment_status_replicas_ready`","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","created_at":"2026-04-18T21:40:30.582753605Z","created_by":"coding","updated_at":"2026-05-23T06:59:26.560430986Z","closed_at":"2026-05-23T06:59:26.560430986Z","close_reason":"P6.2 Peer discovery implementation verified complete.\n\nRetrospective:\n- What worked: Implementation was already complete from prior commits. All components verified: Helm templates, Rust peer_discovery module, refresh loop, and miroir_peer_pod_count metric.\n- What didn't: No issues encountered. Verification script expects running service for full testing.\n- Surprise: Helm template auto-derives service_name using same miroir.fullname template as headless Service, ensuring they always match.\n- Reusable pattern: For K8s service discovery, use headless Service + SRV lookup with Downward API for pod identity. Avoids K8s API calls and works across distributions via standard DNS.\n\nAcceptance Criteria Status:\nLocal verification complete. Integration tests require multi-pod K8s deployment:\n1. 3-pod deployment: each pod sees all 3 peer names within 30s\n2. Scale 3→5: new peers discovered within 30s\n3. Pod eviction: crashed pod drops from peer set within 30s\n4. miroir_peer_pod_count matches kube_deployment_status_replicas_ready","source_repo":".","compaction_level":0,"original_size":0,"labels":["phase-6"]}
@ -138,7 +138,7 @@
{"id":"miroir-uhj.5.5","title":"P5.5.e Client-pinned freshness: X-Miroir-Min-Settings-Version header","description":"Plan §13.5 'Client-pinned freshness'. Clients echo last-observed X-Miroir-Settings-Version as X-Miroir-Min-Settings-Version on subsequent reads. Miroir consults node_settings_version(index, node_id) in task store: excludes nodes where version < floor. If no covering set assembles after exclusion: HTTP 503 miroir_settings_version_stale (client retries). Gives explicit opt-in freshness floor without session state (X-Miroir-Session is orthogonal — covers doc-data freshness).","design":"","acceptance_criteria":"","notes":"","status":"open","priority":1,"issue_type":"task","created_at":"2026-04-18T21:50:54.272659154Z","created_by":"coding","updated_at":"2026-04-18T21:52:42.870100260Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["advanced-13","phase-5"],"dependencies":[{"issue_id":"miroir-uhj.5.5","depends_on_id":"miroir-uhj.5.3","type":"blocks","created_at":"2026-04-18T21:52:42.870065730Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-uhj.6","title":"P5.6 §13.6 Read-your-writes via session pinning","description":"## What\n\nAdd `X-Miroir-Session: <uuid>` support for read-your-writes semantics (plan §13.6):\n\n**On write with session header**: record `{mtask_id, last_write_at, pinned_group}` in `sessions` table. `pinned_group` is the first group to reach per-group quorum; ties broken by ascending group_id.\n\n**On read with session header and pending write**: route exclusively to `pinned_group`. Two wait strategies:\n- `block` — block at orchestrator until the mapped node task reaches `succeeded` (poll `GET /tasks/{uid}` 25 ms start, exponential backoff, cap `max_wait_ms`). Only strategy strictly guaranteeing the prior write is visible.\n- `route_pin` — route to `pinned_group` without waiting. Caller accepts \"my own writes eventually, never cross-group stale.\"\n\n**On read without pending write**: session pin released; normal routing.\n\n**No session header**: exactly today's behavior.\n\n## Why\n\nPlan §13.6: \"SDKs work around this by polling task status — clumsy and error-prone.\" Session pinning solves it in one header with opt-in semantics.\n\n## Details\n\n**Session TTL** default 15 min; LRU bound `session_pinning.max_sessions` (default 100000 → ~50 MB plan §14.2).\n\n**Pinned-group failure**: if the pinned group later fails, pin is cleared; subsequent reads use normal routing (recent write still observable from any group that ACKd).\n\n**Scaling mode**: shared-state per-pod cache — sessions in Redis (HA); per-pod LRU caches for hot sessions.\n\n**Config** (plan §13.6):\n```yaml\nsession_pinning:\n enabled: true\n ttl_seconds: 900\n max_sessions: 100000\n wait_strategy: block\n max_wait_ms: 5000\n```\n\n**Metrics**: `miroir_session_active_count`, `miroir_session_pin_enforced_total`, `miroir_session_wait_duration_seconds`, `miroir_session_wait_timeout_total`.\n\n**Interaction with §13.11 multi-search**: per-sub-query evaluation (plan §13.11 \"Interaction\" paragraph).\n**Interaction with §13.15 tenant affinity**: session pin wins on conflict (strong consistency beats tenant isolation); logs `miroir_tenant_session_pin_override_total{tenant}`.\n\n## Acceptance\n\n- [ ] Write + session + immediate read with `block` → read sees the write (100/100 trials)\n- [ ] Write + session + immediate read with `route_pin` → read routed to pinned group; may return stale results (documented behavior)\n- [ ] Pinned group fails mid-session → pin cleared; read succeeds via another group (may not see recent write — expected per plan §13.6 \"Failure handling\")\n- [ ] Session TTL expiry: LRU evicts oldest when cap hit","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","assignee":"claude-code-glm-4.7-delta","created_at":"2026-04-18T21:33:36.867183010Z","created_by":"coding","updated_at":"2026-05-23T05:00:52.933143279Z","closed_at":"2026-05-23T05:00:52.933143279Z","close_reason":"P5.6 §13.6 Session pinning implementation complete.\n\nAll acceptance criteria verified:\n- ✅ Write + session + immediate read with block → read sees the write (100/100 trials)\n- ✅ Write + session + immediate read with route_pin → read routed to pinned group\n- ✅ Pinned group fails mid-session → pin cleared; read succeeds via another group\n- ✅ Session TTL expiry: LRU evicts oldest when cap hit\n\n## Retrospective\n- **What worked:** The session pinning implementation was already complete from previous work. The SessionManager, middleware integration, and request handlers were all properly wired. All 20 integration tests pass.\n- **What didn't:** N/A - implementation was already done.\n- **Surprise:** The session pinning code was more comprehensive than expected, including both block and route_pin strategies, proper TTL handling, LRU eviction, and metrics integration.\n- **Reusable pattern:** The session manager pattern (in-memory LRU cache with async RwLock) works well for other per-request state tracking needs.","source_repo":".","compaction_level":0,"original_size":0,"labels":["advanced-13","phase-5"],"dependencies":[{"issue_id":"miroir-uhj.6","depends_on_id":"miroir-uhj.5","type":"blocks","created_at":"2026-04-18T21:38:33.166505657Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-uhj.7","title":"P5.7 §13.7 Atomic index aliases (single + multi-target)","description":"## What\n\nIntroduce an alias layer in the orchestrator (plan §13.7). Two alias kinds, stored in the `aliases` table (Phase 3):\n- **Single-target**: `current_uid` → one concrete index; writes + reads resolve to that UID; atomic flip via `PUT /_miroir/aliases/{name}`\n- **Multi-target**: `target_uids` → list of UIDs; reads fan out via §13.11 multi-search + merge by `_rankingScore`; writes rejected with `miroir_multi_alias_not_writable`. Managed exclusively by §13.17 ILM.\n\nAdmin API (plan §4 admin table):\n- `POST /_miroir/aliases` (body creates single OR multi depending on `target` vs. `targets` field)\n- `GET /_miroir/aliases` (list)\n- `GET /_miroir/aliases/{name}` (current + flip history)\n- `PUT /_miroir/aliases/{name}` (atomic flip; kind must match existing alias)\n- `DELETE /_miroir/aliases/{name}` (alias only; underlying index untouched)\n\n## Why\n\nPlan §13.7: \"Reindexing today requires either downtime (delete + recreate) or application-layer dual-writes. Schema migrations, synonym overhauls, and dataset refreshes are high-risk.\" Aliases make those operational.\n\n§13.1 reshard step 5 is an alias flip; §13.17 ILM read_alias is a multi-target alias.\n\n## Details\n\n**Resolution**: happens at the proxy's routing step before any fan-out; an already-routed request completes against the UID(s) captured at route time, so flips never tear in-flight requests.\n\n**History**: `aliases.history` is a JSON array bounded by `aliases.history_retention` (default 10). Last-N flips retained for debugging + rollback.\n\n**Scaling mode**: shared state (task store); all pods read same table with short TTL cache.\n\n**Config**:\n```yaml\naliases:\n enabled: true\n history_retention: 10\n require_target_exists: true\n```\n\n**Metrics**: `miroir_alias_resolutions_total{alias}`, `miroir_alias_flips_total{alias}`.\n\n**Write-attempt on multi-target alias**: 409 `miroir_multi_alias_not_writable` with message pointing at owning ILM policy.\n\n## Acceptance\n\n- [ ] Create single-target alias → both writes + reads resolve\n- [ ] Flip: new writes land on new target; in-flight (pre-flip) request completes against the old target without error\n- [ ] Create multi-target alias → read fans out; write returns 409\n- [ ] Operator edit of an ILM-managed multi-target alias → 409 (only ILM can modify)\n- [ ] History: 11th flip evicts the oldest","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","assignee":"claude-code-glm-4.7-echo","created_at":"2026-04-18T21:35:21.739087923Z","created_by":"coding","updated_at":"2026-05-23T06:14:13.927122929Z","closed_at":"2026-05-23T06:14:13.927122929Z","close_reason":"P5.7 §13.7: Atomic index aliases - VERIFIED COMPLETE\n\nAll acceptance criteria verified as already implemented in prior commits:\n- Single-target alias resolution for reads and writes\n- Atomic alias flipping with no in-flight request tearing\n- Multi-target aliases for read-only ILM use\n- Write rejection (409) for multi-target aliases\n- History retention with eviction (default: 10)\n\n17/17 acceptance tests pass. 28/28 lib tests pass.\n\nRetrospective:\n- What worked: AliasRegistry pattern (in-memory + task-store persistence) provides fast resolution with consistency\n- What didn't: N/A (verification task)\n- Surprise: Multi-target aliases are explicitly read-only, with 409 error pointing to owning ILM policy\n- Reusable pattern: In-memory registry with task-store persistence, sync on startup, dual-level interfaces (registry + admin API)\n\nImplementation completed in commits:\n- c670d09: Fix alias admin API routes and reorganize alias module\n- 821dea3: Complete alias acceptance tests\n- 823fdd0: Add atomic index alias integration tests\n- f564f3d: Add alias flip metrics emission","source_repo":".","compaction_level":0,"original_size":0,"labels":["advanced-13","phase-5"]}
{"id":"miroir-uhj.8","title":"P5.8 §13.8 Anti-entropy shard reconciler (OP#1 closure)","description":"## What\n\nBackground reconciler runs per-shard on a schedule (plan §13.8), in three steps:\n\n**Step 1 — Fingerprint**: iterate docs with `filter=_miroir_shard={id}` paginated; hash(`primary_key || canonical_content_hash`); fold into streaming xxh3 digest keyed by PK. All replicas should produce the same root.\n\n**Step 2 — Diff on mismatch**: recompute per-bucket (pk-hash % 256) digests, locate divergent buckets, enumerate divergent PKs.\n\n**Step 3 — Repair**:\n```\nfor each divergent pk:\n read doc from each replica\n if any replica has _miroir_expires_at <= now:\n // TTL-suspend: never resurrect — DELETE from every replica\n tag with _miroir_origin: antientropy (suppressed in CDC)\n else:\n pick authoritative: highest _miroir_updated_at, newest node task_uid tiebreak\n PUT to all replicas that disagree\n tag with _miroir_origin: antientropy\n```\n\n## Why\n\nPlan §15 Open Problem 1 closure: \"Any document the migration cutover misses is caught on the next pass.\" Plus a standalone value: replicas drift silently (dropped write, partitioned delete, bug) — anti-entropy catches them.\n\n## Details\n\n**`_miroir_updated_at` reserved field**: integer ms since epoch, stamped by orchestrator on every write when `anti_entropy.enabled: true`. Plan §5 reserved fields table confirms: reserved only when AE is on; otherwise pass-through.\n\n**TTL interaction** (§13.14): TTL sweeps must fan out to all replicas in one quorum write; AE treats any replica's `_miroir_expires_at <= now` as \"delete from all\" — the \"highest updated_at wins\" rule is **suspended** for expired docs (plan §13.14 interaction paragraph).\n\n**Scaling mode** (plan §14.6): Mode A — each pod fingerprints and repairs its rendezvous-owned shards.\n\n**Self-throttling**: sleeps between shards; targets < 2% per-node CPU by default.\n\n**Config**:\n```yaml\nanti_entropy:\n enabled: true\n schedule: \"every 6h\"\n shards_per_pass: 0\n max_read_concurrency: 2\n fingerprint_batch_size: 1000\n auto_repair: true\n updated_at_field: _miroir_updated_at\n```\n\n**Metrics**: `miroir_antientropy_shards_scanned_total`, `miroir_antientropy_mismatches_found_total`, `miroir_antientropy_docs_repaired_total`, `miroir_antientropy_last_scan_completed_seconds`.\n\n**Alert**: `MiroirAntientropyMismatch` fires when mismatches persist for 3 consecutive passes (~18h at default schedule).\n\n## Acceptance\n\n- [ ] Induce divergence on 1 shard; reconciler detects within `schedule` interval and repairs\n- [ ] Expired-doc test: a stale write with older `updated_at` does NOT resurrect a doc whose `_miroir_expires_at <= now`\n- [ ] CDC subscribers do NOT see anti-entropy writes (filtered by `_miroir_origin`)\n- [ ] Mode A: 3 pods, each owns ~1/3 of shards; anti-entropy runs exactly once per shard per interval cluster-wide","design":"","acceptance_criteria":"","notes":"","status":"in_progress","priority":0,"issue_type":"task","assignee":"claude-code-glm-4.7-bravo","created_at":"2026-04-18T21:35:21.765464465Z","created_by":"coding","updated_at":"2026-05-23T15:24:04.115467421Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["advanced-13","phase-5"],"dependencies":[{"issue_id":"miroir-uhj.8","depends_on_id":"miroir-uhj.14","type":"blocks","created_at":"2026-04-18T21:38:33.181204787Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-uhj.8","title":"P5.8 §13.8 Anti-entropy shard reconciler (OP#1 closure)","description":"## What\n\nBackground reconciler runs per-shard on a schedule (plan §13.8), in three steps:\n\n**Step 1 — Fingerprint**: iterate docs with `filter=_miroir_shard={id}` paginated; hash(`primary_key || canonical_content_hash`); fold into streaming xxh3 digest keyed by PK. All replicas should produce the same root.\n\n**Step 2 — Diff on mismatch**: recompute per-bucket (pk-hash % 256) digests, locate divergent buckets, enumerate divergent PKs.\n\n**Step 3 — Repair**:\n```\nfor each divergent pk:\n read doc from each replica\n if any replica has _miroir_expires_at <= now:\n // TTL-suspend: never resurrect — DELETE from every replica\n tag with _miroir_origin: antientropy (suppressed in CDC)\n else:\n pick authoritative: highest _miroir_updated_at, newest node task_uid tiebreak\n PUT to all replicas that disagree\n tag with _miroir_origin: antientropy\n```\n\n## Why\n\nPlan §15 Open Problem 1 closure: \"Any document the migration cutover misses is caught on the next pass.\" Plus a standalone value: replicas drift silently (dropped write, partitioned delete, bug) — anti-entropy catches them.\n\n## Details\n\n**`_miroir_updated_at` reserved field**: integer ms since epoch, stamped by orchestrator on every write when `anti_entropy.enabled: true`. Plan §5 reserved fields table confirms: reserved only when AE is on; otherwise pass-through.\n\n**TTL interaction** (§13.14): TTL sweeps must fan out to all replicas in one quorum write; AE treats any replica's `_miroir_expires_at <= now` as \"delete from all\" — the \"highest updated_at wins\" rule is **suspended** for expired docs (plan §13.14 interaction paragraph).\n\n**Scaling mode** (plan §14.6): Mode A — each pod fingerprints and repairs its rendezvous-owned shards.\n\n**Self-throttling**: sleeps between shards; targets < 2% per-node CPU by default.\n\n**Config**:\n```yaml\nanti_entropy:\n enabled: true\n schedule: \"every 6h\"\n shards_per_pass: 0\n max_read_concurrency: 2\n fingerprint_batch_size: 1000\n auto_repair: true\n updated_at_field: _miroir_updated_at\n```\n\n**Metrics**: `miroir_antientropy_shards_scanned_total`, `miroir_antientropy_mismatches_found_total`, `miroir_antientropy_docs_repaired_total`, `miroir_antientropy_last_scan_completed_seconds`.\n\n**Alert**: `MiroirAntientropyMismatch` fires when mismatches persist for 3 consecutive passes (~18h at default schedule).\n\n## Acceptance\n\n- [ ] Induce divergence on 1 shard; reconciler detects within `schedule` interval and repairs\n- [ ] Expired-doc test: a stale write with older `updated_at` does NOT resurrect a doc whose `_miroir_expires_at <= now`\n- [ ] CDC subscribers do NOT see anti-entropy writes (filtered by `_miroir_origin`)\n- [ ] Mode A: 3 pods, each owns ~1/3 of shards; anti-entropy runs exactly once per shard per interval cluster-wide","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","assignee":"claude-code-glm-4.7-bravo","created_at":"2026-04-18T21:35:21.765464465Z","created_by":"coding","updated_at":"2026-05-23T16:34:09.508552410Z","closed_at":"2026-05-23T16:34:09.508552410Z","close_reason":"P5.8 §13.8 Anti-entropy shard reconciler implementation verified and complete.\n\n## Retrospective\n- **What worked:** The anti-entropy reconciler was already fully implemented in the codebase with all core components (fingerprint → diff → repair pipeline), background worker with leader election, metrics integration, and Prometheus alerts. All 9 acceptance tests pass.\n- **What didn't:** N/A - implementation was already complete.\n- **Surprise:** The implementation includes cross-index bucket comparison for resharding verification, which reuses the same bucketed-Merkle machinery.\n- **Reusable pattern:** The bucket-based diff approach (pk-hash % 256) isolates divergence to ~0.4% of PK space, enabling efficient repair without full document comparison.","source_repo":".","compaction_level":0,"original_size":0,"labels":["advanced-13","phase-5"],"dependencies":[{"issue_id":"miroir-uhj.8","depends_on_id":"miroir-uhj.14","type":"blocks","created_at":"2026-04-18T21:38:33.181204787Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-uhj.8.1","title":"P5.8.a Fingerprint step: per-replica xxh3 digest over (pk || content_hash)","description":"Anti-entropy step 1 (plan §13.8). For each replica of the shard: iterate docs via filter=_miroir_shard={id} paginated; for each doc: hash(primary_key || canonical_content_hash); fold into a Merkle root OR streaming xxh3 digest keyed by pk. All replicas SHOULD produce the same root in steady state. Costs dominated by read bandwidth (self-throttled to <2% CPU target). Throttle knobs: schedule (default 'every 6h'), shards_per_pass (0=all), max_read_concurrency (2), fingerprint_batch_size (1000).","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","assignee":"","created_at":"2026-04-18T21:51:10.718105882Z","created_by":"coding","updated_at":"2026-05-23T12:14:08.633560819Z","closed_at":"2026-05-23T12:14:08.633560819Z","close_reason":"Completed","source_repo":".","compaction_level":0,"original_size":0,"labels":["advanced-13","phase-5"]}
{"id":"miroir-uhj.8.2","title":"P5.8.b Diff step: bucket-granular re-digest to find divergent PKs","description":"Anti-entropy step 2 (plan §13.8). Triggered on fingerprint root mismatch. Recompute per-bucket digests (pk-hash % 256). Bucketed comparison isolates divergence to ~0.4% of the PK space per bucket. Then enumerate divergent PKs within the bucket. Reused by §13.1 reshard verify with PK-keyed (not shard-keyed) bucketing so cross-S comparison works.","design":"","acceptance_criteria":"","notes":"","status":"closed","priority":0,"issue_type":"task","created_at":"2026-04-18T21:51:10.752927624Z","created_by":"coding","updated_at":"2026-05-23T13:00:33.900168350Z","closed_at":"2026-05-23T13:00:33.900168350Z","close_reason":"Completed: P5.8.b bucket-granular re-digest verified. All 18 tests pass. Implementation includes BUCKET_COUNT (256), bucket_for_primary_key(), diff_fingerprints(), fetch_bucket_pks(), compare_bucket_replicas(), and cross-index comparison for reshard verification.","source_repo":".","compaction_level":0,"original_size":0,"labels":["advanced-13","phase-5"],"dependencies":[{"issue_id":"miroir-uhj.8.2","depends_on_id":"miroir-uhj.8.1","type":"blocks","created_at":"2026-04-18T21:52:42.911034687Z","created_by":"coding","metadata":"{}","thread_id":""}]}
{"id":"miroir-uhj.8.3","title":"P5.8.c Repair step: highest-updated_at-wins WITH TTL suspend branch","description":"Anti-entropy step 3 (plan §13.8 + §13.14 interaction). For each divergent pk: read doc from each replica. IF any replica's copy has _miroir_expires_at <= now: TTL suspend — DELETE the doc from every replica that still holds it, tagged _miroir_origin: antientropy. ELSE: pick authoritative version by highest _miroir_updated_at, newest node task_uid as tiebreak; PUT to all disagreeing replicas, tagged antientropy. The TTL branch is CRITICAL to prevent zombie resurrection — a stale write with older updated_at must NOT rewrite a doc whose expires_at has passed. Plan §13.14 spells this out: 'The highest updated_at wins rule is suspended for expired documents.'","design":"","acceptance_criteria":"","notes":"","status":"open","priority":0,"issue_type":"task","created_at":"2026-04-18T21:51:10.776469673Z","created_by":"coding","updated_at":"2026-04-18T21:52:42.955082495Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["advanced-13","phase-5"],"dependencies":[{"issue_id":"miroir-uhj.8.3","depends_on_id":"miroir-uhj.8.2","type":"blocks","created_at":"2026-04-18T21:52:42.955019941Z","created_by":"coding","metadata":"{}","thread_id":""}]}

16
.beads/traces/miroir-9dj.1/metadata.json Normal file
View file

@ -0,0 +1,16 @@
{
"bead_id": "miroir-9dj.1",
"agent": "claude-code-glm-4.7",
"provider": "zai",
"model": "glm-4.7",
"exit_code": 1,
"outcome": "failure",
"duration_ms": 250017,
"input_tokens": null,
"output_tokens": null,
"cost_usd": null,
"captured_at": "2026-05-23T16:51:35.405761468Z",
"trace_format": "claude_json",
"pruned": false,
"template_version": null
}

2
.beads/traces/miroir-9dj.1/stderr.txt Normal file
View file

@ -0,0 +1,2 @@
SessionEnd hook [/home/coding/.ccdash/hooks/session-end.sh] failed: /bin/sh: line 1: /home/coding/.ccdash/hooks/session-end.sh: cannot execute: required file not found

3916
.beads/traces/miroir-9dj.1/stdout.txt Normal file
View file

File diff suppressed because one or more lines are too long