jedarden/miroir
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
630 commits 1 branch 0 tags 58 MiB
Commit graph

8 commits

Author SHA1 Message Date
jedarden
1f686c646b Merge remote-tracking branch 'origin/master' 
# Conflicts:
#	.beads/issues.jsonl
#	.beads/traces/bf-5xqk/metadata.json
#	.beads/traces/bf-5xqk/stdout.txt
#	.beads/traces/miroir-9dj/metadata.json
#	.beads/traces/miroir-9dj/stdout.txt
#	.beads/traces/miroir-cdo/metadata.json
#	.beads/traces/miroir-cdo/stdout.txt
#	.beads/traces/miroir-mkk/metadata.json
#	.beads/traces/miroir-mkk/stdout.txt
#	.beads/traces/miroir-r3j/metadata.json
#	.beads/traces/miroir-r3j/stdout.txt
#	.beads/traces/miroir-uhj/metadata.json
#	.beads/traces/miroir-uhj/stdout.txt
#	.beads/traces/miroir-zc2.6/metadata.json
#	.beads/traces/miroir-zc2.6/stdout.txt
#	.needle-predispatch-sha
#	Cargo.lock
#	charts/miroir/Chart.yaml
#	charts/miroir/templates/NOTES.txt
#	charts/miroir/templates/_helpers.tpl
#	charts/miroir/templates/redis-deployment.yaml
#	charts/miroir/templates/serviceaccount.yaml
#	charts/miroir/tests/README.md
#	charts/miroir/values.schema.json
#	charts/miroir/values.yaml
#	crates/miroir-core/Cargo.toml
#	crates/miroir-core/src/config.rs
#	crates/miroir-core/src/hedging.rs
#	crates/miroir-core/src/lib.rs
#	crates/miroir-core/src/merger.rs
#	crates/miroir-core/src/query_planner.rs
#	crates/miroir-core/src/raft_proto/mod.rs
#	crates/miroir-core/src/replica_selection.rs
#	crates/miroir-core/src/router.rs
#	crates/miroir-core/src/scatter.rs
#	crates/miroir-core/src/task_store/mod.rs
#	crates/miroir-core/src/task_store/redis.rs
#	crates/miroir-core/src/task_store/sqlite.rs
#	crates/miroir-core/src/topology.rs
#	crates/miroir-ctl/src/credentials.rs
#	crates/miroir-proxy/Cargo.toml
#	crates/miroir-proxy/src/auth.rs
#	crates/miroir-proxy/src/client.rs
#	crates/miroir-proxy/src/lib.rs
#	crates/miroir-proxy/src/main.rs
#	crates/miroir-proxy/src/middleware.rs
#	crates/miroir-proxy/src/routes/admin.rs
#	crates/miroir-proxy/src/routes/documents.rs
#	crates/miroir-proxy/src/routes/indexes.rs
#	crates/miroir-proxy/src/routes/search.rs
#	crates/miroir-proxy/src/routes/settings.rs
#	crates/miroir-proxy/src/routes/tasks.rs
#	docs/research/score-normalization-at-scale.md
#	notes/miroir-cdo.md
#	notes/miroir-r3j-final-verification.md
#	notes/miroir-r3j-verification.md
#	notes/miroir-r3j.1.md
#	notes/miroir-r3j.md
#	notes/miroir-zc2.1.md
#	notes/miroir-zc2.3.md
#	notes/miroir-zc2.4.md
#	notes/miroir-zc2.5.md
 2026-05-24 05:21:32 -04:00
jedarden
7832d1b578 test(integration): Add integration tests per plan §8 
Add comprehensive integration tests for Miroir with 3 Meilisearch nodes
via docker-compose. Tests cover:

- Document round-trip with distribution verification (1000 docs)
- Search covers all shards (100 docs with unique keywords)
- Facet aggregation across shards (100 docs, 3 colors)
- Offset/limit paging consistency (50 docs, 5×paged vs single)
- Settings broadcast to all nodes (synonyms test)
- Task polling for large batches (500 docs)
- Node failure with RF=2 (requires docker-compose-dev-rf2)

Also added integration test README with setup and running instructions.

Per plan §8: Integration tests validate end-to-end behavior including
document distribution, shard coverage, facet aggregation, paging, settings
broadcast, task polling, and node failure with RF=2.

Closes: miroir-89x (Phase 9 — Testing)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-24 01:29:45 -04:00
jedarden
77ff0b7528 Phase 6 — Horizontal Scaling + HPA (§14): Complete 
Implements the full horizontal scaling architecture with HPA integration
and three coordination modes for background work partitioning.

## §14.1-§14.3 — Per-pod envelope
- Resource limits: 2000m CPU / 3584MiB RAM (2 vCPU / 3.75 GB)
- Memory budget validated for all §13 features
- CPU budget: ~3 kQPS/pod (small), ~1 kQPS/pod (large) at 70%

## §14.4 — Request path HPA
- autoscaling/v2 HPA with CPU 70%, memory 75%
- Custom metrics: miroir_requests_in_flight (Pods/AverageValue: 500)
- Custom metrics: miroir_background_queue_depth (External/Value: 10)
- prometheus-adapter ConfigMap for custom metrics discovery
- Chart dependency on prometheus-adapter (auto-enabled when hpa.enabled=true)
- values.schema.json Rule 2: HPA requires replicas >= 2 AND Redis backend

## §14.5 — Background coordination modes
- Mode A (shard-partitioned): anti_entropy_worker.rs, drift_reconciler.rs
- Mode B (leader-only): mode_b_coordinator.rs + leader_election/
- Mode C (work-queued): mode_c_coordinator.rs + mode_c_worker/
- Peer discovery via headless Service SRV records (15s refresh)

## §14.6 — Per-feature scaling mode wiring
- docs/horizontal-scaling/per-feature.md maps all 21 features to modes
- Forced-mode constraints in values.schema.json (Rules 0-5)

## §14.7 — Deployment sizing matrix
- docs/horizontal-scaling/sizing.md with workload tiers
- Task-store memory accounting for Redis-backed deployments

## §14.8 — Resource-aware configuration defaults
- charts/miroir/values.yaml with envelope-sized defaults
- tests/fixtures/section-14.8-defaults.yaml as reference

## §14.9 — Resource-pressure metrics and alerts
- miroir_memory_pressure, miroir_cpu_throttled_seconds_total
- miroir_request_queue_depth, miroir_background_queue_depth
- miroir_peer_pod_count, miroir_leader, miroir_owned_shards_count
- PrometheusRule with all alerts (MiroirMemoryPressure, etc.)

## §14.10 — Vertical-scaling escape valve
- docs/horizontal-scaling/single-pod.md documents single-pod mode
- tests/fixtures/section-14.10-single-pod-oversized.yaml with 2.13× multiplier

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-24 00:23:57 -04:00
jedarden
6c32dd8efc Phase 0 (miroir-qon): Rust 1.88 upgrade + test infrastructure 
- Bump Rust toolchain from 1.87 to 1.88
- Add testcontainers and arbitrary dependencies for property testing
- Update router with rendezvous hashing improvements
- Fix credential handling in miroir-ctl
- Update reshard and migration modules
- Add Helm chart scaffolding
- Add Redis memory accounting documentation

All Phase 0 DoD checks pass:
- cargo build --all succeeds
- cargo test --all succeeds (103 tests)
- cargo clippy --all-targets --all-features -- -D warnings passes
- cargo fmt --all -- --check passes
- Config round-trip YAML test passes

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-09 02:05:44 -04:00
jedarden
096b43ccab P12.OP4: Implement dfs_query_then_fetch for cross-shard comparability 
Implements the Elasticsearch dfs_query_then_fetch pattern as a pre-query
phase in Miroir to resolve cross-shard score comparability issues caused
by differing local IDF values across shards with skewed document distributions.

Core changes:
- scatter.rs: New PreflightRequest/PreflightResponse types, GlobalIdf
  aggregation, execute_preflight and dfs_query_then_fetch_search functions
- Proxy client: preflight_node implementation for term-frequency gathering
- Search routes: Integration of DFS preflight before main search phase
- Integration test: dfs_skewed_corpus.rs with 10 tests covering aggregation
  and serialization
- Benchmark: dfs_preflight_bench.rs measuring preflight overhead

Validation results (1,443 queries, 10-shard skewed corpus):
- Average Kendall tau: 0.9815 (95% CI: [0.9809, 0.9821])
- Min tau: 0.9523 (zero queries below 0.95 threshold)
- Per-type: common-term +0.84, single-term +0.11, filtered +0.11

The preflight phase adds one network round-trip before the search phase,
with requests parallelized across shards. Estimated overhead: +1-2 RTTs.

Resolves bead miroir-yio: Global-IDF preflight implementation.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-19 03:43:10 -04:00
jedarden
612e7ce0ea P1.5: Implement scatter module with covering-set construction + dispatch trait 
- Add NodeClient trait for HTTP calls to Meilisearch nodes (seam between pure miroir-core and networked miroir-proxy)
- Add ScatterPlan struct containing chosen_group, target_shards, shard_to_node mapping, deadline_ms, hedging_eligible
- Implement plan_search_scatter() pure function that constructs the covering set without I/O
- Implement execute_scatter() async function that fans out to nodes with partial-failure handling
- Add MockNodeClient for testing with pre-programmed responses/errors
- Add unit tests for plan construction, query group rotation, shard-to-node mapping, hedging eligibility, and scatter execution

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-19 00:20:29 -04:00
jedarden
601988829d P0.1: Set up Cargo workspace + toolchain pin 
- Update workspace Cargo.toml: explicit members list, edition 2021, MIT license, rust-version 1.87
- Simplify workspace.dependencies to core shared deps
- Update member crates to use explicit dependency versions where workspace inheritance was removed

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-18 20:52:53 -04:00
jedarden
409f952f59 Add repo hygiene: LICENSE, CHANGELOG, .gitignore 
- LICENSE: MIT (per plan §12)
- CHANGELOG.md: Keep a Changelog 1.1.0 skeleton with [Unreleased]
  and [0.1.0] sections matching the awk extractor from plan §7
- .gitignore: Rust target/, editor junk; Cargo.lock kept in VCS

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-04-18 20:47:36 -04:00