Commit graph

24 commits

Author SHA1 Message Date
jedarden
9c617001ab fix(ci): add make to build dependencies in miroir-release workflow
The rdkafka-sys crate build was failing with 'No such file or directory (os error 2)'
because the make command was missing from the build dependencies. This is required
for building librdkafka statically. Adding make to the apt-get install list.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-02 09:19:00 -04:00
jedarden
03f228c62e fix(ci): ensure c++ symlink exists for rdkafka build
The librdkafka configure script looks for the 'c++' command, not just 'g++'.
Installing g++ should provide the c++ symlink via alternatives, but this
doesn't always work correctly in minimal containers. This adds an explicit
fallback to create the symlink if update-alternatives fails.
2026-07-02 09:13:55 -04:00
jedarden
57e3b186b0 fix(ci): install git in miroir-release build-binaries step
The rust:1.87-slim image doesn't include git, but the build-binaries step
runs 'git clone' which fails with exit code 127 (command not found).
Install git along with other build dependencies before cloning.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-02 08:52:55 -04:00
jedarden
525ff7f9b4 fix(ci): use bash instead of sh in miroir-release build-binaries step
The rust:1.87-slim image doesn't have sh in a location that Argo's
executor can find. Using bash (which is at /usr/bin/bash) fixes
the 'command not found' (exit code 127) error.
2026-07-02 08:41:24 -04:00
jedarden
c94d031f0b fix: explicitly set PATH for cargo in build-binaries step
- Added export PATH=/usr/local/cargo/bin:$PATH to ensure cargo is found
- Fixes exit code 127 (command not found) in build-binaries step
2026-07-02 08:34:40 -04:00
jedarden
2cc0962d11 fix: use /workspace/src in build-binaries step for persistence across steps
- Changed git clone destination from /src to /workspace/src
- This ensures binaries are available to subsequent steps (build-image, create-github-release)
- Fixes exit code 127 failure in miroir-release workflow
2026-07-02 08:30:26 -04:00
jedarden
3adfcf788e fix: update workflow PVC storage size from 5Gi to 6G for Cinder API compatibility
The Rackspace Cinder API requires volume sizes between 5-20 GB.
The previous 5Gi specification was causing provisioning failures.
2026-07-02 08:19:32 -04:00
jedarden
f07ba9575b repo hygiene: remove committed build artifacts and stale config.bak
- Remove coverage/ directory (HTML and lcov files)
- Remove lcov.info and librust_out.rlib build artifacts
- Remove stray file '1' at repo root
- Remove dead config.bak/ module (unreferenced backup)
- Update .gitignore to exclude coverage/, lcov.info, and *.rlib patterns

Verified:
- No references to config.bak or librust_out in codebase
- cargo check --workspace compiles successfully
- notes/, .beads/, tests/, dashboards/ untouched
2026-07-02 07:46:48 -04:00
jedarden
7c5abf09b6 fix(ci): enable kafka-sink feature in CI build and Dockerfile
The kafka-sink Cargo feature existed but was not enabled in production builds,
causing all Kafka CDC events to be silently dropped at runtime.

Changes:
- Add --features miroir-core/kafka-sink to cargo-build in miroir-ci.yaml
- Update Dockerfile comments to reflect the expected build commands
- Add kafka_sink_feature.rs integration test with #[cfg(feature = "kafka-sink")]

The test verifies:
- Feature is enabled (compile-time check)
- CdcManager publish works with Kafka config
- Kafka sink config parses correctly

Fixes plan-gap: kafka-sink feature not enabled in CI build and Dockerfile

Bead-Id: bf-4v4rz
2026-05-31 12:08:39 -04:00
jedarden
252c9e93be fix(ci): use argo-workflow-executor serviceAccount per plan §7
Closes: bf-52l3
2026-05-25 08:28:58 -04:00
jedarden
475b7f0d73 feat(ci): sync miroir-ci workflow from declarative-config
Updated CI workflow includes:
- Helm chart packaging and publishing steps
- Updated tarpaulin coverage task
- Removed PR coverage comment (simplified)
- Added helm-package, helm-publish-ghpages, helm-publish-oci templates

Synced from jedarden/declarative-config to ensure consistency
across the fleet.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 18:35:56 -04:00
jedarden
184ca2bffe feat(ci): add HTML coverage output + PR comments for coverage delta (P9.1)
Updates the CI workflow to:

1. Add HTML coverage report output (plan §8 coverage policy)
   - Previously only generated Lcov + Xml formats
   - Now also outputs Html for browser-based viewing

2. Publish coverage reports as Argo artifacts
   - coverage-html/ directory for interactive browsing
   - cobertura.xml for CI tool integration
   - lcov.info for diff tools

3. Add PR comment showing coverage delta
   - Posts coverage percentage on PRs when revision != main
   - Shows current coverage vs 90% target vs base (main)
   - Includes link to full coverage artifact

4. Generate coverage summary file for PR comment consumption

The coverage gate (--fail-under 90) was already in place; this adds
the visibility (artifacts + PR comments) required by plan §8.

Closes: miroir-89x.1

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 17:02:05 -04:00
jedarden
440a05beb3 feat(k8s): add ArgoCD Application templates for GitOps deployments
Add example ArgoCD Application manifests and supporting files for
deploying Miroir via GitOps following plan §6 pattern.

Includes:
- Application template with placeholders for customization
- Production and development values.yaml examples
- Chart.yaml shim referencing upstream OCI chart
- ExternalSecret template for ESO integration
- Cluster-specific examples (ardenone-cluster, rs-manager)
- README with deployment instructions

Pattern: declarative-config repo holds per-instance values and Chart
shim; ArgoCD syncs from git path rather than inline values.

Closes: miroir-qjt.5
2026-05-24 08:34:24 -04:00
jedarden
77ff0b7528 Phase 6 — Horizontal Scaling + HPA (§14): Complete
Implements the full horizontal scaling architecture with HPA integration
and three coordination modes for background work partitioning.

## §14.1-§14.3 — Per-pod envelope
- Resource limits: 2000m CPU / 3584MiB RAM (2 vCPU / 3.75 GB)
- Memory budget validated for all §13 features
- CPU budget: ~3 kQPS/pod (small), ~1 kQPS/pod (large) at 70%

## §14.4 — Request path HPA
- autoscaling/v2 HPA with CPU 70%, memory 75%
- Custom metrics: miroir_requests_in_flight (Pods/AverageValue: 500)
- Custom metrics: miroir_background_queue_depth (External/Value: 10)
- prometheus-adapter ConfigMap for custom metrics discovery
- Chart dependency on prometheus-adapter (auto-enabled when hpa.enabled=true)
- values.schema.json Rule 2: HPA requires replicas >= 2 AND Redis backend

## §14.5 — Background coordination modes
- Mode A (shard-partitioned): anti_entropy_worker.rs, drift_reconciler.rs
- Mode B (leader-only): mode_b_coordinator.rs + leader_election/
- Mode C (work-queued): mode_c_coordinator.rs + mode_c_worker/
- Peer discovery via headless Service SRV records (15s refresh)

## §14.6 — Per-feature scaling mode wiring
- docs/horizontal-scaling/per-feature.md maps all 21 features to modes
- Forced-mode constraints in values.schema.json (Rules 0-5)

## §14.7 — Deployment sizing matrix
- docs/horizontal-scaling/sizing.md with workload tiers
- Task-store memory accounting for Redis-backed deployments

## §14.8 — Resource-aware configuration defaults
- charts/miroir/values.yaml with envelope-sized defaults
- tests/fixtures/section-14.8-defaults.yaml as reference

## §14.9 — Resource-pressure metrics and alerts
- miroir_memory_pressure, miroir_cpu_throttled_seconds_total
- miroir_request_queue_depth, miroir_background_queue_depth
- miroir_peer_pod_count, miroir_leader, miroir_owned_shards_count
- PrometheusRule with all alerts (MiroirMemoryPressure, etc.)

## §14.10 — Vertical-scaling escape valve
- docs/horizontal-scaling/single-pod.md documents single-pod mode
- tests/fixtures/section-14.10-single-pod-oversized.yaml with 2.13× multiplier

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 00:23:57 -04:00
jedarden
8378292238 Phase 8 CI: Separate bench-check step, rename artifacts→dist
- Split cargo-bench-check into dedicated template (plan §8 regression gate)
- Rename workspace/artifacts → workspace/dist (conventional naming)
- Move bench compilation after test (proper dependency ordering)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 23:50:04 -04:00
jedarden
603b17f708 P8.1: Simplify Dockerfile to scratch-only, update CI to use /workspace/artifacts/
Changes:
- Dockerfile: Remove multi-stage build, now expects pre-built miroir-proxy-linux-amd64
- Dockerfile: Add inline comment documenting the plan §7 cargo-build template
- CI workflow: Change /workspace/dist → /workspace/artifacts to match plan §7
- CI workflow: Update create-github-release to reference /workspace/artifacts

This aligns with plan §7 and §12: scratch base, no libc, minimal attack surface.
The CI builds the static musl binary separately, then Dockerfile copies it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 07:16:34 -04:00
jedarden
90462daa64 P5.5 §13.5: Fix drift_reconciler compilation and complete two-phase settings broadcast
Complete the two-phase settings broadcast with drift reconciler implementation:

- Fix drift_reconciler module compilation (remove unused imports, correct type signatures)
- Complete SettingsBroadcast integration in proxy layer (admin_endpoints.rs)
- Add settings version tracking metrics (middleware.rs)
- Initialize drift_reconciler worker in main.rs
- Fix admin route registration (admin.rs, aliases.rs)

Acceptance tests verify:
1. Normal flow: propose+verify succeed, settings_version increments once
2. Mid-broadcast node failure: reissue succeeds after backoff
3. Out-of-band drift: reconciler detects and repairs within interval_s
4. X-Miroir-Min-Settings-Version floor excludes stale nodes
5. Legacy sequential strategy compatibility

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-22 18:10:10 -04:00
jedarden
9ba6d545ca P11.7: Add quick-start example artifacts (Docker Compose + config)
Adds the on-disk examples referenced by plan §11 "Quick start (local, Docker Compose)":

- examples/docker-compose-dev.yml: 3 Meilisearch nodes + 1 Miroir orchestrator
- examples/dev-config.yaml: Matching Miroir config (16 shards, RF=1)
- examples/README.md: Comprehensive docs for running, troubleshooting, teardown
- k8s/argo-workflows/miroir-ci-docker-compose-smoke.yaml: CI smoke tests

The README.md quick start section already references these examples.

Acceptance:
 docker-compose-dev.yml boots via docker compose up
 dev-config.yaml mounted into Miroir container
 examples/README.md documents usage and teardown
 CI smoke job exercises compose stack (health + index + search tests)
 README.md quick start points to examples/docker-compose-dev.yml

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-20 06:49:05 -04:00
jedarden
5bec6e2bf3 P12: close Phase 12 epic — all 6 open problems triaged and documented
OP#1 (shard migration write safety): chaos-test scope documented; anti-entropy
as the mitigation is complete. Bead miroir-zc2.1 closed.

OP#2 (Raft vs Redis): full crate survey + prototype + benchmark. Decision:
Redis wins, revisit before v2.0. Bead miroir-zc2.2 closed; docs in
docs/research/raft-task-store.md.

OP#3 (resharding 2× load): benchmark confirms 2.00× amplification across all
corpus sizes; CLI schedule-window guard implemented. Bead miroir-zc2.3 closed;
docs in docs/benchmarks/resharding-load.md.

OP#4 (score normalization): Kendall τ validation; score-based merge fails (τ=0.79),
RRF fails (τ=0.14), DFS preflight passes (τ=0.98). Bead miroir-zc2.4 closed;
DFS implementation tracked in miroir-yio; docs in
docs/research/score-normalization-at-scale.md.

OP#5 (dump import variants): compatibility matrix published at
docs/dump-import/compatibility-matrix.md. Bead miroir-zc2.5 closed.

OP#6 (arm64): deferred to v1.x+. Implementation roadmap expanded in
docs/plan/plan.md (commit 7f03fe6). Bead miroir-zc2.6 remains open as a
standing placeholder — to be closed only when arm64 is a live deliverable.

Also: minor unused-variable warning fixes in task_registry.rs, redis.rs,
sqlite.rs; add k8s/openbao-policy.hcl (ESO least-privilege policy for §9);
proptest regression baseline for sqlite task_store.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 19:14:23 -04:00
jedarden
ffe1d63d58 P8: Finalize CI/CD templates, prod ArgoCD app, and CHANGELOG for v0.1.0
- miroir-ci: use cargo fmt --all, add pre-release detection for GitHub releases
- miroir-ci-smoke: fix secret ref to github-token
- miroir-release: rewrite github-release step with gh CLI, build binaries in
  release step, add pre-release flag and resource limits
- miroir-release-ready: fix serviceAccountName to argo-workflow
- miroir-application.yaml: switch prod to Redis backend, 4 Meilisearch replicas
- redis.rs: remove unused conn() helper
- CHANGELOG: date 0.1.0 release, add missing release/prod entries

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 15:09:14 -04:00
jedarden
dcab90d2c9 Add prod ArgoCD Application manifest for ardenone-cluster
Matches the manifest already in declarative-config (commit 3d72934).
OCI Helm chart at ghcr.io/jedarden/charts/miroir, automated sync
with prune + selfHeal + ServerSideApply.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 14:36:31 -04:00
jedarden
42905066cf P8: Fix miroir-release Kaniko build — use shell script instead of sprig expressions
Replace sprig regex template expressions with a shell script approach for
Kaniko destination tags, matching the pattern in miroir-ci.yaml. Pin Kaniko
image to v1.23.0-debug. Fix serviceAccountName from argo-runner to argo-workflow.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 14:33:51 -04:00
jedarden
9b2f11f71b P8: Sync CI/CD templates and ArgoCD Application to miroir repo (plan §6/§7)
Adds miroir-ci WorkflowTemplate (checkout → lint → test → musl build →
Kaniko push + GitHub release, tag-gated), miroir-ci-smoke quick lint+test
template, and miroir-dev ArgoCD Application reference. Updates CHANGELOG.md
with Phase 8 deployment entries.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-19 13:28:07 -04:00
jedarden
2dcfae8822 P8.6: Release mechanics — bump script, release-ready check, PR template, Argo CIs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-19 09:54:26 -04:00