diff --git a/crates/miroir-ctl/src/main.rs b/crates/miroir-ctl/src/main.rs index 150360b..28716e0 100644 --- a/crates/miroir-ctl/src/main.rs +++ b/crates/miroir-ctl/src/main.rs @@ -6,7 +6,15 @@ mod credentials; #[derive(Parser)] #[command(name = "miroir-ctl")] -#[command(about = "Miroir management CLI", long_about = None)] +#[command( + about = "Miroir management CLI", + long_about = "Miroir management CLI + +Runbook documentation for each subcommand is available at: + https://github.com/jedarden/miroir/tree/main/docs/ctl/ + +For local docs, see docs/ctl/*.md in the repository." +)] struct Cli { #[command(subcommand)] command: Commands, diff --git a/docs/ctl/README.md b/docs/ctl/README.md new file mode 100644 index 0000000..8e1294c --- /dev/null +++ b/docs/ctl/README.md @@ -0,0 +1,67 @@ +# miroir-ctl Runbooks + +This directory contains runbooks for each `miroir-ctl` subcommand. Each runbook includes: + +- Purpose (one sentence) +- Preconditions (what must be true before running) +- Examples (common invocations) +- Gotchas (edge cases and warnings) +- See also (plan section references) + +## Runbooks + +| Command | Runbook | Description | +|---------|---------|-------------| +| `status` | [status.md](status.md) | Show cluster health and node status | +| `node` | [node.md](node.md) | Add, remove, drain, and list nodes | +| `rebalance` | [rebalance.md](rebalance.md) | Monitor shard migrations | +| `reshard` | [reshard.md](reshard.md) | Change index shard count | +| `verify` | [verify.md](verify.md) | Check data integrity across replicas | +| `task` | [task.md](task.md) | Monitor background tasks | +| `dump` | [dump.md](dump.md) | Export and import index data | +| `alias` | [alias.md](alias.md) | Manage index aliases | +| `canary` | [canary.md](canary.md) | Control canary deployments | +| `ttl` | [ttl.md](ttl.md) | Manage document expiration policies | +| `cdc` | [cdc.md](cdc.md) | Configure change data capture | +| `shadow` | [shadow.md](shadow.md) | Manage shadow indexing | +| `ui` | [ui.md](ui.md) | Launch the Admin UI | +| `tenant` | [tenant.md](tenant.md) | Manage multi-tenancy | +| `explain` | [explain.md](explain.md) | Analyze query plans | +| `key` | [key.md](key.md) | Manage API keys | + +## Quick Reference + +```bash +# Cluster health +miroir-ctl status --watch + +# Node operations +miroir-ctl node add --id node-3 --address http://node-3:7700 --replica-group 0 +miroir-ctl node drain node-2 +miroir-ctl node remove node-2 + +# Monitor operations +miroir-ctl rebalance status --watch +miroir-ctl task status + +# Data operations +miroir-ctl dump export --index myindex --output myindex.dump +miroir-ctl verify --index myindex + +# Admin UI +miroir-ctl ui +``` + +## Authentication + +All commands require an admin API key. Set it via: + +1. Environment variable: `export MIROIR_ADMIN_API_KEY=...` +2. Credentials file: `~/.config/miroir/credentials` with `[default].admin_api_key` +3. Command line flag: `--admin-key ...` (WARNING: visible in process list) + +## See also + +- [Plan §11](../plan/plan.md#11-onboarding) — common operations +- [Plan §4](../plan/plan.md#4-implementation) — crate layout +- [Troubleshooting](../troubleshooting.md) — common issues diff --git a/docs/ctl/alias.md b/docs/ctl/alias.md new file mode 100644 index 0000000..9029f92 --- /dev/null +++ b/docs/ctl/alias.md @@ -0,0 +1,36 @@ +# `miroir-ctl alias` + +## Purpose +Manage index aliases for zero-downtime index swaps (e.g., for resharding or blue-green deployments). + +## Preconditions +- Source and target indexes must exist +- Admin API key configured + +## Examples + +```bash +# Create an alias pointing to an index +miroir-ctl alias create --name prod --target myindex_v1 + +# List all aliases +miroir-ctl alias list + +# Update an alias to point to a new index (zero-downtime swap) +miroir-ctl alias update --name prod --target myindex_v2 + +# Delete an alias +miroir-ctl alias delete --name prod +``` + +## Gotchas +- **Not yet fully implemented** — see bead miroir-qon for tracking +- Aliases are resolved at query time — no data is copied +- Use aliases for A/B testing: same queries hit different indexes +- Resharding uses aliases internally for the final swap (see `miroir-ctl reshard`) +- Deleting an alias does not delete the underlying index + +## See also +- Plan §13.7 — alias management and atomic swaps +- Plan §13.1 — resharding alias swap phase +- `miroir-ctl reshard` — uses aliases for zero-downtime migration diff --git a/docs/ctl/canary.md b/docs/ctl/canary.md new file mode 100644 index 0000000..f6ad46d --- /dev/null +++ b/docs/ctl/canary.md @@ -0,0 +1,38 @@ +# `miroir-ctl canary` + +## Purpose +Control canary deployments for rolling updates and validation. + +## Preconditions +- Admin API key configured +- Canary feature enabled in config + +## Examples + +```bash +# Start a canary deployment (10% of traffic to new version) +miroir-ctl canary start --version v2.0.0 --percentage 10 + +# Increase canary traffic +miroir-ctl canary set --percentage 25 + +# Check canary status +miroir-ctl canary status + +# Promote canary to 100% (complete rollout) +miroir-ctl canary promote + +# Abort canary and rollback +miroir-ctl canary abort +``` + +## Gotchas +- **Not yet implemented** — see tracking bead for details +- Canary splits traffic at the orchestrator level — not per-node +- Metrics are compared between canary and baseline — automatic abort on regression +- Use `miroir-ctl status` to monitor during canary +- Aborting restores previous routing immediately + +## See also +- Plan §13.18 — canary deployment architecture +- Plan §10 — metrics and observability for canary validation diff --git a/docs/ctl/cdc.md b/docs/ctl/cdc.md new file mode 100644 index 0000000..b0a8c3a --- /dev/null +++ b/docs/ctl/cdc.md @@ -0,0 +1,44 @@ +# `miroir-ctl cdc` + +## Purpose +Configure change data capture (CDC) for streaming index changes to external systems. + +## Preconditions +- External sink configured (Kafka, webhook, etc.) +- CDC feature enabled in config + +## Examples + +```bash +# Enable CDC for an index to a Kafka topic +miroir-ctl cdc create --index myindex --sink kafka:mytopic --format json + +# Enable CDC to a webhook endpoint +miroir-ctl cdc create --index myindex --sink webhook:https://example.com/cdc --format ndjson + +# List CDC streams +miroir-ctl cdc list + +# Get CDC stream status and lag +miroir-ctl cdc status --index myindex + +# Pause a CDC stream +miroir-ctl cdc pause --index myindex + +# Resume a paused stream +miroir-ctl cdc resume --index myindex + +# Delete a CDC stream +miroir-ctl cdc delete --index myindex +``` + +## Gotchas +- **Not yet implemented** — see tracking bead for details +- CDC captures writes, updates, and deletes — not reads +- Lag can build up if sink is slow — check `cdc status` regularly +- Pausing a stream stops delivery but doesn't lose events — they buffer until resume +- Webhook sinks must return 200 OK — retries with exponential backoff on failure + +## See also +- Plan §13.13 — CDC architecture and sink types +- Plan §10 — CDC metrics and monitoring diff --git a/docs/ctl/dump.md b/docs/ctl/dump.md new file mode 100644 index 0000000..1621aa3 --- /dev/null +++ b/docs/ctl/dump.md @@ -0,0 +1,39 @@ +# `miroir-ctl dump` + +## Purpose +Export and import index data for backup, migration, or bulk loading. + +## Preconditions +- For export: index must exist and be healthy +- For import: target index must exist with matching schema + +## Examples + +```bash +# Export an index to a compressed dump file +miroir-ctl dump export --index myindex --output myindex.dump + +# Export with chunking (for large indexes) +miroir-ctl dump export --index myindex --output myindex.dump --chunk-size 10000 + +# Import a dump file +miroir-ctl dump import --input myindex.dump --index myindex + +# Import with throttling (limit write rate) +miroir-ctl dump import --input myindex.dump --index myindex --throttle 5000 + +# Verify dump integrity without importing +miroir-ctl dump verify --input myindex.dump +``` + +## Gotchas +- Export creates a compressed archive — includes documents, settings, and tasks +- Import is idempotent — running twice won't duplicate documents (PK-based upsert) +- Large dumps are chunked automatically — use `--chunk-size` to control memory usage +- Import bypasses the write path's sharding layer — writes directly to target shards +- Use `miroir-ctl task status` to track async import jobs + +## See also +- Plan §13.9 — dump format and chunking +- `miroir-ctl verify` — verify imported data +- `docs/dump-import/` — detailed dump format specification diff --git a/docs/ctl/explain.md b/docs/ctl/explain.md new file mode 100644 index 0000000..ffbad58 --- /dev/null +++ b/docs/ctl/explain.md @@ -0,0 +1,35 @@ +# `miroir-ctl explain` + +## Purpose +Analyze query plans and show how search queries are routed across shards. + +## Preconditions +- Index must exist +- Admin API key configured + +## Examples + +```bash +# Explain a simple search query +miroir-ctl explain --index myindex --q "test query" + +# Explain with filters +miroir-ctl explain --index logs --q "error" --filter "timestamp > 2024-01-01" + +# Explain with ranking details +miroir-ctl explain --index myindex --q "test" --show-ranking + +# Show shard routing for a query +miroir-ctl explain --index myindex --q "test" --show-routing +``` + +## Gotchas +- **Not yet implemented** — see tracking bead for details +- Explain doesn't execute the query — it only shows the plan +- Use `--show-routing` to understand which nodes are contacted +- Ranking details show score breakdown per field +- Explain is useful for debugging slow queries — check per-shard latency + +## See also +- Plan §13.20 — query planning and optimization +- Plan §2 — rendezvous hashing and shard routing diff --git a/docs/ctl/key.md b/docs/ctl/key.md new file mode 100644 index 0000000..a7edf2d --- /dev/null +++ b/docs/ctl/key.md @@ -0,0 +1,48 @@ +# `miroir-ctl key` + +## Purpose +Manage Meilisearch API keys with scoped permissions and tenant isolation. + +## Preconditions +- Admin API key configured +- Key management enabled in config + +## Examples + +```bash +# Create a search-only key for a tenant +miroir-ctl key create --name acme-search --action search --index acme-* + +# Create a key with document write permissions +miroir-ctl key create --name writer --action documentsAdd,indexUpdate --index logs + +# Create a key with expiration +miroir-ctl key create --name temp --action search --expires 2024-12-31 + +# Create a scoped key (UI-only operation, see Plan §13.21) +miroir-ctl key create-scoped --name ui-key --tenant acme + +# List all keys +miroir-ctl key list + +# Get key details (including usage stats) +miroir-ctl key get --name acme-search + +# Delete a key +miroir-ctl key delete --name acme-search + +# Rotate the master key (UI-only, see Plan §13.21) +miroir-ctl key rotate-master +``` + +## Gotchas +- **Partially implemented** — basic CRUD works, scoped keys are UI-only +- Keys are returned once at creation — save the key value immediately +- Deleting a key revokes access immediately — no grace period +- Scoped keys embed tenant and index filters — verify before use +- Master key rotation requires downtime — schedule maintenance window + +## See also +- Plan §13.21 — scoped key rotation and master key management +- Plan §9 — security and key scoping +- Admin UI — preferred interface for key rotation (not CLI) diff --git a/docs/ctl/node.md b/docs/ctl/node.md new file mode 100644 index 0000000..47382a7 --- /dev/null +++ b/docs/ctl/node.md @@ -0,0 +1,39 @@ +# `miroir-ctl node` + +## Purpose +Manage cluster topology — add, remove, drain, and list nodes. + +## Preconditions +- Target node must be running and reachable from the orchestrator +- Node's Meilisearch instance should be healthy before adding +- For removal: node should be drained first to avoid data loss + +## Examples + +```bash +# Add a new node to replica group 0 +miroir-ctl node add --id node-3 --address http://node-3:7700 --replica-group 0 + +# List all nodes with status +miroir-ctl node list + +# Drain a node before removal (migrates its shards elsewhere) +miroir-ctl node drain node-2 + +# Remove a node after draining completes +miroir-ctl node remove node-2 + +# Force remove (dangerous — skips drain, may lose data) +miroir-ctl node remove node-2 --force --yes +``` + +## Gotchas +- `node add` triggers automatic rebalancing — use `miroir-ctl rebalance status --watch` to track +- `node drain` is async — the command returns immediately but migrations continue in background +- Never use `--force` unless the node is permanently dead — you'll lose RF replicas +- `node list` shows the same info as `miroir-ctl status` but in table format + +## See also +- Plan §13.2 — node addition and rebalancing +- Plan §13.3 — node drain and removal +- `miroir-ctl rebalance status` — track migration progress diff --git a/docs/ctl/rebalance.md b/docs/ctl/rebalance.md new file mode 100644 index 0000000..371a001 --- /dev/null +++ b/docs/ctl/rebalance.md @@ -0,0 +1,29 @@ +# `miroir-ctl rebalance` + +## Purpose +Monitor and manage shard migrations triggered by topology changes. + +## Preconditions +- Rebalancing runs automatically after `node add` / `node drain` — no manual start needed +- Admin API key for status queries + +## Examples + +```bash +# Show current rebalance progress +miroir-ctl rebalance status + +# Watch migrations in real-time (use during node operations) +miroir-ctl rebalance status --watch +``` + +## Gotchas +- Rebalancing is automatic — there's no `rebalance start` command +- Migrations are rate-limited to avoid overwhelming nodes — large topology changes may take hours +- Each migration moves one shard from source to destination; a node add triggers ~S/Ng migrations +- Queries continue serving during rebalance — old and new locations are both valid + +## See also +- Plan §13.2 — automatic rebalancing on topology change +- Plan §13.3 — drain-triggered rebalancing +- `miroir-ctl status --watch` — cluster-wide health during migrations diff --git a/docs/ctl/reshard.md b/docs/ctl/reshard.md new file mode 100644 index 0000000..c341dfe --- /dev/null +++ b/docs/ctl/reshard.md @@ -0,0 +1,41 @@ +# `miroir-ctl reshard` + +## Purpose +Change an index's shard count (S) without downtime. Requires full backfill. + +## Preconditions +- Index must exist with sufficient data to warrant resharding +- Resharding must be enabled in config (`resharding.enabled = true`) +- Schedule window configured (or `--force` used) +- Target shard count should be `max_nodes_per_group_ever × 8` — choose generously + +## Examples + +```bash +# Dry run: see what would happen +miroir-ctl reshard start --index myindex --new-shards 128 --dry-run + +# Start resharding within schedule window +miroir-ctl reshard start --index myindex --new-shards 128 --schedule-window off-peak + +# Start with throttled backfill (5000 docs/sec) +miroir-ctl reshard start --index myindex --new-shards 128 --throttle 5000 + +# Force start outside schedule window (not recommended) +miroir-ctl reshard start --index myindex --new-shards 128 --force + +# Check resharding progress +miroir-ctl reshard status --index myindex +``` + +## Gotchas +- Resharding creates a shadow index (`index__reshard_N`) — doubles storage during operation +- Backfill can take hours for large indexes — use `--throttle` to limit load +- The old index is retained for 48h (configurable) before cleanup +- Verify runs before swap — any mismatch aborts the operation +- This is **not** for adding nodes — use `miroir-ctl node add` for that + +## See also +- Plan §13.1 — online resharding architecture +- `~/.config/miroir/config.toml` — `[resharding]` section for windows and throttling +- Admin UI (§13.19) — alternative to CLI for one-off operations diff --git a/docs/ctl/shadow.md b/docs/ctl/shadow.md new file mode 100644 index 0000000..0e6e38e --- /dev/null +++ b/docs/ctl/shadow.md @@ -0,0 +1,35 @@ +# `miroir-ctl shadow` + +## Purpose +Manage shadow indexing for A/B testing and validation without affecting production traffic. + +## Preconditions +- Source index exists and is healthy +- Sufficient capacity for shadow writes (2x write amplification) + +## Examples + +```bash +# Create a shadow index for testing +miroir-ctl shadow create --source prod --shadow test --sync-writes + +# Enable query shadowing (mirror queries to both, compare results) +miroir-ctl shadow query --source prod --shadow test --compare + +# Check shadow index lag +miroir-ctl shadow status --source prod + +# Stop shadowing and delete shadow index +miroir-ctl shadow delete --source prod +``` + +## Gotchas +- **Not yet implemented** — see tracking bead for details +- Shadow writes are synchronous — adds latency to production writes +- Query shadowing is asynchronous — doesn't affect production latency +- Use shadow indexing for schema validation, not load testing (write amplification) +- Delete shadow indexes after testing to free storage + +## See also +- Plan §13.16 — shadow indexing architecture +- `miroir-ctl verify` — compare shadow and source results diff --git a/docs/ctl/status.md b/docs/ctl/status.md new file mode 100644 index 0000000..5dd9cfd --- /dev/null +++ b/docs/ctl/status.md @@ -0,0 +1,27 @@ +# `miroir-ctl status` + +## Purpose +Show cluster health, node status, and ongoing rebalancing operations. + +## Preconditions +- Admin API key configured (`MIROIR_ADMIN_API_KEY` env var, `~/.config/miroir/credentials`, or `--admin-key` flag) +- Miroir orchestrator reachable at `--api-url` (default: `http://localhost:8080`) + +## Examples + +```bash +# One-time status snapshot +miroir-ctl status + +# Continuously refresh every 2 seconds (use during operations) +miroir-ctl status --watch +``` + +## Gotchas +- `--watch` mode clears the terminal each refresh — use `Ctrl+C` to exit +- Degraded nodes show `⚠` but don't necessarily mean downtime — check individual node errors +- `fully_covered: false` means some shards have fewer than RF replicas — check `degraded_node_count` + +## See also +- Plan §10 — health metrics and degraded node detection +- `miroir-ctl node list` — detailed node topology diff --git a/docs/ctl/task.md b/docs/ctl/task.md new file mode 100644 index 0000000..02dd48a --- /dev/null +++ b/docs/ctl/task.md @@ -0,0 +1,34 @@ +# `miroir-ctl task` + +## Purpose +Monitor and manage background tasks (resharding, rebalancing, verification, etc.). + +## Preconditions +- Admin API key configured + +## Examples + +```bash +# List all background tasks +miroir-ctl task list + +# Show status of a specific task +miroir-ctl task status --id abc123 + +# Show status of all tasks (no ID filter) +miroir-ctl task status + +# Cancel a running task +miroir-ctl task cancel --id abc123 +``` + +## Gotchas +- **Not yet implemented** — see bead miroir-qon for tracking +- Task IDs are returned by async commands (`node add`, `reshard start`, etc.) +- Cancellation is best-effort — some tasks complete their current unit before stopping +- Tasks that fail are retried automatically — check `task status` for retry count + +## See also +- Plan §13 — task management across operations +- `miroir-ctl rebalance status` — shortcut for rebalancing tasks +- `miroir-ctl reshard status` — shortcut for resharding tasks diff --git a/docs/ctl/tenant.md b/docs/ctl/tenant.md new file mode 100644 index 0000000..a95f75a --- /dev/null +++ b/docs/ctl/tenant.md @@ -0,0 +1,38 @@ +# `miroir-ctl tenant` + +## Purpose +Manage multi-tenancy by isolating indexes and API keys per tenant. + +## Preconditions +- Admin API key configured +- Multi-tenancy enabled in config + +## Examples + +```bash +# Create a new tenant +miroir-ctl tenant create --name acme --quota indexes=10,docs=1000000 + +# List all tenants +miroir-ctl tenant list + +# Get tenant details and quota usage +miroir-ctl tenant get --name acme + +# Update tenant quota +miroir-ctl tenant update --name acme --quota indexes=20 + +# Delete a tenant (and all its indexes) +miroir-ctl tenant delete --name acme +``` + +## Gotchas +- **Not yet implemented** — see tracking bead for details +- Tenant isolation is logical — all data lives in the same cluster +- Quota enforcement is best-effort — brief overages are possible during batch operations +- Deleting a tenant deletes all its indexes — this is irreversible +- API keys are scoped per tenant — cross-tenant access is impossible + +## See also +- Plan §13.15 — multi-tenancy architecture +- Plan §9 — security and key scoping diff --git a/docs/ctl/ttl.md b/docs/ctl/ttl.md new file mode 100644 index 0000000..ab4eaa0 --- /dev/null +++ b/docs/ctl/ttl.md @@ -0,0 +1,35 @@ +# `miroir-ctl ttl` + +## Purpose +Manage time-to-live (TTL) policies for automatic document expiration. + +## Preconditions +- Index must exist with a timestamp field +- TTL feature enabled in config + +## Examples + +```bash +# Set a TTL policy on an index (documents expire after 30 days) +miroir-ctl ttl set --index logs --field created_at --duration 30d + +# Set TTL with custom check interval +miroir-ctl ttl set --index logs --field created_at --duration 7d --check-interval 1h + +# Get current TTL policy for an index +miroir-ctl ttl get --index logs + +# Remove TTL policy +miroir-ctl ttl remove --index logs +``` + +## Gotchas +- **Not yet implemented** — see tracking bead for details +- TTL is enforced via background tasks — expired documents are deleted on next pass +- Duration format: `30d` (days), `12h` (hours), `15m` (minutes) +- TTL field must be a ISO 8601 timestamp or Unix epoch +- Deleting expired documents is irreversible — backup first if needed + +## See also +- Plan §13.14 — TTL implementation and task scheduling +- `miroir-ctl task status` — monitor TTL cleanup jobs diff --git a/docs/ctl/ui.md b/docs/ctl/ui.md new file mode 100644 index 0000000..32d38b9 --- /dev/null +++ b/docs/ctl/ui.md @@ -0,0 +1,36 @@ +# `miroir-ctl ui` + +## Purpose +Launch the web Admin UI for interactive cluster management. + +## Preconditions +- Admin API key configured +- Browser available (opens localhost:3000 by default) + +## Examples + +```bash +# Launch the Admin UI +miroir-ctl ui + +# Launch on a custom port +miroir-ctl ui --port 8080 + +# Launch without auto-opening browser +miroir-ctl ui --no-open + +# Launch with specific API endpoint +miroir-ctl ui --api-url https://miroir.example.com +``` + +## Gotchas +- UI runs locally — it makes API calls to the orchestrator from your browser +- Credentials are stored in browser local storage — clear after use on shared machines +- Most UI operations have CLI equivalents — prefer CLI for scripts and CI/CD +- UI requires CSRF token exchange — first load may be slower +- Some advanced operations (e.g., scoped key rotation) are UI-only + +## See also +- Plan §13.19 — Admin UI features and architecture +- Plan §13.21 — scoped key rotation (UI-only operation) +- `docs/ctl/*.md` — CLI equivalents for UI operations diff --git a/docs/ctl/verify.md b/docs/ctl/verify.md new file mode 100644 index 0000000..e251c8d --- /dev/null +++ b/docs/ctl/verify.md @@ -0,0 +1,31 @@ +# `miroir-ctl verify` + +## Purpose +Check data integrity across replica groups and detect drift. + +## Preconditions +- Admin API key configured +- Cluster healthy (degraded nodes can cause false positives) + +## Examples + +```bash +# Verify all indexes +miroir-ctl verify + +# Verify specific index +miroir-ctl verify --index myindex + +# Verbose output (shows per-shard details) +miroir-ctl verify --verbose +``` + +## Gotchas +- Verification is read-only — it won't fix drift, only report it +- Large indexes take time to verify — progress is not streamed +- Use `miroir-ctl task status` to check async verification jobs +- Drift is automatically repaired by anti-entropy workers — see `miroir-ctl task status` + +## See also +- Plan §13.8 — anti-entropy and drift detection +- Plan §10 — observability and metrics