docs(miroir-ctl): add subcommand runbooks and help text (P11.4, miroir-uyx.4)

- Created docs/ctl/*.md runbooks for all 16 miroir-ctl subcommands
- Each runbook includes: purpose, preconditions, examples, gotchas, see also
- Added runbook location to --help output
- All runbooks under 50 lines for easy reading

Closes: miroir-uyx.4

crates/miroir-ctl/src/main.rs

@@ -6,7 +6,15 @@ mod credentials;
 
 #[derive(Parser)]
 #[command(name = "miroir-ctl")]
-#[command(about = "Miroir management CLI", long_about = None)]
+#[command(
+    about = "Miroir management CLI",
+    long_about = "Miroir management CLI
+
+Runbook documentation for each subcommand is available at:
+  https://github.com/jedarden/miroir/tree/main/docs/ctl/
+
+For local docs, see docs/ctl/*.md in the repository."
+)]
 struct Cli {
     #[command(subcommand)]
     command: Commands,

docs/ctl/README.md

@@ -0,0 +1,67 @@
+# miroir-ctl Runbooks
+
+This directory contains runbooks for each `miroir-ctl` subcommand. Each runbook includes:
+
+- Purpose (one sentence)
+- Preconditions (what must be true before running)
+- Examples (common invocations)
+- Gotchas (edge cases and warnings)
+- See also (plan section references)
+
+## Runbooks
+
+| Command | Runbook | Description |
+|---------|---------|-------------|
+| `status` | [status.md](status.md) | Show cluster health and node status |
+| `node` | [node.md](node.md) | Add, remove, drain, and list nodes |
+| `rebalance` | [rebalance.md](rebalance.md) | Monitor shard migrations |
+| `reshard` | [reshard.md](reshard.md) | Change index shard count |
+| `verify` | [verify.md](verify.md) | Check data integrity across replicas |
+| `task` | [task.md](task.md) | Monitor background tasks |
+| `dump` | [dump.md](dump.md) | Export and import index data |
+| `alias` | [alias.md](alias.md) | Manage index aliases |
+| `canary` | [canary.md](canary.md) | Control canary deployments |
+| `ttl` | [ttl.md](ttl.md) | Manage document expiration policies |
+| `cdc` | [cdc.md](cdc.md) | Configure change data capture |
+| `shadow` | [shadow.md](shadow.md) | Manage shadow indexing |
+| `ui` | [ui.md](ui.md) | Launch the Admin UI |
+| `tenant` | [tenant.md](tenant.md) | Manage multi-tenancy |
+| `explain` | [explain.md](explain.md) | Analyze query plans |
+| `key` | [key.md](key.md) | Manage API keys |
+
+## Quick Reference
+
+```bash
+# Cluster health
+miroir-ctl status --watch
+
+# Node operations
+miroir-ctl node add --id node-3 --address http://node-3:7700 --replica-group 0
+miroir-ctl node drain node-2
+miroir-ctl node remove node-2
+
+# Monitor operations
+miroir-ctl rebalance status --watch
+miroir-ctl task status
+
+# Data operations
+miroir-ctl dump export --index myindex --output myindex.dump
+miroir-ctl verify --index myindex
+
+# Admin UI
+miroir-ctl ui
+```
+
+## Authentication
+
+All commands require an admin API key. Set it via:
+
+1. Environment variable: `export MIROIR_ADMIN_API_KEY=...`
+2. Credentials file: `~/.config/miroir/credentials` with `[default].admin_api_key`
+3. Command line flag: `--admin-key ...` (WARNING: visible in process list)
+
+## See also
+
+- [Plan §11](../plan/plan.md#11-onboarding) — common operations
+- [Plan §4](../plan/plan.md#4-implementation) — crate layout
+- [Troubleshooting](../troubleshooting.md) — common issues

docs/ctl/alias.md

@@ -0,0 +1,36 @@
+# `miroir-ctl alias`
+
+## Purpose
+Manage index aliases for zero-downtime index swaps (e.g., for resharding or blue-green deployments).
+
+## Preconditions
+- Source and target indexes must exist
+- Admin API key configured
+
+## Examples
+
+```bash
+# Create an alias pointing to an index
+miroir-ctl alias create --name prod --target myindex_v1
+
+# List all aliases
+miroir-ctl alias list
+
+# Update an alias to point to a new index (zero-downtime swap)
+miroir-ctl alias update --name prod --target myindex_v2
+
+# Delete an alias
+miroir-ctl alias delete --name prod
+```
+
+## Gotchas
+- **Not yet fully implemented** — see bead miroir-qon for tracking
+- Aliases are resolved at query time — no data is copied
+- Use aliases for A/B testing: same queries hit different indexes
+- Resharding uses aliases internally for the final swap (see `miroir-ctl reshard`)
+- Deleting an alias does not delete the underlying index
+
+## See also
+- Plan §13.7 — alias management and atomic swaps
+- Plan §13.1 — resharding alias swap phase
+- `miroir-ctl reshard` — uses aliases for zero-downtime migration

docs/ctl/canary.md

@@ -0,0 +1,38 @@
+# `miroir-ctl canary`
+
+## Purpose
+Control canary deployments for rolling updates and validation.
+
+## Preconditions
+- Admin API key configured
+- Canary feature enabled in config
+
+## Examples
+
+```bash
+# Start a canary deployment (10% of traffic to new version)
+miroir-ctl canary start --version v2.0.0 --percentage 10
+
+# Increase canary traffic
+miroir-ctl canary set --percentage 25
+
+# Check canary status
+miroir-ctl canary status
+
+# Promote canary to 100% (complete rollout)
+miroir-ctl canary promote
+
+# Abort canary and rollback
+miroir-ctl canary abort
+```
+
+## Gotchas
+- **Not yet implemented** — see tracking bead for details
+- Canary splits traffic at the orchestrator level — not per-node
+- Metrics are compared between canary and baseline — automatic abort on regression
+- Use `miroir-ctl status` to monitor during canary
+- Aborting restores previous routing immediately
+
+## See also
+- Plan §13.18 — canary deployment architecture
+- Plan §10 — metrics and observability for canary validation

docs/ctl/cdc.md

@@ -0,0 +1,44 @@
+# `miroir-ctl cdc`
+
+## Purpose
+Configure change data capture (CDC) for streaming index changes to external systems.
+
+## Preconditions
+- External sink configured (Kafka, webhook, etc.)
+- CDC feature enabled in config
+
+## Examples
+
+```bash
+# Enable CDC for an index to a Kafka topic
+miroir-ctl cdc create --index myindex --sink kafka:mytopic --format json
+
+# Enable CDC to a webhook endpoint
+miroir-ctl cdc create --index myindex --sink webhook:https://example.com/cdc --format ndjson
+
+# List CDC streams
+miroir-ctl cdc list
+
+# Get CDC stream status and lag
+miroir-ctl cdc status --index myindex
+
+# Pause a CDC stream
+miroir-ctl cdc pause --index myindex
+
+# Resume a paused stream
+miroir-ctl cdc resume --index myindex
+
+# Delete a CDC stream
+miroir-ctl cdc delete --index myindex
+```
+
+## Gotchas
+- **Not yet implemented** — see tracking bead for details
+- CDC captures writes, updates, and deletes — not reads
+- Lag can build up if sink is slow — check `cdc status` regularly
+- Pausing a stream stops delivery but doesn't lose events — they buffer until resume
+- Webhook sinks must return 200 OK — retries with exponential backoff on failure
+
+## See also
+- Plan §13.13 — CDC architecture and sink types
+- Plan §10 — CDC metrics and monitoring

docs/ctl/dump.md

@@ -0,0 +1,39 @@
+# `miroir-ctl dump`
+
+## Purpose
+Export and import index data for backup, migration, or bulk loading.
+
+## Preconditions
+- For export: index must exist and be healthy
+- For import: target index must exist with matching schema
+
+## Examples
+
+```bash
+# Export an index to a compressed dump file
+miroir-ctl dump export --index myindex --output myindex.dump
+
+# Export with chunking (for large indexes)
+miroir-ctl dump export --index myindex --output myindex.dump --chunk-size 10000
+
+# Import a dump file
+miroir-ctl dump import --input myindex.dump --index myindex
+
+# Import with throttling (limit write rate)
+miroir-ctl dump import --input myindex.dump --index myindex --throttle 5000
+
+# Verify dump integrity without importing
+miroir-ctl dump verify --input myindex.dump
+```
+
+## Gotchas
+- Export creates a compressed archive — includes documents, settings, and tasks
+- Import is idempotent — running twice won't duplicate documents (PK-based upsert)
+- Large dumps are chunked automatically — use `--chunk-size` to control memory usage
+- Import bypasses the write path's sharding layer — writes directly to target shards
+- Use `miroir-ctl task status` to track async import jobs
+
+## See also
+- Plan §13.9 — dump format and chunking
+- `miroir-ctl verify` — verify imported data
+- `docs/dump-import/` — detailed dump format specification

docs/ctl/explain.md

@@ -0,0 +1,35 @@
+# `miroir-ctl explain`
+
+## Purpose
+Analyze query plans and show how search queries are routed across shards.
+
+## Preconditions
+- Index must exist
+- Admin API key configured
+
+## Examples
+
+```bash
+# Explain a simple search query
+miroir-ctl explain --index myindex --q "test query"
+
+# Explain with filters
+miroir-ctl explain --index logs --q "error" --filter "timestamp > 2024-01-01"
+
+# Explain with ranking details
+miroir-ctl explain --index myindex --q "test" --show-ranking
+
+# Show shard routing for a query
+miroir-ctl explain --index myindex --q "test" --show-routing
+```
+
+## Gotchas
+- **Not yet implemented** — see tracking bead for details
+- Explain doesn't execute the query — it only shows the plan
+- Use `--show-routing` to understand which nodes are contacted
+- Ranking details show score breakdown per field
+- Explain is useful for debugging slow queries — check per-shard latency
+
+## See also
+- Plan §13.20 — query planning and optimization
+- Plan §2 — rendezvous hashing and shard routing

docs/ctl/key.md

@@ -0,0 +1,48 @@
+# `miroir-ctl key`
+
+## Purpose
+Manage Meilisearch API keys with scoped permissions and tenant isolation.
+
+## Preconditions
+- Admin API key configured
+- Key management enabled in config
+
+## Examples
+
+```bash
+# Create a search-only key for a tenant
+miroir-ctl key create --name acme-search --action search --index acme-*
+
+# Create a key with document write permissions
+miroir-ctl key create --name writer --action documentsAdd,indexUpdate --index logs
+
+# Create a key with expiration
+miroir-ctl key create --name temp --action search --expires 2024-12-31
+
+# Create a scoped key (UI-only operation, see Plan §13.21)
+miroir-ctl key create-scoped --name ui-key --tenant acme
+
+# List all keys
+miroir-ctl key list
+
+# Get key details (including usage stats)
+miroir-ctl key get --name acme-search
+
+# Delete a key
+miroir-ctl key delete --name acme-search
+
+# Rotate the master key (UI-only, see Plan §13.21)
+miroir-ctl key rotate-master
+```
+
+## Gotchas
+- **Partially implemented** — basic CRUD works, scoped keys are UI-only
+- Keys are returned once at creation — save the key value immediately
+- Deleting a key revokes access immediately — no grace period
+- Scoped keys embed tenant and index filters — verify before use
+- Master key rotation requires downtime — schedule maintenance window
+
+## See also
+- Plan §13.21 — scoped key rotation and master key management
+- Plan §9 — security and key scoping
+- Admin UI — preferred interface for key rotation (not CLI)

docs/ctl/node.md

@@ -0,0 +1,39 @@
+# `miroir-ctl node`
+
+## Purpose
+Manage cluster topology — add, remove, drain, and list nodes.
+
+## Preconditions
+- Target node must be running and reachable from the orchestrator
+- Node's Meilisearch instance should be healthy before adding
+- For removal: node should be drained first to avoid data loss
+
+## Examples
+
+```bash
+# Add a new node to replica group 0
+miroir-ctl node add --id node-3 --address http://node-3:7700 --replica-group 0
+
+# List all nodes with status
+miroir-ctl node list
+
+# Drain a node before removal (migrates its shards elsewhere)
+miroir-ctl node drain node-2
+
+# Remove a node after draining completes
+miroir-ctl node remove node-2
+
+# Force remove (dangerous — skips drain, may lose data)
+miroir-ctl node remove node-2 --force --yes
+```
+
+## Gotchas
+- `node add` triggers automatic rebalancing — use `miroir-ctl rebalance status --watch` to track
+- `node drain` is async — the command returns immediately but migrations continue in background
+- Never use `--force` unless the node is permanently dead — you'll lose RF replicas
+- `node list` shows the same info as `miroir-ctl status` but in table format
+
+## See also
+- Plan §13.2 — node addition and rebalancing
+- Plan §13.3 — node drain and removal
+- `miroir-ctl rebalance status` — track migration progress

docs/ctl/rebalance.md

@@ -0,0 +1,29 @@
+# `miroir-ctl rebalance`
+
+## Purpose
+Monitor and manage shard migrations triggered by topology changes.
+
+## Preconditions
+- Rebalancing runs automatically after `node add` / `node drain` — no manual start needed
+- Admin API key for status queries
+
+## Examples
+
+```bash
+# Show current rebalance progress
+miroir-ctl rebalance status
+
+# Watch migrations in real-time (use during node operations)
+miroir-ctl rebalance status --watch
+```
+
+## Gotchas
+- Rebalancing is automatic — there's no `rebalance start` command
+- Migrations are rate-limited to avoid overwhelming nodes — large topology changes may take hours
+- Each migration moves one shard from source to destination; a node add triggers ~S/Ng migrations
+- Queries continue serving during rebalance — old and new locations are both valid
+
+## See also
+- Plan §13.2 — automatic rebalancing on topology change
+- Plan §13.3 — drain-triggered rebalancing
+- `miroir-ctl status --watch` — cluster-wide health during migrations

docs/ctl/reshard.md

@@ -0,0 +1,41 @@
+# `miroir-ctl reshard`
+
+## Purpose
+Change an index's shard count (S) without downtime. Requires full backfill.
+
+## Preconditions
+- Index must exist with sufficient data to warrant resharding
+- Resharding must be enabled in config (`resharding.enabled = true`)
+- Schedule window configured (or `--force` used)
+- Target shard count should be `max_nodes_per_group_ever × 8` — choose generously
+
+## Examples
+
+```bash
+# Dry run: see what would happen
+miroir-ctl reshard start --index myindex --new-shards 128 --dry-run
+
+# Start resharding within schedule window
+miroir-ctl reshard start --index myindex --new-shards 128 --schedule-window off-peak
+
+# Start with throttled backfill (5000 docs/sec)
+miroir-ctl reshard start --index myindex --new-shards 128 --throttle 5000
+
+# Force start outside schedule window (not recommended)
+miroir-ctl reshard start --index myindex --new-shards 128 --force
+
+# Check resharding progress
+miroir-ctl reshard status --index myindex
+```
+
+## Gotchas
+- Resharding creates a shadow index (`index__reshard_N`) — doubles storage during operation
+- Backfill can take hours for large indexes — use `--throttle` to limit load
+- The old index is retained for 48h (configurable) before cleanup
+- Verify runs before swap — any mismatch aborts the operation
+- This is **not** for adding nodes — use `miroir-ctl node add` for that
+
+## See also
+- Plan §13.1 — online resharding architecture
+- `~/.config/miroir/config.toml` — `[resharding]` section for windows and throttling
+- Admin UI (§13.19) — alternative to CLI for one-off operations

docs/ctl/shadow.md

@@ -0,0 +1,35 @@
+# `miroir-ctl shadow`
+
+## Purpose
+Manage shadow indexing for A/B testing and validation without affecting production traffic.
+
+## Preconditions
+- Source index exists and is healthy
+- Sufficient capacity for shadow writes (2x write amplification)
+
+## Examples
+
+```bash
+# Create a shadow index for testing
+miroir-ctl shadow create --source prod --shadow test --sync-writes
+
+# Enable query shadowing (mirror queries to both, compare results)
+miroir-ctl shadow query --source prod --shadow test --compare
+
+# Check shadow index lag
+miroir-ctl shadow status --source prod
+
+# Stop shadowing and delete shadow index
+miroir-ctl shadow delete --source prod
+```
+
+## Gotchas
+- **Not yet implemented** — see tracking bead for details
+- Shadow writes are synchronous — adds latency to production writes
+- Query shadowing is asynchronous — doesn't affect production latency
+- Use shadow indexing for schema validation, not load testing (write amplification)
+- Delete shadow indexes after testing to free storage
+
+## See also
+- Plan §13.16 — shadow indexing architecture
+- `miroir-ctl verify` — compare shadow and source results

docs/ctl/status.md

@@ -0,0 +1,27 @@
+# `miroir-ctl status`
+
+## Purpose
+Show cluster health, node status, and ongoing rebalancing operations.
+
+## Preconditions
+- Admin API key configured (`MIROIR_ADMIN_API_KEY` env var, `~/.config/miroir/credentials`, or `--admin-key` flag)
+- Miroir orchestrator reachable at `--api-url` (default: `http://localhost:8080`)
+
+## Examples
+
+```bash
+# One-time status snapshot
+miroir-ctl status
+
+# Continuously refresh every 2 seconds (use during operations)
+miroir-ctl status --watch
+```
+
+## Gotchas
+- `--watch` mode clears the terminal each refresh — use `Ctrl+C` to exit
+- Degraded nodes show `⚠` but don't necessarily mean downtime — check individual node errors
+- `fully_covered: false` means some shards have fewer than RF replicas — check `degraded_node_count`
+
+## See also
+- Plan §10 — health metrics and degraded node detection
+- `miroir-ctl node list` — detailed node topology

docs/ctl/task.md

@@ -0,0 +1,34 @@
+# `miroir-ctl task`
+
+## Purpose
+Monitor and manage background tasks (resharding, rebalancing, verification, etc.).
+
+## Preconditions
+- Admin API key configured
+
+## Examples
+
+```bash
+# List all background tasks
+miroir-ctl task list
+
+# Show status of a specific task
+miroir-ctl task status --id abc123
+
+# Show status of all tasks (no ID filter)
+miroir-ctl task status
+
+# Cancel a running task
+miroir-ctl task cancel --id abc123
+```
+
+## Gotchas
+- **Not yet implemented** — see bead miroir-qon for tracking
+- Task IDs are returned by async commands (`node add`, `reshard start`, etc.)
+- Cancellation is best-effort — some tasks complete their current unit before stopping
+- Tasks that fail are retried automatically — check `task status` for retry count
+
+## See also
+- Plan §13 — task management across operations
+- `miroir-ctl rebalance status` — shortcut for rebalancing tasks
+- `miroir-ctl reshard status` — shortcut for resharding tasks

docs/ctl/tenant.md

@@ -0,0 +1,38 @@
+# `miroir-ctl tenant`
+
+## Purpose
+Manage multi-tenancy by isolating indexes and API keys per tenant.
+
+## Preconditions
+- Admin API key configured
+- Multi-tenancy enabled in config
+
+## Examples
+
+```bash
+# Create a new tenant
+miroir-ctl tenant create --name acme --quota indexes=10,docs=1000000
+
+# List all tenants
+miroir-ctl tenant list
+
+# Get tenant details and quota usage
+miroir-ctl tenant get --name acme
+
+# Update tenant quota
+miroir-ctl tenant update --name acme --quota indexes=20
+
+# Delete a tenant (and all its indexes)
+miroir-ctl tenant delete --name acme
+```
+
+## Gotchas
+- **Not yet implemented** — see tracking bead for details
+- Tenant isolation is logical — all data lives in the same cluster
+- Quota enforcement is best-effort — brief overages are possible during batch operations
+- Deleting a tenant deletes all its indexes — this is irreversible
+- API keys are scoped per tenant — cross-tenant access is impossible
+
+## See also
+- Plan §13.15 — multi-tenancy architecture
+- Plan §9 — security and key scoping

docs/ctl/ttl.md

@@ -0,0 +1,35 @@
+# `miroir-ctl ttl`
+
+## Purpose
+Manage time-to-live (TTL) policies for automatic document expiration.
+
+## Preconditions
+- Index must exist with a timestamp field
+- TTL feature enabled in config
+
+## Examples
+
+```bash
+# Set a TTL policy on an index (documents expire after 30 days)
+miroir-ctl ttl set --index logs --field created_at --duration 30d
+
+# Set TTL with custom check interval
+miroir-ctl ttl set --index logs --field created_at --duration 7d --check-interval 1h
+
+# Get current TTL policy for an index
+miroir-ctl ttl get --index logs
+
+# Remove TTL policy
+miroir-ctl ttl remove --index logs
+```
+
+## Gotchas
+- **Not yet implemented** — see tracking bead for details
+- TTL is enforced via background tasks — expired documents are deleted on next pass
+- Duration format: `30d` (days), `12h` (hours), `15m` (minutes)
+- TTL field must be a ISO 8601 timestamp or Unix epoch
+- Deleting expired documents is irreversible — backup first if needed
+
+## See also
+- Plan §13.14 — TTL implementation and task scheduling
+- `miroir-ctl task status` — monitor TTL cleanup jobs

docs/ctl/ui.md

@@ -0,0 +1,36 @@
+# `miroir-ctl ui`
+
+## Purpose
+Launch the web Admin UI for interactive cluster management.
+
+## Preconditions
+- Admin API key configured
+- Browser available (opens localhost:3000 by default)
+
+## Examples
+
+```bash
+# Launch the Admin UI
+miroir-ctl ui
+
+# Launch on a custom port
+miroir-ctl ui --port 8080
+
+# Launch without auto-opening browser
+miroir-ctl ui --no-open
+
+# Launch with specific API endpoint
+miroir-ctl ui --api-url https://miroir.example.com
+```
+
+## Gotchas
+- UI runs locally — it makes API calls to the orchestrator from your browser
+- Credentials are stored in browser local storage — clear after use on shared machines
+- Most UI operations have CLI equivalents — prefer CLI for scripts and CI/CD
+- UI requires CSRF token exchange — first load may be slower
+- Some advanced operations (e.g., scoped key rotation) are UI-only
+
+## See also
+- Plan §13.19 — Admin UI features and architecture
+- Plan §13.21 — scoped key rotation (UI-only operation)
+- `docs/ctl/*.md` — CLI equivalents for UI operations

docs/ctl/verify.md

@@ -0,0 +1,31 @@
+# `miroir-ctl verify`
+
+## Purpose
+Check data integrity across replica groups and detect drift.
+
+## Preconditions
+- Admin API key configured
+- Cluster healthy (degraded nodes can cause false positives)
+
+## Examples
+
+```bash
+# Verify all indexes
+miroir-ctl verify
+
+# Verify specific index
+miroir-ctl verify --index myindex
+
+# Verbose output (shows per-shard details)
+miroir-ctl verify --verbose
+```
+
+## Gotchas
+- Verification is read-only — it won't fix drift, only report it
+- Large indexes take time to verify — progress is not streamed
+- Use `miroir-ctl task status` to check async verification jobs
+- Drift is automatically repaired by anti-entropy workers — see `miroir-ctl task status`
+
+## See also
+- Plan §13.8 — anti-entropy and drift detection
+- Plan §10 — observability and metrics