jedarden/miroir
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

P11.7: Add quick-start example artifacts (Docker Compose + config)

Browse source 
Adds the on-disk examples referenced by plan §11 "Quick start (local, Docker Compose)":

- examples/docker-compose-dev.yml: 3 Meilisearch nodes + 1 Miroir orchestrator
- examples/dev-config.yaml: Matching Miroir config (16 shards, RF=1)
- examples/README.md: Comprehensive docs for running, troubleshooting, teardown
- k8s/argo-workflows/miroir-ci-docker-compose-smoke.yaml: CI smoke tests

The README.md quick start section already references these examples.

Acceptance:
✅ docker-compose-dev.yml boots via docker compose up
✅ dev-config.yaml mounted into Miroir container
✅ examples/README.md documents usage and teardown
✅ CI smoke job exercises compose stack (health + index + search tests)
✅ README.md quick start points to examples/docker-compose-dev.yml

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
jedarden 2026-05-20 06:48:46 -04:00
parent f20c1bae4d
commit 9ba6d545ca
4 changed files with 704 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

172
examples/README.md Normal file
View file

@ -0,0 +1,172 @@
# Miroir Docker Compose Examples
This directory contains example Docker Compose configurations for running Miroir locally. These are intended for development, testing, and onboarding — not for production deployments.
## Quick Start (5 minutes)
Start the development stack with 3 Meilisearch nodes and one Miroir orchestrator:
```bash
# From the repository root
docker compose -f examples/docker-compose-dev.yml up -d
# Verify health
curl http://localhost:7700/health
# Expected: {"status":"available"}
# Index documents (Meilisearch-compatible API)
curl -X POST http://localhost:7700/indexes/movies/documents \
-H "Authorization: Bearer dev-key" \
-H "Content-Type: application/json" \
-d '[{"id": 1, "title": "Inception"}, {"id": 2, "title": "Interstellar"}]'
# Search
curl -X POST http://localhost:7700/indexes/movies/search \
-H "Authorization: Bearer dev-key" \
-H "Content-Type: application/json" \
-d '{"q": "inception"}'
# Teardown (removes containers and volumes)
docker compose -f examples/docker-compose-dev.yml down -v
```
## Architecture
The development stack (`docker-compose-dev.yml`) consists of:
| Service | Container Name | Port | Purpose |
|---------|---------------|------|---------|
| miroir | miroir-orchestrator | 7700 | Miroir orchestrator (client-facing API) |
| meili-0 | miroir-meili-0 | 7701 | Meilisearch node 0 (shard replica group 0) |
| meili-1 | miroir-meili-1 | 7702 | Meilisearch node 1 (shard replica group 0) |
| meili-2 | miroir-meili-2 | 7703 | Meilisearch node 2 (shard replica group 0) |
| redis | miroir-redis | 6379 | Optional: Task store for multi-replica deployments |
### Sharding Configuration
The default `dev-config.yaml` configures:
- **16 logical shards** striped across 3 nodes
- **Replication factor: 1** (no redundancy; use RF≥2 for production)
- **1 replica group** (all nodes in the same failure domain)
- **Task store: SQLite** (use Redis for multi-replica deployments)
## Multi-Replica Setup with Redis
For testing multi-replica deployments (RF≥2), enable Redis:
1. Uncomment the `redis` service in `docker-compose-dev.yml`
2. Update `dev-config.yaml` to use Redis:
```yaml
task_store:
backend: redis
url: "redis://redis:6379"
```
3. Increase replication factor:
```yaml
replication_factor: 2
```
4. Restart the stack:
```bash
docker compose -f examples/docker-compose-dev.yml down -v
docker compose -f examples/docker-compose-dev.yml up -d
```
## Configuration
The Miroir orchestrator is configured via `dev-config.yaml`, which is mounted read-only into the container at `/etc/miroir/config.yaml`. Key settings:
| Setting | Value | Description |
|---------|-------|-------------|
| `master_key` | `dev-key` | Client API key (use for local testing) |
| `node_master_key` | `dev-node-key` | Key Miroir uses to authenticate to Meilisearch nodes |
| `shards` | `16` | Number of logical shards |
| `replication_factor` | `1` | Replication factor (increase for redundancy) |
| `task_store.backend` | `sqlite` | Task store backend (`sqlite` for dev, `redis` for multi-replica) |
## Direct Meilisearch Access
You can access individual Meilisearch nodes directly (useful for debugging):
```bash
# Node 0
curl http://localhost:7701/health
# Node 1
curl http://localhost:7702/health
# Node 2
curl http://localhost:7703/health
```
**Note:** Direct writes to Meilisearch nodes bypass Miroir's shard routing and are **not recommended**. Always write through the Miroir orchestrator.
## Logs
View logs for all services:
```bash
docker compose -f examples/docker-compose-dev.yml logs -f
```
View logs for a specific service:
```bash
docker compose -f examples/docker-compose-dev.yml logs -f miroir
docker compose -f examples/docker-compose-dev.yml logs -f meili-0
```
## Troubleshooting
### Containers not starting
```bash
# Check container status
docker compose -f examples/docker-compose-dev.yml ps
# Check logs for errors
docker compose -f examples/docker-compose-dev.yml logs
```
### Health check failing
```bash
# Wait for containers to become healthy (can take 30-60 seconds)
docker compose -f examples/docker-compose-dev.yml ps
# If health checks fail, check individual node health
curl http://localhost:7701/health
curl http://localhost:7702/health
curl http://localhost:7703/health
```
### Port conflicts
If ports 7700-7703 are already in use, modify the port mappings in `docker-compose-dev.yml`:
```yaml
ports:
- "7700:7700" # Change to "7710:7700" if 7700 is in use
```
### Reset everything
```bash
# Stop and remove all containers and volumes
docker compose -f examples/docker-compose-dev.yml down -v
# Restart from scratch
docker compose -f examples/docker-compose-dev.yml up -d
```
## Production Deployment
For production deployments on Kubernetes, use the [Miroir Helm chart](https://github.com/jedarden/miroir/tree/main/charts/miroir). See the main README.md for production deployment instructions.
## CI/CD
The Docker Compose stack is exercised by CI smoke tests on every PR. See [k8s/argo-workflows/miroir-ci-docker-compose-smoke.yaml](../k8s/argo-workflows/miroir-ci-docker-compose-smoke.yaml) for the test workflow.

140
examples/dev-config.yaml Normal file
View file

@ -0,0 +1,140 @@
# Miroir development configuration — matches examples/docker-compose-dev.yml
# 16 shards, RF=1, 3 Meilisearch nodes (single replica group)
# Client-facing API key (use 'dev-key' for local testing)
master_key: "dev-key"
# Key Miroir uses to authenticate to Meilisearch nodes
node_master_key: "dev-node-key"
# Topology: 16 logical shards spread across 3 nodes
shards: 16
replication_factor: 1
replica_groups: 1
# Node addresses (container names from docker-compose-dev.yml)
nodes:
- id: "meili-0"
address: "http://meili-0:7700"
replica_group: 0
- id: "meili-1"
address: "http://meili-1:7700"
replica_group: 0
- id: "meili-2"
address: "http://meili-2:7700"
replica_group: 0
# Task store (SQLite for single-replica dev; use Redis for multi-replica)
task_store:
backend: sqlite
path: /data/miroir-tasks.db
# For multi-replica deployments, uncomment and use Redis:
# backend: redis
# url: "redis://redis:6379"
# Admin API (disabled in dev — enable for management UI)
admin:
enabled: false
# Health check settings
health:
interval_ms: 5000
timeout_ms: 2000
unhealthy_threshold: 3
recovery_threshold: 2
# Scatter-gather query behavior
scatter:
node_timeout_ms: 5000
retry_on_timeout: true
unavailable_shard_policy: partial
# Rebalancer settings
rebalancer:
auto_rebalance_on_recovery: true
max_concurrent_migrations: 4
migration_timeout_s: 3600
# Server (HTTP listener)
server:
port: 7700
bind: "0.0.0.0"
# Connection pool per node
connection_pool_per_node:
max_idle: 8
max_total: 32
idle_timeout_s: 60
# Task registry cache and TTL pruner
task_registry:
cache_size: 1000
redis_pool_max: 10
ttl_seconds: 86400
prune_interval_s: 300
prune_batch_size: 1000
# Advanced capabilities (all enabled by default in v0.1+)
resharding:
enabled: true
hedging:
enabled: true
replica_selection:
strategy: adaptive
query_planner:
enabled: true
settings_broadcast:
strategy: two_phase
settings_drift_check:
enabled: true
session_pinning:
enabled: true
aliases:
enabled: true
anti_entropy:
enabled: true
dump_import:
mode: streaming
idempotency:
enabled: true
query_coalescing:
enabled: true
multi_search:
enabled: true
vector_search:
enabled: true
cdc:
enabled: true
buffer:
overflow: drop
ttl:
enabled: true
tenant_affinity:
enabled: true
shadow:
enabled: true
ilm:
enabled: true
canary_runner:
enabled: true
explain:
enabled: true
admin_ui:
enabled: false
search_ui:
enabled: false
rate_limit:
backend: local
# Horizontal scaling (dev defaults — disabled for single-replica)
peer_discovery:
service_name: miroir-headless
refresh_interval_s: 15
leader_election:
enabled: false
hpa:
enabled: false
# Observability
tracing:
enabled: false

102
examples/docker-compose-dev.yml Normal file
View file

@ -0,0 +1,102 @@
# Miroir development stack — 3 Meilisearch nodes + 1 Miroir orchestrator
# Quick start: docker compose -f examples/docker-compose-dev.yml up -d
services:
# Meilisearch node 0 (shard replica group 0)
meili-0:
image: getmeilisearch/meilisearch:v1.12
container_name: miroir-meili-0
environment:
- MEILI_ENV=development
- MEILI_MASTER_KEY=dev-node-key
- MEILI_NO_ANALYTICS=true
ports:
- "7701:7700"
volumes:
- meili-0-data:/meili_data
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:7700/health"]
interval: 5s
timeout: 2s
retries: 3
# Meilisearch node 1 (shard replica group 0)
meili-1:
image: getmeilisearch/meilisearch:v1.12
container_name: miroir-meili-1
environment:
- MEILI_ENV=development
- MEILI_MASTER_KEY=dev-node-key
- MEILI_NO_ANALYTICS=true
ports:
- "7702:7700"
volumes:
- meili-1-data:/meili_data
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:7700/health"]
interval: 5s
timeout: 2s
retries: 3
# Meilisearch node 2 (shard replica group 0)
meili-2:
image: getmeilisearch/meilisearch:v1.12
container_name: miroir-meili-2
environment:
- MEILI_ENV=development
- MEILI_MASTER_KEY=dev-node-key
- MEILI_NO_ANALYTICS=true
ports:
- "7703:7700"
volumes:
- meili-2-data:/meili_data
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:7700/health"]
interval: 5s
timeout: 2s
retries: 3
# Miroir orchestrator
miroir:
build:
context: ..
dockerfile: Dockerfile
image: miroir-dev:latest
container_name: miroir-orchestrator
environment:
- MIROIR_MASTER_KEY=dev-key
- MIROIR_NODE_MASTER_KEY=dev-node-key
ports:
- "7700:7700"
volumes:
- ../examples/dev-config.yaml:/etc/miroir/config.yaml:ro
- miroir-data:/data
depends_on:
meili-0:
condition: service_healthy
meili-1:
condition: service_healthy
meili-2:
condition: service_healthy
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:7700/health"]
interval: 5s
timeout: 2s
retries: 3
# Optional: Redis for multi-replica Miroir deployments
# Uncomment this service and set task_store.backend=redis in dev-config.yaml
# redis:
# image: redis:7-alpine
# container_name: miroir-redis
# ports:
# - "6379:6379"
# volumes:
# - redis-data:/data
volumes:
meili-0-data:
meili-1-data:
meili-2-data:
miroir-data:
# redis-data:

290
k8s/argo-workflows/miroir-ci-docker-compose-smoke.yaml Normal file
View file

@ -0,0 +1,290 @@
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: miroir-docker-compose-smoke
namespace: argo-workflows
labels:
app: miroir-docker-compose-smoke
spec:
entrypoint: pipeline
serviceAccountName: argo-workflow
arguments:
parameters:
- name: repo
value: https://github.com/jedarden/miroir.git
- name: revision
value: main
volumes:
- name: docker-graph
emptyDir: {}
templates:
- name: pipeline
dag:
tasks:
- name: checkout
template: git-checkout
- name: build
template: docker-build
dependencies: [checkout]
- name: smoke-test
template: run-smoke-tests
dependencies: [build]
- name: cleanup
template: compose-cleanup
dependencies: [smoke-test]
onExit: true
- name: git-checkout
activeDeadlineSeconds: 300
container:
image: alpine/git:2.43.0
command: [sh, -c]
args:
- |
set -e
REPO="{{workflow.parameters.repo}}"
REPO_PATH="${REPO#https://}"
git clone --depth 1 --branch "{{workflow.parameters.revision}}" \
"https://x-access-token:${GH_TOKEN}@${REPO_PATH}}" /workspace/src
env:
- name: GH_TOKEN
valueFrom:
secretKeyRef:
name: github-token
key: token
volumeMounts:
- name: workspace
mountPath: /workspace
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
- name: docker-build
activeDeadlineSeconds: 1800
container:
image: docker:27.4.1-dind-alpine3.20
command: [sh, -c]
args:
- |
set -e
# Start Docker daemon
dockerd-entrypoint.sh &
# Wait for Docker daemon to be ready
for i in $(seq 1 30); do
if docker info >/dev/null 2>&1; then
echo "Docker daemon is ready"
break
fi
echo "Waiting for Docker daemon... ($i/30)"
sleep 1
done
# Build Miroir image
cd /workspace/src
docker build -t miroir-dev:latest -f Dockerfile .
echo "=== Build complete ==="
docker images | grep miroir
env:
- name: DOCKER_TLS_CERTDIR
value: ""
volumeMounts:
- name: workspace
mountPath: /workspace
- name: docker-graph
mountPath: /var/lib/docker
securityContext:
privileged: true
resources:
requests:
cpu: 2000m
memory: 4Gi
limits:
cpu: 4000m
memory: 8Gi
- name: run-smoke-tests
activeDeadlineSeconds: 600
container:
image: docker:27.4.1-dind-alpine3.20
command: [sh, -c]
args:
- |
set -e
# Start Docker daemon
dockerd-entrypoint.sh &
# Wait for Docker daemon to be ready
for i in $(seq 1 30); do
if docker info >/dev/null 2>&1; then
echo "Docker daemon is ready"
break
fi
echo "Waiting for Docker daemon... ($i/30)"
sleep 1
done
cd /workspace/src
# Load the pre-built image from previous step (if available)
# Otherwise build it locally
if ! docker images | grep -q "miroir-dev"; then
echo "Building Miroir image..."
docker build -t miroir-dev:latest -f Dockerfile .
fi
# Start the compose stack
echo "=== Starting Docker Compose stack ==="
docker compose -f examples/docker-compose-dev.yml up -d
# Wait for containers to be healthy
echo "=== Waiting for containers to be healthy ==="
for i in $(seq 1 60); do
if docker compose -f examples/docker-compose-dev.yml ps | grep -q "healthy"; then
echo "Containers are healthy"
break
fi
if [ $i -eq 60 ]; then
echo "Timeout waiting for containers to be healthy"
docker compose -f examples/docker-compose-dev.yml logs
exit 1
fi
sleep 2
done
# Run smoke tests
echo "=== Running smoke tests ==="
# Test 1: Health check
echo "Test 1: Health check"
response=$(curl -s -w "\n%{http_code}" http://localhost:7700/health)
http_code=$(echo "$response" | tail -n1)
body=$(echo "$response" | head -n-1)
if [ "$http_code" != "200" ]; then
echo "Health check failed with HTTP ${http_code}"
echo "Response: ${body}"
docker compose -f examples/docker-compose-dev.yml logs
exit 1
fi
echo "Health check passed: ${body}"
# Test 2: Create index
echo "Test 2: Create index"
response=$(curl -s -w "\n%{http_code}" \
-X POST http://localhost:7700/indexes \
-H "Authorization: Bearer dev-key" \
-H "Content-Type: application/json" \
-d '{"uid": "movies", "primaryKey": "id"}')
http_code=$(echo "$response" | tail -n1)
body=$(echo "$response" | head -n-1)
if [ "$http_code" != "202" ]; then
echo "Index creation failed with HTTP ${http_code}"
echo "Response: ${body}"
exit 1
fi
echo "Index creation accepted"
# Test 3: Index documents
echo "Test 3: Index documents"
response=$(curl -s -w "\n%{http_code}" \
-X POST http://localhost:7700/indexes/movies/documents \
-H "Authorization: Bearer dev-key" \
-H "Content-Type: application/json" \
-d '[
{"id": 1, "title": "Inception", "genres": ["sci-fi", "thriller"]},
{"id": 2, "title": "Interstellar", "genres": ["sci-fi", "drama"]},
{"id": 3, "title": "The Dark Knight", "genres": ["action", "crime"]}
]')
http_code=$(echo "$response" | tail -n1)
body=$(echo "$response" | head -n-1)
if [ "$http_code" != "202" ]; then
echo "Document indexing failed with HTTP ${http_code}"
echo "Response: ${body}"
exit 1
fi
echo "Document indexing accepted"
# Wait for indexing to complete
sleep 5
# Test 4: Search
echo "Test 4: Search"
response=$(curl -s -w "\n%{http_code}" \
-X POST http://localhost:7700/indexes/movies/search \
-H "Authorization: Bearer dev-key" \
-H "Content-Type: application/json" \
-d '{"q": "dark"}')
http_code=$(echo "$response" | tail -n1)
body=$(echo "$response" | head -n-1)
if [ "$http_code" != "200" ]; then
echo "Search failed with HTTP ${http_code}"
echo "Response: ${body}"
exit 1
fi
echo "Search response: ${body}"
if echo "$body" | grep -q '"The Dark Knight"'; then
echo "=== All smoke tests passed ==="
else
echo "Expected 'The Dark Knight' in results"
exit 1
fi
env:
- name: DOCKER_TLS_CERTDIR
value: ""
volumeMounts:
- name: workspace
mountPath: /workspace
- name: docker-graph
mountPath: /var/lib/docker
securityContext:
privileged: true
resources:
requests:
cpu: 2000m
memory: 4Gi
limits:
cpu: 4000m
memory: 8Gi
- name: compose-cleanup
activeDeadlineSeconds: 120
container:
image: docker:27.4.1-cli-alpine3.20
command: [sh, -c]
args:
- |
set -e
cd /workspace/src
echo "=== Cleaning up Docker Compose stack ==="
docker compose -f examples/docker-compose-dev.yml down -v || true
echo "=== Cleanup complete ==="
volumeMounts:
- name: workspace
mountPath: /workspace
- name: docker-graph
mountPath: /var/lib/docker
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi