miroir/examples
jedarden 304879d32a feat(tests): add chaos test scenarios and runbooks (plan §8, P9.4)
Add comprehensive chaos testing infrastructure for Miroir failure scenarios:

- **TestCluster** harness with chaos helpers:
  - `kill_meili()` / `restart_meili()` for node failure simulation
  - `apply_netem()` / `remove_netem()` for network delay injection
  - `kill_miroir()` / `restart_miroir()` for orchestrator failure
  - Docker-compose stack lifecycle management

- **6 chaos test scenarios** (all marked `#[ignore]`):
  1. Kill 1 of 3 nodes (RF=2) → continuous search, no degraded header
  2. Kill 2 of 3 nodes (RF=2) → 503 or partial results with degraded header
  3. Kill 1 of 2 Miroir replicas → zero client-visible downtime
  4. tc netem 500ms delay → searches slow but succeed, no errors
  5. Restart killed node → Miroir detects recovery within health check interval
  6. Kill node mid-rebalance → rebalancer pauses, resumes on recovery

- **Runbooks** in `tests/chaos/runbooks/scenario*.md`:
  - Manual reproduction steps
  - Expected observables (metrics, headers, errors)
  - Recovery procedures
  - HA vs single-instance differences
  - Operator notes and common causes

- **Updated docker-compose files**:
  - Added `CAP_NET_ADMIN` to all Meilisearch containers for tc netem support

Tests are slow (30+ seconds each) and require docker-compose. Run with:
  cargo test --test chaos -- --ignored --test-threads=1

Closes: miroir-89x.4
2026-05-24 10:23:24 -04:00
..
sdk-tests feat(tests): add cross-compatibility SDK tests and standalone Meilisearch 2026-05-24 08:40:58 -04:00
dev-config-rf2.yaml P9.2: Integration test harness with docker-compose 2026-05-23 07:26:55 -04:00
dev-config.yaml P11.7: Add quick-start example artifacts (Docker Compose + config) 2026-05-20 06:49:05 -04:00
docker-compose-dev-rf2.yml feat(tests): add chaos test scenarios and runbooks (plan §8, P9.4) 2026-05-24 10:23:24 -04:00
docker-compose-dev.yml feat(tests): add chaos test scenarios and runbooks (plan §8, P9.4) 2026-05-24 10:23:24 -04:00
README.md P9.2: Integration test harness with docker-compose 2026-05-23 07:33:34 -04:00
sdk-smoke-test.rs feat(multi-search): implement timeout enforcement and acceptance tests (§13.11) 2026-05-24 01:54:20 -04:00

Miroir Docker Compose Examples

This directory contains example Docker Compose configurations for running Miroir locally. These are intended for development, testing, and onboarding — not for production deployments.

Quick Start (5 minutes)

Start the development stack with 3 Meilisearch nodes and one Miroir orchestrator:

# From the repository root
docker compose -f examples/docker-compose-dev.yml up -d

# Verify health
curl http://localhost:7700/health
# Expected: {"status":"available"}

# Index documents (Meilisearch-compatible API)
curl -X POST http://localhost:7700/indexes/movies/documents \
  -H "Authorization: Bearer dev-key" \
  -H "Content-Type: application/json" \
  -d '[{"id": 1, "title": "Inception"}, {"id": 2, "title": "Interstellar"}]'

# Search
curl -X POST http://localhost:7700/indexes/movies/search \
  -H "Authorization: Bearer dev-key" \
  -H "Content-Type: application/json" \
  -d '{"q": "inception"}'

# Teardown (removes containers and volumes)
docker compose -f examples/docker-compose-dev.yml down -v

Architecture

The development stack (docker-compose-dev.yml) consists of:

Service Container Name Port Purpose
miroir miroir-orchestrator 7700 Miroir orchestrator (client-facing API)
meili-0 miroir-meili-0 7701 Meilisearch node 0 (shard replica group 0)
meili-1 miroir-meili-1 7702 Meilisearch node 1 (shard replica group 0)
meili-2 miroir-meili-2 7703 Meilisearch node 2 (shard replica group 0)
redis miroir-redis 6379 Optional: Task store for multi-replica deployments

Sharding Configuration

The default dev-config.yaml configures:

  • 16 logical shards striped across 3 nodes
  • Replication factor: 1 (no redundancy; use RF≥2 for production)
  • 1 replica group (all nodes in the same failure domain)
  • Task store: SQLite (use Redis for multi-replica deployments)

Multi-Replica Setup with Redis

For testing multi-replica deployments (RF≥2), enable Redis:

  1. Uncomment the redis service in docker-compose-dev.yml
  2. Update dev-config.yaml to use Redis:
task_store:
  backend: redis
  url: "redis://redis:6379"
  1. Increase replication factor:
replication_factor: 2
  1. Restart the stack:
docker compose -f examples/docker-compose-dev.yml down -v
docker compose -f examples/docker-compose-dev.yml up -d

Configuration

The Miroir orchestrator is configured via dev-config.yaml, which is mounted read-only into the container at /etc/miroir/config.yaml. Key settings:

Setting Value Description
master_key dev-key Client API key (use for local testing)
node_master_key dev-node-key Key Miroir uses to authenticate to Meilisearch nodes
shards 16 Number of logical shards
replication_factor 1 Replication factor (increase for redundancy)
task_store.backend sqlite Task store backend (sqlite for dev, redis for multi-replica)

Direct Meilisearch Access

You can access individual Meilisearch nodes directly (useful for debugging):

# Node 0
curl http://localhost:7701/health

# Node 1
curl http://localhost:7702/health

# Node 2
curl http://localhost:7703/health

Note: Direct writes to Meilisearch nodes bypass Miroir's shard routing and are not recommended. Always write through the Miroir orchestrator.

Logs

View logs for all services:

docker compose -f examples/docker-compose-dev.yml logs -f

View logs for a specific service:

docker compose -f examples/docker-compose-dev.yml logs -f miroir
docker compose -f examples/docker-compose-dev.yml logs -f meili-0

Troubleshooting

Containers not starting

# Check container status
docker compose -f examples/docker-compose-dev.yml ps

# Check logs for errors
docker compose -f examples/docker-compose-dev.yml logs

Health check failing

# Wait for containers to become healthy (can take 30-60 seconds)
docker compose -f examples/docker-compose-dev.yml ps

# If health checks fail, check individual node health
curl http://localhost:7701/health
curl http://localhost:7702/health
curl http://localhost:7703/health

Port conflicts

If ports 7700-7703 are already in use, modify the port mappings in docker-compose-dev.yml:

ports:
  - "7700:7700"  # Change to "7710:7700" if 7700 is in use

Reset everything

# Stop and remove all containers and volumes
docker compose -f examples/docker-compose-dev.yml down -v

# Restart from scratch
docker compose -f examples/docker-compose-dev.yml up -d

Production Deployment

For production deployments on Kubernetes, use the Miroir Helm chart. See the main README.md for production deployment instructions.

CI/CD

The Docker Compose stack is exercised by CI smoke tests on every PR. See k8s/argo-workflows/miroir-ci-docker-compose-smoke.yaml for the test workflow.

Integration Tests

This Docker Compose setup doubles as the integration test harness. Comprehensive integration tests are located in crates/miroir-proxy/tests/docker_compose_integration.rs with documentation in crates/miroir-proxy/tests/README_integration.md.

Run integration tests:

# Start the stack
docker compose -f examples/docker-compose-dev.yml up -d

# Run all integration tests
cargo test -p miroir-proxy --test docker_compose_integration -- --test-threads=1

# Clean up
docker compose -f examples/docker-compose-dev.yml down -v

Integration test coverage includes:

  • Document round-trip (1000 documents)
  • Search coverage across all 16 shards
  • Facet aggregation
  • Offset/limit pagination
  • Settings broadcast
  • Task polling
  • Health checks
  • Node failure with RF=2 (requires docker-compose-dev-rf2.yml)