miroir/tests/chaos/01_kill_one_node_rf2.md

Chaos Test 01: Kill 1 of 3 Nodes (RF=2)

Objective

Verify that with replication factor 2, the system continues operating when one node fails, with degraded writes warning clients via header.

Preconditions

• 3-node Meilisearch cluster running (RF=2)
• Miroir orchestrator healthy
• Test index with documents indexed

Test Steps

1. Setup: Create test index with data

# Create index
curl -X POST 'http://localhost:7700/indexes' \
  -H 'Authorization: Bearer dev-key' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "uid": "chaos_test_01",
    "primaryKey": "id"
  }'

# Add documents
curl -X POST 'http://localhost:7700/indexes/chaos_test_01/documents' \
  -H 'Authorization: Bearer dev-key' \
  -H 'Content-Type: application/json' \
  --data-binary '[
    {"id": 1, "title": "Document 1", "shard_hint": "node-0"},
    {"id": 2, "title": "Document 2", "shard_hint": "node-1"},
    {"id": 3, "title": "Document 3", "shard_hint": "node-2"}
  ]'

# Wait for indexing
sleep 5

2. Verify baseline: Search works

curl -X POST 'http://localhost:7700/indexes/chaos_test_01/search' \
  -H 'Authorization: Bearer dev-key' \
  -H 'Content-Type: application/json' \
  --data-binary '{"q": "document"}'

# Expected: 3 hits, no degradation header

3. Kill one Meilisearch node

# Kill meili-1
docker stop miroir-meili-1
docker kill miroir-meili-1

# Verify it's down
docker ps | grep miroir-meili-1  # Should return nothing

4. Continuous search test (during failure)

# Run continuous searches while node is down
for i in {1..20}; do
  echo "Search $i:"
  curl -X POST 'http://localhost:7700/indexes/chaos_test_01/search' \
    -H 'Authorization: Bearer dev-key' \
    -H 'Content-Type: application/json' \
    --data-binary '{"q": "document"}' \
    -w '\nHTTP Status: %{http_code}\n' \
    -s
  sleep 1
done

# Expected: All searches succeed (200 OK)
# Expected: May see degraded writes warning header on write operations

5. Test degraded writes (should warn via header)

# Attempt to add documents during degraded state
curl -X POST 'http://localhost:7700/indexes/chaos_test_01/documents' \
  -H 'Authorization: Bearer dev-key' \
  -H 'Content-Type: application/json' \
  -i \
  --data-binary '[{"id": 4, "title": "Document 4 (degraded)"}]'

# Expected: 202 Accepted (task created)
# Expected: Warning header: X-Miroir-Degraded-Writes or similar

6. Verify data consistency

# Wait for task completion
sleep 5

# Search for all documents
curl -X POST 'http://localhost:7700/indexes/chaos_test_01/search' \
  -H 'Authorization: Bearer dev-key' \
  -H 'Content-Type: application/json' \
  --data-binary '{"q": ""}' \
  | jq '.totalHits'

# Expected: 4 documents (all writes succeeded)

7. Cleanup: Restart node

docker start miroir-meili-1

# Wait for health
sleep 10

# Verify cluster recovery
curl -X GET 'http://localhost:7700/health' -H 'Authorization: Bearer dev-key'

Expected Results

Operation	Expected Behavior
Search (read)	All queries succeed, 200 OK
Write (add documents)	Succeeds with 202, includes degradation warning header
Data consistency	No data loss, all documents retrievable
Performance	Slight latency increase, but functional

Success Criteria

• All 20 continuous searches succeed (200 OK)
• Write operations complete with degradation warning header
• No data loss (all 4 documents retrievable)
• Node restart detected by Miroir within health check interval
• Cluster returns to healthy state after node recovery

Cleanup

# Delete test index
curl -X DELETE 'http://localhost:7700/indexes/chaos_test_01' \
  -H 'Authorization: Bearer dev-key'

# Ensure node is running
docker start miroir-meili-1 || true