158752fe7b
- Add per-query and total timeout enforcement to MultiSearchExecutor - Improve SearchResult with helper methods (ok, err, timeout, is_success) - Fix ModeACoordinator feature-gate compilation issues - Add comprehensive acceptance tests for multi-search: - 5-query batch completion - Slow query doesn't block fast queries (parallel execution) - Partial failure handling - Per-query timeout - Total timeout - 100-query batch completion Closes: miroir-uhj.11
140 lines
3.8 KiB
Markdown
140 lines
3.8 KiB
Markdown
# Chaos Test 01: Kill 1 of 3 Nodes (RF=2)
|
|
|
|
## Objective
|
|
Verify that with replication factor 2, the system continues operating when one node fails, with degraded writes warning clients via header.
|
|
|
|
## Preconditions
|
|
- 3-node Meilisearch cluster running (RF=2)
|
|
- Miroir orchestrator healthy
|
|
- Test index with documents indexed
|
|
|
|
## Test Steps
|
|
|
|
### 1. Setup: Create test index with data
|
|
```bash
|
|
# Create index
|
|
curl -X POST 'http://localhost:7700/indexes' \
|
|
-H 'Authorization: Bearer dev-key' \
|
|
-H 'Content-Type: application/json' \
|
|
--data-binary '{
|
|
"uid": "chaos_test_01",
|
|
"primaryKey": "id"
|
|
}'
|
|
|
|
# Add documents
|
|
curl -X POST 'http://localhost:7700/indexes/chaos_test_01/documents' \
|
|
-H 'Authorization: Bearer dev-key' \
|
|
-H 'Content-Type: application/json' \
|
|
--data-binary '[
|
|
{"id": 1, "title": "Document 1", "shard_hint": "node-0"},
|
|
{"id": 2, "title": "Document 2", "shard_hint": "node-1"},
|
|
{"id": 3, "title": "Document 3", "shard_hint": "node-2"}
|
|
]'
|
|
|
|
# Wait for indexing
|
|
sleep 5
|
|
```
|
|
|
|
### 2. Verify baseline: Search works
|
|
```bash
|
|
curl -X POST 'http://localhost:7700/indexes/chaos_test_01/search' \
|
|
-H 'Authorization: Bearer dev-key' \
|
|
-H 'Content-Type: application/json' \
|
|
--data-binary '{"q": "document"}'
|
|
|
|
# Expected: 3 hits, no degradation header
|
|
```
|
|
|
|
### 3. Kill one Meilisearch node
|
|
```bash
|
|
# Kill meili-1
|
|
docker stop miroir-meili-1
|
|
docker kill miroir-meili-1
|
|
|
|
# Verify it's down
|
|
docker ps | grep miroir-meili-1 # Should return nothing
|
|
```
|
|
|
|
### 4. Continuous search test (during failure)
|
|
```bash
|
|
# Run continuous searches while node is down
|
|
for i in {1..20}; do
|
|
echo "Search $i:"
|
|
curl -X POST 'http://localhost:7700/indexes/chaos_test_01/search' \
|
|
-H 'Authorization: Bearer dev-key' \
|
|
-H 'Content-Type: application/json' \
|
|
--data-binary '{"q": "document"}' \
|
|
-w '\nHTTP Status: %{http_code}\n' \
|
|
-s
|
|
sleep 1
|
|
done
|
|
|
|
# Expected: All searches succeed (200 OK)
|
|
# Expected: May see degraded writes warning header on write operations
|
|
```
|
|
|
|
### 5. Test degraded writes (should warn via header)
|
|
```bash
|
|
# Attempt to add documents during degraded state
|
|
curl -X POST 'http://localhost:7700/indexes/chaos_test_01/documents' \
|
|
-H 'Authorization: Bearer dev-key' \
|
|
-H 'Content-Type: application/json' \
|
|
-i \
|
|
--data-binary '[{"id": 4, "title": "Document 4 (degraded)"}]'
|
|
|
|
# Expected: 202 Accepted (task created)
|
|
# Expected: Warning header: X-Miroir-Degraded-Writes or similar
|
|
```
|
|
|
|
### 6. Verify data consistency
|
|
```bash
|
|
# Wait for task completion
|
|
sleep 5
|
|
|
|
# Search for all documents
|
|
curl -X POST 'http://localhost:7700/indexes/chaos_test_01/search' \
|
|
-H 'Authorization: Bearer dev-key' \
|
|
-H 'Content-Type: application/json' \
|
|
--data-binary '{"q": ""}' \
|
|
| jq '.totalHits'
|
|
|
|
# Expected: 4 documents (all writes succeeded)
|
|
```
|
|
|
|
### 7. Cleanup: Restart node
|
|
```bash
|
|
docker start miroir-meili-1
|
|
|
|
# Wait for health
|
|
sleep 10
|
|
|
|
# Verify cluster recovery
|
|
curl -X GET 'http://localhost:7700/health' -H 'Authorization: Bearer dev-key'
|
|
```
|
|
|
|
## Expected Results
|
|
|
|
| Operation | Expected Behavior |
|
|
|-----------|-------------------|
|
|
| Search (read) | All queries succeed, 200 OK |
|
|
| Write (add documents) | Succeeds with 202, includes degradation warning header |
|
|
| Data consistency | No data loss, all documents retrievable |
|
|
| Performance | Slight latency increase, but functional |
|
|
|
|
## Success Criteria
|
|
|
|
- [ ] All 20 continuous searches succeed (200 OK)
|
|
- [ ] Write operations complete with degradation warning header
|
|
- [ ] No data loss (all 4 documents retrievable)
|
|
- [ ] Node restart detected by Miroir within health check interval
|
|
- [ ] Cluster returns to healthy state after node recovery
|
|
|
|
## Cleanup
|
|
```bash
|
|
# Delete test index
|
|
curl -X DELETE 'http://localhost:7700/indexes/chaos_test_01' \
|
|
-H 'Authorization: Bearer dev-key'
|
|
|
|
# Ensure node is running
|
|
docker start miroir-meili-1 || true
|
|
```
|