50400fbe44
Implements the streaming routed dump import flow that routes documents
per-shard instead of broadcasting to all nodes.
Changes:
- Complete dump_import.rs with actual HTTP posting to nodes via NodeClient
- Inject `_miroir_shard` field into documents during routing
- Add proxy routes: POST /_miroir/dumps/import, GET /_miroir/dumps/import/{id}/status
- Wire up miroir-ctl dump import/status commands to call the API
- Add DumpImportPhase enum with as_str/from_str conversions
- Implement parallel flush with buffer_unordered and configurable concurrency
The import manager:
- Parses NDJSON incrementally
- Extracts primary key, computes shard_id via hash(pk) % S
- Routes to target nodes in all replica groups
- Flushes per-node buffers at batch_size intervals
- Tracks import status (phase, documents_processed, bytes_read)
CLI:
- miroir-ctl dump import --file <file> --index <uid> --primary-key <pk>
- miroir-ctl dump status --id <import_id>
Acceptance criteria:
- [ ] 500MB dump imported; no node's transient disk usage exceeds its share
- [ ] Mid-import pod failure: another pod picks up the next chunk
- [ ] Streaming vs broadcast mode produce same post-import content
- [ ] Import rate metric visible in Grafana
Closes: miroir-uhj.9
|
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| Cargo.toml | ||