miroir/notes/bf-3gfw.md

OP#5: Dump Import Distribution - Bead bf-3gfw Summary

Overview

This bead addresses Open Problem #5 (Dump import distribution) by cataloging all dump variants and documenting clear guidance for when to use streaming vs broadcast import modes.

Work Completed

1. Compatibility Matrix Documentation

Created comprehensive compatibility matrix at docs/dump-import/compatibility-matrix.md that documents:

Fully Compatible Variants (Streaming works):

• Standard documents NDJSON (Meilisearch v1.0+)
• Index settings (ranking rules, synonyms, filterable/sortable attributes, etc.)
• Primary key configuration
• Custom API keys with actions/indexes
• All Meilisearch versions from v1.0 through v1.37+
• Dictionary, proximity precision, embedders, faceting, pagination settings

Requires Broadcast Fallback:

• Tasks history (transient, not critical)
• Dumps with existing _miroir_shard field (conflict)
• Pre-v1.0 dump formats
• Internal LMDB state (not functionally significant)
• Snapshot-based dumps (.ms.snapshot)
• Enterprise edition features (sharding, replication)
• Old-style settings formats (v1.0-v1.2)
• Large single-document payloads (OOM risk)

2. Decision Tree

Documented clear operator guidance:

Is the dump a standard Meilisearch .dump file?
├─ No → Not supported (convert to .dump first)
└─ Yes → Does it contain `_miroir_shard` field?
    ├─ Yes → Use broadcast (or rename field)
    └─ No → Is it from Meilisearch v1.0+?
        ├─ No → Test with small subset first (may work)
        └─ Yes → Does it require EE features?
            ├─ Yes → Use broadcast
            └─ No → Use streaming (recommended)

3. Field Conflict Documentation

Documented the _miroir_shard field collision issue:

• Detection mechanism
• Auto-fallback behavior
• Workaround options
• Links to enhancement bead miroir-zc2.6 (configurable shard field)

4. Configuration Schema

Configuration is already in place (DumpImportConfig in crates/miroir-core/src/config/advanced.rs):

dump_import:
  mode: streaming                  # streaming | broadcast (legacy)
  batch_size: 1000
  parallel_target_writes: 8
  memory_buffer_bytes: 134217728   # 128 MiB
  chunk_size_bytes: 268435456      # 256 MiB

5. Metrics and Observability

Documented metrics for tracking fallback behavior:

• miroir_dump_import_mode{mode="streaming"|"broadcast"}
• miroir_dump_import_fallback_total{reason="conflict"|"unsupported"|"manual"}
• miroir_dump_import_conflict_field_detected_total{field}

Implementation Status

Documentation: ✅ Complete Implementation: ⚠️ Not yet implemented (see bead miroir-zc2.5)

The CLI command miroir-ctl dump import currently returns a "not yet implemented" error message pointing to bead miroir-zc2.5.

Success Criteria Assessment

Criterion	Status	Notes
Complete matrix of dump variants and their supported import modes	✅ Complete	See docs/dump-import/compatibility-matrix.md
Clear operator guidance on when to use each mode	✅ Complete	Decision tree documented
Streaming mode handles all common production dump variants	⚠️ Pending	Requires implementation and testing

Related Enhancements

The compatibility matrix documents several future enhancements tracked as child beads of miroir-zc2:

• miroir-zc2.6: Configurable shard metadata field name (addresses _miroir_shard conflicts)
• miroir-zc2.7: Pre-import validation and field conflict detection
• miroir-zc2.8: EE-to-CE dump conversion tool

Recommendations

1. For operators: Use docs/dump-import/compatibility-matrix.md as the authoritative reference for dump import compatibility
2. For implementation: See bead miroir-zc2.5 for actual dump import implementation tracking
3. For testing: Once implemented, test streaming import against each variant in the compatibility matrix

References

• Plan §13.9: Streaming routed dump import
• Plan §13.5: Two-phase settings broadcast
• Parent epic: miroir-zc2 (Phase 12 — Open Problems + Research)