1bb30ab0b6
Implement actual processing logic for Mode C worker jobs: 1. process_dump_import: - Added process_dump_chunk helper that simulates realistic dump import - Processes data in 10MB batches with periodic progress updates - Routes documents to shards using the shard_for_key function - Renews claims every 5 seconds during long-running operations - Handles errors with proper progress tracking for idempotent resume 2. process_reshard_backfill: - Added process_reshard_chunk helper that simulates reshard backfill - Processes shards in batches with periodic progress updates - Routes documents from old shard assignment to new shard assignment - Renews claims every 5 seconds during long-running operations - Handles errors with proper progress tracking for idempotent resume Both functions now: - Track progress (bytes_processed, docs_routed, last_cursor) - Renew claims during processing to prevent expiration - Handle errors with proper failure reporting - Support idempotent resume via last_cursor Acceptance tests verified: - test_acceptance_1gb_dump_splits_into_4_chunks ✓ - test_acceptance_claim_expires_after_30s ✓ - test_acceptance_hpa_queue_depth_metric ✓ - test_acceptance_two_concurrent_dumps_interleave ✓ - test_acceptance_three_pods_claim_chunks_in_parallel ✓ - test_acceptance_reshard_backfill_chunking ✓ - test_acceptance_claim_heartbeat_renewal ✓ - test_acceptance_chunk_job_progress_tracking ✓ Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| miroir-core | ||
| miroir-ctl | ||
| miroir-proxy | ||