Pick an option on each card and press Done; your answer goes straight
back to the session. The raiser's pick is marked. "I'm not sure" is a real answer.
1
In plain terms. PropFlow's stage environment (used for testing, not customers) is missing data because one part of its prod-to-stage sync has been stuck for over a day, and the nightly backup job that's supposed to catch gaps like this hasn't actually run in over two months. Do you want engineering time spent digging into the stuck sync now, or should it just run the backup job to catch stage up and leave the deeper fix for later?
The full question, as raised
stage-replicator (ADR-0110 prod→stage anonymized DynamoDB-stream mirror, propflow-stage-replicator-prod) has one overloaded/stuck shard: Average IteratorAge across its 100 shards is a healthy ~9 min, but Maximum has been pinned at 92-95M ms (25.6-26.4h) and slowly climbing for 20+ hours (alarm flapping OK/ALARM since at least 2026-09-30T17:03, still ALARM now) despite 4 prior merged throttle/cooldown/Node24 fixes (#8467, #12737, #12990, #13054) that resolved earlier incidents of this same alarm. Stream retention is 24h, so this shard's backlog is almost certainly already shedding un-mirrored records. The alarm's own stated backstop, scripts/anon-backfill/run-nightly.ts (ADR-0097/0110 nightly batch), hasn't advanced its DynamoDB ingestion manifest past DAY#2026-07-24 — over 2 months stale, not wired into config/automations.toml, no EventBridge/cron found running it. Separately, propflow-stage-replicator-dlq has 428 metadata-only messages (RetryAttemptsExhausted + UnclassifiedKind drops for PMS_SYNC_CURSOR/HOUSEHOLD) piling up since 2026-09-28 with no depth alarm wired (provision.sh says to run scripts/ensure-dlq-alarms.sh, which looks like it was never done).
Why, and what was tried
The session raising this recommends: Run the nightly catch-up batch, defer the shard fix The backfill closing the historical gap is low-risk and already proven (ran fine through July); isolating which shard/record is stuck needs live log correlation that doesn't fit a bounded alert-triage run, and a 2-month backfill's cost/resource footprint felt like a call that's yours to make, not mine to just run unilaterally. Tried first, unsuccessfully: Pulled CloudWatch IteratorAge Avg/Max/Min over 2-6h windows (Avg ~9min, Max pinned ~25-26h — a single-shard pattern, not a fleet-wide slowdown); confirmed via alarm history the flap has run since at least 2026-09-30T17:03; walked git log + gh pr list for stage-replicator and found 4 already-merged fixes that didn't resolve it; queried the propflow-stage ANON_BACKFILL#MANIFEST partition and found the last ingested day is 2026-07-24; checked the DLQ depth (428, growing since Sep 28) and sampled a message; ran blocked outstanding and grepped for an existing block on this — none found; checked gh pr list for an in-flight smith-fix branch — none open. Stopped short of a live fix because pinpointing the stuck shard's actual record needs deeper per-shard log correlation than a bounded triage run should improvise, and kicking off a multi-month backfill unilaterally is a resource/cost call, not mine to make solo.