Pick an option on each card and press Done; your answer goes straight
back to the session. The raiser's pick is marked. "I'm not sure" is a real answer.
1
In plain terms. Our test/preview copy of production data is missing about a week's worth of information, because the tool meant to automatically catch it back up was never turned on -- it's only run once, months ago. Some of that missing week may already be gone for good. What should I do?
The full question, as raised
stage-replicator (DynamoDB stream -> propflow-stage anonymized mirror, ADR-0110) has held IteratorAge at ~25-28h since 2026-09-24 (prod WCU/day rose ~40% that day, 4-5.4M to 7.2-7.5M, from several dark backfill/index PRs landing that week). Classic DynamoDB Streams retain 24h, so records aging out before the replicator reads them are lost from the stream permanently -- the mirror is missing an unknown slice of writes from the last ~6 days. A related throttle storm (cold-start herd on the spine-index) spiked 2026-09-27 and was fixed by 4 merged PRs (2026-09-27/28); the on-failure DLQ still holds 428 stale RetryAttemptsExhausted messages from that burst (all timestamped 2026-09-28T05:51, not growing since). Despite the throttle fix landing, the backlog never recovered below 24h -- it has been flat since. ADR-0110's own documented backstop, scripts/anon-backfill/run-nightly.ts, is designed to auto-heal exactly this by re-reading prod directly (not the stream) for [latest-manifest-day, today). Confirmed via a direct DynamoDB read of ANON_BACKFILL#MANIFEST: the latest ingested day is 2026-07-24 (runId bootstrap-20260725e -- the one-time cutover bootstrap). No cron, LaunchAgent, GitHub Action, or Temporal schedule anywhere in the repo or on this mini invokes it since -- it has not run in ~68 days. Running it now would re-export/transform/load a ~68-day window from prod: real time and compute against live prod, first run in 2+ months, unknown current runtime.
Why, and what was tried
The session raising this recommends: Catch up the missing week now, and turn on nightly fixes going forward It's the documented backstop, only reads prod and writes stage (never the reverse), is single-flight-locked and self-verifying, and it stops tonight from opening the same gap again Tried first, unsuccessfully: Confirmed the alarm condition against raw CloudWatch metrics (not just the flapping alarm state): IteratorAge has sat at 25-28h for 6 days straight, not climbing further. Traced the onset to 2026-09-24 via daily WCU history on propflow-prod. Read the replicator's own handler/replicator code and ADR-0110 to understand the design. Checked Lambda errors/throttles/concurrency (all healthy right now) and the on-failure SQS DLQ (428 stale messages from a 2026-09-28 burst, not growing). Verified the 2026-09-27 throttle-storm root cause was already fixed via git log (4 merged PRs). Checked config/automations.toml, ~/Library/LaunchAgents, and did a repo-wide grep for any scheduling of the nightly backstop script -- found none. Read the actual DynamoDB manifest row to confirm the backstop's last real run was the 2026-07-25 bootstrap. Did not run the nightly catch-up myself -- it is a real, first-in-months write against a live data pipeline and the sizing/timing is a call worth making deliberately rather than firing blind in a bounded run.