I'm parked on you

Pick an option on each card and press Done; your answer goes straight back to the session. The raiser's pick is marked. "I'm not sure" is a real answer.

1

In plain terms. Should I dig into why some records are stuck in our test-database mirror before clearing them, or just clear them now and move on? Separately, its nightly safety-net job hasn't run in over 2 months — want that turned back on?

The full question, as raised

stage-replicator-iterator-age has read MAXIMUM IteratorAge ~26h continuously since 2026-09-28 (exceeds the 24h DynamoDB Streams retention — the alarm's own documented 'silent gap' condition), while Average/Minimum IteratorAge stay ~0-0.4h (sampleCount ~40-54k/3h) — so this is NOT a systemic backlog, it's a small number of shards stuck on specific records. 14-day trend: Max climbed linearly from ~0h on 2026-09-20/21 to a peak of 28.3h by 2026-09-27, then dropped to ~26h right after a mass DLQ-dispatch burst at 2026-09-28 05:51-05:52 UTC (428 individual records, one per shard, batchSize:1, all condition=RetryAttemptsExhausted after 9 invoke attempts, executedVersion 794) — and has sat flat at ~26h ever since, never clearing. DLQ depth is static at 428 (not growing) and its own propflow-stage-replicator-dlq-depth alarm has been ALARM/armed the whole time. On-failure destination carries metadata only (shard+sequence, no row body, by PII-hygiene design), so I cannot see WHAT shape of record is failing without a deliberate redrive-to-debug. Separately: the alarm description says 'nightly batch is the backstop' (ADR-0110's scripts/anon-backfill/run-nightly.ts) but the stage manifest's latest ingested day is still DAY#2026-07-24 (the one-time bootstrap, ingestedAt 2026-07-25) — no EventBridge rule, Temporal schedule, GitHub cron, or launchd job invokes it anywhere I could find, and ADR-0110 is still Status: Proposed, never accepted. So the documented self-heal for exactly this failure mode has not run in 9+ weeks.

Dig in firstthe raiser's pick Clear it now I'm not sure — go find out

Why, and what was tried

The session raising this recommends: Dig in first
The mirror is 99%+ healthy (avg/min lag is minutes) so this isn't urgent capacity pressure, and the DLQ is static not growing — there's room to find the real cause instead of papering over it with a reset that also nobody can undo once records roll off the 24h stream.
Tried first, unsuccessfully: Pulled IteratorAge Min/Avg/Max/SampleCount over 72h and 14d, alarm state history, DLQ depth + message samples (timestamps/shardIds/conditions), ESM + function config, Lambda Errors/Throttles/ConcurrentExecutions, propflow-stage write-throttle/capacity metrics, stream shard counts (550 total/74 open), the stage manifest's latest ANON_BACKFILL ingested day, and searched the repo + EventBridge + automations.toml for any schedule invoking run-nightly.ts — found none. Could not go further without either redriving a DLQ message (a write action) or running the multi-week nightly-batch catch-up (hours, real prod RCU, raw PII briefly on disk) — both felt like a human's call given the scope.

880f2e26 is parked on this.

Pick an option above, then press Done.