I'm parked on you

Pick an option on each card and press Done; your answer goes straight back to the session. The raiser's pick is marked. "I'm not sure" is a real answer.

1

In plain terms. Should we let 3 of our automatic problem-checks run every 15 minutes instead of every 5, to permanently fix a system that keeps running out of time — even though it means those 3 alerts could arrive up to 10 minutes later? This safety system checks the whole business every 5 minutes for things like stuck maintenance jobs and missing rental units. It's run out of time 3 times now. I just gave it more computing power to buy a few more weeks, but the lasting fix is slowing its 3 slowest checks down.

The full question, as raised

propflow-metrics-alerts-checker-prod's duration alarm has fired 3 times (08-31, 09-12, 09-29), each time fixed by adding capacity (concurrency, then memory 256->512MB, now 512->1024MB in PR #13795). Root cause: table-wide DynamoDB read pressure keeps growing AND new probes keep being added to this one 5-minute EventBridge lambda (unit-count-drift added 09-13, the day after the last memory bump). Durable fix: move the 3 heaviest checks (turnover-stalls, unit-count-drift, conversation-tenant-unknown) off the 5-min cadence to ~15min. This changes on-call assumptions documented in docs/runbooks/sentry-turnover-stall.md ('a second 5-minute tick with the same turnover ID confirms it's a real stall') and docs/runbooks/sentry-appfolio-unit-count-drift.md ('the next 5-minute tick should log recovery'). Should we make this cadence change (accepting ~10min added detection latency on those 3 alerts) now, before a 4th recurrence, or keep buying capacity each time?

Slow those 3 checks down — alerts arrive a bit laterthe raiser's pick Keep buying more computing power each time Hold — let's discuss first I'm not sure — go find out

1 other session waiting on this same answer.

Why, and what was tried

The session raising this recommends: Slow those 3 checks down — alerts arrive a bit later
Buying power only delays the same overload a few weeks each time; slowing the checks is the fix the code's own comments already flagged.
Tried first, unsuccessfully: Bumped memory 512->1024MB (3rd time) to relieve the immediate overload and opened PR #13795 for that; before going further and changing check cadence myself, raising it here since two on-call runbooks assume 5-minute confirmation and that's a product/ops tradeoff, not mine to decide alone.

587629e4 (adopted from 5c076feb) is parked on this. 1 other session waiting on this same answer.

Asked by more than one session. Your answer above goes back to all of them.

587629e4 asked it this way:
propflow-metrics-alerts-checker-prod's duration alarm has now fired 3 times (08-31, 09-12, 09-29). Each fix bought 2-4 weeks: 08-31 went concurrent, 09-12 bumped memory 256->512MB, 09-29 (PR #13795, merged) bumped memory 512->1024MB. Root cause is structural, not a regression: table-wide DynamoDB read pressure keeps growing (ConsumedReadCapacityUnits ~10M/6h on 09-22 -> ~20-46M/6h by 09-29) and a new probe (unit-count-drift, PR #8365) was added the day after the last fix. The durable fix is moving the 3 heaviest checks (turnover-stalls, unit-count-drift, conversation-tenant-unknown — all full-table DDB scans) off the 5-minute EventBridge tick onto a slower, separate schedule. Blocker: two on-call runbooks document 5-minute-tick behavior for two of those checks (sentry-turnover-stall.md: 'a second 5-minute tick with the same turnover ID confirms it's a real stall'; sentry-appfolio-unit-count-drift.md: 'the next 5-minute tick should log...'), so slowing them changes documented detection latency for stall/drift alarms, severity=warning not critical.

Pick an option above, then press Done.