I'm parked on you

Everything I could decide myself is decided. What's left needs you. Each one carries the recommendation of the session that raised it — if you agree, pick it and press Done. A model's opinion, where there is one, appears on the card as a receipt line; cards with no receipt were never put to a model. "I'm not sure" is a real answer and becomes work for me. You don't need to open the session — pressing Done sends your answer back and it picks up.

1

In plain terms. Should we let 3 of our automatic problem-checks run every 15 minutes instead of every 5, to permanently fix a system that keeps running out of time — even though it means those 3 alerts could arrive up to 10 minutes later? This safety system checks the whole business every 5 minutes for things like stuck maintenance jobs and missing rental units. It's run out of time 3 times now. I just gave it more computing power to buy a few more weeks, but the lasting fix is slowing its 3 slowest checks down.

The full question, as raised

propflow-metrics-alerts-checker-prod's duration alarm has fired 3 times (08-31, 09-12, 09-29), each time fixed by adding capacity (concurrency, then memory 256->512MB, now 512->1024MB in PR #13795). Root cause: table-wide DynamoDB read pressure keeps growing AND new probes keep being added to this one 5-minute EventBridge lambda (unit-count-drift added 09-13, the day after the last memory bump). Durable fix: move the 3 heaviest checks (turnover-stalls, unit-count-drift, conversation-tenant-unknown) off the 5-min cadence to ~15min. This changes on-call assumptions documented in docs/runbooks/sentry-turnover-stall.md ('a second 5-minute tick with the same turnover ID confirms it's a real stall') and docs/runbooks/sentry-appfolio-unit-count-drift.md ('the next 5-minute tick should log recovery'). Should we make this cadence change (accepting ~10min added detection latency on those 3 alerts) now, before a 4th recurrence, or keep buying capacity each time?

The session raising this recommends: Slow those 3 checks down — alerts arrive a bit later
Buying power only delays the same overload a few weeks each time; slowing the checks is the fix the code's own comments already flagged.
Tried first, unsuccessfully: Bumped memory 512->1024MB (3rd time) to relieve the immediate overload and opened PR #13795 for that; before going further and changing check cadence myself, raising it here since two on-call runbooks assume 5-minute confirmation and that's a product/ops tradeoff, not mine to decide alone.

Slow those 3 checks down — alerts arrive a bit laterthe raiser's pick Keep buying more computing power each time Hold — let's discuss first I'm not sure — go find out

5c076feb is parked on this.

Pick an option above, then press Done.