Contents: Why it was slow · Timeline · Two queues, one designed · Options · The fix · Proof · Ask
1 · Why it was slow
The card-writing job sat waiting for the quarter-hour clock. A Zillow email lead with no AppFolio card queues a "create card" job with a one-minute delay. Nothing re-runs that job when the minute is up, so it waited for the next 15-minute cron. The robot itself was not slow.
"We already have a guest card in our folder." True, and not enough. PropFlow had its own record of Lori from the moment the email arrived. But Western Slope's texts do not go out through Twilio: Clara types them into the AppFolio inbox through the browser robot. A first text to a new prospect is posted on her AppFolio guest card, so the text needs AppFolio's card id, and our record carries none until the robot creates the card and writes the id back. Our own record cannot send from.
2 · Timeline, Saturday Oct 24 (Mountain)
3 · Two queues, one designed
| Guest-card jobs (today) | Held texts (today) | Held texts (proposed) | |
|---|---|---|---|
| Retry | 3 quick tries, then doubling waits (5 s × 2ⁿ, cap 5 min) | Fixed 1 min, no backoff | Doubling waits 1, 2, 4, 8, cap 15 min (same helper) |
| Budget | 1 h wall clock; 72 h when the robot or login is down | 10 tries, about 9 minutes | The existing 6-hour age limit; no try count |
| Terminal state | "failed", with time and reason | none row sits "held" forever | "failed", same shape |
| Alert | Cron alerts once on rows failed in the last 20 min; stuck alerts at 5 and 10 min | One Sentry "fatal" on the first hold, deduped per company for 7 days; nothing at give-up; no direct #alerts post | One post to #alerts at "failed", through the guest-card queue's path |
| Dead-letter list | Failed-rows count on the cron report; re-drive script | none | Same report and script |
| Wake on dependency | Event-triggered burst exists | none; only the minute sweep | Card landing drains that inquiry's held texts at once |
Why no alert reached #alerts: the first hold is a Sentry error with a constant message, so Sentry groups every episode into one issue and only forwards the first one it ever saw. Running out of tries raises nothing.
4 · Options
A · Reuse the guest-card queue (recommended)
Wire the card landing to wake the held text; run the delayed create-card job at its own +60 s; replace the ten-try cap with doubling waits inside the 6-hour limit; add "failed" with one #alerts post and the re-drive list. One PR, ~4 files, neutral PMS layer, no AppFolio literal outside the adapter.B · Move the text onto a Temporal timer or signal
The first-reply text already runs in the prospect follow-up workflow, so the workflow could sleep and wake on a card-landed signal. Cleaner long-term, but a second retry mechanism next to the outbox, touches the workflow contract, and does nothing for held texts that are not first replies.Recommendation: A. The machinery already exists, is proven on the card queue, and the fix stays inside the PMS layer. B is the right shape when the whole text door moves to Temporal, not for this fix.
5 · The fix, in four lines
- Wake. When the card job writes the card id on the inquiry, drain that inquiry's held texts right then.
- No cron wait. The delayed create-card job runs when its 60 seconds are up, not at the next quarter hour.
- Backoff, no give-up. Held texts retry with doubling waits until the existing 6-hour limit; the ten-try cap goes.
- Fail loudly. At the limit the row is "failed", one post reaches #alerts, and the row shows on the failed list with the guest-card jobs so it can be re-driven.
Fix to live behavior: no switch, live everywhere on merge. Core code gains no PMS literal; the wake hook lives in the neutral PMS layer and the AppFolio adapter stays the only place that knows about guest cards.
6 · Proof ladder
- Read-only replay of Lori's episode and every held-text row since Oct 22 at Western Slope, Situs and Camellia: same holds, texts now wake and send when the card lands, no extra sends.
- Willows live run: a fresh email lead with no card; the first text goes out within a minute of the card landing, with the text id as evidence.
- Give-up path at the Willows: a lead whose card write is forced to fail; one #alerts post, row on the failed list, re-drive sends it.
Unit-covered: the prototype's replay test with a planted defect (wake removed goes red). Not listed as proof.
7 · Ask
Go on option A as one PR? yes / hold. Prototype status: PROTO_STATUS