Incident retro · 2026-07-31

Renewal Outreach Incident 2026-07-31 — Retro & Gap Audit

Notice-to-vacate tenants received a holdover renewal SMS. Blameless retro plus a full gap audit of the renewal outreach paths. Fix merged 31 Jul 2026 (PR #5152).

2 wrong sendstwo notice tenants at Camellia, 8:30 AM MT
~7 weeksexposure since the lapsed nudge shipped (2026-06-09)
0 othersfull 30-day audit of renewal outbounds found no other wrong sends
8 gapsfound in the companion audit: 3 HIGH, 1 MED-HIGH, 2 MED, 1 LOW, 1 in-flight

Incident summary

On 2026-07-31 at 8:30 AM MT, the daily renewal auto-start scan sent a "your lease ended and we haven't received your signed renewal — you can still renew at $X/mo" SMS (renewal_expiry_nudge_sms, milestone lapsed) to two notice tenants at Camellia Apartments who had filed Notice to Vacate. For one of them the PM had canceled the renewal offers in AppFolio on 7/07 and move-out had started with a 7/31 move-out date — Clara still quoted a renewal rate on their move-out day.

PropFlow's own data was correct: both occupancy rows carried reasonForNonRenewal: "Notice to Vacate filed", renewalState: "declined", and unit status notice. The sender never read those fields.

A full 30-day audit of all renewal outbounds confirmed these two sends were the only wrong ones. All other reminders predate signed renewals or went to genuinely expiring, no-notice tenants.

Blameless — every finding below is a property of the system's structure, not of anyone's care. The gate was built correctly; it was built on one of two doors.

Retro

What the code actually does

src/lib/domain/leasing/renewal-auto-start.ts has two tenant-facing send paths and one gate:

PathLineGated on non-renewal stamp?
Holdover branch → sendScanMtmWarning(milestone: 'lapsed'), branch ends continue at 910~873–892No
Final-notice band → sendScanMtmWarning(milestone: '15d') (live-workflow saga, daysLeft <= 15), branch ends continue at 545~519–541No
Normal start → ADR-0108 decision spine (declinedStampGate reading occ.reasonForNonRenewal)1010Yes

Both warning senders sit above the spine and continue past it. sendScanMtmWarning itself checks arm switches, receipt dedup, phone, dry-run, and quiet hours — but never reasonForNonRenewal, never lastObservedNoticeAt/leaseMoveOutDate, and never isRenewalFullySuppressed (ADR-0052 suppression), which also only runs inside the spine. The holdover branch is not one missed gate but an entire un-gated send path.

Why PR #4354 (the NTV gate) missed it

gh pr view 4354: "gate the autonomous path on the tenant's non-renewal stamp", merged 2026-07-23, commit 6d093e27d.

Why tests missed it

renewal-auto-start.test.ts covers the holdover branch thoroughly (~20 cases) and the declined-stamp gate (3 cases) — but the suites are disjoint by fixture: holdover cases never set reasonForNonRenewal; stamp cases never lapse the lease. The missing case cost one line of fixture. No test asserts a negative invariant over the whole function ("a stamped tenant produces zero sends, regardless of window").

Design-doc trail

Timeline

Prevention recommendations (ranked)

  1. Move the disposition check below the tenant fetch, not into each branch — resolve reasonForNonRenewal / notice fields once per tenant and continue there, so every current and future branch inherits it.
  2. Add a whole-function negative invariant test: for {in-window, 15d-band, lapsed} × {stamp set}, assert zero sendSms / startRenewalWorkflow / triggerRenewalSync calls.
  3. Make "tenant-facing send" a single chokepoint in the file — every outbound routes through one helper running disposition + ADR-0052 suppression.
  4. Change the review question from "is this gate correct?" to "what else sends?" — gate PRs must enumerate every send/dispatch site in the touched function, marked gated/not-gated.
  5. Give the spine an entry point for the warn paths ('holdover-warn' in SpineEntryPoint), making the next omission a type error rather than a silent continue.

Gap audit — renewal outreach

Method: read renewal-auto-start.ts end to end; traced every send/write call site through its gate chain; verified each claimed-missing gate by grepping the whole call chain. The known holdover-NTV bug (fixed in PR #5152) is excluded but used as the template.

Ruled out (checked, no gap)

HIGH#1 — Post-conversion warn-key rotation re-texts the MtM warning and can re-run the billing conversion

renewal-auto-start.ts:682 (warnKey), :699 (idempotencyKey), :770 (reoffer)

The proof-of-warning receipt is keyed on whichever saga is open at scan time (openExistingSaga?.id ?? liveWorkflowSaga?.id ?? holdover-${lease.id}), never on the lease — and the conversion itself rotates that key (closes the old saga, mints a new one via reoffer). Next-day re-scan misses the receipt under the new saga id → re-sends the "your lease rolls to month-to-month at $X" SMS to an already-converted tenant; 14 days later the convert re-runs under a different idempotency key: a second convertToMonthToMonth write, a second cancelRenewal (voiding the re-offer it just created), a third saga.

Missing gate: key proof-of-warning and conversion idempotency on (leaseId, leaseEndDate) — the cycle — as claimRenewalMint already does. Confirm against a converted prod occupancy before sizing (whether the lapsed lease stays in-branch depends on rent-roll sync write-back).

HIGH#2 — moveout_scheduled is a detector wired to nothing

external-reconcile.ts:225-235, consumed at external-apply.ts:178-183

The reconciler explicitly classifies "PMS has a move-out scheduled while a renewal saga is still open" — the precise backstop for a move-out the ntv_filed path missed — persists it 'observed', and nothing reads it. If the NTV emit fails or the tenant row doesn't resolve, reasonForNonRenewal is never stamped, and the live saga keeps firing reminder steps at a tenant who is moving out. The system knows and says nothing.

Fix: drive wrapUpWorkflow (machinery exists at external-apply.ts:255) or add to WrapUpEventType. Cheap.

HIGH#3 — PM cancels the offer in AppFolio; the reminder ladder keeps nudging the tenant to sign it

external-apply.ts:186-205

offer_cancelled_externally alone is persisted 'proposed' and by design never closes a saga or wraps up a workflow. An AppFolio-UI cancel is invisible to the workflow (no L4 callback), so the 30d/15d nudges keep firing with the proposed rent and a portal link to an offer that no longer exists. Inconsistency: the scanner honors both applied and proposed external events as a hard skip; the workflow's send path honors neither. Fix: same wrap-up machinery. Cheap.

MEDIUM-HIGH#4 — 90-day re-warn loop for holdovers the converter refuses for reasons other than not_warned

renewal-receipt-ttl.ts:44-51, renewal-auto-start.ts:1547

The "no re-warn loop" argument assumes conversion succeeds ~14 days after warning. If conversion permanently fails (e.g. the AppFolio 422 end-cap loop, unsupported_pms, already_renewed), the receipt expires at day 90 and the tenant is re-warned with the same "your rent goes to $X" text, ~4×/year, indefinitely. The receipt conflates idempotency record and proof-of-warning clock; a permanently-failing conversion needs a circuit breaker (holdoverFailed is counted but never escalated).

MEDIUM#5 — The send chokepoint's declined-stamp gate is fail-OPEN where the scanner's is fail-CLOSED

agent-jobs.ts:322-341

G8 at the chokepoint — the last gate before send_all_to_tportal, which delivers to the tenant portal before any comms suppressor — is failOpen: true and entirely skipped when payload.occupancyId or organizationId is falsy. A transient DDB throttle during a prepare-retry pass for an NTV tenant delivers an offer to the portal of a tenant who gave notice — the exact 2026-06-09 unit-level harm class the module exists to prevent. Fix: fail-closed, and add a lease-cycle gate at the chokepoint.

MEDIUM#6 — HOLDOVER and TRANSFERRED terminals don't resolve the lease cycle

renewal-classify.ts:255-278

A saga closed HOLDOVER by the reconciler (the mapping for both renewed_externally_charges and mtm_managed) leaves the lease-cycle gate passing. A PM who puts a tenant on managed month-to-month before lease end can get a fresh unsolicited offer cycle started by the scanner. Mitigated when the original cycle was scanner-minted (claimRenewalMint on (leaseId, leaseEndDate)), not when it came via the PM /outreach route. Confirm whether /outreach stakes a mint claim before sizing.

IN-FLIGHT FIX#7 — Holdover branch runs zero shared gates

renewal-auto-start.ts:593-910

Beyond NTV/declined, the branch also skips G9 suppression (isRenewalFullySuppressed), G13 transfer, and G11 signer-email; the reoffer closure calls startRenewalWorkflow directly with no gate cascade. The in-flight fix puts the spine in front of the whole branch.

LOW (liveness)#8 — Quiet hours can permanently block the warning

renewal-auto-start.ts:1491

If the daily cron's fixed UTC slot lands inside a property's quiet window, the warn never sends, not_warned holds forever, and conversion never completes; logs only at info. The workflow ladder defers within the day (resolveSendWindowDelayActivity); the scan has no equivalent.

Strong parts (verified, no gap)

Suggested fix ordering

#2 and #3 first (cheap, reuse wrapUpWorkflow). #1 needs scoping against a real converted occupancy — it can compound into repeated PMS billing writes. #5 is a small flag flip + gate addition. #7 shipped in PR #5152.

Status: Fix merged 31 Jul 2026 — PR #5152 — covers finding #7 and the incident's root cause. The remaining findings are open.

What shipped, beyond the reported bug: the lapsed/holdover branch now sits behind a first-class holdover decision-spine entry point rather than running no gate cascade at all, so G8 (non-renewal stamp), G9 (suppression) and G13 (transfer) are enforced before it can text or re-offer. The declined-stamp predicate was widened to read the canonical lease's lastObservedNoticeAt / leaseMoveOutDate alongside the occupancy stamp, and every renewal outbound path — including the triggerRenewalSync send chokepoint — now resolves through that one predicate. Review also closed three holes the original report did not name: the scan's warning SMS never consulted suppression, the send chokepoint still evaluated the narrow predicate, and the 15d warn band silently degraded to the occupancy stamp alone whenever the lease join missed. A drift guard pins the invariant that no renewal outbound path can skip the gate.
PropFlow Docs