A plain-English look at how PropFlow runs lease renewals today, why Wednesday morning went wrong, what's still breaking for real residents, and the lab plan to catch it before it ships — for Fede to decide.
2026-09-03 (architecture) · 2026-09-04 (quality) · covers the renewal system only (not maintenance or turnovers) · written for a non-engineer
Status — executed 2026-09-04| What changed | Plain-English result |
|---|---|
| Second starter removed | The daily job that scans every lease is now the only thing allowed to start a renewal — the newer "cohort walker" can no longer start one at all. |
| Willows can now prove the real path | A new test in the internal test environment (Willows) runs the actual production start sequence end to end, not a stand-in. It's opt-in in CI so it can run without slowing down every change. |
| One rulebook, checked automatically | Every entry point (daily job, staff buttons, watchers) now reads eligibility/timing/email rules from one shared place, and a build-time check fails if a second copy ever creeps back in. This already caught and closed two bugs where a placeholder email address had snuck into the logic. |
| Renewal records are shared safely | Reads of renewal records are now scoped per property but shared underneath, with an automated check that blocks any code from accidentally scanning every property's data at once, plus a load test proving at least 3x headroom at 10x today's volume. |
| Durable message history | Every text, email, and call now logs to a permanent history table with its delivery outcome (scripts/comms-history.ts) — no more "was this actually sent?" guessing. |
| Willows is now a safe sandbox | Willows talks to a recording stand-in for AppFolio instead of a real database connection — it is now physically incapable of writing to a customer's real property records. This is live in production. |
| 30-day replay test on every renewal change | Every renewal pull request now automatically replays 30 days of real past cases. When the three Sept 3 failure types were deliberately re-injected, this test caught all three. |
What the proof work turned up along the way: 66 of 111 Camellia renewal records were missing their AppFolio reference number — a write-back bug, now fixed. A separate check for "offer IDs getting mixed up when several renewals start at once" found no such contamination: all 59 offer IDs mapped cleanly to one unit each. Texts sent from the PM-facing page are now logged like every other message.
Written 2026-09-04. Everything below is either read out of production, read out of the code, or read out of a decision doc. Where something is inferred rather than confirmed, it says so. Unit numbers only — no resident names, emails, or phone numbers appear anywhere below.
Contents
TL;DR
Five decisions need Fede (jump to Section 3.7): what happens when nobody answers; whether an escalation may age; whether an emailed notice can fire a move-out unattended; whether the replay check becomes mandatory and when; and whether the Willows keeps writing into JP's real AppFolio.
How to read a card. "What the resident or PM saw" is the human story. "What actually happened" is what the records show. Then a label:
Unit numbers only. No names.
What the resident/PM experienced. Nothing. Five residents were never contacted about their renewal. The property team got one email about each, months ago, and never heard about it again. Two of those leases end September 30.
What actually happened. AppFolio refuses to create a renewal offer when the signer has no email on file. Our system asks once, gets refused, emails the property team once, and writes a "we already told them" marker that lasts 65 days. A retry job runs every 30 minutes, sees the marker, and skips — it has reported "retried 0, escalated 0 of 7" for weeks. Unit 416's flag has sat untouched since July 7 — 59 days. Units 204 and 416 would get their second-ever reminder around September 16, two weeks before those leases expire. Clara also never made a second attempt to ask the resident directly — she texted once in mid-July, nobody replied, and that was the end of it.
Classification: DESIGN FLAW. Every piece worked as written. The design says "tell a human once, then wait 65 days" — a cadence that only makes sense if humans always act on the first email. Nothing measures whether they did.
Prior art. The once-per-65-days page is deliberate anti-spam (ADR-0108, accepted 2026-07-23). Fede already ruled on the right cadence elsewhere — "reminder 3 times like we do today with leases. Business days only. Once per day max." (2026-08-31) — but it was never applied to held renewals. There's also an open decision from this page's 2026-09-03 review (Q5): should Clara text or call for a missing email before paging a PM? Recommendation was yes, ~70% reduction in the PM queue — still open.
Status today: not fixed. The 65-day marker and one-shot page are both live. The fix Fede has effectively already specified (3 reminders, business days, once a day) exists in the codebase — for a different notification type.
Evidence: retry-failed-prepare.ts:818 (65-day escalation TTL); seven live retry-escalation rows; unit 416's PM flag set 2026-07-07T05:24:49Z.
What the PM experienced. A "flagged for the property manager" marker that means nothing — it never expires, never re-fires, and never shows up as overdue.
What actually happened. The field that records "a PM was flagged" has exactly one writer and zero readers anywhere in the codebase. Unit 416's has been set since July 7. Unit 608's since May 16.
Classification: INCOMPLETE. Somebody built the write half of a follow-up mechanism and never built the read half.
Prior art. None — genuine blank.
Status today: not fixed.
Evidence: sole writer src/lib/temporal/activities/renewal.ts:2432; no readers.
What the resident experienced. Nothing at all. This resident has never been contacted about their renewal, not once, since July 4. Their lease ends September 30.
What actually happened. AppFolio returned an error saying the lease could not be sent to the tenant portal. Our error classifier files that whole family of responses as permanent — never retry, ever — and cannot tell "this resident has no portal account" apart from "AppFolio genuinely rejected this on business grounds." It stopped, emailed a human once (July 8), and never moved again. That was 62 days ago.
Classification: DESIGN FLAW. The classifier does what it was designed to do. The design conflates a fixable setup problem with a genuine refusal, and pairs it with a one-shot notification nothing chases.
Prior art. The transient-vs-permanent error split is a deliberate design in the retry apparatus. Nothing decided what "permanent" should mean when the permanent thing is a missing portal account — genuine blank.
Status today: not fixed.
Evidence: lastError = PMS_REJECTED 422 on the tenant-portal send route; dispatch.ts:216-231, classify-renewal-error.ts.
What the resident experienced. Nothing arrived. Their lease ended August 31 with a live, unsigned offer sitting in AppFolio.
What actually happened. Three channels were tried and all three failed at the carrier: the text came back undelivered, the renewal letter came back undelivered (Twilio error 30005 — the handset is unreachable), and the phone call went unanswered. Fifteen messages were written; not one is confirmed delivered. No flag was raised to anyone. The renewal workflow hit its 60-day timeout on August 7 and died quietly.
Classification: BUG — with a twist. A module exists that computes exactly this — which contact points are dead, whether all of them are, which channels are untried — and it has zero callers. Channels are chosen once when the renewal starts and never re-evaluated; delivery failures are stamped onto the lease record and read by nothing.
Prior art. Directly and explicitly deferred. ADR-0036 (2026-05-27) was written after unit 608, where the phone on file belonged to a former roommate and every phone channel failed silently. It states outright that it does not cover "has a phone, but it's dead or wrong" — that deferred case is unit 111, three months later, exactly the same shape. A kill switch was built 2026-05-17 after a test bug fired 30+ real texts to a canary phone in five minutes — so we do watch for too many sends. We have never watched for zero delivered.
Status today: not fixed. Suspected root cause is confirmed at the symptom level; the code path was never wired.
Evidence: lease channel outcomes (SMS undelivered, letter MMS undelivered, voice attempted/no-answer); PM flag unset; src/lib/domain/leasing/renewal-reachability.ts — evaluateRenewalReachability has no callers on main; channels derived once at renewal-auto-start.ts:1220.
What the PM would experience. A PM adds the missing email in AppFolio — the exact thing we asked them to do — and nothing happens. Nothing notices.
What actually happened. Every phone and email write for a resident funnels through one save function, and that function discards the "did this actually change?" result — the natural hook for "the thing that was blocking this renewal is now unblocked, try again."
Classification: INCOMPLETE. The information exists at the exact right moment and is thrown away.
Prior art. None — genuine blank.
Status today: not fixed.
Evidence: saveTenants, src/lib/data/dynamo/property.ts:681-700 — phone/email "did it change" result discarded.
What the resident experienced. An offer, two follow-ups, two expiry warnings, then nothing. Their leases ended August 31. Nobody from the property ever called.
What actually happened. The renewal workflow ran its full ladder — offer July 10, nudges July 12 and 19, expiry nudges August 1 and 16 — then completed normally at midnight on August 16 and paged nobody. The lease then expired with a live unsigned offer and no human ever knew.
Classification: BUG at the code level, DESIGN DECISION at the policy level — and the two conflict. On 2026-07-17 the "email a PM when outreach runs out and the resident has gone silent" behavior was deliberately removed (Fede is named as the decider in the ADR-0100 addendum), replaced by an office email that only fires when the lease actually flips to month-to-month — which is the second half of this failure (see B2). This directly contradicts Fede's later position that "everything should be automatic" and that a renewal outcome should never just evaporate. It needs re-opening explicitly, not quietly reversing.
Prior art. ADR-0100 addendum, 2026-07-17, decided by Fede. His later rulings contradict it.
Status today: not fixed. The workflow still exits silently.
Evidence: workflow completed 2026-08-16T00:00:07Z for units 420/504/522, no subsequent event; the outreach-exhaustion-quiet-exit commit is dated 2026-07-17.
What the resident experienced. Two texts in August warning them the lease would roll to month-to-month at a higher rate. Then the lease ended — and the rollover never happened. The AppFolio record shows an expired lease plus an unsigned offer, no month-to-month arrangement, nobody billed correctly, and nobody told.
What actually happened. The daily conversion job ran every day from August 31 through September 4 and logged "already renewed, skip" for both units each time. Suspected cause: the job's first check asks AppFolio "is there a renewal on this occupancy?" and sees our own unsigned July 10 offers, concluding the renewal is already handled. No PM was ever notified.
Classification: BUG. The "already handled" check cannot distinguish a signed renewal from an offer we sent that nobody ever answered.
Prior art. ADR-0046 (2026-06-03) is the original "what happens when nobody answers" design: auto-convert the lapsed unsigned lease to month-to-month at a premium rate, update the rent in AppFolio, cancel the stale offer, and auto-send a fresh offer next month. Cancelling the stale offer is exactly the step that would have unblocked these two units, and it's the step that isn't running. The "stand down when AppFolio already shows it handled" rule is separately correct and matches Fede's 2026-09-03 follow-the-PMS-signal ruling — the flaw is the definition of "handled."
Status today: not fixed.
Evidence: daily holdover-scan logs 8/31–9/4, both units; AppFolio last sync 9/1 shows expired lease + unsigned offer, no month-to-month; escalation ledger empty.
What the resident experienced. On August 3 she emailed asking for $1,000 instead of $1,150, pointing out that $1,000 is what the property's own website advertises for comparable studios. Clara told her twice that the team had everything they needed. No human ever replied. On August 16 an automated reminder told her to sign at $1,150 or roll to month-to-month at $1,250 — the exact price she'd asked to move off. She signed at $1,150 anyway on August 30.
What actually happened. Detection worked perfectly — Clara correctly recognized the counter-offer and marked the renewal "negotiating." The escalation then went out as one email to one shared mailbox, twice, with no tracked item, no deadline, no reminder, no expiry. There is no record of a "matter" ever being opened for this resident.
Classification: DESIGN FLAW. Escalation is fire-and-forget by construction. There is no such thing as an overdue escalation in this system.
Prior art — the most decided-and-least-built item in the whole report. ADR-0104 (accepted 2026-07-21) already built the right machine for a different job: an initial notice plus up to three reminders about 24 hours apart, per-property, default ON — covering applications, lease countersigns, and renewal countersigns. A fourth kind, "forwarded question" (added 2026-07-24), has no safety net and is named as a gap in the ADR itself — a resident's counter-offer is a forwarded question. Fede restated the cadence on 2026-08-31 and the philosophy on 2026-08-13 ("escalate as little as possible… when we escalate, we create a policy and learn, and the next time we don't"; "I'll just nag you every day till you reply"). The counter-pressure is real too: on 2026-08-13/14 Fede overturned time-based alerting elsewhere after eleven leases tripped a 24-hour alert for normal wait time — so "add a timer to everything" isn't the answer; the distinction is waiting on a document versus waiting on a person who was asked a question. The reminder-suppression half was addressed by PR #5828 ("hold reminders once a resident answered or a human owns it"), merged 2026-08-17 18:32Z — 28 hours after her wrong-price nudge went out.
Status today: half fixed, and the fixed half is unverified against a live case. The wrong-price nudge can't happen the same way now. The unchased escalation absolutely can — see B4, happening right now.
Evidence: negotiating-since 2026-08-03T14:06:55Z; two escalation events 08-03/08-06; zero tracked-matter rows; PR #5828 merged 2026-08-17.
What the resident experienced. She asked for a physical copy of a 6-month renewal instead of signing digitally. Fourteen days of silence. Her lease ended August 31.
What actually happened. Same shape as B3, live and ongoing. The escalation was recorded August 21 — no tracked item, no re-page, no owner. Compounding it: the term change she needs (12 months to 6) was blocked in AppFolio on August 5, because AppFolio won't change terms while a previous offer is still open and uncancelled.
Classification: DESIGN FLAW (same root as B3), plus a second DESIGN FLAW: an AppFolio "cancel the old offer first" refusal is treated as a dead end rather than a two-step operation.
Prior art. Same blank as B3.
Status today: not fixed. This one is live and needs a person this week.
Evidence: escalation recorded 2026-08-21; term-change blocked 2026-08-05 ("previous document must be cancelled"); no tracked-matter row.
What a resident would experience. A PM answers their question — and the renewal goes quiet for the rest of the cycle, because reminders are still suppressed.
What actually happened. The hold PR #5828 added never lifts early. There is no "the escalation is resolved" signal for it to listen for.
Classification: INCOMPLETE. Documented in the fix's own source file.
Prior art. PR #5828, 2026-08-17 — the gap is named in that PR's own code.
Status today: not fixed.
Evidence: src/lib/domain/leasing/renewal-outreach-hold.ts header comment.
What the PM experienced. A renewal that is done in the system of record but still sitting "escalated / unsigned" on the PropFlow page, inflating every stuck count.
What actually happened. PropFlow's copy of that lease has not re-synced from AppFolio since July 10 — eight weeks. The signature simply isn't visible on our side.
Classification: BUG. A stale-sync gap, and nothing monitors sync freshness per lease.
Prior art. Directly contradicts Fede's 2026-09-03 ruling: when AppFolio shows a renewal handled, Clara closes out automatically — no confirm button. That ruling exists; this path doesn't honor it because the data never arrives.
Status today: not fixed.
Evidence: lease last-synced 2026-07-10T19:30Z; saga still human-review-pending, not closed, as of 2026-09-04.
What the PM experienced. A renewal PropFlow shows as live and awaiting a resident, when AppFolio already recorded it "Did Not Renew."
What actually happened. The external-actor reconciler — whose job is to notice things done directly in AppFolio — did notice. It wrote an evidence record and left it in status "proposed." Last re-observed September 4. Never applied.
Classification: INCOMPLETE. The observe half of ADR-0091 phase 2 works; the apply half is unproven in production.
Prior art. ADR-0091's rollout plan explicitly stages observe-only → board display → saga close-out. This is stuck between phases — again contradicts the follow-the-PMS-signal ruling.
Status today: not fixed — worth a targeted look, since the fix may be one condition.
Evidence: external-event record, unit 517, status "proposed," last seen 2026-09-04.
What the PM experienced. A row on the renewals board asking them to click "confirm" on something AppFolio already settled.
What actually happened. The external-confirm lane is implemented as a one-click human confirmation, not an auto-close.
Classification: DESIGN DECISION, now overridden. A deliberate, conservative rollout choice (ADR-0091 phase 2, human-in-the-loop confirm). Fede's 2026-09-03 ruling explicitly overrides it — system-of-record facts get followed, not confirmed.
Prior art. ADR-0091 (design) vs. the 2026-09-03 ruling (override) — the ruling is newer and wins.
Status today: not fixed — policy and implementation are out of step.
What the resident/PM experienced. On August 31 she emailed a proper 30-day notice saying she'd be out September 30. Nothing happened: no renewal close-out, no turnover started, no AppFolio move-out draft.
What actually happened. The classifier read the email perfectly — correctly tagged it as a formal notice to vacate, effective September 30. The code that turns that into action asked for the date, and the only date on file was the literal string "September 30th." The date parser accepts only strict YYYY-MM-DD. It returned nothing, and the firm path bailed with "no move-out date." The PR shipped specifically to fix this case (#6761) contains an inline comment asserting this exact signal carries an ISO date. It does not — the fix was written against a fixture that doesn't match the real record.
Classification: BUG (the date parser), on top of an INCOMPLETE piece — the classifier isn't required to normalize dates to a machine format, so the parser has nothing to work with.
Prior art. PR #6761, 2026-09-01 — attempted, doesn't actually close the case.
Status today: not fixed. Merged, not effective.
Evidence: date regex /^\d{4}-\d{2}-\d{2}$/ at signal-tenant-renewal-bridge.ts:152-163; the raw signal's date field literally reads "September 30th"; PR #6761's own comment at L141-143.
What the PM experienced. The renewal page showed the PM's own reply hanging under nothing — the resident's original notice was invisible.
What actually happened. A "needs review" inbound is a terminal skip — Clara correctly writes no reply, and before the fix wrote no message record either; only a short summary survived. Four PRs merged September 1 to fix this. The code is live on main. The effect for unit 614 is still broken — all 26 message rows on her renewal thread were checked, none carries the new recorder's marker, nothing was added on September 1 or 2. This directly contradicts two Slack posts that declared it fixed and readable.
Classification: BUG, and a verification failure. The fix was declared done without checking the actual record.
Prior art. PRs #6746, #6749, #6750, #6752, all 2026-09-01.
Status today: fixed in code, NOT verified live — and the live evidence says it did not take.
Evidence: all 26 message rows on her renewal thread queried; zero carry the new recorder's writer id; nothing added 09-01 or 09-02.
What the PM experienced. Two truths in one system — a turnover record with a move-out date, an inspection booked, and a calendar event, sitting alongside a lease record that still reads month-to-month, no move-out date, no notice date.
What actually happened. The turnover was created at 15:53:53 on September 1, correctly dated. The lease record was last touched at 06:02 that morning — before the turnover — so nothing on the notice path wrote to it. Suspected: the turnover was actually triggered by the AppFolio rent-roll poller after a human typed the notice into AppFolio, not by our email path — a transition recorded in no event log at all.
Classification: BUG (split-brain write), plus an INCOMPLETE audit trail.
Prior art. Same shape as the archived-saga bug (F1) — a write that lands in one record and not the one the app reads. No doc addresses it as a class.
Status today: not fixed.
Evidence: turnover created 2026-09-01T15:53:53.877Z with move-out date 2026-09-30; lease record end date empty, move-out/notice fields null, last updated 2026-09-01T06:02:35Z.
What happened. The firm notice-to-vacate path is gated behind a switch that is fail-closed — it only runs if a specific value is set exactly right. The policy question behind it is genuinely open; Gera explicitly said he had not picked an option, only asked that the situation be posted in the channel.
Classification: INCOMPLETE (deliberately). The correct kind of guardrail — but parked for weeks with an open decision on it.
Prior art. Decision open as of 2026-09-02. Slack contains two contradictory readings of the live switch value in the same minute — nobody actually knows if it's on.
Status today: not fixed; decision open. See Decision 3 below.
Evidence: isEmailNtvFirmRoutingEnabled(), src/lib/domain/leasing/ntv-gate.ts:24-26; Slack 2026-09-02 18:40Z, two contradictory statements.
What Fede experienced. Asked why not a single notice to vacate shows on the dashboard.
What actually happened. Two separate reasons, both by construction. The dashboard counts notices 100% from the AppFolio rent roll ("units where rent-roll status = Notice") — an emailed notice PropFlow never writes into AppFolio contributes zero, there's no path from Clara's inbox to that number. The renewals board only shows a notice if the renewal record is in a notice state — a month-to-month resident who files a notice is projected as "settled — staying," the exact opposite of the truth.
Classification: DESIGN DECISION for the dashboard (single-source-of-truth discipline, documented and defensible), DESIGN FLAW for the board (the month-to-month projection swallows the notice state).
Prior art. docs/architecture/data-architecture-single-source-discipline.md:337 defines the dashboard number as the rent-roll count — real and defensible, but it means an emailed notice is invisible until somebody types it into AppFolio, colliding with "everything should be automatic."
Status today: not fixed.
What happened (verified via Slack, not re-verified in this pass). A sweep of 1,569 Camellia email signals found six older move-out notices waiting on a human because the sender's email address didn't match a known resident. Units 402 and 603 show a notice date but no move-out date — half-written.
Classification: DESIGN FLAW. Correct to not act on an unmatched sender; wrong that unmatched notices sit in a queue nobody works.
Prior art. None.
Status today: not fixed.
Evidence: Slack 2026-09-01 14:49Z.
What the PM experienced. Renewals showing open and stuck for residents who moved out — one of them 106 days ago. Inflates every stuck-renewal count and every alert.
What actually happened. Nothing in the codebase closes a renewal when its lease is archived. Unit 218's renewal was marked declined May 28 and still wasn't closed. Unit 610's retry counter reached 4 against a maximum of 3.
Classification: BUG. A pure missing behavior — nobody wrote it, so nobody could have tested it.
Prior art. None — genuine blank.
Status today: not fixed.
Evidence: unit 212 archived 2026-09-01, unit 610 archived 2026-08-10, unit 218 archived 2026-06-01; no code path closes a renewal on archive.
What the PM experienced. Seventeen open renewals at Camellia, and — as Gera put it — "nothing is currently driving any of them."
What actually happened. Literally true. Every one of the 17 open renewals is either timed out, completed, or has no underlying process at all. Sixteen healthy renewals are running, and none of them belongs to these 17.
Correction to the Slack numbers: 17 open (not 22), 13 genuinely stuck (not "roughly 11"), and five leases already past their end date, not four — unit 111 was missing from that list.
Classification: the aggregate symptom of A1–E1, not a separate bug.
Status today: not fixed. This set needs a person this week.
What happened. The script that archives a stuck renewal copied the record to an archive key and flipped it out of one index — but never touched the index the app actually reads, and never set the "closed" field. Result: 13 of 15 archived renewals still read as never-closed, and one leaked back into the live list as permanently open.
Important correction to the Slack root-cause post: it claimed nine real Camellia residents were affected, with a knock-on effect on what Clara was allowed to do for them. That is not supported by the data — all 11 Camellia archive rows predate the index change and were never visible to the live queries. The only row that actually leaked belongs to the Willows test property.
Classification: BUG. Both halves fixed same-day.
Status today: fixed and live — both halves. No data cleanup needed.
Evidence: PR #7068 (read-side filter, merged 2026-09-04T16:21Z) and PR #7067 (write path, 16:36Z); full paginated scan of 15 archive rows.
What the resident experienced. The same renewal letter, four times in an hour and a half. (The internal audit reported three; production shows four — three identical letters plus one near-identical outreach text.)
What actually happened. At the time, no renewal-level duplicate protection existed at all — the dedupe key included the process run id, so every retry minted a brand-new key and could never match a previous send. Each send also created its own conversation thread with a raw-timestamp id, so even a per-thread check would have seen four different conversations.
Classification: BUG.
Prior art. Fixed by PR #4023 (July 2026) — an atomic claim per renewal-and-stage before any send, refusing a second attempt before Twilio is touched. Thread ids are now deterministic. The final structural piece landed 2026-09-04 (PR #6902), deleting the second path that could start a renewal.
Status today: fixed and live. This exact class cannot recur. Neither PR #6857 nor #6858 (both merged 9/3) actually touch this guard — credit belongs to #4023 and #6902.
Evidence: four sends 2026-05-26 20:45–22:12; the send-guard is wired at all four dispatch sites in renewal.ts.
What this means. The 7-messages-in-7-days safety net — the thing that would have been the second line of defense against F2 — is deliberately counting-only, never refusing, for renewal letters. It cannot stop a renewal repeat-send, whatever the cause.
What actually happened. Renewal letters are person-keyed, but they pass a flag that means measure-only. The code comment says the ceiling stays stood down "until we have real pooled numbers" (Gera, 2026-09-02).
Classification: DESIGN DECISION, dated and attributed — not a bug — but it means the ceiling provides zero protection for renewals today.
Prior art. Three policy options (A/B/C) were put to Fede and Sean on 2026-09-03 and no decision is recorded.
Status today: decision pending.
Evidence: send-renewal-letter-mms.ts L200-221, L265-289, measureOnly: true.
What this means. A setting exists to turn texting off. The texting code never reads it.
Classification: INCOMPLETE.
Prior art. Raised 2026-09-03, not decided or fixed.
Status today: not fixed.
Evidence: Slack 2026-09-03 03:55Z.
What residents experienced. Seven renewal offers arriving far too soon.
What actually happened. Two independent pieces of code were both auto-starting renewals, and the second used a 120-day window instead of the property's own. Switched off within about 30 minutes; the duplicate starter has since been deleted outright, not just disarmed.
Classification: DESIGN FLAW. Two starters is the root shape; the wrong window is the symptom. This is the fourth distinct time the "two code paths both start the same renewal" class has been fixed.
Prior art. ADR-0108 (accepted 2026-07-23) named this exact class — "duplicated rules reading divergent sources," "25 gates across 8 entry points" — as one of three structural root causes, prescribing one shared pipeline and one owner. The incident happened because that plan hadn't landed yet. Fede's "no half-done migrations" ruling (2026-09-03) came directly out of this.
Status today: fixed and live. PR #6902 (2026-09-04) removed the walker's ability to start a renewal. There is now exactly one starter.
What happened. Part of the same incident. Unit 417's renewal was opened three times in 75 minutes (two cancelled). A shared page-the-PM ledger and an "offer id kept per prepared renewal" guard were both added.
Classification: BUG.
Status today: fixed and live. The new 30-day replay harness now asserts "every prepared renewal keeps its own offer id" specifically because of this.
What happened. A renewal-side text went to a delinquent Camellia resident.
Classification: BUG. A gate for the lapsed branch was shipped dark (PR #6720) — built and deliberately not turned on for customers, per the standard rollout playbook.
Prior art. Suppressions and a receipt deletion were put in place at the time.
Status today: fix shipped dark, not on. A related staleness gap (how old a decision can be before it's re-checked) is open and needs a policy call — see Decision 2.
What happened. On September 3 a June 26 call was reprocessed as if it had just happened. Two promises were graded against September's reality — a tour promise was wrongly declared broken and paged the team; a renewal promise ("I'll text you the renewal details") was marked unfulfillable and paged nobody, recorded silently.
What's actually broken. The trigger that re-grades promises has no staleness check and never looks at whether the conversation is already resolved. A stale re-fire of an old call would produce the exact same result today.
Classification: BUG.
Prior art. Two PRs shipped, neither is the fix. PR #6903 (9/3) adds forensics so the next re-fire names itself — its own header says "NOT A GUARD." PR #6981 (9/4) fixes the tour half only, shipped dark, off at Camellia. The staleness guard does not exist; both PRs' own descriptions say it's a separate follow-up.
Status today: not fixed. The failure mode is silent by design — the renewal case pages nobody, so a recurrence produces no signal at all.
What happened. A tenant asked about the rate; the system treated it as a signed acceptance.
Classification: BUG.
Prior art. Directly caused the intent-evidence module, which now distinguishes "the system confirmed it" from "the resident said it," and flags a conditional yes as a counter-offer rather than an unconditional yes.
Status today: fixed and live.
What happened. A renewal thread led to an unrelated work-order lookup, a work order being closed, and a stand-down text to a vendor.
Classification: BUG (tool surface too wide inside a renewal conversation).
Prior art. Produced a dedicated 5-scenario eval suite guarding this exact class.
Status today: fixed — but the guard is currently RED. Its most recent run (2026-09-02) failed. This is the freshest red signal anywhere in the renewal test estate.
The one code path that pages a human shared a switch with Clara's autonomous sending. Sending was off, so paging was off too — true since April or May, only on Camellia. Produced the standing rule: escalation must never share a switch with autonomous sending. Fixed 2026-06-03 (PR #1896).
The eligibility check accepted any non-blank string; the literal text "n/a" passed, went to AppFolio, and got blind-rejected there. Flagship example in ADR-0108 of the same rule being answered four different ways. Fixed 2026-07-22 (PR #4355); the harder half — an invalid address silently passing the gate — fixed 2026-07-25 (PR #4555). The structural fix (one shared rule pipeline, ADR-0108 D3) is still not landed.
A change worked out legal texting hours from a phone's area code — wrong, because the law measures where the person actually is, and people move and keep their number. Shipped 2026-07-28, reverted 2026-07-29 (PRs #4843/#4844, reverted by #4866) back to property-local time.
The lapsed/holdover branch of the daily scan is a second, independent route into outreach that skipped the main eligibility check entirely — one resident got the text on his actual move-out date. Fixed same day (PR #5152), routed through the central checklist; a 30-day audit confirmed only these two were sent in error, but found eight further open gaps, three rated High — seven of the eight remain open. The instance is fixed; the branch is still structurally separate. Its planned migration into the durable workflow shipped dark then was retired under the "no half-done migrations" call, so the daily scanner's lapsed branch remains the sole owner today, hardened by point fixes only.
Same branch as H0-d, different cause — the eligibility verdict was only checked on the normal entry point, not the lapsed-scan entry point. The resident had been flagged for human review every day since August 15. PR #6720 gates the lapsed branch on the eligibility verdict; a further 2026-09-03 fix re-evaluates eligibility fresh at the post-lapse gates. The related staleness-policy question is open — see Decision 2. (Reported as unit 415; the underlying corpus identifies it by balance, not unit, so treat the unit as unconfirmed.)
Six independent code paths could message a real resident or PM and never write a message record, losing the conversation and Clara's context for it — direct ancestor of unit 614's missing notice (D2). Tracked as its own issue, fixed across four PRs in late July (issue #4802; PRs #4825, #4842, #4859, #4865).
The paging function waited for the email, then reported success unconditionally. Real send failures were swallowed; nobody knew the PM hadn't been told. Fixed (PR #5191).
She told Clara she'd already renewed on paper; the system didn't accept it and kept the renewal open. We trust the PMS as truth (correct) but had no route for "signed outside it, not yet captured." Fixed (PR #5192). Related unresolved case: unit 504 is asking for a paper lease right now and can't get one (B4).
Root cause found on 11 of 19 checked notification surfaces — the system treated a stated intent as a signature. Fede's words at the time: "this tells me there might be a big architecture flaw in how we do reminders, so do a deep inspection and come up with a strong fix that addresses the roots." Produced ADR-0122 (accepted 2026-08-01): only a PMS-verified signature may mark a lease signed; a stated intent is a separate, honestly labelled state; a new "negotiating" category; every notification re-verified immediately before it sends. Fixed, implemented, verified in production — the cleanest root-cause fix in the whole history. Unit 607 (B3) is the next layer of the same problem: detection now works, and nothing chases the human.
Declines recorded by Clara were being lost for residents whose renewal had moved onto the new engine mid-cycle — fixed as an incident in June. The root fix — one permanent, stable address per renewal — shipped 2026-07-23 as opt-in, meaning every property onboarded after that date was silently unprotected against this exact bug because nobody remembered to add them to the list. Fede's "Option B" call on 2026-09-03 flipped it to on-by-default. Also the reason for the standing rule: never hand-migrate a live property's renewal.
The watchdog job that was supposed to notice stuck renewals had itself been broken by an unrelated security fix — nobody noticed until nine offers had stalled. Produced the standing rule "the thing that watches must itself be watched." Fixed after PR #1896 moved the renewal engine onto a hosted durable workflow platform (ADR-0025, 2026-05-14). The shape recurred: on 2026-07-25 the nightly replay check was found running green for three weeks while testing nothing, and on 2026-09-04 the proof run had to be changed to fail loudly on an empty run.
Unit 606's page showed contradictory states for a renewal that had actually finished nearly two days earlier by a different route (an emailed notice to vacate). The page was guessing status from indirect clues instead of reading what the workflow recorded — an audit found 37 of 97 (38%) of Camellia's historical renewals would render the same phantom step. Fixed via ADR-0055 (ratified 2026-06-11): the page becomes a pure display layer, never invents forward steps. "Stale or phantom state on screen" is still the single largest bug class in renewals — 31 fixes in three months.
Only one AppFolio field counted as the trigger, and it was already filled in from earlier — the change to the other field fired nothing. Fixed via ADR-0037 (2026-05-27): both the notice-date and move-out-date fields now count as a trigger. Unit 218's renewal record is one of the three still sitting open today (E1), 106 days later, for a completely different reason — the same unit broken by two different bugs.
Fixed via ADR-0036 (2026-05-27) — email the offer and letter to residents with no phone at all. Explicitly deferred the "has a phone, but it's dead" case — which is unit 111 today (A4), still open.
A resident moving from unit 201 to unit 324 could have received a renewal offer for the wrong unit, or been logged as declined. ADR-0081 (proposed 2026-07-08): check for a pending lease on a different unit before offering; pause and escalate; relabel as "Transferred," not "Declined." A guard shipped the next day. Guard is live; an open decision from 2026-09-03 remains — the same transfer check is still missing from the PM's manual "Approve" button (this is the existing open Decision 3 in "Decisions still open" above).
The check used "is a timestamp missing?" as a stand-in for "is it unsigned?" ADR-0104 addendum, 2026-07-22, now reads AppFolio's real signature-status list instead of guessing.
For 60% of Camellia's renewal history our record had no pointer back to the AppFolio record it belongs to, so anything that needed to reconcile the two had nothing to match on. Found during proof work for the September 3 fix, not by any alert. Fixed. A related loose end: a stray unsigned 12-month offer for unit 417, left in AppFolio by a crashed attempt, was still uncancelled as of the write-up — a person's job, not code's.
Test renewal traffic was reaching JP's production AppFolio database — real cross-environment blast radius, since Camellia and the test property share one AppFolio account. Routed to a test instance the same day (PR #6961), then deliberately reverted the same day: "the Willows is our test property" — test writes into JP's AppFolio under the Willows test property are the intended design. That's Decision 5 below; it needs writing down, because the engineering instinct is to keep separating them.
From ADR-0108 (accepted 2026-07-23), the team's own self-diagnosis backed by a ~50-incident corpus. Listed here because several of this month's incidents are the predicted consequences.
Three different entry points each sequence "start the process, tell AppFolio to prepare, stamp the lease as sent" in three different orders, none atomic. Fail mid-sequence and you get a started-but-hollow renewal, or a lease stamped "sent" when nothing was sent, or — on one path — a real AppFolio offer with no PropFlow record at all, because the reconciler its own error handler names was never built. ADR-0108 decision D1 fixes this structurally. Status: plan accepted, not fully landed.
Because a fresh id is minted on every retry, there's no stable address to send a resident's decision to, and a closed process can't receive one. A decline arriving in the wrong window degrades to a soft note nothing surfaces. Nine of the ~50 catalogued incidents are this class. ADR-0108 decision D2 (stable per-lease identity). Status: plan accepted, not landed.
The named example: "does this signer have an email?" is answered four different ways — one accepts any non-blank string, so an email literally recorded as "n/a" passed the check, went to AppFolio, and got blind-rejected. ADR-0108 decision D3 (one shared pipeline). The "n/a" case itself was fixed by a dedicated deliverable-email check. Status: point fix live; structural fix not landed.
The workflow's own timeout classifies errors by matching words in the error text and fires before the sweep that actually probes what happened — a bad guess beats a real diagnosis, producing false "infrastructure failure" alerts. ADR-0108 decision D4 (one owner per failure class). Status: not landed.
| When | What the resident/PM saw | Class | Status |
|---|---|---|---|
| 2026-04-29 | The first end-to-end renewal failures; produced the very first renewal test script | BUG | fixed |
| 2026-05 (ADR-0046) | Late signers double-charged — a PM manually extending the lease end date collided with AppFolio auto-posting a second recurring charge. Inverse case: a lease expired unsigned and the unit showed occupied with no rent charged at all | DESIGN FLAW | fixed |
| 2026-06 | Duplicate renewal outreach texts — letter and text both firing for the same resident | BUG | fixed (deterministic thread id + hard send block) |
| 2026-06-12 | Four process-layer renewal bugs from a Camellia incident | BUG | fixed |
| 2026-06-23 | The page said "Renewal sent" for renewals that were never sent | BUG (honesty) | fixed — all PM-facing sentences now come from one module |
| 2026-07-02 | Concurrent prepare race at Camellia — the "bootstrap storm" | BUG | fixed |
| 2026-07-08 | The banner on screen and the PM email said different, contradictory things about the same renewal | BUG | fixed |
| ongoing | Renewal dates off by exactly one day; dates rendered against an unnamed timezone | BUG (11 fixes) | fixed repeatedly |
| ongoing | PM escalations routed to the wrong inbox / falling back to an ops address | BUG (9 fixes) | fixed |
| 2026-09-01 | Nightly test caught a real regression: a renewal recorded as executed internally showed as declined on the PM screen (5 of 23 cells) | BUG | caught by automation before customers saw it |
| 2026-09-03 | Namespace migration for the renewal engine: the safeguard against double-messaging was broken going in (a quoting mistake copy-pasted across 34 scripts). Caught in the dry run | BUG (near-miss) | fixed, no customer impact |
| 2026-09-03 | AppFolio's renewal report endpoint returned errors; sync reported 0 of 2 properties | not a bug — transient, self-healed | closed as noise |
| 2026-09-04 | A renewal auto-start test had tomorrow's date typed into it and expired at midnight, turning main red | test-only | fixed |
Excluded deliberately: the Willows test property's 73-day stuck renewal, the synthetic canary lease's recurring prepare-failure alerts, and the Willows-only archive leak. All test traffic, no customer impact.
Across everything above, five shapes account for nearly all of it:
| # | Shape | What it looks like | Where it shows up |
|---|---|---|---|
| 1 | Two writers, one truth | The record the app reads and the record that got written are different ones | Archived renewals, unit 614's split-brain, phantom state — 31 fixes in three months |
| 2 | Escalate and forget | A human is told once, and nothing on earth checks whether they acted | A1, A2, A3, B3, B4, D6 |
| 3 | Silent exit | A process finishes normally in a bad state and pages nobody | B1, B2, E1, G4 |
| 4 | Two paths, one job | Two pieces of code doing the same thing, disagreeing | G1, G2, F2, I1, I3 — fixed at least four separate times with four different mechanisms |
| 5 | Fixture didn't look like reality | The test used a shape a real tenant would never produce | D1's "September 30th", D2's declared-fixed-but-broken |
2.1 There is no "the harness." There are nine. Since April the team has built nine separate families of renewal testing, and not one of them replaced another. They accreted. Here is the whole estate.
| # | What it is | Since | What layer it tests | Runs by itself? | Can it block a merge? | Alive today? |
|---|---|---|---|---|---|---|
| 1 | The fortress suite + the 203-item bug catalogue | Jun 24 / Jul 17 | Decision logic: replies, declines, notices, hand-off to turnover — the deepest corpus of past renewal bugs | Yes — rides the ordinary test run | Yes | Alive and gating. The unglamorous workhorse. |
| 2 | Nightly full-permutation run against the test property | May 16 | Real texts, real durable workflows, real database, against the Willows test property — 19–23 combinations × 3 lease terms | Yes — every night at 9am UTC | Yes, for the nightly job (alerts Slack) | Alive. This is the one that actually caught something this week. |
| 3 | Nightly replay of real closed production renewals | Jul 22 | Downloads the ten most recently finished real renewals, re-runs them against today's code to check nothing drifted | Yes — nightly | No — emails/Slacks after the fact | Alive. Green all week. |
| 4 | The 30-day scan replay | Sep 3 — one day before this week's incidents | Simulates a whole month of the daily renewal scan across three invented properties with three different renewal windows — built specifically to catch a starter using the wrong window | Yes — on every PR touching renewal code | No — deliberately non-blocking "until it earns a few weeks of clean history" | Alive, green, not a gate |
| 5 | The Willows scripts (dry run, seed, reset, live run, verify, policy) | Apr–Sep 3 | The live-run script is closest to "exercise the real production start path end to end" | No. Opt-in only, off by default even on manual trigger — "skipped" on every one of the last 30 nightly runs | No | Effectively dormant. May never have actually executed in CI. |
| 6 | The one-off repro scripts (holdover, bootstrap storm, co-signer, tenant-signed, the "Renewal Failure Gauntlet") | May–Jul | Real AppFolio, real durable workflows, real texts — the deepest, most specific tools we own | No. Only 3 of 15 are wired to CI at all; one hard-codes a laptop file path | No | Alive but hand-cranked. Several were edited September 3 — engineers reached for them during the fire, not before it. |
| 7 | The renewal evals (tool-call, SMS, scoping) | Apr 21 | Does Clara pick the right action and say the right thing — 85 voice scenarios, 52 SMS scenarios, 5 scoping scenarios | Mixed — some on PRs, some weekly, some manual | Partly | Alive. The scoping suite is RED on its freshest run (September 2). |
| 8 | Voice scenarios mined from real Camellia calls | ~Jul | Eight real call shapes replayed against the voice agent with fake tools | Yes, nightly and on relevant PRs | No — labelled "report-only" in its own header | Alive, cannot block anything. One scenario documents a P1 gap open since July 18. |
| 9 | The coworker harness renewal scenario | ~Aug | Does escalating a renewal open a tracked item | No — wired to no CI workflow | No | Alive but structurally cannot observe its own target — the harness suppresses all outbound, and production refuses to save a tracked item unless the notification is verified delivered, so it always hits the refusal path. |
2.2 For each issue: which tool should have caught it, and why it didn't. Reasons drawn from a fixed vocabulary so the pattern is visible: no scenario / wrong layer / stops one step early / fixture doesn't look like reality / disabled, or red and ignored / nothing writes to real AppFolio / the behavior doesn't exist yet, so there's nothing to test.
| Issue | Should have been caught by | Why it wasn't |
|---|---|---|
| A1 — no-email offers, paged once per 65 days | The 30-day scan replay, extended past one retry cycle | No scenario. No harness runs simulated time past a single retry, and the 65-day marker needs two months of clock to ever fire twice. |
| A2 — write-once flag nothing reads | A test asserting the flag is read | The behavior doesn't exist yet. |
| A3 — unit 515 portal refusal treated as permanent | A real-AppFolio scenario returning that specific error | Nothing writes to real AppFolio; also no scenario distinguishes "no portal account" from a real business rejection. |
| A4 — unit 111 all channels dead | A scenario faking carrier delivery failure on every channel | No scenario — harnesses running against test credentials get clean "delivered" receipts back. |
| A5 — contact-info change doesn't retry | A scenario where a PM adds the missing email | No scenario, and the behavior doesn't exist. |
| B1 — silent exit at lease end | Any harness that runs a full cycle to its actual end | Stops one step early — harnesses assert "nudge sent" and stop. |
| B2 — month-to-month blocked by our own offer | The 30-day scan replay, with a lapsed cohort | Fixture doesn't look like reality — the "already handled" cohort models a signed renewal, not our own unanswered offer. |
| B3/B4 — escalations never chased | A test asserting an overdue escalation gets chased | The behavior doesn't exist yet — there's no such thing as an overdue escalation, so nothing to assert against. Product gap and test gap are the same gap. |
| B5 — hold never lifts | A scenario where a PM answers mid-cycle | No scenario, and no "escalation resolved" signal to test with. |
| C1 — unit 607 signed in AppFolio, never noticed | A scenario where the resident signs directly in AppFolio | No scenario — every harness drives the PropFlow-native happy path. Sync freshness per lease is instrumented nowhere. |
| C2 — cancel evidence never applied | An end-to-end test of the reconciler | Stops one step early — existing tests prove the evidence gets recorded, not applied. |
| C3 — "confirm it's handled" button | — | Working as designed; overridden by Fede's ruling. Nothing to catch. |
| D1 — "September 30th" not parsed | An eval feeding real free-text notice emails | Fixture doesn't look like reality — the fix's own comment claims a machine-readable date exists. The test used a synthetic ISO date; real tenants write prose. |
| D2 — notice missing from the thread | A test replaying that exact real inbound | Fixture doesn't look like reality — four PRs passed their tests and the real message still isn't on the thread; declared fixed in Slack twice without checking the record. |
| D3 — turnover says Sept 30, lease says nothing | A test asserting both records agree | No scenario for cross-record consistency. |
| D4 — auto move-out path off | — | Deliberately off; decision open. |
| D5 — notices invisible on the dashboard | A test asserting an emailed notice shows up somewhere | No scenario, and partly by design — the dashboard reads the rent roll only. |
| D6 — six unmatched notices parked | A queue-age alarm | The behavior doesn't exist yet. |
| E1 — moved-out sagas never closed | A test asserting a renewal closes when its lease archives | The behavior doesn't exist yet. |
| F1 — archive script closed nothing | A test asserting an archived record leaves every index | No scenario — caught by a live alert, days later; a regression test shipped with the fix. |
| F2 — letter sent 4× | A test asserting one send survives a retry | No scenario at the time — renewal-level duplicate protection didn't exist in May. Today's guard is live; whether anything regression-tests "a retry does not resend" is unconfirmed. |
| F3 — volume ceiling doesn't apply | — | Deliberate. Nothing to catch. |
| G1/G2 — early offers, duplicate pages, unit 417 | The 30-day scan replay | It was built for exactly this and merged one day too late. Its own header names this incident. Still non-blocking. |
| G3 — delinquent resident texted a quote | The 30-day scan replay, lapsed-branch cohort | No scenario for the lapsed entry point specifically — the eligibility check only ran on the other entry point. |
| G4 — stale June call re-graded in September | A test asserting an old resolved conversation isn't re-graded | The behavior doesn't exist yet — no test covers staleness or re-firing, and the renewal half pages nobody, so a recurrence is invisible in production. |
| G5 — rate question read as acceptance | The renewal evals | Caught after the fact; produced the intent-evidence module and ADR-0122. Now covered. |
| G6 — renewal thread hijacked into closing a work order | The scoping eval suite | Disabled, or red and ignored — the guard exists and its most recent run failed. |
| H0-a — escalation on the wrong switch | Any test asserting a PM gets paged when a renewal stalls | The behavior was silently off — a test would have had to assert on the absence of a message, which nobody does. |
| H0-b — "n/a" email | One shared eligibility rule with its own test | Wrong layer — four separate implementations of one rule; a test on one doesn't cover the others. |
| H0-d/e — lapsed-branch outreach | The 30-day scan replay | No scenario for the second entry point — same root as G1. |
| H0-f — six senders with no record | A test asserting every send writes a record | No scenario. |
| H0-j — dropped decline; fix opt-in for six weeks | A test asserting the protection is on for every property | No scenario for "is this property actually covered?" — an allowlist nobody maintained. |
| H0-k — nine offers stuck; watchdog broken | A test on the watchdog itself | The watcher was unwatched. Recurred twice more (July 25, September 4). |
| I1–I4 — half-starts, dropped replies, divergent rules, wrong alert wins | The permutation-diff harness ADR-0108 requires on every renewal PR | The harness exists and the requirement is written down. The structural fixes it was meant to gate are still not landed. |
2.3 The five structural reasons, ranked.
The goal, in one sentence. Every issue in Section 1 becomes a named, repeatable test that fails on the code as it was the day it happened, and tells the truth about the code as it is today — running on a laptop and in CI, touching no real resident and no real AppFolio.
The rule that makes it cheap: we do not build a new harness. Two replay mechanisms already exist and are complementary. Phase one is fixtures and assertions inside them, not infrastructure.
Everything in Section 1 lands in one or the other.
What we have (counted, not estimated):
| Count | Note | |
|---|---|---|
| Renewal records since launch | ≈137 | 123 live, 15 archived, one double-counted. A floor, not a guarantee — the two archive routes are known to disagree. |
| Camellia renewal conversations — the real corpus | 52 | All text. All genuine residents. |
| Willows renewal conversations | 554 | 496 are synthetic nightly test traffic. Not launch history. |
| Voice renewal conversations | 0 | Every renewal conversation on record is text or email. No voice renewal corpus to replay. |
| Renewal event-log entries | 499 | Across the live records, about 4 events each. |
| AppFolio evidence records | 16 | Latest state per property/occupancy/type — not a full trail. The trail lives in the event logs and automation-run records. |
| Escalation records | 2 stores | A known, deliberate duplication awaiting a planned merge. A corpus builder has to know which one produced a given case. |
What's missing, and it matters:
On resident privacy. Easier than it looks. Renewal records and workflow histories are already id-based — names, phones and emails are never carried in them; they're looked up from the database at run time. The export needs to: run the existing money-field scrubber, swap real ids for readable synthetic ones (e.g. a scenario named for the shape of the incident, not a resident), and refuse to write a fixture that still matches a phone, email or name pattern — reusing the scrubber already in the repo rather than re-deriving it. Same rule the invented-portfolio file already carries in a comment: no real resident, property or phone number appears in it — enforced by code this time.
One named scenario per card in Section 1. The name is the assertion — a failing test should read like a broken promise, not a byte difference. The expected outcome is written against Fede's current rulings, not against what the code does today — most of these will be red when first written, and that's the point.
| Scenario name | Mechanism | Expected outcome under current rulings |
|---|---|---|
no-signer-email-is-chased-not-shelved | 2 | Resident asked for their email more than once; PM reminded three times, business days, once a day — not once per 65 days. |
pm-flag-goes-overdue | 2 | A flag older than the ladder shows up as overdue somewhere a human looks. |
portal-refusal-is-a-setup-problem-not-a-verdict | 2 | A portal-send refusal is retried and named as "no portal account," not shelved as permanent. |
every-channel-dead-raises-a-flag | 2 | All-undelivered across text, letter and voice raises a flag the same day. |
new-email-restarts-a-blocked-renewal | 2 | Adding the missing contact detail re-runs the renewal without a human asking. |
unanswered-offer-does-not-lapse-in-silence | 1 + 2 | At lease end with no reply, something happens — see Decision 1. Not nothing. |
our-own-unsigned-offer-is-not-already-handled | 2 | The month-to-month conversion runs, cancels our stale offer, updates the rent — the chain ADR-0046 already specified. |
counter-offer-escalation-gets-chased | 2 | An unanswered escalation is re-raised on the three-reminder ladder and never ages past the lease end date silently. |
paper-lease-request-gets-an-owner | 2 | Same ladder; the blocked term change surfaces as "cancel the old offer first," a two-step job, not a dead end. |
hold-lifts-when-a-human-answers | 2 | Reminders resume once the escalation is resolved. |
signed-in-appfolio-closes-in-propflow | 1 + 2 | Follows the system of record automatically — no confirm button. Fede's 2026-09-03 ruling. |
external-cancel-evidence-gets-applied | 2 | Evidence moves from proposed to applied and closes the renewal. |
notice-date-in-plain-english-is-understood | 1 | "September 30th" produces a move-out date. |
residents-notice-appears-on-their-thread | 1 + export | The real inbound message is readable on the renewal page. |
notice-writes-the-lease-and-the-turnover | 1 | Both records agree. No split brain. |
an-emailed-notice-is-visible-somewhere | 2 | A notice that only arrived by email shows up on the board (dashboard behavior is a separate decision). |
moved-out-closes-the-renewal | 1 | Archiving a lease closes its renewal. |
archiving-a-renewal-closes-it-everywhere | 1 | Regression guard for the September 4 fix. |
a-retry-does-not-resend-the-letter | 2 | Exactly one delivery survives a mid-send retry — pins the July guard properly. |
only-one-thing-can-start-a-renewal | 2 | Regression guard for September 3; extend to two racing starters. |
a-delinquent-resident-gets-no-quote-from-any-entry-point | 2 | Both entry points honor the eligibility verdict; add a staleness dimension once Decision 2 is made. |
a-stale-call-is-not-re-graded | 1 | Re-firing an old resolved conversation changes nothing. |
a-renewal-conversation-cannot-close-a-work-order | eval suite | Restore the red scoping suite to green first. |
a-half-started-renewal-alerts-loudly | 1 + 2 | Failure between "start" and "prepare" is visible, never silent. |
a-reply-always-has-somewhere-to-land | 1 | A decision arriving mid-retry is never dropped. |
every-property-is-actually-covered | 2 | No protection depends on a list somebody has to remember to update. |
the-run-that-tested-nothing-goes-red | both | An empty run, a stale lock, or zero exported histories fails loudly — the meta-test, earning its place three times over. |
A scenario counts as reproduced only if it passes both halves:
Anything that cannot be made red on the old code goes in a short, explicit "could not reproduce" list with the reason. That list is a finding, not a failure.
No new workflow files.
Steps 1–3 are the bulk. Step 4 is where the most gets learned. Step 5 is the only one that changes what can ship.
Each has a recommendation. Pick from options rather than answering open questions.
Decision 1 — What happens when a resident never answers and the lease is about to end?
The single most consequential one — it's a straight contradiction between two of Fede's own positions. On 2026-07-17 he decided outreach exhaustion should wrap up quietly rather than page a PM, reasoning that a silent resident just rolls to month-to-month and the PM already got an earlier heads-up. On 2026-08-13 he overturned time-based alerting elsewhere, because eleven leases took a normal amount of time and tripped an alert for nothing. Both say fewer alerts — but units 420 and 522 lapsed with nobody told, and the month-to-month rollover that was supposed to make silence safe didn't run (B2). The quiet path is only safe if the fallback works, and it doesn't.
Make the month-to-month conversion actually run — cancel our own stale offer, update the rent, send a fresh offer next month, exactly as ADR-0046 already specified. No new alerting. Silence stays safe because the automatic outcome is real.
Same as A, plus one email to the property when a lease actually lapses unsigned.
Page on outreach exhaustion, as before.
Recommendation: A. It matches "everything should be automatic" and the two anti-alert rulings, it's the design already written down and never finished, and it's the only option that fixes what actually went wrong. B is the safe compromise if the conversion isn't trusted yet — take B for the first month, then drop to A.
Decision 2 — Is an unanswered escalation allowed to age?
Right now, no. An escalation is one email into a mailbox with no deadline, no owner, no expiry. Units 607 and 504 each aged past their lease end. But Fede also said "escalate as little as possible" and already removed time-based alerting once.
Three reminders, business days, once a day max — to a named owner, not a shared mailbox. Then it stops.
If still unanswered at the end, Clara acts on a documented default (e.g. holds the price, holds the offer open) and records what she did.
Escalation is a hand-off; if the team doesn't answer, that's the team's problem.
Recommendation: A now, B as the follow-up. A is already built for lease countersigns — this applies an existing machine to a fourth case ADR-0104 itself names as having no safety net. B is where "Clara becomes the system of work" actually lives, but it needs a policy library that doesn't exist yet.
Decision 3 — Can a notice to vacate read out of an email fire the whole move-out chain with no human check?
Open since September 2 as an existing decision Gera hasn't chosen — the option attributed to him in Slack was retracted two minutes later. Slack also contains two contradictory readings of the live switch value in the same minute — nobody currently knows if it's on.
Decline the renewal, start the turnover, draft the move-out in AppFolio. Risk: a false reading on a staying resident starts a move-out. Note this is already shipped in a narrow form and is non-functional, because it only accepts machine-formatted dates and residents write prose.
Month-to-month residents always go to a human — which is exactly what left unit 614 sitting for two days.
An invented date is half the key used to avoid creating a duplicate move-out draft, so a wrong guess quietly produces a second one when the real date arrives.
Recommendation: B now, A as a fast follow — but A is blocked on something concrete. Before A can work at all, the classifier has to be required to output dates in machine format; today it emits free text, so A silently does nothing. Whichever gets picked, somebody needs to read the live switch value and publish it.
Decision 4 — Does the 30-day scan replay become a required check now? (supersedes the "Decisions still open" item on this same question above — this is the fuller version, with a dated recommendation.)
It's the one tool built for this exact class of bug. Shipped September 3 as a non-blocking check by explicit choice, pending "a few weeks of clean history." Its last ten runs are all green.
Cost: a flaky run blocks a merge. Benefit: the September 3 incident cannot recur through a merge.
A dated commitment, not "when someone remembers."
How a good tool quietly becomes decoration.
Recommendation: B, paired with the meta-test from 3.2 — a check that can silently test nothing is worse than no check, and there are three real instances of that happening.
Decision 5 — Should the Willows write into JP's real AppFolio?
Fede already answered this on September 4 — a fix routing Willows to a test AppFolio instance was merged, and he reverted it, saying "the Willows is our test property." But the engineering instinct will keep pulling the other way, because Camellia and Willows share one AppFolio account and the blast radius is real. This needs writing down as a decision, or it will be re-litigated every time someone sees test data in production.
The only way to test the real AppFolio path at all, which is the single biggest coverage gap today. Recommended with a named guard: every automated AppFolio write must carry the test property's id and refuse on any other, enforced in code, not convention.
Keeps the coverage, caps the blast radius.
Cleanest, and gives up the one thing we most need to test.
Recommendation: A, with the named guard above. Preserves the existing ruling and closes the gap the team's own scorecard has had open since May. Phase one itself writes nothing to AppFolio either way — this decision governs phase two.
1. Should every property get its own separate index of renewal records, instead of sharing one scoped-but-shared index?
A — build the per-property index now (recommended). B — split the shared index into 16 shards instead. C — wait and put an alarm on the current headroom number, revisit later.
2. A property manager's manual click in the app ignores a cancellation notice that's recorded only on the lease, while the automated path already honors it — close that gap or leave it?
Close it (recommended). Leave it as-is.
3. The "Approve" button skips the transfer check that the "Send Renewal" button already has — add it or leave it?
Add it (recommended). Leave it.
4. Should the 30-day replay test become a required, merge-blocking check?
Superseded by Renewals Quality Decision 4 below, which carries the fuller, dated version of this same question.
5–9. Five more decisions came out of the September 4 renewals-quality investigation — what happens when nobody answers, whether an escalation may age, whether an emailed notice can fire a move-out unattended, whether the replay check becomes mandatory and when, and whether the Willows keeps writing into JP's real AppFolio.
See Decision 1, Decision 2, Decision 3, Decision 4, and Decision 5 in the Renewals Quality section below.
The core engine is fine and should stay. Every renewal runs as its own long-lived, durable process (built on a tool called Temporal — think of it as a robot case-worker that can wait weeks for a tenant to reply without forgetting anything) and that part has worked well since spring.
The daily job that kicks off new renewals is also fine — it's just the "starter," not the process itself, and starting renewals is the one thing it should keep doing.
The actual flaw: way too many things are allowed to touch the same renewal. The daily job, a newer "cohort walker," the two staff buttons ("send offer," "approve"), the software that watches AppFolio (the property-management system) for changes, a reconciler, and a retry sweep can all start, write to, or page someone about the same renewal — with no single owner, no shared "did we already do this?" checklist, and the same rules (like "does this timing count as too early," or "whose email do we use") coded up to four different times in four different places.
Wednesday morning was that flaw catching fire. A second starter (the cohort walker) was switched on for Camellia, started working the same tenants the daily job was already handling, and because nothing coordinates the two, the same problem showed up as five different symptoms at once.
The fix isn't a rewrite. It's making the daily job the one and only starter, giving every entry point one shared rulebook instead of four copies, and adding a test that would have caught Wednesday before it shipped — all doable in weeks, not months.
Six different things can start, touch, or page someone about one renewal. A renewal itself is one lease coming up for its yearly (or month-to-month) renewal — the "case" this whole system manages.
flowchart LR
subgraph starters["STARTERS — can open or restart a renewal"]
direction TB
S1["Daily job
(scans every lease, starts eligible renewals)"]
S2["Cohort walker
(newer scanner, same job)"]
S3["Staff button: Send offer"]
S4["Staff button: Approve"]
end
subgraph process["THE PER-RENEWAL PROCESS — one durable case-worker per lease"]
direction TB
P1["Prepare the offer in AppFolio"]
P2["Text / call / email the tenant"]
P3["Wait for a reply, follow up on a timer"]
P4["Record the outcome"]
end
subgraph watchers["WATCHERS — can also write to or page about a renewal"]
direction TB
W1["AppFolio poller
(notices PM actions made by hand)"]
W2["Reconciler
(catches PMS-side changes)"]
W3["Retry sweep
(retries failed offer preps)"]
end
starters --> process
process --> outcomes["AppFolio + tenant + PM inbox"]
watchers --> outcomes
watchers -. "writes into the same case, no shared lock" .-> process
Green arrows below = a single clear owner. Red = contested — more than one thing can act on the same renewal with nothing stopping a collision.
| Component | Added when | What it does | Status |
|---|---|---|---|
| Daily renewal job | Original build | Scans every lease every day, starts a renewal case when one is due | Live, on for Camellia |
| Cohort walker | New, 2026-08-31 built · 2026-09-02 turned on for Camellia | A second, newer scanner meant to eventually replace the daily job | Live for Camellia — turned on the same week the daily job was still running too |
| The per-renewal process (Temporal) | Spring 2026 | Runs one renewal case start to finish: prepares the offer, contacts the tenant, waits, follows up, records the result | Live, solid track record |
| "Send offer" / "Approve" staff buttons | Original build | Let a property manager manually push a renewal forward | Live |
| AppFolio poller | Original build | Notices when a PM handled a renewal by hand, directly in AppFolio, and closes it out here too | Live |
| Reconciler | Added mid-2026 | Compares what AppFolio says really happened against what our records say | Live, look-only for the riskiest cases |
| Retry sweep | Original build | Retries renewals stuck because AppFolio rejected or timed out on preparing the offer | Live |
Imagine two case-workers were both handed the same stack of files overnight, with no note between them saying who already called which client. Some clients got called too soon. The manager got paged twice about the same missing email. And one file got filled out with the wrong name and then just... stopped moving, with nobody noticing.
Ten concrete problems, pulled from a full engineering audit. Each one is either something we can fix without touching the core design ("fix in place"), or something that needs a real redesign.
| # | Flaw, in plain words | Evidence | Fix type |
|---|---|---|---|
| 1 | Multiple starters can begin or restart the same renewal in a different order, with no single atomic "claim" on it | 3 separate starters (daily job, "send offer" button, "approve" button) each sequence their steps differently; a known 2026-07-22 incident half-started a renewal when one step failed mid-sequence | Needs redesign |
| 2 | The stable, permanent address for "this lease's renewal" (so replies always reach the right case) is off by default for every property except the two that were manually turned on | Every new property inherits the old, bug-prone addressing scheme unless someone remembers to flip a switch — this is the root cause of the "tenant's reply got dropped" bug class | Needs redesign (mechanism built, just needs to become the default) |
| 3 | 25 separate eligibility/timing checks live across 8 different entry points, and they don't all run the same checks | The "does this tenant have a valid email" rule alone is implemented 4 different ways with 4 different answers for the same tenant | Needs redesign |
| 4 | Nothing watches for a PM handling a renewal by hand, directly in AppFolio, for most cases | An audit found 5 of 9 "needs your attention" board items were already resolved by a PM in AppFolio — PropFlow just never noticed | Partly fixed (built, rolling out in stages) |
| 5 | A single database partition (a physical storage bucket) has to absorb every renewal read and write platform-wide; it hit its throughput ceiling once already | 4,481 "too many requests" errors over 6 weeks; patched with a 16-way split that is itself sized for today's scale, not 10x more properties | Fix in place (stopgap works now, will need revisiting) |
| 6 | The daily job reads every property on the whole platform, every single day, before narrowing down to the ones it actually needs to check | Read volume scales with total platform size, not with how many renewals are actually due — gets worse every time a new property or a second/third property-management system is added | Fix in place |
| 7 | The core renewal process is one 2,200-line function; the file that writes to AppFolio is 4,000 lines | No one can safely change either without reading the whole thing; every new behavior has to be inserted in exactly the right spot or it can silently corrupt an in-progress renewal | Fix in place (large but mechanical) |
| 8 | About 13 independent on/off switches control renewal behavior, several layered with their own allow-lists | The team has already had to add typo-guards and conflict checks because the switch combinations got too complex to configure safely by hand | Fix in place |
| 9 | The "is this AppFolio failure temporary or permanent" check is written out separately in 4 different places, kept in sync only by a dedicated test | Already caused one real incident (Aug 5–6) where a timing mismatch between two copies stopped failures from self-healing | Fix in place |
| 10 | The half-built "second AppFolio account" config system has its real storage layer deleted — only a test stand-in exists | A comment in the code literally says the real version was deleted and needs to be rebuilt "when needed" — a landmine for onboarding a third property-management account | Fix in place |
Also found, smaller: a generic "task failed" error swallows the real reason before it reaches anyone (source of the 3 vague PM emails above); infrastructure failures sometimes get routed to a property manager instead of engineering; two separate lists track "did we already page someone about this"; message history older than ~3 weeks isn't kept anywhere; nothing tests the real renewal-starting code path against real historical data in CI; the test environment (Willows) can't actually start a real renewal the way production does.
Of the 16 stuck cases: 9 are just waiting on a quiet tenant (offer sent 31–56 days ago, no reply) and 5 are stuck because AppFolio rejected or timed out on preparing the offer, with no automatic way to clear that. Of the 235 bug-fixes, the two biggest repeat-offender categories are 31 fixes for the screen showing a stale/wrong status and 26 fixes for two things racing to open or touch the same renewal at once — both are direct symptoms of "too many writers, one record," the same root cause behind Sept 3.
| Stage a renewal case reaches | # of the 133 that got there |
|---|---|
| Case opened | 82 |
| Waiting on AppFolio to confirm the offer | 71 |
| Outreach sent to tenant | 39 |
| Offer actually prepared in AppFolio | 31 |
| AppFolio prep failed at least once | 30 |
| Escalated to a human | 29 |
| Signed / executed | 13 |
Reading this: of 133 cases, only 31 ever got a fully prepared offer — the biggest single drop-off is AppFolio's own "prepare the offer" step, not anything tenant-side.
This sits on top of a ~50-incident case history the team already built into a July decision document (ADR-0108) — that document is the team's own root-cause diagnosis, and this page's ranked-flaws list above matches it closely; treat that prior work as already-verified, not re-litigated here.
Four ways forward. B is the recommendation.
Patch Wednesday's specific symptoms (dedupe the pages, fix the thread naming) and move on. Not recommended — the underlying flaw (many writers, no single owner) stays, so the next new starter or watcher recreates the same failure in a new shape.
Make the daily job the only thing allowed to start a renewal. Turn off the cohort walker's ability to start one (keep it only for its "is this tenant eligible" read-only judgment — don't delete the work, just stop it from acting). Enforce in code that a lease can only ever have one live renewal case at a time — everyone else has to find the existing one or back off, not start a new one. Put the 25 scattered rules (timing, eligibility, whose email counts) into one shared rulebook every entry point has to use. One shared "did we already page about this?" list. One naming scheme for message threads. Turn on the permanent stable address for every property, not just two. Split the giant 2,200-line function into named, readable pieces. Add a test that replays real past incidents and would have caught Wednesday. No big-bang — each piece ships and proves itself on its own, sized in weeks.
This was already the long-term plan before Wednesday. Only safe after B's single-owner rule and tests exist — otherwise finishing the handoff just repeats Wednesday at a bigger scale.
Rebuild the whole thing around a single entity that owns its own state end to end. Highest cost, and with only 133 renewals ever run, not justified today. Worth revisiting if renewal volume grows 10x.
| Option | Risk right now | Effort | What it fixes | What it leaves |
|---|---|---|---|---|
| A — patch only | Stays high | Small | Wednesday's specific symptoms | The root flaw — next new starter repeats it |
| B — one owner (recommended) | Drops fast | Medium, weeks, staged | Duplicate starts, split rules, split pages, dropped replies | Cross-property partition ceiling (already stopgapped); needs the durable comms-history gap addressed separately |
| C — finish the handoff | Depends on B first | Medium-large | Removes the daily job entirely, simplifies to one starter by design | Nothing new if done after B; repeats Wednesday if done before B |
| D — full rewrite | Low today, but high cost now | Large | Everything, in theory | Cost far exceeds the size of the problem at 133 lifetime renewals |
Q1. Which option do we run with?
Recommended: B — one starter, one owner, fix in place. Revisit C once B's single-owner rule and replay tests are live.
Q2. The cohort walker's ability to start renewals — delete it, or just switch it off for now?
Recommended: delete the start capability, keep the eligibility-judgment code. A switched-off-but-present starter is exactly the kind of thing that gets accidentally re-armed the way Wednesday happened. Removing the capability (not the whole file) closes that door for good; the read-only "is this tenant eligible" logic is still useful and stays.
Q3. Should the permanent stable renewal address be turned on by default for every property, not just opt-in per property?
Recommended: yes, make it the default going forward, with a checklist to confirm existing properties are on it. Today a brand-new property silently inherits the old, bug-prone addressing scheme unless someone remembers to flip a switch — that's backwards from how we normally ship (new behavior off by default, this is the one exception because the "off" state here IS the bug).
Q4. The 5 tenants who got an offer about a month early — let it stand, or retract it?
Recommended: let the offer stand, pause the automatic reminder follow-ups on those 5 so they don't get chased on the wrong clock, and have their property manager reach out personally if needed. Retracting a rent offer already shown to a tenant risks a worse trust hit than an early-but-honest offer.
Q5. When a tenant has no email on file, should Clara try texting or calling to ask for one before paging a property manager?
Recommended: yes. This is the single largest category of property-manager pages today (10 of 14 open "needs attention" items on the board right now are exactly this). Clara already has the texting and calling infrastructure for renewal reminders — it just isn't wired into this specific gap. This would cut the PM queue by roughly 70% with no policy risk (it's a data-collection step, not a judgment call).
| Action | Safety check | Confidence |
|---|---|---|
| Text / letter / call to tenant | Shared duplicate-send guard, keyed per send | Confirmed safe |
| Property-manager page/email | Own duplicate-send guard, plus a 24h re-page window | Confirmed safe |
| Sending the offer to AppFolio itself | Retry logic exists but no confirmed "did this actually go through before we retry" receipt the way tenant texts have | Not confirmed — flagged as open question |
| The lock that stops two starters double-starting the same case | Deliberately lets a case proceed if the lock-check itself errors ("fails open") — documented, but no confirmed second layer of protection was found | Fail-open gap — worth verifying |