Supervised mode: leak audit

Situs incident, Oct 27

What happened: at a company set to Supervised for leasing and maintenance, an approved applicant with a housing voucher emailed her RFTA packet to leasing@. Clara could not read the PDF, asked the Situs team by email, and three minutes later a staff reply to that email was reworded and texted to the applicant. Clara's own reply to the applicant was held correctly. Nothing was approved by a person.

Timeline (Oct 27, Mountain time)

Four defects, one incident

Is this the first leak at Situs?

• First through the escalation path: this is the only matter ever opened at Situs.
• The Supervised hold itself has worked: 58 holds since Oct 19 (56 maintenance texts, 2 prospect emails), none approved, none sent.
• Nine other texts left the Situs Clara line Oct 19 to 21 with no hold row: six "work order created" confirmations (system sends), and three Clara-written texts: a rating request, a reply to a resident about a cleared boiler closet, and a Spanish tour confirmation to a prospect. Whether Supervised was already set on Oct 19 and 20 is unverified (the company row keeps no history for it, and the company on/off switch changed at 13:00 MT on Oct 20, after those texts). Handed to Gera's lane to confirm; not ruled either way here.

Where Supervised is consolidated today

Clara Refactor's askTheTeam seam now fronts all 23 "ask the team" callers, but it is a pass-through: it checks no mode. The decision-request email and the staff-answer relay both still send directly. Folding team notifications into Supervised means one check inside that seam (the matter's module stage) plus the relay fix above; no second gate.

Yes / hold for Fede

1. Team-asks to the customer's staff are held under Supervised, like Clara's replies? Yes, Fede 2026-10-27: "we need to start holding staff replies"; the Situs case "is an example of we should have not replied because we can't help." Replaces the Oct 14 rule. Build: PR A (relay under the hold) and PR B ("handled" replies relay nothing) by session 007; PR C (team-asks through the team-notice hold door) by the Clara Refactor session. Still open: delete the separate Team-notices switch now that notices follow the module? Recommend yes.
2. Fixes ship as bug fixes, no switch, Willows proof then live everywhere. In build, Oct 27.
3. No recovery message to the applicant. Lee already answered her directly and the company is Supervised. Recommend leave it.

Not verified: whether the teaching step saved Lee's line as property knowledge; why the PDF reader was unavailable; Gera's side's intent for the "team notice link" (bridge question q-situs-teamask-2026-10-27, open).

Records: matter_41896f45, conversation dbe8f8e0, hold eschold_95e18598, Twilio SM98792a226973b9b26373c11a62415ce5, Slack #situs-group 1793121922.037549. Code read on main 7765053b4d.

Fix shipped, Oct 28

Live everywhere, no switch. Every staff reply is now read once into one of five outcomes: answer, handled, question, note or unclear. Only the facts in an answer reach the person, in a reply Clara writes, and that reply waits in the Supervised hold like her own. The reword-and-relay step is deleted. Correction: that reword step dates from Oct 4, not Oct 16.

Shipped as: a test that sends leak-shaped staff replies through the real release path; one reader for staff replies that fails closed; the release path rewired to it, with the old relay deleted.

On 208 real and planted staff replies the new reader is right 96% of the time, against 69% for the old phrase list, with zero false relays (the old one had 41). The leak test: 0 of 21 reach the person.

Live matrix in the incident's shape (Fairhaven, Oct 28)

12 of 12 live cells sent nothing to the person. The setup matched the incident: an applicant texts, Clara asks staff by email, staff reply by email, and the company is fully Supervised. Zero texts went to any test applicant, zero left the Fairhaven line all day, there were zero rows in the AppFolio text outbox, and nothing was written to knowledge. All 31 new holds are accounted for. One defect: a late answer on a matter staff had already closed reopened it, and Clara's held draft added "approved", which staff never said. Caveat: the test applicants came in on Clara's own line, so the AppFolio inbox door never applied to their threads; Fairhaven has no AppFolio account to reproduce that entry. Rerun that evening with more fixes live: see Final live rerun.

Every setting was restored and every hold rejected afterwards. Applicants used Twilio test numbers with made-up names. Full table with ids in the fix's description.

Final live rerun, Oct 28 evening

No test reached a person: 0 sends across 20 live cells, with every fix in production. 15 behaved as expected, 2 have a safe defect, 2 sent nothing but behaved differently than planned, and 1 is still a gap. All four are fixed and in production; three are proven live, one fails safe and awaits a test reviewer identity (Oct 29).

Production ran commit 87f7630e0 with ten fixes merged and live since 21:15Z: the fully-Supervised hold (#22881), the one staff-reply reader with the old relay deleted (#22979), and #23191, #23188, #23194, #23198, #23199, #23192, #23214, #23211. The only later commit (99bd1713) touches evals and CI only. Test companies: Fairhaven (fully Supervised) and Willows (Leasing set to Supervised). Applicants used Twilio test numbers with made-up names; staff replied from a corporate test mailbox.

Morning → now

Every cell

Draft ids are eschold_ + the 8 characters shown.

Controls

What this shows: under these fixes, no staff reply in these 20 shapes reached a person. It does not show a clean AppFolio delivery: that needs a test number on exactly one AppFolio card.

Defects and open fixes

• Correction drops what it didn't change (W-C): #23316, fix #23330 open. Related: Clara learns the correction words alone (#23328).
• Half-answer draft adds an unstated status (P1): the check that every fact in Clara's draft traces to staff's words skips status claims like "hasn't come back yet". #23319, fix in build. Related: #23281, #23282.
• Staff question answered straight to staff (c4): nothing reached the applicant, but staff's words are stored in the applicant's thread. #23318, fix in build.
• Slack text card (case 13): #23317, fix #23333 open.
• Smaller: "renewal" hold label on a leasing draft (#23320); the "already closed" note to staff is held, not sent (#23321); the stored answer keeps the signature (#23322); the AppFolio text joins an old conversation and the ask to staff names the person on file (#23323).

Follow-up fixes, Oct 29

All four fixes are merged and in production (last deploy Oct 29, 11:08Z). Three are proven live; the fourth fails safe and waits for a test reviewer. 0 texts or emails to any person across 20 cells on Oct 28 and 6 follow-up cells on Oct 29.

Controls: 0 texts to the test applicants, 0 from either Clara line, 0 new AppFolio outbox rows at either company, 0 emails outside the test mailbox, 0 knowledge changes.

Still open

• Your call: a test Slack member tied to a Willows admin login, so case 13 can be proven live.
• Parked, undecided: re-freezing the Day 0 replay baseline; deleting the separate team-notices switch.
• Follow-up issues: the four smaller items above (#23320–#23323); teaching saves only the correction words (#23328); a wrong status for a part staff did name still passes (#23346); staff questions needing a lookup get only the on-file summary (#23350); three cleanups from #23348's review (#23367, #23368, #23369); facts-check edges (#23281, #23282). #23316–#23319 closed by these fixes.

Every real staff reply, replayed (Oct 28)

69 real staff replies in production history (Aug 14 to Oct 28): 55 from customer staff (Western Slope 53, Situs 2) and 14 from PropFlow's own team on real-company matters. JP&Co staff have never replied to Clara. 15 of the 69 went through the code the fix changed. On those 15, old code wrong: 3, new code wrong: 4, regressions: 0. All 4 new misses are "unclear" on replies from our own team: nothing goes to the person either way, and staff get one "I couldn't tell" line instead of an answer to their question.

Also in the corpus and left out: 128 replies at the two test companies, 6 desk asks staff never answered, 1 out-of-office. Two of the PropFlow-team emails are the same words sent twice, two minutes apart.

(a) Old code got it wrong, new code gets it right

PropFlow closing notes, for information (the old code never read these): "Closed administratively… no relay to the resident" → handled; "PM judgment… prospect poking the bot" → note; "not interested… no outbound" → note (matter_73b3db5d, matter_a225d287, matter_da2bfab9). A fourth, the voucher decision on matter_8fde2f40, reads as an answer with two clean fact rows and no staff name.

(b) Both right

Of the old code's 7 relays, 2 told the person something staff never said ("I've already reviewed the RFTA packet"; a fresh "confirmed"). The other add-ons were Clara's own tour offers, which the new draft step can add too.

(c) Regressions: new code changes a case the old code got right

None. No reply that the old code handled correctly is now sent to the person wrongly or dropped.

(d) New code is unsure where the old code was right

The cost: on 4 of 15 replies (3 different texts) staff would get "I couldn't tell what to do with your last reply" instead of an answer to their question, or the prospect would wait for a second reply. Nothing would reach anyone wrongly.

Paths the fix did not touch (50 replies)

None of these ran the old phrase list or the new reader, so old and new behave the same: 47 sent nothing to the person, and in 3 the staff member texted the person directly. Run through the reader anyway, it reads 46 of 50 correctly. The 4 misses, which would matter only if these paths were wired:

Gap: on a staff text mirrored from AppFolio, the reader has no idea that staff already texted the person. Today that only settles the matter, so nothing leaks.

The nine Situs texts (Oct 19 to 21) and the Oct 27 message

None of the nine is a staff reply. All are Clara's own texts from the Situs line, so this fix does not touch them:

How this was run

• Read-only against production: database writes and every outbound host blocked. A planted check (one database write, one Twilio send, one SendGrid send, one Graph send) was blocked 4 of 4 and failed the run. The real runs ended with 0 blocked attempts.
• New: the shipped reader (Sonnet 5.5), with the same inputs production builds, on the subscription. Old: the frozen phrase list (reconstructed, since production never logged its label) plus what the records show was sent.
• Grader: Opus 5.5 on the subscription, blind to which side was which. Control: 10 labels flipped, 10 caught. Every flip pointed toward sending, the easy kind to catch.
• Not replayed: Clara's final drafted wording (the draft step writes to the thread). Fact rows were checked against the staff words instead; 1 unfaithful fact, on an unwired path, shown above. Holds use today's Supervised settings, not the settings on the day.

How the gate works

Every text, email and call goes through one of three "doors". Each door checks Off. Each has a Supervised backstop that refuses an unapproved send, but only when the sender names its module and plays by the door's rules. AppFolio writes and calendar writes have no door.

Ordinary Clara replies name no module at the door; they are held one step earlier. A bug there has no backstop.

Leasing

Every prospect message Clara starts on her own is held. The leaks are AppFolio writes, AppFolio's own application email, and three side doors.

Maintenance

Every message to a resident is held, emails included. The work order itself is not: Clara's AppFolio writes run in Supervised. That follows Gera's 10-01 ruling ("only the message waits"); Fede's 10-16 ruling on the assign makes it a leak.

Renewals

Every text, letter and email to a resident is held. The AppFolio renewal offer is not. Where the company lets Clara send paperwork, AppFolio prepares the offer (and likely emails the resident to sign) while Clara's text waits for review.

Collections (short pass)

Reminders wait for review in Supervised and only go out on Live. One narrow 🔴: a reminder approved by the system overnight still sends in the morning even if the company moved from Live to Supervised in between. The eviction email to escalationOwnerEmail is 🟡, or 🔴 if a customer typed a real owner's address there (not read). Collections never writes to AppFolio.

Company not live yet: no renewal or collections path checks it, and neither do the text or voice doors. A company that is not live but has Renewals on would still get renewal calls, texts and AppFolio writes. Only the renewal email is held.

Tests

Nine tests assert a leak as correct. Each fix below has to flip one of them, so name it in the PR.

• regression/renewals/supervised-paperwork.permutations.regression.test.ts:324 (renewal offer written in Supervised)
• renewal-auto-start.test.ts:2161 ("Supervised scans like Live")
• tech-sms-dispatch-supervised.test.ts:409-440 and maintenance-persist-assigns-paged-vendor.test.ts (vendor assigned while the page is held)
• process-envelope-maintenance-drafts.test.ts:399 (vendor reply sent)
• email-lead-engine-reply.test.ts:213 (card written on a held turn)
• module-gate-sms-drafts-backstop.test.ts:142 and outbound-call-supervised.test.ts:166 (an unmarked vendor text or call sends)
• email-supervised-backstop.test.ts:164 (an email that does not opt in sends)

Structural checks. A ratchet counts every "none" sender and goes red on a new one. Nothing checks who an exempt send goes to: a new "team alert" or staff-only email to a prospect passes every check today.

What to fix

Rule for every row: it is a fix, so it ships with no switch, live at every customer on merge. Proof is a Supervised run at the Fairhaven or Willows sandbox showing the write parked and nothing delivered. Rows 2 to 4 wait on decision 1.

Decisions

1. Supervised holds every AppFolio write, not only the assign? Declined for now (feature switches hold writes). Recommend yes. This replaces Gera's 10-01 "actions happen right away". Unblocks PRs 2 to 4.
2. Supervised covers calls Clara starts (renewal day-0 call, AI callback)? Parked. Recommend yes. A call Clara starts can wait; an incoming call cannot.
3. "High" priority work orders keep skipping Supervised? Parked. Recommend hold to emergency only. "High" covers no AC or a dead fridge, and was added by a session, not by you or Gera.
4. The staff-reply label may carry text Clara wrote? Yes, in progress. Recommend no. Today it carries Clara's rewording and, in one path, a full Clara reply.
5. At Leasing Off, Clara still replies to prospects: the tools refuse, the words do not. Keep, or disclose and stop like Maintenance? No (Sept 13 rule). Recommend stop. The ruling that lets her answer was written for Maintenance and is still open in code.

Corrections

• pmsSideEffectMode is read. The earlier finding "nothing reads it" was wrong. pms/side-effects/dispatcher.ts:154 and mode.ts:88 read it. It gates one write: the first-reply note on the card. It defaults to shadow. The guest-card drain does not read it.
• Maintenance email replies are held now. The Oct 3 evidence page says they are not in the build; all three email lanes hold them.
• Staff tour emails are ⚪, not 🔴. Today's Situs inspection rated them 🔴; Fede ruled today that staff notices may flow in Supervised.

Method and limits

Why a new page: the audit spans Leasing, Maintenance and Renewals, and no existing page covers them together. The Situs inspection and the Oct 3 evidence page cover Leasing and the build.

Method: four independent Opus passes over main (Leasing, Maintenance, Renewals, and raw send calls across all modules), each walking from the send or write back to its trigger. Counts are paths per module; the 🔴 list counts kinds. The listing publish appears under both Leasing and Maintenance.

Not verified:

• Which companies have each side switch on, for example AppFolio applications, showing sync, Clara sending renewal paperwork, calls set to Clara, the callback, the publish switch. No database reads were made, so whether each "other switch only" row leaks today is unknown.
• What AppFolio does on its own after a write, for example emailing the resident on a renewal prepare. The robot service is not in this repo.
• Whether the open rulings on "dark still answers" and on the staff-reply label have closed.
• Whether an incoming call in Supervised with calls to the team can still book a tour. Only a prompt line stops it.