RCA — Aug 15, 2026: Clara's new "booking guard" rewrote real customer replies, and Camellia's team emails bounced

Written 2026-08-15 evening (MT) · Owner: Fede · Status: production restored to pre-guard state; root causes confirmed; long-term fixes NOT started — deliberately. This page is the handoff. All times Mountain (MT) unless marked UTC.

1. Timeline

2. Incident A — the booking guard

2.1 What it was

A post-turn check in Clara's reply loop (agents/clara/lib/agent/no-fabricated-confirmations.ts, called from conversation-manager.ts). Pattern-matching (regex) over the reply: if it saw a clock time + the word "tour"/"pickup", or a confirmation verb, or a promise-to-book, and no booking tool had returned ok/success in that turn, it threw away the whole reply and sent fixed copy: "I've passed this to the property team and they'll confirm a time with you. Nothing is scheduled yet — I don't want to give you a time that isn't real." A copy of the discarded draft was parked on the thread with a [BOOKING-GUARD-BLOCKED …] tag — which the conversations UI rendered as if it had been sent (the "show blocked drafts as blocked" UI change planned after #5764 never shipped).

2.2 Who introduced it, when, why

PR #5731, merged Aug 13 10:22 PM MT by fede-propflow — a Claude Code session working the Willows journey-harness bench, where Clara had (on the synthetic bench) stated a key-pickup time no tool produced. The fix was reasonable in intent; the shipping decision was the failure: blocking on day one, fleet-wide, no shadow mode, no flag, no replay against real conversations. Prompt half also rendered into 6 voice-agent prompts.

2.3 Why it fired on true sentences (root causes, ranked)

1. Shipped blocking with no trial period. An observe-only week would have surfaced every false positive at zero customer cost.
2. Never tested against real conversation history. All evidence was synthetic: hand-written unit sentences, a promptfoo gauntlet, the Willows bench. The deterministic replay technique existed (built for the escalation guard in #5769) and was never pointed at this guard.
3. The replacement was maximally destructive. Discard the whole reply, substitute copy asserting "nothing is scheduled yet" — which in all 3 real fires was itself the false statement.
4. The slot-offer exemption never worked in production. check_availability and get_tour_slots return no ok/success field, so a real availability call could never license the times Clara quoted. Cause of 14 of the 28 monthly false positives (machine recount from the replay data; an earlier hand tally said 15) and of Alexis's block.
5. "Don't promise to book" was implemented without its precondition ("…if you cannot call the tool"). Clara had schedule_tour/book_tour available; the guard never checked.
6. Clock time + "tour" in one clause = "offered a slot". Cancellations, reminders, restatements all have that shape (Ciara, Noa).
7. Pre-agent state changes are invisible to the machinery. Ciara's tour was cancelled by the deterministic tour-confirmation handler 4.2 s before her agent turn. Corrected mechanism (traced 2026-08-15 evening, session archaeology + code + prod trace): the earlier "queue-boundary race" hypothesis is refuted — router and agent turn are one invocation, and the handler's plain-English note ("Tour on … has been cancelled") very likely did reach Clara (her blocked reply mirrors it). What failed: the legacy handler returns its note with no action field, and every machine consumer — the tool-catalog strip, the #5644 transcript-persist, the guard's grounding — filters on action ∈ DEFINITIVE_TOUR_ACTIONS, so all three ignored it (handle-tour-confirmation.ts cancel branch vs capabilities/leasing.ts, tour-pipeline-audit.ts). The #5644 session knew: it flagged the legacy confirm/cancel handler as a third decider "not folded into the new mechanism … a follow-up". Fix is one field: have the legacy handler emit the modern action vocabulary.
8. No kill switch short of a code revert + two deploys (web app and Lambda).

2.4 Numbers (all from replaying the real guard code over real production traces, read-only, no AI)

The four false-positive mechanisms in the 28: (A) slot tools can't ground — 15; (B) tour real but booked outside the turn's tool blocks (pipeline action / reschedule recorded after the reply) — 6; (C) no claim at all, just an offer to book — 4; (D) Clara honestly saying she can't confirm — 1; (E) Clara replying to a system notification email — 2. These A–E labels are from the in-session hand-read of the 28 threads and aren't reproducible from the saved replay files; the replay's own machine reason-codes are 25 no-booking-tool-fired / 2 slot-offered-without-tool / 1 booking-tool-errored.

2.5 Why every check said it was safe

• #5731/#5767: unit tests hand-inject succeededTools — none built from a real tool result, so the check_availability shape mismatch was invisible. The Willows harness cannot observe a guard block at all (its client keeps only tool_execution events); when the bench went red, the AI judge blamed the model.
• #5769: ran a real-history replay — of the other guard (0/166 real-person fires, all true positives). That number felt like proof and was about a different predicate.
• Sentry: bench and real blocks share one issue; pipeline-lab isn't a synthetic tag and sms-stress is tagged production. 40+ bench events buried the 3 real ones.
• Repo rule ("done for an agent-loop change = harness green against real history") satisfied on paper, not in substance.

2.6 What we did (all live)

• Threads: Alexis aabe42f4-… and Ciara 8258d02a-… set to human takeover 1:13 PM; Ciara SMS sent 1:19 PM (Fede-approved text); Alexis emailed the exact Clara draft that had been blocked (Fede-approved).
• #5790 (revert ledger) merged 1:24 PM; #5791 (revert guard + follow-ups; 31 files, +22/−3,527 lines per GitHub; only conflict resolutions were keeping later, unrelated imports and deleting two guard-only marker tests + the guard's registry id) merged 2:31 PM. Both deploy targets verified on the merge commit 2:46 PM.
• Repo-wide: the forbidden sentence and the guard predicate no longer exist in code.
• Camellia full-history replay harness built (branch fede/camellia-guard-replay, not pushed: scripts/eval/replay-camellia-guards.ts, npm run replay:camellia-guards; reports kept local — they contain names/phones).

3. Incident B — Camellia team emails

3.1 "Tour cancelled" email to the team: never existed

Verified by a read of the whole code history and of all 11 Camellia cancellations since April (9 earlier + today's 2): applyCancel() only writes the tour history, signals Temporal, and deletes the Outlook calendar hold. No email, no Slack, no outbound-log row. The "team update" email that exists (notify_leasing_team, kinds on-the-way / running-late / …) has no "cancelled" type. The Outlook deletion can't notify either: the calendar event is created without attendees, and today's two tours had no outlookEventId at all. Nothing merged in the last two days touches this path. It is a missing feature, not a regression. What does exist and works: "Tour confirmed: <name> — Camellia Apartments · <when>" on booking.

3.2 Microsoft bouncing our email provider

Clara's team notifications go out via SendGrid on a shared sending pool (no dedicated IP on the account). Today Microsoft 365 (JP's inbox) blocklisted two of those pool addresses: 167.89.40.80 and 50.31.49.42 — bounce 550 5.7.511 Access denied, banned sender. In SendGrid's 30-day log, every bounce to a jp-co.com address is from today: 4 total (Montez confirmation 11:57 AM; two signature reminders 1:00 PM — later retried OK; Caleb cancellation notice 3:53 PM). Nothing on our side changed. Why nobody knew: the SendGrid event webhook (/api/sendgrid/event) acts only on renewal-tagged bounces; every other bounce is acknowledged and dropped. No fallback path existed.

Disputing with Microsoft: (1) Microsoft delist portal (sender.office.com) for both addresses; (2) SendGrid support ticket — they own the pool and delist at scale; ask for a dedicated IP.

4. Also surfaced today (not root causes, need owners)

• Blocked-draft UX never shipped — parked drafts render in the conversations UI as if sent (planned after #5764).
• Anthropic prod key 429s 2:10–10:10 AM; consumer unidentified; needs the console usage page.
• Ciara's turn ran with empty completedActions after the handler cancelled her tour — Clara was right by inference, not by being told (mechanism unconfirmed).
• Bench ≠ real separation in Sentry (see 2.5).
• Nikolas Espinal (Apr 17 cancellation) prospect row not found under its personId — data-integrity loose thread.

5. Root cause, one paragraph, plain English

Clara got a new safety net that decides "is this appointment real?" by one test only: did Clara call a booking tool in this exact message. True things that arrive any other way counted as lies. It was turned on for every real customer the day it was written, without a trial period and without ever being run against a single past real conversation, so nobody learned that its most common trigger is a true sentence. When it fired it deleted Clara's whole answer and sent a fixed apology that was itself false. Separately, the "cancellation email to the team" that Fede expected never existed, and the day's attempts to email the team ran into Microsoft blocklisting our provider's shared sending addresses — silently, because we only watch bounces on renewal emails.

6. Prevention — the rules to adopt (proposed for CLAUDE.md / review lens)

1. Real-history replay is the merge gate for anything that can alter customer-facing text. Deterministic replay of that exact code over ≥30 days of real-person conversations (the Camellia harness now exists), reporting turns / fires / false-positive count, with every real-person fire listed and labelled in the PR body. "Unit tests pass" is not evidence about production text.
2. Observe-only first, always. ≥3 days in production recording verdicts and paging on real-person hits before any guard is allowed to change what a customer receives.
3. Pattern-matching (regex) may never replace customer text. It may annotate, park, page, hold. Replacing needs grounded evidence of falsity, a model-scored judgment, or a human. (Fede: "no regex on the future version.")
4. Every guard has a one-command rollback that reaches every deploy target (web app + Lambdas), and the rollback is exercised before the guard ships.
5. Bench and real are separated in telemetry (source/property on the Sentry extra; pipeline-lab synthetic), and the journey harness keeps safety_check events and asserts guard_blocked=false on truthful fixtures.
6. Bounces to a team inbox alert same-day, for every email kind, not just renewals.
7. Two delivery paths for team notifications: the property's own connected mailbox first (Microsoft-native), SendGrid as fallback; retry across paths on bounce; request a dedicated SendGrid IP.

7. Fix-forward — decisions for Fede (multiple choice, recommendation marked)

7.1 What replaces the guard, if anything?

7.2 Team notification on cancellation

Build the missing feature: "Tour cancelled: <name> — <property> · <when>" fired from applyCancel() (deterministic, no AI), logged on the thread as pm_tour_cancellation_email, mirroring the confirmation email. Small. Recommend yes.

7.3 Email delivery hardening

Per §6 items 6–7. Recommend all three (alerts, mailbox-first, dedicated IP + delist).

8. Handoff — open items