Duplicate reschedule notifications: the agent re-executed a move it couldn't see

RCA · 2026-08-10 · Camellia Apartments, prod conversation 8258d02a · incident fix shipped in PR #5644

TL;DR — A prospect moved her tour to 2:00 PM by SMS. The PM got the same "Tour rescheduled" email twice, plus the leasing-activity Slack post twice. The duplicate email was only the symptom: the ingestion pipeline had auto-applied the move, but its only evidence to Clara was an ephemeral prompt block that dies at the end of the turn — so on the prospect's "Thank you!!" one turn later, Clara's own anti-dropped-reschedule guardrail read the transcript as an unexecuted move and re-executed it. The fix persists every pipeline-applied tour change into the transcript as a first-class record, so the "don't re-act" signal finally survives turn boundaries the way the "always act" rules do.

What the PM saw

Time (MDT)	Event
2:10 PM	Prospect asks about Saturday tours; books 1:00 PM; identifies as Ciara Baird.
2:12 PM	Tour confirmed at 1:00 PM · PM notification #0 (correct — a fresh booking).
2:31:53	Prospect: "would it be possible to move the appointment to 2pm?"
2:31:58	PM email #1: "Tour rescheduled … 2:00 PM" — sent by the ingestion pipeline, which auto-applied the move before Clara's turn ran.
2:32:00	Clara (words only, no tool call): "Your tour has been moved to 2:00 PM."
2:32:25	Prospect: "Thank you!!"
2:32:31	Clara calls `reschedule_tour(Aug 15, 2:00 PM)` — the same slot. A second `rescheduled` history event is written.
2:32:37	PM email #2 — identical — plus the duplicate Slack activity post.

The tour row confirms it: two rescheduled history events to the identical time, 39 seconds apart.

Root cause — five mechanisms lined up

1 · The pipeline moves the tour deterministically (by design).Since the July 8 "Jose" dropped-reschedule incident (PR #3345), a concrete, available counter-proposal is applied by the ingestion state machine, not left to the agent's judgment. This part worked perfectly — including notifying the PM.

2 · The proof is ephemeral.Clara learns about the move via a [COMPLETED ACTIONS] block injected into that turn's prompt only — never persisted to the conversation. The next turn rebuilds the transcript from persisted messages, where the move left zero trace.

3 · She couldn't have called the tool that turn even if she wanted to.The same-turn conflict guard strips reschedule_tour from her tool list when the pipeline already acted — structurally forcing the words-only reply her own guardrails classify as a hard failure.

4 · The durable rules point one way.The Jose-incident guardrail — "never just agree in words; a spoken confirmation with no reschedule_tour call leaves everything at the old time" — reads from the persisted transcript and survives turn boundaries. The counter-rule ("if it's already handled, don't re-call") reads from the ephemeral block and omitted "rescheduled" from its action list. Repair rule durable, suppression rule dead: on "Thank you!!", Clara did exactly what her prompt ordered and re-executed the move.

5 · The notification dedup keys on the event timestamp.Every reschedule appends a history event with a fresh timestamp — even a same-slot one — so pmConfirmNotifiedAt can never recognize a duplicate. Second event, second email, second Slack post.

Why our test suite was green through all of this

The reschedule gauntlet (9 live prod scenarios, revived in July) verified the tour moved — but after both writes the time is correct. Its PM check was existence-only (reads the latest notification), and its tool checks are cumulative booleans. Nothing counted anything, and no scenario sent a follow-up message after a successful reschedule. Unit tests had the same shape: every pipeline test asserts one call's outcome; the PM sender was asserted as "was called," never "called once." A structural repeat of the exact blindness the suite was revived to fix — green runs proving the wrong invariant.

The fix (PR #5644)

Layer	Change
Root cause	Every definitive pipeline tour action now persists as a synthetic `tour_pipeline_action` tool record (system-authored, truthfully labeled) in the conversation. Future turns' transcripts show the move as executed — the repair guardrail sees compliance, not a drop.
Prompt	The suppression rule now includes rescheduled and recognizes the persisted record; the accept-own-offer rule gets a matching "already executed → acknowledge, don't re-call" carve-out. New eval case (CASE 11) scores the exact incident shape.
Guard	The hallucination guard's pending-time-change flag now clears on a pipeline-applied move (it previously stayed latently armed forever).
Idempotency	`reschedule_tour` to the already-confirmed slot is a no-op — no history event, no notification — for any caller (retry, double-tap, race). Defense-in-depth, not the fix.
Harness	New exactly-once checks (`pm_notified_count`, `reschedule_event_count` — settle-polled, fail-fast on excess) and gauntlet scenario RG-10: book → confirm → move → "Thank you!!", asserting one history event, one PM notification, tour still confirmed. This scenario fails on the pre-fix code and gates the nightly forever after.

Verification

RED Incident reproduced on pre-fix prod (run SMS-1786398272219, live SMS against the test property): after "move it to 2pm" then "Thank you!!", the tour carried two rescheduled history events for one logical move — reschedule_event_count: 2 > 1 — DUPLICATE detected. The exact double-write that re-fired the PM's email on Camellia.

GREEN Fix verified end-to-end (run SMS-1786398813913, same scenario against the fix branch's full preview pipeline — webhook → queue → branch-deployed agent Lambda): 15/15 checks. "Thank you!!" got "Looking forward to seeing you Friday!" with no tool call and exactly one reschedule event.

Plus: unit tests for the audit-pair shape and transcript reconstruction, the idempotent no-op vs. a genuine move, the count-check semantics, and the wire-guard boundary; a new prompt-eval case scoring the exact incident shape; typecheck and lint clean; the scenario runs in the nightly gauntlet from now on.

Found along the way, filed separately: the test property's leasing calendar has been disconnected since mid-July, silently muting the suite's calendar/PM-email assertions (Trello: "Reconnect The Willows test property's leasing calendar"); and a ~40-minute prod window today where inbound SMS from unknown senders was dropped between two mainline deploys.

What we're taking away

When a state machine acts on the agent's behalf, the action must be visible wherever the agent's rules look. We wrote durable, transcript-driven rules ordering Clara to always execute, and an ephemeral, single-turn signal telling her when not to. Any pair of guardrails with asymmetric lifetimes will eventually fire against each other. Second lesson, again: a green harness proves only the invariants it actually counts — "notified" and "notified exactly once" are different tests.