0101 — Clara turn integrity: one commit path, one message per turn, no silent failures

Context

On 2026-07-17 a prospect emailed Camellia asking to tour "today (July 17) … maybe 1 pm". Within 41 seconds the system sent him two uncoordinated emails that contradicted each other: a templated "Tour Confirmed — Monday, July 20 at 1:00 PM works perfectly" (wrong date — his move-in date), then a Clara reply narrating the internal mix-up, admitting she couldn't look up times (a prod secret was missing in the serving Lambda), and handing him the building buzz code as a workaround.

Each defect has a point fix (shipped separately). This ADR addresses the architecture that let four independently-minor defects compound into a customer-facing failure — and that absorbed related incidents before (CAM-F-001 dropped reschedule; two-months-silent PM tour SMSes; the renewal "Sent"-while-stranded projection).

The architecture as it exists today

flowchart TD
    subgraph inbound ["Inbound (propflow-inbound-processor-prod Lambda)"]
        A["Email arrives
(Graph webhook → SQS FIFO,
group key = sender email)"] --> B["inbound-router
classification"] B --> C["PRE-AGENT STATE MACHINE
processTourRequest"] B --> G["AGENT LOOP
agent-loop.turn (Clara)"] C --> C1["Haiku extraction
preferredDate / time / moveIn
⚠ no confidence field; prompt says
'resolve relative dates yourself'"] C1 --> C2["reconcileTourReplyDate
⚠ first weekday token in RAW BODY wins
('Move-in Monday…' beat 'today')
⚠ consistency gate exists on reply path only"] C2 --> C3["findTourSlot
bookable?"] C3 -->|yes| C4["applyTourIntent autoConfirmed
⚠ COMMITS on first touch: CONFIRMED tour +
Outlook event — no handshake, no confirmation
⚠ no commit-layer lock; reschedule mints new Tour"] end subgraph writers ["Uncoordinated customer-facing writers"] C4 --> W1["Writer 1: SendGrid template
'Tour Confirmed — works perfectly'"] C4 --> W2["Writer 2: PM confirmation email"] G --> G1["tools: get_tour_slots ⚠ failed —
error string dumped into model context"] G1 --> G2["guards: hallucination-guard
⚠ English-regex, time-only, date-blind;
validated against the WRONG persisted tour"] G2 --> W3["Writer 3: Clara reply via Graph"] V["Vercel PM routes"] --> W4["Writer 4: prospect SMS
⚠ silently dead since May 17
(missing smsFrom, swallowed, HTTP 200)"] T["Temporal workflows"] --> W5["Writer 5: reminder SMSes"] end W1 -.->|"41s apart, contradictory"| W3

Flaws (each with incident evidence)

What production practice says (research summary)

Sierra, Decagon, Intercom Fin, Salesforce Agentforce, and Anthropic's agent guidance converge: the model interprets, plans, and drafts; deterministic code validates and commits; escalation to a human is a first-class action. The published failures (Air Canada — liable for its bot's invented policy; Cursor's fabricated login policy; Project Vend) share one root cause: a customer-visible commitment originating from free-form output with no deterministic chokepoint. Tool-failure practice never lets a raw error reach the model (OpenAI failure_error_function, LangGraph handle_tool_errors); guardrail practice puts output policy in code, channel-symmetric (NeMo output rails, Guardrails AI REFRAIN); say/do practice is transactional outbox + provider-webhook confirmation + "never let the agent be its own witness." Full citations in the RCA thread.

Decision

Adopt a turn-integrity architecture: one pipeline per inbound trigger, one commit chokepoint, one composed customer message, one output gate, one send rail, an explicit turn-health state — and a first-class Anomaly Review Gate that routes big failures to humans without gating the happy path.

Scope: the harness-level machinery — TurnRecord/turn health, the Anomaly Review Gate, typed tool results, the output-gate skeleton (no internal narration, no access-code/secret leakage, no fact tokens outside generated spans), one-message-per-trigger, and the say/do rail — applies to every Clara agent loop in every mode (leasing, maintenance, turnover, renewals, PM chat) on every channel. The per-domain pieces — claim-class templates ("what counts as a fact-bearing sentence and which record verifies it") and domain commit chokepoints — onboard incrementally: leasing tours first (S1–S3, where the incident and the validation corpus are), then maintenance/turnover (vendor dispatch, appointment windows, scope/cost claims — absorbing the existing dispatch/charge guards rather than duplicating them) and renewals (offer terms and dollar figures, complementing the existing Temporal/idempotency gates). A domain whose claim templates haven't landed still gets all universal protections — it just lacks domain-specific claim verification until its slice ships.

flowchart TD
    A["Inbound trigger
(serialized per PERSON, not per email address —
cross-channel races collapse here)"] --> U["UNDERSTAND (LLM)
structured: visitDate + moveInDate as separate fields,
each with an evidence span; raw body never reaches date logic"] U --> P["PROPOSE (durable ProposedAction)
origin lineage: prospect_stated /
machine_derived / machine_derived_override"] P --> V{"VALIDATE (code)
slot handshake · real future date ·
span-overlap + consistency checks (BOTH paths) ·
policy + consent + suppression"} V -->|"conflict / disagreement
(deterministic, never LLM self-report)"| Q["ASK, don't guess
max 2 rounds → then review gate"] V -->|pass| CK["COMMIT chokepoint
tiny synchronous txn: state + OutboundIntent
atomically, then RELEASE — LLM never runs under the lock
idempotency key = fn(personId, actionType, targetSlot)
supersede semantics for reschedule/cancel
per-channel kill switch
first touch commits PROPOSED, never CONFIRMED"] CK --> COMP["COMPOSE — exactly one message
persists concrete payload into OutboundIntent BEFORE send;
template demoted to DEFERRED FALLBACK (60s SLA),
not deleted — customer is never left in silence
channel = prospect's inbound channel
(email→email, text→text, call→text, consent-aware email fallback)"] Q --> COMP COMP --> OG{"OUTPUT GATE (code, all channels)
GENERATE, DON'T CHECK: fact-bearing sentences
are template-generated from the verified record;
gate refuses date/time/code tokens outside those spans
code leakage matched by IDENTITY (actual values +
derivable pointers) + topic lexicon, never by 'shape'
voice: runs on full pre-TTS string
ships OBSERVE-ONLY first; enforcement scoped to
high-risk claim classes after precision is measured"} OG -->|pass| RAIL["SEND RAIL (unified; email migrates in)
outbox relay retries FIXED BYTES, never re-runs the LLM
provider webhooks + active status polling → DeliveryEvent
stall SLA: undelivered 5 min → alert (not 24h)
relay heartbeat monitored"] OG -->|fail| ARG TF["Tool failure
typed result {ok|retryable|fatal, safe_message}
raw error never enters context"] -.->|fatal| ARG subgraph gate ["ANOMALY REVIEW GATE (see section below)"] ARG["Turn → DEGRADED"] end RAIL --> REC["SAY/DO RECONCILER
claimed-but-undelivered (after provider poll + grace) ·
delivered-but-unrecorded → review gate"]

The Anomaly Review Gate (the human-review architecture)

Purpose: big failures like 2026-07-17 reach a human the same hour, without putting the happy path behind approval. Normal turns — clean extraction, tools healthy, gate-passing reply — remain fully autonomous with instant booking and confirmation. The gate is an anomaly path, not a queue in front of the product.

flowchart TD
    S["Detection signals"] --> D{"Dedup + rollup
(existing bake-alert infra:
per-hour bucket + durable 24h key,
keyed on root-cause signature × property)"} S1["tool fatal in a customer turn"] --> S S2["VALIDATE conflict unresolved
after 2 clarify rounds"] --> S S3["output-gate refusal"] --> S S4["say/do mismatch from reconciler"] --> S S5["override-origin proposal +
bare-affirmative confirmation"] --> S S6["explicit human request / frustration / loop
(body-derived signals are UNTRUSTED:
per-sender rate limit — no HITL DoS)"] --> S D --> CB{"Circuit breaker:
degraded fraction over threshold
in window?"} CB -->|"yes — systemic outage"| OUT["OUTAGE MODE (per property/channel)
one rolled-up page ('SendGrid fatal ×47 in 10m'),
per-turn paging stops,
holding messages capped at ONE
per conversation per outage window"] CB -->|"no — isolated anomaly"| ITEM["Review item (deduped)
conversation + turn record + gate verdicts
+ the parked draft attached"] ITEM --> PAGE["#alerts page, tagging
@Fede + @Gera (standing rule)"] ITEM --> HOLD["Customer: fixed holding template
through the rail's consent/quiet-hours gates —
tracked on the conversation: same-reason reply
gets NO second holding message, bumps priority instead"] ITEM --> ACT["Human resolution actions:
approve parked draft · edit & send ·
take over thread · discard + manual outreach"]

Operational rules (each defuses a red-team break):

Principles (each maps to flaws and red-team breaks)

  1. LLM interprets; code validates; one commit chokepoint. (F2, F3) The extractor emits visitDate and moveInDate as separate fields with evidence spans (the exact substrings read); whole-body weekday scanning is deleted; the !consistent → ask gate applies to request and reply paths alike. "Confidence" is never an LLM self-reported scalar (injectable) — it is a deterministic function of disagreement between fields, spans, and re-derivation. Clarifying questions cap at 2 rounds, then the review gate takes the turn. The chokepoint transaction is tiny and synchronous (state + OutboundIntent atomically, release; compose/send run outside the lock off the durable intent), keyed and serialized per person — cross-channel simultaneous inbound (email + phone call) collapses to one winner and one superseded proposal, never two commits. Identity caveat (review finding #1): person resolution is itself a fragile surface (identity-cache drift, unresolved inbound, placeholder cohorts), and a serialization key that splits when identity splits recreates the double-commit. So: when person resolution is missing or ambiguous at inbound time, the lock falls back to the coarser conversation/thread key (correct-but-narrower protection, never none); reliable person-resolution is a named prerequisite with its own validation bar before the per-person key is trusted cross-channel; and supersede looks up prior open proposals by person AND conversation, so a drifted personId cannot orphan a wrong-date tour it can no longer find. Idempotency keys are pure functions of (personId-or-conversationId, actionType, targetSlot) so retries and duplicates both collapse. Reschedule and cancel are first-class transitions through the same chokepoint with supersede semantics on the prior tour, intent, and calendar event (the CAM-F-001 class). Same-day carve-out (review finding #5): "first touch = PROPOSED" would make a same-day request ("today ~1pm") wait on a confirm round-trip that can outlive the tour window — so a first-touch commit MAY auto-confirm when all of: origin lineage is prospect_stated (never machine-derived or override), the slot is validated bookable, every consistency check passes, and the requested time is within office hours. That carve-out still flows through the chokepoint and the output gate; anything less than all-four parks as PROPOSED. A per-channel kill switch lives inside the chokepoint so one bug can be contained without stopping all booking everywhere.

  2. One inbound trigger → exactly one customer-facing message, channel-matched. (F1, F4) When a composer will reply (email always), it is the only primary writer — and the standalone template is demoted to a deferred fallback, not deleted: if no composed payload reaches the provider within the SLA, the outbox releases the minimal safe template. Channel rule per product decision 2026-07-17: email→email, text→text, phone call→text, with consent-aware email fallback for callers who can't receive SMS.

  3. Typed tool results; raw errors never reach the model or the customer. (F6) Every tool returns {ok | retryable | fatal, safe_message}; retryable → bounded retry under the stable idempotency key; fatal → review gate. The model sees at most a curated sentence.

  4. Turn health with a first-class abort — the Anomaly Review Gate. (F7) As specified above: detection signals, dedup/rollup on existing bake-alert infra, circuit breaker + outage mode, fate-isolated holding messages, tagged pages, and human resolution actions on the parked draft.

  5. Channel-symmetric output gate in code — generate, don't check. (F5) A post-hoc claim-checker over prose is a trap both ways: deterministic checkers are paraphrase-, negation-, and language-blind (the current guard proves it), and an LLM checker is an injectable witness. Invert the burden: on committing turns, fact-bearing sentences (dates, times, addresses, codes) are template-generated from the verified record; the gate refuses date/time/code tokens outside the generated spans — a string diff, not prose understanding. Date-token definition (review finding #2): the refusal lexicon for dates is the calendar resolver's locale token tables in the reply's language — weekday names, today/tomorrow/relative words, ordinals, and numeric date shapes — not numeric tokens only, so "see you next Monday" against a record saying today is refused, not slipped. False positives are handled by construction, not tuning: legitimate day-words belong inside generated spans (fact sentences are template-generated, so any correct date mention is already a span), and non-committing turns aren't gated for dates at all. Access-code policy keeps the current prompt's stricter rule (prospects never receive codes automatically — "confirmed + imminent" was a regression enabling book-harvest-cancel); code requests route by verified sender identity (staff / resident / prospect). Code leakage is detected by identity, not shape (exact/normalized match on actual code values and derivable pointers like last-4-of-office-phone, plus a topic-lexicon block) — a 4-digit buzz code is invisible to entropy filters. On voice, the gate runs on the fully-composed pre-TTS string; committing turns buffer the sentence before speaking. The gate ships in observe-only mode — verdicts logged, never blocking (an observe-only rollout of a single new gate, not a dual-path fork) — with a hard enforcement trigger (review finding #3): 200 observed committing-turn verdicts per claim class or 4 weeks, whichever comes first, then scoped enforcement with the per-channel kill switch as the fast rollback lever — real volume is low (the RCA path saw 5 events in 11 days), and "enforce when precision supports it" with no trigger is observe-only-forever. Enforcement scopes to high-risk claim classes (asserted tour date/time, access codes, dollar figures, confirmations), with normalization ("1 PM" ≡ "13:00") and tool-result provenance so knowledge-base facts aren't false-flagged.

  6. Say/do closure. (F8, F10) Outbound intent written transactionally with state; the outbox relay retries fixed bytes (the persisted composed payload), never re-runs the LLM; all channels — email included, sized honestly as the bulk of the migration (~36 direct SendGrid call sites, no email quiet-hours or consent gate today) — join the enveloped rail; provider webhooks plus active status-API polling with a grace window feed the reconciler (a single lost webhook must not page); stall SLA is minutes (undelivered in 5 → alert), and the relay itself is heartbeat-monitored so a dead relay is an alarm, not a 24h-later discovery. No code path may report success it cannot evidence.

  7. One copy of the harness. (F9) agents/clara/lib fork consolidation; guards and gates exist in exactly one module. Sequencing (review finding, minor): the output gate, typed tool results, and review-gate code are born in a single shared module in S2 — imported by both the fork and src/lib — even though full fork consolidation trails in S5; the safety-enforcing code is never built twice or allowed to drift while it's the thing enforcing safety. Voice's escape hatch becomes real: transfer_to_number (coded, currently disabled — no VOICE_PM_TRANSFER_NUMBER) ships before S2 claims voice coverage; voice gates the commitment (structured booking under timeout), not live prose.

  8. Provenance lineage, not a provenance boolean. (Red-team "laundering," observed in prod: all four other affected prospects agreed to times offered without slot data.) Proposals carry origin lineage; a bare affirmative against a machine_derived_override proposal does not commit — it triggers a restating confirmation ("Just to confirm — Monday the 20th, or today?"). Only a confirmation restating the fact in the prospect's own words upgrades an override-origin proposal. Prospect-derived text (notes, aiNotes, summaries) never enters trusted context tags on any later hop.

  9. Language is a first-class dimension, end to end. (Product decision 2026-07-17; red-team found the current guard is English-regex-blind and the resolver English-token-only.) Camellia's inquiry forms carry a locale field and its prospect base includes Spanish speakers — every layer must handle at minimum en + es natively, extensibly per property locale: the extractor reads any-language text (validated empirically); the calendar resolver carries locale token tables (hoy/mañana/lunes…domingo, accented and unaccented, abbreviations) — an unresolvable verified span is an ASK, never a guess; clarifying questions, holding templates, and the generate-don't-check fact templates ship localized variants selected by ADR-0089 language provenance — a Spanish-speaking prospect gets the safe fallback and the confirmation in Spanish, and the output gate's generated-span matching works because the spans themselves are generated in the reply's language. A language the system doesn't carry templates for degrades to ASK/review, never to silent English or a skipped gate. Validation bar: the same wrong-date-0% standard applies to every supported locale before it's declared supported.

Entity classification (ADR-0027)

Single-writer conformance (ADR-0079; review finding — canonical entities must name their writer, cf. the writerless-Organization lesson): each canonical entity below has exactly one registered writer.

Entity Class Naming Single writer (ADR-0079) Spine trace (canonical) OR derived-from / rebuilt-by / drift-tolerance (derived)
ProposedAction canonical bare name the UNDERSTAND/PROPOSE stage (turn-pipeline.propose) via personId + propertyId; carries origin lineage + extractor evidence spans
OutboundIntent canonical bare name the COMMIT chokepoint (turn-pipeline.commit) — sole minter; the outbox relay only transitions delivery status via conversationId/personId; written transactionally with its state change; stores the concrete composed payload before any send attempt
TurnRecord canonical bare name the turn runtime (turn-pipeline.turn) — opened at trigger, closed at terminal state via conversationId; one per inbound trigger; health state, gate verdicts, clarify count
DeliveryEventSnapshot derived *Snapshot suffix n/a (derived) from: provider webhooks + active status polling · Rebuilt by: webhook ingestion + reconciler · Drift tolerance: ≤1h for alerting-relevant states (stall SLA is minutes; the 24h figure applies only to historical backfill)

Migration (revised per red-team: flags everywhere, no slice enabled before its safety net)

Every slice lands behind a feature flag, default off, with an explicit "worse-than-today?" check before enablement — and every flag carries a removal trigger: when a slice's replacement path graduates (its validation bar passes and the flag has been on in prod for one clean week), the superseded path and the flag itself are deleted in that same slice's closing PR, never left as a permanent fork (one-source-of-truth rule). Concretely: S1's graduation deletes the whole-body weekday scan and the old date-resolution branch; S2's deletes the raw-error-into-context path; S3's deletes the pre-agent machine's direct commit path (the ADR-0090 fork dies here); S4's deletes the direct-SendGrid call sites it migrates.

Consequences

Trace architecture & UX impact

The existing per-turn trace stack (trace collector, withDeliveryTrace, the conversation /trace route, admin agent-traces) is kept, not replaced — TurnRecord formalizes what traces capture ad hoc and references the same turn. Additive changes: traces gain structured stages (extraction with evidence spans, validation verdicts, commit decision, output-gate verdict — logged from observe-only mode on, turn health, delivery events); typed tool results are recorded as both the raw internal error (full debugging fidelity) and the curated safe_message the model actually saw. UX changes: (1) the conversation timeline shows real delivery states per outbound (queued/sent/delivered/failed chips from DeliveryEvent) plus parked drafts and holding messages as first-class items — the UI can no longer display success for a send that never happened; (2) the Anomaly Review Gate's review item is the trace + parked draft + gate verdicts with approve / edit-and-send / take-over actions, making the trace view the evidence pane of the review surface (standard detail-page layout); (3) voice traces gain a pre-TTS gate stage. All customer-facing wording on these surfaces follows the no-technical-language rule — a PM sees "Waiting for your review — Clara couldn't verify a tour time," never gate/verdict/DEGRADED vocabulary.

Alternatives considered

Validation bar (how "safe" gets proven)

Empirical, not asserted, and adversarial-until-dry: each domain's pipeline must survive (1) a benchmark corpus with programmatic ground truth (per supported locale), (2) a full replay of every real prod scenario in history at original timestamps, and (3) iterated adversarial break rounds that continue until a full honest round produces zero wrong-commit kills. The first break campaign against the leasing prototype proved why the last leg is non-negotiable: after 635 benchmark cases at 0% wrong dates, an adversarial round produced 74 reproducible wrong-date kills (week-offset arithmetic gaps, a missing English weekday+ordinal cross-check, a tag-conditional consistency-guard bypass, truncation burial, quoted-history dates, cross-field contradictions) — every one a named, cheap fix, and every one invisible to non-adversarial benchmarking. Standing invariants those kills bought: the span-vs-date cross-check runs on every branch that yields a bookable date (never tag-conditional); relative-offset phrases ("a week from Tuesday") are resolver arithmetic, not dropped tokens; weekday+ordinal consistency is checked in every supported language; extraction sees the full body (no fixed-length slice) with quoted history stripped and quoted-only dates ineligible; structured-field vs free-text disagreement asks; benchmark-only validation is never sufficient to declare a claim class safe.

Red-team record

Thirteen operational breaks (B1–B13) and eleven adversarial breaks were filed against the draft; the material ones and their dispositions: B1 suppress-then-crash silence → deferred-template fallback + payload-before-send (P2, gate rules); B2 escalation flood → bake-alert dedup + circuit breaker + outage mode; B3 holding-message fate-sharing/loop → rail-gated holding messages, one-per-cause, provider-down skip; B4 cross-channel race → per-person serialization + supersede + stable keys (P1); B5 voice → commitment-gating, pre-TTS buffering, transfer_to_number prerequisite (P7); B6 clarify loop → 2-round cap + observe-only conversion instrumentation; B7 lossy webhooks → status polling + grace + relay heartbeat + minutes-scale stall SLA (P6); B8 duplicate sends on retry → business-stable idempotency keys; B9 gate false positives → observe-only mode + normalization + provenance + scoped classes (P5); B10 chokepoint SPOF/lock-across-LLM → tiny txn, compose outside lock, per-channel kill switch, outbox lands with chokepoint (P1, S3); B11 reschedule orphans → supersede semantics (P1); B12 injectable frustration trigger → untrusted-signal rate limits (gate); B13 migration seams → flags + S1-suppression-gated-behind-S2 + PROPOSED-never-"Confirmed" invariant. Adversarial: extractor conflation/evidence spans and request-path gate (P1); generate-don't-check and identity-based code detection (P5); access-code regression reverted (P5); provenance laundering (P8); ASK-loop injection cap (P1); tainted-notes containment (P8).