Grounded Claims

Updated Aug 15, 2026 · DECIDED: Option B (Fede, 2026-08-14) — claims ledger now, escalation family first; C adopted selectively for the highest-stakes sentences · build under way

In plain terms

Every fabrication incident we've had is the same event: Clara states something about the system's state — "it's booked," "the team has it," "we have openings today" — that no record backs. Prompt rules can't stop this; the model writes plausible text, and plausible isn't the same as true. The fix that works is structural: a claim about system state doesn't reach a customer unless a record backs it. We already built this for bookings. This page decides how far to take it.

The evidence, one day's worth

Bookings (fixed today): the booking guard exists precisely because Clara once invented a confirmed key pickup — and today we fixed its opposite failure, where truthful tour restatements were being rewritten because their grounding wasn't wired in. Both directions of the same lesson: the checker is only as good as the record feed.
Escalations (unguarded): "That's already in the queue! The leasing team is working on the unit-201 switch — they have it." Nothing was in any queue. No rail exists for this claim family — even the booking guard's own fallback copy promises a handoff without creating one.
Availability (unguarded): a Spanish-speaking real caller was told "we have availability today" when the slot list held nothing for that day.

Adversarial verification, 2026-08-14: three attack passes over two harness runs plus a production sweep. Most claimed defects fell as grader error; these survived.

Decision: how do we kill the class?

The options are a ladder, not rivals — the question is how far to commit.

A · Patch the next family only. Add escalation claims to the existing booking guard (claim detector + escalation-record grounding, and the fallback copy creates the escalation it promises). Days of work, smallest blast radius — but the next unguarded family (availability, fees, dates) waits for its own incident. B · Claims ledger.Recommended One per-turn "grounded facts ledger" every producer registers into (tool results, context blocks, escalation writes), and one post-turn checker owning all claim families as data — booked, escalated, available — each family ~50 lines instead of a new 800-line guard. Includes A as its first migration. The [C-1] lesson institutionalized: a context block that asserts state must register itself, so today's over-block bug can't recur by construction. Roughly a 1–2 week refactor. C · Receipts-first composition. Go further: state-bearing sentences are rendered from records (the tour row, the escalation write's result), and free prose never carries a confirmation at all — fabrication becomes structurally impossible instead of caught. Strongest guarantee; biggest change to how replies are composed, and post-hoc deletion bugs (the scope-erasure template) go away with it. Weeks, staged by claim family.

Recommendation

B now, with A as its first shipped increment (escalations are actively unguarded and observed failing). Adopt C selectively where the stakes are highest — confirmations and handoff promises — as claim families migrate. The harness validity fixes land in parallel so every step is measurable against the replayed real-caller corpus.

Notes for the decision

Production sweep results (appended after the decision)

All real conversations since Jul 15 (237 threads, 1,266 outbound Clara messages, cross-checked against tool traces): 25 escalation-status claims — 21 fully backed by records, 2 ungrounded, 2 borderline.

One real harm case: Jul 17, a prospect calling about parking was told "the team will follow up, usually same day." No handoff tool fired; her ask landed in a notes field nobody watches; no human ever contacted her. She had conditioned a tour on the answer — she never came back.
The second ungrounded claim came from the booking guard's own fallback text ("I've passed this to the property team") — the safety copy promising a handoff it doesn't create.
Every claim made alongside a real handoff tool was truthful, 21 of 21 — evidence the ledger approach works where it's wired. Every ungrounded claim came from the voice agent free-styling reassurance, which is what the phone-lane design (pre-call grounding + rails + post-call repair that creates the promised escalation) closes. Latency constraint honored: nothing new runs mid-call.

Correction — Aug 15 evening

Everything in the section below was reverted the same day. The booking guard (PR #5731) rewrote three real Camellia customers' correct replies with canned text; replay over real history showed 0 true positives. Both it and increment 1 of this architecture (#5769) are out of production (PRs #5790, #5791). Decision on what replaces them is deliberately paused for analysis first. Full account: RCA — Aug 15.

Shipped overnight (Aug 15) — where this stood at 1 AM (superseded, kept as history)

Plain English

Two things are live in production this morning. First, the booking-honesty guard stopped rewriting real tour confirmations (it was eating 42 good replies a day, one to a real caller) — since the fix went live at 8:23 PM MT it has fired zero times. Second, the first slice of the "grounded claims" design is live for texts and emails: when Clara tells someone "the team has this" and nothing is on record, the system now creates the handoff for real and lets her sentence stand; only if that fails does it correct her out loud. No promise ships without a record behind it. Phone calls are untouched (by design — nothing new runs mid-call).

How we know it is safe. Rather than trust unit tests, we replayed the new check over every text/email reply Clara sent in the last month — 2,594 real turns — with the exact code that shipped, no AI involved.

First pass: it would have fired 12 times, and 8 of those were wrong (it flagged ordinary sentences like "the property manager will call you at your scheduled time"). Those four mistake-classes were fixed and pinned as tests before merge.
After the fixes: 4 fires, all correct — the same shape each time: the handoff email failed to send, and Clara still said "the property team has been alerted." Now the system retries the handoff, and if it still can't, tells the person the truth.
On the 166 replies to real people in that month: zero fires. Every real handoff sentence Clara wrote over text had a record behind it. The only ungrounded handoff promise a real person received over text came from the old guard's own fallback text — the follow-on already named above.
Reviewer's independent finding, also fixed before merge: a booked tour must not switch the check off for the whole conversation (that would have re-opened exactly the July harm case's scenario).

Also fixed: the test bench itself. Last night's "Clara invented a signed lease" verdicts on the tour stage were the bench's fault (it re-used an identity that genuinely had signed a lease in an earlier run), not Clara's. The bench can now start a stage on a fresh identity.

What is next (in order): make the old booking guard's fallback text create the handoff it promises (or stop promising) · the phone-call lane — pre-call grounding, prompt rails, and an after-call auditor that creates any handoff the agent promised, with zero added mid-call latency · re-run the tour stage on the fixed bench for the honest before/after number (this morning's attempt was blocked by production API rate limits — being retried).

Context: booking-guard fix PR #5767 (merged) · ledger PR #5769 (merged) · bench fix PR #5771 (merged) · fake-escalation card fnUGgrf8 · availability card 0nBHa43P · scope-erasure card efGLobwt · harness validity card 7VKa0hXH.