Honesty Gate — Overnight Numbers

You asked for real data instead of theory, from a real harness, with no simulation and no live side effects. Overnight we built the actual gate (merged, off by default, three PRs), ran it through two real harnesses — the sandboxed conversation-replay harness (real inbound loop, real model, outbound suppressed) and an offline replay of the gate over every real customer turn from the last 30 days — and labeled every flag adversarially (two independent reviewers per flag + a tie-breaker, all with the full conversation record). This page is the numbers, next to each open decision on the redesign page.

What got built and merged tonight (all off by default)

• The gate itself — deterministic policies + the re-landed "make it true" escalation check + a fast model judge that sees the full turn (tools, results, prior context). Two modes only: off (default, prod today) and enforce. Code review removed the "shadow in prod" middle mode as a forbidden permanent side-path — measurement is offline instead. 7 review rounds, including a real catch: the repair path would have bypassed the standing "escalated turn ships the handoff template only" rule; fixed before merge.
• Replay harness extension — the existing sandboxed conversation-replay harness now records gate verdicts and per-stage latency, plus a report command.
• Offline trace replay — a script that runs the exact shipped gate module over saved real turns (never touching customers), with redacted reports.

Finding 1 — how often Clara actually says something unsupported

Out of 451 evaluated real turns in 30 days, the gate flagged 51. Adversarial labeling (defaulting to "the gate is wrong" unless the record clearly shows an unsupported claim) split them:

What the 19 real catches look like (each verified against the full record): an invented $6.50 admin fee; a wrong deposit amount contradicting the $500 on file; a wrong tour date stated as fact; a tour "confirmed" with no booking made; a reschedule to 2 PM told to a real customer with the system still showing 1 PM; "I've passed this to the property team" with no escalation ever fired; an invented disability-accommodation detail (fair-housing adjacent); a follow-up promised after the escalation tool had failed.

Both false-alarm causes are identified and fixable, so 37% is a floor, not the ceiling:

• 28 of 32 false alarms are voice, and most trace to two input gaps: the judge misreads call-transfer tools (a transfer that succeeded looks "unconfirmed" to it), and it never sees the property knowledge base — so a correct $300 deposit/$38 fee answer from the Willows KB looks invented. Feed it those two inputs and voice precision rises sharply.
• The rest are future-tense phrasing ("I'll pass this along") graded as if it claimed the action already happened.

What the gate misses (false negatives): re-judging 40 turns the gate passed found 11 with real unsupported claims — almost all voice-call summaries asserting prices/quotes where the record can't confirm what was said. Coverage on voice needs transcript-grade input, not the call summary.

Finding 2 — latency (your constraint: zero added to voice)

Practical read: the judge fits async shapes today — flag after send for review, or hold-then-send on email where seconds don't matter — and fits inline SMS only after caching/trimming work proves p95 under ~1.5s.

Finding 3 — cost

235 judge calls over 30 days of full traffic ≈ 504k input + 77k output tokens on Haiku ≈ about $1/month at today's volume. The overnight measurement itself ran on the subscription (no metered keys). Cost is not a factor in this decision.

Finding 4 — what a review queue would actually feel like

• Holding every flag: ~1.7 items/day across the whole portfolio (51/month; Camellia ~19, Willows ~29). At 37% precision, ~1/day is noise — annoying but triageable; each labeled item feeds the Grading Playground and tunes the gate.
• Holding only text channels (SMS+email): ~13/month, 9 real — a queue where most items deserve attention from day one.
• Enforcing only the deterministic "make it true" repair (fire the missed escalation, let the sentence stand): ~1–4/month, labeled real — the same pattern that scored 0/166 false positives in August.

Bonus findings the harness surfaced (pre-existing, not the gate)

• The renewal-offer tool failed 15 of its 20 calls in 30 days (75% error rate) — the single least reliable high-stakes tool in the window. Separately worth fixing regardless of any gate decision.
• Under connection errors, a shared circuit breaker makes Clara send a canned fallback ("Clara hit a snag mid-reply…") across all conversations for 30s at a time — the first harness run tripped it constantly. A real prod resilience behavior, now documented with logs.

The decisions, now with numbers

Suggested next step

One increment, fully measurable: fix the two judge input gaps → re-run this replay → if SMS/email precision holds ≥80%, turn on repair-only enforcement (the 0/166 pattern) for text channels at one property, with every fire feeding the Grading Playground. Everything else stays off until its own replay clears the bar. The 19 labeled catches are queued for your 👍/👎 in the Playground to turn model labels into human ground truth.

Method & honesty notes

Sources: three merged PRs (gate; harness extension; offline replay — repo docs/planning/policy-gate-spec.md holds the spec); offline replay over all 489 real-person turns July 18–Aug 17 from production traces (read-only; drafts for guard-blocked turns are the as-sent text); sandboxed harness runs over the 55-conversation/199-turn eligible corpus (real loop, outbound suppressed, writes to the dev table only); labels from two independent adversarial model reviewers per flag + tie-breaker — not yet human-verified; treat precision as an estimate until you grade the 19 in the Playground. Judge ran with a relaxed 15s measurement timeout (the shipped 2.5s default hides 74% of signal — that number is itself a finding). Voice rows judge the call summary, not a full transcript. Harness latency deltas are from the harness environment, not prod. No customer was contacted; nothing is enabled in production; the gate remains off everywhere.

Measurement 2 — Aug 30 to Sep 13

We ran the same check again, three and a half weeks later, this time only on real customer messages and calls (not test traffic): 1,502 turns from Camellia, Yale 25 Station, and Western Slope. We left out the Willows test property's 881 turns — that's our practice property, not real customers.

The rate of Clara saying something made-up dropped from about 4 in every 100 conversation turns (Aug 17) to about 1 in every 200 (Sep 13):

We found 8 real problems this time, not 19 — but we also checked far fewer turns by hand, so this is a floor, not the full picture. Most were Western Slope voice calls where Clara said a tour was booked or confirmed when it wasn't; one was a text that quoted a wide rent range instead of the one real price on file. Western Slope's call volume here is almost entirely Fede's own practice calls before go-live, not real prospects, so don't read it as customer traffic yet.

Three reasons this number understates the real rate, not overstates it: the automatic checker timed out on 56–92% of its checks and skipped them entirely, so its "16% real" figure is counting fewer checks than it should; the 0.53 only comes from the turns that got flagged plus a spot-check of 40 "looked fine" turns out of roughly 1,485 — most passed turns were never hand-reviewed; and 12 turns were lost to a technical parsing error and never checked at all.

Source: ~/agents/006/kb-honesty-measurement-2026-09-13.md and the matching redacted .json. Method: real production turns replayed through the same gate as the Aug 17 run, each flagged turn double-reviewed blind by two AI reviewers with a third settling disagreements, plus a 40-turn spot-check of turns the gate passed. No prompt, knowledge base, or code changed; nothing was sent to a customer; the gate remains off everywhere.