PropFlowAI · ADR-0088 implementation · July 14, 2026
Say–do detection: before / after
Where the platform stood before today's three implementation PRs, and where it stands now.
The failure class: Clara says a tour was rescheduled, booked, or cancelled without the matching
tool call ever firing (incident CAM-F-001, July 11). ADR-0088's decision: extend the existing
Sherlock fake-detector — one detection brain — to leasing/tour on both voice and text, plus three
channel-drift protections. Maintenance paths untouched throughout, per scope lock.
3 PRs open · CI green
ADR accepted & merged — final number ADR-0090
Maintenance untouched
No prod API keys used — validation ran on subscription
Coverage map
The detection surface, before and after — every cell traces to a verified repo read or an agent report from this session.
| Surface | Before | After |
|---|
| Voice · post-call |
No reconciliation of any kind. Realness ledger covered 5 maintenance tools, zero leasing/tour keys. All 5 text guards voice-blind. ElevenLabs' own per-call judge missed the real incident. A voice-booked tour and a silently-failed booking were indistinguishable. |
Every voice call is claim-audited at call end against a fresh Tour read. Two shapes caught: said-but-no-tool (the CAM-F-001 shape) and tool-fired-but-state-disagrees. Fail-closed verdicts, one durable shame row per fake, ledger widened +4 tour tools. |
| Text · regression net |
Followthrough checking covered exactly 1 tool (renewal_escalate) on 1 dataset. The helper had zero unit tests. |
4 tools across 3 commitment families (reschedule, booking, cancellation) incl. passive shapes ("you're all booked for…"). 8 multi-turn golden cases, 63 unit tests, k=3 repeat wired for CI/nightly. |
| Channel drift |
Fork D1 live in prod: email tour confirmations written to history as via voice_handler (sms). No PR template. No guard against hardcoded channel provenance. No SMS↔email parity test. |
D1 fixed — history now says via tour_confirmation_handler (email); channel is a required param (a default was the re-introduction vector). Repo-wide provenance drift guard + multi-turn SMS↔email parity test + PR-template parity question. |
The three PRs
One concern per PR. All three: full suite green, tsgo clean, adversarially audited.
Voice post-call tour reconciliation HOLD for #3729
PR #3773 · 4 commits · new modules voice-tour-claims / verify / reconcile + production trigger
Before
- Detection ended at maintenance; leasing/tour voice claims never verified
- Ledger stamps existed for 5 maintenance tools only
After
- Sherlock's claim → fresh-read → fail-closed core pointed at real calls; same shame-row surface, zero new detection systems
- Causal (time-ordered, consume-once) claim↔tool correlation; retractions pinned as not-fakes
- Ledger-empty discriminator: TTL-expired/outage reads unverifiable, never a false accusation
- Checks
effect:* stamps (success), not tool:* (mere firing); shame receipts PII-scrubbed
Evidence: 28,170 tests green · 98.85% branch coverage on new modules · maintenance entries pinned byte-identical by a drift guard · trigger fail-soft proven via observed logs, double-layered so it can't break call handling.
Text followthrough net → leasing/tour HOLD for #3729
PR #3770 · 2 commits · assert-tool-followthrough generalized + new golden dataset
Before
- One commitment family → one tool, renewals only
- Hallucinated tour claims in SMS/email invisible to evals
After
- Reschedule / booking / cancellation claims must be backed by the matching tool in the same turn
- 8 golden cases incl. the exact CAM-F-001 shape, earlier-turn promises, retraction safety
- Renewals behavior pinned byte-identical (12-input comparison vs. main)
- Live eval deliberately gated out of PR CI until a supervised first run
Evidence: 63 unit tests · subscription-harness k=3 scorecard 24/24 posted on the PR · adversarial probes now correctly fail all hallucinated-claim shapes.
Channel-drift protections + D1 fix HOLD for #3729
PR #3771 · 4 commits · 2 new drift guards + PR template + provenance fix
Before
- Email confirms/cancels stamped
voice_handler (sms) into durable Tour history — false provenance in prod
- Nothing stopped the next hardcoded-channel literal from landing
After
- Real channel threaded from the router; honest trigger name; ~34 call sites updated
- Provenance guard scans the repo for trigger+channel literal pairs into shared writers (6-entry justified allowlist, stale-pin + scan-health self-checks)
- Parity test: identical fixtures through SMS and email, multi-turn propose→confirm/cancel, 3-iteration stability
- Every PR now answers: "Does this apply equally to SMS and email? Voice equivalent, or why not?"
Evidence: 10 tests fail when the D1 fix is reverted (captured twice) · guard catches a synthetic new violation · zero false positives repo-wide.
What the verification loop caught
Chronological. None of these were found by the builders' own green test runs — each needed an independent adversarial layer.
- k=3 probe"You're all booked for 3 PM tomorrow!" with no tool call passed vacuously — the booking family missed passive phrasing, Clara's most common shape. Fixed + probes re-run: now fails correctly.
- Audit · #3770PR CI would have live-run the never-tested eval with real Anthropic key spend, contradicting the PR body. Gated out of CI mode; body corrected.
- Mutation testThe "tool args must not trip the check" test survived having the filtering deleted — vacuous. Replaced with a fixture proven to bite.
- Audit · #3773Demonstrated wrong verdict: a hallucinated claim whose own tool call failed borrowed a later claim's success and read verified — a false negative in the exact direction the detector exists for. Ordinal matching replaced with causal, consume-once correlation + 7 regression tests.
- Audit · #3773TTL-expired ledger would have produced false accusations on late webhook redelivery. Ledger-empty discriminator: no-signal → unverifiable, never shamed.
- claude-botVerifier read
tool:* (fired) instead of effect:* (succeeded) — a fired-but-failed tool would verify a false claim. Now requires the success stamp.
- claude-botProspect PII was flowing into durable shame rows. Receipts narrowed to non-PII fields.
- claude-botA
channel = 'sms' default was a silent D1 re-introduction vector. Param made required, no default.
- Both auditsADR number collision: main already has an ADR-0088 (turn-settled event stream, PR #3742). #3729's ADR needs renumbering (→ 0089) before merge, then a one-sweep reference update across all three PRs. Your call, with the #3729 review.
How it was verified
28,170
full-suite tests green on the largest branch (0 failures, all three PRs)
20 / 20
sabotage mutations caught across the two audited detector PRs (1 survivor found → fixed)
24 / 24
k=3 × 8 multi-turn cases passed, graded by the PRs' own assertion functions
98.85%
branch coverage, new voice modules (threshold 94%)
$0
prod / dev Anthropic API spend — live validation ran on your Claude subscription
10
tests proven to fail on pre-fix D1 code (captured, then restored green)
Method: each PR was built by its own agent, then independently attacked by a separate auditor —
mutation testing (break the code, require the tests to notice), vacuity checks, false-claim tracing
of every PR-body assertion, and a three-simulator k=3 harness generating Clara's turns from the real
prompt + tool definitions. Honest caveats: simulators are Claude-family models roleplaying the provider
call, not the exact production API call; the first live promptfoo run of the new eval is deliberately
deferred to a supervised session.
Open — waiting on you
- ADR PR #3729 — your accept/reject, plus the 0088→0089 renumber (collision is real, verified on main). On accept I'll sweep references and merge all three PRs in order.
- Golden-set eval flake on #3771 — pre-existing LLM-eval flake on an ambiguous fixture; zero diff on the files it tests (verified). Rerunning; not a blocker unless you want it chased separately.
- First supervised live run of the new tour-followthrough eval (needs a non-prod key or a nightly slot).
- Prototype B stays dormant per the ADR — the detection net shipping here is the tripwire; its first confirmed post-#3699 text failure re-opens forced tool_choice.