ADR-0090 — Tool-Call Followthrough Architecture (the Say/Do Gap)

• Status: Accepted (Fede, 2026-07-14). Renumbered 0088→0089→0090: 0088 was taken by the conversation-turn-settled event stream ADR (PR #3742); 0089 was taken by the language-aware outbound construction ADR (PR #3780), which merged first and already has code references.
• Date: 2026-07-14
• Deciders: Fede; Gera
• Relates:
  • PR #3697–#3700 (2026-07-13 reliability hardening), especially #3699 — deterministic acceptance-persistence (persist the tenant's ACCEPTED offer terms at the moment of acceptance, rather than re-deriving them from Clara's own later claim) + a generalized followthrough guard (EndConversation's "duty undischarged" check — observed firing live during this ADR's own experiment, see below).
  • Branch fede/agentic-replay-harness (experimental, not merged) — the replay harness, Prototype B (forced tool_choice), Prototype C (send_reply/reply-as-tool), and the model-call-params fix this ADR's evidence was generated against.
  • ADR-0022 (channel adapter architecture — the real unification event this ADR's channel-drift findings are measured against), ADR-0054 (identity resolution — three unrelated mechanisms + a voice precedence rule, never claimed complete), ADR-0057 (the same promise-without-delivery defect class, previously fixed but explicitly scoped to one voice sub-path).
  • Channel-drift audit, 2026-07-13 (session-local; findings folded into §5 below — not committed elsewhere as a standalone doc).

Context

The failure class. Clara sometimes verbally agrees to, or claims she performed, an action on a text turn (SMS/email) without emitting the matching tool call — she says the reschedule/booking/cancellation happened without doing it. The anchor incident: CAM-F-001, 2026-07-11 — a tenant's tour-reschedule request was verbally acknowledged by Clara with a since-stale time, but the corresponding reschedule_tour tool call never fired, so the tour record was never updated to match what the tenant was told.

This is a canonical class, not a one-off. Prior research (external + internal, synthesized before this ADR) puts hard numbers on it:

• τ-bench: tool-use task success rate <50%, and pass^8 <25% — even a model that usually gets a multi-step tool-use task right does not get it right consistently across repeated attempts. Reliability, not raw capability, is the binding constraint.
• τ²-bench: an 18–25 percentage point success-rate drop specifically on coordination-shaped tasks (multi-step handoffs, state that must be kept in sync across turns) — the exact shape of a "tenant accepts an offer → the acceptance must be persisted" turn.
• No verified evidence a model upgrade alone closes this gap. This session's own model A/B (§2) is consistent with that: mixed, not a clean win.
• Deterministic external gates improve success by +10–12 percentage points on both benchmark tiers — i.e., mechanisms OUTSIDE the model's own judgment (forced tool choice, deterministic pre-processing pipelines) measurably help; the model alone, however prompted, does not reliably self-correct this failure class.
• LLM-judge monitoring of followthrough has AUROC <0.65 — barely better than chance at telling a genuine completed action apart from a claimed-but-not-done one after the fact. This directly predicts (and, per §4 below, is directly confirmed by) why ElevenLabs' own per-call tour_scheduling_accuracy judge missed the real voice incident.

What's already shipped, tonight, ahead of this ADR. PR #3697–#3700 landed a deterministic-pipeline-first response to CAM-F-001's class: #3699 in particular persists a tenant's accepted offer terms at acceptance time (not re-derived from whatever Clara's final reply happens to claim) and generalizes the "did you actually discharge what you promised" guard. That guard is not theoretical — it fired live, unprompted, during this ADR's own replay experiment (§2): on a maintenance healthy-corpus case, the loop logged "REFUSED silence ... duty undischarged: You promised to text the tenant something but the matching send tool was not called in the same turn." That is the mechanism this ADR is deciding whether to widen, replace, or supplement.

The question this ADR answers: given the deterministic pipeline already shipped, do we ALSO need a model-loop-level backstop (forced tool_choice / reply-as-tool), and if so, which one, at what cost, and does it reach voice at all?

Experiment evidence (2026-07-13/14, fede/agentic-replay-harness)

Full methodology, raw per-case-run data (PII-bearing, never committed), and the harness code are on the branch. Summary here is caseId/score-only.

Per-failure-case scorecard (3 real failures × 4 archs × 2 models × k=3 = 72 runs, 0 harness errors):

Healthy-corpus regression (36/39 cases ran — 3 skipped, see limitations; 4 archs × sonnet-4-6 only × k=1 = 144 runs, 0 errors):

current itself "regresses" on 47% of healthy cases with zero prototype changes — this is a noisy baseline from replay-environment-fidelity gaps (synthesized property/unit data vs. the real historical environment), not genuine drift. Read the new-vs-current column only: forced-tool-choice adds 2 new regressions — at or near the noise floor. Reply-as-tool/both add 10 new regressions each — a real signal, not noise.

Latency: forced-tool-choice ≈ free (same or fewer API round trips than current on mutation turns — it forces choice on one call, not every call). Reply-as-tool costs +1.5 round trips and +25–35% wallclock on mutation turns (the mandatory separate send_reply call after every tool result) — the "+1 round trip" tax the mechanism was already expected to carry, now measured.

k/model caveats. Failures ran the full requested matrix (4 archs × 2 models × k=3). Healthy ran 4 archs × 1 model (sonnet-4-6) × k=1 — cut on wall-clock time, not dollars (total session spend ≈ $5.56; the $150 budget guard never triggered). There is no flake-consistency data on healthy and no sonnet-5 comparison on healthy — a real gap, not a finding.

Dev workspace API cap. Hit mid-session (400, "reached your specified workspace API usage limits... regain access 2026-08-01") after the real-corpus sweep finished, mid-synthetic-pack — a shared dev/subscription-key cap, not something this run alone exhausted (total spend well under any per-session limit). Production is verified unaffected — separate key, confirmed live with a 200 probe.

Voice

Mechanisms B (forced tool_choice) and C (reply-as-tool), and their combination D, structurally do not apply to voice. ElevenLabs' hosted runtime generates live voice turns itself; there is no in-process Claude tool-calling loop on PropFlow's side for these mechanisms to attach to (the same architectural fact ADR-0022 and the channel-drift audit both independently confirm — voice is a SessionChannelAdapter with lifecycle hooks, not a participant in conversation-manager.ts's loop).

What the voice-side audit found:

• The June auto-cancel incident is a deterministic keyword→cancel state-machine bug in a SHARED handler — one fix on that shared handler covers both SMS and voice call paths for that specific defect. Not everything about voice reliability needs a voice-native fix.
• All 5 post-generation text guards are voice-blind: the hallucination guard, fair-housing check, PII strip, turnover-fabrication guard, and runaway-message-ceiling all run inline inside handleIncomingMessage — a function voice never calls. Voice's only defense against exactly this incident class is an advisory prompt line, not an enforced check. The hallucination guard's own docstring falsely claims it runs "before Clara's text hits Twilio (SMS) or the voice TTS layer" — it does not.
• The realness-ledger is the closest thing voice has to a followthrough check, and it doesn't reach leasing/tour at all. TOOL_LIVE_EFFECT_ENTITY stamps an "effect landed" breadcrumb (distinct from "tool fired") for exactly 5 hardcoded maintenance/work-order tools, with a 1-hour TTL, and zero leasing/tour keys — a voice-booked tour and a voice tour-booking that silently failed are indistinguishable via the one mechanism built for this exact problem class.
• 21 ElevenLabs eval criteria are computed on every call, with zero alerting wired to any of them — and tour_scheduling_accuracy (a per-call LLM judge over the transcript) missed the real failure. This is a direct, concrete instance of the AUROC<0.65 research finding above, not a coincidence.

Recommended voice mechanism: post-call entity-diff reconciliation, extending the existing realness-ledger rather than inventing new voice infrastructure — widen TOOL_LIVE_EFFECT_ENTITY to cover leasing/tour tools, and after each call ends, diff what the transcript claims happened against the actual Tour/WorkOrder state, using the ledger's existing breadcrumb pattern as the template. This is the voice-side analogue of #3699's "persist on acceptance, verify against ground truth" shape (§Alternatives, option e), not a from-scratch design.

Channel-drift protections

Channel unification is real where it was deliberately built — the router/pipeline layer (ADR-0022) — and genuinely absent downstream of it, in the say-do-specific guard/audit layers ADR-0022 never addressed. The channel-drift audit confirmed 12 divergences (see the confirmed forks below); it also ranked concrete, cheap protections against the repo's existing *.drift.test.ts enforcement pattern:

Confirmed forks this ADR's recommendation must account for: D2 (email tenants have no automatic maintenance work-order filing path at all — Clara's own prompt unconditionally tells her she lacks a create tool, which is false for email); D3 (the older, un-hardened, phone-keyed confirm/cancel state machine can commit a stale/wrong time with live side effects — calendar sync + PM notify — on SMS before the #3699-hardened pipeline runs on the same turn); R2/R3 (all 5 text guards are voice-blind, and the hallucination guard's docstring falsely claims voice coverage); D1/R4/R5 (audit-trail gaps: mislabeled trigger/channel on email, the deterministic pipeline's own mutations never produce a Conversation tool_use row at all, and voice's ledger has no leasing keys); R6 (decline can silently escalate to cancel, channel-uniform).

Prior art — the say–do detector already substantially exists (reuse, don't rebuild)

Review feedback (Gera, 2026-07-14) surfaced that this ADR's first draft proposed detection mechanisms without citing the in-house infrastructure that already implements the same shape. Verified against the repo:

• The fake-detector (src/lib/domain/evals/fake-detector/detect.ts, the "Sherlock" checkpoint in Maestro): claim → independent fresh read of the real source (Twilio / AppFolio L4 / DDB — never the run's own stored actual) → fail-closed verified | fake | unverifiable, writing one EVAL_SHAME# row per fake with the contradicting read attached. This is exactly "Clara said she did X — did it actually happen," already deterministic and already live for maintenance claims.
• expectation-shame (src/lib/domain/evals/fake-detector/expectation-shame.ts): the expectation-driven second audit that catches the silent-omission variant (a promised dispatch SMS that never sent produces no claim — this module walks the graded expectations instead). Built from two real incidents of precisely this ADR's failure class.
• assert-tool-followthrough (evals/lib/assert-tool-followthrough.ts): commitment-phrase → must-have-matching-tool_use grading, today scoped to renewal_escalate only.
• The realness-ledger (§Voice): the voice-side breadcrumb half of the same idea, today scoped to 5 maintenance tools.

Principle adopted from this review: one detection brain, not three. The recommendations below are re-stated as extensions of the fake-detector family rather than new systems. The replay harness on fede/agentic-replay-harness remains what it was built as — a one-off experiment instrument for this ADR's evidence — not a production detection system; ongoing text regression coverage belongs in the existing eval/golden checkpoint machinery (mustNotHappen / tool-fired checkpoints), extended to leasing/tour.

Decision

Recommendation (proposal — Fede & Gera decide; revised after Gera's 2026-07-14 review):

1. Deterministic-pipeline-first stays the primary layer (already shipped, #3697–#3700). This experiment's strongest single data point — CAM-F-001 passing 23/24 — is consistent with that bet paying off for the incident that motivated it.
2. Detection before prevention: extend the fake-detector family to leasing/tour, on both text and voice.
   • Voice post-call reconciliation = Sherlock's claim→independent-verify core pointed at Tour/WorkOrder state after real calls, with the realness-ledger's breadcrumbs as one input (widen TOOL_LIVE_EFFECT_ENTITY to tour tools) — not a new detector. Same fail-closed classify, same shame-row output surface, new claim extractors + verifiers for tour entities, plus a production (non-eval) trigger at call end.
   • Text: extend assert-tool-followthrough beyond renewal_escalate to leasing/tour commitments, and add leasing/tour mustNotHappen/tool-fired golden checkpoints — the standing regression net for this class, replacing any notion of productionizing the branch harness.
3. Defer the forced-tool_choice backstop (Prototype B) behind a tripwire. It measured nearly free (+2 regressions ≈ noise, no latency cost), but it is also this ADR's weakest-justified build: Open question #4 concedes no corpus case reproduces a gap the shipped pipeline still misses. With the detection net from #2 in place, the honest sequencing is: the first confirmed post-#3699 text say–do failure caught by the detector re-opens Prototype B, whose implementation stays ready on the experiment branch (classifier scope-widening past tour-offer-acceptance shapes remains the prerequisite if/when it ships).
4. Ship channel-drift protections #1–#3 immediately (PR-template question, CI grep, channel-parity.drift.test.ts) — cheap, and each maps to a confirmed fork already sitting in production.
5. Explicitly do NOT ship reply-as-tool (Prototype C) platform-wide. It is the vendor-blessed reference pattern for exactly this problem, but the data doesn't support it here: no unique fix over Option B on either case B already fails to fix (CAM-F-002, CAM-S-001), a real regression signal (+10 new vs. current, not noise-level like B's +2), and the highest measured cost (+1.5 round trips, +25–35% wallclock on mutation turns).

What would change this verdict:

• A reproducing text failure — a case where the shipped deterministic pipeline (#3699) itself still misses the say-do gap under live replay. Today's corpus doesn't have one (CAM-F-001 didn't reproduce); until it does, there's no direct evidence #3699 needs a text-side backstop at all, only a plausible one.
• Reply-as-tool's regressions disappearing under a prompt-iteration pass paired with a genuine k=3 run on the full healthy corpus (today's k=1 single-model run cannot distinguish flake from real regression with confidence — see limitations, §2).

Consequences

Commits us to:

• Extending the fake-detector family (claim extractors + independent verifiers for Tour entities, a production post-call trigger, assert-tool-followthrough + golden checkpoints widened to leasing/tour) — real work, but on one existing detection brain instead of three new systems.
• Widening TOOL_LIVE_EFFECT_ENTITY past its deliberately-narrow inception scope (§ "what stays legitimately channel-specific" in the audit).
• Prioritizing 3 cheap-but-not-zero-effort drift protections that don't fix anything by themselves — they prevent the next instance of an already-confirmed bug class.
• Keeping Prototype B maintained-but-dormant on the experiment branch until the detection net produces a reproducing post-#3699 text failure (the tripwire in Decision #3).

Makes easier: future audits of "did channel X get this fix" have a written coverage-boundary doc and a standing parity test to point to, instead of a reactive dig.

Makes harder / explicitly NOT foreclosed: reply-as-tool is deferred on this evidence, not rejected forever — the "what would change the verdict" criteria above are the reopening condition, not a permanent no.

Explicitly out of scope for this ADR (see Open questions): CAM-F-002's actual fix (a conversational/language bug, not an architecture gap), CAM-S-001's actual fix (a prompt/policy/eval change, not a tool-forcing gap), and any reply-CONTENT quality gate — this ADR only covers whether a tool call fires, not whether what Clara says is correct once it does.

Alternatives considered

• (a) Status quo — deterministic pipeline only, no model-loop backstop. Evidence: CAM-F-001 consistent-with-fixed. Cost: zero, already shipped. Gap: no backstop for whatever the pipeline can't parse (CAM-F-002's shape) and no coverage at all for cases the pipeline was never designed to reach. Kept as the foundation, not sufficient alone — hence (b).
• (b) Targeted forced tool_choice (Prototype B), scope widened. Cheap, safe (+2 regressions ≈ noise), zero measurable latency cost. Current scope misses CAM-F-002's shape. Recommended, contingent on the scope-widening work actually landing.
• (c) Reply-as-tool (Prototype C), platform-wide default. Vendor-blessed (Anthropic cookbook reference pattern for exactly this problem). Measured cost: +1.5 round trips / +25–35% wallclock on mutation turns; +10 new healthy regressions (real, not noise). No unique fix demonstrated over (b) on either case (b) already fails on. Rejected as a platform-wide default on this evidence — see "what would change the verdict."
• (d) Both B + C. Inherits (c)'s cost and regression profile without a demonstrated unique benefit over (b) alone — same fail/fail/pass pattern across the 3 real cases as (b) and (c) individually. Rejected for the same reason as (c).
• (e) Post-conversation reconciliation (Gera). The stale-time extraction trap is the load-bearing risk: a naive "extract whatever Clara's final reply claims" pass would have extracted the wrong time on CAM-F-001 (2:00 PM, the stale time her broken reply actually stated) — it must extract the accepted offer (what the tenant agreed to), not the final claim (what Clara said afterward). This is precisely the insight #3699's deterministic acceptance-persistence already encodes for text. Also unresolved: no clean SMS/email "conversation ended" signal to trigger a reconciliation pass; mid-conversation staleness (a tenant asking "confirmed for X?" before reconciliation runs gets a stale-DB answer — a new say-do-gap shape, not a fix for the old one); a double-booking window before the slot is held. Deferred — sound direction for voice specifically (§Voice, where it's the only option that reaches that channel), but for text it should extend #3699's already-proven persist-on-accept pattern rather than become a from-scratch offline architecture.
• (f) Bake the next 5 days of slots into the prompt (Gera). Real read-side latency win (skips a check_availability round trip), composes with the repo's existing HMAC-signed slot tokens (issueSlotToken/verifySlotToken). Does not close the write-side gap this ADR is about — the failure is "didn't call the mutation tool after acceptance," not "didn't know what was available." Orthogonal, not a substitute — could ship independently as a latency optimization regardless of this ADR's outcome.
• (g) Model upgrade alone (e.g., adopt sonnet-5 broadly). Config traps first: sonnet-5 rejects a non-default temperature/top_p/top_k with a 400, and defaults to adaptive thinking that can silently consume a fixed max_tokens budget down to an empty reply — both real integration bugs this session's own model-call-params.ts fix had to close before sonnet-5 could even be A/B'd. Behavioral evidence once fixed: 3/3 regression on concession-naming, 3/3 win on objection-handling — a genuine mixed bag, not a clean improvement. Consistent with the research verdict that no upgrade alone closes this class. Rejected as a standalone fix — may still be worth adopting for its own separate merits once the config traps are handled platform-wide, but not as a substitute for the mitigations above.

Open questions

1. Dev workspace API cap — raise the existing cap vs. cut a dedicated key for the remaining consistency runs (k=3 / 2-models on the full healthy corpus, the synthetic pack). Production is confirmed unaffected (separate key, live 200 probe).
2. CAM-F-002's actual fix path — presents as a language/conversational-looping bug in how Clara handles a preference correction, not a tool-forcing gap. Flagged as its own investigation (Gera's bug #1 territory), not resolved by this ADR.
3. CAM-S-001's actual fix path — a prompt/policy/eval change (what Clara should say when escalation is right but no new contact path exists), not an architecture change. Not resolved by this ADR.
4. Corpus curation gap — no case in today's corpus reproduces the exact incident class (a say-do gap the already-shipped deterministic pipeline itself still misses) under live replay; CAM-F-001 didn't reproduce post-#3699. Until a reproducing text case exists, the case for a text-side backstop beyond #3699 is plausible, not demonstrated.