0058 — Agent actions must be reliable by construction: every tool an agent can call really does what it claims

• Status: Proposed — generalizes the ADR-0057 invariant from two hand-fixed tools into a tool taxonomy + a three-layer enforcement ladder. No code in this ADR beyond the ADR-0057 follow-up already landed on fede/unknown-caller-always-notify (the escalate_to_human voice-wording fix + its wording tests). The taxonomy field, the generalized guard, the voice reality-eval, and the reviewer lens are each scoped as follow-up work below.
• Date: 2026-06-16
• Deciders: Fede, Jose
• Supersedes nothing; builds on: ADR-0057 (the unknown_caller promise-without-delivery fix), the Tools Platform catalog (ADR-0028), the AppFolio tools eval (evals/run-tools-eval.ts), and the cloud reviewer's "no mock theater" gate (.github/reviewer-lenses/evals-and-tests.md §H).

Context

Goal: Clara's actions are trustworthy by construction — she can never tell a caller or PM that something happened unless the tool actually did it. Reliability here is not "fewer bugs"; it is the invariant that a promise Clara speaks is always backed by a real side effect. The failure mode below is what breaks that invariant; the taxonomy + guard restore it.

ADR-0057 found two tools on the unknown_caller voice path that looked like they acted but didn't: capture_unknown_caller_note (wrote a dashboard note nobody watched, no notification) and escalate_to_human (stamped transferredTo + said "Transferring call…" while placing zero telephony and paging no one below high urgency). It fixed those two by hand and fenced them with per-tool guards.

But the class is wider than two tools. A wider audit of 1,377 voice calls in ADR-0057's own context surfaced the same shape elsewhere (lost work orders, unsent renewal texts). The root failure mode is structural, not incidental:

An agent (Clara) is given a tool. The tool's name and the prompt tell her it does X. The handler does not do X — it records a row, returns a hardcoded success string, or no-ops — and Clara faithfully tells a human "X is done." The lie is invisible because the unit test mocks the handler and goes green.

The standing follow-up to ADR-0057 (this branch) removed the last hand-findable instance of the wording half: escalate_to_human no longer narrates a transfer it cannot perform, on any channel. With that done, the remaining work is to make the class impossible to reintroduce, rather than waiting for the next production incident to surface the next instance.

The mental model that organizes the fix (Fede/Jose, 2026-06-16): there are two kinds of "tool," and one of them should not be a tool at all.

• A real action happens now, synchronously, with a real-world side effect — paging a human, placing a transfer, sending an SMS, telling a caller in a gas emergency to hang up and dial 911. Clara invokes it and her wording must be derived from what the tool confirms it did.
• Deferred / "we'll do it later" work — taking a message, drafting notes, opening a work order off the call's content — is the maestro's job, done after the call ends (the maintenance maintenanceWorkflow already runs gather → judge → dispatch → persist on signalCallEnded, ADR-0031). Clara does not need a tool that hands her a fake "done" mid-call; she needs prompt language that frames the capability ("I'll take this down and make sure the team gets it"), while the maestro reads the transcript post-call and performs the real delivery. A tool that exists only to record intent for later is a tool whose success is always a guess — so it should not be surfaced to the agent as an action.

Decision

Adopt a tool effect-class taxonomy for every tool surfaced to an agent, and enforce the ADR-0057 invariant per class through a three-layer ladder.

1. Effect-class taxonomy (a field on the agent-surfaced ToolSpec)

Every tool reachable by an agent (voice or SMS/text Clara) is classed:

• real-sync — performs a real-world side effect during the turn and returns a delivery receipt (a typed confirmation: delivered/emailDelivered/messageId/transfer leg/etc.). Clara's spoken or written wording is derived from the receipt, never hardcoded ahead of it. Examples: escalate_to_human (pages PM, returns emailDelivered), the EL-native transfer_to_number / transfer_to_agent system tools, an outbound SMS send, a 911-safety flag.
• record-deferred — writes local state that a post-call / maestro process acts on. Its agent-facing wording MUST be future or neutral tense ("I'll make sure the team gets this"), never past-tense "done." Preferred form: not a tool at all — the maestro reads the conversation post-call and performs the delivery, and the agent gets prompt language instead of a tool. A record-deferred tool is permitted only when the maestro genuinely needs an explicit in-call signal that transcript-reading can't reconstruct; that exception must be justified in the PR.
• readonly — lookups, no side effect. Lying here is a wrong-answer problem (covered by domain evals), not a fake-action problem.

The class lives on the spec so all three enforcement layers reference one contract.

2. Three-layer enforcement ladder

Layer 1 — Cheap CI structural fence (every PR, $0). Generalize the per-tool wording⇒delivery guard ADR-0057 hand-wrote (unknown-caller-promise-delivery.repro.test.ts, the escalate wording tests) into a single drift test over every agent-surfaced handler:

• If a handler's return message can match the follow-up / notification / transfer promise pattern, the tool MUST be classed real-sync AND its return MUST carry a delivery-confirmation field.
• A record-deferred tool's return wording must be future/neutral (no past-tense "done"/"sent"/"transferred"/"opened").
• A tool with no effect-class is a failure (forces the author to classify).

This makes "the agent promises X with no verifiable side effect" fail to merge, for all tools, not just the two we already caught.

Layer 2 — Deterministic reality eval against the appfolio-45 bench (daily canary + pre-cutover gate). The machinery already exists for the AppFolio L4 catalog: evals/run-tools-eval.ts fires each tool against the real bench and asserts a real response shape, and src/__tests__/tools-eval-fixture-coverage.test.ts fails CI if a catalog tool has no fixture. Extend this to the voice/agent webhook tools — fire the real handler → real integration and assert the side-effect actually happened. Crucially, the assertion must read the external source of truth, never PropFlow's own state (workflow-preference, Gera 2026-06-16, docs/planning/real-source-funnel-bench-handoff.md): a "message delivered" check queries the Twilio/SES API for the delivery, a "work order created" check queries AppFolio for the WO — it does NOT self-validate from the PropFlow DDB row that the handler just wrote, and never falls back to a local value on a read failure. Self-validation is how a silently-dropped message (Temporal down) passed green. Wire a daily run as a launchd canary alongside the spine-health / pms-sync canaries, and require it green before any live voice-tool cutover. This is the layer that catches "the handler is shaped right but the integration is dead" — which neither a unit test nor Layer 1 can see.

Layer 3 — Reviewer-lens rule (codifies the gate culturally). The cloud reviewer already enforces "no mock theater" (reviewer-lenses/evals-and-tests.md §H). Add one rule to the tools-platform lens: any tool surfaced to an agent must be real-sync (with a receipt + a Layer-2 reality fixture) or record-deferred (future-tense wording, no fake "done"); a handler that returns a hardcoded success/promise string with no verifiable side effect is a merge blocker, and a new record-deferred tool must justify why it isn't maestro-level deferred work. This puts the human/bot reviewer on the same contract the tests enforce.

Entity classification

No new entities. This ADR adds a classification field to existing ToolSpecs and three test/eval/reviewer surfaces. No data-model change.

Consequences

• Easier: a whole class of "fake tool" bugs becomes un-mergeable (Layer 1) and un-deployable-while-broken (Layer 2); the agent's promises are true by construction; the next instance of the ADR-0057 class is caught at PR time, not by an angry customer follow-up.
• Harder / commits us to: classifying every agent-surfaced tool (one-time sweep); a real-bench fixture per real-sync voice tool (Layer 2 costs real sends on the appfolio-45 bench — the same cost the existing tools-eval already pays); a daily canary to own and watch; and a discipline shift — new "take a note / do it later" capabilities default to maestro post-call work + prompt language, not a new tool.
• Migration of existing tools: capture_unknown_caller_note is the canonical record-deferred candidate to re-examine — ADR-0057 made it real-sync (synchronous in-call email) to satisfy the invariant, but the maestro model would instead have the post-call worker deliver it and let Clara speak in future tense. That re-shaping is explicitly out of scope here (ADR-0057 shipped the synchronous form; revisiting it is a separate decision once this taxonomy is adopted), and is the worked example of the real-sync-vs-record-deferred trade in the wild.

Alternatives considered

• Keep fixing instances by hand as they surface (status quo post-ADR-0057). Rejected — the class re-appears (the 1,377-call audit found several); hand-fixing is reactive and each instance ships a production lie first.
• Layer 1 only (static guard, no bench eval). Rejected as insufficient alone — a handler can be shaped correctly (returns a delivered field) while the integration behind it 404s; only a real-bench eval (Layer 2) catches that. Layer 1 is necessary but not sufficient.
• Layer 2 only (daily eval, no static guard). Rejected — a daily eval catches the regression up to 24h after it merges; the cheap PR-time fence (Layer 1) should stop it from merging at all. Belt and suspenders.
• Make everything real-sync (no deferred class). Rejected — it forces synchronous side effects into the call turn (latency on voice) for work that genuinely belongs post-call, and it contradicts the maestro architecture already in place for maintenance (ADR-0031). The point of the taxonomy is that deferred work should mostly not be a tool, not that it should be a synchronous one.

Steps to reproduce (the class this ADR closes)

In production (the ADR-0057 incidents, real callers, test data excluded): an inbound voice call routes to a sibling whose tool promises a human action; the tool records a row or returns a success string but performs no notification/transfer/send; Clara tells the caller it's handled; no human is reached. Pinned historically by src/__tests__/unknown-caller-promise-delivery.repro.test.ts + unknown-caller-escalate-no-transfer.repro.test.ts.

In the test suite (what each layer would add):

• Layer 1 — a drift test that, given a synthetic handler returning a promise string with no delivery field and no real-sync class, fails (RED) — proving the fence bites.
• Layer 2 — a voice-tool fixture in evals/run-tools-eval.ts that fires escalate_to_human against appfolio-45 and asserts a real emailDelivered:true (or the page receipt), failing if the integration is down.
• Layer 3 — a reviewer-lens line item the bot cites when a PR adds an agent tool without an effect-class or with a hardcoded success string.

Out of scope here (separate tickets): the re-shaping of capture_unknown_caller_note from real-sync to maestro-deferred; the parallel promise-without-delivery gaps the ADR-0057 audit surfaced in other flows (lost work orders, unsent renewal texts, issueType misclassification).