Answer anything on file

Decision doc · Accepted 2026-08-03 · companions: deposit postmortem · channel-split decision

Decisions (Fede, 2026-08-03): Q1 = A, schema-derived injected knowledge block. Q2 = no output guard — rely on injection + generated evals (risk accepted: no runtime check stands between a small model's bad day and an outbox; the gauntlet and replay corpus are the containment). Q3 = straight to general — the narrow deposit-fix worktree is retired.

Ship gate (Fede): hard proof that the new solution is better than current prod — on all eligible real past conversations plus synthetic data — before anything ships. Protocol below; nothing merges until the proof artifacts are attached and reviewed.

The ask (Fede, 2026-08-03): stop fixing this one deposit field. Make Clara able to answer anything that's in the property knowledge base on email/SMS — and route anything that isn't — without hand-writing every fact into a tool description. Three premises are taken as direction, not open questions:

General, not per-field. The fix must cover every knowledge-base fact, current and future — not "deposit" as a special case.
Email/SMS optimize for correctness, not latency. The tool-split that started this was a voice-latency fix. Text channels have seconds of budget; spending tokens and a lookup on being right is the correct trade.
No hand-enumerated tool descriptions. Anything a human must remember to update when a schema grows is a future incident.

What the knowledge base actually holds

Block	Shape	Examples	Reaches email today?
`pricingDetails`	~19 structured fields + `customFees`	securityDeposit, petFee/Deposit/Rent, adminFee, applicationFee, parking, storageFee, holdingDeposit, transferFee, mtmFee, earlyTerminationMultiplier, nsfFee, utilitySetupFee, unauthorizedPetFee, keyFobDeposit, lateFee	No — only via tools the model may not call
`sections`	free-text policy cards (doc-ingested, human-confirmed)	Pet Policy, Smoking, Laundry, Lease Terms	Yes — rendered as PROPERTY POLICIES
`concessions`	free text	"1 month free on 12-mo leases"	Yes — LEASING SPECIALS
hours / links / phones	structured	office hours, application link	Yes

So the gap is precisely the structured facts block — the one with the dollar amounts, where being wrong hurts most. Voice already injects it ({{pricing_fees}}, INJECT-4); email deliberately excludes it from the prompt and reaches it only through tool selection.

In plain terms

Clara's email brain already gets handed the property's policies, specials, and office hours on a card. The fee sheet — the thing prospects ask about most — was left off the card, and she was told "go look it up if you think you need to," with a catalog whose labels didn't say where the deposit was. The fix is to put the whole fee sheet on the card, generated from the database so it can never go stale, and mark every blank line as "not on file" so she knows when to hand off to a human instead of guessing.

Q1 — How should knowledge-base facts reach the model on email/SMS?

A. Schema-derived knowledge block, injected every turnRecommended

One renderer walks the pricingDetails schema (plus sections, concessions, hours) and writes a PROPERTY KNOWLEDGE block into the email/SMS system prompt. Fields with values are listed ("Security deposit: $300"); fields without are listed as "not on file". A new schema field shows up in the prompt automatically the day it's added — no description to update, no tool to re-catalog. Tools remain only for genuinely volatile data: unit availability, tour slots, term pricing. This is exactly the architecture that kept voice correct through the whole incident, extended to text.

Cost: the structured-fee block is ~20 short lines — roughly 150–250 tokens (estimate; measured with the real tokenizer before ship) on an email/SMS system prompt already around 6–8k tokens (the ~630-line leasing prompt + policies + specials already injected today) — a ~2–3% increase. The block is stable per property, so prompt caching absorbs most of the marginal cost across turns. If customFees/sections ever balloon, the renderer caps the block and falls back to a lookup tool for the tail — bloat has a designed escape hatch.

B. Keep the pull model; generate tool descriptions from payload schemas

Descriptions become build artifacts derived from what handlers return, so they can never under-describe again. Least change to the runtime. But retrieval stays model-mediated — the model must still choose to look, and a prompt rule can still blind it, which is what actually happened (RULE 7.0 mattered more than the description gap).

C. One lookup_property_fact tool, force-called before every leasing reply

Deterministic pre-fetch: code (not the model) runs the knowledge lookup at turn start and attaches the result. Equivalent grounding to A, but adds a round trip and a new mechanism where A reuses the proven injection pattern. Worth it only if the knowledge block ever grows too large to inject wholesale.

Q2 — How do we enforce "only cite what's on file"?

Injection makes correct answers possible; it doesn't stop a small model from inventing a number anyway (Haiku fabricated "$1,350 = one month's rent" under the always-route rule). The hallucination guard already verifies dollar amounts against renewal offers — the same pattern can verify leasing replies: any dollar amount / numeric fact in an outbound email/SMS leasing reply must trace to the injected knowledge block or a tool result from this turn.

Hard gate from day oneRecommended

An unsourced figure blocks the send; the turn re-generates once with the violation named, and if it recurs the reply routes to the team. Matches the rip-the-bandaid preference. Scope: leasing replies on text channels only — renewal/work-order numbers keep their existing guards, so legitimate figures from other subsystems aren't caught.

Shadow first

Log violations for a period, review false-positive rate, then flip to blocking. Safer against over-blocking legitimate composed math ("$300 deposit + $38 app fee = $338"), at the cost of a window where fabrication is possible.

No output guard — rely on injection + evals

Cheapest; leaves nothing at runtime between a small model's bad day and a prospect's inbox.

Q3 — Does the held deposit fix ship first, or fold into this?

The narrow fix (RULE 7.0 deposit clause + tool description + eval cases 13/13b) sits proven but unshipped in worktree propflowai-deposit-quote, with one Haiku tour-pushiness regression open. The general fix replaces most of it: the injection block supersedes the description edit, and a general "answer what's on file, route what isn't" rule supersedes the deposit-specific clause.

Go straight to the general fix; retire the narrow worktreeRecommended

One coherent cutover: injection + general prompt rule + guard + generated eval gauntlet, one PR. The deposit incident becomes one generated case among many. Avoids shipping a deposit-shaped rule we'd delete weeks later, and avoids burning an iteration on the narrow fix's Haiku regression. Deposit asks keep escalating (current behavior) until it lands — tolerable at observed volume (1 avoidable escalation in 30 days).

Ship the narrow fix now, general fix behind it

Stops deposit escalations this week; costs finishing the narrow fix's regression work and a second prompt churn when the general rule replaces the deposit clause. Two prompt changes for Haiku to wobble through instead of one.

The eval that makes this stick — generated, not hand-written

"Deposit ON FILE → quote it" was one hand-authored case; the general fix gets a generated knowledge-grounding gauntlet:

For every field in the pricingDetails schema, two cases are generated automatically: value on file → the reply must cite it; value absent → the reply must route, and must not contain any invented figure. Add a field to the schema and its test cases exist the same day — the eval can no longer lag the data model.
Section-content probes: questions answerable from each ingested policy section → answered from the section, never contradicted.
The always-route floor stays tested: deposit assistance / vouchers / qualification-dependent asks route regardless of what's on file (the fair-housing core is untouched).
Both model tiers, repeat-sampled (3×), on the subscription runner — the Haiku wobble class is caught by design, not by luck.
A replay arm: the existing evals/replay-harness/ (real prod conversations through the real loop, parity + judge) runs baseline-vs-candidate over the 117-conversation corpus before the cutover PR merges — the strongest available evidence that nothing else moved.

What this closes structurally: the incident's three root causes each lose their mechanism. Descriptions can't under-describe (nothing behavioral lives in them); channels can't diverge on facts (both read the same rendered knowledge); and the eval can't lag the schema (cases are generated from it). The remaining human-authored surface is the route-vs-answer policy — one rule, stated once.

Proof protocol — the ship gate

Two arms, both run against the candidate build and current prod (baseline), same inputs, same graders. The PR does not merge until both pass and Fede has reviewed the artifacts.

Arm 1 — every eligible real conversation we've ever had

The existing evals/replay-harness/ runs the full eligible corpus — 117 real customer email/SMS conversations, 291 inbound turns — through the real inbound loop, baseline vs candidate (~640 model calls per arm).
Pass bar: every diverged turn is classified. Divergences must fall only in the intended class (a knowledge-answerable question that prod routed, refused, or answered without grounding). Each intended divergence must be judge-approved and hand-reviewed; zero unintended divergences — identical replies and identical tool-call sets everywhere else.
Known fidelity limit, stated up front: replays run as a synthetic unknown sender, so tenant-identity-dependent turns are weaker evidence; those diverged turns get mandatory hand review rather than judge-only.

Arm 2 — synthetic, generated from the schema

The generated knowledge-grounding gauntlet: every pricingDetails field × on-file ("must cite it") and absent ("must route, must not invent") × Sonnet and Haiku × 3 repeats — roughly 230 graded runs, on the subscription runner.
The existing 25-case leasing-concerns gauntlet, both models, 3 repeats, baseline vs candidate: no regression on any currently-passing case (tour-pushiness and fair-housing routing explicitly watched); known-flaky cases compared by pass-rate, not single samples.
Because Q2 = no output guard, the fabrication cases are the only line of defense — they are blocking, not advisory.

Proof artifacts (parity report, divergence review, gauntlet JSONs both arms) attach to the cutover PR. This PR does not auto-merge regardless of green — Fede reviews the proof.

Team-side follow-through

"Not on file" becomes visible. Every field the renderer marks absent is a fact a human will be asked to answer by email forever. A per-property completeness view (Yale currently has no security deposit on file at all) turns each blank into a 2-minute data-entry task that permanently deletes an escalation class.
Escalation queue owner — unchanged from the postmortem: routes that remain legitimate (not-on-file, assistance, qualification) still need a response-time owner.

Evidence base: prompt/tool paths verified in-repo 2026-08-03 (context builders conversation-manager.ts:3160/3348, voice injection leasing-context-injection.ts:371, no forced tool_choice or fee router anywhere in the conversation loop). Corpus and replay figures from evals/replay-harness/ README census 2026-08-02. Token-cost figure is an estimate until measured.