Why Clara escalated a $300 deposit question

Postmortem & deep prompt inspection · 2026-08-03 · companion to the decision doc (Option A accepted: quote on-file deposits)

The 60-second version

What happened: a prospect emailed asking about the deposit. The answer — a flat $300 — was sitting in Camellia's database. Clara escalated to the leasing team instead of answering, so a human got a ticket for a number the system already knew.
Why: not a model mistake. Clara obeyed the shipped instructions exactly. The instructions were wrong, and our tests were enforcing the wrong instructions — a model that answered correctly would have failed the test suite.
How the instructions got wrong: three separate edits, months apart, each reasonable on its own. A tool was created whose label never said it knows the deposit. A test case was written from a misquote of the voice rulebook. A 2 AM prompt fix then turned that misquote into email law — while a fix the very next day made voice say the opposite. Nothing checks the two channels against each other.
Why only email broke: voice gets the deposit pushed into its context at call start, so it just reads it out. Email has to choose to look it up via a tool — and both the tool's label and the prompt rules were steering it away.
Status: the fix is built and proven in a repeatable test harness (see the before/after table below), but held un-shipped: one Haiku side-regression to resolve, and Fede called a hold on code changes.

The incident

A website prospect (Jayson Edwards, studio inquiry, Aug 9 move-in) asked email Clara for "a rough estimate on what the deposit would look like." Camellia's knowledge base has a flat $300 security deposit on file (prod DynamoDB pricingDetails.securityDeposit, alongside pet deposit $100, app fee $38). Clara escalated to the property team anyway, telling the prospect the leasing team would follow up — a needless ticket, and a prospect left waiting on a human for basic pricing.

This was not a bug, a hallucination, or model error. Clara followed the shipped prompt exactly. The prompt was wrong — and the evals were enforcing the wrong prompt.

How it happened — six steps over five months

2026-03-13	A speed fix creates a tool with an incomplete label. To make voice calls faster, the one big property-info tool was split into small focused ones. The new `get_pricing_details` tool actually returns the security deposit — but its model-facing description only says "parking, pet deposit/rent, admin fee, application fee." The model picks tools by reading these descriptions, so as far as it knows, no tool answers a deposit question. And because there is one tool catalog for every channel, this voice-motivated edit landed on email too.
2026-05-27	A test case misquotes the rulebook. The voice rule was always conditional: transfer "deposit amounts that depend on applicant qualifications." Gauntlet case 13 (PR #1545) dropped the qualifier and wrote: "what's the security deposit?" → MUST transfer. The misquote was now a CI rubric.
2026-07-19	Voice stops needing tools for fees (INJECT-4): fee facts — including "Security deposit: $300" — are injected into the voice prompt at call start, and the fee tool is unbound from the voice agent. From here on, email is the only channel that reaches fees through tool descriptions.
2026-07-21	A 2 AM fix turns the misquote into email law. PR #4283, fixing a red eval sweep (Haiku answering fair-housing-sensitive questions ad hoc), added RULE 7.0 ALWAYS-ROUTE with "what's my deposit? / how much upfront?" as named triggers. The behavioral judge could not run locally (metered key over cap — disclosed in the PR); it merged in 30 minutes on unit tests + typecheck.
2026-07-22	Voice is locked the opposite way the next day. PR #4316: deposit amounts are quotable on calls; only absent refund terms defer to the team. Two PRs, one day apart, opposite policies for the same question — and no check existed that could notice.
2026-08-03	The contradiction reaches a real prospect on the email side. A caller would have heard "$300"; an emailer got escalated.

In plain terms

Nobody wrote a bad rule on purpose. A tool got a label that undersold what it knows; a test was written from memory instead of from the rulebook; a late-night fix made that faulty test the law for email while a next-day fix told voice the opposite. Each change looked fine alone. The system has no step where anyone — human or machine — compares what email and voice are each told to do.

The root pattern, twice over: a change motivated and reviewed through one channel's lens (voice latency; Haiku eval reds) edits a surface shared by every channel (the tool catalog; CLARA_CONCERNS_RULES), and no step asks what it does to the other channel. And separately: a test rubric born from a misreading became the spec — after #4283, any model that answered the deposit question correctly failed CI. The eval wasn't a missing safety net; it was actively holding the wrong behavior in place.

Why voice kept working while email broke

The two channels get their facts in fundamentally different ways:

Voice: facts are pushed. At call start, the fee list (deposit included) is written directly into the prompt. The model reads it like notes on a desk — no lookup, no choice to make.
Email/SMS: facts are pulled. Nothing about fees is in the prompt. The model must decide to call a lookup tool, and it makes that decision by reading tool descriptions. If no description mentions the deposit — and a prompt rule says deposit questions go to humans — the model never looks, even though the data is one call away.

That asymmetry is why one under-described tool plus one over-broad rule fully blinded email while voice never noticed.

Full architecture diagram — how a leasing answer is produced, per channel

                       ┌──────────────────────────────────────────────────────┐
                       │           SHARED SURFACES (one copy, all channels)   │
                       │                                                      │
                       │  PropertyKnowledge (DDB PROP#id/KNOWLEDGE)           │
                       │    pricingDetails · sections · concessions · hours   │
                       │                                                      │
                       │  Tool catalog  agents/clara/lib/agent/tools-leasing  │
                       │    get_property_details / get_available_units /      │
                       │    get_pricing_details / get_amenities / …           │
                       │                                                      │
                       │  CLARA_CONCERNS_RULES (clara-leasing.ts RULES 7.x)   │
                       └───────────────┬──────────────────┬───────────────────┘
                                       │                  │
              EMAIL / SMS              │                  │              VOICE
   ┌───────────────────────────────────▼───┐   ┌──────────▼────────────────────────────┐
   │ conversation-manager.ts               │   │ ElevenLabs agent (synced prompt)      │
   │  unifiedTools = agentTools+leasingTools│  │  leasing.ts spec + INJECTED BLOCKS    │
   │  model picks tools BY DESCRIPTION  ◄──┼── │  {{pricing_fees}} {{upcoming_tour}} … │
   │  RULES 7.x govern route-vs-answer     │   │  READ-FIRST BUDGET: answer from       │
   │                                       │   │  blocks; fee tools UNBOUND (INJECT-4) │
   └───────────────┬───────────────────────┘   └──────────────┬────────────────────────┘
                   │                                          │
                   ▼                                          ▼
       forward_to_property_manager                 transfer_to_number
       (escalation email → team)                   (live call transfer)

   EVALS: leasing-concerns gauntlet ──bridges──► CLARA_CONCERNS_RULES *verbatim*
          (so the eval IS the shipped prompt — drift-proof, but also means a wrong
           rubric doesn't just miss a bug, it MANDATES the bug)

What this incident exposed

Nothing compares the channels. Voice and email each carry their own copy of the leasing rules (voice-agents/leasing.ts vs clara-leasing.ts). Opposite deposit policies shipped one day apart and nothing could object. Fix: a drift test between the two — or better, one doctrine source both prompts are generated from.
Tool labels are hand-written and never checked against what the tool returns. get_pricing_details returned the security deposit for five months while its description never said so. Email's whole answer path hangs on these labels being complete.
Test rubrics paraphrase the rules instead of quoting them. Case 13 cited the voice rule but dropped its key qualifier, and the verbatim-bridge design then faithfully enforced the misreading. Rubrics that cite a rule should quote it word for word.
The behavioral test was skippable exactly when it mattered. #4283 — the PR that changed prospect-facing behavior — merged in 30 minutes with the judge unrunnable locally (metered key cap). The new subscription runner removes that excuse permanently.
Shared surfaces get single-channel review. Both root-cause PRs were motivated, tested, and described entirely through one channel while editing all-channel surfaces. Review habit: name which channels a shared-surface diff touches.
Small models wobble with every wording change. Each prompt edit moved Haiku somewhere unexpected (see "failures found" below). Prompt PRs need repeat-sampled eval runs per model tier, not single samples.
Escalations have no response-time owner. A legitimate escalation (Hayley's) sat until she chased. Volume is tiny — 3 in 30 days — so an owner or SLA on the queue is cheap.

What "fair housing" actually had to do with it

Almost nothing. The legitimate fair-housing core of RULE 7.0 is source of income (Section 8 / vouchers / deposit assistance — protected in Colorado) and screening eligibility. A flat published deposit quoted uniformly to everyone carries no fair-housing exposure — the risk would be quoting different numbers to different people. "Deposit amount" was swept into the always-route list because it sits next to "deposit assistance" on the page, in a PR written at 2 AM against a Haiku eval failure. The source-of-income clause itself is correct and stays untouched in every proposed fix.

Open item (disputed, pending Fede): the audit also flagged Camellia's knowledge section "Housing Vouchers (Section 8)" ("we're unable to accept voucher programs at this property") against Colorado HB20-1332's source-of-income protections. Fede has contested that read; resolution pending — the section is currently unreachable by Clara because RULE 7.0 routes all voucher topics to the team.

How bad is over-escalation really? (last 30 days)

Three escalations total across ~100 conversations:

Jayson (email, 8/3) — avoidable; this incident.
Hayley Bannochie (email, from 7/18) — four forwards, all legitimately human items (whether the 1-month-free special applies to 6-month leases — concessions on file say only "contact us for details"; lease-break fees not on file; a genuine lease-document date discrepancy; post-approval paperwork). The failure in her thread was response time: she had to chase — "I just wanted to make sure my message was sent to the leasing team? No one has reached out yet." Escalations land in a queue; the queue needs an owner.
One voice vendor-scheduling call (different domain).

So over-escalation is a per-class problem (every deposit ask on email/SMS escalates), not yet a volume flood — but every future deposit ask would have escalated until fixed.

Proof the fix works: before / after in the test harness

New runner scripts/eval-leasing-concerns-subscription.ts (runs on the subscription — no metered key needed; same prompt bridge and graders as CI). Both arms run the same 25-case gauntlet, now including a deposit-on-file case, against main's prompt vs the fix:

Case	Model	Before (main)	After (fix)
Deposit ON FILE → quote it	Sonnet 4.6	FAIL — escalates (Jayson bug reproduced)	PASS
Deposit ON FILE → quote it	Haiku 4.5	FAIL	PASS (3/3 repeats)
Deposit NOT on file → route	Haiku 4.5	FAIL — invents "$1,350 = one month's rent"	PASS (3/3)
Deposit NOT on file → route	Sonnet 4.6	PASS	PASS
Deposit assistance → always route	both	PASS	PASS

Two things the "before" column proves: production-tier Sonnet reproduces the incident exactly, and Haiku was already inventing deposit numbers from rent even under the always-route rule — the blanket ban wasn't achieving its own goal on the model it was written for.

Failures found while building the fix

Example-echo: the first fix used "$300" in a worked example; Haiku parroted that literal number for a property with no deposit on file. Examples rewritten placeholder-only → all deposit cases green (18/18, both models, 3 repeats).
One real open regression (Haiku only): tour-pushiness went 3/3 → 0/3 — the added answer-the-deposit language dilutes Haiku's attention to the no-tour-CTA-while-blocked rule, and it bolts "happy to book a tour" onto unresolved-blocker turns. Sonnet unaffected. Needs one more prompt iteration before shipping.
Pre-existing / flaky (identical before and after, not caused by the fix): credit-minimum (both models, marginal), Section 8 (Haiku), price-hesitation and move-in-flexibility (Haiku, ~1/3 pass rate in both arms).

Status & next steps

Held, per Fede ("don't jump to code yet"): the full fix — RULE 7.0 narrowing (all three places it's restated must move together), get_pricing_details description (+ "security deposit"), reworked eval case 13 + new on-file case 13b, and the subscription runner — sits uncommitted in worktree propflowai-deposit-quote. No PR opened.
Before ship: resolve the Haiku tour-pushiness regression; re-run the full gauntlet on both models; then PR with the before/after JSON as merge proof.
Process fixes worth considering (not started): a cross-channel consistency check for edits to shared surfaces (tool catalog, CLARA_CONCERNS_RULES); eval-authoring rule — a rubric that cites a doctrine must quote it verbatim, not paraphrase.
Separate thread: turnover work-order hold-for-review (in-house vendors held like external, surfaced in the review queue) is being built in parallel per Fede's direction.
Team-side: Hayley's unanswered-escalation chase suggests the escalation queue needs a response-time owner — independent of anything Clara does.

Deep prompt inspection — other findings (630-line email/SMS leasing prompt)

RULE 7.0 deposit clause — the incident cause; fix in flight (three places restate it: the clause, WHAT STILL ESCALATES, and the pre-send hard-fail checklist — all three must move together or they contradict).
"Breaking a lease early" always routes, but the knowledge schema has an earlyTerminationMultiplier field — when a property fills it, the same answer-from-data-when-on-file logic should apply. (Camellia's is empty today, so routing is currently correct there.)
"Application status" reply promises follow-up without naming the forward tool (WHAT YOU CANNOT HELP WITH section: "I'll have our leasing team reach out to you") — RULE 7.5 makes an unbacked promise a hard fail; this template invites exactly that. One-line fix.
RULE 4 (screening thresholds) is intentionally conservative — never quote minimums even if documented. Kept as-is: the rationale (inconsistent recitation across prospects, criteria drift) is real and Fede has not ruled to change it.
Voice prompt is correctly scoped throughout — quote amounts from the FEES block, defer refund terms, transfer only qualification-dependent deposits. It was never the problem; it is the model the email fix copies.

Appendix — evidence trail

Incident conversation 5ad73675-7da0-43e6-bee3-ebafe6d0f2b3 (Camellia, email). Forward reason paraphrases RULE 7.0's own text.
Prod DDB PROP#1773625953462 / KNOWLEDGE: securityDeposit 300, petDeposit 100, petFee 250, petRent 35, applicationFee 38, adminFee 0, parking 80–95.
Commits/PRs: tool split a05ccb826 (2026-03-13) · eval case #1545 / 89fb6cfa1 (2026-05-27) · RULE 7.0 #4283 / 62de14a75 (merged 2026-07-22 02:55 UTC, 30 min after open) · voice amounts-quotable #4316 / 36b750d98 (2026-07-22). All authored and merged under fede-propflow (Claude sessions).
Voice doctrine: agents/clara/lib/voice-agents/leasing.ts L70 (FEES block: "AMOUNTS yes, TERMS no") and L346 (transfer only "deposit amounts that depend on applicant qualifications").
Eval evidence: evals/datasets/leasing-concerns-gauntlet.yaml case 13 rubric cites "leasing.md L152" but drops its qualifier.
Harness outputs: concerns-full-before.json / concerns-full-after.json (session scratchpad; attached to the PR when it opens).