Application-Link Incident & the Tool-vs-Context Decision
What happened (all verified against the production trace)
send_application_link tool. The loop ships only the final round's text; the second round was a 44-token wrap-up ("The application link has been sent to you…"). The prospect got the wrap-up; the answer was discarded. Discard site pinned in conversation-manager.ts (text alongside a tool call is kept only as a fallback for an empty final round). Same code path serves email, SMS, and voice. A red regression test reproducing this exactly is pushed on branch fede/loop-text-drop-repro.
POLICY_GATE_MODE unset). See the Honesty Layer deep inspection — this incident is a third real-customer data point for its thesis.
The finding that reframes the fix
Clara's system prompt and the tool have been contradicting each other since July 19. The prompt line (written July 1 with PR #2756, when the tool really did send its own message) still orders: "call send_application_link — NEVER paste the link above into your reply as text… After calling it, just tell the prospect it's on the way." The July 19 redesign (#4152) changed the tool to send nothing and instruct "include the link inline" — but this prompt line was never updated. On every explicit application ask since, the model has been told two opposite things; in this incident the system prompt won. Clara followed her instructions. Session trail: the tool's origin session (July 1, PR #2756 — moved the link out of plain prompt context, where it had lived since May 20 #1240, because inline pasting couldn't stamp the dedup flag and the post-tour auto-sender double-sent), and the July 19-20 redesign session (Carly Sprague "already sent" incident → PRs #4147/#4152/#4153/#4168).
The application link is already in Clara's email/SMS prompt context on every turn. The tool fetches nothing the model lacks — it reads the same leasing settings the prompt is built from. The tool's only remaining side effect (a 24-hour "already shared" stamp on the prospect) has exactly one reader: the function that writes it.
Origin verified: the tool was never built for phone callers (no voice binding exists; the voice agent has no such action). The remembered "text a link to a caller mid-call" feature is the separate virtual-tour-link tool (ADR-0104).
Decisions
D1 — The tool in email/SMS lanes
Owner concurrence 2026-08-19: the "don't paste, call the tool" prompt line is "probably an outdated instruction and makes no sense."
A. Remove it; let Clara include the link from context. Delete the "don't paste, call the tool" prompt line; retire the tool from email/SMS (post-tour auto-sender untouched). Kills the failure mode structurally and saves a full model round-trip. The dedup stamp has no other consumer, so nothing is lost. Ships under the same testing gate as D2 (this is a prompt-behavior change with production blast radius).
D2 — Reply assembly becomes an explicit, tested contract (the root architectural fix; needed regardless of D1)
Owner direction 2026-08-19: no hacks or quick fixes — a comprehensive solution. The flaw is that the agent loop has no defined contract for what constitutes "the reply" when the model produces text across multiple rounds; today's behavior (keep only the final round, stash earlier text as a fallback) is an implicit convention that silently loses composed answers on every channel.
A. Design and ship an outbound-composition contract for the agent loop, one seam, all channels. Scope: (i) a written contract in the loop — every customer-visible text block the model emits across rounds is part of the drafted reply unless explicitly marked internal; assembly is one deterministic, unit-pinned function, not scattered extraction; (ii) the model's prompt documents the same contract (final message stands alone; on email never announce a separate send — links inline), so model and harness agree on the rules; (iii) the trace records the assembled draft and what each round contributed, so future incidents are inspectable; (iv) migration proof, not vibes — see testing gate below. The existing red repro test becomes one case in a new assembly test family (multi-round text, text+tool+wrap-up, empty final round, tool-error rounds, multilingual).
Relationship to the honesty-layer redesign: this is upstream of it and complements it — the PolicyGate inspects the composed reply, so composition must be correct first. Ship as its own reviewable unit; do not bundle with gate work.
Testing gate for whichever option (owner requirement, 2026-08-19): full golden eval suites (promptfoo conversation + leasing sets), the multi-turn eval lane, ≥30-day real-conversation replay per the guard-change merge rule (this changes customer-visible text assembly), Willows harness proof, and the red repro test green — all on the subscription runner. No merge on unit tests alone.
D3 — Per-tool URL safety net (interim symptom control only)
Reframed after owner review: a per-tool "reply must contain the URL" check treats the symptom and adds to the guard scatter the honesty-layer redesign wants to retire. The root fixes are D2 (reply assembly) and D1-A (fact in context, no tool round-trip). Claims-vs-reality checking in general belongs in that redesign's single checkpoint, not per-tool rules.
A. Skip the standalone guard; rely on D1-A + D2. If the honesty-layer redesign's increment 1 lands, "reply discussing applying must carry the configured link" becomes one deterministic policy in its registry — add it THERE, with repair-not-block semantics.
D4 — Voice callers who ask about applying (DECIDED 2026-08-19)
No mid-call tool. The offline pipeline sends the link after the call: when post-call processing detects the caller asked to apply / wants the application, the existing application-link sender delivers it to the prospect on their channel (SMS with consent gates / email), deduped against the post-tour automation. NOT built yet — today a voice caller who asks to apply gets no link at all. Implementation queued.
D5 — Classifier hardening
A. Add a prompt rule + eval case: teach the topic labeler to see through automated notification wrappers (AppFolio/Zillow "New Lead" emails) to the human ask inside, and add that shape to the golden eval set (today: zero coverage). D2 also restores the leasing signal these threads normally carry.
D6 — Concession mechanics volunteered on every email (owner-corrected 2026-08-19)
Clara's (discarded) draft recited the free-month mechanics ("the free month is the month after your move-in month, prorated if mid-month…") verbatim from the copy PR #5553 (Aug 11) appends to every rendering of the special. Owner correction: the 2026-08-06 standup discussion happened because someone asked when the month applied — it is an internal mechanic and an on-request answer, not something to state on every email.
A. State the special simply in the always-on specials block ("one month free on new 12-month leases"); keep the full mechanics copy ONLY in the ask-path grounding (the lease-answer block that fires when a prospect asks how it works, which already exists) and the voice spoken rail's one-sentence form. Same testing gate as D1/D2.
Also surfaced (bug cards, not decisions)
| Finding | Detail |
|---|---|
| Stale-label clear may be broken | stampConversationSubtopic(…, null) threw a DynamoDB validation error ("ExpressionAttributeValues must not be empty") when run from a script — if it reproduces in the real workflow, threads upgraded from no-topic keep their stale hidden-label. Unconfirmed in prod; needs a check. |
| Two older lead threads with odd labels | One April lead thread labeled maintenance; one August leasing thread sub-labeled "unrelated". Minor; eyeball when convenient. |
clara_reply kind missing from outbound topic-stamp map | Every clara_reply logs a spurious "untagged kind" warning (no incident impact). |
Method: prod trace + DynamoDB reads; 3 background agents (guard architecture, classifier repro ×5 with live model, 2,698-conversation sweep) + a 4-agent dynamic workflow (tool git archaeology, channel map, past-session search, synthesis). Every number above traces to a cited source in the session log. Related: Honesty Layer deep inspection.